diff --git a/Documentation/PCI/controller/apple-t8140.rst b/Documentation/PCI/controller/apple-t8140.rst index ec0d67dfc594c7..496e9b94d40f60 100644 --- a/Documentation/PCI/controller/apple-t8140.rst +++ b/Documentation/PCI/controller/apple-t8140.rst @@ -1,89 +1,39 @@ .. SPDX-License-Identifier: GPL-2.0-only T8140/J700 experimental radio PCIe bootstrap -============================================ +========================================== The J700 path trains port 0, validates a bounded PIODMA bootstrap request, and enables native ECAM for the MediaTek Wi-Fi and Bluetooth functions. The root bridge forwards their BARs and DART translates endpoint DMA. -This is a bring-up implementation, not general T8140 PCIe support. Build -``CONFIG_PCIE_APPLE_PIODMA_DIAG=y``; the Apple PCIe host driver may be a module. -The bootstrap enables enumeration by default only on machines matching both -``apple,j700`` and ``apple,t8140``, through the matching diagnostic and host -device-tree nodes. No kernel argument is required. To leave the root-only -diagnostic path closed, pass ``pcie_apple_piodma_diag.enumerate=0``. -The read-only ``enumerate`` parameter reports the configured admission request, -not whether the current machine is a Neo or its radios were admitted. Its -default value is also visible on other Macs; the machine and DT checks decide -whether it has any effect. - -The shared kernel can keep suspend, hibernation and kexec support enabled. -Before the host creates PCI children, a short admission critical section -acquires a kexec interlock and registers a PM notifier that refuses sleep -transitions. Its device prepare callback provides a second veto. This also covers -``/dev/snapshot`` files opened before the supplier probes: later image, restore -and suspend ioctls enter device PM without repeating the prepare notifier. -A kexec interlock refuses loading or executing a replacement kernel, -including an already loaded crash kernel. Image unloading remains permitted. -These interlocks apply only to the active J700 bootstrap; other machines do not -acquire them. Merely probing its supplier does not block transitions. The device -prepare callback also checks this guarded pre-publication window. Admission -contention may defer only before PCI children exist; a later failure is a hard -error. Pre-child contention schedules a delayed attachment of this matched -host, so a long-running kexec loader does not require another driver to bind -before admission can retry. Only renewed contention queues another attempt; -successful admission and hardware publication end these retries. Shutdown -blocks late admission. A failure before memory publication releases the guards. Once the -bootstrap retains memory, the protection persists even if enumeration or either -radio's firmware loading fails. Thus the block applies on every default J700 -boot which reaches retained radio admission, regardless of installed firmware. -Failed or uncertain bootstrap state requires a full external hardware reset; -there is no command retry. - -Sleep is not supported on the MacBook Neo yet. With radio support active -(the default), the kernel refuses suspend: lid close and ``systemctl suspend`` -do not put it to sleep. Shut down before putting it in a bag. Booting with -``pcie_apple_piodma_diag.enumerate=0`` removes this refusal and disables Wi-Fi -and Bluetooth; sleep and wake remain unproven on this hardware without radios. - -Before either radio driver can bind, the host disables ASPM, all L1 substates -and Clock PM on the validated radio endpoints' shared PCIe link through the -PCI core. ``CONFIG_PCIEASPM=y`` is required. The global ASPM policy is unchanged; -neither a performance-policy boot argument nor a sysfs policy write is needed. -Booting with ``pcie_aspm=off`` prevents the PCI core from applying this restriction -and radio admission fails closed. - -The supplier publishes radio admission only after both functions pass the -host's identity, cold-state, memory-window and link-power checks. Both radio -probes require that completed admission before accessing PCI or MMIO state; -the host's downstream device-enable callback enforces the same gate for other -drivers. This prevents a later generic rescan from enabling cached children of -a failed admission. Each radio probe reapplies the link-power restriction, so -a recreated ASPM link state cannot inherit an incompatible global policy. -Privileged writes to the per-link PCI sysfs ASPM or Clock PM attributes can -override the restriction; it is reapplied only at probe, not continuously. -Do not enable those link power states while the Neo radios are admitted. - -The PIODMA arena remains allocated until external reset. Standard PCI sysfs -``remove`` writes for the root port and radio functions return ``-EOPNOTSUPP`` -while the host has its retained supplier; the removal attributes remain present. -The same hierarchy refuses userspace ``reset`` and ``reset_subordinate`` -writes, and its host does not hand native PCIe or SHPC slot hot-plug control -to Linux. These restrictions leave other PCI hosts unchanged. -VFIO assignment is refused on this host because it exposes reset operations -outside PCI sysfs. Automatic native AER/DPC port recovery is also withheld: -its bus or link reset has no qualified retained-memory contract here. A hardware -fault requires external reset rather than guessed recovery. -The host and activated Wi-Fi/Bluetooth modules are pinned until reset. Arbitrary -platform-device removal, IOMMU teardown, forced module removal, memory reuse -and arbitrary downstream devices remain unqualified. Do not remove the -controller or its IOMMUs while this experiment is active. +This is a bring-up implementation, not general T8140 PCIe support. +Build ``CONFIG_PCIE_APPLE_PIODMA_DIAG=y``. The board device tree selects the +bootstrap by pointing the host's ``apple,piodma`` property at the engine, and +the kernel then runs it during the host probe. Passing +``pcie_apple_piodma_diag.enumerate=0`` keeps the root-only diagnostic path, +which rejects downstream configuration access. A kernel built without the +option always uses that root-only path. +Build the host and its DART in as well: the supplier retains published DMA +and has no removal contract. Keeping the supplier and host built in avoids +a module dependency on a provider whose lifetime cannot end at unload. + +Suspend-to-idle suspend retains the arena after checking the completed request, +idle FIFO and command state. Resume restores the existing configuration +banks without issuing a second bootstrap. Hibernation, deeper suspend and +kexec are refused while admission or retained DMA ownership is active. The +runtime kexec guard permits the kernel to support kexec on other machines. +Host removal, sysfs reset, hotplug, VFIO assignment and native AER/DPC +recovery are withheld on this retained hierarchy. Sleep is refused until +radio admission completes; the existing suspend-to-idle path remains available +after successful admission. Failed or uncertain bootstrap state +requires a full external hardware reset; there is no command retry. + +The PIODMA arena remains allocated until external reset. Controller removal, +teardown, memory reuse and arbitrary downstream devices are unqualified. +Fresh hardware qualification must check both radio functions across sleep. +Do not remove the controller or its IOMMU while this experiment is active. These lifecycle contracts require further work before production support. -They protect against accidental teardown and are not a security boundary. -Root can still reset through raw configuration writes, including secondary-bus -reset, FLR and Link Disable, change per-link ASPM controls or unbind the root -port's ``pcieport`` driver. Do not perform these operations on an admitted Neo. J700's ``wifi0`` alias and PCI endpoint node allow the public bootloader to pass the unit's own Wi-Fi address through ``local-mac-address``. The zero diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst new file mode 100644 index 00000000000000..d7e63d029de154 --- /dev/null +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +Experimental T8140 video encoding +================================ + +The J700 AVE driver exposes a root-only debugfs firmware transport for +hardware encoding experiments. It is not a V4L2 encoder, has no stable +userspace ABI, and does not make an FFmpeg encoder available. The initial +scope is the existing fixed 320x240 NV12 H.264/HEVC protocol experiment. + +Build ``CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL=m`` on a 16 KiB-page Apple kernel +with debugfs and translated DMA. Suspend, hibernation and kexec must be off. +Load with ``modprobe apple-ave enable=1``; probing is disabled otherwise. + +Firmware and boot requirements +------------------------------ + +Use the public Aurora m1n1 J700 AVE handoff. It reserves the firmware already +loaded by iBoot, fills the device's ``memory-region`` references and exports +``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32`` strings under +``/chosen``. No firmware file is included or requested by this driver. +The current transport requires the qualified 25G76 two-segment layout: +0x14c000 text bytes and 0x130000 data bytes, mapped starting at IOVA +0x10000000000. Every mapping and both content CRCs are checked before use. +Text CRCs are restricted to the two inspected 9003.78.0 images (0x2ce7359e +and 0x69c54454). These checks detect build drift, not firmware authenticity. +Different firmware is unsupported, even if the device has the same SoC. + +The driver holds the DMA, pipe4, pipe5, me0 and me1 power domains and owns +the IPC, queue, log and firmware-requested heap allocations. Debugfs lives +under ``/sys/kernel/debug/285100000.video-codec``: + +* ``ave-status`` reports mappings, protocol phase and reply counters. +* ``ave-start-once`` accepts one explicit firmware start per boot. +* ``ave-command`` accepts one bounded packet and returns a framed reply. +* ``ave-arena`` stages image/reference/output surfaces within owned memory. +* ``ave-poll`` drains later firmware replies without another submission. + +Writes require ``CAP_SYS_RAWIO``. Command packets must come from a trusted, +matching local protocol experiment with all embedded surface addresses +relocated into this boot's owned arena. Root access does not make arbitrary +firmware packets valid. Binary command captures are supplied separately and +are not included in this contribution. + +Lifecycle and verification +-------------------------- + +After the start attempt, the module, DMA buffers and power references remain +pinned until an external hardware reset, including after failure. Do not +force-unload, unbind the DARTs, reuse uncertain buffers or retry a failed +command. Firmware quiescence and normal production teardown are unfinished. + +A transport acknowledgement is not an encoded frame. For physical tests, +record the exact kernel/module, DT and firmware hashes; submit changing +images; validate each codec reply; independently decode the current DMA +output; check every frame's dimensions and quality; then verify Stop/Close, +kernel health and the existing desktop/radio stack. Keep the generated +inputs, bitstreams, decoded pixels, logs and hashes as a repeatable artifact. + +The prior implementation was authored by Ace (Acelogic). This contribution +ports that existing GPL work. Arbitrary resolutions, standard media APIs, +throughput, concurrent clients and other codecs or SoCs are outside this +experiment. diff --git a/Documentation/admin-guide/media/index.rst b/Documentation/admin-guide/media/index.rst index b11737ae6c0468..65aea88dac7ed5 100644 --- a/Documentation/admin-guide/media/index.rst +++ b/Documentation/admin-guide/media/index.rst @@ -43,6 +43,7 @@ Documentation/process/debugging/media_specific_debugging_guide.rst v4l-drivers dvb-drivers + apple-ave **Copyright** |copy| 1999-2020 : LinuxTV Developers diff --git a/Documentation/arch/arm64/apple-j813.rst b/Documentation/arch/arm64/apple-j813.rst index b2fa78079dd46b..a3819bc60a5246 100644 --- a/Documentation/arch/arm64/apple-j813.rst +++ b/Documentation/arch/arm64/apple-j813.rst @@ -5,19 +5,88 @@ Apple J813 initial native Linux boot ===================================== J813 is the 13-inch M5 MacBook Air (T8142, Mac17,3). This configuration -provides an initial RAM-only boot path: one boot CPU, AICv3, architectural -timers, DockChannel console and the framebuffer left by the boot firmware. +provides a RAM-only boot path with AICv3, architectural timers, DockChannel +console and the framebuffer left by the boot firmware. The J813 tree also +describes SMC temperature monitoring, MTP input and the keyboard backlight. -The device tree describes all ten CPU affinities so the loader can identify -the boot CPU. It does not qualify SMP. Keep secondary cores stopped in -m1n1 and use ``maxcpus=1`` for this milestone. The loader removes unavailable -CPUs and fills the release addresses of the remaining nodes. The P-core +The device tree describes all ten CPU affinities. For the initial one-CPU +baseline, keep secondary cores stopped in m1n1 and use ``maxcpus=1``. +The loader removes unavailable CPUs and fills the release addresses of the +remaining nodes. The P-core affinities are 0x10100 through 0x10103, rather than the 0x100 through 0x103 values in the Apple Device Tree. The CPU compatible strings identify T8142 explicitly. The firmware's reused ``sawtooth`` and ``everest`` names do not identify the M5 microarchitecture. +SMP bring-up +============ + +M5 secondary CPUs can return from WFI with cleared general-purpose registers. +Both M5 MIDRs (parts 0x062 and 0x063) select the Apple register-preserving idle +path. Without it, all ten CPUs can start and then panic in their idle tasks. + +SMP also requires the M5-capable m1n1 secondary-start path: dense PMGR CPU +masks, correct reset vectors and per-core stacks. Verify all ten cores there +before preparing the FDT, and use ``maxcpus=10`` for the Linux test. The +existing J813 DT and configuration support this handoff. + +On J813, this path reached the RAM shell with CPUs 0-9 online. A five-second +sleep advanced every CPU's architectural timer count by 1,251 interrupts. +Ten concurrent, individually pinned jobs each hashed 16 MiB of zeros to +``080acf35a507ac9849cfcba47dc2ad83e01b75663a516279c8b9d243b719643e``. +Verify affinity through ``Cpus_allowed_list`` and retain the console output, +kernel configuration hash and payload hashes when repeating the test. +Hotplug, deep sleep and KVM remain unqualified. + +CPU frequency bring-up +====================== + +The T8142 cluster controllers use five-bit P-state requests with busy and +command-state readback. The driver preserves firmware voltage, PLL and +CLPC/PMP configuration, and stops issuing requests after a timeout or an +unexpected command state. Unlike T8140, this initial J813 path does not +register a Linux SMC thermal policy. The DT exposes only states up to the +measured firmware handoff values; higher states remain unqualified. + +State 1 is 300 MHz on both clusters. The firmware SRAM voltage tables begin +at state 2: the E cluster exposes 300/972 MHz, and the P cluster exposes +300/1308/1608/1932/2220/2508/2820/3072/3300/3516 MHz. A dependent decrement +loop timed against CNTPCT_EL0 in m1n1 confirmed this mapping on J813. +``scaling_cur_freq`` reports the requested state, not an independent clock +measurement or a guarantee against firmware throttling. + +The configuration enables the userspace governor. On a fresh ten-core boot, +capture a sweep of both policies to the host serial log:: + + set -e + for p in /sys/devices/system/cpu/cpufreq/policy*; do + cat "$p/affected_cpus" + echo userspace > "$p/scaling_governor" + for round in 1 2; do + for f in $(cat "$p/scaling_available_frequencies"); do + echo "$f" > "$p/scaling_setspeed" + test "$(cat "$p/scaling_cur_freq")" = "$f" + echo "$p $f" + sleep 0.02 + done + done + cat "$p/cpuinfo_max_freq" > "$p/scaling_setspeed" + cat "$p/stats/total_trans" + done + dmesg + +Retain the transcript and payload hashes. Qualification also compared three +pairs of pinned, checksum-verified integer workloads at each cluster's +minimum and maximum states and checked every CPU's timer and pinned SHA-256 +job afterwards. Short frequency sweeps do not qualify sustained thermal +behavior, automatic governors, suspend or states above the handoff limit. + +T8142 can also lose registers on WFIT in delay loops. Frequency transition +polling exposed this as a cleared pointer in the cpufreq driver. T8142 uses +the existing counter delay loop, avoiding low-power waits with live kernel +register state. + Bootloader requirements ======================= @@ -30,8 +99,8 @@ loader is not sufficient. Before entering Linux, the loader must: * Fill the firmware framebuffer's address, size, geometry and format before enabling it. * Prepare DMA protection only for devices enabled in the supplied FDT. The - initial tree exposes no AOP, MTP, PMP or ISP DMA consumers. Their firmware - protection state must remain intact during the native FDT handoff. + ``mtp`` alias enables the MTP DAPF handoff. Leave the protection state of + unrelated AOP, PMP and ISP firmware intact. * Leave the console, interrupt controller and display powered. This minimal DT does not describe their power domains. * Give U-Boot a T8142 memory map covering its MMIO and high DRAM addresses. @@ -51,14 +120,14 @@ On an arm64 Linux build host with the usual kernel build dependencies:: KCONFIG_ALLCONFIG=arch/arm64/configs/j813.config allnoconfig make ARCH=arm64 O="$out" -j"$(nproc)" W=1 Image apple/t8142-j813.dtb -The fragment is deliberately minimal and does not enable KVM or storage -drivers. Add ``CROSS_COMPILE=aarch64-linux-gnu-`` to both commands when +The fragment does not enable KVM or storage drivers. Add +``CROSS_COMPILE=aarch64-linux-gnu-`` to both commands when cross-compiling with GCC from another architecture. With dtschema installed, check the affected bindings and DTB:: make ARCH=arm64 O="$out" CHECK_DTBS=y \ - DT_SCHEMA_FILES=arm/apple.yaml:arm/cpus.yaml:interrupt-controller/apple,aic2.yaml:serial/apple,dockchannel-uart.yaml:serial/samsung_uart.yaml:display/simple-framebuffer.yaml \ + DT_SCHEMA_FILES=arm/apple.yaml:arm/cpus.yaml:cpufreq/apple,cluster-cpufreq.yaml:opp/opp-v2-base.yaml:interrupt-controller/apple,aic2.yaml:serial/apple,dockchannel-uart.yaml:serial/samsung_uart.yaml:display/simple-framebuffer.yaml:apple,pmgr.yaml:apple,smc-hwmon.yaml:apple,dockchannel-hid.yaml:apple,dart.yaml:apple,mailbox.yaml:apple,smc.yaml:apple,pmgr-pwrstate.yaml:apple,s5l-fpwm.yaml:apple,dockchannel.yaml:apple,rtk-helper-asc4.yaml \ apple/t8142-j813.dtb Boot and qualification @@ -103,7 +172,107 @@ RAM boot. Preserve the complete console transcript and a hash manifest; an Image build or successful loader exit alone is not a boot result. Review captures for device identifiers and boot entropy before publishing. -This milestone does not establish SMP, CPU hotplug, KVM guest timers, -cpufreq, suspend, native DCP, GPU acceleration, built-in input, USB, -storage, networking or audio support. The disabled Samsung UART node +These boot tests do not establish CPU hotplug, KVM guest timers, +sustained thermal behavior, suspend, GPU acceleration, USB, +storage, networking or audio support. Input and monitoring require their +separate checks below. The disabled Samsung UART node records its measured resources; the qualified console path is DockChannel. + +Temperature monitoring and built-in input +======================================== + +Temperature monitoring enumerates the running SMC's readable, non-function +``T`` keys with four-byte floating-point values. It exposes their identifiers +through ``/sys/class/hwmon/*/temp*_label`` and millidegrees Celsius through +``temp*_input``. It does not embed a sensor inventory, calibration, inferred +sensor locations, trip temperatures or a new CPU cooling policy. Channel +numbers can change with firmware; identify channels by label. These keys can +include virtual or inactive channels; their number is not a count of physical +sensors, and a readable value does not establish a sensor's location. Read failures, +NaNs and infinities remain errors, rather than fabricated temperatures. +The existing J700 thermal provider and bounded J813 frequency states are +unchanged. Temperature reporting alone does not qualify sustained load. + +MTP provides keyboard and trackpad HID descriptors and identifiers at runtime. +The ``keyboard`` alias lets m1n1 fill ``hid-country-code`` and +``apple,keyboard-layout-id`` from this machine's own boot environment. The +source tree does not assume a US keyboard or contain another machine's layout. +The J813 trackpad uses power method 2 with firmware-owned reset; the host +does not operate an AFE GPIO. Other machines retain their existing sequence. + +The trackpad requires an externally supplied +``/lib/firmware/apple/tpmtfw-j813.bin``. The existing DockChannel firmware +container has a 20-byte little-endian header: magic ``0x46444948`` (``HIDF``), +version 1, header length, data length and interface-byte offset. See +``struct fw_header`` and ``dchid_get_firmware()`` for validation. An interface +offset of zero means no patching. The payload must +be prepared from firmware and any required calibration belonging to the +user's own machine. A filename is a lookup contract, not a firmware payload. +No Apple firmware, captured command buffers or calibration bytes are bundled. +Future IPSW/macOS extraction tools can provide this file without changing the +driver. A missing or invalid file fails trackpad startup; it does not prevent +the keyboard or temperature monitor from registering. + +The loader must preserve the iBoot-loaded MTP image and SRAM, prepare its DMA +protection, and leave MTP ready for Linux's RTKit startup. A prior-stage MTP +client must complete a compatible shutdown before Linux starts; a partially +initialized or crashed coprocessor is not a supported handoff. On J813, U-Boot +must not start and stop MTP before Linux: Linux's later restart crashes the +firmware and takes the keyboard and trackpad down. + +The backlight uses the existing PWM and LED drivers. Its boot brightness is +retained when the initial PWM period is valid, otherwise it starts off. +Userspace controls ``/sys/class/leds/kbd_backlight/brightness`` +on a linear 0--255 scale. No captured brightness curve or factory calibration +is included. + +Physical verification +--------------------- + +Place the user's touch firmware in a private RAM initramfs, then run:: + + sh tools/testing/selftests/drivers/apple/j813-peripherals.sh 30 + +The check reads temperatures twice, exercises and restores backlight +brightness, and counts keyboard and trackpad events during a bounded physical +interaction window. It records no key codes or pointer coordinates. Press +and release keys and move/click the built-in trackpad during that window. +Confirm visible brightness changes separately; sysfs readback is not optical +proof. Preserve stdout, the source revision and SHA-256 hashes of the exact +Image, DTB, configuration, initramfs and bootloaders as the repeatable result. +Keep firmware and raw machine data private. Without physical events the input +check fails rather than crediting device registration as working input. + +Native display +============== + +The internal panel is driven by the display coprocessor (DCP) through the +Apple DRM driver. iBoot leaves DCP running with H17-generation (macOS 26) +firmware mapped through locked DARTs. Linux attaches to that live session: +it sends the standard RTKit wake handshake and never stops, resets or reloads +the coprocessor. No firmware file is loaded by Linux. + +The loader must: + +* reserve every live DCP firmware segment and the boot framebuffer, and + export their IOVAs to the ``dcp``, ``disp0`` and ``disp0_piodma`` nodes; +* enable the display DART (the first ``iommus`` entry of ``disp0``) and the + ``dcp`` node only after doing so. + +The display DARTs keep the firmware's root page-table slots. Linux publishes +its own translations only into vacant slots. On ``apple,t8142-dart``, a +firmware slot is accepted when it already translates every page Linux mapped +there to the same physical address, and later Linux mappings into it must +match the firmware's translation. Firmware buffers DCP later asks Linux to +share are resolved through those firmware tables. + +The DCP firmware reports the panel dimensions at startup; the driver does not +embed a panel timing or geometry. The J813 panel reports 2560x1664; the driver +hides its 64-pixel notch area by default, so the mode is 2560x1600 at 60 Hz. +The primary plane scans out IOMFB surface 2. Backlight control and a cursor +plane are not provided yet. + +Boot the RAM payload as above with ``CONFIG_DRM_APPLE``. On J813 this reached +``DCP booted``, registered ``appledrmfb`` as the console framebuffer, and ran +a KDE Plasma Wayland session with the built-in keyboard and trackpad. The +desktop renders in software; GPU acceleration is separate work. diff --git a/Documentation/arch/arm64/apple-m6-cpu.rst b/Documentation/arch/arm64/apple-m6-cpu.rst new file mode 100644 index 00000000000000..046872e83e09a7 --- /dev/null +++ b/Documentation/arch/arm64/apple-m6-cpu.rst @@ -0,0 +1,29 @@ +M6 CPU bring-up +============== + +J873g has six E CPUs and a shared performance domain containing two P CPUs +and four M CPUs. The T8152 ACC request uses a five-bit hardware state, with +state 2 as the first ADT operating point. Do not use the older voltage or PLL +initialization sequences. + +Failure cases and qualification +------------------------------ + +Before testing, account for incorrect MPIDRs, incomplete spin-table release, +broken cross-CPU interrupts, a stalled architectural timer, mismatched OPP +indices, an unknown inherited request, a stuck busy bit, an unacknowledged +transition, and another agent changing the request. A transition failure must +latch until the policy is recreated. A requested frequency is not a measured +frequency. + +The initial tables expose only states 2 and 3. Higher states require an SMC +thermal policy; this port does not qualify them. The loader must hand over +within the advertised table. Keep firmware voltage, PLL and throttling +configuration intact. + +On physical J873g, capture the source revision, image and DTB hashes, boot log, +online CPU list, per-CPU workload completion, interrupt counters and both +cpufreq policies. Exercise both exposed states in each domain, compare timed +dependent workloads, restore the low states and check the kernel log. Save +the commands and a machine-readable receipt. Native m1n1 SMP results alone +do not qualify Linux scheduling. diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst index fc65c2d61a1de0..23afe611f30398 100644 --- a/Documentation/arch/arm64/index.rst +++ b/Documentation/arch/arm64/index.rst @@ -33,5 +33,6 @@ ARM64 Architecture sve tagged-address-abi tagged-pointers + user4k features diff --git a/Documentation/arch/arm64/user4k.rst b/Documentation/arch/arm64/user4k.rst new file mode 100644 index 00000000000000..d483c565d853b1 --- /dev/null +++ b/Documentation/arch/arm64/user4k.rst @@ -0,0 +1,187 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +=========================================== +Experimental per-process page sizes on arm64 +=========================================== + +The mixed-page implementation lets an address space use a different user +translation granule from the kernel's native physical page size. The kernel +page size remains fixed for the boot. An address space's user granule remains +fixed for its lifetime; exec can create a new address space with another +supported granule. See :doc:`../../mm/user_page_size` for the generic launch +policy and architecture integration contract. + +This is an experimental ABI. The launch commands use private prctl numbers, +not assigned upstream UAPI. The implementation and tests do not establish +complete application, device or hardware qualification. + +Configuration and supported combinations +======================================== + +Enable ``CONFIG_ARM64_USER4K_EXPERIMENTAL`` with 48-bit VA and PA and a native +16K or 64K page size. The option defaults off and selects the generic +``MM_SUBPAGE`` backing and ``ARCH_HAS_USER_PAGE_SIZE`` exec policy support. +LPA2 and ``NO_PAGE_MAPCOUNT`` configurations are excluded by Kconfig. + +The exposed combinations are: + +=================== ================== ============================ +Native kernel size AArch64 default AArch64 alternatives +=================== ================== ============================ +4K 4K None in this implementation +16K 16K 4K, when supported by the CPUs +64K 64K 4K and 16K, subject to support +=================== ================== ============================ + +Capability detection uses the sanitized system-wide granule intersection. +AArch32 executables select 4K when that capability is available and retain +the native granule otherwise, regardless of launch preferences. The latter +case still enforces native mapping/protection alignment. AArch32 mmap2 file +offsets retain their 4K ABI unit in either case. + +An AArch32 executable running under an arm64 kernel is distinct from an +``arch/arm`` kernel. The latter retains its existing native ABI; this series +does not yet provide its alternative-granule backend. Exploiting extra +translation capabilities available only within an asymmetric AArch32 CPU +domain also remains unimplemented. + +Larger-than-native process granules are not exposed. Internal 64K-on-16K +fixtures exercise allocation, mappings and argument staging, but those +fixtures do not enable a public 64K process ABI on a 16K kernel. + +Selecting an executable's page size +=================================== + +Build the development launcher with:: + + make -C tools/aurora-userpages + +For example:: + + tools/aurora-userpages/userpages 4k COMMAND [ARGUMENTS...] + tools/aurora-userpages/userpages --once native COMMAND [ARGUMENTS...] + tools/aurora-userpages/userpages --status + +The ordinary selection establishes an inherited per-thread exec preference. +``--once`` overrides it for one successful exec. A failed exec retains the +pending request; changing a preference never changes existing mappings. +Fork copies the preferences and mappings. Threads sharing an mm have the +same current granule but may establish different preferences for later exec. +``native`` clears an inherited alternative preference; a one-shot native +request can override that preference without clearing it. + +The final ELF loader resolves the granule before committing the new image. +If arguments were staged with another granule, their bytes are transferred +into a new private mm. Temporary argument mappings are bounded by both the +user leaf and native page size. Logical argument-size limits continue to +use the process granule. The interpreter and executable share one mm and +therefore one final granule. ELF segment alignment alone does not opt a +program into an alternative ABI. + +``AT_PAGESZ`` reports the process granule. User virtual address alignment, +protection boundaries and mapping lengths use that granule. Native physical +allocation, PFN arithmetic and file-cache indices continue to use the kernel +page size. Application execution does not require any of the KUnit options +or the ``arm64.user4k_test=1`` diagnostic boot parameter. + +Memory and table ownership +========================== + +Keep the following units separate: + +* ``PAGE_SIZE`` and physical PFNs describe native physical pages. +* ``mm_page_size(mm)`` describes the user mapping/protection unit. +* A folio owns one or more native pages and carries their physical charge. + +Smaller anonymous mappings retain explicit subpage ownership. A reference to +the containing folio alone does not retain an individual reusable slot. +The slot's byte offset must survive fault, COW, reverse mapping, migration, +swap and pinning operations. Newly exposed bytes are zeroed, and changing +one mapping's protection must not widen that change to adjacent slots. + +File offsets use a native cache index plus an explicit byte offset. VMA +split/merge, truncation and reverse mapping must preserve both components. +Physical allocation charges count the actual backing once, independently +of how many user mappings expose it. + +Lower page tables can share a native allocation. Table tokens preserve the +fragment address, while locking and lifetime use the native table owner. +Unissued cached fragments retain owner references. Issued fragments are not +reused within a live owner. Deferred reclamation must preserve the fragment +token through TLB invalidation and the required RCU grace period, including +allocation-failure fallback paths. + +Translation context and invalidation +=================================== + +The arm64 backend supplies per-mm page-table geometry. TTBR1 retains native +kernel tables, while TTBR0 uses the selected user geometry. A granule change +installs and synchronizes a reserved TTBR0 before changing TG0, then +synchronizes the new geometry before installing the user root. Software PAN +also selects the user TG0 even when restoration of TTBR0 is deferred until +uaccess or return to userspace. + +The existing mm ASID is retained. Selecting another granule does not add +an unconditional context-switch TLB flush. User invalidation operands and +range strides use the target mm's geometry; kernel and stage-2 operations +retain their own native geometry. Transition/idmap paths restore native +TG0 and then the active mm's user geometry when returning. + +The current configuration retains the existing 48-bit user address size. +Independent TG0/TG1 fields do not make all other translation controls +independent; the LPA2 exclusion must not be removed merely by changing TG0. + +Sharing, pinning and interface limits +==================================== + +Mixed-granule sharing is permitted. Shared mappings refer to one coherent +backing object, with permissions belonging to each virtual mapping. Shared +futex identity includes the object and byte offset. Private copying is not +an implementation of shared-memory semantics. + +Some interfaces have additional layout or ownership requirements. For +example, a perf ring's first mapper fixes its userspace granule; a mapper +using another granule is rejected. This does not prohibit mixed-granule +memfd or System V shared-memory aliases. + +A legacy page-array pin cannot express every smaller-granule byte range. +Converted consumers use typed fragment references carrying backing, offset, +length and ownership. Unconverted paths must not silently round a pin or DMA +mapping into neighboring slots. Mock IOMMU/VFIO tests validate the recorded +translations and ownership paths, not physical-device DMA on arbitrary +hardware. + +Alternative-granule THP/KSM and hugetlb support, remaining legacy pin +consumers, larger-than-native process activation and broader device +integration still need work. Userfaultfd feature bits and per-range ioctl +masks must be consulted; success for a shmem operation does not imply +support for the corresponding hugetlb operation. Sparse individual fills +can retain an entire native folio; packing behavior is not a throughput +or memory-efficiency guarantee for every workload. + +Test coverage +============= + +The arm64 selftests are in ``tools/testing/selftests/arm64/user4k``. The +``page-contract`` executable checks the exposed userspace contract. Extended +fixtures cover exec policy, AArch32, shared mappings, accounting, reclaim, +I/O and device-facing interfaces. Several are disposable-guest PID1 programs +with companion executables or storage requirements; they are not ordinary +host test commands. + +``CONFIG_ARM64_USER4K_KUNIT_TEST`` and the data/TLB suites exercise internal +geometry and ownership. Some fixtures require ``arm64.user4k_test=1`` and +some hardware translation tests are disabled under KASAN. Check individual +KTAP results and skips rather than treating boot success as a suite pass. +Private IOMMUFD/VFIO test-device options are separate and should remain off +in application kernels. + +The NUMA COW batching case runs only at boot, before userspace can change +its policy prerequisite. Its results remain visible in KUnit debugfs, but +it has no runtime rerun control. Other suites retain their rerun controls. + +Validation must distinguish native/default operation, each exposed +alternative, and AArch32 fallback with 4K support unavailable. Native ARM32 +short-descriptor and LPAE controls check that generic changes preserve the +existing ABI; they do not validate a mixed-page ARM32 backend. Internal +larger-leaf tests likewise do not qualify a public process mode. diff --git a/Documentation/devicetree/bindings/arm/apple/apple,pmgr.yaml b/Documentation/devicetree/bindings/arm/apple/apple,pmgr.yaml index 77d87d8c376e8a..f10b9cbfce597e 100644 --- a/Documentation/devicetree/bindings/arm/apple/apple,pmgr.yaml +++ b/Documentation/devicetree/bindings/arm/apple/apple,pmgr.yaml @@ -43,6 +43,7 @@ properties: - apple,t8122-pmgr - apple,t8132-pmgr - apple,t8140-pmgr + - apple,t8142-pmgr - const: apple,t8103-pmgr - const: syscon - const: simple-mfd diff --git a/Documentation/devicetree/bindings/chosen/apple,dcp-rtkit-quiesced.yaml b/Documentation/devicetree/bindings/chosen/apple,dcp-rtkit-quiesced.yaml new file mode 100644 index 00000000000000..d086bdc05f5ee7 --- /dev/null +++ b/Documentation/devicetree/bindings/chosen/apple,dcp-rtkit-quiesced.yaml @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/chosen/apple,dcp-rtkit-quiesced.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple DCP RTKit loader hand-off + +maintainers: + - Ryan Murray + +description: + The loader records whether it handed a running display coprocessor to + the kernel. It omits this property on a chainload. + +select: + properties: + $nodename: + const: chosen + required: + - $nodename + +properties: + $nodename: + const: chosen + + apple,dcp-rtkit-quiesced: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: + One if the DCP core was running at hand-off, zero if it was not. + The name reflects the loader's RTKit hand-off state, not a request + for Linux to quiesce the coprocessor. + +additionalProperties: true +... diff --git a/Documentation/devicetree/bindings/cpu/apple,t8140-cpuidle.yaml b/Documentation/devicetree/bindings/cpu/apple,t8140-cpuidle.yaml new file mode 100644 index 00000000000000..7d4f9cdbc91829 --- /dev/null +++ b/Documentation/devicetree/bindings/cpu/apple,t8140-cpuidle.yaml @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/cpu/apple,t8140-cpuidle.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple T8140 CPU idle controller + +maintainers: + - Ryan Murray + +description: + Selects the Apple CPU idle driver on T8140 systems. On this SoC the + hardware decides when an idle core is power-gated after WFI; software + cannot request core or cluster power-down, so WFI is the only idle + state. + +properties: + $nodename: + const: cpuidle + + compatible: + const: apple,t8140-cpuidle + +required: + - compatible + +additionalProperties: false + +examples: + - | + cpuidle { + compatible = "apple,t8140-cpuidle"; + }; diff --git a/Documentation/devicetree/bindings/cpufreq/apple,cluster-cpufreq.yaml b/Documentation/devicetree/bindings/cpufreq/apple,cluster-cpufreq.yaml index 8e87c0f6d3d451..a27f4314e0387a 100644 --- a/Documentation/devicetree/bindings/cpufreq/apple,cluster-cpufreq.yaml +++ b/Documentation/devicetree/bindings/cpufreq/apple,cluster-cpufreq.yaml @@ -36,6 +36,8 @@ properties: - const: apple,s5l8960x-cluster-cpufreq - const: apple,s5l8960x-cluster-cpufreq - const: apple,t8140-cluster-cpufreq + - const: apple,t8142-cluster-cpufreq + - const: apple,t8152-cluster-cpufreq - items: - const: apple,t6020-cluster-cpufreq - const: apple,t8112-cluster-cpufreq @@ -43,6 +45,18 @@ properties: reg: maxItems: 1 + apple,fast-die-effort: + $ref: /schemas/types.yaml#/definitions/phandle-array + description: | + PMP report region and cluster effort lane. Increasing firmware effort + imposes a maximum-frequency QoS constraint alongside thermal cooling. + If the PMP is unavailable, cpufreq retains its fallback ceiling. + items: + - items: + - description: PMP report region + - description: Cluster lane (0 for E, 1 for P on T8140) + enum: [0, 1] + '#performance-domain-cells': const: 0 diff --git a/Documentation/devicetree/bindings/display/apple,dcp.yaml b/Documentation/devicetree/bindings/display/apple,dcp.yaml new file mode 100644 index 00000000000000..0fe31267f99e16 --- /dev/null +++ b/Documentation/devicetree/bindings/display/apple,dcp.yaml @@ -0,0 +1,369 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/display/apple,dcp.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple display coprocessor + +maintainers: + - Ryan Murray + +description: + Apple's Display Co-Processor (DCP) drives internal and external displays + through an RTKit mailbox. Its piodma child uses a separate display DART + stream from the coprocessor itself. + +properties: + compatible: + oneOf: + - items: + - enum: + - apple,t8103-dcp + - apple,t8112-dcp + - apple,t6000-dcp + - apple,t6020-dcp + - apple,t8140-dcp + - apple,t8142-dcp + - const: apple,dcp + - items: + - enum: + - apple,t8103-dcpext + - apple,t8112-dcpext + - apple,t6000-dcpext + - apple,t6020-dcpext + - const: apple,dcpext + - items: + - const: apple,t8140-dcpext + - const: apple,dcp + + reg: + minItems: 2 + maxItems: 7 + + reg-names: + minItems: 2 + items: + - const: coproc + - const: disp-0 + - enum: [disp-1, iop-vbar] + - enum: [disp-2, iop-vbar] + - enum: [disp-3, iop-vbar] + - enum: [disp-4, iop-vbar] + - const: iop-vbar + + mboxes: + maxItems: 1 + + mbox-names: + items: + - const: mbox + + iommus: + maxItems: 1 + + power-domains: + maxItems: 1 + + resets: + maxItems: 1 + + clocks: + minItems: 1 + maxItems: 2 + + clock-names: + minItems: 1 + items: + - const: clock-154 + - const: clock-194 + + memory-region: + minItems: 1 + maxItems: 64 + + memory-region-names: + minItems: 1 + maxItems: 64 + + phys: + minItems: 1 + maxItems: 8 + + phy-names: + minItems: 1 + maxItems: 8 + items: + pattern: '^(dp-phy|typec[0-9]+)$' + + apple,bw-scratch: + $ref: /schemas/types.yaml#/definitions/phandle-array + items: + - items: + - description: Phandle of the PMGR bandwidth register provider. + - description: Index into the provider's reg property. + - description: Index in the firmware display register table. + - description: Byte offset of the scratch register within the resource. + description: PMGR bandwidth scratch resource, display index and offset. + + apple,bw-doorbell: + $ref: /schemas/types.yaml#/definitions/phandle-array + items: + - items: + - description: Phandle of the PMGR bandwidth register provider. + - description: Index into the provider's reg property. + - description: Index in the firmware display register table. + description: Optional PMGR bandwidth doorbell resource and display index. + + apple,iomfb-surfaces: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 4 + items: + enum: [0, 1] + description: | + Per-slot enable flags, in slot order. The legacy J700 bootloader + supplies a single zero to select surface zero instead. + + apple,power-lut-data-x: + $ref: /schemas/types.yaml#/definitions/uint8-array + minItems: 4 + maxItems: 4 + description: | + First dimension of the display firmware power lookup table. The + bootloader copies the disp0 ADT property verbatim, as a little-endian + 32-bit word, for transfer to the display firmware. + + apple,power-lut-data-y: + $ref: /schemas/types.yaml#/definitions/uint8-array + minItems: 4 + maxItems: 4 + description: | + Second dimension of the display firmware power lookup table, copied + verbatim from disp0 in the ADT as a little-endian 32-bit word. + + apple,power-lut-data-xindex: + $ref: /schemas/types.yaml#/definitions/uint8-array + minItems: 52 + maxItems: 52 + description: | + First-axis indices of the display firmware power lookup table, copied + verbatim from disp0 in the ADT as thirteen little-endian 32-bit words. + These are firmware table inputs, not a Linux backlight range. + + apple,power-lut-data-yindex: + $ref: /schemas/types.yaml#/definitions/uint8-array + minItems: 4 + maxItems: 4 + description: | + Second-axis index of the display firmware power lookup table, copied + verbatim from disp0 in the ADT as a little-endian 32-bit word. + + apple,power-lut-data-lut: + $ref: /schemas/types.yaml#/definitions/uint8-array + minItems: 52 + maxItems: 52 + description: | + Display firmware power lookup table, copied verbatim from disp0 in + the ADT as thirteen little-endian 32-bit words. Linux passes these + values to the firmware without interpreting or changing them. + + apple,firmware-version: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 3 + maxItems: 5 + description: Version reported by the loaded display firmware. + + apple,firmware-compat: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 3 + maxItems: 5 + description: Loader-provided firmware ABI version used for IOMFB selection. + + apple,connector-type: + enum: [HDMI-A, DP, USB-C] + + apple,dcp-index: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Display coprocessor index on multi-DCP systems. + + apple,dptx-phy: + $ref: /schemas/types.yaml#/definitions/uint32 + description: DisplayPort transmitter PHY selector. + + apple,dptx-die: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Die containing the DisplayPort transmitter. + + apple,notch-height: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Height in pixels reserved for the display notch. + + apple,typec-mux-indices: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 4 + description: Crossbar state for each Type-C display route. + + apple,typec-dptx-phys: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 4 + description: DisplayPort PHY index for each Type-C display route. + + mux-controls: + $ref: /schemas/types.yaml#/definitions/phandle-array + minItems: 1 + maxItems: 8 + + mux-control-names: + minItems: 1 + maxItems: 8 + items: + pattern: '^(dp-xbar|typec[0-9]+)$' + + mux-index: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Crossbar state for a fixed output. + + hdmi-hpd-gpios: + maxItems: 1 + + hdmi-pwren-gpios: + maxItems: 1 + + dp2hdmi-pwren-gpios: + maxItems: 1 + + '#address-cells': + const: 2 + + '#size-cells': + const: 2 + + piodma: + type: object + properties: + iommus: + maxItems: 1 + required: + - iommus + additionalProperties: false + + panel: + type: object + properties: + compatible: + oneOf: + - const: apple,panel + - items: + - pattern: '^apple,panel-j[0-9]+$' + - const: apple,panel + - items: + - pattern: '^apple,panel-j[0-9]+$' + - const: apple,panel-mini-led + - const: apple,panel + width-mm: + description: Physical panel width. + height-mm: + description: Physical panel height. + adj-height-mm: + description: Visible height below the notch. + apple,max-brightness: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Maximum firmware backlight level. + required: + - compatible + additionalProperties: false + + typec-routes: + type: object + properties: + '#address-cells': + const: 1 + '#size-cells': + const: 0 + patternProperties: + '^route@[0-3]$': + type: object + properties: + reg: + maxItems: 1 + mode-switch: + type: boolean + port: + $ref: /schemas/graph.yaml#/properties/port + required: + - reg + - mode-switch + - port + additionalProperties: false + required: + - '#address-cells' + - '#size-cells' + additionalProperties: false + + ports: + $ref: /schemas/graph.yaml#/properties/ports + +required: + - compatible + - reg + - reg-names + - mboxes + - iommus + - apple,bw-scratch + +allOf: + - if: + not: + properties: + compatible: + contains: + enum: ['apple,t8140-dcp', 'apple,t8140-dcpext'] + then: + properties: + apple,power-lut-data-x: false + apple,power-lut-data-y: false + apple,power-lut-data-xindex: false + apple,power-lut-data-yindex: false + apple,power-lut-data-lut: false + - if: + properties: + compatible: + contains: + enum: ['apple,t8140-dcp', 'apple,t8140-dcpext'] + then: + properties: + resets: false + +unevaluatedProperties: false + +examples: + - | + soc { + #address-cells = <2>; + #size-cells = <2>; + + dcp@231c00000 { + compatible = "apple,t8103-dcp", "apple,dcp"; + reg = <0x2 0x31c00000 0x0 0x4000>, + <0x2 0x30000000 0x0 0x3e8000>, + <0x2 0x31320000 0x0 0x4000>, + <0x2 0x31344000 0x0 0x4000>, + <0x2 0x31800000 0x0 0x800000>, + <0x2 0x3b3d0000 0x0 0x4000>; + reg-names = "coproc", "disp-0", "disp-1", "disp-2", + "disp-3", "disp-4"; + mboxes = <&dcp_mbox>; + mbox-names = "mbox"; + iommus = <&dcp_dart 0>; + apple,bw-scratch = <&pmgr_dcp 0 5 0x14>; + apple,bw-doorbell = <&pmgr_dcp 1 6>; + + piodma { + iommus = <&disp_dart 0>; + }; + }; + }; +... diff --git a/Documentation/devicetree/bindings/display/apple,display-subsystem.yaml b/Documentation/devicetree/bindings/display/apple,display-subsystem.yaml new file mode 100644 index 00000000000000..429e68b29d864a --- /dev/null +++ b/Documentation/devicetree/bindings/display/apple,display-subsystem.yaml @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/display/apple,display-subsystem.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple display subsystem + +maintainers: + - Ryan Murray + +description: + Component master for one or more Apple display coprocessors. Each IOMMU + entry describes a scanout DART stream owned by a DCP instance. + +properties: + compatible: + const: apple,display-subsystem + + iommus: + minItems: 1 + maxItems: 8 + +required: + - compatible + - iommus + +unevaluatedProperties: false +... diff --git a/Documentation/devicetree/bindings/gpu/apple,agx.yaml b/Documentation/devicetree/bindings/gpu/apple,agx.yaml index 59989d8bd1cbc5..e7afc4dd87e8f1 100644 --- a/Documentation/devicetree/bindings/gpu/apple,agx.yaml +++ b/Documentation/devicetree/bindings/gpu/apple,agx.yaml @@ -17,6 +17,7 @@ properties: - apple,agx-g13s - apple,agx-g14g - apple,agx-g14s + - apple,agx-t8140 - items: - enum: - apple,agx-g13c @@ -29,22 +30,20 @@ properties: - const: apple,agx-g14s reg: - items: - - description: GPU coprocessor control registers - - description: GPU block MMIO registers + maxItems: 2 reg-names: - items: - - const: asc - - const: sgx + maxItems: 2 power-domains: maxItems: 1 mboxes: - maxItems: 1 + minItems: 1 + maxItems: 2 memory-region: + minItems: 3 items: - description: Region containing GPU MMU TTBs - description: Region containing GPU MMU page tables @@ -56,6 +55,7 @@ properties: - description: Shared global variables with GPU firmware memory-region-names: + minItems: 3 items: - const: ttbs - const: pagetables @@ -64,6 +64,8 @@ properties: - const: hw-cal-b - const: globals + operating-points-v2: true + apple,firmware-abi: $ref: /schemas/types.yaml#/definitions/uint32-array minItems: 3 @@ -77,7 +79,52 @@ required: - reg - mboxes - memory-region - - apple,firmware-abi + +allOf: + - if: + properties: + compatible: + contains: + const: apple,agx-t8140 + then: + properties: + reg: + items: + - description: GFX coprocessor control registers + - description: GPU block MMIO registers + reg-names: + items: + - const: asc + - const: sgx + mboxes: + items: + - description: GFX coprocessor mailbox + - description: GFX1 coprocessor mailbox + memory-region: + maxItems: 3 + memory-region-names: + maxItems: 3 + required: + - power-domains + - operating-points-v2 + else: + properties: + reg: + items: + - description: GPU coprocessor control registers + - description: GPU block MMIO registers + reg-names: + items: + - const: asc + - const: sgx + mboxes: + maxItems: 1 + memory-region: + minItems: 6 + memory-region-names: + minItems: 6 + required: + - apple,firmware-abi additionalProperties: false diff --git a/Documentation/devicetree/bindings/hwmon/apple,smc-hwmon.yaml b/Documentation/devicetree/bindings/hwmon/apple,smc-hwmon.yaml index 076de649627c80..df155f9c34a65b 100644 --- a/Documentation/devicetree/bindings/hwmon/apple,smc-hwmon.yaml +++ b/Documentation/devicetree/bindings/hwmon/apple,smc-hwmon.yaml @@ -40,6 +40,16 @@ properties: compatible: const: apple,smc-hwmon + apple,discover-temperatures: + type: boolean + description: + Enumerate readable, non-function temperature keys from the running SMC + instead of providing a board-specific temperature table. Only keys + beginning with T and reporting a four-byte floating-point value are + exposed. Labels are the key identifiers, without inferred locations. + Channels are reporting-only; their indices depend on the firmware and + must not be used by a static thermal policy. + "#thermal-sensor-cells": const: 1 description: @@ -76,7 +86,6 @@ patternProperties: pattern: "^[A-Za-z0-9]{4}$" description: Writeable key to enable/disable manual fan control - "^power-[A-Za-z0-9]{4}$": $ref: "#/$defs/sensor" unevaluatedProperties: false @@ -89,4 +98,12 @@ patternProperties: $ref: "#/$defs/sensor" unevaluatedProperties: false +allOf: + - if: + required: + - apple,discover-temperatures + then: + patternProperties: + "^temperature-": false + additionalProperties: false diff --git a/Documentation/devicetree/bindings/input/apple,dockchannel-hid.yaml b/Documentation/devicetree/bindings/input/apple,dockchannel-hid.yaml index 3b82308e0b568c..924442134b2482 100644 --- a/Documentation/devicetree/bindings/input/apple,dockchannel-hid.yaml +++ b/Documentation/devicetree/bindings/input/apple,dockchannel-hid.yaml @@ -23,7 +23,9 @@ description: | trackpad itself and asks the host to pulse a reset line when it needs one. With power method 2 the host announces each power transition to the coprocessor, drives the reset line of the trackpad analog front end (AFE) - itself and then confirms the transition. + itself and then confirms the transition. On systems with firmware-managed + reset, the same notifications bracket a firmware-owned transition and the + host waits 50 ms during power-on without operating a GPIO. properties: compatible: @@ -140,7 +142,27 @@ properties: default: 1 description: The power sequencing protocol of the coprocessor firmware, as - described above. Power method 2 requires apple,afe-reset-gpios. + described above. Power method 2 requires apple,afe-reset-gpios + unless apple,firmware-managed-reset is present. + + apple,firmware-managed-reset: + type: boolean + description: + The coprocessor owns the trackpad reset signal. The host sends + power method 2 notifications without controlling an AFE GPIO. + Mutually exclusive with apple,afe-reset-gpios on the parent. + + allOf: + - if: + required: + - apple,firmware-managed-reset + then: + properties: + apple,power-method: + const: 2 + required: + - apple,power-method + - firmware-name stm: type: object @@ -171,6 +193,18 @@ required: additionalProperties: false +allOf: + - if: + properties: + multi-touch: + required: + - apple,firmware-managed-reset + required: + - multi-touch + then: + properties: + apple,afe-reset-gpios: false + examples: - | #include diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml index 7fd21f26df6f89..5a128e38d8f6f4 100644 --- a/Documentation/devicetree/bindings/iommu/apple,dart.yaml +++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml @@ -38,6 +38,7 @@ properties: - apple,t6020-dart - apple,t8122-dart - apple,t8140-dart + - apple,t8142-dart - const: apple,t8110-dart reg: diff --git a/Documentation/devicetree/bindings/mailbox/apple,ascwrap-v6.yaml b/Documentation/devicetree/bindings/mailbox/apple,ascwrap-v6.yaml new file mode 100644 index 00000000000000..4798b62bd84300 --- /dev/null +++ b/Documentation/devicetree/bindings/mailbox/apple,ascwrap-v6.yaml @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/mailbox/apple,ascwrap-v6.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple T8140 ASCWrap v6 mailbox + +maintainers: + - Ryan Murray + +description: + Mailbox interface of the T8140 GPU coprocessors. The wrapper resource + contains the coprocessor control registers and mailbox FIFOs. A separate + resource provides the coprocessor vector-base register. The firmware role + identifies which of the two GPU firmware instances uses the mailbox. + +properties: + compatible: + const: apple,t8140-ascwrap-v6 + + reg: + items: + - description: ASC wrapper registers + - description: Coprocessor vector-base register + + reg-names: + items: + - const: wrapper + - const: iop-vbar + + interrupts: + items: + - description: Send FIFO empty interrupt + - description: Receive FIFO not-empty interrupt + + interrupt-names: + items: + - const: send-empty + - const: recv-not-empty + + '#mbox-cells': + const: 0 + + power-domains: + maxItems: 1 + + apple,firmware-role: + enum: [GFX, GFX1] + +required: + - compatible + - reg + - reg-names + - interrupts + - interrupt-names + - '#mbox-cells' + - apple,firmware-role + +additionalProperties: false + +examples: + - | + #include + #include + + soc { + #address-cells = <2>; + #size-cells = <2>; + + mailbox@482600000 { + compatible = "apple,t8140-ascwrap-v6"; + reg = <0x4 0x82600000 0x0 0x88000>, + <0x4 0x82050000 0x0 0x8>; + reg-names = "wrapper", "iop-vbar"; + interrupts = , + ; + interrupt-names = "send-empty", "recv-not-empty"; + apple,firmware-role = "GFX"; + #mbox-cells = <0>; + }; + }; diff --git a/Documentation/devicetree/bindings/mailbox/apple,mailbox.yaml b/Documentation/devicetree/bindings/mailbox/apple,mailbox.yaml index 32fb3acb8c99fd..e6b31a9afb19ef 100644 --- a/Documentation/devicetree/bindings/mailbox/apple,mailbox.yaml +++ b/Documentation/devicetree/bindings/mailbox/apple,mailbox.yaml @@ -32,6 +32,7 @@ properties: - apple,t8112-asc-mailbox - apple,t8122-asc-mailbox - apple,t8140-asc-mailbox + - apple,t8142-asc-mailbox - apple,t6000-asc-mailbox - apple,t6020-asc-mailbox - const: apple,asc-mailbox-v4 @@ -45,6 +46,14 @@ properties: - apple,t8112-ane-mailbox - const: apple,asc-mailbox-v4 + - description: + ASC mailboxes of the T8140 GPU coprocessors. Unlike the other + T8140 ASC mailboxes, they do not need the application processor + to enable their receive FIFO. + items: + - const: apple,t8140-gfx-asc-mailbox + - const: apple,asc-mailbox-v4 + - description: AOP setup-port mailbox with a separate register layout. const: apple,t8140-aop-setup-mailbox diff --git a/Documentation/devicetree/bindings/media/apple,t8140-ave.yaml b/Documentation/devicetree/bindings/media/apple,t8140-ave.yaml new file mode 100644 index 00000000000000..110ebbc3d09e21 --- /dev/null +++ b/Documentation/devicetree/bindings/media/apple,t8140-ave.yaml @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/apple,t8140-ave.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple T8140 video encoder + +maintainers: + - Miguel Cruz <7362641+Acelogic@users.noreply.github.com> + +description: | + The T8140 AVE coprocessor encodes video using firmware preloaded by the + bootloader. Its two DART instances translate the same stream. The + bootloader supplies two reserved firmware regions and their IOVA mappings. + +properties: + compatible: + const: apple,t8140-ave + + reg: + items: + - description: Codec registers + - description: Mailbox registers + - description: Coprocessor registers + + reg-names: + items: + - const: codec + - const: mailbox + - const: coproc + + interrupts: + maxItems: 5 + + power-domains: + maxItems: 5 + + power-domain-names: + items: + - const: dma + - const: pipe4 + - const: pipe5 + - const: me0 + - const: me1 + + iommus: + maxItems: 2 + + memory-region: + items: + - description: Preloaded firmware text + - description: Preloaded firmware data + +required: + - compatible + - reg + - reg-names + - interrupts + - power-domains + - power-domain-names + - iommus + +additionalProperties: false diff --git a/Documentation/devicetree/bindings/media/apple,t8140-isp.yaml b/Documentation/devicetree/bindings/media/apple,t8140-isp.yaml index dea138b415b7ae..59c6c91fb19d72 100644 --- a/Documentation/devicetree/bindings/media/apple,t8140-isp.yaml +++ b/Documentation/devicetree/bindings/media/apple,t8140-isp.yaml @@ -145,3 +145,45 @@ required: - sensor-presets additionalProperties: false + +examples: + - | + #include + #include + + soc { + #address-cells = <2>; + #size-cells = <2>; + + isp@320000000 { + compatible = "apple,t8140-isp", "apple,isp"; + reg = <0x3 0x20000000 0x0 0x1801000>, + <0x3 0x22920000 0x0 0x1000>, + <0x3 0x22921000 0x0 0x100>, + <0x3 0x63b50000 0x0 0x8000>; + reg-names = "coproc", "mbox", "gpio", "wdt"; + interrupt-parent = <&aic>; + interrupts = ; + iommus = <&isp_dart0 0>, <&isp_dart1 0>, <&isp_dart2 0>; + power-domains = <&ps_isp_sys>, <&ps_isp_cpu>, + <&ps_isp_cpu_core0>, <&ps_isp_cpu_core1>, + <&ps_isp_fe>, <&ps_isp_dprx>, <&ps_isp_secure>, + <&ps_isp_sts0>, <&ps_isp_sts1>, <&ps_isp_pearl>, + <&ps_isp_be>, <&ps_isp_clr>, <&ps_isp_cvd>, + <&ps_isp_cvm>; + memory-region = <&isp_heap>; + memory-region-names = "heap"; + apple,dart-vm-size = <0x0 0xa0000000>; + apple,platform-id = <0x70>; + apple,temporal-filter = <1>; + + sensor-presets { + preset0 { + apple,config-index = <1>; + apple,input-size = <1920 1080>; + apple,output-size = <1920 1080>; + apple,crop = <0 0 1920 1080>; + }; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/mfd/apple,smc.yaml b/Documentation/devicetree/bindings/mfd/apple,smc.yaml index ed6b02086c33ca..31d1940f9d3dc0 100644 --- a/Documentation/devicetree/bindings/mfd/apple,smc.yaml +++ b/Documentation/devicetree/bindings/mfd/apple,smc.yaml @@ -21,6 +21,7 @@ properties: - apple,t6020-smc - apple,t8122-smc - apple,t8140-smc + - apple,t8142-smc - const: apple,t8103-smc - items: - enum: diff --git a/Documentation/devicetree/bindings/mux/apple,display-crossbar.yaml b/Documentation/devicetree/bindings/mux/apple,display-crossbar.yaml new file mode 100644 index 00000000000000..eb80cf382a1880 --- /dev/null +++ b/Documentation/devicetree/bindings/mux/apple,display-crossbar.yaml @@ -0,0 +1,47 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/mux/apple,display-crossbar.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple display crossbar + +maintainers: + - Sven Peter + +description: + Routes display pipelines to the DisplayPort transmitters and inputs + of an Apple Type-C PHY. Each crossbar exposes three mux controls, + selected by the consumer's mux specifier. + +allOf: + - $ref: mux-controller.yaml# + +properties: + compatible: + enum: + - apple,t8103-display-crossbar + - apple,t8112-display-crossbar + - apple,t6000-display-crossbar + - apple,t6020-display-crossbar + + reg: + maxItems: 1 + + '#mux-control-cells': + const: 1 + +required: + - compatible + - reg + - '#mux-control-cells' + +unevaluatedProperties: false + +examples: + - | + mux-controller@b04c000 { + compatible = "apple,t6020-display-crossbar"; + reg = <0xb04c000 0x4000>; + #mux-control-cells = <1>; + }; diff --git a/Documentation/devicetree/bindings/npu/apple,ane.yaml b/Documentation/devicetree/bindings/npu/apple,ane.yaml index ebca460a81bc14..79cf942654f7bc 100644 --- a/Documentation/devicetree/bindings/npu/apple,ane.yaml +++ b/Documentation/devicetree/bindings/npu/apple,ane.yaml @@ -14,7 +14,8 @@ description: SoCs. It accesses memory through three DARTs. On T8103 and T6000 the host drives the engine through its register window. On T602x and T8112 a coprocessor runs the ANE firmware and the host talks to it through an ASC - mailbox. + mailbox. T8140 has two interrupts and one system power domain; its + coprocessor uses a different firmware interface. properties: compatible: @@ -25,6 +26,7 @@ properties: - apple,t6021-ane - apple,t6022-ane - apple,t8112-ane + - apple,t8140-ane reg: minItems: 1 @@ -45,7 +47,8 @@ properties: - const: fuse interrupts: - maxItems: 1 + minItems: 1 + maxItems: 2 interrupt-names: items: @@ -57,8 +60,8 @@ properties: power-domains: description: The power states the engine needs, five on T8103 and T6000, eight on - T602x and seven on T8112. - minItems: 5 + T602x, seven on T8112 and one system domain on T8140. + minItems: 1 maxItems: 8 resets: @@ -80,11 +83,40 @@ required: - reg - reg-names - interrupts - - interrupt-names - iommus - power-domains allOf: + - if: + properties: + compatible: + contains: + const: apple,t8140-ane + then: + properties: + reg: + maxItems: 1 + reg-names: + maxItems: 1 + interrupts: + minItems: 2 + interrupt-names: false + iommus: + minItems: 3 + power-domains: + maxItems: 1 + resets: false + mboxes: false + memory-region: false + else: + properties: + interrupts: + maxItems: 1 + power-domains: + minItems: 5 + required: + - interrupt-names + - if: properties: compatible: @@ -149,6 +181,27 @@ allOf: additionalProperties: false examples: + - | + #include + #include + + soc { + #address-cells = <2>; + #size-cells = <2>; + + ane@400000000 { + compatible = "apple,t8140-ane"; + reg = <0x4 0x0 0x0 0x2000000>; + reg-names = "engine"; + interrupt-parent = <&aic>; + interrupts = , + ; + iommus = <&ane_dart0 0>, <&ane_dart1 0>, <&ane_dart2 0>; + power-domains = <&ps_ane_sys>; + status = "disabled"; + }; + }; + - | #include #include diff --git a/Documentation/devicetree/bindings/nvme/apple,nvme-ans.yaml b/Documentation/devicetree/bindings/nvme/apple,nvme-ans.yaml index 5449db3b250802..fc8a4afd5add47 100644 --- a/Documentation/devicetree/bindings/nvme/apple,nvme-ans.yaml +++ b/Documentation/devicetree/bindings/nvme/apple,nvme-ans.yaml @@ -85,30 +85,7 @@ properties: The SART address filter is documented in iommu/apple,sart.yaml. - apple,rtkit-handoff: - $ref: /schemas/types.yaml#/definitions/string - const: nvme-disabled-v1 - description: | - Added at boot by a bootloader that ran the ANS2 firmware and hands it - over still running instead of shutting it down. It describes the - state that bootloader left behind, not the hardware, and must not be - written into a board device tree source. Its presence states that the - firmware keeps a negotiated RTKit session, that all NVMe controller - DMA has stopped (CC.EN and CSTS.RDY clear), that the SART entries the - bootloader programmed are kept, and that every buffer shared with the - firmware is reserved in the memory map without no-map. - allOf: - - if: - properties: - compatible: - not: - contains: - const: apple,t8140-nvme-ans2 - then: - properties: - apple,rtkit-handoff: false - - if: properties: compatible: diff --git a/Documentation/devicetree/bindings/nvmem/apple,spmi-nvmem.yaml b/Documentation/devicetree/bindings/nvmem/apple,spmi-nvmem.yaml index 7a4114c7fbd197..09cdd776f8d8bd 100644 --- a/Documentation/devicetree/bindings/nvmem/apple,spmi-nvmem.yaml +++ b/Documentation/devicetree/bindings/nvmem/apple,spmi-nvmem.yaml @@ -27,6 +27,9 @@ properties: reg: maxItems: 1 + rtc: + $ref: /schemas/rtc/apple,j700-pmu-rtc.yaml# + required: - compatible - reg diff --git a/Documentation/devicetree/bindings/phy/apple,atcphy.yaml b/Documentation/devicetree/bindings/phy/apple,atcphy.yaml index c42b9c56842967..3e36244ef4d5b3 100644 --- a/Documentation/devicetree/bindings/phy/apple,atcphy.yaml +++ b/Documentation/devicetree/bindings/phy/apple,atcphy.yaml @@ -50,6 +50,11 @@ allOf: reg-names: maxItems: 5 apple,tunable-usb2phy-reg-dflt: false + apple,tunable-dp-common-pre: false + apple,tunable-dp-lpdptx: false + apple,tunable-dp-common-post: false + patternProperties: + "^apple,dp-training-table(-rbr|-hbr|-hbr2|-hbr3)?$": false $defs: apple,tunable: @@ -78,6 +83,7 @@ properties: - const: apple,t8103-atcphy - const: apple,t8103-atcphy - const: apple,t8140-atcphy + - const: apple,t8142-atcphy reg: minItems: 5 @@ -178,6 +184,27 @@ properties: Note that lane here refers to a USB RX and TX pair re-used for DisplayPort and not to an individual DisplayPort differential lane. + apple,tunable-dp-common-pre: + $ref: "#/$defs/apple,tunable" + maxItems: 5461 + description: > + T8140 DisplayPort core setup applied before the LPDPTX list. The + bootloader supplies device-specific values for the core register region. + + apple,tunable-dp-lpdptx: + $ref: "#/$defs/apple,tunable" + maxItems: 5461 + description: > + T8140 DisplayPort setup for the lpdptx register region, applied between + the common-pre and common-post lists. + + apple,tunable-dp-common-post: + $ref: "#/$defs/apple,tunable" + maxItems: 5461 + description: > + T8140 DisplayPort core setup applied after the LPDPTX list. All three + DisplayPort lists are needed before selecting a DisplayPort PHY mode. + apple,tunable-usb2phy-reg-dflt: $ref: "#/$defs/apple,tunable" description: > @@ -185,6 +212,16 @@ properties: T8140, copied from Apple's Device Tree by the bootloader. Unlike the other tunables these are applied to the usb2phy-reg region. +patternProperties: + "^apple,dp-training-table(-rbr|-hbr|-hbr2|-hbr3)?$": + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 16 + maxItems: 16 + description: > + Firmware-provided T8140 drive presets indexed by voltage swing times + four plus pre-emphasis. The unsuffixed table is the required fallback; + a link-rate suffix selects an optional rate-specific override. + required: - compatible - reg diff --git a/Documentation/devicetree/bindings/power/apple,pmc.yaml b/Documentation/devicetree/bindings/power/apple,pmc.yaml new file mode 100644 index 00000000000000..a57b90806511b1 --- /dev/null +++ b/Documentation/devicetree/bindings/power/apple,pmc.yaml @@ -0,0 +1,155 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/power/apple,pmc.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple SoC PMC voter interface + +maintainers: + - Diljit Singh + +description: | + The PMC arbitrates the performance states of shared rails, such as the + DRAM controller and the SoC fabric, between voter agents. Each agent has a + perf-state floor word with one 4-bit state per rail, and each rail has an + interface enable word with one bit per agent. The PMC only honours an + agent's floor on rails where its interface is enabled. + + Static agents are enabled for as long as the PMC is bound. A dynamic + agent is a power domain: it is enabled after its parent domain powers on + and disabled before its parent domain powers off. The host performance + controller's agent is described by a child node that raises its floor + while the CPU clusters or the GPU are busy. + +properties: + compatible: + const: apple,t8140-pmc + + reg: + items: + - description: Perf-state floor words of the voter agents + - description: | + Per-agent words holding the floor the PMC honours for each agent, + and per-rail interface enable words + + reg-names: + items: + - const: voters + - const: rails + + power-domains: + maxItems: 1 + + '#address-cells': + const: 1 + + '#size-cells': + const: 0 + + apple,static-voters: + $ref: /schemas/types.yaml#/definitions/uint32-array + description: Voter agents to enable on every rail. + minItems: 1 + maxItems: 32 + items: + maximum: 31 + + '#cooling-cells': + const: 2 + description: | + The PMC can limit the DRAM controller (DCS) rail. Cooling state 0 leaves + the rail to the agents' votes; each further state lowers the highest + DCS state the PMC honours for them by one, down to the lowest state. + + apple,dcs-power-microwatt: + description: | + Power the system draws at each DCS performance state above what it + draws at the lowest one, from the lowest state up, measured under a + load whose speed follows the DCS state. With it, the DCS limit is a + power actor that the power allocator thermal governor can drive. + minItems: 2 + maxItems: 16 + +patternProperties: + '^voter@[0-9a-f]+$': + type: object + description: A dynamic voter agent, provided as a power domain. + additionalProperties: false + + properties: + reg: + maxItems: 1 + description: Voter agent index. + + label: + description: Name of the power domain. + + '#power-domain-cells': + const: 0 + + power-domains: + maxItems: 1 + description: Domain that must be powered while the agent is enabled. + + required: + - reg + - '#power-domain-cells' + + '^performance-controller@[0-9a-f]+$': + type: object + description: The host performance controller's voter agent. + additionalProperties: false + + properties: + compatible: + const: apple,t8140-clpc + + reg: + maxItems: 1 + description: Voter agent index. + + apple,gpu-power-state: + $ref: /schemas/types.yaml#/definitions/phandle + description: PMGR power state of the GPU, used as a demand input. + + required: + - compatible + - reg + +required: + - compatible + - reg + - reg-names + +additionalProperties: false + +examples: + - | + soc { + #address-cells = <2>; + #size-cells = <2>; + + power-management@302800000 { + compatible = "apple,t8140-pmc"; + reg = <0x3 0x02800000 0x0 0x10000>, + <0x3 0x028aa000 0x0 0x3000>; + reg-names = "voters", "rails"; + #address-cells = <1>; + #size-cells = <0>; + #cooling-cells = <2>; + apple,static-voters = <0 1 3>; + apple,dcs-power-microwatt = <0 919000 1453000 2058000 2407000>; + + performance-controller@1 { + compatible = "apple,t8140-clpc"; + reg = <1>; + }; + + voter@2 { + reg = <2>; + label = "gfx"; + #power-domain-cells = <0>; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/power/apple,pmgr-pwrstate.yaml b/Documentation/devicetree/bindings/power/apple,pmgr-pwrstate.yaml index 0e9e1fbcd3b100..d86c8b68d1b48a 100644 --- a/Documentation/devicetree/bindings/power/apple,pmgr-pwrstate.yaml +++ b/Documentation/devicetree/bindings/power/apple,pmgr-pwrstate.yaml @@ -50,6 +50,7 @@ properties: - apple,t8122-pmgr-pwrstate - apple,t8132-pmgr-pwrstate - apple,t8140-pmgr-pwrstate + - apple,t8142-pmgr-pwrstate - const: apple,t8103-pmgr-pwrstate reg: @@ -78,6 +79,13 @@ properties: Forces this power domain to always be powered up. type: boolean + apple,firmware-owned: + description: + The boot loader hands over the coprocessor in this domain running, and its + firmware owns the domain's power state; writing it faults. The OS keeps the + domain on and never changes its power target, automatic gating or reset. + type: boolean + apple,min-state: description: Specifies the minimum power state for auto-PM. diff --git a/Documentation/devicetree/bindings/power/apple,t8140-pmp-report.yaml b/Documentation/devicetree/bindings/power/apple,t8140-pmp-report.yaml new file mode 100644 index 00000000000000..781afa1ad9d9f1 --- /dev/null +++ b/Documentation/devicetree/bindings/power/apple,t8140-pmp-report.yaml @@ -0,0 +1,101 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/power/apple,t8140-pmp-report.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple T8140 PMP report aperture + +maintainers: + - Ryan Murray + +description: | + The PMP report aperture contains a read window and an update window. + The PMP running bit qualifies the per-cluster fast-die effort. Lane 1 + tracks P-cluster load in Linux measurements. Lane 0 is assigned to the + E cluster from the Apple device tree order and has not been observed + changing. Only documented report entries may be read; the aperture is + not a contiguous readable array of registers. + +properties: + compatible: + const: apple,t8140-pmp-v2-report + + reg: + maxItems: 1 + + power-domains: + maxItems: 1 + + '#address-cells': + const: 1 + + '#size-cells': + const: 0 + +patternProperties: + '^report@[0-9a-f]+$': + type: object + description: | + A device whose power state the host reports to the PMP, provided as a + power domain. The host sets the device's request bit after its parent + domain powers on and clears it before the parent powers off, and waits + for the PMP to acknowledge while the PMP runs. + additionalProperties: false + + properties: + compatible: + items: + - const: apple,t8140-pmp-v2-report-entry + - const: apple,t6000-pmp-v2-report-entry + + reg: + maxItems: 1 + description: PMP soc-device index of the device. + + label: + description: Name of the power domain. + + '#power-domain-cells': + const: 0 + + power-domains: + maxItems: 1 + + apple,always-on: + type: boolean + description: Report the device as on from probe. + + required: + - compatible + - reg + - label + - '#power-domain-cells' + +required: + - compatible + - reg + +additionalProperties: false + +examples: + - | + soc { + #address-cells = <2>; + #size-cells = <2>; + + pmp-report@3003c0000 { + compatible = "apple,t8140-pmp-v2-report"; + reg = <3 0x003c0000 0 0x14000>; + #address-cells = <1>; + #size-cells = <0>; + + report@9 { + compatible = "apple,t8140-pmp-v2-report-entry", + "apple,t6000-pmp-v2-report-entry"; + reg = <0x9>; + label = "pmp-gfx"; + #power-domain-cells = <0>; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/pwm/apple,s5l-fpwm.yaml b/Documentation/devicetree/bindings/pwm/apple,s5l-fpwm.yaml index ae8baa5f7bcdc4..3b3cad92b601fa 100644 --- a/Documentation/devicetree/bindings/pwm/apple,s5l-fpwm.yaml +++ b/Documentation/devicetree/bindings/pwm/apple,s5l-fpwm.yaml @@ -19,6 +19,7 @@ properties: - apple,t8103-fpwm - apple,t8112-fpwm - apple,t8122-fpwm + - apple,t8142-fpwm - apple,t8132-fpwm - apple,t6000-fpwm - apple,t6020-fpwm diff --git a/Documentation/devicetree/bindings/reset/apple,t8103-cio-reset.yaml b/Documentation/devicetree/bindings/reset/apple,t8103-cio-reset.yaml index 5f5be0636b369b..789e55fdaf384d 100644 --- a/Documentation/devicetree/bindings/reset/apple,t8103-cio-reset.yaml +++ b/Documentation/devicetree/bindings/reset/apple,t8103-cio-reset.yaml @@ -19,13 +19,17 @@ description: | the PMGR MMIO region it is represented as a sub-node of the PMGR syscon with reg used as an offset into it. - In both cases the reset id is the CIO port index on the die. + On t8142, a standalone register page contains a shared request register + with alternating request and busy bits for four ports. + + In all cases the reset id is the CIO port index on the die. properties: compatible: oneOf: - enum: - apple,t8103-cio-reset + - apple,t8142-cio-reset - apple,t6000-cio-reset - items: - const: apple,t8112-cio-reset diff --git a/Documentation/devicetree/bindings/rtc/apple,j700-pmu-rtc.yaml b/Documentation/devicetree/bindings/rtc/apple,j700-pmu-rtc.yaml new file mode 100644 index 00000000000000..b1e289c06fe338 --- /dev/null +++ b/Documentation/devicetree/bindings/rtc/apple,j700-pmu-rtc.yaml @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/rtc/apple,j700-pmu-rtc.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple J700 Baku PMU real-time clock + +maintainers: + - Ryan Murray + +description: | + Baku exposes a 65536 Hz counter and an alarm comparator over SPMI. + The clock's whole-second offset is shared with the SMC RTC through an + eight-byte little-endian NVMEM cell. This function is a child of the + PMIC NVMEM provider and uses that provider's SPMI register map. + +allOf: + - $ref: rtc.yaml# + +properties: + compatible: + const: apple,j700-pmu-rtc + + interrupts: + maxItems: 1 + + nvmem-cells: + maxItems: 1 + description: The shared eight-byte whole-second RTC offset. + + nvmem-cell-names: + items: + - const: rtc_offset + + wakeup-source: true + +required: + - compatible + - interrupts + - nvmem-cells + - nvmem-cell-names + +unevaluatedProperties: false + +examples: + - | + #include + #include + + pmic@e { + compatible = "apple,baku-pmic", "apple,spmi-nvmem"; + reg = <0xe SPMI_USID>; + + rtc { + compatible = "apple,j700-pmu-rtc"; + interrupt-parent = <&spmi>; + interrupts = <2 IRQ_TYPE_EDGE_RISING>; + nvmem-cells = <&rtc_offset>; + nvmem-cell-names = "rtc_offset"; + wakeup-source; + }; + }; diff --git a/Documentation/devicetree/bindings/soc/apple/apple,dockchannel.yaml b/Documentation/devicetree/bindings/soc/apple/apple,dockchannel.yaml index 40831b082bf28f..a3362df34e148b 100644 --- a/Documentation/devicetree/bindings/soc/apple/apple,dockchannel.yaml +++ b/Documentation/devicetree/bindings/soc/apple/apple,dockchannel.yaml @@ -28,6 +28,7 @@ properties: - apple,t8112-dockchannel - apple,t8122-dockchannel - apple,t8140-dockchannel + - apple,t8142-dockchannel - const: apple,dockchannel reg: diff --git a/Documentation/devicetree/bindings/soc/apple/apple,pmp.yaml b/Documentation/devicetree/bindings/soc/apple/apple,pmp.yaml new file mode 100644 index 00000000000000..997761ed5ad4de --- /dev/null +++ b/Documentation/devicetree/bindings/soc/apple/apple,pmp.yaml @@ -0,0 +1,94 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/soc/apple/apple,pmp.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple power management co-processor + +maintainers: + - Ryan Murray + +description: | + The PMP runs firmware loaded by the bootloader. Linux starts it through + RTKit and supplies calibration copied from the Apple device tree. The + bootloader must initialize the PMP DART address protection filter before + Linux starts the firmware. Firmware-owned PMC power state must be retained. + +properties: + compatible: + oneOf: + - const: apple,t6000-pmp-v2 + - items: + - const: apple,t8140-pmp-v2 + - const: apple,t6000-pmp-v2 + + reg: + items: + - description: Firmware SRAM + - description: ASC control registers + + reg-names: + items: + - const: pmp + - const: asc + + mboxes: + maxItems: 1 + + mbox-names: + const: mbox + + iommus: + maxItems: 1 + + power-domains: + maxItems: 1 + + apple,board-id: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Board identifier copied by the bootloader. + + apple,dram-vendor-id: + $ref: /schemas/types.yaml#/definitions/uint32 + description: DRAM vendor identifier copied by the bootloader. + + apple,dram-capacity: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Firmware DRAM capacity value copied by the bootloader. + + apple,pio-ranges: + $ref: /schemas/types.yaml#/definitions/uint64-array + minItems: 2 + description: Host physical address and size pairs mapped into firmware PIO space. + +patternProperties: + '^apple,tunable-': + $ref: /schemas/types.yaml#/definitions/uint8-array + description: Opaque firmware calibration copied from the Apple device tree. + +required: + - compatible + - reg + - reg-names + - mboxes + - mbox-names + - iommus + +additionalProperties: false + +examples: + - | + soc { + #address-cells = <2>; + #size-cells = <2>; + + pmp@300500000 { + compatible = "apple,t8140-pmp-v2", "apple,t6000-pmp-v2"; + reg = <3 0x00500000 0 0xa0000>, <3 0x00e00000 0 0x4000>; + reg-names = "pmp", "asc"; + mboxes = <&pmp_mbox>; + mbox-names = "mbox"; + iommus = <&pmp_dart 0>; + }; + }; diff --git a/Documentation/devicetree/bindings/soc/apple/apple,rtk-helper-asc4.yaml b/Documentation/devicetree/bindings/soc/apple/apple,rtk-helper-asc4.yaml index 6db0197b4e8647..da78694e374661 100644 --- a/Documentation/devicetree/bindings/soc/apple/apple,rtk-helper-asc4.yaml +++ b/Documentation/devicetree/bindings/soc/apple/apple,rtk-helper-asc4.yaml @@ -28,12 +28,14 @@ properties: - apple,t8112-mtp - apple,t8122-mtp - apple,t8140-mtp + - apple,t8142-mtp - enum: - apple,t6020-rtk-helper-asc4 - apple,t6030-rtk-helper-asc4 - apple,t8112-rtk-helper-asc4 - apple,t8122-rtk-helper-asc4 - apple,t8140-rtk-helper-asc4 + - apple,t8142-rtk-helper-asc4 - const: apple,mtp - const: apple,rtk-helper-asc4 diff --git a/Documentation/devicetree/bindings/soc/apple/apple,sep.yaml b/Documentation/devicetree/bindings/soc/apple/apple,sep.yaml index 989ee4b9e9078a..83ad3aa31a4b7e 100644 --- a/Documentation/devicetree/bindings/soc/apple/apple,sep.yaml +++ b/Documentation/devicetree/bindings/soc/apple/apple,sep.yaml @@ -33,7 +33,13 @@ properties: maxItems: 1 power-domains: - maxItems: 1 + minItems: 1 + items: + - description: The SEP's own power state. + - description: + A power state the SEP depends on while it runs. On T8140 this is + the GPIO power state; the SEP stops servicing its mailbox for good + once that is switched off. memory-region: maxItems: 1 diff --git a/Documentation/devicetree/bindings/soc/apple/apple,t8140-mcc.yaml b/Documentation/devicetree/bindings/soc/apple/apple,t8140-mcc.yaml new file mode 100644 index 00000000000000..e4c5688a01d7f9 --- /dev/null +++ b/Documentation/devicetree/bindings/soc/apple/apple,t8140-mcc.yaml @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/soc/apple/apple,t8140-mcc.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple T8140 memory cache controller + +maintainers: + - Ryan Murray + +description: + The memory cache controller of the Apple T8140 SoC holds the system level + cache in front of DRAM, split into planes. The boot firmware enables and + powers the cache ways of every plane but leaves allocation disabled, so the + cache holds no data until the operating system allows allocation. + +properties: + compatible: + const: apple,t8140-mcc + + reg: + description: The register block of each cache plane. + minItems: 1 + maxItems: 8 + + apple,boot-dma-devices: + $ref: /schemas/types.yaml#/definitions/phandle-array + items: + maxItems: 1 + description: + Devices that take over DMA the bootloader left running. Cache allocation + is enabled only once each of them is bound to its driver. + +required: + - compatible + - reg + +additionalProperties: false + +examples: + - | + soc { + #address-cells = <2>; + #size-cells = <2>; + + display: display-subsystem { + }; + + memory-controller@220000000 { + compatible = "apple,t8140-mcc"; + reg = <0x2 0x20000000 0x0 0x4000>, + <0x2 0x20040000 0x0 0x4000>, + <0x2 0x20080000 0x0 0x4000>, + <0x2 0x200c0000 0x0 0x4000>; + apple,boot-dma-devices = <&display>; + }; + }; diff --git a/Documentation/devicetree/bindings/sound/apple,t8140-aop-audio.yaml b/Documentation/devicetree/bindings/sound/apple,t8140-aop-audio.yaml index a4e10cbb4deea3..b023080e48c7fc 100644 --- a/Documentation/devicetree/bindings/sound/apple,t8140-aop-audio.yaml +++ b/Documentation/devicetree/bindings/sound/apple,t8140-aop-audio.yaml @@ -20,6 +20,9 @@ description: | The remaining audio data moves over the named ADMAC channels while its AOP service is active. The parent AOP binding describes the firmware transport. +allOf: + - $ref: dai-common.yaml# + properties: compatible: const: apple,t8140-aop-audio @@ -96,7 +99,7 @@ required: - apple,chassis-name - apple,machine-kind -additionalProperties: false +unevaluatedProperties: false examples: - | diff --git a/Documentation/devicetree/bindings/sound/cirrus,cs42l42.yaml b/Documentation/devicetree/bindings/sound/cirrus,cs42l42.yaml index 28a5303cf72046..31cdd309f33ee3 100644 --- a/Documentation/devicetree/bindings/sound/cirrus,cs42l42.yaml +++ b/Documentation/devicetree/bindings/sound/cirrus,cs42l42.yaml @@ -15,6 +15,9 @@ description: high-dynamic-range ADC for audio capture. There is an integrated headset detection block. + On Apple Silicon machines the supplies of the CS42L83 are platform-managed + rails that are not described as regulators, so they are optional there. + properties: compatible: enum: @@ -29,8 +32,6 @@ properties: '#sound-dai-cells': const: 0 - sound-name-prefix: true - VP-supply: description: VP power supply. @@ -192,13 +193,22 @@ properties: required: - compatible - reg - - VP-supply - - VCP-supply - - VD_FILT-supply - - VL-supply - - VA-supply -additionalProperties: false +allOf: + - $ref: dai-common.yaml# + - if: + properties: + compatible: + const: cirrus,cs42l42 + then: + required: + - VP-supply + - VCP-supply + - VD_FILT-supply + - VL-supply + - VA-supply + +unevaluatedProperties: false examples: - | diff --git a/Documentation/devicetree/bindings/spi/apple,qspi.yaml b/Documentation/devicetree/bindings/spi/apple,qspi.yaml new file mode 100644 index 00000000000000..4d33549e7bf396 --- /dev/null +++ b/Documentation/devicetree/bindings/spi/apple,qspi.yaml @@ -0,0 +1,62 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/spi/apple,qspi.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple ARM SoC QSPIMC controller + +maintainers: + - Ryan Murray + +description: | + Apple QSPIMC is a SPI-memory-only controller with a packed command FIFO. + It is distinct from the ordinary Apple SPIMC controller despite serving a + similar set of SPI NOR devices. + +allOf: + - $ref: spi-controller.yaml# + +properties: + compatible: + const: apple,t8140-qspi + + reg: + maxItems: 1 + + clocks: + maxItems: 1 + + interrupts: + maxItems: 1 + + power-domains: + maxItems: 1 + +required: + - compatible + - reg + - clocks + - interrupts + +unevaluatedProperties: false + +examples: + - | + #include + #include + + soc { + #address-cells = <2>; + #size-cells = <2>; + + spi@385110000 { + compatible = "apple,t8140-qspi"; + reg = <0x3 0x85110000 0x0 0x4800>; + interrupt-parent = <&aic>; + interrupts = ; + clocks = <&clk_200m>; + #address-cells = <1>; + #size-cells = <0>; + }; + }; diff --git a/Documentation/devicetree/bindings/spmi/apple,spmi.yaml b/Documentation/devicetree/bindings/spmi/apple,spmi.yaml index f0f7d21d031cab..5a52f808b6f5b8 100644 --- a/Documentation/devicetree/bindings/spmi/apple,spmi.yaml +++ b/Documentation/devicetree/bindings/spmi/apple,spmi.yaml @@ -34,6 +34,7 @@ properties: - apple,t6000-spmi - apple,t8112-spmi - const: apple,spmi + - const: apple,t8142-spmi reg: maxItems: 1 diff --git a/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-acio.yaml b/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-acio.yaml index 7787adb6805ccb..33960878649c21 100644 --- a/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-acio.yaml +++ b/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-acio.yaml @@ -29,6 +29,7 @@ properties: compatible: oneOf: - const: apple,t8103-usb4-acio + - const: apple,t8142-usb4-acio - items: - enum: - apple,t6000-usb4-acio diff --git a/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-nhi.yaml b/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-nhi.yaml index 4f95132cebf303..49b60aaa8deafb 100644 --- a/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-nhi.yaml +++ b/Documentation/devicetree/bindings/thunderbolt/apple,t8103-usb4-nhi.yaml @@ -21,6 +21,7 @@ properties: compatible: oneOf: - const: apple,t8103-usb4-nhi + - const: apple,t8142-usb4-nhi - items: - enum: - apple,t6000-usb4-nhi @@ -32,6 +33,7 @@ properties: - const: apple,t8103-usb4-nhi reg: + minItems: 1 items: - description: NHI registers with the transmit and receive rings - description: @@ -39,6 +41,7 @@ properties: accepted by each receive ring reg-names: + minItems: 1 items: - const: nhi - const: pdf @@ -85,7 +88,8 @@ properties: all other rings which are only used for XDomain connections to other hosts. - T6030, T6031 and T8122 use 24 entries. These are most likely one DART + T8142 uses one DART stream per ring: TX streams 0-11 and RX streams 16-27. + T6030, T6031 and T8122 also use 24 entries. These are most likely one DART stream for each of the 12 transmit and 12 receive rings. apple,thunderbolt-drom: @@ -115,6 +119,23 @@ required: - apple,thunderbolt-drom allOf: + - if: + properties: + compatible: + contains: + const: apple,t8142-usb4-nhi + then: + properties: + reg: + maxItems: 1 + reg-names: + maxItems: 1 + else: + properties: + reg: + minItems: 2 + reg-names: + minItems: 2 - if: properties: compatible: @@ -123,6 +144,7 @@ allOf: - apple,t6030-usb4-nhi - apple,t6031-usb4-nhi - apple,t8122-usb4-nhi + - apple,t8142-usb4-nhi then: properties: iommus: diff --git a/Documentation/devicetree/bindings/usb/apple,dwc3.yaml b/Documentation/devicetree/bindings/usb/apple,dwc3.yaml index 44e8b33576ddac..9dc0adc535bea1 100644 --- a/Documentation/devicetree/bindings/usb/apple,dwc3.yaml +++ b/Documentation/devicetree/bindings/usb/apple,dwc3.yaml @@ -25,6 +25,7 @@ properties: - apple,t6020-dwc3 - apple,t8112-dwc3 - apple,t8140-dwc3 + - apple,t8142-dwc3 - const: apple,t8103-dwc3 - const: apple,t8103-dwc3 diff --git a/Documentation/devicetree/bindings/usb/apple,sn201202x.yaml b/Documentation/devicetree/bindings/usb/apple,sn201202x.yaml index 133a71bbedb7cd..9b1a566fd14373 100644 --- a/Documentation/devicetree/bindings/usb/apple,sn201202x.yaml +++ b/Documentation/devicetree/bindings/usb/apple,sn201202x.yaml @@ -15,6 +15,8 @@ description: | A variant of tps6598x controller also known as Apple ACE3 present on hardware with Apple SoCs starting from the M3. + The controller can be polled when interrupt resources are not supplied. + properties: compatible: enum: @@ -43,8 +45,10 @@ properties: required: - compatible - reg - - interrupts - - interrupt-names + +dependentRequired: + interrupts: [interrupt-names] + interrupt-names: [interrupts] additionalProperties: false diff --git a/Documentation/gpu/apple-dcp-backlight.rst b/Documentation/gpu/apple-dcp-backlight.rst new file mode 100644 index 00000000000000..e486fcf4840df4 --- /dev/null +++ b/Documentation/gpu/apple-dcp-backlight.rst @@ -0,0 +1,163 @@ +.. SPDX-License-Identifier: GPL-2.0-only OR MIT + +Apple DCP H17P backlight integration +=================================== + +The backlight policy implements the clean display specification's BACKLIGHT-1 +through BACKLIGHT-4, with IOMFB-3 and LIFETIME-1 through LIFETIME-3 constraints. +It extends the existing Asahi backlight driver. The measured internal H17P profile carries brightness activation through the +central present queue. It latches the first bounded powerlog hint without +sending a brightness update and leaves registration unavailable if there is +no hint. H17G and older firmware retain their existing paths. Physical loader +level continuity remains a measurement gate. + +Admission and startup +--------------------- + +The measured profile connects the public powerlog parser to +``iomfb_configure_backlight_h17p()`` after linking the CRTC. Generic users of +``dcp_backlight_configure()`` must still supply validated inputs from process +context. Its arguments +are the validated maximum commanded nits, an optional inherited level, an +optional documented ADT default, and a callback that schedules the central +outbound queue. Maximum must fit the backlight core's signed integer range. +An inherited zero is valid and takes precedence over the default. With neither +level available, configuration returns ``-ENODATA``. The policy backlight is registered only after configuration succeeds. +Configuration must precede any existing backlight registration; it returns +``-EBUSY`` if a device has already been published. Configuration does not send a command. + +The device exposes ``apple-panel-bl`` with platform type, linear scale and nits +units. Its maximum is the supplied ceiling; the older 509-nit DAC cap does not +apply to configured H17P policy. Consumer matching accepts the DCP device and its DRM device. +Registration freezes takeover before publishing the backlight. A decoded +powerlog value may seed the state through ``dcp_backlight_seed()`` before that +point. All samples after Linux takes control, including per-frame zeros, are +ignored. The older property and powerlog parsers do not seed an active policy implicitly. +Gather a sample before configuration and pass it as the inherited argument if +asynchronous registration could otherwise win the race. + +``brightness`` is the requested restore target. ``actual_brightness`` is the +last completed commanded level, initially the inherited/default estimate. This +is not a photometric measurement. Updates step immediately; no kernel fade or +initial-brightness/blank-level override is introduced. + +Queue and lifetime contract +--------------------------- + +The callback only schedules/coalesces work. It must not submit an RPC directly, +take DRM modeset locks, or run a present inline in a receive/commit callback. +It remains valid until the central queue and backlight workers are drained at +teardown. Policy access is protected by the brightness spinlock; the callback +is called after dropping it. Configuration also takes the registration mutex. + +The single outbound queue handles normal and brightness-only presents. A static +or inactive screen uses the last *accepted* surface, including its cached +geometry and IOVA. It must hold that surface's framebuffer reference and mapping +before calling ``dcp_backlight_prepare()`` with ``have_surface = true``. A DRM +plane state containing a rejected replacement is not an accepted surface. +The policy's boolean is a caller assertion, not a framebuffer reference. +Allocate any retirement bookkeeping during atomic prepare, never here. + +Prepare reserves a sequence and returns commanded nits. ``-ENODATA`` means no +configuration or surface; ``-EBUSY`` means a prior brightness transaction is +outstanding; ``-EALREADY`` means no brightness change is pending. These results +do not clear pending work. The serializer places the returned level only in +admitted fields of the next valid present. A normal present without a brightness +change disables the measured activation fragment, preserving the existing +physical level without resending a level. +The sequence is a software cookie, not a wire present ID. + +Associate the cookie with the central transaction and its strict wire ID. Call +``dcp_backlight_complete(sequence, true)`` only after both accepted submission +and matching completion. On serialization/submission rejection call it with +``false``; actual remains unchanged and the update is pending again. Wrong or +duplicate sequence callbacks have no effect. On a firmware crash retain the +surface and stop new submissions; there is no local reset/retry ladder. +After a transaction drains, inspect ``dcp_backlight_pending()`` and schedule the +next update or defer until the pipe becomes usable. Requests received during a +transaction remain pending. The central queue must bound static-screen latency +to 200 ms; this still requires a hardware timing test. + +A brightness-only completion never authorizes framebuffer retirement, including +retirement associated with a rejected replacement. Existing generic retention +helpers must be called only for the corresponding accepted replacement. This +policy never arms, releases, or unmaps a framebuffer. + +DPMS and suspend +---------------- + +Internal-panel H17P ``dcp_poweroff()`` and ``dcp_poweron()`` change the policy and +schedule the retained-surface queue. They bypass the legacy pipe-off/on path, +including surface clearing and final release. Until configuration, they warn +and leave the pipe intact; they cannot claim a completed hardware blank. +Startup must independently establish/adopt the live pipe before enabling the +policy. External outputs retain their existing power handling. + +DPMS off, a backlight core blank, or backlight core suspend forces effective nits +to zero while preserving the user target. Brightness changes during any blank +update the target. DPMS on restores it only when core blank/suspend is cleared. +The backlight core's suspend/resume callbacks feed this state; the central PM +worker still owns queue draining, domain lifetime and resume order. Do not call +unqualified H17P sleep/pipe-off methods as a suspend fallback. Keep the last +accepted surface mapped across soft-off and resume. + +Required records and qualification +---------------------------------- + +The following SPEC admission records must supply facts before production use: + +* R-ADT: panel ceiling/calibration in commanded nits and a documented default + if no inherited sample is available. No expected 525-nit ceiling is hardcoded. +* R-BACKLIGHT: takeover sample source, units, validity and timing; decoded + powerlog semantics; commanded-nits conversion, valid update/enable fields, + field offsets, widths and byte order in the present. +* R-SWAP: complete surface layout and successful submit/completion correlation + for normal and retained-surface presents, including inactive CRTC handling. +* R-DPMS: which admitted brightness/property actions blank and restore the live + pipe without clearing its surface, plus wake ordering. No new numeric property + or guessed DPMS payload is emitted by this policy. +* R-SERIALIZE: ordered brightness/present/property/diagnostic timeline and busy + deferral through the central queue; static-screen update latency. +* R-TEARDOWN: safe final release and suspend/resume boundary. Soft-off supplies + no such release boundary. + +Run ``apple-dcp-backlight`` KUnit tests with ``CONFIG_DRM_APPLE_KUNIT_TEST``. +They cover takeover/default/zero handling, range validation, twenty DPMS cycles, +cached restore, both suspend/DPMS orders, static-surface deferral, busy sequencing, +rejection and stale completion. They do not qualify firmware fields or GNOME. +Hardware acceptance requires O-BACKLIGHT, O-BACKLIGHT-TAKEOVER (10 Hz PBwo with +no jump above 0.05 W), O-BACKLIGHT-CRC, O-DPMS (twenty cycles), and O-SUSPEND. + +Measured activation and software replay +-------------------------------------- + +Linux-boundary measurements on the default J700 driver establish an 18-byte +fragment at A408 offsets 0x354 through 0x365. Brightness-enabled presents use +``01 00 00 00 01 01 01 00 01 00`` followed by integer little-endian binary64 +nits. Ordinary presents disable the activation and nits fragment. Its individual control +bytes have no separately inferred semantics. The two binary64 ceiling fields +at 0x36e and 0x376 carry the raw panel maximum on every internal present, +including ordinary frames and frames before a takeover hint arrives. Enabled +updates additionally carry unity at 0x37e and 0x3e6; other scalars are zero. +The complete owned block ends at 0x3ed. The observed 525-nit ceiling is distinct +from the 509-nit exposed software range. Missing ceilings refuse serialization. +The older DAC/power words at 0x32f through 0x33b stay zero on this profile, +including before a takeover hint arrives. Enabled zero nits blanks the +panel while keeping the pipe and retained surface alive. Requests while +blanked change the restore target; unblank restores it through the same queue. + +The measurements include sysfs changes, two GNOME Mutter SetBacklight API +steps and one PowerSaveMode off/on cycle. The latter reaches near-zero panel +power while display-pipe power stays at 605 mW. Cached sysfs actual brightness +on the observed driver is not physical readback while blanked. These records +do not establish clean-driver firmware acceptance, physical takeover continuity, +static-screen latency, suspend/resume or a twenty-cycle hardware gate. + +The native fixture extracts the actual registration, takeover and queue +reservation functions and replays captured fragments under AddressSanitizer +and UndefinedBehaviorSanitizer:: + + python3 tools/testing/apple/check-backlight.py --output /path/to/scratch + +It mocks locks, scheduling, DRM registration and transport. The fragment corpus +retains hashes of clean Linux observations, with runtime addresses omitted. diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst index 7aa2a888690832..fb6128d42aa843 100644 --- a/Documentation/mm/index.rst +++ b/Documentation/mm/index.rst @@ -13,6 +13,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`. physical_memory page_tables process_addrs + user_page_size bootmem page_allocation vmalloc diff --git a/Documentation/mm/user_page_size.rst b/Documentation/mm/user_page_size.rst new file mode 100644 index 00000000000000..62ef6ae39e0aca --- /dev/null +++ b/Documentation/mm/user_page_size.rst @@ -0,0 +1,84 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +============================== +Per-process user mapping sizes +============================== + +The experimental mixed-page implementation separates the user mapping +size of an address space from the kernel's native allocation size. +``PAGE_SIZE`` still describes native physical pages. ``mm_page_size(mm)`` +describes the minimum user mapping and protection unit of that address +space. A process retains one mapping size for the lifetime of its ``mm``. +Exec may create a new ``mm`` with a different size; changing a launch +preference never changes an existing mapping. + +The generic interfaces described here are internal development interfaces. +The experimental launch commands still use private prctl numbers; these +are not assigned upstream UAPI. The arm64 backend's supported modes and +current restrictions are documented in +:doc:`../arch/arm64/user4k`. + +Exec policy and inheritance +=========================== + +``ARCH_HAS_USER_PAGE_SIZE`` enables two per-thread policy fields in +``task_struct``: + +``exec_page_shift`` + A one-shot request for the next exec. Zero means no one-shot request. + +``default_exec_page_shift`` + An inherited preference used when no one-shot request overrides it. + Zero means the architecture's ordinary executable policy. + +Fork and thread creation copy both fields. Subsequent changes affect only +the calling thread, including when it shares an ``mm`` with other threads. +An exec by a non-leader uses the execing thread's policy. The fields do not +belong in ``mm_struct``: threads sharing one address space may launch new +images with different preferences. + +Generic prctl handling validates reserved arguments before calling the +architecture validator. A rejected request leaves both fields unchanged. +Setting a native one-shot request while an inherited alternative is active +stores ``PAGE_SHIFT`` explicitly, so the next exec can override that +alternative. Clearing the inherited preference does not consume a pending +one-shot request. + +Exec stages arguments in a private ``mm``. Its initial allocation receives +the one-shot shift, if any; the final executable loader resolves the +inherited preference and any executable-specific policy. An ELF interpreter +must be loaded using the same final mapping size as its executable. +Failure before committing the new image retains both preferences. +``setup_new_exec()`` consumes the one-shot field after the new image is +committed, while retaining the inherited preference. + +Architecture responsibilities +============================= + +An architecture selecting ``ARCH_HAS_USER_PAGE_SIZE`` must implement: + +* ``arch_exec_page_size_shift(size)``: return zero for a native request, a + validated alternative shift, or a negative errno. Validation includes + the supported hardware and kernel configuration, not just whether the + requested byte size is a power of two. +* ``arch_mm_init_exec(mm, shift)``: initialize the private address space's + geometry before allocating its page-table root. Zero denotes native + geometry. The hook receives the request explicitly and does not read + a separate architecture thread-policy field. +* Binary-loader integration that resolves the final executable policy + before exposing the address space. ``bprm_set_page_shift()`` transfers + staged argument bytes when the final choice changes the initial granule. +* Page-table allocation, walking, invalidation, context switching and MM + operations for every size the architecture exposes. + +The architecture retains control over executable compatibility policy. +For example, the arm64 AArch32 loader prefers 4K where the backend supports +it and retains the native ABI when that capability is absent. This is +separate from running an AArch32 kernel: ``arch/arm`` needs its own backend +integration before it can advertise alternative mapping sizes. The generic +exec policy alone does not provide that backend or expose new sizes there. + +Architectures without this capability retain native mappings and reject +the experimental launch-policy commands. ``ARCH_HAS_MM_PGTABLE`` describes +per-mm page-table accessors; it does not by itself promise an executable +userspace ABI. diff --git a/Documentation/networking/device_drivers/wifi/mt7932-neo.rst b/Documentation/networking/device_drivers/wifi/mt7932-neo.rst index 88773c6a89a07b..a5f0f16a9d46fa 100644 --- a/Documentation/networking/device_drivers/wifi/mt7932-neo.rst +++ b/Documentation/networking/device_drivers/wifi/mt7932-neo.rst @@ -1,74 +1,72 @@ .. SPDX-License-Identifier: GPL-2.0-only -MT7932 radio bring-up on the MacBook Neo +MediaTek MT7932 radio on the MacBook Neo ======================================== J700 MT7932 Wi-Fi uses cfg80211 and NetworkManager with firmware-managed -WPA2-CCMP authentication or open networks. Bluetooth uses an opt-in PCIe -transport and remains disabled by default pending physical qualification. +WPA2-CCMP authentication. Bluetooth uses a transport on the radio's second +PCIe function. -The Wi-Fi import is pinned to -``aurora-silicon/linux-neo-cleanroom-eryk-with-wifi``, commit -``610cb463c03f9bc68a5d020e7ca443af1fda4856``, with configuration, scan IE and -WPA2 AKM compatibility fixes. Existing source notices are retained. +The Wi-Fi driver carries configuration, scan IE and WPA2 AKM compatibility +fixes on top of its original import. Existing source notices are retained. Co-authored by DJ (DjDeveloperr), Ace (Acelogic), and Ryan Murray. Platform and configuration -------------------------- Use the separately reviewed T8140 PCIe bootstrap described in -``Documentation/PCI/controller/apple-t8140.rst``. Enumeration is enabled by -default only on the J700/T8140; ``pcie_apple_piodma_diag.enumerate=0`` disables -it. Public m1n1 must populate the J700 ``wifi0`` endpoint's own -``local-mac-address``; a zero placeholder is rejected. +``Documentation/PCI/controller/apple-t8140.rst``. It runs during boot when +``CONFIG_PCIE_APPLE_PIODMA_DIAG=y``. Public m1n1 must populate the J700 +``wifi0`` endpoint's own ``local-mac-address``; a zero placeholder is rejected. -The original hardware-tested configuration included ``CONFIG_MT7932_FULLMAC=m``, -``CONFIG_CFG80211=y``, ``CONFIG_BT_MTK7932_PCIE=y``, ``CONFIG_BT_BREDR=y``, +The tested configuration includes ``CONFIG_MT7932_FULLMAC=m``, +``CONFIG_CFG80211=m``, ``CONFIG_BT_MTK7932_PCIE=m``, ``CONFIG_BT_BREDR=y``, ``CONFIG_BT_LE=y``, ``CONFIG_CRYPTO_AES=y`` and ``CONFIG_CRYPTO_CMAC=y``. -The shared build supports ``CONFIG_PCIE_APPLE=y`` or ``m``, -``CONFIG_PCIE_APPLE_PIODMA_DIAG=y``, ``CONFIG_PCIEASPM=y``, -``CONFIG_MT7932_FULLMAC=m`` and ``CONFIG_BT_MTK7932_PCIE=m`` with modular -Bluetooth/rfkill and sleep/kexec enabled. The active J700 bootstrap refuses sleep -and kexec at runtime while its arena is retained; other Macs keep their normal -behavior. Firmware loading is selected by both radio drivers. This shared -build still requires physical Neo qualification. - -``CONFIG_CRYPTO_CMAC=m`` is supported with the modular Bluetooth core. Generic -Bluetooth SMP allocates ``cmac(aes)`` through the crypto API. Install the crypto -modules and preserve algorithm autoload; if Bluetooth starts in the initramfs, -include its crypto providers there. With ``CONFIG_CRYPTO_AES=y``, this kernel -also registers its AES CMAC implementation when CMAC is configured as a module. -The earlier tested ``CONFIG_CRYPTO_CMAC=y`` is not a Wi-Fi driver requirement. +The Bluetooth transport can be built with modular Bluetooth and system +sleep. The separately selected bootstrap provider has its own lifecycle +restrictions. A runtime guard refuses kexec while radio admission or retained +bootstrap DMA ownership is active. Kernels may still support kexec elsewhere. +The station uses 2.4 GHz channels 1-13 and the 5 GHz channels 36-64, +100-144 and 149-165 that the cfg80211 regulatory domain permits. Radar (DFS) +channels stay passive: the firmware listens on them while the station is not +associated, but never probes or joins an access point there, and scans made +while associated skip them. A channel whose restrictions the driver cannot +apply (reduced power, no OFDM, PSD limits) is disabled rather than used, and +so is a channel the country package does not permit: every power limit of +that channel in ``policy/.bin`` is left undefined. Use the ordinary cfg80211 regulatory database and applicable country policy. -Kernel country 00 uses firmware XZ and its mandatory ``world-XZ.bin`` package. -An explicit country requires its own package; there is no world fallback. - -The host now disables link power states on the two radio endpoints before -driver binding; no global PCIe ASPM performance policy is required. Load -``mt7932-fullmac`` after the root filesystem and local firmware packages are -available. Bluetooth's gate defaults closed; load ``mt7932_bt_pcie`` with the -default ``enable=0``, validate the cold, unbound ``14c3:793b`` function, enable -``/sys/module/mt7932_bt_pcie/parameters/enable``, then request its PCI probe:: - - echo 1 > /sys/module/mt7932_bt_pcie/parameters/enable - echo YOUR_BT_PCI_BDF > /sys/bus/pci/drivers_probe - -Use the actual ``14c3:793b`` PCI address for ``YOUR_BT_PCI_BDF``. The driver -suppresses its individual bind/unbind attributes, so there is no driver -``bind`` file. For an intentionally enabled cold boot, a file under -``/etc/modprobe.d/`` may instead contain:: - - options mt7932_bt_pcie enable=1 - -Install the target unit's Bluetooth inputs before opening the gate. -Do not reprobe after a failed or uncertain Bluetooth admission. The tested -Wi-Fi driver owns function 0 and Bluetooth owns function 1. - -Use a saved NetworkManager open or WPA2 profile matching the current interface. -Audio in the desktop session requires BlueZ, PipeWire, its PulseAudio -compatibility service, WirePlumber and Bluetooth audio plugins in the logged-in -user's session. +The validated first-admission fallback is kernel country 00 with firmware XZ. +A modular cfg80211 loads ``regulatory.db`` from the root filesystem; a +built-in one tries before the root filesystem is mounted and then needs +``iw reg reload``. + +The radio functions appear while the kernel boots, before the root +filesystem is mounted, and ``mt7932-fullmac`` reads its firmware while +probing. Build it as a module so that udev loads it from the root +filesystem, or provide its firmware in the initramfs. The driver disables +ASPM and clock power management on its link before starting the device, +matching the tested admission sequence, which used the PCIe ASPM +performance policy. + +``mt7932_bt_pcie`` defers its probe until ``mt7932-fullmac`` is bound to +function 0, so the Bluetooth firmware always starts after the Wi-Fi firmware, +as in the tested sequence, however the drivers are loaded. Bluetooth therefore +stays unbound while the Wi-Fi driver is missing or has failed, including when +a failed Wi-Fi function stays bound to keep DMA it could not stop. A device link +makes the driver core unbind Bluetooth before Wi-Fi; unloading +``mt7932_bt_pcie`` does not affect Wi-Fi. Wi-Fi pins its module before publishing +DMA pointers, and refuses function reset while its Bluetooth sibling is live. +The probe rejects a function that is +not cold, which is what now prevents a reprobe after a failed or uncertain +admission. The transport also +disables ASPM on the shared link. Like the Wi-Fi driver, it reads its firmware +while probing, so build it as a module. The tested Wi-Fi driver owns function +0 and Bluetooth owns function 1. + +Use a saved NetworkManager WPA2 profile matching the current interface. +KDE audio requires BlueZ, PipeWire, its PulseAudio compatibility service, +WirePlumber and Bluetooth audio plugins in the logged-in user's session. Local firmware inputs --------------------- @@ -86,70 +84,31 @@ Wi-Fi requests the following files under ``mediatek/mt7932/``: its original BLOB container. * ``config-original.bin``: the bounded J7CF package with the original 64 or 65 configuration records, including all required keys. -* ``policy/world-XZ.bin`` for cfg80211 country 00, or ``policy/.bin`` for - the matching explicit uppercase country code: J7RP containers with - original-derived modes and power tables. A country package is mandatory - whenever that country is set; there is no fallback to the world package. - Relabeling a different country's payload is unsupported. +* ``policy/world-XZ.bin`` and, when available, the matching country package + under ``policy/``: J7RP containers with original-derived modes and power + tables. Relabeling a different country's payload is unsupported. Bluetooth requests these files under ``mediatek/``: * ``MT7932B1_OS_TypeB_2.0.177.0_260706180356.bin`` or the supported fallback - ``MT7932B1_OS_TypeB_0.1.133.0_260128190103.bin`` for ROM1. The fallback is - tried only when the preferred file is absent. ROM0 instead requests - ``MT7932B0_OS_TypeB_0.1.44.0_241001003711.bin``; admitting that variant in - software does not qualify it on physical Neo hardware; -* ``j700-mt7932-btcal.bin`` (the measured input is 388 bytes; the driver accepts - nonempty inputs up to 65535 bytes and relies on HCI setup acceptance); + ``MT7932B1_OS_TypeB_0.1.133.0_260128190103.bin``; +* ``j700-mt7932-btcal.bin`` (388 bytes); * ``MT7932_PTB_IzubaA_0.1.0.0_20251021141303.ptx`` (198 bytes); -* ``j700-mt7932-bdaddr.bin`` (the unit's six-byte Bluetooth address, - most-significant byte first; the driver converts it to HCI byte order). +* ``j700-mt7932-bdaddr.bin`` (the unit's six-byte Bluetooth address). Extraction and packaging require the original local assets. Keep these assets and unit identities out of commits and public test reports. -Keep the installed calibration inputs stable while the Wi-Fi driver is bound: -it requests ``oca2.bin`` again for runtime D7 requests and each association. -``config-original.bin`` and country policies require J7CF/J7RP packaging; copying -unconverted source files to those names does not satisfy the format checks. -If loading Wi-Fi in the initramfs, include every fixed input and the selected -country package there explicitly. Firmware metadata includes a country-file -pattern, but an initramfs builder's glob handling may select only one match; -inspect the resulting image rather than assuming every country is included. - -Load the radio drivers after these inputs are available on the real root. -Wi-Fi reports these distinct failure classes: - -* An initialization probe failure can occur after DMA is published but before - an interface is registered. Only ``initialization failed; DMA retired`` - confirms that the checked reset completed and the unbound driver may be - loaded again. A failed - reset retains the binding and DMA; do not reprobe it. -* ``REGULATORY_BLOCKED`` with ``error=-2`` and ``recovery-required=0`` means - the requested policy file is absent before policy submission. Install that - exact file and bring the interface up or retry a scan/connection. An identical - ``iw reg set`` request alone does not trigger a retry in cfg80211. -* A malformed country package reports ``error=-22`` before submission and - is not retried on interface up/scan/connect. Replace the exact named package - with a valid one, then shut down and start again. An unchanged - ``iw reg set`` request is ignored by cfg80211. If ``recovery-required=1`` - is also present, use the external-reset recovery below instead. -* ``recovery-required=1`` or ``RF_FAILED`` is a latched startup/calibration - failure. Installing a file does not clear it in the bound epoch. A full - external reset is required after a failed or uncertain admission. - Repeatable physical network test -------------------------------- -The manual test activates a saved profile and checks scan, open or WPA2 -authentication, DHCP, gateway replies, DNS, HTTPS and kernel diagnostics. -For a desktop session using Hyprland, use ``--desktop none`` for this network test. -The helper restores the profile's original band even after a failure. It is -not run automatically. Run as root, with a fresh attempt name:: +The manual test activates a saved profile and checks scan, WPA2, DHCP, +gateway replies, DNS, HTTPS, KDE and kernel diagnostics. It is not run +automatically. Run as root, with a fresh attempt name:: python3 tools/testing/selftests/drivers/net/mt7932_e2e.py \ --expected-release "$(uname -r)" --profile "YOUR SAVED PROFILE" \ - --connect --band a --attempt five-ghz --desktop none + --connect --band a --attempt five-ghz Use ``--band bg`` for 2.4 GHz. Require ``WIFI_PHYSICAL_NETWORK_E2E_PASS`` in the JSON artifact printed on completion. The artifact records three traffic @@ -158,44 +117,21 @@ samples and response hashes, omitting network names, addresses and credentials. Limitations ----------- -* Wi-Fi station operation is limited to 2.4 GHz channels 1--13 and 5 GHz - channels 36, 40, 44 and 48, subject to the applicable regulatory rules. - 5 GHz supports 20/40/80 MHz channel layouts within this range. Channels - 149--165 and 6 GHz are not exposed. Authentication supports open networks - and WPA2-PSK with CCMP; WPA3/SAE, 802.1X, required MFP and other ciphers - are unsupported. AP/P2P and roaming are unsupported. -* Sleep is not supported on the MacBook Neo yet. With radio support active - (the default), the retained bootstrap makes the kernel refuse suspend: - lid close and ``systemctl suspend`` do not put it to sleep. Shut down before - putting it in a bag. Booting with ``pcie_apple_piodma_diag.enumerate=0`` - removes this refusal and disables Wi-Fi and Bluetooth; sleep and wake are - still unproven on this hardware without the radios. -* PCI bootstrap memory remains retained until external reset. Sleep and kexec - are refused while it is retained; they are not hardware-qualified on the Neo. - Standard PCI sysfs removal of the root and radio functions is refused; - arbitrary controller/IOMMU teardown and memory reuse remain unqualified. - Wi-Fi shutdown joins host producers before disabling bus mastering and - retains device-visible memory until reset. - Userspace PCI ``reset`` and ``reset_subordinate`` are also refused, and - native slot hot-plug is disabled for this host. Per-link sysfs ASPM/Clock PM - writes can lift the power-state restrictions; they are reapplied only at - probe. Do not enable those link power states while the radios are active. - These guards protect against accidental teardown, not privileged access. - Root can still reset through raw PCI configuration writes (SBR, FLR or Link - Disable), override link power states, or unbind the root port's ``pcieport`` - driver. None of these actions is supported while the retained host is active. -* The Wi-Fi module pins itself before publishing its first DMA ring and cannot - be unloaded normally while DMA is retained. A checked reset during probe - failure may release that pin; a failed reset retains it. Never unload or - reprobe Wi-Fi while Bluetooth is active. Shutdown quiesces host producers - without freeing retained memory; use a full shutdown and external reset - after a failed or uncertain admission. -* Bluetooth PCI removal/quiescence is incomplete. Its software queue limit - does not provide HCI backpressure; saturation can drop an accounted frame. - Both require correction before production use. An activated Bluetooth module - cannot be unloaded normally; forced PCI removal only retires software - callbacks, including the threaded IRQ and MSI vectors, and does not establish - DMA quiescence or release retained ownership. +* PCI bootstrap memory remains retained until external reset. Controller + removal and memory reuse are unqualified. Runtime guards refuse reset and + kexec; suspend-to-idle retains the successfully admitted arena. +* Bluetooth system sleep uses a bounded firmware quiesce/restore handshake + around HCI suspend/resume, preserving its DMA arena. Hibernation is rejected. + Removal first quiesces firmware, then stops work and interrupts, disables + bus mastering and drains pending + PCI transactions before freeing DMA. An unconfirmed drain retains storage + until reset. A controller that stops answering commands, or reports a + hardware error, is reset over HCI and set up again; if that setup fails, + the transport faults. Firmware reloading and warm reprobe are + unsupported; use a fresh cold boot after removing or faulting an active + transport. +* The Bluetooth software queue limit does not provide HCI backpressure; + saturation can drop an accounted frame. * Arbitrary scan IEs, WPA3/SAE, required MFP, AP/P2P, general country-package generation, roaming and long-duration reliability are unqualified. * SCO/headset microphone, LE Audio/ISO and simultaneous headset audio are diff --git a/Documentation/rust/bindgen-layout-checks.rst b/Documentation/rust/bindgen-layout-checks.rst new file mode 100644 index 00000000000000..8ad8c24526b672 --- /dev/null +++ b/Documentation/rust/bindgen-layout-checks.rst @@ -0,0 +1,49 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Checking generated binding layouts +================================== + +Kernel bindings retain bindgen's layout checks. With the configured Rust target, +these are unconditional constant assertions, checked while the bindings crate +is compiled. They compare Rust sizes, alignments and field offsets against the +values obtained from Clang, without executing kernel code. + +Do not disable these checks to make an incompatible binding generator compile. +A type can have the right size and still have incorrect member offsets. For +example, an empty GNU C struct may have size zero and alignment 64; losing that +alignment moves later fields in an enclosing struct without necessarily +changing the enclosing size. + +Known generator requirements +---------------------------- + +Unmodified bindgen 0.73.2 loses explicit alignment on defined zero-sized C +structs. It also omits the named anonymous record member accepted by Clang's +``-fms-extensions`` in a declaration such as:: + + struct payload { unsigned long words[20]; }; + union container { + struct payload; + unsigned long other; + }; + +The generator used for this tree must preserve these layouts. Use a corrected +bindgen build through the normal ``BINDGEN=/path/to/bindgen`` make variable. +``tools/aurora-bindgen/`` contains pinned patches, a private build recipe and +compile-only valid-input tests. It never installs a system-wide tool. The +associated build provenance must retain the generator source revision, +patches, binary digest, and valid-input C/Rust layout checks. A version string +alone does not identify a locally corrected tool. Do not add dummy kernel +storage, hardcode task offsets, or change scheduler configuration to conceal a +binding generator error. + +After a binding layout correction, rebuild the kernel Image and all matching +modules. The generated bindings are consumed by the Rust kernel crate and by +multiple drivers; rebuilding just one module can leave incompatible layouts in +the same kernel. Keep the old Image and its own modules together as a fallback. + +The minimum accepted bindgen version is 0.71.1. Its Rust 1.77 ``offset_of`` +feature enables unconditional const layout assertions; the configured Rust +target 1.85 selects that path rather than ``#[test]``-only functions. The pinned +0.73.2 tool has the same behavior. See the source links and explicit target +accounting in ``tools/aurora-bindgen/README.rst``. diff --git a/Documentation/rust/index.rst b/Documentation/rust/index.rst index b78ed0efa78460..c03031b2410866 100644 --- a/Documentation/rust/index.rst +++ b/Documentation/rust/index.rst @@ -37,6 +37,7 @@ more details. coding-guidelines arch-support testing + bindgen-layout-checks You can also find learning materials for Rust in its section in :doc:`../process/kernel-docs`. diff --git a/Documentation/soc/apple/t8140-pmp-dvfs.rst b/Documentation/soc/apple/t8140-pmp-dvfs.rst new file mode 100644 index 00000000000000..7ec4ae7366ed90 --- /dev/null +++ b/Documentation/soc/apple/t8140-pmp-dvfs.rst @@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +T8140 PMP DVFS diagnostics +========================= + +The existing PMP report provider exposes a read-only ``dvfs`` file under +``/sys/kernel/debug/apple-pmp-/`` on T8140. The provider must be +bound; the file is readable by root. Its output lists ADT report entries +8 through 11 and both raw 64-bit words of each 16-byte entry. Only those four entries +are read. The report aperture is not dumped. + +Reading returns ``EAGAIN`` if the PMP running bit is clear before or after +sampling. Entries are sampled individually, rather than as an atomic +firmware snapshot. A running PMP alone does not establish freshness of +each value. + +The field encoding and conversion to clock rates are not yet established. +Values must not be interpreted as Hz, temperatures, or verified hardware +clock states. This interface changes neither firmware requests nor cache +allocation nor thermal policy. + +CPU temperatures remain available through ``macsmc-hwmon``. CPU cooling +and firmware frequency limits remain under their existing drivers. The +DVFS diagnostic file is not a thermal sensor. diff --git a/MAINTAINERS b/MAINTAINERS index 1103f335b26c74..dacf240284a9bf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1955,6 +1955,7 @@ F: Documentation/devicetree/bindings/display/apple,dcp-typec-routes.yaml F: drivers/gpu/drm/apple/ F: drivers/gpu/drm/apple/dcp-fabric* F: include/linux/soc/apple/dp-tunnel.h +F: tools/testing/selftests/drivers/gpu/apple-afkv2/ APPLE PCIE CONTROLLER DRIVER M: Marc Zyngier @@ -1972,31 +1973,54 @@ APPLE T8140 SOC AND J700 (MACBOOK NEO) SUPPORT M: Ryan Murray W: https://aurora-silicon.github.io S: Maintained +F: Documentation/devicetree/bindings/cpu/apple,t8140-cpuidle.yaml +F: Documentation/devicetree/bindings/cpufreq/apple,cluster-cpufreq.yaml +F: Documentation/devicetree/bindings/power/apple,t8140-pmp-report.yaml +F: Documentation/devicetree/bindings/soc/apple/apple,pmp.yaml +F: drivers/pmdomain/apple/pmp-report.c +F: drivers/soc/apple/pmp.rs +F: include/linux/mt7932.h +F: include/linux/soc/apple/pmp-report.h +F: Documentation/devicetree/bindings/chosen/apple,dcp-rtkit-quiesced.yaml +F: Documentation/devicetree/bindings/display/apple,dcp.yaml +F: Documentation/devicetree/bindings/display/apple,display-subsystem.yaml F: Documentation/PCI/controller/apple-t8140.rst F: Documentation/devicetree/bindings/iio/light/apple,t8140-aop-als.yaml F: Documentation/devicetree/bindings/input/apple,dockchannel-hid.yaml +F: Documentation/devicetree/bindings/mailbox/apple,ascwrap-v6.yaml F: Documentation/devicetree/bindings/media/apple,t8140-isp.yaml +F: Documentation/devicetree/bindings/mux/apple,display-crossbar.yaml F: Documentation/devicetree/bindings/reserved-memory/apple,asc-mem.yaml +F: Documentation/devicetree/bindings/rtc/apple,j700-pmu-rtc.yaml F: Documentation/devicetree/bindings/serial/apple,dockchannel-uart.yaml F: Documentation/devicetree/bindings/soc/apple/apple,dockchannel.yaml F: Documentation/devicetree/bindings/soc/apple/apple,rtk-helper-asc4.yaml F: Documentation/devicetree/bindings/soc/apple/apple,t8140-aop.yaml +F: Documentation/devicetree/bindings/soc/apple/apple,t8140-mcc.yaml F: Documentation/devicetree/bindings/sound/apple,t8140-aop-audio.yaml F: Documentation/devicetree/bindings/usb/vialab,vl122.yaml +F: Documentation/gpu/apple-dcp-backlight.rst F: Documentation/hid/apple-j700-multitouch.rst +F: Documentation/soc/apple/t8140-pmp-dvfs.rst F: Documentation/networking/device_drivers/wifi/mt7932-neo.rst F: arch/arm64/boot/dts/apple/t8140* +F: arch/arm64/configs/j700*.config F: arch/arm64/include/asm/apple-idle.h F: arch/arm64/kernel/idle-apple.S F: drivers/hid/dockchannel-hid/dockchannel-hid-protocol* F: drivers/iio/common/aop_sensors/ -F: drivers/pci/controller/pcie-apple-piodma* +F: drivers/phy/apple/atc-j700-dp.h +F: drivers/rtc/rtc-apple-pmu.c F: drivers/soc/apple/aop.rs +F: drivers/soc/apple/mcc.c F: drivers/tty/serial/apple_dockchannel* +F: drivers/pci/controller/pcie-apple-piodma* F: include/linux/soc/apple/pci-apple-piodma.h F: rust/kernel/iio/common/aop_sensors.rs F: rust/kernel/soc/apple/aop.rs F: sound/soc/apple/t8140_aop_audio.rs +F: tools/testing/apple/ +F: tools/testing/selftests/drivers/gpu/apple-dcp/ APPLETALK NETWORK LAYER L: netdev@vger.kernel.org @@ -2603,9 +2627,7 @@ F: drivers/clk/clk-apple-nco.c F: drivers/cpufreq/apple-soc-cpufreq.c F: drivers/dma/apple-admac.c F: drivers/gpio/gpio-macsmc.c -F: drivers/hwmon/macsmc-hwmon.c -F: drivers/pmdomain/apple/ -F: drivers/pmdomain/apple/.kunitconfig.apple-pmp +F: drivers/hwmon/macsmc-hwmon*.c F: drivers/i2c/busses/i2c-pasemi-core.c F: drivers/i2c/busses/i2c-pasemi-platform.c F: drivers/input/misc/macsmc-input.c @@ -2621,6 +2643,8 @@ F: drivers/nvmem/apple-spmi-nvmem.c F: drivers/phy/apple/ F: drivers/phy/apple/.kunitconfig.apple-atc F: drivers/pinctrl/pinctrl-apple-gpio.c +F: drivers/pmdomain/apple/ +F: drivers/pmdomain/apple/.kunitconfig.apple-pmp F: drivers/power/reset/macsmc-reboot.c F: drivers/power/supply/macsmc-power.c F: drivers/pwm/pwm-apple.c @@ -2644,9 +2668,17 @@ F: include/dt-bindings/pinctrl/apple.h F: include/linux/mfd/macsmc.h F: include/linux/soc/apple/* F: include/uapi/drm/asahi_drm.h +F: include/uapi/linux/apple-sep.h F: net/bluetooth/brcm.c F: net/bluetooth/brcm.h +ARM/APPLE QSPIMC DRIVER +M: Ryan Murray +W: https://aurora-silicon.github.io +S: Maintained +F: Documentation/devicetree/bindings/spi/apple,qspi.yaml +F: drivers/spi/spi-apple-qspi.c + ARM/ARTPEC MACHINE SUPPORT M: Jesper Nilsson M: Lars Persson diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a582d841515373..31b402de5245b0 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -362,6 +362,7 @@ config FIX_EARLYCON_MEM config PGTABLE_LEVELS int + default 4 if ARM64_USER4K_EXPERIMENTAL default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36 default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42 default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) diff --git a/arch/arm64/Kconfig.debug b/arch/arm64/Kconfig.debug index 265c4461031f4d..52fced2ab8355d 100644 --- a/arch/arm64/Kconfig.debug +++ b/arch/arm64/Kconfig.debug @@ -21,3 +21,97 @@ config ARM64_RELOC_TEST tristate "Relocation testing module" source "drivers/hwtracing/coresight/Kconfig" + +config ARM64_USER4K_EXPERIMENTAL + bool "Experimental mixed-granule userspace" + select ARCH_HAS_MM_PGTABLE + select ARCH_HAS_USER_PAGE_SIZE + select MM_SUBPAGE + select MM_ID + select HAVE_ARCH_TLB_REMOVE_TABLE + select ARCH_SUPPORTS_PT_RECLAIM + depends on (ARM64_16K_PAGES || ARM64_64K_PAGES) && ARM64_VA_BITS_48 && ARM64_PA_BITS_48 + depends on !ARM64_LPA2 && !NO_PAGE_MAPCOUNT + default n + help + Build the in-development MM support for alternative 4K user tables + on 16K/64K kernels and 16K user tables on 64K kernels. + Ordinary address spaces and kernel tables retain the native granule. An experimental + one-shot prctl selects a supported granule for the next exec. + This is a development ABI: several MM interfaces and device mappings + still require integration and validation. The private prctl numbers + are not upstream assignments or a stable ABI promise. + Say N for normal kernels. + +config ARM64_USER4K_KUNIT_TEST + bool "Test mixed userspace granules with larger kernel pages" + depends on KUNIT=y && ARM64_USER4K_EXPERIMENTAL + depends on ARM64_PA_BITS_48 + select ARM64_USER4K_DATA_KUNIT_TEST + select IOMMUFD + select IOMMUFD_DRIVER + default n + help + Exercise independent TTBR0 translation granules, subpage permissions + and invalidation while TTBR1 retains the normal native kernel tables. + Hardware translation/uaccess fixtures require arm64.user4k_test=1 + as a second opt-in and are disabled with KASAN. Software MM fixtures + can run with KASAN without borrowing or replacing TTBR0. + Run only in a disposable test VM. This regression harness does not + establish hardware qualification or production ABI readiness. + Say N for normal kernels. + +config ARM64_USER4K_IOMMUFD_TEST + bool "IOMMUFD ioctl harness for mixed userspace granules" + depends on ARM64_USER4K_EXPERIMENTAL && DEBUG_KERNEL + depends on FAULT_INJECTION && RUNTIME_TESTING_MENU + depends on IOMMU_SUPPORT + select GENERIC_PT + select IOMMU_PT + select IOMMU_PT_AMDV1 + select IOMMUFD + select IOMMUFD_TEST + default n + help + Enable the IOMMUFD mock-device ioctls and generic AMDv1 page tables + for the mixed-granule userspace ioctl fixture. This exposes privileged + test interfaces; use only in a disposable test VM. + Say N for normal kernels. + +config ARM64_USER4K_VFIO_ACCOUNT_TEST + bool "VFIO type1 pinning harness for mixed userspace granules" + depends on ARM64_USER4K_EXPERIMENTAL && KUNIT=y + depends on IOMMUFD != m + select VFIO + select VFIO_GROUP + select VFIO_CONTAINER + select VFIO_IOMMU_TYPE1 + select VFIO_TYPE1_ACCOUNT_KUNIT_TEST + default n + help + Enable built-in VFIO type1 accounting and retained-fragment tests + in a disposable mixed-granule guest. Recorded IOMMU translations + do not establish physical-device DMA support. + Say N for normal kernels. + +config ARM64_USER4K_VFIO_IOCTL_TEST + bool "Private VFIO type1 ioctl harness for mixed userspace granules" + depends on ARM64_USER4K_EXPERIMENTAL && KUNIT=y && DEBUG_KERNEL + depends on RUNTIME_TESTING_MENU + depends on IOMMUFD != m + select ARM64_USER4K_VFIO_ACCOUNT_TEST + default n + help + Expose a root-only disposable-guest misc device forwarding real EL0 + arguments to type1 ioctls and supplying recording IOMMU domains. + These privileged test controls perform no physical device DMA. + Say N for normal kernels. + +config ARM64_USER4K_DATA_KUNIT_TEST + bool "Test packed userspace allocation ownership" + depends on KUNIT=y && ARM64_USER4K_EXPERIMENTAL + default n + help + Exercise packed allocation, slot reuse, copying and deferred lifetime. + These tests use native kernel addresses without changing TTBR0, and + can run with KASAN even when the architectural tests are disabled. diff --git a/arch/arm64/boot/dts/apple/t6001.dtsi b/arch/arm64/boot/dts/apple/t6001.dtsi index 95bf2d4d8c6393..f14bd14518042a 100644 --- a/arch/arm64/boot/dts/apple/t6001.dtsi +++ b/arch/arm64/boot/dts/apple/t6001.dtsi @@ -90,10 +90,9 @@ }; /* - * The ANE has run on the M1 Max and the M1 Pro, so enable it here with its + * The ANE has run on the M1 Pro and the M1 Max, so enable it here with its * DARTs and the power states only it uses. t6000.dtsi includes this file. */ -#ifndef APPLE_ANE_UNTESTED &ane { status = "okay"; }; @@ -133,4 +132,3 @@ &ps_ane_set4 { status = "okay"; }; -#endif diff --git a/arch/arm64/boot/dts/apple/t6021.dtsi b/arch/arm64/boot/dts/apple/t6021.dtsi index 7154f4644e749b..53bd36ef4b843f 100644 --- a/arch/arm64/boot/dts/apple/t6021.dtsi +++ b/arch/arm64/boot/dts/apple/t6021.dtsi @@ -87,10 +87,9 @@ }; /* - * The ANE has run on the M2 Max, so enable it here with its mailbox and - * DARTs. The M2 Pro (t6020.dtsi includes this file) has the same die 0 ANE. + * The ANE has run on the M2 Pro and the M2 Max, so enable it here with its + * mailbox and DARTs. t6020.dtsi includes this file. */ -#ifndef APPLE_ANE_UNTESTED &ane { status = "okay"; }; @@ -110,4 +109,3 @@ &ane_mbox { status = "okay"; }; -#endif diff --git a/arch/arm64/boot/dts/apple/t602x-j414-j416.dtsi b/arch/arm64/boot/dts/apple/t602x-j414-j416.dtsi index 98301e37872bb0..1e3160eb1b54b4 100644 --- a/arch/arm64/boot/dts/apple/t602x-j414-j416.dtsi +++ b/arch/arm64/boot/dts/apple/t602x-j414-j416.dtsi @@ -25,8 +25,8 @@ disp0 = &display; disp0_piodma = &disp0_piodma; keyboard = &keyboard; - sio = &sio; sep = &sep; + sio = &sio; /* m1n1 writes this machine's Thunderbolt calibration onto these * aliases. usb4-N-* is current m1n1; usb4_N_* is the 1.6.1 bootloader. */ @@ -367,6 +367,51 @@ apple,pcie-tunnel = <&usb4_2_pcie_tunnel>; }; +/* + * No m1n1 performs the PCIe-C preinit handoff on these machines. Let the + * PCIe driver cold-initialize the Thunderbolt PCIe tunnel ports itself, so + * PCIe devices behind Thunderbolt 3 docks (e.g. their xHCI) come up. + */ +&apciec0 { + apple,pciec-kernel-init; +}; + +&apciec1 { + apple,pciec-kernel-init; +}; + +&apciec2 { + apple,pciec-kernel-init; +}; + +/* + * PCIe-C DART tunables. iBoot adds "dart-tunables-instance-0" to the ADT at + * boot and m1n1 does not copy it for these DARTs. Values taken from the live + * J416c ADT (macOS ioreg), identical for all three ports; offsets are + * relative to the DART's reg. + */ +#define APCIEC_DART_TUNABLES \ + apple,tunable = <0x20c 0xff0000b7 0xe40000b7>, \ + <0x220 0x000f0f0f 0x000f0f0f>, \ + <0x224 0x00ffffff 0x00080808>, \ + <0x300 0x00001f31 0x00000001>, \ + <0x308 0x3ffffffc 0x10000000>, \ + <0x310 0x3ffffffc 0x3ffffffc> + +&apciec0_dart { + APCIEC_DART_TUNABLES; +}; + +&apciec1_dart { + APCIEC_DART_TUNABLES; +}; + +&apciec2_dart { + APCIEC_DART_TUNABLES; +}; + +#undef APCIEC_DART_TUNABLES + /* Keep dptx_phy_ps on until DCP can sequence it during display bring-up. */ &ps_dptx_phy_ps { apple,always-on; diff --git a/arch/arm64/boot/dts/apple/t8103.dtsi b/arch/arm64/boot/dts/apple/t8103.dtsi index fadb7d79268bd4..9c2874c12b732f 100644 --- a/arch/arm64/boot/dts/apple/t8103.dtsi +++ b/arch/arm64/boot/dts/apple/t8103.dtsi @@ -2254,7 +2254,10 @@ #include "t8103-pmgr.dtsi" -#ifndef APPLE_ANE_UNTESTED +/* + * The ANE has run on the M1, so enable it here with its DARTs and the + * power states only it uses. + */ &ane { status = "okay"; }; @@ -2294,4 +2297,3 @@ &ps_ane_set5 { status = "okay"; }; -#endif diff --git a/arch/arm64/boot/dts/apple/t8140-j700-dcpext.dtsi b/arch/arm64/boot/dts/apple/t8140-j700-dcpext.dtsi new file mode 100644 index 00000000000000..69d6eff9aa756f --- /dev/null +++ b/arch/arm64/boot/dts/apple/t8140-j700-dcpext.dtsi @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +/* + * T8140 external display resources measured through the public ADT API. + * + * The controller remains disabled until the loader validates and preserves + * the inherited firmware and its complete dependency group. + */ + +&{/soc} { + dcpext: dcp@416e00000 { + compatible = "apple,t8140-dcpext", "apple,dcp"; + reg = <0x4 0x16e00000 0x0 0x4000>, + <0x4 0x14000000 0x0 0xafc000>, + <0x4 0x16240000 0x0 0x10000>, + <0x4 0x163ac000 0x0 0x4000>, + <0x4 0x16800000 0x0 0x800000>, + <0x3 0x02800000 0x0 0xbc000>, + <0x4 0x16850000 0x0 0x4000>; + reg-names = "coproc", "disp-0", "disp-1", "disp-2", + "disp-3", "disp-4", "iop-vbar"; + mboxes = <&dcpext_mbox>; + mbox-names = "mbox"; + /* First mapper in the measured ADT iommu-parent list. */ + iommus = <&dcpext0_dart 23>; + power-domains = <&ps_dispext0_cpu>; + apple,connector-type = "USB-C"; + status = "disabled"; + + #address-cells = <2>; + #size-cells = <2>; + ranges; + + dispext0_piodma: piodma { + iommus = <&dispext0_dart 16>; + }; + + typec-routes { + #address-cells = <1>; + #size-cells = <0>; + + route@0 { + reg = <0>; + status = "disabled"; + mode-switch; + + port { + dcpext_typec0: endpoint { + remote-endpoint = <&typec0_dcp>; + }; + }; + }; + }; + }; +}; + +&typec0 { + ports { + port@3 { + reg = <3>; + + typec0_dcp: endpoint { + remote-endpoint = <&dcpext_typec0>; + }; + }; + }; +}; + +/* Retain the inherited firmware supply once the loader publishes it. */ +&ps_dispext0_cpu { + apple,always-on; +}; diff --git a/arch/arm64/boot/dts/apple/t8140-j700.dts b/arch/arm64/boot/dts/apple/t8140-j700.dts index 842e1b2bcf5723..affdd0149be647 100644 --- a/arch/arm64/boot/dts/apple/t8140-j700.dts +++ b/arch/arm64/boot/dts/apple/t8140-j700.dts @@ -52,6 +52,8 @@ power-domains = <&ps_disp_cpu>; /* Format properties will be added by loader */ status = "disabled"; + panel = <&panel>; + post-init-providers = <&panel>; }; }; @@ -381,52 +383,6 @@ }; }; }; - - /* - * Once the PMP knows the GPU is on, it votes the DRAM controller - * (DCS) up with GPU load, and the GPU and memory system draw up - * to 2.4 W more. The CPU zones above cannot hold that back: on a - * GPU-bound load the CPUs request little power, so with the - * performance governor the die passed 95 C within 35 s and - * reached 104 C before they throttled. This zone limits the DCS - * vote instead, through the PMC. With it the same load peaks at - * 97 C, as it does without any DCS vote, and a ten-minute - * balanced GPU load loses 1-2% of its throughput. - * - * Tp0t reads at or above every CPU die sensor under GPU load, - * and leads them by about 1 C when the CPUs also run hot. The - * sustainable power is the whole DCS range, so the vote is only - * trimmed above the control trip; the PID gains follow from it - * and the 6 C between the trips. - */ - soc-dcs-thermal { - polling-delay-passive = <500>; - polling-delay = <500>; - thermal-sensors = <&smc_hwmon 6>; /* Tp0t */ - sustainable-power = <2407>; - - trips { - soc_dcs_alert: trip-alert { - temperature = <89000>; - hysteresis = <2000>; - type = "passive"; - }; - - soc_dcs_control: trip-control { - temperature = <95000>; - hysteresis = <2000>; - type = "passive"; - }; - }; - - cooling-maps { - map0 { - trip = <&soc_dcs_control>; - cooling-device = - <&pmc THERMAL_NO_LIMIT THERMAL_NO_LIMIT>; - }; - }; - }; }; }; @@ -470,8 +426,12 @@ &pmc { /* - * System power above DCS state 0 at each DCS state, measured with - * glmark2's terrain scene, GPU-bound, on the J700. + * System power above DCS state 0 at each DCS state, GPU-bound. + * + * No thermal zone limits the DCS rail: the PMP drops its DCS vote + * when the GPU idles, and a power allocator zone on Tp0t made the + * limit oscillate between states 1 and 4 under GPU load. Without + * it Tp0t stays below the 110 C critical trips of the CPU zones. */ apple,dcs-power-microwatt = <0 919000 1453000 2058000 2407000>; status = "okay"; @@ -500,8 +460,6 @@ compatible = "jedec,spi-nor"; reg = <0>; spi-max-frequency = <25000000>; - spi-rx-bus-width = <4>; - spi-tx-bus-width = <4>; partitions { compatible = "fixed-partitions"; @@ -676,7 +634,7 @@ * the SMC key groups and are not otherwise verified. */ - /* Channel 6, the sensor of soc-dcs-thermal: keep it first here. */ + /* Channel 6: reads at or above every CPU die sensor under GPU load. */ temperature-Tp0t { apple,key-id = "Tp0t"; label = "SMC Tp0t"; @@ -1866,7 +1824,7 @@ #include "hwmon-laptop.dtsi" &pcie0 { - /* Experiment: guard stays closed until the retained bootstrap succeeds. */ + /* The PIODMA guard stays closed until the retained bootstrap succeeds. */ apple,piodma = <&pcie0_piodma_diag>; iommu-map = <0x100 &pcie0_dart 1 2>; iommu-map-mask = <0xffff>; @@ -1888,7 +1846,7 @@ }; }; -/* Experimental SID17 bootstrap; arena ownership lasts until full reset. */ +/* SID17 bootstrap; the arena stays owned until a full reset. */ &pcie0_dart { status = "okay"; apple,dma-range = /bits/ 64 <0 0x1000000000>; @@ -1918,3 +1876,31 @@ local-mac-address = [00 00 00 00 00 00]; }; }; + +/* The loader publishes external resources after validating their firmware. */ +#include "t8140-j700-dcpext.dtsi" + +&display { + iommus = <&disp0_dart 0>, <&dispext0_dart 0>; +}; + +&dcpext { + clocks = <&clk_disp0_154>, <&clk_disp0_194>; + clock-names = "clock-154", "clock-194"; + phys = <&atcphy0 PHY_TYPE_DP>; + phy-names = "typec0"; + mux-controls = <&dpxbar0 0>; + mux-control-names = "typec0"; + apple,typec-mux-indices = <2>; + apple,typec-dptx-phys = <0>; + apple,dcp-index = <1>; +}; + +&{/soc/dcp@416e00000/typec-routes/route@0} { + status = "okay"; +}; + +/* Allocate scanout buffers in the window used by the inherited firmware. */ +&dcpext0_dart { + apple,dma-range = <0x100 0x0 0x10 0x0>; +}; diff --git a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi index 6e6098144b757e..8438e4f17e81f9 100644 --- a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi +++ b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi @@ -397,6 +397,7 @@ * firmware owns this state; writing it faults. */ apple,always-on; + apple,firmware-owned; }; diff --git a/arch/arm64/boot/dts/apple/t8140.dtsi b/arch/arm64/boot/dts/apple/t8140.dtsi index a2752e59c07478..89a5a609817e42 100644 --- a/arch/arm64/boot/dts/apple/t8140.dtsi +++ b/arch/arm64/boot/dts/apple/t8140.dtsi @@ -491,7 +491,8 @@ /* PMP soc-device 9, which the PMP acknowledges */ pmp_report_gfx: report@9 { - compatible = "apple,t8140-pmp-v2-report-entry"; + compatible = "apple,t8140-pmp-v2-report-entry", + "apple,t6000-pmp-v2-report-entry"; reg = <0x9>; label = "pmp-gfx"; #power-domain-cells = <0>; diff --git a/arch/arm64/boot/dts/apple/t8142-j813-display.dtsi b/arch/arm64/boot/dts/apple/t8142-j813-display.dtsi new file mode 100644 index 00000000000000..9bd549a5784cc0 --- /dev/null +++ b/arch/arm64/boot/dts/apple/t8142-j813-display.dtsi @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +/* + * J813 internal display pipeline: the DCP coprocessor and its DARTs. + * + * The bootloader leaves DCP running with its firmware mapped through + * locked DART tables. The loader reserves those mappings and enables the + * nodes below; Linux attaches to the running coprocessor without resetting + * it. Resources were read from this machine's Apple Device Tree. + */ + +/ { + aliases { + dcp = &dcp; + disp0 = &display; + disp0_piodma = &disp0_piodma; + }; + + display: display-subsystem { + compatible = "apple,t8142-display-subsystem", "apple,display-subsystem"; + /* disp0_dart must be first since it is locked */ + iommus = <&disp0_dart 0>; + /* generate phandle explicitly for use in loader */ + phandle = <&display>; + }; +}; + +&{/soc} { + pmgr_dcp: power-management@382720000 { + reg = <3 0x82720000 0 0x8>, <3 0x827a4000 0 0x4>; + reg-names = "dcp-bw-scratch", "dcp-bw-doorbell"; + #apple,bw-scratch-cells = <3>; + #apple,bw-doorbell-cells = <2>; + }; + + dcp_mbox: mailbox@286e08000 { + compatible = "apple,t8142-asc-mailbox", "apple,asc-mailbox-v4"; + reg = <2 0x86e08000 0 0x4000>; + interrupts = , + , + , + ; + interrupt-names = "send-empty", "send-not-empty", + "recv-empty", "recv-not-empty"; + #mbox-cells = <0>; + }; + + disp0_dart: iommu@286300000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <2 0x86300000 0 0xc000>; + interrupts = ; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x0 0x10 0x0>; + status = "disabled"; + }; + + dcp_dart: iommu@286340000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <2 0x86340000 0 0x20000>; + interrupts = ; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x0 0x10 0x0>; + }; + + dcp: dcp@286e00000 { + compatible = "apple,t8142-dcp", "apple,dcp"; + mboxes = <&dcp_mbox>; + mbox-names = "mbox"; + iommus = <&dcp_dart 23>; + reg-names = "coproc", "disp-0", "disp-1", "disp-2", + "disp-3", "disp-4", "iop-vbar"; + reg = <2 0x86e00000 0 0x4000>, + <2 0x84000000 0 0xb1c000>, + <2 0x86240000 0 0x10000>, + <2 0x863ac000 0 0x4000>, + <2 0x86800000 0 0x800000>, + <3 0x82700000 0 0x114000>, + <2 0x86850000 0 0x4000>; + apple,bw-scratch = <&pmgr_dcp 0 5 0>; + apple,bw-doorbell = <&pmgr_dcp 1 6>; + /* Scan out the primary plane on IOMFB surface 2. */ + apple,iomfb-surfaces = <0 0 1 0>; + clocks = <&clkref>; + /* generate phandle explicitly for use in loader */ + phandle = <&dcp>; + status = "disabled"; + /* required bus properties for 'piodma' subdevice */ + #address-cells = <2>; + #size-cells = <2>; + + disp0_piodma: piodma { + iommus = <&disp0_dart 16>; + phandle = <&disp0_piodma>; + }; + }; +}; diff --git a/arch/arm64/boot/dts/apple/t8142-j813-peripherals.dtsi b/arch/arm64/boot/dts/apple/t8142-j813-peripherals.dtsi new file mode 100644 index 00000000000000..a6708b1c972dfd --- /dev/null +++ b/arch/arm64/boot/dts/apple/t8142-j813-peripherals.dtsi @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +#include + +/ { + aliases { + mtp = &mtp; + keyboard = &keyboard; + }; + + led-controller { + compatible = "pwm-leds"; + + led-0 { + pwms = <&fpwm0 0 40000>; + label = "kbd_backlight"; + function = LED_FUNCTION_KBD_BACKLIGHT; + color = ; + max-brightness = <255>; + default-state = "keep"; + }; + }; +}; + +&{/soc} { + pmgr: power-management@380700000 { + compatible = "apple,t8142-pmgr", "apple,t8103-pmgr", "syscon", "simple-mfd"; + reg = <3 0x80700000 0 0x4000>; + #address-cells = <1>; + #size-cells = <1>; + + ps_fpwm0: power-controller@330 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0x330 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "fpwm0"; + }; + }; + + fpwm0: pwm@3a5040000 { + compatible = "apple,t8142-fpwm", "apple,s5l-fpwm"; + reg = <3 0xa5040000 0 0x4000>; + power-domains = <&ps_fpwm0>; + clocks = <&clkref>; + #pwm-cells = <2>; + }; + + smc_mbox: mailbox@38c608000 { + compatible = "apple,t8142-asc-mailbox", "apple,asc-mailbox-v4"; + reg = <3 0x8c608000 0 0x4000>; + interrupts = , + , + , + ; + interrupt-names = "send-empty", "send-not-empty", "recv-empty", "recv-not-empty"; + #mbox-cells = <0>; + }; + + smc: smc@38c600000 { + compatible = "apple,t8142-smc", "apple,t8103-smc"; + reg = <3 0x8c600000 0 0x4000>, <3 0x8de00000 0 0x120000>; + reg-names = "smc", "sram"; + mboxes = <&smc_mbox>; + + hwmon { + compatible = "apple,smc-hwmon"; + apple,discover-temperatures; + }; + }; + + mtp: mtp@394600000 { + compatible = "apple,t8142-mtp", "apple,t8142-rtk-helper-asc4", + "apple,mtp", "apple,rtk-helper-asc4"; + /* The second region is firmware SRAM, not the ASC control window. */ + reg = <3 0x94600000 0 0x4000>, <3 0x94c00000 0 0x100000>; + reg-names = "asc", "sram"; + mboxes = <&mtp_mbox>; + iommus = <&mtp_dart 0>; + }; + + mtp_mbox: mailbox@394608000 { + compatible = "apple,t8142-asc-mailbox", "apple,asc-mailbox-v4"; + reg = <3 0x94608000 0 0x4000>; + interrupts = , + , + , + ; + interrupt-names = "send-empty", "send-not-empty", "recv-empty", "recv-not-empty"; + #mbox-cells = <0>; + }; + + mtp_dart: iommu@394800000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <3 0x94800000 0 0xc000>; + interrupts = ; + #iommu-cells = <1>; + /* Keep host allocations clear of firmware's low IOVA mappings. */ + apple,dma-range = <0 0x02000000 0 0x10000000>; + }; + + mtp_dockchannel: fifo@394b14000 { + compatible = "apple,t8142-dockchannel", "apple,dockchannel"; + reg = <3 0x94b14000 0 0x1000>; + reg-names = "irq"; + interrupts = ; + ranges = <0 3 0x94b28000 0x20000>; + nonposted-mmio; + #address-cells = <1>; + #size-cells = <1>; + interrupt-controller; + #interrupt-cells = <2>; + + mtp_hid: input@8000 { + compatible = "apple,dockchannel-hid"; + reg = <0x8000 0x1000>, <0xc000 0x1000>, + <0x0000 0x1000>, <0x4000 0x1000>; + reg-names = "config", "data", "rmt-config", "rmt-data"; + iommus = <&mtp_dart 0>; + interrupt-parent = <&mtp_dockchannel>; + interrupts = <2 IRQ_TYPE_LEVEL_HIGH>, <3 IRQ_TYPE_LEVEL_HIGH>; + interrupt-names = "tx", "rx"; + apple,fifo-size = <0x800>; + apple,helper-cpu = <&mtp>; + + stm { }; + + multi-touch { + firmware-name = "apple/tpmtfw-j813.bin"; + apple,power-method = <2>; + apple,firmware-managed-reset; + }; + + /* Layout is filled by the loader using this machine's own data. */ + keyboard: keyboard { }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/apple/t8142-j813-usb.dtsi b/arch/arm64/boot/dts/apple/t8142-j813-usb.dtsi new file mode 100644 index 00000000000000..e27e40c04e268d --- /dev/null +++ b/arch/arm64/boot/dts/apple/t8142-j813-usb.dtsi @@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +#include +#include + +/ { + aliases { + atcphy0 = &atcphy0; + usb0 = &dwc3_0; + atcphy1 = &atcphy1; + usb1 = &dwc3_1; + }; +}; + +&pmgr { + ps_atc0_common: power-controller@2c8 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0x2c8 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "atc0-common"; + }; +}; + +&{/soc} { + pmgr_nub: power-management@388280000 { + compatible = "apple,t8142-pmgr", "apple,t8103-pmgr", "syscon", "simple-mfd"; + reg = <3 0x88280000 0 0x4000>; + #address-cells = <1>; + #size-cells = <1>; + + ps_nub_spmi_a0: power-controller@70 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0x70 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "nub-spmi-a0"; + }; + + ps_atc0_usb_aon: power-controller@c8 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0xc8 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "atc0-usb-aon"; + }; + + ps_atc0_usb: power-controller@d0 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0xd0 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + power-domains = <&ps_atc0_common>, <&ps_atc0_usb_aon>; + label = "atc0-usb"; + }; + }; + + nub_spmi_a0: spmi@388908000 { + compatible = "apple,t8142-spmi"; + reg = <3 0x88908000 0 0x4000>; + #address-cells = <2>; + #size-cells = <0>; + power-domains = <&ps_nub_spmi_a0>; + + hpm0: usb-pd@c { + compatible = "apple,sn201202x"; + reg = <0xc SPMI_USID>; + + typec0: connector { + compatible = "usb-c-connector"; + label = "USB-C Left-front"; + power-role = "dual"; + data-role = "host"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + typec0_hs: endpoint { + remote-endpoint = <&dwc3_0_hs>; + }; + }; + + port@1 { + reg = <1>; + typec0_ss: endpoint { + remote-endpoint = <&atcphy0_typec>; + }; + }; + }; + }; + }; + }; + + dwc3_0: usb@402280000 { + compatible = "apple,t8142-dwc3", "apple,t8103-dwc3"; + reg = <4 0x02280000 0 0xcd00>, <4 0x0228cd00 0 0x4b00>; + reg-names = "dwc3-core", "dwc3-apple"; + interrupts = ; + dr_mode = "host"; + usb-role-switch; + role-switch-default-mode = "host"; + iommus = <&dwc3_0_dart_0 0>, <&dwc3_0_dart_1 1>; + power-domains = <&ps_atc0_usb>; + resets = <&atcphy0>; + phys = <&atcphy0 PHY_TYPE_USB2>, <&atcphy0 PHY_TYPE_USB3>; + phy-names = "usb2-phy", "usb3-phy"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + dwc3_0_hs: endpoint { + remote-endpoint = <&typec0_hs>; + }; + }; + port@1 { + reg = <1>; + dwc3_0_ss: endpoint { + remote-endpoint = <&atcphy0_usb>; + }; + }; + }; + }; + + dwc3_0_dart_0: iommu@402f00000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <4 0x02f00000 0 0xc000>; + interrupts = ; + power-domains = <&ps_atc0_usb>; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x4000 0 0xffff0000>; + }; + + dwc3_0_dart_1: iommu@402f80000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <4 0x02f80000 0 0xc000>; + interrupts = ; + power-domains = <&ps_atc0_usb>; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x4000 0 0xffff0000>; + }; + + atcphy0: phy@403000000 { + compatible = "apple,t8142-atcphy"; + reg = <4 0x03000000 0 0x4c000>, + <4 0x03050000 0 0x8000>, + <4 0x00000000 0 0x8000>, + <4 0x02a90000 0 0x4000>, + <4 0x02a84000 0 0x4000>; + reg-names = "core", "lpdptx", "axi2af", "usb2phy", "pipehandler"; + #phy-cells = <1>; + #reset-cells = <0>; + orientation-switch; + mode-switch; + power-domains = <&ps_atc0_usb>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + atcphy0_typec: endpoint { + remote-endpoint = <&typec0_ss>; + }; + }; + port@1 { + reg = <1>; + atcphy0_usb: endpoint { + remote-endpoint = <&dwc3_0_ss>; + }; + }; + }; + }; +}; + +&pmgr { + ps_atc1_common: power-controller@2d0 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0x2d0 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "atc1-common"; + }; +}; + +&pmgr_nub { + ps_atc1_usb_aon: power-controller@d8 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0xd8 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + label = "atc1-usb-aon"; + }; + + ps_atc1_usb: power-controller@e0 { + compatible = "apple,t8142-pmgr-pwrstate", "apple,t8103-pmgr-pwrstate"; + reg = <0xe0 4>; + #power-domain-cells = <0>; + #reset-cells = <0>; + power-domains = <&ps_atc1_common>, <&ps_atc1_usb_aon>; + label = "atc1-usb"; + }; +}; + +&nub_spmi_a0 { + hpm1: usb-pd@a { + compatible = "apple,sn201202x"; + reg = <0xa SPMI_USID>; + + typec1: connector { + compatible = "usb-c-connector"; + label = "USB-C Left-rear"; + power-role = "dual"; + data-role = "host"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + typec1_hs: endpoint { + remote-endpoint = <&dwc3_1_hs>; + }; + }; + + port@1 { + reg = <1>; + typec1_ss: endpoint { + remote-endpoint = <&atcphy1_typec>; + }; + }; + }; + }; + }; +}; + +&{/soc} { + dwc3_1: usb@40a280000 { + compatible = "apple,t8142-dwc3", "apple,t8103-dwc3"; + reg = <4 0x0a280000 0 0xcd00>, <4 0x0a28cd00 0 0x4b00>; + reg-names = "dwc3-core", "dwc3-apple"; + interrupts = ; + dr_mode = "host"; + usb-role-switch; + role-switch-default-mode = "host"; + iommus = <&dwc3_1_dart_0 0>, <&dwc3_1_dart_1 1>; + power-domains = <&ps_atc1_usb>; + resets = <&atcphy1>; + phys = <&atcphy1 PHY_TYPE_USB2>, <&atcphy1 PHY_TYPE_USB3>; + phy-names = "usb2-phy", "usb3-phy"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + dwc3_1_hs: endpoint { + remote-endpoint = <&typec1_hs>; + }; + }; + port@1 { + reg = <1>; + dwc3_1_ss: endpoint { + remote-endpoint = <&atcphy1_usb>; + }; + }; + }; + }; + + dwc3_1_dart_0: iommu@40af00000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <4 0x0af00000 0 0xc000>; + interrupts = ; + power-domains = <&ps_atc1_usb>; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x4000 0 0xffff0000>; + }; + + dwc3_1_dart_1: iommu@40af80000 { + compatible = "apple,t8142-dart", "apple,t8110-dart"; + reg = <4 0x0af80000 0 0xc000>; + interrupts = ; + power-domains = <&ps_atc1_usb>; + #iommu-cells = <1>; + apple,dma-range = <0x100 0x4000 0 0xffff0000>; + }; + + atcphy1: phy@40b000000 { + compatible = "apple,t8142-atcphy"; + reg = <4 0x0b000000 0 0x4c000>, + <4 0x0b050000 0 0x8000>, + <4 0x08000000 0 0x8000>, + <4 0x0aa90000 0 0x4000>, + <4 0x0aa84000 0 0x4000>; + reg-names = "core", "lpdptx", "axi2af", "usb2phy", "pipehandler"; + #phy-cells = <1>; + #reset-cells = <0>; + orientation-switch; + mode-switch; + power-domains = <&ps_atc1_usb>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + atcphy1_typec: endpoint { + remote-endpoint = <&typec1_ss>; + }; + }; + port@1 { + reg = <1>; + atcphy1_usb: endpoint { + remote-endpoint = <&dwc3_1_ss>; + }; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/apple/t8142-j813.dts b/arch/arm64/boot/dts/apple/t8142-j813.dts index d56cd1dd2f560c..20df27fa1bfe3b 100644 --- a/arch/arm64/boot/dts/apple/t8142-j813.dts +++ b/arch/arm64/boot/dts/apple/t8142-j813.dts @@ -58,6 +58,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x0>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -66,6 +68,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x1>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -74,6 +78,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x2>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -82,6 +88,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x3>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -90,6 +98,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x4>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -98,6 +108,8 @@ compatible = "apple,t8142-e"; device_type = "cpu"; reg = <0 0x5>; + operating-points-v2 = <&ecluster_opp>; + performance-domains = <&cpufreq_e>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -106,6 +118,8 @@ compatible = "apple,t8142-p"; device_type = "cpu"; reg = <0 0x10100>; + operating-points-v2 = <&pcluster_opp>; + performance-domains = <&cpufreq_p>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -114,6 +128,8 @@ compatible = "apple,t8142-p"; device_type = "cpu"; reg = <0 0x10101>; + operating-points-v2 = <&pcluster_opp>; + performance-domains = <&cpufreq_p>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -122,6 +138,8 @@ compatible = "apple,t8142-p"; device_type = "cpu"; reg = <0 0x10102>; + operating-points-v2 = <&pcluster_opp>; + performance-domains = <&cpufreq_p>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -130,6 +148,8 @@ compatible = "apple,t8142-p"; device_type = "cpu"; reg = <0 0x10103>; + operating-points-v2 = <&pcluster_opp>; + performance-domains = <&cpufreq_p>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ }; @@ -144,6 +164,78 @@ interrupt-names = "phys", "virt", "hyp-phys", "hyp-virt"; }; + /* State 1 is 300 MHz; firmware SRAM OPP tables begin at state 2. */ + /* Cap each cluster at its measured J813 firmware handoff state. */ + ecluster_opp: opp-table-0 { + compatible = "operating-points-v2"; + opp-shared; + + opp-300000000 { + opp-hz = /bits/ 64 <300000000>; + opp-level = <1>; + }; + + opp-972000000 { + opp-hz = /bits/ 64 <972000000>; + opp-level = <2>; + }; + }; + + pcluster_opp: opp-table-1 { + compatible = "operating-points-v2"; + opp-shared; + + opp-300000000 { + opp-hz = /bits/ 64 <300000000>; + opp-level = <1>; + }; + + opp-1308000000 { + opp-hz = /bits/ 64 <1308000000>; + opp-level = <2>; + }; + + opp-1608000000 { + opp-hz = /bits/ 64 <1608000000>; + opp-level = <3>; + }; + + opp-1932000000 { + opp-hz = /bits/ 64 <1932000000>; + opp-level = <4>; + }; + + opp-2220000000 { + opp-hz = /bits/ 64 <2220000000>; + opp-level = <5>; + }; + + opp-2508000000 { + opp-hz = /bits/ 64 <2508000000>; + opp-level = <6>; + }; + + opp-2820000000 { + opp-hz = /bits/ 64 <2820000000>; + opp-level = <7>; + }; + + opp-3072000000 { + opp-hz = /bits/ 64 <3072000000>; + opp-level = <8>; + }; + + opp-3300000000 { + opp-hz = /bits/ 64 <3300000000>; + opp-level = <9>; + }; + + opp-3516000000 { + opp-hz = /bits/ 64 <3516000000>; + opp-level = <10>; + }; + }; + clkref: clock-ref { compatible = "fixed-clock"; #clock-cells = <0>; @@ -157,6 +249,18 @@ ranges; nonposted-mmio; + cpufreq_e: performance-controller@210e20000 { + compatible = "apple,t8142-cluster-cpufreq"; + reg = <0x2 0x10e20000 0 0x1000>; + #performance-domain-cells = <0>; + }; + + cpufreq_p: performance-controller@211e20000 { + compatible = "apple,t8142-cluster-cpufreq"; + reg = <0x2 0x11e20000 0 0x1000>; + #performance-domain-cells = <0>; + }; + aic: interrupt-controller@381000000 { compatible = "apple,t8142-aic3", "apple,t8122-aic3"; interrupt-controller; @@ -186,3 +290,7 @@ }; }; }; + +#include "t8142-j813-display.dtsi" +#include "t8142-j813-peripherals.dtsi" +#include "t8142-j813-usb.dtsi" diff --git a/arch/arm64/boot/dts/apple/t8152-j873g.dts b/arch/arm64/boot/dts/apple/t8152-j873g.dts index bd259e3f7e364a..6b5e14036c98af 100644 --- a/arch/arm64/boot/dts/apple/t8152-j873g.dts +++ b/arch/arm64/boot/dts/apple/t8152-j873g.dts @@ -62,6 +62,8 @@ reg = <0 0x0>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu1: cpu@1 { @@ -70,6 +72,8 @@ reg = <0 0x1>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu2: cpu@2 { @@ -78,6 +82,8 @@ reg = <0 0x2>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu3: cpu@3 { @@ -86,6 +92,8 @@ reg = <0 0x3>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu4: cpu@4 { @@ -94,6 +102,8 @@ reg = <0 0x4>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu5: cpu@5 { @@ -102,6 +112,8 @@ reg = <0 0x5>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_e_opp>; + performance-domains = <&m6_e_cpufreq>; }; cpu6: cpu@10100 { @@ -110,6 +122,8 @@ reg = <0 0x10100>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; cpu7: cpu@10101 { @@ -118,6 +132,8 @@ reg = <0 0x10101>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; cpu8: cpu@40102 { @@ -126,6 +142,8 @@ reg = <0 0x40102>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; cpu9: cpu@40103 { @@ -134,6 +152,8 @@ reg = <0 0x40103>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; cpu10: cpu@40104 { @@ -142,6 +162,8 @@ reg = <0 0x40104>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; cpu11: cpu@40105 { @@ -150,9 +172,35 @@ reg = <0 0x40105>; enable-method = "spin-table"; cpu-release-addr = <0 0>; /* Filled by the loader. */ + operating-points-v2 = <&m6_pm_opp>; + performance-domains = <&m6_pm_cpufreq>; }; }; + m6_e_opp: opp-table-e { + compatible = "operating-points-v2"; + opp-shared; + opp-972000000 { + opp-hz = /bits/ 64 <972000000>; + opp-level = <2>; + }; + opp-1152000000 { + opp-hz = /bits/ 64 <1152000000>; + opp-level = <3>; + }; + }; + m6_pm_opp: opp-table-pm { + compatible = "operating-points-v2"; + opp-shared; + opp-1440000000 { + opp-hz = /bits/ 64 <1440000000>; + opp-level = <2>; + }; + opp-1728000000 { + opp-hz = /bits/ 64 <1728000000>; + opp-level = <3>; + }; + }; timer { compatible = "arm,armv8-timer"; interrupts = , @@ -178,6 +226,17 @@ reg-names = "core", "event"; }; + m6_e_cpufreq: performance-controller@210e20000 { + compatible = "apple,t8152-cluster-cpufreq"; + reg = <2 0x10e20000 0 0x1000>; + #performance-domain-cells = <0>; + }; + m6_pm_cpufreq: performance-controller@211e20000 { + compatible = "apple,t8152-cluster-cpufreq"; + reg = <2 0x11e20000 0 0x1000>; + #performance-domain-cells = <0>; + }; + dockchannel: serial@30812c000 { compatible = "apple,dockchannel-uart"; reg = <3 0x0812c000 0 0x4000>, diff --git a/arch/arm64/configs/asahi.config b/arch/arm64/configs/asahi.config index 1e05c59146c1fa..41797e7774899f 100644 --- a/arch/arm64/configs/asahi.config +++ b/arch/arm64/configs/asahi.config @@ -11,13 +11,13 @@ CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR=y CONFIG_BT_BRCMEXT=y CONFIG_BT_HCIBCM4377=m CONFIG_BT_MTK7932_PCIE=m -CONFIG_PCIE_APPLE=m -CONFIG_PCIE_APPLE_PIODMA_DIAG=y -CONFIG_PCIEASPM=y +CONFIG_PCIE_APPLE=y CONFIG_NVME_APPLE=m CONFIG_MMC_SDHCI_PCI=m CONFIG_BRCMFMAC=m CONFIG_BRCMFMAC_PCIE=y +CONFIG_PCIE_APPLE_PIODMA_DIAG=y +CONFIG_PCIEASPM=y CONFIG_NET_VENDOR_INTEL=y CONFIG_IGB=m CONFIG_IGC=m @@ -72,7 +72,7 @@ CONFIG_APPLE_ADMAC=m CONFIG_APPLE_SIO=m CONFIG_MFD_MACSMC=m CONFIG_COMMON_CLK_APPLE_NCO=m -CONFIG_APPLE_DART=m +CONFIG_APPLE_DART=y CONFIG_APPLE_DOCKCHANNEL=m CONFIG_SERIAL_APPLE_DOCKCHANNEL=y CONFIG_SERIAL_APPLE_DOCKCHANNEL_EARLYCON=y @@ -101,3 +101,8 @@ CONFIG_APPLE_M1_CPU_PMU=y CONFIG_NVMEM_APPLE_EFUSES=m CONFIG_NVMEM_APPLE_SPMI=m CONFIG_MUX_APPLE_DPXBAR=m +CONFIG_INFINIBAND=m +CONFIG_INFINIBAND_USER_ACCESS=m +CONFIG_INFINIBAND_USER_MAD=m +CONFIG_RDMA_RXE=m +CONFIG_RDMA_SIW=m diff --git a/arch/arm64/configs/j700-device-parity.config b/arch/arm64/configs/j700-device-parity.config new file mode 100644 index 00000000000000..55bce7a9ac0af5 --- /dev/null +++ b/arch/arm64/configs/j700-device-parity.config @@ -0,0 +1,1053 @@ +# J700 (MacBook Neo, T8140) device-parity fragment. +# +# Brings the J700 configuration to device and feature parity with a general-purpose +# distribution kernel (reference: Fedora Asahi Remix 44, 7.1.13-402.asahi.fc44.aarch64+16k): +# drivers for devices that can attach over USB or Bluetooth, and common protocols, +# filesystems and features. Most entries are modules. Nothing here changes an option +# that defconfig, asahi.config, j700-radio.config or j700.config set. +# +# Merge after j700.config; see that file for the full sequence. + +# Prerequisites of the entries below: nftables core, hidraw, uinput and the ALSA sequencer +CONFIG_NF_TABLES=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_IPV4=y +CONFIG_NF_TABLES_IPV6=y +CONFIG_NF_TABLES_NETDEV=y +CONFIG_NFT_CT=m +CONFIG_NFT_LOG=m +CONFIG_NFT_LIMIT=m +CONFIG_NFT_MASQ=m +CONFIG_NFT_NAT=m +CONFIG_NFT_REJECT=m +CONFIG_NFT_REJECT_INET=m +CONFIG_NFT_FIB_INET=m +CONFIG_NFT_FIB_IPV4=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_HIDRAW=y +CONFIG_INPUT_UINPUT=m +CONFIG_SND_SEQUENCER=m +CONFIG_SND_HRTIMER=m +CONFIG_SND_SEQ_DUMMY=m +CONFIG_SND_SEQ_HRTIMER_DEFAULT=y + +# HID / input / game controllers (USB + Bluetooth HID devices, xpad, joydev, USB tablets/touchscreens) +CONFIG_DRAGONRISE_FF=y +CONFIG_GREENASIA_FF=y +CONFIG_HID_A4TECH=m +CONFIG_HID_ACCUTOUCH=m +CONFIG_HID_ACRUX=m +CONFIG_HID_ACRUX_FF=y +CONFIG_HID_ALPS=m +CONFIG_HID_APPLEIR=m +CONFIG_HID_AUREAL=m +CONFIG_HID_BATTERY_STRENGTH=y +CONFIG_HID_BELKIN=m +CONFIG_HID_BETOP_FF=m +CONFIG_HID_BIGBEN_FF=m +CONFIG_HID_CHERRY=m +CONFIG_HID_CHICONY=m +CONFIG_HID_CMEDIA=m +CONFIG_HID_CORSAIR=m +CONFIG_HID_COUGAR=m +CONFIG_HID_CP2112=m +CONFIG_HID_CREATIVE_SB0540=m +CONFIG_HID_CYPRESS=m +CONFIG_HID_DRAGONRISE=m +CONFIG_HID_ELAN=m +CONFIG_HID_ELECOM=m +CONFIG_HID_ELO=m +CONFIG_HID_EMS_FF=m +CONFIG_HID_EVISION=m +CONFIG_HID_EZKEY=m +CONFIG_HID_FT260=m +CONFIG_HID_GEMBIRD=m +CONFIG_HID_GFRM=m +CONFIG_HID_GLORIOUS=m +CONFIG_HID_GOOGLE_STADIA_FF=m +CONFIG_HID_GREENASIA=m +CONFIG_HID_GT683R=m +CONFIG_HID_GYRATION=m +CONFIG_HID_HOLTEK=m +CONFIG_HID_HUAWEI=m +CONFIG_HID_ICADE=m +CONFIG_HID_ITE=m +CONFIG_HID_JABRA=m +CONFIG_HID_KENSINGTON=m +CONFIG_HID_KEYTOUCH=m +CONFIG_HID_KYE=m +CONFIG_HID_KYSONA=m +CONFIG_HID_LCPOWER=m +CONFIG_HID_LED=m +CONFIG_HID_LENOVO=m +CONFIG_HID_LENOVO_GO=m +CONFIG_HID_LENOVO_GO_S=m +CONFIG_HID_LETSKETCH=m +CONFIG_HID_LOGITECH=m +CONFIG_HID_LOGITECH_DJ=m +CONFIG_HID_LOGITECH_HIDPP=m +CONFIG_HID_MACALLY=m +CONFIG_HID_MALTRON=m +CONFIG_HID_MAYFLASH=m +CONFIG_HID_MCP2221=m +CONFIG_HID_MEGAWORLD_FF=m +CONFIG_HID_MICROSOFT=m +CONFIG_HID_MONTEREY=m +CONFIG_HID_NINTENDO=m +CONFIG_HID_NTI=m +CONFIG_HID_NTRIG=m +CONFIG_HID_NVIDIA_SHIELD=m +CONFIG_HID_ORTEK=m +CONFIG_HID_PANTHERLORD=m +CONFIG_HID_PENMOUNT=m +CONFIG_HID_PETALYNX=m +CONFIG_HID_PICOLCD=m +CONFIG_HID_PICOLCD_BACKLIGHT=y +CONFIG_HID_PICOLCD_FB=y +CONFIG_HID_PICOLCD_LEDS=y +CONFIG_HID_PID=y +CONFIG_HID_PLANTRONICS=m +CONFIG_HID_PLAYSTATION=m +CONFIG_HID_PRIMAX=m +CONFIG_HID_PRODIKEYS=m +CONFIG_HID_PXRC=m +CONFIG_HID_RAPOO=m +CONFIG_HID_RAZER=m +CONFIG_HID_RETRODE=m +CONFIG_HID_RMI=m +CONFIG_HID_ROCCAT=m +CONFIG_HID_SAITEK=m +CONFIG_HID_SAMSUNG=m +CONFIG_HID_SEMITEK=m +CONFIG_HID_SIGMAMICRO=m +CONFIG_HID_SMARTJOYPLUS=m +CONFIG_HID_SONY=m +CONFIG_HID_SPEEDLINK=m +CONFIG_HID_STEAM=m +CONFIG_HID_STEELSERIES=m +CONFIG_HID_SUNPLUS=m +CONFIG_HID_THINGM=m +CONFIG_HID_THRUSTMASTER=m +CONFIG_HID_TIVO=m +CONFIG_HID_TOPRE=m +CONFIG_HID_TOPSEED=m +CONFIG_HID_TWINHAN=m +CONFIG_HID_U2FZERO=m +CONFIG_HID_UCLOGIC=m +CONFIG_HID_UDRAW_PS3=m +CONFIG_HID_UNIVERSAL_PIDFF=m +CONFIG_HID_VIEWSONIC=m +CONFIG_HID_VIVALDI=m +CONFIG_HID_WACOM=m +CONFIG_HID_WALTOP=m +CONFIG_HID_WIIMOTE=m +CONFIG_HID_WINWING=m +CONFIG_HID_XIAOMI=m +CONFIG_HID_XINMO=m +CONFIG_HID_ZEROPLUS=m +CONFIG_HID_ZYDACRON=m +CONFIG_HOLTEK_FF=y +CONFIG_INPUT_JOYDEV=m +CONFIG_INPUT_JOYSTICK=y +CONFIG_INPUT_TABLET=y +CONFIG_JOYSTICK_IFORCE=m +CONFIG_JOYSTICK_IFORCE_USB=m +CONFIG_JOYSTICK_PXRC=m +CONFIG_JOYSTICK_XPAD=m +CONFIG_JOYSTICK_XPAD_FF=y +CONFIG_JOYSTICK_XPAD_LEDS=y +CONFIG_LOGIG940_FF=y +CONFIG_LOGIRUMBLEPAD2_FF=y +CONFIG_LOGITECH_FF=y +CONFIG_LOGIWHEELS_FF=y +CONFIG_MOUSE_SYNAPTICS_USB=m +CONFIG_NINTENDO_FF=y +CONFIG_NVIDIA_SHIELD_FF=y +CONFIG_PANTHERLORD_FF=y +CONFIG_PLAYSTATION_FF=y +CONFIG_RMI4_CORE=m +CONFIG_RMI4_F03=y +CONFIG_RMI4_F11=y +CONFIG_RMI4_F12=y +CONFIG_RMI4_F1A=y +CONFIG_RMI4_F21=y +CONFIG_RMI4_F30=y +CONFIG_RMI4_F34=y +CONFIG_RMI4_F3A=y +CONFIG_RMI4_F55=y +CONFIG_SMARTJOYPLUS_FF=y +CONFIG_SONY_FF=y +CONFIG_STEAM_FF=y +CONFIG_TABLET_USB_ACECAD=m +CONFIG_TABLET_USB_AIPTEK=m +CONFIG_TABLET_USB_HANWANG=m +CONFIG_TABLET_USB_KBTAB=m +CONFIG_TABLET_USB_PEGASUS=m +CONFIG_THRUSTMASTER_FF=y +CONFIG_TOUCHSCREEN_USB_3M=y +CONFIG_TOUCHSCREEN_USB_COMPOSITE=m +CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y +CONFIG_TOUCHSCREEN_USB_E2I=y +CONFIG_TOUCHSCREEN_USB_EASYTOUCH=y +CONFIG_TOUCHSCREEN_USB_EGALAX=y +CONFIG_TOUCHSCREEN_USB_ELO=y +CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y +CONFIG_TOUCHSCREEN_USB_ETURBO=y +CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y +CONFIG_TOUCHSCREEN_USB_GOTOP=y +CONFIG_TOUCHSCREEN_USB_GUNZE=y +CONFIG_TOUCHSCREEN_USB_IDEALTEK=y +CONFIG_TOUCHSCREEN_USB_IRTOUCH=y +CONFIG_TOUCHSCREEN_USB_ITM=y +CONFIG_TOUCHSCREEN_USB_JASTEC=y +CONFIG_TOUCHSCREEN_USB_NEXIO=y +CONFIG_TOUCHSCREEN_USB_PANJIT=y +CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y +CONFIG_USB_HIDDEV=y +CONFIG_ZEROPLUS_FF=y + +# USB device classes: serial adapters, storage sub-drivers, printers/TMC/WDM, misc USB gadgets, usbmon, USB/IP, card readers, maker adapters +CONFIG_APPLE_MFI_FASTCHARGE=m +CONFIG_DLN2_ADC=m +CONFIG_GNSS_USB=m +CONFIG_GPIB=m +CONFIG_GPIB_AGILENT_82357A=m +CONFIG_GPIB_COMMON=m +CONFIG_GPIB_LPVO=m +CONFIG_GPIB_NI_USB=m +CONFIG_GPIO_DLN2=m +CONFIG_GPIO_MPSSE=m +CONFIG_I2C_CP2615=m +CONFIG_I2C_DIOLAN_U2C=m +CONFIG_I2C_DLN2=m +CONFIG_I2C_TINY_USB=m +CONFIG_MEMSTICK=m +CONFIG_MEMSTICK_REALTEK_USB=m +CONFIG_MFD_DLN2=m +CONFIG_MISC_RTSX_USB=m +CONFIG_MMC_REALTEK_USB=m +CONFIG_MMC_USHC=m +CONFIG_MMC_VUB300=m +CONFIG_MSPRO_BLOCK=m +CONFIG_REALTEK_AUTOPM=y +CONFIG_SENSORS_AQUACOMPUTER_D5NEXT=m +CONFIG_SENSORS_ASUS_ROG_RYUJIN=m +CONFIG_SENSORS_CORSAIR_CPRO=m +CONFIG_SENSORS_CORSAIR_PSU=m +CONFIG_SENSORS_GIGABYTE_WATERFORCE=m +CONFIG_SENSORS_NZXT_KRAKEN2=m +CONFIG_SENSORS_NZXT_KRAKEN3=m +CONFIG_SENSORS_NZXT_SMART2=m +CONFIG_SENSORS_POWERZ=m +CONFIG_SPI_CH341=m +CONFIG_SPI_DLN2=m +CONFIG_USBIP_CORE=m +CONFIG_USBIP_HOST=m +CONFIG_USBIP_VHCI_HCD=m +CONFIG_USBPCWATCHDOG=m +CONFIG_USB_ADUTUX=m +CONFIG_USB_APPLEDISPLAY=m +CONFIG_USB_CHAOSKEY=m +CONFIG_USB_EMI26=m +CONFIG_USB_EMI62=m +CONFIG_USB_EZUSB_FX2=m +CONFIG_USB_IDMOUSE=m +CONFIG_USB_IOWARRIOR=m +CONFIG_USB_ISIGHTFW=m +CONFIG_USB_LCD=m +CONFIG_USB_LD=m +CONFIG_USB_LEGOTOWER=m +CONFIG_USB_MDC800=m +CONFIG_USB_MICROTEK=m +CONFIG_USB_MON=m +CONFIG_USB_PRINTER=m +CONFIG_USB_SERIAL_AIRCABLE=m +CONFIG_USB_SERIAL_ARK3116=m +CONFIG_USB_SERIAL_BELKIN=m +CONFIG_USB_SERIAL_CH341=m +CONFIG_USB_SERIAL_CYBERJACK=m +CONFIG_USB_SERIAL_CYPRESS_M8=m +CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m +CONFIG_USB_SERIAL_EDGEPORT=m +CONFIG_USB_SERIAL_EDGEPORT_TI=m +CONFIG_USB_SERIAL_EMPEG=m +CONFIG_USB_SERIAL_F81232=m +CONFIG_USB_SERIAL_F8153X=m +CONFIG_USB_SERIAL_GARMIN=m +CONFIG_USB_SERIAL_GENERIC=y +CONFIG_USB_SERIAL_IPAQ=m +CONFIG_USB_SERIAL_IPW=m +CONFIG_USB_SERIAL_IR=m +CONFIG_USB_SERIAL_IUU=m +CONFIG_USB_SERIAL_KEYSPAN=m +CONFIG_USB_SERIAL_KEYSPAN_PDA=m +CONFIG_USB_SERIAL_KLSI=m +CONFIG_USB_SERIAL_KOBIL_SCT=m +CONFIG_USB_SERIAL_MCT_U232=m +CONFIG_USB_SERIAL_MOS7720=m +CONFIG_USB_SERIAL_MOS7840=m +CONFIG_USB_SERIAL_NAVMAN=m +CONFIG_USB_SERIAL_OMNINET=m +CONFIG_USB_SERIAL_OPTICON=m +CONFIG_USB_SERIAL_OTI6858=m +CONFIG_USB_SERIAL_PL2303=m +CONFIG_USB_SERIAL_QCAUX=m +CONFIG_USB_SERIAL_QT2=m +CONFIG_USB_SERIAL_QUALCOMM=m +CONFIG_USB_SERIAL_SAFE=m +CONFIG_USB_SERIAL_SAFE_PADDED=y +CONFIG_USB_SERIAL_SIERRAWIRELESS=m +CONFIG_USB_SERIAL_SIMPLE=m +CONFIG_USB_SERIAL_SPCP8X5=m +CONFIG_USB_SERIAL_SSU100=m +CONFIG_USB_SERIAL_SYMBOL=m +CONFIG_USB_SERIAL_TI=m +CONFIG_USB_SERIAL_UPD78F0730=m +CONFIG_USB_SERIAL_VISOR=m +CONFIG_USB_SERIAL_WHITEHEAT=m +CONFIG_USB_SERIAL_XR=m +CONFIG_USB_SERIAL_XSENS_MT=m +CONFIG_USB_SEVSEG=m +CONFIG_USB_SISUSBVGA=m +CONFIG_USB_STORAGE_ALAUDA=m +CONFIG_USB_STORAGE_CYPRESS_ATACB=m +CONFIG_USB_STORAGE_DATAFAB=m +CONFIG_USB_STORAGE_ENE_UB6250=m +CONFIG_USB_STORAGE_FREECOM=m +CONFIG_USB_STORAGE_ISD200=m +CONFIG_USB_STORAGE_JUMPSHOT=m +CONFIG_USB_STORAGE_KARMA=m +CONFIG_USB_STORAGE_ONETOUCH=m +CONFIG_USB_STORAGE_REALTEK=m +CONFIG_USB_STORAGE_SDDR09=m +CONFIG_USB_STORAGE_SDDR55=m +CONFIG_USB_STORAGE_USBAT=m +CONFIG_USB_TMC=m +CONFIG_USB_TRANCEVIBRATOR=m +CONFIG_USB_WDM=m +CONFIG_USB_YUREX=m +CONFIG_W1=m +CONFIG_W1_MASTER_DS2490=m +CONFIG_W1_SLAVE_DS2405=m +CONFIG_W1_SLAVE_DS2406=m +CONFIG_W1_SLAVE_DS2408=m +CONFIG_W1_SLAVE_DS2413=m +CONFIG_W1_SLAVE_DS2423=m +CONFIG_W1_SLAVE_DS2430=m +CONFIG_W1_SLAVE_DS2431=m +CONFIG_W1_SLAVE_DS2433=m +CONFIG_W1_SLAVE_DS2433_CRC=y +CONFIG_W1_SLAVE_DS2438=m +CONFIG_W1_SLAVE_DS2780=m +CONFIG_W1_SLAVE_DS2781=m +CONFIG_W1_SLAVE_DS2805=m +CONFIG_W1_SLAVE_DS28E04=m +CONFIG_W1_SLAVE_SMEM=m +CONFIG_W1_SLAVE_THERM=m +CONFIG_XILLYUSB=m + +# USB network adapters: USB Ethernet/tethering/mobile broadband, USB Wi-Fi dongles, USB CAN adapters +CONFIG_AR5523=m +CONFIG_ATH10K_USB=m +CONFIG_ATH6KL=m +CONFIG_ATH6KL_USB=m +CONFIG_ATH9K_HTC=m +CONFIG_BRCMFMAC_USB=y +CONFIG_CAN_8DEV_USB=m +CONFIG_CAN_EMS_USB=m +CONFIG_CAN_ESD_USB=m +CONFIG_CAN_F81604=m +CONFIG_CAN_GS_USB=m +CONFIG_CAN_ISOTP=m +CONFIG_CAN_J1939=m +CONFIG_CAN_KVASER_USB=m +CONFIG_CAN_MCBA_USB=m +CONFIG_CAN_PEAK_USB=m +CONFIG_CARL9170=m +CONFIG_CARL9170_LEDS=y +CONFIG_LIBERTAS_THINFIRM=m +CONFIG_LIBERTAS_THINFIRM_USB=m +CONFIG_MT7601U=m +CONFIG_MT7663U=m +CONFIG_MT76x0U=m +CONFIG_MT76x2U=m +CONFIG_MT7921U=m +CONFIG_MT7925U=m +CONFIG_RT2500USB=m +CONFIG_RT2800USB=m +CONFIG_RT2800USB_RT33XX=y +CONFIG_RT2800USB_RT3573=y +CONFIG_RT2800USB_RT35XX=y +CONFIG_RT2800USB_RT53XX=y +CONFIG_RT2800USB_RT55XX=y +CONFIG_RT2800USB_UNKNOWN=y +CONFIG_RT2X00=m +CONFIG_RT73USB=m +CONFIG_RTL8187=m +CONFIG_RTL8192DU=m +CONFIG_RTL8XXXU=m +CONFIG_RTW88=m +CONFIG_RTW88_8723DU=m +CONFIG_RTW88_8812AU=m +CONFIG_RTW88_8814AU=m +CONFIG_RTW88_8821AU=m +CONFIG_RTW88_8821CU=m +CONFIG_RTW88_8822BU=m +CONFIG_RTW88_8822CU=m +CONFIG_RTW89=m +CONFIG_RTW89_8851BU=m +CONFIG_RTW89_8852AU=m +CONFIG_RTW89_8852BU=m +CONFIG_RTW89_8852CU=m +CONFIG_USB_ALI_M5632=y +CONFIG_USB_AN2720=y +CONFIG_USB_CATC=m +CONFIG_USB_EPSON2888=y +CONFIG_USB_HSO=m +CONFIG_USB_IPHETH=m +CONFIG_USB_KAWETH=m +CONFIG_USB_KC2190=y +CONFIG_USB_NET_AQC111=m +CONFIG_USB_NET_CDC_EEM=m +CONFIG_USB_NET_CDC_MBIM=m +CONFIG_USB_NET_CH9200=m +CONFIG_USB_NET_CX82310_ETH=m +CONFIG_USB_NET_GL620A=m +CONFIG_USB_NET_HUAWEI_CDC_NCM=m +CONFIG_USB_NET_INT51X1=m +CONFIG_USB_NET_KALMIA=m +CONFIG_USB_NET_QMI_WWAN=m +CONFIG_USB_NET_RNDIS_HOST=m +CONFIG_USB_NET_SR9700=m +CONFIG_USB_SIERRA_NET=m +CONFIG_USB_VL600=m +CONFIG_ZD1211RW=m + +# Storage: optical/SCSI generic, device-mapper (LUKS, LVM, verity, integrity), MD RAID, partition tables +CONFIG_AIX_PARTITION=y +CONFIG_BLK_DEV_SR=m +CONFIG_BSD_DISKLABEL=y +CONFIG_CHR_DEV_SG=m +CONFIG_DM_CACHE=m +CONFIG_DM_CACHE_SMQ=m +CONFIG_DM_CRYPT=m +CONFIG_DM_INTEGRITY=m +CONFIG_DM_RAID=m +CONFIG_DM_SNAPSHOT=m +CONFIG_DM_THIN_PROVISIONING=m +CONFIG_DM_UEVENT=y +CONFIG_DM_VERITY=m +CONFIG_DM_VERITY_FEC=y +CONFIG_DM_WRITECACHE=m +CONFIG_LDM_PARTITION=y +CONFIG_MAC_PARTITION=y +CONFIG_MD_LINEAR=m +CONFIG_MD_RAID0=m +CONFIG_MD_RAID1=m +CONFIG_MD_RAID10=m +CONFIG_MD_RAID456=m +CONFIG_MINIX_SUBPARTITION=y +CONFIG_OSF_PARTITION=y +CONFIG_PARTITION_ADVANCED=y +CONFIG_SGI_PARTITION=y +CONFIG_SOLARIS_X86_PARTITION=y +CONFIG_SUN_PARTITION=y +CONFIG_UNIXWARE_DISKLABEL=y + +# Filesystems for removable media and images, binfmt_misc, SMB client +CONFIG_BINFMT_MISC=m +CONFIG_CIFS=m +CONFIG_CIFS_ALLOW_INSECURE_LEGACY=y +CONFIG_CIFS_DFS_UPCALL=y +CONFIG_CIFS_POSIX=y +CONFIG_CIFS_SWN_UPCALL=y +CONFIG_CIFS_UPCALL=y +CONFIG_CIFS_XATTR=y +CONFIG_EROFS_FS=m +CONFIG_EROFS_FS_BACKED_BY_FILE=y +CONFIG_EROFS_FS_POSIX_ACL=y +CONFIG_EROFS_FS_SECURITY=y +CONFIG_EROFS_FS_XATTR=y +CONFIG_EROFS_FS_ZIP=y +CONFIG_EROFS_FS_ZIP_ACCEL=y +CONFIG_EROFS_FS_ZIP_DEFLATE=y +CONFIG_EROFS_FS_ZIP_LZMA=y +CONFIG_EROFS_FS_ZIP_ZSTD=y +CONFIG_EXFAT_FS=m +CONFIG_F2FS_FS=m +CONFIG_F2FS_FS_COMPRESSION=y +CONFIG_F2FS_FS_LZ4=y +CONFIG_F2FS_FS_LZ4HC=y +CONFIG_F2FS_FS_LZO=y +CONFIG_F2FS_FS_LZORLE=y +CONFIG_F2FS_FS_POSIX_ACL=y +CONFIG_F2FS_FS_SECURITY=y +CONFIG_F2FS_FS_XATTR=y +CONFIG_F2FS_FS_ZSTD=y +CONFIG_F2FS_IOSTAT=y +CONFIG_F2FS_STAT_FS=y +CONFIG_F2FS_UNFAIR_RWSEM=y +CONFIG_HFSPLUS_FS=m +CONFIG_HFS_FS=m +CONFIG_ISO9660_FS=m +CONFIG_JOLIET=y +CONFIG_MSDOS_FS=m +CONFIG_NLS_ASCII=m +CONFIG_NLS_CODEPAGE_1250=m +CONFIG_NLS_CODEPAGE_1251=m +CONFIG_NLS_CODEPAGE_737=m +CONFIG_NLS_CODEPAGE_775=m +CONFIG_NLS_CODEPAGE_850=m +CONFIG_NLS_CODEPAGE_852=m +CONFIG_NLS_CODEPAGE_855=m +CONFIG_NLS_CODEPAGE_857=m +CONFIG_NLS_CODEPAGE_860=m +CONFIG_NLS_CODEPAGE_861=m +CONFIG_NLS_CODEPAGE_862=m +CONFIG_NLS_CODEPAGE_863=m +CONFIG_NLS_CODEPAGE_864=m +CONFIG_NLS_CODEPAGE_865=m +CONFIG_NLS_CODEPAGE_866=m +CONFIG_NLS_CODEPAGE_869=m +CONFIG_NLS_CODEPAGE_874=m +CONFIG_NLS_CODEPAGE_932=m +CONFIG_NLS_CODEPAGE_936=m +CONFIG_NLS_CODEPAGE_949=m +CONFIG_NLS_CODEPAGE_950=m +CONFIG_NLS_ISO8859_13=m +CONFIG_NLS_ISO8859_14=m +CONFIG_NLS_ISO8859_15=m +CONFIG_NLS_ISO8859_2=m +CONFIG_NLS_ISO8859_3=m +CONFIG_NLS_ISO8859_4=m +CONFIG_NLS_ISO8859_5=m +CONFIG_NLS_ISO8859_6=m +CONFIG_NLS_ISO8859_7=m +CONFIG_NLS_ISO8859_8=m +CONFIG_NLS_ISO8859_9=m +CONFIG_NLS_KOI8_R=m +CONFIG_NLS_KOI8_U=m +CONFIG_NLS_MAC_CELTIC=m +CONFIG_NLS_MAC_CENTEURO=m +CONFIG_NLS_MAC_CROATIAN=m +CONFIG_NLS_MAC_CYRILLIC=m +CONFIG_NLS_MAC_GAELIC=m +CONFIG_NLS_MAC_GREEK=m +CONFIG_NLS_MAC_ICELAND=m +CONFIG_NLS_MAC_INUIT=m +CONFIG_NLS_MAC_ROMAN=m +CONFIG_NLS_MAC_ROMANIAN=m +CONFIG_NLS_MAC_TURKISH=m +CONFIG_NLS_UTF8=m +CONFIG_NTFS3_FS=m +CONFIG_NTFS3_FS_POSIX_ACL=y +CONFIG_NTFS3_LZX_XPRESS=y +CONFIG_SQUASHFS_LZ4=y +CONFIG_SQUASHFS_LZO=y +CONFIG_SQUASHFS_XATTR=y +CONFIG_SQUASHFS_XZ=y +CONFIG_SQUASHFS_ZSTD=y +CONFIG_UDF_FS=m +CONFIG_XFS_FS=m +CONFIG_XFS_ONLINE_REPAIR=y +CONFIG_XFS_ONLINE_SCRUB=y +CONFIG_XFS_ONLINE_SCRUB_STATS=y +CONFIG_XFS_POSIX_ACL=y +CONFIG_XFS_QUOTA=y +CONFIG_XFS_RT=y +CONFIG_XFS_SUPPORT_ASCII_CI=y +CONFIG_XFS_SUPPORT_V4=y +CONFIG_ZISOFS=y + +# Network protocols: WireGuard/OpenVPN-DCO/L2TP/PPP/IPsec/kTLS, tunnels, policy routing, nftables+xtables+ipset+ebtables, tc, TCP CC, ss diag, virtual links +CONFIG_BONDING=m +CONFIG_BRIDGE_EBT_802_3=m +CONFIG_BRIDGE_EBT_AMONG=m +CONFIG_BRIDGE_EBT_ARP=m +CONFIG_BRIDGE_EBT_ARPREPLY=m +CONFIG_BRIDGE_EBT_BROUTE=m +CONFIG_BRIDGE_EBT_DNAT=m +CONFIG_BRIDGE_EBT_IP=m +CONFIG_BRIDGE_EBT_IP6=m +CONFIG_BRIDGE_EBT_LIMIT=m +CONFIG_BRIDGE_EBT_LOG=m +CONFIG_BRIDGE_EBT_MARK=m +CONFIG_BRIDGE_EBT_MARK_T=m +CONFIG_BRIDGE_EBT_NFLOG=m +CONFIG_BRIDGE_EBT_PKTTYPE=m +CONFIG_BRIDGE_EBT_REDIRECT=m +CONFIG_BRIDGE_EBT_SNAT=m +CONFIG_BRIDGE_EBT_STP=m +CONFIG_BRIDGE_EBT_T_FILTER=m +CONFIG_BRIDGE_EBT_T_NAT=m +CONFIG_BRIDGE_EBT_VLAN=m +CONFIG_BRIDGE_NF_EBTABLES=m +CONFIG_BRIDGE_NF_EBTABLES_LEGACY=m +CONFIG_CLS_U32_MARK=y +CONFIG_CLS_U32_PERF=y +CONFIG_DEFAULT_CUBIC=y +CONFIG_DUMMY=m +CONFIG_GACT_PROB=y +CONFIG_GENEVE=m +CONFIG_IFB=m +CONFIG_INET6_AH=m +CONFIG_INET6_ESP=m +CONFIG_INET6_ESPINTCP=y +CONFIG_INET6_ESP_OFFLOAD=m +CONFIG_INET6_IPCOMP=m +CONFIG_INET_AH=m +CONFIG_INET_DIAG_DESTROY=y +CONFIG_INET_ESP=m +CONFIG_INET_ESPINTCP=y +CONFIG_INET_ESP_OFFLOAD=m +CONFIG_INET_IPCOMP=m +CONFIG_INET_RAW_DIAG=m +CONFIG_INET_UDP_DIAG=m +CONFIG_IP6_NF_FILTER=m +CONFIG_IP6_NF_IPTABLES_LEGACY=m +CONFIG_IP6_NF_MANGLE=m +CONFIG_IP6_NF_MATCH_AH=m +CONFIG_IP6_NF_MATCH_EUI64=m +CONFIG_IP6_NF_MATCH_FRAG=m +CONFIG_IP6_NF_MATCH_HL=m +CONFIG_IP6_NF_MATCH_IPV6HEADER=m +CONFIG_IP6_NF_MATCH_MH=m +CONFIG_IP6_NF_MATCH_OPTS=m +CONFIG_IP6_NF_MATCH_RPFILTER=m +CONFIG_IP6_NF_MATCH_RT=m +CONFIG_IP6_NF_MATCH_SRH=m +CONFIG_IP6_NF_NAT=m +CONFIG_IP6_NF_RAW=m +CONFIG_IP6_NF_TARGET_HL=m +CONFIG_IP6_NF_TARGET_MASQUERADE=m +CONFIG_IP6_NF_TARGET_NPT=m +CONFIG_IP6_NF_TARGET_REJECT=m +CONFIG_IPV6_GRE=m +CONFIG_IPV6_MULTIPLE_TABLES=y +CONFIG_IPV6_TUNNEL=m +CONFIG_IPV6_VTI=m +CONFIG_IP_ADVANCED_ROUTER=y +CONFIG_IP_MULTIPLE_TABLES=y +CONFIG_IP_NF_ARPFILTER=m +CONFIG_IP_NF_ARPTABLES=m +CONFIG_IP_NF_ARP_MANGLE=m +CONFIG_IP_NF_FILTER=m +CONFIG_IP_NF_IPTABLES_LEGACY=m +CONFIG_IP_NF_MANGLE=m +CONFIG_IP_NF_MATCH_AH=m +CONFIG_IP_NF_MATCH_ECN=m +CONFIG_IP_NF_MATCH_RPFILTER=m +CONFIG_IP_NF_MATCH_TTL=m +CONFIG_IP_NF_NAT=m +CONFIG_IP_NF_RAW=m +CONFIG_IP_NF_TARGET_ECN=m +CONFIG_IP_NF_TARGET_MASQUERADE=m +CONFIG_IP_NF_TARGET_NETMAP=m +CONFIG_IP_NF_TARGET_REDIRECT=m +CONFIG_IP_NF_TARGET_REJECT=m +CONFIG_IP_NF_TARGET_TTL=m +CONFIG_IP_ROUTE_MULTIPATH=y +CONFIG_IP_SET=m +CONFIG_IP_SET_BITMAP_IP=m +CONFIG_IP_SET_BITMAP_IPMAC=m +CONFIG_IP_SET_BITMAP_PORT=m +CONFIG_IP_SET_HASH_IP=m +CONFIG_IP_SET_HASH_IPMAC=m +CONFIG_IP_SET_HASH_IPMARK=m +CONFIG_IP_SET_HASH_IPPORT=m +CONFIG_IP_SET_HASH_IPPORTIP=m +CONFIG_IP_SET_HASH_IPPORTNET=m +CONFIG_IP_SET_HASH_MAC=m +CONFIG_IP_SET_HASH_NET=m +CONFIG_IP_SET_HASH_NETIFACE=m +CONFIG_IP_SET_HASH_NETNET=m +CONFIG_IP_SET_HASH_NETPORT=m +CONFIG_IP_SET_HASH_NETPORTNET=m +CONFIG_IP_SET_LIST_SET=m +CONFIG_L2TP=m +CONFIG_L2TP_ETH=m +CONFIG_L2TP_IP=m +CONFIG_L2TP_V3=y +CONFIG_NETFILTER_NETLINK_ACCT=m +CONFIG_NETFILTER_NETLINK_GLUE_CT=y +CONFIG_NETFILTER_NETLINK_HOOK=m +CONFIG_NETFILTER_NETLINK_LOG=m +CONFIG_NETFILTER_NETLINK_OSF=m +CONFIG_NETFILTER_NETLINK_QUEUE=m +CONFIG_NETFILTER_XTABLES_LEGACY=y +CONFIG_NETFILTER_XT_CONNMARK=m +CONFIG_NETFILTER_XT_MATCH_BPF=m +CONFIG_NETFILTER_XT_MATCH_CGROUP=m +CONFIG_NETFILTER_XT_MATCH_CLUSTER=m +CONFIG_NETFILTER_XT_MATCH_COMMENT=m +CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m +CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m +CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m +CONFIG_NETFILTER_XT_MATCH_CONNMARK=m +CONFIG_NETFILTER_XT_MATCH_CPU=m +CONFIG_NETFILTER_XT_MATCH_DCCP=m +CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m +CONFIG_NETFILTER_XT_MATCH_DSCP=m +CONFIG_NETFILTER_XT_MATCH_ECN=m +CONFIG_NETFILTER_XT_MATCH_ESP=m +CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m +CONFIG_NETFILTER_XT_MATCH_HELPER=m +CONFIG_NETFILTER_XT_MATCH_HL=m +CONFIG_NETFILTER_XT_MATCH_IPCOMP=m +CONFIG_NETFILTER_XT_MATCH_IPRANGE=m +CONFIG_NETFILTER_XT_MATCH_L2TP=m +CONFIG_NETFILTER_XT_MATCH_LENGTH=m +CONFIG_NETFILTER_XT_MATCH_LIMIT=m +CONFIG_NETFILTER_XT_MATCH_MAC=m +CONFIG_NETFILTER_XT_MATCH_MARK=m +CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m +CONFIG_NETFILTER_XT_MATCH_NFACCT=m +CONFIG_NETFILTER_XT_MATCH_OSF=m +CONFIG_NETFILTER_XT_MATCH_OWNER=m +CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m +CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m +CONFIG_NETFILTER_XT_MATCH_POLICY=m +CONFIG_NETFILTER_XT_MATCH_QUOTA=m +CONFIG_NETFILTER_XT_MATCH_RATEEST=m +CONFIG_NETFILTER_XT_MATCH_REALM=m +CONFIG_NETFILTER_XT_MATCH_RECENT=m +CONFIG_NETFILTER_XT_MATCH_SCTP=m +CONFIG_NETFILTER_XT_MATCH_SOCKET=m +CONFIG_NETFILTER_XT_MATCH_STATE=m +CONFIG_NETFILTER_XT_MATCH_STATISTIC=m +CONFIG_NETFILTER_XT_MATCH_STRING=m +CONFIG_NETFILTER_XT_MATCH_TCPMSS=m +CONFIG_NETFILTER_XT_MATCH_TIME=m +CONFIG_NETFILTER_XT_MATCH_U32=m +CONFIG_NETFILTER_XT_NAT=m +CONFIG_NETFILTER_XT_SET=m +CONFIG_NETFILTER_XT_TARGET_AUDIT=m +CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m +CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m +CONFIG_NETFILTER_XT_TARGET_CONNMARK=m +CONFIG_NETFILTER_XT_TARGET_CT=m +CONFIG_NETFILTER_XT_TARGET_DSCP=m +CONFIG_NETFILTER_XT_TARGET_HL=m +CONFIG_NETFILTER_XT_TARGET_HMARK=m +CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m +CONFIG_NETFILTER_XT_TARGET_LED=m +CONFIG_NETFILTER_XT_TARGET_MARK=m +CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m +CONFIG_NETFILTER_XT_TARGET_NETMAP=m +CONFIG_NETFILTER_XT_TARGET_NFLOG=m +CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m +CONFIG_NETFILTER_XT_TARGET_NOTRACK=m +CONFIG_NETFILTER_XT_TARGET_RATEEST=m +CONFIG_NETFILTER_XT_TARGET_REDIRECT=m +CONFIG_NETFILTER_XT_TARGET_TCPMSS=m +CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m +CONFIG_NETFILTER_XT_TARGET_TEE=m +CONFIG_NETFILTER_XT_TARGET_TPROXY=m +CONFIG_NETFILTER_XT_TARGET_TRACE=m +CONFIG_NETLINK_DIAG=m +CONFIG_NET_ACT_BPF=m +CONFIG_NET_ACT_CONNMARK=m +CONFIG_NET_ACT_CSUM=m +CONFIG_NET_ACT_CT=m +CONFIG_NET_ACT_CTINFO=m +CONFIG_NET_ACT_MPLS=m +CONFIG_NET_ACT_NAT=m +CONFIG_NET_ACT_PEDIT=m +CONFIG_NET_ACT_POLICE=m +CONFIG_NET_ACT_SAMPLE=m +CONFIG_NET_ACT_SIMP=m +CONFIG_NET_ACT_SKBEDIT=m +CONFIG_NET_ACT_SKBMOD=m +CONFIG_NET_ACT_TUNNEL_KEY=m +CONFIG_NET_ACT_VLAN=m +CONFIG_NET_CLS_BPF=m +CONFIG_NET_CLS_FLOW=m +CONFIG_NET_CLS_FW=m +CONFIG_NET_CLS_MATCHALL=m +CONFIG_NET_CLS_ROUTE4=m +CONFIG_NET_CLS_U32=m +CONFIG_NET_EMATCH=y +CONFIG_NET_EMATCH_CMP=m +CONFIG_NET_EMATCH_IPSET=m +CONFIG_NET_EMATCH_IPT=m +CONFIG_NET_EMATCH_META=m +CONFIG_NET_EMATCH_NBYTE=m +CONFIG_NET_EMATCH_TEXT=m +CONFIG_NET_EMATCH_U32=m +CONFIG_NET_FOU=m +CONFIG_NET_FOU_IP_TUNNELS=y +CONFIG_NET_IPGRE=m +CONFIG_NET_IPGRE_BROADCAST=y +CONFIG_NET_IPGRE_DEMUX=m +CONFIG_NET_IPIP=m +CONFIG_NET_IPVTI=m +CONFIG_NET_KEY=m +CONFIG_NET_KEY_MIGRATE=y +CONFIG_NET_SCH_CAKE=m +CONFIG_NET_SCH_CHOKE=m +CONFIG_NET_SCH_CODEL=m +CONFIG_NET_SCH_DRR=m +CONFIG_NET_SCH_DUALPI2=m +CONFIG_NET_SCH_ETS=m +CONFIG_NET_SCH_FQ=m +CONFIG_NET_SCH_FQ_CODEL=m +CONFIG_NET_SCH_FQ_PIE=m +CONFIG_NET_SCH_GRED=m +CONFIG_NET_SCH_HFSC=m +CONFIG_NET_SCH_HHF=m +CONFIG_NET_SCH_HTB=m +CONFIG_NET_SCH_MULTIQ=m +CONFIG_NET_SCH_NETEM=m +CONFIG_NET_SCH_PIE=m +CONFIG_NET_SCH_PLUG=m +CONFIG_NET_SCH_PRIO=m +CONFIG_NET_SCH_QFQ=m +CONFIG_NET_SCH_RED=m +CONFIG_NET_SCH_SFB=m +CONFIG_NET_SCH_SFQ=m +CONFIG_NET_SCH_TBF=m +CONFIG_NET_SCH_TEQL=m +CONFIG_NET_TEAM=m +CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m +CONFIG_NET_TEAM_MODE_BROADCAST=m +CONFIG_NET_TEAM_MODE_LOADBALANCE=m +CONFIG_NET_TEAM_MODE_RANDOM=m +CONFIG_NET_TEAM_MODE_ROUNDROBIN=m +CONFIG_NFT_BRIDGE_META=m +CONFIG_NFT_BRIDGE_REJECT=m +CONFIG_NFT_COMPAT=m +CONFIG_NFT_CONNLIMIT=m +CONFIG_NFT_DUP_IPV4=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_DUP_NETDEV=m +CONFIG_NFT_FIB_NETDEV=m +CONFIG_NFT_FLOW_OFFLOAD=m +CONFIG_NFT_FWD_NETDEV=m +CONFIG_NFT_HASH=m +CONFIG_NFT_NUMGEN=m +CONFIG_NFT_OSF=m +CONFIG_NFT_QUEUE=m +CONFIG_NFT_QUOTA=m +CONFIG_NFT_REDIR=m +CONFIG_NFT_REJECT_NETDEV=m +CONFIG_NFT_SOCKET=m +CONFIG_NFT_TPROXY=m +CONFIG_NFT_TUNNEL=m +CONFIG_NFT_XFRM=m +CONFIG_NF_CONNTRACK_AMANDA=m +CONFIG_NF_CONNTRACK_BRIDGE=m +CONFIG_NF_CONNTRACK_FTP=m +CONFIG_NF_CONNTRACK_H323=m +CONFIG_NF_CONNTRACK_IRC=m +CONFIG_NF_CONNTRACK_LABELS=y +CONFIG_NF_CONNTRACK_MARK=y +CONFIG_NF_CONNTRACK_NETBIOS_NS=m +CONFIG_NF_CONNTRACK_PPTP=m +CONFIG_NF_CONNTRACK_PROCFS=y +CONFIG_NF_CONNTRACK_SANE=m +CONFIG_NF_CONNTRACK_SIP=m +CONFIG_NF_CONNTRACK_SNMP=m +CONFIG_NF_CONNTRACK_TFTP=m +CONFIG_NF_CONNTRACK_TIMEOUT=y +CONFIG_NF_CONNTRACK_TIMESTAMP=y +CONFIG_NF_CONNTRACK_ZONES=y +CONFIG_NF_CT_NETLINK=m +CONFIG_NF_CT_NETLINK_HELPER=m +CONFIG_NF_CT_NETLINK_TIMEOUT=m +CONFIG_NF_DUP_IPV4=m +CONFIG_NF_DUP_IPV6=m +CONFIG_NF_DUP_NETDEV=m +CONFIG_NF_FLOW_TABLE=m +CONFIG_NF_FLOW_TABLE_INET=m +CONFIG_NF_FLOW_TABLE_PROCFS=y +CONFIG_NF_LOG_ARP=m +CONFIG_NF_LOG_IPV4=m +CONFIG_NF_NAT_SNMP_BASIC=m +CONFIG_NF_SOCKET_IPV4=m +CONFIG_NF_SOCKET_IPV6=m +CONFIG_NF_TABLES_ARP=y +CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TPROXY_IPV4=m +CONFIG_NF_TPROXY_IPV6=m +CONFIG_OVPN=m +CONFIG_PACKET_DIAG=m +CONFIG_PPP=m +CONFIG_PPPOE=m +CONFIG_PPPOE_HASH_BITS_4=y +CONFIG_PPPOL2TP=m +CONFIG_PPP_ASYNC=m +CONFIG_PPP_BSDCOMP=m +CONFIG_PPP_DEFLATE=m +CONFIG_PPP_FILTER=y +CONFIG_PPP_MPPE=m +CONFIG_PPP_MULTILINK=y +CONFIG_PPP_SYNC_TTY=m +CONFIG_PPTP=m +CONFIG_PSAMPLE=m +CONFIG_TCP_CONG_ADVANCED=y +CONFIG_TCP_CONG_BBR=m +CONFIG_TCP_CONG_BIC=m +CONFIG_TCP_CONG_CDG=m +CONFIG_TCP_CONG_DCTCP=m +CONFIG_TCP_CONG_HSTCP=m +CONFIG_TCP_CONG_HTCP=m +CONFIG_TCP_CONG_HYBLA=m +CONFIG_TCP_CONG_ILLINOIS=m +CONFIG_TCP_CONG_LP=m +CONFIG_TCP_CONG_NV=m +CONFIG_TCP_CONG_SCALABLE=m +CONFIG_TCP_CONG_VEGAS=m +CONFIG_TCP_CONG_VENO=m +CONFIG_TCP_CONG_WESTWOOD=m +CONFIG_TCP_CONG_YEAH=m +CONFIG_TLS=m +CONFIG_UNIX_DIAG=m +CONFIG_VXLAN=m +CONFIG_WIREGUARD=m +CONFIG_XFRM_INTERFACE=m +CONFIG_XFRM_MIGRATE=y +CONFIG_XFRM_STATISTICS=y +CONFIG_XFRM_USER=m + +# Bluetooth: USB dongle drivers, vhci, MSFT/AOSP vendor extensions (USB controllers only) +CONFIG_BT_AOSPEXT=y +CONFIG_BT_ATH3K=m +CONFIG_BT_HCIBCM203X=m +CONFIG_BT_HCIBFUSB=m +CONFIG_BT_HCIBPA10X=m +CONFIG_BT_HCIBTUSB_AUTOSUSPEND=y +CONFIG_BT_HCIVHCI=m +CONFIG_BT_MSFTEXT=y + +# Containers / virtualisation: CRIU/checkpoint, userfaultfd, misc cgroup, KVM host vhost-net/vhost-vsock/vsock +CONFIG_CGROUP_MISC=y +CONFIG_CHECKPOINT_RESTORE=y +CONFIG_USERFAULTFD=y +CONFIG_VHOST_NET=m +CONFIG_VHOST_VSOCK=m +CONFIG_VSOCKETS=m +CONFIG_VSOCKETS_DIAG=m +CONFIG_VSOCKETS_LOOPBACK=m + +# Crypto: AF_ALG user API (cryptsetup/iwd), dm-crypt/fscrypt modes, VeraCrypt/TrueCrypt ciphers+hashes, IPsec AEADs +CONFIG_CRYPTO_ADIANTUM=m +CONFIG_CRYPTO_AEGIS128=m +CONFIG_CRYPTO_AEGIS128_SIMD=y +CONFIG_CRYPTO_BLAKE2B=m +CONFIG_CRYPTO_BLOWFISH=m +CONFIG_CRYPTO_CAMELLIA=m +CONFIG_CRYPTO_CAST5=m +CONFIG_CRYPTO_CAST6=m +CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CRC32=m +CONFIG_CRYPTO_CRC32C=m +CONFIG_CRYPTO_CRYPTD=m +CONFIG_CRYPTO_CTS=m +CONFIG_CRYPTO_ESSIV=m +CONFIG_CRYPTO_HCTR2=m +CONFIG_CRYPTO_LRW=m +CONFIG_CRYPTO_LZ4HC=m +CONFIG_CRYPTO_MD4=m +CONFIG_CRYPTO_NULL=m +CONFIG_CRYPTO_PCBC=m +CONFIG_CRYPTO_RMD160=m +CONFIG_CRYPTO_SEQIV=m +CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_STREEBOG=m +CONFIG_CRYPTO_TWOFISH=m +CONFIG_CRYPTO_USER_API_AEAD=m +CONFIG_CRYPTO_USER_API_HASH=m +CONFIG_CRYPTO_USER_API_SKCIPHER=m +CONFIG_CRYPTO_WP512=m +CONFIG_CRYPTO_XCBC=m +CONFIG_CRYPTO_XXHASH=m + +# Sound: USB audio/MIDI device drivers, MIDI 2.0, virtual MIDI, dummy card, userspace timers +CONFIG_SND_BCD2000=m +CONFIG_SND_DUMMY=m +CONFIG_SND_SEQ_UMP=y +CONFIG_SND_UMP_LEGACY_RAWMIDI=y +CONFIG_SND_USB_6FIRE=m +CONFIG_SND_USB_AUDIO_MIDI_V2=y +CONFIG_SND_USB_CAIAQ=m +CONFIG_SND_USB_CAIAQ_INPUT=y +CONFIG_SND_USB_HIFACE=m +CONFIG_SND_USB_POD=m +CONFIG_SND_USB_PODHD=m +CONFIG_SND_USB_TONEPORT=m +CONFIG_SND_USB_UA101=m +CONFIG_SND_USB_VARIAX=m +CONFIG_SND_UTIMER=y +CONFIG_SND_VIRMIDI=m + +# Video: non-UVC USB webcams (gspca family, Philips pwc), USB displays (DisplayLink USB2 udl, GUD, GM12U320) +CONFIG_DRM_GM12U320=m +CONFIG_DRM_GUD=m +CONFIG_DRM_UDL=m +CONFIG_USB_GL860=m +CONFIG_USB_GSPCA=m +CONFIG_USB_GSPCA_BENQ=m +CONFIG_USB_GSPCA_CONEX=m +CONFIG_USB_GSPCA_CPIA1=m +CONFIG_USB_GSPCA_DTCS033=m +CONFIG_USB_GSPCA_ETOMS=m +CONFIG_USB_GSPCA_FINEPIX=m +CONFIG_USB_GSPCA_JEILINJ=m +CONFIG_USB_GSPCA_JL2005BCD=m +CONFIG_USB_GSPCA_KINECT=m +CONFIG_USB_GSPCA_KONICA=m +CONFIG_USB_GSPCA_MARS=m +CONFIG_USB_GSPCA_MR97310A=m +CONFIG_USB_GSPCA_NW80X=m +CONFIG_USB_GSPCA_OV519=m +CONFIG_USB_GSPCA_OV534=m +CONFIG_USB_GSPCA_OV534_9=m +CONFIG_USB_GSPCA_PAC207=m +CONFIG_USB_GSPCA_PAC7302=m +CONFIG_USB_GSPCA_PAC7311=m +CONFIG_USB_GSPCA_SE401=m +CONFIG_USB_GSPCA_SN9C2028=m +CONFIG_USB_GSPCA_SN9C20X=m +CONFIG_USB_GSPCA_SONIXB=m +CONFIG_USB_GSPCA_SONIXJ=m +CONFIG_USB_GSPCA_SPCA1528=m +CONFIG_USB_GSPCA_SPCA500=m +CONFIG_USB_GSPCA_SPCA501=m +CONFIG_USB_GSPCA_SPCA505=m +CONFIG_USB_GSPCA_SPCA506=m +CONFIG_USB_GSPCA_SPCA508=m +CONFIG_USB_GSPCA_SPCA561=m +CONFIG_USB_GSPCA_SQ905=m +CONFIG_USB_GSPCA_SQ905C=m +CONFIG_USB_GSPCA_SQ930X=m +CONFIG_USB_GSPCA_STK014=m +CONFIG_USB_GSPCA_STK1135=m +CONFIG_USB_GSPCA_STV0680=m +CONFIG_USB_GSPCA_SUNPLUS=m +CONFIG_USB_GSPCA_T613=m +CONFIG_USB_GSPCA_TOPRO=m +CONFIG_USB_GSPCA_TOUPTEK=m +CONFIG_USB_GSPCA_TV8532=m +CONFIG_USB_GSPCA_VC032X=m +CONFIG_USB_GSPCA_VICAM=m +CONFIG_USB_GSPCA_XIRLINK_CIT=m +CONFIG_USB_GSPCA_ZC3XX=m +CONFIG_USB_M5602=m +CONFIG_USB_PWC=m +CONFIG_USB_PWC_INPUT_EVDEV=y +CONFIG_USB_STV06XX=m + +# Other user-facing: Kerberos persistent keyrings, Speakup screen reader, Wine/Proton ntsync +CONFIG_ACCESSIBILITY=y +CONFIG_NTSYNC=m +CONFIG_PERSISTENT_KEYRINGS=y +CONFIG_SPEAKUP=m +CONFIG_SPEAKUP_SYNTH_ACNTSA=m +CONFIG_SPEAKUP_SYNTH_APOLLO=m +CONFIG_SPEAKUP_SYNTH_AUDPTR=m +CONFIG_SPEAKUP_SYNTH_BNS=m +CONFIG_SPEAKUP_SYNTH_DECTLK=m +CONFIG_SPEAKUP_SYNTH_LTLK=m +CONFIG_SPEAKUP_SYNTH_SOFT=m +CONFIG_SPEAKUP_SYNTH_SPKOUT=m +CONFIG_SPEAKUP_SYNTH_TXPRT=m diff --git a/arch/arm64/configs/j700-radio.config b/arch/arm64/configs/j700-radio.config new file mode 100644 index 00000000000000..f5c5d3c3ec0bfe --- /dev/null +++ b/arch/arm64/configs/j700-radio.config @@ -0,0 +1,20 @@ +# J700 retained radio bootstrap; merge after defconfig and asahi.config. +# System sleep is retained; active radio firmware cannot cross kexec. +# Firmware/calibration and runtime admission follow the MT7932 documentation. +# CONFIG_KEXEC is not set +# CONFIG_KEXEC_FILE is not set +# CONFIG_KEXEC_HANDOVER is not set +CONFIG_PCI=y +CONFIG_PCIE_APPLE=y +CONFIG_APPLE_DART=y +CONFIG_PCIE_APPLE_PIODMA_DIAG=y +CONFIG_CFG80211=m +CONFIG_WLAN=y +CONFIG_WLAN_VENDOR_MEDIATEK=y +CONFIG_MT7932_FULLMAC=m +CONFIG_BT=m +CONFIG_BT_BREDR=y +CONFIG_BT_LE=y +CONFIG_BT_MTK7932_PCIE=m +CONFIG_CRYPTO_AES=y +CONFIG_CRYPTO_CMAC=y diff --git a/arch/arm64/configs/j700.config b/arch/arm64/configs/j700.config new file mode 100644 index 00000000000000..976fd65b7161cf --- /dev/null +++ b/arch/arm64/configs/j700.config @@ -0,0 +1,113 @@ +# J700 (MacBook Neo, T8140) base fragment. +# +# Merge after defconfig, asahi.config and j700-radio.config, then add the device-parity fragment: +# make defconfig +# scripts/kconfig/merge_config.sh -m .config arch/arm64/configs/asahi.config arch/arm64/configs/j700-radio.config +# make olddefconfig +# scripts/kconfig/merge_config.sh -m .config arch/arm64/configs/j700.config arch/arm64/configs/j700-device-parity.config +# make olddefconfig +# The result is the configuration the J700 kernel is tested with. + +# Boot NOR (nvram) access for boot-volume switching, and the PMU RTC wake alarm +CONFIG_SPI_APPLE_QSPI=m +CONFIG_RTC_DRV_APPLE_PMU=m + +# Classic Bluetooth profiles (headsets, tethering) +CONFIG_BT_RFCOMM=m +CONFIG_BT_RFCOMM_TTY=y +CONFIG_BT_BNEP=m +CONFIG_BT_BNEP_MC_FILTER=y +CONFIG_BT_BNEP_PROTO_FILTER=y + +# Distribution firmware files are compressed +CONFIG_FW_LOADER_COMPRESS=y +CONFIG_FW_LOADER_COMPRESS_ZSTD=y + +# 48-bit virtual addresses; KASLR is untested on this machine +CONFIG_ARM64_VA_BITS_48=y +# CONFIG_RANDOMIZE_BASE is not set + +# Boot path and platform built in, so storage, keyboard, SMC and USB do not depend on the initramfs +CONFIG_BLK_DEV_NVME=y +CONFIG_NVME_APPLE=y +CONFIG_APPLE_SART=y +CONFIG_APPLE_RTKIT_HELPER=y +CONFIG_BTRFS_FS=y +CONFIG_APPLE_DOCKCHANNEL=y +CONFIG_HID_DOCKCHANNEL=y +CONFIG_HID_APPLE=y +CONFIG_HID_MAGICMOUSE=y +CONFIG_SPMI_APPLE=y +CONFIG_NVMEM_APPLE_SPMI=y +CONFIG_MFD_MACSMC=y +CONFIG_GPIO_MACSMC=y +CONFIG_POWER_RESET_MACSMC=y +CONFIG_SENSORS_MACSMC_HWMON=y +CONFIG_I2C_APPLE=y +CONFIG_PINCTRL_APPLE_GPIO=y +CONFIG_APPLE_WATCHDOG=y +CONFIG_ARM_APPLE_SOC_CPUFREQ=y +CONFIG_PHY_APPLE_ATC=y +CONFIG_USB_DWC3_APPLE=y +CONFIG_TYPEC=y +CONFIG_TYPEC_SN201202X=y +CONFIG_MUX_APPLE_DPXBAR=y +CONFIG_BACKLIGHT_CLASS_DEVICE=y +CONFIG_SYSFB_SIMPLEFB=y +CONFIG_ZRAM=y +CONFIG_UHID=y +CONFIG_RFKILL=y +CONFIG_THERMAL_NETLINK=y +CONFIG_NVME_HWMON=y + +# J700 driver choices +CONFIG_APPLE_CLPC=m +CONFIG_APPLE_PMP=y +CONFIG_DRM_ACCEL=y +CONFIG_DRM_ACCEL_ANE=m +# CONFIG_HID_APPLE_MTP_HAPTIC is not set +CONFIG_IIO=m +# CONFIG_IIO_AOP_SENSOR_LAS is not set +# CONFIG_USB4 is not set + +# Per-process 4K user pages on the 16K kernel (x86 emulation) +CONFIG_ARM64_USER4K_EXPERIMENTAL=y + +# USB Ethernet built in, for a wired link before modules load +CONFIG_USB_USBNET=y +CONFIG_USB_NET_AX8817X=y +CONFIG_USB_NET_AX88179_178A=y +CONFIG_USB_NET_CDC_NCM=y +CONFIG_USB_RTL8152=y +CONFIG_USB_UAS=m + +# Crypto that Bluetooth and Wi-Fi need early +CONFIG_CRYPTO_CCM=y +CONFIG_CRYPTO_ECDH=y +CONFIG_CRYPTO_GCM=y +CONFIG_CRYPTO_HMAC=y +CONFIG_CRYPTO_SHA256=y + +# Utilisation clamping +CONFIG_UCLAMP_TASK=y +CONFIG_UCLAMP_TASK_GROUP=y + +# LSMs named in CONFIG_LSM +CONFIG_SECURITY_LANDLOCK=y +CONFIG_SECURITY_YAMA=y + +# dma-buf heaps and udmabuf (camera and GPU buffer sharing) +CONFIG_DMABUF_HEAPS=y +CONFIG_DMABUF_HEAPS_SYSTEM=y +CONFIG_UDMABUF=y + +# Debugging aids kept in the tested configuration +CONFIG_DYNAMIC_DEBUG=y +CONFIG_FTRACE=y +CONFIG_ENABLE_DEFAULT_TRACERS=y +CONFIG_PM_DEBUG=y +CONFIG_PM_ADVANCED_DEBUG=y + +# Match the tested kernel image: no relative relocation packing, no SCM version suffix +# CONFIG_RELR is not set +# CONFIG_LOCALVERSION_AUTO is not set diff --git a/arch/arm64/configs/j813-usb4.config b/arch/arm64/configs/j813-usb4.config new file mode 100644 index 00000000000000..d754400c6eccd5 --- /dev/null +++ b/arch/arm64/configs/j813-usb4.config @@ -0,0 +1,7 @@ +# Additional build coverage for M5 USB4 controller development. +# Merge with j813.config. This does not enable USB4 device-tree nodes. +CONFIG_PCI=y +CONFIG_PCIE_APPLE=y +CONFIG_USB4=y +CONFIG_USB4_APPLE_SOC=y +CONFIG_RESET_APPLE_CIO=y diff --git a/arch/arm64/configs/j813.config b/arch/arm64/configs/j813.config index d8143807c2e3ee..9091e74f3e9187 100644 --- a/arch/arm64/configs/j813.config +++ b/arch/arm64/configs/j813.config @@ -6,6 +6,11 @@ CONFIG_ARM64_16K_PAGES=y CONFIG_ARM64_VA_BITS_47=y CONFIG_SMP=y CONFIG_NR_CPUS=16 +CONFIG_CPU_FREQ=y +CONFIG_CPU_FREQ_STAT=y +CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y +CONFIG_CPU_FREQ_GOV_USERSPACE=y +CONFIG_ARM_APPLE_SOC_CPUFREQ=y CONFIG_ARM64_VHE=y CONFIG_ARM64_SVE=y CONFIG_ARM64_PAN=y @@ -40,7 +45,63 @@ CONFIG_FB=y CONFIG_FB_DEVICE=y CONFIG_FB_SIMPLE=y CONFIG_FRAMEBUFFER_CONSOLE=y +CONFIG_DRM=y +CONFIG_DRM_APPLE=y +CONFIG_BACKLIGHT_CLASS_DEVICE=y +CONFIG_DRM_FBDEV_EMULATION=y # CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER is not set CONFIG_LOG_BUF_SHIFT=20 CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y +CONFIG_PM=y +CONFIG_MAILBOX=y +CONFIG_APPLE_MAILBOX=y +CONFIG_APPLE_RTKIT=y +CONFIG_APPLE_RTKIT_HELPER=y +CONFIG_APPLE_DOCKCHANNEL=y +CONFIG_IOMMU_SUPPORT=y +CONFIG_APPLE_DART=y +CONFIG_MFD_MACSMC=y +CONFIG_HWMON=y +CONFIG_SENSORS_MACSMC_HWMON=y +CONFIG_INPUT=y +CONFIG_INPUT_EVDEV=y +CONFIG_HID_SUPPORT=y +CONFIG_HID=y +CONFIG_HID_GENERIC=y +CONFIG_HID_DOCKCHANNEL=y +CONFIG_HID_APPLE=y +CONFIG_HID_MAGICMOUSE=y +CONFIG_GPIOLIB=y +CONFIG_NEW_LEDS=y +CONFIG_LEDS_CLASS=y +CONFIG_LEDS_PWM=y +CONFIG_PWM=y +CONFIG_PWM_APPLE=y +CONFIG_FW_LOADER=y +CONFIG_PM_GENERIC_DOMAINS=y +CONFIG_APPLE_PMGR_PWRSTATE=y +CONFIG_SPMI=y +CONFIG_SPMI_APPLE=y +CONFIG_USB_SUPPORT=y +CONFIG_USB=y +CONFIG_USB_XHCI_HCD=y +CONFIG_USB_XHCI_PLATFORM=y +CONFIG_USB_DWC3=y +CONFIG_USB_DWC3_HOST=y +CONFIG_USB_DWC3_APPLE=y +CONFIG_USB_HID=y +CONFIG_BLOCK=y +CONFIG_SCSI=y +CONFIG_BLK_DEV_SD=y +CONFIG_USB_STORAGE=y +CONFIG_USB_UAS=y +CONFIG_PARTITION_ADVANCED=y +CONFIG_EFI_PARTITION=y +CONFIG_MSDOS_PARTITION=y +CONFIG_EXFAT_FS=y +CONFIG_NLS_UTF8=y +CONFIG_TYPEC=y +CONFIG_TYPEC_SN201202X=y +CONFIG_PHY_APPLE_ATC=y +CONFIG_DEBUG_FS=y diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index d0d3cdd5763cab..fff3a9570ad659 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -19,6 +19,11 @@ cpucap_is_possible(const unsigned int cap) "cap must be < ARM64_NCAPS"); switch (cap) { + case ARM64_HAS_USER16K_GRANULE: + return IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) && + IS_ENABLED(CONFIG_ARM64_64K_PAGES); + case ARM64_HAS_USER4K_GRANULE: + return IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL); case ARM64_HAS_EPAN: return IS_ENABLED(CONFIG_ARM64_EPAN); case ARM64_SVE: diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 632f980ec55e56..809e39962fe231 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -155,6 +155,8 @@ #define APPLE_CPU_PART_M2_AVALANCHE_MAX 0x039 #define APPLE_CPU_PART_TAHITI_E 0x060 #define APPLE_CPU_PART_TAHITI_P 0x061 +#define APPLE_CPU_PART_T8142_E 0x062 +#define APPLE_CPU_PART_T8142_P 0x063 #define APPLE_CPU_PART_T8152_E 0x072 #define APPLE_CPU_PART_T8152_P 0x073 @@ -252,6 +254,8 @@ #define MIDR_APPLE_M2_AVALANCHE_MAX MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_AVALANCHE_MAX) #define MIDR_APPLE_TAHITI_E MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_TAHITI_E) #define MIDR_APPLE_TAHITI_P MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_TAHITI_P) +#define MIDR_APPLE_T8142_E MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_T8142_E) +#define MIDR_APPLE_T8142_P MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_T8142_P) #define MIDR_APPLE_T8152_E MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_T8152_E) #define MIDR_APPLE_T8152_P MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_T8152_P) #define MIDR_AMPERE1 MIDR_CPU_MODEL(ARM_CPU_IMP_AMPERE, AMPERE_CPU_PART_AMPERE1) diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h index d2779d604c7be2..23320b49c32082 100644 --- a/arch/arm64/include/asm/elf.h +++ b/arch/arm64/include/asm/elf.h @@ -121,6 +121,9 @@ #define CORE_DUMP_USE_REGSET #define ELF_EXEC_PAGESIZE PAGE_SIZE +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#define ELF_EXEC_PAGESIZE_MM(mm) mm_page_size(mm) +#endif /* * This is the base location for PIE (ET_DYN with INTERP) loads. On @@ -181,6 +184,12 @@ do { \ NEW_AUX_ENT(AT_IGNORE, 0); \ } while (0) +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +struct linux_binprm; +int arm64_bprm_prepare_mm(struct linux_binprm *bprm, bool compat); +#define arch_bprm_prepare_mm arm64_bprm_prepare_mm +#endif + #define ARCH_HAS_SETUP_ADDITIONAL_PAGES struct linux_binprm; extern int arch_setup_additional_pages(struct linux_binprm *bprm, diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 229ee7976f6934..2bd73e773095c1 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -27,7 +27,7 @@ #endif #define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) +#define SWAPPER_PGTABLE_LEVELS (ARM64_NATIVE_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) #define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 5e1211c540ab23..da82ed5f0c41ff 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -8,6 +8,10 @@ #include #define MMCF_AARCH32 0x1 /* mm context flag for AArch32 executables */ +#define MMCF_USER_4K 0x2 /* experimental, immutable mm table geometry */ +#define MMCF_USER_16K 0x4 /* experimental, immutable mm table geometry */ +#define MMCF_USER_64K 0x8 /* internal geometry; larger user ABI not enabled */ +#define MMCF_USER_GRANULE (MMCF_USER_4K | MMCF_USER_16K | MMCF_USER_64K) #define USER_ASID_BIT 48 #define USER_ASID_FLAG (UL(1) << USER_ASID_BIT) @@ -16,6 +20,15 @@ #include #include +/* AArch32 falls back to the native ABI when 4K translation is unavailable. */ +static inline unsigned int arm64_exec_page_shift(bool compat, bool has_4k, + unsigned int native_shift, unsigned int requested_shift) +{ + if (compat) + return has_4k ? 12 : native_shift; + return requested_shift ?: native_shift; +} + typedef struct { atomic64_t id; #ifdef CONFIG_COMPAT @@ -25,8 +38,35 @@ typedef struct { void *vdso; unsigned long flags; u8 pkey_allocation_map; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + /* PTE/PMD/PUD fragment pools; initialized before atomic publication. */ + void *user4k_pt_frag[3]; +#endif } mm_context_t; +static inline bool arm64_context_user4k(const mm_context_t *context) +{ + return IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) && + (context->flags & MMCF_USER_4K); +} + +static inline unsigned int arm64_context_page_shift(const mm_context_t *context) +{ + if (IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL)) { + if (context->flags & MMCF_USER_4K) + return 12; + if (context->flags & MMCF_USER_16K) + return 14; + if (context->flags & MMCF_USER_64K) + return 16; + } + return PAGE_SHIFT; +} + +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#define arch_mm_page_shift(mm) arm64_context_page_shift(&(mm)->context) +#endif + /* * We use atomic64_read() here because the ASID for an 'mm_struct' can * be reallocated when scheduling one of its threads following a diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 803b6875815275..db27ff606555be 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -26,6 +26,17 @@ #include #include +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#define arch_mm_init_exec arch_mm_init_exec +static inline void arch_mm_init_exec(struct mm_struct *mm, unsigned int page_shift) +{ + if (page_shift == 12) + mm->context.flags |= MMCF_USER_4K; + else if (page_shift == 14 && PAGE_SHIFT > 14) + mm->context.flags |= MMCF_USER_16K; +} +#endif + extern bool rodata_full; static inline void contextidr_thread_switch(struct task_struct *next) @@ -53,6 +64,29 @@ static inline void cpu_set_reserved_ttbr0(void) isb(); } +/* + * The caller has installed reserved_pg_dir in TTBR0 and excludes preemption. + * On a granule transition, complete that installation before changing TG0, + * then synchronise the new walk geometry before installing a user root. + * ASIDs remain unique across both granules; switching does not flush them. + */ +static inline void cpu_set_mm_tg0(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + unsigned long tcr = read_sysreg(tcr_el1); + unsigned int shift = mm_page_shift(mm); + unsigned long tg0 = shift == 12 ? TCR_TG0_4K : + (shift == 14 ? TCR_TG0_16K : TCR_TG0_64K); + + if ((tcr & TCR_EL1_TG0_MASK) == tg0) + return; + + isb(); + write_sysreg((tcr & ~TCR_EL1_TG0_MASK) | tg0, tcr_el1); + isb(); +#endif +} + void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) @@ -62,16 +96,21 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) } /* - * Ensure TCR.T0SZ is set to the provided value. + * Set TCR.T0SZ and restore native TG0 for the idmap/transition tables. */ static inline void __cpu_set_tcr_t0sz(unsigned long t0sz) { unsigned long tcr = read_sysreg(tcr_el1); + unsigned long mask = TCR_EL1_T0SZ_MASK; - if ((tcr & TCR_EL1_T0SZ_MASK) == t0sz) +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + mask |= TCR_EL1_TG0_MASK; + t0sz |= IS_ENABLED(CONFIG_ARM64_64K_PAGES) ? TCR_TG0_64K : TCR_TG0_16K; +#endif + if ((tcr & mask) == t0sz) return; - tcr &= ~TCR_EL1_T0SZ_MASK; + tcr &= ~mask; tcr |= t0sz; write_sysreg(tcr, tcr_el1); isb(); @@ -97,6 +136,8 @@ static inline void cpu_uninstall_idmap(void) local_flush_tlb_all(); __cpu_set_tcr_t0sz(TCR_T0SZ(vabits_actual)); + /* SW PAN will restore TTBR0 later, but already needs the mm's TG0. */ + cpu_set_mm_tg0(mm); if (mm != &init_mm && !system_uses_ttbr0_pan()) cpu_switch_mm(mm->pgd, mm); } @@ -168,6 +209,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) { atomic64_set(&mm->context.id, 0); refcount_set(&mm->context.pinned, 0); +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + memset(mm->context.user4k_pt_frag, 0, sizeof(mm->context.user4k_pt_frag)); +#endif /* pkey 0 is the default, so always reserve it. */ mm->context.pkey_allocation_map = BIT(0); @@ -175,6 +219,16 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) return 0; } +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +void arm64_user4k_pt_cache_destroy(struct mm_struct *mm); + +static inline void destroy_context(struct mm_struct *mm) +{ + arm64_user4k_pt_cache_destroy(mm); +} +#define destroy_context destroy_context +#endif + static inline void arch_dup_pkeys(struct mm_struct *oldmm, struct mm_struct *mm) { @@ -241,6 +295,7 @@ static inline void __switch_mm(struct mm_struct *next) */ if (next == &init_mm) { cpu_set_reserved_ttbr0(); + cpu_set_mm_tg0(next); return; } diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 7f7b97e099968e..c454f5ea8c3d15 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -25,6 +25,8 @@ unsigned long mte_copy_tags_to_user(void __user *to, void *from, unsigned long n); int mte_save_tags(struct page *page); void mte_save_page_tags(const void *page_addr, void *tag_storage); +/* Address and size must be aligned to 256 bytes; size may be zero. */ +void mte_save_tags_range(const void *addr, void *tag_storage, unsigned long size); void mte_restore_tags(swp_entry_t entry, struct page *page); void mte_restore_page_tags(void *page_addr, const void *tag_storage); void mte_invalidate_tags(int type, pgoff_t offset); @@ -98,6 +100,8 @@ static inline bool try_page_mte_tagging(struct page *page) void mte_zero_clear_page_tags(void *addr); void mte_sync_tags(pte_t pte, unsigned int nr_pages); void mte_copy_page_tags(void *kto, const void *kfrom); +void mte_copy_subpage_tags_range(void *kto, const void *kfrom, unsigned long size); +void mte_clear_subpage_tags_range(void *addr, unsigned long size); void mte_thread_init_user(void); void mte_thread_switch(struct task_struct *next); void mte_cpu_setup(void); @@ -134,6 +138,12 @@ static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages) static inline void mte_copy_page_tags(void *kto, const void *kfrom) { } +static inline void mte_copy_subpage_tags_range(void *kto, const void *kfrom, unsigned long size) +{ +} +static inline void mte_clear_subpage_tags_range(void *addr, unsigned long size) +{ +} static inline void mte_thread_init_user(void) { } diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 58200de8a221a6..96fa9d8ea23c57 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -10,6 +10,11 @@ #include +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +/* A shared zero mapping must cover the largest supported user granule. */ +#define ARCH_ZERO_PAGE_SIZE (1UL << 16) +#endif + #ifndef __ASSEMBLER__ #include /* for READ_IMPLIES_EXEC */ @@ -26,11 +31,28 @@ void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma); #define __HAVE_ARCH_COPY_USER_HIGHPAGE +#ifdef CONFIG_MM_SUBPAGE +void clear_user_subpage_range(struct page *page, unsigned int offset, + unsigned int size); +int copy_user_subpage_range(struct page *to, unsigned int to_offset, + struct page *from, unsigned int from_offset, + unsigned int size); +void clear_user_subpage(struct page *page, unsigned int offset); +int copy_user_subpage(struct page *to, unsigned int to_offset, + struct page *from, unsigned int from_offset); +#endif + void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr); +struct folio *vma_alloc_zeroed_movable_folio_order(struct vm_area_struct *vma, + unsigned long vaddr, unsigned int order); +#define vma_alloc_zeroed_movable_folio_order vma_alloc_zeroed_movable_folio_order +static inline struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, + unsigned long vaddr) +{ + return vma_alloc_zeroed_movable_folio_order(vma, vaddr, 0); +} #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio bool tag_clear_highpages(struct page *to, int numpages, bool clear_pages); @@ -38,7 +60,14 @@ bool tag_clear_highpages(struct page *to, int numpages, bool clear_pages); #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +/* A table may occupy a 4K fragment of a native 16K physical page. */ +typedef pte_t *pgtable_t; +#define pgtable_to_page(table) virt_to_page(table) +#define pgtable_from_page(page) ((page) ? (pgtable_t)page_address(page) : NULL) +#else typedef struct page *pgtable_t; +#endif int pfn_is_map_memory(unsigned long pfn); diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 1b4509d3382c63..41b81203e575ca 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,7 +15,15 @@ #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#define __HAVE_ARCH_PTE_ALLOC_ONE +#define __HAVE_ARCH_PTE_FREE +#define __HAVE_ARCH_PMD_ALLOC_ONE +#define __HAVE_ARCH_PMD_FREE +#define __HAVE_ARCH_PUD_ALLOC_ONE +#endif #include +#include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -31,6 +39,12 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) pudval_t pudval = PUD_TYPE_TABLE | PUD_TABLE_AF; pudval |= (mm == &init_mm) ? PUD_TABLE_UXN : PUD_TABLE_PXN; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (arm64_mm_alt_granule(mm)) { + set_pud(pudp, __pud(__pa(pmdp) | pudval)); + return; + } +#endif __pud_populate(pudp, __pa(pmdp), pudval); } #else @@ -53,14 +67,24 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_AF; p4dval |= (mm == &init_mm) ? P4D_TABLE_UXN : P4D_TABLE_PXN; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (arm64_mm_alt_granule(mm)) { + set_p4d(p4dp, __p4d(__pa(pudp) | p4dval)); + return; + } +#endif __p4d_populate(p4dp, __pa(pudp), p4dval); } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!pgtable_l4_enabled()) + if (mm_pud_folded(mm)) return; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + arm64_pgtable_free(pud); +#else __pud_free(mm, pud); +#endif } #else static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) @@ -117,7 +141,16 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) { VM_BUG_ON(mm == &init_mm); +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (arm64_mm_alt_granule(mm)) { + set_pmd(pmdp, __pmd(__pa(ptep) | PMD_TYPE_TABLE | + PMD_TABLE_AF | PMD_TABLE_PXN)); + return; + } + __pmd_populate(pmdp, __pa(ptep), +#else __pmd_populate(pmdp, page_to_phys(ptep), +#endif PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_PXN); } diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 72f31800c7030c..cb211cbcdb7fdf 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -28,8 +28,10 @@ * * which gets simplified as : */ +#define ARM64_HW_PGTABLE_LEVELS_FOR(va_bits, page_shift) \ + (((va_bits) - PTDESC_ORDER - 1) / ((page_shift) - PTDESC_ORDER)) #define ARM64_HW_PGTABLE_LEVELS(va_bits) \ - (((va_bits) - PTDESC_ORDER - 1) / PTDESC_TABLE_SHIFT) + ARM64_HW_PGTABLE_LEVELS_FOR(va_bits, PAGE_SHIFT) /* * Size mapped by an entry at level n ( -1 <= n <= 3) @@ -44,7 +46,10 @@ * Rearranging it a bit we get : * (4 - n) * PTDESC_TABLE_SHIFT + PTDESC_ORDER */ -#define ARM64_HW_PGTABLE_LEVEL_SHIFT(n) (PTDESC_TABLE_SHIFT * (4 - (n)) + PTDESC_ORDER) +#define ARM64_HW_PGTABLE_LEVEL_SHIFT_FOR(n, page_shift) \ + (((page_shift) - PTDESC_ORDER) * (4 - (n)) + PTDESC_ORDER) +#define ARM64_HW_PGTABLE_LEVEL_SHIFT(n) \ + ARM64_HW_PGTABLE_LEVEL_SHIFT_FOR(n, PAGE_SHIFT) #define PTRS_PER_PTE (1 << PTDESC_TABLE_SHIFT) @@ -79,7 +84,8 @@ * PGDIR_SHIFT determines the size a top-level page table entry can map * (depending on the configuration, this level can be -1, 0, 1 or 2). */ -#define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) +#define ARM64_NATIVE_PGTABLE_LEVELS ARM64_HW_PGTABLE_LEVELS(VA_BITS) +#define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - ARM64_NATIVE_PGTABLE_LEVELS) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE-1)) #define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT)) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 4dfa42b7d05356..1ce97b836c8ac5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -467,6 +467,8 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } +#include + /* * Hugetlb definitions. */ @@ -711,6 +713,23 @@ static inline void __set_ptes_anysz(struct mm_struct *mm, unsigned long addr, static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { + if (arm64_mm_alt_granule(mm)) { + unsigned int native_pages = DIV_ROUND_UP((pte_phys_mm(mm, pte) & ~PAGE_MASK) + + ((unsigned long)nr << mm_page_shift(mm)), PAGE_SIZE); + + page_table_check_ptes_set(mm, addr, ptep, pte, nr); + __sync_cache_and_tags(pte, native_pages); + for (;;) { + __check_safe_pte_update(mm, ptep, pte); + __set_pte_nosync(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte = pte_advance_pfn_mm(mm, pte, 1); + } + __set_pte_complete(pte); + return; + } __set_ptes_anysz(mm, addr, ptep, pte, nr, PAGE_SIZE); } @@ -946,6 +965,8 @@ static inline pmd_t *pud_pgtable(pud_t pud) static __always_inline bool pgtable_l4_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES)) + return false; if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) return true; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) @@ -955,6 +976,8 @@ static __always_inline bool pgtable_l4_enabled(void) static inline bool mm_pud_folded(const struct mm_struct *mm) { + if (arm64_mm_alt_granule(mm)) + return mm_page_shift(mm) == 16; return !pgtable_l4_enabled(); } #define mm_pud_folded mm_pud_folded @@ -994,6 +1017,8 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) { + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES)) + return (pud_t *)p4dp; /* Ensure that 'p4dp' indexes a page table according to 'addr' */ VM_BUG_ON(((addr >> P4D_SHIFT) ^ ((u64)p4dp >> 3)) % PTRS_PER_P4D); @@ -1216,6 +1241,8 @@ p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr) #endif /* CONFIG_PGTABLE_LEVELS > 4 */ +#include + #define pgd_ERROR(e) \ pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e)) @@ -1261,7 +1288,7 @@ static inline int __ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) { return __ptep_set_access_flags_anysz(vma, address, ptep, entry, dirty, - PAGE_SIZE); + mm_page_size(vma->vm_mm)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1389,7 +1416,7 @@ static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - addr += PAGE_SIZE; + addr += mm_page_size(mm); } } @@ -1402,7 +1429,7 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, pte = __ptep_get_and_clear(mm, addr, ptep); while (--nr) { ptep++; - addr += PAGE_SIZE; + addr += mm_page_size(mm); tmp_pte = __ptep_get_and_clear(mm, addr, ptep); if (pte_dirty(tmp_pte)) pte = pte_mkdirty(pte); @@ -1450,7 +1477,7 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, { unsigned int i; - for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + for (i = 0; i < nr; i++, address += mm_page_size(mm), ptep++) __ptep_set_wrprotect(mm, address, ptep); } @@ -1490,7 +1517,7 @@ static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma, if (--nr == 0) break; ptep++; - addr += PAGE_SIZE; + addr += mm_page_size(vma->vm_mm); } } @@ -1516,9 +1543,11 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, * bits 0-1: present (must be zero) * bits 2: remember PG_anon_exclusive * bit 3: remember uffd-wp state + * bits 4-5: 4K slot offset within a native swap page (MM_SUBPAGE) * bits 6-10: swap type * bit 11: PTE_PRESENT_INVALID (must be zero) * bits 12-61: swap offset + * bits 62-63: high 4K slot offset bits (MM_SUBPAGE, up to 64K) */ #define __SWP_TYPE_SHIFT 6 #define __SWP_TYPE_BITS 5 @@ -1688,11 +1717,11 @@ static __always_inline void contpte_try_fold(struct mm_struct *mm, * We can't fold special mappings, because there is no associated folio. */ - const unsigned long contmask = CONT_PTES - 1; - bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + const unsigned long contmask = cont_ptes_mm(mm) - 1; + bool valign = ((addr >> mm_page_shift(mm)) & contmask) == contmask; if (unlikely(valign)) { - bool palign = (pte_pfn(pte) & contmask) == contmask; + bool palign = ((pte_phys_mm(mm, pte) >> mm_page_shift(mm)) & contmask) == contmask; if (unlikely(palign && pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) @@ -1713,7 +1742,11 @@ static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) if (!pte_valid_cont(pte)) return 1; - return CONT_PTES - (((unsigned long)ptep >> 3) & (CONT_PTES - 1)); + { + unsigned int nr = arm64_cont_ptes(arm64_ptep_page_shift(ptep)); + + return nr - (((unsigned long)ptep >> 3) & (nr - 1)); + } } /* diff --git a/arch/arm64/include/asm/shmparam.h b/arch/arm64/include/asm/shmparam.h index f920e22ec677cd..75120d3a71889b 100644 --- a/arch/arm64/include/asm/shmparam.h +++ b/arch/arm64/include/asm/shmparam.h @@ -12,6 +12,12 @@ */ #define COMPAT_SHMLBA (4 * PAGE_SIZE) +/* SHMLBA also aligns kernel allocations; scale only SysV attachments. */ +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#define arch_shm_attach_align(mm, align) \ + (((align) >> PAGE_SHIFT) << mm_page_shift(mm)) +#endif + #include #endif /* __ASM_SHMPARAM_H */ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 751bd57bc3bac5..21453f3125e0f9 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,6 +10,11 @@ #include +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +void __tlb_remove_table(void *token); +/* Fragment tokens do not carry a private descriptor callback head. */ +#define tlb_table_is_ptdesc(token) (!((unsigned long)(token) & 1UL)) +#endif #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); @@ -76,8 +81,14 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); + struct ptdesc *ptdesc = page_ptdesc(pgtable_to_page(pte)); +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (test_bit(PT_fragmented, &ptdesc->pt_flags.f)) { + tlb_remove_table(tlb, (void *)((unsigned long)pte | 1)); + return; + } +#endif tlb_remove_ptdesc(tlb, ptdesc); } @@ -87,6 +98,12 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (test_bit(PT_fragmented, &ptdesc->pt_flags.f)) { + tlb_remove_table(tlb, (void *)((unsigned long)pmdp | 1)); + return; + } +#endif tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -97,9 +114,15 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, { struct ptdesc *ptdesc = virt_to_ptdesc(pudp); - if (!pgtable_l4_enabled()) + if (mm_pud_folded(tlb->mm)) return; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (test_bit(PT_fragmented, &ptdesc->pt_flags.f)) { + tlb_remove_table(tlb, (void *)((unsigned long)pudp | 1)); + return; + } +#endif tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 57b4eda6a72bba..56062bf906cd41 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -56,8 +57,8 @@ }) /* - * Get translation granule of the system, which is decided by - * PAGE_SIZE. Used by TTL. + * Encode a translation granule for TTL and range operands. Native kernel + * and stage-2 callers use PAGE_SHIFT; userspace callers use their mm. * - 4KB : 1 * - 16KB : 2 * - 64KB : 3 @@ -66,20 +67,25 @@ #define TLBI_TTL_TG_16K 2 #define TLBI_TTL_TG_64K 3 -static inline unsigned long get_trans_granule(void) +static inline unsigned long get_trans_granule_shift(unsigned int page_shift) { - switch (PAGE_SIZE) { - case SZ_4K: + switch (page_shift) { + case 12: return TLBI_TTL_TG_4K; - case SZ_16K: + case 14: return TLBI_TTL_TG_16K; - case SZ_64K: + case 16: return TLBI_TTL_TG_64K; default: return 0; } } +static inline unsigned long get_trans_granule(void) +{ + return get_trans_granule_shift(PAGE_SHIFT); +} + #ifdef CONFIG_ARM64_ERRATUM_4193714 extern cpumask_t sme_active_cpus; @@ -175,13 +181,14 @@ static __always_inline void ipas2e1is(u64 arg) __tlbi(ipas2e1is, arg); } -static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level, - u16 asid) +static __always_inline void __tlbi_level_asid_granule(tlbi_op op, u64 addr, + u32 level, u16 asid, + unsigned int page_shift) { u64 arg = __TLBI_VADDR(addr, asid); if (alternative_has_cap_unlikely(ARM64_HAS_ARMv8_4_TTL) && level <= 3) { - u64 ttl = level | (get_trans_granule() << 2); + u64 ttl = level | (get_trans_granule_shift(page_shift) << 2); FIELD_MODIFY(TLBI_TTL_MASK, &arg, ttl); } @@ -189,6 +196,12 @@ static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level, op(arg); } +static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level, + u16 asid) +{ + __tlbi_level_asid_granule(op, addr, level, asid, PAGE_SHIFT); +} + static inline void __tlbi_level(tlbi_op op, u64 addr, u32 level) { __tlbi_level_asid(op, addr, level, 0); @@ -471,36 +484,45 @@ static __always_inline void ripas2e1is(u64 arg) __tlbi(ripas2e1is, arg); } -static __always_inline void __tlbi_range(tlbi_op op, u64 addr, - u16 asid, int scale, int num, - u32 level, bool lpa2) +static __always_inline void __tlbi_range_granule(tlbi_op op, u64 addr, + u16 asid, int scale, int num, + u32 level, bool lpa2, + unsigned int page_shift) { u64 arg = 0; - arg |= FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : PAGE_SHIFT)); + arg |= FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : page_shift)); arg |= FIELD_PREP(TLBIR_TTL_MASK, level > 3 ? 0 : level); arg |= FIELD_PREP(TLBIR_NUM_MASK, num); arg |= FIELD_PREP(TLBIR_SCALE_MASK, scale); - arg |= FIELD_PREP(TLBIR_TG_MASK, get_trans_granule()); + arg |= FIELD_PREP(TLBIR_TG_MASK, get_trans_granule_shift(page_shift)); arg |= FIELD_PREP(TLBIR_ASID_MASK, asid); op(arg); } -static __always_inline void __flush_tlb_range_op(tlbi_op lop, tlbi_op rop, +/* + * Production callers supply the constant system_supports_tlb_range function, + * which inlines to the existing alternative branch inside the loop. Do not + * evaluate it into a bool outside the loop: that adds runtime flag handling + * to the native path. Recording tests supply constant true/false predicates. + */ +static __always_inline void __flush_tlb_range_op_granule(tlbi_op lop, tlbi_op rop, u64 start, size_t pages, u64 stride, u16 asid, - u32 level, bool lpa2) + u32 level, bool lpa2, + unsigned int page_shift, + bool (*range_supported)(void)) { - u64 addr = start, end = start + pages * PAGE_SIZE; + u64 addr = start, end = start + (pages << page_shift); int scale = 3; while (addr != end) { int num; - pages = (end - addr) >> PAGE_SHIFT; + pages = (end - addr) >> page_shift; - if (!system_supports_tlb_range() || pages == 1) + if (!range_supported() || pages == 1) goto invalidate_one; if (lpa2 && !IS_ALIGNED(addr, SZ_64K)) @@ -508,26 +530,41 @@ static __always_inline void __flush_tlb_range_op(tlbi_op lop, tlbi_op rop, num = __TLBI_RANGE_NUM(pages, scale); if (num >= 0) { - __tlbi_range(rop, addr, asid, scale, num, level, lpa2); - addr += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; + __tlbi_range_granule(rop, addr, asid, scale, num, level, + lpa2, page_shift); + addr += __TLBI_RANGE_PAGES(num, scale) << page_shift; } scale--; continue; invalidate_one: - __tlbi_level_asid(lop, addr, level, asid); + __tlbi_level_asid_granule(lop, addr, level, asid, page_shift); addr += stride; } } +static __always_inline void __flush_tlb_range_op(tlbi_op lop, tlbi_op rop, + u64 start, size_t pages, + u64 stride, u16 asid, + u32 level, bool lpa2) +{ + __flush_tlb_range_op_granule(lop, rop, start, pages, stride, asid, + level, lpa2, PAGE_SHIFT, system_supports_tlb_range); +} + #define __flush_s1_tlb_range_op(op, start, pages, stride, asid, tlb_level) \ __flush_tlb_range_op(op, r##op, start, pages, stride, asid, tlb_level, lpa2_is_enabled()) #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ __flush_tlb_range_op(op, r##op, start, pages, stride, 0, tlb_level, kvm_lpa2_is_enabled()) -static inline bool __flush_tlb_range_limit_excess(unsigned long pages, - unsigned long stride) +#define __flush_user_tlb_range_op(op, start, pages, stride, asid, level, shift) \ + __flush_tlb_range_op_granule(op, r##op, start, pages, stride, asid, \ + level, lpa2_is_enabled(), shift, system_supports_tlb_range) + +static inline bool __flush_tlb_range_limit_excess_granule(unsigned long pages, + unsigned long stride, + unsigned int page_shift) { /* * Assume that the worst case number of DVM ops required to flush a @@ -538,7 +575,13 @@ static inline bool __flush_tlb_range_limit_excess(unsigned long pages, if (system_supports_tlb_range()) return pages > MAX_TLBI_RANGE_PAGES; - return pages >= (MAX_DVM_OPS * stride) >> PAGE_SHIFT; + return pages >= (MAX_DVM_OPS * stride) >> page_shift; +} + +static inline bool __flush_tlb_range_limit_excess(unsigned long pages, + unsigned long stride) +{ + return __flush_tlb_range_limit_excess_granule(pages, stride, PAGE_SHIFT); } typedef unsigned __bitwise tlbf_t; @@ -565,10 +608,11 @@ static __always_inline void __do_flush_tlb_range(struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; unsigned long asid, pages; + unsigned int page_shift = mm_page_shift(mm); - pages = (end - start) >> PAGE_SHIFT; + pages = (end - start) >> page_shift; - if (__flush_tlb_range_limit_excess(pages, stride)) { + if (__flush_tlb_range_limit_excess_granule(pages, stride, page_shift)) { flush_tlb_mm(mm); return; } @@ -582,20 +626,20 @@ static __always_inline void __do_flush_tlb_range(struct vm_area_struct *vma, switch (flags & (TLBF_NOWALKCACHE | TLBF_NOBROADCAST)) { case TLBF_NONE: - __flush_s1_tlb_range_op(vae1is, start, pages, stride, - asid, tlb_level); + __flush_user_tlb_range_op(vae1is, start, pages, stride, + asid, tlb_level, page_shift); break; case TLBF_NOWALKCACHE: - __flush_s1_tlb_range_op(vale1is, start, pages, stride, - asid, tlb_level); + __flush_user_tlb_range_op(vale1is, start, pages, stride, + asid, tlb_level, page_shift); break; case TLBF_NOBROADCAST: /* Combination unused */ BUG(); break; case TLBF_NOWALKCACHE | TLBF_NOBROADCAST: - __flush_s1_tlb_range_op(vale1, start, pages, stride, - asid, tlb_level); + __flush_user_tlb_range_op(vale1, start, pages, stride, + asid, tlb_level, page_shift); break; } @@ -629,16 +673,18 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, * Set the tlb_level to TLBI_TTL_UNKNOWN because we can not get enough * information here. */ - __flush_tlb_range(vma, start, end, PAGE_SIZE, TLBI_TTL_UNKNOWN, TLBF_NONE); + __flush_tlb_range(vma, start, end, mm_page_size(vma->vm_mm), + TLBI_TTL_UNKNOWN, TLBF_NONE); } static inline void __flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr, tlbf_t flags) { - unsigned long start = round_down(uaddr, PAGE_SIZE); - unsigned long end = start + PAGE_SIZE; + unsigned long page_size = mm_page_size(vma->vm_mm); + unsigned long start = round_down(uaddr, page_size); + unsigned long end = start + page_size; - __do_flush_tlb_range(vma, start, end, PAGE_SIZE, 3, + __do_flush_tlb_range(vma, start, end, page_size, 3, TLBF_NOWALKCACHE | flags); } @@ -688,7 +734,7 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b { struct vm_area_struct vma = { .vm_mm = mm, .vm_flags = 0 }; - __flush_tlb_range(&vma, start, end, PAGE_SIZE, 3, + __flush_tlb_range(&vma, start, end, mm_page_size(mm), 3, TLBF_NOWALKCACHE | TLBF_NOSYNC); } diff --git a/arch/arm64/include/asm/user-pgalloc.h b/arch/arm64/include/asm/user-pgalloc.h new file mode 100644 index 00000000000000..3b4dbde0c77367 --- /dev/null +++ b/arch/arm64/include/asm/user-pgalloc.h @@ -0,0 +1,59 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_USER_PGALLOC_H +#define __ASM_USER_PGALLOC_H + +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + +enum arm64_user4k_pt_level { + USER4K_PT_PTE, + USER4K_PT_PMD, + USER4K_PT_PUD, +}; + +/* Explicit geometry entry point; callers must keep each mm's granule fixed. */ +void *arm64_user_pt_alloc_granule(struct mm_struct *mm, + enum arm64_user4k_pt_level level, unsigned int shift); +void *arm64_user4k_pt_alloc(struct mm_struct *mm, enum arm64_user4k_pt_level level); +void arm64_user4k_pt_free(void *table); +void arm64_user4k_pt_cache_destroy(struct mm_struct *mm); +void arm64_pgtable_free(void *table); +void arm64_pte_free_defer(struct mm_struct *mm, pgtable_t table); + +static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm) +{ + if (arm64_mm_alt_granule(mm)) + return arm64_user_pt_alloc_granule(mm, USER4K_PT_PTE, mm_page_shift(mm)); + return __pte_alloc_one_noprof(mm, GFP_PGTABLE_USER); +} +#define pte_alloc_one(...) alloc_hooks(pte_alloc_one_noprof(__VA_ARGS__)) + +static inline void pte_free(struct mm_struct *mm, pgtable_t table) +{ + arm64_pgtable_free(table); +} + +static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return arm64_user_pt_alloc_granule(mm, USER4K_PT_PMD, mm_page_shift(mm)); + return __pmd_alloc_one_noprof(mm, addr); +} +#define pmd_alloc_one(...) alloc_hooks(pmd_alloc_one_noprof(__VA_ARGS__)) + +static inline void pmd_free(struct mm_struct *mm, pmd_t *table) +{ + arm64_pgtable_free(table); +} + +static inline pud_t *pud_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return arm64_user_pt_alloc_granule(mm, USER4K_PT_PUD, mm_page_shift(mm)); + return __pud_alloc_one_noprof(mm, addr); +} +#define pud_alloc_one(...) alloc_hooks(pud_alloc_one_noprof(__VA_ARGS__)) + +#define pte_free_defer arm64_pte_free_defer + +#endif +#endif diff --git a/arch/arm64/include/asm/user-pgtable.h b/arch/arm64/include/asm/user-pgtable.h new file mode 100644 index 00000000000000..7b1d1a94515d00 --- /dev/null +++ b/arch/arm64/include/asm/user-pgtable.h @@ -0,0 +1,187 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_USER_PGTABLE_H +#define __ASM_USER_PGTABLE_H + +/* Included by asm/pgtable.h after the native accessors. */ +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + +/* Alternative tables use 48-bit VA; 64K folds the PUD level. */ +static inline unsigned int arm64_user_level_shift(const struct mm_struct *mm, + unsigned int level) +{ + unsigned int shift = mm_page_shift(mm); + + if (shift == 16 && !level) + level = 1; + return shift + (3 - level) * (shift - 3); +} + +static inline unsigned long arm64_user_table_index(const struct mm_struct *mm, + unsigned long addr, unsigned int level) +{ + unsigned int shift = arm64_user_level_shift(mm, level); + unsigned int bits = level ? mm_page_shift(mm) - 3 : 48 - shift; + + return (addr >> shift) & ((1UL << bits) - 1); +} + +#define p4d_none_mm(mm, p4d) \ + (arm64_mm_alt_granule(mm) ? (!mm_pud_folded(mm) && !p4d_val(p4d)) : p4d_none(p4d)) +#define p4d_bad_mm(mm, p4d) \ + (arm64_mm_alt_granule(mm) ? (!mm_pud_folded(mm) && ((p4d_val(p4d) & P4D_TYPE_MASK) != P4D_TYPE_TABLE)) : p4d_bad(p4d)) +#define p4d_present_mm(mm, p4d) (!p4d_none_mm(mm, p4d)) +static inline void p4d_clear_mm(struct mm_struct *mm, p4d_t *p4d) +{ + if (arm64_mm_alt_granule(mm)) { + if (!mm_pud_folded(mm)) + set_p4d(p4d, __p4d(0)); + } else { + p4d_clear(p4d); + } +} +#define p4d_clear_mm p4d_clear_mm + + +/* pgtable_t preserves the fragment offset; pmd_page() remains its owner. */ +#define pmd_pgtable(pmd) ((pgtable_t)__va(pmd_val(pmd) & ARM64_USER4K_TABLE_ADDR_MASK)) + +static inline void *arm64_user4k_table(u64 descriptor) +{ + /* Unlike native __pXd_to_phys(), retain sub-native table address bits. */ + return __va(descriptor & ARM64_USER4K_TABLE_ADDR_MASK); +} + +static inline pgd_t *pgd_offset_pgd_mm(struct mm_struct *mm, pgd_t *pgd, + unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return pgd + arm64_user_table_index(mm, addr, 0); + return pgd + ((addr >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1)); +} + +#define pgd_offset_pgd_mm pgd_offset_pgd_mm + +static inline pgd_t *pgd_offset_mm(struct mm_struct *mm, unsigned long addr) +{ + return pgd_offset_pgd_mm(mm, mm->pgd, addr); +} + +#define pgd_offset(mm, addr) pgd_offset_mm(mm, addr) + +static inline p4d_t *p4d_offset_mm(struct mm_struct *mm, pgd_t *pgd, + unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return (p4d_t *)pgd; + return p4d_offset(pgd, addr); +} + +#define p4d_offset_mm p4d_offset_mm + +static inline pud_t *pud_offset_mm(struct mm_struct *mm, p4d_t *p4d, + unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) { + if (mm_pud_folded(mm)) + return (pud_t *)p4d; + return (pud_t *)arm64_user4k_table(p4d_val(READ_ONCE(*p4d))) + + arm64_user_table_index(mm, addr, 1); + } + return pud_offset(p4d, addr); +} + +#define pud_offset_mm pud_offset_mm + +static inline pmd_t *pud_pgtable_mm(struct mm_struct *mm, pud_t pud) +{ + if (arm64_mm_alt_granule(mm)) + return arm64_user4k_table(pud_val(pud)); + return pud_pgtable(pud); +} + +#define pud_pgtable_mm pud_pgtable_mm + +static inline pmd_t *pmd_offset_mm(struct mm_struct *mm, pud_t *pud, + unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return (pmd_t *)arm64_user4k_table(pud_val(READ_ONCE(*pud))) + + arm64_user_table_index(mm, addr, 2); + return (pmd_t *)pud_pgtable(READ_ONCE(*pud)) + + ((addr >> PMD_SHIFT) & (PTRS_PER_PMD - 1)); +} + +#define pmd_offset_mm pmd_offset_mm + +/* Use the caller's descriptor snapshots throughout a lockless walk. */ +#define p4d_offset_lockless_mm(mm, pgdp, pgd, addr) \ + (arm64_mm_alt_granule(mm) ? p4d_offset_mm(mm, &(pgd), addr) : \ + p4d_offset_lockless(pgdp, pgd, addr)) +#define pud_offset_lockless_mm(mm, p4dp, p4d, addr) \ + (arm64_mm_alt_granule(mm) ? pud_offset_mm(mm, &(p4d), addr) : \ + pud_offset_lockless(p4dp, p4d, addr)) +#define pmd_offset_lockless_mm(mm, pudp, pud, addr) \ + (arm64_mm_alt_granule(mm) ? pmd_offset_mm(mm, &(pud), addr) : \ + pmd_offset_lockless(pudp, pud, addr)) + +static inline pte_t *pte_offset_kernel_mm(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr) +{ + if (arm64_mm_alt_granule(mm)) + return (pte_t *)arm64_user4k_table(pmd_val(READ_ONCE(*pmd))) + + arm64_user_table_index(mm, addr, 3); + return (pte_t *)pmd_page_vaddr(READ_ONCE(*pmd)) + + ((addr >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)); +} + +#define pte_offset_kernel_mm pte_offset_kernel_mm +#define __pte_map_mm(mm, pmd, addr) pte_offset_kernel_mm(mm, pmd, addr) + +#define pmd_shift_mm(mm) (arm64_mm_alt_granule(mm) ? arm64_user_level_shift(mm, 2) : PMD_SHIFT) +#define pud_shift_mm(mm) (arm64_mm_alt_granule(mm) ? arm64_user_level_shift(mm, 1) : PUD_SHIFT) +#define p4d_shift_mm(mm) (arm64_mm_alt_granule(mm) ? arm64_user_level_shift(mm, 0) : P4D_SHIFT) +#define pmd_size_mm(mm) (1UL << pmd_shift_mm(mm)) +#define pud_size_mm(mm) (1UL << pud_shift_mm(mm)) +#define p4d_size_mm(mm) (1UL << p4d_shift_mm(mm)) +#define pgd_size_mm(mm) (arm64_mm_alt_granule(mm) ? (1UL << arm64_user_level_shift(mm, 0)) : PGDIR_SIZE) +#define pgd_leaf_size_mm(mm, pgd) \ + (arm64_mm_alt_granule(mm) ? pgd_size_mm(mm) : pgd_leaf_size(pgd)) +#define p4d_leaf_size_mm(mm, p4d) \ + (arm64_mm_alt_granule(mm) ? p4d_size_mm(mm) : p4d_leaf_size(p4d)) +#define pud_leaf_size_mm(mm, pud) \ + (arm64_mm_alt_granule(mm) ? pud_size_mm(mm) : pud_leaf_size(pud)) +#define pmd_leaf_size_mm(mm, pmd) \ + (arm64_mm_alt_granule(mm) ? pmd_size_mm(mm) * \ + (pmd_cont(pmd) ? (mm_page_shift(mm) == 12 ? 16 : 32) : 1) : pmd_leaf_size(pmd)) +#define __pte_leaf_size_mm(mm, pmd, pte) \ + (arm64_mm_alt_granule(mm) ? \ + (pte_cont(pte) ? cont_pte_size_mm(mm) : mm_page_size(mm)) : __pte_leaf_size(pmd, pte)) +#define pte_table_bytes_mm(mm) (arm64_mm_alt_granule(mm) ? mm_page_size(mm) : PTRS_PER_PTE * sizeof(pte_t)) +#define pmd_table_bytes_mm(mm) (arm64_mm_alt_granule(mm) ? mm_page_size(mm) : PTRS_PER_PMD * sizeof(pmd_t)) +#define pud_table_bytes_mm(mm) (arm64_mm_alt_granule(mm) ? mm_page_size(mm) : PTRS_PER_PUD * sizeof(pud_t)) +#define pmd_mask_mm(mm) (~(pmd_size_mm(mm) - 1)) +#define pud_mask_mm(mm) (~(pud_size_mm(mm) - 1)) +#define p4d_mask_mm(mm) (~(p4d_size_mm(mm) - 1)) +#define pgd_mask_mm(mm) (~(pgd_size_mm(mm) - 1)) + +static inline unsigned long arm64_user4k_addr_end(unsigned long addr, + unsigned long end, unsigned int shift) +{ + unsigned long size = 1UL << shift; + unsigned long boundary = (addr + size) & ~(size - 1); + + return boundary - 1 < end - 1 ? boundary : end; +} + +/* Native fallbacks retain their architecture-specific folding semantics. */ +#define pgd_addr_end_mm(mm, addr, end) \ + (arm64_mm_alt_granule(mm) ? arm64_user4k_addr_end(addr, end, arm64_user_level_shift(mm, 0)) : pgd_addr_end(addr, end)) +#define p4d_addr_end_mm(mm, addr, end) \ + (arm64_mm_alt_granule(mm) ? (end) : p4d_addr_end(addr, end)) +#define pud_addr_end_mm(mm, addr, end) \ + (arm64_mm_alt_granule(mm) ? arm64_user4k_addr_end(addr, end, arm64_user_level_shift(mm, 1)) : pud_addr_end(addr, end)) +#define pmd_addr_end_mm(mm, addr, end) \ + (arm64_mm_alt_granule(mm) ? arm64_user4k_addr_end(addr, end, arm64_user_level_shift(mm, 2)) : pmd_addr_end(addr, end)) + +#endif /* CONFIG_ARM64_USER4K_EXPERIMENTAL */ +#endif diff --git a/arch/arm64/include/asm/user-pte.h b/arch/arm64/include/asm/user-pte.h new file mode 100644 index 00000000000000..aa4c1e3a5aab96 --- /dev/null +++ b/arch/arm64/include/asm/user-pte.h @@ -0,0 +1,116 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_USER_PTE_H +#define __ASM_USER_PTE_H + +/* Included after native leaf accessors, before their batched setters. */ +#define ARM64_USER4K_TABLE_ADDR_MASK GENMASK_ULL(47, 12) + +static inline bool arm64_mm_user4k(const struct mm_struct *mm) +{ + return mm && arm64_context_user4k(&mm->context); +} + +static inline bool arm64_mm_alt_granule(const struct mm_struct *mm) +{ + return mm && arm64_context_page_shift(&mm->context) != PAGE_SHIFT; +} + +static inline u64 arm64_user_leaf_addr_mask(const struct mm_struct *mm) +{ + return ARM64_USER4K_TABLE_ADDR_MASK & mm_page_mask(mm); +} + +/* PFNs still identify native pages; leaf physical addresses identify slots. */ +static inline phys_addr_t pte_phys_mm(const struct mm_struct *mm, pte_t pte) +{ + if (arm64_mm_alt_granule(mm)) + return pte_val(pte) & arm64_user_leaf_addr_mask(mm); + return __pte_to_phys(pte); +} +#define pte_phys_mm pte_phys_mm + +static inline pte_t phys_pte_mm(const struct mm_struct *mm, phys_addr_t phys, + pgprot_t prot) +{ + if (arm64_mm_alt_granule(mm)) { + VM_BUG_ON(phys & ~arm64_user_leaf_addr_mask(mm)); + return __pte(phys | pgprot_val(prot)); + } + return pfn_pte(phys >> PAGE_SHIFT, prot); +} +#define phys_pte_mm phys_pte_mm + +static inline pgprot_t pte_pgprot_mm(const struct mm_struct *mm, pte_t pte) +{ + if (arm64_mm_alt_granule(mm)) + return __pgprot(pte_val(pte) & ~arm64_user_leaf_addr_mask(mm)); + return pte_pgprot(pte); +} +#define pte_pgprot_mm pte_pgprot_mm + + +#ifdef CONFIG_MM_SUBPAGE +/* Slot bits belong to the PTE, not the native swap-cache index. */ +#define ARM64_SWP_SUBPAGE_MASK (GENMASK_ULL(5, 4) | GENMASK_ULL(63, 62)) + +static inline unsigned int pte_swp_subpage_offset(pte_t pte) +{ + return ((((pte_val(pte) >> 4) & 3) | + ((pte_val(pte) >> 60) & 12)) << 12); +} + +static inline pte_t pte_swp_set_subpage_offset(pte_t pte, unsigned int offset) +{ + VM_BUG_ON(!IS_ALIGNED(offset, SZ_4K) || offset >= PAGE_SIZE); + return __pte((pte_val(pte) & ~ARM64_SWP_SUBPAGE_MASK) | + (((u64)(offset >> 12) & 3) << 4) | + (((u64)(offset >> 12) & 12) << 60)); +} +#endif + +static inline pte_t pte_advance_pfn_mm(const struct mm_struct *mm, pte_t pte, + unsigned long nr) +{ + if (arm64_mm_alt_granule(mm)) + return phys_pte_mm(mm, pte_phys_mm(mm, pte) + (nr << mm_page_shift(mm)), + pte_pgprot_mm(mm, pte)); + return pte_advance_pfn(pte, nr); +} + +#define pte_advance_pfn_mm pte_advance_pfn_mm + +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +unsigned int arm64_ptep_page_shift(pte_t *ptep); +#else +static inline unsigned int arm64_ptep_page_shift(pte_t *ptep) +{ + return PAGE_SHIFT; +} +#endif + +static inline unsigned int arm64_cont_ptes(unsigned int shift) +{ + /* Architectural contiguous group lengths for 4K, 16K and 64K. */ + return shift == 12 ? 16 : shift == 14 ? 128 : 32; +} + +static inline unsigned int cont_ptes_mm(const struct mm_struct *mm) +{ + return arm64_cont_ptes(mm_page_shift(mm)); +} + +static inline unsigned long cont_pte_size_mm(const struct mm_struct *mm) +{ + return (unsigned long)cont_ptes_mm(mm) << mm_page_shift(mm); +} + +static inline unsigned long pte_cont_addr_end_mm(const struct mm_struct *mm, + unsigned long addr, unsigned long end) +{ + unsigned long size = cont_pte_size_mm(mm); + unsigned long boundary = (addr + size) & ~(size - 1); + + return boundary - 1 < end - 1 ? boundary : end; +} + +#endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 432d35b2714cd2..e30ef03e87b3ef 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1661,6 +1661,62 @@ has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope) return cpufeature_matches(val, entry); } +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +#ifdef CONFIG_ARM64_USER4K_KUNIT_TEST +/* Removal-only model for late-CPU verification in a disposable test VM. */ +static int user4k_deny_late_cpu __ro_after_init = -1; + +static int __init user4k_deny_late_cpu_setup(char *str) +{ + int cpu; + + if (kstrtoint(str, 0, &cpu) || cpu <= 0 || cpu >= NR_CPUS) + return -EINVAL; + user4k_deny_late_cpu = cpu; + return 0; +} +early_param("arm64.user4k_deny_late_cpu", user4k_deny_late_cpu_setup); +#ifdef CONFIG_ARM64_64K_PAGES +static int user16k_deny_late_cpu __ro_after_init = -1; + +static int __init user16k_deny_late_cpu_setup(char *str) +{ + int cpu; + + if (kstrtoint(str, 0, &cpu) || cpu <= 0 || cpu >= NR_CPUS) + return -EINVAL; + user16k_deny_late_cpu = cpu; + return 0; +} +early_param("arm64.user16k_deny_late_cpu", user16k_deny_late_cpu_setup); +#endif +#endif + +static bool has_user4k_granule(const struct arm64_cpu_capabilities *entry, + int scope) +{ +#ifdef CONFIG_ARM64_USER4K_KUNIT_TEST + if (scope == SCOPE_LOCAL_CPU && system_capabilities_finalized() && + smp_processor_id() == user4k_deny_late_cpu) + return false; +#endif + return has_cpuid_feature(entry, scope); +} + +#ifdef CONFIG_ARM64_64K_PAGES +static bool has_user16k_granule(const struct arm64_cpu_capabilities *entry, + int scope) +{ +#ifdef CONFIG_ARM64_USER4K_KUNIT_TEST + if (scope == SCOPE_LOCAL_CPU && system_capabilities_finalized() && + smp_processor_id() == user16k_deny_late_cpu) + return false; +#endif + return has_cpuid_feature(entry, scope); +} +#endif +#endif + const struct cpumask *system_32bit_el0_cpumask(void) { if (!system_supports_32bit_el0()) @@ -2391,8 +2447,11 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap) * clear the tags or set PG_mte_tagged. */ if (!cleared_zero_page) { + unsigned long offset; + cleared_zero_page = true; - mte_clear_page_tags(lm_alias(empty_zero_page)); + for (offset = 0; offset < sizeof(empty_zero_page); offset += PAGE_SIZE) + mte_clear_page_tags(lm_alias(empty_zero_page + offset)); } kasan_init_hw_tags_cpu(); @@ -2529,6 +2588,28 @@ test_has_gicv5_legacy(const struct arm64_cpu_capabilities *entry, int scope) } static const struct arm64_cpu_capabilities arm64_features[] = { +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + { + .desc = "4K userspace translation granule", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_USER4K_GRANULE, + .matches = has_user4k_granule, + __ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN4, + ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN, + ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX) + }, +#ifdef CONFIG_ARM64_64K_PAGES + { + .desc = "16K userspace translation granule", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_USER16K_GRANULE, + .matches = has_user16k_granule, + __ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN16, + ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN, + ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX) + }, +#endif +#endif { .capability = ARM64_ALWAYS_BOOT, .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, diff --git a/arch/arm64/kernel/elfcore.c b/arch/arm64/kernel/elfcore.c index b735f4c2fe5ea8..32ddc36838fe35 100644 --- a/arch/arm64/kernel/elfcore.c +++ b/arch/arm64/kernel/elfcore.c @@ -2,8 +2,10 @@ #include #include +#include #include #include +#include #include #include @@ -17,9 +19,60 @@ static unsigned long mte_vma_tag_dump_size(struct core_vma_metadata *m) { - return (m->dump_size >> PAGE_SHIFT) * MTE_PAGE_TAG_STORAGE; + return (m->dump_size / MTE_GRANULE_SIZE) * MTE_TAG_SIZE / 8; } +#ifdef CONFIG_MM_SUBPAGE +static int mte_dump_fragment_tags(struct coredump_params *cprm, + unsigned long start, unsigned long len) +{ + unsigned long size = mm_user_fragment_size(current->mm); + unsigned long tag_size = (size / MTE_GRANULE_SIZE) * MTE_TAG_SIZE / 8; + unsigned long addr; + void *tags = NULL; + int ret = 1; + + for (addr = start; addr < start + len; addr += size) { + struct user_page_fragment fragment; + struct page *page; + + if (get_dump_fragment(addr, &fragment) != 1) { + dump_skip(cprm, tag_size); + goto next; + } + page = folio_page(fragment.folio, fragment.offset >> PAGE_SHIFT); + if (!page_mte_tagged(page)) { + release_user_fragments(&fragment, 1, false); + dump_skip(cprm, tag_size); + goto next; + } + if (!tags) { + tags = mte_allocate_tag_storage(); + if (!tags) { + release_user_fragments(&fragment, 1, false); + ret = 0; + break; + } + } + mte_save_tags_range(folio_address(fragment.folio) + fragment.offset, tags, size); + release_user_fragments(&fragment, 1, false); + if (!dump_emit(cprm, tags, tag_size)) { + ret = 0; + break; + } +next: + if (fatal_signal_pending(current) || freezing(current)) { + ret = 0; + break; + } + cond_resched(); + } + if (tags) + mte_free_tag_storage(tags); + return ret; +} +#endif + /* Derived from dump_user_range(); start/end must be page-aligned */ static int mte_dump_tag_range(struct coredump_params *cprm, unsigned long start, unsigned long len) @@ -29,6 +82,11 @@ static int mte_dump_tag_range(struct coredump_params *cprm, void *tags = NULL; int locked = 0; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return mte_dump_fragment_tags(cprm, start, len); +#endif + for (addr = start; addr < start + len; addr += PAGE_SIZE) { struct page *page = get_dump_page(addr, &locked); diff --git a/arch/arm64/kernel/idle.c b/arch/arm64/kernel/idle.c index 972164e78874c4..14cde8592c86e4 100644 --- a/arch/arm64/kernel/idle.c +++ b/arch/arm64/kernel/idle.c @@ -41,9 +41,9 @@ early_param("idle", setup_idle); #ifdef CONFIG_ARCH_APPLE /* - * Apple T8140 (A18 Pro) secondary CPUs and T8152 (M6) CPUs can return from + * Apple T8140 (A18 Pro), T8142 (M5), and T8152 (M6) CPUs can return from * WFI with general-purpose registers cleared. T8140 also loses FP/SIMD - * state; conservatively preserve it on both parts. The key is enabled once + * state; conservatively preserve it on all three parts. The key is enabled once * at boot on those parts; everywhere else the check is a patched-out branch. */ static DEFINE_STATIC_KEY_FALSE(apple_wfi_loses_regs); @@ -51,6 +51,8 @@ static DEFINE_STATIC_KEY_FALSE(apple_wfi_loses_regs); static const struct midr_range apple_wfi_loses_regs_cpus[] __initconst = { MIDR_ALL_VERSIONS(MIDR_APPLE_TAHITI_E), MIDR_ALL_VERSIONS(MIDR_APPLE_TAHITI_P), + MIDR_ALL_VERSIONS(MIDR_APPLE_T8142_E), + MIDR_ALL_VERSIONS(MIDR_APPLE_T8142_P), MIDR_ALL_VERSIONS(MIDR_APPLE_T8152_E), MIDR_ALL_VERSIONS(MIDR_APPLE_T8152_P), {} diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 6874b16d065767..fb4eba0626e5a5 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -69,21 +70,23 @@ void mte_sync_tags(pte_t pte, unsigned int nr_pages) smp_wmb(); } -int memcmp_pages(struct page *page1, struct page *page2) +int memcmp_pages_range(struct page *page1, unsigned int offset1, + struct page *page2, unsigned int offset2, size_t len) { char *addr1, *addr2; int ret; - addr1 = page_address(page1); - addr2 = page_address(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); + addr1 = page_address(page1) + offset1; + addr2 = page_address(page2) + offset2; + ret = memcmp(addr1, addr2, len); - if (!system_supports_mte() || ret) + if (!len || !system_supports_mte() || ret) return ret; /* - * If the page content is identical but at least one of the pages is - * tagged, return non-zero to avoid KSM merging. If only one of the + * If the byte ranges are identical but at least one page is tagged, + * only the same physical range is interchangeable. Avoid merging or + * replacing independently tagged ranges. If only one of the * pages is tagged, __set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ @@ -93,6 +96,11 @@ int memcmp_pages(struct page *page1, struct page *page2) return ret; } +int memcmp_pages(struct page *page1, struct page *page2) +{ + return memcmp_pages_range(page1, 0, page2, 0, PAGE_SIZE); +} + static inline void __mte_enable_kernel(const char *mode, unsigned long tcf) { /* Enable MTE Sync Mode for EL1. */ @@ -456,11 +464,24 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, while (len) { struct vm_area_struct *vma; - unsigned long tags, offset; + unsigned long tags = 0, offset; void *maddr; - struct page *page = get_user_page_vma_remote(mm, addr, - gup_flags, &vma); + struct page *page; struct folio *folio; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment = {}; + + if (mm_page_size(mm) != PAGE_SIZE) { + unsigned long bytes = min(len, PAGE_SIZE / MTE_GRANULE_SIZE) * + MTE_GRANULE_SIZE; + int got = get_user_fragment_vma_remote(mm, addr, bytes, + gup_flags, &fragment, &vma); + + page = got > 0 ? folio_page(fragment.folio, + fragment.offset >> PAGE_SHIFT) : ERR_PTR(got ?: -EFAULT); + } else +#endif + page = get_user_page_vma_remote(mm, addr, gup_flags, &vma); if (IS_ERR(page)) { err = PTR_ERR(page); @@ -476,8 +497,7 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, */ if (!(vma->vm_flags & VM_MTE)) { err = -EOPNOTSUPP; - put_page(page); - break; + goto release; } folio = page_folio(page); @@ -490,18 +510,33 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, /* limit access to the end of the page */ offset = offset_in_page(addr); tags = min(len, (PAGE_SIZE - offset) / MTE_GRANULE_SIZE); +#ifdef CONFIG_MM_SUBPAGE + if (fragment.folio) { + offset = offset_in_page(fragment.offset); + tags = fragment.length / MTE_GRANULE_SIZE; + } +#endif maddr = page_address(page); if (write) { tags = mte_copy_tags_from_user(maddr + offset, buf, tags); - set_page_dirty_lock(page); +#ifdef CONFIG_MM_SUBPAGE + if (!fragment.folio) +#endif + set_page_dirty_lock(page); } else { tags = mte_copy_tags_to_user(buf, maddr + offset, tags); } - put_page(page); +release: +#ifdef CONFIG_MM_SUBPAGE + if (fragment.folio) + release_user_fragments(&fragment, 1, write && !err); + else +#endif + put_page(page); /* error accessing the tracer's buffer */ - if (!tags) + if (err || !tags) break; len -= tags; diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1288766a68bf6c..622dc2034664cd 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -38,11 +38,32 @@ struct ftr_set_desc { #define FIELD(n, s, f) { .name = n, .shift = s, .width = 4, .filter = f } +#if defined(CONFIG_ARM64_USER4K_KUNIT_TEST) && !defined(CONFIG_ARM64_4K_PAGES) +/* VM capability-model test: only remove 4K support, never fabricate it. */ +static bool __init user4k_granule_test_filter(u64 val) +{ + return val == ID_AA64MMFR0_EL1_TGRAN4_NI; +} + +#ifdef CONFIG_ARM64_64K_PAGES +static bool __init user16k_granule_test_filter(u64 val) +{ + return val == ID_AA64MMFR0_EL1_TGRAN16_NI; +} +#endif +#endif + static const struct ftr_set_desc mmfr0 __prel64_initconst = { .name = "id_aa64mmfr0", .override = &id_aa64mmfr0_override, .fields = { FIELD("ecv", ID_AA64MMFR0_EL1_ECV_SHIFT, NULL), +#if defined(CONFIG_ARM64_USER4K_KUNIT_TEST) && !defined(CONFIG_ARM64_4K_PAGES) + FIELD("tgran4", ID_AA64MMFR0_EL1_TGRAN4_SHIFT, user4k_granule_test_filter), +#ifdef CONFIG_ARM64_64K_PAGES + FIELD("tgran16", ID_AA64MMFR0_EL1_TGRAN16_SHIFT, user16k_granule_test_filter), +#endif +#endif {} }, }; @@ -59,11 +80,22 @@ static bool __init mmfr1_vh_filter(u64 val) val == 0); } +#if defined(CONFIG_ARM64_USER4K_KUNIT_TEST) && defined(CONFIG_ARM64_SW_TTBR0_PAN) +/* VM-only coverage of deferred TTBR0 restore on CPUs which implement PAN. */ +static bool __init user4k_pan_test_filter(u64 val) +{ + return val == 0; +} +#endif + static const struct ftr_set_desc mmfr1 __prel64_initconst = { .name = "id_aa64mmfr1", .override = &id_aa64mmfr1_override, .fields = { FIELD("vh", ID_AA64MMFR1_EL1_VH_SHIFT, mmfr1_vh_filter), +#if defined(CONFIG_ARM64_USER4K_KUNIT_TEST) && defined(CONFIG_ARM64_SW_TTBR0_PAN) + FIELD("pan", ID_AA64MMFR1_EL1_PAN_SHIFT, user4k_pan_test_filter), +#endif {} }, }; diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index a852264958c36b..c7dd0ba8d800e8 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -243,7 +243,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, phys_addr_t fdt) static char const chosen_str[] __initconst = "/chosen"; u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; - int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int root_level = 4 - ARM64_NATIVE_PGTABLE_LEVELS; int va_bits = VA_BITS; int chosen; void *fdt_mapped = map_fdt(fdt); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index ab1beffa8d7a44..a326135b164d4f 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -7,11 +7,13 @@ * Copyright (C) 2012 ARM Ltd. */ #include +#include #include #include #include #include #include +#include #include #include #include @@ -884,6 +886,33 @@ unsigned long arch_align_stack(unsigned long sp) return sp & ~0xf; } +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +int arm64_bprm_prepare_mm(struct linux_binprm *bprm, bool compat) +{ + unsigned int requested = current->exec_page_shift ?: + current->default_exec_page_shift; + unsigned int shift = arm64_exec_page_shift(compat, + cpus_have_final_cap(ARM64_HAS_USER4K_GRANULE), PAGE_SHIFT, + requested); + + return bprm_set_page_shift(bprm, shift); +} + +/* Return zero for the native default, or a validated alternative shift. */ +int arch_exec_page_size_shift(unsigned long size) +{ + if (!size || size == PAGE_SIZE) + return 0; + if (size == SZ_4K) + return cpus_have_final_cap(ARM64_HAS_USER4K_GRANULE) ? + 12 : -EOPNOTSUPP; + if (PAGE_SIZE > SZ_16K && size == SZ_16K) + return cpus_have_final_cap(ARM64_HAS_USER16K_GRANULE) ? + 14 : -EOPNOTSUPP; + return -EINVAL; +} +#endif + #ifdef CONFIG_COMPAT int compat_elf_check_arch(const struct elf32_hdr *hdr) { @@ -911,10 +940,10 @@ int compat_elf_check_arch(const struct elf32_hdr *hdr) */ void arch_setup_new_exec(void) { - unsigned long mmflags = 0; + unsigned long mmflags = current->mm->context.flags & MMCF_USER_GRANULE; if (is_compat_task()) { - mmflags = MMCF_AARCH32; + mmflags |= MMCF_AARCH32; /* * Restrict the CPU affinity mask for a 32-bit task so that diff --git a/arch/arm64/kernel/sys.c b/arch/arm64/kernel/sys.c index f08408b6e8263c..bdd2531554f270 100644 --- a/arch/arm64/kernel/sys.c +++ b/arch/arm64/kernel/sys.c @@ -22,6 +22,11 @@ SYSCALL_DEFINE6(mmap, unsigned long, addr, unsigned long, len, unsigned long, prot, unsigned long, flags, unsigned long, fd, unsigned long, off) { +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return ksys_mmap_user(addr, len, prot, flags, fd, off); +#endif + if (offset_in_page(off) != 0) return -EINVAL; diff --git a/arch/arm64/kernel/sys32.c b/arch/arm64/kernel/sys32.c index 12a948f3a50437..ba66b65f506f4b 100644 --- a/arch/arm64/kernel/sys32.c +++ b/arch/arm64/kernel/sys32.c @@ -7,6 +7,7 @@ #include #include +#include #include #include @@ -51,6 +52,11 @@ COMPAT_SYSCALL_DEFINE6(aarch32_mmap2, unsigned long, addr, unsigned long, len, unsigned long, prot, unsigned long, flags, unsigned long, fd, unsigned long, off_4k) { +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return ksys_mmap_user(addr, len, prot, flags, fd, off_4k << 12); +#endif + if (off_4k & (~PAGE_MASK >> 12)) return -EINVAL; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index e1ac876200a3d6..688b82c0cf30da 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -66,9 +66,10 @@ * We require that __hyp_bss_start and __bss_start are aligned, and enforce it * with an assertion. But the BSS_SECTION macro places an empty .sbss section * between them, which can in some cases cause the linker to misalign them. To - * work around the issue, force a page alignment for __bss_start. + * work around the issue, align __bss_start to the BSS section's alignment, + * which can exceed PAGE_SIZE for the mixed-granule zero page. */ -#define SBSS_ALIGN PAGE_SIZE +#define SBSS_ALIGN MAX(PAGE_SIZE, ALIGNOF(.bss)) #else /* CONFIG_KVM */ #define HYPERVISOR_EXTABLE #define HYPERVISOR_RODATA_SECTIONS diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c index e278e060e78a9e..e0c5e615b5e5d2 100644 --- a/arch/arm64/lib/delay.c +++ b/arch/arm64/lib/delay.c @@ -14,6 +14,7 @@ #include #include +#include #define USECS_TO_CYCLES(time_usecs) \ xloops_to_cycles((time_usecs) * 0x10C7UL) @@ -42,6 +43,18 @@ void __delay(unsigned long cycles) { cycles_t start = __delay_cycles(); + /* + * T8142 can clear registers on WFIT just as it does on WFI. A delay + * may run with live kernel SIMD state, so the idle task's save/flush + * path is not applicable here. Keep the core awake for short delays. + */ + if (IS_ENABLED(CONFIG_ARCH_APPLE)) { + u32 midr = read_cpuid_id() & MIDR_CPU_MODEL_MASK; + + if (midr == MIDR_APPLE_T8142_E || midr == MIDR_APPLE_T8142_P) + goto counter_delay; + } + if (alternative_has_cap_unlikely(ARM64_HAS_WFXT)) { u64 end = start + cycles; @@ -60,6 +73,7 @@ void __delay(unsigned long cycles) wfe(); } +counter_delay: while ((__delay_cycles() - start) < cycles) cpu_relax(); } diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 5018ac03b6bf39..8402e81b714749 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -80,6 +80,52 @@ SYM_FUNC_START(mte_copy_page_tags) ret SYM_FUNC_END(mte_copy_page_tags) +#ifdef CONFIG_MM_SUBPAGE +/* Copy one aligned power-of-two extent, preserving neighbouring tags. */ +SYM_FUNC_START(mte_copy_subpage_tags_range) + mov x6, x2 + add x2, x0, x2 + multitag_transfer_size x4, x5 + cmp x4, x6 + b.hi 2f +1: ldgm x3, [x1] + stgm x3, [x0] + add x0, x0, x4 + add x1, x1, x4 + cmp x0, x2 + b.ne 1b + ret +2: mov x3, xzr +3: ldg x3, [x1] + stg x3, [x0] + add x0, x0, #MTE_GRANULE_SIZE + add x1, x1, #MTE_GRANULE_SIZE + cmp x0, x2 + b.ne 3b + ret +SYM_FUNC_END(mte_copy_subpage_tags_range) + +/* Clear an aligned extent without changing data or neighbouring tags. */ +SYM_FUNC_START(mte_clear_subpage_tags_range) + mov x4, x1 + add x1, x0, x1 + multitag_transfer_size x2, x3 + cmp x2, x4 + b.hi 2f +1: stgm xzr, [x0] + add x0, x0, x2 + cmp x0, x1 + b.ne 1b + ret +2: mov x2, xzr +3: stg x2, [x0] + add x0, x0, #MTE_GRANULE_SIZE + cmp x0, x1 + b.ne 3b + ret +SYM_FUNC_END(mte_clear_subpage_tags_range) +#endif + /* * Read tags from a user buffer (one tag per byte) and set the corresponding * tags at the given kernel address. Used by PTRACE_POKEMTETAGS. @@ -137,6 +183,19 @@ SYM_FUNC_END(mte_copy_tags_to_user) * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes */ SYM_FUNC_START(mte_save_page_tags) + mov x2, #PAGE_SIZE + b mte_save_tags_range +SYM_FUNC_END(mte_save_page_tags) + +/* + * Save tags for a 256-byte-aligned range. + * x0 - start address + * x1 - tag storage, x2 / 32 bytes + * x2 - byte length, a multiple of 256 + */ +SYM_FUNC_START(mte_save_tags_range) + cbz x2, 3f + add x6, x0, x2 multitag_transfer_size x7, x5 1: mov x2, #0 @@ -149,11 +208,12 @@ SYM_FUNC_START(mte_save_page_tags) str x2, [x1], #8 - tst x0, #(PAGE_SIZE - 1) - b.ne 1b + cmp x0, x6 + b.lo 1b +3: ret -SYM_FUNC_END(mte_save_page_tags) +SYM_FUNC_END(mte_save_tags_range) /* * Restore the tags in a page diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index c26489cf96cd90..757f7eb07ccac6 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -12,6 +12,9 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o obj-$(CONFIG_ARM64_GCS) += gcs.o +obj-$(CONFIG_ARM64_USER4K_KUNIT_TEST) += user4k-test.o user4k-tlb-test.o +obj-$(CONFIG_ARM64_USER4K_DATA_KUNIT_TEST) += user4k-data-test.o +obj-$(CONFIG_ARM64_USER4K_EXPERIMENTAL) += user4k-pgalloc.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 0f4a28b874692d..cd3f56b9427083 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -38,6 +39,34 @@ static unsigned long *pinned_asid_map; #define ctxid2asid(asid) ((asid) & ~ASID_MASK) #define asid2ctxid(asid, genid) ((asid) | (genid)) +#ifdef CONFIG_ARM64_USER4K_KUNIT_TEST +/* Test-only capacity reduction; hardware ASID width and normal boots are unchanged. */ +static bool user4k_test_small_asids; + +static int __init user4k_test_asid_bits_setup(char *str) +{ + unsigned int bits; + + if (kstrtouint(str, 0, &bits) || bits != 8) + return -EINVAL; + user4k_test_small_asids = true; + return 0; +} +early_param("arm64.user4k_test_asid_bits", user4k_test_asid_bits_setup); + +/* Read-only evidence that an EL0 stress run actually crossed ASID rollover. */ +static int user4k_asid_generation_get(char *buffer, const struct kernel_param *kp) +{ + return sysfs_emit(buffer, "%llu\n", (unsigned long long)atomic64_read(&asid_generation)); +} + +static const struct kernel_param_ops user4k_asid_generation_ops = { + .get = user4k_asid_generation_get, +}; +module_param_cb(user4k_asid_generation, &user4k_asid_generation_ops, NULL, 0444); +MODULE_PARM_DESC(user4k_asid_generation, "Current ASID generation for mixed-granule VM tests"); +#endif + /* Get the ASIDBits supported by the current CPU */ static u32 get_cpu_asid_bits(void) { @@ -268,6 +297,13 @@ void check_and_switch_context(struct mm_struct *mm) */ if (!system_uses_ttbr0_pan()) cpu_switch_mm(mm->pgd, mm); +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + else { + /* uaccess/EL0 return restore only TTBRs, so select TG0 here. */ + cpu_set_reserved_ttbr0_nosync(); + cpu_set_mm_tg0(mm); + } +#endif } unsigned long arm64_mm_context_get(struct mm_struct *mm) @@ -365,6 +401,7 @@ void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm) ttbr1 |= FIELD_PREP(TTBRx_EL1_ASID_MASK, asid); cpu_set_reserved_ttbr0_nosync(); + cpu_set_mm_tg0(mm); write_sysreg(ttbr1, ttbr1_el1); write_sysreg(ttbr0, ttbr0_el1); isb(); @@ -401,6 +438,12 @@ arch_initcall(asids_update_limit); static int asids_init(void) { asid_bits = get_cpu_asid_bits(); +#ifdef CONFIG_ARM64_USER4K_KUNIT_TEST + if (user4k_test_small_asids) { + pr_info("user4k test: limiting ASID allocator to 8 bits\n"); + asid_bits = 8; + } +#endif atomic64_set(&asid_generation, ASID_FIRST_VERSION); asid_map = bitmap_zalloc(NUM_USER_ASIDS, GFP_KERNEL); if (!asid_map) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 2de12656b4d8ba..b3fcca267148bc 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -8,6 +8,25 @@ #include #include +static inline unsigned int contpte_nr(pte_t *ptep) +{ + return arm64_cont_ptes(arm64_ptep_page_shift(ptep)); +} + +static inline phys_addr_t contpte_phys(pte_t pte, unsigned int shift) +{ + if (IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) && shift != PAGE_SHIFT) + return pte_val(pte) & (ARM64_USER4K_TABLE_ADDR_MASK & ~((1ULL << shift) - 1)); + return __pte_to_phys(pte); +} + +static inline pgprot_t contpte_prot(pte_t pte, unsigned int shift) +{ + if (IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) && shift != PAGE_SHIFT) + return __pgprot(pte_val(pte) & ~(ARM64_USER4K_TABLE_ADDR_MASK & ~((1ULL << shift) - 1))); + return pte_pgprot(pte); +} + static inline bool mm_is_user(struct mm_struct *mm) { /* @@ -23,10 +42,10 @@ static inline bool mm_is_user(struct mm_struct *mm) static inline pte_t *contpte_align_down(pte_t *ptep) { - return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); + return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * contpte_nr(ptep)); } -static inline pte_t *contpte_align_addr_ptep(unsigned long *start, +static inline pte_t *contpte_align_addr_ptep(struct mm_struct *mm, unsigned long *start, unsigned long *end, pte_t *ptep, unsigned int nr) { @@ -36,10 +55,10 @@ static inline pte_t *contpte_align_addr_ptep(unsigned long *start, * single VMA and a single page table. */ if (pte_cont(__ptep_get(ptep + nr - 1))) - *end = ALIGN(*end, CONT_PTE_SIZE); + *end = ALIGN(*end, cont_pte_size_mm(mm)); if (pte_cont(__ptep_get(ptep))) { - *start = ALIGN_DOWN(*start, CONT_PTE_SIZE); + *start = ALIGN_DOWN(*start, cont_pte_size_mm(mm)); ptep = contpte_align_down(ptep); } @@ -49,16 +68,19 @@ static inline pte_t *contpte_align_addr_ptep(unsigned long *start, static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { + unsigned int cont_bytes = sizeof(*ptep) * cont_ptes_mm(mm); + /* * Unfold any partially covered contpte block at the beginning and end * of the range. */ - if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + if (!IS_ALIGNED((unsigned long)ptep, cont_bytes) || nr < cont_ptes_mm(mm)) contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - if (ptep + nr != contpte_align_down(ptep + nr)) { - unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); + /* The exclusive end can lie beyond the allocated page-table backing. */ + if (!IS_ALIGNED((unsigned long)(ptep + nr), cont_bytes)) { + unsigned long last_addr = addr + mm_page_size(mm) * (nr - 1); pte_t *last_ptep = ptep + nr - 1; contpte_try_unfold(mm, last_addr, last_ptep, @@ -75,10 +97,11 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, int i; start_ptep = ptep = contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); + start_addr = addr = ALIGN_DOWN(addr, cont_pte_size_mm(mm)); + pte = phys_pte_mm(mm, ALIGN_DOWN(pte_phys_mm(mm, pte), cont_pte_size_mm(mm)), + pte_pgprot_mm(mm, pte)); - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { + for (i = 0; i < cont_ptes_mm(mm); i++, ptep++, addr += mm_page_size(mm)) { pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); if (pte_dirty(ptent)) @@ -225,10 +248,10 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, */ if (!system_supports_bbml2_noabort()) - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, 3, + __flush_tlb_range(&vma, start_addr, addr, mm_page_size(mm), 3, TLBF_NOWALKCACHE); - __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); + __set_ptes(mm, start_addr, start_ptep, pte, cont_ptes_mm(mm)); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, @@ -269,24 +292,26 @@ void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, page = pte_page(pte); folio = page_folio(page); folio_start = addr - (page - &folio->page) * PAGE_SIZE; + if (arm64_mm_alt_granule(mm)) + folio_start -= pte_phys_mm(mm, pte) & ~PAGE_MASK; folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; - cont_start = ALIGN_DOWN(addr, CONT_PTE_SIZE); - cont_end = cont_start + CONT_PTE_SIZE; + cont_start = ALIGN_DOWN(addr, cont_pte_size_mm(mm)); + cont_end = cont_start + cont_pte_size_mm(mm); if (folio_start > cont_start || folio_end < cont_end) return; - pfn = ALIGN_DOWN(pte_pfn(pte), CONT_PTES); - prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); - expected_pte = pfn_pte(pfn, prot); + pfn = ALIGN_DOWN(pte_phys_mm(mm, pte) >> mm_page_shift(mm), cont_ptes_mm(mm)); + prot = pte_pgprot_mm(mm, pte_mkold(pte_mkclean(pte))); + expected_pte = phys_pte_mm(mm, pfn << mm_page_shift(mm), prot); orig_ptep = ptep; ptep = contpte_align_down(ptep); - for (i = 0; i < CONT_PTES; i++) { + for (i = 0; i < cont_ptes_mm(mm); i++) { subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); if (!pte_same(subpte, expected_pte)) return; - expected_pte = pte_advance_pfn(expected_pte, 1); + expected_pte = pte_advance_pfn_mm(mm, expected_pte, 1); ptep++; } @@ -319,17 +344,18 @@ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) * feet. */ + unsigned int nr = contpte_nr(ptep); pte_t pte; int i; ptep = contpte_align_down(ptep); - for (i = 0; i < CONT_PTES; i++, ptep++) { + for (i = 0; i < nr; i++, ptep++) { pte = __ptep_get(ptep); if (pte_dirty(pte)) { orig_pte = pte_mkdirty(orig_pte); - for (; i < CONT_PTES; i++, ptep++) { + for (; i < nr; i++, ptep++) { pte = __ptep_get(ptep); if (pte_young(pte)) { orig_pte = pte_mkyoung(orig_pte); @@ -343,7 +369,7 @@ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) orig_pte = pte_mkyoung(orig_pte); i++; ptep++; - for (; i < CONT_PTES; i++, ptep++) { + for (; i < nr; i++, ptep++) { pte = __ptep_get(ptep); if (pte_dirty(pte)) { orig_pte = pte_mkdirty(orig_pte); @@ -359,11 +385,12 @@ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) EXPORT_SYMBOL_GPL(contpte_ptep_get); static inline bool contpte_is_consistent(pte_t pte, unsigned long pfn, + unsigned int shift, pgprot_t orig_prot) { - pgprot_t prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + pgprot_t prot = contpte_prot(pte_mkold(pte_mkclean(pte)), shift); - return pte_valid_cont(pte) && pte_pfn(pte) == pfn && + return pte_valid_cont(pte) && (contpte_phys(pte, shift) >> shift) == pfn && pgprot_val(prot) == pgprot_val(orig_prot); } @@ -386,6 +413,8 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) * because it is not part of a contpte range. */ + unsigned int shift = arm64_ptep_page_shift(orig_ptep); + unsigned int nr = arm64_cont_ptes(shift); pgprot_t orig_prot; unsigned long pfn; pte_t orig_pte; @@ -399,22 +428,22 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) if (!pte_valid_cont(orig_pte)) return orig_pte; - orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + orig_prot = contpte_prot(pte_mkold(pte_mkclean(orig_pte)), shift); ptep = contpte_align_down(orig_ptep); - pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + pfn = (contpte_phys(orig_pte, shift) >> shift) - (orig_ptep - ptep); - for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + for (i = 0; i < nr; i++, ptep++, pfn++) { pte = __ptep_get(ptep); - if (!contpte_is_consistent(pte, pfn, orig_prot)) + if (!contpte_is_consistent(pte, pfn, shift, orig_prot)) goto retry; if (pte_dirty(pte)) { orig_pte = pte_mkdirty(orig_pte); - for (; i < CONT_PTES; i++, ptep++, pfn++) { + for (; i < nr; i++, ptep++, pfn++) { pte = __ptep_get(ptep); - if (!contpte_is_consistent(pte, pfn, orig_prot)) + if (!contpte_is_consistent(pte, pfn, shift, orig_prot)) goto retry; if (pte_young(pte)) { @@ -430,10 +459,10 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) i++; ptep++; pfn++; - for (; i < CONT_PTES; i++, ptep++, pfn++) { + for (; i < nr; i++, ptep++, pfn++) { pte = __ptep_get(ptep); - if (!contpte_is_consistent(pte, pfn, orig_prot)) + if (!contpte_is_consistent(pte, pfn, shift, orig_prot)) goto retry; if (pte_dirty(pte)) { @@ -468,16 +497,16 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, if (!mm_is_user(mm)) return __set_ptes(mm, addr, ptep, pte, nr); - end = addr + (nr << PAGE_SHIFT); - pfn = pte_pfn(pte); - prot = pte_pgprot(pte); + end = addr + (nr << mm_page_shift(mm)); + pfn = pte_phys_mm(mm, pte) >> mm_page_shift(mm); + prot = pte_pgprot_mm(mm, pte); do { - next = pte_cont_addr_end(addr, end); - nr = (next - addr) >> PAGE_SHIFT; - pte = pfn_pte(pfn, prot); + next = pte_cont_addr_end_mm(mm, addr, end); + nr = (next - addr) >> mm_page_shift(mm); + pte = phys_pte_mm(mm, pfn << mm_page_shift(mm), prot); - if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) + if (((addr | next | (pfn << mm_page_shift(mm))) & (cont_pte_size_mm(mm) - 1)) == 0) pte = pte_mkcont(pte); else pte = pte_mknoncont(pte); @@ -512,6 +541,8 @@ EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); bool contpte_test_and_clear_young_ptes(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned int nr) { + struct mm_struct *mm = vma->vm_mm; + /* * ptep_clear_flush_young() technically requires us to clear the access * flag for a _single_ pte. However, the core-mm code actually tracks @@ -524,11 +555,11 @@ bool contpte_test_and_clear_young_ptes(struct vm_area_struct *vma, * of the same large folio in a single VMA and a single page table. */ - unsigned long end = addr + nr * PAGE_SIZE; + unsigned long end = addr + nr * mm_page_size(mm); bool young = false; - ptep = contpte_align_addr_ptep(&addr, &end, ptep, nr); - for (; addr != end; ptep++, addr += PAGE_SIZE) + ptep = contpte_align_addr_ptep(vma->vm_mm, &addr, &end, ptep, nr); + for (; addr != end; ptep++, addr += mm_page_size(mm)) young |= __ptep_test_and_clear_young(vma, addr, ptep); return young; @@ -538,19 +569,21 @@ EXPORT_SYMBOL_GPL(contpte_test_and_clear_young_ptes); bool contpte_clear_flush_young_ptes(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned int nr) { + struct mm_struct *mm = vma->vm_mm; + bool young; young = contpte_test_and_clear_young_ptes(vma, addr, ptep, nr); if (young) { - unsigned long end = addr + nr * PAGE_SIZE; + unsigned long end = addr + nr * mm_page_size(mm); - contpte_align_addr_ptep(&addr, &end, ptep, nr); + contpte_align_addr_ptep(vma->vm_mm, &addr, &end, ptep, nr); /* * See comment in __ptep_clear_flush_young(); same rationale for * eliding the trailing DSB applies here. */ - __flush_tlb_range(vma, addr, end, PAGE_SIZE, 3, + __flush_tlb_range(vma, addr, end, mm_page_size(mm), 3, TLBF_NOWALKCACHE | TLBF_NOSYNC); } @@ -580,6 +613,17 @@ void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned int nr, cydp_t flags) { + struct mm_struct *mm = vma->vm_mm; + unsigned long start = addr; + unsigned long end = start + nr * mm_page_size(mm); + + /* Smaller leaves may be discarded independently within the folio. */ + if (mm_page_size(mm) < PAGE_SIZE && (flags & CYDP_CLEAR_DIRTY)) { + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + return; + } + /* * We can safely clear access/dirty without needing to unfold from * the architectures perspective, even when contpte is set. If the @@ -590,15 +634,13 @@ void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, * when it is covered by a single folio, we can get away with * clearing access/dirty for the whole block. */ - unsigned long start = addr; - unsigned long end = start + nr * PAGE_SIZE; - - ptep = contpte_align_addr_ptep(&start, &end, ptep, nr); - __clear_young_dirty_ptes(vma, start, ptep, (end - start) / PAGE_SIZE, flags); + ptep = contpte_align_addr_ptep(vma->vm_mm, &start, &end, ptep, nr); + __clear_young_dirty_ptes(vma, start, ptep, (end - start) >> mm_page_shift(mm), flags); } EXPORT_SYMBOL_GPL(contpte_clear_young_dirty_ptes); -static bool contpte_all_subptes_match_access_flags(pte_t *ptep, pte_t entry) +static bool contpte_all_subptes_match_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { pte_t *cont_ptep = contpte_align_down(ptep); /* @@ -609,7 +651,7 @@ static bool contpte_all_subptes_match_access_flags(pte_t *ptep, pte_t entry) pteval_t entry_cmp = pte_val(entry) & cmp_mask; int i; - for (i = 0; i < CONT_PTES; i++) { + for (i = 0; i < cont_ptes_mm(mm); i++) { pteval_t pte_cmp = pte_val(__ptep_get(cont_ptep + i)) & cmp_mask; if (pte_cmp != entry_cmp) @@ -623,6 +665,8 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) { + struct mm_struct *mm = vma->vm_mm; + unsigned long start_addr; pte_t orig_pte; int i; @@ -651,7 +695,7 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, * no-op, and when any sub-PTE mismatches, proceed to update the whole * range. */ - if (contpte_all_subptes_match_access_flags(ptep, entry)) + if (contpte_all_subptes_match_access_flags(mm, ptep, entry)) return 0; /* @@ -672,7 +716,7 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, * and instead flush the whole range at the end. */ ptep = contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + start_addr = addr = ALIGN_DOWN(addr, cont_pte_size_mm(mm)); /* * We are not advancing entry because __ptep_set_access_flags() @@ -680,13 +724,13 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, * for the whole contpte block and returned early, pte_same() * within __ptep_set_access_flags() is likely false. */ - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + for (i = 0; i < cont_ptes_mm(mm); i++, ptep++, addr += mm_page_size(mm)) __ptep_set_access_flags(vma, addr, ptep, entry, 0); if (dirty) __flush_tlb_range(vma, start_addr, - start_addr + CONT_PTE_SIZE, - PAGE_SIZE, 3, + start_addr + cont_pte_size_mm(mm), + mm_page_size(mm), 3, TLBF_NOWALKCACHE | TLBF_NOBROADCAST); } else { __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c index cd5912ba617b70..b81ccf42b22d19 100644 --- a/arch/arm64/mm/copypage.c +++ b/arch/arm64/mm/copypage.c @@ -8,6 +8,7 @@ #include #include +#include #include #include @@ -72,3 +73,54 @@ void copy_user_highpage(struct page *to, struct page *from, flush_dcache_page(to); } EXPORT_SYMBOL_GPL(copy_user_highpage); + +#ifdef CONFIG_MM_SUBPAGE +void clear_user_subpage_range(struct page *page, unsigned int offset, unsigned int size) +{ + void *addr = page_address(page) + offset; + + VM_BUG_ON(!is_power_of_2(size) || size < SZ_4K || size > PAGE_SIZE); + VM_BUG_ON(!IS_ALIGNED(offset, size) || offset > PAGE_SIZE - size); + memset(addr, 0, size); + if (system_supports_mte() && page_mte_tagged(page)) + mte_clear_subpage_tags_range(addr, size); + flush_dcache_page(page); +} + +int copy_user_subpage_range(struct page *to, unsigned int to_offset, + struct page *from, unsigned int from_offset, unsigned int size) +{ + void *kto = page_address(to), *kfrom = page_address(from); + + VM_BUG_ON(!is_power_of_2(size) || size < SZ_4K || size > PAGE_SIZE); + VM_BUG_ON(!IS_ALIGNED(to_offset, size) || to_offset > PAGE_SIZE - size); + VM_BUG_ON(!IS_ALIGNED(from_offset, size) || from_offset > PAGE_SIZE - size); + if (PageHWPoison(from) || copy_mc_to_kernel(kto + to_offset, kfrom + from_offset, size)) + return -EHWPOISON; + if (kasan_hw_tags_enabled()) + page_kasan_tag_reset(to); + if (system_supports_mte()) { + if (page_mte_tagged(from)) { + if (try_page_mte_tagging(to)) + mte_clear_page_tags(kto); + mte_copy_subpage_tags_range(kto + to_offset, kfrom + from_offset, size); + set_page_mte_tagged(to); + } else if (page_mte_tagged(to)) { + mte_clear_subpage_tags_range(kto + to_offset, size); + } + } + flush_dcache_page(to); + return 0; +} +/* Existing 4K fault and ownership callers keep their fixed-size contract. */ +void clear_user_subpage(struct page *page, unsigned int offset) +{ + clear_user_subpage_range(page, offset, SZ_4K); +} + +int copy_user_subpage(struct page *to, unsigned int to_offset, + struct page *from, unsigned int from_offset) +{ + return copy_user_subpage_range(to, to_offset, from, from_offset, SZ_4K); +} +#endif diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 739800835920c1..6a37063417c4ce 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -150,7 +150,7 @@ static void show_pte(unsigned long addr) } pr_alert("%s pgtable: %luk pages, %llu-bit VAs, pgdp=%016lx\n", - mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, + mm == &init_mm ? "swapper" : "user", mm_page_size(mm) / SZ_1K, vabits_actual, mm_to_pgd_phys(mm)); pgdp = pgd_offset(mm, addr); pgd = READ_ONCE(*pgdp); @@ -165,25 +165,25 @@ static void show_pte(unsigned long addr) if (pgd_none(pgd) || pgd_bad(pgd)) break; - p4dp = p4d_offset(pgdp, addr); + p4dp = p4d_offset_mm(mm, pgdp, addr); p4d = READ_ONCE(*p4dp); pr_cont(", p4d=%016llx", p4d_val(p4d)); - if (p4d_none(p4d) || p4d_bad(p4d)) + if (p4d_none_mm(mm, p4d) || p4d_bad_mm(mm, p4d)) break; - pudp = pud_offset(p4dp, addr); + pudp = pud_offset_mm(mm, p4dp, addr); pud = READ_ONCE(*pudp); pr_cont(", pud=%016llx", pud_val(pud)); if (pud_none(pud) || pud_bad(pud)) break; - pmdp = pmd_offset(pudp, addr); + pmdp = pmd_offset_mm(mm, pudp, addr); pmd = READ_ONCE(*pmdp); pr_cont(", pmd=%016llx", pmd_val(pmd)); if (pmd_none(pmd) || pmd_bad(pmd)) break; - ptep = pte_offset_map(pmdp, addr); + ptep = pte_offset_map_mm(mm, pmdp, addr); if (!ptep) break; @@ -241,19 +241,15 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct *vma, * flush_tlb_fix_spurious_fault(). */ if (dirty) { - switch (pgsize) { - case PAGE_SIZE: + if (pgsize == mm_page_size(vma->vm_mm)) { level = 3; - break; - case PMD_SIZE: + } else if (pgsize == pmd_size_mm(vma->vm_mm)) { level = 2; - break; #ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: + } else if (pgsize == pud_size_mm(vma->vm_mm)) { level = 1; - break; #endif - default: + } else { level = TLBI_TTL_UNKNOWN; WARN_ON(1); } @@ -1002,8 +998,8 @@ NOKPROBE_SYMBOL(do_sp_pc_abort); /* * Used during anonymous page fault handling. */ -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) +struct folio *vma_alloc_zeroed_movable_folio_order(struct vm_area_struct *vma, + unsigned long vaddr, unsigned int order) { gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO; @@ -1015,7 +1011,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, if (vma->vm_flags & VM_MTE) flags |= __GFP_ZEROTAGS; - return vma_alloc_folio(flags, 0, vma, vaddr); + return vma_alloc_folio(flags, order, vma, vaddr); } bool tag_clear_highpages(struct page *page, int numpages, bool clear_pages) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c5c5425791da72..b670d27a898cc3 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -77,7 +77,7 @@ static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr, p4d_t p4d = READ_ONCE(*p4dp); pud_t *pudp; - if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) && + if (pgtable_l4_enabled() && !p4d_none(p4d) && p4d_page_paddr(p4d) != __pa_symbol(bm_pud)) { /* * We only end up here if the kernel mapping and the fixmap diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 97987f850a33c3..1e5f314b542d5c 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -368,8 +368,9 @@ void __init arch_mm_preinit(void) * Selected page table levels should match when derived from * scratch using the virtual address range and page size. */ - BUILD_BUG_ON(ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS) != - CONFIG_PGTABLE_LEVELS); + BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS != + (IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) ? 4 : + ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS))); if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) { extern int sysctl_overcommit_memory; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 09fd9da4eeed41..976d115d72f3f2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -2253,8 +2253,8 @@ pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, * in cases where cpu is affected with errata #2645198. */ if (pte_accessible(vma->vm_mm, pte) && pte_user_exec(pte)) - __flush_tlb_range(vma, addr, nr * PAGE_SIZE, - PAGE_SIZE, 3, TLBF_NOWALKCACHE); + __flush_tlb_range(vma, addr, addr + nr * mm_page_size(vma->vm_mm), + mm_page_size(vma->vm_mm), 3, TLBF_NOWALKCACHE); } return pte; diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index bf5110b91e2faa..3b361b95322e1b 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -16,14 +16,19 @@ #include static struct kmem_cache *pgd_cache __ro_after_init; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL +static struct kmem_cache *user4k_pgd_cache __ro_after_init; +static struct kmem_cache *user16k_pgd_cache __ro_after_init; +static struct kmem_cache *user64k_pgd_cache __ro_after_init; +#endif static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; - if (CONFIG_PGTABLE_LEVELS == 4) + if (ARM64_NATIVE_PGTABLE_LEVELS == 4) return !pgtable_l4_enabled(); - if (CONFIG_PGTABLE_LEVELS == 5) + if (ARM64_NATIVE_PGTABLE_LEVELS == 5) return !pgtable_l5_enabled(); return false; } @@ -32,6 +37,14 @@ pgd_t *pgd_alloc(struct mm_struct *mm) { gfp_t gfp = GFP_PGTABLE_USER; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (arm64_context_user4k(&mm->context)) + return kmem_cache_alloc(user4k_pgd_cache, gfp); + if (arm64_mm_alt_granule(mm) && mm_page_shift(mm) == 14) + return kmem_cache_alloc(user16k_pgd_cache, gfp); + if (arm64_mm_alt_granule(mm) && mm_page_shift(mm) == 16) + return kmem_cache_alloc(user64k_pgd_cache, gfp); +#endif if (pgdir_is_page_size()) return __pgd_alloc(mm, 0); else @@ -40,6 +53,20 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (arm64_context_user4k(&mm->context)) { + kmem_cache_free(user4k_pgd_cache, pgd); + return; + } + if (arm64_mm_alt_granule(mm) && mm_page_shift(mm) == 14) { + kmem_cache_free(user16k_pgd_cache, pgd); + return; + } + if (arm64_mm_alt_granule(mm) && mm_page_shift(mm) == 16) { + kmem_cache_free(user64k_pgd_cache, pgd); + return; + } +#endif if (pgdir_is_page_size()) __pgd_free(mm, pgd); else @@ -48,6 +75,16 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) void __init pgtable_cache_init(void) { +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (PAGE_SHIFT < 16) + user64k_pgd_cache = kmem_cache_create("user64k_pgd_cache", SZ_64K, SZ_64K, + SLAB_PANIC, NULL); + if (PAGE_SHIFT != 14) + user16k_pgd_cache = kmem_cache_create("user16k_pgd_cache", SZ_16K, SZ_16K, + SLAB_PANIC, NULL); + user4k_pgd_cache = kmem_cache_create("user4k_pgd_cache", SZ_4K, SZ_4K, + SLAB_PANIC, NULL); +#endif if (pgdir_is_page_size()) return; diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 22866b49be3720..7f947589a3b4d7 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -291,7 +291,7 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) cbnz cpu, __idmap_kpti_secondary -#if CONFIG_PGTABLE_LEVELS > 4 +#if ARM64_NATIVE_PGTABLE_LEVELS > 4 stp x29, x30, [sp, #-32]! mov x29, sp stp x19, x20, [sp, #16] @@ -352,7 +352,7 @@ alternative_else_nop_endif /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] -#if CONFIG_PGTABLE_LEVELS > 4 +#if ARM64_NATIVE_PGTABLE_LEVELS > 4 ldp x19, x20, [sp, #16] ldp x29, x30, [sp], #32 #endif @@ -360,39 +360,39 @@ alternative_else_nop_endif .Lderef_pgd: /* P4D */ - .if CONFIG_PGTABLE_LEVELS > 4 + .if ARM64_NATIVE_PGTABLE_LEVELS > 4 p4d .req x30 pte_to_phys cur_p4dp, pgd kpti_map_pgtbl p4d, 0 kpti_mk_tbl_ng p4d, PTRS_PER_P4D b .Lnext_pgd - .else /* CONFIG_PGTABLE_LEVELS <= 4 */ + .else /* ARM64_NATIVE_PGTABLE_LEVELS <= 4 */ p4d .req pgd .set .Lnext_p4d, .Lnext_pgd .endif .Lderef_p4d: /* PUD */ - .if CONFIG_PGTABLE_LEVELS > 3 + .if ARM64_NATIVE_PGTABLE_LEVELS > 3 pud .req x10 pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD b .Lnext_p4d - .else /* CONFIG_PGTABLE_LEVELS <= 3 */ + .else /* ARM64_NATIVE_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd .endif .Lderef_pud: /* PMD */ - .if CONFIG_PGTABLE_LEVELS > 2 + .if ARM64_NATIVE_PGTABLE_LEVELS > 2 pmd .req x13 pte_to_phys cur_pmdp, pud kpti_map_pgtbl pmd, 2 kpti_mk_tbl_ng pmd, PTRS_PER_PMD b .Lnext_pud - .else /* CONFIG_PGTABLE_LEVELS <= 2 */ + .else /* ARM64_NATIVE_PGTABLE_LEVELS <= 2 */ pmd .req pgd .set .Lnext_pmd, .Lnext_pgd .endif diff --git a/arch/arm64/mm/user4k-data-test.c b/arch/arm64/mm/user4k-data-test.c new file mode 100644 index 00000000000000..76656736074b1c --- /dev/null +++ b/arch/arm64/mm/user4k-data-test.c @@ -0,0 +1,1907 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Backing ownership tests; these slots are not fault/rmap-owned mappings. */ +#include +#include +#include +#include +#include +#include +#include +#ifdef CONFIG_NET +#include +#if IS_ENABLED(CONFIG_NET_SOCK_MSG) && IS_ENABLED(CONFIG_INET) +#include +#endif +#ifdef CONFIG_INET +#include +#include +#endif +#include +#endif +#include +#include + +#define DATA_SLOTS MM_SUBPAGES_PER_PAGE + +struct data_fixture { + struct mm_subpage_pool *pool; + struct mm_subpage *held[2 * DATA_SLOTS + 2]; +}; + +static void data_fixture_free(void *arg) +{ + struct data_fixture *fixture = arg; + unsigned int i; + + if (fixture->pool) + mm_subpage_pool_close(fixture->pool); + for (i = 0; i < ARRAY_SIZE(fixture->held); i++) + if (fixture->held[i]) + mm_subpage_put(fixture->held[i]); + if (fixture->pool) + mm_subpage_pool_put(fixture->pool); +} + +static struct data_fixture *data_fixture_create_granule(struct kunit *test, unsigned int shift) +{ + struct data_fixture *fixture = kunit_kzalloc(test, sizeof(*fixture), GFP_KERNEL); + + if (!fixture) + return NULL; + fixture->pool = mm_subpage_pool_create_granule(GFP_KERNEL, shift); + if (!fixture->pool) + return NULL; + if (kunit_add_action_or_reset(test, data_fixture_free, fixture)) + return NULL; + return fixture; +} + +static struct data_fixture *data_fixture_create(struct kunit *test) +{ + return data_fixture_create_granule(test, MM_SUBPAGE_SHIFT); +} + +static void put_held(struct data_fixture *fixture, unsigned int index) +{ + mm_subpage_put(fixture->held[index]); + fixture->held[index] = NULL; +} + +static struct mm_subpage *hold_alloc(struct data_fixture *fixture, unsigned int index) +{ + struct mm_subpage *slot = mm_subpage_alloc(fixture->pool); + + if (!IS_ERR(slot)) + fixture->held[index] = slot; + return slot; +} + +static int supply_backing(struct mm_subpage_pool *pool) +{ + struct folio *folio = folio_alloc(GFP_KERNEL, 0); + int ret; + + if (!folio) + return -ENOMEM; + /* Check that stale allocation contents never become exposed. */ + memset(folio_address(folio), 0xcc, PAGE_SIZE); + ret = mm_subpage_pool_add_folio(pool, folio, GFP_KERNEL); + if (ret) + folio_put(folio); + return ret; +} + +static int supply_backing_at(struct mm_subpage_pool *pool, unsigned int offset) +{ + struct folio *folio = folio_alloc(GFP_KERNEL, 0); + int ret; + + if (!folio) + return -ENOMEM; + ret = mm_subpage_pool_add_folio_at(pool, folio, offset, GFP_KERNEL); + if (ret) + folio_put(folio); + return ret; +} + +static struct mm_subpage *hold_alloc_at(struct data_fixture *fixture, unsigned int index, + unsigned int offset) +{ + struct mm_subpage *slot = mm_subpage_alloc_at(fixture->pool, offset); + + if (!IS_ERR(slot)) + fixture->held[index] = slot; + return slot; +} + +static void *slot_address(struct mm_subpage *slot) +{ + return folio_address(mm_subpage_folio(slot)) + mm_subpage_offset(slot); +} + +static void user4k_data_reuse_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct folio *folio; + struct mm_subpage *slot; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(fixture->pool)), -EAGAIN); + KUNIT_ASSERT_EQ(test, supply_backing(fixture->pool), 0); + KUNIT_EXPECT_EQ(test, supply_backing(fixture->pool), -EEXIST); + for (i = 0; i < DATA_SLOTS; i++) { + slot = hold_alloc(fixture, i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + folio = mm_subpage_folio(slot); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)i + 1); + KUNIT_EXPECT_PTR_EQ(test, folio, mm_subpage_folio(fixture->held[0])); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), i * SZ_4K); + KUNIT_EXPECT_EQ(test, mm_subpage_phys(slot), PFN_PHYS(folio_pfn(folio)) + i * SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0, SZ_4K), NULL); + memset(slot_address(slot), 0x10 + i, SZ_4K); + } + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 1UL); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(fixture->pool)), -EAGAIN); + /* A pin, alias or deferred-TLB holder keeps this slot unavailable. */ + fixture->held[DATA_SLOTS] = fixture->held[1]; + mm_subpage_get(fixture->held[DATA_SLOTS]); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)DATA_SLOTS + 1); + put_held(fixture, 1); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)DATA_SLOTS); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(fixture->pool)), -EAGAIN); + put_held(fixture, DATA_SLOTS); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)DATA_SLOTS - 1); + slot = hold_alloc(fixture, DATA_SLOTS + 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), (unsigned int)SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0, SZ_4K), NULL); + for (i = 0; i < DATA_SLOTS; i++) { + if (i != 1) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(fixture->held[i]), + 0x10 + i, SZ_4K), NULL); + } + mm_subpage_pool_close(fixture->pool); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(fixture->pool)), -ESHUTDOWN); + KUNIT_EXPECT_EQ(test, supply_backing(fixture->pool), -ESHUTDOWN); + for (i = 0; i < DATA_SLOTS; i++) + if (i != 1) + put_held(fixture, i); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 1UL); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + put_held(fixture, DATA_SLOTS + 1); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +static void user4k_data_copy_test(struct kunit *test) +{ + struct data_fixture *source = data_fixture_create(test); + struct data_fixture *dest = data_fixture_create(test); + struct mm_subpage *copy; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, source); + KUNIT_ASSERT_NOT_NULL(test, dest); + KUNIT_ASSERT_EQ(test, supply_backing(source->pool), 0); + KUNIT_ASSERT_EQ(test, supply_backing(dest->pool), 0); + for (i = 0; i < DATA_SLOTS; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc(source, i)); + memset(slot_address(source->held[i]), 0x50 + i, SZ_4K); + } + source->held[DATA_SLOTS] = source->held[1]; + mm_subpage_get(source->held[DATA_SLOTS]); + copy = mm_subpage_copy(dest->pool, source->held[1]); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy); + dest->held[0] = copy; + KUNIT_EXPECT_NE(test, mm_subpage_phys(copy), mm_subpage_phys(source->held[1])); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(copy), 0x51, SZ_4K), NULL); + memset(slot_address(copy), 0xee, SZ_4K); + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(source->held[i]), 0x50 + i, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_copy(source->pool, source->held[1])), -EAGAIN); + /* Copy into a free neighboring slot in the same physical owner. */ + put_held(source, 2); + copy = mm_subpage_copy(source->pool, source->held[1]); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy); + source->held[2] = copy; + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(copy), mm_subpage_folio(source->held[1])); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(copy), 2U * SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(copy), 0x51, SZ_4K), NULL); + memset(slot_address(copy), 0xdd, SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(source->held[1]), 0x51, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(source->held[3]), 0x53, SZ_4K), NULL); +} + +static void user4k_data_lookup_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct mm_subpage *slot, *found; + struct folio *folio; + phys_addr_t phys; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + folio = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, folio); + phys = PFN_PHYS(folio_pfn(folio)); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_get_from_phys(phys), NULL); + if (mm_subpage_pool_add_folio(fixture->pool, folio, GFP_KERNEL)) { + folio_put(folio); + KUNIT_FAIL(test, "failed to supply lookup backing"); + return; + } + /* A published owner does not expose uninitialised or unissued slots. */ + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_get_from_phys(phys + i * SZ_4K), NULL); + for (i = 0; i < DATA_SLOTS; i++) { + slot = hold_alloc(fixture, i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + memset(slot_address(slot), 0x90 + i, SZ_4K); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 2 * (int)i + 1); + if (!i) + KUNIT_EXPECT_EQ(test, mm_subpage_pool_add_folio(fixture->pool, folio, + GFP_KERNEL), -EEXIST); + found = mm_subpage_get_from_phys(phys + i * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, found); + fixture->held[i + DATA_SLOTS] = found; + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 2 * (int)i + 2); + KUNIT_EXPECT_PTR_EQ(test, found, slot); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_get_from_phys(phys + i * SZ_4K + 1), NULL); + } + /* A second pool must not zero or acquire already owned backing. */ + KUNIT_EXPECT_EQ(test, mm_subpage_pool_add_folio(fixture->pool, folio, GFP_KERNEL), + -EINVAL); + for (i = 0; i < DATA_SLOTS; i++) { + put_held(fixture, i); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(fixture->held[i + DATA_SLOTS]), + 0x90 + i, SZ_4K), NULL); + } + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(fixture->pool)), -EAGAIN); + put_held(fixture, DATA_SLOTS + 1); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_get_from_phys(phys + SZ_4K), NULL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc(fixture, 1)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(fixture->held[1]), 0, SZ_4K), NULL); + mm_subpage_pool_close(fixture->pool); + /* Keep the physical allocation stable while checking final unbinding. */ + folio_get(folio); + put_held(fixture, 1); + for (i = 0; i < DATA_SLOTS; i++) + if (i != 1) + put_held(fixture, i + DATA_SLOTS); + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_get_from_phys(phys + i * SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + folio_put(folio); +} + +static void user4k_data_exact_slot_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct mm_subpage *source, *copy, *reused; + struct folio *first; + static const unsigned int neighbors[] = { 0, SZ_4K, 3 * SZ_4K }; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_EXPECT_EQ(test, supply_backing_at(fixture->pool, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc_at(fixture->pool, PAGE_SIZE)), -EINVAL); + KUNIT_ASSERT_EQ(test, supply_backing_at(fixture->pool, SZ_8K), 0); + source = hold_alloc_at(fixture, 0, SZ_8K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, source); + first = mm_subpage_folio(source); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(source), (unsigned int)SZ_8K); + memset(slot_address(source), 0xa7, SZ_4K); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_copy_at(fixture->pool, source, 1)), -EINVAL); + /* Three free neighbors cannot satisfy a request for this occupied slot. */ + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_copy_at(fixture->pool, source, SZ_8K)), -EAGAIN); + KUNIT_EXPECT_EQ(test, supply_backing(fixture->pool), -EEXIST); + KUNIT_ASSERT_EQ(test, supply_backing_at(fixture->pool, SZ_8K), 0); + KUNIT_EXPECT_EQ(test, supply_backing_at(fixture->pool, SZ_8K), -EEXIST); + copy = mm_subpage_copy_at(fixture->pool, source, SZ_8K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy); + fixture->held[1] = copy; + KUNIT_EXPECT_PTR_NE(test, mm_subpage_folio(copy), first); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(copy), (unsigned int)SZ_8K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(copy), 0xa7, SZ_4K), NULL); + memset(slot_address(copy), 0x4b, SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(source), 0xa7, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 2UL); + for (i = 0; i < ARRAY_SIZE(neighbors); i++) { + struct mm_subpage *slot = hold_alloc_at(fixture, i + 2, neighbors[i]); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(slot), first); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), neighbors[i]); + memset(slot_address(slot), 0x30 + i, SZ_4K); + } + KUNIT_EXPECT_EQ(test, folio_ref_count(first), 4); + fixture->held[5] = source; + mm_subpage_get(source); + put_held(fixture, 0); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc_at(fixture->pool, SZ_8K)), -EAGAIN); + put_held(fixture, 5); + reused = hold_alloc_at(fixture, 6, SZ_8K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, reused); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(reused), first); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(reused), 0, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, folio_ref_count(first), 4); + for (i = 0; i < ARRAY_SIZE(neighbors); i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(fixture->held[i + 2]), + 0x30 + i, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(copy), 0x4b, SZ_4K), NULL); + mm_subpage_pool_close(fixture->pool); + for (i = 2; i <= 4; i++) + put_held(fixture, i); + put_held(fixture, 6); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 1UL); + put_held(fixture, 1); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +static void user4k_data_close_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct mm_subpage *slot; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_EQ(test, supply_backing(fixture->pool), 0); + /* Closing with no issued slots must release that entire unused owner. */ + mm_subpage_pool_close(fixture->pool); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); + fixture = data_fixture_create(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_EQ(test, supply_backing(fixture->pool), 0); + slot = hold_alloc(fixture, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + memset(slot_address(slot), 0xa7, SZ_4K); + mm_subpage_pool_close(fixture->pool); + mm_subpage_pool_put(fixture->pool); + fixture->pool = NULL; + /* The originating pool/mm reference is gone; a child's slot remains live. */ + fixture->held[1] = slot; + mm_subpage_get(slot); + put_held(fixture, 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0xa7, SZ_4K), NULL); + put_held(fixture, 1); +} + +struct deferred_slot { + struct rcu_head rcu; + struct mm_subpage *slot; + struct completion done; +}; + +static void deferred_slot_put(struct rcu_head *rcu) +{ + struct deferred_slot *pending = container_of(rcu, struct deferred_slot, rcu); + + mm_subpage_put(pending->slot); + complete(&pending->done); +} + +static void data_rcu_barrier(void *unused) +{ + rcu_barrier(); +} + +static void user4k_data_deferred_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct deferred_slot *pending = kunit_kzalloc(test, sizeof(*pending), GFP_KERNEL); + struct mm_subpage *probe; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_NOT_NULL(test, pending); + KUNIT_ASSERT_EQ(test, supply_backing(fixture->pool), 0); + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc(fixture, i)); + init_completion(&pending->done); + pending->slot = fixture->held[0]; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, data_rcu_barrier, NULL), 0); + mm_subpage_get(pending->slot); + rcu_read_lock(); + call_rcu(&pending->rcu, deferred_slot_put); + put_held(fixture, 0); + /* The delayed reference must prevent recycling before the grace period. */ + probe = mm_subpage_alloc(fixture->pool); + for (i = 1; i < DATA_SLOTS; i++) + put_held(fixture, i); + rcu_read_unlock(); + if (!IS_ERR(probe)) + fixture->held[DATA_SLOTS] = probe; + KUNIT_EXPECT_EQ(test, PTR_ERR(probe), -EAGAIN); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&pending->done, 5 * HZ), 0UL); + /* The callback retired the final live slot and the whole native owner. */ + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +static void user4k_data_backing_reject_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct folio *folio; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + folio = folio_alloc(GFP_KERNEL, 1); + KUNIT_ASSERT_NOT_NULL(test, folio); + ret = mm_subpage_pool_add_folio(fixture->pool, folio, GFP_KERNEL); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + if (ret) + folio_put(folio); + folio = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_get(folio); + ret = mm_subpage_pool_add_folio(fixture->pool, folio, GFP_KERNEL); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 2); + folio_put(folio); + if (ret) + folio_put(folio); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +struct data_worker { + struct mm_subpage_pool *pool; + struct task_struct *task; + struct completion ready, go, done; + unsigned int id, checks; + int error; +}; + +static struct mm_subpage *alloc_with_backing(struct mm_subpage_pool *pool) +{ + unsigned int tries; + + for (tries = 0; tries < 128; tries++) { + struct mm_subpage *slot = mm_subpage_alloc(pool); + int ret; + + if (!IS_ERR(slot) || PTR_ERR(slot) != -EAGAIN) + return slot; + ret = supply_backing(pool); + if (ret && ret != -EEXIST) + return ERR_PTR(ret); + cond_resched(); + } + return ERR_PTR(-EAGAIN); +} + +static int data_worker_run(void *arg) +{ + struct data_worker *worker = arg; + struct mm_subpage *slot = NULL; + unsigned int round; + u8 pattern = 0; + + for (round = 0; round < 64; round++) { + slot = alloc_with_backing(worker->pool); + if (IS_ERR(slot)) { + worker->error = PTR_ERR(slot); + slot = NULL; + break; + } + pattern = 1 + round + worker->id * 64; + memset(slot_address(slot), pattern, SZ_4K); + mm_subpage_get(slot); + mm_subpage_put(slot); + schedule_timeout_uninterruptible(1); + if (memchr_inv(slot_address(slot), pattern, SZ_4K)) { + worker->error = -EIO; + break; + } + worker->checks++; + if (round != 63) { + mm_subpage_put(slot); + slot = NULL; + } + } + complete(&worker->ready); + wait_for_completion(&worker->go); + if (slot) { + if (memchr_inv(slot_address(slot), pattern, SZ_4K)) + worker->error = -EIO; + mm_subpage_put(slot); + } + complete(&worker->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void data_worker_stop(void *arg) +{ + struct data_worker *worker = arg; + + complete_all(&worker->go); + kthread_stop(worker->task); +} + +static void user4k_data_concurrent_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct data_worker *workers = kunit_kcalloc(test, 2, sizeof(*workers), GFP_KERNEL); + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_NOT_NULL(test, workers); + for (i = 0; i < 2; i++) { + workers[i].pool = fixture->pool; + workers[i].id = i; + init_completion(&workers[i].ready); + init_completion(&workers[i].go); + init_completion(&workers[i].done); + workers[i].task = kthread_run(data_worker_run, &workers[i], "user4k-data-%u", i); + KUNIT_ASSERT_FALSE(test, IS_ERR(workers[i].task)); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, data_worker_stop, &workers[i]), 0); + } + for (i = 0; i < 2; i++) + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].ready, 10 * HZ), 0UL); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 1UL); + mm_subpage_pool_close(fixture->pool); + for (i = 0; i < 2; i++) + complete(&workers[i].go); + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].done, 10 * HZ), 0UL); + KUNIT_EXPECT_EQ(test, workers[i].error, 0); + KUNIT_EXPECT_EQ(test, workers[i].checks, 64U); + } + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +struct lookup_worker { + struct task_struct *task; + struct completion observed; + phys_addr_t phys; + unsigned long checks; + int error; +}; + +static int lookup_worker_run(void *arg) +{ + struct lookup_worker *worker = arg; + bool observed = false; + + while (!kthread_should_stop()) { + struct mm_subpage *slot = mm_subpage_get_from_phys(worker->phys); + + if (slot) { + unsigned char *data = slot_address(slot); + unsigned char value = READ_ONCE(*data); + + if (mm_subpage_phys(slot) != worker->phys || + (value != 0x5a && value != 0xab) || + memchr_inv(data, value, SZ_4K)) + worker->error = -EINVAL; + worker->checks++; + mm_subpage_put(slot); + if (!observed) { + complete(&worker->observed); + observed = true; + } + if (worker->error) + break; + } + cond_resched(); + } + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void lookup_worker_stop(void *arg) +{ + struct lookup_worker *worker = arg; + + if (worker->task) { + kthread_stop(worker->task); + worker->task = NULL; + } +} + +static void user4k_data_lookup_race_test(struct kunit *test) +{ + struct data_fixture *fixture = data_fixture_create(test); + struct lookup_worker *worker = kunit_kzalloc(test, sizeof(*worker), GFP_KERNEL); + unsigned int copied = 0, i; + unsigned long deadline; + struct folio *folio; + struct mm_subpage *found; + + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_NOT_NULL(test, worker); + KUNIT_ASSERT_EQ(test, supply_backing(fixture->pool), 0); + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc(fixture, i)); + memset(slot_address(fixture->held[0]), 0x5a, SZ_4K); + memset(slot_address(fixture->held[1]), 0xab, SZ_4K); + worker->phys = mm_subpage_phys(fixture->held[2]); + put_held(fixture, 2); + init_completion(&worker->observed); + worker->task = kthread_run(lookup_worker_run, worker, "user4k-lookup"); + KUNIT_ASSERT_FALSE(test, IS_ERR(worker->task)); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, lookup_worker_stop, worker), 0); + if (num_online_cpus() > 1) { + unsigned int cpu = cpumask_next(cpumask_first(cpu_online_mask), cpu_online_mask); + + KUNIT_ASSERT_EQ(test, set_cpus_allowed_ptr(worker->task, cpumask_of(cpu)), 0); + } + /* Alternate complete contents; a reader must never see a partial copy. */ + deadline = jiffies + 30 * HZ; + while (copied < 2048 && time_before(jiffies, deadline)) { + struct mm_subpage *dest = mm_subpage_copy(fixture->pool, + fixture->held[copied & 1]); + bool observed = true; + phys_addr_t phys; + + if (PTR_ERR_OR_ZERO(dest) == -EAGAIN) { + /* Let a scheduled-out lookup holder retire its reference. */ + schedule_timeout_uninterruptible(1); + continue; + } + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, dest); + phys = mm_subpage_phys(dest); + if (!copied) + observed = wait_for_completion_timeout(&worker->observed, 5 * HZ) != 0; + mm_subpage_put(dest); + KUNIT_ASSERT_TRUE(test, observed); + KUNIT_ASSERT_EQ(test, phys, worker->phys); + copied++; + cond_resched(); + } + /* Retire the owner while lookup is still running, then verify unbinding. */ + folio = mm_subpage_folio(fixture->held[0]); + folio_get(folio); + mm_subpage_pool_close(fixture->pool); + for (i = 0; i < DATA_SLOTS; i++) + if (i != 2) + put_held(fixture, i); + lookup_worker_stop(worker); + found = mm_subpage_get_from_phys(worker->phys); + KUNIT_EXPECT_PTR_EQ(test, found, NULL); + if (found) + mm_subpage_put(found); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + folio_put(folio); + KUNIT_EXPECT_EQ(test, worker->error, 0); + KUNIT_EXPECT_GT(test, worker->checks, 0UL); + KUNIT_EXPECT_EQ(test, copied, 2048U); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(fixture->pool), 0UL); +} + +static void user4k_data_swapcache_reuse_test(struct kunit *test) +{ + struct data_fixture *f = data_fixture_create(test); + struct mm_subpage *attempt; + struct folio *folio; + unsigned int i; + bool intact; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * MM_SUBPAGE_SIZE)); + memset(slot_address(f->held[i]), 0x60 + i, MM_SUBPAGE_SIZE); + } + folio = mm_subpage_folio(f->held[0]); + put_held(f, 1); + /* Synthetic cache membership only; no swap entry or I/O is fabricated. */ + __folio_set_swapbacked(folio); + folio_set_swapcache(folio); + attempt = mm_subpage_alloc_at(f->pool, MM_SUBPAGE_SIZE); + intact = !memchr_inv(folio_address(folio) + MM_SUBPAGE_SIZE, 0x61, MM_SUBPAGE_SIZE); + folio_clear_swapcache(folio); + __folio_clear_swapbacked(folio); + /* Restore the real state before assertions can unwind the fixture. */ + if (!IS_ERR(attempt)) + f->held[1] = attempt; + KUNIT_EXPECT_TRUE(test, intact); + KUNIT_ASSERT_EQ(test, PTR_ERR_OR_ZERO(attempt), -EAGAIN); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, MM_SUBPAGE_SIZE)); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[i]), + i == 1 ? 0 : 0x60 + i, MM_SUBPAGE_SIZE), NULL); +} + +static void restore_folio_free(void *arg) +{ + struct folio *folio = arg; + + /* Data-only fixtures never create a real swap entry or mapping. */ + folio_clear_swapcache(folio); + __folio_clear_swapbacked(folio); + folio_put(folio); +} + +static struct mm_subpage *restore_test_slot_granule(struct folio *folio, unsigned int offset, unsigned int shift) +{ + struct mm_subpage *slot; + + folio_lock(folio); + __folio_set_swapbacked(folio); + folio_set_swapcache(folio); + slot = mm_subpage_restore_granule(folio, offset, shift, GFP_KERNEL); + folio_clear_swapcache(folio); + __folio_clear_swapbacked(folio); + folio_unlock(folio); + return slot; +} + +static struct mm_subpage *restore_test_slot(struct folio *folio, unsigned int offset) +{ + return restore_test_slot_granule(folio, offset, MM_SUBPAGE_SHIFT); +} + +static void test_data_restore_granule(struct kunit *test, unsigned int shift) +{ + struct data_fixture *f = data_fixture_create_granule(test, shift); + struct folio *folio; + struct mm_subpage *slot; + unsigned int i, size = 1U << shift; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * size)); + memset(slot_address(f->held[i]), 0x70 + i, size); + } + folio = mm_subpage_folio(f->held[0]); + folio_get(folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, restore_folio_free, folio), 0); + put_held(f, 1); + slot = restore_test_slot_granule(folio, size, shift); + if (!IS_ERR(slot)) + f->held[1] = slot; + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), (unsigned int)size); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 5); + slot = restore_test_slot_granule(folio, 0, shift); + if (!IS_ERR(slot)) + f->held[4] = slot; + KUNIT_ASSERT_PTR_EQ(test, slot, f->held[0]); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 6); + put_held(f, 4); + /* Retire the entire old owner, retaining only a native backing reference. */ + for (i = 0; i < 4; i++) + put_held(f, i); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(f->pool), 0UL); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + for (i = 0; i < 4; i++) { + slot = restore_test_slot_granule(folio, i * size, shift); + if (!IS_ERR(slot)) + f->held[i] = slot; + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(slot), folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0x70 + i, + size), NULL); + } + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 5); + KUNIT_EXPECT_EQ(test, PTR_ERR(restore_test_slot_granule(folio, 1, shift)), -EINVAL); + KUNIT_EXPECT_EQ(test, PTR_ERR(restore_test_slot_granule(folio, PAGE_SIZE, shift)), -EINVAL); + if (shift != MM_SUBPAGE_SHIFT) { + KUNIT_EXPECT_EQ(test, PTR_ERR(restore_test_slot(folio, 0)), -EINVAL); + KUNIT_EXPECT_EQ(test, PTR_ERR(restore_test_slot_granule(folio, SZ_4K, shift)), -EINVAL); + } + +} + +static void user4k_data_restore_test(struct kunit *test) +{ + test_data_restore_granule(test, MM_SUBPAGE_SHIFT); +} + +static void user16k_data_restore_test(struct kunit *test) +{ + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_data_restore_granule(test, 14); +} + +struct restore_worker { + struct folio *folio; + struct task_struct *task; + struct completion done; + unsigned int checks; + int error; +}; + +static int restore_worker_run(void *arg) +{ + struct restore_worker *w = arg; + unsigned long deadline = jiffies + 20 * HZ; + + while (!kthread_should_stop() && w->checks < 1024 && time_before(jiffies, deadline)) { + struct mm_subpage *slot; + + folio_lock(w->folio); + slot = mm_subpage_restore(w->folio, MM_SUBPAGE_SIZE, GFP_KERNEL); + folio_unlock(w->folio); + if (PTR_ERR_OR_ZERO(slot) == -EAGAIN) { + cond_resched(); + continue; + } + if (IS_ERR(slot)) { + w->error = PTR_ERR(slot); + break; + } + if (memchr_inv(slot_address(slot), 0x83, MM_SUBPAGE_SIZE)) + w->error = -EIO; + /* Retirement deliberately occurs without the folio lock. */ + mm_subpage_put(slot); + if (w->error) + break; + w->checks++; + cond_resched(); + } + complete(&w->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void restore_worker_stop(void *arg) +{ + struct restore_worker *w = arg; + + if (w->task) { + kthread_stop(w->task); + w->task = NULL; + } +} + +static void user4k_data_restore_race_test(struct kunit *test) +{ + struct folio *folio = folio_alloc(GFP_KERNEL, 0); + struct restore_worker *workers; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, restore_folio_free, folio), 0); + memset(folio_address(folio), 0x83, PAGE_SIZE); + __folio_mark_uptodate(folio); + __folio_set_swapbacked(folio); + folio_set_swapcache(folio); + workers = kunit_kcalloc(test, 2, sizeof(*workers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, workers); + for (i = 0; i < 2; i++) { + workers[i].folio = folio; + init_completion(&workers[i].done); + workers[i].task = kthread_run(restore_worker_run, &workers[i], "user4k-restore-%u", i); + if (IS_ERR(workers[i].task)) { + workers[i].task = NULL; + KUNIT_FAIL(test, "restore worker creation failed"); + return; + } + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, restore_worker_stop, &workers[i]), 0); + } + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].done, 25 * HZ), 0UL); + restore_worker_stop(&workers[i]); + KUNIT_EXPECT_EQ(test, workers[i].error, 0); + KUNIT_EXPECT_EQ(test, workers[i].checks, 1024U); + } + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); +} + +static void test_data_migration_granule(struct kunit *test, unsigned int shift) +{ + unsigned int mode, size = 1U << shift; + + /* Success, rollback, all PTEs discarded, and partial-folio success. */ + for (mode = 0; mode < 4; mode++) { + struct data_fixture *f = data_fixture_create_granule(test, shift); + struct folio *src, *dst, *target; + unsigned int i, nr = mode == 3 ? 3 : 4; + int ret, src_refs = 0, dst_refs = 0, blocked = 0; + bool unpublished = true, active = false; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < nr; i++) { + unsigned int index = mode == 3 && i ? i + 1 : i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, index * size)); + memset(slot_address(f->held[i]), 0x80 + index, size); + } + src = mm_subpage_folio(f->held[0]); + folio_get(src); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, restore_folio_free, src), 0); + dst = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, dst); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, restore_folio_free, dst), 0); + folio_lock(src); + folio_lock(dst); + ret = mm_subpage_migrate_prepare(src, dst); + if (!ret) { + struct mm_subpage *probe; + + src_refs = folio_ref_count(src); + dst_refs = folio_ref_count(dst); + active = mm_subpage_migrating(src) && mm_subpage_migrating(dst); + probe = mm_subpage_alloc_at(f->pool, size); + blocked = PTR_ERR_OR_ZERO(probe); + if (!IS_ERR(probe)) + mm_subpage_put(probe); + /* Emulate migration entries: their old mapping refs are gone. */ + for (i = 0; i < nr; i++) + put_held(f, i); + probe = mm_subpage_get_from_phys(PFN_PHYS(folio_pfn(src))); + unpublished &= !probe; + if (probe) + mm_subpage_put(probe); + probe = mm_subpage_get_from_phys(PFN_PHYS(folio_pfn(dst))); + unpublished &= !probe; + if (probe) + mm_subpage_put(probe); + if (mode != 1 && mode != 2) { + copy_user_highpage(&dst->page, &src->page, 0, NULL); + __folio_mark_uptodate(dst); + } + target = mode == 1 ? src : dst; + if (mode != 2) + for (i = 0; i < nr; i++) { + unsigned int index = mode == 3 && i ? i + 1 : i; + + f->held[i] = mm_subpage_migrate_restore(target, index * size); + } + mm_subpage_migrate_finish(src); + mm_subpage_migrate_finish(dst); + } + folio_unlock(dst); + folio_unlock(src); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, active); + KUNIT_EXPECT_TRUE(test, unpublished); + KUNIT_EXPECT_EQ(test, src_refs, (int)nr + 1); + KUNIT_EXPECT_EQ(test, dst_refs, 1); + KUNIT_EXPECT_EQ(test, blocked, -EAGAIN); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(f->pool), mode == 2 ? 0UL : 1UL); + KUNIT_EXPECT_EQ(test, folio_ref_count(src), mode == 1 ? (int)nr + 1 : 1); + KUNIT_EXPECT_EQ(test, folio_ref_count(dst), mode == 0 || mode == 3 ? (int)nr + 1 : 1); + if (mode != 2) + for (i = 0; i < nr; i++) { + unsigned int index = mode == 3 && i ? i + 1 : i; + + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(f->held[i]), mode == 1 ? src : dst); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[i]), + 0x80 + index, size), NULL); + } + if (mode == 3) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 3, size)); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(f->held[3]), dst); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[3]), 0, + size), NULL); + } + } +} + +static void user4k_data_migration_owner_test(struct kunit *test) +{ + test_data_migration_granule(test, MM_SUBPAGE_SHIFT); +} + +static void user16k_data_migration_owner_test(struct kunit *test) +{ + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_data_migration_granule(test, 14); +} + +#ifdef CONFIG_ARM64_MTE +static void test_slot_tags_set(void *addr, unsigned int tag) +{ + unsigned long value = (unsigned long)tag << MTE_TAG_SHIFT; + unsigned int offset; + + for (offset = 0; offset < MM_SUBPAGE_SIZE; offset += MTE_GRANULE_SIZE) + asm volatile(".arch_extension memtag\n\tstg %0, [%1]" + : : "r" (value), "r" (addr + offset) : "memory"); +} + +static bool test_slot_tags_equal(void *addr, unsigned int expected) +{ + unsigned int offset; + + for (offset = 0; offset < MM_SUBPAGE_SIZE; offset += MTE_GRANULE_SIZE) { + unsigned long value = 0; + + asm volatile(".arch_extension memtag\n\tldg %0, [%1]" + : "+r" (value) : "r" (addr + offset) : "memory"); + if (((value >> MTE_TAG_SHIFT) & 0xf) != expected) + return false; + } + return true; +} +#endif + +static void user4k_data_mte_test(struct kunit *test) +{ +#ifdef CONFIG_ARM64_MTE + struct data_fixture *f; + struct folio *folio; + unsigned int i; + + if (!system_supports_mte()) { + kunit_skip(test, "MTE is not available on this CPU"); + return; + } + f = data_fixture_create(test); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * MM_SUBPAGE_SIZE)); + folio = mm_subpage_folio(f->held[0]); + KUNIT_ASSERT_TRUE(test, try_page_mte_tagging(&folio->page)); + mte_clear_page_tags(folio_address(folio)); + for (i = 0; i < 4; i++) { + memset(slot_address(f->held[i]), 0x40 + i, MM_SUBPAGE_SIZE); + test_slot_tags_set(slot_address(f->held[i]), i + 1); + } + set_page_mte_tagged(&folio->page); + put_held(f, 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, MM_SUBPAGE_SIZE)); + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(slot_address(f->held[i]), i == 1 ? 0 : i + 1)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[i]), + i == 1 ? 0 : 0x40 + i, MM_SUBPAGE_SIZE), NULL); + } + put_held(f, 1); + f->held[1] = mm_subpage_copy_at(f->pool, f->held[0], MM_SUBPAGE_SIZE); + if (IS_ERR(f->held[1])) { + f->held[1] = NULL; + KUNIT_FAIL(test, "tagged slot copy failed"); + return; + } + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(slot_address(f->held[i]), i == 1 ? 1 : i + 1)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[i]), + i == 1 ? 0x40 : 0x40 + i, MM_SUBPAGE_SIZE), NULL); + } + /* A fresh destination must initialise tags outside the copied slot too. */ + KUNIT_ASSERT_EQ(test, supply_backing_at(f->pool, 3 * MM_SUBPAGE_SIZE), 0); + f->held[4] = mm_subpage_copy_at(f->pool, f->held[2], 3 * MM_SUBPAGE_SIZE); + if (IS_ERR(f->held[4])) { + f->held[4] = NULL; + KUNIT_FAIL(test, "fresh tagged slot copy failed"); + return; + } + folio = mm_subpage_folio(f->held[4]); + KUNIT_EXPECT_TRUE(test, page_mte_tagged(&folio->page)); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(folio_address(folio) + + i * MM_SUBPAGE_SIZE, i == 3 ? 3 : 0)); +#else + kunit_skip(test, "MTE is not configured"); +#endif +} + +static void extent_folio_put(void *arg) +{ + folio_put(arg); +} + +static struct folio *compare_folio_alloc(struct kunit *test) +{ + struct folio *folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 0); + + if (folio && kunit_add_action_or_reset(test, extent_folio_put, folio)) + return NULL; + return folio; +} + +static void user_page_range_compare_test(struct kunit *test) +{ + const unsigned int sizes[] = { SZ_4K, SZ_16K, PAGE_SIZE }; + struct folio *a = compare_folio_alloc(test), *b = compare_folio_alloc(test); + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + unsigned int len = sizes[i], offset = PAGE_SIZE - len; + char *from = folio_address(a), *to = folio_address(b); + + if (len > PAGE_SIZE) + continue; + memset(from, 0x11, PAGE_SIZE); + memset(to, 0x22, PAGE_SIZE); + memset(from, 0x5a, len); + memset(to + offset, 0x5a, len); + KUNIT_EXPECT_EQ(test, memcmp_pages_range(&a->page, 0, &b->page, offset, len), 0); + KUNIT_EXPECT_EQ(test, !memcmp_pages(&a->page, &b->page), len == PAGE_SIZE); + to[PAGE_SIZE - 1] ^= 1; + KUNIT_EXPECT_NE(test, memcmp_pages_range(&a->page, 0, &b->page, offset, len), 0); + } +} + +static void user_page_tagged_range_compare_test(struct kunit *test) +{ +#ifdef CONFIG_ARM64_MTE + struct folio *a, *b; + unsigned int len; + + if (!system_supports_mte()) { + kunit_skip(test, "MTE is not available on this CPU"); + return; + } + a = compare_folio_alloc(test); + b = compare_folio_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + KUNIT_ASSERT_EQ(test, memcmp_pages(&a->page, &b->page), 0); + KUNIT_ASSERT_TRUE(test, try_page_mte_tagging(&a->page)); + mte_clear_page_tags(folio_address(a)); + set_page_mte_tagged(&a->page); + KUNIT_EXPECT_NE(test, memcmp_pages(&a->page, &b->page), 0); + for (len = SZ_4K; len <= PAGE_SIZE; len *= 4) { + KUNIT_EXPECT_EQ(test, memcmp_pages_range(&a->page, 0, &a->page, 0, len), 0); + KUNIT_EXPECT_NE(test, memcmp_pages_range(&a->page, 0, &b->page, 0, len), 0); + if (len < PAGE_SIZE) + KUNIT_EXPECT_NE(test, + memcmp_pages_range(&a->page, 0, &a->page, len, len), 0); + } + KUNIT_EXPECT_EQ(test, memcmp_pages_range(&a->page, 0, &b->page, 0, 0), 0); + KUNIT_ASSERT_TRUE(test, try_page_mte_tagging(&b->page)); + mte_clear_page_tags(folio_address(b)); + set_page_mte_tagged(&b->page); + KUNIT_EXPECT_NE(test, memcmp_pages_range(&a->page, 0, &b->page, 0, SZ_4K), 0); +#else + kunit_skip(test, "MTE is not configured"); +#endif +} + +static void user4k_data_copy_extent_test(struct kunit *test) +{ + const unsigned int sizes[] = { SZ_4K, SZ_16K, SZ_64K }; + + for (unsigned int n = 0; n < ARRAY_SIZE(sizes); n++) { + struct folio *source, *dest; + unsigned int size = sizes[n], to_offset; + void *from, *to; + + if (size > PAGE_SIZE) + continue; + source = folio_alloc(GFP_KERNEL | __GFP_ZERO, 0); + KUNIT_ASSERT_NOT_NULL(test, source); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, extent_folio_put, source), 0); + dest = folio_alloc(GFP_KERNEL | __GFP_ZERO, 0); + KUNIT_ASSERT_NOT_NULL(test, dest); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, extent_folio_put, dest), 0); + from = folio_address(source); + to = folio_address(dest); + to_offset = PAGE_SIZE - size; + for (unsigned int i = 0; i < DATA_SLOTS; i++) + memset(from + i * SZ_4K, 0x40 + i, SZ_4K); + for (unsigned int tagged = 0; tagged < 2; tagged++) { + memset(to, 0xac, PAGE_SIZE); +#ifdef CONFIG_ARM64_MTE + if (system_supports_mte()) { + if (try_page_mte_tagging(&dest->page)) + mte_clear_page_tags(to); + for (unsigned int i = 0; i < DATA_SLOTS; i++) + test_slot_tags_set(to + i * SZ_4K, 9); + set_page_mte_tagged(&dest->page); + if (tagged) { + if (try_page_mte_tagging(&source->page)) + mte_clear_page_tags(from); + for (unsigned int i = 0; i < DATA_SLOTS; i++) + test_slot_tags_set(from + i * SZ_4K, 1 + i % 15); + set_page_mte_tagged(&source->page); + } + } +#endif + KUNIT_ASSERT_EQ(test, copy_user_subpage_range(&dest->page, to_offset, + &source->page, 0, size), 0); + for (unsigned int i = 0; i < DATA_SLOTS; i++) { + unsigned int offset = i * SZ_4K; + bool copied = offset >= to_offset; + unsigned int source_index = copied ? (offset - to_offset) / SZ_4K : 0; + + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(to + offset, + copied ? 0x40 + source_index : 0xac, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(from + offset, 0x40 + i, SZ_4K), NULL); +#ifdef CONFIG_ARM64_MTE + if (system_supports_mte()) + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(to + offset, + copied ? (tagged ? 1 + source_index % 15 : 0) : 9)); +#endif + } + clear_user_subpage_range(&dest->page, to_offset, size); + for (unsigned int i = 0; i < DATA_SLOTS; i++) { + unsigned int offset = i * SZ_4K; + bool cleared = offset >= to_offset; + + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(to + offset, cleared ? 0 : 0xac, + SZ_4K), NULL); +#ifdef CONFIG_ARM64_MTE + if (system_supports_mte()) + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(to + offset, cleared ? 0 : 9)); +#endif + } + } + } +} + +#ifdef CONFIG_NET +struct skb_test_state { + struct folio *folio; +#ifdef CONFIG_INET + struct socket *socket; +#endif + struct sk_buff *skbs[5]; +}; + +static void skb_test_cleanup(void *arg) +{ + struct skb_test_state *state = arg; + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(state->skbs); i++) { + if (state->skbs[i]) + state->skbs[i]->sk = NULL; + kfree_skb(state->skbs[i]); + } +#ifdef CONFIG_INET + if (state->socket) + sock_release(state->socket); +#endif + /* This fixture marks ownership only; it installs no anonymous rmap. */ + if (state->folio) + state->folio->mapping = NULL; +} + +static void user4k_data_skb_test(struct kunit *test) +{ + struct data_fixture *f = data_fixture_create(test); + struct skb_test_state *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct mm_subpage *attempt; + struct iov_iter iter; + struct bio_vec bvec; + unsigned char *bytes = kunit_kmalloc(test, SZ_4K, GFP_KERNEL); + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_NOT_NULL(test, bytes); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * SZ_4K)); + memset(slot_address(f->held[i]), 0x51 + i, SZ_4K); + } + state->folio = mm_subpage_folio(f->held[0]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, skb_test_cleanup, state), 0); + state->folio->mapping = (void *)FOLIO_MAPPING_ANON; + state->skbs[0] = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[0]); + state->skbs[0]->ip_summed = CHECKSUM_UNNECESSARY; + bvec_set_page(&bvec, &state->folio->page, SZ_4K, SZ_4K); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, SZ_4K); + KUNIT_ASSERT_EQ(test, skb_splice_from_iter(state->skbs[0], &iter, SZ_4K), (ssize_t)SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, skb_zcopy(state->skbs[0])); + state->skbs[1] = pskb_copy(state->skbs[0], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[1]); + /* Append a distinct quarter while the callback object has two owners. */ + bvec_set_page(&bvec, &state->folio->page, SZ_4K, 2 * SZ_4K); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, SZ_4K); + KUNIT_ASSERT_EQ(test, skb_splice_from_iter(state->skbs[0], &iter, SZ_4K), (ssize_t)SZ_4K); + KUNIT_EXPECT_PTR_NE(test, skb_zcopy(state->skbs[0]), skb_zcopy(state->skbs[1])); + state->skbs[2] = skb_clone(state->skbs[1], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[2]); + state->skbs[3] = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[3]); + skb_split(state->skbs[0], state->skbs[3], SZ_2K); + state->skbs[4] = skb_copy(state->skbs[3], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[4]); + put_held(f, 1); + put_held(f, 2); + attempt = mm_subpage_alloc_at(f->pool, SZ_4K); + if (!IS_ERR(attempt)) + f->held[1] = attempt; + KUNIT_EXPECT_EQ(test, PTR_ERR_OR_ZERO(attempt), -EAGAIN); + for (i = 0; i < 2; i++) { + kfree_skb(state->skbs[i]); + state->skbs[i] = NULL; + } + KUNIT_ASSERT_EQ(test, skb_copy_bits(state->skbs[2], 0, bytes, SZ_4K), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(bytes, 0x52, SZ_4K), NULL); + KUNIT_ASSERT_EQ(test, skb_copy_bits(state->skbs[3], SZ_2K, bytes, SZ_4K), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(bytes, 0x53, SZ_4K), NULL); + for (i = 2; i < 4; i++) { + kfree_skb(state->skbs[i]); + state->skbs[i] = NULL; + } + /* A full data copy must not keep ownership of the original quarters. */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, SZ_4K)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 2, 2 * SZ_4K)); + memset(slot_address(f->held[1]), 0xe1, SZ_4K); + memset(slot_address(f->held[2]), 0xe2, SZ_4K); + KUNIT_ASSERT_EQ(test, skb_copy_bits(state->skbs[4], SZ_2K, bytes, SZ_4K), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(bytes, 0x53, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); +} +#ifdef CONFIG_INET +static void test_data_skb_prune_granule(struct kunit *test, unsigned int shift) +{ + struct data_fixture *f = data_fixture_create_granule(test, shift); + struct skb_test_state *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct sk_buff *skb; + struct bio_vec bvec; + struct iov_iter iter; + unsigned int i, size = 1U << shift; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * size)); + state->folio = mm_subpage_folio(f->held[0]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, skb_test_cleanup, state), 0); + KUNIT_ASSERT_EQ(test, sock_create_kern(&init_net, AF_INET, SOCK_STREAM, 0, &state->socket), 0); + state->folio->mapping = (void *)FOLIO_MAPPING_ANON; + state->skbs[0] = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[0]); + skb = state->skbs[0]; + /* A live socket association prevents trim from condensing data into the head. */ + skb->sk = state->socket->sk; + skb->ip_summed = CHECKSUM_UNNECESSARY; + bvec_set_page(&bvec, &state->folio->page, 16, size - SZ_4K + 16); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, 16); + KUNIT_ASSERT_EQ(test, skb_splice_from_iter(skb, &iter, 16), (ssize_t)16); + for (i = 0; i < 1024; i++) { + bvec_set_page(&bvec, &state->folio->page, size, (1 + i % 3) * size); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, size); + KUNIT_ASSERT_EQ(test, skb_splice_from_iter(skb, &iter, size), (ssize_t)size); + /* Four fixture refs, two frag refs, two current quarter refs. */ + KUNIT_ASSERT_EQ(test, folio_ref_count(state->folio), 8); + KUNIT_ASSERT_EQ(test, pskb_trim(skb, 16), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 7); + } + skb->sk = NULL; + kfree_skb(skb); + state->skbs[0] = NULL; + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); +} + +static void user4k_data_skb_prune_test(struct kunit *test) +{ + test_data_skb_prune_granule(test, MM_SUBPAGE_SHIFT); +} + +static void user16k_data_skb_prune_test(struct kunit *test) +{ + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_data_skb_prune_granule(test, 14); +} + +#endif + +static void user4k_data_skb_checksum_test(struct kunit *test) +{ + struct data_fixture *f = data_fixture_create(test); + struct skb_test_state *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct bio_vec bvec[2]; + struct iov_iter iter; + unsigned char *bytes; + unsigned int i, owned; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * SZ_4K)); + state->folio = mm_subpage_folio(f->held[0]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, skb_test_cleanup, state), 0); + bytes = folio_address(state->folio); + for (i = 0; i < PAGE_SIZE; i++) + bytes[i] = i * 13 + (i >> 12) * 31; + for (owned = 0; owned < 2; owned++) { + struct sk_buff *skb; + + state->folio->mapping = owned ? (void *)FOLIO_MAPPING_ANON : NULL; + state->skbs[0] = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->skbs[0]); + skb = state->skbs[0]; + skb->ip_summed = CHECKSUM_NONE; + bvec_set_page(&bvec[0], &state->folio->page, SZ_4K - 13, 13); + bvec_set_page(&bvec[1], &state->folio->page, SZ_4K, SZ_4K); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 2, SZ_8K - 13); + KUNIT_ASSERT_EQ(test, skb_splice_from_iter(skb, &iter, SZ_8K - 13), + (ssize_t)(SZ_8K - 13)); + KUNIT_EXPECT_EQ(test, (__force u16)csum_fold(skb->csum), + (__force u16)csum_fold(skb_checksum(skb, 0, skb->len, 0))); + kfree_skb(skb); + state->skbs[0] = NULL; + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); + } +} + +#endif + +struct refs_test_state { + struct folio *folio; + struct mm_subpage_refs *refs[3]; +}; + +static void refs_test_cleanup(void *arg) +{ + struct refs_test_state *state = arg; + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(state->refs); i++) + mm_subpage_refs_put(state->refs[i]); + if (state->folio) + state->folio->mapping = NULL; +} + +static bool refs_test_keep(const struct mm_subpage *slot, void *context) +{ + unsigned long mask = *(unsigned long *)context; + + return mask & BIT(mm_subpage_offset(slot) >> MM_SUBPAGE_SHIFT); +} + +static void user4k_data_refset_test(struct kunit *test) +{ + struct data_fixture *f = data_fixture_create(test); + struct refs_test_state *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct mm_subpage *attempt; + struct page *page; + unsigned long mask; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < DATA_SLOTS; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * SZ_4K)); + state->folio = mm_subpage_folio(f->held[0]); + page = &state->folio->page; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, refs_test_cleanup, state), 0); + /* Ownership fixture only; no anonymous rmap is installed. */ + state->folio->mapping = (void *)FOLIO_MAPPING_ANON; + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, + SZ_4K + 17, SZ_8K - 34, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 2); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, SZ_4K, SZ_4K, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 2); + state->refs[1] = mm_subpage_refs_get(state->refs[0]); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[1], page, 3 * SZ_4K, SZ_4K, GFP_KERNEL), 0); + KUNIT_EXPECT_PTR_NE(test, state->refs[0], state->refs[1]); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 5); + mask = BIT(2); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[1], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 3); + state->refs[2] = mm_subpage_refs_get(state->refs[0]); + mask = BIT(1); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[2], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_PTR_NE(test, state->refs[0], state->refs[2]); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 4); + KUNIT_EXPECT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, PAGE_SIZE, 1, GFP_KERNEL), -EINVAL); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 4); + mask = 0; + for (i = 1; i < 3; i++) { + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[i], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_PTR_EQ(test, state->refs[i], NULL); + } + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 2); + put_held(f, 1); + attempt = mm_subpage_alloc_at(f->pool, SZ_4K); + if (!IS_ERR(attempt)) + f->held[1] = attempt; + KUNIT_ASSERT_EQ(test, PTR_ERR_OR_ZERO(attempt), -EAGAIN); + mm_subpage_refs_put(state->refs[0]); + state->refs[0] = NULL; + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, SZ_4K)); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS); + for (i = 0; i < 1024; i++) { + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, 0, PAGE_SIZE, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 2 * (int)DATA_SLOTS); + mask = BIT(i % DATA_SLOTS); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[0], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS + 1); + } + mm_subpage_refs_put(state->refs[0]); + state->refs[0] = NULL; + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)DATA_SLOTS); +} + +#if IS_ENABLED(CONFIG_NET_SOCK_MSG) && IS_ENABLED(CONFIG_INET) +struct skmsg_test_state { + struct folio *folio; + struct socket *socket; + struct sk_msg messages[2]; + struct sk_msg borrowed; +}; + +static void skmsg_test_cleanup(void *arg) +{ + struct skmsg_test_state *state = arg; + unsigned int i; + + if (state->socket) { + for (i = 0; i < ARRAY_SIZE(state->messages); i++) + sk_msg_free(state->socket->sk, &state->messages[i]); + sk_msg_subpages_release(&state->borrowed); + sock_release(state->socket); + } + if (state->folio) + state->folio->mapping = NULL; +} + +static void test_data_skmsg_granule(struct kunit *test, unsigned int shift) +{ + struct data_fixture *f = data_fixture_create_granule(test, shift); + struct skmsg_test_state *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct mm_subpage *attempt; + struct sk_msg *src, *dst; + struct page *page; + struct sock *sk; + struct bio_vec bvec; + struct iov_iter iter; + unsigned int i, size = 1U << shift; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (i = 0; i < 4; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * size)); + memset(slot_address(f->held[i]), 0x60 + i, size); + } + state->folio = mm_subpage_folio(f->held[0]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, skmsg_test_cleanup, state), 0); + KUNIT_ASSERT_EQ(test, sock_create_kern(&init_net, AF_INET, SOCK_STREAM, 0, &state->socket), 0); + state->folio->mapping = (void *)FOLIO_MAPPING_ANON; + page = &state->folio->page; + sk = state->socket->sk; + src = &state->messages[0]; + dst = &state->messages[1]; + sk_msg_init(src); + sk_msg_init(dst); + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, (size / 2) - 13, size + (size / 2) + 13), 0); + sk_mem_charge(sk, (size / 2) - 13); + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, size, 2 * size), 0); + sk_mem_charge(sk, size); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 8); + KUNIT_ASSERT_EQ(test, sk_msg_clone(sk, dst, src, 0, size + (size / 2) - 13), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 11); + sk_msg_free_partial(sk, src, (size / 2) - 13); + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, size, 0), 0); + sk_mem_charge(sk, size); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 11); + sk_msg_free(sk, dst); + put_held(f, 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, size)); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 8); + sk_msg_free_partial(sk, src, size); + for (i = 0; i < 1024; i++) { + unsigned int index = (i + 1) & 3; + + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, size, index * size), 0); + sk_mem_charge(sk, size); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 8); + sk_msg_free_partial(sk, src, size); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 7); + } + sk_msg_free(sk, src); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); + /* A borrowed descriptor copy has an independent ownership lifetime. */ + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, size, 0), 0); + sk_mem_charge(sk, size); + memcpy(&state->borrowed, src, sizeof(*src)); + sk_msg_subpages_get(&state->borrowed); + sk_msg_free(sk, src); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 5); + put_held(f, 0); + attempt = mm_subpage_alloc_at(f->pool, 0); + if (!IS_ERR(attempt)) + f->held[0] = attempt; + KUNIT_ASSERT_EQ(test, PTR_ERR_OR_ZERO(attempt), -EAGAIN); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(sg_virt(&state->borrowed.sg.data[0]), 0x60, size), NULL); + sk_msg_subpages_release(&state->borrowed); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 0, 0)); + /* Full transfer moves the set, and zero-copy iterator trim releases its references. */ + KUNIT_ASSERT_EQ(test, sk_msg_page_add(src, page, size, 3 * size), 0); + sk_mem_charge(sk, size); + sk_msg_xfer_full(dst, src); + KUNIT_EXPECT_EQ(test, src->sg.size, 0U); + sk_msg_free(sk, dst); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); + bvec_set_page(&bvec, page, size - 17, 3 * size + 17); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, size - 17); + KUNIT_ASSERT_EQ(test, sk_msg_zerocopy_from_iter(sk, &iter, src, size - 17), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 6); + sk_msg_trim(sk, src, 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), 4); +} + +static void user4k_data_skmsg_test(struct kunit *test) +{ + test_data_skmsg_granule(test, MM_SUBPAGE_SHIFT); +} + +static void user16k_data_skmsg_test(struct kunit *test) +{ + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_data_skmsg_granule(test, 14); +} +#endif + +static void user16k_data_owner_test(struct kunit *test) +{ + struct data_fixture *f, *dest, *small; + struct mm_subpage *slot, *found; + unsigned int count = PAGE_SIZE / SZ_16K; + + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + f = data_fixture_create_granule(test, 14); + dest = data_fixture_create_granule(test, 14); + small = data_fixture_create(test); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, dest); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + KUNIT_ASSERT_EQ(test, supply_backing(dest->pool), 0); + KUNIT_ASSERT_EQ(test, supply_backing(small->pool), 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc(small, 0)); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc_at(f->pool, SZ_4K)), -EINVAL); + for (unsigned int i = 0; i < count; i++) { + slot = hold_alloc_at(f, i, i * SZ_16K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_shift(slot), 14U); + KUNIT_EXPECT_EQ(test, mm_subpage_size(slot), (unsigned int)SZ_16K); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), i * (unsigned int)SZ_16K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0, SZ_16K), NULL); + memset(slot_address(slot), 0x30 + i, SZ_16K); + for (unsigned int quarter = 0; quarter < 4; quarter++) { + found = mm_subpage_get_from_phys(mm_subpage_phys(slot) + quarter * SZ_4K); + KUNIT_ASSERT_PTR_EQ(test, found, slot); + mm_subpage_put(found); + } + } + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(f->pool)), -EAGAIN); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_copy(dest->pool, small->held[0])), -EINVAL); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_copy(small->pool, f->held[0])), -EINVAL); + slot = mm_subpage_copy_at(dest->pool, f->held[count - 1], SZ_16K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + dest->held[0] = slot; + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(slot), 0x30 + count - 1, SZ_16K), NULL); + /* An interior-byte lookup holds the whole identity across release/reuse. */ + f->held[count] = mm_subpage_get_from_phys(mm_subpage_phys(f->held[1]) + 3 * SZ_4K); + KUNIT_ASSERT_PTR_EQ(test, f->held[count], f->held[1]); + put_held(f, 1); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc_at(f->pool, SZ_16K)), -EAGAIN); + put_held(f, count); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, SZ_16K)); + for (unsigned int i = 0; i < count; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(slot_address(f->held[i]), + i == 1 ? 0 : 0x30 + i, SZ_16K), NULL); + mm_subpage_pool_close(f->pool); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc(f->pool)), -ESHUTDOWN); + for (unsigned int i = 0; i < count; i++) + put_held(f, i); + KUNIT_EXPECT_EQ(test, mm_subpage_pool_backing_pages(f->pool), 0UL); +} + +static void user16k_data_refset_test(struct kunit *test) +{ + struct data_fixture *f; + struct refs_test_state *state; + unsigned int count = PAGE_SIZE / SZ_16K; + unsigned long mask; + struct page *page; + + if (PAGE_SIZE <= SZ_16K) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + f = data_fixture_create_granule(test, 14); + state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, supply_backing(f->pool), 0); + for (unsigned int i = 0; i < count; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * SZ_16K)); + state->folio = mm_subpage_folio(f->held[0]); + page = &state->folio->page; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, refs_test_cleanup, state), 0); + /* Metadata-only ownership fixture: no VMA or rmap is published. */ + state->folio->mapping = (void *)FOLIO_MAPPING_ANON; + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, + SZ_16K - 1024, 2048, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count + 2); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, 0, PAGE_SIZE, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count * 2); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_add(&state->refs[0], page, 0, PAGE_SIZE, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count * 2); + state->refs[1] = mm_subpage_refs_get(state->refs[0]); + mask = BIT(SZ_16K >> MM_SUBPAGE_SHIFT); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[1], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count * 2 + 1); + mask = 0; + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[0], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count + 1); + put_held(f, 1); + KUNIT_EXPECT_EQ(test, PTR_ERR(mm_subpage_alloc_at(f->pool, SZ_16K)), -EAGAIN); + KUNIT_ASSERT_EQ(test, mm_subpage_refs_prune(&state->refs[1], refs_test_keep, &mask, GFP_KERNEL), 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, SZ_16K)); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->folio), (int)count); +} + +static void user16k_data_mte_test(struct kunit *test) +{ +#ifdef CONFIG_ARM64_MTE + struct data_fixture *f; + struct folio *folio; + struct mm_subpage *copy; + unsigned int count = PAGE_SIZE / SZ_16K; + int ret; + + if (PAGE_SIZE <= SZ_16K || !system_supports_mte()) { + kunit_skip(test, "requires MTE and native backing larger than 16K"); + return; + } + f = data_fixture_create_granule(test, 14); + KUNIT_ASSERT_NOT_NULL(test, f); + folio = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, folio); + if (try_page_mte_tagging(&folio->page)) + mte_clear_page_tags(folio_address(folio)); + for (unsigned int i = 0; i < DATA_SLOTS; i++) + test_slot_tags_set(folio_address(folio) + i * SZ_4K, 9); + set_page_mte_tagged(&folio->page); + ret = mm_subpage_pool_add_folio(f->pool, folio, GFP_KERNEL); + if (ret) + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + for (unsigned int i = 0; i < count; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, i, i * SZ_16K)); + for (unsigned int i = 0; i < DATA_SLOTS; i++) { + void *address = folio_address(folio) + i * SZ_4K; + + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(address, 0)); + memset(address, 0x50 + i, SZ_4K); + test_slot_tags_set(address, 1 + i % 15); + } + put_held(f, 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, hold_alloc_at(f, 1, SZ_16K)); + for (unsigned int i = 0; i < DATA_SLOTS; i++) { + bool cleared = i >= 4 && i < 8; + void *address = folio_address(folio) + i * SZ_4K; + + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(address, cleared ? 0 : 1 + i % 15)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(address, cleared ? 0 : 0x50 + i, SZ_4K), NULL); + } + put_held(f, 1); + copy = mm_subpage_copy_at(f->pool, f->held[count - 1], SZ_16K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy); + f->held[1] = copy; + for (unsigned int i = 0; i < DATA_SLOTS; i++) { + unsigned int source = i >= 4 && i < 8 ? i + DATA_SLOTS - 8 : i; + void *address = folio_address(folio) + i * SZ_4K; + + KUNIT_EXPECT_TRUE(test, test_slot_tags_equal(address, 1 + source % 15)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(address, 0x50 + source, SZ_4K), NULL); + } +#else + kunit_skip(test, "MTE is not configured"); +#endif +} + +static void expect_file_coverage(struct kunit *test, struct folio *folio, + unsigned int leaves, unsigned int first, unsigned int tail, + unsigned int final) +{ + struct mm_subpage_mapcounts counts; + unsigned int i; + + KUNIT_EXPECT_EQ(test, mm_subpage_file_mapcounts(&folio->page, &counts), leaves); + for (i = 0; i < DATA_SLOTS; i++) { + unsigned int expected = i == 0 ? first : + i == DATA_SLOTS - 1 ? final : i >= DATA_SLOTS - 4 ? tail : 0; + + KUNIT_EXPECT_EQ_MSG(test, counts.slots[i], expected, "coverage slot %u", i); + } +} + +static void user16k_file_coverage_test(struct kunit *test) +{ + struct folio *folio; + phys_addr_t base, tail, last; + + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative file coverage requires larger native backing"); + return; + } + folio = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, extent_folio_put, folio), 0); + base = PFN_PHYS(folio_pfn(folio)); + tail = base + PAGE_SIZE - SZ_16K; + last = base + PAGE_SIZE - SZ_4K; + expect_file_coverage(test, folio, 0, 0, 0, 0); + __mm_subpage_file_map_add(tail, SZ_16K); + expect_file_coverage(test, folio, 1, 0, 1, 1); + __mm_subpage_file_map_add(last, SZ_4K); + expect_file_coverage(test, folio, 2, 0, 1, 2); + __mm_subpage_file_map_add(tail, SZ_16K); + expect_file_coverage(test, folio, 3, 0, 2, 3); + __mm_subpage_file_map_add(base, SZ_4K); + expect_file_coverage(test, folio, 4, 1, 2, 3); + __mm_subpage_file_map_del(tail, SZ_16K); + expect_file_coverage(test, folio, 3, 1, 1, 2); + __mm_subpage_file_map_del(last, SZ_4K); + expect_file_coverage(test, folio, 2, 1, 1, 1); + __mm_subpage_file_map_del(tail, SZ_16K); + expect_file_coverage(test, folio, 1, 1, 0, 0); + __mm_subpage_file_map_del(base, SZ_4K); + expect_file_coverage(test, folio, 0, 0, 0, 0); +} + +static struct kunit_case user4k_data_cases[] = { + KUNIT_CASE(user4k_data_reuse_test), + KUNIT_CASE(user4k_data_refset_test), +#if IS_ENABLED(CONFIG_NET_SOCK_MSG) && IS_ENABLED(CONFIG_INET) + KUNIT_CASE(user4k_data_skmsg_test), + KUNIT_CASE(user16k_data_skmsg_test), +#endif +#ifdef CONFIG_NET + KUNIT_CASE(user4k_data_skb_test), + KUNIT_CASE(user4k_data_skb_checksum_test), + +#ifdef CONFIG_INET + KUNIT_CASE(user4k_data_skb_prune_test), + KUNIT_CASE(user16k_data_skb_prune_test), +#endif +#endif + KUNIT_CASE(user4k_data_mte_test), + KUNIT_CASE(user4k_data_copy_extent_test), + KUNIT_CASE(user_page_range_compare_test), + KUNIT_CASE(user_page_tagged_range_compare_test), + KUNIT_CASE(user16k_data_owner_test), + KUNIT_CASE(user16k_file_coverage_test), + KUNIT_CASE(user16k_data_refset_test), + KUNIT_CASE(user16k_data_mte_test), + KUNIT_CASE(user4k_data_swapcache_reuse_test), + KUNIT_CASE(user4k_data_restore_test), + KUNIT_CASE(user16k_data_restore_test), + KUNIT_CASE(user4k_data_restore_race_test), + KUNIT_CASE(user4k_data_migration_owner_test), + KUNIT_CASE(user16k_data_migration_owner_test), + KUNIT_CASE(user4k_data_copy_test), + KUNIT_CASE(user4k_data_lookup_test), + KUNIT_CASE(user4k_data_exact_slot_test), + KUNIT_CASE(user4k_data_close_test), + KUNIT_CASE(user4k_data_deferred_test), + KUNIT_CASE(user4k_data_backing_reject_test), + KUNIT_CASE(user4k_data_concurrent_test), + KUNIT_CASE(user4k_data_lookup_race_test), + {} +}; + +static void user4k_data_suite_exit(struct kunit_suite *suite) +{ + /* Drain batched metadata frees before the disposable guest powers off. */ + kvfree_rcu_barrier(); +} + +static struct kunit_suite user4k_data_suite = { + .name = "arm64-user4k-data", + .test_cases = user4k_data_cases, + .suite_exit = user4k_data_suite_exit, +}; + +kunit_test_suite(user4k_data_suite); +MODULE_LICENSE("GPL"); diff --git a/arch/arm64/mm/user4k-pgalloc.c b/arch/arm64/mm/user4k-pgalloc.c new file mode 100644 index 00000000000000..7a3e34205760c0 --- /dev/null +++ b/arch/arm64/mm/user4k-pgalloc.c @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Smaller hardware tables share native owners, including safely retired slots. */ +#include +#include +#include +#include +#include +#include + +struct fragment_pool { + spinlock_t lock; + refcount_t refs; + struct list_head available; + unsigned int shift; + bool closed; +}; + +struct fragment_slot { + struct rcu_head rcu; + void *table; +}; + +struct fragment_owner { + struct fragment_pool *pool; + struct ptdesc *pt; + struct list_head available; + unsigned long free_mask; + unsigned long dirty_mask; + struct fragment_slot slots[]; +}; + +unsigned int arm64_ptep_page_shift(pte_t *ptep) +{ + struct ptdesc *pt = virt_to_ptdesc(ptep); + + return test_bit(PT_fragmented, &pt->pt_flags.f) ? pt->pt_frag_shift : PAGE_SHIFT; +} +EXPORT_SYMBOL_GPL(arm64_ptep_page_shift); + +static void fragment_owner_free(struct ptdesc *pt) +{ + clear_bit(PT_fragmented, &pt->pt_flags.f); + pagetable_dtor_free(pt); +} + +static void fragment_owner_free_rcu(struct rcu_head *rcu) +{ + fragment_owner_free(container_of(rcu, struct ptdesc, pt_rcu_head)); +} + +static void fragment_pool_put(struct fragment_pool *pool) +{ + if (refcount_dec_and_test(&pool->refs)) { + WARN_ON_ONCE(!list_empty(&pool->available)); + kfree(pool); + } +} + +/* Active table allocation already holds the mm alive. Its cache reference + * remains until destroy_context, after all allocators have stopped. Owners + * independently retain the pool for callbacks after that point. + */ +static struct fragment_pool *fragment_pool_get(struct mm_struct *mm, + enum arm64_user4k_pt_level level, unsigned int shift) +{ + struct fragment_pool *pool, *new; + + pool = smp_load_acquire(&mm->context.user4k_pt_frag[level]); + if (pool) + return pool->shift == shift ? pool : NULL; + new = kzalloc(sizeof(*new), GFP_KERNEL | __GFP_ACCOUNT); + if (!new) + return NULL; + spin_lock_init(&new->lock); + refcount_set(&new->refs, 1); + INIT_LIST_HEAD(&new->available); + new->shift = shift; + pool = cmpxchg(&mm->context.user4k_pt_frag[level], NULL, new); + if (!pool) + return new; + fragment_pool_put(new); + return pool->shift == shift ? pool : NULL; +} + +static struct ptdesc *fragment_pt_alloc(struct mm_struct *mm, + enum arm64_user4k_pt_level level, unsigned int shift) +{ + struct ptdesc *pt; + bool constructed; + + pt = pagetable_alloc(GFP_PGTABLE_USER, shift > PAGE_SHIFT ? shift - PAGE_SHIFT : 0); + if (!pt) + return NULL; + switch (level) { + case USER4K_PT_PTE: + constructed = pagetable_pte_ctor(mm, pt); + break; + case USER4K_PT_PMD: + constructed = pagetable_pmd_ctor(mm, pt); + break; + case USER4K_PT_PUD: + pagetable_pud_ctor(pt); + constructed = true; + break; + default: + constructed = false; + } + if (!constructed) { + pagetable_free(pt); + return NULL; + } + pt->pt_frag_shift = shift; + set_bit(PT_fragmented, &pt->pt_flags.f); + atomic_set(&pt->pt_frag_refcount, 1); + return pt; +} + +/* The caller holds pool->lock; its active mm retains the pool. */ +static void *fragment_take(struct fragment_pool *pool, bool *zero) +{ + struct fragment_owner *owner; + unsigned int slot; + + if (pool->closed || list_empty(&pool->available)) + return NULL; + owner = list_first_entry(&pool->available, struct fragment_owner, available); + slot = __ffs(owner->free_mask); + owner->free_mask &= ~BIT(slot); + *zero = owner->dirty_mask & BIT(slot); + owner->dirty_mask &= ~BIT(slot); + atomic_inc(&owner->pt->pt_frag_refcount); + if (!owner->free_mask) + list_del_init(&owner->available); + return owner->slots[slot].table; +} + +void *arm64_user_pt_alloc_granule(struct mm_struct *mm, + enum arm64_user4k_pt_level level, unsigned int shift) +{ + struct fragment_pool *pool; + struct fragment_owner *owner; + struct ptdesc *pt; + void *table; + unsigned int nr, i; + unsigned long flags; + bool zero = false; + + if (shift != 12 && shift != 14 && shift != 16) + return NULL; + if (WARN_ON_ONCE((unsigned int)level >= ARRAY_SIZE(mm->context.user4k_pt_frag))) + return NULL; + if (shift >= PAGE_SHIFT) { + pt = fragment_pt_alloc(mm, level, shift); + return pt ? ptdesc_address(pt) : NULL; + } + pool = fragment_pool_get(mm, level, shift); + if (!pool) + return NULL; + spin_lock_irqsave(&pool->lock, flags); + table = fragment_take(pool, &zero); + spin_unlock_irqrestore(&pool->lock, flags); + if (table) + goto out; + + nr = PAGE_SIZE >> shift; + owner = kzalloc(struct_size(owner, slots, nr), GFP_KERNEL | __GFP_ACCOUNT); + if (!owner) + goto out; + pt = fragment_pt_alloc(mm, level, shift); + if (!pt) { + kfree(owner); + goto out; + } + owner->pool = pool; + owner->pt = pt; + INIT_LIST_HEAD(&owner->available); + owner->free_mask = GENMASK(nr - 1, 1); + for (i = 0; i < nr; i++) + owner->slots[i].table = ptdesc_address(pt) + (i << shift); + /* This word is unused by arm64 page-table constructors and walkers. */ + pt->__page_mapping = (unsigned long)owner; + + spin_lock_irqsave(&pool->lock, flags); + /* Reuse supply published while allocation slept, including retired slots. */ + table = fragment_take(pool, &zero); + if (!table && !pool->closed) { + refcount_inc(&pool->refs); /* Native owner's lifetime reference. */ + list_add_tail(&owner->available, &pool->available); + table = owner->slots[0].table; + owner = NULL; + } + spin_unlock_irqrestore(&pool->lock, flags); + if (owner) { + pt->__page_mapping = 0; + fragment_owner_free(pt); + kfree(owner); + } +out: + /* Clear only recycled slots that are actually allocated again. The issued + * reference excludes other allocators and keeps this owner alive here. + */ + if (table && zero) + memset(table, 0, 1UL << shift); + return table; +} + +void *arm64_user4k_pt_alloc(struct mm_struct *mm, enum arm64_user4k_pt_level level) +{ + return arm64_user_pt_alloc_granule(mm, level, 12); +} + +/* Only unpublished tables or tables past the required walker grace arrive here. */ +void arm64_user4k_pt_free(void *table) +{ + struct ptdesc *pt = virt_to_ptdesc(table); + struct fragment_owner *owner; + struct fragment_pool *pool; + unsigned int slot; + unsigned long flags; + bool last; + + VM_BUG_ON(!test_bit(PT_fragmented, &pt->pt_flags.f)); + VM_BUG_ON(!IS_ALIGNED((unsigned long)table, 1UL << pt->pt_frag_shift)); + VM_BUG_ON(atomic_read(&pt->pt_frag_refcount) <= 0); + if (pt->pt_frag_shift >= PAGE_SHIFT) { + if (test_and_clear_bit(PT_frag_defer, &pt->pt_flags.f)) + call_rcu(&pt->pt_rcu_head, fragment_owner_free_rcu); + else + fragment_owner_free(pt); + return; + } + owner = (void *)pt->__page_mapping; + pool = owner->pool; + slot = ((unsigned long)table & ~PAGE_MASK) >> pt->pt_frag_shift; + spin_lock_irqsave(&pool->lock, flags); + VM_BUG_ON(owner->free_mask & BIT(slot)); + owner->free_mask |= BIT(slot); + owner->dirty_mask |= BIT(slot); + last = atomic_dec_and_test(&pt->pt_frag_refcount); + if (last) + list_del_init(&owner->available); + else if (!pool->closed && list_empty(&owner->available)) + list_add_tail(&owner->available, &pool->available); + spin_unlock_irqrestore(&pool->lock, flags); + if (last) { + pt->__page_mapping = 0; + fragment_owner_free(pt); + kfree(owner); + fragment_pool_put(pool); + } +} + +void arm64_pgtable_free(void *table) +{ + struct ptdesc *pt = virt_to_ptdesc(table); + + if (test_bit(PT_fragmented, &pt->pt_flags.f)) + arm64_user4k_pt_free(table); + else { + VM_BUG_ON(!PAGE_ALIGNED(table)); + pagetable_dtor_free(pt); + } +} + +void arm64_user4k_pt_cache_destroy(struct mm_struct *mm) +{ + struct fragment_pool *pool; + unsigned int level; + unsigned long flags; + + /* No allocator may still access this mm; deferred owners are independent. */ + for (level = 0; level < ARRAY_SIZE(mm->context.user4k_pt_frag); level++) { + pool = mm->context.user4k_pt_frag[level]; + if (!pool) + continue; + mm->context.user4k_pt_frag[level] = NULL; + spin_lock_irqsave(&pool->lock, flags); + pool->closed = true; + spin_unlock_irqrestore(&pool->lock, flags); + fragment_pool_put(pool); + } +} + +static void fragment_slot_free_rcu(struct rcu_head *rcu) +{ + struct fragment_slot *slot = container_of(rcu, struct fragment_slot, rcu); + + arm64_user4k_pt_free(slot->table); +} + +void arm64_pte_free_defer(struct mm_struct *mm, pgtable_t table) +{ + struct ptdesc *pt = virt_to_ptdesc(table); + + if (test_bit(PT_fragmented, &pt->pt_flags.f)) { + if (pt->pt_frag_shift < PAGE_SHIFT) { + struct fragment_owner *owner = (void *)pt->__page_mapping; + unsigned int slot = ((unsigned long)table & ~PAGE_MASK) >> + pt->pt_frag_shift; + + /* Pending callbacks retain their issued references and contents. */ + call_rcu(&owner->slots[slot].rcu, fragment_slot_free_rcu); + } else { + set_bit(PT_frag_defer, &pt->pt_flags.f); + arm64_user4k_pt_free(table); + } + } else { + call_rcu(&pt->pt_rcu_head, fragment_owner_free_rcu); + } +} + +void __tlb_remove_table(void *token) +{ + if ((unsigned long)token & 1) + arm64_user4k_pt_free((void *)((unsigned long)token & ~1UL)); + else + pagetable_dtor_free(token); +} diff --git a/arch/arm64/mm/user4k-test.c b/arch/arm64/mm/user4k-test.c new file mode 100644 index 00000000000000..d5349f54f6d7c4 --- /dev/null +++ b/arch/arm64/mm/user4k-test.c @@ -0,0 +1,14451 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Architectural prerequisite for mixed-granule userspace. + * + * Packed architectural tables and isolated Linux mms are separate fixtures. + * PAGE_SIZE, struct page, all ordinary mms and TTBR1 retain native geometry. Only bounded + * kernel workers may borrow the isolated Linux mms. Never pass + * packed architectural tables to generic MM walkers or expose them to a task. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../../drivers/iommu/iommufd/iommufd_private.h" +#include "../../../drivers/iommu/iommufd/io_pagetable.h" +#include "../../../mm/internal.h" +#include "../../../mm/vma.h" +#include "../../../mm/swap.h" +#ifdef CONFIG_FUTEX +#include "../../../kernel/futex/futex.h" +#endif + +#define USER4K_SHIFT 12 +#define TEST_SLOTS MM_SUBPAGES_PER_PAGE +#define USER4K_SIZE BIT(USER4K_SHIFT) +#define TEST_VA (4UL * SZ_1M) +#define TEST_LEVELS 4 +#define PROBE_SLOTS 4 +#define TEST_ROUNDS 8 +#define TEST_PROT (PTE_TYPE_PAGE | PTE_USER | PTE_AF | PTE_NG | \ + PTE_SHARED | PTE_PXN | PTE_UXN | PTE_ATTRINDX(MT_NORMAL)) + +static bool user4k_test_enabled; + +static int __init user4k_test_setup(char *str) +{ + bool enabled; + int ret = kstrtobool(str, &enabled); + + if (ret) + return ret; + /* Software MM tests are sanitizable; raw/borrowed TTBR0 tests stay gated. */ + if (enabled && IS_ENABLED(CONFIG_KASAN)) + return -EINVAL; + user4k_test_enabled = enabled; + return 0; +} +early_param("arm64.user4k_test", user4k_test_setup); + +struct test_tables { + u64 *level[TEST_LEVELS]; + unsigned int shift; +}; + +struct test_result { + u64 native[PROBE_SLOTS]; + u64 read[PROBE_SLOTS]; + u64 write[PROBE_SLOTS]; + u64 replaced[PROBE_SLOTS]; + u64 replaced_read; + u64 kernel; + u64 tcr; + u64 ttbr1; + u64 restored_tcr; + u64 restored_ttbr0; +}; + +static void test_free_page(void *ptr) +{ + free_page((unsigned long)ptr); +} + +static void *test_alloc_page(struct kunit *test) +{ + void *ptr = (void *)get_zeroed_page(GFP_KERNEL); + + if (!ptr) + return NULL; + if (kunit_add_action_or_reset(test, test_free_page, ptr)) + return NULL; + return ptr; +} + +static void test_put_compound(void *ptr) +{ + put_page(ptr); +} + +static void *test_alloc_granule(struct kunit *test, unsigned int shift) +{ + struct page *page; + + if (shift <= PAGE_SHIFT) + return test_alloc_page(test); + page = alloc_pages(GFP_KERNEL | __GFP_ZERO | __GFP_COMP, shift - PAGE_SHIFT); + if (!page) + return NULL; + if (kunit_add_action_or_reset(test, test_put_compound, page)) + return NULL; + return page_address(page); +} + +static unsigned int test_index(unsigned long va, unsigned int shift, + unsigned int level) +{ + unsigned int bits = shift - PTDESC_ORDER; + + return (va >> ARM64_HW_PGTABLE_LEVEL_SHIFT_FOR(level, shift)) & (BIT(bits) - 1); +} + +static int test_alloc_tables(struct kunit *test, struct test_tables *tables, + unsigned int shift) +{ + int level; + void *packed = NULL; + + tables->shift = shift; + if (shift == USER4K_SHIFT) { + /* Also test table address bits 13:12, not just leaf addresses. */ + packed = test_alloc_page(test); + if (!packed) + return -ENOMEM; + } + for (level = 0; level < TEST_LEVELS; level++) { + tables->level[level] = packed ? packed + level * USER4K_SIZE : + test_alloc_page(test); + if (!tables->level[level]) + return -ENOMEM; + } + for (level = 0; level < TEST_LEVELS - 1; level++) { + unsigned int index = test_index(TEST_VA, shift, level); + + tables->level[level][index] = + virt_to_phys(tables->level[level + 1]) | PMD_TYPE_TABLE; + } + return 0; +} + +static void user4k_geometry_test(struct kunit *test) +{ + static const unsigned int shifts4k[] = { 39, 30, 21, 12 }; + static const unsigned int shifts16k[] = { 47, 36, 25, 14 }; + static const unsigned int shifts64k[] = { 55, 42, 29, 16 }; + unsigned int level; + + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(48, 12), 4); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(48, 14), 4); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(48, 16), 3); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(42, 16), 2); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(39, 12), 3); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVELS_FOR(47, 14), 3); + for (level = 0; level < TEST_LEVELS; level++) { + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVEL_SHIFT_FOR(level, 12), + shifts4k[level]); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVEL_SHIFT_FOR(level, 14), + shifts16k[level]); + KUNIT_EXPECT_EQ(test, ARM64_HW_PGTABLE_LEVEL_SHIFT(level), + PAGE_SHIFT == 16 ? shifts64k[level] : shifts16k[level]); + KUNIT_EXPECT_EQ(test, test_index(BIT(shifts4k[level]), 12, level), 1U); + KUNIT_EXPECT_EQ(test, test_index(BIT(shifts4k[level]) - 1, 12, level), 0U); + } +} + +static void test_leaf_encoding(struct kunit *test, unsigned int shift) +{ + struct mm_struct *mm = kunit_kzalloc(test, sizeof(*mm), GFP_KERNEL); + const phys_addr_t base = GENMASK_ULL(47, PAGE_SHIFT); + pgprot_t prot = __pgprot(TEST_PROT | PTE_RDONLY); + unsigned int i; + pte_t pte; + + KUNIT_ASSERT_NOT_NULL(test, mm); + pte = phys_pte_mm(mm, base, prot); + KUNIT_EXPECT_EQ(test, pte_val(pte), pte_val(pfn_pte(base >> PAGE_SHIFT, prot))); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), base); + KUNIT_EXPECT_EQ(test, pgprot_val(pte_pgprot_mm(mm, pte)), pgprot_val(prot)); + mm->context.flags |= shift == 12 ? MMCF_USER_4K : MMCF_USER_16K; + for (i = 0; i < (PAGE_SIZE >> shift); i++) { + phys_addr_t phys = base + i * (1UL << shift); + + pte = phys_pte_mm(mm, phys, prot); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), phys); + KUNIT_EXPECT_EQ(test, pte_pfn(pte), (unsigned long)(base >> PAGE_SHIFT)); + KUNIT_EXPECT_EQ(test, pgprot_val(pte_pgprot_mm(mm, pte)), pgprot_val(prot)); + if (i + 1 < (PAGE_SIZE >> shift)) + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte_advance_pfn_mm(mm, pte, 1)), + phys + (1UL << shift)); + pte = pte_modify(pte, __pgprot(TEST_PROT)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), phys); + KUNIT_EXPECT_FALSE(test, pte_val(pte) & PTE_RDONLY); + } +} + +static void user4k_leaf_encoding_test(struct kunit *test) +{ + test_leaf_encoding(test, 12); +} + +static void user16k_leaf_encoding_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative leaves require larger native backing"); + return; + } + test_leaf_encoding(test, 14); +} + + +/* Caller masks all asynchronous exceptions; this function cannot sleep. */ +static void test_switch_ttbr0(u64 ttbr, u64 tcr) +{ + /* The zero root is valid for both tested granules. */ + cpu_set_reserved_ttbr0(); + local_flush_tlb_all(); + write_sysreg(tcr, tcr_el1); + isb(); + write_sysreg(ttbr, ttbr0_el1); + isb(); +} + +static u64 test_translate(unsigned long va, bool write) +{ + if (write) + asm volatile("at s1e0w, %0" : : "r" (va) : "memory"); + else + asm volatile("at s1e0r, %0" : : "r" (va) : "memory"); + isb(); + return read_sysreg_par(); +} + +#define MM_TEST_MAPPINGS 11 + +struct test_mm { + struct kunit *test; + struct mm_struct *mm; + unsigned long address[MM_TEST_MAPPINGS]; + phys_addr_t phys[MM_TEST_MAPPINGS]; +}; + +static pte_t *test_mm_lookup(struct mm_struct *mm, unsigned long addr, + spinlock_t **ptl); + +static void test_mm_hardware(struct test_mm *ctx) +{ + struct kunit *test = ctx->test; + struct vm_area_struct vma = TLB_FLUSH_VMA(ctx->mm, 0); + struct arch_tlbflush_unmap_batch batch = {}; + pte_t *leaves[4]; + u64 invalidated[3][4]; + u64 result[MM_TEST_MAPPINGS], writes[MM_TEST_MAPPINGS]; + u64 old_tcr, old_ttbr0, old_ttbr1, old_par, tcr, kernel; + u64 old_id = atomic64_read(&ctx->mm->context.id); + u64 restored_tcr, restored_ttbr0, retained_ttbr1; + unsigned long flags; + unsigned int i, step; + + for (i = 0; i < ARRAY_SIZE(leaves); i++) { + spinlock_t *ptl; + + leaves[i] = test_mm_lookup(ctx->mm, ctx->address[i], &ptl); + if (leaves[i]) + pte_unmap_unlock(leaves[i], ptl); + KUNIT_ASSERT_NOT_NULL(test, leaves[i]); + } + + preempt_disable(); + flags = local_daif_save(); + old_tcr = read_sysreg(tcr_el1); + old_ttbr0 = read_sysreg(ttbr0_el1); + old_ttbr1 = read_sysreg(ttbr1_el1); + old_par = read_sysreg_par(); + if ((old_tcr & (TCR_EL1_DS | TCR_EL1_TG1_MASK | TCR_EL1_A1)) != + (TCR_TG1_16K | TCR_EL1_A1)) { + local_daif_restore(flags); + preempt_enable(); + kunit_skip(test, "requires 16K TTBR1, DS=0, and A1=1"); + return; + } + tcr = (old_tcr & ~(TCR_EL1_TG0_MASK | TCR_EL1_T0SZ_MASK | TCR_EL1_EPD0)) | + TCR_TG0_4K | TCR_T0SZ(48); + test_switch_ttbr0(virt_to_phys(ctx->mm->pgd), tcr); + /* A1 selects the saved TTBR1 ASID for these private test translations. */ + atomic64_set(&ctx->mm->context.id, old_ttbr1 >> 48); + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + result[i] = test_translate(ctx->address[i], false); + writes[i] = test_translate(ctx->address[i], true); + } + for (step = 0; step < ARRAY_SIZE(invalidated); step++) { + unsigned int first = step == 2 ? 3 : 1; + unsigned int last = step == 1 ? 3 : first + 1; + + for (i = first; i < last; i++) + WRITE_ONCE(*leaves[i], __pte(0)); + switch (step) { + case 0: + __flush_tlb_page(&vma, ctx->address[first] + 1, + TLBF_NONOTIFY | TLBF_NOBROADCAST); + break; + case 1: + __flush_tlb_range(&vma, ctx->address[first], ctx->address[last], + SZ_4K, 3, TLBF_NONOTIFY); + break; + case 2: + arch_tlbbatch_add_pending(&batch, ctx->mm, ctx->address[first], + ctx->address[first] + SZ_4K); + arch_tlbbatch_flush(&batch); + break; + } + isb(); + for (i = 0; i < ARRAY_SIZE(leaves); i++) + invalidated[step][i] = test_translate(ctx->address[i], false); + for (i = first; i < last; i++) + WRITE_ONCE(*leaves[i], __pte(ctx->phys[i] | TEST_PROT)); + dsb(ishst); + isb(); + /* Repopulate before the next independent invalidation. */ + for (i = 0; i < ARRAY_SIZE(leaves); i++) + test_translate(ctx->address[i], false); + } + asm volatile("at s1e1r, %0" : : "r" (ctx->mm->pgd) : "memory"); + isb(); + kernel = read_sysreg_par(); + retained_ttbr1 = read_sysreg(ttbr1_el1); + test_switch_ttbr0(old_ttbr0, old_tcr); + restored_tcr = read_sysreg(tcr_el1); + restored_ttbr0 = read_sysreg(ttbr0_el1); + write_sysreg(old_par, par_el1); + isb(); + atomic64_set(&ctx->mm->context.id, old_id); + local_daif_restore(flags); + preempt_enable(); + + KUNIT_EXPECT_EQ(test, restored_tcr, old_tcr); + KUNIT_EXPECT_EQ(test, restored_ttbr0, old_ttbr0); + KUNIT_EXPECT_EQ(test, retained_ttbr1, old_ttbr1); + KUNIT_EXPECT_EQ(test, kernel & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, kernel & SYS_PAR_EL1_PA, + virt_to_phys(ctx->mm->pgd) & SYS_PAR_EL1_PA); + for (step = 0; step < ARRAY_SIZE(invalidated); step++) { + for (i = 0; i < ARRAY_SIZE(leaves); i++) { + bool hole = step == 0 ? i == 1 : step == 1 ? i == 1 || i == 2 : i == 3; + + KUNIT_EXPECT_EQ(test, invalidated[step][i] & SYS_PAR_EL1_F, (u64)hole); + if (!hole) + KUNIT_EXPECT_EQ(test, invalidated[step][i] & SYS_PAR_EL1_PA, + ctx->phys[i]); + } + } + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + KUNIT_EXPECT_EQ(test, result[i] & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, result[i] & SYS_PAR_EL1_PA, ctx->phys[i]); + KUNIT_EXPECT_EQ(test, writes[i] & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, writes[i] & SYS_PAR_EL1_PA, ctx->phys[i]); + } +} + +struct test_walk { + struct test_mm *ctx; + unsigned long size; + unsigned long holes; + unsigned int present; +}; + +/* Callbacks run under the PTE lock; record errors without invoking KUnit. */ +static int test_walk_pte(pte_t *pte, unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + struct test_walk *state = walk->private; + u64 value = pte_val(ptep_get(pte)); + unsigned int i; + + if (next - addr != state->size) + return -EINVAL; + if (!value) { + state->holes += next - addr; + return 0; + } + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + if (state->ctx->address[i] != addr) + continue; + if (value != (state->ctx->phys[i] | TEST_PROT)) + return -EINVAL; + state->present++; + return 0; + } + return -EINVAL; +} + +static int test_walk_hole(unsigned long addr, unsigned long next, int depth, + struct mm_walk *walk) +{ + struct test_walk *state = walk->private; + + state->holes += next - addr; + return 0; +} + +static const struct mm_walk_ops test_walk_ops = { + .pte_entry = test_walk_pte, + .pte_hole = test_walk_hole, +}; + +static pte_t *test_mm_lookup(struct mm_struct *mm, unsigned long addr, + spinlock_t **ptl) +{ + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + if (pgd_none(*pgd) || pgd_bad(*pgd)) + return NULL; + p4d = p4d_offset_mm(mm, pgd, addr); + if (p4d_none_mm(mm, *p4d) || p4d_bad_mm(mm, *p4d)) + return NULL; + pud = pud_offset_mm(mm, p4d, addr); + if (pud_none(*pud) || pud_bad(*pud)) + return NULL; + pmd = pmd_offset_mm(mm, pud, addr); + if (pmd_none(*pmd) || pmd_bad(*pmd)) + return NULL; + return pte_offset_map_lock(mm, pmd, addr, ptl); +} + +static void test_mm_destroy(void *ptr) +{ + struct test_mm *ctx = ptr; + struct mm_struct *mm = ctx->mm; + struct mmu_gather tlb; + unsigned int i; + + if (!mm) + return; + mmap_write_lock(mm); + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, ctx->address[i], &ptl); + + if (pte) { + /* Test-owned physical pages, not faulted/rmap-owned pages. */ + pte_clear(mm, ctx->address[i], pte); + pte_unmap_unlock(pte, ptl); + } + } + /* Exercise the real generic gather/teardown path, not a test walker. */ + tlb_gather_mmu_fullmm(&tlb, mm); + free_pgd_range(&tlb, 0, 1UL << 48, 0, 0); + tlb_finish_mmu(&tlb); + mmap_write_unlock(mm); + KUNIT_EXPECT_EQ(ctx->test, mm_pgtables_bytes(mm), 0UL); + mmput(mm); + ctx->mm = NULL; +} + +static unsigned int test_mm_backing_pages(struct test_mm *ctx) +{ + struct page *owners[3 * MM_TEST_MAPPINGS]; + unsigned int count = 0, total = 0, i, j, level; + struct mm_struct *mm = ctx->mm; + + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + unsigned long addr = ctx->address[i]; + p4d_t *p4d = p4d_offset_mm(mm, pgd_offset(mm, addr), addr); + pud_t *pud = pud_offset_mm(mm, p4d, addr); + pmd_t *pmd = pmd_offset_mm(mm, pud, addr); + pte_t *pte = pte_offset_kernel_mm(mm, pmd, addr); + struct page *pages[] = { virt_to_page(pud), virt_to_page(pmd), virt_to_page(pte) }; + + for (level = mm_pud_folded(mm) ? 1 : 0; level < ARRAY_SIZE(pages); level++) { + pages[level] = compound_head(pages[level]); + for (j = 0; j < count; j++) + if (owners[j] == pages[level]) + break; + if (j == count) { + owners[count++] = pages[level]; + total += 1U << compound_order(pages[level]); + } + } + } + return total; +} + +/* Only valid for a private, empty mm with no ASID or hardware references. */ +static int test_mm_select_granule(struct mm_struct *mm, unsigned int shift) +{ + pgd_t *native_pgd = mm->pgd, *new_pgd; + unsigned long flag = shift == 12 ? MMCF_USER_4K : + shift == 14 ? MMCF_USER_16K : MMCF_USER_64K; + + mm->context.flags |= flag; + new_pgd = pgd_alloc(mm); + mm->context.flags &= ~flag; + if (!new_pgd) + return -ENOMEM; + pgd_free(mm, native_pgd); + mm->pgd = new_pgd; + mm->context.flags |= flag; + return 0; +} + +static int test_mm_select4k(struct mm_struct *mm) +{ + return test_mm_select_granule(mm, 12); +} + +static void test_mm_tables(struct kunit *test, unsigned int shift, bool hardware) +{ + struct test_mm *ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + bool small = shift != PAGE_SHIFT; + unsigned int pmd_shift = small ? shift + (shift - 3) : PMD_SHIFT; + unsigned int pud_shift = small ? shift + 2 * (shift - 3) : PUD_SHIFT; + unsigned int pgd_shift = small ? (shift == 16 ? pud_shift : + shift + 3 * (shift - 3)) : PGDIR_SHIFT; + unsigned long size = 1UL << shift; + struct mm_struct *mm; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + ctx->test = test; + for (i = 0; i < 4; i++) + ctx->address[i] = TEST_VA + i * size; + ctx->address[4] = (1UL << pmd_shift) - size; + ctx->address[5] = 1UL << pmd_shift; + ctx->address[6] = (1UL << pud_shift) - size; + ctx->address[7] = 1UL << pud_shift; + ctx->address[8] = (1UL << pgd_shift) - size; + ctx->address[9] = 1UL << pgd_shift; + ctx->address[10] = (1UL << 48) - size; + if (pud_shift == pgd_shift) { + ctx->address[6] -= size; + ctx->address[7] += size; + } + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + void *data = test_alloc_granule(test, shift); + + KUNIT_ASSERT_NOT_NULL(test, data); + ctx->phys[i] = virt_to_phys(data) + (shift < PAGE_SHIFT ? (i % (PAGE_SIZE >> shift)) * size : 0); + } + mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, mm); + ctx->mm = mm; + /* Register after the data allocations: detach tables before freeing data. */ + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_mm_destroy, ctx), 0); + KUNIT_EXPECT_EQ(test, mm_page_shift(mm), (unsigned int)PAGE_SHIFT); + if (small) { + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, shift), 0); + KUNIT_EXPECT_EQ(test, (unsigned long)mm->pgd & (size - 1), 0UL); + } + KUNIT_EXPECT_EQ(test, mm_page_shift(mm), shift); + if (small) + KUNIT_EXPECT_EQ(test, mm_pud_folded(mm), shift == 16); + KUNIT_EXPECT_EQ(test, mm_page_size(mm), size); + KUNIT_EXPECT_EQ(test, mm_page_mask(mm), ~(size - 1)); + KUNIT_EXPECT_EQ(test, mm_page_shift(&init_mm), (unsigned int)PAGE_SHIFT); + KUNIT_EXPECT_EQ(test, pmd_addr_end_mm(mm, (1UL << pmd_shift) - 1, + (1UL << pmd_shift) + size), + 1UL << pmd_shift); + KUNIT_EXPECT_EQ(test, pgd_addr_end_mm(mm, ULONG_MAX - size, 0), 0UL); + + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + unsigned long addr = ctx->address[i]; + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d = p4d_alloc(mm, pgd, addr); + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + + KUNIT_ASSERT_NOT_NULL(test, p4d); + pud = pud_alloc(mm, p4d, addr); + KUNIT_ASSERT_NOT_NULL(test, pud); + pmd = pmd_alloc(mm, pud, addr); + KUNIT_ASSERT_NOT_NULL(test, pmd); + pte = pte_alloc_map_lock(mm, pmd, addr, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + /* Do not assert while holding the spinlock/RCU section. */ + set_pte(pte, __pte(ctx->phys[i] | TEST_PROT)); + pte_unmap_unlock(pte, ptl); + } + /* Installed table bytes and physical backing (including cached slots) differ. */ + { + unsigned int shifts[] = { pmd_shift, pud_shift, pgd_shift }; + unsigned int tables = 0, owners = 0; + + for (unsigned int level = 0; level < (mm_pud_folded(mm) ? 2 : 3); level++) { + unsigned int unique = 0; + + for (unsigned int a = 0; a < MM_TEST_MAPPINGS; a++) { + unsigned int b; + + for (b = 0; b < a; b++) + if ((ctx->address[a] >> shifts[level]) == + (ctx->address[b] >> shifts[level])) + break; + unique += b == a; + } + tables += unique; + if (shift < PAGE_SHIFT) + owners += DIV_ROUND_UP(unique, PAGE_SIZE >> shift); + else + owners += unique << (shift - PAGE_SHIFT); + } + KUNIT_EXPECT_EQ(test, mm_pgtables_bytes(mm), tables * size); + KUNIT_EXPECT_EQ(test, test_mm_backing_pages(ctx), owners); + } + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, ctx->address[i], &ptl); + u64 value; + + KUNIT_ASSERT_NOT_NULL(test, pte); + value = pte_val(ptep_get(pte)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, value, ctx->phys[i] | TEST_PROT); + } + { + struct vm_area_struct *vma = vm_area_alloc(mm); + struct test_walk state = { .ctx = ctx, .size = size }; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + vma->vm_start = 0; + vma->vm_end = 1UL << 48; + mmap_write_lock(mm); + ret = walk_page_range_vma(vma, 0, 1UL << 48, &test_walk_ops, &state); + mmap_write_unlock(mm); + vm_area_free(vma); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, state.present, MM_TEST_MAPPINGS); + KUNIT_EXPECT_EQ(test, state.holes, (1UL << 48) - MM_TEST_MAPPINGS * size); + } + if (hardware) + test_mm_hardware(ctx); + { + struct mmu_gather tlb; + unsigned long before = mm_pgtables_bytes(mm); + unsigned long boundary = 1UL << pmd_shift; + + mmap_write_lock(mm); + /* An adjacent VMA's unaligned ceiling must keep the shared table. */ + tlb_gather_mmu_fullmm(&tlb, mm); + free_pgd_range(&tlb, 0, boundary - size, 0, boundary - size); + tlb_finish_mmu(&tlb); + KUNIT_EXPECT_EQ(test, mm_pgtables_bytes(mm), before); + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + spinlock_t *ptl; + pte_t *pte; + + if (ctx->address[i] >= boundary) + continue; + pte = test_mm_lookup(mm, ctx->address[i], &ptl); + if (pte) { + pte_clear(mm, ctx->address[i], pte); + pte_unmap_unlock(pte, ptl); + } + } + tlb_gather_mmu_fullmm(&tlb, mm); + free_pgd_range(&tlb, 0, boundary, 0, boundary); + tlb_finish_mmu(&tlb); + mmap_write_unlock(mm); + KUNIT_EXPECT_EQ(test, mm_pgtables_bytes(mm), before - size); + /* The next table and the higher levels must still be walkable. */ + for (i = 0; i < MM_TEST_MAPPINGS; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, ctx->address[i], &ptl); + u64 value; + + if (ctx->address[i] < boundary) { + if (pte) + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_PTR_EQ(test, pte, NULL); + continue; + } + KUNIT_ASSERT_NOT_NULL(test, pte); + value = pte_val(ptep_get(pte)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, value, ctx->phys[i] | TEST_PROT); + } + } + test_mm_destroy(ctx); +} + +static void user4k_mm_tables_test(struct kunit *test) +{ + test_mm_tables(test, 12, false); +} + +static void native_mm_tables_test(struct kunit *test) +{ + test_mm_tables(test, PAGE_SHIFT, false); +} + +static void user16k_mm_tables_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative tables require larger native backing"); + return; + } + test_mm_tables(test, 14, false); +} + +static void user64k_mm_tables_test(struct kunit *test) +{ + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "64K is already the native geometry"); + return; + } + test_mm_tables(test, 16, false); +} + +static void test_free_vma(void *ptr) +{ + vm_area_free(ptr); +} + +static void user4k_anon_read_fault_test(struct kunit *test) +{ + struct test_mm *ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + struct vm_area_struct *vma; + vm_flags_t vm_flags = VM_READ | VM_MAYREAD; + unsigned int i, j; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + ctx->test = test; + ctx->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, ctx->mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_mm_destroy, ctx), 0); + KUNIT_ASSERT_EQ(test, test_mm_select4k(ctx->mm), 0); + for (i = 0; i < 4; i++) + ctx->address[i] = TEST_VA + i * SZ_4K; + vma = vm_area_alloc(ctx->mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + PAGE_SIZE; + vma_set_anonymous(vma); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + vm_flags |= VM_NOHUGEPAGE; +#endif + vm_flags_init(vma, vm_flags); + vma->vm_page_prot = vm_get_page_prot(vm_flags); + /* This VMA is a private fault fixture, not inserted into the VMA tree. */ + for (i = 0; i < 4; i++) { + vm_fault_t ret; + + mmap_read_lock(ctx->mm); + ret = handle_mm_fault(vma, ctx->address[i] + 1, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(ctx->mm); + KUNIT_ASSERT_EQ(test, ret, (vm_fault_t)0); + for (j = 0; j < 4; j++) { + spinlock_t *ptl; + pte_t *ptep = test_mm_lookup(ctx->mm, ctx->address[j], &ptl); + pte_t pte; + + KUNIT_ASSERT_NOT_NULL(test, ptep); + pte = ptep_get(ptep); + pte_unmap_unlock(ptep, ptl); + if (j > i) { + KUNIT_EXPECT_TRUE(test, pte_none(pte)); + continue; + } + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_TRUE(test, pte_special(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_TRUE(test, is_zero_pfn(pte_pfn(pte))); + } + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(ctx->mm, MM_ANONPAGES), 0L); +} + +static void user4k_mm_hardware_test(struct kunit *test) +{ + if (!user4k_test_enabled) { + kunit_skip(test, "requires arm64.user4k_test=1 in a disposable VM"); + return; + } + if (!system_supports_4kb_granule() || system_supports_cnp() || + cpus_have_final_cap(ARM64_HAS_S1PIE) || + cpus_have_final_cap(ARM64_HAS_S1POE)) { + kunit_skip(test, "requires 4K, no CnP, and direct PTE permissions"); + return; + } + test_mm_tables(test, 12, true); +} + +struct fragment_fixture { + struct mm_struct *mm; + void *table[TEST_SLOTS + 2]; + struct page *pinned[3]; +}; + +static void fragment_fixture_destroy(void *data) +{ + struct fragment_fixture *f = data; + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(f->table); i++) + if (f->table[i]) + arm64_pgtable_free(f->table[i]); + if (f->mm) + mmput(f->mm); + rcu_barrier(); + for (i = 0; i < ARRAY_SIZE(f->pinned); i++) + if (f->pinned[i]) + put_page(f->pinned[i]); +} + +static void test_fragment_lifetime(struct kunit *test, unsigned int shift) +{ + struct fragment_fixture *f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + struct ptdesc *pte_owner; + unsigned int i, slots = PAGE_SIZE >> shift; + unsigned long size = 1UL << shift; + unsigned int read_shift = 0; + int pending_refs; + bool live; + + KUNIT_ASSERT_NOT_NULL(test, f); + f->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, f->mm); + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, fragment_fixture_destroy, f), 0); + /* Direct allocator fixture: the mm's native root is never changed or used. */ + for (i = 0; i < slots + 2; i++) { + enum arm64_user4k_pt_level level = i < slots ? USER4K_PT_PTE : i - slots + 1; + + f->table[i] = arm64_user_pt_alloc_granule(f->mm, level, shift); + KUNIT_ASSERT_NOT_NULL(test, f->table[i]); + KUNIT_EXPECT_TRUE(test, IS_ALIGNED((unsigned long)f->table[i], size)); + KUNIT_EXPECT_EQ(test, arm64_ptep_page_shift(f->table[i]), shift); + KUNIT_EXPECT_EQ(test, arm64_ptep_page_shift(f->table[i] + size - sizeof(pte_t)), shift); + KUNIT_EXPECT_EQ(test, pte_batch_hint(f->table[i], __pte(TEST_PROT | PTE_CONT)), + IS_ENABLED(CONFIG_ARM64_CONTPTE) ? arm64_cont_ptes(shift) : 1U); + if (!i && PAGE_SHIFT == 16) { + void *cached = f->mm->context.user4k_pt_frag[level]; + + KUNIT_EXPECT_PTR_EQ(test, arm64_user_pt_alloc_granule(f->mm, level, + shift == 12 ? 14 : 12), NULL); + KUNIT_EXPECT_PTR_EQ(test, f->mm->context.user4k_pt_frag[level], cached); + } + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(f->table[i], 0, size), NULL); + if (i == 0 || i >= slots) { + unsigned int slot = i == 0 ? 0 : i - slots + 1; + + f->pinned[slot] = virt_to_page(f->table[i]); + get_page(f->pinned[slot]); + } else { + KUNIT_EXPECT_PTR_EQ(test, virt_to_page(f->table[i]), f->pinned[0]); + KUNIT_EXPECT_EQ(test, (unsigned long)f->table[i] - + (unsigned long)f->table[0], i * (unsigned long)size); + } + memset(f->table[i], 0x50 + i, size); + } + for (i = 0; i < slots + 2; i++) { + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(f->table[i], 0x50 + i, size), NULL); + memset(f->table[i], 0, size); + } + /* Pointer-only walkers must inspect all 128 entries of a 16K group. */ + if (IS_ENABLED(CONFIG_ARM64_CONTPTE)) { + pte_t *ptes = f->table[0]; + unsigned int nr = arm64_cont_ptes(shift), j; + pte_t observed; + + for (j = 0; j < nr; j++) + ptes[j] = pte_mkold(pte_mkclean(__pte(TEST_PROT | PTE_CONT | + (SZ_4M + (j << shift))))); + ptes[nr - 1] = pte_mkyoung(pte_mkdirty(ptes[nr - 1])); + observed = ptep_get(ptes + 1); + KUNIT_EXPECT_TRUE(test, pte_young(observed)); + KUNIT_EXPECT_TRUE(test, pte_dirty(observed)); + observed = ptep_get_lockless(ptes + 1); + KUNIT_EXPECT_TRUE(test, pte_young(observed)); + KUNIT_EXPECT_TRUE(test, pte_dirty(observed)); + KUNIT_EXPECT_EQ(test, pte_val(observed) & + (ARM64_USER4K_TABLE_ADDR_MASK & ~(size - 1)), + (u64)(SZ_4M + size)); + memset(ptes, 0, size); + } + KUNIT_EXPECT_PTR_NE(test, f->pinned[0], f->pinned[1]); + KUNIT_EXPECT_PTR_NE(test, f->pinned[1], f->pinned[2]); + pte_owner = page_ptdesc(f->pinned[0]); + KUNIT_EXPECT_EQ(test, atomic_read(&pte_owner->pt_frag_refcount), (int)slots); + /* Refill holes in the same owner, without retaining old table contents. */ + for (i = 1; i < slots; i++) { + void *old = f->table[i]; + + memset(old, 0x6a, size); + arm64_pgtable_free(old); + f->table[i] = arm64_user_pt_alloc_granule(f->mm, USER4K_PT_PTE, shift); + KUNIT_ASSERT_NOT_NULL(test, f->table[i]); + KUNIT_EXPECT_PTR_EQ(test, f->table[i], old); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(f->table[i], 0, size), NULL); + } + /* Begin the reader before queuing a callback, so it cannot finish early. */ + rcu_read_lock(); + arm64_pte_free_defer(f->mm, f->table[0]); + f->table[0] = NULL; + pending_refs = atomic_read(&pte_owner->pt_frag_refcount); + for (i = 1; i < slots - 1; i++) { + arm64_pgtable_free(f->table[i]); + f->table[i] = NULL; + } + /* Pending slot zero keeps the owner alive through this reader. */ + arm64_pgtable_free(f->table[slots - 1]); + f->table[slots - 1] = NULL; + live = folio_test_pgtable(page_folio(f->pinned[0])); + read_shift = arm64_ptep_page_shift(page_address(f->pinned[0])); + rcu_read_unlock(); + KUNIT_EXPECT_EQ(test, pending_refs, (int)slots); + KUNIT_EXPECT_TRUE(test, live); + KUNIT_EXPECT_EQ(test, read_shift, shift); + rcu_barrier(); + KUNIT_EXPECT_FALSE(test, folio_test_pgtable(page_folio(f->pinned[0]))); + KUNIT_EXPECT_EQ(test, page_count(f->pinned[0]), 1); + for (i = slots; i < slots + 2; i++) { + struct ptdesc *pt = virt_to_ptdesc(f->table[i]); + + arm64_pgtable_free(f->table[i]); + f->table[i] = NULL; + /* A wholly empty owner is released without waiting for mm teardown. */ + KUNIT_EXPECT_EQ(test, atomic_read(&pt->pt_frag_refcount), 0); + KUNIT_EXPECT_FALSE(test, folio_test_pgtable(ptdesc_folio(pt))); + } + mmput(f->mm); + f->mm = NULL; + for (i = 1; i < 3; i++) { + KUNIT_EXPECT_FALSE(test, folio_test_pgtable(page_folio(f->pinned[i]))); + KUNIT_EXPECT_EQ(test, page_count(f->pinned[i]), 1); + } +} + +/* Direct table-owner test; no larger-than-native user mm is enabled here. */ +static void user64k_table_owner_test(struct kunit *test) +{ + struct fragment_fixture *f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + unsigned int i, offset, order = 16 - PAGE_SHIFT; + unsigned int shift; + bool live; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, fragment_fixture_destroy, f), 0); + f->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, f->mm); + for (i = 0; i < 3; i++) { + struct ptdesc *pt; + + f->table[i] = arm64_user_pt_alloc_granule(f->mm, i, 16); + KUNIT_ASSERT_NOT_NULL(test, f->table[i]); + pt = virt_to_ptdesc(f->table[i]); + f->pinned[i] = ptdesc_page(pt); + get_page(f->pinned[i]); + KUNIT_EXPECT_TRUE(test, IS_ALIGNED((unsigned long)f->table[i], SZ_64K)); + KUNIT_EXPECT_EQ(test, compound_order(f->pinned[i]), order); + KUNIT_EXPECT_EQ(test, atomic_read(&pt->pt_frag_refcount), 1); + KUNIT_EXPECT_PTR_EQ(test, f->mm->context.user4k_pt_frag[i], NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(f->table[i], 0, SZ_64K), NULL); + /* Include the last entry of every constituent native page. */ + for (offset = PAGE_SIZE - sizeof(pte_t); offset < SZ_64K; offset += PAGE_SIZE) { + void *entry = f->table[i] + offset; + + KUNIT_EXPECT_PTR_EQ(test, virt_to_ptdesc(entry), pt); + KUNIT_EXPECT_EQ(test, arm64_ptep_page_shift(entry), 16U); + KUNIT_EXPECT_EQ(test, pte_batch_hint(entry, __pte(TEST_PROT | PTE_CONT)), + 32U - ((offset / sizeof(pte_t)) % 32)); + if (i == USER4K_PT_PTE) + KUNIT_EXPECT_PTR_EQ(test, ptep_lockptr(f->mm, entry), + ptep_lockptr(f->mm, f->table[i])); + if (i == USER4K_PT_PMD) + KUNIT_EXPECT_PTR_EQ(test, pmd_lockptr(f->mm, entry), + pmd_lockptr(f->mm, f->table[i])); + } + memset(f->table[i], 0x51 + i, SZ_64K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(f->table[i], 0x51 + i, SZ_64K), NULL); + memset(f->table[i], 0, SZ_64K); + } + /* Contiguous readers on the last native page must still use 64K geometry. */ + if (IS_ENABLED(CONFIG_ARM64_CONTPTE)) { + pte_t *ptes = f->table[0] + SZ_64K - 32 * sizeof(pte_t); + pte_t observed; + unsigned int j; + + for (j = 0; j < 32; j++) + ptes[j] = pte_mkold(pte_mkclean(__pte(TEST_PROT | PTE_CONT | + (SZ_4M + (j << 16))))); + ptes[31] = pte_mkyoung(pte_mkdirty(ptes[31])); + observed = ptep_get(ptes + 1); + KUNIT_EXPECT_TRUE(test, pte_young(observed)); + KUNIT_EXPECT_TRUE(test, pte_dirty(observed)); + observed = ptep_get_lockless(ptes + 1); + KUNIT_EXPECT_TRUE(test, pte_young(observed)); + KUNIT_EXPECT_TRUE(test, pte_dirty(observed)); + KUNIT_EXPECT_EQ(test, pte_val(observed) & + (ARM64_USER4K_TABLE_ADDR_MASK & ~(SZ_64K - 1UL)), + (u64)(SZ_4M + SZ_64K)); + memset(f->table[0], 0, SZ_64K); + } + rcu_read_lock(); + arm64_pte_free_defer(f->mm, f->table[0]); + live = folio_test_pgtable(page_folio(f->pinned[0])); + shift = arm64_ptep_page_shift(f->table[0] + SZ_64K - sizeof(pte_t)); + f->table[0] = NULL; + rcu_read_unlock(); + KUNIT_EXPECT_TRUE(test, live); + KUNIT_EXPECT_EQ(test, shift, 16U); + rcu_barrier(); + for (i = 0; i < 3; i++) { + if (f->table[i]) { + arm64_pgtable_free(f->table[i]); + f->table[i] = NULL; + } + KUNIT_EXPECT_FALSE(test, folio_test_pgtable(page_folio(f->pinned[i]))); + KUNIT_EXPECT_EQ(test, page_count(f->pinned[i]), 1); + KUNIT_EXPECT_EQ(test, compound_order(f->pinned[i]), order); + } +} + +static void user4k_fragment_lifetime_test(struct kunit *test) +{ + test_fragment_lifetime(test, 12); +} + +static void user16k_fragment_lifetime_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K fragments require larger native backing"); + return; + } + test_fragment_lifetime(test, 14); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void test_put_mm(void *ptr) +{ + mmput(ptr); +} + +static void native_pgtable_deposit_test(struct kunit *test) +{ + struct mm_struct *mm = mm_alloc(); + pgtable_t first, second, out1, out2; + pmd_t *pmd; + spinlock_t *ptl; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_put_mm, mm), 0); + pmd = pmd_alloc_one(mm, 0); + KUNIT_ASSERT_NOT_NULL(test, pmd); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, arm64_pgtable_free, pmd), 0); + first = pte_alloc_one(mm); + KUNIT_ASSERT_NOT_NULL(test, first); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, arm64_pgtable_free, first), 0); + second = pte_alloc_one(mm); + KUNIT_ASSERT_NOT_NULL(test, second); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, arm64_pgtable_free, second), 0); + ptl = pmd_lock(mm, pmd); + pgtable_trans_huge_deposit(mm, pmd, first); + pgtable_trans_huge_deposit(mm, pmd, second); + out1 = pgtable_trans_huge_withdraw(mm, pmd); + out2 = pgtable_trans_huge_withdraw(mm, pmd); + spin_unlock(ptl); + KUNIT_EXPECT_PTR_EQ(test, out1, second); + KUNIT_EXPECT_PTR_EQ(test, out2, first); +} +#endif + +static void test_data_slot_put(void *ptr) +{ + mm_subpage_put(ptr); +} + +static void test_data_pool_free(void *ptr) +{ + mm_subpage_pool_close(ptr); + mm_subpage_pool_put(ptr); +} + +static bool test_alloc_data_slots(struct kunit *test, struct test_mm *ctx) +{ + struct mm_subpage_pool *pool = mm_subpage_pool_create(GFP_KERNEL); + struct folio *folio; + unsigned int i; + int ret; + + if (!pool) + return false; + if (kunit_add_action_or_reset(test, test_data_pool_free, pool)) + return false; + folio = folio_alloc(GFP_KERNEL, 0); + if (!folio) + return false; + ret = mm_subpage_pool_add_folio(pool, folio, GFP_KERNEL); + if (ret) { + folio_put(folio); + return false; + } + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc(pool); + + if (IS_ERR(slot)) + return false; + if (kunit_add_action_or_reset(test, test_data_slot_put, slot)) + return false; + ctx->phys[i] = mm_subpage_phys(slot); + } + return true; +} + +/* These private mappings are borrowed only for bounded, fault-disabled uaccess. */ +static struct test_mm *test_usercopy_mm(struct kunit *test, bool small) +{ + struct test_mm *ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + unsigned long size = small ? SZ_4K : PAGE_SIZE; + unsigned int i; + + if (!ctx) + return NULL; + ctx->test = test; + if (small && !test_alloc_data_slots(test, ctx)) + return NULL; + for (i = 0; i < 4; i++) { + if (!small) { + void *data = test_alloc_page(test); + + if (!data) + return NULL; + ctx->phys[i] = virt_to_phys(data); + } + ctx->address[i] = TEST_VA + i * size; + memset(phys_to_virt(ctx->phys[i]), (small ? 0x40 : 0x80) + i, 64); + } + ctx->mm = mm_alloc(); + if (!ctx->mm) + return NULL; + if (kunit_add_action_or_reset(test, test_mm_destroy, ctx)) + return NULL; + if (small && test_mm_select4k(ctx->mm)) + return NULL; + for (i = 0; i < 4; i++) { + unsigned long addr = ctx->address[i]; + pgd_t *pgd = pgd_offset(ctx->mm, addr); + p4d_t *p4d = p4d_alloc(ctx->mm, pgd, addr); + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + + if (!p4d) + return NULL; + pud = pud_alloc(ctx->mm, p4d, addr); + if (!pud) + return NULL; + pmd = pmd_alloc(ctx->mm, pud, addr); + if (!pmd) + return NULL; + pte = pte_alloc_map_lock(ctx->mm, pmd, addr, &ptl); + if (!pte) + return NULL; + if (i != 2) + set_pte(pte, phys_pte_mm(ctx->mm, ctx->phys[i], + __pgprot(TEST_PROT | (i ? 0 : PTE_RDONLY)))); + pte_unmap_unlock(pte, ptl); + } + return ctx; +} + +static void user4k_rmap_walk_test(struct kunit *test) +{ + struct test_mm *ctx = test_usercopy_mm(test, true); + struct vm_area_struct *vma; + unsigned int i, j, sync; + spinlock_t *ptl; + pte_t *pte; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + vma = vm_area_alloc(ctx->mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + 2 * PAGE_SIZE; + /* A later alias must not extend a native-owner range walk's bound. */ + ctx->address[4] = TEST_VA + PAGE_SIZE; + ctx->phys[4] = ctx->phys[0]; + pte = test_mm_lookup(ctx->mm, ctx->address[4], &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + set_pte(pte, phys_pte_mm(ctx->mm, ctx->phys[4], __pgprot(TEST_PROT))); + pte_unmap_unlock(pte, ptl); + /* Same physical slot at a different virtual byte offset and native index. */ + ctx->address[5] = TEST_VA + PAGE_SIZE + SZ_4K; + ctx->phys[5] = ctx->phys[3]; + pte = test_mm_lookup(ctx->mm, ctx->address[5], &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + set_pte(pte, phys_pte_mm(ctx->mm, ctx->phys[5], __pgprot(TEST_PROT))); + pte_unmap_unlock(pte, ptl); + for (sync = 0; sync < 2; sync++) { + struct page_vma_mapped_walk whole = { + .pfn = ctx->phys[0] >> PAGE_SHIFT, + .nr_pages = 1, + .vma = vma, + .address = TEST_VA, + .flags = sync ? PVMW_SYNC : 0, + }; + unsigned int count = 0; + bool valid = true; + + /* A native-owner walk must visit every mapped 4K occupant. */ + while (page_vma_mapped_walk(&whole)) { + unsigned int index = (whole.address - TEST_VA) / SZ_4K; + + if (index >= 4 || index == 2 || !whole.pte || + pte_phys_mm(ctx->mm, ptep_get(whole.pte)) != ctx->phys[index]) + valid = false; + if (++count > 4) { + page_vma_mapped_walk_done(&whole); + break; + } + } + KUNIT_EXPECT_TRUE(test, valid); + KUNIT_EXPECT_EQ(test, count, 3U); + for (i = 0; i < 4; i++) { + for (j = 0; j < 6; j++) { + struct rmap_walk_range range = { + .address = ctx->address[j], + .subpage = true, + .subpage_offset = offset_in_page(ctx->phys[i]), + }; + DEFINE_FOLIO_RMAP_WALK(one, page_folio(pfn_to_page(PHYS_PFN(ctx->phys[i]))), + vma, range, sync ? PVMW_SYNC : 0); + struct mm_subpage *slot = NULL; + bool found = page_vma_mapped_walk(&one); + phys_addr_t phys = 0; + + if (found) { + phys = pte_phys_mm(ctx->mm, ptep_get(one.pte)); + slot = mm_subpage_get_from_phys(phys); + page_vma_mapped_walk_done(&one); + } + KUNIT_EXPECT_EQ(test, found, (i == j && j != 2) || + (j == 4 && i == 0) || (j == 5 && i == 3)); + if (found) { + KUNIT_EXPECT_EQ(test, phys, ctx->phys[i]); + KUNIT_EXPECT_NOT_NULL(test, slot); + if (slot) { + KUNIT_EXPECT_EQ(test, mm_subpage_phys(slot), phys); + mm_subpage_put(slot); + } + } + } + } + } +} + +static int test_map_alias(struct test_mm *ctx, unsigned int index, + unsigned long addr, phys_addr_t phys) +{ + struct mm_struct *mm = ctx->mm; + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d = p4d_alloc(mm, pgd, addr); + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + + ctx->address[index] = addr; + ctx->phys[index] = phys; + if (!p4d) + return -ENOMEM; + pud = pud_alloc(mm, p4d, addr); + if (!pud) + return -ENOMEM; + pmd = pmd_alloc(mm, pud, addr); + if (!pmd) + return -ENOMEM; + pte = pte_alloc_map_lock(mm, pmd, addr, &ptl); + if (!pte) + return -ENOMEM; + set_pte(pte, pte_mkspecial(phys_pte_mm(mm, phys, __pgprot(TEST_PROT)))); + pte_unmap_unlock(pte, ptl); + return 0; +} + +#ifdef CONFIG_PERF_EVENTS +static void test_perf_page_size(struct kunit *test, unsigned int shift) +{ + const unsigned long addresses[] = { + SZ_1M, SZ_1G + SZ_1M, (1UL << 39) + SZ_1M, + }; + struct test_mm *ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + void *data = test_alloc_granule(test, PAGE_SHIFT); + unsigned long flags; + u64 present, absent; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + KUNIT_ASSERT_NOT_NULL(test, data); + ctx->test = test; + ctx->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, ctx->mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_mm_destroy, ctx), 0); + if (shift != PAGE_SHIFT) + KUNIT_ASSERT_EQ(test, test_mm_select_granule(ctx->mm, shift), 0); + for (i = 0; i < ARRAY_SIZE(addresses); i++) + KUNIT_ASSERT_EQ(test, test_map_alias(ctx, i, addresses[i], virt_to_phys(data)), 0); + for (i = 0; i < ARRAY_SIZE(addresses); i++) { + local_irq_save(flags); + present = perf_get_pgtable_size(ctx->mm, addresses[i]); + absent = perf_get_pgtable_size(ctx->mm, addresses[i] + (1UL << shift)); + local_irq_restore(flags); + KUNIT_EXPECT_EQ(test, present, 1ULL << shift); + KUNIT_EXPECT_EQ(test, absent, 0ULL); + } +} + +static void user4k_perf_page_size_test(struct kunit *test) +{ + test_perf_page_size(test, 12); +} + +static void user16k_perf_page_size_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_perf_page_size(test, 14); +} + +static void native_perf_page_size_test(struct kunit *test) +{ + test_perf_page_size(test, PAGE_SHIFT); +} +#endif + +/* Physical-range lookup only; coarse anonymous ownership is a separate step. */ +static void user64k_rmap_lookup_test(struct kunit *test) +{ + struct test_mm *ctx; + struct vm_area_struct *vma; + void *data; + unsigned long pfn, per_leaf = SZ_64K / PAGE_SIZE; + unsigned int mode, sync, i; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires userspace leaf larger than native page"); + return; + } + ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, ctx); + ctx->test = test; + data = test_alloc_granule(test, 17); + KUNIT_ASSERT_NOT_NULL(test, data); + pfn = virt_to_phys(data) >> PAGE_SHIFT; + ctx->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, ctx->mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_mm_destroy, ctx), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(ctx->mm, 16), 0); + vma = vm_area_alloc(ctx->mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = (1UL << 29) - SZ_64K; + vma->vm_end = vma->vm_start + 4 * SZ_64K; + vma->vm_pgoff = 4096; + page_folio(virt_to_page(data))->index = vma->vm_pgoff; + for (i = 0; i < 3; i++) + KUNIT_ASSERT_EQ(test, test_map_alias(ctx, i, vma->vm_start + i * SZ_64K, + virt_to_phys(data) + (i == 1 ? SZ_64K : 0)), 0); + + for (mode = 0; mode < 2; mode++) { + if (mode) { + for (i = 0; i < 3; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(ctx->mm, ctx->address[i], &ptl); + + KUNIT_ASSERT_NOT_NULL(test, pte); + set_pte(pte, swp_entry_to_pte(make_readable_migration_entry( + ctx->phys[i] >> PAGE_SHIFT))); + pte_unmap_unlock(pte, ptl); + } + } + for (sync = 0; sync < 2; sync++) { + unsigned int flags = (sync ? PVMW_SYNC : 0) | + (mode ? PVMW_MIGRATION : 0); + + for (i = 0; i < 2 * per_leaf; i++) { + struct page_vma_mapped_walk walk = { + .pfn = pfn + i, .nr_pages = 1, + /* Deliberately invalid pgoff: single-page callers may omit it. */ + .pgoff = 0, .vma = vma, + .address = vma->vm_start + i * PAGE_SIZE, + .flags = flags, + }; + unsigned int count = 0; + bool valid = true; + + while (page_vma_mapped_walk(&walk)) { + valid &= walk.pte && walk.address == + vma->vm_start + (i / per_leaf) * SZ_64K; + if (++count > 1) { + page_vma_mapped_walk_done(&walk); + break; + } + } + KUNIT_EXPECT_TRUE(test, valid); + KUNIT_EXPECT_EQ(test, count, 1U); +#ifdef CONFIG_MEMORY_FAILURE + if (!mode) + KUNIT_EXPECT_EQ(test, page_mapped_in_vma(pfn_to_page(pfn + i), vma), + vma->vm_start + i * PAGE_SIZE); +#endif + } + /* Two native pages straddle a coarse-leaf and PTE-table boundary. */ + { + struct page_vma_mapped_walk walk = { + .pfn = pfn + per_leaf - 1, .nr_pages = 2, + .pgoff = vma->vm_pgoff + per_leaf - 1, .vma = vma, + .address = vma->vm_start + SZ_64K - PAGE_SIZE, + .flags = flags, + }; + unsigned int count = 0; + bool valid = true; + + KUNIT_EXPECT_EQ(test, vma_address_end(&walk), vma->vm_start + 2 * SZ_64K); + while (page_vma_mapped_walk(&walk)) { + valid &= walk.address == vma->vm_start + count * SZ_64K; + if (++count > 2) { + page_vma_mapped_walk_done(&walk); + break; + } + } + KUNIT_EXPECT_TRUE(test, valid); + KUNIT_EXPECT_EQ(test, count, 2U); + KUNIT_EXPECT_TRUE(test, walk.flags & PVMW_PGTABLE_CROSSED); + } + for (i = 0; i < 2; i++) { + struct page_vma_mapped_walk walk = { + .pfn = i ? pfn + 2 * per_leaf : pfn - 1, + .nr_pages = 1, .vma = vma, + .address = vma->vm_start, .flags = flags, + }; + bool found = page_vma_mapped_walk(&walk); + + if (found) + page_vma_mapped_walk_done(&walk); + KUNIT_EXPECT_FALSE(test, found); + } + } + } +} + +static void user4k_rmap_boundary_test(struct kunit *test) +{ + struct test_mm *ctx = test_usercopy_mm(test, true); + struct vm_area_struct *vma; + unsigned int i, count = 0; + struct page_vma_mapped_walk walk; + bool valid = true; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + vma = vm_area_alloc(ctx->mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = SZ_2M - SZ_4K; + vma->vm_end = vma->vm_start + PAGE_SIZE; + vma->vm_pgoff = 37; + page_folio(pfn_to_page(ctx->phys[0] >> PAGE_SHIFT))->index = vma->vm_pgoff; + for (i = 0; i < 3; i++) + KUNIT_ASSERT_EQ(test, test_map_alias(ctx, 4 + i, vma->vm_start + i * SZ_4K, + ctx->phys[i]), 0); + walk = (struct page_vma_mapped_walk) { + .pfn = ctx->phys[0] >> PAGE_SHIFT, + .pgoff = vma->vm_pgoff, + .nr_pages = 1, + .vma = vma, + .address = vma->vm_start, + .flags = PVMW_SYNC, + }; + while (page_vma_mapped_walk(&walk)) { + if (count >= 3 || walk.address != vma->vm_start + count * SZ_4K || + pte_phys_mm(ctx->mm, ptep_get(walk.pte)) != ctx->phys[count]) + valid = false; + if (++count > 3) { + page_vma_mapped_walk_done(&walk); + break; + } + } + KUNIT_EXPECT_TRUE(test, valid); + KUNIT_EXPECT_EQ(test, count, 3U); + KUNIT_EXPECT_TRUE(test, walk.flags & PVMW_PGTABLE_CROSSED); +#ifdef CONFIG_MEMORY_FAILURE + { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(ctx->mm, vma->vm_start, &ptl); + + KUNIT_ASSERT_NOT_NULL(test, pte); + ptep_get_and_clear(ctx->mm, vma->vm_start, pte); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, + page_mapped_in_vma(pfn_to_page(ctx->phys[0] >> PAGE_SHIFT), vma), + vma->vm_start + SZ_4K); + } +#endif +} + +static void native_rmap_walk_test(struct kunit *test) +{ + struct test_mm *ctx = test_usercopy_mm(test, false); + struct vm_area_struct *vma; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + vma = vm_area_alloc(ctx->mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + 4 * PAGE_SIZE; + for (i = 0; i < 4; i++) { + struct page_vma_mapped_walk walk = { + .pfn = ctx->phys[i] >> PAGE_SHIFT, + .nr_pages = 1, + .vma = vma, + .address = ctx->address[i], + .flags = PVMW_SYNC, + }; + unsigned int count = 0; + bool valid = true; + + while (page_vma_mapped_walk(&walk)) { + if (walk.address != ctx->address[i] || + pte_phys_mm(ctx->mm, ptep_get(walk.pte)) != ctx->phys[i]) + valid = false; + if (++count > 1) { + page_vma_mapped_walk_done(&walk); + break; + } + } + KUNIT_EXPECT_TRUE(test, valid); + KUNIT_EXPECT_EQ(test, count, i != 2 ? 1U : 0U); +#ifdef CONFIG_MEMORY_FAILURE + page_folio(pfn_to_page(ctx->phys[i] >> PAGE_SHIFT))->index = i; + KUNIT_EXPECT_EQ(test, page_mapped_in_vma(pfn_to_page(ctx->phys[i] >> PAGE_SHIFT), vma), + i != 2 ? ctx->address[i] : (unsigned long)-EFAULT); +#endif + } +} + +static void user4k_vma_offsets_test(struct kunit *test) +{ + struct vm_area_struct *vma = kunit_kzalloc(test, sizeof(*vma), GFP_KERNEL); + unsigned int slot, length, step; + + KUNIT_ASSERT_NOT_NULL(test, vma); + vma->vm_start = (1UL << 48) - 8 * PAGE_SIZE; + for (slot = 0; slot < TEST_SLOTS; slot++) { + for (length = 1; length <= 8; length++) { + struct vm_page_offset initial = { ULONG_MAX - 3, slot * SZ_4K }; + struct vm_page_offset pos; + + vma->vm_end = vma->vm_start + length * SZ_4K; + vma_set_page_offset(vma, initial); + KUNIT_EXPECT_EQ(test, vma_last_pgoff(vma), + initial.index + (slot + length - 1) / TEST_SLOTS); + for (step = 0; step < length; step++) { + unsigned long addr = vma->vm_start + step * SZ_4K; + + pos = vma_page_offset_at(vma, addr); + KUNIT_EXPECT_EQ(test, pos.index, initial.index + (slot + step) / TEST_SLOTS); + KUNIT_EXPECT_EQ(test, pos.offset, ((slot + step) % TEST_SLOTS) * (unsigned int)SZ_4K); + KUNIT_EXPECT_EQ(test, linear_page_index(vma, addr), pos.index); + KUNIT_EXPECT_EQ(test, vma_address_at_offset(vma, pos), addr); + } + pos = vma_page_offset_at(vma, vma->vm_end); + KUNIT_EXPECT_EQ(test, vma_address_at_offset(vma, pos), (unsigned long)-EFAULT); + pos = initial; + pos.index--; + KUNIT_EXPECT_EQ(test, vma_address_at_offset(vma, pos), (unsigned long)-EFAULT); + if (slot) { + pos = initial; + pos.offset--; + KUNIT_EXPECT_EQ(test, vma_address_at_offset(vma, pos), (unsigned long)-EFAULT); + } + } + } +} + +static void user4k_vma_intervals_test(struct kunit *test) +{ + struct vm_area_struct *vmas = kunit_kcalloc(test, 3, sizeof(*vmas), GFP_KERNEL); + struct rb_root_cached root = RB_ROOT_CACHED; + struct vm_area_struct *vma; + unsigned int i, count = 0; + + KUNIT_ASSERT_NOT_NULL(test, vmas); + for (i = 0; i < 3; i++) { + vmas[i].vm_start = TEST_VA + i * SZ_8K; + vmas[i].vm_end = vmas[i].vm_start + SZ_4K; + vma_set_page_offset(&vmas[i], (struct vm_page_offset) { 77, i * SZ_4K }); + vma_interval_tree_insert(&vmas[i], &root); + } + vma_interval_tree_foreach(vma, &root, 77, 77) { + unsigned long address = vma_address_at_offset(vma, + (struct vm_page_offset) { 77, SZ_4K }); + + KUNIT_EXPECT_EQ(test, address, + vma == &vmas[1] ? vma->vm_start : (unsigned long)-EFAULT); + count++; + } + KUNIT_EXPECT_EQ(test, count, 3U); + KUNIT_EXPECT_PTR_EQ(test, vma_interval_tree_iter_first(&root, 76, 76), NULL); + KUNIT_EXPECT_PTR_EQ(test, vma_interval_tree_iter_first(&root, 78, 78), NULL); + for (i = 0; i < 3; i++) + vma_interval_tree_remove(&vmas[i], &root); + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&root.rb_root)); +} + +static void user4k_vma_mm_free(void *arg) +{ + mmput(arg); +} + +/* These private mms have no concurrent VMA writers after dropping the lock. */ +static struct vm_area_struct *test_vma_lookup(struct mm_struct *mm, unsigned long addr) +{ + struct vm_area_struct *vma; + + mmap_read_lock(mm); + vma = vma_lookup(mm, addr); + mmap_read_unlock(mm); + return vma; +} + +static void user4k_vma_split_test(struct kunit *test) +{ + struct mm_struct *mm = mm_alloc(); + struct vm_area_struct *vma, *middle, *part; + unsigned long start = TEST_VA + SZ_4K; + vma_flags_t flags; + int ret; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + KUNIT_ASSERT_EQ(test, test_mm_select4k(mm), 0); + vma = vm_area_alloc(mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + /* Dummy vm_ops keep insert_vm_struct from replacing this file-like offset. */ + vma->vm_start = start; + vma->vm_end = start + 3 * SZ_4K; + vm_flags_init(vma, VM_READ | VM_MAYREAD | VM_MAYWRITE); + vma_set_page_offset(vma, (struct vm_page_offset) { 77, 3 * SZ_4K }); + mmap_write_lock(mm); + ret = insert_vm_struct(mm, vma); + if (ret) { + mmap_write_unlock(mm); + vm_area_free(vma); + KUNIT_FAIL(test, "insert_vm_struct: %d", ret); + return; + } + { + VMA_ITERATOR(vmi, mm, start); + + vma_iter_load(&vmi); + flags = vma->flags; + vma_flags_set(&flags, VMA_WRITE_BIT); + middle = vma_modify_flags(&vmi, vma, vma, start + SZ_4K, + start + 2 * SZ_4K, &flags); + } + mmap_write_unlock(mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, middle); + KUNIT_EXPECT_EQ(test, mm->map_count, 3); + for (i = 0; i < 3; i++) { + part = test_vma_lookup(mm, start + i * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, part); + KUNIT_EXPECT_EQ(test, part->vm_start, start + i * SZ_4K); + KUNIT_EXPECT_EQ(test, part->vm_end, start + (i + 1) * SZ_4K); + KUNIT_EXPECT_EQ(test, part->vm_pgoff, 77UL + (i + 3) / TEST_SLOTS); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(part), ((i + 3) % TEST_SLOTS) * (unsigned int)SZ_4K); + } +} + +static struct mm_struct *test_vma_mm_granule(struct kunit *test, unsigned int shift) +{ + struct mm_struct *mm = mm_alloc(); + + if (!mm) + return NULL; + if (kunit_add_action_or_reset(test, user4k_vma_mm_free, mm)) + return NULL; + if (test_mm_select_granule(mm, shift)) + return NULL; + return mm; +} + +static struct mm_struct *test_vma_mm(struct kunit *test) +{ + return test_vma_mm_granule(test, 12); +} + +static struct vm_area_struct *test_vma_add(struct mm_struct *mm, unsigned long start, + unsigned long len, struct vm_page_offset pos, + vm_flags_t flags) +{ + struct vm_area_struct *vma = vm_area_alloc(mm); + int ret; + + if (!vma) + return ERR_PTR(-ENOMEM); + vma->vm_start = start; + vma->vm_end = start + len; + vm_flags_init(vma, flags | VM_MAYREAD | VM_MAYWRITE); + vma_set_page_offset(vma, pos); + mmap_write_lock(mm); + ret = insert_vm_struct(mm, vma); + mmap_write_unlock(mm); + if (ret) { + vm_area_free(vma); + return ERR_PTR(ret); + } + return vma; +} + +static void user4k_vma_merge_test(struct kunit *test) +{ + unsigned long start = TEST_VA + SZ_4K; + unsigned int scenario, i; + + for (scenario = 0; scenario < 4; scenario++) { + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vmas[3], *merged; + struct vm_page_offset initial = { 77, 3 * SZ_4K }; + struct vm_page_offset expected; + unsigned long left = scenario >= 2 ? SZ_4K : 0; + unsigned long right = scenario == 1 || scenario == 3 ? 2 * SZ_4K : 3 * SZ_4K; + vma_flags_t flags; + + KUNIT_ASSERT_NOT_NULL(test, mm); + for (i = 0; i < 3; i++) { + struct vm_page_offset pos = vm_page_offset_add(initial, i * SZ_4K); + vm_flags_t bits = VM_READ; + + if (i == 1) + bits |= VM_WRITE; + if ((!i && scenario >= 2) || (i == 2 && scenario == 1)) + bits |= VM_EXEC; + /* Same native index, wrong byte offset: this must not merge. */ + if (i == 2 && scenario == 3) + pos.offset = 0; + vmas[i] = test_vma_add(mm, start + i * SZ_4K, SZ_4K, pos, bits); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vmas[i]); + } + flags = vmas[1]->flags; + vma_flags_clear(&flags, VMA_WRITE_BIT); + mmap_write_lock(mm); + { + VMA_ITERATOR(vmi, mm, start + SZ_4K); + + vma_iter_load(&vmi); + merged = vma_modify_flags(&vmi, vmas[0], vmas[1], start + SZ_4K, + start + 2 * SZ_4K, &flags); + if (!IS_ERR(merged)) { + vma_start_write(merged); + merged->flags = flags; + } + } + mmap_write_unlock(mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, merged); + expected = vm_page_offset_add(initial, left); + KUNIT_EXPECT_EQ(test, mm->map_count, scenario == 0 ? 1 : scenario == 3 ? 3 : 2); + KUNIT_EXPECT_EQ(test, merged->vm_start, start + left); + KUNIT_EXPECT_EQ(test, merged->vm_end, start + right); + KUNIT_EXPECT_EQ(test, merged->vm_pgoff, expected.index); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(merged), expected.offset); + } +} + +static void user4k_vma_partial_merge_test(struct kunit *test) +{ + unsigned long start = TEST_VA; + unsigned int side; + + for (side = 0; side < 2; side++) { + struct mm_struct *mm = test_vma_mm(test); + struct vm_page_offset initial = { 77, 2 * SZ_4K }; + struct vm_area_struct *first, *second, *merged, *remaining; + unsigned long first_len = side ? SZ_4K : 2 * SZ_4K; + vma_flags_t flags; + struct vm_page_offset expected; + + KUNIT_ASSERT_NOT_NULL(test, mm); + first = test_vma_add(mm, start, first_len, initial, VM_READ | (side ? 0 : VM_WRITE)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, first); + second = test_vma_add(mm, start + first_len, 3 * SZ_4K - first_len, + vm_page_offset_add(initial, first_len), VM_READ | (side ? VM_WRITE : 0)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, second); + flags = side ? first->flags : second->flags; + mmap_write_lock(mm); + { + VMA_ITERATOR(vmi, mm, start + SZ_4K); + + vma_iter_load(&vmi); + merged = vma_modify_flags(&vmi, first, side ? second : first, + start + SZ_4K, start + 2 * SZ_4K, &flags); + } + mmap_write_unlock(mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, merged); + KUNIT_EXPECT_EQ(test, mm->map_count, 2); + KUNIT_EXPECT_EQ(test, merged->vm_start, start + (side ? 0 : SZ_4K)); + KUNIT_EXPECT_EQ(test, merged->vm_end, start + (side ? 2 : 3) * SZ_4K); + expected = vm_page_offset_add(initial, side ? 0 : SZ_4K); + KUNIT_EXPECT_EQ(test, merged->vm_pgoff, expected.index); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(merged), expected.offset); + remaining = test_vma_lookup(mm, start + (side ? 2 * SZ_4K : 0)); + KUNIT_ASSERT_NOT_NULL(test, remaining); + expected = vm_page_offset_add(initial, side ? 2 * SZ_4K : 0); + KUNIT_EXPECT_EQ(test, remaining->vm_pgoff, expected.index); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(remaining), expected.offset); + } +} + +static void user4k_vma_copy_shrink_test(struct kunit *test) +{ + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *source, *dest; + struct vm_page_offset initial = { 77, 3 * SZ_4K }; + unsigned long start = TEST_VA + SZ_4K; + unsigned long target = TEST_VA + 16 * PAGE_SIZE + SZ_4K; + bool need_locks = false; + unsigned int i; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + source = test_vma_add(mm, start, PAGE_SIZE, initial, VM_READ); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, source); + for (i = 0; i < 3; i++) { + unsigned long addr = i == 2 ? target - SZ_4K : target + i * SZ_4K; + struct vm_page_offset pos = vm_page_offset_add(initial, i == 2 ? 0 : (i + 1) * SZ_4K); + + mmap_write_lock(mm); + dest = copy_vma(&source, addr, SZ_4K, pos, &need_locks); + mmap_write_unlock(mm); + KUNIT_ASSERT_NOT_NULL(test, dest); + KUNIT_EXPECT_EQ(test, mm->map_count, 2); + KUNIT_EXPECT_EQ(test, dest->vm_start, i == 2 ? target - SZ_4K : target); + KUNIT_EXPECT_EQ(test, dest->vm_end, i ? target + 2 * SZ_4K : target + SZ_4K); + KUNIT_EXPECT_EQ(test, dest->vm_pgoff, 77UL + (i == 2 ? 3UL : 4UL) / TEST_SLOTS); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(dest), ((i == 2 ? 3U : 4U) % TEST_SLOTS) * SZ_4K); + KUNIT_EXPECT_EQ(test, source->vm_pgoff, initial.index); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(source), initial.offset); + } + mmap_write_lock(mm); + { + VMA_ITERATOR(vmi, mm, dest->vm_start); + struct vm_page_offset pos = vma_page_offset_at(dest, target); + + vma_iter_load(&vmi); + ret = vma_shrink(&vmi, dest, target, dest->vm_end, pos); + } + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, dest->vm_start, target); + KUNIT_EXPECT_EQ(test, dest->vm_end, target + 2 * SZ_4K); + KUNIT_EXPECT_EQ(test, dest->vm_pgoff, 77UL + 4 / TEST_SLOTS); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(dest), (4 % TEST_SLOTS) * SZ_4K); +} + +struct anon_slot_fixture { + struct mm_struct *mm; + struct vm_area_struct *vma; + struct mm_subpage_pool *pool; + struct mm_subpage *slots[4]; + bool mapped[4]; +}; + +static void anon_slot_fixture_free(void *arg) +{ + struct anon_slot_fixture *f = arg; + struct mmu_gather tlb; + unsigned int i; + + if (f->pool) + mm_subpage_pool_close(f->pool); + if (f->mm) { + mmap_write_lock(f->mm); + for (i = 0; i < 4; i++) { + unsigned long addr = TEST_VA + i * SZ_4K; + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(f->mm, addr, &ptl); + + if (!pte) + continue; + ptep_get_and_clear_full(f->mm, addr, pte, 0); + if (f->mapped[i]) { + struct vm_area_struct *vma = vma_lookup(f->mm, addr); + + mm_subpage_remove_anon_rmap(f->slots[i], vma); + dec_mm_counter(f->mm, MM_ANONPAGES); + f->mapped[i] = false; + } + pte_unmap_unlock(pte, ptl); + } + tlb_gather_mmu_fullmm(&tlb, f->mm); + free_pgd_range(&tlb, 0, 1UL << 48, 0, 0); + tlb_finish_mmu(&tlb); + mmap_write_unlock(f->mm); + } + /* No slot reference is released before the PTEs and TLBs are retired. */ + for (i = 0; i < 4; i++) + if (f->slots[i]) + mm_subpage_put(f->slots[i]); + if (f->pool) + mm_subpage_pool_put(f->pool); + if (f->mm) + mmput(f->mm); + memset(f, 0, sizeof(*f)); +} + +static struct anon_slot_fixture *anon_slot_fixture_create_from(struct kunit *test, + struct anon_slot_fixture *parent) +{ + struct anon_slot_fixture *f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + struct vm_area_struct *vma; + struct folio *folio; + int ret; + + if (!f || kunit_add_action_or_reset(test, anon_slot_fixture_free, f)) + return NULL; + f->mm = mm_alloc(); + if (!f->mm || test_mm_select4k(f->mm)) + return NULL; + f->pool = mm_subpage_pool_create(GFP_KERNEL); + if (!f->pool) + return NULL; + folio = folio_alloc(GFP_KERNEL, 0); + if (!folio) + return NULL; + ret = mm_subpage_pool_add_folio(f->pool, folio, GFP_KERNEL); + if (ret) { + folio_put(folio); + return NULL; + } + vma = vm_area_alloc(f->mm); + if (!vma) + return NULL; + vma_set_anonymous(vma); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + 2 * PAGE_SIZE; + vm_flags_init(vma, VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE); + vma_set_page_prot(vma); + if (parent) { + mmap_write_lock(parent->mm); + mmap_write_lock_nested(f->mm, SINGLE_DEPTH_NESTING); + vma_set_page_offset(vma, vma_page_offset_at(parent->vma, TEST_VA)); + ret = anon_vma_fork(vma, parent->vma); + } else { + mmap_write_lock(f->mm); + ret = 0; + } + if (!ret && parent) { + VMA_ITERATOR(vmi, f->mm, TEST_VA); + + vma_start_write(vma); + ret = vma_iter_store_gfp(&vmi, vma, GFP_KERNEL); + if (!ret) + f->mm->map_count++; + } else if (!ret) { + ret = insert_vm_struct(f->mm, vma); + } + if (!ret) { + f->vma = vma; + ret = anon_vma_prepare(vma); + } + if (!f->vma) + unlink_anon_vmas(vma); + mmap_write_unlock(f->mm); + if (parent) + mmap_write_unlock(parent->mm); + if (!f->vma) + vm_area_free(vma); + return ret ? NULL : f; +} + +static struct anon_slot_fixture *anon_slot_fixture_create(struct kunit *test) +{ + return anon_slot_fixture_create_from(test, NULL); +} + +static void user4k_anon_rmap_test(struct kunit *test) +{ + struct anon_slot_fixture *f = anon_slot_fixture_create(test); + struct folio *folio = NULL; + unsigned int i; + vm_flags_t flags; + int ret, bad_offset = 0, bad_index = 0; + + KUNIT_ASSERT_NOT_NULL(test, f); + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc_at(f->pool, i * SZ_4K); + unsigned long addr = TEST_VA + i * SZ_4K; + struct test_mm tables = { .mm = f->mm }; + spinlock_t *ptl; + pte_t *pte; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + f->slots[i] = slot; + folio = mm_subpage_folio(slot); + /* Allocate the table first; clear the test leaf before rmap attachment. */ + KUNIT_ASSERT_EQ(test, test_map_alias(&tables, 0, addr, mm_subpage_phys(slot)), 0); + mmap_read_lock(f->mm); + folio_lock(folio); + pte = test_mm_lookup(f->mm, addr, &ptl); + if (!pte) { + folio_unlock(folio); + mmap_read_unlock(f->mm); + KUNIT_FAIL(test, "missing allocated PTE"); + return; + } + pte_clear(f->mm, addr, pte); + if (!i) + bad_offset = mm_subpage_add_anon_rmap(slot, f->vma, addr + 1); + ret = mm_subpage_add_anon_rmap(slot, f->vma, addr); + if (!ret) { + f->mapped[i] = true; + inc_mm_counter(f->mm, MM_ANONPAGES); + set_pte_at(f->mm, addr, pte, phys_pte_mm(f->mm, mm_subpage_phys(slot), + __pgprot(TEST_PROT | PTE_RDONLY))); + if (!i) + bad_index = mm_subpage_add_anon_rmap(slot, f->vma, addr + PAGE_SIZE); + } + pte_unmap_unlock(pte, ptl); + folio_unlock(folio); + mmap_read_unlock(f->mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 1); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)i + 1); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)i + 1); + } + KUNIT_EXPECT_EQ(test, bad_offset, -EINVAL); + KUNIT_EXPECT_EQ(test, bad_index, -EINVAL); + KUNIT_EXPECT_EQ(test, folio->index, TEST_VA >> PAGE_SHIFT); + KUNIT_EXPECT_FALSE(test, PageAnonExclusive(&folio->page)); + KUNIT_EXPECT_EQ(test, folio_expected_ref_count(folio), folio_ref_count(folio)); + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 1); + for (i = 0; i < 4; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(f->mm, TEST_VA + i * SZ_4K, &ptl); + bool young; + + KUNIT_ASSERT_NOT_NULL(test, pte); + young = pte_young(ptep_get(pte)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_FALSE(test, young); + } + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 0); +} + +/* The fixture takes one existing slot reference, including on failure. */ +static int anon_slot_fixture_map(struct anon_slot_fixture *f, + unsigned int index, struct mm_subpage *slot) +{ + unsigned long addr = TEST_VA + index * SZ_4K; + struct test_mm tables = { .mm = f->mm }; + struct folio *folio = mm_subpage_folio(slot); + spinlock_t *ptl; + pte_t *pte; + int ret; + + f->slots[index] = slot; + ret = test_map_alias(&tables, 0, addr, mm_subpage_phys(slot)); + if (ret) + return ret; + mmap_read_lock(f->mm); + folio_lock(folio); + pte = test_mm_lookup(f->mm, addr, &ptl); + if (!pte) { + ret = -ENOMEM; + goto unlock; + } + pte_clear(f->mm, addr, pte); + ret = mm_subpage_add_anon_rmap(slot, f->vma, addr); + if (!ret) { + f->mapped[index] = true; + inc_mm_counter(f->mm, MM_ANONPAGES); + set_pte_at(f->mm, addr, pte, phys_pte_mm(f->mm, mm_subpage_phys(slot), + __pgprot(TEST_PROT | PTE_RDONLY))); + } + pte_unmap_unlock(pte, ptl); +unlock: + folio_unlock(folio); + mmap_read_unlock(f->mm); + return ret; +} + +static void user4k_anon_fork_rmap_test(struct kunit *test) +{ + struct anon_slot_fixture *parent = anon_slot_fixture_create(test); + struct anon_slot_fixture *child; + struct mm_subpage *slot; + struct folio *folio; + vm_flags_t flags; + int ret; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, parent); + child = anon_slot_fixture_create_from(test, parent); + KUNIT_ASSERT_NOT_NULL(test, child); + for (i = 0; i < 4; i++) { + slot = mm_subpage_alloc_at(parent->pool, i * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(parent, i, slot), 0); + mm_subpage_get(slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(child, i, slot), 0); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 2); + } + folio = mm_subpage_folio(slot); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 8); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 8); + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 2); + anon_slot_fixture_free(parent); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 4); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 4); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(child->slots[i]), 1); + /* The child mappings and root remain valid after the parent disappears. */ + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 0); +} + +static void user4k_nonlinear_rmap_reuse_test(struct kunit *test) +{ + struct anon_slot_fixture *a = anon_slot_fixture_create(test); + struct anon_slot_fixture *b = anon_slot_fixture_create(test); + struct anon_slot_fixture *c = anon_slot_fixture_create(test); + struct mm_subpage *slot; + struct folio *folio; + struct anon_vma_chain *avc; + phys_addr_t phys; + vm_flags_t flags; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + KUNIT_ASSERT_NOT_NULL(test, c); + slot = mm_subpage_alloc_at(a->pool, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(a, 0, slot), 0); + folio = mm_subpage_folio(slot); + slot = mm_subpage_alloc_at(a->pool, 3 * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + phys = mm_subpage_phys(slot); +#ifdef CONFIG_FAILSLAB + { + unsigned int pending; + + mmap_read_lock(b->mm); + folio_lock(folio); + WRITE_ONCE(current->fail_nth, 1); + ret = mm_subpage_add_anon_rmap(slot, b->vma, TEST_VA + SZ_4K); + pending = READ_ONCE(current->fail_nth); + WRITE_ONCE(current->fail_nth, 0); + /* Keep cleanup valid even if allocation unexpectedly succeeds. */ + if (!ret) + mm_subpage_remove_anon_rmap(slot, b->vma); + folio_unlock(folio); + mmap_read_unlock(b->mm); + KUNIT_EXPECT_EQ(test, pending, 0U); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 0); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 1); + } +#endif + /* Allocation failure left the slot available for the same operation. */ + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(b, 1, slot), 0); + anon_slot_fixture_free(b); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 1); + slot = mm_subpage_alloc_at(a->pool, 3 * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_phys(slot), phys); + + /* Give this empty VMA a different logical native index as well. */ + mmap_write_lock(c->mm); + anon_vma_lock_write(c->vma->anon_vma); + list_for_each_entry(avc, &c->vma->anon_vma_chain, same_vma) + anon_vma_interval_tree_remove(avc, &avc->anon_vma->rb_root); + c->vma->vm_pgoff += 9; + list_for_each_entry(avc, &c->vma->anon_vma_chain, same_vma) + anon_vma_interval_tree_insert(avc, &avc->anon_vma->rb_root); + anon_vma_unlock_write(c->vma->anon_vma); + mmap_write_unlock(c->mm); +#ifdef CONFIG_FAILSLAB + /* Retiring a previous root also needs failure-atomic bookkeeping. */ + mmap_read_lock(c->mm); + folio_lock(folio); + WRITE_ONCE(current->fail_nth, 1); + ret = mm_subpage_add_anon_rmap(slot, c->vma, TEST_VA + 2 * SZ_4K); + WRITE_ONCE(current->fail_nth, 0); + if (!ret) + mm_subpage_remove_anon_rmap(slot, c->vma); + folio_unlock(folio); + mmap_read_unlock(c->mm); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 0); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 1); +#endif + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(c, 2, slot), 0); + mmap_read_lock(c->mm); + KUNIT_EXPECT_EQ(test, page_address_in_vma(folio, &folio->page, c->vma), + TEST_VA + 2 * SZ_4K); + mmap_read_unlock(c->mm); + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 2); + /* The original root may disappear before the other slot's last mapping. */ + anon_slot_fixture_free(a); + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(c->slots[2]), 1); +} + +static void user4k_dup_mmap_test(struct kunit *test) +{ + struct anon_slot_fixture *parent = anon_slot_fixture_create(test); + struct mm_struct *child; + struct folio *folio; + unsigned int i; + int ret = 0; + + KUNIT_ASSERT_NOT_NULL(test, parent); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc_at(parent->pool, i * SZ_4K); + unsigned long addr = TEST_VA + i * SZ_4K; + spinlock_t *ptl; + pte_t *pte; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(parent, i, slot), 0); + pte = test_mm_lookup(parent->mm, addr, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + set_pte_at(parent->mm, addr, pte, + pte_mkwrite_novma(pte_mkdirty(ptep_get(pte)))); + pte_unmap_unlock(pte, ptl); + } + /* Split into four 4K VMAs plus a trailing hole, preserving COW flags. */ + mmap_write_lock(parent->mm); + for (i = 1; i < 4; i += 2) { + unsigned long addr = TEST_VA + i * SZ_4K; + VMA_ITERATOR(vmi, parent->mm, addr); + struct vm_area_struct *vma = vma_iter_load(&vmi); + vma_flags_t flags = vma->flags; + + vma_flags_set(&flags, VMA_DONTDUMP_BIT); + vma = vma_modify_flags(&vmi, vma, vma, addr, addr + SZ_4K, &flags); + if (IS_ERR(vma)) { + ret = PTR_ERR(vma); + break; + } + } + mmap_write_unlock(parent->mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, parent->mm->map_count, 5); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, child->map_count, 5); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 4UL); + folio = mm_subpage_folio(parent->slots[0]); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 8); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 8); + for (i = 0; i < 4; i++) { + unsigned long addr = TEST_VA + i * SZ_4K; + struct vm_area_struct *vma = test_vma_lookup(child, addr); + unsigned int side; + + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(vma), i * (unsigned int)SZ_4K); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(parent->slots[i]), 2); + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : parent->mm; + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, addr, &ptl); + pte_t entry; + + KUNIT_ASSERT_NOT_NULL(test, pte); + entry = ptep_get(pte); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_FALSE(test, pte_write(entry)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, entry), + mm_subpage_phys(parent->slots[i])); + } + } + anon_slot_fixture_free(parent); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 4); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 4); + /* KUnit's child-mm cleanup now exercises ordinary exit_mmap(). */ +} + +static void test_subpage_put(void *arg) +{ + mm_subpage_put(arg); +} + +static void user4k_anon_root_lifetime_test(struct kunit *test) +{ + struct anon_slot_fixture *old = anon_slot_fixture_create(test); + struct anon_slot_fixture *other = anon_slot_fixture_create(test); + struct mm_subpage *slot; + struct anon_vma *root; + struct folio *folio; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, old); + KUNIT_ASSERT_NOT_NULL(test, other); + slot = mm_subpage_alloc_at(old->pool, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(old, 0, slot), 0); + folio = mm_subpage_folio(slot); + root = old->vma->anon_vma->root; + mm_subpage_get(slot); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, slot), 0); + anon_slot_fixture_free(old); + /* Only the owner now pins the root; no VMA or PTE survives. */ + KUNIT_EXPECT_EQ(test, atomic_read(&root->refcount), 1); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 0); + mm_subpage_get(slot); + ret = anon_slot_fixture_map(other, 0, slot); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 0); + KUNIT_EXPECT_EQ(test, atomic_read(&root->refcount), 1); +} + +static void test_folio_put(void *arg) +{ + folio_put(arg); +} + +static void user4k_data_tlb_batch_test(struct kunit *test) +{ + struct anon_slot_fixture *f = anon_slot_fixture_create(test); + struct folio *native = folio_alloc(GFP_KERNEL, 2); + struct mmu_gather tlb; + struct folio *backing; + unsigned int i; + bool full = false; + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, native), 0); + for (i = 0; i < 4; i++) { + f->slots[i] = mm_subpage_alloc_at(f->pool, i * SZ_4K); + if (IS_ERR(f->slots[i])) { + f->slots[i] = NULL; + KUNIT_FAIL(test, "slot allocation failed"); + return; + } + } + backing = mm_subpage_folio(f->slots[0]); + memset(folio_address(backing), 0x5a, SZ_4K); + tlb_gather_mmu(&tlb, f->mm); + tlb_change_page_size(&tlb, MM_SUBPAGE_SIZE); + tlb_flush_pte_range(&tlb, TEST_VA, PAGE_SIZE); + for (i = 0; i < 64; i++) { + mm_subpage_get(f->slots[i % 4]); + if (__tlb_remove_subpage(&tlb, f->slots[i % 4])) { + full = true; + tlb_flush_mmu(&tlb); + tlb_flush_pte_range(&tlb, TEST_VA, PAGE_SIZE); + } + if (i == 12) { + /* Exercise the native count payload (3 << 2 has tag bit 2). */ + folio_ref_add(native, 3); + if (__tlb_remove_folio_pages(&tlb, &native->page, 3, false)) { + full = true; + tlb_flush_mmu(&tlb); + tlb_flush_pte_range(&tlb, TEST_VA, PAGE_SIZE); + } + } + } + mm_subpage_put(f->slots[0]); + f->slots[0] = NULL; + /* Without memory pressure these fit in the local plus one dynamic batch. */ + if (!full) { + struct mm_subpage *extra = mm_subpage_alloc_at(f->pool, 0); + + KUNIT_EXPECT_EQ(test, PTR_ERR(extra), -EAGAIN); + if (!IS_ERR(extra)) + mm_subpage_put(extra); + KUNIT_EXPECT_EQ(test, folio_ref_count(backing), 67); + KUNIT_EXPECT_EQ(test, folio_ref_count(native), 4); + KUNIT_EXPECT_GT(test, tlb.batch_count, 0U); + } + tlb_finish_mmu(&tlb); + KUNIT_EXPECT_EQ(test, folio_ref_count(backing), 3); + KUNIT_EXPECT_EQ(test, folio_ref_count(native), 1); + f->slots[0] = mm_subpage_alloc_at(f->pool, 0); + if (IS_ERR(f->slots[0])) { + f->slots[0] = NULL; + KUNIT_FAIL(test, "slot was not reusable after TLB completion"); + return; + } + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(f->slots[0]), backing); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(backing), 0, SZ_4K), NULL); + for (i = 0; i < 4; i++) { + struct mm_subpage *extra = mm_subpage_alloc_at(f->pool, i * SZ_4K); + + KUNIT_EXPECT_EQ(test, PTR_ERR(extra), -EAGAIN); + if (!IS_ERR(extra)) + mm_subpage_put(extra); + } +} + +static void user4k_anon_zap_test(struct kunit *test) +{ + struct anon_slot_fixture *f = anon_slot_fixture_create(test); + struct mm_subpage *slots[4]; + struct folio *folio; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, f); + for (i = 0; i < 4; i++) { + slots[i] = mm_subpage_alloc_at(f->pool, i * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slots[i]); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(f, i, slots[i]), 0); + mm_subpage_get(slots[i]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, slots[i]), 0); + } + folio = mm_subpage_folio(slots[0]); + mmap_read_lock(f->mm); + zap_vma_range(f->vma, TEST_VA + SZ_4K, SZ_4K); + f->mapped[1] = false; + f->slots[1] = NULL; + mmap_read_unlock(f->mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 3); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 7); + for (i = 0; i < 4; i++) { + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(f->mm, TEST_VA + i * SZ_4K, &ptl); + bool present; + + KUNIT_ASSERT_NOT_NULL(test, pte); + present = pte_present(ptep_get(pte)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, present, i != 1); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slots[i]), i == 1 ? 0 : 1); + } + /* Include both the existing hole and a trailing run of empty entries. */ + mmap_read_lock(f->mm); + zap_vma(f->vma); + for (i = 0; i < 4; i++) { + f->mapped[i] = false; + f->slots[i] = NULL; + } + mmap_read_unlock(f->mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 4); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slots[i]), 0); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(f->mm, MM_ANONPAGES), 0UL); +} + +static void user4k_anon_writable_slots_test(struct kunit *test) +{ + struct anon_slot_fixture *f = anon_slot_fixture_create(test); + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, f); + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc_at(f->pool, i * SZ_4K); + unsigned long addr = TEST_VA + i * SZ_4K; + spinlock_t *ptl; + pte_t *pte; + bool writable; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(f, i, slot), 0); + pte = test_mm_lookup(f->mm, addr, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + set_pte_at(f->mm, addr, pte, + pte_mkwrite_novma(pte_mkdirty(ptep_get(pte)))); + writable = pte_write(ptep_get(pte)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_TRUE(test, writable); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 1); + } + KUNIT_EXPECT_EQ(test, folio_mapcount(mm_subpage_folio(f->slots[0])), 4); +} + +static struct vm_area_struct *test_fault_vma_in_mm(struct mm_struct *mm, + unsigned long start, unsigned long len) +{ + struct vm_area_struct *vma; + vm_flags_t flags = VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE; + int ret; + + if (!mm) + return NULL; + vma = vm_area_alloc(mm); + if (!vma) + return NULL; + vma_set_anonymous(vma); + vma->vm_start = start; + vma->vm_end = start + len; + vm_flags_init(vma, flags); + vma->vm_page_prot = vm_get_page_prot(flags); + mmap_write_lock(mm); + ret = insert_vm_struct(mm, vma); + mmap_write_unlock(mm); + if (ret) { + vm_area_free(vma); + return NULL; + } + return vma; +} + +struct coarse_anon_fixture { + struct mm_struct *mm[2]; + bool attached[2]; + struct vm_area_struct *vma[2]; + struct folio *folio; + unsigned long mapped[2]; + unsigned int leaves; + bool pinned; +}; + +static void coarse_anon_fixture_free(void *ptr) +{ + struct coarse_anon_fixture *f = ptr; + unsigned int i, nr = SZ_64K / PAGE_SIZE; + + if (f->pinned) { + unpin_user_page(folio_page(f->folio, nr - 1)); + f->pinned = false; + } + for (i = 0; i < 2; i++) { + unsigned int leaf; + + if (!f->mm[i]) + continue; + mmap_write_lock(f->mm[i]); + for (leaf = 0; leaf < (f->leaves ?: 1); leaf++) { + unsigned long addr = TEST_VA + leaf * SZ_64K; + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(f->mm[i], addr, &ptl); + + if (pte) { + ptep_get_and_clear_full(f->mm[i], addr, pte, 0); + if (f->mapped[i] & BIT(leaf)) { + folio_remove_rmap_pte(f->folio, + folio_page(f->folio, leaf * nr), f->vma[i]); + dec_mm_counter(f->mm[i], MM_ANONPAGES); + } + pte_unmap_unlock(pte, ptl); + } + if (f->mapped[i] & BIT(leaf)) { + folio_put_refs(f->folio, nr); + f->mapped[i] &= ~BIT(leaf); + } + } + mmap_write_unlock(f->mm[i]); + if (f->vma[i] && !f->attached[i]) { + unlink_anon_vmas(f->vma[i]); + vm_area_free(f->vma[i]); + } + mmput(f->mm[i]); + f->vma[i] = NULL; + f->mm[i] = NULL; + } + if (f->folio) { + folio_put(f->folio); + f->folio = NULL; + } +} + +static void expect_coarse_anon_state(struct kunit *test, struct folio *folio, + unsigned int maps, bool exclusive) +{ + unsigned int i, nr = folio_nr_pages(folio); + + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)(maps * nr)); + for (i = 0; i < nr; i++) { + struct page *page = folio_page(folio, i); + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, (atomic_read(&page->_mapcount) + 1), (int)maps); + KUNIT_EXPECT_EQ(test, PageAnonExclusive(page), exclusive); + } +} + +static void user64k_anon_rmap_span_test(struct kunit *test) +{ + struct coarse_anon_fixture *f; + unsigned int i, nr = SZ_64K / PAGE_SIZE; + pte_t entry, *pte; + spinlock_t *ptl; + int ret, dup_ret, share_ret; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires coarse anonymous leaf"); + return; + } + f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, coarse_anon_fixture_free, f), 0); + f->folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 16 - PAGE_SHIFT); + KUNIT_ASSERT_NOT_NULL(test, f->folio); + for (i = 0; i < 2; i++) { + struct test_mm tables = {}; + + f->mm[i] = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, f->mm[i]); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(f->mm[i], 16), 0); + if (!i) { + f->vma[i] = test_fault_vma_in_mm(f->mm[i], TEST_VA, SZ_64K); + f->attached[i] = !!f->vma[i]; + } else { + f->vma[i] = vm_area_alloc(f->mm[i]); + if (f->vma[i]) { + vma_set_anonymous(f->vma[i]); + f->vma[i]->vm_start = TEST_VA; + f->vma[i]->vm_end = TEST_VA + SZ_64K; + vm_flags_init(f->vma[i], f->vma[0]->vm_flags); + f->vma[i]->vm_page_prot = f->vma[0]->vm_page_prot; + vma_set_page_offset(f->vma[i], vma_page_offset_at(f->vma[0], TEST_VA)); + } + } + KUNIT_ASSERT_NOT_NULL(test, f->vma[i]); + tables.mm = f->mm[i]; + KUNIT_ASSERT_EQ(test, test_map_alias(&tables, 0, TEST_VA, PFN_PHYS(folio_pfn(f->folio))), 0); + pte = test_mm_lookup(f->mm[i], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + pte_clear(f->mm[i], TEST_VA, pte); + pte_unmap_unlock(pte, ptl); + } + mmap_write_lock(f->mm[0]); + ret = anon_vma_prepare(f->vma[0]); + mmap_write_unlock(f->mm[0]); + KUNIT_ASSERT_EQ(test, ret, 0); + mmap_write_lock(f->mm[0]); + mmap_write_lock_nested(f->mm[1], SINGLE_DEPTH_NESTING); + ret = anon_vma_fork(f->vma[1], f->vma[0]); + if (!ret) { + VMA_ITERATOR(vmi, f->mm[1], TEST_VA); + + vma_start_write(f->vma[1]); + ret = vma_iter_store_gfp(&vmi, f->vma[1], GFP_KERNEL); + if (!ret) { + f->mm[1]->map_count++; + f->attached[1] = true; + } + } + mmap_write_unlock(f->mm[1]); + mmap_write_unlock(f->mm[0]); + KUNIT_ASSERT_EQ(test, ret, 0); + + pte = test_mm_lookup(f->mm[0], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + folio_add_new_anon_rmap(f->folio, f->vma[0], TEST_VA, RMAP_EXCLUSIVE); + folio_ref_add(f->folio, nr); /* Fixture retains its original allocation ref. */ + inc_mm_counter(f->mm[0], MM_ANONPAGES); + f->mapped[0] = true; + entry = pte_mkwrite(pte_mkdirty(phys_pte_mm(f->mm[0], PFN_PHYS(folio_pfn(f->folio)), + f->vma[0]->vm_page_prot)), f->vma[0]); + set_ptes(f->mm[0], TEST_VA, pte, entry, 1); + pte_unmap_unlock(pte, ptl); + expect_coarse_anon_state(test, f->folio, 1, true); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), (int)nr + 1); + + mm_flags_set(MMF_HAS_PINNED, f->mm[0]); + ret = try_grab_folio(page_folio(folio_page(f->folio, nr - 1)), 1, FOLL_PIN); + f->pinned = !ret; + KUNIT_ASSERT_EQ(test, ret, 0); + pte = test_mm_lookup(f->mm[0], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + ptep_set_wrprotect(f->mm[0], TEST_VA, pte); + raw_write_seqcount_begin(&f->mm[0]->write_protect_seq); + dup_ret = folio_try_dup_anon_rmap_pte(f->folio, &f->folio->page, f->vma[1], f->vma[0]); + raw_write_seqcount_end(&f->mm[0]->write_protect_seq); + entry = ptep_get_and_clear(f->mm[0], TEST_VA, pte); + share_ret = folio_try_share_anon_rmap_pte(f->folio, &f->folio->page, f->vma[0]); + set_ptes(f->mm[0], TEST_VA, pte, entry, 1); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, dup_ret, -EBUSY); + KUNIT_EXPECT_EQ(test, share_ret, -EBUSY); + expect_coarse_anon_state(test, f->folio, 1, true); + unpin_user_page(folio_page(f->folio, nr - 1)); + f->pinned = false; + + pte = test_mm_lookup(f->mm[0], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + entry = ptep_get_and_clear(f->mm[0], TEST_VA, pte); + share_ret = folio_try_share_anon_rmap_pte(f->folio, &f->folio->page, f->vma[0]); + set_ptes(f->mm[0], TEST_VA, pte, pte_wrprotect(entry), 1); + pte_unmap_unlock(pte, ptl); + KUNIT_ASSERT_EQ(test, share_ret, 0); + expect_coarse_anon_state(test, f->folio, 1, false); + + pte = test_mm_lookup(f->mm[0], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + raw_write_seqcount_begin(&f->mm[0]->write_protect_seq); + dup_ret = folio_try_dup_anon_rmap_pte(f->folio, &f->folio->page, f->vma[1], f->vma[0]); + raw_write_seqcount_end(&f->mm[0]->write_protect_seq); + if (!dup_ret) { + folio_ref_add(f->folio, nr); + inc_mm_counter(f->mm[1], MM_ANONPAGES); + f->mapped[1] = true; + } + pte_unmap_unlock(pte, ptl); + KUNIT_ASSERT_EQ(test, dup_ret, 0); + pte = test_mm_lookup(f->mm[1], TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + set_ptes(f->mm[1], TEST_VA, pte, pte_wrprotect(entry), 1); + pte_unmap_unlock(pte, ptl); + expect_coarse_anon_state(test, f->folio, 2, false); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), (int)(2 * nr + 1)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(f->mm[0], MM_ANONPAGES), 1L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(f->mm[1], MM_ANONPAGES), 1L); + + /* Exercise ordinary unmapping of each shared coarse leaf. */ + for (i = 0; i < 2; i++) { + mmap_read_lock(f->mm[i]); + zap_vma_range(f->vma[i], TEST_VA, SZ_64K); + f->mapped[i] = 0; + mmap_read_unlock(f->mm[i]); + expect_coarse_anon_state(test, f->folio, 1 - i, false); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), (int)((1 - i) * nr + 1)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(f->mm[i], MM_ANONPAGES), 0L); + } + + /* Keep an observation ref while cleanup releases the empty address spaces. */ + { + struct folio *folio = f->folio; + + folio_get(folio); + coarse_anon_fixture_free(f); + expect_coarse_anon_state(test, folio, 0, false); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + folio_put(folio); + } +} + +static void user64k_anon_zap_batch_test(struct kunit *test) +{ + struct coarse_anon_fixture *f; + struct test_mm tables = {}; + struct vm_area_struct *vma; + struct mm_struct *mm; + unsigned int i, nr = SZ_64K / PAGE_SIZE, batch; + pte_t entry, *pte; + spinlock_t *ptl; + int ret; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires coarse anonymous leaves"); + return; + } + f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, coarse_anon_fixture_free, f), 0); + f->leaves = 2; + f->folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 17 - PAGE_SHIFT); + KUNIT_ASSERT_NOT_NULL(test, f->folio); + mm = f->mm[0] = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, mm); + /* Partial unmap can enqueue this folio on the memcg split queue. */ + KUNIT_ASSERT_EQ(test, mem_cgroup_charge(f->folio, mm, GFP_KERNEL), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, 16), 0); + vma = f->vma[0] = test_fault_vma_in_mm(mm, TEST_VA, 2 * SZ_64K); + f->attached[0] = !!vma; + KUNIT_ASSERT_NOT_NULL(test, vma); + tables.mm = mm; + for (i = 0; i < 2; i++) { + unsigned long addr = TEST_VA + i * SZ_64K; + + KUNIT_ASSERT_EQ(test, test_map_alias(&tables, i, addr, + PFN_PHYS(folio_pfn(f->folio)) + i * SZ_64K), 0); + pte = test_mm_lookup(mm, addr, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + pte_clear(mm, addr, pte); + pte_unmap_unlock(pte, ptl); + } + mmap_write_lock(mm); + ret = anon_vma_prepare(vma); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + pte = test_mm_lookup(mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + folio_add_new_anon_rmap(f->folio, vma, TEST_VA, RMAP_EXCLUSIVE); + folio_ref_add(f->folio, 2 * nr); + add_mm_counter(mm, MM_ANONPAGES, 2); + f->mapped[0] = BIT(0) | BIT(1); + entry = pte_mkwrite(pte_mkdirty(phys_pte_mm(mm, + PFN_PHYS(folio_pfn(f->folio)), vma->vm_page_prot)), vma); + set_ptes(mm, TEST_VA, pte, entry, 2); + batch = folio_subpage_pte_batch(vma, f->folio, pte, entry, 2, 0); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, batch, 2U); + KUNIT_EXPECT_EQ(test, folio_mapcount(f->folio), (int)(2 * nr)); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), (int)(2 * nr + 1)); + + /* The reference walker must consume logical PTEs and native mapcounts. */ + { + vm_flags_t flags; + bool young; + + folio_lock(f->folio); + ret = folio_referenced(f->folio, 1, NULL, &flags); + folio_unlock(f->folio); + KUNIT_EXPECT_EQ(test, ret, 1); + pte = test_mm_lookup(mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + young = pte_young(ptep_get(pte)) || pte_young(ptep_get(pte + 1)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_FALSE(test, young); + folio_lock(f->folio); + ret = folio_referenced(f->folio, 1, NULL, &flags); + folio_unlock(f->folio); + KUNIT_EXPECT_EQ(test, ret, 0); + } + + /* Removing one logical leaf must leave the adjacent leaf's native span. */ + mmap_read_lock(mm); + zap_vma_range(vma, TEST_VA, SZ_64K); + f->mapped[0] &= ~BIT(0); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(f->folio), (int)nr); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), (int)nr + 1); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 1L); + for (i = 0; i < 2 * nr; i++) + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, atomic_read(&folio_page(f->folio, i)->_mapcount), + i < nr ? -1 : 0); + pte = test_mm_lookup(mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + ret = pte_none(ptep_get(pte)) && pte_present(ptep_get(pte + 1)); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_TRUE(test, ret); + + /* Restore the first leaf, then remove both through one batched walk. */ + pte = test_mm_lookup(mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + folio_add_anon_rmap_pte(f->folio, &f->folio->page, vma, + TEST_VA, RMAP_EXCLUSIVE); + folio_ref_add(f->folio, nr); + inc_mm_counter(mm, MM_ANONPAGES); + f->mapped[0] |= BIT(0); + set_ptes(mm, TEST_VA, pte, entry, 1); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, folio_mapcount(f->folio), (int)(2 * nr)); + mmap_read_lock(mm); + zap_vma_range(vma, TEST_VA, 2 * SZ_64K); + f->mapped[0] = 0; + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(f->folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(f->folio), 1); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0L); + for (i = 0; i < 2 * nr; i++) + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, atomic_read(&folio_page(f->folio, i)->_mapcount), -1); +} + +static void user64k_file_rmap_span_test(struct kunit *test) +{ + struct mm_struct *mm; + struct vm_area_struct *vma; + struct folio *folio; + unsigned int i, nr = SZ_64K / PAGE_SIZE; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires coarse nonanonymous leaf"); + return; + } + mm = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, mm); + vma = vm_area_alloc(mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + 2 * SZ_64K; + folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 16 - PAGE_SHIFT); + KUNIT_ASSERT_NOT_NULL(test, folio); + /* Private metadata-only fixture: no file-cache promotion or fault is implied. */ + folio_lock(folio); + folio_add_file_rmap_pte(folio, &folio->page, vma); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)nr); + for (i = 0; i < nr; i++) + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, atomic_read(&folio_page(folio, i)->_mapcount) + 1, 1); + folio_dup_file_rmap_pte(folio, &folio->page, vma); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)(2 * nr)); + for (i = 0; i < nr; i++) + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, atomic_read(&folio_page(folio, i)->_mapcount) + 1, 2); + folio_remove_rmap_pte(folio, &folio->page, vma); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)nr); + folio_remove_rmap_pte(folio, &folio->page, vma); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + for (i = 0; i < nr; i++) + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + KUNIT_EXPECT_EQ(test, atomic_read(&folio_page(folio, i)->_mapcount), -1); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + folio_put(folio); +} + +static struct vm_area_struct *test_fault_vma(struct kunit *test, + unsigned long start, unsigned long len) +{ + return test_fault_vma_in_mm(test_vma_mm(test), start, len); +} + +static struct vm_area_struct *test_fault_vma_granule(struct kunit *test, + unsigned int shift, unsigned long start, unsigned long len) +{ + return test_fault_vma_in_mm(test_vma_mm_granule(test, shift), start, len); +} + +static void test_anon_write_fault_granule(struct kunit *test, unsigned int shift) +{ + unsigned long size = 1UL << shift; + unsigned int shape; + + /* Swap/reclaim integration is a later gate, before any ABI activation. */ + KUNIT_ASSERT_EQ(test, total_swap_pages, 0UL); + for (shape = 0; shape < 2; shape++) { + unsigned long start = TEST_VA + (shape ? size : 0); + unsigned long len = shape ? size : 2 * PAGE_SIZE; + struct vm_area_struct *vma = test_fault_vma_granule(test, shift, start, len); + struct folio *backing = NULL; + unsigned int i, mapped = 0; + vm_fault_t ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(vma->vm_mm); + ret = handle_mm_fault(vma, TEST_VA + size + 3, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, ret, (vm_fault_t)0); + /* Remove transient LRU-add references before checking ownership. */ + lru_add_drain_all(); + for (i = 0; i < (PAGE_SIZE >> shift); i++) { + unsigned long addr = TEST_VA + i * size; + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(vma->vm_mm, addr, &ptl); + struct mm_subpage *slot = NULL; + bool expected = !shape || i == 1; + bool present, writable = false, zero = false, identity = false; + + KUNIT_ASSERT_NOT_NULL(test, pte); + present = pte_present(ptep_get(pte)); + if (present) { + slot = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, ptep_get(pte))); + writable = pte_write(ptep_get(pte)); + if (slot) { + struct folio *folio = mm_subpage_folio(slot); + + if (!backing) + backing = folio; + identity = folio == backing && + mm_subpage_phys(slot) == + pte_phys_mm(vma->vm_mm, ptep_get(pte)) && + (shape || mm_subpage_offset(slot) == i * size); + zero = !memchr_inv(folio_address(folio) + mm_subpage_offset(slot), + 0, size); + mm_subpage_put(slot); + } + mapped++; + } + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_EQ(test, present, expected); + if (expected) { + KUNIT_EXPECT_TRUE(test, writable); + KUNIT_EXPECT_TRUE(test, identity); + KUNIT_EXPECT_TRUE(test, zero); + } + } + KUNIT_ASSERT_NOT_NULL(test, backing); + KUNIT_EXPECT_EQ(test, mapped, shape ? 1U : (PAGE_SIZE >> shift)); + KUNIT_EXPECT_EQ(test, folio_mapcount(backing), (int)mapped); + KUNIT_EXPECT_EQ(test, folio_ref_count(backing), (int)mapped); + KUNIT_EXPECT_TRUE(test, folio_test_lru(backing)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), + (unsigned long)mapped); + } +} + +static void user4k_anon_write_fault_test(struct kunit *test) +{ + test_anon_write_fault_granule(test, 12); +} + +static void user16k_anon_write_fault_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative faults require larger native backing"); + return; + } + test_anon_write_fault_granule(test, 14); +} + +static void user4k_swap_encoding_test(struct kunit *test) +{ + const unsigned long offsets[] = { 1, 2, (1UL << __SWP_OFFSET_BITS) - 2 }; + const unsigned int types[] = { 0, 1, __SWP_TYPE_MASK }; + unsigned int i, j, slot, flags; + + for (i = 0; i < ARRAY_SIZE(offsets); i++) { + for (j = 0; j < ARRAY_SIZE(types); j++) { + for (slot = 0; slot < PAGE_SIZE / SZ_4K; slot++) { + for (flags = 0; flags < 4; flags++) { + softleaf_t entry = swp_entry(types[j], offsets[i]); + pte_t pte = softleaf_to_pte(entry), moved; + + pte = pte_swp_set_subpage_offset(pte, slot * SZ_4K); + if (flags & 1) + pte = pte_swp_mkexclusive(pte); + if (flags & 2) + pte = pte_swp_mkuffd_wp(pte); + KUNIT_EXPECT_FALSE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(pte).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(pte), slot * (unsigned int)SZ_4K); + moved = pte_move_swp_offset(pte, 1); + KUNIT_EXPECT_EQ(test, swp_offset(softleaf_from_pte(moved)), offsets[i] + 1); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(moved), slot * (unsigned int)SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_same(pte_move_swp_offset(moved, -1), pte)); + moved = pte_swp_set_subpage_offset(pte, ((slot + 1) % (PAGE_SIZE / SZ_4K)) * SZ_4K); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(moved).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(moved), ((slot + 1) % (PAGE_SIZE / SZ_4K)) * (unsigned int)SZ_4K); + } + } + } + } +} + +static pte_t test_fault_entry(struct kunit *test, struct mm_struct *mm, unsigned long addr) +{ + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, addr, &ptl); + pte_t entry = __pte(0); + + KUNIT_EXPECT_NOT_NULL(test, pte); + if (pte) { + entry = ptep_get(pte); + pte_unmap_unlock(pte, ptl); + } + return entry; +} + + +static void test_file_put(void *file) +{ + fput(file); +} + +static struct file *test_cache_file(struct kunit *test, unsigned long size, bool fill) +{ + struct file *file = shmem_file_setup("user4k-file-test", size, EMPTY_VMA_FLAGS); + unsigned char *buf; + loff_t pos = 0; + unsigned int i; + + if (IS_ERR(file)) + return file; + if (kunit_add_action_or_reset(test, test_file_put, file)) + return ERR_PTR(-ENOMEM); + if (!fill) + return file; + buf = kunit_kmalloc(test, size, GFP_KERNEL); + if (!buf) + return ERR_PTR(-ENOMEM); + for (i = 0; i < size; i++) + buf[i] = 0x31 + (i / SZ_4K); + if (kernel_write(file, buf, size, &pos) != size) + return ERR_PTR(-EIO); + return file; +} + +static struct vm_area_struct *test_file_vma_granule(struct kunit *test, struct file *file, + unsigned int shift, bool shared, unsigned long start, unsigned long len, + unsigned long offset) +{ + struct mm_struct *mm; + struct vm_area_struct *vma; + vm_flags_t flags = VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE; + int ret; + + mm = test_vma_mm_granule(test, shift); + if (!mm) + return NULL; + vma = vm_area_alloc(mm); + if (!vma) + return NULL; + vma->vm_start = start; + vma->vm_end = start + len; + vma_set_page_offset(vma, (struct vm_page_offset) { + .index = offset >> PAGE_SHIFT, .offset = offset & ~PAGE_MASK, + }); + if (shared) + flags |= VM_SHARED | VM_MAYSHARE; + vm_flags_init(vma, flags); + vma->vm_file = get_file(file); + vma->vm_page_prot = vm_get_page_prot(flags); + mmap_write_lock(mm); + ret = vfs_mmap(file, vma); + if (!ret) { + vma_set_page_prot(vma); + ret = insert_vm_struct(mm, vma); + } + mmap_write_unlock(mm); + if (ret) { + fput(vma->vm_file); + vm_area_free(vma); + return NULL; + } + return vma; +} + +static struct vm_area_struct *test_file_vma(struct kunit *test, struct file *file, + bool native, bool shared, unsigned long start, unsigned long len, + unsigned long offset) +{ + return test_file_vma_granule(test, file, native ? PAGE_SHIFT : 12, + shared, start, len, offset); +} + +static void user4k_mmap_descriptor_test(struct kunit *test) +{ + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vma; + struct vm_area_desc desc; + + KUNIT_ASSERT_NOT_NULL(test, mm); + vma = vm_area_alloc(mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_free_vma, vma), 0); + vma->vm_start = TEST_VA; + vma->vm_end = TEST_VA + PAGE_SIZE; + vma_set_page_offset(vma, (struct vm_page_offset) { 7, SZ_4K }); + compat_set_desc_from_vma(&desc, NULL, vma); + KUNIT_EXPECT_EQ(test, desc.pgoff, 7UL); + KUNIT_EXPECT_EQ(test, desc.subpage_offset, (unsigned int)SZ_4K); + desc.pgoff = 19; + desc.subpage_offset = 3 * SZ_4K; + KUNIT_ASSERT_EQ(test, __compat_vma_mmap(&desc, vma), 0); + KUNIT_EXPECT_TRUE(test, vm_page_offset_equal(vma_page_offset_at(vma, TEST_VA), + (struct vm_page_offset) { 19, 3 * SZ_4K })); +} + +static void user4k_mmap_region_offsets_test(struct kunit *test) +{ + struct file *file; + struct mm_struct *mm; + struct vm_area_struct *vma; + unsigned long start = TEST_VA + SZ_4K; + vm_flags_t flags = VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | + VM_SHARED | VM_MAYSHARE | VM_NORESERVE; + unsigned long result[4]; + unsigned int i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mapping fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, 8 * PAGE_SIZE, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + /* Only the full native-sized lengths are used before accounting conversion. */ + kthread_use_mm(mm); + mmap_write_lock(mm); + result[0] = mmap_region(file, start + PAGE_SIZE, PAGE_SIZE, flags, + (struct vm_page_offset) { 1, SZ_4K }, NULL); + result[1] = mmap_region(file, start + 2 * PAGE_SIZE, PAGE_SIZE, flags, + (struct vm_page_offset) { 2, SZ_4K }, NULL); + result[2] = mmap_region(file, start, PAGE_SIZE, flags, + (struct vm_page_offset) { 0, SZ_4K }, NULL); + /* Adjacent virtual bytes with a different file offset must not merge. */ + result[3] = mmap_region(file, start + 3 * PAGE_SIZE, PAGE_SIZE, flags, + (struct vm_page_offset) { 3, 2 * SZ_4K }, NULL); + mmap_write_unlock(mm); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, result[0], start + PAGE_SIZE); + KUNIT_ASSERT_EQ(test, result[1], start + 2 * PAGE_SIZE); + KUNIT_ASSERT_EQ(test, result[2], start); + KUNIT_ASSERT_EQ(test, result[3], start + 3 * PAGE_SIZE); + KUNIT_EXPECT_EQ(test, mm->map_count, 2); + vma = test_vma_lookup(mm, start); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma->vm_start, start); + KUNIT_EXPECT_EQ(test, vma->vm_end, start + 3 * PAGE_SIZE); + for (i = 0; i < 12; i++) + KUNIT_EXPECT_TRUE(test, vm_page_offset_equal(vma_page_offset_at(vma, start + i * SZ_4K), + (struct vm_page_offset) { (i + 1) / TEST_SLOTS, ((i + 1) % TEST_SLOTS) * SZ_4K })); + vma = test_vma_lookup(mm, start + 3 * PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(vma), 2U * SZ_4K); +} + +#ifdef CONFIG_PROC_FS +/* Exercise the actual status/statm producers without publishing a fake task. */ +extern void task_mem(struct seq_file *m, struct mm_struct *mm); +extern unsigned long task_vsize(struct mm_struct *mm); +extern unsigned long task_statm(struct mm_struct *mm, unsigned long *shared, + unsigned long *text, unsigned long *data, unsigned long *resident); + +static void expect_status_kb(struct kunit *test, struct seq_file *seq, + const char *name, unsigned long expected) +{ + const char *field = strstr(seq->buf, name); + unsigned long value = ULONG_MAX; + + KUNIT_ASSERT_NOT_NULL(test, field); + KUNIT_EXPECT_EQ(test, sscanf(field + strlen(name), "%lu", &value), 1); + KUNIT_EXPECT_EQ_MSG(test, value, expected, "%s", name); +} + +static void user4k_accounting_readers_test(struct kunit *test) +{ + struct mm_struct *small = test_vma_mm(test); + struct mm_struct *native = mm_alloc(); + struct seq_file seq = {}; + unsigned long shared, text, data, resident, size, maxrss = 0; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, native), 0); + seq.size = 2048; + seq.buf = kunit_kzalloc(test, seq.size, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, seq.buf); + /* Same logical counts, deliberately not multiples of a native page. */ + for (i = 0; i < 2; i++) { + struct mm_struct *mm = i ? native : small; + unsigned long kb = i ? PAGE_SIZE / 1024 : 4; + + mm->total_vm = 11; + mm->hiwater_vm = 13; + mm->data_vm = 5; + mm->stack_vm = 2; + mm->exec_vm = 3; + mm->locked_vm = mm_pages_to_account(mm, 1); + atomic64_set(&mm->pinned_vm, 2); + mm->hiwater_rss = 10; + mm->start_code = TEST_VA + mm_page_size(mm); + mm->end_code = mm->start_code + 2 * mm_page_size(mm); + percpu_counter_set(&mm->rss_stat[MM_ANONPAGES], 5); + percpu_counter_set(&mm->rss_stat[MM_FILEPAGES], 2); + percpu_counter_set(&mm->rss_stat[MM_SHMEMPAGES], 1); + percpu_counter_set(&mm->rss_stat[MM_SWAPENTS], 3); + seq.count = 0; + task_mem(&seq, mm); + KUNIT_EXPECT_FALSE(test, seq_has_overflowed(&seq)); + seq.buf[min(seq.count, seq.size - 1)] = '\0'; + expect_status_kb(test, &seq, "VmPeak:", 13 * kb); + expect_status_kb(test, &seq, "VmSize:", 11 * kb); + expect_status_kb(test, &seq, "VmLck:", kb); + expect_status_kb(test, &seq, "VmPin:", 2 * PAGE_SIZE / 1024); + expect_status_kb(test, &seq, "VmHWM:", 10 * kb); + expect_status_kb(test, &seq, "VmRSS:", 8 * kb); + expect_status_kb(test, &seq, "RssAnon:", 5 * kb); + expect_status_kb(test, &seq, "RssFile:", 2 * kb); + expect_status_kb(test, &seq, "RssShmem:", kb); + expect_status_kb(test, &seq, "VmData:", 5 * kb); + expect_status_kb(test, &seq, "VmStk:", 2 * kb); + expect_status_kb(test, &seq, "VmExe:", 2 * kb); + expect_status_kb(test, &seq, "VmLib:", kb); + expect_status_kb(test, &seq, "VmSwap:", 3 * kb); + KUNIT_EXPECT_EQ(test, task_vsize(mm), 11 * kb * 1024); + { + struct vm_area_struct vma = { .vm_mm = mm }; + + KUNIT_EXPECT_EQ(test, vma_kernel_pagesize(&vma), kb * 1024); + KUNIT_EXPECT_EQ(test, vma_mmu_pagesize(&vma), kb * 1024); + } + } + /* Maxima must compare bytes, not incompatible 4K/16K counts. */ + setmax_mm_hiwater_rss(&maxrss, small); + KUNIT_EXPECT_EQ(test, maxrss, 40UL); + setmax_mm_hiwater_rss(&maxrss, native); + KUNIT_EXPECT_EQ(test, maxrss, 10UL * PAGE_SIZE / 1024); + setmax_mm_hiwater_rss(&maxrss, small); + KUNIT_EXPECT_EQ(test, maxrss, 10UL * PAGE_SIZE / 1024); + small->hiwater_rss = 10 * TEST_SLOTS + 1; + setmax_mm_hiwater_rss(&maxrss, small); + KUNIT_EXPECT_EQ(test, maxrss, (10UL * TEST_SLOTS + 1) * 4); + + KUNIT_EXPECT_EQ(test, mm_pages_to_native(small, 0), 0UL); + KUNIT_EXPECT_EQ(test, mm_pages_to_native(small, 1), 1UL); + KUNIT_EXPECT_EQ(test, mm_pages_to_native(small, 5), DIV_ROUND_UP(5UL, TEST_SLOTS)); + KUNIT_EXPECT_EQ(test, mm_pages_for_reader(native, small, 5), 5UL * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, mm_pages_for_reader(small, native, 5), DIV_ROUND_UP(5UL, TEST_SLOTS)); + KUNIT_EXPECT_EQ(test, mm_pages_for_reader(small, small, 5), 5UL); + KUNIT_EXPECT_EQ(test, mm_pages_for_reader(native, native, 5), 5UL); + KUNIT_EXPECT_EQ(test, mm_pages_to_shift(small, 5, MM_ACCOUNT_SHIFT), 5UL); + KUNIT_EXPECT_EQ(test, mm_pages_to_shift(native, 5, MM_ACCOUNT_SHIFT), 5UL * TEST_SLOTS); + + /* A kernel reader defaults to native units; partial pages round up. */ + size = task_statm(small, &shared, &text, &data, &resident); + KUNIT_EXPECT_EQ(test, size, DIV_ROUND_UP(11UL, TEST_SLOTS)); + KUNIT_EXPECT_EQ(test, shared, 1UL); + KUNIT_EXPECT_EQ(test, text, 1UL); + KUNIT_EXPECT_EQ(test, data, DIV_ROUND_UP(7UL, TEST_SLOTS)); + KUNIT_EXPECT_EQ(test, resident, DIV_ROUND_UP(8UL, TEST_SLOTS)); + if (user4k_test_enabled || IS_ENABLED(CONFIG_KASAN)) { + kthread_use_mm(small); + size = task_statm(native, &shared, &text, &data, &resident); + kthread_unuse_mm(small); + KUNIT_EXPECT_EQ(test, size, 11UL * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, shared, 3UL * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, text, 2UL * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, data, 7UL * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, resident, 8UL * TEST_SLOTS); + } + /* Synthetic counters have no page references; restore before mmput. */ + for (i = 0; i < 2; i++) { + struct mm_struct *mm = i ? native : small; + unsigned int counter; + + for (counter = 0; counter < NR_MM_COUNTERS; counter++) + percpu_counter_set(&mm->rss_stat[counter], 0); + mm->total_vm = mm->data_vm = mm->stack_vm = mm->exec_vm = 0; + mm->locked_vm = 0; + atomic64_set(&mm->pinned_vm, 0); + } +} +#endif + +static void mixed_locked_account_units_test(struct kunit *test) +{ + unsigned int shift; + unsigned long native_units = PAGE_SIZE >> MM_ACCOUNT_SHIFT; + + for (shift = 12; shift <= 16; shift += 2) { + struct mm_struct *mm = mm_alloc(); + unsigned long size = 1UL << shift; + unsigned long base, limit; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + if (shift != PAGE_SHIFT) + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, shift), 0); + base = mm_pages_to_account(mm, 1); + mmap_write_lock(mm); + /* Model one locked VMA leaf and two independently charged native pins. */ + mm->locked_vm = base; + ret = __account_locked_vm(mm, 1, true, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->locked_vm, base + native_units); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(mm), size + PAGE_SIZE); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(mm), + DIV_ROUND_UP(size + PAGE_SIZE, PAGE_SIZE)); + ret = __account_locked_vm(mm, 1, true, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->locked_vm, base + 2 * native_units); + ret = __account_locked_vm(mm, 1, false, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(mm), size + PAGE_SIZE); + ret = __account_locked_vm(mm, 1, false, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->locked_vm, base); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(mm), size); + + /* A refused charge must leave the exact pre-existing amount intact. */ + limit = task_rlimit(current, RLIMIT_MEMLOCK) >> MM_ACCOUNT_SHIFT; + mm->locked_vm = limit; + ret = __account_locked_vm(mm, 1, true, current, false); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm->locked_vm, limit); + mm->locked_vm = ULONG_MAX - native_units + 1; + ret = __account_locked_vm(mm, 1, true, current, true); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm->locked_vm, ULONG_MAX - native_units + 1); + mm->locked_vm = 0; + if (native_units > 1) { + ret = __account_locked_vm(mm, + (ULONG_MAX >> (PAGE_SHIFT - MM_ACCOUNT_SHIFT)) + 1, + true, current, true); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm->locked_vm, 0UL); + } + mmap_write_unlock(mm); + } +} + +/* Prior shmem fixtures can refund global commit from deferred fput work. */ +static s64 test_commit_baseline(struct kunit *test) +{ + s64 before = percpu_counter_sum(&vm_committed_as), after; + + /* No mmap, file, or filesystem locks are held at these call sites. */ + flush_delayed_fput(); + after = percpu_counter_sum(&vm_committed_as); + if (after != before) + kunit_info(test, "drained deferred file refunds: commit delta %lld", after - before); + return after; +} + +static void user64k_commit_accounting_test(struct kunit *test) +{ + struct mm_struct *mm; + s64 before; + unsigned int i; + unsigned long native_per_leaf; + int ret; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires userspace granule larger than native"); + return; + } + mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, 16), 0); + native_per_leaf = SZ_64K / PAGE_SIZE; + before = test_commit_baseline(test); + mmap_write_lock(mm); + for (i = 0; i < 9; i++) { + ret = mm_account_memory(mm, mm, 1); + KUNIT_EXPECT_EQ(test, ret, 0); + if (ret) + break; + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i + 1); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)((i + 1) * native_per_leaf)); + } + /* Multiplication overflow must not mutate either reservation counter. */ + ret = mm_account_memory(mm, mm, LONG_MAX / native_per_leaf + 1); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)(i * native_per_leaf)); + mm_acct_memory(mm, 3); + i += 3; + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)(i * native_per_leaf)); + while (i) { + mm_unacct_memory(mm, 1); + i--; + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)(i * native_per_leaf)); + } + mmap_write_unlock(mm); +} + +#ifdef CONFIG_PAGE_TABLE_CHECK +static void expect_coarse_file_counts(struct kunit *test, void *data, + unsigned int phase) +{ + unsigned int offset; + + for (offset = 0; offset < 2 * SZ_64K; offset += SZ_4K) { + int anon = -1, file = -1, expected; + int ret = page_table_check_get_counts(virt_to_page(data + offset), + offset & ~PAGE_MASK, &anon, &file); + + /* Phase0: both coarse leaves;1:plusalias;2:firstclear;3:bothclear;4:allclear. */ + expected = phase < 2 || (phase == 2 && offset >= SZ_64K); + if (phase >= 1 && phase <= 3 && offset == 2 * SZ_64K - SZ_4K) + expected++; + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, anon, 0); + KUNIT_EXPECT_EQ_MSG(test, file, expected, "phase=%u offset=%u", phase, offset); + } +} +#endif + +static void user64k_page_table_check_test(struct kunit *test) +{ +#ifdef CONFIG_PAGE_TABLE_CHECK + struct mm_struct *coarse, *small; + void *data; + pte_t ptes[2] = {}, alias = __pte(0), first, last; + int anon, file; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires userspace leaf larger than native page"); + return; + } + data = test_alloc_granule(test, 17); + KUNIT_ASSERT_NOT_NULL(test, data); + if (page_table_check_get_counts(virt_to_page(data), 0, &anon, &file)) { + kunit_skip(test, "requires page-table-check enabled at boot"); + return; + } + coarse = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, coarse), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(coarse, 16), 0); + small = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, small), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(small, 12), 0); + first = phys_pte_mm(coarse, virt_to_phys(data), __pgprot(TEST_PROT)); + last = phys_pte_mm(small, virt_to_phys(data) + 2 * SZ_64K - SZ_4K, + __pgprot(TEST_PROT)); + + /* Exercise checker hooks directly; no actual coarse file fault is claimed. */ + page_table_check_ptes_set(coarse, TEST_VA, ptes, first, 2); + ptes[0] = first; + ptes[1] = pte_advance_pfn_mm(coarse, first, 1); + expect_coarse_file_counts(test, data, 0); + page_table_check_ptes_set(small, TEST_VA, &alias, last, 1); + alias = last; + expect_coarse_file_counts(test, data, 1); + page_table_check_pte_clear(coarse, TEST_VA, ptes[0]); + ptes[0] = __pte(0); + expect_coarse_file_counts(test, data, 2); + page_table_check_pte_clear(coarse, TEST_VA + SZ_64K, ptes[1]); + ptes[1] = __pte(0); + expect_coarse_file_counts(test, data, 3); + page_table_check_pte_clear(small, TEST_VA, alias); + expect_coarse_file_counts(test, data, 4); +#else + kunit_skip(test, "requires CONFIG_PAGE_TABLE_CHECK"); +#endif +} + +static void user4k_commit_accounting_test(struct kunit *test) +{ + struct mm_struct *mm = test_vma_mm(test); + s64 before; + int ret; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, mm); + before = test_commit_baseline(test); + mmap_write_lock(mm); + for (i = 0; i < 9; i++) { + ret = mm_account_memory(mm, mm, 1); + KUNIT_EXPECT_EQ(test, ret, 0); + if (ret) + break; + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i + 1); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)DIV_ROUND_UP(i + 1, TEST_SLOTS)); + } + if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) { + unsigned long old = mm->committed_user_pages; + s64 committed = percpu_counter_sum(&vm_committed_as); + + ret = mm_account_memory(mm, mm, (totalram_pages() + total_swap_pages + 1) * TEST_SLOTS); + KUNIT_EXPECT_EQ(test, ret, -ENOMEM); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, old); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as), committed); + } + /* Native physical-page callers must charge four user granules per page. */ + ret = __account_locked_vm(mm, 1, true, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->locked_vm, (unsigned long)TEST_SLOTS); + ret = __account_locked_vm(mm, 1, false, current, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->locked_vm, 0UL); + + /* Refund one granule at a time: splitting must not multiply refunds. */ + while (i) { + mm_unacct_memory(mm, 1); + i--; + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)i); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, + (s64)DIV_ROUND_UP(i, TEST_SLOTS)); + } + mmap_write_unlock(mm); +} + +/* Prefaulting must stop at each process-page boundary, not native pages. */ +static void user_granule_prefault_test(struct kunit *test) +{ + unsigned int shift; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mapping fixture requires hardware opt-in"); + return; + } + for (shift = 12; shift <= PAGE_SHIFT; shift += 2) { + struct mm_struct *mm = test_vma_mm_granule(test, shift); + unsigned long size = 1UL << shift, start = TEST_VA; + unsigned long flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE; + unsigned long mapped[3]; + size_t left[3][4] = {}; + int phase, ret = 0; + + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + mapped[0] = vm_mmap(NULL, start, size, PROT_READ | PROT_WRITE, flags, 0); + mapped[2] = vm_mmap(NULL, start + 2 * size, size, + PROT_READ | PROT_WRITE, flags, 0); + mapped[1] = start + size; + if (mapped[0] == start && mapped[2] == start + 2 * size) { + for (phase = 0; phase < 3; phase++) { + char __user *addr = (char __user *)(start + 17); + size_t len = 3 * size - 17; + + /* Missing middle, then read-only, then fully writable. */ + if (phase) { + mapped[1] = vm_mmap(NULL, start + size, size, + PROT_READ | (phase == 2 ? PROT_WRITE : 0), + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, 0); + if (mapped[1] != start + size) + break; + } + left[phase][0] = fault_in_safe_writeable(addr, len); + left[phase][1] = fault_in_readable(addr, len); + left[phase][2] = fault_in_writeable(addr, len); + left[phase][3] = fault_in_subpage_writeable(addr, len); + } + ret = vm_munmap(start, 3 * size); + } + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped[0], start); + KUNIT_ASSERT_EQ(test, mapped[1], start + size); + KUNIT_ASSERT_EQ(test, mapped[2], start + 2 * size); + KUNIT_EXPECT_EQ(test, ret, 0); + for (phase = 0; phase < 3; phase++) { + KUNIT_EXPECT_EQ(test, left[phase][0], phase < 2 ? 2 * size : 0UL); + KUNIT_EXPECT_EQ(test, left[phase][1], phase == 0 ? 2 * size : 0UL); + KUNIT_EXPECT_EQ(test, left[phase][2], phase < 2 ? 2 * size : 0UL); + KUNIT_EXPECT_EQ(test, left[phase][3], phase < 2 ? 2 * size : 0UL); + } + } +} + +static void user4k_mmap_accounting_test(struct kunit *test) +{ + struct mm_struct *mm; + struct vm_area_struct *vma; + unsigned long start = TEST_VA + SZ_4K, mapped; + vm_flags_t flags = VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE; + s64 before; + unsigned int i; + int ret = 0; + vm_fault_t fault = 0; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mapping fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + before = test_commit_baseline(test); + kthread_use_mm(mm); + mmap_write_lock(mm); + mapped = mmap_region(NULL, start, 3 * SZ_4K, flags, + (struct vm_page_offset) { start >> PAGE_SHIFT, SZ_4K }, NULL); + mmap_write_unlock(mm); + if (!IS_ERR_VALUE(mapped)) { + mmap_read_lock(mm); + vma = vma_lookup(mm, start); + fault = handle_mm_fault(vma, start, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + } + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, start); + KUNIT_ASSERT_EQ(test, fault & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_EQ(test, mm->total_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm->data_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 3UL); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, 1LL); + /* Remove the middle first, then the two neighbors. */ + for (i = 0; i < 3; i++) { + unsigned int slot = i == 0 ? 1 : i == 1 ? 0 : 2; + VMA_ITERATOR(vmi, mm, start + slot * SZ_4K); + + kthread_use_mm(mm); + mmap_write_lock(mm); + ret = do_vmi_munmap(&vmi, mm, start + slot * SZ_4K, SZ_4K, NULL, false); + mmap_write_unlock(mm); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->total_vm, 2UL - i); + KUNIT_EXPECT_EQ(test, mm->data_vm, 2UL - i); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 2UL - i); + KUNIT_EXPECT_EQ(test, percpu_counter_sum(&vm_committed_as) - before, i == 2 ? 0LL : 1LL); + } + KUNIT_EXPECT_EQ(test, mm->map_count, 0); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0L); +} + +static vm_fault_t test_file_fault(struct vm_area_struct *vma, unsigned long addr, bool write) +{ + vm_fault_t ret; + + mmap_read_lock(vma->vm_mm); + ret = handle_mm_fault(vma, addr, FAULT_FLAG_REMOTE | + (write ? FAULT_FLAG_WRITE : 0), NULL); + mmap_read_unlock(vma->vm_mm); + return ret; +} + +static void user4k_vm_mmap_test(struct kunit *test) +{ + struct mm_struct *mm; + struct file *file; + struct vm_area_struct *vma; + unsigned long start = TEST_VA + SZ_4K; + unsigned long anon_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE; + unsigned long file_flags = MAP_SHARED | MAP_FIXED_NOREPLACE; + unsigned long result[7]; + int unmap[4]; + pte_t pte; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mapping fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, 2 * PAGE_SIZE, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + mm->mmap_base = TEST_VA + 2 * PAGE_SIZE + SZ_4K; + mm_flags_clear(MMF_TOPDOWN, mm); + kthread_use_mm(mm); + result[0] = vm_mmap(NULL, start, SZ_4K, PROT_READ | PROT_WRITE, anon_flags, 0); + result[1] = vm_mmap(file, start + 2 * SZ_4K, SZ_4K, PROT_READ | PROT_WRITE, + file_flags, 3 * SZ_4K); + result[2] = vm_mmap(file, start + 3 * SZ_4K, SZ_4K, PROT_READ | PROT_WRITE, + file_flags, 4 * SZ_4K); + result[3] = vm_mmap(NULL, start, SZ_4K, PROT_READ, anon_flags, 0); + result[4] = vm_mmap(NULL, start + 17, SZ_4K, PROT_READ, anon_flags, 0); + result[5] = vm_mmap(file, start + SZ_4K, SZ_4K, PROT_READ, file_flags, SZ_4K + 1); + result[6] = vm_mmap(NULL, 0, SZ_4K + 1, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, result[0], start); + KUNIT_ASSERT_EQ(test, result[1], start + 2 * SZ_4K); + KUNIT_ASSERT_EQ(test, result[2], start + 3 * SZ_4K); + KUNIT_EXPECT_EQ(test, result[3], (unsigned long)-EEXIST); + KUNIT_EXPECT_EQ(test, result[4], (unsigned long)-EINVAL); + KUNIT_EXPECT_EQ(test, result[5], (unsigned long)-EINVAL); + KUNIT_ASSERT_EQ(test, result[6], mm->mmap_base); + KUNIT_EXPECT_EQ(test, mm->total_vm, 5UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 3UL); + vma = test_vma_lookup(mm, start); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, start, true) & VM_FAULT_ERROR, 0U); + vma = test_vma_lookup(mm, start + 2 * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma->vm_end - vma->vm_start, 2UL * SZ_4K); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(vma), 3U * SZ_4K); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, start + 2 * SZ_4K, false) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, mm, start + 2 * SZ_4K); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, *(u8 *)phys_to_virt(pte_phys_mm(mm, pte)), (u8)0x34); + pte = test_fault_entry(test, mm, start + 3 * SZ_4K); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, *(u8 *)phys_to_virt(pte_phys_mm(mm, pte)), (u8)0x35); + kthread_use_mm(mm); + unmap[0] = vm_munmap(start + 2 * SZ_4K, 1); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, unmap[0], 0); + vma = test_vma_lookup(mm, start + 3 * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma->vm_pgoff, 4UL / TEST_SLOTS); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(vma), (4U % TEST_SLOTS) * SZ_4K); + KUNIT_EXPECT_EQ(test, mm->total_vm, 4UL); + kthread_use_mm(mm); + unmap[1] = vm_munmap(start, SZ_4K); + unmap[2] = vm_munmap(start + 3 * SZ_4K, SZ_4K); + unmap[3] = vm_munmap(result[6], 2 * SZ_4K); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, unmap[1], 0); + KUNIT_EXPECT_EQ(test, unmap[2], 0); + KUNIT_EXPECT_EQ(test, unmap[3], 0); + KUNIT_EXPECT_EQ(test, mm->total_vm, 0UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 0UL); + KUNIT_EXPECT_EQ(test, mm->map_count, 0); +} + +static void test_mlock_drain_all(struct kunit *test, struct folio *folio) +{ + unsigned int before, after; + bool counted; + + lru_add_drain(); + counted = folio_test_unevictable(folio); + before = counted ? folio->mlock_count : 0; + /* A sleeping fixture can enqueue mlock work on more than one CPU. */ + lru_add_drain_all(); + after = folio_test_unevictable(folio) ? folio->mlock_count : 0; + if (counted && before != after) + kunit_info(test, "remote mlock drain changed count %u -> %u", before, after); +} + +static void user4k_mlock_counts_test(struct kunit *test) +{ + struct mm_struct *mm = test_vma_mm(test); + struct mm_subpage *slot; + struct folio *folio; + unsigned long addr; + unsigned int i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mlock requires hardware opt-in"); + return; + } + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE | MAP_LOCKED, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, test_fault_entry(test, mm, TEST_VA))); + KUNIT_ASSERT_NOT_NULL(test, slot); + folio = mm_subpage_folio(slot); + folio_get(folio); + mm_subpage_put(slot); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + test_mlock_drain_all(test, folio); + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + KUNIT_EXPECT_EQ(test, folio->mlock_count, TEST_SLOTS); + KUNIT_EXPECT_EQ(test, mm->locked_vm, (unsigned long)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) { + int ret; + + kthread_use_mm(mm); + ret = vm_munmap(TEST_VA + i * SZ_4K, SZ_4K); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + test_mlock_drain_all(test, folio); + /* mlock_count shares storage with the evictable LRU links. */ + if (i != TEST_SLOTS - 1) + KUNIT_EXPECT_EQ(test, folio->mlock_count, TEST_SLOTS - 1 - i); + KUNIT_EXPECT_EQ(test, mm->locked_vm, TEST_SLOTS - 1UL - i); + KUNIT_EXPECT_EQ(test, folio_test_mlocked(folio), i != TEST_SLOTS - 1); + } +} + +static void test_mlock_enable_lru_cache(void *unused) +{ + lru_cache_enable(); +} + +static void user4k_mlock_shared_test(struct kunit *test) +{ + struct mm_struct *small = test_vma_mm(test), *native = mm_alloc(); + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct folio *folio; + unsigned long addr; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + if (native) + mmput(native); + kunit_skip(test, "borrowed-mm mlock requires hardware opt-in"); + return; + } + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, native), 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + /* Exact counts require synchronous LRU/lock accounting. Normal deferred + * batches may legitimately miss locks while the folio is off LRU; + * user4k_mlock_repair_test checks that reclaim repairs that case. + */ + lru_cache_disable(); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + test_mlock_enable_lru_cache, NULL), 0); + kthread_use_mm(small); + addr = vm_mmap(file, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_LOCKED | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(small); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + kthread_use_mm(native); + addr = vm_mmap(file, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_LOCKED | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(native); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + test_mlock_drain_all(test, folio); + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + KUNIT_EXPECT_EQ(test, folio->mlock_count, TEST_SLOTS + 1); + kthread_use_mm(small); + ret = vm_munmap(TEST_VA + SZ_4K, SZ_4K); + kthread_unuse_mm(small); + KUNIT_ASSERT_EQ(test, ret, 0); + test_mlock_drain_all(test, folio); + KUNIT_EXPECT_EQ(test, folio->mlock_count, TEST_SLOTS); + kthread_use_mm(native); + ret = vm_munmap(TEST_VA, PAGE_SIZE); + kthread_unuse_mm(native); + KUNIT_ASSERT_EQ(test, ret, 0); + test_mlock_drain_all(test, folio); + KUNIT_EXPECT_EQ(test, folio->mlock_count, TEST_SLOTS - 1); + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + kthread_use_mm(small); + ret = vm_munmap(TEST_VA, PAGE_SIZE); + kthread_unuse_mm(small); + KUNIT_ASSERT_EQ(test, ret, 0); + test_mlock_drain_all(test, folio); + KUNIT_EXPECT_FALSE(test, folio_test_mlocked(folio)); +} + +static void test_mlock_putback(void *folio) +{ + folio_putback_lru(folio); +} + +static void user4k_mlock_repair_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + unsigned int mode, walker; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mlock requires hardware opt-in"); + return; + } + for (mode = 0; mode < (PAGE_SHIFT > 14 ? 3U : 2U); mode++) { + unsigned long size = 1UL << shifts[mode]; + + for (walker = 0; walker < 2; walker++) { + struct mm_struct *mm = test_vma_mm_granule(test, shifts[mode]); + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct folio *folio; + unsigned long addr, offset; + vm_flags_t flags = 0; + int maps, ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + test_folio_put, folio), 0); + lru_add_drain_all(); + KUNIT_ASSERT_TRUE(test, folio_isolate_lru(folio)); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + test_mlock_putback, folio), 0); + /* Two aliases leave a locked mapping even after native unmap. + * Force deferred mlocks to run while the folio is isolated. + */ + for (offset = 0; offset < 2 * PAGE_SIZE; offset += PAGE_SIZE) { + kthread_use_mm(mm); + addr = vm_mmap(file, TEST_VA + offset, PAGE_SIZE, + PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_LOCKED | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA + offset); + } + lru_add_drain_all(); + KUNIT_EXPECT_FALSE(test, folio_test_lru(folio)); + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + kunit_release_action(test, test_mlock_putback, folio); + lru_add_drain_all(); + KUNIT_ASSERT_TRUE(test, folio_test_unevictable(folio)); + KUNIT_EXPECT_EQ(test, folio->mlock_count, 0U); + + kthread_use_mm(mm); + ret = vm_munmap(TEST_VA, size); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + KUNIT_EXPECT_FALSE(test, folio_test_mlocked(folio)); + maps = folio_mapcount(folio); + KUNIT_EXPECT_EQ(test, maps, (int)(2 * PAGE_SIZE / size) - 1); + folio_lock(folio); + if (walker) + ret = folio_referenced(folio, 1, NULL, &flags); + else + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + lru_add_drain_all(); + if (walker) { + KUNIT_EXPECT_GE(test, ret, 0); + KUNIT_EXPECT_TRUE(test, flags & VM_LOCKED); + } + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), maps); + for (offset = size; offset < 2 * PAGE_SIZE; offset += size) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + offset); + + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_PTR_EQ(test, + page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(mm, pte)))), + folio); + } + kthread_use_mm(mm); + ret = vm_munmap(TEST_VA, 2 * PAGE_SIZE); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + KUNIT_EXPECT_FALSE(test, folio_test_mlocked(folio)); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + kunit_info(test, "%uK %s repairs deferred mlock without unmapping", + 1U << (shifts[mode] - 10), walker ? "referenced" : "unmap"); + } + } +} + +asmlinkage long __arm64_sys_mremap(const struct pt_regs *regs); + +static unsigned long test_mremap(unsigned long old, unsigned long old_len, + unsigned long new_len, unsigned long flags, unsigned long new) +{ + struct pt_regs regs = { .regs = { old, old_len, new_len, flags, new } }; + + return __arm64_sys_mremap(®s); +} + +static int test_iova_bitmap_fill(struct iova_bitmap *bitmap, unsigned long iova, + size_t length, void *opaque) +{ + unsigned long unit = *(unsigned long *)opaque; + + iova_bitmap_set(bitmap, iova + 3 * unit, length - 10 * unit); + /* A second write advances backwards into an already populated range. */ + iova_bitmap_set(bitmap, iova, 5 * unit); + return 0; +} + +/* Real IOMMUFD lifecycle with a translation-recording IOMMU, no physical DMA. */ +struct test_fragment_domain { + struct iommu_domain domain; + struct xarray translations; + unsigned long fail_after; + unsigned long largest_map; + unsigned long mapped; + unsigned long map_calls; +}; + +struct test_fragment_iopt { + struct io_pagetable table; + struct test_fragment_domain domains[2]; + struct kunit *test; +}; + +static int test_fragment_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t phys, size_t pgsize, size_t pgcount, int prot, + gfp_t gfp, size_t *mapped) +{ + struct test_fragment_domain *mock = container_of(domain, typeof(*mock), domain); + unsigned long length = pgsize * pgcount, done; + int rc; + + mock->map_calls++; + mock->largest_map = max(mock->largest_map, length); + for (done = 0; done < length; done += SZ_4K) { + if (mock->fail_after && mock->mapped >= mock->fail_after) + return -ENOMEM; + rc = xa_insert(&mock->translations, (iova + done) >> 12, + xa_mk_value((phys + done) >> 12), gfp); + if (rc) + return rc; + *mapped += SZ_4K; + mock->mapped += SZ_4K; + } + return 0; +} + +static size_t test_fragment_unmap(struct iommu_domain *domain, unsigned long iova, + size_t pgsize, size_t pgcount, struct iommu_iotlb_gather *gather) +{ + struct test_fragment_domain *mock = container_of(domain, typeof(*mock), domain); + unsigned long length = pgsize * pgcount, done; + + for (done = 0; done < length; done += SZ_4K) { + if (!xa_erase(&mock->translations, (iova + done) >> 12)) + break; + mock->mapped -= SZ_4K; + } + return done; +} + +static phys_addr_t test_fragment_translate(struct iommu_domain *domain, dma_addr_t iova) +{ + struct test_fragment_domain *mock = container_of(domain, typeof(*mock), domain); + void *entry = xa_load(&mock->translations, iova >> 12); + + return entry ? ((phys_addr_t)xa_to_value(entry) << 12) + (iova & (SZ_4K - 1)) : 0; +} + +static void test_fragment_sync(struct iommu_domain *domain, struct iommu_iotlb_gather *gather) +{ +} + +static const struct iommu_domain_ops test_fragment_domain_ops = { + .map_pages = test_fragment_map, + .unmap_pages = test_fragment_unmap, + .iova_to_phys = test_fragment_translate, + .iotlb_sync = test_fragment_sync, +}; + +static void test_fragment_iopt_free(void *data) +{ + struct test_fragment_iopt *fixture = data; + struct iommu_domain *domain; + unsigned long unmapped, index; + unsigned int i; + + iopt_unmap_all(&fixture->table, &unmapped); + while ((domain = xa_find(&fixture->table.domains, &(unsigned long){0}, + ULONG_MAX, XA_PRESENT))) + iopt_table_remove_domain(&fixture->table, domain); + iopt_destroy_table(&fixture->table); + for (i = 0; i < ARRAY_SIZE(fixture->domains); i++) { + struct test_fragment_domain *mock = &fixture->domains[i]; + void *entry; + + KUNIT_EXPECT_TRUE(fixture->test, xa_empty(&mock->translations)); + xa_for_each(&mock->translations, index, entry) + xa_erase(&mock->translations, index); + xa_destroy(&mock->translations); + } +} + +static struct test_fragment_iopt *test_fragment_iopt_alloc(struct kunit *test) +{ + struct test_fragment_iopt *fixture = kunit_kzalloc(test, sizeof(*fixture), GFP_KERNEL); + unsigned int i; + + if (!fixture) + return NULL; + fixture->test = test; + iopt_init_table(&fixture->table); + for (i = 0; i < ARRAY_SIZE(fixture->domains); i++) { + struct test_fragment_domain *mock = &fixture->domains[i]; + + xa_init(&mock->translations); + mock->domain.ops = &test_fragment_domain_ops; + mock->domain.type = IOMMU_DOMAIN_UNMANAGED; + mock->domain.pgsize_bitmap = SZ_4K; + mock->domain.geometry.aperture_end = ULONG_MAX; + } + if (kunit_add_action_or_reset(test, test_fragment_iopt_free, fixture)) + return NULL; + return fixture; +} + +static int test_fragment_map_user(struct test_fragment_iopt *fixture, + struct mm_struct *mm, unsigned long va, unsigned long iova, unsigned long length) +{ + struct iommufd_ctx ctx = {}; + int rc; + + kthread_use_mm(mm); + rc = iopt_map_user_pages(&ctx, &fixture->table, &iova, (void __user *)va, + length, IOMMU_READ | IOMMU_WRITE, 0); + kthread_unuse_mm(mm); + return rc; +} + +static void user4k_iommufd_lifetime_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14, 16 }, mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm IOMMUFD requires hardware opt-in"); + return; + } + for (mode = 0; mode < ARRAY_SIZE(shifts); mode++) { + unsigned long size = 1UL << shifts[mode]; + unsigned long len = 2 * max(size, PAGE_SIZE); + unsigned long dest = TEST_VA + 32 * PAGE_SIZE + size; + unsigned long iova = SZ_1M, alias = SZ_16M, addr, moved, missed, unmapped; + struct mm_struct *mm; + struct test_fragment_iopt *fixture; + struct iopt_pages *pages; + struct iopt_area *area; + unsigned char *buffer; + unsigned long off, pinned; + LIST_HEAD(copy); + int rc; + + if (mode && shifts[mode] == PAGE_SHIFT) + continue; + mm = test_vma_mm_granule(test, shifts[mode]); + KUNIT_ASSERT_NOT_NULL(test, mm); + fixture = test_fragment_iopt_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + buffer = kunit_kmalloc(test, len, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, buffer); + memset(buffer, 0x31, len); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + missed = IS_ERR_VALUE(addr) ? len : copy_to_user((void __user *)addr, buffer, len); + moved = missed ? 0 : test_mremap(addr, len, len, MREMAP_FIXED | MREMAP_MAYMOVE, dest); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, moved, dest); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[0].domain), 0); + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, dest, iova, len), 0); + pinned = atomic64_read(&mm->pinned_vm); + KUNIT_EXPECT_EQ(test, pinned, len / PAGE_SIZE); + KUNIT_EXPECT_GE(test, fixture->domains[0].largest_map, PAGE_SIZE); + for (off = 0; off < len; off += size) { + pte_t pte = test_fault_entry(test, mm, dest + off); + phys_addr_t actual = iommu_iova_to_phys(&fixture->domains[0].domain, iova + off); + + KUNIT_EXPECT_EQ(test, actual, pte_phys_mm(mm, pte)); + } + KUNIT_ASSERT_EQ(test, iopt_get_pages(&fixture->table, iova + size, len - size, ©), 0); + rc = iopt_map_pages(&fixture->table, ©, len - size, &alias, IOMMU_READ | IOMMU_WRITE, 0); + iopt_free_pages_list(©); + KUNIT_ASSERT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, (unsigned long)atomic64_read(&mm->pinned_vm), pinned); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[1].domain), 0); + KUNIT_EXPECT_EQ(test, (unsigned long)atomic64_read(&mm->pinned_vm), pinned); + + /* Replace the original VA: mappings and copied IOAS must retain old pins. */ + kthread_use_mm(mm); + addr = vm_mmap(NULL, dest, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, 0); + memset(buffer, 0x5a, len); + missed = IS_ERR_VALUE(addr) ? len : copy_to_user((void __user *)dest, buffer, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, missed, 0UL); + down_read(&fixture->table.iova_rwsem); + area = iopt_area_iter_first(&fixture->table, iova, iova); + pages = area->pages; + /* Force the domain-backed reader for the native control too. */ + memset(buffer, 0x77, len); + rc = iopt_pages_rw_access(pages, 0, buffer, len, IOMMUFD_ACCESS_RW_WRITE); + up_read(&fixture->table.iova_rwsem); + KUNIT_ASSERT_EQ(test, rc, 0); + for (off = 0; off < len; off += SZ_4K) { + phys_addr_t phys = iommu_iova_to_phys(&fixture->domains[1].domain, iova + off); + void *ptr = kmap_local_page(pfn_to_page(PHYS_PFN(phys))); + bool correct = !memchr_inv(ptr + offset_in_page(phys), 0x77, SZ_4K); + + kunmap_local(ptr); + KUNIT_EXPECT_TRUE(test, correct); + } + kthread_use_mm(mm); + missed = copy_from_user(buffer, (void __user *)dest, len); + /* Local CPU access retains the existing copy_from_user fast path. */ + rc = iopt_pages_rw_access(pages, 0, buffer, len, 0); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, missed, 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0x5a, len), NULL); + KUNIT_ASSERT_EQ(test, iopt_unmap_iova(&fixture->table, iova, len, &unmapped), 0); + KUNIT_EXPECT_EQ(test, unmapped, len); + KUNIT_EXPECT_EQ(test, (unsigned long)atomic64_read(&mm->pinned_vm), (len - (size < PAGE_SIZE ? 0 : size)) / PAGE_SIZE); + iopt_table_remove_domain(&fixture->table, &fixture->domains[0].domain); + KUNIT_EXPECT_EQ(test, fixture->domains[1].mapped, len - size); + KUNIT_ASSERT_EQ(test, iopt_unmap_iova(&fixture->table, alias, len - size, &unmapped), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + kunit_release_action(test, test_fragment_iopt_free, fixture); + kunit_release_action(test, user4k_vma_mm_free, mm); + kunit_info(test, "IOMMUFD %luK: relocated PA, batching, copy, two domains, VA replacement and unpin passed", size / SZ_1K); + } +} + +static void user4k_iommufd_rollback_test(struct kunit *test) +{ + struct mm_struct *mm; + struct test_fragment_iopt *fixture; + unsigned long addr, iova = SZ_1M, size = SZ_4K; + int rc; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm IOMMUFD requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + fixture = test_fragment_iopt_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[0].domain), 0); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[1].domain), 0); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, 3 * size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + rc = vm_munmap(TEST_VA + size, size); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, test_fragment_map_user(fixture, mm, addr, iova, 3 * size), -EFAULT); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, 0UL); + KUNIT_EXPECT_EQ(test, fixture->domains[1].mapped, 0UL); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA + size, size, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA + size); + KUNIT_EXPECT_LT(test, test_fragment_map_user(fixture, mm, TEST_VA, iova, 3 * size), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA + size, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA + size); + fixture->domains[1].fail_after = size; + KUNIT_EXPECT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, iova, 3 * size), -ENOMEM); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, 0UL); + KUNIT_EXPECT_EQ(test, fixture->domains[1].mapped, 0UL); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + fixture->domains[1].fail_after = 0; + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, iova, 3 * size), 0); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, 3 * size); + KUNIT_EXPECT_EQ(test, fixture->domains[1].mapped, 3 * size); + kunit_release_action(test, test_fragment_iopt_free, fixture); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + kunit_release_action(test, user4k_vma_mm_free, mm); +} + +static void user4k_iommufd_granule_test(struct kunit *test) +{ + struct mm_struct *mm; + struct test_fragment_iopt *fixture; + unsigned long addr, unmapped, cut = SZ_1M + PAGE_SIZE - 1; + int rc; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm IOMMUFD requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + fixture = test_fragment_iopt_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + fixture->domains[0].domain.pgsize_bitmap = PAGE_SIZE; + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[0].domain), 0); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, 3 * PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + /* The device cannot cover a misaligned fragment by widening the mapping. */ + KUNIT_EXPECT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA + SZ_4K, SZ_1M, PAGE_SIZE), -EINVAL); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, 0UL); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + KUNIT_EXPECT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, SZ_1M, SZ_4K), -EINVAL); + /* Full contiguous native spans still work, including legacy area splitting. */ + KUNIT_ASSERT_EQ(test, iopt_disable_large_pages(&fixture->table), 0); + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, SZ_1M, 2 * PAGE_SIZE), 0); + KUNIT_EXPECT_EQ(test, fixture->domains[0].largest_map, PAGE_SIZE); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 2LL); + KUNIT_ASSERT_EQ(test, iopt_cut_iova(&fixture->table, &cut, 1), 0); + rc = iopt_unmap_iova(&fixture->table, SZ_1M, PAGE_SIZE, &unmapped); + KUNIT_ASSERT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 1LL); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, PAGE_SIZE); + KUNIT_ASSERT_EQ(test, iopt_unmap_all(&fixture->table, &unmapped), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + /* A closed IOVA interval ending at ULONG_MAX must not wrap at map/unmap. */ + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, + ULONG_MAX - 2 * PAGE_SIZE + 1, 2 * PAGE_SIZE), 0); + KUNIT_ASSERT_EQ(test, iopt_unmap_all(&fixture->table, &unmapped), 0); + KUNIT_EXPECT_EQ(test, unmapped, 2 * PAGE_SIZE); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + kunit_release_action(test, test_fragment_iopt_free, fixture); + kunit_release_action(test, user4k_vma_mm_free, mm); +} + +static void user4k_iommufd_shared_test(struct kunit *test) +{ + struct mm_struct *mm; + struct test_fragment_iopt *fixture; + struct file *file; + struct folio *folio; + unsigned long addr, unmapped, iova = SZ_1M; + unsigned char *buffer; + loff_t pos = 0; + phys_addr_t phys; + void *ptr; + int rc; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm IOMMUFD requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + file = test_cache_file(test, 2 * PAGE_SIZE, false); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + fixture = test_fragment_iopt_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + buffer = kunit_kmalloc(test, PAGE_SIZE, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, buffer); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[0].domain), 0); + kthread_use_mm(mm); + addr = vm_mmap(file, TEST_VA, 2 * SZ_4K, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, SZ_4K); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, iova, 2 * SZ_4K), 0); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_ASSERT_TRUE(test, folio_maybe_dma_pinned(folio)); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 1LL); + folio_lock(folio); + folio_clear_dirty_for_io(folio); + folio_unlock(folio); + phys = iommu_iova_to_phys(&fixture->domains[0].domain, iova); + KUNIT_EXPECT_EQ(test, phys, PFN_PHYS(folio_pfn(folio)) + SZ_4K); + ptr = kmap_local_folio(folio, SZ_4K); + memset(ptr, 0x93, 2 * SZ_4K); + kunmap_local(ptr); + kthread_use_mm(mm); + rc = vm_munmap(TEST_VA, 2 * SZ_4K); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, rc, 0); + /* A failed second-domain attachment must retain the first domain's pins. */ + fixture->domains[1].fail_after = SZ_4K; + KUNIT_EXPECT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[1].domain), -ENOMEM); + KUNIT_EXPECT_EQ(test, fixture->domains[0].mapped, 2 * SZ_4K); + KUNIT_EXPECT_EQ(test, fixture->domains[1].mapped, 0UL); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 1LL); + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(folio)); + fixture->domains[1].fail_after = 0; + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[1].domain), 0); + KUNIT_ASSERT_EQ(test, iopt_unmap_all(&fixture->table, &unmapped), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + KUNIT_EXPECT_FALSE(test, folio_maybe_dma_pinned(folio)); + KUNIT_EXPECT_TRUE(test, folio_test_dirty(folio)); + KUNIT_ASSERT_EQ(test, kernel_read(file, buffer, PAGE_SIZE, &pos), (ssize_t)PAGE_SIZE); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + SZ_4K, 0x93, 2 * SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + 3 * SZ_4K, 0, PAGE_SIZE - 3 * SZ_4K), NULL); + kunit_release_action(test, test_fragment_iopt_free, fixture); + kunit_release_action(test, user4k_vma_mm_free, mm); +} + +static void test_iommufd_exit_account(struct kunit *test, unsigned int shift) +{ + struct mm_struct *mm; + struct test_fragment_iopt *fixture; + struct iopt_pages *pages; + struct iopt_area *area; + unsigned long addr, unmapped; + LIST_HEAD(keep); + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm IOMMUFD requires hardware opt-in"); + return; + } + mm = test_vma_mm_granule(test, shift); + KUNIT_ASSERT_NOT_NULL(test, mm); + fixture = test_fragment_iopt_alloc(test); + KUNIT_ASSERT_NOT_NULL(test, fixture); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, 2 * PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_EQ(test, test_fragment_map_user(fixture, mm, TEST_VA, SZ_1M, 2 * PAGE_SIZE), 0); + down_read(&fixture->table.iova_rwsem); + area = iopt_area_iter_first(&fixture->table, SZ_1M, SZ_1M); + pages = area->pages; + mutex_lock(&pages->mutex); + /* Select VFIO-compatible accounting before the lazy range acquires pins. */ + pages->account_mode = IOPT_PAGES_ACCOUNT_MM; + mutex_unlock(&pages->mutex); + up_read(&fixture->table.iova_rwsem); + KUNIT_ASSERT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[0].domain), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 2LL); + KUNIT_EXPECT_EQ(test, mm->locked_vm, 2 * (PAGE_SIZE >> MM_ACCOUNT_SHIFT)); + /* Keep the provider, and thus mm_count, after mm_users reaches zero. */ + KUNIT_ASSERT_EQ(test, iopt_get_pages(&fixture->table, SZ_1M, 2 * PAGE_SIZE, &keep), 0); + kunit_release_action(test, user4k_vma_mm_free, mm); + KUNIT_EXPECT_EQ(test, atomic_read(&mm->mm_users), 0); + KUNIT_EXPECT_EQ(test, iopt_table_add_domain(&fixture->table, &fixture->domains[1].domain), 0); + KUNIT_EXPECT_EQ(test, iopt_unmap_all(&fixture->table, &unmapped), 0); + KUNIT_EXPECT_EQ(test, atomic64_read(&mm->pinned_vm), 0LL); + KUNIT_EXPECT_EQ(test, mm->locked_vm, 0UL); + KUNIT_EXPECT_EQ(test, pages->npinned, 0UL); + KUNIT_EXPECT_EQ(test, pages->last_npinned, 0UL); + iopt_free_pages_list(&keep); + kunit_release_action(test, test_fragment_iopt_free, fixture); +} + +static void user4k_iommufd_exit_account_test(struct kunit *test) +{ + test_iommufd_exit_account(test, PAGE_SHIFT); + test_iommufd_exit_account(test, 12); +} + +static void user4k_iova_bitmap_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + unsigned int mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bitmap requires hardware opt-in"); + return; + } + for (mode = 0; mode < (PAGE_SHIFT > 14 ? 3U : 2U); mode++) { + struct mm_struct *mm = test_vma_mm_granule(test, shifts[mode]); + unsigned long size = 1UL << shifts[mode], len = 4 * size; + unsigned long old = TEST_VA, dest = TEST_VA + 32 * PAGE_SIZE + size; + unsigned long start = dest + size - sizeof(u64), bytes = 2 * size; + unsigned long unit = SZ_4K, iova = 1UL << 32, addr, moved; + unsigned char *actual, *expected; + struct iova_bitmap *bitmap; + unsigned long missed; + int ret = 0; + + KUNIT_ASSERT_NOT_NULL(test, mm); + actual = kunit_kzalloc(test, len, GFP_KERNEL); + expected = kunit_kzalloc(test, len, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, actual); + KUNIT_ASSERT_NOT_NULL(test, expected); + kthread_use_mm(mm); + addr = vm_mmap(NULL, old, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + missed = IS_ERR_VALUE(addr) ? len : copy_to_user((void __user *)addr, actual, len); + moved = missed ? 0 : test_mremap(old, len, len, + MREMAP_FIXED | MREMAP_MAYMOVE, dest); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, old); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_ASSERT_EQ(test, moved, dest); + /* VA and PA offsets differ after moving by one alternative leaf. */ + if (size < PAGE_SIZE) { + pte_t pte = test_fault_entry(test, mm, dest); + + KUNIT_EXPECT_NE(test, pte_phys_mm(mm, pte) & ~PAGE_MASK, + dest & ~PAGE_MASK); + } + kthread_use_mm(mm); + bitmap = iova_bitmap_alloc(iova, bytes * 8 * unit, unit, + (u64 __user *)start); + if (!IS_ERR(bitmap)) { + ret = iova_bitmap_for_each(bitmap, &unit, test_iova_bitmap_fill); + iova_bitmap_free(bitmap); + } else { + ret = PTR_ERR(bitmap); + } + missed = copy_from_user(actual, (void __user *)dest, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, missed, 0UL); + memset(expected + size - sizeof(u64), 0xff, bytes); + expected[size - sizeof(u64) + bytes - 1] = 1; + KUNIT_EXPECT_EQ(test, memcmp(actual, expected, len), 0); + kunit_release_action(test, user4k_vma_mm_free, mm); + kunit_info(test, "IOVA bitmap relocated %luK user leaves checked", size / SZ_1K); + } +} + +static void test_iova_free(void *bitmap) +{ + iova_bitmap_free(bitmap); +} + +static void test_iova_kvfree(void *buffer) +{ + kvfree(buffer); +} + +static void user4k_iova_bitmap_window_test(struct kunit *test) +{ + struct mm_struct *mm; + unsigned long capacity = PAGE_SIZE / sizeof(struct user_page_fragment); + unsigned long bytes = (capacity + 3) * SZ_4K, len = bytes + 2 * SZ_4K; + unsigned long start = TEST_VA + SZ_4K - sizeof(u64), unit = SZ_4K; + unsigned long addr, missed; + unsigned char *actual; + struct iova_bitmap *bitmap; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bitmap requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + actual = kvzalloc(len, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, actual); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_iova_kvfree, actual), 0); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + bitmap = iova_bitmap_alloc(0, bytes * 8 * unit, unit, (u64 __user *)start); + ret = IS_ERR(bitmap) ? PTR_ERR(bitmap) : + iova_bitmap_for_each(bitmap, &unit, test_iova_bitmap_fill); + if (!IS_ERR(bitmap)) + iova_bitmap_free(bitmap); + missed = IS_ERR_VALUE(addr) ? len : copy_from_user(actual, (void __user *)addr, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(actual, 0, SZ_4K - sizeof(u64)), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(actual + SZ_4K - sizeof(u64), 0xff, bytes - 1), NULL); + KUNIT_EXPECT_EQ(test, actual[SZ_4K - sizeof(u64) + bytes - 1], (unsigned char)1); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(actual + SZ_4K - sizeof(u64) + bytes, + 0, SZ_4K + sizeof(u64)), NULL); + kunit_release_action(test, user4k_vma_mm_free, mm); + kunit_info(test, "IOVA bitmap crossed %lu-record window and revisited first byte", capacity); +} + +static void user4k_iova_bitmap_retained_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + unsigned int mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bitmap requires hardware opt-in"); + return; + } + for (mode = 0; mode < (PAGE_SHIFT > 14 ? 3U : 2U); mode++) { + struct mm_struct *mm = test_vma_mm_granule(test, shifts[mode]); + unsigned long size = 1UL << shifts[mode], len = 2 * size; + struct file *file = test_cache_file(test, 4 * PAGE_SIZE, false); + unsigned char *buffer = kunit_kzalloc(test, len, GFP_KERNEL); + unsigned long addr, replacement = 0, missed = 0; + struct iova_bitmap *bitmap; + struct folio *folio; + loff_t pos = size; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_NOT_NULL(test, buffer); + kthread_use_mm(mm); + addr = vm_mmap(file, TEST_VA, len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, size); + bitmap = iova_bitmap_alloc(0, len * 8 * SZ_4K, SZ_4K, + (u64 __user *)TEST_VA); + if (!IS_ERR(bitmap)) + iova_bitmap_set(bitmap, 3 * SZ_4K, SZ_4K); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_iova_free, bitmap), 0); + folio = filemap_get_folio(file->f_mapping, size >> PAGE_SHIFT); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_ASSERT_TRUE(test, folio_maybe_dma_pinned(folio)); + kthread_use_mm(mm); + ret = vm_munmap(TEST_VA, len); + if (!ret) { + replacement = vm_mmap(NULL, TEST_VA, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + if (!IS_ERR_VALUE(replacement)) { + memset(buffer, 0x5a, len); + missed = copy_to_user((void __user *)replacement, buffer, len); + } + } + /* The cached pins still refer to the old file, not the new VMA. */ + iova_bitmap_set(bitmap, 9 * SZ_4K, 2 * SZ_4K); + if (!ret && !IS_ERR_VALUE(replacement)) + missed |= copy_from_user(buffer, (void __user *)replacement, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, replacement, TEST_VA); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0x5a, len), NULL); + kunit_release_action(test, test_iova_free, bitmap); + KUNIT_EXPECT_FALSE(test, folio_maybe_dma_pinned(folio)); + KUNIT_EXPECT_TRUE(test, folio_test_dirty(folio)); + KUNIT_ASSERT_EQ(test, kernel_read(file, buffer, len, &pos), (ssize_t)len); + KUNIT_EXPECT_EQ(test, buffer[0], (unsigned char)8); + KUNIT_EXPECT_EQ(test, buffer[1], (unsigned char)6); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + 2, 0, len - 2), NULL); + kunit_release_action(test, user4k_vma_mm_free, mm); + kunit_info(test, "IOVA bitmap %luK shared backing retained across unmap/replacement", size / SZ_1K); + } +} + +static long test_mprotect(unsigned long start, unsigned long len, unsigned long prot); + +static void user4k_iova_bitmap_partial_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + unsigned int mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bitmap requires hardware opt-in"); + return; + } + for (mode = 0; mode < (PAGE_SHIFT > 14 ? 3U : 2U); mode++) { + struct mm_struct *mm = test_vma_mm_granule(test, shifts[mode]); + unsigned long size = 1UL << shifts[mode], len = 3 * size; + unsigned char *buffer = kunit_kzalloc(test, len, GFP_KERNEL); + unsigned long addr, refill, missed; + struct iova_bitmap *bitmap; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_NULL(test, buffer); + kthread_use_mm(mm); + addr = vm_mmap(NULL, TEST_VA, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + missed = IS_ERR_VALUE(addr) ? len : copy_to_user((void __user *)addr, buffer, len); + ret = vm_munmap(TEST_VA + size, size); + bitmap = iova_bitmap_alloc(0, len * 8 * SZ_4K, SZ_4K, (u64 __user *)TEST_VA); + if (!IS_ERR(bitmap)) + iova_bitmap_set(bitmap, 0, len * 8 * SZ_4K); + missed |= copy_from_user(buffer, (void __user *)TEST_VA, size); + missed |= copy_from_user(buffer + 2 * size, (void __user *)(TEST_VA + 2 * size), size); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, addr, TEST_VA); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_iova_free, bitmap), 0); + /* Preserve the existing void API: pin failure stops at the valid prefix. */ + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0xff, size), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + 2 * size, 0, size), NULL); + kthread_use_mm(mm); + refill = vm_mmap(NULL, TEST_VA + size, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + iova_bitmap_set(bitmap, 0, len * 8 * SZ_4K); + missed = copy_from_user(buffer, (void __user *)TEST_VA, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, refill, TEST_VA + size); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0xff, len), NULL); + kunit_release_action(test, test_iova_free, bitmap); + memset(buffer, 0, len); + kthread_use_mm(mm); + missed = copy_to_user((void __user *)TEST_VA, buffer, len); + ret = test_mprotect(TEST_VA, len, PROT_READ); + bitmap = iova_bitmap_alloc(0, len * 8 * SZ_4K, SZ_4K, (u64 __user *)TEST_VA); + if (!IS_ERR(bitmap)) { + iova_bitmap_set(bitmap, 0, len * 8 * SZ_4K); + iova_bitmap_free(bitmap); + } + missed |= copy_from_user(buffer, (void __user *)TEST_VA, len); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, missed, 0UL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bitmap); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0, len), NULL); + kunit_release_action(test, user4k_vma_mm_free, mm); + kunit_info(test, "IOVA bitmap %luK partial pin, retry and read-only checks passed", size / SZ_1K); + } +} + +static u64 test_table_descriptor(struct mm_struct *mm, unsigned long addr, bool pud) +{ + pgd_t *pgdp = pgd_offset(mm, addr); + p4d_t *p4dp = p4d_offset_mm(mm, pgdp, addr); + pud_t *pudp = pud_offset_mm(mm, p4dp, addr); + + return pud ? pud_val(*pudp) : pmd_val(*pmd_offset_mm(mm, pudp, addr)); +} + +static void user4k_mremap_move_test(struct kunit *test) +{ + static const struct { unsigned long old, new, len; unsigned int level; } cases[] = { + { TEST_VA + SZ_4K, 3 * SZ_2M + 2 * SZ_4K, 3 * SZ_4K, 0 }, + { SZ_2M - SZ_4K, SZ_1G - 2 * SZ_4K, 3 * SZ_4K, 0 }, + { SZ_1G - SZ_4K, BIT_ULL(39) - 2 * SZ_4K, 3 * SZ_4K, 0 }, + { 4 * SZ_2M, 8 * SZ_2M, SZ_2M, 1 }, + { 2UL * SZ_1G, 4UL * SZ_1G, SZ_1G, 2 }, + }; + unsigned int shape; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mremap fixture requires hardware opt-in"); + return; + } + for (shape = 0; shape < ARRAY_SIZE(cases); shape++) { + struct mm_struct *mm = test_vma_mm(test); + unsigned long old = cases[shape].old, new = cases[shape].new; + unsigned long len = cases[shape].len; + unsigned long offsets[] = { 0, SZ_4K, len - SZ_4K }; + unsigned long result, warm, moved, vm_before, commit_before, rss_before; + unsigned long flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE; + unsigned long old_left[3], new_left[3]; + phys_addr_t phys[3]; + u8 readback[3], value = 0x79, byte; + u64 table = 0; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, mm); + if (cases[shape].level) + flags |= MAP_NORESERVE; + kthread_use_mm(mm); + warm = vm_mmap(NULL, SZ_1M, SZ_4K, PROT_READ | PROT_WRITE, flags, 0); + result = vm_mmap(NULL, old, len, PROT_READ | PROT_WRITE, flags, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, warm, SZ_1M); + KUNIT_ASSERT_EQ(test, result, old); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, warm, &value, 1, FOLL_WRITE), 1); + for (i = 0; i < 3; i++) { + value = 0x80 + i; + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, old + offsets[i], &value, 1, FOLL_WRITE), 1); + phys[i] = pte_phys_mm(mm, test_fault_entry(test, mm, old + offsets[i])); + } + if (cases[shape].level) { + table = test_table_descriptor(mm, old, cases[shape].level == 2); + /* Warm-up consumed the first fragment; the moved table is packed. */ + KUNIT_EXPECT_NE(test, table & (3UL * SZ_4K), 0ULL); + } + vm_before = mm->total_vm; + commit_before = mm->committed_user_pages; + rss_before = get_mm_rss_sum(mm); + kthread_use_mm(mm); + pagefault_disable(); + for (i = 0; i < 3; i++) + old_left[i] = copy_from_user(&byte, (void __user *)(old + offsets[i]), 1); + pagefault_enable(); + moved = test_mremap(old, len, len, MREMAP_FIXED | MREMAP_MAYMOVE, new); + pagefault_disable(); + for (i = 0; i < 3; i++) { + new_left[i] = copy_from_user(&readback[i], (void __user *)(new + offsets[i]), 1); + old_left[i] += !copy_from_user(&byte, (void __user *)(old + offsets[i]), 1); + } + pagefault_enable(); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, moved, new); + KUNIT_EXPECT_EQ(test, mm->total_vm, vm_before); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, commit_before); + KUNIT_EXPECT_EQ(test, get_mm_rss_sum(mm), rss_before); + for (i = 0; i < 3; i++) { + KUNIT_EXPECT_EQ(test, old_left[i], 0UL); + KUNIT_EXPECT_EQ(test, new_left[i], 0UL); + KUNIT_EXPECT_EQ(test, readback[i], (u8)(0x80 + i)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, test_fault_entry(test, mm, new + offsets[i])), phys[i]); + } + if (cases[shape].level) + KUNIT_EXPECT_EQ(test, test_table_descriptor(mm, new, cases[shape].level == 2), table); + } +} + +static void user4k_stack_relocate_test(struct kunit *test) +{ + const unsigned long shifts[] = { SZ_4K, SZ_2M + SZ_4K, SZ_1G + 3 * SZ_4K }; + unsigned int shape, i; + + for (shape = 0; shape < ARRAY_SIZE(shifts); shape++) { + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vma; + struct vm_page_offset pos; + unsigned long top, old, new; + phys_addr_t phys[3]; + u8 bytes[3] = { 0x42, 0x73, 0xb5 }; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, create_init_stack_vma(mm, &vma, &top), 0); + old = vma->vm_end - 3 * SZ_4K; + new = old - shifts[shape]; + mmap_write_lock(mm); + ret = expand_stack_locked(vma, old); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->total_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 3UL); + for (i = 0; i < 3; i++) { + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, old + i * SZ_4K, + &bytes[i], 1, FOLL_WRITE), 1); + phys[i] = pte_phys_mm(mm, test_fault_entry(test, mm, old + i * SZ_4K)); + } + pos = vma_page_offset_at(vma, old); + mmap_write_lock(mm); + ret = relocate_vma_down(vma, shifts[shape]); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, vma->vm_start, new); + KUNIT_EXPECT_EQ(test, vma->vm_end, new + 3 * SZ_4K); + KUNIT_EXPECT_TRUE(test, vm_page_offset_equal(vma_page_offset_at(vma, new), pos)); + KUNIT_EXPECT_EQ(test, mm->total_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 3UL); + for (i = 0; i < 3; i++) { + u8 byte = 0; + + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, new + i * SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, bytes[i]); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, test_fault_entry(test, mm, new + i * SZ_4K)), phys[i]); + } + /* Grow after relocation: keep the file-like anon offset linear. */ + mmap_write_lock(mm); + vm_flags_clear(vma, VM_STACK_INCOMPLETE_SETUP); + ret = expand_stack_locked(vma, new - SZ_4K); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm->total_vm, 4UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 4UL); + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, new - SZ_4K, &bytes[0], 1, FOLL_WRITE), 1); + KUNIT_EXPECT_TRUE(test, vm_page_offset_equal(vma_page_offset_at(vma, new), pos)); + } +} + +static void user4k_mremap_cow_test(struct kunit *test) +{ + struct mm_struct *parent, *child; + unsigned long old = TEST_VA + SZ_4K, new = SZ_2M - 2 * SZ_4K, mapped; + unsigned long moved; + u8 value = 0x29, byte; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mremap fixture requires hardware opt-in"); + return; + } + parent = test_vma_mm(test); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, parent); + KUNIT_ASSERT_NOT_NULL(test, child); + kthread_use_mm(parent); + mapped = vm_mmap(NULL, old, 3 * SZ_4K, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(parent); + KUNIT_ASSERT_EQ(test, mapped, old); + KUNIT_ASSERT_EQ(test, access_remote_vm(parent, old + SZ_4K, &value, 1, FOLL_WRITE), 1); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + kthread_use_mm(child); + /* Move only the middle quarter, leaving both neighbors mapped. */ + moved = test_mremap(old + SZ_4K, SZ_4K, SZ_4K, + MREMAP_FIXED | MREMAP_MAYMOVE, new); + kthread_unuse_mm(child); + KUNIT_ASSERT_EQ(test, moved, new); + value = 0xa4; + KUNIT_EXPECT_EQ(test, access_remote_vm(child, new, &value, 1, FOLL_WRITE), 1); + KUNIT_EXPECT_EQ(test, access_remote_vm(parent, old + SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)0x29); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, new, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, value); + KUNIT_EXPECT_EQ(test, mm_pages_to_bytes(child, child->total_vm), 3UL * SZ_4K); + KUNIT_EXPECT_EQ(test, child->committed_user_pages, 3UL); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, old, &value, 1, 0), 1); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, old + 2 * SZ_4K, &value, 1, 0), 1); +} + +static void user4k_mremap_resize_test(struct kunit *test) +{ + struct mm_struct *mm; + unsigned long old = TEST_VA + SZ_4K, new = TEST_VA + 8 * PAGE_SIZE + 2 * SZ_4K; + unsigned long res[7]; + u8 value = 0x98, byte = 0; + int unmap[2]; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mremap fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + res[0] = vm_mmap(NULL, old, SZ_4K, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + res[1] = test_mremap(old, SZ_4K, 3 * SZ_4K - 1, 0, 0); + res[2] = test_mremap(old + 1, SZ_4K, SZ_4K, 0, 0); + res[3] = test_mremap(old, 3 * SZ_4K, 3 * SZ_4K, + MREMAP_FIXED | MREMAP_MAYMOVE, new + 1); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, res[0], old); + KUNIT_ASSERT_EQ(test, res[1], old); + KUNIT_EXPECT_EQ(test, res[2], (unsigned long)-EINVAL); + KUNIT_EXPECT_EQ(test, res[3], (unsigned long)-EINVAL); + KUNIT_EXPECT_EQ(test, mm->total_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 3UL); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, old + SZ_4K, &value, 1, FOLL_WRITE), 1); + kthread_use_mm(mm); + res[4] = test_mremap(old, 3 * SZ_4K, 2 * SZ_4K, 0, 0); + res[5] = test_mremap(old, 2 * SZ_4K, 2 * SZ_4K, + MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP, new); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, res[4], old); + KUNIT_ASSERT_EQ(test, res[5], new); + KUNIT_EXPECT_EQ(test, mm->total_vm, 4UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 4UL); + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, new + SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, value); + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, old + SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)0); + kthread_use_mm(mm); + unmap[0] = vm_munmap(old, 2 * SZ_4K); + unmap[1] = vm_munmap(new, 2 * SZ_4K); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, unmap[0], 0); + KUNIT_EXPECT_EQ(test, unmap[1], 0); + KUNIT_EXPECT_EQ(test, mm->total_vm, 0UL); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, 0UL); +} + +static int test_reject_mremap(struct vm_area_struct *vma) +{ + return -EIO; +} + +static void user4k_cold_batch_test(struct kunit *test) +{ + struct mm_struct *mm, *child; + struct vm_area_struct *vma; + unsigned long start = TEST_VA, mapped; + spinlock_t *ptl; + pte_t *ptep, entry; + u8 byte = 0x68; + unsigned int i; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm cold fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_NULL(test, child); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, start, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, start); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, start, &byte, 1, FOLL_WRITE), 1); + /* Only the last quarter is young: batching must aggregate that bit. */ + mmap_read_lock(mm); + vma = vma_lookup(mm, start); + for (i = 0; i < 3; i++) { + unsigned long addr = start + i * SZ_4K; + + ptep = test_mm_lookup(mm, addr, &ptl); + if (!ptep) { + mmap_read_unlock(mm); + KUNIT_FAIL(test, "missing cold test PTE"); + return; + } + entry = ptep_clear_flush(vma, addr, ptep); + set_pte_at(mm, addr, ptep, pte_mkold(entry)); + pte_unmap_unlock(ptep, ptl); + } + mmap_read_unlock(mm); + kthread_use_mm(mm); + ret = do_madvise(mm, start + 3 * SZ_4K, SZ_4K, MADV_COLD); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, pte_young(test_fault_entry(test, mm, start + 3 * SZ_4K))); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + kthread_use_mm(mm); + ret = do_madvise(mm, start, PAGE_SIZE, MADV_COLD); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, pte_young(test_fault_entry(test, mm, start + 3 * SZ_4K))); + kunit_release_action(test, user4k_vma_mm_free, child); + kthread_use_mm(mm); + ret = do_madvise(mm, start, PAGE_SIZE, MADV_COLD); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret, 0); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_FALSE(test, pte_young(test_fault_entry(test, mm, start + i * SZ_4K))); + byte = 0; + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, start, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)0x68); +} + +static void user4k_prefault_dirty_test(struct kunit *test) +{ + struct mm_struct *mm; + struct folio *folio; + unsigned long start = TEST_VA + SZ_4K, mapped; + spinlock_t *ptl; + pte_t *ptep, entry; + u8 byte = 0x7a; + int locked = 1; + long ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm prefault fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, start, SZ_4K, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, start); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, start, &byte, 1, FOLL_WRITE), 1); + mmap_read_lock(mm); + ptep = test_mm_lookup(mm, start, &ptl); + if (!ptep) { + mmap_read_unlock(mm); + KUNIT_FAIL(test, "missing prefault test PTE"); + return; + } + entry = ptep_clear_flush(vma_lookup(mm, start), start, ptep); + folio = page_folio(pte_page(entry)); + folio_get(folio); + set_pte_at(mm, start, ptep, pte_mkclean(entry)); + folio_clear_dirty(folio); + pte_unmap_unlock(ptep, ptl); + mmap_read_unlock(mm); + kthread_use_mm(mm); + mmap_read_lock(mm); + ret = faultin_page_range(mm, start, start + SZ_4K, true, &locked); + if (locked) + mmap_read_unlock(mm); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret, 1L); + KUNIT_EXPECT_TRUE(test, folio_test_dirty(folio)); + folio_put(folio); +} + +struct test_fsync_record { + loff_t start, end; + unsigned int calls; + int datasync; +}; + +static int test_fsync_range(struct file *file, loff_t start, loff_t end, int datasync) +{ + struct test_fsync_record *record = file->private_data; + + *record = (struct test_fsync_record) { start, end, record->calls + 1, datasync }; + return 0; +} + +asmlinkage long __arm64_sys_msync(const struct pt_regs *regs); + +static void user4k_msync_offsets_test(struct kunit *test) +{ + struct file *file; + struct vm_area_struct *vma; + struct file_operations ops; + const struct file_operations *original_ops; + void *original_private; + struct test_fsync_record records[3] = {}, record = {}; + struct pt_regs regs = { .regs = { TEST_VA + SZ_4K, 1, MS_SYNC } }; + long result[3]; + unsigned int i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm sync fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, 3 * PAGE_SIZE, false); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + 5 * SZ_4K, 3 * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, vma); + original_ops = file->f_op; + original_private = file->private_data; + ops = *original_ops; + ops.fsync = test_fsync_range; + file->f_op = &ops; + file->private_data = &record; + kthread_use_mm(vma->vm_mm); + for (i = 0; i < ARRAY_SIZE(result); i++) { + regs.regs[0] = vma->vm_start + i * SZ_4K; + result[i] = __arm64_sys_msync(®s); + records[i] = record; + } + kthread_unuse_mm(vma->vm_mm); + file->f_op = original_ops; + file->private_data = original_private; + for (i = 0; i < ARRAY_SIZE(result); i++) { + KUNIT_EXPECT_EQ(test, result[i], 0L); + KUNIT_EXPECT_EQ(test, records[i].calls, i + 1); + KUNIT_EXPECT_EQ(test, records[i].start, (loff_t)(3 + i) * SZ_4K); + KUNIT_EXPECT_EQ(test, records[i].end, (loff_t)(4 + i) * SZ_4K - 1); + KUNIT_EXPECT_EQ(test, records[i].datasync, 1); + } +} + +static void user4k_mremap_file_test(struct kunit *test) +{ + struct mm_struct *mm; + struct vm_area_struct *vma; + struct file *file; + struct vm_operations_struct rejecting_ops; + const struct vm_operations_struct *original_ops; + unsigned long old = TEST_VA + SZ_4K, new = SZ_1G - SZ_4K; + unsigned long mapped, moved; + u8 bytes[3]; + unsigned int i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mremap fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, 2 * PAGE_SIZE, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + mapped = vm_mmap(file, old, 3 * SZ_4K, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, 3 * SZ_4K); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, old); + for (i = 0; i < 3; i++) + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, old + i * SZ_4K, &bytes[i], 1, 0), 1); + vma = test_vma_lookup(mm, old); + KUNIT_ASSERT_NOT_NULL(test, vma); + original_ops = vma->vm_ops; + rejecting_ops = *original_ops; + rejecting_ops.mremap = test_reject_mremap; + mmap_write_lock(mm); + vma->vm_ops = &rejecting_ops; + mmap_write_unlock(mm); + kthread_use_mm(mm); + moved = test_mremap(old, 3 * SZ_4K, 3 * SZ_4K, + MREMAP_FIXED | MREMAP_MAYMOVE, new); + kthread_unuse_mm(mm); + /* Restore stack-owned callbacks before any assertion can leave this scope. */ + mmap_write_lock(mm); + { + VMA_ITERATOR(vmi, mm, 0); + struct vm_area_struct *part; + + for_each_vma(vmi, part) + if (part->vm_ops == &rejecting_ops) + part->vm_ops = original_ops; + } + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, moved, (unsigned long)-EIO); + for (i = 0; i < 3; i++) { + u8 byte = 0; + + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, old + i * SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, bytes[i]); + } + KUNIT_EXPECT_EQ(test, mm->map_count, 1); + KUNIT_EXPECT_EQ(test, mm->total_vm, 3UL); + kthread_use_mm(mm); + moved = test_mremap(old, 3 * SZ_4K, 3 * SZ_4K, + MREMAP_FIXED | MREMAP_MAYMOVE, new); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, moved, new); + vma = test_vma_lookup(mm, new); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma_subpage_offset(vma), 3U * SZ_4K); + for (i = 0; i < 3; i++) { + u8 byte = 0; + + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, new + i * SZ_4K, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, bytes[i]); + } +} + +static void user4k_exec_arguments_test(struct kunit *test) +{ + unsigned int native; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm stack setup requires hardware opt-in"); + return; + } + for (native = 0; native < 2; native++) { + struct linux_binprm *bprm = kunit_kzalloc(test, sizeof(*bprm), GFP_KERNEL); + struct mm_struct *mm = native ? mm_alloc() : test_vma_mm(test); + unsigned long size, mask, initial, copied, new_top; + unsigned long a_len = 2 * PAGE_SIZE + SZ_4K + 31, b_len = SZ_4K + 43; + char *a, *b, *buf, *too_long; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, bprm); + KUNIT_ASSERT_NOT_NULL(test, mm); + if (native) + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + size = mm_page_size(mm); + mask = mm_page_mask(mm); + a_len = min(a_len, 16 * size + 31); + bprm->mm = mm; + KUNIT_ASSERT_EQ(test, create_init_stack_vma(mm, &bprm->vma, &bprm->p), 0); + initial = bprm->p; + bprm->argmin = initial - SZ_2M; + bprm->rlim_stack.rlim_cur = SZ_8M; + bprm->rlim_stack.rlim_max = RLIM_INFINITY; + a = kunit_kmalloc(test, a_len + 1, GFP_KERNEL); + b = kunit_kmalloc(test, b_len + 1, GFP_KERNEL); + buf = kunit_kmalloc(test, a_len + b_len + 7, GFP_KERNEL); + too_long = kunit_kmalloc(test, 32 * size + 1, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + KUNIT_ASSERT_NOT_NULL(test, buf); + KUNIT_ASSERT_NOT_NULL(test, too_long); + memset(a, 'A', a_len); + a[a_len] = 0; + memset(b, 'b', b_len); + b[b_len] = 0; + memset(too_long, 'x', 32 * size); + too_long[32 * size] = 0; + KUNIT_EXPECT_EQ(test, copy_string_kernel(too_long, bprm), -E2BIG); + KUNIT_EXPECT_EQ(test, bprm->p, initial); + KUNIT_ASSERT_EQ(test, copy_string_kernel("tail", bprm), 0); + bprm->exec = bprm->p; + KUNIT_ASSERT_EQ(test, copy_string_kernel(a, bprm), 0); + KUNIT_ASSERT_EQ(test, copy_string_kernel(b, bprm), 0); + bprm->argc = 3; + copied = initial - bprm->p; + KUNIT_EXPECT_EQ(test, copied, a_len + b_len + 7); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, bprm->p, buf, copied, 0), (int)copied); + KUNIT_EXPECT_MEMEQ(test, buf, b, b_len + 1); + KUNIT_EXPECT_MEMEQ(test, buf + b_len + 1, a, a_len + 1); + KUNIT_EXPECT_STREQ(test, buf + b_len + a_len + 2, "tail"); + new_top = (bprm->vma->vm_end - SZ_2M - size) & mask; + kthread_use_mm(mm); + ret = setup_arg_pages(bprm, new_top, EXSTACK_DISABLE_X); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + /* Subtract < PAGE_SIZE, then round down to 16 bytes and up to a leaf. */ + KUNIT_EXPECT_LE(test, bprm->vma->vm_end, new_top); + KUNIT_EXPECT_LE(test, new_top - bprm->vma->vm_end, PAGE_SIZE); + KUNIT_EXPECT_EQ(test, bprm->vma->vm_end & ~mask, 0UL); + KUNIT_EXPECT_EQ(test, mm->start_stack, bprm->p); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, bprm->p, buf, copied, 0), (int)copied); + KUNIT_EXPECT_MEMEQ(test, buf, b, b_len + 1); + KUNIT_EXPECT_MEMEQ(test, buf + b_len + 1, a, a_len + 1); + KUNIT_EXPECT_STREQ(test, buf + b_len + a_len + 2, "tail"); + /* /proc/PID/cmdline and environ read the stack with FOLL_ANON. */ + memset(buf, 0, copied); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, bprm->p, buf, copied, FOLL_ANON), + (int)copied); + KUNIT_EXPECT_MEMEQ(test, buf, b, b_len + 1); + KUNIT_EXPECT_STREQ(test, buf + b_len + a_len + 2, "tail"); + /* Interpreter argument removal scans several user granules. */ + initial = bprm->p; + KUNIT_ASSERT_EQ(test, remove_arg_zero(bprm), 0); + KUNIT_EXPECT_EQ(test, bprm->p, initial + b_len + 1); + KUNIT_ASSERT_EQ(test, remove_arg_zero(bprm), 0); + KUNIT_EXPECT_EQ(test, bprm->p, initial + b_len + a_len + 2); + KUNIT_ASSERT_EQ(test, remove_arg_zero(bprm), 0); + KUNIT_EXPECT_EQ(test, bprm->p, initial + copied); + KUNIT_EXPECT_EQ(test, bprm->argc, 0); + KUNIT_EXPECT_EQ(test, bprm->vma_pages, 0UL); + } +} + +static struct vm_area_struct *test_special_vma(struct mm_struct *mm, + const struct vm_special_mapping *sm, unsigned long start, unsigned long len, + vm_flags_t flags) +{ + struct vm_area_struct *vma; + + mmap_write_lock(mm); + vma = _install_special_mapping(mm, start, len, flags, sm); + mmap_write_unlock(mm); + return vma; +} + +static void user4k_special_mapping_test(struct kunit *test) +{ + struct vm_special_mapping *sm = kunit_kzalloc(test, sizeof(*sm), GFP_KERNEL); + struct page **pages = kunit_kcalloc(test, 3, sizeof(*pages), GFP_KERNEL); + void *data[2] = { test_alloc_page(test), test_alloc_page(test) }; + struct mm_struct *small = test_vma_mm(test), *cold = test_vma_mm(test); + struct mm_struct *native = mm_alloc(); + struct vm_area_struct *vma[3]; + unsigned long start = SZ_2M - SZ_4K; + unsigned int i, side; + u8 byte; + + KUNIT_ASSERT_NOT_NULL(test, sm); + KUNIT_ASSERT_NOT_NULL(test, pages); + KUNIT_ASSERT_NOT_NULL(test, data[0]); + KUNIT_ASSERT_NOT_NULL(test, data[1]); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, cold); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, native), 0); + /* Reverse the supplied native pages: folio indices cannot identify them. */ + pages[0] = virt_to_page(data[1]); + pages[1] = virt_to_page(data[0]); + sm->pages = pages; + sm->name = "[user4k-special-test]"; + for (i = 0; i < 2 * TEST_SLOTS; i++) + memset(page_address(pages[i / TEST_SLOTS]) + (i % TEST_SLOTS) * SZ_4K, 0x41 + i, SZ_4K); + vma[0] = test_special_vma(small, sm, start, 2 * PAGE_SIZE, VM_READ | VM_MAYREAD | VM_MAYWRITE); + vma[1] = test_special_vma(cold, sm, start, 2 * PAGE_SIZE, VM_READ | VM_MAYREAD | VM_MAYWRITE); + vma[2] = test_special_vma(native, sm, TEST_VA, 2 * PAGE_SIZE, VM_READ | VM_MAYREAD | VM_MAYWRITE); + for (i = 0; i < 3; i++) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma[i]); + for (i = 0; i < 2 * TEST_SLOTS; i++) { + unsigned long addr = start + i * SZ_4K; + + KUNIT_ASSERT_EQ(test, access_remote_vm(small, addr, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)(0x41 + i)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small, test_fault_entry(test, small, addr)), + page_to_phys(pages[i / TEST_SLOTS]) + (i % TEST_SLOTS) * SZ_4K); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(small, MM_FILEPAGES), 2UL * TEST_SLOTS); + /* Both present-PTE and cold-PTE forced writes copy only their quarter. */ + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? cold : small; + + byte = 0xe1 + side; + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, start + 2 * SZ_4K + 7, + &byte, 1, FOLL_WRITE | FOLL_FORCE), 1); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 1UL); + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, start + 2 * SZ_4K + 7, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)(0xe1 + side)); + KUNIT_EXPECT_EQ(test, access_remote_vm(native, TEST_VA + 2 * SZ_4K + 7, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)0x43); + } + { + struct user_page_fragment fragment; + long ret = pin_user_fragments_remote(small, start + 5 * SZ_4K + 13, + 7, 0, &fragment, 1); + + KUNIT_ASSERT_EQ(test, ret, 1L); + KUNIT_EXPECT_NOT_NULL(test, fragment.subpage); + KUNIT_EXPECT_EQ(test, fragment.offset, (5 * SZ_4K + 13UL) & ~PAGE_MASK); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(fragment.folio) + fragment.offset, + 0x46, fragment.length), NULL); + release_user_fragments(&fragment, 1, false); + } +} + +static vm_fault_t test_callback_fault(struct vm_fault *vmf) +{ + struct page **pages = vmf->vma->vm_private_data; + + if (vmf->pgoff >= 2) + return VM_FAULT_SIGBUS; + + vmf->page = pages[vmf->pgoff]; + get_page(vmf->page); + return 0; +} + +static const struct vm_operations_struct test_callback_vm_ops = { + .fault = test_callback_fault, +}; + +static void test_callback_mapping(struct kunit *test, unsigned int shift) +{ + void *data[2] = { test_alloc_page(test), test_alloc_page(test) }; + struct page **pages = kunit_kcalloc(test, 2, sizeof(*pages), GFP_KERNEL); + struct file *file = test_cache_file(test, 0, false); + struct vm_area_struct *vma; + unsigned long size = 1UL << shift; + unsigned long offset = shift == PAGE_SHIFT ? 0 : size; + unsigned long len = 2 * PAGE_SIZE - offset; + unsigned long addr; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, data[0]); + KUNIT_ASSERT_NOT_NULL(test, data[1]); + KUNIT_ASSERT_NOT_NULL(test, pages); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + pages[0] = virt_to_page(data[1]); + pages[1] = virt_to_page(data[0]); + for (i = 0; i < 2 * PAGE_SIZE / size; i++) + memset(page_address(pages[(i * size) >> PAGE_SHIFT]) + + offset_in_page(i * size), 0x51 + i, size); + + /* The file identifies the VMA; the callback supplies independent pages. */ + vma = test_file_vma_granule(test, file, shift, false, TEST_VA, len, offset); + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_write_lock(vma->vm_mm); + vma->vm_ops = &test_callback_vm_ops; + vma->vm_private_data = pages; + mmap_write_unlock(vma->vm_mm); + for (addr = vma->vm_start; addr < vma->vm_end; addr += size) { + unsigned long source = offset + addr - vma->vm_start; + u8 byte; + + KUNIT_ASSERT_EQ(test, test_file_fault(vma, addr, false) & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, + test_fault_entry(test, vma->vm_mm, addr)), + page_to_phys(pages[source >> PAGE_SHIFT]) + offset_in_page(source)); + KUNIT_ASSERT_EQ(test, access_remote_vm(vma->vm_mm, addr + 7, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)(0x51 + source / size)); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_FILEPAGES), len / size); +} + +static void native_callback_mapping_test(struct kunit *test) +{ + test_callback_mapping(test, PAGE_SHIFT); +} + +static void user4k_callback_mapping_test(struct kunit *test) +{ + test_callback_mapping(test, 12); +} + +static void user16k_callback_mapping_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative requires a larger native granule"); + return; + } + test_callback_mapping(test, 14); +} + +static void user4k_insert_range_test(struct kunit *test) +{ + unsigned int native; + + for (native = 0; native < 2; native++) { + void *data[2] = { test_alloc_page(test), test_alloc_page(test) }; + struct page *pages[2]; + struct mm_struct *mm = native ? mm_alloc() : test_vma_mm(test); + struct vm_area_struct *vma; + unsigned long size, start, offset, length, num; + unsigned int i; + int ret, invalid[4]; + + KUNIT_ASSERT_NOT_NULL(test, data[0]); + KUNIT_ASSERT_NOT_NULL(test, data[1]); + KUNIT_ASSERT_NOT_NULL(test, mm); + if (native) + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + pages[0] = virt_to_page(data[1]); + pages[1] = virt_to_page(data[0]); + size = mm_page_size(mm); + start = pmd_size_mm(mm) - size; + offset = native ? 0 : 3 * SZ_4K; + length = 2 * PAGE_SIZE - offset; + vma = test_vma_add(mm, start, length, (struct vm_page_offset) {}, + VM_READ | VM_MAYREAD | VM_MIXEDMAP | VM_IO); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + mmap_write_lock(mm); + invalid[0] = vm_insert_pages_range(vma, start + 1, pages, 2, offset, length); + invalid[1] = vm_insert_pages_range(vma, start, pages, 2, offset + 1, length); + invalid[2] = vm_insert_pages_range(vma, start, pages, 2, offset, length + size); + invalid[3] = vm_insert_pages_range(vma, start, pages, ULONG_MAX, 0, length); + ret = vm_insert_pages_range(vma, start, pages, 2, offset, length); + mmap_write_unlock(mm); + for (i = 0; i < ARRAY_SIZE(invalid); i++) + KUNIT_EXPECT_EQ(test, invalid[i], -EINVAL); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = 0; i < length / size; i++) { + unsigned long source = offset + i * size; + + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, + test_fault_entry(test, mm, start + i * size)), + page_to_phys(pages[source >> PAGE_SHIFT]) + offset_in_page(source)); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_FILEPAGES), length / size); + /* The original native-page-count API also expands all user leaves. */ + start += 4 * PAGE_SIZE; + vma = test_vma_add(mm, start, 2 * PAGE_SIZE, (struct vm_page_offset) {}, + VM_READ | VM_MAYREAD | VM_MIXEDMAP | VM_IO); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + num = 2; + mmap_write_lock(mm); + ret = vm_insert_pages(vma, start, pages, &num); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, num, 0UL); + for (i = 0; i < 2 * PAGE_SIZE / size; i++) + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, + test_fault_entry(test, mm, start + i * size)), + page_to_phys(pages[(i * size) >> PAGE_SHIFT]) + offset_in_page(i * size)); + num = 2; + mmap_write_lock(mm); + ret = vm_insert_pages(vma, start, pages, &num); + mmap_write_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, -EBUSY); + KUNIT_EXPECT_EQ(test, num, 2UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_FILEPAGES), + (length + 2 * PAGE_SIZE) / size); + start += 4 * PAGE_SIZE; + offset = native ? PAGE_SIZE : 3 * SZ_4K; + length = 2 * PAGE_SIZE - offset; + vma = test_vma_add(mm, start, length, + (struct vm_page_offset) { .index = offset >> PAGE_SHIFT, + .offset = offset_in_page(offset) }, + VM_READ | VM_MAYREAD | VM_MIXEDMAP | VM_IO); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + mmap_write_lock(mm); + ret = vm_map_pages(vma, pages, 2); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = 0; i < length / size; i++) { + unsigned long source = offset + i * size; + + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, + test_fault_entry(test, mm, start + i * size)), + page_to_phys(pages[source >> PAGE_SHIFT]) + offset_in_page(source)); + } + kunit_release_action(test, user4k_vma_mm_free, mm); + KUNIT_EXPECT_EQ(test, page_ref_count(pages[0]), 1); + KUNIT_EXPECT_EQ(test, page_ref_count(pages[1]), 1); + } +} + +static void user4k_remap_pfn_range_test(struct kunit *test) +{ + struct folio *folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 1); + + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + for (unsigned int shift = 12; shift <= PAGE_SHIFT; shift += 2) { + struct mm_struct *mm = test_vma_mm_granule(test, shift); + bool native = shift == PAGE_SHIFT; + unsigned long size, boundaries[4]; + + KUNIT_ASSERT_NOT_NULL(test, mm); + size = mm_page_size(mm); + boundaries[0] = pmd_size_mm(mm); + boundaries[1] = pud_size_mm(mm); + boundaries[2] = pgd_size_mm(mm); + boundaries[3] = 2 * pmd_size_mm(mm); + for (unsigned int shape = 0; shape < ARRAY_SIZE(boundaries); shape++) { + unsigned long start = boundaries[shape] - size; + unsigned long length = PAGE_SIZE + size; + unsigned int physical_offset = native || !shape ? 0 : size; + struct vm_area_struct *vma; + int ret; + + if (shape && boundaries[shape] == boundaries[shape - 1]) + continue; + vma = test_vma_add(mm, start, length, (struct vm_page_offset) {}, + VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | + VM_SHARED | VM_MAYSHARE); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + mmap_write_lock(mm); + if (shape >= 2) { + phys_addr_t base = PFN_PHYS(folio_pfn(folio)) + physical_offset; + unsigned long available = folio_size(folio) - physical_offset; + + vma_set_page_offset(vma, (struct vm_page_offset) { + .offset = physical_offset, + }); + if (shape == 2) { + KUNIT_EXPECT_EQ(test, + vm_iomap_memory(vma, base, PAGE_SIZE), -EINVAL); + ret = vm_iomap_memory(vma, base, available); + } else { + struct vm_area_desc desc; + + compat_set_desc_from_vma(&desc, NULL, vma); + mmap_action_simple_ioremap(&desc, base, PAGE_SIZE); + KUNIT_EXPECT_EQ(test, + __compat_vma_mmap(&desc, vma), -EINVAL); + mmap_action_simple_ioremap(&desc, base, available); + ret = __compat_vma_mmap(&desc, vma); + } + physical_offset *= 2; + } else if (physical_offset) + ret = remap_pfn_range_offset(vma, start, folio_pfn(folio), + physical_offset, length, vma->vm_page_prot); + else + ret = remap_pfn_range(vma, start, folio_pfn(folio), length, + vma->vm_page_prot); + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + for (unsigned long offset = 0; offset < length; offset += size) { + pte_t pte = test_fault_entry(test, mm, start + offset); + + struct follow_pfnmap_args args = { + .vma = vma, .address = start + offset + 31, + }; + phys_addr_t expected = PFN_PHYS(folio_pfn(folio)) + + physical_offset + offset + 31; + int ret; + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_TRUE(test, pte_special(pte)); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), + PFN_PHYS(folio_pfn(folio)) + + physical_offset + offset); + + mmap_read_lock(mm); + ret = follow_pfnmap_start(&args); + if (!ret) + follow_pfnmap_end(&args); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, native ? 0 : -EOPNOTSUPP); + args.allow_subpage = true; + mmap_read_lock(mm); + ret = follow_pfnmap_start(&args); + if (!ret) { + KUNIT_EXPECT_EQ(test, args.pfn, (unsigned long)PHYS_PFN(expected)); + KUNIT_EXPECT_EQ(test, args.offset, (unsigned int)offset_in_page(expected)); + KUNIT_EXPECT_EQ(test, args.addr_mask, mm_page_mask(mm)); + KUNIT_EXPECT_TRUE(test, args.special); + follow_pfnmap_end(&args); + } + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + } + KUNIT_EXPECT_TRUE(test, pte_none(test_fault_entry(test, mm, start - size))); + KUNIT_EXPECT_TRUE(test, pte_none(test_fault_entry(test, mm, start + length))); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_FILEPAGES), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0UL); + kunit_release_action(test, user4k_vma_mm_free, mm); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + } +} + +#ifdef CONFIG_HAVE_IOREMAP_PROT +static void test_iounmap(void *arg) +{ + iounmap((void __iomem *)arg); +} + +static const struct vm_operations_struct test_iomem_ops = { + .access = generic_access_phys, +}; +#endif + +static void user4k_iomem_access_test(struct kunit *test) +{ +#if defined(CONFIG_HAVE_IOREMAP_PROT) && defined(CONFIG_OF) + struct device_node *node; + struct resource resource; + struct mm_struct *mm; + struct vm_area_struct *vma; + void __iomem *uart; + unsigned long start = SZ_2M + PAGE_SIZE - SZ_4K; + u8 reference[16], buffer[32]; + vm_fault_t fault; + int ret; + + if (!of_machine_is_compatible("linux,dummy-virt")) { + kunit_skip(test, "requires disposable QEMU virt PL011 identification registers"); + return; + } + node = of_find_compatible_node(NULL, NULL, "arm,pl011"); + KUNIT_ASSERT_NOT_NULL(test, node); + ret = of_address_to_resource(node, 0, &resource); + uart = ret ? NULL : of_iomap(node, 0); + of_node_put(node); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_NOT_NULL(test, uart); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_iounmap, + (void *)uart), 0); + KUNIT_ASSERT_GE(test, resource_size(&resource), (resource_size_t)SZ_4K); + memcpy_fromio(reference, uart + 0xff0, sizeof(reference)); + KUNIT_ASSERT_NOT_NULL(test, memchr_inv(reference, 0, sizeof(reference))); + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + vma = test_vma_add(mm, start, 2 * SZ_4K, (struct vm_page_offset) {}, + VM_READ | VM_MAYREAD | VM_SHARED | VM_MAYSHARE | VM_IO | VM_PFNMAP); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_ops = &test_iomem_ops; + vma->vm_page_prot = pgprot_device(vm_get_page_prot(vma->vm_flags)); + mmap_read_lock(mm); + fault = vmf_insert_pfn_prot_mkwrite_offset(vma, start, + PHYS_PFN(resource.start), offset_in_page(resource.start), + vma->vm_page_prot, false); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)VM_FAULT_NOPAGE); + memset(buffer, 0x6d, sizeof(buffer)); + mmap_read_lock(mm); + ret = generic_access_phys(vma, start + 0xff0, buffer, sizeof(buffer), 0); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, 16); + KUNIT_EXPECT_MEMEQ(test, buffer, reference, sizeof(reference)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + 16, 0x6d, 16), NULL); + memset(buffer, 0x5a, sizeof(buffer)); + ret = access_remote_vm(mm, start + 0xff0, buffer, sizeof(buffer), 0); + KUNIT_EXPECT_EQ(test, ret, 16); + KUNIT_EXPECT_MEMEQ(test, buffer, reference, sizeof(reference)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer + 16, 0x5a, 16), NULL); + /* Writes must fail before touching the read-only device mapping. */ + mmap_read_lock(mm); + ret = generic_access_phys(vma, start + 0xff0, buffer, 1, FOLL_WRITE); + mmap_read_unlock(mm); + KUNIT_EXPECT_LT(test, ret, 0); +#else + kunit_skip(test, "requires OF and iomem access support"); +#endif +} + +static void user4k_pfn_offset_test(struct kunit *test) +{ + void *data = test_alloc_page(test); + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vma; + unsigned long start = SZ_2M - SZ_4K; + unsigned long pfn; + unsigned int i; + vm_fault_t result[4], invalid[3], write; + + KUNIT_ASSERT_NOT_NULL(test, data); + KUNIT_ASSERT_NOT_NULL(test, mm); + pfn = page_to_pfn(virt_to_page(data)); + vma = test_vma_add(mm, start, PAGE_SIZE, (struct vm_page_offset) {}, + VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | VM_SHARED | VM_MAYSHARE | + VM_PFNMAP | VM_IO); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + /* Model a write-notify VMA: initial PTEs are read-only. */ + vma->vm_page_prot = vm_get_page_prot(VM_READ | VM_SHARED); + mmap_read_lock(mm); + for (i = 0; i < 4; i++) + result[i] = vmf_insert_pfn_prot_mkwrite_offset(vma, start + i * SZ_4K, + pfn, (3 - i) * SZ_4K, vma->vm_page_prot, false); + invalid[0] = vmf_insert_pfn_prot_mkwrite_offset(vma, start + 1, pfn, + 0, vma->vm_page_prot, false); + invalid[1] = vmf_insert_pfn_prot_mkwrite_offset(vma, start, pfn, + 1, vma->vm_page_prot, false); + invalid[2] = vmf_insert_pfn_prot_mkwrite_offset(vma, start, pfn, + PAGE_SIZE, vma->vm_page_prot, false); + write = vmf_insert_pfn_prot_mkwrite_offset(vma, start + SZ_4K, pfn, + 2 * SZ_4K, vma->vm_page_prot, true); + mmap_read_unlock(mm); + for (i = 0; i < 4; i++) { + pte_t entry = test_fault_entry(test, mm, start + i * SZ_4K); + + KUNIT_EXPECT_EQ(test, result[i], (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, entry), PFN_PHYS(pfn) + (3 - i) * SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_special(entry)); + KUNIT_EXPECT_EQ(test, !!pte_write(entry), i == 1); + } + for (i = 0; i < ARRAY_SIZE(invalid); i++) + KUNIT_EXPECT_EQ(test, invalid[i], (vm_fault_t)VM_FAULT_SIGBUS); + KUNIT_EXPECT_EQ(test, write, (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_FILEPAGES), 0UL); + KUNIT_EXPECT_EQ(test, page_ref_count(virt_to_page(data)), 1); + kunit_release_action(test, user4k_vma_mm_free, mm); + KUNIT_EXPECT_EQ(test, page_ref_count(virt_to_page(data)), 1); +} + +static void user16k_insert_shared_test(struct kunit *test) +{ + struct mm_struct *middle, *small, *native; + struct vm_area_struct *vmas[3]; + struct mm_subpage_mapcounts counts; + struct page *page; + void *data; + vm_fault_t result; + unsigned int i; + vm_flags_t flags = VM_READ | VM_MAYREAD | VM_MIXEDMAP | VM_IO; + + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "mixed16K file insertion requires larger native backing"); + return; + } + data = test_alloc_page(test); + KUNIT_ASSERT_NOT_NULL(test, data); + page = virt_to_page(data); + middle = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, middle); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, middle), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(middle, 14), 0); + small = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, small); + native = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, native), 0); + vmas[0] = test_vma_add(middle, TEST_VA, PAGE_SIZE, (struct vm_page_offset) {}, flags); + vmas[1] = test_vma_add(small, TEST_VA + SZ_4K, SZ_4K, (struct vm_page_offset) {}, flags); + vmas[2] = test_vma_add(native, TEST_VA, PAGE_SIZE, (struct vm_page_offset) {}, flags); + for (i = 0; i < 3; i++) { + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vmas[i]); + vmas[i]->vm_page_prot = vm_get_page_prot(vmas[i]->vm_flags); + } + for (i = 0; i < PAGE_SIZE / SZ_16K; i++) { + unsigned long address = TEST_VA + i * SZ_16K; + + mmap_read_lock(middle); + result = vmf_insert_page_offset(vmas[0], address, page, i * SZ_16K); + mmap_read_unlock(middle); + KUNIT_ASSERT_EQ(test, result, (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, pte_phys_mm(middle, test_fault_entry(test, middle, address)), + page_to_phys(page) + i * SZ_16K); + } + mmap_read_lock(small); + result = vmf_insert_page_offset(vmas[1], vmas[1]->vm_start, page, PAGE_SIZE - SZ_4K); + mmap_read_unlock(small); + KUNIT_ASSERT_EQ(test, result, (vm_fault_t)VM_FAULT_NOPAGE); + mmap_read_lock(native); + result = vmf_insert_page_offset(vmas[2], vmas[2]->vm_start, page, 0); + mmap_read_unlock(native); + KUNIT_ASSERT_EQ(test, result, (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, mm_subpage_file_mapcounts(page, &counts), 5U); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, counts.slots[i], i == TEST_SLOTS - 1 ? 2U : 1U); + KUNIT_EXPECT_EQ(test, folio_mapcount(page_folio(page)), 6); + KUNIT_EXPECT_EQ(test, page_ref_count(page), 7); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(middle, MM_FILEPAGES), 4UL); + /* Real mm teardown removes whole16K leaves and preserves both aliases. */ + kunit_release_action(test, user4k_vma_mm_free, middle); + KUNIT_EXPECT_EQ(test, mm_subpage_file_mapcounts(page, &counts), 1U); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, counts.slots[i], i == TEST_SLOTS - 1 ? 1U : 0U); + KUNIT_EXPECT_EQ(test, folio_mapcount(page_folio(page)), 2); + KUNIT_EXPECT_EQ(test, page_ref_count(page), 3); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small, test_fault_entry(test, small, vmas[1]->vm_start)), + page_to_phys(page) + PAGE_SIZE - SZ_4K); + KUNIT_EXPECT_EQ(test, pte_phys_mm(native, test_fault_entry(test, native, vmas[2]->vm_start)), + page_to_phys(page)); + kunit_release_action(test, user4k_vma_mm_free, small); + KUNIT_EXPECT_EQ(test, mm_subpage_file_mapcounts(page, &counts), 0U); + KUNIT_EXPECT_EQ(test, folio_mapcount(page_folio(page)), 1); + kunit_release_action(test, user4k_vma_mm_free, native); + KUNIT_EXPECT_EQ(test, folio_mapcount(page_folio(page)), 0); + KUNIT_EXPECT_EQ(test, page_ref_count(page), 1); +} + +static void user4k_insert_fragment_test(struct kunit *test) +{ + void *data = test_alloc_page(test); + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vma; + unsigned long start = TEST_VA + SZ_4K; + struct page *page; + vm_fault_t result[4], invalid[4]; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, data); + KUNIT_ASSERT_NOT_NULL(test, mm); + page = virt_to_page(data); + vma = test_vma_add(mm, start, PAGE_SIZE, (struct vm_page_offset) {}, + VM_READ | VM_MAYREAD | VM_MIXEDMAP | VM_IO); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + mmap_read_lock(mm); + for (i = 0; i < 4; i++) + result[i] = vmf_insert_page_offset(vma, start + i * SZ_4K, page, (3 - i) * SZ_4K); + invalid[0] = vmf_insert_page_offset(vma, start + 1, page, 0); + invalid[1] = vmf_insert_page_offset(vma, start, page, 1); + invalid[2] = vmf_insert_page_offset(vma, start, page, PAGE_SIZE); + invalid[3] = vmf_insert_page_offset(vma, vma->vm_end, page, 0); + mmap_read_unlock(mm); + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_EQ(test, result[i], (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, invalid[i], (vm_fault_t)VM_FAULT_SIGBUS); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, test_fault_entry(test, mm, start + i * SZ_4K)), + page_to_phys(page) + (3 - i) * SZ_4K); + } + KUNIT_EXPECT_EQ(test, page_ref_count(page), 5); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_FILEPAGES), 4UL); + kunit_release_action(test, user4k_vma_mm_free, mm); + KUNIT_EXPECT_EQ(test, page_ref_count(page), 1); +} + +#if defined(CONFIG_GENERIC_GETTIMEOFDAY) && defined(CONFIG_VDSO_GETRANDOM) +static void user4k_vdso_mapping_test(struct kunit *test) +{ + struct mm_struct *mm; + struct linux_binprm bprm = {}; + unsigned long code, base, left[8]; + u8 bytes[8], magic[4]; + unsigned int i; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm vDSO fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + mm->mmap_base = TEST_VA + SZ_4K; + mm_flags_clear(MMF_TOPDOWN, mm); + kthread_use_mm(mm); + ret = arch_setup_additional_pages(&bprm, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + code = (unsigned long)mm->context.vdso; + KUNIT_ASSERT_NE(test, code, 0UL); + base = code - VDSO_NR_PAGES * PAGE_SIZE; + KUNIT_EXPECT_EQ(test, base & ~PAGE_MASK, SZ_4K); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, code, magic, sizeof(magic), 0), (int)sizeof(magic)); + KUNIT_EXPECT_MEMEQ(test, magic, "\177ELF", 4); + /* VVAR forbids remote GUP; ordinary uaccess must still fault it in. */ + kthread_use_mm(mm); + for (i = 0; i < 4; i++) { + left[i] = copy_from_user(&bytes[i], (void __user *)(base + i * SZ_4K), 1); + left[4 + i] = copy_from_user(&bytes[4 + i], + (void __user *)(base + VDSO_RNG_PAGE_OFFSET * PAGE_SIZE + i * SZ_4K), 1); + } + kthread_unuse_mm(mm); + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_EQ(test, left[i], 0UL); + KUNIT_EXPECT_EQ(test, left[4 + i], 0UL); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, test_fault_entry(test, mm, base + i * SZ_4K)), + page_to_phys(virt_to_page(vdso_k_time_data)) + i * SZ_4K); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, test_fault_entry(test, mm, + base + VDSO_RNG_PAGE_OFFSET * PAGE_SIZE + i * SZ_4K)), + page_to_phys(virt_to_page(vdso_k_rng_data)) + i * SZ_4K); + } + KUNIT_EXPECT_EQ(test, access_remote_vm(mm, base, magic, sizeof(magic), 0), 0); +} + +#endif + +static void user4k_remote_access_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA + SZ_4K, 3 * SZ_4K); + struct mm_struct *child = test_vma_mm(test); + struct vm_area_struct *child_vma; + unsigned long start = TEST_VA + SZ_4K; + u8 *buf = kunit_kmalloc(test, 3 * SZ_4K, GFP_KERNEL); + int ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, child); + KUNIT_ASSERT_NOT_NULL(test, buf); + memset(buf, 0x5a, 3 * SZ_4K); + KUNIT_ASSERT_EQ(test, access_remote_vm(vma->vm_mm, start, buf, 3 * SZ_4K, FOLL_WRITE), + 3 * (int)SZ_4K); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + /* Deliberately straddle quarters whose physical/virtual offsets differ. */ + memset(buf, 0x96, SZ_4K + 34); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + SZ_4K - 17, + buf, SZ_4K + 34, FOLL_WRITE), (int)SZ_4K + 34); + memset(buf, 0, 3 * SZ_4K); + KUNIT_EXPECT_EQ(test, access_remote_vm(vma->vm_mm, start, buf, 3 * SZ_4K, 0), + 3 * (int)SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0x5a, 3 * SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start, buf, 3 * SZ_4K, 0), 3 * (int)SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0x5a, SZ_4K - 17), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + SZ_4K - 17, 0x96, SZ_4K + 34), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + 2 * SZ_4K + 17, 0x5a, SZ_4K - 17), NULL); + child_vma = test_vma_lookup(child, start); + KUNIT_ASSERT_NOT_NULL(test, child_vma); + mmap_write_lock(child); + vm_flags_clear(child_vma, VM_WRITE); + mmap_write_unlock(child); + buf[0] = 0x27; + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + 1, buf, 1, FOLL_WRITE), 0); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + 1, buf, 1, + FOLL_WRITE | FOLL_FORCE), 1); + buf[0] = 0; + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + 1, buf, 1, 0), 1); + KUNIT_EXPECT_EQ(test, buf[0], (u8)0x27); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + 3 * SZ_4K - 5, buf, 20, 0), 5); + KUNIT_EXPECT_EQ(test, access_remote_vm(child, start + 3 * SZ_4K, buf, 20, 0), 0); +} + +static void user4k_remote_file_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 2 * PAGE_SIZE, true); + struct vm_area_struct *small, *native; + u8 buf[34]; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, 3 * SZ_4K, 2 * SZ_4K); + native = test_file_vma(test, file, true, true, TEST_VA, 2 * PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_EXPECT_EQ(test, access_remote_vm(small->vm_mm, TEST_VA + 3 * SZ_4K - 17, + buf, sizeof(buf), 0), (int)sizeof(buf)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0x34, 17), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + 17, 0x35, 17), NULL); + memset(buf, 0xa6, sizeof(buf)); + KUNIT_EXPECT_EQ(test, access_remote_vm(small->vm_mm, TEST_VA + 3 * SZ_4K - 17, + buf, sizeof(buf), FOLL_WRITE), (int)sizeof(buf)); + memset(buf, 0, sizeof(buf)); + KUNIT_EXPECT_EQ(test, access_remote_vm(native->vm_mm, TEST_VA + 4 * SZ_4K - 17, + buf, sizeof(buf), 0), (int)sizeof(buf)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0xa6, sizeof(buf)), NULL); +#ifdef CONFIG_CROSS_MEMORY_ATTACH + { + struct kvec vec = { .iov_base = buf, .iov_len = sizeof(buf) }; + struct iov_iter iter; + + memset(buf, 0xb7, sizeof(buf)); + iov_iter_kvec(&iter, ITER_SOURCE, &vec, 1, sizeof(buf)); + KUNIT_EXPECT_EQ(test, process_vm_rw_fragments(small->vm_mm, + TEST_VA + 3 * SZ_4K - 17, sizeof(buf), &iter, true), 0); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 0UL); + memset(buf, 0, sizeof(buf)); + KUNIT_EXPECT_EQ(test, access_remote_vm(native->vm_mm, TEST_VA + 4 * SZ_4K - 17, + buf, sizeof(buf), 0), (int)sizeof(buf)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0xb7, sizeof(buf)), NULL); + } +#endif + mmap_write_lock(small->vm_mm); + vm_flags_clear(small, VM_WRITE); + mmap_write_unlock(small->vm_mm); + KUNIT_EXPECT_EQ(test, access_remote_vm(small->vm_mm, small->vm_start, buf, 1, + FOLL_WRITE | FOLL_FORCE), 0); +#ifdef CONFIG_CROSS_MEMORY_ATTACH + { + struct kvec vec = { .iov_base = buf, .iov_len = sizeof(buf) }; + struct iov_iter iter; + + iov_iter_kvec(&iter, ITER_SOURCE, &vec, 1, sizeof(buf)); + KUNIT_EXPECT_EQ(test, process_vm_rw_fragments(small->vm_mm, + small->vm_start, sizeof(buf), &iter, true), -EFAULT); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), sizeof(buf)); + } +#endif +} + +#ifdef CONFIG_CROSS_MEMORY_ATTACH +static void user4k_process_vm_test(struct kunit *test) +{ + /* Exceed the stack batch, cross table boundaries and include a partial tail. */ + unsigned long start = SZ_2M - SZ_4K, len = 21 * SZ_4K; + struct vm_area_struct *vma = test_fault_vma(test, start, len); + struct mm_struct *child = test_vma_mm(test); + struct iov_iter iter; + struct kvec vec; + u8 *buf = kunit_kmalloc(test, len + 17, GFP_KERNEL); + int ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, child); + KUNIT_ASSERT_NOT_NULL(test, buf); + memset(buf, 0x61, len); + vec = (struct kvec) { .iov_base = buf, .iov_len = len }; + iov_iter_kvec(&iter, ITER_SOURCE, &vec, 1, len); + ret = process_vm_rw_fragments(vma->vm_mm, start, len, &iter, true); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, iov_iter_count(&iter), 0UL); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + memset(buf, 0xc7, len); + iov_iter_kvec(&iter, ITER_SOURCE, &vec, 1, len - 34); + ret = process_vm_rw_fragments(child, start + 17, len - 34, &iter, true); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 0UL); + memset(buf, 0, len); + iov_iter_kvec(&iter, ITER_DEST, &vec, 1, len); + ret = process_vm_rw_fragments(vma->vm_mm, start, len, &iter, false); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0x61, len), NULL); + vec.iov_len = len + 17; + iov_iter_kvec(&iter, ITER_DEST, &vec, 1, len + 17); + ret = process_vm_rw_fragments(child, start, len + 17, &iter, false); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 17UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf, 0x61, 17), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + 17, 0xc7, len - 34), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + len - 17, 0x61, 17), NULL); + /* A shorter local iterator ends successfully before the remote range. */ + iov_iter_kvec(&iter, ITER_DEST, &vec, 1, 23); + ret = process_vm_rw_fragments(child, start + 13, len, &iter, false); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 0UL); +} +#endif + +#ifdef CONFIG_FUTEX +static void test_stop_kthread(void *arg); + +struct test_futex_waiter { + struct mm_struct *mm; + unsigned long address; + struct completion ready; + struct completion done; + int result; +}; + +static int test_futex_wait_run(void *arg) +{ + struct test_futex_waiter *w = arg; + ktime_t deadline = ktime_add_ms(ktime_get(), 10000); + + kthread_use_mm(w->mm); + complete(&w->ready); + w->result = futex_wait((u32 __user *)w->address, FLAGS_SIZE_32 | FLAGS_SHARED, + 0x33333333, &deadline, FUTEX_BITSET_MATCH_ANY); + kthread_unuse_mm(w->mm); + complete(&w->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void user4k_futex_wake_test(struct kunit *test) +{ + struct file *file; + struct vm_area_struct *vmas[2]; + unsigned long addresses[2] = { TEST_VA + SZ_4K + 64, TEST_VA + 2 * SZ_4K + 64 }; + unsigned int round; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm futex fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, PAGE_SIZE, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vmas[0] = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, SZ_4K, 2 * SZ_4K); + vmas[1] = test_file_vma(test, file, true, true, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, vmas[0]); + KUNIT_ASSERT_NOT_NULL(test, vmas[1]); + /* Both directions use the real wait queue and atomic user-value read. */ + for (round = 0; round < 2; round++) { + struct test_futex_waiter *w = kunit_kzalloc(test, sizeof(*w), GFP_KERNEL); + struct task_struct *task; + unsigned long until; + int woke = 0, wrong; + + KUNIT_ASSERT_NOT_NULL(test, w); + w->mm = vmas[round]->vm_mm; + w->address = addresses[round]; + init_completion(&w->ready); + init_completion(&w->done); + task = kthread_run(test_futex_wait_run, w, "user4k-futex-%u", round); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, task); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_stop_kthread, task), 0); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&w->ready, 5 * HZ), 0UL); + until = jiffies + 5 * HZ; + kthread_use_mm(vmas[round ^ 1]->vm_mm); + wrong = futex_wake((u32 __user *)(addresses[round ^ 1] + 4), + FLAGS_SIZE_32 | FLAGS_SHARED, 1, FUTEX_BITSET_MATCH_ANY); + do { + woke = futex_wake((u32 __user *)addresses[round ^ 1], + FLAGS_SIZE_32 | FLAGS_SHARED, 1, FUTEX_BITSET_MATCH_ANY); + if (woke || completion_done(&w->done)) + break; + schedule_timeout_uninterruptible(1); + } while (time_before(jiffies, until)); + kthread_unuse_mm(vmas[round ^ 1]->vm_mm); + KUNIT_EXPECT_EQ(test, wrong, 0); + KUNIT_EXPECT_EQ(test, woke, 1); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&w->done, 12 * HZ), 0UL); + KUNIT_EXPECT_EQ(test, w->result, 0); + kunit_release_action(test, test_stop_kthread, task); + } +} + +static void user4k_futex_keys_test(struct kunit *test) +{ + struct file *file; + struct vm_area_struct *small, *native; + union futex_key keys[2], other; + unsigned int i; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm futex fixture requires hardware opt-in"); + return; + } + file = test_cache_file(test, 2 * PAGE_SIZE, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, 3 * SZ_4K, 2 * SZ_4K); + native = test_file_vma(test, file, true, true, TEST_VA, 2 * PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, native); + for (i = 0; i < 3; i++) { + unsigned long va = small->vm_start + i * SZ_4K + 64; + unsigned long fileoff = (i + 2) * SZ_4K + 64; + + kthread_use_mm(small->vm_mm); + ret = get_futex_key((u32 __user *)va, FLAGS_SIZE_32 | FLAGS_SHARED, + &keys[0], FUTEX_READ); + kthread_unuse_mm(small->vm_mm); + KUNIT_ASSERT_EQ(test, ret, 0); + kthread_use_mm(native->vm_mm); + ret = get_futex_key((u32 __user *)(TEST_VA + fileoff), + FLAGS_SIZE_32 | FLAGS_SHARED, &keys[1], FUTEX_READ); + kthread_unuse_mm(native->vm_mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, futex_match(&keys[0], &keys[1])); + KUNIT_EXPECT_EQ(test, keys[0].shared.pgoff, fileoff / PAGE_SIZE); + KUNIT_EXPECT_EQ(test, keys[0].both.offset, (unsigned int)(fileoff % PAGE_SIZE) | FUT_OFF_INODE); + kthread_use_mm(small->vm_mm); + ret = get_futex_key((u32 __user *)(va + 4), FLAGS_SIZE_32 | FLAGS_SHARED, + &other, FUTEX_READ); + kthread_unuse_mm(small->vm_mm); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_FALSE(test, futex_match(&keys[0], &other)); + } + mmap_write_lock(small->vm_mm); + vm_flags_clear(small, VM_WRITE); + mmap_write_unlock(small->vm_mm); + kthread_use_mm(small->vm_mm); + ret = get_futex_key((u32 __user *)(small->vm_start + 2 * SZ_4K + 64), + FLAGS_SIZE_32 | FLAGS_SHARED, &other, FUTEX_READ); + kthread_unuse_mm(small->vm_mm); + KUNIT_EXPECT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, futex_match(&keys[1], &other)); +} +#endif + +static void test_swap_usage_mapping_free(void *arg) +{ + struct address_space *mapping = arg; + + xa_destroy(&mapping->i_pages); +} + +static void user4k_shmem_swap_usage_test(struct kunit *test) +{ + struct address_space *mapping = kunit_kzalloc(test, sizeof(*mapping), GFP_KERNEL); + void *ret; + + KUNIT_ASSERT_NOT_NULL(test, mapping); + xa_init_flags(&mapping->i_pages, XA_FLAGS_LOCK_IRQ); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_swap_usage_mapping_free, mapping), 0); + /* Synthetic swap values exercise accounting without allocating swap slots. */ + ret = xa_store(&mapping->i_pages, 1, xa_mk_value(1), GFP_KERNEL); + KUNIT_ASSERT_FALSE(test, xa_is_err(ret)); + if (IS_ENABLED(CONFIG_XARRAY_MULTI)) { + ret = xa_store_range(&mapping->i_pages, 4, 7, + xa_mk_value(2), GFP_KERNEL); + KUNIT_ASSERT_FALSE(test, xa_is_err(ret)); + } else { + unsigned long index; + + /* Without multi-index entries, shmem stores one value per page. */ + for (index = 4; index <= 7; index++) { + ret = xa_store(&mapping->i_pages, index, + xa_mk_value(2), GFP_KERNEL); + KUNIT_ASSERT_FALSE(test, xa_is_err(ret)); + } + } + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, PAGE_SIZE + SZ_4K, + PAGE_SIZE + 3 * SZ_4K), 2UL * SZ_4K); + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, 3 * SZ_4K, + 2 * PAGE_SIZE + SZ_4K), PAGE_SIZE); + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, 4 * PAGE_SIZE + SZ_4K, + 7 * PAGE_SIZE + 3 * SZ_4K), 3 * PAGE_SIZE + 2UL * SZ_4K); + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, 5 * PAGE_SIZE + SZ_4K, + 5 * PAGE_SIZE + 2 * SZ_4K), (unsigned long)SZ_4K); + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, 0, SZ_4K), 0UL); + KUNIT_EXPECT_EQ(test, shmem_partial_swap_usage_bytes(mapping, PAGE_SIZE, PAGE_SIZE), 0UL); +} + +static void test_file_mapcounts(struct kunit *test, struct folio *folio, + const unsigned int expected[TEST_SLOTS], unsigned int native) +{ + struct mm_subpage_mapcounts counts; + unsigned int i, total = 0; + + for (i = 0; i < TEST_SLOTS; i++) + total += expected[i]; + KUNIT_EXPECT_EQ(test, mm_subpage_file_mapcounts(&folio->page, &counts), total); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, counts.slots[i], expected[i]); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)(total + native)); +} + +static void user4k_file_mapcounts_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct vm_area_struct *shared, *private, *native, *cvma; + struct mm_struct *child; + struct folio *folio; + int ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shared = test_file_vma(test, file, false, true, TEST_VA, 3 * SZ_4K, SZ_4K); + private = test_file_vma(test, file, false, false, TEST_VA, 3 * SZ_4K, SZ_4K); + native = test_file_vma(test, file, true, true, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, shared); + KUNIT_ASSERT_NOT_NULL(test, private); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, test_file_fault(shared, TEST_VA, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(private, TEST_VA, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA, false) & VM_FAULT_ERROR, 0U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 2, 2, 2 }, 1); + KUNIT_ASSERT_EQ(test, test_file_fault(private, TEST_VA + 2 * SZ_4K, true) & + VM_FAULT_ERROR, 0U); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 2, 2, 1 }, 1); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, private->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 3, 3, 1 }, 1); + KUNIT_ASSERT_EQ(test, test_file_fault(cvma, TEST_VA + SZ_4K, true) & VM_FAULT_ERROR, 0U); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 3, 2, 1 }, 1); + kunit_release_action(test, user4k_vma_mm_free, child); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 2, 2, 1 }, 1); + kunit_release_action(test, user4k_vma_mm_free, private->vm_mm); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 1, 1, 1 }, 1); + kunit_release_action(test, user4k_vma_mm_free, native->vm_mm); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 1, 1, 1 }, 0); + kunit_release_action(test, user4k_vma_mm_free, shared->vm_mm); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 0, 0, 0 }, 0); +} + +/* A unique RAM-backed test inode: exercise generic file faults without + * making unsupported filesystem promotion part of this mapping fixture. + */ +static int test_coarse_read_folio(struct file *file, struct folio *folio) +{ + unsigned long i; + loff_t base = (loff_t)folio->index << PAGE_SHIFT; + loff_t size = i_size_read(file_inode(file)); + u8 *data = folio_address(folio); + + for (i = 0; i < folio_size(folio); i++) + data[i] = base + i < size ? 0x31 + ((base + i) / SZ_4K) : 0; + __folio_mark_uptodate(folio); + folio_unlock(folio); + return 0; +} + +static struct file *test_coarse_cache_file(struct kunit *test, unsigned int order, + unsigned long size) +{ + static const struct address_space_operations aops = { + .read_folio = test_coarse_read_folio, + .dirty_folio = filemap_dirty_folio, + }; + static const struct file_operations fops = { + .mmap = generic_file_mmap, + .read_iter = generic_file_read_iter, + .llseek = generic_file_llseek, + }; + struct file *file = anon_inode_create_getfile("user-coarse-file", &fops, + NULL, O_RDWR, NULL); + struct folio *folio; + int ret; + + if (IS_ERR(file)) + return file; + if (kunit_add_action_or_reset(test, test_file_put, file)) + return ERR_PTR(-ENOMEM); + file->f_mapping->a_ops = &aops; + mapping_set_large_folios(file->f_mapping); + i_size_write(file_inode(file), size); + folio = folio_alloc(GFP_KERNEL, order); + if (!folio) + return ERR_PTR(-ENOMEM); + ret = filemap_add_folio(file->f_mapping, folio, 0, GFP_KERNEL); + if (!ret) + test_coarse_read_folio(file, folio); /* Unlocks the installed folio. */ + folio_put(folio); + return ret ? ERR_PTR(ret) : file; +} + +static bool test_coarse_file_enabled(struct kunit *test) +{ + if (PAGE_SHIFT >= 16 || + (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN))) { + kunit_skip(test, "requires internal larger user granule and borrowed-mm opt-in"); + return false; + } + return true; +} + +static void test_promote_unpin(void *data) +{ + struct user_page_fragment *fragment = data; + + if (fragment->folio) { + release_user_fragments(fragment, 1, true); + fragment->folio = NULL; + } +} + +static void user64k_file_promote_test(struct kunit *test) +{ + unsigned int pinned; + + if (!test_coarse_file_enabled(test)) + return; + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "cache promotion currently uses the collapse implementation"); + return; + } + for (pinned = 0; pinned < 2; pinned++) { + struct file *file = test_cache_file(test, 2 * SZ_64K, true); + struct vm_area_struct *coarse, *small; + struct user_page_fragment *fragment; + struct folio *folio; + phys_addr_t old[SZ_64K / PAGE_SIZE], untouched; + unsigned int nr = SZ_64K / PAGE_SIZE, i; + unsigned long offset, modified = SZ_64K / 2 + 19; + pte_t entry; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + coarse = test_file_vma_granule(test, file, 16, true, TEST_VA, SZ_64K, SZ_64K); + small = test_file_vma(test, file, false, true, TEST_VA, SZ_64K, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_NOT_NULL(test, small); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + untouched = page_to_phys(&folio->page); + folio_put(folio); + for (i = 0; i < nr; i++) { + folio = filemap_get_folio(file->f_mapping, nr + i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_order(folio), 0U); + old[i] = page_to_phys(&folio->page); + folio_put(folio); + } + for (offset = 0; offset < SZ_64K; offset += SZ_4K) + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + if (pinned) { + fragment = kunit_kzalloc(test, sizeof(*fragment), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, fragment); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_promote_unpin, + fragment), 0); + KUNIT_ASSERT_EQ(test, pin_user_fragments_remote(small->vm_mm, + TEST_VA + modified, 1, FOLL_WRITE, fragment, 1), 1L); + *(u8 *)(folio_address(fragment->folio) + fragment->offset) = 0xa7; + KUNIT_EXPECT_EQ(test, collapse_file_user_page(coarse->vm_mm, file, nr), -EBUSY); + /* A pin in the middle forces rollback of earlier isolated folios. */ + for (i = 0; i < nr; i++) { + folio = filemap_get_folio(file->f_mapping, nr + i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, page_to_phys(&folio->page), old[i]); + KUNIT_EXPECT_EQ(test, folio_order(folio), 0U); + folio_put(folio); + } + KUNIT_EXPECT_EQ(test, *(u8 *)(folio_address(fragment->folio) + + fragment->offset), (u8)0xa7); + kunit_release_action(test, test_promote_unpin, fragment); + } + KUNIT_ASSERT_EQ(test, collapse_file_user_page(coarse->vm_mm, file, nr), 0); + folio = filemap_get_folio(file->f_mapping, nr); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_EXPECT_EQ(test, folio_order(folio), 16U - PAGE_SHIFT); + KUNIT_EXPECT_EQ(test, folio->index, (pgoff_t)nr); + KUNIT_EXPECT_TRUE(test, folio_test_dirty(folio)); + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, (unsigned long)2 * nr); + for (offset = 0; offset < SZ_64K; offset++) { + u8 expected = pinned && offset == modified ? 0xa7 : 0x41 + offset / SZ_4K; + + if (((u8 *)folio_address(folio))[offset] != expected) { + KUNIT_FAIL(test, "promoted data mismatch at %lu", offset); + break; + } + } + /* An already suitable folio is an idempotent success, including with a GET. */ + KUNIT_EXPECT_EQ(test, collapse_file_user_page(coarse->vm_mm, file, nr), 0); + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA, false) & VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, coarse->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(coarse->vm_mm, entry), page_to_phys(&folio->page)); + for (offset = 0; offset < SZ_64K; offset += SZ_4K) { + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, small->vm_mm, TEST_VA + offset); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, entry), + page_to_phys(&folio->page) + offset); + } + { + struct folio *first = filemap_get_folio(file->f_mapping, 0); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, first); + KUNIT_EXPECT_EQ(test, page_to_phys(&first->page), untouched); + folio_put(first); + } + kunit_release_action(test, user4k_vma_mm_free, coarse->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)nr + 1); + } +} + +static void user64k_shmem_fault_test(struct kunit *test) +{ + unsigned int mode; + + if (!test_coarse_file_enabled(test)) + return; + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "coarse shmem promotion requires collapse support"); + return; + } + for (mode = 0; mode < 4; mode++) { + bool seeded = mode & 1, shared = mode < 2; + struct file *file = test_cache_file(test, SZ_64K, seeded); + struct vm_area_struct *vma; + struct folio *cache, *mapped; + unsigned long offset; + pte_t entry; + vm_fault_t fault; + unsigned int attempt; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma_granule(test, file, 16, shared, TEST_VA, SZ_64K, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + for (attempt = 0; attempt < 8; attempt++) { + fault = test_file_fault(vma, TEST_VA, true); + if (!(fault & VM_FAULT_NOPAGE)) + break; + } + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY | VM_FAULT_NOPAGE), 0U); + entry = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_TRUE(test, pte_write(entry)); + mapped = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(vma->vm_mm, entry)))); + KUNIT_EXPECT_EQ(test, folio_size(mapped), SZ_64K); + KUNIT_EXPECT_EQ(test, !!folio_test_anon(mapped), !shared); + for (offset = 0; offset < SZ_64K; offset += SZ_4K) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mapped) + offset, + seeded ? 0x31 + offset / SZ_4K : 0, SZ_4K), NULL); + cache = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, cache); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, cache), 0); + KUNIT_EXPECT_EQ(test, folio_size(cache), SZ_64K); + if (shared) + KUNIT_EXPECT_PTR_EQ(test, mapped, cache); + else { + KUNIT_EXPECT_PTR_NE(test, mapped, cache); + memset(folio_address(mapped), 0xc9, SZ_64K); + for (offset = 0; offset < SZ_64K; offset += SZ_4K) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(cache) + offset, + seeded ? 0x31 + offset / SZ_4K : 0, SZ_4K), NULL); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, + shared ? MM_SHMEMPAGES : MM_ANONPAGES), 1L); + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, SZ_64K / PAGE_SIZE); + { + u8 byte = 0x83, observed = 0; + + KUNIT_EXPECT_EQ(test, access_remote_vm(vma->vm_mm, TEST_VA + SZ_64K - 1, + &byte, 1, FOLL_WRITE), 1); + KUNIT_EXPECT_EQ(test, access_remote_vm(vma->vm_mm, TEST_VA + SZ_64K - 1, + &observed, 1, 0), 1); + KUNIT_EXPECT_EQ(test, observed, byte); + KUNIT_EXPECT_EQ(test, ((u8 *)folio_address(cache))[SZ_64K - 1], + shared ? byte : (u8)(seeded ? 0x40 : 0)); + } + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(cache), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(cache), (int)(SZ_64K / PAGE_SIZE) + 1); + } +} + +static void user64k_shmem_fault_busy_test(struct kunit *test) +{ + unsigned int mode; + + if (!test_coarse_file_enabled(test)) + return; + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "coarse shmem promotion requires collapse support"); + return; + } + for (mode = 0; mode < 3; mode++) { + struct file *file = test_cache_file(test, SZ_64K, mode != 0); + struct vm_area_struct *vma; + struct folio *held; + vm_fault_t fault; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma_granule(test, file, 16, true, TEST_VA, SZ_64K, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + if (!mode) { + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_REMOTE | + FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT, NULL); + mmap_assert_locked(vma->vm_mm); + mmap_read_unlock(vma->vm_mm); + KUNIT_EXPECT_EQ(test, fault, (vm_fault_t)VM_FAULT_RETRY); + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, 0UL); + } else if (mode == 1) { + held = filemap_get_folio(file->f_mapping, SZ_64K / (2 * PAGE_SIZE)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, held); + folio_lock(held); + fault = test_file_fault(vma, TEST_VA, false); + folio_unlock(held); + folio_put(held); + KUNIT_EXPECT_EQ(test, fault, (vm_fault_t)VM_FAULT_NOPAGE); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SHMEMPAGES), 0L); + } else { + struct vm_area_struct *small = test_file_vma(test, file, false, true, + TEST_VA, SZ_64K, 0); + struct user_page_fragment *fragment; + + KUNIT_ASSERT_NOT_NULL(test, small); + fragment = kunit_kzalloc(test, sizeof(*fragment), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, fragment); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_promote_unpin, + fragment), 0); + KUNIT_ASSERT_EQ(test, pin_user_fragments_remote(small->vm_mm, + TEST_VA + SZ_64K / 2, 1, FOLL_WRITE, fragment, 1), 1L); + fault = test_file_fault(vma, TEST_VA, false); + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_SIGBUS); + KUNIT_EXPECT_EQ(test, *(u8 *)(folio_address(fragment->folio) + + fragment->offset), (u8)0x39); + kunit_release_action(test, test_promote_unpin, fragment); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + } + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & + (VM_FAULT_ERROR | VM_FAULT_RETRY | VM_FAULT_NOPAGE), 0U); + KUNIT_EXPECT_TRUE(test, pte_present(test_fault_entry(test, vma->vm_mm, TEST_VA))); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + } +} + +static void user64k_shmem_fault_hole_test(struct kunit *test) +{ +#ifdef CONFIG_USERFAULTFD + unsigned int adjacent; + + if (!test_coarse_file_enabled(test)) + return; + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "coarse shmem promotion requires collapse support"); + return; + } + for (adjacent = 0; adjacent < 2; adjacent++) { + struct file *file = test_cache_file(test, 2 * SZ_64K, false); + struct vm_area_struct *coarse, *missing, *seed; + struct folio *folio; + vm_fault_t fault; + bool new; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + coarse = test_file_vma_granule(test, file, 16, true, TEST_VA, SZ_64K, 0); + missing = test_file_vma(test, file, false, true, TEST_VA, SZ_4K, + adjacent ? SZ_64K : PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_NOT_NULL(test, missing); + seed = test_file_vma(test, file, false, true, TEST_VA, SZ_4K, 0); + KUNIT_ASSERT_NOT_NULL(test, seed); + mmap_read_lock(seed->vm_mm); + folio = shmem_uffd_prepare_folio(seed, TEST_VA, &new); + mmap_read_unlock(seed->vm_mm); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, new); + memset(folio_address(folio), 0x6f, SZ_4K); + shmem_uffd_mark_present(folio, 0, BIT(0)); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), + GENMASK(PAGE_SIZE / SZ_4K - 1, 1)); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + /* Model the registration flag used by the collapse interval check. */ + mmap_write_lock(missing->vm_mm); + vma_start_write(missing); + vm_flags_set(missing, VM_UFFD_MISSING); + mmap_write_unlock(missing->vm_mm); + fault = test_file_fault(coarse, TEST_VA, false); + mmap_write_lock(missing->vm_mm); + vma_start_write(missing); + vm_flags_clear(missing, VM_UFFD_MISSING); + mmap_write_unlock(missing->vm_mm); + /* A real coarse fault may populate holes covered by its entire leaf. */ + KUNIT_EXPECT_EQ(test, fault & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA, false) & + (VM_FAULT_ERROR | VM_FAULT_RETRY | VM_FAULT_NOPAGE), 0U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), SZ_64K); + folio_lock(folio); + KUNIT_EXPECT_FALSE(test, shmem_uffd_range_missing(folio, 1, 0, SZ_4K)); + folio_unlock(folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0x6f, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0, + SZ_64K - SZ_4K), NULL); + folio_put(folio); + kunit_release_action(test, user4k_vma_mm_free, coarse->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, missing->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, seed->vm_mm); + } +#else + kunit_skip(test, "requires userfaultfd registration flags"); +#endif +} + +static void user64k_file_leaf_test(struct kunit *test) +{ + struct file *file; + struct vm_area_struct *coarse, *native, *small, *child_vma; + struct mm_struct *coarse_mm, *child; + struct folio *folio; + unsigned long offset; + unsigned int nr = (2 * SZ_64K) / PAGE_SIZE; + unsigned int small_nr = 2 * SZ_64K / SZ_4K - 1; + unsigned int maps = 2 * nr + small_nr; + u8 byte = 0xc7, observed; + int ret; + + if (!test_coarse_file_enabled(test)) + return; + file = test_coarse_cache_file(test, 17 - PAGE_SHIFT, 2 * SZ_64K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + coarse = test_file_vma_granule(test, file, 16, true, TEST_VA, 2 * SZ_64K, 0); + native = test_file_vma(test, file, true, true, TEST_VA, 2 * SZ_64K, 0); + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + 2 * SZ_64K - SZ_4K, SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_NOT_NULL(test, small); + coarse_mm = coarse->vm_mm; + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA + SZ_64K, false) & + VM_FAULT_ERROR, 0U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + /* One cache folio supplies two user leaves, with native references per leaf. */ + for (offset = 0; offset < 2 * SZ_64K; offset += SZ_64K) { + pte_t entry = test_fault_entry(test, coarse_mm, TEST_VA + offset); + + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(coarse_mm, entry), + page_to_phys(&folio->page) + offset); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(coarse_mm, MM_FILEPAGES), 2L); + for (offset = 0; offset < 2 * SZ_64K; offset += PAGE_SIZE) + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + for (offset = SZ_4K; offset < 2 * SZ_64K; offset += SZ_4K) + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)(nr + maps + 1)); + KUNIT_ASSERT_EQ(test, access_remote_vm(coarse_mm, TEST_VA + SZ_64K + 3 * SZ_4K, + &byte, 1, FOLL_WRITE), 1); + KUNIT_ASSERT_EQ(test, access_remote_vm(small->vm_mm, TEST_VA + SZ_64K + 3 * SZ_4K, + &observed, 1, 0), 1); + KUNIT_EXPECT_EQ(test, observed, byte); + byte = 0x98; + KUNIT_ASSERT_EQ(test, access_remote_vm(small->vm_mm, TEST_VA + 7 * SZ_4K, + &byte, 1, FOLL_WRITE), 1); + KUNIT_ASSERT_EQ(test, access_remote_vm(native->vm_mm, TEST_VA + 7 * SZ_4K, + &observed, 1, 0), 1); + KUNIT_EXPECT_EQ(test, observed, byte); + KUNIT_EXPECT_EQ(test, ((u8 *)folio_address(folio))[7 * SZ_4K], byte); + + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, coarse_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + /* Ordinary file VMAs are lazily populated in the child after fork. */ + mmap_read_lock(child); + child_vma = find_vma(child, TEST_VA); + mmap_read_unlock(child); + KUNIT_ASSERT_NOT_NULL(test, child_vma); + KUNIT_ASSERT_EQ(test, test_file_fault(child_vma, TEST_VA, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(child_vma, TEST_VA + SZ_64K, false) & + VM_FAULT_ERROR, 0U); + maps += nr; + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + kthread_use_mm(coarse_mm); + ret = vm_munmap(TEST_VA, SZ_64K); + kthread_unuse_mm(coarse_mm); + KUNIT_ASSERT_EQ(test, ret, 0); + maps -= SZ_64K / PAGE_SIZE; + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(coarse_mm, MM_FILEPAGES), 1L); + KUNIT_EXPECT_TRUE(test, pte_present(test_fault_entry(test, child, TEST_VA))); + + /* The native-rounded range must remove every overlapping coarse alias. */ + unmap_mapping_range(file->f_mapping, SZ_64K + SZ_4K, SZ_4K, 0); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, coarse_mm, TEST_VA + SZ_64K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, child, TEST_VA + SZ_64K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, native->vm_mm, TEST_VA + SZ_64K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, small->vm_mm, TEST_VA + SZ_64K + SZ_4K))); + KUNIT_EXPECT_TRUE(test, pte_present(test_fault_entry(test, small->vm_mm, + TEST_VA + SZ_64K + PAGE_SIZE))); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(coarse_mm, MM_FILEPAGES), 0L); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), + (int)(maps - 2 * SZ_64K / PAGE_SIZE - 1 - PAGE_SIZE / SZ_4K)); + + /* An exact subpage hole keeps neighboring small leaves in that native page. */ + mmap_read_lock(coarse_mm); + coarse = find_vma(coarse_mm, TEST_VA + SZ_64K); + mmap_read_unlock(coarse_mm); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA + SZ_64K, false) & + VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(child_vma, TEST_VA + SZ_64K, false) & + VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA + SZ_64K, false) & + VM_FAULT_ERROR, 0U); + for (offset = 0; offset < PAGE_SIZE; offset += SZ_4K) + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + SZ_64K + offset, + false) & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + unmap_mapping_subpage_hole(file->f_mapping, SZ_64K + SZ_4K, SZ_64K + 2 * SZ_4K); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, coarse_mm, TEST_VA + SZ_64K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, child, TEST_VA + SZ_64K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, native->vm_mm, TEST_VA + SZ_64K))); + for (offset = 0; offset < PAGE_SIZE; offset += SZ_4K) + KUNIT_EXPECT_EQ(test, !!pte_present(test_fault_entry(test, small->vm_mm, + TEST_VA + SZ_64K + offset)), offset != SZ_4K); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)(maps - 2 * SZ_64K / PAGE_SIZE - 2)); + kunit_release_action(test, user4k_vma_mm_free, coarse_mm); + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, native->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)nr + 1); +} + +static void user64k_file_cow_test(struct kunit *test) +{ + unsigned int cold; + + if (!test_coarse_file_enabled(test)) + return; + for (cold = 0; cold < 2; cold++) { + struct file *file = test_coarse_cache_file(test, 17 - PAGE_SHIFT, 2 * SZ_64K); + struct vm_area_struct *vma; + struct folio *cache, *anon; + unsigned long offset; + pte_t entry; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma_granule(test, file, 16, false, TEST_VA, SZ_64K, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + if (!cold) + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_TRUE(test, pte_write(entry)); + anon = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(vma->vm_mm, entry)))); + KUNIT_EXPECT_TRUE(test, folio_test_anon(anon)); + KUNIT_EXPECT_EQ(test, folio_size(anon), SZ_64K); + folio_get(anon); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, anon), 0); + for (offset = 0; offset < SZ_64K; offset += SZ_4K) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(anon) + offset, + 0x41 + offset / SZ_4K, SZ_4K), NULL); + memset(folio_address(anon), 0x93, SZ_64K); + cache = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, cache); + for (offset = 0; offset < 2 * SZ_64K; offset += SZ_4K) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(cache) + offset, + 0x31 + offset / SZ_4K, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, folio_mapcount(cache), 0); + folio_put(cache); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_FILEPAGES), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 1L); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(anon), (int)(SZ_64K / PAGE_SIZE)); + KUNIT_EXPECT_EQ(test, folio_ref_count(anon), (int)(SZ_64K / PAGE_SIZE) + 1); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(anon), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(anon), 1); + } +} + +static void user64k_file_bounds_test(struct kunit *test) +{ + struct file *file; + struct vm_area_struct *vma; + struct folio *folio; + unsigned int mode; + unsigned long size = SZ_64K + PAGE_SIZE + 19; + pte_t entry; + + if (!test_coarse_file_enabled(test)) + return; + /* Too-small backing and a physically unaligned source cannot form a leaf. */ + for (mode = 0; mode < 2; mode++) { + file = test_coarse_cache_file(test, mode ? 17 - PAGE_SHIFT : 0, 2 * SZ_64K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma_granule(test, file, 16, false, TEST_VA, SZ_64K, + mode ? PAGE_SIZE : 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_TRUE(test, test_file_fault(vma, TEST_VA, false) & VM_FAULT_SIGBUS); + KUNIT_EXPECT_TRUE(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_SIGBUS); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, vma->vm_mm, TEST_VA))); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + folio_put(folio); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + } + file = test_coarse_cache_file(test, 17 - PAGE_SHIFT, size); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma_granule(test, file, 16, true, TEST_VA, 3 * SZ_64K, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA + SZ_64K, false) & VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, vma->vm_mm, TEST_VA + SZ_64K); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(vma->vm_mm, entry)) + + PAGE_SIZE + 19, 0, SZ_64K - PAGE_SIZE - 19), NULL); + KUNIT_EXPECT_TRUE(test, test_file_fault(vma, TEST_VA + 2 * SZ_64K, false) & VM_FAULT_SIGBUS); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_FILEPAGES), 2L); +} + +static void user4k_file_shared_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 4 * PAGE_SIZE, true); + struct vm_area_struct *small, *native; + struct folio *folio; + unsigned int i; + pte_t entry; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + native = test_file_vma(test, file, true, true, TEST_VA, 4 * PAGE_SIZE, 0); + /* Virtual and file offsets deliberately have different native alignment. */ + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + 3 * PAGE_SIZE, 2 * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start + SZ_4K, false) & + VM_FAULT_ERROR, 0U); + /* Fault-around keeps its 64K window and populates all ready cache folios. */ + for (i = 0; i < 12; i++) { + unsigned long addr = small->vm_start + i * SZ_4K; + struct vm_page_offset pos = vma_page_offset_at(small, addr); + + entry = test_fault_entry(test, small->vm_mm, addr); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + folio = filemap_get_folio(file->f_mapping, pos.index); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, entry), + page_to_phys(folio_file_page(folio, pos.index)) + pos.offset); + KUNIT_EXPECT_EQ(test, *(u8 *)phys_to_virt(pte_phys_mm(small->vm_mm, entry)), + (u8)(0x33 + i)); + folio_put(folio); + } + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA, true) & VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, native->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + memset(phys_to_virt(pte_phys_mm(native->vm_mm, entry)) + 2 * SZ_4K, 0x72, SZ_4K); + entry = test_fault_entry(test, small->vm_mm, small->vm_start); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(small->vm_mm, entry)), + 0x72, SZ_4K), NULL); + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start + SZ_4K, true) & + VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, small->vm_mm, small->vm_start + SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_write(entry)); + memset(phys_to_virt(pte_phys_mm(small->vm_mm, entry)), 0x84, SZ_4K); + entry = test_fault_entry(test, native->vm_mm, TEST_VA); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(native->vm_mm, entry)) + + 3 * SZ_4K, 0x84, SZ_4K), NULL); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, native->vm_mm); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + folio_put(folio); +} + +static void user4k_file_cow_test(struct kunit *test) +{ + unsigned int cold; + + for (cold = 0; cold < 2; cold++) { + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct vm_area_struct *vma; + struct folio *cache; + pte_t entry; + unsigned int i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma(test, file, false, false, TEST_VA + SZ_4K, + 3 * SZ_4K, SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, vma); + if (!cold) + KUNIT_ASSERT_EQ(test, test_file_fault(vma, vma->vm_start, false) & + VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, vma->vm_start + SZ_4K, true) & + VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, vma->vm_mm, vma->vm_start + SZ_4K); + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_TRUE(test, pte_write(entry)); + KUNIT_EXPECT_TRUE(test, folio_test_anon(page_folio(pte_page(entry)))); + /* Private COW can pack into any physical quarter of its backing. */ + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(vma->vm_mm, entry)), + 0x33, SZ_4K), NULL); + memset(phys_to_virt(pte_phys_mm(vma->vm_mm, entry)), 0x91, SZ_4K); + cache = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, cache); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(cache) + i * SZ_4K, + 0x31 + i, SZ_4K), NULL); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(cache), 0); + folio_put(cache); + } +} + +static void user4k_file_eof_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, SZ_4K + 37, false); + struct vm_area_struct *small, *native; + unsigned int i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + small = test_file_vma(test, file, false, true, TEST_VA, PAGE_SIZE, 0); + native = test_file_vma(test, file, true, true, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + SZ_4K, true) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA + 3 * SZ_4K, false) & VM_FAULT_ERROR, 0U); + for (i = 0; i < 4; i++) { + vm_fault_t ret = test_file_fault(small, TEST_VA + i * SZ_4K, false); + + KUNIT_EXPECT_EQ(test, ret & VM_FAULT_ERROR, i < 2 ? 0U : VM_FAULT_SIGBUS); + KUNIT_EXPECT_EQ(test, pte_present(test_fault_entry(test, small->vm_mm, + TEST_VA + i * SZ_4K)), i < 2); + } +} + + +static const struct vm_operations_struct test_filemap_fault_ops = { + .fault = filemap_fault, + .page_mkwrite = filemap_page_mkwrite, +}; + +static void user4k_filemap_fault_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 2 * SZ_4K + 17, true); + struct vm_area_struct *vma; + unsigned int i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma(test, file, false, true, TEST_VA + 3 * SZ_4K, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + /* Exercise generic filemap_fault against an already populated cache. */ + vma->vm_ops = &test_filemap_fault_ops; + KUNIT_ASSERT_EQ(test, test_file_fault(vma, vma->vm_start + SZ_4K, true) & VM_FAULT_ERROR, 0U); + for (i = 0; i < 3; i++) { + pte_t pte = test_fault_entry(test, vma->vm_mm, vma->vm_start + i * SZ_4K); + + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, *(u8 *)phys_to_virt(pte_phys_mm(vma->vm_mm, pte)), (u8)(0x31 + i)); + } + KUNIT_EXPECT_EQ(test, test_file_fault(vma, vma->vm_start + 3 * SZ_4K, false) & + VM_FAULT_ERROR, VM_FAULT_SIGBUS); +} + +static void user4k_file_truncate_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 2 * PAGE_SIZE, true); + struct vm_area_struct *small, *cow, *native; + struct iattr attr = { .ia_valid = ATTR_SIZE, .ia_size = SZ_4K + 37 }; + unsigned int i; + int ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + PAGE_SIZE, SZ_4K); + cow = test_file_vma(test, file, false, false, TEST_VA, PAGE_SIZE, 0); + native = test_file_vma(test, file, true, false, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, cow); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(cow, TEST_VA + 3 * SZ_4K, true) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA, true) & VM_FAULT_ERROR, 0U); + inode_lock(file_inode(file)); + ret = notify_change(&nop_mnt_idmap, file->f_path.dentry, &attr, NULL); + inode_unlock(file_inode(file)); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = 0; i < 4; i++) { + unsigned long addr = small->vm_start + i * SZ_4K; + + if (i) + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, small->vm_mm, addr))); + KUNIT_EXPECT_EQ(test, test_file_fault(small, addr, false) & VM_FAULT_ERROR, + i ? VM_FAULT_SIGBUS : 0U); + } + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, cow->vm_mm, TEST_VA + 3 * SZ_4K))); + KUNIT_EXPECT_EQ(test, test_file_fault(cow, TEST_VA + 3 * SZ_4K, false) & VM_FAULT_ERROR, + VM_FAULT_SIGBUS); + KUNIT_EXPECT_TRUE(test, pte_present(test_fault_entry(test, native->vm_mm, TEST_VA))); + KUNIT_EXPECT_EQ(test, test_file_fault(native, TEST_VA + 3 * SZ_4K, false) & VM_FAULT_ERROR, 0U); +} + +static void user4k_file_invalidate_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 2 * PAGE_SIZE, true); + struct vm_area_struct *vma; + unsigned int i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + 4 * SZ_4K, PAGE_SIZE - SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, vma->vm_start, false) & VM_FAULT_ERROR, 0U); + unmap_mapping_pages(file->f_mapping, 0, 1, false); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_EQ(test, pte_present(test_fault_entry(test, vma->vm_mm, + vma->vm_start + i * SZ_4K)), i != 0); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, vma->vm_start, false) & VM_FAULT_ERROR, 0U); + unmap_mapping_pages(file->f_mapping, 1, 1, false); + for (i = 0; i < 4; i++) + KUNIT_EXPECT_EQ(test, pte_present(test_fault_entry(test, vma->vm_mm, + vma->vm_start + i * SZ_4K)), i == 0); +} + +static void user4k_file_fork_hole_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 2 * PAGE_SIZE, true); + struct vm_area_struct *shared, *private, *cvma; + struct mm_struct *child; + struct inode *inode; + unsigned int i; + pte_t copied; + int ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shared = test_file_vma(test, file, false, true, TEST_VA, 2 * PAGE_SIZE, 0); + private = test_file_vma(test, file, false, false, TEST_VA, 2 * PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, shared); + KUNIT_ASSERT_NOT_NULL(test, private); + KUNIT_ASSERT_EQ(test, test_file_fault(shared, TEST_VA, false) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(private, TEST_VA, false) & VM_FAULT_ERROR, 0U); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, private->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + mmap_read_lock(child); + cvma = find_vma(child, TEST_VA); + mmap_read_unlock(child); + KUNIT_ASSERT_NOT_NULL(test, cvma); + KUNIT_ASSERT_EQ(test, test_file_fault(cvma, TEST_VA + SZ_4K, true) & VM_FAULT_ERROR, 0U); + copied = test_fault_entry(test, child, TEST_VA + SZ_4K); + memset(phys_to_virt(pte_phys_mm(child, copied)), 0x82, SZ_4K); + inode = file_inode(file); + /* A sub-native-page hole zeros shared bytes but retains private COW. */ + ret = vfs_fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, SZ_4K, SZ_4K); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, i_size_read(inode), (loff_t)(2 * PAGE_SIZE)); + for (i = 0; i < 2 * TEST_SLOTS; i++) { + pte_t pte; + + KUNIT_ASSERT_EQ(test, test_file_fault(shared, TEST_VA + i * SZ_4K, false) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, shared->vm_mm, TEST_VA + i * SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(shared->vm_mm, pte)), + i == 1 ? 0 : 0x31 + i, SZ_4K), NULL); + } + KUNIT_EXPECT_EQ(test, pte_phys_mm(child, test_fault_entry(test, child, TEST_VA + SZ_4K)), + pte_phys_mm(child, copied)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(child, copied)), 0x82, SZ_4K), NULL); + /* A full cache-page hole exercises the file interval-tree invalidation. */ + ret = vfs_fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, PAGE_SIZE, PAGE_SIZE); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = TEST_SLOTS; i < 2 * TEST_SLOTS; i++) { + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, shared->vm_mm, TEST_VA + i * SZ_4K))); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, child, TEST_VA + i * SZ_4K))); + } +} + +static void test_zero_cow_granule(struct kunit *test, unsigned int shift) +{ + unsigned long size = 1UL << shift; + struct vm_area_struct *vma = test_fault_vma_granule(test, shift, TEST_VA, PAGE_SIZE); + struct mm_struct *mm; + struct mm_subpage *slot; + pte_t before[4], after; + unsigned int i; + vm_fault_t ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mm = vma->vm_mm; + for (i = 0; i < 4; i++) { + mmap_read_lock(mm); + ret = handle_mm_fault(vma, TEST_VA + i * size, + FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, (vm_fault_t)0); + before[i] = test_fault_entry(test, mm, TEST_VA + i * size); + KUNIT_ASSERT_TRUE(test, pte_present(before[i])); + KUNIT_ASSERT_TRUE(test, is_zero_pfn(pte_pfn(before[i]))); + KUNIT_ASSERT_FALSE(test, pte_write(before[i])); + } + mmap_read_lock(mm); + ret = handle_mm_fault(vma, TEST_VA + size + 7, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, (vm_fault_t)0); + lru_add_drain_all(); + for (i = 0; i < 4; i++) { + after = test_fault_entry(test, mm, TEST_VA + i * size); + if (i != 1) { + KUNIT_EXPECT_TRUE(test, pte_same(before[i], after)); + continue; + } + KUNIT_ASSERT_TRUE(test, pte_write(after)); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, after)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), + (unsigned int)offset_in_page(pte_phys_mm(mm, after))); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 1); + KUNIT_EXPECT_EQ(test, folio_ref_count(mm_subpage_folio(slot)), 2); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0, size), NULL); + mm_subpage_put(slot); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 1UL); +} + +static void user4k_zero_cow_test(struct kunit *test) +{ + test_zero_cow_granule(test, 12); +} + +static void user16k_zero_cow_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative faults require larger native backing"); + return; + } + test_zero_cow_granule(test, 14); +} + +static void test_cow_reuse_granule(struct kunit *test, unsigned int shift) +{ + unsigned long size = 1UL << shift; + unsigned int mode; + + for (mode = 0; mode < 2; mode++) { + struct vm_area_struct *vma = test_fault_vma_granule(test, shift, TEST_VA, PAGE_SIZE); + struct mm_subpage *held = NULL; + struct mm_struct *child; + phys_addr_t original[4]; + unsigned int i; + vm_fault_t fault; + int ret; + pte_t pte; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + for (i = 0; i < 4; i++) { + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + i * size); + original[i] = pte_phys_mm(vma->vm_mm, pte); + memset(phys_to_virt(original[i]), 0xb0 + i, size); + } + if (mode) { + held = mm_subpage_get_from_phys(original[2]); + KUNIT_ASSERT_NOT_NULL(test, held); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, held), 0); + } + child = test_vma_mm_granule(test, shift); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + kunit_release_action(test, user4k_vma_mm_free, child); + for (i = 0; i < 4; i++) { + unsigned long address = TEST_VA + i * size; + + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + pte = test_fault_entry(test, vma->vm_mm, address); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte) == original[i], + !(mode && i == 2)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(vma->vm_mm, pte)), + 0xb0 + i, size), NULL); + memset(phys_to_virt(pte_phys_mm(vma->vm_mm, pte)), 0xd0 + i, size); + } + if (held) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(mm_subpage_phys(held)), + 0xb2, size), NULL); + } +} + +static void user4k_cow_reuse_test(struct kunit *test) +{ + test_cow_reuse_granule(test, 12); +} + +static void user16k_cow_reuse_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative faults require larger native backing"); + return; + } + test_cow_reuse_granule(test, 14); +} + +static void test_fork_cow_granule(struct kunit *test, unsigned int shift) +{ + unsigned long size = 1UL << shift; + struct vm_area_struct *parent = test_fault_vma_granule(test, shift, TEST_VA, 4 * size); + struct mm_struct *child = test_vma_mm_granule(test, shift); + struct vm_area_struct *cvma; + struct mm_subpage *held[4]; + pte_t before[4], after; + unsigned int i, side; + vm_fault_t fault; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, parent); + KUNIT_ASSERT_NOT_NULL(test, child); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + for (i = 0; i < 4; i++) { + before[i] = test_fault_entry(test, parent->vm_mm, TEST_VA + i * size); + held[i] = mm_subpage_get_from_phys(pte_phys_mm(parent->vm_mm, before[i])); + KUNIT_ASSERT_NOT_NULL(test, held[i]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, held[i]), 0); + memset(folio_address(mm_subpage_folio(held[i])) + mm_subpage_offset(held[i]), + 0x31 + i, size); + } + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + mmap_read_lock(child); + fault = handle_mm_fault(cvma, TEST_VA + size + 15, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + lru_add_drain_all(); + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : parent->vm_mm; + + for (i = 0; i < 4; i++) { + struct mm_subpage *slot; + bool changed = side && i == 1; + + after = test_fault_entry(test, mm, TEST_VA + i * size); + KUNIT_EXPECT_EQ(test, !!pte_write(after), changed); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, after) != mm_subpage_phys(held[i]), changed); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, after)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), + (unsigned int)offset_in_page(pte_phys_mm(mm, after))); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0x31 + i, size), NULL); + if (changed) { + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 1); + KUNIT_EXPECT_EQ(test, folio_ref_count(mm_subpage_folio(slot)), 2); + memset(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0x77, size); + } + mm_subpage_put(slot); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 4UL); + } + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(held[i]), i == 1 ? 1 : 2); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(held[i])) + + mm_subpage_offset(held[i]), 0x31 + i, size), NULL); + } + KUNIT_EXPECT_EQ(test, folio_ref_count(mm_subpage_folio(held[0])), 11); +} + +static void user4k_fork_cow_test(struct kunit *test) +{ + test_fork_cow_granule(test, 12); +} + +static void user16k_fork_cow_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative faults require larger native backing"); + return; + } + test_fork_cow_granule(test, 14); +} + +static void user4k_folio_walk_geometry_test(struct kunit *test) +{ + unsigned int shifts[] = { 12, PAGE_SHIFT, 14 }; + unsigned int which; + + for (which = 0; which < ARRAY_SIZE(shifts); which++) { + unsigned int shift = shifts[which], leaf; + struct vm_area_struct *vma; + struct mm_struct *mm; + unsigned long start, size = 1UL << shift; + vm_fault_t fault; + + if (which == 2 && PAGE_SHIFT <= 14) + continue; + mm = shift == PAGE_SHIFT ? mm_alloc() : test_vma_mm_granule(test, shift); + KUNIT_ASSERT_NOT_NULL(test, mm); + if (shift == PAGE_SHIFT) + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + start = pmd_size_mm(mm) - size; + vma = test_fault_vma_in_mm(mm, start, 2 * size); + KUNIT_ASSERT_NOT_NULL(test, vma); + for (leaf = 0; leaf < 2; leaf++) { + unsigned long address = start + leaf * size; + struct folio_walk fw; + struct folio *folio; + struct page *page = NULL; + pte_t expected, walked = __pte(0); + bool zero = false; + + mmap_read_lock(mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + mmap_read_lock(mm); + folio = folio_walk_start(&fw, vma, address, 0); + if (folio) + folio_walk_end(&fw, vma); + mmap_read_unlock(mm); + KUNIT_EXPECT_PTR_EQ(test, folio, NULL); + mmap_read_lock(mm); + folio = folio_walk_start(&fw, vma, address, FW_ZEROPAGE); + if (folio) { + zero = is_zero_folio(folio) && !fw.page && fw.level == FW_LEVEL_PTE; + folio_walk_end(&fw, vma); + } + mmap_read_unlock(mm); + KUNIT_EXPECT_TRUE(test, zero); + + mmap_read_lock(mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + expected = test_fault_entry(test, mm, address); + mmap_read_lock(mm); + folio = folio_walk_start(&fw, vma, address + size - 1, 0); + if (folio) { + page = fw.page; + walked = fw.pte; + folio_walk_end(&fw, vma); + } + mmap_read_unlock(mm); + KUNIT_EXPECT_PTR_EQ(test, page, pfn_to_page(pte_pfn(expected))); + KUNIT_EXPECT_TRUE(test, pte_same(walked, expected)); + KUNIT_EXPECT_PTR_EQ(test, folio, page_folio(pfn_to_page(pte_pfn(expected)))); + } + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +static void user4k_fragment_locked_pin_test(struct kunit *test) +{ + unsigned int shifts[] = { 12, PAGE_SHIFT, 14 }; + unsigned int which; + + for (which = 0; which < ARRAY_SIZE(shifts); which++) { + unsigned int shift = shifts[which]; + struct mm_struct *mm; + struct vm_area_struct *vma, *found; + struct user_page_fragment fragment = { }; + unsigned long address; + int ret; + + if (which == 2 && PAGE_SHIFT <= 14) + continue; + mm = shift == PAGE_SHIFT ? mm_alloc() : test_vma_mm_granule(test, shift); + KUNIT_ASSERT_NOT_NULL(test, mm); + if (shift == PAGE_SHIFT) + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + vma = test_fault_vma_in_mm(mm, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + address = TEST_VA + (1UL << shift) - sizeof(long); + mmap_read_lock(mm); + ret = pin_user_fragment_vma_remote(mm, address, sizeof(long), + FOLL_WRITE | FOLL_NOFAULT, &fragment, &found); + mmap_assert_locked(mm); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + KUNIT_EXPECT_PTR_EQ(test, found, NULL); + mmap_read_lock(mm); + ret = pin_user_fragment_vma_remote(mm, address, sizeof(long), + FOLL_WRITE | FOLL_LONGTERM, &fragment, &found); + mmap_assert_locked(mm); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + KUNIT_EXPECT_PTR_EQ(test, found, NULL); + mmap_read_lock(mm); + ret = pin_user_fragment_vma_remote(mm, address, sizeof(long), + FOLL_WRITE | FOLL_NOWAIT, &fragment, &found); + mmap_assert_locked(mm); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + KUNIT_EXPECT_PTR_EQ(test, found, NULL); + /* Only the first fragment is returned, while retaining the VMA lock. */ + mmap_read_lock(mm); + ret = pin_user_fragment_vma_remote(mm, address, 2 * sizeof(long), + FOLL_WRITE, &fragment, &found); + mmap_assert_locked(mm); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, 1); + if (ret != 1) + continue; + KUNIT_EXPECT_PTR_EQ(test, found, vma); + KUNIT_EXPECT_TRUE(test, fragment.pinned); + KUNIT_EXPECT_EQ(test, fragment.length, (unsigned int)sizeof(long)); + KUNIT_EXPECT_EQ(test, !!fragment.subpage, shift != PAGE_SHIFT); + release_user_fragments(&fragment, 1, false); + /* With backing present, NOFAULT pins the exact same leaf successfully. */ + mmap_read_lock(mm); + ret = pin_user_fragment_vma_remote(mm, address, sizeof(long), + FOLL_WRITE | FOLL_NOFAULT, &fragment, &found); + mmap_assert_locked(mm); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, ret, 1); + if (ret == 1) { + KUNIT_EXPECT_PTR_EQ(test, found, vma); + release_user_fragments(&fragment, 1, true); + } + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +static void user16k_fragment_offset_test(struct kunit *test) +{ + struct vm_area_struct *vma; + struct user_page_fragment fragment = {}; + unsigned long address = TEST_VA + 3 * SZ_16K + SZ_4K + 13; + long ret; + + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "16K alternative fragments require larger native backing"); + return; + } + vma = test_fault_vma_granule(test, 14, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + ret = pin_user_fragments_remote(vma->vm_mm, address, 31, FOLL_WRITE, &fragment, 1); + KUNIT_EXPECT_EQ(test, ret, 1L); + if (ret != 1) + return; + KUNIT_EXPECT_NOT_NULL(test, fragment.subpage); + KUNIT_EXPECT_EQ(test, fragment.offset, 3UL * SZ_16K + SZ_4K + 13); + KUNIT_EXPECT_EQ(test, fragment.length, 31U); + if (fragment.subpage) + KUNIT_EXPECT_EQ(test, mm_subpage_shift(fragment.subpage), 14U); + memset(folio_address(fragment.folio) + fragment.offset, 0x9c, fragment.length); + release_user_fragments(&fragment, 1, true); + { + u8 buf[33]; + + ret = access_remote_vm(vma->vm_mm, address - 1, buf, sizeof(buf), 0); + KUNIT_ASSERT_EQ(test, ret, (long)sizeof(buf)); + KUNIT_EXPECT_EQ(test, buf[0], (u8)0); + KUNIT_EXPECT_EQ(test, buf[32], (u8)0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buf + 1, 0x9c, 31), NULL); + } +} + +static void user4k_anon_exit_test(struct kunit *test) +{ + struct anon_slot_fixture *f = anon_slot_fixture_create(test); + struct mm_subpage *held; + struct anon_vma *root; + struct folio *folio; + struct mm_struct *mm; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, f); + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc_at(f->pool, i * SZ_4K); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(f, i, slot), 0); + } + held = f->slots[0]; + folio = mm_subpage_folio(held); + root = f->vma->anon_vma->root; + mm_subpage_get(held); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, held), 0); + mm_subpage_pool_close(f->pool); + mm_subpage_pool_put(f->pool); + /* Leave all four mapping refs and the tables to the real exit path. */ + mm = f->mm; + memset(f, 0, sizeof(*f)); + mmput(mm); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(held), 0); + KUNIT_EXPECT_EQ(test, atomic_read(&root->refcount), 1); + for (i = 1; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_get_from_phys(mm_subpage_phys(held) + i * SZ_4K); + + KUNIT_EXPECT_PTR_EQ(test, slot, NULL); + if (slot) + mm_subpage_put(slot); + } +} + +#ifdef CONFIG_ARM64_CONTPTE +struct contpte_fixture { + struct mm_struct *mm; + struct folio *folio; + unsigned int nr; +}; + +static void contpte_fixture_free(void *arg) +{ + struct contpte_fixture *f = arg; + struct mmu_gather tlb; + unsigned int i; + + if (f->mm) { + mmap_write_lock(f->mm); + for (i = 0; i <= f->nr; i++) { + unsigned long addr = TEST_VA + i * mm_page_size(f->mm); + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(f->mm, addr, &ptl); + + if (pte) { + pte_clear(f->mm, addr, pte); + pte_unmap_unlock(pte, ptl); + } + } + tlb_gather_mmu_fullmm(&tlb, f->mm); + free_pgd_range(&tlb, 0, 1UL << 48, 0, 0); + tlb_finish_mmu(&tlb); + mmap_write_unlock(f->mm); + mmput(f->mm); + } + if (f->folio) + folio_put(f->folio); +} + +static void test_contpte_granule(struct kunit *test, unsigned int shift) +{ + struct contpte_fixture *f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + struct vm_area_struct vma = {}; + struct test_mm tables = {}; + struct { + bool initial, addresses, hints, gathered, aged, access_all, unfolded; + bool permissions, refolded, holes, full_clear, sentinel; + } ok; + unsigned long size; + phys_addr_t phys; + pte_t base, saved, observed; + pte_t *pte; + spinlock_t *ptl; + unsigned int i; + int access_ret, access_same_ret, access_noop_ret; +#ifdef CONFIG_PERF_EVENTS + unsigned long irqflags; + u64 reported_size; +#endif + + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, contpte_fixture_free, f), 0); + f->mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, f->mm); + if (shift != PAGE_SHIFT) + KUNIT_ASSERT_EQ(test, test_mm_select_granule(f->mm, shift), 0); + f->nr = cont_ptes_mm(f->mm); + size = mm_page_size(f->mm); + f->folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, + get_order(cont_pte_size_mm(f->mm))); + KUNIT_ASSERT_NOT_NULL(test, f->folio); + phys = PFN_PHYS(folio_pfn(f->folio)); + tables.mm = f->mm; + KUNIT_ASSERT_EQ(test, test_map_alias(&tables, 0, TEST_VA, phys), 0); + vma.vm_mm = f->mm; + base = phys_pte_mm(f->mm, phys, __pgprot(TEST_PROT | PTE_RDONLY)); + memset(&ok, 1, sizeof(ok)); + mmap_read_lock(f->mm); + pte = test_mm_lookup(f->mm, TEST_VA, &ptl); + if (!pte) { + mmap_read_unlock(f->mm); + KUNIT_FAIL(test, "missing PTE table"); + return; + } + pte_clear(f->mm, TEST_VA, pte); + /* The adjacent entry must never be included in this contiguous group. */ + saved = pte_mkspecial(base); + set_pte(pte + f->nr, saved); + set_ptes(f->mm, TEST_VA, pte, base, f->nr); + for (i = 0; i < f->nr; i++) { + observed = __ptep_get(pte + i); + ok.initial &= pte_cont(observed); + ok.addresses &= pte_phys_mm(f->mm, observed) == phys + i * size; + ok.hints &= pte_batch_hint(pte + i, observed) == f->nr - i; + } +#ifdef CONFIG_PERF_EVENTS + local_irq_save(irqflags); + reported_size = perf_get_pgtable_size(f->mm, TEST_VA + size); + local_irq_restore(irqflags); +#endif + __set_pte(pte + 3, pte_mkdirty(__ptep_get(pte + 3))); + observed = ptep_get(pte + 1); + ok.gathered &= pte_dirty(observed) && pte_young(observed); + observed = ptep_get_lockless(pte + 2); + ok.gathered &= pte_dirty(observed) && pte_young(observed) && + pte_phys_mm(f->mm, observed) == phys + 2 * size; + clear_young_dirty_ptes(&vma, TEST_VA + 3 * size, pte + 3, 1, + CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY); + for (i = 0; i < f->nr; i++) { + observed = __ptep_get(pte + i); + if (size < PAGE_SIZE) + ok.aged &= !pte_cont(observed) && + pte_young(observed) == (i != 3) && pte_dirty(observed) == (i != 3); + else + ok.aged &= !pte_young(observed) && !pte_dirty(observed) && + pte_cont(observed); + } + observed = pte_mkdirty(pte_mkyoung(__ptep_get(pte + 3))); + access_same_ret = ptep_set_access_flags(&vma, TEST_VA + 3 * size, + pte + 3, observed, 1); + for (i = 0; i < f->nr; i++) { + observed = __ptep_get(pte + i); + ok.access_all &= pte_cont(observed) == (size >= PAGE_SIZE) && pte_young(observed) && + pte_dirty(observed) && !pte_write(observed); + } + access_noop_ret = ptep_set_access_flags(&vma, TEST_VA + 3 * size, + pte + 3, ptep_get(pte + 3), 1); + observed = pte_mkwrite_novma(pte_mkdirty(pte_mkyoung(__ptep_get(pte + 3)))); + access_ret = ptep_set_access_flags(&vma, TEST_VA + 3 * size, pte + 3, + observed, 1); + for (i = 0; i < f->nr; i++) { + observed = __ptep_get(pte + i); + ok.unfolded &= !pte_cont(observed); + ok.permissions &= pte_write(observed) == (i == 3); + } + set_pte_at(f->mm, TEST_VA + 3 * size, pte + 3, + pte_advance_pfn_mm(f->mm, base, 3)); + /* Last-entry installation must fold using the physical byte offset too. */ + set_pte_at(f->mm, TEST_VA + (f->nr - 1) * size, pte + f->nr - 1, + pte_advance_pfn_mm(f->mm, base, f->nr - 1)); + for (i = 0; i < f->nr; i++) + ok.refolded &= pte_cont(__ptep_get(pte + i)); + get_and_clear_full_ptes(f->mm, TEST_VA + 3 * size, pte + 3, 2, 0); + for (i = 0; i < f->nr; i++) { + observed = __ptep_get(pte + i); + ok.holes &= pte_none(observed) == (i == 3 || i == 4); + if (i != 3 && i != 4) + ok.holes &= !pte_cont(observed) && + pte_phys_mm(f->mm, observed) == phys + i * size; + } + set_ptes(f->mm, TEST_VA + 3 * size, pte + 3, + pte_advance_pfn_mm(f->mm, base, 3), 2); + set_pte_at(f->mm, TEST_VA + (f->nr - 1) * size, pte + f->nr - 1, + pte_advance_pfn_mm(f->mm, base, f->nr - 1)); + for (i = 0; i < f->nr; i++) + ok.refolded &= pte_cont(__ptep_get(pte + i)); + clear_full_ptes(f->mm, TEST_VA, pte, f->nr, 0); + for (i = 0; i < f->nr; i++) + ok.full_clear &= pte_none(__ptep_get(pte + i)); + ok.sentinel = pte_same(__ptep_get(pte + f->nr), saved); + pte_unmap_unlock(pte, ptl); + mmap_read_unlock(f->mm); + KUNIT_EXPECT_TRUE(test, ok.initial); + KUNIT_EXPECT_TRUE(test, ok.addresses); + KUNIT_EXPECT_TRUE(test, ok.hints); + KUNIT_EXPECT_TRUE(test, ok.gathered); + KUNIT_EXPECT_TRUE(test, ok.aged); + KUNIT_EXPECT_TRUE(test, ok.access_all); + KUNIT_EXPECT_EQ(test, access_same_ret, 1); + KUNIT_EXPECT_EQ(test, access_noop_ret, 0); + KUNIT_EXPECT_EQ(test, access_ret, 1); + KUNIT_EXPECT_TRUE(test, ok.unfolded); + KUNIT_EXPECT_TRUE(test, ok.permissions); + KUNIT_EXPECT_TRUE(test, ok.refolded); + KUNIT_EXPECT_TRUE(test, ok.holes); + KUNIT_EXPECT_TRUE(test, ok.full_clear); + KUNIT_EXPECT_TRUE(test, ok.sentinel); +#ifdef CONFIG_PERF_EVENTS + KUNIT_EXPECT_EQ(test, reported_size, (u64)cont_pte_size_mm(f->mm)); +#endif +} + +static void user4k_contpte_test(struct kunit *test) +{ + test_contpte_granule(test, 12); +} + +static void user16k_contpte_test(struct kunit *test) +{ + if (PAGE_SHIFT <= 14) { + kunit_skip(test, "requires native backing larger than 16K"); + return; + } + test_contpte_granule(test, 14); +} + +static void native_contpte_test(struct kunit *test) +{ + test_contpte_granule(test, PAGE_SHIFT); +} +#endif + +struct usercopy_worker { + struct test_mm *ctx[2]; + struct completion done; + unsigned int id; + unsigned int checks; + unsigned int idmaps; + int error; + u64 tcr; +}; + +static int test_usercopy_worker(void *arg) +{ + struct usercopy_worker *worker = arg; + u64 kernel_ttbr = read_sysreg(ttbr1_el1) & ~TTBRx_EL1_ASID_MASK; + unsigned int round, i, byte; + bool borrowed = false; + struct test_mm *ctx = NULL; + + for (round = 0; round < 16; round++) { + unsigned int small = (round + worker->id) & 1; + unsigned int cpu = cpumask_first(cpu_online_mask); + + ctx = worker->ctx[small]; + kthread_use_mm(ctx->mm); + borrowed = true; + if ((round + worker->id) & 2) { + unsigned int next = cpumask_next(cpu, cpu_online_mask); + + if (next < nr_cpu_ids) + cpu = next; + } + if (set_cpus_allowed_ptr(current, cpumask_of(cpu))) { + worker->error = 1; + break; + } + /* Exercise a real scheduler switch while borrowing this mm. */ + schedule_timeout_uninterruptible(1); + if (small && round < 4) { + unsigned long flags; + bool native; + + preempt_disable(); + flags = local_daif_save(); + if (round & 2) + cpu_install_ttbr0(phys_to_ttbr(__pa_symbol(idmap_pg_dir)), + TCR_T0SZ(IDMAP_VA_BITS)); + else + cpu_install_idmap(); + native = (read_sysreg(tcr_el1) & (TCR_EL1_TG0_MASK | TCR_EL1_T0SZ_MASK)) == + (TCR_TG0_16K | TCR_T0SZ(IDMAP_VA_BITS)); + cpu_uninstall_idmap(); + local_daif_restore(flags); + preempt_enable(); + if (!native) { + worker->error = 6; + break; + } + worker->idmaps++; + } + worker->tcr = read_sysreg(tcr_el1); + if ((worker->tcr & TCR_EL1_TG0_MASK) != (small ? TCR_TG0_4K : TCR_TG0_16K) || + (worker->tcr & TCR_EL1_T0SZ_MASK) != TCR_T0SZ(48) || + (worker->tcr & TCR_EL1_TG1_MASK) != TCR_TG1_16K || + (read_sysreg(ttbr1_el1) & ~TTBRx_EL1_ASID_MASK) != kernel_ttbr) { + worker->error = 2; + break; + } + for (i = 0; i < 4; i++) { + unsigned long offset = 64 + worker->id * sizeof(u64); + void __user *address = (void __user *)ctx->address[i]; + u64 value = 0xa5000000 | (worker->id << 16) | (round << 8) | i; + u64 *backing = phys_to_virt(ctx->phys[i]) + offset; + unsigned long read_left, write_left; + u8 data[8]; + + /* No VMA/fault path may be entered for these private mappings. */ + pagefault_disable(); + read_left = copy_from_user(data, address, sizeof(data)); + write_left = copy_to_user(address + offset, &value, sizeof(value)); + pagefault_enable(); + if (read_left != (i == 2 ? sizeof(data) : 0) || + write_left != (i == 0 || i == 2 ? sizeof(value) : 0)) { + worker->error = 3; + goto out; + } + if (i != 2) { + for (byte = 0; byte < sizeof(data); byte++) { + if (data[byte] != (small ? 0x40 : 0x80) + i) { + worker->error = 4; + goto out; + } + } + } + if (READ_ONCE(*backing) != (i == 0 || i == 2 ? 0 : value)) { + worker->error = 5; + goto out; + } + worker->checks++; + } + kthread_unuse_mm(ctx->mm); + borrowed = false; + } +out: + if (borrowed) + kthread_unuse_mm(ctx->mm); + complete(&worker->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void test_stop_kthread(void *arg) +{ + kthread_stop(arg); +} + +struct cow_reader { + struct mm_struct *parent; + struct mm_struct *child; + struct completion ready; + struct completion copied; + struct completion done; + u8 observed[8]; + unsigned long read_left[8], write_left[8]; + int error; + unsigned int checks; +}; + +static int test_cow_reader(void *arg) +{ + struct cow_reader *r = arg; + unsigned int i, side; + + kthread_use_mm(r->child); + /* Resolve fork's old PTEs normally, then cache pre-COW translations. */ + for (i = 0; i < 4; i++) { + u8 value = 0; + unsigned long left; + + left = copy_from_user(&value, (void __user *)(TEST_VA + i * SZ_4K + 64), 1); + if (left || value != 0x41 + i) + r->error = 1; + } + complete(&r->ready); + if (!wait_for_completion_timeout(&r->copied, 10 * HZ)) { + r->error = 2; + goto out; + } + for (side = 0; side < 2; side++) { + if (side) { + kthread_unuse_mm(r->child); + kthread_use_mm(r->parent); + } + for (i = 0; i < 4; i++) { + void __user *addr = (void __user *)(TEST_VA + i * SZ_4K + 64); + u8 value = 0, changed = 0x99; + unsigned long read_left, write_left; + + pagefault_disable(); + read_left = copy_from_user(&value, addr, 1); + write_left = copy_to_user(addr, &changed, 1); + pagefault_enable(); + r->observed[side * 4 + i] = value; + r->read_left[side * 4 + i] = read_left; + r->write_left[side * 4 + i] = write_left; + if (read_left || value != 0x41 + i || + write_left != (side == 0 && i == 1 ? 0 : 1)) + r->error = 3; + r->checks++; + } + } + kthread_unuse_mm(r->parent); + goto done; +out: + kthread_unuse_mm(r->child); +done: + complete(&r->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void user4k_fault_uaccess_test(struct kunit *test) +{ + struct vm_area_struct *parent, *child_vma; + struct mm_struct *child; + struct cow_reader *reader; + struct task_struct *task; + unsigned int i, cpu; + vm_fault_t fault; + int ret; + u64 tcr = read_sysreg(tcr_el1); + + if (!user4k_test_enabled || !system_supports_4kb_granule() || + system_supports_cnp() || cpus_have_final_cap(ARM64_HAS_S1PIE) || + cpus_have_final_cap(ARM64_HAS_S1POE) || + (tcr & (TCR_EL1_DS | TCR_EL1_A1 | TCR_EL1_T0SZ_MASK)) != + (TCR_EL1_A1 | TCR_T0SZ(48))) { + kunit_skip(test, "requires opt-in and compatible 4K TTBR0 controls"); + return; + } + parent = test_fault_vma(test, TEST_VA, PAGE_SIZE); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, parent); + KUNIT_ASSERT_NOT_NULL(test, child); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + for (i = 0; i < 4; i++) { + pte_t entry = test_fault_entry(test, parent->vm_mm, TEST_VA + i * SZ_4K); + + memset(phys_to_virt(pte_phys_mm(parent->vm_mm, entry)), 0x41 + i, SZ_4K); + } + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + child_vma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, child_vma); + reader = kunit_kzalloc(test, sizeof(*reader), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, reader); + reader->parent = parent->vm_mm; + reader->child = child; + init_completion(&reader->ready); + init_completion(&reader->copied); + init_completion(&reader->done); + task = kthread_create(test_cow_reader, reader, "user4k-cow-reader"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, task); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_stop_kthread, task), 0); + cpu = cpumask_last(cpu_online_mask); + kthread_bind(task, cpu); + wake_up_process(task); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&reader->ready, 10 * HZ), 0UL); + mmap_read_lock(child); + fault = handle_mm_fault(child_vma, TEST_VA + SZ_4K, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + complete(&reader->copied); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&reader->done, 10 * HZ), 0UL); + if (reader->error) + for (i = 0; i < 8; i++) + kunit_info(test, "access %u value=%#x read_left=%lu write_left=%lu\n", + i, reader->observed[i], reader->read_left[i], reader->write_left[i]); + KUNIT_EXPECT_EQ(test, reader->error, 0); + KUNIT_EXPECT_EQ(test, reader->checks, 8U); + { + pte_t entry = test_fault_entry(test, child, TEST_VA + SZ_4K); + + KUNIT_EXPECT_EQ(test, *(u8 *)(phys_to_virt(pte_phys_mm(child, entry)) + 64), (u8)0x99); + entry = test_fault_entry(test, parent->vm_mm, TEST_VA + SZ_4K); + KUNIT_EXPECT_EQ(test, *(u8 *)(phys_to_virt(pte_phys_mm(parent->vm_mm, entry)) + 64), (u8)0x42); + } +} + +static void user4k_context_uaccess_test(struct kunit *test) +{ + struct test_mm *ctx[2]; + struct usercopy_worker *workers; + unsigned int i; + u64 tcr = read_sysreg(tcr_el1); + + if (!user4k_test_enabled) { + kunit_skip(test, "requires arm64.user4k_test=1 in a disposable VM"); + return; + } + if (!system_supports_4kb_granule() || system_supports_cnp() || + cpus_have_final_cap(ARM64_HAS_S1PIE) || cpus_have_final_cap(ARM64_HAS_S1POE) || + (tcr & (TCR_EL1_DS | TCR_EL1_A1 | TCR_EL1_T0SZ_MASK)) != + (TCR_EL1_A1 | TCR_T0SZ(48))) { + kunit_skip(test, "requires 4K, no CnP/PIE/POE, DS=0, A1=1 and 48-bit TTBR0"); + return; + } + ctx[0] = test_usercopy_mm(test, false); + KUNIT_ASSERT_NOT_NULL(test, ctx[0]); + ctx[1] = test_usercopy_mm(test, true); + KUNIT_ASSERT_NOT_NULL(test, ctx[1]); + workers = kunit_kcalloc(test, 2, sizeof(*workers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, workers); + for (i = 0; i < 2; i++) { + struct task_struct *task; + + workers[i].ctx[0] = ctx[0]; + workers[i].ctx[1] = ctx[1]; + workers[i].id = i; + init_completion(&workers[i].done); + task = kthread_run(test_usercopy_worker, &workers[i], "user4k-copy-%u", i); + KUNIT_ASSERT_FALSE(test, IS_ERR(task)); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_stop_kthread, task), 0); + } + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].done, 10 * HZ), 0UL); + KUNIT_EXPECT_EQ_MSG(test, workers[i].error, 0, "worker %u TCR=%#llx", i, workers[i].tcr); + KUNIT_EXPECT_EQ(test, workers[i].checks, 64U); + KUNIT_EXPECT_EQ(test, workers[i].idmaps, 2U); + } + kunit_info(test, "two migrating kthreads, 32 borrowed-mm switches, %s PAN\n", + system_uses_ttbr0_pan() ? "software" : "hardware"); +} + +struct zap_reader { + struct mm_struct *mm; + struct task_struct *task; + struct completion ready; + struct completion proceed; + struct completion done; + unsigned int cpu; + unsigned int checks; + bool zapped; + int error; +}; + +static int test_zap_reader(void *arg) +{ + struct zap_reader *reader = arg; + unsigned int phase, i; + + reader->error = set_cpus_allowed_ptr(current, cpumask_of(reader->cpu)); + if (reader->error) { + complete(&reader->ready); + goto out; + } + kthread_use_mm(reader->mm); + for (phase = 0; phase < 2; phase++) { + if (phase) { + complete(&reader->ready); + wait_for_completion(&reader->proceed); + if (!reader->zapped) + break; + } + for (i = 0; i < 4; i++) { + u64 value = 0, expected = 0xa5a50000UL + i; + unsigned long left; + + pagefault_disable(); + left = copy_from_user(&value, + (void __user *)((unsigned long)TEST_VA + i * SZ_4K), sizeof(value)); + pagefault_enable(); + if (phase && i == 1) { + if (!left) + reader->error = -EUCLEAN; + } else if (left || value != expected) { + reader->error = -EFAULT; + } + reader->checks++; + } + } + kthread_unuse_mm(reader->mm); +out: + complete(&reader->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void test_stop_zap_reader(void *arg) +{ + struct zap_reader *reader = arg; + + complete(&reader->proceed); + kthread_stop(reader->task); +} + +static void user4k_anon_remote_zap_test(struct kunit *test) +{ + struct anon_slot_fixture *f; + struct zap_reader *readers; + unsigned int i, cpu; + u64 tcr = read_sysreg(tcr_el1); + + if (!user4k_test_enabled || num_online_cpus() < 2) { + kunit_skip(test, "requires arm64.user4k_test=1 and two CPUs in a disposable VM"); + return; + } + if (!system_supports_4kb_granule() || system_supports_cnp() || + cpus_have_final_cap(ARM64_HAS_S1PIE) || cpus_have_final_cap(ARM64_HAS_S1POE) || + (tcr & (TCR_EL1_DS | TCR_EL1_A1 | TCR_EL1_T0SZ_MASK)) != + (TCR_EL1_A1 | TCR_T0SZ(48))) { + kunit_skip(test, "requires 4K, no CnP/PIE/POE, DS=0, A1=1 and 48-bit TTBR0"); + return; + } + f = anon_slot_fixture_create(test); + KUNIT_ASSERT_NOT_NULL(test, f); + for (i = 0; i < 4; i++) { + struct mm_subpage *slot = mm_subpage_alloc_at(f->pool, i * SZ_4K); + u64 value = 0xa5a50000UL + i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + memcpy(folio_address(mm_subpage_folio(slot)) + mm_subpage_offset(slot), + &value, sizeof(value)); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(f, i, slot), 0); + } + readers = kunit_kcalloc(test, 2, sizeof(*readers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, readers); + cpu = cpumask_first(cpu_online_mask); + for (i = 0; i < 2; i++) { + readers[i].mm = f->mm; + readers[i].cpu = cpu; + init_completion(&readers[i].ready); + init_completion(&readers[i].proceed); + init_completion(&readers[i].done); + readers[i].task = kthread_run(test_zap_reader, &readers[i], "user4k-zap-%u", i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, readers[i].task); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_stop_zap_reader, + &readers[i]), 0); + cpu = cpumask_next(cpu, cpu_online_mask); + } + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&readers[i].ready, 10 * HZ), 0UL); + KUNIT_ASSERT_EQ(test, readers[i].error, 0); + } + /* Keep an explicit non-mapping reference for fixture cleanup. */ + mm_subpage_get(f->slots[1]); + mmap_read_lock(f->mm); + zap_vma_range(f->vma, TEST_VA + SZ_4K, SZ_4K); + f->mapped[1] = false; + mmap_read_unlock(f->mm); + for (i = 0; i < 2; i++) { + readers[i].zapped = true; + complete(&readers[i].proceed); + } + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&readers[i].done, 10 * HZ), 0UL); + KUNIT_EXPECT_EQ(test, readers[i].error, 0); + KUNIT_EXPECT_EQ(test, readers[i].checks, 8U); + } +} + +static u64 *test_leaf(struct test_tables *tables, unsigned long va) +{ + return &tables->level[3][test_index(va, tables->shift, 3)]; +} + +/* Explicitly 4K, without the native PAGE_SHIFT or a granule/level hint. */ +static void test_invalidate_4k(unsigned long va, u64 ttbr1) +{ + u64 operand = (va >> USER4K_SHIFT) | (ttbr1 & GENMASK_ULL(63, 48)); + + dsb(ishst); + __tlbi(vae1, operand); + dsb(ish); + isb(); +} + +static void user4k_translation_test(struct kunit *test) +{ + struct test_tables small, native; + struct test_result *result; + unsigned long flags; + void *data, *other; + phys_addr_t data_pa, other_pa; + u64 old_tcr, old_ttbr0, old_ttbr1, old_par, tcr4k, tcr16k; + unsigned int round, slot; + + if (!user4k_test_enabled) { + kunit_skip(test, "requires arm64.user4k_test=1 in a disposable VM"); + return; + } + if (!system_supports_4kb_granule() || system_supports_cnp() || + cpus_have_final_cap(ARM64_HAS_S1PIE) || + cpus_have_final_cap(ARM64_HAS_S1POE)) { + kunit_skip(test, "requires 4K, no CnP, and direct PTE permissions"); + return; + } + + KUNIT_ASSERT_EQ(test, PAGE_SHIFT, 14); + KUNIT_ASSERT_EQ(test, test_alloc_tables(test, &small, USER4K_SHIFT), 0); + KUNIT_ASSERT_EQ(test, test_alloc_tables(test, &native, PAGE_SHIFT), 0); + data = test_alloc_page(test); + other = test_alloc_page(test); + KUNIT_ASSERT_NOT_NULL(test, data); + KUNIT_ASSERT_NOT_NULL(test, other); + data_pa = virt_to_phys(data); + other_pa = virt_to_phys(other); + result = kunit_kcalloc(test, TEST_ROUNDS, sizeof(*result), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, result); + + /* One native mapping and four independently described subpages. */ + *test_leaf(&native, TEST_VA) = other_pa | TEST_PROT; + *test_leaf(&small, TEST_VA) = data_pa | TEST_PROT | PTE_RDONLY; + *test_leaf(&small, TEST_VA + USER4K_SIZE) = + (data_pa + USER4K_SIZE) | TEST_PROT; + /* Slot 2 deliberately unmapped. Slot 3 aliases slot 0, writable. */ + *test_leaf(&small, TEST_VA + 3 * USER4K_SIZE) = data_pa | TEST_PROT; + + preempt_disable(); + flags = local_daif_save(); + old_tcr = read_sysreg(tcr_el1); + old_ttbr0 = read_sysreg(ttbr0_el1); + old_ttbr1 = read_sysreg(ttbr1_el1); + old_par = read_sysreg_par(); + /* DS affects both address ranges: never toggle it under a live TTBR1. */ + if ((old_tcr & (TCR_EL1_DS | TCR_EL1_TG1_MASK | TCR_EL1_A1)) != + (TCR_TG1_16K | TCR_EL1_A1)) { + local_daif_restore(flags); + preempt_enable(); + kunit_skip(test, "requires 16K TTBR1, DS=0, and A1=1"); + return; + } + tcr4k = (old_tcr & ~(TCR_EL1_TG0_MASK | TCR_EL1_T0SZ_MASK | + TCR_EL1_EPD0)) | TCR_TG0_4K | TCR_T0SZ(48); + tcr16k = (tcr4k & ~TCR_EL1_TG0_MASK) | TCR_TG0_16K; + + /* No allocations, logging, KUnit assertions or user access in this window. */ + for (round = 0; round < TEST_ROUNDS; round++) { + struct test_result *r = &result[round]; + + test_switch_ttbr0(virt_to_phys(native.level[0]), tcr16k); + for (slot = 0; slot < PROBE_SLOTS; slot++) + r->native[slot] = test_translate(TEST_VA + slot * USER4K_SIZE, false); + test_switch_ttbr0(virt_to_phys(small.level[0]), tcr4k); + r->tcr = read_sysreg(tcr_el1); + r->ttbr1 = read_sysreg(ttbr1_el1); + for (slot = 0; slot < PROBE_SLOTS; slot++) { + unsigned long va = TEST_VA + slot * USER4K_SIZE; + + r->read[slot] = test_translate(va, false); + r->write[slot] = test_translate(va, true); + } + + /* Break-before-make one 4K PTE and invalidate exactly that VA. */ + WRITE_ONCE(*test_leaf(&small, TEST_VA + USER4K_SIZE), 0); + test_invalidate_4k(TEST_VA + USER4K_SIZE, old_ttbr1); + WRITE_ONCE(*test_leaf(&small, TEST_VA + USER4K_SIZE), + (other_pa + USER4K_SIZE) | TEST_PROT | PTE_RDONLY); + dsb(ishst); + isb(); + for (slot = 0; slot < PROBE_SLOTS; slot++) + r->replaced[slot] = test_translate(TEST_VA + slot * USER4K_SIZE, true); + r->replaced_read = test_translate(TEST_VA + USER4K_SIZE, false); + asm volatile("at s1e1r, %0" : : "r" (data) : "memory"); + isb(); + r->kernel = read_sysreg_par(); + + test_switch_ttbr0(old_ttbr0, old_tcr); + r->restored_tcr = read_sysreg(tcr_el1); + r->restored_ttbr0 = read_sysreg(ttbr0_el1); + /* Private tables are inactive and their translations have been flushed. */ + WRITE_ONCE(*test_leaf(&small, TEST_VA + USER4K_SIZE), + (data_pa + USER4K_SIZE) | TEST_PROT); + } + write_sysreg(old_par, par_el1); + isb(); + local_daif_restore(flags); + preempt_enable(); + + for (round = 0; round < TEST_ROUNDS; round++) { + struct test_result *r = &result[round]; + + KUNIT_EXPECT_EQ(test, r->tcr, tcr4k); + KUNIT_EXPECT_EQ(test, r->ttbr1, old_ttbr1); + KUNIT_EXPECT_EQ(test, r->restored_tcr, old_tcr); + KUNIT_EXPECT_EQ(test, r->restored_ttbr0, old_ttbr0); + KUNIT_EXPECT_EQ(test, r->kernel & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, r->kernel & SYS_PAR_EL1_PA, data_pa); + KUNIT_EXPECT_EQ(test, r->replaced_read & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, r->replaced_read & SYS_PAR_EL1_PA, + other_pa + USER4K_SIZE); + for (slot = 0; slot < PROBE_SLOTS; slot++) { + KUNIT_EXPECT_EQ(test, r->native[slot] & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, r->native[slot] & SYS_PAR_EL1_PA, + other_pa + slot * USER4K_SIZE); + if (slot == 2) { + KUNIT_EXPECT_EQ(test, + FIELD_GET(SYS_PAR_EL1_FST, r->read[slot]), 7ULL); + KUNIT_EXPECT_EQ(test, r->read[slot] & SYS_PAR_EL1_F, 1ULL); + KUNIT_EXPECT_EQ(test, r->write[slot] & SYS_PAR_EL1_F, 1ULL); + KUNIT_EXPECT_EQ(test, r->replaced[slot] & SYS_PAR_EL1_F, 1ULL); + continue; + } + KUNIT_EXPECT_EQ(test, r->read[slot] & SYS_PAR_EL1_F, 0ULL); + KUNIT_EXPECT_EQ(test, r->read[slot] & SYS_PAR_EL1_PA, + data_pa + (slot == 1 ? USER4K_SIZE : 0)); + KUNIT_EXPECT_EQ(test, r->write[slot] & SYS_PAR_EL1_F, + slot == 0 ? 1ULL : 0ULL); + KUNIT_EXPECT_EQ(test, r->replaced[slot] & SYS_PAR_EL1_F, + slot == 3 ? 0ULL : 1ULL); + } + } + kunit_info(test, "16K kernel, 4K EL0 translation/permissions; %u switch cycles\n", + TEST_ROUNDS); +} + +static struct folio *test_fault_folio(struct mm_struct *mm, unsigned long address) +{ + spinlock_t *ptl; + pte_t *pte = test_mm_lookup(mm, address, &ptl); + struct folio *folio = NULL; + + if (pte) { + pte_t entry = ptep_get(pte); + + if (pte_present(entry) && !pte_special(entry)) { + folio = page_folio(pfn_to_page(pte_pfn(entry))); + folio_get(folio); + } + pte_unmap_unlock(pte, ptl); + } + return folio; +} + +static bool test_reclaim_swapcache(swp_entry_t entry) +{ + unsigned int tries; + + for (tries = 0; tries < 100; tries++) { + struct folio *folio = swap_cache_get_folio(entry); + LIST_HEAD(folios); + + if (!folio) + return true; + folio_wait_writeback(folio); + lru_add_drain_all(); + if (folio_isolate_lru(folio)) + list_add(&folio->lru, &folios); + folio_put(folio); + /* Only the swap cache and reclaim isolation may retain this folio. */ + reclaim_pages(&folios); + schedule_timeout_uninterruptible(1); + } + return false; +} + +static void test_swap_tags_set(struct folio *folio) +{ +#ifdef CONFIG_ARM64_MTE + unsigned int offset; + + if (!system_supports_mte()) + return; + if (try_page_mte_tagging(&folio->page)) + mte_clear_page_tags(folio_address(folio)); + for (offset = 0; offset < PAGE_SIZE; offset += MTE_GRANULE_SIZE) { + unsigned long tag = (1UL + offset / SZ_4K) << MTE_TAG_SHIFT; + + asm volatile(".arch_extension memtag\n\tstg %0, [%1]" + : : "r" (tag), "r" (folio_address(folio) + offset) : "memory"); + } + set_page_mte_tagged(&folio->page); +#endif +} + +static bool test_swap_tags_match_at(struct folio *folio, unsigned int start, + unsigned int slot) +{ +#ifdef CONFIG_ARM64_MTE + unsigned int offset; + + if (!system_supports_mte()) + return true; + if (!page_mte_tagged(&folio->page)) + return false; + for (offset = start; offset < start + SZ_4K; offset += MTE_GRANULE_SIZE) { + unsigned long tag = 0; + + asm volatile(".arch_extension memtag\n\tldg %0, [%1]" + : "+r" (tag) : "r" (folio_address(folio) + offset) : "memory"); + if (((tag >> MTE_TAG_SHIFT) & 0xf) != ((slot + 1) & 0xf)) + return false; + } +#endif + return true; +} + +static bool test_swap_tags_match(struct folio *folio, unsigned int slot) +{ + return test_swap_tags_match_at(folio, slot * SZ_4K, slot); +} + +struct fragment_test_pins { + struct user_page_fragment fragments[2 * TEST_SLOTS]; + unsigned long count; +}; + +static void fragment_test_unpin(void *arg) +{ + struct fragment_test_pins *pins = arg; + + release_user_fragments(pins->fragments, pins->count, true); + pins->count = 0; +} + +static long fragment_test_pin(struct fragment_test_pins *pins, struct mm_struct *mm, + unsigned long start, unsigned long length, unsigned int flags, + unsigned long capacity) +{ + long ret; + + fragment_test_unpin(pins); + ret = pin_user_fragments_remote(mm, start, length, flags, pins->fragments, capacity); + pins->count = ret > 0 ? ret : 0; + return ret; +} + +static void user4k_fragment_pin_test(struct kunit *test) +{ + unsigned int native; + + for (native = 0; native < 2; native++) { + struct fragment_test_pins *pins; + struct vm_area_struct *vma; + struct mm_struct *mm, *child; + unsigned long size, total = 0; + unsigned int i, nr; + long ret; + + if (native) { + mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + vma = test_fault_vma_in_mm(mm, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + } else { + vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + mm = vma->vm_mm; + } + size = mm_page_size(mm); + nr = PAGE_SIZE / size; + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + ret = fragment_test_pin(pins, mm, TEST_VA, 1, FOLL_WRITE | FOLL_NOFAULT, ARRAY_SIZE(pins->fragments)); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + ret = fragment_test_pin(pins, mm, TEST_VA + 123, PAGE_SIZE - 200, FOLL_WRITE, ARRAY_SIZE(pins->fragments)); + KUNIT_ASSERT_EQ(test, ret, (long)nr); + for (i = 0; i < nr; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + unsigned int offset = i ? i * size : 123; + unsigned int length = size - (!i ? 123 : 0) - (i == nr - 1 ? 77 : 0); + + KUNIT_EXPECT_EQ(test, f->offset, (unsigned long)offset); + KUNIT_EXPECT_EQ(test, f->length, length); + KUNIT_EXPECT_EQ(test, !!f->subpage, !native); + memset(folio_address(f->folio) + f->offset, 0x71 + i, f->length); + total += f->length; + } + KUNIT_EXPECT_EQ(test, total, PAGE_SIZE - 200); + fragment_test_unpin(pins); + child = native ? mm_alloc() : test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + if (native) + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, child), 0); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0L); + /* Ordinary references retain shared COW backing, as native FOLL_GET does. */ + ret = get_user_fragments_remote(child, TEST_VA + 123, PAGE_SIZE - 200, + 0, pins->fragments, ARRAY_SIZE(pins->fragments)); + pins->count = ret > 0 ? ret : 0; + KUNIT_ASSERT_EQ(test, ret, (long)nr); + for (i = 0; i < nr; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + pte_t parent_pte = test_fault_entry(test, mm, TEST_VA + i * size); + pte_t child_pte = test_fault_entry(test, child, TEST_VA + i * size); + + KUNIT_EXPECT_FALSE(test, f->pinned); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, parent_pte), pte_phys_mm(child, child_pte)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x71 + i, f->length), NULL); + } + fragment_test_unpin(pins); + /* Read pins must unshare COW mappings without enabling PTE writes. */ + ret = fragment_test_pin(pins, child, TEST_VA + 123, PAGE_SIZE - 200, FOLL_NOFAULT, ARRAY_SIZE(pins->fragments)); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + ret = fragment_test_pin(pins, child, TEST_VA + 123, PAGE_SIZE - 200, 0, ARRAY_SIZE(pins->fragments)); + KUNIT_ASSERT_EQ(test, ret, (long)nr); + for (i = 0; i < nr; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + pte_t pte = test_fault_entry(test, child, TEST_VA + i * size); + + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x71 + i, f->length), NULL); + if (!native) { + pte_t parent_pte = test_fault_entry(test, mm, TEST_VA + i * size); + + KUNIT_EXPECT_TRUE(test, mm_subpage_is_exclusive(f->subpage)); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, pte), + pte_phys_mm(mm, parent_pte)); + if (i) + KUNIT_EXPECT_NE(test, mm_subpage_phys(f->subpage), + mm_subpage_phys(pins->fragments[0].subpage)); + } + } + fragment_test_unpin(pins); + /* Return a valid partial result when the byte range reaches an unmapped VMA. */ + ret = fragment_test_pin(pins, mm, TEST_VA, PAGE_SIZE + size, FOLL_WRITE, ARRAY_SIZE(pins->fragments)); + KUNIT_EXPECT_EQ(test, ret, (long)nr); + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, mm, TEST_VA, PAGE_SIZE, FOLL_WRITE, 1); + KUNIT_EXPECT_EQ(test, ret, 1L); + KUNIT_EXPECT_EQ(test, pins->fragments[0].length, (unsigned int)size); + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, mm, TEST_VA, size, FOLL_LONGTERM, ARRAY_SIZE(pins->fragments)); + KUNIT_EXPECT_EQ(test, ret, 1L); + ret = fragment_test_pin(pins, mm, ULONG_MAX - 15, ULONG_MAX, 0, ARRAY_SIZE(pins->fragments)); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +#ifdef CONFIG_BLOCK +struct bio_fragment_test { + struct bio *bio; + struct bio *clone; + struct user_page_fragment held[TEST_SLOTS]; + unsigned int count; +}; + +static void bio_fragment_test_free(void *arg) +{ + struct bio_fragment_test *state = arg; + + if (state->clone) + bio_put(state->clone); + if (state->bio) { + bio_release_pages(state->bio, false); + bio_put(state->bio); + } + release_user_fragments(state->held, state->count, false); +} + +static void user4k_bio_fragments_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct bio_fragment_test *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct iov_iter iter; + unsigned int i, bytes = round_down(PAGE_SIZE - 200, 512); + long nr; + int ret; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bio fixture requires hardware opt-in"); + return; + } + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, bio_fragment_test_free, state), 0); + /* Populate real writable anonymous mappings and retain independent GETs. */ + nr = get_user_fragments_remote(vma->vm_mm, TEST_VA, PAGE_SIZE, FOLL_WRITE, + state->held, ARRAY_SIZE(state->held)); + state->count = nr > 0 ? nr : 0; + KUNIT_ASSERT_EQ(test, nr, (long)TEST_SLOTS); + state->bio = bio_alloc(NULL, 8, REQ_OP_READ, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->bio); + iov_iter_ubuf(&iter, ITER_DEST, (void __user *)(TEST_VA + 123), PAGE_SIZE - 200); + kthread_use_mm(vma->vm_mm); + ret = bio_iov_vecs_to_alloc(&iter, BIO_MAX_VECS); + kthread_unuse_mm(vma->vm_mm); + KUNIT_EXPECT_EQ(test, ret, (int)TEST_SLOTS); + kthread_use_mm(vma->vm_mm); + ret = bio_iov_iter_get_pages(state->bio, &iter, 511); + kthread_unuse_mm(vma->vm_mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, state->bio->bi_iter.bi_size, bytes); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), PAGE_SIZE - 200 - bytes); + KUNIT_EXPECT_EQ(test, state->bio->bi_nr_user_fragments, (int)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, state->bio->bi_vcnt, 1); + KUNIT_EXPECT_EQ(test, state->bio->bi_io_vec[0].bv_offset, 123U); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(state->held[i].subpage), 1U); + state->clone = bio_alloc_clone(NULL, state->bio, GFP_KERNEL, &fs_bio_set); + KUNIT_ASSERT_NOT_NULL(test, state->clone); + KUNIT_EXPECT_PTR_EQ(test, state->clone->bi_user_fragments, NULL); + KUNIT_EXPECT_PTR_EQ(test, state->clone->bi_io_vec, state->bio->bi_io_vec); + /* Completion ownership must outlive all originating VMAs. */ + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(state->held[i].subpage), 1U); + bio_put(state->clone); + state->clone = NULL; + bio_release_pages(state->bio, false); + KUNIT_EXPECT_PTR_EQ(test, state->bio->bi_user_fragments, NULL); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(state->held[i].subpage), 0U); +} + +static void user4k_bio_trim_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, SZ_4K); + struct bio_fragment_test *state = kunit_kzalloc(test, sizeof(*state), GFP_KERNEL); + struct iov_iter iter; + unsigned int i; + long nr; + int ret, refs; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm bio fixture requires hardware opt-in"); + return; + } + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, state); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, bio_fragment_test_free, state), 0); + nr = get_user_fragments_remote(vma->vm_mm, TEST_VA, SZ_4K, FOLL_WRITE, state->held, 1); + state->count = nr > 0 ? nr : 0; + KUNIT_ASSERT_EQ(test, nr, 1L); + refs = folio_ref_count(state->held[0].folio); + state->bio = bio_alloc(NULL, 4, REQ_OP_READ, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, state->bio); + for (i = 0; i < 1024; i++) { + iov_iter_ubuf(&iter, ITER_DEST, (void __user *)(TEST_VA + SZ_4K - 256), 512); + kthread_use_mm(vma->vm_mm); + ret = bio_iov_iter_get_pages(state->bio, &iter, 511); + kthread_unuse_mm(vma->vm_mm); + KUNIT_ASSERT_EQ(test, ret, -EFAULT); + KUNIT_EXPECT_EQ(test, iov_iter_count(&iter), 512UL); + KUNIT_EXPECT_EQ(test, state->bio->bi_vcnt, 0); + KUNIT_EXPECT_PTR_EQ(test, state->bio->bi_user_fragments, NULL); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(state->held[0].subpage), 0U); + KUNIT_EXPECT_EQ(test, folio_ref_count(state->held[0].folio), refs); + bio_reset(state->bio, NULL, REQ_OP_READ); + } +} +#endif + +static void user4k_fragment_longterm_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct fragment_test_pins *pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + phys_addr_t before; + bool movable; + unsigned int i; + long ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + ret = fragment_test_pin(pins, vma->vm_mm, TEST_VA, PAGE_SIZE, FOLL_WRITE, ARRAY_SIZE(pins->fragments)); + KUNIT_ASSERT_EQ(test, ret, (long)TEST_SLOTS); + before = PFN_PHYS(folio_pfn(pins->fragments[0].folio)); + movable = !folio_is_longterm_pinnable(pins->fragments[0].folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(pins->fragments[i].folio) + pins->fragments[i].offset, + 0x91 + i, pins->fragments[i].length); + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, vma->vm_mm, TEST_VA, PAGE_SIZE, + FOLL_WRITE | FOLL_LONGTERM, ARRAY_SIZE(pins->fragments)); + KUNIT_ASSERT_EQ(test, ret, (long)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + + KUNIT_EXPECT_TRUE(test, folio_is_longterm_pinnable(f->folio)); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(f->subpage), 1U); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x91 + i, f->length), NULL); + } + if (movable) + KUNIT_EXPECT_NE(test, PFN_PHYS(folio_pfn(pins->fragments[0].folio)), before); + kunit_info(test, "long-term source movable=%u migrated=%u\n", movable, + PFN_PHYS(folio_pfn(pins->fragments[0].folio)) != before); + fragment_test_unpin(pins); + /* A long-term flag must never silently create an ordinary GET. */ + ret = get_user_fragments_remote(vma->vm_mm, TEST_VA, SZ_4K, FOLL_LONGTERM, + pins->fragments, 1); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); +} + +static void user4k_file_fragments_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct vm_area_struct *shared, *private; + struct fragment_test_pins *pins; + phys_addr_t shared_phys; + long ret; + unsigned int i; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shared = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, 3 * SZ_4K, SZ_4K); + private = test_file_vma(test, file, false, false, TEST_VA + SZ_4K, 3 * SZ_4K, SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, shared); + KUNIT_ASSERT_NOT_NULL(test, private); + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + ret = fragment_test_pin(pins, shared->vm_mm, shared->vm_start + 17, + 3 * SZ_4K - 33, FOLL_WRITE, 8); + KUNIT_ASSERT_EQ(test, ret, 3L); + shared_phys = PFN_PHYS(folio_pfn(pins->fragments[0].folio)); + for (i = 0; i < 3; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + + KUNIT_EXPECT_PTR_EQ(test, f->subpage, NULL); + KUNIT_EXPECT_EQ(test, f->offset, (unsigned long)((i + 1) * SZ_4K + (!i ? 17 : 0))); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x32 + i, f->length), NULL); + } + /* File pins stay live across PTE removal and mm exit. */ + kunit_release_action(test, user4k_vma_mm_free, shared->vm_mm); + for (i = 0; i < 3; i++) + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(pins->fragments[i].folio)); + fragment_test_unpin(pins); + ret = get_user_fragments_remote(private->vm_mm, private->vm_start, 3 * SZ_4K, + 0, pins->fragments, 8); + pins->count = ret > 0 ? ret : 0; + KUNIT_ASSERT_EQ(test, ret, 3L); + for (i = 0; i < 3; i++) { + KUNIT_EXPECT_PTR_EQ(test, pins->fragments[i].subpage, NULL); + KUNIT_EXPECT_EQ(test, PFN_PHYS(folio_pfn(pins->fragments[i].folio)), shared_phys); + } + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, private->vm_mm, private->vm_start, 3 * SZ_4K, 0, 8); + KUNIT_ASSERT_EQ(test, ret, 3L); + for (i = 0; i < 3; i++) { + struct user_page_fragment *f = &pins->fragments[i]; + + KUNIT_EXPECT_NOT_NULL(test, f->subpage); + KUNIT_EXPECT_NE(test, PFN_PHYS(folio_pfn(f->folio)), shared_phys); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x32 + i, f->length), NULL); + KUNIT_EXPECT_FALSE(test, pte_write(test_fault_entry(test, private->vm_mm, + private->vm_start + i * SZ_4K))); + } + fragment_test_unpin(pins); +} + +/* Exercise the same implementation used by the arm64 syscall entry. */ +asmlinkage long __arm64_sys_mprotect(const struct pt_regs *regs); + +static long test_mprotect(unsigned long start, unsigned long len, unsigned long prot) +{ + struct pt_regs regs = { .regs = { start, len, prot } }; + + return __arm64_sys_mprotect(®s); +} + +static unsigned long test_copy_word(unsigned long addr, u64 *value, bool write) +{ + unsigned long left; + + pagefault_disable(); + if (write) + left = copy_to_user((void __user *)addr, value, sizeof(*value)); + else + left = copy_from_user(value, (void __user *)addr, sizeof(*value)); + pagefault_enable(); + return left; +} + +static void user4k_mprotect_test(struct kunit *test) +{ + const unsigned long starts[] = { + TEST_VA, SZ_2M - SZ_4K, SZ_1G - SZ_4K, BIT(39) - SZ_4K, TEST_VA, + }; + unsigned int shape; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm permission fixture requires hardware opt-in"); + return; + } + for (shape = 0; shape < ARRAY_SIZE(starts); shape++) { + struct file *file = shape == 4 ? test_cache_file(test, PAGE_SIZE, true) : NULL; + struct mm_struct *mm = test_vma_mm(test); + unsigned long start = starts[shape], mapped; + unsigned long flags = MAP_FIXED_NOREPLACE | (file ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS); + phys_addr_t before[4] = { }; + long protect[4] = { }; + unsigned long initial = 0, final = 0, read_ro = 0, write_ro = 0, read_none = 0; + u64 value, ro_value = 0; + unsigned int i; + vm_fault_t fault = 0; + + KUNIT_ASSERT_NOT_NULL(test, mm); + if (shape == 4) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + kthread_use_mm(mm); + mapped = vm_mmap(file, start, PAGE_SIZE, PROT_READ | PROT_WRITE, flags, 0); + if (IS_ERR_VALUE(mapped)) + goto detach; + for (i = 0; i < 4; i++) { + struct vm_area_struct *vma = test_vma_lookup(mm, start + i * SZ_4K); + + fault |= test_file_fault(vma, start + i * SZ_4K, true); + if (fault & VM_FAULT_ERROR) + goto detach; + before[i] = pte_phys_mm(mm, test_fault_entry(test, mm, start + i * SZ_4K)); + value = 0x88110000 + i; + initial |= test_copy_word(start + i * SZ_4K, &value, true); + } + /* Keep the mm attached across permission changes and cached accesses. */ + protect[0] = test_mprotect(start + SZ_4K, SZ_4K, PROT_READ); + value = 0xaa; + write_ro = test_copy_word(start + SZ_4K, &value, true); + final |= test_copy_word(start, &value, true); + final |= test_copy_word(start + 3 * SZ_4K, &value, true); + protect[1] = test_mprotect(start + 2 * SZ_4K, SZ_4K, PROT_NONE); + read_none = test_copy_word(start + 2 * SZ_4K, &value, false); + read_ro = test_copy_word(start + SZ_4K, &ro_value, false); + protect[2] = test_mprotect(start, PAGE_SIZE, PROT_READ); + protect[3] = test_mprotect(start, PAGE_SIZE, PROT_READ | PROT_WRITE); + for (i = 0; i < 4; i++) { + value = 0x99220000 + i; + final |= test_copy_word(start + i * SZ_4K, &value, true); + } +detach: + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, start); + KUNIT_ASSERT_EQ(test, fault & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_EQ(test, initial, 0UL); + KUNIT_EXPECT_EQ(test, final, 0UL); + KUNIT_EXPECT_EQ(test, write_ro, (unsigned long)sizeof(u64)); + KUNIT_EXPECT_EQ(test, read_none, (unsigned long)sizeof(u64)); + KUNIT_EXPECT_EQ(test, read_ro, 0UL); + KUNIT_EXPECT_EQ(test, ro_value, 0x88110001ULL); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, start + i * SZ_4K); + + KUNIT_EXPECT_EQ(test, protect[i], 0L); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), before[i]); + } + KUNIT_EXPECT_EQ(test, mm->total_vm, (unsigned long)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, mm->map_count, 1); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +#ifdef CONFIG_NUMA_BALANCING +static void user_subpage_numa_cow_batch_test(struct kunit *test) +{ + unsigned int shift; + + for (shift = 12; shift < PAGE_SHIFT; shift += 2) { + unsigned long size = 1UL << shift, start = TEST_VA; + struct vm_area_struct *vma; + struct mm_struct *child = test_vma_mm_granule(test, shift); + struct mm_struct *mm = test_vma_mm_granule(test, shift); + phys_addr_t before; + unsigned long marked, mapped; + long ret; + unsigned int i; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_NULL(test, child); + /* Use mmap accounting because the child will partially unmap. */ + kthread_use_mm(mm); + mapped = vm_mmap(NULL, start, 4 * size, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, start); + vma = test_vma_lookup(mm, start); + KUNIT_ASSERT_NOT_NULL(test, vma); + for (i = 0; i < 4; i++) + KUNIT_ASSERT_EQ(test, test_file_fault(vma, start + i * size, true), + (vm_fault_t)0); + before = pte_phys_mm(mm, test_fault_entry(test, mm, start + size)); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0L); + kthread_use_mm(child); + ret = vm_munmap(start + size, size); + kthread_unuse_mm(child); + KUNIT_ASSERT_EQ(test, ret, 0L); + /* Reuse the now-private slot without replacing its native owner. */ + KUNIT_ASSERT_EQ(test, test_file_fault(vma, start + size, true), (vm_fault_t)0); + KUNIT_ASSERT_EQ(test, pte_phys_mm(mm, + test_fault_entry(test, mm, start + size)), before); + kthread_use_mm(mm); + ret = test_mprotect(start, 4 * size, PROT_READ); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0L); + vma = test_vma_lookup(mm, start); + KUNIT_ASSERT_NOT_NULL(test, vma); + /* Permit local-node hinting, which normally skips a single user. */ + mmget(mm); + mmap_read_lock(mm); + marked = change_prot_numa(vma, start, start + 4 * size); + mmap_read_unlock(mm); + mmput(mm); + KUNIT_EXPECT_EQ(test, marked, 1UL); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, start + i * size); + + KUNIT_EXPECT_EQ(test, !!pte_protnone(pte), i == 1); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + } + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} +#endif + +static void user4k_mprotect_cow_batch_test(struct kunit *test) +{ + struct mm_struct *mm, *child; + struct vm_area_struct *vma; + struct fragment_test_pins *pins; + phys_addr_t before[4]; + unsigned int i; + unsigned long mapped; + long ret[2]; + int copied; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm permission fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_NULL(test, child); + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, TEST_VA); + vma = test_vma_lookup(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, fragment_test_pin(pins, mm, TEST_VA + SZ_4K, SZ_4K, FOLL_WRITE, 8), 1L); + for (i = 0; i < 4; i++) + before[i] = pte_phys_mm(mm, test_fault_entry(test, mm, TEST_VA + i * SZ_4K)); + uprobe_start_dup_mmap(); + copied = dup_mmap(child, mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, copied, 0); + kthread_use_mm(mm); + ret[0] = test_mprotect(TEST_VA, PAGE_SIZE, PROT_READ); + ret[1] = test_mprotect(TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret[0], 0L); + KUNIT_EXPECT_EQ(test, ret[1], 0L); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), before[i]); + /* One native folio contains both exclusive and shared COW slots. */ + KUNIT_EXPECT_EQ(test, pte_write(pte), i == 1); + } + fragment_test_unpin(pins); +} + +static void user4k_mprotect_accounting_test(struct kunit *test) +{ + struct mm_struct *mm = test_vma_mm(test); + unsigned long mapped, committed[4], data[4]; + s64 global[4], before; + long ret[4]; + unsigned int i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm permission fixture requires hardware opt-in"); + return; + } + KUNIT_ASSERT_NOT_NULL(test, mm); + before = test_commit_baseline(test); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, 0); + if (!IS_ERR_VALUE(mapped)) { + ret[0] = test_mprotect(TEST_VA + SZ_4K, SZ_4K, PROT_READ); + committed[0] = mm->committed_user_pages; + data[0] = mm->data_vm; + global[0] = percpu_counter_sum(&vm_committed_as) - before; + ret[1] = test_mprotect(TEST_VA, PAGE_SIZE, PROT_READ); + committed[1] = mm->committed_user_pages; + data[1] = mm->data_vm; + global[1] = percpu_counter_sum(&vm_committed_as) - before; + ret[2] = test_mprotect(TEST_VA + 2 * SZ_4K, SZ_4K, PROT_READ | PROT_WRITE); + committed[2] = mm->committed_user_pages; + data[2] = mm->data_vm; + global[2] = percpu_counter_sum(&vm_committed_as) - before; + ret[3] = vm_munmap(TEST_VA, PAGE_SIZE); + committed[3] = mm->committed_user_pages; + data[3] = mm->data_vm; + global[3] = percpu_counter_sum(&vm_committed_as) - before; + } + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, TEST_VA); + for (i = 0; i < 4; i++) { + unsigned long expected = i == 0 ? TEST_SLOTS - 1 : i == 2 ? 1 : 0; + + KUNIT_EXPECT_EQ(test, ret[i], 0L); + KUNIT_EXPECT_EQ(test, committed[i], expected); + KUNIT_EXPECT_EQ(test, data[i], expected); + KUNIT_EXPECT_EQ(test, global[i], expected ? 1LL : 0LL); + } + KUNIT_EXPECT_EQ(test, mm->map_count, 0); +} + +static void user4k_fragment_permissions_test(struct kunit *test) +{ + unsigned long start = TEST_VA + SZ_4K; + struct vm_area_struct *vma = test_fault_vma(test, start, SZ_4K); + struct fragment_test_pins *pins; + struct user_page_fragment *fragment; + struct mm_struct *mm; + spinlock_t *ptl; + pte_t *ptep; + long ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mm = vma->vm_mm; + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + fragment = &pins->fragments[0]; + { + struct page *page = NULL; + int locked = 1; + + mmap_read_lock(mm); + ret = pin_user_pages_remote(mm, start, 1, 0, &page, &locked); + if (locked) + mmap_read_unlock(mm); + if (ret == 1) + unpin_user_page(page); + KUNIT_EXPECT_EQ(test, ret, -EOPNOTSUPP); + } + ret = fragment_test_pin(pins, mm, start + 17, 31, 0, 8); + KUNIT_ASSERT_EQ(test, ret, 1L); + KUNIT_EXPECT_TRUE(test, is_zero_folio(fragment->folio)); + KUNIT_EXPECT_PTR_EQ(test, fragment->subpage, NULL); + /* Zero-page PTEs alias the first physical 4K despite this VA's slot 1. */ + KUNIT_EXPECT_EQ(test, fragment->offset, 17UL); + KUNIT_EXPECT_EQ(test, fragment->length, 31U); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(fragment->folio) + fragment->offset, + 0, fragment->length), NULL); + ret = fragment_test_pin(pins, mm, start + 17, 31, FOLL_WRITE, 8); + KUNIT_ASSERT_EQ(test, ret, 1L); + KUNIT_EXPECT_NOT_NULL(test, fragment->subpage); + /* COW backing may use a different quarter from the virtual address. */ + KUNIT_EXPECT_EQ(test, page_to_phys(&fragment->folio->page) + fragment->offset, + pte_phys_mm(mm, test_fault_entry(test, mm, start)) + 17); + memset(folio_address(fragment->folio) + fragment->offset, 0xa4, fragment->length); + fragment_test_unpin(pins); + mmap_write_lock(mm); + vm_flags_clear(vma, VM_WRITE); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + ptep = test_mm_lookup(mm, start, &ptl); + if (ptep) { + ptep_set_wrprotect(mm, start, ptep); + pte_unmap_unlock(ptep, ptl); + } + flush_tlb_range(vma, start, start + SZ_4K); + mmap_write_unlock(mm); + KUNIT_ASSERT_NOT_NULL(test, ptep); + ret = fragment_test_pin(pins, mm, start + 17, 31, FOLL_WRITE, 8); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + ret = fragment_test_pin(pins, mm, start + 17, 31, FOLL_WRITE | FOLL_FORCE, 8); + KUNIT_ASSERT_EQ(test, ret, 1L); + KUNIT_EXPECT_FALSE(test, pte_write(test_fault_entry(test, mm, start))); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(fragment->folio) + fragment->offset, + 0xa4, fragment->length), NULL); + fragment_test_unpin(pins); + mmap_write_lock(mm); + vm_flags_clear(vma, VM_READ); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + mmap_write_unlock(mm); + ret = fragment_test_pin(pins, mm, start + 17, 31, 0, 8); + KUNIT_EXPECT_EQ(test, ret, -EFAULT); + ret = fragment_test_pin(pins, mm, start + 17, 31, FOLL_FORCE, 8); + KUNIT_EXPECT_EQ(test, ret, 1L); + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, mm, start, 1, FOLL_GET, 8); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); +} + +static void test_subpage_unpin(void *arg) +{ + mm_subpage_unpin(arg); +} + +static void user4k_slot_pin_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct mm_struct *child; + struct mm_subpage *slots[4]; + struct folio *folio, *dst; + unsigned int i; + vm_fault_t fault; + spinlock_t *ptl; + pte_t *ptep, pte; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + for (i = 0; i < 4; i++) { + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + i * SZ_4K); + slots[i] = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slots[i]); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, slots[i]), 0); + KUNIT_EXPECT_TRUE(test, mm_subpage_is_exclusive(slots[i])); + memset(folio_address(mm_subpage_folio(slots[i])) + i * SZ_4K, 0x61 + i, SZ_4K); + } + folio = mm_subpage_folio(slots[1]); + test_swap_tags_set(folio); + for (i = 0; i < 2; i++) { + ptep = test_mm_lookup(vma->vm_mm, TEST_VA + SZ_4K, &ptl); + KUNIT_ASSERT_NOT_NULL(test, ptep); + ret = mm_subpage_pin(slots[1]); + pte_unmap_unlock(ptep, ptl); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_unpin, slots[1]), 0); + } + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(slots[1]), 2U); + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(folio)); + KUNIT_EXPECT_FALSE(test, mm_subpage_try_share(slots[1])); + dst = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, dst); + folio_lock(folio); + folio_lock(dst); + ret = mm_subpage_migrate_prepare(folio, dst); + /* Finish even unexpected success so a failing assertion cannot leak metadata. */ + mm_subpage_migrate_finish(folio); + mm_subpage_migrate_finish(dst); + folio_unlock(dst); + folio_unlock(folio); + folio_put(dst); + KUNIT_EXPECT_EQ(test, ret, -EBUSY); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = 0; i < 4; i++) { + struct mm_subpage *copy; + + pte = test_fault_entry(test, child, TEST_VA + i * SZ_4K); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(child, pte) == mm_subpage_phys(slots[i]), i != 1); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slots[i]), i == 1 ? 1 : 2); + KUNIT_EXPECT_EQ(test, mm_subpage_is_exclusive(slots[i]), i == 1); + copy = mm_subpage_get_from_phys(pte_phys_mm(child, pte)); + KUNIT_ASSERT_NOT_NULL(test, copy); + KUNIT_EXPECT_EQ(test, mm_subpage_is_exclusive(copy), i == 1); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(copy)) + mm_subpage_offset(copy), + 0x61 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match_at(mm_subpage_folio(copy), + mm_subpage_offset(copy), i)); + mm_subpage_put(copy); + } + /* A read pin on a shared slot must request unsharing, not pin shared COW data. */ + ptep = test_mm_lookup(vma->vm_mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, ptep); + ret = mm_subpage_pin(slots[0]); + pte_unmap_unlock(ptep, ptl); + if (!ret) + mm_subpage_unpin(slots[0]); + KUNIT_EXPECT_EQ(test, ret, -EAGAIN); + /* Re-enabling writes must retain the exact backing of an exclusive pinned slot. */ + ptep = test_mm_lookup(vma->vm_mm, TEST_VA + SZ_4K, &ptl); + KUNIT_ASSERT_NOT_NULL(test, ptep); + ptep_set_wrprotect(vma->vm_mm, TEST_VA + SZ_4K, ptep); + pte_unmap_unlock(ptep, ptl); + flush_tlb_range(vma, TEST_VA + SZ_4K, TEST_VA + 2 * SZ_4K); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA + SZ_4K, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + SZ_4K); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte), mm_subpage_phys(slots[1])); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + *((u8 *)folio_address(folio) + SZ_4K) = 0xab; + pte = test_fault_entry(test, child, TEST_VA + SZ_4K); + KUNIT_EXPECT_EQ(test, *(u8 *)phys_to_virt(pte_phys_mm(child, pte)), (u8)0x62); + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slots[1]), 0); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(slots[1]), 2U); + KUNIT_EXPECT_EQ(test, *((u8 *)folio_address(folio) + SZ_4K), (u8)0xab); + kunit_release_action(test, test_subpage_unpin, slots[1]); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(slots[1]), 1U); + kunit_release_action(test, test_subpage_unpin, slots[1]); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(slots[1]), 0U); + KUNIT_EXPECT_FALSE(test, folio_maybe_dma_pinned(folio)); +} + +#ifdef CONFIG_MIGRATION +static struct folio *test_migration_alloc(struct folio *src, unsigned long private) +{ + return folio_alloc(GFP_HIGHUSER_MOVABLE, 0); +} + +asmlinkage long __arm64_sys_madvise(const struct pt_regs *regs); + +static long test_madvise(unsigned long addr, unsigned long size, unsigned long behavior) +{ + struct pt_regs regs = { .regs = { addr, size, behavior } }; + + return __arm64_sys_madvise(®s); +} +#endif + +static void user_granule_large_vma_split_test(struct kunit *test) +{ + unsigned int shifts[] = { 12, 14 }, i; + + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) || + (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN))) { + kunit_skip(test, "requires THP boundary splitting and borrowed-mm execution"); + return; + } + for (i = 0; i < ARRAY_SIZE(shifts); i++) { + struct mm_struct *mm; + struct vm_area_struct *vma; + unsigned long size = 1UL << shifts[i]; + unsigned long base = 4UL * HPAGE_PMD_SIZE, mapped; + pte_t pte; + long ret; + + if (shifts[i] >= PAGE_SHIFT) + continue; + mm = mm_alloc(); + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, shifts[i]), 0); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, base, 2UL * HPAGE_PMD_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE | MAP_NORESERVE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, base); + vma = test_vma_lookup(mm, base); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, base, true) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, base + size, true) & VM_FAULT_ERROR, 0U); + /* The generic VMA split probes the native THP boundary for this range. */ + kthread_use_mm(mm); + ret = test_mprotect(base + size, size, PROT_READ); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0L); + pte = test_fault_entry(test, mm, base + size); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + pte = test_fault_entry(test, mm, base); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + kthread_use_mm(mm); + ret = test_mprotect(base + size, size, PROT_READ | PROT_WRITE); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, ret, 0L); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +static void user_granule_brk_extent_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + unsigned int count = PAGE_SHIFT > 14 ? 3 : 2, i; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm brk fixture requires hardware opt-in"); + return; + } + for (i = 0; i < count; i++) { + struct mm_struct *mm = mm_alloc(); + struct vm_area_struct *vma; + unsigned long size = 1UL << shifts[i], neighbor; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, user4k_vma_mm_free, mm), 0); + KUNIT_ASSERT_EQ(test, test_mm_select_granule(mm, shifts[i]), 0); + kthread_use_mm(mm); + ret = vm_brk_flags(TEST_VA, size + 1, false); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0); + vma = test_vma_lookup(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_EQ(test, vma->vm_start, TEST_VA); + KUNIT_EXPECT_EQ(test, vma->vm_end, TEST_VA + 2 * size); + KUNIT_EXPECT_EQ(test, mm->total_vm, 2UL); + /* FEX reserves exactly this page after its initial ELF/BSS break. */ + kthread_use_mm(mm); + neighbor = vm_mmap(NULL, TEST_VA + 2 * size, size, PROT_NONE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, 0); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, neighbor, TEST_VA + 2 * size); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +#ifdef CONFIG_MIGRATION +static void user4k_droppable_reclaim_test(struct kunit *test) +{ + unsigned int mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm droppable fixture requires hardware opt-in"); + return; + } + /* Dirty discard, retained GET, retained PIN, migration, zero-page COW. */ + for (mode = 0; mode < 5; mode++) { + struct mm_struct *mm = test_vma_mm(test); + struct vm_area_struct *vma; + struct user_page_fragment held = {}; + struct folio *folio; + unsigned int i, succeeded = 0; + unsigned long mapped; + long ret; + pte_t pte; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_ANONYMOUS | MAP_DROPPABLE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, TEST_VA); + vma = test_vma_lookup(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_EXPECT_TRUE(test, vma->vm_flags & VM_WIPEONFORK); + KUNIT_EXPECT_TRUE(test, vma->vm_flags & VM_DONTDUMP); + if (mode == 4) { + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, mm, TEST_VA); + KUNIT_EXPECT_TRUE(test, is_zero_pfn(pte_pfn(pte))); + } + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + if (mode != 4) + for (i = 1; i < PAGE_SIZE / SZ_4K; i++) + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA + i * SZ_4K, + true) & VM_FAULT_ERROR, 0U); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_FALSE(test, folio_test_swapbacked(folio)); + memset(folio_address(folio), 0xa7, PAGE_SIZE); + folio_put(folio); + if (mode == 1 || mode == 2) { + ret = mode == 1 ? get_user_fragments_remote(mm, TEST_VA, SZ_4K, 0, &held, 1) : + pin_user_fragments_remote(mm, TEST_VA, SZ_4K, 0, &held, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + } + if (mode == 3) { + lru_add_drain_all(); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + ret = isolate_folio_to_list(folio, &folios); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 1L); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_EXPECT_EQ(test, ret, 0L); + KUNIT_EXPECT_EQ(test, succeeded, 1U); + } + lru_add_drain_all(); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + KUNIT_EXPECT_FALSE(test, folio_test_swapbacked(folio)); + KUNIT_EXPECT_FALSE(test, folio_test_swapcache(folio)); + if (mode == 1 || mode == 2) { + pte = test_fault_entry(test, mm, TEST_VA); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, *(unsigned char *)folio_address(folio), 0xa7); + release_user_fragments(&held, 1, false); + folio_lock(folio); + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + } + folio_put(folio); + for (i = 0; i < (mode == 4 ? 1 : PAGE_SIZE / SZ_4K); i++) { + pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_none(pte)); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0L); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0, PAGE_SIZE), NULL); + KUNIT_EXPECT_FALSE(test, folio_test_swapbacked(folio)); + folio_put(folio); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +static void test_release_fragment(void *arg) +{ + release_user_fragments(arg, 1, false); +} + +static void user4k_lazyfree_test(struct kunit *test) +{ + unsigned int mode; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm advice requires hardware opt-in"); + return; + } + /* Partial, redirty, retained GET, typed write, all leaves, migration. */ + for (mode = 0; mode < 11; mode++) { + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct mm_struct *mm; + struct user_page_fragment *held; + struct folio *folio; + struct mm_subpage *slot; + unsigned int i, succeeded = 0; + long ret; + pte_t pte; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_NULL(test, vma); + held = kunit_kzalloc(test, sizeof(*held), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, held); + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, test_release_fragment, held), 0); + mm = vma->vm_mm; + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + lru_add_drain_all(); + if (mode == 6) { + ret = get_user_fragments_remote(mm, TEST_VA, SZ_4K, 0, held, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + } + kthread_use_mm(mm); + ret = test_madvise(TEST_VA, + (mode == 4 || mode == 9 || mode == 10) ? PAGE_SIZE : SZ_4K, + MADV_FREE); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, ret, 0L); + pte = test_fault_entry(test, mm, TEST_VA); + KUNIT_EXPECT_EQ(test, pte_dirty(pte), mode == 6); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_is_lazyfree(slot), mode != 6); + mm_subpage_put(slot); + if (mode == 1) + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + + if (mode == 2 || mode == 3 || mode == 8 || mode == 10) { + ret = get_user_fragments_remote(mm, TEST_VA, SZ_4K, + mode == 3 ? FOLL_WRITE : 0, held, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + if (mode == 3 || mode == 8) { + memset(folio_address(held->folio) + held->offset, 0xd3, SZ_4K); + release_user_fragments(held, 1, true); + } + } + if (mode == 7) { + ret = pin_user_fragments_remote(mm, TEST_VA, SZ_4K, 0, held, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + } + if (mode == 5 || mode == 9) { + lru_add_drain_all(); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + ret = isolate_folio_to_list(folio, &folios); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 1L); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_EXPECT_EQ(test, ret, 0L); + KUNIT_EXPECT_EQ(test, succeeded, 1U); + } + lru_add_drain_all(); + /* A retained GET can coexist with a temporary migration PTE. */ + folio = held->folio; + if (folio) + folio_get(folio); + else + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + try_to_unmap(folio, TTU_SYNC); + if (mode == 10) { + /* A temporary reader leaves clean whole-folio advice intact. */ + KUNIT_EXPECT_FALSE(test, folio_test_swapbacked(folio)); + KUNIT_EXPECT_FALSE(test, folio_test_dirty(folio)); + KUNIT_EXPECT_FALSE(test, pte_none(test_fault_entry(test, mm, TEST_VA))); + release_user_fragments(held, 1, false); + try_to_unmap(folio, TTU_SYNC); + } + folio_unlock(folio); + folio_put(folio); + for (i = 0; i < 4; i++) { + pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + /* Migration may refold dirty neighbors and retain an advised slot. */ + if (mode == 5 && !i && pte_present(pte)) { + KUNIT_EXPECT_TRUE(test, pte_dirty(pte)); + continue; + } + KUNIT_EXPECT_EQ_MSG(test, pte_none(pte), + (mode == 4 || mode == 9 || mode == 10) || + (i == 0 && (mode == 0 || mode == 5)), + "mode=%u slot=%u pte=%llx", mode, i, + (unsigned long long)pte_val(pte)); + } + if (mode == 2 || mode == 6 || mode == 7) + release_user_fragments(held, 1, false); + kunit_release_action(test, test_release_fragment, held); + kunit_kfree(test, held); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +static void user4k_mprotect_migration_test(struct kunit *test) +{ + unsigned int file_case; + + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm permission fixture requires hardware opt-in"); + return; + } + for (file_case = 0; file_case < 2; file_case++) { + struct mm_struct *mm = test_vma_mm(test); + struct file *file = file_case ? test_cache_file(test, PAGE_SIZE, true) : NULL; + struct vm_area_struct *vma; + struct folio *src, *dst; + unsigned long mapped; + unsigned int i; + int prepared, unmapped = -1; + long protected = -EINVAL; + bool offsets = true; + phys_addr_t phys; + + KUNIT_ASSERT_NOT_NULL(test, mm); + if (file_case) + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + kthread_use_mm(mm); + mapped = vm_mmap(file, TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | (file ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS), 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, TEST_VA); + vma = test_vma_lookup(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + dst = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, dst); + src = test_fault_folio(mm, TEST_VA); + if (!src) { + folio_put(dst); + KUNIT_FAIL(test, "missing migration source"); + return; + } + phys = PFN_PHYS(folio_pfn(src)); + folio_lock(src); + folio_lock(dst); + prepared = mm_subpage_migrate_prepare(src, dst); + if (!prepared) { + try_to_migrate(src, TTU_SYNC); + unmapped = folio_mapcount(src); + kthread_use_mm(mm); + protected = test_mprotect(TEST_VA, PAGE_SIZE, PROT_READ); + kthread_unuse_mm(mm); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + + offsets &= softleaf_is_migration(softleaf_from_pte(pte)) && + !softleaf_is_migration_write(softleaf_from_pte(pte)) && + pte_swp_subpage_offset(pte) == i * SZ_4K; + } + remove_migration_ptes(src, src, 0); + mm_subpage_migrate_finish(src); + mm_subpage_migrate_finish(dst); + } + folio_unlock(dst); + folio_unlock(src); + folio_put(dst); + folio_put(src); + KUNIT_ASSERT_EQ(test, prepared, 0); + KUNIT_EXPECT_EQ(test, unmapped, 0); + KUNIT_EXPECT_EQ(test, protected, 0L); + KUNIT_EXPECT_TRUE(test, offsets); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte), phys + i * SZ_4K); + } + } +} + +/* Independent roots and VA offsets must not change physical slot identity. */ +static void user4k_nonlinear_rmap_test(struct kunit *test) +{ + struct anon_slot_fixture *a = anon_slot_fixture_create(test); + struct anon_slot_fixture *b = anon_slot_fixture_create(test); + struct mm_struct *child = test_vma_mm(test); + struct mm_subpage *slot; + struct folio *folio, *dst; + phys_addr_t phys; + vm_flags_t flags; + unsigned int succeeded = 0, side; + int ret, unmapped = -1; + bool isolated; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + KUNIT_ASSERT_NOT_NULL(test, child); + slot = mm_subpage_alloc_at(a->pool, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + folio = mm_subpage_folio(slot); + /* Unlike the lightweight rmap fixture, real LRU migration needs a charge. */ + ret = mem_cgroup_charge(folio, a->mm, GFP_KERNEL); + if (ret) { + mm_subpage_put(slot); + KUNIT_FAIL(test, "cannot charge nonlinear fixture folio"); + return; + } + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(a, 0, slot), 0); + phys = mm_subpage_phys(slot); + slot = mm_subpage_alloc_at(a->pool, 3 * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + /* Same native folio, different root and different VA/physical offset. */ + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(b, 1, slot), 0); + KUNIT_EXPECT_PTR_EQ(test, mm_subpage_folio(slot), folio); + memset(folio_address(folio), 0x53, SZ_4K); + memset(folio_address(folio) + 3 * SZ_4K, 0xa7, SZ_4K); + folio_lock(folio); + ret = folio_referenced(folio, 1, NULL, &flags); + folio_unlock(folio); + KUNIT_EXPECT_EQ(test, ret, 2); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, b->mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 2); + KUNIT_EXPECT_EQ(test, pte_phys_mm(child, + test_fault_entry(test, child, TEST_VA + SZ_4K)), phys + 3 * SZ_4K); + kunit_release_action(test, user4k_vma_mm_free, child); + + /* Rollback must find migration entries through both roots. */ + dst = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, dst); + folio_get(folio); + folio_lock(folio); + folio_lock(dst); + ret = mm_subpage_migrate_prepare(folio, dst); + if (!ret) { + try_to_migrate(folio, TTU_SYNC); + unmapped = folio_mapcount(folio); + remove_migration_ptes(folio, folio, 0); + mm_subpage_migrate_finish(folio); + mm_subpage_migrate_finish(dst); + } + folio_unlock(dst); + folio_unlock(folio); + folio_put(dst); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, unmapped, 0); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 2); + + /* Successful migration exercises copied identities in the new owner. */ + folio_add_lru(folio); + lru_add_drain_all(); + folio_get(folio); + isolated = isolate_folio_to_list(folio, &folios); + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + if (ret) { + KUNIT_FAIL(test, "nonlinear migration failed: %d", ret); + return; + } + /* Refresh fixture references before assertions/cleanup after migration. */ + for (side = 0; side < 2; side++) { + struct anon_slot_fixture *f = side ? b : a; + pte_t pte = test_fault_entry(test, f->mm, TEST_VA + side * SZ_4K); + + f->slots[side] = NULL; + f->mapped[side] = false; + if (!pte_present(pte)) + continue; + slot = mm_subpage_get_from_phys(pte_phys_mm(f->mm, pte)); + if (!slot) + continue; + f->slots[side] = slot; + f->mapped[side] = true; + mm_subpage_put(slot); + } + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, succeeded, 1U); + for (side = 0; side < 2; side++) { + struct anon_slot_fixture *f = side ? b : a; + + slot = f->slots[side]; + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), side ? 3U * SZ_4K : 0U); + KUNIT_EXPECT_NE(test, mm_subpage_phys(slot), phys + (side ? 3 * SZ_4K : 0)); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), 1); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), side ? 0xa7 : 0x53, SZ_4K), NULL); + } +} + +static void user4k_migration_entries_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct mm_struct *child = test_vma_mm(test); + struct folio *src, *dst; + unsigned int i, side; + vm_fault_t fault; + bool encoded = true, copied = true, exact = true; + int ret, fork_ret = -EINVAL, unmapped = -1; + + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_NOT_NULL(test, child); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + dst = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, dst); + src = test_fault_folio(vma->vm_mm, TEST_VA); + if (!src) { + folio_put(dst); + KUNIT_FAIL(test, "missing source folio"); + return; + } + folio_lock(src); + folio_lock(dst); + ret = mm_subpage_migrate_prepare(src, dst); + if (!ret) { + try_to_migrate(src, TTU_SYNC); + unmapped = folio_mapcount(src); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, vma->vm_mm, TEST_VA + i * SZ_4K); + struct rmap_walk_range range = { + .address = TEST_VA + i * SZ_4K, + .subpage = true, + .subpage_offset = i * SZ_4K, + }; + DEFINE_FOLIO_RMAP_WALK(walk, src, vma, range, + PVMW_SYNC | PVMW_MIGRATION); + unsigned int count = 0; + + encoded &= softleaf_is_migration_write(softleaf_from_pte(pte)) && + pte_swp_subpage_offset(pte) == i * SZ_4K; + while (page_vma_mapped_walk(&walk)) { + exact &= walk.address == TEST_VA + i * SZ_4K; + count++; + } + exact &= count == 1; + } + uprobe_start_dup_mmap(); + fork_ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + if (!fork_ret) { + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : vma->vm_mm; + + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + + copied &= softleaf_is_migration_read(softleaf_from_pte(pte)) && + pte_swp_subpage_offset(pte) == i * SZ_4K; + } + } + } + /* Drop one migration entry while the other mm still retains it. */ + mmap_read_lock(vma->vm_mm); + zap_vma_range(vma, TEST_VA + SZ_4K, SZ_4K); + mmap_read_unlock(vma->vm_mm); + remove_migration_ptes(src, src, 0); + mm_subpage_migrate_finish(src); + mm_subpage_migrate_finish(dst); + } + folio_unlock(dst); + folio_unlock(src); + folio_put(dst); + folio_put(src); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, fork_ret, 0); + KUNIT_EXPECT_EQ(test, unmapped, 0); + KUNIT_EXPECT_TRUE(test, encoded); + KUNIT_EXPECT_TRUE(test, copied); + KUNIT_EXPECT_TRUE(test, exact); + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : vma->vm_mm; + + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), side ? (unsigned long)TEST_SLOTS : TEST_SLOTS - 1UL); + for (i = 0; i < 4; i++) { + pte_t pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + struct mm_subpage *slot; + + if (!side && i == 1) { + KUNIT_EXPECT_TRUE(test, pte_none(pte)); + continue; + } + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), i * (unsigned int)SZ_4K); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), i == 1 ? 1 : 2); + mm_subpage_put(slot); + } + } +} + +static struct folio *test_file_migration_alloc(struct folio *src, unsigned long private) +{ + return folio_alloc(GFP_HIGHUSER_MOVABLE, folio_order(src)); +} + +static void user64k_file_migration_test(struct kunit *test) +{ + unsigned int mode; + + if (!test_coarse_file_enabled(test)) + return; + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "coarse shmem promotion requires collapse support"); + return; + } + for (mode = 0; mode < 6; mode++) { + bool shmem = mode & 1, fail = mode >= 2, pin = mode >= 4; + bool isolated; + unsigned long len = shmem ? SZ_64K : 2 * SZ_64K; + unsigned long base = shmem ? SZ_64K : 0, offset; + unsigned int nr = len / PAGE_SIZE, succeeded = 0; + unsigned int maps = 2 * nr + len / SZ_4K - 1; + struct file *file = shmem ? test_cache_file(test, 2 * SZ_64K, true) : + test_coarse_cache_file(test, 17 - PAGE_SHIFT, 2 * SZ_64K); + struct vm_area_struct *coarse, *native, *small; + struct folio *folio; + struct user_page_fragment *fragment = NULL; + phys_addr_t old_phys, phys; + int ret; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + coarse = test_file_vma_granule(test, file, 16, true, TEST_VA, len, base); + native = test_file_vma(test, file, true, true, TEST_VA, len, base); + /* Shifted file offset and a small-page table boundary in the alias. */ + small = test_file_vma(test, file, false, true, SZ_2M - SZ_4K, + len - SZ_4K, base + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_NOT_NULL(test, small); + for (offset = 0; offset < len; offset += SZ_64K) + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA + offset, shmem) & + VM_FAULT_ERROR, 0U); + for (offset = 0; offset < len; offset += PAGE_SIZE) + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA + offset, shmem) & + VM_FAULT_ERROR, 0U); + for (offset = 0; offset < len - SZ_4K; offset += SZ_4K) + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start + offset, + shmem) & VM_FAULT_ERROR, 0U); + lru_add_drain_all(); + folio = filemap_get_folio(file->f_mapping, base >> PAGE_SHIFT); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), len); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)(nr + maps + 1)); + old_phys = page_to_phys(&folio->page); + if (pin) { + fragment = kunit_kzalloc(test, sizeof(*fragment), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, fragment); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + test_promote_unpin, fragment), 0); + KUNIT_ASSERT_EQ(test, pin_user_fragments_remote(small->vm_mm, + small->vm_end - SZ_4K + 37, 1, 0, fragment, 1), 1L); + KUNIT_EXPECT_PTR_EQ(test, fragment->folio, folio); + KUNIT_EXPECT_EQ(test, fragment->offset, len - SZ_4K + 37); + } + isolated = isolate_folio_to_list(folio, &folios); + /* Retain either a GET or a typed PIN to force migration rollback. */ + if (!fail || pin || !isolated) + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_file_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + if (pin) { + KUNIT_EXPECT_EQ(test, page_to_phys(&fragment->folio->page), old_phys); + KUNIT_EXPECT_EQ(test, *(u8 *)(folio_address(fragment->folio) + + fragment->offset), + (u8)(0x31 + (base + len - SZ_4K) / SZ_4K)); + kunit_release_action(test, test_promote_unpin, fragment); + } else if (fail) { + folio_put(folio); + } + KUNIT_EXPECT_EQ_MSG(test, ret, fail ? 1 : 0, "mode=%u", mode); + KUNIT_EXPECT_EQ(test, succeeded, fail ? 0U : nr); + folio = filemap_get_folio(file->f_mapping, base >> PAGE_SHIFT); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + phys = page_to_phys(&folio->page); + KUNIT_EXPECT_EQ(test, phys == old_phys, fail); + KUNIT_EXPECT_EQ(test, folio_size(folio), len); + KUNIT_EXPECT_EQ(test, folio->index, base >> PAGE_SHIFT); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)maps); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)(nr + maps + 1)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(coarse->vm_mm, + shmem ? MM_SHMEMPAGES : MM_FILEPAGES), (long)(len / SZ_64K)); + for (offset = 0; offset < len; offset += SZ_4K) { + pte_t pte; + + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + offset, + 0x31 + (base + offset) / SZ_4K, SZ_4K), NULL); + if (offset) { + pte = test_fault_entry(test, small->vm_mm, + small->vm_start + offset - SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, !!pte_write(pte), shmem); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, pte), phys + offset); + } + if (!(offset % PAGE_SIZE)) { + pte = test_fault_entry(test, native->vm_mm, TEST_VA + offset); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, !!pte_write(pte), shmem); + KUNIT_EXPECT_EQ(test, pte_phys_mm(native->vm_mm, pte), phys + offset); + } + if (!(offset % SZ_64K)) { + pte = test_fault_entry(test, coarse->vm_mm, TEST_VA + offset); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, !!pte_write(pte), shmem); + KUNIT_EXPECT_EQ(test, pte_phys_mm(coarse->vm_mm, pte), phys + offset); + } + } + kunit_release_action(test, user4k_vma_mm_free, coarse->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, native->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)nr + 1); + } +} + +static void user64k_anon_migration_test(struct kunit *test) +{ + unsigned int mode, nr = SZ_64K / PAGE_SIZE; + + if (!test_coarse_file_enabled(test)) + return; + for (mode = 0; mode < 5; mode++) { + bool forked = mode & 1, fail = mode >= 2, pin = mode == 4; + struct vm_area_struct *vma, *cvma = NULL; + struct mm_struct *child = NULL; + struct user_page_fragment *fragment = NULL; + struct folio *folio; + phys_addr_t old_phys, phys; + unsigned int i, succeeded = 0; + pte_t pte; + bool isolated; + int ret; + LIST_HEAD(folios); + + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA + SZ_64K - 1, true) & + VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + old_phys = pte_phys_mm(vma->vm_mm, pte); + folio = page_folio(pfn_to_page(PHYS_PFN(old_phys))); + for (i = 0; i < nr; i++) + memset(page_address(folio_page(folio, i)), 0x51 + i, PAGE_SIZE); + if (forked) { + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + } + if (pin) { + fragment = kunit_kzalloc(test, sizeof(*fragment), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, fragment); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + test_promote_unpin, fragment), 0); + KUNIT_ASSERT_EQ(test, pin_user_fragments_remote(vma->vm_mm, + TEST_VA + SZ_64K - 1, 1, FOLL_WRITE, fragment, 1), 1L); + KUNIT_EXPECT_PTR_EQ(test, fragment->folio, folio); + KUNIT_EXPECT_EQ(test, fragment->offset, (unsigned int)SZ_64K - 1); + } + lru_add_drain_all(); + folio_get(folio); + isolated = isolate_folio_to_list(folio, &folios); + if (!fail || pin || !isolated) + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_file_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + if (pin) + kunit_release_action(test, test_promote_unpin, fragment); + else if (fail) + folio_put(folio); + KUNIT_EXPECT_EQ_MSG(test, ret, fail ? 1 : 0, "mode=%u", mode); + KUNIT_EXPECT_EQ(test, succeeded, fail ? 0U : nr); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, !!pte_write(pte), !forked); + phys = pte_phys_mm(vma->vm_mm, pte); + KUNIT_EXPECT_EQ(test, phys == old_phys, fail); + folio = page_folio(pfn_to_page(PHYS_PFN(phys))); + folio_get(folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + expect_coarse_anon_state(test, folio, forked ? 2 : 1, !forked); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)(nr * (forked ? 2 : 1) + 1)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 1L); + if (forked) { + struct folio *copied; + + pte = test_fault_entry(test, child, TEST_VA); + KUNIT_EXPECT_EQ(test, pte_phys_mm(child, pte), phys); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 1L); + /* A restored shared migration entry must still perform full-leaf COW. */ + KUNIT_ASSERT_EQ(test, test_file_fault(cvma, TEST_VA + SZ_64K - 1, + true) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, child, TEST_VA); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, pte), phys); + copied = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(child, pte)))); + expect_coarse_anon_state(test, copied, 1, true); + for (i = 0; i < nr; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(copied, i)), + 0x51 + i, PAGE_SIZE), NULL); + memset(folio_address(copied), 0xa7, SZ_64K); + kunit_release_action(test, user4k_vma_mm_free, child); + } + for (i = 0; i < nr; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(folio, i)), + 0x51 + i, PAGE_SIZE), NULL); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + } +} + +static void user4k_file_migration_test(struct kunit *test) +{ + unsigned int fail; + + for (fail = 0; fail < 2; fail++) { + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct vm_area_struct *small, *native; + struct folio *folio; + phys_addr_t old_phys; + unsigned int i, succeeded = 0; + int ret; + bool isolated; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + small = test_file_vma(test, file, false, true, SZ_2M - SZ_4K, PAGE_SIZE, 0); + native = test_file_vma(test, file, true, true, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_NOT_NULL(test, native); + for (i = 0; i < 4; i++) + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start + i * SZ_4K, + true) & VM_FAULT_ERROR, 0U); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA, true) & VM_FAULT_ERROR, 0U); + lru_add_drain_all(); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ [0 ... TEST_SLOTS - 1] = 1 }, 1); + old_phys = PFN_PHYS(folio_pfn(folio)); + isolated = isolate_folio_to_list(folio, &folios); + if (!fail || !isolated) + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, MIGRATE_SYNC, + MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + if (fail) + folio_put(folio); + KUNIT_EXPECT_EQ(test, ret, fail ? 1 : 0); + KUNIT_EXPECT_EQ(test, succeeded, fail ? 0U : 1U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ [0 ... TEST_SLOTS - 1] = 1 }, 1); + for (i = 0; i < 4; i++) { + pte_t entry = test_fault_entry(test, small->vm_mm, small->vm_start + i * SZ_4K); + + KUNIT_ASSERT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_TRUE(test, pte_write(entry)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, entry), + PFN_PHYS(folio_pfn(folio)) + i * SZ_4K); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, entry) == old_phys + i * SZ_4K, !!fail); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0x31 + i, SZ_4K), NULL); + } + KUNIT_EXPECT_EQ(test, pte_phys_mm(native->vm_mm, + test_fault_entry(test, native->vm_mm, TEST_VA)), PFN_PHYS(folio_pfn(folio))); + /* File reclaim must visit every 4K alias, including across PTE tables. */ + folio_lock(folio); + try_to_unmap(folio, TTU_SYNC); + KUNIT_EXPECT_FALSE(test, folio_mapped(folio)); + test_file_mapcounts(test, folio, (unsigned int[TEST_SLOTS]){ 0, 0, 0, 0 }, 0); + folio_unlock(folio); + folio_put(folio); + for (i = 0; i < 4; i++) { + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, small->vm_mm, + small->vm_start + i * SZ_4K))); + } + for (i = 0; i < 4; i++) + KUNIT_ASSERT_EQ(test, test_file_fault(small, small->vm_start + i * SZ_4K, + false) & VM_FAULT_ERROR, 0U); + } +} + +static void user4k_migration_test(struct kunit *test) +{ + const unsigned long boundaries[] = { + SZ_2M - SZ_4K, SZ_2M, SZ_1G - SZ_4K, SZ_1G, + BIT(39) - SZ_4K, BIT(39), + }; + unsigned int mode; + + /* Dense/sparse, forks, refcount rollback, batched TLBI and table edges. */ + for (mode = 0; mode < 16; mode++) { + bool edge = mode >= 6 && mode < 12; + bool locked = mode >= 12; + bool sparse = mode == 1 || edge || mode == 13; + bool forked = mode == 2 || mode == 4 || edge || mode == 14; + bool fail = mode == 3 || mode == 4 || mode == 15; + unsigned long start = edge ? boundaries[mode - 6] : + TEST_VA + (sparse ? SZ_4K : 0); + unsigned int nr = sparse ? 1 : 4, i, side, succeeded = 0; + unsigned int offsets[TEST_SLOTS]; + struct vm_area_struct *vma = test_fault_vma(test, start, nr * SZ_4K); + struct mm_struct *child = NULL; + struct folio *folio; + phys_addr_t old_phys; + vm_fault_t fault; + int ret; + bool isolated; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_NULL(test, vma); + if (locked) { + mmap_write_lock(vma->vm_mm); + vm_flags_set(vma, VM_LOCKED); + vma->vm_mm->locked_vm = nr; + mmap_write_unlock(vma->vm_mm); + } + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, start, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + folio = test_fault_folio(vma->vm_mm, start); + KUNIT_ASSERT_NOT_NULL(test, folio); + old_phys = PFN_PHYS(folio_pfn(folio)); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, 0x41 + i + mode, SZ_4K); + test_swap_tags_set(folio); + for (i = 0; i < nr; i++) + offsets[i] = offset_in_page(pte_phys_mm(vma->vm_mm, + test_fault_entry(test, vma->vm_mm, start + i * SZ_4K))); + folio_put(folio); + if (forked) { + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + } + lru_add_drain_all(); + folio = test_fault_folio(vma->vm_mm, start); + KUNIT_ASSERT_NOT_NULL(test, folio); + isolated = isolate_folio_to_list(folio, &folios); + /* An extra native reference forces the normal migration rollback. */ + if (!fail || !isolated) + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + mode == 5 ? MIGRATE_ASYNC : MIGRATE_SYNC, + MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + if (fail) + folio_put(folio); + KUNIT_EXPECT_EQ(test, ret, fail ? 1 : 0); + KUNIT_EXPECT_EQ(test, succeeded, fail ? 0U : 1U); + for (side = 0; side <= (unsigned int)forked; side++) { + struct mm_struct *mm = side ? child : vma->vm_mm; + + for (i = 0; i < nr; i++) { + unsigned int index = offsets[i] / SZ_4K; + pte_t pte = test_fault_entry(test, mm, start + i * SZ_4K); + struct mm_subpage *slot; + + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte) == old_phys + index * SZ_4K, fail); + KUNIT_EXPECT_EQ(test, !!pte_write(pte), !forked); + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + folio = mm_subpage_folio(slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), index * (unsigned int)SZ_4K); + KUNIT_EXPECT_EQ(test, mm_subpage_mapcount(slot), forked ? 2 : 1); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + index * SZ_4K, + 0x41 + index + mode, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, index)); + mm_subpage_put(slot); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), (unsigned long)nr); + } + if (locked) { + lru_add_drain_all(); + folio = test_fault_folio(vma->vm_mm, start); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, folio_test_mlocked(folio)); + if (!fail) + KUNIT_EXPECT_EQ(test, folio->mlock_count, nr); + folio_put(folio); + } + if (child) + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + } +} +struct fragment_pin_worker { + struct mm_struct *mm; + struct task_struct *task; + struct completion ready; + unsigned long checks; + int error; +}; + +static int fragment_pin_worker_run(void *arg) +{ + struct fragment_pin_worker *worker = arg; + struct user_page_fragment fragments[4]; + + while (!kthread_should_stop()) { + long nr = pin_user_fragments_remote(worker->mm, TEST_VA, PAGE_SIZE, + 0, fragments, ARRAY_SIZE(fragments)); + unsigned int i; + + if (nr != 4) { + worker->error = nr < 0 ? nr : -EIO; + } else { + for (i = 0; i < nr; i++) { + struct user_page_fragment *f = &fragments[i]; + + if (memchr_inv(folio_address(f->folio) + f->offset, 0x51 + i, f->length)) + worker->error = -EILSEQ; + worker->checks++; + } + } + if (nr > 0) + release_user_fragments(fragments, nr, false); + complete_all(&worker->ready); + if (worker->error) + break; + cond_resched(); + } + return 0; +} + +static void fragment_pin_worker_stop(void *arg) +{ + struct fragment_pin_worker *worker = arg; + + if (worker->task) { + kthread_stop(worker->task); + put_task_struct(worker->task); + worker->task = NULL; + } +} + +static void user4k_fragment_pin_race_test(struct kunit *test) +{ + struct vm_area_struct *vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + struct fragment_test_pins *pins; + struct fragment_pin_worker *workers; + unsigned int i, round, migrated = 0, busy = 0; + long ret; + + KUNIT_ASSERT_NOT_NULL(test, vma); + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + ret = fragment_test_pin(pins, vma->vm_mm, TEST_VA, PAGE_SIZE, FOLL_WRITE, 4); + KUNIT_ASSERT_EQ(test, ret, 4L); + for (i = 0; i < 4; i++) + memset(folio_address(pins->fragments[i].folio) + pins->fragments[i].offset, + 0x51 + i, pins->fragments[i].length); + fragment_test_unpin(pins); + workers = kunit_kcalloc(test, 2, sizeof(*workers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, workers); + for (i = 0; i < 2; i++) { + workers[i].mm = vma->vm_mm; + init_completion(&workers[i].ready); + workers[i].task = kthread_create(fragment_pin_worker_run, &workers[i], + "user4k-pin-%u", i); + if (IS_ERR(workers[i].task)) { + workers[i].task = NULL; + KUNIT_FAIL(test, "pin worker creation failed"); + return; + } + get_task_struct(workers[i].task); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_pin_worker_stop, &workers[i]), 0); + wake_up_process(workers[i].task); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].ready, 10 * HZ), 0UL); + } + for (round = 0; round < 32; round++) { + struct mm_struct *child = test_vma_mm(test); + unsigned long addr = TEST_VA + (round % 4) * SZ_4K; + struct vm_area_struct *cvma; + struct folio *folio; + unsigned int succeeded = 0; + vm_fault_t fault; + struct user_page_fragment write_fragment; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0L); + cvma = test_vma_lookup(child, addr); + KUNIT_ASSERT_NOT_NULL(test, cvma); + mmap_read_lock(child); + fault = handle_mm_fault(cvma, addr, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + ret = pin_user_fragments_remote(child, addr, 1, FOLL_WRITE, &write_fragment, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + *((u8 *)folio_address(write_fragment.folio) + write_fragment.offset) = 0xa0 + round; + release_user_fragments(&write_fragment, 1, true); + lru_add_drain_all(); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + ret = isolate_folio_to_list(folio, &folios); + folio_put(folio); + if (ret) { + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_ASYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_EXPECT_TRUE(test, ret == 0 || ret == 1); + migrated += succeeded; + busy += ret == 1; + } + kunit_release_action(test, user4k_vma_mm_free, child); + } + for (i = 0; i < 2; i++) { + fragment_pin_worker_stop(&workers[i]); + KUNIT_EXPECT_EQ(test, workers[i].error, 0); + KUNIT_EXPECT_GE(test, workers[i].checks, 4UL); + } + kunit_info(test, "32 fork/COW/exit cycles against two pin workers; migrations=%u not_migrated=%u\n", + migrated, busy); +} + +#endif + +static int test_swap_control(const char *operation) +{ + char *argv[] = { "/init", (char *)operation, NULL }; + char *envp[] = { NULL }; + + return call_usermodehelper(argv[0], argv, envp, UMH_WAIT_PROC); +} + +static void test_reenable_swap(void *arg) +{ + bool *disabled = arg; + + if (*disabled) + test_swap_control("--swapon-helper"); +} + +/* Exclusive slots must remain packed after read swap-in and later writes. */ +static void test_user4k_swap_exclusive(struct kunit *test, unsigned int mode) +{ + struct vm_area_struct *vma; + struct folio *folio; + struct mm_subpage *held = NULL; + bool *disabled; + phys_addr_t restored[TEST_SLOTS]; + swp_entry_t entry; + vm_fault_t fault; + unsigned int i; + int ret, mapped, hold_err = 0; + pte_t pte; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + lru_add_drain_all(); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, 0xc0 + i, SZ_4K); + test_swap_tags_set(folio); + if (mode == 5) { + held = mm_subpage_get_from_phys(PFN_PHYS(folio_pfn(folio)) + 2 * SZ_4K); + if (held) { + hold_err = kunit_add_action_or_reset(test, test_subpage_put, held); + if (hold_err) + held = NULL; + } + } + ret = folio_alloc_swap(folio); + entry = folio->swap; + if (!ret) + try_to_unmap(folio, TTU_SYNC); + mapped = folio_mapcount(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, mapped, 0); + if (mode == 5) { + KUNIT_ASSERT_EQ(test, hold_err, 0); + KUNIT_ASSERT_NOT_NULL(test, held); + } + for (i = 0; i < TEST_SLOTS; i++) { + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + i * SZ_4K); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(softleaf_from_pte(pte))); + KUNIT_EXPECT_TRUE(test, pte_swp_exclusive(pte)); + } + if (mode >= 3) { + struct mm_struct *child = test_vma_mm(test); + + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + kunit_release_action(test, user4k_vma_mm_free, child); + for (i = 0; i < TEST_SLOTS; i++) { + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + i * SZ_4K); + KUNIT_EXPECT_FALSE(test, pte_swp_exclusive(pte)); + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), 1U); + } + } + if (mode == 1 || mode == 2 || mode == 4 || mode == 6) + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + if (mode == 2 || mode == 6) { + ret = test_swap_control("--swapoff-helper"); + *disabled = !total_swap_pages; + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, *disabled); + } + for (i = 0; i < TEST_SLOTS; i++) { + struct mm_subpage *slot; + unsigned long address = TEST_VA + i * SZ_4K; + + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + pte = test_fault_entry(test, vma->vm_mm, address); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + restored[i] = pte_phys_mm(vma->vm_mm, pte); + slot = mm_subpage_get_from_phys(restored[i]); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_is_exclusive(slot), !(held && i == 2)); + KUNIT_EXPECT_EQ(test, restored[i], restored[0] + i * SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0xc0 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(mm_subpage_folio(slot), i)); + mm_subpage_put(slot); + } + for (i = 0; i < TEST_SLOTS; i++) { + unsigned long address = TEST_VA + i * SZ_4K; + + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + pte = test_fault_entry(test, vma->vm_mm, address); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + if (held && i == 2) { + KUNIT_EXPECT_NE(test, pte_phys_mm(vma->vm_mm, pte), restored[i]); + memset(phys_to_virt(pte_phys_mm(vma->vm_mm, pte)), 0xdd, SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(held)) + + mm_subpage_offset(held), 0xc2, SZ_4K), NULL); + } else { + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte), restored[i]); + } + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), (unsigned long)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SWAPENTS), 0UL); + if (*disabled) { + ret = test_swap_control("--swapon-helper"); + if (!ret) + *disabled = false; + KUNIT_EXPECT_EQ(test, ret, 0); + } +} + +static void user4k_swap_exclusive_cached_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 0); +} + +static void user4k_swap_exclusive_disk_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 1); +} + +static void user4k_swap_exclusive_swapoff_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 2); +} + +static void user4k_swap_reuse_cached_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 3); +} + +static void user4k_swap_reuse_disk_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 4); +} + +static void user4k_swap_reuse_get_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 5); +} + +static void user4k_swap_reuse_swapoff_test(struct kunit *test) +{ + test_user4k_swap_exclusive(test, 6); +} + +static void test_user4k_swap_roundtrip(struct kunit *test, unsigned int mode, u8 base) +{ + const bool evict = mode != 0; + struct vm_area_struct *parent, *cvma; + struct mm_struct *child; + struct folio *folio; + swp_entry_t entry = { 0 }; + unsigned int i, count; + int err, mapped; + vm_fault_t fault; + pte_t pte; + + if (!total_swap_pages) { + kunit_skip(test, "rerun this suite after enabling the disposable guest swap disk"); + return; + } + parent = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, parent); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + lru_add_drain_all(); + folio = test_fault_folio(parent->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, base + i, SZ_4K); + test_swap_tags_set(folio); + if (mode == 2) { + LIST_HEAD(folios); + + folio_unlock(folio); + err = folio_isolate_lru(folio) ? 0 : -EBUSY; + if (!err) + list_add(&folio->lru, &folios); + folio_put(folio); + /* Let the normal reclaim path allocate swap and unmap all slots. */ + reclaim_pages(&folios); + mapped = 0; + for (i = 0; i < TEST_SLOTS; i++) { + pte = test_fault_entry(test, parent->vm_mm, TEST_VA + i * SZ_4K); + mapped += pte_present(pte); + } + entry = softleaf_from_pte(pte); + } else { + err = folio_alloc_swap(folio); + if (!err) { + entry = folio->swap; + try_to_unmap(folio, TTU_SYNC | (evict ? TTU_BATCH_FLUSH : 0)); + } + mapped = folio_mapcount(folio); + folio_unlock(folio); + try_to_unmap_flush(); + folio_put(folio); + } + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_ASSERT_EQ(test, mapped, 0); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent->vm_mm, MM_ANONPAGES), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent->vm_mm, MM_SWAPENTS), (unsigned long)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) { + pte = test_fault_entry(test, parent->vm_mm, TEST_VA + i * SZ_4K); + KUNIT_ASSERT_FALSE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(pte).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(pte), i * (unsigned int)SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_swp_exclusive(pte)); + } + /* Include extended counts when a native page has 16 user slots. */ + count = swp_swapcount(entry); + KUNIT_EXPECT_EQ(test, count, (unsigned int)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), 1U); + if (evict) + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + + /* Fork while all slots are nonpresent, preserving each byte offset. */ + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + err = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, err, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_SWAPENTS), (unsigned long)TEST_SLOTS); + /* Include extended counts when a native page has 16 user slots. */ + count = swp_swapcount(entry); + KUNIT_EXPECT_EQ(test, count, 2U * TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), 2U); + for (i = 0; i < TEST_SLOTS; i++) { + unsigned long address = TEST_VA + ((i * 5 + 2) % TEST_SLOTS) * SZ_4K; + struct mm_subpage *slot; + spinlock_t *ptl; + pte_t *ptep; + bool intact = false; + + pte = test_fault_entry(test, parent->vm_mm, address); + KUNIT_EXPECT_FALSE(test, pte_swp_exclusive(pte)); + pte = test_fault_entry(test, child, address); + KUNIT_EXPECT_FALSE(test, pte_swp_exclusive(pte)); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(pte).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(pte), ((i * 5 + 2) % TEST_SLOTS) * (unsigned int)SZ_4K); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, address, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + KUNIT_EXPECT_EQ(test, !!(fault & VM_FAULT_MAJOR), evict && !i); + ptep = test_mm_lookup(parent->vm_mm, address, &ptl); + KUNIT_ASSERT_NOT_NULL(test, ptep); + pte = ptep_get(ptep); + slot = pte_present(pte) ? mm_subpage_get_from_phys(pte_phys_mm(parent->vm_mm, pte)) : NULL; + if (slot) { + intact = !memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), base + ((i * 5 + 2) % TEST_SLOTS), SZ_4K) && + test_swap_tags_match(mm_subpage_folio(slot), ((i * 5 + 2) % TEST_SLOTS)); + mm_subpage_put(slot); + } + pte_unmap_unlock(ptep, ptl); + KUNIT_EXPECT_TRUE(test, intact); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent->vm_mm, MM_ANONPAGES), (unsigned long)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent->vm_mm, MM_SWAPENTS), 0UL); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), 1U); + /* A write fault must swap in and then COW just the child's selected slot. */ + mmap_read_lock(child); + fault = handle_mm_fault(cvma, TEST_VA + SZ_4K, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + pte = test_fault_entry(test, child, TEST_VA + SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + folio = test_fault_folio(child, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + { + unsigned int offset = pte_phys_mm(child, pte) - PFN_PHYS(folio_pfn(folio)); + + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + offset, + base + 1, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match_at(folio, offset, 1)); + memset(folio_address(folio) + offset, 0xa7, SZ_4K); + } + folio_put(folio); + folio = test_fault_folio(parent->vm_mm, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + { + pte_t parent_pte = test_fault_entry(test, parent->vm_mm, TEST_VA + SZ_4K); + unsigned int offset = pte_phys_mm(parent->vm_mm, parent_pte) - + PFN_PHYS(folio_pfn(folio)); + + KUNIT_EXPECT_NE(test, pte_phys_mm(child, pte), + pte_phys_mm(parent->vm_mm, parent_pte)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + offset, + base + 1, SZ_4K), NULL); + } + folio_put(folio); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 1UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_SWAPENTS), TEST_SLOTS - 1UL); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), i == 1 ? 0U : 1U); + /* Exit with three swapped slots, then tear down the restored parent. */ + kunit_release_action(test, user4k_vma_mm_free, child); + /* Include extended counts when a native page has 16 user slots. */ + count = swp_swapcount(entry); + KUNIT_EXPECT_EQ(test, count, 0U); + for (i = 0; i < TEST_SLOTS; i++) + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, i * SZ_4K), 0U); + kunit_release_action(test, user4k_vma_mm_free, parent->vm_mm); +} + +static void user4k_swap_mremap_test(struct kunit *test) +{ + struct mm_struct *mm; + struct folio *folio; + unsigned long old = TEST_VA, new = SZ_1G - SZ_4K, mapped, moved; + pte_t saved[TEST_SLOTS]; + u8 byte = 0x55; + int ret; + unsigned int i; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + if (!user4k_test_enabled && !IS_ENABLED(CONFIG_KASAN)) { + kunit_skip(test, "borrowed-mm mremap fixture requires hardware opt-in"); + return; + } + mm = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, mm); + kthread_use_mm(mm); + mapped = vm_mmap(NULL, old, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, mapped, old); + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, old, &byte, 1, FOLL_WRITE), 1); + folio = test_fault_folio(mm, old); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, 0x81 + i, SZ_4K); + test_swap_tags_set(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + for (i = 0; i < TEST_SLOTS; i++) { + saved[i] = test_fault_entry(test, mm, old + i * SZ_4K); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(softleaf_from_pte(saved[i]))); + } + kthread_use_mm(mm); + moved = test_mremap(old, PAGE_SIZE, PAGE_SIZE, MREMAP_FIXED | MREMAP_MAYMOVE, new); + kthread_unuse_mm(mm); + KUNIT_ASSERT_EQ(test, moved, new); + KUNIT_EXPECT_EQ(test, mm->committed_user_pages, (unsigned long)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), (unsigned long)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) { + pte_t entry = test_fault_entry(test, mm, new + i * SZ_4K); + + KUNIT_EXPECT_EQ(test, softleaf_from_pte(entry).val, softleaf_from_pte(saved[i]).val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(entry), i * (unsigned int)SZ_4K); + } + for (i = 0; i < TEST_SLOTS; i++) { + unsigned int slot = (i + 2) % TEST_SLOTS; + unsigned long addr = new + slot * SZ_4K; + + KUNIT_ASSERT_EQ(test, access_remote_vm(mm, addr, &byte, 1, 0), 1); + KUNIT_EXPECT_EQ(test, byte, (u8)(0x81 + slot)); + folio = test_fault_folio(mm, addr); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, slot)); + folio_put(folio); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), (unsigned long)TEST_SLOTS); +} + +static void user4k_swap_pin_test(struct kunit *test) +{ + struct vm_area_struct *vma; + struct fragment_test_pins *pins; + struct folio *folio; + phys_addr_t pinned_phys; + unsigned int i; + vm_fault_t fault; + long ret; + pte_t pte; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + ret = fragment_test_pin(pins, vma->vm_mm, TEST_VA, PAGE_SIZE, FOLL_WRITE, 4); + KUNIT_ASSERT_EQ(test, ret, 4L); + for (i = 0; i < 4; i++) + memset(folio_address(pins->fragments[i].folio) + pins->fragments[i].offset, + 0xb1 + i, pins->fragments[i].length); + test_swap_tags_set(pins->fragments[0].folio); + fragment_test_unpin(pins); + ret = fragment_test_pin(pins, vma->vm_mm, TEST_VA + SZ_4K, SZ_4K, FOLL_WRITE, 1); + KUNIT_ASSERT_EQ(test, ret, 1L); + pinned_phys = mm_subpage_phys(pins->fragments[0].subpage); + folio = pins->fragments[0].folio; + folio_lock(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + KUNIT_ASSERT_EQ(test, ret, 0L); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA + SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte), pinned_phys); + KUNIT_EXPECT_TRUE(test, mm_subpage_is_exclusive(pins->fragments[0].subpage)); + KUNIT_EXPECT_EQ(test, mm_subpage_pincount(pins->fragments[0].subpage), 1U); + memset(folio_address(folio) + SZ_4K, 0xd2, SZ_4K); + fragment_test_unpin(pins); + /* Once released, the formerly pinned PTE can be swapped normally. */ + folio = test_fault_folio(vma->vm_mm, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + try_to_unmap(folio, TTU_SYNC); + ret = folio_mapcount(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0L); + for (i = 0; i < 4; i++) { + unsigned long addr = TEST_VA + i * SZ_4K; + + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, addr, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + folio = test_fault_folio(vma->vm_mm, addr); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + i == 1 ? 0xd2 : 0xb1 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, i)); + folio_put(folio); + } +} + +#ifdef CONFIG_MIGRATION +static void user4k_swap_migration_test(struct kunit *test) +{ + struct vm_area_struct *vma, *cvma; + struct mm_struct *child; + struct folio *folio; + phys_addr_t old_phys; + unsigned int i, side, succeeded = 0; + vm_fault_t fault; + int ret; + LIST_HEAD(folios); + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + vma = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + old_phys = PFN_PHYS(folio_pfn(folio)); + folio_lock(folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, 0xa1 + i, SZ_4K); + test_swap_tags_set(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + mmap_read_lock(child); + fault = handle_mm_fault(cvma, TEST_VA + SZ_4K, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + lru_add_drain_all(); + folio = test_fault_folio(child, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, folio_test_swapcache(folio)); + ret = isolate_folio_to_list(folio, &folios) ? 0 : -EBUSY; + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, succeeded, 1U); + /* One present slot and seven swap entries must all follow the new cache folio. */ + for (side = 0; side < 2; side++) { + struct vm_area_struct *area = side ? cvma : vma; + struct mm_struct *mm = area->vm_mm; + + for (i = 0; i < TEST_SLOTS; i++) { + unsigned long addr = TEST_VA + i * SZ_4K; + + mmap_read_lock(mm); + fault = handle_mm_fault(area, addr, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + folio = test_fault_folio(mm, addr); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_NE(test, PFN_PHYS(folio_pfn(folio)), old_phys); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0xa1 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, i)); + folio_put(folio); + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), (unsigned long)TEST_SLOTS); + } +} +#endif + +static void user4k_swap_cached_test(struct kunit *test) +{ + test_user4k_swap_roundtrip(test, 0, 0x51); +} + +static void user4k_swap_disk_test(struct kunit *test) +{ + test_user4k_swap_roundtrip(test, 1, 0x61); +} + +static void user4k_swap_reclaim_test(struct kunit *test) +{ + test_user4k_swap_roundtrip(test, 2, 0x71); +} + +static void user4k_swap_stress_test(struct kunit *test) +{ + unsigned int i; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + for (i = 0; i < 32; i++) + test_user4k_swap_roundtrip(test, i % 3, 0x21 + i * 5); + kunit_info(test, "32 swap/fork/COW/exit cycles with distinct data patterns\n"); +} + +static void user4k_swapoff_test(struct kunit *test) +{ + struct vm_area_struct *parent, *cvma; + struct mm_struct *child; + struct folio *folio; + bool *disabled; + vm_fault_t fault; + unsigned int i, side; + swp_entry_t entry; + int ret; + pte_t pte; + LIST_HEAD(folios); + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + parent = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, parent); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + lru_add_drain_all(); + folio = test_fault_folio(parent->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + for (i = 0; i < TEST_SLOTS; i++) + memset(folio_address(folio) + i * SZ_4K, 0x91 + i, SZ_4K); + test_swap_tags_set(folio); + ret = folio_isolate_lru(folio) ? 0 : -EBUSY; + if (!ret) + list_add(&folio->lru, &folios); + folio_put(folio); + reclaim_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + pte = test_fault_entry(test, parent->vm_mm, TEST_VA); + entry = softleaf_from_pte(pte); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent->vm_mm, MM_SWAPENTS), (unsigned long)TEST_SLOTS); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_SWAPENTS), (unsigned long)TEST_SLOTS); + ret = test_swap_control("--swapoff-helper"); + *disabled = !total_swap_pages; + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, *disabled); + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : parent->vm_mm; + + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), (unsigned long)TEST_SLOTS); + for (i = 0; i < TEST_SLOTS; i++) { + pte = test_fault_entry(test, mm, TEST_VA + i * SZ_4K); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte) & ~PAGE_MASK, + i * (unsigned long)SZ_4K); + folio = test_fault_folio(mm, TEST_VA + i * SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0x91 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, i)); + folio_put(folio); + } + } + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + mmap_read_lock(child); + fault = handle_mm_fault(cvma, TEST_VA + 3 * SZ_4K, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + pte = test_fault_entry(test, child, TEST_VA + 3 * SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, pte), pte_phys_mm(parent->vm_mm, + test_fault_entry(test, parent->vm_mm, TEST_VA + 3 * SZ_4K))); + ret = test_swap_control("--swapon-helper"); + if (!ret) + *disabled = false; + KUNIT_EXPECT_EQ(test, ret, 0); +} + +static void user4k_swapoff_boundaries_test(struct kunit *test) +{ + const unsigned long boundaries[] = { SZ_2M, SZ_1G, 1UL << 39 }; + bool *disabled; + unsigned int boundary; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + for (boundary = 0; boundary < ARRAY_SIZE(boundaries); boundary++) { + unsigned long start = boundaries[boundary] - SZ_4K; + struct vm_area_struct *parent = test_fault_vma(test, start, 2 * SZ_4K); + struct mm_struct *child; + unsigned int i, side; + unsigned int offsets[2]; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, parent); + for (i = 0; i < 2; i++) { + unsigned long addr = start + i * SZ_4K; + unsigned int offset; + struct folio *folio; + vm_fault_t fault; + swp_entry_t entry; + LIST_HEAD(folios); + + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, addr, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + offset = offset_in_page(pte_phys_mm(parent->vm_mm, + test_fault_entry(test, parent->vm_mm, addr))); + offsets[i] = offset; + lru_add_drain_all(); + folio = test_fault_folio(parent->vm_mm, addr); + KUNIT_ASSERT_NOT_NULL(test, folio); + memset(folio_address(folio) + offset, 0xb0 + boundary * 2 + i, SZ_4K); + test_swap_tags_set(folio); + ret = folio_isolate_lru(folio) ? 0 : -EBUSY; + if (!ret) + list_add(&folio->lru, &folios); + folio_put(folio); + reclaim_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + entry = softleaf_from_pte(test_fault_entry(test, parent->vm_mm, addr)); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + } + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = test_swap_control("--swapoff-helper"); + *disabled = !total_swap_pages; + KUNIT_ASSERT_EQ(test, ret, 0); + for (side = 0; side < 2; side++) { + struct mm_struct *mm = side ? child : parent->vm_mm; + + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 2UL); + for (i = 0; i < 2; i++) { + unsigned long addr = start + i * SZ_4K; + unsigned int offset = offsets[i]; + pte_t pte = test_fault_entry(test, mm, addr); + struct folio *folio; + + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_FALSE(test, pte_write(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, pte) & ~PAGE_MASK, + (unsigned long)offset); + folio = test_fault_folio(mm, addr); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + offset, + 0xb0 + boundary * 2 + i, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(folio, offset / SZ_4K)); + folio_put(folio); + } + } + ret = test_swap_control("--swapon-helper"); + if (!ret) + *disabled = false; + KUNIT_ASSERT_EQ(test, ret, 0); + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, parent->vm_mm); + } +} + +struct live_swap_reader { + struct mm_struct *mm; + struct task_struct *task; + struct completion ready, resume, entered, done; + unsigned int id, checks; + int error; +}; + +static int live_swap_reader_run(void *arg) +{ + struct live_swap_reader *r = arg; + unsigned int phase, i; + + kthread_use_mm(r->mm); + for (phase = 0; phase < 2; phase++) { + if (phase) + complete(&r->entered); + for (i = 0; i < 4; i++) { + u8 value = 0; + void __user *addr = (void __user *)(TEST_VA + i * SZ_4K + 64 + r->id); + + /* Resolve AF, swap and COW faults through the real uaccess path. */ + if (copy_from_user(&value, addr, 1) || value != 0xd1 + i) + r->error = phase ? 2 : 1; + r->checks++; + } + if (!phase) { + complete(&r->ready); + if (!wait_for_completion_timeout(&r->resume, 15 * HZ)) { + r->error = 3; + goto out; + } + } + } + { + u8 value = 0xe0 + r->id; + void __user *addr = (void __user *)(TEST_VA + SZ_4K + 64 + r->id); + + if (copy_to_user(addr, &value, 1)) + r->error = 4; + r->checks++; + } +out: + kthread_unuse_mm(r->mm); + complete(&r->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void live_swap_reader_stop(void *arg) +{ + struct live_swap_reader *r = arg; + + if (r->task) { + complete_all(&r->resume); + kthread_stop(r->task); + r->task = NULL; + } +} + +static void test_live_fault_roundtrip(struct kunit *test, unsigned int mode) +{ + bool migration = mode != 0; + struct vm_area_struct *parent; + struct mm_struct *child; + struct live_swap_reader *readers; + struct folio *folio; + vm_fault_t fault; + unsigned int i, cpu; + swp_entry_t entry; + u64 tcr = read_sysreg(tcr_el1); + int ret; + LIST_HEAD(folios); + + if ((!migration && !total_swap_pages) || !user4k_test_enabled || num_online_cpus() < 2 || + !system_supports_4kb_granule() || system_supports_cnp() || + cpus_have_final_cap(ARM64_HAS_S1PIE) || cpus_have_final_cap(ARM64_HAS_S1POE) || + (tcr & (TCR_EL1_DS | TCR_EL1_A1 | TCR_EL1_T0SZ_MASK)) != + (TCR_EL1_A1 | TCR_T0SZ(48))) { + kunit_skip(test, "requires two CPUs, compatible opt-in 4K TTBR0 controls, and swap for reclaim"); + return; + } + parent = test_fault_vma(test, TEST_VA, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, parent); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault, (vm_fault_t)0); + folio = test_fault_folio(parent->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + for (i = 0; i < 4; i++) + memset(folio_address(folio) + i * SZ_4K, 0xd1 + i, SZ_4K); + folio_put(folio); + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + readers = kunit_kcalloc(test, 2, sizeof(*readers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, readers); + cpu = cpumask_first(cpu_online_mask); + for (i = 0; i < 2; i++) { + readers[i].mm = child; + readers[i].id = i; + init_completion(&readers[i].ready); + init_completion(&readers[i].resume); + init_completion(&readers[i].entered); + init_completion(&readers[i].done); + readers[i].task = kthread_create(live_swap_reader_run, &readers[i], + "user4k-swap-read-%u", i); + if (IS_ERR(readers[i].task)) { + readers[i].task = NULL; + KUNIT_FAIL(test, "live swap worker creation failed"); + return; + } + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, live_swap_reader_stop, + &readers[i]), 0); + kthread_bind(readers[i].task, cpu); + cpu = cpumask_next(cpu, cpu_online_mask); + wake_up_process(readers[i].task); + } + for (i = 0; i < 2; i++) + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&readers[i].ready, 10 * HZ), 0UL); + lru_add_drain_all(); + folio = test_fault_folio(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + if (mode == 2) { +#ifdef CONFIG_MIGRATION + struct folio *dst = folio_alloc(GFP_KERNEL, 0); + bool entered = true, blocked = true, unmapped = false; + + if (!dst) { + folio_put(folio); + KUNIT_FAIL(test, "migration waiter destination allocation failed"); + return; + } + folio_lock(folio); + folio_lock(dst); + ret = mm_subpage_migrate_prepare(folio, dst); + if (!ret) { + try_to_migrate(folio, TTU_SYNC); + unmapped = !folio_mapped(folio); + for (i = 0; i < 2; i++) + complete(&readers[i].resume); + for (i = 0; i < 2; i++) { + entered &= !!wait_for_completion_timeout(&readers[i].entered, HZ); + if (wait_for_completion_timeout(&readers[i].done, max(1UL, HZ / 20UL))) { + blocked = false; + complete(&readers[i].done); + } + } + remove_migration_ptes(folio, folio, 0); + mm_subpage_migrate_finish(folio); + mm_subpage_migrate_finish(dst); + } + folio_unlock(dst); + folio_unlock(folio); + folio_put(dst); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_TRUE(test, entered); + KUNIT_EXPECT_TRUE(test, blocked); + KUNIT_EXPECT_TRUE(test, unmapped); +#endif + } else if (migration) { +#ifdef CONFIG_MIGRATION + unsigned int succeeded = 0; + phys_addr_t old_phys = PFN_PHYS(folio_pfn(folio)); + + ret = isolate_folio_to_list(folio, &folios) ? 0 : -EBUSY; + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_ASYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, succeeded, 1U); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, test_fault_entry(test, child, TEST_VA)), old_phys); +#endif + } else { + ret = folio_isolate_lru(folio) ? 0 : -EBUSY; + if (!ret) + list_add(&folio->lru, &folios); + folio_put(folio); + reclaim_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + entry = softleaf_from_pte(test_fault_entry(test, child, TEST_VA)); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + } + for (i = 0; i < 2; i++) + complete(&readers[i].resume); + for (i = 0; i < 2; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&readers[i].done, 15 * HZ), 0UL); + live_swap_reader_stop(&readers[i]); + KUNIT_EXPECT_EQ(test, readers[i].error, 0); + KUNIT_EXPECT_EQ(test, readers[i].checks, 9U); + } + folio = test_fault_folio(child, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + { + pte_t pte = test_fault_entry(test, child, TEST_VA + SZ_4K); + unsigned long offset = pte_phys_mm(child, pte) - PFN_PHYS(folio_pfn(folio)); + + for (i = 0; i < 2; i++) + KUNIT_EXPECT_EQ(test, *((u8 *)folio_address(folio) + offset + 64 + i), + (u8)(0xe0 + i)); + } + folio_put(folio); + mmap_read_lock(parent->vm_mm); + fault = handle_mm_fault(parent, TEST_VA + SZ_4K, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(parent->vm_mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + folio = test_fault_folio(parent->vm_mm, TEST_VA + SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0xd2, SZ_4K), NULL); + folio_put(folio); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_SWAPENTS), 0UL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 4UL); + kunit_info(test, "two CPUs access through uaccess after %s, then write one child slot\n", + mode == 2 ? "migration-entry rollback" : migration ? "migration" : "reclaim"); +} + +static void user4k_live_swap_fault_test(struct kunit *test) +{ + test_live_fault_roundtrip(test, 0); +} + +#ifdef CONFIG_MIGRATION +static void user4k_live_migration_test(struct kunit *test) +{ + test_live_fault_roundtrip(test, 1); +} + +static void user4k_migration_wait_test(struct kunit *test) +{ + test_live_fault_roundtrip(test, 2); +} +#endif + +/* Transfer PTE ownership to normal mm teardown, which also handles swap. */ +static int anon_slot_fixture_handoff(struct kunit *test, struct anon_slot_fixture *f) +{ + struct mm_struct *mm = f->mm; + + f->mm = NULL; + f->vma = NULL; + memset(f->slots, 0, sizeof(f->slots)); + memset(f->mapped, 0, sizeof(f->mapped)); + return kunit_add_action_or_reset(test, user4k_vma_mm_free, mm); +} + +static void test_nonlinear_swap(struct kunit *test, bool evict) +{ + unsigned int first; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + for (first = 0; first < 2; first++) { + struct anon_slot_fixture *a = anon_slot_fixture_create(test); + struct anon_slot_fixture *b = anon_slot_fixture_create(test); + struct vm_area_struct *vmas[2]; + struct mm_struct *child; + struct mm_subpage *slot; + struct folio *folio; + swp_entry_t entry = { }; + unsigned int side; + vm_fault_t fault; + int err, mapped; + + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + vmas[0] = a->vma; + vmas[1] = b->vma; + slot = mm_subpage_alloc_at(a->pool, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + folio = mm_subpage_folio(slot); + err = mem_cgroup_charge(folio, a->mm, GFP_KERNEL); + if (err) { + mm_subpage_put(slot); + KUNIT_FAIL(test, "cannot charge swap fixture: %d", err); + return; + } + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(a, 0, slot), 0); + folio_lock(folio); + mm_subpage_set_exclusive(slot); + folio_unlock(folio); + slot = mm_subpage_alloc_at(a->pool, 3 * SZ_4K); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, slot); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_map(b, 1, slot), 0); + folio_lock(folio); + mm_subpage_set_exclusive(slot); + folio_unlock(folio); + memset(folio_address(folio), 0x5b, SZ_4K); + memset(folio_address(folio) + 3 * SZ_4K, 0xd3, SZ_4K); + test_swap_tags_set(folio); + folio_mark_dirty(folio); + folio_add_lru(folio); + lru_add_drain_all(); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_handoff(test, a), 0); + KUNIT_ASSERT_EQ(test, anon_slot_fixture_handoff(test, b), 0); + folio_get(folio); + folio_lock(folio); + err = folio_alloc_swap(folio); + if (!err) { + entry = folio->swap; + try_to_unmap(folio, TTU_SYNC); + } + mapped = folio_mapcount(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_ASSERT_EQ(test, mapped, 0); + for (side = 0; side < 2; side++) { + pte_t pte = test_fault_entry(test, vmas[side]->vm_mm, + TEST_VA + side * SZ_4K); + + KUNIT_EXPECT_EQ(test, softleaf_from_pte(pte).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(pte), side ? 3U * SZ_4K : 0U); + KUNIT_EXPECT_EQ(test, swap_subpage_count(entry, side ? 3 * SZ_4K : 0), 1U); + } + if (evict) + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(entry)); + /* Preserve physical offsets when a nonpresent nonlinear slot is forked. */ + child = test_vma_mm(test); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + err = dup_mmap(child, vmas[1]->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, err, 0); + for (side = 0; side < 2; side++) { + unsigned int which = side ^ first, physical = which ? 3 : 0; + struct vm_area_struct *vma = vmas[which]; + unsigned long address = TEST_VA + which * SZ_4K; + pte_t pte; + + mmap_read_lock(vma->vm_mm); + fault = handle_mm_fault(vma, address, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(vma->vm_mm); + KUNIT_ASSERT_EQ_MSG(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), + (vm_fault_t)0, "evict=%d first=%u side=%u", evict, first, which); + KUNIT_EXPECT_EQ(test, !!(fault & VM_FAULT_MAJOR), evict && !side); + pte = test_fault_entry(test, vma->vm_mm, address); + slot = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_EQ(test, mm_subpage_offset(slot), physical * (unsigned int)SZ_4K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), which ? 0xd3 : 0x5b, SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, test_swap_tags_match(mm_subpage_folio(slot), physical)); + mm_subpage_put(slot); + } + /* A child's write must copy only its slot at the new virtual offset. */ + { + struct vm_area_struct *vma = test_vma_lookup(child, TEST_VA + SZ_4K); + pte_t pte; + + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(child); + fault = handle_mm_fault(vma, TEST_VA + SZ_4K, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(child); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + pte = test_fault_entry(test, child, TEST_VA + SZ_4K); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + slot = mm_subpage_get_from_phys(pte_phys_mm(child, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0xd3, SZ_4K), NULL); + memset(folio_address(mm_subpage_folio(slot)) + mm_subpage_offset(slot), 0x91, SZ_4K); + mm_subpage_put(slot); + pte = test_fault_entry(test, vmas[1]->vm_mm, TEST_VA + SZ_4K); + slot = mm_subpage_get_from_phys(pte_phys_mm(vmas[1]->vm_mm, pte)); + KUNIT_ASSERT_NOT_NULL(test, slot); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(mm_subpage_folio(slot)) + + mm_subpage_offset(slot), 0xd3, SZ_4K), NULL); + mm_subpage_put(slot); + } + kunit_release_action(test, user4k_vma_mm_free, child); + kunit_release_action(test, user4k_vma_mm_free, vmas[1]->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, vmas[0]->vm_mm); + } +} + +static void user4k_nonlinear_swap_cached_test(struct kunit *test) +{ + test_nonlinear_swap(test, false); +} + +static void user4k_nonlinear_swap_disk_test(struct kunit *test) +{ + test_nonlinear_swap(test, true); +} + +#ifdef CONFIG_USERFAULTFD +/* The context is attached only during the call; ordinary mm cleanup owns the VMAs. */ +static ssize_t test_uffd_move(struct vm_area_struct *dst, unsigned long dst_addr, + unsigned long src_addr, unsigned long size) +{ + struct userfaultfd_ctx ctx = { .mm = dst->vm_mm }; + ssize_t ret; + + init_rwsem(&ctx.map_changing_lock); + atomic_set(&ctx.mmap_changing, 0); + mmap_write_lock(ctx.mm); + vma_start_write(dst); + dst->vm_userfaultfd_ctx.ctx = &ctx; + mmap_write_unlock(ctx.mm); + ret = move_pages(&ctx, dst_addr, src_addr, size, 0); + mmap_write_lock(ctx.mm); + vma_start_write(dst); + dst->vm_userfaultfd_ctx.ctx = NULL; + mmap_write_unlock(ctx.mm); + return ret; +} + +#ifdef CONFIG_MIGRATION +struct move_migration_worker { + struct userfaultfd_ctx ctx; + struct vm_area_struct *vma[2]; + struct task_struct *task; + struct completion ready; + unsigned long address[2], size; + unsigned long moves, busy; + int error; +}; + +static int move_migration_worker_run(void *arg) +{ + struct move_migration_worker *w = arg; + + while (!kthread_should_stop()) { + unsigned int side = w->moves & 1; + ssize_t ret = move_pages(&w->ctx, w->address[side ^ 1], + w->address[side], w->size, 0); + + if (ret == w->size) + WRITE_ONCE(w->moves, w->moves + 1); + else if (ret == -EAGAIN || ret == -EBUSY) + w->busy++; + else + w->error = ret < 0 ? ret : -EIO; + if (w->moves || w->error) + complete_all(&w->ready); + if (w->error) { + /* Keep the task valid until the owner joins it. */ + set_current_state(TASK_INTERRUPTIBLE); + if (!kthread_should_stop()) + schedule(); + __set_current_state(TASK_RUNNING); + } + /* Leave bounded gaps so migration can complete between contended moves. */ + if (ret == w->size) + usleep_range(1000, 2000); + else + cond_resched(); + } + return 0; +} + +static void move_migration_worker_stop(void *arg) +{ + struct move_migration_worker *w = arg; + unsigned int i; + + if (w->task) { + kthread_stop(w->task); + w->task = NULL; + } + mmap_write_lock(w->ctx.mm); + for (i = 0; i < 2; i++) { + vma_start_write(w->vma[i]); + w->vma[i]->vm_userfaultfd_ctx.ctx = NULL; + } + mmap_write_unlock(w->ctx.mm); +} + +static void user4k_move_migration_race_test(struct kunit *test) +{ + unsigned int shift; + + for (shift = 12; shift < PAGE_SHIFT; shift += 2) { + struct mm_struct *mm = test_vma_mm_granule(test, shift); + struct move_migration_worker *w; + unsigned long size = 1UL << shift, i, initial_moves; + unsigned int round, migrated = 0, failed = 0; + pte_t pte; + vm_fault_t fault; + + KUNIT_ASSERT_NOT_NULL(test, mm); + w = kunit_kzalloc(test, sizeof(*w), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, w); + w->ctx.mm = mm; + init_rwsem(&w->ctx.map_changing_lock); + atomic_set(&w->ctx.mmap_changing, 0); + init_completion(&w->ready); + w->size = size; + w->address[0] = TEST_VA + 3 * size; + w->address[1] = TEST_VA + SZ_128M + size; + w->vma[0] = test_fault_vma_in_mm(mm, TEST_VA, PAGE_SIZE); + w->vma[1] = test_fault_vma_in_mm(mm, TEST_VA + SZ_128M, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, w->vma[0]); + KUNIT_ASSERT_NOT_NULL(test, w->vma[1]); + for (i = 0; i < PAGE_SIZE; i += size) { + mmap_read_lock(mm); + fault = handle_mm_fault(w->vma[0], TEST_VA + i, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + pte = test_fault_entry(test, mm, TEST_VA + i); + memset(phys_to_virt(pte_phys_mm(mm, pte)), 0x60 + i / size, size); + } + /* Both directions are registered before starting the concurrent mover. */ + mmap_write_lock(mm); + for (i = 0; i < 2; i++) { + vma_start_write(w->vma[i]); + w->vma[i]->vm_userfaultfd_ctx.ctx = &w->ctx; + } + mmap_write_unlock(mm); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, move_migration_worker_stop, w), 0); + w->task = kthread_create(move_migration_worker_run, w, "user4k-move"); + if (IS_ERR(w->task)) { + w->task = NULL; + KUNIT_FAIL(test, "MOVE worker creation failed"); + return; + } + if (num_online_cpus() > 1) + kthread_bind(w->task, cpumask_last(cpu_online_mask)); + wake_up_process(w->task); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&w->ready, 10 * HZ), 0UL); + initial_moves = READ_ONCE(w->moves); + lru_add_drain_all(); + for (round = 0; round < 64; round++) { + struct folio *folio; + unsigned int succeeded = 0; + bool isolated; + int ret; + LIST_HEAD(folios); + + lru_add_drain_all(); + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + isolated = isolate_folio_to_list(folio, &folios); + folio_put(folio); + KUNIT_ASSERT_TRUE_MSG(test, isolated, "round=%u", round); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, + MIGRATE_SYNC, MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_EXPECT_GE(test, ret, 0); + migrated += succeeded; + failed += ret > 0; + cond_resched(); + } + kunit_release_action(test, move_migration_worker_stop, w); + kunit_info(test, "%uK MOVE/migration: moves=%lu retries=%lu migrated=%u busy=%u\n", + 1U << (shift - 10), w->moves, w->busy, migrated, failed); + KUNIT_EXPECT_EQ(test, w->error, 0); + KUNIT_EXPECT_GT(test, w->moves, initial_moves); + KUNIT_EXPECT_GT(test, migrated, 0U); + for (i = 0; i < PAGE_SIZE; i += size) { + unsigned long addr = i == 3 * size ? w->address[w->moves & 1] : TEST_VA + i; + + pte = test_fault_entry(test, mm, addr); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(mm, pte)), + 0x60 + i / size, size), NULL); + } + KUNIT_EXPECT_TRUE(test, pte_none(test_fault_entry(test, mm, w->address[(w->moves & 1) ^ 1]))); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} + +#endif + +static void user4k_move_swap_get_test(struct kunit *test) +{ + unsigned int variant; + + if (!total_swap_pages) { + kunit_skip(test, "requires the disposable guest swap runner"); + return; + } + for (variant = 0; variant < PAGE_SHIFT - 12; variant++) { + unsigned int shift = 12 + (variant / 2) * 2; + bool forked = variant & 1; + unsigned long size = 1UL << shift; + unsigned long src_addr = TEST_VA + 3 * size; + unsigned long dst_addr = TEST_VA + SZ_128M + size; + struct mm_struct *mm = test_vma_mm_granule(test, shift); + struct vm_area_struct *src, *dst; + struct mm_subpage *held; + struct folio *folio; + phys_addr_t physical; + swp_entry_t entry = { }; + pte_t before, after; + vm_fault_t fault; + unsigned long i; + int err, mapped; + + KUNIT_ASSERT_NOT_NULL(test, mm); + src = test_fault_vma_in_mm(mm, TEST_VA, PAGE_SIZE); + dst = test_fault_vma_in_mm(mm, TEST_VA + SZ_128M, PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, src); + KUNIT_ASSERT_NOT_NULL(test, dst); + for (i = 0; i < PAGE_SIZE; i += size) { + mmap_read_lock(mm); + fault = handle_mm_fault(src, TEST_VA + i, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + before = test_fault_entry(test, mm, TEST_VA + i); + memset(phys_to_virt(pte_phys_mm(mm, before)), 0x50 + i / size, size); + } + before = test_fault_entry(test, mm, src_addr); + physical = pte_phys_mm(mm, before); + held = mm_subpage_get_from_phys(physical); + KUNIT_ASSERT_NOT_NULL(test, held); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_subpage_put, held), 0); + KUNIT_ASSERT_EQ(test, mm_subpage_offset(held), (unsigned int)(3 * size)); + folio = mm_subpage_folio(held); + lru_add_drain_all(); + folio_lock(folio); + err = folio_alloc_swap(folio); + if (!err) { + entry = folio->swap; + try_to_unmap(folio, TTU_SYNC); + } + mapped = folio_mapcount(folio); + folio_unlock(folio); + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_ASSERT_EQ(test, mapped, 0); + KUNIT_ASSERT_EQ(test, mm_subpage_mapcount(held), 0); + KUNIT_ASSERT_EQ(test, swap_subpage_count(entry, 3 * size), 1U); + before = test_fault_entry(test, mm, src_addr); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(softleaf_from_pte(before))); + if (forked) { + struct mm_struct *child = test_vma_mm_granule(test, shift); + + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + err = dup_mmap(child, mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_EXPECT_EQ(test, test_uffd_move(dst, dst_addr, src_addr, size), + (ssize_t)-EBUSY); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(test_fault_entry(test, mm, + src_addr)).val, entry.val); + kunit_release_action(test, user4k_vma_mm_free, child); + before = test_fault_entry(test, mm, src_addr); + KUNIT_EXPECT_FALSE(test, pte_swp_exclusive(before)); + KUNIT_ASSERT_EQ(test, swap_subpage_count(entry, 3 * size), 1U); + } + KUNIT_ASSERT_EQ(test, test_uffd_move(dst, dst_addr, src_addr, size), (ssize_t)size); + after = test_fault_entry(test, mm, dst_addr); + KUNIT_EXPECT_EQ(test, softleaf_from_pte(after).val, entry.val); + KUNIT_EXPECT_EQ(test, pte_swp_subpage_offset(after), (unsigned int)(3 * size)); + KUNIT_EXPECT_TRUE(test, pte_none(test_fault_entry(test, mm, src_addr))); + /* Restore with the retained metadata already rebound to the new root. */ + mmap_read_lock(mm); + fault = handle_mm_fault(dst, dst_addr, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + after = test_fault_entry(test, mm, dst_addr); + KUNIT_ASSERT_TRUE(test, pte_present(after)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, after), physical); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(physical), 0x53, size), NULL); + KUNIT_EXPECT_EQ(test, mm_subpage_is_exclusive(held), !forked); + /* Fork-cleared exclusivity requires COW while a GET is retained. */ + mmap_read_lock(mm); + fault = handle_mm_fault(dst, dst_addr, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + after = test_fault_entry(test, mm, dst_addr); + KUNIT_ASSERT_TRUE(test, pte_write(after)); + if (forked) + KUNIT_EXPECT_NE(test, pte_phys_mm(mm, after), physical); + else + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, after), physical); + memset(phys_to_virt(pte_phys_mm(mm, after)), 0xa9, size); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(physical), + forked ? 0x53 : 0xa9, size), NULL); + for (i = 0; i < PAGE_SIZE; i += size) { + if (i == 3 * size) + continue; + mmap_read_lock(mm); + fault = handle_mm_fault(src, TEST_VA + i, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), (vm_fault_t)0); + after = test_fault_entry(test, mm, TEST_VA + i); + KUNIT_ASSERT_TRUE(test, pte_present(after)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(phys_to_virt(pte_phys_mm(mm, after)), + 0x50 + i / size, size), NULL); + } + kunit_release_action(test, test_subpage_put, held); + kunit_release_action(test, user4k_vma_mm_free, mm); + } +} +#endif + +static struct kunit_case user4k_swap_cases[] = { + KUNIT_CASE(user4k_swap_reuse_get_test), + KUNIT_CASE(user4k_swap_reuse_swapoff_test), + KUNIT_CASE(user4k_swap_reuse_cached_test), + KUNIT_CASE(user4k_swap_reuse_disk_test), + KUNIT_CASE(user4k_swap_exclusive_cached_test), + KUNIT_CASE(user4k_swap_exclusive_disk_test), + KUNIT_CASE(user4k_swap_exclusive_swapoff_test), + KUNIT_CASE(user4k_swap_cached_test), + KUNIT_CASE(user4k_swap_pin_test), + KUNIT_CASE(user4k_swap_mremap_test), +#ifdef CONFIG_MIGRATION + KUNIT_CASE(user4k_swap_migration_test), +#endif + KUNIT_CASE(user4k_swap_disk_test), + KUNIT_CASE(user4k_swap_reclaim_test), + KUNIT_CASE(user4k_swapoff_test), + KUNIT_CASE(user4k_swap_stress_test), + KUNIT_CASE(user4k_swapoff_boundaries_test), + KUNIT_CASE(user4k_live_swap_fault_test), + {} +}; + +static void user4k_swap_suite_exit(struct kunit_suite *suite) +{ + kvfree_rcu_barrier(); +} + +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) +static void test_shmem_uffd_swap(struct kunit *test, unsigned int mode) +{ + const unsigned long missing = GENMASK(PAGE_SIZE / SZ_4K - 1, mode == 3 ? 2 : 1); + struct swap_iocb *plug = NULL; + struct file *file; + struct folio *folio; + void *entry; + bool *disabled; + int ret; + LIST_HEAD(folios); + + if (!total_swap_pages) { + kunit_skip(test, "requires disposable guest swap"); + return; + } + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + file = test_cache_file(test, PAGE_SIZE, false); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + ret = shmem_get_folio(file_inode(file), 0, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + memset(folio_address(folio), 0xb6, mode == 3 ? 2 * SZ_4K : SZ_4K); + ret = shmem_uffd_set_missing(folio, 0, missing); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + /* Match pageout(): submitted data is clean unless writeout redirties it. */ + KUNIT_EXPECT_TRUE(test, folio_clear_dirty_for_io(folio)); + ret = shmem_writeout(folio, &plug, &folios); + if (plug) + swap_write_unplug(plug); + if (ret == AOP_WRITEPAGE_ACTIVATE) + folio_unlock(folio); + folio_wait_writeback(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_TRUE(test, list_empty(&folios)); + entry = filemap_get_entry(file->f_mapping, 0); + if (entry && !xa_is_value(entry)) + folio_put(entry); + KUNIT_ASSERT_TRUE(test, xa_is_value(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(radix_to_swp_entry(entry))); + KUNIT_EXPECT_EQ(test, xa_to_value(xa_load(&SHMEM_I(file_inode(file))->uffd_missing, 0)), missing); + if (mode == 1) { + shmem_truncate_range(file_inode(file), 0, PAGE_SIZE - 1); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + } else if (mode == 3) { + shmem_truncate_range(file_inode(file), 0, SZ_4K - 1); + } else if (mode == 2) { + ret = test_swap_control("--swapoff-helper"); + *disabled = ret == 0; + KUNIT_ASSERT_EQ(test, ret, 0); + } + ret = shmem_get_folio(file_inode(file), 0, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), + mode == 1 ? 0UL : mode == 3 ? missing | BIT(0) : missing); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), + mode == 1 || mode == 3 ? 0 : 0xb6, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, + mode == 3 ? 0xb6 : 0, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + 2 * SZ_4K, + 0, PAGE_SIZE - 2 * SZ_4K), NULL); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + folio_unlock(folio); + folio_put(folio); +} + +static void user4k_shmem_uffd_swapin_test(struct kunit *test) +{ + test_shmem_uffd_swap(test, 0); +} + +static void user4k_shmem_uffd_swapped_truncate_test(struct kunit *test) +{ + test_shmem_uffd_swap(test, 1); +} + +static void user4k_shmem_uffd_swapped_partial_hole_test(struct kunit *test) +{ + test_shmem_uffd_swap(test, 3); +} + +static void user4k_shmem_uffd_swapoff_test(struct kunit *test) +{ + test_shmem_uffd_swap(test, 2); +} + +#if defined(CONFIG_FAILSLAB) && defined(CONFIG_FAULT_INJECTION_DEBUG_FS) +static void test_disable_failslab(void *unused) +{ + test_swap_control("--failslab-off-helper"); +} +#endif + +static void user4k_shmem_uffd_swap_alloc_fail_test(struct kunit *test) +{ +#if defined(CONFIG_FAILSLAB) && defined(CONFIG_FAULT_INJECTION_DEBUG_FS) + unsigned int attempt; + + if (!total_swap_pages) { + kunit_skip(test, "requires disposable guest swap"); + return; + } + KUNIT_ASSERT_EQ(test, test_swap_control("--failslab-on-helper"), 0); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_disable_failslab, NULL), 0); + for (attempt = 1; attempt <= 2; attempt++) { + pgoff_t index = attempt == 1 ? 1 : 64; + const unsigned long missing = GENMASK(PAGE_SIZE / SZ_4K - 1, 1); + struct file *file = test_cache_file(test, (index + 1) * PAGE_SIZE, false); + struct swap_iocb *plug = NULL; + struct folio *folio; + unsigned int pending; + unsigned long pfn; + void *entry; + int ret; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + ret = shmem_get_folio(file_inode(file), index, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + memset(folio_address(folio), 0xb6, SZ_4K); + ret = shmem_uffd_set_missing(folio, index, missing); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + folio = filemap_lock_folio(file->f_mapping, index); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + pfn = folio_pfn(folio); + KUNIT_EXPECT_TRUE(test, folio_clear_dirty_for_io(folio)); + /* Fail this task's selected allocation and its allocator retries. */ + WRITE_ONCE(current->make_it_fail, 1); + WRITE_ONCE(current->fail_nth, attempt); + ret = shmem_writeout(folio, &plug, &folios); + pending = READ_ONCE(current->fail_nth); + WRITE_ONCE(current->fail_nth, 0); + WRITE_ONCE(current->make_it_fail, 0); + KUNIT_EXPECT_EQ(test, pending, 0U); + KUNIT_EXPECT_EQ(test, ret, AOP_WRITEPAGE_ACTIVATE); + if (ret != AOP_WRITEPAGE_ACTIVATE) { + if (plug) + swap_write_unplug(plug); + folio_wait_writeback(folio); + folio_put(folio); + return; + } + KUNIT_EXPECT_TRUE(test, folio_test_locked(folio)); + KUNIT_EXPECT_TRUE(test, folio_test_dirty(folio)); + KUNIT_EXPECT_FALSE(test, folio_test_swapcache(folio)); + KUNIT_EXPECT_PTR_EQ(test, folio->mapping, file->f_mapping); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, index), missing); + /* A failed multi-level insertion may retain an empty internal node. */ + { + unsigned long cursor = 0; + + KUNIT_EXPECT_PTR_EQ(test, xa_find(&SHMEM_I(file_inode(file))->uffd_missing, + &cursor, ULONG_MAX, XA_PRESENT), NULL); + } + KUNIT_EXPECT_PTR_EQ(test, plug, NULL); + KUNIT_EXPECT_TRUE(test, list_empty(&folios)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xb6, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0, PAGE_SIZE - SZ_4K), NULL); + /* No restart or cache repair: retry exactly the retained folio. */ + KUNIT_EXPECT_EQ(test, folio_pfn(folio), pfn); + KUNIT_EXPECT_TRUE(test, folio_clear_dirty_for_io(folio)); + ret = shmem_writeout(folio, &plug, &folios); + if (plug) + swap_write_unplug(plug); + if (ret == AOP_WRITEPAGE_ACTIVATE) + folio_unlock(folio); + folio_wait_writeback(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + entry = filemap_get_entry(file->f_mapping, index); + if (entry && !xa_is_value(entry)) + folio_put(entry); + KUNIT_ASSERT_TRUE(test, xa_is_value(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(radix_to_swp_entry(entry))); + KUNIT_EXPECT_EQ(test, xa_to_value(xa_load(&SHMEM_I(file_inode(file))->uffd_missing, index)), missing); + ret = shmem_get_folio(file_inode(file), index, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, index), missing); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xb6, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0, PAGE_SIZE - SZ_4K), NULL); + folio_unlock(folio); + folio_put(folio); + kunit_info(test, "failed metadata allocation %u at index %lu; same-folio retry and disk swap-in preserved bytes/mask", attempt, index); + } +#else + kunit_skip(test, "requires FAILSLAB and fault-injection debugfs"); +#endif +} + +#ifdef CONFIG_MEMCG +static void test_memcg_put(void *memcg) +{ + mem_cgroup_put(memcg); +} +#endif + +static void user4k_shmem_uffd_swap_charge_test(struct kunit *test) +{ +#ifdef CONFIG_MEMCG + struct cgroup *cgroup; + struct cgroup_subsys_state *css; + struct mem_cgroup *memcg, *saved; + struct file *file; + struct folio *folio; + struct swap_iocb *plug = NULL; + void *entry; + const unsigned long missing = GENMASK(PAGE_SIZE / SZ_4K - 1, 1); + int ret; + LIST_HEAD(folios); + + if (!total_swap_pages || mem_cgroup_disabled()) { + kunit_skip(test, "requires disposable guest swap and memcg"); + return; + } + KUNIT_ASSERT_EQ(test, test_swap_control("--memcg-setup-helper"), 0); + cgroup = cgroup_get_from_path("/uffd-metadata"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, cgroup); + css = cgroup_get_e_css(cgroup, &memory_cgrp_subsys); + cgroup_put(cgroup); + KUNIT_ASSERT_NOT_NULL(test, css); + memcg = mem_cgroup_from_css(css); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_memcg_put, memcg), 0); + KUNIT_ASSERT_PTR_NE(test, memcg, root_mem_cgroup); + file = test_cache_file(test, 65 * PAGE_SIZE, false); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + saved = set_active_memcg(memcg); + ret = shmem_get_folio(file_inode(file), 64, 0, &folio, SGP_CACHE); + set_active_memcg(saved); + KUNIT_ASSERT_EQ(test, ret, 0); + rcu_read_lock(); + KUNIT_EXPECT_PTR_EQ(test, folio_memcg(folio), memcg); + rcu_read_unlock(); + ret = shmem_uffd_set_missing(folio, 64, missing); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + folio = filemap_lock_folio(file->f_mapping, 64); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, folio_clear_dirty_for_io(folio)); + /* Reclaim from the root charge domain, not the folio's cgroup. */ + saved = set_active_memcg(root_mem_cgroup); + ret = shmem_writeout(folio, &plug, &folios); + set_active_memcg(saved); + if (plug) + swap_write_unplug(plug); + if (ret == AOP_WRITEPAGE_ACTIVATE) + folio_unlock(folio); + folio_wait_writeback(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + rcu_read_lock(); + entry = rcu_dereference(SHMEM_I(file_inode(file))->uffd_missing.xa_head); + if (xa_is_node(entry)) + KUNIT_EXPECT_PTR_EQ(test, mem_cgroup_from_virt(xa_to_node(entry)), memcg); + rcu_read_unlock(); + KUNIT_ASSERT_TRUE(test, xa_is_node(entry)); + entry = filemap_get_entry(file->f_mapping, 64); + if (entry && !xa_is_value(entry)) + folio_put(entry); + KUNIT_ASSERT_TRUE(test, xa_is_value(entry)); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(radix_to_swp_entry(entry))); + ret = shmem_get_folio(file_inode(file), 64, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 64), missing); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + folio_unlock(folio); + folio_put(folio); +#else + kunit_skip(test, "requires memcg"); +#endif +} + +static bool test_coarse_swap_enabled(struct kunit *test) +{ + if (!test_coarse_file_enabled(test)) + return false; + if (!total_swap_pages || !IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + kunit_skip(test, "requires disposable coarse-shmem swap runner"); + return false; + } + return true; +} + +static bool test_coarse_swap_out(struct kunit *test, struct file *file, pgoff_t index) +{ + unsigned int tries; + + for (tries = 0; tries < 100; tries++) { + struct folio *folio; + void *entry = filemap_get_entry(file->f_mapping, index); + LIST_HEAD(folios); + + if (xa_is_value(entry)) + return test_reclaim_swapcache(radix_to_swp_entry(entry)); + if (!entry) { + KUNIT_FAIL(test, "reclaim removed data without a swap entry"); + return false; + } + folio = entry; + lru_add_drain_all(); + if (folio_isolate_lru(folio)) + list_add(&folio->lru, &folios); + folio_put(folio); + /* Real reclaim owns any folios produced by a large-folio split. */ + reclaim_pages(&folios); + schedule_timeout_uninterruptible(1); + } + KUNIT_FAIL(test, "file index %lu did not reach swap", index); + return false; +} + +static void test_coarse_cold_shmem(struct kunit *test, bool promoted) +{ + struct file *file; + struct vm_area_struct *coarse, *small; + struct folio *folio; + unsigned int nr = SZ_64K / PAGE_SIZE, i; + unsigned long offset; + vm_fault_t fault; + + if (!test_coarse_swap_enabled(test)) + return; + file = test_cache_file(test, SZ_64K, true); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + coarse = test_file_vma_granule(test, file, 16, true, TEST_VA, SZ_64K, 0); + small = test_file_vma(test, file, false, true, TEST_VA + SZ_4K, + SZ_64K - SZ_4K, SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, coarse); + KUNIT_ASSERT_NOT_NULL(test, small); + if (promoted) { + KUNIT_ASSERT_EQ(test, test_file_fault(coarse, TEST_VA, false) & VM_FAULT_ERROR, 0U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), SZ_64K); + folio_put(folio); + } + for (offset = SZ_4K; offset < SZ_64K; offset += SZ_4K) + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + lru_add_drain_all(); + for (i = 0; i < nr; i++) + KUNIT_ASSERT_TRUE(test, test_coarse_swap_out(test, file, i)); + /* XArray swap entries and absent swapcache prove this is a cold refault. */ + for (i = 0; i < nr; i++) { + void *entry = filemap_get_entry(file->f_mapping, i); + struct folio *cached = NULL; + + KUNIT_EXPECT_TRUE(test, xa_is_value(entry)); + if (entry && !xa_is_value(entry)) { + folio_put(entry); + return; + } + KUNIT_ASSERT_NOT_NULL(test, entry); + cached = swap_cache_get_folio(radix_to_swp_entry(entry)); + KUNIT_EXPECT_PTR_EQ(test, cached, NULL); + if (cached) + folio_put(cached); + } + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, 0UL); + KUNIT_EXPECT_EQ(test, SHMEM_I(file_inode(file))->swapped, (unsigned long)nr); + fault = test_file_fault(coarse, TEST_VA, false); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY | VM_FAULT_NOPAGE), 0U); + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_MAJOR); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_EXPECT_EQ(test, folio_size(folio), SZ_64K); + for (offset = 0; offset < SZ_64K; offset += SZ_4K) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + offset, + 0x31 + offset / SZ_4K, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, (unsigned long)nr); + KUNIT_EXPECT_EQ(test, SHMEM_I(file_inode(file))->swapped, 0UL); + for (offset = SZ_4K; offset < SZ_64K; offset += SZ_4K) { + pte_t entry; + + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA + offset, false) & + VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, small->vm_mm, TEST_VA + offset); + KUNIT_EXPECT_EQ(test, pte_phys_mm(small->vm_mm, entry), + page_to_phys(&folio->page) + offset); + } + kunit_release_action(test, user4k_vma_mm_free, coarse->vm_mm); + kunit_release_action(test, user4k_vma_mm_free, small->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), (int)nr + 1); +} + +static void user64k_shmem_cold_constituents_test(struct kunit *test) +{ + test_coarse_cold_shmem(test, false); +} + +static void user64k_shmem_cold_promoted_test(struct kunit *test) +{ + test_coarse_cold_shmem(test, true); +} + +#if defined(CONFIG_FAILSLAB) && defined(CONFIG_FAIL_PAGE_ALLOC) && defined(CONFIG_MEMCG) +static void test_coarse_injection_off(void *unused) +{ + test_swap_control("--coarse-injection-off"); +} + +#endif + +static void user64k_shmem_fault_alloc_fail_test(struct kunit *test) +{ +#if defined(CONFIG_FAILSLAB) && defined(CONFIG_FAIL_PAGE_ALLOC) && defined(CONFIG_MEMCG) + unsigned int mode; + + if (!test_coarse_swap_enabled(test)) + return; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_coarse_injection_off, NULL), 0); + for (mode = 0; mode < 3; mode++) { + struct file *file = test_cache_file(test, SZ_64K, false); + struct vm_area_struct *vma; + struct folio *folio; + phys_addr_t original[SZ_64K / PAGE_SIZE]; + unsigned char *buffer; + loff_t pos = 0; + unsigned int i, pending = 0; + vm_fault_t result; + bool prior_fail; + char operation[80]; + int enter = 0, leave = 0; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + /* Make the zero-limit charge case nonblocking and avoid an OOM kill. */ + if (mode == 2) + mapping_set_gfp_mask(file->f_mapping, + mapping_gfp_mask(file->f_mapping) & ~__GFP_DIRECT_RECLAIM); + buffer = kunit_kmalloc(test, SZ_64K, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, buffer); + memset(buffer, 0x95, SZ_64K); + KUNIT_ASSERT_EQ(test, kernel_write(file, buffer, SZ_64K, &pos), (ssize_t)SZ_64K); + vma = test_file_vma_granule(test, file, 16, true, TEST_VA, SZ_64K, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + for (i = 0; i < SZ_64K / PAGE_SIZE; i++) { + folio = filemap_get_folio(file->f_mapping, i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_order(folio), 0U); + original[i] = page_to_phys(&folio->page); + folio_put(folio); + } + if (mode < 2) { + KUNIT_ASSERT_EQ(test, test_swap_control(mode ? "--coarse-page-on" : + "--coarse-slab-on"), 0); + } else { + snprintf(operation, sizeof(operation), "--coarse-memcg-enter=%d", + task_pid_nr(current)); + enter = test_swap_control(operation); + } + /* Direct callback isolates backing allocation from page-table allocation. */ + if (!enter) { + struct vm_fault vmf = { .vma = vma, .address = TEST_VA, + .pgoff = 0, .flags = FAULT_FLAG_REMOTE }; + + mmap_read_lock(vma->vm_mm); + prior_fail = current->make_it_fail; + if (!mode) + WRITE_ONCE(current->fail_nth, 1); + else if (mode == 1) + current->make_it_fail = true; + result = vma->vm_ops->fault(&vmf); + pending = READ_ONCE(current->fail_nth); + WRITE_ONCE(current->fail_nth, 0); + current->make_it_fail = prior_fail; + mmap_read_unlock(vma->vm_mm); + /* Keep unexpected success safe for test cleanup. */ + if (vmf.page) { + if (result & VM_FAULT_LOCKED) + unlock_page(vmf.page); + put_page(vmf.page); + } + } else { + result = 0; + } + if (mode == 2) { + snprintf(operation, sizeof(operation), "--coarse-memcg-leave=%d", + task_pid_nr(current)); + leave = test_swap_control(operation); + } else { + leave = test_swap_control("--coarse-injection-off"); + } + KUNIT_ASSERT_EQ(test, enter, 0); + KUNIT_ASSERT_EQ(test, leave, 0); + KUNIT_EXPECT_EQ(test, pending, 0U); + KUNIT_ASSERT_TRUE(test, result & VM_FAULT_OOM); + for (i = 0; i < SZ_64K / PAGE_SIZE; i++) { + folio = filemap_get_folio(file->f_mapping, i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_order(folio), 0U); + KUNIT_EXPECT_EQ(test, page_to_phys(&folio->page), original[i]); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0x95, PAGE_SIZE), NULL); + folio_put(folio); + } + KUNIT_EXPECT_EQ(test, file->f_mapping->nrpages, SZ_64K / PAGE_SIZE); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SHMEMPAGES), 0L); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & + (VM_FAULT_ERROR | VM_FAULT_RETRY | VM_FAULT_NOPAGE), 0U); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), SZ_64K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0x95, SZ_64K), NULL); + folio_put(folio); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + kunit_info(test, "coarse allocation failure mode %u preserved data and recovered", mode); + } +#else + kunit_skip(test, "requires slab/page fault injection and memcg"); +#endif +} + +static void user64k_anon_swap_test(struct kunit *test) +{ + unsigned int mode, nr = SZ_64K / PAGE_SIZE; + + if (!test_coarse_swap_enabled(test)) + return; + for (mode = 0; mode < 7; mode++) { + struct vm_area_struct *vma, *cvma = NULL; + struct mm_struct *child = NULL; + struct folio *folio, *held = NULL; + swp_entry_t entry, part; + unsigned int i, tries; + pte_t pte; + bool *disabled; + bool warm = !mode || mode == 6; + vm_fault_t fault; + int err; + + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + folio = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(vma->vm_mm, pte)))); + for (i = 0; i < SZ_64K / SZ_4K; i++) + memset(folio_address(folio) + i * SZ_4K, 0x31 + i, SZ_4K); + if (mode == 2 || mode == 6) { + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + err = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, err, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + } + if (warm) { + folio_get(folio); + held = folio; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, held), 0); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + KUNIT_EXPECT_EQ(test, min_order_for_split(folio), 16U - PAGE_SHIFT); +#endif + folio_lock(folio); + KUNIT_EXPECT_EQ(test, folio_anon_min_order(folio), 16U - PAGE_SHIFT); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + KUNIT_EXPECT_EQ(test, split_huge_page_to_list_to_order(&folio->page, NULL, 0), + -EBUSY); + KUNIT_EXPECT_EQ(test, folio_order(folio), 16U - PAGE_SHIFT); + expect_coarse_anon_state(test, folio, child ? 2 : 1, !child); +#endif + err = folio_alloc_swap(folio); + if (!err) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + KUNIT_ASSERT_EQ(test, err, 0); + } else { + for (tries = 0; tries < 100; tries++) { + LIST_HEAD(folios); + + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + if (!pte_present(pte)) + break; + folio = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(vma->vm_mm, pte)))); + folio_get(folio); + lru_add_drain_all(); + if (folio_isolate_lru(folio)) + list_add(&folio->lru, &folios); + folio_put(folio); + reclaim_pages(&folios); + schedule_timeout_uninterruptible(1); + } + } + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_FALSE_MSG(test, pte_present(pte), "mode=%u", mode); + entry = softleaf_from_pte(pte); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_EXPECT_TRUE(test, IS_ALIGNED(swp_offset(entry), nr)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SWAPENTS), 1L); + if (mode == 3) { + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + err = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, err, 0); + } + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), child ? 2 : 1); + if (!warm) + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(part)); + } + if (mode == 3) { + /* Exit a swapped child: every constituent count must be released. */ + kunit_release_action(test, user4k_vma_mm_free, child); + child = NULL; + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), 1); + } + } + if (mode == 5) { +#if defined(CONFIG_FAIL_PAGE_ALLOC) && defined(CONFIG_FAULT_INJECTION) + bool old = current->make_it_fail; + + KUNIT_ASSERT_EQ(test, test_swap_control("--coarse-page-on"), 0); + current->make_it_fail = true; + fault = test_file_fault(vma, TEST_VA, false); + current->make_it_fail = old; + err = test_swap_control("--coarse-injection-off"); + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_OOM); + KUNIT_EXPECT_TRUE(test, pte_same(test_fault_entry(test, vma->vm_mm, TEST_VA), pte)); + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), 1); + } +#endif + } + if (mode == 4) { + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + err = test_swap_control("--swapoff-helper"); + if (!err) + *disabled = true; + KUNIT_ASSERT_EQ(test, err, 0); + } else { + fault = test_file_fault(vma, TEST_VA + SZ_64K - 1, false); + KUNIT_ASSERT_EQ_MSG(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), 0U, + "mode=%u fault=%x", mode, fault); + if (!warm && mode != 5) + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_MAJOR); + } + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE_MSG(test, pte_present(pte), "mode=%u", mode); + folio = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(vma->vm_mm, pte)))); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + if (held) { + KUNIT_EXPECT_PTR_EQ(test, folio, held); /* Warm aligned cache: no copy. */ + } else { + folio_get(folio); + held = folio; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, held), 0); + } + for (i = 0; i < SZ_64K / SZ_4K; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0x31 + i, SZ_4K), NULL); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SWAPENTS), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 1L); + if (child) { + struct folio *copied; + + KUNIT_ASSERT_EQ(test, test_file_fault(cvma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + pte = test_fault_entry(test, child, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, pte), page_to_phys(&folio->page)); + KUNIT_EXPECT_TRUE(test, pte_write(pte)); + copied = page_folio(pfn_to_page(PHYS_PFN(pte_phys_mm(child, pte)))); + expect_coarse_anon_state(test, copied, 1, true); + for (i = 0; i < SZ_64K / SZ_4K; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(copied) + i * SZ_4K, + 0x31 + i, SZ_4K), NULL); + memset(folio_address(copied), 0xa7, SZ_64K); + for (i = 0; i < SZ_64K / SZ_4K; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0x31 + i, SZ_4K), NULL); + kunit_release_action(test, user4k_vma_mm_free, child); + } + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + if (held) { + KUNIT_EXPECT_EQ(test, folio_mapcount(held), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(held), 1); + } + for (i = 0; i < nr; i++) { + struct swap_info_struct *si; + + part = entry; + part.val += i; + si = get_swap_device(part); + if (si) { + KUNIT_EXPECT_FALSE(test, swap_entry_swapped(si, part)); + put_swap_device(si); + } else { + KUNIT_EXPECT_EQ(test, mode, 4U); + } + } + if (mode == 4) + kunit_release_action(test, test_reenable_swap, disabled); + kunit_info(test, "completed coarse anonymous swap mode %u", mode); + } +} + +/* The caller's private fixture has no concurrent PTE writers during reclaim. */ +static void test_coarse_anon_reclaim_once(struct vm_area_struct *vma) +{ + struct folio *folio = test_fault_folio(vma->vm_mm, vma->vm_start); + LIST_HEAD(folios); + + if (!folio) + return; + lru_add_drain_all(); + if (folio_isolate_lru(folio)) + list_add(&folio->lru, &folios); + folio_put(folio); + reclaim_pages(&folios); + schedule_timeout_uninterruptible(1); +} + +static void user64k_anon_swap_limit_test(struct kunit *test) +{ +#ifdef CONFIG_MEMCG + struct vm_area_struct *vma; + struct cgroup *cgroup; + struct cgroup_subsys_state *css; + struct mem_cgroup *memcg; + struct folio *folio; + swp_entry_t entry, part; + phys_addr_t original; + unsigned int i, tries, nr = SZ_64K / PAGE_SIZE; + long before, denied; + char operation[80]; + vm_fault_t fault; + pte_t pte; + int ret, leave; + + if (!test_coarse_swap_enabled(test)) + return; + if (mem_cgroup_disabled()) { + kunit_skip(test, "requires memory cgroups"); + return; + } + snprintf(operation, sizeof(operation), "--coarse-swap-enter=%d", task_pid_nr(current)); + KUNIT_ASSERT_EQ(test, test_swap_control(operation), 0); + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + fault = vma ? test_file_fault(vma, TEST_VA, true) : VM_FAULT_OOM; + snprintf(operation, sizeof(operation), "--coarse-swap-leave=%d", task_pid_nr(current)); + leave = test_swap_control(operation); + KUNIT_ASSERT_EQ(test, leave, 0); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, fault & VM_FAULT_ERROR, 0U); + cgroup = cgroup_get_from_path("/coarse-swap-limit"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, cgroup); + css = cgroup_get_e_css(cgroup, &memory_cgrp_subsys); + cgroup_put(cgroup); + KUNIT_ASSERT_NOT_NULL(test, css); + memcg = mem_cgroup_from_css(css); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_memcg_put, memcg), 0); + KUNIT_ASSERT_PTR_NE(test, memcg, root_mem_cgroup); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + rcu_read_lock(); + KUNIT_EXPECT_PTR_EQ(test, folio_memcg(folio), memcg); + rcu_read_unlock(); + original = page_to_phys(&folio->page); + memset(folio_address(folio), 0xc9, SZ_64K); + before = atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]); + folio_lock(folio); + ret = folio_alloc_swap(folio); + KUNIT_EXPECT_FALSE(test, folio_test_swapcache(folio)); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, -ENOMEM); + denied = atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]); + KUNIT_EXPECT_GT(test, denied, before); + KUNIT_EXPECT_EQ(test, page_counter_read(&memcg->swap), 0UL); + /* Exercise vmscan's failed-allocation/split fallback without a retained GET. */ + for (tries = 0; tries < 8; tries++) + test_coarse_anon_reclaim_once(vma); + KUNIT_EXPECT_GT(test, atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]), denied); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte), original); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xc9, SZ_64K), NULL); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)nr); + folio_put(folio); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 1L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SWAPENTS), 0L); + KUNIT_EXPECT_EQ(test, page_counter_read(&memcg->swap), 0UL); + KUNIT_ASSERT_EQ(test, test_swap_control("--coarse-swap-unlimit"), 0); + for (tries = 0; tries < 100; tries++) { + test_coarse_anon_reclaim_once(vma); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + if (!pte_present(pte)) + break; + } + KUNIT_ASSERT_FALSE(test, pte_present(pte)); + entry = softleaf_from_pte(pte); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_EXPECT_EQ(test, page_counter_read(&memcg->swap), (unsigned long)nr); + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), 1); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(part)); + } + fault = test_file_fault(vma, TEST_VA, false); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), 0U); + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_MAJOR); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xc9, SZ_64K), NULL); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + KUNIT_EXPECT_EQ(test, page_counter_read(&memcg->swap), 0UL); +#else + kunit_skip(test, "requires memory cgroups"); +#endif +} + +struct coarse_swap_worker { + struct mm_struct *mm; + struct task_struct *task; + struct completion ready, go, entered, done; + u8 *buffer; + unsigned int id; + bool *disabled; + int error; +}; + +static int coarse_swap_worker_run(void *arg) +{ + struct coarse_swap_worker *w = arg; + unsigned int i; + u8 value = 0x91 + w->id; + + complete(&w->ready); + if (!wait_for_completion_timeout(&w->go, 15 * HZ) || kthread_should_stop()) { + w->error = -ETIMEDOUT; + goto out; + } + if (!w->mm) { + complete(&w->entered); + w->error = test_swap_control("--swapoff-helper"); + if (!w->error) + WRITE_ONCE(*w->disabled, true); + goto out; + } + kthread_use_mm(w->mm); + complete(&w->entered); + if (copy_from_user(w->buffer, (void __user *)TEST_VA, SZ_64K)) { + w->error = 1; + goto unuse; + } + for (i = 0; i < SZ_64K; i++) { + if (w->buffer[i] != (u8)(0x31 + i / SZ_4K)) { + w->error = 2; + goto unuse; + } + } + if (copy_to_user((void __user *)(TEST_VA + SZ_64K - 1), &value, 1) || + copy_from_user(w->buffer, (void __user *)TEST_VA, SZ_64K)) { + w->error = 3; + goto unuse; + } + for (i = 0; i < SZ_64K; i++) { + u8 expected = i == SZ_64K - 1 ? value : 0x31 + i / SZ_4K; + + if (w->buffer[i] != expected) { + w->error = 4; + break; + } + } +unuse: + kthread_unuse_mm(w->mm); +out: + complete(&w->done); + while (!kthread_should_stop()) + schedule_timeout_interruptible(1); + return 0; +} + +static void coarse_swap_worker_stop(void *arg) +{ + struct coarse_swap_worker *w = arg; + + if (w->task) { + complete_all(&w->go); + kthread_stop(w->task); + w->task = NULL; + } +} + +static void user64k_anon_swap_race_test(struct kunit *test) +{ + unsigned int round, nr = SZ_64K / PAGE_SIZE; + + if (!test_coarse_swap_enabled(test)) + return; + if (num_online_cpus() < 2 || !system_supports_64kb_granule()) { + kunit_skip(test, "requires two CPUs and 64K TTBR0 support"); + return; + } + for (round = 0; round < 8; round++) { + struct vm_area_struct *vma; + struct mm_struct *child; + struct coarse_swap_worker *workers; + struct folio *folio, *blocker; + struct swap_info_struct *si; + swp_entry_t entry, part; + bool *disabled, started = true, blocked; + bool off_started = false, cold = round & 1; + unsigned int i, j, tries; + pte_t pte; + int ret, cpu; + + disabled = kunit_kzalloc(test, sizeof(*disabled), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, disabled); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_reenable_swap, disabled), 0); + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + for (i = 0; i < SZ_64K / SZ_4K; i++) + memset(folio_address(folio) + i * SZ_4K, 0x31 + i, SZ_4K); + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, vma->vm_mm); + uprobe_end_dup_mmap(); + if (!ret) { + folio_lock(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + } + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_FALSE(test, pte_present(pte)); + entry = softleaf_from_pte(pte); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), 2); + if (cold) + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(part)); + } + part = entry; + part.val += nr - 1; + blocker = read_swap_cache_async(part, GFP_KERNEL, vma, + TEST_VA + SZ_64K - PAGE_SIZE, NULL); + KUNIT_ASSERT_NOT_NULL(test, blocker); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, blocker), 0); + KUNIT_EXPECT_EQ(test, folio_size(blocker), cold ? PAGE_SIZE : (unsigned long)SZ_64K); + workers = kunit_kcalloc(test, 3, sizeof(*workers), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, workers); + cpu = cpumask_first(cpu_online_mask); + for (i = 0; i < 3; i++) { + workers[i].mm = i < 2 ? (i ? child : vma->vm_mm) : NULL; + workers[i].id = i; + workers[i].disabled = disabled; + workers[i].buffer = kunit_kmalloc(test, SZ_64K, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, workers[i].buffer); + init_completion(&workers[i].ready); + init_completion(&workers[i].go); + init_completion(&workers[i].entered); + init_completion(&workers[i].done); + workers[i].task = kthread_create(coarse_swap_worker_run, &workers[i], + "user64k-swap-%u", i); + if (IS_ERR(workers[i].task)) { + workers[i].task = NULL; + KUNIT_FAIL(test, "coarse swap worker creation failed"); + return; + } + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + coarse_swap_worker_stop, &workers[i]), 0); + if (i < 2) { + kthread_bind(workers[i].task, cpu); + cpu = cpumask_next(cpu, cpu_online_mask); + } + wake_up_process(workers[i].task); + } + for (i = 0; i < 3; i++) + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].ready, 10 * HZ), 0UL); + folio_lock(blocker); + for (i = 0; i < 3; i++) + complete(&workers[i].go); + for (i = 0; i < 3; i++) + started &= !!wait_for_completion_timeout(&workers[i].entered, HZ); + for (tries = 0; tries < 3 * HZ; tries++) { + si = get_swap_device(entry); + if (si) { + off_started = !(READ_ONCE(si->flags) & SWP_WRITEOK); + put_swap_device(si); + } + if (off_started) + break; + schedule_timeout_uninterruptible(1); + } + blocked = !completion_done(&workers[0].done) && + !completion_done(&workers[1].done) && !completion_done(&workers[2].done); + /* Never execute a fatal assertion while retaining the blocking lock. */ + folio_unlock(blocker); + KUNIT_EXPECT_TRUE(test, started); + KUNIT_EXPECT_TRUE(test, off_started); + KUNIT_EXPECT_TRUE(test, blocked); + for (i = 0; i < 3; i++) { + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&workers[i].done, 20 * HZ), 0UL); + KUNIT_EXPECT_EQ_MSG(test, workers[i].error, 0, "round=%u worker=%u", round, i); + kunit_release_action(test, coarse_swap_worker_stop, &workers[i]); + } + KUNIT_EXPECT_TRUE(test, *disabled); + for (i = 0; i < 2; i++) { + struct mm_struct *mm = i ? child : vma->vm_mm; + + folio = test_fault_folio(mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + for (j = 0; j < SZ_64K; j++) { + u8 expected = j == SZ_64K - 1 ? 0x91 + i : 0x31 + j / SZ_4K; + + if (((u8 *)folio_address(folio))[j] != expected) { + KUNIT_FAIL(test, "round=%u side=%u byte=%u", round, i, j); + break; + } + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_SWAPENTS), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 1L); + kunit_release_action(test, user4k_vma_mm_free, mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), folio == blocker ? 2 : 1); + kunit_release_action(test, test_folio_put, folio); + } + KUNIT_EXPECT_FALSE(test, folio_test_swapcache(blocker)); + KUNIT_EXPECT_EQ(test, folio_ref_count(blocker), 1); + kunit_release_action(test, test_folio_put, blocker); + kunit_release_action(test, test_reenable_swap, disabled); + KUNIT_ASSERT_GT(test, total_swap_pages, 0UL); + } + kunit_info(test, "8 controlled warm/cold fault+COW+swapoff rounds completed"); +} + +struct coarse_swap_reservation { + struct folio **folios; + unsigned int count; +}; + +static void test_coarse_release_reservation(struct folio *folio) +{ + folio_lock(folio); + folio_put_swap(folio, NULL); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); +} + +static void test_coarse_release_reservations(void *arg) +{ + struct coarse_swap_reservation *r = arg; + unsigned int i; + + for (i = 0; i < r->count; i++) { + if (!r->folios[i]) + continue; + test_coarse_release_reservation(r->folios[i]); + r->folios[i] = NULL; + cond_resched(); + } +} + +static void test_coarse_swap_refusal(struct kunit *test, struct vm_area_struct *vma, + phys_addr_t original) +{ + struct folio *folio = test_fault_folio(vma->vm_mm, TEST_VA); + pte_t pte; + unsigned int i; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + ret = folio_alloc_swap(folio); + KUNIT_EXPECT_FALSE(test, folio_test_swapcache(folio)); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, -ENOMEM); + for (i = 0; i < 4; i++) + test_coarse_anon_reclaim_once(vma); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_TRUE(test, pte_present(pte)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(vma->vm_mm, pte), original); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xd6, SZ_64K), NULL); + folio_put(folio); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_ANONPAGES), 1L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(vma->vm_mm, MM_SWAPENTS), 0L); +} + +static void user64k_anon_swap_fragment_test(struct kunit *test) +{ + struct coarse_swap_reservation *r; + struct vm_area_struct *vma; + struct folio *folio; + unsigned int i, nr = SZ_64K / PAGE_SIZE, released = 0; + unsigned long hole, free_before; + phys_addr_t original; + swp_entry_t entry, part; + vm_fault_t fault; + pte_t pte; + int ret = 0; + + if (!test_coarse_swap_enabled(test)) + return; + /* Exhaust only the bounded disposable disk owned by this test's PID1. */ + KUNIT_ASSERT_EQ(test, test_swap_control("--coarse-fixture-check"), 0); + KUNIT_ASSERT_LT(test, total_swap_pages, (unsigned long)(SZ_256M / PAGE_SIZE)); + KUNIT_ASSERT_GT(test, total_swap_pages, 2UL * SWAPFILE_CLUSTER); + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + original = page_to_phys(&folio->page); + memset(folio_address(folio), 0xd6, SZ_64K); + folio_put(folio); + r = kunit_kzalloc(test, sizeof(*r), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, r); + r->folios = kunit_kcalloc(test, total_swap_pages + 1, sizeof(*r->folios), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, r->folios); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_coarse_release_reservations, r), 0); + /* Each cache-only folio has one explicit slot owner, so allocation cannot reclaim it. */ + for (i = 0; i <= total_swap_pages; i++) { + folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, 0); + KUNIT_ASSERT_NOT_NULL(test, folio); + ret = mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL); + if (ret) { + folio_put(folio); + KUNIT_FAIL(test, "reservation memory charge failed"); + return; + } + __folio_set_swapbacked(folio); + __folio_mark_uptodate(folio); + folio_lock(folio); + ret = folio_alloc_swap(folio); + if (!ret) { + ret = folio_dup_swap(folio, NULL); + if (ret) + folio_free_swap(folio); + } + folio_unlock(folio); + if (ret) { + folio_put(folio); + break; + } + r->folios[r->count++] = folio; + cond_resched(); + } + KUNIT_ASSERT_EQ(test, ret, -ENOMEM); + KUNIT_ASSERT_EQ(test, atomic_long_read(&nr_swap_pages), 0L); + test_coarse_swap_refusal(test, vma, original); + /* Retain exactly one native entry in each naturally aligned coarse group. */ + for (i = 0; i < r->count; i++) { + folio = r->folios[i]; + if (swp_offset(folio->swap) % nr == 1) + continue; + test_coarse_release_reservation(folio); + r->folios[i] = NULL; + } + free_before = atomic_long_read(&nr_swap_pages); + KUNIT_EXPECT_GT(test, free_before, total_swap_pages / 2); + KUNIT_EXPECT_LT(test, free_before, total_swap_pages); + test_coarse_swap_refusal(test, vma, original); + /* Free one aligned group near the end, without freeing a whole native-order cluster. */ + hole = round_down(total_swap_pages + 1, (unsigned long)SWAPFILE_CLUSTER) - nr; + for (i = 0; i < r->count; i++) { + folio = r->folios[i]; + if (!folio || swp_offset(folio->swap) < hole || swp_offset(folio->swap) >= hole + nr) + continue; + test_coarse_release_reservation(folio); + r->folios[i] = NULL; + released++; + } + KUNIT_ASSERT_EQ(test, released, 1U); + KUNIT_EXPECT_EQ(test, atomic_long_read(&nr_swap_pages), (long)free_before + 1); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + folio_lock(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ_MSG(test, ret, 0, "a complete aligned group is free in an order-0 cluster"); + pte = test_fault_entry(test, vma->vm_mm, TEST_VA); + KUNIT_ASSERT_FALSE(test, pte_present(pte)); + entry = softleaf_from_pte(pte); + KUNIT_ASSERT_TRUE(test, softleaf_is_swap(entry)); + KUNIT_EXPECT_EQ(test, swp_offset(entry), hole); + for (i = 0; i < r->count; i++) { + if (r->folios[i]) + KUNIT_EXPECT_EQ(test, swp_swapcount(r->folios[i]->swap), 1); + } + kunit_release_action(test, test_coarse_release_reservations, r); + for (i = 0; i < nr; i++) { + part = entry; + part.val += i; + KUNIT_EXPECT_EQ(test, swp_swapcount(part), 1); + KUNIT_ASSERT_TRUE(test, test_reclaim_swapcache(part)); + } + fault = test_file_fault(vma, TEST_VA, false); + KUNIT_ASSERT_EQ(test, fault & (VM_FAULT_ERROR | VM_FAULT_RETRY), 0U); + KUNIT_EXPECT_TRUE(test, fault & VM_FAULT_MAJOR); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xd6, SZ_64K), NULL); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + kunit_info(test, "exhaustion, scattered free space and one-group recovery checked"); +} + +struct coarse_swap_io_mode { + struct swap_info_struct *si; + bool synchronous; +}; + +static void test_coarse_restore_io_mode(void *arg) +{ + struct coarse_swap_io_mode *mode = arg; + + spin_lock(&mode->si->lock); + if (mode->synchronous) + mode->si->flags |= SWP_SYNCHRONOUS_IO; + else + mode->si->flags &= ~SWP_SYNCHRONOUS_IO; + spin_unlock(&mode->si->lock); + put_swap_device(mode->si); +} + +static void user64k_anon_swap_extent_test(struct kunit *test) +{ + unsigned int mode, i; + + if (!test_coarse_swap_enabled(test)) + return; + KUNIT_ASSERT_EQ(test, test_swap_control("--coarse-fixture-check"), 0); + for (mode = 0; mode < 2; mode++) { + struct vm_area_struct *vma; + struct coarse_swap_io_mode *saved; + struct folio *folio; + unsigned long nr = SZ_64K / PAGE_SIZE, run = nr; + swp_entry_t entry; + sector_t first, next; + bool file; + int ret; + + vma = test_fault_vma_granule(test, 16, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + folio = test_fault_folio(vma->vm_mm, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, folio); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_folio_put, folio), 0); + for (i = 0; i < SZ_64K / SZ_4K; i++) + memset(folio_address(folio) + i * SZ_4K, 0x63 + i, SZ_4K); + folio_lock(folio); + ret = folio_alloc_swap(folio); + if (!ret) + try_to_unmap(folio, TTU_SYNC); + folio_unlock(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, folio_mapcount(folio), 0); + entry = folio->swap; + saved = kunit_kzalloc(test, sizeof(*saved), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, saved); + saved->si = get_swap_device(entry); + KUNIT_ASSERT_NOT_NULL(test, saved->si); + saved->synchronous = saved->si->flags & SWP_SYNCHRONOUS_IO; + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_coarse_restore_io_mode, saved), 0); + file = !(saved->si->flags & SWP_BLKDEV); + KUNIT_ASSERT_FALSE(test, saved->si->flags & SWP_FS_OPS); + first = swap_extent_sector(entry, &run); + if (file) { + unsigned long remaining; + swp_entry_t part = entry; + + KUNIT_ASSERT_LT_MSG(test, run, nr, "fragmented fixture must cross an extent"); + part.val += run; + remaining = nr - run; + next = swap_extent_sector(part, &remaining); + KUNIT_EXPECT_NE(test, next, first + (run << (PAGE_SHIFT - 9))); + } else { + KUNIT_EXPECT_EQ(test, run, nr); + } + /* The verified disposable fixture owns this otherwise idle swap area. */ + spin_lock(&saved->si->lock); + if (mode) + saved->si->flags |= SWP_SYNCHRONOUS_IO; + else + saved->si->flags &= ~SWP_SYNCHRONOUS_IO; + spin_unlock(&saved->si->lock); + folio_lock(folio); + folio_clear_dirty_for_io(folio); + __swap_writepage(folio, NULL); + folio_wait_writeback(folio); + folio_lock(folio); + KUNIT_EXPECT_FALSE(test, folio_test_dirty(folio)); + /* A full-folio read must reconstruct all physical runs, not warm cache. */ + memset(folio_address(folio), 0, SZ_64K); + folio_clear_uptodate(folio); + swap_read_folio(folio, NULL); + folio_lock(folio); + KUNIT_EXPECT_TRUE(test, folio_test_uptodate(folio)); + for (i = 0; i < SZ_64K / SZ_4K; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + i * SZ_4K, + 0x63 + i, SZ_4K), NULL); + folio_unlock(folio); + kunit_release_action(test, test_coarse_restore_io_mode, saved); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & VM_FAULT_ERROR, 0U); + kunit_release_action(test, user4k_vma_mm_free, vma->vm_mm); + lru_add_drain_all(); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), 0); + KUNIT_EXPECT_EQ(test, folio_ref_count(folio), 1); + kunit_info(test, "64K full-folio %s I/O: file=%u first-run=%lu/%lu native pages", + mode ? "synchronous" : "asynchronous", file, run, nr); + } +} + +static struct kunit_case user4k_coarse_shmem_cases[] = { + KUNIT_CASE(user64k_anon_swap_extent_test), + KUNIT_CASE(user64k_anon_swap_test), + KUNIT_CASE(user64k_anon_swap_limit_test), + KUNIT_CASE(user64k_anon_swap_fragment_test), + KUNIT_CASE(user64k_anon_swap_race_test), + KUNIT_CASE(user64k_shmem_cold_constituents_test), + KUNIT_CASE(user64k_shmem_cold_promoted_test), + KUNIT_CASE(user64k_shmem_fault_alloc_fail_test), + {} +}; + +static struct kunit_suite user4k_coarse_shmem_suite = { + .name = "arm64-user4k-coarse-shmem", + .test_cases = user4k_coarse_shmem_cases, + .suite_exit = user4k_swap_suite_exit, +}; + +static struct kunit_case user4k_shmem_swap_cases[] = { + KUNIT_CASE(user4k_shmem_uffd_swap_alloc_fail_test), + KUNIT_CASE(user4k_shmem_uffd_swap_charge_test), + KUNIT_CASE(user4k_shmem_uffd_swapin_test), + KUNIT_CASE(user4k_shmem_uffd_swapped_truncate_test), + KUNIT_CASE(user4k_shmem_uffd_swapoff_test), + KUNIT_CASE(user4k_shmem_uffd_swapped_partial_hole_test), + {} +}; + +static struct kunit_suite user4k_shmem_swap_suite = { + .name = "arm64-user4k-shmem-swap", + .test_cases = user4k_shmem_swap_cases, + .suite_exit = user4k_swap_suite_exit, +}; +#endif + +static struct kunit_case user4k_nonlinear_swap_cases[] = { +#ifdef CONFIG_USERFAULTFD + KUNIT_CASE(user4k_move_swap_get_test), +#ifdef CONFIG_MIGRATION + KUNIT_CASE(user4k_move_migration_race_test), +#endif +#endif + KUNIT_CASE(user4k_nonlinear_swap_cached_test), + KUNIT_CASE(user4k_nonlinear_swap_disk_test), + {} +}; + +static struct kunit_suite user4k_nonlinear_swap_suite = { + .name = "arm64-user4k-nonlinear-swap", + .test_cases = user4k_nonlinear_swap_cases, + .suite_exit = user4k_swap_suite_exit, +}; + +static struct kunit_suite user4k_swap_suite = { + .name = "arm64-user4k-swap", + .test_cases = user4k_swap_cases, + .suite_exit = user4k_swap_suite_exit, +}; + +static void user4k_thp_eligibility_test(struct kunit *test) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + struct mm_struct *small = test_vma_mm(test), *native = mm_alloc(); + const enum tva_type types[] = { + TVA_PAGEFAULT, TVA_SMAPS, TVA_KHUGEPAGED, TVA_FORCED_COLLAPSE, + }; + struct vm_area_struct *vma, *normal; + vm_flags_t flags = VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | + VM_HUGEPAGE; + + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + user4k_vma_mm_free, native), 0); + KUNIT_ASSERT_NOT_NULL(test, small); + vma = test_vma_add(small, SZ_1G, SZ_1G, (struct vm_page_offset) {}, flags); + normal = test_vma_add(native, SZ_1G, SZ_1G, (struct vm_page_offset) {}, flags); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vma); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, normal); + /* Neither user advice nor forced collapse can bypass geometry limits. */ + KUNIT_EXPECT_TRUE(test, vma_thp_disabled(vma, flags, false)); + KUNIT_EXPECT_TRUE(test, vma_thp_disabled(vma, flags, true)); + KUNIT_EXPECT_FALSE(test, vma_thp_disabled(normal, flags, false)); + KUNIT_EXPECT_FALSE(test, vma_thp_disabled(normal, flags, true)); + for (unsigned int i = 0; i < ARRAY_SIZE(types); i++) + KUNIT_EXPECT_EQ(test, __thp_vma_allowable_orders(vma, flags, + types[i], THP_ORDERS_ALL_ANON), 0UL); +#else + kunit_skip(test, "requires transparent hugepage support"); +#endif +} + +static void user4k_aslr_window_test(struct kunit *test) +{ + struct mm_struct *small = test_vma_mm(test), *native = mm_alloc(); + const unsigned long windows[] = { SZ_8M, SZ_32M, SZ_1G }; + + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, + user4k_vma_mm_free, native), 0); + KUNIT_ASSERT_NOT_NULL(test, small); + for (unsigned int i = 0; i < ARRAY_SIZE(windows); i++) { + unsigned long mask = (windows[i] >> PAGE_SHIFT) - 1; + unsigned long converted = mm_aslr_mask(small, mask); + + KUNIT_EXPECT_EQ(test, mm_aslr_mask(native, mask), mask); + KUNIT_EXPECT_EQ(test, (converted + 1) << mm_page_shift(small), + windows[i]); + KUNIT_EXPECT_EQ(test, converted & (TEST_SLOTS - 1), + (unsigned long)TEST_SLOTS - 1); + } +} + +static void user4k_exec_granule_policy_test(struct kunit *test) +{ + /* Model all native granules, including AArch32 with NO 4K support. */ + const unsigned int native[] = { 12, 14, 16 }; + + for (unsigned int i = 0; i < ARRAY_SIZE(native); i++) { + KUNIT_EXPECT_EQ(test, arm64_exec_page_shift(true, false, native[i], 0), + native[i]); + KUNIT_EXPECT_EQ(test, arm64_exec_page_shift(true, false, native[i], 12), + native[i]); + KUNIT_EXPECT_EQ(test, arm64_exec_page_shift(true, true, native[i], 0), + 12U); + KUNIT_EXPECT_EQ(test, arm64_exec_page_shift(false, true, native[i], 0), + native[i]); + KUNIT_EXPECT_EQ(test, arm64_exec_page_shift(false, true, native[i], 12), + 12U); + } +} + +static void user_granule_zero_region_test(struct kunit *test) +{ + unsigned long base_pfn = zero_pfn(0); + unsigned int shift, i; + u8 *buffer; + + KUNIT_EXPECT_TRUE(test, IS_ALIGNED(PFN_PHYS(base_pfn), SZ_64K)); + KUNIT_EXPECT_FALSE(test, is_zero_pfn(base_pfn - 1)); + KUNIT_EXPECT_FALSE(test, is_zero_pfn(base_pfn + SZ_64K / PAGE_SIZE)); + for (i = 0; i < SZ_64K / PAGE_SIZE; i++) { + KUNIT_EXPECT_TRUE(test, is_zero_pfn(base_pfn + i)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(pfn_to_page(base_pfn + i)), + 0, PAGE_SIZE), NULL); + } + buffer = kunit_kmalloc(test, SZ_64K, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, buffer); + for (shift = 12; shift <= 16; shift += 2) { + struct vm_area_struct *vma; + struct mm_struct *mm; + unsigned long size = 1UL << shift, left; + vm_fault_t ret; + spinlock_t *ptl; + pte_t *pte, entry; + + if ((shift == 12 && !system_supports_4kb_granule()) || + (shift == 14 && !system_supports_16kb_granule()) || + (shift == 16 && !system_supports_64kb_granule())) { + kunit_info(test, "hardware zero mapping skipped for shift %u", shift); + continue; + } + /* Internal coarse mm: the public opt-in remains disabled. */ + mm = test_vma_mm_granule(test, shift); + KUNIT_ASSERT_NOT_NULL(test, mm); + vma = test_fault_vma_in_mm(mm, TEST_VA, size); + KUNIT_ASSERT_NOT_NULL(test, vma); + mmap_read_lock(mm); + ret = handle_mm_fault(vma, TEST_VA + size - 1, FAULT_FLAG_REMOTE, NULL); + mmap_read_unlock(mm); + KUNIT_ASSERT_EQ(test, ret, (vm_fault_t)0); + pte = test_mm_lookup(mm, TEST_VA, &ptl); + KUNIT_ASSERT_NOT_NULL(test, pte); + entry = ptep_get(pte); + pte_unmap_unlock(pte, ptl); + KUNIT_EXPECT_TRUE(test, pte_present(entry)); + KUNIT_EXPECT_TRUE(test, pte_special(entry)); + KUNIT_EXPECT_FALSE(test, pte_write(entry)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(mm, entry), PFN_PHYS(base_pfn)); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0L); + + memset(buffer, 0xa5, size); + kthread_use_mm(mm); + left = copy_from_user(buffer, (void __user *)TEST_VA, size); + kthread_unuse_mm(mm); + KUNIT_EXPECT_EQ(test, left, 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(buffer, 0, size), NULL); + mmap_read_lock(mm); + zap_vma_range(vma, TEST_VA, size); + mmap_read_unlock(mm); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(mm, MM_ANONPAGES), 0L); + } +} + +#ifdef CONFIG_COREDUMP +static void user_granule_core_offsets_test(struct kunit *test) +{ + struct mm_struct *mm = kunit_kzalloc(test, sizeof(*mm), GFP_KERNEL); + struct vm_area_struct *vma = kunit_kzalloc(test, sizeof(*vma), GFP_KERNEL); + unsigned int shift; + + KUNIT_ASSERT_NOT_NULL(test, mm); + KUNIT_ASSERT_NOT_NULL(test, vma); + vma->vm_mm = mm; + for (shift = 12; shift <= 16; shift += 2) { + unsigned long bytes = (1UL << 55) + 3 * SZ_64K + (1UL << shift); + + mm->context.flags = shift == 12 ? MMCF_USER_4K : + shift == 14 ? MMCF_USER_16K : MMCF_USER_64K; + vma_set_page_offset(vma, (struct vm_page_offset) { + .index = bytes >> PAGE_SHIFT, .offset = bytes & ~PAGE_MASK, + }); + KUNIT_EXPECT_EQ(test, core_vma_pgoff(vma), bytes >> shift); + KUNIT_EXPECT_EQ(test, mm_user_fragment_size(mm), min(1UL << shift, PAGE_SIZE)); + vma_set_page_offset(vma, (struct vm_page_offset) {}); + KUNIT_EXPECT_EQ(test, core_vma_pgoff(vma), 0UL); + } +} +#endif + +static void user64k_fragment_ownership_test(struct kunit *test) +{ + struct user_page_fragment fragments[SZ_64K / PAGE_SIZE] = {}; + struct mm_struct *parent, *child; + struct vm_area_struct *vma, *cvma; + struct folio *original, *copied; + unsigned int i, nr = ARRAY_SIZE(fragments); + long count; + int ret, refs; + pte_t entry; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires a user leaf larger than native pages"); + return; + } + parent = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, parent); + vma = test_fault_vma_in_mm(parent, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, false) & VM_FAULT_ERROR, 0U); + count = get_user_fragments_remote(parent, TEST_VA + 17, SZ_64K - 34, + 0, fragments, nr); + if (count > 0) { + for (i = 0; i < count; i++) { + struct user_page_fragment *f = &fragments[i]; + unsigned int bytes = PAGE_SIZE - (i == 0 || i == nr - 1 ? 17 : 0); + + KUNIT_EXPECT_TRUE(test, is_zero_folio(f->folio)); + KUNIT_EXPECT_EQ(test, folio_pfn(f->folio), zero_pfn(TEST_VA) + i); + KUNIT_EXPECT_PTR_EQ(test, f->subpage, NULL); + KUNIT_EXPECT_EQ(test, f->offset, i == 0 ? 17UL : 0UL); + KUNIT_EXPECT_EQ(test, f->length, bytes); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0, f->length), NULL); + } + release_user_fragments(fragments, count, false); + } + KUNIT_ASSERT_EQ(test, count, (long)nr); + + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA, true) & VM_FAULT_ERROR, 0U); + entry = test_fault_entry(test, parent, TEST_VA); + original = page_folio(pfn_to_page(pte_pfn(entry))); + for (i = 0; i < nr; i++) + memset(page_address(folio_page(original, i)), 0x51 + i, PAGE_SIZE); + lru_add_drain_all(); + refs = folio_ref_count(original); + count = get_user_fragments_remote(parent, TEST_VA + 17, SZ_64K - 34, + 0, fragments, nr); + if (count > 0) { + KUNIT_EXPECT_EQ(test, folio_ref_count(original), refs + (int)count); + for (i = 0; i < count; i++) { + struct user_page_fragment *f = &fragments[i]; + + KUNIT_EXPECT_PTR_EQ(test, f->folio, original); + KUNIT_EXPECT_EQ(test, f->offset, i * PAGE_SIZE + (i == 0 ? 17UL : 0UL)); + KUNIT_EXPECT_LE(test, offset_in_page(f->offset) + f->length, PAGE_SIZE); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(f->folio) + f->offset, + 0x51 + i, f->length), NULL); + } + release_user_fragments(fragments, count, false); + } + KUNIT_ASSERT_EQ(test, count, (long)nr); + KUNIT_EXPECT_EQ(test, folio_ref_count(original), refs); + + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + /* A read pin after fork must unshare the complete coarse leaf. */ + count = pin_user_fragments_remote(child, TEST_VA, SZ_64K, 0, fragments, nr); + if (count > 0) { + copied = fragments[0].folio; + KUNIT_EXPECT_PTR_NE(test, copied, original); + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(copied)); + KUNIT_EXPECT_FALSE(test, pte_write(test_fault_entry(test, child, TEST_VA))); + for (i = 0; i < count; i++) { + KUNIT_EXPECT_PTR_EQ(test, fragments[i].folio, copied); + KUNIT_EXPECT_PTR_EQ(test, fragments[i].subpage, NULL); + KUNIT_EXPECT_TRUE(test, fragments[i].pinned); + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(copied, i))); + KUNIT_EXPECT_EQ(test, fragments[i].offset, i * PAGE_SIZE); + KUNIT_EXPECT_EQ(test, fragments[i].length, (unsigned int)PAGE_SIZE); + } + mmap_read_lock(child); + zap_vma_range(cvma, TEST_VA, SZ_64K); + mmap_read_unlock(child); + KUNIT_EXPECT_EQ(test, folio_mapcount(copied), 0); + for (i = 0; i < count; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(copied) + i * PAGE_SIZE, + 0x51 + i, PAGE_SIZE), NULL); + release_user_fragments(fragments, count, false); + } + KUNIT_ASSERT_EQ(test, count, (long)nr); + for (i = 0; i < nr; i++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(original) + i * PAGE_SIZE, + 0x51 + i, PAGE_SIZE), NULL); +} + +static void user64k_anon_fault_cow_test(struct kunit *test) +{ + struct mm_struct *parent, *child; + struct vm_area_struct *vma, *cvma; + pte_t orig[2], copied; + struct folio *folio; + unsigned int i, j, nr = SZ_64K / PAGE_SIZE; + int ret; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires coarse anonymous faults"); + return; + } + parent = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, parent); + vma = test_fault_vma_in_mm(parent, TEST_VA, 2 * SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + /* Leaf0 exercises zero-page COW; leaf1 exercises a direct write fault. */ + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA + SZ_64K - 1, false) & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_TRUE(test, pte_special(test_fault_entry(test, parent, TEST_VA))); + for (i = 0; i < 2; i++) { + unsigned long addr = TEST_VA + i * SZ_64K; + + KUNIT_ASSERT_EQ(test, test_file_fault(vma, addr + SZ_64K - 1, true) & VM_FAULT_ERROR, 0U); + orig[i] = test_fault_entry(test, parent, addr); + KUNIT_ASSERT_TRUE(test, pte_present(orig[i])); + KUNIT_EXPECT_TRUE(test, pte_write(orig[i])); + KUNIT_EXPECT_FALSE(test, pte_special(orig[i])); + folio = page_folio(pfn_to_page(pte_pfn(orig[i]))); + KUNIT_EXPECT_EQ(test, folio_size(folio), (unsigned long)SZ_64K); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)nr); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0, SZ_64K), NULL); + for (j = 0; j < nr; j++) { + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(folio, j))); + memset(page_address(folio_page(folio, j)), 0x31 + i * nr + j, PAGE_SIZE); + } + } + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent, MM_ANONPAGES), 2L); + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + cvma = test_vma_lookup(child, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, cvma); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 2L); + for (i = 0; i < 2; i++) { + unsigned long addr = TEST_VA + i * SZ_64K; + struct folio *old = page_folio(pfn_to_page(pte_pfn(orig[i]))); + + KUNIT_EXPECT_EQ(test, pte_phys_mm(child, test_fault_entry(test, child, addr)), + pte_phys_mm(parent, orig[i])); + KUNIT_EXPECT_FALSE(test, pte_write(test_fault_entry(test, parent, addr))); + KUNIT_EXPECT_EQ(test, folio_mapcount(old), (int)(2 * nr)); + KUNIT_ASSERT_EQ(test, test_file_fault(cvma, addr + SZ_64K - 1, true) & VM_FAULT_ERROR, 0U); + copied = test_fault_entry(test, child, addr); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, copied), pte_phys_mm(parent, orig[i])); + folio = page_folio(pfn_to_page(pte_pfn(copied))); + KUNIT_EXPECT_EQ(test, folio_mapcount(old), (int)nr); + KUNIT_EXPECT_EQ(test, folio_mapcount(folio), (int)nr); + for (j = 0; j < nr; j++) { + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(folio, j)), + 0x31 + i * nr + j, PAGE_SIZE), NULL); + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(folio, j))); + } + memset(folio_address(folio), 0xa5, SZ_64K); + for (j = 0; j < nr; j++) + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(old, j)), + 0x31 + i * nr + j, PAGE_SIZE), NULL); + /* Parent reuse must restore exclusivity across the entire leaf. */ + lru_add_drain_all(); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, addr + SZ_64K - 1, true) & VM_FAULT_ERROR, 0U); + KUNIT_EXPECT_EQ(test, pte_phys_mm(parent, test_fault_entry(test, parent, addr)), + pte_phys_mm(parent, orig[i])); + for (j = 0; j < nr; j++) + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(old, j))); + } + if (system_supports_64kb_granule()) { + long protect[2]; + bool writable = false; + + kthread_use_mm(parent); + protect[0] = test_mprotect(TEST_VA, 2 * SZ_64K, PROT_READ); + kthread_unuse_mm(parent); + for (i = 0; i < 2; i++) + writable |= pte_write(test_fault_entry(test, parent, TEST_VA + i * SZ_64K)); + KUNIT_EXPECT_EQ(test, protect[0], 0L); + KUNIT_EXPECT_FALSE(test, writable); + kthread_use_mm(parent); + protect[1] = test_mprotect(TEST_VA, 2 * SZ_64K, PROT_READ | PROT_WRITE); + kthread_unuse_mm(parent); + KUNIT_EXPECT_EQ(test, protect[1], 0L); + for (i = 0; i < 2; i++) + KUNIT_EXPECT_TRUE(test, pte_write(test_fault_entry(test, parent, TEST_VA + i * SZ_64K))); + vma = test_vma_lookup(parent, TEST_VA); + KUNIT_ASSERT_NOT_NULL(test, vma); + } + mmap_read_lock(child); + zap_vma_range(cvma, TEST_VA, 2 * SZ_64K); + mmap_read_unlock(child); + mmap_read_lock(parent); + zap_vma_range(vma, TEST_VA, 2 * SZ_64K); + mmap_read_unlock(parent); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 0L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent, MM_ANONPAGES), 0L); +} + +struct coarse_test_pin { + struct page *page; +}; + +static void coarse_test_unpin(void *arg) +{ + struct coarse_test_pin *pin = arg; + + if (pin->page) { + unpin_user_page(pin->page); + pin->page = NULL; + } +} + +static void user64k_pinned_fork_test(struct kunit *test) +{ + struct mm_struct *parent, *child; + struct vm_area_struct *vma; + struct coarse_test_pin *pin; + struct folio *old, *new; + pte_t original, copied; + unsigned int i, nr = SZ_64K / PAGE_SIZE; + int ret; + + if (PAGE_SHIFT >= 16) { + kunit_skip(test, "requires pinned coarse leaf"); + return; + } + parent = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, parent); + vma = test_fault_vma_in_mm(parent, TEST_VA, SZ_64K); + KUNIT_ASSERT_NOT_NULL(test, vma); + KUNIT_ASSERT_EQ(test, test_file_fault(vma, TEST_VA + SZ_64K - 1, true) & VM_FAULT_ERROR, 0U); + original = test_fault_entry(test, parent, TEST_VA); + old = page_folio(pfn_to_page(pte_pfn(original))); + for (i = 0; i < nr; i++) + memset(page_address(folio_page(old, i)), 0x70 + i, PAGE_SIZE); + pin = kunit_kzalloc(test, sizeof(*pin), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pin); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, coarse_test_unpin, pin), 0); + mm_flags_set(MMF_HAS_PINNED, parent); + ret = try_grab_folio(old, 1, FOLL_PIN); + if (!ret) + pin->page = folio_page(old, nr - 1); + KUNIT_ASSERT_EQ(test, ret, 0); + child = test_vma_mm_granule(test, 16); + KUNIT_ASSERT_NOT_NULL(test, child); + uprobe_start_dup_mmap(); + ret = dup_mmap(child, parent); + uprobe_end_dup_mmap(); + KUNIT_ASSERT_EQ(test, ret, 0); + copied = test_fault_entry(test, child, TEST_VA); + KUNIT_EXPECT_NE(test, pte_phys_mm(child, copied), pte_phys_mm(parent, original)); + KUNIT_EXPECT_EQ(test, pte_phys_mm(parent, test_fault_entry(test, parent, TEST_VA)), + pte_phys_mm(parent, original)); + new = page_folio(pfn_to_page(pte_pfn(copied))); + KUNIT_EXPECT_EQ(test, folio_mapcount(old), (int)nr); + KUNIT_EXPECT_EQ(test, folio_mapcount(new), (int)nr); + for (i = 0; i < nr; i++) { + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(new, i)), + 0x70 + i, PAGE_SIZE), NULL); + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(new, i))); + KUNIT_EXPECT_TRUE(test, PageAnonExclusive(folio_page(old, i))); + } + memset(page_address(pin->page), 0xe3, PAGE_SIZE); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(page_address(folio_page(new, nr - 1)), + 0x70 + nr - 1, PAGE_SIZE), NULL); + coarse_test_unpin(pin); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(parent, MM_ANONPAGES), 1L); + KUNIT_EXPECT_EQ(test, get_mm_counter_sum(child, MM_ANONPAGES), 1L); +} + +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) +static void user4k_shmem_partial_hole_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 3 * PAGE_SIZE, true); + struct folio *folio; + loff_t start = PAGE_SIZE + SZ_4K + 17, end = PAGE_SIZE + 3 * SZ_4K + 16; + unsigned int i; + unsigned char byte = 0xa5; + loff_t write_pos = PAGE_SIZE + 2 * SZ_4K + 23; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shmem_truncate_range(file_inode(file), start, end); + folio = filemap_lock_folio(file->f_mapping, 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 1), BIT(2)); + for (i = 0; i < PAGE_SIZE; i++) { + unsigned char expected = (i + PAGE_SIZE >= start && i + PAGE_SIZE <= end) ? + 0 : 0x31 + (PAGE_SIZE + i) / SZ_4K; + + if (((unsigned char *)folio_address(folio))[i] != expected) { + KUNIT_FAIL(test, "partial hole changed byte %u outside its range", i); + break; + } + } + /* Resident hole metadata has no dynamically allocated inode record. */ + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + folio_unlock(folio); + folio_put(folio); + shmem_truncate_range(file_inode(file), PAGE_SIZE + SZ_4K, PAGE_SIZE + 2 * SZ_4K - 1); + KUNIT_ASSERT_EQ(test, kernel_write(file, &byte, 1, &write_pos), 1L); + folio = filemap_lock_folio(file->f_mapping, 1); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 1), BIT(1)); + KUNIT_EXPECT_EQ(test, ((unsigned char *)folio_address(folio))[2 * SZ_4K + 23], byte); + folio_unlock(folio); + folio_put(folio); +} + +/* A surviving writable pin exposes bytes even after its PTE is revoked. */ +static void user4k_shmem_pinned_hole_test(struct kunit *test) +{ + unsigned int native; + + for (native = 0; native < 2; native++) { + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct vm_area_struct *vma; + struct fragment_test_pins *pins; + struct folio *folio; + unsigned long offset = native ? 0 : SZ_4K; + unsigned long length = native ? PAGE_SIZE : SZ_4K; + unsigned long pfn; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + vma = test_file_vma(test, file, native, true, TEST_VA, length, offset); + KUNIT_ASSERT_NOT_NULL(test, vma); + pins = kunit_kzalloc(test, sizeof(*pins), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pins); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, fragment_test_unpin, pins), 0); + KUNIT_ASSERT_EQ(test, fragment_test_pin(pins, vma->vm_mm, TEST_VA, + length, FOLL_WRITE | FOLL_LONGTERM, 1), 1L); + folio = pins->fragments[0].folio; + pfn = folio_pfn(folio); + KUNIT_ASSERT_TRUE(test, folio_maybe_dma_pinned(folio)); + shmem_truncate_range(file_inode(file), SZ_4K, 2 * SZ_4K - 1); + KUNIT_EXPECT_FALSE(test, pte_present(test_fault_entry(test, vma->vm_mm, TEST_VA))); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0, SZ_4K), NULL); + /* Model a completed device write through the original pin. */ + memset(folio_address(folio) + SZ_4K, 0xb7, SZ_4K); + folio_lock(folio); + KUNIT_EXPECT_FALSE(test, shmem_uffd_range_missing(folio, 0, SZ_4K, SZ_4K)); + folio_unlock(folio); + fragment_test_unpin(pins); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, folio_pfn(folio), pfn); + KUNIT_EXPECT_FALSE(test, shmem_uffd_range_missing(folio, 0, SZ_4K, SZ_4K)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0xb7, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0x31, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + 2 * SZ_4K, 0x33, SZ_4K), NULL); + folio_unlock(folio); + folio_put(folio); + /* With the external writer gone, another hole can be missing. */ + shmem_truncate_range(file_inode(file), SZ_4K, 2 * SZ_4K - 1); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_TRUE(test, shmem_uffd_range_missing(folio, 0, SZ_4K, SZ_4K)); + folio_unlock(folio); + folio_put(folio); + } +} + +static void test_unpin_folio(void *folio) +{ + unpin_folio(folio); +} + +/* The direct memfd API must publish cache exposure without a user PTE. */ +static void user4k_shmem_memfd_pin_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct folio *pin, *folio; + pgoff_t offset; + long ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shmem_truncate_range(file_inode(file), SZ_4K, 2 * SZ_4K - 1); + ret = memfd_pin_folios(file, SZ_4K, 2 * SZ_4K - 1, &pin, 1, &offset); + KUNIT_ASSERT_EQ(test, ret, 1L); + KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test, test_unpin_folio, pin), 0); + KUNIT_EXPECT_EQ(test, offset, (pgoff_t)SZ_4K); + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(pin)); + memset(folio_address(pin) + offset, 0xc8, SZ_4K); + folio_lock(pin); + KUNIT_EXPECT_FALSE(test, shmem_uffd_range_missing(pin, 0, SZ_4K, SZ_4K)); + folio_unlock(pin); + kunit_release_action(test, test_unpin_folio, pin); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_FALSE(test, shmem_uffd_range_missing(folio, 0, SZ_4K, SZ_4K)); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0xc8, SZ_4K), NULL); + folio_unlock(folio); + folio_put(folio); +} + +static void user4k_shmem_replace_validity_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, true); + struct folio *old, *new; + unsigned long old_pfn; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + shmem_truncate_range(file_inode(file), SZ_4K, 2 * SZ_4K - 1); + new = folio_alloc(GFP_KERNEL, 0); + KUNIT_ASSERT_NOT_NULL(test, new); + folio_lock(new); + old = filemap_lock_folio(file->f_mapping, 0); + if (IS_ERR(old)) { + folio_unlock(new); + folio_put(new); + KUNIT_FAIL(test, "could not lock source cache folio"); + return; + } + old_pfn = folio_pfn(old); + memcpy(folio_address(new), folio_address(old), PAGE_SIZE); + __folio_set_swapbacked(new); + __folio_mark_uptodate(new); + folio_mark_dirty(new); + replace_page_cache_folio(old, new); + folio_add_lru(new); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(new, 0), BIT(1)); + KUNIT_EXPECT_NE(test, folio_pfn(new), old_pfn); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(new), 0x31, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(new) + SZ_4K, 0, SZ_4K), NULL); + folio_unlock(old); + folio_put(old); + folio_unlock(new); + folio_put(new); +} + +static void user4k_shmem_uffd_alias_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, false); + const unsigned long all = GENMASK(PAGE_SIZE / SZ_4K - 1, 0); + struct vm_area_struct *small, *native; + struct folio *folio; + int ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + ret = shmem_get_folio(file_inode(file), 0, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = shmem_uffd_set_missing(folio, 0, all); + KUNIT_EXPECT_TRUE(test, shmem_uffd_has_missing_native_page(folio)); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + small = test_file_vma(test, file, false, true, TEST_VA, SZ_4K, SZ_4K); + KUNIT_ASSERT_NOT_NULL(test, small); + KUNIT_ASSERT_EQ(test, test_file_fault(small, TEST_VA, false) & VM_FAULT_ERROR, 0U); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), all & ~BIT(1)); + KUNIT_EXPECT_FALSE(test, shmem_uffd_has_missing_native_page(folio)); + folio_unlock(folio); + folio_put(folio); + + native = test_file_vma(test, file, true, true, TEST_VA, PAGE_SIZE, 0); + KUNIT_ASSERT_NOT_NULL(test, native); + KUNIT_ASSERT_EQ(test, test_file_fault(native, TEST_VA, false) & VM_FAULT_ERROR, 0U); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), 0UL); + folio_unlock(folio); + folio_put(folio); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); +} + +static void user4k_shmem_uffd_state_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, 3 * PAGE_SIZE, false); + const unsigned long all = GENMASK(PAGE_SIZE / SZ_4K - 1, 0); + struct folio *folio; + unsigned long mask; + unsigned int i; + int ret; + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + for (i = 0; i < 3; i++) { + ret = shmem_get_folio(file_inode(file), i, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, i), 0UL); + ret = shmem_uffd_set_missing(folio, i, all & ~BIT(0)); + KUNIT_EXPECT_EQ(test, ret, 0); + shmem_uffd_mark_present(folio, i, BIT(1)); + mask = shmem_uffd_missing_mask(folio, i); + KUNIT_EXPECT_EQ(test, mask, all & ~GENMASK(1, 0)); + ret = shmem_uffd_set_missing(folio, i, BIT(PAGE_SIZE / SZ_4K)); + KUNIT_EXPECT_EQ(test, ret, -EINVAL); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, i), mask); + folio_unlock(folio); + folio_put(folio); + } + /* Real truncate removes one index without discarding neighbouring state. */ + shmem_truncate_range(file_inode(file), PAGE_SIZE, 2 * PAGE_SIZE - 1); + KUNIT_EXPECT_PTR_EQ(test, xa_load(&SHMEM_I(file_inode(file))->uffd_missing, 1), NULL); + for (i = 0; i < 3; i += 2) { + folio = filemap_lock_folio(file->f_mapping, i); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, i), all & ~GENMASK(1, 0)); + folio_unlock(folio); + folio_put(folio); + } + /* Generic batch deletion must not strand inode metadata either. */ + truncate_inode_pages(file->f_mapping, 0); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); + ret = shmem_get_folio(file_inode(file), 1, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 1), 0UL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0, PAGE_SIZE), NULL); + folio_unlock(folio); + folio_put(folio); +} + +#ifdef CONFIG_MIGRATION +static void user4k_shmem_uffd_migration_test(struct kunit *test) +{ + struct file *file = test_cache_file(test, PAGE_SIZE, false); + const unsigned long missing = GENMASK(PAGE_SIZE / SZ_4K - 1, 1); + struct folio *folio; + unsigned long old_pfn; + unsigned int succeeded = 0; + bool isolated; + int ret; + LIST_HEAD(folios); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + ret = shmem_get_folio(file_inode(file), 0, 0, &folio, SGP_CACHE); + KUNIT_ASSERT_EQ(test, ret, 0); + memset(folio_address(folio), 0xa5, SZ_4K); + folio_mark_dirty(folio); + ret = shmem_uffd_set_missing(folio, 0, missing); + old_pfn = folio_pfn(folio); + folio_unlock(folio); + folio_put(folio); + KUNIT_ASSERT_EQ(test, ret, 0); + lru_add_drain_all(); + folio = filemap_get_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + isolated = isolate_folio_to_list(folio, &folios); + folio_put(folio); + KUNIT_ASSERT_TRUE(test, isolated); + ret = migrate_pages(&folios, test_migration_alloc, NULL, 0, MIGRATE_SYNC, + MR_SYSCALL, &succeeded); + putback_movable_pages(&folios); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_EQ(test, succeeded, 1U); + folio = filemap_lock_folio(file->f_mapping, 0); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folio); + KUNIT_EXPECT_NE(test, folio_pfn(folio), old_pfn); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), missing); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio), 0xa5, SZ_4K), NULL); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(folio_address(folio) + SZ_4K, 0, + PAGE_SIZE - SZ_4K), NULL); + shmem_uffd_mark_present(folio, 0, missing); + KUNIT_EXPECT_EQ(test, shmem_uffd_missing_mask(folio, 0), 0UL); + folio_unlock(folio); + folio_put(folio); + KUNIT_EXPECT_TRUE(test, xa_empty(&SHMEM_I(file_inode(file))->uffd_missing)); +} +#endif +#endif + +static struct kunit_case user4k_test_cases[] = { + KUNIT_CASE(user4k_exec_granule_policy_test), + KUNIT_CASE(user4k_aslr_window_test), + KUNIT_CASE(user4k_thp_eligibility_test), + KUNIT_CASE(user4k_geometry_test), + KUNIT_CASE(user4k_leaf_encoding_test), + KUNIT_CASE(user16k_leaf_encoding_test), + KUNIT_CASE(user4k_rmap_walk_test), + KUNIT_CASE(user4k_rmap_boundary_test), + KUNIT_CASE(native_rmap_walk_test), + KUNIT_CASE(user64k_rmap_lookup_test), + KUNIT_CASE(user64k_anon_rmap_span_test), + KUNIT_CASE(user64k_anon_zap_batch_test), + KUNIT_CASE(user_granule_zero_region_test), +#ifdef CONFIG_COREDUMP + KUNIT_CASE(user_granule_core_offsets_test), +#endif + KUNIT_CASE(user64k_fragment_ownership_test), + KUNIT_CASE(user64k_anon_fault_cow_test), + KUNIT_CASE(user64k_pinned_fork_test), + KUNIT_CASE(user64k_file_rmap_span_test), + KUNIT_CASE(user4k_vma_offsets_test), + KUNIT_CASE(user4k_vma_intervals_test), + KUNIT_CASE(user4k_vma_split_test), + KUNIT_CASE(user4k_vma_merge_test), + KUNIT_CASE(user4k_vma_partial_merge_test), + KUNIT_CASE(user4k_vma_copy_shrink_test), + KUNIT_CASE(user4k_anon_rmap_test), + KUNIT_CASE(user4k_anon_fork_rmap_test), + KUNIT_CASE(user4k_nonlinear_rmap_reuse_test), + KUNIT_CASE(user4k_dup_mmap_test), + KUNIT_CASE(user4k_anon_root_lifetime_test), + KUNIT_CASE(user4k_data_tlb_batch_test), + KUNIT_CASE(user4k_anon_zap_test), + KUNIT_CASE(user4k_anon_writable_slots_test), + KUNIT_CASE(user4k_anon_write_fault_test), + KUNIT_CASE(user16k_anon_write_fault_test), + KUNIT_CASE(user4k_mlock_counts_test), + KUNIT_CASE(user4k_mlock_shared_test), + KUNIT_CASE(user4k_mlock_repair_test), + KUNIT_CASE(user_granule_large_vma_split_test), + KUNIT_CASE(user_granule_brk_extent_test), +#ifdef CONFIG_MIGRATION + KUNIT_CASE(user4k_droppable_reclaim_test), + KUNIT_CASE(user4k_lazyfree_test), +#endif + KUNIT_CASE(user4k_zero_cow_test), + KUNIT_CASE(user16k_zero_cow_test), + KUNIT_CASE(user4k_shmem_swap_usage_test), +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + KUNIT_CASE(user4k_shmem_partial_hole_test), + KUNIT_CASE(user4k_shmem_pinned_hole_test), + KUNIT_CASE(user4k_shmem_memfd_pin_test), + KUNIT_CASE(user4k_shmem_replace_validity_test), + KUNIT_CASE(user4k_shmem_uffd_alias_test), + KUNIT_CASE(user4k_shmem_uffd_state_test), +#ifdef CONFIG_MIGRATION + KUNIT_CASE(user4k_shmem_uffd_migration_test), +#endif +#endif + KUNIT_CASE(user4k_file_mapcounts_test), + KUNIT_CASE(user64k_file_promote_test), + KUNIT_CASE(user64k_shmem_fault_test), + KUNIT_CASE(user64k_shmem_fault_busy_test), + KUNIT_CASE(user64k_shmem_fault_hole_test), + KUNIT_CASE(user64k_file_leaf_test), + KUNIT_CASE(user64k_file_cow_test), + KUNIT_CASE(user64k_file_bounds_test), + KUNIT_CASE(user4k_file_shared_test), + KUNIT_CASE(user4k_iommufd_lifetime_test), + KUNIT_CASE(user4k_iommufd_rollback_test), + KUNIT_CASE(user4k_iommufd_granule_test), + KUNIT_CASE(user4k_iommufd_shared_test), + KUNIT_CASE(user4k_iommufd_exit_account_test), + KUNIT_CASE(user4k_iova_bitmap_test), + KUNIT_CASE(user4k_iova_bitmap_window_test), + KUNIT_CASE(user4k_iova_bitmap_retained_test), + KUNIT_CASE(user4k_iova_bitmap_partial_test), + KUNIT_CASE(user4k_mremap_move_test), + KUNIT_CASE(user4k_mremap_resize_test), + KUNIT_CASE(user4k_stack_relocate_test), + KUNIT_CASE(user4k_exec_arguments_test), + KUNIT_CASE(user4k_special_mapping_test), + KUNIT_CASE(native_callback_mapping_test), + KUNIT_CASE(user4k_callback_mapping_test), + KUNIT_CASE(user16k_callback_mapping_test), + KUNIT_CASE(user4k_insert_fragment_test), + KUNIT_CASE(user16k_insert_shared_test), + KUNIT_CASE(user4k_insert_range_test), + KUNIT_CASE(user4k_pfn_offset_test), + KUNIT_CASE(user4k_remap_pfn_range_test), + KUNIT_CASE(user4k_iomem_access_test), +#if defined(CONFIG_GENERIC_GETTIMEOFDAY) && defined(CONFIG_VDSO_GETRANDOM) + KUNIT_CASE(user4k_vdso_mapping_test), +#endif + KUNIT_CASE(user4k_mremap_cow_test), + KUNIT_CASE(user4k_mremap_file_test), + KUNIT_CASE(user4k_msync_offsets_test), + KUNIT_CASE(user4k_prefault_dirty_test), + KUNIT_CASE(user4k_cold_batch_test), + KUNIT_CASE(user4k_remote_access_test), + KUNIT_CASE(user4k_remote_file_test), +#ifdef CONFIG_CROSS_MEMORY_ATTACH + KUNIT_CASE(user4k_process_vm_test), +#endif +#ifdef CONFIG_FUTEX + KUNIT_CASE(user4k_futex_keys_test), + KUNIT_CASE(user4k_futex_wake_test), +#endif + KUNIT_CASE(user4k_mmap_descriptor_test), + KUNIT_CASE(user4k_mmap_region_offsets_test), + KUNIT_CASE(user4k_commit_accounting_test), + KUNIT_CASE(mixed_locked_account_units_test), + KUNIT_CASE(user64k_commit_accounting_test), + KUNIT_CASE(user64k_page_table_check_test), +#ifdef CONFIG_PROC_FS + KUNIT_CASE(user4k_accounting_readers_test), +#endif + KUNIT_CASE(user_granule_prefault_test), + KUNIT_CASE(user4k_mmap_accounting_test), + KUNIT_CASE(user4k_vm_mmap_test), + KUNIT_CASE(user4k_mprotect_test), + KUNIT_CASE(user4k_mprotect_cow_batch_test), +#ifdef CONFIG_NUMA_BALANCING + KUNIT_CASE(user_subpage_numa_cow_batch_test), +#endif + KUNIT_CASE(user4k_mprotect_accounting_test), + KUNIT_CASE(user4k_file_cow_test), + KUNIT_CASE(user4k_file_eof_test), + KUNIT_CASE(user4k_file_truncate_test), + KUNIT_CASE(user4k_filemap_fault_test), + KUNIT_CASE(user4k_file_invalidate_test), + KUNIT_CASE(user4k_file_fragments_test), + KUNIT_CASE(user4k_file_fork_hole_test), + KUNIT_CASE(user4k_swap_encoding_test), + KUNIT_CASE(user4k_cow_reuse_test), + KUNIT_CASE(user16k_cow_reuse_test), + KUNIT_CASE(user4k_fork_cow_test), + KUNIT_CASE(user16k_fork_cow_test), + KUNIT_CASE(user4k_slot_pin_test), + KUNIT_CASE(user4k_fragment_pin_test), + KUNIT_CASE(user4k_fragment_locked_pin_test), + KUNIT_CASE(user4k_folio_walk_geometry_test), + KUNIT_CASE(user4k_fragment_longterm_test), +#ifdef CONFIG_BLOCK + KUNIT_CASE(user4k_bio_fragments_test), + KUNIT_CASE(user4k_bio_trim_test), +#endif + KUNIT_CASE(user4k_fragment_permissions_test), + KUNIT_CASE(user16k_fragment_offset_test), +#ifdef CONFIG_MIGRATION + KUNIT_CASE(user4k_migration_test), + KUNIT_CASE(user4k_file_migration_test), + KUNIT_CASE(user64k_file_migration_test), + KUNIT_CASE(user64k_anon_migration_test), + KUNIT_CASE(user4k_fragment_pin_race_test), + KUNIT_CASE(user4k_migration_entries_test), + KUNIT_CASE(user4k_nonlinear_rmap_test), + KUNIT_CASE(user4k_mprotect_migration_test), + KUNIT_CASE(user4k_live_migration_test), + KUNIT_CASE(user4k_migration_wait_test), +#endif + KUNIT_CASE(user4k_anon_exit_test), + KUNIT_CASE(user4k_anon_remote_zap_test), +#ifdef CONFIG_ARM64_CONTPTE + KUNIT_CASE(user4k_contpte_test), + KUNIT_CASE(user16k_contpte_test), + KUNIT_CASE(native_contpte_test), +#endif +#ifdef CONFIG_PERF_EVENTS + KUNIT_CASE(user4k_perf_page_size_test), + KUNIT_CASE(user16k_perf_page_size_test), + KUNIT_CASE(native_perf_page_size_test), +#endif + KUNIT_CASE(user4k_mm_tables_test), + KUNIT_CASE(user16k_mm_tables_test), + KUNIT_CASE(user64k_mm_tables_test), + KUNIT_CASE(native_mm_tables_test), + KUNIT_CASE(user4k_anon_read_fault_test), + KUNIT_CASE(user4k_mm_hardware_test), + KUNIT_CASE(user4k_fragment_lifetime_test), + KUNIT_CASE(user16k_fragment_lifetime_test), + KUNIT_CASE(user64k_table_owner_test), +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + KUNIT_CASE(native_pgtable_deposit_test), +#endif + KUNIT_CASE(user4k_translation_test), + KUNIT_CASE(user4k_context_uaccess_test), + KUNIT_CASE(user4k_fault_uaccess_test), + {} +}; + +static struct kunit_suite user4k_test_suite = { + .name = "arm64-user4k", + .test_cases = user4k_test_cases, +}; + +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) +kunit_test_suites(&user4k_test_suite, &user4k_swap_suite, &user4k_shmem_swap_suite, + &user4k_coarse_shmem_suite, + &user4k_nonlinear_swap_suite); +#else +kunit_test_suites(&user4k_test_suite, &user4k_swap_suite, &user4k_nonlinear_swap_suite); +#endif + +MODULE_LICENSE("GPL"); +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING"); diff --git a/arch/arm64/mm/user4k-tlb-test.c b/arch/arm64/mm/user4k-tlb-test.c new file mode 100644 index 00000000000000..e5671256b7c76e --- /dev/null +++ b/arch/arm64/mm/user4k-tlb-test.c @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Decode emitted operands without executing TLBI instructions. */ +#include +#include +#include +#include + +struct tlbi_test_state { + u64 next; + u64 end; + u64 stride; + u16 asid; + u32 level; + unsigned int shift; + unsigned int single_ops; + unsigned int range_ops; + bool lpa2; +}; + +static unsigned int test_tg(unsigned int shift) +{ + return shift == 12 ? 1 : shift == 14 ? 2 : 3; +} + +static void record_single(u64 arg) +{ + struct kunit *test = kunit_get_current_test(); + struct tlbi_test_state *state = test->priv; + u64 address = (arg & GENMASK_ULL(43, 0)) << 12; + u64 hint = 0; + + if (alternative_has_cap_unlikely(ARM64_HAS_ARMv8_4_TTL) && state->level <= 3) + hint = (test_tg(state->shift) << 2) | state->level; + KUNIT_EXPECT_EQ(test, address, state->next); + KUNIT_EXPECT_EQ(test, arg >> 48, (u64)state->asid); + KUNIT_EXPECT_EQ(test, FIELD_GET(TLBI_TTL_MASK, arg), hint); + state->next += state->stride; + state->single_ops++; + KUNIT_EXPECT_LE(test, state->next, state->end); +} + +static void record_range(u64 arg) +{ + struct kunit *test = kunit_get_current_test(); + struct tlbi_test_state *state = test->priv; + u64 address = FIELD_GET(TLBIR_BADDR_MASK, arg) << (state->lpa2 ? 16 : state->shift); + u64 count = (FIELD_GET(TLBIR_NUM_MASK, arg) + 1) << + (5 * FIELD_GET(TLBIR_SCALE_MASK, arg) + 1); + + KUNIT_EXPECT_EQ(test, address, state->next); + KUNIT_EXPECT_EQ(test, arg >> 48, (u64)state->asid); + KUNIT_EXPECT_EQ(test, FIELD_GET(TLBIR_TG_MASK, arg), (u64)test_tg(state->shift)); + KUNIT_EXPECT_EQ(test, FIELD_GET(TLBIR_TTL_MASK, arg), + state->level > 3 ? 0ULL : (u64)state->level); + if (state->lpa2) + KUNIT_EXPECT_TRUE(test, IS_ALIGNED(address, SZ_64K)); + state->next += count << state->shift; + state->range_ops++; + KUNIT_EXPECT_LE(test, state->next, state->end); +} + +static bool range_supported(void) +{ + return true; +} + +static bool range_unsupported(void) +{ + return false; +} + +static void test_flush(struct kunit *test, unsigned int shift, u64 start, + unsigned long pages, u64 stride, u32 level, bool lpa2, + bool has_range) +{ + struct tlbi_test_state state = { + .next = start, + .end = start + (pages << shift), + .stride = stride, + .asid = 0xa55a, + .level = level, + .shift = shift, + .lpa2 = lpa2, + }; + + test->priv = &state; + __flush_tlb_range_op_granule(record_single, record_range, start, pages, + stride, state.asid, level, lpa2, shift, + has_range ? range_supported : range_unsupported); + KUNIT_EXPECT_EQ(test, state.next, state.end); + if (has_range) { + KUNIT_EXPECT_LE(test, state.single_ops + state.range_ops, 20U); + if (pages >= 32) + KUNIT_EXPECT_GT(test, state.range_ops, 0U); + } else { + KUNIT_EXPECT_EQ(test, state.range_ops, 0U); + KUNIT_EXPECT_EQ(test, state.single_ops, (unsigned int)((pages << shift) / stride)); + } + test->priv = NULL; +} + +static void user4k_tlbi_operands_test(struct kunit *test) +{ + static const unsigned long counts[] = { + 0, 1, 2, 3, 31, 32, 33, 63, 64, 65, 2047, 2048, 2049, + 65535, 65536, 65537, MAX_TLBI_RANGE_PAGES - 1, MAX_TLBI_RANGE_PAGES, + }; + static const u32 levels[] = { 2, 3, TLBI_TTL_UNKNOWN }; + unsigned int shift, i, j; + + /* Force both algorithms using recording callbacks, independent of CPU caps. */ + for (shift = 12; shift <= 16; shift += 2) { + u64 size = 1UL << shift; + + for (j = 0; j < ARRAY_SIZE(levels); j++) { + for (i = 0; i < ARRAY_SIZE(counts); i++) { + unsigned long pages = counts[i]; + + test_flush(test, shift, SZ_4M + size, pages, size, + levels[j], false, true); + if (shift != 16) + test_flush(test, shift, SZ_4M + size, pages, size, + levels[j], true, true); + if (pages <= 65) + test_flush(test, shift, SZ_4M + size, pages, size, + levels[j], false, false); + } + } + /* Preserve a larger leaf stride when range instructions are absent. */ + test_flush(test, shift, SZ_1G, 3 * 512, 512 * size, 2, false, false); + /* Last virtual page must not lose high address bits. */ + test_flush(test, shift, (1ULL << 48) - size, 1, size, 3, false, false); + } +} + +static void user4k_tlb_gather_test(struct kunit *test) +{ + struct mm_struct *mm = kunit_kzalloc(test, sizeof(*mm), GFP_KERNEL); + pte_t entries[3] = {}; + unsigned int small; + + KUNIT_ASSERT_NOT_NULL(test, mm); + for (small = 0; small < 2; small++) { + struct mmu_gather tlb = { .mm = mm, .start = ULONG_MAX }; + unsigned long size = small ? SZ_4K : PAGE_SIZE; + unsigned long address = SZ_4M + size; + + mm->context.flags = small ? MMCF_USER_4K : 0; + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), size); + tlb_remove_tlb_entries(&tlb, entries, 3, address); + KUNIT_EXPECT_EQ(test, tlb.start, address); + KUNIT_EXPECT_EQ(test, tlb.end, address + 3 * size); + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), size); + KUNIT_EXPECT_EQ(test, tlb_get_level(&tlb), 3); + tlb.cleared_pmds = 1; + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), size); + KUNIT_EXPECT_EQ(test, tlb_get_level(&tlb), TLBI_TTL_UNKNOWN); + tlb.cleared_ptes = 0; + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), small ? SZ_2M : PMD_SIZE); + tlb.cleared_pmds = 0; + tlb.cleared_puds = 1; + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), small ? SZ_1G : PUD_SIZE); + tlb.cleared_puds = 0; + tlb.cleared_p4ds = 1; + KUNIT_EXPECT_EQ(test, tlb_get_unmap_size(&tlb), small ? 1UL << 39 : P4D_SIZE); + tlb.freed_tables = 1; + KUNIT_EXPECT_EQ(test, tlb_get_level(&tlb), TLBI_TTL_UNKNOWN); + } +} + +static struct kunit_case user4k_tlb_test_cases[] = { + KUNIT_CASE(user4k_tlbi_operands_test), + KUNIT_CASE(user4k_tlb_gather_test), + {} +}; + +static struct kunit_suite user4k_tlb_test_suite = { + .name = "arm64-user4k-tlb", + .test_cases = user4k_tlb_test_cases, +}; + +kunit_test_suite(user4k_tlb_test_suite); + +MODULE_LICENSE("GPL"); diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index de92d63fdbc763..432df2ce9ddfd4 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -69,6 +69,8 @@ HAS_TIDCP1 HAS_TLB_RANGE HAS_TSO_APPLE HAS_TSO_FIXED +HAS_USER4K_GRANULE +HAS_USER16K_GRANULE HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 3af979fb41d38d..3155eed81c2ecb 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -926,13 +926,15 @@ static int is_call_insn(uprobe_opcode_t *insn) * Verification callback used by int3_update uprobe_write calls to make sure * the underlying instruction is as expected - either int3 or call. */ -static int verify_insn(struct page *page, unsigned long vaddr, uprobe_opcode_t *new_opcode, +static int verify_insn(struct page *page, unsigned long vaddr, unsigned int page_offset, + uprobe_opcode_t *new_opcode, int nbytes, void *data) { struct write_opcode_ctx *ctx = data; uprobe_opcode_t old_opcode[5]; - uprobe_copy_from_page(page, ctx->base, (uprobe_opcode_t *) &old_opcode, 5); + uprobe_copy_from_page(page, page_offset - (vaddr - ctx->base), + (uprobe_opcode_t *) &old_opcode, 5); switch (ctx->expect) { case EXPECT_SWBP: diff --git a/block/bio.c b/block/bio.c index 5f10900b3f42a1..19805b79cf0b26 100644 --- a/block/bio.c +++ b/block/bio.c @@ -3,6 +3,7 @@ * Copyright (C) 2001 Jens Axboe */ #include +#include #include #include #include @@ -246,6 +247,10 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, #endif #ifdef CONFIG_BLK_DEV_INTEGRITY bio->bi_integrity = NULL; +#endif +#ifdef CONFIG_MM_SUBPAGE + bio->bi_user_fragments = NULL; + bio->bi_nr_user_fragments = 0; #endif bio->bi_vcnt = 0; @@ -1159,6 +1164,18 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) { struct folio_iter fi; +#ifdef CONFIG_MM_SUBPAGE + if (bio->bi_user_fragments) { + release_user_fragments(bio->bi_user_fragments, + bio->bi_nr_user_fragments, mark_dirty); + kfree(bio->bi_user_fragments); + bio->bi_user_fragments = NULL; + bio->bi_nr_user_fragments = 0; + bio_clear_flag(bio, BIO_PAGE_PINNED); + return; + } +#endif + bio_for_each_folio_all(fi, bio) { size_t nr_pages; @@ -1220,6 +1237,137 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter, return 0; } +#ifdef CONFIG_MM_SUBPAGE +static bool bio_subpage_iter(const struct iov_iter *iter) +{ + return user_backed_iter(iter) && current->mm && + mm_page_size(current->mm) < PAGE_SIZE; +} + +int bio_iov_user_vecs(const struct iov_iter *iter, int max_vecs) +{ + struct iov_iter walk = *iter; + unsigned long size; + unsigned int count = 0; + + if (!bio_subpage_iter(iter)) + return iov_iter_npages(iter, max_vecs); + size = mm_user_fragment_size(current->mm); + while (iov_iter_count(&walk) && count < max_vecs) { + size_t len, offset, nr; + + iov_iter_advance(&walk, 0); + len = min(iter_iov_len(&walk), iov_iter_count(&walk)); + if (WARN_ON_ONCE(!len)) + return max_vecs; + offset = (unsigned long)iter_iov_addr(&walk) & (size - 1); + nr = len / size + DIV_ROUND_UP(offset + (len & (size - 1)), size); + if (nr >= max_vecs - count) + return max_vecs; + count += nr; + iov_iter_advance(&walk, len); + } + return count; +} +EXPORT_SYMBOL_GPL(bio_iov_user_vecs); + +/* Pins belong to the submitting bio, just as with ordinary BIO_PAGE_PINNED. + * Split/cloned bios borrow its vectors and complete before their owner does. + */ +static int bio_iov_iter_get_fragments(struct bio *bio, struct iov_iter *iter, + unsigned int align_mask) +{ + unsigned int capacity = bio->bi_max_vecs, total = 0; + unsigned int flags = iov_iter_rw(iter) == READ ? FOLL_WRITE : 0; + int ret = 0; + + if (WARN_ON_ONCE(bio->bi_vcnt || bio->bi_user_fragments || !capacity)) + return -EINVAL; + if (iter->nofault) + flags |= FOLL_NOFAULT; + bio->bi_user_fragments = kmalloc_array(capacity, + sizeof(*bio->bi_user_fragments), GFP_KERNEL); + if (!bio->bi_user_fragments) + return -ENOMEM; + while (iov_iter_count(iter) && bio->bi_nr_user_fragments < capacity && + total < BIO_MAX_SIZE) { + unsigned int i, bytes = 0; + long nr; + + iov_iter_advance(iter, 0); + nr = pin_user_fragments_remote(current->mm, + (unsigned long)iter_iov_addr(iter), + min3(iter_iov_len(iter), iov_iter_count(iter), + (size_t)(BIO_MAX_SIZE - total)), flags, + bio->bi_user_fragments + bio->bi_nr_user_fragments, + capacity - bio->bi_nr_user_fragments); + if (nr <= 0) { + ret = nr ?: -EFAULT; + break; + } + for (i = 0; i < nr; i++) { + struct user_page_fragment *fragment = + &bio->bi_user_fragments[bio->bi_nr_user_fragments++]; + struct page *page = folio_page(fragment->folio, + fragment->offset >> PAGE_SHIFT); + unsigned int offset = offset_in_page(fragment->offset); + struct bio_vec *prev = bio->bi_vcnt ? + &bio->bi_io_vec[bio->bi_vcnt - 1] : NULL; + + /* Coalesce physically adjacent quarters within one folio. */ + if (prev && page_folio(prev->bv_page) == fragment->folio && + page_to_phys(prev->bv_page) + prev->bv_offset + prev->bv_len == + page_to_phys(page) + offset) + prev->bv_len += fragment->length; + else + bvec_set_page(&bio->bi_io_vec[bio->bi_vcnt++], page, + fragment->length, offset); + bytes += fragment->length; + } + iov_iter_advance(iter, bytes); + total += bytes; + } + /* Trim pins separately from vectors: several pins may share one bvec. */ + if (total & align_mask) { + unsigned int trim = total & align_mask, left = trim; + + while (left) { + struct user_page_fragment *fragment = + &bio->bi_user_fragments[bio->bi_nr_user_fragments - 1]; + unsigned int bytes = min(left, fragment->length); + + if (bytes == fragment->length) { + release_user_fragments(fragment, 1, false); + bio->bi_nr_user_fragments--; + } else { + fragment->length -= bytes; + } + left -= bytes; + } + left = trim; + while (left) { + struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1]; + unsigned int bytes = min(left, bv->bv_len); + + bv->bv_len -= bytes; + if (!bv->bv_len) + bio->bi_vcnt--; + left -= bytes; + } + iov_iter_revert(iter, trim); + total -= trim; + } + bio->bi_iter.bi_size = total; + if (!total) { + kfree(bio->bi_user_fragments); + bio->bi_user_fragments = NULL; + return ret ?: -EFAULT; + } + bio_set_flag(bio, BIO_PAGE_PINNED); + return 0; +} +#endif + /** * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to @@ -1255,6 +1403,11 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter, return 0; } +#ifdef CONFIG_MM_SUBPAGE + if (bio_subpage_iter(iter)) + return bio_iov_iter_get_fragments(bio, iter, len_align_mask); +#endif + if (iov_iter_extract_will_pin(iter)) bio_set_flag(bio, BIO_PAGE_PINNED); if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) diff --git a/block/blk-map.c b/block/blk-map.c index 768549f19f97ec..b98eed276aea25 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -260,7 +260,7 @@ static int bio_copy_user_iov(struct request *rq, struct rq_map_data *map_data, static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, gfp_t gfp_mask) { - unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); + unsigned int nr_vecs = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS); struct bio *bio; int ret; diff --git a/drivers/accel/ane/ane.h b/drivers/accel/ane/ane.h index eac81bdab9f748..c9d18fb13f5213 100644 --- a/drivers/accel/ane/ane.h +++ b/drivers/accel/ane/ane.h @@ -17,6 +17,15 @@ #include "ane_stats.h" +#define ANE_DART_MAX 3 +#define ANE_DART_SCRATCH_MAX 16 + +struct ane_dart_scratch { + u64 iova; + struct page *page; + struct drm_mm_node *node; +}; + struct ane_device { struct drm_device drm; struct device *dev; @@ -62,6 +71,9 @@ struct ane_device { * nodes (see ane_reclaim_preserved). Registered under engine_lock. */ struct list_head preserved_list; + /* Fault-drain DMA backing, retained until completion or a safe reset. */ + struct ane_dart_scratch scratch[ANE_DART_SCRATCH_MAX]; + int scratch_pages; bool removed; /* @@ -135,15 +147,6 @@ struct ane_request { u32 bar[DRM_ANE_TILE_COUNT]; }; -/* - * Drop the wedge pin (module refcount) without requiring a successful - * recovery. Used by ane_drm_postclose so a wedged engine cannot leave - * the module un-unloadable after a clean session close. Recovery stays - * safe to call afterwards; if it returns error, the postclose path has - * already unlocked the module count so the operator can swap the ko. - */ -void ane_wedge_clear(struct ane_device *ane); - /* * Reclaim wedge-preserved BO ranges (unmap, release nodes and pages). * Only legal once DMA is provably quiescent: after a successful @@ -156,15 +159,6 @@ void ane_boost_exit(struct ane_device *ane); void ane_boost_begin(struct ane_device *ane); void ane_boost_end(struct ane_device *ane); -#define ANE_DART_MAX 3 -#define ANE_DART_SCRATCH_MAX 16 - -struct ane_dart_scratch { - u64 iova; - struct page *page; - struct drm_mm_node *node; -}; - int ane_dart_init(struct ane_device *ane); void ane_dart_mask(struct ane_device *ane); void ane_dart_unmask(struct ane_device *ane); diff --git a/drivers/accel/ane/ane_drv.c b/drivers/accel/ane/ane_drv.c index 70f0d9bd9b3657..c98f2348e82f1b 100644 --- a/drivers/accel/ane/ane_drv.c +++ b/drivers/accel/ane/ane_drv.c @@ -360,6 +360,8 @@ void ane_reclaim_preserved(struct ane_device *ane) unsigned long count = 0; lockdep_assert_held(&ane->engine_lock); + ane_dart_release_scratch(ane, ane->scratch, ane->scratch_pages); + ane->scratch_pages = 0; list_for_each_entry_safe(p, tmp, &ane->preserved_list, list) { u64 iova = p->mm->start; @@ -796,21 +798,7 @@ static int ane_drm_open(struct drm_device *drm, struct drm_file *file) return 0; } -/* - * Per-session teardown: every file context open is a fresh boundary. - * Anything the previous session left on the device (wedged flag, tm - * completion counters, in-flight recovery state) belongs to that - * session and must not survive into the next one. - * - * drm_gem_release runs before us and has dropped every handle this - * file held. BOs leaked by `ane_gem_free_object under wedge` stay on - * ane->bo_list until module teardown (their backing `drm_gem_object`s - * are already gone, and touching them here would be use-after-free). - * We do not try to drain them; we only clear the session-leaving - * state (wedged + tm counters + recovering flag) so the next opener - * sees a clean device. The orphan BOs remain inert: they are not in - * any drm_file's handle table. - */ +/* Retry recovery on close, retaining quarantine if DMA cannot be quiesced. */ static void ane_drm_postclose(struct drm_device *drm, struct drm_file *file) { struct ane_device *ane = drm->dev_private; @@ -821,14 +809,6 @@ static void ane_drm_postclose(struct drm_device *drm, struct drm_file *file) return; } - /* Drop the wedge pin first: a wedged engine that cannot be - * recovered in software must NOT keep the module pinned, or the - * operator cannot unload the ko for an updated build without a - * reboot. ane_wedge_clear releases both the wedged flag and the - * module_refcount; ane_tm_recover then attempts a power-cycle - * recovery (succeeds only if the engine is actually idle-able). - */ - ane_wedge_clear(ane); ane_tm_recover(ane); mutex_unlock(&ane->engine_lock); diff --git a/drivers/accel/ane/ane_stats.h b/drivers/accel/ane/ane_stats.h index 4a2c40784817f4..768fe97846edc3 100644 --- a/drivers/accel/ane/ane_stats.h +++ b/drivers/accel/ane/ane_stats.h @@ -148,7 +148,8 @@ static inline void ane_stats_counters_init(struct ane_stats_counters *ctrs, * * A period never starts before the previous one folded: a caller * sample can be stale (taken before the transition), so the latch is - * max(submit_ns, max_end). + * max(submit_ns, max_end), with max_end read under the transition + * sentinel. */ static inline u64 ane_stats_begin(struct ane_stats_counters *ctrs, struct ane_stats_ring *ring, @@ -157,12 +158,8 @@ static inline u64 ane_stats_begin(struct ane_stats_counters *ctrs, u64 ticket = atomic64_fetch_add(1ull, &ring->head) + 1ull; struct ane_stats_ring_entry *e = &ring->slots[(size_t)(ticket - 1ull) & ring->mask]; - u64 latch = atomic64_read(&ctrs->max_end); - u32 cur; - - if (submit_ns > latch) - latch = submit_ns; - cur = atomic_read(&ctrs->inflight); + u64 latch; + u32 cur = atomic_read(&ctrs->inflight); for (;;) { if (cur == ANE_STATS_INFLIGHT_TRANS) { @@ -173,6 +170,18 @@ static inline u64 ane_stats_begin(struct ane_stats_counters *ctrs, cur = atomic_read(&ctrs->inflight); continue; } + /* + * Read max_end only now. The drainer of the previous + * period fed its end sample into max_end before its + * release store of inflight = 0, and every other + * member fed before its decrement. The fully ordered + * cmpxchg above read that 0, so this read sees every + * feed and the latch is never below the previous + * fold end: periods cannot overlap. + */ + latch = atomic64_read(&ctrs->max_end); + if (submit_ns > latch) + latch = submit_ns; atomic64_set_release(&ctrs->last_busy_end, latch); atomic_set_release(&ctrs->inflight, 1u); break; diff --git a/drivers/accel/ane/ane_tm.c b/drivers/accel/ane/ane_tm.c index 703d595b612a99..ba234d57cdf74f 100644 --- a/drivers/accel/ane/ane_tm.c +++ b/drivers/accel/ane/ane_tm.c @@ -265,8 +265,7 @@ int ane_tm_execute(struct ane_device *ane, struct ane_request *req) u32 finished = 0; u32 fault_status = 0; u64 fault_iova = 0; - struct ane_dart_scratch scratch[ANE_DART_SCRATCH_MAX]; - int scratch_pages = 0; + struct ane_dart_scratch *scratch = ane->scratch; bool faulted = false; int err, status; @@ -298,10 +297,10 @@ int ane_tm_execute(struct ane_device *ane, struct ane_request *req) err = status; if (ane_dart_faulted(ane, &fault_iova, &fault_status)) { faulted = true; - if (scratch_pages >= ANE_DART_SCRATCH_MAX) + if (ane->scratch_pages >= ANE_DART_SCRATCH_MAX) break; if (ane_dart_drain_fault(ane, fault_iova, scratch, - &scratch_pages)) + &ane->scratch_pages)) break; /* The retried transaction now completes; poll on. */ continue; @@ -321,7 +320,7 @@ int ane_tm_execute(struct ane_device *ane, struct ane_request *req) if (err || ane_dart_faulted(ane, NULL, NULL)) { /* Drain failed or the drained program never reached * completion: the engine is not idle and access is - * off the table. wedge releases the scratch. + * off the table. Retain scratch backing until reset. */ err = err ?: -EIO; goto wedge; @@ -334,11 +333,12 @@ int ane_tm_execute(struct ane_device *ane, struct ane_request *req) */ rmb(); tq_write32(ane, TQ_STATUS(req->qid), 0x0); - ane_dart_release_scratch(ane, scratch, scratch_pages); + ane_dart_release_scratch(ane, scratch, ane->scratch_pages); ane_dart_unmask(ane); dev_err(ane->dev, "DART fault contained: status=%#x iova=%#llx scratch=%d\n", - fault_status, fault_iova, scratch_pages); + fault_status, fault_iova, ane->scratch_pages); + ane->scratch_pages = 0; return -EIO; } if (err) @@ -356,7 +356,6 @@ int ane_tm_execute(struct ane_device *ane, struct ane_request *req) return 0; wedge: - ane_dart_release_scratch(ane, scratch, scratch_pages); ane_dart_unmask(ane); if (atomic_xchg(&ane->wedged, 1) == 0) { __module_get(THIS_MODULE); @@ -644,27 +643,3 @@ int ane_tm_recover(struct ane_device *ane) } return 0; } - -/* - * Drop the wedge pin on demand. The recovery path in ane_tm_recover - * does an atomic_xchg + module_put, but only on its success branch - - * a real -110 with the engine stuck leaves ane->wedged set and the - * module refcount one too high, so rmmod fails forever and the bug - * becomes a kernel-brick-without-reboot. ane_drm_postclose calls - * this unconditionally so every session close releases the pin, - * letting the operator unload the ko for an updated build even when - * the engine itself cannot be cleared in software. - * - * Pair count: must follow an ane_wedge_set() that called - * __module_get(THIS_MODULE). Today the only such set is in the wedge: - * branch of ane_tm_execute(); future sets must call this in their - * clean-up path too. - */ -void ane_wedge_clear(struct ane_device *ane) -{ - if (atomic_xchg(&ane->wedged, 0)) { - module_put(THIS_MODULE); - dev_info(ane->dev, - "wedge pin released (engine state unknown)\n"); - } -} diff --git a/drivers/bluetooth/Kconfig b/drivers/bluetooth/Kconfig index 975801c66fe30c..38482dff3ea25e 100644 --- a/drivers/bluetooth/Kconfig +++ b/drivers/bluetooth/Kconfig @@ -37,13 +37,27 @@ config BT_MTK7932_PCIE is mounted and read their firmware when probed, so say M, like the Wi-Fi driver. System suspend uses the firmware sleep handshake and preserves DMA - storage. The PCIe bootstrap retains its own PM and kexec blockers. - Hibernation and automatic firmware recovery are unsupported. + storage. Hibernation and firmware reloading are unsupported. A + controller that stops answering commands is reset over HCI and set + up again; if that setup fails, the transport stays faulted. Removal drains the PCI function before releasing device-visible memory; an unconfirmed drain retains it until a full hardware reset. To build as a module, choose M. The module is mt7932_bt_pcie. +config BT_MTK7932_PCIE_KUNIT_TEST + bool "KUnit tests for the MT7932 Bluetooth transport" if !KUNIT_ALL_TESTS + depends on KUNIT=y && BT_MTK7932_PCIE + default KUNIT_ALL_TESTS + help + Build KUnit tests into the MT7932 Bluetooth transport. They check, + without radio hardware, that HCI packets whose length does not + match their header are rejected in both directions, and that a + board file with a bad layout or checksum is never sent to the + controller. + + If unsure, say N. + config BT_HCIBTUSB tristate "HCI USB driver" depends on USB diff --git a/drivers/bluetooth/mt7932_bt_pcie.c b/drivers/bluetooth/mt7932_bt_pcie.c index 5f812cc42039e6..f4a7e39232d7b6 100644 --- a/drivers/bluetooth/mt7932_bt_pcie.c +++ b/drivers/bluetooth/mt7932_bt_pcie.c @@ -1,7 +1,9 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * J700 MT7932 Bluetooth PCIe transport. - * Authors: DJ (DjDeveloperr), Ace (Acelogic), and Ryan Murray. + * Bluetooth transport for the MediaTek MT7932 radio in the Apple MacBook Neo + * (J700), whose Bluetooth controller is the second function of the radio's + * PCIe device. Transport research by DJ (DjDeveloperr), Ace (Acelogic) and + * Ryan Murray. * * Firmware and board inputs are supplied locally. System sleep preserves the * transport arena and uses the firmware quiesce/restore handshake. Unload @@ -20,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -103,6 +106,14 @@ enum bt7932_pm_state { BT7932_RESTORING, }; +enum bt7932_recovery { + BT7932_RECOVERY_NONE, + /* A Hardware Error event was injected and not yet handled. */ + BT7932_RECOVERY_REQUESTED, + /* The core is closing and reopening the device. */ + BT7932_RECOVERY_REOPENING, +}; + struct bt7932 { struct pci_dev *pdev; void __iomem *bar; @@ -132,8 +143,8 @@ struct bt7932 { bool enabled, regions, vectors, irq_requested, retained; bool ipc_ready, opened, awake, waking, fault, registered; bool stopping, runtime_forbidden; + enum bt7932_recovery recovery; enum bt7932_pm_state pm_state; - u64 tx_packets, rx_packets, diagnostics; }; static void bt7932_ring_doorbell(struct bt7932 *bt, unsigned int bit) @@ -170,7 +181,7 @@ static void bt7932_fault_locked(struct bt7932 *bt, const char *reason) dev_err(&bt->pdev->dev, "terminal transport fault: %s; owned DMA retained, external reset required\n", reason); for (channel = 0; channel < BT7932_CHANNELS; channel++) - dev_err(&bt->pdev->dev, "ring%u host=%u firmware=%u previous=%u\n", channel, + dev_dbg(&bt->pdev->dev, "ring%u host=%u firmware=%u previous=%u\n", channel, bt7932_counter(bt, bt->host_transfer, channel), bt7932_counter(bt, bt->fw_transfer, channel), bt->rings[channel].previous); pci_clear_master(bt->pdev); @@ -402,8 +413,8 @@ static int bt7932_arena(struct bt7932 *bt) context[0x51] = 13; context[0x53] = 2; dma_wmb(); - dev_info(&bt->pdev->dev, "owned SID2 arena=%pad bytes=%zu image=%pad bytes=%zu, both low32\n", - &bt->arena_dma, bt->arena_size, &bt->image_dma, bt->image_size); + dev_dbg(&bt->pdev->dev, "owned SID2 arena=%pad bytes=%zu image=%pad bytes=%zu, both low32\n", + &bt->arena_dma, bt->arena_size, &bt->image_dma, bt->image_size); return 0; } @@ -576,7 +587,6 @@ static void bt7932_tx_work(struct work_struct *work) ring->host = (ring->host + 1) & BT7932_MASK; bt7932_publish_counter(bt, bt->host_transfer, channel, ring->host); bt7932_ring_doorbell(bt, geometry[channel].doorbell); - bt->tx_packets++; bt->hdev->stat.byte_tx += skb->len; mutex_unlock(&bt->lock); kfree_skb(skb); @@ -621,10 +631,7 @@ static int bt7932_rx_locked(struct bt7932 *bt, unsigned int channel) hci_skb_pkt_type(skb) = geo->packet; skb_put_data(skb, descriptor + 16, length); bt->hdev->stat.byte_rx += length; - bt->rx_packets++; hci_recv_frame(bt->hdev, skb); - } else if (!geo->packet) { - bt->diagnostics++; } slot = (slot + 1) & BT7932_MASK; } @@ -717,7 +724,7 @@ static int bt7932_download(struct bt7932 *bt) } writel(0x188d1807, bt->bar + BT7932_CONNECTION_SELECTOR); writel(1, bt->bar + BT7932_SEMAPHORE_RELEASE); - dev_info(&bt->pdev->dev, "BT_IMAGE_ACCEPTED response1 boot0->2; image remains retained\n"); + dev_dbg(&bt->pdev->dev, "BT_IMAGE_ACCEPTED response1 boot0->2; image remains retained\n"); return 0; } @@ -731,7 +738,7 @@ static int bt7932_ipc_state(struct bt7932 *bt, u8 *state) value = readl(bt->bar + BT7932_STATE); if (value == U32_MAX) return -EIO; - /* The reviewed early-state rule replaces memory with the BAR byte. */ + /* Until the state in memory passes 1, the BAR register holds it. */ *state = value & 0xff; } return 0; @@ -814,11 +821,11 @@ static void bt7932_channels(struct bt7932 *bt) bt7932_publish_counter(bt, bt->host_transfer, channel, BT7932_DEPTH - 1); bt7932_ring_doorbell(bt, geometry[channel].doorbell); } - /* The reviewed cold IPC transition starts ACTIVE; no initial sleep ACK. */ + /* The IPC starts active after a cold start; no sleep ACK is due. */ bt->last_sleep = 0; WRITE_ONCE(bt->awake, true); WRITE_ONCE(bt->ipc_ready, true); - dev_info(&bt->pdev->dev, "BT_IPC_READY ten retained advertisements;127 RX offers per channel\n"); + dev_dbg(&bt->pdev->dev, "BT_IPC_READY ten retained advertisements;127 RX offers per channel\n"); } static int bt7932_open(struct hci_dev *hdev) @@ -832,6 +839,9 @@ static int bt7932_open(struct hci_dev *hdev) ret = -EHOSTDOWN; else WRITE_ONCE(bt->opened, true); + /* Closing the device drops an injected event that was still queued. */ + if (bt->recovery == BT7932_RECOVERY_REQUESTED) + bt->recovery = BT7932_RECOVERY_NONE; mutex_unlock(&bt->lock); return ret; } @@ -938,29 +948,72 @@ static int bt7932_setup(struct hci_dev *hdev) HCI_INIT_TIMEOUT); if (ret) goto fault; - dev_info(&bt->pdev->dev, "BT_HCI_SETUP_OK Reset/calibration/PTX/address completed in order\n"); + dev_dbg(&bt->pdev->dev, "BT_HCI_SETUP_OK Reset/calibration/PTX/address completed in order\n"); + mutex_lock(&bt->lock); + if (bt->recovery == BT7932_RECOVERY_REOPENING) + dev_info(&bt->pdev->dev, "controller recovered\n"); + bt->recovery = BT7932_RECOVERY_NONE; + mutex_unlock(&bt->lock); return 0; fault: dev_err(&bt->pdev->dev, "BT_HCI_SETUP_FAILED stage=%s opcode=%04x status=%d\n", stage, opcode, ret); + /* + * The transport is intact, so a requested recovery or a later power-on + * may try again. A controller that fails the setup of its recovery, + * though, does not come back without a reset of the PCI function. + */ mutex_lock(&bt->lock); - bt7932_fault_locked(bt, "HCI setup command failed; no retry"); + if (bt->recovery == BT7932_RECOVERY_REOPENING) + bt7932_fault_locked(bt, "controller recovery failed"); mutex_unlock(&bt->lock); return ret; } +/* + * Called when a command times out, and through the "reset" sysfs attribute. + * While the transport rings are consistent, recover the controller the + * standard way: inject a Hardware Error event, so that the core closes the + * device, resetting the controller, and opens it again, repeating setup. + * Allow one attempt until a setup succeeds, so that a controller that no + * longer answers ends in a fault instead of a reset loop. + */ static void bt7932_reset(struct hci_dev *hdev) { struct bt7932 *bt = hci_get_drvdata(hdev); + bool recover; mutex_lock(&bt->lock); - bt7932_fault_locked(bt, "HCI requested recovery; cold reset required"); + recover = !bt->fault && !bt->stopping && + bt->recovery == BT7932_RECOVERY_NONE && + bt->pm_state == BT7932_RUNNING; + if (recover) + bt->recovery = BT7932_RECOVERY_REQUESTED; mutex_unlock(&bt->lock); + if (!recover) + return; + dev_warn(&bt->pdev->dev, "resetting the controller\n"); + if (hci_reset_dev(hdev)) { + mutex_lock(&bt->lock); + if (bt->recovery == BT7932_RECOVERY_REQUESTED) + bt->recovery = BT7932_RECOVERY_NONE; + mutex_unlock(&bt->lock); + } } +/* + * The core closes and reopens the device after this, for a Hardware Error + * event from the controller as for one injected above. That reopening is + * the recovery attempt. + */ static void bt7932_hw_error(struct hci_dev *hdev, u8 code) { - bt7932_reset(hdev); + struct bt7932 *bt = hci_get_drvdata(hdev); + + mutex_lock(&bt->lock); + bt->recovery = BT7932_RECOVERY_REOPENING; + mutex_unlock(&bt->lock); + bt_dev_err(hdev, "hardware error 0x%2.2x", code); } static int bt7932_iommu(struct bt7932 *bt) @@ -1003,7 +1056,7 @@ static int bt7932_iommu(struct bt7932 *bt) if (ret) dev_err(dev, "BT_IOMMU_REJECT 32-bit DMA mask status=%d\n", ret); else - dev_info(dev, "BT_IOMMU_ADMITTED RID101->SID2 matched translated DART\n"); + dev_dbg(dev, "BT_IOMMU_ADMITTED RID101->SID2 matched translated DART\n"); } of_node_put(target); return ret; @@ -1037,8 +1090,8 @@ static void bt7932_cleanup(struct bt7932 *bt, bool release_dma) } /* - * Bluetooth starts after the Wi-Fi function has initialized its firmware. - * Wait until mt7932-fullmac is bound to function 0, then link the + * Admission was only qualified with the Wi-Fi function already running its + * firmware. Wait until mt7932-fullmac is bound to function 0, then link the * two functions so that the driver core unbinds this one first. */ static int bt7932_wait_for_wifi(struct pci_dev *pdev) @@ -1076,10 +1129,13 @@ static int bt7932_probe(struct pci_dev *pdev, const struct pci_device_id *id) !of_machine_is_compatible("apple,t8140") || pci_domain_nr(pdev->bus) || pdev->bus->number != 1 || pdev->devfn != PCI_DEVFN(0, 1)) return -ENODEV; + ret = apple_piodma_radio_check(pdev); + if (ret) + return ret; ret = bt7932_wait_for_wifi(pdev); if (ret) return ret; - /* The radio transport requires ASPM and clock PM off on this link. */ + /* Admission was qualified with ASPM and clock PM off on this link. */ ret = pci_disable_link_state(pdev, PCIE_LINK_STATE_ALL); if (ret) dev_warn(&pdev->dev, "link power management left as found: %d\n", ret); @@ -1120,7 +1176,7 @@ static int bt7932_probe(struct pci_dev *pdev, const struct pci_device_id *id) ret = -ENOMEM; goto fail; } - dev_info(&pdev->dev, "BT_BAR0 resource=%pR\n", &pdev->resource[0]); + dev_dbg(&pdev->dev, "BT_BAR0 resource=%pR\n", &pdev->resource[0]); ret = pci_read_config_dword(pdev, BT7932_BOOT_CONFIG, &boot); if (ret || boot == U32_MAX || (boot & 0xf) || ((boot >> 4) & 7) > 1) { dev_err(&pdev->dev, "BT_IDENTITY_REJECT config_status=%d boot=%#x\n", ret, boot); @@ -1137,7 +1193,7 @@ static int bt7932_probe(struct pci_dev *pdev, const struct pci_device_id *id) ret = -ENODEV; goto fail; } - dev_info(&pdev->dev, "BT_COLD_ADMISSION ROM=%u bootstate0 chip7932 SID2\n", bt->rom); + dev_dbg(&pdev->dev, "BT_COLD_ADMISSION ROM=%u bootstate0 chip7932 SID2\n", bt->rom); ret = bt7932_read_inputs(bt); if (ret) goto fail; @@ -1198,7 +1254,7 @@ static int bt7932_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (ret < 0) goto retained_fault; bt->registered = true; - dev_info(&pdev->dev, "BT_HCI_REGISTERED; setup/discovery/connection remain separate gates\n"); + dev_dbg(&pdev->dev, "BT_HCI_REGISTERED; setup/discovery/connection remain separate gates\n"); return 0; retained_fault: mutex_lock(&bt->lock); @@ -1421,7 +1477,7 @@ static struct pci_driver bt7932_driver = { }; module_pci_driver(bt7932_driver); -MODULE_DESCRIPTION("Experimental standalone J700 MT7932 Bluetooth PCIe transport"); +MODULE_DESCRIPTION("MediaTek MT7932 Bluetooth PCIe transport for the Apple MacBook Neo"); MODULE_LICENSE("GPL"); MODULE_FIRMWARE(BT7932_FW_B0); MODULE_FIRMWARE(BT7932_FW_B1); @@ -1429,3 +1485,7 @@ MODULE_FIRMWARE(BT7932_FW_B1_FALLBACK); MODULE_FIRMWARE(BT7932_PTX); MODULE_FIRMWARE(BT7932_CAL); MODULE_FIRMWARE(BT7932_ADDR); + +#if IS_ENABLED(CONFIG_BT_MTK7932_PCIE_KUNIT_TEST) +#include "mt7932_bt_pcie_test.c" +#endif diff --git a/drivers/bluetooth/mt7932_bt_pcie_test.c b/drivers/bluetooth/mt7932_bt_pcie_test.c new file mode 100644 index 00000000000000..748dcabde731cc --- /dev/null +++ b/drivers/bluetooth/mt7932_bt_pcie_test.c @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * KUnit tests for the checks mt7932_bt_pcie applies to HCI packets and to the + * board file before either reaches the controller or the HCI core. Built as + * part of mt7932_bt_pcie.c, so the tests reach its static helpers. + */ +#include + +static void bt7932_hci_length_command_test(struct kunit *test) +{ + /* HCI_Reset (0x0c03) and a command carrying two parameter bytes. */ + static const u8 reset[] = { 0x03, 0x0c, 0x00 }; + static const u8 command[] = { 0x1a, 0xfc, 0x02, 0x11, 0x22, 0x33 }; + + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_COMMAND_PKT, reset, sizeof(reset))); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_COMMAND_PKT, reset, 2)); + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_COMMAND_PKT, command, 5)); + /* One byte short of, and one byte past, the declared length. */ + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_COMMAND_PKT, command, 4)); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_COMMAND_PKT, command, 6)); +} + +static void bt7932_hci_length_event_test(struct kunit *test) +{ + /* Command Complete for HCI_Reset with status 0, then one spare byte. */ + static const u8 event[] = { 0x0e, 0x04, 0x01, 0x03, 0x0c, 0x00, 0xff }; + + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_EVENT_PKT, event, 6)); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_EVENT_PKT, event, 5)); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_EVENT_PKT, event, 7)); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_EVENT_PKT, event, 1)); +} + +static void bt7932_hci_length_data_test(struct kunit *test) +{ + /* ACL and ISO lengths are 16-bit; ISO keeps two flag bits on top. */ + static const u8 acl[] = { 0x01, 0x20, 0x02, 0x00, 0xaa, 0xbb }; + static const u8 sco[] = { 0x01, 0x00, 0x01, 0xcc }; + static const u8 iso[] = { 0x01, 0x00, 0x01, 0xc0, 0xdd }; + static const u8 long_acl[] = { 0x01, 0x20, 0x00, 0x01 }; + + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_ACLDATA_PKT, acl, sizeof(acl))); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_ACLDATA_PKT, acl, 5)); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_ACLDATA_PKT, acl, 3)); + /* A 256-byte ACL payload must not be satisfied by its header alone. */ + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_ACLDATA_PKT, long_acl, sizeof(long_acl))); + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_SCODATA_PKT, sco, sizeof(sco))); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_SCODATA_PKT, sco, 3)); + KUNIT_EXPECT_TRUE(test, bt7932_hci_length(HCI_ISODATA_PKT, iso, sizeof(iso))); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_ISODATA_PKT, iso, 4)); +} + +static void bt7932_hci_length_type_test(struct kunit *test) +{ + static const u8 packet[] = { 0x00, 0x00, 0x00, 0x00 }; + + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(HCI_VENDOR_PKT, packet, sizeof(packet))); + KUNIT_EXPECT_FALSE(test, bt7932_hci_length(0, packet, sizeof(packet))); +} + +#define BT7932_TEST_PTX_SIZE 198 + +/* A well-formed container with synthetic record contents. */ +static void bt7932_test_ptx(u8 *data) +{ + static const u32 types[] = { 0x101, 0x201, 0x301, 0x401 }; + static const u32 sizes[] = { 14, 44, 44, 0 }; + u32 offset = 96, sum, i, j; + + memset(data, 0, BT7932_TEST_PTX_SIZE); + memcpy(data, "BLOB", 4); + put_unaligned_le32(96, data + 4); + put_unaligned_le16(1, data + 8); + put_unaligned_le16(ARRAY_SIZE(types), data + 10); + for (i = 0; i < ARRAY_SIZE(types); i++) { + u8 *record = data + 16 + i * 20; + + for (j = 0, sum = 0; j < sizes[i]; j++) { + data[offset + j] = 0x40 + i * 16 + j; + sum += data[offset + j]; + } + put_unaligned_le32(types[i], record); + put_unaligned_le32(offset, record + 4); + put_unaligned_le32(sizes[i], record + 8); + put_unaligned_le32(sum, record + 12); + offset += sizes[i]; + } +} + +static void bt7932_validate_ptx_test(struct kunit *test) +{ + u8 *data = kunit_kzalloc(test, BT7932_TEST_PTX_SIZE + 1, GFP_KERNEL); + + KUNIT_ASSERT_NOT_NULL(test, data); + bt7932_test_ptx(data); + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE), 0); + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE - 1), -EINVAL); + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE + 1), -EINVAL); + + /* A payload byte that no longer matches its record's checksum. */ + data[120]++; + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE), -EINVAL); + + bt7932_test_ptx(data); + data[0] = 'b'; + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE), -EINVAL); + + /* A record of the wrong type in the first slot. */ + bt7932_test_ptx(data); + put_unaligned_le32(0x201, data + 16); + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE), -EINVAL); + + /* A record length other than the one its type has. */ + bt7932_test_ptx(data); + put_unaligned_le32(45, data + 16 + 2 * 20 + 8); + KUNIT_EXPECT_EQ(test, bt7932_validate_ptx(data, BT7932_TEST_PTX_SIZE), -EINVAL); +} + +static struct kunit_case bt7932_test_cases[] = { + KUNIT_CASE(bt7932_hci_length_command_test), + KUNIT_CASE(bt7932_hci_length_event_test), + KUNIT_CASE(bt7932_hci_length_data_test), + KUNIT_CASE(bt7932_hci_length_type_test), + KUNIT_CASE(bt7932_validate_ptx_test), + {} +}; + +static struct kunit_suite bt7932_test_suite = { + .name = "mt7932-bt-pcie", + .test_cases = bt7932_test_cases, +}; + +kunit_test_suite(bt7932_test_suite); diff --git a/drivers/cpufreq/apple-soc-cpufreq.c b/drivers/cpufreq/apple-soc-cpufreq.c index 5dc0eef7531e32..86273ba1d64d80 100644 --- a/drivers/cpufreq/apple-soc-cpufreq.c +++ b/drivers/cpufreq/apple-soc-cpufreq.c @@ -13,7 +13,7 @@ #include #include #include -#include +#include #include #include #include @@ -525,7 +525,7 @@ static void apple_fast_die_sample(struct apple_fast_die *fd) u8 effort; int ret; - ret = apple_neo_pmp_report_fast_die_effort(fd->report, fd->lane, &effort); + ret = apple_pmp_report_fast_die_effort(fd->report, fd->lane, &effort); if (!ret) { limit = apple_fast_die_limit(fd, effort); fd->waiting = false; diff --git a/drivers/cpuidle/cpuidle-apple.c b/drivers/cpuidle/cpuidle-apple.c index 6c02bd4197c263..64b45dcbafb31a 100644 --- a/drivers/cpuidle/cpuidle-apple.c +++ b/drivers/cpuidle/cpuidle-apple.c @@ -127,10 +127,12 @@ static int apple_cpuidle_probe(struct platform_device *pdev) /* * CPU PD writes CYC_OVRD, which is an undefined instruction on * T8140. Its WFI already lets the hardware power-gate idle - * secondary cores, so expose only the WFI state. + * secondary cores, so both entries must use WFI. Keep index 1: + * the generic s2idle selector skips fallback index 0, otherwise + * tick_freeze() and timekeeping suspension are never reached. */ if (of_machine_is_compatible("apple,t8140")) - apple_idle_driver.state_count = STATE_WFI + 1; + apple_idle_driver.states[STATE_PWRDOWN] = apple_idle_driver.states[STATE_WFI]; return cpuidle_register(&apple_idle_driver, NULL); } diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index 5c5dc8672fb814..76082bc71f7eba 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -225,9 +225,23 @@ static struct file_system_type dma_buf_fs_type = { .kill_sb = kill_anon_super, }; +static int dma_buf_mmap_range_check(struct dma_buf *dmabuf, + struct vm_area_struct *vma, unsigned long pgoff, unsigned int suboffset) +{ + unsigned long offset; + + if (check_mul_overflow(pgoff, PAGE_SIZE, &offset) || + check_add_overflow(offset, (unsigned long)suboffset, &offset)) + return -EOVERFLOW; + if (offset > dmabuf->size || vma->vm_end - vma->vm_start > dmabuf->size - offset) + return -EINVAL; + return 0; +} + static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma) { struct dma_buf *dmabuf; + int ret; if (!is_dma_buf_file(file)) return -EINVAL; @@ -239,9 +253,9 @@ static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma) return -EINVAL; /* check for overflowing the buffer's size */ - if (vma->vm_pgoff + vma_pages(vma) > - dmabuf->size >> PAGE_SHIFT) - return -EINVAL; + ret = dma_buf_mmap_range_check(dmabuf, vma, vma->vm_pgoff, vma_subpage_offset(vma)); + if (ret) + return ret; DMA_BUF_TRACE(trace_dma_buf_mmap_internal, dmabuf); @@ -1602,6 +1616,8 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, "DMA_BUF"); int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma, unsigned long pgoff) { + int ret; + if (WARN_ON(!dmabuf || !vma)) return -EINVAL; @@ -1609,18 +1625,13 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma, if (!dmabuf->ops->mmap) return -EINVAL; - /* check for offset overflow */ - if (pgoff + vma_pages(vma) < pgoff) - return -EOVERFLOW; - - /* check for overflowing the buffer's size */ - if (pgoff + vma_pages(vma) > - dmabuf->size >> PAGE_SHIFT) - return -EINVAL; + ret = dma_buf_mmap_range_check(dmabuf, vma, pgoff, 0); + if (ret) + return ret; /* readjust the vma */ vma_set_file(vma, dmabuf->file); - vma->vm_pgoff = pgoff; + vma_set_page_offset(vma, (struct vm_page_offset) { .index = pgoff }); DMA_BUF_TRACE(trace_dma_buf_mmap, dmabuf); diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index a359aac4557990..37ec7bc84a24ee 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -174,7 +174,10 @@ static vm_fault_t cma_heap_vm_fault(struct vm_fault *vmf) if (vmf->pgoff >= buffer->pagecount) return VM_FAULT_SIGBUS; - return vmf_insert_pfn(vma, vmf->address, page_to_pfn(buffer->pages[vmf->pgoff])); + return vmf_insert_pfn_prot_mkwrite_offset(vma, vmf->address, + page_to_pfn(buffer->pages[vmf->pgoff]), + vma_page_offset_at(vma, vmf->address).offset, + vma->vm_page_prot, false); } static const struct vm_operations_struct dma_heap_vm_ops = { diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 03c2b87cb11126..a1797ae903004e 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -228,6 +228,7 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) struct sg_table *table = &buffer->sg_table; unsigned long addr = vma->vm_start; unsigned long pgoff = vma->vm_pgoff; + unsigned int offset = vma_subpage_offset(vma); struct scatterlist *sg; pgprot_t prot; int i, ret; @@ -247,17 +248,18 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) for (; sg && addr < vma->vm_end; sg = sg_next(sg)) { unsigned long n = (sg->length >> PAGE_SHIFT) - pgoff; struct page *page = sg_page(sg) + pgoff; - unsigned long size = n << PAGE_SHIFT; + unsigned long size = (n << PAGE_SHIFT) - offset; - if (addr + size > vma->vm_end) - size = vma->vm_end - addr; + size = min(size, vma->vm_end - addr); - ret = remap_pfn_range(vma, addr, page_to_pfn(page), size, prot); + ret = remap_pfn_range_offset(vma, addr, page_to_pfn(page), offset, + size, prot); if (ret) return ret; addr += size; pgoff = 0; + offset = 0; } return 0; diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c index 2e7b5d4d11f429..71efcf40d91695 100644 --- a/drivers/dma-buf/udmabuf.c +++ b/drivers/dma-buf/udmabuf.c @@ -45,45 +45,45 @@ struct udmabuf { pgoff_t *offsets; }; -static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) +static vm_fault_t udmabuf_insert_pfn(struct vm_area_struct *vma, unsigned long addr) { - struct vm_area_struct *vma = vmf->vma; struct udmabuf *ubuf = vma->vm_private_data; - pgoff_t pgoff = vmf->pgoff; - unsigned long addr, pfn; - vm_fault_t ret; + struct vm_page_offset pos = vma_page_offset_at(vma, addr); + unsigned long pfn; - if (pgoff >= ubuf->pagecount) + if (pos.index >= ubuf->pagecount) return VM_FAULT_SIGBUS; - pfn = folio_pfn(ubuf->folios[pgoff]); - pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT; + pfn = folio_pfn(ubuf->folios[pos.index]); + pfn += ubuf->offsets[pos.index] >> PAGE_SHIFT; + return vmf_insert_pfn_prot_mkwrite_offset(vma, addr, pfn, pos.offset, + vma->vm_page_prot, false); +} + +static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + unsigned long addr; + vm_fault_t ret; - ret = vmf_insert_pfn(vma, vmf->address, pfn); + ret = udmabuf_insert_pfn(vma, vmf->address); if (ret & VM_FAULT_ERROR) return ret; /* pre fault */ - pgoff = vma->vm_pgoff; addr = vma->vm_start; - for (; addr < vma->vm_end; pgoff++, addr += PAGE_SIZE) { + for (; addr < vma->vm_end; addr += mm_page_size(vma->vm_mm)) { if (addr == vmf->address) continue; - if (WARN_ON(pgoff >= ubuf->pagecount)) - break; - - pfn = folio_pfn(ubuf->folios[pgoff]); - pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT; - /** - * If the below vmf_insert_pfn() fails, we do not return an + * If the below insertion fails, we do not return an * error here during this pre-fault step. However, an error * will be returned if the failure occurs when the addr is * truly accessed. */ - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) + if (udmabuf_insert_pfn(vma, addr) & VM_FAULT_ERROR) break; } diff --git a/drivers/dma/apple-admac.c b/drivers/dma/apple-admac.c index a1aeaf3e4dc8f5..dfaeeba9d8e83e 100644 --- a/drivers/dma/apple-admac.c +++ b/drivers/dma/apple-admac.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -42,6 +43,10 @@ #define T603X_UNK_28_VAL 0x200000 +/* + * One bit per channel: a write to START sets the channel's enable bit, a write + * to STOP clears it, and START reads back the channels that are enabled. + */ #define REG_TX_START 0x0000 #define REG_TX_STOP 0x0004 #define REG_RX_START 0x0008 @@ -474,6 +479,35 @@ static void admac_stop_chan(struct admac_chan *adchan) dev_dbg(adchan->host->dev, "ch%d stop\n", adchan->no); } +/* + * Generous bound for a channel to wind down after its enable bit was cleared: + * it only has to finish the burst in flight, a few frames at audio rates. + */ +#define ADMAC_STOP_TIMEOUT_US 100000 + +static void admac_wait_stopped(struct admac_chan *adchan) +{ + struct admac_data *ad = adchan->host; + u32 bit = BIT(adchan->no / 2); + void __iomem *reg; + u32 val; + + switch (admac_chan_direction(adchan->no)) { + case DMA_MEM_TO_DEV: + reg = ad->base + REG_TX_START; + break; + case DMA_DEV_TO_MEM: + reg = ad->base + REG_RX_START; + break; + default: + return; + } + + if (readl_poll_timeout(reg, val, !(val & bit), 10, ADMAC_STOP_TIMEOUT_US)) + dev_warn_ratelimited(ad->dev, "ch%d still enabled %d us after stop\n", + adchan->no, ADMAC_STOP_TIMEOUT_US); +} + static void admac_reset_rings(struct admac_chan *adchan) { struct admac_data *ad = adchan->host; @@ -564,6 +598,13 @@ static void admac_synchronize(struct dma_chan *chan) unsigned long flags; LIST_HEAD(head); + /* + * terminate_all() only requests the stop: make sure the channel no + * longer touches the memory of the terminated descriptors before the + * client is allowed to reuse or free it. + */ + admac_wait_stopped(adchan); + spin_lock_irqsave(&adchan->lock, flags); list_splice_tail_init(&adchan->to_free, &head); spin_unlock_irqrestore(&adchan->lock, flags); diff --git a/drivers/gpu/drm/Kconfig b/drivers/gpu/drm/Kconfig index 781765bbdcab3f..325223d75e5c80 100644 --- a/drivers/gpu/drm/Kconfig +++ b/drivers/gpu/drm/Kconfig @@ -368,6 +368,4 @@ source "drivers/gpu/drm/apple/Kconfig.tests" config DRM_PANEL_ORIENTATION_QUIRKS tristate -source "drivers/gpu/drm/asahi-neo/Kconfig" -source "drivers/gpu/drm/apple-neo/Kconfig" diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile index d2c2eef1a276b3..846f2da7c81a08 100644 --- a/drivers/gpu/drm/Makefile +++ b/drivers/gpu/drm/Makefile @@ -263,6 +263,4 @@ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) $(obj)/%.hdrtest: $(src)/%.h FORCE $(call if_changed_dep,hdrtest) -obj-$(CONFIG_DRM_ASAHI_NEO) += asahi-neo/ -obj-$(CONFIG_DRM_APPLE_NEO) += apple-neo/ diff --git a/drivers/gpu/drm/apple-neo/Kconfig b/drivers/gpu/drm/apple-neo/Kconfig deleted file mode 100644 index e24927ca4648c4..00000000000000 --- a/drivers/gpu/drm/apple-neo/Kconfig +++ /dev/null @@ -1,43 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only OR MIT -config DRM_APPLE_NEO - tristate "DRM Support for the Apple T8140 display controller" - depends on DRM && OF && ARM64 - depends on ARCH_APPLE || COMPILE_TEST - depends on APPLE_RTKIT - depends on TYPEC - depends on OF_ADDRESS - select APPLE_PMP_REPORT - select DRM_CLIENT_SELECTION - select DRM_KMS_HELPER - select DRM_KMS_DMA_HELPER - select DRM_GEM_DMA_HELPER - select VIDEOMODE_HELPERS - depends on BACKLIGHT_CLASS_DEVICE - select MULTIPLEXER - help - Say Y for native display support on the MacBook Neo. - -config DRM_APPLE_NEO_AUDIO - bool "DisplayPort/HDMI Audio support" - default y - depends on DRM_APPLE_NEO - depends on SND - select SND_PCM - select SND_DMAENGINE_PCM - -config DRM_APPLE_NEO_DEBUG - bool "Enable additional driver debugging" - depends on DRM_APPLE_NEO - depends on EXPERT # only for developers - -config DRM_APPLE_NEO_KUNIT_TEST - tristate "KUnit tests for Apple display controller" if !KUNIT_ALL_TESTS - depends on KUNIT - default KUNIT_ALL_TESTS - help - Build tests for the Apple display driver's AFK ring parser and - backlight policy (takeover, DPMS, suspend and present completion). - The tests cover the header strides used by older firmware and - larger rings, as well as malformed ring size combinations. - This option is intended for developer test kernels. - Also test the Type-C modeset, hotplug and service ownership state. diff --git a/drivers/gpu/drm/apple-neo/Makefile b/drivers/gpu/drm/apple-neo/Makefile deleted file mode 100644 index 89e62139560007..00000000000000 --- a/drivers/gpu/drm/apple-neo/Makefile +++ /dev/null @@ -1,23 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only OR MIT - -CFLAGS_trace.o = -I$(src) - -appledrm_neo-y := apple_drv.o -appledrm_neo-y += afk.o dcp.o dcp_backlight.o dptxep.o iomfb.o parser.o systemep.o -appledrm_neo-$(CONFIG_DRM_APPLE_NEO_AUDIO) += audio.o -appledrm_neo-$(CONFIG_DRM_APPLE_NEO_AUDIO) += av.o -appledrm_neo-y += connector.o -appledrm_neo-y += ibootep.o -appledrm_neo-y += iomfb_v12_3.o -appledrm_neo-y += iomfb_v13_3.o -appledrm_neo-y += iomfb_h17p.o -appledrm_neo-$(CONFIG_DRM_APPLE_NEO_KUNIT_TEST) += iomfb_h17p_test.o -appledrm_neo-y += epic/dpavservep.o -appledrm_neo-y += plane.o - - -appledrm_neo-$(CONFIG_TRACING) += trace.o - -obj-$(CONFIG_DRM_APPLE_NEO) += appledrm_neo.o -obj-$(CONFIG_DRM_APPLE_NEO_KUNIT_TEST) += afk_test.o -obj-$(CONFIG_DRM_APPLE_NEO_KUNIT_TEST) += dcp_backlight_test.o diff --git a/drivers/gpu/drm/apple-neo/afk-service-test.c b/drivers/gpu/drm/apple-neo/afk-service-test.c deleted file mode 100644 index 65345b991d2e4f..00000000000000 --- a/drivers/gpu/drm/apple-neo/afk-service-test.c +++ /dev/null @@ -1,142 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#include -#include - -#include "afk.h" - -static const struct neo_apple_epic_service_ops reusable_ops = { - .name = "reusable", - .reusable = true, -}; - -static const struct neo_apple_epic_service_ops permanent_ops = { - .name = "permanent", -}; - -static int neo_afk_service_test_init(struct kunit *test) -{ - struct neo_apple_epic_service *service; - - service = kunit_kzalloc(test, sizeof(*service), GFP_KERNEL); - if (!service) - return -ENOMEM; - spin_lock_init(&service->lock); - test->priv = service; - return 0; -} - -static void neo_afk_test_retire(struct neo_apple_epic_service *service) -{ - unsigned long flags; - - spin_lock_irqsave(&service->lock, flags); - service->enabled = false; - service->torndown = true; - spin_unlock_irqrestore(&service->lock, flags); -} - -static void neo_afk_service_many_generations_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - unsigned int i; - - /* Repeated unplug/replug must not consume an unbounded slot budget. */ - for (i = 0; i < 4 * AFK_MAX_CHANNEL; i++) { - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, i)); - KUNIT_EXPECT_EQ(test, service->channel, i); - KUNIT_EXPECT_TRUE(test, neo_afk_service_matches(service, i)); - neo_afk_test_retire(service); - } -} - -static void neo_afk_service_reader_pin_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 1)); - KUNIT_ASSERT_PTR_EQ(test, neo_afk_service_get(service), service); /* owner */ - KUNIT_ASSERT_PTR_EQ(test, neo_afk_service_get(service), service); /* reader */ - neo_afk_test_retire(service); - neo_afk_service_put(service); /* owner detaches while EDID is being read */ - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - KUNIT_EXPECT_EQ(test, service->channel, 1U); - KUNIT_EXPECT_PTR_EQ(test, neo_afk_service_get(service), NULL); - neo_afk_service_put(service); - KUNIT_EXPECT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); -} - -static void neo_afk_service_pending_reply_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 1)); - set_bit(3, service->cmd_map); - neo_afk_test_retire(service); - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - KUNIT_EXPECT_TRUE(test, neo_afk_service_matches(service, 1)); - KUNIT_EXPECT_FALSE(test, neo_afk_service_matches(service, 2)); - /* A timed-out command retains its DMA and slot until the late reply. */ - service->cmds[3].free_on_ack = true; - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - clear_bit(3, service->cmd_map); - KUNIT_EXPECT_FALSE(test, neo_afk_service_matches(service, 1)); - KUNIT_EXPECT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); -} - -static void neo_afk_service_completed_reader_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 1)); - KUNIT_ASSERT_PTR_EQ(test, neo_afk_service_get(service), service); - set_bit(0, service->cmd_map); - neo_afk_test_retire(service); - /* Reply arrived, but the waiting thread has not consumed it yet. */ - service->cmds[0].done = true; - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - clear_bit(0, service->cmd_map); - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - neo_afk_service_put(service); - KUNIT_EXPECT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); -} - -static void neo_afk_service_tag_sequence_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 1)); - service->cmd_tag = 0xa9; - service->cookie = test; - neo_afk_test_retire(service); - KUNIT_EXPECT_TRUE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); - KUNIT_EXPECT_EQ(test, service->cmd_tag, (u8)0xa9); - KUNIT_EXPECT_PTR_EQ(test, service->cookie, NULL); -} - -static void neo_afk_service_unconverted_owner_test(struct kunit *test) -{ - struct neo_apple_epic_service *service = test->priv; - - KUNIT_ASSERT_TRUE(test, neo_afk_service_reinit(service, NULL, &permanent_ops, 1)); - neo_afk_test_retire(service); - KUNIT_EXPECT_FALSE(test, neo_afk_service_reinit(service, NULL, &reusable_ops, 2)); -} - -static struct kunit_case neo_afk_service_cases[] = { - KUNIT_CASE(neo_afk_service_many_generations_test), - KUNIT_CASE(neo_afk_service_reader_pin_test), - KUNIT_CASE(neo_afk_service_pending_reply_test), - KUNIT_CASE(neo_afk_service_completed_reader_test), - KUNIT_CASE(neo_afk_service_tag_sequence_test), - KUNIT_CASE(neo_afk_service_unconverted_owner_test), - { } -}; - -static struct kunit_suite neo_afk_service_suite = { - .name = "apple-afk-service-lifetime", - .init = neo_afk_service_test_init, - .test_cases = neo_afk_service_cases, -}; - -kunit_test_suite(neo_afk_service_suite); -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple-neo/afk.c b/drivers/gpu/drm/apple-neo/afk.c deleted file mode 100644 index 935fcbf483b1a8..00000000000000 --- a/drivers/gpu/drm/apple-neo/afk.c +++ /dev/null @@ -1,1398 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2022 Sven Peter */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "afk.h" -#include "trace.h" - -struct neo_afk_receive_message_work { - struct neo_apple_dcp_afkep *ep; - u64 message; - struct work_struct work; -}; - -#define RBEP_TYPE GENMASK(63, 48) - -enum rbep_msg_type { - RBEP_INIT = 0x80, - RBEP_INIT_ACK = 0xa0, - RBEP_GETBUF = 0x89, - RBEP_GETBUF_ACK = 0xa1, - RBEP_INIT_TX = 0x8a, - RBEP_INIT_RX = 0x8b, - RBEP_START = 0xa3, - RBEP_START_ACK = 0x86, - RBEP_SEND = 0xa2, - RBEP_RECV = 0x85, - RBEP_SHUTDOWN = 0xc0, - RBEP_SHUTDOWN_ACK = 0xc1, -}; - -#define BLOCK_SHIFT 6 - -#define GETBUF_SIZE GENMASK(31, 16) -#define GETBUF_TAG GENMASK(15, 0) -#define GETBUF_ACK_DVA GENMASK(47, 0) - -#define INITRB_OFFSET GENMASK(47, 32) -#define INITRB_SIZE GENMASK(31, 16) -#define INITRB_TAG GENMASK(15, 0) - -#define SEND_WPTR GENMASK(31, 0) - -static void neo_afk_send(struct neo_apple_dcp_afkep *ep, u64 message) -{ - neo_dcp_send_message(ep->neo_dcp, ep->endpoint, message); -} - -static void neo_afk_release_context(void *data) -{ - struct neo_apple_dcp_afkep *afkep = data; - - neo_afk_quiesce(afkep); - if (!afkep->neo_dcp->retain_dma) - kfree(afkep); -} - -struct neo_apple_dcp_afkep *neo_afk_init(struct neo_apple_dcp *neo_dcp, u32 endpoint, - const struct neo_apple_epic_service_ops *ops) -{ - struct neo_apple_dcp_afkep *afkep; - int ret, i; - - afkep = kzalloc_obj(*afkep); - if (!afkep) - return ERR_PTR(-ENOMEM); - - afkep->ops = ops; - afkep->neo_dcp = neo_dcp; - afkep->endpoint = endpoint; - afkep->wq = alloc_ordered_workqueue("apple-dcp-afkep%02x", - WQ_MEM_RECLAIM, endpoint); - if (!afkep->wq) { - ret = -ENOMEM; - goto out_free_afkep; - } - - init_completion(&afkep->started); - init_completion(&afkep->stopped); - spin_lock_init(&afkep->lock); - for (i = 0; i < AFK_MAX_CHANNEL; i++) - spin_lock_init(&afkep->services[i].lock); - ret = devm_add_action_or_reset(neo_dcp->dev, neo_afk_release_context, afkep); - if (ret) - return ERR_PTR(ret); - - return afkep; - -out_free_afkep: - kfree(afkep); - return ERR_PTR(ret); -} - -void neo_afk_quiesce(struct neo_apple_dcp_afkep *afkep) -{ - if (IS_ERR_OR_NULL(afkep)) - return; - if (afkep->wq) { - destroy_workqueue(afkep->wq); - afkep->wq = NULL; - } - debugfs_remove_recursive(afkep->debugfs_entry); - afkep->debugfs_entry = NULL; -} - -void neo_afk_shutdown(struct neo_apple_dcp_afkep *afkep) -{ - unsigned long flags; - - neo_afk_send(afkep, FIELD_PREP(RBEP_TYPE, RBEP_SHUTDOWN)); - int ret; - - ret = wait_for_completion_timeout(&afkep->stopped, msecs_to_jiffies(1000)); - if (ret <= 0) { - dev_err(afkep->neo_dcp->dev, "Timed out shutting down AFK endpoint %02x", afkep->endpoint); - } - - spin_lock_irqsave(&afkep->lock, flags); - afkep->stopping = true; - spin_unlock_irqrestore(&afkep->lock, flags); - neo_afk_quiesce(afkep); - if (afkep->rx_scratch) { - devm_kfree(afkep->neo_dcp->dev, afkep->rx_scratch); - afkep->rx_scratch = NULL; - } -} - -int neo_afk_start(struct neo_apple_dcp_afkep *ep) -{ - int ret; - - reinit_completion(&ep->started); - ret = apple_rtkit_start_ep(ep->neo_dcp->rtk, ep->endpoint); - if (ret) - return ret; - neo_afk_send(ep, FIELD_PREP(RBEP_TYPE, RBEP_INIT)); - - ret = wait_for_completion_timeout(&ep->started, msecs_to_jiffies(1000)); - if (ret <= 0) - return -ETIMEDOUT; - else - return 0; -} - -static void neo_afk_getbuf(struct neo_apple_dcp_afkep *ep, u64 message) -{ - u32 size = FIELD_GET(GETBUF_SIZE, message) << BLOCK_SHIFT; - u16 tag = FIELD_GET(GETBUF_TAG, message); - u64 reply; - - trace_neo_afk_getbuf(ep, size, tag); - - if (ep->bfr) { - dev_err(ep->neo_dcp->dev, - "Got GETBUF message but buffer already exists\n"); - return; - } - - /* An unstopped adopted session owns this ring until the machine resets. */ - if (READ_ONCE(ep->neo_dcp->retain_dma)) - ep->bfr = dma_alloc_coherent(ep->neo_dcp->dev, size, &ep->bfr_dma, GFP_KERNEL); - else - ep->bfr = dmam_alloc_coherent(ep->neo_dcp->dev, size, &ep->bfr_dma, GFP_KERNEL); - if (!ep->bfr) { - dev_err(ep->neo_dcp->dev, "Failed to allocate %d bytes buffer\n", - size); - return; - } - - ep->bfr_size = size; - ep->bfr_tag = tag; - - reply = FIELD_PREP(RBEP_TYPE, RBEP_GETBUF_ACK); - reply |= FIELD_PREP(GETBUF_ACK_DVA, ep->bfr_dma); - neo_afk_send(ep, reply); -} - -static void neo_afk_init_rxtx(struct neo_apple_dcp_afkep *ep, u64 message, - struct neo_afk_ringbuffer *bfr) -{ - u32 base = FIELD_GET(INITRB_OFFSET, message) << BLOCK_SHIFT; - u32 size = FIELD_GET(INITRB_SIZE, message) << BLOCK_SHIFT; - u16 tag = FIELD_GET(INITRB_TAG, message); - u32 bufsz, end, stride; - u8 *hdr; - - if (tag != ep->bfr_tag) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: expected tag 0x%x but got 0x%x\n", - ep->endpoint, ep->bfr_tag, tag); - return; - } - - if (bfr->ready) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: buffer is already initialized\n", - ep->endpoint); - return; - } - - if (!ep->bfr || base >= ep->bfr_size || size < sizeof(u32)) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: invalid ring base 0x%x or size 0x%x\n", - ep->endpoint, base, size); - return; - } - - end = base + size; - if (end > ep->bfr_size) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: requested end 0x%x > max size 0x%lx\n", - ep->endpoint, end, ep->bfr_size); - return; - } - - hdr = (u8 *)ep->bfr + base; - dma_rmb(); - bufsz = le32_to_cpu(READ_ONCE(*(__le32 *)hdr)); - if (neo_afk_ring_stride(size, bufsz, &stride)) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: invalid ring payload size 0x%x of 0x%x\n", - ep->endpoint, bufsz, size); - return; - } - if (bufsz < BIT(BLOCK_SHIFT)) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: ring cannot hold an entry\n", - ep->endpoint); - return; - } - if (bfr == &ep->rxbfr) { - /* Receive dispatch cannot recurse on the ordered endpoint queue. */ - ep->rx_scratch = devm_kmalloc(ep->neo_dcp->dev, bufsz, GFP_KERNEL); - if (!ep->rx_scratch) - return; - } - - bfr->rptr = (__le32 *)(hdr + stride); - bfr->wptr = (__le32 *)(hdr + 2 * stride); - bfr->buf = hdr + 3 * stride; - bfr->bufsz = bufsz; - bfr->stride = stride; - bfr->ready = true; - - if (ep->rxbfr.ready && ep->txbfr.ready) - neo_afk_send(ep, FIELD_PREP(RBEP_TYPE, RBEP_START)); -} - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_DEBUG) -static void neo_afk_populate_service_debugfs(struct neo_apple_epic_service *srv); -static void neo_afk_remove_service_debugfs(struct neo_apple_epic_service *srv); -#else -static void neo_afk_populate_service_debugfs(struct neo_apple_epic_service *srv) -{ -} -static void neo_afk_remove_service_debugfs(struct neo_apple_epic_service *srv) -{ -} -#endif - -static const struct neo_apple_epic_service_ops * -neo_afk_match_service(struct neo_apple_dcp_afkep *ep, const char *name) -{ - const struct neo_apple_epic_service_ops *ops; - - if (!name || !name[0]) - return NULL; - if (!ep->ops) - return NULL; - - for (ops = ep->ops; ops->name[0]; ops++) { - if (strcmp(ops->name, name)) - continue; - - return ops; - } - - return NULL; -} - -static struct neo_apple_epic_service *neo_afk_epic_find_service(struct neo_apple_dcp_afkep *ep, - u32 channel) -{ - struct neo_apple_epic_service *service; - u32 i; - - for (i = 0; i < ep->num_channels; i++) { - service = &ep->services[i]; - if (neo_afk_service_matches(service, channel)) - return service; - } - - return NULL; -} - -static void neo_afk_recv_handle_init(struct neo_apple_dcp_afkep *ep, u32 channel, - u8 *payload, size_t payload_size) -{ - char name[32]; - s64 neo_epic_unit = -1; - struct neo_apple_epic_service *service = NULL; - u32 ch_idx; - const char *service_name = name; - const char *neo_epic_name = NULL, *neo_epic_class = NULL; - const struct neo_apple_epic_service_ops *ops; - struct neo_dcp_parse_ctx ctx; - u8 *props = payload + sizeof(name); - size_t props_size = payload_size - sizeof(name); - - if (neo_afk_epic_find_service(ep, channel)) - return; - - if (payload_size < sizeof(name)) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: payload too small: %lx\n", - ep->endpoint, payload_size); - return; - } - - strscpy(name, payload, sizeof(name)); - - /* - * in DCP firmware 13.2 DCP reports interface-name as name which starts - * with "dispext%d" using -1 s ID for "dcp". In the 12.3 firmware - * EPICProviderClass was used. If the init call has props parse them and - * use EPICProviderClass to match the service. - */ - if (props_size > 36) { - int ret = neo_parse(props, props_size, &ctx); - if (ret) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: Failed to parse service init props for %s\n", - ep->endpoint, name); - return; - } - ret = neo_parse_epic_service_init(&ctx, &neo_epic_name, &neo_epic_class, &neo_epic_unit, - ep->neo_dcp->fw_compat == DCP_FIRMWARE_H17P); - if (ret) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: failed to extract init props: %d\n", - ep->endpoint, ret); - return; - } - service_name = neo_epic_class; - } else { - service_name = name; - } - - if (ep->match_epic_name) - ops = neo_afk_match_service(ep, neo_epic_name); - else - ops = neo_afk_match_service(ep, service_name); - - if (!ops) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: unable to match service %s on channel %d\n", - ep->endpoint, service_name, channel); - goto free; - } - - for (ch_idx = 0; ch_idx < AFK_MAX_CHANNEL; ch_idx++) { - struct neo_apple_epic_service *candidate = &ep->services[ch_idx]; - - if (neo_afk_service_reinit(candidate, ep, ops, channel)) { - service = candidate; - break; - } - } - if (!service) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: no reusable service slots\n", - ep->endpoint); - goto free; - } - ep->num_channels = max(ep->num_channels, ch_idx + 1); - ops->init(service, neo_epic_name, neo_epic_class, neo_epic_unit); - dev_info(ep->neo_dcp->dev, "AFK[ep:%02x]: new service %s on channel %d\n", - ep->endpoint, service_name, channel); - - neo_afk_populate_service_debugfs(service); - -free: - kfree(neo_epic_name); - kfree(neo_epic_class); -} - -static void neo_afk_recv_handle_teardown(struct neo_apple_dcp_afkep *ep, u32 channel) -{ - struct neo_apple_epic_service *service; - const struct neo_apple_epic_service_ops *ops; - unsigned long flags; - - service = neo_afk_epic_find_service(ep, channel); - if (!service) { - dev_warn(ep->neo_dcp->dev, "AFK[ep:%02x]: teardown for disabled channel %u\n", - ep->endpoint, channel); - return; - } - - neo_afk_remove_service_debugfs(service); - - spin_lock_irqsave(&service->lock, flags); - if (service->torndown) { - spin_unlock_irqrestore(&service->lock, flags); - return; - } - /* Outstanding commands remain discoverable until their replies arrive. */ - service->torndown = true; - service->enabled = false; - ops = service->ops; - spin_unlock_irqrestore(&service->lock, flags); - - if (ops->teardown) - ops->teardown(service); -} - -static void neo_afk_recv_handle_reply(struct neo_apple_dcp_afkep *ep, u32 channel, - u16 tag, void *payload, size_t payload_size) -{ - struct neo_epic_cmd *cmd = payload; - struct neo_apple_epic_service *service; - unsigned long flags; - u8 idx = tag & 0xff; - void *rxbuf, *txbuf; - dma_addr_t rxbuf_dma, txbuf_dma; - size_t rxlen, txlen; - - service = neo_afk_epic_find_service(ep, channel); - if (!service) { - dev_warn(ep->neo_dcp->dev, "AFK[ep:%02x]: command reply on disabled channel %u\n", - ep->endpoint, channel); - return; - } - - if (payload_size < sizeof(*cmd)) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: command reply on channel %d too small: %ld\n", - ep->endpoint, channel, payload_size); - return; - } - - if (idx >= MAX_PENDING_CMDS) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: command reply on channel %d out of range: %d\n", - ep->endpoint, channel, idx); - return; - } - - spin_lock_irqsave(&service->lock, flags); - if (!test_bit(idx, service->cmd_map) || service->cmds[idx].done) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: command reply on channel %d already handled\n", - ep->endpoint, channel); - spin_unlock_irqrestore(&service->lock, flags); - return; - } - - if (tag != service->cmds[idx].tag) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: command reply on channel %d has invalid tag: expected 0x%04x != 0x%04x\n", - ep->endpoint, channel, tag, service->cmds[idx].tag); - spin_unlock_irqrestore(&service->lock, flags); - return; - } - - service->cmds[idx].done = true; - service->cmds[idx].retcode = le32_to_cpu(cmd->retcode); - if (service->cmds[idx].free_on_ack) { - /* defer freeing until we're no longer in atomic context */ - rxbuf = service->cmds[idx].rxbuf; - txbuf = service->cmds[idx].txbuf; - rxlen = service->cmds[idx].rxlen; - txlen = service->cmds[idx].txlen; - rxbuf_dma = service->cmds[idx].rxbuf_dma; - txbuf_dma = service->cmds[idx].txbuf_dma; - bitmap_release_region(service->cmd_map, idx, 0); - } else { - rxbuf = txbuf = NULL; - rxlen = txlen = 0; - } - if (service->cmds[idx].completion) - complete(service->cmds[idx].completion); - - spin_unlock_irqrestore(&service->lock, flags); - - if (rxbuf && rxlen) - dma_free_coherent(ep->neo_dcp->dev, rxlen, rxbuf, rxbuf_dma); - if (txbuf && txlen) - dma_free_coherent(ep->neo_dcp->dev, txlen, txbuf, txbuf_dma); -} - -struct neo_epic_std_service_ap_call { - __le32 unk0; - __le32 unk1; - __le32 type; - __le32 len; - __le32 magic; - u8 _unk[48]; -} __attribute__((packed)); - -static void neo_afk_recv_handle_std_service(struct neo_apple_dcp_afkep *ep, u32 channel, - u32 type, struct neo_epic_hdr *ehdr, - struct neo_epic_sub_hdr *eshdr, - void *payload, size_t payload_size) -{ - struct neo_apple_epic_service *service = neo_afk_epic_find_service(ep, channel); - - if (!service) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: std service notify on disabled channel %u\n", - ep->endpoint, channel); - return; - } - if (service->torndown) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: std service notify on torn down service " - "(chan:%u)\n", ep->endpoint, channel); - return; - } - - if (type == EPIC_TYPE_NOTIFY && eshdr->category == EPIC_CAT_NOTIFY) { - struct neo_epic_std_service_ap_call *call = payload; - size_t call_size; - void *reply; - int ret; - - if (payload_size < sizeof(*call)) - return; - - call_size = le32_to_cpu(call->len); - if (payload_size < sizeof(*call) + call_size) - return; - - if (!service->ops->call) - return; - reply = kzalloc(payload_size, GFP_KERNEL); - if (!reply) - return; - - ret = service->ops->call(service, le32_to_cpu(call->type), - payload + sizeof(*call), call_size, - reply + sizeof(*call), call_size); - if (ret) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) apcall %u handler returned %d, sending NO reply\n", - ep->endpoint, service->ops->name, channel, - le32_to_cpu(call->type), ret); - kfree(reply); - return; - } - - memcpy(reply, call, sizeof(*call)); - neo_afk_send_epic(ep, channel, le16_to_cpu(eshdr->tag), - EPIC_TYPE_NOTIFY_ACK, EPIC_CAT_REPLY, - EPIC_SUBTYPE_STD_SERVICE, reply, payload_size); - kfree(reply); - - return; - } - - if (type == EPIC_TYPE_NOTIFY && eshdr->category == EPIC_CAT_REPORT) { - if (service->ops->report) - service->ops->report(service, le16_to_cpu(eshdr->type), - payload, payload_size); - return; - } - - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: channel %d received unhandled standard service message: %x / %x\n", - ep->endpoint, channel, type, eshdr->category); - print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, payload, - payload_size, true); -} - -static bool neo_afk_validate_h17p_header(const u8 *data, size_t size) -{ - /* The measured length counts bytes after the first eight header bytes. */ - return size >= sizeof(struct neo_epic_hdr) + - sizeof(struct neo_epic_sub_hdr_h17p) && - get_unaligned_le32(data + 4) == size - 8; -} - -static void neo_afk_recv_handle(struct neo_apple_dcp_afkep *ep, u32 channel, u32 type, - u8 *data, size_t data_size) -{ - struct neo_apple_epic_service *service; - struct neo_epic_hdr *ehdr = (struct neo_epic_hdr *)data; - struct neo_epic_sub_hdr *eshdr = - (struct neo_epic_sub_hdr *)(data + sizeof(*ehdr)); - struct neo_epic_sub_hdr h17p_eshdr; - size_t hdr_len = sizeof(*ehdr) + sizeof(*eshdr); - u16 subtype; - u8 *payload; - size_t payload_size; - - if (ep->neo_dcp->fw_compat == DCP_FIRMWARE_H17P) { - const struct neo_epic_sub_hdr_h17p *c = - (const struct neo_epic_sub_hdr_h17p *)(data + sizeof(*ehdr)); - - hdr_len = sizeof(*ehdr) + sizeof(*c); - if (data_size < hdr_len) - goto too_small; - if (ep->neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !neo_afk_validate_h17p_header(data, data_size)) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: invalid compact message length\n", - ep->endpoint); - return; - } - - /* - * H17P multiplexes every service of an endpoint onto queue - * channel 0 and carries the real channel in the EPIC header - * instead: byte 0 is a sequence counter, byte 1 a flags byte - * and bytes 2-3 the channel. ep:20 announces "system" as - * channel 1 and "powerlog-service" as channel 3, then sends the - * mNits/uAmps/iDAC backlight reports on channel 3. Taking the - * queue channel at face value binds "system" to 0 and makes - * every later announce and report look like a duplicate. - */ - channel = le16_to_cpup((__le16 *)(data + 2)); - - /* - * Normalise into the wide sub-header the rest of this file - * expects, mapping H17P's announce subtype onto the canonical - * one so no downstream check has to know the difference. - */ - memset(&h17p_eshdr, 0, sizeof(h17p_eshdr)); - h17p_eshdr.category = c->category; - h17p_eshdr.type = cpu_to_le16(c->type == EPIC_SUBTYPE_ANNOUNCE_H17P ? - EPIC_SUBTYPE_ANNOUNCE : c->type); - h17p_eshdr.tag = cpu_to_le16(le32_to_cpu(c->tag)); - eshdr = &h17p_eshdr; - } - - if (data_size < hdr_len) - goto too_small; - - subtype = le16_to_cpu(eshdr->type); - payload = data + hdr_len; - payload_size = data_size - hdr_len; - - trace_neo_afk_recv_handle(ep, channel, type, data_size, ehdr, eshdr); - - service = neo_afk_epic_find_service(ep, channel); - - if (!service) { - if (type != EPIC_TYPE_NOTIFY && type != EPIC_TYPE_REPLY) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: expected notify but got 0x%x on channel %d\n", - ep->endpoint, type, channel); - return; - } - if (eshdr->category != EPIC_CAT_REPORT) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: expected report but got 0x%x on channel %d\n", - ep->endpoint, eshdr->category, channel); - return; - } - if (subtype == EPIC_SUBTYPE_TEARDOWN) { - dev_dbg(ep->neo_dcp->dev, - "AFK[ep:%02x]: teardown without service on channel %d\n", - ep->endpoint, channel); - return; - } - if (subtype != EPIC_SUBTYPE_ANNOUNCE) { - dev_err(ep->neo_dcp->dev, - "AFK[ep:%02x]: expected announce but got 0x%x on channel %d\n", - ep->endpoint, subtype, channel); - return; - } - - return neo_afk_recv_handle_init(ep, channel, payload, payload_size); - } - - if (!service) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: channel %d has no service\n", - ep->endpoint, channel); - return; - } - - if (type == EPIC_TYPE_NOTIFY && eshdr->category == EPIC_CAT_REPORT && - subtype == EPIC_SUBTYPE_TEARDOWN) - return neo_afk_recv_handle_teardown(ep, channel); - - if (type == EPIC_TYPE_REPLY && eshdr->category == EPIC_CAT_REPLY) - return neo_afk_recv_handle_reply(ep, channel, - le16_to_cpu(eshdr->tag), payload, - payload_size); - - if (subtype == EPIC_SUBTYPE_STD_SERVICE) - return neo_afk_recv_handle_std_service( - ep, channel, type, ehdr, eshdr, payload, payload_size); - - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: channel %d received unhandled message " - "(type %x subtype %x)\n", ep->endpoint, channel, type, subtype); - print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, payload, - payload_size, true); - return; - -too_small: - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: payload too small: %lx\n", - ep->endpoint, data_size); -} - -static bool neo_afk_recv(struct neo_apple_dcp_afkep *ep) -{ - struct neo_afk_qe *hdr; - u32 rptr, wptr; - u32 magic, size, channel, type; - - if (!ep->rxbfr.ready) { - dev_err(ep->neo_dcp->dev, "AFK[ep:%02x]: got RECV but not ready\n", - ep->endpoint); - return false; - } - - rptr = le32_to_cpu(*ep->rxbfr.rptr); - wptr = le32_to_cpu(*ep->rxbfr.wptr); - trace_neo_afk_recv_rwptr_pre(ep, rptr, wptr); - - if (rptr == wptr) - return false; - - if (rptr > (ep->rxbfr.bufsz - sizeof(*hdr))) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: rptr out of bounds: 0x%x > 0x%lx\n", - ep->endpoint, rptr, ep->rxbfr.bufsz - sizeof(*hdr)); - return false; - } - - dma_rmb(); - - hdr = ep->rxbfr.buf + rptr; - magic = le32_to_cpu(hdr->magic); - size = le32_to_cpu(hdr->size); - trace_neo_afk_recv_qe(ep, rptr, magic, size); - - if (magic != QE_MAGIC) { - dev_warn(ep->neo_dcp->dev, "AFK[ep:%02x]: invalid queue entry magic: 0x%x\n", - ep->endpoint, magic); - return false; - } - - /* - * If there's not enough space for the payload the co-processor inserted - * the current dummy queue entry and we have to advance to the next one - * which will contain the real data. - */ - if (rptr + size + sizeof(*hdr) > ep->rxbfr.bufsz) { - rptr = 0; - hdr = ep->rxbfr.buf + rptr; - magic = le32_to_cpu(hdr->magic); - size = le32_to_cpu(hdr->size); - trace_neo_afk_recv_qe(ep, rptr, magic, size); - - if (magic != QE_MAGIC) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: invalid next queue entry magic: 0x%x\n", - ep->endpoint, magic); - return false; - } - } - - if (rptr + size + sizeof(*hdr) > ep->rxbfr.bufsz) { - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: queue entry out of bounds: 0x%lx > 0x%lx\n", - ep->endpoint, rptr + size + sizeof(*hdr), ep->rxbfr.bufsz); - return false; - } - - channel = le32_to_cpu(hdr->channel); - type = le32_to_cpu(hdr->type); - /* - * Publishing rptr releases this entry to the firmware. Handlers can - * sleep and send replies, so retain a private copy before allowing - * the producer to reuse the ring storage. - */ - memcpy(ep->rx_scratch, hdr->data, size); - - rptr = neo_afk_ring_advance(rptr, sizeof(*hdr) + size, - ep->rxbfr.bufsz, ep->rxbfr.stride); - - dma_mb(); - - *ep->rxbfr.rptr = cpu_to_le32(rptr); - trace_neo_afk_recv_rwptr_post(ep, rptr, wptr); - - neo_afk_recv_handle(ep, channel, type, ep->rx_scratch, size); - - return true; -} - -static void neo_afk_receive_message_worker(struct work_struct *work_) -{ - struct neo_afk_receive_message_work *work; - u16 type; - - work = container_of(work_, struct neo_afk_receive_message_work, work); - - type = FIELD_GET(RBEP_TYPE, work->message); - switch (type) { - case RBEP_INIT_ACK: - break; - - case RBEP_START_ACK: - complete_all(&work->ep->started); - break; - - case RBEP_SHUTDOWN_ACK: - complete_all(&work->ep->stopped); - break; - - case RBEP_GETBUF: - neo_afk_getbuf(work->ep, work->message); - break; - - case RBEP_INIT_TX: - neo_afk_init_rxtx(work->ep, work->message, &work->ep->txbfr); - break; - - case RBEP_INIT_RX: - neo_afk_init_rxtx(work->ep, work->message, &work->ep->rxbfr); - break; - - case RBEP_RECV: - while (neo_afk_recv(work->ep)) - ; - break; - - default: - dev_err(work->ep->neo_dcp->dev, - "Received unknown AFK message type: 0x%x\n", type); - } - - kfree(work); -} - -int neo_afk_receive_message(struct neo_apple_dcp_afkep *ep, u64 message) -{ - struct neo_afk_receive_message_work *work; - unsigned long flags; - - if (!ep) - return -ENODEV; - - // TODO: comment why decoupling from rtkit thread is required here - work = kzalloc(sizeof(*work), GFP_KERNEL); - if (!work) - return -ENOMEM; - - work->ep = ep; - work->message = message; - INIT_WORK(&work->work, neo_afk_receive_message_worker); - spin_lock_irqsave(&ep->lock, flags); - if (ep->stopping) { - spin_unlock_irqrestore(&ep->lock, flags); - kfree(work); - return -ESHUTDOWN; - } - queue_work(ep->wq, &work->work); - spin_unlock_irqrestore(&ep->lock, flags); - - return 0; -} - -int neo_afk_send_epic(struct neo_apple_dcp_afkep *ep, u32 channel, u16 tag, - enum neo_epic_type etype, enum neo_epic_category ecat, u8 stype, - const void *payload, size_t payload_len) -{ - u32 rptr, wptr; - struct neo_afk_qe *hdr, *hdr2; - struct neo_epic_hdr *ehdr; - struct neo_epic_sub_hdr *eshdr; - unsigned long flags; - size_t total_epic_size, total_size; - int ret; - - /* - * H17P frames EPIC messages differently (see afk_recv_handle()): an - * 8-byte sub-header, with the service channel carried in the EPIC - * header on queue channel 0. Only the receive side of that format is - * known, and the 24-byte framing below would be misparsed, so send - * nothing. The display path needs no AP-initiated EPIC traffic: the - * callers are optional (verbose firmware logging, EDID copy, audio, - * DPTX, debugfs) or replies to calls only DPTX services make. - */ - if (ep->neo_dcp->fw_compat == DCP_FIRMWARE_H17P) { - dev_dbg_once(ep->neo_dcp->dev, - "AFK[ep:%02x]: not sending EPIC message on H17P\n", - ep->endpoint); - return -EOPNOTSUPP; - } - - spin_lock_irqsave(&ep->lock, flags); - if (ep->stopping) { - ret = -ESHUTDOWN; - goto out; - } - - dma_rmb(); - rptr = le32_to_cpu(*ep->txbfr.rptr); - wptr = le32_to_cpu(*ep->txbfr.wptr); - trace_neo_afk_send_rwptr_pre(ep, rptr, wptr); - total_epic_size = sizeof(*ehdr) + sizeof(*eshdr) + payload_len; - total_size = sizeof(*hdr) + total_epic_size; - - hdr = hdr2 = NULL; - - /* - * We need to figure out how to place the entire headers and payload - * into the ring buffer: - * - If the write pointer is in front of the read pointer we just need - * enough space inbetween to store everything. - * - If the read pointer has already wrapper around the end of the - * buffer we can - * a) either store the entire payload at the writer pointer if - * there's enough space until the end, - * b) or just store the queue entry at the write pointer to indicate - * that we need to wrap to the start and then store the headers - * and the payload at the beginning of the buffer. The queue - * header has to be store twice in this case. - * In either case we have to ensure that there's always enough space - * so that we don't accidentally overwrite other buffers. - */ - if (wptr < rptr) { - /* - * If wptr < rptr we can't wrap around and only have to make - * sure that there's enough space for the entire payload. - */ - if (wptr + total_size > rptr) { - ret = -ENOMEM; - goto out; - } - - hdr = ep->txbfr.buf + wptr; - wptr += sizeof(*hdr); - } else { - /* We need enough space to place at least a queue entry */ - if (wptr + sizeof(*hdr) > ep->txbfr.bufsz) { - ret = -ENOMEM; - goto out; - } - - /* - * If we can place a single queue entry but not the full payload - * we need to place one queue entry at the end of the ring - * buffer and then another one together with the entire - * payload at the beginning. - */ - if (wptr + total_size > ep->txbfr.bufsz) { - /* - * Ensure there's space for the queue entry at the - * beginning - */ - if (sizeof(*hdr) > rptr) { - ret = -ENOMEM; - goto out; - } - - /* - * Place two queue entries to indicate we want to wrap - * around to the firmware. - */ - hdr = ep->txbfr.buf + wptr; - hdr2 = ep->txbfr.buf; - wptr = sizeof(*hdr); - - /* Ensure there's enough space for the entire payload */ - if (wptr + total_epic_size > rptr) { - ret = -ENOMEM; - goto out; - } - } else { - /* We have enough space to place the entire payload */ - hdr = ep->txbfr.buf + wptr; - wptr += sizeof(*hdr); - } - } - /* - * At this point we're guaranteed that hdr (and possibly hdr2) point - * to a buffer large enough to fit the queue entry and that we have - * enough space at wptr to store the payload. - */ - - hdr->magic = cpu_to_le32(QE_MAGIC); - hdr->size = cpu_to_le32(total_epic_size); - hdr->channel = cpu_to_le32(channel); - hdr->type = cpu_to_le32(etype); - if (hdr2) - memcpy(hdr2, hdr, sizeof(*hdr)); - - ehdr = ep->txbfr.buf + wptr; - memset(ehdr, 0, sizeof(*ehdr)); - ehdr->version = 2; - ehdr->seq = cpu_to_le16(ep->qe_seq++); - ehdr->timestamp = cpu_to_le64(0); - wptr += sizeof(*ehdr); - - eshdr = ep->txbfr.buf + wptr; - memset(eshdr, 0, sizeof(*eshdr)); - eshdr->length = cpu_to_le32(payload_len); - eshdr->version = 4; - eshdr->category = ecat; - eshdr->type = cpu_to_le16(stype); - eshdr->timestamp = cpu_to_le64(0); - eshdr->tag = cpu_to_le16(tag); - if (ecat == EPIC_CAT_REPLY) - eshdr->inline_len = cpu_to_le32(payload_len - 4); - else - eshdr->inline_len = cpu_to_le32(0); - wptr += sizeof(*eshdr); - - memcpy(ep->txbfr.buf + wptr, payload, payload_len); - wptr += payload_len; - wptr = neo_afk_ring_advance(wptr, 0, ep->txbfr.bufsz, - ep->txbfr.stride); - trace_neo_afk_send_rwptr_post(ep, rptr, wptr); - - /* Publish the completed entry before allowing the consumer to read it. */ - dma_wmb(); - *ep->txbfr.wptr = cpu_to_le32(wptr); - neo_afk_send(ep, FIELD_PREP(RBEP_TYPE, RBEP_SEND) | - FIELD_PREP(SEND_WPTR, wptr)); - ret = 0; - -out: - spin_unlock_irqrestore(&ep->lock, flags); - return ret; -} - -int neo_afk_send_command(struct neo_apple_epic_service *service, u8 type, - const void *payload, size_t payload_len, void *output, - size_t output_len, u32 *retcode) -{ - return neo_afk_send_command_timeout(service, type, payload, payload_len, - output, output_len, retcode, - MSEC_PER_SEC); -} - -int neo_afk_send_command_timeout(struct neo_apple_epic_service *service, u8 type, - const void *payload, size_t payload_len, - void *output, size_t output_len, u32 *retcode, - unsigned int timeout_ms) -{ - struct neo_epic_cmd cmd; - void *rxbuf, *txbuf; - dma_addr_t rxbuf_dma, txbuf_dma; - unsigned long flags; - int ret, idx; - u16 tag; - struct neo_apple_dcp_afkep *ep; - DECLARE_COMPLETION_ONSTACK(completion); - - service = neo_afk_service_get(service); - if (!service) - return -ENODEV; - ep = service->ep; - if (READ_ONCE(ep->neo_dcp->quiescing)) { - ret = -ENODEV; - goto err_put_service; - } - - rxbuf = dma_alloc_coherent(ep->neo_dcp->dev, output_len, &rxbuf_dma, - GFP_KERNEL); - if (!rxbuf) { - ret = -ENOMEM; - goto err_put_service; - } - txbuf = dma_alloc_coherent(ep->neo_dcp->dev, payload_len, &txbuf_dma, - GFP_KERNEL); - if (!txbuf) { - ret = -ENOMEM; - goto err_free_rxbuf; - } - - memcpy(txbuf, payload, payload_len); - - memset(&cmd, 0, sizeof(cmd)); - cmd.retcode = cpu_to_le32(0); - cmd.rxbuf = cpu_to_le64(rxbuf_dma); - cmd.rxlen = cpu_to_le32(output_len); - cmd.txbuf = cpu_to_le64(txbuf_dma); - cmd.txlen = cpu_to_le32(payload_len); - - spin_lock_irqsave(&service->lock, flags); - if (service->torndown || !service->enabled) { - ret = -ENODEV; - goto err_unlock; - } - idx = bitmap_find_free_region(service->cmd_map, MAX_PENDING_CMDS, 0); - if (idx < 0) { - ret = -ENOSPC; - goto err_unlock; - } - - tag = (service->cmd_tag & 0xff) << 8; - tag |= idx & 0xff; - service->cmd_tag++; - - service->cmds[idx].tag = tag; - service->cmds[idx].rxbuf = rxbuf; - service->cmds[idx].txbuf = txbuf; - service->cmds[idx].rxbuf_dma = rxbuf_dma; - service->cmds[idx].txbuf_dma = txbuf_dma; - service->cmds[idx].rxlen = output_len; - service->cmds[idx].txlen = payload_len; - service->cmds[idx].free_on_ack = false; - service->cmds[idx].done = false; - service->cmds[idx].completion = &completion; - init_completion(&completion); - - ret = neo_afk_send_epic(service->ep, service->channel, tag, - EPIC_TYPE_COMMAND, EPIC_CAT_COMMAND, type, &cmd, - sizeof(cmd)); - spin_unlock_irqrestore(&service->lock, flags); - if (ret) - goto err_free_cmd; - - ret = wait_for_completion_timeout(&completion, - msecs_to_jiffies(timeout_ms)); - - if (ret <= 0) { - spin_lock_irqsave(&service->lock, flags); - /* - * Check again while we're inside the lock to make sure - * the command wasn't completed just after - * wait_for_completion_timeout returned. - */ - if (!service->cmds[idx].done) { - service->cmds[idx].completion = NULL; - service->cmds[idx].free_on_ack = true; - spin_unlock_irqrestore(&service->lock, flags); - dev_warn(ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) command type 0x%x tag 0x%04x timed out after %u ms\n", - ep->endpoint, service->ops->name, service->channel, - type, tag, timeout_ms); - neo_afk_service_put(service); - return -ETIMEDOUT; - } - spin_unlock_irqrestore(&service->lock, flags); - } - - ret = 0; - if (retcode) - *retcode = service->cmds[idx].retcode; - if (output && output_len) - memcpy(output, rxbuf, output_len); - -err_free_cmd: - spin_lock_irqsave(&service->lock, flags); - service->cmds[idx].completion = NULL; - bitmap_release_region(service->cmd_map, idx, 0); -err_unlock: - spin_unlock_irqrestore(&service->lock, flags); - dma_free_coherent(ep->neo_dcp->dev, payload_len, txbuf, txbuf_dma); -err_free_rxbuf: - dma_free_coherent(ep->neo_dcp->dev, output_len, rxbuf, rxbuf_dma); -err_put_service: - neo_afk_service_put(service); - return ret; -} - -int neo_afk_service_call(struct neo_apple_epic_service *service, u16 group, u32 command, - const void *data, size_t data_len, size_t data_pad, - void *output, size_t output_len, size_t output_pad) -{ - return neo_afk_service_call_timeout(service, group, command, data, data_len, - data_pad, output, output_len, - output_pad, MSEC_PER_SEC); -} - -int neo_afk_service_call_timeout(struct neo_apple_epic_service *service, u16 group, - u32 command, const void *data, size_t data_len, - size_t data_pad, void *output, size_t output_len, - size_t output_pad, unsigned int timeout_ms) -{ - struct neo_epic_service_call *call; - void *bfr; - size_t bfr_len = max(data_len + data_pad, output_len + output_pad) + - sizeof(*call); - int ret; - u32 retcode; - u32 retlen; - - if (!service) - return -ENODEV; - - bfr = kzalloc(bfr_len, GFP_KERNEL); - if (!bfr) - return -ENOMEM; - - call = bfr; - - memset(call, 0, sizeof(*call)); - call->group = cpu_to_le16(group); - call->command = cpu_to_le32(command); - call->data_len = cpu_to_le32(data_len + data_pad); - call->magic = cpu_to_le32(EPIC_SERVICE_CALL_MAGIC); - - memcpy(bfr + sizeof(*call), data, data_len); - - ret = neo_afk_send_command_timeout(service, EPIC_SUBTYPE_STD_SERVICE, bfr, - bfr_len, bfr, bfr_len, &retcode, - timeout_ms); - if (ret) { - dev_warn(service->ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) service call group %u cmd %u failed to complete: %d\n", - service->ep->endpoint, service->ops->name, - service->channel, group, command, ret); - goto out; - } - if (retcode) { - dev_warn(service->ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) service call group %u cmd %u returned retcode 0x%08x (reply data_len %u)\n", - service->ep->endpoint, service->ops->name, - service->channel, group, command, retcode, - le32_to_cpu(call->data_len)); - ret = -EINVAL; - goto out; - } - if (le32_to_cpu(call->magic) != EPIC_SERVICE_CALL_MAGIC || - le16_to_cpu(call->group) != group || - le32_to_cpu(call->command) != command) { - dev_warn(service->ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) service call reply mismatch: sent magic 0x%08x group %u cmd %u, got magic 0x%08x group %u cmd %u\n", - service->ep->endpoint, service->ops->name, - service->channel, EPIC_SERVICE_CALL_MAGIC, group, - command, le32_to_cpu(call->magic), - le16_to_cpu(call->group), le32_to_cpu(call->command)); - ret = -EINVAL; - goto out; - } - dev_dbg(service->ep->neo_dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) service call group %u cmd %u ok (reply data_len %u)\n", - service->ep->endpoint, service->ops->name, service->channel, - group, command, le32_to_cpu(call->data_len)); - - retlen = le32_to_cpu(call->data_len); - if (output_len < retlen) - retlen = output_len; - if (output && output_len) { - memset(output, 0, output_len); - memcpy(output, bfr + sizeof(*call), retlen); - } - -out: - kfree(bfr); - return ret; -} - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_DEBUG) - -#define AFK_DEBUGFS_MAX_REPLY 8192 - -static ssize_t service_call_write_file(struct file *file, const char __user *user_buf, - size_t count, loff_t *ppos) -{ - struct neo_apple_epic_service *srv = file->private_data; - void *buf; - int ret; - struct { - u32 group; - u32 command; - } call_info; - - if (count < sizeof(call_info)) - return -EINVAL; - if (!srv->debugfs.scratch) { - srv->debugfs.scratch = \ - devm_kzalloc(srv->ep->neo_dcp->dev, AFK_DEBUGFS_MAX_REPLY, GFP_KERNEL); - if (!srv->debugfs.scratch) - return -ENOMEM; - } - - if (copy_from_user(&call_info, user_buf, sizeof(call_info))) - return -EFAULT; - user_buf += sizeof(call_info); - count -= sizeof(call_info); - - buf = kmalloc(count, GFP_KERNEL); - if (!buf) - return -ENOMEM; - if (copy_from_user(buf, user_buf, count)) { - kfree(buf); - return -EFAULT; - } - - memset(srv->debugfs.scratch, 0, AFK_DEBUGFS_MAX_REPLY); - dma_mb(); - - ret = neo_afk_service_call(srv, call_info.group, call_info.command, buf, count, 0, - srv->debugfs.scratch, AFK_DEBUGFS_MAX_REPLY, 0); - kfree(buf); - - if (ret < 0) - return ret; - - return count + sizeof(call_info); -} - -static ssize_t service_call_read_file(struct file *file, char __user *user_buf, - size_t count, loff_t *ppos) -{ - struct neo_apple_epic_service *srv = file->private_data; - - if (!srv->debugfs.scratch) - return -EINVAL; - - return simple_read_from_buffer(user_buf, count, ppos, - srv->debugfs.scratch, AFK_DEBUGFS_MAX_REPLY); -} - -static const struct file_operations service_call_fops = { - .open = simple_open, - .write = service_call_write_file, - .read = service_call_read_file, -}; - -static ssize_t service_raw_call_write_file(struct file *file, const char __user *user_buf, - size_t count, loff_t *ppos) -{ - struct neo_apple_epic_service *srv = file->private_data; - u32 retcode; - int ret; - - if (!count) - return 0; - if (count > AFK_DEBUGFS_MAX_REPLY) - return -E2BIG; - - if (!srv->debugfs.scratch) { - srv->debugfs.scratch = \ - devm_kzalloc(srv->ep->neo_dcp->dev, AFK_DEBUGFS_MAX_REPLY, GFP_KERNEL); - if (!srv->debugfs.scratch) - return -ENOMEM; - } - - memset(srv->debugfs.scratch, 0, AFK_DEBUGFS_MAX_REPLY); - if (copy_from_user(srv->debugfs.scratch, user_buf, count)) - return -EFAULT; - - ret = neo_afk_send_command(srv, EPIC_SUBTYPE_STD_SERVICE, srv->debugfs.scratch, count, - srv->debugfs.scratch, AFK_DEBUGFS_MAX_REPLY, &retcode); - if (ret < 0) - return ret; - if (retcode) - return -EINVAL; - - return count; -} - -static ssize_t service_raw_call_read_file(struct file *file, char __user *user_buf, - size_t count, loff_t *ppos) -{ - struct neo_apple_epic_service *srv = file->private_data; - - if (!srv->debugfs.scratch) - return -EINVAL; - - return simple_read_from_buffer(user_buf, count, ppos, - srv->debugfs.scratch, AFK_DEBUGFS_MAX_REPLY); -} - -static const struct file_operations service_raw_call_fops = { - .open = simple_open, - .write = service_raw_call_write_file, - .read = service_raw_call_read_file, -}; - -static void neo_afk_populate_service_debugfs(struct neo_apple_epic_service *srv) -{ - if (!srv->ep->debugfs_entry || !srv->ops) - return; - - if (strcmp(srv->ops->name, "DCPAVAudioInterface") == 0) { - srv->debugfs.entry = debugfs_create_dir(srv->ops->name, - srv->ep->debugfs_entry); - debugfs_create_file("call", 0600, srv->debugfs.entry, srv, - &service_call_fops); - debugfs_create_file("raw_call", 0600, srv->debugfs.entry, srv, - &service_raw_call_fops); - } -} - -static void neo_afk_remove_service_debugfs(struct neo_apple_epic_service *srv) -{ - if (srv->debugfs.entry) { - debugfs_remove_recursive(srv->debugfs.entry); - srv->debugfs.entry = NULL; - } -} - -#endif diff --git a/drivers/gpu/drm/apple-neo/afk.h b/drivers/gpu/drm/apple-neo/afk.h deleted file mode 100644 index 3ec1f0cddcf37b..00000000000000 --- a/drivers/gpu/drm/apple-neo/afk.h +++ /dev/null @@ -1,347 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * AFK (Apple Firmware Kit) EPIC (EndPoint Interface Client) support - */ -/* Copyright 2022 Sven Peter */ - -#ifndef _DRM_APPLE_DCP_AFK_H -#define _DRM_APPLE_DCP_AFK_H - -#include -#include -#include -#include -#include -#include -#include - -#include "dcp.h" - -#define AFK_MAX_CHANNEL 16 -#define MAX_PENDING_CMDS 16 - -struct neo_apple_epic_service_ops; -struct neo_apple_dcp_afkep; - -struct neo_epic_cmd_info { - u16 tag; - - void *rxbuf; - void *txbuf; - dma_addr_t rxbuf_dma; - dma_addr_t txbuf_dma; - size_t rxlen; - size_t txlen; - - u32 retcode; - bool done; - bool free_on_ack; - struct completion *completion; -}; - -struct neo_apple_epic_service { - const struct neo_apple_epic_service_ops *ops; - struct neo_apple_dcp_afkep *ep; - - struct neo_epic_cmd_info cmds[MAX_PENDING_CMDS]; - DECLARE_BITMAP(cmd_map, MAX_PENDING_CMDS); - u8 cmd_tag; - spinlock_t lock; - /* Pins held by owners and synchronous callers; protected by lock. */ - unsigned int users; - - u32 channel; - bool enabled; - bool torndown; - - void *cookie; - - struct { - struct dentry *entry; - u8 *scratch; - } debugfs; -}; - -enum neo_epic_subtype; - -struct neo_apple_epic_service_ops { - const char name[32]; - /* All retained pointers must be pinned before enabling slot reuse. */ - bool reusable; - - void (*init)(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit); - int (*call)(struct neo_apple_epic_service *service, u32 idx, - const void *data, size_t data_size, void *reply, - size_t reply_size); - int (*report)(struct neo_apple_epic_service *service, enum neo_epic_subtype type, - const void *data, size_t data_size); - void (*teardown)(struct neo_apple_epic_service *service); -}; - -static inline int neo_afk_ring_stride(u32 total, u32 bufsz, u32 *stride) -{ - u32 header, block; - - if (bufsz >= total) - return -EINVAL; - header = total - bufsz; - block = header / 3; - if (header % 3 || block < 0x40 || block % 0x40 || - bufsz < block || bufsz % block) - return -EINVAL; - - *stride = block; - return 0; -} - -static inline u32 neo_afk_ring_advance(u32 ptr, u32 bytes, u32 bufsz, - u32 stride) -{ - u32 next = roundup(ptr + bytes, stride); - - return next == bufsz ? 0 : next; -} -static inline struct neo_apple_epic_service *neo_afk_service_get(struct neo_apple_epic_service *service) -{ - unsigned long flags; - bool available; - - if (!service) - return NULL; - - spin_lock_irqsave(&service->lock, flags); - available = service->enabled && !service->torndown; - if (available) - service->users++; - spin_unlock_irqrestore(&service->lock, flags); - - return available ? service : NULL; -} - -static inline void neo_afk_service_put(struct neo_apple_epic_service *service) -{ - unsigned long flags; - - spin_lock_irqsave(&service->lock, flags); - if (!WARN_ON(!service->users)) - service->users--; - spin_unlock_irqrestore(&service->lock, flags); -} - -static inline void neo_afk_service_disable(struct neo_apple_epic_service *service) -{ - unsigned long flags; - - spin_lock_irqsave(&service->lock, flags); - service->enabled = false; - spin_unlock_irqrestore(&service->lock, flags); -} - -static inline bool neo_afk_service_matches(struct neo_apple_epic_service *service, - u32 channel) -{ - unsigned long flags; - bool found; - - spin_lock_irqsave(&service->lock, flags); - /* Teardown does not cancel commands or relinquish their DMA. */ - found = service->channel == channel && - (service->enabled || - !bitmap_empty(service->cmd_map, MAX_PENDING_CMDS)); - spin_unlock_irqrestore(&service->lock, flags); - return found; -} - -/* The caller holds service->lock. */ -static inline bool neo_afk_service_reclaimable(struct neo_apple_epic_service *service) -{ - return !service->ops || - (service->ops->reusable && service->torndown && - !service->enabled && !service->users && - bitmap_empty(service->cmd_map, MAX_PENDING_CMDS)); -} - -static inline bool -neo_afk_service_reinit(struct neo_apple_epic_service *service, - struct neo_apple_dcp_afkep *ep, - const struct neo_apple_epic_service_ops *ops, u32 channel) -{ - unsigned long flags; - bool reusable; - - spin_lock_irqsave(&service->lock, flags); - reusable = neo_afk_service_reclaimable(service); - if (reusable) { - service->enabled = true; - service->torndown = false; - service->ops = ops; - service->ep = ep; - service->channel = channel; - service->cookie = NULL; - /* Keep the command-tag sequence across service generations. */ - } - spin_unlock_irqrestore(&service->lock, flags); - return reusable; -} - -struct neo_afk_ringbuffer_header { - __le32 bufsz; - u32 unk; - u32 _pad1[14]; - __le32 rptr; - u32 _pad2[15]; - __le32 wptr; - u32 _pad3[15]; -}; - -struct neo_afk_qe { -#define QE_MAGIC 0x20504f49 // ' POI' - __le32 magic; - __le32 size; - __le32 channel; - __le32 type; - u8 data[]; -}; - -struct neo_epic_hdr { - u8 version; - __le16 seq; - u8 _pad; - __le32 unk; - __le64 timestamp; -} __attribute__((packed)); - -struct neo_epic_sub_hdr { - __le32 length; - u8 version; - u8 category; - __le16 type; - __le64 timestamp; - __le16 tag; - __le16 unk; - __le32 inline_len; -} __attribute__((packed)); - -/* - * H17P compresses the EPIC sub-header from 24 bytes to 8 and drops the - * separate length/timestamp: the announce for the "system" service puts its - * name at data + 0x18, not data + 0x28. An announce carries type 0x11 and a - * standard-service report carries the usual 0xc0. - */ -struct neo_epic_sub_hdr_h17p { - u8 type; - u8 category; - __le16 flags; - __le32 tag; -} __packed; - -#define EPIC_SUBTYPE_ANNOUNCE_H17P 0x11 - -struct neo_epic_cmd { - __le32 retcode; - __le64 rxbuf; - __le64 txbuf; - __le32 rxlen; - __le32 txlen; - u8 rxcookie; - u8 txcookie; -} __attribute__((packed)); - -struct neo_epic_service_call { - u8 _pad0[2]; - __le16 group; - __le32 command; - __le32 data_len; -#define EPIC_SERVICE_CALL_MAGIC 0x69706378 - __le32 magic; - u8 _pad1[48]; -} __attribute__((packed)); -static_assert(sizeof(struct neo_epic_service_call) == 64); - -enum neo_epic_type { - EPIC_TYPE_NOTIFY = 0, - EPIC_TYPE_COMMAND = 3, - EPIC_TYPE_REPLY = 4, - EPIC_TYPE_NOTIFY_ACK = 8, -}; - -enum neo_epic_category { - EPIC_CAT_REPORT = 0x00, - EPIC_CAT_NOTIFY = 0x10, - EPIC_CAT_REPLY = 0x20, - EPIC_CAT_COMMAND = 0x30, -}; - -enum neo_epic_subtype { - EPIC_SUBTYPE_ANNOUNCE = 0x30, - EPIC_SUBTYPE_TEARDOWN = 0x32, - EPIC_SUBTYPE_STD_SERVICE = 0xc0, -}; - -struct neo_afk_ringbuffer { - bool ready; - __le32 *rptr; - __le32 *wptr; - void *buf; - size_t bufsz; - u32 stride; -}; - - -struct neo_apple_dcp_afkep { - struct neo_apple_dcp *neo_dcp; - - u32 endpoint; - struct workqueue_struct *wq; - - struct completion started; - struct completion stopped; - - void *bfr; - u16 bfr_tag; - size_t bfr_size; - dma_addr_t bfr_dma; - - struct neo_afk_ringbuffer txbfr; - struct neo_afk_ringbuffer rxbfr; - /* Private receive entry, owned by the ordered endpoint worker. */ - void *rx_scratch; - - spinlock_t lock; - u16 qe_seq; - bool stopping; /* lock: no new receive work after shutdown. */ - - const struct neo_apple_epic_service_ops *ops; - struct neo_apple_epic_service services[AFK_MAX_CHANNEL]; - u32 num_channels; - - struct dentry *debugfs_entry; - - bool match_epic_name; -}; - -struct neo_apple_dcp_afkep *neo_afk_init(struct neo_apple_dcp *neo_dcp, u32 endpoint, - const struct neo_apple_epic_service_ops *ops); -int neo_afk_start(struct neo_apple_dcp_afkep *ep); -void neo_afk_shutdown(struct neo_apple_dcp_afkep *ep); -void neo_afk_quiesce(struct neo_apple_dcp_afkep *ep); -int neo_afk_receive_message(struct neo_apple_dcp_afkep *ep, u64 message); -int neo_afk_send_epic(struct neo_apple_dcp_afkep *ep, u32 channel, u16 tag, - enum neo_epic_type etype, enum neo_epic_category ecat, u8 stype, - const void *payload, size_t payload_len); -int neo_afk_send_command(struct neo_apple_epic_service *service, u8 type, - const void *payload, size_t payload_len, void *output, - size_t output_len, u32 *retcode); -int neo_afk_send_command_timeout(struct neo_apple_epic_service *service, u8 type, - const void *payload, size_t payload_len, - void *output, size_t output_len, u32 *retcode, - unsigned int timeout_ms); -int neo_afk_service_call(struct neo_apple_epic_service *service, u16 group, u32 command, - const void *data, size_t data_len, size_t data_pad, - void *output, size_t output_len, size_t output_pad); -int neo_afk_service_call_timeout(struct neo_apple_epic_service *service, u16 group, - u32 command, const void *data, size_t data_len, - size_t data_pad, void *output, size_t output_len, - size_t output_pad, unsigned int timeout_ms); -#endif diff --git a/drivers/gpu/drm/apple-neo/afk_test.c b/drivers/gpu/drm/apple-neo/afk_test.c deleted file mode 100644 index ba71901920cc1f..00000000000000 --- a/drivers/gpu/drm/apple-neo/afk_test.c +++ /dev/null @@ -1,44 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2026 Ryan Murray */ - -#include -#include - -#include "afk.h" - -static void neo_afk_ring_stride_test(struct kunit *test) -{ - u32 stride; - - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10c0, 0x1000, &stride), 0); - KUNIT_EXPECT_EQ(test, stride, 0x40U); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x1180, 0x1000, &stride), 0); - KUNIT_EXPECT_EQ(test, stride, 0x80U); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x1000, 0x1000, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10bd, 0x1000, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10c1, 0x1000, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0xc0, 0, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10c3, 0x1000, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10c0, 0x1001, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x11c0, 0x1040, &stride), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x1180, 0x1000, &stride), 0); - KUNIT_EXPECT_EQ(test, neo_afk_ring_advance(0xf80, 0x80, 0x1000, stride), 0U); - KUNIT_EXPECT_EQ(test, neo_afk_ring_advance(0xe80, 0x80, 0x1000, stride), - 0xf00U); - KUNIT_EXPECT_EQ(test, neo_afk_ring_stride(0x10c0, 0x1000, &stride), 0); - KUNIT_EXPECT_EQ(test, neo_afk_ring_advance(0xfc0, 0x40, 0x1000, stride), 0U); -} - -static struct kunit_case neo_afk_ring_cases[] = { - KUNIT_CASE(neo_afk_ring_stride_test), - {} -}; - -static struct kunit_suite neo_afk_ring_suite = { - .name = "apple-afk-ring", - .test_cases = neo_afk_ring_cases, -}; - -kunit_test_suite(neo_afk_ring_suite); - -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple-neo/apple_drv.c b/drivers/gpu/drm/apple-neo/apple_drv.c deleted file mode 100644 index 6d665800061af8..00000000000000 --- a/drivers/gpu/drm/apple-neo/apple_drv.c +++ /dev/null @@ -1,988 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ -/* Based on meson driver which is - * Copyright (C) 2016 BayLibre, SAS - * Author: Neil Armstrong - * Copyright (C) 2015 Amlogic, Inc. All rights reserved. - * Copyright (C) 2014 Endless Mobile - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "dcp.h" -#include "plane.h" - -#define DRIVER_NAME "apple" -#define DRIVER_DESC "Apple display controller DRM driver" - -#define MAX_COPROCESSORS 8 - -struct neo_apple_drm_private { - struct drm_device drm; -}; - -DEFINE_DRM_GEM_DMA_FOPS(apple_fops); - -#define DART_PAGE_SIZE 16384 - -static int neo_apple_drm_gem_dumb_create(struct drm_file *file_priv, - struct drm_device *drm, - struct drm_mode_create_dumb *args) -{ - args->pitch = ALIGN(DIV_ROUND_UP(args->width * args->bpp, 8), 64); - args->size = round_up(args->pitch * args->height, DART_PAGE_SIZE); - - return drm_gem_dma_dumb_create_internal(file_priv, drm, args); -} - -#ifdef CONFIG_DRM_FBDEV_EMULATION -static int neo_apple_drm_fbdev_probe(struct drm_fb_helper *helper, - struct drm_fb_helper_surface_size *sizes) -{ - struct drm_crtc *crtc; - - drm_for_each_crtc(crtc, helper->dev) { - struct neo_apple_dcp *neo_dcp; - - neo_dcp = platform_get_drvdata(to_apple_crtc(crtc)->neo_dcp); - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - /* H17P applies the stored fourth byte as straight alpha. */ - sizes->surface_bpp = 32; - sizes->surface_depth = 32; - break; - } - } - - return drm_fbdev_dma_driver_fbdev_probe(helper, sizes); -} -#endif - -/* - * A compositor or boot splash is closing the device. drm_release() has - * already taken its file off the file list, so the Type-C fabric can tell - * whether any other one still has the device open. - */ -static void neo_apple_drm_postclose(struct drm_device *drm, struct drm_file *file) -{ - if (file->was_master) - neo_dcp_typec_reorder(); -} - -static const struct drm_driver neo_apple_drm_driver = { - DRM_GEM_DMA_DRIVER_OPS_WITH_DUMB_CREATE(neo_apple_drm_gem_dumb_create), -#ifdef CONFIG_DRM_FBDEV_EMULATION - .fbdev_probe = neo_apple_drm_fbdev_probe, -#else - DRM_FBDEV_DMA_DRIVER_OPS, -#endif - .postclose = neo_apple_drm_postclose, - .name = DRIVER_NAME, - .desc = DRIVER_DESC, - .major = 1, - .minor = 0, - .driver_features = DRIVER_MODESET | DRIVER_GEM | DRIVER_ATOMIC | DRIVER_SYNCOBJ | DRIVER_SYNCOBJ_TIMELINE, - .fops = &apple_fops, -}; - -static enum drm_connector_status -neo_apple_connector_detect(struct drm_connector *connector, bool force) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(connector); - - return neo_apple_connector->connected ? connector_status_connected : - connector_status_disconnected; -} - -static void neo_apple_connector_oob_hotplug(struct drm_connector *connector, - enum drm_connector_status status) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(connector); - - if (status == connector_status_connected) - neo_dcp_dptx_connect_oob(neo_apple_connector->neo_dcp, 0); - else if (status == connector_status_disconnected) - neo_dcp_dptx_disconnect_oob(neo_apple_connector->neo_dcp, 0); - else if (status == connector_status_unknown) - neo_dcp_retrain_oob(neo_apple_connector); - else - dev_err(&neo_apple_connector->neo_dcp->dev, "unexpected connector status" - ":0x%x in oob_hotplug event\n", (u32)status); -} - -static void neo_apple_crtc_atomic_enable(struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct drm_crtc_state *crtc_state; - crtc_state = drm_atomic_get_new_crtc_state(state, crtc); - - if (crtc_state->active_changed && crtc_state->active) { - struct neo_apple_crtc *neo_apple_crtc = to_apple_crtc(crtc); - neo_dcp_poweron(neo_apple_crtc->neo_dcp); - /* Force the CTM to be set on first swap */ - crtc_state->color_mgmt_changed = true; - } - - if (crtc_state->active) - neo_dcp_crtc_atomic_modeset(crtc, state); -} - -static void neo_apple_crtc_atomic_disable(struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct drm_crtc_state *crtc_state; - struct neo_apple_crtc *neo_apple_crtc = to_apple_crtc(crtc); - - crtc_state = drm_atomic_get_new_crtc_state(state, crtc); - - if (crtc_state->active_changed && !crtc_state->active) { - neo_dcp_poweroff(neo_apple_crtc->neo_dcp); - } - - if (crtc->state->event && !crtc->state->active) { - spin_lock_irq(&crtc->dev->event_lock); - drm_crtc_send_vblank_event(crtc, crtc->state->event); - spin_unlock_irq(&crtc->dev->event_lock); - - crtc->state->event = NULL; - } -} - -static void neo_apple_crtc_atomic_begin(struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct neo_apple_crtc *neo_apple_crtc = to_apple_crtc(crtc); - unsigned long flags; - - if (crtc->state->event) { - spin_lock_irqsave(&crtc->dev->event_lock, flags); - neo_apple_crtc->event = crtc->state->event; - spin_unlock_irqrestore(&crtc->dev->event_lock, flags); - crtc->state->event = NULL; - } -} - -static void neo_apple_crtc_cleanup(struct drm_crtc *crtc) -{ - drm_crtc_cleanup(crtc); - kfree(to_apple_crtc(crtc)); -} - -static int neo_apple_crtc_parse_crc_source(const char *source, bool *enabled) -{ - int ret = 0; - - if (!source) { - *enabled = false; - } else if (strcmp(source, "auto") == 0) { - *enabled = true; - } else { - *enabled = false; - ret = -EINVAL; - } - - return ret; -} - -static int neo_apple_crtc_set_crc_source(struct drm_crtc *crtc, const char *source) -{ - bool enabled = false; - - int ret = neo_apple_crtc_parse_crc_source(source, &enabled); - - if (!ret) - neo_dcp_set_crc(crtc, enabled); - - return ret; -} - -static int neo_apple_crtc_verify_crc_source(struct drm_crtc *crtc, - const char *source, - size_t *values_cnt) -{ - bool enabled; - - if (neo_apple_crtc_parse_crc_source(source, &enabled) < 0) { - pr_warn("dcp: Invalid CRC source name %s\n", source); - return -EINVAL; - } - - *values_cnt = 1; - - return 0; -} - -static const char * const neo_apple_crtc_crc_sources[] = {"auto"}; - -static const char *const * neo_apple_crtc_get_crc_sources(struct drm_crtc *crtc, - size_t *count) -{ - *count = ARRAY_SIZE(neo_apple_crtc_crc_sources); - return neo_apple_crtc_crc_sources; -} - -static const struct drm_crtc_funcs neo_apple_crtc_funcs = { - .atomic_destroy_state = drm_atomic_helper_crtc_destroy_state, - .atomic_duplicate_state = drm_atomic_helper_crtc_duplicate_state, - .destroy = neo_apple_crtc_cleanup, - .page_flip = drm_atomic_helper_page_flip, - .reset = drm_atomic_helper_crtc_reset, - .set_config = drm_atomic_helper_set_config, - .set_crc_source = neo_apple_crtc_set_crc_source, - .verify_crc_source = neo_apple_crtc_verify_crc_source, - .get_crc_sources = neo_apple_crtc_get_crc_sources, - -}; - -static const struct drm_mode_config_funcs apple_mode_config_funcs = { - .atomic_check = drm_atomic_helper_check, - .atomic_commit = drm_atomic_helper_commit, - .fb_create = drm_gem_fb_create, -}; - -static void apple_atomic_commit_tail(struct drm_atomic_state *state) -{ - struct drm_plane *plane; - struct drm_plane_state *old, *new; - int i; - - /* Retain displaced scanout even when the CRTC stays inactive. */ - for_each_oldnew_plane_in_state(state, plane, old, new, i) { - struct neo_apple_plane_state *apple_state = to_apple_plane_state(new); - struct neo_dcp_fb_reference *entry = apple_state->retirement; - - if (!entry) - continue; - - apple_state->retirement = NULL; - neo_dcp_retain_framebuffer(to_apple_crtc(old->crtc)->neo_dcp, entry); - } - - drm_atomic_helper_commit_tail_rpm(state); -} - -static const struct drm_mode_config_helper_funcs apple_mode_config_helpers = { - .atomic_commit_tail = apple_atomic_commit_tail, -}; - -static void appledrm_neo_connector_cleanup(struct drm_connector *connector) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(connector); - - drm_connector_cleanup(connector); - kfree(neo_apple_connector->color_elements.data); - kfree(neo_apple_connector->timing_elements.data); - kfree(neo_apple_connector->display_attributes.data); - kfree(neo_apple_connector->transport.data); - kfree(neo_apple_connector); -} - -static const struct drm_connector_funcs neo_apple_connector_funcs = { - .fill_modes = drm_helper_probe_single_connector_modes, - .destroy = appledrm_neo_connector_cleanup, - .reset = drm_atomic_helper_connector_reset, - .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state, - .atomic_destroy_state = drm_atomic_helper_connector_destroy_state, - .detect = neo_apple_connector_detect, - .debugfs_init = neo_apple_connector_debugfs_init, - .oob_hotplug_event = neo_apple_connector_oob_hotplug, -}; - -/* - * A Type-C connector is attached to every pipeline that can drive its port, so - * the atomic helper cannot pick one on its own. Follow the fabric's choice. - */ -static struct drm_encoder * -neo_apple_connector_atomic_best_encoder(struct drm_connector *conn, - struct drm_atomic_state *state) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(conn); - struct drm_connector_state *conn_state; - struct drm_encoder *encoder; - - /* - * Type-C tunnels are routed before the modeset and cannot move here. - * Reject stale topology and explicit assignments to another pipeline. - */ - if (neo_apple_connector->port_encoder) { - conn_state = drm_atomic_get_new_connector_state(state, conn); - if (!conn_state || !conn_state->crtc || - READ_ONCE(neo_apple_connector->neo_dcp) != - to_apple_crtc(conn_state->crtc)->neo_dcp) - return NULL; - return neo_apple_connector->port_encoder; - } - - drm_connector_for_each_possible_encoder(conn, encoder) - return encoder; - - return NULL; -} - -static const struct drm_connector_helper_funcs neo_apple_connector_helper_funcs = { - .get_modes = neo_dcp_get_modes, - .mode_valid = neo_dcp_mode_valid, - .atomic_best_encoder = neo_apple_connector_atomic_best_encoder, -}; - -static const struct drm_crtc_helper_funcs neo_apple_crtc_helper_funcs = { - .atomic_begin = neo_apple_crtc_atomic_begin, - .atomic_check = neo_dcp_crtc_atomic_check, - .atomic_flush = neo_dcp_flush, - .atomic_enable = neo_apple_crtc_atomic_enable, - .atomic_disable = neo_apple_crtc_atomic_disable, - .mode_fixup = neo_dcp_crtc_mode_fixup, -}; - -static int neo_apple_connector_create(struct drm_device *drm, - struct platform_device *neo_dcp, - struct neo_apple_crtc *crtc, - struct apple_encoder *encoder, - int connector_type, bool attach_fwnode) -{ - struct neo_apple_connector *connector; - int ret; - - connector = kzalloc_obj(*connector); - if (!connector) - return -ENOMEM; - - mutex_init(&connector->chunk_lock); - drm_connector_helper_add(&connector->base, - &neo_apple_connector_helper_funcs); - if (attach_fwnode) - connector->base.fwnode = fwnode_handle_get(neo_dcp->dev.fwnode); - - ret = drm_connector_init(drm, &connector->base, &neo_apple_connector_funcs, - connector_type); - if (ret) - goto err_free; - ret = drm_connector_attach_vrr_capable_property(&connector->base); - if (ret) - goto err_cleanup; - - connector->base.polled = DRM_CONNECTOR_POLL_HPD; - connector->connected = false; - connector->neo_dcp = neo_dcp; - INIT_WORK(&connector->hotplug_wq, neo_dcp_hotplug); - - ret = drm_connector_attach_encoder(&connector->base, &encoder->base); - if (ret) - goto err_cleanup; - neo_dcp_link(neo_dcp, crtc, connector); - - return 0; - -err_cleanup: - drm_connector_cleanup(&connector->base); - kfree(connector); - return ret; -err_free: - fwnode_handle_put(connector->base.fwnode); - kfree(connector); - return ret; -} - -static int apple_probe_per_dcp(struct device *dev, - struct drm_device *drm, - struct platform_device *neo_dcp, - int num, bool neo_dcp_ext, - struct drm_encoder **encoder, - u32 *crtc_mask) -{ - struct neo_apple_crtc *crtc; - struct apple_encoder *enc; - struct drm_plane *planes[DCP_MAX_PLANES]; - unsigned long *neo_iomfb_surfaces = neo_dcp_get_iomfb_surfaces(neo_dcp); - int ret; - int connector_type; - u32 surf; - int zpos = 0; - bool supports_l10r = !neo_dcp_fw_compat_is_12_x(neo_dcp); - struct neo_apple_dcp *neo_dcp_data = platform_get_drvdata(neo_dcp); - bool supports_xrgb2101010; - enum drm_plane_type plane_type; - - supports_xrgb2101010 = neo_dcp_data->fw_compat != DCP_FIRMWARE_H17P || - neo_dcp_data->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G; - if (bitmap_empty(neo_iomfb_surfaces, DCP_MAX_PLANES)) - return dev_err_probe(dev, -EINVAL, "No usable display surfaces\n"); - - for_each_set_bit(surf, neo_iomfb_surfaces, DCP_MAX_PLANES) { - plane_type = (zpos == 0) ? DRM_PLANE_TYPE_PRIMARY : DRM_PLANE_TYPE_OVERLAY; - planes[zpos] = neo_apple_plane_init(drm, 1U << num, surf, - supports_l10r, - supports_xrgb2101010, - plane_type); - if (IS_ERR(planes[zpos])) - return PTR_ERR(planes[zpos]); - - ret = drm_plane_create_zpos_immutable_property(planes[zpos], zpos); - if (ret) - return ret; - - zpos++; - } - - /* - * Even though we have an overlay plane, we cannot expose it to legacy - * userspace for cursors as we cannot make the same guarantees as ye olde - * hardware cursor planes such userspace would expect us to. Modern userspace - * knows what to do with overlays. - */ - crtc = kzalloc(sizeof(*crtc), GFP_KERNEL); - if (!crtc) - return -ENOMEM; - ret = drm_crtc_init_with_planes(drm, &crtc->base, planes[0], NULL, - &neo_apple_crtc_funcs, NULL); - if (ret) - return ret; - - drm_crtc_helper_add(&crtc->base, &neo_apple_crtc_helper_funcs); - drm_crtc_enable_color_mgmt(&crtc->base, 0, true, 0); - - crtc->neo_dcp = neo_dcp; - *crtc_mask = drm_crtc_mask(&crtc->base); - connector_type = neo_dcp_get_connector_type(neo_dcp); - - /* - * Type-C outputs are physical ports, not pipelines: several DCPs may be - * able to drive one port, and which one does can change at runtime. - * Those connectors are created per port once every pipeline is known, - * so a pipeline whose only output is Type-C gets no connector here -- - * and no encoder either, because the port's encoder spans every - * pipeline that can drive it. - */ - if (connector_type == DRM_MODE_CONNECTOR_USB) { - neo_dcp_link(neo_dcp, crtc, NULL); - *encoder = NULL; - return 0; - } - - enc = drmm_simple_encoder_alloc(drm, struct apple_encoder, base, - DRM_MODE_ENCODER_TMDS); - if (IS_ERR(enc)) - return PTR_ERR(enc); - enc->base.possible_crtcs = *crtc_mask; - *encoder = &enc->base; - - return neo_apple_connector_create(drm, neo_dcp, crtc, enc, connector_type, - neo_dcp_ext); -} - -/* - * Create a connector per physical Type-C port and attach it to every - * pipeline that can drive it. The M2 Pro/Max laptops append a second - * connector per port for a dock's second DP tunnel, preserving primary - * connector names. - */ -static int apple_probe_typec_ports(struct drm_device *drm, - struct platform_device **neo_dcp, - u32 *crtc_mask, int num_dcp) -{ - unsigned int idx, nr_ports = neo_dcp_typec_nr_ports(); - int i, ret; - - /* Keep the existing physical-port connector numbers stable. On the - * M2 Pro/Max laptops, append one more connector per port for a second - * USB4 DP tunnel. - */ - for (idx = 0; idx < nr_ports * (neo_dcp_typec_dual_stream() ? 2 : 1); - idx++) { - struct neo_apple_connector *connector; - struct apple_encoder *enc; - unsigned int port_idx = idx % nr_ports; - bool secondary = idx >= nr_ports; - u32 mask = 0; - - connector = kzalloc_obj(*connector); - if (!connector) - return -ENOMEM; - - mutex_init(&connector->chunk_lock); - drm_connector_helper_add(&connector->base, - &neo_apple_connector_helper_funcs); - - ret = drm_connector_init(drm, &connector->base, - &neo_apple_connector_funcs, - DRM_MODE_CONNECTOR_DisplayPort); - if (ret) { - kfree(connector); - return ret; - } - - ret = drm_connector_attach_vrr_capable_property(&connector->base); - if (ret) - return ret; - - connector->base.polled = DRM_CONNECTOR_POLL_HPD; - connector->connected = false; - connector->neo_dcp = NULL; - INIT_WORK(&connector->hotplug_wq, neo_dcp_hotplug); - - for (i = 0; i < num_dcp; i++) { - if (!neo_dcp_typec_port_has_candidate(port_idx, neo_dcp[i])) - continue; - /* - * A dock's second stream (DPIN1) always runs on the - * lowest-indexed Type-C-only pipeline (dcpext1). Fix the - * connector to it: compositors read possible_crtcs once - * and would otherwise pair it with a pipeline the stream - * can never be routed to. - */ - if (secondary) { - if (neo_dcp_is_typec_only(neo_dcp[i]) && - (!mask || crtc_mask[i] < mask)) - mask = crtc_mask[i]; - continue; - } - mask |= crtc_mask[i]; - } - - if (!mask) { - drm_warn(drm, "Type-C port %u has no display pipeline\n", - port_idx); - return -ENODEV; - } - - /* - * One encoder for the port, spanning every pipeline that can - * drive it. Attaching one encoder per pipeline would describe - * the same hardware, but userspace takes the CRTCs a connector - * can use to be what all of its encoders have in common, and - * single-pipeline encoders have nothing in common -- the port - * ends up with no usable CRTC and the monitor is left dark. - */ - enc = drmm_simple_encoder_alloc(drm, struct apple_encoder, base, - DRM_MODE_ENCODER_TMDS); - if (IS_ERR(enc)) - return PTR_ERR(enc); - enc->base.possible_crtcs = mask; - - ret = drm_connector_attach_encoder(&connector->base, &enc->base); - if (ret) - return ret; - connector->port_encoder = &enc->base; - connector->candidate_crtcs = mask; - - neo_dcp_typec_port_set_connector(port_idx, secondary, connector); - } - - return 0; -} - -static int apple_get_fb_resource(struct device *dev, const char *name, - struct resource *fb_r) -{ - int idx, ret = -ENODEV; - struct device_node *node; - - idx = of_property_match_string(dev->of_node, "memory-region-names", name); - - node = of_parse_phandle(dev->of_node, "memory-region", idx); - if (!node) { - dev_err(dev, "reserved-memory node '%s' not found\n", name); - return -ENODEV; - } - - if (!of_device_is_available(node)) { - dev_err(dev, "reserved-memory node '%s' is unavailable\n", name); - goto err; - } - - if (!of_device_is_compatible(node, "framebuffer")) { - dev_err(dev, "reserved-memory node '%s' is incompatible\n", - node->full_name); - goto err; - } - - ret = of_address_to_resource(node, 0, fb_r); - -err: - of_node_put(node); - return ret; -} - -static const struct of_device_id neo_apple_dcp_id_tbl[] = { - { .compatible = "apple,dcp" }, - { .compatible = "apple,dcpext" }, - {}, -}; - -/* - * The T8140 external display coprocessor needs firmware support this driver - * does not have yet. Leave it out of the display subsystem, so that an - * enabled node cannot hold back the internal panel's component master. - */ -static bool neo_apple_dcp_usable(struct device_node *np) -{ - return of_device_is_available(np) && - !of_device_is_compatible(np, "apple,t8140-dcpext"); -} - -static int neo_apple_drm_init_dcp(struct device *dev) -{ - struct neo_apple_drm_private *apple = dev_get_drvdata(dev); - struct platform_device *neo_dcp[MAX_COPROCESSORS]; - struct drm_encoder *encoder[MAX_COPROCESSORS]; - u32 crtc_mask[MAX_COPROCESSORS] = {}; - struct device_node *np; - u64 timeout; - int i, ret, num_dcp = 0; - - for_each_matching_node(np, neo_apple_dcp_id_tbl) { - bool neo_dcp_ext; - if (!neo_apple_dcp_usable(np)) - continue; - neo_dcp_ext = of_device_is_compatible(np, "apple,dcpext") || - of_property_present(np, "phys"); - - neo_dcp[num_dcp] = of_find_device_by_node(np); - if (!neo_dcp[num_dcp]) - continue; - - device_link_add(dev, &neo_dcp[num_dcp]->dev, DL_FLAG_AUTOREMOVE_SUPPLIER); - - ret = apple_probe_per_dcp(dev, &apple->drm, neo_dcp[num_dcp], - num_dcp, neo_dcp_ext, &encoder[num_dcp], - &crtc_mask[num_dcp]); - if (ret) - continue; - - num_dcp++; - } - - if (num_dcp < 1) - return -ENODEV; - - /* - * Build the Type-C port connectors before starting any pipeline: a - * pipeline can be handed a route as soon as the Type-C mux notifies, - * and it needs somewhere to report the display. - */ - ret = apple_probe_typec_ports(&apple->drm, neo_dcp, crtc_mask, num_dcp); - if (ret) - return ret; - - for (i = 0; i < num_dcp; i++) { - ret = neo_dcp_start(neo_dcp[i]); - if (ret) - dev_warn(dev, "DCP[%d] failed to start: %d\n", i, ret); - } - - /* - * Starting DPTX might take some time. - */ - timeout = get_jiffies_64() + msecs_to_jiffies(3000); - - for (i = 0; i < num_dcp; ++i) { - u64 jiffies = get_jiffies_64(); - u64 wait = time_after_eq64(jiffies, timeout) ? - 0 : - timeout - jiffies; - ret = neo_dcp_wait_ready(neo_dcp[i], wait); - /* There is nothing we can do if a dcp/dcpext does not boot - * (successfully). Ignoring it should not do any harm now. - * Needs to reevaluated when adding dcpext support. - */ - if (ret) - dev_warn(dev, "DCP[%d] not ready: %d\n", i, ret); - } - /* HACK: Wait for dcp* to settle before a modeset */ - msleep(100); - - return 0; -} - -static int neo_apple_drm_init(struct device *dev) -{ - struct neo_apple_drm_private *apple; - struct resource fb_r; - resource_size_t fb_size; - int ret; - - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(42)); - if (ret) - return ret; - - ret = apple_get_fb_resource(dev, "framebuffer", &fb_r); - if (ret) - return ret; - - apple = devm_drm_dev_alloc(dev, &neo_apple_drm_driver, - struct neo_apple_drm_private, drm); - if (IS_ERR(apple)) - return PTR_ERR(apple); - - dev_set_drvdata(dev, apple); - - ret = component_bind_all(dev, apple); - if (ret) - return ret; - - ret = drmm_mode_config_init(&apple->drm); - if (ret) - goto err_unbind; - - /* - * IOMFB::UPPipeDCP_H13P::verify_surfaces produces the error "plane - * requires a minimum of 32x32 for the source buffer" if smaller - */ - apple->drm.mode_config.min_width = 32; - apple->drm.mode_config.min_height = 32; - - /* - * TODO: this is the max framebuffer size not the maximal supported - * output resolution. DCP reports the maximal framebuffer size take it - * from there. - * Hardcode it for now to the M1 Max DCP reported 'MaxSrcBufferWidth' - * and 'MaxSrcBufferHeight' of 16384. - */ - apple->drm.mode_config.max_width = 16384; - apple->drm.mode_config.max_height = 16384; - - apple->drm.mode_config.funcs = &apple_mode_config_funcs; - apple->drm.mode_config.helper_private = &apple_mode_config_helpers; - - ret = neo_apple_drm_init_dcp(dev); - if (ret) - goto err_unbind; - - drm_mode_config_reset(&apple->drm); - - fb_size = fb_r.end - fb_r.start + 1; - ret = aperture_remove_conflicting_devices(fb_r.start, fb_size, - neo_apple_drm_driver.name); - if (ret) { - dev_err(dev, "Failed remove fb: %d\n", ret); - goto err_unbind; - } - - ret = drm_dev_register(&apple->drm, 0); - if (ret) - goto err_unbind; - - /* the fabric keeps order only once registered: catch up on probe */ - neo_dcp_typec_reorder(); - - drm_client_setup_with_fourcc(&apple->drm, DRM_FORMAT_XRGB8888); - - return 0; - -err_unbind: - component_unbind_all(dev, NULL); - return ret; -} - -static void neo_apple_drm_uninit(struct device *dev) -{ - struct neo_apple_drm_private *apple = dev_get_drvdata(dev); - - drm_dev_unplug(&apple->drm); - drm_atomic_helper_shutdown(&apple->drm); - - component_unbind_all(dev, NULL); - - dev_set_drvdata(dev, NULL); -} - -static int neo_apple_drm_bind(struct device *dev) -{ - return neo_apple_drm_init(dev); -} - -static void neo_apple_drm_unbind(struct device *dev) -{ - neo_apple_drm_uninit(dev); -} - -const struct component_master_ops neo_apple_drm_ops = { - .bind = neo_apple_drm_bind, - .unbind = neo_apple_drm_unbind, -}; - -static int add_dcp_components(struct device *dev, - struct component_match **matchptr) -{ - struct device_node *np, *endpoint, *port; - int num = 0; - - for_each_matching_node(np, neo_apple_dcp_id_tbl) { - if (neo_apple_dcp_usable(np)) { - drm_of_component_match_add(dev, matchptr, - component_compare_of, np); - num++; - for_each_endpoint_of_node(np, endpoint) { - port = of_graph_get_remote_port_parent(endpoint); - if (!port) - continue; - -#if !IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) - if (of_device_is_compatible(port, "apple,dpaudio")) { - of_node_put(port); - continue; - } -#endif - - /* - * The ATC phy driver is not part of the component - * collection for the Apple display-subsystem so - * ignore it here. - */ - if (of_device_is_compatible(port, "apple,t8103-atcphy")) { - of_node_put(port); - continue; - } - - if (of_device_is_available(port)) - drm_of_component_match_add(dev, matchptr, - component_compare_of, - port); - of_node_put(port); - } - } - } - - return num; -} - -static int apple_platform_probe(struct platform_device *pdev) -{ - struct device *mdev = &pdev->dev; - struct component_match *match = NULL; - int num_dcp; - - /* add DCP components, handle less than 1 as probe error */ - num_dcp = add_dcp_components(mdev, &match); - if (num_dcp < 1) - return -ENODEV; - - return component_master_add_with_match(mdev, &neo_apple_drm_ops, match); -} - -static void apple_platform_remove(struct platform_device *pdev) -{ - component_master_del(&pdev->dev, &neo_apple_drm_ops); -} - -static const struct of_device_id of_match[] = { - { .compatible = "apple,t8140-display-subsystem" }, - {} -}; -MODULE_DEVICE_TABLE(of, of_match); - -#ifdef CONFIG_PM_SLEEP -static int apple_platform_suspend(struct device *dev) -{ - struct neo_apple_drm_private *apple = dev_get_drvdata(dev); - - if (apple) - return drm_mode_config_helper_suspend(&apple->drm); - - return 0; -} - -static int apple_platform_resume(struct device *dev) -{ - struct neo_apple_drm_private *apple = dev_get_drvdata(dev); - - if (apple) - drm_mode_config_helper_resume(&apple->drm); - - return 0; -} - -static const struct dev_pm_ops apple_platform_pm_ops = { - .suspend = apple_platform_suspend, - .resume = apple_platform_resume, -}; -#endif - -static struct platform_driver apple_platform_driver = { - .driver = { - .name = "apple-drm-neo", - .of_match_table = of_match, -#ifdef CONFIG_PM_SLEEP - .pm = &apple_platform_pm_ops, -#endif - }, - .probe = apple_platform_probe, - .remove = apple_platform_remove, -}; - - - -static int __init appledrm_neo_register(void) -{ - int ret; - - /* The generic audio compatible belongs to the legacy driver on other SoCs. */ - if (!of_machine_is_compatible("apple,t8140") || drm_firmware_drivers_only()) - return -ENODEV; - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) - ret = neo_dcp_audio_register(); - if (ret) - return ret; -#endif - ret = neo_dcp_register(); - if (ret) - goto unregister_audio; - ret = platform_driver_register(&apple_platform_driver); - if (!ret) - return 0; - - neo_dcp_unregister(); -unregister_audio: -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) - neo_dcp_audio_unregister(); -#endif - return ret; -} - -static void __exit appledrm_neo_unregister(void) -{ -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) - neo_dcp_audio_unregister(); -#endif - neo_dcp_unregister(); - platform_driver_unregister(&apple_platform_driver); -} - -module_init(appledrm_neo_register); -module_exit(appledrm_neo_unregister); - -MODULE_AUTHOR("Asahi Linux contributors"); -MODULE_DESCRIPTION(DRIVER_DESC); -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple-neo/audio.c b/drivers/gpu/drm/apple-neo/audio.c deleted file mode 100644 index f9eea5e6fa7b07..00000000000000 --- a/drivers/gpu/drm/apple-neo/audio.c +++ /dev/null @@ -1,815 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * DCP Audio Bits - * - * Copyright (C) The Asahi Linux Contributors - * - * TODO: - * - figure some nice identification of the sound card (in case - * there's many DCP instances) - */ - -#define DEBUG - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "av.h" -#include "dcp.h" -#include "audio.h" -#include "parser.h" - -#define DCPAUD_ELEMENTS_MAXSIZE 16384 -#define DCPAUD_PRODUCTATTRS_MAXSIZE 1024 - -struct neo_dcp_audio { - struct device *dev; - struct device *neo_dcp_dev; - struct device *dma_dev; - struct device_link *dma_link; - struct dma_chan *chan; - struct snd_card *card; - struct snd_jack *jack; - struct snd_pcm_substream *substream; - unsigned int open_cookie; - bool open_unplugged; - - struct mutex data_lock; - bool neo_dcp_connected; /// dcp status keep for delayed initialization - bool connected; - unsigned int connection_cookie; - - struct snd_pcm_chmap_elem selected_chmap; - struct neo_dcp_sound_cookie selected_cookie; - void *elements; - void *productattrs; - - struct snd_pcm_chmap *chmap_info; -}; - -static const struct snd_pcm_hardware neo_dcp_pcm_hw = { - .info = SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID | - SNDRV_PCM_INFO_INTERLEAVED, - .formats = SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S20_LE | - SNDRV_PCM_FMTBIT_S24_LE | SNDRV_PCM_FMTBIT_S32_LE, - .rates = SNDRV_PCM_RATE_CONTINUOUS, - .rate_min = 0, - .rate_max = UINT_MAX, - .channels_min = 1, - .channels_max = 16, - .buffer_bytes_max = SIZE_MAX, - .period_bytes_min = 4096, /* TODO */ - .period_bytes_max = SIZE_MAX, - .periods_min = 2, - .periods_max = UINT_MAX, -}; - -/* - * Without a sink the DCP has no audio elements to derive constraints from. - * Offer a plain stereo format so that the PCM can still be opened and - * configured, as HDMI PCMs on other platforms can. Userspace (PipeWire, - * PulseAudio) probes the PCM once when the card appears and drops the - * output for good if that fails, so a monitor attached later would never - * show up. Starting the stream still fails until a sink is connected. - */ -static const struct snd_pcm_hardware neo_dcp_pcm_hw_unplugged = { - .info = SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID | - SNDRV_PCM_INFO_INTERLEAVED, - .formats = SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S24_LE | - SNDRV_PCM_FMTBIT_S32_LE, - .rates = SNDRV_PCM_RATE_32000 | SNDRV_PCM_RATE_44100 | - SNDRV_PCM_RATE_48000, - .rate_min = 32000, - .rate_max = 48000, - .channels_min = 2, - .channels_max = 2, - .buffer_bytes_max = SIZE_MAX, - .period_bytes_min = 4096, - .period_bytes_max = SIZE_MAX, - .periods_min = 2, - .periods_max = UINT_MAX, -}; - -static int neo_dcpaud_read_remote_info(struct neo_dcp_audio *neo_dcpaud) -{ - int ret; - - ret = neo_dcp_audiosrv_get_elements(neo_dcpaud->neo_dcp_dev, neo_dcpaud->elements, - DCPAUD_ELEMENTS_MAXSIZE); - if (ret < 0) - return ret; - - ret = neo_dcp_audiosrv_get_product_attrs(neo_dcpaud->neo_dcp_dev, neo_dcpaud->productattrs, - DCPAUD_PRODUCTATTRS_MAXSIZE); - if (ret < 0) - return ret; - - return 0; -} - -static int neo_dcpaud_interval_bitmask(struct snd_interval *i, - unsigned int mask) -{ - struct snd_interval range; - if (!mask) - return -EINVAL; - - snd_interval_any(&range); - range.min = __ffs(mask); - range.max = __fls(mask); - return snd_interval_refine(i, &range); -} - -extern const struct snd_pcm_hw_constraint_list snd_pcm_known_rates; - -static void neo_dcpaud_fill_fmt_sieve(struct snd_pcm_hw_params *params, - struct neo_dcp_sound_format_mask *sieve) -{ - struct snd_interval *c = hw_param_interval(params, - SNDRV_PCM_HW_PARAM_CHANNELS); - struct snd_interval *r = hw_param_interval(params, - SNDRV_PCM_HW_PARAM_RATE); - struct snd_mask *f = hw_param_mask(params, - SNDRV_PCM_HW_PARAM_FORMAT); - int i; - - sieve->nchans = GENMASK(c->max, c->min); - sieve->formats = f->bits[0] | ((u64) f->bits[1]) << 32; /* TODO: don't open-code */ - - for (i = 0; i < snd_pcm_known_rates.count; i++) { - unsigned int rate = snd_pcm_known_rates.list[i]; - - if (snd_interval_test(r, rate)) - sieve->rates |= 1u << i; - } -} - -static void neo_dcpaud_consult_elements(struct neo_dcp_audio *neo_dcpaud, - struct snd_pcm_hw_params *params, - struct neo_dcp_sound_format_mask *hits) -{ - struct neo_dcp_sound_format_mask sieve; - struct neo_dcp_parse_ctx elements = { - .neo_dcp = dev_get_drvdata(neo_dcpaud->neo_dcp_dev), - .blob = neo_dcpaud->elements + 4, - .len = DCPAUD_ELEMENTS_MAXSIZE - 4, - .pos = 0, - }; - - neo_dcpaud_fill_fmt_sieve(params, &sieve); - dev_dbg(neo_dcpaud->dev, "elements in: %llx %x %x\n", sieve.formats, sieve.nchans, sieve.rates); - neo_parse_sound_constraints(&elements, &sieve, hits); - dev_dbg(neo_dcpaud->dev, "elements out: %llx %x %x\n", hits->formats, hits->nchans, hits->rates); -} - -static int neo_dcpaud_select_cookie(struct neo_dcp_audio *neo_dcpaud, - struct snd_pcm_hw_params *params) -{ - struct neo_dcp_sound_format_mask sieve; - struct neo_dcp_parse_ctx elements = { - .neo_dcp = dev_get_drvdata(neo_dcpaud->neo_dcp_dev), - .blob = neo_dcpaud->elements + 4, - .len = DCPAUD_ELEMENTS_MAXSIZE - 4, - .pos = 0, - }; - - neo_dcpaud_fill_fmt_sieve(params, &sieve); - return neo_parse_sound_mode(&elements, &sieve, &neo_dcpaud->selected_chmap, - &neo_dcpaud->selected_cookie); -} - -static int neo_dcpaud_rule_channels(struct snd_pcm_hw_params *params, - struct snd_pcm_hw_rule *rule) -{ - struct neo_dcp_audio *neo_dcpaud = rule->private; - struct snd_interval *c = hw_param_interval(params, - SNDRV_PCM_HW_PARAM_CHANNELS); - struct neo_dcp_sound_format_mask hits = {0, 0, 0}; - - neo_dcpaud_consult_elements(neo_dcpaud, params, &hits); - - return neo_dcpaud_interval_bitmask(c, hits.nchans); -} - -static int neo_dcpaud_refine_fmt_mask(struct snd_mask *m, u64 mask) -{ - struct snd_mask mask_mask; - - if (!mask) - return -EINVAL; - mask_mask.bits[0] = mask; - mask_mask.bits[1] = mask >> 32; - - return snd_mask_refine(m, &mask_mask); -} - -static int neo_dcpaud_rule_format(struct snd_pcm_hw_params *params, - struct snd_pcm_hw_rule *rule) -{ - struct neo_dcp_audio *neo_dcpaud = rule->private; - struct snd_mask *f = hw_param_mask(params, - SNDRV_PCM_HW_PARAM_FORMAT); - struct neo_dcp_sound_format_mask hits; - - neo_dcpaud_consult_elements(neo_dcpaud, params, &hits); - - return neo_dcpaud_refine_fmt_mask(f, hits.formats); -} - -static int neo_dcpaud_rule_rate(struct snd_pcm_hw_params *params, - struct snd_pcm_hw_rule *rule) -{ - struct neo_dcp_audio *neo_dcpaud = rule->private; - struct snd_interval *r = hw_param_interval(params, - SNDRV_PCM_HW_PARAM_RATE); - struct neo_dcp_sound_format_mask hits; - - neo_dcpaud_consult_elements(neo_dcpaud, params, &hits); - - return snd_interval_rate_bits(r, hits.rates); -} - -static int neo_dcpaud_init_dma(struct neo_dcp_audio *neo_dcpaud) -{ - struct dma_chan *chan; - if (neo_dcpaud->chan) - return 0; - - chan = of_dma_request_slave_channel(neo_dcpaud->dev->of_node, "tx"); - /* squelch dma channel request errors, the driver will try again alter */ - if (!chan) { - dev_warn(neo_dcpaud->dev, "audio TX DMA channel request failed\n"); - return -ENXIO; - } else if (chan == ERR_PTR(-EPROBE_DEFER)) { - dev_info(neo_dcpaud->dev, "audio TX DMA channel is not ready yet\n"); - return -ENXIO; - } else if (IS_ERR(chan)) { - dev_warn(neo_dcpaud->dev, "audio TX DMA channel request failed: %ld\n", PTR_ERR(chan)); - return PTR_ERR(chan); - } - neo_dcpaud->chan = chan; - - snd_pcm_set_managed_buffer(neo_dcpaud->substream, SNDRV_DMA_TYPE_DEV_IRAM, - neo_dcpaud->chan->device->dev, 1024 * 1024, - SIZE_MAX); - - return 0; -} - -static int neo_dcp_pcm_open(struct snd_pcm_substream *substream) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - struct snd_dmaengine_dai_dma_data dma_data = { - .flags = SND_DMAENGINE_PCM_DAI_FLAG_PACK, - }; - struct snd_pcm_hardware hw; - int ret; - - mutex_lock(&neo_dcpaud->data_lock); - ret = neo_dcpaud_init_dma(neo_dcpaud); - if (ret < 0) { - mutex_unlock(&neo_dcpaud->data_lock); - return ret; - } - - neo_dcpaud->open_unplugged = !neo_dcpaud->connected; - neo_dcpaud->open_cookie = neo_dcpaud->connection_cookie; - mutex_unlock(&neo_dcpaud->data_lock); - - if (neo_dcpaud->open_unplugged) { - hw = neo_dcp_pcm_hw_unplugged; - goto refine; - } - - ret = neo_dcpaud_read_remote_info(neo_dcpaud); - if (ret < 0) - return ret; - - snd_pcm_hw_rule_add(substream->runtime, 0, SNDRV_PCM_HW_PARAM_FORMAT, - neo_dcpaud_rule_format, neo_dcpaud, - SNDRV_PCM_HW_PARAM_CHANNELS, SNDRV_PCM_HW_PARAM_RATE, -1); - snd_pcm_hw_rule_add(substream->runtime, 0, SNDRV_PCM_HW_PARAM_CHANNELS, - neo_dcpaud_rule_channels, neo_dcpaud, - SNDRV_PCM_HW_PARAM_FORMAT, SNDRV_PCM_HW_PARAM_RATE, -1); - snd_pcm_hw_rule_add(substream->runtime, 0, SNDRV_PCM_HW_PARAM_RATE, - neo_dcpaud_rule_rate, neo_dcpaud, - SNDRV_PCM_HW_PARAM_FORMAT, SNDRV_PCM_HW_PARAM_CHANNELS, -1); - - hw = neo_dcp_pcm_hw; -refine: - hw.info = SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID | - SNDRV_PCM_INFO_INTERLEAVED; - hw.periods_min = 2; - hw.periods_max = UINT_MAX; - hw.period_bytes_min = 256; - hw.period_bytes_max = SIZE_MAX; // TODO dma_get_max_seg_size(dma_dev); - hw.buffer_bytes_max = SIZE_MAX; - hw.fifo_size = 16; - ret = snd_dmaengine_pcm_refine_runtime_hwparams(substream, &dma_data, - &hw, neo_dcpaud->chan); - if (ret) - return ret; - substream->runtime->hw = hw; - - return snd_dmaengine_pcm_open(substream, neo_dcpaud->chan); -} - -static int neo_dcp_pcm_close(struct snd_pcm_substream *substream) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - neo_dcpaud->selected_chmap.channels = 0; - neo_dcpaud->open_unplugged = false; - - return snd_dmaengine_pcm_close(substream); -} - -static int neo_dcpaud_connection_up(struct neo_dcp_audio *neo_dcpaud) -{ - bool ret; - mutex_lock(&neo_dcpaud->data_lock); - ret = neo_dcpaud->connected && - neo_dcpaud->open_cookie == neo_dcpaud->connection_cookie; - mutex_unlock(&neo_dcpaud->data_lock); - return ret; -} - -static int neo_dcp_pcm_hw_params(struct snd_pcm_substream *substream, - struct snd_pcm_hw_params *params) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - struct dma_slave_config slave_config; - struct dma_chan *chan = snd_dmaengine_pcm_get_chan(substream); - int ret; - - if (neo_dcpaud->open_unplugged) - return 0; - - if (!neo_dcpaud_connection_up(neo_dcpaud)) - return -ENXIO; - - ret = neo_dcpaud_select_cookie(neo_dcpaud, params); - if (ret < 0) - return ret; - if (!ret) - return -EINVAL; - - memset(&slave_config, 0, sizeof(slave_config)); - ret = snd_hwparams_to_dma_slave_config(substream, params, &slave_config); - dev_info(neo_dcpaud->dev, "snd_hwparams_to_dma_slave_config: %d\n", ret); - if (ret < 0) - return ret; - - slave_config.direction = DMA_MEM_TO_DEV; - /* - * The data entry from the DMA controller to the DPA peripheral - * is 32-bit wide no matter the actual sample size. - */ - slave_config.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; - - ret = dmaengine_slave_config(chan, &slave_config); - dev_info(neo_dcpaud->dev, "dmaengine_slave_config: %d\n", ret); - return ret; -} - -static int neo_dcp_pcm_hw_free(struct snd_pcm_substream *substream) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - - if (!neo_dcpaud_connection_up(neo_dcpaud)) - return 0; - - return neo_dcp_audiosrv_unprepare(neo_dcpaud->neo_dcp_dev); -} - -static int neo_dcp_pcm_prepare(struct snd_pcm_substream *substream) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - - /* alsa-lib prepares right after hw_params; see dcp_pcm_hw_unplugged */ - if (neo_dcpaud->open_unplugged) - return 0; - - if (!neo_dcpaud_connection_up(neo_dcpaud)) - return -ENXIO; - - return neo_dcp_audiosrv_prepare(neo_dcpaud->neo_dcp_dev, - &neo_dcpaud->selected_cookie); -} - -static int neo_dcp_pcm_trigger(struct snd_pcm_substream *substream, int cmd) -{ - struct neo_dcp_audio *neo_dcpaud = substream->pcm->private_data; - int ret; - - switch (cmd) { - case SNDRV_PCM_TRIGGER_START: - case SNDRV_PCM_TRIGGER_RESUME: - if (!neo_dcpaud_connection_up(neo_dcpaud)) - return -ENXIO; - - WARN_ON(pm_runtime_get_sync(neo_dcpaud->dev) < 0); - ret = neo_dcp_audiosrv_startlink(neo_dcpaud->neo_dcp_dev, - &neo_dcpaud->selected_cookie); - if (ret < 0) - return ret; - break; - - case SNDRV_PCM_TRIGGER_STOP: - case SNDRV_PCM_TRIGGER_SUSPEND: - break; - - default: - return -EINVAL; - } - - ret = snd_dmaengine_pcm_trigger(substream, cmd); - if (ret < 0) - return ret; - - switch (cmd) { - case SNDRV_PCM_TRIGGER_START: - case SNDRV_PCM_TRIGGER_RESUME: - break; - - case SNDRV_PCM_TRIGGER_STOP: - case SNDRV_PCM_TRIGGER_SUSPEND: - ret = neo_dcp_audiosrv_stoplink(neo_dcpaud->neo_dcp_dev); - pm_runtime_mark_last_busy(neo_dcpaud->dev); - __pm_runtime_put_autosuspend(neo_dcpaud->dev); - if (ret < 0) - return ret; - break; - } - - return 0; -} - -struct snd_pcm_ops neo_dcp_playback_ops = { - .open = neo_dcp_pcm_open, - .close = neo_dcp_pcm_close, - .hw_params = neo_dcp_pcm_hw_params, - .hw_free = neo_dcp_pcm_hw_free, - .prepare = neo_dcp_pcm_prepare, - .trigger = neo_dcp_pcm_trigger, - .pointer = snd_dmaengine_pcm_pointer, -}; - -// Transitional workaround: for the chmap control TLV, advertise options -// copied from hdmi-codec.c -#include "hdmi-codec-chmap.h" - -static int neo_dcpaud_chmap_ctl_get(struct snd_kcontrol *kcontrol, - struct snd_ctl_elem_value *ucontrol) -{ - struct snd_pcm_chmap *info = snd_kcontrol_chip(kcontrol); - struct neo_dcp_audio *neo_dcpaud = info->private_data; - unsigned int i; - - for (i = 0; i < info->max_channels; i++) - ucontrol->value.integer.value[i] = \ - (i < neo_dcpaud->selected_chmap.channels) ? - neo_dcpaud->selected_chmap.map[i] : SNDRV_CHMAP_UNKNOWN; - - return 0; -} - - -static int neo_dcpaud_create_chmap_ctl(struct neo_dcp_audio *neo_dcpaud) -{ - struct snd_pcm *pcm = neo_dcpaud->substream->pcm; - struct snd_pcm_chmap *chmap_info; - int ret; - - ret = snd_pcm_add_chmap_ctls(pcm, SNDRV_PCM_STREAM_PLAYBACK, NULL, - neo_dcp_pcm_hw.channels_max, 0, &chmap_info); - if (ret < 0) - return ret; - - chmap_info->kctl->get = neo_dcpaud_chmap_ctl_get; - chmap_info->chmap = hdmi_codec_8ch_chmaps; - chmap_info->private_data = neo_dcpaud; - - return 0; -} - -static int neo_dcpaud_create_pcm(struct neo_dcp_audio *neo_dcpaud) -{ - struct snd_card *card = neo_dcpaud->card; - struct snd_pcm *pcm; - int ret; - -#define NUM_PLAYBACK 1 -#define NUM_CAPTURE 0 - - ret = snd_pcm_new(card, card->shortname, 0, NUM_PLAYBACK, NUM_CAPTURE, &pcm); - if (ret) - return ret; - - snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &neo_dcp_playback_ops); - neo_dcpaud->substream = pcm->streams[SNDRV_PCM_STREAM_PLAYBACK].substream; - pcm->nonatomic = true; - pcm->private_data = neo_dcpaud; - strscpy(pcm->name, card->shortname, sizeof(pcm->name)); - - return 0; -} - -/* expects to be called with data_lock locked and unlocks it */ -static void neo_dcpaud_report_hotplug(struct neo_dcp_audio *neo_dcpaud, bool connected) -{ - struct snd_pcm_substream *substream = neo_dcpaud->substream; - - if (!neo_dcpaud->card || neo_dcpaud->connected == connected) { - mutex_unlock(&neo_dcpaud->data_lock); - return; - } - - neo_dcpaud->connected = connected; - if (connected) - neo_dcpaud->connection_cookie++; - mutex_unlock(&neo_dcpaud->data_lock); - - snd_jack_report(neo_dcpaud->jack, connected ? SND_JACK_AVOUT : 0); - - if (!connected) { - snd_pcm_stream_lock(substream); - if (substream->runtime) - snd_pcm_stop(substream, SNDRV_PCM_STATE_DISCONNECTED); - snd_pcm_stream_unlock(substream); - } -} - -static int neo_dcpaud_create_jack(struct neo_dcp_audio *neo_dcpaud) -{ - struct snd_card *card = neo_dcpaud->card; - - return snd_jack_new(card, "HDMI/DP", SND_JACK_AVOUT, - &neo_dcpaud->jack, true, false); -} - -static void neo_dcpaud_set_card_names(struct neo_dcp_audio *neo_dcpaud) -{ - struct snd_card *card = neo_dcpaud->card; - - strscpy(card->driver, "apple_dcp", sizeof(card->driver)); - strscpy(card->longname, "Apple DisplayPort", sizeof(card->longname)); - strscpy(card->shortname, "Apple DisplayPort", sizeof(card->shortname)); -} - -#ifdef CONFIG_SND_DEBUG -static void neo_dcpaud_expose_debugfs_blob(struct neo_dcp_audio *neo_dcpaud, const char *name, void *base, size_t size) -{ - struct debugfs_blob_wrapper *wrapper; - wrapper = devm_kzalloc(neo_dcpaud->dev, sizeof(*wrapper), GFP_KERNEL); - if (!wrapper) - return; - wrapper->data = base; - wrapper->size = size; - debugfs_create_blob(name, 0600, neo_dcpaud->card->debugfs_root, wrapper); -} -#else -static void neo_dcpaud_expose_debugfs_blob(struct neo_dcp_audio *neo_dcpaud, const char *name, void *base, size_t size) {} -#endif - -extern bool neo_hdmi_audio; - -static int neo_dcpaud_init_snd_card(struct neo_dcp_audio *neo_dcpaud) -{ - int ret; - if (!neo_hdmi_audio) - return -ENODEV; - - - ret = snd_card_new(neo_dcpaud->dev, SNDRV_DEFAULT_IDX1, SNDRV_DEFAULT_STR1, - THIS_MODULE, 0, &neo_dcpaud->card); - if (ret) - return ret; - - neo_dcpaud_set_card_names(neo_dcpaud); - - ret = neo_dcpaud_create_pcm(neo_dcpaud); - if (ret) - goto err_free_card; - - ret = neo_dcpaud_create_chmap_ctl(neo_dcpaud); - if (ret) - goto err_free_card; - - ret = neo_dcpaud_create_jack(neo_dcpaud); - if (ret) - goto err_free_card; - - ret = snd_card_register(neo_dcpaud->card); - if (ret) - goto err_free_card; - - return 0; -err_free_card: - dev_warn(neo_dcpaud->dev, "Failed to initialize sound card: %d\n", ret); - snd_card_free(neo_dcpaud->card); - neo_dcpaud->card = NULL; - return ret; -} - -void neo_dcpaud_connect(struct platform_device *pdev, bool connected) -{ - struct neo_dcp_audio *neo_dcpaud = platform_get_drvdata(pdev); - - mutex_lock(&neo_dcpaud->data_lock); - - neo_dcpaud_report_hotplug(neo_dcpaud, connected); -} - -void neo_dcpaud_disconnect(struct platform_device *pdev) -{ - struct neo_dcp_audio *neo_dcpaud = platform_get_drvdata(pdev); - - mutex_lock(&neo_dcpaud->data_lock); - - neo_dcpaud_report_hotplug(neo_dcpaud, false); -} - -static int neo_dcpaud_comp_bind(struct device *dev, struct device *main, void *data) -{ - struct neo_dcp_audio *neo_dcpaud = dev_get_drvdata(dev); - struct device_node *endpoint, *neo_dcp_node = NULL; - struct platform_device *neo_dcp_pdev, *dma_pdev; - struct of_phandle_args dma_spec; - int index; - int ret; - - pm_runtime_get_noresume(dev); - pm_runtime_set_active(dev); - - ret = devm_pm_runtime_enable(dev); - if (ret) - return dev_err_probe(dev, ret, "Failed to enable runtime PM: %d\n", ret); - - /* find linked DCP instance */ - endpoint = of_graph_get_endpoint_by_regs(dev->of_node, 0, 0); - if (endpoint) { - neo_dcp_node = of_graph_get_remote_port_parent(endpoint); - of_node_put(endpoint); - } - if (!neo_dcp_node || !of_device_is_available(neo_dcp_node)) { - of_node_put(neo_dcp_node); - dev_info(dev, "No audio support\n"); - goto rpm_put; - } - - index = of_property_match_string(dev->of_node, "dma-names", "tx"); - if (index < 0) { - dev_err(dev, "No dma-names property\n"); - goto rpm_put; - } - - if (of_parse_phandle_with_args(dev->of_node, "dmas", "#dma-cells", index, - &dma_spec) || !dma_spec.np) { - dev_err(dev, "Failed to parse dmas property\n"); - goto rpm_put; - } - - neo_dcp_pdev = of_find_device_by_node(neo_dcp_node); - of_node_put(neo_dcp_node); - if (!neo_dcp_pdev) { - dev_info(dev, "No DP/HDMI audio device, dcp not ready\n"); - goto rpm_put; - } - neo_dcpaud->neo_dcp_dev = &neo_dcp_pdev->dev; - - dma_pdev = of_find_device_by_node(dma_spec.np); - of_node_put(dma_spec.np); - if (!dma_pdev) { - dev_info(dev, "No DMA device\n"); - goto rpm_put; - } - neo_dcpaud->dma_dev = &dma_pdev->dev; - - neo_dcpaud->dma_link = device_link_add(dev, neo_dcpaud->dma_dev, - DL_FLAG_PM_RUNTIME | - DL_FLAG_RPM_ACTIVE | - DL_FLAG_STATELESS); - - /* ignore errors to prevent audio issues affecting the display side */ - ret = neo_dcpaud_init_snd_card(neo_dcpaud); - - if (!ret) { - neo_dcpaud_expose_debugfs_blob(neo_dcpaud, "selected_cookie", &neo_dcpaud->selected_cookie, - sizeof(neo_dcpaud->selected_cookie)); - neo_dcpaud_expose_debugfs_blob(neo_dcpaud, "elements", neo_dcpaud->elements, - DCPAUD_ELEMENTS_MAXSIZE); - neo_dcpaud_expose_debugfs_blob(neo_dcpaud, "product_attrs", neo_dcpaud->productattrs, - DCPAUD_PRODUCTATTRS_MAXSIZE); - } - -rpm_put: - pm_runtime_put(dev); - - return 0; -} - -static void neo_dcpaud_comp_unbind(struct device *dev, struct device *main, - void *data) -{ - struct neo_dcp_audio *neo_dcpaud = dev_get_drvdata(dev); - - /* snd_card_free_when_closed() checks for NULL */ - snd_card_free_when_closed(neo_dcpaud->card); - - if (neo_dcpaud->dma_link) - device_link_del(neo_dcpaud->dma_link); -} - -static const struct component_ops neo_dcpaud_comp_ops = { - .bind = neo_dcpaud_comp_bind, - .unbind = neo_dcpaud_comp_unbind, -}; - -static int neo_dcpaud_probe(struct platform_device *pdev) -{ - struct neo_dcp_audio *neo_dcpaud; - - neo_dcpaud = devm_kzalloc(&pdev->dev, sizeof(*neo_dcpaud), GFP_KERNEL); - if (!neo_dcpaud) - return -ENOMEM; - - neo_dcpaud->elements = devm_kzalloc(&pdev->dev, DCPAUD_ELEMENTS_MAXSIZE, - GFP_KERNEL); - if (!neo_dcpaud->elements) - return -ENOMEM; - - neo_dcpaud->productattrs = devm_kzalloc(&pdev->dev, DCPAUD_PRODUCTATTRS_MAXSIZE, - GFP_KERNEL); - if (!neo_dcpaud->productattrs) - return -ENOMEM; - - neo_dcpaud->dev = &pdev->dev; - mutex_init(&neo_dcpaud->data_lock); - platform_set_drvdata(pdev, neo_dcpaud); - - return component_add(&pdev->dev, &neo_dcpaud_comp_ops); -} - -static void neo_dcpaud_remove(struct platform_device *pdev) -{ - component_del(&pdev->dev, &neo_dcpaud_comp_ops); -} - -static void neo_dcpaud_shutdown(struct platform_device *pdev) -{ - component_del(&pdev->dev, &neo_dcpaud_comp_ops); -} - -static __maybe_unused int neo_dcpaud_suspend(struct device *dev) -{ - /* - * Using snd_power_change_state() does not work since the sound card - * is what resumes runtime PM. - */ - - return 0; -} - -static __maybe_unused int neo_dcpaud_resume(struct device *dev) -{ - return 0; -} - -static DEFINE_RUNTIME_DEV_PM_OPS(neo_dcpaud_pm_ops, neo_dcpaud_suspend, neo_dcpaud_resume, NULL); - -static const struct of_device_id neo_dcpaud_of_match[] = { - { .compatible = "apple,dpaudio" }, - {} -}; - -static struct platform_driver neo_dcpaud_driver = { - .driver = { - .name = "dcp-dp-audio-neo", - .of_match_table = neo_dcpaud_of_match, - .pm = pm_ptr(&neo_dcpaud_pm_ops), - }, - .probe = neo_dcpaud_probe, - .remove = neo_dcpaud_remove, - .shutdown = neo_dcpaud_shutdown, -}; - -int __init neo_dcp_audio_register(void) -{ - return platform_driver_register(&neo_dcpaud_driver); -} - -void neo_dcp_audio_unregister(void) -{ - platform_driver_unregister(&neo_dcpaud_driver); -} diff --git a/drivers/gpu/drm/apple-neo/audio.h b/drivers/gpu/drm/apple-neo/audio.h deleted file mode 100644 index a57bb7ba7e9f02..00000000000000 --- a/drivers/gpu/drm/apple-neo/audio.h +++ /dev/null @@ -1,20 +0,0 @@ -#ifndef __AUDIO_H__ -#define __AUDIO_H__ - -#include - -struct device; -struct platform_device; -struct neo_dcp_sound_cookie; - -int neo_dcp_audiosrv_prepare(struct device *dev, struct neo_dcp_sound_cookie *cookie); -int neo_dcp_audiosrv_startlink(struct device *dev, struct neo_dcp_sound_cookie *cookie); -int neo_dcp_audiosrv_stoplink(struct device *dev); -int neo_dcp_audiosrv_unprepare(struct device *dev); -int neo_dcp_audiosrv_get_elements(struct device *dev, void *elements, size_t maxsize); -int neo_dcp_audiosrv_get_product_attrs(struct device *dev, void *attrs, size_t maxsize); - -void neo_dcpaud_connect(struct platform_device *pdev, bool connected); -void neo_dcpaud_disconnect(struct platform_device *pdev); - -#endif /* __AUDIO_H__ */ diff --git a/drivers/gpu/drm/apple-neo/av.c b/drivers/gpu/drm/apple-neo/av.c deleted file mode 100644 index de16792c95f2ec..00000000000000 --- a/drivers/gpu/drm/apple-neo/av.c +++ /dev/null @@ -1,434 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2023 Martin PoviÅ¡er */ - -// #define DEBUG - -#include -#include -#include -#include -#include -#include -#include -#include - -#include "audio.h" -#include "afk.h" -#include "av.h" -#include "dcp.h" -#include "dcp-internal.h" - -struct neo_dcp_av_audio_cmds { - /* commands in group 0*/ - u32 open; - u32 close; - u32 prepare; - u32 start_link; - u32 stop_link; - u32 unprepare; - /* commands in group 1*/ - u32 get_elements; - u32 get_product_attrs; -}; - -static const struct neo_dcp_av_audio_cmds neo_dcp_av_audio_cmds_v12_3 = { - .open = 6, - .close = 7, - .prepare = 8, - .start_link = 9, - .stop_link = 12, - .unprepare = 13, - .get_elements = 18, - .get_product_attrs = 20, -}; - -static const struct neo_dcp_av_audio_cmds neo_dcp_av_audio_cmds_v13_5 = { - .open = 4, - .close = 5, - .prepare = 6, - .start_link = 7, - .stop_link = 10, - .unprepare = 11, - .get_elements = 16, - .get_product_attrs = 18, -}; - -struct audiosrv_data { - struct platform_device *neo_audio_dev; - bool plugged; - struct mutex plug_lock; - - struct neo_apple_epic_service *srv; - struct rw_semaphore srv_rwsem; - /* Workqueue for starting the audio service */ - struct work_struct start_av_service_wq; - - struct neo_dcp_av_audio_cmds cmds; - - bool warned_get_elements; - bool warned_get_product_attrs; - bool is_open; -}; - -static void neo_av_interface_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ -} - -static void neo_av_interface_teardown(struct neo_apple_epic_service *service) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - struct audiosrv_data *asrv = neo_dcp->audiosrv; - - neo_afk_service_disable(service); - - mutex_lock(&asrv->plug_lock); - - asrv->plugged = false; - if (asrv->neo_audio_dev) - neo_dcpaud_disconnect(asrv->neo_audio_dev); - - mutex_unlock(&asrv->plug_lock); -} - -static void neo_av_audiosrv_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - struct audiosrv_data *asrv = neo_dcp->audiosrv; - - mutex_lock(&asrv->plug_lock); - - down_write(&asrv->srv_rwsem); - asrv->srv = service; - up_write(&asrv->srv_rwsem); - - asrv->plugged = true; - mutex_unlock(&asrv->plug_lock); - schedule_work(&asrv->start_av_service_wq); -} - -static void neo_av_audiosrv_teardown(struct neo_apple_epic_service *service) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - struct audiosrv_data *asrv = neo_dcp->audiosrv; - - mutex_lock(&asrv->plug_lock); - - down_write(&asrv->srv_rwsem); - asrv->srv = NULL; - up_write(&asrv->srv_rwsem); - - asrv->plugged = false; - if (asrv->neo_audio_dev) - neo_dcpaud_disconnect(asrv->neo_audio_dev); - - mutex_unlock(&asrv->plug_lock); -} - -int neo_dcp_audiosrv_prepare(struct device *dev, struct neo_dcp_sound_cookie *cookie) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_afk_service_call(asrv->srv, 0, asrv->cmds.prepare, cookie, - sizeof(*cookie), 64 - sizeof(*cookie), NULL, 0, - 64); - up_write(&asrv->srv_rwsem); - - return ret; -} - -int neo_dcp_audiosrv_startlink(struct device *dev, struct neo_dcp_sound_cookie *cookie) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_afk_service_call(asrv->srv, 0, asrv->cmds.start_link, cookie, - sizeof(*cookie), 64 - sizeof(*cookie), NULL, 0, - 64); - up_write(&asrv->srv_rwsem); - - return ret; -} - -int neo_dcp_audiosrv_stoplink(struct device *dev) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_afk_service_call(asrv->srv, 0, asrv->cmds.stop_link, NULL, 0, 64, - NULL, 0, 64); - up_write(&asrv->srv_rwsem); - - return ret; -} - -int neo_dcp_audiosrv_unprepare(struct device *dev) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_afk_service_call(asrv->srv, 0, asrv->cmds.unprepare, NULL, 0, 64, - NULL, 0, 64); - up_write(&asrv->srv_rwsem); - - return ret; -} - -static int -neo_dcp_audiosrv_osobject_call(struct neo_apple_epic_service *service, u16 group, - u32 command, void *output, size_t output_maxsize, - size_t *output_size) -{ - struct { - __le64 max_size; - u8 _pad1[24]; - __le64 used_size; - u8 _pad2[8]; - } __attribute__((packed)) *hdr; - static_assert(sizeof(*hdr) == 48); - size_t bfr_len = output_maxsize + sizeof(*hdr); - void *bfr __free(kfree) = NULL; - int ret; - - bfr = kzalloc(bfr_len, GFP_KERNEL); - if (!bfr) - return -ENOMEM; - - hdr = bfr; - hdr->max_size = cpu_to_le64(output_maxsize); - ret = neo_afk_service_call(service, group, command, hdr, sizeof(*hdr), output_maxsize, - bfr, sizeof(*hdr) + output_maxsize, 0); - if (ret) - return ret; - - if (output) - memcpy(output, bfr + sizeof(*hdr), output_maxsize); - - if (output_size) - *output_size = le64_to_cpu(hdr->used_size); - - return 0; -} - -int neo_dcp_audiosrv_get_elements(struct device *dev, void *elements, size_t maxsize) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - size_t size; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_dcp_audiosrv_osobject_call(asrv->srv, 1, asrv->cmds.get_elements, - elements, maxsize, &size); - up_write(&asrv->srv_rwsem); - - if (ret && !asrv->warned_get_elements) { - dev_err(dev, "audiosrv: error getting elements: %d\n", ret); - asrv->warned_get_elements = true; - } else if (!ret) { - dev_dbg(dev, "audiosrv: got %zd bytes worth of elements\n", size); - } - - return ret; -} - -int neo_dcp_audiosrv_get_product_attrs(struct device *dev, void *attrs, size_t maxsize) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - struct audiosrv_data *asrv = neo_dcp->audiosrv; - size_t size; - int ret; - - down_write(&asrv->srv_rwsem); - ret = neo_dcp_audiosrv_osobject_call(asrv->srv, 1, - asrv->cmds.get_product_attrs, attrs, - maxsize, &size); - up_write(&asrv->srv_rwsem); - - if (ret && !asrv->warned_get_product_attrs) { - dev_err(dev, "audiosrv: error getting product attributes: %d\n", ret); - asrv->warned_get_product_attrs = true; - } else if (!ret) { - dev_dbg(dev, "audiosrv: got %zd bytes worth of product attributes\n", size); - } - - return ret; -} - -static int neo_av_audiosrv_report(struct neo_apple_epic_service *service, u32 idx, - const void *data, size_t data_size) -{ - dev_dbg(service->ep->neo_dcp->dev, "got audio report %d size %zx\n", idx, data_size); -#ifdef DEBUG - print_hex_dump(KERN_DEBUG, "audio report: ", DUMP_PREFIX_NONE, 16, 1, data, data_size, true); -#endif - - return 0; -} - -static const struct neo_apple_epic_service_ops neo_avep_ops[] = { - { - .name = "DCPAVSimpleVideoInterface", - .init = neo_av_interface_init, - .teardown = neo_av_interface_teardown, - }, - { - .name = "DCPAVAudioInterface", - .init = neo_av_audiosrv_init, - .report = neo_av_audiosrv_report, - .teardown = neo_av_audiosrv_teardown, - }, - {} -}; - -void neo_av_service_connect(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_epic_service *service; - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - scoped_guard(rwsem_write, &asrv->srv_rwsem) { - if (!asrv->srv) - return; - service = asrv->srv; - } - - /* open AV audio service */ - dev_info(neo_dcp->dev, "%s: starting audio service, plugged:%d\n", __func__, asrv->plugged); - if (asrv->is_open) - return; - - ret = neo_afk_service_call(service, 0, asrv->cmds.open, NULL, 0, 32, - NULL, 0, 32); - if (ret) { - dev_err(neo_dcp->dev, "error opening audio service: %d\n", ret); - return; - } - mutex_lock(&asrv->plug_lock); - asrv->is_open = true; - - if (asrv->neo_audio_dev) - neo_dcpaud_connect(asrv->neo_audio_dev, asrv->plugged); - mutex_unlock(&asrv->plug_lock); -} - -void neo_av_service_disconnect(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_epic_service *service; - struct audiosrv_data *asrv = neo_dcp->audiosrv; - int ret; - - scoped_guard(rwsem_write, &asrv->srv_rwsem) { - if (!asrv->srv) - return; - service = asrv->srv; - } - - /* close AV audio service */ - dev_info(neo_dcp->dev, "%s: stopping audio service\n", __func__); - if (!asrv->is_open) - return; - - mutex_lock(&asrv->plug_lock); - - if (asrv->neo_audio_dev) - neo_dcpaud_disconnect(asrv->neo_audio_dev); - - mutex_unlock(&asrv->plug_lock); - - ret = neo_afk_service_call(service, 0, asrv->cmds.close, NULL, 0, 16, - NULL, 0, 16); - if (ret) { - dev_err(neo_dcp->dev, "error closing audio service: %d\n", ret); - } - if (service->torndown) - neo_afk_service_disable(service); - asrv->is_open = false; -} - -static void neo_av_work_service_start(struct work_struct *work) -{ - struct audiosrv_data *audiosrv_data; - struct neo_apple_dcp *neo_dcp; - - audiosrv_data = container_of(work, struct audiosrv_data, start_av_service_wq); - - scoped_guard(rwsem_read, &audiosrv_data->srv_rwsem) { - if (!audiosrv_data->srv || - !audiosrv_data->srv->ep || - !audiosrv_data->srv->ep->neo_dcp) { - pr_err("%s: dcp: av: NULL ptr during startup\n", __func__); - return; - } - neo_dcp = audiosrv_data->srv->ep->neo_dcp; - } - - neo_av_service_connect(neo_dcp); -} - -int neo_avep_init(struct neo_apple_dcp *neo_dcp) -{ - struct audiosrv_data *audiosrv_data; - struct platform_device *neo_audio_pdev; - struct device *dev = neo_dcp->dev; - struct device_node *endpoint, *neo_audio_node = NULL; - - audiosrv_data = devm_kzalloc(neo_dcp->dev, sizeof(*audiosrv_data), GFP_KERNEL); - if (!audiosrv_data) - return -ENOMEM; - init_rwsem(&audiosrv_data->srv_rwsem); - mutex_init(&audiosrv_data->plug_lock); - - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - audiosrv_data->cmds = neo_dcp_av_audio_cmds_v12_3; - break; - case DCP_FIRMWARE_V_13_5: - audiosrv_data->cmds = neo_dcp_av_audio_cmds_v13_5; - break; - default: - dev_err(neo_dcp->dev, "Audio not supported for firmware\n"); - return -ENODEV; - } - - neo_dcp->audiosrv = audiosrv_data; - INIT_WORK(&audiosrv_data->start_av_service_wq, neo_av_work_service_start); - - endpoint = of_graph_get_endpoint_by_regs(dev->of_node, 0, 0); - if (endpoint) { - neo_audio_node = of_graph_get_remote_port_parent(endpoint); - of_node_put(endpoint); - } - if (!neo_audio_node || !of_device_is_available(neo_audio_node)) { - of_node_put(neo_audio_node); - dev_info(dev, "No audio support\n"); - return 0; - } - - neo_audio_pdev = of_find_device_by_node(neo_audio_node); - of_node_put(neo_audio_node); - if (!neo_audio_pdev) { - dev_info(dev, "No DP/HDMI audio device not ready\n"); - return 0; - } - neo_dcp->audiosrv->neo_audio_dev = neo_audio_pdev; - - device_link_add(&neo_audio_pdev->dev, dev, - DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME); - - neo_dcp->neo_avep = neo_afk_init(neo_dcp, AV_ENDPOINT, neo_avep_ops); - if (IS_ERR(neo_dcp->neo_avep)) - return PTR_ERR(neo_dcp->neo_avep); - neo_dcp->neo_avep->debugfs_entry = neo_dcp->ep_debugfs[AV_ENDPOINT - 0x20]; - return neo_afk_start(neo_dcp->neo_avep); -} diff --git a/drivers/gpu/drm/apple-neo/av.h b/drivers/gpu/drm/apple-neo/av.h deleted file mode 100644 index 83d5c76f2a486d..00000000000000 --- a/drivers/gpu/drm/apple-neo/av.h +++ /dev/null @@ -1,17 +0,0 @@ -#ifndef __AV_H__ -#define __AV_H__ - -#include "parser.h" - -//int avep_audiosrv_startlink(struct apple_dcp *dcp, struct dcp_sound_cookie *cookie); -//int avep_audiosrv_stoplink(struct apple_dcp *dcp); - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) -void neo_av_service_connect(struct neo_apple_dcp *neo_dcp); -void neo_av_service_disconnect(struct neo_apple_dcp *neo_dcp); -#else -static inline void neo_av_service_connect(struct neo_apple_dcp *neo_dcp) { } -static inline void neo_av_service_disconnect(struct neo_apple_dcp *neo_dcp) { } -#endif - -#endif /* __AV_H__ */ diff --git a/drivers/gpu/drm/apple-neo/connector.c b/drivers/gpu/drm/apple-neo/connector.c deleted file mode 100644 index 713b3f4927151b..00000000000000 --- a/drivers/gpu/drm/apple-neo/connector.c +++ /dev/null @@ -1,156 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0+ OR MIT -/* - * Copyright (C) The Asahi Linux Contributors - */ - -#include "connector.h" - -#include "linux/err.h" -#include -#include -#include -#include -#include - -#include - -#include "dcp-internal.h" - -enum neo_dcp_chunk_type { - DCP_CHUNK_COLOR_ELEMENTS, - DCP_CHUNK_TIMING_ELELMENTS, - DCP_CHUNK_DISPLAY_ATTRIBUTES, - DCP_CHUNK_TRANSPORT, - DCP_CHUNK_NUM_TYPES, -}; - -static int chunk_show(struct seq_file *m, - enum neo_dcp_chunk_type chunk_type) -{ - struct neo_apple_connector *apple_con = m->private; - struct neo_dcp_chunks *chunk = NULL; - - mutex_lock(&apple_con->chunk_lock); - - switch (chunk_type) { - case DCP_CHUNK_COLOR_ELEMENTS: - chunk = &apple_con->color_elements; - break; - case DCP_CHUNK_TIMING_ELELMENTS: - chunk = &apple_con->timing_elements; - break; - case DCP_CHUNK_DISPLAY_ATTRIBUTES: - chunk = &apple_con->display_attributes; - break; - case DCP_CHUNK_TRANSPORT: - chunk = &apple_con->transport; - break; - default: - break; - } - - if (chunk) - seq_write(m, chunk->data, chunk->length); - - mutex_unlock(&apple_con->chunk_lock); - - return 0; -} - -#define CONNECTOR_DEBUGFS_ENTRY(name, type) \ -static int chunk_ ## name ## _show(struct seq_file *m, void *data) \ -{ \ - return chunk_show(m, type); \ -} \ -static int chunk_ ## name ## _open(struct inode *inode, struct file *file) \ -{ \ - return single_open(file, chunk_ ## name ## _show, inode->i_private); \ -} \ -static const struct file_operations chunk_ ## name ## _fops = { \ - .owner = THIS_MODULE, \ - .open = chunk_ ## name ## _open, \ - .read = seq_read, \ - .llseek = seq_lseek, \ - .release = single_release, \ -} - -CONNECTOR_DEBUGFS_ENTRY(color, DCP_CHUNK_COLOR_ELEMENTS); -CONNECTOR_DEBUGFS_ENTRY(timing, DCP_CHUNK_TIMING_ELELMENTS); -CONNECTOR_DEBUGFS_ENTRY(display_attribs, DCP_CHUNK_DISPLAY_ATTRIBUTES); -CONNECTOR_DEBUGFS_ENTRY(transport, DCP_CHUNK_TRANSPORT); - -static void neo_dcp_afk_debugfs_root(struct platform_device *pdev, int ep, struct dentry *root) -{ -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_DEBUG) - struct dentry *entry = NULL; - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - switch (ep) { - case AV_ENDPOINT: - entry = debugfs_create_dir("avep", root); - break; - default: - break; - } - - if (!IS_ERR_OR_NULL(entry)) - neo_dcp->ep_debugfs[ep - 0x20] = entry; -#endif -} - -void neo_apple_connector_debugfs_init(struct drm_connector *connector, struct dentry *root) -{ - struct neo_apple_connector *apple_con = to_apple_connector(connector); - - debugfs_create_file("ColorElements", 0444, root, apple_con, - &chunk_color_fops); - debugfs_create_file("TimingElements", 0444, root, apple_con, - &chunk_timing_fops); - debugfs_create_file("DisplayAttributes", 0444, root, apple_con, - &chunk_display_attribs_fops); - debugfs_create_file("Transport", 0444, root, apple_con, - &chunk_transport_fops); - - switch (connector->connector_type) { - case DRM_MODE_CONNECTOR_DisplayPort: - case DRM_MODE_CONNECTOR_HDMIA: - neo_dcp_afk_debugfs_root(apple_con->neo_dcp, AV_ENDPOINT, root); - break; - default: - break; - } -} - -static void neo_dcp_connector_set_dict(struct neo_apple_connector *connector, - struct neo_dcp_chunks *dict, - struct neo_dcp_chunks *chunks) -{ - /* - * Not devm: the buffer is allocated by whichever DCP received the - * chunks, but a Type-C port can move to another pipeline before it is - * replaced, and freeing it against the connector's current DCP then - * fails devres_destroy() -- one WARN and one leak per dictionary. - */ - kfree(dict->data); - - *dict = *chunks; -} - -void neo_dcp_connector_update_dict(struct neo_apple_connector *connector, const char *key, - struct neo_dcp_chunks *chunks) -{ - mutex_lock(&connector->chunk_lock); - if (!strcmp(key, "ColorElements")) - neo_dcp_connector_set_dict(connector, &connector->color_elements, chunks); - else if (!strcmp(key, "TimingElements")) - neo_dcp_connector_set_dict(connector, &connector->timing_elements, chunks); - else if (!strcmp(key, "DisplayAttributes")) - neo_dcp_connector_set_dict(connector, &connector->display_attributes, chunks); - else if (!strcmp(key, "Transport")) - neo_dcp_connector_set_dict(connector, &connector->transport, chunks); - - chunks->data = NULL; - chunks->length = 0; - - mutex_unlock(&connector->chunk_lock); -} diff --git a/drivers/gpu/drm/apple-neo/connector.h b/drivers/gpu/drm/apple-neo/connector.h deleted file mode 100644 index 4d91fc2639eac6..00000000000000 --- a/drivers/gpu/drm/apple-neo/connector.h +++ /dev/null @@ -1,64 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* "Copyright" 2021 Alyssa Rosenzweig */ - -#ifndef __APPLE_CONNECTOR_H__ -#define __APPLE_CONNECTOR_H__ - -#include - -#include -#include "drm/drm_connector.h" -#include "drm/drm_edid.h" - -struct neo_apple_connector; - -#include "dcp-internal.h" - -void neo_dcp_hotplug(struct work_struct *work); -void neo_dcp_retrain_oob(struct neo_apple_connector *connector); - -/* - * How many DCP pipelines may offer a route to one Type-C port. The Type-C mux - * class caps the mode-switch providers per connector (TYPEC_MUX_MAX_DEVS) and - * the ATC PHY already claims one of those slots, so this is generous. - */ -struct neo_apple_connector { - struct drm_connector base; - bool connected; - - /* The pipeline currently driving this connector, NULL if unrouted. */ - struct platform_device *neo_dcp; - - /* - * A Type-C connector is a physical port rather than a fixed pipeline, - * and it has one encoder covering every pipeline that can drive it. - * One encoder per pipeline would describe the same hardware, but - * userspace takes the CRTCs a connector can use to be what all of its - * encoders have in common, and single-pipeline encoders have nothing - * in common. - */ - struct drm_encoder *port_encoder; - - /* the CRTCs of every pipeline that can drive the port */ - u32 candidate_crtcs; - - const struct drm_edid *drm_edid; - - /* Workqueue for sending hotplug events to the associated device */ - struct work_struct hotplug_wq; - - struct mutex chunk_lock; - - struct neo_dcp_chunks color_elements; - struct neo_dcp_chunks timing_elements; - struct neo_dcp_chunks display_attributes; - struct neo_dcp_chunks transport; -}; - -#define to_apple_connector(x) container_of(x, struct neo_apple_connector, base) - -void neo_apple_connector_debugfs_init(struct drm_connector *connector, struct dentry *root); - -void neo_dcp_connector_update_dict(struct neo_apple_connector *connector, const char *key, - struct neo_dcp_chunks *chunks); -#endif diff --git a/drivers/gpu/drm/apple-neo/dcp-internal.h b/drivers/gpu/drm/apple-neo/dcp-internal.h deleted file mode 100644 index e698052013ad79..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp-internal.h +++ /dev/null @@ -1,475 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#ifndef __APPLE_DCP_INTERNAL_H__ -#define __APPLE_DCP_INTERNAL_H__ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "dptxep.h" -#include "dcp_backlight.h" -#include "iomfb.h" -#include "iomfb_h17p.h" -#include "iomfb-state.h" -#include "iomfb_v12_3.h" -#include "iomfb_v13_3.h" -#include "epic/dpavservep.h" - -#define DCP_MAX_PLANES 4 -#define DCP_MAX_TYPEC_ROUTES 4 - -struct neo_apple_dcp; -struct neo_apple_dcp_afkep; - -/* Snapshot of a completed present, independent of the latest DRM state. */ -struct neo_iomfb_scanout_h17p { - struct neo_dcp_swap_submit_req_h17p request; - struct drm_framebuffer *fb[SWAP_SURFACES]; -}; - -/* One operation, including all of its nested replies and present completion. */ -struct neo_iomfb_transaction { - struct list_head link; - void (*start)(struct neo_apple_dcp *neo_dcp, struct neo_iomfb_transaction *transaction); - void (*release)(struct neo_iomfb_transaction *transaction); - struct neo_dcp_backlight_present backlight; - bool backlight_reserved; - bool backlight_failed; - bool brightness_only; - bool completed; -}; -struct neo_apple_dcp_typec_port; - -struct neo_apple_dcp_typec_route { - struct neo_apple_dcp *neo_dcp; - struct neo_apple_dcp_typec_port *port; - struct list_head port_link; - struct phy *phy; - struct mux_control *xbar; - struct typec_mux_dev *typec_mux; - u32 neo_dptx_phy; - u32 mux_index; - bool selected; - /* crossbar output actually selected: xbar (dpphy) or a Thunderbolt dpin */ - unsigned int tunnel_dpin; - struct mux_control *active_xbar; - bool tunnel; - /* tunnel: crossbar brought up (at DidChangeLinkConfiguration) */ - bool xbar_up; -}; - -bool neo_dcp_is_typec_output(struct neo_apple_dcp *neo_dcp); -void neo_dcp_swap_watchdog_arm(struct neo_apple_dcp *neo_dcp); -void neo_dcp_swap_watchdog_complete(struct neo_apple_dcp *neo_dcp); -bool neo_dcp_is_usb4_output(struct neo_apple_dcp *neo_dcp); -void neo_dcp_retry_placeholder_edid(struct neo_apple_dcp *neo_dcp, - const struct drm_edid *drm_edid); - -struct dcpav_service_epic; - -enum neo_dcp_firmware_version { - DCP_FIRMWARE_UNKNOWN, - DCP_FIRMWARE_V_12_3, - DCP_FIRMWARE_V_13_5, - DCP_FIRMWARE_H17P, -}; - -enum { - SYSTEM_ENDPOINT = 0x20, - TEST_ENDPOINT = 0x21, - DCP_EXPERT_ENDPOINT = 0x22, - DISP0_ENDPOINT = 0x23, - DPAVSERV_ENDPOINT = 0x28, - AV_ENDPOINT = 0x29, - DPTX_ENDPOINT = 0x2a, - HDCP_ENDPOINT = 0x2b, - REMOTE_ALLOC_ENDPOINT = 0x2d, - IOMFB_ENDPOINT = 0x37, -}; - -/* Temporary backing for a chunked transfer via setDCPAVPropStart/Chunk/End */ -struct neo_dcp_chunks { - size_t length; - void *data; -}; - -#define DCP_MAX_MAPPINGS (128) /* should be enough */ -#define MAX_DISP_REGISTERS (7) - -struct neo_dcp_mem_descriptor { - size_t size; - void *buf; - dma_addr_t dva; - struct sg_table map; - u64 reg; - /* live iommu mappings installed by dcpep_cb_map_piodma() */ - bool piodma_mapped; -}; - -/* Limit on call stack depth (arbitrary). Some nesting is required */ -#define DCP_MAX_CALL_DEPTH 8 - -typedef void (*neo_dcp_callback_t)(struct neo_apple_dcp *, void *, void *); - -struct neo_dcp_channel { - neo_dcp_callback_t callbacks[DCP_MAX_CALL_DEPTH]; - void *cookies[DCP_MAX_CALL_DEPTH]; - void *output[DCP_MAX_CALL_DEPTH]; - /* Header of each pending AP command record, as dcp_push() wrote it */ - struct neo_dcp_packet_header header[DCP_MAX_CALL_DEPTH]; - u16 end[DCP_MAX_CALL_DEPTH]; - - /* Current depth of the call stack. Less than DCP_MAX_CALL_DEPTH */ - u8 depth; - /* Already warned about busy channel */ - bool warned_busy; -}; - -struct neo_dcp_fb_reference { - struct list_head head; - struct drm_framebuffer *fb; - /* - * Id of the swap that unbinds @fb, i.e. the one that puts its - * replacement on screen. Only meaningful once @armed is set: the - * firmware assigns the id in the swap_start reply, which happens after - * the atomic commit that displaced @fb has already queued this entry. - */ - u32 swap_id; - bool armed; -}; - -#define MAX_NOTCH_HEIGHT 160 - -struct neo_dcp_brightness { - struct backlight_device *bl_dev; - u32 maximum; - u32 dac; - int nits; - int scale; - bool update; -}; - -struct audiosrv_data; - -/** laptop/AiO integrated panel parameters from DT */ -struct neo_dcp_panel { - /// panel width in millimeter - int width_mm; - /// panel height in millimeter - int height_mm; - /// panel has a mini-LED backlight - bool has_mini_led; -}; - -enum neo_dcp_iomfb_method_profile { - DCP_IOMFB_METHODS_DEFAULT, - DCP_IOMFB_METHODS_H17G, -}; - -struct neo_apple_dcp_hw_data { - u32 num_dptx_ports; - enum neo_dcp_iomfb_method_profile neo_iomfb_method_profile; - /* - * The bootloader leaves the coprocessor running. Attach to it with a - * standard RTKit wake instead of starting or restarting the ASC. - */ - bool adopt_live_session; - enum neo_dcp_firmware_version firmware_compat; - /* Firmware-visible clock aperture, independent of the AP PMGR window. */ - const struct resource *firmware_clock; - u32 firmware_scratch; - u32 firmware_request; -}; - -/* TODO: move IOMFB members to its own struct */ -struct neo_apple_dcp { - /* An adopted session has no confirmed firmware DMA stop boundary. */ - bool retain_dma; - bool quiescing; - bool drm_retained; - struct device *dev; - struct platform_device *piodma; - bool piodma_created; - struct iommu_domain *iommu_dom; - /* which of the nine cumulative A031 notify-client states to send next */ - unsigned int a031_step; - struct apple_rtkit *rtk; - struct neo_apple_crtc *crtc; - struct neo_apple_connector *connector; - - struct neo_apple_dcp_hw_data hw; - - /* firmware version and compatible firmware version */ - enum neo_dcp_firmware_version fw_compat; - - /* Coprocessor control register */ - void __iomem *coproc_reg; - - /* DCP has crashed */ - bool crashed; - - DECLARE_BITMAP(neo_iomfb_surfaces, DCP_MAX_PLANES); - - /************* IOMFB ************************************************** - * everything below is mostly used inside IOMFB but it could make * - * sense to keep some of the members in apple_dcp. * - **********************************************************************/ - - /* clock rate request by dcp in */ - struct clk *clk; - struct clk *clk_194; - - /* DCP shared memory */ - void *shmem; - - /* Display registers mappable to the DCP */ - struct resource *disp_registers[MAX_DISP_REGISTERS]; - unsigned int nr_disp_registers; - - struct resource disp_bw_scratch_res; - struct resource disp_bw_doorbell_res; - u32 disp_bw_scratch_index; - u32 disp_bw_scratch_offset; - u32 disp_bw_doorbell_index; - u32 disp_bw_doorbell_offset; - - u32 index; - - /* Bitmap of memory descriptors used for mappings made by the DCP */ - DECLARE_BITMAP(memdesc_map, DCP_MAX_MAPPINGS); - - /* Indexed table of memory descriptors */ - struct neo_dcp_mem_descriptor memdesc[DCP_MAX_MAPPINGS]; - - struct neo_dcp_channel ch_cmd, ch_oobcmd; - struct neo_dcp_channel ch_cb, ch_oobcb, ch_async, ch_oobasync; - - /* iomfb EP callback handlers */ - const neo_iomfb_cb_handler *cb_handlers; - - /* Active chunked transfer. There can only be one at a time. */ - struct neo_dcp_chunks chunks; - - /* Queued swap. Owned by the DCP to avoid per-swap memory allocation */ - union { - struct neo_dcp_swap_submit_req_v12_3 v12_3; - struct neo_dcp_swap_submit_req_v13_3 v13_3; - struct neo_dcp_swap_submit_req_h17p h17p; - } swap; - - /* swap id of the last completed swap */ - u32 last_swap_id; - /* last_swap_id is only meaningful after the first swap completes */ - bool have_swap_complete; - ktime_t swap_start; - u64 swap_submit_timestamp; - - /* Current display mode */ - struct neo_dcp_mode_state mode_state; - /* One HPD pulse after a placeholder EDID, per Type-C connection. */ - bool placeholder_retried; - u64 typec_generation; /* hpd_mutex: identifies the current connection */ - u64 placeholder_generation; - struct delayed_work placeholder_edid_wq; - bool use_timestamps; - bool vrr_enabled; - struct neo_dcp_set_digital_out_mode_req mode; - - /* completion for active turning true */ - struct completion start_done; - - /* Is the DCP booted? */ - bool active; - /* A successful initial enable keeps the H17P pipe running across DPMS. */ - bool pipe_enabled_h17p; - - /* eDP display without DP-HDMI conversion */ - bool main_display; - - /* clear all surfaces on init */ - bool surfaces_cleared; - - /* enable CRC calculation */ - bool crc_enabled; - - /* Modes valid for the connected display */ - struct neo_dcp_display_mode *modes; - unsigned int nr_modes; - - /* Attributes of the connector */ - int connector_type; - int fixed_connector_type; - - /* Attributes of the connected display */ - int width_mm, height_mm; - - unsigned notch_height; - - /* Workqueue for sending vblank events when a dcp swap is not possible */ - struct work_struct vblank_wq; - - /* Completes a Type-C swap that DCP dropped, and recovers the pipe. */ - struct delayed_work swap_watchdog_wq; - unsigned int swap_watchdog_retrains; - - /* List of referenced framebuffers pending a completed replacement swap. - * The commit and RTKit work queues share it under this lock. - */ - struct list_head swapped_out_fbs; - /* Protects swapped_out_fbs. */ - struct mutex swapped_out_fbs_lock; - - struct neo_dcp_brightness brightness; - /* Workqueue for updating the initial brightness */ - struct work_struct bl_register_wq; - struct mutex bl_register_mutex; - /* Workqueue for updating the brightness */ - struct work_struct bl_update_wq; - /* Registers the H17P backlight when the loader reports no level. */ - struct delayed_work bl_fallback_wq; - - /* integrated panel if present */ - struct neo_dcp_panel panel; - - struct neo_apple_dcp_afkep *neo_systemep; - struct completion neo_systemep_done; - - struct neo_apple_dcp_afkep *neo_ibootep; - struct neo_apple_dcp_afkep *dcpavservep; - struct neo_dcpavserv neo_dcpavserv; - - struct neo_apple_dcp_afkep *neo_avep; - struct audiosrv_data *audiosrv; - - struct neo_apple_dcp_afkep *neo_dptxep; - - struct neo_dptx_port neo_dptxport[2]; - - /* debugfs entries */ - struct dentry *ep_debugfs[0x20]; - - /* these fields are output port specific */ - struct phy *phy; - struct phy *fixed_phy; - struct mux_control *xbar; - struct typec_mux *typec_mux; - struct neo_apple_dcp_typec_route typec_routes[DCP_MAX_TYPEC_ROUTES]; - struct neo_apple_dcp_typec_route *active_typec_route; - u32 nr_typec_routes; - bool phy_managed_by_typec; - bool typec_cable_connected; - /* DPTX feeds a Thunderbolt DP IN adapter, not the Type-C PHY lanes */ - bool neo_dptx_tunnel; - /* DFP port in the DPTX target: 0 = dpphy, 1 = dpin0, 2 = dpin1 */ - u8 neo_dptx_dfp_port; - /* wakes/sleeps the Thunderbolt DP IN adapter from DCP Activate/Deactivate */ - int (*tb_dpin_set_active)(void *ctx, bool active); - void *tb_dpin_ctx; - /* - * Serializes the Thunderbolt DP IN callback and tunnel crossbar state - * between DCP apcalls and tunnel teardown; never held while waiting - * for DCP. - */ - struct mutex tb_lock; - bool tb_clock_ok; - /* CRTC powered off while the Type-C cable stays attached */ - bool typec_crtc_off; - /* IOMFB reports its video interface ready after DPTX link training. */ - struct completion typec_iomfb_hpd_ready; - struct delayed_work typec_reconnect_wq; - struct delayed_work typec_fabric_retrain_wq; - u32 typec_reconnect_tries; - - struct gpio_desc *hdmi_hpd; - struct gpio_desc *hdmi_pwren; - struct gpio_desc *dp2hdmi_pwren; - - struct mutex hpd_mutex; - - u32 neo_dptx_phy; - u32 neo_dptx_die; - u32 fixed_dptx_phy; - u32 fixed_mux_index; - bool fixed_route_selected; - struct neo_apple_connector *fixed_connector; - struct neo_apple_connector *typec_connector; - int hdmi_hpd_irq; - - /* H17P policy shared by backlight, commit, and reply workers. */ - struct { - /* Protects state and callback installation. */ - spinlock_t lock; - struct neo_dcp_backlight_state state; - void (*kick)(struct neo_apple_dcp *neo_dcp); - /* Rate limit for firmware re-send requests. */ - unsigned long resent_at; - bool resent; - } backlight; - - /* Staging wire record; serialization never mutates the atomic inputs. */ - struct neo_dcp_present_h17p present_h17p; - struct neo_dcp_present_state_h17p present_state_h17p; - DECLARE_BITMAP(unknown_callbacks, IOMFB_MAX_CB); - /* Runtime callbacks whose record size has been reported once. */ - DECLARE_BITMAP(sized_callbacks, IOMFB_MAX_CB); - - /* Serializes H17P transmit preparation with RTKit receive callbacks. */ - struct { - /* Protects queue, channel stacks, and the current owner. */ - struct mutex lock; - struct list_head pending; - struct neo_iomfb_transaction *active; - struct task_struct *owner; - struct work_struct work; - struct delayed_work timeout; - struct delayed_work backlight_retry; - unsigned long deadline; - unsigned int queued; - bool stopped; - bool backlight_queued; - atomic_t opaque_x_state; - struct neo_iomfb_scanout_h17p *scanout; - struct neo_iomfb_scanout_h17p *next_scanout; - } neo_iomfb; -}; - -void neo_iomfb_scanout_complete_h17p(struct neo_apple_dcp *neo_dcp); -void neo_iomfb_present_backlight_h17p(struct neo_apple_dcp *neo_dcp); -bool neo_iomfb_present_brightness_only_h17p(struct neo_apple_dcp *neo_dcp); -bool neo_iomfb_present_complete_h17p(struct neo_apple_dcp *neo_dcp); -void neo_iomfb_present_failed_h17p(struct neo_apple_dcp *neo_dcp); -bool neo_iomfb_apply_backlight_h17p(struct neo_apple_dcp *neo_dcp, - const struct neo_dcp_swap_submit_req_h17p *request, - struct neo_dcp_present_h17p *wire); -void neo_iomfb_apply_opaque_x_h17p(struct neo_apple_dcp *neo_dcp, neo_dcp_callback_t callback, - void *cookie); -void neo_iomfb_opaque_x_reset_h17p(struct neo_apple_dcp *neo_dcp); -int neo_iomfb_configure_backlight_h17p(struct neo_apple_dcp *neo_dcp, u32 maximum, - bool inherited_valid, u32 inherited, - bool default_valid, u32 default_nits); -void neo_iomfb_queue_init(struct neo_apple_dcp *neo_dcp); -void neo_iomfb_queue_stop(struct neo_apple_dcp *neo_dcp); -bool neo_iomfb_queue_drain(struct neo_apple_dcp *neo_dcp, unsigned long timeout); -int neo_iomfb_queue(struct neo_apple_dcp *neo_dcp, struct neo_iomfb_transaction *transaction); -void neo_iomfb_queue_crc_h17p(struct neo_apple_dcp *neo_dcp, u32 swap_id); - -void neo_dcp_drm_crtc_page_flip(struct neo_apple_dcp *neo_dcp, ktime_t now); -void neo_dcp_handle_hotplug_actions(struct neo_apple_dcp *neo_dcp, unsigned int action); - -int neo_dcp_backlight_register(struct neo_apple_dcp *neo_dcp); -int neo_dcp_backlight_update(struct neo_apple_dcp *neo_dcp); -bool neo_dcp_has_panel(struct neo_apple_dcp *neo_dcp); - -#define DCP_AUDIO_MAX_CHANS 15 - -#endif /* __APPLE_DCP_INTERNAL_H__ */ diff --git a/drivers/gpu/drm/apple-neo/dcp-link.h b/drivers/gpu/drm/apple-neo/dcp-link.h deleted file mode 100644 index 400b58ef5660de..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp-link.h +++ /dev/null @@ -1,95 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ -/* Copyright The Asahi Linux Contributors */ - -#ifndef __APPLE_DCP_LINK_H__ -#define __APPLE_DCP_LINK_H__ - -#include -#include -#include - -#include - -/* - * H17-generation DCP firmware uses the ordinary IOMFB transport once the - * INIT/SetShmem handshake is done: the same six fixed channels (CMD 0x00000, - * OOBCMD 0x08000, ASYNC 0x40000, OOBASYNC 0x48000, CB 0x60000, - * OOBCB 0x68000), selected by the context field of the message. Records are - * allocated on each channel's stack at - * off_next = off_cur + align_up(12 + in_len + out_len, 0x40) - * which is the rule dcp_packet_start() and DCP_PACKET_ALIGNMENT implement in - * iomfb.c. - * - * This header only describes the callback payloads whose layout differs from - * older firmware. - */ - -#define APPLE_DCP_H17P_CLOCK_ID_154 0x154 -#define APPLE_DCP_H17P_CLOCK_ID_194 0x194 - -struct neo_apple_dcp_h17p_clock_request { - __le32 service; - __le32 clock_id; -} __packed; - -struct neo_apple_dcp_h17p_rt_bw_config { - __le64 scratch; - __le64 clock_request; -} __packed; - -struct neo_apple_dcp_h17p_rt_bw_request { - __le32 config_null; -} __packed; - -struct neo_apple_dcp_h17p_rt_bw_reply { - struct neo_apple_dcp_h17p_rt_bw_config config; - __le32 status; -} __packed; - -static_assert(sizeof(struct neo_apple_dcp_h17p_clock_request) == 0x08); -static_assert(sizeof(struct neo_apple_dcp_h17p_rt_bw_config) == 0x10); -static_assert(sizeof(struct neo_apple_dcp_h17p_rt_bw_request) == 0x04); -static_assert(sizeof(struct neo_apple_dcp_h17p_rt_bw_reply) == 0x14); - -static inline int -neo_apple_dcp_h17p_clock_rate(const struct neo_apple_dcp_h17p_clock_request *request, - u64 rate_154, u64 rate_194, u64 *rate) -{ - if (!request || !rate) - return -EINVAL; - - switch (le32_to_cpu(request->clock_id)) { - case APPLE_DCP_H17P_CLOCK_ID_154: - *rate = rate_154; - return 0; - case APPLE_DCP_H17P_CLOCK_ID_194: - *rate = rate_194; - return 0; - default: - return -ENOENT; - } -} - -static inline void -neo_apple_dcp_h17p_rt_bw_encode(struct neo_apple_dcp_h17p_rt_bw_config *config, - u64 scratch, u64 clock_request) -{ - config->scratch = cpu_to_le64(scratch); - config->clock_request = cpu_to_le64(clock_request); -} - -/* D003: fill in the bandwidth config unless the firmware passed it as null */ -static inline int -neo_apple_dcp_h17p_rt_bw_encode_reply(const struct neo_apple_dcp_h17p_rt_bw_request *request, - struct neo_apple_dcp_h17p_rt_bw_reply *reply, - u64 scratch, u64 clock_request) -{ - if (le32_to_cpu(request->config_null) & 1) - return -EINVAL; - - neo_apple_dcp_h17p_rt_bw_encode(&reply->config, scratch, clock_request); - reply->status = cpu_to_le32(0); - return 0; -} - -#endif /* __APPLE_DCP_LINK_H__ */ diff --git a/drivers/gpu/drm/apple-neo/dcp.c b/drivers/gpu/drm/apple-neo/dcp.c deleted file mode 100644 index d44d7a52fd096b..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp.c +++ /dev/null @@ -1,3986 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include - -#include "afk.h" -#include "av.h" -#include "dcp.h" -#include "dcp-internal.h" -#include "iomfb.h" -#include "parser.h" -#include "trace.h" - -#define APPLE_DCP_COPROC_CPU_CONTROL 0x44 -#define APPLE_DCP_COPROC_CPU_CONTROL_RUN BIT(4) - -struct neo_dcp_session_ref { - struct list_head link; - struct device *dev; - struct iommu_group *group; -}; - -static LIST_HEAD(neo_dcp_retained_sessions); -static DEFINE_MUTEX(neo_dcp_retained_sessions_lock); - -static bool neo_dcp_session_retained(struct device *dev) -{ - struct neo_dcp_session_ref *session; - bool retained = false; - - mutex_lock(&neo_dcp_retained_sessions_lock); - list_for_each_entry(session, &neo_dcp_retained_sessions, link) { - if (dev_fwnode(session->dev) == dev_fwnode(dev)) { - retained = true; - break; - } - } - mutex_unlock(&neo_dcp_retained_sessions_lock); - return retained; -} - -static int neo_dcp_pin_live_session(struct neo_apple_dcp *neo_dcp) -{ - struct neo_dcp_session_ref *session; - - if (!neo_dcp->hw.adopt_live_session || neo_dcp->fw_compat != DCP_FIRMWARE_H17P || - neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G || neo_dcp->retain_dma) - return 0; - session = kzalloc_obj(*session); - if (!session) - return -ENOMEM; - - /* Keep the DMA device, domain usage and driver code valid until reset. */ - session->dev = get_device(neo_dcp->dev); - session->group = iommu_group_get(neo_dcp->dev); - __module_get(THIS_MODULE); - WRITE_ONCE(neo_dcp->retain_dma, true); - mutex_lock(&neo_dcp_retained_sessions_lock); - list_add_tail(&session->link, &neo_dcp_retained_sessions); - mutex_unlock(&neo_dcp_retained_sessions_lock); - dev_info(neo_dcp->dev, "live H17P DMA resources retained until reboot; rebind unsupported\n"); - return 0; -} - -static void neo_dcp_release_dma_domain(void *data) -{ - struct neo_apple_dcp *neo_dcp = data; - - if (!neo_dcp->retain_dma) - iommu_device_unuse_default_domain(neo_dcp->dev); -} - -static void neo_dcp_release_context(void *data) -{ - struct neo_apple_dcp *neo_dcp = data; - - if (!neo_dcp->retain_dma) - kfree(neo_dcp); -} - -static void neo_dcp_release_rtkit(void *data) -{ - struct neo_apple_dcp *neo_dcp = data; - - if (neo_dcp->retain_dma) { - WRITE_ONCE(neo_dcp->quiescing, true); - apple_rtkit_free_retaining_buffers(neo_dcp->rtk); - } else { - apple_rtkit_free(neo_dcp->rtk); - } - neo_dcp->rtk = NULL; -} - -#define DCP_BOOT_TIMEOUT msecs_to_jiffies(1000) - -static bool show_notch; -module_param(show_notch, bool, 0644); -MODULE_PARM_DESC(show_notch, "Use the full display height and shows the notch"); - -bool neo_hdmi_audio; -module_param(neo_hdmi_audio, bool, 0644); -MODULE_PARM_DESC(neo_hdmi_audio, "Enable unstable HDMI audio support"); - -static bool unstable_edid = true; -module_param(unstable_edid, bool, 0644); -MODULE_PARM_DESC(unstable_edid, "Enable unstable EDID retrival support"); - -struct neo_apple_dcp_typec_port { - struct list_head link; - struct list_head routes; - struct device_node *connector_np; - struct neo_apple_dcp_typec_route *owner; - struct neo_apple_dcp_typec_route *secondary_owner; - /* Keep a port on its last DCP while that pipeline remains free. */ - struct neo_apple_dcp_typec_route *preferred_route; - /* Ignore the USB4 fallback immediately following this port's DP teardown. */ - unsigned long dp_release_deadline; - /* DRM connector for this physical port, driven by whichever DCP owns it */ - struct neo_apple_connector *connector; - /* A second logical stream through this port's USB4 dock. */ - struct neo_apple_connector *secondary_connector; - /* last mux state acted on, to collapse the per-candidate notifications */ - struct typec_altmode *applied_alt; - unsigned long applied_mode; - u32 applied_status; - u32 applied_conf; - bool applied_valid; - bool hpd; - /* Direct DP-alt: the port is in DP mode, routed or not, and its HPD. */ - bool dp_wanted; - bool dp_hpd; - /* dcp_typec_rebalance_locked()'s plan for the port's two streams */ - struct neo_apple_dcp_typec_route *target; - struct neo_apple_dcp_typec_route *secondary_target; - /* left out of the plan: its planned pipeline is held by a tunnel */ - bool plan_dark; -}; - -static DEFINE_MUTEX(neo_dcp_typec_fabric_lock); -static LIST_HEAD(neo_dcp_typec_ports); - -static int neo_dcp_dptx_connect(struct neo_apple_dcp *neo_dcp, u32 port); -static void disconnected_hpd_event(struct neo_apple_connector *connector); - -bool neo_dcp_is_typec_output(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->active_typec_route || - neo_dcp->fixed_connector_type == DRM_MODE_CONNECTOR_USB; -} - -bool neo_dcp_is_usb4_output(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->active_typec_route && neo_dcp->active_typec_route->tunnel; -} - -static bool neo_dcp_typec_route_is_dp(const struct typec_mux_state *state) -{ - return state->alt && state->alt->svid == USB_TYPEC_DP_SID && - state->mode >= TYPEC_DP_STATE_A && - state->mode <= TYPEC_DP_STATE_F; -} - -/* Keep a live fixed output on its own pipeline. */ -static bool neo_dcp_typec_route_fixed_output_busy(struct neo_apple_dcp_typec_route *route) -{ - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - - if (neo_dcp->fixed_connector_type == DRM_MODE_CONNECTOR_USB) - return false; - if (neo_dcp->fixed_connector && neo_dcp->fixed_connector->connected) - return true; - if (neo_dcp->hdmi_hpd && gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) - return true; - - return false; -} - -static bool neo_dcp_typec_route_available(struct neo_apple_dcp_typec_route *route) -{ - return !route->neo_dcp->active_typec_route && - !neo_dcp_typec_route_fixed_output_busy(route); -} - -/* - * Machines whose USB4 docks carry two independent DP streams through one - * port: DPIN0 drives the hybrid dcpext0, DPIN1 the Type-C-only dcpext1, and - * each port has a second connector for the DPIN1 stream. - */ -bool neo_dcp_typec_dual_stream(void) -{ - return false; -} - -bool neo_dcp_is_typec_only(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return !neo_dcp->fixed_phy; -} - -/* - * Can this route feed the given connector without changing its - * possible_crtcs? On dual-stream machines those are fixed at probe, since - * compositors read them once when the connector appears and pair the - * connector with a CRTC before this fabric has routed it. - */ -static bool neo_dcp_typec_route_fits(struct neo_apple_dcp_typec_route *route, - struct neo_apple_connector *connector) -{ - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - - if (!neo_dcp_typec_dual_stream() || !connector || !neo_dcp->crtc) - return true; - return connector->candidate_crtcs & drm_crtc_mask(&neo_dcp->crtc->base); -} - -static int neo_dcp_typec_route_activate(struct neo_apple_dcp_typec_route *route, - struct mux_control *xbar); -static int neo_dcp_typec_route_deactivate(struct neo_apple_dcp_typec_route *route); -static int neo_dcp_dpxbar_preselect(struct mux_control *mux, int state); -static int neo_dcp_dptx_disconnect(struct neo_apple_dcp *neo_dcp, u32 port); - -static int neo_dcp_dpxbar_tunnel_select_source(struct mux_control *mux, int state) -{ - typeof(&apple_dpxbar_tunnel_select_source) select = - symbol_get(apple_dpxbar_tunnel_select_source); - int ret; - - if (!select) - return -ENOENT; - ret = select(mux, state); - symbol_put(apple_dpxbar_tunnel_select_source); - return ret; -} - -/* - * For a port without a prior owner, rank pipelines by CRTC index. A pipeline - * whose fixed output is live is not a candidate at all, so a hybrid is only - * ever ranked here when it is genuinely free. - */ -static unsigned int neo_dcp_typec_route_score(struct neo_apple_dcp_typec_route *route) -{ - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - - if (!neo_dcp->crtc) - return UINT_MAX - 1; - - return drm_crtc_index(&neo_dcp->crtc->base); -} - -static int neo_dcp_typec_route_activate(struct neo_apple_dcp_typec_route *route, - struct mux_control *xbar) -{ - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - struct neo_apple_connector *connector = - xbar != route->xbar && route->tunnel_dpin == 1 ? - route->port->secondary_connector : route->port->connector; - int ret; - - /* - * The fixed output's HPD handler leaves disconnects to DCP, so the port - * can still be marked connected to a display that is gone. Release it - * (a no-op otherwise), or connecting the borrowed route returns early. - */ - neo_dcp_dptx_disconnect(neo_dcp, 0); - - if (neo_dcp->fixed_route_selected) { - ret = mux_control_deselect(neo_dcp->xbar); - if (ret) - return ret; - neo_dcp->fixed_route_selected = false; - } - - /* - * Thunderbolt DP IN: the crossbar connection may only be brought up - * once DCP has configured the link and the tunnel pixel clock runs, - * see dcp_tunnel_crossbar_up(). Just remember the output here. - */ - if (xbar != route->xbar && route->mux_index) { - /* - * The T602X crossbar must point a DP IN at its pipeline before - * DCP probes AUX; other crossbars select it at link-up and - * report -EOPNOTSUPP here. - */ - ret = neo_dcp_dpxbar_tunnel_select_source(xbar, route->mux_index); - if (ret == -EOPNOTSUPP) - ret = 0; - } else { - ret = xbar == route->xbar ? - mux_control_select(xbar, route->mux_index) : 0; - } - if (ret) { - if (neo_dcp->xbar) { - int restore_ret; - - restore_ret = mux_control_select(neo_dcp->xbar, - neo_dcp->fixed_mux_index); - if (!restore_ret) - neo_dcp->fixed_route_selected = true; - else - dev_err(neo_dcp->dev, - "failed to restore fixed display route: %d\n", - restore_ret); - } - return ret; - } - - neo_dcp->phy = route->phy; - neo_dcp->neo_dptx_phy = route->neo_dptx_phy; - neo_dcp->connector_type = DRM_MODE_CONNECTOR_USB; - if (connector) { - WRITE_ONCE(connector->neo_dcp, to_platform_device(neo_dcp->dev)); - neo_dcp->typec_connector = connector; - neo_dcp->connector = connector; - - /* - * Narrow the port to the pipeline now driving it. The encoder - * spans every pipeline that could, which is what lets userspace - * see the port as usable at all -- but only one of them is - * routed to the display, and userspace has no way to tell which. - * Offering it the choice makes it pair the port with a pipeline - * holding a different monitor's mode list, and the modeset is - * rejected with no way for it to recover. The hotplug that - * follows publishes the actual binding on dual-stream machines - * too. Clients must re-read encoder topology after a hotplug. - */ - if (connector->port_encoder && neo_dcp->crtc) - connector->port_encoder->possible_crtcs = - drm_crtc_mask(&neo_dcp->crtc->base); - } - neo_dcp->active_typec_route = route; - scoped_guard(mutex, &neo_dcp->tb_lock) { - route->active_xbar = xbar; - route->tunnel = xbar != route->xbar; - route->xbar_up = !route->tunnel; - neo_dcp->neo_dptx_tunnel = route->tunnel; - /* crossbar controls are dpphy, dpin0, dpin1: same order as the DFP port */ - neo_dcp->neo_dptx_dfp_port = route->tunnel ? xbar - &route->xbar->chip->mux[0] : 0; - neo_dcp->tb_clock_ok = false; - } - route->selected = true; - - dev_info(neo_dcp->dev, "allocated Type-C DPTX PHY %u\n", route->neo_dptx_phy); - return 0; -} - -static int neo_dcp_typec_route_deactivate(struct neo_apple_dcp_typec_route *route) -{ - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - struct neo_apple_connector *connector = neo_dcp->typec_connector; - struct mux_control *active_xbar = route->active_xbar; - bool was_tunnel = route->tunnel; - int ret = 0; - - /* - * Under tb_lock so a DCP link (re)configuration can neither select the - * crossbar nor restart the tunnel pixel clock behind our back. - */ - scoped_guard(mutex, &neo_dcp->tb_lock) { - if (!route->tunnel || route->xbar_up) - ret = mux_control_deselect(route->active_xbar ?: route->xbar); - else - neo_dcp_dpxbar_preselect(route->active_xbar, MUX_IDLE_DISCONNECT); - if (ret) - return ret; - route->xbar_up = false; - - if (route->tunnel && neo_dcp->phy) { - /* the tunnel pixel clock must not outlive the tunnel */ - typeof(&apple_atc_dp_tunnel_rate) stop = - symbol_get(apple_atc_dp_tunnel_rate); - - if (stop) { - stop(neo_dcp->phy, route->tunnel_dpin, 0); - symbol_put(apple_atc_dp_tunnel_rate); - } - } - route->active_xbar = NULL; - route->tunnel = false; - neo_dcp->neo_dptx_tunnel = false; - neo_dcp->neo_dptx_dfp_port = 0; - neo_dcp->tb_dpin_set_active = NULL; - neo_dcp->tb_dpin_ctx = NULL; - neo_dcp->tb_clock_ok = false; - } - if (was_tunnel && route->mux_index) { - int sel = neo_dcp_dpxbar_tunnel_select_source(active_xbar, -1); - - if (sel && sel != -EOPNOTSUPP) - dev_warn(neo_dcp->dev, "DP tunnel source reset failed: %d\n", sel); - } - route->selected = false; - if (neo_dcp->active_typec_route == route) - neo_dcp->active_typec_route = NULL; - - if (connector && connector->neo_dcp == to_platform_device(neo_dcp->dev)) { - /* - * Until the port is activated again it has no pipeline behind - * it, and nothing can read modes or EDID from it. Report it - * disconnected for that window: leaving a connected connector - * whose ->dcp is NULL lets anything probing it in between -- - * a compositor starting up while the fabric is still settling - * -- see an output it cannot get a mode for, and give up on - * it. The new pipeline marks it connected again once the - * display has come back up on it. - */ - WRITE_ONCE(connector->connected, false); - /* hotplug work queued before this checks for it */ - WRITE_ONCE(connector->neo_dcp, NULL); - - /* Unrouted: the port could go to any of its pipelines again. */ - if (connector->port_encoder) - connector->port_encoder->possible_crtcs = - connector->candidate_crtcs; - } - neo_dcp->typec_connector = NULL; - neo_dcp->connector = neo_dcp->fixed_connector; - - if (neo_dcp->fixed_connector_type != DRM_MODE_CONNECTOR_USB) { - neo_dcp->connector_type = neo_dcp->fixed_connector_type; - - /* - * Only hand the pipeline back to its fixed output if that output - * is live. Re-targeting the DPTX endpoint at the fixed PHY while - * nothing is attached there leaves DCP unable to train a link on a - * later Type-C target: it answers DEVICE_NOT_STARTED and every - * following DPTX call times out. Park on a Type-C PHY instead, - * for the same reason the USB-C-only case does below. - */ - if (neo_dcp->hdmi_hpd && gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) { - neo_dcp->phy = neo_dcp->fixed_phy; - neo_dcp->neo_dptx_phy = neo_dcp->fixed_dptx_phy; - - if (neo_dcp->xbar) { - ret = mux_control_select(neo_dcp->xbar, - neo_dcp->fixed_mux_index); - if (ret) - return ret; - neo_dcp->fixed_route_selected = true; - } - } else { - neo_dcp->phy = neo_dcp->typec_routes[0].phy; - neo_dcp->neo_dptx_phy = neo_dcp->typec_routes[0].neo_dptx_phy; - } - } else { - /* Keep DPTX endpoint discovery working before a cable is attached. */ - neo_dcp->phy = neo_dcp->typec_routes[0].phy; - neo_dcp->neo_dptx_phy = neo_dcp->typec_routes[0].neo_dptx_phy; - neo_dcp->connector_type = DRM_MODE_CONNECTOR_USB; - } - - return 0; -} - -static void neo_dcp_typec_retrain_work(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = - container_of(to_delayed_work(work), struct neo_apple_dcp, - typec_fabric_retrain_wq); - - struct neo_apple_connector *connector = READ_ONCE(neo_dcp->typec_connector); - - if (READ_ONCE(neo_dcp->active_typec_route) && connector) - neo_dcp_retrain_oob(connector); -} - -static void neo_dcp_typec_retrain_active_routes(void) -{ - struct neo_apple_dcp_typec_port *port; - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - if (port->owner) - mod_delayed_work(system_freezable_wq, - &port->owner->neo_dcp->typec_fabric_retrain_wq, - msecs_to_jiffies(200)); - if (port->secondary_owner) - mod_delayed_work(system_freezable_wq, - &port->secondary_owner->neo_dcp->typec_fabric_retrain_wq, - msecs_to_jiffies(200)); - } -} - -static struct neo_apple_dcp_typec_route * -neo_dcp_typec_port_route(struct neo_apple_dcp_typec_port *port, struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_dcp_typec_route *route; - - list_for_each_entry(route, &port->routes, port_link) - if (route->neo_dcp == neo_dcp) - return route; - - return NULL; -} - -/* - * The DRM device once it has bound, that is once every pipeline has its - * CRTC and every port its connectors; NULL before. - */ -static struct drm_device *neo_dcp_typec_drm(void) -{ - struct neo_apple_dcp_typec_port *port; - struct neo_apple_dcp_typec_route *route; - struct drm_device *drm = NULL; - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - if (!port->connector || !port->secondary_connector) - return NULL; - list_for_each_entry(route, &port->routes, port_link) { - if (!route->neo_dcp->crtc) - return NULL; - drm = route->neo_dcp->crtc->base.dev; - } - } - - return drm; -} - -/* - * Has a compositor (or boot splash) taken the display? It paired its - * connectors with CRTCs when it started and keeps that pairing, also - * across a VT switch, where it drops DRM master but keeps the device - * open; moving routes under it would hand connectors to pipelines driving - * other displays. So the routes are frozen while any open file has ever - * been master, and thaw when the last such file is closed. - * - * Taken under dcp_typec_fabric_lock, next to the routing it decides, so - * the order is the fabric lock, then filelist_mutex. Nothing nests them - * the other way: filelist_mutex only covers list edits and walks, and - * drm_release() drops it before drm_file_free() calls postclose, which is - * where a close takes the fabric lock. drm_open() makes a file master - * before adding it here, but its owner cannot have paired anything before - * open() returns, and it re-probes on the hotplugs the moves send. - */ -static bool neo_dcp_typec_frozen(struct drm_device *drm) -{ - struct drm_file *file; - - guard(mutex)(&drm->filelist_mutex); - list_for_each_entry(file, &drm->filelist, lhead) { - /* set under master_mutex, and only ever from false to true */ - if (READ_ONCE(file->was_master)) - return true; - } - - return false; -} - -/* Are the routes kept in compositor pairing order right now? */ -static bool neo_dcp_typec_keep_order(void) -{ - struct drm_device *drm; - - if (!neo_dcp_typec_dual_stream()) - return false; - drm = neo_dcp_typec_drm(); - - return drm && READ_ONCE(drm->registered) && !neo_dcp_typec_frozen(drm); -} - -/* A Thunderbolt tunnel holds @dcp's pipeline: it never moves. */ -static bool neo_dcp_typec_tunnel_held(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->active_typec_route && neo_dcp->active_typec_route->tunnel; -} - -/* - * Give @dcp's pipeline to the first stream, in connector order, that - * wants one, has none yet and has it in its possible_crtcs. A tunnel - * stream wants one once it is set up, or as @arriving/@dpin. A direct - * DP-alt stream wants one once its sink asserts HPD: only then can its - * connector read connected, and a compositor pairs only those. - */ -static void neo_dcp_typec_plan_pipeline(struct neo_apple_dcp *neo_dcp, struct drm_crtc *crtc, - struct neo_apple_dcp_typec_port *arriving, - unsigned int dpin) -{ - struct neo_apple_dcp_typec_port *port; - bool secondary = false; - - /* primary connectors in port order, then the DPIN1 ones */ - do { - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - struct neo_apple_dcp_typec_route **slot = secondary ? - &port->secondary_target : &port->target; - struct neo_apple_connector *connector = secondary ? - port->secondary_connector : port->connector; - struct neo_apple_dcp_typec_route *route = - neo_dcp_typec_port_route(port, neo_dcp); - bool wants; - - if (secondary) - wants = port->secondary_owner || - (port == arriving && dpin == 1); - else - wants = (port->owner && port->owner->tunnel) || - (port->dp_wanted && port->dp_hpd && - !port->plan_dark) || - (port == arriving && dpin == 0); - - if (!wants || *slot || !route || - !(connector->candidate_crtcs & drm_crtc_mask(crtc))) - continue; - - *slot = route; - return; - } - secondary = !secondary; - } while (secondary); -} - -/* - * The pairing a compositor starting now would make, wherever the streams - * sit at the moment: pipelines in CRTC index order, each to the first - * stream that wants it. A pipeline whose fixed output is live is left to - * that output, unless a tunnel holds it: the output cannot have it back - * then, and its connector reads disconnected. - * - * Tunnels are planned in connector order like any other stream, not - * seated on the pipelines they hold: the compositor pairs their connectors - * by order too, and seating them would plan the direct streams around - * pairings it never makes. A direct stream planned onto a pipeline a - * tunnel holds cannot have it and stays dark, so it is left out and the - * pass re-run, or every stream after it would be planned one pipeline - * off. Streams only ever drop out, so that settles within one pass per - * port. It also brings a tunnel's plan onto the pipeline it holds - * wherever direct streams ahead of it were in the way. A tunnel still - * planned elsewhere cannot be brought there by any direct stream dropping - * out, and the compositor pairs its connector wrongly whatever they do. - */ -static void neo_dcp_typec_plan(struct drm_device *drm, - struct neo_apple_dcp_typec_port *arriving, unsigned int dpin) -{ - struct neo_apple_dcp_typec_port *port; - struct drm_crtc *crtc; - bool again; - - list_for_each_entry(port, &neo_dcp_typec_ports, link) - port->plan_dark = false; - - do { - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - port->target = NULL; - port->secondary_target = NULL; - } - - /* CRTCs are listed in index order */ - drm_for_each_crtc(crtc, drm) { - struct neo_apple_dcp *neo_dcp = - platform_get_drvdata(to_apple_crtc(crtc)->neo_dcp); - - if (neo_dcp->nr_typec_routes && - (neo_dcp_typec_tunnel_held(neo_dcp) || - !neo_dcp_typec_route_fixed_output_busy(&neo_dcp->typec_routes[0]))) - neo_dcp_typec_plan_pipeline(neo_dcp, crtc, arriving, dpin); - } - - again = false; - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - if (port == arriving || !port->target || - (port->owner && port->owner->tunnel) || - !neo_dcp_typec_tunnel_held(port->target->neo_dcp)) - continue; - port->plan_dark = true; - again = true; - } - } while (again); -} - -/* Replay HPD to the pipeline a direct DP-alt port has just been given. */ -static void neo_dcp_typec_port_attach(struct neo_apple_dcp_typec_port *port) -{ - struct neo_apple_dcp *neo_dcp = port->owner->neo_dcp; - - port->hpd = port->dp_hpd; - if (!port->hpd) - return; - - WRITE_ONCE(neo_dcp->typec_cable_connected, true); - if (neo_dcp->typec_connector) - neo_dcp_dptx_connect_oob(to_platform_device(neo_dcp->dev), 0); -} - -static struct neo_apple_dcp_typec_route * -neo_dcp_typec_lowest_free(struct neo_apple_dcp_typec_port *port) -{ - struct neo_apple_dcp_typec_route *candidate, *best = NULL; - unsigned int best_score = UINT_MAX; - - list_for_each_entry(candidate, &port->routes, port_link) { - unsigned int score; - - if (!neo_dcp_typec_route_available(candidate)) - continue; - /* - * Prefer the lowest free CRTC index for a stable allocation order. - */ - score = neo_dcp_typec_route_score(candidate); - if (score < best_score) { - best = candidate; - best_score = score; - } - } - - return best; -} - -/* - * Route the direct DP-alt ports left waiting for a pipeline. Their DP - * state is not reported again while it stays the same, so nothing else - * would. Each goes back to the pipeline it last had if that is free, as - * a compositor keeps a reconnected connector's CRTC, and otherwise takes - * the lowest free one, the CRTC a compositor gives a new connector. - */ -static void neo_dcp_typec_route_waiting(void) -{ - struct neo_apple_dcp_typec_port *port; - struct neo_apple_dcp_typec_route *route; - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - if (port->owner || !port->dp_wanted || !port->dp_hpd) - continue; - - route = port->preferred_route; - if (!route || !neo_dcp_typec_route_available(route)) - route = neo_dcp_typec_lowest_free(port); - if (!route || neo_dcp_typec_route_activate(route, route->xbar)) - continue; - port->owner = route; - port->dp_release_deadline = 0; - neo_dcp_typec_port_attach(port); - } -} - -/* - * Prefer connector order when placing Type-C streams before a DRM master - * starts. This is an allocation policy, not a promise about the CRTC that - * userspace must choose: each connected encoder advertises only its actual - * route, and atomic encoder selection checks that ownership. Route changes - * publish the new mask through hotplug; the plan uses candidate_crtcs so - * narrowing the advertised mask does not remove hardware routing choices. - * - * Only direct DP-alt routes move: each goes to exactly its planned - * pipeline. A Thunderbolt tunnel never moves once set up, so where one - * sits on a pipeline planned for a direct stream, or the plan has nothing - * for it, that stream stays unrouted and its connector disconnected. - * Placing it on some other pipeline would have the compositor cross both - * displays; dark is the better failure. @arriving/@dpin is a tunnel - * stream asking for a pipeline, planned like any other; its route is - * returned for apple_dcp_tb_dp_tunnel() to set up, or NULL if the plan - * has none for it or another tunnel holds that one. - * - * A move is an unplug and replug. Every route that moves is taken down - * first, so that two never share a pipeline midway, then each goes up on - * its new pipeline with its HPD replayed; the hotplugs this sends make - * fbdev and userspace re-probe. - */ -static struct neo_apple_dcp_typec_route * -neo_dcp_typec_rebalance_locked(struct neo_apple_dcp_typec_port *arriving, - unsigned int dpin) -{ - struct drm_device *drm = neo_dcp_typec_drm(); - struct neo_apple_dcp_typec_route *planned = NULL; - struct neo_apple_dcp_typec_port *port; - - lockdep_assert_held(&neo_dcp_typec_fabric_lock); - - neo_dcp_typec_plan(drm, arriving, dpin); - if (arriving) { - planned = dpin ? arriving->secondary_target : arriving->target; - /* it will be refused: plan as if it had not asked */ - if (planned && neo_dcp_typec_tunnel_held(planned->neo_dcp)) { - planned = NULL; - neo_dcp_typec_plan(drm, NULL, 0); - } - } - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - struct neo_apple_dcp_typec_route *owner = port->owner; - struct neo_apple_dcp *neo_dcp; - - if (!owner || owner->tunnel || owner == port->target) - continue; - - neo_dcp = owner->neo_dcp; - dev_info(neo_dcp->dev, "re-routing %pOF from %s to %s for Type-C connector order\n", - port->connector_np, dev_name(neo_dcp->dev), - port->target ? dev_name(port->target->neo_dcp->dev) : "none"); - if (port->hpd || neo_dcp->typec_cable_connected || - (neo_dcp->typec_connector && neo_dcp->typec_connector->connected)) - neo_dcp_dptx_disconnect_oob(to_platform_device(neo_dcp->dev), 0); - port->hpd = false; - if (neo_dcp_typec_route_deactivate(owner) && owner->selected) { - /* still routed: leave the display where it is */ - neo_dcp_typec_port_attach(port); - continue; - } - port->owner = NULL; - - /* as after a DP exit, hand the hybrid back to a live HDMI */ - if (neo_dcp->hdmi_hpd && neo_dcp->active && - gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) - neo_dcp_dptx_connect(neo_dcp, 0); - } - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - struct neo_apple_dcp_typec_route *route = port->target; - - if (port->owner || !port->dp_wanted || !port->dp_hpd || - port == arriving) - continue; - - if (!route || !neo_dcp_typec_route_available(route) || - neo_dcp_typec_route_activate(route, route->xbar)) { - /* - * Dark for now. Its unchanged DP state is not reported - * again, so it is placed when the plan next runs, or - * once a compositor owns the display, when a pipeline - * is freed (dcp_typec_route_waiting()). - */ - port->applied_valid = false; - continue; - } - port->owner = route; - port->preferred_route = route; - port->dp_release_deadline = 0; - neo_dcp_typec_port_attach(port); - } - - return planned; -} - -/* - * A route has let its pipeline go. Until a compositor owns the display - * the plan places everything anew; after that the pipeline goes to a port - * left waiting for one. - */ -static void neo_dcp_typec_pipeline_freed(void) -{ - if (!neo_dcp_typec_dual_stream()) - return; - - if (neo_dcp_typec_keep_order()) - neo_dcp_typec_rebalance_locked(NULL, 0); - else - neo_dcp_typec_route_waiting(); -} - -/* - * Re-run the pairing pass from outside the fabric: once DRM is registered, - * as ports routed before that could not follow it, and when the last - * compositor or boot splash has closed the device, as the next one pairs - * the connectors from scratch. - */ -void neo_dcp_typec_reorder(void) -{ - if (!neo_dcp_typec_dual_stream()) - return; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - if (neo_dcp_typec_keep_order()) - neo_dcp_typec_rebalance_locked(NULL, 0); -} - -static int neo_dcp_typec_route_set(struct typec_mux_dev *mux, - struct typec_mux_state *state) -{ - struct neo_apple_dcp_typec_route *route = typec_mux_get_drvdata(mux); - struct neo_apple_dcp_typec_port *port = route->port; - struct neo_apple_dcp_typec_route *best = NULL; - bool is_dp = neo_dcp_typec_route_is_dp(state); - struct typec_displayport_data *dp_data = is_dp ? state->data : NULL; - u32 dp_status = dp_data ? dp_data->status : 0; - u32 dp_conf = dp_data ? dp_data->conf : 0; - bool hpd, was_counted; - int ret = 0; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - /* - * Every candidate route for this port is notified with the same state, - * so only the first to arrive does the work; the others return early. - * - * Deliberately not a nominated coordinator: fwnode_typec_mux_get() - * caps the providers one connector may have and drops the remainder - * without a word, so a nominated route might never be called at all -- - * and the port would then never be routed. - */ - if (port->applied_valid && port->applied_alt == state->alt && - port->applied_mode == state->mode && - port->applied_status == dp_status && port->applied_conf == dp_conf) - return 0; - - port->applied_alt = state->alt; - port->applied_mode = state->mode; - port->applied_status = dp_status; - port->applied_conf = dp_conf; - /* Failed route acquisition must remain retryable on the next update. */ - port->applied_valid = false; - - /* did the pairing pass count this port's direct stream so far? */ - was_counted = port->dp_wanted && port->dp_hpd; - - if (!is_dp) { - port->dp_wanted = false; - port->dp_hpd = false; - - /* a Thunderbolt/USB4 DP tunnel is torn down by its own path */ - if (port->owner && port->owner->tunnel) { - port->applied_valid = true; - return 0; - } - if (port->owner) { - struct neo_apple_dcp *neo_dcp = port->owner->neo_dcp; - - port->preferred_route = port->owner; - if (port->hpd || neo_dcp->typec_cable_connected || - (neo_dcp->typec_connector && - neo_dcp->typec_connector->connected)) - neo_dcp_dptx_disconnect_oob(to_platform_device(neo_dcp->dev), 0); - port->hpd = false; - ret = neo_dcp_typec_route_deactivate(port->owner); - if (ret) - return ret; - port->owner = NULL; - port->dp_release_deadline = jiffies + msecs_to_jiffies(10000); - if (neo_dcp->hdmi_hpd && neo_dcp->active && - gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) - neo_dcp_dptx_connect(neo_dcp, 0); - neo_dcp_typec_pipeline_freed(); - } else if (was_counted && neo_dcp_typec_keep_order()) { - /* a stream the plan left dark is gone: plan the rest */ - neo_dcp_typec_rebalance_locked(NULL, 0); - } - - /* - * A port leaving DP can report SAFE/NONE before falling back to USB4. - * Resetting every other live CRTC for that same cable removal blanks - * unaffected displays. Keep the guard across the Type-C state sequence; - * a later, independent USB4 attach still gets recovery. - */ - if (state->mode == TYPEC_MODE_USB4) { - if (!port->dp_release_deadline || - time_after_eq(jiffies, port->dp_release_deadline)) - neo_dcp_typec_retrain_active_routes(); - port->dp_release_deadline = 0; - } - port->applied_valid = true; - return 0; - } - - /* - * A Thunderbolt DP tunnel still owns the port (its teardown is on the - * way): don't act on or remember this state; the tunnel teardown drops - * what was recorded so the next update is applied. - */ - if (port->owner && port->owner->tunnel) { - port->applied_valid = false; - return 0; - } - - hpd = dp_data && (dp_data->status & DP_STATUS_HPD_STATE); - port->dp_wanted = true; - port->dp_hpd = hpd; - - /* - * Until a compositor owns the display, a direct DP-alt stream has a - * pipeline only while its sink asserts HPD, and the pairing pass - * places it: HPD coming or going is the stream connecting or - * disconnecting as far as a compositor can tell. Otherwise the port - * takes the lowest free pipeline on DP entry, as a compositor does. - */ - if (neo_dcp_typec_keep_order()) { - struct neo_apple_dcp_typec_route *owner = port->owner; - - /* it connects or disconnects, or is not routed as it should be */ - if (hpd != was_counted || !owner != !hpd) - neo_dcp_typec_rebalance_locked(NULL, 0); - if (!port->owner) { - if (hpd) - return -EBUSY; - port->applied_valid = true; - return 0; - } - /* just attached, its HPD replayed: nothing left to apply */ - if (port->owner != owner) { - port->applied_valid = true; - return 0; - } - } - - if (!port->owner) { - if (port->preferred_route && - neo_dcp_typec_route_available(port->preferred_route)) - best = port->preferred_route; - - if (!best) - best = neo_dcp_typec_lowest_free(port); - - if (!best) - return -EBUSY; - - ret = neo_dcp_typec_route_activate(best, best->xbar); - if (ret) - return ret; - port->owner = best; - port->dp_release_deadline = 0; - } - - - if (!hpd && port->hpd) { - neo_dcp_dptx_disconnect_oob(to_platform_device(port->owner->neo_dcp->dev), 0); - } else if (hpd && !port->hpd) { - struct neo_apple_dcp *neo_dcp = port->owner->neo_dcp; - - WRITE_ONCE(neo_dcp->typec_cable_connected, true); - if (neo_dcp->typec_connector) - neo_dcp_dptx_connect_oob(to_platform_device(neo_dcp->dev), 0); - } else if (hpd && dp_data && (dp_data->status & DP_STATUS_IRQ_HPD)) { - struct neo_apple_dcp *neo_dcp = port->owner->neo_dcp; - - if (neo_dcp->typec_connector) - neo_dcp_retrain_oob(neo_dcp->typec_connector); - } - port->hpd = hpd; - port->applied_valid = true; - - return 0; -} - -/* - * Crossbar connection up/down, looked up at runtime so appledrm_neo does not - * require the crossbar driver to be built. - */ -static int neo_dcp_dpxbar_link(struct mux_control *mux, bool up) -{ - typeof(&apple_dpxbar_link_up) fn; - int ret; - - fn = up ? symbol_get(apple_dpxbar_link_up) : - symbol_get(apple_dpxbar_link_down); - if (!fn) - return -ENOENT; - ret = fn(mux); - if (up) - symbol_put(apple_dpxbar_link_up); - else - symbol_put(apple_dpxbar_link_down); - return ret; -} - -static int neo_dcp_dpxbar_preselect(struct mux_control *mux, int state) -{ - typeof(&apple_dpxbar_preselect) fn = symbol_get(apple_dpxbar_preselect); - int ret; - - if (!fn) - return -ENOENT; - ret = fn(mux, state); - symbol_put(apple_dpxbar_preselect); - return ret; -} - -/* - * Thunderbolt DP IN, before DCP hears about the display: point the DP IN - * output at this pipeline and wake the ATC's DP clock path. The crossbar - * connection itself still only comes up in dcp_tunnel_crossbar_up(), but an - * output left at its idle source (dispext0) only suits the pipeline that is - * dispext0: any other one activates the link, gets no answer from the sink - * and gives up with DEVICE_NOT_RESPONDING before it ever sets a link rate. - */ -static void neo_dcp_tunnel_prepare(struct neo_apple_dcp_typec_route *route, - struct mux_control *xbar) -{ - typeof(&apple_atc_dp_tunnel_open) open; - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - int ret; - - ret = neo_dcp_dpxbar_preselect(xbar, route->mux_index); - if (ret && ret != -EOPNOTSUPP) - dev_warn(neo_dcp->dev, "DP tunnel crossbar preselect failed: %d\n", ret); - - open = symbol_get(apple_atc_dp_tunnel_open); - if (!open) - return; - ret = open(neo_dcp->phy); - symbol_put(apple_atc_dp_tunnel_open); - if (ret && ret != -EOPNOTSUPP) - dev_warn(neo_dcp->dev, "DP tunnel PHY open failed: %d\n", ret); -} - -/* DP IN adapter handshake through the thunderbolt glue; tb_lock held */ -static int neo_dcp_tunnel_dpin_locked(struct neo_apple_dcp *neo_dcp, bool active) -{ - lockdep_assert_held(&neo_dcp->tb_lock); - if (!neo_dcp->tb_dpin_set_active) - return -ENODEV; - return neo_dcp->tb_dpin_set_active(neo_dcp->tb_dpin_ctx, active); -} - -/* - * Thunderbolt DP IN, from DCP's DidChangeLinkConfiguration once a link rate - * is set: bring the crossbar connection up (FIFO/PCLK/ATC enables) now that - * the tunnel pixel clock runs, and re-assert the DP IN adapter's - * DPTX_INACTIVE=0 afterwards. The first time this is the mux selection; after - * a re-link only the clocks are brought back (the mux selection and the ATC - * output enable are kept, see dcp_tunnel_crossbar_down()). - * Runs inside a DCP apcall: only tb_lock, which nobody holds across a DCP call. - */ -int neo_dcp_tunnel_crossbar_up(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_dcp_typec_route *route; - int ret; - - guard(mutex)(&neo_dcp->tb_lock); - route = neo_dcp->active_typec_route; - if (!route || !route->tunnel || !route->active_xbar) - return -ENODEV; - if (!neo_dcp->tb_clock_ok) { - dev_warn(neo_dcp->dev, "no DP tunnel pixel clock, crossbar left down\n"); - return -EIO; - } - if (!route->xbar_up) { - /* never block a DCP call on the mux semaphore */ - ret = mux_control_try_select(route->active_xbar, route->mux_index); - if (!ret) - route->xbar_up = true; - } else { - ret = neo_dcp_dpxbar_link(route->active_xbar, true); - } - if (ret) { - dev_warn(neo_dcp->dev, "DP tunnel crossbar up failed: %d\n", ret); - return ret; - } - return neo_dcp_tunnel_dpin_locked(neo_dcp, true); -} - -/* - * Thunderbolt DP IN, from DCP's WillChangeLinkConfiguration on an established - * link: DP IN inactive, crossbar clocks down (mux selection and ATC output - * enable kept). DP IN goes active again in dcp_tunnel_crossbar_up(). - */ -int neo_dcp_tunnel_crossbar_down(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_dcp_typec_route *route; - - guard(mutex)(&neo_dcp->tb_lock); - route = neo_dcp->active_typec_route; - if (!route || !route->tunnel || !route->xbar_up) - return 0; - neo_dcp_tunnel_dpin_locked(neo_dcp, false); - return neo_dcp_dpxbar_link(route->active_xbar, false); -} - -/* - * Thunderbolt DP IN, from DCP's SetLinkRate: start (rate != 0) or stop the - * tunnel pixel clock. A stopped clock also takes the crossbar connection down. - */ -int neo_dcp_tunnel_set_rate(struct neo_apple_dcp *neo_dcp, struct phy *phy, u32 link_rate) -{ - typeof(&apple_atc_dp_tunnel_rate) fn; - struct neo_apple_dcp_typec_route *route; - int ret; - - guard(mutex)(&neo_dcp->tb_lock); - if (!neo_dcp->neo_dptx_tunnel) - return -ENODEV; - fn = symbol_get(apple_atc_dp_tunnel_rate); - if (!fn) { - dev_err(neo_dcp->dev, "phy-apple-atc not loaded, no DP tunnel clock\n"); - return -ENOENT; - } - route = neo_dcp->active_typec_route; - if (!route) { - symbol_put(apple_atc_dp_tunnel_rate); - return -ENODEV; - } - if (!link_rate && route->xbar_up) - neo_dcp_dpxbar_link(route->active_xbar, false); - ret = fn(phy, route->tunnel_dpin, link_rate); - symbol_put(apple_atc_dp_tunnel_rate); - neo_dcp->tb_clock_ok = !ret && link_rate; - if (ret) - dev_warn(neo_dcp->dev, "DP tunnel pixel clock (rate 0x%x) failed: %d\n", - link_rate, ret); - return ret; -} - -/* Thunderbolt DP IN: DCP Activate/Deactivate */ -int neo_dcp_tunnel_dpin_activate(struct neo_apple_dcp *neo_dcp, bool active) -{ - guard(mutex)(&neo_dcp->tb_lock); - if (!neo_dcp->neo_dptx_tunnel) - return 0; - return neo_dcp_tunnel_dpin_locked(neo_dcp, active); -} - -/* - * Thunderbolt DP tunnels: the host router's DP IN adapters sit behind the - * crossbar's dpin0/dpin1 outputs of the port's ATC. When the Thunderbolt - * connection manager has set up a tunnel from one of them, route a free - * display pipeline there and tell DCP a display is attached, so it trains - * the link (and completes DPRX) through the tunnel. - */ -int neo_apple_dcp_tb_dp_tunnel(struct device_node *connector_np, unsigned int dpin, - bool active, int (*set_active)(void *ctx, bool active), - void *ctx) -{ - struct neo_apple_dcp_typec_port *port = NULL, *pos; - struct neo_apple_dcp_typec_route *candidate, *best = NULL, *planned = NULL; - struct neo_apple_dcp_typec_route **slot; - unsigned int best_score = UINT_MAX; - struct mux_control *ctl; - struct neo_apple_dcp *neo_dcp; - bool ordered; - int ret; - - if (!connector_np || dpin > 1) - return -EINVAL; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - list_for_each_entry(pos, &neo_dcp_typec_ports, link) { - if (pos->connector_np == connector_np) { - port = pos; - break; - } - } - if (!port) - return -ENODEV; - slot = dpin ? &port->secondary_owner : &port->owner; - - if (!active) { - if (!*slot || !(*slot)->tunnel || - (*slot)->tunnel_dpin != dpin) - return 0; - neo_dcp = (*slot)->neo_dcp; - if (port->hpd || neo_dcp->typec_cable_connected) - neo_dcp_dptx_disconnect_oob(to_platform_device(neo_dcp->dev), 0); - ret = neo_dcp_typec_route_deactivate(*slot); - if (ret) { - /* the caller's context is going away regardless */ - scoped_guard(mutex, &neo_dcp->tb_lock) { - neo_dcp->neo_dptx_tunnel = false; - neo_dcp->tb_dpin_set_active = NULL; - neo_dcp->tb_dpin_ctx = NULL; - } - return ret; - } - *slot = NULL; - port->hpd = !!(port->owner || port->secondary_owner); - /* re-apply the next Type-C mux state in full */ - port->applied_valid = false; - if (neo_dcp->hdmi_hpd && neo_dcp->active && - gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) - neo_dcp_dptx_connect(neo_dcp, 0); - neo_dcp_typec_pipeline_freed(); - return 0; - } - - if (*slot) { - if ((*slot)->tunnel && (*slot)->tunnel_dpin == dpin) - return 0; - dev_warn((*slot)->neo_dcp->dev, - "port already routed, not taking DP tunnel dpin%u\n", dpin); - return -EBUSY; - } - if (port->owner && !port->owner->tunnel) - return -EBUSY; - - /* - * Until a compositor owns the display, the pairing pass decides where - * the stream goes, moving direct DP-alt routes out of its way, and - * whether it gets a pipeline at all. - */ - ordered = neo_dcp_typec_keep_order(); - if (ordered) - planned = neo_dcp_typec_rebalance_locked(port, dpin); - - list_for_each_entry(candidate, &port->routes, port_link) { - unsigned int score; - - if (ordered && candidate != planned) - continue; - if (!neo_dcp_typec_route_available(candidate)) - continue; - if (!neo_dcp_typec_route_fits(candidate, dpin ? - port->secondary_connector : - port->connector)) - continue; - score = neo_dcp_typec_route_score(candidate); - /* - * DPIN0 prefers the hybrid dcpext0, which completes tunneled - * link training. On dual-stream machines DPIN1 is confined to - * dcpext1 by its connector's possible_crtcs, so both pipelines - * drive independent streams through one dock. - */ - if (neo_dcp_typec_dual_stream() && dpin == 0 && - !candidate->neo_dcp->fixed_phy && score < UINT_MAX - 100) - score += 100; - if (score < best_score) { - best = candidate; - best_score = score; - } - } - if (!best) { - ret = -EBUSY; - goto err_reorder; - } - - /* The route's crossbar control is dpphy (0); dpin0/dpin1 are 1/2. */ - if (best->xbar != &best->xbar->chip->mux[0] || - best->xbar->chip->controllers < 3) { - ret = -EOPNOTSUPP; - goto err_reorder; - } - ctl = &best->xbar->chip->mux[1 + dpin]; - - neo_dcp = best->neo_dcp; - scoped_guard(mutex, &neo_dcp->tb_lock) { - neo_dcp->tb_dpin_set_active = set_active; - neo_dcp->tb_dpin_ctx = ctx; - } - best->tunnel_dpin = dpin; - ret = neo_dcp_typec_route_activate(best, ctl); - if (ret) { - scoped_guard(mutex, &neo_dcp->tb_lock) { - neo_dcp->tb_dpin_set_active = NULL; - neo_dcp->tb_dpin_ctx = NULL; - } - goto err_reorder; - } - *slot = best; - /* the port is in USB4 mode, not DP-alt */ - port->dp_wanted = false; - neo_dcp_tunnel_prepare(best, ctl); - - dev_info(neo_dcp->dev, "display routed to Thunderbolt DP tunnel dpin%u\n", dpin); - - /* - * The DP IN adapter may only be woken (DPTX_INACTIVE=0) while DCP - * drives the DPTX, i.e. from DCP's Activate call; waking it earlier - * hangs the machine. dptxep calls set_active back from Activate and - * Deactivate (set above, before the route became a tunnel). - */ - if (!neo_dcp->typec_connector) - dev_warn(neo_dcp->dev, "no Type-C connector for the DP tunnel\n"); - WRITE_ONCE(neo_dcp->typec_cable_connected, true); - port->hpd = true; - if (neo_dcp->typec_connector) - neo_dcp_dptx_connect_oob(to_platform_device(neo_dcp->dev), 0); - - return 0; - -err_reorder: - /* the pass kept a pipeline for this stream: give it to the others */ - if (planned) - neo_dcp_typec_rebalance_locked(NULL, 0); - return ret; -} -EXPORT_SYMBOL_GPL(neo_apple_dcp_tb_dp_tunnel); - -static struct neo_apple_dcp_typec_port * -neo_dcp_typec_port_get(struct device_node *connector_np) -{ - struct neo_apple_dcp_typec_port *port, *pos; - - lockdep_assert_held(&neo_dcp_typec_fabric_lock); - - list_for_each_entry(port, &neo_dcp_typec_ports, link) { - if (port->connector_np == connector_np) { - of_node_put(connector_np); - return port; - } - } - - port = kzalloc_obj(*port); - if (!port) { - of_node_put(connector_np); - return NULL; - } - - INIT_LIST_HEAD(&port->routes); - port->connector_np = connector_np; - - /* - * Insert in device-tree order rather than DCP probe order. The list - * index becomes the DRM connector index, and userspace keys its - * per-monitor configuration (scale, rotation, layout) on the connector - * name -- so it has to mean the same physical port on every boot. - */ - list_for_each_entry(pos, &neo_dcp_typec_ports, link) - if (strcmp(of_node_full_name(connector_np), - of_node_full_name(pos->connector_np)) < 0) - break; - list_add_tail(&port->link, &pos->link); - return port; -} - -static struct neo_apple_dcp_typec_port *neo_dcp_typec_port_by_index(unsigned int idx) -{ - struct neo_apple_dcp_typec_port *port; - unsigned int i = 0; - - lockdep_assert_held(&neo_dcp_typec_fabric_lock); - - list_for_each_entry(port, &neo_dcp_typec_ports, link) - if (i++ == idx) - return port; - - return NULL; -} - -unsigned int neo_dcp_typec_nr_ports(void) -{ - struct neo_apple_dcp_typec_port *port; - unsigned int n = 0; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - list_for_each_entry(port, &neo_dcp_typec_ports, link) - n++; - - return n; -} - -struct device_node *neo_dcp_typec_port_of_node(unsigned int idx) -{ - struct neo_apple_dcp_typec_port *port; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - port = neo_dcp_typec_port_by_index(idx); - - return port ? port->connector_np : NULL; -} - -bool neo_dcp_typec_port_has_candidate(unsigned int idx, struct platform_device *pdev) -{ - struct neo_apple_dcp_typec_port *port; - struct neo_apple_dcp_typec_route *route; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - port = neo_dcp_typec_port_by_index(idx); - if (!port) - return false; - - list_for_each_entry(route, &port->routes, port_link) - if (route->neo_dcp->dev == &pdev->dev) - return true; - - return false; -} - -void neo_dcp_typec_port_set_connector(unsigned int idx, bool secondary, - struct neo_apple_connector *connector) -{ - struct neo_apple_dcp_typec_port *port; - struct neo_apple_dcp_typec_route *owner; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - port = neo_dcp_typec_port_by_index(idx); - if (!port) - return; - - if (secondary) { - port->secondary_connector = connector; - owner = port->secondary_owner; - } else { - port->connector = connector; - owner = port->owner; - } - - /* - * The port may already have been routed, either before DRM bound or - * while these connectors were being created. Adopt that owner now, - * otherwise its display would be reported on the pipeline's fixed - * connector instead of the port it is actually plugged into. - */ - if (owner) { - struct neo_apple_dcp *neo_dcp = owner->neo_dcp; - - if (neo_dcp->crtc && connector->port_encoder) - connector->port_encoder->possible_crtcs = - drm_crtc_mask(&neo_dcp->crtc->base); - - connector->neo_dcp = to_platform_device(neo_dcp->dev); - neo_dcp->typec_connector = connector; - neo_dcp->connector = connector; - neo_dcp->connector_type = DRM_MODE_CONNECTOR_USB; - } -} - -static void neo_dcp_typec_route_unregister(void *data) -{ - struct neo_apple_dcp_typec_route *route = data; - struct neo_apple_dcp_typec_port *port = route->port; - - typec_mux_unregister(route->typec_mux); - - guard(mutex)(&neo_dcp_typec_fabric_lock); - if (port->preferred_route == route) - port->preferred_route = NULL; - if (port->owner == route) { - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - - if (port->hpd || neo_dcp->typec_cable_connected) - neo_dcp_dptx_disconnect_oob(to_platform_device(neo_dcp->dev), 0); - port->hpd = false; - neo_dcp_typec_route_deactivate(route); - port->owner = NULL; - } - if (port->secondary_owner == route) { - struct neo_apple_dcp *neo_dcp = route->neo_dcp; - - if (port->hpd || neo_dcp->typec_cable_connected) - neo_dcp_dptx_disconnect_oob(to_platform_device(neo_dcp->dev), 0); - neo_dcp_typec_route_deactivate(route); - port->secondary_owner = NULL; - } - port->hpd = !!(port->owner || port->secondary_owner); - list_del(&route->port_link); - if (list_empty(&port->routes)) { - list_del(&port->link); - of_node_put(port->connector_np); - kfree(port); - } -} - -static int neo_dcp_register_typec_routes(struct neo_apple_dcp *neo_dcp) -{ - struct device_node *routes __free(device_node) = - of_get_child_by_name(neo_dcp->dev->of_node, "typec-routes"); - struct device *dev = neo_dcp->dev; - u32 route_index; - int ret; - - if (!routes) - return 0; - - for_each_available_child_of_node_scoped(routes, route_np) { - struct neo_apple_dcp_typec_route *route; - struct device_node *endpoint __free(device_node) = NULL; - struct device_node *connector_np; - struct typec_mux_desc desc = {}; - const char *name, *mux_name; - - if (neo_dcp->nr_typec_routes == DCP_MAX_TYPEC_ROUTES) - return dev_err_probe(dev, -E2BIG, "Too many Type-C display routes\n"); - - ret = of_property_read_u32(route_np, "reg", &route_index); - if (ret) - return dev_err_probe(dev, ret, "%pOF: missing route index\n", route_np); - if (route_index >= DCP_MAX_TYPEC_ROUTES) - return dev_err_probe(dev, -EINVAL, "%pOF: invalid route index %u\n", - route_np, route_index); - - name = devm_kasprintf(dev, GFP_KERNEL, "typec%u", route_index); - if (!name) - return -ENOMEM; - - /* - * The DT lookups above are per-DCP, but the typec_mux class is - * global. Several DCPs can offer a route to the same Type-C port, - * so the registered mux needs a name unique across all of them. - */ - mux_name = devm_kasprintf(dev, GFP_KERNEL, "%s-typec%u", - dev_name(dev), route_index); - if (!mux_name) - return -ENOMEM; - - route = &neo_dcp->typec_routes[neo_dcp->nr_typec_routes]; - route->neo_dcp = neo_dcp; - INIT_LIST_HEAD(&route->port_link); - route->phy = devm_phy_get(dev, name); - if (IS_ERR(route->phy)) - return dev_err_probe(dev, PTR_ERR(route->phy), - "%pOF: failed to get DP PHY\n", route_np); - - route->xbar = devm_mux_control_get(dev, name); - if (IS_ERR(route->xbar)) - return dev_err_probe(dev, PTR_ERR(route->xbar), - "%pOF: failed to get display crossbar\n", route_np); - - ret = of_property_read_u32_index(dev->of_node, "apple,typec-mux-indices", - route_index, &route->mux_index); - if (ret) - return dev_err_probe(dev, ret, "%pOF: missing crossbar state\n", - route_np); - - ret = of_property_read_u32_index(dev->of_node, "apple,typec-dptx-phys", - route_index, &route->neo_dptx_phy); - if (ret) - return dev_err_probe(dev, ret, "%pOF: missing DPTX PHY index\n", - route_np); - - endpoint = of_graph_get_next_endpoint(route_np, NULL); - if (!endpoint) - return dev_err_probe(dev, -EINVAL, - "%pOF: missing Type-C graph endpoint\n", - route_np); - connector_np = of_graph_get_remote_port_parent(endpoint); - if (!connector_np) - return dev_err_probe(dev, -EINVAL, - "%pOF: missing Type-C connector\n", - route_np); - - mutex_lock(&neo_dcp_typec_fabric_lock); - route->port = neo_dcp_typec_port_get(connector_np); - if (route->port) - list_add_tail(&route->port_link, &route->port->routes); - mutex_unlock(&neo_dcp_typec_fabric_lock); - if (!route->port) - return -ENOMEM; - - desc.fwnode = of_fwnode_handle(route_np); - desc.set = neo_dcp_typec_route_set; - desc.name = mux_name; - desc.drvdata = route; - route->typec_mux = typec_mux_register(dev, &desc); - if (IS_ERR(route->typec_mux)) { - mutex_lock(&neo_dcp_typec_fabric_lock); - list_del(&route->port_link); - if (list_empty(&route->port->routes)) { - list_del(&route->port->link); - of_node_put(route->port->connector_np); - kfree(route->port); - } - mutex_unlock(&neo_dcp_typec_fabric_lock); - return dev_err_probe(dev, PTR_ERR(route->typec_mux), - "%pOF: failed to register Type-C route\n", route_np); - } - - ret = devm_add_action_or_reset(dev, neo_dcp_typec_route_unregister, route); - if (ret) - return ret; - - if (!neo_dcp->phy) - neo_dcp->phy = route->phy; - neo_dcp->nr_typec_routes++; - } - - if (!neo_dcp->nr_typec_routes) - return dev_err_probe(dev, -EINVAL, "Type-C route container is empty\n"); - - neo_dcp->phy_managed_by_typec = true; - return 0; -} - - -/* copied and simplified from drm_vblank.c */ -static void send_vblank_event(struct drm_device *dev, - struct drm_pending_vblank_event *e, - u64 seq, ktime_t now) -{ - struct timespec64 tv; - - if (e->event.base.type != DRM_EVENT_FLIP_COMPLETE) - return; - - tv = ktime_to_timespec64(now); - e->event.vbl.sequence = seq; - /* - * e->event is a user space structure, with hardcoded unsigned - * 32-bit seconds/microseconds. This is safe as we always use - * monotonic timestamps since linux-4.15 - */ - e->event.vbl.tv_sec = tv.tv_sec; - e->event.vbl.tv_usec = tv.tv_nsec / 1000; - - /* - * Use the same timestamp for any associated fence signal to avoid - * mismatch in timestamps for vsync & fence events triggered by the - * same HW event. Frameworks like SurfaceFlinger in Android expects the - * retire-fence timestamp to match exactly with HW vsync as it uses it - * for its software vsync modeling. - */ - drm_send_event_timestamp_locked(dev, &e->base, now); -} - -/** - * dcp_crtc_send_page_flip_event - helper to send vblank event after pageflip - * - * Compensate for unknown slack between page flip and arrival of the - * swap_complete callback. Minimal observed duration on DCP with HDMI output - * was around 2.3 ms. If the fb swap was submitted closer to the expected - * swap_complete it gets a penalty of one frame duration. This is on the border - * of unreasonable considering that Apple advertises support for 240 Hz (frame - * duration of 4.167 ms). - * It is unreasonable considering kwin's kms commit scheduling. Kwin commits - * 1.5 ms + the mode's vblank time before the expected next page flip - * completion. This results in presenting at half the display's rate for HDMI - * outputs. - * This might be a difference between dcp and dcpext. - */ -static void neo_dcp_crtc_send_page_flip_event(struct neo_apple_crtc *crtc, - struct drm_pending_vblank_event *e, - ktime_t now, ktime_t start) -{ - struct drm_device *dev = crtc->base.dev; - u64 seq; - unsigned int pipe = drm_crtc_index(&crtc->base); - ktime_t flip; - - seq = 0; - if (start != KTIME_MIN) { - s64 delta = ktime_us_delta(now, start); - if (delta <= 500) - flip = now; - else if (delta >= 2500) - flip = ktime_sub_us(now, 1000); - else - flip = ktime_sub_us(now, (delta - 500) / 2); - } else { - flip = now; - } - e->pipe = pipe; - send_vblank_event(dev, e, seq, flip); -} - -/* HACK: moved here to avoid circular dependency between apple_drv and dcp */ -void neo_dcp_drm_crtc_vblank(struct neo_apple_crtc *crtc) -{ - unsigned long flags; - - spin_lock_irqsave(&crtc->base.dev->event_lock, flags); - if (crtc->event) { - drm_crtc_send_vblank_event(&crtc->base, crtc->event); - crtc->event = NULL; - } - spin_unlock_irqrestore(&crtc->base.dev->event_lock, flags); -} - -void neo_dcp_drm_crtc_page_flip(struct neo_apple_dcp *neo_dcp, ktime_t now) -{ - unsigned long flags; - struct neo_apple_crtc *crtc = neo_dcp->crtc; - - spin_lock_irqsave(&crtc->base.dev->event_lock, flags); - if (crtc->event) { - if (crtc->event->event.base.type == DRM_EVENT_FLIP_COMPLETE) - neo_dcp_crtc_send_page_flip_event(crtc, crtc->event, now, neo_dcp->swap_start); - else - drm_crtc_send_vblank_event(&crtc->base, crtc->event); - crtc->event = NULL; - neo_dcp->swap_start = KTIME_MIN; - } - spin_unlock_irqrestore(&crtc->base.dev->event_lock, flags); -} - -void neo_dcp_set_dimensions(struct neo_apple_dcp *neo_dcp) -{ - int i; - int width_mm = neo_dcp->width_mm; - int height_mm = neo_dcp->height_mm; - - if (width_mm == 0 || height_mm == 0) { - width_mm = neo_dcp->panel.width_mm; - height_mm = neo_dcp->panel.height_mm; - } - - /* Set the connector info */ - if (neo_dcp->connector) { - struct drm_connector *connector = &neo_dcp->connector->base; - - mutex_lock(&connector->dev->mode_config.mutex); - connector->display_info.width_mm = width_mm; - connector->display_info.height_mm = height_mm; - mutex_unlock(&connector->dev->mode_config.mutex); - } - - /* - * Fix up any probed modes. Modes are created when parsing - * TimingElements, dimensions are calculated when parsing - * DisplayAttributes, and TimingElements may be sent first - */ - for (i = 0; i < neo_dcp->nr_modes; ++i) { - neo_dcp->modes[i].mode.width_mm = width_mm; - neo_dcp->modes[i].mode.height_mm = height_mm; - } -} - -bool neo_dcp_has_panel(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->panel.width_mm > 0; -} - -int neo_dcp_set_crc(struct drm_crtc *crtc, bool enabled) -{ - struct neo_apple_crtc *ac = to_apple_crtc(crtc); - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(ac->neo_dcp); - - WRITE_ONCE(neo_dcp->crc_enabled, enabled); - - return 0; -} - -/* - * Helper to send a DRM vblank event. We do not know how call swap_submit_dcp - * without surfaces. To avoid timeouts in drm_atomic_helper_wait_for_vblanks - * send a vblank event via a workqueue. - */ -static void neo_dcp_delayed_vblank(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp; - - struct neo_apple_crtc *crtc; - - neo_dcp = container_of(work, struct neo_apple_dcp, vblank_wq); - /* RTKit can report a crash before the pipeline has a CRTC. */ - crtc = READ_ONCE(neo_dcp->crtc); - if (!crtc) - return; - mdelay(5); - neo_dcp_drm_crtc_vblank(crtc); -} - -static struct neo_apple_dcp_afkep *neo_dcp_afkep(struct neo_apple_dcp *neo_dcp, u8 endpoint) -{ - switch (endpoint) { - case AV_ENDPOINT: - return neo_dcp->neo_avep; - case SYSTEM_ENDPOINT: - return neo_dcp->neo_systemep; - case DISP0_ENDPOINT: - return neo_dcp->neo_ibootep; - case DPAVSERV_ENDPOINT: - return neo_dcp->dcpavservep; - case DPTX_ENDPOINT: - return neo_dcp->neo_dptxep; - default: - return NULL; - } -} - -#define DCP_SWAP_WATCHDOG_MS 1000 -#define DCP_SWAP_WATCHDOG_RETRAINS 5 - -/* - * DCP drops swaps without completing them while an external pipe is not - * enabled, for instance after a modeset that raced a Type-C sink which had - * not asserted HPD yet. Userspace would then wait for the flip until the - * commit times out, stalling every output it drives. Complete the flip, - * mark the mode invalid so later commits do not wait for DCP, and let the - * hotplug worker re-apply the active CRTC. - */ -static void neo_dcp_swap_watchdog(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = - container_of(to_delayed_work(work), struct neo_apple_dcp, - swap_watchdog_wq); - - dev_warn(neo_dcp->dev, "swap not completed, retraining the display\n"); - neo_dcp_mode_invalidate(&neo_dcp->mode_state); - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); - if (neo_dcp->connector && - neo_dcp->swap_watchdog_retrains++ < DCP_SWAP_WATCHDOG_RETRAINS) - schedule_work(&neo_dcp->connector->hotplug_wq); -} - -void neo_dcp_swap_watchdog_arm(struct neo_apple_dcp *neo_dcp) -{ - if (neo_dcp_is_typec_output(neo_dcp)) - mod_delayed_work(system_wq, &neo_dcp->swap_watchdog_wq, - msecs_to_jiffies(DCP_SWAP_WATCHDOG_MS)); -} - -void neo_dcp_swap_watchdog_complete(struct neo_apple_dcp *neo_dcp) -{ - cancel_delayed_work(&neo_dcp->swap_watchdog_wq); - neo_dcp->swap_watchdog_retrains = 0; -} - -static void neo_dcp_recv_msg(void *cookie, u8 endpoint, u64 message) -{ - struct neo_apple_dcp *neo_dcp = cookie; - struct neo_apple_dcp_afkep *ep; - - if (READ_ONCE(neo_dcp->quiescing)) - return; - - trace_neo_dcp_recv_msg(neo_dcp, endpoint, message); - - /* - * H17P starts endpoints that have no AFK instance behind them (the - * remote allocator endpoint, for one). Log their messages rather than - * dereferencing a NULL afkep or warning about an unknown endpoint. - */ - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && endpoint != IOMFB_ENDPOINT && - !neo_dcp_afkep(neo_dcp, endpoint)) { - dev_dbg_ratelimited(neo_dcp->dev, "ep %#04x: %#llx\n", endpoint, - message); - return; - } - - switch (endpoint) { - case IOMFB_ENDPOINT: - return neo_iomfb_recv_msg(neo_dcp, message); - case AV_ENDPOINT: - ep = neo_dcp->neo_avep; - break; - case SYSTEM_ENDPOINT: - ep = neo_dcp->neo_systemep; - break; - case DISP0_ENDPOINT: - ep = neo_dcp->neo_ibootep; - break; - case DPAVSERV_ENDPOINT: - ep = neo_dcp->dcpavservep; - break; - case DPTX_ENDPOINT: - ep = neo_dcp->neo_dptxep; - break; - default: - ep = NULL; - break; - } - - if (!ep) { - dev_warn_ratelimited(neo_dcp->dev, - "dropping message for unhandled endpoint %#x\n", - endpoint); - return; - } - - neo_afk_receive_message(ep, message); -} - -static void neo_dcp_rtk_crashed(void *cookie, const void *crashlog, size_t crashlog_size) -{ - struct neo_apple_dcp *neo_dcp = cookie; - - if (READ_ONCE(neo_dcp->quiescing)) - return; - - neo_dcp->crashed = true; - dev_err(neo_dcp->dev, "DCP has crashed\n"); - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - /* No in-place restart exists; the last scanout stays retained. */ - dev_err(neo_dcp->dev, "a reboot is required to restore the display\n"); - /* - * Nothing will complete a present that was in flight. Before - * the pipeline is linked there is no event to complete. - */ - if (READ_ONCE(neo_dcp->crtc)) - schedule_work(&neo_dcp->vblank_wq); - } - if (neo_dcp->connector) { - neo_dcp->connector->connected = 0; - drm_edid_free(neo_dcp->connector->drm_edid); - neo_dcp->connector->drm_edid = NULL; - schedule_work(&neo_dcp->connector->hotplug_wq); - } - complete(&neo_dcp->start_done); -} - -/* - * Check that every IOMMU page of [iova, iova + size) translates to the same - * offset from @phys as it has from @iova. - */ -static int neo_dcp_check_iova_contiguous(struct iommu_domain *domain, - dma_addr_t iova, size_t size, - phys_addr_t phys) -{ - unsigned long pgsize = domain->pgsize_bitmap ? - 1UL << __ffs(domain->pgsize_bitmap) : PAGE_SIZE; - dma_addr_t end = iova + size; - dma_addr_t addr = iova; - - while (addr < end) { - if (iommu_iova_to_phys(domain, addr) != phys + (addr - iova)) - return -EINVAL; - addr = ALIGN_DOWN(addr, pgsize) + pgsize; - } - - return 0; -} - -static int neo_dcp_rtk_shmem_setup(void *cookie, struct apple_rtkit_shmem *bfr) -{ - struct neo_apple_dcp *neo_dcp = cookie; - int ret; - - if (bfr->iova) { - struct iommu_domain *domain = - iommu_get_domain_for_dev(neo_dcp->dev); - phys_addr_t phy_addr; - - if (!domain) - return -ENOMEM; - - // TODO: get map from device-tree - phy_addr = iommu_iova_to_phys(domain, bfr->iova); - if (!phy_addr) - return -ENOMEM; - - /* memremap() below needs the whole buffer physically contiguous */ - ret = neo_dcp_check_iova_contiguous(domain, bfr->iova, bfr->size, - phy_addr); - if (ret) { - dev_err(neo_dcp->dev, - "shmem_setup: iova %pad (%#zx bytes) is not physically contiguous\n", - &bfr->iova, bfr->size); - return ret; - } - - /* Firmware writes without CPU cache maintenance. */ - bfr->buffer = memremap(phy_addr, bfr->size, MEMREMAP_WC); - if (!bfr->buffer) - return -ENOMEM; - - bfr->is_mapped = true; - dev_info(neo_dcp->dev, - "shmem_setup: iova: %lx -> pa: %lx -> iomem: %lx\n", - (uintptr_t)bfr->iova, (uintptr_t)phy_addr, - (uintptr_t)bfr->buffer); - } else { - bfr->buffer = dma_alloc_coherent(neo_dcp->dev, bfr->size, - &bfr->iova, GFP_KERNEL); - if (!bfr->buffer) - return -ENOMEM; - - dev_info(neo_dcp->dev, "shmem_setup: iova: %lx, buffer: %lx\n", - (uintptr_t)bfr->iova, (uintptr_t)bfr->buffer); - } - - return 0; -} - -static void neo_dcp_rtk_shmem_destroy(void *cookie, struct apple_rtkit_shmem *bfr) -{ - struct neo_apple_dcp *neo_dcp = cookie; - - if (bfr->is_mapped) - memunmap(bfr->buffer); - else - dma_free_coherent(neo_dcp->dev, bfr->size, bfr->buffer, bfr->iova); -} - -static struct apple_rtkit_ops rtkit_ops = { - .crashed = neo_dcp_rtk_crashed, - .recv_message = neo_dcp_recv_msg, - .shmem_setup = neo_dcp_rtk_shmem_setup, - .shmem_destroy = neo_dcp_rtk_shmem_destroy, -}; - -void neo_dcp_send_message(struct neo_apple_dcp *neo_dcp, u8 endpoint, u64 message) -{ - if (READ_ONCE(neo_dcp->quiescing)) - return; - trace_neo_dcp_send_msg(neo_dcp, endpoint, message); - apple_rtkit_send_message(neo_dcp->rtk, endpoint, message, NULL, - true); -} - -int neo_dcp_crtc_atomic_check(struct drm_crtc *crtc, struct drm_atomic_state *state) -{ - struct platform_device *pdev = to_apple_crtc(crtc)->neo_dcp; - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - struct drm_crtc_state *crtc_state; - bool needs_modeset; - - if (neo_dcp->crashed) - return -EINVAL; - - crtc_state = drm_atomic_get_new_crtc_state(state, crtc); - - needs_modeset = drm_atomic_crtc_needs_modeset(crtc_state) || - !READ_ONCE(neo_dcp->mode_state.valid); - if (!needs_modeset && (!neo_dcp->connector || !neo_dcp->connector->connected)) { - /* - * Resume restores the mode before the firmware reports the - * display back, so a plane-only commit lands here while the - * connector is still marked disconnected. Rejecting it makes - * the compositor fail every flip and give up on the output; - * dcp_flush() defers the commit until the link returns. - */ - dev_dbg(neo_dcp->dev, - "crtc_atomic_check: deferring commit, link still down\n"); - } - - return 0; -} - -int neo_dcp_get_connector_type(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp->fixed_connector_type; -} - -bool neo_dcp_has_typec_routes(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp->nr_typec_routes; -} - -#define DPTX_CONNECT_TIMEOUT msecs_to_jiffies(2000) -#define DPTX_TUNNEL_CONNECT_TIMEOUT msecs_to_jiffies(8000) -#define DPTX_RECONNECT_DELAY msecs_to_jiffies(1000) -#define DPTX_RECONNECT_RETRIES 5 - -static int neo_dcp_dptx_connect(struct neo_apple_dcp *neo_dcp, u32 port) -{ - unsigned long timeout; - int ret = 0; - - if (!neo_dcp->phy) { - dev_warn(neo_dcp->dev, "dcp_dptx_connect: missing phy\n"); - return -ENODEV; - } - dev_info(neo_dcp->dev, - "%s(port=%d) target=%u:%u typec=%d route=%s conn_type=%d connected=%d\n", - __func__, port, neo_dcp->neo_dptx_die, neo_dcp->neo_dptx_phy, - neo_dcp_is_typec_output(neo_dcp), - neo_dcp->active_typec_route ? "borrowed" : "fixed", - neo_dcp->connector_type, neo_dcp->neo_dptxport[port].connected); - - mutex_lock(&neo_dcp->hpd_mutex); - if (!neo_dcp->neo_dptxport[port].enabled) { - dev_warn(neo_dcp->dev, "dcp_dptx_connect: dptx service for port %d not enabled\n", port); - ret = -ENODEV; - goto out_unlock; - } - - if (neo_dcp->neo_dptxport[port].connected) - goto out_unlock; - - reinit_completion(&neo_dcp->neo_dptxport[port].linkcfg_completion); - neo_dcp->neo_dptxport[port].atcphy = neo_dcp->phy; - ret = neo_dptxport_validate_connection(neo_dcp->neo_dptxport[port].service, - neo_dcp->neo_dptx_dfp_port, - neo_dcp->neo_dptx_phy, neo_dcp->neo_dptx_die); - if (ret) { - dev_err(neo_dcp->dev, - "dcp_dptx_connect: failed to validate DPTX target %u:%u: %d\n", - neo_dcp->neo_dptx_die, neo_dcp->neo_dptx_phy, ret); - goto out_unlock; - } - - ret = neo_dptxport_connect(neo_dcp->neo_dptxport[port].service, - neo_dcp->neo_dptx_dfp_port, - neo_dcp->neo_dptx_phy, neo_dcp->neo_dptx_die, - neo_dcp_is_typec_output(neo_dcp)); - if (ret) { - dev_err(neo_dcp->dev, - "dcp_dptx_connect: failed to connect DPTX target %u:%u: %d\n", - neo_dcp->neo_dptx_die, neo_dcp->neo_dptx_phy, ret); - goto out_unlock; - } - - ret = neo_dptxport_request_display(neo_dcp->neo_dptxport[port].service); - if (ret) { - dev_err(neo_dcp->dev, - "dcp_dptx_connect: failed to request display: %d\n", - ret); - goto out_release; - } - neo_dcp->neo_dptxport[port].connected = true; - if (neo_dcp_is_typec_output(neo_dcp)) { - if (neo_dcp_is_usb4_output(neo_dcp) && false) - ret = neo_dptxport_set_hpd_timeout(neo_dcp->neo_dptxport[port].service, - true, 8000); - else - ret = neo_dptxport_set_hpd(neo_dcp->neo_dptxport[port].service, true); - if (ret) { - dev_err(neo_dcp->dev, - "dcp_dptx_connect: failed to assert Type-C HPD: %d\n", - ret); - neo_dcp->neo_dptxport[port].connected = false; - goto out_release; - } - } - - mutex_unlock(&neo_dcp->hpd_mutex); - timeout = neo_dcp_is_usb4_output(neo_dcp) && false ? - DPTX_TUNNEL_CONNECT_TIMEOUT : DPTX_CONNECT_TIMEOUT; - ret = wait_for_completion_timeout(&neo_dcp->neo_dptxport[port].linkcfg_completion, - timeout); - if (!ret) { - dev_err(neo_dcp->dev, - "dcp_dptx_connect: timed out waiting for port %u link configuration\n", - port); - ret = -ETIMEDOUT; - goto out_disconnect; - } - - dev_dbg(neo_dcp->dev, "dcp_dptx_connect: waited %d ms for link\n", - jiffies_to_msecs(timeout - ret)); - - usleep_range(5, 10); - - if (neo_dcp->connector_type == DRM_MODE_CONNECTOR_DisplayPort) - neo_dptxport_set_hpd(neo_dcp->neo_dptxport[port].service, true); - - if (neo_dcp->neo_avep) - neo_av_service_connect(neo_dcp); - - return 0; - -out_disconnect: - mutex_lock(&neo_dcp->hpd_mutex); - neo_dcp->neo_dptxport[port].connected = false; -out_release: - neo_dptxport_release_display(neo_dcp->neo_dptxport[port].service); - -out_unlock: - mutex_unlock(&neo_dcp->hpd_mutex); - return ret; -} - -static bool neo_dcp_edid_is_placeholder(const struct drm_edid *drm_edid) -{ - const u8 *raw = (const u8 *)drm_edid_raw(drm_edid); - static const u8 name[] = "Non-PnP"; - unsigned int i; - - if (!raw) - return false; - - /* EDID contains interior NUL bytes, so this cannot use strnstr(). */ - for (i = 0; i + sizeof(name) - 1 <= sizeof(struct edid); i++) { - if (!memcmp(raw + i, name, sizeof(name) - 1)) - return true; - } - - return false; -} - -void neo_dcp_retry_placeholder_edid(struct neo_apple_dcp *neo_dcp, - const struct drm_edid *drm_edid) -{ - guard(mutex)(&neo_dcp->hpd_mutex); - if (!neo_dcp_is_typec_output(neo_dcp) || !neo_dcp->typec_cable_connected || - neo_dcp->placeholder_retried) - return; - if (!neo_dcp_edid_is_placeholder(drm_edid)) - return; - - neo_dcp->placeholder_retried = true; - neo_dcp->placeholder_generation = neo_dcp->typec_generation; - schedule_delayed_work(&neo_dcp->placeholder_edid_wq, msecs_to_jiffies(300)); -} - -static void neo_dcp_placeholder_edid_work(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = - container_of(to_delayed_work(work), struct neo_apple_dcp, - placeholder_edid_wq); - struct neo_apple_epic_service *service; - u64 generation; - int ret; - - mutex_lock(&neo_dcp->hpd_mutex); - generation = neo_dcp->placeholder_generation; - if (!neo_dcp->typec_cable_connected || !neo_dcp->neo_dptxport[0].connected || - !neo_dcp->neo_dptxport[0].enabled || generation != neo_dcp->typec_generation) - goto out_unlock; - service = neo_dcp->neo_dptxport[0].service; - - /* - * Some adapters answer the first connection with a 1024x768 - * placeholder and publish the panel EDID only after HPD drops - * and returns. One pulse; a second placeholder is left alone. - */ - ret = neo_dptxport_set_hpd(service, false); - if (ret) { - dev_info(neo_dcp->dev, "placeholder EDID: HPD drop failed: %d\n", - ret); - goto out_unlock; - } - mutex_unlock(&neo_dcp->hpd_mutex); - - msleep(1000); - - mutex_lock(&neo_dcp->hpd_mutex); - if (!neo_dcp->typec_cable_connected || !neo_dcp->neo_dptxport[0].connected || - generation != neo_dcp->typec_generation) - goto out_unlock; - - ret = neo_dptxport_set_hpd(service, true); - if (ret) - dev_info(neo_dcp->dev, "placeholder EDID: HPD assert failed: %d\n", - ret); -out_unlock: - mutex_unlock(&neo_dcp->hpd_mutex); -} - -static void neo_dcp_typec_reconnect_work(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = - container_of(to_delayed_work(work), struct neo_apple_dcp, - typec_reconnect_wq); - int ret; - - if (!READ_ONCE(neo_dcp->typec_cable_connected)) - return; - - ret = neo_dcp_dptx_connect(neo_dcp, 0); - if (!ret) { - neo_dcp->typec_reconnect_tries = 0; - return; - } - - if (++neo_dcp->typec_reconnect_tries < - (neo_dcp_is_usb4_output(neo_dcp) && false ? - 1 : DPTX_RECONNECT_RETRIES)) { - mod_delayed_work(system_freezable_wq, &neo_dcp->typec_reconnect_wq, - DPTX_RECONNECT_DELAY); - return; - } - - dev_err(neo_dcp->dev, "Type-C DPTX reconnect failed after %u retries: %d\n", - neo_dcp->typec_reconnect_tries, ret); -} - -static void disconnected_hpd_event(struct neo_apple_connector *con) -{ - if (con && con->connected) { - con->connected = 0; - drm_edid_free(con->drm_edid); - con->drm_edid = NULL; - drm_kms_helper_connector_hotplug_event(&con->base); - } -} - -static int neo_dcp_dptx_disconnect(struct neo_apple_dcp *neo_dcp, u32 port) -{ - dev_info(neo_dcp->dev, "%s(port=%d)\n", __func__, port); - - mutex_lock(&neo_dcp->hpd_mutex); - if (neo_dcp->neo_dptxport[port].enabled && neo_dcp->neo_dptxport[port].connected) { - neo_dptxport_release_display(neo_dcp->neo_dptxport[port].service); - neo_dcp->neo_dptxport[port].connected = false; - } - mutex_unlock(&neo_dcp->hpd_mutex); - - return 0; -} - -int neo_dcp_dptx_connect_oob(struct platform_device *pdev, u32 port) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - int ret; - - if (neo_dcp_is_typec_output(neo_dcp)) { - cancel_delayed_work_sync(&neo_dcp->placeholder_edid_wq); - guard(mutex)(&neo_dcp->hpd_mutex); - neo_dcp->typec_generation++; - WRITE_ONCE(neo_dcp->typec_cable_connected, true); - neo_dcp->typec_reconnect_tries = 0; - neo_dcp->placeholder_retried = false; - cancel_delayed_work(&neo_dcp->typec_reconnect_wq); - } - - ret = neo_dcp_dptx_connect(neo_dcp, port); - if (ret && neo_dcp_is_typec_output(neo_dcp)) - mod_delayed_work(system_freezable_wq, &neo_dcp->typec_reconnect_wq, - DPTX_RECONNECT_DELAY); - - return ret; -} - -int neo_dcp_dptx_disconnect_oob(struct platform_device *pdev, u32 port) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - if (neo_dcp_is_typec_output(neo_dcp)) { - scoped_guard(mutex, &neo_dcp->hpd_mutex) { - WRITE_ONCE(neo_dcp->typec_cable_connected, false); - neo_dcp->typec_generation++; - } - WRITE_ONCE(neo_dcp->typec_crtc_off, false); - reinit_completion(&neo_dcp->typec_iomfb_hpd_ready); - cancel_delayed_work_sync(&neo_dcp->typec_reconnect_wq); - cancel_delayed_work_sync(&neo_dcp->placeholder_edid_wq); - } - - disconnected_hpd_event(neo_dcp->connector); - - if (neo_dcp->neo_avep) - neo_av_service_disconnect(neo_dcp); - - if (neo_dcp->neo_dptxport[port].enabled) - neo_dptxport_set_hpd(neo_dcp->neo_dptxport[port].service, false); - - return neo_dcp_dptx_disconnect(neo_dcp, port); -} - -static irqreturn_t neo_dcp_dp2hdmi_hpd(int irq, void *data) -{ - struct neo_apple_dcp *neo_dcp = data; - bool connected; - - guard(mutex)(&neo_dcp_typec_fabric_lock); - - if (READ_ONCE(neo_dcp->active_typec_route)) { - /* - * Until a compositor owns the display, a live HDMI output - * takes its pipeline back from a direct DP-alt route: the - * compositor pairs the HDMI connector with it first. - */ - if (neo_dcp_typec_keep_order() && - gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) { - msleep(500); - if (gpiod_get_value_cansleep(neo_dcp->hdmi_hpd)) - neo_dcp_typec_rebalance_locked(NULL, 0); - } - return IRQ_HANDLED; - } - connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - - /* do nothing on disconnect and trust that dcp detects it itself. - * Parallel disconnect HPDs result drm disabling the CRTC even when it - * should not. - * The interrupt should be changed to rising but for now the disconnect - * IRQs might be helpful for debugging. - */ - dev_info(neo_dcp->dev, "DP2HDMI HPD irq, connected:%d\n", connected); - - if (connected) { - msleep(500); - connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - dev_info(neo_dcp->dev, "DP2HDMI HPD irq, 500ms debounce: connected:%d\n", connected); - } - - if (connected) - neo_dcp_dptx_connect(neo_dcp, 0); - - return IRQ_HANDLED; -} - -void neo_dcp_link(struct platform_device *pdev, struct neo_apple_crtc *crtc, - struct neo_apple_connector *connector) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - WRITE_ONCE(neo_dcp->crtc, crtc); - - /* - * Type-C connectors belong to physical ports and are bound by the - * display fabric when a pipeline takes a route, so a pipeline with no - * fixed output simply has no connector until then. - */ - if (!connector) - return; - - neo_dcp->fixed_connector = connector; - if (!neo_dcp->active_typec_route) { - neo_dcp->connector = connector; - neo_dcp->connector_type = neo_dcp->fixed_connector_type; - } -} - - -bool neo_dcp_fw_compat_is_12_x(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp->fw_compat == DCP_FIRMWARE_V_12_3; -} - -unsigned long* neo_dcp_get_iomfb_surfaces(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp->neo_iomfb_surfaces; -} - -int neo_dcp_start(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - int ret; - - if (!neo_dcp->rtk) - return -ENODEV; - - init_completion(&neo_dcp->start_done); - - /* - * The coprocessor start can fail (or be recovered from) without tearing - * the DRM side down, so this can be reached with no RTKit instance at - * all. Starting an endpoint then dereferences NULL inside - * apple_rtkit_start_ep() and takes the machine with it. - */ - if (!neo_dcp->rtk || IS_ERR(neo_dcp->rtk)) { - dev_err(neo_dcp->dev, - "cannot start endpoints: RTKit is not initialised\n"); - return -ENODEV; - } - - /* start RTKit endpoints */ - ret = neo_systemep_init(neo_dcp); - if (ret) - dev_warn(neo_dcp->dev, "Failed to start system endpoint: %d\n", ret); - - if (unstable_edid && !neo_dcp_has_panel(neo_dcp)) { - ret = dpavservep_init(neo_dcp); - if (ret) - dev_warn(neo_dcp->dev, "Failed to start DPAVSERV endpoint: %d", - ret); - } - - if (neo_dcp->phy && neo_dcp->fw_compat >= DCP_FIRMWARE_V_13_5) { - ret = neo_ibootep_init(neo_dcp); - if (ret) - dev_warn(neo_dcp->dev, "Failed to start IBOOT endpoint: %d\n", - ret); - - ret = neo_dptxep_init(neo_dcp); - if (ret) { - dev_warn(neo_dcp->dev, "Failed to start DPTX endpoint: %d\n", - ret); -#ifdef DCP_DPTX_DISCONNECT_ON_INIT - /* - * This disconnect / connect cycle on init is only necessary - * when using dcp0 on j473, j474s and presumedly j475c. - * Since dcp0 is not used at the moment let's avoid this - * since it is possibly the cause for startup issues. - */ - } else if (neo_dcp->neo_dptxport[0].enabled) { - bool connected; - /* force disconnect on start - necessary if the display - * is already up from m1n1 - */ - neo_dptxport_set_hpd(neo_dcp->neo_dptxport[0].service, false); - neo_dptxport_release_display(neo_dcp->neo_dptxport[0].service); - usleep_range(10 * USEC_PER_MSEC, 25 * USEC_PER_MSEC); - - connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - dev_info(neo_dcp->dev, "%s: DP2HDMI HPD connected:%d\n", __func__, connected); - - // necessary on j473/j474 but not on j314c - if (connected) - neo_dcp_dptx_connect(neo_dcp, 0); -#endif - } - } else if (neo_dcp->phy) { - dev_warn(neo_dcp->dev, "OS firmware incompatible with dptxport EP\n"); - } - ret = neo_iomfb_start_rtkit(neo_dcp); - if (ret) - dev_err(neo_dcp->dev, "Failed to start IOMFB endpoint: %d\n", ret); - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) - if (neo_hdmi_audio) { - ret = neo_avep_init(neo_dcp); - if (ret) - dev_warn(neo_dcp->dev, "Failed to start AV endpoint: %d", ret); - ret = 0; - } -#endif - - return ret; -} - -static void _dcp_poweroff(struct neo_apple_dcp *neo_dcp) -{ - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_poweroff_v12_3(neo_dcp); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_poweroff_v13_3(neo_dcp); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_poweroff_h17p(neo_dcp); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } -} - -static int neo_dcp_enable_dp2hdmi_hpd(struct neo_apple_dcp *neo_dcp) -{ - if (neo_dcp_is_typec_output(neo_dcp)) { - if (READ_ONCE(neo_dcp->typec_cable_connected)) - neo_dcp_dptx_connect(neo_dcp, 0); - } else if (neo_dcp->hdmi_hpd) { - /* Check HPD before enabling the edge-triggered IRQ. */ - bool connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - dev_info(neo_dcp->dev, "%s: DP2HDMI HPD connected:%d\n", __func__, connected); - - if (connected) - neo_dcp_dptx_connect(neo_dcp, 0); - else - _dcp_poweroff(neo_dcp); - } - - if (neo_dcp->hdmi_hpd_irq) - enable_irq(neo_dcp->hdmi_hpd_irq); - - return 0; -} - -int neo_dcp_wait_ready(struct platform_device *pdev, u64 timeout) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - int ret; - - if (neo_dcp->crashed) - return -ENODEV; - if (neo_dcp->active) - return neo_dcp_enable_dp2hdmi_hpd(neo_dcp); - if (timeout <= 0) - return -ETIMEDOUT; - - ret = wait_for_completion_timeout(&neo_dcp->start_done, timeout); - if (ret < 0) - return ret; - - if (neo_dcp->crashed) - return -ENODEV; - - if (neo_dcp->active) - neo_dcp_enable_dp2hdmi_hpd(neo_dcp); - - return neo_dcp->active ? 0 : -ETIMEDOUT; -} - -static void __maybe_unused neo_dcp_sleep(struct neo_apple_dcp *neo_dcp) -{ - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_sleep_v12_3(neo_dcp); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_sleep_v13_3(neo_dcp); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_sleep_h17p(neo_dcp); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } -} - -#define DCP_BL_FALLBACK_DELAY msecs_to_jiffies(2000) - -static bool neo_dcp_uses_soft_dpms(struct neo_apple_dcp *neo_dcp) -{ - return (neo_dcp_backlight_active(neo_dcp) && neo_dcp_has_panel(neo_dcp)) || - (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - neo_dcp->connector_type == DRM_MODE_CONNECTOR_eDP); -} - -void neo_dcp_poweron(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - bool wait_for_typec_hpd = false; - unsigned long remaining; - int ret; - - if (neo_dcp_uses_soft_dpms(neo_dcp)) { - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !READ_ONCE(neo_dcp->pipe_enabled_h17p)) { - neo_iomfb_poweron_h17p(neo_dcp); - if (!READ_ONCE(neo_dcp->pipe_enabled_h17p)) { - dev_err(neo_dcp->dev, "initial display pipe enable failed\n"); - WRITE_ONCE(neo_dcp->crashed, true); - return; - } - /* The powerlog takeover report normally precedes this. */ - if (neo_dcp_has_panel(neo_dcp)) - queue_delayed_work(system_freezable_wq, - &neo_dcp->bl_fallback_wq, - DCP_BL_FALLBACK_DELAY); - } - ret = neo_dcp_backlight_dpms(neo_dcp, true); - if (ret) - dev_warn(neo_dcp->dev, "backlight restore unavailable: %d\n", ret); - return; - } - - if (neo_dcp_is_typec_output(neo_dcp)) { - wait_for_typec_hpd = READ_ONCE(neo_dcp->typec_crtc_off) && - READ_ONCE(neo_dcp->typec_cable_connected); - WRITE_ONCE(neo_dcp->typec_crtc_off, false); - - /* - * A Type-C CRTC disable releases its DPTX session. Re-establish it - * synchronously before IOMFB is powered back on. - */ - if (READ_ONCE(neo_dcp->typec_cable_connected)) { - cancel_delayed_work(&neo_dcp->typec_reconnect_wq); - neo_dcp->typec_reconnect_tries = 0; - ret = neo_dcp_dptx_connect(neo_dcp, 0); - if (ret) - mod_delayed_work(system_freezable_wq, - &neo_dcp->typec_reconnect_wq, - DPTX_RECONNECT_DELAY); - else if (wait_for_typec_hpd) { - remaining = wait_for_completion_timeout( - &neo_dcp->typec_iomfb_hpd_ready, - msecs_to_jiffies(3000)); - if (!remaining) - dev_warn(neo_dcp->dev, - "Type-C IOMFB hotplug not ready on wake\n"); - } - } - } else if (neo_dcp->hdmi_hpd) { - bool connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - dev_info(neo_dcp->dev, "%s: DP2HDMI HPD connected:%d\n", __func__, connected); - - if (connected) - neo_dcp_dptx_connect(neo_dcp, 0); - } - - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_poweron_v12_3(neo_dcp); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_poweron_v13_3(neo_dcp); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_poweron_h17p(neo_dcp); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } - if (neo_dcp->neo_avep) - neo_av_service_connect(neo_dcp); -} - -void neo_dcp_poweroff(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - int ret; - - cancel_delayed_work(&neo_dcp->swap_watchdog_wq); - - /* Internal H17P DPMS is a brightness present, never a pipe stop. */ - if (neo_dcp_uses_soft_dpms(neo_dcp)) { - ret = neo_dcp_backlight_dpms(neo_dcp, false); - if (ret) - dev_warn(neo_dcp->dev, "backlight blank unavailable: %d\n", ret); - return; - } - - if (neo_dcp->neo_avep) - neo_av_service_disconnect(neo_dcp); - - /* - * Powering a Type-C CRTC off drops DCP's synthetic HPD, and the firmware - * reports that as an unplug. The display is still attached: keep the - * connector connected (see dcpep_cb_hotplug()) and let dcp_poweron() - * re-establish the DPTX session. Recreating it here instead makes the - * display vanish and come back, and compositors light a returning - * display, so DPMS off never sticks. Cable removal is reported through - * the Type-C mux. - */ - if (neo_dcp_is_typec_output(neo_dcp)) { - reinit_completion(&neo_dcp->typec_iomfb_hpd_ready); - if (READ_ONCE(neo_dcp->typec_cable_connected)) - WRITE_ONCE(neo_dcp->typec_crtc_off, true); - /* dcp_poweron() reconnects the link on DPMS wake. */ - cancel_delayed_work(&neo_dcp->typec_reconnect_wq); - } - - _dcp_poweroff(neo_dcp); - - if (neo_dcp_is_typec_output(neo_dcp)) { - /* DCP owns a synthetic HPD for Type-C. Release it with the CRTC. */ - if (neo_dcp->neo_dptxport[0].enabled && neo_dcp->neo_dptxport[0].connected) { - ret = neo_dptxport_set_hpd(neo_dcp->neo_dptxport[0].service, false); - if (ret) - dev_warn(neo_dcp->dev, - "failed to deassert Type-C DPTX HPD: %d\n", ret); - neo_dcp_dptx_disconnect(neo_dcp, 0); - } - } else if (neo_dcp->hdmi_hpd) { - bool connected = gpiod_get_value_cansleep(neo_dcp->hdmi_hpd); - if (!connected) { - disconnected_hpd_event(neo_dcp->connector); - neo_dcp_dptx_disconnect(neo_dcp, 0); - } - } -} - -static void neo_dcp_work_register_backlight(struct work_struct *work) -{ - int ret; - struct neo_apple_dcp *neo_dcp; - - neo_dcp = container_of(work, struct neo_apple_dcp, bl_register_wq); - - mutex_lock(&neo_dcp->bl_register_mutex); - if (neo_dcp->brightness.bl_dev) - goto out_unlock; - - /* try to register backlight device, */ - ret = neo_dcp_backlight_register(neo_dcp); - if (ret == -ENODATA) - goto out_unlock; - if (ret) { - dev_err(neo_dcp->dev, "Unable to register backlight device: %d\n", ret); - /* - * The H17P policy encodes the panel ceiling into every present - * and keeps working without a class device; only older - * firmware stops sending brightness here. - */ - if (!neo_dcp_backlight_active(neo_dcp)) - neo_dcp->brightness.maximum = 0; - } - -out_unlock: - mutex_unlock(&neo_dcp->bl_register_mutex); -} - -static void neo_dcp_work_update_backlight(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp; - - neo_dcp = container_of(work, struct neo_apple_dcp, bl_update_wq); - - neo_dcp_backlight_update(neo_dcp); -} - -/* - * H17P never publishes the panel level, and the loader's level arrives only - * as an optional powerlog report. Without one, register the backlight at - * the middle of the panel range. That value is only reported: no level is - * presented until userspace writes one, so the panel keeps the loader's - * level. - */ -static void neo_dcp_work_backlight_fallback(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = container_of(to_delayed_work(work), - struct neo_apple_dcp, bl_fallback_wq); - u32 nits = neo_dcp->brightness.maximum / 2; - int ret; - - if (READ_ONCE(neo_dcp->quiescing) || READ_ONCE(neo_dcp->crashed) || - neo_dcp_backlight_active(neo_dcp)) - return; - - ret = neo_iomfb_configure_backlight_h17p(neo_dcp, neo_dcp->brightness.maximum, - false, 0, true, nits); - if (!ret) - dev_info(neo_dcp->dev, - "no loader brightness reported; backlight starts at %u nits without a panel change\n", - nits); - else if (ret != -EBUSY) - dev_warn(neo_dcp->dev, "backlight registration failed: %d\n", ret); -} - -static void neo_dcp_release_piodma_iommu_dev(struct neo_apple_dcp *neo_dcp) -{ - if (neo_dcp->retain_dma) { - WRITE_ONCE(neo_dcp->quiescing, true); - return; - } - if (neo_dcp->piodma) { - if (neo_dcp->piodma_created) - of_platform_device_destroy(&neo_dcp->piodma->dev, NULL); - else - put_device(&neo_dcp->piodma->dev); - } - neo_dcp->piodma = NULL; - neo_dcp->iommu_dom = NULL; - neo_dcp->piodma_created = false; -} - -void neo_dcp_retain_framebuffer(struct platform_device *pdev, - struct neo_dcp_fb_reference *entry) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - list_add_tail(&entry->head, &neo_dcp->swapped_out_fbs); - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); -} - -void neo_dcp_arm_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id) -{ - struct neo_dcp_fb_reference *entry; - - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - list_for_each_entry(entry, &neo_dcp->swapped_out_fbs, head) { - if (entry->armed) - continue; - entry->swap_id = swap_id; - entry->armed = true; - } - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); -} - -void neo_dcp_unarm_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id) -{ - struct neo_dcp_fb_reference *entry; - - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - list_for_each_entry(entry, &neo_dcp->swapped_out_fbs, head) - if (entry->armed && entry->swap_id == swap_id) - entry->armed = false; - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); -} - -void neo_dcp_release_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id) -{ - struct neo_dcp_fb_reference *entry, *tmp; - LIST_HEAD(completed); - - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - list_for_each_entry_safe(entry, tmp, &neo_dcp->swapped_out_fbs, head) { - if (!entry->armed || entry->swap_id != swap_id) - continue; - list_move_tail(&entry->head, &completed); - } - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); - - list_for_each_entry_safe(entry, tmp, &completed, head) { - list_del(&entry->head); - drm_framebuffer_put(entry->fb); - kfree(entry); - } -} - -void neo_dcp_release_all_retained_framebuffers(struct neo_apple_dcp *neo_dcp) -{ - struct neo_dcp_fb_reference *entry, *tmp; - LIST_HEAD(completed); - - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - list_splice_init(&neo_dcp->swapped_out_fbs, &completed); - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); - - list_for_each_entry_safe(entry, tmp, &completed, head) { - list_del(&entry->head); - drm_framebuffer_put(entry->fb); - kfree(entry); - } -} - -static int neo_dcp_create_piodma_iommu_dev(struct neo_apple_dcp *neo_dcp) -{ - int ret; - struct device_node *node __free(device_node) = of_get_child_by_name(neo_dcp->dev->of_node, "piodma"); - - if (!node) - return dev_err_probe(neo_dcp->dev, -ENODEV, - "Failed to get piodma child DT node\n"); - - neo_dcp->piodma = of_find_device_by_node(node); - neo_dcp->piodma_created = !neo_dcp->piodma; - if (neo_dcp->piodma_created) - neo_dcp->piodma = of_platform_device_create(node, NULL, neo_dcp->dev); - if (!neo_dcp->piodma) - return dev_err_probe(neo_dcp->dev, -ENODEV, - "Failed to create piodma pdev for %pOF\n", node); - - ret = dma_set_mask_and_coherent(&neo_dcp->piodma->dev, DMA_BIT_MASK(42)); - if (ret) - goto err_destroy_pdev; - - ret = of_dma_configure(&neo_dcp->piodma->dev, node, true); - if (ret) { - ret = dev_err_probe(neo_dcp->dev, ret, - "Failed to configure IOMMU child DMA\n"); - goto err_destroy_pdev; - } - - neo_dcp->iommu_dom = iommu_get_domain_for_dev(&neo_dcp->piodma->dev); - if (IS_ERR(neo_dcp->iommu_dom)) { - ret = dev_err_probe(neo_dcp->dev, PTR_ERR(neo_dcp->iommu_dom), - "Failed to get default iommu domain for " - "piodma device\n"); - neo_dcp->iommu_dom = NULL; - goto err_destroy_pdev; - } - - return 0; -err_destroy_pdev: - neo_dcp_release_piodma_iommu_dev(neo_dcp); - return ret; -} - -static int neo_dcp_get_bw_scratch_reg(struct neo_apple_dcp *neo_dcp, u32 expected) -{ - struct of_phandle_args ph_args; - u32 addr_idx, disp_idx, offset; - int ret; - - ret = of_parse_phandle_with_args(neo_dcp->dev->of_node, "apple,bw-scratch", - "#apple,bw-scratch-cells", 0, &ph_args); - if (ret < 0) { - dev_err(neo_dcp->dev, "Failed to read 'apple,bw-scratch': %d\n", ret); - return ret; - } - - if (ph_args.args_count != 3) { - dev_err(neo_dcp->dev, "Unexpected 'apple,bw-scratch' arg count %d\n", - ph_args.args_count); - ret = -EINVAL; - goto err_of_node_put; - } - - addr_idx = ph_args.args[0]; - disp_idx = ph_args.args[1]; - offset = ph_args.args[2]; - - if (disp_idx != expected || disp_idx >= MAX_DISP_REGISTERS) { - dev_err(neo_dcp->dev, "Unexpected disp_reg value in 'apple,bw-scratch': %d\n", - disp_idx); - ret = -EINVAL; - goto err_of_node_put; - } - - ret = of_address_to_resource(ph_args.np, addr_idx, &neo_dcp->disp_bw_scratch_res); - if (ret < 0) { - dev_err(neo_dcp->dev, "Failed to get 'apple,bw-scratch' resource %d from %pOF\n", - addr_idx, ph_args.np); - goto err_of_node_put; - } - if (offset > resource_size(&neo_dcp->disp_bw_scratch_res) - 4) { - ret = -EINVAL; - goto err_of_node_put; - } - - neo_dcp->disp_registers[disp_idx] = &neo_dcp->disp_bw_scratch_res; - neo_dcp->disp_bw_scratch_index = disp_idx; - neo_dcp->disp_bw_scratch_offset = offset; - ret = 0; - -err_of_node_put: - of_node_put(ph_args.np); - return ret; -} - -static int neo_dcp_get_bw_doorbell_reg(struct neo_apple_dcp *neo_dcp, u32 expected) -{ - struct of_phandle_args ph_args; - u32 addr_idx, disp_idx; - int ret; - - ret = of_parse_phandle_with_args(neo_dcp->dev->of_node, "apple,bw-doorbell", - "#apple,bw-doorbell-cells", 0, &ph_args); - if (ret < 0) { - dev_err(neo_dcp->dev, "Failed to read 'apple,bw-doorbell': %d\n", ret); - return ret; - } - - if (ph_args.args_count != 2) { - dev_err(neo_dcp->dev, "Unexpected 'apple,bw-doorbell' arg count %d\n", - ph_args.args_count); - ret = -EINVAL; - goto err_of_node_put; - } - - addr_idx = ph_args.args[0]; - disp_idx = ph_args.args[1]; - - if (disp_idx != expected || disp_idx >= MAX_DISP_REGISTERS) { - dev_err(neo_dcp->dev, "Unexpected disp_reg value in 'apple,bw-doorbell': %d\n", - disp_idx); - ret = -EINVAL; - goto err_of_node_put; - } - - ret = of_address_to_resource(ph_args.np, addr_idx, &neo_dcp->disp_bw_doorbell_res); - if (ret < 0) { - dev_err(neo_dcp->dev, "Failed to get 'apple,bw-doorbell' resource %d from %pOF\n", - addr_idx, ph_args.np); - goto err_of_node_put; - } - neo_dcp->disp_bw_doorbell_index = disp_idx; - neo_dcp->disp_registers[disp_idx] = &neo_dcp->disp_bw_doorbell_res; - ret = 0; - -err_of_node_put: - of_node_put(ph_args.np); - return ret; -} - -static int neo_dcp_get_disp_regs(struct neo_apple_dcp *neo_dcp) -{ - struct platform_device *pdev = to_platform_device(neo_dcp->dev); - int count = 0; - int i, ret; - char name[16]; - const char *reg_name; - struct resource *res; - - if (of_property_present(neo_dcp->dev->of_node, "reg-names")) { - ret = of_property_count_strings(neo_dcp->dev->of_node, "reg-names"); - if (ret < 0) - return ret; - - for (i = 0; i < ret; i++) { - if (of_property_read_string_index(neo_dcp->dev->of_node, - "reg-names", i, ®_name)) - return -EINVAL; - if (!strncmp(reg_name, "disp-", 5)) - count++; - } - } else { - count = pdev->num_resources - 1; - } - - if (count <= 0 || count > MAX_DISP_REGISTERS) - return -EINVAL; - - for (i = 0; i < count; ++i) { - if (of_property_present(neo_dcp->dev->of_node, "reg-names")) { - snprintf(name, sizeof(name), "disp-%d", i); - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, - name); - } else { - res = platform_get_resource(pdev, IORESOURCE_MEM, 1 + i); - } - if (!res) - return -EINVAL; - neo_dcp->disp_registers[i] = res; - } - - if (neo_dcp->hw.firmware_clock) { - /* The internal firmware's last aperture follows five display windows. */ - if (count != 5 || resource_size(neo_dcp->hw.firmware_clock) < 4 || - neo_dcp->hw.firmware_scratch > resource_size(neo_dcp->hw.firmware_clock) - 4 || - neo_dcp->hw.firmware_request > resource_size(neo_dcp->hw.firmware_clock) - 4) - return -EINVAL; - - neo_dcp->disp_bw_scratch_res = *neo_dcp->hw.firmware_clock; - neo_dcp->disp_bw_scratch_index = count; - neo_dcp->disp_bw_scratch_offset = neo_dcp->hw.firmware_scratch; - neo_dcp->disp_bw_doorbell_res = neo_dcp->disp_bw_scratch_res; - neo_dcp->disp_bw_doorbell_res.start += neo_dcp->hw.firmware_request; - neo_dcp->disp_bw_doorbell_res.end = neo_dcp->disp_bw_doorbell_res.start + 3; - neo_dcp->disp_registers[count] = &neo_dcp->disp_bw_scratch_res; - neo_dcp->nr_disp_registers = count + 1; - return 0; - } - - /* load pmgr bandwidth scratch resource and offset */ - ret = neo_dcp_get_bw_scratch_reg(neo_dcp, count); - if (ret < 0) - return ret; - count += 1; - - /* load pmgr bandwidth doorbell resource if present (only on t8103) */ - if (of_property_present(neo_dcp->dev->of_node, "apple,bw-doorbell")) { - ret = neo_dcp_get_bw_doorbell_reg(neo_dcp, count); - if (ret < 0) - return ret; - count += 1; - } - - neo_dcp->nr_disp_registers = count; - return 0; -} - -#define DCP_FW_VERSION_MIN_LEN 3 -#define DCP_FW_VERSION_MAX_LEN 5 -#define DCP_FW_VERSION_STR_LEN (DCP_FW_VERSION_MAX_LEN * 4) - -static int neo_dcp_read_fw_version(struct device *dev, const char *name, - char *version_str) -{ - u32 ver[DCP_FW_VERSION_MAX_LEN]; - int len_str; - int len; - - len = of_property_read_variable_u32_array(dev->of_node, name, ver, - DCP_FW_VERSION_MIN_LEN, - DCP_FW_VERSION_MAX_LEN); - - switch (len) { - case 3: - len_str = scnprintf(version_str, DCP_FW_VERSION_STR_LEN, - "%d.%d.%d", ver[0], ver[1], ver[2]); - break; - case 4: - len_str = scnprintf(version_str, DCP_FW_VERSION_STR_LEN, - "%d.%d.%d.%d", ver[0], ver[1], ver[2], - ver[3]); - break; - case 5: - len_str = scnprintf(version_str, DCP_FW_VERSION_STR_LEN, - "%d.%d.%d.%d.%d", ver[0], ver[1], ver[2], - ver[3], ver[4]); - break; - default: - len_str = strscpy(version_str, "UNKNOWN", - DCP_FW_VERSION_STR_LEN); - if (len >= 0) - len = -EOVERFLOW; - break; - } - - if (len_str >= DCP_FW_VERSION_STR_LEN) - dev_warn(dev, "'%s' truncated: '%s'\n", name, version_str); - - return len; -} - -static enum neo_dcp_firmware_version neo_dcp_check_firmware_version(struct device *dev) -{ - const struct neo_apple_dcp_hw_data *hw = of_device_get_match_data(dev); - char compat_str[DCP_FW_VERSION_STR_LEN]; - char fw_str[DCP_FW_VERSION_STR_LEN]; - int ret; - - /* - * SoCs introduced with H17-generation firmware pin its interface; the - * loader may not recognise their boot firmware version. - */ - if (hw->firmware_compat != DCP_FIRMWARE_UNKNOWN) - return hw->firmware_compat; - - /* firmware version is just informative */ - neo_dcp_read_fw_version(dev, "apple,firmware-version", fw_str); - - ret = neo_dcp_read_fw_version(dev, "apple,firmware-compat", compat_str); - if (ret < 0) { - dev_err(dev, "Could not read 'apple,firmware-compat': %d\n", ret); - return DCP_FIRMWARE_UNKNOWN; - } - - if (strncmp(compat_str, "12.3.0", sizeof(compat_str)) == 0) - return DCP_FIRMWARE_V_12_3; - /* - * m1n1 reports firmware version 13.5 as compatible with 13.3. This is - * only true for the iomfb endpoint. The interface for the dptx-port - * endpoint changed between 13.3 and 13.5. The driver will only support - * firmware 13.5. Check the actual firmware version for compat version - * 13.3 until m1n1 reports 13.5 as "firmware-compat". - */ - else if ((strncmp(compat_str, "13.3.0", sizeof(compat_str)) == 0) && - (strncmp(fw_str, "13.5.0", sizeof(compat_str)) == 0)) - return DCP_FIRMWARE_V_13_5; - else if (strncmp(compat_str, "13.5.0", sizeof(compat_str)) == 0) - return DCP_FIRMWARE_V_13_5; - - dev_err(dev, "DCP firmware-compat %s (FW: %s) is not supported\n", - compat_str, fw_str); - - return DCP_FIRMWARE_UNKNOWN; -} - -static int neo_dcp_connector_type_from_dt(struct device_node *np) -{ - if (of_property_match_string(np, "apple,connector-type", "HDMI-A") >= 0) - return DRM_MODE_CONNECTOR_HDMIA; - if (of_property_match_string(np, "apple,connector-type", "DP") >= 0) - return DRM_MODE_CONNECTOR_DisplayPort; - if (of_property_match_string(np, "apple,connector-type", "USB-C") >= 0) - return DRM_MODE_CONNECTOR_USB; - - return DRM_MODE_CONNECTOR_Unknown; -} - -static void neo_dcp_disable_typec_work(struct neo_apple_dcp *neo_dcp, bool release_cable) -{ - scoped_guard(mutex, &neo_dcp->hpd_mutex) { - if (release_cable) - WRITE_ONCE(neo_dcp->typec_cable_connected, false); - neo_dcp->typec_generation++; - } - /* Block new enqueues as well as draining users of the AFK endpoints. */ - disable_delayed_work_sync(&neo_dcp->typec_reconnect_wq); - disable_delayed_work_sync(&neo_dcp->placeholder_edid_wq); - disable_delayed_work_sync(&neo_dcp->typec_fabric_retrain_wq); -} - -static void neo_dcp_enable_typec_work(struct neo_apple_dcp *neo_dcp) -{ - enable_delayed_work(&neo_dcp->typec_reconnect_wq); - enable_delayed_work(&neo_dcp->placeholder_edid_wq); - enable_delayed_work(&neo_dcp->typec_fabric_retrain_wq); - /* A cable can be routed before the DRM component binds. */ - if (READ_ONCE(neo_dcp->typec_cable_connected)) - mod_delayed_work(system_freezable_wq, &neo_dcp->typec_reconnect_wq, 0); -} - -static int neo_dcp_comp_bind(struct device *dev, struct device *main, void *data) -{ - struct device_node *panel_np; - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - u32 cpu_ctrl; - int ret; - - if (READ_ONCE(neo_dcp->quiescing)) - return dev_err_probe(dev, -EBUSY, - "Live DCP session cannot be rebound; reboot required\n"); - - /* A timed-out prior session may still scan these mappings. */ - mutex_lock(&neo_dcp->swapped_out_fbs_lock); - ret = list_empty(&neo_dcp->swapped_out_fbs) ? 0 : -EBUSY; - mutex_unlock(&neo_dcp->swapped_out_fbs_lock); - if (ret) - return dev_err_probe(dev, ret, - "Previous scanout has not been stopped\n"); - - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(42)); - if (ret) - return ret; - - neo_dcp->coproc_reg = devm_platform_ioremap_resource_byname(to_platform_device(dev), "coproc"); - if (IS_ERR(neo_dcp->coproc_reg)) - return PTR_ERR(neo_dcp->coproc_reg); - - /* - * Display coprocessors running H17-generation firmware are left running - * by the bootloader with their firmware mapped through a locked DART. - * Linux attaches to that session with a standard RTKit INIT/HELLO - * exchange and must not stop or restart the ASC. - */ - if (neo_dcp->hw.adopt_live_session) { - cpu_ctrl = readl_relaxed(neo_dcp->coproc_reg + - APPLE_DCP_COPROC_CPU_CONTROL); - if (!(cpu_ctrl & APPLE_DCP_COPROC_CPU_CONTROL_RUN)) - return dev_err_probe(dev, -EOPNOTSUPP, - "DCP was not left running by the bootloader\n"); - } - - if (neo_dcp->index || neo_dcp->neo_dptx_phy || neo_dcp->neo_dptx_die) - dev_info(dev, "DCP index:%u dptx target phy: %u dptx die: %u\n", - neo_dcp->index, neo_dcp->neo_dptx_phy, neo_dcp->neo_dptx_die); - - if (!show_notch) - ret = of_property_read_u32(dev->of_node, "apple,notch-height", - &neo_dcp->notch_height); - - if (neo_dcp->notch_height > MAX_NOTCH_HEIGHT) - neo_dcp->notch_height = MAX_NOTCH_HEIGHT; - if (neo_dcp->notch_height > 0) - dev_info(dev, "Detected display with notch of %u pixel\n", neo_dcp->notch_height); - - /* initialize brightness scale to a sensible default to avoid divide by 0*/ - neo_dcp->brightness.scale = 65536; - panel_np = of_get_compatible_child(dev->of_node, "apple,panel-mini-led"); - if (panel_np) - neo_dcp->panel.has_mini_led = true; - else - panel_np = of_get_compatible_child(dev->of_node, "apple,panel"); - - if (panel_np) { - const char height_prop[2][16] = { "adj-height-mm", "height-mm" }; - - if (of_device_is_available(panel_np)) { - ret = of_property_read_u32(panel_np, "apple,max-brightness", - &neo_dcp->brightness.maximum); - if (ret) - dev_err(dev, "Missing property 'apple,max-brightness'\n"); - } - - of_property_read_u32(panel_np, "width-mm", &neo_dcp->panel.width_mm); - /* use adjusted height as long as the notch is hidden */ - of_property_read_u32(panel_np, height_prop[!neo_dcp->notch_height], - &neo_dcp->panel.height_mm); - - of_node_put(panel_np); - neo_dcp->fixed_connector_type = DRM_MODE_CONNECTOR_eDP; - neo_dcp->connector_type = DRM_MODE_CONNECTOR_eDP; - INIT_WORK(&neo_dcp->bl_register_wq, neo_dcp_work_register_backlight); - mutex_init(&neo_dcp->bl_register_mutex); - INIT_WORK(&neo_dcp->bl_update_wq, neo_dcp_work_update_backlight); - } - - ret = neo_dcp_create_piodma_iommu_dev(neo_dcp); - if (ret) - return dev_err_probe(dev, ret, - "Failed to create PIODMA iommu child device\n"); - if (!neo_dcp->iommu_dom) { - ret = dev_err_probe(dev, -EPROBE_DEFER, - "PIODMA iommu domain is unavailable\n"); - goto err_piodma; - } - - ret = neo_dcp_get_disp_regs(neo_dcp); - if (ret) { - dev_err(dev, "failed to find display registers\n"); - goto err_piodma; - } - - neo_dcp->clk = devm_clk_get(dev, NULL); - if (IS_ERR(neo_dcp->clk)) { - ret = dev_err_probe(dev, PTR_ERR(neo_dcp->clk), - "Unable to find clock\n"); - goto err_piodma; - } - neo_dcp->clk_194 = devm_clk_get_optional(dev, "clock-194"); - if (IS_ERR(neo_dcp->clk_194)) { - ret = dev_err_probe(dev, PTR_ERR(neo_dcp->clk_194), - "Unable to find clock 0x194\n"); - goto err_piodma; - } - - bitmap_zero(neo_dcp->memdesc_map, DCP_MAX_MAPPINGS); - // TDOD: mem_desc IDs start at 1, for simplicity just skip '0' entry - set_bit(0, neo_dcp->memdesc_map); - - INIT_WORK(&neo_dcp->vblank_wq, neo_dcp_delayed_vblank); - INIT_DELAYED_WORK(&neo_dcp->swap_watchdog_wq, neo_dcp_swap_watchdog); - - if (!neo_dcp->hw.adopt_live_session) { - cpu_ctrl = - readl_relaxed(neo_dcp->coproc_reg + APPLE_DCP_COPROC_CPU_CONTROL); - writel_relaxed(cpu_ctrl | APPLE_DCP_COPROC_CPU_CONTROL_RUN, - neo_dcp->coproc_reg + APPLE_DCP_COPROC_CPU_CONTROL); - } - - /* Registering the mailbox can already expose RTKit buffer requests. */ - ret = neo_dcp_pin_live_session(neo_dcp); - if (ret) - goto err_piodma; - neo_dcp->rtk = apple_rtkit_init(dev, neo_dcp, "mbox", 0, &rtkit_ops); - if (IS_ERR(neo_dcp->rtk)) { - ret = dev_err_probe(dev, PTR_ERR(neo_dcp->rtk), - "Failed to initialize RTKit\n"); - goto err_piodma; - } - ret = devm_add_action_or_reset(dev, neo_dcp_release_rtkit, neo_dcp); - if (ret) - goto err_piodma; - - if (neo_dcp->hw.adopt_live_session) - dev_info(dev, "negotiating RTKit with the running DCP\n"); - - ret = apple_rtkit_wake(neo_dcp->rtk); - if (ret) { - ret = dev_err_probe(dev, ret, "Failed to boot RTKit: %d\n", ret); - goto err_piodma; - } - neo_dcp_enable_typec_work(neo_dcp); - return 0; - -err_piodma: - neo_dcp_release_piodma_iommu_dev(neo_dcp); - return ret; -} - -/* - * We need to shutdown DCP before tearing down the display subsystem. Otherwise - * the DCP will crash and briefly flash a green screen of death. - */ -static void neo_dcp_comp_unbind(struct device *dev, struct device *main, void *data) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - - if (!neo_dcp) - return; - - if (neo_dcp->retain_dma) { - /* No firmware stop is acknowledged: detach host users, never DMA. */ - WRITE_ONCE(neo_dcp->quiescing, true); - if (neo_dcp->crtc && !neo_dcp->drm_retained) { - drm_dev_get(neo_dcp->crtc->base.dev); - neo_dcp->drm_retained = true; - } - neo_iomfb_queue_stop(neo_dcp); - neo_afk_quiesce(neo_dcp->neo_avep); - neo_afk_quiesce(neo_dcp->neo_dptxep); - neo_afk_quiesce(neo_dcp->neo_ibootep); - neo_afk_quiesce(neo_dcp->neo_systemep); - neo_afk_quiesce(neo_dcp->dcpavservep); - cancel_delayed_work_sync(&neo_dcp->bl_fallback_wq); - if (neo_dcp->connector_type == DRM_MODE_CONNECTOR_eDP) { - cancel_work_sync(&neo_dcp->bl_register_wq); - cancel_work_sync(&neo_dcp->bl_update_wq); - } - cancel_delayed_work_sync(&neo_dcp->typec_reconnect_wq); - cancel_delayed_work_sync(&neo_dcp->typec_fabric_retrain_wq); - cancel_work_sync(&neo_dcp->vblank_wq); - if (neo_dcp->connector) - cancel_work_sync(&neo_dcp->connector->hotplug_wq); - if (neo_dcp->typec_connector && neo_dcp->typec_connector != neo_dcp->connector) - cancel_work_sync(&neo_dcp->typec_connector->hotplug_wq); - dev_warn(dev, "retaining live DCP DMA resources after unbind; reboot required\n"); - return; - } - - if (neo_dcp->hdmi_hpd_irq) - disable_irq(neo_dcp->hdmi_hpd_irq); - - neo_dcp_disable_typec_work(neo_dcp, true); - typec_mux_put(neo_dcp->typec_mux); - - if (neo_dcp->neo_avep) { - neo_av_service_disconnect(neo_dcp); - neo_afk_shutdown(neo_dcp->neo_avep); - neo_dcp->neo_avep = NULL; - } - - if (neo_dcp->neo_dptxep) { - neo_afk_shutdown(neo_dcp->neo_dptxep); - neo_dcp->neo_dptxep = NULL; - } - - if (neo_dcp->neo_ibootep) { - neo_afk_shutdown(neo_dcp->neo_ibootep); - neo_dcp->neo_ibootep = NULL; - } - - if (neo_dcp->neo_systemep) { - neo_afk_shutdown(neo_dcp->neo_systemep); - neo_dcp->neo_systemep = NULL; - } - - if (neo_dcp->dcpavservep) { - neo_afk_shutdown(neo_dcp->dcpavservep); - dpavservep_detach(neo_dcp); - neo_dcp->dcpavservep = NULL; - } - - if (neo_dcp->shmem) - neo_iomfb_shutdown(neo_dcp); - - neo_iomfb_queue_stop(neo_dcp); - neo_dcp_release_piodma_iommu_dev(neo_dcp); - - cancel_delayed_work_sync(&neo_dcp->bl_fallback_wq); - if (neo_dcp->connector_type == DRM_MODE_CONNECTOR_eDP) { - cancel_work_sync(&neo_dcp->bl_register_wq); - cancel_work_sync(&neo_dcp->bl_update_wq); - } - cancel_delayed_work_sync(&neo_dcp->swap_watchdog_wq); - cancel_work_sync(&neo_dcp->vblank_wq); - - devm_clk_put(dev, neo_dcp->clk); - neo_dcp->clk = NULL; - /* optional and possibly NULL, released with the bind's devres group */ - neo_dcp->clk_194 = NULL; -} - -static const struct component_ops neo_dcp_comp_ops = { - .bind = neo_dcp_comp_bind, - .unbind = neo_dcp_comp_unbind, -}; - -static int neo_dcp_platform_probe(struct platform_device *pdev) -{ - const struct neo_apple_dcp_hw_data *hw = of_device_get_match_data(&pdev->dev); - enum neo_dcp_firmware_version fw_compat; - struct device *dev = &pdev->dev; - struct neo_apple_dcp *neo_dcp; - int ret, surf, num_surfs; - u32 surf_en; - u32 mux_index; - - if (neo_dcp_session_retained(dev)) - return dev_err_probe(dev, -EBUSY, "Previous live DCP session requires a reboot\n"); - - /* Not part of the display subsystem yet; see apple_dcp_usable(). */ - if (of_device_is_compatible(dev->of_node, "apple,t8140-dcpext")) { - dev_info(dev, "external display coprocessor not supported yet\n"); - return -ENODEV; - } - - fw_compat = neo_dcp_check_firmware_version(dev); - if (fw_compat == DCP_FIRMWARE_UNKNOWN) - return -ENODEV; - - /* Check for "apple,bw-scratch" to avoid probing appledrm_neo with outdated - * device trees. This prevents replacing simpledrm and ending up without - * display. - */ - if (!of_property_present(dev->of_node, "apple,bw-scratch") && - !hw->firmware_clock) - return dev_err_probe(dev, -ENODEV, "Incompatible devicetree! " - "Use devicetree matching this kernel.\n"); - - neo_dcp = kzalloc_obj(*neo_dcp); - if (!neo_dcp) - return -ENOMEM; - ret = devm_add_action_or_reset(dev, neo_dcp_release_context, neo_dcp); - if (ret) - return ret; - - INIT_LIST_HEAD(&neo_dcp->swapped_out_fbs); - mutex_init(&neo_dcp->swapped_out_fbs_lock); - spin_lock_init(&neo_dcp->backlight.lock); - neo_iomfb_queue_init(neo_dcp); - - neo_dcp->fw_compat = fw_compat; - neo_dcp->dev = dev; - /* - * Type-C and Thunderbolt routes can be activated as soon as they are - * registered below, before the DRM device binds. - */ - mutex_init(&neo_dcp->hpd_mutex); - mutex_init(&neo_dcp->tb_lock); - spin_lock_init(&neo_dcp->mode_state.lock); - spin_lock_init(&neo_dcp->neo_dcpavserv.lock); - neo_dcp->hw = *(struct neo_apple_dcp_hw_data *)of_device_get_match_data(dev); - - /* The domain use must survive driver-core DMA cleanup for a live session. */ - ret = iommu_device_use_default_domain(dev); - if (ret) - return ret; - ret = devm_add_action_or_reset(dev, neo_dcp_release_dma_domain, neo_dcp); - if (ret) - return ret; - neo_dcp->fixed_connector_type = neo_dcp_connector_type_from_dt(dev->of_node); - neo_dcp->connector_type = neo_dcp->fixed_connector_type; - of_property_read_u32(dev->of_node, "apple,dcp-index", &neo_dcp->index); - of_property_read_u32(dev->of_node, "apple,dptx-phy", &neo_dcp->neo_dptx_phy); - of_property_read_u32(dev->of_node, "apple,dptx-die", &neo_dcp->neo_dptx_die); - neo_dcp->fixed_dptx_phy = neo_dcp->neo_dptx_phy; - init_completion(&neo_dcp->typec_iomfb_hpd_ready); - INIT_DELAYED_WORK(&neo_dcp->typec_reconnect_wq, - neo_dcp_typec_reconnect_work); - INIT_DELAYED_WORK(&neo_dcp->placeholder_edid_wq, - neo_dcp_placeholder_edid_work); - INIT_DELAYED_WORK(&neo_dcp->typec_fabric_retrain_wq, - neo_dcp_typec_retrain_work); - INIT_DELAYED_WORK(&neo_dcp->bl_fallback_wq, neo_dcp_work_backlight_fallback); - /* Balanced by enable at successful component bind. */ - disable_delayed_work(&neo_dcp->typec_reconnect_wq); - disable_delayed_work(&neo_dcp->placeholder_edid_wq); - disable_delayed_work(&neo_dcp->typec_fabric_retrain_wq); - - platform_set_drvdata(pdev, neo_dcp); - - neo_dcp->phy = devm_phy_optional_get(dev, "dp-phy"); - if (IS_ERR(neo_dcp->phy)) { - dev_err(dev, "Failed to get dp-phy: %ld\n", PTR_ERR(neo_dcp->phy)); - return PTR_ERR(neo_dcp->phy); - } - neo_dcp->fixed_phy = neo_dcp->phy; - - bitmap_zero(neo_dcp->neo_iomfb_surfaces, DCP_MAX_PLANES); - if (!of_property_present(dev->of_node, "apple,iomfb-surfaces")) - num_surfs = 0; - else - num_surfs = of_property_count_elems_of_size(dev->of_node, - "apple,iomfb-surfaces", - sizeof(u32)); - - if (num_surfs == 0 || num_surfs == -ENODATA) { - set_bit(0, neo_dcp->neo_iomfb_surfaces); - set_bit(1, neo_dcp->neo_iomfb_surfaces); - } else if (num_surfs < 0) { - return num_surfs; - } else if (num_surfs > DCP_MAX_PLANES) { - dev_err(dev, "Number of iomfb-surfaces (%d) exceeds DCP_MAX_PLANES\n", - num_surfs); - return -EINVAL; - } - - surf = 0; - of_property_for_each_u32(dev->of_node, "apple,iomfb-surfaces", surf_en) { - if (surf_en) - set_bit(surf, neo_dcp->neo_iomfb_surfaces); - surf++; - } - /* The qualified J700 loader names surface zero rather than enabling it. */ - if (of_machine_is_compatible("apple,j700") && - of_device_is_compatible(dev->of_node, "apple,t8140-dcp") && - num_surfs == 1 && - !of_property_read_u32(dev->of_node, "apple,iomfb-surfaces", &surf_en) && - surf_en == 0) - set_bit(0, neo_dcp->neo_iomfb_surfaces); - - if (neo_dcp->phy) { - int ret; - /* - * Request DP2HDMI related GPIOs as optional for DP-altmode - * compatibility. J180D misses a dp2hdmi-pwren GPIO in the - * template ADT. TODO: check device ADT - */ - neo_dcp->hdmi_hpd = devm_gpiod_get_optional(dev, "hdmi-hpd", GPIOD_IN); - if (IS_ERR(neo_dcp->hdmi_hpd)) - return PTR_ERR(neo_dcp->hdmi_hpd); - if (neo_dcp->hdmi_hpd) { - int irq = gpiod_to_irq(neo_dcp->hdmi_hpd); - if (irq < 0) { - dev_err(dev, "failed to translate HDMI hpd GPIO to IRQ\n"); - return irq; - } - neo_dcp->hdmi_hpd_irq = irq; - - ret = devm_request_threaded_irq(dev, neo_dcp->hdmi_hpd_irq, - NULL, neo_dcp_dp2hdmi_hpd, - IRQF_ONESHOT | IRQF_NO_AUTOEN | - IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING, - "dp2hdmi-hpd-irq", neo_dcp); - if (ret < 0) { - dev_err(dev, "failed to request HDMI hpd irq %d: %d\n", - irq, ret); - return ret; - } - } - - /* - * Power DP2HDMI on as it is required for the HPD irq. - * TODO: check if one is sufficient for the hpd to save power - * on battery powered Macbooks. - */ - neo_dcp->hdmi_pwren = devm_gpiod_get_optional(dev, "hdmi-pwren", GPIOD_OUT_HIGH); - if (IS_ERR(neo_dcp->hdmi_pwren)) - return PTR_ERR(neo_dcp->hdmi_pwren); - - neo_dcp->dp2hdmi_pwren = devm_gpiod_get_optional(dev, "dp2hdmi-pwren", GPIOD_OUT_HIGH); - if (IS_ERR(neo_dcp->dp2hdmi_pwren)) - return PTR_ERR(neo_dcp->dp2hdmi_pwren); - - /* - * A DCP may have both a fixed HDMI/DP route and allocatable Type-C - * routes. Keep the fixed route selected until the allocator borrows - * this otherwise-idle pipeline for a Type-C display. - */ - ret = neo_dcp->fixed_phy ? - of_property_read_u32(dev->of_node, "mux-index", &mux_index) : - -ENODATA; - if (!ret) { - neo_dcp->fixed_mux_index = mux_index; - neo_dcp->xbar = devm_mux_control_get(dev, "dp-xbar"); - if (IS_ERR(neo_dcp->xbar)) { - dev_err(dev, "Failed to get dp-xbar: %ld\n", PTR_ERR(neo_dcp->xbar)); - return PTR_ERR(neo_dcp->xbar); - } - ret = mux_control_select(neo_dcp->xbar, mux_index); - if (ret) - dev_warn(dev, "mux_control_select failed: %d\n", ret); - else - neo_dcp->fixed_route_selected = true; - - /* - * Switch atcphy to DP-only. should move to a Macbook Pro - * 14-/16-inch specific DP-to-HDMI drm_bridge. - */ - neo_dcp->typec_mux = fwnode_typec_mux_get(dev_fwnode(neo_dcp->dev)); - if (!IS_ERR_OR_NULL(neo_dcp->typec_mux)) { - struct typec_altmode alt = { - .svid = USB_TYPEC_DP_SID, - }; - struct typec_mux_state state = { - .alt = &alt, - .mode = TYPEC_DP_STATE_C, - }; - int ret = typec_mux_set(neo_dcp->typec_mux, &state); - dev_info(dev, "typec_mux_set() returned: %d\n", ret); - if (!ret) - neo_dcp->phy_managed_by_typec = true; - } else { - dev_info(dev, "fwnode_typec_mux_get() returned: %ld\n", - IS_ERR(neo_dcp->typec_mux) ? PTR_ERR(neo_dcp->typec_mux) : 0); - neo_dcp->typec_mux = NULL; - } - } - } - - ret = neo_dcp_register_typec_routes(neo_dcp); - if (ret) - return ret; - - return component_add(&pdev->dev, &neo_dcp_comp_ops); -} - -static void neo_dcp_platform_remove(struct platform_device *pdev) -{ - component_del(&pdev->dev, &neo_dcp_comp_ops); -} - -static void neo_dcp_platform_shutdown(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - if (neo_dcp && neo_dcp->hw.adopt_live_session && - neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - /* - * This firmware keeps scanning across soft DPMS and has no - * qualified stop sequence. Component unbind releases the RTKit - * devres group and the piodma domain while firmware may still - * access them. Stop host submissions but keep the component and - * its DMA resources alive until the system resets. - */ - neo_iomfb_queue_stop(neo_dcp); - cancel_delayed_work_sync(&neo_dcp->bl_fallback_wq); - cancel_work_sync(&neo_dcp->bl_register_wq); - cancel_work_sync(&neo_dcp->bl_update_wq); - return; - } - - component_del(&pdev->dev, &neo_dcp_comp_ops); -} - -/* - * The display stays powered across system sleep, and the blanking present - * queued by the DRM and backlight suspend handlers must reach the firmware - * before its mailbox interrupt is suspended. - */ -static void neo_dcp_drain_for_sleep(struct neo_apple_dcp *neo_dcp) -{ - int pass; - - if (!neo_dcp->hw.adopt_live_session || neo_dcp->fw_compat != DCP_FIRMWARE_H17P || - neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return; - - /* - * A completed present can schedule one more brightness present, and a - * rejected one a delayed retry; run a pending retry now instead of - * letting it fire after the mailbox has suspended. Four passes cover - * the first attempt and the policy's three retries. - */ - for (pass = 0; pass < 4; pass++) { - flush_delayed_work(&neo_dcp->neo_iomfb.backlight_retry); - if (neo_dcp_has_panel(neo_dcp)) - flush_work(&neo_dcp->bl_update_wq); - if (!neo_iomfb_queue_drain(neo_dcp, msecs_to_jiffies(500))) { - dev_warn(neo_dcp->dev, "display updates still pending at suspend\n"); - return; - } - /* - * A crashed or stopped DCP takes no more presents, so a level - * still pending there can never be sent and is not waited for. - * Checking after the drain, which returns at once for such a - * DCP, also covers a crash during the pass. - */ - if (READ_ONCE(neo_dcp->crashed) || READ_ONCE(neo_dcp->neo_iomfb.stopped)) - return; - if (!neo_dcp_backlight_active(neo_dcp) || !neo_dcp_backlight_pending(neo_dcp)) - return; - } - dev_warn(neo_dcp->dev, "brightness change still pending at suspend\n"); -} - -static int neo_dcp_platform_suspend(struct device *dev) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - - neo_dcp_drain_for_sleep(neo_dcp); - /* - * The Type-C route reports cable removal through - * dcp_dptx_disconnect_oob(). A DP tunnel kept through the sleep stays - * connected, and resume powers the CRTC back up through dcp_poweron() - * as after DPMS off. - */ - neo_dcp_disable_typec_work(neo_dcp, false); - cancel_delayed_work_sync(&neo_dcp->swap_watchdog_wq); - - if (neo_dcp->neo_avep) - neo_av_service_disconnect(neo_dcp); - - if (neo_dcp->hdmi_hpd_irq) { - disable_irq(neo_dcp->hdmi_hpd_irq); - if (!neo_dcp->active_typec_route) { - disconnected_hpd_event(neo_dcp->connector); - neo_dcp_dptx_disconnect(neo_dcp, 0); - } - } - /* - * Set the device as a wakeup device, which forces its power - * domains to stay on. We need this as we do not support full - * shutdown properly yet. - */ - device_set_wakeup_path(dev); - - return 0; -} - -static int neo_dcp_platform_resume(struct device *dev) -{ - struct neo_apple_dcp *neo_dcp = dev_get_drvdata(dev); - - neo_dcp_enable_typec_work(neo_dcp); - if (neo_dcp->hdmi_hpd_irq) - enable_irq(neo_dcp->hdmi_hpd_irq); - - if (neo_dcp->neo_avep) - neo_av_service_connect(neo_dcp); - - return 0; -} - -static DEFINE_SIMPLE_DEV_PM_OPS(neo_dcp_platform_pm_ops, - neo_dcp_platform_suspend, neo_dcp_platform_resume); - - -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_t6020 = { - .num_dptx_ports = 1, -}; - -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_t8112 = { - .num_dptx_ports = 2, -}; - -/* The internal T8140 endpoint uses the measured H17P method profile. */ -static const struct resource t8140_firmware_clock = - DEFINE_RES_MEM(0x302800000, 0xbc000); - -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_t8140 = { - .num_dptx_ports = 0, - .adopt_live_session = true, - .firmware_compat = DCP_FIRMWARE_H17P, - .firmware_clock = &t8140_firmware_clock, - .firmware_scratch = 0x20000, - .firmware_request = 0x68000, -}; - -/* - * M5 (T8142) runs H17-generation DCP firmware with the H17G method numbering - * and is left running by the bootloader. - */ -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_t8142 = { - .num_dptx_ports = 0, - .neo_iomfb_method_profile = DCP_IOMFB_METHODS_H17G, - .adopt_live_session = true, - .firmware_compat = DCP_FIRMWARE_H17P, -}; - -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_dcp = { - .num_dptx_ports = 0, -}; - -static const struct neo_apple_dcp_hw_data neo_apple_dcp_hw_dcpext = { - .num_dptx_ports = 2, -}; - -static const struct of_device_id of_match[] = { - { .compatible = "apple,t8140-dcp", .data = &neo_apple_dcp_hw_t8140, }, - {} -}; -MODULE_DEVICE_TABLE(of, of_match); - -static struct platform_driver apple_platform_driver = { - .driver_managed_dma = true, - .probe = neo_dcp_platform_probe, - .remove = neo_dcp_platform_remove, - .shutdown = neo_dcp_platform_shutdown, - .driver = { - .name = "apple-dcp-neo", - .of_match_table = of_match, - .pm = pm_sleep_ptr(&neo_dcp_platform_pm_ops), - }, -}; - -int __init neo_dcp_register(void) -{ - return platform_driver_register(&apple_platform_driver); -} - -void neo_dcp_unregister(void) -{ - platform_driver_unregister(&apple_platform_driver); -} diff --git a/drivers/gpu/drm/apple-neo/dcp.h b/drivers/gpu/drm/apple-neo/dcp.h deleted file mode 100644 index 7acf095f7b5022..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp.h +++ /dev/null @@ -1,108 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#ifndef __APPLE_DCP_H__ -#define __APPLE_DCP_H__ - -#include -#include -#include - -#include "connector.h" -#include "dcp-internal.h" -#include "parser.h" - -struct neo_apple_crtc { - struct drm_crtc base; - struct drm_pending_vblank_event *event; - bool vsync_disabled; - - /* Reference to the DCP device owning this CRTC */ - struct platform_device *neo_dcp; -}; - -#define to_apple_crtc(x) container_of(x, struct neo_apple_crtc, base) - -struct apple_encoder { - struct drm_encoder base; -}; - -#define to_apple_encoder(x) container_of(x, struct apple_encoder, base) - -void neo_dcp_poweroff(struct platform_device *pdev); -void neo_dcp_poweron(struct platform_device *pdev); -int neo_dcp_set_crc(struct drm_crtc *crtc, bool enabled); -int neo_dcp_crtc_atomic_check(struct drm_crtc *crtc, struct drm_atomic_state *state); -int neo_dcp_get_connector_type(struct platform_device *pdev); -bool neo_dcp_has_typec_routes(struct platform_device *pdev); - -/* - * The Type-C display fabric. Ports are enumerated in device-tree order, not - * DCP probe order, so a given physical port keeps the same DRM connector index - * across boots -- userspace keys its per-monitor configuration on that name. - */ -unsigned int neo_dcp_typec_nr_ports(void); -struct device_node *neo_dcp_typec_port_of_node(unsigned int idx); -bool neo_dcp_typec_port_has_candidate(unsigned int idx, struct platform_device *pdev); -void neo_dcp_typec_port_set_connector(unsigned int idx, bool secondary, - struct neo_apple_connector *connector); -bool neo_dcp_typec_dual_stream(void); -void neo_dcp_typec_reorder(void); -bool neo_dcp_is_typec_only(struct platform_device *pdev); -bool neo_dcp_fw_compat_is_12_x(struct platform_device *pdev); -unsigned long* neo_dcp_get_iomfb_surfaces(struct platform_device *pdev); -void neo_dcp_link(struct platform_device *pdev, struct neo_apple_crtc *apple, - struct neo_apple_connector *connector); -int neo_dcp_start(struct platform_device *pdev); -int neo_dcp_wait_ready(struct platform_device *pdev, u64 timeout); -void neo_dcp_flush(struct drm_crtc *crtc, struct drm_atomic_state *state); -void neo_dcp_retain_framebuffer(struct platform_device *pdev, - struct neo_dcp_fb_reference *entry); -void neo_dcp_arm_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id); -void neo_dcp_unarm_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id); -void neo_dcp_release_retained_framebuffers(struct neo_apple_dcp *neo_dcp, u32 swap_id); -void neo_dcp_release_all_retained_framebuffers(struct neo_apple_dcp *neo_dcp); -bool neo_dcp_is_initialized(struct platform_device *pdev); -void neo_apple_crtc_vblank(struct neo_apple_crtc *apple); -void neo_dcp_drm_crtc_vblank(struct neo_apple_crtc *crtc); -int neo_dcp_get_modes(struct drm_connector *connector); -enum drm_mode_status neo_dcp_mode_valid(struct drm_connector *connector, - const struct drm_display_mode *mode); -int neo_dcp_crtc_atomic_modeset(struct drm_crtc *crtc, - struct drm_atomic_state *state); -bool neo_dcp_crtc_mode_fixup(struct drm_crtc *crtc, - const struct drm_display_mode *mode, - struct drm_display_mode *adjusted_mode); -void neo_dcp_set_dimensions(struct neo_apple_dcp *neo_dcp); -void neo_dcp_send_message(struct neo_apple_dcp *neo_dcp, u8 endpoint, u64 message); - -int neo_dcp_dptx_connect_oob(struct platform_device *pdev, u32 port); -int neo_dcp_dptx_disconnect_oob(struct platform_device *pdev, u32 port); - -struct neo_apple_dcp; -struct phy; -/* Thunderbolt DP tunnels, called from the DPTX endpoint */ -int neo_dcp_tunnel_crossbar_up(struct neo_apple_dcp *neo_dcp); -int neo_dcp_tunnel_crossbar_down(struct neo_apple_dcp *neo_dcp); -int neo_dcp_tunnel_set_rate(struct neo_apple_dcp *neo_dcp, struct phy *phy, u32 link_rate); -int neo_dcp_tunnel_dpin_activate(struct neo_apple_dcp *neo_dcp, bool active); - -int neo_iomfb_start_rtkit(struct neo_apple_dcp *neo_dcp); -void neo_iomfb_shutdown(struct neo_apple_dcp *neo_dcp); -/* rtkit message handler for IOMFB messages */ -void neo_iomfb_recv_msg(struct neo_apple_dcp *neo_dcp, u64 message); - -int neo_systemep_init(struct neo_apple_dcp *neo_dcp); -int neo_dptxep_init(struct neo_apple_dcp *neo_dcp); -int neo_ibootep_init(struct neo_apple_dcp *neo_dcp); -int dpavservep_init(struct neo_apple_dcp *neo_dcp); -int neo_avep_init(struct neo_apple_dcp *neo_dcp); - - -int __init neo_dcp_register(void); -void neo_dcp_unregister(void); - -int __init neo_dcp_audio_register(void); -void neo_dcp_audio_unregister(void); - -#endif diff --git a/drivers/gpu/drm/apple-neo/dcp_backlight.c b/drivers/gpu/drm/apple-neo/dcp_backlight.c deleted file mode 100644 index f9fad8ca2af323..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp_backlight.c +++ /dev/null @@ -1,476 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright (C) The Asahi Linux Contributors */ - -#include -#include -#include -#include - -#include -#include -#include -#include "linux/jiffies.h" - -#include "dcp.h" -#include "dcp-internal.h" - -#define MIN_BRIGHTNESS_PART1 2U -#define MAX_BRIGHTNESS_PART1 99U -#define MIN_BRIGHTNESS_PART2 103U -#define MAX_BRIGHTNESS_PART2 510U - -/* - * lookup for display brightness 2 to 99 nits - * */ -static u32 brightness_part1[] = { - 0x0000000, 0x0810038, 0x0f000bd, 0x143011c, - 0x1850165, 0x1bc01a1, 0x1eb01d4, 0x2140200, - 0x2380227, 0x2590249, 0x2770269, 0x2930285, - 0x2ac02a0, 0x2c402b8, 0x2d902cf, 0x2ee02e4, - 0x30102f8, 0x314030b, 0x325031c, 0x335032d, - 0x345033d, 0x354034d, 0x362035b, 0x3700369, - 0x37d0377, 0x38a0384, 0x3960390, 0x3a2039c, - 0x3ad03a7, 0x3b803b3, 0x3c303bd, 0x3cd03c8, - 0x3d703d2, 0x3e103dc, 0x3ea03e5, 0x3f303ef, - 0x3fc03f8, 0x4050400, 0x40d0409, 0x4150411, - 0x41d0419, 0x4250421, 0x42d0429, 0x4340431, - 0x43c0438, 0x443043f, 0x44a0446, 0x451044d, - 0x4570454, 0x45e045b, 0x4640461, 0x46b0468, - 0x471046e, 0x4770474, 0x47d047a, 0x4830480, - 0x4890486, 0x48e048b, 0x4940491, 0x4990497, - 0x49f049c, 0x4a404a1, 0x4a904a7, 0x4ae04ac, - 0x4b304b1, 0x4b804b6, 0x4bd04bb, 0x4c204c0, - 0x4c704c5, 0x4cc04c9, 0x4d004ce, 0x4d504d3, - 0x4d904d7, 0x4de04dc, 0x4e204e0, 0x4e704e4, - 0x4eb04e9, 0x4ef04ed, 0x4f304f1, 0x4f704f5, - 0x4fb04f9, 0x4ff04fd, 0x5030501, 0x5070505, - 0x50b0509, 0x50f050d, 0x5130511, 0x5160515, - 0x51a0518, 0x51e051c, 0x5210520, 0x5250523, - 0x5290527, 0x52c052a, 0x52f052e, 0x5330531, - 0x5360535, 0x53a0538, 0x53d053b, 0x540053f, - 0x5440542, 0x5470545, 0x54a0548, 0x54d054c, - 0x550054f, 0x5530552, 0x5560555, 0x5590558, - 0x55c055b, 0x55f055e, 0x5620561, 0x5650564, - 0x5680567, 0x56b056a, 0x56e056d, 0x571056f, - 0x5740572, 0x5760575, 0x5790578, 0x57c057b, - 0x57f057d, 0x5810580, 0x5840583, 0x5870585, - 0x5890588, 0x58c058b, 0x58f058d -}; - -static u32 brightness_part12[] = { 0x58f058d, 0x59d058f }; - -/* - * lookup table for display brightness 103.3 to 510 nits - * */ -static u32 brightness_part2[] = { - 0x59d058f, 0x5b805ab, 0x5d105c5, 0x5e805dd, - 0x5fe05f3, 0x6120608, 0x625061c, 0x637062e, - 0x6480640, 0x6580650, 0x6680660, 0x677066f, - 0x685067e, 0x693068c, 0x6a00699, 0x6ac06a6, - 0x6b806b2, 0x6c406be, 0x6cf06ca, 0x6da06d5, - 0x6e506df, 0x6ef06ea, 0x6f906f4, 0x70206fe, - 0x70c0707, 0x7150710, 0x71e0719, 0x7260722, - 0x72f072a, 0x7370733, 0x73f073b, 0x7470743, - 0x74e074a, 0x7560752, 0x75d0759, 0x7640760, - 0x76b0768, 0x772076e, 0x7780775, 0x77f077c, - 0x7850782, 0x78c0789, 0x792078f, 0x7980795, - 0x79e079b, 0x7a407a1, 0x7aa07a7, 0x7af07ac, - 0x7b507b2, 0x7ba07b8, 0x7c007bd, 0x7c507c2, - 0x7ca07c8, 0x7cf07cd, 0x7d407d2, 0x7d907d7, - 0x7de07dc, 0x7e307e1, 0x7e807e5, 0x7ec07ea, - 0x7f107ef, 0x7f607f3, 0x7fa07f8, 0x7fe07fc -}; - - -bool neo_dcp_backlight_active(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - READ_ONCE(neo_dcp->backlight.state.ready); -} - -int neo_dcp_backlight_takeover(struct neo_apple_dcp *neo_dcp, u32 millinits) -{ - u32 maximum, nits; - int ret; - - if (neo_dcp_backlight_active(neo_dcp)) - return 0; - /* H17P takes nits up to the panel ceiling, not the older DAC table's. */ - maximum = neo_dcp->brightness.maximum; - ret = neo_dcp_bl_takeover_nits(maximum, millinits, &nits); - if (ret) - return ret; - - /* Keep the first bounded hint. Later reports are not user requests. */ - return neo_iomfb_configure_backlight_h17p(neo_dcp, maximum, true, nits, false, 0); -} - -static int neo_dcp_get_brightness(struct backlight_device *bd) -{ - struct neo_apple_dcp *neo_dcp = bl_get_data(bd); - unsigned long flags; - u32 actual; - - if (READ_ONCE(neo_dcp->quiescing)) - return -ENODEV; - - if (neo_dcp_backlight_active(neo_dcp)) { - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - actual = neo_dcp->backlight.state.actual; - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return actual; - } - - return neo_dcp->brightness.nits; -} - -int neo_dcp_backlight_configure(struct neo_apple_dcp *neo_dcp, u32 maximum, - bool inherited_valid, u32 inherited, - bool default_valid, u32 default_nits, - void (*kick)(struct neo_apple_dcp *neo_dcp)) -{ - unsigned long flags; - int ret; - - if (neo_dcp->fw_compat != DCP_FIRMWARE_H17P || !neo_dcp_has_panel(neo_dcp) || - !neo_dcp->crtc || !kick) - return -EINVAL; - - mutex_lock(&neo_dcp->bl_register_mutex); - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - if (neo_dcp->backlight.state.ready || neo_dcp->brightness.bl_dev) { - ret = -EBUSY; - } else { - ret = neo_dcp_bl_init(&neo_dcp->backlight.state, maximum, - inherited_valid, inherited, - default_valid, default_nits); - if (!ret) - neo_dcp->backlight.kick = kick; - } - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - mutex_unlock(&neo_dcp->bl_register_mutex); - if (!ret) - schedule_work(&neo_dcp->bl_register_wq); - return ret; -} - -bool neo_dcp_backlight_seed(struct neo_apple_dcp *neo_dcp, u32 nits) -{ - unsigned long flags; - bool seeded; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - seeded = neo_dcp_bl_seed(&neo_dcp->backlight.state, nits); - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return seeded; -} - -static void neo_dcp_backlight_kick(struct neo_apple_dcp *neo_dcp) -{ - void (*kick)(struct neo_apple_dcp *neo_dcp) = READ_ONCE(neo_dcp->backlight.kick); - - if (kick) - kick(neo_dcp); -} - -int neo_dcp_backlight_dpms(struct neo_apple_dcp *neo_dcp, bool on) -{ - unsigned long flags; - int ret; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - ret = neo_dcp_bl_dpms(&neo_dcp->backlight.state, on); - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - if (!ret) - neo_dcp_backlight_kick(neo_dcp); - return ret; -} - -int neo_dcp_backlight_prepare(struct neo_apple_dcp *neo_dcp, bool have_surface, - struct neo_dcp_backlight_present *present) -{ - unsigned long flags; - int ret; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - ret = neo_dcp_bl_prepare(&neo_dcp->backlight.state, have_surface, present); - /* No reservation: carry the last completed level on an ordinary present. */ - if (ret == -EALREADY) - present->nits = neo_dcp->backlight.state.actual; - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return ret; -} - -bool neo_dcp_backlight_complete(struct neo_apple_dcp *neo_dcp, u64 sequence, bool accepted) -{ - unsigned long flags; - bool completed; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - completed = neo_dcp_bl_complete(&neo_dcp->backlight.state, sequence, accepted); - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - /* Queue retries after the central queue has finished this transaction. */ - return completed; -} - -bool neo_dcp_backlight_pending(struct neo_apple_dcp *neo_dcp) -{ - unsigned long flags; - bool pending; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - pending = neo_dcp->backlight.state.dirty && !neo_dcp->backlight.state.in_flight; - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return pending; -} - -/* - * Called from the receive path, which must not queue a transaction itself; - * the caller schedules the update worker when this returns true. Requests - * are honoured at most once per second, so a firmware that repeats the - * request after every brightness present cannot keep the queue busy. - */ -bool neo_dcp_backlight_resend(struct neo_apple_dcp *neo_dcp) -{ - unsigned long flags; - bool resend = false; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - if (!neo_dcp->backlight.resent || - time_after(jiffies, neo_dcp->backlight.resent_at + HZ)) { - resend = neo_dcp_bl_resend(&neo_dcp->backlight.state); - if (resend) { - neo_dcp->backlight.resent = true; - neo_dcp->backlight.resent_at = jiffies; - } - } - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return resend; -} - -unsigned int neo_dcp_backlight_retry_delay(struct neo_apple_dcp *neo_dcp) -{ - unsigned long flags; - unsigned int delay; - - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - delay = neo_dcp_bl_retry_delay(&neo_dcp->backlight.state); - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return delay; -} - -#define SCALE_FACTOR (1 << 10) - -static u32 interpolate(int val, int min, int max, u32 *tbl, size_t tbl_size) -{ - u32 frac; - u64 low, high; - u32 interpolated = (tbl_size - 1) * ((val - min) * SCALE_FACTOR) / (max - min); - - size_t index = interpolated / SCALE_FACTOR; - - if (WARN(index + 1 >= tbl_size, "invalid index %zu for brightness %u\n", index, val)) - return tbl[tbl_size / 2]; - - frac = interpolated & (SCALE_FACTOR - 1); - low = tbl[index]; - high = tbl[index + 1]; - - return ((frac * high) + ((SCALE_FACTOR - frac) * low)) / SCALE_FACTOR; -} - -static u32 calculate_dac(struct neo_apple_dcp *neo_dcp, int val) -{ - u32 dac; - - if (val <= MIN_BRIGHTNESS_PART1) - return 16 * brightness_part1[0]; - else if (val == MAX_BRIGHTNESS_PART1) - return 16 * brightness_part1[ARRAY_SIZE(brightness_part1) - 1]; - else if (val == MIN_BRIGHTNESS_PART2) - return 16 * brightness_part2[0]; - else if (val >= MAX_BRIGHTNESS_PART2) - return brightness_part2[ARRAY_SIZE(brightness_part2) - 1]; - - if (val < MAX_BRIGHTNESS_PART1) { - dac = interpolate(val, MIN_BRIGHTNESS_PART1, MAX_BRIGHTNESS_PART1, - brightness_part1, ARRAY_SIZE(brightness_part1)); - } else if (val > MIN_BRIGHTNESS_PART2) { - dac = interpolate(val, MIN_BRIGHTNESS_PART2, MAX_BRIGHTNESS_PART2, - brightness_part2, ARRAY_SIZE(brightness_part2)); - } else { - dac = interpolate(val, MAX_BRIGHTNESS_PART1, MIN_BRIGHTNESS_PART2, - brightness_part12, ARRAY_SIZE(brightness_part12)); - } - - return 16 * dac; -} - -static int drm_crtc_set_brightness(struct neo_apple_dcp *neo_dcp) -{ - struct drm_atomic_state *state; - struct drm_crtc_state *crtc_state; - struct drm_modeset_acquire_ctx ctx; - struct drm_crtc *crtc = &neo_dcp->crtc->base; - int ret = 0; - - state = drm_atomic_state_alloc(crtc->dev); - if (!state) - return -ENOMEM; - - drm_modeset_acquire_init(&ctx, DRM_MODESET_ACQUIRE_INTERRUPTIBLE); - state->acquire_ctx = &ctx; - -retry: - crtc_state = drm_atomic_get_crtc_state(state, crtc); - if (IS_ERR(crtc_state)) { - ret = PTR_ERR(crtc_state); - goto out; - } - - if (!neo_dcp->brightness.update) - goto out; - - /* Re-present the current scanout when brightness is the only change. */ - ret = drm_atomic_add_affected_planes(state, crtc); - if (ret) - goto out; - - crtc_state->color_mgmt_changed |= true; - - ret = drm_atomic_commit(state); - -out: - if (ret == -EDEADLK) { - drm_atomic_state_clear(state); - ret = drm_modeset_backoff(&ctx); - if (!ret) - goto retry; - } - - drm_atomic_state_put(state); - drm_modeset_drop_locks(&ctx); - drm_modeset_acquire_fini(&ctx); - - return ret; -} - -int neo_dcp_backlight_update(struct neo_apple_dcp *neo_dcp) -{ - if (READ_ONCE(neo_dcp->quiescing)) - return -ENODEV; - if (neo_dcp_backlight_active(neo_dcp)) { - neo_dcp_backlight_kick(neo_dcp); - return 0; - } - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - return -ENODATA; - - /* - * Do not actively try to change brightness if no mode is set. - * TODO: should this be reflected the in backlight's power property? - * defer this hopefully until it becomes irrelevant due to proper - * drm integrated backlight handling - */ - if (!READ_ONCE(neo_dcp->mode_state.valid)) - return 0; - - /* Wait 1 vblank cycle in the hope an atomic swap has already updated - * the brightness */ - msleep((1001 + 23) / 24); // 42ms for 23.976 fps - - return drm_crtc_set_brightness(neo_dcp); -} - -static int neo_dcp_set_brightness(struct backlight_device *bd) -{ - int ret = 0; - struct neo_apple_dcp *neo_dcp = bl_get_data(bd); - int brightness = backlight_get_brightness(bd); - unsigned long flags; - - if (READ_ONCE(neo_dcp->quiescing)) - return -ENODEV; - - if (neo_dcp_backlight_active(neo_dcp)) { - /* Preserve the requested level even while the core forces zero. */ - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - ret = neo_dcp_bl_request(&neo_dcp->backlight.state, - bd->props.brightness, - backlight_is_blank(bd), - bd->props.state & BL_CORE_SUSPENDED); - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - if (!ret) - neo_dcp_backlight_kick(neo_dcp); - return ret; - } - - ret = drm_modeset_lock_single_interruptible(&neo_dcp->crtc->base.mutex); - if (ret) - return ret; - - neo_dcp->brightness.dac = calculate_dac(neo_dcp, brightness); - neo_dcp->brightness.update = true; - - drm_modeset_unlock(&neo_dcp->crtc->base.mutex); - - return neo_dcp_backlight_update(neo_dcp); -} - -static bool neo_dcp_backlight_controls_device(struct backlight_device *bd, - struct device *display_dev) -{ - struct neo_apple_dcp *neo_dcp = bl_get_data(bd); - - return !display_dev || display_dev == neo_dcp->dev || - (neo_dcp->crtc && display_dev == neo_dcp->crtc->base.dev->dev); -} - -static const struct backlight_ops neo_dcp_backlight_ops = { - .options = BL_CORE_SUSPENDRESUME, - .get_brightness = neo_dcp_get_brightness, - .update_status = neo_dcp_set_brightness, - .controls_device = neo_dcp_backlight_controls_device, -}; - -int neo_dcp_backlight_register(struct neo_apple_dcp *neo_dcp) -{ - struct device *dev = neo_dcp->dev; - struct backlight_device *bl_dev; - unsigned long flags; - struct backlight_properties props = { - .type = BACKLIGHT_PLATFORM, - .brightness = neo_dcp->brightness.nits, - .scale = BACKLIGHT_SCALE_LINEAR, - }; - props.max_brightness = min(neo_dcp->brightness.maximum, MAX_BRIGHTNESS_PART2 - 1); - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !neo_dcp_backlight_active(neo_dcp)) - return -ENODATA; - if (neo_dcp_backlight_active(neo_dcp)) { - spin_lock_irqsave(&neo_dcp->backlight.lock, flags); - if (!neo_dcp->backlight.state.ready) { - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - return -ENODATA; - } - /* Freeze takeover before userspace can discover the device. */ - neo_dcp->backlight.state.controlled = true; - props.brightness = neo_dcp->backlight.state.target; - props.max_brightness = neo_dcp->backlight.state.maximum; - spin_unlock_irqrestore(&neo_dcp->backlight.lock, flags); - } - - bl_dev = devm_backlight_device_register(dev, "apple-panel-bl", dev, neo_dcp, - &neo_dcp_backlight_ops, &props); - if (IS_ERR(bl_dev)) - return PTR_ERR(bl_dev); - - neo_dcp->brightness.bl_dev = bl_dev; - if (!neo_dcp_backlight_active(neo_dcp)) - neo_dcp->brightness.dac = calculate_dac(neo_dcp, neo_dcp->brightness.nits); - - return 0; -} diff --git a/drivers/gpu/drm/apple-neo/dcp_backlight_test.c b/drivers/gpu/drm/apple-neo/dcp_backlight_test.c deleted file mode 100644 index 0c2123db25287c..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp_backlight_test.c +++ /dev/null @@ -1,379 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2026 Ryan Murray */ - -#include -#include - -#include "dcp_backlight.h" - -static void backlight_millinits_takeover_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - u32 nits = 123; - - KUNIT_EXPECT_EQ(test, neo_dcp_bl_takeover_nits(0, 100000, &nits), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_takeover_nits(U32_MAX, 100000, &nits), -EINVAL); - KUNIT_EXPECT_EQ(test, nits, 123U); - /* Reports above the panel ceiling clamp to it. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(525, 525999, &nits), 0); - KUNIT_EXPECT_EQ(test, nits, 525U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(509, 509001, &nits), 0); - KUNIT_EXPECT_EQ(test, nits, 509U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(509, 1018999, &nits), 0); - KUNIT_EXPECT_EQ(test, nits, 509U); - /* More than twice the ceiling is not a panel level. */ - nits = 123; - KUNIT_EXPECT_EQ(test, neo_dcp_bl_takeover_nits(509, 1019000, &nits), -ERANGE); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_takeover_nits(509, U32_MAX, &nits), -ERANGE); - KUNIT_EXPECT_EQ(test, nits, 123U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(509, 100999, &nits), 0); - KUNIT_EXPECT_EQ(test, nits, 100U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 509, true, nits, false, 0), 0); - KUNIT_EXPECT_EQ(test, state.actual, 100U); - KUNIT_EXPECT_FALSE(test, state.dirty); - /* Registration freezes the hint before exposing the controls. */ - state.controlled = true; - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 400)); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 0)); - KUNIT_EXPECT_EQ(test, state.target, 100U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(509, 0, &nits), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 509, true, nits, false, 0), 0); - KUNIT_EXPECT_EQ(test, state.target, 0U); - KUNIT_EXPECT_FALSE(test, state.dirty); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_takeover_nits(509, 509000, &nits), 0); - KUNIT_EXPECT_EQ(test, nits, 509U); -} - -static void backlight_takeover_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state = {}; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, true, 36), 0); - KUNIT_EXPECT_EQ(test, state.target, 140U); - KUNIT_EXPECT_EQ(test, state.actual, 140U); - KUNIT_EXPECT_FALSE(test, state.dirty); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_seed(&state, 400)); - KUNIT_EXPECT_EQ(test, state.target, 400U); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 601)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 400, false, false), 0); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 0)); - KUNIT_EXPECT_EQ(test, state.target, 400U); - KUNIT_EXPECT_EQ(test, state.actual, 400U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 0, true, 36), 0); - KUNIT_EXPECT_EQ(test, state.target, 0U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, false, 0, true, 36), 0); - KUNIT_EXPECT_EQ(test, state.target, 36U); -} - -static void backlight_range_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state = {}; - - KUNIT_EXPECT_EQ(test, neo_dcp_bl_init(&state, 600, false, 0, false, 0), -ENODATA); - KUNIT_EXPECT_FALSE(test, state.ready); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_request(&state, 140, false, false), -ENODATA); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_dpms(&state, false), -ENODATA); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_init(&state, 0, true, 0, false, 0), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_init(&state, U32_MAX, true, 0, false, 0), -EINVAL); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_init(&state, 600, true, 601, false, 0), -ERANGE); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, false, 0), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_request(&state, 601, false, false), -ERANGE); - KUNIT_EXPECT_EQ(test, state.target, 140U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 600, false, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 600U); -} - -static void backlight_dpms_cycles_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - int i; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 400, false, 0), 0); - for (i = 0; i < 20; i++) { - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_EQ(test, state.actual, 0U); - KUNIT_EXPECT_EQ(test, state.target, 400U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 400U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_EQ(test, state.actual, 400U); - } - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 0U); - KUNIT_EXPECT_EQ(test, state.target, 400U); -} - -static void backlight_blanked_target_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 400, false, 0), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 140, false, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_EQ(test, state.target, 140U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 140U); - KUNIT_EXPECT_EQ(test, state.actual, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_EQ(test, state.actual, 140U); -} - -static void backlight_suspend_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - int off_first; - - for (off_first = 0; off_first < 2; off_first++) { - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, false, 0), 0); - if (off_first) - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 140, true, true), 0); - if (!off_first) - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 36, true, true), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 0U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 0U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 36, false, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 0U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 36U); - KUNIT_EXPECT_EQ(test, state.target, 36U); - } - /* A core blank also keeps the panel dark independently of DRM DPMS. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 400, true, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 0U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 400, false, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_effective(&state), 400U); -} - -static void backlight_surface_and_busy_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present first, second; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, false, 0), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_prepare(&state, true, &first), -EALREADY); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 36, false, false), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_prepare(&state, false, &first), -ENODATA); - KUNIT_EXPECT_TRUE(test, state.dirty); - KUNIT_EXPECT_FALSE(test, state.in_flight); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &first), 0); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_prepare(&state, true, &second), -EBUSY); - KUNIT_EXPECT_EQ(test, state.actual, 140U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 400, false, false), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, first.sequence, true)); - KUNIT_EXPECT_EQ(test, state.actual, 36U); - KUNIT_EXPECT_EQ(test, state.target, 400U); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &second), 0); - KUNIT_EXPECT_EQ(test, second.nits, 400U); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_complete(&state, first.sequence, true)); - KUNIT_EXPECT_TRUE(test, state.in_flight); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, second.sequence, true)); - KUNIT_EXPECT_EQ(test, state.actual, 400U); -} - -static void backlight_reject_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, false, 0), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 400, false, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, false)); - KUNIT_EXPECT_EQ(test, state.actual, 140U); - KUNIT_EXPECT_TRUE(test, state.dirty); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_EQ(test, state.target, 400U); - KUNIT_EXPECT_EQ(test, state.actual, 0U); -} - -static void backlight_inflight_dpms_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 600, true, 140, false, 0), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_TRUE(test, state.dirty); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 140U); -} - -static void backlight_resend_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - /* Nothing is re-sent before Linux controls the level. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, true, 140, false, 0), 0); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_resend(&state)); - KUNIT_EXPECT_FALSE(test, state.dirty); - - state.controlled = true; - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_resend(&state)); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_resend(&state)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 140U); - /* The present in flight already carries the level. */ - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_resend(&state)); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_FALSE(test, state.dirty); - - /* A blanked panel is re-sent dark, keeping the restore target. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_resend(&state)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_EQ(test, state.target, 140U); -} - -static void backlight_fallback_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - /* A registration default is reported, never presented on its own. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - KUNIT_EXPECT_FALSE(test, state.level_known); - state.controlled = true; - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_resend(&state)); - KUNIT_EXPECT_FALSE(test, state.dirty); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), -EALREADY); - - /* A rejected userspace level does not become known. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 300, false, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 300U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, false)); - KUNIT_EXPECT_FALSE(test, state.level_known); - - /* Once a present carrying it completes, the level is re-sendable. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_TRUE(test, state.level_known); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_resend(&state)); - - /* DPMS has to light the panel at some level: wake uses the default. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - state.controlled = true; - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 0U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_dpms(&state, true), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 262U); -} - -static void backlight_fallback_same_level_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - /* A userspace write of the reported default is still presented. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 262, false, false), 0); - KUNIT_EXPECT_TRUE(test, state.dirty); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 262U); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, true)); - KUNIT_EXPECT_TRUE(test, state.level_known); - - /* Once the level is known, repeating it presents nothing. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 262, false, false), 0); - KUNIT_EXPECT_FALSE(test, state.dirty); - KUNIT_EXPECT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), -EALREADY); - - /* A rejected present leaves it unknown, so the same write is retried. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 262, false, false), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_complete(&state, present.sequence, false)); - KUNIT_EXPECT_FALSE(test, state.level_known); - state.dirty = false; - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 262, false, false), 0); - KUNIT_EXPECT_TRUE(test, state.dirty); - - /* An inherited level is known from the start. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, true, 180, false, 0), 0); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_request(&state, 180, false, false), 0); - KUNIT_EXPECT_FALSE(test, state.dirty); -} - -static void backlight_seed_known_test(struct kunit *test) -{ - struct neo_dcp_backlight_state state; - struct neo_dcp_backlight_present present; - - /* A loader sample seeded over a registration default is known. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - KUNIT_EXPECT_FALSE(test, state.level_known); - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_seed(&state, 180)); - KUNIT_EXPECT_TRUE(test, state.level_known); - state.controlled = true; - KUNIT_EXPECT_TRUE(test, neo_dcp_bl_resend(&state)); - KUNIT_ASSERT_EQ(test, neo_dcp_bl_prepare(&state, true, &present), 0); - KUNIT_EXPECT_EQ(test, present.nits, 180U); - - /* A refused sample leaves the default unknown. */ - KUNIT_ASSERT_EQ(test, neo_dcp_bl_init(&state, 525, false, 0, true, 262), 0); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 526)); - KUNIT_EXPECT_FALSE(test, state.level_known); - state.controlled = true; - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_seed(&state, 180)); - KUNIT_EXPECT_FALSE(test, state.level_known); - KUNIT_EXPECT_FALSE(test, neo_dcp_bl_resend(&state)); -} - -static struct kunit_case backlight_cases[] = { - KUNIT_CASE(backlight_millinits_takeover_test), - KUNIT_CASE(backlight_takeover_test), - KUNIT_CASE(backlight_range_test), - KUNIT_CASE(backlight_dpms_cycles_test), - KUNIT_CASE(backlight_blanked_target_test), - KUNIT_CASE(backlight_suspend_test), - KUNIT_CASE(backlight_surface_and_busy_test), - KUNIT_CASE(backlight_reject_test), - KUNIT_CASE(backlight_inflight_dpms_test), - KUNIT_CASE(backlight_resend_test), - KUNIT_CASE(backlight_fallback_test), - KUNIT_CASE(backlight_fallback_same_level_test), - KUNIT_CASE(backlight_seed_known_test), - {} -}; - -static struct kunit_suite backlight_suite = { - .name = "apple-dcp-backlight", - .test_cases = backlight_cases, -}; - -kunit_test_suite(backlight_suite); - -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple-neo/dcp_trace.c b/drivers/gpu/drm/apple-neo/dcp_trace.c deleted file mode 100644 index d18e71af73a74d..00000000000000 --- a/drivers/gpu/drm/apple-neo/dcp_trace.c +++ /dev/null @@ -1,3 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#define CREATE_TRACE_POINTS -#include "dcp_trace.h" \ No newline at end of file diff --git a/drivers/gpu/drm/apple-neo/dptxep.c b/drivers/gpu/drm/apple-neo/dptxep.c deleted file mode 100644 index 486195d97fe082..00000000000000 --- a/drivers/gpu/drm/apple-neo/dptxep.c +++ /dev/null @@ -1,878 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2022 Sven Peter */ - -#include -#include -#include -#include -#include -#include -#include - -#include - -#include "afk.h" -#include "dcp.h" -#include "dptxep.h" -#include "parser.h" -#include "trace.h" - -struct dcpdptx_connection_cmd { - __le32 unk; - __le32 target; -} __attribute__((packed)); - -struct dcpdptx_hotplug_cmd { - u8 _pad0[16]; - __le32 unk; -} __attribute__((packed)); - -struct neo_dptxport_apcall_link_rate { - __le32 retcode; - u8 _unk0[12]; - __le32 link_rate; - u8 _unk1[12]; -} __attribute__((packed)); - -struct neo_dptxport_apcall_lane_count { - __le32 retcode; - u8 _unk0[12]; - __le64 lane_count; - u8 _unk1[8]; -} __attribute__((packed)); - -struct neo_dptxport_apcall_set_active_lane_count { - __le32 retcode; - u8 _unk0[12]; - __le64 lane_count; - u8 _unk1[8]; -} __packed; - -struct neo_dptxport_apcall_get_support { - __le32 retcode; - u8 _unk0[12]; - __le32 supported; - u8 _unk1[12]; -} __attribute__((packed)); - -struct neo_dptxport_apcall_max_drive_settings { - __le32 retcode; - u8 _unk0[12]; - __le32 max_drive_settings[2]; - u8 _unk1[8]; -}; - -struct neo_dptxport_apcall_drive_settings { - __le32 retcode; - u8 _unk0[12]; - __le32 unk1; - __le32 unk2; - __le32 unk3; - __le32 unk4; - __le32 unk5; - __le32 unk6; - __le32 unk7; -}; - -struct neo_dptxport_apcall_set_tiled { - __le32 retcode; -}; - -/* - * Ported from aurora-silicon/linux#8: a Thunderbolt DP tunnel uses the same - * plain CORE|ATC|DIE|CONNECTED target as a direct alt-mode PHY. CORE is the - * DFP port (0 = dpphy, 1/2 = dpin0/dpin1, dcp->dptx_dfp_port), ATC is the - * route's own ATC index -- no separate DPIN field. - */ -static u32 neo_dptxport_remote_target(struct neo_apple_dcp *neo_dcp, u8 core, u8 atc, - u8 die) -{ - return FIELD_PREP(DCPDPTX_REMOTE_PORT_CORE, core) | - FIELD_PREP(DCPDPTX_REMOTE_PORT_ATC, atc) | - FIELD_PREP(DCPDPTX_REMOTE_PORT_DIE, die) | - DCPDPTX_REMOTE_PORT_CONNECTED; -} - -int neo_dptxport_validate_connection(struct neo_apple_epic_service *service, u8 core, - u8 atc, u8 die) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct dcpdptx_connection_cmd cmd, resp; - int ret; - u32 target = neo_dptxport_remote_target(service->ep->neo_dcp, core, atc, die); - - /* - * attributes: role (0 = direct PHY, 1 = Thunderbolt/USB4 DP IN) | - * supportsHPD << 8. A real, hardware-validated reference - * implementation of the equivalent Thunderbolt DP tunnel mechanism - * on a different SoC (aurora-silicon/linux#8) adds this exact role - * bit for its tunnel routes; our own analog-DPIN path has never set - * it, always sending a plain direct-PHY attributes value even - * though this is a genuinely USB4-tunneled connection. DCP firmware - * appears to treat a failed AUX probe as fatal for role=0 but keeps - * training regardless for role=1 (a Thunderbolt tunnel's AUX - * proxying may not respond as fast as a direct PHY's): repeated - * test runs with substantially different Activate/crossbar ordering - * all reached the identical INACTIVE_SINK_DETECTED/DPRX-timeout - * outcome, which points at a signal we are not sending at all - * rather than a timing issue fixable by reordering. - */ - u32 attrs = 0x100 | (neo_dcp_is_usb4_output(service->ep->neo_dcp) ? 1 : 0); - - trace_neo_dptxport_validate_connection(neo_dptx, core, atc, die); - neo_dptx->validate_calls++; - dev_dbg(service->ep->neo_dcp->dev, - "DPTX validate: call #%u this boot target=0x%x core=%u atc=%u die=%u attrs=0x%x caller=%pS\n", - neo_dptx->validate_calls, target, core, atc, die, - attrs, __builtin_return_address(0)); - - cmd.target = cpu_to_le32(target); - cmd.unk = cpu_to_le32(attrs); - ret = neo_afk_service_call(service, 0, 12, &cmd, sizeof(cmd), 40, &resp, - sizeof(resp), 40); - if (ret) - return ret; - - if (le32_to_cpu(resp.target) != target) { - dev_warn(service->ep->neo_dcp->dev, - "validate_connection: target reply 0x%x (sent 0x%x)\n", - le32_to_cpu(resp.target), target); - return -EINVAL; - } - if (le32_to_cpu(resp.unk) != attrs) { - /* only the Thunderbolt DP IN role is allowed to differ */ - if (!neo_dcp_is_usb4_output(service->ep->neo_dcp)) { - dev_warn(service->ep->neo_dcp->dev, - "validate_connection: attrs reply 0x%x (sent 0x%x), rejecting\n", - le32_to_cpu(resp.unk), attrs); - return -EINVAL; - } - dev_info(service->ep->neo_dcp->dev, - "validate_connection: attrs reply 0x%x (sent 0x%x)\n", - le32_to_cpu(resp.unk), attrs); - } - - return 0; -} - -int neo_dptxport_connect(struct neo_apple_epic_service *service, u8 core, u8 atc, - u8 die, bool supports_hpd) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct dcpdptx_connection_cmd cmd, resp; - /* same role bit as dptxport_validate_connection() above */ - u32 unk_field = (supports_hpd ? DCPDPTX_REMOTE_PORT_SUPPORTS_HPD : 0) | - (neo_dcp_is_usb4_output(service->ep->neo_dcp) ? 1 : 0); - int ret; - u32 target = neo_dptxport_remote_target(service->ep->neo_dcp, core, atc, die); - - trace_neo_dptxport_connect(neo_dptx, core, atc, die); - neo_dptx->connect_calls++; - dev_dbg(service->ep->neo_dcp->dev, - "DPTX connect: call #%u this boot target=0x%x unk=0x%x caller=%pS\n", - neo_dptx->connect_calls, target, unk_field, - __builtin_return_address(0)); - - cmd.target = cpu_to_le32(target); - cmd.unk = cpu_to_le32(unk_field); - ret = neo_afk_service_call(service, 0, 11, &cmd, sizeof(cmd), 24, &resp, - sizeof(resp), 24); - if (ret) - return ret; - - if (le32_to_cpu(resp.target) != target) { - dev_warn(service->ep->neo_dcp->dev, - "connect: target reply 0x%x (sent 0x%x)\n", - le32_to_cpu(resp.target), target); - return -EINVAL; - } - if (le32_to_cpu(resp.unk) != unk_field) - dev_notice(service->ep->neo_dcp->dev, "unexpected unk field in reply: 0x%x (0x%x)\n", - le32_to_cpu(resp.unk), unk_field); - - return 0; -} - -int neo_dptxport_request_display(struct neo_apple_epic_service *service) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - int ret; - - neo_dptx->request_calls++; - dev_dbg(service->ep->neo_dcp->dev, - "DPTX request_display: call #%u this boot caller=%pS\n", - neo_dptx->request_calls, __builtin_return_address(0)); - ret = neo_afk_service_call(service, 0, 6, NULL, 0, 16, NULL, 0, 16); - dev_dbg(service->ep->neo_dcp->dev, - "DPTX request_display: call #%u result=%d\n", - neo_dptx->request_calls, ret); - return ret; -} - -int neo_dptxport_release_display(struct neo_apple_epic_service *service) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - int ret; - - neo_dptx->release_calls++; - dev_dbg(service->ep->neo_dcp->dev, - "DPTX release_display: call #%u this boot caller=%pS\n", - neo_dptx->release_calls, __builtin_return_address(0)); - ret = neo_afk_service_call(service, 0, 7, NULL, 0, 16, NULL, 0, 16); - dev_dbg(service->ep->neo_dcp->dev, - "DPTX release_display: call #%u result=%d\n", - neo_dptx->release_calls, ret); - return ret; -} - -int neo_dptxport_set_hpd_timeout(struct neo_apple_epic_service *service, bool hpd, - unsigned int timeout_ms) -{ - struct dcpdptx_hotplug_cmd cmd, resp; - int ret; - - memset(&cmd, 0, sizeof(cmd)); - - if (hpd) - cmd.unk = cpu_to_le32(1); - - ret = neo_afk_service_call_timeout(service, 8, 8, &cmd, sizeof(cmd), 12, - &resp, sizeof(resp), 12, timeout_ms); - if (ret) - return ret; - if (le32_to_cpu(resp.unk) != hpd) { - dev_warn(service->ep->neo_dcp->dev, - "set_hpd: unk reply 0x%x (sent hpd=%d)\n", - le32_to_cpu(resp.unk), hpd); - return -EINVAL; - } - return 0; -} - -int neo_dptxport_set_hpd(struct neo_apple_epic_service *service, bool hpd) -{ - return neo_dptxport_set_hpd_timeout(service, hpd, MSEC_PER_SEC); -} - -static int -neo_dptxport_call_get_max_drive_settings(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_dptxport_apcall_max_drive_settings *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(0); - reply->max_drive_settings[0] = cpu_to_le32(0x3); - reply->max_drive_settings[1] = cpu_to_le32(0x3); - - return 0; -} - -static int -neo_dptxport_call_get_drive_settings(struct neo_apple_epic_service *service, - const void *request_, size_t request_size, - void *reply_, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - const struct neo_dptxport_apcall_drive_settings *request = request_; - struct neo_dptxport_apcall_drive_settings *reply = reply_; - - if (reply_size < sizeof(*reply) || request_size < sizeof(*request)) - return -EINVAL; - - *reply = *request; - - /* Clear the rest of the buffer */ - memset(reply_ + sizeof(*reply), 0, reply_size - sizeof(*reply)); - - /* - * retcode appears to be lane count, seeing 2 for USB-C dp alt mode - * with lanes splitted for DP/USB3. - */ - if (le32_to_cpu(reply->retcode) != neo_dptx->lane_count) - dev_err(service->ep->neo_dcp->dev, - "get_drive_settings: unexpected retcode %d\n", - reply->retcode); - - reply->retcode = cpu_to_le32(neo_dptx->lane_count); - reply->unk5 = cpu_to_le32(neo_dptx->drive_settings[0]); - reply->unk6 = cpu_to_le32(0); - reply->unk7 = cpu_to_le32(neo_dptx->drive_settings[1]); - - return 0; -} - -static int -neo_dptxport_call_set_drive_settings(struct neo_apple_epic_service *service, - const void *request_, size_t request_size, - void *reply_, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - const struct neo_dptxport_apcall_drive_settings *request = request_; - struct neo_dptxport_apcall_drive_settings *reply = reply_; - - if (reply_size < sizeof(*reply) || request_size < sizeof(*request)) - return -EINVAL; - - *reply = *request; - reply->retcode = cpu_to_le32(0); - - dev_info(service->ep->neo_dcp->dev, "set_drive_settings: %d:%d:%d:%d:%d:%d:%d\n", - request->unk1, request->unk2, request->unk3, request->unk4, - request->unk5, request->unk6, request->unk7); - - neo_dptx->drive_settings[0] = le32_to_cpu(reply->unk5); - neo_dptx->drive_settings[1] = le32_to_cpu(reply->unk7); - - return 0; -} - -static int neo_dptxport_call_get_max_link_rate(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_dptxport_apcall_link_rate *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(0); - reply->link_rate = cpu_to_le32(LINK_RATE_HBR3); - - return 0; -} - -static int neo_dptxport_call_get_max_lane_count(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_dptxport_apcall_lane_count *reply = reply_; - struct neo_dptx_port *neo_dptx = service->cookie; - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - union phy_configure_opts phy_ops; - int ret; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - if (!neo_dptx->atcphy) { - /* USB4 DP IN: no ATC DP PHY to validate. */ - neo_dptx->lane_count = 4; - reply->retcode = cpu_to_le32(0); - reply->lane_count = cpu_to_le64(4); - dev_info(neo_dcp->dev, "get_max_lane_count: USB4 DP IN, 4 lanes\n"); - return 0; - } - - ret = phy_validate(neo_dptx->atcphy, PHY_MODE_DP, 0, &phy_ops); - if (ret < 0) { - dev_err(neo_dcp->dev, "phy_validate failed: %d\n", ret); - reply->retcode = cpu_to_le32(1); - reply->lane_count = cpu_to_le64(0); - } else { - if (phy_ops.dp.lanes < 2) { - // phy_validate might return 0 lanes if atc phy is not - // yet switched to DP mode - dev_dbg(neo_dcp->dev, "get_max_lane_count: phy lanes: %d\n", - phy_ops.dp.lanes); - // default to 4 lanes - neo_dptx->lane_count = 4; - } else { - neo_dptx->lane_count = phy_ops.dp.lanes; - } - reply->retcode = cpu_to_le32(0); - reply->lane_count = cpu_to_le64(neo_dptx->lane_count); - } - - return 0; -} - -static int neo_dptxport_call_set_active_lane_count(struct neo_apple_epic_service *service, - const void *data, size_t data_size, - void *reply_, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - const struct neo_dptxport_apcall_set_active_lane_count *request = data; - struct neo_dptxport_apcall_set_active_lane_count *reply = reply_; - int ret = 0; - int retcode = 0; - - if (reply_size < sizeof(*reply)) - return -1; - if (data_size < sizeof(*request)) - return -1; - - u64 lane_count = le64_to_cpu(request->lane_count); - - if (neo_dptx->lane_count < lane_count) - dev_err(neo_dcp->dev, "set_active_lane_count: unexpected lane " - "count:%llu phy: %d\n", lane_count, neo_dptx->lane_count); - - switch (lane_count) { - case 0 ... 2: - case 4: - neo_dptx->phy_ops.dp.lanes = lane_count; - /* USB4 DP IN has no ATC PHY; still accept the lane count. */ - neo_dptx->phy_ops.dp.set_lanes = - neo_dcp_is_usb4_output(neo_dcp) || (neo_dcp->neo_dptx_phy > 3); - break; - default: - dev_err(neo_dcp->dev, "set_active_lane_count: invalid lane count:%llu\n", lane_count); - retcode = 1; - lane_count = 0; - break; - } - - if (neo_dptx->phy_ops.dp.set_lanes) { - if (neo_dptx->atcphy && !neo_dcp_is_usb4_output(neo_dcp)) { - ret = phy_configure(neo_dptx->atcphy, &neo_dptx->phy_ops); - if (ret) - return ret; - } - neo_dptx->phy_ops.dp.set_lanes = 0; - neo_dptx->lane_count = lane_count; - } - - reply->retcode = cpu_to_le32(retcode); - reply->lane_count = cpu_to_le64(lane_count); - - if (lane_count > 0) { - dev_info(neo_dcp->dev, "USB4/DPTX: SET_ACTIVE_LANE_COUNT %llu\n", - lane_count); - /* - * dcp_usb4_drm_allowed() (usb4_force_dptx) used to gate this - * for the old manual-training sysfs knob (module_param_cb - * usb4_dptx_train), removed wholesale in 0dc9f50 when - * apple_dcp_tb_dp_tunnel() replaced it with automatic tunnel - * detection -- but usb4_force_dptx itself, and this gate, - * were left behind with no way left to ever set it true. - * dcp_dptx_connect()'s single wait_for_completion_timeout() - * on linkcfg_completion is now shared by both the alt-mode - * and USB4-tunnel paths (also part of 0dc9f50's unification), - * so gating it here meant every tunneled connect timed out - * after a fully successful DPRX/AUX handshake and lane - * negotiation -- firmware does not send FORCE_HOTPLUG_DETECT - * (the only other completion site) for a tunnel. The - * reference (aurora-silicon/linux#8) completes - * linkcfg_completion here unconditionally; usb4_lane_completion - * has no consumer anywhere in this tree, so this drops both the - * dead gate and the dead completion and matches the reference. - */ - complete(&neo_dptx->linkcfg_completion); - } - - return ret; -} - -static int neo_dptxport_call_get_link_rate(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct neo_dptxport_apcall_link_rate *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(0); - reply->link_rate = cpu_to_le32(neo_dptx->link_rate); - - return 0; -} - -static int -neo_dptxport_call_will_change_link_config(struct neo_apple_epic_service *service) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - - neo_dptx->phy_ops.dp.set_lanes = 0; - neo_dptx->phy_ops.dp.set_rate = 0; - neo_dptx->phy_ops.dp.set_voltages = 0; - - return 0; -} - -static int -neo_dptxport_call_did_change_link_config(struct neo_apple_epic_service *service) -{ - /* - * Ported from aurora-silicon/linux#8: bringing the tunnel crossbar - * connection up (dcp_tunnel_crossbar_up()) now happens in - * dptxport_call(), after this succeeds, gated on - * dptx_tunnel && link_rate -- not inside this handler, and its - * result is never propagated back to DCP as a call failure. - */ - - /* assume the link config did change and wait a little bit */ - mdelay(10); - - return 0; -} - -static int neo_dptxport_call_set_link_rate(struct neo_apple_epic_service *service, - const void *data, size_t data_size, - void *reply_, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - const struct neo_dptxport_apcall_link_rate *request = data; - struct neo_dptxport_apcall_link_rate *reply = reply_; - u32 link_rate, phy_link_rate; - bool phy_set_rate = false; - int ret; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - if (data_size < sizeof(*request)) - return -EINVAL; - - link_rate = le32_to_cpu(request->link_rate); - trace_neo_dptxport_call_set_link_rate(neo_dptx, link_rate); - dev_info(service->ep->neo_dcp->dev, "DPTXPort: SET_LINK_RATE 0x%x\n", - link_rate); - - switch (link_rate) { - case LINK_RATE_RBR: - phy_link_rate = 1620; - phy_set_rate = true; - break; - case LINK_RATE_HBR: - phy_link_rate = 2700; - phy_set_rate = true; - break; - case LINK_RATE_HBR2: - phy_link_rate = 5400; - phy_set_rate = true; - break; - case LINK_RATE_HBR3: - phy_link_rate = 8100; - phy_set_rate = true; - break; - case 0: - phy_link_rate = 0; - phy_set_rate = true; - break; - default: - dev_err(service->ep->neo_dcp->dev, - "DPTXPort: Unsupported link rate 0x%x requested\n", - link_rate); - link_rate = 0; - phy_set_rate = false; - break; - } - - if (phy_set_rate) { - /* - * Ported from aurora-silicon/linux#8: a Thunderbolt DP tunnel - * starts/stops its own pixel clock instead of configuring the - * PHY directly, and never fails the apcall over it -- a - * failed clock just keeps the crossbar connection down - * (dcp->tb_clock_ok), which dcp_tunnel_crossbar_up() checks. - */ - if (neo_dptx->atcphy && service->ep->neo_dcp->neo_dptx_tunnel) { - neo_dcp_tunnel_set_rate(service->ep->neo_dcp, neo_dptx->atcphy, - link_rate); - } else if (neo_dptx->atcphy) { - neo_dptx->phy_ops.dp.link_rate = phy_link_rate; - neo_dptx->phy_ops.dp.set_rate = 1; - ret = phy_configure(neo_dptx->atcphy, &neo_dptx->phy_ops); - if (ret) - return ret; - } - - neo_dptx->link_rate = neo_dptx->pending_link_rate = link_rate; - } - - //dptx->pending_link_rate = link_rate; - reply->retcode = cpu_to_le32(0); - reply->link_rate = cpu_to_le32(link_rate); - - return 0; -} - -static int neo_dptxport_call_get_supports_hpd(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - struct neo_dptxport_apcall_get_support *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(0); - /* - * Analog DPIN CORE=1 already uses AFK set_hpd (returns 0). - * Denying HPD here made request_display ACTIVATE then 22/24 - * DEVICE_NOT_STARTED (~5.5s). Advertise HPD so firmware uses - * that path instead of waiting for a PHY start. - */ - reply->supported = cpu_to_le32(neo_dcp_is_typec_output(neo_dcp) ? 1 : 0); - dev_info(neo_dcp->dev, "DPTXPort: GET_SUPPORTS_HPD %u usb4=%d\n", - le32_to_cpu(reply->supported), neo_dcp_is_usb4_output(neo_dcp)); - return 0; -} - -static int -neo_dptxport_call_get_supports_downspread(struct neo_apple_epic_service *service, - void *reply_, size_t reply_size) -{ - struct neo_dptxport_apcall_get_support *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(0); - reply->supported = cpu_to_le32(0); - return 0; -} - -static int neo_dptxport_call_set_tiled_display_hint(void *reply_, - size_t reply_size) -{ - struct neo_dptxport_apcall_set_tiled *reply = reply_; - - if (reply_size < sizeof(*reply)) - return -EINVAL; - - reply->retcode = cpu_to_le32(1); - return 0; -} - -static int -neo_dptxport_call_activate(struct neo_apple_epic_service *service, - const void *data, size_t data_size, - void *reply, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - - /* - * Ported from aurora-silicon/linux#8: no crossbar, and no PHY mode - * change for a tunnel (dptx->atcphy must stay in USB4/TBT mode the - * whole connection, see dcp_tunnel_set_rate()). The DP IN adapter is - * only woken (DPTX_INACTIVE=0) here, via dcp_tunnel_dpin_activate(); - * waking it earlier hangs the machine. Activate always replies - * success to DCP. - */ - if (neo_dptx->atcphy && !neo_dcp->phy_managed_by_typec) - phy_set_mode_ext(neo_dptx->atcphy, PHY_MODE_DP, neo_dcp->index); - if (neo_dcp->neo_dptx_tunnel) - neo_dcp_tunnel_dpin_activate(neo_dcp, true); - - memcpy(reply, data, min(reply_size, data_size)); - if (reply_size >= 4) - memset(reply, 0, 4); - - return 0; -} - -static int -neo_dptxport_call_deactivate(struct neo_apple_epic_service *service, - const void *data, size_t data_size, - void *reply, size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - - dev_info(neo_dcp->dev, "DPTXPort: DEACTIVATE\n"); - if (neo_dcp->neo_dptx_tunnel) - neo_dcp_tunnel_dpin_activate(neo_dcp, false); - if (neo_dptx->atcphy && !neo_dcp->phy_managed_by_typec) - phy_set_mode_ext(neo_dptx->atcphy, PHY_MODE_INVALID, 0); - - memcpy(reply, data, min(reply_size, data_size)); - if (reply_size >= 4) - memset(reply, 0, 4); - - return 0; -} - -static int neo_dptxport_call(struct neo_apple_epic_service *service, u32 idx, - const void *data, size_t data_size, void *reply, - size_t reply_size) -{ - struct neo_dptx_port *neo_dptx = service->cookie; - trace_neo_dptxport_apcall(neo_dptx, idx, data_size); - dev_dbg(service->ep->neo_dcp->dev, "DPTXPort: APCALL %u (%zu bytes)\n", - idx, data_size); - if (data_size) - print_hex_dump_debug("DPTXPort: apcall data: ", - DUMP_PREFIX_OFFSET, 16, 1, data, - min(data_size, (size_t)64), true); - - switch (idx) { - case DPTX_APCALL_WILL_CHANGE_LINKG_CONFIG: - /* - * Ported from aurora-silicon/linux#8: a re-link on an - * established tunnel takes the crossbar connection down - * first. - */ - if (service->ep->neo_dcp->neo_dptx_tunnel && neo_dptx->link_rate) - neo_dcp_tunnel_crossbar_down(service->ep->neo_dcp); - return neo_dptxport_call_will_change_link_config(service); - case DPTX_APCALL_DID_CHANGE_LINK_CONFIG: { - int ret = neo_dptxport_call_did_change_link_config(service); - - if (!ret && service->ep->neo_dcp->neo_dptx_tunnel && neo_dptx->link_rate) - neo_dcp_tunnel_crossbar_up(service->ep->neo_dcp); - return ret; - } - case DPTX_APCALL_GET_MAX_LINK_RATE: - return neo_dptxport_call_get_max_link_rate(service, reply, - reply_size); - case DPTX_APCALL_GET_LINK_RATE: - return neo_dptxport_call_get_link_rate(service, reply, reply_size); - case DPTX_APCALL_SET_LINK_RATE: - return neo_dptxport_call_set_link_rate(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_GET_MAX_LANE_COUNT: - return neo_dptxport_call_get_max_lane_count(service, reply, reply_size); - case DPTX_APCALL_GET_ACTIVE_LANE_COUNT: { - struct neo_dptxport_apcall_lane_count *lc = reply; - - if (reply_size < sizeof(*lc)) - return -EINVAL; - lc->retcode = cpu_to_le32(0); - lc->lane_count = cpu_to_le64(neo_dptx->lane_count); - return 0; - } - case DPTX_APCALL_SET_ACTIVE_LANE_COUNT: - return neo_dptxport_call_set_active_lane_count(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_GET_SUPPORTS_HPD: - return neo_dptxport_call_get_supports_hpd(service, reply, - reply_size); - case DPTX_APCALL_GET_SUPPORTS_DOWN_SPREAD: - return neo_dptxport_call_get_supports_downspread(service, reply, - reply_size); - case DPTX_APCALL_GET_MAX_DRIVE_SETTINGS: - return neo_dptxport_call_get_max_drive_settings(service, reply, - reply_size); - case DPTX_APCALL_DEVICE_NOT_RESPONDING: - case DPTX_APCALL_DEVICE_BUSY_TIMEOUT: - case DPTX_APCALL_DEVICE_NOT_STARTED: - dev_warn(service->ep->neo_dcp->dev, - "DPTXPort: firmware reports link fault %u on target %u:%u\n", - idx, service->ep->neo_dcp->neo_dptx_die, - service->ep->neo_dcp->neo_dptx_phy); - memcpy(reply, data, min(reply_size, data_size)); - return 0; - case DPTX_APCALL_SET_TILED_DISPLAY_HINTS: - memcpy(reply, data, min(reply_size, data_size)); - return neo_dptxport_call_set_tiled_display_hint(reply, reply_size); - case DPTX_APCALL_GET_DRIVE_SETTINGS: - return neo_dptxport_call_get_drive_settings(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_SET_DRIVE_SETTINGS: - return neo_dptxport_call_set_drive_settings(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_ACTIVATE: - return neo_dptxport_call_activate(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_DEACTIVATE: - return neo_dptxport_call_deactivate(service, data, data_size, - reply, reply_size); - case DPTX_APCALL_FORCE_HOTPLUG_DETECT: - dev_info(service->ep->neo_dcp->dev, - "DPTXPort: FORCE_HOTPLUG_DETECT\n"); - complete(&neo_dptx->linkcfg_completion); - memcpy(reply, data, min(reply_size, data_size)); - if (reply_size >= 4) - memset(reply, 0, 4); - return 0; - case DPTX_APCALL_INACTIVE_SINK_DETECTED: - /* - * Normal prelude to link training on USB4. Ack and wait for - * SET_ACTIVE_LANE_COUNT. Treating this as failure made - * firmware DEACTIVATE after it had already set lanes/rate. - */ - dev_info(service->ep->neo_dcp->dev, - "DPTXPort: INACTIVE_SINK_DETECTED (keep waiting for lanes)\n"); - memcpy(reply, data, min(reply_size, data_size)); - if (reply_size >= 4) - memset(reply, 0, 4); - return 0; - default: - /* just try to ACK and hope for the best... */ - dev_info(service->ep->neo_dcp->dev, "DPTXPort: acking unhandled call %u\n", - idx); - fallthrough; - case DPTX_APCALL_GET_DOWN_SPREAD: - case DPTX_APCALL_SET_DOWN_SPREAD: - memcpy(reply, data, min(reply_size, data_size)); - if (reply_size >= 4) - memset(reply, 0, 4); - return 0; - } -} - -static void neo_dptxport_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ - - if (strcmp(name, "dcpdptx-port-epic")) - return; - if (strcmp(class, "AppleDCPDPTXRemotePort")) - return; - - trace_neo_dptxport_init(service->ep->neo_dcp, unit); - - switch (unit) { - case 0: - case 1: - if (service->ep->neo_dcp->neo_dptxport[unit].enabled) { - dev_err(service->ep->neo_dcp->dev, - "DPTXPort: unit %lld already exists\n", unit); - return; - } - service->ep->neo_dcp->neo_dptxport[unit].unit = unit; - service->ep->neo_dcp->neo_dptxport[unit].service = service; - service->ep->neo_dcp->neo_dptxport[unit].enabled = true; - service->cookie = (void *)&service->ep->neo_dcp->neo_dptxport[unit]; - complete(&service->ep->neo_dcp->neo_dptxport[unit].enable_completion); - break; - default: - dev_err(service->ep->neo_dcp->dev, "DPTXPort: invalid unit %lld\n", - unit); - } -} - -static const struct neo_apple_epic_service_ops neo_dptxep_ops[] = { - { - .name = "AppleDCPDPTXRemotePort", - .init = neo_dptxport_init, - .call = neo_dptxport_call, - }, - {} -}; - -int neo_dptxep_init(struct neo_apple_dcp *neo_dcp) -{ - int ret; - u32 port; - unsigned long timeout = msecs_to_jiffies(1000); - - init_completion(&neo_dcp->neo_dptxport[0].enable_completion); - init_completion(&neo_dcp->neo_dptxport[1].enable_completion); - init_completion(&neo_dcp->neo_dptxport[0].linkcfg_completion); - init_completion(&neo_dcp->neo_dptxport[1].linkcfg_completion); - - neo_dcp->neo_dptxep = neo_afk_init(neo_dcp, DPTX_ENDPOINT, neo_dptxep_ops); - if (IS_ERR(neo_dcp->neo_dptxep)) - return PTR_ERR(neo_dcp->neo_dptxep); - - ret = neo_afk_start(neo_dcp->neo_dptxep); - if (ret) - return ret; - - for (port = 0; port < neo_dcp->hw.num_dptx_ports; port++) { - ret = wait_for_completion_timeout(&neo_dcp->neo_dptxport[port].enable_completion, - timeout); - if (!ret) - return -ETIMEDOUT; - else if (ret < 0) - return ret; - timeout = ret; - } - - return 0; -} diff --git a/drivers/gpu/drm/apple-neo/dptxep.h b/drivers/gpu/drm/apple-neo/dptxep.h deleted file mode 100644 index e39e73ef409361..00000000000000 --- a/drivers/gpu/drm/apple-neo/dptxep.h +++ /dev/null @@ -1,81 +0,0 @@ -#ifndef __APPLE_DCP_DPTXEP_H__ -#define __APPLE_DCP_DPTXEP_H__ - -#include -#include - -enum neo_dptx_apcall { - DPTX_APCALL_ACTIVATE = 0, - DPTX_APCALL_DEACTIVATE = 1, - DPTX_APCALL_GET_MAX_DRIVE_SETTINGS = 2, - DPTX_APCALL_SET_DRIVE_SETTINGS = 3, - DPTX_APCALL_GET_DRIVE_SETTINGS = 4, - DPTX_APCALL_WILL_CHANGE_LINKG_CONFIG = 5, - DPTX_APCALL_DID_CHANGE_LINK_CONFIG = 6, - DPTX_APCALL_GET_MAX_LINK_RATE = 7, - DPTX_APCALL_GET_LINK_RATE = 8, - DPTX_APCALL_SET_LINK_RATE = 9, - DPTX_APCALL_GET_MAX_LANE_COUNT = 10, - DPTX_APCALL_GET_ACTIVE_LANE_COUNT = 11, - DPTX_APCALL_SET_ACTIVE_LANE_COUNT = 12, - DPTX_APCALL_GET_SUPPORTS_DOWN_SPREAD = 13, - DPTX_APCALL_GET_DOWN_SPREAD = 14, - DPTX_APCALL_SET_DOWN_SPREAD = 15, - DPTX_APCALL_GET_SUPPORTS_LANE_MAPPING = 16, - DPTX_APCALL_SET_LANE_MAP = 17, - DPTX_APCALL_GET_SUPPORTS_HPD = 18, - DPTX_APCALL_FORCE_HOTPLUG_DETECT = 19, - DPTX_APCALL_INACTIVE_SINK_DETECTED = 20, - DPTX_APCALL_SET_TILED_DISPLAY_HINTS = 21, - DPTX_APCALL_DEVICE_NOT_RESPONDING = 22, - DPTX_APCALL_DEVICE_BUSY_TIMEOUT = 23, - DPTX_APCALL_DEVICE_NOT_STARTED = 24, -}; - -#define DCPDPTX_REMOTE_PORT_CORE GENMASK(3, 0) -#define DCPDPTX_REMOTE_PORT_ATC GENMASK(7, 4) -#define DCPDPTX_REMOTE_PORT_DIE GENMASK(11, 8) -#define DCPDPTX_REMOTE_PORT_CONNECTED BIT(15) -#define DCPDPTX_REMOTE_PORT_SUPPORTS_HPD BIT(8) - -enum neo_dptx_link_rate { - LINK_RATE_RBR = 0x06, - LINK_RATE_HBR = 0x0a, - LINK_RATE_HBR2 = 0x14, - LINK_RATE_HBR3 = 0x1e, -}; - -struct neo_apple_epic_service; - -struct neo_dptx_port { - bool enabled, connected; - struct completion enable_completion; - struct completion linkcfg_completion; - u32 unit; - struct neo_apple_epic_service *service; - union phy_configure_opts phy_ops; - struct phy *atcphy; - struct mux_control *mux; - u32 lane_count; - u32 link_rate, pending_link_rate; - u32 drive_settings[2]; - /* - * Per-boot invocation counters for the validate/connect/request/ - * release AFK calls, used together with the caller return address - * logged alongside them to correlate call ordering when diagnosing - * link-training sequencing issues. - */ - u32 validate_calls, connect_calls, request_calls, release_calls; -}; - -int neo_dptxport_validate_connection(struct neo_apple_epic_service *service, u8 core, - u8 atc, u8 die); -int neo_dptxport_connect(struct neo_apple_epic_service *service, u8 core, u8 atc, - u8 die, bool supports_hpd); -int neo_dptxport_request_display(struct neo_apple_epic_service *service); -int neo_dptxport_release_display(struct neo_apple_epic_service *service); -int neo_dptxport_set_hpd(struct neo_apple_epic_service *service, bool hpd); -int neo_dptxport_set_hpd_timeout(struct neo_apple_epic_service *service, bool hpd, - unsigned int timeout_ms); - -#endif diff --git a/drivers/gpu/drm/apple-neo/epic/dpavservep.c b/drivers/gpu/drm/apple-neo/epic/dpavservep.c deleted file mode 100644 index 8ec6b858822c93..00000000000000 --- a/drivers/gpu/drm/apple-neo/epic/dpavservep.c +++ /dev/null @@ -1,302 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#include "dpavservep.h" - -#include - -#include -#include -#include - -#include "../afk.h" -#include "../dcp.h" -#include "../dcp-internal.h" -#include "../trace.h" - -static void neo_dcpavserv_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - unsigned long flags; - - trace_neo_dcpavserv_init(neo_dcp, unit); - - if (unit == 0 && name && !strcmp(name, "dcpav-service-epic")) { - spin_lock_irqsave(&neo_dcp->neo_dcpavserv.lock, flags); - if (neo_dcp->neo_dcpavserv.enabled) { - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - dev_err(neo_dcp->dev, - "DCPAVSERV: unit %lld already exists\n", unit); - return; - } - neo_dcp->neo_dcpavserv.service = neo_afk_service_get(service); - neo_dcp->neo_dcpavserv.enabled = true; - service->cookie = &neo_dcp->neo_dcpavserv; - complete(&neo_dcp->neo_dcpavserv.enable_completion); - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - } -} - -static void neo_dcpavserv_teardown(struct neo_apple_epic_service *service) -{ - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - unsigned long flags; - bool owned = false; - - spin_lock_irqsave(&neo_dcp->neo_dcpavserv.lock, flags); - if (neo_dcp->neo_dcpavserv.service == service) { - neo_dcp->neo_dcpavserv.enabled = false; - neo_dcp->neo_dcpavserv.service = NULL; - reinit_completion(&neo_dcp->neo_dcpavserv.enable_completion); - owned = true; - } - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - if (owned) - neo_afk_service_put(service); -} - -void dpavservep_detach(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_epic_service *service; - unsigned long flags; - - spin_lock_irqsave(&neo_dcp->neo_dcpavserv.lock, flags); - service = neo_dcp->neo_dcpavserv.service; - neo_dcp->neo_dcpavserv.service = NULL; - neo_dcp->neo_dcpavserv.enabled = false; - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - if (service) { - neo_afk_service_disable(service); - neo_afk_service_put(service); - } -} - -static void dcpdpserv_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ -} - -static void dcpdpserv_teardown(struct neo_apple_epic_service *service) -{ - neo_afk_service_disable(service); -} - -struct neo_dcpavserv_status_report { - u32 unk00[4]; - u8 flag0; - u8 flag1; - u8 flag2; - u8 flag3; - u32 unk14[3]; - u32 status; - u32 unk24[3]; -} __packed; - -struct neo_dpavserv_copy_edid_cmd { - __le64 max_size; - u8 _pad1[24]; - __le64 used_size; - u8 _pad2[8]; -} __packed; - -#define EDID_LEADING_DATA_SIZE 8 -#define EDID_BLOCK_SIZE 128 -#define EDID_EXT_BLOCK_COUNT_OFFSET 0x7E -#define EDID_MAX_SIZE SZ_32K -#define EDID_BUF_SIZE (EDID_LEADING_DATA_SIZE + EDID_MAX_SIZE) - -struct neo_dpavserv_copy_edid_resp { - __le64 max_size; - u8 _pad1[24]; - __le64 used_size; - u8 _pad2[8]; - u8 data[]; -} __packed; -static_assert(sizeof(struct neo_dpavserv_copy_edid_resp) == 48); - -static int neo_parse_report(struct neo_apple_epic_service *service, enum neo_epic_subtype type, - const void *data, size_t data_size) -{ -#if defined(DEBUG) - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - const struct neo_epic_service_call *call; - const void *payload; - size_t payload_size; - - dev_dbg(neo_dcp->dev, "dcpavserv[ch:%u]: report type:%02x len:%zu\n", - service->channel, type, data_size); - - if (type != EPIC_SUBTYPE_STD_SERVICE) - return 0; - - if (data_size < sizeof(*call)) - return 0; - - call = data; - - if (le32_to_cpu(call->magic) != EPIC_SERVICE_CALL_MAGIC) { - dev_warn(neo_dcp->dev, "dcpavserv[ch:%u]: report magic 0x%08x != 0x%08x\n", - service->channel, le32_to_cpu(call->magic), EPIC_SERVICE_CALL_MAGIC); - return 0; - } - - payload_size = data_size - sizeof(*call); - if (payload_size < le32_to_cpu(call->data_len)) { - dev_warn(neo_dcp->dev, "dcpavserv[ch:%u]: report payload size %zu call len %u\n", - service->channel, payload_size, le32_to_cpu(call->data_len)); - return 0; - } - payload_size = le32_to_cpu(call->data_len); - payload = data + sizeof(*call); - - if (le16_to_cpu(call->group) == 2 && le16_to_cpu(call->command) == 0) { - if (payload_size == sizeof(struct neo_dcpavserv_status_report)) { - const struct neo_dcpavserv_status_report *stat = payload; - dev_info(neo_dcp->dev, "dcpavserv[ch:%u]: flags: 0x%02x,0x%02x,0x%02x,0x%02x status:%u\n", - service->channel, stat->flag0, stat->flag1, - stat->flag2, stat->flag3, stat->status); - } else { - dev_dbg(neo_dcp->dev, "dcpavserv[ch:%u]: report payload size %zu\n", service->channel, payload_size); - } - } else { - print_hex_dump(KERN_DEBUG, "dcpavserv report: ", DUMP_PREFIX_NONE, - 16, 1, payload, payload_size, true); - } -#endif - - return 0; -} - -static int neo_dcpavserv_report(struct neo_apple_epic_service *service, - enum neo_epic_subtype type, const void *data, - size_t data_size) -{ - return neo_parse_report(service, type, data, data_size); -} - -static int dcpdpserv_report(struct neo_apple_epic_service *service, - enum neo_epic_subtype type, const void *data, - size_t data_size) -{ - return neo_parse_report(service, type, data, data_size); -} - -static const struct drm_edid *neo_dcpavserv_read_edid(struct neo_apple_epic_service *service) -{ - struct neo_dpavserv_copy_edid_cmd cmd; - struct neo_dpavserv_copy_edid_resp *resp __free(kfree) = NULL; - size_t resp_size = sizeof(*resp) + EDID_BUF_SIZE; - int num_blocks; - u64 data_size; - int ret; - - memset(&cmd, 0, sizeof(cmd)); - cmd.max_size = cpu_to_le64(EDID_BUF_SIZE); - resp = kzalloc(resp_size, GFP_KERNEL); - if (!resp) - return ERR_PTR(-ENOMEM); - - ret = neo_afk_service_call(service, 1, 7, &cmd, sizeof(cmd), EDID_BUF_SIZE, resp, - resp_size, 0); - if (ret < 0) - return ERR_PTR(ret); - - if (le64_to_cpu(resp->max_size) != EDID_BUF_SIZE) - return ERR_PTR(-EIO); - - // print_hex_dump(KERN_DEBUG, "dpavserv EDID cmd: ", DUMP_PREFIX_NONE, - // 16, 1, resp, 192, true); - - data_size = le64_to_cpu(resp->used_size); - if (data_size < EDID_LEADING_DATA_SIZE + EDID_BLOCK_SIZE || - data_size > EDID_BUF_SIZE) - return ERR_PTR(-EIO); - - /* - * An HDMI 2.x sink using the HF-EEODB announces fewer extension blocks - * in the base block than it sends, so the payload may be longer than - * the base block says. Accept any whole number of blocks at least that - * long and leave the block count to drm_edid, which knows about - * HF-EEODB. - */ - num_blocks = resp->data[EDID_LEADING_DATA_SIZE + EDID_EXT_BLOCK_COUNT_OFFSET]; - if ((1 + num_blocks) * EDID_BLOCK_SIZE > data_size - EDID_LEADING_DATA_SIZE || - (data_size - EDID_LEADING_DATA_SIZE) % EDID_BLOCK_SIZE) - return ERR_PTR(-EIO); - - return drm_edid_alloc(resp->data + EDID_LEADING_DATA_SIZE, - data_size - EDID_LEADING_DATA_SIZE); -} - -const struct drm_edid *neo_dcpavserv_copy_edid(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_epic_service *service; - const struct drm_edid *edid; - unsigned long flags; - - spin_lock_irqsave(&neo_dcp->neo_dcpavserv.lock, flags); - service = neo_afk_service_get(neo_dcp->neo_dcpavserv.service); - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - if (!service) - return ERR_PTR(-ENODEV); - - edid = neo_dcpavserv_read_edid(service); - spin_lock_irqsave(&neo_dcp->neo_dcpavserv.lock, flags); - if (neo_dcp->neo_dcpavserv.service != service) { - if (!IS_ERR(edid)) - drm_edid_free(edid); - edid = ERR_PTR(-ENODEV); - } - spin_unlock_irqrestore(&neo_dcp->neo_dcpavserv.lock, flags); - neo_afk_service_put(service); - return edid; -} - -static const struct neo_apple_epic_service_ops dpavservep_ops[] = { - { - .name = "dcpav-service-epic", - .reusable = true, - .init = neo_dcpavserv_init, - .teardown = neo_dcpavserv_teardown, - .report = neo_dcpavserv_report, - }, - { - .name = "dcpdp-service-epic", - .reusable = true, - .init = dcpdpserv_init, - .teardown = dcpdpserv_teardown, - .report = dcpdpserv_report, - }, - {}, -}; - -int dpavservep_init(struct neo_apple_dcp *neo_dcp) -{ - struct neo_apple_dcp_afkep *ep; - int ret; - - init_completion(&neo_dcp->neo_dcpavserv.enable_completion); - - ep = neo_afk_init(neo_dcp, DPAVSERV_ENDPOINT, dpavservep_ops); - if (IS_ERR(ep)) - return PTR_ERR(ep); - neo_dcp->dcpavservep = ep; - - neo_dcp->dcpavservep->match_epic_name = true; - - ret = neo_afk_start(neo_dcp->dcpavservep); - if (ret) { - neo_afk_shutdown(ep); - dpavservep_detach(neo_dcp); - neo_dcp->dcpavservep = NULL; - return ret; - } - - ret = wait_for_completion_timeout(&neo_dcp->neo_dcpavserv.enable_completion, - msecs_to_jiffies(1000)); - if (ret >= 0) - return 0; - - return ret; -} diff --git a/drivers/gpu/drm/apple-neo/epic/dpavservep.h b/drivers/gpu/drm/apple-neo/epic/dpavservep.h deleted file mode 100644 index f8caa5262a523a..00000000000000 --- a/drivers/gpu/drm/apple-neo/epic/dpavservep.h +++ /dev/null @@ -1,26 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#ifndef _DRM_APPLE_EPIC_DPAVSERV_H -#define _DRM_APPLE_EPIC_DPAVSERV_H - -#include -#include -#include - -struct drm_edid; -struct neo_apple_epic_service; -struct neo_apple_dcp; - -struct neo_dcpavserv { - spinlock_t lock; /* Protects the service pointer and its owner pin. */ - bool enabled; - struct completion enable_completion; - u32 unit; - struct neo_apple_epic_service *service; -}; - -const struct drm_edid *neo_dcpavserv_copy_edid(struct neo_apple_dcp *neo_dcp); -void dpavservep_detach(struct neo_apple_dcp *neo_dcp); - -#endif /* _DRM_APPLE_EPIC_DPAVSERV_H */ diff --git a/drivers/gpu/drm/apple-neo/hdmi-codec-chmap.h b/drivers/gpu/drm/apple-neo/hdmi-codec-chmap.h deleted file mode 100644 index f98e1e86b89602..00000000000000 --- a/drivers/gpu/drm/apple-neo/hdmi-codec-chmap.h +++ /dev/null @@ -1,123 +0,0 @@ -// copied from sound/soc/codecs/hdmi-codec.c - -#include - -/* Channel maps for multi-channel playbacks, up to 8 n_ch */ -static const struct snd_pcm_chmap_elem hdmi_codec_8ch_chmaps[] = { - { .channels = 2, /* CA_ID 0x00 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR } }, - { .channels = 4, /* CA_ID 0x01 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA } }, - { .channels = 4, /* CA_ID 0x02 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC } }, - { .channels = 4, /* CA_ID 0x03 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC } }, - { .channels = 6, /* CA_ID 0x04 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 6, /* CA_ID 0x05 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 6, /* CA_ID 0x06 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 6, /* CA_ID 0x07 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 6, /* CA_ID 0x08 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR } }, - { .channels = 6, /* CA_ID 0x09 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR } }, - { .channels = 6, /* CA_ID 0x0A */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR } }, - { .channels = 6, /* CA_ID 0x0B */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR } }, - { .channels = 8, /* CA_ID 0x0C */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 8, /* CA_ID 0x0D */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 8, /* CA_ID 0x0E */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 8, /* CA_ID 0x0F */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RC, SNDRV_CHMAP_NA } }, - { .channels = 8, /* CA_ID 0x10 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RLC, SNDRV_CHMAP_RRC } }, - { .channels = 8, /* CA_ID 0x11 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RLC, SNDRV_CHMAP_RRC } }, - { .channels = 8, /* CA_ID 0x12 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RLC, SNDRV_CHMAP_RRC } }, - { .channels = 8, /* CA_ID 0x13 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_RL, SNDRV_CHMAP_RR, - SNDRV_CHMAP_RLC, SNDRV_CHMAP_RRC } }, - { .channels = 8, /* CA_ID 0x14 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x15 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x16 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x17 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x18 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x19 */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1A */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1B */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1C */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1D */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1E */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { .channels = 8, /* CA_ID 0x1F */ - .map = { SNDRV_CHMAP_FL, SNDRV_CHMAP_FR, SNDRV_CHMAP_LFE, - SNDRV_CHMAP_FC, SNDRV_CHMAP_NA, SNDRV_CHMAP_NA, - SNDRV_CHMAP_FLC, SNDRV_CHMAP_FRC } }, - { } -}; diff --git a/drivers/gpu/drm/apple-neo/ibootep.c b/drivers/gpu/drm/apple-neo/ibootep.c deleted file mode 100644 index a84b527ca10031..00000000000000 --- a/drivers/gpu/drm/apple-neo/ibootep.c +++ /dev/null @@ -1,29 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2023 */ - -#include - -#include "afk.h" -#include "dcp.h" - -static void disp_service_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ -} - - -static const struct neo_apple_epic_service_ops neo_ibootep_ops[] = { - { - .name = "disp0-service", - .init = disp_service_init, - }, - {} -}; - -int neo_ibootep_init(struct neo_apple_dcp *neo_dcp) -{ - neo_dcp->neo_ibootep = neo_afk_init(neo_dcp, DISP0_ENDPOINT, neo_ibootep_ops); - neo_afk_start(neo_dcp->neo_ibootep); - - return 0; -} diff --git a/drivers/gpu/drm/apple-neo/iomfb-state-test.c b/drivers/gpu/drm/apple-neo/iomfb-state-test.c deleted file mode 100644 index 5b597580e36069..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb-state-test.c +++ /dev/null @@ -1,127 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#include -#include - -#include "iomfb-state.h" - -static void neo_dcp_mode_success_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = true; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - KUNIT_EXPECT_FALSE(test, state.valid); - KUNIT_EXPECT_EQ(test, neo_dcp_mode_finish(&state, true, &connected), 0U); - KUNIT_EXPECT_TRUE(test, state.valid); - KUNIT_EXPECT_FALSE(test, state.changing); -} - -static void neo_dcp_mode_failure_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { .valid = true }; - bool connected = true; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - neo_dcp_mode_finish(&state, false, &connected); - KUNIT_EXPECT_FALSE(test, state.valid); - KUNIT_EXPECT_FALSE(test, state.changing); -} - -static void neo_dcp_mode_deferred_disconnect_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = true; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - KUNIT_EXPECT_EQ(test, neo_dcp_mode_hotplug(&state, false, &connected), 0U); - KUNIT_EXPECT_TRUE(test, connected); - KUNIT_EXPECT_EQ(test, neo_dcp_mode_finish(&state, true, &connected), - (unsigned int)(DCP_HOTPLUG_VBLANK | DCP_HOTPLUG_NOTIFY)); - KUNIT_EXPECT_FALSE(test, connected); - KUNIT_EXPECT_FALSE(test, state.valid); - KUNIT_EXPECT_FALSE(test, state.hotplug_pending); -} - -static void neo_dcp_mode_deferred_connect_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = false; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - neo_dcp_mode_hotplug(&state, true, &connected); - KUNIT_EXPECT_EQ(test, neo_dcp_mode_finish(&state, true, &connected), - (unsigned int)DCP_HOTPLUG_NOTIFY); - KUNIT_EXPECT_TRUE(test, connected); - KUNIT_EXPECT_FALSE(test, state.valid); - /* A recovery modeset without another link change can now succeed. */ - neo_dcp_mode_begin(&state); - neo_dcp_mode_finish(&state, true, &connected); - KUNIT_EXPECT_TRUE(test, state.valid); -} - -static void neo_dcp_mode_transient_disconnect_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = true; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - neo_dcp_mode_hotplug(&state, false, &connected); - neo_dcp_mode_hotplug(&state, true, &connected); - KUNIT_EXPECT_EQ(test, neo_dcp_mode_finish(&state, true, &connected), - (unsigned int)DCP_HOTPLUG_NOTIFY); - KUNIT_EXPECT_TRUE(test, connected); - KUNIT_EXPECT_FALSE(test, state.valid); -} - -static void neo_dcp_mode_invalidation_order_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = true; - - spin_lock_init(&state.lock); - /* Invalidation just before completion must win. */ - neo_dcp_mode_begin(&state); - neo_dcp_mode_invalidate(&state); - neo_dcp_mode_finish(&state, true, &connected); - KUNIT_EXPECT_FALSE(test, state.valid); - /* So must invalidation just after completion. */ - neo_dcp_mode_begin(&state); - neo_dcp_mode_finish(&state, true, &connected); - neo_dcp_mode_invalidate(&state); - KUNIT_EXPECT_FALSE(test, state.valid); -} - -static void neo_dcp_mode_disconnected_test(struct kunit *test) -{ - struct neo_dcp_mode_state state = { }; - bool connected = false; - - spin_lock_init(&state.lock); - neo_dcp_mode_begin(&state); - neo_dcp_mode_finish(&state, true, &connected); - KUNIT_EXPECT_FALSE(test, state.valid); -} - -static struct kunit_case neo_dcp_mode_cases[] = { - KUNIT_CASE(neo_dcp_mode_success_test), - KUNIT_CASE(neo_dcp_mode_failure_test), - KUNIT_CASE(neo_dcp_mode_deferred_disconnect_test), - KUNIT_CASE(neo_dcp_mode_deferred_connect_test), - KUNIT_CASE(neo_dcp_mode_transient_disconnect_test), - KUNIT_CASE(neo_dcp_mode_invalidation_order_test), - KUNIT_CASE(neo_dcp_mode_disconnected_test), - { } -}; - -static struct kunit_suite neo_dcp_mode_suite = { - .name = "apple-iomfb-mode-state", - .test_cases = neo_dcp_mode_cases, -}; - -kunit_test_suite(neo_dcp_mode_suite); -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple-neo/iomfb-state.h b/drivers/gpu/drm/apple-neo/iomfb-state.h deleted file mode 100644 index 800146a8c236f3..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb-state.h +++ /dev/null @@ -1,110 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ -#ifndef __APPLE_IOMFB_STATE_H__ -#define __APPLE_IOMFB_STATE_H__ - -#include - -/* Shared by the atomic commit worker and the firmware callback thread. */ -struct neo_dcp_mode_state { - spinlock_t lock; /* Protects all state and hotplug connector updates. */ - bool valid; - bool changing; - bool invalidated; - bool hotplug_pending; - bool connected_pending; -}; - -enum neo_dcp_hotplug_action { - DCP_HOTPLUG_NONE = 0, - DCP_HOTPLUG_VBLANK = BIT(0), - DCP_HOTPLUG_NOTIFY = BIT(1), -}; - -static inline void neo_dcp_mode_begin(struct neo_dcp_mode_state *state) -{ - unsigned long flags; - - spin_lock_irqsave(&state->lock, flags); - state->valid = false; - state->changing = true; - state->invalidated = false; - state->hotplug_pending = false; - spin_unlock_irqrestore(&state->lock, flags); -} - -static inline void neo_dcp_mode_invalidate(struct neo_dcp_mode_state *state) -{ - unsigned long flags; - - spin_lock_irqsave(&state->lock, flags); - state->valid = false; - state->invalidated = true; - spin_unlock_irqrestore(&state->lock, flags); -} - -/* The caller holds state->lock. */ -static inline unsigned int -neo_dcp_mode_apply_hotplug(struct neo_dcp_mode_state *state, bool connected, - bool *connector_connected) -{ - unsigned int action = DCP_HOTPLUG_NONE; - - if (!connected) { - state->valid = false; - action |= DCP_HOTPLUG_VBLANK; - } - if (connector_connected && *connector_connected != connected) { - *connector_connected = connected; - state->valid = false; - action |= DCP_HOTPLUG_NOTIFY; - } - return action; -} - -static inline unsigned int -neo_dcp_mode_hotplug(struct neo_dcp_mode_state *state, bool connected, - bool *connector_connected) -{ - unsigned long flags; - unsigned int action = DCP_HOTPLUG_NONE; - - spin_lock_irqsave(&state->lock, flags); - if (state->changing) { - state->hotplug_pending = true; - state->connected_pending = connected; - /* A later connect must not erase a link loss during this call. */ - if (!connected) - state->invalidated = true; - } else { - action = neo_dcp_mode_apply_hotplug(state, connected, - connector_connected); - } - spin_unlock_irqrestore(&state->lock, flags); - return action; -} - -static inline unsigned int -neo_dcp_mode_finish(struct neo_dcp_mode_state *state, bool completed, - bool *connector_connected) -{ - unsigned long flags; - unsigned int action = DCP_HOTPLUG_NONE; - - spin_lock_irqsave(&state->lock, flags); - state->valid = completed && !state->invalidated; - state->changing = false; - if (state->hotplug_pending) { - state->hotplug_pending = false; - action = neo_dcp_mode_apply_hotplug(state, state->connected_pending, - connector_connected); - } - if (connector_connected && !*connector_connected) - state->valid = false; - /* A transient unplug also needs recovery if the final state is up. */ - if (state->invalidated && connector_connected && *connector_connected) - action |= DCP_HOTPLUG_NOTIFY; - spin_unlock_irqrestore(&state->lock, flags); - return action; -} - -#endif diff --git a/drivers/gpu/drm/apple-neo/iomfb.c b/drivers/gpu/drm/apple-neo/iomfb.c deleted file mode 100644 index 7e8fdc055eff38..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb.c +++ /dev/null @@ -1,1588 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "dcp.h" -#include "dcp-internal.h" -#include "iomfb.h" -#include "iomfb_internal.h" -#include "parser.h" -#include "trace.h" - -static int neo_dcp_tx_offset(enum neo_dcp_context_id id) -{ - switch (id) { - case DCP_CONTEXT_CB: - case DCP_CONTEXT_CMD: - return 0x00000; - case DCP_CONTEXT_OOBCB: - case DCP_CONTEXT_OOBCMD: - return 0x08000; - default: - return -EINVAL; - } -} - -static int neo_dcp_channel_offset(enum neo_dcp_context_id id) -{ - switch (id) { - case DCP_CONTEXT_ASYNC: - return 0x40000; - case DCP_CONTEXT_OOBASYNC: - return 0x48000; - case DCP_CONTEXT_CB: - return 0x60000; - case DCP_CONTEXT_OOBCB: - return 0x68000; - default: - return neo_dcp_tx_offset(id); - } -} - -static inline u64 dcpep_set_shmem(u64 dart_va) -{ - return FIELD_PREP(IOMFB_MESSAGE_TYPE, IOMFB_MESSAGE_TYPE_SET_SHMEM) | - FIELD_PREP(IOMFB_SHMEM_FLAG, IOMFB_SHMEM_FLAG_VALUE) | - FIELD_PREP(IOMFB_SHMEM_DVA, dart_va); -} - -static inline u64 dcpep_msg(enum neo_dcp_context_id id, u32 length, u16 offset) -{ - return FIELD_PREP(IOMFB_MESSAGE_TYPE, IOMFB_MESSAGE_TYPE_MSG) | - FIELD_PREP(IOMFB_MSG_CONTEXT, id) | - FIELD_PREP(IOMFB_MSG_OFFSET, offset) | - FIELD_PREP(IOMFB_MSG_LENGTH, length); -} - -static inline u64 dcpep_ack(enum neo_dcp_context_id id) -{ - return dcpep_msg(id, 0, 0) | IOMFB_MSG_ACK; -} - -/* - * A channel is busy if we have sent a message that has yet to be - * acked. The driver must not sent a message to a busy channel. - */ -static bool neo_dcp_channel_busy(struct neo_dcp_channel *ch) -{ - return (ch->depth != 0); -} - -/* - * Get the context ID passed to the DCP for a command we push. The rule is - * simple: callback contexts are used when replying to the DCP, command - * contexts are used otherwise. That corresponds to a non/zero call stack - * depth. This rule frees the caller from tracking the call context manually. - */ -static enum neo_dcp_context_id neo_dcp_call_context(struct neo_apple_dcp *neo_dcp, bool oob) -{ - u8 depth = oob ? neo_dcp->ch_oobcmd.depth : neo_dcp->ch_cmd.depth; - - if (depth) - return oob ? DCP_CONTEXT_OOBCB : DCP_CONTEXT_CB; - else - return oob ? DCP_CONTEXT_OOBCMD : DCP_CONTEXT_CMD; -} - -/* Get a channel for a context */ -static struct neo_dcp_channel *neo_dcp_get_channel(struct neo_apple_dcp *neo_dcp, - enum neo_dcp_context_id context) -{ - switch (context) { - case DCP_CONTEXT_CB: - return &neo_dcp->ch_cb; - case DCP_CONTEXT_CMD: - return &neo_dcp->ch_cmd; - case DCP_CONTEXT_OOBCB: - return &neo_dcp->ch_oobcb; - case DCP_CONTEXT_OOBCMD: - return &neo_dcp->ch_oobcmd; - case DCP_CONTEXT_ASYNC: - return &neo_dcp->ch_async; - case DCP_CONTEXT_OOBASYNC: - return &neo_dcp->ch_oobasync; - default: - return NULL; - } -} - -/* Get the start of a packet: after the end of the previous packet */ -static u16 neo_dcp_packet_start(struct neo_dcp_channel *ch, u8 depth) -{ - if (depth > 0) - return ch->end[depth - 1]; - else - return 0; -} - -/* Pushes and pops the depth of the call stack with safety checks */ -static u8 neo_dcp_push_depth(u8 *depth) -{ - u8 ret = (*depth)++; - - WARN_ON(ret >= DCP_MAX_CALL_DEPTH); - return ret; -} - -static u8 neo_dcp_pop_depth(u8 *depth) -{ - WARN_ON((*depth) == 0); - - return --(*depth); -} - -/* Older firmware and the H17G method profile retain their existing transport. */ -static bool neo_iomfb_uses_queue(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G; -} - -static bool neo_iomfb_channels_idle(struct neo_apple_dcp *neo_dcp) -{ - return !neo_dcp->ch_cmd.depth && !neo_dcp->ch_cb.depth && - !neo_dcp->ch_oobcmd.depth && !neo_dcp->ch_oobcb.depth && - !neo_dcp->ch_async.depth && !neo_dcp->ch_oobasync.depth; -} - -enum neo_iomfb_opaque_x_state { - IOMFB_OPAQUE_X_WAITING, - IOMFB_OPAQUE_X_NEEDED, - IOMFB_OPAQUE_X_QUEUED, - IOMFB_OPAQUE_X_READY, -}; - -struct neo_iomfb_opaque_x_transaction { - struct neo_iomfb_transaction transaction; -}; - -static void neo_iomfb_opaque_x_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction); -static void neo_iomfb_opaque_x_release(struct neo_iomfb_transaction *transaction); - -static int neo_iomfb_enqueue_opaque_x(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_opaque_x_transaction *opaque; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (atomic_read(&neo_dcp->neo_iomfb.opaque_x_state) != IOMFB_OPAQUE_X_NEEDED) - return 0; - if (neo_dcp->neo_iomfb.queued >= 32) - return -EBUSY; - opaque = kzalloc_obj(*opaque); - if (!opaque) - return -ENOMEM; - if (atomic_cmpxchg(&neo_dcp->neo_iomfb.opaque_x_state, IOMFB_OPAQUE_X_NEEDED, - IOMFB_OPAQUE_X_QUEUED) != IOMFB_OPAQUE_X_NEEDED) { - kfree(opaque); - return 0; - } - - opaque->transaction.start = neo_iomfb_opaque_x_start; - opaque->transaction.release = neo_iomfb_opaque_x_release; - neo_dcp->neo_iomfb.queued++; - list_add(&opaque->transaction.link, &neo_dcp->neo_iomfb.pending); - return 0; -} - -static void neo_iomfb_discard_pending(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_transaction *transaction, *next; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - list_for_each_entry_safe(transaction, next, &neo_dcp->neo_iomfb.pending, link) { - list_del(&transaction->link); - neo_dcp->neo_iomfb.queued--; - if (transaction->brightness_only) - neo_dcp->neo_iomfb.backlight_queued = false; - transaction->release(transaction); - } -} - -static void neo_iomfb_queue_advance(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_transaction *transaction = neo_dcp->neo_iomfb.active; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (READ_ONCE(neo_dcp->crashed) || neo_dcp->neo_iomfb.stopped) { - neo_iomfb_discard_pending(neo_dcp); - /* The active operation may still be visible to firmware. */ - return; - } - if (!neo_iomfb_channels_idle(neo_dcp) || neo_dcp->present_state_h17p.pending) - return; - - if (transaction) { - neo_dcp->neo_iomfb.active = NULL; - cancel_delayed_work(&neo_dcp->neo_iomfb.timeout); - if (transaction->brightness_only) - neo_dcp->neo_iomfb.backlight_queued = false; - /* A new level requested during a successful present stays pending. */ - if (neo_dcp_backlight_pending(neo_dcp)) { - if (transaction->completed) { - schedule_work(&neo_dcp->bl_update_wq); - } else if (transaction->backlight_failed) { - unsigned int delay = neo_dcp_backlight_retry_delay(neo_dcp); - - if (delay) - mod_delayed_work(system_wq, &neo_dcp->neo_iomfb.backlight_retry, - msecs_to_jiffies(delay)); - else - dev_warn_ratelimited(neo_dcp->dev, - "backlight retry limit reached\n"); - } - } - transaction->release(transaction); - } - if (neo_iomfb_enqueue_opaque_x(neo_dcp)) { - WRITE_ONCE(neo_dcp->crashed, true); - neo_iomfb_discard_pending(neo_dcp); - schedule_work(&neo_dcp->vblank_wq); - return; - } - if (!list_empty(&neo_dcp->neo_iomfb.pending)) - schedule_work(&neo_dcp->neo_iomfb.work); -} - -static void neo_iomfb_queue_work(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = container_of(work, struct neo_apple_dcp, neo_iomfb.work); - struct neo_iomfb_transaction *transaction; - - mutex_lock(&neo_dcp->neo_iomfb.lock); - if (READ_ONCE(neo_dcp->crashed) || neo_dcp->neo_iomfb.stopped) { - neo_iomfb_discard_pending(neo_dcp); - goto unlock; - } - if (neo_dcp->neo_iomfb.active || !neo_iomfb_channels_idle(neo_dcp) || - neo_dcp->present_state_h17p.pending || list_empty(&neo_dcp->neo_iomfb.pending)) - goto unlock; - - transaction = list_first_entry(&neo_dcp->neo_iomfb.pending, - struct neo_iomfb_transaction, link); - list_del(&transaction->link); - neo_dcp->neo_iomfb.queued--; - neo_dcp->neo_iomfb.active = transaction; - WRITE_ONCE(neo_dcp->neo_iomfb.owner, current); - neo_dcp->neo_iomfb.deadline = jiffies + msecs_to_jiffies(10000); - mod_delayed_work(system_wq, &neo_dcp->neo_iomfb.timeout, msecs_to_jiffies(10000)); - transaction->start(neo_dcp, transaction); - WRITE_ONCE(neo_dcp->neo_iomfb.owner, NULL); - neo_iomfb_queue_advance(neo_dcp); -unlock: - mutex_unlock(&neo_dcp->neo_iomfb.lock); -} - -static void neo_iomfb_queue_timeout(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = container_of(to_delayed_work(work), - struct neo_apple_dcp, neo_iomfb.timeout); - - mutex_lock(&neo_dcp->neo_iomfb.lock); - if (neo_dcp->neo_iomfb.active && time_before(jiffies, neo_dcp->neo_iomfb.deadline)) { - mod_delayed_work(system_wq, &neo_dcp->neo_iomfb.timeout, - neo_dcp->neo_iomfb.deadline - jiffies); - } else if (neo_dcp->neo_iomfb.active) { - WRITE_ONCE(neo_dcp->crashed, true); - dev_err(neo_dcp->dev, "IOMFB transaction timed out\n"); - neo_iomfb_discard_pending(neo_dcp); - schedule_work(&neo_dcp->vblank_wq); - } - mutex_unlock(&neo_dcp->neo_iomfb.lock); -} - -static void neo_iomfb_opaque_x_complete(struct neo_apple_dcp *neo_dcp, void *out, - void *cookie) -{ - u32 status = out ? *(u32 *)out : ~0U; - - (void)cookie; - - if (status) { - dev_err(neo_dcp->dev, "opaque X property failed: %u\n", status); - WRITE_ONCE(neo_dcp->crashed, true); - schedule_work(&neo_dcp->vblank_wq); - return; - } - - atomic_set(&neo_dcp->neo_iomfb.opaque_x_state, IOMFB_OPAQUE_X_READY); -} - -static void neo_iomfb_opaque_x_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction) -{ - (void)transaction; - neo_iomfb_apply_opaque_x_h17p(neo_dcp, neo_iomfb_opaque_x_complete, NULL); -} - -static void neo_iomfb_opaque_x_release(struct neo_iomfb_transaction *transaction) -{ - kfree(container_of(transaction, struct neo_iomfb_opaque_x_transaction, - transaction)); -} - -void neo_iomfb_opaque_x_reset_h17p(struct neo_apple_dcp *neo_dcp) -{ - atomic_set(&neo_dcp->neo_iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING); -} - -static void neo_iomfb_backlight_retry(struct work_struct *work) -{ - struct neo_apple_dcp *neo_dcp = container_of(to_delayed_work(work), struct neo_apple_dcp, - neo_iomfb.backlight_retry); - - if (!READ_ONCE(neo_dcp->crashed) && !READ_ONCE(neo_dcp->neo_iomfb.stopped) && - neo_dcp_backlight_pending(neo_dcp)) - schedule_work(&neo_dcp->bl_update_wq); -} - -void neo_iomfb_queue_init(struct neo_apple_dcp *neo_dcp) -{ - mutex_init(&neo_dcp->neo_iomfb.lock); - INIT_LIST_HEAD(&neo_dcp->neo_iomfb.pending); - INIT_WORK(&neo_dcp->neo_iomfb.work, neo_iomfb_queue_work); - INIT_DELAYED_WORK(&neo_dcp->neo_iomfb.timeout, neo_iomfb_queue_timeout); - INIT_DELAYED_WORK(&neo_dcp->neo_iomfb.backlight_retry, neo_iomfb_backlight_retry); - atomic_set(&neo_dcp->neo_iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING); -} - -void neo_iomfb_queue_stop(struct neo_apple_dcp *neo_dcp) -{ - if (!neo_iomfb_uses_queue(neo_dcp)) - return; - - mutex_lock(&neo_dcp->neo_iomfb.lock); - neo_dcp->neo_iomfb.stopped = true; - neo_iomfb_discard_pending(neo_dcp); - mutex_unlock(&neo_dcp->neo_iomfb.lock); - cancel_work_sync(&neo_dcp->neo_iomfb.work); - cancel_delayed_work_sync(&neo_dcp->neo_iomfb.timeout); - cancel_delayed_work_sync(&neo_dcp->neo_iomfb.backlight_retry); -} - -/* - * Wait until every queued transaction, including a present's completion, has - * finished. A stopped or crashed queue has nothing left to deliver. - */ -bool neo_iomfb_queue_drain(struct neo_apple_dcp *neo_dcp, unsigned long timeout) -{ - unsigned long deadline = jiffies + timeout; - bool idle; - - if (!neo_iomfb_uses_queue(neo_dcp)) - return true; - - for (;;) { - mutex_lock(&neo_dcp->neo_iomfb.lock); - idle = READ_ONCE(neo_dcp->crashed) || neo_dcp->neo_iomfb.stopped || - (!neo_dcp->neo_iomfb.active && list_empty(&neo_dcp->neo_iomfb.pending) && - !neo_dcp->present_state_h17p.pending); - mutex_unlock(&neo_dcp->neo_iomfb.lock); - if (idle) - return true; - if (time_after(jiffies, deadline)) - return false; - usleep_range(2000, 4000); - } -} - -int neo_iomfb_queue(struct neo_apple_dcp *neo_dcp, struct neo_iomfb_transaction *transaction) -{ - int ret = 0; - - mutex_lock(&neo_dcp->neo_iomfb.lock); - if (READ_ONCE(neo_dcp->crashed) || neo_dcp->neo_iomfb.stopped) { - ret = -EIO; - } else if (transaction->brightness_only && neo_dcp->neo_iomfb.backlight_queued) { - ret = -EALREADY; - } else if (neo_dcp->neo_iomfb.queued >= 32) { - ret = -EBUSY; - } else { - if (transaction->brightness_only) - neo_dcp->neo_iomfb.backlight_queued = true; - neo_dcp->neo_iomfb.queued++; - list_add_tail(&transaction->link, &neo_dcp->neo_iomfb.pending); - schedule_work(&neo_dcp->neo_iomfb.work); - } - mutex_unlock(&neo_dcp->neo_iomfb.lock); - return ret; -} - -struct neo_iomfb_crc_transaction { - struct neo_iomfb_transaction transaction; - u32 swap_id; -}; - -static void neo_iomfb_crc_complete(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) -{ - struct neo_iomfb_crc_transaction *crc = cookie; - struct neo_dcp_packet_header *header; - u32 value; - - /* A106 has no input and a single little-endian blend-output CRC. */ - if (!out) - goto invalid; - header = out - sizeof(*header); - if (header->in_len || header->out_len != sizeof(__le32)) - goto invalid; - value = get_unaligned_le32(out); - if (READ_ONCE(neo_dcp->crc_enabled)) - drm_crtc_add_crc_entry(&neo_dcp->crtc->base, true, crc->swap_id, &value); - return; - -invalid: - dev_err(neo_dcp->dev, "invalid blend CRC response\n"); - WRITE_ONCE(neo_dcp->crashed, true); -} - -static void neo_iomfb_crc_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction) -{ - static const struct neo_dcp_method_entry method = { - .tag = { 'A', '1', '0', '6' }, - .name = "read_blend_crc", - }; - struct neo_iomfb_crc_transaction *crc = container_of(transaction, - struct neo_iomfb_crc_transaction, - transaction); - - if (READ_ONCE(neo_dcp->crc_enabled)) - neo_dcp_push(neo_dcp, false, &method, 0, sizeof(__le32), NULL, - neo_iomfb_crc_complete, crc); -} - -static void neo_iomfb_crc_release(struct neo_iomfb_transaction *transaction) -{ - kfree(container_of(transaction, struct neo_iomfb_crc_transaction, transaction)); -} - -void neo_iomfb_queue_crc_h17p(struct neo_apple_dcp *neo_dcp, u32 swap_id) -{ - struct neo_iomfb_crc_transaction *crc; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (!READ_ONCE(neo_dcp->crc_enabled) || READ_ONCE(neo_dcp->crashed) || - neo_dcp->neo_iomfb.stopped || neo_dcp->neo_iomfb.queued >= 32) - return; - crc = kzalloc_obj(*crc); - if (!crc) - return; - crc->swap_id = swap_id; - crc->transaction.start = neo_iomfb_crc_start; - crc->transaction.release = neo_iomfb_crc_release; - /* Read the completed frame before another queued present changes it. */ - list_add(&crc->transaction.link, &neo_dcp->neo_iomfb.pending); - neo_dcp->neo_iomfb.queued++; - /* The receiver advances the queue only after acknowledging completion. */ -} - -struct neo_iomfb_command { - struct neo_iomfb_transaction transaction; - struct neo_dcp_method_entry method; - neo_dcp_callback_t callback; - void *cookie; - u32 in_len; - u32 out_len; - bool oob; - u8 data[]; -}; - -static void neo_iomfb_command_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction) -{ - struct neo_iomfb_command *command = container_of(transaction, - struct neo_iomfb_command, transaction); - - neo_dcp_push(neo_dcp, command->oob, &command->method, command->in_len, - command->out_len, command->data, command->callback, command->cookie); -} - -static void neo_iomfb_command_release(struct neo_iomfb_transaction *transaction) -{ - kfree(container_of(transaction, struct neo_iomfb_command, transaction)); -} - -static int neo_iomfb_queue_command(struct neo_apple_dcp *neo_dcp, bool oob, - const struct neo_dcp_method_entry *method, - u32 in_len, u32 out_len, void *data, - neo_dcp_callback_t callback, void *cookie) -{ - struct neo_iomfb_command *command; - int ret; - - if ((u64)sizeof(struct neo_dcp_packet_header) + in_len + out_len > 0x8000) - return -EMSGSIZE; - command = kzalloc(struct_size(command, data, in_len), GFP_KERNEL); - if (!command) - return -ENOMEM; - command->transaction.start = neo_iomfb_command_start; - command->transaction.release = neo_iomfb_command_release; - command->method = *method; - command->callback = callback; - command->cookie = cookie; - command->in_len = in_len; - command->out_len = out_len; - command->oob = oob; - if (in_len) - memcpy(command->data, data, in_len); - ret = neo_iomfb_queue(neo_dcp, &command->transaction); - if (ret) - neo_iomfb_command_release(&command->transaction); - return ret; -} - -/* Call a DCP function given by a tag */ -void neo_dcp_push(struct neo_apple_dcp *neo_dcp, bool oob, const struct neo_dcp_method_entry *call, - u32 in_len, u32 out_len, void *data, neo_dcp_callback_t cb, - void *cookie) -{ - struct neo_dcp_method_entry resolved = *call; - - /* Reply chains stay on the serialized receiver; callers enqueue copies. */ - if (neo_iomfb_uses_queue(neo_dcp) && READ_ONCE(neo_dcp->neo_iomfb.owner) != current) { - if (neo_iomfb_queue_command(neo_dcp, oob, call, in_len, out_len, - data, cb, cookie)) - WRITE_ONCE(neo_dcp->crashed, true); - return; - } - - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && READ_ONCE(neo_dcp->crashed)) - return; - - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G && - call->tag_h17g[0]) - memcpy(resolved.tag, call->tag_h17g, sizeof(resolved.tag)); - call = &resolved; - - enum neo_dcp_context_id context = neo_dcp_call_context(neo_dcp, oob); - struct neo_dcp_channel *ch = neo_dcp_get_channel(neo_dcp, context); - - struct neo_dcp_packet_header header = { - .in_len = in_len, - .out_len = out_len, - - /* Tag is reversed due to endianness of the fourcc */ - .tag[0] = call->tag[3], - .tag[1] = call->tag[2], - .tag[2] = call->tag[1], - .tag[3] = call->tag[0], - }; - - u8 depth; - u16 offset; - - if (neo_iomfb_uses_queue(neo_dcp) && - (!ch || ch->depth >= DCP_MAX_CALL_DEPTH || - (u64)sizeof(header) + in_len + out_len > 0x8000 || - neo_dcp_packet_start(ch, ch->depth) > - 0x8000 - ALIGN(sizeof(header) + in_len + out_len, - DCP_PACKET_ALIGNMENT))) { - dev_err(neo_dcp->dev, "invalid IOMFB command envelope\n"); - WRITE_ONCE(neo_dcp->crashed, true); - return; - } - depth = neo_dcp_push_depth(&ch->depth); - offset = neo_dcp_packet_start(ch, depth); - - void *out = neo_dcp->shmem + neo_dcp_tx_offset(context) + offset; - void *out_data = out + sizeof(header); - size_t data_len = sizeof(header) + in_len + out_len; - - memcpy(out, &header, sizeof(header)); - - if (in_len > 0) - memcpy(out_data, data, in_len); - /* An unwritten status must not look like a successful response. */ - if (out_len) - memset(out_data + in_len, 0xff, out_len); - - trace_neo_iomfb_push(neo_dcp, call, context, offset, depth); - - ch->callbacks[depth] = cb; - ch->cookies[depth] = cookie; - ch->output[depth] = out + sizeof(header) + in_len; - ch->header[depth] = header; - ch->end[depth] = offset + ALIGN(data_len, DCP_PACKET_ALIGNMENT); - - neo_dcp_send_message(neo_dcp, IOMFB_ENDPOINT, - dcpep_msg(context, data_len, offset)); -} - -/* Parse a callback tag "D123" into the ID 123. Returns -EINVAL on failure. */ -int neo_dcp_parse_tag(char tag[4]) -{ - u32 d[3]; - int i; - - if (tag[3] != 'D') - return -EINVAL; - - for (i = 0; i < 3; ++i) { - d[i] = (u32)(tag[i] - '0'); - - if (d[i] > 9) - return -EINVAL; - } - - return d[0] + (d[1] * 10) + (d[2] * 100); -} - -/* Ack a callback from the DCP */ -void neo_dcp_ack(struct neo_apple_dcp *neo_dcp, enum neo_dcp_context_id context) -{ - struct neo_dcp_channel *ch = neo_dcp_get_channel(neo_dcp, context); - - neo_dcp_pop_depth(&ch->depth); - neo_dcp_send_message(neo_dcp, IOMFB_ENDPOINT, - dcpep_ack(context)); -} - -/* - * Helper to send a DRM hotplug event. The DCP is accessed from a single - * (RTKit) thread. To handle hotplug callbacks, we need to call - * drm_kms_helper_hotplug_event, which does an atomic commit (via DCP) and - * waits for vblank (a DCP callback). That means we deadlock if we call from - * the RTKit thread! Instead, move the call to another thread via a workqueue. - */ -static int neo_dcp_retrain_active_crtc(struct neo_apple_connector *connector) -{ - struct drm_device *dev = connector->base.dev; - struct drm_modeset_acquire_ctx ctx; - struct drm_crtc *crtc; - int ret; - - DRM_MODESET_LOCK_ALL_BEGIN(dev, ctx, 0, ret); - - crtc = connector->base.state ? connector->base.state->crtc : NULL; - if (crtc && crtc->state && crtc->state->active) - ret = drm_atomic_helper_reset_crtc(crtc, &ctx); - else - ret = 0; - - DRM_MODESET_LOCK_ALL_END(dev, ctx, ret); - - return ret; -} - -void neo_dcp_handle_hotplug_actions(struct neo_apple_dcp *neo_dcp, unsigned int action) -{ - if (action & DCP_HOTPLUG_VBLANK) - schedule_work(&neo_dcp->vblank_wq); - if ((action & DCP_HOTPLUG_NOTIFY) && neo_dcp->connector) - schedule_work(&neo_dcp->connector->hotplug_wq); -} - -void neo_dcp_retrain_oob(struct neo_apple_connector *connector) -{ - struct platform_device *pdev = READ_ONCE(connector->neo_dcp); - struct neo_apple_dcp *neo_dcp; - - /* a Type-C port with no pipeline behind it has nothing to retrain */ - if (!pdev || !READ_ONCE(connector->connected)) - return; - neo_dcp = platform_get_drvdata(pdev); - - /* - * Bringing up another high-speed Type-C route can disturb an active DPTX - * stream without changing its HPD state. Invalidate the IOMFB mode and - * use the normal hotplug worker to replay the active CRTC from process - * context; sending a synthetic disconnect would tear down the connector. - */ - neo_dcp_mode_invalidate(&neo_dcp->mode_state); - schedule_work(&connector->hotplug_wq); -} - -void neo_dcp_hotplug(struct work_struct *work) -{ - struct neo_apple_connector *connector; - struct platform_device *pdev; - struct neo_apple_dcp *neo_dcp; - int ret; - - connector = container_of(work, struct neo_apple_connector, hotplug_wq); - - pdev = READ_ONCE(connector->neo_dcp); - if (!pdev) { /* a Type-C port unrouted after this was queued */ - drm_kms_helper_connector_hotplug_event(&connector->base); - return; - } - neo_dcp = platform_get_drvdata(pdev); - dev_info(neo_dcp->dev, "%s() connected:%d valid_mode:%d nr_modes:%u\n", __func__, - connector->connected, READ_ONCE(neo_dcp->mode_state.valid), neo_dcp->nr_modes); - - if (!connector->connected) { - drm_edid_free(connector->drm_edid); - connector->drm_edid = NULL; - } - - /* - * DCP defers link training until we set a display mode. But we set - * display modes from atomic_flush, so userspace needs to trigger a - * flush, or the CRTC gets no signal. - */ - if (connector->base.state && !READ_ONCE(neo_dcp->mode_state.valid) && connector->connected && - !(neo_dcp_is_usb4_output(neo_dcp) && false)) { - drm_connector_set_link_status_property(&connector->base, - DRM_MODE_LINK_STATUS_BAD); - - /* - * A short Type-C route interruption can leave the DRM CRTC active - * while DCP has discarded its display mode. Userspace is then free - * to keep submitting plane-only commits, none of which retrains the - * link. Re-apply the active CRTC state once the DPTX link-config - * callback has completed. - */ - ret = neo_dcp_retrain_active_crtc(connector); - if (ret) - dev_warn(neo_dcp->dev, - "failed to retrain active CRTC after hotplug: %d\n", - ret); - } - - drm_kms_helper_connector_hotplug_event(&connector->base); -} - -static void dcpep_handle_cb(struct neo_apple_dcp *neo_dcp, enum neo_dcp_context_id context, - void *data, u32 length, u16 offset) -{ - struct device *dev = neo_dcp->dev; - struct neo_dcp_packet_header *hdr = data; - void *in, *out; - int tag; - struct neo_dcp_channel *ch = neo_dcp_get_channel(neo_dcp, context); - u8 depth; - bool handled; - - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - (length < sizeof(*hdr) || !ch || ch->depth >= DCP_MAX_CALL_DEPTH || - offset >= 0x8000 || length > 0x8000 - offset)) { - dev_err(dev, "invalid IOMFB callback envelope\n"); - neo_dcp->crashed = true; - return; - } - - tag = neo_dcp_parse_tag(hdr->tag); - handled = tag >= 0 && tag < IOMFB_MAX_CB && neo_dcp->cb_handlers && - neo_dcp->cb_handlers[tag]; - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - (tag < 0 || tag >= IOMFB_MAX_CB || - (u64)sizeof(*hdr) + hdr->in_len + hdr->out_len != length || - (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - handled && - !neo_iomfb_check_callback_h17p(neo_dcp, tag, hdr->in_len, hdr->out_len)))) { - dev_err(dev, "unqualified IOMFB callback %d (%u, %u)\n", - tag, hdr->in_len, hdr->out_len); - neo_dcp->crashed = true; - return; - } - - in = data + sizeof(*hdr); - out = in + hdr->in_len; - - if (!handled) { - if (!neo_iomfb_uses_queue(neo_dcp) || - !test_and_set_bit(tag, neo_dcp->unknown_callbacks)) - dev_warn(dev, "received unknown callback %c%c%c%c\n", - hdr->tag[3], hdr->tag[2], hdr->tag[1], hdr->tag[0]); - /* - * Leaving a callback unanswered wedges the coprocessor: it - * waits for the ack forever and the outer call never returns. - * On H17P answer it with a zeroed output instead. - */ - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P) { - if (hdr->out_len) - memset(out, 0, hdr->out_len); - depth = neo_dcp_push_depth(&ch->depth); - ch->output[depth] = out; - ch->end[depth] = offset + - ALIGN(length, DCP_PACKET_ALIGNMENT); - neo_dcp_ack(neo_dcp, context); - } - return; - } - - // TODO: verify that in_len and out_len match our prototypes - // for now just clear the out data to have at least consistent results - if (hdr->out_len) - memset(out, 0, hdr->out_len); - - depth = neo_dcp_push_depth(&ch->depth); - ch->output[depth] = out; - ch->end[depth] = offset + ALIGN(length, DCP_PACKET_ALIGNMENT); - - if (neo_dcp->cb_handlers[tag](neo_dcp, tag, out, in)) - neo_dcp_ack(neo_dcp, context); -} - -static void dcpep_handle_ack(struct neo_apple_dcp *neo_dcp, enum neo_dcp_context_id context) -{ - struct neo_dcp_channel *ch = neo_dcp_get_channel(neo_dcp, context); - const struct neo_dcp_packet_header *record, *sent; - void *cookie, *out; - neo_dcp_callback_t cb; - - if (!ch) { - dev_warn(neo_dcp->dev, "ignoring ack on context %X\n", context); - return; - } - - if (!ch->depth) { - dev_warn(neo_dcp->dev, "ignoring ack on idle context %X\n", context); - return; - } - - neo_dcp_pop_depth(&ch->depth); - - cb = ch->callbacks[ch->depth]; - cookie = ch->cookies[ch->depth]; - out = ch->output[ch->depth]; - sent = &ch->header[ch->depth]; - - ch->callbacks[ch->depth] = NULL; - ch->cookies[ch->depth] = NULL; - ch->output[ch->depth] = NULL; - - /* - * An ack completes the innermost pending command on its context. The - * firmware writes the reply in place into that AP command record, whose - * address dcp_push() saved; the ack itself carries neither offset nor - * length on any firmware. A command sent from within a callback is - * acked on the callback context, whose receive area holds the callback - * record rather than the reply, so the ack cannot be used to find it. - * - * On older firmware the lengths in the record header must still be - * the ones that were sent; otherwise the record was overwritten and - * its output cannot be trusted. The tag is left out of the check: it - * has only been seen preserved on 13.5 firmware. - */ - if (neo_dcp->fw_compat != DCP_FIRMWARE_H17P) { - record = out - sent->in_len - sizeof(*record); - if (record->in_len != sent->in_len || - record->out_len != sent->out_len) { - dev_err(neo_dcp->dev, - "corrupt %c%c%c%c command record on context %X\n", - sent->tag[3], sent->tag[2], sent->tag[1], - sent->tag[0], context); - WRITE_ONCE(neo_dcp->crashed, true); - /* - * Still complete the command, without an output, so - * that its callback drops the cookie reference and - * wakes the waiter instead of leaving it to time out. - */ - out = NULL; - } - } - - if (cb) - cb(neo_dcp, out, cookie); -} - -static void dcpep_got_msg(struct neo_apple_dcp *neo_dcp, u64 message) -{ - enum neo_dcp_context_id ctx_id; - u16 offset; - u32 length; - int channel_offset; - void *data; - - ctx_id = FIELD_GET(IOMFB_MSG_CONTEXT, message); - offset = FIELD_GET(IOMFB_MSG_OFFSET, message); - length = FIELD_GET(IOMFB_MSG_LENGTH, message); - - channel_offset = neo_dcp_channel_offset(ctx_id); - - if (channel_offset < 0) { - dev_warn(neo_dcp->dev, "invalid context received %u\n", ctx_id); - return; - } - - data = neo_dcp->shmem + channel_offset + offset; - - if (FIELD_GET(IOMFB_MSG_ACK, message)) - dcpep_handle_ack(neo_dcp, ctx_id); - else - dcpep_handle_cb(neo_dcp, ctx_id, data, length, offset); -} - -int neo_dcp_get_modes(struct drm_connector *connector) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(connector); - struct platform_device *pdev = READ_ONCE(neo_apple_connector->neo_dcp); - struct neo_apple_dcp *neo_dcp; - - struct drm_device *dev = connector->dev; - struct drm_display_mode *mode; - u16 min_vfreq = 0, max_vfreq = 0; - bool vrr_capable = false; - int i; - - /* A Type-C port has no pipeline while the fabric is moving it. */ - if (!pdev) - return 0; - neo_dcp = platform_get_drvdata(pdev); - - for (i = 0; i < neo_dcp->nr_modes; ++i) { - if (neo_dcp->modes[i].vrr) { - u16 lo = neo_dcp->modes[i].min_vrr >> 16; - u16 hi = neo_dcp->modes[i].max_vrr >> 16; - - if (!min_vfreq || lo < min_vfreq) - min_vfreq = lo; - if (hi > max_vfreq) - max_vfreq = hi; - } - vrr_capable |= neo_dcp->modes[i].vrr; - mode = drm_mode_duplicate(dev, &neo_dcp->modes[i].mode); - - if (!mode) { - dev_err(dev->dev, "Failed to duplicate display mode\n"); - return 0; - } - - drm_mode_probed_add(connector, mode); - } - drm_connector_set_vrr_capable_property(connector, vrr_capable); - - /* H17P sends no EPIC commands; see afk_send_epic(). */ - if (neo_dcp->nr_modes && neo_dcp->fw_compat != DCP_FIRMWARE_H17P && - !neo_apple_connector->drm_edid) { - const struct drm_edid *edid; - edid = neo_dcpavserv_copy_edid(neo_dcp); - if (IS_ERR_OR_NULL(edid)) { - /* - * An internal panel has no AV service and so no EDID; - * do not report that on every mode probe. - */ - if (neo_dcp_has_panel(neo_dcp) && PTR_ERR(edid) == -ENODEV) - dev_dbg(neo_dcp->dev, "no EDID source for the panel\n"); - else - dev_info(neo_dcp->dev, "copy_edid failed: %pe\n", edid); - } else { - drm_edid_free(neo_apple_connector->drm_edid); - neo_apple_connector->drm_edid = edid; - } - } - if (neo_dcp->nr_modes && neo_apple_connector->drm_edid) { - drm_edid_connector_update(connector, neo_apple_connector->drm_edid); - neo_dcp_retry_placeholder_edid(neo_dcp, neo_apple_connector->drm_edid); - } - - /* - * An internal panel has no EDID, so nothing fills in the refresh range - * that userspace needs before it will drive VRR. Supply the range DCP - * reported for the mode. This has to follow the EDID update, which - * resets display_info. - */ - if (vrr_capable && max_vfreq && - !connector->display_info.monitor_range.max_vfreq) { - connector->display_info.monitor_range.min_vfreq = min_vfreq; - connector->display_info.monitor_range.max_vfreq = max_vfreq; - } - - return neo_dcp->nr_modes; -} - -/* The user may own drm_display_mode, so we need to search for our copy */ -struct neo_dcp_display_mode *neo_lookup_mode(struct neo_apple_dcp *neo_dcp, - const struct drm_display_mode *mode) -{ - int i; - - for (i = 0; i < neo_dcp->nr_modes; ++i) { - if (drm_mode_match(mode, &neo_dcp->modes[i].mode, - DRM_MODE_MATCH_TIMINGS | - DRM_MODE_MATCH_CLOCK)) - return &neo_dcp->modes[i]; - } - - return NULL; -} - -/* - * H17P keeps the mode the bootloader programmed and never sends - * set_digital_out_mode (see DCP_INHERIT_BOOT_MODE in iomfb_template.c), so a - * modeset to any other mode would silently not be applied. The bootloader - * brings the panel up in its native timing, which is the mode the firmware - * scores highest and enumerate_modes() marks preferred; offer only that one. - */ -static bool neo_dcp_mode_settable(struct neo_apple_dcp *neo_dcp, - const struct drm_display_mode *mode) -{ - struct neo_dcp_display_mode *neo_dcp_mode = neo_lookup_mode(neo_dcp, mode); - - if (!neo_dcp_mode) - return false; - - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P) - return neo_dcp_mode->mode.type & DRM_MODE_TYPE_PREFERRED; - - return true; -} - -enum drm_mode_status neo_dcp_mode_valid(struct drm_connector *connector, - const struct drm_display_mode *mode) -{ - struct neo_apple_connector *neo_apple_connector = to_apple_connector(connector); - struct platform_device *pdev = READ_ONCE(neo_apple_connector->neo_dcp); - struct neo_apple_dcp *neo_dcp; - - if (!pdev) - return MODE_ERROR; - neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp_mode_settable(neo_dcp, mode) ? MODE_OK : MODE_BAD; -} - -int neo_dcp_crtc_atomic_modeset(struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct neo_apple_crtc *neo_apple_crtc = to_apple_crtc(crtc); - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(neo_apple_crtc->neo_dcp); - struct drm_crtc_state *crtc_state; - unsigned int action; - int ret = -EIO; - bool modeset; - - crtc_state = drm_atomic_get_new_crtc_state(state, crtc); - if (!crtc_state) - return 0; - - modeset = drm_atomic_crtc_needs_modeset(crtc_state) || - !READ_ONCE(neo_dcp->mode_state.valid); - - if (!modeset) - return 0; - - /* ignore no mode, poweroff is handled elsewhere */ - if (crtc_state->mode.hdisplay == 0 && crtc_state->mode.vdisplay == 0) - return 0; - - neo_dcp_mode_begin(&neo_dcp->mode_state); - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - ret = neo_iomfb_modeset_v12_3(neo_dcp, crtc_state); - break; - case DCP_FIRMWARE_V_13_5: - ret = neo_iomfb_modeset_v13_3(neo_dcp, crtc_state); - break; - case DCP_FIRMWARE_H17P: - ret = neo_iomfb_modeset_h17p(neo_dcp, crtc_state); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", - neo_dcp->fw_compat); - break; - } - - action = neo_dcp_mode_finish(&neo_dcp->mode_state, !ret, neo_dcp->connector ? - &neo_dcp->connector->connected : NULL); - neo_dcp_handle_hotplug_actions(neo_dcp, action); - - return ret; -} - -bool neo_dcp_crtc_mode_fixup(struct drm_crtc *crtc, - const struct drm_display_mode *mode, - struct drm_display_mode *adjusted_mode) -{ - struct neo_apple_crtc *neo_apple_crtc = to_apple_crtc(crtc); - struct platform_device *pdev = neo_apple_crtc->neo_dcp; - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - /* TODO: support synthesized modes through scaling */ - return neo_dcp_mode_settable(neo_dcp, mode); -} - - -static void neo_iomfb_scanout_release_h17p(struct neo_iomfb_scanout_h17p *scanout) -{ - unsigned int i; - - if (!scanout) - return; - for (i = 0; i < SWAP_SURFACES; i++) - if (scanout->fb[i]) - drm_framebuffer_put(scanout->fb[i]); - kfree(scanout); -} - -static struct neo_iomfb_scanout_h17p * -neo_iomfb_scanout_prepare_h17p(struct neo_apple_dcp *neo_dcp, struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct neo_iomfb_scanout_h17p *scanout; - struct drm_plane *plane; - struct drm_plane_state *old_state, *new_state; - unsigned int i, slot; - int index; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - scanout = kzalloc_obj(*scanout); - if (!scanout) - return NULL; - if (neo_dcp->neo_iomfb.scanout) { - scanout->request = neo_dcp->neo_iomfb.scanout->request; - for (i = 0; i < SWAP_SURFACES; i++) { - scanout->fb[i] = neo_dcp->neo_iomfb.scanout->fb[i]; - if (scanout->fb[i]) - drm_framebuffer_get(scanout->fb[i]); - } - } - - for_each_oldnew_plane_in_state(state, plane, old_state, new_state, index) { - if (old_state->crtc != crtc && new_state->crtc != crtc) - continue; - slot = to_apple_plane(plane)->neo_iomfb_surf; - if (slot >= SWAP_SURFACES) { - neo_iomfb_scanout_release_h17p(scanout); - return NULL; - } - if (scanout->fb[slot]) - drm_framebuffer_put(scanout->fb[slot]); - scanout->fb[slot] = NULL; - if (new_state->crtc == crtc && new_state->visible && new_state->fb) { - scanout->fb[slot] = new_state->fb; - drm_framebuffer_get(new_state->fb); - } - } - return scanout; -} - -void neo_iomfb_scanout_complete_h17p(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_scanout_h17p *scanout = neo_dcp->neo_iomfb.next_scanout; - struct neo_iomfb_scanout_h17p *previous = neo_dcp->neo_iomfb.scanout; - struct neo_dcp_swap_submit_req_h17p *request = &neo_dcp->swap.h17p; - unsigned int i; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (!scanout) - return; - - /* Keep common present arguments, including the null output pointers. */ - scanout->request = *request; - /* Preserve unchanged planes when only a subset was presented. */ - for (i = 0; i < SWAP_SURFACES; i++) { - if (!(request->swap.swap_enabled & BIT(i)) && previous) { - scanout->request.surf[i] = previous->request.surf[i]; - scanout->request.surf_iova[i] = previous->request.surf_iova[i]; - scanout->request.swap.src_rect[i] = previous->request.swap.src_rect[i]; - scanout->request.swap.dst_rect[i] = previous->request.swap.dst_rect[i]; - scanout->request.swap.surf_ids[i] = previous->request.swap.surf_ids[i]; - scanout->request.swap.surf_flags[i] = previous->request.swap.surf_flags[i]; - scanout->request.swap.surf_unk[i] = previous->request.swap.surf_unk[i]; - } - scanout->request.surf_null[i] = !scanout->fb[i]; - } - /* Brightness re-presents all pinned surfaces with a fresh swap ID. */ - /* Replay only a background established by a completed present. */ - scanout->request.swap.swap_enabled = - request->swap.swap_enabled & IOMFB_SET_BACKGROUND; - if (previous) - scanout->request.swap.swap_enabled |= - previous->request.swap.swap_enabled & IOMFB_SET_BACKGROUND; - for (i = 0; i < SWAP_SURFACES; i++) - if (scanout->fb[i]) - scanout->request.swap.swap_enabled |= BIT(i); - scanout->request.swap.swap_completed = scanout->request.swap.swap_enabled; - if (!(request->swap.swap_enabled & IOMFB_SET_BACKGROUND) && previous) - scanout->request.swap.bg_color = previous->request.swap.bg_color; - neo_dcp->neo_iomfb.scanout = scanout; - neo_dcp->neo_iomfb.next_scanout = NULL; - neo_iomfb_scanout_release_h17p(previous); -} - -bool neo_iomfb_present_brightness_only_h17p(struct neo_apple_dcp *neo_dcp) -{ - return neo_dcp->neo_iomfb.active && neo_dcp->neo_iomfb.active->brightness_only; -} - -bool neo_iomfb_apply_backlight_h17p(struct neo_apple_dcp *neo_dcp, - const struct neo_dcp_swap_submit_req_h17p *request, - struct neo_dcp_present_h17p *wire) -{ - struct neo_iomfb_transaction *transaction = neo_dcp->neo_iomfb.active; - struct neo_dcp_backlight_present present; - bool have_surface = false; - unsigned int i; - int ret; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (!neo_dcp_backlight_active(neo_dcp)) { - if (!neo_dcp_has_panel(neo_dcp)) - return true; - if (!neo_dcp->brightness.maximum) - return false; - neo_iomfb_encode_backlight_h17p(wire, 0, - neo_dcp->brightness.maximum, false); - return true; - } - if (!transaction || !neo_dcp->brightness.maximum) - return false; - for (i = 0; i < SWAP_SURFACES; i++) - have_surface |= !request->surf_null[i]; - ret = neo_dcp_backlight_prepare(neo_dcp, have_surface, &present); - if (ret && ret != -EALREADY) - return false; - if (!ret) { - transaction->backlight = present; - transaction->backlight_reserved = true; - } - neo_iomfb_encode_backlight_h17p(wire, present.nits, - neo_dcp->brightness.maximum, !ret); - return true; -} - -void neo_iomfb_present_failed_h17p(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_transaction *transaction = neo_dcp->neo_iomfb.active; - - if (transaction && transaction->backlight_reserved) { - neo_dcp_backlight_complete(neo_dcp, transaction->backlight.sequence, false); - transaction->backlight_reserved = false; - transaction->backlight_failed = true; - } -} - -bool neo_iomfb_present_complete_h17p(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_transaction *transaction = neo_dcp->neo_iomfb.active; - - lockdep_assert_held(&neo_dcp->neo_iomfb.lock); - if (transaction) { - transaction->completed = true; - if (transaction->backlight_reserved) { - neo_dcp_backlight_complete(neo_dcp, transaction->backlight.sequence, true); - transaction->backlight_reserved = false; - } - if (transaction->brightness_only) - return false; - } - atomic_cmpxchg(&neo_dcp->neo_iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING, - IOMFB_OPAQUE_X_NEEDED); - neo_iomfb_scanout_complete_h17p(neo_dcp); - return true; -} - -static void neo_iomfb_backlight_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction) -{ - struct neo_iomfb_scanout_h17p *scanout = neo_dcp->neo_iomfb.scanout; - bool have_surface = false; - unsigned int i; - - if (!neo_dcp_backlight_pending(neo_dcp) || !READ_ONCE(neo_dcp->mode_state.valid) || !scanout || - !neo_dcp->connector || !neo_dcp->connector->connected) - return; - for (i = 0; i < SWAP_SURFACES; i++) - have_surface |= !!scanout->fb[i]; - if (!have_surface) - return; - - neo_dcp->swap.h17p = scanout->request; - neo_iomfb_present_backlight_h17p(neo_dcp); -} - -static void neo_iomfb_backlight_release(struct neo_iomfb_transaction *transaction) -{ - kfree(transaction); -} - -static void neo_iomfb_backlight_kick(struct neo_apple_dcp *neo_dcp) -{ - struct neo_iomfb_transaction *transaction; - - transaction = kzalloc_obj(*transaction); - if (!transaction) - return; - transaction->start = neo_iomfb_backlight_start; - transaction->release = neo_iomfb_backlight_release; - transaction->brightness_only = true; - if (neo_iomfb_queue(neo_dcp, transaction)) - kfree(transaction); -} - -int neo_iomfb_configure_backlight_h17p(struct neo_apple_dcp *neo_dcp, u32 maximum, - bool inherited_valid, u32 inherited, - bool default_valid, u32 default_nits) -{ - if (!neo_iomfb_uses_queue(neo_dcp)) - return -EINVAL; - return neo_dcp_backlight_configure(neo_dcp, maximum, inherited_valid, inherited, - default_valid, default_nits, neo_iomfb_backlight_kick); -} - -struct neo_iomfb_atomic_transaction { - struct neo_iomfb_transaction transaction; - struct drm_atomic_state *state; - struct drm_crtc *crtc; - struct neo_apple_dcp *neo_dcp; - struct neo_iomfb_scanout_h17p *scanout; -}; - -static void neo_iomfb_atomic_start(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_transaction *transaction) -{ - struct neo_iomfb_atomic_transaction *atomic = container_of(transaction, - struct neo_iomfb_atomic_transaction, transaction); - - if (!READ_ONCE(neo_dcp->mode_state.valid) || !neo_dcp->connector || !neo_dcp->connector->connected) { - schedule_work(&neo_dcp->vblank_wq); - return; - } - atomic->scanout = neo_iomfb_scanout_prepare_h17p(neo_dcp, atomic->crtc, atomic->state); - if (!atomic->scanout) { - WRITE_ONCE(neo_dcp->crashed, true); - schedule_work(&neo_dcp->vblank_wq); - return; - } - neo_dcp->neo_iomfb.next_scanout = atomic->scanout; - neo_iomfb_flush_h17p(neo_dcp, atomic->crtc, atomic->state); -} - -static void neo_iomfb_atomic_release(struct neo_iomfb_transaction *transaction) -{ - struct neo_iomfb_atomic_transaction *atomic = container_of(transaction, - struct neo_iomfb_atomic_transaction, transaction); - - if (atomic->scanout && atomic->neo_dcp->neo_iomfb.scanout != atomic->scanout) { - if (atomic->neo_dcp->neo_iomfb.next_scanout == atomic->scanout) - atomic->neo_dcp->neo_iomfb.next_scanout = NULL; - neo_iomfb_scanout_release_h17p(atomic->scanout); - } - drm_atomic_state_put(atomic->state); - kfree(atomic); -} - -static void neo_iomfb_queue_atomic(struct neo_apple_dcp *neo_dcp, struct drm_crtc *crtc, - struct drm_atomic_state *state) -{ - struct neo_iomfb_atomic_transaction *atomic; - - atomic = kzalloc_obj(*atomic); - if (!atomic) - goto failed; - atomic->transaction.start = neo_iomfb_atomic_start; - atomic->transaction.release = neo_iomfb_atomic_release; - atomic->state = drm_atomic_state_get(state); - atomic->crtc = crtc; - atomic->neo_dcp = neo_dcp; - if (!neo_iomfb_queue(neo_dcp, &atomic->transaction)) - return; - neo_iomfb_atomic_release(&atomic->transaction); -failed: - WRITE_ONCE(neo_dcp->crashed, true); - schedule_work(&neo_dcp->vblank_wq); -} - -void neo_dcp_flush(struct drm_crtc *crtc, struct drm_atomic_state *state) -{ - struct platform_device *pdev = to_apple_crtc(crtc)->neo_dcp; - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - if (neo_iomfb_uses_queue(neo_dcp)) { - neo_iomfb_queue_atomic(neo_dcp, crtc, state); - return; - } - - /* - * DCP does not complete swaps after a link loss. A plane-only commit - * arriving between the reconnect callback and the required modeset - * would otherwise leave an unsignalled flip event in front of the - * recovery commit. Modesets set valid_mode before reaching flush. - * - * The same applies while a connector is still disconnected: resume - * re-runs the modeset, which marks the mode valid again, before the - * firmware has reported the display back. - */ - if (!READ_ONCE(neo_dcp->mode_state.valid) || !neo_dcp->connector || !neo_dcp->connector->connected) { - schedule_work(&neo_dcp->vblank_wq); - return; - } - - if (neo_dcp_channel_busy(&neo_dcp->ch_cmd)) - { - if (!neo_dcp->ch_cmd.warned_busy) { - dev_err(neo_dcp->dev, "unexpected busy command channel\n"); - neo_dcp->ch_cmd.warned_busy = true; - } - /* HACK: issue a delayed vblank event to avoid timeouts in - * drm_atomic_helper_wait_for_vblanks(). - */ - schedule_work(&neo_dcp->vblank_wq); - return; - } else if (neo_dcp->ch_cmd.warned_busy) { - neo_dcp->ch_cmd.warned_busy = false; - } - - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_flush_v12_3(neo_dcp, crtc, state); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_flush_v13_3(neo_dcp, crtc, state); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_flush_h17p(neo_dcp, crtc, state); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } -} - -static void neo_iomfb_start(struct neo_apple_dcp *neo_dcp) -{ - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_start_v12_3(neo_dcp); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_start_v13_3(neo_dcp); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_start_h17p(neo_dcp); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } -} - -bool neo_dcp_is_initialized(struct platform_device *pdev) -{ - struct neo_apple_dcp *neo_dcp = platform_get_drvdata(pdev); - - return neo_dcp->active; -} - -static void neo_iomfb_recv_message(struct neo_apple_dcp *neo_dcp, u64 message) -{ - enum dcpep_type type = FIELD_GET(IOMFB_MESSAGE_TYPE, message); - - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && READ_ONCE(neo_dcp->crashed)) - return; - - if (type == IOMFB_MESSAGE_TYPE_INITIALIZED) { - /* - * H17P reports its interface version (bits 63:48) and a - * firmware hash in the high half of the InitComplete word. - * The transport below it is unchanged. - */ - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P) - dev_info(neo_dcp->dev, - "IOMFB: init complete %#llx (version %llu)\n", - message, - FIELD_GET(GENMASK_ULL(63, 48), message)); - neo_iomfb_start(neo_dcp); - } else if (type == IOMFB_MESSAGE_TYPE_MSG) - dcpep_got_msg(neo_dcp, message); - else - dev_warn(neo_dcp->dev, "Ignoring unknown message %llx\n", message); -} - -void neo_iomfb_recv_msg(struct neo_apple_dcp *neo_dcp, u64 message) -{ - if (!neo_iomfb_uses_queue(neo_dcp)) { - neo_iomfb_recv_message(neo_dcp, message); - return; - } - - mutex_lock(&neo_dcp->neo_iomfb.lock); - WRITE_ONCE(neo_dcp->neo_iomfb.owner, current); - if (!neo_dcp->neo_iomfb.stopped) - neo_iomfb_recv_message(neo_dcp, message); - WRITE_ONCE(neo_dcp->neo_iomfb.owner, NULL); - neo_iomfb_queue_advance(neo_dcp); - mutex_unlock(&neo_dcp->neo_iomfb.lock); -} - -int neo_iomfb_start_rtkit(struct neo_apple_dcp *neo_dcp) -{ - dma_addr_t shmem_iova; - int ret; - - if (neo_iomfb_uses_queue(neo_dcp)) { - mutex_lock(&neo_dcp->neo_iomfb.lock); - if (neo_dcp->neo_iomfb.active || neo_dcp->neo_iomfb.scanout) { - mutex_unlock(&neo_dcp->neo_iomfb.lock); - return -EBUSY; - } - neo_dcp->neo_iomfb.stopped = false; - mutex_unlock(&neo_dcp->neo_iomfb.lock); - } - - /* - * H17P firmware expects the remote allocator endpoint to be started - * before IOMFB. A firmware subsystem whose endpoint was never started - * has nothing answering it on the AP side, and without this one the - * display DART faults. - */ - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P) { - ret = apple_rtkit_start_ep(neo_dcp->rtk, REMOTE_ALLOC_ENDPOINT); - if (ret) - return ret; - } - ret = apple_rtkit_start_ep(neo_dcp->rtk, IOMFB_ENDPOINT); - if (ret) - return ret; - - neo_dcp->shmem = dma_alloc_coherent(neo_dcp->dev, DCP_SHMEM_SIZE, &shmem_iova, - GFP_KERNEL); - if (!neo_dcp->shmem) - return -ENOMEM; - - neo_dcp_send_message(neo_dcp, IOMFB_ENDPOINT, dcpep_set_shmem(shmem_iova)); - - return 0; -} - -void neo_iomfb_shutdown(struct neo_apple_dcp *neo_dcp) -{ - /* We're going down */ - neo_dcp->active = false; - neo_dcp_mode_invalidate(&neo_dcp->mode_state); - - switch (neo_dcp->fw_compat) { - case DCP_FIRMWARE_V_12_3: - neo_iomfb_shutdown_v12_3(neo_dcp); - break; - case DCP_FIRMWARE_V_13_5: - neo_iomfb_shutdown_v13_3(neo_dcp); - break; - case DCP_FIRMWARE_H17P: - neo_iomfb_shutdown_h17p(neo_dcp); - break; - default: - WARN_ONCE(true, "Unexpected firmware version: %u\n", neo_dcp->fw_compat); - break; - } -} diff --git a/drivers/gpu/drm/apple-neo/iomfb.h b/drivers/gpu/drm/apple-neo/iomfb.h deleted file mode 100644 index ef432d78cac774..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb.h +++ /dev/null @@ -1,408 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#ifndef __APPLE_DCPEP_H__ -#define __APPLE_DCPEP_H__ - -#include - -#include "version_utils.h" - -/* Fixed size of shared memory between DCP and AP */ -#define DCP_SHMEM_SIZE 0x100000 - -/* DCP message contexts */ -enum neo_dcp_context_id { - /* Callback */ - DCP_CONTEXT_CB = 0, - - /* Command */ - DCP_CONTEXT_CMD = 2, - - /* Asynchronous */ - DCP_CONTEXT_ASYNC = 3, - - /* Out-of-band callback */ - DCP_CONTEXT_OOBCB = 4, - - /* Out-of-band command */ - DCP_CONTEXT_OOBCMD = 6, - - /* Out-of-band Asynchronous */ - DCP_CONTEXT_OOBASYNC = 7, - - DCP_NUM_CONTEXTS -}; - -/* RTKit endpoint message types */ -enum dcpep_type { - /* Set shared memory */ - IOMFB_MESSAGE_TYPE_SET_SHMEM = 0, - - /* DCP is initialized */ - IOMFB_MESSAGE_TYPE_INITIALIZED = 1, - - /* Remote procedure call */ - IOMFB_MESSAGE_TYPE_MSG = 2, -}; - -/* - * IOMFB supports the setting of a number of parameters - * that alter various aspects of the connected sink's - * behaviour at runtime. - */ -enum neo_iomfb_parameter { - IOMFBPARAM_ADAPTIVE_SYNC = 14, -}; - -#define IOMFB_MESSAGE_TYPE GENMASK_ULL( 3, 0) - -/* Message */ -#define IOMFB_MSG_LENGTH GENMASK_ULL(63, 32) -#define IOMFB_MSG_OFFSET GENMASK_ULL(31, 16) -#define IOMFB_MSG_CONTEXT GENMASK_ULL(11, 8) -#define IOMFB_MSG_ACK BIT_ULL(6) - -/* Set shmem */ -#define IOMFB_SHMEM_DVA GENMASK_ULL(63, 16) -#define IOMFB_SHMEM_FLAG GENMASK_ULL( 7, 4) -#define IOMFB_SHMEM_FLAG_VALUE 4 - -struct neo_dcp_packet_header { - char tag[4]; - u32 in_len; - u32 out_len; -} __packed; - -#define DCP_IS_NULL(ptr) ((ptr) ? 1 : 0) -#define DCP_PACKET_ALIGNMENT (0x40) - -enum neo_iomfb_property_id { - IOMFB_PROPERTY_NITS = 15, // divide by Brightness_Scale -}; - -#define IOMFB_BRIGHTNESS_MIN 0x10000000 - -/* Structures used in v12.0 firmware */ - -#define SWAP_SURFACES 4 - -struct neo_dcp_iouserclient { - /* Handle for the IOUserClient. macOS sets this to a kernel VA. */ - u64 handle; - u32 unk; - u8 flag1; - u8 flag2; - u8 padding[2]; -} __packed; - -/* - * Update background color to struct dcp_swap.bg_color - */ -#define IOMFB_SET_BACKGROUND BIT(31) - -struct neo_dcp_allocate_bandwidth_req { - u64 unk1; - u64 unk2; - u64 unk3; - u8 unk1_null; - u8 unk2_null; - u8 padding[8]; -} __packed; - -struct neo_dcp_allocate_bandwidth_resp { - u64 unk1; - u64 unk2; - u32 ret; -} __packed; - -struct neo_dcp_rt_bandwidth { - u64 unk1; - u64 reg_scratch; - u64 reg_doorbell; - u32 unk2; - u32 doorbell_bit; - u32 padding[7]; -} __packed; - -struct frame_sync_props { - u8 unk[28]; -}; - -struct neo_dcp_set_frame_sync_props_req { - struct frame_sync_props props; - u8 frame_sync_props_null; - u8 padding[3]; -} __packed; - -struct neo_dcp_set_frame_sync_props_resp { - struct frame_sync_props props; -} __packed; - -/* Method calls */ - -enum dcpep_method { - dcpep_late_init_signal, - dcpep_setup_video_limits, - dcpep_set_create_dfb, - dcpep_start_signal, - dcpep_swap_start, - dcpep_swap_submit, - dcpep_set_display_device, - dcpep_set_digital_out_mode, - dcpep_create_default_fb, - dcpep_set_display_refresh_properties, - dcpep_flush_supports_power, - dcpep_set_power_state, - dcpep_first_client_open, - dcpep_register_dfb_surface, - dcpep_set_parameter_dcp, - dcpep_enable_disable_video_power_savings, - dcpep_is_main_display, - iomfbep_a131_pmu_service_matched, - iomfbep_a132_backlight_service_matched, - iomfbep_a358_vi_set_temperature_hint, - iomfbep_get_color_remap_mode, - iomfbep_last_client_close, - iomfbep_abort_swaps_dcp, - iomfbep_set_matrix, - /* - * H17P default-framebuffer queries, issued from inside - * start_hardware_boot (D121) between create_default_fb and - * setup_video_limits. - */ - dcpep_get_dfb_compression_info, - dcpep_get_dfb_info, - dcpep_get_dfb_layout, - dcpep_get_dfb_state, - dcpep_set_dfb_dimensions, - dcpep_commit_dfb_info, - dcpep_dfb_query, - dcpep_dfb_ready_query, - /* - * H17P pipe-configuration calls made before the first swap. Most - * names are unknown; they are identified by method number. - */ - dcpep_pipe_cfg_415, - dcpep_pipe_cfg_031, - dcpep_pipe_cfg_414, - dcpep_pipe_query_478, - dcpep_pipe_query_474, - dcpep_pipe_set_476, - dcpep_pipe_set_428, - dcpep_num_methods -}; - -#define IOMFB_METHOD(tag, name) [name] = { #name, { tag[0], tag[1], tag[2], tag[3] } } -#define IOMFB_METHOD_H17(tag, h17g, name) [name] = { #name, \ - { tag[0], tag[1], tag[2], tag[3] }, \ - { h17g[0], h17g[1], h17g[2], h17g[3] } } - -struct neo_dcp_method_entry { - const char *name; - char tag[4]; - /* H17G shares the H17P transport but changes selected method IDs. */ - char tag_h17g[4]; -}; - -#define IOMFB_MAX_CB (1000) - -bool neo_iomfb_validate_callback_h17p(int tag, u32 in_len, u32 out_len); -struct neo_apple_dcp; -bool neo_iomfb_check_callback_h17p(struct neo_apple_dcp *neo_dcp, int tag, u32 in_len, - u32 out_len); - -typedef bool (*neo_iomfb_cb_handler)(struct neo_apple_dcp *, int, void *, void *); - -/* Prototypes */ - -struct neo_dcp_set_digital_out_mode_req { - u32 color_mode_id; - u32 timing_mode_id; -} __packed; - -struct neo_dcp_map_buf_req { - u64 buffer; - u8 unk; - u8 buf_null; - u8 vaddr_null; - u8 dva_null; -} __packed; - -struct neo_dcp_unmap_buf_resp { - u64 buffer; - u64 vaddr; - u64 dva; - u8 unk; - u8 buf_null; -} __packed; - -struct neo_dcp_allocate_buffer_req { - u32 unk0; - u64 size; - u32 unk2; - u8 paddr_null; - u8 dva_null; - u8 dva_size_null; - u8 padding; -} __packed; - -struct neo_dcp_allocate_buffer_resp { - u64 paddr; - u64 dva; - u64 dva_size; - u32 mem_desc_id; -} __packed; - -struct neo_dcp_map_physical_req { - u64 paddr; - u64 size; - u32 flags; - u8 dva_null; - u8 dva_size_null; - u8 padding[2]; -} __packed; - -struct neo_dcp_map_physical_resp { - u64 dva; - u64 dva_size; - u32 mem_desc_id; -} __packed; - -struct neo_dcp_get_uint_prop_req { - char obj[4]; - char key[0x40]; - u64 value; - u8 value_null; - u8 padding[3]; -} __packed; - -struct neo_dcp_get_uint_prop_resp { - u64 value; - u8 ret; - u8 padding[3]; -} __packed; - -struct neo_iomfb_sr_set_property_int_req { - char obj[4]; - char key[0x40]; - u64 value; - u8 value_null; - u8 padding[3]; -} __packed; - -struct neo_iomfb_set_fx_prop_req { - char obj[4]; - char key[0x40]; - u32 value; -} __packed; - -struct neo_dcp_set_power_state_resp { - u32 unkint; - u32 ret; -} __packed; - -struct neo_dcp_set_dcpav_prop_chunk_req { - char data[0x1000]; - u32 offset; - u32 length; -} __packed; - -struct neo_dcp_set_dcpav_prop_end_req { - char key[0x40]; -} __packed; - -struct neo_dcp_set_parameter_dcp { - u32 param; - u32 value[8]; - u32 count; -} __packed; - -struct neo_dcp_swap_complete_intent_gated { - u32 swap_id; - u8 unkBool; - u32 unkInt; - u32 width; - u32 height; -} __packed; - -struct neo_dcp_read_edt_data_req { - char key[0x40]; - u32 count; - u32 value[8]; -} __packed; - -struct neo_dcp_read_edt_data_resp { - u32 value[8]; - u8 ret; -} __packed; - -struct neo_iomfb_property { - u32 id; - u32 value; -} __packed; - -struct neo_iomfb_get_color_remap_mode_req { - u32 mode; - u8 mode_null; - u8 padding[3]; -} __packed; - -struct neo_iomfb_get_color_remap_mode_resp { - u32 mode; - u32 ret; -} __packed; - -struct neo_iomfb_last_client_close_req { - u8 unkint_null; - u8 padding[3]; -} __packed; - -struct neo_iomfb_last_client_close_resp { - u32 unkint; -} __packed; - -struct io_user_client { - u64 addr; - u32 unk; - u8 flag1; - u8 flag2; - u8 pad[2]; -} __packed; - -struct neo_iomfb_abort_swaps_dcp_req { - struct io_user_client client; - u8 client_null; - u8 pad[3]; -} __packed; - -struct neo_iomfb_abort_swaps_dcp_resp { - struct io_user_client client; - u32 ret; -} __packed; - -struct neo_iomfb_set_matrix_req { - u32 location; - u64 matrix[9]; - u8 matrix_null; - u8 padding[3]; -} __packed; - -struct neo_iomfb_set_matrix_resp { - u32 ret; -} __packed; - -struct dcpep_get_tiling_state_req { - u32 event; - u32 param; - u32 value; - u8 value_null; - u8 padding[3]; -} __packed; - -struct dcpep_get_tiling_state_resp { - u32 value; - u32 ret; -} __packed; - -#endif diff --git a/drivers/gpu/drm/apple-neo/iomfb_internal.h b/drivers/gpu/drm/apple-neo/iomfb_internal.h deleted file mode 100644 index 65a564f3a32b8e..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_internal.h +++ /dev/null @@ -1,114 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#include -#include - -#include "dcp-internal.h" - -struct neo_apple_dcp; - -typedef void (*neo_dcp_callback_t)(struct neo_apple_dcp *, void *, void *); - - -#define DCP_THUNK_VOID(func, handle) \ - static void func(struct neo_apple_dcp *neo_dcp, bool oob, neo_dcp_callback_t cb, \ - void *cookie) \ - { \ - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[handle], 0, 0, NULL, cb, cookie); \ - } - -#define DCP_THUNK_OUT(func, handle, T) \ - static void func(struct neo_apple_dcp *neo_dcp, bool oob, neo_dcp_callback_t cb, \ - void *cookie) \ - { \ - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[handle], 0, sizeof(T), NULL, cb, cookie); \ - } - -#define DCP_THUNK_IN(func, handle, T) \ - static void func(struct neo_apple_dcp *neo_dcp, bool oob, T *data, \ - neo_dcp_callback_t cb, void *cookie) \ - { \ - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[handle], sizeof(T), 0, data, cb, cookie); \ - } - -#define DCP_THUNK_INOUT(func, handle, T_in, T_out) \ - static void func(struct neo_apple_dcp *neo_dcp, bool oob, T_in *data, \ - neo_dcp_callback_t cb, void *cookie) \ - { \ - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[handle], sizeof(T_in), sizeof(T_out), data, \ - cb, cookie); \ - } - -#define IOMFB_THUNK_INOUT(name) \ - static void neo_iomfb_ ## name(struct neo_apple_dcp *neo_dcp, bool oob, \ - struct neo_iomfb_ ## name ## _req *data, \ - neo_dcp_callback_t cb, void *cookie) \ - { \ - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[iomfbep_ ## name], \ - sizeof(struct neo_iomfb_ ## name ## _req), \ - sizeof(struct neo_iomfb_ ## name ## _resp), \ - data, cb, cookie); \ - } - -/* - * Define type-safe trampolines. Define typedefs to enforce type-safety on the - * input data (so if the types don't match, gcc errors out). - */ - -#define TRAMPOLINE_VOID(func, handler) \ - static bool __maybe_unused func(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) \ - { \ - trace_neo_iomfb_callback(neo_dcp, tag, #handler); \ - handler(neo_dcp); \ - return true; \ - } - -#define TRAMPOLINE_IN(func, handler, T_in) \ - typedef void (*callback_##handler)(struct neo_apple_dcp *, T_in *); \ - \ - static bool __maybe_unused func(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) \ - { \ - callback_##handler cb = handler; \ - \ - trace_neo_iomfb_callback(neo_dcp, tag, #handler); \ - cb(neo_dcp, in); \ - return true; \ - } - -#define TRAMPOLINE_INOUT(func, handler, T_in, T_out) \ - typedef T_out (*callback_##handler)(struct neo_apple_dcp *, T_in *); \ - \ - static bool __maybe_unused func(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) \ - { \ - T_out *typed_out = out; \ - callback_##handler cb = handler; \ - \ - trace_neo_iomfb_callback(neo_dcp, tag, #handler); \ - *typed_out = cb(neo_dcp, in); \ - return true; \ - } - -#define TRAMPOLINE_OUT(func, handler, T_out) \ - static bool __maybe_unused func(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) \ - { \ - T_out *typed_out = out; \ - \ - trace_neo_iomfb_callback(neo_dcp, tag, #handler); \ - *typed_out = handler(neo_dcp); \ - return true; \ - } - -/* Call a DCP function given by a tag */ -void neo_dcp_push(struct neo_apple_dcp *neo_dcp, bool oob, const struct neo_dcp_method_entry *call, - u32 in_len, u32 out_len, void *data, neo_dcp_callback_t cb, - void *cookie); - -/* Parse a callback tag "D123" into the ID 123. Returns -EINVAL on failure. */ -int neo_dcp_parse_tag(char tag[4]); - -void neo_dcp_ack(struct neo_apple_dcp *neo_dcp, enum neo_dcp_context_id context); - -/* The user may own drm_display_mode, so we need to search for our copy */ -struct neo_dcp_display_mode *neo_lookup_mode(struct neo_apple_dcp *neo_dcp, - const struct drm_display_mode *mode); diff --git a/drivers/gpu/drm/apple-neo/iomfb_plane.h b/drivers/gpu/drm/apple-neo/iomfb_plane.h deleted file mode 100644 index f2a191de45bdbc..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_plane.h +++ /dev/null @@ -1,109 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * Copyright (C) The Asahi Linux Contributors - */ - - -#ifndef __APPLE_IOMFB_PLANE_H__ -#define __APPLE_IOMFB_PLANE_H__ - -#include - -#include - -#define DCP_SURF_MAX_PLANES 3 - -#define DCP_FORMAT_BGRA fourcc_code('A', 'R', 'G', 'B') -#define DCP_FORMAT_RGBA fourcc_code('A', 'B', 'G', 'R') - -#define DCP_FORMAT_W30R fourcc_code('r', '0', '3', 'w') // wide gamut packed 10-bit RGB without alpha -#define DCP_FORMAT_L10R fourcc_code('r', '0', '1', 'l') // full range packed 10-bit RGB with alpha - -#define DCP_FORMAT_420V fourcc_code('v', '0', '2', '4') // NV12 video range 2 plane 8-bit YCbCr -#define DCP_FORMAT_420F fourcc_code('f', '0', '2', '4') // NV12 full range 2 plane 8-bit YCbCr -#define DCP_FORMAT_422V fourcc_code('v', '2', '2', '4') // NV16 video range 2 plane 8-bit YCbCr -#define DCP_FORMAT_422F fourcc_code('f', '2', '2', '4') // NV16 full range 2 plane 8-bit YCbCr -#define DCP_FORMAT_444V fourcc_code('v', '4', '4', '4') // NV24 video range 2 plane 8-bit YCbCr -#define DCP_FORMAT_444F fourcc_code('f', '4', '4', '4') // NV24 full range 2 plane 8-bit YCbCr - -#define DCP_FORMAT_X420 fourcc_code('0', '2', '4', 'x') // P010 video range 2 plane 10-bit YCbCR -#define DCP_FORMAT_X422 fourcc_code('2', '2', '4', 'x') // P210 video range 2 plane 10-bit YCbCR -#define DCP_FORMAT_X444 fourcc_code('4', '4', '4', 'x') // P410 video range 2 plane 10-bit YCbCR - -#define DCP_FORMAT_XF20 fourcc_code('0', '2', 'f', 'x') // P010 full range 2 plane 10-bit YCbCR -#define DCP_FORMAT_XF22 fourcc_code('2', '2', 'f', 'x') // P210 full range 2 plane 10-bit YCbCR -#define DCP_FORMAT_XF44 fourcc_code('4', '4', 'f', 'x') // P410 full range 2 plane 10-bit YCbCR - -enum neo_dcp_colorspace { - DCP_COLORSPACE_BG_SRGB = 0, - DCP_COLORSPACE_BT601 = 1, - DCP_COLORSPACE_BT709 = 2, - DCP_COLORSPACE_BG_BT2020 = 9, - DCP_COLORSPACE_NATIVE = 12, -}; - -enum neo_dcp_xfer_func { - DCP_XFER_FUNC_BT601 = 1, - DCP_XFER_FUNC_BT1886 = 2, - DCP_XFER_FUNC_SDR = 13, - DCP_XFER_FUNC_HDR = 16, -}; - -struct neo_dcp_rect { - u32 x; - u32 y; - u32 w; - u32 h; -} __packed; - -/* Information describing a plane of a planar compressed surface */ -struct neo_dcp_plane_info { - u32 width; - u32 height; - u32 base; - u32 offset; - u32 stride; - u32 size; - u16 tile_size; - u8 tile_w; - u8 tile_h; - u32 unk[13]; -} __packed; - -struct neo_dcp_component_types { - u8 count; - u8 types[7]; -} __packed; - -/* Information describing a surface */ -struct neo_dcp_surface { - u8 is_tiled; - u8 is_tearing_allowed; - u8 is_premultiplied; - u32 plane_cnt; - u32 plane_cnt2; - u32 format; /* DCP fourcc */ - u32 ycbcr_matrix; - u8 xfer_func; - u8 colorspace; - u32 stride; - u16 pix_size; - u8 pel_w; - u8 pel_h; - u32 offset; - u32 width; - u32 height; - u32 buf_size; - u64 protection_opts; - u32 surface_id; - struct neo_dcp_component_types comp_types[DCP_SURF_MAX_PLANES]; - u64 has_comp; - struct neo_dcp_plane_info planes[DCP_SURF_MAX_PLANES]; - u64 has_planes; - u32 compression_info[DCP_SURF_MAX_PLANES][13]; - u64 has_compr_info; - u32 unk_num; - u32 unk_denom; -} __packed; - -#endif /* __APPLE_IOMFB_PLANE_H__ */ diff --git a/drivers/gpu/drm/apple-neo/iomfb_template.h b/drivers/gpu/drm/apple-neo/iomfb_template.h deleted file mode 100644 index 1a2a7b7ff5cb95..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_template.h +++ /dev/null @@ -1,285 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -/* - * This file is intended to be included multiple times with IOMFB_VER - * defined to declare DCP firmware version dependent structs. - */ - -#ifdef DCP_FW_VER - -#include - -#include - -#include "iomfb.h" -#include "iomfb_plane.h" -#include "plane.h" -#include "version_utils.h" - -struct DCP_FW_NAME(neo_dcp_swap) { - /* IOMobileFramebuffer timestamp types 1 through 7, in wire order. */ - u64 timestamp[7]; - u64 unk_38; - - u64 flags1; - u64 flags2; - -#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) - /* - * H17P inserts 0x48 bytes between flags2 and the swap ID, which moves - * to +0x98. Every following field shifts with it: surf_ids +0x9c, - * src_rect +0xac, surf_flags +0xec, dst_rect +0x10c, - * swap_enabled/completed +0x14c/+0x150, bg_color +0x154. The gap holds - * two small u32 counters (at +0x50 and +0x78) that are not modelled. - */ - u8 h17p_pre_swap_id[0x48]; -#endif - - u32 swap_id; - - u32 surf_ids[SWAP_SURFACES]; - struct neo_dcp_rect src_rect[SWAP_SURFACES]; - u32 surf_flags[SWAP_SURFACES]; - u32 surf_unk[SWAP_SURFACES]; - struct neo_dcp_rect dst_rect[SWAP_SURFACES]; - u32 swap_enabled; - u32 swap_completed; - - u32 bg_color; - u8 unk_110[0x1b8]; - u32 unk_2c8; - u8 unk_2cc[0x14]; - u32 unk_2e0; -#if DCP_FW_VER < DCP_FW_VERSION(13, 2, 0) - u16 unk_2e2; -#else - u8 unk_2e2[3]; -#endif - u64 bl_unk; - u32 bl_value; // min value is 0x10000000 - u8 bl_power; // constant 0x40 for on - u8 unk_2f3[0x2d]; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u8 unk_320[0x13f]; - u64 unk_1; -#endif -#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) - /* - * The H17P A408 request is 0xe9c bytes; with a 0x22c-byte surface, - * SWAP_SURFACES = 4 and no surf2 block, that leaves this tail. - */ - u8 h17p_tail[0xda]; -#endif -} __packed; - -/* Information describing a surface */ -struct DCP_FW_NAME(neo_dcp_surface) { - struct neo_dcp_surface base; -#if DCP_FW_VER < DCP_FW_VERSION(13, 2, 0) - u8 padding[7]; -#else - u8 padding[47]; -#endif -} __packed; - -/* Prototypes */ - -struct DCP_FW_NAME(neo_dcp_swap_submit_req) { - struct DCP_FW_NAME(neo_dcp_swap) swap; - struct DCP_FW_NAME(neo_dcp_surface) surf[SWAP_SURFACES]; - u64 surf_iova[SWAP_SURFACES]; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u64 unk_u64_a[SWAP_SURFACES]; - /* - * H17P carries no surf2 block: its trailing u8 null-flag run is 8 bytes - * (swap_null + surf_null[4] + three tail bools). - */ -#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) - struct DCP_FW_NAME(neo_dcp_surface) surf2[5]; - u64 surf2_iova[5]; -#endif -#endif - u8 unkbool; - u64 unkdouble; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u64 unkU64; - u8 unkbool2; -#endif - u32 clear; // or maybe switch to default fb? -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u32 unkU32Ptr; -#endif - u8 swap_null; - u8 surf_null[SWAP_SURFACES]; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) -#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) - u8 surf2_null[5]; -#endif -#endif - u8 unkoutbool_null; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u8 unknown_pointer_null; - u8 unknown_output_null; -#endif -#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) - u8 padding[1]; -#endif -} __packed; - -struct DCP_FW_NAME(neo_dcp_swap_submit_resp) { - u8 unkoutbool; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u32 unkU32out; -#endif - u32 ret; - u8 padding[3]; -} __packed; - -/* - * A406 swap_start / A473 set_power_state. H17P shrank both requests: the - * 0x10-byte struct dcp_iouserclient became a bare u64 handle and the - * swap_start response no longer echoes it. swap_start is 0x10 in / 0x8 out, - * set_power_state 0x10 in. - */ -#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) -struct DCP_FW_NAME(neo_dcp_swap_start_req) { - u32 swap_id; - u64 client; /* IOUserClient kernel VA */ - u8 client_null; - u8 padding[3]; -} __packed; - -struct DCP_FW_NAME(neo_dcp_swap_start_resp) { - u32 swap_id; - u32 ret; -} __packed; - -struct DCP_FW_NAME(neo_dcp_set_power_state_req) { - u64 unklong; - u32 unkint; /* widened from u8 on H17P */ - u8 unkint_null; - u8 padding[3]; -} __packed; - -static_assert(sizeof(struct DCP_FW_NAME(neo_dcp_swap_start_req)) == 0x10); -static_assert(sizeof(struct DCP_FW_NAME(neo_dcp_swap_start_resp)) == 0x8); -static_assert(sizeof(struct DCP_FW_NAME(neo_dcp_set_power_state_req)) == 0x10); -#else -struct DCP_FW_NAME(neo_dcp_swap_start_req) { - u32 swap_id; - struct neo_dcp_iouserclient client; - u8 swap_id_null; - u8 client_null; - u8 padding[2]; -} __packed; - -struct DCP_FW_NAME(neo_dcp_swap_start_resp) { - u32 swap_id; - struct neo_dcp_iouserclient client; - u32 ret; -} __packed; - -struct DCP_FW_NAME(neo_dcp_set_power_state_req) { - u64 unklong; - u8 unkbool; - u8 unkint_null; - u8 padding[2]; -} __packed; -#endif - -/* - * D201 map_piodma reply. H17P declares a 16-byte output, not 20, laid out as - * { u32 buffer index, u64 dva, u32 pad } rather than { u64 vaddr, u64 dva, - * u32 ret }. With the old layout the DVA lands four bytes late and the - * display engine DMAs to a bogus address. - */ -#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) -struct DCP_FW_NAME(neo_dcp_map_buf_resp) { - u32 buffer; - u64 dva; - u32 unk; -} __packed; -static_assert(sizeof(struct DCP_FW_NAME(neo_dcp_map_buf_resp)) == 0x10); -#else -struct DCP_FW_NAME(neo_dcp_map_buf_resp) { - u64 vaddr; - u64 dva; - u32 ret; -} __packed; -#endif - -struct DCP_FW_NAME(dc_swap_complete_resp) { - u32 swap_id; /* +0x000 */ - u8 unkbool; /* +0x004 */ -#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) - /* - * H17P: swap_complete_ap_gated(u32, bool, SwapCompleteData const *, - * SwapInfoBlob const *, u32, bool) is 0x730 bytes on the wire, 89 more - * than the pre-H17P record: the SwapCompleteData grew from 8 to 0x22, - * swap_info became eight 0xe0-byte SwapInfoBlobs rather than one - * 0x6c5 blob, and the tail gained a flag and three pad bytes. Only - * swap_info_count of those blobs are populated; the rest are - * uninitialised DCP buffer memory. - */ - u32 hol_swap_ids[4]; /* +0x005 */ - u32 hol_arg; /* +0x015 */ - u32 hol_count; /* +0x019 */ - u8 hol_valid; /* +0x01d */ - u64 displayed_data; /* +0x01e, unaligned */ - u8 notify_displayed; /* +0x026 */ - u8 swap_info[8][0xe0]; /* +0x027, blob[0]+0x38 == swap_id */ - u32 swap_info_count; /* +0x727, max 8 */ - u8 completion_flag; /* +0x72b */ - u8 swap_data_null; /* +0x72c, 0 => data at +0x005 valid */ - u8 padding[3]; /* +0x72d, to sizeof == 0x730 */ -#else - u64 swap_data; -#if DCP_FW_VER < DCP_FW_VERSION(13, 2, 0) - u8 swap_info[0x6c4]; -#else - u8 swap_info[0x6c5]; -#endif - u32 unkint; - u8 swap_info_null; -#endif -} __packed; - -struct DCP_FW_NAME(neo_dcp_map_reg_req) { - char obj[4]; - u32 index; - u32 flags; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u8 unk_u64_null; -#endif - u8 addr_null; - u8 length_null; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u8 padding[1]; -#else - u8 padding[2]; -#endif -} __packed; - -struct DCP_FW_NAME(neo_dcp_map_reg_resp) { -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - u64 dva; -#endif - u64 addr; - u64 length; - u32 ret; -} __packed; - - -struct neo_apple_dcp; - -int DCP_FW_NAME(neo_iomfb_modeset)(struct neo_apple_dcp *neo_dcp, - struct drm_crtc_state *crtc_state); -void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc *crtc, struct drm_atomic_state *state); -void DCP_FW_NAME(neo_iomfb_poweron)(struct neo_apple_dcp *neo_dcp); -void DCP_FW_NAME(neo_iomfb_poweroff)(struct neo_apple_dcp *neo_dcp); -void DCP_FW_NAME(neo_iomfb_sleep)(struct neo_apple_dcp *neo_dcp); -void DCP_FW_NAME(neo_iomfb_start)(struct neo_apple_dcp *neo_dcp); -void DCP_FW_NAME(neo_iomfb_shutdown)(struct neo_apple_dcp *neo_dcp); - -#endif diff --git a/drivers/gpu/drm/apple-neo/iomfb_v12_3.c b/drivers/gpu/drm/apple-neo/iomfb_v12_3.c deleted file mode 100644 index 52036ead11de3c..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_v12_3.c +++ /dev/null @@ -1,109 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#include "iomfb_v12_3.h" -#include "iomfb_v13_3.h" -#include "iomfb_h17p.h" -#include "version_utils.h" - -static const struct neo_dcp_method_entry neo_dcp_methods[dcpep_num_methods] = { - IOMFB_METHOD("A000", dcpep_late_init_signal), - IOMFB_METHOD("A029", dcpep_setup_video_limits), - IOMFB_METHOD("A131", iomfbep_a131_pmu_service_matched), - IOMFB_METHOD("A132", iomfbep_a132_backlight_service_matched), - IOMFB_METHOD("A357", dcpep_set_create_dfb), - IOMFB_METHOD("A358", iomfbep_a358_vi_set_temperature_hint), - IOMFB_METHOD("A401", dcpep_start_signal), - IOMFB_METHOD("A407", dcpep_swap_start), - IOMFB_METHOD("A408", dcpep_swap_submit), - IOMFB_METHOD("A410", dcpep_set_display_device), - IOMFB_METHOD("A411", dcpep_is_main_display), - IOMFB_METHOD("A412", dcpep_set_digital_out_mode), - IOMFB_METHOD("A422", iomfbep_set_matrix), - IOMFB_METHOD("A426", iomfbep_get_color_remap_mode), - IOMFB_METHOD("A439", dcpep_set_parameter_dcp), - IOMFB_METHOD("A443", dcpep_create_default_fb), - IOMFB_METHOD("A447", dcpep_enable_disable_video_power_savings), - IOMFB_METHOD("A454", dcpep_first_client_open), - IOMFB_METHOD("A455", iomfbep_last_client_close), - IOMFB_METHOD("A460", dcpep_set_display_refresh_properties), - IOMFB_METHOD("A463", dcpep_flush_supports_power), - IOMFB_METHOD("A464", iomfbep_abort_swaps_dcp), - IOMFB_METHOD("A468", dcpep_set_power_state), -}; - -#define DCP_FW v12_3 -#define DCP_FW_VER DCP_FW_VERSION(12, 3, 0) - -#include "iomfb_template.c" - -static const neo_iomfb_cb_handler cb_handlers[IOMFB_MAX_CB] = { - [0] = trampoline_true, /* did_boot_signal */ - [1] = trampoline_true, /* did_power_on_signal */ - [2] = trampoline_nop, /* will_power_off_signal */ - [3] = trampoline_rt_bandwidth, - [100] = iomfbep_cb_match_pmu_service, - [101] = trampoline_zero, /* get_display_default_stride */ - [102] = trampoline_nop, /* set_number_property */ - [103] = trampoline_nop, /* set_boolean_property */ - [106] = trampoline_nop, /* remove_property */ - [107] = trampoline_true, /* create_provider_service */ - [108] = trampoline_true, /* create_product_service */ - [109] = trampoline_true, /* create_pmu_service */ - [110] = trampoline_true, /* create_iomfb_service */ - [111] = trampoline_create_backlight_service, - [116] = dcpep_cb_boot_1, - [117] = trampoline_false, /* is_dark_boot */ - [118] = trampoline_false, /* is_dark_boot / is_waking_from_hibernate*/ - [120] = trampoline_read_edt_data, - [122] = trampoline_prop_start, - [123] = trampoline_prop_chunk, - [124] = trampoline_prop_end, - [201] = trampoline_map_piodma, - [202] = trampoline_unmap_piodma, - [206] = iomfbep_cb_match_pmu_service_2, - [207] = iomfbep_cb_match_backlight_service, - [208] = trampoline_get_time, - [211] = trampoline_nop, /* update_backlight_factor_prop */ - [300] = trampoline_pr_publish, - [401] = trampoline_get_uint_prop, - [404] = trampoline_nop, /* sr_set_uint_prop */ - [406] = trampoline_set_fx_prop, - [408] = trampoline_get_frequency, - [411] = trampoline_map_reg, - [413] = trampoline_true, /* sr_set_property_dict */ - [414] = trampoline_sr_set_property_int, - [415] = trampoline_true, /* sr_set_property_bool */ - [451] = trampoline_allocate_buffer, - [452] = trampoline_map_physical, - [456] = trampoline_release_mem_desc, - [552] = trampoline_true, /* set_property_dict_0 */ - [561] = trampoline_true, /* set_property_dict */ - [563] = trampoline_true, /* set_property_int */ - [565] = trampoline_true, /* set_property_bool */ - [567] = trampoline_true, /* set_property_str */ - [574] = trampoline_zero, /* power_up_dart */ - [576] = trampoline_hotplug, - [577] = trampoline_nop, /* powerstate_notify */ - [582] = trampoline_true, /* create_default_fb_surface */ - [584] = trampoline_nop, /* IOMobileFramebufferAP::clear_default_surface */ - [588] = trampoline_nop, /* resize_default_fb_surface_gated */ - [589] = trampoline_swap_complete, - [591] = trampoline_swap_complete_intent_gated, - [592] = trampoline_abort_swap_ap_gated, - [593] = trampoline_enable_backlight_message_ap_gated, - [594] = trampoline_nop, /* IOMobileFramebufferAP::setSystemConsoleMode */ - [596] = trampoline_false, /* IOMobileFramebufferAP::isDFBAllocated */ - [597] = trampoline_false, /* IOMobileFramebufferAP::preserveContents */ - [598] = trampoline_nop, /* find_swap_function_gated */ -}; - -void DCP_FW_NAME(neo_iomfb_start)(struct neo_apple_dcp *neo_dcp) -{ - neo_dcp->cb_handlers = cb_handlers; - - neo_dcp_start_signal(neo_dcp, false, neo_dcp_started, NULL); -} - -#undef DCP_FW_VER -#undef DCP_FW diff --git a/drivers/gpu/drm/apple-neo/iomfb_v12_3.h b/drivers/gpu/drm/apple-neo/iomfb_v12_3.h deleted file mode 100644 index 7359685d981fe5..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_v12_3.h +++ /dev/null @@ -1,17 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#ifndef __APPLE_IOMFB_V12_3_H__ -#define __APPLE_IOMFB_V12_3_H__ - -#include "version_utils.h" - -#define DCP_FW v12_3 -#define DCP_FW_VER DCP_FW_VERSION(12, 3, 0) - -#include "iomfb_template.h" - -#undef DCP_FW_VER -#undef DCP_FW - -#endif /* __APPLE_IOMFB_V12_3_H__ */ diff --git a/drivers/gpu/drm/apple-neo/iomfb_v13_3.c b/drivers/gpu/drm/apple-neo/iomfb_v13_3.c deleted file mode 100644 index 0ae62cd345561a..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_v13_3.c +++ /dev/null @@ -1,114 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#include "iomfb_v12_3.h" -#include "iomfb_v13_3.h" -#include "iomfb_h17p.h" -#include "version_utils.h" - -static const struct neo_dcp_method_entry neo_dcp_methods[dcpep_num_methods] = { - IOMFB_METHOD("A000", dcpep_late_init_signal), - IOMFB_METHOD("A029", dcpep_setup_video_limits), - IOMFB_METHOD("A131", iomfbep_a131_pmu_service_matched), - IOMFB_METHOD("A132", iomfbep_a132_backlight_service_matched), - IOMFB_METHOD("A373", dcpep_set_create_dfb), - IOMFB_METHOD("A374", iomfbep_a358_vi_set_temperature_hint), - IOMFB_METHOD("A401", dcpep_start_signal), - IOMFB_METHOD("A407", dcpep_swap_start), - IOMFB_METHOD("A408", dcpep_swap_submit), - IOMFB_METHOD("A410", dcpep_set_display_device), - IOMFB_METHOD("A411", dcpep_is_main_display), - IOMFB_METHOD("A412", dcpep_set_digital_out_mode), - IOMFB_METHOD("A422", iomfbep_set_matrix), - IOMFB_METHOD("A426", iomfbep_get_color_remap_mode), - IOMFB_METHOD("A441", dcpep_set_parameter_dcp), - IOMFB_METHOD("A445", dcpep_create_default_fb), - IOMFB_METHOD("A449", dcpep_enable_disable_video_power_savings), - IOMFB_METHOD("A456", dcpep_first_client_open), - IOMFB_METHOD("A457", iomfbep_last_client_close), - IOMFB_METHOD("A463", dcpep_set_display_refresh_properties), - IOMFB_METHOD("A466", dcpep_flush_supports_power), - IOMFB_METHOD("A467", iomfbep_abort_swaps_dcp), - IOMFB_METHOD("A472", dcpep_set_power_state), -}; - -#define DCP_FW v13_3 -#define DCP_FW_VER DCP_FW_VERSION(13, 3, 0) - -#include "iomfb_template.c" - -static const neo_iomfb_cb_handler cb_handlers[IOMFB_MAX_CB] = { - [0] = trampoline_true, /* did_boot_signal */ - [1] = trampoline_true, /* did_power_on_signal */ - [2] = trampoline_nop, /* will_power_off_signal */ - [3] = trampoline_rt_bandwidth, - [6] = trampoline_set_frame_sync_props, - [100] = iomfbep_cb_match_pmu_service, - [101] = trampoline_zero, /* get_display_default_stride */ - [102] = trampoline_nop, /* set_number_property */ - [103] = trampoline_nop, /* trigger_user_cal_loader */ - [104] = trampoline_nop, /* set_boolean_property */ - [107] = trampoline_nop, /* remove_property */ - [108] = trampoline_true, /* create_provider_service */ - [109] = trampoline_true, /* create_product_service */ - [110] = trampoline_true, /* create_pmu_service */ - [111] = trampoline_true, /* create_iomfb_service */ - [112] = trampoline_create_backlight_service, - [113] = trampoline_true, /* create_nvram_service? */ - [114] = trampoline_get_tiling_state, - [115] = trampoline_false, /* set_tiling_state */ - [120] = dcpep_cb_boot_1, - [121] = trampoline_false, /* is_dark_boot */ - [122] = trampoline_false, /* is_dark_boot / is_waking_from_hibernate*/ - [124] = trampoline_read_edt_data, - [126] = trampoline_prop_start, - [127] = trampoline_prop_chunk, - [128] = trampoline_prop_end, - [129] = trampoline_allocate_bandwidth, - [201] = trampoline_map_piodma, - [202] = trampoline_unmap_piodma, - [206] = iomfbep_cb_match_pmu_service_2, - [207] = iomfbep_cb_match_backlight_service, - [208] = trampoline_nop, /* update_backlight_factor_prop */ - [209] = trampoline_get_time, - [300] = trampoline_pr_publish, - [401] = trampoline_get_uint_prop, - [404] = trampoline_nop, /* sr_set_uint_prop */ - [406] = trampoline_set_fx_prop, - [408] = trampoline_get_frequency, - [411] = trampoline_map_reg, - [413] = trampoline_true, /* sr_set_property_dict */ - [414] = trampoline_sr_set_property_int, - [415] = trampoline_true, /* sr_set_property_bool */ - [451] = trampoline_allocate_buffer, - [452] = trampoline_map_physical, - [454] = trampoline_release_mem_desc, - [552] = trampoline_true, /* set_property_dict_0 */ - [561] = trampoline_true, /* set_property_dict */ - [563] = trampoline_true, /* set_property_int */ - [565] = trampoline_true, /* set_property_bool */ - [567] = trampoline_true, /* set_property_str */ - [574] = trampoline_zero, /* power_up_dart */ - [576] = trampoline_hotplug, - [577] = trampoline_nop, /* powerstate_notify */ - [582] = trampoline_true, /* create_default_fb_surface */ - [584] = trampoline_nop, /* IOMobileFramebufferAP::clear_default_surface */ - [588] = trampoline_nop, /* resize_default_fb_surface_gated */ - [589] = trampoline_swap_complete, - [591] = trampoline_swap_complete_intent_gated, - [592] = trampoline_abort_swap_ap_gated, - [593] = trampoline_enable_backlight_message_ap_gated, - [594] = trampoline_nop, /* IOMobileFramebufferAP::setSystemConsoleMode */ - [596] = trampoline_false, /* IOMobileFramebufferAP::isDFBAllocated */ - [597] = trampoline_false, /* IOMobileFramebufferAP::preserveContents */ - [598] = trampoline_nop, /* find_swap_function_gated */ -}; -void DCP_FW_NAME(neo_iomfb_start)(struct neo_apple_dcp *neo_dcp) -{ - neo_dcp->cb_handlers = cb_handlers; - - neo_dcp_start_signal(neo_dcp, false, neo_dcp_started, NULL); -} - -#undef DCP_FW_VER -#undef DCP_FW diff --git a/drivers/gpu/drm/apple-neo/iomfb_v13_3.h b/drivers/gpu/drm/apple-neo/iomfb_v13_3.h deleted file mode 100644 index bbb3156b40f893..00000000000000 --- a/drivers/gpu/drm/apple-neo/iomfb_v13_3.h +++ /dev/null @@ -1,17 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#ifndef __APPLE_IOMFB_V13_3_H__ -#define __APPLE_IOMFB_V13_3_H__ - -#include "version_utils.h" - -#define DCP_FW v13_3 -#define DCP_FW_VER DCP_FW_VERSION(13, 3, 0) - -#include "iomfb_template.h" - -#undef DCP_FW_VER -#undef DCP_FW - -#endif /* __APPLE_IOMFB_V13_3_H__ */ diff --git a/drivers/gpu/drm/apple-neo/parser.c b/drivers/gpu/drm/apple-neo/parser.c deleted file mode 100644 index 6b7bee8c1a00e7..00000000000000 --- a/drivers/gpu/drm/apple-neo/parser.c +++ /dev/null @@ -1,1203 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#include -#include -#include -#include -#include -#include - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) -#include // for sound format masks -#endif - -#include "parser.h" -#include "trace.h" - -#define DCP_PARSE_HEADER 0xd3 - -enum neo_dcp_parse_type { - DCP_TYPE_DICTIONARY = 1, - DCP_TYPE_ARRAY = 2, - DCP_TYPE_INT64 = 4, - DCP_TYPE_STRING = 9, - DCP_TYPE_BLOB = 10, - DCP_TYPE_BOOL = 11 -}; - -struct neo_dcp_parse_tag { - unsigned int size : 24; - enum neo_dcp_parse_type type : 5; - unsigned int padding : 2; - bool last : 1; -} __packed; - -static const void *neo_parse_bytes(struct neo_dcp_parse_ctx *ctx, size_t count) -{ - const void *ptr; - - if (ctx->pos > ctx->len || count > ctx->len - ctx->pos) - return ERR_PTR(-EINVAL); - - ptr = ctx->blob + ctx->pos; - ctx->pos += count; - return ptr; -} - -static const u32 *neo_parse_u32(struct neo_dcp_parse_ctx *ctx) -{ - return neo_parse_bytes(ctx, sizeof(u32)); -} - -static const struct neo_dcp_parse_tag *neo_parse_tag(struct neo_dcp_parse_ctx *ctx) -{ - const struct neo_dcp_parse_tag *tag; - - /* Align to 32-bits */ - if (ctx->pos > U32_MAX - 3) - return ERR_PTR(-EINVAL); - ctx->pos = round_up(ctx->pos, 4); - - tag = neo_parse_bytes(ctx, sizeof(struct neo_dcp_parse_tag)); - - if (IS_ERR(tag)) - return tag; - - if (tag->padding) - return ERR_PTR(-EINVAL); - - return tag; -} - -static const struct neo_dcp_parse_tag *neo_parse_tag_of_type(struct neo_dcp_parse_ctx *ctx, - enum neo_dcp_parse_type type) -{ - const struct neo_dcp_parse_tag *tag = neo_parse_tag(ctx); - - if (IS_ERR(tag)) - return tag; - - if (tag->type != type) - return ERR_PTR(-EINVAL); - - return tag; -} - -/* Bound stack use when ignoring nested firmware dictionaries. */ -#define DCP_PARSE_MAX_DEPTH 32 - -static int skip_value(struct neo_dcp_parse_ctx *handle, unsigned int depth) -{ - const struct neo_dcp_parse_tag *tag; - const void *data; - unsigned int count, i; - int ret; - - if (depth >= DCP_PARSE_MAX_DEPTH) - return -EINVAL; - - tag = neo_parse_tag(handle); - if (IS_ERR(tag)) - return PTR_ERR(tag); - - switch (tag->type) { - case DCP_TYPE_DICTIONARY: - case DCP_TYPE_ARRAY: - count = tag->size; - if (tag->type == DCP_TYPE_DICTIONARY) - count *= 2; - - for (i = 0; i < count; ++i) { - ret = skip_value(handle, depth + 1); - if (ret) - return ret; - } - return 0; - - case DCP_TYPE_INT64: - data = neo_parse_bytes(handle, sizeof(s64)); - break; - - case DCP_TYPE_STRING: - case DCP_TYPE_BLOB: - data = neo_parse_bytes(handle, tag->size); - break; - - case DCP_TYPE_BOOL: - return 0; - - default: - return -EINVAL; - } - - return IS_ERR(data) ? PTR_ERR(data) : 0; -} - -static int skip(struct neo_dcp_parse_ctx *handle) -{ - return skip_value(handle, 0); -} - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) -static int skip_pair(struct neo_dcp_parse_ctx *handle) -{ - int ret; - - ret = skip(handle); - if (ret) - return ret; - - return skip(handle); -} - -static bool consume_string(struct neo_dcp_parse_ctx *ctx, const char *specimen) -{ - struct neo_dcp_parse_ctx probe = *ctx; - const struct neo_dcp_parse_tag *tag; - const char *key; - - tag = neo_parse_tag_of_type(&probe, DCP_TYPE_STRING); - if (IS_ERR(tag) || tag->size != strlen(specimen)) - return false; - - key = neo_parse_bytes(&probe, tag->size); - if (IS_ERR(key) || memcmp(key, specimen, tag->size)) - return false; - - *ctx = probe; - return true; -} -#endif - -/* Caller must free the result */ -static char *neo_parse_string(struct neo_dcp_parse_ctx *handle) -{ - const struct neo_dcp_parse_tag *tag = neo_parse_tag_of_type(handle, DCP_TYPE_STRING); - const char *in; - char *out; - - if (IS_ERR(tag)) - return (void *)tag; - - in = neo_parse_bytes(handle, tag->size); - if (IS_ERR(in)) - return (void *)in; - - out = kmalloc(tag->size + 1, GFP_KERNEL); - if (!out) - return ERR_PTR(-ENOMEM); - - memcpy(out, in, tag->size); - out[tag->size] = '\0'; - return out; -} - -static int neo_parse_int_bound(struct neo_dcp_parse_ctx *handle, s64 *value, s64 min, s64 max) -{ - const void *tag = neo_parse_tag_of_type(handle, DCP_TYPE_INT64); - const void *in; - - if (IS_ERR(tag)) - return PTR_ERR(tag); - - in = neo_parse_bytes(handle, sizeof(s64)); - - if (IS_ERR(in)) - return PTR_ERR(in); - - memcpy(value, in, sizeof(*value)); - - if (*value < min || *value > max) - return -EINVAL; - - return 0; -} - -static int neo_parse_int(struct neo_dcp_parse_ctx *handle, s64 *value) -{ - return neo_parse_int_bound(handle, value, S64_MIN, S64_MAX); -} - -static int neo_parse_bool(struct neo_dcp_parse_ctx *handle, bool *b) -{ - const struct neo_dcp_parse_tag *tag = neo_parse_tag_of_type(handle, DCP_TYPE_BOOL); - - if (IS_ERR(tag)) - return PTR_ERR(tag); - - *b = !!tag->size; - return 0; -} - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) -static int neo_parse_blob(struct neo_dcp_parse_ctx *handle, size_t size, u8 const **blob) -{ - const struct neo_dcp_parse_tag *tag = neo_parse_tag_of_type(handle, DCP_TYPE_BLOB); - const u8 *out; - - if (IS_ERR(tag)) - return PTR_ERR(tag); - - if (tag->size < size) - return -EINVAL; - - out = neo_parse_bytes(handle, tag->size); - - if (IS_ERR(out)) - return PTR_ERR(out); - - *blob = out; - return 0; -} -#endif - -struct iterator { - struct neo_dcp_parse_ctx *handle; - u32 idx, len; -}; - -static int iterator_begin(struct neo_dcp_parse_ctx *handle, struct iterator *it, - bool dict) -{ - const struct neo_dcp_parse_tag *tag; - enum neo_dcp_parse_type type = dict ? DCP_TYPE_DICTIONARY : DCP_TYPE_ARRAY; - - *it = (struct iterator) { - .handle = handle, - .idx = 0 - }; - - tag = neo_parse_tag_of_type(it->handle, type); - if (IS_ERR(tag)) - return PTR_ERR(tag); - - it->len = tag->size; - return 0; -} - -int neo_parse(const void *blob, size_t size, struct neo_dcp_parse_ctx *ctx) -{ - const u32 *header; - - if (size > U32_MAX) - return -EINVAL; - - *ctx = (struct neo_dcp_parse_ctx) { - .blob = blob, - .len = size, - .pos = 0, - }; - - header = neo_parse_u32(ctx); - if (IS_ERR(header)) - return PTR_ERR(header); - - if (*header != DCP_PARSE_HEADER) - return -EINVAL; - - return 0; -} - -static int neo_parse_dimension(struct neo_dcp_parse_ctx *handle, struct dimension *dim) -{ - struct iterator it; - int ret = 0; - - *dim = (struct dimension) { - .active = -1, - .total = -1, - .front_porch = -1, - .sync_width = -1, - .precise_sync_rate = -1, - }; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - - if (IS_ERR(key)) - ret = PTR_ERR(key); - else if (!strcmp(key, "Active")) - ret = neo_parse_int_bound(it.handle, &dim->active, 0, U16_MAX); - else if (!strcmp(key, "Total")) - ret = neo_parse_int_bound(it.handle, &dim->total, 0, U16_MAX); - else if (!strcmp(key, "FrontPorch")) - ret = neo_parse_int_bound(it.handle, &dim->front_porch, 0, U16_MAX); - else if (!strcmp(key, "SyncWidth")) - ret = neo_parse_int_bound(it.handle, &dim->sync_width, 0, U16_MAX); - else if (!strcmp(key, "PreciseSyncRate")) - ret = neo_parse_int_bound(it.handle, &dim->precise_sync_rate, 1, U32_MAX); - else - ret = skip(it.handle); - - if (!IS_ERR_OR_NULL(key)) - kfree(key); - - if (ret) - return ret; - } - - if (dim->active <= 0 || dim->total < dim->active || - dim->front_porch < 0 || dim->sync_width < 0 || - dim->front_porch + dim->sync_width > dim->total - dim->active) - return -EINVAL; - - return 0; -} - -struct color_mode { - s64 colorimetry; - s64 depth; - s64 dynamic_range; - s64 eotf; - s64 id; - s64 pixel_encoding; - s64 score; -}; - -static int fill_color_mode(struct neo_dcp_color_mode *color, - struct color_mode *cmode) -{ - if (cmode->id < 0 || cmode->id > U32_MAX) - return -EINVAL; - - if (color->score >= cmode->score) - return 0; - - if (cmode->colorimetry < 0 || cmode->colorimetry >= DCP_COLORIMETRY_COUNT) - return -EINVAL; - if (cmode->depth < 8 || cmode->depth > 12) - return -EINVAL; - if (cmode->dynamic_range < 0 || cmode->dynamic_range >= DCP_COLOR_YCBCR_RANGE_COUNT) - return -EINVAL; - if (cmode->eotf < 0 || cmode->eotf >= DCP_EOTF_COUNT) - return -EINVAL; - if (cmode->pixel_encoding < 0 || cmode->pixel_encoding >= DCP_COLOR_FORMAT_COUNT) - return -EINVAL; - - color->score = cmode->score; - color->id = cmode->id; - color->eotf = cmode->eotf; - color->format = cmode->pixel_encoding; - color->colorimetry = cmode->colorimetry; - color->range = cmode->dynamic_range; - color->depth = cmode->depth; - - return 0; -} - -static int neo_parse_color_modes(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_display_mode *out) -{ - struct iterator outer_it; - int ret = 0; - out->sdr_444.score = -1; - out->sdr_rgb.score = -1; - out->sdr.score = -1; - out->best.score = -1; - - ret = iterator_begin(handle, &outer_it, false); - if (ret) - return ret; - - for (; outer_it.idx < outer_it.len; ++outer_it.idx) { - struct iterator it; - bool is_virtual = true; - struct color_mode cmode = { - .colorimetry = -1, - .depth = -1, - .dynamic_range = -1, - .eotf = -1, - .id = -1, - .pixel_encoding = -1, - .score = -1, - }; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - - if (IS_ERR(key)) - ret = PTR_ERR(key); - else if (!strcmp(key, "Colorimetry")) - ret = neo_parse_int(it.handle, &cmode.colorimetry); - else if (!strcmp(key, "Depth")) - ret = neo_parse_int(it.handle, &cmode.depth); - else if (!strcmp(key, "DynamicRange")) - ret = neo_parse_int(it.handle, &cmode.dynamic_range); - else if (!strcmp(key, "EOTF")) - ret = neo_parse_int(it.handle, &cmode.eotf); - else if (!strcmp(key, "ID")) - ret = neo_parse_int(it.handle, &cmode.id); - else if (!strcmp(key, "IsVirtual")) - ret = neo_parse_bool(it.handle, &is_virtual); - else if (!strcmp(key, "PixelEncoding")) - ret = neo_parse_int(it.handle, &cmode.pixel_encoding); - else if (!strcmp(key, "Score")) - ret = neo_parse_int(it.handle, &cmode.score); - else - ret = skip(it.handle); - - if (!IS_ERR_OR_NULL(key)) - kfree(key); - - if (ret) - return ret; - } - - /* Skip virtual or partial entries */ - if (is_virtual || cmode.score < 0 || cmode.id < 0) - continue; - - trace_neo_iomfb_color_mode(handle->neo_dcp, cmode.id, cmode.score, - cmode.depth, cmode.colorimetry, - cmode.eotf, cmode.dynamic_range, - cmode.pixel_encoding); - - if (cmode.eotf == DCP_EOTF_SDR_GAMMA) { - if (cmode.pixel_encoding == DCP_COLOR_FORMAT_RGB && - cmode.depth <= 10) - fill_color_mode(&out->sdr_rgb, &cmode); - else if (cmode.pixel_encoding == DCP_COLOR_FORMAT_YCBCR444 && - cmode.depth <= 10) - fill_color_mode(&out->sdr_444, &cmode); - fill_color_mode(&out->sdr, &cmode); - } - fill_color_mode(&out->best, &cmode); - } - - return 0; -} - -/* - * Calculate the pixel clock for a mode given the 16:16 fixed-point refresh - * rate. The pixel clock is the refresh rate times the pixel count. DRM - * specifies the clock in kHz. The intermediate result may overflow a u32, so - * use a u64 where required. - */ -static u64 calculate_clock(struct dimension *horiz, struct dimension *vert) -{ - u32 pixels = horiz->total * vert->total; - u64 clock = mul_u32_u32(pixels, vert->precise_sync_rate); - - return DIV_ROUND_CLOSEST_ULL(clock >> 16, 1000); -} - -static int neo_parse_mode(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_display_mode *out, s64 *score, int width_mm, - int height_mm, unsigned notch_height, bool internal) -{ - int ret = 0; - struct iterator it; - struct dimension horiz = { .active = -1 }, vert = { .active = -1 }; - s64 min_vrr = 0, max_vrr = 0; - u64 clock; - s64 id = -1; - s64 best_color_mode = -1; - bool is_virtual = false; - struct drm_display_mode *mode = &out->mode; - - *out = (struct neo_dcp_display_mode) { - .sdr_rgb.score = -1, - .sdr_444.score = -1, - .sdr.score = -1, - .best.score = -1, - }; - *score = -1; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - - if (IS_ERR(key)) - ret = PTR_ERR(key); - else if (is_virtual) - ret = skip(it.handle); - else if (!strcmp(key, "HorizontalAttributes")) - ret = neo_parse_dimension(it.handle, &horiz); - else if (!strcmp(key, "VerticalAttributes")) - ret = neo_parse_dimension(it.handle, &vert); - else if (!strcmp(key, "MinimumVariableRefreshRate")) - ret = neo_parse_int_bound(it.handle, &min_vrr, 0, U32_MAX); - else if (!strcmp(key, "MaximumVariableRefreshRate")) - ret = neo_parse_int_bound(it.handle, &max_vrr, 0, U32_MAX); - else if (!strcmp(key, "ColorModes")) - ret = neo_parse_color_modes(it.handle, out); - else if (!strcmp(key, "ID")) - ret = neo_parse_int_bound(it.handle, &id, 0, U32_MAX); - else if (!strcmp(key, "IsVirtual")) - ret = neo_parse_bool(it.handle, &is_virtual); - else if (!strcmp(key, "Score")) - ret = neo_parse_int_bound(it.handle, score, 0, S64_MAX); - else - ret = skip(it.handle); - - if (!IS_ERR_OR_NULL(key)) - kfree(key); - - if (ret) { - trace_neo_iomfb_parse_mode_fail(id, &horiz, &vert, best_color_mode, is_virtual, *score); - return ret; - } - } - if (out->sdr_rgb.score >= 0) - best_color_mode = out->sdr_rgb.id; - else if (out->sdr_444.score >= 0) - best_color_mode = out->sdr_444.id; - else if (out->sdr.score >= 0) - best_color_mode = out->sdr.id; - else if (out->best.score >= 0) - best_color_mode = out->best.id; - - trace_neo_iomfb_parse_mode_success(id, &horiz, &vert, best_color_mode, - is_virtual, *score); - - /* - * Reject modes without valid color mode. - */ - if (best_color_mode < 0 || id < 0 || *score < 0 || - horiz.active <= 0 || vert.active <= notch_height || - vert.precise_sync_rate <= 0) - return -EINVAL; - - /* - * We need to skip virtual modes. In some cases, virtual modes are "too - * big" for the monitor and can cause breakage. It is unclear why the - * DCP reports these modes at all. Treat as a recoverable error. - */ - if (is_virtual) - return -EINVAL; - - /* - * An internal ProMotion panel carries no EDID or DisplayID, so DCP - * reports no adaptive-sync range for it. Assume the ProMotion floor of - * 24 Hz up to whatever rate the mode itself advertises. - */ - if (internal && vert.precise_sync_rate >> 16 == 120) { - out->min_vrr = 24 << 16; - out->max_vrr = vert.precise_sync_rate; - out->vrr = true; - } - - /* Refresh rates are reported by DCP as 16.16 fixed-point Hz. */ - if (min_vrr && max_vrr > min_vrr) { - out->min_vrr = min_vrr; - out->max_vrr = max_vrr; - out->vrr = true; - } - - clock = calculate_clock(&horiz, &vert); - if (!clock || clock > INT_MAX) - return -EINVAL; - - vert.active -= notch_height; - vert.sync_width += notch_height; - - /* From here we must succeed. Start filling out the mode. */ - *mode = (struct drm_display_mode) { - .type = DRM_MODE_TYPE_DRIVER, - .clock = clock, - - .vdisplay = vert.active, - .vsync_start = vert.active + vert.front_porch, - .vsync_end = vert.active + vert.front_porch + vert.sync_width, - .vtotal = vert.total, - - .hdisplay = horiz.active, - .hsync_start = horiz.active + horiz.front_porch, - .hsync_end = horiz.active + horiz.front_porch + - horiz.sync_width, - .htotal = horiz.total, - - .width_mm = width_mm, - .height_mm = height_mm, - }; - - drm_mode_set_name(mode); - - out->timing_mode_id = id; - out->color_mode_id = best_color_mode; - - trace_neo_iomfb_timing_mode(handle->neo_dcp, id, *score, horiz.active, - vert.active, vert.precise_sync_rate, - best_color_mode); - - return 0; -} - -struct neo_dcp_display_mode *neo_enumerate_modes(struct neo_dcp_parse_ctx *handle, - unsigned int *count, int width_mm, - int height_mm, unsigned notch_height, - bool internal) -{ - struct iterator it; - int ret; - struct neo_dcp_display_mode *mode, *modes; - struct neo_dcp_display_mode *best_mode = NULL; - s64 score, best_score = -1; - - ret = iterator_begin(handle, &it, false); - - if (ret) - return ERR_PTR(ret); - - /* Start with a worst case allocation. */ - modes = kcalloc(it.len, sizeof(*modes), GFP_KERNEL); - *count = 0; - - if (!modes) - return ERR_PTR(-ENOMEM); - - for (; it.idx < it.len; ++it.idx) { - struct neo_dcp_parse_ctx entry = *it.handle; - - mode = &modes[*count]; - ret = neo_parse_mode(it.handle, mode, &score, width_mm, height_mm, - notch_height, internal); - - /* Errors for a single mode are recoverable -- just skip it. */ - if (ret) { - /* A failed parse can stop in the middle of the dictionary. */ - *it.handle = entry; - ret = skip(it.handle); - if (ret) { - kfree(modes); - return ERR_PTR(ret); - } - continue; - } - - /* Process a successful mode */ - (*count)++; - - if (score > best_score) { - best_score = score; - best_mode = mode; - } - } - - if (best_mode != NULL) - best_mode->mode.type |= DRM_MODE_TYPE_PREFERRED; - - return modes; -} - -int neo_parse_display_attributes(struct neo_dcp_parse_ctx *handle, int *width_mm, - int *height_mm) -{ - int ret = 0; - struct iterator it; - s64 width_cm = 0, height_cm = 0; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - - if (IS_ERR(key)) - ret = PTR_ERR(key); - else if (!strcmp(key, "MaxHorizontalImageSize")) - ret = neo_parse_int(it.handle, &width_cm); - else if (!strcmp(key, "MaxVerticalImageSize")) - ret = neo_parse_int(it.handle, &height_cm); - else - ret = skip(it.handle); - - if (!IS_ERR_OR_NULL(key)) - kfree(key); - - if (ret) - return ret; - } - - /* 1cm = 10mm */ - *width_mm = 10 * width_cm; - *height_mm = 10 * height_cm; - - return 0; -} - -int neo_parse_epic_service_init(struct neo_dcp_parse_ctx *handle, const char **name, - const char **class, s64 *unit, bool h17p_keys) -{ - int ret = 0; - struct iterator it; - bool parsed_unit = false; - bool parsed_name = false; - bool parsed_class = false; - - *name = ERR_PTR(-ENOENT); - *class = ERR_PTR(-ENOENT); - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - - if (IS_ERR(key)) { - ret = PTR_ERR(key); - break; - } - - /* - * H17P advertises its system service with lower-case keys -- - * "compartment", "name", "interface-id" -- where earlier - * firmware uses EPICName/EPICProviderClass/EPICUnit. Without - * them the service never registers, the firmware has no system - * service to ask and idles the panel, so no vblank and no swap - * ever completes. Only H17P callers accept them. If a - * dictionary carries both spellings, the later value wins and - * the earlier string is freed. - */ - if (!strcmp(key, "EPICName") || - (h17p_keys && !strcmp(key, "compartment"))) { - if (parsed_name) - kfree(*name); - *name = neo_parse_string(it.handle); - if (IS_ERR(*name)) - ret = PTR_ERR(*name); - else - parsed_name = true; - } else if (!strcmp(key, "EPICProviderClass") || - (h17p_keys && !strcmp(key, "name"))) { - if (parsed_class) - kfree(*class); - *class = neo_parse_string(it.handle); - if (IS_ERR(*class)) - ret = PTR_ERR(*class); - else - parsed_class = true; - } else if (!strcmp(key, "EPICUnit") || - (h17p_keys && !strcmp(key, "interface-id"))) { - ret = neo_parse_int(it.handle, unit); - if (!ret) - parsed_unit = true; - } else { - ret = skip(it.handle); - } - - kfree(key); - if (ret) - break; - } - - if (!parsed_unit || !parsed_name || !parsed_class) - ret = -ENOENT; - - if (ret) { - if (!IS_ERR(*name)) { - kfree(*name); - *name = ERR_PTR(ret); - } - if (!IS_ERR(*class)) { - kfree(*class); - *class = ERR_PTR(ret); - } - } - - return ret; -} - -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_AUDIO) -static int neo_parse_sample_rate_bit(struct neo_dcp_parse_ctx *handle, unsigned int *ratebit) -{ - s64 rate; - int ret = neo_parse_int(handle, &rate); - - if (ret) - return ret; - - *ratebit = snd_pcm_rate_to_rate_bit(rate); - if (*ratebit == SNDRV_PCM_RATE_KNOT) { - /* - * The rate wasn't recognized, and unless we supply - * a supplementary constraint, the SNDRV_PCM_RATE_KNOT bit - * will allow any rate. So clear it. - */ - *ratebit = 0; - } - - return 0; -} - -static int neo_parse_sample_fmtbit(struct neo_dcp_parse_ctx *handle, u64 *fmtbit) -{ - s64 sample_size; - int ret = neo_parse_int(handle, &sample_size); - - if (ret) - return ret; - - switch (sample_size) { - case 16: - *fmtbit = SNDRV_PCM_FMTBIT_S16; - break; - case 20: - *fmtbit = SNDRV_PCM_FMTBIT_S20; - break; - case 24: - *fmtbit = SNDRV_PCM_FMTBIT_S24; - break; - case 32: - *fmtbit = SNDRV_PCM_FMTBIT_S32; - break; - default: - *fmtbit = 0; - break; - } - - return 0; -} - -static struct { - const char *label; - u8 type; -} chan_position_names[] = { - { "Front Left", SNDRV_CHMAP_FL }, - { "Front Right", SNDRV_CHMAP_FR }, - { "Rear Left", SNDRV_CHMAP_RL }, - { "Rear Right", SNDRV_CHMAP_RR }, - { "Front Center", SNDRV_CHMAP_FC }, - { "Low Frequency Effects", SNDRV_CHMAP_LFE }, - { "Rear Center", SNDRV_CHMAP_RC }, - { "Front Left Center", SNDRV_CHMAP_FLC }, - { "Front Right Center", SNDRV_CHMAP_FRC }, - { "Rear Left Center", SNDRV_CHMAP_RLC }, - { "Rear Right Center", SNDRV_CHMAP_RRC }, - { "Front Left Wide", SNDRV_CHMAP_FLW }, - { "Front Right Wide", SNDRV_CHMAP_FRW }, - { "Front Left High", SNDRV_CHMAP_FLH }, - { "Front Center High", SNDRV_CHMAP_FCH }, - { "Front Right High", SNDRV_CHMAP_FRH }, - { "Top Center", SNDRV_CHMAP_TC }, -}; - -static void append_chmap(struct snd_pcm_chmap_elem *chmap, u8 type) -{ - if (!chmap || chmap->channels >= ARRAY_SIZE(chmap->map)) - return; - - chmap->map[chmap->channels] = type; - chmap->channels++; -} - -static int neo_parse_chmap(struct neo_dcp_parse_ctx *handle, struct snd_pcm_chmap_elem *chmap) -{ - struct iterator it; - int i, ret; - - if (!chmap) - return skip(handle); - - chmap->channels = 0; - - ret = iterator_begin(handle, &it, false); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - for (i = 0; i < ARRAY_SIZE(chan_position_names); i++) - if (consume_string(it.handle, chan_position_names[i].label)) - break; - - if (i == ARRAY_SIZE(chan_position_names)) { - ret = skip(it.handle); - if (ret) - return ret; - - append_chmap(chmap, SNDRV_CHMAP_UNKNOWN); - continue; - } - - append_chmap(chmap, chan_position_names[i].type); - } - - return 0; -} - -static int neo_parse_chan_layout_element(struct neo_dcp_parse_ctx *handle, - unsigned int *nchans_out, - struct snd_pcm_chmap_elem *chmap) -{ - struct iterator it; - int ret; - s64 nchans = 0; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - if (consume_string(it.handle, "ActiveChannelCount")) - ret = neo_parse_int(it.handle, &nchans); - else if (consume_string(it.handle, "ChannelLayout")) - ret = neo_parse_chmap(it.handle, chmap); - else - ret = skip_pair(it.handle); - - if (ret) - return ret; - } - - if (nchans_out) - *nchans_out = nchans; - - return 0; -} - -static int neo_parse_nchans_mask(struct neo_dcp_parse_ctx *handle, unsigned int *mask) -{ - struct iterator it; - int ret; - - *mask = 0; - - ret = iterator_begin(handle, &it, false); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - int nchans; - - ret = neo_parse_chan_layout_element(it.handle, &nchans, NULL); - if (ret) - return ret; - *mask |= 1 << nchans; - } - - return 0; -} - -static int neo_parse_avep_element(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_sound_format_mask *sieve, - struct neo_dcp_sound_format_mask *hits) -{ - struct neo_dcp_sound_format_mask mask = {0, 0, 0}; - struct iterator it; - int ret; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - if (consume_string(handle, "StreamSampleRate")) - ret = neo_parse_sample_rate_bit(it.handle, &mask.rates); - else if (consume_string(handle, "SampleSize")) - ret = neo_parse_sample_fmtbit(it.handle, &mask.formats); - else if (consume_string(handle, "AudioChannelLayoutElements")) - ret = neo_parse_nchans_mask(it.handle, &mask.nchans); - else - ret = skip_pair(it.handle); - - if (ret) - return ret; - } - - trace_neo_avep_sound_mode(handle->neo_dcp, mask.rates, mask.formats, mask.nchans); - - if (!(mask.rates & sieve->rates) || !(mask.formats & sieve->formats) || - !(mask.nchans & sieve->nchans)) - return 0; - - if (hits) { - hits->rates |= mask.rates; - hits->formats |= mask.formats; - hits->nchans |= mask.nchans; - } - - return 1; -} - -static int neo_parse_mode_in_avep_element(struct neo_dcp_parse_ctx *handle, - unsigned int selected_nchans, - struct snd_pcm_chmap_elem *chmap, - struct neo_dcp_sound_cookie *cookie) -{ - struct iterator it; - struct neo_dcp_parse_ctx save_handle; - int ret; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - if (consume_string(it.handle, "AudioChannelLayoutElements")) { - struct iterator inner_it; - int nchans; - - ret = iterator_begin(it.handle, &inner_it, false); - if (ret) - return ret; - - for (; inner_it.idx < inner_it.len; ++inner_it.idx) { - save_handle = *it.handle; - ret = neo_parse_chan_layout_element(inner_it.handle, - &nchans, NULL); - if (ret) - return ret; - - if (nchans != selected_nchans) - continue; - - /* - * Now that we know this layout matches the - * selected channel number, reread the element - * and fill in the channel map. - */ - *inner_it.handle = save_handle; - ret = neo_parse_chan_layout_element(inner_it.handle, - NULL, chmap); - if (ret) - return ret; - } - } else if (consume_string(it.handle, "ElementData")) { - const u8 *blob; - - ret = neo_parse_blob(it.handle, sizeof(*cookie), &blob); - if (ret) - return ret; - - if (cookie) - memcpy(cookie, blob, sizeof(*cookie)); - } else { - ret = skip_pair(it.handle); - if (ret) - return ret; - } - } - - return 0; -} - -int neo_parse_sound_constraints(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_sound_format_mask *sieve, - struct neo_dcp_sound_format_mask *hits) -{ - int ret; - struct iterator it; - - if (hits) { - hits->rates = 0; - hits->formats = 0; - hits->nchans = 0; - } - - ret = iterator_begin(handle, &it, false); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - ret = neo_parse_avep_element(it.handle, sieve, hits); - - if (ret < 0) - return ret; - } - - return 0; -} - -int neo_parse_sound_mode(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_sound_format_mask *sieve, - struct snd_pcm_chmap_elem *chmap, - struct neo_dcp_sound_cookie *cookie) -{ - struct neo_dcp_parse_ctx save_handle; - struct iterator it; - int ret; - - ret = iterator_begin(handle, &it, false); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - save_handle = *it.handle; - ret = neo_parse_avep_element(it.handle, sieve, NULL); - - if (!ret) - continue; - - if (ret < 0) - return ret; - - ret = neo_parse_mode_in_avep_element(&save_handle, __ffs(sieve->nchans), - chmap, cookie); - if (ret < 0) - return ret; - return 1; - } - - return 0; -} -#endif - -int neo_parse_system_log_mnits(struct neo_dcp_parse_ctx *handle, struct neo_dcp_system_ev_mnits *entry) -{ - struct iterator it; - int ret; - s64 mnits = -1; - s64 idac = -1; - s64 timestamp = -1; - bool type_match = false; - - ret = iterator_begin(handle, &it, true); - if (ret) - return ret; - - for (; it.idx < it.len; ++it.idx) { - char *key = neo_parse_string(it.handle); - if (IS_ERR(key)) { - ret = PTR_ERR(key); - } else if (!strcmp(key, "mNits")) { - ret = neo_parse_int(it.handle, &mnits); - } else if (!strcmp(key, "iDAC")) { - ret = neo_parse_int(it.handle, &idac); - } else if (!strcmp(key, "logEvent")) { - const char *value = neo_parse_string(it.handle); - - ret = IS_ERR(value) ? PTR_ERR(value) : 0; - if (!ret) { - type_match = strcmp(value, "Display (Event Forward)") == 0; - kfree(value); - } - } else if (!strcmp(key, "timestamp")) { - ret = neo_parse_int(it.handle, ×tamp); - } else { - ret = skip(it.handle); - } - - if (!IS_ERR_OR_NULL(key)) - kfree(key); - - if (ret) { - pr_err("dcp parser: failed to parse mNits sys event\n"); - return ret; - } - } - - if (!type_match || mnits < 0 || idac < 0 || timestamp < 0) - return -EINVAL; - if (mnits > U32_MAX) - return -ERANGE; - - entry->millinits = mnits; - entry->idac = idac; - entry->timestamp = timestamp; - - return 0; -} diff --git a/drivers/gpu/drm/apple-neo/parser.h b/drivers/gpu/drm/apple-neo/parser.h deleted file mode 100644 index d62c33c3234166..00000000000000 --- a/drivers/gpu/drm/apple-neo/parser.h +++ /dev/null @@ -1,142 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2021 Alyssa Rosenzweig */ - -#ifndef __APPLE_DCP_PARSER_H__ -#define __APPLE_DCP_PARSER_H__ - -/* For mode parsing */ -#include - -struct neo_apple_dcp; - -struct neo_dcp_parse_ctx { - struct neo_apple_dcp *neo_dcp; - const void *blob; - u32 pos, len; -}; - -enum neo_dcp_color_eotf { - DCP_EOTF_SDR_GAMMA = 0, // "SDR gamma" - DCP_EOTF_HDR_GAMMA = 1, // "HDR gamma" - DCP_EOTF_ST_2084 = 2, // "ST 2084 (PQ)" - DCP_EOTF_BT_2100 = 3, // "BT.2100 (HLG)" - DCP_EOTF_COUNT -}; - -enum neo_dcp_color_format { - DCP_COLOR_FORMAT_RGB = 0, // "RGB" - DCP_COLOR_FORMAT_YCBCR420 = 1, // "YUV 4:2:0" - DCP_COLOR_FORMAT_YCBCR422 = 3, // "YUV 4:2:2" - DCP_COLOR_FORMAT_YCBCR444 = 2, // "YUV 4:4:4" - DCP_COLOR_FORMAT_DV_NATIVE = 4, // "DolbyVision (native)" - DCP_COLOR_FORMAT_DV_HDMI = 5, // "DolbyVision (HDMI)" - DCP_COLOR_FORMAT_YCBCR422_DP = 6, // "YCbCr 4:2:2 (DP tunnel)" - DCP_COLOR_FORMAT_YCBCR422_HDMI = 7, // "YCbCr 4:2:2 (HDMI tunnel)" - DCP_COLOR_FORMAT_DV_LL_YCBCR422 = 8, // "DolbyVision LL YCbCr 4:2:2" - DCP_COLOR_FORMAT_DV_LL_YCBCR422_DP = 9, // "DolbyVision LL YCbCr 4:2:2 (DP)" - DCP_COLOR_FORMAT_DV_LL_YCBCR422_HDMI = 10, // "DolbyVision LL YCbCr 4:2:2 (HDMI)" - DCP_COLOR_FORMAT_DV_LL_YCBCR444 = 11, // "DolbyVision LL YCbCr 4:4:4" - DCP_COLOR_FORMAT_DV_LL_RGB422 = 12, // "DolbyVision LL RGB 4:2:2" - DCP_COLOR_FORMAT_GRGB_BLUE_422 = 13, // "GRGB as YCbCr422 (Even line blue)" - DCP_COLOR_FORMAT_GRGB_RED_422 = 14, // "GRGB as YCbCr422 (Even line red)" - DCP_COLOR_FORMAT_COUNT -}; - -enum neo_dcp_colorimetry { - DCP_COLORIMETRY_BT601 = 0, // "SMPTE 170M/BT.601" - DCP_COLORIMETRY_BT709 = 1, // "BT.701" - DCP_COLORIMETRY_XVYCC_601 = 2, // "xvYCC601" - DCP_COLORIMETRY_XVYCC_709 = 3, // "xvYCC709" - DCP_COLORIMETRY_SYCC_601 = 4, // "sYCC601" - DCP_COLORIMETRY_ADOBE_YCC_601 = 5, // "AdobeYCC601" - DCP_COLORIMETRY_BT2020_CYCC = 6, // "BT.2020 (c)" - DCP_COLORIMETRY_BT2020_YCC = 7, // "BT.2020 (nc)" - DCP_COLORIMETRY_VSVDB = 8, // "DolbyVision VSVDB" - DCP_COLORIMETRY_BT2020_RGB = 9, // "BT.2020 (RGB)" - DCP_COLORIMETRY_SRGB = 10, // "sRGB" - DCP_COLORIMETRY_SCRGB = 11, // "scRGB" - DCP_COLORIMETRY_SCRGB_FIXED = 12, // "scRGBfixed" - DCP_COLORIMETRY_ADOBE_RGB = 13, // "AdobeRGB" - DCP_COLORIMETRY_DCI_P3_RGB_D65 = 14, // "DCI-P3 (D65)" - DCP_COLORIMETRY_DCI_P3_RGB_THEATER = 15, // "DCI-P3 (Theater)" - DCP_COLORIMETRY_RGB = 16, // "Default RGB" - DCP_COLORIMETRY_COUNT -}; - -enum neo_dcp_color_range { - DCP_COLOR_YCBCR_RANGE_FULL = 0, - DCP_COLOR_YCBCR_RANGE_LIMITED = 1, - DCP_COLOR_YCBCR_RANGE_COUNT -}; - -struct neo_dcp_color_mode { - s64 score; - u32 id; - enum neo_dcp_color_eotf eotf; - enum neo_dcp_color_format format; - enum neo_dcp_colorimetry colorimetry; - enum neo_dcp_color_range range; - u8 depth; -}; - -/* - * Represents a single display mode. These mode objects are populated at - * runtime based on the TimingElements dictionary sent by the DCP. - */ -struct neo_dcp_display_mode { - struct drm_display_mode mode; - u32 color_mode_id; - u32 timing_mode_id; - u32 min_vrr; - u32 max_vrr; - struct neo_dcp_color_mode sdr_rgb; - struct neo_dcp_color_mode sdr_444; - struct neo_dcp_color_mode sdr; - struct neo_dcp_color_mode best; - bool vrr; -}; - -struct dimension { - s64 total, front_porch, sync_width, active; - s64 precise_sync_rate; -}; - -int neo_parse(const void *blob, size_t size, struct neo_dcp_parse_ctx *ctx); -struct neo_dcp_display_mode *neo_enumerate_modes(struct neo_dcp_parse_ctx *handle, - unsigned int *count, int width_mm, - int height_mm, unsigned notch_height, - bool internal); -int neo_parse_display_attributes(struct neo_dcp_parse_ctx *handle, int *width_mm, - int *height_mm); -int neo_parse_epic_service_init(struct neo_dcp_parse_ctx *handle, const char **name, - const char **class, s64 *unit, bool h17p_keys); - -struct neo_dcp_sound_format_mask { - u64 formats; /* SNDRV_PCM_FMTBIT_* */ - unsigned int rates; /* SNDRV_PCM_RATE_* */ - unsigned int nchans; -}; - -struct neo_dcp_sound_cookie { - u8 data[24]; -}; - -struct snd_pcm_chmap_elem; -int neo_parse_sound_constraints(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_sound_format_mask *sieve, - struct neo_dcp_sound_format_mask *hits); -int neo_parse_sound_mode(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_sound_format_mask *sieve, - struct snd_pcm_chmap_elem *chmap, - struct neo_dcp_sound_cookie *cookie); - -struct neo_dcp_system_ev_mnits { - u32 timestamp; - u32 millinits; - u32 idac; -}; - -int neo_parse_system_log_mnits(struct neo_dcp_parse_ctx *handle, - struct neo_dcp_system_ev_mnits *entry); - -#endif diff --git a/drivers/gpu/drm/apple-neo/plane.c b/drivers/gpu/drm/apple-neo/plane.c deleted file mode 100644 index 41e7215578f50c..00000000000000 --- a/drivers/gpu/drm/apple-neo/plane.c +++ /dev/null @@ -1,561 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * Copyright (C) The Asahi Linux Contributors - */ - -#include "plane.h" - -#include "dcp-internal.h" -#include "iomfb_internal.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#define FRAC_16_16(mult, div) (((mult) << 16) / (div)) - -static int neo_apple_plane_atomic_check(struct drm_plane *plane, - struct drm_atomic_state *state) -{ - struct drm_plane_state *new_plane_state; - struct drm_crtc_state *crtc_state; - struct drm_rect *dst; - int ret; - - new_plane_state = drm_atomic_get_new_plane_state(state, plane); - - if (!new_plane_state->crtc) - return 0; - - crtc_state = drm_atomic_get_crtc_state(state, new_plane_state->crtc); - if (IS_ERR(crtc_state)) - return PTR_ERR(crtc_state); - - /* - * DCP limits downscaling to 2x and upscaling to 4x. Attempting to - * scale outside these bounds errors out when swapping. - * - * This function also takes care of clipping the src/dest rectangles, - * which is required for correct operation. Partially off-screen - * surfaces may appear corrupted. - * - * DCP does not distinguish plane types in the hardware, so we set - * can_position. If the primary plane does not fill the screen, the - * hardware will fill in zeroes (black). - */ - ret = drm_atomic_helper_check_plane_state(new_plane_state, crtc_state, - FRAC_16_16(1, 2), - FRAC_16_16(4, 1), - true, true); - if (ret < 0) - return ret; - - if (!new_plane_state->visible) - return 0; - - /* - * DCP does not allow a surface to clip off the screen, and will crash - * if any blended surface is smaller than 32x32. Reject the atomic op - * if the plane will crash DCP. - * - * This is most pertinent to cursors. Userspace should fall back to - * software cursors if the plane check is rejected. - */ - dst = &new_plane_state->dst; - if (drm_rect_width(dst) < 32 || drm_rect_height(dst) < 32) { - dev_err_once(state->dev->dev, - "Plane operation would have crashed DCP! Rejected!\n\ - DCP requires 32x32 of every plane to be within screen space.\n\ - Your compositor asked to overlay [%dx%d, %dx%d] on %dx%d.\n\ - This is not supported, and your compositor should have\n\ - switched to software compositing when this operation failed.\n\ - You should not have noticed this at all. If your screen\n\ - froze/hitched, or your compositor crashed, please report\n\ - this to the your compositor's developers. We will not\n\ - throw this error again until you next reboot.\n", - dst->x1, dst->y1, dst->x2, dst->y2, - crtc_state->mode.hdisplay, crtc_state->mode.vdisplay); - return -EINVAL; - } - - /* - * Pitches have to be 64-byte aligned. - */ - for (u32 i = 0; i < new_plane_state->fb->format->num_planes; i++) - if (new_plane_state->fb->pitches[i] & 63) - return -EINVAL; - - /* - * FIXME: dcp can currently only use multi-planar buffers using the same - * object for all planes. It has a mandatory iommu so it should - * be no problem to map multiple objects "linearly" into DCP - * virtual address space and calculate the offsets accordingly. - * Or maybe it can accept multiple BOs via the per plane field - * `base`. - */ - if (new_plane_state->fb->format->num_planes > 1) { - const struct drm_gem_object *first = new_plane_state->fb->obj[0]; - for (u32 i = 1; i < new_plane_state->fb->format->num_planes; i++) - if (new_plane_state->fb->obj[i] != NULL && - new_plane_state->fb->obj[i] != first) - return -EINVAL; - } - - return 0; -} - -/* - * DRM specifies rectangles as start and end coordinates. DCP specifies - * rectangles as a start coordinate and a width/height. Convert a DRM rectangle - * to a DCP rectangle. - */ -static struct neo_dcp_rect drm_to_dcp_rect(const struct drm_rect *rect) -{ - return (struct neo_dcp_rect){ .x = rect->x1, - .y = rect->y1, - .w = drm_rect_width(rect), - .h = drm_rect_height(rect), - }; -} - -static struct neo_dcp_rect drm_to_dcp_rect_fp(const struct drm_rect *fp_rect) -{ - struct drm_rect rect; - drm_rect_fp_to_int(&rect, fp_rect); - return drm_to_dcp_rect(&rect); -} - -static u32 drm_format_to_dcp(u32 drm, enum drm_color_range range) -{ - bool fr = range == DRM_COLOR_YCBCR_FULL_RANGE; - switch (drm) { - case DRM_FORMAT_XRGB8888: - case DRM_FORMAT_ARGB8888: - return DCP_FORMAT_BGRA; - - case DRM_FORMAT_XBGR8888: - case DRM_FORMAT_ABGR8888: - return DCP_FORMAT_RGBA; - - case DRM_FORMAT_XRGB2101010: - case DRM_FORMAT_ARGB2101010: - return DCP_FORMAT_L10R; - - /* semi planar YCbCr formats, limited and full range */ - case DRM_FORMAT_NV12: - return fr ? DCP_FORMAT_420F : DCP_FORMAT_420V; - case DRM_FORMAT_NV16: - return fr ? DCP_FORMAT_422F : DCP_FORMAT_422V; - case DRM_FORMAT_NV24: - return fr ? DCP_FORMAT_444F : DCP_FORMAT_444V; - - /* semi planar 10-bit YCbCr formats, limited and full range */ - case DRM_FORMAT_P010: - return fr ? DCP_FORMAT_XF20 : DCP_FORMAT_X420; - case DRM_FORMAT_P210: - return fr ? DCP_FORMAT_XF22 : DCP_FORMAT_X422; - /* - * TODO: missing DRM fourcc for P410 - */ -#if defined(DRM_FORMAT_P410) - case DRM_FORMAT_P410: - return fr ? DCP_FORMAT_XF44 : DCP_FORMAT_X444; -#endif - } - - pr_warn("DRM format %X not supported in DCP\n", drm); - return 0; -} - -static enum neo_dcp_xfer_func get_xfer_func(bool is_yuv, enum drm_color_encoding enc) -{ - if (!is_yuv) - return DCP_XFER_FUNC_SDR; - - switch (enc) { - case DRM_COLOR_YCBCR_BT601: - return DCP_XFER_FUNC_BT601; - case DRM_COLOR_YCBCR_BT709: - case DRM_COLOR_YCBCR_BT2020: - return DCP_XFER_FUNC_BT1886; - default: - return DCP_XFER_FUNC_SDR; - } -} - -static enum neo_dcp_colorspace get_colorspace(bool is_yuv, - enum drm_color_encoding enc) -{ - if (!is_yuv) - return DCP_COLORSPACE_NATIVE; - - switch (enc) { - case DRM_COLOR_YCBCR_BT601: - return DCP_COLORSPACE_BT601; - case DRM_COLOR_YCBCR_BT709: - return DCP_COLORSPACE_BT709; - case DRM_COLOR_YCBCR_BT2020: - return DCP_COLORSPACE_BG_BT2020; - default: - return DCP_COLORSPACE_NATIVE; - } -} - -static void neo_apple_plane_atomic_update(struct drm_plane *plane, - struct drm_atomic_state *state) -{ - struct drm_plane_state *base = drm_atomic_get_new_plane_state(state, plane); - struct neo_apple_plane_state *new_state; - struct drm_gem_dma_object *obj; - bool is_premultiplied = false; - - if (!base) - return; - - new_state = to_apple_plane_state(base); - - if (!base->fb) { - memset(&new_state->surf, 0, sizeof(new_state->surf)); - return; - } - - struct drm_framebuffer *fb = base->fb; - const struct drm_format_info *fmt = fb->format; - /* - * DCP doesn't support XBGR8 / XRGB8 / XBGR2101010 natively. Blending as - * pre-multiplied alpha with a black background can be used as - * workaround for the bottommost plane. - */ - if (fmt->format == DRM_FORMAT_XRGB8888 || - fmt->format == DRM_FORMAT_XBGR8888 || - fmt->format == DRM_FORMAT_XBGR2101010) - is_premultiplied = true; - - new_state->src_rect = drm_to_dcp_rect_fp(&base->src); - new_state->dst_rect = drm_to_dcp_rect(&base->dst); - - new_state->surf = (struct neo_dcp_surface){ - .is_premultiplied = is_premultiplied, - .plane_cnt = fb->format->num_planes, - .plane_cnt2 = fb->format->num_planes, - .format = drm_format_to_dcp(fmt->format, base->color_range), - .xfer_func = get_xfer_func(fmt->is_yuv, base->color_encoding), - .colorspace = get_colorspace(fmt->is_yuv, base->color_encoding), - .stride = fb->pitches[0], - .width = fb->width, - .height = fb->height, - .buf_size = fb->height * fb->pitches[0], - // .surface_id = req->swap.surf_ids[l], - - /* Only used for compressed or multiplanar surfaces */ - .pix_size = 1, - .pel_w = 1, - .pel_h = 1, - .has_comp = 1, - .has_planes = 1, - }; - - /* Populate plane information for planar formats */ - struct neo_dcp_surface *surf = &new_state->surf; - for (int i = 0; fb->format->num_planes && i < fb->format->num_planes; i++) { - u32 width = drm_format_info_plane_width(fb->format, fb->width, i); - u32 height = drm_format_info_plane_height(fb->format, fb->height, i); - u32 bh = drm_format_info_block_height(fb->format, i); - u32 bw = drm_format_info_block_width(fb->format, i); - - surf->planes[i] = (struct neo_dcp_plane_info){ - .width = width, - .height = height, - .base = fb->offsets[i] - fb->offsets[0], - .offset = fb->offsets[i] - fb->offsets[0], - .stride = fb->pitches[i], - .size = height * fb->pitches[i], - .tile_size = bw * bh, - .tile_w = bw, - .tile_h = bh, - }; - - if (i > 0) - surf->buf_size += surf->planes[i].size; - } - - /* the obvious helper call drm_fb_dma_get_gem_addr() adjusts - * the address for source x/y offsets. Since IOMFB has a direct - * support source position prefer that. - */ - obj = drm_fb_dma_get_gem_obj(base->fb, 0); - if (obj) - new_state->iova = obj->dma_addr + base->fb->offsets[0]; -} - -static void neo_apple_plane_cleanup_fb(struct drm_plane *plane, - struct drm_plane_state *state) -{ - struct neo_apple_plane_state *apple_state = to_apple_plane_state(state); - struct neo_dcp_fb_reference *entry = apple_state->retirement; - - if (!entry) - return; - - apple_state->retirement = NULL; - drm_framebuffer_put(entry->fb); - kfree(entry); -} - -static int neo_apple_plane_prepare_fb(struct drm_plane *plane, - struct drm_plane_state *state) -{ - struct drm_plane_state *old; - struct neo_dcp_fb_reference *entry; - int ret; - - ret = drm_gem_plane_helper_prepare_fb(plane, state); - if (ret) - return ret; - - old = drm_atomic_get_old_plane_state(state->state, plane); - if (!old || !old->crtc || !old->fb || old->fb == state->fb) - return 0; - - entry = kzalloc_obj(*entry); - if (!entry) - return -ENOMEM; - - drm_framebuffer_get(old->fb); - entry->fb = old->fb; - to_apple_plane_state(state)->retirement = entry; - return 0; -} - -static const struct drm_plane_helper_funcs apple_primary_plane_helper_funcs = { - .prepare_fb = neo_apple_plane_prepare_fb, - .cleanup_fb = neo_apple_plane_cleanup_fb, - .atomic_check = neo_apple_plane_atomic_check, - .atomic_update = neo_apple_plane_atomic_update, - .get_scanout_buffer = drm_fb_dma_get_scanout_buffer, -}; - -static const struct drm_plane_helper_funcs neo_apple_plane_helper_funcs = { - .prepare_fb = neo_apple_plane_prepare_fb, - .cleanup_fb = neo_apple_plane_cleanup_fb, - .atomic_check = neo_apple_plane_atomic_check, - .atomic_update = neo_apple_plane_atomic_update, -}; - -// Duplicate drm_atomic_helper_plane_reset but allocate struct apple_plane_state -static void neo_apple_plane_reset(struct drm_plane *plane) -{ - struct neo_apple_plane_state *state = to_apple_plane_state(plane->state); - if (state) { - neo_apple_plane_cleanup_fb(plane, &state->base); - __drm_atomic_helper_plane_destroy_state(&state->base); - } - - kfree(state); - plane->state = NULL; - state = kzalloc(sizeof(*state), GFP_KERNEL); - if (state) - __drm_atomic_helper_plane_reset(plane, &state->base); -} - -static struct drm_plane_state * -neo_apple_plane_duplicate_state(struct drm_plane *plane) -{ - struct neo_apple_plane_state *neo_apple_plane_state, *old_apple_plane_state; - - if (!plane->state) - return NULL; - - old_apple_plane_state = to_apple_plane_state(plane->state); - neo_apple_plane_state = kzalloc(sizeof(*neo_apple_plane_state), GFP_KERNEL); - if (!neo_apple_plane_state) - return NULL; - - __drm_atomic_helper_plane_duplicate_state(plane, &neo_apple_plane_state->base); - - neo_apple_plane_state->surf = old_apple_plane_state->surf; - neo_apple_plane_state->src_rect = old_apple_plane_state->src_rect; - neo_apple_plane_state->dst_rect = old_apple_plane_state->dst_rect; - neo_apple_plane_state->iova = old_apple_plane_state->iova; - - return &neo_apple_plane_state->base; -} - -static void neo_apple_plane_destroy_state(struct drm_plane *plane, - struct drm_plane_state *state) -{ - neo_apple_plane_cleanup_fb(plane, state); - drm_atomic_helper_plane_destroy_state(plane, state); -} - -static const struct drm_plane_funcs neo_apple_plane_funcs = { - .update_plane = drm_atomic_helper_update_plane, - .disable_plane = drm_atomic_helper_disable_plane, - .reset = neo_apple_plane_reset, - .atomic_duplicate_state = neo_apple_plane_duplicate_state, - .atomic_destroy_state = neo_apple_plane_destroy_state, -}; - -/* - * Table of supported formats, mapping from DRM fourccs to DCP fourccs. - * - * For future work, DCP supports more formats not listed, including YUV - * formats, an extra RGBA format, and a biplanar RGB10_A8 format (fourcc b3a8) - * used for HDR. - * - * Note: we don't have non-alpha formats but userspace breaks without XRGB. It - * doesn't matter for the primary plane, but cursors/overlays must not - * advertise formats without alpha. - */ -static const u32 neo_dcp_primary_formats[] = { - DRM_FORMAT_XRGB2101010, - DRM_FORMAT_ARGB2101010, - DRM_FORMAT_XRGB8888, - DRM_FORMAT_ARGB8888, - DRM_FORMAT_XBGR8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_NV12, - DRM_FORMAT_NV16, - DRM_FORMAT_NV24, - DRM_FORMAT_P010, - DRM_FORMAT_P210, -#if defined(DRM_FORMAT_P410) - DRM_FORMAT_P410, -#endif -}; - -/* H17P 10-bit XRGB alpha semantics have not been qualified. */ -static const u32 neo_dcp_primary_formats_h17p[] = { - DRM_FORMAT_ARGB2101010, - DRM_FORMAT_XRGB8888, - DRM_FORMAT_ARGB8888, - DRM_FORMAT_XBGR8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_NV12, - DRM_FORMAT_NV16, - DRM_FORMAT_NV24, - DRM_FORMAT_P010, - DRM_FORMAT_P210, -#if defined(DRM_FORMAT_P410) - DRM_FORMAT_P410, -#endif -}; - -static const u32 neo_dcp_overlay_formats[] = { - DRM_FORMAT_ARGB2101010, - DRM_FORMAT_ARGB8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_NV12, - DRM_FORMAT_NV16, - DRM_FORMAT_NV24, - DRM_FORMAT_P010, - DRM_FORMAT_P210, -#if defined(DRM_FORMAT_P410) - DRM_FORMAT_P410, -#endif -}; - -/* - * Formats for the 12.x firmware which does not support "l10r" / ARGB2101010 - */ -static const u32 neo_dcp_primary_formats_12_x[] = { - DRM_FORMAT_XRGB8888, - DRM_FORMAT_ARGB8888, - DRM_FORMAT_XBGR8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_NV12, - DRM_FORMAT_NV16, - DRM_FORMAT_NV24, - DRM_FORMAT_P010, - DRM_FORMAT_P210, -#if defined(DRM_FORMAT_P410) - DRM_FORMAT_P410, -#endif -}; - -static const u32 neo_dcp_overlay_formats_12_x[] = { - DRM_FORMAT_ARGB8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_NV12, - DRM_FORMAT_NV16, - DRM_FORMAT_NV24, - DRM_FORMAT_P010, - DRM_FORMAT_P210, -#if defined(DRM_FORMAT_P410) - DRM_FORMAT_P410, -#endif -}; - -u64 neo_apple_format_modifiers[] = { - DRM_FORMAT_MOD_LINEAR, - DRM_FORMAT_MOD_INVALID -}; - -struct drm_plane *neo_apple_plane_init(struct drm_device *dev, - unsigned long possible_crtcs, - u32 neo_iomfb_surf, - bool supports_l10r, - bool supports_xrgb2101010, - enum drm_plane_type type) -{ - struct neo_apple_plane *plane; - const u32 *fmts; - u32 num_fmts; - - switch (type) { - case DRM_PLANE_TYPE_PRIMARY: - if (supports_l10r && supports_xrgb2101010) { - fmts = neo_dcp_primary_formats; - num_fmts = ARRAY_SIZE(neo_dcp_primary_formats); - } else if (supports_l10r) { - fmts = neo_dcp_primary_formats_h17p; - num_fmts = ARRAY_SIZE(neo_dcp_primary_formats_h17p); - } else { - fmts = neo_dcp_primary_formats_12_x; - num_fmts = ARRAY_SIZE(neo_dcp_primary_formats_12_x); - } - plane = drmm_universal_plane_alloc(dev, struct neo_apple_plane, base, possible_crtcs, - &neo_apple_plane_funcs, fmts, num_fmts, - neo_apple_format_modifiers, type, NULL); - break; - case DRM_PLANE_TYPE_OVERLAY: - case DRM_PLANE_TYPE_CURSOR: - if (supports_l10r) { - fmts = neo_dcp_overlay_formats; - num_fmts = ARRAY_SIZE(neo_dcp_overlay_formats); - } else { - fmts = neo_dcp_overlay_formats_12_x; - num_fmts = ARRAY_SIZE(neo_dcp_overlay_formats_12_x); - } - plane = drmm_universal_plane_alloc(dev, struct neo_apple_plane, base, possible_crtcs, - &neo_apple_plane_funcs, fmts, num_fmts, - neo_apple_format_modifiers, type, NULL); - break; - default: - return ERR_PTR(-EINVAL); - } - - if (IS_ERR(plane)) - return ERR_PTR(PTR_ERR(plane)); - - drm_plane_create_color_properties(&plane->base, - (1 << DRM_COLOR_ENCODING_MAX) - 1, - (1 << DRM_COLOR_RANGE_MAX) - 1, - DRM_COLOR_YCBCR_BT709, - DRM_COLOR_YCBCR_LIMITED_RANGE); - - if (type == DRM_PLANE_TYPE_PRIMARY) - drm_plane_helper_add(&plane->base, &apple_primary_plane_helper_funcs); - else - drm_plane_helper_add(&plane->base, &neo_apple_plane_helper_funcs); - - plane->neo_iomfb_surf = neo_iomfb_surf; - - return &plane->base; -} diff --git a/drivers/gpu/drm/apple-neo/plane.h b/drivers/gpu/drm/apple-neo/plane.h deleted file mode 100644 index ed8080a29915cc..00000000000000 --- a/drivers/gpu/drm/apple-neo/plane.h +++ /dev/null @@ -1,42 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * Copyright (C) The Asahi Linux Contributors - */ - -#ifndef __APPLE_PLANE_H__ -#define __APPLE_PLANE_H__ - -#include - -#include - -#include "iomfb_plane.h" - -struct neo_apple_plane { - struct drm_plane base; - u32 neo_iomfb_surf; -}; - -#define to_apple_plane(x) container_of(x, struct neo_apple_plane, base) - -struct neo_dcp_fb_reference; - -struct neo_apple_plane_state { - struct drm_plane_state base; - struct neo_dcp_surface surf; - struct neo_dcp_rect src_rect; - struct neo_dcp_rect dst_rect; - u64 iova; - struct neo_dcp_fb_reference *retirement; -}; - -#define to_apple_plane_state(x) container_of(x, struct neo_apple_plane_state, base) - -struct drm_plane *neo_apple_plane_init(struct drm_device *dev, - unsigned long possible_crtcs, - u32 neo_iomfb_surf, - bool supports_l10r, - bool supports_xrgb2101010, - enum drm_plane_type type); - -#endif /* __APPLE_PLANE_H__ */ diff --git a/drivers/gpu/drm/apple-neo/systemep.c b/drivers/gpu/drm/apple-neo/systemep.c deleted file mode 100644 index 38a8f28b9c4537..00000000000000 --- a/drivers/gpu/drm/apple-neo/systemep.c +++ /dev/null @@ -1,150 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright 2022 Sven Peter */ - -#include - -#include "afk.h" -#include "dcp.h" -#include "parser.h" - -static bool enable_verbose_logging; -module_param(enable_verbose_logging, bool, 0644); -MODULE_PARM_DESC(enable_verbose_logging, "Enable DCP firmware verbose logging"); - -/* - * Serialized setProperty("gAFKConfigLogMask", 0xffff) IPC call which - * will set the DCP firmware log level to the most verbose setting - */ -#define SYSTEM_SET_PROPERTY 0x43 -static const u8 setprop_gAFKConfigLogMask_ffff[] = { - 0x14, 0x00, 0x00, 0x00, 0x67, 0x41, 0x46, 0x4b, 0x43, 0x6f, - 0x6e, 0x66, 0x69, 0x67, 0x4c, 0x6f, 0x67, 0x4d, 0x61, 0x73, - 0x6b, 0x00, 0x00, 0x00, 0xd3, 0x00, 0x00, 0x00, 0x40, 0x00, - 0x00, 0x84, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 -}; - -struct neo_systemep_work { - struct neo_apple_epic_service *service; - struct work_struct work; -}; - -static void system_log_work(struct work_struct *work_) -{ - struct neo_systemep_work *work = - container_of(work_, struct neo_systemep_work, work); - - neo_afk_send_command(work->service, SYSTEM_SET_PROPERTY, - setprop_gAFKConfigLogMask_ffff, - sizeof(setprop_gAFKConfigLogMask_ffff), NULL, - sizeof(setprop_gAFKConfigLogMask_ffff), NULL); - complete(&work->service->ep->neo_dcp->neo_systemep_done); - kfree(work); -} - -static void system_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ - struct neo_systemep_work *work; - - if (!enable_verbose_logging) - return; - - /* - * We're called from the service message handler thread and can't - * dispatch blocking message from there. - */ - work = kzalloc(sizeof(*work), GFP_KERNEL); - if (!work) - return; - - work->service = service; - INIT_WORK(&work->work, system_log_work); - schedule_work(&work->work); -} - -static void powerlog_init(struct neo_apple_epic_service *service, const char *name, - const char *class, s64 unit) -{ -} - -static int powerlog_report(struct neo_apple_epic_service *service, enum neo_epic_subtype type, - const void *data, size_t data_size) -{ - struct neo_dcp_system_ev_mnits mnits; - struct neo_dcp_parse_ctx neo_parse_ctx; - struct neo_apple_dcp *neo_dcp = service->ep->neo_dcp; - int ret; - - /* Later reports must not overwrite a latched hint or Linux's target. */ - if (neo_dcp_backlight_active(neo_dcp)) - return 0; - - dev_dbg(neo_dcp->dev, "systemep[ch:%u]: report type:%02x len:%zu\n", - service->channel, type, data_size); - - if (type != EPIC_SUBTYPE_STD_SERVICE) - return 0; - - ret = neo_parse(data, data_size, &neo_parse_ctx); - if (ret) { - dev_warn(service->ep->neo_dcp->dev, "systemep: failed to parse report: %d\n", ret); - return ret; - } - - ret = neo_parse_system_log_mnits(&neo_parse_ctx, &mnits); - if (ret) { - /* ignore parse errors in the case dcp sends unknown log events */ - dev_dbg(neo_dcp->dev, "systemep: failed to parse mNits event: %d\n", ret); - return 0; - } - - dev_dbg(neo_dcp->dev, "systemep: mNits event: Nits: %u.%03u, iDAC: %u\n", - mnits.millinits / 1000, mnits.millinits % 1000, mnits.idac); - if (neo_dcp->fw_compat == DCP_FIRMWARE_H17P && - neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - if (!neo_dcp_has_panel(neo_dcp)) - return 0; - ret = neo_dcp_backlight_takeover(neo_dcp, mnits.millinits); - if (ret && ret != -EBUSY) - dev_dbg(neo_dcp->dev, "backlight takeover hint unavailable: %d\n", ret); - return 0; - } - - neo_dcp->brightness.nits = mnits.millinits / 1000; - - return 0; -} - -static const struct neo_apple_epic_service_ops neo_systemep_ops[] = { - { - .name = "system", - .init = system_init, - }, - { - .name = "powerlog-service", - .init = powerlog_init, - .report = powerlog_report, - }, - {} -}; - -int neo_systemep_init(struct neo_apple_dcp *neo_dcp) -{ - init_completion(&neo_dcp->neo_systemep_done); - - neo_dcp->neo_systemep = neo_afk_init(neo_dcp, SYSTEM_ENDPOINT, neo_systemep_ops); - neo_afk_start(neo_dcp->neo_systemep); - - if (!enable_verbose_logging) - return 0; - - /* - * Timeouts aren't really fatal here: in the worst case we just weren't - * able to enable additional debug prints inside DCP - */ - if (!wait_for_completion_timeout(&neo_dcp->neo_systemep_done, - msecs_to_jiffies(MSEC_PER_SEC))) - dev_err(neo_dcp->dev, "systemep: couldn't enable verbose logs\n"); - - return 0; -} diff --git a/drivers/gpu/drm/apple-neo/trace.c b/drivers/gpu/drm/apple-neo/trace.c deleted file mode 100644 index 6f40d5a583df01..00000000000000 --- a/drivers/gpu/drm/apple-neo/trace.c +++ /dev/null @@ -1,9 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0+ OR MIT -/* - * Tracepoints for Apple DCP driver - * - * Copyright (C) The Asahi Linux Contributors - */ - -#define CREATE_TRACE_POINTS -#include "trace.h" diff --git a/drivers/gpu/drm/apple-neo/trace.h b/drivers/gpu/drm/apple-neo/trace.h deleted file mode 100644 index 1cfeeaaebe7ae5..00000000000000 --- a/drivers/gpu/drm/apple-neo/trace.h +++ /dev/null @@ -1,629 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright (C) The Asahi Linux Contributors */ - -#undef TRACE_SYSTEM -#define TRACE_SYSTEM neo_dcp - -#if !defined(_TRACE_DCP_H) || defined(TRACE_HEADER_MULTI_READ) -#define _TRACE_DCP_H - -#include "afk.h" -#include "dptxep.h" -#include "dcp-internal.h" -#include "parser.h" - -#include -#include -#include - -#define show_dcp_endpoint(ep) \ - __print_symbolic(ep, { SYSTEM_ENDPOINT, "system" }, \ - { TEST_ENDPOINT, "test" }, \ - { DCP_EXPERT_ENDPOINT, "dcpexpert" }, \ - { DISP0_ENDPOINT, "disp0" }, \ - { DPTX_ENDPOINT, "dptxport" }, \ - { HDCP_ENDPOINT, "hdcp" }, \ - { REMOTE_ALLOC_ENDPOINT, "remotealloc" }, \ - { IOMFB_ENDPOINT, "iomfb" }) -#define print_epic_type(etype) \ - __print_symbolic(etype, { EPIC_TYPE_NOTIFY, "notify" }, \ - { EPIC_TYPE_COMMAND, "command" }, \ - { EPIC_TYPE_REPLY, "reply" }, \ - { EPIC_TYPE_NOTIFY_ACK, "notify-ack" }) - -#define print_epic_category(ecat) \ - __print_symbolic(ecat, { EPIC_CAT_REPORT, "report" }, \ - { EPIC_CAT_NOTIFY, "notify" }, \ - { EPIC_CAT_REPLY, "reply" }, \ - { EPIC_CAT_COMMAND, "command" }) - -#define show_dptxport_apcall(idx) \ - __print_symbolic( \ - idx, { DPTX_APCALL_ACTIVATE, "activate" }, \ - { DPTX_APCALL_DEACTIVATE, "deactivate" }, \ - { DPTX_APCALL_GET_MAX_DRIVE_SETTINGS, \ - "get_max_drive_settings" }, \ - { DPTX_APCALL_SET_DRIVE_SETTINGS, "set_drive_settings" }, \ - { DPTX_APCALL_GET_DRIVE_SETTINGS, "get_drive_settings" }, \ - { DPTX_APCALL_WILL_CHANGE_LINKG_CONFIG, \ - "will_change_link_config" }, \ - { DPTX_APCALL_DID_CHANGE_LINK_CONFIG, \ - "did_change_link_config" }, \ - { DPTX_APCALL_GET_MAX_LINK_RATE, "get_max_link_rate" }, \ - { DPTX_APCALL_GET_LINK_RATE, "get_link_rate" }, \ - { DPTX_APCALL_SET_LINK_RATE, "set_link_rate" }, \ - { DPTX_APCALL_GET_MAX_LANE_COUNT, \ - "get_max_lane_count" }, \ - { DPTX_APCALL_GET_ACTIVE_LANE_COUNT, \ - "get_active_lane_count" }, \ - { DPTX_APCALL_SET_ACTIVE_LANE_COUNT, \ - "set_active_lane_count" }, \ - { DPTX_APCALL_GET_SUPPORTS_DOWN_SPREAD, \ - "get_supports_downspread" }, \ - { DPTX_APCALL_GET_DOWN_SPREAD, "get_downspread" }, \ - { DPTX_APCALL_SET_DOWN_SPREAD, "set_downspread" }, \ - { DPTX_APCALL_GET_SUPPORTS_LANE_MAPPING, \ - "get_supports_lane_mapping" }, \ - { DPTX_APCALL_SET_LANE_MAP, "set_lane_map" }, \ - { DPTX_APCALL_GET_SUPPORTS_HPD, "get_supports_hpd" }, \ - { DPTX_APCALL_FORCE_HOTPLUG_DETECT, "force_hotplug_detect" }, \ - { DPTX_APCALL_INACTIVE_SINK_DETECTED, \ - "inactive_sink_detected" }, \ - { DPTX_APCALL_SET_TILED_DISPLAY_HINTS, \ - "set_tiled_display_hints" }, \ - { DPTX_APCALL_DEVICE_NOT_RESPONDING, \ - "device_not_responding" }, \ - { DPTX_APCALL_DEVICE_BUSY_TIMEOUT, "device_busy_timeout" }, \ - { DPTX_APCALL_DEVICE_NOT_STARTED, "device_not_started" }) - -TRACE_EVENT(neo_dcp_recv_msg, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u8 endpoint, u64 message), - TP_ARGS(neo_dcp, endpoint, message), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dcp->dev)) - __field(u8, endpoint) - __field(u64, message)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = endpoint; - __entry->message = message;), - - TP_printk("%s: endpoint 0x%x (%s): received message 0x%016llx", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->message)); - -TRACE_EVENT(neo_dcp_send_msg, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u8 endpoint, u64 message), - TP_ARGS(neo_dcp, endpoint, message), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dcp->dev)) - __field(u8, endpoint) - __field(u64, message)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = endpoint; - __entry->message = message;), - - TP_printk("%s: endpoint 0x%x (%s): will send message 0x%016llx", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->message)); - -TRACE_EVENT( - neo_afk_getbuf, TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 size, u16 tag), - TP_ARGS(ep, size, tag), - - TP_STRUCT__entry(__string(devname, dev_name(ep->neo_dcp->dev)) - __field(u8, endpoint) __field(u32, size) - __field(u16, tag)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = ep->endpoint; __entry->size = size; - __entry->tag = tag;), - - TP_printk( - "%s: endpoint 0x%x (%s): get buffer with size 0x%x and tag 0x%x", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->size, - __entry->tag)); - -DECLARE_EVENT_CLASS(neo_afk_rwptr_template, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 wptr), - TP_ARGS(ep, rptr, wptr), - - TP_STRUCT__entry(__string(devname, dev_name(ep->neo_dcp->dev)) - __field(u8, endpoint) __field(u32, rptr) - __field(u32, wptr)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = ep->endpoint; - __entry->rptr = rptr; __entry->wptr = wptr;), - - TP_printk("%s: endpoint 0x%x (%s): rptr 0x%x, wptr 0x%x", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->rptr, - __entry->wptr)); - -DEFINE_EVENT(neo_afk_rwptr_template, neo_afk_recv_rwptr_pre, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 wptr), - TP_ARGS(ep, rptr, wptr)); -DEFINE_EVENT(neo_afk_rwptr_template, neo_afk_recv_rwptr_post, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 wptr), - TP_ARGS(ep, rptr, wptr)); -DEFINE_EVENT(neo_afk_rwptr_template, neo_afk_send_rwptr_pre, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 wptr), - TP_ARGS(ep, rptr, wptr)); -DEFINE_EVENT(neo_afk_rwptr_template, neo_afk_send_rwptr_post, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 wptr), - TP_ARGS(ep, rptr, wptr)); - -TRACE_EVENT( - neo_afk_recv_qe, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 rptr, u32 magic, u32 size), - TP_ARGS(ep, rptr, magic, size), - - TP_STRUCT__entry(__string(devname, dev_name(ep->neo_dcp->dev)) - __field(u8, endpoint) __field(u32, rptr) - __field(u32, magic) - __field(u32, size)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = ep->endpoint; __entry->rptr = rptr; - __entry->magic = magic; __entry->size = size;), - - TP_printk("%s: endpoint 0x%x (%s): QE rptr 0x%x, magic 0x%x, size 0x%x", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->rptr, - __entry->magic, __entry->size)); - -TRACE_EVENT( - neo_afk_recv_handle, - TP_PROTO(struct neo_apple_dcp_afkep *ep, u32 channel, u32 type, - u32 data_size, struct neo_epic_hdr *ehdr, - struct neo_epic_sub_hdr *eshdr), - TP_ARGS(ep, channel, type, data_size, ehdr, eshdr), - - TP_STRUCT__entry(__string(devname, dev_name(ep->neo_dcp->dev)) __field( - u8, endpoint) __field(u32, channel) __field(u32, type) - __field(u32, data_size) __field(u8, category) - __field(u16, subtype) - __field(u16, tag)), - - TP_fast_assign(__assign_str(devname); - __entry->endpoint = ep->endpoint; - __entry->channel = channel; __entry->type = type; - __entry->data_size = data_size; - __entry->category = eshdr->category, - __entry->subtype = le16_to_cpu(eshdr->type), - __entry->tag = le16_to_cpu(eshdr->tag)), - - TP_printk( - "%s: endpoint 0x%x (%s): channel 0x%x, type 0x%x (%s), data_size 0x%x, category: 0x%x (%s), subtype: 0x%x, seq: 0x%x", - __get_str(devname), __entry->endpoint, - show_dcp_endpoint(__entry->endpoint), __entry->channel, - __entry->type, print_epic_type(__entry->type), - __entry->data_size, __entry->category, - print_epic_category(__entry->category), __entry->subtype, - __entry->tag)); - -TRACE_EVENT(neo_iomfb_callback, - TP_PROTO(struct neo_apple_dcp *neo_dcp, int tag, const char *name), - TP_ARGS(neo_dcp, tag, name), - - TP_STRUCT__entry( - __string(devname, dev_name(neo_dcp->dev)) - __field(int, tag) - __field(const char *, name) - ), - - TP_fast_assign( - __assign_str(devname); - __entry->tag = tag; __entry->name = name; - ), - - TP_printk("%s: Callback D%03d %s", __get_str(devname), __entry->tag, - __entry->name)); - -TRACE_EVENT(neo_iomfb_push, - TP_PROTO(struct neo_apple_dcp *neo_dcp, - const struct neo_dcp_method_entry *method, int context, - int offset, int depth), - TP_ARGS(neo_dcp, method, context, offset, depth), - - TP_STRUCT__entry( - __string(devname, dev_name(neo_dcp->dev)) - __string(name, method->name) - __field(int, context) - __field(int, offset) - __field(int, depth)), - - TP_fast_assign( - __assign_str(devname); - __assign_str(name); - __entry->context = context; __entry->offset = offset; - __entry->depth = depth; - ), - - TP_printk("%s: Method %s: context %u, offset %u, depth %u", - __get_str(devname), __get_str(name), __entry->context, - __entry->offset, __entry->depth)); - -TRACE_EVENT(neo_iomfb_swap_submit, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 swap_id), - TP_ARGS(neo_dcp, swap_id), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, swap_id) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->swap_id = swap_id; - ), - TP_printk("dcp=%llx, swap_id=%d", - __entry->neo_dcp, - __entry->swap_id) -); - -TRACE_EVENT(neo_iomfb_vrr_timestamps, - TP_PROTO(struct neo_apple_dcp *neo_dcp, bool enabled, u64 timestamp, - u64 submit_timestamp), - TP_ARGS(neo_dcp, enabled, timestamp, submit_timestamp), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(bool, enabled) - __field(u64, timestamp) - __field(u64, submit_timestamp) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->enabled = enabled; - __entry->timestamp = timestamp; - __entry->submit_timestamp = submit_timestamp; - ), - TP_printk("dcp=%llx, enabled=%u, timestamp=%llu, previous=%llu", - __entry->neo_dcp, __entry->enabled, __entry->timestamp, - __entry->submit_timestamp) -); - -TRACE_EVENT(neo_iomfb_swap_complete, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 swap_id), - TP_ARGS(neo_dcp, swap_id), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, swap_id) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->swap_id = swap_id; - ), - TP_printk("dcp=%llx, swap_id=%d", - __entry->neo_dcp, - __entry->swap_id - ) -); - -TRACE_EVENT(neo_iomfb_swap_complete_intent_gated, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 swap_id, u32 width, u32 height), - TP_ARGS(neo_dcp, swap_id, width, height), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, swap_id) - __field(u32, width) - __field(u32, height) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->swap_id = swap_id; - __entry->height = height; - __entry->width = width; - ), - TP_printk("dcp=%llx, swap_id=%u %ux%u", - __entry->neo_dcp, - __entry->swap_id, - __entry->width, - __entry->height - ) -); - -TRACE_EVENT(neo_iomfb_abort_swap_ap_gated, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 swap_id), - TP_ARGS(neo_dcp, swap_id), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, swap_id) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->swap_id = swap_id; - ), - TP_printk("dcp=%llx, swap_id=%u", - __entry->neo_dcp, - __entry->swap_id - ) -); - -DECLARE_EVENT_CLASS(neo_iomfb_parse_mode_template, - TP_PROTO(s64 id, struct dimension *horiz, struct dimension *vert, s64 best_color_mode, bool is_virtual, s64 score), - TP_ARGS(id, horiz, vert, best_color_mode, is_virtual, score), - - TP_STRUCT__entry(__field(s64, id) - __field_struct(struct dimension, horiz) - __field_struct(struct dimension, vert) - __field(s64, best_color_mode) - __field(bool, is_virtual) - __field(s64, score)), - - TP_fast_assign(__entry->id = id; - __entry->horiz = *horiz; - __entry->vert = *vert; - __entry->best_color_mode = best_color_mode; - __entry->is_virtual = is_virtual; - __entry->score = score;), - - TP_printk("id: %lld, best_color_mode: %lld, resolution:%lldx%lld virtual: %d, score: %lld", - __entry->id, __entry->best_color_mode, - __entry->horiz.active, __entry->vert.active, - __entry->is_virtual, __entry->score)); - -DEFINE_EVENT(neo_iomfb_parse_mode_template, neo_iomfb_parse_mode_success, - TP_PROTO(s64 id, struct dimension *horiz, struct dimension *vert, s64 best_color_mode, bool is_virtual, s64 score), - TP_ARGS(id, horiz, vert, best_color_mode, is_virtual, score)); - -DEFINE_EVENT(neo_iomfb_parse_mode_template, neo_iomfb_parse_mode_fail, - TP_PROTO(s64 id, struct dimension *horiz, struct dimension *vert, s64 best_color_mode, bool is_virtual, s64 score), - TP_ARGS(id, horiz, vert, best_color_mode, is_virtual, score)); - -TRACE_EVENT(neo_dcpavserv_init, TP_PROTO(struct neo_apple_dcp *neo_dcp, u64 unit), - TP_ARGS(neo_dcp, unit), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dcp->dev)) - __field(u64, unit)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = unit;), - - TP_printk("%s: dcpav-service unit %lld initialized", __get_str(devname), - __entry->unit)); - -TRACE_EVENT(neo_dptxport_init, TP_PROTO(struct neo_apple_dcp *neo_dcp, u64 unit), - TP_ARGS(neo_dcp, unit), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dcp->dev)) - __field(u64, unit)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = unit;), - - TP_printk("%s: dptxport unit %lld initialized", __get_str(devname), - __entry->unit)); - -TRACE_EVENT( - neo_dptxport_apcall, - TP_PROTO(struct neo_dptx_port *neo_dptx, int idx, size_t len), - TP_ARGS(neo_dptx, idx, len), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dptx->service->ep->neo_dcp->dev)) - __field(u32, unit) __field(int, idx) __field(size_t, len)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = neo_dptx->unit; __entry->idx = idx; __entry->len = len;), - - TP_printk("%s: dptx%d: AP Call %d (%s) with len %lu", __get_str(devname), - __entry->unit, - __entry->idx, show_dptxport_apcall(__entry->idx), __entry->len)); - -TRACE_EVENT( - neo_dptxport_validate_connection, - TP_PROTO(struct neo_dptx_port *neo_dptx, u8 core, u8 atc, u8 die), - TP_ARGS(neo_dptx, core, atc, die), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dptx->service->ep->neo_dcp->dev)) - __field(u32, unit) __field(u8, core) __field(u8, atc) __field(u8, die)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = neo_dptx->unit; __entry->core = core; __entry->atc = atc; __entry->die = die;), - - TP_printk("%s: dptx%d: core %d, atc %d, die %d", __get_str(devname), - __entry->unit, __entry->core, __entry->atc, __entry->die)); - -TRACE_EVENT( - neo_dptxport_connect, - TP_PROTO(struct neo_dptx_port *neo_dptx, u8 core, u8 atc, u8 die), - TP_ARGS(neo_dptx, core, atc, die), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dptx->service->ep->neo_dcp->dev)) - __field(u32, unit) __field(u8, core) __field(u8, atc) __field(u8, die)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = neo_dptx->unit; __entry->core = core; __entry->atc = atc; __entry->die = die;), - - TP_printk("%s: dptx%d: core %d, atc %d, die %d", __get_str(devname), - __entry->unit, __entry->core, __entry->atc, __entry->die)); - -TRACE_EVENT( - neo_dptxport_call_set_link_rate, - TP_PROTO(struct neo_dptx_port *neo_dptx, u32 link_rate), - TP_ARGS(neo_dptx, link_rate), - - TP_STRUCT__entry(__string(devname, dev_name(neo_dptx->service->ep->neo_dcp->dev)) - __field(u32, unit) - __field(u32, link_rate)), - - TP_fast_assign(__assign_str(devname); - __entry->unit = neo_dptx->unit; - __entry->link_rate = link_rate;), - - TP_printk("%s: dptx%d: link rate 0x%x", __get_str(devname), __entry->unit, - __entry->link_rate)); - -TRACE_EVENT(neo_iomfb_brightness, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 nits), - TP_ARGS(neo_dcp, nits), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, nits) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->nits = nits; - ), - TP_printk("dcp=%llx, nits=%u (raw=0x%05x)", - __entry->neo_dcp, - __entry->nits >> 16, - __entry->nits - ) -); - -#define show_eotf(eotf) \ - __print_symbolic(eotf, { 0, "SDR gamma"}, \ - { 1, "HDR gamma"}, \ - { 2, "ST 2084 (PQ)"}, \ - { 3, "BT.2100 (HLG)"}, \ - { 4, "unexpected"}) - -#define show_encoding(enc) \ - __print_symbolic(enc, { 0, "RGB"}, \ - { 1, "YUV 4:2:0"}, \ - { 3, "YUV 4:2:2"}, \ - { 2, "YUV 4:4:4"}, \ - { 4, "DolbyVision (native)"}, \ - { 5, "DolbyVision (HDMI)"}, \ - { 6, "YCbCr 4:2:2 (DP tunnel)"}, \ - { 7, "YCbCr 4:2:2 (HDMI tunnel)"}, \ - { 8, "DolbyVision LL YCbCr 4:2:2"}, \ - { 9, "DolbyVision LL YCbCr 4:2:2 (DP)"}, \ - {10, "DolbyVision LL YCbCr 4:2:2 (HDMI)"}, \ - {11, "DolbyVision LL YCbCr 4:4:4"}, \ - {12, "DolbyVision LL RGB 4:2:2"}, \ - {13, "GRGB as YCbCr422 (Even line blue)"}, \ - {14, "GRGB as YCbCr422 (Even line red)"}, \ - {15, "unexpected"}) - -#define show_colorimetry(col) \ - __print_symbolic(col, { 0, "SMPTE 170M/BT.601"}, \ - { 1, "BT.701"}, \ - { 2, "xvYCC601"}, \ - { 3, "xvYCC709"}, \ - { 4, "sYCC601"}, \ - { 5, "AdobeYCC601"}, \ - { 6, "BT.2020 (c)"}, \ - { 7, "BT.2020 (nc)"}, \ - { 8, "DolbyVision VSVDB"}, \ - { 9, "BT.2020 (RGB)"}, \ - {10, "sRGB"}, \ - {11, "scRGB"}, \ - {12, "scRGBfixed"}, \ - {13, "AdobeRGB"}, \ - {14, "DCI-P3 (D65)"}, \ - {15, "DCI-P3 (Theater)"}, \ - {16, "Default RGB"}, \ - {17, "unexpected"}) - -#define show_range(range) \ - __print_symbolic(range, { 0, "Full"}, \ - { 1, "Limited"}, \ - { 2, "unexpected"}) - -TRACE_EVENT(neo_iomfb_color_mode, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 id, u32 score, u32 depth, - u32 colorimetry, u32 eotf, u32 range, u32 pixel_enc), - TP_ARGS(neo_dcp, id, score, depth, colorimetry, eotf, range, pixel_enc), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, id) - __field(u32, score) - __field(u32, depth) - __field(u32, colorimetry) - __field(u32, eotf) - __field(u32, range) - __field(u32, pixel_enc) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->id = id; - __entry->score = score; - __entry->depth = depth; - __entry->colorimetry = min_t(u32, colorimetry, 17U); - __entry->eotf = min_t(u32, eotf, 4U); - __entry->range = min_t(u32, range, 2U); - __entry->pixel_enc = min_t(u32, pixel_enc, 15U); - ), - TP_printk("dcp=%llx, id=%u, score=%u, depth=%u, colorimetry=%s, eotf=%s, range=%s, pixel_enc=%s", - __entry->neo_dcp, - __entry->id, - __entry->score, - __entry->depth, - show_colorimetry(__entry->colorimetry), - show_eotf(__entry->eotf), - show_range(__entry->range), - show_encoding(__entry->pixel_enc) - ) -); - -TRACE_EVENT(neo_iomfb_timing_mode, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 id, u32 score, u32 width, - u32 height, u32 clock, u32 color_mode), - TP_ARGS(neo_dcp, id, score, width, height, clock, color_mode), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, id) - __field(u32, score) - __field(u32, width) - __field(u32, height) - __field(u32, clock) - __field(u32, color_mode) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->id = id; - __entry->score = score; - __entry->width = width; - __entry->height = height; - __entry->clock = clock; - __entry->color_mode = color_mode; - ), - TP_printk("dcp=%llx, id=%u, score=%u, %ux%u@%u.%u, color_mode=%u", - __entry->neo_dcp, - __entry->id, - __entry->score, - __entry->width, - __entry->height, - __entry->clock >> 16, - ((__entry->clock & 0xffff) * 1000) >> 16, - __entry->color_mode - ) -); - -TRACE_EVENT(neo_avep_sound_mode, - TP_PROTO(struct neo_apple_dcp *neo_dcp, u32 rates, u64 formats, unsigned int nchans), - TP_ARGS(neo_dcp, rates, formats, nchans), - TP_STRUCT__entry( - __field(u64, neo_dcp) - __field(u32, rates) - __field(u64, formats) - __field(unsigned int, nchans) - ), - TP_fast_assign( - __entry->neo_dcp = (u64)neo_dcp; - __entry->rates = rates; - __entry->formats = formats; - __entry->nchans = nchans; - ), - TP_printk("dcp=%llx, rates=%#x, formats=%#llx, nchans=%#x", - __entry->neo_dcp, - __entry->rates, - __entry->formats, - __entry->nchans - ) -); - -#endif /* _TRACE_DCP_H */ - -/* This part must be outside protection */ - -#undef TRACE_INCLUDE_FILE -#define TRACE_INCLUDE_FILE trace - -#undef TRACE_INCLUDE_PATH -#define TRACE_INCLUDE_PATH . - -#include diff --git a/drivers/gpu/drm/apple-neo/version_utils.h b/drivers/gpu/drm/apple-neo/version_utils.h deleted file mode 100644 index 5a33ce1db61c47..00000000000000 --- a/drivers/gpu/drm/apple-neo/version_utils.h +++ /dev/null @@ -1,15 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* Copyright The Asahi Linux Contributors */ - -#ifndef __APPLE_VERSION_UTILS_H__ -#define __APPLE_VERSION_UTILS_H__ - -#include -#include - -#define DCP_FW_UNION(u) (u).DCP_FW -#define DCP_FW_SUFFIX CONCATENATE(_, DCP_FW) -#define DCP_FW_NAME(name) CONCATENATE(name, DCP_FW_SUFFIX) -#define DCP_FW_VERSION(x, y, z) ( ((x) << 16) | ((y) << 8) | (z) ) - -#endif /*__APPLE_VERSION_UTILS_H__*/ diff --git a/drivers/gpu/drm/apple/Kconfig b/drivers/gpu/drm/apple/Kconfig index ea7f24a22bc8a9..32173fef7127f3 100644 --- a/drivers/gpu/drm/apple/Kconfig +++ b/drivers/gpu/drm/apple/Kconfig @@ -35,8 +35,9 @@ config DRM_APPLE_DEBUG depends on EXPERT # only for developers config DRM_APPLE_KUNIT_TEST - tristate "KUnit tests for Apple display state" if !KUNIT_ALL_TESTS + tristate "KUnit tests for Apple display controller" if !KUNIT_ALL_TESTS depends on KUNIT + select DRM_KUNIT_TEST_HELPERS if DRM_APPLE default KUNIT_ALL_TESTS help Test modeset and hotplug state transitions without display hardware, @@ -44,6 +45,11 @@ config DRM_APPLE_KUNIT_TEST failed modesets. These tests can run on other architectures and do not require the Apple display driver to be enabled. Also test service ownership and pending commands across disconnect and reannouncement. + With the Apple display driver enabled, also build tests for its AFK + ring parser (header strides used by older firmware and larger rings, + malformed ring size combinations), the H17P backlight policy + (takeover, DPMS, suspend, fades and present completion) and the + system resume of a CRTC whose mode is no longer admissible. config DRM_APPLE_DISPLAY_KUNIT_TEST bool "KUnit tests for Apple display mode probing" if !KUNIT_ALL_TESTS diff --git a/drivers/gpu/drm/apple/Makefile b/drivers/gpu/drm/apple/Makefile index 2690627fcb4827..dfd6fde66c387f 100644 --- a/drivers/gpu/drm/apple/Makefile +++ b/drivers/gpu/drm/apple/Makefile @@ -2,12 +2,12 @@ CFLAGS_trace.o = -I$(src) -appledrm-y := apple_drv.o +appledrm-y := apple_drv.o apple_resume.o appledrm-y += afk.o dcp.o dcp-fabric.o dcp_backlight.o dptxep.o iomfb.o parser.o systemep.o appledrm-y += dcp-hdmi.o appledrm-$(CONFIG_DRM_APPLE_AUDIO) += audio.o appledrm-$(CONFIG_DRM_APPLE_AUDIO) += av.o -appledrm-y += connector.o +appledrm-y += connector.o dcp-display-metadata.o dcp-180hz.o appledrm-y += ibootep.o appledrm-y += dcpext_scanout.o appledrm-y += dcpext_drm.o @@ -16,6 +16,9 @@ appledrm-y += iomfb_v13_3.o appledrm-y += iomfb_v14_7_tpl.o appledrm-y += iomfb_v14_7.o iomfb_v14_7_link.o appledrm-y += iomfb_v26_6.o iomfb_v26_6_link.o +appledrm-y += iomfb_h17p.o +appledrm-$(CONFIG_DRM_APPLE_KUNIT_TEST) += iomfb_h17p_test.o +appledrm-$(CONFIG_DRM_APPLE_KUNIT_TEST) += apple_resume_test.o appledrm-y += epic/dpavservep.o appledrm-y += plane.o @@ -30,3 +33,7 @@ appledrm-$(CONFIG_DRM_APPLE_DISPLAY_KUNIT_TEST) += connector-edid-test.o appledrm-$(CONFIG_DRM_APPLE_DISPLAY_KUNIT_TEST) += hotplug-test.o appledrm-$(CONFIG_DRM_APPLE_DISPLAY_KUNIT_TEST) += external-hold-test.o appledrm-$(CONFIG_DRM_APPLE_DISPLAY_KUNIT_TEST) += v14-board-test.o + +# afk-service-test.o is built from drivers/gpu/drm/Makefile. +obj-$(CONFIG_DRM_APPLE_KUNIT_TEST) += afk_test.o +obj-$(CONFIG_DRM_APPLE_KUNIT_TEST) += dcp_backlight_test.o diff --git a/drivers/gpu/drm/apple/afk-service-test.c b/drivers/gpu/drm/apple/afk-service-test.c index 4a5f07fa9b15f2..75a352bec0481c 100644 --- a/drivers/gpu/drm/apple/afk-service-test.c +++ b/drivers/gpu/drm/apple/afk-service-test.c @@ -3,6 +3,7 @@ #include #include "afk.h" +#include "dptx-service.h" static const struct apple_epic_service_ops reusable_ops = { .name = "reusable", @@ -100,6 +101,23 @@ static void afk_service_completed_reader_test(struct kunit *test) KUNIT_EXPECT_TRUE(test, afk_service_reinit(service, NULL, &reusable_ops, 2)); } +static void afk_service_close_pin_test(struct kunit *test) +{ + struct apple_epic_service *service = test->priv; + + KUNIT_EXPECT_PTR_EQ(test, afk_service_pin(NULL), NULL); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(service, NULL, &reusable_ops, 1)); + KUNIT_ASSERT_PTR_EQ(test, afk_service_get(service), service); /* owner */ + afk_test_retire(service); + /* A close issued after teardown still pins the owner's service. */ + KUNIT_ASSERT_PTR_EQ(test, afk_service_pin(service), service); + afk_service_put(service); /* owner detaches */ + KUNIT_EXPECT_FALSE(test, afk_service_reinit(service, NULL, &reusable_ops, 2)); + KUNIT_EXPECT_EQ(test, service->channel, 1U); + afk_service_put(service); + KUNIT_EXPECT_TRUE(test, afk_service_reinit(service, NULL, &reusable_ops, 2)); +} + static void afk_service_tag_sequence_test(struct kunit *test) { struct apple_epic_service *service = test->priv; @@ -122,11 +140,110 @@ static void afk_service_unconverted_owner_test(struct kunit *test) KUNIT_EXPECT_FALSE(test, afk_service_reinit(service, NULL, &reusable_ops, 2)); } +static int afk_test_prepare(struct apple_epic_service *service, + const char *name, const char *class, s64 unit) +{ + return 0; +} + +static const struct apple_epic_service_ops prepared_ops = { + .name = "prepared", + .prepare = afk_test_prepare, + .teardown = dptx_service_unbind, +}; + +static void afk_service_admission_test(struct kunit *test) +{ + struct apple_epic_service *service = test->priv; + + KUNIT_ASSERT_TRUE(test, afk_service_reinit(service, NULL, &prepared_ops, 1)); + /* Neither callback dispatch nor command callers can observe preparation. */ + KUNIT_EXPECT_FALSE(test, afk_service_matches(service, 1)); + KUNIT_EXPECT_PTR_EQ(test, afk_service_get(service), NULL); + /* A rejected announcement or failed OPEN consumes no service slot. */ + afk_service_discard(service); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(service, NULL, &prepared_ops, 2)); + /* Cookie-less services are valid: admission does not test cookie != NULL. */ + afk_service_publish(service); + KUNIT_EXPECT_TRUE(test, afk_service_matches(service, 2)); + KUNIT_ASSERT_PTR_EQ(test, afk_service_get(service), service); + afk_service_put(service); +} + +static void dptx_service_identity_test(struct kunit *test) +{ + const char *name = "dcpdptx-port-epic"; + const char *class = "AppleDCPDPTXRemotePort"; + + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, 3, true, 3, 1), 0); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, 0, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, -1, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, 3, true, 3, 2), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(NULL, class, 3, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, NULL, 3, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port("wrong", class, 3, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, "wrong", 3, true, 3, 1), -EINVAL); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, 1, false, 3, 2), 1); + KUNIT_EXPECT_EQ(test, dptx_service_port(name, class, 2, false, 3, 2), -EINVAL); +} + +static void dptx_service_duplicate_close_test(struct kunit *test) +{ + struct apple_epic_service *first = test->priv; + struct apple_epic_service *next; + struct dptx_port port = {}; + + next = kunit_kzalloc(test, sizeof(*next), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, next); + spin_lock_init(&next->lock); + init_completion(&port.enable_completion); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(first, NULL, &prepared_ops, 1)); + KUNIT_ASSERT_EQ(test, dptx_service_bind(first, &port, 3), 0); + /* An observer during preparation/OPEN sees no available port or owner. */ + KUNIT_EXPECT_FALSE(test, dptx_service_available(&port)); + KUNIT_EXPECT_PTR_EQ(test, port.service, NULL); + KUNIT_EXPECT_PTR_EQ(test, first->cookie, &port); + /* Failed OPEN does not publish anything and permits a fresh attempt. */ + afk_service_discard(first); + KUNIT_EXPECT_FALSE(test, port.enabled); + KUNIT_EXPECT_PTR_EQ(test, port.service, NULL); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(first, NULL, &prepared_ops, 1)); + KUNIT_ASSERT_EQ(test, dptx_service_bind(first, &port, 3), 0); + afk_service_publish(first); + KUNIT_EXPECT_FALSE(test, port.enabled); + dptx_service_publish(first); + KUNIT_EXPECT_TRUE(test, dptx_service_available(&port)); + KUNIT_EXPECT_PTR_EQ(test, port.service->cookie, &port); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(next, NULL, &prepared_ops, 2)); + KUNIT_EXPECT_EQ(test, dptx_service_bind(next, &port, 3), -EBUSY); + afk_service_discard(next); + KUNIT_EXPECT_TRUE(test, port.enabled); + KUNIT_EXPECT_PTR_EQ(test, port.service, first); + /* CLOSE invalidates a connected session without publishing a replacement. */ + port.connected = true; + afk_test_retire(first); + dptx_service_unbind(first); + KUNIT_EXPECT_FALSE(test, port.enabled); + KUNIT_EXPECT_FALSE(test, port.connected); + KUNIT_EXPECT_PTR_EQ(test, first->cookie, &port); + KUNIT_ASSERT_TRUE(test, afk_service_reinit(next, NULL, &prepared_ops, 3)); + KUNIT_EXPECT_EQ(test, dptx_service_bind(next, &port, 3), -EBUSY); + afk_service_discard(next); + /* Old work between validate/connect/request still names the old session. */ + KUNIT_EXPECT_PTR_EQ(test, port.service, first); + KUNIT_EXPECT_PTR_EQ(test, afk_service_get(port.service), NULL); + KUNIT_EXPECT_FALSE(test, port.enabled); +} + static struct kunit_case afk_service_cases[] = { + KUNIT_CASE(afk_service_admission_test), + KUNIT_CASE(dptx_service_identity_test), + KUNIT_CASE(dptx_service_duplicate_close_test), KUNIT_CASE(afk_service_many_generations_test), KUNIT_CASE(afk_service_reader_pin_test), KUNIT_CASE(afk_service_pending_reply_test), KUNIT_CASE(afk_service_completed_reader_test), + KUNIT_CASE(afk_service_close_pin_test), KUNIT_CASE(afk_service_tag_sequence_test), KUNIT_CASE(afk_service_unconverted_owner_test), { } @@ -139,4 +256,5 @@ static struct kunit_suite afk_service_suite = { }; kunit_test_suite(afk_service_suite); +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING"); MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple/afk.c b/drivers/gpu/drm/apple/afk.c index 604b1972705e8a..5daa26990a02bb 100644 --- a/drivers/gpu/drm/apple/afk.c +++ b/drivers/gpu/drm/apple/afk.c @@ -6,15 +6,21 @@ #include #include #include +#include #include #include #include #include +#include +#include #include #include +#include + #include "afk.h" #include "afk_reply.h" +#include "afkv2-codec.h" #include "trace.h" struct afk_receive_message_work { @@ -52,18 +58,54 @@ enum rbep_msg_type { #define SEND_WPTR GENMASK(31, 0) +static void afk_service_worker(struct work_struct *work); +static void afk_discard_pending_open(struct apple_epic_service *service); + +void afk_service_retry(struct apple_dcp_afkep *ep) +{ + unsigned long flags; + + if (!ep || !ep->wire_v2 || ep != ep->dcp->dcpavservep) + return; + spin_lock_irqsave(&ep->lock, flags); + if (!ep->stopping) + queue_work(ep->wq, &ep->service_work); + spin_unlock_irqrestore(&ep->lock, flags); +} +EXPORT_SYMBOL_IF_KUNIT(afk_service_retry); + static void afk_send(struct apple_dcp_afkep *ep, u64 message) { dcp_send_message(ep->dcp, ep->endpoint, message); } +static void afk_release_context(void *data) +{ + struct apple_dcp_afkep *afkep = data; + + afk_quiesce(afkep); + if (!afkep->dcp->retain_dma) { + kvfree(afkep->command_tags); + kfree(afkep); + } +} + struct apple_dcp_afkep *afk_init(struct apple_dcp *dcp, u32 endpoint, const struct apple_epic_service_ops *ops) { + bool h17p = dcp->fw_compat == DCP_FIRMWARE_H17P; struct apple_dcp_afkep *afkep; int ret, i; - afkep = devm_kzalloc(dcp->dev, sizeof(*afkep), GFP_KERNEL); + /* + * H17P: an endpoint of a retained session outlives the binding, so it + * is freed by its own release action. Other firmware keeps the + * device-managed lifetime it always had. + */ + if (h17p) + afkep = kzalloc_obj(*afkep); + else + afkep = devm_kzalloc(dcp->dev, sizeof(*afkep), GFP_KERNEL); if (!afkep) return ERR_PTR(-ENOMEM); @@ -77,21 +119,49 @@ struct apple_dcp_afkep *afk_init(struct apple_dcp *dcp, u32 endpoint, goto out_free_afkep; } - // TODO: devm_ for wq - + INIT_WORK(&afkep->service_work, afk_service_worker); init_completion(&afkep->started); init_completion(&afkep->stopped); spin_lock_init(&afkep->lock); for (i = 0; i < AFK_MAX_CHANNEL; i++) spin_lock_init(&afkep->services[i].lock); + if (h17p) { + ret = devm_add_action_or_reset(dcp->dev, afk_release_context, afkep); + if (ret) + return ERR_PTR(ret); + } return afkep; out_free_afkep: - devm_kfree(dcp->dev, afkep); + if (h17p) + kfree(afkep); + else + devm_kfree(dcp->dev, afkep); return ERR_PTR(ret); } +void afk_quiesce(struct apple_dcp_afkep *afkep) +{ + unsigned long flags; + u32 i; + + if (IS_ERR_OR_NULL(afkep)) + return; + spin_lock_irqsave(&afkep->lock, flags); + afkep->stopping = true; + spin_unlock_irqrestore(&afkep->lock, flags); + if (afkep->wq) { + destroy_workqueue(afkep->wq); + afkep->wq = NULL; + } + for (i = 0; i < afkep->num_channels; i++) + if (afkep->services[i].pending_open) + afk_discard_pending_open(&afkep->services[i]); + debugfs_remove_recursive(afkep->debugfs_entry); + afkep->debugfs_entry = NULL; +} + void afk_shutdown(struct apple_dcp_afkep *afkep) { unsigned long flags; @@ -107,7 +177,10 @@ void afk_shutdown(struct apple_dcp_afkep *afkep) spin_lock_irqsave(&afkep->lock, flags); afkep->stopping = true; spin_unlock_irqrestore(&afkep->lock, flags); - destroy_workqueue(afkep->wq); + if (afkep->dcp->fw_compat == DCP_FIRMWARE_H17P) + afk_quiesce(afkep); + else + destroy_workqueue(afkep->wq); if (afkep->rx_scratch) { devm_kfree(afkep->dcp->dev, afkep->rx_scratch); afkep->rx_scratch = NULL; @@ -154,6 +227,14 @@ int afk_start(struct apple_dcp_afkep *ep) { int ret; + /* Allocate before firmware can announce services or issue commands. */ + if (ep->wire_v2 && !ep->command_tags) { + ep->command_tags = kvcalloc(AFKV2_INTERFACES, sizeof(*ep->command_tags), + GFP_KERNEL); + if (!ep->command_tags) + return -ENOMEM; + } + reinit_completion(&ep->started); ret = apple_rtkit_start_ep(ep->dcp->rtk, ep->endpoint); if (ret) @@ -163,6 +244,11 @@ int afk_start(struct apple_dcp_afkep *ep) afk_send(ep, FIELD_PREP(RBEP_TYPE, RBEP_INIT)); ret = wait_for_completion_timeout(&ep->started, msecs_to_jiffies(1000)); + if (ret <= 0 && ep->dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) + dev_warn(ep->dcp->dev, + "external AFK startup ep=%#x tx_ready=%u rx_ready=%u bytes=%zu\n", + ep->endpoint, ep->txbfr.ready, ep->rxbfr.ready, ep->bfr_size); if (ret <= 0) return -ETIMEDOUT; else @@ -171,7 +257,7 @@ int afk_start(struct apple_dcp_afkep *ep) static void afk_getbuf(struct apple_dcp_afkep *ep, u64 message) { - u16 size = FIELD_GET(GETBUF_SIZE, message) << BLOCK_SHIFT; + u32 size = FIELD_GET(GETBUF_SIZE, message) << BLOCK_SHIFT; u16 tag = FIELD_GET(GETBUF_TAG, message); u64 reply; @@ -183,8 +269,11 @@ static void afk_getbuf(struct apple_dcp_afkep *ep, u64 message) return; } - ep->bfr = dmam_alloc_coherent(ep->dcp->dev, size, &ep->bfr_dma, - GFP_KERNEL); + /* An unstopped adopted session owns this ring until the machine resets. */ + if (READ_ONCE(ep->dcp->retain_dma)) + ep->bfr = dma_alloc_coherent(ep->dcp->dev, size, &ep->bfr_dma, GFP_KERNEL); + else + ep->bfr = dmam_alloc_coherent(ep->dcp->dev, size, &ep->bfr_dma, GFP_KERNEL); if (!ep->bfr) { dev_err(ep->dcp->dev, "Failed to allocate %d bytes buffer\n", size); @@ -202,10 +291,11 @@ static void afk_getbuf(struct apple_dcp_afkep *ep, u64 message) static void afk_init_rxtx(struct apple_dcp_afkep *ep, u64 message, struct afk_ringbuffer *bfr) { - u16 base = FIELD_GET(INITRB_OFFSET, message) << BLOCK_SHIFT; - u16 size = FIELD_GET(INITRB_SIZE, message) << BLOCK_SHIFT; + u32 base = FIELD_GET(INITRB_OFFSET, message) << BLOCK_SHIFT; + u32 size = FIELD_GET(INITRB_SIZE, message) << BLOCK_SHIFT; u16 tag = FIELD_GET(INITRB_TAG, message); - u32 bufsz, end; + u32 bufsz, end, stride; + u8 *hdr; if (tag != ep->bfr_tag) { dev_err(ep->dcp->dev, "AFK[ep:%02x]: expected tag 0x%x but got 0x%x\n", @@ -219,10 +309,10 @@ static void afk_init_rxtx(struct apple_dcp_afkep *ep, u64 message, return; } - if (base >= ep->bfr_size) { + if (!ep->bfr || base >= ep->bfr_size || size < sizeof(u32)) { dev_err(ep->dcp->dev, - "AFK[ep:%02x]: requested base 0x%x >= max size 0x%lx\n", - ep->endpoint, base, ep->bfr_size); + "AFK[ep:%02x]: invalid ring base 0x%x or size 0x%x\n", + ep->endpoint, base, size); return; } @@ -234,12 +324,13 @@ static void afk_init_rxtx(struct apple_dcp_afkep *ep, u64 message, return; } - bfr->hdr = ep->bfr + base; - bufsz = le32_to_cpu(bfr->hdr->bufsz); - if (bufsz + sizeof(*bfr->hdr) != size) { + hdr = (u8 *)ep->bfr + base; + dma_rmb(); + bufsz = le32_to_cpu(READ_ONCE(*(__le32 *)hdr)); + if (afk_ring_stride(size, bufsz, &stride)) { dev_err(ep->dcp->dev, - "AFK[ep:%02x]: ring buffer size 0x%x != expected 0x%lx\n", - ep->endpoint, bufsz, sizeof(*bfr->hdr)); + "AFK[ep:%02x]: invalid ring payload size 0x%x of 0x%x\n", + ep->endpoint, bufsz, size); return; } if (bufsz < BIT(BLOCK_SHIFT)) { @@ -254,8 +345,11 @@ static void afk_init_rxtx(struct apple_dcp_afkep *ep, u64 message, return; } - bfr->buf = bfr->hdr + 1; + bfr->rptr = (__le32 *)(hdr + stride); + bfr->wptr = (__le32 *)(hdr + 2 * stride); + bfr->buf = hdr + 3 * stride; bfr->bufsz = bufsz; + bfr->stride = stride; bfr->ready = true; if (ep->rxbfr.ready && ep->txbfr.ready) @@ -309,12 +403,78 @@ static struct apple_epic_service *afk_epic_find_service(struct apple_dcp_afkep * return NULL; } +static int afk_send_epic_token(struct apple_dcp_afkep *ep, u32 channel, u16 tag, + enum epic_type etype, enum epic_category ecat, u8 stype, + const void *payload, size_t payload_len, u64 token); + +/* Pending identities are owned only by the ordered endpoint workqueue. */ +static void afk_discard_pending_open(struct apple_epic_service *service) +{ + service->pending_open = false; + kfree(service->pending_name); + kfree(service->pending_class); + service->pending_name = NULL; + service->pending_class = NULL; + afk_service_discard(service); +} + +static void afk_retry_service_init(struct apple_dcp_afkep *ep) +{ + u32 i; + + for (i = 0; i < ep->num_channels; i++) { + struct apple_epic_service *service = &ep->services[i]; + const struct apple_epic_service_ops *ops = service->ops; + int ret; + + if (!service->pending_open) + continue; + ret = ops->revalidate(service, service->pending_name, + service->pending_class, service->pending_unit); + if (ret) { + afk_discard_pending_open(service); + continue; + } + /* Old replies keep their original slot, tag and buffers until drained. */ + if (afk_epic_find_service(ep, service->channel)) + continue; + ret = afk_send_epic(ep, service->channel, 0, EPIC_TYPE_NOTIFY, + EPIC_CAT_REPORT, AFKV2_OPEN, NULL, 0); + if (ret) { + dev_err(ep->dcp->dev, "AFKV2 interface %u deferred OPEN failed: %d\n", + service->channel, ret); + afk_discard_pending_open(service); + continue; + } + service->pending_open = false; + afk_service_publish(service); + ops->init(service, service->pending_name, service->pending_class, + service->pending_unit); + if (service->enabled) + afk_populate_service_debugfs(service); + kfree(service->pending_name); + kfree(service->pending_class); + service->pending_name = NULL; + service->pending_class = NULL; + } +} + +static void afk_service_worker(struct work_struct *work) +{ + struct apple_dcp_afkep *ep = container_of(work, struct apple_dcp_afkep, service_work); + + if (!READ_ONCE(ep->stopping)) + afk_retry_service_init(ep); +} + static void afk_recv_handle_init(struct apple_dcp_afkep *ep, u32 channel, u8 *payload, size_t payload_size) { char name[32]; s64 epic_unit = -1; - struct apple_epic_service *service = NULL; + struct apple_epic_service *service = NULL, *previous; + unsigned long flags; + bool retired = false; u32 ch_idx; const char *service_name = name; const char *epic_name = NULL, *epic_class = NULL; @@ -323,8 +483,25 @@ static void afk_recv_handle_init(struct apple_dcp_afkep *ep, u32 channel, u8 *props = payload + sizeof(name); size_t props_size = payload_size - sizeof(name); - if (afk_epic_find_service(ep, channel)) - return; + previous = afk_epic_find_service(ep, channel); + if (previous) { + spin_lock_irqsave(&previous->lock, flags); + retired = previous->torndown; + spin_unlock_irqrestore(&previous->lock, flags); + if (!ep->wire_v2 || ep != ep->dcp->dcpavservep || + !retired || !previous->ops->revalidate) + return; + } + for (ch_idx = 0; ch_idx < ep->num_channels; ch_idx++) { + struct apple_epic_service *pending = &ep->services[ch_idx]; + + if (!pending->pending_open || pending->channel != channel) + continue; + if (!pending->ops->revalidate(pending, pending->pending_name, + pending->pending_class, pending->pending_unit)) + return; + afk_discard_pending_open(pending); + } if (payload_size < sizeof(name)) { dev_err(ep->dcp->dev, "AFK[ep:%02x]: payload too small: %lx\n", @@ -348,7 +525,8 @@ static void afk_recv_handle_init(struct apple_dcp_afkep *ep, u32 channel, ep->endpoint, name); return; } - ret = parse_epic_service_init(&ctx, &epic_name, &epic_class, &epic_unit); + ret = parse_epic_service_init(&ctx, &epic_name, &epic_class, &epic_unit, + ep->dcp->fw_compat == DCP_FIRMWARE_H17P); if (ret) { dev_err(ep->dcp->dev, "AFK[ep:%02x]: failed to extract init props: %d\n", @@ -393,7 +571,54 @@ static void afk_recv_handle_init(struct apple_dcp_afkep *ep, u32 channel, goto free; } ep->num_channels = max(ep->num_channels, ch_idx + 1); - ops->init(service, epic_name, epic_class, epic_unit); + if (ops->prepare) { + int ret = ops->prepare(service, epic_name, epic_class, epic_unit); + + if (ret) { + if (ops->optional && ret == -EOPNOTSUPP) + dev_info(ep->dcp->dev, + "AFK optional %s (%s) is unsupported on channel %u\n", + ops->name, service_name, channel); + else + dev_err(ep->dcp->dev, "AFK service %s admission failed: %d\n", + service_name, ret); + afk_service_discard(service); + goto free; + } + } + if (previous && retired && ops->revalidate) { + /* Preserve the admitted identity and its original provider ticket. */ + service->pending_open = true; + service->pending_name = epic_name; + service->pending_class = epic_class; + service->pending_unit = epic_unit; + afk_service_retry(ep); + return; + } + if (ep->wire_v2) { + int ret = afk_send_epic(ep, channel, 0, EPIC_TYPE_NOTIFY, + EPIC_CAT_REPORT, AFKV2_OPEN, NULL, 0); + + if (ret) { + unsigned long flags; + + if (ops->prepare) { + afk_service_discard(service); + } else { + spin_lock_irqsave(&service->lock, flags); + service->enabled = false; + service->torndown = true; + spin_unlock_irqrestore(&service->lock, flags); + } + dev_err(ep->dcp->dev, "AFKV2 interface %u OPEN failed: %d\n", + channel, ret); + goto free; + } + } + if (ops->prepare) + afk_service_publish(service); + if (ops->init) + ops->init(service, epic_name, epic_class, epic_unit); dev_info(ep->dcp->dev, "AFK[ep:%02x]: new service %s on channel %d\n", ep->endpoint, service_name, channel); @@ -410,6 +635,14 @@ static void afk_recv_handle_teardown(struct apple_dcp_afkep *ep, u32 channel) const struct apple_epic_service_ops *ops; unsigned long flags; + /* Accepted CLOSE/CLOSED supersedes unpublished recovery for this interface. */ + for (u32 i = 0; i < ep->num_channels; i++) { + struct apple_epic_service *pending = &ep->services[i]; + + if (pending->pending_open && pending->channel == channel) + afk_discard_pending_open(pending); + } + service = afk_epic_find_service(ep, channel); if (!service) { dev_warn(ep->dcp->dev, "AFK[ep:%02x]: teardown for disabled channel %u\n", @@ -442,7 +675,7 @@ static void afk_recv_handle_reply(struct apple_dcp_afkep *ep, u32 channel, struct epic_cmd *cmd = payload; struct apple_epic_service *service; unsigned long flags; - u8 idx = tag & 0xff; + u8 idx = tag & (ep->wire_v2 ? 0xf : 0xff); void *rxbuf, *txbuf; dma_addr_t rxbuf_dma, txbuf_dma; size_t rxlen, txlen; @@ -513,14 +746,12 @@ static void afk_recv_handle_reply(struct apple_dcp_afkep *ep, u32 channel, dma_free_coherent(ep->dcp->dev, txlen, txbuf, txbuf_dma); } +/* A call from DCP: one word ahead of the call header AP sends. */ struct epic_std_service_ap_call { __le32 unk0; - __le32 unk1; - __le32 type; - __le32 len; - __le32 magic; - u8 _unk[48]; + struct epic_service_call call; } __attribute__((packed)); +static_assert(sizeof(struct epic_std_service_ap_call) == 68); static void afk_recv_handle_std_service(struct apple_dcp_afkep *ep, u32 channel, u32 type, struct epic_hdr *ehdr, @@ -563,30 +794,60 @@ static void afk_recv_handle_std_service(struct apple_dcp_afkep *ep, u32 channel, if (type == EPIC_TYPE_NOTIFY && eshdr->category == EPIC_CAT_NOTIFY) { struct epic_std_service_ap_call *call = payload; size_t call_size; + u32 command; void *reply; + u16 group; int ret; if (payload_size < sizeof(*call)) return; - call_size = le32_to_cpu(call->len); + call_size = le32_to_cpu(call->call.data_len); if (payload_size < sizeof(*call) + call_size) return; - if (!service->ops->call) + group = le16_to_cpu(call->call.group); + command = le32_to_cpu(call->call.command); + + /* + * H17P firmware: command numbers are per group, so a + * group-less .call handler would take another group's call + * for a group 0 one. Without call_v2 there is no handler for + * it, and AFK cannot guess a reply, so it gets none, as a call + * without any handler. Other firmware hands every group's + * command to .call, as it always has. + */ + if (!service->ops->call_v2 && + (!service->ops->call || + (group && ep->dcp->fw_compat == DCP_FIRMWARE_H17P))) { + if (group && ep->dcp->fw_compat == DCP_FIRMWARE_H17P) + dev_warn_ratelimited(ep->dcp->dev, + "AFK[ep:%02x]: %s (chan:%u) apcall %u:%u has no handler, sending NO reply\n", + ep->endpoint, service->ops->name, + channel, group, command); return; + } reply = kzalloc(payload_size, GFP_KERNEL); if (!reply) return; - ret = service->ops->call(service, le32_to_cpu(call->type), - payload + sizeof(*call), call_size, - reply + sizeof(*call), call_size); + if (service->ops->call_v2) + ret = service->ops->call_v2(service, group, command, + payload + sizeof(*call), + call_size, + reply + sizeof(*call), + call_size); + else + ret = service->ops->call(service, command, + payload + sizeof(*call), + call_size, + reply + sizeof(*call), + call_size); if (ret) { dev_warn(ep->dcp->dev, - "AFK[ep:%02x]: %s (chan:%u) apcall %u handler returned %d, sending NO reply\n", + "AFK[ep:%02x]: %s (chan:%u) apcall %u:%u handler returned %d, sending NO reply\n", ep->endpoint, service->ops->name, channel, - le32_to_cpu(call->type), ret); + group, command, ret); kfree(reply); return; } @@ -610,8 +871,270 @@ static void afk_recv_handle_std_service(struct apple_dcp_afkep *ep, u32 channel, dev_err(ep->dcp->dev, "AFK[ep:%02x]: channel %d received unhandled standard service message: %x / %x\n", ep->endpoint, channel, type, eshdr->category); - print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, payload, - payload_size, true); + if (ep->dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) + print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, + payload, payload_size, true); +} + +static bool afk_validate_h17p_header(const u8 *data, size_t size) +{ + /* The measured length counts bytes after the first eight header bytes. */ + return size >= sizeof(struct epic_hdr) + + sizeof(struct epic_sub_hdr_h17p) && + get_unaligned_le32(data + 4) == size - 8; +} + +static void afk_v2_status(struct apple_dcp_afkep *ep, + const struct afkv2_frame *frame, + const struct afkv2_command *command, u32 status) +{ + u8 record[AFKV2_HEADER_SIZE + AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE]; + size_t size; + int ret; + + ret = afkv2_encode_error_reply(record, sizeof(record), 0, frame, command, + status, &size); + if (!ret) + ret = afk_send_epic_token(ep, frame->interface, command->id, + EPIC_TYPE_REPLY, EPIC_CAT_REPLY, frame->type, + record + AFKV2_HEADER_SIZE, + size - AFKV2_HEADER_SIZE, frame->token); + if (ret) + dev_warn(ep->dcp->dev, "AFKV2 error reply failed: %d\n", ret); +} + +static int afk_v2_access_owned(struct apple_epic_service *service, u64 dva, + void *buffer, size_t size, bool to_peer, bool copy) +{ + unsigned long flags; + unsigned int i; + int ret = -EACCES; + + if (!service || !service->ep->wire_v2 || size > SZ_1M || (copy && size && !buffer)) + return -EINVAL; + if (!size) + return 0; + spin_lock_irqsave(&service->lock, flags); + for (i = 0; i < MAX_PENDING_CMDS; i++) { + struct epic_cmd_info *cmd = &service->cmds[i]; + void *owned = NULL; + u64 offset; + + if (!test_bit(i, service->cmd_map) || cmd->done) + continue; + if (cmd->rxbuf && afkv2_owned_span(cmd->rxbuf_dma, cmd->rxlen, dva, size)) { + offset = dva - cmd->rxbuf_dma; + owned = cmd->rxbuf + offset; + } else if (cmd->txbuf && afkv2_owned_span(cmd->txbuf_dma, cmd->txlen, dva, size)) { + offset = dva - cmd->txbuf_dma; + owned = cmd->txbuf + offset; + } + if (!owned) + continue; + if (!copy) { + ret = 0; + break; + } + if (to_peer) { + memcpy(owned, buffer, size); + dma_wmb(); + } else { + dma_rmb(); + memcpy(buffer, owned, size); + } + ret = 0; + break; + } + spin_unlock_irqrestore(&service->lock, flags); + if (ret) + dev_warn_ratelimited(service->ep->dcp->dev, + "AFKV2 unowned buffer: ep=%u interface=%u size=%zu write=%u\n", + service->ep->endpoint, service->channel, size, to_peer); + return ret; +} + +int afk_v2_copy_owned(struct apple_epic_service *service, u64 dva, + void *buffer, size_t size, bool to_peer) +{ + return afk_v2_access_owned(service, dva, buffer, size, to_peer, true); +} + +static void afk_v2_callback(struct apple_dcp_afkep *ep, + const struct afkv2_frame *frame, + const struct afkv2_command *command) +{ + struct apple_epic_service *service; + const struct epic_service_call *call; + const u8 *request = command->inline_data; + u8 *owned_request = NULL, *reply = NULL; + size_t request_size, response_size, response_body; + size_t body_size, reply_size; + u32 status = 0xe00002c7; + int ret; + + dev_dbg(ep->dcp->dev, "AFKV2 callback sizes: tx=%u rx=%u value=%u inline=%zu\n", + command->tx_size, command->rx_size, command->value, command->inline_size); + if (frame->type != EPIC_SUBTYPE_STD_SERVICE || + afkv2_callback_lengths(command, sizeof(*call) + sizeof(u32), + SZ_1M - AFKV2_COMMAND_SIZE, + &request_size, &response_size)) + goto reject; + service = afk_service_get(afk_epic_find_service(ep, frame->interface)); + if (!service || READ_ONCE(service->torndown) || + (!service->ops->call && !service->ops->call_v2)) { + if (service) + afk_service_put(service); + goto reject; + } + if (command->oob) { + /* Reject an unowned response destination before any callback side effect. */ + ret = afk_v2_access_owned(service, command->rx_address, NULL, + response_size, true, false); + if (ret) + goto release; + owned_request = kmalloc(request_size, GFP_KERNEL); + if (!owned_request) { + status = 0xe00002bd; + goto release; + } + ret = afk_v2_copy_owned(service, command->tx_address, + owned_request, request_size, false); + if (ret) + goto release; + request = owned_request; + } + call = (const void *)request; + body_size = request_size - sizeof(*call); + if ((le16_to_cpu(call->group) && !service->ops->call_v2) || + le32_to_cpu(call->magic) != EPIC_SERVICE_CALL_MAGIC || + le32_to_cpu(call->data_len) != body_size) + goto release; + response_body = response_size - sizeof(*call); + reply = kzalloc(AFKV2_COMMAND_SIZE + response_size, GFP_KERNEL); + if (!reply) { + status = 0xe00002bd; + goto release; + } + reply[1] = command->id; + memcpy(reply + AFKV2_COMMAND_SIZE, call, sizeof(*call)); + ((struct epic_service_call *)(reply + AFKV2_COMMAND_SIZE))->data_len = + cpu_to_le32(response_body); + dev_dbg(ep->dcp->dev, "AFKV2 callback begin: group=%u command=%u in=%zu out=%zu\n", + le16_to_cpu(call->group), le32_to_cpu(call->command), body_size, response_body); + if (service->ops->call_v2) + ret = service->ops->call_v2(service, le16_to_cpu(call->group), + le32_to_cpu(call->command), + request + sizeof(*call), body_size, + reply + AFKV2_COMMAND_SIZE + sizeof(*call), + response_body); + else + ret = service->ops->call(service, le32_to_cpu(call->command), + request + sizeof(*call), body_size, + reply + AFKV2_COMMAND_SIZE + sizeof(*call), response_body); + dev_dbg(ep->dcp->dev, "AFKV2 callback end: group=%u command=%u result=%d\n", + le16_to_cpu(call->group), le32_to_cpu(call->command), ret); + if (ret) + goto release; + reply_size = AFKV2_COMMAND_SIZE + response_size; + if (command->oob) { + ret = afk_v2_copy_owned(service, command->rx_address, + reply + AFKV2_COMMAND_SIZE, response_size, true); + if (ret) + goto release; + ret = afkv2_encode_oob_command(reply, reply_size, command->id, + command->rx_address, response_size, + command->tx_address, command->tx_size); + if (ret) + goto release; + afkv2_put32(reply + 4, 0); + reply_size = AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE; + } + ret = afk_send_epic_token(ep, frame->interface, command->id, EPIC_TYPE_REPLY, + EPIC_CAT_REPLY, frame->type, reply, reply_size, frame->token); + kfree(reply); + kfree(owned_request); + afk_service_put(service); + if (ret) + dev_warn(ep->dcp->dev, "AFKV2 callback reply failed: %d\n", ret); + return; +release: + kfree(reply); + kfree(owned_request); + afk_service_put(service); +reject: + afk_v2_status(ep, frame, command, status); +} + +static void afk_v2_receive(struct apple_dcp_afkep *ep, u32 channel, u32 type, + const void *data, size_t size) +{ + struct afkv2_frame frame; + struct afkv2_command response; + struct apple_epic_service *service; + struct epic_cmd legacy = {}; + unsigned long flags; + unsigned int slot; + int ret; + + if (channel || type || afkv2_decode_frame(data, size, false, &frame) || + !frame.interface) + return; + if (frame.category == AFKV2_REPORT) { + if (frame.type == EPIC_SUBTYPE_ANNOUNCE_H17P) + afk_recv_handle_init(ep, frame.interface, (void *)frame.payload, + frame.payload_size); + else if ((frame.type == AFKV2_CLOSE && frame.interface_flags == 1) || + (ep == ep->dcp->dcpavservep && afkv2_closed_report(&frame))) + afk_recv_handle_teardown(ep, frame.interface); + return; + } + if (afkv2_decode_command(frame.payload, frame.payload_size, &response)) + return; + if (frame.category == AFKV2_COMMAND) { + afk_v2_callback(ep, &frame, &response); + return; + } + service = afk_epic_find_service(ep, frame.interface); + if (!service) + return; + spin_lock_irqsave(&service->lock, flags); + if (bitmap_empty(service->cmd_map, MAX_PENDING_CMDS)) { + spin_unlock_irqrestore(&service->lock, flags); + return; + } + service->users++; + spin_unlock_irqrestore(&service->lock, flags); + slot = response.id & 0xf; + spin_lock_irqsave(&service->lock, flags); + ret = -EPROTO; + if (slot < MAX_PENDING_CMDS && test_bit(slot, service->cmd_map) && + !service->cmds[slot].done && response.id == service->cmds[slot].tag) { + struct epic_cmd_info *cmd = &service->cmds[slot]; + + if (ep->inline_v2) + ret = afkv2_check_inline_response(&frame, &response, service->channel, + cmd->type, cmd->tag, cmd->rxlen); + else + ret = afkv2_check_response(&frame, &response, service->channel, + cmd->type, cmd->tag, cmd->rxbuf_dma, + cmd->rxlen, cmd->txbuf_dma, cmd->txlen); + if (!ret) { + cmd->reply_length = response.value ? 0 : + (ep->inline_v2 ? response.inline_size : response.rx_size); + if (ep->inline_v2 && !response.value && cmd->reply_length) + memcpy(cmd->rxbuf, response.inline_data, cmd->reply_length); + } + } + spin_unlock_irqrestore(&service->lock, flags); + if (!ret) { + legacy.retcode = cpu_to_le32(response.value); + afk_recv_handle_reply(ep, frame.interface, response.id, + &legacy, sizeof(legacy)); + } else { + dev_warn(ep->dcp->dev, "AFKV2 reply rejected: %d\n", ret); + } + afk_service_put(service); } static void afk_recv_handle(struct apple_dcp_afkep *ep, u32 channel, u32 type, @@ -621,20 +1144,67 @@ static void afk_recv_handle(struct apple_dcp_afkep *ep, u32 channel, u32 type, struct epic_hdr *ehdr = (struct epic_hdr *)data; struct epic_sub_hdr *eshdr = (struct epic_sub_hdr *)(data + sizeof(*ehdr)); + struct epic_sub_hdr h17p_eshdr; + size_t hdr_len = sizeof(*ehdr) + sizeof(*eshdr); u16 subtype; - u8 *payload = data + sizeof(*ehdr) + sizeof(*eshdr); + u8 *payload; size_t payload_size; - if (data_size < sizeof(*ehdr) + sizeof(*eshdr)) { - dev_err(ep->dcp->dev, "AFK[ep:%02x]: payload too small: %lx\n", - ep->endpoint, data_size); + if (ep->wire_v2) { + afk_v2_receive(ep, channel, type, data, data_size); return; } - payload_size = data_size - sizeof(*ehdr) - sizeof(*eshdr); + + if (ep->dcp->fw_compat == DCP_FIRMWARE_H17P) { + const struct epic_sub_hdr_h17p *c = + (const struct epic_sub_hdr_h17p *)(data + sizeof(*ehdr)); + + hdr_len = sizeof(*ehdr) + sizeof(*c); + if (data_size < hdr_len) + goto too_small; + if (ep->dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !afk_validate_h17p_header(data, data_size)) { + dev_err(ep->dcp->dev, + "AFK[ep:%02x]: invalid compact message length\n", + ep->endpoint); + return; + } + + /* + * H17P multiplexes every service of an endpoint onto queue + * channel 0 and carries the real channel in the EPIC header + * instead: byte 0 is a sequence counter, byte 1 a flags byte + * and bytes 2-3 the channel. ep:20 announces "system" as + * channel 1 and "powerlog-service" as channel 3, then sends the + * mNits/uAmps/iDAC backlight reports on channel 3. Taking the + * queue channel at face value binds "system" to 0 and makes + * every later announce and report look like a duplicate. + */ + channel = le16_to_cpup((__le16 *)(data + 2)); + + /* + * Normalise into the wide sub-header the rest of this file + * expects, mapping H17P's announce subtype onto the canonical + * one so no downstream check has to know the difference. + */ + memset(&h17p_eshdr, 0, sizeof(h17p_eshdr)); + h17p_eshdr.category = c->category; + h17p_eshdr.type = cpu_to_le16(c->type == EPIC_SUBTYPE_ANNOUNCE_H17P ? + EPIC_SUBTYPE_ANNOUNCE : c->type); + h17p_eshdr.tag = cpu_to_le16(le32_to_cpu(c->tag)); + eshdr = &h17p_eshdr; + } + + if (data_size < hdr_len) + goto too_small; + subtype = le16_to_cpu(eshdr->type); + payload = data + hdr_len; + payload_size = data_size - hdr_len; trace_afk_recv_handle(ep, channel, type, data_size, ehdr, eshdr); + service = afk_epic_find_service(ep, channel); if (!service) { @@ -696,8 +1266,15 @@ static void afk_recv_handle(struct apple_dcp_afkep *ep, u32 channel, u32 type, dev_err(ep->dcp->dev, "AFK[ep:%02x]: channel %d received unhandled message " "(type %x subtype %x)\n", ep->endpoint, channel, type, subtype); - print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, payload, - payload_size, true); + if (ep->dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) + print_hex_dump(KERN_INFO, "AFK: ", DUMP_PREFIX_NONE, 16, 1, + payload, payload_size, true); + return; + +too_small: + dev_err(ep->dcp->dev, "AFK[ep:%02x]: payload too small: %lx\n", + ep->endpoint, data_size); } static bool afk_recv(struct apple_dcp_afkep *ep) @@ -712,8 +1289,8 @@ static bool afk_recv(struct apple_dcp_afkep *ep) return false; } - rptr = le32_to_cpu(ep->rxbfr.hdr->rptr); - wptr = le32_to_cpu(ep->rxbfr.hdr->wptr); + rptr = le32_to_cpu(*ep->rxbfr.rptr); + wptr = le32_to_cpu(*ep->rxbfr.wptr); trace_afk_recv_rwptr_pre(ep, rptr, wptr); if (rptr == wptr) @@ -775,15 +1352,15 @@ static bool afk_recv(struct apple_dcp_afkep *ep) */ memcpy(ep->rx_scratch, hdr->data, size); - rptr = ALIGN(rptr + sizeof(*hdr) + size, 1 << BLOCK_SHIFT); - if (WARN_ON(rptr > ep->rxbfr.bufsz)) - rptr = 0; - if (rptr == ep->rxbfr.bufsz) + rptr = afk_ring_advance(rptr, sizeof(*hdr) + size, + ep->rxbfr.bufsz, ep->rxbfr.stride); + if (ep->dcp->fw_compat != DCP_FIRMWARE_H17P && + WARN_ON(rptr > ep->rxbfr.bufsz)) rptr = 0; dma_mb(); - ep->rxbfr.hdr->rptr = cpu_to_le32(rptr); + *ep->rxbfr.rptr = cpu_to_le32(rptr); trace_afk_recv_rwptr_post(ep, rptr, wptr); afk_recv_handle(ep, channel, type, ep->rx_scratch, size); @@ -799,6 +1376,7 @@ static void afk_receive_message_worker(struct work_struct *work_) work = container_of(work_, struct afk_receive_message_work, work); type = FIELD_GET(RBEP_TYPE, work->message); + switch (type) { case RBEP_INIT_ACK: break; @@ -868,9 +1446,9 @@ int afk_receive_message(struct apple_dcp_afkep *ep, u64 message) return 0; } -int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, - enum epic_type etype, enum epic_category ecat, u8 stype, - const void *payload, size_t payload_len) +static int afk_send_epic_token(struct apple_dcp_afkep *ep, u32 channel, u16 tag, + enum epic_type etype, enum epic_category ecat, u8 stype, + const void *payload, size_t payload_len, u64 token) { u32 rptr, wptr; struct afk_qe *hdr, *hdr2; @@ -878,21 +1456,102 @@ int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, struct epic_sub_hdr *eshdr; unsigned long flags; size_t total_epic_size, total_size; + u8 v2_command[AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE]; + u8 v2_category = 0; + u16 v2_flags = 0; int ret; + /* External H17P services use AFKV2; the internal profile has no EPIC sender. */ + if (ep->dcp->fw_compat == DCP_FIRMWARE_H17P && !ep->wire_v2) { + dev_dbg_once(ep->dcp->dev, + "AFK[ep:%02x]: not sending EPIC message on H17P\n", + ep->endpoint); + return -EOPNOTSUPP; + } + + if (ep->wire_v2) { + if (!channel || channel > U16_MAX || tag > U8_MAX || payload_len > SZ_1M) + return -EINVAL; + if (ecat == EPIC_CAT_REPORT) { + if ((stype != AFKV2_OPEN && stype != AFKV2_CLOSE) || payload_len) + return -EOPNOTSUPP; + v2_flags = 1; + } else if (ecat == EPIC_CAT_COMMAND) { + if (ep->inline_v2) { + struct afkv2_command command; + + if (stype != EPIC_SUBTYPE_STD_SERVICE || + afkv2_decode_command(payload, payload_len, &command) || + command.oob || command.id != tag) + return -EINVAL; + } else { + const struct epic_cmd *cmd = payload; + + if (stype != EPIC_SUBTYPE_STD_SERVICE || !cmd || + payload_len != sizeof(*cmd)) + return -EINVAL; + ret = afkv2_encode_oob_command(v2_command, sizeof(v2_command), tag, + le64_to_cpu(cmd->rxbuf), + le32_to_cpu(cmd->rxlen), + le64_to_cpu(cmd->txbuf), + le32_to_cpu(cmd->txlen)); + if (ret) + return ret; + payload = v2_command; + payload_len = sizeof(v2_command); + } + v2_category = AFKV2_COMMAND; + } else if (ecat == EPIC_CAT_REPLY) { + v2_category = AFKV2_RESPONSE; + } else { + return -EOPNOTSUPP; + } + } + spin_lock_irqsave(&ep->lock, flags); if (ep->stopping) { ret = -ESHUTDOWN; goto out; } + if (ep->wire_v2 && + (!ep->txbfr.ready || !ep->txbfr.rptr || !ep->txbfr.wptr || !ep->txbfr.buf)) { + ret = -ESHUTDOWN; + goto out; + } + dma_rmb(); - rptr = le32_to_cpu(ep->txbfr.hdr->rptr); - wptr = le32_to_cpu(ep->txbfr.hdr->wptr); + rptr = le32_to_cpu(*ep->txbfr.rptr); + wptr = le32_to_cpu(*ep->txbfr.wptr); trace_afk_send_rwptr_pre(ep, rptr, wptr); - total_epic_size = sizeof(*ehdr) + sizeof(*eshdr) + payload_len; + total_epic_size = sizeof(*ehdr) + + (ep->wire_v2 ? sizeof(struct epic_sub_hdr_h17p) : sizeof(*eshdr)) + + payload_len; total_size = sizeof(*hdr) + total_epic_size; + if (ep->wire_v2) { + u32 available, needed; + + if (!ep->txbfr.ready || !ep->txbfr.stride || + (ep->txbfr.stride & (ep->txbfr.stride - 1)) || + rptr >= ep->txbfr.bufsz || wptr >= ep->txbfr.bufsz || + !IS_ALIGNED(rptr, ep->txbfr.stride) || + !IS_ALIGNED(wptr, ep->txbfr.stride) || + total_size >= ep->txbfr.bufsz) { + ret = -EPROTO; + goto out; + } + available = rptr > wptr ? rptr - wptr : ep->txbfr.bufsz - wptr + rptr; + needed = ALIGN(total_size, ep->txbfr.stride); + if (total_size > ep->txbfr.bufsz - wptr) + needed += ep->txbfr.bufsz - wptr; + /* Retain an empty slot; equal read/write pointers mean empty. */ + if (needed >= available) { + ret = -ENOSPC; + goto out; + } + } + hdr = hdr2 = NULL; /* @@ -973,42 +1632,54 @@ int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, hdr->magic = cpu_to_le32(QE_MAGIC); hdr->size = cpu_to_le32(total_epic_size); - hdr->channel = cpu_to_le32(channel); - hdr->type = cpu_to_le32(etype); + hdr->channel = cpu_to_le32(ep->wire_v2 ? 0 : channel); + hdr->type = cpu_to_le32(ep->wire_v2 ? 0 : etype); if (hdr2) memcpy(hdr2, hdr, sizeof(*hdr)); - ehdr = ep->txbfr.buf + wptr; - memset(ehdr, 0, sizeof(*ehdr)); - ehdr->version = 2; - ehdr->seq = cpu_to_le16(ep->qe_seq++); - ehdr->timestamp = cpu_to_le64(0); - wptr += sizeof(*ehdr); - - eshdr = ep->txbfr.buf + wptr; - memset(eshdr, 0, sizeof(*eshdr)); - eshdr->length = cpu_to_le32(payload_len); - eshdr->version = 4; - eshdr->category = ecat; - eshdr->type = cpu_to_le16(stype); - eshdr->timestamp = cpu_to_le64(0); - eshdr->tag = cpu_to_le16(tag); - if (ecat == EPIC_CAT_REPLY) - eshdr->inline_len = cpu_to_le32(payload_len - 4); - else - eshdr->inline_len = cpu_to_le32(0); - wptr += sizeof(*eshdr); - - memcpy(ep->txbfr.buf + wptr, payload, payload_len); - wptr += payload_len; - wptr = ALIGN(wptr, 1 << BLOCK_SHIFT); - if (wptr == ep->txbfr.bufsz) - wptr = 0; + if (ep->wire_v2) { + size_t written; + + ret = afkv2_encode_frame(ep->txbfr.buf + wptr, total_epic_size, + ep->qe_seq++, channel, token, stype, v2_category, + v2_flags, payload, payload_len, &written); + if (ret) + goto out; + wptr += written; + } else { + ehdr = ep->txbfr.buf + wptr; + memset(ehdr, 0, sizeof(*ehdr)); + ehdr->version = 2; + ehdr->seq = cpu_to_le16(ep->qe_seq++); + ehdr->timestamp = cpu_to_le64(0); + wptr += sizeof(*ehdr); + + eshdr = ep->txbfr.buf + wptr; + memset(eshdr, 0, sizeof(*eshdr)); + eshdr->length = cpu_to_le32(payload_len); + eshdr->version = 4; + eshdr->category = ecat; + eshdr->type = cpu_to_le16(stype); + eshdr->timestamp = cpu_to_le64(0); + eshdr->tag = cpu_to_le16(tag); + if (ecat == EPIC_CAT_REPLY) + eshdr->inline_len = cpu_to_le32(payload_len - 4); + else + eshdr->inline_len = cpu_to_le32(0); + wptr += sizeof(*eshdr); + } + + if (!ep->wire_v2) { + memcpy(ep->txbfr.buf + wptr, payload, payload_len); + wptr += payload_len; + } + wptr = afk_ring_advance(wptr, 0, ep->txbfr.bufsz, + ep->txbfr.stride); trace_afk_send_rwptr_post(ep, rptr, wptr); /* Publish the completed entry before allowing the consumer to read it. */ dma_wmb(); - ep->txbfr.hdr->wptr = cpu_to_le32(wptr); + *ep->txbfr.wptr = cpu_to_le32(wptr); afk_send(ep, FIELD_PREP(RBEP_TYPE, RBEP_SEND) | FIELD_PREP(SEND_WPTR, wptr)); ret = 0; @@ -1018,11 +1689,41 @@ int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, return ret; } +int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, + enum epic_type etype, enum epic_category ecat, u8 stype, + const void *payload, size_t payload_len) +{ + return afk_send_epic_token(ep, channel, tag, etype, ecat, stype, + payload, payload_len, 0); +} + +static u16 afk_command_tag(struct apple_epic_service *service, u8 slot) +{ + struct apple_dcp_afkep *ep = service->ep; + unsigned long flags; + u16 tag; + + lockdep_assert_held(&service->lock); + if (!ep->wire_v2) + return ((service->cmd_tag++ & 0xff) << 8) | slot; + + /* Send already nests the endpoint lock below the service lock. */ + spin_lock_irqsave(&ep->lock, flags); + tag = afkv2_command_id(ep->command_tags[service->channel]++, slot); + spin_unlock_irqrestore(&ep->lock, flags); + return tag; +} + /* * Send one EPIC command and wait up to @timeout_ms for its reply. With * @reply_len set, only the bytes the firmware reported are copied to @output * (the rest is zeroed) and *@reply_len receives that count; without it the * whole receive buffer is copied, as the original callers expect. + * + * AFKV2 endpoints (@ep->wire_v2, external H17P) carry the reply length in + * the reply itself: with @reply_len set, only that many bytes are copied, + * a short successful reply fails with -EMSGSIZE, and nothing is copied for + * a nonzero status. */ static int afk_send_command_full(struct apple_epic_service *service, u8 type, const void *payload, size_t payload_len, @@ -1030,6 +1731,8 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, size_t *reply_len, unsigned int timeout_ms) { struct epic_cmd cmd; + void *inline_command = NULL; + size_t copy_len = output_len; void *rxbuf, *txbuf; dma_addr_t rxbuf_dma, txbuf_dma; unsigned long flags; @@ -1047,6 +1750,21 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, if (!service) return -ENODEV; ep = service->ep; + if (ep->wire_v2 && (!service->channel || service->channel >= AFKV2_INTERFACES || + !ep->command_tags)) { + ret = -ENODEV; + goto err_put_service; + } + if (READ_ONCE(ep->dcp->quiescing)) { + ret = -ENODEV; + goto err_put_service; + } + + if (ep->wire_v2 && (payload_len > SZ_1M || output_len > SZ_1M || + (payload_len && !payload))) { + ret = -EINVAL; + goto err_put_service; + } rxbuf = dma_alloc_coherent(ep->dcp->dev, output_len, &rxbuf_dma, GFP_KERNEL); @@ -1062,6 +1780,17 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, } memcpy(txbuf, payload, payload_len); + if (ep->inline_v2) { + if (payload_len > AFKV2_MAX_PAYLOAD - AFKV2_COMMAND_SIZE) { + ret = -E2BIG; + goto err_free_txbuf; + } + inline_command = kmalloc(AFKV2_COMMAND_SIZE + payload_len, GFP_KERNEL); + if (!inline_command) { + ret = -ENOMEM; + goto err_free_txbuf; + } + } memset(&cmd, 0, sizeof(cmd)); cmd.retcode = cpu_to_le32(0); @@ -1081,11 +1810,11 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, goto err_unlock; } - tag = (service->cmd_tag & 0xff) << 8; - tag |= idx & 0xff; - service->cmd_tag++; + tag = afk_command_tag(service, idx); service->cmds[idx].tag = tag; + service->cmds[idx].type = type; + service->cmds[idx].reply_length = 0; service->cmds[idx].rxbuf = rxbuf; service->cmds[idx].txbuf = txbuf; service->cmds[idx].rxbuf_dma = rxbuf_dma; @@ -1099,9 +1828,21 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, service->cmds[idx].completion = &completion; init_completion(&completion); - ret = afk_send_epic(service->ep, service->channel, tag, - EPIC_TYPE_COMMAND, EPIC_CAT_COMMAND, type, &cmd, - sizeof(cmd)); + if (ep->inline_v2) { + ret = afkv2_encode_inline_command(inline_command, + AFKV2_COMMAND_SIZE + payload_len, + tag, payload, payload_len, output_len); + if (!ret) + ret = afk_send_epic(ep, service->channel, tag, EPIC_TYPE_COMMAND, + EPIC_CAT_COMMAND, type, inline_command, + AFKV2_COMMAND_SIZE + payload_len); + } else { + ret = afk_send_epic(service->ep, service->channel, tag, + EPIC_TYPE_COMMAND, EPIC_CAT_COMMAND, type, &cmd, + sizeof(cmd)); + } + kfree(inline_command); + inline_command = NULL; spin_unlock_irqrestore(&service->lock, flags); if (ret) goto err_free_cmd; @@ -1124,7 +1865,7 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, "AFK[ep:%02x]: %s (chan:%u) command type 0x%x tag 0x%04x timed out after %u ms\n", ep->endpoint, service->ops->name, service->channel, type, tag, timeout_ms); - if (ep->dcp->external) { + if (ep->dcp->external && ep->txbfr.ready && ep->rxbfr.ready) { /* * Shared ring state, not coprocessor MMIO: unequal * TX pointers mean the firmware has not drained @@ -1134,10 +1875,10 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, dev_warn(ep->dcp->dev, "AFK[ep:%02x]: TX r/w %#x/%#x RX r/w %#x/%#x running %d crashed %d\n", ep->endpoint, - le32_to_cpu(READ_ONCE(ep->txbfr.hdr->rptr)), - le32_to_cpu(READ_ONCE(ep->txbfr.hdr->wptr)), - le32_to_cpu(READ_ONCE(ep->rxbfr.hdr->rptr)), - le32_to_cpu(READ_ONCE(ep->rxbfr.hdr->wptr)), + le32_to_cpu(READ_ONCE(*ep->txbfr.rptr)), + le32_to_cpu(READ_ONCE(*ep->txbfr.wptr)), + le32_to_cpu(READ_ONCE(*ep->rxbfr.rptr)), + le32_to_cpu(READ_ONCE(*ep->rxbfr.wptr)), apple_rtkit_is_running(ep->dcp->rtk), apple_rtkit_is_crashed(ep->dcp->rtk)); } @@ -1150,7 +1891,19 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, ret = 0; if (retcode) *retcode = service->cmds[idx].retcode; - if (reply_len) { + if (ep->wire_v2) { + if (reply_len) + copy_len = service->cmds[idx].reply_length; + if (!service->cmds[idx].retcode && + service->cmds[idx].reply_length < copy_len) { + ret = -EMSGSIZE; + } else if (output && output_len && !service->cmds[idx].retcode) { + dma_rmb(); + memcpy(output, rxbuf, copy_len); + if (reply_len) + *reply_len = copy_len; + } + } else if (reply_len) { ret = afk_reply_size_valid(service->cmds[idx].reply_len_valid, service->cmds[idx].reply_len, output_len, 0); @@ -1172,6 +1925,8 @@ static int afk_send_command_full(struct apple_epic_service *service, u8 type, bitmap_release_region(service->cmd_map, idx, 0); err_unlock: spin_unlock_irqrestore(&service->lock, flags); +err_free_txbuf: + kfree(inline_command); dma_free_coherent(ep->dcp->dev, payload_len, txbuf, txbuf_dma); err_free_rxbuf: dma_free_coherent(ep->dcp->dev, output_len, rxbuf, rxbuf_dma); @@ -1210,6 +1965,11 @@ int afk_send_command_with_reply_len(struct apple_epic_service *service, u8 type, MSEC_PER_SEC); } +/* + * One EPIC standard-service call. With @reply_len set, the reply body + * length is validated and returned: against the received bytes on the + * legacy wire, with afkv2_service_reply_size() on AFKV2 endpoints. + */ static int afk_service_call_full(struct apple_epic_service *service, u16 group, u32 command, const void *data, size_t data_len, size_t data_pad, void *output, size_t output_len, @@ -1218,7 +1978,8 @@ static int afk_service_call_full(struct apple_epic_service *service, u16 group, { struct epic_service_call *call; void *bfr; - size_t bfr_len, input_size, output_size, received = 0; + size_t bfr_len, input_size, output_size, received = 0, result_size; + bool v2; int ret; u32 retcode; u32 retlen; @@ -1227,6 +1988,7 @@ static int afk_service_call_full(struct apple_epic_service *service, u16 group, *reply_len = 0; if (!service) return -ENODEV; + v2 = service->ep->wire_v2; if (check_add_overflow(data_len, data_pad, &input_size) || check_add_overflow(output_len, output_pad, &output_size) || check_add_overflow(max(input_size, output_size), sizeof(*call), @@ -1242,12 +2004,15 @@ static int afk_service_call_full(struct apple_epic_service *service, u16 group, memset(call, 0, sizeof(*call)); call->group = cpu_to_le16(group); call->command = cpu_to_le32(command); - call->data_len = cpu_to_le32(data_len + data_pad); + call->data_len = cpu_to_le32(service->ep->inline_v2 ? data_len : + data_len + data_pad); call->magic = cpu_to_le32(EPIC_SERVICE_CALL_MAGIC); memcpy(bfr + sizeof(*call), data, data_len); + /* Inline carries only the request; its response capacity is independent. */ ret = afk_send_command_full(service, EPIC_SUBTYPE_STD_SERVICE, bfr, + service->ep->inline_v2 ? sizeof(*call) + data_len : bfr_len, bfr, bfr_len, &retcode, reply_len ? &received : NULL, timeout_ms); if (ret) { @@ -1266,7 +2031,13 @@ static int afk_service_call_full(struct apple_epic_service *service, u16 group, ret = -EINVAL; goto out; } - if (reply_len) { + if (reply_len && v2) { + ret = afkv2_service_reply_size(bfr, received, group, command, + output_len, &result_size); + if (ret) + goto out; + *reply_len = result_size; + } else if (reply_len) { ret = afk_reply_size_valid(true, received, bfr_len, sizeof(*call)); if (ret) goto out; @@ -1289,7 +2060,7 @@ static int afk_service_call_full(struct apple_epic_service *service, u16 group, group, command, le32_to_cpu(call->data_len)); retlen = le32_to_cpu(call->data_len); - if (reply_len) { + if (reply_len && !v2) { ret = afk_service_body_size_valid(received, sizeof(*call), retlen, output_len); if (ret) @@ -1339,6 +2110,17 @@ int afk_service_call_with_reply_len(struct apple_epic_service *service, reply_len, MSEC_PER_SEC); } +/* The AFKV2 (external H17P) name of afk_service_call_with_reply_len(). */ +int afk_service_call_len(struct apple_epic_service *service, u16 group, + u32 command, const void *data, size_t data_len, + size_t data_pad, void *output, size_t output_len, + size_t output_pad, size_t *received) +{ + return afk_service_call_with_reply_len(service, group, command, data, + data_len, data_pad, output, + output_len, output_pad, received); +} + #if IS_ENABLED(CONFIG_DRM_APPLE_DEBUG) #define AFK_DEBUGFS_MAX_REPLY 8192 diff --git a/drivers/gpu/drm/apple/afk.h b/drivers/gpu/drm/apple/afk.h index f0a2ce2e70c676..1186ec19f2f758 100644 --- a/drivers/gpu/drm/apple/afk.h +++ b/drivers/gpu/drm/apple/afk.h @@ -9,20 +9,27 @@ #include #include +#include #include #include +#include #include #include "dcp.h" #define AFK_MAX_CHANNEL 16 #define MAX_PENDING_CMDS 16 +#define AFKV2_INTERFACES BIT(16) struct apple_epic_service_ops; struct apple_dcp_afkep; +void afk_service_retry(struct apple_dcp_afkep *ep); + struct epic_cmd_info { u16 tag; + u8 type; + size_t reply_length; void *rxbuf; void *txbuf; @@ -46,6 +53,7 @@ struct apple_epic_service { struct epic_cmd_info cmds[MAX_PENDING_CMDS]; DECLARE_BITMAP(cmd_map, MAX_PENDING_CMDS); + /* Legacy EPIC sequence; AFKV2 history belongs to the interface. */ u8 cmd_tag; spinlock_t lock; /* Pins held by owners and synchronous callers; protected by lock. */ @@ -55,8 +63,15 @@ struct apple_epic_service { bool enabled; bool torndown; + /* Provider-owned prepare/init ticket; AFK core never interprets it. */ + struct { u64 generation, epoch; } admission; void *cookie; + /* Admitted reannouncement waiting for the previous channel to drain. */ + bool pending_open; + const char *pending_name, *pending_class; + s64 pending_unit; + struct { struct dentry *entry; u8 *scratch; @@ -69,17 +84,53 @@ struct apple_epic_service_ops { const char name[32]; /* All retained pointers must be pinned before enabling slot reuse. */ bool reusable; - + /* EOPNOTSUPP from prepare means intentional absence only for this role. */ + bool optional; + + /* + * Optional admission, before OPEN/publication. Must not retain service + * on failure; teardown must accept a failed/partial preparation. + */ + int (*prepare)(struct apple_epic_service *service, const char *name, + const char *class, s64 unit); + /* Check the original prepare ticket before a deferred OPEN. */ + int (*revalidate)(struct apple_epic_service *service, const char *name, + const char *class, s64 unit); void (*init)(struct apple_epic_service *service, const char *name, const char *class, s64 unit); int (*call)(struct apple_epic_service *service, u32 idx, const void *data, size_t data_size, void *reply, size_t reply_size); + int (*call_v2)(struct apple_epic_service *service, u16 group, u32 command, + const void *data, size_t size, void *reply, size_t room); int (*report)(struct apple_epic_service *service, enum epic_subtype type, const void *data, size_t data_size); void (*teardown)(struct apple_epic_service *service); }; +static inline int afk_ring_stride(u32 total, u32 bufsz, u32 *stride) +{ + u32 header, block; + + if (bufsz >= total) + return -EINVAL; + header = total - bufsz; + block = header / 3; + if (header % 3 || block < 0x40 || block % 0x40 || + bufsz < block || bufsz % block) + return -EINVAL; + + *stride = block; + return 0; +} + +static inline u32 afk_ring_advance(u32 ptr, u32 bytes, u32 bufsz, + u32 stride) +{ + u32 next = roundup(ptr + bytes, stride); + + return next == bufsz ? 0 : next; +} static inline struct apple_epic_service *afk_service_get(struct apple_epic_service *service) { unsigned long flags; @@ -105,6 +156,26 @@ static inline void afk_service_put(struct apple_epic_service *service) if (!WARN_ON(!service->users)) service->users--; spin_unlock_irqrestore(&service->lock, flags); + afk_service_retry(service->ep); +} + +/* + * Take another pin on a service the caller already keeps pinned, e.g. an + * owner pointer read under the owner's lock. Unlike afk_service_get() this + * also succeeds after teardown, so a close can still be issued. + */ +static inline struct apple_epic_service *afk_service_pin(struct apple_epic_service *service) +{ + unsigned long flags; + + if (!service) + return NULL; + + spin_lock_irqsave(&service->lock, flags); + service->users++; + spin_unlock_irqrestore(&service->lock, flags); + + return service; } static inline void afk_service_disable(struct apple_epic_service *service) @@ -116,6 +187,17 @@ static inline void afk_service_disable(struct apple_epic_service *service) spin_unlock_irqrestore(&service->lock, flags); } +/* Retire local ownership while keeping timed-out commands discoverable. */ +static inline void afk_service_retire(struct apple_epic_service *service) +{ + unsigned long flags; + + spin_lock_irqsave(&service->lock, flags); + service->enabled = false; + service->torndown = true; + spin_unlock_irqrestore(&service->lock, flags); +} + static inline bool afk_service_matches(struct apple_epic_service *service, u32 channel) { @@ -151,18 +233,42 @@ afk_service_reinit(struct apple_epic_service *service, spin_lock_irqsave(&service->lock, flags); reusable = afk_service_reclaimable(service); if (reusable) { - service->enabled = true; + service->enabled = !ops->prepare; service->torndown = false; service->ops = ops; service->ep = ep; service->channel = channel; service->cookie = NULL; - /* Keep the command-tag sequence across service generations. */ + /* Keep legacy slot history; AFKV2 history belongs to the interface. */ } spin_unlock_irqrestore(&service->lock, flags); return reusable; } +/* Only used before a prepared service has been published to callers. */ +static inline void afk_service_discard(struct apple_epic_service *service) +{ + unsigned long flags; + + if (service->ops->teardown) + service->ops->teardown(service); + spin_lock_irqsave(&service->lock, flags); + service->enabled = false; + service->torndown = true; + service->cookie = NULL; + service->ops = NULL; + spin_unlock_irqrestore(&service->lock, flags); +} + +static inline void afk_service_publish(struct apple_epic_service *service) +{ + unsigned long flags; + + spin_lock_irqsave(&service->lock, flags); + service->enabled = true; + spin_unlock_irqrestore(&service->lock, flags); +} + struct afk_ringbuffer_header { __le32 bufsz; u32 unk; @@ -201,6 +307,21 @@ struct epic_sub_hdr { __le32 inline_len; } __attribute__((packed)); +/* + * H17P compresses the EPIC sub-header from 24 bytes to 8 and drops the + * separate length/timestamp: the announce for the "system" service puts its + * name at data + 0x18, not data + 0x28. An announce carries type 0x11 and a + * standard-service report carries the usual 0xc0. + */ +struct epic_sub_hdr_h17p { + u8 type; + u8 category; + __le16 flags; + __le32 tag; +} __packed; + +#define EPIC_SUBTYPE_ANNOUNCE_H17P 0x11 + struct epic_cmd { __le32 retcode; __le64 rxbuf; @@ -244,17 +365,26 @@ enum epic_subtype { struct afk_ringbuffer { bool ready; - struct afk_ringbuffer_header *hdr; - u32 rptr; + __le32 *rptr; + __le32 *wptr; void *buf; size_t bufsz; + u32 stride; }; + struct apple_dcp_afkep { struct apple_dcp *dcp; u32 endpoint; + /* Set before afk_start; protocol selection is per endpoint. */ + bool wire_v2; + /* External H17P DPAV only; other endpoints retain owned-DMA commands. */ + bool inline_v2; + /* Fixed 16-bit interface domain; survives slot and channel reuse. */ + u8 *command_tags; struct workqueue_struct *wq; + struct work_struct service_work; struct completion started; struct completion stopped; @@ -284,8 +414,11 @@ struct apple_dcp_afkep { struct apple_dcp_afkep *afk_init(struct apple_dcp *dcp, u32 endpoint, const struct apple_epic_service_ops *ops); +int afk_v2_copy_owned(struct apple_epic_service *service, u64 dva, + void *buffer, size_t size, bool to_peer); int afk_start(struct apple_dcp_afkep *ep); void afk_shutdown(struct apple_dcp_afkep *ep); +void afk_quiesce(struct apple_dcp_afkep *ep); int afk_receive_message(struct apple_dcp_afkep *ep, u64 message); int afk_send_epic(struct apple_dcp_afkep *ep, u32 channel, u16 tag, enum epic_type etype, enum epic_category ecat, u8 stype, @@ -312,4 +445,9 @@ int afk_service_call_with_reply_len(struct apple_epic_service *service, u16 group, u32 command, const void *data, size_t data_len, size_t data_pad, void *output, size_t output_len, size_t output_pad, size_t *reply_len); +int afk_service_call_len(struct apple_epic_service *service, u16 group, + u32 command, const void *data, size_t data_len, + size_t data_pad, void *output, size_t output_len, + size_t output_pad, size_t *received); + #endif diff --git a/drivers/gpu/drm/apple/afk_test.c b/drivers/gpu/drm/apple/afk_test.c new file mode 100644 index 00000000000000..e58a9526420458 --- /dev/null +++ b/drivers/gpu/drm/apple/afk_test.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Ryan Murray */ + +#include +#include + +#include "afk.h" + +static void afk_ring_stride_test(struct kunit *test) +{ + u32 stride; + + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10c0, 0x1000, &stride), 0); + KUNIT_EXPECT_EQ(test, stride, 0x40U); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x1180, 0x1000, &stride), 0); + KUNIT_EXPECT_EQ(test, stride, 0x80U); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x1000, 0x1000, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10bd, 0x1000, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10c1, 0x1000, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0xc0, 0, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10c3, 0x1000, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10c0, 0x1001, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x11c0, 0x1040, &stride), -EINVAL); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x1180, 0x1000, &stride), 0); + KUNIT_EXPECT_EQ(test, afk_ring_advance(0xf80, 0x80, 0x1000, stride), 0U); + KUNIT_EXPECT_EQ(test, afk_ring_advance(0xe80, 0x80, 0x1000, stride), + 0xf00U); + KUNIT_EXPECT_EQ(test, afk_ring_stride(0x10c0, 0x1000, &stride), 0); + KUNIT_EXPECT_EQ(test, afk_ring_advance(0xfc0, 0x40, 0x1000, stride), 0U); +} + +static struct kunit_case afk_ring_cases[] = { + KUNIT_CASE(afk_ring_stride_test), + {} +}; + +static struct kunit_suite afk_ring_suite = { + .name = "apple-afk-ring", + .test_cases = afk_ring_cases, +}; + +kunit_test_suite(afk_ring_suite); + +MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple/afkv2-codec.h b/drivers/gpu/drm/apple/afkv2-codec.h new file mode 100644 index 00000000000000..c42788a1b2a74e --- /dev/null +++ b/drivers/gpu/drm/apple/afkv2-codec.h @@ -0,0 +1,352 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* Copyright 2026 Aurora Silicon */ +/* Wire layout: public Linux PR254 and m1n1 AFKV2 implementations. */ +#ifndef AURORA_AFKV2_CODEC_H +#define AURORA_AFKV2_CODEC_H + +#include +#include +#include + +#define AFKV2_HEADER_SIZE 24U +#define AFKV2_COMMAND_SIZE 8U +#define AFKV2_OOB_SIZE 24U +#define AFKV2_MAX_PAYLOAD 1048576U +#define AFKV2_REPORT 0U +#define AFKV2_COMMAND 1U +#define AFKV2_RESPONSE 2U +#define AFKV2_OPEN 0x12U +#define AFKV2_CLOSE 0x13U +#define AFKV2_CLOSED 0x14U + +struct afkv2_frame { + u8 sequence; + u8 transport_flags; + u16 interface; + u64 token; + u8 type; + u8 category; + u16 interface_flags; + u32 reserved; + const u8 *payload; + size_t payload_size; + size_t padding_size; +}; + +struct afkv2_command { + u8 id; + u32 value; + bool oob; + u64 rx_address; + u64 tx_address; + u32 rx_size; + u32 tx_size; + const u8 *inline_data; + size_t inline_size; +}; + +static inline u16 afkv2_get16(const u8 *p) +{ + return (u16)p[0] | (u16)p[1] << 8; +} + +static inline u32 afkv2_get32(const u8 *p) +{ + return (u32)p[0] | (u32)p[1] << 8 | (u32)p[2] << 16 | (u32)p[3] << 24; +} + +static inline u64 afkv2_get64(const u8 *p) +{ + return (u64)afkv2_get32(p) | (u64)afkv2_get32(p + 4) << 32; +} + +static inline void afkv2_put16(u8 *p, u16 value) +{ + p[0] = value; + p[1] = value >> 8; +} + +static inline void afkv2_put32(u8 *p, u32 value) +{ + p[0] = value; + p[1] = value >> 8; + p[2] = value >> 16; + p[3] = value >> 24; +} + +static inline void afkv2_put64(u8 *p, u64 value) +{ + afkv2_put32(p, value); + afkv2_put32(p + 4, value >> 32); +} + +/* Decode metadata only. Never follow an address supplied by the peer. */ +static inline int afkv2_decode_frame(const void *record, size_t size, + bool allow_padding, struct afkv2_frame *out) +{ + const u8 *p = record; + u32 length; + struct afkv2_frame result = {0}; + + if (!record || !out || size < AFKV2_HEADER_SIZE) + return -EINVAL; + if (size > AFKV2_HEADER_SIZE + AFKV2_MAX_PAYLOAD) + return -E2BIG; + length = afkv2_get32(p + 4); + if (length < 16 || length - 16 > AFKV2_MAX_PAYLOAD || length > size - 8) + return -EINVAL; + if (!allow_padding && length != size - 8) + return -EINVAL; + if (p[17] > AFKV2_RESPONSE) + return -EPROTO; + result.sequence = p[0]; + result.transport_flags = p[1]; + result.interface = afkv2_get16(p + 2); + result.token = afkv2_get64(p + 8); + result.type = p[16]; + result.category = p[17]; + result.interface_flags = afkv2_get16(p + 18); + result.reserved = afkv2_get32(p + 20); + result.payload = p + AFKV2_HEADER_SIZE; + result.payload_size = length - 16; + result.padding_size = size - 8 - length; + *out = result; + return 0; +} + +/* Closed notification: flags occupy the low byte; the high byte is opaque. */ +static inline bool afkv2_closed_report(const struct afkv2_frame *frame) +{ + return frame && !frame->token && frame->category == AFKV2_REPORT && + frame->type == AFKV2_CLOSED && + (frame->interface_flags & 0xff) == 1 && frame->payload_size == 1 && + frame->payload && !frame->payload[0]; +} + +static inline int afkv2_encode_frame(void *record, size_t capacity, u8 sequence, + u16 interface, u64 token, u8 type, u8 category, + u16 flags, const void *payload, size_t payload_size, + size_t *written) +{ + u8 *p = record; + + if (!record || !written || !interface || category > AFKV2_RESPONSE || + (payload_size && !payload)) + return -EINVAL; + if (payload_size > AFKV2_MAX_PAYLOAD) + return -E2BIG; + if (capacity < AFKV2_HEADER_SIZE || payload_size > capacity - AFKV2_HEADER_SIZE) + return -ENOSPC; + /* Move first so an in-buffer payload is safe even when it overlaps. */ + if (payload_size) + memmove(p + AFKV2_HEADER_SIZE, payload, payload_size); + memset(p, 0, AFKV2_HEADER_SIZE); + p[0] = sequence; + afkv2_put16(p + 2, interface); + afkv2_put32(p + 4, payload_size + 16); + afkv2_put64(p + 8, token); + p[16] = type; + p[17] = category; + afkv2_put16(p + 18, flags); + *written = AFKV2_HEADER_SIZE + payload_size; + return 0; +} + +static inline int afkv2_encode_open(void *record, size_t capacity, u8 sequence, + u16 interface, size_t *written) +{ + return afkv2_encode_frame(record, capacity, sequence, interface, 0, + AFKV2_OPEN, AFKV2_REPORT, 1, NULL, 0, written); +} + +static inline bool afkv2_valid_span(u64 address, u32 size) +{ + if (!size) + return !address; + return address && size <= AFKV2_MAX_PAYLOAD && + address <= ~(u64)0 - (size - 1); +} + +static inline int afkv2_encode_oob_command(void *payload, size_t capacity, + u8 id, u64 rx_address, u32 rx_size, + u64 tx_address, u32 tx_size) +{ + u8 *p = payload; + + if (!payload || !afkv2_valid_span(rx_address, rx_size) || + !afkv2_valid_span(tx_address, tx_size)) + return -EINVAL; + if (capacity < AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE) + return -ENOSPC; + memset(p, 0, AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE); + p[0] = 1; + p[1] = id; + afkv2_put32(p + 4, rx_size); + afkv2_put64(p + 8, rx_address); + afkv2_put64(p + 16, tx_address); + afkv2_put32(p + 24, rx_size); + afkv2_put32(p + 28, tx_size); + return 0; +} + +/* Inline requests carry a response capacity separately from their body. */ +static inline int afkv2_encode_inline_command(void *payload, size_t capacity, + u8 id, const void *body, + size_t size, u32 response_capacity) +{ + u8 *p = payload; + + if (!payload || (size && !body) || size > AFKV2_MAX_PAYLOAD - AFKV2_COMMAND_SIZE || + response_capacity > AFKV2_MAX_PAYLOAD) + return -EINVAL; + if (capacity < AFKV2_COMMAND_SIZE || size > capacity - AFKV2_COMMAND_SIZE) + return -ENOSPC; + if (size) + memmove(p + AFKV2_COMMAND_SIZE, body, size); + memset(p, 0, AFKV2_COMMAND_SIZE); + p[1] = id; + afkv2_put32(p + 4, response_capacity); + return 0; +} + +static inline int afkv2_decode_command(const void *payload, size_t size, + struct afkv2_command *out) +{ + const u8 *p = payload; + struct afkv2_command result = {0}; + size_t header = AFKV2_COMMAND_SIZE; + + if (!payload || !out || size < header || p[0] & ~1U || afkv2_get16(p + 2)) + return -EINVAL; + result.id = p[1]; + result.value = afkv2_get32(p + 4); + result.oob = p[0] & 1; + if (result.oob) { + header += AFKV2_OOB_SIZE; + if (size < header) + return -EINVAL; + result.rx_address = afkv2_get64(p + 8); + result.tx_address = afkv2_get64(p + 16); + result.rx_size = afkv2_get32(p + 24); + result.tx_size = afkv2_get32(p + 28); + /* A failed response may retain addresses with zero lengths. */ + if ((result.rx_size && !afkv2_valid_span(result.rx_address, result.rx_size)) || + (result.tx_size && !afkv2_valid_span(result.tx_address, result.tx_size))) + return -EINVAL; + } + result.inline_data = p + header; + result.inline_size = size - header; + if (result.inline_size > AFKV2_MAX_PAYLOAD) + return -E2BIG; + *out = result; + return 0; +} + +/* Requests and responses may have different owned-buffer capacities. */ +static inline int afkv2_callback_lengths(const struct afkv2_command *command, + size_t minimum, size_t limit, + size_t *request, size_t *response) +{ + size_t in, out; + + if (!command || !request || !response || minimum > limit) + return -EINVAL; + in = command->oob ? command->tx_size : command->inline_size; + out = command->oob ? command->rx_size : command->inline_size; + if (in < minimum || out < minimum || in > limit || out > limit || + in > command->value || out > command->value) + return -EINVAL; + *request = in; + *response = out; + return 0; +} + +/* Correlate only; no buffer is read, written or freed by this helper. */ +static inline int afkv2_check_response(const struct afkv2_frame *frame, + const struct afkv2_command *response, + u16 interface, u8 type, u8 id, + u64 rx_address, u32 rx_capacity, + u64 tx_address, u32 tx_capacity) +{ + if (!frame || !response || frame->category != AFKV2_RESPONSE || + frame->interface != interface || frame->type != type || response->id != id) + return -EPROTO; + if (response->value) + return 0; + if (!response->oob || response->inline_size || + response->rx_address != rx_address || response->tx_address != tx_address || + response->rx_size > rx_capacity || response->tx_size > tx_capacity) + return -EPROTO; + return 0; +} + +/* Kept separate so DPAV inline replies cannot loosen owned-DMA checks. + * Transport/interface flags and the reserved word are opaque firmware data. + */ +static inline int afkv2_check_inline_response(const struct afkv2_frame *frame, + const struct afkv2_command *response, + u16 interface, u8 type, u8 id, + size_t capacity) +{ + if (!frame || !response || frame->category != AFKV2_RESPONSE || + frame->interface != interface || frame->type != type || response->id != id || + frame->token || response->oob || response->inline_size > capacity) + return -EPROTO; + return 0; +} + +/* The standard service envelope precedes the actual, possibly short reply. */ +static inline int afkv2_service_reply_size(const void *body, size_t received, + u16 group, u32 command, size_t capacity, + size_t *size) +{ + const u8 *p = body; + u32 length; + + if (!body || !size || received < 64 || afkv2_get32(p + 12) != 0x69706378 || + afkv2_get16(p + 2) != group || afkv2_get32(p + 4) != command) + return -EPROTO; + length = afkv2_get32(p + 8); + if (length > received - 64 || length > capacity) + return -EMSGSIZE; + *size = length; + return 0; +} + +/* Reject a peer command without touching its advertised DMA buffers. */ +static inline int afkv2_encode_error_reply(void *record, size_t capacity, u8 sequence, + const struct afkv2_frame *request, + const struct afkv2_command *command, + u32 status, size_t *written) +{ + u8 payload[AFKV2_COMMAND_SIZE + AFKV2_OOB_SIZE] = {0}; + size_t size = AFKV2_COMMAND_SIZE; + + if (!request || !command || request->category != AFKV2_COMMAND || !status) + return -EINVAL; + payload[0] = command->oob ? 1 : 0; + payload[1] = command->id; + afkv2_put32(payload + 4, status); + if (command->oob) { + afkv2_put64(payload + 8, command->rx_address); + afkv2_put64(payload + 16, command->tx_address); + size += AFKV2_OOB_SIZE; + } + return afkv2_encode_frame(record, capacity, sequence, request->interface, + request->token, request->type, AFKV2_RESPONSE, 0, + payload, size, written); +} + +static inline u8 afkv2_command_id(u8 generation, u8 slot) +{ + return ((generation & 15) << 4) | (slot & 15); +} + +static inline bool afkv2_owned_span(u64 base, size_t length, u64 address, size_t size) +{ + if (!length || base > ~(u64)0 - (length - 1)) + return false; + return address >= base && size <= length && address - base <= length - size; +} + +#endif diff --git a/drivers/gpu/drm/apple/apple_drv.c b/drivers/gpu/drm/apple/apple_drv.c index d53e5654393e6f..50b19346d641c4 100644 --- a/drivers/gpu/drm/apple/apple_drv.c +++ b/drivers/gpu/drm/apple/apple_drv.c @@ -42,6 +42,7 @@ #include +#include "apple_resume.h" #include "dcp.h" #include "plane.h" @@ -68,6 +69,29 @@ static int apple_drm_gem_dumb_create(struct drm_file *file_priv, return drm_gem_dma_dumb_create_internal(file_priv, drm, args); } +#ifdef CONFIG_DRM_FBDEV_EMULATION +static int apple_drm_fbdev_probe(struct drm_fb_helper *helper, + struct drm_fb_helper_surface_size *sizes) +{ + struct drm_crtc *crtc; + + drm_for_each_crtc(crtc, helper->dev) { + struct apple_dcp *dcp; + + dcp = platform_get_drvdata(to_apple_crtc(crtc)->dcp); + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + /* H17P applies the stored fourth byte as straight alpha. */ + sizes->surface_bpp = 32; + sizes->surface_depth = 32; + break; + } + } + + return drm_fbdev_dma_driver_fbdev_probe(helper, sizes); +} +#endif + /* * A compositor or boot splash is closing the device. drm_release() has * already taken its file off the file list, so the Type-C fabric can tell @@ -81,7 +105,11 @@ static void apple_drm_postclose(struct drm_device *drm, struct drm_file *file) static const struct drm_driver apple_drm_driver = { DRM_GEM_DMA_DRIVER_OPS_WITH_DUMB_CREATE(apple_drm_gem_dumb_create), +#ifdef CONFIG_DRM_FBDEV_EMULATION + .fbdev_probe = apple_drm_fbdev_probe, +#else DRM_FBDEV_DMA_DRIVER_OPS, +#endif .postclose = apple_drm_postclose, .name = DRIVER_NAME, .desc = DRIVER_DESC, @@ -145,7 +173,8 @@ static void apple_crtc_atomic_enable(struct drm_crtc *crtc, } if (crtc_state->active && !dcp_crtc_route_ready(crtc, state, true)) goto failed; - if (crtc_state->active && !dcp->main_display && !apple_crtc->fabric_held) { + if (crtc_state->active && !dcp->main_display && !apple_crtc->fabric_held && + dcp->fw_compat != DCP_FIRMWARE_H17P) { /* Raise the fabric before external scanout starts. */ if (apple_pmgr_fabric_hold()) dev_warn(dcp->dev, "fabric raise not confirmed before scanout\n"); @@ -155,6 +184,9 @@ static void apple_crtc_atomic_enable(struct drm_crtc *crtc, (crtc_state->active_changed || dcp_crtc_needs_route_start(dcp))) { dcp_poweron(apple_crtc->dcp); crtc_state->color_mgmt_changed = true; + } else if (crtc_state->active && drm_atomic_crtc_needs_modeset(crtc_state)) { + /* A cable generation can change while the CRTC stays active. */ + dcp_external_recover(to_apple_crtc(crtc)->dcp); } if (crtc_state->active) dcp_crtc_atomic_modeset(crtc, state); @@ -177,10 +209,11 @@ static void apple_crtc_atomic_disable(struct drm_crtc *crtc, struct drm_atomic_state *state) { struct drm_crtc_state *crtc_state; + struct apple_crtc *apple_crtc = to_apple_crtc(crtc); + crtc_state = drm_atomic_get_new_crtc_state(state, crtc); if (crtc_state->active_changed && !crtc_state->active) { - struct apple_crtc *apple_crtc = to_apple_crtc(crtc); dcp_poweroff(apple_crtc->dcp); if (apple_crtc->fabric_held) { apple_crtc->fabric_held = false; @@ -317,8 +350,29 @@ static const struct drm_mode_config_funcs apple_mode_config_funcs = { .fb_create = drm_gem_fb_create, }; +static void apple_atomic_commit_tail(struct drm_atomic_state *state) +{ + struct drm_plane *plane; + struct drm_plane_state *old, *new; + int i; + + /* Retain displaced scanout even when the CRTC stays inactive. */ + for_each_oldnew_plane_in_state(state, plane, old, new, i) { + struct apple_plane_state *apple_state = to_apple_plane_state(new); + struct dcp_fb_reference *entry = apple_state->retirement; + + if (!entry) + continue; + + apple_state->retirement = NULL; + dcp_retain_framebuffer(to_apple_crtc(old->crtc)->dcp, entry); + } + + drm_atomic_helper_commit_tail_rpm(state); +} + static const struct drm_mode_config_helper_funcs apple_mode_config_helpers = { - .atomic_commit_tail = drm_atomic_helper_commit_tail_rpm, + .atomic_commit_tail = apple_atomic_commit_tail, }; static void appledrm_connector_cleanup(struct drm_connector *connector) @@ -372,6 +426,7 @@ apple_connector_atomic_best_encoder(struct drm_connector *conn, struct drm_atomic_state *state) { struct apple_connector *apple_connector = to_apple_connector(conn); + struct drm_connector_state *conn_state; struct drm_encoder *encoder; /* @@ -380,9 +435,23 @@ apple_connector_atomic_best_encoder(struct drm_connector *conn, * the fabric routes within: on dual-stream machines, and where the * route follows the CRTC (see dcp_typec_follow_crtc()). So it is the * only answer there is. + * + * H17P (T8140, T8142): Type-C tunnels are routed before the modeset + * and cannot move here. Reject stale topology and explicit assignments + * to another pipeline. */ - if (apple_connector->port_encoder) + if (apple_connector->port_encoder) { + struct apple_dcp *target; + + conn_state = drm_atomic_get_new_connector_state(state, conn); + target = conn_state && conn_state->crtc ? + platform_get_drvdata(to_apple_crtc(conn_state->crtc)->dcp) : NULL; + if (target && target->fw_compat == DCP_FIRMWARE_H17P && + READ_ONCE(apple_connector->dcp) != + to_apple_crtc(conn_state->crtc)->dcp) + return NULL; return apple_connector->port_encoder; + } drm_connector_for_each_possible_encoder(conn, encoder) return encoder; @@ -473,8 +542,16 @@ static int apple_probe_per_dcp(struct device *dev, u32 surf; int zpos = 0; bool supports_l10r = !dcp_fw_compat_is_12_x(dcp); + struct apple_dcp *dcp_data = platform_get_drvdata(dcp); + bool supports_xrgb2101010; enum drm_plane_type plane_type; + supports_xrgb2101010 = dcp_data->fw_compat != DCP_FIRMWARE_H17P || + dcp_data->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G; + if (dcp_data->fw_compat == DCP_FIRMWARE_H17P && + bitmap_empty(iomfb_surfaces, DCP_MAX_PLANES)) + return dev_err_probe(dev, -EINVAL, "No usable display surfaces\n"); + if (dcp_fw_compat_is_14_7(dcp)) { planes[0] = apple_plane_init_v14_7(drm, 1U << num, 0, DRM_PLANE_TYPE_PRIMARY); @@ -491,7 +568,9 @@ static int apple_probe_per_dcp(struct device *dev, break; plane_type = (zpos == 0) ? DRM_PLANE_TYPE_PRIMARY : DRM_PLANE_TYPE_OVERLAY; planes[zpos] = apple_plane_init(drm, 1U << num, surf, - supports_l10r, plane_type); + supports_l10r, + supports_xrgb2101010, + plane_type); if (IS_ERR(planes[zpos])) return PTR_ERR(planes[zpos]); @@ -509,6 +588,8 @@ static int apple_probe_per_dcp(struct device *dev, * knows what to do with overlays. */ crtc = kzalloc(sizeof(*crtc), GFP_KERNEL); + if (!crtc) + return -ENOMEM; ret = drm_crtc_init_with_planes(drm, &crtc->base, planes[0], NULL, &apple_crtc_funcs, NULL); if (ret) @@ -730,6 +811,28 @@ static void apple_drm_quiesce_connectors(struct drm_device *drm) drm_connector_list_iter_end(&iter); } +/* Does any pipeline of @drm run H17-generation firmware (T8140, T8142)? */ +static bool apple_drm_has_h17p(struct drm_device *drm) +{ + struct drm_crtc *crtc; + + drm_for_each_crtc(crtc, drm) { + struct apple_dcp *dcp = platform_get_drvdata(to_apple_crtc(crtc)->dcp); + + if (dcp && dcp->fw_compat == DCP_FIRMWARE_H17P) + return true; + } + return false; +} + +/* A supported firmware-described external provider participates in component matching. */ +static bool apple_dcp_usable(struct device_node *np) +{ + return of_device_is_available(np) && + (!of_device_is_compatible(np, "apple,t8140-dcpext") || + dcp_h17p_external_supported(np)); +} + static int apple_drm_init_dcp(struct device *dev) { struct apple_drm_private *apple = dev_get_drvdata(dev); @@ -742,17 +845,20 @@ static int apple_drm_init_dcp(struct device *dev) for_each_matching_node(np, apple_dcp_id_tbl) { bool dcp_ext; - if (!apple_dcp_in_drm(np)) - continue; - if (!of_device_is_available(np)) { - of_node_put(np); + if (!apple_dcp_in_drm(np) || !apple_dcp_usable(np)) continue; - } dcp_ext = of_device_is_compatible(np, "apple,dcpext") || + of_device_is_compatible(np, "apple,t8140-dcpext") || of_property_present(np, "phys"); + /* component_bind_all() established the matched provider lifetime. */ + if (of_device_is_compatible(np, "apple,t8140-dcpext") && + !dcp_h17p_external_ready(np)) { + of_node_put(np); + return -ENODEV; + } + dcp[num_dcp] = of_find_device_by_node(np); - of_node_put(np); if (!dcp[num_dcp]) continue; /* A failed T8122 dcpext: bound, but no pipeline. */ @@ -768,8 +874,14 @@ static int apple_drm_init_dcp(struct device *dev) ret = apple_probe_per_dcp(dev, &apple->drm, dcp[num_dcp], num_dcp, dcp_ext, &encoder[num_dcp], &crtc_mask[num_dcp]); - if (ret) + if (ret) { + if (of_device_is_compatible(np, "apple,t8140-dcpext")) { + put_device(&dcp[num_dcp]->dev); + of_node_put(np); + return ret; + } continue; + } num_dcp++; } @@ -906,7 +1018,11 @@ static void apple_drm_uninit(struct device *dev) struct apple_drm_private *apple = dev_get_drvdata(dev); apple_drm_quiesce_connectors(&apple->drm); - drm_dev_unregister(&apple->drm); + /* H17P keeps its live scanout across unbind: unplug, as aurora-wip. */ + if (apple_drm_has_h17p(&apple->drm)) + drm_dev_unplug(&apple->drm); + else + drm_dev_unregister(&apple->drm); drm_atomic_helper_shutdown(&apple->drm); component_unbind_all(dev, NULL); @@ -937,7 +1053,7 @@ static int add_dcp_components(struct device *dev, int num = 0; for_each_matching_node(np, apple_dcp_id_tbl) { - if (apple_dcp_in_drm(np) && of_device_is_available(np)) { + if (apple_dcp_in_drm(np) && apple_dcp_usable(np)) { drm_of_component_match_add(dev, matchptr, component_compare_of, np); num++; @@ -971,7 +1087,6 @@ static int add_dcp_components(struct device *dev, of_node_put(port); } } - of_node_put(np); } return num; @@ -979,9 +1094,6 @@ static int add_dcp_components(struct device *dev, static int apple_platform_probe(struct platform_device *pdev) { - if (of_machine_is_compatible("apple,t8140")) - return -ENODEV; - struct device *mdev = &pdev->dev; struct component_match *match = NULL; int num_dcp; @@ -1004,6 +1116,8 @@ static const struct of_device_id of_match[] = { { .compatible = "apple,t6030-display-subsystem" }, { .compatible = "apple,t8122-display-subsystem" }, { .compatible = "apple,t6031-display-subsystem" }, + { .compatible = "apple,t8140-display-subsystem" }, + { .compatible = "apple,t8142-display-subsystem" }, {} }; MODULE_DEVICE_TABLE(of, of_match); @@ -1022,9 +1136,18 @@ static int apple_platform_suspend(struct device *dev) static int apple_platform_resume(struct device *dev) { struct apple_drm_private *apple = dev_get_drvdata(dev); + bool dropped = false; - if (apple) + if (apple) { + /* H17P (T8140, T8142) only, as on aurora-wip. */ + if (apple->drm.mode_config.suspend_state && + apple_drm_has_h17p(&apple->drm)) + dropped = apple_drm_drop_stale_crtcs(apple->drm.mode_config.suspend_state); drm_mode_config_helper_resume(&apple->drm); + /* Have userspace set the mode of a CRTC resumed disabled again. */ + if (dropped) + drm_kms_helper_hotplug_event(&apple->drm); + } return 0; } diff --git a/drivers/gpu/drm/apple/apple_resume.c b/drivers/gpu/drm/apple/apple_resume.c new file mode 100644 index 00000000000000..52dabd11d3e510 --- /dev/null +++ b/drivers/gpu/drm/apple/apple_resume.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Ryan Murray */ + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "apple_resume.h" + +/* + * A mode can be admissible only while the link it was proven on is up: the + * external H17P 180 Hz mode needs the current DPTX link training, and system + * sleep releases that link. Restoring such a CRTC as it was fails the whole + * resume commit in its mode fixup, which leaves every output dark until + * userspace reacts to the next hotplug. Restore that CRTC disabled instead. + * Userspace still believes the CRTC is on, and the DCP need not report the + * same display again when its link comes back, so mark the link of each of + * its connectors bad: userspace then sets the mode again, on the hotplug + * event the caller sends, as after a failed link retraining. + * + * The suspend state holds every CRTC, plane and connector already, so it is + * edited directly rather than through helpers that need the modeset locks. + * + * Returns true if a CRTC is to be resumed disabled. + */ +bool apple_drm_drop_stale_crtcs(struct drm_atomic_state *state) +{ + const struct drm_crtc_helper_funcs *funcs; + struct drm_connector_state *conn_state; + struct drm_plane_state *plane_state; + struct drm_crtc_state *crtc_state; + struct drm_connector *connector; + struct drm_plane *plane; + struct drm_crtc *crtc; + bool dropped = false; + int i, j; + + for_each_new_crtc_in_state(state, crtc, crtc_state, i) { + funcs = crtc->helper_private; + if (!crtc_state->enable || !funcs || !funcs->mode_fixup || + funcs->mode_fixup(crtc, &crtc_state->mode, + &crtc_state->adjusted_mode)) + continue; + + drm_dbg_kms(crtc->dev, + "[CRTC:%d:%s] " DRM_MODE_FMT " needs a new link, resuming disabled\n", + crtc->base.id, crtc->name, DRM_MODE_ARG(&crtc_state->mode)); + + for_each_new_plane_in_state(state, plane, plane_state, j) { + if (plane_state->crtc != crtc) + continue; + crtc_state->plane_mask &= ~drm_plane_mask(plane); + plane_state->crtc = NULL; + drm_atomic_set_fb_for_plane(plane_state, NULL); + } + for_each_new_connector_in_state(state, connector, conn_state, j) { + if (conn_state->crtc != crtc) + continue; + /* Detaching from a CRTC already in the state cannot fail. */ + if (WARN_ON(drm_atomic_set_crtc_for_connector(conn_state, NULL))) + return true; + /* + * Nor does a connector without a CRTC keep an encoder: + * the resume check warns about one left routed. + */ + if (conn_state->best_encoder) + crtc_state->encoder_mask &= + ~drm_encoder_mask(conn_state->best_encoder); + conn_state->best_encoder = NULL; + conn_state->link_status = DRM_LINK_STATUS_BAD; + } + crtc_state->active = false; + dropped = true; + if (WARN_ON(drm_atomic_set_mode_for_crtc(crtc_state, NULL))) + break; + } + + return dropped; +} diff --git a/drivers/gpu/drm/apple/apple_resume.h b/drivers/gpu/drm/apple/apple_resume.h new file mode 100644 index 00000000000000..de0dffe2dc8bd5 --- /dev/null +++ b/drivers/gpu/drm/apple/apple_resume.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* Copyright 2026 Ryan Murray */ + +#ifndef __APPLE_RESUME_H__ +#define __APPLE_RESUME_H__ + +#include + +struct drm_atomic_state; + +bool apple_drm_drop_stale_crtcs(struct drm_atomic_state *state); + +#endif /* __APPLE_RESUME_H__ */ diff --git a/drivers/gpu/drm/apple/apple_resume_test.c b/drivers/gpu/drm/apple/apple_resume_test.c new file mode 100644 index 00000000000000..7f89c07c152946 --- /dev/null +++ b/drivers/gpu/drm/apple/apple_resume_test.c @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Ryan Murray */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "apple_resume.h" + +/* One pipe: a primary plane scanning a framebuffer out to one connector. */ +struct resume_test { + struct drm_device drm; + struct drm_plane *plane; + struct drm_crtc *crtc; + struct drm_encoder encoder; + struct drm_connector connector; + struct drm_framebuffer fb; + /* Whether the mode fixup admits the mode, as a current link would. */ + bool admissible; +}; + +static struct resume_test *to_resume_test(struct drm_device *drm) +{ + return container_of(drm, struct resume_test, drm); +} + +static bool resume_test_mode_fixup(struct drm_crtc *crtc, + const struct drm_display_mode *mode, + struct drm_display_mode *adjusted_mode) +{ + return to_resume_test(crtc->dev)->admissible; +} + +static const struct drm_crtc_helper_funcs resume_test_crtc_helpers = { + .mode_fixup = resume_test_mode_fixup, +}; + +/* Scanout itself is not under test; the plane helpers require an update. */ +static void resume_test_plane_update(struct drm_plane *plane, + struct drm_atomic_state *state) +{ +} + +static const struct drm_plane_helper_funcs resume_test_plane_helpers = { + .atomic_update = resume_test_plane_update, +}; + +static const struct drm_connector_helper_funcs resume_test_connector_helpers = { +}; + +static const struct drm_connector_funcs resume_test_connector_funcs = { + .atomic_destroy_state = drm_atomic_helper_connector_destroy_state, + .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state, + .reset = drm_atomic_helper_connector_reset, +}; + +static void resume_test_fb_destroy(struct drm_framebuffer *fb) +{ + drm_framebuffer_cleanup(fb); +} + +static const struct drm_framebuffer_funcs resume_test_fb_funcs = { + .destroy = resume_test_fb_destroy, +}; + +static void resume_test_teardown(void *data) +{ + struct resume_test *priv = data; + + /* Release the scanout reference before the device goes away. */ + drm_atomic_helper_shutdown(&priv->drm); + drm_framebuffer_put(&priv->fb); +} + +static struct resume_test *resume_test_init(struct kunit *test) +{ + struct drm_modeset_acquire_ctx ctx; + struct drm_connector_state *conn_state; + struct drm_plane_state *plane_state; + struct drm_crtc_state *crtc_state; + struct drm_atomic_state *state; + struct drm_display_mode *mode; + struct resume_test *priv; + struct drm_device *drm; + struct device *dev; + int ret; + + dev = drm_kunit_helper_alloc_device(test); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, dev); + priv = drm_kunit_helper_alloc_drm_device(test, dev, struct resume_test, drm, + DRIVER_MODESET | DRIVER_ATOMIC); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, priv); + drm = &priv->drm; + priv->admissible = true; + + priv->plane = drm_kunit_helper_create_primary_plane(test, drm, NULL, + &resume_test_plane_helpers, + NULL, 0, NULL); + priv->crtc = drm_kunit_helper_create_crtc(test, drm, priv->plane, NULL, NULL, + &resume_test_crtc_helpers); + ret = drmm_encoder_init(drm, &priv->encoder, NULL, DRM_MODE_ENCODER_TMDS, NULL); + KUNIT_ASSERT_EQ(test, ret, 0); + priv->encoder.possible_crtcs = drm_crtc_mask(priv->crtc); + ret = drmm_connector_init(drm, &priv->connector, &resume_test_connector_funcs, + DRM_MODE_CONNECTOR_USB, NULL); + KUNIT_ASSERT_EQ(test, ret, 0); + drm_connector_helper_add(&priv->connector, &resume_test_connector_helpers); + ret = drm_connector_attach_encoder(&priv->connector, &priv->encoder); + KUNIT_ASSERT_EQ(test, ret, 0); + drm_mode_config_reset(drm); + + priv->fb.dev = drm; + priv->fb.format = drm_format_info(DRM_FORMAT_XRGB8888); + priv->fb.width = 1920; + priv->fb.height = 1080; + priv->fb.pitches[0] = 1920 * 4; + priv->fb.modifier = DRM_FORMAT_MOD_LINEAR; + ret = drm_framebuffer_init(drm, &priv->fb, &resume_test_fb_funcs); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = kunit_add_action_or_reset(test, resume_test_teardown, priv); + KUNIT_ASSERT_EQ(test, ret, 0); + + mode = drm_kunit_display_mode_from_cea_vic(test, drm, 16); + KUNIT_ASSERT_NOT_NULL(test, mode); + + /* Put by hand: a state kept to the end of the test pins old scanout. */ + drm_modeset_acquire_init(&ctx, 0); + state = drm_atomic_state_alloc(drm); + KUNIT_ASSERT_NOT_NULL(test, state); + state->acquire_ctx = &ctx; + conn_state = drm_atomic_get_connector_state(state, &priv->connector); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, conn_state); + KUNIT_ASSERT_EQ(test, drm_atomic_set_crtc_for_connector(conn_state, priv->crtc), 0); + crtc_state = drm_atomic_get_crtc_state(state, priv->crtc); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, crtc_state); + KUNIT_ASSERT_EQ(test, drm_atomic_set_mode_for_crtc(crtc_state, mode), 0); + crtc_state->active = true; + plane_state = drm_atomic_get_plane_state(state, priv->plane); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, plane_state); + KUNIT_ASSERT_EQ(test, drm_atomic_set_crtc_for_plane(plane_state, priv->crtc), 0); + drm_atomic_set_fb_for_plane(plane_state, &priv->fb); + plane_state->crtc_w = 1920; + plane_state->crtc_h = 1080; + plane_state->src_w = 1920 << 16; + plane_state->src_h = 1080 << 16; + ret = drm_atomic_commit(state); + drm_atomic_state_put(state); + drm_modeset_drop_locks(&ctx); + drm_modeset_acquire_fini(&ctx); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_TRUE(test, priv->crtc->state->active); + + return priv; +} + +/* The failure the stale CRTC filter exists for: one bad mode fails resume. */ +static void resume_stale_mode_fails_unfiltered_test(struct kunit *test) +{ + struct resume_test *priv = resume_test_init(test); + + KUNIT_ASSERT_EQ(test, drm_mode_config_helper_suspend(&priv->drm), 0); + priv->admissible = false; + KUNIT_EXPECT_EQ(test, drm_mode_config_helper_resume(&priv->drm), -EINVAL); + KUNIT_EXPECT_FALSE(test, priv->crtc->state->enable); +} + +static void resume_stale_crtc_disabled_test(struct kunit *test) +{ + struct resume_test *priv = resume_test_init(test); + struct drm_connector_state *conn_state; + struct drm_crtc_state *crtc_state; + struct drm_atomic_state *state; + + KUNIT_ASSERT_EQ(test, drm_mode_config_helper_suspend(&priv->drm), 0); + priv->admissible = false; + state = priv->drm.mode_config.suspend_state; + /* The caller tells userspace about the CRTC it had and lost. */ + KUNIT_EXPECT_TRUE(test, apple_drm_drop_stale_crtcs(state)); + + /* The resume check warns about an encoder routed without a CRTC. */ + conn_state = drm_atomic_get_new_connector_state(state, &priv->connector); + crtc_state = drm_atomic_get_new_crtc_state(state, priv->crtc); + KUNIT_ASSERT_NOT_NULL(test, conn_state); + KUNIT_ASSERT_NOT_NULL(test, crtc_state); + KUNIT_EXPECT_NULL(test, conn_state->best_encoder); + KUNIT_EXPECT_EQ(test, crtc_state->encoder_mask, 0U); + + KUNIT_EXPECT_EQ(test, drm_mode_config_helper_resume(&priv->drm), 0); + + KUNIT_EXPECT_FALSE(test, priv->crtc->state->enable); + KUNIT_EXPECT_FALSE(test, priv->crtc->state->active); + KUNIT_EXPECT_EQ(test, priv->crtc->state->plane_mask, 0U); + KUNIT_EXPECT_EQ(test, priv->crtc->state->connector_mask, 0U); + KUNIT_EXPECT_NULL(test, priv->plane->state->crtc); + KUNIT_EXPECT_NULL(test, priv->plane->state->fb); + KUNIT_EXPECT_NULL(test, priv->connector.state->crtc); + /* Userspace must modeset again rather than flip onto a dead CRTC. */ + KUNIT_EXPECT_EQ(test, priv->connector.state->link_status, + DRM_LINK_STATUS_BAD); + /* Only the creator's reference is left: the saved scanout was dropped. */ + KUNIT_EXPECT_EQ(test, drm_framebuffer_read_refcount(&priv->fb), 1U); +} + +static void resume_admissible_crtc_restored_test(struct kunit *test) +{ + struct resume_test *priv = resume_test_init(test); + + KUNIT_ASSERT_EQ(test, drm_mode_config_helper_suspend(&priv->drm), 0); + KUNIT_EXPECT_FALSE(test, + apple_drm_drop_stale_crtcs(priv->drm.mode_config.suspend_state)); + KUNIT_EXPECT_EQ(test, drm_mode_config_helper_resume(&priv->drm), 0); + + KUNIT_EXPECT_TRUE(test, priv->crtc->state->enable); + KUNIT_EXPECT_TRUE(test, priv->crtc->state->active); + KUNIT_EXPECT_PTR_EQ(test, priv->plane->state->crtc, priv->crtc); + KUNIT_EXPECT_PTR_EQ(test, priv->plane->state->fb, &priv->fb); + KUNIT_EXPECT_PTR_EQ(test, priv->connector.state->crtc, priv->crtc); + KUNIT_EXPECT_EQ(test, priv->connector.state->link_status, + DRM_LINK_STATUS_GOOD); + KUNIT_EXPECT_EQ(test, drm_framebuffer_read_refcount(&priv->fb), 2U); +} + +/* A DPMS-off CRTC keeps its mode, so it is checked like an active one. */ +static void resume_stale_dpms_off_crtc_disabled_test(struct kunit *test) +{ + struct resume_test *priv = resume_test_init(test); + struct drm_modeset_acquire_ctx ctx; + struct drm_crtc_state *crtc_state; + struct drm_atomic_state *state; + int ret; + + drm_modeset_acquire_init(&ctx, 0); + state = drm_atomic_state_alloc(&priv->drm); + KUNIT_ASSERT_NOT_NULL(test, state); + state->acquire_ctx = &ctx; + crtc_state = drm_atomic_get_crtc_state(state, priv->crtc); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, crtc_state); + crtc_state->active = false; + ret = drm_atomic_commit(state); + drm_atomic_state_put(state); + drm_modeset_drop_locks(&ctx); + drm_modeset_acquire_fini(&ctx); + KUNIT_ASSERT_EQ(test, ret, 0); + + KUNIT_ASSERT_EQ(test, drm_mode_config_helper_suspend(&priv->drm), 0); + priv->admissible = false; + KUNIT_EXPECT_TRUE(test, + apple_drm_drop_stale_crtcs(priv->drm.mode_config.suspend_state)); + KUNIT_EXPECT_EQ(test, drm_mode_config_helper_resume(&priv->drm), 0); + KUNIT_EXPECT_FALSE(test, priv->crtc->state->enable); + KUNIT_EXPECT_NULL(test, priv->connector.state->crtc); + KUNIT_EXPECT_EQ(test, priv->connector.state->link_status, + DRM_LINK_STATUS_BAD); + KUNIT_EXPECT_EQ(test, drm_framebuffer_read_refcount(&priv->fb), 1U); +} + +static struct kunit_case resume_cases[] = { + KUNIT_CASE(resume_stale_mode_fails_unfiltered_test), + KUNIT_CASE(resume_stale_crtc_disabled_test), + KUNIT_CASE(resume_admissible_crtc_restored_test), + KUNIT_CASE(resume_stale_dpms_off_crtc_disabled_test), + {} +}; + +static struct kunit_suite resume_suite = { + .name = "apple-drm-resume", + .test_cases = resume_cases, +}; + +kunit_test_suite(resume_suite); diff --git a/drivers/gpu/drm/apple/audio.c b/drivers/gpu/drm/apple/audio.c index 66a83ecea86fac..cfdfd606cf129e 100644 --- a/drivers/gpu/drm/apple/audio.c +++ b/drivers/gpu/drm/apple/audio.c @@ -27,12 +27,20 @@ #include "av.h" #include "dcp.h" +#include "dcp-internal.h" #include "audio.h" #include "parser.h" #define DCPAUD_ELEMENTS_MAXSIZE 16384 #define DCPAUD_PRODUCTATTRS_MAXSIZE 1024 +/* + * SIO takes one of its 64 descriptor slots per period, shared by all its + * channels, and up to three DP audio outputs use one SIO. A stream with + * more periods fails START with -ENOMEM. + */ +#define DCPAUD_PERIODS_MAX 16 + struct dcp_audio { struct device *dev; struct device *dcp_dev; @@ -43,6 +51,7 @@ struct dcp_audio { struct snd_jack *jack; struct snd_pcm_substream *substream; unsigned int open_cookie; + bool open_unplugged; struct mutex data_lock; bool dcp_connected; /// dcp status keep for delayed initialization @@ -51,6 +60,11 @@ struct dcp_audio { struct snd_pcm_chmap_elem selected_chmap; struct dcp_sound_cookie selected_cookie; + /* stream runs without a sink, see dcpaud_void_start() */ + bool void_running; + ktime_t void_start; + u64 void_periods; + struct delayed_work void_work; void *elements; void *productattrs; @@ -71,9 +85,42 @@ static const struct snd_pcm_hardware dcp_pcm_hw = { .period_bytes_min = 4096, /* TODO */ .period_bytes_max = SIZE_MAX, .periods_min = 2, - .periods_max = UINT_MAX, + .periods_max = DCPAUD_PERIODS_MAX, }; +/* + * Without a sink the DCP has no audio elements to derive constraints from. + * Offer a plain stereo format so that the PCM can still be opened and + * configured, as HDMI PCMs on other platforms can. Userspace (PipeWire, + * PulseAudio) probes the PCM once when the card appears and drops the + * output for good if that fails, so a monitor attached later would never + * show up. The stream then plays into the void, see dcpaud_void_start(). + */ +static const struct snd_pcm_hardware dcp_pcm_hw_unplugged = { + .info = SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID | + SNDRV_PCM_INFO_INTERLEAVED, + .formats = SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S24_LE | + SNDRV_PCM_FMTBIT_S32_LE, + .rates = SNDRV_PCM_RATE_32000 | SNDRV_PCM_RATE_44100 | + SNDRV_PCM_RATE_48000, + .rate_min = 32000, + .rate_max = 48000, + .channels_min = 2, + .channels_max = 2, + .buffer_bytes_max = SIZE_MAX, + .period_bytes_min = 4096, + .period_bytes_max = SIZE_MAX, + .periods_min = 2, + .periods_max = DCPAUD_PERIODS_MAX, +}; + +static bool dcpaud_h17p(struct dcp_audio *dcpaud) +{ + struct apple_dcp *dcp = dev_get_drvdata(dcpaud->dcp_dev); + + return dcp && dcp->fw_compat == DCP_FIRMWARE_H17P; +} + static int dcpaud_read_remote_info(struct dcp_audio *dcpaud) { int ret; @@ -251,16 +298,25 @@ static int dcp_pcm_open(struct snd_pcm_substream *substream) mutex_lock(&dcpaud->data_lock); ret = dcpaud_init_dma(dcpaud); - if (ret < 0) + if (ret < 0) { + mutex_unlock(&dcpaud->data_lock); return ret; + } - if (!dcpaud->connected) { + /* H17P (T8140): the PCM opens without a sink, as on aurora-wip. */ + dcpaud->open_unplugged = !dcpaud->connected && dcpaud_h17p(dcpaud); + if (!dcpaud->connected && !dcpaud->open_unplugged) { mutex_unlock(&dcpaud->data_lock); return -ENXIO; } dcpaud->open_cookie = dcpaud->connection_cookie; mutex_unlock(&dcpaud->data_lock); + if (dcpaud->open_unplugged) { + hw = dcp_pcm_hw_unplugged; + goto refine; + } + ret = dcpaud_read_remote_info(dcpaud); if (ret < 0) return ret; @@ -276,10 +332,11 @@ static int dcp_pcm_open(struct snd_pcm_substream *substream) SNDRV_PCM_HW_PARAM_FORMAT, SNDRV_PCM_HW_PARAM_CHANNELS, -1); hw = dcp_pcm_hw; +refine: hw.info = SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_MMAP_VALID | SNDRV_PCM_INFO_INTERLEAVED; hw.periods_min = 2; - hw.periods_max = UINT_MAX; + hw.periods_max = DCPAUD_PERIODS_MAX; hw.period_bytes_min = 256; hw.period_bytes_max = SIZE_MAX; // TODO dma_get_max_seg_size(dma_dev); hw.buffer_bytes_max = SIZE_MAX; @@ -297,6 +354,7 @@ static int dcp_pcm_close(struct snd_pcm_substream *substream) { struct dcp_audio *dcpaud = substream->pcm->private_data; dcpaud->selected_chmap.channels = 0; + dcpaud->open_unplugged = false; return snd_dmaengine_pcm_close(substream); } @@ -319,8 +377,9 @@ static int dcp_pcm_hw_params(struct snd_pcm_substream *substream, struct dma_chan *chan = snd_dmaengine_pcm_get_chan(substream); int ret; - if (!dcpaud_connection_up(dcpaud)) - return -ENXIO; + /* no sink: the stream runs without DMA, see dcpaud_void_start() */ + if (dcpaud->open_unplugged || !dcpaud_connection_up(dcpaud)) + return 0; ret = dcpaud_select_cookie(dcpaud, params); if (ret < 0) @@ -360,13 +419,80 @@ static int dcp_pcm_prepare(struct snd_pcm_substream *substream) { struct dcp_audio *dcpaud = substream->pcm->private_data; - if (!dcpaud_connection_up(dcpaud)) - return -ENXIO; + /* alsa-lib prepares right after hw_params; see dcp_pcm_hw_unplugged */ + if (dcpaud->open_unplugged || !dcpaud_connection_up(dcpaud)) + return 0; return dcp_audiosrv_prepare(dcpaud->dcp_dev, &dcpaud->selected_cookie); } +static u64 dcpaud_void_period_ns(struct snd_pcm_runtime *runtime) +{ + return div_u64((u64)runtime->period_size * NSEC_PER_SEC, runtime->rate); +} + +static void dcpaud_void_work(struct work_struct *work) +{ + struct dcp_audio *dcpaud = container_of(to_delayed_work(work), + struct dcp_audio, void_work); + struct snd_pcm_substream *substream = dcpaud->substream; + ktime_t next; + s64 delay; + + snd_pcm_period_elapsed(substream); + + snd_pcm_stream_lock_irq(substream); + if (dcpaud->void_running) { + /* the next period boundary, so that periods do not drift */ + next = ktime_add_ns(dcpaud->void_start, + ++dcpaud->void_periods * + dcpaud_void_period_ns(substream->runtime)); + delay = ktime_to_ns(ktime_sub(next, ktime_get())); + schedule_delayed_work(&dcpaud->void_work, + delay > 0 ? nsecs_to_jiffies(delay) : 0); + } + snd_pcm_stream_unlock_irq(substream); +} + +/* + * Without a sink, DCP has no audio link to start. Failing START does not + * stop userspace from trying: PipeWire retries it on every wakeup, and a + * stream that never starts never makes it wait, so it retries at about + * 130000 times a second for as long as something plays (a monitor + * without audio, or none attached). Play into the void instead, as HDMI + * PCMs elsewhere do: the stream runs on the clock, without DMA, and its + * samples are dropped. A sink attached later gets the next stream opened. + */ +static int dcpaud_void_start(struct snd_pcm_substream *substream) +{ + struct dcp_audio *dcpaud = substream->pcm->private_data; + u64 period_ns = dcpaud_void_period_ns(substream->runtime); + + dev_dbg(dcpaud->dev, "no sink, playing into the void\n"); + dcpaud->void_running = true; + dcpaud->void_start = ktime_get(); + dcpaud->void_periods = 0; + schedule_delayed_work(&dcpaud->void_work, nsecs_to_jiffies(period_ns)); + + return 0; +} + +static snd_pcm_uframes_t dcp_pcm_pointer(struct snd_pcm_substream *substream) +{ + struct dcp_audio *dcpaud = substream->pcm->private_data; + struct snd_pcm_runtime *runtime = substream->runtime; + u64 frames; + + if (!dcpaud->void_running) + return snd_dmaengine_pcm_pointer(substream); + + frames = mul_u64_u32_div(ktime_to_ns(ktime_sub(ktime_get(), + dcpaud->void_start)), + runtime->rate, NSEC_PER_SEC); + return do_div(frames, runtime->buffer_size); +} + static int dcp_pcm_trigger(struct snd_pcm_substream *substream, int cmd) { struct dcp_audio *dcpaud = substream->pcm->private_data; @@ -375,8 +501,8 @@ static int dcp_pcm_trigger(struct snd_pcm_substream *substream, int cmd) switch (cmd) { case SNDRV_PCM_TRIGGER_START: case SNDRV_PCM_TRIGGER_RESUME: - if (!dcpaud_connection_up(dcpaud)) - return -ENXIO; + if (dcpaud->open_unplugged || !dcpaud_connection_up(dcpaud)) + return dcpaud_void_start(substream); WARN_ON(pm_runtime_get_sync(dcpaud->dev) < 0); ret = dcp_audiosrv_startlink(dcpaud->dcp_dev, @@ -387,6 +513,11 @@ static int dcp_pcm_trigger(struct snd_pcm_substream *substream, int cmd) case SNDRV_PCM_TRIGGER_STOP: case SNDRV_PCM_TRIGGER_SUSPEND: + if (dcpaud->void_running) { + /* sync_stop waits for the work */ + dcpaud->void_running = false; + return 0; + } break; default: @@ -415,6 +546,19 @@ static int dcp_pcm_trigger(struct snd_pcm_substream *substream, int cmd) return 0; } +/* + * ALSA calls this once a stream has stopped, outside the stream lock, before + * the next hw_params, prepare or hw_free and at close. + */ +static int dcp_pcm_sync_stop(struct snd_pcm_substream *substream) +{ + struct dcp_audio *dcpaud = substream->pcm->private_data; + + cancel_delayed_work_sync(&dcpaud->void_work); + + return 0; +} + struct snd_pcm_ops dcp_playback_ops = { .open = dcp_pcm_open, .close = dcp_pcm_close, @@ -422,7 +566,8 @@ struct snd_pcm_ops dcp_playback_ops = { .hw_free = dcp_pcm_hw_free, .prepare = dcp_pcm_prepare, .trigger = dcp_pcm_trigger, - .pointer = snd_dmaengine_pcm_pointer, + .sync_stop = dcp_pcm_sync_stop, + .pointer = dcp_pcm_pointer, }; // Transitional workaround: for the chmap control TLV, advertise options @@ -504,7 +649,8 @@ static void dcpaud_report_hotplug(struct dcp_audio *dcpaud, bool connected) if (!connected) { snd_pcm_stream_lock(substream); - if (substream->runtime) + /* a stream without a sink plays on, see dcpaud_void_start() */ + if (substream->runtime && !dcpaud->void_running) snd_pcm_stop(substream, SNDRV_PCM_STATE_DISCONNECTED); snd_pcm_stream_unlock(substream); } @@ -716,6 +862,7 @@ static int dcpaud_probe(struct platform_device *pdev) dcpaud->dev = &pdev->dev; mutex_init(&dcpaud->data_lock); + INIT_DELAYED_WORK(&dcpaud->void_work, dcpaud_void_work); platform_set_drvdata(pdev, dcpaud); return component_add(&pdev->dev, &dcpaud_comp_ops); diff --git a/drivers/gpu/drm/apple/av.c b/drivers/gpu/drm/apple/av.c index 563a02bb4e5519..f79fbd8e32614d 100644 --- a/drivers/gpu/drm/apple/av.c +++ b/drivers/gpu/drm/apple/av.c @@ -96,12 +96,21 @@ static void av_audiosrv_init(struct apple_epic_service *service, const char *nam { struct apple_dcp *dcp = service->ep->dcp; struct audiosrv_data *asrv = dcp->audiosrv; + struct apple_epic_service *old; + + /* The owner pin keeps the slot from being recycled while published. */ + service = afk_service_get(service); + if (!service) + return; mutex_lock(&asrv->plug_lock); down_write(&asrv->srv_rwsem); + old = asrv->srv; asrv->srv = service; up_write(&asrv->srv_rwsem); + if (old) + afk_service_put(old); asrv->plugged = true; mutex_unlock(&asrv->plug_lock); @@ -112,18 +121,27 @@ static void av_audiosrv_teardown(struct apple_epic_service *service) { struct apple_dcp *dcp = service->ep->dcp; struct audiosrv_data *asrv = dcp->audiosrv; + bool owned; mutex_lock(&asrv->plug_lock); down_write(&asrv->srv_rwsem); - asrv->srv = NULL; + owned = asrv->srv == service; + if (owned) + asrv->srv = NULL; up_write(&asrv->srv_rwsem); - asrv->plugged = false; - if (asrv->audio_dev) - dcpaud_disconnect(asrv->audio_dev); + /* A stale generation must not unplug its successor. */ + if (owned) { + asrv->plugged = false; + if (asrv->audio_dev) + dcpaud_disconnect(asrv->audio_dev); + } mutex_unlock(&asrv->plug_lock); + + if (owned) + afk_service_put(service); } int dcp_audiosrv_prepare(struct device *dev, struct dcp_sound_cookie *cookie) @@ -279,11 +297,13 @@ static int av_audiosrv_report(struct apple_epic_service *service, u32 idx, static const struct apple_epic_service_ops avep_ops[] = { { .name = "DCPAVSimpleVideoInterface", + .reusable = true, .init = av_interface_init, .teardown = av_interface_teardown, }, { .name = "DCPAVAudioInterface", + .reusable = true, .init = av_audiosrv_init, .report = av_audiosrv_report, .teardown = av_audiosrv_teardown, @@ -298,21 +318,21 @@ void av_service_connect(struct apple_dcp *dcp) int ret; scoped_guard(rwsem_write, &asrv->srv_rwsem) { - if (!asrv->srv) + service = afk_service_pin(asrv->srv); + if (!service) return; - service = asrv->srv; } /* open AV audio service */ dev_info(dcp->dev, "%s: starting audio service, plugged:%d\n", __func__, asrv->plugged); if (asrv->is_open) - return; + goto out; ret = afk_service_call(service, 0, asrv->cmds.open, NULL, 0, 32, NULL, 0, 32); if (ret) { dev_err(dcp->dev, "error opening audio service: %d\n", ret); - return; + goto out; } mutex_lock(&asrv->plug_lock); asrv->is_open = true; @@ -320,6 +340,8 @@ void av_service_connect(struct apple_dcp *dcp) if (asrv->audio_dev) dcpaud_connect(asrv->audio_dev, asrv->plugged); mutex_unlock(&asrv->plug_lock); +out: + afk_service_put(service); } void av_service_disconnect(struct apple_dcp *dcp) @@ -329,15 +351,15 @@ void av_service_disconnect(struct apple_dcp *dcp) int ret; scoped_guard(rwsem_write, &asrv->srv_rwsem) { - if (!asrv->srv) + service = afk_service_pin(asrv->srv); + if (!service) return; - service = asrv->srv; } /* close AV audio service */ dev_info(dcp->dev, "%s: stopping audio service\n", __func__); if (!asrv->is_open) - return; + goto out; mutex_lock(&asrv->plug_lock); @@ -354,6 +376,8 @@ void av_service_disconnect(struct apple_dcp *dcp) if (service->torndown) afk_service_disable(service); asrv->is_open = false; +out: + afk_service_put(service); } static void av_work_service_start(struct work_struct *work) diff --git a/drivers/gpu/drm/apple/connector.h b/drivers/gpu/drm/apple/connector.h index fe4a60d7155997..01a113a5cfa648 100644 --- a/drivers/gpu/drm/apple/connector.h +++ b/drivers/gpu/drm/apple/connector.h @@ -55,6 +55,10 @@ struct apple_connector { struct mutex edid_lock; u64 edid_generation; bool edid_live; + bool drm_edid_firmware_derived; + bool edid_invalidation_pending; + u64 drm_edid_generation; + u64 drm_edid_service_epoch; /* Workqueue for sending hotplug events to the associated device */ struct work_struct hotplug_wq; diff --git a/drivers/gpu/drm/apple/dcp-180hz-policy.h b/drivers/gpu/drm/apple/dcp-180hz-policy.h new file mode 100644 index 00000000000000..d23aa57c685e28 --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-180hz-policy.h @@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DCP_180HZ_POLICY_H +#define APPLE_DCP_180HZ_POLICY_H + +#include +#include +#include + +/* Only this verified uncompressed configuration is admitted. */ +static inline bool dcp_180hz_payload(u32 rate, u32 lanes, u32 clock) +{ + return rate == 0x1e && lanes == 2 && clock == 414090 && + (u64)clock * 24 <= (u64)8100000 * lanes * 8 / 10; +} + +static inline bool dcp_180hz_dtd(const u8 *d, const struct drm_display_mode *m) +{ + int ha = d[2] | (d[4] & 0xf0) << 4; + int hb = d[3] | (d[4] & 0x0f) << 8; + int va = d[5] | (d[7] & 0xf0) << 4; + int vb = d[6] | (d[7] & 0x0f) << 8; + int hf = d[8] | (d[11] & 0xc0) << 2; + int hs = d[9] | (d[11] & 0x30) << 4; + int vf = (d[10] >> 4) | (d[11] & 0x0c) << 2; + int vs = (d[10] & 0x0f) | (d[11] & 0x03) << 4; + u32 flags = (d[17] & 2 ? DRM_MODE_FLAG_PHSYNC : DRM_MODE_FLAG_NHSYNC) | + (d[17] & 4 ? DRM_MODE_FLAG_PVSYNC : DRM_MODE_FLAG_NVSYNC); + + return ((d[0] | d[1] << 8) * 10 == m->clock) && m->clock == 414090 && + (d[17] & 0x98) == 0x18 && ha == 1920 && va == 1080 && + ha + hb == 2080 && va + vb == 1106 && + m->hdisplay == ha && m->htotal == ha + hb && + m->hsync_start == ha + hf && m->hsync_end == ha + hf + hs && + m->vdisplay == va && m->vtotal == va + vb && + m->vsync_start == va + vf && m->vsync_end == va + vf + vs && + m->flags == flags; +} + +/* Bounded actual EDID only; no serial field is consulted. */ +static inline bool dcp_180hz_edid(const u8 *p, size_t len, + const struct drm_display_mode *mode) +{ + static const u8 header[] = { 0, 255, 255, 255, 255, 255, 255, 0 }; + bool name = false, timing = false, forum = false; + unsigned int names = 0, i, off, end; + + if (!p || !mode || len != 256 || memcmp(p, header, 8) || p[126] != 1 || + p[8] != 0x04 || p[9] != 0x72 || p[10] != 0x4f || p[11] != 0x0b || + p[18] != 1 || p[19] < 3 || p[19] > 4 || !(p[20] & 0x80)) + return false; + for (off = 0; off < len; off += 128) { + u8 sum = 0; + + for (i = 0; i < 128; i++) + sum += p[off + i]; + if (sum) + return false; + } + for (off = 54; off + 18 <= 126; off += 18) { + const u8 *d = p + off; + + if (d[0] || d[1]) { + timing |= dcp_180hz_dtd(d, mode); + } else if (!d[2] && d[3] == 0xfc && !d[4]) { + unsigned int n = 13; + + names++; + while (n && (d[5 + n - 1] == ' ' || d[5 + n - 1] == '\n' || + d[5 + n - 1] == 0)) + n--; + name = (n == 8 && !memcmp(d + 5, "QG270 S3", 8)) || + (n == 7 && !memcmp(d + 5, "QG270S3", 7)); + } + } + p += 128; + if (p[0] != 2 || p[1] < 3 || p[2] < 4 || p[2] > 127) + return false; + end = p[2]; + for (off = 4; off < end;) { + const u8 *db = p + off; + unsigned int size = (db[0] & 31) + 1; + + if (size > end - off) + return false; + if ((db[0] >> 5) == 3 && size >= 4 && + db[1] == 0xd8 && db[2] == 0x5d && db[3] == 0xc4) { + /* HF-VSDB overrides the older HDMI VSDB clock, as drm_edid does. */ + if (forum || size < 8 || db[4] != 1 || !(db[6] & 0x80) || + (u32)db[5] * 5000 < (u32)mode->clock) + return false; + forum = true; + } + off += size; + } + for (off = end; off + 18 <= 127; off += 18) + if (p[off] || p[off + 1]) + timing |= dcp_180hz_dtd(p + off, mode); + return name && names == 1 && timing && forum; +} + +#endif diff --git a/drivers/gpu/drm/apple/dcp-180hz.c b/drivers/gpu/drm/apple/dcp-180hz.c new file mode 100644 index 00000000000000..8eb4166aa6dbb6 --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-180hz.c @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +#include +#include +#include +#include "connector.h" +#include "afk.h" +#include "dcp.h" +#include "parser.h" +#include "dcp-180hz.h" +#include "dcp-180hz-policy.h" + +static bool dcp_180hz_direct_port(struct apple_epic_service *service) +{ + struct apple_dcp *dcp; + struct dptx_port *port; + + if (!service || !service->ep || !service->cookie) + return false; + dcp = service->ep->dcp; + port = service->cookie; + return dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && + service->ep->endpoint == DPTX_ENDPOINT_V2 && service->ep->wire_v2 && + port == &dcp->dptxport[0] && READ_ONCE(port->enabled) && + !READ_ONCE(port->closed) && READ_ONCE(port->service) == service && + READ_ONCE(service->enabled) && port->atcphy && port->atcphy == dcp->phy && + dcp->active_typec_route && !dcp->dptx_tunnel && !dcp_is_usb4_output(dcp); +} + +void dcp_180hz_invalidate(struct apple_dcp *dcp) +{ + unsigned long flags; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + dcp->rate_proof.sequence++; + dcp->rate_proof.rate.valid = false; + dcp->rate_proof.lanes.valid = false; + dcp->rate_proof.retries = 0; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); +} + +struct dcp_180hz_token dcp_180hz_config_begin(struct apple_epic_service *service, bool lanes) +{ + struct apple_dcp *dcp = service->ep->dcp; + struct dcp_180hz_token token = {}; + unsigned long flags; + + /* RX callbacks never acquire lifecycle/HPD/AV mutexes. */ + spin_lock_irqsave(&dcp->metadata.lock, flags); + token.generation = READ_ONCE(dcp->typec_generation); + token.sequence = dcp->rate_proof.sequence; + token.connection_sequence = dcp->rate_proof.connection_sequence; + token.phy = dcp->phy; + token.route = dcp->active_typec_route; + token.service = service; + token.valid = !READ_ONCE(dcp->quiescing) && dcp_180hz_direct_port(service) && + READ_ONCE(dcp->typec_cable_connected) && + token.generation == dcp->metadata.epoch.generation; + if (lanes) + dcp->rate_proof.lanes.valid = false; + else + dcp->rate_proof.rate.valid = false; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + return token; +} + +/* Preserve the receipt-start registry ticket even when it awaited OPEN. */ +static bool dcp_180hz_timing_epoch(const struct dcp_display_mode *mode, u64 epoch) +{ + if (mode->automatic_service_pending) + return mode->unsafe_rgb8_service_epoch != U64_MAX && + epoch == mode->unsafe_rgb8_service_epoch + 1; + return epoch && mode->unsafe_rgb8_service_epoch == epoch; +} + +void dcp_180hz_config_complete(struct apple_epic_service *service, bool lanes, + struct dcp_180hz_token token, u32 value, bool success) +{ + struct apple_dcp *dcp = service->ep->dcp; + struct dcp_180hz_fact *fact; + bool notify = false; + unsigned long flags; + + if (!success) { + dcp_180hz_invalidate(dcp); + return; + } + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (token.valid && token.service == service && + token.sequence == dcp->rate_proof.sequence && + token.generation == READ_ONCE(dcp->typec_generation) && + token.connection_sequence == dcp->rate_proof.connection_sequence && + READ_ONCE(dcp->typec_cable_connected) && + token.phy == dcp->phy && token.route == dcp->active_typec_route && + dcp_180hz_direct_port(service) && !READ_ONCE(dcp->quiescing)) { + fact = lanes ? &dcp->rate_proof.lanes : &dcp->rate_proof.rate; + *fact = (struct dcp_180hz_fact) { .token = token, .value = value, .valid = true }; + notify = dcp->rate_proof.rate.valid && dcp->rate_proof.lanes.valid; + dcp->rate_proof.retries = 0; + } + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (notify) + dcp_180hz_notify(dcp); +} + +static bool dcp_180hz_fact_current(struct apple_dcp *dcp, const struct dcp_180hz_fact *fact) +{ + const struct dcp_180hz_token *t = &fact->token; + + return fact->valid && t->generation == dcp->typec_generation && + t->sequence == dcp->rate_proof.sequence && + t->connection_sequence == dcp->rate_proof.connection_sequence && t->phy == dcp->phy && + t->route == dcp->active_typec_route && t->service == dcp->dptxport[0].service; +} + +/* Catalog withdrawal needs a DRM refresh even when no exception is ready. */ +void dcp_180hz_catalog_changed(struct apple_dcp *dcp) +{ + struct apple_connector *con = READ_ONCE(dcp->connector); + + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17P_EXTERNAL || + !con || READ_ONCE(dcp->quiescing) || !READ_ONCE(con->connected) || + READ_ONCE(con->dcp) != to_platform_device(dcp->dev) || + READ_ONCE(con->base.registration_state) != DRM_CONNECTOR_REGISTERED) + return; + schedule_work(&con->hotplug_wq); +} + +/* Timing and configuration can arrive in either order, before AV admission. */ +void dcp_180hz_notify(struct apple_dcp *dcp) +{ + struct apple_connector *con = READ_ONCE(dcp->connector); + unsigned long flags; + bool ready; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + ready = dcp->metadata.epoch.connected && READ_ONCE(dcp->typec_cable_connected) && + !READ_ONCE(dcp->quiescing) && READ_ONCE(dcp->nr_modes) && + dcp_180hz_direct_port(dcp->dptxport[0].service) && + dcp_180hz_fact_current(dcp, &dcp->rate_proof.rate) && + dcp_180hz_fact_current(dcp, &dcp->rate_proof.lanes) && + dcp_180hz_payload(dcp->rate_proof.rate.value, dcp->rate_proof.lanes.value, 414090); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (ready && con && READ_ONCE(con->connected) && READ_ONCE(dcp->dptxport[0].connected) && + READ_ONCE(con->dcp) == to_platform_device(dcp->dev) && + READ_ONCE(con->base.registration_state) == DRM_CONNECTOR_REGISTERED) + schedule_work(&con->hotplug_wq); +} + +bool dcp_180hz_mode_current(struct apple_dcp *dcp, const struct dcp_display_mode *mode) +{ + struct apple_connector *con = dcp->connector; + const struct edid *raw; + + lockdep_assert_held(&dcp->hpd_mutex); + lockdep_assert_held(&dcp->dcpavserv.lock); + lockdep_assert_held(&dcp->metadata.lock); + if (!con || dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17P_EXTERNAL || + !dcp_180hz_direct_port(dcp->dptxport[0].service) || + !mode->unsafe_rgb8_ticket.associated || + mode->unsafe_rgb8_ticket.generation != dcp->typec_generation || + mode->automatic_connection_sequence != dcp->rate_proof.connection_sequence || + mode->automatic_catalog_sequence != dcp->rate_proof.catalog_sequence || + !dcp->metadata.epoch.connected || !dcp->typec_cable_connected || dcp->quiescing || + READ_ONCE(con->dcp) != to_platform_device(dcp->dev) || + !dcpavserv_edid_service_current(dcp, con->drm_edid_service_epoch) || + !dcp_180hz_timing_epoch(mode, dcp->dcpavserv.epoch) || + !con->drm_edid || con->drm_edid_firmware_derived || + READ_ONCE(con->edid_invalidation_pending) || + con->drm_edid_generation != dcp->typec_generation || + !dcp->dptxport[0].connected || !dcp->dptxport[0].enabled || dcp->dptxport[0].closed || + !dcp_180hz_fact_current(dcp, &dcp->rate_proof.rate) || + !dcp_180hz_fact_current(dcp, &dcp->rate_proof.lanes) || + !dcp_180hz_payload(dcp->rate_proof.rate.value, dcp->rate_proof.lanes.value, mode->mode.clock) || + mode->sdr_rgb.depth != 8 || mode->sdr_rgb.format != DCP_COLOR_FORMAT_RGB || + mode->sdr_rgb.range != DCP_COLOR_YCBCR_RANGE_FULL || + mode->sdr_rgb.eotf != DCP_EOTF_SDR_GAMMA || mode->sdr_rgb.colorimetry != DCP_COLORIMETRY_SRGB) + return false; + raw = drm_edid_raw(con->drm_edid); + /* The transport validates extent/checksums; this quirk accepts only its 256-byte form. */ + return raw && raw->extensions == 1 && + dcp_180hz_edid((const u8 *)raw, 256, &mode->mode); +} + +bool dcp_180hz_pending(struct apple_dcp *dcp, const struct dcp_display_mode *mode) +{ + struct apple_connector *con = dcp->connector; + const struct edid *raw; + + if (!con || !dcp->active_typec_route || dcp_is_usb4_output(dcp) || dcp->dptx_tunnel || + !mode->unsafe_rgb8_ticket.associated || + mode->unsafe_rgb8_ticket.generation != dcp->typec_generation || + mode->automatic_connection_sequence != dcp->rate_proof.connection_sequence || + mode->automatic_catalog_sequence != dcp->rate_proof.catalog_sequence || + (!dcp_180hz_timing_epoch(mode, dcp->dcpavserv.epoch) && + !(mode->automatic_service_pending && !dcp->dcpavserv.enabled && + mode->unsafe_rgb8_service_epoch == dcp->dcpavserv.epoch))) + return false; + if (!con->drm_edid) + return true; + raw = drm_edid_raw(con->drm_edid); + return raw && raw->extensions == 1 && dcp_180hz_edid((const u8 *)raw, 256, &mode->mode); +} + +static void dcp_180hz_retry_work(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(to_delayed_work(work), struct apple_dcp, rate_retry_wq); + unsigned long flags; + bool current_generation; + + guard(mutex)(&dcp->hpd_mutex); + spin_lock_irqsave(&dcp->metadata.lock, flags); + current_generation = dcp->typec_cable_connected && !dcp->quiescing && + dcp->rate_proof.retry_generation == dcp->typec_generation; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (current_generation && dcp->connector && + READ_ONCE(dcp->connector->dcp) == to_platform_device(dcp->dev)) + schedule_work(&dcp->connector->hotplug_wq); +} + +void dcp_180hz_retry(struct apple_dcp *dcp) +{ + unsigned long flags; + bool retry = false; + + if (!dcp->rate_retry_initialized || READ_ONCE(dcp->quiescing)) + return; + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (dcp->rate_proof.retry_generation != READ_ONCE(dcp->typec_generation)) { + dcp->rate_proof.retry_generation = READ_ONCE(dcp->typec_generation); + dcp->rate_proof.retries = 0; + } + if (READ_ONCE(dcp->typec_cable_connected) && dcp->rate_proof.retries < 8) { + dcp->rate_proof.retries++; + retry = true; + } + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (retry) + mod_delayed_work(system_freezable_wq, &dcp->rate_retry_wq, msecs_to_jiffies(300)); +} + +static void dcp_180hz_cancel(void *data) +{ + struct apple_dcp *dcp = data; + + WRITE_ONCE(dcp->rate_retry_initialized, false); + cancel_delayed_work_sync(&dcp->rate_retry_wq); +} + +int dcp_180hz_init(struct apple_dcp *dcp) +{ + int ret; + + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17P_EXTERNAL) + return 0; + INIT_DELAYED_WORK(&dcp->rate_retry_wq, dcp_180hz_retry_work); + ret = devm_add_action_or_reset(dcp->dev, dcp_180hz_cancel, dcp); + if (!ret) + dcp->rate_retry_initialized = true; + return ret; +} diff --git a/drivers/gpu/drm/apple/dcp-180hz.h b/drivers/gpu/drm/apple/dcp-180hz.h new file mode 100644 index 00000000000000..4995dae4cf6489 --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-180hz.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DCP_180HZ_H +#define APPLE_DCP_180HZ_H +#include +struct apple_dcp; +struct apple_epic_service; +struct dcp_display_mode; +struct dcp_180hz_token { + /* Successful physical configuration is independent of AV discovery. */ + u64 generation, sequence, connection_sequence; + const void *phy, *route, *service; + bool valid; +}; +struct dcp_180hz_fact { + struct dcp_180hz_token token; + u32 value; + bool valid; +}; +struct dcp_180hz_proof { + u64 sequence, connection_sequence, catalog_sequence, retry_generation; + unsigned int retries; + struct dcp_180hz_fact rate, lanes; +}; +struct dcp_180hz_token dcp_180hz_config_begin(struct apple_epic_service *service, bool lanes); +void dcp_180hz_config_complete(struct apple_epic_service *service, bool lanes, + struct dcp_180hz_token token, u32 value, bool success); +void dcp_180hz_invalidate(struct apple_dcp *dcp); +void dcp_180hz_notify(struct apple_dcp *dcp); +void dcp_180hz_catalog_changed(struct apple_dcp *dcp); +bool dcp_180hz_mode_current(struct apple_dcp *dcp, const struct dcp_display_mode *mode); +bool dcp_180hz_pending(struct apple_dcp *dcp, const struct dcp_display_mode *mode); +void dcp_180hz_retry(struct apple_dcp *dcp); +int dcp_180hz_init(struct apple_dcp *dcp); +#endif diff --git a/drivers/gpu/drm/apple/dcp-display-metadata.c b/drivers/gpu/drm/apple/dcp-display-metadata.c new file mode 100644 index 00000000000000..cc4c69736d0986 --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-display-metadata.c @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +#include +#include +#include +#include +#include + +#include "dcp-display-metadata.h" + +static int dcp_metadata_timing(struct detailed_timing *timing, + const struct drm_display_mode *mode, + const struct dcp_display_metadata *metadata) +{ + struct detailed_pixel_timing *pixel = &timing->data.pixel_data; + u32 hblank, vblank, hoff, voff, hsync, vsync; + u32 width = metadata->width_cm * 10, height = metadata->height_cm * 10; + u32 hflags = mode->flags & (DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_NHSYNC); + u32 vflags = mode->flags & (DRM_MODE_FLAG_PVSYNC | DRM_MODE_FLAG_NVSYNC); + + if (mode->clock <= 0 || mode->clock > 655350 || mode->clock % 10 || + mode->hdisplay <= 0 || mode->vdisplay <= 0 || + mode->hdisplay > 4095 || mode->vdisplay > 4095 || + mode->htotal <= mode->hdisplay || mode->vtotal <= mode->vdisplay || + mode->hsync_start < mode->hdisplay || mode->hsync_end <= mode->hsync_start || + mode->hsync_end > mode->htotal || mode->vsync_start < mode->vdisplay || + mode->vsync_end <= mode->vsync_start || mode->vsync_end > mode->vtotal || + (hflags != DRM_MODE_FLAG_PHSYNC && hflags != DRM_MODE_FLAG_NHSYNC) || + (vflags != DRM_MODE_FLAG_PVSYNC && vflags != DRM_MODE_FLAG_NVSYNC) || + (mode->flags & ~(DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_NHSYNC | + DRM_MODE_FLAG_PVSYNC | DRM_MODE_FLAG_NVSYNC))) + return -EINVAL; + hblank = mode->htotal - mode->hdisplay; + vblank = mode->vtotal - mode->vdisplay; + hoff = mode->hsync_start - mode->hdisplay; + voff = mode->vsync_start - mode->vdisplay; + hsync = mode->hsync_end - mode->hsync_start; + vsync = mode->vsync_end - mode->vsync_start; + if (hblank > 4095 || vblank > 4095 || hoff > 1023 || hsync > 1023 || + voff > 63 || vsync > 63 || width > 4095 || height > 4095) + return -EINVAL; + + timing->pixel_clock = cpu_to_le16(mode->clock / 10); + pixel->hactive_lo = mode->hdisplay; + pixel->hblank_lo = hblank; + pixel->hactive_hblank_hi = (mode->hdisplay >> 8) << 4 | hblank >> 8; + pixel->vactive_lo = mode->vdisplay; + pixel->vblank_lo = vblank; + pixel->vactive_vblank_hi = (mode->vdisplay >> 8) << 4 | vblank >> 8; + pixel->hsync_offset_lo = hoff; + pixel->hsync_pulse_width_lo = hsync; + pixel->vsync_offset_pulse_width_lo = (voff & 15) << 4 | (vsync & 15); + pixel->hsync_vsync_offset_pulse_width_hi = (hoff >> 8) << 6 | (hsync >> 8) << 4 | + (voff >> 4) << 2 | vsync >> 4; + pixel->width_mm_lo = width; + pixel->height_mm_lo = height; + pixel->width_height_mm_hi = (width >> 8) << 4 | height >> 8; + pixel->misc = DRM_EDID_PT_SEPARATE_SYNC; + if (hflags == DRM_MODE_FLAG_PHSYNC) + pixel->misc |= DRM_EDID_PT_HSYNC_POSITIVE; + if (vflags == DRM_MODE_FLAG_PVSYNC) + pixel->misc |= DRM_EDID_PT_VSYNC_POSITIVE; + return 0; +} + +static int dcp_metadata_text(struct detailed_timing *timing, u8 type, const char *text) +{ + size_t length = strnlen(text, 14); + u8 *out = timing->data.other_data.data.str.str; + + if (!length || length > 13) + return -EINVAL; + for (size_t i = 0; i < length; i++) + if (text[i] < 0x20 || text[i] > 0x7e) + return -EINVAL; + timing->data.other_data.type = type; + memset(out, ' ', 13); + memcpy(out, text, length); + if (length < 13) + out[length] = '\n'; + return 0; +} + +static s64 dcp_metadata_side(const u32 *a, const u32 *b, const u32 *point) +{ + return ((s64)b[0] - a[0]) * ((s64)point[1] - a[1]) - + ((s64)b[1] - a[1]) * ((s64)point[0] - a[0]); +} + +/* Build only a base block: the authoritative firmware mode list stays separate. */ +static int dcp_metadata_block(struct edid *edid, const struct dcp_display_metadata *metadata, + const struct drm_display_mode *mode) +{ + u32 coordinate[8], manufacturer = 0, gamma; + u8 checksum = 0; + int ret; + + if (!metadata->width_cm || metadata->width_cm > 255 || + !metadata->height_cm || metadata->height_cm > 255 || + metadata->product > U16_MAX || metadata->week > 53 || + metadata->year < 1990 || metadata->year > 2245 || + metadata->gamma < 65536 || metadata->gamma > (354U << 16) / 100 || + strnlen(metadata->manufacturer, 4) != 3) + return -EINVAL; + for (unsigned int i = 0; i < 3; i++) { + if (metadata->manufacturer[i] < 'A' || metadata->manufacturer[i] > 'Z') + return -EINVAL; + manufacturer = manufacturer << 5 | (metadata->manufacturer[i] - '@'); + } + for (unsigned int i = 0; i < 8; i++) { + u32 value = i < 6 ? metadata->chromaticity[i / 2][i % 2] : metadata->white[i - 6]; + + if (!value || value >= 65536) + return -EINVAL; + coordinate[i] = DIV_ROUND_CLOSEST(value, 64); + if (!coordinate[i] || coordinate[i] > 1023) + return -EINVAL; + if ((i & 1) && coordinate[i - 1] + coordinate[i] > 1024) + return -EINVAL; + } + /* Degenerate primaries or a white point outside their triangle are not usable. */ + { + s64 area = dcp_metadata_side(coordinate, coordinate + 2, coordinate + 4); + + if (!area) + return -EINVAL; + for (unsigned int i = 0; i < 3; i++) { + s64 side = dcp_metadata_side(coordinate + 2 * i, + coordinate + 2 * ((i + 1) % 3), coordinate + 6); + + if ((area > 0 && side < 0) || (area < 0 && side > 0)) + return -EINVAL; + } + } + gamma = DIV_ROUND_CLOSEST(metadata->gamma * 100, 65536) - 100; + if (gamma > 254) + return -EINVAL; + + memset(edid, 0, sizeof(*edid)); + memcpy(edid->header, "\x00\xff\xff\xff\xff\xff\xff\x00", 8); + edid->product_id.manufacturer_name = cpu_to_be16(manufacturer); + edid->product_id.product_code = cpu_to_le16(metadata->product); + edid->product_id.serial_number = cpu_to_le32(metadata->serial); + edid->product_id.week_of_manufacture = metadata->week; + edid->product_id.year_of_manufacture = metadata->year - 1990; + edid->version = 1; + edid->revision = 4; + /* No claim about connector interface, bit depth, HDR, audio or VRR. */ + edid->input = DRM_EDID_INPUT_DIGITAL; + edid->width_cm = metadata->width_cm; + edid->height_cm = metadata->height_cm; + edid->gamma = gamma; + edid->features = DRM_EDID_FEATURE_PREFERRED_TIMING | DRM_EDID_FEATURE_RGB; + edid->red_green_lo = (coordinate[0] & 3) << 6 | (coordinate[1] & 3) << 4 | + (coordinate[2] & 3) << 2 | (coordinate[3] & 3); + edid->blue_white_lo = (coordinate[4] & 3) << 6 | (coordinate[5] & 3) << 4 | + (coordinate[6] & 3) << 2 | (coordinate[7] & 3); + edid->red_x = coordinate[0] >> 2; + edid->red_y = coordinate[1] >> 2; + edid->green_x = coordinate[2] >> 2; + edid->green_y = coordinate[3] >> 2; + edid->blue_x = coordinate[4] >> 2; + edid->blue_y = coordinate[5] >> 2; + edid->white_x = coordinate[6] >> 2; + edid->white_y = coordinate[7] >> 2; + memset(edid->standard_timings, 1, sizeof(edid->standard_timings)); + ret = dcp_metadata_timing(&edid->detailed_timings[0], mode, metadata); + if (!ret) + ret = dcp_metadata_text(&edid->detailed_timings[1], EDID_DETAIL_MONITOR_NAME, + metadata->name); + if (!ret) + ret = dcp_metadata_text(&edid->detailed_timings[2], EDID_DETAIL_MONITOR_STRING, + "DCP metadata"); + if (!ret) + ret = dcp_metadata_text(&edid->detailed_timings[3], + metadata->serial_text[0] ? EDID_DETAIL_MONITOR_SERIAL : + EDID_DETAIL_MONITOR_STRING, + metadata->serial_text[0] ? + metadata->serial_text : "FW derived"); + if (ret) + return ret; + for (unsigned int i = 0; i < sizeof(*edid) - 1; i++) + checksum += ((const u8 *)edid)[i]; + edid->checksum = -checksum; + return 0; +} + +const struct drm_edid *dcp_metadata_edid(const struct dcp_display_metadata *metadata, + const struct drm_display_mode *mode) +{ + struct edid block; + const struct drm_edid *edid; + int ret = dcp_metadata_block(&block, metadata, mode); + + if (ret) + return ERR_PTR(ret); + edid = drm_edid_alloc(&block, sizeof(block)); + if (!edid) + return ERR_PTR(-ENOMEM); + if (!drm_edid_valid(edid)) { + drm_edid_free(edid); + return ERR_PTR(-EINVAL); + } + return edid; +} diff --git a/drivers/gpu/drm/apple/dcp-display-metadata.h b/drivers/gpu/drm/apple/dcp-display-metadata.h new file mode 100644 index 00000000000000..537994420ebd6e --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-display-metadata.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DCP_DISPLAY_METADATA_H +#define APPLE_DCP_DISPLAY_METADATA_H + +#include + +/* Values reported by DisplayAttributes, not a copy of the sink's EDID. */ +struct dcp_display_metadata { + u32 width_cm, height_cm; + u32 chromaticity[3][2]; + u32 white[2]; + u32 gamma; + u32 product, serial, week, year; + char manufacturer[4]; + char name[14]; + char serial_text[14]; +}; + +struct drm_edid; +struct drm_display_mode; +const struct drm_edid *dcp_metadata_edid(const struct dcp_display_metadata *metadata, + const struct drm_display_mode *mode); + +#endif diff --git a/drivers/gpu/drm/apple/dcp-external-bw.h b/drivers/gpu/drm/apple/dcp-external-bw.h new file mode 100644 index 00000000000000..99430d3e4d883f --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-external-bw.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* Copyright 2026 Aurora Silicon */ +#ifndef APPLE_DCP_EXTERNAL_BW_H +#define APPLE_DCP_EXTERNAL_BW_H + +#include + +#define DCP_EXTERNAL_FW_CELLS 3 + +/* The external H17P wire contract is supported only for this firmware release. */ +static inline bool +apple_dcp_external_firmware_supported(const u32 *firmware, int firmware_cells, + const u32 *compat, int compat_cells) +{ + return firmware && compat && firmware_cells == DCP_EXTERNAL_FW_CELLS && + compat_cells == DCP_EXTERNAL_FW_CELLS && + firmware[0] == 26 && firmware[1] == 6 && firmware[2] == 2 && + compat[0] == 26 && compat[1] == 6 && compat[2] == 2; +} + +static inline bool apple_dcp_external_bw_valid(u64 size, int scratch, int request) +{ + if (size < sizeof(u32) || scratch < 0 || request < 0) + return false; + if ((scratch & 3) || (request & 3) || scratch == request) + return false; + return (u64)scratch <= size - sizeof(u32) && + (u64)request <= size - sizeof(u32); +} + +static inline bool apple_dcp_external_clock_window(u64 start, u64 size, + u64 expected_start, u64 expected_size) +{ + return start == expected_start && size == expected_size; +} + + +#endif diff --git a/drivers/gpu/drm/apple/dcp-fabric.c b/drivers/gpu/drm/apple/dcp-fabric.c index 78a436b23193a6..80c205b8a1f67f 100644 --- a/drivers/gpu/drm/apple/dcp-fabric.c +++ b/drivers/gpu/drm/apple/dcp-fabric.c @@ -62,6 +62,12 @@ struct apple_dcp_typec_port { u32 applied_status; u32 applied_conf; bool applied_valid; + /* + * H17P: SAFE preparation retains this owner until the set phase + * completes (dcp_typec_route_prepare() and its finish and commit). + */ + bool safe_pending; + bool safe_finish_ready; bool hpd; /* Direct DP-alt: the port is in DP mode, routed or not, and its HPD. */ bool dp_wanted; @@ -162,6 +168,7 @@ static bool dcp_typec_route_busy(struct apple_dcp_typec_route *route) static bool dcp_typec_route_available(struct apple_dcp_typec_route *route) { + struct apple_dcp_typec_port *port; struct dcp_fabric_pipeline pipeline = { .owned = !!route->dcp->active_typec_route || route->dcp->tb_retiring, .fixed_busy = dcp_typec_route_busy(route), @@ -169,6 +176,12 @@ static bool dcp_typec_route_available(struct apple_dcp_typec_route *route) struct dcp_fabric_policy policy = { .dual_stream = dcp_typec_dual_stream() }; + /* Only H17P routes prepare SAFE: their pipeline stays owned until commit. */ + list_for_each_entry(port, &dcp_typec_ports, link) { + if (port->safe_pending && port->owner && + port->owner->dcp == route->dcp) + return false; + } pipeline.presence = dcp_hdmi_presence(route->dcp); return dcp_fabric_available(&pipeline, &policy); } @@ -274,6 +287,9 @@ static int dcp_typec_route_activate(struct apple_dcp_typec_route *route, route->port->secondary_connector : route->port->connector; int ret; + /* External H17P: a new route starts a new firmware-metadata connection. */ + dcp_firmware_metadata_connection(dcp, READ_ONCE(dcp->typec_generation), false); + /* * The fixed output's HPD handler leaves disconnects to DCP, so the port * can still be marked connected to a display that is gone. Release it @@ -359,6 +375,8 @@ static int dcp_typec_route_activate(struct apple_dcp_typec_route *route, } dcp->active_typec_route = route; scoped_guard(mutex, &dcp->tb_lock) { + route->deactivate_xbar_done = false; + route->deactivate_error = 0; route->active_xbar = xbar; route->tunnel = xbar != route->xbar; route->xbar_up = !route->tunnel; @@ -375,7 +393,7 @@ static int dcp_typec_route_activate(struct apple_dcp_typec_route *route, return 0; } -static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *route) +static int dcp_typec_route_release(struct apple_dcp_typec_route *route) { struct apple_dcp *dcp = route->dcp; struct apple_connector *connector = dcp->typec_connector; @@ -389,16 +407,36 @@ static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *route) * crossbar nor restart the tunnel pixel clock behind our back. */ scoped_guard(mutex, &dcp->tb_lock) { - if (!route->tunnel || route->xbar_up) - ret = mux_control_deselect(route->active_xbar ?: route->xbar); - else - dcp_dpxbar_preselect(route->active_xbar, MUX_IDLE_DISCONNECT); - /* mux_control_deselect releases its semaphore even on failure. */ - route->xbar_up = false; - if (ret) - dev_warn(dcp->dev, "crossbar deselect failed: %d\n", ret); - /* Ownership is released; let the caller release its route owner too. */ - ret = 0; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + if (route->deactivate_error) + return route->deactivate_error; + if (!route->deactivate_xbar_done) { + if (!route->tunnel || route->xbar_up) + ret = mux_control_deselect(route->active_xbar ?: route->xbar); + else + ret = dcp_dpxbar_preselect(route->active_xbar, + MUX_IDLE_DISCONNECT); + /* Deselect always unlocks: never consume that selection twice. */ + route->deactivate_xbar_done = true; + route->xbar_up = false; + if (ret) { + /* Idle is unproven; retain the owner and the real error. */ + route->deactivate_error = ret; + return ret; + } + } + } else { + if (!route->tunnel || route->xbar_up) + ret = mux_control_deselect(route->active_xbar ?: route->xbar); + else + dcp_dpxbar_preselect(route->active_xbar, MUX_IDLE_DISCONNECT); + /* mux_control_deselect releases its semaphore even on failure. */ + route->xbar_up = false; + if (ret) + dev_warn(dcp->dev, "crossbar deselect failed: %d\n", ret); + /* Ownership is released; let the caller release its route owner too. */ + ret = 0; + } if (route->tunnel && dcp->phy) { /* the tunnel pixel clock must not outlive the tunnel */ @@ -424,6 +462,8 @@ static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *route) if (sel && sel != -EOPNOTSUPP) dev_warn(dcp->dev, "DP tunnel source reset failed: %d\n", sel); } + if (dcp_is_external_h17p(dcp)) + dpavservep_detach(dcp); route->selected = false; ended_attachment = dcp_modes_end_typec(dcp, route); if (dcp->active_typec_route == route) @@ -495,6 +535,28 @@ static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *route) return 0; } +static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *route) +{ + struct apple_dcp *dcp = route->dcp; + int ret; + + if (dcp->fw_compat != DCP_FIRMWARE_H17P) + return dcp_typec_route_release(route); + + /* + * External H17P: the route stays owned until its firmware stop has + * completed; a failed stop keeps it, with its error. + */ + guard(mutex)(&dcp->external_lock); + if (dcp_is_external_h17p(dcp) && + (dcp->external_error || (dcp->external_power_attempted && !dcp->external_stopped))) + return dcp->external_error ?: -EAGAIN; + + dcp_firmware_metadata_connection(dcp, READ_ONCE(dcp->typec_generation), false); + ret = dcp_typec_route_release(route); + return ret; +} + /* * The DRM device once it has bound, that is once every pipeline has its * CRTC and every port its connectors; NULL before. @@ -1239,6 +1301,10 @@ static int dcp_typec_port_busy(struct apple_dcp_typec_port *port, { const char *label = NULL; + /* H17P: every pipeline drives another display; -EBUSY means retry. */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return -ENOSPC; + if (port->busy_noted) return -EBUSY; port->busy_noted = true; @@ -1249,6 +1315,93 @@ static int dcp_typec_port_busy(struct apple_dcp_typec_port *port, return -EBUSY; } +/* + * H17P (T8140, T8142): the mux transaction's SAFE phases. Preparation + * disconnects the display and stops new admission, while the route keeps + * its owner until the PHY has gone SAFE (dcp_typec_route_finish()) and the + * transaction commits (dcp_typec_route_commit()). Other processors register + * only dcp_typec_route_set(). + */ +static int dcp_typec_route_prepare(struct typec_mux_dev *mux, + struct typec_mux_state *state) +{ + struct apple_dcp_typec_route *route = typec_mux_get_drvdata(mux); + struct apple_dcp_typec_port *port = route->port; + struct apple_dcp *dcp; + int ret; + + guard(mutex)(&dcp_typec_fabric_lock); + if (state->mode != TYPEC_STATE_SAFE) + return port->safe_pending ? -EAGAIN : 0; + if (!port->owner || port->owner->tunnel) + return 0; + dcp = port->owner->dcp; + /* Stop new admission, but retain the owner until the set phase succeeds. */ + if (!port->safe_pending) + port->safe_finish_ready = false; + port->safe_pending = true; + port->applied_valid = false; + port->dp_wanted = false; + port->dp_hpd = false; + /* HPD absence alone is not proof of a completed firmware shutdown. */ + if (dcp_is_external_h17p(dcp)) { + scoped_guard(mutex, &dcp->external_lock) { + if (dcp->external_error) + return dcp->external_error; + if (!dcp->external_cable && dcp->external_stopped && + !READ_ONCE(dcp->dptxport[0].connected)) { + port->hpd = false; + return 0; + } + } + } + ret = dcp_dptx_disconnect_oob(to_platform_device(dcp->dev), 0); + if (ret) + return ret; + port->hpd = false; + return 0; +} + +static int dcp_typec_route_finish(struct typec_mux_dev *mux, + struct typec_mux_state *state) +{ + struct apple_dcp_typec_port *port = + ((struct apple_dcp_typec_route *)typec_mux_get_drvdata(mux))->port; + int ret; + + guard(mutex)(&dcp_typec_fabric_lock); + if (state->mode != TYPEC_STATE_SAFE || !port->safe_pending || + port->safe_finish_ready) + return 0; + ret = port->owner ? dcp_typec_route_deactivate(port->owner) : 0; + if (ret) + return ret; + port->safe_finish_ready = true; + return 0; +} + +static void dcp_typec_route_commit(struct typec_mux_dev *mux, + struct typec_mux_state *state) +{ + struct apple_dcp_typec_port *port = + ((struct apple_dcp_typec_route *)typec_mux_get_drvdata(mux))->port; + + guard(mutex)(&dcp_typec_fabric_lock); + if (state->mode != TYPEC_STATE_SAFE || !port->safe_pending) + return; + if (port->owner) + port->preferred_route = port->owner; + port->owner = NULL; + port->safe_pending = false; + port->safe_finish_ready = false; + port->applied_alt = state->alt; + port->applied_mode = state->mode; + port->applied_status = 0; + port->applied_conf = 0; + port->applied_valid = true; + dcp_typec_pipeline_freed(); +} + static int dcp_typec_route_set(struct typec_mux_dev *mux, struct typec_mux_state *state) { @@ -1269,6 +1422,13 @@ static int dcp_typec_route_set(struct typec_mux_dev *mux, guard(mutex)(&dcp_typec_fabric_lock); + /* H17P only: a SAFE transition is still being prepared or finished. */ + if (state->mode == TYPEC_STATE_SAFE && port->safe_pending) + return 0; + /* The previous SAFE transition has not finished; retry. */ + if (port->safe_pending) + return -EAGAIN; + /* * Every candidate route for this port is notified with the same state, * so only the first to arrive does the work; the others return early. @@ -1278,7 +1438,8 @@ static int dcp_typec_route_set(struct typec_mux_dev *mux, * without a word, so a nominated route might never be called at all -- * and the port would then never be routed. */ - if (port->applied_valid && port->applied_alt == state->alt && + if (port->applied_valid && !(dp_status & DP_STATUS_IRQ_HPD) && + port->applied_alt == state->alt && port->applied_mode == state->mode && port->applied_status == dp_status && port->applied_conf == dp_conf) return 0; @@ -1332,6 +1493,7 @@ static int dcp_typec_route_set(struct typec_mux_dev *mux, dcp_typec_rebalance_locked(NULL, 0); } + port->safe_pending = false; /* Data-only USB4 changes do not invalidate other display routes. */ port->applied_valid = true; return 0; @@ -1378,6 +1540,9 @@ static int dcp_typec_route_set(struct typec_mux_dev *mux, if (!port->owner) { best = dcp_typec_free_route(port); + /* H17P: every pipeline is driving another port's display. */ + if (!best && route->dcp->fw_compat == DCP_FIRMWARE_H17P) + return -ENOSPC; if (!best) return hpd ? dcp_typec_port_busy(port, route->dcp) : -EBUSY; @@ -1401,17 +1566,21 @@ static int dcp_typec_route_set(struct typec_mux_dev *mux, /* * A T6030 external processor services the request itself, and * reads the display again if it changed; elsewhere the active - * mode is applied again. + * mode is applied again. H17P always applies it again. */ - if (dcp->typec_connector && dcp->external_native) - dcp_external_sink_irq(dcp); - else if (dcp->typec_connector && - !READ_ONCE(dcp->typec_connector->connected)) { - ret = dcp_dptx_recover_irq(dcp); - if (ret) - return ret; - } else if (dcp->typec_connector) + if (dcp->typec_connector && dcp->fw_compat == DCP_FIRMWARE_H17P) dcp_retrain_oob(dcp->typec_connector); + else if (dcp->typec_connector && dcp->external_native) + dcp_external_sink_irq(dcp); + else if (dcp->typec_connector && !dcp_typec_sink_relink(dcp)) { + if (!READ_ONCE(dcp->typec_connector->connected)) { + ret = dcp_dptx_recover_irq(dcp); + if (ret) + return ret; + } else { + dcp_retrain_oob(dcp->typec_connector); + } + } } port->hpd = hpd; port->applied_valid = true; @@ -1591,7 +1760,8 @@ int dcp_tunnel_crossbar_up(struct apple_dcp *dcp, u32 unit) if (unit) return dcp_tunnel_split_up_locked(dcp); route = dcp->active_typec_route; - if (!route || !route->tunnel || !route->active_xbar) + if (!route || !route->tunnel || !route->active_xbar || + route->deactivate_xbar_done) return -ENODEV; if (!dcp_fabric_callback_valid(route->tunnel_generation, dcp->tb_generation)) return -ESTALE; @@ -3258,7 +3428,9 @@ static void dcp_typec_route_unregister(void *data) if (!route->tunnel && (port->hpd || dcp->typec_cable_connected)) dcp_dptx_disconnect_oob(to_platform_device(dcp->dev), 0); port->hpd = false; - dcp_typec_route_deactivate(route); + /* External H17P: a failed firmware stop keeps the route owned. */ + if (dcp_typec_route_deactivate(route) && dcp_is_external_h17p(dcp)) + return; port->owner = NULL; } if (port->secondary_owner == route) { @@ -3266,7 +3438,8 @@ static void dcp_typec_route_unregister(void *data) if (!route->tunnel && (port->hpd || dcp->typec_cable_connected)) dcp_dptx_disconnect_oob(to_platform_device(dcp->dev), 0); - dcp_typec_route_deactivate(route); + if (dcp_typec_route_deactivate(route) && dcp_is_external_h17p(dcp)) + return; port->secondary_owner = NULL; } WRITE_ONCE(route->dcp->tb_retiring, false); @@ -3444,6 +3617,11 @@ static int dcp_register_typec_routes_present(void *data) desc.fwnode = of_fwnode_handle(route_np); desc.set = dcp_typec_route_set; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + desc.prepare = dcp_typec_route_prepare; + desc.finish = dcp_typec_route_finish; + desc.commit = dcp_typec_route_commit; + } desc.name = mux_name; desc.drvdata = route; route->typec_mux = typec_mux_register(dev, &desc); @@ -3900,7 +4078,8 @@ void dcp_link(struct platform_device *pdev, struct apple_crtc *crtc, { struct apple_dcp *dcp = platform_get_drvdata(pdev); - dcp->crtc = crtc; + /* dcp_delayed_vblank() reads this after an early RTKit crash. */ + WRITE_ONCE(dcp->crtc, crtc); /* * Type-C connectors belong to physical ports and are bound by the diff --git a/drivers/gpu/drm/apple/dcp-fabric.h b/drivers/gpu/drm/apple/dcp-fabric.h index 94ac176d96db3c..5bb70348091145 100644 --- a/drivers/gpu/drm/apple/dcp-fabric.h +++ b/drivers/gpu/drm/apple/dcp-fabric.h @@ -46,6 +46,12 @@ struct apple_dcp_typec_route { bool tunnel; /* tunnel: crossbar brought up (at DidChangeLinkConfiguration) */ bool xbar_up; + /* + * H17P: the lower mux release is consumed even when idle programming + * fails, and that error retains the route's owner. + */ + bool deactivate_xbar_done; + int deactivate_error; }; bool dcp_is_typec_output(struct apple_dcp *dcp); diff --git a/drivers/gpu/drm/apple/dcp-internal.h b/drivers/gpu/drm/apple/dcp-internal.h index e5e40af46e233e..4d706327793c04 100644 --- a/drivers/gpu/drm/apple/dcp-internal.h +++ b/drivers/gpu/drm/apple/dcp-internal.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -18,7 +19,12 @@ #include "dcp-fabric.h" #include "dcp-fabric-session.h" #include "dptxep.h" +#include "dcp_backlight.h" +#include "dcp-display-metadata.h" +#include "dcp-metadata-epoch.h" +#include "dcp-180hz.h" #include "iomfb.h" +#include "iomfb_h17p.h" #include "iomfb-state.h" #include "iomfb_v12_3.h" #include "iomfb_v13_3.h" @@ -31,10 +37,32 @@ struct apple_dcp_afkep; +/* Snapshot of a completed present, independent of the latest DRM state. */ +struct iomfb_scanout_h17p { + struct dcp_swap_submit_req_h17p request; + struct drm_framebuffer *fb[SWAP_SURFACES]; +}; + +/* One operation, including all of its nested replies and present completion. */ +struct iomfb_transaction { + struct list_head link; + void (*start)(struct apple_dcp *dcp, struct iomfb_transaction *transaction); + void (*release)(struct iomfb_transaction *transaction); + struct dcp_backlight_present backlight; + bool backlight_reserved; + bool backlight_failed; + bool brightness_only; + bool presentation; + bool completed; +}; +struct apple_dcp_typec_port; + void dcp_swap_watchdog_arm(struct apple_dcp *dcp); void dcp_swap_watchdog_complete(struct apple_dcp *dcp); void dcp_retry_placeholder_edid(struct apple_dcp *dcp, const struct drm_edid *drm_edid); +bool dcp_typec_relink(struct apple_dcp *dcp); +bool dcp_typec_sink_relink(struct apple_dcp *dcp); struct dcpav_service_epic; @@ -51,6 +79,7 @@ enum dcp_firmware_version { * endpoints speak the 14.7 protocol (dcp_fw_is_14_7()). */ DCP_FIRMWARE_V_14_7_TEMPLATE, + DCP_FIRMWARE_H17P, }; enum { @@ -60,9 +89,12 @@ enum { DISP0_ENDPOINT = 0x23, /* DPAV controller: publishes the port service used by DPTX 0x2a. */ DPAV_CTRL_ENDPOINT = 0x24, + DPTX_CONTROLLER_ENDPOINT = 0x24, DPAVSERV_ENDPOINT = 0x28, + DPAVSERV_ENDPOINT_V2 = 0x2a, AV_ENDPOINT = 0x29, DPTX_ENDPOINT = 0x2a, + DPTX_ENDPOINT_V2 = 0x2c, HDCP_ENDPOINT = 0x2b, REMOTE_ALLOC_ENDPOINT = 0x2d, IOMFB_ENDPOINT = 0x37, @@ -73,6 +105,14 @@ struct dcp_chunks { u64 modes_generation; size_t length; void *data; + size_t received; + bool contiguous; + bool payload_complete; + struct dcp_metadata_ticket metadata_ticket; + u64 av_service_epoch; /* Captured at property start, never at publication. */ + struct dcp_metadata_ticket mode_ticket; + u64 automatic_connection_sequence; + bool automatic_service_pending; }; #define DCP_MAX_MAPPINGS (128) /* should be enough */ @@ -84,6 +124,8 @@ struct dcp_mem_descriptor { dma_addr_t dva; struct sg_table map; u64 reg; + /* live iommu mappings installed by dcpep_cb_map_piodma() */ + bool piodma_mapped; }; /* Limit on call stack depth (arbitrary). Some nesting is required */ @@ -95,6 +137,8 @@ struct dcp_channel { dcp_callback_t callbacks[DCP_MAX_CALL_DEPTH]; void *cookies[DCP_MAX_CALL_DEPTH]; void *output[DCP_MAX_CALL_DEPTH]; + /* Header of each pending AP command record, as dcp_push() wrote it */ + struct dcp_packet_header header[DCP_MAX_CALL_DEPTH]; u16 end[DCP_MAX_CALL_DEPTH]; /* Current depth of the call stack. Less than DCP_MAX_CALL_DEPTH */ @@ -106,7 +150,14 @@ struct dcp_channel { struct dcp_fb_reference { struct list_head head; struct drm_framebuffer *fb; + /* + * Id of the swap that unbinds @fb, i.e. the one that puts its + * replacement on screen. Only meaningful once @armed is set: the + * firmware assigns the id in the swap_start reply, which happens after + * the atomic commit that displaced @fb has already queued this entry. + */ u32 swap_id; + bool armed; }; #define MAX_NOTCH_HEIGHT 160 @@ -132,9 +183,27 @@ struct dcp_panel { bool has_mini_led; }; +enum dcp_iomfb_method_profile { + DCP_IOMFB_METHODS_DEFAULT, + DCP_IOMFB_METHODS_H17G, + DCP_IOMFB_METHODS_H17P_EXTERNAL, +}; + struct apple_dcp_hw_data { u32 num_dptx_ports; bool t6020_tunnel_flow; + enum dcp_iomfb_method_profile iomfb_method_profile; + /* + * The bootloader leaves the coprocessor running. Attach to it with a + * standard RTKit wake instead of starting or restarting the ASC. + */ + bool adopt_live_session; + enum dcp_firmware_version firmware_compat; + /* Firmware-visible clock aperture, independent of the AP PMGR window. */ + const struct resource *firmware_clock; + u32 firmware_display_windows; + u32 firmware_scratch; + u32 firmware_request; }; /* TODO: move IOMFB members to its own struct */ @@ -143,14 +212,41 @@ struct dcp_hdmi; struct apple_dcp { struct dcp_fabric_pipeline fabric; + /* An adopted session has no confirmed firmware DMA stop boundary. */ + bool retain_dma; + bool quiescing; + bool drm_retained; struct device *dev; struct platform_device *piodma; + bool piodma_created; struct iommu_domain *iommu_dom; + /* which of the nine cumulative A031 notify-client states to send next */ + unsigned int a031_step; struct apple_rtkit *rtk; struct apple_crtc *crtc; struct apple_connector *connector; struct apple_dcp_hw_data hw; + bool external_prepared; + /* Serializes external connection, power, and CRTC generation changes. */ + struct mutex external_lock; + u64 external_generation; + u64 external_crtc_generation; + /* iomfb.lock: epoch used by receive-side deferred stop requests. */ + u64 external_rx_generation; + u64 external_stop_generation; + bool external_cable; + /* RX tombstone: initially unknown; only physical OOB events change it. */ + bool external_physically_absent; + bool external_running; + bool external_power_attempted; + /* iomfb.lock: replay the latest callback after activation succeeds. */ + bool external_hotplug_pending; + bool external_hotplug_connected; + bool external_stopped; + int external_error; + struct work_struct external_stop_work; + /* firmware version and compatible firmware version */ enum dcp_firmware_version fw_compat; @@ -215,6 +311,7 @@ struct apple_dcp { /* clock rate request by dcp in */ struct clk *clk; + struct clk *clk_194; /* DCP shared memory */ void *shmem; @@ -246,16 +343,31 @@ struct apple_dcp { /* Active chunked transfer. There can only be one at a time. */ struct dcp_chunks chunks; + struct dcp_180hz_proof rate_proof; + struct delayed_work rate_retry_wq; + bool rate_retry_initialized; + struct { + /* Protects connection tickets and cached metadata snapshots. */ + spinlock_t lock; + struct dcp_metadata_epoch epoch; + struct dcp_display_metadata attributes; + struct drm_display_mode preferred; + struct dcp_metadata_ticket attributes_ticket, timing_ticket; + bool attributes_valid, timing_valid, hpd_seen; + } metadata; /* Queued swap. Owned by the DCP to avoid per-swap memory allocation */ union { struct dcp_swap_submit_req_v12_3 v12_3; struct dcp_swap_submit_req_v13_3 v13_3; struct dcp_swap_submit_req_v14_7_0 v14_7_0; + struct dcp_swap_submit_req_h17p h17p; } swap; /* swap id of the last completed swap */ u32 last_swap_id; + /* last_swap_id is only meaningful after the first swap completes */ + bool have_swap_complete; ktime_t swap_start; u64 swap_submit_timestamp; @@ -270,6 +382,12 @@ struct apple_dcp { u64 crtc_unplugs; u64 placeholder_generation; struct delayed_work placeholder_edid_wq; + /* replugs replayed for a Type-C sink that lost its link */ + u32 relink_tries; + bool relink_active; + unsigned long relink_done; + u64 relink_generation; + struct delayed_work typec_relink_wq; bool use_timestamps; bool vrr_enabled; struct dcp_set_digital_out_mode_req mode; @@ -279,6 +397,10 @@ struct apple_dcp { /* Is the DCP booted? */ bool active; + /* A successful initial enable keeps the H17P pipe running across DPMS. */ + bool pipe_enabled_h17p; + /* Messages from the coprocessor are held in the mailbox for sleep. */ + bool rx_held; /* eDP display without DP-HDMI conversion */ bool main_display; @@ -318,10 +440,12 @@ struct apple_dcp { struct delayed_work swap_watchdog_wq; unsigned int swap_watchdog_retrains; - /* List of referenced drm_framebuffers which can be unreferenced - * on the next successfully completed swap. + /* List of referenced framebuffers pending a completed replacement swap. + * The commit and RTKit work queues share it under this lock. */ struct list_head swapped_out_fbs; + /* Protects swapped_out_fbs. */ + struct mutex swapped_out_fbs_lock; struct dcp_brightness brightness; /* Workqueue for updating the initial brightness */ @@ -329,6 +453,8 @@ struct apple_dcp { struct mutex bl_register_mutex; /* Workqueue for updating the brightness */ struct work_struct bl_update_wq; + /* Registers the H17P backlight when the loader reports no level. */ + struct delayed_work bl_fallback_wq; /* integrated panel if present */ struct dcp_panel panel; @@ -344,6 +470,8 @@ struct apple_dcp { struct audiosrv_data *audiosrv; struct apple_dcp_afkep *dptxep; + struct apple_dcp_afkep *dptx_controller_ep; + struct completion dptx_controller_ready; struct apple_dcp_afkep *dpavctrlep; struct apple_epic_service *dpav_ctrl; @@ -404,6 +532,8 @@ struct apple_dcp { bool typec_follow_start; bool typec_follow_retiring; u64 typec_follow_gen; + /* dcp_poweron() reconnects DPTX; its own modeset follows the HPD */ + bool typec_poweron_hpd; /* IOMFB reports its video interface ready after DPTX link training. */ struct completion typec_iomfb_hpd_ready; struct delayed_work typec_reconnect_wq; @@ -434,11 +564,86 @@ struct apple_dcp { struct delayed_work hdmi_recheck_wq; /* T6030 HDMI converter service requests, see dcp-hdmi.c */ struct dcp_hdmi *hdmi; + + /* H17P policy shared by backlight, commit, and reply workers. */ + struct { + /* Protects state and callback installation. */ + spinlock_t lock; + struct dcp_backlight_state state; + void (*kick)(struct apple_dcp *dcp); + /* Steps a running fade; fade_start is its start in jiffies. */ + struct delayed_work fade_work; + unsigned long fade_start; + /* Rate limit for firmware re-send requests. */ + unsigned long resent_at; + bool resent; + } backlight; + + /* Staging wire record; serialization never mutates the atomic inputs. */ + struct dcp_present_h17p present_h17p; + struct dcp_present_state_h17p present_state_h17p; + DECLARE_BITMAP(unknown_callbacks, IOMFB_MAX_CB); + + /* Serializes H17P transmit preparation with RTKit receive callbacks. */ + struct { + /* Protects queue, channel stacks, and the current owner. */ + struct mutex lock; + struct list_head pending; + struct iomfb_transaction *active; + struct task_struct *owner; + struct work_struct work; + struct delayed_work timeout; + struct delayed_work backlight_retry; + unsigned long deadline; + unsigned int queued; + bool stopped; + bool presentation_paused; + bool backlight_queued; + atomic_t opaque_x_state; + struct iomfb_scanout_h17p *scanout; + struct iomfb_scanout_h17p *next_scanout; + } iomfb; }; +void dcp_firmware_metadata_connection(struct apple_dcp *dcp, u64 generation, bool connected); +void dcp_firmware_metadata_hotplug(struct apple_dcp *dcp, bool connected); +void dcp_firmware_metadata_begin(struct apple_dcp *dcp); +void dcp_firmware_metadata_attributes(struct apple_dcp *dcp, + const struct dcp_display_metadata *attributes); +void dcp_firmware_metadata_timing(struct apple_dcp *dcp); +bool dcp_firmware_metadata_current(struct apple_dcp *dcp, u64 generation); +const struct drm_edid *dcp_firmware_edid(struct apple_dcp *dcp, u64 *generation); + +void iomfb_scanout_complete_h17p(struct apple_dcp *dcp); +void iomfb_present_backlight_h17p(struct apple_dcp *dcp); +bool iomfb_present_brightness_only_h17p(struct apple_dcp *dcp); +bool iomfb_present_complete_h17p(struct apple_dcp *dcp); +void iomfb_present_failed_h17p(struct apple_dcp *dcp); +bool iomfb_apply_backlight_h17p(struct apple_dcp *dcp, + const struct dcp_swap_submit_req_h17p *request, + struct dcp_present_h17p *wire); +void iomfb_apply_opaque_x_h17p(struct apple_dcp *dcp, dcp_callback_t callback, + void *cookie); +void iomfb_opaque_x_reset_h17p(struct apple_dcp *dcp); +int iomfb_configure_backlight_h17p(struct apple_dcp *dcp, u32 maximum, + bool inherited_valid, u32 inherited, + bool default_valid, u32 default_nits); +void iomfb_queue_init(struct apple_dcp *dcp); +void iomfb_queue_stop(struct apple_dcp *dcp); +bool iomfb_queue_drain(struct apple_dcp *dcp, unsigned long timeout); +int iomfb_queue_settle(struct apple_dcp *dcp, unsigned long timeout); +void iomfb_pause_presentation(struct apple_dcp *dcp, bool pause); +int iomfb_external_power_h17p(struct apple_dcp *dcp, bool on); +int iomfb_queue(struct apple_dcp *dcp, struct iomfb_transaction *transaction); +void iomfb_queue_crc_h17p(struct apple_dcp *dcp, u32 swap_id); + void dcp_drm_crtc_page_flip(struct apple_dcp *dcp, ktime_t now); void dcp_handle_hotplug_actions(struct apple_dcp *dcp, unsigned int action); +bool dcp_external_hotplug(struct apple_dcp *dcp, bool connected); +int dcp_backlight_init(struct apple_dcp *dcp, struct device *dev); +void dcp_backlight_stop(struct apple_dcp *dcp); +void dcp_backlight_finish_fade(struct apple_dcp *dcp); int dcp_backlight_register(struct apple_dcp *dcp); int dcp_backlight_update(struct apple_dcp *dcp); s32 dcp_ext_backlight_value(struct apple_dcp *dcp); @@ -469,6 +674,12 @@ static inline bool dcp_ext_dptx_607(const struct apple_dcp *dcp) return dcp->fw_compat == DCP_FIRMWARE_V_14_7_TEMPLATE; } +/* The T8140 external processor (dispext0) running H17P firmware. */ +static inline bool dcp_is_external_h17p(const struct apple_dcp *dcp) +{ + return dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL; +} + #define DCP_AUDIO_MAX_CHANS 15 #endif /* __APPLE_DCP_INTERNAL_H__ */ diff --git a/drivers/gpu/drm/apple/dcp-link.h b/drivers/gpu/drm/apple/dcp-link.h index 6f27757be27953..1bcca315c42578 100644 --- a/drivers/gpu/drm/apple/dcp-link.h +++ b/drivers/gpu/drm/apple/dcp-link.h @@ -761,27 +761,33 @@ apple_dcp_h17p_rt_bw_encode(struct apple_dcp_h17p_rt_bw_config *config, config->clock_request = cpu_to_le64(clock_request); } +/* D003: fill in the bandwidth config unless the firmware passed it as null */ static inline int -apple_dcp_h17p_rt_bw_encode_reply(struct apple_dcp_link_rpc_view *view, +apple_dcp_h17p_rt_bw_encode_reply(const struct apple_dcp_h17p_rt_bw_request *request, + struct apple_dcp_h17p_rt_bw_reply *reply, u64 scratch, u64 clock_request) { - const struct apple_dcp_h17p_rt_bw_request *request; - struct apple_dcp_h17p_rt_bw_reply *reply; - - if (!view || view->call != APPLE_DCP_LINK_CALL_D003) - return -EINVAL; - if (view->input_size < sizeof(*request) || - view->output_size < sizeof(*reply)) - return -EMSGSIZE; - - request = view->input; if (le32_to_cpu(request->config_null) & 1) return -EINVAL; - reply = view->output; apple_dcp_h17p_rt_bw_encode(&reply->config, scratch, clock_request); reply->status = cpu_to_le32(0); return 0; } +/* The same, for a call record seen through a DCPLink RPC view. */ +static inline int +apple_dcp_h17p_rt_bw_encode_reply_view(struct apple_dcp_link_rpc_view *view, + u64 scratch, u64 clock_request) +{ + if (!view || view->call != APPLE_DCP_LINK_CALL_D003) + return -EINVAL; + if (view->input_size < sizeof(struct apple_dcp_h17p_rt_bw_request) || + view->output_size < sizeof(struct apple_dcp_h17p_rt_bw_reply)) + return -EMSGSIZE; + + return apple_dcp_h17p_rt_bw_encode_reply(view->input, view->output, + scratch, clock_request); +} + #endif /* __APPLE_DCP_LINK_H__ */ diff --git a/drivers/gpu/drm/apple/dcp-metadata-epoch.h b/drivers/gpu/drm/apple/dcp-metadata-epoch.h new file mode 100644 index 00000000000000..c2ab0c22a74e0b --- /dev/null +++ b/drivers/gpu/drm/apple/dcp-metadata-epoch.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DCP_METADATA_EPOCH_H +#define APPLE_DCP_METADATA_EPOCH_H + +#include + +/* Caller serializes these fields with the metadata snapshot lock. */ +struct dcp_metadata_epoch { + u64 generation; + bool connected; + bool associated_once; + bool invalidated; +}; + +struct dcp_metadata_ticket { + u64 generation; + bool associated; +}; + +static inline void dcp_metadata_epoch_connection(struct dcp_metadata_epoch *epoch, + u64 generation, bool connected) +{ + if (epoch->associated_once && + (!connected || generation != epoch->generation)) + epoch->invalidated = true; + if (connected) + epoch->associated_once = true; + epoch->generation = generation; + epoch->connected = connected; +} + +static inline struct dcp_metadata_ticket +dcp_metadata_begin(const struct dcp_metadata_epoch *epoch) +{ + return (struct dcp_metadata_ticket) { + .generation = epoch->generation, + .associated = epoch->connected && !epoch->invalidated, + }; +} + +static inline bool dcp_metadata_matches(const struct dcp_metadata_epoch *epoch, + struct dcp_metadata_ticket ticket) +{ + return ticket.associated && epoch->connected && !epoch->invalidated && + ticket.generation == epoch->generation; +} + +#endif diff --git a/drivers/gpu/drm/apple/dcp.c b/drivers/gpu/drm/apple/dcp.c index 03a3737c9b2507..7b01eaf6d59a78 100644 --- a/drivers/gpu/drm/apple/dcp.c +++ b/drivers/gpu/drm/apple/dcp.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -17,10 +18,12 @@ #include #include #include +#include #include #include #include #include +#include #include #include #include @@ -33,6 +36,7 @@ #include #include +#include #include #include #include @@ -47,6 +51,8 @@ #include "dcp-hdmi.h" #include "dcpext_scanout.h" #include "dcp-internal.h" +#include "dcp-external-bw.h" +#include "dptx-service.h" #include "iomfb.h" #include "ibootep.h" #include "parser.h" @@ -54,6 +60,9 @@ #define APPLE_DCP_COPROC_CPU_CONTROL 0x44 #define APPLE_DCP_COPROC_CPU_CONTROL_RUN BIT(4) +#define APPLE_DCP_COPROC_CPU_STATUS 0x48 +#define APPLE_DCP_COPROC_CPU_STATUS_STOPPED BIT(1) +#define APPLE_DCP_COPROC_CPU_STATUS_IDLE BIT(5) #define DCP_BOOT_TIMEOUT msecs_to_jiffies(1000) @@ -74,6 +83,31 @@ static bool unstable_edid = true; module_param(unstable_edid, bool, 0644); MODULE_PARM_DESC(unstable_edid, "Enable unstable EDID retrival support"); +/* + * H17-generation firmware (T8140 H17P, T8142 H17G) runs aurora-wip's + * processor lifecycle, below in the H17P section. Shared entry points + * hand those processors over to it. + */ +static void dcp_recv_msg_h17p(struct apple_dcp *dcp, u8 endpoint, u64 message); +static void dcp_rtk_crashed_h17p(struct apple_dcp *dcp); +static int dcp_rtk_shmem_setup_h17p(struct apple_dcp *dcp, struct apple_rtkit_shmem *bfr); +static int dcp_crtc_atomic_check_h17p(struct drm_crtc *crtc, struct drm_atomic_state *state); +static int dcp_dptx_connect_h17p(struct apple_dcp *dcp, u32 port); +static int dcp_dptx_disconnect_h17p(struct apple_dcp *dcp, u32 port); +static int dcp_dptx_connect_oob_h17p(struct platform_device *pdev, u32 port); +static int dcp_dptx_disconnect_oob_h17p(struct platform_device *pdev, u32 port); +static int dcp_start_h17p(struct platform_device *pdev); +static void dcp_poweron_h17p(struct platform_device *pdev); +static void dcp_poweroff_h17p(struct platform_device *pdev); +static void dcp_disable_typec_work_h17p(struct apple_dcp *dcp, bool release_cable); +static void dcp_enable_typec_work_h17p(struct apple_dcp *dcp); +static int dcp_comp_bind_h17p(struct device *dev, struct device *main, void *data); +static void dcp_comp_unbind_h17p(struct device *dev, struct device *main, void *data); +static int dcp_platform_probe_h17p(struct platform_device *pdev); +static void dcp_platform_shutdown_h17p(struct platform_device *pdev); +static int dcp_platform_suspend_h17p(struct device *dev); +static int dcp_platform_resume_h17p(struct device *dev); + /* copied and simplified from drm_vblank.c */ static void send_vblank_event(struct drm_device *dev, struct drm_pending_vblank_event *e, @@ -230,9 +264,15 @@ static void dcp_delayed_vblank(struct work_struct *work) { struct apple_dcp *dcp; + struct apple_crtc *crtc; + dcp = container_of(work, struct apple_dcp, vblank_wq); + /* RTKit can report a crash before the pipeline has a CRTC. */ + crtc = READ_ONCE(dcp->crtc); + if (!crtc) + return; mdelay(5); - dcp_drm_crtc_vblank(dcp->crtc); + dcp_drm_crtc_vblank(crtc); } #define DCP_SWAP_WATCHDOG_MS 1000 @@ -277,6 +317,9 @@ static void dcp_recv_msg(void *cookie, u8 endpoint, u64 message) { struct apple_dcp *dcp = cookie; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_recv_msg_h17p(dcp, endpoint, message); + trace_dcp_recv_msg(dcp, endpoint, message); /* A failed T8122 dcpext: its firmware may still talk, nobody listens. */ @@ -310,6 +353,9 @@ static void dcp_rtk_crashed(void *cookie, const void *crashlog, size_t crashlog_ { struct apple_dcp *dcp = cookie; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_rtk_crashed_h17p(dcp); + /* * A failed T8122 dcpext: its start never ran (start_done is not set * up) and it has no connector. Its firmware may still report a crash. @@ -333,6 +379,9 @@ static int dcp_rtk_shmem_setup(void *cookie, struct apple_rtkit_shmem *bfr) { struct apple_dcp *dcp = cookie; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_rtk_shmem_setup_h17p(dcp, bfr); + if (bfr->iova) { struct iommu_domain *domain = iommu_get_domain_for_dev(dcp->dev); @@ -390,6 +439,15 @@ void dcp_send_message(struct apple_dcp *dcp, u8 endpoint, u64 message) { int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + if (READ_ONCE(dcp->quiescing)) + return; + trace_dcp_send_msg(dcp, endpoint, message); + apple_rtkit_send_message(dcp->rtk, endpoint, message, NULL, + true); + return; + } + trace_dcp_send_msg(dcp, endpoint, message); /* * The adopted 14.7 session shares this mailbox with the panel link. @@ -411,6 +469,8 @@ int dcp_crtc_atomic_check(struct drm_crtc *crtc, struct drm_atomic_state *state) struct drm_crtc_state *crtc_state; bool needs_modeset; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_crtc_atomic_check_h17p(crtc, state); if (dcp->fw_compat == DCP_FIRMWARE_V_26_6) return iomfb_v26_6_atomic_check(dcp, crtc, state); if (dcp->fw_compat == DCP_FIRMWARE_V_14_7) @@ -947,6 +1007,9 @@ int dcp_dptx_connect(struct apple_dcp *dcp, u32 port) struct dcp_fabric_session session; int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_dptx_connect_h17p(dcp, port); + scoped_guard(mutex, &dcp->hpd_mutex) session = dcp_session_locked(dcp); ret = dcp_dptx_connect_session(dcp, port, session, !!session.cookie, false); @@ -1066,6 +1129,111 @@ static void dcp_typec_reconnect_work(struct work_struct *work) dcp->typec_reconnect_tries, ret); } +#define DPTX_RELINK_RETRIES 3 +#define DPTX_RELINK_UNPLUG_MS 1000 +/* a monitor pulses HPD up to about 4 s after it lost the link */ +#define DPTX_RELINK_SETTLE_MS 4000 + +/* + * A USB-C monitor with a built-in hub can lose the DP link when the hub + * brings its SuperSpeed link up right around DCP's link training for the + * first modeset. DCP may then report the sink as unplugged two seconds + * later although it still asserts HPD, or keep driving the dead link + * without noticing while the monitor pulses HPD to be detected again. + * Neither is recovered by anything else: replay an unplug and replug of + * the sink, which retrains the link once the hub has settled, as a cable + * replug does. + * + * The retries cover one connection or one DPMS wake: the replay itself + * does not reset them, dcp_poweron() does. + */ +static bool dcp_typec_relink_start(struct apple_dcp *dcp, const char *why) +{ + u64 generation = READ_ONCE(dcp->typec_generation); + + /* H17P and native external pipes recover through their own lifecycle */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P || dcp->external_native || + !dcp_is_typec_output(dcp) || dcp_is_usb4_output(dcp) || + dcp_is_external_h17p(dcp) || + !READ_ONCE(dcp->typec_cable_connected) || + READ_ONCE(dcp->typec_crtc_off) || !dcp->typec_connector) + return false; + + if (dcp->relink_generation != generation) { + dcp->relink_generation = generation; + dcp->relink_tries = 0; + WRITE_ONCE(dcp->relink_active, false); + } + /* the unplug a running relink replays makes DCP report one too */ + if (READ_ONCE(dcp->relink_active)) + return true; + if (dcp->relink_tries >= DPTX_RELINK_RETRIES) + return false; + + dcp->relink_tries++; + WRITE_ONCE(dcp->relink_active, true); + dev_info(dcp->dev, "%s, relinking the Type-C sink (%u/%u)\n", why, + dcp->relink_tries, DPTX_RELINK_RETRIES); + mod_delayed_work(system_freezable_wq, &dcp->typec_relink_wq, 0); + + return true; +} + +/* DCP reported an unplug of a Type-C sink that still asserts HPD */ +bool dcp_typec_relink(struct apple_dcp *dcp) +{ + return dcp_typec_relink_start(dcp, "DCP dropped a sink that asserts HPD"); +} + +/* + * The sink asked for attention: IRQ_HPD or a coalesced HPD pulse. Right + * after a relink that is the sink's late reaction to the loss the relink + * already handled; another relink would only interrupt the new link. + */ +bool dcp_typec_sink_relink(struct apple_dcp *dcp) +{ + unsigned long done = READ_ONCE(dcp->relink_done); + + if (done && time_before(jiffies, done + msecs_to_jiffies(DPTX_RELINK_SETTLE_MS)) && + READ_ONCE(dcp->relink_generation) == READ_ONCE(dcp->typec_generation)) { + dev_info(dcp->dev, "Type-C sink requested attention right after a relink, ignored\n"); + return true; + } + + return dcp_typec_relink_start(dcp, "Type-C sink requested attention"); +} + +static void dcp_typec_relink_work(struct work_struct *work) +{ + struct apple_dcp *dcp = + container_of(to_delayed_work(work), struct apple_dcp, + typec_relink_wq); + struct platform_device *pdev = to_platform_device(dcp->dev); + u64 generation; + + scoped_guard(mutex, &dcp->hpd_mutex) + generation = dcp->typec_generation; + if (generation != dcp->relink_generation || + !READ_ONCE(dcp->typec_cable_connected)) + goto out; + + dcp_dptx_disconnect_oob(pdev, 0); + msleep(DPTX_RELINK_UNPLUG_MS); + + /* a real unplug or replug meanwhile takes over */ + scoped_guard(mutex, &dcp->hpd_mutex) { + if (dcp->typec_generation != generation + 1) + goto out; + } + dcp_dptx_connect_oob(pdev, 0); + + scoped_guard(mutex, &dcp->hpd_mutex) + dcp->relink_generation = dcp->typec_generation; + WRITE_ONCE(dcp->relink_done, jiffies); +out: + WRITE_ONCE(dcp->relink_active, false); +} + static void disconnected_hpd_event(struct apple_connector *con) { apple_connector_edid_set_live(con, false); @@ -1116,6 +1284,9 @@ int dcp_dptx_disconnect(struct apple_dcp *dcp, u32 port) { int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_dptx_disconnect_h17p(dcp, port); + /* Release the caller's RemotePort service, not the downstream DFP port. */ dev_info(dcp->dev, "%s(port=%d)\n", __func__, port); @@ -1133,6 +1304,9 @@ int dcp_dptx_connect_oob(struct platform_device *pdev, u32 port) struct apple_dcp *dcp = platform_get_drvdata(pdev); int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_dptx_connect_oob_h17p(pdev, port); + if (dcp_is_typec_output(dcp)) { cancel_delayed_work_sync(&dcp->placeholder_edid_wq); guard(mutex)(&dcp->hpd_mutex); @@ -1159,6 +1333,9 @@ int dcp_dptx_disconnect_oob(struct platform_device *pdev, u32 port) { struct apple_dcp *dcp = platform_get_drvdata(pdev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_dptx_disconnect_oob_h17p(pdev, port); + if (dcp_is_typec_output(dcp)) { scoped_guard(mutex, &dcp->hpd_mutex) { WRITE_ONCE(dcp->typec_cable_connected, false); @@ -1329,6 +1506,9 @@ int dcp_start(struct platform_device *pdev) struct apple_dcp *dcp = platform_get_drvdata(pdev); int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_start_h17p(pdev); + init_completion(&dcp->start_done); if (dcp->fw_compat == DCP_FIRMWARE_V_26_6) return iomfb_v26_6_start(dcp); @@ -1436,6 +1616,9 @@ static void _dcp_poweroff(struct apple_dcp *dcp) case DCP_FIRMWARE_V_14_7: iomfb_v14_7_poweroff(dcp); break; + case DCP_FIRMWARE_H17P: + iomfb_poweroff_h17p(dcp); + break; default: WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); break; @@ -1524,6 +1707,9 @@ static void __maybe_unused dcp_sleep(struct apple_dcp *dcp) case DCP_FIRMWARE_V_14_7: iomfb_v14_7_poweroff(dcp); break; + case DCP_FIRMWARE_H17P: + iomfb_sleep_h17p(dcp); + break; default: WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); break; @@ -1558,6 +1744,9 @@ void dcp_poweron(struct platform_device *pdev) unsigned long remaining; int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_poweron_h17p(pdev); + if (dcp_is_typec_output(dcp)) { /* T8122 dcpext only, see dcp_poweroff() */ if (dcp->ext_failsafe) { @@ -1568,6 +1757,12 @@ void dcp_poweron(struct platform_device *pdev) READ_ONCE(dcp->typec_cable_connected); WRITE_ONCE(dcp->typec_crtc_off, false); WRITE_ONCE(dcp->typec_follow_start, false); + /* + * A DPMS wake retrains the link like a replug: give it a fresh + * relink budget. No relink runs while the CRTC was off. + */ + if (wait_for_typec_hpd) + dcp->relink_tries = 0; /* * A Type-C CRTC disable releases its DPTX session. Re-establish it @@ -1576,6 +1771,8 @@ void dcp_poweron(struct platform_device *pdev) if (READ_ONCE(dcp->typec_cable_connected)) { cancel_delayed_work(&dcp->typec_reconnect_wq); dcp->typec_reconnect_tries = 0; + /* the modeset of this commit follows, see dcpep_cb_hotplug() */ + WRITE_ONCE(dcp->typec_poweron_hpd, true); ret = dcp_dptx_connect(dcp, 0); if (ret) dcp_queue_typec_reconnect(dcp, DPTX_RECONNECT_DELAY); @@ -1587,6 +1784,7 @@ void dcp_poweron(struct platform_device *pdev) dev_warn(dcp->dev, "Type-C IOMFB hotplug not ready on wake\n"); } + WRITE_ONCE(dcp->typec_poweron_hpd, false); } } else if (dcp->hdmi_hpd) { bool connected = gpiod_get_value_cansleep(dcp->hdmi_hpd); @@ -1626,6 +1824,9 @@ void dcp_poweroff(struct platform_device *pdev) bool display_gone = false; int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_poweroff_h17p(pdev); + cancel_delayed_work(&dcp->swap_watchdog_wq); if (dcp->avep) av_service_disconnect(dcp); @@ -1955,11 +2156,19 @@ static int dcp_read_fw_version(struct device *dev, const char *name, static enum dcp_firmware_version dcp_check_firmware_version(struct device *dev) { + const struct apple_dcp_hw_data *hw = of_device_get_match_data(dev); const struct dcp_v14_board *v14_board; char compat_str[DCP_FW_VERSION_STR_LEN]; char fw_str[DCP_FW_VERSION_STR_LEN]; int ret; + /* + * SoCs introduced with H17-generation firmware pin its interface; the + * loader may not recognise their boot firmware version. + */ + if (hw && hw->firmware_compat != DCP_FIRMWARE_UNKNOWN) + return hw->firmware_compat; + /* firmware version is just informative */ dcp_read_fw_version(dev, "apple,firmware-version", fw_str); @@ -2052,6 +2261,8 @@ static int dcp_connector_type_from_dt(struct device_node *np) static void dcp_disable_typec_work(struct apple_dcp *dcp, bool release_cable) { + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_disable_typec_work_h17p(dcp, release_cable); scoped_guard(mutex, &dcp->hpd_mutex) { dcp->typec_work_enabled = false; if (release_cable) @@ -2061,6 +2272,7 @@ static void dcp_disable_typec_work(struct apple_dcp *dcp, bool release_cable) /* Block new enqueues as well as draining users of the AFK endpoints. */ disable_delayed_work_sync(&dcp->typec_reconnect_wq); disable_delayed_work_sync(&dcp->placeholder_edid_wq); + disable_delayed_work_sync(&dcp->typec_relink_wq); disable_delayed_work_sync(&dcp->hdmi_settle_wq); disable_delayed_work_sync(&dcp->hdmi_recheck_wq); dcp_hdmi_disable(dcp); @@ -2068,8 +2280,11 @@ static void dcp_disable_typec_work(struct apple_dcp *dcp, bool release_cable) static void dcp_enable_typec_work(struct apple_dcp *dcp) { + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_enable_typec_work_h17p(dcp); enable_delayed_work(&dcp->typec_reconnect_wq); enable_delayed_work(&dcp->placeholder_edid_wq); + enable_delayed_work(&dcp->typec_relink_wq); enable_delayed_work(&dcp->hdmi_settle_wq); enable_delayed_work(&dcp->hdmi_recheck_wq); dcp_hdmi_enable(dcp); @@ -2285,6 +2500,9 @@ static int dcp_comp_bind(struct device *dev, struct device *main, void *data) struct apple_dcp *dcp = dev_get_drvdata(dev); int ret; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_comp_bind_h17p(dev, main, data); + if (READ_ONCE(dcp->ext_dead)) return 0; ret = dcp_comp_bind_pipe(dev, main, data); @@ -2303,6 +2521,8 @@ static void dcp_comp_unbind(struct device *dev, struct device *main, void *data) { struct apple_dcp *dcp = dev_get_drvdata(dev); + if (dcp && dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_comp_unbind_h17p(dev, main, data); if (!dcp || READ_ONCE(dcp->ext_dead)) return; @@ -2453,9 +2673,6 @@ static int dcp_native_routes_error(struct apple_dcp *dcp, int err) static int dcp_platform_probe_pipe(struct platform_device *pdev) { - if (of_machine_is_compatible("apple,t8140")) - return -ENODEV; - enum dcp_firmware_version fw_compat; struct device *dev = &pdev->dev; struct apple_dcp *dcp; @@ -2500,7 +2717,7 @@ static int dcp_platform_probe_pipe(struct platform_device *pdev) mutex_init(&dcp->tb_lock); dcp_fabric_init(dcp); spin_lock_init(&dcp->mode_state.lock); - spin_lock_init(&dcp->dcpavserv.lock); + mutex_init(&dcp->dcpavserv.lock); dcp->hw = *(struct apple_dcp_hw_data *)of_device_get_match_data(dev); dcp->fixed_connector_type = dcp_connector_type_from_dt(dev->of_node); dcp->connector_type = dcp->fixed_connector_type; @@ -2513,9 +2730,11 @@ static int dcp_platform_probe_pipe(struct platform_device *pdev) dcp_typec_reconnect_work); INIT_DELAYED_WORK(&dcp->placeholder_edid_wq, dcp_placeholder_edid_work); + INIT_DELAYED_WORK(&dcp->typec_relink_wq, dcp_typec_relink_work); /* Balanced by enable at successful component bind. */ disable_delayed_work(&dcp->typec_reconnect_wq); disable_delayed_work(&dcp->placeholder_edid_wq); + disable_delayed_work(&dcp->typec_relink_wq); /* * A native external pipe's bind enables these; firmware callbacks may * come before it. The 14.7 panel never uses them, but suspend cancels @@ -2802,13 +3021,35 @@ static int dcp_ext_route_suppliers(struct device *dev) * deferring while its suppliers may still appear (bounded by the deferred * probe timeout); on any other failure join as an unusable pipe. */ +static void dcp_unuse_dma_domain(void *data) +{ + iommu_device_unuse_default_domain(data); +} + static int dcp_platform_probe(struct platform_device *pdev) { + const struct apple_dcp_hw_data *hw = of_device_get_match_data(&pdev->dev); struct device *dev = &pdev->dev; struct apple_dcp *dcp; void *group; int ret; + if (hw && hw->firmware_compat == DCP_FIRMWARE_H17P) + return dcp_platform_probe_h17p(pdev); + + /* + * The driver manages DMA ownership itself (driver_managed_dma), so + * that a retained H17P session keeps its domain use across unbind. + * Every other processor uses its default domain while bound, as the + * driver core would. + */ + ret = iommu_device_use_default_domain(dev); + if (ret) + return ret; + ret = devm_add_action_or_reset(dev, dcp_unuse_dma_domain, dev); + if (ret) + return ret; + if (!of_device_is_compatible(dev->of_node, "apple,t8122-dcpext")) return dcp_platform_probe_pipe(pdev); @@ -2872,6 +3113,9 @@ static void dcp_platform_shutdown(struct platform_device *pdev) { struct apple_dcp *dcp = platform_get_drvdata(pdev); + if (dcp && dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_platform_shutdown_h17p(pdev); + if (dcp && dcp->external && !dcp->external_native) return; component_del(&pdev->dev, &dcp_comp_ops); @@ -2955,6 +3199,8 @@ static int dcp_platform_suspend(struct device *dev) { struct apple_dcp *dcp = dev_get_drvdata(dev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_platform_suspend_h17p(dev); if (READ_ONCE(dcp->ext_dead)) return 0; @@ -3011,6 +3257,8 @@ static int dcp_platform_resume(struct device *dev) { struct apple_dcp *dcp = dev_get_drvdata(dev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_platform_resume_h17p(dev); if (READ_ONCE(dcp->ext_dead)) return 0; @@ -3028,54 +3276,2485 @@ static int dcp_platform_resume(struct device *dev) return 0; } -static const struct dev_pm_ops dcp_platform_pm_ops = { - .prepare = pm_sleep_ptr(dcp_platform_prepare), - .complete = pm_sleep_ptr(dcp_platform_complete), - SYSTEM_SLEEP_PM_OPS(dcp_platform_suspend, dcp_platform_resume) -}; +/* --------------------------------------------------------------------------- + * H17-generation firmware: T8140 (H17P, internal and the external dispext0) + * and T8142 (H17G). aurora-wip's processor lifecycle. + * ------------------------------------------------------------------------- */ -static const struct apple_dcp_hw_data apple_dcp_hw_t6020 = { - .num_dptx_ports = 1, - .t6020_tunnel_flow = true, +struct dcp_session_ref { + struct list_head link; + struct device *dev; + struct iommu_group *group; }; -static const struct apple_dcp_hw_data apple_dcp_hw_t6020_dcpext = { - .num_dptx_ports = 2, - .t6020_tunnel_flow = true, -}; +static LIST_HEAD(dcp_retained_sessions); +static DEFINE_MUTEX(dcp_retained_sessions_lock); -static const struct apple_dcp_hw_data apple_dcp_hw_t8112 = { - .num_dptx_ports = 2, -}; +static bool dcp_session_retained(struct device *dev) +{ + struct dcp_session_ref *session; + bool retained = false; -static const struct apple_dcp_hw_data apple_dcp_hw_dcp = { - .num_dptx_ports = 0, -}; + mutex_lock(&dcp_retained_sessions_lock); + list_for_each_entry(session, &dcp_retained_sessions, link) { + if (dev_fwnode(session->dev) == dev_fwnode(dev)) { + retained = true; + break; + } + } + mutex_unlock(&dcp_retained_sessions_lock); + return retained; +} -static const struct apple_dcp_hw_data apple_dcp_hw_t6030 = { - .num_dptx_ports = 1, -}; +static int dcp_pin_live_session(struct apple_dcp *dcp) +{ + struct dcp_session_ref *session; -static const struct apple_dcp_hw_data apple_dcp_hw_t8122 = { - .num_dptx_ports = 2, -}; + if ((!dcp->hw.adopt_live_session && + dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) || + dcp->fw_compat != DCP_FIRMWARE_H17P || + dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G || dcp->retain_dma) + return 0; + session = kzalloc_obj(*session); + if (!session) + return -ENOMEM; -/* The internal panel only, through the 14.x IOMFB, as on T6030. */ -static const struct apple_dcp_hw_data apple_dcp_hw_t6031 = { - .num_dptx_ports = 1, -}; + /* Keep the DMA device, domain usage and driver code valid until reset. */ + session->dev = get_device(dcp->dev); + session->group = iommu_group_get(dcp->dev); + __module_get(THIS_MODULE); + WRITE_ONCE(dcp->retain_dma, true); + mutex_lock(&dcp_retained_sessions_lock); + list_add_tail(&session->link, &dcp_retained_sessions); + mutex_unlock(&dcp_retained_sessions_lock); + dev_info(dcp->dev, "live H17P DMA resources retained until reboot; rebind unsupported\n"); + return 0; +} -static const struct apple_dcp_hw_data apple_dcp_hw_t6030_dcpext = { - .num_dptx_ports = 1, -}; +static void dcp_release_dma_domain(void *data) +{ + struct apple_dcp *dcp = data; -static const struct apple_dcp_hw_data apple_dcp_hw_t8122_dcpext = { - .num_dptx_ports = 2, -}; + if (!dcp->retain_dma) + iommu_device_unuse_default_domain(dcp->dev); +} -static const struct apple_dcp_hw_data apple_dcp_hw_dcpext = { - .num_dptx_ports = 2, +static void dcp_release_context(void *data) +{ + struct apple_dcp *dcp = data; + + if (!dcp->retain_dma) + kfree(dcp); +} + +static void dcp_release_rtkit(void *data) +{ + struct apple_dcp *dcp = data; + + if (dcp->retain_dma) { + WRITE_ONCE(dcp->quiescing, true); + apple_rtkit_free_retaining_buffers(dcp->rtk); + } else { + apple_rtkit_free(dcp->rtk); + } + dcp->rtk = NULL; +} + +bool dcp_h17p_external_supported(struct device_node *np) +{ + u32 firmware[DCP_EXTERNAL_FW_CELLS], compat[DCP_EXTERNAL_FW_CELLS]; + int firmware_cells, compat_cells; + + if (!of_machine_is_compatible("apple,j700") || + !of_device_is_compatible(np, "apple,t8140-dcpext")) + return false; + firmware_cells = of_property_read_variable_u32_array( + np, "apple,firmware-version", firmware, + DCP_EXTERNAL_FW_CELLS, DCP_EXTERNAL_FW_CELLS); + compat_cells = of_property_read_variable_u32_array( + np, "apple,firmware-compat", compat, + DCP_EXTERNAL_FW_CELLS, DCP_EXTERNAL_FW_CELLS); + return apple_dcp_external_firmware_supported(firmware, firmware_cells, + compat, compat_cells); +} + +bool dcp_h17p_external_ready(struct device_node *np) +{ + struct platform_device *pdev; + struct apple_dcp *dcp; + bool ready = false; + + if (!dcp_h17p_external_supported(np)) + return false; + pdev = of_find_device_by_node(np); + if (!pdev) + return false; + /* + * Called after component binding. component_add() may synchronously + * bind the master before the platform probe itself returns, so the + * driver core's bound flag is not a readiness condition here. + */ + dcp = platform_get_drvdata(pdev); + ready = dcp && READ_ONCE(dcp->external_prepared) && + !READ_ONCE(dcp->quiescing) && !READ_ONCE(dcp->crashed); + put_device(&pdev->dev); + return ready; +} + +static bool dcp_firmware_metadata_enabled(struct apple_dcp *dcp) +{ + return dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL; +} + +void dcp_firmware_metadata_connection(struct apple_dcp *dcp, u64 generation, bool connected) +{ + unsigned long flags; + + if (!dcp_firmware_metadata_enabled(dcp)) + return; + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (generation != dcp->metadata.epoch.generation || !connected) { + dcp->rate_proof.connection_sequence++; + dcp->rate_proof.sequence++; + dcp->rate_proof.rate.valid = false; + dcp->rate_proof.lanes.valid = false; + dcp->metadata.attributes_valid = false; + dcp->metadata.timing_valid = false; + dcp->metadata.hpd_seen = false; + } + dcp_metadata_epoch_connection(&dcp->metadata.epoch, generation, connected); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); +} + +void dcp_firmware_metadata_hotplug(struct apple_dcp *dcp, bool connected) +{ + struct apple_connector *con = READ_ONCE(dcp->connector); + unsigned long flags; + + if (!dcp_firmware_metadata_enabled(dcp)) + return; + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (connected) { + dcp->metadata.hpd_seen = true; + /* + * Fresh proof may precede HPD assertion after a same-cable loss. + * Keep invalidated firmware metadata and old receipt tickets revoked. + */ + if (READ_ONCE(dcp->typec_cable_connected) && + dcp->metadata.epoch.generation == READ_ONCE(dcp->typec_generation)) + dcp->metadata.epoch.connected = true; + } else if (dcp->metadata.hpd_seen) { + dcp->metadata.hpd_seen = false; + dcp->rate_proof.connection_sequence++; + dcp->rate_proof.sequence++; + dcp->rate_proof.rate.valid = false; + dcp->rate_proof.lanes.valid = false; + if (con && READ_ONCE(con->dcp) == to_platform_device(dcp->dev)) + WRITE_ONCE(con->edid_invalidation_pending, true); + dcp_metadata_epoch_connection(&dcp->metadata.epoch, + dcp->metadata.epoch.generation, false); + dcp->metadata.attributes_valid = false; + dcp->metadata.timing_valid = false; + } + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (connected) + dcp_180hz_notify(dcp); +} + +void dcp_firmware_metadata_begin(struct apple_dcp *dcp) +{ + unsigned long flags; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + dcp->chunks.metadata_ticket = dcp_metadata_begin(&dcp->metadata.epoch); + dcp->chunks.received = 0; + dcp->chunks.contiguous = true; + dcp->chunks.payload_complete = false; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); +} + +static void dcp_firmware_metadata_notify(struct apple_dcp *dcp) +{ + unsigned long flags; + bool ready; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + ready = dcp->metadata.attributes_valid && dcp->metadata.timing_valid && + dcp_metadata_matches(&dcp->metadata.epoch, dcp->metadata.attributes_ticket) && + dcp_metadata_matches(&dcp->metadata.epoch, dcp->metadata.timing_ticket); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (ready && dcp->connector && READ_ONCE(dcp->connector->connected)) + schedule_work(&dcp->connector->hotplug_wq); +} + +static bool dcp_firmware_metadata_complete(struct apple_dcp *dcp) +{ + return dcp->chunks.contiguous && dcp->chunks.payload_complete && + dcp_metadata_matches(&dcp->metadata.epoch, dcp->chunks.metadata_ticket); +} + +void dcp_firmware_metadata_attributes(struct apple_dcp *dcp, + const struct dcp_display_metadata *attributes) +{ + unsigned long flags; + + if (!dcp_firmware_metadata_enabled(dcp)) + return; + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (dcp_metadata_matches(&dcp->metadata.epoch, dcp->chunks.metadata_ticket)) { + dcp->metadata.attributes_valid = attributes && dcp_firmware_metadata_complete(dcp); + if (dcp->metadata.attributes_valid) { + dcp->metadata.attributes = *attributes; + dcp->metadata.attributes_ticket = dcp->chunks.metadata_ticket; + } + } + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + dcp_firmware_metadata_notify(dcp); +} + +void dcp_firmware_metadata_timing(struct apple_dcp *dcp) +{ + unsigned long flags; + unsigned int i; + + if (!dcp_firmware_metadata_enabled(dcp)) + return; + spin_lock_irqsave(&dcp->metadata.lock, flags); + if (!dcp_metadata_matches(&dcp->metadata.epoch, dcp->chunks.metadata_ticket)) + goto unlock; + dcp->metadata.timing_valid = false; + if (!dcp_firmware_metadata_complete(dcp)) + goto unlock; + for (i = 0; i < dcp->nr_modes; i++) { + if (!(dcp->modes[i].mode.type & DRM_MODE_TYPE_PREFERRED) || + !dcp->modes[i].metadata_timing_valid) + continue; + dcp->metadata.preferred = dcp->modes[i].mode; + dcp->metadata.timing_ticket = dcp->chunks.metadata_ticket; + dcp->metadata.timing_valid = true; + break; + } +unlock: + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + dcp_firmware_metadata_notify(dcp); +} + +bool dcp_firmware_metadata_current(struct apple_dcp *dcp, u64 generation) +{ + unsigned long flags; + bool valid; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + valid = !dcp->metadata.epoch.invalidated && dcp->metadata.epoch.connected && + generation == dcp->metadata.epoch.generation; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + return valid; +} + +const struct drm_edid *dcp_firmware_edid(struct apple_dcp *dcp, u64 *generation) +{ + struct dcp_display_metadata attributes; + struct drm_display_mode preferred; + const struct drm_edid *edid; + unsigned long flags; + bool valid; + + if (!dcp_firmware_metadata_enabled(dcp)) + return ERR_PTR(-ENODEV); + spin_lock_irqsave(&dcp->metadata.lock, flags); + valid = dcp->metadata.attributes_valid && dcp->metadata.timing_valid && + dcp_metadata_matches(&dcp->metadata.epoch, dcp->metadata.attributes_ticket) && + dcp_metadata_matches(&dcp->metadata.epoch, dcp->metadata.timing_ticket); + if (valid) { + attributes = dcp->metadata.attributes; + preferred = dcp->metadata.preferred; + *generation = dcp->metadata.epoch.generation; + } + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (!valid) + return ERR_PTR(-ENODATA); + edid = dcp_metadata_edid(&attributes, &preferred); + spin_lock_irqsave(&dcp->metadata.lock, flags); + valid = !dcp->metadata.epoch.invalidated && dcp->metadata.epoch.connected && + *generation == dcp->metadata.epoch.generation; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (!valid && !IS_ERR(edid)) { + drm_edid_free(edid); + return ERR_PTR(-ESTALE); + } + return edid; +} + +/* + * H17P: deferred to the connector hotplug work, which updates the connector + * with dcp_update_connector_dimensions(). + */ +void dcp_set_dimensions_h17p(struct apple_dcp *dcp) +{ + int i; + int width_mm = dcp->width_mm; + int height_mm = dcp->height_mm; + + if (width_mm == 0 || height_mm == 0) { + width_mm = dcp->panel.width_mm; + height_mm = dcp->panel.height_mm; + } + + /* + * This callback runs on the ordered RTKit RX worker. A mode probe + * can hold mode_config.mutex while waiting for an AFK reply which + * that same worker still has to dispatch. Defer the DRM update. + */ + if (dcp->connector && !READ_ONCE(dcp->quiescing)) + schedule_work(&dcp->connector->hotplug_wq); + + /* + * Fix up any probed modes. Modes are created when parsing + * TimingElements, dimensions are calculated when parsing + * DisplayAttributes, and TimingElements may be sent first + */ + for (i = 0; i < dcp->nr_modes; ++i) { + dcp->modes[i].mode.width_mm = width_mm; + dcp->modes[i].mode.height_mm = height_mm; + } +} + +void dcp_update_connector_dimensions(struct apple_dcp *dcp, + struct drm_connector *connector) +{ + int width_mm = READ_ONCE(dcp->width_mm); + int height_mm = READ_ONCE(dcp->height_mm); + + lockdep_assert_held(&connector->dev->mode_config.mutex); + if (!width_mm || !height_mm) { + width_mm = dcp->panel.width_mm; + height_mm = dcp->panel.height_mm; + } + connector->display_info.width_mm = width_mm; + connector->display_info.height_mm = height_mm; +} + + +static struct apple_dcp_afkep *dcp_afkep(struct apple_dcp *dcp, u8 endpoint) +{ + /* DPAV 0x2a on H17P overlaps the legacy DPTX endpoint number. */ + if (dcp->dcpavservep && endpoint == dcp->dcpavservep->endpoint) + return dcp->dcpavservep; + if (dcp->dptxep && endpoint == dcp->dptxep->endpoint) + return dcp->dptxep; + switch (endpoint) { + case AV_ENDPOINT: + return dcp->avep; + case SYSTEM_ENDPOINT: + return dcp->systemep; + case DISP0_ENDPOINT: + return dcp->ibootep; + case DPTX_CONTROLLER_ENDPOINT: + return dcp->dptx_controller_ep; + default: + return NULL; + } +} + + +static void dcp_recv_msg_h17p(struct apple_dcp *dcp, u8 endpoint, u64 message) +{ + struct apple_dcp_afkep *ep; + + if (READ_ONCE(dcp->quiescing)) + return; + + trace_dcp_recv_msg(dcp, endpoint, message); + + /* + * H17P starts endpoints that have no AFK instance behind them (the + * remote allocator endpoint, for one). Log their messages rather than + * dereferencing a NULL afkep or warning about an unknown endpoint. + */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P && endpoint != IOMFB_ENDPOINT && + !dcp_afkep(dcp, endpoint)) { + dev_dbg_ratelimited(dcp->dev, "ep %#04x: %#llx\n", endpoint, + message); + return; + } + + if (endpoint == IOMFB_ENDPOINT) + return iomfb_recv_msg(dcp, message); + + ep = dcp_afkep(dcp, endpoint); + + if (!ep) { + dev_warn_ratelimited(dcp->dev, + "dropping message for unhandled endpoint %#x\n", + endpoint); + return; + } + + afk_receive_message(ep, message); +} + +static void dcp_rtk_crashed_h17p(struct apple_dcp *dcp) +{ + if (READ_ONCE(dcp->quiescing)) + return; + + dcp->crashed = true; + dev_err(dcp->dev, "DCP has crashed\n"); + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + /* No in-place restart exists; the last scanout stays retained. */ + dev_err(dcp->dev, "a reboot is required to restore the display\n"); + /* + * Nothing will complete a present that was in flight. Before + * the pipeline is linked there is no event to complete. + */ + if (READ_ONCE(dcp->crtc)) + schedule_work(&dcp->vblank_wq); + } + if (dcp->connector) { + dcp->connector->connected = 0; + /* The cached EDID is owned under edid_lock. */ + apple_connector_invalidate_edid(dcp->connector); + schedule_work(&dcp->connector->hotplug_wq); + } + complete(&dcp->start_done); +} + +/* + * Check that every IOMMU page of [iova, iova + size) translates to the same + * offset from @phys as it has from @iova. + */ +static int dcp_check_iova_contiguous(struct iommu_domain *domain, + dma_addr_t iova, size_t size, + phys_addr_t phys) +{ + unsigned long pgsize = domain->pgsize_bitmap ? + 1UL << __ffs(domain->pgsize_bitmap) : PAGE_SIZE; + dma_addr_t end = iova + size; + dma_addr_t addr = iova; + + while (addr < end) { + if (iommu_iova_to_phys(domain, addr) != phys + (addr - iova)) + return -EINVAL; + addr = ALIGN_DOWN(addr, pgsize) + pgsize; + } + + return 0; +} + +static int dcp_rtk_shmem_setup_h17p(struct apple_dcp *dcp, struct apple_rtkit_shmem *bfr) +{ + int ret; + + if (bfr->iova) { + struct iommu_domain *domain = + iommu_get_domain_for_dev(dcp->dev); + phys_addr_t phy_addr; + + if (!domain) + return -ENOMEM; + + // TODO: get map from device-tree + phy_addr = iommu_iova_to_phys(domain, bfr->iova); + if (!phy_addr) + return -ENOMEM; + + /* memremap() below needs the whole buffer physically contiguous */ + ret = dcp_check_iova_contiguous(domain, bfr->iova, bfr->size, + phy_addr); + if (ret) { + dev_err(dcp->dev, + "shmem_setup: iova %pad (%#zx bytes) is not physically contiguous\n", + &bfr->iova, bfr->size); + return ret; + } + + /* Firmware writes without CPU cache maintenance. */ + bfr->buffer = memremap(phy_addr, bfr->size, MEMREMAP_WC); + if (!bfr->buffer) + return -ENOMEM; + + bfr->is_mapped = true; + dev_info(dcp->dev, + "shmem_setup: iova: %lx -> pa: %lx -> iomem: %lx\n", + (uintptr_t)bfr->iova, (uintptr_t)phy_addr, + (uintptr_t)bfr->buffer); + } else { + bfr->buffer = dma_alloc_coherent(dcp->dev, bfr->size, + &bfr->iova, GFP_KERNEL); + if (!bfr->buffer) + return -ENOMEM; + + dev_info(dcp->dev, "shmem_setup: iova: %lx, buffer: %lx\n", + (uintptr_t)bfr->iova, (uintptr_t)bfr->buffer); + } + + return 0; +} + +static const struct apple_rtkit_ops external_h17p_rtkit_ops = { + .flags = APPLE_RTKIT_COPROC_PLACES_BUFFERS, + .crashed = dcp_rtk_crashed, + .recv_message = dcp_recv_msg, + .shmem_setup = dcp_rtk_shmem_setup, + .shmem_destroy = dcp_rtk_shmem_destroy, +}; + +static int dcp_crtc_atomic_check_h17p(struct drm_crtc *crtc, struct drm_atomic_state *state) +{ + struct platform_device *pdev = to_apple_crtc(crtc)->dcp; + struct apple_dcp *dcp = platform_get_drvdata(pdev); + struct drm_crtc_state *crtc_state; + bool needs_modeset; + + if (dcp->crashed) + return -EINVAL; + + crtc_state = drm_atomic_get_new_crtc_state(state, crtc); + + /* External H17P does not program adaptive sync. */ + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + crtc_state->vrr_enabled) + return -EINVAL; + + needs_modeset = drm_atomic_crtc_needs_modeset(crtc_state) || + !READ_ONCE(dcp->mode_state.valid); + if (!needs_modeset && (!dcp->connector || !dcp->connector->connected)) { + /* + * Resume restores the mode before the firmware reports the + * display back, so a plane-only commit lands here while the + * connector is still marked disconnected. Rejecting it makes + * the compositor fail every flip and give up on the output; + * dcp_flush() defers the commit until the link returns. + */ + dev_dbg(dcp->dev, + "crtc_atomic_check: deferring commit, link still down\n"); + } + + return 0; +} + +static int dcp_dptx_connect_raw(struct apple_dcp *dcp, u32 port, bool *waiting_service) +{ + unsigned long timeout; + int ret = 0; + + if (waiting_service) + *waiting_service = false; + if (!dcp->phy) { + dev_warn(dcp->dev, "dcp_dptx_connect: missing phy\n"); + return -ENODEV; + } + dev_dbg(dcp->dev, + "%s(port=%d) target=%u:%u typec=%d route=%s conn_type=%d connected=%d\n", + __func__, port, dcp->dptx_die, dcp->dptx_phy, + dcp_is_typec_output(dcp), + dcp->active_typec_route ? "borrowed" : "fixed", + dcp->connector_type, dcp->dptxport[port].connected); + + mutex_lock(&dcp->hpd_mutex); + if (!dptx_service_available(&dcp->dptxport[port])) { + dev_warn(dcp->dev, "dcp_dptx_connect: dptx service for port %d not enabled\n", port); + ret = -ENODEV; + /* No firmware request has been made; the first publication can retry. */ + if (waiting_service && !READ_ONCE(dcp->dptxport[port].closed)) { + *waiting_service = true; + ret = -EAGAIN; + } + goto out_unlock; + } + + if (dcp->dptxport[port].connected) + goto out_unlock; + + reinit_completion(&dcp->dptxport[port].linkcfg_completion); + dcp->dptxport[port].atcphy = dcp->phy; + ret = dptxport_validate_connection(dcp->dptxport[port].service, + dcp->dptx_dfp_port, + dcp->dptx_phy, dcp->dptx_die); + if (ret) { + dev_err(dcp->dev, + "dcp_dptx_connect: failed to validate DPTX target %u:%u: %d\n", + dcp->dptx_die, dcp->dptx_phy, ret); + goto out_unlock; + } + + ret = dptxport_connect(dcp->dptxport[port].service, + dcp->dptx_dfp_port, + dcp->dptx_phy, dcp->dptx_die, + dcp_is_typec_output(dcp)); + if (ret) { + dev_err(dcp->dev, + "dcp_dptx_connect: failed to connect DPTX target %u:%u: %d\n", + dcp->dptx_die, dcp->dptx_phy, ret); + goto out_unlock; + } + + ret = dptxport_request_display(dcp->dptxport[port].service); + if (ret) { + dev_err(dcp->dev, + "dcp_dptx_connect: failed to request display: %d\n", + ret); + goto out_release; + } + dcp->dptxport[port].connected = true; + /* Not a T602x USB4 tunnel: the plain HPD assertion and timeout apply. */ + if (dcp_is_typec_output(dcp)) { + ret = dptxport_set_hpd(dcp->dptxport[port].service, true); + if (ret) { + dev_err(dcp->dev, + "dcp_dptx_connect: failed to assert Type-C HPD: %d\n", + ret); + dcp->dptxport[port].connected = false; + goto out_release; + } + } + + mutex_unlock(&dcp->hpd_mutex); + timeout = DPTX_CONNECT_TIMEOUT; + ret = wait_for_completion_timeout(&dcp->dptxport[port].linkcfg_completion, + timeout); + if (!ret) { + dev_err(dcp->dev, + "dcp_dptx_connect: timed out waiting for port %u link configuration\n", + port); + ret = -ETIMEDOUT; + goto out_disconnect; + } + + dev_dbg(dcp->dev, "dcp_dptx_connect: waited %d ms for link\n", + jiffies_to_msecs(timeout - ret)); + + usleep_range(5, 10); + + if (dcp->connector_type == DRM_MODE_CONNECTOR_DisplayPort) + dptxport_set_hpd(dcp->dptxport[port].service, true); + + if (dcp->avep) + av_service_connect(dcp); + + return 0; + +out_disconnect: + mutex_lock(&dcp->hpd_mutex); + dcp->dptxport[port].connected = false; +out_release: + dptxport_release_display(dcp->dptxport[port].service); + +out_unlock: + mutex_unlock(&dcp->hpd_mutex); + return ret; +} + +static void dcp_external_report_disconnected(struct apple_dcp *dcp) +{ + unsigned int action; + + guard(mutex)(&dcp->iomfb.lock); + /* A paused callback from the old sink must not be replayed on wake. */ + dcp->external_hotplug_pending = false; + action = dcp_mode_hotplug(&dcp->mode_state, false, + dcp->connector ? &dcp->connector->connected : NULL); + if (dcp->connector) { + WRITE_ONCE(dcp->connector->edid_invalidation_pending, true); + action |= DCP_HOTPLUG_NOTIFY; + } + /* The existing worker delivers DRM notifications outside our locks. */ + dcp_handle_hotplug_actions(dcp, action); +} + +static void dcp_external_publish_generation(struct apple_dcp *dcp) +{ + lockdep_assert_held(&dcp->external_lock); + guard(mutex)(&dcp->iomfb.lock); + dcp->external_rx_generation = dcp->external_generation; +} + +/* external_lock is never taken by the serialized IOMFB receive callbacks. */ +static int dcp_external_link_locked(struct apple_dcp *dcp) +{ + bool waiting_service; + int ret; + + lockdep_assert_held(&dcp->external_lock); + if (!dcp->external_prepared) + return -EAGAIN; + if (dcp->external_error) + return dcp->external_error; + if (!dcp->external_cable) + return -ENODEV; + ret = dcp_dptx_connect_raw(dcp, 0, &waiting_service); + if (ret && !waiting_service) + dcp->external_error = ret; + return ret; +} + +static int dcp_external_start_locked(struct apple_dcp *dcp) +{ + int ret; + + lockdep_assert_held(&dcp->external_lock); + ret = dcp_external_link_locked(dcp); + if (ret || dcp->external_running) + return ret; + iomfb_pause_presentation(dcp, true); + /* Even a timed-out first activation can have changed firmware state. */ + dcp->external_power_attempted = true; + ret = iomfb_external_power_h17p(dcp, true); + if (ret) { + dcp->external_error = ret; + return ret; + } + dcp->external_running = true; + dcp->external_stopped = false; + WRITE_ONCE(dcp->pipe_enabled_h17p, true); + WRITE_ONCE(dcp->typec_crtc_off, false); + iomfb_pause_presentation(dcp, false); + return 0; +} + +static int dcp_dptx_connect_h17p(struct apple_dcp *dcp, u32 port) +{ + if (!dcp_is_external_h17p(dcp)) + return dcp_dptx_connect_raw(dcp, port, NULL); + guard(mutex)(&dcp->external_lock); + return dcp_external_link_locked(dcp); +} + +/* The generation token belongs to the CRTC that was actually enabled. */ +static int dcp_external_stop_locked(struct apple_dcp *dcp, u64 generation) +{ + int ret; + + lockdep_assert_held(&dcp->external_lock); + if (generation != dcp->external_generation) + return 0; + if (dcp->external_error) + return dcp->external_error; + if (dcp->external_stopped && !READ_ONCE(dcp->dptxport[0].connected)) + return 0; + iomfb_pause_presentation(dcp, true); + dcp->external_running = false; + /* Route setup can call this before IOMFB or its shared memory exists. */ + if (dcp->external_stopped || !dcp->external_power_attempted) + goto release_link; + ret = iomfb_queue_settle(dcp, msecs_to_jiffies(500)); + if (!ret) + ret = iomfb_external_power_h17p(dcp, false); + if (!ret) + ret = iomfb_queue_settle(dcp, msecs_to_jiffies(500)); + if (ret) + goto failed; + /* This is a transport acknowledgment, not permission to free scanout. */ + WRITE_ONCE(dcp->pipe_enabled_h17p, false); +release_link: + scoped_guard(mutex, &dcp->hpd_mutex) { + if (dcp->dptxport[0].connected) { + if (!dptx_service_available(&dcp->dptxport[0])) { + ret = -ENODEV; + goto failed; + } + ret = dptxport_set_hpd(dcp->dptxport[0].service, false); + if (!ret) + ret = dptxport_release_display(dcp->dptxport[0].service); + if (ret) + goto failed; + dcp->dptxport[0].connected = false; + } + } + dcp->external_stopped = true; + WRITE_ONCE(dcp->typec_crtc_off, true); + if (dcp->avep) + av_service_disconnect(dcp); + return 0; +failed: + /* A retry could race a late reply. Keep this session blocked and owned. */ + dcp->external_error = ret; + return ret; +} + +static int dcp_external_disconnect(struct apple_dcp *dcp, bool physical) +{ + int ret; + + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + guard(mutex)(&dcp->external_lock); + /* Invalidate physical EDID before waiting for firmware power/link teardown. */ + if (physical) { + WRITE_ONCE(dcp->external_physically_absent, true); + scoped_guard(mutex, &dcp->hpd_mutex) { + WRITE_ONCE(dcp->typec_cable_connected, false); + dcp->typec_generation++; + dcp_firmware_metadata_connection(dcp, dcp->typec_generation, false); + } + /* HPD is fenced; retire the AV owner after releasing hpd_mutex. */ + dpavservep_detach(dcp); + } + ret = dcp_external_stop_locked(dcp, dcp->external_generation); + if (physical) { + dcp->external_cable = false; + dcp->external_generation++; + dcp_external_publish_generation(dcp); + /* A missing cable is reported even when its hardware remains owned. */ + dcp_external_report_disconnected(dcp); + } + return ret; +} + +static void dcp_external_stop_work(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(work, struct apple_dcp, external_stop_work); + + u64 generation; + + scoped_guard(mutex, &dcp->iomfb.lock) + generation = dcp->external_stop_generation; + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + guard(mutex)(&dcp->external_lock); + if (generation == dcp->external_generation) { + /* The deferred disconnect also covers missing out-of-band HPD. */ + dpavservep_detach(dcp); + dcp_external_stop_locked(dcp, generation); + dcp_external_report_disconnected(dcp); + } +} + +bool dcp_external_hotplug(struct apple_dcp *dcp, bool connected) +{ + if (!dcp_is_external_h17p(dcp)) + return false; + /* No lifecycle or HPD mutex may be acquired from the RX callback. */ + lockdep_assert_held(&dcp->iomfb.lock); + if (READ_ONCE(dcp->quiescing) || READ_ONCE(dcp->external_physically_absent)) + return true; + if (dcp->iomfb.presentation_paused) { + dcp->external_hotplug_pending = true; + dcp->external_hotplug_connected = connected; + /* Publish initial/new-cable discovery, never same-cable DPMS wake. */ + if (connected && READ_ONCE(dcp->external_cable) && + !READ_ONCE(dcp->external_error) && + (!READ_ONCE(dcp->external_power_attempted) || + (READ_ONCE(dcp->external_stopped) && + READ_ONCE(dcp->external_rx_generation) != + READ_ONCE(dcp->external_crtc_generation) && + READ_ONCE(dcp->dptxport[0].connected)))) { + unsigned int action = dcp_mode_hotplug(&dcp->mode_state, true, + dcp->connector ? + &dcp->connector->connected : NULL); + + dcp_handle_hotplug_actions(dcp, action); + } + return true; + } + if (!connected && !READ_ONCE(dcp->iomfb.presentation_paused)) { + /* Fence admission in RX before a busy actor can delay teardown. */ + dcp_firmware_metadata_hotplug(dcp, false); + dcp->external_stop_generation = dcp->external_rx_generation; + schedule_work(&dcp->external_stop_work); + return true; + } + return READ_ONCE(dcp->iomfb.presentation_paused); +} + + +static void dcp_placeholder_edid_work_h17p(struct work_struct *work) +{ + struct apple_dcp *dcp = + container_of(to_delayed_work(work), struct apple_dcp, + placeholder_edid_wq); + struct apple_epic_service *service; + u64 generation; + int ret; + + mutex_lock(&dcp->hpd_mutex); + generation = dcp->placeholder_generation; + if (!dcp->typec_cable_connected || !dcp->dptxport[0].connected || + !dptx_service_available(&dcp->dptxport[0]) || generation != dcp->typec_generation) + goto out_unlock; + service = dcp->dptxport[0].service; + + /* + * Some adapters answer the first connection with a 1024x768 + * placeholder and publish the panel EDID only after HPD drops + * and returns. One pulse; a second placeholder is left alone. + */ + ret = dptxport_set_hpd(service, false); + if (ret) { + dev_info(dcp->dev, "placeholder EDID: HPD drop failed: %d\n", + ret); + goto out_unlock; + } + mutex_unlock(&dcp->hpd_mutex); + + msleep(1000); + + mutex_lock(&dcp->hpd_mutex); + if (!dcp->typec_cable_connected || !dcp->dptxport[0].connected || + generation != dcp->typec_generation) + goto out_unlock; + + ret = dptxport_set_hpd(service, true); + if (ret) + dev_info(dcp->dev, "placeholder EDID: HPD assert failed: %d\n", + ret); +out_unlock: + mutex_unlock(&dcp->hpd_mutex); +} + +static void dcp_typec_reconnect_work_h17p(struct work_struct *work) +{ + struct apple_dcp *dcp = + container_of(to_delayed_work(work), struct apple_dcp, + typec_reconnect_wq); + int ret; + + if (!READ_ONCE(dcp->typec_cable_connected)) + return; + + ret = dcp_dptx_connect(dcp, 0); + if (!ret) { + dcp->typec_reconnect_tries = 0; + return; + } + + /* Not a T602x USB4 tunnel: the full retry budget applies. */ + if (++dcp->typec_reconnect_tries < DPTX_RECONNECT_RETRIES) { + mod_delayed_work(system_freezable_wq, &dcp->typec_reconnect_wq, + DPTX_RECONNECT_DELAY); + return; + } + + dev_err(dcp->dev, "Type-C DPTX reconnect failed after %u retries: %d\n", + dcp->typec_reconnect_tries, ret); +} + +static void disconnected_hpd_event_h17p(struct apple_connector *con) +{ + if (!con) + return; + + /* May run with DRM modeset locks held; defer property/cache cleanup. */ + WRITE_ONCE(con->connected, false); + WRITE_ONCE(con->edid_invalidation_pending, true); + schedule_work(&con->hotplug_wq); +} + +static int dcp_dptx_disconnect_h17p(struct apple_dcp *dcp, u32 port) +{ + if (dcp_is_external_h17p(dcp)) + return dcp_external_disconnect(dcp, false); + + dev_info(dcp->dev, "%s(port=%d)\n", __func__, port); + + mutex_lock(&dcp->hpd_mutex); + if (dptx_service_available(&dcp->dptxport[port]) && dcp->dptxport[port].connected) { + dptxport_release_display(dcp->dptxport[port].service); + dcp->dptxport[port].connected = false; + } + mutex_unlock(&dcp->hpd_mutex); + + return 0; +} + +static int dcp_dptx_connect_oob_h17p(struct platform_device *pdev, u32 port) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + int ret; + + if (dcp_is_external_h17p(dcp)) { + bool new_cable; + + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + cancel_work_sync(&dcp->external_stop_work); + guard(mutex)(&dcp->external_lock); + new_cable = !dcp->external_cable; + if (new_cable) + dcp->external_generation++; + dcp->external_cable = true; + WRITE_ONCE(dcp->external_physically_absent, false); + dcp_external_publish_generation(dcp); + scoped_guard(mutex, &dcp->hpd_mutex) { + WRITE_ONCE(dcp->typec_cable_connected, true); + if (new_cable) { + dcp->typec_generation++; + dcp_firmware_metadata_connection(dcp, dcp->typec_generation, true); + } + dcp->placeholder_retried = false; + } + return dcp_external_link_locked(dcp); + } + + if (dcp_is_typec_output(dcp)) { + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + guard(mutex)(&dcp->hpd_mutex); + dcp->typec_generation++; + WRITE_ONCE(dcp->typec_cable_connected, true); + dcp_firmware_metadata_connection(dcp, dcp->typec_generation, true); + dcp->typec_reconnect_tries = 0; + dcp->placeholder_retried = false; + cancel_delayed_work(&dcp->typec_reconnect_wq); + } + + ret = dcp_dptx_connect(dcp, port); + if (ret && dcp_is_typec_output(dcp)) + mod_delayed_work(system_freezable_wq, &dcp->typec_reconnect_wq, + DPTX_RECONNECT_DELAY); + + return ret; +} + +static int dcp_dptx_disconnect_oob_h17p(struct platform_device *pdev, u32 port) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + + if (dcp_is_external_h17p(dcp)) + return dcp_external_disconnect(dcp, true); + + if (dcp_is_typec_output(dcp)) { + scoped_guard(mutex, &dcp->hpd_mutex) { + WRITE_ONCE(dcp->typec_cable_connected, false); + dcp->typec_generation++; + dcp_firmware_metadata_connection(dcp, dcp->typec_generation, false); + } + WRITE_ONCE(dcp->typec_crtc_off, false); + reinit_completion(&dcp->typec_iomfb_hpd_ready); + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + } + + disconnected_hpd_event_h17p(dcp->connector); + + if (dcp->avep) + av_service_disconnect(dcp); + + if (dptx_service_available(&dcp->dptxport[port])) + dptxport_set_hpd(dcp->dptxport[port].service, false); + + return dcp_dptx_disconnect(dcp, port); +} + +static int dcp_start_h17p(struct platform_device *pdev) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + int ret; + + if (!dcp->rtk) + return -ENODEV; + + init_completion(&dcp->start_done); + + /* + * The coprocessor start can fail (or be recovered from) without tearing + * the DRM side down, so this can be reached with no RTKit instance at + * all. Starting an endpoint then dereferences NULL inside + * apple_rtkit_start_ep() and takes the machine with it. + */ + if (!dcp->rtk || IS_ERR(dcp->rtk)) { + dev_err(dcp->dev, + "cannot start endpoints: RTKit is not initialised\n"); + return -ENODEV; + } + + /* Make the H17P allocator available before external service discovery. */ + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL) { + ret = apple_rtkit_start_ep(dcp->rtk, REMOTE_ALLOC_ENDPOINT); + if (ret) + return dev_err_probe(dcp->dev, ret, "Failed to start remote allocator\n"); + } + + /* start RTKit endpoints */ + ret = systemep_init(dcp); + if (ret) + dev_warn(dcp->dev, "Failed to start system endpoint: %d\n", ret); + + if (unstable_edid && !dcp_has_panel(dcp)) { + ret = dpavservep_init(dcp); + if (ret) + dev_warn(dcp->dev, "Failed to start DPAVSERV endpoint: %d", + ret); + } + + if (dcp->phy && dcp->fw_compat >= DCP_FIRMWARE_V_13_5) { + ret = ibootep_init(dcp); + if (ret) + dev_warn(dcp->dev, "Failed to start IBOOT endpoint: %d\n", + ret); + + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + ret = dptx_controller_init(dcp); + if (ret) + return dev_err_probe(dcp->dev, ret, + "Failed to start DPTX controller\n"); + } + + ret = dptxep_init(dcp); + if (ret) { + dev_warn(dcp->dev, "Failed to start DPTX endpoint: %d\n", + ret); +#ifdef DCP_DPTX_DISCONNECT_ON_INIT + /* + * This disconnect / connect cycle on init is only necessary + * when using dcp0 on j473, j474s and presumedly j475c. + * Since dcp0 is not used at the moment let's avoid this + * since it is possibly the cause for startup issues. + */ + } else if (dptx_service_available(&dcp->dptxport[0])) { + bool connected; + /* force disconnect on start - necessary if the display + * is already up from m1n1 + */ + dptxport_set_hpd(dcp->dptxport[0].service, false); + dptxport_release_display(dcp->dptxport[0].service); + usleep_range(10 * USEC_PER_MSEC, 25 * USEC_PER_MSEC); + + connected = gpiod_get_value_cansleep(dcp->hdmi_hpd); + dev_info(dcp->dev, "%s: DP2HDMI HPD connected:%d\n", __func__, connected); + + // necessary on j473/j474 but not on j314c + if (connected) + dcp_dptx_connect(dcp, 0); +#endif + } + } else if (dcp->phy) { + dev_warn(dcp->dev, "OS firmware incompatible with dptxport EP\n"); + } + ret = iomfb_start_rtkit(dcp); + if (ret) + dev_err(dcp->dev, "Failed to start IOMFB endpoint: %d\n", ret); + +#if IS_ENABLED(CONFIG_DRM_APPLE_AUDIO) + if (hdmi_audio) { + ret = avep_init(dcp); + if (ret) + dev_warn(dcp->dev, "Failed to start AV endpoint: %d", ret); + ret = 0; + } +#endif + + return ret; +} + +#define DCP_BL_FALLBACK_DELAY msecs_to_jiffies(2000) + +static bool dcp_uses_soft_dpms(struct apple_dcp *dcp) +{ + return (dcp_backlight_active(dcp) && dcp_has_panel(dcp)) || + (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + dcp->connector_type == DRM_MODE_CONNECTOR_eDP); +} + +static void dcp_poweron_h17p(struct platform_device *pdev) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + bool wait_for_typec_hpd = false; + unsigned long remaining; + int ret; + + if (dcp_is_external_h17p(dcp)) { + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + cancel_work_sync(&dcp->external_stop_work); + guard(mutex)(&dcp->external_lock); + ret = dcp_external_start_locked(dcp); + if (!ret) + dcp->external_crtc_generation = dcp->external_generation; + else + dev_err(dcp->dev, "external display activation failed: %d\n", ret); + return; + } + + if (dcp_uses_soft_dpms(dcp)) { + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !READ_ONCE(dcp->pipe_enabled_h17p)) { + iomfb_poweron_h17p(dcp); + if (!READ_ONCE(dcp->pipe_enabled_h17p)) { + dev_err(dcp->dev, "initial display pipe enable failed\n"); + WRITE_ONCE(dcp->crashed, true); + return; + } + /* The powerlog takeover report normally precedes this. */ + if (dcp_has_panel(dcp)) + queue_delayed_work(system_freezable_wq, + &dcp->bl_fallback_wq, + DCP_BL_FALLBACK_DELAY); + } + ret = dcp_backlight_dpms(dcp, true); + if (ret) + dev_warn(dcp->dev, "backlight restore unavailable: %d\n", ret); + return; + } + + if (dcp_is_typec_output(dcp)) { + wait_for_typec_hpd = READ_ONCE(dcp->typec_crtc_off) && + READ_ONCE(dcp->typec_cable_connected); + WRITE_ONCE(dcp->typec_crtc_off, false); + + /* + * A Type-C CRTC disable releases its DPTX session. Re-establish it + * synchronously before IOMFB is powered back on. + */ + if (READ_ONCE(dcp->typec_cable_connected)) { + cancel_delayed_work(&dcp->typec_reconnect_wq); + dcp->typec_reconnect_tries = 0; + ret = dcp_dptx_connect(dcp, 0); + if (ret) + mod_delayed_work(system_freezable_wq, + &dcp->typec_reconnect_wq, + DPTX_RECONNECT_DELAY); + else if (wait_for_typec_hpd) { + remaining = wait_for_completion_timeout( + &dcp->typec_iomfb_hpd_ready, + msecs_to_jiffies(3000)); + if (!remaining) + dev_warn(dcp->dev, + "Type-C IOMFB hotplug not ready on wake\n"); + } + } + } else if (dcp->hdmi_hpd) { + bool connected = gpiod_get_value_cansleep(dcp->hdmi_hpd); + dev_info(dcp->dev, "%s: DP2HDMI HPD connected:%d\n", __func__, connected); + + if (connected) + dcp_dptx_connect(dcp, 0); + } + + switch (dcp->fw_compat) { + case DCP_FIRMWARE_V_12_3: + iomfb_poweron_v12_3(dcp); + break; + case DCP_FIRMWARE_V_13_5: + iomfb_poweron_v13_3(dcp); + break; + case DCP_FIRMWARE_H17P: + iomfb_poweron_h17p(dcp); + break; + default: + WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); + break; + } + if (dcp->avep) + av_service_connect(dcp); +} + +/* Same-CRTC helper dependencies order this token refresh before later disables. */ +void dcp_external_recover(struct platform_device *pdev) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + + if (dcp_is_external_h17p(dcp)) + dcp_poweron(pdev); +} + +static void dcp_poweroff_h17p(struct platform_device *pdev) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + int ret; + + if (dcp_is_external_h17p(dcp)) { + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_delayed_work_sync(&dcp->placeholder_edid_wq); + guard(mutex)(&dcp->external_lock); + ret = dcp_external_stop_locked(dcp, dcp->external_crtc_generation); + if (ret) + dev_err(dcp->dev, "external display stop failed: %d\n", ret); + return; + } + + cancel_delayed_work(&dcp->swap_watchdog_wq); + + /* Internal H17P DPMS is a brightness present, never a pipe stop. */ + if (dcp_uses_soft_dpms(dcp)) { + ret = dcp_backlight_dpms(dcp, false); + if (ret) + dev_warn(dcp->dev, "backlight blank unavailable: %d\n", ret); + return; + } + + if (dcp->avep) + av_service_disconnect(dcp); + + /* + * Powering a Type-C CRTC off drops DCP's synthetic HPD, and the firmware + * reports that as an unplug. The display is still attached: keep the + * connector connected (see dcpep_cb_hotplug()) and let dcp_poweron() + * re-establish the DPTX session. Recreating it here instead makes the + * display vanish and come back, and compositors light a returning + * display, so DPMS off never sticks. Cable removal is reported through + * the Type-C mux. + */ + if (dcp_is_typec_output(dcp)) { + reinit_completion(&dcp->typec_iomfb_hpd_ready); + if (READ_ONCE(dcp->typec_cable_connected)) + WRITE_ONCE(dcp->typec_crtc_off, true); + /* dcp_poweron() reconnects the link on DPMS wake. */ + cancel_delayed_work(&dcp->typec_reconnect_wq); + } + + _dcp_poweroff(dcp); + + if (dcp_is_typec_output(dcp)) { + /* DCP owns a synthetic HPD for Type-C. Release it with the CRTC. */ + if (dptx_service_available(&dcp->dptxport[0]) && dcp->dptxport[0].connected) { + ret = dptxport_set_hpd(dcp->dptxport[0].service, false); + if (ret) + dev_warn(dcp->dev, + "failed to deassert Type-C DPTX HPD: %d\n", ret); + dcp_dptx_disconnect(dcp, 0); + } + } else if (dcp->hdmi_hpd) { + bool connected = gpiod_get_value_cansleep(dcp->hdmi_hpd); + if (!connected) { + disconnected_hpd_event_h17p(dcp->connector); + dcp_dptx_disconnect(dcp, 0); + } + } +} + +static void dcp_work_register_backlight_h17p(struct work_struct *work) +{ + int ret; + struct apple_dcp *dcp; + + dcp = container_of(work, struct apple_dcp, bl_register_wq); + + mutex_lock(&dcp->bl_register_mutex); + if (dcp->brightness.bl_dev) + goto out_unlock; + + /* try to register backlight device, */ + ret = dcp_backlight_register(dcp); + if (ret == -ENODATA) + goto out_unlock; + if (ret) { + dev_err(dcp->dev, "Unable to register backlight device: %d\n", ret); + /* + * The H17P policy encodes the panel ceiling into every present + * and keeps working without a class device; only older + * firmware stops sending brightness here. + */ + if (!dcp_backlight_active(dcp)) + dcp->brightness.maximum = 0; + } + +out_unlock: + mutex_unlock(&dcp->bl_register_mutex); +} + +/* + * H17P never publishes the panel level, and the loader's level arrives only + * as an optional powerlog report. Without one, register the backlight at + * the middle of the panel range. That value is only reported: no level is + * presented until userspace writes one, so the panel keeps the loader's + * level. + */ +static void dcp_work_backlight_fallback(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(to_delayed_work(work), + struct apple_dcp, bl_fallback_wq); + u32 nits = dcp->brightness.maximum / 2; + int ret; + + if (READ_ONCE(dcp->quiescing) || READ_ONCE(dcp->crashed) || + dcp_backlight_active(dcp)) + return; + + ret = iomfb_configure_backlight_h17p(dcp, dcp->brightness.maximum, + false, 0, true, nits); + if (!ret) + dev_info(dcp->dev, + "no loader brightness reported; backlight starts at %u nits without a panel change\n", + nits); + else if (ret != -EBUSY) + dev_warn(dcp->dev, "backlight registration failed: %d\n", ret); +} + +static void dcp_release_piodma_iommu_dev(struct apple_dcp *dcp) +{ + if (dcp->retain_dma) { + WRITE_ONCE(dcp->quiescing, true); + return; + } + if (dcp->piodma) { + if (dcp->piodma_created) + of_platform_device_destroy(&dcp->piodma->dev, NULL); + else + put_device(&dcp->piodma->dev); + } + dcp->piodma = NULL; + dcp->iommu_dom = NULL; + dcp->piodma_created = false; +} + +void dcp_retain_framebuffer(struct platform_device *pdev, + struct dcp_fb_reference *entry) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + + mutex_lock(&dcp->swapped_out_fbs_lock); + list_add_tail(&entry->head, &dcp->swapped_out_fbs); + mutex_unlock(&dcp->swapped_out_fbs_lock); +} + +void dcp_arm_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id) +{ + struct dcp_fb_reference *entry; + + mutex_lock(&dcp->swapped_out_fbs_lock); + list_for_each_entry(entry, &dcp->swapped_out_fbs, head) { + if (entry->armed) + continue; + entry->swap_id = swap_id; + entry->armed = true; + } + mutex_unlock(&dcp->swapped_out_fbs_lock); +} + +void dcp_unarm_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id) +{ + struct dcp_fb_reference *entry; + + mutex_lock(&dcp->swapped_out_fbs_lock); + list_for_each_entry(entry, &dcp->swapped_out_fbs, head) + if (entry->armed && entry->swap_id == swap_id) + entry->armed = false; + mutex_unlock(&dcp->swapped_out_fbs_lock); +} + +void dcp_release_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id) +{ + struct dcp_fb_reference *entry, *tmp; + LIST_HEAD(completed); + + mutex_lock(&dcp->swapped_out_fbs_lock); + list_for_each_entry_safe(entry, tmp, &dcp->swapped_out_fbs, head) { + if (!entry->armed || entry->swap_id != swap_id) + continue; + list_move_tail(&entry->head, &completed); + } + mutex_unlock(&dcp->swapped_out_fbs_lock); + + list_for_each_entry_safe(entry, tmp, &completed, head) { + list_del(&entry->head); + drm_framebuffer_put(entry->fb); + kfree(entry); + } +} + +void dcp_release_all_retained_framebuffers(struct apple_dcp *dcp) +{ + struct dcp_fb_reference *entry, *tmp; + LIST_HEAD(completed); + + mutex_lock(&dcp->swapped_out_fbs_lock); + list_splice_init(&dcp->swapped_out_fbs, &completed); + mutex_unlock(&dcp->swapped_out_fbs_lock); + + list_for_each_entry_safe(entry, tmp, &completed, head) { + list_del(&entry->head); + drm_framebuffer_put(entry->fb); + kfree(entry); + } +} + + +static int dcp_create_piodma_iommu_dev_h17p(struct apple_dcp *dcp) +{ + int ret; + struct device_node *node __free(device_node) = of_get_child_by_name(dcp->dev->of_node, "piodma"); + + if (!node) + return dev_err_probe(dcp->dev, -ENODEV, + "Failed to get piodma child DT node\n"); + + dcp->piodma = of_find_device_by_node(node); + dcp->piodma_created = !dcp->piodma; + if (dcp->piodma_created) + dcp->piodma = of_platform_device_create(node, NULL, dcp->dev); + if (!dcp->piodma) + return dev_err_probe(dcp->dev, -ENODEV, + "Failed to create piodma pdev for %pOF\n", node); + + ret = dma_set_mask_and_coherent(&dcp->piodma->dev, DMA_BIT_MASK(42)); + if (ret) + goto err_destroy_pdev; + + ret = of_dma_configure(&dcp->piodma->dev, node, true); + if (ret) { + ret = dev_err_probe(dcp->dev, ret, + "Failed to configure IOMMU child DMA\n"); + goto err_destroy_pdev; + } + + dcp->iommu_dom = iommu_get_domain_for_dev(&dcp->piodma->dev); + if (IS_ERR(dcp->iommu_dom)) { + ret = dev_err_probe(dcp->dev, PTR_ERR(dcp->iommu_dom), + "Failed to get default iommu domain for " + "piodma device\n"); + dcp->iommu_dom = NULL; + goto err_destroy_pdev; + } + + return 0; +err_destroy_pdev: + dcp_release_piodma_iommu_dev(dcp); + return ret; +} + +static int dcp_get_disp_regs_h17p(struct apple_dcp *dcp) +{ + struct platform_device *pdev = to_platform_device(dcp->dev); + int count = 0; + int i, ret; + char name[16]; + const char *reg_name; + struct resource *res; + + if (of_property_present(dcp->dev->of_node, "reg-names")) { + ret = of_property_count_strings(dcp->dev->of_node, "reg-names"); + if (ret < 0) + return ret; + + for (i = 0; i < ret; i++) { + if (of_property_read_string_index(dcp->dev->of_node, + "reg-names", i, ®_name)) + return -EINVAL; + if (!strncmp(reg_name, "disp-", 5)) + count++; + } + } else { + count = pdev->num_resources - 1; + } + + if (count <= 0 || count > MAX_DISP_REGISTERS) + return -EINVAL; + + for (i = 0; i < count; ++i) { + if (of_property_present(dcp->dev->of_node, "reg-names")) { + snprintf(name, sizeof(name), "disp-%d", i); + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, + name); + } else { + res = platform_get_resource(pdev, IORESOURCE_MEM, 1 + i); + } + if (!res) + return -EINVAL; + dcp->disp_registers[i] = res; + } + + if (dcp->hw.firmware_clock) { + unsigned int windows = dcp->hw.firmware_display_windows; + bool external = dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL; + + if (!windows || windows >= MAX_DISP_REGISTERS) + return -EINVAL; + if (external) { + /* The external clock aperture is already described by disp-4. */ + if (count != windows + 1) + return -EINVAL; + res = dcp->disp_registers[windows]; + if (!apple_dcp_external_clock_window(res->start, + resource_size(res), + dcp->hw.firmware_clock->start, + resource_size(dcp->hw.firmware_clock))) + return -EINVAL; + count = windows; + } + if (count != windows || resource_size(dcp->hw.firmware_clock) < 4 || + dcp->hw.firmware_scratch > resource_size(dcp->hw.firmware_clock) - 4 || + dcp->hw.firmware_request > resource_size(dcp->hw.firmware_clock) - 4) + return -EINVAL; + + dcp->disp_bw_scratch_res = *dcp->hw.firmware_clock; + dcp->disp_bw_scratch_index = count; + dcp->disp_bw_scratch_offset = dcp->hw.firmware_scratch; + dcp->disp_bw_doorbell_res = dcp->disp_bw_scratch_res; + dcp->disp_bw_doorbell_res.start += dcp->hw.firmware_request; + dcp->disp_bw_doorbell_res.end = dcp->disp_bw_doorbell_res.start + 3; + dcp->disp_registers[count] = &dcp->disp_bw_scratch_res; + dcp->nr_disp_registers = count + 1; + return 0; + } + + /* load pmgr bandwidth scratch resource and offset */ + ret = dcp_get_bw_scratch_reg(dcp, count); + if (ret < 0) + return ret; + count += 1; + + /* load pmgr bandwidth doorbell resource if present (only on t8103) */ + if (of_property_present(dcp->dev->of_node, "apple,bw-doorbell")) { + ret = dcp_get_bw_doorbell_reg(dcp, count); + if (ret < 0) + return ret; + count += 1; + } + + dcp->nr_disp_registers = count; + return 0; +} + +static void dcp_disable_typec_work_h17p(struct apple_dcp *dcp, bool release_cable) +{ + scoped_guard(mutex, &dcp->hpd_mutex) { + if (release_cable) + WRITE_ONCE(dcp->typec_cable_connected, false); + dcp->typec_generation++; + dcp_firmware_metadata_connection(dcp, dcp->typec_generation, false); + } + if (release_cable && dcp_is_external_h17p(dcp)) + dpavservep_detach(dcp); + /* Block new enqueues as well as draining users of the AFK endpoints. */ + disable_delayed_work_sync(&dcp->typec_reconnect_wq); + disable_delayed_work_sync(&dcp->placeholder_edid_wq); + cancel_work_sync(&dcp->external_stop_work); +} + +static void dcp_enable_typec_work_h17p(struct apple_dcp *dcp) +{ + enable_delayed_work(&dcp->typec_reconnect_wq); + enable_delayed_work(&dcp->placeholder_edid_wq); + /* A cable can be routed before the DRM component binds. */ + if (READ_ONCE(dcp->typec_cable_connected)) + mod_delayed_work(system_freezable_wq, &dcp->typec_reconnect_wq, 0); +} + +static bool dcp_external_is_idle(struct apple_dcp *dcp) +{ + u32 control = readl_relaxed(dcp->coproc_reg + APPLE_DCP_COPROC_CPU_CONTROL); + u32 status = readl_relaxed(dcp->coproc_reg + APPLE_DCP_COPROC_CPU_STATUS); + + /* Same externally inherited idle state accepted by the public loader. */ + return (control & APPLE_DCP_COPROC_CPU_CONTROL_RUN) && + (status & APPLE_DCP_COPROC_CPU_STATUS_STOPPED) && + (status & APPLE_DCP_COPROC_CPU_STATUS_IDLE); +} + +static int dcp_comp_bind_h17p(struct device *dev, struct device *main, void *data) +{ + struct device_node *panel_np; + struct apple_dcp *dcp = dev_get_drvdata(dev); + u32 cpu_ctrl; + int ret; + + if (READ_ONCE(dcp->quiescing)) + return dev_err_probe(dev, -EBUSY, + "Live DCP session cannot be rebound; reboot required\n"); + + if (dcp->external_prepared) + return 0; + + /* A timed-out prior session may still scan these mappings. */ + mutex_lock(&dcp->swapped_out_fbs_lock); + ret = list_empty(&dcp->swapped_out_fbs) ? 0 : -EBUSY; + mutex_unlock(&dcp->swapped_out_fbs_lock); + if (ret) + return dev_err_probe(dev, ret, + "Previous scanout has not been stopped\n"); + + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(42)); + if (ret) + return ret; + + dcp->coproc_reg = devm_platform_ioremap_resource_byname(to_platform_device(dev), "coproc"); + if (IS_ERR(dcp->coproc_reg)) + return PTR_ERR(dcp->coproc_reg); + + /* + * Display coprocessors running H17-generation firmware are left running + * by the bootloader with their firmware mapped through a locked DART. + * Linux attaches to that session with a standard RTKit INIT/HELLO + * exchange and must not stop or restart the ASC. + */ + if (dcp->hw.adopt_live_session) { + cpu_ctrl = readl_relaxed(dcp->coproc_reg + + APPLE_DCP_COPROC_CPU_CONTROL); + if (!(cpu_ctrl & APPLE_DCP_COPROC_CPU_CONTROL_RUN)) + return dev_err_probe(dev, -EOPNOTSUPP, + "DCP was not left running by the bootloader\n"); + } + + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + !dcp_external_is_idle(dcp)) + return dev_err_probe(dev, -EBUSY, + "External DCP is not in the inherited idle state\n"); + + if (dcp->index || dcp->dptx_phy || dcp->dptx_die) + dev_info(dev, "DCP index:%u dptx target phy: %u dptx die: %u\n", + dcp->index, dcp->dptx_phy, dcp->dptx_die); + + if (!show_notch) + ret = of_property_read_u32(dev->of_node, "apple,notch-height", + &dcp->notch_height); + + if (dcp->notch_height > MAX_NOTCH_HEIGHT) + dcp->notch_height = MAX_NOTCH_HEIGHT; + if (dcp->notch_height > 0) + dev_info(dev, "Detected display with notch of %u pixel\n", dcp->notch_height); + + /* initialize brightness scale to a sensible default to avoid divide by 0*/ + dcp->brightness.scale = 65536; + panel_np = of_get_compatible_child(dev->of_node, "apple,panel-mini-led"); + if (panel_np) + dcp->panel.has_mini_led = true; + else + panel_np = of_get_compatible_child(dev->of_node, "apple,panel"); + + if (panel_np) { + const char height_prop[2][16] = { "adj-height-mm", "height-mm" }; + + if (of_device_is_available(panel_np)) { + ret = of_property_read_u32(panel_np, "apple,max-brightness", + &dcp->brightness.maximum); + if (ret) + dev_err(dev, "Missing property 'apple,max-brightness'\n"); + } + + of_property_read_u32(panel_np, "width-mm", &dcp->panel.width_mm); + /* use adjusted height as long as the notch is hidden */ + of_property_read_u32(panel_np, height_prop[!dcp->notch_height], + &dcp->panel.height_mm); + + of_node_put(panel_np); + dcp->fixed_connector_type = DRM_MODE_CONNECTOR_eDP; + dcp->connector_type = DRM_MODE_CONNECTOR_eDP; + INIT_WORK(&dcp->bl_register_wq, dcp_work_register_backlight_h17p); + mutex_init(&dcp->bl_register_mutex); + INIT_WORK(&dcp->bl_update_wq, dcp_work_update_backlight); + } + + ret = dcp_create_piodma_iommu_dev_h17p(dcp); + if (ret) + return dev_err_probe(dev, ret, + "Failed to create PIODMA iommu child device\n"); + if (!dcp->iommu_dom) { + ret = dev_err_probe(dev, -EPROBE_DEFER, + "PIODMA iommu domain is unavailable\n"); + goto err_piodma; + } + + ret = dcp_get_disp_regs_h17p(dcp); + if (ret) { + dev_err(dev, "failed to find display registers\n"); + goto err_piodma; + } + + dcp->clk = devm_clk_get(dev, NULL); + if (IS_ERR(dcp->clk)) { + ret = dev_err_probe(dev, PTR_ERR(dcp->clk), + "Unable to find clock\n"); + goto err_piodma; + } + dcp->clk_194 = devm_clk_get_optional(dev, "clock-194"); + if (IS_ERR(dcp->clk_194)) { + ret = dev_err_probe(dev, PTR_ERR(dcp->clk_194), + "Unable to find clock 0x194\n"); + goto err_piodma; + } + + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + (!dcp->clk_194 || !clk_get_rate(dcp->clk) || + !clk_get_rate(dcp->clk_194))) { + ret = dev_err_probe(dev, -EINVAL, + "External display clocks are unavailable\n"); + goto err_piodma; + } + + bitmap_zero(dcp->memdesc_map, DCP_MAX_MAPPINGS); + // TDOD: mem_desc IDs start at 1, for simplicity just skip '0' entry + set_bit(0, dcp->memdesc_map); + + INIT_WORK(&dcp->vblank_wq, dcp_delayed_vblank); + INIT_DELAYED_WORK(&dcp->swap_watchdog_wq, dcp_swap_watchdog); + + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + /* Pin before the first write or mailbox wake can expose DMA. */ + ret = dcp_pin_live_session(dcp); + if (ret) + goto err_piodma; + if (!dcp_external_is_idle(dcp)) { + ret = dev_err_probe(dev, -EBUSY, + "External DCP changed state before startup\n"); + goto err_piodma; + } + } + + if (!dcp->hw.adopt_live_session) { + cpu_ctrl = + readl_relaxed(dcp->coproc_reg + APPLE_DCP_COPROC_CPU_CONTROL); + writel_relaxed(cpu_ctrl | APPLE_DCP_COPROC_CPU_CONTROL_RUN, + dcp->coproc_reg + APPLE_DCP_COPROC_CPU_CONTROL); + } + + /* Registering the mailbox can already expose RTKit buffer requests. */ + ret = dcp_pin_live_session(dcp); + if (ret) + goto err_piodma; + /* External H17P supplies shared-buffer addresses during RTKit startup. */ + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) + dcp->rtk = apple_rtkit_init(dev, dcp, "mbox", 0, + &external_h17p_rtkit_ops); + else + dcp->rtk = apple_rtkit_init(dev, dcp, "mbox", 0, &rtkit_ops); + if (IS_ERR(dcp->rtk)) { + ret = dev_err_probe(dev, PTR_ERR(dcp->rtk), + "Failed to initialize RTKit\n"); + goto err_piodma; + } + ret = devm_add_action_or_reset(dev, dcp_release_rtkit, dcp); + if (ret) + goto err_piodma; + + if (dcp->hw.adopt_live_session) + dev_info(dev, "negotiating RTKit with the running DCP\n"); + + /* Announce host readiness before awaiting external IOP readiness. */ + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) + ret = apple_rtkit_wake_early_ap(dcp->rtk); + else + ret = apple_rtkit_wake(dcp->rtk); + if (ret) { + ret = dev_err_probe(dev, ret, "Failed to boot RTKit: %d\n", ret); + goto err_piodma; + } + dcp_enable_typec_work_h17p(dcp); + return 0; + +err_piodma: + dcp_release_piodma_iommu_dev(dcp); + return ret; +} + +/* + * We need to shutdown DCP before tearing down the display subsystem. Otherwise + * the DCP will crash and briefly flash a green screen of death. + */ +static void dcp_comp_unbind_h17p(struct device *dev, struct device *main, void *data) +{ + struct apple_dcp *dcp = dev_get_drvdata(dev); + + if (!dcp) + return; + + if (dcp->retain_dma) { + /* No firmware stop is acknowledged: detach host users, never DMA. */ + scoped_guard(mutex, &dcp->iomfb.lock) + WRITE_ONCE(dcp->quiescing, true); + cancel_work_sync(&dcp->external_stop_work); + if (dcp->crtc && !dcp->drm_retained) { + drm_dev_get(dcp->crtc->base.dev); + dcp->drm_retained = true; + } + iomfb_queue_stop(dcp); + afk_quiesce(dcp->avep); + afk_quiesce(dcp->dptxep); + afk_quiesce(dcp->dptx_controller_ep); + afk_quiesce(dcp->ibootep); + afk_quiesce(dcp->systemep); + afk_quiesce(dcp->dcpavservep); + cancel_delayed_work_sync(&dcp->bl_fallback_wq); + dcp_backlight_stop(dcp); + if (dcp->connector_type == DRM_MODE_CONNECTOR_eDP) { + cancel_work_sync(&dcp->bl_register_wq); + cancel_work_sync(&dcp->bl_update_wq); + } + cancel_delayed_work_sync(&dcp->typec_reconnect_wq); + cancel_work_sync(&dcp->vblank_wq); + if (dcp->connector) + cancel_work_sync(&dcp->connector->hotplug_wq); + if (dcp->typec_connector && dcp->typec_connector != dcp->connector) + cancel_work_sync(&dcp->typec_connector->hotplug_wq); + dev_warn(dev, "retaining live DCP DMA resources after unbind; reboot required\n"); + return; + } + + if (dcp->hdmi_hpd_irq) + disable_irq(dcp->hdmi_hpd_irq); + + dcp_disable_typec_work_h17p(dcp, true); + typec_mux_put(dcp->typec_mux); + + if (dcp->avep) { + av_service_disconnect(dcp); + afk_shutdown(dcp->avep); + dcp->avep = NULL; + } + + if (dcp->dptxep) { + afk_shutdown(dcp->dptxep); + dcp->dptxep = NULL; + } + + if (!IS_ERR_OR_NULL(dcp->dptx_controller_ep)) { + afk_shutdown(dcp->dptx_controller_ep); + dcp->dptx_controller_ep = NULL; + } + + if (dcp->ibootep) { + afk_shutdown(dcp->ibootep); + dcp->ibootep = NULL; + } + + if (dcp->systemep) { + afk_shutdown(dcp->systemep); + dcp->systemep = NULL; + } + + if (dcp->dcpavservep) { + afk_shutdown(dcp->dcpavservep); + dpavservep_detach(dcp); + dcp->dcpavservep = NULL; + } + + if (dcp->shmem) + iomfb_shutdown(dcp); + + iomfb_queue_stop(dcp); + dcp_release_piodma_iommu_dev(dcp); + + cancel_delayed_work_sync(&dcp->bl_fallback_wq); + dcp_backlight_stop(dcp); + if (dcp->connector_type == DRM_MODE_CONNECTOR_eDP) { + cancel_work_sync(&dcp->bl_register_wq); + cancel_work_sync(&dcp->bl_update_wq); + } + cancel_delayed_work_sync(&dcp->swap_watchdog_wq); + cancel_work_sync(&dcp->vblank_wq); + + devm_clk_put(dev, dcp->clk); + dcp->clk = NULL; + /* optional and possibly NULL, released with the bind's devres group */ + dcp->clk_194 = NULL; +} + +static int dcp_platform_probe_h17p(struct platform_device *pdev) +{ + const struct apple_dcp_hw_data *hw = of_device_get_match_data(&pdev->dev); + enum dcp_firmware_version fw_compat; + struct device *dev = &pdev->dev; + struct apple_dcp *dcp; + int ret, surf, num_surfs; + u32 surf_en; + u32 mux_index; + + if (dcp_session_retained(dev)) + return dev_err_probe(dev, -EBUSY, "Previous live DCP session requires a reboot\n"); + + /* Reject unsupported external firmware before replacing the boot display. */ + if (of_device_is_compatible(dev->of_node, "apple,t8140-dcpext") && + !dcp_h17p_external_supported(dev->of_node)) { + dev_err(dev, "Unsupported external display hardware or firmware handoff\n"); + return -ENODEV; + } + + fw_compat = dcp_check_firmware_version(dev); + if (fw_compat == DCP_FIRMWARE_UNKNOWN) + return -ENODEV; + + /* Check for "apple,bw-scratch" to avoid probing appledrm with outdated + * device trees. This prevents replacing simpledrm and ending up without + * display. + */ + if (!of_property_present(dev->of_node, "apple,bw-scratch") && + !hw->firmware_clock) + return dev_err_probe(dev, -ENODEV, "Incompatible devicetree! " + "Use devicetree matching this kernel.\n"); + + dcp = kzalloc_obj(*dcp); + if (!dcp) + return -ENOMEM; + ret = devm_add_action_or_reset(dev, dcp_release_context, dcp); + if (ret) + return ret; + + INIT_LIST_HEAD(&dcp->swapped_out_fbs); + mutex_init(&dcp->swapped_out_fbs_lock); + ret = dcp_backlight_init(dcp, dev); + if (ret) + return ret; + iomfb_queue_init(dcp); + + dcp->fw_compat = fw_compat; + dcp->dev = dev; + /* + * Type-C and Thunderbolt routes can be activated as soon as they are + * registered below, before the DRM device binds. + */ + /* The shared Type-C fabric takes the catalog lock and HDMI presence. */ + dcp_modes_init(dcp); + ret = devm_add_action_or_reset(dev, dcp_modes_release, dcp); + if (ret) + return ret; + dcp_fabric_init(dcp); + mutex_init(&dcp->hpd_mutex); + mutex_init(&dcp->external_lock); + INIT_WORK(&dcp->external_stop_work, dcp_external_stop_work); + mutex_init(&dcp->tb_lock); + spin_lock_init(&dcp->mode_state.lock); + spin_lock_init(&dcp->metadata.lock); + mutex_init(&dcp->dcpavserv.lock); + dcp->hw = *(struct apple_dcp_hw_data *)of_device_get_match_data(dev); + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + if (!dcp->hw.firmware_clock || + !apple_dcp_external_bw_valid(resource_size(dcp->hw.firmware_clock), + dcp->hw.firmware_scratch, + dcp->hw.firmware_request)) + return dev_err_probe(dev, -EINVAL, + "Invalid external profile bandwidth resources\n"); + iomfb_pause_presentation(dcp, true); + } + + + /* The domain use must survive driver-core DMA cleanup for a live session. */ + ret = iommu_device_use_default_domain(dev); + if (ret) + return ret; + ret = devm_add_action_or_reset(dev, dcp_release_dma_domain, dcp); + if (ret) + return ret; + dcp->fixed_connector_type = dcp_connector_type_from_dt(dev->of_node); + dcp->connector_type = dcp->fixed_connector_type; + of_property_read_u32(dev->of_node, "apple,dcp-index", &dcp->index); + of_property_read_u32(dev->of_node, "apple,dptx-phy", &dcp->dptx_phy); + of_property_read_u32(dev->of_node, "apple,dptx-die", &dcp->dptx_die); + dcp->fixed_dptx_phy = dcp->dptx_phy; + init_completion(&dcp->typec_iomfb_hpd_ready); + INIT_DELAYED_WORK(&dcp->typec_reconnect_wq, + dcp_typec_reconnect_work_h17p); + INIT_DELAYED_WORK(&dcp->placeholder_edid_wq, + dcp_placeholder_edid_work_h17p); + INIT_DELAYED_WORK(&dcp->bl_fallback_wq, dcp_work_backlight_fallback); + /* Balanced by enable at successful component bind. */ + disable_delayed_work(&dcp->typec_reconnect_wq); + disable_delayed_work(&dcp->placeholder_edid_wq); + + platform_set_drvdata(pdev, dcp); + + dcp->phy = devm_phy_optional_get(dev, "dp-phy"); + if (IS_ERR(dcp->phy)) { + dev_err(dev, "Failed to get dp-phy: %ld\n", PTR_ERR(dcp->phy)); + return PTR_ERR(dcp->phy); + } + dcp->fixed_phy = dcp->phy; + + bitmap_zero(dcp->iomfb_surfaces, DCP_MAX_PLANES); + if (!of_property_present(dev->of_node, "apple,iomfb-surfaces")) + num_surfs = 0; + else + num_surfs = of_property_count_elems_of_size(dev->of_node, + "apple,iomfb-surfaces", + sizeof(u32)); + + if (num_surfs == 0 || num_surfs == -ENODATA) { + set_bit(0, dcp->iomfb_surfaces); + set_bit(1, dcp->iomfb_surfaces); + } else if (num_surfs < 0) { + return num_surfs; + } else if (num_surfs > DCP_MAX_PLANES) { + dev_err(dev, "Number of iomfb-surfaces (%d) exceeds DCP_MAX_PLANES\n", + num_surfs); + return -EINVAL; + } + + surf = 0; + of_property_for_each_u32(dev->of_node, "apple,iomfb-surfaces", surf_en) { + if (surf_en) + set_bit(surf, dcp->iomfb_surfaces); + surf++; + } + /* The J700 loader names surface zero rather than enabling it. */ + if (of_machine_is_compatible("apple,j700") && + of_device_is_compatible(dev->of_node, "apple,t8140-dcp") && + num_surfs == 1 && + !of_property_read_u32(dev->of_node, "apple,iomfb-surfaces", &surf_en) && + surf_en == 0) + set_bit(0, dcp->iomfb_surfaces); + + if (dcp->phy) { + int ret; + /* + * Request DP2HDMI related GPIOs as optional for DP-altmode + * compatibility. J180D misses a dp2hdmi-pwren GPIO in the + * template ADT. TODO: check device ADT + */ + dcp->hdmi_hpd = devm_gpiod_get_optional(dev, "hdmi-hpd", GPIOD_IN); + if (IS_ERR(dcp->hdmi_hpd)) + return PTR_ERR(dcp->hdmi_hpd); + if (dcp->hdmi_hpd) { + int irq = gpiod_to_irq(dcp->hdmi_hpd); + if (irq < 0) { + dev_err(dev, "failed to translate HDMI hpd GPIO to IRQ\n"); + return irq; + } + dcp->hdmi_hpd_irq = irq; + + ret = devm_request_threaded_irq(dev, dcp->hdmi_hpd_irq, + NULL, dcp_dp2hdmi_hpd, + IRQF_ONESHOT | IRQF_NO_AUTOEN | + IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING, + "dp2hdmi-hpd-irq", dcp); + if (ret < 0) { + dev_err(dev, "failed to request HDMI hpd irq %d: %d\n", + irq, ret); + return ret; + } + } + + /* + * Power DP2HDMI on as it is required for the HPD irq. + * TODO: check if one is sufficient for the hpd to save power + * on battery powered Macbooks. + */ + dcp->hdmi_pwren = devm_gpiod_get_optional(dev, "hdmi-pwren", GPIOD_OUT_HIGH); + if (IS_ERR(dcp->hdmi_pwren)) + return PTR_ERR(dcp->hdmi_pwren); + + dcp->dp2hdmi_pwren = devm_gpiod_get_optional(dev, "dp2hdmi-pwren", GPIOD_OUT_HIGH); + if (IS_ERR(dcp->dp2hdmi_pwren)) + return PTR_ERR(dcp->dp2hdmi_pwren); + + /* + * A DCP may have both a fixed HDMI/DP route and allocatable Type-C + * routes. Keep the fixed route selected until the allocator borrows + * this otherwise-idle pipeline for a Type-C display. + */ + ret = dcp->fixed_phy ? + of_property_read_u32(dev->of_node, "mux-index", &mux_index) : + -ENODATA; + if (!ret) { + dcp->fixed_mux_index = mux_index; + dcp->xbar = devm_mux_control_get(dev, "dp-xbar"); + if (IS_ERR(dcp->xbar)) { + dev_err(dev, "Failed to get dp-xbar: %ld\n", PTR_ERR(dcp->xbar)); + return PTR_ERR(dcp->xbar); + } + ret = mux_control_select(dcp->xbar, mux_index); + if (ret) + dev_warn(dev, "mux_control_select failed: %d\n", ret); + else + dcp->fixed_route_selected = true; + + /* + * Switch atcphy to DP-only. should move to a Macbook Pro + * 14-/16-inch specific DP-to-HDMI drm_bridge. + */ + dcp->typec_mux = fwnode_typec_mux_get(dev_fwnode(dcp->dev)); + if (!IS_ERR_OR_NULL(dcp->typec_mux)) { + struct typec_altmode alt = { + .svid = USB_TYPEC_DP_SID, + }; + struct typec_mux_state state = { + .alt = &alt, + .mode = TYPEC_DP_STATE_C, + }; + int ret = typec_mux_set(dcp->typec_mux, &state); + dev_info(dev, "typec_mux_set() returned: %d\n", ret); + if (!ret) + dcp->phy_managed_by_typec = true; + } else { + dev_info(dev, "fwnode_typec_mux_get() returned: %ld\n", + IS_ERR(dcp->typec_mux) ? PTR_ERR(dcp->typec_mux) : 0); + dcp->typec_mux = NULL; + } + } + } + + ret = dcp_register_typec_routes(dcp); + if (ret) + return ret; + + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + /* Complete optional startup before the DRM master can depend on it. */ + ret = dcp_comp_bind_h17p(dev, NULL, NULL); + if (ret) { + dcp_disable_typec_work_h17p(dcp, true); + return ret; + } + WRITE_ONCE(dcp->external_prepared, true); + } + + ret = component_add(&pdev->dev, &dcp_comp_ops); + if (ret && dcp->external_prepared) { + dcp_disable_typec_work_h17p(dcp, true); + } + return ret; +} + +static void dcp_platform_shutdown_h17p(struct platform_device *pdev) +{ + struct apple_dcp *dcp = platform_get_drvdata(pdev); + + if (dcp && dcp->retain_dma && dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + dcp_comp_unbind_h17p(&pdev->dev, NULL, NULL); + return; + } + + if (dcp && dcp->hw.adopt_live_session && + dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + /* + * This firmware keeps scanning across soft DPMS and has no + * known stop sequence. Component unbind releases the RTKit + * devres group and the piodma domain while firmware may still + * access them. Stop host submissions but keep the component and + * its DMA resources alive until the system resets. + */ + iomfb_queue_stop(dcp); + cancel_delayed_work_sync(&dcp->bl_fallback_wq); + dcp_backlight_stop(dcp); + cancel_work_sync(&dcp->bl_register_wq); + cancel_work_sync(&dcp->bl_update_wq); + return; + } + + component_del(&pdev->dev, &dcp_comp_ops); +} + +/* + * The display stays powered across system sleep, and the blanking present + * queued by the DRM and backlight suspend handlers must reach the firmware + * before its mailbox interrupt is suspended. + */ +static void dcp_drain_for_sleep(struct apple_dcp *dcp) +{ + int pass; + + if (!dcp->hw.adopt_live_session || dcp->fw_compat != DCP_FIRMWARE_H17P || + dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return; + + /* + * A completed present can schedule one more brightness present, and a + * rejected one a delayed retry; run a pending retry now instead of + * letting it fire after the mailbox has suspended. Four passes cover + * the first attempt and the policy's three retries. A fade still + * running ends at its target first. + */ + if (dcp_has_panel(dcp)) + dcp_backlight_finish_fade(dcp); + for (pass = 0; pass < 4; pass++) { + flush_delayed_work(&dcp->iomfb.backlight_retry); + if (dcp_has_panel(dcp)) + flush_work(&dcp->bl_update_wq); + if (!iomfb_queue_drain(dcp, msecs_to_jiffies(500))) { + dev_warn(dcp->dev, "display updates still pending at suspend\n"); + return; + } + /* + * A crashed or stopped DCP takes no more presents, so a level + * still pending there can never be sent and is not waited for. + * Checking after the drain, which returns at once for such a + * DCP, also covers a crash during the pass. + */ + if (READ_ONCE(dcp->crashed) || READ_ONCE(dcp->iomfb.stopped)) + return; + if (!dcp_backlight_active(dcp) || !dcp_backlight_pending(dcp)) + return; + } + dev_warn(dcp->dev, "brightness change still pending at suspend\n"); +} + +/* + * DPMS off leaves the internal H17P pipe scanning out (see dcp_poweroff()), + * and while it scans out the firmware publishes a property to the AP on every + * frame. The mailbox interrupt is not suspended, so each of those messages + * wakes a CPU out of suspend-to-idle about 60 times a second. Hold the + * messages in the mailbox for system sleep and process them at resume. + * apple-drm is a consumer of this device, so its suspend handler has blanked + * the panel before this runs and its resume handler restores the CRTC after. + */ +static void dcp_hold_for_sleep(struct apple_dcp *dcp) +{ + if (!dcp_uses_soft_dpms(dcp) || dcp->fw_compat != DCP_FIRMWARE_H17P || + dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G || + READ_ONCE(dcp->crashed)) + return; + + iomfb_pause_presentation(dcp, true); + if (iomfb_queue_settle(dcp, msecs_to_jiffies(500))) { + iomfb_pause_presentation(dcp, false); + return; + } + apple_rtkit_hold_rx(dcp->rtk); + dcp->rx_held = true; +} + +static void dcp_release_after_sleep(struct apple_dcp *dcp) +{ + if (!dcp->rx_held) + return; + dcp->rx_held = false; + apple_rtkit_resume_rx(dcp->rtk); + iomfb_pause_presentation(dcp, false); +} + +static int dcp_platform_suspend_h17p(struct device *dev) +{ + struct apple_dcp *dcp = dev_get_drvdata(dev); + + if (dcp_is_external_h17p(dcp)) { + int ret; + + dcp_disable_typec_work_h17p(dcp, false); + ret = dcp_external_disconnect(dcp, false); + if (ret) { + dcp_enable_typec_work_h17p(dcp); + return ret; + } + device_set_wakeup_path(dev); + return 0; + } + + dcp_drain_for_sleep(dcp); + /* + * The Type-C route reports cable removal through + * dcp_dptx_disconnect_oob(). A DP tunnel kept through the sleep stays + * connected, and resume powers the CRTC back up through dcp_poweron() + * as after DPMS off. + */ + dcp_disable_typec_work_h17p(dcp, false); + cancel_delayed_work_sync(&dcp->swap_watchdog_wq); + + if (dcp->avep) + av_service_disconnect(dcp); + + if (dcp->hdmi_hpd_irq) { + disable_irq(dcp->hdmi_hpd_irq); + if (!dcp->active_typec_route) { + disconnected_hpd_event_h17p(dcp->connector); + dcp_dptx_disconnect_h17p(dcp, 0); + } + } + /* + * Set the device as a wakeup device, which forces its power + * domains to stay on. We need this as we do not support full + * shutdown properly yet. + */ + device_set_wakeup_path(dev); + /* Last: the steps above still talk to the coprocessor. */ + dcp_hold_for_sleep(dcp); + + return 0; +} + +static int dcp_platform_resume_h17p(struct device *dev) +{ + struct apple_dcp *dcp = dev_get_drvdata(dev); + + dcp_release_after_sleep(dcp); + dcp_enable_typec_work_h17p(dcp); + if (dcp->hdmi_hpd_irq) + enable_irq(dcp->hdmi_hpd_irq); + + if (dcp->avep) + av_service_connect(dcp); + + return 0; +} + +static const struct dev_pm_ops dcp_platform_pm_ops = { + .prepare = pm_sleep_ptr(dcp_platform_prepare), + .complete = pm_sleep_ptr(dcp_platform_complete), + SYSTEM_SLEEP_PM_OPS(dcp_platform_suspend, dcp_platform_resume) +}; + + +static const struct apple_dcp_hw_data apple_dcp_hw_t6020 = { + .num_dptx_ports = 1, + .t6020_tunnel_flow = true, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t6020_dcpext = { + .num_dptx_ports = 2, + .t6020_tunnel_flow = true, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t8112 = { + .num_dptx_ports = 2, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_dcp = { + .num_dptx_ports = 0, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t6030 = { + .num_dptx_ports = 1, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t8122 = { + .num_dptx_ports = 2, +}; + +/* The internal panel only, through the 14.x IOMFB, as on T6030. */ +static const struct apple_dcp_hw_data apple_dcp_hw_t6031 = { + .num_dptx_ports = 1, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t6030_dcpext = { + .num_dptx_ports = 1, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_t8122_dcpext = { + .num_dptx_ports = 2, +}; + +static const struct apple_dcp_hw_data apple_dcp_hw_dcpext = { + .num_dptx_ports = 2, +}; + +/* The internal T8140 endpoint uses the H17P method profile. */ +static const struct resource t8140_firmware_clock = + DEFINE_RES_MEM(0x302800000, 0xbc000); + +static const struct apple_dcp_hw_data apple_dcp_hw_t8140 = { + .num_dptx_ports = 0, + .adopt_live_session = true, + .firmware_compat = DCP_FIRMWARE_H17P, + .firmware_clock = &t8140_firmware_clock, + .firmware_display_windows = 5, + .firmware_scratch = 0x20000, + .firmware_request = 0x68000, +}; + +/* Separate from the established internal-panel H17P profile. */ +static const struct apple_dcp_hw_data apple_dcp_hw_t8140_external = { + .num_dptx_ports = 1, + .iomfb_method_profile = DCP_IOMFB_METHODS_H17P_EXTERNAL, + .adopt_live_session = false, + .firmware_compat = DCP_FIRMWARE_H17P, + .firmware_clock = &t8140_firmware_clock, + .firmware_display_windows = 4, + /* Native PMGR BWR agent 2 and boost agent 3 for DISPEXT0. */ + .firmware_scratch = 0x28000, + .firmware_request = 0x70000, +}; + +/* + * M5 (T8142) runs H17-generation DCP firmware with the H17G method numbering + * and is left running by the bootloader. + */ +static const struct apple_dcp_hw_data apple_dcp_hw_t8142 = { + .num_dptx_ports = 0, + .iomfb_method_profile = DCP_IOMFB_METHODS_H17G, + .adopt_live_session = true, + .firmware_compat = DCP_FIRMWARE_H17P, }; static const struct of_device_id of_match[] = { @@ -3087,6 +5766,9 @@ static const struct of_device_id of_match[] = { { .compatible = "apple,t8122-dcp", .data = &apple_dcp_hw_t8122, }, { .compatible = "apple,t6031-dcp", .data = &apple_dcp_hw_t6031, }, { .compatible = "apple,t8122-dcpext", .data = &apple_dcp_hw_t8122_dcpext, }, + { .compatible = "apple,t8140-dcp", .data = &apple_dcp_hw_t8140, }, + { .compatible = "apple,t8140-dcpext", .data = &apple_dcp_hw_t8140_external, }, + { .compatible = "apple,t8142-dcp", .data = &apple_dcp_hw_t8142, }, { .compatible = "apple,dcp", .data = &apple_dcp_hw_dcp, }, { .compatible = "apple,dcpext", .data = &apple_dcp_hw_dcpext, }, {} @@ -3094,6 +5776,7 @@ static const struct of_device_id of_match[] = { MODULE_DEVICE_TABLE(of, of_match); static struct platform_driver apple_platform_driver = { + .driver_managed_dma = true, .probe = dcp_platform_probe, .remove = dcp_platform_remove, .shutdown = dcp_platform_shutdown, diff --git a/drivers/gpu/drm/apple/dcp.h b/drivers/gpu/drm/apple/dcp.h index 9cbd5926a36ab8..0fc12de4db11f7 100644 --- a/drivers/gpu/drm/apple/dcp.h +++ b/drivers/gpu/drm/apple/dcp.h @@ -21,6 +21,9 @@ struct apple_atomic_state { #define to_apple_atomic_state(s) container_of(s, struct apple_atomic_state, base) +bool dcp_h17p_external_supported(struct device_node *np); +bool dcp_h17p_external_ready(struct device_node *np); + struct apple_crtc { struct drm_crtc base; struct drm_pending_vblank_event *event; @@ -44,6 +47,7 @@ struct apple_encoder { void dcp_poweroff(struct platform_device *pdev); int dcp_typec_prepare_route(struct apple_dcp *dcp); void dcp_poweron(struct platform_device *pdev); +void dcp_external_recover(struct platform_device *pdev); int dcp_set_crc(struct drm_crtc *crtc, bool enabled); int dcp_crtc_atomic_check(struct drm_crtc *crtc, struct drm_atomic_state *state); int dcp_get_connector_type(struct platform_device *pdev); @@ -54,6 +58,12 @@ unsigned long* dcp_get_iomfb_surfaces(struct platform_device *pdev); int dcp_start(struct platform_device *pdev); int dcp_wait_ready(struct platform_device *pdev, u64 timeout); void dcp_flush(struct drm_crtc *crtc, struct drm_atomic_state *state); +void dcp_retain_framebuffer(struct platform_device *pdev, + struct dcp_fb_reference *entry); +void dcp_arm_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id); +void dcp_unarm_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id); +void dcp_release_retained_framebuffers(struct apple_dcp *dcp, u32 swap_id); +void dcp_release_all_retained_framebuffers(struct apple_dcp *dcp); bool dcp_is_initialized(struct platform_device *pdev); void apple_crtc_vblank(struct apple_crtc *apple); void dcp_drm_crtc_vblank(struct apple_crtc *crtc); @@ -71,6 +81,10 @@ bool dcp_crtc_mode_fixup(struct drm_crtc *crtc, const struct drm_display_mode *mode, struct drm_display_mode *adjusted_mode); void dcp_set_dimensions(struct apple_dcp *dcp, u64 generation); +/* H17P (T8140, T8142) */ +void dcp_set_dimensions_h17p(struct apple_dcp *dcp); +void dcp_update_connector_dimensions(struct apple_dcp *dcp, + struct drm_connector *connector); void dcp_send_message(struct apple_dcp *dcp, u8 endpoint, u64 message); int dcp_dptx_connect_oob(struct platform_device *pdev, u32 port); @@ -93,6 +107,7 @@ void iomfb_recv_msg(struct apple_dcp *dcp, u64 message); int systemep_init(struct apple_dcp *dcp); int dptxep_init(struct apple_dcp *dcp); +int dptx_controller_init(struct apple_dcp *dcp); int ibootep_init(struct apple_dcp *dcp); int dpavservep_init(struct apple_dcp *dcp); int avep_init(struct apple_dcp *dcp); diff --git a/drivers/gpu/drm/apple/dcp_backlight.c b/drivers/gpu/drm/apple/dcp_backlight.c index 055aecc8af709f..2bdd58652de0a8 100644 --- a/drivers/gpu/drm/apple/dcp_backlight.c +++ b/drivers/gpu/drm/apple/dcp_backlight.c @@ -85,13 +85,259 @@ static u32 brightness_part2[] = { }; +/* + * A new level fades in over DCP_BL_FADE_MS, one brightness-only present per + * step at most. Steps that come faster than presents complete merge, so the + * step only bounds how smooth the fade can be. + */ +#define DCP_BL_FADE_MS 250U +#define DCP_BL_FADE_STEP_MS 16U + +bool dcp_backlight_active(struct apple_dcp *dcp) +{ + return dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + READ_ONCE(dcp->backlight.state.ready); +} + +int dcp_backlight_takeover(struct apple_dcp *dcp, u32 millinits) +{ + u32 maximum, nits; + int ret; + + if (dcp_backlight_active(dcp)) + return 0; + /* H17P takes nits up to the panel ceiling, not the older DAC table's. */ + maximum = dcp->brightness.maximum; + ret = dcp_bl_takeover_nits(maximum, millinits, &nits); + if (ret) + return ret; + + /* Keep the first bounded hint. Later reports are not user requests. */ + return iomfb_configure_backlight_h17p(dcp, maximum, true, nits, false, 0); +} + static int dcp_get_brightness(struct backlight_device *bd) { struct apple_dcp *dcp = bl_get_data(bd); + unsigned long flags; + u32 actual; + + if (READ_ONCE(dcp->quiescing)) + return -ENODEV; + + if (dcp_backlight_active(dcp)) { + spin_lock_irqsave(&dcp->backlight.lock, flags); + actual = dcp->backlight.state.actual; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return actual; + } return dcp->brightness.nits; } +int dcp_backlight_configure(struct apple_dcp *dcp, u32 maximum, + bool inherited_valid, u32 inherited, + bool default_valid, u32 default_nits, + void (*kick)(struct apple_dcp *dcp)) +{ + unsigned long flags; + int ret; + + if (dcp->fw_compat != DCP_FIRMWARE_H17P || !dcp_has_panel(dcp) || + !dcp->crtc || !kick) + return -EINVAL; + + mutex_lock(&dcp->bl_register_mutex); + spin_lock_irqsave(&dcp->backlight.lock, flags); + if (dcp->backlight.state.ready || dcp->brightness.bl_dev) { + ret = -EBUSY; + } else { + ret = dcp_bl_init(&dcp->backlight.state, maximum, + inherited_valid, inherited, + default_valid, default_nits); + if (!ret) { + dcp->backlight.state.fade = true; + dcp->backlight.kick = kick; + } + } + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + mutex_unlock(&dcp->bl_register_mutex); + if (!ret) + schedule_work(&dcp->bl_register_wq); + return ret; +} + +bool dcp_backlight_seed(struct apple_dcp *dcp, u32 nits) +{ + unsigned long flags; + bool seeded; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + seeded = dcp_bl_seed(&dcp->backlight.state, nits); + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return seeded; +} + +static void dcp_backlight_kick(struct apple_dcp *dcp) +{ + void (*kick)(struct apple_dcp *dcp) = READ_ONCE(dcp->backlight.kick); + + if (kick) + kick(dcp); +} + +static void dcp_backlight_fade_work(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(to_delayed_work(work), + struct apple_dcp, + backlight.fade_work); + unsigned long flags; + bool running, kick; + u32 elapsed; + + if (READ_ONCE(dcp->quiescing) || READ_ONCE(dcp->crashed) || + READ_ONCE(dcp->iomfb.stopped)) + return; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + elapsed = jiffies_to_msecs(jiffies - dcp->backlight.fade_start); + running = dcp_bl_fade_step(&dcp->backlight.state, elapsed, + DCP_BL_FADE_MS); + /* A rejected present is retried with backoff, carrying the new level. */ + kick = dcp->backlight.state.dirty && !dcp->backlight.state.retries; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + + if (kick) + dcp_backlight_kick(dcp); + if (running) + queue_delayed_work(system_wq, &dcp->backlight.fade_work, + msecs_to_jiffies(DCP_BL_FADE_STEP_MS)); +} + +void dcp_backlight_stop(struct apple_dcp *dcp) +{ + cancel_delayed_work_sync(&dcp->backlight.fade_work); +} + +static void dcp_backlight_release(void *data) +{ + dcp_backlight_stop(data); +} + +/* + * The backlight device is a devres of @dev, registered after probe, so it is + * unregistered before this action runs, and no request can start a fade + * afterwards. + */ +int dcp_backlight_init(struct apple_dcp *dcp, struct device *dev) +{ + spin_lock_init(&dcp->backlight.lock); + INIT_DELAYED_WORK(&dcp->backlight.fade_work, dcp_backlight_fade_work); + return devm_add_action_or_reset(dev, dcp_backlight_release, dcp); +} + +/* Present the target of a running fade at once, e.g. before system sleep. */ +void dcp_backlight_finish_fade(struct apple_dcp *dcp) +{ + unsigned long flags; + bool kick; + + cancel_delayed_work_sync(&dcp->backlight.fade_work); + spin_lock_irqsave(&dcp->backlight.lock, flags); + dcp_bl_fade_step(&dcp->backlight.state, DCP_BL_FADE_MS, DCP_BL_FADE_MS); + kick = dcp->backlight.state.dirty; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + if (kick) + dcp_backlight_kick(dcp); +} + +int dcp_backlight_dpms(struct apple_dcp *dcp, bool on) +{ + unsigned long flags; + int ret; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + ret = dcp_bl_dpms(&dcp->backlight.state, on); + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + if (!ret) + dcp_backlight_kick(dcp); + return ret; +} + +int dcp_backlight_prepare(struct apple_dcp *dcp, bool have_surface, + struct dcp_backlight_present *present) +{ + unsigned long flags; + int ret; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + ret = dcp_bl_prepare(&dcp->backlight.state, have_surface, present); + /* No reservation: carry the last completed level on an ordinary present. */ + if (ret == -EALREADY) + present->nits = dcp->backlight.state.actual; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return ret; +} + +bool dcp_backlight_complete(struct apple_dcp *dcp, u64 sequence, bool accepted) +{ + unsigned long flags; + bool completed; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + completed = dcp_bl_complete(&dcp->backlight.state, sequence, accepted); + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + /* Queue retries after the central queue has finished this transaction. */ + return completed; +} + +bool dcp_backlight_pending(struct apple_dcp *dcp) +{ + unsigned long flags; + bool pending; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + pending = dcp->backlight.state.dirty && !dcp->backlight.state.in_flight; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return pending; +} + +/* + * Called from the receive path, which must not queue a transaction itself; + * the caller schedules the update worker when this returns true. Requests + * are honoured at most once per second, so a firmware that repeats the + * request after every brightness present cannot keep the queue busy. + */ +bool dcp_backlight_resend(struct apple_dcp *dcp) +{ + unsigned long flags; + bool resend = false; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + if (!dcp->backlight.resent || + time_after(jiffies, dcp->backlight.resent_at + HZ)) { + resend = dcp_bl_resend(&dcp->backlight.state); + if (resend) { + dcp->backlight.resent = true; + dcp->backlight.resent_at = jiffies; + } + } + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return resend; +} + +unsigned int dcp_backlight_retry_delay(struct apple_dcp *dcp) +{ + unsigned long flags; + unsigned int delay; + + spin_lock_irqsave(&dcp->backlight.lock, flags); + delay = dcp_bl_retry_delay(&dcp->backlight.state); + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return delay; +} + #define SCALE_FACTOR (1 << 10) static u32 interpolate(int val, int min, int max, u32 *tbl, size_t tbl_size) @@ -187,8 +433,71 @@ static int drm_crtc_set_brightness(struct apple_dcp *dcp) return ret; } +/* + * H17P (T8142 H17G): brightness-only commits re-present the scanout and back + * off within the call. + */ +static int drm_crtc_set_brightness_h17p(struct apple_dcp *dcp) +{ + struct drm_atomic_state *state; + struct drm_crtc_state *crtc_state; + struct drm_modeset_acquire_ctx ctx; + struct drm_crtc *crtc = &dcp->crtc->base; + int ret = 0; + + state = drm_atomic_state_alloc(crtc->dev); + if (!state) + return -ENOMEM; + + drm_modeset_acquire_init(&ctx, DRM_MODESET_ACQUIRE_INTERRUPTIBLE); + state->acquire_ctx = &ctx; + +retry: + crtc_state = drm_atomic_get_crtc_state(state, crtc); + if (IS_ERR(crtc_state)) { + ret = PTR_ERR(crtc_state); + goto out; + } + + if (!dcp->brightness.update) + goto out; + + /* Re-present the current scanout when brightness is the only change. */ + ret = drm_atomic_add_affected_planes(state, crtc); + if (ret) + goto out; + + crtc_state->color_mgmt_changed |= true; + + ret = drm_atomic_commit(state); + +out: + if (ret == -EDEADLK) { + drm_atomic_state_clear(state); + ret = drm_modeset_backoff(&ctx); + if (!ret) + goto retry; + } + + drm_atomic_state_put(state); + drm_modeset_drop_locks(&ctx); + drm_modeset_acquire_fini(&ctx); + + return ret; +} + int dcp_backlight_update(struct apple_dcp *dcp) { + if (READ_ONCE(dcp->quiescing)) + return -ENODEV; + if (dcp_backlight_active(dcp)) { + dcp_backlight_kick(dcp); + return 0; + } + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + return -ENODATA; + /* * Do not actively try to change brightness if no mode is set. * TODO: should this be reflected the in backlight's power property? @@ -202,9 +511,57 @@ int dcp_backlight_update(struct apple_dcp *dcp) * the brightness */ msleep((1001 + 23) / 24); // 42ms for 23.976 fps + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return drm_crtc_set_brightness_h17p(dcp); return drm_crtc_set_brightness(dcp); } +static int dcp_set_brightness_h17p(struct backlight_device *bd) +{ + int ret = 0; + struct apple_dcp *dcp = bl_get_data(bd); + int brightness = backlight_get_brightness(bd); + unsigned long flags; + + if (READ_ONCE(dcp->quiescing)) + return -ENODEV; + + if (dcp_backlight_active(dcp)) { + struct dcp_backlight_state *state = &dcp->backlight.state; + bool fade; + u32 old; + + /* Preserve the requested level even while the core forces zero. */ + spin_lock_irqsave(&dcp->backlight.lock, flags); + old = state->target; + ret = dcp_bl_request(state, bd->props.brightness, + backlight_is_blank(bd), + bd->props.state & BL_CORE_SUSPENDED); + /* A new level restarts the fade from the level presented last. */ + fade = !ret && state->fading && state->target != old; + if (fade) + dcp->backlight.fade_start = jiffies; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + if (fade) + queue_delayed_work(system_wq, &dcp->backlight.fade_work, + msecs_to_jiffies(DCP_BL_FADE_STEP_MS)); + else if (!ret) + dcp_backlight_kick(dcp); + return ret; + } + + ret = drm_modeset_lock_single_interruptible(&dcp->crtc->base.mutex); + if (ret) + return ret; + + dcp->brightness.dac = calculate_dac(dcp, brightness); + dcp->brightness.update = true; + + drm_modeset_unlock(&dcp->crtc->base.mutex); + + return dcp_backlight_update(dcp); +} + static int dcp_set_brightness(struct backlight_device *bd) { int ret = 0; @@ -212,6 +569,9 @@ static int dcp_set_brightness(struct backlight_device *bd) struct drm_modeset_acquire_ctx ctx; int brightness = backlight_get_brightness(bd); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_set_brightness_h17p(bd); + drm_modeset_acquire_init(&ctx, DRM_MODESET_ACQUIRE_INTERRUPTIBLE); ret = drm_modeset_lock(&dcp->crtc->base.mutex, &ctx); if (ret == -EDEADLK) { @@ -232,30 +592,67 @@ static int dcp_set_brightness(struct backlight_device *bd) return dcp_backlight_update(dcp); } +static bool dcp_backlight_controls_device(struct backlight_device *bd, + struct device *display_dev) +{ + struct apple_dcp *dcp = bl_get_data(bd); + + return !display_dev || display_dev == dcp->dev || + (dcp->crtc && display_dev == dcp->crtc->base.dev->dev); +} + static const struct backlight_ops dcp_backlight_ops = { .options = BL_CORE_SUSPENDRESUME, .get_brightness = dcp_get_brightness, .update_status = dcp_set_brightness, }; +/* H17P: fbdev blanking follows only this display. */ +static const struct backlight_ops dcp_backlight_ops_h17p = { + .options = BL_CORE_SUSPENDRESUME, + .get_brightness = dcp_get_brightness, + .update_status = dcp_set_brightness, + .controls_device = dcp_backlight_controls_device, +}; + int dcp_backlight_register(struct apple_dcp *dcp) { struct device *dev = dcp->dev; struct backlight_device *bl_dev; + unsigned long flags; struct backlight_properties props = { .type = BACKLIGHT_PLATFORM, .brightness = dcp->brightness.nits, .scale = BACKLIGHT_SCALE_LINEAR, }; props.max_brightness = min(dcp->brightness.maximum, MAX_BRIGHTNESS_PART2 - 1); + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !dcp_backlight_active(dcp)) + return -ENODATA; + if (dcp_backlight_active(dcp)) { + spin_lock_irqsave(&dcp->backlight.lock, flags); + if (!dcp->backlight.state.ready) { + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + return -ENODATA; + } + /* Freeze takeover before userspace can discover the device. */ + dcp->backlight.state.controlled = true; + props.brightness = dcp->backlight.state.target; + props.max_brightness = dcp->backlight.state.maximum; + spin_unlock_irqrestore(&dcp->backlight.lock, flags); + } bl_dev = devm_backlight_device_register(dev, "apple-panel-bl", dev, dcp, + dcp->fw_compat == DCP_FIRMWARE_H17P ? + &dcp_backlight_ops_h17p : &dcp_backlight_ops, &props); if (IS_ERR(bl_dev)) return PTR_ERR(bl_dev); dcp->brightness.bl_dev = bl_dev; - dcp->brightness.dac = calculate_dac(dcp, dcp->brightness.nits); + if (!dcp_backlight_active(dcp)) + dcp->brightness.dac = calculate_dac(dcp, dcp->brightness.nits); return 0; } diff --git a/drivers/gpu/drm/apple-neo/dcp_backlight.h b/drivers/gpu/drm/apple/dcp_backlight.h similarity index 56% rename from drivers/gpu/drm/apple-neo/dcp_backlight.h rename to drivers/gpu/drm/apple/dcp_backlight.h index 4b73a54fd29866..92ac07b2e9d9d4 100644 --- a/drivers/gpu/drm/apple-neo/dcp_backlight.h +++ b/drivers/gpu/drm/apple/dcp_backlight.h @@ -6,13 +6,20 @@ #include #include +#include #include #include -/* Commanded nits, independent of firmware encoding. No fade policy. */ -struct neo_dcp_backlight_state { +/* + * Commanded nits, independent of firmware encoding. target is the requested + * level and level the one presented while the panel is lit. They differ only + * while a fade runs; the caller supplies the fade's clock. + */ +struct dcp_backlight_state { u32 maximum; u32 target; + u32 level; + u32 fade_from; u32 actual; u32 sent_nits; u64 sequence; @@ -24,6 +31,10 @@ struct neo_dcp_backlight_state { bool suspended; bool dirty; bool in_flight; + /* Fade between lit levels instead of jumping to them. */ + bool fade; + /* A fade from fade_from to target is running. */ + bool fading; /* * The panel shows a level Linux knows: the loader's reported level, or * one a completed present carried. A default used for registration @@ -32,7 +43,7 @@ struct neo_dcp_backlight_state { bool level_known; }; -struct neo_dcp_backlight_present { +struct dcp_backlight_present { u64 sequence; u32 nits; }; @@ -42,7 +53,7 @@ struct neo_dcp_backlight_present { * ceiling describes the brightest level the panel can show, so clamp it; one * more than twice the ceiling is not a credible panel level. */ -static inline int neo_dcp_bl_takeover_nits(u32 maximum, u32 millinits, u32 *nits) +static inline int dcp_bl_takeover_nits(u32 maximum, u32 millinits, u32 *nits) { if (!maximum || maximum > INT_MAX / 2000) return -EINVAL; @@ -56,7 +67,7 @@ static inline int neo_dcp_bl_takeover_nits(u32 maximum, u32 millinits, u32 *nits * An inherited level comes from a loader report. A default level is only * reported to userspace; it is never presented on its own (see level_known). */ -static inline int neo_dcp_bl_init(struct neo_dcp_backlight_state *state, u32 maximum, +static inline int dcp_bl_init(struct dcp_backlight_state *state, u32 maximum, bool inherited_valid, u32 inherited, bool default_valid, u32 default_nits) { @@ -70,9 +81,10 @@ static inline int neo_dcp_bl_init(struct neo_dcp_backlight_state *state, u32 max if (nits > maximum) return -ERANGE; - *state = (struct neo_dcp_backlight_state) { + *state = (struct dcp_backlight_state) { .maximum = maximum, .target = nits, + .level = nits, .actual = nits, .ready = true, .level_known = inherited_valid, @@ -84,26 +96,45 @@ static inline int neo_dcp_bl_init(struct neo_dcp_backlight_state *state, u32 max * A decoded powerlog sample can seed takeover only before Linux control. * It reports the level the panel shows, so the level is then known. */ -static inline bool neo_dcp_bl_seed(struct neo_dcp_backlight_state *state, u32 nits) +static inline bool dcp_bl_seed(struct dcp_backlight_state *state, u32 nits) { if (!state->ready || state->controlled || nits > state->maximum) return false; state->target = nits; + state->level = nits; state->actual = nits; state->level_known = true; return true; } -static inline u32 neo_dcp_bl_effective(const struct neo_dcp_backlight_state *state) +static inline bool dcp_bl_lit(const struct dcp_backlight_state *state) { - if (state->dpms_off || state->core_blank || state->suspended) - return 0; - return state->target; + return !state->dpms_off && !state->core_blank && !state->suspended; } -static inline int neo_dcp_bl_request(struct neo_dcp_backlight_state *state, u32 nits, +static inline u32 dcp_bl_effective(const struct dcp_backlight_state *state) +{ + return dcp_bl_lit(state) ? state->level : 0; +} + +/* Stop a running fade and present the target directly. */ +static inline void dcp_bl_jump(struct dcp_backlight_state *state) +{ + state->fading = false; + state->level = state->target; + state->dirty = true; +} + +/* + * With fades enabled, a new level for a panel that stays lit at a known level + * starts a fade from the level presented last; dcp_bl_fade_step() moves it. + * Blanking, unblanking and a level the panel may not show still jump. + */ +static inline int dcp_bl_request(struct dcp_backlight_state *state, u32 nits, bool core_blank, bool suspended) { + bool was_lit = dcp_bl_lit(state); + if (!state->ready) return -ENODATA; if (nits > state->maximum) @@ -119,12 +150,45 @@ static inline int neo_dcp_bl_request(struct neo_dcp_backlight_state *state, u32 state->target = nits; state->core_blank = core_blank; state->suspended = suspended; - state->dirty = true; + if (state->fade && state->level_known && was_lit && + dcp_bl_lit(state) && state->level != nits) { + state->fade_from = state->level; + state->fading = true; + } else { + dcp_bl_jump(state); + } } return 0; } -static inline int neo_dcp_bl_dpms(struct neo_dcp_backlight_state *state, bool on) +/* + * Move a running fade to @elapsed of @duration, in any one unit. The level + * follows a straight line from the level presented when the fade started, + * and the last step is the target itself. Returns true while the fade runs. + */ +static inline bool dcp_bl_fade_step(struct dcp_backlight_state *state, + u32 elapsed, u32 duration) +{ + u32 from = state->fade_from, to = state->target, level; + + if (!state->fading) + return false; + if (elapsed >= duration) { + level = to; + state->fading = false; + } else if (to >= from) { + level = from + div_u64(mul_u32_u32(to - from, elapsed), duration); + } else { + level = from - div_u64(mul_u32_u32(from - to, elapsed), duration); + } + if (state->level != level) { + state->level = level; + state->dirty = true; + } + return state->fading; +} + +static inline int dcp_bl_dpms(struct dcp_backlight_state *state, bool on) { if (!state->ready) return -ENODATA; @@ -132,7 +196,7 @@ static inline int neo_dcp_bl_dpms(struct neo_dcp_backlight_state *state, bool on state->retries = 0; if (state->dpms_off != !on) { state->dpms_off = !on; - state->dirty = true; + dcp_bl_jump(state); } return 0; } @@ -142,7 +206,7 @@ static inline int neo_dcp_bl_dpms(struct neo_dcp_backlight_state *state, bool on * Linux controls it and knows it, unless a present already carries or will * carry it. A default level is never sent this way. */ -static inline bool neo_dcp_bl_resend(struct neo_dcp_backlight_state *state) +static inline bool dcp_bl_resend(struct dcp_backlight_state *state) { if (!state->ready || !state->controlled || !state->level_known || state->dirty || state->in_flight) @@ -156,9 +220,9 @@ static inline bool neo_dcp_bl_resend(struct neo_dcp_backlight_state *state) * prepared replacement. Soft-off never releases it. A brightness-only present * must use the last accepted surface, not a rejected replacement's DRM state. */ -static inline int neo_dcp_bl_prepare(struct neo_dcp_backlight_state *state, +static inline int dcp_bl_prepare(struct dcp_backlight_state *state, bool have_surface, - struct neo_dcp_backlight_present *present) + struct dcp_backlight_present *present) { if (!state->ready || !have_surface) return -ENODATA; @@ -170,7 +234,7 @@ static inline int neo_dcp_bl_prepare(struct neo_dcp_backlight_state *state, state->controlled = true; state->in_flight = true; state->dirty = false; - state->sent_nits = neo_dcp_bl_effective(state); + state->sent_nits = dcp_bl_effective(state); present->nits = state->sent_nits; present->sequence = ++state->sequence; return 0; @@ -181,7 +245,7 @@ static inline int neo_dcp_bl_prepare(struct neo_dcp_backlight_state *state, * Rejection/serialization failure cancels the reservation without changing * actual. This result is NEVER a framebuffer retirement authorization. */ -static inline bool neo_dcp_bl_complete(struct neo_dcp_backlight_state *state, +static inline bool dcp_bl_complete(struct dcp_backlight_state *state, u64 sequence, bool accepted) { if (!state->in_flight || sequence != state->sequence) @@ -200,7 +264,7 @@ static inline bool neo_dcp_bl_complete(struct neo_dcp_backlight_state *state, } /* Three deferred retries per user request; a permanent rejection stays dirty. */ -static inline unsigned int neo_dcp_bl_retry_delay(const struct neo_dcp_backlight_state *state) +static inline unsigned int dcp_bl_retry_delay(const struct dcp_backlight_state *state) { if (!state->ready || !state->dirty || state->in_flight || !state->retries || state->retries > 3) @@ -208,26 +272,26 @@ static inline unsigned int neo_dcp_bl_retry_delay(const struct neo_dcp_backlight return 100U << (state->retries - 1); } -struct neo_apple_dcp; +struct apple_dcp; /* * kick only queues work; it must not submit a top-level RPC from the caller. * The central IOMFB queue owns surface references, retries, and wire encoding. * Configure before registration and keep the callback valid until teardown. */ -bool neo_dcp_backlight_active(struct neo_apple_dcp *neo_dcp); -int neo_dcp_backlight_takeover(struct neo_apple_dcp *neo_dcp, u32 millinits); -int neo_dcp_backlight_configure(struct neo_apple_dcp *neo_dcp, u32 maximum, +bool dcp_backlight_active(struct apple_dcp *dcp); +int dcp_backlight_takeover(struct apple_dcp *dcp, u32 millinits); +int dcp_backlight_configure(struct apple_dcp *dcp, u32 maximum, bool inherited_valid, u32 inherited, bool default_valid, u32 default_nits, - void (*kick)(struct neo_apple_dcp *neo_dcp)); -bool neo_dcp_backlight_seed(struct neo_apple_dcp *neo_dcp, u32 nits); -int neo_dcp_backlight_dpms(struct neo_apple_dcp *neo_dcp, bool on); -int neo_dcp_backlight_prepare(struct neo_apple_dcp *neo_dcp, bool have_surface, - struct neo_dcp_backlight_present *present); -bool neo_dcp_backlight_complete(struct neo_apple_dcp *neo_dcp, u64 sequence, bool accepted); -bool neo_dcp_backlight_pending(struct neo_apple_dcp *neo_dcp); -bool neo_dcp_backlight_resend(struct neo_apple_dcp *neo_dcp); -unsigned int neo_dcp_backlight_retry_delay(struct neo_apple_dcp *neo_dcp); + void (*kick)(struct apple_dcp *dcp)); +bool dcp_backlight_seed(struct apple_dcp *dcp, u32 nits); +int dcp_backlight_dpms(struct apple_dcp *dcp, bool on); +int dcp_backlight_prepare(struct apple_dcp *dcp, bool have_surface, + struct dcp_backlight_present *present); +bool dcp_backlight_complete(struct apple_dcp *dcp, u64 sequence, bool accepted); +bool dcp_backlight_pending(struct apple_dcp *dcp); +bool dcp_backlight_resend(struct apple_dcp *dcp); +unsigned int dcp_backlight_retry_delay(struct apple_dcp *dcp); #endif /* __APPLE_DCP_BACKLIGHT_H__ */ diff --git a/drivers/gpu/drm/apple/dcp_backlight_test.c b/drivers/gpu/drm/apple/dcp_backlight_test.c new file mode 100644 index 00000000000000..a6fb45958b1684 --- /dev/null +++ b/drivers/gpu/drm/apple/dcp_backlight_test.c @@ -0,0 +1,531 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Ryan Murray */ + +#include +#include + +#include "dcp_backlight.h" + +static void backlight_millinits_takeover_test(struct kunit *test) +{ + struct dcp_backlight_state state; + u32 nits = 123; + + KUNIT_EXPECT_EQ(test, dcp_bl_takeover_nits(0, 100000, &nits), -EINVAL); + KUNIT_EXPECT_EQ(test, dcp_bl_takeover_nits(U32_MAX, 100000, &nits), -EINVAL); + KUNIT_EXPECT_EQ(test, nits, 123U); + /* Reports above the panel ceiling clamp to it. */ + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(525, 525999, &nits), 0); + KUNIT_EXPECT_EQ(test, nits, 525U); + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(509, 509001, &nits), 0); + KUNIT_EXPECT_EQ(test, nits, 509U); + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(509, 1018999, &nits), 0); + KUNIT_EXPECT_EQ(test, nits, 509U); + /* More than twice the ceiling is not a panel level. */ + nits = 123; + KUNIT_EXPECT_EQ(test, dcp_bl_takeover_nits(509, 1019000, &nits), -ERANGE); + KUNIT_EXPECT_EQ(test, dcp_bl_takeover_nits(509, U32_MAX, &nits), -ERANGE); + KUNIT_EXPECT_EQ(test, nits, 123U); + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(509, 100999, &nits), 0); + KUNIT_EXPECT_EQ(test, nits, 100U); + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 509, true, nits, false, 0), 0); + KUNIT_EXPECT_EQ(test, state.actual, 100U); + KUNIT_EXPECT_FALSE(test, state.dirty); + /* Registration freezes the hint before exposing the controls. */ + state.controlled = true; + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 400)); + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 0)); + KUNIT_EXPECT_EQ(test, state.target, 100U); + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(509, 0, &nits), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 509, true, nits, false, 0), 0); + KUNIT_EXPECT_EQ(test, state.target, 0U); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_ASSERT_EQ(test, dcp_bl_takeover_nits(509, 509000, &nits), 0); + KUNIT_EXPECT_EQ(test, nits, 509U); +} + +static void backlight_takeover_test(struct kunit *test) +{ + struct dcp_backlight_state state = {}; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, true, 36), 0); + KUNIT_EXPECT_EQ(test, state.target, 140U); + KUNIT_EXPECT_EQ(test, state.actual, 140U); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_EXPECT_TRUE(test, dcp_bl_seed(&state, 400)); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 601)); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 0)); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_EXPECT_EQ(test, state.actual, 400U); + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 0, true, 36), 0); + KUNIT_EXPECT_EQ(test, state.target, 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, false, 0, true, 36), 0); + KUNIT_EXPECT_EQ(test, state.target, 36U); +} + +static void backlight_range_test(struct kunit *test) +{ + struct dcp_backlight_state state = {}; + + KUNIT_EXPECT_EQ(test, dcp_bl_init(&state, 600, false, 0, false, 0), -ENODATA); + KUNIT_EXPECT_FALSE(test, state.ready); + KUNIT_EXPECT_EQ(test, dcp_bl_request(&state, 140, false, false), -ENODATA); + KUNIT_EXPECT_EQ(test, dcp_bl_dpms(&state, false), -ENODATA); + KUNIT_EXPECT_EQ(test, dcp_bl_init(&state, 0, true, 0, false, 0), -EINVAL); + KUNIT_EXPECT_EQ(test, dcp_bl_init(&state, U32_MAX, true, 0, false, 0), -EINVAL); + KUNIT_EXPECT_EQ(test, dcp_bl_init(&state, 600, true, 601, false, 0), -ERANGE); + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, false, 0), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_request(&state, 601, false, false), -ERANGE); + KUNIT_EXPECT_EQ(test, state.target, 140U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 600, false, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 600U); +} + +static void backlight_dpms_cycles_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + int i; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 400, false, 0), 0); + for (i = 0; i < 20; i++) { + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 0U); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 400U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 400U); + } + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_EXPECT_EQ(test, state.target, 400U); +} + +static void backlight_blanked_target_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 400, false, 0), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 140, false, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.target, 140U); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 140U); + KUNIT_EXPECT_EQ(test, state.actual, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 140U); +} + +static void backlight_suspend_test(struct kunit *test) +{ + struct dcp_backlight_state state; + int off_first; + + for (off_first = 0; off_first < 2; off_first++) { + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, false, 0), 0); + if (off_first) + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 140, true, true), 0); + if (!off_first) + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 36, true, true), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 36, false, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 36U); + KUNIT_EXPECT_EQ(test, state.target, 36U); + } + /* A core blank also keeps the panel dark independently of DRM DPMS. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, true, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 400U); +} + +static void backlight_surface_and_busy_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present first, second; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, false, 0), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, true, &first), -EALREADY); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 36, false, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, false, &first), -ENODATA); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_EXPECT_FALSE(test, state.in_flight); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &first), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, true, &second), -EBUSY); + KUNIT_EXPECT_EQ(test, state.actual, 140U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, first.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 36U); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &second), 0); + KUNIT_EXPECT_EQ(test, second.nits, 400U); + KUNIT_EXPECT_FALSE(test, dcp_bl_complete(&state, first.sequence, true)); + KUNIT_EXPECT_TRUE(test, state.in_flight); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, second.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 400U); +} + +static void backlight_reject_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, false, 0), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, false)); + KUNIT_EXPECT_EQ(test, state.actual, 140U); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_EXPECT_FALSE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_EXPECT_EQ(test, state.actual, 0U); +} + +static void backlight_inflight_dpms_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 140, false, 0), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 140U); +} + +static void backlight_resend_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + /* Nothing is re-sent before Linux controls the level. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, true, 140, false, 0), 0); + KUNIT_EXPECT_FALSE(test, dcp_bl_resend(&state)); + KUNIT_EXPECT_FALSE(test, state.dirty); + + state.controlled = true; + KUNIT_EXPECT_TRUE(test, dcp_bl_resend(&state)); + KUNIT_EXPECT_FALSE(test, dcp_bl_resend(&state)); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 140U); + /* The present in flight already carries the level. */ + KUNIT_EXPECT_FALSE(test, dcp_bl_resend(&state)); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_FALSE(test, state.dirty); + + /* A blanked panel is re-sent dark, keeping the restore target. */ + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_TRUE(test, dcp_bl_resend(&state)); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_EQ(test, state.target, 140U); +} + +static void backlight_fallback_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + /* A registration default is reported, never presented on its own. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + KUNIT_EXPECT_FALSE(test, state.level_known); + state.controlled = true; + KUNIT_EXPECT_FALSE(test, dcp_bl_resend(&state)); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, true, &present), -EALREADY); + + /* A rejected userspace level does not become known. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 300, false, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 300U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, false)); + KUNIT_EXPECT_FALSE(test, state.level_known); + + /* Once a present carrying it completes, the level is re-sendable. */ + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_TRUE(test, state.level_known); + KUNIT_EXPECT_TRUE(test, dcp_bl_resend(&state)); + + /* DPMS has to light the panel at some level: wake uses the default. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + state.controlled = true; + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 262U); +} + +static void backlight_fallback_same_level_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + /* A userspace write of the reported default is still presented. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 262, false, false), 0); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 262U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_TRUE(test, state.level_known); + + /* Once the level is known, repeating it presents nothing. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 262, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, true, &present), -EALREADY); + + /* A rejected present leaves it unknown, so the same write is retried. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 262, false, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, false)); + KUNIT_EXPECT_FALSE(test, state.level_known); + state.dirty = false; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 262, false, false), 0); + KUNIT_EXPECT_TRUE(test, state.dirty); + + /* An inherited level is known from the start. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, true, 180, false, 0), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 180, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.dirty); +} + +static void backlight_seed_known_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + /* A loader sample seeded over a registration default is known. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + KUNIT_EXPECT_FALSE(test, state.level_known); + KUNIT_EXPECT_TRUE(test, dcp_bl_seed(&state, 180)); + KUNIT_EXPECT_TRUE(test, state.level_known); + state.controlled = true; + KUNIT_EXPECT_TRUE(test, dcp_bl_resend(&state)); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 180U); + + /* A refused sample leaves the default unknown. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 526)); + KUNIT_EXPECT_FALSE(test, state.level_known); + state.controlled = true; + KUNIT_EXPECT_FALSE(test, dcp_bl_seed(&state, 180)); + KUNIT_EXPECT_FALSE(test, state.level_known); + KUNIT_EXPECT_FALSE(test, dcp_bl_resend(&state)); +} + +static void backlight_fade_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 100, false, 0), 0); + state.fade = true; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_TRUE(test, state.fading); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_EXPECT_EQ(test, state.target, 400U); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 100U); + + /* The start of the fade is the level already shown. */ + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 0, 250)); + KUNIT_EXPECT_FALSE(test, state.dirty); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 125, 250)); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 250U); + + /* Steps during a present merge into the next one. */ + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 150, 250)); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 200, 250)); + KUNIT_EXPECT_EQ(test, dcp_bl_prepare(&state, true, &present), -EBUSY); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 250U); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 340U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + + /* The last step is exactly the target, however late it runs. */ + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 1000, 250)); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 400U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_EXPECT_EQ(test, state.actual, 400U); + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 250, 250)); + KUNIT_EXPECT_FALSE(test, state.dirty); + + /* Fading down reaches the target too, including zero. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 0, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 249, 250)); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 2U); + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 250, 250)); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); +} + +static void backlight_fade_restart_test(struct kunit *test) +{ + struct dcp_backlight_state state; + + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 100, false, 0), 0); + state.fade = true; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 500, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 100, 250)); + KUNIT_EXPECT_EQ(test, state.level, 260U); + + /* Repeating the target keeps the running fade. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 500, false, false), 0); + KUNIT_EXPECT_TRUE(test, state.fading); + KUNIT_EXPECT_EQ(test, state.fade_from, 100U); + + /* A new level replaces it, starting from the level presented last. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 60, false, false), 0); + KUNIT_EXPECT_TRUE(test, state.fading); + KUNIT_EXPECT_EQ(test, state.fade_from, 260U); + KUNIT_EXPECT_EQ(test, state.level, 260U); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 125, 250)); + KUNIT_EXPECT_EQ(test, state.level, 160U); + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 250, 250)); + KUNIT_EXPECT_EQ(test, state.level, 60U); + + /* A new level equal to the one presented ends the fade there. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 300, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 125, 250)); + KUNIT_EXPECT_EQ(test, state.level, 180U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 180, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 180U); + + /* No overflow over the whole range a panel ceiling may have. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, INT_MAX, true, 0, false, 0), 0); + state.fade = true; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, INT_MAX, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 249, 250)); + KUNIT_EXPECT_EQ(test, state.level, (u32)div_u64((u64)INT_MAX * 249, 250)); + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 250, 250)); + KUNIT_EXPECT_EQ(test, state.level, (u32)INT_MAX); +} + +static void backlight_fade_blank_test(struct kunit *test) +{ + struct dcp_backlight_state state; + struct dcp_backlight_present present; + + /* DPMS off and on during a fade jump, as without fades. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 100, false, 0), 0); + state.fade = true; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 125, 250)); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_TRUE(test, state.dirty); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_EXPECT_FALSE(test, dcp_bl_fade_step(&state, 200, 250)); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 0U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + KUNIT_ASSERT_EQ(test, dcp_bl_dpms(&state, true), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 400U); + KUNIT_EXPECT_TRUE(test, dcp_bl_complete(&state, present.sequence, true)); + + /* A level set while blanked is shown at once on unblank. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, true, false), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 200, true, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 200, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 200U); + + /* Suspend during a fade blanks at once; resume shows the target. */ + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 500, false, false), 0); + KUNIT_EXPECT_TRUE(test, dcp_bl_fade_step(&state, 50, 250)); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 500, true, true), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 0U); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 500, false, false), 0); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 500U); + + /* A level the panel may not show yet is presented directly. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 525, false, 0, true, 262), 0); + state.fade = true; + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 100, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_ASSERT_EQ(test, dcp_bl_prepare(&state, true, &present), 0); + KUNIT_EXPECT_EQ(test, present.nits, 100U); + + /* Without fades a new level is presented directly. */ + KUNIT_ASSERT_EQ(test, dcp_bl_init(&state, 600, true, 100, false, 0), 0); + KUNIT_ASSERT_EQ(test, dcp_bl_request(&state, 400, false, false), 0); + KUNIT_EXPECT_FALSE(test, state.fading); + KUNIT_EXPECT_EQ(test, dcp_bl_effective(&state), 400U); +} + +static struct kunit_case backlight_cases[] = { + KUNIT_CASE(backlight_millinits_takeover_test), + KUNIT_CASE(backlight_takeover_test), + KUNIT_CASE(backlight_range_test), + KUNIT_CASE(backlight_dpms_cycles_test), + KUNIT_CASE(backlight_blanked_target_test), + KUNIT_CASE(backlight_suspend_test), + KUNIT_CASE(backlight_surface_and_busy_test), + KUNIT_CASE(backlight_reject_test), + KUNIT_CASE(backlight_inflight_dpms_test), + KUNIT_CASE(backlight_resend_test), + KUNIT_CASE(backlight_fallback_test), + KUNIT_CASE(backlight_fallback_same_level_test), + KUNIT_CASE(backlight_seed_known_test), + KUNIT_CASE(backlight_fade_test), + KUNIT_CASE(backlight_fade_restart_test), + KUNIT_CASE(backlight_fade_blank_test), + {} +}; + +static struct kunit_suite backlight_suite = { + .name = "apple-dcp-backlight", + .test_cases = backlight_cases, +}; + +kunit_test_suite(backlight_suite); + +MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/gpu/drm/apple/dptx-j700-training.h b/drivers/gpu/drm/apple/dptx-j700-training.h new file mode 100644 index 00000000000000..a4632cd363abdf --- /dev/null +++ b/drivers/gpu/drm/apple/dptx-j700-training.h @@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* Copyright 2026 Aurora Silicon */ +#ifndef APPLE_DPTX_J700_TRAINING_H +#define APPLE_DPTX_J700_TRAINING_H + +/* Active lanes may be zero during the firmware's training reset. */ +static int dptx_j700_validate_lanes(struct dptx_port *port, u64 requested) +{ + union phy_configure_opts capacity = {}; + int ret; + + if (!port || !port->atcphy) + return -ENODEV; + if (requested > 4 || requested == 3) + return -EINVAL; + ret = phy_validate(port->atcphy, PHY_MODE_DP, 0, &capacity); + if (ret) + return ret; + return requested <= capacity.dp.lanes ? 0 : -EINVAL; +} + +static int dptx_j700_training(struct apple_epic_service *service, bool write, + const void *data, size_t size, void *reply, size_t room) +{ + struct dptx_port *port = service->cookie; + union phy_configure_opts options = {}; + u32 count; + int ret; + + if (!port || !port->atcphy) + return -ENODEV; + if (size < 32 || room < 32) + return -EINVAL; + count = get_unaligned_le32(data + (write ? 16 : 0)); + if (count > 4 || (write && (!count || count == 3)) || room < 32 + 12 * count || + (write && size < 32 + 12 * count)) + return -EINVAL; + if (write) { + ret = phy_validate(port->atcphy, PHY_MODE_DP, 0, &options); + if (ret) + return ret; + if (count > options.dp.lanes) + return -EINVAL; + for (unsigned int i = 0; i < count; i++) { + const u8 *entry = data + 32 + 12 * i; + + if (get_unaligned_le32(entry) || get_unaligned_le32(entry + 4) > 3 || + get_unaligned_le32(entry + 8) > 3) + return -EINVAL; + options.dp.voltage[i] = get_unaligned_le32(entry + 4); + options.dp.pre[i] = get_unaligned_le32(entry + 8); + } + options.dp.lanes = count; + options.dp.set_voltages = true; + ret = phy_configure(port->atcphy, &options); + if (ret) + return ret; + memcpy(port->trained_voltage, options.dp.voltage, sizeof(port->trained_voltage)); + memcpy(port->trained_pre, options.dp.pre, sizeof(port->trained_pre)); + } + memset(reply, 0, room); + put_unaligned_le32(count, reply + (write ? 16 : 0)); + for (unsigned int i = 0; i < count; i++) { + put_unaligned_le32(port->trained_voltage[i], reply + 36 + 12 * i); + put_unaligned_le32(port->trained_pre[i], reply + 40 + 12 * i); + } + return 0; +} + +#endif diff --git a/drivers/gpu/drm/apple/dptx-service.h b/drivers/gpu/drm/apple/dptx-service.h new file mode 100644 index 00000000000000..3d990a9ca1ea11 --- /dev/null +++ b/drivers/gpu/drm/apple/dptx-service.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DPTX_SERVICE_H +#define APPLE_DPTX_SERVICE_H + +#include "afk.h" +#include "dptxep.h" + +static inline int dptx_service_port(const char *name, const char *class, + s64 unit, bool wire_v2, u32 index, + u32 num_ports) +{ + if (!name || !class || strcmp(name, "dcpdptx-port-epic") || + strcmp(class, "AppleDCPDPTXRemotePort")) + return -EINVAL; + if (wire_v2) + return num_ports == 1 && unit == index ? 0 : -EINVAL; + return unit >= 0 && unit < 2 ? unit : -EINVAL; +} + +static inline bool dptx_service_available(struct dptx_port *port) +{ + /* Pairs with publication before readers use the service and its cookie. */ + return smp_load_acquire(&port->enabled); +} + +static inline int dptx_service_bind(struct apple_epic_service *service, + struct dptx_port *port, s64 unit) +{ + if (port->service) + return -EBUSY; + port->unit = unit; + /* No owner is published until OPEN and AFK publication succeed. */ + service->cookie = port; + return 0; +} + +static inline void dptx_service_publish(struct apple_epic_service *service) +{ + struct dptx_port *port = service->cookie; + + port->service = service; + port->connected = false; + /* Pair with dcp.c readers before they dereference the service cookie. */ + smp_store_release(&port->enabled, true); +} + +static inline void dptx_service_unbind(struct apple_epic_service *service) +{ + struct dptx_port *port = service->cookie; + + if (port && port->service == service) { + WRITE_ONCE(port->closed, true); + /* Make terminal state visible to readers of availability. */ + smp_store_release(&port->enabled, false); + WRITE_ONCE(port->connected, false); + reinit_completion(&port->enable_completion); + } + /* + * Keep the published owner and cookie valid for retained callers. Once + * this session closes, bind fails closed until reboot: + * an in-flight connect must never switch to a new firmware session. + */ +} + +#endif diff --git a/drivers/gpu/drm/apple/dptxep.c b/drivers/gpu/drm/apple/dptxep.c index 2fc97c208d8f39..f1953d06f42869 100644 --- a/drivers/gpu/drm/apple/dptxep.c +++ b/drivers/gpu/drm/apple/dptxep.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -17,9 +18,23 @@ #include "dcp.h" #include "dcp-hdmi.h" #include "dptxep.h" +#include "dcp-180hz.h" +#include "dptx-service.h" #include "parser.h" #include "trace.h" +#include "dptx-j700-training.h" + +/* + * The 1080p180 rate proof (dcp-180hz.c) belongs to H17-generation + * firmware, whose probe sets up its lock; other firmware has none. + */ +static void dptx_180hz_invalidate(struct apple_dcp *dcp) +{ + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + dcp_180hz_invalidate(dcp); +} + struct dcpdptx_connection_cmd { __le32 unk; __le32 target; @@ -230,6 +245,7 @@ int dptxport_release_display(struct apple_epic_service *service) struct dptx_port *dptx = service->cookie; int ret; + dptx_180hz_invalidate(service->ep->dcp); dptx->release_calls++; dev_dbg(service->ep->dcp->dev, "DPTX release_display: call #%u this boot caller=%pS\n", @@ -286,6 +302,13 @@ dptxport_call_get_max_drive_settings(struct apple_epic_service *service, return -EINVAL; reply->retcode = cpu_to_le32(0); + if (service->ep->wire_v2) { + memset(reply, 0, reply_size); + put_unaligned_le32(3, reply_ + 20); + put_unaligned_le32(3, reply_ + 24); + return 0; + } + reply->max_drive_settings[0] = cpu_to_le32(0x3); reply->max_drive_settings[1] = cpu_to_le32(0x3); @@ -427,7 +450,11 @@ static int dptxport_call_set_active_lane_count(struct apple_epic_service *servic u64 lane_count = le64_to_cpu(request->lane_count); - if (dptx->lane_count < lane_count) + if (service->ep->wire_v2) { + ret = dptx_j700_validate_lanes(dptx, lane_count); + if (ret) + return ret; + } else if (dptx->lane_count < lane_count) dev_err(dcp->dev, "set_active_lane_count: unexpected lane " "count:%llu phy: %d\n", lane_count, dptx->lane_count); @@ -437,7 +464,7 @@ static int dptxport_call_set_active_lane_count(struct apple_epic_service *servic dptx->phy_ops.dp.lanes = lane_count; /* USB4 DP IN has no ATC PHY; still accept the lane count. */ dptx->phy_ops.dp.set_lanes = - dcp_is_usb4_output(dcp) || (dcp->dptx_phy > 3); + service->ep->wire_v2 || dcp_is_usb4_output(dcp) || (dcp->dptx_phy > 3); break; default: dev_err(dcp->dev, "set_active_lane_count: invalid lane count:%llu\n", lane_count); @@ -507,6 +534,7 @@ dptxport_call_will_change_link_config(struct apple_epic_service *service) { struct dptx_port *dptx = service->cookie; + dptx_180hz_invalidate(service->ep->dcp); dptx->phy_ops.dp.set_lanes = 0; dptx->phy_ops.dp.set_rate = 0; dptx->phy_ops.dp.set_voltages = 0; @@ -726,6 +754,7 @@ dptxport_call_deactivate(struct apple_epic_service *service, struct dptx_port *dptx = service->cookie; struct apple_dcp *dcp = service->ep->dcp; + dptx_180hz_invalidate(dcp); dev_info(dcp->dev, "DPTXPort: DEACTIVATE\n"); if (dcp->dptx_tunnel) dcp_tunnel_dpin_activate(dcp, dptx->unit, false); @@ -741,15 +770,58 @@ dptxport_call_deactivate(struct apple_epic_service *service, return 0; } +static int dptxport_call_link_config(struct apple_epic_service *service, bool lanes, + const void *data, size_t size, void *reply, size_t room) +{ + struct dcp_180hz_token token; + struct dptx_port *port = service->cookie; + bool success = false; + u32 value = 0; + int ret; + + if (service->ep->dcp->fw_compat != DCP_FIRMWARE_H17P) + return lanes ? + dptxport_call_set_active_lane_count(service, data, size, + reply, room) : + dptxport_call_set_link_rate(service, data, size, reply, + room); + + token = dcp_180hz_config_begin(service, lanes); + if (lanes) { + ret = dptxport_call_set_active_lane_count(service, data, size, reply, room); + if (!ret && size >= sizeof(struct dptxport_apcall_set_active_lane_count) && + room >= sizeof(struct dptxport_apcall_set_active_lane_count)) { + const struct dptxport_apcall_set_active_lane_count *request = data; + const struct dptxport_apcall_set_active_lane_count *response = reply; + + value = le64_to_cpu(request->lane_count); + success = !le32_to_cpu(response->retcode) && value == 2 && port->lane_count == 2; + } + } else { + ret = dptxport_call_set_link_rate(service, data, size, reply, room); + if (!ret && size >= sizeof(struct dptxport_apcall_link_rate) && + room >= sizeof(struct dptxport_apcall_link_rate)) { + const struct dptxport_apcall_link_rate *request = data; + const struct dptxport_apcall_link_rate *response = reply; + + value = le32_to_cpu(request->link_rate); + success = !le32_to_cpu(response->retcode) && value == LINK_RATE_HBR3 && port->link_rate == value; + } + } + dcp_180hz_config_complete(service, lanes, token, value, success); + return ret; +} + static int dptxport_call(struct apple_epic_service *service, u32 idx, const void *data, size_t data_size, void *reply, size_t reply_size) { struct dptx_port *dptx = service->cookie; - trace_dptxport_apcall(dptx, idx, data_size); + trace_dptxport_apcall(dptx, 0, idx, data_size); dev_dbg(service->ep->dcp->dev, "DPTXPort: APCALL %u (%zu bytes)\n", idx, data_size); - if (data_size) + if (data_size && service->ep->dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) print_hex_dump_debug("DPTXPort: apcall data: ", DUMP_PREFIX_OFFSET, 16, 1, data, min(data_size, (size_t)64), true); @@ -783,8 +855,8 @@ static int dptxport_call(struct apple_epic_service *service, u32 idx, case DPTX_APCALL_GET_LINK_RATE: return dptxport_call_get_link_rate(service, reply, reply_size); case DPTX_APCALL_SET_LINK_RATE: - return dptxport_call_set_link_rate(service, data, data_size, - reply, reply_size); + return dptxport_call_link_config(service, false, data, data_size, + reply, reply_size); case DPTX_APCALL_GET_MAX_LANE_COUNT: return dptxport_call_get_max_lane_count(service, reply, reply_size); case DPTX_APCALL_GET_ACTIVE_LANE_COUNT: { @@ -797,8 +869,8 @@ static int dptxport_call(struct apple_epic_service *service, u32 idx, return 0; } case DPTX_APCALL_SET_ACTIVE_LANE_COUNT: - return dptxport_call_set_active_lane_count(service, data, data_size, - reply, reply_size); + return dptxport_call_link_config(service, true, data, data_size, + reply, reply_size); case DPTX_APCALL_GET_SUPPORTS_HPD: return dptxport_call_get_supports_hpd(service, reply, reply_size); @@ -811,6 +883,7 @@ static int dptxport_call(struct apple_epic_service *service, u32 idx, case DPTX_APCALL_DEVICE_NOT_RESPONDING: case DPTX_APCALL_DEVICE_BUSY_TIMEOUT: case DPTX_APCALL_DEVICE_NOT_STARTED: + dptx_180hz_invalidate(service->ep->dcp); dev_warn(service->ep->dcp->dev, "DPTXPort: firmware reports link fault %u on target %u:%u\n", idx, service->ep->dcp->dptx_die, @@ -862,6 +935,9 @@ static int dptxport_call(struct apple_epic_service *service, u32 idx, memset(reply, 0, 4); return 0; default: + if (service->ep->dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) + return -EOPNOTSUPP; /* just try to ACK and hope for the best... */ dev_info(service->ep->dcp->dev, "DPTXPort: acking unhandled call %u\n", idx); @@ -875,6 +951,20 @@ static int dptxport_call(struct apple_epic_service *service, u32 idx, } } +static int dptxport_prepare(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + struct apple_dcp *dcp = service->ep->dcp; + int port = dptx_service_port(name, class, unit, service->ep->wire_v2, + dcp->index, dcp->hw.num_dptx_ports); + + if (port < 0) + return port; + trace_dptxport_init(dcp, unit); + return dptx_service_bind(service, &dcp->dptxport[port], unit); +} + +/* M1/M2/M3 (12.3, 13.5 and 14.7 firmware): integration's port binding. */ static void dptxport_init(struct apple_epic_service *service, const char *name, const char *class, s64 unit) { @@ -916,6 +1006,115 @@ static void dptxport_init(struct apple_epic_service *service, const char *name, } } +/* H17-generation firmware: dptxport_prepare() bound the port before OPEN. */ +static void dptxport_h17_init(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + struct dptx_port *port = service->cookie; + + dptx_service_publish(service); + complete(&port->enable_completion); + /* Publication may follow the startup wait or the bounded retry window. */ + if (service->ep->dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + READ_ONCE(service->ep->dcp->typec_cable_connected) && + !READ_ONCE(service->ep->dcp->quiescing) && + !READ_ONCE(service->ep->dcp->external_error)) + mod_delayed_work(system_freezable_wq, + &service->ep->dcp->typec_reconnect_wq, 0); +} + +static int dptxport_v2_group_call(struct apple_epic_service *service, u16 group, u32 command, + const void *data, size_t size, void *reply, size_t room) +{ + if (group == 1 && command == DPTX_APCALL_SET_DRIVE_SETTINGS) + return dptx_j700_training(service, true, data, size, reply, room); + if (group == 1 && command == DPTX_APCALL_GET_DRIVE_SETTINGS) + return dptx_j700_training(service, false, data, size, reply, room); + if (group) + return -EOPNOTSUPP; + if (command == DPTX_APCALL_SET_DRIVE_SETTINGS) + return -EOPNOTSUPP; + if (command == DPTX_APCALL_GET_VIRTUAL_DEVICE_MODE) { + if (room < sizeof(struct dptxport_apcall_get_support)) + return -EINVAL; + memset(reply, 0, room); + return 0; + } + if (command == DPTX_APCALL_GET_DRIVE_SETTINGS) + return dptx_j700_training(service, false, data, size, reply, room); + return dptxport_call(service, command, data, size, reply, room); +} + +/* + * Group 1 calls 3 and 4 come around link (re)configuration, audio prepare + * included. They carry a lane count at +0x10 in (1, 3) and at +0x00 in + * (1, 4); the J700 code reads (1, 3) as drive settings with levels per + * lane from +0x20, and its group 0 GET (0, 4) has the count at +0x00. + * macOS 13 acks both with the request, (1, 3) with +0x00..0x03 + * cleared and (1, 4) with +0x10..0x13 and +0x20 to the end cleared, so + * what DCP left there does not go back (42 of 42 traced pairs on t6020, + * 13.5 firmware: TV through a USB-C to HDMI adapter, Dell over TB). + */ +static int dptxport_call_group1(u32 command, const void *data, + size_t data_size, void *reply, size_t reply_size) +{ + size_t size = min(reply_size, data_size); + + memcpy(reply, data, size); + if (command == 3) { + memset(reply, 0, min_t(size_t, size, 0x4)); + } else if (size > 0x10) { + memset(reply + 0x10, 0, min_t(size_t, size - 0x10, 0x4)); + if (size > 0x20) + memset(reply + 0x20, 0, size - 0x20); + } + return 0; +} + +static int dptxport_group_call(struct apple_epic_service *service, u16 group, + u32 command, const void *data, size_t size, + void *reply, size_t room) +{ + struct dptx_port *dptx = service->cookie; + + /* Keep native AFK-v2 training and its refusal policy distinct. */ + if (service->ep->wire_v2) + return dptxport_v2_group_call(service, group, command, data, + size, reply, room); + + if (!group) + return dptxport_call(service, command, data, size, reply, room); + + trace_dptxport_apcall(dptx, group, command, size); + dev_dbg(service->ep->dcp->dev, "DPTXPort: APCALL %u:%u (%zu bytes)\n", + group, command, size); + if (size) + print_hex_dump_debug("DPTXPort: apcall data: ", + DUMP_PREFIX_OFFSET, 16, 1, data, + min(size, (size_t)64), true); + + if (group == 1 && (command == 3 || command == 4)) + return dptxport_call_group1(command, data, size, reply, room); + + /* + * Echo it, as group 0 acks an unhandled call: no reply might stall + * DCP's caller. + */ + dev_warn_ratelimited(service->ep->dcp->dev, + "DPTXPort: unknown AP call %u:%u, echoing it\n", + group, command); + memcpy(reply, data, min(room, size)); + return 0; +} + +static void dptxport_teardown(struct apple_epic_service *service) +{ + dcp_180hz_invalidate(service->ep->dcp); + dptx_service_unbind(service); +} + +/* 12.3 and 13.5 firmware. */ static const struct apple_epic_service_ops dptxep_ops[] = { { .name = "AppleDCPDPTXRemotePort", @@ -925,6 +1124,33 @@ static const struct apple_epic_service_ops dptxep_ops[] = { {} }; +/* H17-generation firmware (T8140 H17P, T8142 H17G). */ +static const struct apple_epic_service_ops dptxep_h17_ops[] = { + { + .name = "AppleDCPDPTXRemotePort", + .prepare = dptxport_prepare, + .teardown = dptxport_teardown, + .init = dptxport_h17_init, + .call = dptxport_call, + .call_v2 = dptxport_group_call, + }, + {} +}; + +static void dptx_controller_service_init(struct apple_epic_service *service, + const char *name, const char *class, s64 unit) +{ + complete(&service->ep->dcp->dptx_controller_ready); +} + +static const struct apple_epic_service_ops dptx_controller_ops[] = { + { + .name = "AppleDCPDPTXController", + .init = dptx_controller_service_init, + }, + {} +}; + /* 14.7 also announces the port under its EPIC names. */ static const struct apple_epic_service_ops dptxep_v14_7_ops[] = { { @@ -945,8 +1171,35 @@ static const struct apple_epic_service_ops dptxep_v14_7_ops[] = { {} }; +int dptx_controller_init(struct apple_dcp *dcp) +{ + struct apple_dcp_afkep *ep; + int ret; + + init_completion(&dcp->dptx_controller_ready); + ep = afk_init(dcp, DPTX_CONTROLLER_ENDPOINT, dptx_controller_ops); + if (IS_ERR(ep)) + return PTR_ERR(ep); + /* Controller interfaces require the same OPEN handshake as the port. */ + ep->wire_v2 = dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL; + dcp->dptx_controller_ep = ep; + + ret = afk_start(ep); + if (ret) + return ret; + if (!wait_for_completion_timeout(&dcp->dptx_controller_ready, + msecs_to_jiffies(1000))) + return -ETIMEDOUT; + + return 0; +} + int dptxep_init(struct apple_dcp *dcp) { + struct apple_dcp_afkep *ep; + bool wire_v2 = dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL; int ret; u32 port; unsigned long timeout = msecs_to_jiffies(1000); @@ -956,11 +1209,20 @@ int dptxep_init(struct apple_dcp *dcp) init_completion(&dcp->dptxport[0].linkcfg_completion); init_completion(&dcp->dptxport[1].linkcfg_completion); - dcp->dptxep = afk_init(dcp, DPTX_ENDPOINT, - dcp_fw_is_14_7(dcp) ? - dptxep_v14_7_ops : dptxep_ops); - if (IS_ERR(dcp->dptxep)) - return PTR_ERR(dcp->dptxep); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + ep = afk_init(dcp, wire_v2 ? DPTX_ENDPOINT_V2 : DPTX_ENDPOINT, + dptxep_h17_ops); + else + ep = afk_init(dcp, DPTX_ENDPOINT, + dcp_fw_is_14_7(dcp) ? dptxep_v14_7_ops : dptxep_ops); + if (IS_ERR(ep)) { + /* As integration: other firmware keeps the error pointer. */ + if (dcp->fw_compat != DCP_FIRMWARE_H17P) + dcp->dptxep = ep; + return PTR_ERR(ep); + } + ep->wire_v2 = wire_v2; + dcp->dptxep = ep; ret = afk_start(dcp->dptxep); if (ret) @@ -981,13 +1243,17 @@ int dptxep_init(struct apple_dcp *dcp) "DPTX port %u not announced yet\n", port); return 0; } + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + dev_warn(dcp->dev, "DPTX transport ready but port %u service not announced\n", + port); return -ETIMEDOUT; - } else if (ret < 0) + } else if (ret < 0) { return ret; + } timeout = ret; } - return 0; + return dcp_180hz_init(dcp); } static void dpav_ctrl_bind(struct apple_epic_service *service, const char *name, diff --git a/drivers/gpu/drm/apple/dptxep.h b/drivers/gpu/drm/apple/dptxep.h index b6360621351db6..db9cedaa48f4a6 100644 --- a/drivers/gpu/drm/apple/dptxep.h +++ b/drivers/gpu/drm/apple/dptxep.h @@ -30,6 +30,7 @@ enum dptx_apcall { DPTX_APCALL_DEVICE_NOT_RESPONDING = 22, DPTX_APCALL_DEVICE_BUSY_TIMEOUT = 23, DPTX_APCALL_DEVICE_NOT_STARTED = 24, + DPTX_APCALL_GET_VIRTUAL_DEVICE_MODE = 26, }; #define DCPDPTX_REMOTE_PORT_CORE GENMASK(3, 0) @@ -49,6 +50,8 @@ struct apple_epic_service; struct dptx_port { bool enabled, connected; + /* A published session closed; unlike initial discovery, this is terminal. */ + bool closed; struct completion enable_completion; struct completion linkcfg_completion; u32 unit; @@ -59,6 +62,7 @@ struct dptx_port { u32 lane_count; u32 link_rate, pending_link_rate; u32 drive_settings[2]; + u32 trained_voltage[4], trained_pre[4]; /* * Per-boot invocation counters for the validate/connect/request/ * release AFK calls, used together with the caller return address diff --git a/drivers/gpu/drm/apple/epic/dpavserv-edid.h b/drivers/gpu/drm/apple/epic/dpavserv-edid.h new file mode 100644 index 00000000000000..31baaaea7070d0 --- /dev/null +++ b/drivers/gpu/drm/apple/epic/dpavserv-edid.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#ifndef APPLE_DPAVSERV_EDID_H +#define APPLE_DPAVSERV_EDID_H + +#include "../afkv2-codec.h" + +#define DPAV_EDID_HEADER_SIZE 48U +#define EDID_LEADING_DATA_SIZE 8U +#define EDID_BLOCK_SIZE 128U +#define EDID_MAX_SIZE 32768U +#define EDID_BUF_SIZE (EDID_LEADING_DATA_SIZE + EDID_MAX_SIZE) + +/* Validate only bytes actually returned by firmware, never zero-fill padding. */ +static inline int dpavserv_edid_size(const void *response, size_t received, + size_t *edid_size) +{ + static const u8 header[] = { 0, 255, 255, 255, 255, 255, 255, 0 }; + const u8 *p = response; + u64 used; + size_t length, offset, i; + + if (!response || !edid_size || received < DPAV_EDID_HEADER_SIZE) + return -EIO; + if (afkv2_get64(p) != EDID_BUF_SIZE) + return -EIO; + used = afkv2_get64(p + 32); + if (used < EDID_LEADING_DATA_SIZE + EDID_BLOCK_SIZE || used > EDID_BUF_SIZE || + used > received - DPAV_EDID_HEADER_SIZE) + return -EIO; + length = used - EDID_LEADING_DATA_SIZE; + p += DPAV_EDID_HEADER_SIZE + EDID_LEADING_DATA_SIZE; + if (length % EDID_BLOCK_SIZE || memcmp(p, header, sizeof(header)) || + ((size_t)p[126] + 1) * EDID_BLOCK_SIZE > length) + return -EIO; + /* Permit extra whole blocks for HF-EEODB, as drm_edid supports it. */ + for (offset = 0; offset < length; offset += EDID_BLOCK_SIZE) { + u8 sum = 0; + + for (i = 0; i < EDID_BLOCK_SIZE; i++) + sum += p[offset + i]; + if (sum) + return -EIO; + } + *edid_size = length; + return 0; +} + +#endif diff --git a/drivers/gpu/drm/apple/epic/dpavservep.c b/drivers/gpu/drm/apple/epic/dpavservep.c index fac08e53d2b378..527b88160e32e6 100644 --- a/drivers/gpu/drm/apple/epic/dpavservep.c +++ b/drivers/gpu/drm/apple/epic/dpavservep.c @@ -3,6 +3,9 @@ #include "dpavservep.h" #include "dpavservep-edid.h" +#include "dpavserv-edid.h" + +#define EDID_EXT_BLOCK_COUNT_OFFSET 0x7E #include @@ -13,20 +16,181 @@ #include "../afk.h" #include "../dcp.h" #include "../dcp-internal.h" +#include "../connector.h" #include "../trace.h" +static void dcpavserv_notify_edid(struct apple_epic_service *service) +{ + struct apple_dcp *dcp = service->ep->dcp; + struct apple_connector *connector = READ_ONCE(dcp->connector); + + if (!service->ep->wire_v2 || READ_ONCE(dcp->quiescing) || !connector || + !READ_ONCE(dcp->nr_modes) || !READ_ONCE(connector->connected) || + READ_ONCE(connector->base.registration_state) != DRM_CONNECTOR_REGISTERED || + READ_ONCE(connector->dcp) != to_platform_device(dcp->dev)) + return; + /* get_modes may already have run before this service was announced. */ + schedule_work(&connector->hotplug_wq); +} + +static bool dcpavserv_identity(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + if (!name || strcmp(name, "dcpav-service-epic")) + return false; + if (!service->ep->wire_v2) + return unit == 0; + /* H17P's lower-case interface-id is dynamic, not a physical-port unit. */ + return service->ep->endpoint == DPAVSERV_ENDPOINT_V2 && class && + (!strcmp(class, "AppleDCPDP2HDMI") || + !strcmp(class, "DCPDP13Service")) && service->channel && + service->channel <= U16_MAX && unit == service->channel; +} + +static bool dcpavserv_admission_current(struct apple_dcp *dcp, u64 generation) +{ + lockdep_assert_held(&dcp->hpd_mutex); + /* Initial service discovery may precede the first Type-C attachment. */ + return !READ_ONCE(dcp->quiescing) && !READ_ONCE(dcp->external_physically_absent) && + generation == dcp->typec_generation && + (!generation || dcp->typec_cable_connected); +} + +/* Admission runs before AFKV2 OPEN and before readers can pin the service. */ +static int dcpavserv_prepare(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + struct apple_dcp *dcp = service->ep->dcp; + bool v2 = service->ep->wire_v2; + int ret = 0; + + if (!dcpavserv_identity(service, name, class, unit)) + return -ENODEV; + if (v2) + mutex_lock(&dcp->hpd_mutex); + mutex_lock(&dcp->dcpavserv.lock); + if (dcp->dcpavserv.enabled) + ret = -EBUSY; + else if (v2 && !dcpavserv_admission_current(dcp, dcp->typec_generation)) + ret = -ENODEV; + else { + service->admission.generation = v2 ? dcp->typec_generation : 0; + service->admission.epoch = dcp->dcpavserv.epoch; + } + mutex_unlock(&dcp->dcpavserv.lock); + if (v2) + mutex_unlock(&dcp->hpd_mutex); + return ret; +} + +static int dcpavserv_revalidate(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + struct apple_dcp *dcp = service->ep->dcp; + int ret = 0; + + if (!dcpavserv_identity(service, name, class, unit)) + return -ENODEV; + mutex_lock(&dcp->hpd_mutex); + mutex_lock(&dcp->dcpavserv.lock); + if (dcp->dcpavserv.enabled) + ret = -EBUSY; + else if (service->admission.epoch != dcp->dcpavserv.epoch || + !dcpavserv_admission_current(dcp, service->admission.generation)) + ret = -ENODEV; + mutex_unlock(&dcp->dcpavserv.lock); + mutex_unlock(&dcp->hpd_mutex); + return ret; +} + +static void dcpavserv_init_h17p(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + struct apple_dcp *dcp = service->ep->dcp; + bool v2 = service->ep->wire_v2; + bool published = false; + + trace_dcpavserv_init(dcp, unit); + if (v2) + mutex_lock(&dcp->hpd_mutex); + mutex_lock(&dcp->dcpavserv.lock); + if (dcpavserv_identity(service, name, class, unit) && + !dcp->dcpavserv.enabled && + service->admission.epoch == dcp->dcpavserv.epoch && + (!v2 || dcpavserv_admission_current(dcp, service->admission.generation))) { + dcp->dcpavserv.service = afk_service_get(service); + if (dcp->dcpavserv.service) { + dcp->dcpavserv.enabled = true; + dcp->dcpavserv.epoch++; + service->cookie = &dcp->dcpavserv; + complete(&dcp->dcpavserv.enable_completion); + published = true; + } + } + mutex_unlock(&dcp->dcpavserv.lock); + if (v2) + mutex_unlock(&dcp->hpd_mutex); + if (!published) { + /* A generation/registry change during OPEN cannot enable stale work. */ + afk_service_retire(service); + return; + } + if (v2) { + dev_dbg(dcp->dev, "DCPAVSERV: AFKV2 EDID service ready on channel %u\n", + service->channel); + dcpavserv_notify_edid(service); + } +} + +/* Defer DRM cleanup: AFK callbacks must not take mode_config.mutex. */ +static void dcpavserv_invalidate_edid(struct apple_dcp *dcp) +{ + struct apple_connector *connector = READ_ONCE(dcp->connector); + + lockdep_assert_held(&dcp->dcpavserv.lock); + if (!connector || READ_ONCE(connector->dcp) != to_platform_device(dcp->dev)) + return; + WRITE_ONCE(connector->edid_invalidation_pending, true); + if (!READ_ONCE(dcp->quiescing) && + READ_ONCE(connector->base.registration_state) == DRM_CONNECTOR_REGISTERED) + schedule_work(&connector->hotplug_wq); +} + +static void dcpavserv_teardown_h17p(struct apple_epic_service *service) +{ + struct apple_dcp *dcp = service->ep->dcp; + bool owned = false; + + mutex_lock(&dcp->dcpavserv.lock); + if (dcp->dcpavserv.service == service) { + dcp->dcpavserv.enabled = false; + dcp->dcpavserv.service = NULL; + dcp->dcpavserv.epoch++; + dcpavserv_invalidate_edid(dcp); + reinit_completion(&dcp->dcpavserv.enable_completion); + owned = true; + } + mutex_unlock(&dcp->dcpavserv.lock); + if (owned) + afk_service_put(service); +} + +/* + * Every firmware other than H17P (M1/M2/M3): integration's service + * lifetime. The H17P variants below add admission, service epochs and + * connector EDID invalidation. + */ static void dcpavserv_init(struct apple_epic_service *service, const char *name, const char *class, s64 unit) { struct apple_dcp *dcp = service->ep->dcp; - unsigned long flags; trace_dcpavserv_init(dcp, unit); if (unit == 0 && name && !strcmp(name, "dcpav-service-epic")) { - spin_lock_irqsave(&dcp->dcpavserv.lock, flags); + mutex_lock(&dcp->dcpavserv.lock); if (dcp->dcpavserv.enabled) { - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); + mutex_unlock(&dcp->dcpavserv.lock); dev_err(dcp->dev, "DCPAVSERV: unit %lld already exists\n", unit); return; @@ -35,17 +199,16 @@ static void dcpavserv_init(struct apple_epic_service *service, const char *name, dcp->dcpavserv.enabled = true; service->cookie = &dcp->dcpavserv; complete(&dcp->dcpavserv.enable_completion); - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); + mutex_unlock(&dcp->dcpavserv.lock); } } static void dcpavserv_teardown(struct apple_epic_service *service) { struct apple_dcp *dcp = service->ep->dcp; - unsigned long flags; bool owned = false; - spin_lock_irqsave(&dcp->dcpavserv.lock, flags); + mutex_lock(&dcp->dcpavserv.lock); if (dcp->dcpavserv.service == service) { dcp->dcpavserv.enabled = false; dcp->dcpavserv.service = NULL; @@ -53,7 +216,7 @@ static void dcpavserv_teardown(struct apple_epic_service *service) reinit_completion(&dcp->dcpavserv.enable_completion); owned = true; } - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); + mutex_unlock(&dcp->dcpavserv.lock); if (owned) afk_service_put(service); } @@ -61,18 +224,42 @@ static void dcpavserv_teardown(struct apple_epic_service *service) void dpavservep_detach(struct apple_dcp *dcp) { struct apple_epic_service *service; - unsigned long flags; - spin_lock_irqsave(&dcp->dcpavserv.lock, flags); + if (dcp->fw_compat != DCP_FIRMWARE_H17P) { + mutex_lock(&dcp->dcpavserv.lock); + service = dcp->dcpavserv.service; + dcp->dcpavserv.service = NULL; + dcp->dcpavserv.enabled = false; + dcp->dcpavserv.opened = NULL; + mutex_unlock(&dcp->dcpavserv.lock); + if (service) { + afk_service_disable(service); + afk_service_put(service); + } + return; + } + + mutex_lock(&dcp->dcpavserv.lock); service = dcp->dcpavserv.service; dcp->dcpavserv.service = NULL; dcp->dcpavserv.enabled = false; - dcp->dcpavserv.opened = NULL; - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); + /* Also reject an in-flight prepare/OPEN with no published owner yet. */ + dcp->dcpavserv.epoch++; + reinit_completion(&dcp->dcpavserv.enable_completion); if (service) { - afk_service_disable(service); - afk_service_put(service); + afk_service_retire(service); + dcpavserv_invalidate_edid(dcp); } + mutex_unlock(&dcp->dcpavserv.lock); + if (service) + afk_service_put(service); +} + +static int dcpdpserv_prepare(struct apple_epic_service *service, const char *name, + const char *class, s64 unit) +{ + /* No production command transport is needed for this H17P service. */ + return service->ep->wire_v2 ? -EOPNOTSUPP : 0; } static void dcpdpserv_init(struct apple_epic_service *service, const char *name, @@ -172,6 +359,7 @@ static int dcpdpserv_report(struct apple_epic_service *service, return parse_report(service, type, data, data_size); } +/* Integration's bounded read, for every firmware other than H17P. */ static const struct drm_edid *dcpavserv_read_edid(struct apple_epic_service *service) { struct dpavserv_copy_edid_cmd cmd; @@ -199,48 +387,175 @@ static const struct drm_edid *dcpavserv_read_edid(struct apple_epic_service *ser return drm_edid_alloc(resp->data + DPAVSERV_EDID_LEADING_SIZE, edid_size); } -const struct drm_edid *dcpavserv_copy_edid(struct apple_dcp *dcp) +static const struct drm_edid *dcpavserv_read_edid_legacy(struct apple_epic_service *service) { - struct apple_epic_service *service; - const struct drm_edid *edid; - unsigned long flags; + struct dpavserv_copy_edid_cmd cmd; + struct dpavserv_copy_edid_resp *resp __free(kfree) = NULL; + size_t resp_size = sizeof(*resp) + EDID_BUF_SIZE; + int num_blocks; + u64 data_size; + int ret; - spin_lock_irqsave(&dcp->dcpavserv.lock, flags); - service = afk_service_get(dcp->dcpavserv.service); - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); - if (!service) - return ERR_PTR(-ENODEV); + memset(&cmd, 0, sizeof(cmd)); + cmd.max_size = cpu_to_le64(EDID_BUF_SIZE); + resp = kzalloc(resp_size, GFP_KERNEL); + if (!resp) + return ERR_PTR(-ENOMEM); + + ret = afk_service_call(service, 1, 7, &cmd, sizeof(cmd), EDID_BUF_SIZE, resp, + resp_size, 0); + if (ret < 0) + return ERR_PTR(ret); + + if (le64_to_cpu(resp->max_size) != EDID_BUF_SIZE) + return ERR_PTR(-EIO); + + // print_hex_dump(KERN_DEBUG, "dpavserv EDID cmd: ", DUMP_PREFIX_NONE, + // 16, 1, resp, 192, true); + + data_size = le64_to_cpu(resp->used_size); + if (data_size < EDID_LEADING_DATA_SIZE + EDID_BLOCK_SIZE || + data_size > EDID_BUF_SIZE) + return ERR_PTR(-EIO); /* - * The 14.7 firmware of an external processor answers EDID requests - * once its service has been opened; open each announced instance once. - * - * Only on Aurora's DCPLink session (T6030). The T8122 dcpext, which - * Linux starts and drives through the IOMFB template, reads the EDID - * directly, as in AsahiLinux/linux#607. Its firmware never answers the - * open call and drops the display link instead, about 50 ms after it, - * which brings the display back 4 s later: hotplug in a loop. + * An HDMI 2.x sink using the HF-EEODB announces fewer extension blocks + * in the base block than it sends, so the payload may be longer than + * the base block says. Accept any whole number of blocks at least that + * long and leave the block count to drm_edid, which knows about + * HF-EEODB. */ - if (dcp->fw_compat == DCP_FIRMWARE_V_14_7 && - READ_ONCE(dcp->dcpavserv.opened) != service) { - int ret = afk_service_call(service, 4, 6, NULL, 0, 32, NULL, 0, 32); + num_blocks = resp->data[EDID_LEADING_DATA_SIZE + EDID_EXT_BLOCK_COUNT_OFFSET]; + if ((1 + num_blocks) * EDID_BLOCK_SIZE > data_size - EDID_LEADING_DATA_SIZE || + (data_size - EDID_LEADING_DATA_SIZE) % EDID_BLOCK_SIZE) + return ERR_PTR(-EIO); - if (ret) { - afk_service_put(service); - return ERR_PTR(ret); + return drm_edid_alloc(resp->data + EDID_LEADING_DATA_SIZE, + data_size - EDID_LEADING_DATA_SIZE); +} + +static const struct drm_edid *dcpavserv_read_edid_inline(struct apple_epic_service *service) +{ + struct dpavserv_copy_edid_cmd cmd; + struct dpavserv_copy_edid_resp *resp __free(kfree) = NULL; + size_t resp_size = sizeof(*resp) + EDID_BUF_SIZE; + size_t received, edid_size; + int ret; + + memset(&cmd, 0, sizeof(cmd)); + cmd.max_size = cpu_to_le64(EDID_BUF_SIZE); + resp = kzalloc(resp_size, GFP_KERNEL); + if (!resp) + return ERR_PTR(-ENOMEM); + + ret = afk_service_call_len(service, 1, 7, &cmd, sizeof(cmd), EDID_BUF_SIZE, + resp, resp_size, 0, &received); + if (ret < 0) + return ERR_PTR(ret); + + ret = dpavserv_edid_size(resp, received, &edid_size); + if (ret) + return ERR_PTR(ret); + + return drm_edid_alloc(resp->data + EDID_LEADING_DATA_SIZE, edid_size); +} + +bool dcpavserv_edid_current(struct apple_dcp *dcp, u64 generation) +{ + lockdep_assert_held(&dcp->hpd_mutex); + return dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL || + (dcp->typec_cable_connected && generation == dcp->typec_generation && + !READ_ONCE(dcp->quiescing) && !READ_ONCE(dcp->external_physically_absent)); +} + +bool dcpavserv_edid_service_current(struct apple_dcp *dcp, u64 epoch) +{ + lockdep_assert_held(&dcp->dcpavserv.lock); + return dcp->dcpavserv.enabled && dcp->dcpavserv.service && + epoch == dcp->dcpavserv.epoch && + (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17P_EXTERNAL || + !READ_ONCE(dcp->external_physically_absent)); +} + +const struct drm_edid *dcpavserv_copy_edid(struct apple_dcp *dcp, u64 generation, + u64 *epoch) +{ + struct apple_epic_service *service; + const struct drm_edid *edid; + + if (dcp->fw_compat != DCP_FIRMWARE_H17P) { + mutex_lock(&dcp->dcpavserv.lock); + service = afk_service_get(dcp->dcpavserv.service); + *epoch = dcp->dcpavserv.epoch; + mutex_unlock(&dcp->dcpavserv.lock); + if (!service) + return ERR_PTR(-ENODEV); + + /* + * The 14.7 firmware of an external processor answers EDID + * requests once its service has been opened; open each + * announced instance once. + * + * Only on Aurora's DCPLink session (T6030). The T8122 dcpext, + * which Linux starts and drives through the IOMFB template, + * reads the EDID directly, as in AsahiLinux/linux#607. Its + * firmware never answers the open call and drops the display + * link instead, about 50 ms after it, which brings the display + * back 4 s later: hotplug in a loop. + */ + if (dcp->fw_compat == DCP_FIRMWARE_V_14_7 && + READ_ONCE(dcp->dcpavserv.opened) != service) { + int ret = afk_service_call(service, 4, 6, NULL, 0, 32, NULL, 0, 32); + + if (ret) { + afk_service_put(service); + return ERR_PTR(ret); + } + WRITE_ONCE(dcp->dcpavserv.opened, service); } - WRITE_ONCE(dcp->dcpavserv.opened, service); + + edid = dcpavserv_read_edid(service); + mutex_lock(&dcp->dcpavserv.lock); + if (dcp->dcpavserv.service != service) { + if (!IS_ERR(edid)) + drm_edid_free(edid); + edid = ERR_PTR(-ENODEV); + } + mutex_unlock(&dcp->dcpavserv.lock); + afk_service_put(service); + return edid; } - edid = dcpavserv_read_edid(service); - spin_lock_irqsave(&dcp->dcpavserv.lock, flags); - if (dcp->dcpavserv.service != service) { + scoped_guard(mutex, &dcp->hpd_mutex) { + if (!dcpavserv_edid_current(dcp, generation)) + return ERR_PTR(-ESTALE); + } + mutex_lock(&dcp->dcpavserv.lock); + service = dcp->dcpavserv.enabled ? afk_service_get(dcp->dcpavserv.service) : NULL; + *epoch = dcp->dcpavserv.epoch; + mutex_unlock(&dcp->dcpavserv.lock); + if (!service) + return ERR_PTR(-ENODEV); + + edid = service->ep->wire_v2 ? dcpavserv_read_edid_inline(service) : + dcpavserv_read_edid_legacy(service); + mutex_lock(&dcp->dcpavserv.lock); + if (dcp->dcpavserv.service != service || + !dcpavserv_edid_service_current(dcp, *epoch)) { if (!IS_ERR(edid)) drm_edid_free(edid); edid = ERR_PTR(-ENODEV); } - spin_unlock_irqrestore(&dcp->dcpavserv.lock, flags); + mutex_unlock(&dcp->dcpavserv.lock); afk_service_put(service); + scoped_guard(mutex, &dcp->hpd_mutex) { + if (!dcpavserv_edid_current(dcp, generation)) { + if (!IS_ERR(edid)) + drm_edid_free(edid); + edid = ERR_PTR(-ESTALE); + } + } return edid; } @@ -262,6 +577,29 @@ static const struct apple_epic_service_ops dpavservep_ops[] = { {}, }; +/* H17P (T8140, T8142): admission before OPEN, and the optional DP service. */ +static const struct apple_epic_service_ops dpavservep_h17p_ops[] = { + { + .name = "dcpav-service-epic", + .reusable = true, + .prepare = dcpavserv_prepare, + .revalidate = dcpavserv_revalidate, + .init = dcpavserv_init_h17p, + .teardown = dcpavserv_teardown_h17p, + .report = dcpavserv_report, + }, + { + .name = "dcpdp-service-epic", + .reusable = true, + .optional = true, + .prepare = dcpdpserv_prepare, + .init = dcpdpserv_init, + .teardown = dcpdpserv_teardown, + .report = dcpdpserv_report, + }, + {}, +}; + int dpavservep_init(struct apple_dcp *dcp) { struct apple_dcp_afkep *ep; @@ -269,9 +607,17 @@ int dpavservep_init(struct apple_dcp *dcp) init_completion(&dcp->dcpavserv.enable_completion); - ep = afk_init(dcp, DPAVSERV_ENDPOINT, dpavservep_ops); + ep = afk_init(dcp, dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL ? + DPAVSERV_ENDPOINT_V2 : DPAVSERV_ENDPOINT, + dcp->fw_compat == DCP_FIRMWARE_H17P ? + dpavservep_h17p_ops : dpavservep_ops); if (IS_ERR(ep)) return PTR_ERR(ep); + /* External H17P uses AFKV2 service calls with host-owned OOB buffers. */ + ep->wire_v2 = dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL; + ep->inline_v2 = false; dcp->dcpavservep = ep; dcp->dcpavservep->match_epic_name = true; @@ -284,10 +630,19 @@ int dpavservep_init(struct apple_dcp *dcp) return ret; } + /* Admission may arrive after startup; exact guards and late notify own it. */ + if (ep->wire_v2) + return 0; + ret = wait_for_completion_timeout(&dcp->dcpavserv.enable_completion, msecs_to_jiffies(1000)); - if (ret >= 0) + /* Integration: a missing announcement is not an error elsewhere. */ + if (dcp->fw_compat != DCP_FIRMWARE_H17P) return 0; + if (!ret) { + dev_warn(dcp->dev, "DCPAVSERV: timed out waiting for EDID service announcement\n"); + return -ETIMEDOUT; + } - return ret; + return 0; } diff --git a/drivers/gpu/drm/apple/epic/dpavservep.h b/drivers/gpu/drm/apple/epic/dpavservep.h index 319471d272d99a..001e211feeb6b4 100644 --- a/drivers/gpu/drm/apple/epic/dpavservep.h +++ b/drivers/gpu/drm/apple/epic/dpavservep.h @@ -5,7 +5,7 @@ #define _DRM_APPLE_EPIC_DPAVSERV_H #include -#include +#include #include struct drm_edid; @@ -13,7 +13,11 @@ struct apple_epic_service; struct apple_dcp; struct dcpavserv { - spinlock_t lock; /* Protects the service pointer and its owner pin. */ + /* Process context only: serializes retirement and EDID publication. + * Nest below mode_config.mutex and hpd_mutex; never wait for firmware. + */ + struct mutex lock; + u64 epoch; /* Changes even when the same service slot is reused. */ bool enabled; struct completion enable_completion; u32 unit; @@ -22,7 +26,10 @@ struct dcpavserv { struct apple_epic_service *opened; }; -const struct drm_edid *dcpavserv_copy_edid(struct apple_dcp *dcp); +bool dcpavserv_edid_current(struct apple_dcp *dcp, u64 generation); +bool dcpavserv_edid_service_current(struct apple_dcp *dcp, u64 epoch); +const struct drm_edid *dcpavserv_copy_edid(struct apple_dcp *dcp, u64 generation, + u64 *epoch); void dpavservep_detach(struct apple_dcp *dcp); #endif /* _DRM_APPLE_EPIC_DPAVSERV_H */ diff --git a/drivers/gpu/drm/apple/external-hold-test.c b/drivers/gpu/drm/apple/external-hold-test.c index 1830c1846d06bd..88a4f8b742d1a6 100644 --- a/drivers/gpu/drm/apple/external-hold-test.c +++ b/drivers/gpu/drm/apple/external-hold-test.c @@ -151,7 +151,7 @@ static int hold_test_init(struct kunit *test) mutex_init(&dcp->hpd_mutex); mutex_init(&dcp->tb_lock); spin_lock_init(&dcp->mode_state.lock); - spin_lock_init(&dcp->dcpavserv.lock); + mutex_init(&dcp->dcpavserv.lock); init_completion(&dcp->typec_iomfb_hpd_ready); INIT_DELAYED_WORK(&dcp->external_retry_wq, dcp_external_retry_work); INIT_DELAYED_WORK(&dcp->typec_reconnect_wq, hold_relink_work); diff --git a/drivers/gpu/drm/apple/ibootep.c b/drivers/gpu/drm/apple/ibootep.c index beb40ff3113ccb..d757ddc47edc74 100644 --- a/drivers/gpu/drm/apple/ibootep.c +++ b/drivers/gpu/drm/apple/ibootep.c @@ -734,10 +734,29 @@ static const struct apple_epic_service_ops ibootep_ops[] = { {} }; +static const struct apple_epic_service_ops external_ibootep_ops[] = { + { + .name = "dispext-service", + .init = disp_service_init, + }, + {} +}; + int ibootep_init(struct apple_dcp *dcp) { + const struct apple_epic_service_ops *ops = ibootep_ops; int ret; + /* H17P firmware (T8140, T8142): aurora-wip's service ops and errors. */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL) + ops = external_ibootep_ops; + dcp->ibootep = afk_init(dcp, DISP0_ENDPOINT, ops); + if (IS_ERR(dcp->ibootep)) + return PTR_ERR(dcp->ibootep); + return afk_start(dcp->ibootep); + } + if (!of_machine_is_compatible("apple,t6030") && !of_machine_is_compatible("apple,t6031") && !of_machine_is_compatible("apple,t6032") && diff --git a/drivers/gpu/drm/apple/iomfb.c b/drivers/gpu/drm/apple/iomfb.c index 40f410828e1a5e..f2673d637279ca 100644 --- a/drivers/gpu/drm/apple/iomfb.c +++ b/drivers/gpu/drm/apple/iomfb.c @@ -14,8 +14,10 @@ #include #include #include +#include #include #include +#include #include #include @@ -157,11 +159,487 @@ static u8 dcp_pop_depth(u8 *depth) return --(*depth); } +/* Older firmware and the H17G method profile retain their existing transport. */ +static bool iomfb_uses_queue(struct apple_dcp *dcp) +{ + return dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G; +} + +static bool iomfb_channels_idle(struct apple_dcp *dcp) +{ + return !dcp->ch_cmd.depth && !dcp->ch_cb.depth && + !dcp->ch_oobcmd.depth && !dcp->ch_oobcb.depth && + !dcp->ch_async.depth && !dcp->ch_oobasync.depth; +} + +enum iomfb_opaque_x_state { + IOMFB_OPAQUE_X_WAITING, + IOMFB_OPAQUE_X_NEEDED, + IOMFB_OPAQUE_X_QUEUED, + IOMFB_OPAQUE_X_READY, +}; + +struct iomfb_opaque_x_transaction { + struct iomfb_transaction transaction; +}; + +static void iomfb_opaque_x_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction); +static void iomfb_opaque_x_release(struct iomfb_transaction *transaction); + +static int iomfb_enqueue_opaque_x(struct apple_dcp *dcp) +{ + struct iomfb_opaque_x_transaction *opaque; + + lockdep_assert_held(&dcp->iomfb.lock); + /* DCPEXT rejects enableGPAlphaDiv; retain the panel-only workaround. */ + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL) + return 0; + if (atomic_read(&dcp->iomfb.opaque_x_state) != IOMFB_OPAQUE_X_NEEDED) + return 0; + if (dcp->iomfb.queued >= 32) + return -EBUSY; + opaque = kzalloc_obj(*opaque); + if (!opaque) + return -ENOMEM; + if (atomic_cmpxchg(&dcp->iomfb.opaque_x_state, IOMFB_OPAQUE_X_NEEDED, + IOMFB_OPAQUE_X_QUEUED) != IOMFB_OPAQUE_X_NEEDED) { + kfree(opaque); + return 0; + } + + opaque->transaction.start = iomfb_opaque_x_start; + opaque->transaction.release = iomfb_opaque_x_release; + dcp->iomfb.queued++; + list_add(&opaque->transaction.link, &dcp->iomfb.pending); + return 0; +} + +static void iomfb_discard_pending(struct apple_dcp *dcp) +{ + struct iomfb_transaction *transaction, *next; + + lockdep_assert_held(&dcp->iomfb.lock); + list_for_each_entry_safe(transaction, next, &dcp->iomfb.pending, link) { + list_del(&transaction->link); + dcp->iomfb.queued--; + if (transaction->brightness_only) + dcp->iomfb.backlight_queued = false; + transaction->release(transaction); + } +} + +static void iomfb_queue_advance(struct apple_dcp *dcp) +{ + struct iomfb_transaction *transaction = dcp->iomfb.active; + + lockdep_assert_held(&dcp->iomfb.lock); + if (READ_ONCE(dcp->crashed) || dcp->iomfb.stopped) { + iomfb_discard_pending(dcp); + /* The active operation may still be visible to firmware. */ + return; + } + if (!iomfb_channels_idle(dcp) || dcp->present_state_h17p.pending) + return; + + if (transaction) { + dcp->iomfb.active = NULL; + cancel_delayed_work(&dcp->iomfb.timeout); + if (transaction->brightness_only) + dcp->iomfb.backlight_queued = false; + /* A new level requested during a successful present stays pending. */ + if (dcp_backlight_pending(dcp)) { + if (transaction->completed) { + schedule_work(&dcp->bl_update_wq); + } else if (transaction->backlight_failed) { + unsigned int delay = dcp_backlight_retry_delay(dcp); + + if (delay) + mod_delayed_work(system_wq, &dcp->iomfb.backlight_retry, + msecs_to_jiffies(delay)); + else + dev_warn_ratelimited(dcp->dev, + "backlight retry limit reached\n"); + } + } + transaction->release(transaction); + } + if (iomfb_enqueue_opaque_x(dcp)) { + WRITE_ONCE(dcp->crashed, true); + iomfb_discard_pending(dcp); + schedule_work(&dcp->vblank_wq); + return; + } + if (!list_empty(&dcp->iomfb.pending)) + schedule_work(&dcp->iomfb.work); +} + +static void iomfb_queue_work(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(work, struct apple_dcp, iomfb.work); + struct iomfb_transaction *transaction; + + mutex_lock(&dcp->iomfb.lock); + if (READ_ONCE(dcp->crashed) || dcp->iomfb.stopped) { + iomfb_discard_pending(dcp); + goto unlock; + } + if (dcp->iomfb.active || !iomfb_channels_idle(dcp) || + dcp->present_state_h17p.pending || list_empty(&dcp->iomfb.pending)) + goto unlock; + + transaction = list_first_entry(&dcp->iomfb.pending, + struct iomfb_transaction, link); + list_del(&transaction->link); + dcp->iomfb.queued--; + dcp->iomfb.active = transaction; + WRITE_ONCE(dcp->iomfb.owner, current); + dcp->iomfb.deadline = jiffies + msecs_to_jiffies(10000); + mod_delayed_work(system_wq, &dcp->iomfb.timeout, msecs_to_jiffies(10000)); + transaction->start(dcp, transaction); + WRITE_ONCE(dcp->iomfb.owner, NULL); + iomfb_queue_advance(dcp); +unlock: + mutex_unlock(&dcp->iomfb.lock); +} + +static void iomfb_queue_timeout(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(to_delayed_work(work), + struct apple_dcp, iomfb.timeout); + + mutex_lock(&dcp->iomfb.lock); + if (dcp->iomfb.active && time_before(jiffies, dcp->iomfb.deadline)) { + mod_delayed_work(system_wq, &dcp->iomfb.timeout, + dcp->iomfb.deadline - jiffies); + } else if (dcp->iomfb.active) { + WRITE_ONCE(dcp->crashed, true); + dev_err(dcp->dev, + "IOMFB transaction timed out: cmd=%u oob=%u present=%u accepted=%u swap=%u\n", + dcp->ch_cmd.depth, dcp->ch_oobcmd.depth, + dcp->present_state_h17p.pending, dcp->present_state_h17p.accepted, + dcp->present_state_h17p.swap_id); + if (dcp->ch_cmd.depth && dcp->ch_cmd.depth <= DCP_MAX_CALL_DEPTH) { + const struct dcp_packet_header *h = + &dcp->ch_cmd.header[dcp->ch_cmd.depth - 1]; + + dev_err(dcp->dev, "pending IOMFB %c%c%c%c in=%u out=%u\n", + h->tag[3], h->tag[2], h->tag[1], h->tag[0], + h->in_len, h->out_len); + } + iomfb_discard_pending(dcp); + schedule_work(&dcp->vblank_wq); + } + mutex_unlock(&dcp->iomfb.lock); +} + +static void iomfb_opaque_x_complete(struct apple_dcp *dcp, void *out, + void *cookie) +{ + u32 status = out ? *(u32 *)out : ~0U; + + (void)cookie; + + if (status) { + dev_err(dcp->dev, "opaque X property failed: %u\n", status); + WRITE_ONCE(dcp->crashed, true); + schedule_work(&dcp->vblank_wq); + return; + } + + atomic_set(&dcp->iomfb.opaque_x_state, IOMFB_OPAQUE_X_READY); +} + +static void iomfb_opaque_x_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction) +{ + (void)transaction; + iomfb_apply_opaque_x_h17p(dcp, iomfb_opaque_x_complete, NULL); +} + +static void iomfb_opaque_x_release(struct iomfb_transaction *transaction) +{ + kfree(container_of(transaction, struct iomfb_opaque_x_transaction, + transaction)); +} + +void iomfb_opaque_x_reset_h17p(struct apple_dcp *dcp) +{ + atomic_set(&dcp->iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING); +} + +static void iomfb_backlight_retry(struct work_struct *work) +{ + struct apple_dcp *dcp = container_of(to_delayed_work(work), struct apple_dcp, + iomfb.backlight_retry); + + if (!READ_ONCE(dcp->crashed) && !READ_ONCE(dcp->iomfb.stopped) && + dcp_backlight_pending(dcp)) + schedule_work(&dcp->bl_update_wq); +} + +void iomfb_queue_init(struct apple_dcp *dcp) +{ + mutex_init(&dcp->iomfb.lock); + INIT_LIST_HEAD(&dcp->iomfb.pending); + INIT_WORK(&dcp->iomfb.work, iomfb_queue_work); + INIT_DELAYED_WORK(&dcp->iomfb.timeout, iomfb_queue_timeout); + INIT_DELAYED_WORK(&dcp->iomfb.backlight_retry, iomfb_backlight_retry); + atomic_set(&dcp->iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING); +} + +void iomfb_queue_stop(struct apple_dcp *dcp) +{ + if (!iomfb_uses_queue(dcp)) + return; + + mutex_lock(&dcp->iomfb.lock); + dcp->iomfb.stopped = true; + iomfb_discard_pending(dcp); + mutex_unlock(&dcp->iomfb.lock); + cancel_work_sync(&dcp->iomfb.work); + cancel_delayed_work_sync(&dcp->iomfb.timeout); + cancel_delayed_work_sync(&dcp->iomfb.backlight_retry); +} + +/* + * Wait until every queued transaction, including a present's completion, has + * finished. A stopped or crashed queue has nothing left to deliver. + */ +bool iomfb_queue_drain(struct apple_dcp *dcp, unsigned long timeout) +{ + unsigned long deadline = jiffies + timeout; + bool idle; + + if (!iomfb_uses_queue(dcp)) + return true; + + for (;;) { + mutex_lock(&dcp->iomfb.lock); + idle = READ_ONCE(dcp->crashed) || dcp->iomfb.stopped || + (!dcp->iomfb.active && list_empty(&dcp->iomfb.pending) && + !dcp->present_state_h17p.pending); + mutex_unlock(&dcp->iomfb.lock); + if (idle) + return true; + if (time_after(jiffies, deadline)) + return false; + usleep_range(2000, 4000); + } +} + +/* Admission is independent of transport: replies and control RPCs remain live. */ +void iomfb_pause_presentation(struct apple_dcp *dcp, bool pause) +{ + guard(mutex)(&dcp->iomfb.lock); + dcp->iomfb.presentation_paused = pause; + if (!pause && dcp->external_hotplug_pending) { + unsigned int action; + + dcp->external_hotplug_pending = false; + action = dcp_mode_hotplug(&dcp->mode_state, dcp->external_hotplug_connected, + dcp->connector ? &dcp->connector->connected : NULL); + dcp_handle_hotplug_actions(dcp, action); + } +} + +/* A dead transport is not proof that DMA or a pending present has stopped. */ +int iomfb_queue_settle(struct apple_dcp *dcp, unsigned long timeout) +{ + unsigned long deadline = jiffies + timeout; + bool idle; + + for (;;) { + mutex_lock(&dcp->iomfb.lock); + if (READ_ONCE(dcp->crashed) || dcp->iomfb.stopped) { + mutex_unlock(&dcp->iomfb.lock); + return -EIO; + } + idle = !dcp->iomfb.active && !dcp->iomfb.queued && + list_empty(&dcp->iomfb.pending) && + !dcp->present_state_h17p.pending && iomfb_channels_idle(dcp); + mutex_unlock(&dcp->iomfb.lock); + if (idle) + return 0; + if (time_after_eq(jiffies, deadline)) + return -ETIMEDOUT; + usleep_range(2000, 4000); + } +} + +int iomfb_queue(struct apple_dcp *dcp, struct iomfb_transaction *transaction) +{ + int ret = 0; + + mutex_lock(&dcp->iomfb.lock); + if (READ_ONCE(dcp->crashed) || dcp->iomfb.stopped) { + ret = -EIO; + } else if (transaction->presentation && dcp->iomfb.presentation_paused) { + ret = -EAGAIN; + } else if (transaction->brightness_only && dcp->iomfb.backlight_queued) { + ret = -EALREADY; + } else if (dcp->iomfb.queued >= 32) { + ret = -EBUSY; + } else { + if (transaction->brightness_only) + dcp->iomfb.backlight_queued = true; + dcp->iomfb.queued++; + list_add_tail(&transaction->link, &dcp->iomfb.pending); + schedule_work(&dcp->iomfb.work); + } + mutex_unlock(&dcp->iomfb.lock); + return ret; +} + +struct iomfb_crc_transaction { + struct iomfb_transaction transaction; + u32 swap_id; +}; + +static void iomfb_crc_complete(struct apple_dcp *dcp, void *out, void *cookie) +{ + struct iomfb_crc_transaction *crc = cookie; + struct dcp_packet_header *header; + u32 value; + + /* A106 has no input and a single little-endian blend-output CRC. */ + if (!out) + goto invalid; + header = out - sizeof(*header); + if (header->in_len || header->out_len != sizeof(__le32)) + goto invalid; + value = get_unaligned_le32(out); + if (READ_ONCE(dcp->crc_enabled)) + drm_crtc_add_crc_entry(&dcp->crtc->base, true, crc->swap_id, &value); + return; + +invalid: + dev_err(dcp->dev, "invalid blend CRC response\n"); + WRITE_ONCE(dcp->crashed, true); +} + +static void iomfb_crc_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction) +{ + static const struct dcp_method_entry method = { + .tag = { 'A', '1', '0', '6' }, + .name = "read_blend_crc", + }; + struct iomfb_crc_transaction *crc = container_of(transaction, + struct iomfb_crc_transaction, + transaction); + + if (READ_ONCE(dcp->crc_enabled)) + dcp_push(dcp, false, &method, 0, sizeof(__le32), NULL, + iomfb_crc_complete, crc); +} + +static void iomfb_crc_release(struct iomfb_transaction *transaction) +{ + kfree(container_of(transaction, struct iomfb_crc_transaction, transaction)); +} + +void iomfb_queue_crc_h17p(struct apple_dcp *dcp, u32 swap_id) +{ + struct iomfb_crc_transaction *crc; + + lockdep_assert_held(&dcp->iomfb.lock); + if (!READ_ONCE(dcp->crc_enabled) || READ_ONCE(dcp->crashed) || + dcp->iomfb.stopped || dcp->iomfb.presentation_paused || dcp->iomfb.queued >= 32) + return; + crc = kzalloc_obj(*crc); + if (!crc) + return; + crc->swap_id = swap_id; + crc->transaction.start = iomfb_crc_start; + crc->transaction.release = iomfb_crc_release; + /* Read the completed frame before another queued present changes it. */ + list_add(&crc->transaction.link, &dcp->iomfb.pending); + dcp->iomfb.queued++; + /* The receiver advances the queue only after acknowledging completion. */ +} + +struct iomfb_command { + struct iomfb_transaction transaction; + struct dcp_method_entry method; + dcp_callback_t callback; + void *cookie; + u32 in_len; + u32 out_len; + bool oob; + u8 data[]; +}; + +static void iomfb_command_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction) +{ + struct iomfb_command *command = container_of(transaction, + struct iomfb_command, transaction); + + dcp_push(dcp, command->oob, &command->method, command->in_len, + command->out_len, command->data, command->callback, command->cookie); +} + +static void iomfb_command_release(struct iomfb_transaction *transaction) +{ + kfree(container_of(transaction, struct iomfb_command, transaction)); +} + +static int iomfb_queue_command(struct apple_dcp *dcp, bool oob, + const struct dcp_method_entry *method, + u32 in_len, u32 out_len, void *data, + dcp_callback_t callback, void *cookie) +{ + struct iomfb_command *command; + int ret; + + if ((u64)sizeof(struct dcp_packet_header) + in_len + out_len > 0x8000) + return -EMSGSIZE; + command = kzalloc(struct_size(command, data, in_len), GFP_KERNEL); + if (!command) + return -ENOMEM; + command->transaction.start = iomfb_command_start; + command->transaction.release = iomfb_command_release; + command->method = *method; + command->callback = callback; + command->cookie = cookie; + command->in_len = in_len; + command->out_len = out_len; + command->oob = oob; + if (in_len) + memcpy(command->data, data, in_len); + ret = iomfb_queue(dcp, &command->transaction); + if (ret) + iomfb_command_release(&command->transaction); + return ret; +} + + /* Call a DCP function given by a tag */ void dcp_push(struct apple_dcp *dcp, bool oob, const struct dcp_method_entry *call, u32 in_len, u32 out_len, void *data, dcp_callback_t cb, void *cookie) { + struct dcp_method_entry resolved = *call; + + /* Reply chains stay on the serialized receiver; callers enqueue copies. */ + if (iomfb_uses_queue(dcp) && READ_ONCE(dcp->iomfb.owner) != current) { + if (iomfb_queue_command(dcp, oob, call, in_len, out_len, + data, cb, cookie)) + WRITE_ONCE(dcp->crashed, true); + return; + } + + if (dcp->fw_compat == DCP_FIRMWARE_H17P && READ_ONCE(dcp->crashed)) + return; + + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G && + call->tag_h17g[0]) + memcpy(resolved.tag, call->tag_h17g, sizeof(resolved.tag)); + call = &resolved; + enum dcp_context_id context = dcp_call_context(dcp, oob); struct dcp_channel *ch = dcp_get_channel(dcp, context); @@ -176,8 +654,21 @@ void dcp_push(struct apple_dcp *dcp, bool oob, const struct dcp_method_entry *ca .tag[3] = call->tag[0], }; - u8 depth = dcp_push_depth(&ch->depth); - u16 offset = dcp_packet_start(ch, depth); + u8 depth; + u16 offset; + + if (iomfb_uses_queue(dcp) && + (!ch || ch->depth >= DCP_MAX_CALL_DEPTH || + (u64)sizeof(header) + in_len + out_len > 0x8000 || + dcp_packet_start(ch, ch->depth) > + 0x8000 - ALIGN(sizeof(header) + in_len + out_len, + DCP_PACKET_ALIGNMENT))) { + dev_err(dcp->dev, "invalid IOMFB command envelope\n"); + WRITE_ONCE(dcp->crashed, true); + return; + } + depth = dcp_push_depth(&ch->depth); + offset = dcp_packet_start(ch, depth); void *out = dcp->shmem + dcp_tx_offset(context) + offset; void *out_data = out + sizeof(header); @@ -187,12 +678,19 @@ void dcp_push(struct apple_dcp *dcp, bool oob, const struct dcp_method_entry *ca if (in_len > 0) memcpy(out_data, data, in_len); + /* + * H17P: an unwritten status must not look like a successful + * response. + */ + if (out_len && dcp->fw_compat == DCP_FIRMWARE_H17P) + memset(out_data + in_len, 0xff, out_len); trace_iomfb_push(dcp, call, context, offset, depth); ch->callbacks[depth] = cb; ch->cookies[depth] = cookie; ch->output[depth] = out + sizeof(header) + in_len; + ch->header[depth] = header; ch->end[depth] = offset + ALIGN(data_len, DCP_PACKET_ALIGNMENT); dcp_send_message(dcp, IOMFB_ENDPOINT, @@ -328,6 +826,103 @@ void dcp_route_failure_notify(struct apple_connector *connector) schedule_work(&connector->hotplug_wq); } + +/* Does @connector belong to a DRM device of H17-generation pipelines? */ +static bool dcp_connector_h17p(struct apple_connector *connector) +{ + struct platform_device *pdev = READ_ONCE(connector->dcp); + struct drm_crtc *crtc; + + if (pdev) + return ((struct apple_dcp *)platform_get_drvdata(pdev))->fw_compat == + DCP_FIRMWARE_H17P; + drm_for_each_crtc(crtc, connector->base.dev) { + struct apple_dcp *dcp = platform_get_drvdata(to_apple_crtc(crtc)->dcp); + + if (dcp && dcp->fw_compat == DCP_FIRMWARE_H17P) + return true; + } + return false; +} + +/* + * The cached EDID itself is owned under edid_lock, which integration's + * route and sink changes take to drop it (apple_connector_invalidate_edid()). + */ +static void dcp_clear_connector_edid(struct apple_connector *connector) +{ + lockdep_assert_held(&connector->base.dev->mode_config.mutex); + drm_edid_connector_update(&connector->base, NULL); + apple_connector_invalidate_edid(connector); + connector->drm_edid_firmware_derived = false; +} + +static void dcp_consume_edid_invalidation(struct apple_connector *connector) +{ + lockdep_assert_held(&connector->base.dev->mode_config.mutex); + /* Preserve a disconnect even if the port reconnected before probing. */ + if (xchg(&connector->edid_invalidation_pending, false)) + dcp_clear_connector_edid(connector); +} + +/* H17P (T8140, T8142): aurora-wip's hotplug work. */ +static void dcp_hotplug_h17p(struct apple_connector *connector) +{ + struct platform_device *pdev; + struct apple_dcp *dcp; + int ret; + + scoped_guard(mutex, &connector->base.dev->mode_config.mutex) + dcp_consume_edid_invalidation(connector); + + pdev = READ_ONCE(connector->dcp); + if (!pdev) { /* a Type-C port unrouted after this was queued */ + scoped_guard(mutex, &connector->base.dev->mode_config.mutex) { + if (!READ_ONCE(connector->dcp) && connector->drm_edid_firmware_derived) { + drm_edid_connector_update(&connector->base, NULL); + connector->drm_edid_firmware_derived = false; + } + } + drm_kms_helper_connector_hotplug_event(&connector->base); + return; + } + dcp = platform_get_drvdata(pdev); + dev_info(dcp->dev, "%s() connected:%d valid_mode:%d nr_modes:%u\n", __func__, + connector->connected, READ_ONCE(dcp->mode_state.valid), dcp->nr_modes); + + scoped_guard(mutex, &connector->base.dev->mode_config.mutex) { + if (READ_ONCE(connector->dcp) == pdev) + dcp_update_connector_dimensions(dcp, &connector->base); + if (!READ_ONCE(connector->connected)) + dcp_clear_connector_edid(connector); + } + + /* + * DCP defers link training until we set a display mode. But we set + * display modes from atomic_flush, so userspace needs to trigger a + * flush, or the CRTC gets no signal. + */ + if (connector->base.state && !READ_ONCE(dcp->mode_state.valid) && connector->connected) { + drm_connector_set_link_status_property(&connector->base, + DRM_MODE_LINK_STATUS_BAD); + + /* + * A short Type-C route interruption can leave the DRM CRTC active + * while DCP has discarded its display mode. Userspace is then free + * to keep submitting plane-only commits, none of which retrains the + * link. Re-apply the active CRTC state once the DPTX link-config + * callback has completed. + */ + ret = dcp_retrain_active_crtc(connector); + if (ret) + dev_warn(dcp->dev, + "failed to retrain active CRTC after hotplug: %d\n", + ret); + } + + drm_kms_helper_connector_hotplug_event(&connector->base); +} + void dcp_hotplug(struct work_struct *work) { struct apple_connector *connector; @@ -338,6 +933,10 @@ void dcp_hotplug(struct work_struct *work) int ret; connector = container_of(work, struct apple_connector, hotplug_wq); + if (dcp_connector_h17p(connector)) { + dcp_hotplug_h17p(connector); + return; + } reasons = atomic_xchg(&connector->hotplug_reasons, 0); notify_only = (reasons & DCP_HOTPLUG_ROUTE_FAILURE) && !(reasons & DCP_HOTPLUG_READY); @@ -361,8 +960,16 @@ void dcp_hotplug(struct work_struct *work) * display modes from atomic_flush, so userspace needs to trigger a * flush, or the CRTC gets no signal. */ + /* + * Not for a Type-C CRTC powered off with the display attached: it has + * no link, and dcp_poweron() retrains one on wake. DCP tears its AV + * service down at power off, which lands here; a link reported bad + * then makes wlroots replace the output on wake, which powers the + * CRTC off and on again, and that wake does the same, forever. + */ if (!notify_only && connector->base.state && !READ_ONCE(dcp->mode_state.valid) && connector->connected && + !READ_ONCE(dcp->typec_crtc_off) && !(dcp_uses_t6020_tunnel_flow(dcp))) { drm_connector_set_link_status_property(&connector->base, DRM_MODE_LINK_STATUS_BAD); @@ -386,6 +993,172 @@ void dcp_hotplug(struct work_struct *work) apple_connector_backlight_sync(connector); } +/* + * J700/25G83 asks D134 after publishing its external video interface. + * The observed async record has no input and a four-byte result. Use the + * existing zero reply for this record only; its feature remains unsupported. + */ +static bool dcp_external_zero_reply(struct apple_dcp *dcp, int tag, + enum dcp_context_id context, + u32 in_len, u32 out_len) +{ + return tag == 134 && context == DCP_CONTEXT_ASYNC && + in_len == 0 && out_len == sizeof(u32) && + dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + of_machine_is_compatible("apple,j700") && + of_device_is_compatible(dcp->dev->of_node, "apple,t8140-dcpext"); +} + +static void dcpep_handle_cb_h17p(struct apple_dcp *dcp, enum dcp_context_id context, + void *data, u32 length, u16 offset) +{ + struct device *dev = dcp->dev; + struct dcp_packet_header *hdr = data; + void *in, *out; + int tag; + struct dcp_channel *ch = dcp_get_channel(dcp, context); + u8 depth; + bool handled, zero_reply; + + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + (length < sizeof(*hdr) || !ch || ch->depth >= DCP_MAX_CALL_DEPTH || + offset >= 0x8000 || length > 0x8000 - offset)) { + dev_err(dev, "invalid IOMFB callback envelope\n"); + dcp->crashed = true; + return; + } + + tag = dcp_parse_tag(hdr->tag); + + + handled = tag >= 0 && tag < IOMFB_MAX_CB && dcp->cb_handlers && + dcp->cb_handlers[tag]; + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + (tag < 0 || tag >= IOMFB_MAX_CB || + (u64)sizeof(*hdr) + hdr->in_len + hdr->out_len != length || + (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + handled && + !iomfb_check_callback_h17p(dcp, tag, hdr->in_len, hdr->out_len)))) { + dev_err(dev, "unsupported IOMFB callback %d (%u, %u)\n", + tag, hdr->in_len, hdr->out_len); + dcp->crashed = true; + return; + } + + in = data + sizeof(*hdr); + out = in + hdr->in_len; + + if (!handled) { + zero_reply = dcp_external_zero_reply(dcp, tag, context, + hdr->in_len, hdr->out_len); + if (!zero_reply && + (!iomfb_uses_queue(dcp) || + !test_and_set_bit(tag, dcp->unknown_callbacks))) + dev_warn(dev, "received unknown callback %c%c%c%c (%u, %u) context=%u\n", + hdr->tag[3], hdr->tag[2], hdr->tag[1], hdr->tag[0], + hdr->in_len, hdr->out_len, context); + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && !zero_reply) { + /* Retain the failed session without inventing a successful reply. */ + dcp->crashed = true; + complete(&dcp->start_done); + return; + } + /* + * Leaving a callback unanswered wedges the coprocessor: it + * waits for the ack forever and the outer call never returns. + * On H17P answer it with a zeroed output instead. + */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + if (hdr->out_len) + memset(out, 0, hdr->out_len); + depth = dcp_push_depth(&ch->depth); + ch->output[depth] = out; + ch->end[depth] = offset + + ALIGN(length, DCP_PACKET_ALIGNMENT); + dcp_ack(dcp, context); + } + return; + } + + // TODO: verify that in_len and out_len match our prototypes + // for now just clear the out data to have at least consistent results + if (hdr->out_len) + memset(out, 0, hdr->out_len); + + depth = dcp_push_depth(&ch->depth); + ch->output[depth] = out; + ch->end[depth] = offset + ALIGN(length, DCP_PACKET_ALIGNMENT); + + if (dcp->cb_handlers[tag](dcp, tag, out, in)) + dcp_ack(dcp, context); +} + +/* H17P: the reply is written in place into the record dcp_push() saved. */ +static void dcpep_handle_ack_h17p(struct apple_dcp *dcp, enum dcp_context_id context) +{ + struct dcp_channel *ch = dcp_get_channel(dcp, context); + const struct dcp_packet_header *record, *sent; + void *cookie, *out; + dcp_callback_t cb; + + if (!ch) { + dev_warn(dcp->dev, "ignoring ack on context %X\n", context); + return; + } + + if (!ch->depth) { + dev_warn(dcp->dev, "ignoring ack on idle context %X\n", context); + return; + } + + dcp_pop_depth(&ch->depth); + + cb = ch->callbacks[ch->depth]; + cookie = ch->cookies[ch->depth]; + out = ch->output[ch->depth]; + sent = &ch->header[ch->depth]; + + ch->callbacks[ch->depth] = NULL; + ch->cookies[ch->depth] = NULL; + ch->output[ch->depth] = NULL; + + /* + * An ack completes the innermost pending command on its context. The + * firmware writes the reply in place into that AP command record, whose + * address dcp_push() saved; the ack itself carries neither offset nor + * length on any firmware. A command sent from within a callback is + * acked on the callback context, whose receive area holds the callback + * record rather than the reply, so the ack cannot be used to find it. + * + * On older firmware the lengths in the record header must still be + * the ones that were sent; otherwise the record was overwritten and + * its output cannot be trusted. The tag is left out of the check: it + * has only been seen preserved on 13.5 firmware. + */ + if (dcp->fw_compat != DCP_FIRMWARE_H17P) { + record = out - sent->in_len - sizeof(*record); + if (record->in_len != sent->in_len || + record->out_len != sent->out_len) { + dev_err(dcp->dev, + "corrupt %c%c%c%c command record on context %X\n", + sent->tag[3], sent->tag[2], sent->tag[1], + sent->tag[0], context); + WRITE_ONCE(dcp->crashed, true); + /* + * Still complete the command, without an output, so + * that its callback drops the cookie reference and + * wakes the waiter instead of leaving it to time out. + */ + out = NULL; + } + } + + if (cb) + cb(dcp, out, cookie); +} + static void dcpep_handle_cb(struct apple_dcp *dcp, enum dcp_context_id context, void *data, u32 length, u16 offset) { @@ -464,6 +1237,14 @@ static void dcpep_got_msg(struct apple_dcp *dcp, u64 message) data = dcp->shmem + channel_offset + offset; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + if (FIELD_GET(IOMFB_MSG_ACK, message)) + dcpep_handle_ack_h17p(dcp, ctx_id); + else + dcpep_handle_cb_h17p(dcp, ctx_id, data, length, offset); + return; + } + if (FIELD_GET(IOMFB_MSG_ACK, message)) dcpep_handle_ack(dcp, ctx_id, data, length); else @@ -634,20 +1415,335 @@ int dcp_attributes_replace(struct apple_dcp *dcp, struct dcp_parse_ctx *handle, if (dcp->connector) schedule_work(&dcp->connector->bl_sync_wq); } - return 0; + return 0; +} + +void dcp_modes_release(void *data) +{ + struct apple_dcp *dcp = data; + + disable_work_sync(&dcp->dimensions_wq); + guard(mutex)(&dcp->modes_lock); + dcp->modes_generation++; + dcp->modes_admitted = false; + kfree(dcp->modes); + dcp->modes = NULL; + dcp->nr_modes = 0; +} + + +/* Retained arrays are not freed while mode readers can still refer to them. */ +static struct dcp_display_mode *dcp_modes_snapshot(struct apple_dcp *dcp, + unsigned int *count) +{ + struct dcp_display_mode *modes; + unsigned long flags; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + modes = dcp->modes; + *count = dcp->nr_modes; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + return modes; +} + +/* Caller holds hpd_mutex -> dcpavserv.lock -> metadata.lock. */ +static bool dcp_exception_mode_current_locked(struct apple_dcp *dcp, + const struct dcp_display_mode *mode) +{ + if (!mode->automatic_rgb8_quirk) + return true; + return dcp_180hz_mode_current(dcp, mode); +} + +static bool dcp_publish_mode(struct apple_dcp *dcp, struct drm_connector *connector, + const struct dcp_display_mode *source, + struct drm_display_mode *copy) +{ + unsigned long flags; + bool eligible; + + /* Ordinary modes keep their existing publication policy. */ + if (!source->automatic_rgb8_quirk) { + drm_mode_probed_add(connector, copy); + return true; + } + /* External get_modes holds hpd_mutex after its final EDID generation check. */ + /* Allocation happened outside metadata.lock; recheck before linking. */ + spin_lock_irqsave(&dcp->metadata.lock, flags); + eligible = dcp->connector && &dcp->connector->base == connector && + dcp_exception_mode_current_locked(dcp, source); + if (eligible) + /* drm_mode_probed_add only links a list under mode_config.mutex. */ + drm_mode_probed_add(connector, copy); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (!eligible) + drm_mode_destroy(connector->dev, copy); + return eligible; +} + +/* H17P (T8140, T8142): aurora-wip's mode and EDID publication. */ +static int dcp_get_modes_h17p(struct drm_connector *connector) +{ + struct apple_connector *apple_connector = to_apple_connector(connector); + struct platform_device *pdev = READ_ONCE(apple_connector->dcp); + struct apple_dcp *dcp; + + struct drm_device *dev = connector->dev; + struct drm_display_mode *mode; + struct dcp_display_mode *modes; + unsigned int nr_modes, published = 0; + u16 min_vfreq = 0, max_vfreq = 0; + bool vrr_capable = false; + bool physical_edid; + bool pending_180hz = false, published_180hz = false; + u64 edid_generation = 0; + u64 service_epoch = 0; + const struct drm_edid *placeholder; + unsigned int i; + + dcp_consume_edid_invalidation(apple_connector); + + /* A Type-C port has no pipeline while the fabric is moving it. */ + if (!pdev) { + if (apple_connector->drm_edid_firmware_derived) { + drm_edid_connector_update(connector, NULL); + apple_connector->drm_edid_firmware_derived = false; + } + return 0; + } + dcp = platform_get_drvdata(pdev); + modes = dcp_modes_snapshot(dcp, &nr_modes); + + physical_edid = dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL; + /* get_modes holds mode_config.mutex; HPD paths never acquire it. */ + if (physical_edid) { + mutex_lock(&dcp->hpd_mutex); + edid_generation = dcp->typec_generation; + } + mutex_lock(&dcp->dcpavserv.lock); + /* Consume CLOSE invalidation under the publication/retirement lock. */ + dcp_consume_edid_invalidation(apple_connector); + if (apple_connector->drm_edid && + (!dcpavserv_edid_service_current(dcp, apple_connector->drm_edid_service_epoch) || + (physical_edid && apple_connector->drm_edid_generation != edid_generation))) + dcp_clear_connector_edid(apple_connector); + if (physical_edid) { + if (!dcpavserv_edid_current(dcp, edid_generation)) { + dcp_clear_connector_edid(apple_connector); + goto edid_done; + } + } + + /* Internal H17P still has no supported EDID command transport. */ + if (nr_modes && !apple_connector->drm_edid && + (dcp->fw_compat != DCP_FIRMWARE_H17P || + (dcp->dcpavservep && dcp->dcpavservep->wire_v2))) { + const struct drm_edid *edid; + /* Let physical disconnect advance the generation during the request. */ + mutex_unlock(&dcp->dcpavserv.lock); + if (physical_edid) + mutex_unlock(&dcp->hpd_mutex); + edid = dcpavserv_copy_edid(dcp, edid_generation, &service_epoch); + if (physical_edid) + mutex_lock(&dcp->hpd_mutex); + mutex_lock(&dcp->dcpavserv.lock); + if (physical_edid) { + if (!dcpavserv_edid_current(dcp, edid_generation) || + READ_ONCE(apple_connector->dcp) != pdev) { + if (!IS_ERR_OR_NULL(edid)) + drm_edid_free(edid); + dcp_clear_connector_edid(apple_connector); + goto edid_done; + } + } + if (!IS_ERR_OR_NULL(edid) && + !dcpavserv_edid_service_current(dcp, service_epoch)) { + drm_edid_free(edid); + dcp_clear_connector_edid(apple_connector); + goto edid_done; + } + /* Retirement may have queued cleanup while this request waited. */ + dcp_consume_edid_invalidation(apple_connector); + if (IS_ERR_OR_NULL(edid)) { + /* + * There is no AV service for an internal panel, nor + * for an external display while its link is down; + * do not report that on every mode probe. + */ + if (PTR_ERR(edid) == -ENODEV) + dev_dbg(dcp->dev, "no EDID source\n"); + else + dev_warn(dcp->dev, "copy_edid failed: %pe\n", edid); + } else { + scoped_guard(mutex, &apple_connector->edid_lock) { + drm_edid_free(apple_connector->drm_edid); + apple_connector->drm_edid = edid; + } + apple_connector->drm_edid_generation = edid_generation; + apple_connector->drm_edid_service_epoch = service_epoch; + } + } + if (nr_modes && apple_connector->drm_edid) { + int update; + + /* Integration's route changes drop the cached EDID under edid_lock. */ + scoped_guard(mutex, &apple_connector->edid_lock) + update = apple_connector->drm_edid ? + drm_edid_connector_update(connector, apple_connector->drm_edid) : + -ENODATA; + + if (update) { + dev_warn(dcp->dev, "raw EDID publication failed: %d\n", update); + dcp_clear_connector_edid(apple_connector); + } else { + apple_connector->drm_edid_firmware_derived = false; + } + } else { + u64 generation = 0; + const struct drm_edid *derived = nr_modes ? + dcp_firmware_edid(dcp, &generation) : ERR_PTR(-ENODATA); + + if (!IS_ERR(derived)) { + bool log = !apple_connector->drm_edid_firmware_derived || + apple_connector->drm_edid_generation != generation; + int update; + + apple_connector->drm_edid_firmware_derived = true; + apple_connector->drm_edid_generation = generation; + update = drm_edid_connector_update(connector, derived); + drm_edid_free(derived); + if (update || !dcp_firmware_metadata_current(dcp, generation)) { + drm_edid_connector_update(connector, NULL); + apple_connector->drm_edid_firmware_derived = false; + } else if (log) { + dev_info(dcp->dev, "using firmware-derived EDID metadata for connection %llu\n", + generation); + } + } else if (apple_connector->drm_edid_firmware_derived) { + drm_edid_connector_update(connector, NULL); + apple_connector->drm_edid_firmware_derived = false; + } + } + +edid_done: + /* The 180 Hz exception checks below read the cached EDID. */ + mutex_lock(&apple_connector->edid_lock); + /* Publish after a possibly delayed raw read, under its physical generation. */ + for (i = 0; i < nr_modes; ++i) { + mode = drm_mode_duplicate(dev, &modes[i].mode); + if (!mode) { + dev_err(dev->dev, "Failed to duplicate display mode\n"); + published = 0; + goto modes_done; + } + pending_180hz |= modes[i].automatic_rgb8_quirk && + dcp_180hz_pending(dcp, &modes[i]); + if (!dcp_publish_mode(dcp, connector, &modes[i], mode)) + continue; + published_180hz |= modes[i].automatic_rgb8_quirk; + published++; + if (modes[i].vrr) { + u16 lo = modes[i].min_vrr >> 16; + u16 hi = modes[i].max_vrr >> 16; + + if (!min_vfreq || lo < min_vfreq) + min_vfreq = lo; + if (hi > max_vfreq) + max_vfreq = hi; + } + vrr_capable |= modes[i].vrr; + } + drm_connector_set_vrr_capable_property(connector, vrr_capable); +modes_done: + mutex_unlock(&apple_connector->edid_lock); + mutex_unlock(&dcp->dcpavserv.lock); + if (physical_edid) + mutex_unlock(&dcp->hpd_mutex); + if (pending_180hz && !published_180hz) + dcp_180hz_retry(dcp); + scoped_guard(mutex, &apple_connector->edid_lock) + placeholder = apple_connector->drm_edid ? + drm_edid_dup(apple_connector->drm_edid) : NULL; + if (placeholder) { + dcp_retry_placeholder_edid(dcp, placeholder); + drm_edid_free(placeholder); + } + + /* + * An internal panel has no EDID, so nothing fills in the refresh range + * that userspace needs before it will drive VRR. Supply the range DCP + * reported for the mode. This has to follow the EDID update, which + * resets display_info. + */ + if (vrr_capable && max_vfreq && + !connector->display_info.monitor_range.max_vfreq) { + connector->display_info.monitor_range.min_vfreq = min_vfreq; + connector->display_info.monitor_range.max_vfreq = max_vfreq; + } + + return published; } -void dcp_modes_release(void *data) +/* The user may own drm_display_mode, so we need to search for our copy */ +struct dcp_display_mode *lookup_mode_h17p(struct apple_dcp *dcp, + const struct drm_display_mode *mode) { - struct apple_dcp *dcp = data; + struct apple_connector *con; + struct dcp_display_mode *modes; + unsigned int count, i; + unsigned long flags; + bool eligible; - disable_work_sync(&dcp->dimensions_wq); - guard(mutex)(&dcp->modes_lock); - dcp->modes_generation++; - dcp->modes_admitted = false; - kfree(dcp->modes); - dcp->modes = NULL; - dcp->nr_modes = 0; + modes = dcp_modes_snapshot(dcp, &count); + for (i = 0; i < count; ++i) { + if (drm_mode_match(mode, &modes[i].mode, + DRM_MODE_MATCH_TIMINGS | + DRM_MODE_MATCH_CLOCK)) { + if (!modes[i].automatic_rgb8_quirk) + return &modes[i]; + mutex_lock(&dcp->hpd_mutex); + mutex_lock(&dcp->dcpavserv.lock); + con = READ_ONCE(dcp->connector); + /* The exception check reads the cached EDID. */ + if (con) + mutex_lock(&con->edid_lock); + spin_lock_irqsave(&dcp->metadata.lock, flags); + eligible = dcp_exception_mode_current_locked(dcp, &modes[i]); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + if (con) + mutex_unlock(&con->edid_lock); + mutex_unlock(&dcp->dcpavserv.lock); + mutex_unlock(&dcp->hpd_mutex); + if (eligible) + return &modes[i]; + } + } + + return NULL; +} + +/* + * H17P keeps the mode the bootloader programmed and never sends + * set_digital_out_mode (see DCP_INHERIT_BOOT_MODE in iomfb_template.c), so a + * modeset to any other mode would silently not be applied. The bootloader + * brings the panel up in its native timing, which is the mode the firmware + * scores highest and enumerate_modes() marks preferred; offer only that one. + */ +static bool dcp_mode_settable(struct apple_dcp *dcp, + const struct drm_display_mode *mode) +{ + struct dcp_display_mode *dcp_mode = lookup_mode_h17p(dcp, mode); + + if (!dcp_mode) + return false; + + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) + return dcp_mode->mode.type & DRM_MODE_TYPE_PREFERRED; + + return true; } int dcp_get_modes(struct drm_connector *connector) @@ -663,6 +1759,9 @@ int dcp_get_modes(struct drm_connector *connector) unsigned int count; int i; + if (dcp_connector_h17p(apple_connector)) + return dcp_get_modes_h17p(connector); + /* A Type-C port has no pipeline while the fabric is moving it. */ if (!pdev) { drm_edid_connector_update(connector, NULL); @@ -698,11 +1797,11 @@ int dcp_get_modes(struct drm_connector *connector) if (count) { const struct drm_edid *edid; - u64 generation; + u64 generation, epoch; /* The RPC can sleep while a disconnect or route change clears EDID. */ if (apple_connector_edid_begin(apple_connector, pdev, &generation)) { - edid = dcpavserv_copy_edid(dcp); + edid = dcpavserv_copy_edid(dcp, 0, &epoch); if (IS_ERR_OR_NULL(edid)) dev_info(dcp->dev, "copy_edid failed: %pe\n", edid); else @@ -762,6 +1861,14 @@ static bool lookup_mode_locked(struct apple_dcp *dcp, bool lookup_mode(struct apple_dcp *dcp, const struct drm_display_mode *mode, struct dcp_display_mode *out) { + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + struct dcp_display_mode *found = lookup_mode_h17p(dcp, mode); + + if (found && out) + *out = *found; + return found; + } + guard(mutex)(&dcp->modes_lock); return lookup_mode_locked(dcp, mode, out); } @@ -777,12 +1884,71 @@ enum drm_mode_status dcp_mode_valid(struct drm_connector *connector, return MODE_ERROR; dcp = platform_get_drvdata(pdev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_mode_settable(dcp, mode) ? MODE_OK : MODE_BAD; + guard(mutex)(&dcp->modes_lock); if (!dcp_modes_for_connector(dcp, apple_connector)) return MODE_BAD; return lookup_mode_locked(dcp, mode, NULL) ? MODE_OK : MODE_BAD; } +/* H17P (T8140, T8142): aurora-wip's modeset admission. */ +static int dcp_crtc_atomic_modeset_h17p(struct drm_crtc *crtc, + struct drm_atomic_state *state) +{ + struct apple_crtc *apple_crtc = to_apple_crtc(crtc); + struct apple_dcp *dcp = platform_get_drvdata(apple_crtc->dcp); + struct drm_crtc_state *crtc_state; + unsigned int action; + int ret = -EIO; + bool modeset; + + /* Disconnect cannot pass the admission check before these RPCs finish. */ + guard(mutex)(&dcp->external_lock); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && + (!dcp->external_running || dcp->external_error || READ_ONCE(dcp->quiescing) || + READ_ONCE(dcp->crashed) || READ_ONCE(dcp->iomfb.presentation_paused))) + return -EPIPE; + + crtc_state = drm_atomic_get_new_crtc_state(state, crtc); + if (!crtc_state) + return 0; + + modeset = drm_atomic_crtc_needs_modeset(crtc_state) || + !READ_ONCE(dcp->mode_state.valid); + + if (!modeset) + return 0; + + /* ignore no mode, poweroff is handled elsewhere */ + if (crtc_state->mode.hdisplay == 0 && crtc_state->mode.vdisplay == 0) + return 0; + + dcp_mode_begin(&dcp->mode_state); + switch (dcp->fw_compat) { + case DCP_FIRMWARE_V_12_3: + ret = iomfb_modeset_v12_3(dcp, crtc_state); + break; + case DCP_FIRMWARE_V_13_5: + ret = iomfb_modeset_v13_3(dcp, crtc_state); + break; + case DCP_FIRMWARE_H17P: + ret = iomfb_modeset_h17p(dcp, crtc_state); + break; + default: + WARN_ONCE(true, "Unexpected firmware version: %u\n", + dcp->fw_compat); + break; + } + + action = dcp_mode_finish(&dcp->mode_state, !ret, dcp->connector ? + &dcp->connector->connected : NULL); + dcp_handle_hotplug_actions(dcp, action); + + return ret; +} + int dcp_crtc_atomic_modeset(struct drm_crtc *crtc, struct drm_atomic_state *state) { @@ -793,6 +1959,9 @@ int dcp_crtc_atomic_modeset(struct drm_crtc *crtc, int ret = -EIO; bool modeset; + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_crtc_atomic_modeset_h17p(crtc, state); + crtc_state = drm_atomic_get_new_crtc_state(state, crtc); if (!crtc_state) return 0; @@ -912,6 +2081,9 @@ bool dcp_crtc_mode_fixup(struct drm_crtc *crtc, struct platform_device *pdev = apple_crtc->dcp; struct apple_dcp *dcp = platform_get_drvdata(pdev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + return dcp_mode_settable(dcp, mode); + /* * TODO: support synthesized modes through scaling * @@ -923,11 +2095,366 @@ bool dcp_crtc_mode_fixup(struct drm_crtc *crtc, } +static void iomfb_scanout_release_h17p(struct iomfb_scanout_h17p *scanout) +{ + unsigned int i; + + if (!scanout) + return; + for (i = 0; i < SWAP_SURFACES; i++) + if (scanout->fb[i]) + drm_framebuffer_put(scanout->fb[i]); + kfree(scanout); +} + +static struct iomfb_scanout_h17p * +iomfb_scanout_prepare_h17p(struct apple_dcp *dcp, struct drm_crtc *crtc, + struct drm_atomic_state *state) +{ + struct iomfb_scanout_h17p *scanout; + struct drm_plane *plane; + struct drm_plane_state *old_state, *new_state; + unsigned int i, slot; + int index; + + lockdep_assert_held(&dcp->iomfb.lock); + scanout = kzalloc_obj(*scanout); + if (!scanout) + return NULL; + if (dcp->iomfb.scanout) { + scanout->request = dcp->iomfb.scanout->request; + for (i = 0; i < SWAP_SURFACES; i++) { + scanout->fb[i] = dcp->iomfb.scanout->fb[i]; + if (scanout->fb[i]) + drm_framebuffer_get(scanout->fb[i]); + } + } + + for_each_oldnew_plane_in_state(state, plane, old_state, new_state, index) { + if (old_state->crtc != crtc && new_state->crtc != crtc) + continue; + slot = to_apple_plane(plane)->iomfb_surf; + if (slot >= SWAP_SURFACES) { + iomfb_scanout_release_h17p(scanout); + return NULL; + } + if (scanout->fb[slot]) + drm_framebuffer_put(scanout->fb[slot]); + scanout->fb[slot] = NULL; + if (new_state->crtc == crtc && new_state->visible && new_state->fb) { + scanout->fb[slot] = new_state->fb; + drm_framebuffer_get(new_state->fb); + } + } + return scanout; +} + +void iomfb_scanout_complete_h17p(struct apple_dcp *dcp) +{ + struct iomfb_scanout_h17p *scanout = dcp->iomfb.next_scanout; + struct iomfb_scanout_h17p *previous = dcp->iomfb.scanout; + struct dcp_swap_submit_req_h17p *request = &dcp->swap.h17p; + unsigned int i; + + lockdep_assert_held(&dcp->iomfb.lock); + if (!scanout) + return; + + /* Keep common present arguments, including the null output pointers. */ + scanout->request = *request; + /* Preserve unchanged planes when only a subset was presented. */ + for (i = 0; i < SWAP_SURFACES; i++) { + if (!(request->swap.swap_enabled & BIT(i)) && previous) { + scanout->request.surf[i] = previous->request.surf[i]; + scanout->request.surf_iova[i] = previous->request.surf_iova[i]; + scanout->request.swap.src_rect[i] = previous->request.swap.src_rect[i]; + scanout->request.swap.dst_rect[i] = previous->request.swap.dst_rect[i]; + scanout->request.swap.surf_ids[i] = previous->request.swap.surf_ids[i]; + scanout->request.swap.surf_flags[i] = previous->request.swap.surf_flags[i]; + scanout->request.swap.surf_unk[i] = previous->request.swap.surf_unk[i]; + } + scanout->request.surf_null[i] = !scanout->fb[i]; + } + /* Brightness re-presents all pinned surfaces with a fresh swap ID. */ + /* Replay only a background established by a completed present. */ + scanout->request.swap.swap_enabled = + request->swap.swap_enabled & IOMFB_SET_BACKGROUND; + if (previous) + scanout->request.swap.swap_enabled |= + previous->request.swap.swap_enabled & IOMFB_SET_BACKGROUND; + for (i = 0; i < SWAP_SURFACES; i++) + if (scanout->fb[i]) + scanout->request.swap.swap_enabled |= BIT(i); + scanout->request.swap.swap_completed = scanout->request.swap.swap_enabled; + if (!(request->swap.swap_enabled & IOMFB_SET_BACKGROUND) && previous) + scanout->request.swap.bg_color = previous->request.swap.bg_color; + dcp->iomfb.scanout = scanout; + dcp->iomfb.next_scanout = NULL; + iomfb_scanout_release_h17p(previous); +} + +bool iomfb_present_brightness_only_h17p(struct apple_dcp *dcp) +{ + return dcp->iomfb.active && dcp->iomfb.active->brightness_only; +} + +bool iomfb_apply_backlight_h17p(struct apple_dcp *dcp, + const struct dcp_swap_submit_req_h17p *request, + struct dcp_present_h17p *wire) +{ + struct iomfb_transaction *transaction = dcp->iomfb.active; + struct dcp_backlight_present present; + bool have_surface = false; + unsigned int i; + int ret; + + lockdep_assert_held(&dcp->iomfb.lock); + if (!dcp_backlight_active(dcp)) { + if (!dcp_has_panel(dcp)) + return true; + if (!dcp->brightness.maximum) + return false; + iomfb_encode_backlight_h17p(wire, 0, + dcp->brightness.maximum, false); + return true; + } + if (!transaction || !dcp->brightness.maximum) + return false; + for (i = 0; i < SWAP_SURFACES; i++) + have_surface |= !request->surf_null[i]; + ret = dcp_backlight_prepare(dcp, have_surface, &present); + if (ret && ret != -EALREADY) + return false; + if (!ret) { + transaction->backlight = present; + transaction->backlight_reserved = true; + } + iomfb_encode_backlight_h17p(wire, present.nits, + dcp->brightness.maximum, !ret); + return true; +} + +void iomfb_present_failed_h17p(struct apple_dcp *dcp) +{ + struct iomfb_transaction *transaction = dcp->iomfb.active; + + if (transaction && transaction->backlight_reserved) { + dcp_backlight_complete(dcp, transaction->backlight.sequence, false); + transaction->backlight_reserved = false; + transaction->backlight_failed = true; + } +} + +bool iomfb_present_complete_h17p(struct apple_dcp *dcp) +{ + struct iomfb_transaction *transaction = dcp->iomfb.active; + + lockdep_assert_held(&dcp->iomfb.lock); + if (transaction) { + transaction->completed = true; + if (transaction->backlight_reserved) { + dcp_backlight_complete(dcp, transaction->backlight.sequence, true); + transaction->backlight_reserved = false; + } + if (transaction->brightness_only) + return false; + } + atomic_cmpxchg(&dcp->iomfb.opaque_x_state, IOMFB_OPAQUE_X_WAITING, + IOMFB_OPAQUE_X_NEEDED); + iomfb_scanout_complete_h17p(dcp); + return true; +} + +static void iomfb_backlight_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction) +{ + struct iomfb_scanout_h17p *scanout = dcp->iomfb.scanout; + bool have_surface = false; + unsigned int i; + + if (!dcp_backlight_pending(dcp) || !READ_ONCE(dcp->mode_state.valid) || !scanout || + !dcp->connector || !dcp->connector->connected) + return; + for (i = 0; i < SWAP_SURFACES; i++) + have_surface |= !!scanout->fb[i]; + if (!have_surface) + return; + + dcp->swap.h17p = scanout->request; + iomfb_present_backlight_h17p(dcp); +} + +static void iomfb_backlight_release(struct iomfb_transaction *transaction) +{ + kfree(transaction); +} + +static void iomfb_backlight_kick(struct apple_dcp *dcp) +{ + struct iomfb_transaction *transaction; + + transaction = kzalloc_obj(*transaction); + if (!transaction) + return; + transaction->start = iomfb_backlight_start; + transaction->release = iomfb_backlight_release; + transaction->brightness_only = true; + transaction->presentation = true; + if (iomfb_queue(dcp, transaction)) + kfree(transaction); +} + +int iomfb_configure_backlight_h17p(struct apple_dcp *dcp, u32 maximum, + bool inherited_valid, u32 inherited, + bool default_valid, u32 default_nits) +{ + if (!iomfb_uses_queue(dcp)) + return -EINVAL; + return dcp_backlight_configure(dcp, maximum, inherited_valid, inherited, + default_valid, default_nits, iomfb_backlight_kick); +} + +struct iomfb_atomic_transaction { + struct iomfb_transaction transaction; + struct drm_atomic_state *state; + struct drm_crtc *crtc; + struct apple_dcp *dcp; + struct iomfb_scanout_h17p *scanout; +}; + +static void iomfb_atomic_start(struct apple_dcp *dcp, + struct iomfb_transaction *transaction) +{ + struct iomfb_atomic_transaction *atomic = container_of(transaction, + struct iomfb_atomic_transaction, transaction); + + if (!READ_ONCE(dcp->mode_state.valid) || !dcp->connector || !dcp->connector->connected) { + schedule_work(&dcp->vblank_wq); + return; + } + atomic->scanout = iomfb_scanout_prepare_h17p(dcp, atomic->crtc, atomic->state); + if (!atomic->scanout) { + WRITE_ONCE(dcp->crashed, true); + schedule_work(&dcp->vblank_wq); + return; + } + dcp->iomfb.next_scanout = atomic->scanout; + iomfb_flush_h17p(dcp, atomic->crtc, atomic->state); +} + +static void iomfb_atomic_release(struct iomfb_transaction *transaction) +{ + struct iomfb_atomic_transaction *atomic = container_of(transaction, + struct iomfb_atomic_transaction, transaction); + + if (atomic->scanout && atomic->dcp->iomfb.scanout != atomic->scanout) { + if (atomic->dcp->iomfb.next_scanout == atomic->scanout) + atomic->dcp->iomfb.next_scanout = NULL; + iomfb_scanout_release_h17p(atomic->scanout); + } + drm_atomic_state_put(atomic->state); + kfree(atomic); +} + +static void iomfb_queue_atomic(struct apple_dcp *dcp, struct drm_crtc *crtc, + struct drm_atomic_state *state) +{ + struct iomfb_atomic_transaction *atomic; + int ret; + + if (READ_ONCE(dcp->iomfb.presentation_paused)) { + schedule_work(&dcp->vblank_wq); + return; + } + atomic = kzalloc_obj(*atomic); + if (!atomic) + goto failed; + atomic->transaction.start = iomfb_atomic_start; + atomic->transaction.presentation = true; + atomic->transaction.release = iomfb_atomic_release; + atomic->state = drm_atomic_state_get(state); + atomic->crtc = crtc; + atomic->dcp = dcp; + ret = iomfb_queue(dcp, &atomic->transaction); + if (!ret) + return; + iomfb_atomic_release(&atomic->transaction); + if (ret == -EAGAIN) { + schedule_work(&dcp->vblank_wq); + return; + } +failed: + WRITE_ONCE(dcp->crashed, true); + schedule_work(&dcp->vblank_wq); +} + +/* H17P (T8140, T8142): aurora-wip's flush. */ +static void dcp_flush_h17p(struct drm_crtc *crtc, struct drm_atomic_state *state) +{ + struct platform_device *pdev = to_apple_crtc(crtc)->dcp; + struct apple_dcp *dcp = platform_get_drvdata(pdev); + + if (iomfb_uses_queue(dcp)) { + iomfb_queue_atomic(dcp, crtc, state); + return; + } + + /* + * DCP does not complete swaps after a link loss. A plane-only commit + * arriving between the reconnect callback and the required modeset + * would otherwise leave an unsignalled flip event in front of the + * recovery commit. Modesets set valid_mode before reaching flush. + * + * The same applies while a connector is still disconnected: resume + * re-runs the modeset, which marks the mode valid again, before the + * firmware has reported the display back. + */ + if (!READ_ONCE(dcp->mode_state.valid) || !dcp->connector || !dcp->connector->connected) { + schedule_work(&dcp->vblank_wq); + return; + } + + if (dcp_channel_busy(&dcp->ch_cmd)) + { + if (!dcp->ch_cmd.warned_busy) { + dev_err(dcp->dev, "unexpected busy command channel\n"); + dcp->ch_cmd.warned_busy = true; + } + /* HACK: issue a delayed vblank event to avoid timeouts in + * drm_atomic_helper_wait_for_vblanks(). + */ + schedule_work(&dcp->vblank_wq); + return; + } else if (dcp->ch_cmd.warned_busy) { + dcp->ch_cmd.warned_busy = false; + } + + switch (dcp->fw_compat) { + case DCP_FIRMWARE_V_12_3: + iomfb_flush_v12_3(dcp, crtc, state); + break; + case DCP_FIRMWARE_V_13_5: + iomfb_flush_v13_3(dcp, crtc, state); + break; + case DCP_FIRMWARE_H17P: + iomfb_flush_h17p(dcp, crtc, state); + break; + default: + WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); + break; + } +} + void dcp_flush(struct drm_crtc *crtc, struct drm_atomic_state *state) { struct platform_device *pdev = to_apple_crtc(crtc)->dcp; struct apple_dcp *dcp = platform_get_drvdata(pdev); + if (dcp->fw_compat == DCP_FIRMWARE_H17P) { + dcp_flush_h17p(crtc, state); + return; + } + if (dcp->fw_compat == DCP_FIRMWARE_V_26_6) { iomfb_v26_6_flush(dcp, crtc, state); return; @@ -997,6 +2524,9 @@ static void iomfb_start(struct apple_dcp *dcp) case DCP_FIRMWARE_V_14_7_TEMPLATE: iomfb_start_v14_7_0(dcp); break; + case DCP_FIRMWARE_H17P: + iomfb_start_h17p(dcp); + break; default: WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); break; @@ -1010,29 +2540,82 @@ bool dcp_is_initialized(struct platform_device *pdev) return dcp->active; } -void iomfb_recv_msg(struct apple_dcp *dcp, u64 message) +static void iomfb_recv_message(struct apple_dcp *dcp, u64 message) { enum dcpep_type type = FIELD_GET(IOMFB_MESSAGE_TYPE, message); - if (type == IOMFB_MESSAGE_TYPE_INITIALIZED) + if (dcp->fw_compat == DCP_FIRMWARE_H17P && READ_ONCE(dcp->crashed)) + return; + + if (type == IOMFB_MESSAGE_TYPE_INITIALIZED) { + /* + * H17P reports its interface version (bits 63:48) and a + * firmware hash in the high half of the InitComplete word. + * The transport below it is unchanged. + */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P) + dev_info(dcp->dev, + "IOMFB: init complete %#llx (version %llu)\n", + message, + FIELD_GET(GENMASK_ULL(63, 48), message)); iomfb_start(dcp); - else if (type == IOMFB_MESSAGE_TYPE_MSG) + } else if (type == IOMFB_MESSAGE_TYPE_MSG) dcpep_got_msg(dcp, message); else dev_warn(dcp->dev, "Ignoring unknown message %llx\n", message); } +void iomfb_recv_msg(struct apple_dcp *dcp, u64 message) +{ + if (!iomfb_uses_queue(dcp)) { + iomfb_recv_message(dcp, message); + return; + } + + mutex_lock(&dcp->iomfb.lock); + WRITE_ONCE(dcp->iomfb.owner, current); + if (!dcp->iomfb.stopped) + iomfb_recv_message(dcp, message); + WRITE_ONCE(dcp->iomfb.owner, NULL); + iomfb_queue_advance(dcp); + mutex_unlock(&dcp->iomfb.lock); +} + int iomfb_start_rtkit(struct apple_dcp *dcp) { dma_addr_t shmem_iova; int ret; + if (iomfb_uses_queue(dcp)) { + mutex_lock(&dcp->iomfb.lock); + if (dcp->iomfb.active || dcp->iomfb.scanout) { + mutex_unlock(&dcp->iomfb.lock); + return -EBUSY; + } + dcp->iomfb.stopped = false; + mutex_unlock(&dcp->iomfb.lock); + } + + /* + * H17P firmware expects the remote allocator endpoint to be started + * before IOMFB. A firmware subsystem whose endpoint was never started + * has nothing answering it on the AP side, and without this one the + * display DART faults. + */ + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17P_EXTERNAL) { + ret = apple_rtkit_start_ep(dcp->rtk, REMOTE_ALLOC_ENDPOINT); + if (ret) + return ret; + } ret = apple_rtkit_start_ep(dcp->rtk, IOMFB_ENDPOINT); if (ret) return ret; dcp->shmem = dma_alloc_coherent(dcp->dev, DCP_SHMEM_SIZE, &shmem_iova, GFP_KERNEL); + if (!dcp->shmem && dcp->fw_compat == DCP_FIRMWARE_H17P) + return -ENOMEM; dcp_send_message(dcp, IOMFB_ENDPOINT, dcpep_set_shmem(shmem_iova)); @@ -1055,6 +2638,9 @@ void iomfb_shutdown(struct apple_dcp *dcp) case DCP_FIRMWARE_V_14_7_TEMPLATE: iomfb_shutdown_v14_7_0(dcp); break; + case DCP_FIRMWARE_H17P: + iomfb_shutdown_h17p(dcp); + break; default: WARN_ONCE(true, "Unexpected firmware version: %u\n", dcp->fw_compat); break; diff --git a/drivers/gpu/drm/apple/iomfb.h b/drivers/gpu/drm/apple/iomfb.h index 2f022a0320a219..e4e6ea533da220 100644 --- a/drivers/gpu/drm/apple/iomfb.h +++ b/drivers/gpu/drm/apple/iomfb.h @@ -156,6 +156,7 @@ enum dcpep_method { dcpep_flush_supports_power, dcpep_set_power_state, dcpep_first_client_open, + dcpep_register_dfb_surface, dcpep_set_parameter_dcp, dcpep_enable_disable_video_power_savings, dcpep_is_main_display, @@ -166,18 +167,51 @@ enum dcpep_method { iomfbep_last_client_close, iomfbep_abort_swaps_dcp, iomfbep_set_matrix, + /* + * H17P default-framebuffer queries, issued from inside + * start_hardware_boot (D121) between create_default_fb and + * setup_video_limits. + */ + dcpep_get_dfb_compression_info, + dcpep_get_dfb_info, + dcpep_get_dfb_layout, + dcpep_get_dfb_state, + dcpep_set_dfb_dimensions, + dcpep_commit_dfb_info, + dcpep_dfb_query, + dcpep_dfb_ready_query, + /* + * H17P pipe-configuration calls made before the first swap. Most + * names are unknown; they are identified by method number. + */ + dcpep_pipe_cfg_415, + dcpep_pipe_cfg_031, + dcpep_pipe_cfg_414, + dcpep_pipe_query_478, + dcpep_pipe_query_474, + dcpep_pipe_set_476, + dcpep_pipe_set_428, dcpep_num_methods }; #define IOMFB_METHOD(tag, name) [name] = { #name, { tag[0], tag[1], tag[2], tag[3] } } +#define IOMFB_METHOD_H17(tag, h17g, name) [name] = { #name, \ + { tag[0], tag[1], tag[2], tag[3] }, \ + { h17g[0], h17g[1], h17g[2], h17g[3] } } struct dcp_method_entry { const char *name; char tag[4]; + /* H17G shares the H17P transport but changes selected method IDs. */ + char tag_h17g[4]; }; #define IOMFB_MAX_CB (1000) + +bool iomfb_validate_callback_h17p(int tag, u32 in_len, u32 out_len); struct apple_dcp; +bool iomfb_check_callback_h17p(struct apple_dcp *dcp, int tag, u32 in_len, + u32 out_len); typedef bool (*iomfb_cb_handler)(struct apple_dcp *, int, void *, void *); @@ -196,6 +230,7 @@ struct dcp_map_buf_req { u8 dva_null; } __packed; +/* 12.3, 13.5 and 14.7; H17P's is in iomfb_template.h. */ struct dcp_map_buf_resp { u64 vaddr; u64 dva; @@ -242,6 +277,7 @@ struct dcp_map_physical_resp { u32 mem_desc_id; } __packed; +/* 12.3, 13.5 and 14.7; H17P's are in iomfb_template.h. */ struct dcp_swap_start_req { u32 swap_id; struct dcp_iouserclient client; @@ -284,6 +320,7 @@ struct iomfb_set_fx_prop_req { u32 value; } __packed; +/* 12.3, 13.5 and 14.7; H17P's is in iomfb_template.h. */ struct dcp_set_power_state_req { u64 unklong; u8 unkbool; diff --git a/drivers/gpu/drm/apple-neo/iomfb_h17p.c b/drivers/gpu/drm/apple/iomfb_h17p.c similarity index 64% rename from drivers/gpu/drm/apple-neo/iomfb_h17p.c rename to drivers/gpu/drm/apple/iomfb_h17p.c index a27b79d96ed6cf..6c0b780674206d 100644 --- a/drivers/gpu/drm/apple-neo/iomfb_h17p.c +++ b/drivers/gpu/drm/apple/iomfb_h17p.c @@ -10,7 +10,7 @@ #include "iomfb_v13_3.h" #include "version_utils.h" -struct neo_dcp_callback_size { +struct dcp_callback_size { u32 in_len; u32 out_len; bool valid; @@ -29,7 +29,7 @@ struct neo_dcp_callback_size { { .in_len = (in), .out_len = (out), .valid = true, .minimum = true } /* R-IOMFB-startup-run21 and R-ANALYTICS Linux boundary observations. */ -static const struct neo_dcp_callback_size callback_sizes[IOMFB_MAX_CB] = { +static const struct dcp_callback_size callback_sizes[IOMFB_MAX_CB] = { [0] = { 0x0, 0x4, true }, [1] = { 0x0, 0x4, true }, [2] = DCP_CB_AT_LEAST(0x0, 0x0), @@ -102,9 +102,9 @@ static const struct neo_dcp_callback_size callback_sizes[IOMFB_MAX_CB] = { [599] = { 0x0, 0x0, true }, }; -bool neo_iomfb_validate_callback_h17p(int tag, u32 in_len, u32 out_len) +bool iomfb_validate_callback_h17p(int tag, u32 in_len, u32 out_len) { - const struct neo_dcp_callback_size *size; + const struct dcp_callback_size *size; if (tag < 0 || tag >= ARRAY_SIZE(callback_sizes)) return false; @@ -116,35 +116,35 @@ bool neo_iomfb_validate_callback_h17p(int tag, u32 in_len, u32 out_len) return size->in_len == in_len && size->out_len == out_len; } -struct neo_dcp_h17p_hotplug_request { +struct dcp_h17p_hotplug_request { __le64 connected; u8 tiled_display[0x4c]; u8 tiled_display_null; u8 padding[3]; } __packed; -struct neo_dcp_h17p_swap_info_request { +struct dcp_h17p_swap_info_request { u8 swap_info[0xe0]; u8 swap_info_null; u8 padding[3]; } __packed; -static_assert(sizeof(struct neo_dcp_h17p_hotplug_request) == 0x58); -static_assert(sizeof(struct neo_dcp_h17p_swap_info_request) == 0xe4); +static_assert(sizeof(struct dcp_h17p_hotplug_request) == 0x58); +static_assert(sizeof(struct dcp_h17p_swap_info_request) == 0xe4); /* D590 swap_complete_ap_gated is 0x730 bytes on the wire. */ static_assert(sizeof(struct dc_swap_complete_resp_h17p) == 0x730); static_assert(offsetof(struct dc_swap_complete_resp_h17p, swap_id) == 0); -void neo_iomfb_serialize_present_h17p(struct neo_dcp_present_h17p *wire, - const struct neo_dcp_swap_submit_req_h17p *request) +void iomfb_serialize_present_h17p(struct dcp_present_h17p *wire, + const struct dcp_swap_submit_req_h17p *request) { - size_t tail_offset = offsetof(struct neo_dcp_swap_submit_req_h17p, surf_iova); + size_t tail_offset = offsetof(struct dcp_swap_submit_req_h17p, surf_iova); size_t tail_size = sizeof(*request) - tail_offset; static_assert(sizeof(request->swap) == sizeof(wire->swap) + 2); static_assert(sizeof(request->surf) == sizeof(wire->surf)); static_assert(sizeof(*request) - - offsetof(struct neo_dcp_swap_submit_req_h17p, surf_iova) == + offsetof(struct dcp_swap_submit_req_h17p, surf_iova) == sizeof(wire->tail) - 2); memcpy(wire->swap, &request->swap, sizeof(wire->swap)); @@ -154,7 +154,7 @@ void neo_iomfb_serialize_present_h17p(struct neo_dcp_present_h17p *wire, } /* Integer commanded nits are exact binary64 values; no kernel FP is needed. */ -static u64 neo_iomfb_nits_binary64_h17p(u32 nits) +static u64 iomfb_nits_binary64_h17p(u32 nits) { u64 value = 0; unsigned int exponent; @@ -167,11 +167,11 @@ static u64 neo_iomfb_nits_binary64_h17p(u32 nits) return value; } -void neo_iomfb_encode_backlight_h17p(struct neo_dcp_present_h17p *wire, u32 nits, +void iomfb_encode_backlight_h17p(struct dcp_present_h17p *wire, u32 nits, u32 maximum, bool update) { - u64 ceiling = neo_iomfb_nits_binary64_h17p(maximum); - u64 unity = neo_iomfb_nits_binary64_h17p(1); + u64 ceiling = iomfb_nits_binary64_h17p(maximum); + u64 unity = iomfb_nits_binary64_h17p(1); /* This profile does not use the older DAC/power words. */ memset(wire->swap + 0x32f, 0, 0xd); @@ -188,12 +188,12 @@ void neo_iomfb_encode_backlight_h17p(struct neo_dcp_present_h17p *wire, u32 nits wire->swap[0x359] = 1; wire->swap[0x35a] = 1; put_unaligned_le16(1, wire->swap + 0x35c); - put_unaligned_le64(neo_iomfb_nits_binary64_h17p(nits), wire->swap + 0x35e); + put_unaligned_le64(iomfb_nits_binary64_h17p(nits), wire->swap + 0x35e); put_unaligned_le64(unity, wire->swap + 0x37e); put_unaligned_le64(unity, wire->swap + 0x3e6); } -static const struct neo_dcp_method_entry neo_dcp_methods[dcpep_num_methods] = { +static const struct dcp_method_entry dcp_methods[dcpep_num_methods] = { IOMFB_METHOD("A000", dcpep_late_init_signal), IOMFB_METHOD_H17("A025", "A029", dcpep_setup_video_limits), /* (0,0) */ IOMFB_METHOD("A131", iomfbep_a131_pmu_service_matched), /* nested in D206 */ @@ -254,48 +254,48 @@ static const struct neo_dcp_method_entry neo_dcp_methods[dcpep_num_methods] = { #include "iomfb_template.c" -void neo_iomfb_apply_opaque_x_h17p(struct neo_apple_dcp *neo_dcp, neo_dcp_callback_t callback, +void iomfb_apply_opaque_x_h17p(struct apple_dcp *dcp, dcp_callback_t callback, void *cookie) { - static const struct neo_dcp_method_entry method = { + static const struct dcp_method_entry method = { .name = "apply_property", .tag = { 'A', '3', '5', '2' }, }; - struct neo_dcp_apply_property_h17p property = neo_dcp_opaque_x_property_h17p(); + struct dcp_apply_property_h17p property = dcp_opaque_x_property_h17p(); - neo_dcp_push(neo_dcp, false, &method, sizeof(property), sizeof(u32), &property, + dcp_push(dcp, false, &method, sizeof(property), sizeof(u32), &property, callback, cookie); } -static bool trampoline_rt_bandwidth_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_rt_bandwidth_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { /* * The bandwidth scratch and doorbell come from the DCP's DT node; the * apple,bw-doorbell binding has no offset cell. */ - u64 scratch = neo_dcp->disp_bw_scratch_res.start + - neo_dcp->disp_bw_scratch_offset; - u64 clock_request = neo_dcp->disp_bw_doorbell_res.start; + u64 scratch = dcp->disp_bw_scratch_res.start + + dcp->disp_bw_scratch_offset; + u64 clock_request = dcp->disp_bw_doorbell_res.start; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - if (neo_apple_dcp_h17p_rt_bw_encode_reply(in, out, scratch, clock_request)) - dev_warn(neo_dcp->dev, "D003 bandwidth request is malformed\n"); + trace_iomfb_callback(dcp, tag, __func__); + if (apple_dcp_h17p_rt_bw_encode_reply(in, out, scratch, clock_request)) + dev_warn(dcp->dev, "D003 bandwidth request is malformed\n"); return true; } -static bool trampoline_get_frequency_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_get_frequency_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_apple_dcp_h17p_clock_request *request = in; + const struct apple_dcp_h17p_clock_request *request = in; u64 rate; int ret; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - ret = neo_apple_dcp_h17p_clock_rate(request, clk_get_rate(neo_dcp->clk), - clk_get_rate(neo_dcp->clk_194), &rate); + trace_iomfb_callback(dcp, tag, __func__); + ret = apple_dcp_h17p_clock_rate(request, clk_get_rate(dcp->clk), + clk_get_rate(dcp->clk_194), &rate); if (ret) { - dev_warn(neo_dcp->dev, "unknown display clock ID %#x\n", + dev_warn(dcp->dev, "unknown display clock ID %#x\n", le32_to_cpu(request->clock_id)); rate = 0; } @@ -304,62 +304,62 @@ static bool trampoline_get_frequency_h17p(struct neo_apple_dcp *neo_dcp, int tag return true; } -static bool trampoline_analytics_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_analytics_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_zero(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_zero(dcp, tag, out, in); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - /* The dispatcher has validated the measured D114 reply size. */ + trace_iomfb_callback(dcp, tag, __func__); + /* The dispatcher has validated the D114 reply size. */ memset(out, 0, 0x1004); *(u8 *)out = 'd'; return true; } -static bool trampoline_hotplug_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_hotplug_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_h17p_hotplug_request *request = in; + const struct dcp_h17p_hotplug_request *request = in; if (!(request->tiled_display_null & 1)) memcpy(out, request->tiled_display, sizeof(request->tiled_display)); - return trampoline_hotplug(neo_dcp, tag, out, in); + return trampoline_hotplug(dcp, tag, out, in); } -static bool trampoline_swap_info_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_swap_info_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_h17p_swap_info_request *request = in; + const struct dcp_h17p_swap_info_request *request = in; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); if (!(request->swap_info_null & 1)) memcpy(out, request->swap_info, sizeof(request->swap_info)); return true; } -struct neo_dcp_h17p_provider_request { +struct dcp_h17p_provider_request { u8 scope[4]; char name[64]; __le32 capacity; u8 reserved[4]; } __packed; -struct neo_dcp_h17p_provider_reply { +struct dcp_h17p_provider_reply { u8 data[0xc00]; __le32 length; } __packed; -struct neo_dcp_h17p_provider_property { +struct dcp_h17p_provider_property { const char *name; const char *dt_name; u32 length; const u8 *legacy_data; }; -static_assert(sizeof(struct neo_dcp_h17p_provider_request) == 0x4c); -static_assert(sizeof(struct neo_dcp_h17p_provider_reply) == 0xc04); +static_assert(sizeof(struct dcp_h17p_provider_request) == 0x4c); +static_assert(sizeof(struct dcp_h17p_provider_reply) == 0xc04); /* Compatibility data matches the independently decoded J700 disp0 ADT. */ static const u8 j700_provider_data_0[] = { @@ -390,8 +390,8 @@ static const u8 j700_provider_data_4[] = { 0xa2, 0x19, 0x00, 0x00 }; -/* Only these provider transfers have been measured against the disp0 ADT. */ -static const struct neo_dcp_h17p_provider_property provider_properties[] = { +/* Only these provider transfers are known to match the disp0 ADT. */ +static const struct dcp_h17p_provider_property provider_properties[] = { { "power-lut-data-x", "apple,power-lut-data-x", 4, j700_provider_data_0 }, { "power-lut-data-y", "apple,power-lut-data-y", 4, j700_provider_data_1 }, { "power-lut-data-xindex", "apple,power-lut-data-xindex", 52, j700_provider_data_2 }, @@ -400,14 +400,14 @@ static const struct neo_dcp_h17p_provider_property provider_properties[] = { { "power-lut-vbatt-cur-nominal", "apple,power-lut-vbatt-cur-nominal", 0, NULL }, }; -static const struct neo_dcp_h17p_provider_property * -neo_dcp_provider_property_h17p(const struct neo_dcp_h17p_provider_request *request) +static const struct dcp_h17p_provider_property * +dcp_provider_property_h17p(const struct dcp_h17p_provider_request *request) { unsigned int i; if (memcmp(request->scope, "VORP", sizeof(request->scope)) || le32_to_cpu(request->capacity) != - sizeof_field(struct neo_dcp_h17p_provider_reply, data) || + sizeof_field(struct dcp_h17p_provider_reply, data) || memchr_inv(request->reserved, 0, sizeof(request->reserved)) || strnlen(request->name, sizeof(request->name)) == sizeof(request->name)) return NULL; @@ -419,41 +419,45 @@ neo_dcp_provider_property_h17p(const struct neo_dcp_h17p_provider_request *reque return NULL; } -static bool trampoline_provider_property_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_provider_property_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_h17p_provider_property *property; - struct neo_dcp_h17p_provider_reply *reply = out; - struct device_node *node = neo_dcp->dev->of_node; + const struct dcp_h17p_provider_property *property; + struct dcp_h17p_provider_reply *reply = out; + struct device_node *node = dcp->dev->of_node; int length, ret; /* The dispatcher already cleared the exact H17G output length. */ - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) return true; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - property = neo_dcp_provider_property_h17p(in); + trace_iomfb_callback(dcp, tag, __func__); + property = dcp_provider_property_h17p(in); if (!property) { - dev_err(neo_dcp->dev, "unqualified D400 provider property request\n"); + dev_err(dcp->dev, "unsupported D400 provider property request\n"); goto fail; } length = of_property_count_u8_elems(node, property->dt_name); if (!property->length) { - /* Only this absent property has a measured empty reply. */ + /* Only this absent property has a known empty reply. */ if (of_property_present(node, property->dt_name)) goto fail; } else if (!of_property_present(node, property->dt_name) && of_machine_is_compatible("apple,j700") && - of_device_is_compatible(node, "apple,t8140-dcp")) { - /* Qualified older loaders lack the newly described table properties. */ + (of_device_is_compatible(node, "apple,t8140-dcp") || + (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && + of_device_is_compatible(node, "apple,t8140-dcpext")))) { + /* Older loaders lack the newly described table properties. */ memset(reply, 0, sizeof(*reply)); memcpy(reply->data, property->legacy_data, property->length); reply->length = cpu_to_le32(property->length); - dev_warn_once(neo_dcp->dev, "using J700 provider data for an older loader\n"); + dev_warn_once(dcp->dev, + "missing provider property %s (%u bytes); using the J700 defaults\n", + property->dt_name, property->length); return true; } else if (length != (int)property->length) { - dev_err(neo_dcp->dev, "invalid provider property %s: %d bytes\n", + dev_err(dcp->dev, "invalid provider property %s: %d bytes\n", property->dt_name, length); goto fail; } @@ -469,21 +473,21 @@ static bool trampoline_provider_property_h17p(struct neo_apple_dcp *neo_dcp, int return true; fail: - WRITE_ONCE(neo_dcp->crashed, true); + WRITE_ONCE(dcp->crashed, true); return false; } -static bool trampoline_luma_span_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_luma_span_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { /* Preserve the older profile's already-cleared unknown-callback reply. */ - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) return true; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); if (get_unaligned_le32(in)) { - dev_err(neo_dcp->dev, "unqualified D574 range selector\n"); - WRITE_ONCE(neo_dcp->crashed, true); + dev_err(dcp->dev, "unsupported D574 range selector\n"); + WRITE_ONCE(dcp->crashed, true); return false; } @@ -492,49 +496,49 @@ static bool trampoline_luma_span_h17p(struct neo_apple_dcp *neo_dcp, int tag, return true; } -static bool trampoline_get_uint_prop_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_get_uint_prop_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_get_uint_prop_req *request = in; + const struct dcp_get_uint_prop_req *request = in; - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_get_uint_prop(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_get_uint_prop(dcp, tag, out, in); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); /* The observed startup query uses Asahi's existing Temperature reply. */ if (memcmp(request->obj, "SUMP", sizeof(request->obj)) || strnlen(request->key, sizeof(request->key)) == sizeof(request->key) || strcmp(request->key, "Temperature") || request->value_null || memchr_inv(request->padding, 0, sizeof(request->padding))) { - dev_err(neo_dcp->dev, "unqualified D401 integer property query\n"); - WRITE_ONCE(neo_dcp->crashed, true); + dev_err(dcp->dev, "unsupported D401 integer property query\n"); + WRITE_ONCE(dcp->crashed, true); return false; } put_unaligned_le64(3029, out); - *((u8 *)out + offsetof(struct neo_dcp_get_uint_prop_resp, ret)) = true; + *((u8 *)out + offsetof(struct dcp_get_uint_prop_resp, ret)) = true; return true; } -static bool trampoline_allocate_buffer_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_allocate_buffer_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_allocate_buffer_req *wire = in; - struct neo_dcp_allocate_buffer_req request = { 0 }; - struct neo_dcp_allocate_buffer_resp reply; + const struct dcp_allocate_buffer_req *wire = in; + struct dcp_allocate_buffer_req request = { 0 }; + struct dcp_allocate_buffer_resp reply; - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_allocate_buffer(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_allocate_buffer(dcp, tag, out, in); static_assert(sizeof(*wire) == 0x14); static_assert(sizeof(reply) == 0x1c); - static_assert(offsetof(struct neo_dcp_allocate_buffer_req, size) == 4); - static_assert(offsetof(struct neo_dcp_allocate_buffer_req, unk2) == 12); - static_assert(offsetof(struct neo_dcp_allocate_buffer_req, paddr_null) == 16); + static_assert(offsetof(struct dcp_allocate_buffer_req, size) == 4); + static_assert(offsetof(struct dcp_allocate_buffer_req, unk2) == 12); + static_assert(offsetof(struct dcp_allocate_buffer_req, paddr_null) == 16); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); request.size = get_unaligned_le64(&wire->size); - reply = dcpep_cb_allocate_buffer(neo_dcp, &request); + reply = dcpep_cb_allocate_buffer(dcp, &request); if (!reply.dva_size) { /* Exhaustion and allocation errors use the all-zero reply. */ memset(out, 0, sizeof(reply)); @@ -548,26 +552,26 @@ static bool trampoline_allocate_buffer_h17p(struct neo_apple_dcp *neo_dcp, int t return true; } -static bool trampoline_map_piodma_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_map_piodma_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_map_buf_req *wire = in; - struct neo_dcp_map_buf_req request = { 0 }; - struct neo_dcp_map_buf_resp_h17p reply = { 0 }; + const struct dcp_map_buf_req *wire = in; + struct dcp_map_buf_req request = { 0 }; + struct dcp_map_buf_resp_h17p reply = { 0 }; u64 buffer; - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_map_piodma(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_map_piodma(dcp, tag, out, in); static_assert(sizeof(*wire) == 0xc); static_assert(sizeof(reply) == 0x10); - static_assert(offsetof(struct neo_dcp_map_buf_req, unk) == 8); - static_assert(offsetof(struct neo_dcp_map_buf_req, dva_null) == 11); + static_assert(offsetof(struct dcp_map_buf_req, unk) == 8); + static_assert(offsetof(struct dcp_map_buf_req, dva_null) == 11); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); buffer = get_unaligned_le64(&wire->buffer); request.buffer = buffer; - if (!dcpep_map_piodma(neo_dcp, &request, &reply)) { + if (!dcpep_map_piodma(dcp, &request, &reply)) { /* A rejected mapping echoes the descriptor with a zero DVA. */ put_unaligned_le32(buffer, out); put_unaligned_le64(0, (u8 *)out + 4); @@ -581,47 +585,47 @@ static bool trampoline_map_piodma_h17p(struct neo_apple_dcp *neo_dcp, int tag, return true; } -static bool trampoline_unmap_piodma_h17p(struct neo_apple_dcp *neo_dcp, int tag, +static bool trampoline_unmap_piodma_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - const struct neo_dcp_unmap_buf_resp *wire = in; - struct neo_dcp_unmap_buf_resp request = { 0 }; + const struct dcp_unmap_buf_resp *wire = in; + struct dcp_unmap_buf_resp request = { 0 }; - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_unmap_piodma(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_unmap_piodma(dcp, tag, out, in); static_assert(sizeof(*wire) == 0x1a); - static_assert(offsetof(struct neo_dcp_unmap_buf_resp, vaddr) == 8); - static_assert(offsetof(struct neo_dcp_unmap_buf_resp, dva) == 16); - static_assert(offsetof(struct neo_dcp_unmap_buf_resp, buf_null) == 25); + static_assert(offsetof(struct dcp_unmap_buf_resp, vaddr) == 8); + static_assert(offsetof(struct dcp_unmap_buf_resp, dva) == 16); + static_assert(offsetof(struct dcp_unmap_buf_resp, buf_null) == 25); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); request.buffer = get_unaligned_le64(&wire->buffer); request.vaddr = get_unaligned_le64(&wire->vaddr); request.dva = get_unaligned_le64(&wire->dva); request.unk = wire->unk; request.buf_null = wire->buf_null; - dcpep_unmap_piodma(neo_dcp, &request); + dcpep_unmap_piodma(dcp, &request); return true; } static bool -trampoline_release_mem_desc_h17p(struct neo_apple_dcp *neo_dcp, int tag, void *out, +trampoline_release_mem_desc_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { u32 id; - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - return trampoline_release_mem_desc(neo_dcp, tag, out, in); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + return trampoline_release_mem_desc(dcp, tag, out, in); - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); id = get_unaligned_le32(in); - *(u8 *)out = dcpep_cb_release_mem_desc(neo_dcp, &id); + *(u8 *)out = dcpep_cb_release_mem_desc(dcp, &id); return true; } /* H17P callback numbering is not a uniform shift of the v13.5 table. */ -static const neo_iomfb_cb_handler cb_handlers[IOMFB_MAX_CB] = { +static const iomfb_cb_handler cb_handlers[IOMFB_MAX_CB] = { [0] = dcpep_cb_d000_h17p, /* acked after a nested A033 */ [1] = trampoline_true, [2] = trampoline_nop, @@ -694,38 +698,116 @@ static const neo_iomfb_cb_handler cb_handlers[IOMFB_MAX_CB] = { [599] = trampoline_nop, /* find_swap_function_gated */ }; -/* - * Validate a callback record and, the first time each runtime callback whose - * length is only bounded arrives with a different length, report the length - * actually seen. Those reports are the first capture of these records. - */ -bool neo_iomfb_check_callback_h17p(struct neo_apple_dcp *neo_dcp, int tag, u32 in_len, +bool iomfb_check_callback_h17p(struct apple_dcp *dcp, int tag, u32 in_len, u32 out_len) { - const struct neo_dcp_callback_size *size; + return iomfb_validate_callback_h17p(tag, in_len, out_len); +} - if (!neo_iomfb_validate_callback_h17p(tag, in_len, out_len)) +static_assert(sizeof(struct dcp_set_power_state_resp) == 8); + +struct dcp_external_power_cookie { + struct dcp_wait_cookie wait; + bool on; +}; + +/* A473 echoes the accepted state with zero status; DMA ownership is unchanged. */ +static void iomfb_external_power_complete(struct apple_dcp *dcp, void *out, void *data) +{ + struct dcp_external_power_cookie *cookie = data; + const struct dcp_packet_header *header; + int ret = -EPROTO; + + if (out) { + header = out - sizeof(struct dcp_set_power_state_req_h17p) - sizeof(*header); + if (header->in_len == sizeof(struct dcp_set_power_state_req_h17p) && + header->out_len == sizeof(struct dcp_set_power_state_resp)) { + u64 reply = get_unaligned_le64(out); + + ret = reply == (u64)cookie->on ? 0 : -EREMOTEIO; + } + } + WRITE_ONCE(cookie->wait.status, ret); + complete(&cookie->wait.done); + kref_put(&cookie->wait.refcount, release_wait_cookie); +} + +static bool iomfb_external_reply_size(void *out, u32 in_len, u32 out_len) +{ + const struct dcp_packet_header *header; + + if (!out) return false; - size = &callback_sizes[tag]; - if (size->minimum && - (in_len != size->in_len || out_len != size->out_len) && - !test_and_set_bit(tag, neo_dcp->sized_callbacks)) - dev_info(neo_dcp->dev, - "callback D%03d record is %#x/%#x bytes; its handler uses %#x/%#x\n", - tag, in_len, out_len, size->in_len, size->out_len); - return true; + header = out - in_len - sizeof(*header); + return header->in_len == in_len && header->out_len == out_len; +} + +static void iomfb_external_parameter_complete(struct apple_dcp *dcp, void *out, void *data) +{ + struct dcp_set_power_state_req_h17p request = { .unklong = 1 }; + + if (!iomfb_external_reply_size(out, sizeof(struct dcp_set_parameter_dcp), sizeof(u32))) { + iomfb_external_power_complete(dcp, NULL, data); + return; + } + dcp_set_power_state(dcp, true, &request, iomfb_external_power_complete, data); +} + +static void iomfb_external_device_complete(struct apple_dcp *dcp, void *out, void *data) +{ + struct dcp_set_parameter_dcp parameter = { + .param = IOMFBPARAM_ADAPTIVE_SYNC, + .count = 3, + }; + + if (!iomfb_external_reply_size(out, sizeof(u32), sizeof(u32))) { + iomfb_external_power_complete(dcp, NULL, data); + return; + } + /* Preserve the existing external power-on preparation and payload semantics. */ + dcp_set_parameter_dcp(dcp, false, ¶meter, iomfb_external_parameter_complete, data); +} + +int iomfb_external_power_h17p(struct apple_dcp *dcp, bool on) +{ + struct dcp_set_power_state_req_h17p request = { .unklong = on }; + struct dcp_external_power_cookie *cookie; + int ret; + + if (READ_ONCE(dcp->crashed) || READ_ONCE(dcp->iomfb.stopped)) + return -EIO; + cookie = kzalloc_obj(*cookie); + if (!cookie) + return -ENOMEM; + cookie->on = on; + cookie->wait.status = -ETIMEDOUT; + init_completion(&cookie->wait.done); + kref_init(&cookie->wait.refcount); + kref_get(&cookie->wait.refcount); + if (on) { + u32 handle = 2; + + iomfb_opaque_x_reset_h17p(dcp); + dcp_set_display_device(dcp, false, &handle, iomfb_external_device_complete, cookie); + } else { + dcp_set_power_state(dcp, true, &request, iomfb_external_power_complete, cookie); + } + ret = wait_for_completion_timeout(&cookie->wait.done, msecs_to_jiffies(500)); + ret = ret ? (int)READ_ONCE(cookie->wait.status) : -ETIMEDOUT; + kref_put(&cookie->wait.refcount, release_wait_cookie); + return ret; } -void neo_iomfb_start_h17p(struct neo_apple_dcp *neo_dcp) +void iomfb_start_h17p(struct apple_dcp *dcp) { - WRITE_ONCE(neo_dcp->pipe_enabled_h17p, false); - neo_dcp->cb_handlers = cb_handlers; - neo_dcp_start_signal(neo_dcp, false, neo_dcp_started, NULL); + WRITE_ONCE(dcp->pipe_enabled_h17p, false); + dcp->cb_handlers = cb_handlers; + dcp_start_signal(dcp, false, dcp_started, NULL); } -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_KUNIT_TEST) +#if IS_ENABLED(CONFIG_DRM_APPLE_KUNIT_TEST) /* Returns the first handled tag without reply bounds, or -1. */ -int neo_iomfb_h17p_first_unbounded_callback(void) +int iomfb_h17p_first_unbounded_callback(void) { int tag; diff --git a/drivers/gpu/drm/apple-neo/iomfb_h17p.h b/drivers/gpu/drm/apple/iomfb_h17p.h similarity index 58% rename from drivers/gpu/drm/apple-neo/iomfb_h17p.h rename to drivers/gpu/drm/apple/iomfb_h17p.h index 3f9c26c84a99ee..75856d1921562b 100644 --- a/drivers/gpu/drm/apple-neo/iomfb_h17p.h +++ b/drivers/gpu/drm/apple/iomfb_h17p.h @@ -11,13 +11,13 @@ #include "iomfb_template.h" -static_assert(sizeof(struct neo_dcp_swap_h17p) == 0x58a); -static_assert(sizeof(struct neo_dcp_surface_h17p) == 0x22c); -static_assert(sizeof(struct neo_dcp_swap_submit_req_h17p) == 0xe9c); -static_assert(sizeof(struct neo_dcp_swap_submit_resp_h17p) == 0x0c); +static_assert(sizeof(struct dcp_swap_h17p) == 0x58a); +static_assert(sizeof(struct dcp_surface_h17p) == 0x22c); +static_assert(sizeof(struct dcp_swap_submit_req_h17p) == 0xe9c); +static_assert(sizeof(struct dcp_swap_submit_resp_h17p) == 0x0c); /* Owned by the RTKit receive thread, independently of DRM state lifetime. */ -struct neo_dcp_present_state_h17p { +struct dcp_present_state_h17p { u32 swap_id; bool pending; bool accepted; @@ -29,7 +29,7 @@ struct neo_dcp_present_state_h17p { }; static inline bool -neo_dcp_present_begin_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id) +dcp_present_begin_h17p(struct dcp_present_state_h17p *state, u32 swap_id) { if (state->pending) return false; @@ -41,7 +41,7 @@ neo_dcp_present_begin_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id } static inline bool -neo_dcp_present_submit_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id, +dcp_present_submit_h17p(struct dcp_present_state_h17p *state, u32 swap_id, bool accepted) { if (!state->pending || state->accepted || state->swap_id != swap_id) @@ -57,7 +57,7 @@ neo_dcp_present_submit_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_i * recorded and the reply ends the present instead. */ static inline bool -neo_dcp_present_abort_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id) +dcp_present_abort_h17p(struct dcp_present_state_h17p *state, u32 swap_id) { if (!state->pending || state->swap_id != swap_id) return false; @@ -74,14 +74,14 @@ neo_dcp_present_abort_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id /* Whether the firmware aborted @swap_id; a completion for it is stale. */ static inline bool -neo_dcp_present_was_aborted_h17p(const struct neo_dcp_present_state_h17p *state, +dcp_present_was_aborted_h17p(const struct dcp_present_state_h17p *state, u32 swap_id) { return state->has_last_aborted && state->last_aborted == swap_id; } static inline bool -neo_dcp_present_complete_h17p(struct neo_dcp_present_state_h17p *state, u32 swap_id) +dcp_present_complete_h17p(struct dcp_present_state_h17p *state, u32 swap_id) { if (!state->pending || !state->accepted || state->swap_id != swap_id) return false; @@ -91,39 +91,39 @@ neo_dcp_present_complete_h17p(struct neo_dcp_present_state_h17p *state, u32 swap } /* The H17P swap boundary is independent of the template's native layout. */ -struct neo_dcp_present_h17p { +struct dcp_present_h17p { u8 swap[0x588]; - struct neo_dcp_surface_h17p surf[SWAP_SURFACES]; + struct dcp_surface_h17p surf[SWAP_SURFACES]; u8 tail[0x64]; } __packed; -static_assert(sizeof(struct neo_dcp_present_h17p) == 0xe9c); -static_assert(offsetof(struct neo_dcp_present_h17p, surf) == 0x588); -static_assert(offsetof(struct neo_dcp_present_h17p, tail) == 0xe38); +static_assert(sizeof(struct dcp_present_h17p) == 0xe9c); +static_assert(offsetof(struct dcp_present_h17p, surf) == 0x588); +static_assert(offsetof(struct dcp_present_h17p, tail) == 0xe38); -struct neo_dcp_apply_property_h17p { +struct dcp_apply_property_h17p { u32 property; u32 value; } __packed; -static_assert(sizeof(struct neo_dcp_apply_property_h17p) == 0x8); +static_assert(sizeof(struct dcp_apply_property_h17p) == 0x8); -static inline struct neo_dcp_apply_property_h17p neo_dcp_opaque_x_property_h17p(void) +static inline struct dcp_apply_property_h17p dcp_opaque_x_property_h17p(void) { - return (struct neo_dcp_apply_property_h17p) { + return (struct dcp_apply_property_h17p) { .property = 0x49, .value = 0, }; } -void neo_iomfb_encode_backlight_h17p(struct neo_dcp_present_h17p *wire, u32 nits, +void iomfb_encode_backlight_h17p(struct dcp_present_h17p *wire, u32 nits, u32 maximum, bool update); -void neo_iomfb_serialize_present_h17p(struct neo_dcp_present_h17p *wire, - const struct neo_dcp_swap_submit_req_h17p *request); +void iomfb_serialize_present_h17p(struct dcp_present_h17p *wire, + const struct dcp_swap_submit_req_h17p *request); -#if IS_ENABLED(CONFIG_DRM_APPLE_NEO_KUNIT_TEST) -int neo_iomfb_h17p_first_unbounded_callback(void); +#if IS_ENABLED(CONFIG_DRM_APPLE_KUNIT_TEST) +int iomfb_h17p_first_unbounded_callback(void); #endif #undef DCP_FW_VER diff --git a/drivers/gpu/drm/apple-neo/iomfb_h17p_test.c b/drivers/gpu/drm/apple/iomfb_h17p_test.c similarity index 54% rename from drivers/gpu/drm/apple-neo/iomfb_h17p_test.c rename to drivers/gpu/drm/apple/iomfb_h17p_test.c index b506d133ed7b3e..c5955836700ca4 100644 --- a/drivers/gpu/drm/apple-neo/iomfb_h17p_test.c +++ b/drivers/gpu/drm/apple/iomfb_h17p_test.c @@ -9,9 +9,9 @@ static void h17p_surface_wire_test(struct kunit *test) { - struct neo_dcp_swap_submit_req_h17p *request; - struct neo_dcp_present_h17p *wire; - struct neo_dcp_surface *surface; + struct dcp_swap_submit_req_h17p *request; + struct dcp_present_h17p *wire; + struct dcp_surface *surface; u64 iova = 0x123456789aULL; request = kunit_kzalloc(test, sizeof(*request), GFP_KERNEL); @@ -40,11 +40,13 @@ static void h17p_surface_wire_test(struct kunit *test) request->surf_null[2] = true; request->surf_null[3] = true; - neo_iomfb_serialize_present_h17p(wire, request); + iomfb_serialize_present_h17p(wire, request); KUNIT_EXPECT_MEMEQ(test, wire->swap, &request->swap, sizeof(wire->swap)); KUNIT_EXPECT_EQ(test, get_unaligned_le32((u8 *)wire + 0x593), DCP_FORMAT_BGRA); + KUNIT_EXPECT_EQ(test, ((u8 *)wire)[0x59b], (u8)DCP_XFER_FUNC_SDR); + KUNIT_EXPECT_EQ(test, ((u8 *)wire)[0x59c], (u8)DCP_COLORSPACE_NATIVE); KUNIT_EXPECT_EQ(test, get_unaligned_le32((u8 *)wire + 0x59d), 10176U); KUNIT_EXPECT_EQ(test, get_unaligned_le32((u8 *)wire + 0x5a9), 2408U); KUNIT_EXPECT_EQ(test, get_unaligned_le32((u8 *)wire + 0x5ad), 1506U); @@ -58,11 +60,17 @@ static void h17p_surface_wire_test(struct kunit *test) KUNIT_EXPECT_EQ(test, get_unaligned_le64((u8 *)wire + 0xe38), iova); KUNIT_EXPECT_EQ(test, wire->tail[sizeof(wire->tail) - 2], 0U); KUNIT_EXPECT_EQ(test, wire->tail[sizeof(wire->tail) - 1], 0U); + + /* Source-tag changes must reach the wire without changing transfer. */ + surface->colorspace = DCP_COLORSPACE_BG_SRGB; + iomfb_serialize_present_h17p(wire, request); + KUNIT_EXPECT_EQ(test, ((u8 *)wire)[0x59b], (u8)DCP_XFER_FUNC_SDR); + KUNIT_EXPECT_EQ(test, ((u8 *)wire)[0x59c], (u8)DCP_COLORSPACE_BG_SRGB); } static void h17p_opaque_x_parameter_test(struct kunit *test) { - struct neo_dcp_apply_property_h17p property = neo_dcp_opaque_x_property_h17p(); + struct dcp_apply_property_h17p property = dcp_opaque_x_property_h17p(); KUNIT_EXPECT_EQ(test, property.property, 0x49U); KUNIT_EXPECT_EQ(test, property.value, 0U); @@ -71,69 +79,69 @@ static void h17p_opaque_x_parameter_test(struct kunit *test) static void h17p_callback_bounds_test(struct kunit *test) { /* Every installed handler is reachable without failing the session. */ - KUNIT_EXPECT_EQ(test, neo_iomfb_h17p_first_unbounded_callback(), -1); + KUNIT_EXPECT_EQ(test, iomfb_h17p_first_unbounded_callback(), -1); /* Startup observations stay exact. */ - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(114, 0x1044, 0x1004)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(114, 0x1044, 0x1008)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(590, 0x734, 0x0)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(114, 0x1044, 0x1004)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(114, 0x1044, 0x1008)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(590, 0x734, 0x0)); /* Runtime callbacks must hold what their handlers read and write. */ - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(209, 0x0, 0x8)); - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(209, 0x4, 0xc)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(209, 0x0, 0x4)); - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(593, 0x4, 0x0)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(593, 0x2, 0x0)); - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(594, 0x4, 0x0)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(594, 0x0, 0x0)); - KUNIT_EXPECT_TRUE(test, neo_iomfb_validate_callback_h17p(585, 0xe4, 0xe0)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(585, 0xe0, 0xe0)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(209, 0x0, 0x8)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(209, 0x4, 0xc)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(209, 0x0, 0x4)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(593, 0x4, 0x0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(593, 0x2, 0x0)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(594, 0x4, 0x0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(594, 0x0, 0x0)); + KUNIT_EXPECT_TRUE(test, iomfb_validate_callback_h17p(585, 0xe4, 0xe0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(585, 0xe0, 0xe0)); /* Tags without a handler or out of range never validate. */ - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(5, 0x0, 0x0)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(-1, 0x0, 0x0)); - KUNIT_EXPECT_FALSE(test, neo_iomfb_validate_callback_h17p(IOMFB_MAX_CB, 0x0, 0x0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(5, 0x0, 0x0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(-1, 0x0, 0x0)); + KUNIT_EXPECT_FALSE(test, iomfb_validate_callback_h17p(IOMFB_MAX_CB, 0x0, 0x0)); } static void h17p_present_abort_test(struct kunit *test) { - struct neo_dcp_present_state_h17p state = {}; + struct dcp_present_state_h17p state = {}; /* An abort after acceptance ends the present without a completion. */ - KUNIT_ASSERT_TRUE(test, neo_dcp_present_begin_h17p(&state, 7)); - KUNIT_ASSERT_TRUE(test, neo_dcp_present_submit_h17p(&state, 7, true)); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_abort_h17p(&state, 6)); + KUNIT_ASSERT_TRUE(test, dcp_present_begin_h17p(&state, 7)); + KUNIT_ASSERT_TRUE(test, dcp_present_submit_h17p(&state, 7, true)); + KUNIT_EXPECT_FALSE(test, dcp_present_abort_h17p(&state, 6)); KUNIT_EXPECT_TRUE(test, state.pending); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_was_aborted_h17p(&state, 7)); - KUNIT_EXPECT_TRUE(test, neo_dcp_present_abort_h17p(&state, 7)); + KUNIT_EXPECT_FALSE(test, dcp_present_was_aborted_h17p(&state, 7)); + KUNIT_EXPECT_TRUE(test, dcp_present_abort_h17p(&state, 7)); KUNIT_EXPECT_FALSE(test, state.pending); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_complete_h17p(&state, 7)); + KUNIT_EXPECT_FALSE(test, dcp_present_complete_h17p(&state, 7)); /* A late completion for it is recognisable as stale, not unexpected. */ - KUNIT_EXPECT_TRUE(test, neo_dcp_present_was_aborted_h17p(&state, 7)); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_was_aborted_h17p(&state, 6)); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_abort_h17p(&state, 7)); + KUNIT_EXPECT_TRUE(test, dcp_present_was_aborted_h17p(&state, 7)); + KUNIT_EXPECT_FALSE(test, dcp_present_was_aborted_h17p(&state, 6)); + KUNIT_EXPECT_FALSE(test, dcp_present_abort_h17p(&state, 7)); /* An abort before the submit reply leaves the reply to finish it. */ - KUNIT_ASSERT_TRUE(test, neo_dcp_present_begin_h17p(&state, 8)); - KUNIT_EXPECT_TRUE(test, neo_dcp_present_abort_h17p(&state, 8)); + KUNIT_ASSERT_TRUE(test, dcp_present_begin_h17p(&state, 8)); + KUNIT_EXPECT_TRUE(test, dcp_present_abort_h17p(&state, 8)); KUNIT_EXPECT_TRUE(test, state.pending); KUNIT_EXPECT_TRUE(test, state.aborted); - KUNIT_ASSERT_TRUE(test, neo_dcp_present_submit_h17p(&state, 8, true)); + KUNIT_ASSERT_TRUE(test, dcp_present_submit_h17p(&state, 8, true)); KUNIT_EXPECT_FALSE(test, state.pending); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_complete_h17p(&state, 8)); + KUNIT_EXPECT_FALSE(test, dcp_present_complete_h17p(&state, 8)); /* The next present starts clean and completes normally. */ - KUNIT_ASSERT_TRUE(test, neo_dcp_present_begin_h17p(&state, 9)); + KUNIT_ASSERT_TRUE(test, dcp_present_begin_h17p(&state, 9)); KUNIT_EXPECT_FALSE(test, state.aborted); - KUNIT_ASSERT_TRUE(test, neo_dcp_present_submit_h17p(&state, 9, true)); + KUNIT_ASSERT_TRUE(test, dcp_present_submit_h17p(&state, 9, true)); KUNIT_EXPECT_TRUE(test, state.pending); /* A stale completion for 8 neither matches nor disturbs present 9. */ - KUNIT_EXPECT_FALSE(test, neo_dcp_present_complete_h17p(&state, 8)); - KUNIT_EXPECT_TRUE(test, neo_dcp_present_was_aborted_h17p(&state, 8)); + KUNIT_EXPECT_FALSE(test, dcp_present_complete_h17p(&state, 8)); + KUNIT_EXPECT_TRUE(test, dcp_present_was_aborted_h17p(&state, 8)); KUNIT_EXPECT_TRUE(test, state.pending); - KUNIT_EXPECT_TRUE(test, neo_dcp_present_complete_h17p(&state, 9)); + KUNIT_EXPECT_TRUE(test, dcp_present_complete_h17p(&state, 9)); KUNIT_EXPECT_FALSE(test, state.pending); - KUNIT_EXPECT_FALSE(test, neo_dcp_present_was_aborted_h17p(&state, 9)); + KUNIT_EXPECT_FALSE(test, dcp_present_was_aborted_h17p(&state, 9)); } static struct kunit_case h17p_present_cases[] = { diff --git a/drivers/gpu/drm/apple/iomfb_internal.h b/drivers/gpu/drm/apple/iomfb_internal.h index bd55bfd475f3ed..ce0d7e0867d0be 100644 --- a/drivers/gpu/drm/apple/iomfb_internal.h +++ b/drivers/gpu/drm/apple/iomfb_internal.h @@ -112,3 +112,6 @@ void dcp_ack(struct apple_dcp *dcp, enum dcp_context_id context); /* The user may own drm_display_mode, so we need to search for our copy */ bool lookup_mode(struct apple_dcp *dcp, const struct drm_display_mode *mode, struct dcp_display_mode *out); +/* H17P (T8140, T8142): the catalog entry itself, see iomfb.c. */ +struct dcp_display_mode *lookup_mode_h17p(struct apple_dcp *dcp, + const struct drm_display_mode *mode); diff --git a/drivers/gpu/drm/apple/iomfb_template.c b/drivers/gpu/drm/apple/iomfb_template.c index 7c1f6c76b22693..783ef63094734b 100644 --- a/drivers/gpu/drm/apple/iomfb_template.c +++ b/drivers/gpu/drm/apple/iomfb_template.c @@ -4,6 +4,15 @@ * Copyright The Asahi Linux Contributors */ +/* + * H17-generation firmware (T8140 H17P, T8142 H17G) is driven by the template + * aurora-wip carries for it; the 12.3, 13.5 and 14.7 instances below keep + * integration's. + */ +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) +#include "iomfb_template_h17p.c" +#else + #include #include #include @@ -1082,6 +1091,8 @@ void DCP_FW_NAME(iomfb_sleep)(struct apple_dcp *dcp) static void dcpep_cb_hotplug(struct apple_dcp *dcp, u64 *connected) { struct apple_connector *connector = dcp->connector; + /* sampled before complete_all() lets dcp_poweron() clear it */ + bool poweron_hpd = READ_ONCE(dcp->typec_poweron_hpd); unsigned int action; /* DCP issues hotplug_gated callbacks after SetPowerState() calls on @@ -1112,6 +1123,12 @@ static void dcpep_cb_hotplug(struct apple_dcp *dcp, u64 *connected) schedule_work(&dcp->vblank_wq); return; } + /* DCP lost a sink which is still attached, see dcp_typec_relink() */ + if (!(*connected) && dcp_typec_relink(dcp)) { + dcp_mode_invalidate(&dcp->mode_state); + schedule_work(&dcp->vblank_wq); + return; + } if (dcp_is_typec_output(dcp) && *connected && dcp->nr_modes) complete_all(&dcp->typec_iomfb_hpd_ready); @@ -1129,8 +1146,13 @@ static void dcpep_cb_hotplug(struct apple_dcp *dcp, u64 *connected) * A Type-C sink can assert HPD only after a modeset has already failed, * as a TV behind a DP-to-HDMI converter does when it wakes from standby. * The connector state does not change then; re-apply the mode anyway. + * + * Not for the HPD dcp_poweron() waits for: the modeset of that commit + * has not started yet, rather than failed. Reporting the link as bad + * then makes wlroots replace the output, which powers the CRTC off and + * on again, and the next wake does the same, forever. */ - if (*connected && dcp_is_typec_output(dcp) && + if (*connected && dcp_is_typec_output(dcp) && !poweron_hpd && !READ_ONCE(dcp->mode_state.valid) && !READ_ONCE(dcp->mode_state.changing)) { dcp->swap_watchdog_retrains = 0; @@ -1636,3 +1658,4 @@ void DCP_FW_NAME(iomfb_shutdown)(struct apple_dcp *dcp) dcp_set_power_state(dcp, false, &req, NULL, NULL); } +#endif /* DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) */ diff --git a/drivers/gpu/drm/apple/iomfb_template.h b/drivers/gpu/drm/apple/iomfb_template.h index 6f5824c8043f3b..b9e3fe9e7f2aec 100644 --- a/drivers/gpu/drm/apple/iomfb_template.h +++ b/drivers/gpu/drm/apple/iomfb_template.h @@ -24,10 +24,21 @@ struct DCP_FW_NAME(dcp_swap) { u64 flags1; u64 flags2; -#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) +#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) && DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) u8 unk_v14_7[0x48]; #endif +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) + /* + * H17P inserts 0x48 bytes between flags2 and the swap ID, which moves + * to +0x98. Every following field shifts with it: surf_ids +0x9c, + * src_rect +0xac, surf_flags +0xec, dst_rect +0x10c, + * swap_enabled/completed +0x14c/+0x150, bg_color +0x154. The gap holds + * two small u32 counters (at +0x50 and +0x78) that are not modelled. + */ + u8 h17p_pre_swap_id[0x48]; +#endif + u32 swap_id; u32 surf_ids[SWAP_SURFACES]; @@ -41,7 +52,7 @@ struct DCP_FW_NAME(dcp_swap) { u32 bg_color; u8 unk_110[0x1b8]; u32 unk_2c8; -#if DCP_FW_VER < DCP_FW_VERSION(14, 7, 0) +#if DCP_FW_VER < DCP_FW_VERSION(14, 7, 0) || DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) u8 unk_2cc[0x14]; u32 unk_2e0; #else @@ -58,7 +69,7 @@ struct DCP_FW_NAME(dcp_swap) { #else u8 unk_2e2[3]; #endif -#if DCP_FW_VER < DCP_FW_VERSION(14, 7, 0) +#if DCP_FW_VER < DCP_FW_VERSION(14, 7, 0) || DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) u64 bl_unk; #else u32 bl_unk_v14_7; /* the 13.x bl_unk position; not written */ @@ -68,11 +79,18 @@ struct DCP_FW_NAME(dcp_swap) { u8 unk_2f3[0x2d]; #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) u8 unk_320[0x13f]; -#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) +#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) && DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) u8 unk_14_7_2[0x30]; #endif u64 unk_1; #endif +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) + /* + * The H17P A408 request is 0xe9c bytes; with a 0x22c-byte surface, + * SWAP_SURFACES = 4 and no surf2 block, that leaves this tail. + */ + u8 h17p_tail[0xda]; +#endif } __packed; /* Information describing a surface */ @@ -93,8 +111,14 @@ struct DCP_FW_NAME(dcp_swap_submit_req) { u64 surf_iova[SWAP_SURFACES]; #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) u64 unk_u64_a[SWAP_SURFACES]; + /* + * H17P carries no surf2 block: its trailing u8 null-flag run is 8 bytes + * (swap_null + surf_null[4] + three tail bools). + */ +#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) struct DCP_FW_NAME(dcp_surface) surf2[5]; u64 surf2_iova[5]; +#endif #endif u8 unkbool; u64 unkdouble; @@ -106,20 +130,27 @@ struct DCP_FW_NAME(dcp_swap_submit_req) { #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) u32 unkU32Ptr; #endif -#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) +#if DCP_FW_VER >= DCP_FW_VERSION(14, 7, 0) && DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) u8 padding_14_7[0x234]; #endif u8 swap_null; u8 surf_null[SWAP_SURFACES]; #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) +#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) u8 surf2_null[5]; +#endif #endif u8 unkoutbool_null; -#if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) + u8 unknown_pointer_null; + u8 unknown_output_null; +#elif DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) u8 unkU32Ptr_null; u8 unkU32out_null; #endif +#if DCP_FW_VER < DCP_FW_VERSION(26, 0, 0) u8 padding[1]; +#endif } __packed; struct DCP_FW_NAME(dcp_swap_submit_resp) { @@ -131,9 +162,104 @@ struct DCP_FW_NAME(dcp_swap_submit_resp) { u8 padding[3]; } __packed; -struct DCP_FW_NAME(dc_swap_complete_resp) { +/* + * A406 swap_start / A473 set_power_state. H17P shrank both requests: the + * 0x10-byte struct dcp_iouserclient became a bare u64 handle and the + * swap_start response no longer echoes it. swap_start is 0x10 in / 0x8 out, + * set_power_state 0x10 in. + */ +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) +struct DCP_FW_NAME(dcp_swap_start_req) { + u32 swap_id; + u64 client; /* IOUserClient kernel VA */ + u8 client_null; + u8 padding[3]; +} __packed; + +struct DCP_FW_NAME(dcp_swap_start_resp) { + u32 swap_id; + u32 ret; +} __packed; + +struct DCP_FW_NAME(dcp_set_power_state_req) { + u64 unklong; + u32 unkint; /* widened from u8 on H17P */ + u8 unkint_null; + u8 padding[3]; +} __packed; + +static_assert(sizeof(struct DCP_FW_NAME(dcp_swap_start_req)) == 0x10); +static_assert(sizeof(struct DCP_FW_NAME(dcp_swap_start_resp)) == 0x8); +static_assert(sizeof(struct DCP_FW_NAME(dcp_set_power_state_req)) == 0x10); +#else +struct DCP_FW_NAME(dcp_swap_start_req) { u32 swap_id; + struct dcp_iouserclient client; + u8 swap_id_null; + u8 client_null; + u8 padding[2]; +} __packed; + +struct DCP_FW_NAME(dcp_swap_start_resp) { + u32 swap_id; + struct dcp_iouserclient client; + u32 ret; +} __packed; + +struct DCP_FW_NAME(dcp_set_power_state_req) { + u64 unklong; u8 unkbool; + u8 unkint_null; + u8 padding[2]; +} __packed; +#endif + +/* + * D201 map_piodma reply. H17P declares a 16-byte output, not 20, laid out as + * { u32 buffer index, u64 dva, u32 pad } rather than { u64 vaddr, u64 dva, + * u32 ret }. With the old layout the DVA lands four bytes late and the + * display engine DMAs to a bogus address. + */ +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) +struct DCP_FW_NAME(dcp_map_buf_resp) { + u32 buffer; + u64 dva; + u32 unk; +} __packed; +static_assert(sizeof(struct DCP_FW_NAME(dcp_map_buf_resp)) == 0x10); +#else +struct DCP_FW_NAME(dcp_map_buf_resp) { + u64 vaddr; + u64 dva; + u32 ret; +} __packed; +#endif + +struct DCP_FW_NAME(dc_swap_complete_resp) { + u32 swap_id; /* +0x000 */ + u8 unkbool; /* +0x004 */ +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) + /* + * H17P: swap_complete_ap_gated(u32, bool, SwapCompleteData const *, + * SwapInfoBlob const *, u32, bool) is 0x730 bytes on the wire, 89 more + * than the pre-H17P record: the SwapCompleteData grew from 8 to 0x22, + * swap_info became eight 0xe0-byte SwapInfoBlobs rather than one + * 0x6c5 blob, and the tail gained a flag and three pad bytes. Only + * swap_info_count of those blobs are populated; the rest are + * uninitialised DCP buffer memory. + */ + u32 hol_swap_ids[4]; /* +0x005 */ + u32 hol_arg; /* +0x015 */ + u32 hol_count; /* +0x019 */ + u8 hol_valid; /* +0x01d */ + u64 displayed_data; /* +0x01e, unaligned */ + u8 notify_displayed; /* +0x026 */ + u8 swap_info[8][0xe0]; /* +0x027, blob[0]+0x38 == swap_id */ + u32 swap_info_count; /* +0x727, max 8 */ + u8 completion_flag; /* +0x72b */ + u8 swap_data_null; /* +0x72c, 0 => data at +0x005 valid */ + u8 padding[3]; /* +0x72d, to sizeof == 0x730 */ +#else u64 swap_data; #if DCP_FW_VER < DCP_FW_VERSION(13, 2, 0) u8 swap_info[0x6c4]; @@ -142,6 +268,7 @@ struct DCP_FW_NAME(dc_swap_complete_resp) { #endif u32 unkint; u8 swap_info_null; +#endif } __packed; struct DCP_FW_NAME(dcp_map_reg_req) { diff --git a/drivers/gpu/drm/apple-neo/iomfb_template.c b/drivers/gpu/drm/apple/iomfb_template_h17p.c similarity index 50% rename from drivers/gpu/drm/apple-neo/iomfb_template.c rename to drivers/gpu/drm/apple/iomfb_template_h17p.c index def92694ff5726..1dd691dd315b0f 100644 --- a/drivers/gpu/drm/apple-neo/iomfb_template.c +++ b/drivers/gpu/drm/apple/iomfb_template_h17p.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -36,10 +37,10 @@ /* Register defines used in bandwidth setup structure */ #define REG_DOORBELL_BIT(idx) (2 + (idx)) -static_assert(offsetof(struct DCP_FW_NAME(neo_dcp_swap), timestamp[6]) == 0x30); -static_assert(offsetof(struct DCP_FW_NAME(neo_dcp_swap), flags1) == 0x40); +static_assert(offsetof(struct DCP_FW_NAME(dcp_swap), timestamp[6]) == 0x30); +static_assert(offsetof(struct DCP_FW_NAME(dcp_swap), flags1) == 0x40); -struct neo_dcp_wait_cookie { +struct dcp_wait_cookie { struct kref refcount; struct completion done; u32 status; @@ -47,15 +48,15 @@ struct neo_dcp_wait_cookie { static void release_wait_cookie(struct kref *ref) { - struct neo_dcp_wait_cookie *cookie; - cookie = container_of(ref, struct neo_dcp_wait_cookie, refcount); + struct dcp_wait_cookie *cookie; + cookie = container_of(ref, struct dcp_wait_cookie, refcount); kfree(cookie); } -DCP_THUNK_OUT(neo_iomfb_a131_pmu_service_matched, iomfbep_a131_pmu_service_matched, u32); -DCP_THUNK_OUT(neo_iomfb_a132_backlight_service_matched, iomfbep_a132_backlight_service_matched, u32); -DCP_THUNK_OUT(neo_iomfb_a358_vi_set_temperature_hint, iomfbep_a358_vi_set_temperature_hint, u32); +DCP_THUNK_OUT(iomfb_a131_pmu_service_matched, iomfbep_a131_pmu_service_matched, u32); +DCP_THUNK_OUT(iomfb_a132_backlight_service_matched, iomfbep_a132_backlight_service_matched, u32); +DCP_THUNK_OUT(iomfb_a358_vi_set_temperature_hint, iomfbep_a358_vi_set_temperature_hint, u32); IOMFB_THUNK_INOUT(set_matrix); IOMFB_THUNK_INOUT(get_color_remap_mode); @@ -63,37 +64,37 @@ IOMFB_THUNK_INOUT(last_client_close); IOMFB_THUNK_INOUT(abort_swaps_dcp); #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER -static void neo_dcp_swap_submit(struct neo_apple_dcp *neo_dcp, bool oob, - struct neo_dcp_swap_submit_req_h17p *request, - neo_dcp_callback_t cb, void *cookie) +static void dcp_swap_submit(struct apple_dcp *dcp, bool oob, + struct dcp_swap_submit_req_h17p *request, + dcp_callback_t cb, void *cookie) { const void *data = request; - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - neo_iomfb_serialize_present_h17p(&neo_dcp->present_h17p, request); - if (!neo_iomfb_apply_backlight_h17p(neo_dcp, request, &neo_dcp->present_h17p)) { + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + iomfb_serialize_present_h17p(&dcp->present_h17p, request); + if (!iomfb_apply_backlight_h17p(dcp, request, &dcp->present_h17p)) { if (cb) - cb(neo_dcp, NULL, cookie); + cb(dcp, NULL, cookie); return; } - data = &neo_dcp->present_h17p; + data = &dcp->present_h17p; } - neo_dcp_push(neo_dcp, oob, &neo_dcp_methods[dcpep_swap_submit], sizeof(*request), - sizeof(struct neo_dcp_swap_submit_resp_h17p), (void *)data, cb, cookie); + dcp_push(dcp, oob, &dcp_methods[dcpep_swap_submit], sizeof(*request), + sizeof(struct dcp_swap_submit_resp_h17p), (void *)data, cb, cookie); } #else -DCP_THUNK_INOUT(neo_dcp_swap_submit, dcpep_swap_submit, - struct DCP_FW_NAME(neo_dcp_swap_submit_req), - struct DCP_FW_NAME(neo_dcp_swap_submit_resp)); +DCP_THUNK_INOUT(dcp_swap_submit, dcpep_swap_submit, + struct DCP_FW_NAME(dcp_swap_submit_req), + struct DCP_FW_NAME(dcp_swap_submit_resp)); #endif -DCP_THUNK_INOUT(neo_dcp_swap_start, dcpep_swap_start, struct DCP_FW_NAME(neo_dcp_swap_start_req), - struct DCP_FW_NAME(neo_dcp_swap_start_resp)); +DCP_THUNK_INOUT(dcp_swap_start, dcpep_swap_start, struct DCP_FW_NAME(dcp_swap_start_req), + struct DCP_FW_NAME(dcp_swap_start_resp)); -DCP_THUNK_INOUT(neo_dcp_set_power_state, dcpep_set_power_state, - struct DCP_FW_NAME(neo_dcp_set_power_state_req), - struct neo_dcp_set_power_state_resp); +DCP_THUNK_INOUT(dcp_set_power_state, dcpep_set_power_state, + struct DCP_FW_NAME(dcp_set_power_state_req), + struct dcp_set_power_state_resp); /* * H17P firmware expects set_power_state (A473) on the out-of-band command @@ -132,24 +133,24 @@ DCP_THUNK_INOUT(neo_dcp_set_power_state, dcpep_set_power_state, #define DCP_HAS_CLIENT_TEARDOWN true #endif -DCP_THUNK_INOUT(neo_dcp_set_digital_out_mode, dcpep_set_digital_out_mode, - struct neo_dcp_set_digital_out_mode_req, u32); +DCP_THUNK_INOUT(dcp_set_digital_out_mode, dcpep_set_digital_out_mode, + struct dcp_set_digital_out_mode_req, u32); -DCP_THUNK_INOUT(neo_dcp_set_display_device, dcpep_set_display_device, u32, u32); +DCP_THUNK_INOUT(dcp_set_display_device, dcpep_set_display_device, u32, u32); -DCP_THUNK_OUT(neo_dcp_set_display_refresh_properties, +DCP_THUNK_OUT(dcp_set_display_refresh_properties, dcpep_set_display_refresh_properties, u32); #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) -DCP_THUNK_INOUT(neo_dcp_late_init_signal, dcpep_late_init_signal, u32, u32); +DCP_THUNK_INOUT(dcp_late_init_signal, dcpep_late_init_signal, u32, u32); #else -DCP_THUNK_OUT(neo_dcp_late_init_signal, dcpep_late_init_signal, u32); +DCP_THUNK_OUT(dcp_late_init_signal, dcpep_late_init_signal, u32); #endif -DCP_THUNK_IN(neo_dcp_flush_supports_power, dcpep_flush_supports_power, u32); -DCP_THUNK_OUT(neo_dcp_create_default_fb, dcpep_create_default_fb, u32); -DCP_THUNK_OUT(neo_dcp_start_signal, dcpep_start_signal, u32); -DCP_THUNK_VOID(neo_dcp_setup_video_limits, dcpep_setup_video_limits); -DCP_THUNK_VOID(neo_dcp_set_create_dfb, dcpep_set_create_dfb); +DCP_THUNK_IN(dcp_flush_supports_power, dcpep_flush_supports_power, u32); +DCP_THUNK_OUT(dcp_create_default_fb, dcpep_create_default_fb, u32); +DCP_THUNK_OUT(dcp_start_signal, dcpep_start_signal, u32); +DCP_THUNK_VOID(dcp_setup_video_limits, dcpep_setup_video_limits); +DCP_THUNK_VOID(dcp_set_create_dfb, dcpep_set_create_dfb); #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* @@ -157,38 +158,38 @@ DCP_THUNK_VOID(neo_dcp_set_create_dfb, dcpep_set_create_dfb); * The getters take a zeroed input of the expected size; the firmware only * checks the size. */ -struct neo_dcp_dfb_info_h17p { +struct dcp_dfb_info_h17p { u8 data[0x2c]; } __packed; -struct neo_dcp_dfb_layout_req_h17p { +struct dcp_dfb_layout_req_h17p { u8 data[0xc]; } __packed; -struct neo_dcp_dfb_dimensions_h17p { +struct dcp_dfb_dimensions_h17p { u32 width; u32 height; } __packed; -DCP_THUNK_INOUT(neo_dcp_get_dfb_compression_info, dcpep_get_dfb_compression_info, +DCP_THUNK_INOUT(dcp_get_dfb_compression_info, dcpep_get_dfb_compression_info, u64, u64); -DCP_THUNK_INOUT(neo_dcp_get_dfb_info, dcpep_get_dfb_info, - struct neo_dcp_dfb_info_h17p, struct neo_dcp_dfb_info_h17p); -DCP_THUNK_INOUT(neo_dcp_get_dfb_layout, dcpep_get_dfb_layout, - struct neo_dcp_dfb_layout_req_h17p, u64); -DCP_THUNK_INOUT(neo_dcp_get_dfb_state, dcpep_get_dfb_state, u32, u64); -DCP_THUNK_IN(neo_dcp_set_dfb_dimensions, dcpep_set_dfb_dimensions, - struct neo_dcp_dfb_dimensions_h17p); -DCP_THUNK_VOID(neo_dcp_commit_dfb_info, dcpep_commit_dfb_info); -DCP_THUNK_OUT(neo_dcp_dfb_query, dcpep_dfb_query, u32); -DCP_THUNK_OUT(neo_dcp_dfb_ready_query, dcpep_dfb_ready_query, u32); +DCP_THUNK_INOUT(dcp_get_dfb_info, dcpep_get_dfb_info, + struct dcp_dfb_info_h17p, struct dcp_dfb_info_h17p); +DCP_THUNK_INOUT(dcp_get_dfb_layout, dcpep_get_dfb_layout, + struct dcp_dfb_layout_req_h17p, u64); +DCP_THUNK_INOUT(dcp_get_dfb_state, dcpep_get_dfb_state, u32, u64); +DCP_THUNK_IN(dcp_set_dfb_dimensions, dcpep_set_dfb_dimensions, + struct dcp_dfb_dimensions_h17p); +DCP_THUNK_VOID(dcp_commit_dfb_info, dcpep_commit_dfb_info); +DCP_THUNK_OUT(dcp_dfb_query, dcpep_dfb_query, u32); +DCP_THUNK_OUT(dcp_dfb_ready_query, dcpep_dfb_ready_query, u32); /* * Pipe-configuration calls the firmware expects between the boot chain and * the first swap. Most of them have no known name; they are identified by * method number and wire size. */ -struct neo_dcp_pipe_cfg_415 { +struct dcp_pipe_cfg_415 { u8 data[0xc]; } __packed; @@ -197,110 +198,110 @@ struct neo_dcp_pipe_cfg_415 { * notification clients the AP wants delivered. Clients that are not * registered here never send the callbacks they gate. */ -struct neo_dcp_pipe_cfg_031 { +struct dcp_pipe_cfg_031 { __le32 clients[27]; } __packed; -static_assert(sizeof(struct neo_dcp_pipe_cfg_031) == 0x6c); - -DCP_THUNK_INOUT(neo_dcp_pipe_cfg_415, dcpep_pipe_cfg_415, - struct neo_dcp_pipe_cfg_415, struct neo_dcp_pipe_cfg_415); -DCP_THUNK_IN(neo_dcp_pipe_cfg_031, dcpep_pipe_cfg_031, struct neo_dcp_pipe_cfg_031); -DCP_THUNK_INOUT(neo_dcp_pipe_cfg_414, dcpep_pipe_cfg_414, u64, u64); -DCP_THUNK_OUT(neo_dcp_pipe_query_478, dcpep_pipe_query_478, u32); -DCP_THUNK_OUT(neo_dcp_pipe_query_474, dcpep_pipe_query_474, u32); -DCP_THUNK_IN(neo_dcp_pipe_set_476, dcpep_pipe_set_476, u32); -DCP_THUNK_INOUT(neo_dcp_pipe_set_428, dcpep_pipe_set_428, u32, u32); +static_assert(sizeof(struct dcp_pipe_cfg_031) == 0x6c); + +DCP_THUNK_INOUT(dcp_pipe_cfg_415, dcpep_pipe_cfg_415, + struct dcp_pipe_cfg_415, struct dcp_pipe_cfg_415); +DCP_THUNK_IN(dcp_pipe_cfg_031, dcpep_pipe_cfg_031, struct dcp_pipe_cfg_031); +DCP_THUNK_INOUT(dcp_pipe_cfg_414, dcpep_pipe_cfg_414, u64, u64); +DCP_THUNK_OUT(dcp_pipe_query_478, dcpep_pipe_query_478, u32); +DCP_THUNK_OUT(dcp_pipe_query_474, dcpep_pipe_query_474, u32); +DCP_THUNK_IN(dcp_pipe_set_476, dcpep_pipe_set_476, u32); +DCP_THUNK_INOUT(dcp_pipe_set_428, dcpep_pipe_set_428, u32, u32); #endif -DCP_THUNK_VOID(neo_dcp_first_client_open, dcpep_first_client_open); +DCP_THUNK_VOID(dcp_first_client_open, dcpep_first_client_open); -DCP_THUNK_INOUT(neo_dcp_set_parameter_dcp, dcpep_set_parameter_dcp, - struct neo_dcp_set_parameter_dcp, u32); +DCP_THUNK_INOUT(dcp_set_parameter_dcp, dcpep_set_parameter_dcp, + struct dcp_set_parameter_dcp, u32); -DCP_THUNK_INOUT(neo_dcp_enable_disable_video_power_savings, +DCP_THUNK_INOUT(dcp_enable_disable_video_power_savings, dcpep_enable_disable_video_power_savings, u32, int); -DCP_THUNK_OUT(neo_dcp_is_main_display, dcpep_is_main_display, u32); +DCP_THUNK_OUT(dcp_is_main_display, dcpep_is_main_display, u32); /* DCP callback handlers */ -static void dcpep_cb_nop(struct neo_apple_dcp *neo_dcp) +static void dcpep_cb_nop(struct apple_dcp *dcp) { /* No operation */ } -static u8 dcpep_cb_true(struct neo_apple_dcp *neo_dcp) +static u8 dcpep_cb_true(struct apple_dcp *dcp) { return true; } -static u8 dcpep_cb_false(struct neo_apple_dcp *neo_dcp) +static u8 dcpep_cb_false(struct apple_dcp *dcp) { return false; } -static u32 dcpep_cb_zero(struct neo_apple_dcp *neo_dcp) +static u32 dcpep_cb_zero(struct apple_dcp *dcp) { return 0; } -static void dcpep_cb_swap_complete(struct neo_apple_dcp *neo_dcp, +static void dcpep_cb_swap_complete(struct apple_dcp *dcp, struct DCP_FW_NAME(dc_swap_complete_resp) *resp) { ktime_t now = ktime_get(); bool retire = true; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !neo_dcp_present_complete_h17p(&neo_dcp->present_state_h17p, resp->swap_id)) { + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !dcp_present_complete_h17p(&dcp->present_state_h17p, resp->swap_id)) { /* * Abort semantics are not captured on H17P. If an aborted * present completes after all, its framebuffers were already * unarmed and its event signalled; record it and carry on. */ - if (neo_dcp_present_was_aborted_h17p(&neo_dcp->present_state_h17p, + if (dcp_present_was_aborted_h17p(&dcp->present_state_h17p, resp->swap_id)) { - dev_warn_ratelimited(neo_dcp->dev, + dev_warn_ratelimited(dcp->dev, "completion for aborted present %u ignored\n", resp->swap_id); return; } - dev_err(neo_dcp->dev, "unexpected present completion %u\n", resp->swap_id); - neo_dcp->crashed = true; + dev_err(dcp->dev, "unexpected present completion %u\n", resp->swap_id); + dcp->crashed = true; return; } - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - retire = neo_iomfb_present_complete_h17p(neo_dcp); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + retire = iomfb_present_complete_h17p(dcp); #endif - trace_neo_iomfb_swap_complete(neo_dcp, resp->swap_id); - neo_dcp->last_swap_id = resp->swap_id; - neo_dcp_swap_watchdog_complete(neo_dcp); + trace_iomfb_swap_complete(dcp, resp->swap_id); + dcp->last_swap_id = resp->swap_id; + dcp_swap_watchdog_complete(dcp); if (retire) { - neo_dcp_release_retained_framebuffers(neo_dcp, resp->swap_id); - neo_dcp_drm_crtc_page_flip(neo_dcp, now); + dcp_release_retained_framebuffers(dcp, resp->swap_id); + dcp_drm_crtc_page_flip(dcp, now); } - if (READ_ONCE(neo_dcp->crc_enabled)) { + if (READ_ONCE(dcp->crc_enabled)) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - neo_iomfb_queue_crc_h17p(neo_dcp, resp->swap_id); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + iomfb_queue_crc_h17p(dcp, resp->swap_id); return; } #endif u32 crc32 = 0; - drm_crtc_add_crc_entry(&neo_dcp->crtc->base, true, resp->swap_id, &crc32); + drm_crtc_add_crc_entry(&dcp->crtc->base, true, resp->swap_id, &crc32); } } /* special */ -static void complete_vi_set_temperature_hint(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void complete_vi_set_temperature_hint(struct apple_dcp *dcp, void *out, void *cookie) { // ack D100 cb_match_pmu_service - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } -static bool iomfbep_cb_match_pmu_service(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) +static bool iomfbep_cb_match_pmu_service(struct apple_dcp *dcp, int tag, void *out, void *in) { - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - neo_iomfb_a358_vi_set_temperature_hint(neo_dcp, false, + trace_iomfb_callback(dcp, tag, __func__); + iomfb_a358_vi_set_temperature_hint(dcp, false, complete_vi_set_temperature_hint, NULL); @@ -316,27 +317,27 @@ static bool iomfbep_cb_match_pmu_service(struct neo_apple_dcp *neo_dcp, int tag, * to report success. Without it the firmware has no surface, programs the * display engine from a null base and the piodma DART faults. */ -struct neo_dcp_h17p_dfb_surface_req { +struct dcp_h17p_dfb_surface_req { u8 surface[0x22c]; u8 surface_null; u8 padding[3]; } __packed; -static_assert(sizeof(struct neo_dcp_h17p_dfb_surface_req) == 0x230); +static_assert(sizeof(struct dcp_h17p_dfb_surface_req) == 0x230); /* * A472 has a 4-byte output; with out_len 0 the firmware has nowhere to write * its result and stops answering. */ -DCP_THUNK_INOUT(neo_dcp_register_dfb_surface, dcpep_register_dfb_surface, - struct neo_dcp_h17p_dfb_surface_req, u32); +DCP_THUNK_INOUT(dcp_register_dfb_surface, dcpep_register_dfb_surface, + struct dcp_h17p_dfb_surface_req, u32); -static void neo_dcp_ack_dfb_surface(struct neo_apple_dcp *neo_dcp) +static void dcp_ack_dfb_surface(struct apple_dcp *dcp) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; u8 *succ = ch->output[ch->depth - 1]; *succ = true; - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } /* @@ -345,7 +346,7 @@ static void neo_dcp_ack_dfb_surface(struct neo_apple_dcp *neo_dcp) * default surface ID. The bytes at +0x29..+0x2c are an opaque word whose * meaning is not known. */ -static void neo_dcp_fill_dfb_surface(struct neo_apple_dcp *neo_dcp, u8 *s) +static void dcp_fill_dfb_surface(struct apple_dcp *dcp, u8 *s) { s[0x002] = 0x01; memcpy(s + 0x00b, "8a3b", 4); @@ -358,7 +359,7 @@ static void neo_dcp_fill_dfb_surface(struct neo_apple_dcp *neo_dcp, u8 *s) s[0x051] = 0x01; s[0x149] = 0x01; - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { /* Internal H17P default registration differs from the older profile. */ memcpy(s + 11, "ARGB", 4); put_unaligned_le32(0x00e44000, s + 41); @@ -371,17 +372,17 @@ static void neo_dcp_fill_dfb_surface(struct neo_apple_dcp *neo_dcp, u8 *s) * AXI read error. Scanout does not need it, since every swap carries its own * surface. Just acknowledge the callback. */ -static void complete_dfb_surface(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void complete_dfb_surface(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_ack_dfb_surface(neo_dcp); + dcp_ack_dfb_surface(dcp); } -static bool iomfbep_cb_create_dfb_surface(struct neo_apple_dcp *neo_dcp, int tag, +static bool iomfbep_cb_create_dfb_surface(struct apple_dcp *dcp, int tag, void *out, void *in) { - struct neo_dcp_h17p_dfb_surface_req *req; + struct dcp_h17p_dfb_surface_req *req; - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); /* * The request is too large for the stack this deep in the RTKit @@ -393,9 +394,9 @@ static bool iomfbep_cb_create_dfb_surface(struct neo_apple_dcp *neo_dcp, int tag if (!req) return true; - neo_dcp_fill_dfb_surface(neo_dcp, req->surface); + dcp_fill_dfb_surface(dcp, req->surface); - neo_dcp_register_dfb_surface(neo_dcp, false, req, complete_dfb_surface, NULL); + dcp_register_dfb_surface(dcp, false, req, complete_dfb_surface, NULL); kfree(req); /* deferred ACK: the A472 completion reports success for us */ @@ -403,87 +404,87 @@ static bool iomfbep_cb_create_dfb_surface(struct neo_apple_dcp *neo_dcp, int tag } #endif -static void complete_pmu_service_matched(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void complete_pmu_service_matched(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; u8 *succ = ch->output[ch->depth - 1]; *succ = true; // ack D206 cb_match_pmu_service_2 - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } -static bool iomfbep_cb_match_pmu_service_2(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) +static bool iomfbep_cb_match_pmu_service_2(struct apple_dcp *dcp, int tag, void *out, void *in) { - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); - neo_iomfb_a131_pmu_service_matched(neo_dcp, false, complete_pmu_service_matched, + iomfb_a131_pmu_service_matched(dcp, false, complete_pmu_service_matched, out); // return false for deferred ACK return false; } -static void complete_backlight_service_matched(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void complete_backlight_service_matched(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; u8 *succ = ch->output[ch->depth - 1]; *succ = true; // ack D206 cb_match_backlight_service - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } -static bool iomfbep_cb_match_backlight_service(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) +static bool iomfbep_cb_match_backlight_service(struct apple_dcp *dcp, int tag, void *out, void *in) { - trace_neo_iomfb_callback(neo_dcp, tag, __func__); + trace_iomfb_callback(dcp, tag, __func__); - if (!neo_dcp_has_panel(neo_dcp)) { + if (!dcp_has_panel(dcp)) { u8 *succ = out; *succ = true; return true; } - neo_iomfb_a132_backlight_service_matched(neo_dcp, false, complete_backlight_service_matched, out); + iomfb_a132_backlight_service_matched(dcp, false, complete_backlight_service_matched, out); // return false for deferred ACK return false; } -static void neo_iomfb_cb_pr_publish(struct neo_apple_dcp *neo_dcp, struct neo_iomfb_property *prop) +static void iomfb_cb_pr_publish(struct apple_dcp *dcp, struct iomfb_property *prop) { switch (prop->id) { case IOMFB_PROPERTY_NITS: { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - /* Measured H17P takeover comes from the powerlog hint interface. */ - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + /* The H17P takeover level comes from the powerlog hint interface. */ + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) break; #endif - if (neo_dcp_has_panel(neo_dcp)) { - neo_dcp->brightness.nits = prop->value / neo_dcp->brightness.scale; + if (dcp_has_panel(dcp)) { + dcp->brightness.nits = prop->value / dcp->brightness.scale; /* notify backlight device of the initial brightness */ - if (!neo_dcp->brightness.bl_dev && neo_dcp->brightness.maximum > 0) - schedule_work(&neo_dcp->bl_register_wq); - trace_neo_iomfb_brightness(neo_dcp, prop->value); + if (!dcp->brightness.bl_dev && dcp->brightness.maximum > 0) + schedule_work(&dcp->bl_register_wq); + trace_iomfb_brightness(dcp, prop->value); } break; } default: - dev_dbg(neo_dcp->dev, "pr_publish: id: %d = %u\n", prop->id, prop->value); + dev_dbg(dcp->dev, "pr_publish: id: %d = %u\n", prop->id, prop->value); } } -static struct neo_dcp_get_uint_prop_resp -dcpep_cb_get_uint_prop(struct neo_apple_dcp *neo_dcp, struct neo_dcp_get_uint_prop_req *req) +static struct dcp_get_uint_prop_resp +dcpep_cb_get_uint_prop(struct apple_dcp *dcp, struct dcp_get_uint_prop_req *req) { - struct neo_dcp_get_uint_prop_resp resp = (struct neo_dcp_get_uint_prop_resp){ + struct dcp_get_uint_prop_resp resp = (struct dcp_get_uint_prop_resp){ .value = 0 }; - if (neo_dcp->panel.has_mini_led && + if (dcp->panel.has_mini_led && memcmp(req->obj, "SUMP", sizeof(req->obj)) == 0) { /* "PMUS */ if (strncmp(req->key, "Temperature", sizeof(req->key)) == 0) { /* @@ -498,20 +499,20 @@ dcpep_cb_get_uint_prop(struct neo_apple_dcp *neo_dcp, struct neo_dcp_get_uint_pr return resp; } -static u8 iomfbep_cb_sr_set_property_int(struct neo_apple_dcp *neo_dcp, - struct neo_iomfb_sr_set_property_int_req *req) +static u8 iomfbep_cb_sr_set_property_int(struct apple_dcp *dcp, + struct iomfb_sr_set_property_int_req *req) { if (memcmp(req->obj, "FMOI", sizeof(req->obj)) == 0) { /* "IOMF */ if (strncmp(req->key, "Brightness_Scale", sizeof(req->key)) == 0) { if (!req->value_null) - neo_dcp->brightness.scale = req->value; + dcp->brightness.scale = req->value; } } return 1; } -static void iomfbep_cb_set_fx_prop(struct neo_apple_dcp *neo_dcp, struct neo_iomfb_set_fx_prop_req *req) +static void iomfbep_cb_set_fx_prop(struct apple_dcp *dcp, struct iomfb_set_fx_prop_req *req) { // TODO: trace this, see if there properties which needs to used later } @@ -521,36 +522,36 @@ static void iomfbep_cb_set_fx_prop(struct neo_apple_dcp *neo_dcp, struct neo_iom * PIODMA is separate from the main DCP and uses own IOVA space on a dedicated * stream of the display DART, rather than the expected DCP DART. */ -static bool dcpep_map_piodma(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_map_buf_req *req, - struct DCP_FW_NAME(neo_dcp_map_buf_resp) * resp) +static bool dcpep_map_piodma(struct apple_dcp *dcp, + struct dcp_map_buf_req *req, + struct DCP_FW_NAME(dcp_map_buf_resp) * resp) { - struct neo_dcp_mem_descriptor *memdesc; + struct dcp_mem_descriptor *memdesc; struct sg_table *map; size_t size; ssize_t ret; - if (req->buffer >= ARRAY_SIZE(neo_dcp->memdesc)) + if (req->buffer >= ARRAY_SIZE(dcp->memdesc)) goto reject; - memdesc = &neo_dcp->memdesc[req->buffer]; + memdesc = &dcp->memdesc[req->buffer]; map = &memdesc->map; size = ALIGN(memdesc->size, SZ_16K); - if (!test_bit(req->buffer, neo_dcp->memdesc_map) || !map->sgl || - memdesc->piodma_mapped || !neo_dcp->iommu_dom) + if (!test_bit(req->buffer, dcp->memdesc_map) || !map->sgl || + memdesc->piodma_mapped || !dcp->iommu_dom) goto reject; /* use the piodma iommu domain to map against the right IOMMU */ - ret = iommu_map_sgtable(neo_dcp->iommu_dom, memdesc->dva, map, + ret = iommu_map_sgtable(dcp->iommu_dom, memdesc->dva, map, IOMMU_READ | IOMMU_WRITE); if (ret != (ssize_t)size) { - dev_err(neo_dcp->dev, + dev_err(dcp->dev, "iommu_map_sgtable() returned %zd instead of %zu\n", ret, size); if (ret > 0) - iommu_unmap(neo_dcp->iommu_dom, memdesc->dva, ret); + iommu_unmap(dcp->iommu_dom, memdesc->dva, ret); goto reject; } @@ -558,27 +559,27 @@ static bool dcpep_map_piodma(struct neo_apple_dcp *neo_dcp, memdesc->piodma_mapped = true; /* the H17P reply echoes the buffer index */ - *resp = (struct DCP_FW_NAME(neo_dcp_map_buf_resp)) { + *resp = (struct DCP_FW_NAME(dcp_map_buf_resp)) { .buffer = req->buffer, .dva = memdesc->dva, }; #else - *resp = (struct DCP_FW_NAME(neo_dcp_map_buf_resp)) { .dva = memdesc->dva }; + *resp = (struct DCP_FW_NAME(dcp_map_buf_resp)) { .dva = memdesc->dva }; #endif return true; reject: - dev_err(neo_dcp->dev, "denying map of invalid buffer %llx for piodma\n", + dev_err(dcp->dev, "denying map of invalid buffer %llx for piodma\n", req->buffer); return false; } -static struct DCP_FW_NAME(neo_dcp_map_buf_resp) dcpep_cb_map_piodma(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_map_buf_req *req) +static struct DCP_FW_NAME(dcp_map_buf_resp) dcpep_cb_map_piodma(struct apple_dcp *dcp, + struct dcp_map_buf_req *req) { - struct DCP_FW_NAME(neo_dcp_map_buf_resp) resp = { 0 }; + struct DCP_FW_NAME(dcp_map_buf_resp) resp = { 0 }; - if (dcpep_map_piodma(neo_dcp, req, &resp)) + if (dcpep_map_piodma(dcp, req, &resp)) return resp; #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) @@ -589,22 +590,22 @@ static struct DCP_FW_NAME(neo_dcp_map_buf_resp) dcpep_cb_map_piodma(struct neo_a return resp; } -static bool dcpep_unmap_piodma(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_unmap_buf_resp *request) +static bool dcpep_unmap_piodma(struct apple_dcp *dcp, + struct dcp_unmap_buf_resp *request) { - struct neo_dcp_mem_descriptor *memdesc; + struct dcp_mem_descriptor *memdesc; size_t size, unmapped; - if (request->buffer >= ARRAY_SIZE(neo_dcp->memdesc)) { - dev_warn(neo_dcp->dev, "unmap request for out of range buffer %llu\n", + if (request->buffer >= ARRAY_SIZE(dcp->memdesc)) { + dev_warn(dcp->dev, "unmap request for out of range buffer %llu\n", request->buffer); return false; } - memdesc = &neo_dcp->memdesc[request->buffer]; + memdesc = &dcp->memdesc[request->buffer]; - if (!test_bit(request->buffer, neo_dcp->memdesc_map) || !memdesc->buf || - !neo_dcp->iommu_dom) + if (!test_bit(request->buffer, dcp->memdesc_map) || !memdesc->buf || + !dcp->iommu_dom) goto not_mapped; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER if (!memdesc->piodma_mapped) @@ -612,7 +613,7 @@ static bool dcpep_unmap_piodma(struct neo_apple_dcp *neo_dcp, #endif if (memdesc->dva != request->dva) { - dev_warn(neo_dcp->dev, + dev_warn(dcp->dev, "unmap buffer %llu address mismatch memdesc.dva:%llx dva:%llx\n", request->buffer, memdesc->dva, request->dva); @@ -621,9 +622,9 @@ static bool dcpep_unmap_piodma(struct neo_apple_dcp *neo_dcp, /* use the piodma iommu domain to unmap from the right IOMMU */ size = ALIGN(memdesc->size, SZ_16K); - unmapped = iommu_unmap(neo_dcp->iommu_dom, memdesc->dva, size); + unmapped = iommu_unmap(dcp->iommu_dom, memdesc->dva, size); if (unmapped != size) { - dev_err(neo_dcp->dev, "iommu_unmap() returned %zu instead of %zu\n", + dev_err(dcp->dev, "iommu_unmap() returned %zu instead of %zu\n", unmapped, size); return false; } @@ -631,16 +632,16 @@ static bool dcpep_unmap_piodma(struct neo_apple_dcp *neo_dcp, return true; not_mapped: - dev_warn(neo_dcp->dev, + dev_warn(dcp->dev, "unmap for non-mapped buffer %llu iova:0x%08llx\n", request->buffer, request->dva); return false; } -static void dcpep_cb_unmap_piodma(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_unmap_buf_resp *request) +static void dcpep_cb_unmap_piodma(struct apple_dcp *dcp, + struct dcp_unmap_buf_resp *request) { - dcpep_unmap_piodma(neo_dcp, request); + dcpep_unmap_piodma(dcp, request); } /* @@ -648,12 +649,12 @@ static void dcpep_cb_unmap_piodma(struct neo_apple_dcp *neo_dcp, * physically contiguous, however we should save the sgtable in case the * buffer needs to be later mapped for PIODMA. */ -static struct neo_dcp_allocate_buffer_resp -dcpep_cb_allocate_buffer(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_allocate_buffer_req *req) +static struct dcp_allocate_buffer_resp +dcpep_cb_allocate_buffer(struct apple_dcp *dcp, + struct dcp_allocate_buffer_req *req) { - struct neo_dcp_allocate_buffer_resp resp = { 0 }; - struct neo_dcp_mem_descriptor allocated = { 0 }; + struct dcp_allocate_buffer_resp resp = { 0 }; + struct dcp_mem_descriptor allocated = { 0 }; size_t size, rounded; u32 id; int ret; @@ -667,28 +668,28 @@ dcpep_cb_allocate_buffer(struct neo_apple_dcp *neo_dcp, return resp; size = round_down(rounded, SZ_16K); - id = find_first_zero_bit(neo_dcp->memdesc_map, DCP_MAX_MAPPINGS); + id = find_first_zero_bit(dcp->memdesc_map, DCP_MAX_MAPPINGS); if (id >= DCP_MAX_MAPPINGS) { - dev_warn(neo_dcp->dev, "DCP overflowed mapping table, ignoring\n"); + dev_warn(dcp->dev, "DCP overflowed mapping table, ignoring\n"); return resp; } - allocated.buf = dma_alloc_coherent(neo_dcp->dev, size, &allocated.dva, + allocated.buf = dma_alloc_coherent(dcp->dev, size, &allocated.dva, GFP_KERNEL); if (!allocated.buf) return resp; - ret = dma_get_sgtable(neo_dcp->dev, &allocated.map, allocated.buf, + ret = dma_get_sgtable(dcp->dev, &allocated.map, allocated.buf, allocated.dva, size); if (ret) { - dma_free_coherent(neo_dcp->dev, size, allocated.buf, allocated.dva); + dma_free_coherent(dcp->dev, size, allocated.buf, allocated.dva); return resp; } /* Callbacks are serialized; publish only a fully initialized descriptor. */ - neo_dcp->memdesc[id] = allocated; - set_bit(id, neo_dcp->memdesc_map); + dcp->memdesc[id] = allocated; + set_bit(id, dcp->memdesc_map); resp.mem_desc_id = id; resp.dva_size = allocated.size; resp.dva = allocated.dva; @@ -705,25 +706,25 @@ dcpep_cb_allocate_buffer(struct neo_apple_dcp *neo_dcp, return resp; } -static u8 dcpep_cb_release_mem_desc(struct neo_apple_dcp *neo_dcp, u32 *mem_desc_id) +static u8 dcpep_cb_release_mem_desc(struct apple_dcp *dcp, u32 *mem_desc_id) { - struct neo_dcp_mem_descriptor *memdesc; + struct dcp_mem_descriptor *memdesc; size_t size, unmapped; u32 id = *mem_desc_id; if (id >= DCP_MAX_MAPPINGS) { - dev_warn(neo_dcp->dev, + dev_warn(dcp->dev, "unmap request for out of range mem_desc_id %u", id); return 0; } - if (!test_bit(id, neo_dcp->memdesc_map)) { - dev_warn(neo_dcp->dev, "unmap request for unused mem_desc_id %u\n", + if (!test_bit(id, dcp->memdesc_map)) { + dev_warn(dcp->dev, "unmap request for unused mem_desc_id %u\n", id); return 0; } - memdesc = &neo_dcp->memdesc[id]; + memdesc = &dcp->memdesc[id]; size = ALIGN(memdesc->size, SZ_16K); /* @@ -735,30 +736,30 @@ static u8 dcpep_cb_release_mem_desc(struct neo_apple_dcp *neo_dcp, u32 *mem_desc * sets piodma_mapped, so older firmware is unaffected. */ if (memdesc->piodma_mapped) { - dev_warn(neo_dcp->dev, + dev_warn(dcp->dev, "releasing buffer %u while still mapped for piodma; unmapping first\n", id); - if (!neo_dcp->iommu_dom) + if (!dcp->iommu_dom) return 0; - unmapped = iommu_unmap(neo_dcp->iommu_dom, memdesc->dva, size); + unmapped = iommu_unmap(dcp->iommu_dom, memdesc->dva, size); if (unmapped != size) { - dev_err(neo_dcp->dev, + dev_err(dcp->dev, "failed to unmap buffer %u before release: %zu of %zu\n", id, unmapped, size); return 0; } memdesc->piodma_mapped = false; } - clear_bit(id, neo_dcp->memdesc_map); + clear_bit(id, dcp->memdesc_map); if (memdesc->buf) { sg_free_table(&memdesc->map); - dma_free_coherent(neo_dcp->dev, size, memdesc->buf, memdesc->dva); + dma_free_coherent(dcp->dev, size, memdesc->buf, memdesc->dva); memdesc->buf = NULL; memset(&memdesc->map, 0, sizeof(memdesc->map)); } else if (memdesc->size) { /* A register range mapped by map_physical */ - dma_unmap_resource(neo_dcp->dev, memdesc->dva, memdesc->size, + dma_unmap_resource(dcp->dev, memdesc->dva, memdesc->size, DMA_BIDIRECTIONAL, 0); memdesc->reg = 0; } @@ -770,12 +771,12 @@ static u8 dcpep_cb_release_mem_desc(struct neo_apple_dcp *neo_dcp, u32 *mem_desc } /* Validate that the specified region is a display register */ -static bool is_disp_register(struct neo_apple_dcp *neo_dcp, u64 start, u64 end) +static bool is_disp_register(struct apple_dcp *dcp, u64 start, u64 end) { int i; - for (i = 0; i < neo_dcp->nr_disp_registers; ++i) { - struct resource *r = neo_dcp->disp_registers[i]; + for (i = 0; i < dcp->nr_disp_registers; ++i) { + struct resource *r = dcp->disp_registers[i]; if ((start >= r->start) && (end <= r->end)) return true; @@ -790,8 +791,8 @@ static bool is_disp_register(struct neo_apple_dcp *neo_dcp, u64 start, u64 end) * sr_map_device_memory_with_index, so we bounds check against that to guard * safe against malicious coprocessors. */ -static struct neo_dcp_map_physical_resp -dcpep_cb_map_physical(struct neo_apple_dcp *neo_dcp, struct neo_dcp_map_physical_req *req) +static struct dcp_map_physical_resp +dcpep_cb_map_physical(struct apple_dcp *dcp, struct dcp_map_physical_req *req) { u64 size, end; dma_addr_t dva; @@ -800,53 +801,53 @@ dcpep_cb_map_physical(struct neo_apple_dcp *neo_dcp, struct neo_dcp_map_physical /* Both values come from the coprocessor; reject wrapping ranges. */ if (!req->size || check_add_overflow(req->size, 4095ULL, &size) || check_add_overflow(req->paddr, round_down(size, 4096) - 1, &end) || - !is_disp_register(neo_dcp, req->paddr, end)) { - dev_err(neo_dcp->dev, "refusing to map phys address %llx size %llx\n", + !is_disp_register(dcp, req->paddr, end)) { + dev_err(dcp->dev, "refusing to map phys address %llx size %llx\n", req->paddr, req->size); - return (struct neo_dcp_map_physical_resp){}; + return (struct dcp_map_physical_resp){}; } size = round_down(size, 4096); - id = find_first_zero_bit(neo_dcp->memdesc_map, DCP_MAX_MAPPINGS); + id = find_first_zero_bit(dcp->memdesc_map, DCP_MAX_MAPPINGS); if (id >= DCP_MAX_MAPPINGS) { - dev_warn(neo_dcp->dev, "DCP overflowed mapping table, ignoring\n"); - return (struct neo_dcp_map_physical_resp){}; + dev_warn(dcp->dev, "DCP overflowed mapping table, ignoring\n"); + return (struct dcp_map_physical_resp){}; } - dva = dma_map_resource(neo_dcp->dev, req->paddr, size, DMA_BIDIRECTIONAL, 0); - if (dma_mapping_error(neo_dcp->dev, dva)) { - dev_err(neo_dcp->dev, "failed to map phys address %llx size %llx\n", + dva = dma_map_resource(dcp->dev, req->paddr, size, DMA_BIDIRECTIONAL, 0); + if (dma_mapping_error(dcp->dev, dva)) { + dev_err(dcp->dev, "failed to map phys address %llx size %llx\n", req->paddr, size); - return (struct neo_dcp_map_physical_resp){}; + return (struct dcp_map_physical_resp){}; } - set_bit(id, neo_dcp->memdesc_map); - neo_dcp->memdesc[id].size = size; - neo_dcp->memdesc[id].reg = req->paddr; - neo_dcp->memdesc[id].dva = dva; + set_bit(id, dcp->memdesc_map); + dcp->memdesc[id].size = size; + dcp->memdesc[id].reg = req->paddr; + dcp->memdesc[id].dva = dva; - return (struct neo_dcp_map_physical_resp){ + return (struct dcp_map_physical_resp){ .dva_size = size, .mem_desc_id = id, .dva = dva, }; } -static u64 dcpep_cb_get_frequency(struct neo_apple_dcp *neo_dcp) +static u64 dcpep_cb_get_frequency(struct apple_dcp *dcp) { - return clk_get_rate(neo_dcp->clk); + return clk_get_rate(dcp->clk); } -static struct DCP_FW_NAME(neo_dcp_map_reg_resp) dcpep_cb_map_reg(struct neo_apple_dcp *neo_dcp, - struct DCP_FW_NAME(neo_dcp_map_reg_req) *req) +static struct DCP_FW_NAME(dcp_map_reg_resp) dcpep_cb_map_reg(struct apple_dcp *dcp, + struct DCP_FW_NAME(dcp_map_reg_req) *req) { - if (req->index >= neo_dcp->nr_disp_registers) { - dev_warn(neo_dcp->dev, "attempted to read invalid reg index %u\n", + if (req->index >= dcp->nr_disp_registers) { + dev_warn(dcp->dev, "attempted to read invalid reg index %u\n", req->index); - return (struct DCP_FW_NAME(neo_dcp_map_reg_resp)){ .ret = 1 }; + return (struct DCP_FW_NAME(dcp_map_reg_resp)){ .ret = 1 }; } else { - struct resource *rsrc = neo_dcp->disp_registers[req->index]; + struct resource *rsrc = dcp->disp_registers[req->index]; #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* * H17P reaches these apertures physically, not through its @@ -856,12 +857,12 @@ static struct DCP_FW_NAME(neo_dcp_map_reg_resp) dcpep_cb_map_reg(struct neo_appl */ dma_addr_t dva = rsrc->start; #elif DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) - dma_addr_t dva = dma_map_resource(neo_dcp->dev, rsrc->start, resource_size(rsrc), + dma_addr_t dva = dma_map_resource(dcp->dev, rsrc->start, resource_size(rsrc), DMA_BIDIRECTIONAL, 0); WARN_ON(dva == DMA_MAPPING_ERROR); #endif - return (struct DCP_FW_NAME(neo_dcp_map_reg_resp)){ + return (struct DCP_FW_NAME(dcp_map_reg_resp)){ .addr = rsrc->start, .length = resource_size(rsrc), #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) @@ -871,31 +872,31 @@ static struct DCP_FW_NAME(neo_dcp_map_reg_resp) dcpep_cb_map_reg(struct neo_appl } } -static struct neo_dcp_read_edt_data_resp -dcpep_cb_read_edt_data(struct neo_apple_dcp *neo_dcp, struct neo_dcp_read_edt_data_req *req) +static struct dcp_read_edt_data_resp +dcpep_cb_read_edt_data(struct apple_dcp *dcp, struct dcp_read_edt_data_req *req) { /* Observe boot-property requests without inventing firmware timings. */ - if (neo_dcp->fixed_connector_type != DRM_MODE_CONNECTOR_eDP) - dev_info(neo_dcp->dev, - "read_edt_data key=%.*s count=%u default0=%#x ret=0\n", + if (dcp->fixed_connector_type != DRM_MODE_CONNECTOR_eDP) + dev_dbg(dcp->dev, + "read_edt_data key=%.*s count=%u default0=%#x ret=0\n", (int)sizeof(req->key), req->key, req->count, req->value[0]); - return (struct neo_dcp_read_edt_data_resp){ + return (struct dcp_read_edt_data_resp){ .value[0] = req->value[0], .ret = 0, }; } -static void iomfbep_cb_enable_backlight_message_ap_gated(struct neo_apple_dcp *neo_dcp, +static void iomfbep_cb_enable_backlight_message_ap_gated(struct apple_dcp *dcp, u8 *enabled) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER /* H17P carries the level only in presents; ask for one more. */ - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - if (neo_dcp_has_panel(neo_dcp) && neo_dcp_backlight_active(neo_dcp) && - neo_dcp_backlight_resend(neo_dcp)) - schedule_work(&neo_dcp->bl_update_wq); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + if (dcp_has_panel(dcp) && dcp_backlight_active(dcp) && + dcp_backlight_resend(dcp)) + schedule_work(&dcp->bl_update_wq); return; } #endif @@ -904,17 +905,17 @@ static void iomfbep_cb_enable_backlight_message_ap_gated(struct neo_apple_dcp *n * DCP seems to set an invalid iDAC value after coming out of DPMS. * syslog: "[BrightnessLCD.cpp:743][AFK]nitsToDBV: iDAC out of range" */ - neo_dcp->brightness.update = true; + dcp->brightness.update = true; /* Backlight handling is only set up for panels described in the DT. */ - if (neo_dcp_has_panel(neo_dcp)) - schedule_work(&neo_dcp->bl_update_wq); + if (dcp_has_panel(dcp)) + schedule_work(&dcp->bl_update_wq); } /* Chunked data transfer for property dictionaries */ -static u8 dcpep_cb_prop_start(struct neo_apple_dcp *neo_dcp, u32 *length) +static u8 dcpep_cb_prop_start(struct apple_dcp *dcp, u32 *length) { - if (neo_dcp->chunks.data != NULL) { - dev_warn(neo_dcp->dev, "ignoring spurious transfer start\n"); + if (dcp->chunks.data != NULL) { + dev_warn(dcp->dev, "ignoring spurious transfer start\n"); return false; } @@ -925,170 +926,262 @@ static u8 dcpep_cb_prop_start(struct neo_apple_dcp *neo_dcp, u32 *length) * Property blobs are a few KiB, so bound it. */ if (*length == 0 || *length > SZ_1M) { - dev_warn(neo_dcp->dev, "rejecting %u byte property transfer\n", + dev_warn(dcp->dev, "rejecting %u byte property transfer\n", *length); return false; } #endif - neo_dcp->chunks.length = *length; - neo_dcp->chunks.data = kzalloc(*length, GFP_KERNEL); + dcp->chunks.length = *length; + dcp->chunks.av_service_epoch = 0; + dcp->chunks.automatic_service_pending = false; + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL) { + /* Match the receipt's start, not a replacement service at its end. */ + mutex_lock(&dcp->dcpavserv.lock); + dcp->chunks.av_service_epoch = dcp->dcpavserv.epoch; + dcp->chunks.automatic_service_pending = + !dcp->dcpavserv.enabled && !dcp->dcpavserv.service; + dcp_firmware_metadata_begin(dcp); + { + unsigned long flags; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + dcp->chunks.automatic_connection_sequence = dcp->rate_proof.connection_sequence; + dcp->chunks.mode_ticket = dcp_metadata_begin(&dcp->metadata.epoch); + dcp->chunks.mode_ticket.associated = + dcp->metadata.epoch.generation == READ_ONCE(dcp->typec_generation) && + READ_ONCE(dcp->typec_cable_connected); + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + } + mutex_unlock(&dcp->dcpavserv.lock); + } else { + dcp_firmware_metadata_begin(dcp); + } + dcp->chunks.data = kzalloc(*length, GFP_KERNEL); - if (!neo_dcp->chunks.data) { - dev_warn(neo_dcp->dev, "failed to allocate chunks\n"); + if (!dcp->chunks.data) { + dev_warn(dcp->dev, "failed to allocate chunks\n"); return false; } return true; } -static u8 dcpep_cb_prop_chunk(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_set_dcpav_prop_chunk_req *req) +static u8 dcpep_cb_prop_chunk(struct apple_dcp *dcp, + struct dcp_set_dcpav_prop_chunk_req *req) { - if (!neo_dcp->chunks.data) { - dev_warn(neo_dcp->dev, "ignoring spurious chunk\n"); + if (!dcp->chunks.data) { + dev_warn(dcp->dev, "ignoring spurious chunk\n"); return false; } - if (req->offset + req->length > neo_dcp->chunks.length) { - dev_warn(neo_dcp->dev, "ignoring overflowing chunk\n"); + if (req->offset > dcp->chunks.length || req->length > sizeof(req->data) || + req->length > dcp->chunks.length - req->offset) { + dcp->chunks.contiguous = false; + dev_warn(dcp->dev, "ignoring overflowing chunk\n"); return false; } - memcpy(neo_dcp->chunks.data + req->offset, req->data, req->length); + if (req->offset != dcp->chunks.received) + dcp->chunks.contiguous = false; + if (dcp->chunks.contiguous) + dcp->chunks.received += req->length; + memcpy(dcp->chunks.data + req->offset, req->data, req->length); return true; } -static bool dcpep_process_chunks(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_set_dcpav_prop_end_req *req) +static bool dcpep_process_chunks(struct apple_dcp *dcp, + struct dcp_set_dcpav_prop_end_req *req) { - struct neo_dcp_parse_ctx ctx; + struct dcp_parse_ctx ctx; int ret; - if (!neo_dcp->chunks.data) { - dev_warn(neo_dcp->dev, "ignoring spurious end\n"); + if (!dcp->chunks.data) { + dev_warn(dcp->dev, "ignoring spurious end\n"); return false; } + /* Validate serialized extent outside the metadata spinlock. */ + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL) + dcp->chunks.payload_complete = dcp->chunks.contiguous && + dcp_property_complete(dcp->chunks.data, dcp->chunks.received, + dcp->chunks.length); + /* used just as opaque pointer for tracing */ - ctx.neo_dcp = neo_dcp; + ctx.dcp = dcp; - ret = neo_parse(neo_dcp->chunks.data, neo_dcp->chunks.length, &ctx); + ret = parse(dcp->chunks.data, dcp->chunks.length, &ctx); if (ret) { - dev_warn(neo_dcp->dev, "bad header on dcpav props\n"); + dev_warn(dcp->dev, "bad header on dcpav props\n"); + /* A failed replacement cannot keep an unsafe catalog admitted. */ + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && + !strcmp(req->key, "TimingElements")) { + unsigned long flags; + + spin_lock_irqsave(&dcp->metadata.lock, flags); + dcp->rate_proof.catalog_sequence++; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + dcp_180hz_catalog_changed(dcp); + } return false; } if (!strcmp(req->key, "TimingElements")) { - neo_dcp->modes = neo_enumerate_modes(&ctx, &neo_dcp->nr_modes, - neo_dcp->width_mm, neo_dcp->height_mm, - neo_dcp->notch_height, - neo_dcp->fixed_connector_type == - DRM_MODE_CONNECTOR_eDP); - - if (IS_ERR(neo_dcp->modes)) { - dev_warn(neo_dcp->dev, "failed to parse modes\n"); - neo_dcp->modes = NULL; - neo_dcp->nr_modes = 0; + struct dcp_display_mode *modes; + struct dcp_metadata_ticket rate_ticket = {}; + u64 rate_service_epoch = 0; + unsigned int nr_modes = 0, i; + unsigned long flags; + + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && + dcp->chunks.payload_complete && dcp->chunks.mode_ticket.associated) { + ctx.automatic_1080p180_rgb8 = true; + rate_ticket = dcp->chunks.mode_ticket; + rate_service_epoch = dcp->chunks.av_service_epoch; + } + modes = enumerate_modes(&ctx, &nr_modes, + dcp->width_mm, dcp->height_mm, + dcp->notch_height, + dcp->fixed_connector_type == + DRM_MODE_CONNECTOR_eDP, + dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL); + + if (IS_ERR(modes)) { + dev_warn(dcp->dev, "failed to parse modes\n"); + modes = NULL; + nr_modes = 0; + } else { + for (i = 0; i < nr_modes; i++) + if (modes[i].automatic_rgb8_quirk) { + modes[i].unsafe_rgb8_ticket = rate_ticket; + modes[i].unsafe_rgb8_service_epoch = rate_service_epoch; + modes[i].automatic_connection_sequence = dcp->chunks.automatic_connection_sequence; + modes[i].automatic_service_pending = dcp->chunks.automatic_service_pending; + } + } + /* Publish one catalog revision with its immutable receipt tickets. */ + spin_lock_irqsave(&dcp->metadata.lock, flags); + dcp->rate_proof.catalog_sequence++; + for (i = 0; i < nr_modes; i++) + if (modes[i].automatic_rgb8_quirk) + modes[i].automatic_catalog_sequence = dcp->rate_proof.catalog_sequence; + dcp->modes = modes; + dcp->nr_modes = nr_modes; + spin_unlock_irqrestore(&dcp->metadata.lock, flags); + /* Notify removals as well as newly eligible modes, outside the lock. */ + dcp_180hz_catalog_changed(dcp); + if (!modes) { + dcp_firmware_metadata_timing(dcp); return false; } - if (neo_dcp->nr_modes == 0) - dev_warn(neo_dcp->dev, "TimingElements without valid modes!\n"); + if (nr_modes == 0) + dev_warn(dcp->dev, "TimingElements without valid modes!\n"); + dcp_180hz_notify(dcp); + dcp_firmware_metadata_timing(dcp); } else if (!strcmp(req->key, "DisplayAttributes")) { - ret = neo_parse_display_attributes(&ctx, &neo_dcp->width_mm, - &neo_dcp->height_mm); + struct dcp_parse_ctx metadata_ctx = ctx; + + ret = parse_display_attributes_h17p(&ctx, &dcp->width_mm, + &dcp->height_mm); if (ret) { - dev_warn(neo_dcp->dev, "failed to parse display attribs\n"); + dev_warn(dcp->dev, "failed to parse display attribs\n"); return false; } - neo_dcp_set_dimensions(neo_dcp); + dcp_set_dimensions_h17p(dcp); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17P_EXTERNAL) { + struct dcp_display_metadata metadata; + + ret = parse_display_metadata(&metadata_ctx, &metadata); + dcp_firmware_metadata_attributes(dcp, ret ? NULL : &metadata); + } } return true; } -static u8 dcpep_cb_prop_end(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_set_dcpav_prop_end_req *req) +static u8 dcpep_cb_prop_end(struct apple_dcp *dcp, + struct dcp_set_dcpav_prop_end_req *req) { - u8 resp = dcpep_process_chunks(neo_dcp, req); + u8 resp = dcpep_process_chunks(dcp, req); - if (neo_dcp->fixed_connector_type != DRM_MODE_CONNECTOR_eDP) - dev_info(neo_dcp->dev, - "DCP property key=%.*s bytes=%zu accepted=%u nr_modes=%u\n", - (int)sizeof(req->key), req->key, neo_dcp->chunks.length, - resp, neo_dcp->nr_modes); + if (dcp->fixed_connector_type != DRM_MODE_CONNECTOR_eDP) + dev_dbg(dcp->dev, + "DCP property key=%.*s bytes=%zu accepted=%u nr_modes=%u\n", + (int)sizeof(req->key), req->key, dcp->chunks.length, + resp, dcp->nr_modes); /* move chunked data to connector to provide it via debugfs */ - neo_dcp_connector_update_dict(neo_dcp->connector, req->key, &neo_dcp->chunks); - neo_dcp->chunks.data = NULL; - neo_dcp->chunks.length = 0; + dcp_connector_update_dict(dcp->connector, req->key, &dcp->chunks); + dcp->chunks.data = NULL; + dcp->chunks.length = 0; return resp; } /* Boot sequence */ -static void boot_done(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_done(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; u8 *succ = ch->output[ch->depth - 1]; - dev_dbg(neo_dcp->dev, "boot done\n"); + dev_dbg(dcp->dev, "boot done\n"); *succ = true; - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } -static void boot_5b(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_5b(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_set_display_refresh_properties(neo_dcp, false, boot_done, NULL); + dcp_set_display_refresh_properties(dcp, false, boot_done, NULL); } -static void boot_5(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_5(struct apple_dcp *dcp, void *out, void *cookie) { #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* H17P expects A380 right before set_display_refresh_properties */ - neo_dcp_dfb_ready_query(neo_dcp, false, boot_5b, NULL); + dcp_dfb_ready_query(dcp, false, boot_5b, NULL); #else - boot_5b(neo_dcp, out, cookie); + boot_5b(dcp, out, cookie); #endif } -static void boot_4b(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_4b(struct apple_dcp *dcp, void *out, void *cookie) { #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - u32 value = neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G; + u32 value = dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G; #else u32 value = 1; #endif - neo_dcp_late_init_signal(neo_dcp, false, &value, boot_5, NULL); + dcp_late_init_signal(dcp, false, &value, boot_5, NULL); #else - neo_dcp_late_init_signal(neo_dcp, false, boot_5, NULL); + dcp_late_init_signal(dcp, false, boot_5, NULL); #endif } -static void boot_4(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_4(struct apple_dcp *dcp, void *out, void *cookie) { #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* H17P expects A033 between flush_supports_power and late_init_signal */ - neo_dcp_dfb_query(neo_dcp, false, boot_4b, NULL); + dcp_dfb_query(dcp, false, boot_4b, NULL); #else - boot_4b(neo_dcp, out, cookie); + boot_4b(dcp, out, cookie); #endif } -static void boot_3(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_3(struct apple_dcp *dcp, void *out, void *cookie) { u32 v_true = true; - neo_dcp_flush_supports_power(neo_dcp, false, &v_true, boot_4, NULL); + dcp_flush_supports_power(dcp, false, &v_true, boot_4, NULL); } -static void boot_2(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_2(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_setup_video_limits(neo_dcp, false, boot_3, NULL); + dcp_setup_video_limits(dcp, false, boot_3, NULL); } #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) @@ -1099,14 +1192,14 @@ static void boot_2(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) * get_dfb_state -> set_dfb_dimensions(w, h) -> commit_dfb_info * Without it the firmware never finishes its own default framebuffer setup. */ -static void boot_1_11(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_11(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_commit_dfb_info(neo_dcp, false, boot_2, NULL); + dcp_commit_dfb_info(dcp, false, boot_2, NULL); } -static void boot_1_10(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_10(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_dfb_dimensions_h17p dim; + struct dcp_dfb_dimensions_h17p dim; /* * get_dfb_state returns the default framebuffer dimensions, which are @@ -1114,116 +1207,116 @@ static void boot_1_10(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) */ memcpy(&dim, out, sizeof(dim)); if (!dim.width || !dim.height || dim.width > 16384 || dim.height > 16384) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; - dev_err(neo_dcp->dev, "invalid default framebuffer dimensions %ux%u\n", + dev_err(dcp->dev, "invalid default framebuffer dimensions %ux%u\n", dim.width, dim.height); *(u8 *)ch->output[ch->depth - 1] = false; - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); return; } - dev_dbg(neo_dcp->dev, "default framebuffer dimensions %ux%u\n", + dev_dbg(dcp->dev, "default framebuffer dimensions %ux%u\n", dim.width, dim.height); - neo_dcp_set_dfb_dimensions(neo_dcp, false, &dim, boot_1_11, NULL); + dcp_set_dfb_dimensions(dcp, false, &dim, boot_1_11, NULL); } -static void boot_1_9(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_9(struct apple_dcp *dcp, void *out, void *cookie) { u32 zero = 0; - neo_dcp_get_dfb_state(neo_dcp, false, &zero, boot_1_10, NULL); + dcp_get_dfb_state(dcp, false, &zero, boot_1_10, NULL); } -static void boot_1_8(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_8(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_dfb_layout_req_h17p req = {}; + struct dcp_dfb_layout_req_h17p req = {}; - neo_dcp_get_dfb_layout(neo_dcp, false, &req, boot_1_9, NULL); + dcp_get_dfb_layout(dcp, false, &req, boot_1_9, NULL); } -static void boot_1_7(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_7(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_dfb_info_h17p info = {}; + struct dcp_dfb_info_h17p info = {}; - neo_dcp_get_dfb_info(neo_dcp, false, &info, boot_1_8, NULL); + dcp_get_dfb_info(dcp, false, &info, boot_1_8, NULL); } -static void boot_1_6(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_6(struct apple_dcp *dcp, void *out, void *cookie) { u64 zero = 0; - neo_dcp_get_dfb_compression_info(neo_dcp, false, &zero, boot_1_7, NULL); + dcp_get_dfb_compression_info(dcp, false, &zero, boot_1_7, NULL); } #endif -static void boot_1_5(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void boot_1_5(struct apple_dcp *dcp, void *out, void *cookie) { #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) - neo_dcp_create_default_fb(neo_dcp, false, boot_1_6, NULL); + dcp_create_default_fb(dcp, false, boot_1_6, NULL); #else - neo_dcp_create_default_fb(neo_dcp, false, boot_2, NULL); + dcp_create_default_fb(dcp, false, boot_2, NULL); #endif } #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) -static void d000_done(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void d000_done(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_channel *ch = &neo_dcp->ch_cb; + struct dcp_channel *ch = &dcp->ch_cb; u8 *succ = ch->output[ch->depth - 1]; *succ = true; - neo_dcp_ack(neo_dcp, DCP_CONTEXT_CB); + dcp_ack(dcp, DCP_CONTEXT_CB); } /* * The firmware expects D000 to be answered only after a nested A033 * (dfb_query), and then with 1. Defer the ACK until that call completes. */ -static bool dcpep_cb_d000_h17p(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) +static bool dcpep_cb_d000_h17p(struct apple_dcp *dcp, int tag, void *out, void *in) { - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - neo_dcp_dfb_query(neo_dcp, false, d000_done, NULL); + trace_iomfb_callback(dcp, tag, __func__); + dcp_dfb_query(dcp, false, d000_done, NULL); return false; } #endif /* Use special function signature to defer the ACK */ -static bool dcpep_cb_boot_1(struct neo_apple_dcp *neo_dcp, int tag, void *out, void *in) +static bool dcpep_cb_boot_1(struct apple_dcp *dcp, int tag, void *out, void *in) { - trace_neo_iomfb_callback(neo_dcp, tag, __func__); - neo_dcp_set_create_dfb(neo_dcp, false, boot_1_5, NULL); + trace_iomfb_callback(dcp, tag, __func__); + dcp_set_create_dfb(dcp, false, boot_1_5, NULL); return false; } -static struct neo_dcp_allocate_bandwidth_resp dcpep_cb_allocate_bandwidth(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_allocate_bandwidth_req *req) +static struct dcp_allocate_bandwidth_resp dcpep_cb_allocate_bandwidth(struct apple_dcp *dcp, + struct dcp_allocate_bandwidth_req *req) { - return (struct neo_dcp_allocate_bandwidth_resp){ + return (struct dcp_allocate_bandwidth_resp){ .unk1 = req->unk1, .unk2 = req->unk2, .ret = 1, }; } -static struct neo_dcp_rt_bandwidth dcpep_cb_rt_bandwidth(struct neo_apple_dcp *neo_dcp) +static struct dcp_rt_bandwidth dcpep_cb_rt_bandwidth(struct apple_dcp *dcp) { - struct neo_dcp_rt_bandwidth rt_bw = (struct neo_dcp_rt_bandwidth){ + struct dcp_rt_bandwidth rt_bw = (struct dcp_rt_bandwidth){ .reg_scratch = 0, .reg_doorbell = 0, .doorbell_bit = 0, }; - if (neo_dcp->disp_bw_scratch_index) { - u32 offset = neo_dcp->disp_bw_scratch_offset; - u32 index = neo_dcp->disp_bw_scratch_index; - rt_bw.reg_scratch = neo_dcp->disp_registers[index]->start + offset; + if (dcp->disp_bw_scratch_index) { + u32 offset = dcp->disp_bw_scratch_offset; + u32 index = dcp->disp_bw_scratch_index; + rt_bw.reg_scratch = dcp->disp_registers[index]->start + offset; } - if (neo_dcp->disp_bw_doorbell_index) { - u32 index = neo_dcp->disp_bw_doorbell_index; - rt_bw.reg_doorbell = neo_dcp->disp_registers[index]->start; - rt_bw.doorbell_bit = REG_DOORBELL_BIT(neo_dcp->index); + if (dcp->disp_bw_doorbell_index) { + u32 index = dcp->disp_bw_doorbell_index; + rt_bw.reg_doorbell = dcp->disp_registers[index]->start; + rt_bw.doorbell_bit = REG_DOORBELL_BIT(dcp->index); /* * This is most certainly not padding. t8103-dcp crashes without * setting this immediately during modeset on 12.3 and 13.5 @@ -1249,19 +1342,19 @@ static struct neo_dcp_rt_bandwidth dcpep_cb_rt_bandwidth(struct neo_apple_dcp *n */ #define DCP_FRAME_SYNC_IDENTITY 0x10000 -struct neo_dcp_set_frame_sync_props_req_h17p { +struct dcp_set_frame_sync_props_req_h17p { u8 unk[0x54]; } __packed; -struct neo_dcp_set_frame_sync_props_resp_h17p { +struct dcp_set_frame_sync_props_resp_h17p { u8 unk[0x50]; } __packed; -static struct neo_dcp_set_frame_sync_props_resp_h17p -dcpep_cb_set_frame_sync_props_h17p(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_set_frame_sync_props_req_h17p *req) +static struct dcp_set_frame_sync_props_resp_h17p +dcpep_cb_set_frame_sync_props_h17p(struct apple_dcp *dcp, + struct dcp_set_frame_sync_props_req_h17p *req) { - struct neo_dcp_set_frame_sync_props_resp_h17p resp = {}; + struct dcp_set_frame_sync_props_resp_h17p resp = {}; static const u16 identity_at[] = { 0x24, 0x28, 0x2c }; unsigned int i; @@ -1275,20 +1368,20 @@ dcpep_cb_set_frame_sync_props_h17p(struct neo_apple_dcp *neo_dcp, } #endif -static struct neo_dcp_set_frame_sync_props_resp -dcpep_cb_set_frame_sync_props(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_set_frame_sync_props_req *req) +static struct dcp_set_frame_sync_props_resp +dcpep_cb_set_frame_sync_props(struct apple_dcp *dcp, + struct dcp_set_frame_sync_props_req *req) { - return (struct neo_dcp_set_frame_sync_props_resp){}; + return (struct dcp_set_frame_sync_props_resp){}; } /* Callback to get the current time as milliseconds since the UNIX epoch */ -static u64 dcpep_cb_get_time(struct neo_apple_dcp *neo_dcp) +static u64 dcpep_cb_get_time(struct apple_dcp *dcp) { return ktime_to_ms(ktime_get_real()); } -struct neo_dcp_swap_cookie { +struct dcp_swap_cookie { struct kref refcount; struct completion done; u32 status; @@ -1297,8 +1390,8 @@ struct neo_dcp_swap_cookie { static void release_swap_cookie(struct kref *ref) { - struct neo_dcp_swap_cookie *cookie; - cookie = container_of(ref, struct neo_dcp_swap_cookie, refcount); + struct dcp_swap_cookie *cookie; + cookie = container_of(ref, struct dcp_swap_cookie, refcount); kfree(cookie); } @@ -1314,12 +1407,12 @@ static void release_swap_cookie(struct kref *ref) * format two bytes late and its surface check rejects the swap as an * unsupported format. * - * H17G retains this public packing path. The measured H17P profile uses + * H17G retains this public packing path. The H17P profile uses * a separate wire record with the SPEC's 0x588 swap boundary. */ -static void neo_dcp_h17p_prepare_swap(struct neo_apple_dcp *neo_dcp) +static void dcp_h17p_prepare_swap(struct apple_dcp *dcp) { - struct DCP_FW_NAME(neo_dcp_swap_submit_req) *req = &DCP_FW_UNION(neo_dcp->swap); + struct DCP_FW_NAME(dcp_swap_submit_req) *req = &DCP_FW_UNION(dcp->swap); unsigned int i; /* @@ -1337,8 +1430,8 @@ static void neo_dcp_h17p_prepare_swap(struct neo_apple_dcp *neo_dcp) req->swap.h17p_pre_swap_id[0x00] = 2; req->swap.h17p_pre_swap_id[0x28] = 3; - /* The measured H17P layout is serialized without shifting its surfaces. */ - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + /* The H17P layout is serialized without shifting its surfaces. */ + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) return; for (i = 0; i < SWAP_SURFACES; i++) { @@ -1370,33 +1463,33 @@ static void neo_dcp_h17p_prepare_swap(struct neo_apple_dcp *neo_dcp) } } #else -static void neo_dcp_h17p_prepare_swap(struct neo_apple_dcp *neo_dcp) { } +static void dcp_h17p_prepare_swap(struct apple_dcp *dcp) { } #endif -static bool neo_dcp_present_retires(struct neo_apple_dcp *neo_dcp) +static bool dcp_present_retires(struct apple_dcp *dcp) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - return !neo_iomfb_present_brightness_only_h17p(neo_dcp); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + return !iomfb_present_brightness_only_h17p(dcp); #endif return true; } -static void neo_dcp_present_failed(struct neo_apple_dcp *neo_dcp) +static void dcp_present_failed(struct apple_dcp *dcp) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - neo_iomfb_present_failed_h17p(neo_dcp); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + iomfb_present_failed_h17p(dcp); #endif } -static bool neo_dcp_present_begin(struct neo_apple_dcp *neo_dcp, u32 swap_id) +static bool dcp_present_begin(struct apple_dcp *dcp, u32 swap_id) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !neo_dcp_present_begin_h17p(&neo_dcp->present_state_h17p, swap_id)) { - dev_err(neo_dcp->dev, "overlapping present %u\n", swap_id); - neo_dcp->crashed = true; + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !dcp_present_begin_h17p(&dcp->present_state_h17p, swap_id)) { + dev_err(dcp->dev, "overlapping present %u\n", swap_id); + dcp->crashed = true; return false; } #endif @@ -1408,19 +1501,19 @@ static bool neo_dcp_present_begin(struct neo_apple_dcp *neo_dcp, u32 swap_id) * aborted it before replying (already reported), or -EPROTO if no present * was expecting this reply. */ -static int neo_dcp_present_submit(struct neo_apple_dcp *neo_dcp, u32 swap_id, bool accepted) +static int dcp_present_submit(struct apple_dcp *dcp, u32 swap_id, bool accepted) { #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - !neo_dcp_present_submit_h17p(&neo_dcp->present_state_h17p, swap_id, accepted)) { - dev_err(neo_dcp->dev, "unexpected present submission %u\n", swap_id); - neo_dcp->crashed = true; + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + !dcp_present_submit_h17p(&dcp->present_state_h17p, swap_id, accepted)) { + dev_err(dcp->dev, "unexpected present submission %u\n", swap_id); + dcp->crashed = true; return -EPROTO; } /* An abort received before this reply turns acceptance into failure. */ - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && - accepted && neo_dcp->present_state_h17p.aborted) { - dev_warn_ratelimited(neo_dcp->dev, "firmware aborted present %u\n", + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + accepted && dcp->present_state_h17p.aborted) { + dev_warn_ratelimited(dcp->dev, "firmware aborted present %u\n", swap_id); return -ECANCELED; } @@ -1428,9 +1521,9 @@ static int neo_dcp_present_submit(struct neo_apple_dcp *neo_dcp, u32 swap_id, bo return 0; } -static void neo_dcp_prepare_clear_swap(struct neo_apple_dcp *neo_dcp, void *request) +static void dcp_prepare_clear_swap(struct apple_dcp *dcp, void *request) { - typeof(DCP_FW_UNION(neo_dcp->swap)) *swap = request; + typeof(DCP_FW_UNION(dcp->swap)) *swap = request; /* Clear surfaces. */ memset(swap, 0, sizeof(*swap)); @@ -1446,7 +1539,7 @@ static void neo_dcp_prepare_clear_swap(struct neo_apple_dcp *neo_dcp, void *requ * subsequent update on poweron an actual change and restore the * brightness. */ - if (neo_dcp_has_panel(neo_dcp)) { + if (dcp_has_panel(dcp)) { swap->swap.bl_unk = 1; swap->swap.bl_value = 0; swap->swap.bl_power = 0; @@ -1465,26 +1558,26 @@ static void neo_dcp_prepare_clear_swap(struct neo_apple_dcp *neo_dcp, void *requ #endif } -static void neo_dcp_swap_cleared(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) +static void dcp_swap_cleared(struct apple_dcp *dcp, void *data, void *cookie) { - struct DCP_FW_NAME(neo_dcp_swap_submit_resp) *resp = data; - struct neo_dcp_swap_cookie *info = cookie; - u32 swap_id = DCP_FW_UNION(neo_dcp->swap).swap.swap_id; + struct DCP_FW_NAME(dcp_swap_submit_resp) *resp = data; + struct dcp_swap_cookie *info = cookie; + u32 swap_id = DCP_FW_UNION(dcp->swap).swap.swap_id; u32 status = resp ? resp->ret : ~0U; int ret; - ret = neo_dcp_present_submit(neo_dcp, swap_id, !status); + ret = dcp_present_submit(dcp, swap_id, !status); if (ret) status = ~0U; if (status) { /* An aborted present has already been reported. */ if (ret != -ECANCELED) - dev_err(neo_dcp->dev, "swap_clear failed! status %u\n", status); - neo_dcp_present_failed(neo_dcp); - if (neo_dcp_present_retires(neo_dcp)) { - neo_dcp_unarm_retained_framebuffers(neo_dcp, swap_id); - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dev_err(dcp->dev, "swap_clear failed! status %u\n", status); + dcp_present_failed(dcp); + if (dcp_present_retires(dcp)) { + dcp_unarm_retained_framebuffers(dcp, swap_id); + dcp_drm_crtc_vblank(dcp->crtc); } } if (info) { @@ -1494,11 +1587,11 @@ static void neo_dcp_swap_cleared(struct neo_apple_dcp *neo_dcp, void *data, void } } -static void neo_dcp_swap_clear_started(struct neo_apple_dcp *neo_dcp, void *data, +static void dcp_swap_clear_started(struct apple_dcp *dcp, void *data, void *cookie) { - struct DCP_FW_NAME(neo_dcp_swap_start_resp) *resp = data; - struct neo_dcp_swap_cookie *info = cookie; + struct DCP_FW_NAME(dcp_swap_start_resp) *resp = data; + struct dcp_swap_cookie *info = cookie; if (!resp || resp->ret) { if (info) { @@ -1506,41 +1599,41 @@ static void neo_dcp_swap_clear_started(struct neo_apple_dcp *neo_dcp, void *data complete(&info->done); kref_put(&info->refcount, release_swap_cookie); } - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dcp_drm_crtc_vblank(dcp->crtc); return; } - if (!neo_dcp_present_begin(neo_dcp, resp->swap_id)) { + if (!dcp_present_begin(dcp, resp->swap_id)) { if (info) { WRITE_ONCE(info->status, ~0U); complete(&info->done); kref_put(&info->refcount, release_swap_cookie); } - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dcp_drm_crtc_vblank(dcp->crtc); return; } #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - neo_dcp_prepare_clear_swap(neo_dcp, &DCP_FW_UNION(neo_dcp->swap)); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + dcp_prepare_clear_swap(dcp, &DCP_FW_UNION(dcp->swap)); #endif - DCP_FW_UNION(neo_dcp->swap).swap.swap_id = resp->swap_id; - if (neo_dcp_present_retires(neo_dcp)) - neo_dcp_arm_retained_framebuffers(neo_dcp, resp->swap_id); + DCP_FW_UNION(dcp->swap).swap.swap_id = resp->swap_id; + if (dcp_present_retires(dcp)) + dcp_arm_retained_framebuffers(dcp, resp->swap_id); if (info) info->swap_id = resp->swap_id; - neo_dcp_h17p_prepare_swap(neo_dcp); - neo_dcp_swap_submit(neo_dcp, false, &DCP_FW_UNION(neo_dcp->swap), neo_dcp_swap_cleared, cookie); + dcp_h17p_prepare_swap(dcp); + dcp_swap_submit(dcp, false, &DCP_FW_UNION(dcp->swap), dcp_swap_cleared, cookie); } -static void neo_dcp_on_final(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void dcp_on_final(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_wait_cookie *wait = cookie; + struct dcp_wait_cookie *wait = cookie; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G && + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G && out && get_unaligned_le64(out) == 1) - WRITE_ONCE(neo_dcp->pipe_enabled_h17p, true); + WRITE_ONCE(dcp->pipe_enabled_h17p, true); #endif if (wait) { @@ -1549,18 +1642,18 @@ static void neo_dcp_on_final(struct neo_apple_dcp *neo_dcp, void *out, void *coo } } -static void neo_dcp_on_set_power_state(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void dcp_on_set_power_state(struct apple_dcp *dcp, void *out, void *cookie) { - struct DCP_FW_NAME(neo_dcp_set_power_state_req) req = { + struct DCP_FW_NAME(dcp_set_power_state_req) req = { .unklong = 1, }; - neo_dcp_set_power_state(neo_dcp, DCP_POWER_OOB, &req, neo_dcp_on_final, cookie); + dcp_set_power_state(dcp, DCP_POWER_OOB, &req, dcp_on_final, cookie); } -static void neo_dcp_on_set_parameter(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void dcp_on_set_parameter(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_set_parameter_dcp param = { + struct dcp_set_parameter_dcp param = { .param = IOMFBPARAM_ADAPTIVE_SYNC, .value = { 0 }, #if DCP_FW_VER >= DCP_FW_VERSION(13, 2, 0) @@ -1570,19 +1663,19 @@ static void neo_dcp_on_set_parameter(struct neo_apple_dcp *neo_dcp, void *out, v #endif }; - neo_dcp_set_parameter_dcp(neo_dcp, false, ¶m, neo_dcp_on_set_power_state, cookie); + dcp_set_parameter_dcp(dcp, false, ¶m, dcp_on_set_power_state, cookie); } -void DCP_FW_NAME(neo_iomfb_poweron)(struct neo_apple_dcp *neo_dcp) +void DCP_FW_NAME(iomfb_poweron)(struct apple_dcp *dcp) { - struct neo_dcp_wait_cookie *cookie; + struct dcp_wait_cookie *cookie; int ret; u32 handle; - dev_info(neo_dcp->dev, "dcp_poweron() starting\n"); + dev_info(dcp->dev, "dcp_poweron() starting\n"); #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - neo_iomfb_opaque_x_reset_h17p(neo_dcp); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + iomfb_opaque_x_reset_h17p(dcp); #endif cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); @@ -1595,7 +1688,7 @@ void DCP_FW_NAME(neo_iomfb_poweron)(struct neo_apple_dcp *neo_dcp) /* increase refcount to ensure the receiver has a reference */ kref_get(&cookie->refcount); - if (neo_dcp->main_display) { + if (dcp->main_display) { #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* * H17P takes set_display_device (A411) once, in the @@ -1603,44 +1696,44 @@ void DCP_FW_NAME(neo_iomfb_poweron)(struct neo_apple_dcp *neo_dcp) * is a call the firmware stops answering, which wedges the * command channel and with it every later swap. */ - neo_dcp_on_set_power_state(neo_dcp, NULL, cookie); + dcp_on_set_power_state(dcp, NULL, cookie); #else handle = 0; - neo_dcp_set_display_device(neo_dcp, false, &handle, neo_dcp_on_set_power_state, + dcp_set_display_device(dcp, false, &handle, dcp_on_set_power_state, cookie); #endif } else { handle = 2; - neo_dcp_set_display_device(neo_dcp, false, &handle, - neo_dcp_on_set_parameter, cookie); + dcp_set_display_device(dcp, false, &handle, + dcp_on_set_parameter, cookie); } ret = wait_for_completion_timeout(&cookie->done, msecs_to_jiffies(10000)); if (ret == 0) { - dev_warn(neo_dcp->dev, "wait for power timed out, connector will be broken\n"); + dev_warn(dcp->dev, "wait for power timed out, connector will be broken\n"); } else if (ret > 0) { int msecs = jiffies_to_msecs(ret); if (msecs > 6000) - dev_info(neo_dcp->dev, "dcp_set_power_state_req returned, %d ms remaining\n", msecs); + dev_info(dcp->dev, "dcp_set_power_state_req returned, %d ms remaining\n", msecs); else - dev_warn(neo_dcp->dev, "dcp_set_power_state_req returned, %d ms remaining\n", msecs); + dev_warn(dcp->dev, "dcp_set_power_state_req returned, %d ms remaining\n", msecs); } else { - drm_connector_set_link_status_property(&neo_dcp->connector->base, + drm_connector_set_link_status_property(&dcp->connector->base, DRM_MODE_LINK_STATUS_BAD); - dev_warn(neo_dcp->dev, "wait for completion error: %d\n", ret); + dev_warn(dcp->dev, "wait for completion error: %d\n", ret); } kref_put(&cookie->refcount, release_wait_cookie);; /* Force a brightness update after poweron, to restore the brightness */ - neo_dcp->brightness.update = true; + dcp->brightness.update = true; } -static void complete_set_powerstate(struct neo_apple_dcp *neo_dcp, void *out, +static void complete_set_powerstate(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_dcp_set_power_state_resp *resp = out; - struct neo_dcp_wait_cookie *wait = cookie; + struct dcp_set_power_state_resp *resp = out; + struct dcp_wait_cookie *wait = cookie; if (wait) { WRITE_ONCE(wait->status, resp ? resp->ret : ~0U); @@ -1649,9 +1742,9 @@ static void complete_set_powerstate(struct neo_apple_dcp *neo_dcp, void *out, } } -static bool neo_dcp_power_stop_confirmed(struct neo_dcp_wait_cookie *cookie) +static bool dcp_power_stop_confirmed(struct dcp_wait_cookie *cookie) { - /* H17P stop semantics still require hardware qualification. */ + /* H17P does not confirm that scanout has stopped. */ #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER return false; #else @@ -1659,38 +1752,38 @@ static bool neo_dcp_power_stop_confirmed(struct neo_dcp_wait_cookie *cookie) #endif } -static void last_client_closed_poff(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void last_client_closed_poff(struct apple_dcp *dcp, void *out, void *cookie) { - struct DCP_FW_NAME(neo_dcp_set_power_state_req) power_req = { + struct DCP_FW_NAME(dcp_set_power_state_req) power_req = { .unklong = 0, }; - neo_dcp_set_power_state(neo_dcp, DCP_POWER_OOB, &power_req, complete_set_powerstate, + dcp_set_power_state(dcp, DCP_POWER_OOB, &power_req, complete_set_powerstate, cookie); } -static void aborted_swaps_dcp_poff(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void aborted_swaps_dcp_poff(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_iomfb_last_client_close_req last_client_req = {}; - neo_iomfb_last_client_close(neo_dcp, false, &last_client_req, + struct iomfb_last_client_close_req last_client_req = {}; + iomfb_last_client_close(dcp, false, &last_client_req, last_client_closed_poff, cookie); } -void DCP_FW_NAME(neo_iomfb_poweroff)(struct neo_apple_dcp *neo_dcp) +void DCP_FW_NAME(iomfb_poweroff)(struct apple_dcp *dcp) { int ret, swap_id; - struct neo_iomfb_abort_swaps_dcp_req abort_req = { + struct iomfb_abort_swaps_dcp_req abort_req = { .client = { .flag2 = 1, }, }; - struct neo_dcp_swap_cookie *cookie; - struct neo_dcp_wait_cookie *poff_cookie; - struct DCP_FW_NAME(neo_dcp_swap_start_req) swap_req = { 0 }; - typeof(DCP_FW_UNION(neo_dcp->swap)) *swap = &DCP_FW_UNION(neo_dcp->swap); + struct dcp_swap_cookie *cookie; + struct dcp_wait_cookie *poff_cookie; + struct DCP_FW_NAME(dcp_swap_start_req) swap_req = { 0 }; + typeof(DCP_FW_UNION(dcp->swap)) *swap = &DCP_FW_UNION(dcp->swap); #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - /* The measured profile has no safe pipe-off or surface-free transition. */ - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + /* H17P has no known safe pipe-off or surface-free transition. */ + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) return; #endif @@ -1704,13 +1797,13 @@ void DCP_FW_NAME(neo_iomfb_poweroff)(struct neo_apple_dcp *neo_dcp) kref_get(&cookie->refcount); #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G) - neo_dcp_prepare_clear_swap(neo_dcp, swap); + if (dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G) + dcp_prepare_clear_swap(dcp, swap); #else - neo_dcp_prepare_clear_swap(neo_dcp, swap); + dcp_prepare_clear_swap(dcp, swap); #endif - neo_dcp_swap_start(neo_dcp, false, &swap_req, neo_dcp_swap_clear_started, cookie); + dcp_swap_start(dcp, false, &swap_req, dcp_swap_clear_started, cookie); /* * On unplug the firmware powers the external pipe down on its own and @@ -1725,9 +1818,9 @@ void DCP_FW_NAME(neo_iomfb_poweroff)(struct neo_apple_dcp *neo_dcp) ret = 0; kref_put(&cookie->refcount, release_swap_cookie); if (ret <= 0) - dev_warn(neo_dcp->dev, "%s: clear swap timed out\n", __func__); + dev_warn(dcp->dev, "%s: clear swap timed out\n", __func__); else - dev_dbg(neo_dcp->dev, "%s: clear swap submitted: %u\n", __func__, + dev_dbg(dcp->dev, "%s: clear swap submitted: %u\n", __func__, swap_id); poff_cookie = kzalloc(sizeof(*poff_cookie), GFP_KERNEL); @@ -1740,60 +1833,60 @@ void DCP_FW_NAME(neo_iomfb_poweroff)(struct neo_apple_dcp *neo_dcp) kref_get(&poff_cookie->refcount); if (DCP_HAS_CLIENT_TEARDOWN) { - neo_iomfb_abort_swaps_dcp(neo_dcp, false, &abort_req, + iomfb_abort_swaps_dcp(dcp, false, &abort_req, aborted_swaps_dcp_poff, poff_cookie); } else { - dev_dbg_once(neo_dcp->dev, + dev_dbg_once(dcp->dev, "power-off: skipping abort_swaps/last_client_close\n"); - last_client_closed_poff(neo_dcp, NULL, poff_cookie); + last_client_closed_poff(dcp, NULL, poff_cookie); } ret = wait_for_completion_timeout(&poff_cookie->done, msecs_to_jiffies(1000)); if (ret == 0) - dev_warn(neo_dcp->dev, "setPowerState(0) timeout %u ms\n", 1000); + dev_warn(dcp->dev, "setPowerState(0) timeout %u ms\n", 1000); else if (ret > 0) { - dev_dbg(neo_dcp->dev, + dev_dbg(dcp->dev, "setPowerState(0) finished with %d ms to spare", jiffies_to_msecs(ret)); - if (neo_dcp_power_stop_confirmed(poff_cookie)) - neo_dcp_release_all_retained_framebuffers(neo_dcp); + if (dcp_power_stop_confirmed(poff_cookie)) + dcp_release_all_retained_framebuffers(dcp); else - dev_warn(neo_dcp->dev, "scanout stop was not confirmed\n"); + dev_warn(dcp->dev, "scanout stop was not confirmed\n"); } kref_put(&poff_cookie->refcount, release_wait_cookie); - dev_info(neo_dcp->dev, "dcp_poweroff() done\n"); + dev_info(dcp->dev, "dcp_poweroff() done\n"); } -static void last_client_closed_sleep(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void last_client_closed_sleep(struct apple_dcp *dcp, void *out, void *cookie) { - struct DCP_FW_NAME(neo_dcp_set_power_state_req) power_req = { + struct DCP_FW_NAME(dcp_set_power_state_req) power_req = { .unklong = 0, }; - neo_dcp_set_power_state(neo_dcp, DCP_POWER_OOB, &power_req, complete_set_powerstate, cookie); + dcp_set_power_state(dcp, DCP_POWER_OOB, &power_req, complete_set_powerstate, cookie); } -static void aborted_swaps_dcp_sleep(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void aborted_swaps_dcp_sleep(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_iomfb_last_client_close_req req = { 0 }; - neo_iomfb_last_client_close(neo_dcp, false, &req, last_client_closed_sleep, cookie); + struct iomfb_last_client_close_req req = { 0 }; + iomfb_last_client_close(dcp, false, &req, last_client_closed_sleep, cookie); } -void DCP_FW_NAME(neo_iomfb_sleep)(struct neo_apple_dcp *neo_dcp) +void DCP_FW_NAME(iomfb_sleep)(struct apple_dcp *dcp) { int ret; - struct neo_iomfb_abort_swaps_dcp_req req = { + struct iomfb_abort_swaps_dcp_req req = { .client = { .flag2 = 1, }, }; - struct neo_dcp_wait_cookie *cookie; + struct dcp_wait_cookie *cookie; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) return; #endif @@ -1807,26 +1900,26 @@ void DCP_FW_NAME(neo_iomfb_sleep)(struct neo_apple_dcp *neo_dcp) kref_get(&cookie->refcount); if (DCP_HAS_CLIENT_TEARDOWN) { - neo_iomfb_abort_swaps_dcp(neo_dcp, false, &req, aborted_swaps_dcp_sleep, + iomfb_abort_swaps_dcp(dcp, false, &req, aborted_swaps_dcp_sleep, cookie); } else { - dev_dbg_once(neo_dcp->dev, + dev_dbg_once(dcp->dev, "sleep: skipping abort_swaps/last_client_close\n"); - last_client_closed_sleep(neo_dcp, NULL, cookie); + last_client_closed_sleep(dcp, NULL, cookie); } ret = wait_for_completion_timeout(&cookie->done, msecs_to_jiffies(1000)); if (ret == 0) - dev_warn(neo_dcp->dev, "setDCPPower(0) timeout %u ms\n", 1000); + dev_warn(dcp->dev, "setDCPPower(0) timeout %u ms\n", 1000); kref_put(&cookie->refcount, release_wait_cookie); - dev_info(neo_dcp->dev, "dcp_sleep() done\n"); + dev_info(dcp->dev, "dcp_sleep() done\n"); } -static void dcpep_cb_hotplug(struct neo_apple_dcp *neo_dcp, u64 *connected) +static void dcpep_cb_hotplug(struct apple_dcp *dcp, u64 *connected) { - struct neo_apple_connector *connector = neo_dcp->connector; + struct apple_connector *connector = dcp->connector; unsigned int action; /* DCP issues hotplug_gated callbacks after SetPowerState() calls on @@ -1836,10 +1929,10 @@ static void dcpep_cb_hotplug(struct neo_apple_dcp *neo_dcp, u64 *connected) * is one example. dcp_is_main_display() returns true on devices with * integrated display. Ignore the hotplug_gated() callbacks there. */ - if (neo_dcp->main_display) + if (dcp->main_display) return; /* - * Report firmware hotplug independently of the USB4 PHY experiment. + * Report firmware hotplug independently of the USB4 PHY. * Reassigning lpdptxphy blanked eDP even with these callbacks ignored; * suppressing connector notifications does not protect the panel. * Mode probing still uses this DCP's firmware modes, and mode_valid @@ -1850,37 +1943,41 @@ static void dcpep_cb_hotplug(struct neo_apple_dcp *neo_dcp, u64 *connected) * Same for the unplug a Type-C output reports after its CRTC was * powered off with the cable still attached (see dcp_poweroff()). */ - if (!(*connected) && READ_ONCE(neo_dcp->typec_crtc_off) && - READ_ONCE(neo_dcp->typec_cable_connected)) { - dev_dbg(neo_dcp->dev, "cb_hotplug() ignoring unplug of powered-off Type-C output\n"); - neo_dcp_mode_invalidate(&neo_dcp->mode_state); - schedule_work(&neo_dcp->vblank_wq); + if (!(*connected) && READ_ONCE(dcp->typec_crtc_off) && + READ_ONCE(dcp->typec_cable_connected)) { + dev_dbg(dcp->dev, "cb_hotplug() ignoring unplug of powered-off Type-C output\n"); + dcp_mode_invalidate(&dcp->mode_state); + schedule_work(&dcp->vblank_wq); return; } - if (neo_dcp_is_typec_output(neo_dcp) && *connected && neo_dcp->nr_modes) - complete_all(&neo_dcp->typec_iomfb_hpd_ready); + if (dcp_is_typec_output(dcp) && *connected && dcp->nr_modes) + complete_all(&dcp->typec_iomfb_hpd_ready); - action = neo_dcp_mode_hotplug(&neo_dcp->mode_state, !!(*connected), + if (dcp_external_hotplug(dcp, !!(*connected))) + return; + + dcp_firmware_metadata_hotplug(dcp, !!(*connected)); + action = dcp_mode_hotplug(&dcp->mode_state, !!(*connected), connector ? &connector->connected : NULL); /* * A Type-C sink can assert HPD only after a modeset has already failed, * as a TV behind a DP-to-HDMI converter does when it wakes from standby. * The connector state does not change then; re-apply the mode anyway. */ - if (*connected && neo_dcp_is_typec_output(neo_dcp) && - !READ_ONCE(neo_dcp->mode_state.valid) && - !READ_ONCE(neo_dcp->mode_state.changing)) { - neo_dcp->swap_watchdog_retrains = 0; + if (*connected && dcp_is_typec_output(dcp) && + !READ_ONCE(dcp->mode_state.valid) && + !READ_ONCE(dcp->mode_state.changing)) { + dcp->swap_watchdog_retrains = 0; action |= DCP_HOTPLUG_NOTIFY; } - neo_dcp_handle_hotplug_actions(neo_dcp, action); + dcp_handle_hotplug_actions(dcp, action); } static void -dcpep_cb_swap_complete_intent_gated(struct neo_apple_dcp *neo_dcp, - struct neo_dcp_swap_complete_intent_gated *info) +dcpep_cb_swap_complete_intent_gated(struct apple_dcp *dcp, + struct dcp_swap_complete_intent_gated *info) { - trace_neo_iomfb_swap_complete_intent_gated(neo_dcp, info->swap_id, + trace_iomfb_swap_complete_intent_gated(dcp, info->swap_id, info->width, info->height); } @@ -1890,41 +1987,41 @@ dcpep_cb_swap_complete_intent_gated(struct neo_apple_dcp *neo_dcp, * keep the displaced framebuffers for the next completed present and signal * the DRM event, so neither the queue nor userspace waits for a completion. */ -static void neo_dcp_present_aborted(struct neo_apple_dcp *neo_dcp, u32 swap_id) +static void dcp_present_aborted(struct apple_dcp *dcp, u32 swap_id) { - if (!neo_dcp_present_abort_h17p(&neo_dcp->present_state_h17p, swap_id)) { - dev_warn_ratelimited(neo_dcp->dev, + if (!dcp_present_abort_h17p(&dcp->present_state_h17p, swap_id)) { + dev_warn_ratelimited(dcp->dev, "abort for present %u, which is not in flight\n", swap_id); return; } /* The submit reply has not arrived; it finishes the present. */ - if (neo_dcp->present_state_h17p.pending) + if (dcp->present_state_h17p.pending) return; - dev_warn_ratelimited(neo_dcp->dev, "firmware aborted present %u\n", swap_id); + dev_warn_ratelimited(dcp->dev, "firmware aborted present %u\n", swap_id); /* No completion follows, so the swap watchdog must not wait for one. */ - neo_dcp_swap_watchdog_complete(neo_dcp); - neo_dcp_present_failed(neo_dcp); - if (neo_dcp_present_retires(neo_dcp)) { - neo_dcp_unarm_retained_framebuffers(neo_dcp, swap_id); - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dcp_swap_watchdog_complete(dcp); + dcp_present_failed(dcp); + if (dcp_present_retires(dcp)) { + dcp_unarm_retained_framebuffers(dcp, swap_id); + dcp_drm_crtc_vblank(dcp->crtc); } } #endif static void -dcpep_cb_abort_swap_ap_gated(struct neo_apple_dcp *neo_dcp, u32 *swap_id) +dcpep_cb_abort_swap_ap_gated(struct apple_dcp *dcp, u32 *swap_id) { - trace_neo_iomfb_abort_swap_ap_gated(neo_dcp, *swap_id); + trace_iomfb_abort_swap_ap_gated(dcp, *swap_id); #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) - neo_dcp_present_aborted(neo_dcp, *swap_id); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) + dcp_present_aborted(dcp, *swap_id); #endif } static struct dcpep_get_tiling_state_resp -dcpep_cb_get_tiling_state(struct neo_apple_dcp *neo_dcp, +dcpep_cb_get_tiling_state(struct apple_dcp *dcp, struct dcpep_get_tiling_state_req *req) { return (struct dcpep_get_tiling_state_resp){ @@ -1933,7 +2030,7 @@ dcpep_cb_get_tiling_state(struct neo_apple_dcp *neo_dcp, }; } -static u8 dcpep_cb_create_backlight_service(struct neo_apple_dcp *neo_dcp) +static u8 dcpep_cb_create_backlight_service(struct apple_dcp *dcp) { #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) /* @@ -1943,10 +2040,10 @@ static u8 dcpep_cb_create_backlight_service(struct neo_apple_dcp *neo_dcp) * connector; answering false makes the firmware report that it could * not find the backlight service. */ - return neo_dcp_has_panel(neo_dcp) || - neo_dcp->connector_type == DRM_MODE_CONNECTOR_eDP; + return dcp_has_panel(dcp) || + dcp->connector_type == DRM_MODE_CONNECTOR_eDP; #else - return neo_dcp_has_panel(neo_dcp); + return dcp_has_panel(dcp); #endif } @@ -1957,56 +2054,56 @@ TRAMPOLINE_OUT(trampoline_zero, dcpep_cb_zero, u32); TRAMPOLINE_IN(trampoline_swap_complete, dcpep_cb_swap_complete, struct DCP_FW_NAME(dc_swap_complete_resp)); TRAMPOLINE_INOUT(trampoline_get_uint_prop, dcpep_cb_get_uint_prop, - struct neo_dcp_get_uint_prop_req, struct neo_dcp_get_uint_prop_resp); + struct dcp_get_uint_prop_req, struct dcp_get_uint_prop_resp); TRAMPOLINE_IN(trampoline_set_fx_prop, iomfbep_cb_set_fx_prop, - struct neo_iomfb_set_fx_prop_req) + struct iomfb_set_fx_prop_req) TRAMPOLINE_INOUT(trampoline_map_piodma, dcpep_cb_map_piodma, - struct neo_dcp_map_buf_req, struct DCP_FW_NAME(neo_dcp_map_buf_resp)); + struct dcp_map_buf_req, struct DCP_FW_NAME(dcp_map_buf_resp)); TRAMPOLINE_IN(trampoline_unmap_piodma, dcpep_cb_unmap_piodma, - struct neo_dcp_unmap_buf_resp); + struct dcp_unmap_buf_resp); TRAMPOLINE_INOUT(trampoline_sr_set_property_int, iomfbep_cb_sr_set_property_int, - struct neo_iomfb_sr_set_property_int_req, u8); + struct iomfb_sr_set_property_int_req, u8); TRAMPOLINE_INOUT(trampoline_allocate_buffer, dcpep_cb_allocate_buffer, - struct neo_dcp_allocate_buffer_req, - struct neo_dcp_allocate_buffer_resp); + struct dcp_allocate_buffer_req, + struct dcp_allocate_buffer_resp); TRAMPOLINE_INOUT(trampoline_map_physical, dcpep_cb_map_physical, - struct neo_dcp_map_physical_req, struct neo_dcp_map_physical_resp); + struct dcp_map_physical_req, struct dcp_map_physical_resp); TRAMPOLINE_INOUT(trampoline_release_mem_desc, dcpep_cb_release_mem_desc, u32, u8); TRAMPOLINE_INOUT(trampoline_map_reg, dcpep_cb_map_reg, - struct DCP_FW_NAME(neo_dcp_map_reg_req), - struct DCP_FW_NAME(neo_dcp_map_reg_resp)); + struct DCP_FW_NAME(dcp_map_reg_req), + struct DCP_FW_NAME(dcp_map_reg_resp)); TRAMPOLINE_INOUT(trampoline_read_edt_data, dcpep_cb_read_edt_data, - struct neo_dcp_read_edt_data_req, struct neo_dcp_read_edt_data_resp); + struct dcp_read_edt_data_req, struct dcp_read_edt_data_resp); TRAMPOLINE_INOUT(trampoline_prop_start, dcpep_cb_prop_start, u32, u8); TRAMPOLINE_INOUT(trampoline_prop_chunk, dcpep_cb_prop_chunk, - struct neo_dcp_set_dcpav_prop_chunk_req, u8); + struct dcp_set_dcpav_prop_chunk_req, u8); TRAMPOLINE_INOUT(trampoline_prop_end, dcpep_cb_prop_end, - struct neo_dcp_set_dcpav_prop_end_req, u8); + struct dcp_set_dcpav_prop_end_req, u8); TRAMPOLINE_INOUT(trampoline_allocate_bandwidth, dcpep_cb_allocate_bandwidth, - struct neo_dcp_allocate_bandwidth_req, struct neo_dcp_allocate_bandwidth_resp); + struct dcp_allocate_bandwidth_req, struct dcp_allocate_bandwidth_resp); TRAMPOLINE_OUT(trampoline_rt_bandwidth, dcpep_cb_rt_bandwidth, - struct neo_dcp_rt_bandwidth); + struct dcp_rt_bandwidth); TRAMPOLINE_INOUT(trampoline_set_frame_sync_props, dcpep_cb_set_frame_sync_props, - struct neo_dcp_set_frame_sync_props_req, - struct neo_dcp_set_frame_sync_props_resp); + struct dcp_set_frame_sync_props_req, + struct dcp_set_frame_sync_props_resp); #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) TRAMPOLINE_INOUT(trampoline_set_frame_sync_props_h17p, dcpep_cb_set_frame_sync_props_h17p, - struct neo_dcp_set_frame_sync_props_req_h17p, - struct neo_dcp_set_frame_sync_props_resp_h17p); + struct dcp_set_frame_sync_props_req_h17p, + struct dcp_set_frame_sync_props_resp_h17p); #endif TRAMPOLINE_OUT(trampoline_get_frequency, dcpep_cb_get_frequency, u64); TRAMPOLINE_OUT(trampoline_get_time, dcpep_cb_get_time, u64); TRAMPOLINE_IN(trampoline_hotplug, dcpep_cb_hotplug, u64); TRAMPOLINE_IN(trampoline_swap_complete_intent_gated, dcpep_cb_swap_complete_intent_gated, - struct neo_dcp_swap_complete_intent_gated); + struct dcp_swap_complete_intent_gated); TRAMPOLINE_IN(trampoline_abort_swap_ap_gated, dcpep_cb_abort_swap_ap_gated, u32); TRAMPOLINE_IN(trampoline_enable_backlight_message_ap_gated, iomfbep_cb_enable_backlight_message_ap_gated, u8); -TRAMPOLINE_IN(trampoline_pr_publish, neo_iomfb_cb_pr_publish, - struct neo_iomfb_property); +TRAMPOLINE_IN(trampoline_pr_publish, iomfb_cb_pr_publish, + struct iomfb_property); TRAMPOLINE_INOUT(trampoline_get_tiling_state, dcpep_cb_get_tiling_state, struct dcpep_get_tiling_state_req, struct dcpep_get_tiling_state_resp); TRAMPOLINE_OUT(trampoline_create_backlight_service, dcpep_cb_create_backlight_service, u8); @@ -2016,67 +2113,67 @@ TRAMPOLINE_OUT(trampoline_create_backlight_service, dcpep_cb_create_backlight_se * complete event so we need to fake a vblank event early to avoid a hang. */ -static void neo_dcp_swapped(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) +static void dcp_swapped(struct apple_dcp *dcp, void *data, void *cookie) { int ret; - struct DCP_FW_NAME(neo_dcp_swap_submit_resp) *resp = data; - u32 swap_id = DCP_FW_UNION(neo_dcp->swap).swap.swap_id; + struct DCP_FW_NAME(dcp_swap_submit_resp) *resp = data; + u32 swap_id = DCP_FW_UNION(dcp->swap).swap.swap_id; u32 status = resp ? resp->ret : ~0U; - ret = neo_dcp_present_submit(neo_dcp, swap_id, !status); + ret = dcp_present_submit(dcp, swap_id, !status); if (ret) status = ~0U; if (status) { /* An aborted present has already been reported. */ if (ret != -ECANCELED) - dev_err(neo_dcp->dev, "swap failed! status %u\n", status); - neo_dcp_present_failed(neo_dcp); - if (neo_dcp_present_retires(neo_dcp)) { - neo_dcp_unarm_retained_framebuffers(neo_dcp, swap_id); - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dev_err(dcp->dev, "swap failed! status %u\n", status); + dcp_present_failed(dcp); + if (dcp_present_retires(dcp)) { + dcp_unarm_retained_framebuffers(dcp, swap_id); + dcp_drm_crtc_vblank(dcp->crtc); } return; } - neo_dcp->swap_start = ktime_get(); - neo_dcp->swap_submit_timestamp = arch_timer_read_counter(); - neo_dcp_swap_watchdog_arm(neo_dcp); + dcp->swap_start = ktime_get(); + dcp->swap_submit_timestamp = arch_timer_read_counter(); + dcp_swap_watchdog_arm(dcp); } -static void neo_dcp_swap_started(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) +static void dcp_swap_started(struct apple_dcp *dcp, void *data, void *cookie) { - struct DCP_FW_NAME(neo_dcp_swap_start_resp) *resp = data; + struct DCP_FW_NAME(dcp_swap_start_resp) *resp = data; if (!resp || resp->ret) { - dev_err(neo_dcp->dev, "swap_start was rejected\n"); - if (neo_dcp_present_retires(neo_dcp)) - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dev_err(dcp->dev, "swap_start was rejected\n"); + if (dcp_present_retires(dcp)) + dcp_drm_crtc_vblank(dcp->crtc); return; } - if (!neo_dcp_present_begin(neo_dcp, resp->swap_id)) { - if (neo_dcp_present_retires(neo_dcp)) - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + if (!dcp_present_begin(dcp, resp->swap_id)) { + if (dcp_present_retires(dcp)) + dcp_drm_crtc_vblank(dcp->crtc); return; } - DCP_FW_UNION(neo_dcp->swap).swap.swap_id = resp->swap_id; - if (neo_dcp_present_retires(neo_dcp)) - neo_dcp_arm_retained_framebuffers(neo_dcp, resp->swap_id); + DCP_FW_UNION(dcp->swap).swap.swap_id = resp->swap_id; + if (dcp_present_retires(dcp)) + dcp_arm_retained_framebuffers(dcp, resp->swap_id); - trace_neo_iomfb_swap_submit(neo_dcp, resp->swap_id); - neo_dcp_h17p_prepare_swap(neo_dcp); - neo_dcp_swap_submit(neo_dcp, false, &DCP_FW_UNION(neo_dcp->swap), neo_dcp_swapped, NULL); + trace_iomfb_swap_submit(dcp, resp->swap_id); + dcp_h17p_prepare_swap(dcp); + dcp_swap_submit(dcp, false, &DCP_FW_UNION(dcp->swap), dcp_swapped, NULL); } /* Helpers to modeset and swap, used to flush */ -static void do_swap(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) +static void do_swap(struct apple_dcp *dcp, void *data, void *cookie) { - struct DCP_FW_NAME(neo_dcp_swap_start_req) start_req = { 0 }; + struct DCP_FW_NAME(dcp_swap_start_req) start_req = { 0 }; - if (READ_ONCE(neo_dcp->mode_state.valid) && neo_dcp->connector && - READ_ONCE(neo_dcp->connector->connected)) - neo_dcp_swap_start(neo_dcp, false, &start_req, neo_dcp_swap_started, NULL); + if (READ_ONCE(dcp->mode_state.valid) && dcp->connector && + READ_ONCE(dcp->connector->connected)) + dcp_swap_start(dcp, false, &start_req, dcp_swap_started, NULL); else - neo_dcp_drm_crtc_vblank(neo_dcp->crtc); + dcp_drm_crtc_vblank(dcp->crtc); } #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) @@ -2085,24 +2182,29 @@ static void do_swap(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) * 16.16 fixed point), the same identity value the D006 reply carries. */ #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER -void neo_iomfb_present_backlight_h17p(struct neo_apple_dcp *neo_dcp) +void iomfb_present_backlight_h17p(struct apple_dcp *dcp) { - do_swap(neo_dcp, NULL, NULL); + do_swap(dcp, NULL, NULL); } #endif -static void neo_dcp_pre_swap_a428(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void dcp_post_swap_a428(struct apple_dcp *dcp, void *out, void *cookie) +{ + do_swap(dcp, out, cookie); +} + +static void dcp_pre_swap_a428(struct apple_dcp *dcp, void *out, void *cookie) { u32 identity = 0x10000; - neo_dcp_pipe_set_428(neo_dcp, false, &identity, do_swap, cookie); + dcp_pipe_set_428(dcp, false, &identity, dcp_post_swap_a428, cookie); } #endif -static void complete_set_digital_out_mode(struct neo_apple_dcp *neo_dcp, void *data, +static void complete_set_digital_out_mode(struct apple_dcp *dcp, void *data, void *cookie) { - struct neo_dcp_wait_cookie *wait = cookie; + struct dcp_wait_cookie *wait = cookie; if (wait) { complete(&wait->done); @@ -2111,17 +2213,17 @@ static void complete_set_digital_out_mode(struct neo_apple_dcp *neo_dcp, void *d } /* DCP applies Adaptive Sync changes when the display mode is reselected. */ -static void neo_dcp_on_set_adaptive_sync(struct neo_apple_dcp *neo_dcp, void *out, +static void dcp_on_set_adaptive_sync(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_set_digital_out_mode(neo_dcp, false, &neo_dcp->mode, + dcp_set_digital_out_mode(dcp, false, &dcp->mode, complete_set_digital_out_mode, cookie); } -static void neo_dcp_set_adaptive_sync(struct neo_apple_dcp *neo_dcp, u32 min_vrr, +static void dcp_set_adaptive_sync(struct apple_dcp *dcp, u32 min_vrr, void *cookie) { - struct neo_dcp_set_parameter_dcp param = { + struct dcp_set_parameter_dcp param = { .param = IOMFBPARAM_ADAPTIVE_SYNC, .value = { min_vrr, /* minRR, 16.16 fixed-point Hz */ @@ -2135,26 +2237,26 @@ static void neo_dcp_set_adaptive_sync(struct neo_apple_dcp *neo_dcp, u32 min_vrr #endif }; - neo_dcp_set_parameter_dcp(neo_dcp, false, ¶m, neo_dcp_on_set_adaptive_sync, + dcp_set_parameter_dcp(dcp, false, ¶m, dcp_on_set_adaptive_sync, cookie); } -int DCP_FW_NAME(neo_iomfb_modeset)(struct neo_apple_dcp *neo_dcp, +int DCP_FW_NAME(iomfb_modeset)(struct apple_dcp *dcp, struct drm_crtc_state *crtc_state) { - struct neo_dcp_display_mode *mode; - struct neo_dcp_wait_cookie *cookie; - struct neo_dcp_color_mode *cmode = NULL; + struct dcp_display_mode *mode; + struct dcp_wait_cookie *cookie; + struct dcp_color_mode *cmode = NULL; int ret; - mode = neo_lookup_mode(neo_dcp, &crtc_state->mode); + mode = lookup_mode_h17p(dcp, &crtc_state->mode); if (!mode) { - dev_err(neo_dcp->dev, "no match for " DRM_MODE_FMT "\n", + dev_err(dcp->dev, "no match for " DRM_MODE_FMT "\n", DRM_MODE_ARG(&crtc_state->mode)); return -EIO; } - dev_info(neo_dcp->dev, + dev_info(dcp->dev, "set_digital_out_mode(color:%d timing:%d) " DRM_MODE_FMT "\n", mode->color_mode_id, mode->timing_mode_id, DRM_MODE_ARG(&crtc_state->mode)); @@ -2167,19 +2269,19 @@ int DCP_FW_NAME(neo_iomfb_modeset)(struct neo_apple_dcp *neo_dcp, else if (mode->color_mode_id == mode->best.id) cmode = &mode->best; if (cmode) - dev_info(neo_dcp->dev, + dev_info(dcp->dev, "set_digital_out_mode() color mode depth:%hhu format:%u " "colorimetry:%u eotf:%u range:%u vrr:%u\n", cmode->depth, cmode->format, cmode->colorimetry, cmode->eotf, cmode->range, mode->vrr); - neo_dcp->mode = (struct neo_dcp_set_digital_out_mode_req){ + dcp->mode = (struct dcp_set_digital_out_mode_req){ .color_mode_id = mode->color_mode_id, .timing_mode_id = mode->timing_mode_id }; /* Built-in ProMotion panels require timestamps even in fixed-120 mode. */ - neo_dcp->use_timestamps = mode->vrr && neo_dcp->main_display; + dcp->use_timestamps = mode->vrr && dcp->main_display; cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); if (!cookie) { @@ -2192,17 +2294,21 @@ int DCP_FW_NAME(neo_iomfb_modeset)(struct neo_apple_dcp *neo_dcp, /* increase refcount to ensure the receiver has a reference */ kref_get(&cookie->refcount); - neo_dcp->swap_submit_timestamp = 0; - - if (DCP_INHERIT_BOOT_MODE) { - dev_dbg(neo_dcp->dev, "inheriting the bootloader's display mode\n"); - complete_set_digital_out_mode(neo_dcp, NULL, cookie); - } else if (mode->vrr) - neo_dcp_set_adaptive_sync(neo_dcp, + dcp->swap_submit_timestamp = 0; + + if (DCP_INHERIT_BOOT_MODE && + dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) { + dev_dbg(dcp->dev, "inheriting the bootloader's display mode\n"); + complete_set_digital_out_mode(dcp, NULL, cookie); + } else if (mode->vrr && + dcp->hw.iomfb_method_profile != + DCP_IOMFB_METHODS_H17P_EXTERNAL) + dcp_set_adaptive_sync(dcp, crtc_state->vrr_enabled ? mode->min_vrr : 0, cookie); else - neo_dcp_set_digital_out_mode(neo_dcp, false, &neo_dcp->mode, + dcp_set_digital_out_mode(dcp, false, &dcp->mode, complete_set_digital_out_mode, cookie); /* @@ -2213,34 +2319,34 @@ int DCP_FW_NAME(neo_iomfb_modeset)(struct neo_apple_dcp *neo_dcp, ret = wait_for_completion_timeout(&cookie->done, msecs_to_jiffies(8500)); - dev_info(neo_dcp->dev, "set_digital_out_mode finished:%d\n", ret); + dev_info(dcp->dev, "set_digital_out_mode finished:%d\n", ret); if (ret == 0) { - dev_info(neo_dcp->dev, "set_digital_out_mode timed out\n"); + dev_info(dcp->dev, "set_digital_out_mode timed out\n"); kref_put(&cookie->refcount, release_wait_cookie); return -EIO; } else if (ret < 0) { - dev_info(neo_dcp->dev, + dev_info(dcp->dev, "waiting on set_digital_out_mode failed:%d\n", ret); kref_put(&cookie->refcount, release_wait_cookie); return -EIO; } else { - dev_dbg(neo_dcp->dev, + dev_dbg(dcp->dev, "set_digital_out_mode finished with %d to spare\n", jiffies_to_msecs(ret)); } kref_put(&cookie->refcount, release_wait_cookie); - neo_dcp->vrr_enabled = mode->vrr && crtc_state->vrr_enabled; + dcp->vrr_enabled = mode->vrr && crtc_state->vrr_enabled; return 0; } -void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc *crtc, struct drm_atomic_state *state) +void DCP_FW_NAME(iomfb_flush)(struct apple_dcp *dcp, struct drm_crtc *crtc, struct drm_atomic_state *state) { struct drm_plane *plane; struct drm_plane_state *new_state, *old_state; struct drm_crtc_state *crtc_state; - struct DCP_FW_NAME(neo_dcp_swap_submit_req) *req = &DCP_FW_UNION(neo_dcp->swap); + struct DCP_FW_NAME(dcp_swap_submit_req) *req = &DCP_FW_UNION(dcp->swap); int plane_idx, l; int has_surface = 0; bool update_brightness; @@ -2264,21 +2370,21 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc * Clear all surfaces on startup. The boot framebuffer in surface 0 * sticks around. */ - if (!neo_dcp->surfaces_cleared) { + if (!dcp->surfaces_cleared) { req->swap.swap_enabled = IOMFB_SET_BACKGROUND | 0x7; req->swap.bg_color = 0xFF000000; - neo_dcp->surfaces_cleared = true; + dcp->surfaces_cleared = true; } for_each_oldnew_plane_in_state(state, plane, old_state, new_state, plane_idx) { - struct neo_apple_plane_state *apple_state = to_apple_plane_state(new_state); - struct neo_apple_plane *apl_plane = to_apple_plane(plane); + struct apple_plane_state *apple_state = to_apple_plane_state(new_state); + struct apple_plane *apl_plane = to_apple_plane(plane); /* skip planes not for this crtc */ if (old_state->crtc != crtc && new_state->crtc != crtc) continue; - l = apl_plane->neo_iomfb_surf; + l = apl_plane->iomfb_surf; req->swap.swap_enabled |= BIT(l); if (!new_state->fb || !new_state->visible) { @@ -2290,8 +2396,8 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc req->swap.src_rect[l] = apple_state->src_rect; req->swap.dst_rect[l] = apple_state->dst_rect; - if (neo_dcp->notch_height > 0) - req->swap.dst_rect[l].y += neo_dcp->notch_height; + if (dcp->notch_height > 0) + req->swap.dst_rect[l].y += dcp->notch_height; req->surf_iova[l] = apple_state->iova; req->surf[l].base = apple_state->surf; @@ -2301,15 +2407,27 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc * the contained colorimetry information to provide native * colors. */ - if (neo_dcp->connector_type == DRM_MODE_CONNECTOR_eDP && + if (dcp->connector_type == DRM_MODE_CONNECTOR_eDP && req->surf[l].base.colorspace == DCP_COLORSPACE_BG_SRGB) req->surf[l].base.colorspace = DCP_COLORSPACE_NATIVE; + +#if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) + /* H17P external SDR RGB needs an sRGB source tag. Native + * tagging washes out the same pixel values on this profile. + * Keep YUV, other source spaces and other profiles unchanged. + */ + if (dcp->hw.iomfb_method_profile == + DCP_IOMFB_METHODS_H17P_EXTERNAL && + !new_state->fb->format->is_yuv && + req->surf[l].base.colorspace == DCP_COLORSPACE_NATIVE) + req->surf[l].base.colorspace = DCP_COLORSPACE_BG_SRGB; +#endif } if (!has_surface && !crtc_state->color_mgmt_changed) { if (crtc_state->enable && crtc_state->active && !crtc_state->planes_changed) { - schedule_work(&neo_dcp->vblank_wq); + schedule_work(&dcp->vblank_wq); return; } @@ -2319,8 +2437,8 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc req->clear = 1; } - if (has_surface && (neo_dcp->use_timestamps || neo_dcp->vrr_enabled)) { - u64 submit_timestamp = neo_dcp->swap_submit_timestamp; + if (has_surface && (dcp->use_timestamps || dcp->vrr_enabled)) { + u64 submit_timestamp = dcp->swap_submit_timestamp; u64 timestamp = arch_timer_read_counter(); /* @@ -2337,7 +2455,7 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc req->swap.timestamp[1] = submit_timestamp; req->swap.timestamp[6] = timestamp; - trace_neo_iomfb_vrr_timestamps(neo_dcp, neo_dcp->vrr_enabled, timestamp, + trace_iomfb_vrr_timestamps(dcp, dcp->vrr_enabled, timestamp, submit_timestamp); } @@ -2345,20 +2463,20 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc req->swap.swap_completed = req->swap.swap_enabled; /* update brightness if changed */ - update_brightness = neo_dcp_has_panel(neo_dcp) && neo_dcp->brightness.update; + update_brightness = dcp_has_panel(dcp) && dcp->brightness.update; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER update_brightness = update_brightness && - neo_dcp->hw.neo_iomfb_method_profile == DCP_IOMFB_METHODS_H17G; + dcp->hw.iomfb_method_profile == DCP_IOMFB_METHODS_H17G; #endif if (update_brightness) { req->swap.bl_unk = 1; - req->swap.bl_value = neo_dcp->brightness.dac; + req->swap.bl_value = dcp->brightness.dac; req->swap.bl_power = 0x40; - neo_dcp->brightness.update = false; + dcp->brightness.update = false; } if (crtc_state->color_mgmt_changed) { - struct neo_iomfb_set_matrix_req mat = { + struct iomfb_set_matrix_req mat = { .location = 9, }; @@ -2370,22 +2488,22 @@ void DCP_FW_NAME(neo_iomfb_flush)(struct neo_apple_dcp *neo_dcp, struct drm_crtc } #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) - neo_iomfb_set_matrix(neo_dcp, false, &mat, neo_dcp_pre_swap_a428, NULL); + iomfb_set_matrix(dcp, false, &mat, dcp_pre_swap_a428, NULL); #else - neo_iomfb_set_matrix(neo_dcp, false, &mat, do_swap, NULL); + iomfb_set_matrix(dcp, false, &mat, do_swap, NULL); #endif } else - do_swap(neo_dcp, NULL, NULL); + do_swap(dcp, NULL, NULL); } -static void res_is_main_display(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void res_is_main_display(struct apple_dcp *dcp, void *out, void *cookie) { - struct neo_apple_connector *connector; + struct apple_connector *connector; bool result; /* No reply: the DCP is marked crashed, so only end the start-up wait. */ if (!out) { - complete(&neo_dcp->start_done); + complete(&dcp->start_done); return; } @@ -2395,23 +2513,23 @@ static void res_is_main_display(struct neo_apple_dcp *neo_dcp, void *out, void * */ result = *(u8 *)out; - dev_info(neo_dcp->dev, "DCP is_main_display: %d\n", result); + dev_info(dcp->dev, "DCP is_main_display: %d\n", result); - neo_dcp->main_display = result; + dcp->main_display = result; - connector = neo_dcp->connector; + connector = dcp->connector; if (connector) { - connector->connected = neo_dcp->nr_modes > 0; + connector->connected = dcp->nr_modes > 0; schedule_work(&connector->hotplug_wq); } - neo_dcp->active = true; - complete(&neo_dcp->start_done); + dcp->active = true; + complete(&dcp->start_done); } -static void init_3(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void init_3(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_is_main_display(neo_dcp, false, res_is_main_display, NULL); + dcp_is_main_display(dcp, false, res_is_main_display, NULL); } #if DCP_FW_VER >= DCP_FW_VERSION(26, 0, 0) @@ -2424,13 +2542,13 @@ static void init_3(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) * A412 is not in the table: its result feeds res_is_main_display(), so it * stays on the existing path. */ -enum neo_dcp_client_step_kind { +enum dcp_client_step_kind { DCP_STEP_A445, DCP_STEP_A411, DCP_STEP_A415, DCP_STEP_A450, DCP_STEP_A031, DCP_STEP_A414, DCP_STEP_A478, DCP_STEP_A457, DCP_STEP_A474, DCP_STEP_A476, }; -static const u8 neo_dcp_client_open_seq[] = { +static const u8 dcp_client_open_seq[] = { DCP_STEP_A445, DCP_STEP_A411, DCP_STEP_A445, DCP_STEP_A415, DCP_STEP_A450, DCP_STEP_A031, DCP_STEP_A031, DCP_STEP_A031, DCP_STEP_A031, @@ -2440,40 +2558,40 @@ static const u8 neo_dcp_client_open_seq[] = { DCP_STEP_A478, DCP_STEP_A476, DCP_STEP_A478, }; -static void neo_dcp_client_open_run(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void dcp_client_open_run(struct apple_dcp *dcp, void *out, void *cookie) { unsigned long idx = (unsigned long)cookie; - neo_dcp_callback_t next = neo_dcp_client_open_run; + dcp_callback_t next = dcp_client_open_run; void *ncookie = (void *)(idx + 1); - if (idx >= ARRAY_SIZE(neo_dcp_client_open_seq)) { - init_3(neo_dcp, NULL, NULL); + if (idx >= ARRAY_SIZE(dcp_client_open_seq)) { + init_3(dcp, NULL, NULL); return; } - switch (neo_dcp_client_open_seq[idx]) { + switch (dcp_client_open_seq[idx]) { case DCP_STEP_A445: { u32 zero = 0; - neo_dcp_get_dfb_state(neo_dcp, false, &zero, next, ncookie); + dcp_get_dfb_state(dcp, false, &zero, next, ncookie); break; } case DCP_STEP_A411: { u32 handle = 0; - neo_dcp_set_display_device(neo_dcp, false, &handle, next, ncookie); + dcp_set_display_device(dcp, false, &handle, next, ncookie); break; } case DCP_STEP_A415: { - struct neo_dcp_pipe_cfg_415 cfg = {}; + struct dcp_pipe_cfg_415 cfg = {}; - neo_dcp_pipe_cfg_415(neo_dcp, false, &cfg, next, ncookie); + dcp_pipe_cfg_415(dcp, false, &cfg, next, ncookie); break; } case DCP_STEP_A450: { u32 val = 0; - neo_dcp_enable_disable_video_power_savings(neo_dcp, false, &val, next, + dcp_enable_disable_video_power_savings(dcp, false, &val, next, ncookie); break; } @@ -2494,33 +2612,33 @@ static void neo_dcp_client_open_run(struct neo_apple_dcp *neo_dcp, void *out, vo { 6, 7, 8, 10, 11, 12, 25 }, { 6, 7, 8, 10, 11, 12, 25, 26 }, }; - struct neo_dcp_pipe_cfg_031 cfg = {}; - unsigned int step = neo_dcp->a031_step; + struct dcp_pipe_cfg_031 cfg = {}; + unsigned int step = dcp->a031_step; unsigned int i; if (step >= ARRAY_SIZE(notify_clients)) step = ARRAY_SIZE(notify_clients) - 1; for (i = 0; i < step; i++) cfg.clients[i] = cpu_to_le32(notify_clients[step][i]); - neo_dcp->a031_step++; + dcp->a031_step++; - neo_dcp_pipe_cfg_031(neo_dcp, false, &cfg, next, ncookie); + dcp_pipe_cfg_031(dcp, false, &cfg, next, ncookie); break; } case DCP_STEP_A414: { u64 val = 0; - neo_dcp_pipe_cfg_414(neo_dcp, false, &val, next, ncookie); + dcp_pipe_cfg_414(dcp, false, &val, next, ncookie); break; } case DCP_STEP_A478: - neo_dcp_pipe_query_478(neo_dcp, false, next, ncookie); + dcp_pipe_query_478(dcp, false, next, ncookie); break; case DCP_STEP_A457: - neo_dcp_first_client_open(neo_dcp, false, next, ncookie); + dcp_first_client_open(dcp, false, next, ncookie); break; case DCP_STEP_A474: - neo_dcp_pipe_query_474(neo_dcp, false, next, ncookie); + dcp_pipe_query_474(dcp, false, next, ncookie); break; case DCP_STEP_A476: { /* @@ -2531,59 +2649,59 @@ static void neo_dcp_client_open_run(struct neo_apple_dcp *neo_dcp, void *out, vo */ u32 state = 1; - neo_dcp_pipe_set_476(neo_dcp, false, &state, next, ncookie); + dcp_pipe_set_476(dcp, false, &state, next, ncookie); break; } default: - dev_warn(neo_dcp->dev, "unknown client-open step %u\n", - neo_dcp_client_open_seq[idx]); - init_3(neo_dcp, NULL, NULL); + dev_warn(dcp->dev, "unknown client-open step %u\n", + dcp_client_open_seq[idx]); + init_3(dcp, NULL, NULL); break; } } -static void init_1(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void init_1(struct apple_dcp *dcp, void *out, void *cookie) { /* every client-open sequence starts from the empty A031 state */ - neo_dcp->a031_step = 0; - neo_dcp_client_open_run(neo_dcp, NULL, (void *)0UL); + dcp->a031_step = 0; + dcp_client_open_run(dcp, NULL, (void *)0UL); } #else -static void init_2(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void init_2(struct apple_dcp *dcp, void *out, void *cookie) { - neo_dcp_first_client_open(neo_dcp, false, init_3, NULL); + dcp_first_client_open(dcp, false, init_3, NULL); } -static void init_1(struct neo_apple_dcp *neo_dcp, void *out, void *cookie) +static void init_1(struct apple_dcp *dcp, void *out, void *cookie) { u32 val = 0; - neo_dcp_enable_disable_video_power_savings(neo_dcp, false, &val, init_2, NULL); + dcp_enable_disable_video_power_savings(dcp, false, &val, init_2, NULL); } #endif -static void neo_dcp_started(struct neo_apple_dcp *neo_dcp, void *data, void *cookie) +static void dcp_started(struct apple_dcp *dcp, void *data, void *cookie) { - struct neo_iomfb_get_color_remap_mode_req color_remap = - (struct neo_iomfb_get_color_remap_mode_req){ + struct iomfb_get_color_remap_mode_req color_remap = + (struct iomfb_get_color_remap_mode_req){ .mode = 6, }; - dev_info(neo_dcp->dev, "DCP booted\n"); + dev_info(dcp->dev, "DCP booted\n"); - neo_iomfb_get_color_remap_mode(neo_dcp, false, &color_remap, init_1, cookie); + iomfb_get_color_remap_mode(dcp, false, &color_remap, init_1, cookie); } -void DCP_FW_NAME(neo_iomfb_shutdown)(struct neo_apple_dcp *neo_dcp) +void DCP_FW_NAME(iomfb_shutdown)(struct apple_dcp *dcp) { - struct DCP_FW_NAME(neo_dcp_set_power_state_req) req = { + struct DCP_FW_NAME(dcp_set_power_state_req) req = { /* defaults are ok */ }; - struct neo_dcp_wait_cookie *cookie; + struct dcp_wait_cookie *cookie; int ret; #if DCP_FW_VERSION(26, 0, 0) <= DCP_FW_VER - if (neo_dcp->hw.neo_iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { - neo_iomfb_queue_stop(neo_dcp); + if (dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + iomfb_queue_stop(dcp); return; } #endif @@ -2595,11 +2713,11 @@ void DCP_FW_NAME(neo_iomfb_shutdown)(struct neo_apple_dcp *neo_dcp) init_completion(&cookie->done); kref_init(&cookie->refcount); kref_get(&cookie->refcount); - neo_dcp_set_power_state(neo_dcp, DCP_POWER_OOB, &req, complete_set_powerstate, cookie); + dcp_set_power_state(dcp, DCP_POWER_OOB, &req, complete_set_powerstate, cookie); ret = wait_for_completion_timeout(&cookie->done, msecs_to_jiffies(1000)); - if (ret > 0 && neo_dcp_power_stop_confirmed(cookie)) - neo_dcp_release_all_retained_framebuffers(neo_dcp); + if (ret > 0 && dcp_power_stop_confirmed(cookie)) + dcp_release_all_retained_framebuffers(dcp); else - dev_warn(neo_dcp->dev, "shutdown scanout stop was not confirmed\n"); + dev_warn(dcp->dev, "shutdown scanout stop was not confirmed\n"); kref_put(&cookie->refcount, release_wait_cookie); } diff --git a/drivers/gpu/drm/apple/iomfb_v12_3.c b/drivers/gpu/drm/apple/iomfb_v12_3.c index 092a3878562e40..cae54a1d1a2755 100644 --- a/drivers/gpu/drm/apple/iomfb_v12_3.c +++ b/drivers/gpu/drm/apple/iomfb_v12_3.c @@ -4,6 +4,7 @@ #include "iomfb_v12_3.h" #include "iomfb_v13_3.h" #include "iomfb_v14_7_tpl.h" +#include "iomfb_h17p.h" #include "version_utils.h" static const struct dcp_method_entry dcp_methods[dcpep_num_methods] = { diff --git a/drivers/gpu/drm/apple/iomfb_v13_3.c b/drivers/gpu/drm/apple/iomfb_v13_3.c index 05624058537f8d..f47505fbbdc310 100644 --- a/drivers/gpu/drm/apple/iomfb_v13_3.c +++ b/drivers/gpu/drm/apple/iomfb_v13_3.c @@ -4,6 +4,7 @@ #include "iomfb_v12_3.h" #include "iomfb_v13_3.h" #include "iomfb_v14_7_tpl.h" +#include "iomfb_h17p.h" #include "version_utils.h" static const struct dcp_method_entry dcp_methods[dcpep_num_methods] = { diff --git a/drivers/gpu/drm/apple/iomfb_v14_7.c b/drivers/gpu/drm/apple/iomfb_v14_7.c index d7dcc2ca70fa66..0e39ab4b7478e6 100644 --- a/drivers/gpu/drm/apple/iomfb_v14_7.c +++ b/drivers/gpu/drm/apple/iomfb_v14_7.c @@ -2308,7 +2308,7 @@ static int dcp_v14_mode(struct apple_dcp *dcp, struct apple_dcp_v14 *v14) return ret; ctx.dcp = dcp; modes = enumerate_modes(&ctx, &count, dcp->panel.width_mm, dcp->panel.height_mm, - dcp->notch_height, true); + dcp->notch_height, true, false); if (IS_ERR(modes)) return PTR_ERR(modes); for (i = 0; i < count; i++) { diff --git a/drivers/gpu/drm/apple/iomfb_v14_7_tpl.c b/drivers/gpu/drm/apple/iomfb_v14_7_tpl.c index f59c016cd95c0a..b634d86bde2770 100644 --- a/drivers/gpu/drm/apple/iomfb_v14_7_tpl.c +++ b/drivers/gpu/drm/apple/iomfb_v14_7_tpl.c @@ -11,6 +11,7 @@ #include "iomfb_v12_3.h" #include "iomfb_v13_3.h" #include "iomfb_v14_7_tpl.h" +#include "iomfb_h17p.h" #include "version_utils.h" static const struct dcp_method_entry dcp_methods[dcpep_num_methods] = { diff --git a/drivers/gpu/drm/apple/iomfb_v14_7_tpl.h b/drivers/gpu/drm/apple/iomfb_v14_7_tpl.h index 888ff9ef4dc966..47590b4354232a 100644 --- a/drivers/gpu/drm/apple/iomfb_v14_7_tpl.h +++ b/drivers/gpu/drm/apple/iomfb_v14_7_tpl.h @@ -13,6 +13,15 @@ #include "version_utils.h" +/* + * dcp-internal.h includes this header, also from inside another template + * instantiation (iomfb_h17p.c): keep that one's DCP_FW. + */ +#pragma push_macro("DCP_FW") +#pragma push_macro("DCP_FW_VER") +#undef DCP_FW +#undef DCP_FW_VER + #define DCP_FW v14_7_0 #define DCP_FW_VER DCP_FW_VERSION(14, 7, 0) @@ -21,4 +30,7 @@ #undef DCP_FW_VER #undef DCP_FW +#pragma pop_macro("DCP_FW_VER") +#pragma pop_macro("DCP_FW") + #endif /* __APPLE_IOMFB_V14_7_TPL_H__ */ diff --git a/drivers/gpu/drm/apple/iomfb_v26_6.c b/drivers/gpu/drm/apple/iomfb_v26_6.c index a65d88d89087e6..5a26c08d2d6ef6 100644 --- a/drivers/gpu/drm/apple/iomfb_v26_6.c +++ b/drivers/gpu/drm/apple/iomfb_v26_6.c @@ -1302,7 +1302,7 @@ static int dcp_v26_mode(struct apple_dcp *dcp, struct apple_dcp_v26 *v26) return ret; ctx.dcp = dcp; modes = enumerate_modes(&ctx, &count, dcp->panel.width_mm, dcp->panel.height_mm, - dcp->notch_height, true); + dcp->notch_height, true, false); if (IS_ERR(modes)) return PTR_ERR(modes); for (i = 0; i < count; i++) { diff --git a/drivers/gpu/drm/apple/parser-test.c b/drivers/gpu/drm/apple/parser-test.c index 26ef32e13a4796..96a119bff535e0 100644 --- a/drivers/gpu/drm/apple/parser-test.c +++ b/drivers/gpu/drm/apple/parser-test.c @@ -182,7 +182,7 @@ static struct dcp_display_mode *parse_blob(struct parser_blob *blob, if (ret) return ERR_PTR(ret); ctx.dcp = &parser_dcp; - return enumerate_modes(&ctx, count, 500, 300, notch, internal); + return enumerate_modes(&ctx, count, 500, 300, notch, internal, false); } static void parser_score_presence(struct kunit *test) diff --git a/drivers/gpu/drm/apple/parser.c b/drivers/gpu/drm/apple/parser.c index 689df138df9a14..2dfd85a19be3cf 100644 --- a/drivers/gpu/drm/apple/parser.c +++ b/drivers/gpu/drm/apple/parser.c @@ -325,15 +325,50 @@ int parse(const void *blob, size_t size, struct dcp_parse_ctx *ctx) return 0; } +bool dcp_property_complete(const void *blob, size_t received, size_t advertised) +{ + struct dcp_parse_ctx ctx; + + if (!blob || !received || received > U32_MAX - 3 || + received > advertised || advertised - received > 1) + return false; + /* H17P may count one unsent byte after the aligned serialized object. */ + if (advertised != received && (received & 3)) + return false; + /* Bound decoding to bytes actually delivered, never allocation padding. */ + if (parse(blob, received, &ctx) || skip(&ctx)) + return false; + return ctx.pos == received || round_up(ctx.pos, 4) == received; +} + +/* + * @polarity: external H17P only, also read SyncPolarity and PixelRepetition. + * An automatic 1080p180 exception catalog also records duplicate keys. + */ static int parse_dimension(struct dcp_parse_ctx *handle, struct dimension *dim, - bool require_rate) + bool require_rate, bool polarity) { + static const char * const dup_fields[] = { + "Active", "Total", "FrontPorch", "SyncWidth", "PreciseSyncRate", + }; + unsigned int seen = 0; struct iterator it; unsigned int fields = 0; int ret = 0; + dim->sync_polarity = -1; + dim->pixel_repetition = -1; + dcp_parse_foreach_in_dict(handle, it) { char *key = parse_string(it.handle); + unsigned int field; + + if (!IS_ERR(key) && handle->automatic_1080p180_rgb8) + for (field = 0; field < ARRAY_SIZE(dup_fields); field++) + if (!strcmp(key, dup_fields[field])) { + handle->ambiguous_180hz |= !!(seen & (1U << field)); + seen |= 1U << field; + } if (IS_ERR(key)) { ret = PTR_ERR(key); @@ -353,6 +388,17 @@ static int parse_dimension(struct dcp_parse_ctx *handle, struct dimension *dim, ret = parse_int_bound(it.handle, &dim->precise_sync_rate, require_rate ? 1 : 0, U32_MAX); fields |= BIT(4); + } else if (polarity && !strcmp(key, "SyncPolarity")) { + if (dim->sync_polarity >= 0) + ret = -EINVAL; + else + ret = parse_int_bound(it.handle, &dim->sync_polarity, 0, 1); + } else if (polarity && !strcmp(key, "PixelRepetition")) { + if (dim->pixel_repetition >= 0) + ret = -EINVAL; + else + ret = parse_int_bound(it.handle, &dim->pixel_repetition, + 0, U32_MAX); } else { ret = skip(it.handle); } @@ -410,10 +456,117 @@ static int fill_color_mode(struct dcp_color_mode *color, return 0; } +/* External SDR scanout must not select a wide-gamut mode by score alone. */ +static unsigned int external_sdr_rank(const struct color_mode *color) +{ + if (color->id < 0 || color->id > U32_MAX || + color->eotf != DCP_EOTF_SDR_GAMMA || + color->pixel_encoding != DCP_COLOR_FORMAT_RGB || + color->dynamic_range != DCP_COLOR_YCBCR_RANGE_FULL || + (color->depth != 8 && color->depth != 10)) + return 0; + + switch (color->colorimetry) { + case DCP_COLORIMETRY_SRGB: + case DCP_COLORIMETRY_BT709: + return 2; + case DCP_COLORIMETRY_RGB: + return 1; + default: + return 0; + } +} + +/* Firmware lists color element IDs which this timing cannot safely use. */ +#define DCP_TIMING_COLOR_ID_LIMIT 64 + +struct timing_color_ids { + u32 ids[DCP_TIMING_COLOR_ID_LIMIT]; + unsigned int count; +}; + +struct timing_color_restrictions { + struct timing_color_ids unsafe; + struct timing_color_ids dsc_required; +}; + +static bool timing_color_contains(const struct timing_color_ids *ids, u32 id) +{ + unsigned int i; + + for (i = 0; i < ids->count; i++) + if (ids->ids[i] == id) + return true; + return false; +} + +static int parse_timing_color_ids(struct dcp_parse_ctx *handle, + struct timing_color_ids *ids) +{ + struct iterator it; + int ret; + + ret = iterator_begin(handle, &it, false); + if (ret) + return ret; + if (it.len > DCP_TIMING_COLOR_ID_LIMIT) + return -E2BIG; + for (; it.idx < it.len; it.idx++) { + s64 id; + + ret = parse_int_bound(handle, &id, 0, U32_MAX); + if (ret) + return ret; + if (timing_color_contains(ids, id)) + return -EINVAL; + ids->ids[ids->count++] = id; + } + return 0; +} + +static int parse_timing_color_restrictions(struct dcp_parse_ctx *handle, + struct timing_color_restrictions *limits) +{ + struct iterator it; + bool unsafe_seen = false, dsc_seen = false, colors_seen = false; + int ret; + + ret = iterator_begin(handle, &it, true); + if (ret) + return ret; + for (; it.idx < it.len; it.idx++) { + char *key = parse_string(handle); + + if (IS_ERR(key)) + return PTR_ERR(key); + if (!strcmp(key, "UnsafeColorElementIDs")) { + ret = unsafe_seen ? -EINVAL : + parse_timing_color_ids(handle, &limits->unsafe); + unsafe_seen = true; + } else if (!strcmp(key, "DSCRequiredColorElementIDs")) { + ret = dsc_seen ? -EINVAL : + parse_timing_color_ids(handle, &limits->dsc_required); + dsc_seen = true; + } else if (!strcmp(key, "ColorModes")) { + ret = colors_seen ? -EINVAL : skip(handle); + colors_seen = true; + } else { + ret = skip(handle); + } + kfree(key); + if (ret) + return ret; + } + return 0; +} + static int parse_color_modes(struct dcp_parse_ctx *handle, - struct dcp_display_mode *out) + struct dcp_display_mode *out, bool external_sdr, + const struct timing_color_restrictions *limits, + struct dcp_color_mode *unsafe_rgb8) { struct iterator outer_it; + unsigned int best_sdr_rank = 0, best_unsafe_rank = 0; int ret = 0; out->sdr_444.score = -1; out->sdr_rgb.score = -1; @@ -421,6 +574,11 @@ static int parse_color_modes(struct dcp_parse_ctx *handle, out->best.score = -1; dcp_parse_foreach_in_array(handle, outer_it) { + static const char * const dup_fields[] = { + "Colorimetry", "Depth", "DynamicRange", "EOTF", "ID", + "PixelEncoding", "Score", "IsVirtual", + }; + unsigned int seen = 0; struct iterator it; bool is_virtual = true; struct color_mode cmode = { @@ -430,6 +588,14 @@ static int parse_color_modes(struct dcp_parse_ctx *handle, dcp_parse_foreach_in_dict(handle, it) { char *key = parse_string(it.handle); + unsigned int field; + + if (!IS_ERR(key) && handle->automatic_1080p180_rgb8) + for (field = 0; field < ARRAY_SIZE(dup_fields); field++) + if (!strcmp(key, dup_fields[field])) { + handle->ambiguous_180hz |= !!(seen & (1U << field)); + seen |= 1U << field; + } if (IS_ERR(key)) ret = PTR_ERR(key); @@ -469,6 +635,37 @@ static int parse_color_modes(struct dcp_parse_ctx *handle, cmode.eotf, cmode.dynamic_range, cmode.pixel_encoding); + /* External H17P: bounded SDR selection and timing color limits. */ + if (external_sdr) { + unsigned int rank; + + if (timing_color_contains(&limits->dsc_required, cmode.id)) + continue; + rank = external_sdr_rank(&cmode); + if (timing_color_contains(&limits->unsafe, cmode.id)) { + /* + * Collect only a timing-local bounded RGB8 candidate. + * Admission still requires current EDID and link proof. + */ + if (unsafe_rgb8 && cmode.depth == 8 && rank && + cmode.colorimetry == DCP_COLORIMETRY_SRGB && + rank >= best_unsafe_rank) { + if (rank > best_unsafe_rank) + unsafe_rgb8->score = -1; + fill_color_mode(unsafe_rgb8, &cmode); + best_unsafe_rank = rank; + } + continue; + } + if (!rank || rank < best_sdr_rank) + continue; + if (rank > best_sdr_rank) + out->sdr_rgb.score = -1; + fill_color_mode(&out->sdr_rgb, &cmode); + best_sdr_rank = rank; + continue; + } + if (cmode.eotf == DCP_EOTF_SDR_GAMMA) { if (cmode.pixel_encoding == DCP_COLOR_FORMAT_RGB && cmode.depth <= 10) @@ -498,18 +695,35 @@ static u64 calculate_clock(struct dimension *horiz, struct dimension *vert) return DIV_ROUND_CLOSEST_ULL(clock >> 16, 1000); } +/* + * @external_sdr: the external H17P (T8140 dispext0): bounded SDR color + * selection, timing color restrictions, IsPreferred/IsInterlaced and sync + * polarity, and no adaptive-sync range. + */ static int parse_mode(struct dcp_parse_ctx *handle, struct dcp_display_mode *out, s64 *score, int width_mm, - int height_mm, unsigned notch_height, bool internal) + int height_mm, unsigned notch_height, bool internal, + bool external_sdr) { + static const char * const dup_fields[] = { + "ID", "Score", "MinimumVariableRefreshRate", "MaximumVariableRefreshRate", + }; + unsigned int seen = 0; int ret = 0; struct iterator it; - struct dimension horiz = {}, vert = {}; + struct dimension horiz = { .sync_polarity = -1, .pixel_repetition = -1 }; + struct dimension vert = { .sync_polarity = -1, .pixel_repetition = -1 }; + struct timing_color_restrictions limits = {}; + struct dcp_color_mode unsafe_rgb8 = { .score = -1 }; + bool have_horiz = false, have_vert = false, have_virtual = false; + bool duplicate_scope = false; u64 clock; s64 min_vrr = 0, max_vrr = 0; s64 id = -1; s64 best_color_mode = -1; bool is_virtual = false; + bool preferred = false, interlaced = false; + bool have_preferred = false, have_interlaced = false; struct drm_display_mode *mode = &out->mode; *out = (struct dcp_display_mode) {}; @@ -518,29 +732,67 @@ static int parse_mode(struct dcp_parse_ctx *handle, out->sdr.score = -1; out->best.score = -1; *score = -1; + handle->ambiguous_180hz = false; + + if (external_sdr) { + struct dcp_parse_ctx scan = *handle; + + /* ColorModes can precede either list: collect restrictions first. */ + ret = parse_timing_color_restrictions(&scan, &limits); + if (ret) + return ret; + } dcp_parse_foreach_in_dict(handle, it) { char *key = parse_string(it.handle); + unsigned int field; + + if (!IS_ERR(key) && handle->automatic_1080p180_rgb8) + for (field = 0; field < ARRAY_SIZE(dup_fields); field++) + if (!strcmp(key, dup_fields[field])) { + handle->ambiguous_180hz |= !!(seen & (1U << field)); + seen |= 1U << field; + } if (IS_ERR(key)) ret = PTR_ERR(key); else if (is_virtual) skip(it.handle); - else if (!strcmp(key, "HorizontalAttributes")) - ret = parse_dimension(it.handle, &horiz, false); - else if (!strcmp(key, "VerticalAttributes")) - ret = parse_dimension(it.handle, &vert, true); - else if (!strcmp(key, "MinimumVariableRefreshRate")) + else if (!strcmp(key, "HorizontalAttributes")) { + duplicate_scope |= have_horiz; + have_horiz = true; + ret = parse_dimension(it.handle, &horiz, false, external_sdr); + } else if (!strcmp(key, "VerticalAttributes")) { + duplicate_scope |= have_vert; + have_vert = true; + ret = parse_dimension(it.handle, &vert, true, external_sdr); + } else if (!strcmp(key, "MinimumVariableRefreshRate")) ret = parse_int_bound(it.handle, &min_vrr, 0, U32_MAX); else if (!strcmp(key, "MaximumVariableRefreshRate")) ret = parse_int_bound(it.handle, &max_vrr, 0, U32_MAX); else if (!strcmp(key, "ColorModes")) - ret = parse_color_modes(it.handle, out); + ret = parse_color_modes(it.handle, out, external_sdr, &limits, + handle->automatic_1080p180_rgb8 ? + &unsafe_rgb8 : NULL); else if (!strcmp(key, "ID")) ret = parse_int(it.handle, &id); - else if (!strcmp(key, "IsVirtual")) + else if (!strcmp(key, "IsVirtual")) { + duplicate_scope |= have_virtual; + have_virtual = true; ret = parse_bool(it.handle, &is_virtual); - else if (!strcmp(key, "Score")) + } else if (external_sdr && !strcmp(key, "IsPreferred")) { + if (have_preferred) + ret = -EINVAL; + else + ret = parse_bool(it.handle, &preferred); + have_preferred = true; + } else if (external_sdr && !strcmp(key, "IsInterlaced")) { + if (have_interlaced) + ret = -EINVAL; + else + ret = parse_bool(it.handle, &interlaced); + have_interlaced = true; + } else if (!strcmp(key, "Score")) ret = parse_int(it.handle, score); else skip(it.handle); @@ -553,6 +805,27 @@ static int parse_mode(struct dcp_parse_ctx *handle, return ret; } } + /* + * Never replace a normal candidate or create a timing/color pairing. + * Only the bounded automatic policy may publish or use this candidate. + * Its EDID identity and successful link callbacks must remain current. + * The observed 2080x1106 total and 11796583/65536 Hz are deliberate: + * do not widen this to arbitrary nominal-180 Hz blanking or rates. + */ + if (external_sdr && !internal && + handle->automatic_1080p180_rgb8 && + !handle->ambiguous_180hz && + out->sdr_rgb.score < 0 && unsafe_rgb8.score >= 0 && + !duplicate_scope && have_horiz && have_vert && have_virtual && !is_virtual && + have_interlaced && !interlaced && !notch_height && + horiz.active == 1920 && vert.active == 1080 && + horiz.total == 2080 && vert.total == 1106 && + vert.precise_sync_rate == 11796583 && + horiz.pixel_repetition == 0 && vert.pixel_repetition == 0 && + horiz.sync_polarity >= 0 && vert.sync_polarity >= 0) { + out->sdr_rgb = unsafe_rgb8; + out->automatic_rgb8_quirk = handle->automatic_1080p180_rgb8; + } if (out->sdr_rgb.score >= 0) best_color_mode = out->sdr_rgb.id; else if (out->sdr_444.score >= 0) @@ -597,8 +870,12 @@ static int parse_mode(struct dcp_parse_ctx *handle, out->vrr = true; } - /* Refresh rates are reported by DCP as 16.16 fixed-point Hz. */ - if (min_vrr && max_vrr > min_vrr) { + /* + * Refresh rates are reported by DCP as 16.16 fixed-point Hz. + * External H17P skips adaptive-sync programming: keep the fixed timing + * available, but do not advertise its sink VRR range as driver support. + */ + if (!external_sdr && min_vrr && max_vrr > min_vrr) { out->min_vrr = min_vrr; out->max_vrr = max_vrr; out->vrr = true; @@ -627,6 +904,13 @@ static int parse_mode(struct dcp_parse_ctx *handle, .height_mm = height_mm, }; + if (external_sdr && horiz.sync_polarity >= 0 && vert.sync_polarity >= 0) + mode->flags = (horiz.sync_polarity ? DRM_MODE_FLAG_PHSYNC : DRM_MODE_FLAG_NHSYNC) | + (vert.sync_polarity ? DRM_MODE_FLAG_PVSYNC : DRM_MODE_FLAG_NVSYNC); + out->metadata_timing_valid = external_sdr && !out->automatic_rgb8_quirk && + have_preferred && preferred && + have_interlaced && !interlaced && horiz.pixel_repetition == 0 && + vert.pixel_repetition == 0 && horiz.sync_polarity >= 0 && vert.sync_polarity >= 0; drm_mode_set_name(mode); out->timing_mode_id = id; @@ -639,11 +923,110 @@ static int parse_mode(struct dcp_parse_ctx *handle, return 0; } +/* + * Inspect command identities before filtering can discard conflicting siblings. + * Only automatic exceptions use this conservative receipt-wide check. + */ +static bool dcp_180hz_color_ids_current(struct dcp_parse_ctx *handle, u32 id) +{ + static const char * const fields[] = { + "ID", "Depth", "DynamicRange", "EOTF", "PixelEncoding", "Colorimetry", + }; + struct iterator colors; + + if (iterator_begin(handle, &colors, false)) + return false; + for (; colors.idx < colors.len; colors.idx++) { + struct color_mode color = { .id = -1, .depth = -1, .dynamic_range = -1, + .eotf = -1, .pixel_encoding = -1, .colorimetry = -1 }; + s64 *values[] = { &color.id, &color.depth, &color.dynamic_range, + &color.eotf, &color.pixel_encoding, &color.colorimetry }; + bool is_virtual = true, virtual_seen = false; + unsigned int seen = 0; + struct iterator entry; + + if (iterator_begin(handle, &entry, true)) + return false; + for (; entry.idx < entry.len; entry.idx++) { + char *key = parse_string(handle); + unsigned int field; + int ret; + + if (IS_ERR(key)) + return false; + for (field = 0; field < ARRAY_SIZE(fields); field++) + if (!strcmp(key, fields[field])) + break; + if (field < ARRAY_SIZE(fields)) { + ret = seen & (1U << field) ? -EINVAL : + parse_int_bound(handle, values[field], 0, U32_MAX); + seen |= 1U << field; + } else if (!strcmp(key, "IsVirtual")) { + ret = virtual_seen ? -EINVAL : parse_bool(handle, &is_virtual); + virtual_seen = true; + } else { + ret = skip(handle); + } + kfree(key); + if (ret) + return false; + } + if (color.id == id && + (is_virtual || !virtual_seen || color.depth != 8 || + color.dynamic_range != DCP_COLOR_YCBCR_RANGE_FULL || + color.eotf != DCP_EOTF_SDR_GAMMA || + color.pixel_encoding != DCP_COLOR_FORMAT_RGB || + color.colorimetry != DCP_COLORIMETRY_SRGB)) + return false; + } + return true; +} + +static bool dcp_180hz_ids_current(struct dcp_parse_ctx handle, + const struct dcp_display_mode *mode) +{ + struct iterator timings; + unsigned int matches = 0; + + if (iterator_begin(&handle, &timings, false)) + return false; + for (; timings.idx < timings.len; timings.idx++) { + struct iterator entry; + + if (iterator_begin(&handle, &entry, true)) + return false; + for (; entry.idx < entry.len; entry.idx++) { + char *key = parse_string(&handle); + int ret = 0; + + if (IS_ERR(key)) + return false; + if (!strcmp(key, "ID")) { + s64 id; + + ret = parse_int_bound(&handle, &id, 0, U32_MAX); + if (!ret && id == mode->timing_mode_id) + matches++; + } else if (!strcmp(key, "ColorModes")) { + if (!dcp_180hz_color_ids_current(&handle, mode->color_mode_id)) + ret = -EINVAL; + } else { + ret = skip(&handle); + } + kfree(key); + if (ret || matches > 1) + return false; + } + } + return matches == 1; +} + struct dcp_display_mode *enumerate_modes(struct dcp_parse_ctx *handle, unsigned int *count, int width_mm, int height_mm, unsigned notch_height, - bool internal) + bool internal, bool external_sdr) { + struct dcp_parse_ctx catalog = *handle; struct iterator it; int ret; struct dcp_display_mode *mode, *modes; @@ -677,7 +1060,7 @@ struct dcp_display_mode *enumerate_modes(struct dcp_parse_ctx *handle, mode = &modes[*count]; ret = parse_mode(it.handle, mode, &score, width_mm, height_mm, - notch_height, internal); + notch_height, internal, external_sdr); *handle = next; /* Errors for a single mode are recoverable -- just skip it. */ @@ -687,7 +1070,7 @@ struct dcp_display_mode *enumerate_modes(struct dcp_parse_ctx *handle, /* Process a successful mode */ (*count)++; - if (score > best_score) { + if (!mode->automatic_rgb8_quirk && score > best_score) { best_score = score; best_mode = mode; } @@ -696,6 +1079,17 @@ struct dcp_display_mode *enumerate_modes(struct dcp_parse_ctx *handle, if (best_mode != NULL) best_mode->mode.type |= DRM_MODE_TYPE_PREFERRED; + if (handle->automatic_1080p180_rgb8) { + unsigned int i, kept = 0; + + for (i = 0; i < *count; i++) { + if (modes[i].automatic_rgb8_quirk && + !dcp_180hz_ids_current(catalog, &modes[i])) + continue; + modes[kept++] = modes[i]; + } + *count = kept; + } return modes; } @@ -715,7 +1109,7 @@ int replace_modes(struct dcp_parse_ctx *handle, return ret; next = enumerate_modes(handle, &next_count, width_mm, height_mm, - notch_height, internal); + notch_height, internal, false); if (IS_ERR(next)) return PTR_ERR(next); if (it.len && !next_count) { @@ -769,8 +1163,278 @@ int parse_display_attributes(struct dcp_parse_ctx *handle, int *width_mm, return 0; } +/* H17P (T8140, T8142): DisplayAttributes as aurora-wip reads them. */ +int parse_display_attributes_h17p(struct dcp_parse_ctx *handle, int *width_mm, + int *height_mm) +{ + int ret = 0; + struct iterator it; + s64 width_cm = 0, height_cm = 0; + + ret = iterator_begin(handle, &it, true); + if (ret) + return ret; + + for (; it.idx < it.len; ++it.idx) { + char *key = parse_string(it.handle); + + if (IS_ERR(key)) + ret = PTR_ERR(key); + else if (!strcmp(key, "MaxHorizontalImageSize")) + ret = parse_int(it.handle, &width_cm); + else if (!strcmp(key, "MaxVerticalImageSize")) + ret = parse_int(it.handle, &height_cm); + else + ret = skip(it.handle); + + if (!IS_ERR_OR_NULL(key)) + kfree(key); + + if (ret) + return ret; + } + + /* 1cm = 10mm */ + *width_mm = 10 * width_cm; + *height_mm = 10 * height_cm; + + return 0; +} + +/* Metadata strings must fit an EDID text descriptor without hidden suffixes. */ +static int parse_metadata_string(struct dcp_parse_ctx *handle, char *out, size_t size) +{ + const struct dcp_parse_tag *tag = parse_tag_of_type(handle, DCP_TYPE_STRING); + const u8 *value; + + if (IS_ERR(tag)) + return PTR_ERR(tag); + if (!tag->size || tag->size >= size) + return -EINVAL; + value = parse_bytes(handle, tag->size); + if (IS_ERR(value)) + return PTR_ERR(value); + for (unsigned int i = 0; i < tag->size; i++) + if (value[i] < 0x20 || value[i] > 0x7e) + return -EINVAL; + memcpy(out, value, tag->size); + out[tag->size] = 0; + return 0; +} + +static int parse_metadata_number(struct dcp_parse_ctx *handle, u32 *out, u32 min, u32 max) +{ + s64 value; + int ret = parse_int_bound(handle, &value, min, max); + + if (!ret) + *out = value; + return ret; +} + +static int parse_metadata_point(struct dcp_parse_ctx *handle, u32 point[2], u32 *gamma) +{ + struct iterator it; + unsigned int seen = 0; + int ret = iterator_begin(handle, &it, true); + + if (ret) + return ret; + for (; it.idx < it.len; it.idx++) { + char key[64]; + + ret = parse_metadata_string(handle, key, sizeof(key)); + if (ret) + return ret; + if (!strcmp(key, "X")) { + if (seen & 1) + return -EINVAL; + seen |= 1; + ret = parse_metadata_number(handle, &point[0], 1, 65535); + } else if (!strcmp(key, "Y")) { + if (seen & 2) + return -EINVAL; + seen |= 2; + ret = parse_metadata_number(handle, &point[1], 1, 65535); + } else if (gamma && !strcmp(key, "Gamma")) { + if (seen & 4) + return -EINVAL; + seen |= 4; + ret = parse_metadata_number(handle, gamma, 65536, (354U << 16) / 100); + } else { + ret = skip(handle); + } + if (ret) + return ret; + } + if (seen != (gamma ? 7 : 3) || point[0] + point[1] > 65536) + return -EINVAL; + return 0; +} + +static int parse_metadata_chromaticity(struct dcp_parse_ctx *handle, + struct dcp_display_metadata *out) +{ + static const char * const names[] = { "Red", "Green", "Blue" }; + struct iterator it; + unsigned int seen = 0; + int ret = iterator_begin(handle, &it, true); + + if (ret) + return ret; + for (; it.idx < it.len; it.idx++) { + char key[64]; + unsigned int color; + + ret = parse_metadata_string(handle, key, sizeof(key)); + if (ret) + return ret; + for (color = 0; color < ARRAY_SIZE(names); color++) + if (!strcmp(key, names[color])) + break; + if (color < ARRAY_SIZE(names)) { + if (seen & 1U << color) + return -EINVAL; + seen |= 1U << color; + ret = parse_metadata_point(handle, out->chromaticity[color], NULL); + } else { + ret = skip(handle); + } + if (ret) + return ret; + } + return seen == 7 ? 0 : -EINVAL; +} + +static int parse_metadata_product(struct dcp_parse_ctx *handle, struct dcp_display_metadata *out) +{ + struct iterator it; + unsigned int seen = 0; + u32 legacy = 0; + int ret = iterator_begin(handle, &it, true); + + if (ret) + return ret; + for (; it.idx < it.len; it.idx++) { + char key[64]; + + ret = parse_metadata_string(handle, key, sizeof(key)); + if (ret) + return ret; + if (!strcmp(key, "ManufacturerID")) { + if (seen & 1) + return -EINVAL; + seen |= 1; + ret = parse_metadata_string(handle, out->manufacturer, + sizeof(out->manufacturer)); + } else if (!strcmp(key, "ProductName")) { + if (seen & 2) + return -EINVAL; + seen |= 2; + ret = parse_metadata_string(handle, out->name, sizeof(out->name)); + } else if (!strcmp(key, "ProductID")) { + if (seen & 4) + return -EINVAL; + seen |= 4; + ret = parse_metadata_number(handle, &out->product, 0, U16_MAX); + } else if (!strcmp(key, "WeekOfManufacture")) { + if (seen & 8) + return -EINVAL; + seen |= 8; + ret = parse_metadata_number(handle, &out->week, 0, 53); + } else if (!strcmp(key, "YearOfManufacture")) { + if (seen & 16) + return -EINVAL; + seen |= 16; + ret = parse_metadata_number(handle, &out->year, 1990, 2245); + } else if (!strcmp(key, "SerialNumber")) { + if (seen & 64) + return -EINVAL; + seen |= 64; + ret = parse_metadata_number(handle, &out->serial, 0, U32_MAX); + } else if (!strcmp(key, "AlphanumericSerialNumber")) { + if (seen & 128) + return -EINVAL; + seen |= 128; + ret = parse_metadata_string(handle, out->serial_text, + sizeof(out->serial_text)); + } else if (!strcmp(key, "LegacyManufacturerID")) { + if (seen & 32) + return -EINVAL; + seen |= 32; + ret = parse_metadata_number(handle, &legacy, 0, 0x7fff); + } else { + ret = skip(handle); + } + if (ret) + return ret; + } + if ((seen & 31) != 31 || strlen(out->manufacturer) != 3) + return -EINVAL; + for (unsigned int i = 0; i < 3; i++) + if (out->manufacturer[i] < 'A' || out->manufacturer[i] > 'Z') + return -EINVAL; + if ((seen & 32) && legacy != (u32)((out->manufacturer[0] - '@') << 10 | + (out->manufacturer[1] - '@') << 5 | + (out->manufacturer[2] - '@'))) + return -EINVAL; + return 0; +} + +int parse_display_metadata(struct dcp_parse_ctx *handle, struct dcp_display_metadata *out) +{ + struct dcp_display_metadata parsed = {}; + struct iterator it; + unsigned int seen = 0; + int ret = iterator_begin(handle, &it, true); + + if (ret) + return ret; + for (; it.idx < it.len; it.idx++) { + char key[64]; + + ret = parse_metadata_string(handle, key, sizeof(key)); + if (ret) + return ret; + if (!strcmp(key, "MaxHorizontalImageSize")) { + if (seen & 1) + return -EINVAL; + seen |= 1; + ret = parse_metadata_number(handle, &parsed.width_cm, 1, 255); + } else if (!strcmp(key, "MaxVerticalImageSize")) { + if (seen & 2) + return -EINVAL; + seen |= 2; + ret = parse_metadata_number(handle, &parsed.height_cm, 1, 255); + } else if (!strcmp(key, "Chromaticity")) { + if (seen & 4) + return -EINVAL; + seen |= 4; + ret = parse_metadata_chromaticity(handle, &parsed); + } else if (!strcmp(key, "DefaultWhitePoint")) { + if (seen & 8) + return -EINVAL; + seen |= 8; + ret = parse_metadata_point(handle, parsed.white, &parsed.gamma); + } else if (!strcmp(key, "ProductAttributes")) { + if (seen & 16) + return -EINVAL; + seen |= 16; + ret = parse_metadata_product(handle, &parsed); + } else { + ret = skip(handle); + } + if (ret) + return ret; + } + if (seen != 31) + return -EINVAL; + *out = parsed; + return 0; +} + int parse_epic_service_init(struct dcp_parse_ctx *handle, const char **name, - const char **class, s64 *unit) + const char **class, s64 *unit, bool h17p_keys) { int ret = 0; struct iterator it; @@ -789,19 +1453,36 @@ int parse_epic_service_init(struct dcp_parse_ctx *handle, const char **name, break; } - if (!strcmp(key, "EPICName")) { + /* + * H17P advertises its system service with lower-case keys -- + * "compartment", "name", "interface-id" -- where earlier + * firmware uses EPICName/EPICProviderClass/EPICUnit. Without + * them the service never registers, the firmware has no system + * service to ask and idles the panel, so no vblank and no swap + * ever completes. Only H17P callers accept them. If a + * dictionary carries both spellings, the later value wins and + * the earlier string is freed. + */ + if (!strcmp(key, "EPICName") || + (h17p_keys && !strcmp(key, "compartment"))) { + if (parsed_name) + kfree(*name); *name = parse_string(it.handle); if (IS_ERR(*name)) ret = PTR_ERR(*name); else parsed_name = true; - } else if (!strcmp(key, "EPICProviderClass")) { + } else if (!strcmp(key, "EPICProviderClass") || + (h17p_keys && !strcmp(key, "name"))) { + if (parsed_class) + kfree(*class); *class = parse_string(it.handle); if (IS_ERR(*class)) ret = PTR_ERR(*class); else parsed_class = true; - } else if (!strcmp(key, "EPICUnit")) { + } else if (!strcmp(key, "EPICUnit") || + (h17p_keys && !strcmp(key, "interface-id"))) { ret = parse_int(it.handle, unit); if (!ret) parsed_unit = true; diff --git a/drivers/gpu/drm/apple/parser.h b/drivers/gpu/drm/apple/parser.h index 18919ed777b307..610e601d947fd9 100644 --- a/drivers/gpu/drm/apple/parser.h +++ b/drivers/gpu/drm/apple/parser.h @@ -7,6 +7,9 @@ /* For mode parsing */ #include +#include "dcp-display-metadata.h" +#include "dcp-metadata-epoch.h" + struct apple_dcp; struct apple_connector; struct apple_dcp_typec_route; @@ -15,6 +18,9 @@ struct dcp_parse_ctx { struct apple_dcp *dcp; const void *blob; u32 pos, len; + /* External H17P candidates require separate publication/lookup proof. */ + bool automatic_1080p180_rgb8; + bool ambiguous_180hz; }; enum dcp_color_eotf { @@ -96,18 +102,30 @@ struct dcp_display_mode { struct dcp_color_mode sdr; struct dcp_color_mode best; bool vrr; + bool metadata_timing_valid; + bool automatic_rgb8_quirk; + u64 automatic_connection_sequence; + u64 automatic_catalog_sequence; + bool automatic_service_pending; + /* Immutable receipt provenance; never retag a cached exception. */ + struct dcp_metadata_ticket unsafe_rgb8_ticket; + u64 unsafe_rgb8_service_epoch; }; struct dimension { s64 total, front_porch, sync_width, active; s64 precise_sync_rate; + s64 sync_polarity; + s64 pixel_repetition; }; int parse(const void *blob, size_t size, struct dcp_parse_ctx *ctx); +bool dcp_property_complete(const void *blob, size_t received, size_t advertised); struct dcp_display_mode *enumerate_modes(struct dcp_parse_ctx *handle, unsigned int *count, int width_mm, int height_mm, unsigned notch_height, - bool internal); + bool internal, bool external_sdr); +int parse_display_metadata(struct dcp_parse_ctx *handle, struct dcp_display_metadata *out); /* Caller holds modes_lock while checking connector ownership. */ bool dcp_modes_for_connector(struct apple_dcp *dcp, struct apple_connector *connector); @@ -134,8 +152,10 @@ int replace_modes(struct dcp_parse_ctx *handle, bool internal); int parse_display_attributes(struct dcp_parse_ctx *handle, int *width_mm, int *height_mm, bool *backlight_control); +int parse_display_attributes_h17p(struct dcp_parse_ctx *handle, int *width_mm, + int *height_mm); int parse_epic_service_init(struct dcp_parse_ctx *handle, const char **name, - const char **class, s64 *unit); + const char **class, s64 *unit, bool h17p_keys); struct dcp_sound_format_mask { u64 formats; /* SNDRV_PCM_FMTBIT_* */ diff --git a/drivers/gpu/drm/apple/plane.c b/drivers/gpu/drm/apple/plane.c index fbb1311e0bc5cd..48fbca01f4f9b8 100644 --- a/drivers/gpu/drm/apple/plane.c +++ b/drivers/gpu/drm/apple/plane.c @@ -5,6 +5,8 @@ #include "plane.h" +#include "dcp.h" +#include "dcp-internal.h" #include "iomfb_internal.h" #include @@ -13,6 +15,7 @@ #include #include #include +#include #include #include @@ -292,13 +295,78 @@ static void apple_plane_atomic_update(struct drm_plane *plane, new_state->iova = obj->dma_addr + base->fb->offsets[0]; } +static void apple_plane_cleanup_fb(struct drm_plane *plane, + struct drm_plane_state *state) +{ + struct apple_plane_state *apple_state = to_apple_plane_state(state); + struct dcp_fb_reference *entry = apple_state->retirement; + + if (!entry) + return; + + apple_state->retirement = NULL; + drm_framebuffer_put(entry->fb); + kfree(entry); +} + +/* Does @plane scan out on an H17-generation pipeline (T8140, T8142)? */ +static bool apple_plane_h17p(struct drm_plane *plane) +{ + struct drm_crtc *crtc; + + drm_for_each_crtc(crtc, plane->dev) { + struct apple_dcp *dcp; + + if (!(plane->possible_crtcs & drm_crtc_mask(crtc))) + continue; + dcp = platform_get_drvdata(to_apple_crtc(crtc)->dcp); + return dcp && dcp->fw_compat == DCP_FIRMWARE_H17P; + } + return false; +} + +/* + * H17P: a displaced framebuffer stays referenced until the swap that + * replaces it completes (dcp_release_retained_framebuffers()). + */ +static int apple_plane_prepare_fb(struct drm_plane *plane, + struct drm_plane_state *state) +{ + struct drm_plane_state *old; + struct dcp_fb_reference *entry; + int ret; + + ret = drm_gem_plane_helper_prepare_fb(plane, state); + if (ret) + return ret; + + old = drm_atomic_get_old_plane_state(state->state, plane); + if (!old || !old->crtc || !old->fb || old->fb == state->fb) + return 0; + if (!apple_plane_h17p(plane)) + return 0; + + entry = kzalloc_obj(*entry); + if (!entry) + return -ENOMEM; + + drm_framebuffer_get(old->fb); + entry->fb = old->fb; + to_apple_plane_state(state)->retirement = entry; + return 0; +} + static const struct drm_plane_helper_funcs apple_primary_plane_helper_funcs = { + .prepare_fb = apple_plane_prepare_fb, + .cleanup_fb = apple_plane_cleanup_fb, .atomic_check = apple_plane_atomic_check, .atomic_update = apple_plane_atomic_update, .get_scanout_buffer = drm_fb_dma_get_scanout_buffer, }; static const struct drm_plane_helper_funcs apple_plane_helper_funcs = { + .prepare_fb = apple_plane_prepare_fb, + .cleanup_fb = apple_plane_cleanup_fb, .atomic_check = apple_plane_atomic_check, .atomic_update = apple_plane_atomic_update, }; @@ -307,8 +375,10 @@ static const struct drm_plane_helper_funcs apple_plane_helper_funcs = { static void apple_plane_reset(struct drm_plane *plane) { struct apple_plane_state *state = to_apple_plane_state(plane->state); - if (state) + if (state) { + apple_plane_cleanup_fb(plane, &state->base); __drm_atomic_helper_plane_destroy_state(&state->base); + } kfree(state); plane->state = NULL; @@ -333,23 +403,29 @@ apple_plane_duplicate_state(struct drm_plane *plane) __drm_atomic_helper_plane_duplicate_state(plane, &apple_plane_state->base); apple_plane_state->surf = old_apple_plane_state->surf; + /* H17P presents re-use the cached scanout geometry. */ + if (apple_plane_h17p(plane)) { + apple_plane_state->src_rect = old_apple_plane_state->src_rect; + apple_plane_state->dst_rect = old_apple_plane_state->dst_rect; + apple_plane_state->iova = old_apple_plane_state->iova; + } return &apple_plane_state->base; } -// void apple_plane_destroy_state(struct drm_plane *plane, -// struct drm_plane_state *state) -// { -// drm_atomic_helper_plane_destroy_state(plane, state); -// } +static void apple_plane_destroy_state(struct drm_plane *plane, + struct drm_plane_state *state) +{ + apple_plane_cleanup_fb(plane, state); + drm_atomic_helper_plane_destroy_state(plane, state); +} static const struct drm_plane_funcs apple_plane_funcs = { .update_plane = drm_atomic_helper_update_plane, .disable_plane = drm_atomic_helper_disable_plane, .reset = apple_plane_reset, .atomic_duplicate_state = apple_plane_duplicate_state, - // .atomic_destroy_state = apple_plane_destroy_state, - .atomic_destroy_state = drm_atomic_helper_plane_destroy_state, + .atomic_destroy_state = apple_plane_destroy_state, }; /* @@ -380,6 +456,23 @@ static const u32 dcp_primary_formats[] = { #endif }; +/* The H17P alpha semantics of 10-bit XRGB are unknown. */ +static const u32 dcp_primary_formats_h17p[] = { + DRM_FORMAT_ARGB2101010, + DRM_FORMAT_XRGB8888, + DRM_FORMAT_ARGB8888, + DRM_FORMAT_XBGR8888, + DRM_FORMAT_ABGR8888, + DRM_FORMAT_NV12, + DRM_FORMAT_NV16, + DRM_FORMAT_NV24, + DRM_FORMAT_P010, + DRM_FORMAT_P210, +#if defined(DRM_FORMAT_P410) + DRM_FORMAT_P410, +#endif +}; + static const u32 dcp_overlay_formats[] = { DRM_FORMAT_ARGB2101010, DRM_FORMAT_ARGB8888, @@ -434,6 +527,7 @@ struct drm_plane *apple_plane_init(struct drm_device *dev, unsigned long possible_crtcs, u32 iomfb_surf, bool supports_l10r, + bool supports_xrgb2101010, enum drm_plane_type type) { struct apple_plane *plane; @@ -442,9 +536,12 @@ struct drm_plane *apple_plane_init(struct drm_device *dev, switch (type) { case DRM_PLANE_TYPE_PRIMARY: - if (supports_l10r) { + if (supports_l10r && supports_xrgb2101010) { fmts = dcp_primary_formats; num_fmts = ARRAY_SIZE(dcp_primary_formats); + } else if (supports_l10r) { + fmts = dcp_primary_formats_h17p; + num_fmts = ARRAY_SIZE(dcp_primary_formats_h17p); } else { fmts = dcp_primary_formats_12_x; num_fmts = ARRAY_SIZE(dcp_primary_formats_12_x); diff --git a/drivers/gpu/drm/apple/plane.h b/drivers/gpu/drm/apple/plane.h index 2d1a03e91f38bc..ebb2420bf4041d 100644 --- a/drivers/gpu/drm/apple/plane.h +++ b/drivers/gpu/drm/apple/plane.h @@ -19,12 +19,15 @@ struct apple_plane { #define to_apple_plane(x) container_of(x, struct apple_plane, base) +struct dcp_fb_reference; + struct apple_plane_state { struct drm_plane_state base; struct dcp_surface surf; struct dcp_rect src_rect; struct dcp_rect dst_rect; u64 iova; + struct dcp_fb_reference *retirement; }; #define to_apple_plane_state(x) container_of(x, struct apple_plane_state, base) @@ -33,6 +36,7 @@ struct drm_plane *apple_plane_init(struct drm_device *dev, unsigned long possible_crtcs, u32 iomfb_surf, bool supports_l10r, + bool supports_xrgb2101010, enum drm_plane_type type); struct drm_plane *apple_plane_init_v14_7(struct drm_device *dev, unsigned long possible_crtcs, diff --git a/drivers/gpu/drm/apple/systemep.c b/drivers/gpu/drm/apple/systemep.c index 9fe7a0ce495aab..a32f0a0bcf6a0f 100644 --- a/drivers/gpu/drm/apple/systemep.c +++ b/drivers/gpu/drm/apple/systemep.c @@ -75,6 +75,10 @@ static int powerlog_report(struct apple_epic_service *service, enum epic_subtype struct apple_dcp *dcp = service->ep->dcp; int ret; + /* Later reports must not overwrite a latched hint or Linux's target. */ + if (dcp_backlight_active(dcp)) + return 0; + dev_dbg(dcp->dev, "systemep[ch:%u]: report type:%02x len:%zu\n", service->channel, type, data_size); @@ -96,6 +100,15 @@ static int powerlog_report(struct apple_epic_service *service, enum epic_subtype dev_dbg(dcp->dev, "systemep: mNits event: Nits: %u.%03u, iDAC: %u\n", mnits.millinits / 1000, mnits.millinits % 1000, mnits.idac); + if (dcp->fw_compat == DCP_FIRMWARE_H17P && + dcp->hw.iomfb_method_profile != DCP_IOMFB_METHODS_H17G) { + if (!dcp_has_panel(dcp)) + return 0; + ret = dcp_backlight_takeover(dcp, mnits.millinits); + if (ret && ret != -EBUSY) + dev_dbg(dcp->dev, "backlight takeover hint unavailable: %d\n", ret); + return 0; + } dcp->brightness.nits = mnits.millinits / 1000; diff --git a/drivers/gpu/drm/apple/trace.h b/drivers/gpu/drm/apple/trace.h index c0acac6ee690ab..cbb3f4080fd573 100644 --- a/drivers/gpu/drm/apple/trace.h +++ b/drivers/gpu/drm/apple/trace.h @@ -109,11 +109,11 @@ TRACE_EVENT(dcp_send_msg, show_dcp_endpoint(__entry->endpoint), __entry->message)); TRACE_EVENT( - afk_getbuf, TP_PROTO(struct apple_dcp_afkep *ep, u16 size, u16 tag), + afk_getbuf, TP_PROTO(struct apple_dcp_afkep *ep, u32 size, u16 tag), TP_ARGS(ep, size, tag), TP_STRUCT__entry(__string(devname, dev_name(ep->dcp->dev)) - __field(u8, endpoint) __field(u16, size) + __field(u8, endpoint) __field(u32, size) __field(u16, tag)), TP_fast_assign(__assign_str(devname); @@ -398,18 +398,21 @@ TRACE_EVENT(dptxport_init, TP_PROTO(struct apple_dcp *dcp, u64 unit), TRACE_EVENT( dptxport_apcall, - TP_PROTO(struct dptx_port *dptx, int idx, size_t len), - TP_ARGS(dptx, idx, len), + TP_PROTO(struct dptx_port *dptx, u16 group, int idx, size_t len), + TP_ARGS(dptx, group, idx, len), TP_STRUCT__entry(__string(devname, dev_name(dptx->service->ep->dcp->dev)) - __field(u32, unit) __field(int, idx) __field(size_t, len)), + __field(u32, unit) __field(u16, group) __field(int, idx) + __field(size_t, len)), TP_fast_assign(__assign_str(devname); - __entry->unit = dptx->unit; __entry->idx = idx; __entry->len = len;), + __entry->unit = dptx->unit; __entry->group = group; + __entry->idx = idx; __entry->len = len;), - TP_printk("%s: dptx%d: AP Call %d (%s) with len %lu", __get_str(devname), - __entry->unit, - __entry->idx, show_dptxport_apcall(__entry->idx), __entry->len)); + TP_printk("%s: dptx%d: AP Call %u:%d (%s) with len %lu", __get_str(devname), + __entry->unit, __entry->group, __entry->idx, + __entry->group ? "-" : show_dptxport_apcall(__entry->idx), + __entry->len)); TRACE_EVENT( dptxport_validate_connection, diff --git a/drivers/gpu/drm/asahi-neo/Kconfig b/drivers/gpu/drm/asahi-neo/Kconfig deleted file mode 100644 index 14005176f72bf7..00000000000000 --- a/drivers/gpu/drm/asahi-neo/Kconfig +++ /dev/null @@ -1,14 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 -config DRM_ASAHI_NEO - tristate "Asahi Neo (Apple T8140 G17 GPU)" - depends on RUST && DRM=y - depends on ARM64 && ARCH_APPLE && MMU && IOMMU_SUPPORT - depends on PAGE_SIZE_16KB - select RUST_DRM_SCHED - select RUST_DRM_GEM_SHMEM_HELPER - select RUST_DRM_GPUVM - select RUST_APPLE_RTKIT - select WANT_DEV_COREDUMP - help - Native graphics support for the T8140 G17P GPU in the MacBook Neo. - Uses an independent command interface from the M1/M2/M3 driver. diff --git a/drivers/gpu/drm/asahi-neo/Makefile b/drivers/gpu/drm/asahi-neo/Makefile deleted file mode 100644 index 055d0b797b254d..00000000000000 --- a/drivers/gpu/drm/asahi-neo/Makefile +++ /dev/null @@ -1,3 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_DRM_ASAHI_NEO) += asahi_neo.o -asahi_neo-y := asahi_core.o sysfs.o diff --git a/drivers/gpu/drm/asahi-neo/alloc.rs b/drivers/gpu/drm/asahi-neo/alloc.rs deleted file mode 100644 index 5e55ce5ad975a6..00000000000000 --- a/drivers/gpu/drm/asahi-neo/alloc.rs +++ /dev/null @@ -1,1087 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU kernel object allocator. -//! -//! This kernel driver needs to manage a large number of GPU objects, in both firmware/kernel -//! address space and user address space. This module implements a simple grow-only heap allocator -//! based on the DRM MM range allocator, and a debug allocator that allocates each object as a -//! separate GEM object. -//! -//! Allocations may optionally have debugging enabled, which adds preambles that store metadata -//! about the allocation. This is useful for live debugging using the hypervisor or postmortem -//! debugging with a GPU memory snapshot, since it makes it easier to identify use-after-free and -//! caching issues. - -use kernel::{ - drm_neo::mm, - error::Result, - prelude::*, - str::CString, // -}; - -use crate::debug::*; -use crate::driver::{ - AsahiDevRef, - AsahiDevice, // -}; -use crate::fw::types::Zeroable; -use crate::mmu; -use crate::object::{ - GpuArray, - GpuObject, - GpuOnlyArray, - GpuStruct, - GpuWeakPointer, // -}; -use crate::util::RangeExt; - -use core::cmp::Ordering; -use core::fmt; -use core::fmt::{ - Debug, - Formatter, // -}; -use core::marker::PhantomData; -use core::mem; -use core::ops::Range; -use core::ptr::NonNull; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Alloc; - -#[cfg(not(CONFIG_DRM_ASAHI_NEO_DEBUG_ALLOCATOR))] -/// The driver-global allocator type -pub(crate) type DefaultAllocator = HeapAllocator; - -#[cfg(not(CONFIG_DRM_ASAHI_NEO_DEBUG_ALLOCATOR))] -/// The driver-global allocation type -pub(crate) type DefaultAllocation = HeapAllocation; - -#[cfg(CONFIG_DRM_ASAHI_NEO_DEBUG_ALLOCATOR)] -/// The driver-global allocator type -pub(crate) type DefaultAllocator = SimpleAllocator; - -#[cfg(CONFIG_DRM_ASAHI_NEO_DEBUG_ALLOCATOR)] -/// The driver-global allocation type -pub(crate) type DefaultAllocation = SimpleAllocation; - -/// Represents a raw allocation (without any type information). -pub(crate) trait RawAllocation { - /// Returns the CPU-side pointer (if CPU mapping is enabled) as a byte non-null pointer. - fn ptr(&self) -> Option>; - /// Returns the GPU VA pointer as a u64. - fn gpu_ptr(&self) -> u64; - /// Returns the AsahiDevice that owns this allocation. - fn device(&self) -> &AsahiDevice; -} - -/// Represents a typed allocation. -pub(crate) trait Allocation: Debug { - /// Returns the typed CPU-side pointer (if CPU mapping is enabled). - fn ptr(&self) -> Option>; - /// Returns the GPU VA pointer as a u64. - fn gpu_ptr(&self) -> u64; - /// Returns the size of the allocation in bytes. - fn size(&self) -> usize; - /// Returns the AsahiDevice that owns this allocation. - fn device(&self) -> &AsahiDevice; -} - -/// A generic typed allocation wrapping a RawAllocation. -/// -/// This is currently the only Allocation implementation, since it is shared by all allocators. -/// -/// # Invariants -/// The alloaction at `alloc` must have a size equal or greater than `alloc_size` plus `debug_offset` plus `padding`. -pub(crate) struct GenericAlloc { - alloc: U, - alloc_size: usize, - debug_offset: usize, - padding: usize, - tag: u32, - pad_word: u32, - _p: PhantomData, -} - -impl Allocation for GenericAlloc { - /// Returns a pointer to the inner (usable) part of the allocation. - fn ptr(&self) -> Option> { - // SAFETY: self.debug_offset is always within the allocation per the invariant, so is safe to add - // to the base pointer. - unsafe { self.alloc.ptr().map(|p| p.add(self.debug_offset).cast()) } - } - /// Returns the GPU pointer to the inner (usable) part of the allocation. - fn gpu_ptr(&self) -> u64 { - self.alloc.gpu_ptr() + self.debug_offset as u64 - } - /// Returns the size of the inner (usable) part of the allocation. - fn size(&self) -> usize { - self.alloc_size - } - fn device(&self) -> &AsahiDevice { - self.alloc.device() - } -} - -impl Debug for GenericAlloc { - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - f.debug_struct(core::any::type_name::>()) - .field("ptr", &format_args!("{:?}", self.ptr())) - .field("gpu_ptr", &format_args!("{:#X?}", self.gpu_ptr())) - .field("size", &format_args!("{:#X?}", self.size())) - .finish() - } -} - -/// Debugging data associated with an allocation, when debugging is enabled. -#[repr(C)] -struct AllocDebugData { - state: u32, - tag: u32, - size: u64, - base_gpuva: u64, - obj_gpuva: u64, - name: [u8; 0x20], -} - -/// Magic flag indicating a live allocation. -const STATE_LIVE: u32 = u32::from_le_bytes(*b"LIVE"); -/// Magic flag indicating a freed allocation. -const STATE_DEAD: u32 = u32::from_le_bytes(*b"DEAD"); - -/// Marker byte to identify when firmware/GPU write beyond the end of an allocation. -const GUARD_MARKER: u32 = 0x93939393; - -impl Drop for GenericAlloc { - fn drop(&mut self) { - let debug_len = mem::size_of::(); - if self.debug_offset >= debug_len { - if let Some(p) = self.alloc.ptr() { - // SAFETY: self.debug_offset is always greater than the alloc size per - // the invariant, and greater than debug_len as checked above. - unsafe { - let p = p.as_ptr().add(self.debug_offset - debug_len); - (p as *mut u32).write(STATE_DEAD); - } - } - } - if debug_enabled(DebugFlags::FillAllocations) { - if let Some(p) = self.ptr() { - // SAFETY: Writing to our inner base pointer with our known inner size is safe. - unsafe { (p.as_ptr() as *mut u8).write_bytes(0xde, self.size()) }; - } - } - if self.padding != 0 { - if let Some(p) = self.ptr() { - // SAFETY: Per the invariant, we have at least `self.padding` bytes trailing - // the inner base pointer, after `size()` bytes. - let guard = unsafe { - core::slice::from_raw_parts( - (p.as_ptr() as *mut u8 as *const u8).add(self.size()), - self.padding, - ) - }; - let mut first_err = None; - let mut last_err = 0; - for (i, p) in guard.iter().enumerate() { - if *p != (self.pad_word >> (8 * (i & 3))) as u8 { - if first_err.is_none() { - first_err = Some(i); - } - last_err = i; - } - } - if let Some(start) = first_err { - dev_warn!( - self.device().as_ref(), - "Allocator: Corruption after object of type {}/{:#x} at {:#x}:{:#x} + {:#x}..={:#x}\n", - core::any::type_name::(), - self.tag, - self.gpu_ptr(), - self.size(), - start, - last_err, - ); - } - } - } - } -} - -static_assert!(mem::size_of::() == 0x40); - -/// A trait representing an allocator. -pub(crate) trait Allocator { - /// The raw allocation type used by this allocator. - type Raw: RawAllocation; - // TODO: Needs associated_type_defaults - // type Allocation = GenericAlloc; - - /// Returns whether CPU-side mapping is enabled. - fn cpu_maps(&self) -> bool; - /// Returns the minimum alignment for allocations. - fn min_align(&self) -> usize; - /// Allocate an object of the given size in bytes with the given alignment. - fn alloc(&mut self, size: usize, align: usize) -> Result; - - /// Returns a tuple of (count, size) of how much garbage (freed but not yet reusable objects) - /// exists in this allocator. Optional. - fn garbage(&self) -> (usize, usize) { - (0, 0) - } - /// Collect garbage for this allocator, up to the given object count. Optional. - fn collect_garbage(&mut self, _count: usize) {} - - /// Allocate a new GpuStruct object. See [`GpuObject::new`]. - #[inline(never)] - fn new_object( - &mut self, - inner: T, - callback: impl for<'a> FnOnce(&'a T) -> T::Raw<'a>, - ) -> Result>> { - GpuObject::>::new(self.alloc_object()?, inner, callback) - } - - /// Allocate a new GpuStruct object. See [`GpuObject::new_default`]. - #[inline(never)] - fn new_default( - &mut self, - ) -> Result>> - where - for<'a> ::Raw<'a>: Default + Zeroable, - { - GpuObject::>::new_default(self.alloc_object()?) - } - - /// Allocate a new GpuStruct object. See [`GpuObject::new_init`]. - #[inline(never)] - fn new_init<'a, T: GpuStruct, R: PinInit, F>, E, F>( - &mut self, - inner_init: impl Init, - raw_init: impl FnOnce(&'a T, GpuWeakPointer) -> R, - ) -> Result>> - where - kernel::error::Error: core::convert::From, - kernel::error::Error: core::convert::From, - { - GpuObject::>::new_init_prealloc( - self.alloc_object()?, - |_p| inner_init, - raw_init, - ) - } - - /// Allocate a generic buffer of the given size and alignment, applying the debug features if - /// enabled to tag it and detect overflows. - fn alloc_generic( - &mut self, - size: usize, - align: usize, - tag: Option, - ) -> Result> { - let padding = if debug_enabled(DebugFlags::DetectOverflows) { - size - } else { - 0 - }; - - let ret: GenericAlloc = - if self.cpu_maps() && debug_enabled(debug::DebugFlags::DebugAllocations) { - let debug_align = self.min_align().max(align); - let debug_len = mem::size_of::(); - let debug_offset = (debug_len * 2 + debug_align - 1) & !(debug_align - 1); - - let alloc = self.alloc(size + debug_offset + padding, align)?; - - let mut debug = AllocDebugData { - state: STATE_LIVE, - tag: tag.unwrap_or(0), - size: size as u64, - base_gpuva: alloc.gpu_ptr(), - obj_gpuva: alloc.gpu_ptr() + debug_offset as u64, - name: [0; 0x20], - }; - - let name = core::any::type_name::().as_bytes(); - let len = name.len().min(debug.name.len() - 1); - debug.name[..len].copy_from_slice(&name[..len]); - - if let Some(p) = alloc.ptr() { - // SAFETY: Per the size calculations above, this pointer math and the - // writes never exceed the allocation size. - unsafe { - let p = p.as_ptr(); - p.write_bytes(0x42, debug_offset - 2 * debug_len); - let cur = p.add(debug_offset - debug_len) as *mut AllocDebugData; - let prev = p.add(debug_offset - 2 * debug_len) as *mut AllocDebugData; - prev.copy_from(cur, 1); - cur.copy_from(&debug, 1); - }; - } - - GenericAlloc { - alloc, - alloc_size: size, - debug_offset, - tag: tag.unwrap_or(0), - pad_word: tag.unwrap_or(GUARD_MARKER) | 0x81818181, - padding, - _p: PhantomData, - } - } else { - GenericAlloc { - alloc: self.alloc(size + padding, align)?, - alloc_size: size, - debug_offset: 0, - tag: tag.unwrap_or(0), - pad_word: tag.unwrap_or(GUARD_MARKER) | 0x81818181, - padding, - _p: PhantomData, - } - }; - - if debug_enabled(DebugFlags::FillAllocations) { - if let Some(p) = ret.ptr() { - // SAFETY: Writing to our inner base pointer with our known inner size is safe. - unsafe { (p.as_ptr() as *mut u8).write_bytes(0xaa, ret.size()) }; - } - } - - if padding != 0 { - if let Some(p) = ret.ptr() { - // SAFETY: Per the invariant, we have at least `self.padding` bytes trailing - // the inner base pointer, after `size()` bytes. - let guard = unsafe { - core::slice::from_raw_parts_mut( - (p.as_ptr() as *mut u8).add(ret.size()), - padding, - ) - }; - for (i, p) in guard.iter_mut().enumerate() { - *p = (ret.pad_word >> (8 * (i & 3))) as u8; - } - } - } - - Ok(ret) - } - - /// Allocate an object of a given type, without actually initializing the allocation. - /// - /// This is useful to directly call [`GpuObject::new_*`], without borrowing a reference to the - /// allocator for the entire duration (e.g. if further allocations need to happen inside the - /// callbacks). - fn alloc_object(&mut self) -> Result> { - let size = mem::size_of::>(); - let align = mem::align_of::>(); - - self.alloc_generic(size, align, None) - } - - /// Allocate an empty `GpuArray` of a given type and length. - fn array_empty( - &mut self, - count: usize, - ) -> Result>> { - let size = mem::size_of::() * count; - let align = mem::align_of::(); - - let alloc = self.alloc_generic(size, align, None)?; - GpuArray::>::empty(alloc, count) - } - - /// Allocate an empty `GpuArray` of a given type and length. - fn array_empty_tagged( - &mut self, - count: usize, - tag: &[u8; 4], - ) -> Result>> { - let size = mem::size_of::() * count; - let align = mem::align_of::(); - - let alloc = self.alloc_generic(size, align, Some(u32::from_le_bytes(*tag)))?; - GpuArray::>::empty(alloc, count) - } - - /// Allocate an empty `GpuOnlyArray` of a given type and length. - fn array_gpuonly( - &mut self, - count: usize, - ) -> Result>> { - let size = mem::size_of::() * count; - let align = mem::align_of::(); - - let alloc = self.alloc_generic(size, align, None)?; - GpuOnlyArray::>::new(alloc, count) - } -} - -/// A simple allocation backed by a separate GEM object. -/// -/// # Invariants -/// `ptr` is either None or a valid, non-null pointer to the CPU view of the object. -/// `gpu_ptr` is the GPU-side VA of the object. -pub(crate) struct SimpleAllocation { - dev: AsahiDevRef, - ptr: Option>, - gpu_ptr: u64, - _mapping: mmu::KernelMapping, - obj: crate::gem::ObjectRef, -} - -/// SAFETY: `SimpleAllocation` just points to raw memory and should be safe to send across threads. -unsafe impl Send for SimpleAllocation {} -/// SAFETY: `SimpleAllocation` just points to raw memory and should be safe to share across threads. -unsafe impl Sync for SimpleAllocation {} - -impl Drop for SimpleAllocation { - fn drop(&mut self) { - mod_dev_dbg!( - self.device(), - "SimpleAllocator: drop object @ {:#x}\n", - self.gpu_ptr() - ); - if debug_enabled(DebugFlags::FillAllocations) { - if let Ok(mut vmap) = self.obj.vmap() { - vmap.memset(0x42); - } - } - } -} - -impl RawAllocation for SimpleAllocation { - fn ptr(&self) -> Option> { - self.ptr - } - fn gpu_ptr(&self) -> u64 { - self.gpu_ptr - } - fn device(&self) -> &AsahiDevice { - &self.dev - } -} - -/// A simple allocator that allocates each object as its own GEM object, aligned to the end of a -/// page. -/// -/// This is very slow, but it has the advantage that over-reads by the firmware or GPU will fault on -/// the guard page after the allocation, which can be useful to validate that the firmware's or -/// GPU's idea of object size what we expect. -pub(crate) struct SimpleAllocator { - dev: AsahiDevRef, - range: Range, - prot: mmu::Prot, - vm: mmu::Vm, - min_align: usize, - cpu_maps: bool, -} - -impl SimpleAllocator { - /// Create a new `SimpleAllocator` for a given address range and `Vm`. - #[allow(dead_code)] - #[allow(clippy::too_many_arguments)] - pub(crate) fn new( - dev: &AsahiDevice, - vm: &mmu::Vm, - range: Range, - min_align: usize, - prot: mmu::Prot, - _block_size: usize, - mut cpu_maps: bool, - _name: fmt::Arguments<'_>, - _keep_garbage: bool, - ) -> Result { - if debug_enabled(DebugFlags::ForceCPUMaps) { - cpu_maps = true; - } - Ok(SimpleAllocator { - dev: dev.into(), - vm: vm.clone(), - range, - prot, - min_align, - cpu_maps, - }) - } -} - -impl Allocator for SimpleAllocator { - type Raw = SimpleAllocation; - - fn cpu_maps(&self) -> bool { - self.cpu_maps - } - - fn min_align(&self) -> usize { - self.min_align - } - - #[inline(never)] - fn alloc(&mut self, size: usize, align: usize) -> Result { - let size_aligned = (size + mmu::UAT_PGSZ - 1) & !mmu::UAT_PGMSK; - let align = self.min_align.max(align); - let offset = (size_aligned - size) & !(align - 1); - - mod_dev_dbg!( - &self.dev, - "SimpleAllocator::new: size={:#x} size_al={:#x} al={:#x} off={:#x}\n", - size, - size_aligned, - align, - offset - ); - - let mut obj = crate::gem::new_kernel_object(&self.dev, size_aligned)?; - let p = obj.vmap()?.as_mut_ptr() as *mut u8; - if debug_enabled(DebugFlags::FillAllocations) { - obj.vmap()?.memset(0xde); - } - let mapping = obj.map_into_range( - &self.vm, - self.range.clone(), - self.min_align.max(mmu::UAT_PGSZ) as u64, - self.prot, - true, - )?; - - let iova = mapping.iova(); - - // SAFETY: Per the math above to calculate `size_aligned`, this can never overflow. - let ptr = unsafe { p.add(offset) }; - let gpu_ptr = iova + offset as u64; - - mod_dev_dbg!( - &self.dev, - "SimpleAllocator::new -> {:#?} / {:#?} | {:#x} / {:#x}\n", - p, - ptr, - iova, - gpu_ptr - ); - - Ok(SimpleAllocation { - dev: self.dev.clone(), - ptr: NonNull::new(ptr), - gpu_ptr, - _mapping: mapping, - obj, - }) - } -} - -/// Inner data for an allocation from the heap allocator. -/// -/// This is wrapped in an `mm::Node`. -pub(crate) struct HeapAllocationInner { - dev: AsahiDevRef, - ptr: Option>, - real_size: usize, -} - -/// SAFETY: `HeapAllocationInner` just points to raw memory and should be safe to send across threads. -unsafe impl Send for HeapAllocationInner {} -/// SAFETY: `HeapAllocationInner` just points to raw memory and should be safe to share between threads. -unsafe impl Sync for HeapAllocationInner {} - -/// Outer view of a heap allocation. -/// -/// This uses an Option<> so we can move the internal `Node` into the garbage pool when it gets -/// dropped. -/// -/// # Invariants -/// The `Option` must always be `Some(...)` while this object is alive. -pub(crate) struct HeapAllocation(Option>); - -impl Drop for HeapAllocation { - fn drop(&mut self) { - let node = self.0.take().unwrap(); - let size = node.size(); - let alloc = node.alloc_ref(); - - alloc.with(|a| { - if let Some(garbage) = a.garbage.as_mut() { - if garbage.push(node, GFP_KERNEL).is_err() { - dev_err!( - &a.dev.as_ref(), - "HeapAllocation[{}]::drop: Failed to keep garbage\n", - &*a.name, - ); - } - a.total_garbage += size as usize; - None - } else { - // We need to ensure node survives this scope, since dropping it - // will try to take the mm lock and deadlock us - Some(node) - } - }); - } -} - -impl mm::AllocInner for HeapAllocatorInner { - fn drop_object( - &mut self, - start: u64, - _size: u64, - _color: usize, - obj: &mut HeapAllocationInner, - ) { - /* real_size == 0 means it's a guard node */ - if obj.real_size > 0 { - mod_dev_dbg!( - obj.dev, - "HeapAllocator[{}]: drop object @ {:#x} ({} bytes)\n", - &*self.name, - start, - obj.real_size, - ); - self.allocated -= obj.real_size; - } - } -} - -impl RawAllocation for HeapAllocation { - // SAFETY: This function must always return a valid pointer. - // Since the HeapAllocation contains a reference to the - // backing_objects array that contains the object backing this pointer, - // and objects are only ever added to it, this pointer is guaranteed to - // remain valid for the lifetime of the HeapAllocation. - fn ptr(&self) -> Option> { - self.0.as_ref().unwrap().ptr - } - // SAFETY: This function must always return a valid GPU pointer. - // See the explanation in ptr(). - fn gpu_ptr(&self) -> u64 { - self.0.as_ref().unwrap().start() - } - fn device(&self) -> &AsahiDevice { - &self.0.as_ref().unwrap().dev - } -} - -/// Inner data for a heap allocator which uses the DRM MM range allocator to manage the heap. -/// -/// This is wrapped by an `mm::Allocator`. -struct HeapAllocatorInner { - dev: AsahiDevRef, - allocated: usize, - backing_objects: KVec<(crate::gem::ObjectRef, mmu::KernelMapping, u64)>, - garbage: Option>>, - total_garbage: usize, - name: CString, -} - -/// A heap allocator which uses the DRM MM range allocator to manage its objects. -/// -/// The heap is composed of a series of GEM objects. This implementation only ever grows the heap, -/// never shrinks it. -pub(crate) struct HeapAllocator { - dev: AsahiDevRef, - range: Range, - top: u64, - prot: mmu::Prot, - vm: mmu::Vm, - min_align: usize, - block_size: usize, - cpu_maps: bool, - guard_nodes: KVec>, - mm: mm::Allocator, - name: CString, - garbage: Option>>, -} - -impl HeapAllocator { - /// Create a new HeapAllocator for a given `Vm` and address range. - #[allow(dead_code)] - #[allow(clippy::too_many_arguments)] - pub(crate) fn new( - dev: &AsahiDevice, - vm: &mmu::Vm, - range: Range, - min_align: usize, - prot: mmu::Prot, - block_size: usize, - mut cpu_maps: bool, - name: fmt::Arguments<'_>, - keep_garbage: bool, - ) -> Result { - if !min_align.is_power_of_two() { - return Err(EINVAL); - } - if debug_enabled(DebugFlags::ForceCPUMaps) { - cpu_maps = true; - } - - let name = CString::try_from_fmt(name)?; - - let inner = HeapAllocatorInner { - dev: dev.into(), - allocated: 0, - backing_objects: KVec::new(), - // TODO: This clearly needs a try_clone() or similar - name: CString::try_from_fmt(fmt!("{}", &*name))?, - garbage: if keep_garbage { - Some(KVec::new()) - } else { - None - }, - total_garbage: 0, - }; - - let mm = mm::Allocator::new(range.start, range.range(), inner)?; - - Ok(HeapAllocator { - dev: dev.into(), - vm: vm.clone(), - top: range.start, - range, - prot, - min_align, - block_size: block_size.max(min_align), - cpu_maps, - guard_nodes: KVec::new(), - mm, - name, - garbage: if keep_garbage { - Some({ - let mut v = KVec::new(); - v.reserve(128, GFP_KERNEL)?; - v - }) - } else { - None - }, - }) - } - - /// Add a new backing block of the given size to this heap. - /// - /// If CPU mapping is enabled, this also adds a guard node to the range allocator to ensure that - /// objects cannot straddle backing block boundaries, since we cannot easily create a contiguous - /// CPU VA mapping for them. This can create some fragmentation. If CPU mapping is disabled, we - /// skip the guard blocks, since the GPU view of the heap is always contiguous. - #[inline(never)] - fn add_block(&mut self, size: usize) -> Result { - let size_aligned = (size + mmu::UAT_PGSZ - 1) & !mmu::UAT_PGMSK; - - mod_dev_dbg!( - &self.dev, - "HeapAllocator[{}]::add_block: size={:#x} size_al={:#x}\n", - &*self.name, - size, - size_aligned, - ); - - if self.top.saturating_add(size_aligned as u64) > self.range.end { - dev_err!( - self.dev.as_ref(), - "HeapAllocator[{}]::add_block: Exhausted VA space\n", - &*self.name, - ); - } - - let mut obj = crate::gem::new_kernel_object(&self.dev, size_aligned)?; - if self.cpu_maps && debug_enabled(DebugFlags::FillAllocations) { - obj.vmap()?.memset(0xde); - } - - let gpu_ptr = self.top; - let mapping = obj - .map_at(&self.vm, gpu_ptr, self.prot, self.cpu_maps) - .inspect_err(|err| { - dev_err!( - self.dev.as_ref(), - "HeapAllocator[{}]::add_block: Failed to map at {:#x} ({:?})\n", - &*self.name, - gpu_ptr, - err - ); - })?; - - if self.cpu_maps { - // Create virtual mapping here ahead of time so that the vmap() in - // alloc_inner() does not take the the object's dma_resv lock while - // the mm lock is locked. mmu::Vm requires the opposite lock order. - obj.vmap()?; - } - - self.mm - .with_inner(|inner| inner.backing_objects.reserve(1, GFP_KERNEL))?; - - let mut new_top = self.top + size_aligned as u64; - if self.cpu_maps { - let guard = self.min_align.max(mmu::UAT_PGSZ); - mod_dev_dbg!( - self.dev, - "HeapAllocator[{}]::add_block: Adding guard node {:#x}:{:#x}\n", - &*self.name, - new_top, - guard - ); - - let inner = HeapAllocationInner { - dev: self.dev.clone(), - ptr: None, - real_size: 0, - }; - - let node = match self.mm.reserve_node(inner, new_top, guard as u64, 0) { - Ok(a) => a, - Err(a) => { - dev_err!( - self.dev.as_ref(), - "HeapAllocator[{}]::add_block: Failed to reserve guard node {:#x}:{:#x}: {:?}\n", - &*self.name, - guard, - new_top, - a - ); - return Err(EIO); - } - }; - - self.guard_nodes.push(node, GFP_KERNEL)?; - - new_top += guard as u64; - } - mod_dev_dbg!( - &self.dev, - "HeapAllocator[{}]::add_block: top={:#x}\n", - &*self.name, - new_top - ); - - self.mm.with_inner(|inner| { - inner - .backing_objects - .push((obj, mapping, gpu_ptr), GFP_KERNEL) - })?; - - self.top = new_top; - - cls_dev_dbg!( - MemStats, - &self.dev, - "{} Heap: grow to {} bytes\n", - &*self.name, - self.top - self.range.start - ); - - Ok(()) - } - - /// Find the backing object index that backs a given GPU address. - fn find_obj(&mut self, addr: u64) -> Result { - self.mm.with_inner(|inner| { - inner - .backing_objects - .binary_search_by(|obj| { - let start = obj.2; - let end = obj.2 + obj.0.size() as u64; - if start > addr { - Ordering::Greater - } else if end <= addr { - Ordering::Less - } else { - Ordering::Equal - } - }) - .or(Err(ENOENT)) - }) - } - - fn alloc_inner(&mut self, size: usize, align: usize) -> Result { - if align != 0 && !align.is_power_of_two() { - return Err(EINVAL); - } - let align = self.min_align.max(align); - let size_aligned = (size + align - 1) & !(align - 1); - - mod_dev_dbg!( - &self.dev, - "HeapAllocator[{}]::new: size={:#x} size_al={:#x}\n", - &*self.name, - size, - size_aligned, - ); - - let inner = HeapAllocationInner { - dev: self.dev.clone(), - ptr: None, - real_size: size, - }; - - let mut node = match self.mm.insert_node_generic( - inner, - size_aligned as u64, - align as u64, - 0, - mm::InsertMode::Best, - ) { - Ok(a) => a, - Err(a) => { - dev_err!( - &self.dev.as_ref(), - "HeapAllocator[{}]::new: Failed to insert node of size {:#x} / align {:#x}: {:?}\n", - &*self.name, size_aligned, align, a - ); - return Err(a); - } - }; - - self.mm.with_inner(|inner| inner.allocated += size); - - let mut new_object = false; - let start = node.start(); - let end = start + node.size(); - if end > self.top { - if start > self.top { - dev_warn!( - self.dev.as_ref(), - "HeapAllocator[{}]::alloc: top={:#x}, start={:#x}\n", - &*self.name, - self.top, - start - ); - } - let block_size = self.block_size.max((end - self.top) as usize); - self.add_block(block_size)?; - new_object = true; - } - assert!(end <= self.top); - - if self.cpu_maps { - mod_dev_dbg!( - self.dev, - "HeapAllocator[{}]::alloc: mapping to CPU\n", - &*self.name - ); - - let idx = if new_object { - None - } else { - Some(match self.find_obj(start) { - Ok(a) => a, - Err(_) => { - dev_warn!( - self.dev.as_ref(), - "HeapAllocator[{}]::alloc: Failed to find object at {:#x}\n", - &*self.name, - start - ); - return Err(EIO); - } - }) - }; - let (obj_start, obj_size, p) = self.mm.with_inner(|inner| -> Result<_> { - let idx = idx.unwrap_or(inner.backing_objects.len() - 1); - let obj = &mut inner.backing_objects[idx]; - let p = obj.0.vmap()?.as_mut_ptr() as *mut u8; - Ok((obj.2, obj.0.size(), p)) - })?; - assert!(obj_start <= start); - assert!(obj_start + obj_size as u64 >= end); - node.as_mut().inner_mut().ptr = - // SAFETY: Per the asserts above, this offset is always within the allocation. - NonNull::new(unsafe { p.add((start - obj_start) as usize) }); - mod_dev_dbg!( - self.dev, - "HeapAllocator[{}]::alloc: CPU pointer = {:?}\n", - &*self.name, - node.ptr - ); - } - - mod_dev_dbg!( - self.dev, - "HeapAllocator[{}]::alloc: Allocated {:#x} bytes @ {:#x}\n", - &*self.name, - end - start, - start - ); - - Ok(HeapAllocation(Some(node))) - } -} - -impl Allocator for HeapAllocator { - type Raw = HeapAllocation; - - fn cpu_maps(&self) -> bool { - self.cpu_maps - } - - fn min_align(&self) -> usize { - self.min_align - } - - fn alloc(&mut self, size: usize, align: usize) -> Result { - let ret = self.alloc_inner(size, align); - - if ret.is_err() { - dev_warn!( - self.dev.as_ref(), - "HeapAllocator[{}]::alloc: Allocation of {:#x}({:#x}) size object failed\n", - &*self.name, - size, - align - ); - } - ret - } - - fn garbage(&self) -> (usize, usize) { - self.mm.with_inner(|inner| { - if let Some(g) = inner.garbage.as_ref() { - (g.len(), inner.total_garbage) - } else { - (0, 0) - } - }) - } - - fn collect_garbage(&mut self, mut count: usize) { - if let Some(garbage) = self.garbage.as_mut() { - garbage.clear(); - - while count > 0 { - let block = count.min(garbage.capacity()); - assert!(block > 0); - - // Take the garbage out of the inner block, so we can safely drop it without deadlocking - self.mm.with_inner(|inner| { - if let Some(g) = inner.garbage.as_mut() { - for node in g.drain(0..block) { - inner.total_garbage -= node.size() as usize; - garbage - .push(node, GFP_KERNEL) - .expect("push() failed after reserve()"); - } - } - }); - - count -= block; - // Now drop it - garbage.clear(); - } - } - } -} - -impl Drop for HeapAllocatorInner { - fn drop(&mut self) { - mod_dev_dbg!( - self.dev, - "HeapAllocator[{}]: dropping allocator\n", - &*self.name - ); - if self.allocated > 0 { - // This should never happen - dev_crit!( - self.dev.as_ref(), - "HeapAllocator[{}]: dropping with {} bytes allocated\n", - &*self.name, - self.allocated - ); - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/asahi_core.rs b/drivers/gpu/drm/asahi-neo/asahi_core.rs deleted file mode 100644 index 483d1ea2b17d09..00000000000000 --- a/drivers/gpu/drm/asahi-neo/asahi_core.rs +++ /dev/null @@ -1,93 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![recursion_limit = "2048"] - -//! Driver for the Apple AGX GPUs found in Apple Silicon SoCs. - -mod alloc; -mod buffer; -mod channel; -mod cleanup; -#[cfg(CONFIG_DEV_COREDUMP)] -mod crashdump; -mod debug; -mod driver; -mod event; -mod file; -mod float; -mod fw; -mod gem; -mod g17; -mod gpu; -mod hw; -mod initdata; -mod mem; -mod microseq; -mod mmu; -mod object; -mod pgtable; -mod queue; -mod regs; -mod slotalloc; -mod stats; -mod sysfs_exports; -mod util; -#[cfg(CONFIG_DRM_ASAHI_NEO_MAPLE_TREE)] -mod vm; -mod workqueue; - -use kernel::prelude::*; - -#[pin_data] -struct DriverModule { - // Field drop order unregisters/joins driver producers before queue teardown. - #[pin] - _driver: kernel::driver::Registration>, - _cleanup: cleanup::QueueOwner, -} - -impl kernel::InPlaceModule for DriverModule { - fn init(module: &'static kernel::ThisModule) -> impl PinInit { - try_pin_init!(Self { - // Initialize the queue first, before registration can invoke probe. - _cleanup: cleanup::QueueOwner::new()?, - _driver <- kernel::driver::Registration::new( - ::NAME, - module, - ), - }) - } -} - -module! { - type: DriverModule, - name: "asahi_neo", - description: "G17 GPU driver for Apple T8140", - license: "Dual MIT/GPL", - params: { - debug_flags: u64 { - default: 0, - // permissions: 0o644, - description: "Debug flags", - }, - fault_control: u32 { - default: 0xb, - // permissions: 0, - description: "Fault control (0x0: hard faults, 0xb: macOS default)", - }, - initial_tvb_size: usize { - default: 0x8, - // permissions: 0o644, - description: "Initial TVB size in blocks", - }, - robust_isolation: u32 { - default: 0, - // permissions: 0o644, - description: "Fully isolate GPU contexts (limits performance)", - }, - stats_export: u8 { - default: 1, - // permissions: 0, - description: "Export AGX firmware stats to sysfs (1 = on, 0 = off)", - }, - }, -} diff --git a/drivers/gpu/drm/asahi-neo/buffer.rs b/drivers/gpu/drm/asahi-neo/buffer.rs deleted file mode 100644 index ba3823098af645..00000000000000 --- a/drivers/gpu/drm/asahi-neo/buffer.rs +++ /dev/null @@ -1,809 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Tiled Vertex Buffer management -//! -//! This module manages the Tiled Vertex Buffer, also known as the Parameter Buffer (in imgtec -//! parlance) or the tiler heap (on other architectures). This buffer holds transformed primitive -//! data between the vertex/tiling stage and the fragment stage. -//! -//! On AGX, the buffer is a heap of 128K blocks split into 32K pages (which must be aligned to a -//! multiple of 32K in VA space). The buffer can be shared between multiple render jobs, and each -//! will allocate pages from it during vertex processing and return them during fragment processing. -//! -//! If the buffer runs out of free pages, the vertex pass stops and a partial fragment pass occurs, -//! spilling the intermediate render target state to RAM (a partial render). This is all managed -//! transparently by the firmware. Since partial renders are less efficient, the kernel must grow -//! the heap in response to feedback from the firmware to avoid partial renders in the future. -//! Currently, we only ever grow the heap, and never shrink it. -//! -//! AGX also supports memoryless render targets, which can be used for intermediate results within -//! a render pass. To support partial renders, it seems the GPU/firmware has the ability to borrow -//! pages from the TVB buffer as a temporary render target buffer. Since this happens during a -//! partial render itself, if the buffer runs out of space, it requires synchronous growth in -//! response to a firmware interrupt. This is not currently supported, but may be in the future, -//! though it is unclear whether it is worth the effort. -//! -//! This module is also in charge of managing the temporary objects associated with a single render -//! pass, which includes the top-level tile array, the tail pointer cache, preemption buffers, and -//! other miscellaneous structures collectively managed as a "scene". -//! -//! To avoid runaway memory usage, there is a maximum size for buffers (at that point it's unlikely -//! that partial renders will incur much overhead over the buffer data access itself). This is -//! different depending on whether memoryless render targets are in use, and is currently hardcoded. -//! to the most common value used by macOS. - -use crate::debug::*; -use crate::fw::buffer; -use crate::fw::types::*; -use crate::util::*; -use crate::{ - alloc, - fw, - gpu, - hw, - mmu, - slotalloc, // -}; -use core::sync::atomic::Ordering; -use kernel::new_mutex; -use kernel::prelude::*; -use kernel::sync::{ - Arc, - Mutex, // -}; -use kernel::{ - c_str, - static_lock_class, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Buffer; - -/// There are 127 GPU/firmware-side buffer manager slots (yes, 127, not 128). -const NUM_BUFFERS: u32 = 127; - -/// Page size bits for buffer pages (32K). VAs must be aligned to this size. -pub(crate) const PAGE_SHIFT: usize = 15; -/// Page size for buffer pages. -pub(crate) const PAGE_SIZE: usize = 1 << PAGE_SHIFT; -/// Number of pages in a buffer block, which should be contiguous in VA space. -pub(crate) const PAGES_PER_BLOCK: usize = 4; -/// Size of a buffer block. -pub(crate) const BLOCK_SIZE: usize = PAGE_SIZE * PAGES_PER_BLOCK; - -/// Metadata about the tiling configuration for a scene. This is computed in the `render` module. -/// based on dimensions, tile size, and other info. -pub(crate) struct TileInfo { - /// Tile count in the X dimension. Tiles are always 32x32. - pub(crate) tiles_x: u32, - /// Tile count in the Y dimension. Tiles are always 32x32. - pub(crate) tiles_y: u32, - /// Total tile count. - pub(crate) tiles: u32, - /// Micro-tile width (16 or 32). - pub(crate) utile_width: u32, - /// Micro-tile height (16 or 32). - pub(crate) utile_height: u32, - // Macro-tiles in the X dimension. Always 4. - //pub(crate) mtiles_x: u32, - // Macro-tiles in the Y dimension. Always 4. - //pub(crate) mtiles_y: u32, - /// Tiles per macro-tile in the X dimension. - pub(crate) tiles_per_mtile_x: u32, - /// Tiles per macro-tile in the Y dimension. - pub(crate) tiles_per_mtile_y: u32, - // Total tiles per macro-tile. - //pub(crate) tiles_per_mtile: u32, - /// Micro-tiles per macro-tile in the X dimension. - pub(crate) utiles_per_mtile_x: u32, - /// Micro-tiles per macro-tile in the Y dimension. - pub(crate) utiles_per_mtile_y: u32, - // Total micro-tiles per macro-tile. - //pub(crate) utiles_per_mtile: u32, - /// Size of the top-level tilemap, in bytes (for all layers, one cluster). - pub(crate) tilemap_size: usize, - /// Size of the Tail Pointer Cache, in bytes (for all layers * clusters). - pub(crate) tpc_size: usize, - /// Number of blocks in the clustering meta buffer (for clustering) per layer. - pub(crate) meta1_layer_stride: u32, - /// Number of blocks in the clustering meta buffer (for clustering). - pub(crate) meta1_blocks: u32, - /// Layering metadata size. - pub(crate) layermeta_size: usize, - /// Minimum number of TVB blocks for this render. - pub(crate) min_tvb_blocks: usize, - /// Tiling parameter structure passed to firmware. - pub(crate) params: fw::vertex::raw::TilingParameters, -} - -/// A single scene, representing a render pass and its required buffers. -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct Scene { - object: GpuObject, - slot: u32, - rebind: bool, - preempt2_off: usize, - preempt3_off: usize, - // Note: these are dead code only on some version variants. - // It's easier to do this than to propagate the version conditionals everywhere. - #[allow(dead_code)] - meta1_off: usize, - #[allow(dead_code)] - meta2_off: usize, - #[allow(dead_code)] - meta3_off: usize, - #[allow(dead_code)] - meta4_off: usize, -} - -#[versions(AGX_NEO)] -impl Scene::ver { - /// Returns true if the buffer was bound to a fresh manager slot, and therefore needs an init - /// command before a render. - pub(crate) fn rebind(&self) -> bool { - self.rebind - } - - /// Returns the buffer manager slot this scene's buffer was bound to. - pub(crate) fn slot(&self) -> u32 { - self.slot - } - - /// Returns the GPU pointer to the [`buffer::Scene::ver`]. - pub(crate) fn gpu_pointer(&self) -> GpuPointer<'_, buffer::Scene::ver> { - self.object.gpu_pointer() - } - - /// Returns the GPU weak pointer to the [`buffer::Scene::ver`]. - pub(crate) fn weak_pointer(&self) -> GpuWeakPointer { - self.object.weak_pointer() - } - - /// Returns the GPU weak pointer to the kernel-side temp buffer. - /// (purpose unknown...) - pub(crate) fn kernel_buffer_pointer(&self) -> GpuWeakPointer<[u8]> { - self.object.buffer.inner.lock().kernel_buffer.weak_pointer() - } - - /// Returns the GPU pointer to the `buffer::Info::ver` object associated with this Scene. - pub(crate) fn buffer_pointer(&self) -> GpuPointer<'_, buffer::Info::ver> { - // SAFETY: We can't return the strong pointer directly since its lifetime crosses a lock, - // but we know its lifetime will be valid as long as &self since we hold a reference to the - // buffer, so just construct the strong pointer with the right lifetime here. - unsafe { self.weak_buffer_pointer().upgrade() } - } - - /// Returns the GPU weak pointer to the `buffer::Info::ver` object associated with this Scene. - pub(crate) fn weak_buffer_pointer(&self) -> GpuWeakPointer { - self.object.buffer.inner.lock().info.weak_pointer() - } - - /// Returns the GPU pointer to the TVB heap metadata buffer. - pub(crate) fn tvb_heapmeta_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object.tvb_heapmeta.gpu_pointer() - } - - /// Returns the GPU pointer to the layer metadata buffer. - pub(crate) fn tvb_layermeta_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object.tvb_heapmeta.gpu_offset_pointer(0x200) - } - - /// Returns the GPU pointer to the top-level TVB tilemap buffer. - pub(crate) fn tvb_tilemap_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object.tvb_tilemap.gpu_pointer() - } - - /// Returns the GPU pointer to the Tail Pointer Cache buffer. - pub(crate) fn tpc_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object.tpc.gpu_pointer() - } - - /// Returns the GPU pointer to the first preemption scratch buffer. - pub(crate) fn preempt_buf_1_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object.preempt_buf.gpu_pointer() - } - - /// Returns the GPU pointer to the second preemption scratch buffer. - pub(crate) fn preempt_buf_2_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object - .preempt_buf - .gpu_offset_pointer(self.preempt2_off) - } - - /// Returns the GPU pointer to the third preemption scratch buffer. - pub(crate) fn preempt_buf_3_pointer(&self) -> GpuPointer<'_, &'_ [u8]> { - self.object - .preempt_buf - .gpu_offset_pointer(self.preempt3_off) - } - - /// Returns the GPU pointer to the per-cluster tilemap buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn cluster_tilemaps_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.tilemaps.gpu_pointer()) - } - - /// Returns the GPU pointer to the clustering layer metadata buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn tvb_cluster_layermeta_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.meta.gpu_pointer()) - } - - /// Returns the GPU pointer to the clustering metadata 1 buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn meta_1_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.meta.gpu_offset_pointer(self.meta1_off)) - } - - /// Returns the GPU pointer to the clustering metadata 2 buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn meta_2_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.meta.gpu_offset_pointer(self.meta2_off)) - } - - /// Returns the GPU pointer to the clustering metadata 3 buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn meta_3_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.meta.gpu_offset_pointer(self.meta3_off)) - } - - /// Returns the GPU pointer to the clustering metadata 4 buffer, if clustering is enabled. - #[allow(dead_code)] - pub(crate) fn meta_4_pointer(&self) -> Option> { - self.object - .clustering - .as_ref() - .map(|c| c.meta.gpu_offset_pointer(self.meta4_off)) - } -} - -#[versions(AGX_NEO)] -impl Drop for Scene::ver { - fn drop(&mut self) { - let mut inner = self.object.buffer.inner.lock(); - assert_ne!(inner.active_scenes, 0); - inner.active_scenes -= 1; - - if inner.active_scenes == 0 { - mod_pr_debug!( - "Buffer: no scenes left, dropping slot {}", - inner.active_slot.take().unwrap().slot() - ); - inner.active_slot = None; - } - } -} - -/// Inner data for a single TVB buffer object. -#[versions(AGX_NEO)] -struct BufferInner { - info: GpuObject, - ualloc: Arc>, - ualloc_priv: Arc>, - blocks: KVec>, - max_blocks: usize, - max_blocks_nomemless: usize, - mgr: BufferManager::ver, - active_scenes: usize, - active_slot: Option>, - last_token: Option, - tpc: Option>>, - kernel_buffer: GpuArray, - stats: GpuObject, - cfg: &'static hw::HwConfig, - preempt1_size: usize, - preempt2_size: usize, - preempt3_size: usize, - num_clusters: usize, -} - -/// Locked and reference counted TVB buffer. -#[versions(AGX_NEO)] -pub(crate) struct Buffer { - inner: Arc>, -} - -#[versions(AGX_NEO)] -impl Buffer::ver { - /// Create a new Buffer for a given VM, given the per-VM allocators. - pub(crate) fn new( - gpu: &dyn gpu::GpuManager, - alloc: &mut gpu::KernelAllocators, - ualloc: Arc>, - ualloc_priv: Arc>, - mgr: &BufferManager::ver, - ) -> Result { - // These are the typical max numbers on macOS. - // 8GB machines have this halved. - let max_size: usize = 862_322_688; // bytes - let max_size_nomemless = max_size / 3; - - let max_blocks = max_size / BLOCK_SIZE; - let max_blocks_nomemless = max_size_nomemless / BLOCK_SIZE; - let max_pages = max_blocks * PAGES_PER_BLOCK; - let max_pages_nomemless = max_blocks_nomemless * PAGES_PER_BLOCK; - - let num_clusters = gpu.get_dyncfg().id.num_clusters as usize; - let num_clusters_adj = if num_clusters > 1 { - num_clusters + 1 - } else { - 1 - }; - - let preempt1_size = num_clusters_adj * gpu.get_cfg().preempt1_size; - let preempt2_size = num_clusters_adj * gpu.get_cfg().preempt2_size; - let preempt3_size = num_clusters_adj * gpu.get_cfg().preempt3_size; - - let shared = &mut alloc.shared; - let info = alloc.private.new_init( - { - let ualloc_priv = &ualloc_priv; - try_init!(buffer::Info::ver { - block_ctl: shared.new_default::()?, - counter: shared.new_default::()?, - page_list: ualloc_priv.lock().array_empty_tagged(max_pages, b"PLST")?, - block_list: ualloc_priv - .lock() - .array_empty_tagged(max_blocks * 2, b"BLST")?, - }) - }, - |inner, _p| { - try_init!(buffer::raw::Info::ver { - gpu_counter: 0x0, - unk_4: 0, - last_id: 0x0, - cur_id: -1, - unk_10: 0x0, - gpu_counter2: 0x0, - unk_18: 0x0, - #[ver(V < V13_0B4 || G >= G14X)] - unk_1c: 0x0, - page_list: inner.page_list.gpu_pointer(), - page_list_size: (4 * max_pages).try_into()?, - page_count: AtomicU32::new(0), - max_blocks: max_blocks.try_into()?, - block_count: AtomicU32::new(0), - unk_38: 0x0, - block_list: inner.block_list.gpu_pointer(), - block_ctl: inner.block_ctl.gpu_pointer(), - last_page: AtomicU32::new(0), - gpu_page_ptr1: 0x0, - gpu_page_ptr2: 0x0, - unk_58: 0x0, - block_size: BLOCK_SIZE as u32, - unk_60: U64(0x0), - counter: inner.counter.gpu_pointer(), - unk_70: 0x0, - unk_74: 0x0, - unk_78: 0x0, - unk_7c: 0x0, - unk_80: 0x1, - max_pages: max_pages.try_into()?, - max_pages_nomemless: max_pages_nomemless.try_into()?, - unk_8c: 0x0, - unk_90: Default::default(), - }) - }, - )?; - - // Technically similar to Scene below, let's play it safe. - let kernel_buffer = alloc.shared.array_empty_tagged(0x40, b"KBUF")?; - let stats = alloc - .shared - .new_object(Default::default(), |_inner| buffer::raw::Stats { - reset: AtomicU32::from(1), - ..Default::default() - })?; - - Ok(Buffer::ver { - inner: Arc::pin_init( - new_mutex!(BufferInner::ver { - info, - ualloc, - ualloc_priv, - blocks: KVec::new(), - max_blocks, - max_blocks_nomemless, - mgr: mgr.clone(), - active_scenes: 0, - active_slot: None, - last_token: None, - tpc: None, - kernel_buffer, - stats, - cfg: gpu.get_cfg(), - preempt1_size, - preempt2_size, - preempt3_size, - num_clusters, - }), - GFP_KERNEL, - )?, - }) - } - - /// Returns the total block count allocated to this Buffer. - pub(crate) fn block_count(&self) -> u32 { - self.inner.lock().blocks.len() as u32 - } - - /// Automatically grow the Buffer based on feedback from the statistics. - pub(crate) fn auto_grow(&self) -> Result { - let inner = self.inner.lock(); - - let used_pages = inner.stats.with(|raw, _inner| { - let used = raw.max_pages.load(Ordering::Relaxed); - raw.reset.store(1, Ordering::Release); - used as usize - }); - - let need_blocks = (used_pages * 2) - .div_ceil(PAGES_PER_BLOCK) - .min(inner.max_blocks_nomemless); - let want_blocks = (used_pages * 3) - .div_ceil(PAGES_PER_BLOCK) - .min(inner.max_blocks_nomemless); - - let cur_count = inner.blocks.len(); - - if need_blocks <= cur_count { - Ok(false) - } else { - // Grow to 3x requested size (same logic as macOS) - core::mem::drop(inner); - self.ensure_blocks(want_blocks)?; - Ok(true) - } - } - - /// Synchronously grow the Buffer. - pub(crate) fn sync_grow(&self) { - let inner = self.inner.lock(); - - let cur_count = inner.blocks.len(); - core::mem::drop(inner); - if self.ensure_blocks(cur_count + 10).is_err() { - pr_err!("BufferManager: Failed to grow buffer synchronously\n"); - } - } - - /// Ensure that the buffer has at least a certain minimum size in blocks. - pub(crate) fn ensure_blocks(&self, min_blocks: usize) -> Result { - let mut inner = self.inner.lock(); - - let cur_count = inner.blocks.len(); - if cur_count >= min_blocks { - return Ok(false); - } - if min_blocks > inner.max_blocks { - return Err(ENOMEM); - } - - let add_blocks = min_blocks - cur_count; - let new_count = min_blocks; - - let mut new_blocks: KVec> = KVec::new(); - - // Allocate the new blocks first, so if it fails they will be dropped - let mut ualloc = inner.ualloc.lock(); - for _i in 0..add_blocks { - new_blocks.push(ualloc.array_gpuonly(BLOCK_SIZE)?, GFP_KERNEL)?; - } - core::mem::drop(ualloc); - - // Then actually commit them - inner.blocks.reserve(add_blocks, GFP_KERNEL)?; - - for (i, block) in new_blocks.into_iter().enumerate() { - let page_num = (block.gpu_va().get() >> PAGE_SHIFT) as u32; - - inner - .blocks - .push(block, GFP_KERNEL) - .expect("push() failed after reserve()"); - inner.info.block_list[2 * (cur_count + i)] = page_num; - for j in 0..PAGES_PER_BLOCK { - inner.info.page_list[(cur_count + i) * PAGES_PER_BLOCK + j] = page_num + j as u32; - } - } - - inner.info.block_ctl.with(|raw, _inner| { - raw.total.store(new_count as u32, Ordering::SeqCst); - raw.wptr.store(new_count as u32, Ordering::SeqCst); - }); - - /* Only do this update if the buffer manager is idle (which means we own it) */ - if inner.active_scenes == 0 { - let page_count = (new_count * PAGES_PER_BLOCK) as u32; - inner.info.with(|raw, _inner| { - raw.page_count.store(page_count, Ordering::Relaxed); - raw.block_count.store(new_count as u32, Ordering::Relaxed); - raw.last_page.store(page_count - 1, Ordering::Relaxed); - }); - } - - Ok(true) - } - - /// Create a new [`Scene::ver`] (render pass) using this buffer. - pub(crate) fn new_scene( - &self, - alloc: &mut gpu::KernelAllocators, - tile_info: &TileInfo, - ) -> Result { - let mut inner = self.inner.lock(); - - let tilemap_size = tile_info.tilemap_size; - let tpc_size = tile_info.tpc_size; - - // TODO: what is this exactly? - mod_pr_debug!("Buffer: Allocating TVB buffers\n"); - - // This seems to be a list, with 4x2 bytes of headers and 8 bytes per entry. - // On single-cluster devices, the used length always seems to be 1. - // On M1 Ultra, it can grow and usually doesn't exceed 64 entries. - // macOS allocates a whole 64K * 0x80 for this, so let's go with - // that to be safe... - let user_buffer = inner.ualloc.lock().array_empty_tagged( - if inner.num_clusters > 1 { - 0x10080 - } else { - 0x80 - }, - b"UBUF", - )?; - - let tvb_heapmeta = inner - .ualloc - .lock() - .array_empty_tagged(0x200 + tile_info.layermeta_size, b"HMTA")?; - let tvb_tilemap = inner - .ualloc - .lock() - .array_empty_tagged(tilemap_size, b"TMAP")?; - - mod_pr_debug!("Buffer: Allocating misc buffers\n"); - let preempt_buf = inner.ualloc.lock().array_empty_tagged( - inner.preempt1_size + inner.preempt2_size + inner.preempt3_size, - b"PRMT", - )?; - - let tpc = match inner.tpc.as_ref() { - Some(buf) if buf.len() >= tpc_size => buf.clone(), - _ => { - // MacOS allocates this as shared GPU+FW, but - // priv seems to work and might be faster? - // Needs to be FW-writable anyway, so ualloc - // won't work. - let buf = Arc::new( - inner.ualloc_priv.lock().array_empty_tagged( - (tpc_size + mmu::UAT_PGMSK) & !mmu::UAT_PGMSK, - b"TPC ", - )?, - GFP_KERNEL, - )?; - inner.tpc = Some(buf.clone()); - buf - } - }; - - let mut clmeta_size = 0; - let mut meta1_size = 0; - let mut meta2_size = 0; - let mut meta3_size = 0; - - let clustering = if inner.num_clusters > 1 { - let cfg = inner.cfg.clustering.as_ref().unwrap(); - - clmeta_size = tile_info.layermeta_size * cfg.max_splits; - // Maybe: (4x4 macro tiles + 1 global page)*n, 32bit each (17*4*n) - // Unused on t602x? - meta1_size = align(tile_info.meta1_blocks as usize * cfg.meta1_blocksize, 0x80); - meta2_size = align(cfg.meta2_size, 0x80); - meta3_size = align(cfg.meta3_size, 0x80); - let meta4_size = cfg.meta4_size; - - let meta_size = clmeta_size + meta1_size + meta2_size + meta3_size + meta4_size; - - mod_pr_debug!("Buffer: Allocating clustering buffers\n"); - let tilemaps = inner - .ualloc - .lock() - .array_empty_tagged(cfg.max_splits * tilemap_size, b"CTMP")?; - let meta = inner.ualloc.lock().array_empty_tagged(meta_size, b"CMTA")?; - Some(buffer::ClusterBuffers { tilemaps, meta }) - } else { - None - }; - - // Could be made strong, but we wind up with a deadlock if we try to grab the - // pointer through the inner.buffer path inside the closure. - let stats_pointer = inner.stats.weak_pointer(); - - let _gpu = &mut alloc.gpu; - - // macOS allocates this as private. However, the firmware does not - // DC CIVAC this before reading it (like it does most other things), - // which causes odd cache incoherency bugs when combined with - // speculation on the firmware side (maybe). This doesn't happen - // on macOS because these structs are a circular pool that is mapped - // already initialized. Just mark this shared for now. - let scene = alloc.shared.new_init( - try_init!(buffer::Scene::ver { - user_buffer: user_buffer, - buffer: self.clone(), - tvb_heapmeta: tvb_heapmeta, - tvb_tilemap: tvb_tilemap, - tpc: tpc, - clustering: clustering, - preempt_buf: preempt_buf, - #[ver(G >= G14X)] - control_word: _gpu.array_empty_tagged(1, b"CWRD")?, - }), - |inner, _p| { - try_init!(buffer::raw::Scene::ver { - #[ver(G >= G14X)] - control_word: inner.control_word.gpu_pointer(), - #[ver(G >= G14X)] - control_word2: inner.control_word.gpu_pointer(), - pass_page_count: AtomicU32::new(0), - unk_4: 0, - unk_8: U64(0), - unk_10: U64(0), - user_buffer: inner.user_buffer.gpu_pointer(), - unk_20: 0, - #[ver(V >= V13_3)] - unk_28: U64(0), - stats: stats_pointer, - total_page_count: AtomicU32::new(0), - #[ver(G < G14X)] - unk_30: U64(0), - #[ver(G < G14X)] - unk_38: U64(0), - }) - }, - )?; - - let mut rebind = false; - - if inner.active_slot.is_none() { - assert_eq!(inner.active_scenes, 0); - - let slot = inner.mgr.0.get_inner(inner.last_token, |inner, mgr| { - inner.owners[mgr.slot() as usize] = Some(self.clone()); - Ok(()) - })?; - rebind = slot.changed(); - - mod_pr_debug!("Buffer: assigning slot {} (rebind={})", slot.slot(), rebind); - - inner.last_token = Some(slot.token()); - inner.active_slot = Some(slot); - } - - inner.active_scenes += 1; - - Ok(Scene::ver { - object: scene, - slot: inner.active_slot.as_ref().unwrap().slot(), - rebind, - preempt2_off: inner.preempt1_size, - preempt3_off: inner.preempt1_size + inner.preempt2_size, - meta1_off: clmeta_size, - meta2_off: clmeta_size + meta1_size, - meta3_off: clmeta_size + meta1_size + meta2_size, - meta4_off: clmeta_size + meta1_size + meta2_size + meta3_size, - }) - } - - /// Increment the buffer manager usage count. Should we done once we know the Scene is ready - /// to be committed and used in commands submitted to the GPU. - pub(crate) fn increment(&self) { - let inner = self.inner.lock(); - inner.info.counter.with(|raw, _inner| { - // We could use fetch_add, but the non-LSE atomic - // sequence Rust produces confuses the hypervisor. - // We have inner locked anyway, so this is not racy. - let v = raw.count.load(Ordering::Relaxed); - raw.count.store(v + 1, Ordering::Relaxed); - }); - } - - pub(crate) fn any_ref(&self) -> Arc { - self.inner.clone() - } -} - -#[versions(AGX_NEO)] -impl Clone for Buffer::ver { - fn clone(&self) -> Self { - Buffer::ver { - inner: self.inner.clone(), - } - } -} - -#[versions(AGX_NEO)] -struct BufferSlotInner(); - -#[versions(AGX_NEO)] -impl slotalloc::SlotItem for BufferSlotInner::ver { - type Data = BufferManagerInner::ver; - - fn release(&mut self, data: &mut Self::Data, slot: u32) { - mod_pr_debug!("BufferManager: Released slot {}\n", slot); - data.owners[slot as usize] = None; - } -} - -/// Inner data for the buffer manager, to be protected by the SlotAllocator lock. -#[versions(AGX_NEO)] -pub(crate) struct BufferManagerInner { - owners: KVec>, -} - -/// The GPU-global buffer manager, used to allocate and release buffer slots from the pool. -#[versions(AGX_NEO)] -pub(crate) struct BufferManager(slotalloc::SlotAllocator); - -#[versions(AGX_NEO)] -impl BufferManager::ver { - pub(crate) fn new() -> Result { - let mut owners = KVec::new(); - for _i in 0..(NUM_BUFFERS as usize) { - owners.push(None, GFP_KERNEL)?; - } - Ok(BufferManager::ver(slotalloc::SlotAllocator::new( - NUM_BUFFERS, - BufferManagerInner::ver { owners }, - |_inner, _slot| Some(BufferSlotInner::ver()), - c_str!("BufferManager::SlotAllocator"), - static_lock_class!(), - static_lock_class!(), - )?)) - } - - /// Signals a Buffer to synchronously grow. - pub(crate) fn grow(&self, slot: u32) { - match self - .0 - .with_inner(|inner| inner.owners[slot as usize].as_ref().cloned()) - { - Some(owner) => { - pr_err!( - "BufferManager: Unexpected grow request for slot {}. This might deadlock. Please report this bug.\n", - slot - ); - owner.sync_grow(); - } - None => { - pr_err!( - "BufferManager: Received grow request for empty slot {}\n", - slot - ); - } - } - } -} - -#[versions(AGX_NEO)] -impl Clone for BufferManager::ver { - fn clone(&self) -> Self { - BufferManager::ver(self.0.clone()) - } -} diff --git a/drivers/gpu/drm/asahi-neo/channel.rs b/drivers/gpu/drm/asahi-neo/channel.rs deleted file mode 100644 index 590d2e5fa3aae9..00000000000000 --- a/drivers/gpu/drm/asahi-neo/channel.rs +++ /dev/null @@ -1,720 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU ring buffer channels -//! -//! The GPU firmware use a set of ring buffer channels to receive commands from the driver and send -//! it notifications and status messages. -//! -//! These ring buffers mostly follow uniform conventions, so they share the same base -//! implementation. - -use crate::debug::*; -use crate::driver::{ - AsahiDevRef, - AsahiDevice, // -}; -use crate::fw::channels::*; -use crate::fw::initdata::{ - raw, - ChannelRing, // -}; -use crate::fw::types::*; -use crate::{ - buffer, - event, - gpu, - mem, - module_parameters, // -}; -use core::sync::atomic::Ordering; -use kernel::{ - c_str, - prelude::*, - sync::Arc, - time::{ - delay::fsleep, - Delta, - Instant, - Monotonic, // - }, -}; - -pub(crate) use crate::fw::channels::PipeType; - -/// A receive (FW->driver) channel. -pub(crate) struct RxChannel -where - for<'a> ::Raw<'a>: Debug + Default + Zeroable, -{ - ring: ChannelRing, - // FIXME: needs feature(generic_const_exprs) - //rptr: [u32; T::SUB_CHANNELS], - rptr: [u32; 6], - count: u32, -} - -impl RxChannel -where - for<'a> ::Raw<'a>: Debug + Default + Zeroable, -{ - /// Allocates a new receive channel with a given message count. - pub(crate) fn new(alloc: &mut gpu::KernelAllocators, count: usize) -> Result> { - Ok(RxChannel { - ring: ChannelRing { - state: alloc.shared.new_default()?, - ring: alloc.shared.array_empty(T::SUB_CHANNELS * count)?, - }, - rptr: Default::default(), - count: count as u32, - }) - } - - /// Receives a message on the specified sub-channel index, optionally leaving in the ring - /// buffer. - /// - /// Returns None if the channel is empty. - fn get_or_peek(&mut self, index: usize, peek: bool) -> Option { - self.ring.state.with(|raw, _inner| { - let wptr = T::wptr(raw, index); - let rptr = &mut self.rptr[index]; - if wptr == *rptr { - None - } else { - let off = self.count as usize * index; - let msg = self.ring.ring[off + *rptr as usize]; - if !peek { - *rptr = (*rptr + 1) % self.count; - T::set_rptr(raw, index, *rptr); - } - Some(msg) - } - }) - } - - /// Receives a message on the specified sub-channel index, and dequeues it from the ring buffer. - /// - /// Returns None if the channel is empty. - pub(crate) fn get(&mut self, index: usize) -> Option { - self.get_or_peek(index, false) - } - - /// Peeks a message on the specified sub-channel index, leaving it in the ring buffer. - /// - /// Returns None if the channel is empty. - pub(crate) fn peek(&mut self, index: usize) -> Option { - self.get_or_peek(index, true) - } -} - -/// A transmit (driver->FW) channel. -pub(crate) struct TxChannel -where - for<'a> ::Raw<'a>: Debug + Default + Zeroable, -{ - ring: ChannelRing, - wptr: u32, - count: u32, -} - -impl TxChannel -where - for<'a> ::Raw<'a>: Debug + Default + Zeroable, -{ - /// Allocates a new cached transmit channel with a given message count. - pub(crate) fn new(alloc: &mut gpu::KernelAllocators, count: usize) -> Result> { - Ok(TxChannel { - ring: ChannelRing { - state: alloc.shared.new_default()?, - ring: alloc.private.array_empty(count)?, - }, - wptr: 0, - count: count as u32, - }) - } - - /// Allocates a new uncached transmit channel with a given message count. - pub(crate) fn new_uncached( - alloc: &mut gpu::KernelAllocators, - count: usize, - ) -> Result> { - Ok(TxChannel { - ring: ChannelRing { - state: alloc.shared.new_default()?, - ring: alloc.shared.array_empty(count)?, - }, - wptr: 0, - count: count as u32, - }) - } - - /// Send a message to the ring, returning a cookie with the ring buffer position. - /// - /// This will poll/block if the ring is full, which we don't really expect to happen. - pub(crate) fn put(&mut self, msg: &U) -> u32 { - self.ring.state.with(|raw, _inner| { - let next_wptr = (self.wptr + 1) % self.count; - let mut rptr = T::rptr(raw); - if next_wptr == rptr { - pr_err!( - "TX ring buffer is full! Waiting... ({}, {})\n", - next_wptr, - rptr - ); - // TODO: block properly on incoming messages? - while next_wptr == rptr { - fsleep(Delta::from_millis(8)); - rptr = T::rptr(raw); - } - } - self.ring.ring[self.wptr as usize] = *msg; - mem::sync(); - T::set_wptr(raw, next_wptr); - self.wptr = next_wptr; - }); - self.wptr - } - - /// Wait for a previously submitted message to be popped off of the ring by the GPU firmware. - /// - /// This busy-loops, and is intended to be used for rare cases when we need to block for - /// completion of a cache management or invalidation operation synchronously (which - /// the firmware normally completes fast enough not to be worth sleeping for). - /// If the poll takes longer than 10ms, this switches to sleeping between polls. - pub(crate) fn wait_for(&mut self, wptr: u32, timeout_ms: i64) -> Result { - const MAX_FAST_POLL: i64 = 10; - let start = Instant::::now(); - let timeout_ms = timeout_ms.max(1); - let timeout_fast = Delta::from_millis(timeout_ms.min(MAX_FAST_POLL)); - let timeout_slow = Delta::from_millis(timeout_ms); - self.ring.state.with(|raw, _inner| { - while start.elapsed() < timeout_fast { - if T::rptr(raw) == wptr { - return Ok(()); - } - mem::sync(); - } - while start.elapsed() < timeout_slow { - if T::rptr(raw) == wptr { - return Ok(()); - } - fsleep(Delta::from_millis(5)); - mem::sync(); - } - Err(ETIMEDOUT) - }) - } -} - -/// Device Control channel for global device management commands. -#[versions(AGX_NEO)] -pub(crate) struct DeviceControlChannel { - dev: AsahiDevRef, - ch: TxChannel, -} - -#[versions(AGX_NEO)] -impl DeviceControlChannel::ver { - const COMMAND_TIMEOUT_MS: i64 = 1000; - - /// Allocate a new Device Control channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ) -> Result { - Ok(DeviceControlChannel::ver { - dev: dev.into(), - ch: TxChannel::::new(alloc, 0x100)?, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Submits a Device Control command. - pub(crate) fn send(&mut self, msg: &DeviceControlMsg::ver) -> u32 { - cls_dev_dbg!(DeviceControlCh, self.dev, "DeviceControl: {:?}\n", msg); - self.ch.put(msg) - } - - /// Waits for a previously submitted Device Control command to complete. - pub(crate) fn wait_for(&mut self, wptr: u32) -> Result { - self.ch.wait_for(wptr, Self::COMMAND_TIMEOUT_MS) - } -} - -/// Pipe channel to submit WorkQueue execution requests. -#[versions(AGX_NEO)] -pub(crate) struct PipeChannel { - dev: AsahiDevRef, - ch: TxChannel, -} - -#[versions(AGX_NEO)] -impl PipeChannel::ver { - /// Allocate a new Pipe submission channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ) -> Result { - Ok(PipeChannel::ver { - dev: dev.into(), - ch: TxChannel::::new(alloc, 0x100)?, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Submits a Pipe kick command to the firmware. - pub(crate) fn send(&mut self, msg: &PipeMsg::ver) { - cls_dev_dbg!(PipeCh, self.dev, "Pipe: {:?}\n", msg); - self.ch.put(msg); - } -} - -/// Firmware Control channel, used for secure cache flush requests. -pub(crate) struct FwCtlChannel { - dev: AsahiDevRef, - ch: TxChannel, -} - -impl FwCtlChannel { - const COMMAND_TIMEOUT_MS: i64 = 1000; - - /// Allocate a new Firmware Control channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ) -> Result { - Ok(FwCtlChannel { - dev: dev.into(), - ch: TxChannel::::new_uncached(alloc, 0x100)?, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Submits a Firmware Control command to the firmware. - pub(crate) fn send(&mut self, msg: &FwCtlMsg) -> u32 { - cls_dev_dbg!(FwCtlCh, self.dev, "FwCtl: {:?}\n", msg); - self.ch.put(msg) - } - - /// Waits for a previously submitted Firmware Control command to complete. - pub(crate) fn wait_for(&mut self, wptr: u32) -> Result { - self.ch.wait_for(wptr, Self::COMMAND_TIMEOUT_MS) - } -} - -/// Event channel, used to notify the driver of command completions, GPU faults and errors, and -/// other events. -#[versions(AGX_NEO)] -pub(crate) struct EventChannel { - dev: AsahiDevRef, - ch: RxChannel, - ev_mgr: Arc, - buf_mgr: buffer::BufferManager::ver, - gpu: Option>, -} - -#[versions(AGX_NEO)] -impl EventChannel::ver { - /// Allocate a new Event channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ev_mgr: Arc, - buf_mgr: buffer::BufferManager::ver, - ) -> Result { - Ok(EventChannel::ver { - dev: dev.into(), - ch: RxChannel::::new(alloc, 0x100)?, - ev_mgr, - buf_mgr, - gpu: None, - }) - } - - /// Registers the managing `Gpu` instance that will handle events on this channel. - pub(crate) fn set_manager(&mut self, gpu: Arc) { - self.gpu = Some(gpu); - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Polls for new Event messages on this ring. - pub(crate) fn poll(&mut self) { - while let Some(msg) = self.ch.get(0) { - // SAFETY: The raw view is always valid for all bit patterns. - let tag = unsafe { msg.raw.0 }; - match tag { - 0..=EVENT_MAX => { - // SAFETY: Since we have checked the tag to be in range, - // accessing the enum view is valid. - let msg = unsafe { msg.msg }; - - cls_dev_dbg!(EventCh, self.dev, "Event: {:?}\n", msg); - match msg { - EventMsg::Fault => match self.gpu.as_ref() { - Some(gpu) => gpu.handle_fault(), - None => { - dev_crit!( - self.dev.as_ref(), - "EventChannel: No GPU manager available!\n" - ) - } - }, - EventMsg::Timeout { - counter, - unk_8, - event_slot, - } => match self.gpu.as_ref() { - Some(gpu) => gpu.handle_timeout(counter, event_slot, unk_8), - None => { - dev_crit!( - self.dev.as_ref(), - "EventChannel: No GPU manager available!\n" - ) - } - }, - EventMsg::Flag { firing, .. } => { - for (i, flags) in firing.iter().enumerate() { - for j in 0..32 { - if flags & (1u32 << j) != 0 { - self.ev_mgr.signal((i * 32 + j) as u32); - } - } - } - } - EventMsg::GrowTVB { - vm_slot, - buffer_slot, - counter, - } => match self.gpu.as_ref() { - Some(gpu) => { - self.buf_mgr.grow(buffer_slot); - gpu.ack_grow(buffer_slot, vm_slot, counter); - } - None => { - dev_crit!( - self.dev.as_ref(), - "EventChannel: No GPU manager available!\n" - ) - } - }, - EventMsg::ChannelError { - error_type, - pipe_type, - event_slot, - event_value, - } => match self.gpu.as_ref() { - Some(gpu) => { - let error_type = match error_type { - 0 => ChannelErrorType::MemoryError, - 1 => ChannelErrorType::DMKill, - 2 => ChannelErrorType::Aborted, - 3 => ChannelErrorType::Unk3, - a => ChannelErrorType::Unknown(a), - }; - gpu.handle_channel_error( - error_type, - pipe_type, - event_slot, - event_value, - ); - } - None => { - dev_crit!( - self.dev.as_ref(), - "EventChannel: No GPU manager available!\n" - ) - } - }, - msg => { - dev_crit!(self.dev.as_ref(), "Unknown event message: {:?}\n", msg); - } - } - } - _ => { - // SAFETY: The raw view is always valid for all bit patterns. - dev_warn!(self.dev.as_ref(), "Unknown event message: {:?}\n", unsafe { - msg.raw - }); - } - } - } - } -} - -/// Firmware Log channel. This one is pretty special, since it has 6 sub-channels (for different log -/// levels), and it also uses a side buffer to actually hold the log messages, only passing around -/// pointers in the main buffer. -pub(crate) struct FwLogChannel { - dev: AsahiDevRef, - ch: RxChannel, - payload_buf: GpuArray, -} - -impl FwLogChannel { - const RING_SIZE: usize = 0x100; - const BUF_SIZE: usize = 0x100; - - /// Allocate a new Firmware Log channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ) -> Result { - Ok(FwLogChannel { - dev: dev.into(), - ch: RxChannel::::new(alloc, Self::RING_SIZE)?, - payload_buf: alloc - .shared - .array_empty(Self::BUF_SIZE * FwLogChannelState::SUB_CHANNELS)?, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Returns the GPU pointers to the firmware log payload buffer. - pub(crate) fn get_buf(&self) -> GpuWeakPointer<[RawFwLogPayloadMsg]> { - self.payload_buf.weak_pointer() - } - - /// Polls for new log messages on all sub-rings. - pub(crate) fn poll(&mut self) { - for i in 0..=FwLogChannelState::SUB_CHANNELS - 1 { - while let Some(msg) = self.ch.peek(i) { - cls_dev_dbg!(FwLogCh, self.dev, "FwLog{}: {:?}\n", i, msg); - if msg.msg_type != 2 { - dev_warn!(self.dev.as_ref(), "Unknown FWLog{} message: {:?}\n", i, msg); - self.ch.get(i); - continue; - } - if msg.msg_index.0 as usize >= Self::BUF_SIZE { - dev_warn!( - self.dev.as_ref(), - "FWLog{} message index out of bounds: {:?}\n", - i, - msg - ); - self.ch.get(i); - continue; - } - let index = Self::BUF_SIZE * i + msg.msg_index.0 as usize; - let payload = &self.payload_buf.as_slice()[index]; - if payload.msg_type != 3 { - dev_warn!( - self.dev.as_ref(), - "Unknown FWLog{} payload: {:?}\n", - i, - payload - ); - self.ch.get(i); - continue; - } - let msg = if let Some(end) = payload.msg.iter().position(|&r| r == 0) { - CStr::from_bytes_with_nul(&(*payload.msg)[..end + 1]) - .unwrap_or(c_str!("cstr_err")) - } else { - dev_warn!( - self.dev.as_ref(), - "FWLog{} payload not NUL-terminated: {:?}\n", - i, - payload - ); - self.ch.get(i); - continue; - }; - match i { - 0 => dev_dbg!(self.dev.as_ref(), "FWLog: {}\n", msg), - 1 => dev_info!(self.dev.as_ref(), "FWLog: {}\n", msg), - 2 => dev_notice!(self.dev.as_ref(), "FWLog: {}\n", msg), - 3 => dev_warn!(self.dev.as_ref(), "FWLog: {}\n", msg), - 4 => dev_err!(self.dev.as_ref(), "FWLog: {}\n", msg), - 5 => dev_crit!(self.dev.as_ref(), "FWLog: {}\n", msg), - _ => (), - }; - self.ch.get(i); - } - } - } -} - -pub(crate) struct KTraceChannel { - dev: AsahiDevRef, - ch: RxChannel, -} - -/// KTrace channel, used to receive detailed execution trace markers from the firmware. -/// We currently disable this in initdata, so no messages are expected here at this time. -impl KTraceChannel { - /// Allocate a new KTrace channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - ) -> Result { - Ok(KTraceChannel { - dev: dev.into(), - ch: RxChannel::::new(alloc, 0x200)?, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Polls for new KTrace messages on this ring. - pub(crate) fn poll(&mut self) { - while let Some(msg) = self.ch.get(0) { - cls_dev_dbg!(KTraceCh, self.dev, "KTrace: {:?}\n", msg); - } - } -} - -/// Statistics channel, reporting power-related statistics to the driver. -/// Debug-logs the messages and, when the `stats_export` module parameter -/// is enabled, also updates the shared `StatsSnapshot` exported via -/// `/sys/class/drm_neo/card*/device/agx_stats`. -#[versions(AGX_NEO)] -pub(crate) struct StatsChannel { - dev: AsahiDevRef, - ch: RxChannel, - snap: Arc, - /// Rate of firmware stats-message timestamps (Hz). All stats timestamps - /// run on the chip base clock (`HwConfig::base_clock_hz`, 24 MHz on every - /// supported SoC), not nanoseconds. - ts_hz: u64, - /// Last `Utilization` message timestamp (raw base-clock ticks), for - /// utilization-weighted `busy_ns` integration. - last_util_ts: u64, -} - -#[versions(AGX_NEO)] -impl StatsChannel::ver { - /// Allocate a new Statistics channel. - pub(crate) fn new( - dev: &AsahiDevice, - alloc: &mut gpu::KernelAllocators, - snap: Arc, - ts_hz: u64, - ) -> Result { - Ok(StatsChannel::ver { - dev: dev.into(), - ch: RxChannel::::new(alloc, 0x100)?, - snap, - ts_hz, - last_util_ts: 0, - }) - } - - /// Returns the raw `ChannelRing` structure to pass to firmware. - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - self.ch.ring.to_raw() - } - - /// Polls for new statistics messages on this ring. - pub(crate) fn poll(&mut self) { - while let Some(msg) = self.ch.get(0) { - // SAFETY: The raw view is always valid for all bit patterns. - let tag = unsafe { msg.raw.0 }; - match tag { - 0..=STATS_MAX::ver => { - // SAFETY: Since we have checked the tag to be in range, - // accessing the enum view is valid. - let msg = unsafe { msg.msg }; - cls_dev_dbg!(StatsCh, self.dev, "Stats: {:?}\n", msg); - if *module_parameters::stats_export.value() != 0 { - // Retain the latest values for /sys/class/drm_neo/card*/ - // device/agx_stats. Read-only with respect to - // scheduling or power behaviour: the firmware already - // sent the message, we only keep what it said. - match &msg { - StatsMsg::ver::Utilization { - timestamp, - util1, - util2, - util3, - util4, - } => { - self.snap.util1.store(*util1, Ordering::Relaxed); - self.snap.util2.store(*util2, Ordering::Relaxed); - self.snap.util3.store(*util3, Ordering::Relaxed); - self.snap.util4.store(*util4, Ordering::Relaxed); - // busy_ns: utilization-weighted busy time in - // nanoseconds. Each Utilization window - // contributes its duration scaled by the - // busiest subqueue's percentage. MEASURED on - // T6021 (round-3 window): util1..4 read 100 - // across a saturated 30 s matmul and 0-2 on - // an idle desktop, while FwBusy timestamp - // deltas (the prior producer) covered 0.17% - // of that matmul. The timestamps are base- - // clock ticks (24 MHz), so convert to ns - // once here. The kernel runtime has no - // 128-bit division, so split whole seconds - // from the tick remainder: whole seconds - // contribute util * 1e7 ns each (1e9 / 100 - // exactly), the remainder contributes - // rem * util * 1e9 / (100 * ts_hz), which - // cannot overflow u64 (rem < ts_hz <= 24e6 - // on supported SoCs). - let util = [*util1, *util2, *util3, *util4] - .into_iter() - .max() - .unwrap_or(0) - .min(100) as u64; - let ts = timestamp.0; - if self.last_util_ts != 0 && ts > self.last_util_ts { - let delta = ts - self.last_util_ts; - let whole_s = delta / self.ts_hz; - let rem = delta % self.ts_hz; - let busy = whole_s - .saturating_mul(util) - .saturating_mul(10_000_000) - .saturating_add( - rem * util * 1_000_000_000 / (100 * self.ts_hz), - ); - self.snap.busy_ns.fetch_add(busy, Ordering::Relaxed); - } - self.last_util_ts = ts; - } - StatsMsg::ver::PowerState { pstate, .. } => { - self.snap.pstate.store(*pstate, Ordering::Relaxed); - } - StatsMsg::ver::AvgPower { avg_power, .. } => { - self.snap.avg_power_mw.store(*avg_power, Ordering::Relaxed); - } - StatsMsg::ver::Temperature { - raw_value, scale, .. - } => { - self.snap - .temperature_raw - .store(*raw_value, Ordering::Relaxed); - self.snap.temperature_scale.store(*scale, Ordering::Relaxed); - } - _ => {} - } - } - } - _ => { - // SAFETY: The raw view is always valid for all bit patterns. - pr_warn!("Unknown stats message: {:?}\n", unsafe { msg.raw }); - } - } - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/crashdump.rs b/drivers/gpu/drm/asahi-neo/crashdump.rs deleted file mode 100644 index 148cae7aa9b460..00000000000000 --- a/drivers/gpu/drm/asahi-neo/crashdump.rs +++ /dev/null @@ -1,307 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU crash dump formatter -//! -//! Takes a raw dump of firmware/kernel mapped pages from `pgtable` and formats it into -//! an ELF core dump suitable for dumping into userspace. - -use core::mem::size_of; - -use kernel::{ - devcoredump::DevCoreDump, - error::Result, - page::{Page, PAGE_MASK, PAGE_SHIFT, PAGE_SIZE}, - prelude::*, - types::Owned, - uapi, -}; - -use crate::hw; -use crate::pgtable::{self, DumpedPage, Prot, UAT_PGSZ}; -use crate::util::align; - -pub(crate) struct CrashDump { - headers: KVVec, - pages: KVVec>, -} - -const NOTE_NAME_AGX: &str = "AGX"; -const NOTE_AGX_DUMP_INFO: u32 = 1; - -const NOTE_NAME_RTKIT: &str = "RTKIT"; -const NOTE_RTKIT_CRASHLOG: u32 = 1; - -#[repr(C)] -pub(crate) struct AGXDumpInfo { - initdata_address: u64, - chip_id: u32, - gpu_gen: hw::GpuGen, - gpu_variant: hw::GpuVariant, - gpu_rev: hw::GpuRevision, - total_active_cores: u32, - firmware_version: [u32; 6], - reserved: u32, -} - -struct ELFNote { - name: &'static str, - ty: u32, - data: KVVec, -} - -pub(crate) struct CrashDumpBuilder { - page_dump: KVVec, - notes: KVec, -} - -/// Helper to convert ELF headers into byte slices -/// TODO: Hook this up into kernel::AsBytes somehow -/// -/// # Safety -/// -/// Types implementing this trait must have no padding bytes. -unsafe trait AsBytes: Sized { - fn as_bytes(&self) -> &[u8] { - // SAFETY: This trait is only implemented for types with no padding bytes - unsafe { core::slice::from_raw_parts(self as *const _ as *const u8, size_of::()) } - } - fn slice_as_bytes(slice: &[Self]) -> &[u8] { - // SAFETY: This trait is only implemented for types with no padding bytes - unsafe { - core::slice::from_raw_parts(slice.as_ptr() as *const u8, core::mem::size_of_val(slice)) - } - } -} - -// SAFETY: This type has no padding -unsafe impl AsBytes for uapi::Elf64_Ehdr {} -// SAFETY: This type has no padding -unsafe impl AsBytes for uapi::Elf64_Phdr {} -// SAFETY: This type has no padding -unsafe impl AsBytes for uapi::Elf64_Nhdr {} -// SAFETY: This type has no padding -unsafe impl AsBytes for AGXDumpInfo {} - -const FIRMWARE_ENTRYPOINT: u64 = 0xFFFFFF8000000000u64; - -impl CrashDumpBuilder { - pub(crate) fn new(page_dump: KVVec) -> Result { - Ok(CrashDumpBuilder { - page_dump, - notes: KVec::new(), - }) - } - - pub(crate) fn add_agx_info( - &mut self, - cfg: &hw::HwConfig, - dyncfg: &hw::DynConfig, - initdata_address: u64, - ) -> Result { - let mut info = AGXDumpInfo { - chip_id: cfg.chip_id, - gpu_gen: dyncfg.id.gpu_gen, - gpu_variant: dyncfg.id.gpu_variant, - gpu_rev: dyncfg.id.gpu_rev, - total_active_cores: dyncfg.id.total_active_cores, - firmware_version: [0; 6], - reserved: 0, - initdata_address, - }; - info.firmware_version[..dyncfg.firmware_version.len().min(6)] - .copy_from_slice(&dyncfg.firmware_version); - - let mut data = KVVec::new(); - data.extend_from_slice(info.as_bytes(), GFP_KERNEL)?; - - self.notes.push( - ELFNote { - name: NOTE_NAME_AGX, - ty: NOTE_AGX_DUMP_INFO, - data, - }, - GFP_KERNEL, - )?; - Ok(()) - } - - pub(crate) fn add_crashlog(&mut self, crashlog: &[u8]) -> Result { - let mut data = KVVec::new(); - data.extend_from_slice(crashlog, GFP_KERNEL)?; - - self.notes.push( - ELFNote { - name: NOTE_NAME_RTKIT, - ty: NOTE_RTKIT_CRASHLOG, - data, - }, - GFP_KERNEL, - )?; - - Ok(()) - } - - pub(crate) fn finalize(self) -> Result { - let CrashDumpBuilder { page_dump, notes } = self; - - let mut ehdr: uapi::Elf64_Ehdr = Default::default(); - - ehdr.e_ident[uapi::EI_MAG0 as usize..=uapi::EI_MAG3 as usize].copy_from_slice(b"\x7fELF"); - ehdr.e_ident[uapi::EI_CLASS as usize] = uapi::ELFCLASS64 as u8; - ehdr.e_ident[uapi::EI_DATA as usize] = uapi::ELFDATA2LSB as u8; - ehdr.e_ident[uapi::EI_VERSION as usize] = uapi::EV_CURRENT as u8; - ehdr.e_type = uapi::ET_CORE as u16; - ehdr.e_machine = uapi::EM_AARCH64 as u16; - ehdr.e_version = uapi::EV_CURRENT; - ehdr.e_entry = FIRMWARE_ENTRYPOINT; - ehdr.e_ehsize = core::mem::size_of::() as u16; - ehdr.e_phentsize = core::mem::size_of::() as u16; - - let phdr_offset = core::mem::size_of::(); - - // PHDRs come after the ELF header - ehdr.e_phoff = phdr_offset as u64; - - let mut phdrs = KVVec::new(); - - // First PHDR is the NOTE section - phdrs.push( - uapi::Elf64_Phdr { - p_type: uapi::PT_NOTE, - p_flags: uapi::PF_R, - p_align: 1, - ..Default::default() - }, - GFP_KERNEL, - )?; - - // Generate the page phdrs. The offset will be fixed up later. - let mut off: usize = 0; - let mut next = None; - let mut pages: KVVec> = KVVec::new(); - - for mut page in page_dump { - let vaddr = page.iova; - let paddr = page.pte & pgtable::PTE_ADDR_BITS; - let flags = Prot::from_pte(page.pte).elf_flags(); - let valid = page.data.is_some(); - let cur = (vaddr, paddr, flags, valid); - if Some(cur) != next { - phdrs.push( - uapi::Elf64_Phdr { - p_type: uapi::PT_LOAD, - p_offset: if valid { off as u64 } else { 0 }, - p_vaddr: vaddr, - p_paddr: paddr, - p_filesz: if valid { UAT_PGSZ as u64 } else { 0 }, - p_memsz: UAT_PGSZ as u64, - p_flags: flags, - p_align: UAT_PGSZ as u64, - }, - GFP_KERNEL, - )?; - if valid { - off += UAT_PGSZ; - } - } else { - let ph = phdrs.last_mut().unwrap(); - ph.p_memsz += UAT_PGSZ as u64; - if valid { - ph.p_filesz += UAT_PGSZ as u64; - off += UAT_PGSZ; - } - } - if let Some(data_page) = page.data.take() { - pages.push(data_page, GFP_KERNEL)?; - } - next = Some(( - vaddr + UAT_PGSZ as u64, - paddr + UAT_PGSZ as u64, - flags, - valid, - )); - } - - ehdr.e_phnum = phdrs.len() as u16; - - let note_offset = phdr_offset + size_of::() * phdrs.len(); - - let mut note_data: KVVec = KVVec::new(); - - for note in notes { - let hdr = uapi::Elf64_Nhdr { - n_namesz: note.name.len() as u32 + 1, - n_descsz: note.data.len() as u32, - n_type: note.ty, - }; - note_data.extend_from_slice(hdr.as_bytes(), GFP_KERNEL)?; - note_data.extend_from_slice(note.name.as_bytes(), GFP_KERNEL)?; - note_data.push(0, GFP_KERNEL)?; - while note_data.len() & 3 != 0 { - note_data.push(0, GFP_KERNEL)?; - } - note_data.extend_from_slice(¬e.data, GFP_KERNEL)?; - while note_data.len() & 3 != 0 { - note_data.push(0, GFP_KERNEL)?; - } - } - - // NOTE section comes after the PHDRs - phdrs[0].p_offset = note_offset as u64; - phdrs[0].p_filesz = note_data.len() as u64; - - // Align data section to the page size - let data_offset = align(note_offset + note_data.len(), UAT_PGSZ); - - // Fix up data PHDR offsets - for phdr in &mut phdrs[1..] { - phdr.p_offset += data_offset as u64; - } - - // Build ELF header buffer - let mut headers: KVVec = KVVec::from_elem(0, data_offset, GFP_KERNEL)?; - - headers[0..size_of::()].copy_from_slice(ehdr.as_bytes()); - headers[phdr_offset..phdr_offset + phdrs.len() * size_of::()] - .copy_from_slice(AsBytes::slice_as_bytes(&phdrs)); - headers[note_offset..note_offset + note_data.len()].copy_from_slice(¬e_data); - - Ok(CrashDump { headers, pages }) - } -} - -impl DevCoreDump for CrashDump { - fn read(&self, buf: &mut [u8], mut offset: usize) -> Result { - let mut read = 0; - let mut left = buf.len(); - if offset < self.headers.len() { - let block = left.min(self.headers.len() - offset); - buf[..block].copy_from_slice(&self.headers[offset..offset + block]); - read += block; - offset += block; - left -= block; - } - if left == 0 { - return Ok(read); - } - offset -= self.headers.len(); // Offset from the page area - - while left > 0 { - let page_index = offset >> PAGE_SHIFT; - let page_offset = offset & !PAGE_MASK; - let block = left.min(PAGE_SIZE - page_offset); - let Some(page) = self.pages.get(page_index) else { - break; - }; - let slice = &mut buf[read..read + block]; - // SAFETY: We own the page, and the slice guarantees the - // dst length is sufficient. - unsafe { page.read_raw(slice.as_mut_ptr(), page_offset, slice.len())? }; - read += block; - offset += block; - left -= block; - } - - Ok(read) - } -} diff --git a/drivers/gpu/drm/asahi-neo/debug.rs b/drivers/gpu/drm/asahi-neo/debug.rs deleted file mode 100644 index 50628ade5ab8e4..00000000000000 --- a/drivers/gpu/drm/asahi-neo/debug.rs +++ /dev/null @@ -1,142 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![allow(dead_code)] - -//! Debug enable/disable flags and convenience macros - -#[allow(unused_imports)] -pub(crate) use super::{ - cls_dev_dbg, - cls_pr_debug, - debug, - mod_dev_dbg, - mod_pr_debug, // -}; -use crate::module_parameters; -use core::sync::atomic::{ - AtomicU64, - Ordering, // -}; - -static DEBUG_FLAGS: AtomicU64 = AtomicU64::new(0); - -/// Debug flag bit indices -pub(crate) enum DebugFlags { - // 0-4: Memory-related debug - Mmu = 0, - PgTable = 1, - Alloc = 2, - Gem = 3, - Object = 4, - - // 5-7: Firmware objects and resources - Event = 5, - Buffer = 6, - WorkQueue = 7, - - // 8-13: DRM interface, rendering, compute, GPU globals - Gpu = 8, - File = 9, - Queue = 10, - Render = 11, - Compute = 12, - Errors = 13, - - // 14-15: Misc stats - MemStats = 14, - TVBStats = 15, - - // 16-22: Channels - FwLogCh = 16, - KTraceCh = 17, - StatsCh = 18, - EventCh = 19, - PipeCh = 20, - DeviceControlCh = 21, - FwCtlCh = 22, - - // 32-35: Allocator debugging - FillAllocations = 32, - DebugAllocations = 33, - DetectOverflows = 34, - ForceCPUMaps = 35, - - // 36-: Behavior flags - ConservativeTlbi = 36, - KeepGpuPowered = 37, - WaitForPowerOff = 38, - NoGpuRecovery = 39, - DisableClustering = 40, - - // 48-: Misc - Debug0 = 48, - Debug1 = 49, - Debug2 = 50, - Debug3 = 51, - Debug4 = 52, - Debug5 = 53, - Debug6 = 54, - Debug7 = 55, - - VerboseFaults = 61, - AllowUnknownOverrides = 62, - OopsOnGpuCrash = 63, -} - -/// Update the cached global debug flags from the module parameter -pub(crate) fn update_debug_flags() { - let flags = *module_parameters::debug_flags.value(); - - DEBUG_FLAGS.store(flags, Ordering::Relaxed); -} - -/// Check whether debug is enabled for a given flag -#[inline(always)] -pub(crate) fn debug_enabled(flag: DebugFlags) -> bool { - DEBUG_FLAGS.load(Ordering::Relaxed) & 1 << (flag as usize) != 0 -} - -/// Run some code only if debug is enabled for the calling module -#[macro_export] -macro_rules! debug { - ($($arg:tt)*) => { - if $crate::debug::debug_enabled(DEBUG_CLASS) { - $($arg)* - } - }; -} - -/// pr_info!() if debug is enabled for the calling module -#[macro_export] -macro_rules! mod_pr_debug ( - ($($arg:tt)*) => ( - $crate::debug! { ::kernel::pr_info! ( $($arg)* ); } - ) -); - -/// dev_info!() if debug is enabled for the calling module -#[macro_export] -macro_rules! mod_dev_dbg ( - ($dev:expr, $($arg:tt)*) => ( - $crate::debug! { ::kernel::dev_info! ( $dev.as_ref(), $($arg)* ); } - ) -); - -/// pr_info!() if debug is enabled for a specific module -#[macro_export] -macro_rules! cls_pr_debug ( - ($cls:ident, $($arg:tt)*) => ( - if $crate::debug::debug_enabled($crate::debug::DebugFlags::$cls) { - ::kernel::pr_info! ( $($arg)* ); - } - ) -); - -/// dev_info!() if debug is enabled for a specific module -#[macro_export] -macro_rules! cls_dev_dbg ( - ($cls:ident, $dev:expr, $($arg:tt)*) => ( - if $crate::debug::debug_enabled($crate::debug::DebugFlags::$cls) { - ::kernel::dev_info! ( $dev.as_ref(), $($arg)* ); - } - ) -); diff --git a/drivers/gpu/drm/asahi-neo/driver.rs b/drivers/gpu/drm/asahi-neo/driver.rs deleted file mode 100644 index 0c502a21d312e7..00000000000000 --- a/drivers/gpu/drm/asahi-neo/driver.rs +++ /dev/null @@ -1,289 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Top-level GPU driver implementation. - -use crate::module_parameters; -use kernel::bindings; -use kernel::{ - c_str, - device::Core, - dma::{ - Device, - DmaMask, // - }, - drm_neo, - drm_neo::ioctl, - of, - platform, - prelude::*, - sync::{ - aref::ARef, - Arc, // - }, // -}; - -use crate::{ - debug, - file, - g17, - gem::AsahiObject, - gpu, - hw, - regs, // -}; - -use kernel::macros::vtable; - -/// Holds a reference to the top-level GPU object. -#[pin_data] -pub(crate) struct AsahiData { - #[pin] - pub(crate) gpu: Arc, - pub(crate) pdev: ARef, - pub(crate) resources: regs::Resources, -} - -unsafe impl Send for AsahiData {} -unsafe impl Sync for AsahiData {} - -pub(crate) struct AsahiDriver { - drm_neo: ARef>, - /// Raw `struct device *` of the platform device. Stored once at probe - /// time so the `Drop` impl can find the sysfs file on unregister. - raw_dev: *mut bindings::device, - raw_stats: *mut core::ffi::c_void, - // Keep the allocation alive until sysfs readers have drained. - stats: Option>, -} - -unsafe impl Send for AsahiDriver {} -unsafe impl Sync for AsahiDriver {} - -/// Convenience type alias for the DRM device type for this driver. -pub(crate) type AsahiDevice = drm_neo::device::Device; -pub(crate) type AsahiDevRef = ARef; - -/// DRM Driver metadata -const INFO: drm_neo::driver::DriverInfo = drm_neo::driver::DriverInfo { - major: 0, - minor: 0, - patchlevel: 0, - name: c_str!("asahi"), - desc: c_str!("Apple AGX Graphics"), -}; - -/// DRM Driver implementation for `AsahiDriver`. -#[vtable] -impl drm_neo::driver::Driver for AsahiDriver { - /// Our `DeviceData` type, reference-counted - type Data = AsahiData; - /// Our `File` type. - type File = file::File; - /// Our `Object` type. - type Object = drm_neo::gem::shmem::Object; - - const INFO: drm_neo::driver::DriverInfo = INFO; - const MODULE: Option<&'static kernel::ThisModule> = Some(&crate::THIS_MODULE); - const FEATURES: u32 = drm_neo::driver::FEAT_GEM - | drm_neo::driver::FEAT_RENDER - | drm_neo::driver::FEAT_SYNCOBJ - | drm_neo::driver::FEAT_SYNCOBJ_TIMELINE - | drm_neo::driver::FEAT_GEM_GPUVA; - - kernel::declare_drm_neo_ioctls! { - (ASAHI_NEO_GET_PARAMS, drm_asahi_neo_get_params, - ioctl::RENDER_ALLOW, crate::file::File::get_params), - (ASAHI_NEO_GET_TIME, drm_asahi_neo_get_time, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::get_time), - (ASAHI_NEO_VM_CREATE, drm_asahi_neo_vm_create, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::vm_create), - (ASAHI_NEO_VM_DESTROY, drm_asahi_neo_vm_destroy, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::vm_destroy), - (ASAHI_NEO_VM_BIND, drm_asahi_neo_vm_bind, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::vm_bind), - (ASAHI_NEO_GEM_CREATE, drm_asahi_neo_gem_create, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_create), - (ASAHI_NEO_GEM_MMAP_OFFSET, drm_asahi_neo_gem_mmap_offset, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_mmap_offset), - (ASAHI_NEO_GEM_BIND_OBJECT, drm_asahi_neo_gem_bind_object, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_bind_object), - (ASAHI_NEO_QUEUE_CREATE, drm_asahi_neo_queue_create, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::queue_create), - (ASAHI_NEO_QUEUE_DESTROY, drm_asahi_neo_queue_destroy, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::queue_destroy), - (ASAHI_NEO_SUBMIT, drm_asahi_neo_submit, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::submit), - (ASAHI_NEO_GEM_MADVISE, drm_asahi_neo_gem_madvise, - ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_madvise), - } -} - -/// Firmware interface selected by the hardware compatible. -pub(crate) enum ProbeConfig { - Legacy(&'static hw::HwConfig), - G17(&'static g17::Config), -} - -// OF Device ID table. -kernel::of_device_table!( - OF_TABLE, - MODULE_OF_TABLE, - ::IdInfo, - [ - ( - of::DeviceId::new(c_str!("apple,agx-t8140")), - ProbeConfig::G17(&hw::t8140::CONFIG) - ), - ] -); - -/// Platform Driver implementation for `AsahiDriver`. -impl platform::Driver for AsahiDriver { - type IdInfo = ProbeConfig; - const OF_ID_TABLE: Option> = Some(&OF_TABLE); - - fn unbind(_pdev: &platform::Device, this: Pin<&Self>) { - if let Some(gpu) = (*this.drm_neo).gpu.as_any().downcast_ref::() { - // Driver data is dropped after devres, so CPU control must happen in unbind. - gpu.shutdown(); - } - } - - /// Device probe function. - fn probe( - pdev: &platform::Device, - info: Option<&Self::IdInfo>, - ) -> impl PinInit { - debug::update_debug_flags(); - - dev_info!(pdev.as_ref(), "Probing...\n"); - - let cfg = match info.ok_or(ENODEV)? { - ProbeConfig::Legacy(cfg) => *cfg, - ProbeConfig::G17(cfg) => return Self::probe_g17(pdev, cfg), - }; - - unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(cfg.uat_oas)?)? }; - - let res = regs::Resources::new(pdev)?; - - // Initialize misc MMIO - res.init_mmio()?; - - // Start the coprocessor CPU, so UAT can initialize the handoff - regs::Resources::start_cpu(pdev)?; - - let fwnode = pdev.as_ref().fwnode().ok_or(EIO)?; - let compat: KVec = fwnode - .property_read_array_vec(c_str!("apple,firmware-compat"), 3)? - .required_by(pdev.as_ref())?; - - // SAFETY: GPU construction uses the DRM allocation but not its driver data. - // No userspace or driver-data work is published before init_data() succeeds. - let drm_neo: ARef = unsafe { drm_neo::device::Device::new_uninit(pdev.as_ref())? }; - - let gpu = match (cfg.gpu_gen, cfg.gpu_variant, compat.as_slice()) { - (hw::GpuGen::G13, _, &[12, 3, 0]) => { - gpu::GpuManagerG13V12_3::new(&drm_neo.clone(), &res, cfg)? as Arc - } - (hw::GpuGen::G14, hw::GpuVariant::G, &[12, 4, 0]) => { - gpu::GpuManagerG14V12_4::new(&drm_neo.clone(), &res, cfg)? as Arc - } - (hw::GpuGen::G13, _, &[13, 5, 0]) => { - gpu::GpuManagerG13V13_5::new(&drm_neo.clone(), &res, cfg)? as Arc - } - (hw::GpuGen::G14, hw::GpuVariant::G, &[13, 5, 0]) => { - gpu::GpuManagerG14V13_5::new(&drm_neo.clone(), &res, cfg)? as Arc - } - (hw::GpuGen::G14, _, &[13, 5, 0]) => { - gpu::GpuManagerG14XV13_5::new(&drm_neo.clone(), &res, cfg)? as Arc - } - _ => { - dev_info!( - pdev.as_ref(), - "Unsupported GPU/firmware combination ({:?}, {:?}, {:?})\n", - cfg.gpu_gen, - cfg.gpu_variant, - compat - ); - return Err(ENODEV); - } - }; - - let data = try_pin_init!(AsahiData { - gpu, - pdev: pdev.into(), - resources: res, - }); - - // SAFETY: This is the sole initializer; driver data has not been accessed yet. - unsafe { drm_neo.init_data(data)? }; - - (*drm_neo).gpu.init()?; - - drm_neo::driver::Registration::new_foreign_owned(&drm_neo, pdev.as_ref(), 0)?; - - // Register the sysfs file on the platform device. Must happen after - // the DRM device is registered so the device is fully bound. - let raw_dev = pdev.as_ref().as_raw(); - let stats = (*drm_neo).gpu.stats_snapshot(); - let snapshot = stats.as_ref().map_or(core::ptr::null(), Arc::as_ptr); - let raw_stats = crate::sysfs_exports::register( - raw_dev, - snapshot, - stats.is_some() && *module_parameters::stats_export.value() != 0, - )?; - - Ok(Self { drm_neo, raw_dev, raw_stats, stats }) - } -} - -impl AsahiDriver { - fn probe_g17(pdev: &platform::Device, cfg: &'static g17::Config) -> Result { - // SAFETY: The GPU performs DMA through a UAT whose output width is part of the SoC config. - unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(cfg.uat_oas)?)? }; - let res = regs::Resources::new(pdev)?; - // SAFETY: GPU construction cannot access driver data before init_data(). - // Failed construction releases the allocation without dropping uninitialized data. - let drm_neo: ARef = unsafe { drm_neo::device::Device::new_uninit(pdev.as_ref())? }; - let gpu = g17::Gpu::new(pdev, &drm_neo, cfg, res.clone())?; - let data_gpu = gpu.clone() as Arc; - let data = try_pin_init!(AsahiData { - gpu: data_gpu, - pdev: pdev.into(), - resources: res, - }); - // SAFETY: This is the sole initializer; no driver-data users have been published. - if let Err(error) = unsafe { drm_neo.init_data(data) } { - gpu.shutdown(); - return Err(error); - } - if let Err(error) = drm_neo::driver::Registration::new_foreign_owned(&drm_neo, pdev.as_ref(), 0) { - gpu.shutdown(); - return Err(error); - } - // G17's firmware channel format has no supported stats decoder yet. - let raw_dev = pdev.as_ref().as_raw(); - let raw_stats = match crate::sysfs_exports::register( - raw_dev, core::ptr::null(), false, - ) { - Ok(handle) => handle, - Err(error) => { - gpu.shutdown(); - return Err(error); - } - }; - Ok(Self { drm_neo, raw_dev, raw_stats, stats: None }) - } -} - -impl Drop for AsahiDriver { - fn drop(&mut self) { - crate::sysfs_exports::unregister(self.raw_dev, self.raw_stats); - drop(self.stats.take()); - if let Some(gpu) = (*self.drm_neo).gpu.as_any().downcast_ref::() { - // Also covers failure to allocate platform driver data after the probe body returned. - gpu.shutdown(); - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/event.rs b/drivers/gpu/drm/asahi-neo/event.rs deleted file mode 100644 index edd7d701e665cd..00000000000000 --- a/drivers/gpu/drm/asahi-neo/event.rs +++ /dev/null @@ -1,257 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU event manager -//! -//! The GPU firmware manages work completion by using event objects (Apple calls them "stamps"), -//! which are monotonically incrementing counters. There are a fixed number of objects, and -//! they are managed with a `SlotAllocator`. -//! -//! This module manages the set of available events and lets users compute expected values. -//! It also manages signaling owners when the GPU firmware reports that an event fired. - -use crate::debug::*; -use crate::fw::types::*; -use crate::{ - gpu, - slotalloc, - workqueue, // -}; -use core::cmp; -use core::sync::atomic::Ordering; -use kernel::prelude::*; -use kernel::sync::Arc; -use kernel::{ - c_str, - static_lock_class, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Event; - -/// Number of events managed by the firmware. -const NUM_EVENTS: u32 = 128; - -/// Inner data associated with a given event slot. -pub(crate) struct EventInner { - /// CPU pointer to the driver notification event stamp - stamp: *const AtomicU32, - /// GPU pointer to the driver notification event stamp - gpu_stamp: GpuWeakPointer, - /// GPU pointer to the firmware-internal event stamp - gpu_fw_stamp: GpuWeakPointer, -} - -/// SAFETY: The event slots are safe to send across threads. -unsafe impl Send for EventInner {} - -/// Alias for an event token, which allows requesting the same event. -pub(crate) type Token = slotalloc::SlotToken; -/// Alias for an allocated `Event` that has a slot. -pub(crate) type Event = slotalloc::Guard; - -/// Represents a given stamp value for an event. -#[derive(Eq, PartialEq, Copy, Clone, Debug)] -#[repr(transparent)] -pub(crate) struct EventValue(u32); - -impl EventValue { - /// Returns the `EventValue` that succeeds this one. - pub(crate) fn next(&self) -> EventValue { - EventValue(self.0.wrapping_add(0x100)) - } - - /// Increments this `EventValue` in place. - pub(crate) fn increment(&mut self) { - self.0 = self.0.wrapping_add(0x100); - } - - /* Not used - /// Increments this `EventValue` in place by a certain count. - pub(crate) fn add(&mut self, val: u32) { - self.0 = self - .0 - .wrapping_add(val.checked_mul(0x100).expect("Adding too many events")); - } - */ - - /// Increments this `EventValue` in place by a certain count. - pub(crate) fn sub(&mut self, val: u32) { - self.0 = self - .0 - .wrapping_sub(val.checked_mul(0x100).expect("Subtracting too many events")); - } - - /// Computes the delta between this event and another event. - pub(crate) fn delta(&self, other: &EventValue) -> i32 { - (self.0.wrapping_sub(other.0) as i32) >> 8 - } -} - -impl PartialOrd for EventValue { - fn partial_cmp(&self, other: &Self) -> Option { - Some(self.cmp(other)) - } -} - -impl Ord for EventValue { - fn cmp(&self, other: &Self) -> cmp::Ordering { - self.delta(other).cmp(&0) - } -} - -impl EventInner { - /// Returns the GPU pointer to the driver notification stamp - pub(crate) fn stamp_pointer(&self) -> GpuWeakPointer { - self.gpu_stamp - } - - /// Returns the GPU pointer to the firmware internal stamp - pub(crate) fn fw_stamp_pointer(&self) -> GpuWeakPointer { - self.gpu_fw_stamp - } - - /// Fetches the current event value from shared memory - pub(crate) fn current(&self) -> EventValue { - // SAFETY: The pointer is always valid as constructed in - // EventManager below, and outside users cannot construct - // new EventInners, nor move or copy them, and Guards as - // returned by the SlotAllocator hold a reference to the - // SlotAllocator containing the EventManagerInner, which - // keeps the GpuObject the stamp is contained within alive. - EventValue(unsafe { &*self.stamp }.load(Ordering::Acquire)) - } -} - -impl slotalloc::SlotItem for EventInner { - type Data = EventManagerInner; - - fn release(&mut self, data: &mut Self::Data, slot: u32) { - mod_pr_debug!("EventManager: Released slot {}\n", slot); - data.owners[slot as usize] = None; - } -} - -/// Inner data for the event manager, to be protected by the SlotAllocator lock. -pub(crate) struct EventManagerInner { - stamps: GpuArray, - fw_stamps: GpuArray, - // Note: Use dyn to avoid having to version this entire module. - owners: KVec>>, -} - -/// Top-level EventManager object. -pub(crate) struct EventManager { - alloc: slotalloc::SlotAllocator, -} - -impl EventManager { - /// Create a new EventManager. - #[inline(never)] - pub(crate) fn new(alloc: &mut gpu::KernelAllocators) -> Result { - let mut owners = KVec::new(); - for _i in 0..(NUM_EVENTS as usize) { - owners.push(None, GFP_KERNEL)?; - } - let inner = EventManagerInner { - stamps: alloc.shared.array_empty(NUM_EVENTS as usize)?, - fw_stamps: alloc.private.array_empty(NUM_EVENTS as usize)?, - owners, - }; - - for slot in 0..NUM_EVENTS { - inner.stamps[slot as usize] - .0 - .store(slot << 24, Ordering::Relaxed); - } - - Ok(EventManager { - alloc: slotalloc::SlotAllocator::new( - NUM_EVENTS, - inner, - |inner: &mut EventManagerInner, slot| { - Some(EventInner { - stamp: &inner.stamps[slot as usize].0, - gpu_stamp: inner.stamps.weak_item_pointer(slot as usize), - gpu_fw_stamp: inner.fw_stamps.weak_item_pointer(slot as usize), - }) - }, - c_str!("EventManager::SlotAllocator"), - static_lock_class!(), - static_lock_class!(), - )?, - }) - } - - /// Gets a free `Event`, optionally trying to reuse the last one allocated by this caller. - pub(crate) fn get( - &self, - token: Option, - owner: Arc, - ) -> Result { - let ev = self.alloc.get_inner(token, |inner, ev| { - mod_pr_debug!( - "EventManager: Registered owner {:p} on slot {}\n", - &*owner, - ev.slot() - ); - inner.owners[ev.slot() as usize] = Some(owner); - Ok(()) - })?; - Ok(ev) - } - - /// Signals an event by slot, indicating completion (of one or more commands). - pub(crate) fn signal(&self, slot: u32) { - match self - .alloc - .with_inner(|inner| inner.owners[slot as usize].as_ref().cloned()) - { - Some(owner) => { - owner.signal(); - } - None => { - mod_pr_debug!("EventManager: Received event for empty slot {}\n", slot); - } - } - } - - /// Marks the owner of an event as having lost its work due to a GPU error. - pub(crate) fn mark_error(&self, slot: u32, wait_value: u32, error: workqueue::WorkError) { - match self - .alloc - .with_inner(|inner| inner.owners[slot as usize].as_ref().cloned()) - { - Some(owner) => { - owner.mark_error(EventValue(wait_value), error); - } - None => { - pr_err!("Received error for empty slot {}\n", slot); - } - } - } - - /// Returns a reference to the workqueue owning an event. - pub(crate) fn get_owner( - &self, - slot: u32, - ) -> Option> { - self.alloc - .with_inner(|inner| inner.owners[slot as usize].as_ref().cloned()) - } - - /// Fail all commands, used when the GPU crashes. - pub(crate) fn fail_all(&self, error: workqueue::WorkError) { - let mut owners: KVec> = KVec::new(); - - self.alloc.with_inner(|inner| { - for wq in inner.owners.iter().filter_map(|o| o.as_ref()).cloned() { - if owners.push(wq, GFP_KERNEL).is_err() { - pr_err!("Failed to signal failure to WorkQueue\n"); - } - } - }); - - for wq in owners { - wq.fail_all(error); - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/file.rs b/drivers/gpu/drm/asahi-neo/file.rs deleted file mode 100644 index 2634e9535d9e19..00000000000000 --- a/drivers/gpu/drm/asahi-neo/file.rs +++ /dev/null @@ -1,1332 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![allow(clippy::unusual_byte_groupings)] - -//! File implementation, which represents a single DRM client. -//! -//! This is in charge of managing the resources associated with one GPU client, including an -//! arbitrary number of submission queues and Vm objects, and reporting hardware/driver -//! information to userspace and accepting submissions. - -use crate::debug::*; -use crate::driver::AsahiDevice; -use crate::{ - alloc, - buffer, - driver, - gem, - mmu, - module_parameters, - queue, - util::{ - align, - align_down, - gcd, - AnyBitPattern, - RangeExt, - Reader, // - }, // -}; -use core::mem::MaybeUninit; -use core::ops::Deref; -use core::ops::Range; -use core::ptr::addr_of_mut; -use kernel::bindings; -use kernel::dma_fence::RawDmaFence; -use kernel::drm_neo::gem::BaseObject; -use kernel::error::code::*; -use kernel::new_mutex; -use kernel::prelude::*; -use kernel::sync::{ - Arc, - Mutex, // -}; -use kernel::time::NSEC_PER_SEC; -use kernel::uaccess::{ - UserPtr, - UserSlice, // -}; -use kernel::{ - dma_fence, - drm_neo, - uapi, - xarray, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::File; - -pub(crate) const MAX_COMMANDS_PER_SUBMISSION: u32 = 64; - -/// A client instance of an `mmu::Vm` address space. -struct Vm { - shrinker: Option>, - ualloc: Arc>, - ualloc_priv: Arc>, - vm: mmu::Vm, - user_range: Range, - kernel_range: Range, - _dummy_mapping: mmu::KernelMapping, -} - -impl Drop for Vm { - fn drop(&mut self) { - self.shrinker = None; - self.vm.close_idle_reclaim(); - if let Some(status) = self.vm.status() { - // Retained graphs may keep this status alive; the client's mirror - // object must not outlive the client's VM handle. - status.release_mirror(); - } - if self.vm.status().is_some() { - // Tracked jobs can outlive their file's VM handle. Their last guard performs - // these unmaps once no accepted work can still dereference the mappings. - if let Err(error) = self - .vm - .unmap_user_ranges(self.user_range.clone(), self.kernel_range.clone()) - { - pr_err!("Vm::Drop: deferred user unmap failed: {:?}\n", error); - } - self.vm.bo_deferred_cleanup(); - return; - } - // When the user Vm is dropped, unmap everything in the user range - let left_range = self.user_range.start..self.kernel_range.start; - let right_range = self.kernel_range.end..self.user_range.end; - - if !left_range.is_empty() - && self - .vm - .unmap_range(left_range.start, left_range.range()) - .is_err() - { - pr_err!("Vm::Drop: vm.unmap_range() failed\n"); - } - if !right_range.is_empty() - && self - .vm - .unmap_range(right_range.start, right_range.range()) - .is_err() - { - pr_err!("Vm::Drop: vm.unmap_range() failed\n"); - } - - self.vm.bo_deferred_cleanup(); - } -} - -/// Sync object from userspace. -pub(crate) struct SyncItem { - pub(crate) syncobj: drm_neo::syncobj::SyncObj, - pub(crate) fence: Option, - pub(crate) chain_fence: Option, - pub(crate) timeline_value: u64, -} - -impl SyncItem { - fn parse_one(file: &DrmFile, data: uapi::drm_asahi_neo_sync, out: bool) -> Result { - match data.sync_type { - uapi::drm_asahi_neo_sync_type_DRM_ASAHI_NEO_SYNC_SYNCOBJ => { - if data.timeline_value != 0 { - cls_pr_debug!(Errors, "Non-timeline sync object with a nonzero value\n"); - return Err(EINVAL); - } - let syncobj = drm_neo::syncobj::SyncObj::lookup_handle(file, data.handle)?; - - Ok(SyncItem { - fence: if out { - None - } else { - Some(syncobj.fence_get().ok_or_else(|| { - cls_pr_debug!(Errors, "Failed to get fence from sync object\n"); - EINVAL - })?) - }, - syncobj, - chain_fence: None, - timeline_value: data.timeline_value, - }) - } - uapi::drm_asahi_neo_sync_type_DRM_ASAHI_NEO_SYNC_TIMELINE_SYNCOBJ => { - let syncobj = drm_neo::syncobj::SyncObj::lookup_handle(file, data.handle)?; - let fence = if out { - None - } else { - syncobj - .fence_get() - .ok_or_else(|| { - cls_pr_debug!( - Errors, - "Failed to get fence from timeline sync object\n" - ); - EINVAL - })? - .chain_find_seqno(data.timeline_value)? - }; - - Ok(SyncItem { - fence, - syncobj, - chain_fence: if out { - Some(dma_fence::FenceChain::new()?) - } else { - None - }, - timeline_value: data.timeline_value, - }) - } - _ => { - cls_pr_debug!(Errors, "Invalid sync type {}\n", data.sync_type); - Err(EINVAL) - } - } - } - - fn parse_array( - file: &DrmFile, - ptr: u64, - in_count: u32, - out_count: u32, - ) -> Result> { - let count = in_count.checked_add(out_count).ok_or(EOVERFLOW)?; - - const STRIDE: usize = core::mem::size_of::(); - let size = STRIDE.checked_mul(count as usize).ok_or(EOVERFLOW)?; - let mut vec = KVec::with_capacity(count as usize, GFP_KERNEL)?; - - // SAFETY: We only read this once, so there are no TOCTOU issues. - let mut reader = UserSlice::new(UserPtr::from_addr(ptr as _), size).reader(); - - for i in 0..count { - let mut sync: MaybeUninit = MaybeUninit::uninit(); - - // SAFETY: The size of `sync` is STRIDE - reader.read_raw(unsafe { - core::slice::from_raw_parts_mut(sync.as_mut_ptr() as *mut MaybeUninit, STRIDE) - })?; - - // SAFETY: All bit patterns in the struct are valid - let sync = unsafe { sync.assume_init() }; - - vec.push(SyncItem::parse_one(file, sync, i >= in_count)?, GFP_KERNEL)?; - } - - Ok(vec) - } -} - -#[derive(Clone)] -pub(crate) enum Object { - TimestampBuffer(Arc), -} - -/// State associated with a client. -// #[pin_data] -pub(crate) struct File { - id: u64, - /// Available VM range for the user - user_range: Range, - // #[pin] - vms: xarray::XArray>, - // #[pin] - queues: xarray::XArray>>>, - // #[pin] - objects: xarray::XArray>, -} - -/// Convenience type alias for our DRM `File` type. -pub(crate) type DrmFile = drm_neo::File; - -/// Minimum reserved AS for kernel mappings -const VM_KERNEL_MIN_SIZE: u64 = 0x20000000; - -impl drm_neo::file::DriverFile for File { - type Driver = driver::AsahiDriver; - - /// Create a new `File` instance for a fresh client. - fn open(device: &AsahiDevice) -> Result>> { - debug::update_debug_flags(); - - let gpu = &device.gpu; - let id = gpu.ids().file.next(); - let user_range = gpu.uat_geometry().user_usable_range(); - - mod_dev_dbg!(device, "[File {}]: DRM device opened\n", id); - Ok(KBox::pin_init(File::new(id, user_range), GFP_KERNEL)?) - } - - fn as_raw(&self) -> *mut bindings::drm_file { - todo!() - } -} - -// SAFETY: All bit patterns are valid by construction. -unsafe impl AnyBitPattern for uapi::drm_asahi_neo_gem_bind_op {} - -impl File { - fn new(id: u64, user_range: Range) -> impl PinInit { - unsafe { - pin_init::pin_init_from_closure(move |slot: *mut Self| { - let raw_vms = addr_of_mut!((*slot).vms); - xarray::XArray::>::new(xarray::AllocKind::Alloc1) - .__pinned_init(raw_vms)?; - - let raw_queues = addr_of_mut!((*slot).queues); - xarray::XArray::>>>::new( - xarray::AllocKind::Alloc1, - ) - .__pinned_init(raw_queues)?; - - let raw_objects = addr_of_mut!((*slot).objects); - xarray::XArray::>::new(xarray::AllocKind::Alloc1) - .__pinned_init(raw_objects)?; - - (*slot).id = id; - (*slot).user_range = user_range; - Ok(()) - }) - } - } - - fn vms(self: Pin<&Self>) -> Pin<&xarray::XArray>> { - // SAFETY: Structural pinned projection for vms. - // We never move out of this field. - unsafe { self.map_unchecked(|s| &s.vms) } - } - - #[allow(clippy::type_complexity)] - fn queues(self: Pin<&Self>) -> Pin<&xarray::XArray>>>> { - // SAFETY: Structural pinned projection for queues. - // We never move out of this field. - unsafe { self.map_unchecked(|s| &s.queues) } - } - - fn objects(self: Pin<&Self>) -> Pin<&xarray::XArray>> { - // SAFETY: Structural pinned projection for objects. - // We never move out of this field. - unsafe { self.map_unchecked(|s| &s.objects) } - } - - /// IOCTL: get_param: Get a driver parameter value. - pub(crate) fn get_params( - device: &AsahiDevice, - data: &uapi::drm_asahi_neo_get_params, - file: &DrmFile, - ) -> Result { - mod_dev_dbg!(device, "[File {}]: IOCTL: get_params\n", file.inner().id); - - let gpu = &device.gpu; - - match data.param_group { - uapi::DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS if gpu.supports_vm_status() => { - return Self::get_vm_status(data, file); - } - uapi::DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS_MIRROR if gpu.supports_vm_status() => { - return Self::register_vm_status_mirror(data, file); - } - uapi::DRM_ASAHI_NEO_PARAM_GROUP_QUEUE_LIMITS => { - if data.pad != 0 - || data.size < core::mem::size_of::() as u64 - { - return Err(EINVAL); - } - let limits = gpu.queue_limits().ok_or(EINVAL)?; - let words = [limits.max_queues, limits.max_in_flight_per_queue, 0, 0]; - UserSlice::new( - UserPtr::from_addr(data.pointer as _), - core::mem::size_of_val(&words), - ) - .writer() - .write(&words)?; - return Ok(0); - } - _ => {} - } - - if data.param_group != 0 || data.pad != 0 { - cls_pr_debug!(Errors, "get_params: Invalid arguments\n"); - return Err(EINVAL); - } - - if gpu.is_crashed() { - return Err(ENODEV); - } - - let mut params = uapi::drm_asahi_neo_params_global { - features: 0, - - vm_start: file.inner().user_range.start, - vm_end: file.inner().user_range.end, - vm_kernel_min_size: VM_KERNEL_MIN_SIZE, - - max_commands_per_submission: MAX_COMMANDS_PER_SUBMISSION, - max_attachments: crate::microseq::MAX_ATTACHMENTS as u32, - - command_timestamp_frequency_hz: 1_000_000_000, // User timestamps always in nanoseconds - - ..pin_init::zeroed() - }; - - gpu.get_params(&mut params)?; - - if gpu.supports_vm_status() { - params.features |= uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_VM_STATUS as u64 - | uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_VM_STATUS_MIRROR as u64; - } - if gpu.queue_limits().is_some() { - params.features |= uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_INDEPENDENT_QUEUES as u64; - } - - if *module_parameters::fault_control.value() == 0xb { - params.features |= uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_SOFT_FAULTS as u64; - } - - let size = core::mem::size_of::().min(data.size.try_into()?); - - // SAFETY: We only write to this userptr once, so there are no TOCTOU issues. - let mut params_writer = - UserSlice::new(UserPtr::from_addr(data.pointer as _), size).writer(); - - // SAFETY: `size` is at most the sizeof of `params` - params_writer.write_slice(unsafe { - core::slice::from_raw_parts(¶ms as *const _ as *const u8, size) - })?; - - Ok(0) - } - - /// Query only file-owned VM state; this does not start recovery or wait for work. - fn get_vm_status(data: &uapi::drm_asahi_neo_get_params, file: &DrmFile) -> Result { - let size = core::mem::size_of::(); - if data.pad != 0 || data.size < size as u64 { - return Err(EINVAL); - } - let user = UserPtr::from_addr(data.pointer as _); - let [vm_id, flags, _, pad] = UserSlice::new(user, size).reader().read::<[u32; 4]>()?; - if flags != 0 || pad != 0 { - return Err(EINVAL); - } - let status = Self::vm_status(file, vm_id)?; - UserSlice::new(user, size) - .writer() - .write(&[vm_id, 0, status.reported_error() as u32, 0])?; - Ok(0) - } - - /// Register a file-owned GEM word as the mirror of a file-owned VM's status. - fn register_vm_status_mirror(data: &uapi::drm_asahi_neo_get_params, file: &DrmFile) -> Result { - let size = core::mem::size_of::(); - if data.pad != 0 || data.size < size as u64 { - return Err(EINVAL); - } - let [vm_id, handle, offset, flags] = - UserSlice::new(UserPtr::from_addr(data.pointer as _), size) - .reader() - .read::<[u32; 4]>()?; - if flags != 0 || handle == 0 || offset % core::mem::align_of::() as u32 != 0 { - return Err(EINVAL); - } - let status = Self::vm_status(file, vm_id)?; - let owner = gem::ObjectRef::new(gem::Object::lookup_handle(file, handle)?); - status.attach_mirror(owner, offset as usize)?; - Ok(0) - } - - fn vm_status(file: &DrmFile, vm_id: u32) -> Result> { - let vms = file.inner().vms(); - let guard = vms.lock(); - Ok(guard - .get(vm_id as usize) - .ok_or(ENOENT)? - .vm - .status() - .ok_or(EINVAL)? - .clone()) - } - - /// IOCTL: vm_create: Create a new `Vm`. - pub(crate) fn vm_create( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_vm_create, - file: &DrmFile, - ) -> Result { - let kernel_range = data.kernel_start..data.kernel_end; - let user_range = file.inner().user_range.clone(); - - // Validate requested kernel range - if !user_range.is_superset(kernel_range.clone()) - || kernel_range.range() < VM_KERNEL_MIN_SIZE - || kernel_range.start & (mmu::UAT_PGMSK as u64) != 0 - || kernel_range.end & (mmu::UAT_PGMSK as u64) != 0 - { - cls_pr_debug!(Errors, "vm_create: Invalid kernel range\n"); - return Err(EINVAL); - } - - // Align to buffer::PAGE_SIZE so the allocators are happy - let kernel_range = align(kernel_range.start, buffer::PAGE_SIZE as u64) - ..align_down(kernel_range.end, buffer::PAGE_SIZE as u64); - - let kernel_half_size = align_down(kernel_range.range() >> 1, buffer::PAGE_SIZE as u64); - let kernel_gpu_range = kernel_range.start..(kernel_range.start + kernel_half_size); - let kernel_gpufw_range = kernel_gpu_range.end..kernel_range.end; - - let gpu = &device.gpu; - let file_id = file.inner().id; - let vm = gpu.new_vm(kernel_range.clone())?; - - let vm_xa = file.inner().vms(); - let resv = vm_xa.lock().reserve_limit(1..=u32::MAX, GFP_KERNEL)?; - let id: u32 = resv.index().try_into()?; - - mod_dev_dbg!(device, "[File {} VM {}]: VM Create\n", file_id, id); - mod_dev_dbg!( - device, - "[File {} VM {}]: Creating allocators\n", - file_id, - id - ); - let ualloc = Arc::pin_init( - new_mutex!(alloc::DefaultAllocator::new( - device, - &vm, - kernel_gpu_range, - buffer::PAGE_SIZE, - mmu::PROT_GPU_SHARED_RW, - 512 * 1024, - true, - fmt!("File {} VM {} GPU Shared", file_id, id), - false, - )?), - GFP_KERNEL, - )?; - let ualloc_priv = Arc::pin_init( - new_mutex!(alloc::DefaultAllocator::new( - device, - &vm, - kernel_gpufw_range, - buffer::PAGE_SIZE, - mmu::PROT_GPU_FW_PRIV_RW, - 64 * 1024, - true, - fmt!("File {} VM {} GPU FW Private", file_id, id), - false, - )?), - GFP_KERNEL, - )?; - - mod_dev_dbg!( - device, - "[File {} VM {}]: Creating dummy object\n", - file_id, - id - ); - let mut dummy_obj = gem::new_kernel_object(device, 0x4000)?; - dummy_obj.vmap()?.memset(0); - let dummy_mapping = dummy_obj.map_at( - &vm, - gpu.uat_geometry().unk_page(), - mmu::PROT_GPU_SHARED_RW, - true, - )?; - - mod_dev_dbg!(device, "[File {} VM {}]: VM created\n", file_id, id); - resv.fill(KBox::new( - Vm { - shrinker: if vm.residency_reclaim_enabled() { - Some(mmu::VmShrinker::new(&vm)?) - } else { - None - }, - ualloc, - ualloc_priv, - vm, - user_range, - kernel_range, - _dummy_mapping: dummy_mapping, - }, - GFP_KERNEL, - )?)?; - - data.vm_id = id; - - Ok(0) - } - - /// IOCTL: vm_destroy: Destroy a `Vm`. - pub(crate) fn vm_destroy( - _device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_vm_destroy, - file: &DrmFile, - ) -> Result { - let vm = file.inner().vms().remove(data.vm_id as usize); - if vm.is_none() { - Err(ENOENT) - } else { - Ok(0) - } - } - - /// IOCTL: gem_create: Create a new GEM object. - pub(crate) fn gem_create( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_create, - file: &DrmFile, - ) -> Result { - mod_dev_dbg!( - device, - "[File {}]: IOCTL: gem_create size={:#x?}\n", - file.inner().id, - data.size - ); - - if (data.flags - & !(uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_WRITEBACK - | uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE)) - != 0 - || (data.flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE == 0 - && data.vm_id != 0) - { - cls_pr_debug!(Errors, "gem_create: Invalid arguments\n"); - return Err(EINVAL); - } - - let resv_gem; - let mut reclaim_cpu = false; - let resv_obj = if data.flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE != 0 { - let vms = file.inner().vms(); - let vms = vms.lock(); - let vm = &vms.get(data.vm_id.try_into()?).ok_or(ENOENT)?.vm; - reclaim_cpu = vm.residency_reclaim_enabled(); - resv_gem = vm.get_resv_obj(); - Some(resv_gem.deref()) - } else { - None - }; - - let gem = gem::new_object(device, data.size.try_into()?, data.flags, resv_obj, reclaim_cpu)?; - - let handle = gem.create_handle(file)?; - data.handle = handle; - - mod_dev_dbg!( - device, - "[File {}]: IOCTL: gem_create size={:#x} handle={:#x?}\n", - file.inner().id, - data.size, - data.handle - ); - - Ok(0) - } - - /// IOCTL: gem_mmap_offset: Assign an mmap offset to a GEM object. - pub(crate) fn gem_mmap_offset( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_mmap_offset, - file: &DrmFile, - ) -> Result { - mod_dev_dbg!( - device, - "[File {}]: IOCTL: gem_mmap_offset handle={:#x?}\n", - file.inner().id, - data.handle - ); - - if data.flags != 0 { - cls_pr_debug!(Errors, "gem_mmap_offset: Unexpected flags\n"); - return Err(EINVAL); - } - - let gem = gem::Object::lookup_handle(file, data.handle)?; - data.offset = gem.create_mmap_offset()?; - Ok(0) - } - - /// IOCTL: gem_madvise: Mark a VM-private object purgeable or needed. - pub(crate) fn gem_madvise( - _device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_madvise, - file: &DrmFile, - ) -> Result { - if data.pad != 0 || data.madv > uapi::drm_asahi_neo_gem_madv_DRM_ASAHI_NEO_GEM_MADV_DONTNEED { - return Err(EINVAL); - } - let gem = gem::Object::lookup_handle(file, data.handle)?; - data.retained = gem::madvise( - &gem, - data.madv == uapi::drm_asahi_neo_gem_madv_DRM_ASAHI_NEO_GEM_MADV_DONTNEED, - )?; - Ok(0) - } - - /// IOCTL: vm_bind: Map or unmap memory into a Vm. - pub(crate) fn vm_bind( - device: &AsahiDevice, - data: &uapi::drm_asahi_neo_vm_bind, - file: &DrmFile, - ) -> Result { - mod_dev_dbg!( - device, - "[File {} VM {}]: IOCTL: vm_bind\n", - file.inner().id, - data.vm_id, - ); - - if data.stride == 0 || data.pad != 0 { - cls_pr_debug!(Errors, "vm_bind: Unexpected headers\n"); - return Err(EINVAL); - } - - if device.gpu.as_any().is::() { - return Self::vm_bind_g17(data, file); - } - - let vm_id = data.vm_id.try_into()?; - - let mut vec = KVec::new(); - let size = usize::try_from(data.stride)? - .checked_mul(usize::try_from(data.num_binds)?) - .ok_or(EOVERFLOW)?; - let reader = UserSlice::new(UserPtr::from_addr(data.userptr as _), size).reader(); - reader.read_all(&mut vec, GFP_KERNEL)?; - let mut reader = Reader::new(&vec); - - for _i in 0..data.num_binds { - let bind: uapi::drm_asahi_neo_gem_bind_op = reader.read_up_to(data.stride as usize)?; - Self::do_gem_bind_unbind(vm_id, &bind, file)?; - } - - Ok(0) - } - - /// Validate all operations and retain their objects before changing a tracked VM. - fn vm_bind_g17(data: &uapi::drm_asahi_neo_vm_bind, file: &DrmFile) -> Result { - let count = usize::try_from(data.num_binds)?; - let stride = usize::try_from(data.stride)?; - let size = stride.checked_mul(count).ok_or(EOVERFLOW)?; - let mut bytes = KVec::new(); - UserSlice::new(UserPtr::from_addr(data.userptr as _), size) - .reader() - .read_all(&mut bytes, GFP_KERNEL)?; - let (vm, user_range, kernel_range) = { - let vms = file.inner().vms(); - let guard = vms.lock(); - let entry = guard.get(data.vm_id as usize).ok_or(ENOENT)?; - ( - entry.vm.clone(), - entry.user_range.clone(), - entry.kernel_range.clone(), - ) - }; - let mut batch = mmu::PreparedUserBindBatch::new(count)?; - let mut unmaps = 0; - // A fresh reader per declared stride also skips extension bytes in longer records. - for bytes in bytes.chunks_exact(stride) { - let op: uapi::drm_asahi_neo_gem_bind_op = Reader::new(bytes).read_up_to(stride)?; - if op.range == 0 || (op.addr | op.range | op.offset) & mmu::UAT_PGMSK as u64 != 0 { - return Err(EINVAL); - } - let range = op.addr..op.addr.checked_add(op.range).ok_or(EINVAL)?; - if !user_range.is_superset(range.clone()) - || kernel_range.overlaps(range.clone()) - || vm.driver_range_overlaps(range.clone()) - { - return Err(EINVAL); - } - if op.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_UNBIND != 0 { - if op.offset != 0 - || op.flags != uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_UNBIND - || op.handle != 0 - { - return Err(EINVAL); - } - batch.push_unmap(vm.prepare_user_unmap(op.addr, op.range)?)?; - unmaps += 1; - continue; - } - vm.wait_for_user_map_admission()?; - if op.flags - & !(uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_READ - | uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_WRITE - | uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_SINGLE_PAGE) - != 0 - { - return Err(EINVAL); - } - let read = op.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_READ != 0; - let write = op.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_WRITE != 0; - let prot = match (read, write) { - (true, true) => mmu::PROT_GPU_SHARED_RW, - (true, false) => mmu::PROT_GPU_SHARED_RO, - (false, true) => mmu::PROT_GPU_SHARED_WO, - (false, false) => return Err(EINVAL), - }; - let single_page = op.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_SINGLE_PAGE != 0; - let object = gem::Object::lookup_handle(file, op.handle)?; - let accessed = if single_page { - mmu::UAT_PGSZ as u64 - } else { - op.range - }; - if op.offset.checked_add(accessed).ok_or(EINVAL)? > u64::try_from(object.size())? { - return Err(EINVAL); - } - vm.validate_context_binding(op.addr, op.range, single_page)?; - batch.push_map(vm.prepare_bind_object( - &object, - op.addr, - op.range, - op.offset, - prot, - single_page, - )?)?; - } - vm.reserve_deferred_user_unmaps(unmaps)?; - vm.commit_prepared_user_bind_batch(&mut batch)?; - vm.bo_deferred_cleanup(); - Ok(0) - } - - pub(crate) fn do_gem_bind_unbind( - vm_id: usize, - data: &uapi::drm_asahi_neo_gem_bind_op, - file: &DrmFile, - ) -> Result { - if data.range == 0 { - return Err(EINVAL); - } - if (data.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_UNBIND) != 0 { - Self::do_gem_unbind(vm_id, data, file) - } else { - Self::do_gem_bind(vm_id, data, file) - } - } - - pub(crate) fn do_gem_bind( - vm_id: usize, - data: &uapi::drm_asahi_neo_gem_bind_op, - file: &DrmFile, - ) -> Result { - if (data.addr | data.range | data.offset) as usize & mmu::UAT_PGMSK != 0 { - cls_pr_debug!( - Errors, - "gem_bind: Addr/range/offset not page aligned: {:#x} {:#x}\n", - data.addr, - data.range - ); - return Err(EINVAL); // Must be page aligned - } - - if (data.flags - & !(uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_READ - | uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_WRITE - | uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_SINGLE_PAGE)) - != 0 - { - cls_pr_debug!(Errors, "gem_bind: Invalid flags {:#x}\n", data.flags); - return Err(EINVAL); - } - - let single_page = data.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_SINGLE_PAGE != 0; - - let bo = gem::Object::lookup_handle(file, data.handle)?; - - let start = data.addr; - let end = data.addr.checked_add(data.range).ok_or(EINVAL)?; - let range = start..end; - - let bo_accessed_size = if single_page { - mmu::UAT_PGMSK as u64 - } else { - data.range - }; - let end_off = data.offset.checked_add(bo_accessed_size).ok_or(EINVAL)?; - if end_off as usize > bo.size() { - return Err(EINVAL); - } - - if !file.inner().user_range.is_superset(range.clone()) { - cls_pr_debug!( - Errors, - "gem_bind: Invalid map range {:#x}..{:#x} (not contained in user range)\n", - start, - end - ); - return Err(EINVAL); // Invalid map range - } - - let prot = if data.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_READ != 0 { - if data.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_WRITE != 0 { - mmu::PROT_GPU_SHARED_RW - } else { - mmu::PROT_GPU_SHARED_RO - } - } else if data.flags & uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_WRITE != 0 { - mmu::PROT_GPU_SHARED_WO - } else { - cls_pr_debug!( - Errors, - "gem_bind: Must specify read or write (flags: {:#x})\n", - data.flags - ); - return Err(EINVAL); // Must specify one of DRM_ASAHI_NEO_BIND_{READ,WRITE} - }; - - let vms_xa = file.inner().vms(); - let guard = vms_xa.lock(); - let guarded_vm = guard.get(vm_id).ok_or(ENOENT)?; - - // Clone it immediately so we aren't holding the XArray lock - let vm = guarded_vm.vm.clone(); - let kernel_range = guarded_vm.kernel_range.clone(); - let _ = guarded_vm; - core::mem::drop(guard); - - if kernel_range.overlaps(range) { - cls_pr_debug!( - Errors, - "gem_bind: Invalid map range {:#x}..{:#x} (intrudes in kernel range)\n", - start, - end - ); - return Err(EINVAL); - } - - vm.bind_object(&bo, data.addr, data.range, data.offset, prot, single_page)?; - - vm.bo_deferred_cleanup(); - - Ok(0) - } - - pub(crate) fn do_gem_unbind( - vm_id: usize, - data: &uapi::drm_asahi_neo_gem_bind_op, - file: &DrmFile, - ) -> Result { - if data.offset != 0 - || data.flags != uapi::drm_asahi_neo_bind_flags_DRM_ASAHI_NEO_BIND_UNBIND - || data.handle != 0 - { - cls_pr_debug!(Errors, "gem_unbind: offset/flags/handle not zero\n"); - return Err(EINVAL); - } - - if (data.addr | data.range) as usize & mmu::UAT_PGMSK != 0 { - cls_pr_debug!( - Errors, - "gem_bind: Addr/range/offset not page aligned: {:#x} {:#x}\n", - data.addr, - data.range - ); - return Err(EINVAL); // Must be page aligned - } - - let start = data.addr; - let end = data.addr.checked_add(data.range).ok_or(EINVAL)?; - let range = start..end; - - if !file.inner().user_range.is_superset(range.clone()) { - cls_pr_debug!( - Errors, - "gem_bind: Invalid unmap range {:#x}..{:#x} (not contained in user range)\n", - start, - end - ); - return Err(EINVAL); // Invalid map range - } - - let vms_xa = file.inner().vms(); - let guard = vms_xa.lock(); - let guarded_vm = guard.get(vm_id).ok_or(ENOENT)?; - - // Clone it immediately so we aren't holding the XArray lock - let vm = guarded_vm.vm.clone(); - let kernel_range = guarded_vm.kernel_range.clone(); - let _ = guarded_vm; - core::mem::drop(guard); - - if kernel_range.overlaps(range.clone()) { - cls_pr_debug!( - Errors, - "gem_bind: Invalid unmap range {:#x}..{:#x} (intrudes in kernel range)\n", - start, - end - ); - return Err(EINVAL); - } - - vm.unmap_range(range.start, range.range())?; - - vm.bo_deferred_cleanup(); - - Ok(0) - } - - pub(crate) fn unbind_gem_object(file: &DrmFile, bo: &gem::Object) -> Result { - // TODO: use iter() - let mut index = 0; - loop { - let vms = file.inner().vms(); - let item = vms.find(index, usize::MAX); - match item { - Some((idx, file_vm)) => { - // Clone since we can't hold the xarray spinlock while - // calling drop_mappings() - let vm = file_vm.borrow().vm.clone(); - core::mem::drop(file_vm); - vm.drop_mappings(bo)?; - if idx == usize::MAX { - break; - } - index = idx + 1; - } - None => break, - } - } - Ok(()) - } - - /// IOCTL: gem_bind_object: Map or unmap a GEM object as a special object. - pub(crate) fn gem_bind_object( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_bind_object, - file: &DrmFile, - ) -> Result { - mod_dev_dbg!( - device, - "[File {} VM {}]: IOCTL: gem_bind_object op={:?} handle={:#x?} flags={:#x?} {:#x?}:{:#x?} object_handle={:#x?}\n", - file.inner().id, - data.vm_id, - data.op, - data.handle, - data.flags, - data.offset, - data.range, - data.object_handle - ); - - if data.pad != 0 { - cls_pr_debug!(Errors, "gem_bind_object: Unexpected pad\n"); - return Err(EINVAL); - } - - if data.vm_id != 0 { - cls_pr_debug!(Errors, "gem_bind_object: Unexpected vm_id\n"); - return Err(EINVAL); - } - - match data.op { - uapi::drm_asahi_neo_bind_object_op_DRM_ASAHI_NEO_BIND_OBJECT_OP_BIND => { - Self::do_gem_bind_object(device, data, file) - } - uapi::drm_asahi_neo_bind_object_op_DRM_ASAHI_NEO_BIND_OBJECT_OP_UNBIND => { - Self::do_gem_unbind_object(device, data, file) - } - _ => { - cls_pr_debug!(Errors, "gem_bind_object: Invalid op {}\n", data.op); - Err(EINVAL) - } - } - } - - pub(crate) fn do_gem_bind_object( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_bind_object, - file: &DrmFile, - ) -> Result { - if (data.range | data.offset) as usize & mmu::UAT_PGMSK != 0 { - cls_pr_debug!( - Errors, - "gem_bind_object: Range/offset not page aligned: {:#x} {:#x}\n", - data.range, - data.offset - ); - return Err(EINVAL); // Must be page aligned - } - - if data.flags != uapi::drm_asahi_neo_bind_object_flags_DRM_ASAHI_NEO_BIND_OBJECT_USAGE_TIMESTAMPS { - cls_pr_debug!(Errors, "gem_bind_object: Invalid flags {:#x}\n", data.flags); - return Err(EINVAL); - } - - let offset = data.offset.try_into()?; - let end_offset = data - .offset - .checked_add(data.range) - .ok_or(EINVAL)? - .try_into()?; - let bo = gem::ObjectRef::new(gem::Object::lookup_handle(file, data.handle)?); - - let mapping = Arc::new( - device.gpu.map_timestamp_buffer(bo, offset..end_offset)?, - GFP_KERNEL, - )?; - let obj = KBox::new(Object::TimestampBuffer(mapping), GFP_KERNEL)?; - let handle = file - .inner() - .objects() - .lock() - .insert_limit(1..=u32::MAX, obj, GFP_KERNEL); - // StoreError owns the rejected object. Convert it only after the - // temporary XArray guard is gone: dropping a mapping may sleep. - let handle = handle? as u64; - - data.object_handle = handle as u32; - Ok(0) - } - - pub(crate) fn do_gem_unbind_object( - _device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_gem_bind_object, - file: &DrmFile, - ) -> Result { - if data.range != 0 || data.offset != 0 { - cls_pr_debug!( - Errors, - "gem_unbind_object: Range/offset not zero: {:#x} {:#x}\n", - data.range, - data.offset - ); - return Err(EINVAL); - } - - if data.flags != 0 { - cls_pr_debug!( - Errors, - "gem_unbind_object: Invalid flags {:#x}\n", - data.flags - ); - return Err(EINVAL); - } - - if data.handle != 0 { - cls_pr_debug!( - Errors, - "gem_unbind_object: Invalid handle {}\n", - data.handle - ); - return Err(EINVAL); - } - - let object = file.inner().objects().remove(data.object_handle as usize); - if object.is_none() { - Err(ENOENT) - } else { - Ok(0) - } - } - - /// IOCTL: queue_create: Create a new command submission queue of a given type. - pub(crate) fn queue_create( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_queue_create, - file: &DrmFile, - ) -> Result { - let file_id = file.inner().id; - - mod_dev_dbg!( - device, - "[File {} VM {}]: Creating queue prio={:?} flags={:#x?}\n", - file_id, - data.vm_id, - data.priority, - data.flags, - ); - - if data.flags != 0 || data.priority > uapi::drm_asahi_neo_priority_DRM_ASAHI_NEO_PRIORITY_REALTIME { - cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); - return Err(EINVAL); - } - - // TODO: Allow with CAP_SYS_NICE - if data.priority >= uapi::drm_asahi_neo_priority_DRM_ASAHI_NEO_PRIORITY_HIGH { - cls_pr_debug!(Errors, "queue_create: Invalid priority\n"); - return Err(EINVAL); - } - - let queues_xa = file.inner().queues(); - let resv = queues_xa.lock().reserve_limit(1..=u32::MAX, GFP_KERNEL)?; - let vms_xa = file.inner().vms(); - let guard = vms_xa.lock(); - let file_vm = guard.get(data.vm_id.try_into()?).ok_or(ENOENT)?; - let vm = file_vm.vm.clone(); - let ualloc = file_vm.ualloc.clone(); - let ualloc_priv = file_vm.ualloc_priv.clone(); - // Drop the vms lock eagerly - let _ = file_vm; - core::mem::drop(guard); - - let queue = device.gpu.new_queue( - vm, - ualloc, - ualloc_priv, - // TODO: Plumb deeper the enum - uapi::drm_asahi_neo_priority_DRM_ASAHI_NEO_PRIORITY_REALTIME - data.priority, - data.usc_exec_base, - )?; - - data.queue_id = resv.index().try_into()?; - resv.fill(Arc::pin_init(new_mutex!(queue), GFP_KERNEL)?)?; - - Ok(0) - } - - /// IOCTL: queue_destroy: Destroy a command submission queue. - pub(crate) fn queue_destroy( - _device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_queue_destroy, - file: &DrmFile, - ) -> Result { - // grab the queue so the xarray spinlock is dropped first - let queue = file.inner().queues().remove(data.queue_id as usize); - if queue.is_none() { - Err(ENOENT) - } else { - Ok(0) - } - } - - /// IOCTL: submit: Submit GPU work to a command submission queue. - pub(crate) fn submit( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_submit, - file: &DrmFile, - ) -> Result { - debug::update_debug_flags(); - - if data.flags != 0 || data.pad != 0 { - cls_pr_debug!(Errors, "submit: Invalid arguments\n"); - return Err(EINVAL); - } - - let gpu = &device.gpu; - if let Some(manager) = gpu.manager() { - manager.update_globals(); - } - - // Upgrade to Arc to drop the XArray lock early - let queue: Arc>> = file - .inner() - .queues() - .lock() - .get(data.queue_id.try_into()?) - .ok_or(ENOENT)? - .into(); - - let id = gpu.ids().submission.next(); - mod_dev_dbg!( - device, - "[File {} Queue {}]: IOCTL: submit (submission ID: {})\n", - file.inner().id, - data.queue_id, - id - ); - - mod_dev_dbg!( - device, - "[File {} Queue {}]: IOCTL: submit({}): Parsing syncs\n", - file.inner().id, - data.queue_id, - id - ); - let mut vec = KVec::new(); - let copy_commands = |bytes: &mut KVec| { - UserSlice::new( - UserPtr::from_addr(data.cmdbuf as _), - data.cmdbuf_size as usize, - ) - .reader() - .read_all(bytes, GFP_KERNEL) - }; - let guarded = gpu.as_any().is::(); - // Once G17 output syncobjs are resolved, its queue must install a fence - // on every failure. Take the fallible userspace snapshot before that point. - if guarded { - copy_commands(&mut vec)?; - } - let syncs = - SyncItem::parse_array(file, data.syncs, data.in_sync_count, data.out_sync_count)?; - - mod_dev_dbg!( - device, - "[File {} Queue {}]: IOCTL: submit({}): Parsing commands\n", - file.inner().id, - data.queue_id, - id - ); - - // Copy the command buffer into the kernel. Because we need to iterate - // the command buffer twice, we do this in one big copy_from_user to - // avoid TOCTOU issues. - if !guarded { - copy_commands(&mut vec)?; - } - - let objects = file.inner().objects(); - let ret = queue - .lock() - .submit(id, syncs, data.in_sync_count as usize, &vec, objects); - - if ret.is_ok() { - if let Some((util_min, duration_ms)) = gpu.syncobj_wait_hint() { - file.set_syncobj_wait_hint(util_min, duration_ms); - } - } - - match ret { - Err(ERESTARTSYS) => Err(ERESTARTSYS), - Err(e) => { - dev_info!( - device.as_ref(), - "[File {} Queue {}]: IOCTL: submit failed! (submission ID: {} err: {:?})\n", - file.inner().id, - data.queue_id, - id, - e - ); - Err(e) - } - Ok(()) => Ok(0), - } - } - - /// IOCTL: get_time: Get the current GPU timer value. - pub(crate) fn get_time( - device: &AsahiDevice, - data: &mut uapi::drm_asahi_neo_get_time, - _file: &DrmFile, - ) -> Result { - if data.flags != 0 { - cls_pr_debug!(Errors, "get_time: Unexpected flags\n"); - return Err(EINVAL); - } - - // TODO: Do this on device-init for perf. - let gpu = &device.gpu; - let frequency_hz = gpu.base_clock_hz() as u64; - let ts_gcd = gcd(frequency_hz, NSEC_PER_SEC as u64); - - let num = (NSEC_PER_SEC as u64) / ts_gcd; - let den = frequency_hz / ts_gcd; - - let raw: u64; - - // SAFETY: Assembly only loads the timer - unsafe { - core::arch::asm!( - "mrs {x}, CNTPCT_EL0", - x = out(reg) raw - ); - } - - data.gpu_timestamp = (raw * num) / den; - - Ok(0) - } -} - -impl Drop for File { - fn drop(&mut self) { - mod_pr_debug!("[File {}]: Closing...\n", self.id); - } -} diff --git a/drivers/gpu/drm/asahi-neo/float.rs b/drivers/gpu/drm/asahi-neo/float.rs deleted file mode 100644 index 46c05699d77225..00000000000000 --- a/drivers/gpu/drm/asahi-neo/float.rs +++ /dev/null @@ -1,388 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Basic soft floating-point support -//! -//! The GPU firmware requires a large number of power-related configuration values, many of which -//! are IEEE 754 32-bit floating point values. These values change not only between GPU/SoC -//! variants, but also between specific hardware platforms using these SoCs, so they must be -//! derived from device tree properties. There are many redundant values computed from the same -//! inputs with simple add/sub/mul/div calculations, plus a few values that are actually specific -//! to each individual device depending on its binning and fused voltage configuration, so it -//! doesn't make sense to store the final values to be passed to the firmware in the device tree. -//! -//! Therefore, we need a way to perform floating-point calculations in the kernel. -//! -//! Using the actual FPU from kernel mode is asking for trouble, since there is no way to bound -//! the execution of FPU instructions to a controlled section of code without outright putting it -//! in its own compilation unit, which is quite painful for Rust. Since these calculations only -//! have to happen at initialization time and there is no need for performance, let's use a simple -//! software float implementation instead. -//! -//! This implementation makes no attempt to be fully IEEE754 compliant, but it's good enough and -//! gives bit-identical results to macOS in the vast majority of cases, with one or two exceptions -//! related to slightly non-compliant rounding. - -use core::ops; -use kernel::{ - of, - prelude::*, // -}; - -/// An IEEE754-compatible floating point number implemented in software. -#[derive(Default, Debug, Copy, Clone)] -#[repr(transparent)] -pub(crate) struct F32(u32); - -// SAFETY: F32 is a transparent repr of `u32` and therefore zeroable -unsafe impl Zeroable for F32 {} - -#[derive(Default, Debug, Copy, Clone)] -struct F32U { - sign: bool, - exp: i32, - frac: i64, -} - -impl F32 { - /// Convert a raw 32-bit representation into an F32 - pub(crate) const fn from_bits(u: u32) -> F32 { - F32(u) - } - - // Convert a `f32` value into an F32 - // - // This must ONLY be used in const context. Use the `f32!{}` macro to do it safely. - #[doc(hidden)] - pub(crate) const fn from_f32(v: f32) -> F32 { - F32(v.to_bits()) - } - - // Convert an F32 into a `f32` value - // - // For testing only. - #[doc(hidden)] - #[cfg(test)] - pub(crate) fn to_f32(self) -> f32 { - f32::from_bits(self.0) - } - - const fn unpack(&self) -> F32U { - F32U { - sign: self.0 & (1 << 31) != 0, - exp: ((self.0 >> 23) & 0xff) as i32 - 127, - frac: (((self.0 & 0x7fffff) | 0x800000) as i64) << 9, - } - .norm() - } -} - -/// Safely construct an `F32` out of a constant floating-point value. -/// -/// This ensures that the conversion happens in const context, so no floating point operations are -/// emitted. -#[macro_export] -macro_rules! f32 { - ([$($val:expr),*]) => {{ - [$(f32!($val)),*] - }}; - ($val:expr) => {{ - const _K: $crate::float::F32 = $crate::float::F32::from_f32($val); - _K - }}; -} - -impl ops::Neg for F32 { - type Output = F32; - - fn neg(self) -> F32 { - F32(self.0 ^ (1 << 31)) - } -} - -impl ops::Add for F32 { - type Output = F32; - - fn add(self, rhs: F32) -> F32 { - self.unpack().add(rhs.unpack()).pack() - } -} - -impl ops::Sub for F32 { - type Output = F32; - - fn sub(self, rhs: F32) -> F32 { - self.unpack().add((-rhs).unpack()).pack() - } -} - -impl ops::Mul for F32 { - type Output = F32; - - fn mul(self, rhs: F32) -> F32 { - self.unpack().mul(rhs.unpack()).pack() - } -} - -impl ops::Div for F32 { - type Output = F32; - - fn div(self, rhs: F32) -> F32 { - self.unpack().div(rhs.unpack()).pack() - } -} - -macro_rules! from_ints { - ($u:ty, $i:ty) => { - impl From<$i> for F32 { - fn from(v: $i) -> F32 { - F32U::from_i64(v as i64).pack() - } - } - impl From<$u> for F32 { - fn from(v: $u) -> F32 { - F32U::from_u64(v as u64).pack() - } - } - }; -} - -from_ints!(u8, i8); -from_ints!(u16, i16); -from_ints!(u32, i32); -from_ints!(u64, i64); - -impl F32U { - const INFINITY: F32U = f32!(f32::INFINITY).unpack(); - const NEG_INFINITY: F32U = f32!(f32::NEG_INFINITY).unpack(); - - fn from_i64(v: i64) -> F32U { - F32U { - sign: v < 0, - exp: 32, - frac: v.abs(), - } - .norm() - } - - fn from_u64(mut v: u64) -> F32U { - let mut exp = 32; - if v >= (1 << 63) { - exp = 31; - v >>= 1; - } - F32U { - sign: false, - exp, - frac: v as i64, - } - .norm() - } - - fn shr(&mut self, shift: i32) { - if shift > 63 { - self.exp = 0; - self.frac = 0; - } else { - self.frac >>= shift; - } - } - - fn align(a: &mut F32U, b: &mut F32U) { - if a.exp > b.exp { - b.shr(a.exp - b.exp); - b.exp = a.exp; - } else { - a.shr(b.exp - a.exp); - a.exp = b.exp; - } - } - - fn mul(self, other: F32U) -> F32U { - F32U { - sign: self.sign != other.sign, - exp: self.exp + other.exp, - frac: ((self.frac >> 8) * (other.frac >> 8)) >> 16, - } - } - - fn div(self, other: F32U) -> F32U { - if other.frac == 0 || self.is_inf() { - if self.sign { - F32U::NEG_INFINITY - } else { - F32U::INFINITY - } - } else { - F32U { - sign: self.sign != other.sign, - exp: self.exp - other.exp, - frac: ((self.frac << 24) / (other.frac >> 8)), - } - } - } - - fn add(mut self, mut other: F32U) -> F32U { - F32U::align(&mut self, &mut other); - if self.sign == other.sign { - self.frac += other.frac; - } else { - self.frac -= other.frac; - } - if self.frac < 0 { - self.sign = !self.sign; - self.frac = -self.frac; - } - self - } - - const fn norm(mut self) -> F32U { - let lz = self.frac.leading_zeros() as i32; - if lz > 31 { - self.frac <<= lz - 31; - self.exp -= lz - 31; - } else if lz < 31 { - self.frac >>= 31 - lz; - self.exp += 31 - lz; - } - - if self.is_zero() { - return F32U { - sign: self.sign, - frac: 0, - exp: 0, - }; - } - self - } - - const fn is_zero(&self) -> bool { - self.frac == 0 || self.exp < -126 - } - - const fn is_inf(&self) -> bool { - self.exp > 127 - } - - const fn pack(mut self) -> F32 { - self = self.norm(); - if !self.is_zero() { - self.frac += 0x100; - self = self.norm(); - } - - if self.is_inf() { - if self.sign { - return f32!(f32::NEG_INFINITY); - } else { - return f32!(f32::INFINITY); - } - } else if self.is_zero() { - if self.sign { - return f32!(-0.0); - } else { - return f32!(0.0); - } - } - - F32(if self.sign { 1u32 << 31 } else { 0u32 } - | ((self.exp + 127) as u32) << 23 - | ((self.frac >> 9) & 0x7fffff) as u32) - } -} - -impl<'a> TryFrom> for F32 { - type Error = Error; - - fn try_from(p: of::Property<'_>) -> core::result::Result { - let bits: u32 = p.try_into()?; - Ok(F32::from_bits(bits)) - } -} - -impl of::PropertyUnit for F32 { - const UNIT_SIZE: usize = 4; - - fn from_bytes(data: &[u8]) -> Result { - Ok(F32::from_bits(::from_bytes(data)?)) - } -} - -// TODO: Make this an actual test and figure out how to make it run. -#[cfg(test)] -mod tests { - #[test] - fn test_all() { - fn add(a: f32, b: f32) { - println!( - "{} + {} = {} {}", - a, - b, - (F32::from_f32(a) + F32::from_f32(b)).to_f32(), - a + b - ); - } - fn sub(a: f32, b: f32) { - println!( - "{} - {} = {} {}", - a, - b, - (F32::from_f32(a) - F32::from_f32(b)).to_f32(), - a - b - ); - } - fn mul(a: f32, b: f32) { - println!( - "{} * {} = {} {}", - a, - b, - (F32::from_f32(a) * F32::from_f32(b)).to_f32(), - a * b - ); - } - fn div(a: f32, b: f32) { - println!( - "{} / {} = {} {}", - a, - b, - (F32::from_f32(a) / F32::from_f32(b)).to_f32(), - a / b - ); - } - - fn test(a: f32, b: f32) { - add(a, b); - sub(a, b); - mul(a, b); - div(a, b); - } - - test(1.123, 7.567); - test(1.123, 1.456); - test(7.567, 1.123); - test(1.123, -7.567); - test(1.123, -1.456); - test(7.567, -1.123); - test(-1.123, -7.567); - test(-1.123, -1.456); - test(-7.567, -1.123); - test(1000.123, 0.001); - test(1000.123, 0.0000001); - test(0.0012, 1000.123); - test(0.0000001, 1000.123); - test(0., 0.); - test(0., 1.); - test(1., 0.); - test(1., 1.); - test(2., f32::INFINITY); - test(2., f32::NEG_INFINITY); - test(f32::INFINITY, 2.); - test(f32::NEG_INFINITY, 2.); - test(f32::NEG_INFINITY, 2.); - test(f32::MAX, 2.); - test(f32::MIN, 2.); - test(f32::MIN_POSITIVE, 2.); - test(2., f32::MAX); - test(2., f32::MIN); - test(2., f32::MIN_POSITIVE); - } -} diff --git a/drivers/gpu/drm/asahi-neo/fw/buffer.rs b/drivers/gpu/drm/asahi-neo/fw/buffer.rs deleted file mode 100644 index 04892ddd06fc0b..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/buffer.rs +++ /dev/null @@ -1,184 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU tiled vertex buffer control firmware structures - -use super::types::*; -use super::workqueue; -use crate::{ - default_zeroed, - no_debug, - trivial_gpustruct, // -}; -use kernel::sync::Arc; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct BlockControl { - pub(crate) total: AtomicU32, - pub(crate) wptr: AtomicU32, - pub(crate) unk: AtomicU32, - pub(crate) pad: Pad<0x34>, - } - default_zeroed!(BlockControl); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Counter { - pub(crate) count: AtomicU32, - __pad: Pad<0x3c>, - } - default_zeroed!(Counter); - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct Stats { - pub(crate) max_pages: AtomicU32, - pub(crate) max_b: AtomicU32, - pub(crate) overflow_count: AtomicU32, - pub(crate) gpu_c: AtomicU32, - pub(crate) __pad0: Pad<0x10>, - pub(crate) reset: AtomicU32, - pub(crate) __pad1: Pad<0x1c>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Info<'a> { - pub(crate) gpu_counter: u32, - pub(crate) unk_4: u32, - pub(crate) last_id: i32, - pub(crate) cur_id: i32, - pub(crate) unk_10: u32, - pub(crate) gpu_counter2: u32, - pub(crate) unk_18: u32, - - #[ver(V < V13_0B4 || G >= G14X)] - pub(crate) unk_1c: u32, - - pub(crate) page_list: GpuPointer<'a, &'a [u32]>, - pub(crate) page_list_size: u32, - pub(crate) page_count: AtomicU32, - pub(crate) max_blocks: u32, - pub(crate) block_count: AtomicU32, - pub(crate) unk_38: u32, - pub(crate) block_list: GpuPointer<'a, &'a [u32]>, - pub(crate) block_ctl: GpuPointer<'a, super::BlockControl>, - pub(crate) last_page: AtomicU32, - pub(crate) gpu_page_ptr1: u32, - pub(crate) gpu_page_ptr2: u32, - pub(crate) unk_58: u32, - pub(crate) block_size: u32, - pub(crate) unk_60: U64, - pub(crate) counter: GpuPointer<'a, super::Counter>, - pub(crate) unk_70: u32, - pub(crate) unk_74: u32, - pub(crate) unk_78: u32, - pub(crate) unk_7c: u32, - pub(crate) unk_80: u32, - pub(crate) max_pages: u32, - pub(crate) max_pages_nomemless: u32, - pub(crate) unk_8c: u32, - pub(crate) unk_90: Array<0x30, u8>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Scene<'a> { - #[ver(G >= G14X)] - pub(crate) control_word: GpuPointer<'a, &'a [u32]>, - #[ver(G >= G14X)] - pub(crate) control_word2: GpuPointer<'a, &'a [u32]>, - pub(crate) pass_page_count: AtomicU32, - pub(crate) unk_4: u32, - pub(crate) unk_8: U64, - pub(crate) unk_10: U64, - pub(crate) user_buffer: GpuPointer<'a, &'a [u8]>, - pub(crate) unk_20: u32, - #[ver(V >= V13_3)] - pub(crate) unk_28: U64, - pub(crate) stats: GpuWeakPointer, - pub(crate) total_page_count: AtomicU32, - #[ver(G < G14X)] - pub(crate) unk_30: U64, // pad - #[ver(G < G14X)] - pub(crate) unk_38: U64, // pad - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct InitBuffer<'a> { - pub(crate) tag: workqueue::CommandType, - pub(crate) vm_slot: u32, - pub(crate) buffer_slot: u32, - pub(crate) unk_c: u32, - pub(crate) block_count: u32, - pub(crate) buffer: GpuPointer<'a, super::Info::ver>, - pub(crate) stamp_value: EventValue, - } -} - -trivial_gpustruct!(BlockControl); -trivial_gpustruct!(Counter); -trivial_gpustruct!(Stats); - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct Info { - pub(crate) block_ctl: GpuObject, - pub(crate) counter: GpuObject, - pub(crate) page_list: GpuArray, - pub(crate) block_list: GpuArray, -} - -#[versions(AGX_NEO)] -impl GpuStruct for Info::ver { - type Raw<'a> = raw::Info::ver<'a>; -} - -pub(crate) struct ClusterBuffers { - pub(crate) tilemaps: GpuArray, - pub(crate) meta: GpuArray, -} - -#[versions(AGX_NEO)] -pub(crate) struct Scene { - pub(crate) user_buffer: GpuArray, - pub(crate) buffer: crate::buffer::Buffer::ver, - pub(crate) tvb_heapmeta: GpuArray, - pub(crate) tvb_tilemap: GpuArray, - pub(crate) tpc: Arc>, - pub(crate) clustering: Option, - pub(crate) preempt_buf: GpuArray, - #[ver(G >= G14X)] - pub(crate) control_word: GpuArray, -} - -#[versions(AGX_NEO)] -no_debug!(Scene::ver); - -#[versions(AGX_NEO)] -impl GpuStruct for Scene::ver { - type Raw<'a> = raw::Scene::ver<'a>; -} - -#[versions(AGX_NEO)] -pub(crate) struct InitBuffer { - pub(crate) scene: Arc, -} - -#[versions(AGX_NEO)] -no_debug!(InitBuffer::ver); - -#[versions(AGX_NEO)] -impl workqueue::Command for InitBuffer::ver {} - -#[versions(AGX_NEO)] -impl GpuStruct for InitBuffer::ver { - type Raw<'a> = raw::InitBuffer::ver<'a>; -} diff --git a/drivers/gpu/drm/asahi-neo/fw/channels.rs b/drivers/gpu/drm/asahi-neo/fw/channels.rs deleted file mode 100644 index 65d7c6367bedd8..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/channels.rs +++ /dev/null @@ -1,443 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU communication channel firmware structures (ring buffers) - -use super::types::*; -use crate::default_zeroed; -use core::sync::atomic::Ordering; -use kernel::static_assert; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct ChannelState<'a> { - pub(crate) read_ptr: AtomicU32, - __pad0: Pad<0x1c>, - pub(crate) write_ptr: AtomicU32, - __pad1: Pad<0xc>, - _p: PhantomData<&'a ()>, - } - default_zeroed!(<'a>, ChannelState<'a>); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct FwCtlChannelState<'a> { - pub(crate) read_ptr: AtomicU32, - __pad0: Pad<0xc>, - pub(crate) write_ptr: AtomicU32, - __pad1: Pad<0xc>, - _p: PhantomData<&'a ()>, - } - default_zeroed!(<'a>, FwCtlChannelState<'a>); -} - -pub(crate) trait RxChannelState: GpuStruct + Debug + Default -where - for<'a> ::Raw<'a>: Default + Zeroable, -{ - const SUB_CHANNELS: usize; - - fn wptr(raw: &Self::Raw<'_>, index: usize) -> u32; - fn set_rptr(raw: &Self::Raw<'_>, index: usize, rptr: u32); -} - -#[derive(Debug, Default)] -pub(crate) struct ChannelState {} - -impl GpuStruct for ChannelState { - type Raw<'a> = raw::ChannelState<'a>; -} - -impl RxChannelState for ChannelState { - const SUB_CHANNELS: usize = 1; - - fn wptr(raw: &Self::Raw<'_>, _index: usize) -> u32 { - raw.write_ptr.load(Ordering::Acquire) - } - - fn set_rptr(raw: &Self::Raw<'_>, _index: usize, rptr: u32) { - raw.read_ptr.store(rptr, Ordering::Release); - } -} - -#[derive(Debug, Default)] -pub(crate) struct FwLogChannelState {} - -impl GpuStruct for FwLogChannelState { - type Raw<'a> = Array<6, raw::ChannelState<'a>>; -} - -impl RxChannelState for FwLogChannelState { - const SUB_CHANNELS: usize = 6; - - fn wptr(raw: &Self::Raw<'_>, index: usize) -> u32 { - raw[index].write_ptr.load(Ordering::Acquire) - } - - fn set_rptr(raw: &Self::Raw<'_>, index: usize, rptr: u32) { - raw[index].read_ptr.store(rptr, Ordering::Release); - } -} - -#[derive(Debug, Default)] -pub(crate) struct FwCtlChannelState {} - -impl GpuStruct for FwCtlChannelState { - type Raw<'a> = raw::FwCtlChannelState<'a>; -} - -pub(crate) trait TxChannelState: GpuStruct + Debug + Default { - fn rptr(raw: &Self::Raw<'_>) -> u32; - fn set_wptr(raw: &Self::Raw<'_>, wptr: u32); -} - -impl TxChannelState for ChannelState { - fn rptr(raw: &Self::Raw<'_>) -> u32 { - raw.read_ptr.load(Ordering::Acquire) - } - - fn set_wptr(raw: &Self::Raw<'_>, wptr: u32) { - raw.write_ptr.store(wptr, Ordering::Release); - } -} - -impl TxChannelState for FwCtlChannelState { - fn rptr(raw: &Self::Raw<'_>) -> u32 { - raw.read_ptr.load(Ordering::Acquire) - } - - fn set_wptr(raw: &Self::Raw<'_>, wptr: u32) { - raw.write_ptr.store(wptr, Ordering::Release); - } -} - -#[derive(Debug, Copy, Clone, Default, PartialEq, Eq)] -#[repr(u32)] -pub(crate) enum PipeType { - #[default] - Vertex = 0, - Fragment = 1, - Compute = 2, -} - -#[versions(AGX_NEO)] -#[derive(Debug, Copy, Clone, Default)] -#[repr(C)] -pub(crate) struct RunWorkQueueMsg { - pub(crate) pipe_type: PipeType, - pub(crate) work_queue: Option>, - pub(crate) wptr: u32, - pub(crate) event_slot: u32, - pub(crate) is_new: bool, - #[ver(V >= V13_2 && G == G14)] - pub(crate) __pad: Pad<0x2b>, - #[ver(V < V13_2 || G != G14)] - pub(crate) __pad: Pad<0x1b>, -} - -#[versions(AGX_NEO)] -pub(crate) type PipeMsg = RunWorkQueueMsg::ver; - -#[versions(AGX_NEO)] -pub(crate) const DEVICECONTROL_SZ: usize = { - #[ver(V < V13_2 || G != G14)] - { - 0x2c - } - #[ver(V >= V13_2 && G == G14)] - { - 0x3c - } -}; - -// TODO: clean up when arbitrary_enum_discriminant is stable -// https://github.com/rust-lang/rust/issues/60553 - -#[versions(AGX_NEO)] -#[derive(Debug, Copy, Clone)] -#[repr(C, u32)] -#[allow(dead_code)] -pub(crate) enum DeviceControlMsg { - Unk00(Array), - Unk01(Array), - Unk02(Array), - Unk03(Array), - Unk04(Array), - Unk05(Array), - Unk06(Array), - Unk07(Array), - Unk08(Array), - Unk09(Array), - Unk0a(Array), - Unk0b(Array), - Unk0c(Array), - #[ver(V >= V13_3)] - Unk0d(Array), - GrowTVBAck { - unk_4: u32, - buffer_slot: u32, - vm_slot: u32, - counter: u32, - subpipe: u32, - halt_count: U64, - __pad: Pad<{ DEVICECONTROL_SZ::ver - 0x1c }>, - }, - RecoverChannel { - pipe_type: u32, - work_queue: GpuWeakPointer, - event_value: u32, - __pad: Pad<{ DEVICECONTROL_SZ::ver - 0x10 }>, - }, - IdlePowerOff { - val: u32, - __pad: Pad<{ DEVICECONTROL_SZ::ver - 0x4 }>, - }, - Unk10(Array), - Unk11(Array), - Unk12(Array), - Unk13(Array), - Unk14(Array), // Init? - Unk15(Array), // Enable something - Unk16(Array), // Disable something - DestroyContext { - unk_4: u32, - ctx_23: u8, - #[ver(V < V13_3)] - __pad0: Pad<3>, - unk_c: U32, - unk_10: U32, - ctx_0: u8, - ctx_1: u8, - ctx_4: u8, - #[ver(V < V13_3)] - __pad1: Pad<1>, - #[ver(V < V13_3)] - unk_18: u32, - gpu_context: Option>, - #[ver(V < V13_3)] - __pad2: Pad<{ DEVICECONTROL_SZ::ver - 0x20 }>, - #[ver(V >= V13_3)] - __pad2: Pad<{ DEVICECONTROL_SZ::ver - 0x18 }>, - }, - Unk18(Array), - Initialize(Pad), // Update RegionC -} - -#[versions(AGX_NEO)] -static_assert!(core::mem::size_of::() == 4 + DEVICECONTROL_SZ::ver); - -#[versions(AGX_NEO)] -default_zeroed!(DeviceControlMsg::ver); - -#[derive(Copy, Clone, Default, Debug)] -#[repr(C)] -#[allow(dead_code)] -pub(crate) struct FwCtlMsg { - pub(crate) addr: U64, - pub(crate) unk_8: u32, - pub(crate) slot: u32, - pub(crate) page_count: u16, - pub(crate) unk_12: u16, -} - -pub(crate) const EVENT_SZ: usize = 0x34; - -#[derive(Debug, Copy, Clone, PartialEq, Eq)] -#[repr(C, u32)] -#[allow(dead_code)] -pub(crate) enum ChannelErrorType { - MemoryError, - DMKill, - Aborted, - Unk3, - Unknown(u32), -} - -#[derive(Debug, Copy, Clone)] -#[repr(C, u32)] -#[allow(dead_code)] -pub(crate) enum EventMsg { - Fault, - Flag { - firing: [u32; 4], - unk_14: u16, - }, - Unk2(Array), - Unk3(Array), - Timeout { - counter: u32, - unk_8: u32, - event_slot: i32, - }, - Unk5(Array), - Unk6(Array), - GrowTVB { - vm_slot: u32, - buffer_slot: u32, - counter: u32, - }, - ChannelError { - error_type: u32, - pipe_type: u32, - event_slot: u32, - event_value: u32, - }, - // Max discriminant: 0x8 -} - -static_assert!(core::mem::size_of::() == 4 + EVENT_SZ); - -pub(crate) const EVENT_MAX: u32 = 0x8; - -#[derive(Copy, Clone)] -#[repr(C)] -pub(crate) union RawEventMsg { - pub(crate) raw: (u32, Array), - pub(crate) msg: EventMsg, -} - -default_zeroed!(RawEventMsg); - -#[derive(Debug, Copy, Clone, Default)] -#[repr(C)] -pub(crate) struct RawFwLogMsg { - pub(crate) msg_type: u32, - __pad0: u32, - pub(crate) msg_index: U64, - __pad1: Pad<0x28>, -} - -#[derive(Debug, Copy, Clone, Default)] -#[repr(C)] -pub(crate) struct RawFwLogPayloadMsg { - pub(crate) msg_type: u32, - pub(crate) seq_no: u32, - pub(crate) timestamp: U64, - pub(crate) msg: Array<0xc8, u8>, -} - -#[derive(Debug, Copy, Clone, Default)] -#[repr(C)] -pub(crate) struct RawKTraceMsg { - pub(crate) msg_type: u32, - pub(crate) timestamp: U64, - pub(crate) args: Array<4, U64>, - pub(crate) code: u8, - pub(crate) channel: u8, - __pad: Pad<1>, - pub(crate) thread: u8, - pub(crate) unk_flag: U64, -} - -#[versions(AGX_NEO)] -pub(crate) const STATS_SZ: usize = { - #[ver(V < V13_0B4)] - { - 0x2c - } - #[ver(V >= V13_0B4)] - { - 0x3c - } -}; - -#[versions(AGX_NEO)] -#[derive(Debug, Copy, Clone)] -#[repr(C, u32)] -#[allow(dead_code)] -pub(crate) enum StatsMsg { - Power { - // 0x00 - __pad: Pad<0x18>, - power: U64, - }, - Unk1(Array<{ STATS_SZ::ver }, u8>), - PowerOn { - // 0x02 - off_time: U64, - }, - PowerOff { - // 0x03 - on_time: U64, - }, - Utilization { - // 0x04 - timestamp: U64, - util1: u32, - util2: u32, - util3: u32, - util4: u32, - }, - Unk5(Array<{ STATS_SZ::ver }, u8>), - Unk6(Array<{ STATS_SZ::ver }, u8>), - Unk7(Array<{ STATS_SZ::ver }, u8>), - Unk8(Array<{ STATS_SZ::ver }, u8>), - AvgPower { - // 0x09 - active_cs: U64, - unk2: u32, - unk3: u32, - unk4: u32, - avg_power: u32, - }, - Temperature { - // 0x0a - __pad: Pad<0x8>, - raw_value: u32, - scale: u32, - tmin: u32, - tmax: u32, - }, - PowerState { - // 0x0b - timestamp: U64, - last_busy_ts: U64, - active: u32, - poweroff: u32, - unk1: u32, - pstate: u32, - unk2: u32, - unk3: u32, - }, - FwBusy { - // 0x0c - timestamp: U64, - busy: u32, - }, - PState { - // 0x0d - __pad: Pad<0x8>, - ps_min: u32, - unk1: u32, - ps_max: u32, - unk2: u32, - }, - TempSensor { - // 0x0e - __pad: Pad<0x4>, - sensor_id: u32, - raw_value: u32, - scale: u32, - tmin: u32, - tmax: u32, - }, // Max discriminant: 0xe -} - -#[versions(AGX_NEO)] -static_assert!(core::mem::size_of::() == 4 + STATS_SZ::ver); - -#[versions(AGX_NEO)] -pub(crate) const STATS_MAX: u32 = 0xe; - -#[versions(AGX_NEO)] -#[derive(Copy, Clone)] -#[repr(C)] -pub(crate) union RawStatsMsg { - pub(crate) raw: (u32, Array<{ STATS_SZ::ver }, u8>), - pub(crate) msg: StatsMsg::ver, -} - -#[versions(AGX_NEO)] -default_zeroed!(RawStatsMsg::ver); diff --git a/drivers/gpu/drm/asahi-neo/fw/compute.rs b/drivers/gpu/drm/asahi-neo/fw/compute.rs deleted file mode 100644 index 916356298418a0..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/compute.rs +++ /dev/null @@ -1,120 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU compute job firmware structures - -use super::types::*; -use super::{ - event, - job, - workqueue, // -}; -use crate::{ - microseq, - mmu, // -}; -use kernel::sync::Arc; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters1<'a> { - pub(crate) preempt_buf1: GpuPointer<'a, &'a [u8]>, - pub(crate) cdm_ctrl_stream_base: U64, - pub(crate) preempt_buf2: GpuPointer<'a, &'a [u8]>, - pub(crate) preempt_buf3: GpuPointer<'a, &'a [u8]>, - pub(crate) preempt_buf4: GpuPointer<'a, &'a [u8]>, - pub(crate) preempt_buf5: GpuPointer<'a, &'a [u8]>, - pub(crate) usc_exec_base_cp: U64, - pub(crate) unk_38: U64, - pub(crate) helper_program: u32, - pub(crate) unk_44: u32, - pub(crate) helper_arg: U64, - pub(crate) helper_cfg: u32, - pub(crate) unk_54: u32, - pub(crate) unk_58: u32, - pub(crate) unk_5c: u32, - pub(crate) iogpu_unk_40: u32, - pub(crate) __pad: Pad<0xfc>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters2<'a> { - #[ver(V >= V13_0B4)] - pub(crate) unk_0_0: u32, - pub(crate) unk_0: Array<0x24, u8>, - pub(crate) preempt_buf1: GpuPointer<'a, &'a [u8]>, - pub(crate) cdm_ctrl_stream_end: U64, - pub(crate) unk_34: Array<0x20, u8>, - pub(crate) unk_g14x: u32, - pub(crate) unk_58: u32, - #[ver(V < V13_0B4)] - pub(crate) unk_5c: u32, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RunCompute<'a> { - pub(crate) tag: workqueue::CommandType, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - pub(crate) unk_4: u32, - pub(crate) vm_slot: u32, - pub(crate) notifier: GpuPointer<'a, event::Notifier::ver>, - pub(crate) unk_pointee: u32, - #[ver(G < G14X)] - pub(crate) __pad0: Array<0x50, u8>, - #[ver(G < G14X)] - pub(crate) job_params1: JobParameters1<'a>, - #[ver(G >= G14X)] - pub(crate) registers: job::raw::RegisterArray, - pub(crate) __pad1: Array<0x20, u8>, - pub(crate) microsequence: GpuPointer<'a, &'a [u8]>, - pub(crate) microsequence_size: u32, - pub(crate) job_params2: JobParameters2::ver<'a>, - pub(crate) encoder_params: job::raw::EncoderParams, - pub(crate) meta: job::raw::JobMeta, - pub(crate) command_time: U64, - pub(crate) timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) user_timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) client_sequence: u8, - pub(crate) pad_2d1: Array<3, u8>, - pub(crate) unk_2d4: u32, - pub(crate) unk_2d8: u8, - #[ver(V >= V13_0B4)] - pub(crate) context_store_req: U64, - #[ver(V >= V13_0B4)] - pub(crate) context_store_compl: U64, - #[ver(V >= V13_0B4)] - pub(crate) unk_2e9: Array<0x14, u8>, - #[ver(V >= V13_0B4)] - pub(crate) unk_flag: U32, - #[ver(V >= V13_0B4)] - pub(crate) unk_pad: Array<0x10, u8>, - } -} - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct RunCompute { - pub(crate) notifier: Arc>, - pub(crate) preempt_buf: GpuArray, - pub(crate) micro_seq: microseq::MicroSequence, - pub(crate) vm_bind: mmu::VmBind, - pub(crate) timestamps: Arc>, - pub(crate) user_timestamps: job::UserTimestamps, -} - -#[versions(AGX_NEO)] -impl GpuStruct for RunCompute::ver { - type Raw<'a> = raw::RunCompute::ver<'a>; -} - -#[versions(AGX_NEO)] -impl workqueue::Command for RunCompute::ver {} diff --git a/drivers/gpu/drm/asahi-neo/fw/event.rs b/drivers/gpu/drm/asahi-neo/fw/event.rs deleted file mode 100644 index caeb7b5317f379..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/event.rs +++ /dev/null @@ -1,104 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU events control structures & stamps - -use super::types::*; -use crate::{ - default_zeroed, - trivial_gpustruct, // -}; -use core::sync::atomic::Ordering; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug, Clone, Copy, Default)] - #[repr(C)] - pub(crate) struct LinkedListHead { - pub(crate) prev: Option>, - pub(crate) next: Option>, - } - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct NotifierList { - pub(crate) list_head: LinkedListHead, - pub(crate) unkptr_10: U64, - } - default_zeroed!(NotifierList); - - #[versions(AGX_NEO)] - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct NotifierState { - unk_14: u32, - unk_18: U64, - unk_20: u32, - vm_slot: u32, - has_vtx: u32, - pstamp_vtx: Array<4, U64>, - has_frag: u32, - pstamp_frag: Array<4, U64>, - has_comp: u32, - pstamp_comp: Array<4, U64>, - #[ver(G >= G14 && V < V13_0B4)] - unk_98_g14_0: Array<0x14, u8>, - in_list: u32, - list_head: LinkedListHead, - #[ver(G >= G14 && V < V13_0B4)] - unk_a8_g14_0: Pad<4>, - #[ver(V >= V13_0B4)] - pub(crate) unk_buf: Array<0x8, u8>, // Init to all-ff - } - - #[versions(AGX_NEO)] - impl Default for NotifierState::ver { - fn default() -> Self { - #[allow(unused_mut)] - // SAFETY: All bit patterns are valid for this type. - let mut s: Self = unsafe { core::mem::zeroed() }; - #[ver(V >= V13_0B4)] - s.unk_buf = Array::new([0xff; 0x8]); - s - } - } - - #[derive(Debug)] - #[repr(transparent)] - pub(crate) struct Threshold(AtomicU64); - default_zeroed!(Threshold); - - impl Threshold { - pub(crate) fn increase(&self, amount: u32) { - // We could use fetch_add, but the non-LSE atomic - // sequence Rust produces confuses the hypervisor. - let v = self.0.load(Ordering::Relaxed); - self.0.store(v + (amount as u64), Ordering::Relaxed); - } - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Notifier<'a> { - pub(crate) threshold: GpuPointer<'a, super::Threshold>, - pub(crate) generation: AtomicU32, - pub(crate) cur_count: AtomicU32, - pub(crate) unk_10: AtomicU32, - pub(crate) state: NotifierState::ver, - } -} - -trivial_gpustruct!(Threshold); -trivial_gpustruct!(NotifierList); - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct Notifier { - pub(crate) threshold: GpuObject, -} - -#[versions(AGX_NEO)] -impl GpuStruct for Notifier::ver { - type Raw<'a> = raw::Notifier::ver<'a>; -} diff --git a/drivers/gpu/drm/asahi-neo/fw/fragment.rs b/drivers/gpu/drm/asahi-neo/fw/fragment.rs deleted file mode 100644 index 8c38be4679ed24..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/fragment.rs +++ /dev/null @@ -1,296 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU fragment job firmware structures - -use super::types::*; -use super::{ - event, - job, - workqueue, // -}; -use crate::{ - buffer, - fw, - microseq, - mmu, // -}; -use kernel::sync::Arc; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct BackgroundProgram { - pub(crate) rsrc_spec: U64, - pub(crate) address: U64, - } - - #[derive(Debug, Clone, Copy, Default)] - #[repr(C)] - pub(crate) struct EotProgram { - pub(crate) unk_0: U64, - pub(crate) unk_8: u32, - pub(crate) rsrc_spec: u32, - pub(crate) unk_10: u32, - pub(crate) address: u32, - pub(crate) unk_18: u32, - pub(crate) unk_1c_padding: u32, - } - - impl EotProgram { - pub(crate) fn new(rsrc_spec: u32, address: u32) -> EotProgram { - EotProgram { - rsrc_spec, - address, - ..Default::default() - } - } - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct ArrayAddr { - pub(crate) ptr: U64, - pub(crate) unk_padding: U64, - } - - #[versions(AGX_NEO)] - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct AuxFBInfo { - pub(crate) isp_ctl: u32, - pub(crate) unk2: u32, - pub(crate) width: u32, - pub(crate) height: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk3: U64, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters1<'a> { - pub(crate) utile_config: u32, - pub(crate) unk_4: u32, - pub(crate) bg: BackgroundProgram, - pub(crate) ppp_multisamplectl: U64, - pub(crate) isp_scissor_base: U64, - pub(crate) isp_dbias_base: U64, - pub(crate) aux_fb_info: AuxFBInfo::ver, - pub(crate) isp_zls_pixels: U64, - pub(crate) isp_oclqry_base: U64, - pub(crate) zls_ctrl: U64, - - #[ver(G >= G14)] - pub(crate) unk_58_g14_0: U64, - #[ver(G >= G14)] - pub(crate) unk_58_g14_8: U64, - - pub(crate) z_load: U64, - pub(crate) z_store: U64, - pub(crate) s_load: U64, - pub(crate) s_store: U64, - - #[ver(G >= G14)] - pub(crate) unk_68_g14_0: Array<0x20, u8>, - - pub(crate) z_load_stride: U64, - pub(crate) z_store_stride: U64, - pub(crate) s_load_stride: U64, - pub(crate) s_store_stride: U64, - pub(crate) z_load_comp: U64, - pub(crate) z_load_comp_stride: U64, - pub(crate) z_store_comp: U64, - pub(crate) z_store_comp_stride: U64, - pub(crate) s_load_comp: U64, - pub(crate) s_load_comp_stride: U64, - pub(crate) s_store_comp: U64, - pub(crate) s_store_comp_stride: U64, - pub(crate) tvb_tilemap: GpuPointer<'a, &'a [u8]>, - pub(crate) tvb_layermeta: GpuPointer<'a, &'a [u8]>, - pub(crate) mtile_stride_dwords: U64, - pub(crate) tvb_heapmeta: GpuPointer<'a, &'a [u8]>, - pub(crate) tile_config: U64, - pub(crate) aux_fb: GpuPointer<'a, &'a [u8]>, - pub(crate) unk_108: Array<0x6, U64>, - pub(crate) usc_exec_base_isp: U64, - pub(crate) unk_140: U64, - pub(crate) helper_program: u32, - pub(crate) unk_14c: u32, - pub(crate) helper_arg: U64, - pub(crate) unk_158: U64, - pub(crate) unk_160: U64, - - #[ver(G < G14)] - pub(crate) __pad: Pad<0x1d8>, - #[ver(G >= G14)] - pub(crate) __pad: Pad<0x1a8>, - #[ver(V < V13_0B4)] - pub(crate) __pad1: Pad<0x8>, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters2 { - pub(crate) eot_rsrc_spec: u32, - pub(crate) eot_usc: u32, - pub(crate) unk_8: u32, - pub(crate) unk_c: u32, - pub(crate) isp_merge_upper_x: F32, - pub(crate) isp_merge_upper_y: F32, - pub(crate) unk_18: U64, - pub(crate) utiles_per_mtile_y: u16, - pub(crate) utiles_per_mtile_x: u16, - pub(crate) unk_24: u32, - pub(crate) tile_counts: u32, - pub(crate) tib_blocks: u32, - pub(crate) isp_bgobjdepth: u32, - pub(crate) isp_bgobjvals: u32, - pub(crate) unk_38: u32, - pub(crate) unk_3c: u32, - pub(crate) helper_cfg: u32, - pub(crate) __pad: Pad<0xac>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters3 { - pub(crate) isp_dbias_base: ArrayAddr, - pub(crate) isp_scissor_base: ArrayAddr, - pub(crate) isp_oclqry_base: U64, - pub(crate) unk_118: U64, - pub(crate) unk_120: Array<0x25, U64>, - pub(crate) unk_partial_bg: BackgroundProgram, - pub(crate) unk_258: U64, - pub(crate) unk_260: U64, - pub(crate) unk_268: U64, - pub(crate) unk_270: U64, - pub(crate) partial_bg: BackgroundProgram, - pub(crate) zls_ctrl: U64, - pub(crate) unk_290: U64, - pub(crate) z_load: U64, - pub(crate) z_partial_stride: U64, - pub(crate) z_partial_comp_stride: U64, - pub(crate) z_store: U64, - pub(crate) z_partial: U64, - pub(crate) z_partial_comp: U64, - pub(crate) s_load: U64, - pub(crate) s_partial_stride: U64, - pub(crate) s_partial_comp_stride: U64, - pub(crate) s_store: U64, - pub(crate) s_partial: U64, - pub(crate) s_partial_comp: U64, - pub(crate) unk_2f8: Array<2, U64>, - pub(crate) tib_blocks: u32, - pub(crate) unk_30c: u32, - pub(crate) aux_fb_info: AuxFBInfo::ver, - pub(crate) tile_config: U64, - pub(crate) unk_328_padding: Array<0x8, u8>, - pub(crate) unk_partial_eot: EotProgram, - pub(crate) partial_eot: EotProgram, - pub(crate) isp_bgobjdepth: u32, - pub(crate) isp_bgobjvals: u32, - pub(crate) sample_size: u32, - pub(crate) unk_37c: u32, - pub(crate) unk_380: U64, - pub(crate) unk_388: U64, - - #[ver(V >= V13_0B4)] - pub(crate) unk_390_0: U64, - - pub(crate) isp_zls_pixels: U64, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RunFragment<'a> { - pub(crate) tag: workqueue::CommandType, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - pub(crate) vm_slot: u32, - pub(crate) unk_8: u32, - pub(crate) microsequence: GpuPointer<'a, &'a [u8]>, - pub(crate) microsequence_size: u32, - pub(crate) notifier: GpuPointer<'a, event::Notifier::ver>, - pub(crate) buffer: GpuPointer<'a, fw::buffer::Info::ver>, - pub(crate) scene: GpuPointer<'a, fw::buffer::Scene::ver>, - pub(crate) unk_buffer_buf: GpuWeakPointer<[u8]>, - pub(crate) tvb_tilemap: GpuPointer<'a, &'a [u8]>, - pub(crate) ppp_multisamplectl: U64, - pub(crate) samples: u32, - pub(crate) tiles_per_mtile_y: u16, - pub(crate) tiles_per_mtile_x: u16, - pub(crate) unk_50: U64, - pub(crate) unk_58: U64, - pub(crate) isp_merge_upper_x: F32, - pub(crate) isp_merge_upper_y: F32, - pub(crate) unk_68: U64, - pub(crate) tile_count: U64, - - #[ver(G < G14X)] - pub(crate) job_params1: JobParameters1::ver<'a>, - #[ver(G < G14X)] - pub(crate) job_params2: JobParameters2, - #[ver(G >= G14X)] - pub(crate) registers: job::raw::RegisterArray, - - pub(crate) job_params3: JobParameters3::ver, - pub(crate) unk_758_flag: u32, - pub(crate) unk_75c_flag: u32, - pub(crate) unk_buf: Array<0x110, u8>, - pub(crate) busy_flag: u32, - pub(crate) tvb_overflow_count: u32, - pub(crate) unk_878: u32, - pub(crate) encoder_params: job::raw::EncoderParams, - pub(crate) process_empty_tiles: u32, - pub(crate) no_clear_pipeline_textures: u32, - pub(crate) msaa_zs: u32, - pub(crate) unk_pointee: u32, - #[ver(V >= V13_3)] - pub(crate) unk_v13_3: u32, - pub(crate) meta: job::raw::JobMeta, - pub(crate) unk_after_meta: u32, - pub(crate) unk_buf_0: U64, - pub(crate) unk_buf_8: U64, - pub(crate) unk_buf_10: U64, - pub(crate) command_time: U64, - pub(crate) timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) user_timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) client_sequence: u8, - pub(crate) pad_925: Array<3, u8>, - pub(crate) unk_928: u32, - pub(crate) unk_92c: u8, - - #[ver(V >= V13_0B4)] - pub(crate) unk_ts: U64, - - #[ver(V >= V13_0B4)] - pub(crate) unk_92d_8: Array<0x1b, u8>, - } -} - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct RunFragment { - pub(crate) notifier: Arc>, - pub(crate) scene: Arc, - pub(crate) micro_seq: microseq::MicroSequence, - pub(crate) vm_bind: mmu::VmBind, - pub(crate) aux_fb: GpuArray, - pub(crate) timestamps: Arc>, - pub(crate) user_timestamps: job::UserTimestamps, -} - -#[versions(AGX_NEO)] -impl GpuStruct for RunFragment::ver { - type Raw<'a> = raw::RunFragment::ver<'a>; -} - -#[versions(AGX_NEO)] -impl workqueue::Command for RunFragment::ver {} diff --git a/drivers/gpu/drm/asahi-neo/fw/initdata.rs b/drivers/gpu/drm/asahi-neo/fw/initdata.rs deleted file mode 100644 index 09943b57d68152..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/initdata.rs +++ /dev/null @@ -1,1352 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU initialization / global structures - -use super::channels; -use super::types::*; -use crate::{ - default_zeroed, - gem, - mmu, - no_debug, - trivial_gpustruct, // -}; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct ChannelRing { - pub(crate) state: Option>, - pub(crate) ring: Option>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct PipeChannels { - pub(crate) vtx: ChannelRing, - pub(crate) frag: ChannelRing, - pub(crate) comp: ChannelRing, - } - #[versions(AGX_NEO)] - default_zeroed!(PipeChannels::ver); - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct FwStatusFlags { - pub(crate) halt_count: AtomicU64, - __pad0: Pad<0x8>, - pub(crate) halted: AtomicU32, - __pad1: Pad<0xc>, - pub(crate) resume: AtomicU32, - __pad2: Pad<0xc>, - pub(crate) unk_40: u32, - __pad3: Pad<0xc>, - pub(crate) unk_ctr: u32, - __pad4: Pad<0xc>, - pub(crate) unk_60: u32, - __pad5: Pad<0xc>, - pub(crate) unk_70: u32, - __pad6: Pad<0xc>, - } - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct FwStatus { - pub(crate) fwctl_channel: ChannelRing, - pub(crate) flags: FwStatusFlags, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataShared1 { - pub(crate) table: Array<16, i32>, - pub(crate) unk_44: Array<0x60, u8>, - pub(crate) unk_a4: u32, - pub(crate) unk_a8: u32, - } - default_zeroed!(HwDataShared1); - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct HwDataShared2Curve { - pub(crate) unk_0: u32, - pub(crate) unk_4: u32, - pub(crate) t1: Array<16, u16>, - pub(crate) t2: Array<16, i16>, - pub(crate) t3: Array<8, Array<16, i32>>, - } - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct HwDataShared2G14 { - pub(crate) unk_0: Array<5, u32>, - pub(crate) unk_14: u32, - pub(crate) unk_18: Array<8, u32>, - pub(crate) curve1: HwDataShared2Curve, - pub(crate) curve2: HwDataShared2Curve, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataShared2 { - pub(crate) table: Array<10, i32>, - pub(crate) unk_28: Array<0x10, u8>, - pub(crate) g14: HwDataShared2G14, - pub(crate) unk_500: u32, - pub(crate) unk_504: u32, - pub(crate) unk_508: u32, - pub(crate) unk_50c: u32, - } - default_zeroed!(HwDataShared2); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataShared3 { - pub(crate) unk_0: u32, - pub(crate) unk_4: u32, - pub(crate) unk_8: u32, - pub(crate) table: Array<16, u32>, - pub(crate) unk_4c: u32, - } - default_zeroed!(HwDataShared3); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataA130Extra { - pub(crate) unk_0: Array<0x38, u8>, - pub(crate) unk_38: u32, - pub(crate) unk_3c: u32, - pub(crate) gpu_se_inactive_threshold: u32, - pub(crate) unk_44: u32, - pub(crate) gpu_se_engagement_criteria: i32, - pub(crate) gpu_se_reset_criteria: u32, - pub(crate) unk_50: u32, - pub(crate) unk_54: u32, - pub(crate) unk_58: u32, - pub(crate) unk_5c: u32, - pub(crate) gpu_se_filter_a_neg: F32, - pub(crate) gpu_se_filter_1_a_neg: F32, - pub(crate) gpu_se_filter_a: F32, - pub(crate) gpu_se_filter_1_a: F32, - pub(crate) gpu_se_ki_dt: F32, - pub(crate) gpu_se_ki_1_dt: F32, - pub(crate) unk_78: F32, - pub(crate) unk_7c: F32, - pub(crate) gpu_se_kp: F32, - pub(crate) gpu_se_kp_1: F32, - pub(crate) unk_88: u32, - pub(crate) unk_8c: u32, - pub(crate) max_pstate_scaled_1: u32, - pub(crate) unk_94: u32, - pub(crate) unk_98: u32, - pub(crate) unk_9c: F32, - pub(crate) unk_a0: u32, - pub(crate) unk_a4: u32, - pub(crate) gpu_se_filter_time_constant_ms: u32, - pub(crate) gpu_se_filter_time_constant_1_ms: u32, - pub(crate) gpu_se_filter_time_constant_clks: U64, - pub(crate) gpu_se_filter_time_constant_1_clks: U64, - pub(crate) unk_c0: u32, - pub(crate) unk_c4: F32, - pub(crate) unk_c8: Array<0x4c, u8>, - pub(crate) unk_114: F32, - pub(crate) unk_118: u32, - pub(crate) unk_11c: u32, - pub(crate) unk_120: u32, - pub(crate) unk_124: u32, - pub(crate) max_pstate_scaled_2: u32, - pub(crate) unk_12c: Array<0x8c, u8>, - } - default_zeroed!(HwDataA130Extra); - - #[repr(C)] - pub(crate) struct T81xxData { - pub(crate) unk_d8c: u32, - pub(crate) unk_d90: u32, - pub(crate) unk_d94: u32, - pub(crate) unk_d98: u32, - pub(crate) unk_d9c: F32, - pub(crate) unk_da0: u32, - pub(crate) unk_da4: F32, - pub(crate) unk_da8: u32, - pub(crate) unk_dac: F32, - pub(crate) unk_db0: u32, - pub(crate) unk_db4: u32, - pub(crate) unk_db8: F32, - pub(crate) unk_dbc: F32, - pub(crate) unk_dc0: u32, - pub(crate) unk_dc4: u32, - pub(crate) unk_dc8: u32, - pub(crate) max_pstate_scaled: u32, - } - default_zeroed!(T81xxData); - - #[versions(AGX_NEO)] - #[derive(Default, Copy, Clone)] - #[repr(C)] - pub(crate) struct PowerZone { - pub(crate) val: F32, - pub(crate) target: u32, - pub(crate) target_off: u32, - pub(crate) filter_tc_x4: u32, - pub(crate) filter_tc_xperiod: u32, - #[ver(V >= V13_0B4)] - pub(crate) unk_10: u32, - #[ver(V >= V13_0B4)] - pub(crate) unk_14: u32, - pub(crate) filter_a_neg: F32, - pub(crate) filter_a: F32, - pub(crate) pad: u32, - } - - #[versions(AGX_NEO)] - const MAX_CORES_PER_CLUSTER: usize = { - #[ver(G >= G14X)] - { - 16 - } - #[ver(G < G14X)] - { - 8 - } - }; - - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct AuxLeakCoef { - pub(crate) afr_1: Array<2, F32>, - pub(crate) cs_1: Array<2, F32>, - pub(crate) afr_2: Array<2, F32>, - pub(crate) cs_2: Array<2, F32>, - } - - #[versions(AGX_NEO)] - #[repr(C)] - pub(crate) struct HwDataA { - pub(crate) unk_0: u32, - pub(crate) clocks_per_period: u32, - - #[ver(V >= V13_0B4)] - pub(crate) clocks_per_period_2: u32, - - pub(crate) unk_8: u32, - pub(crate) pwr_status: AtomicU32, - pub(crate) unk_10: F32, - pub(crate) unk_14: u32, - pub(crate) unk_18: u32, - pub(crate) unk_1c: u32, - pub(crate) unk_20: u32, - pub(crate) unk_24: u32, - pub(crate) actual_pstate: u32, - pub(crate) tgt_pstate: u32, - pub(crate) unk_30: u32, - pub(crate) cur_pstate: u32, - pub(crate) unk_38: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_3c_0: u32, - - pub(crate) base_pstate_scaled: u32, - pub(crate) unk_40: u32, - pub(crate) max_pstate_scaled: u32, - pub(crate) unk_48: u32, - pub(crate) min_pstate_scaled: u32, - pub(crate) freq_mhz: F32, - pub(crate) unk_54: Array<0x20, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_74_0: u32, - - pub(crate) sram_k: Array<0x10, F32>, - pub(crate) unk_b4: Array<0x100, u8>, - pub(crate) unk_1b4: u32, - pub(crate) temp_c: u32, - pub(crate) avg_power_mw: u32, - pub(crate) update_ts: U64, - pub(crate) unk_1c8: u32, - pub(crate) unk_1cc: Array<0x478, u8>, - pub(crate) pad_644: Pad<0x8>, - pub(crate) unk_64c: u32, - pub(crate) unk_650: u32, - pub(crate) pad_654: u32, - pub(crate) pwr_filter_a_neg: F32, - pub(crate) pad_65c: u32, - pub(crate) pwr_filter_a: F32, - pub(crate) pad_664: u32, - pub(crate) pwr_integral_gain: F32, - pub(crate) pad_66c: u32, - pub(crate) pwr_integral_min_clamp: F32, - pub(crate) max_power_1: F32, - pub(crate) pwr_proportional_gain: F32, - pub(crate) pad_67c: u32, - pub(crate) pwr_pstate_related_k: F32, - pub(crate) pwr_pstate_max_dc_offset: i32, - pub(crate) unk_688: u32, - pub(crate) max_pstate_scaled_2: u32, - pub(crate) pad_690: u32, - pub(crate) unk_694: u32, - pub(crate) max_power_2: u32, - pub(crate) pad_69c: Pad<0x18>, - pub(crate) unk_6b4: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_6b8_0: Array<0x10, u8>, - - pub(crate) max_pstate_scaled_3: u32, - pub(crate) unk_6bc: u32, - pub(crate) pad_6c0: Pad<0x14>, - pub(crate) ppm_filter_tc_periods_x4: u32, - pub(crate) unk_6d8: u32, - pub(crate) pad_6dc: u32, - pub(crate) ppm_filter_a_neg: F32, - pub(crate) pad_6e4: u32, - pub(crate) ppm_filter_a: F32, - pub(crate) pad_6ec: u32, - pub(crate) ppm_ki_dt: F32, - pub(crate) pad_6f4: u32, - pub(crate) pwr_integral_min_clamp_2: u32, - pub(crate) unk_6fc: F32, - pub(crate) ppm_kp: F32, - pub(crate) pad_704: u32, - pub(crate) unk_708: u32, - pub(crate) pwr_min_duty_cycle: u32, - pub(crate) max_pstate_scaled_4: u32, - pub(crate) unk_714: u32, - pub(crate) pad_718: u32, - pub(crate) unk_71c: F32, - pub(crate) max_power_3: u32, - pub(crate) cur_power_mw_2: u32, - pub(crate) ppm_filter_tc_ms: u32, - pub(crate) unk_72c: u32, - - #[ver(V >= V13_0B4)] - pub(crate) ppm_filter_tc_clks: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_730_4: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_730_8: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_730_c: u32, - - pub(crate) unk_730: F32, - pub(crate) unk_734: u32, - pub(crate) unk_738: u32, - pub(crate) unk_73c: u32, - pub(crate) unk_740: u32, - pub(crate) unk_744: u32, - pub(crate) unk_748: Array<0x4, F32>, - pub(crate) unk_758: u32, - pub(crate) perf_tgt_utilization: u32, - pub(crate) pad_760: u32, - pub(crate) perf_boost_min_util: u32, - pub(crate) perf_boost_ce_step: u32, - pub(crate) perf_reset_iters: u32, - pub(crate) pad_770: u32, - pub(crate) unk_774: u32, - pub(crate) unk_778: u32, - pub(crate) perf_filter_drop_threshold: u32, - pub(crate) perf_filter_a_neg: F32, - pub(crate) perf_filter_a2_neg: F32, - pub(crate) perf_filter_a: F32, - pub(crate) perf_filter_a2: F32, - pub(crate) perf_ki: F32, - pub(crate) perf_ki2: F32, - pub(crate) perf_integral_min_clamp: F32, - pub(crate) unk_79c: F32, - pub(crate) perf_kp: F32, - pub(crate) perf_kp2: F32, - pub(crate) boost_state_unk_k: F32, - pub(crate) base_pstate_scaled_2: u32, - pub(crate) max_pstate_scaled_5: u32, - pub(crate) base_pstate_scaled_3: u32, - pub(crate) pad_7b8: u32, - pub(crate) perf_cur_utilization: F32, - pub(crate) perf_tgt_utilization_2: u32, - pub(crate) pad_7c4: Pad<0x18>, - pub(crate) unk_7dc: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_7e0_0: Array<0x10, u8>, - - pub(crate) base_pstate_scaled_4: u32, - pub(crate) pad_7e4: u32, - pub(crate) unk_7e8: Array<0x14, u8>, - pub(crate) unk_7fc: F32, - pub(crate) pwr_min_duty_cycle_2: F32, - pub(crate) max_pstate_scaled_6: F32, - pub(crate) max_freq_mhz: u32, - pub(crate) pad_80c: u32, - pub(crate) unk_810: u32, - pub(crate) pad_814: u32, - pub(crate) pwr_min_duty_cycle_3: u32, - pub(crate) unk_81c: u32, - pub(crate) pad_820: u32, - pub(crate) min_pstate_scaled_4: F32, - pub(crate) max_pstate_scaled_7: u32, - pub(crate) unk_82c: u32, - pub(crate) unk_alpha_neg: F32, - pub(crate) unk_alpha: F32, - pub(crate) unk_838: u32, - pub(crate) unk_83c: u32, - pub(crate) pad_840: Pad<0x2c>, - pub(crate) unk_86c: u32, - pub(crate) fast_die0_sensor_mask: U64, - #[ver(G >= G14X)] - pub(crate) fast_die1_sensor_mask: U64, - pub(crate) fast_die0_release_temp_cc: u32, - pub(crate) unk_87c: i32, - pub(crate) unk_880: u32, - pub(crate) unk_884: u32, - pub(crate) pad_888: u32, - pub(crate) unk_88c: u32, - pub(crate) pad_890: u32, - pub(crate) unk_894: F32, - pub(crate) pad_898: u32, - pub(crate) fast_die0_ki_dt: F32, - pub(crate) pad_8a0: u32, - pub(crate) unk_8a4: u32, - pub(crate) unk_8a8: F32, - pub(crate) fast_die0_kp: F32, - pub(crate) pad_8b0: u32, - pub(crate) unk_8b4: u32, - pub(crate) pwr_min_duty_cycle_4: u32, - pub(crate) max_pstate_scaled_8: u32, - pub(crate) max_pstate_scaled_9: u32, - pub(crate) fast_die0_prop_tgt_delta: u32, - pub(crate) unk_8c8: u32, - pub(crate) unk_8cc: u32, - pub(crate) pad_8d0: Pad<0x14>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_8e4_0: Array<0x10, u8>, - - pub(crate) unk_8e4: u32, - pub(crate) unk_8e8: u32, - pub(crate) max_pstate_scaled_10: u32, - pub(crate) unk_8f0: u32, - pub(crate) unk_8f4: u32, - pub(crate) pad_8f8: u32, - pub(crate) pad_8fc: u32, - pub(crate) unk_900: Array<0x24, u8>, - - pub(crate) unk_coef_a1: Array<8, Array>, - pub(crate) unk_coef_a2: Array<8, Array>, - - pub(crate) pad_b24: Pad<0x70>, - pub(crate) max_pstate_scaled_11: u32, - pub(crate) freq_with_off: u32, - pub(crate) unk_b9c: u32, - pub(crate) unk_ba0: U64, - pub(crate) unk_ba8: U64, - pub(crate) unk_bb0: u32, - pub(crate) unk_bb4: u32, - - #[ver(V >= V13_3)] - pub(crate) pad_bb8_0: Pad<0x200>, - #[ver(V >= V13_5)] - pub(crate) pad_bb8_200: Pad<0x8>, - - pub(crate) pad_bb8: Pad<0x74>, - pub(crate) unk_c2c: u32, - pub(crate) power_zone_count: u32, - pub(crate) max_power_4: u32, - pub(crate) max_power_5: u32, - pub(crate) max_power_6: u32, - pub(crate) unk_c40: u32, - pub(crate) unk_c44: F32, - pub(crate) avg_power_target_filter_a_neg: F32, - pub(crate) avg_power_target_filter_a: F32, - pub(crate) avg_power_target_filter_tc_x4: u32, - pub(crate) avg_power_target_filter_tc_xperiod: u32, - - #[ver(V >= V13_0B4)] - pub(crate) avg_power_target_filter_tc_clks: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_c58_4: u32, - - pub(crate) power_zones: Array<5, PowerZone::ver>, - pub(crate) avg_power_filter_tc_periods_x4: u32, - pub(crate) unk_cfc: u32, - pub(crate) unk_d00: u32, - pub(crate) avg_power_filter_a_neg: F32, - pub(crate) unk_d08: u32, - pub(crate) avg_power_filter_a: F32, - pub(crate) unk_d10: u32, - pub(crate) avg_power_ki_dt: F32, - pub(crate) unk_d18: u32, - pub(crate) unk_d1c: u32, - pub(crate) unk_d20: F32, - pub(crate) avg_power_kp: F32, - pub(crate) unk_d28: u32, - pub(crate) unk_d2c: u32, - pub(crate) avg_power_min_duty_cycle: u32, - pub(crate) max_pstate_scaled_12: u32, - pub(crate) max_pstate_scaled_13: u32, - pub(crate) unk_d3c: u32, - pub(crate) max_power_7: F32, - pub(crate) max_power_8: u32, - pub(crate) unk_d48: u32, - pub(crate) avg_power_filter_tc_ms: u32, - pub(crate) unk_d50: u32, - - #[ver(V >= V13_0B4)] - pub(crate) avg_power_filter_tc_clks: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_d54_4: Array<0xc, u8>, - - pub(crate) unk_d54: Array<0x10, u8>, - pub(crate) max_pstate_scaled_14: u32, - pub(crate) unk_d68: Array<0x24, u8>, - - pub(crate) t81xx_data: T81xxData, - - pub(crate) unk_dd0: Array<0x40, u8>, - - #[ver(V >= V13_2)] - pub(crate) unk_e10_pad: Array<0x10, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_e10_0: HwDataA130Extra, - - pub(crate) unk_e10: Array<0xc, u8>, - - pub(crate) fast_die0_sensor_mask_2: U64, - #[ver(G >= G14X)] - pub(crate) fast_die1_sensor_mask_2: U64, - - pub(crate) unk_e24: u32, - pub(crate) unk_e28: u32, - pub(crate) unk_e2c: Pad<0x1c>, - pub(crate) unk_coef_b1: Array<8, Array>, - pub(crate) unk_coef_b2: Array<8, Array>, - - #[ver(G >= G14X)] - pub(crate) pad_1048_0: Pad<0x600>, - - pub(crate) pad_1048: Pad<0x5e4>, - - pub(crate) fast_die0_sensor_mask_alt: U64, - #[ver(G >= G14X)] - pub(crate) fast_die1_sensor_mask_alt: U64, - #[ver(V < V13_0B4)] - pub(crate) fast_die0_sensor_present: U64, - - pub(crate) unk_163c: u32, - - pub(crate) unk_1640: Array<0x2000, u8>, - - #[ver(G >= G14X)] - pub(crate) unk_3640_0: Array<0x2000, u8>, - - pub(crate) unk_3640: u32, - pub(crate) unk_3644: u32, - pub(crate) hws1: HwDataShared1, - - #[ver(V >= V13_0B4)] - pub(crate) unk_hws2: Array<16, u16>, - - pub(crate) hws2: HwDataShared2, - pub(crate) unk_3c00: u32, - pub(crate) unk_3c04: u32, - pub(crate) hws3: HwDataShared3, - pub(crate) unk_3c58: Array<0x3c, u8>, - pub(crate) unk_3c94: u32, - pub(crate) unk_3c98: U64, - pub(crate) unk_3ca0: U64, - pub(crate) unk_3ca8: U64, - pub(crate) unk_3cb0: U64, - pub(crate) ts_last_idle: U64, - pub(crate) ts_last_poweron: U64, - pub(crate) ts_last_poweroff: U64, - pub(crate) unk_3cd0: U64, - pub(crate) unk_3cd8: U64, - - #[ver(V >= V13_0B4)] - pub(crate) unk_3ce0_0: u32, - - pub(crate) unk_3ce0: u32, - pub(crate) unk_3ce4: u32, - pub(crate) unk_3ce8: u32, - pub(crate) unk_3cec: u32, - pub(crate) unk_3cf0: u32, - pub(crate) core_leak_coef: Array<8, F32>, - pub(crate) sram_leak_coef: Array<8, F32>, - - #[ver(V >= V13_0B4)] - pub(crate) aux_leak_coef: AuxLeakCoef, - #[ver(V >= V13_0B4)] - pub(crate) unk_3d34_0: Array<0x18, u8>, - - pub(crate) unk_3d34: Array<0x38, u8>, - } - #[versions(AGX_NEO)] - default_zeroed!(HwDataA::ver); - #[versions(AGX_NEO)] - no_debug!(HwDataA::ver); - - #[derive(Debug, Default, Clone, Copy)] - #[repr(C)] - pub(crate) struct IOMapping { - pub(crate) phys_addr: U64, - pub(crate) virt_addr: U64, - pub(crate) total_size: u32, - pub(crate) element_size: u32, - pub(crate) readwrite: U64, - } - - #[versions(AGX_NEO)] - const IO_MAPPING_COUNT: usize = { - #[ver(V < V13_0B4)] - { - 0x14 - } - #[ver(V >= V13_0B4 && V < V13_3)] - { - 0x17 - } - #[ver(V >= V13_3 && V < V13_5)] - { - 0x18 - } - #[ver(V >= V13_5)] - { - 0x19 - } - }; - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataBAuxPStates { - pub(crate) cs_max_pstate: u32, - pub(crate) cs_frequencies: Array<0x10, u32>, - pub(crate) cs_voltages: Array<0x10, Array<0x2, u32>>, - pub(crate) cs_voltages_sram: Array<0x10, Array<0x2, u32>>, - pub(crate) cs_unkpad: u32, - pub(crate) afr_max_pstate: u32, - pub(crate) afr_frequencies: Array<0x8, u32>, - pub(crate) afr_voltages: Array<0x8, Array<0x2, u32>>, - pub(crate) afr_voltages_sram: Array<0x8, Array<0x2, u32>>, - pub(crate) afr_unkpad: u32, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct HwDataB { - #[ver(V < V13_0B4)] - pub(crate) unk_0: U64, - - pub(crate) unk_8: U64, - - #[ver(V < V13_0B4)] - pub(crate) unk_10: U64, - - pub(crate) unk_18: U64, - pub(crate) unk_20: U64, - pub(crate) unk_28: U64, - pub(crate) unk_30: U64, - pub(crate) timestamp_area_base: U64, - pub(crate) pad_40: Pad<0x20>, - - #[ver(V < V13_0B4)] - pub(crate) yuv_matrices: Array<0xf, Array<3, Array<4, i16>>>, - - #[ver(V >= V13_0B4)] - pub(crate) yuv_matrices: Array<0x3f, Array<3, Array<4, i16>>>, - - pub(crate) pad_1c8: Pad<0x8>, - pub(crate) io_mappings: Array, - - #[ver(V >= V13_0B4)] - pub(crate) sgx_sram_ptr: U64, - - pub(crate) chip_id: u32, - pub(crate) unk_454: u32, - pub(crate) unk_458: u32, - pub(crate) unk_45c: u32, - pub(crate) unk_460: u32, - pub(crate) unk_464: u32, - pub(crate) unk_468: u32, - pub(crate) unk_46c: u32, - pub(crate) unk_470: u32, - pub(crate) unk_474: u32, - pub(crate) unk_478: u32, - pub(crate) unk_47c: u32, - pub(crate) unk_480: u32, - pub(crate) unk_484: u32, - pub(crate) unk_488: u32, - pub(crate) unk_48c: u32, - pub(crate) base_clock_khz: u32, - pub(crate) power_sample_period: u32, - pub(crate) pad_498: Pad<0x4>, - pub(crate) unk_49c: u32, - pub(crate) unk_4a0: u32, - pub(crate) unk_4a4: u32, - pub(crate) pad_4a8: Pad<0x4>, - pub(crate) unk_4ac: u32, - pub(crate) pad_4b0: Pad<0x8>, - pub(crate) unk_4b8: u32, - pub(crate) unk_4bc: Array<0x4, u8>, - pub(crate) unk_4c0: u32, - pub(crate) unk_4c4: u32, - pub(crate) unk_4c8: u32, - pub(crate) unk_4cc: u32, - pub(crate) unk_4d0: u32, - pub(crate) unk_4d4: u32, - pub(crate) unk_4d8: Array<0x4, u8>, - pub(crate) unk_4dc: u32, - pub(crate) unk_4e0: U64, - pub(crate) unk_4e8: u32, - pub(crate) unk_4ec: u32, - pub(crate) unk_4f0: u32, - pub(crate) unk_4f4: u32, - pub(crate) unk_4f8: u32, - pub(crate) unk_4fc: u32, - pub(crate) unk_500: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_504_0: u32, - - pub(crate) unk_504: u32, - pub(crate) unk_508: u32, - pub(crate) unk_50c: u32, - pub(crate) unk_510: u32, - pub(crate) unk_514: u32, - pub(crate) unk_518: u32, - pub(crate) unk_51c: u32, - pub(crate) unk_520: u32, - pub(crate) unk_524: u32, - pub(crate) unk_528: u32, - pub(crate) unk_52c: u32, - pub(crate) unk_530: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_534_0: u32, - - pub(crate) unk_534: u32, - pub(crate) unk_538: u32, - - pub(crate) num_frags: u32, - pub(crate) unk_540: u32, - pub(crate) unk_544: u32, - pub(crate) unk_548: u32, - pub(crate) unk_54c: u32, - pub(crate) unk_550: u32, - pub(crate) unk_554: u32, - pub(crate) uat_ttb_base: U64, - pub(crate) gpu_core_id: u32, - pub(crate) gpu_rev_id: u32, - pub(crate) num_cores: u32, - pub(crate) max_pstate: u32, - - #[ver(V < V13_0B4)] - pub(crate) num_pstates: u32, - - pub(crate) frequencies: Array<0x10, u32>, - pub(crate) voltages: Array<0x10, [u32; 0x8]>, - pub(crate) voltages_sram: Array<0x10, [u32; 0x8]>, - - #[ver(V >= V13_3)] - pub(crate) unk_9f4_0: Pad<64>, - - pub(crate) sram_k: Array<0x10, F32>, - pub(crate) unk_9f4: Array<0x10, u32>, - pub(crate) rel_max_powers: Array<0x10, u32>, - pub(crate) rel_boost_freqs: Array<0x10, u32>, - - #[ver(V >= V13_3)] - pub(crate) unk_arr_0: Array<32, u32>, - - #[ver(V < V13_0B4)] - pub(crate) min_sram_volt: u32, - - #[ver(V < V13_0B4)] - pub(crate) unk_ab8: u32, - - #[ver(V < V13_0B4)] - pub(crate) unk_abc: u32, - - #[ver(V < V13_0B4)] - pub(crate) unk_ac0: u32, - - #[ver(V >= V13_0B4)] - pub(crate) aux_ps: HwDataBAuxPStates, - - #[ver(V >= V13_3)] - pub(crate) pad_ac4_0: Array<0x44c, u8>, - - pub(crate) pad_ac4: Pad<0x8>, - pub(crate) unk_acc: u32, - pub(crate) unk_ad0: u32, - pub(crate) pad_ad4: Pad<0x10>, - pub(crate) unk_ae4: Array<0x4, u32>, - pub(crate) pad_af4: Pad<0x4>, - pub(crate) unk_af8: u32, - pub(crate) pad_afc: Pad<0x8>, - pub(crate) unk_b04: u32, - pub(crate) unk_b08: u32, - pub(crate) unk_b0c: u32, - - #[ver(G >= G14X)] - pub(crate) pad_b10_0: Array<0x8, u8>, - - pub(crate) unk_b10: u32, - pub(crate) timer_offset: U64, - pub(crate) unk_b1c: u32, - pub(crate) unk_b20: u32, - pub(crate) unk_b24: u32, - pub(crate) unk_b28: u32, - pub(crate) unk_b2c: u32, - pub(crate) unk_b30: u32, - pub(crate) unk_b34: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_b38_0: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_b38_4: u32, - - #[ver(V >= V13_3)] - pub(crate) unk_b38_8: u32, - - pub(crate) unk_b38: Array<0xc, u32>, - pub(crate) unk_b68: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_b6c: Array<0xd0, u8>, - - #[ver(G >= G14X)] - pub(crate) unk_c3c_0: Array<0x8, u8>, - - #[ver(G < G14X && V >= V13_5)] - pub(crate) unk_c3c_8: Array<0x10, u8>, - - #[ver(V >= V13_5)] - pub(crate) unk_c3c_18: Array<0x20, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_c3c: u32, - } - #[versions(AGX_NEO)] - default_zeroed!(HwDataB::ver); - - #[derive(Debug)] - #[repr(C, packed)] - pub(crate) struct GpuStatsVtx { - // This changes all the time and we don't use it, let's just make it a big buffer - pub(crate) opaque: Array<0x3000, u8>, - } - default_zeroed!(GpuStatsVtx); - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct GpuStatsFrag { - // This changes all the time and we don't use it, let's just make it a big buffer - // except for these two fields which may need init. - #[ver(G >= G14X)] - pub(crate) unk1_0: Array<0x910, u8>, - pub(crate) unk1: Array<0x100, u8>, - pub(crate) cur_stamp_id: i32, - pub(crate) unk2: Array<0x14, u8>, - pub(crate) unk_id: i32, - pub(crate) unk3: Array<0x1000, u8>, - } - - #[versions(AGX_NEO)] - impl Default for GpuStatsFrag::ver { - fn default() -> Self { - Self { - #[ver(G >= G14X)] - unk1_0: Default::default(), - unk1: Default::default(), - cur_stamp_id: -1, - unk2: Default::default(), - unk_id: -1, - unk3: Default::default(), - } - } - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct GpuGlobalStatsVtx { - pub(crate) total_cmds: u32, - pub(crate) stats: GpuStatsVtx, - } - default_zeroed!(GpuGlobalStatsVtx); - - #[versions(AGX_NEO)] - #[derive(Debug, Default)] - #[repr(C)] - pub(crate) struct GpuGlobalStatsFrag { - pub(crate) total_cmds: u32, - pub(crate) unk_4: u32, - pub(crate) stats: GpuStatsFrag::ver, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct GpuStatsComp { - // This changes all the time and we don't use it, let's just make it a big buffer - pub(crate) opaque: Array<0x3000, u8>, - } - default_zeroed!(GpuStatsComp); - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RuntimeScratch { - pub(crate) unk_280: Array<0x6800, u8>, - pub(crate) unk_6a80: u32, - pub(crate) gpu_idle: u32, - pub(crate) unkpad_6a88: Pad<0x14>, - pub(crate) unk_6a9c: u32, - pub(crate) unk_ctr0: u32, - pub(crate) unk_ctr1: u32, - pub(crate) unk_6aa8: u32, - pub(crate) unk_6aac: u32, - pub(crate) unk_ctr2: u32, - pub(crate) unk_6ab4: u32, - pub(crate) unk_6ab8: u32, - pub(crate) unk_6abc: u32, - pub(crate) unk_6ac0: u32, - pub(crate) unk_6ac4: u32, - pub(crate) unk_ctr3: u32, - pub(crate) unk_6acc: u32, - pub(crate) unk_6ad0: u32, - pub(crate) unk_6ad4: u32, - pub(crate) unk_6ad8: u32, - pub(crate) unk_6adc: u32, - pub(crate) unk_6ae0: u32, - pub(crate) unk_6ae4: u32, - pub(crate) unk_6ae8: u32, - pub(crate) unk_6aec: u32, - pub(crate) unk_6af0: u32, - pub(crate) unk_ctr4: u32, - pub(crate) unk_ctr5: u32, - pub(crate) unk_6afc: u32, - pub(crate) pad_6b00: Pad<0x38>, - - #[ver(G >= G14X)] - pub(crate) pad_6b00_extra: Array<0x4800, u8>, - - pub(crate) unk_6b38: u32, - pub(crate) pad_6b3c: Pad<0x84>, - } - #[versions(AGX_NEO)] - default_zeroed!(RuntimeScratch::ver); - - #[versions(AGX_NEO)] - #[repr(C)] - pub(crate) struct RuntimePointers<'a> { - pub(crate) pipes: Array<4, PipeChannels::ver>, - - pub(crate) device_control: - ChannelRing, - pub(crate) event: ChannelRing, - pub(crate) fw_log: ChannelRing, - pub(crate) ktrace: ChannelRing, - pub(crate) stats: ChannelRing, - - pub(crate) __pad0: Pad<0x50>, - pub(crate) unk_160: U64, - pub(crate) unk_168: U64, - pub(crate) stats_vtx: GpuPointer<'a, super::GpuGlobalStatsVtx>, - pub(crate) stats_frag: GpuPointer<'a, super::GpuGlobalStatsFrag::ver>, - pub(crate) stats_comp: GpuPointer<'a, super::GpuStatsComp>, - pub(crate) hwdata_a: GpuPointer<'a, super::HwDataA::ver>, - pub(crate) unkptr_190: GpuPointer<'a, &'a [u8]>, - pub(crate) unkptr_198: GpuPointer<'a, &'a [u8]>, - pub(crate) hwdata_b: GpuPointer<'a, super::HwDataB::ver>, - pub(crate) hwdata_b_2: GpuPointer<'a, super::HwDataB::ver>, - pub(crate) fwlog_buf: Option>, - pub(crate) unkptr_1b8: GpuPointer<'a, &'a [u8]>, - - #[ver(G < G14X)] - pub(crate) unkptr_1c0: GpuPointer<'a, &'a [u8]>, - #[ver(G < G14X)] - pub(crate) unkptr_1c8: GpuPointer<'a, &'a [u8]>, - - pub(crate) unk_1d0: u32, - pub(crate) unk_1d4: u32, - pub(crate) unk_1d8: Array<0x3c, u8>, - pub(crate) buffer_mgr_ctl_gpu_addr: U64, - pub(crate) buffer_mgr_ctl_fw_addr: U64, - pub(crate) __pad1: Pad<0x5c>, - pub(crate) gpu_scratch: RuntimeScratch::ver, - } - #[versions(AGX_NEO)] - no_debug!(RuntimePointers::ver<'_>); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct PendingStamp { - pub(crate) info: AtomicU32, - pub(crate) wait_value: AtomicU32, - } - default_zeroed!(PendingStamp); - - #[derive(Debug, Clone, Copy)] - #[repr(C, packed)] - pub(crate) struct FaultInfo { - pub(crate) unk_0: u32, - pub(crate) unk_4: u32, - pub(crate) queue_uuid: u32, - pub(crate) unk_c: u32, - pub(crate) unk_10: u32, - pub(crate) unk_14: u32, - } - default_zeroed!(FaultInfo); - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct PowerZoneGlobal { - pub(crate) target: u32, - pub(crate) target_off: u32, - pub(crate) filter_tc: u32, - } - default_zeroed!(PowerZoneGlobal); - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Globals { - pub(crate) ktrace_enable: u32, - pub(crate) unk_4: Array<0x20, u8>, - - #[ver(V >= V13_2)] - pub(crate) unk_24_0: u32, - - pub(crate) unk_24: u32, - - #[ver(V >= V13_0B4)] - pub(crate) debug: u32, - - #[ver(V >= V13_3)] - pub(crate) unk_28_4: u32, - - pub(crate) unk_28: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_2c_0: u32, - - pub(crate) unk_2c: u32, - pub(crate) unk_30: u32, - pub(crate) unk_34: u32, - pub(crate) unk_38: Array<0x1c, u8>, - - // pub(crate) sub: GlobalsSub::ver, - pub(crate) unk_54: u16, - pub(crate) unk_56: u16, - pub(crate) unk_58: u16, - pub(crate) unk_5a: U32, - pub(crate) unk_5e: U32, - pub(crate) unk_62: U32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_66_0: Array<0xc, u8>, - - pub(crate) unk_66: U32, - pub(crate) unk_6a: Array<0x16, u8>, - // end GlobalsSub::ver - - pub(crate) unk_80: Array<0xf80, u8>, - pub(crate) unk_1000: Array<0x7000, u8>, - pub(crate) unk_8000: Array<0x900, u8>, - - #[ver(G >= G14X)] - pub(crate) unk_8900_pad: Array<0x484c, u8>, - - #[ver(V >= V13_3)] - pub(crate) unk_8900_pad2: Array<0x54, u8>, - - pub(crate) unk_8900: u32, - pub(crate) pending_submissions: AtomicU32, - pub(crate) max_power: u32, - pub(crate) max_pstate_scaled: u32, - pub(crate) max_pstate_scaled_2: u32, - pub(crate) unk_8914: u32, - pub(crate) unk_8918: u32, - pub(crate) max_pstate_scaled_3: u32, - pub(crate) unk_8920: u32, - pub(crate) power_zone_count: u32, - pub(crate) avg_power_filter_tc_periods: u32, - pub(crate) avg_power_ki_dt: F32, - pub(crate) avg_power_kp: F32, - pub(crate) avg_power_min_duty_cycle: u32, - pub(crate) avg_power_target_filter_tc: u32, - pub(crate) power_zones: Array<5, PowerZoneGlobal>, - pub(crate) unk_8978: Array<0x44, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_89bc_0: Array<0x3c, u8>, - - pub(crate) unk_89bc: u32, - pub(crate) fast_die0_release_temp: u32, - pub(crate) unk_89c4: i32, - pub(crate) fast_die0_prop_tgt_delta: u32, - pub(crate) fast_die0_kp: F32, - pub(crate) fast_die0_ki_dt: F32, - pub(crate) unk_89d4: Array<0xc, u8>, - pub(crate) unk_89e0: u32, - pub(crate) max_power_2: u32, - pub(crate) ppm_kp: F32, - pub(crate) ppm_ki_dt: F32, - pub(crate) unk_89f0: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_89f4_0: Array<0x8, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_89f4_8: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_89f4_c: Array<0x50, u8>, - - #[ver(V >= V13_3)] - pub(crate) unk_89f4_5c: Array<0xc, u8>, - - pub(crate) unk_89f4: u32, - pub(crate) hws1: HwDataShared1, - pub(crate) hws2: HwDataShared2, - - #[ver(V >= V13_0B4)] - pub(crate) idle_off_standby_timer: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_hws2_4: Array<0x8, F32>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_hws2_24: u32, - - pub(crate) unk_hws2_28: u32, - - pub(crate) hws3: HwDataShared3, - pub(crate) unk_9004: Array<8, u8>, - pub(crate) unk_900c: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_9010_0: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_9010_4: Array<0x14, u8>, - - pub(crate) unk_9010: Array<0x2c, u8>, - pub(crate) unk_903c: u32, - pub(crate) unk_9040: Array<0xc0, u8>, - pub(crate) unk_9100: Array<0x6f00, u8>, - pub(crate) unk_10000: Array<0xe50, u8>, - pub(crate) unk_10e50: u32, - pub(crate) unk_10e54: Array<0x2c, u8>, - - #[ver((G >= G14X && V < V13_3) || (G <= G14 && V >= V13_3))] - pub(crate) unk_x_pad: Array<0x4, u8>, - - // bit 0: sets sgx_reg 0x17620 - // bit 1: sets sgx_reg 0x17630 - pub(crate) fault_control: u32, - pub(crate) do_init: u32, - pub(crate) unk_10e88: Array<0x188, u8>, - pub(crate) idle_ts: U64, - pub(crate) idle_unk: U64, - pub(crate) progress_check_interval_3d: u32, - pub(crate) progress_check_interval_ta: u32, - pub(crate) progress_check_interval_cl: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_1102c_0: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_1102c_4: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_1102c_8: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_1102c_c: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_1102c_10: u32, - - pub(crate) unk_1102c: u32, - pub(crate) idle_off_delay_ms: AtomicU32, - pub(crate) fender_idle_off_delay_ms: u32, - pub(crate) fw_early_wake_timeout_ms: u32, - #[ver(V == V13_3)] - pub(crate) ps_pad_0: Pad<0x8>, - pub(crate) pending_stamps: Array<0x100, PendingStamp>, - #[ver(V != V13_3)] - pub(crate) ps_pad_0: Pad<0x8>, - pub(crate) unkpad_ps: Pad<0x78>, - pub(crate) unk_117bc: u32, - pub(crate) fault_info: FaultInfo, - pub(crate) counter: u32, - pub(crate) unk_118dc: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_118e0_0: Array<0x9c, u8>, - - #[ver(G >= G14X)] - pub(crate) unk_118e0_9c: Array<0x580, u8>, - - #[ver(V >= V13_3)] - pub(crate) unk_118e0_9c_x: Array<0x8, u8>, - - pub(crate) cl_context_switch_timeout_ms: u32, - - #[ver(V >= V13_0B4)] - pub(crate) cl_kill_timeout_ms: u32, - - pub(crate) cdm_context_store_latency_threshold: u32, - pub(crate) unk_118e8: u32, - pub(crate) unk_118ec: Array<0x400, u8>, - pub(crate) unk_11cec: Array<0x54, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_11d40: Array<0x19c, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_11edc: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_11ee0: Array<0x1c, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_11efc: u32, - - #[ver(V >= V13_3)] - pub(crate) unk_11f00: Array<0x280, u8>, - } - #[versions(AGX_NEO)] - default_zeroed!(Globals::ver); - - #[derive(Debug, Default, Clone, Copy)] - #[repr(C, packed)] - pub(crate) struct UatLevelInfo { - pub(crate) unk_3: u8, - pub(crate) unk_1: u8, - pub(crate) unk_2: u8, - pub(crate) index_shift: u8, - pub(crate) num_entries: u16, - pub(crate) unk_4: u16, - pub(crate) unk_8: U64, - pub(crate) unk_10: U64, - pub(crate) index_mask: U64, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct InitData<'a> { - #[ver(V >= V13_0B4)] - pub(crate) ver_info: Array<0x4, u16>, - - pub(crate) unk_buf: GpuPointer<'a, &'a [u8]>, - pub(crate) unk_8: u32, - pub(crate) unk_c: u32, - pub(crate) runtime_pointers: GpuPointer<'a, super::RuntimePointers::ver>, - pub(crate) globals: GpuPointer<'a, super::Globals::ver>, - pub(crate) fw_status: GpuPointer<'a, super::FwStatus>, - pub(crate) uat_page_size: u16, - pub(crate) uat_page_bits: u8, - pub(crate) uat_num_levels: u8, - pub(crate) uat_level_info: Array<0x3, UatLevelInfo>, - pub(crate) __pad0: Pad<0x14>, - pub(crate) host_mapped_fw_allocations: u32, - pub(crate) unk_ac: u32, - pub(crate) unk_b0: u32, - pub(crate) unk_b4: u32, - pub(crate) unk_b8: u32, - } -} - -#[derive(Debug)] -pub(crate) struct ChannelRing -where - for<'a> ::Raw<'a>: Debug, -{ - pub(crate) state: GpuObject, - pub(crate) ring: GpuArray, -} - -impl ChannelRing -where - for<'a> ::Raw<'a>: Debug, -{ - pub(crate) fn to_raw(&self) -> raw::ChannelRing { - raw::ChannelRing { - state: Some(self.state.weak_pointer()), - ring: Some(self.ring.weak_pointer()), - } - } -} - -trivial_gpustruct!(FwStatus); -trivial_gpustruct!(GpuGlobalStatsVtx); -#[versions(AGX_NEO)] -trivial_gpustruct!(GpuGlobalStatsFrag::ver); -trivial_gpustruct!(GpuStatsComp); - -#[versions(AGX_NEO)] -trivial_gpustruct!(HwDataA::ver); - -#[versions(AGX_NEO)] -trivial_gpustruct!(HwDataB::ver); - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct Stats { - pub(crate) vtx: GpuObject, - pub(crate) frag: GpuObject, - pub(crate) comp: GpuObject, -} - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct RuntimePointers { - pub(crate) stats: Stats::ver, - - pub(crate) hwdata_a: GpuObject, - pub(crate) unkptr_190: GpuArray, - pub(crate) unkptr_198: GpuArray, - pub(crate) hwdata_b: GpuObject, - - pub(crate) unkptr_1b8: GpuArray, - pub(crate) unkptr_1c0: GpuArray, - pub(crate) unkptr_1c8: GpuArray, - - pub(crate) buffer_mgr_ctl: gem::ObjectRef, - pub(crate) buffer_mgr_ctl_low_mapping: Option, - pub(crate) buffer_mgr_ctl_high_mapping: Option, -} - -#[versions(AGX_NEO)] -impl GpuStruct for RuntimePointers::ver { - type Raw<'a> = raw::RuntimePointers::ver<'a>; -} - -#[versions(AGX_NEO)] -trivial_gpustruct!(Globals::ver); - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct InitData { - pub(crate) unk_buf: GpuArray, - pub(crate) runtime_pointers: GpuObject, - pub(crate) globals: GpuObject, - pub(crate) fw_status: GpuObject, -} - -#[versions(AGX_NEO)] -impl GpuStruct for InitData::ver { - type Raw<'a> = raw::InitData::ver<'a>; -} diff --git a/drivers/gpu/drm/asahi-neo/fw/job.rs b/drivers/gpu/drm/asahi-neo/fw/job.rs deleted file mode 100644 index e4f2f9225ea050..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/job.rs +++ /dev/null @@ -1,160 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Common GPU job firmware structures - -use super::types::*; -use crate::{ - default_zeroed, - mmu, - trivial_gpustruct, // -}; -use kernel::prelude::Result; -use kernel::sync::Arc; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct JobMeta { - pub(crate) unk_0: u16, - pub(crate) unk_2: u8, - pub(crate) no_preemption: u8, - pub(crate) stamp: GpuWeakPointer, - pub(crate) fw_stamp: GpuWeakPointer, - pub(crate) stamp_value: EventValue, - pub(crate) stamp_slot: u32, - pub(crate) evctl_index: u32, - pub(crate) flush_stamps: u32, - pub(crate) uuid: u32, - pub(crate) event_seq: u32, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct EncoderParams { - pub(crate) unk_8: u32, - pub(crate) sync_grow: u32, - pub(crate) unk_10: u32, - pub(crate) encoder_id: u32, - pub(crate) unk_18: u32, - pub(crate) unk_mask: u32, - pub(crate) sampler_array: U64, - pub(crate) sampler_count: u32, - pub(crate) sampler_max: u32, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobTimestamps { - pub(crate) start: AtomicU64, - pub(crate) end: AtomicU64, - } - default_zeroed!(JobTimestamps); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RenderTimestamps { - pub(crate) vtx: JobTimestamps, - pub(crate) frag: JobTimestamps, - } - default_zeroed!(RenderTimestamps); - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Register { - pub(crate) number: u32, - pub(crate) value: U64, - } - default_zeroed!(Register); - - impl Register { - fn new(number: u32, value: u64) -> Register { - Register { - number, - value: U64(value), - } - } - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RegisterArray { - pub(crate) registers: Array<128, Register>, - pub(crate) pad: Array<0x100, u8>, - - pub(crate) addr: GpuWeakPointer>, - pub(crate) count: u16, - pub(crate) length: u16, - pub(crate) unk_pad: u32, - } - - impl RegisterArray { - pub(crate) fn new( - self_ptr: GpuWeakPointer>, - cb: impl FnOnce(&mut RegisterArray), - ) -> RegisterArray { - let mut array = RegisterArray { - registers: Default::default(), - pad: Default::default(), - addr: self_ptr, - count: 0, - length: 0, - unk_pad: 0, - }; - - cb(&mut array); - - array - } - - pub(crate) fn add(&mut self, number: u32, value: u64) { - self.registers[self.count as usize] = Register::new(number, value); - self.count += 1; - self.length += core::mem::size_of::() as u16; - } - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct TimestampPointers<'a> { - pub(crate) start_addr: Option>, - pub(crate) end_addr: Option>, - } -} - -trivial_gpustruct!(JobTimestamps); -trivial_gpustruct!(RenderTimestamps); - -#[derive(Debug)] -pub(crate) struct UserTimestamp { - pub(crate) mapping: Arc, - pub(crate) offset: usize, -} - -#[derive(Debug, Default)] -pub(crate) struct UserTimestamps { - pub(crate) start: Option, - pub(crate) end: Option, -} - -impl UserTimestamps { - pub(crate) fn any(&self) -> bool { - self.start.is_some() || self.end.is_some() - } - - pub(crate) fn pointers(&self) -> Result> { - Ok(raw::TimestampPointers { - start_addr: self - .start - .as_ref() - .map(|a| GpuPointer::from_mapping(&a.mapping, a.offset)) - .transpose()?, - end_addr: self - .end - .as_ref() - .map(|a| GpuPointer::from_mapping(&a.mapping, a.offset)) - .transpose()?, - }) - } -} diff --git a/drivers/gpu/drm/asahi-neo/fw/microseq.rs b/drivers/gpu/drm/asahi-neo/fw/microseq.rs deleted file mode 100644 index af46b1073af7d9..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/microseq.rs +++ /dev/null @@ -1,412 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU firmware microsequence operations - -use super::types::*; -use super::{ - buffer, - compute, - fragment, - initdata, - job, - vertex, - workqueue, // -}; -use crate::default_zeroed; - -pub(crate) trait Operation {} - -#[derive(Debug, Copy, Clone)] -#[repr(u32)] -enum OpCode { - WaitForIdle = 0x01, - WaitForIdle2 = 0x02, - RetireStamp = 0x18, - #[allow(dead_code)] - Timestamp = 0x19, - StartVertex = 0x22, - FinalizeVertex = 0x23, - StartFragment = 0x24, - FinalizeFragment = 0x25, - StartCompute = 0x29, - FinalizeCompute = 0x2a, -} - -#[derive(Debug, Copy, Clone)] -#[repr(u32)] -pub(crate) enum Pipe { - Vertex = 1 << 0, - Fragment = 1 << 8, - Compute = 1 << 15, -} - -pub(crate) const MAX_ATTACHMENTS: usize = 16; - -#[derive(Debug, Clone, Copy)] -#[repr(C)] -pub(crate) struct Attachment { - pub(crate) address: U64, - pub(crate) size: u32, - pub(crate) unk_c: u16, - pub(crate) unk_e: u16, -} -default_zeroed!(Attachment); - -#[derive(Debug, Clone, Copy, Default)] -#[repr(C)] -pub(crate) struct Attachments { - pub(crate) list: Array, - pub(crate) count: u32, -} - -#[derive(Debug, Copy, Clone)] -#[repr(transparent)] -pub(crate) struct OpHeader(u32); - -impl OpHeader { - const fn new(opcode: OpCode) -> OpHeader { - OpHeader(opcode as u32) - } - const fn with_args(opcode: OpCode, args: u32) -> OpHeader { - OpHeader(opcode as u32 | args) - } -} - -macro_rules! simple_op { - ($name:ident) => { - #[allow(dead_code)] - #[derive(Debug, Copy, Clone)] - pub(crate) struct $name(OpHeader); - - impl $name { - pub(crate) const HEADER: $name = $name(OpHeader::new(OpCode::$name)); - } - }; -} - -pub(crate) mod op { - use super::*; - - simple_op!(StartVertex); - simple_op!(FinalizeVertex); - simple_op!(StartFragment); - simple_op!(FinalizeFragment); - simple_op!(StartCompute); - simple_op!(FinalizeCompute); - simple_op!(WaitForIdle2); - - #[allow(dead_code)] - #[derive(Debug, Copy, Clone)] - pub(crate) struct RetireStamp(OpHeader); - impl RetireStamp { - pub(crate) const HEADER: RetireStamp = - RetireStamp(OpHeader::with_args(OpCode::RetireStamp, 0x40000000)); - } - - #[allow(dead_code)] - #[derive(Debug, Copy, Clone)] - pub(crate) struct WaitForIdle(OpHeader); - impl WaitForIdle { - pub(crate) const fn new(pipe: Pipe) -> WaitForIdle { - WaitForIdle(OpHeader::with_args(OpCode::WaitForIdle, (pipe as u32) << 8)) - } - } - - #[allow(dead_code)] - #[derive(Debug, Copy, Clone)] - pub(crate) struct Timestamp(OpHeader); - impl Timestamp { - #[allow(dead_code)] - pub(crate) const fn new(flag: bool) -> Timestamp { - Timestamp(OpHeader::with_args(OpCode::Timestamp, (flag as u32) << 31)) - } - } -} - -#[derive(Debug)] -#[repr(C)] -pub(crate) struct WaitForIdle { - pub(crate) header: op::WaitForIdle, -} - -impl Operation for WaitForIdle {} - -#[derive(Debug)] -#[repr(C)] -pub(crate) struct WaitForIdle2 { - pub(crate) header: op::WaitForIdle2, -} - -impl Operation for WaitForIdle2 {} - -#[derive(Debug)] -#[repr(C)] -pub(crate) struct RetireStamp { - pub(crate) header: op::RetireStamp, -} - -impl Operation for RetireStamp {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct Timestamp<'a> { - pub(crate) header: op::Timestamp, - pub(crate) command_time: GpuWeakPointer, - pub(crate) ts_pointers: GpuWeakPointer>, - // Unused? - pub(crate) update_ts: GpuWeakPointer>>, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) user_ts_pointers: GpuWeakPointer>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_ts: GpuWeakPointer, - - pub(crate) uuid: u32, - pub(crate) unk_30_padding: u32, -} - -#[versions(AGX_NEO)] -impl<'a> Operation for Timestamp::ver<'a> {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct StartVertex<'a> { - pub(crate) header: op::StartVertex, - pub(crate) tiling_params: Option>, - pub(crate) job_params1: Option>>, - #[ver(G >= G14X)] - pub(crate) registers: GpuWeakPointer, - pub(crate) buffer: GpuWeakPointer, - pub(crate) scene: GpuWeakPointer, - pub(crate) stats: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) vm_slot: u32, - pub(crate) unk_38: u32, - pub(crate) event_generation: u32, - pub(crate) buffer_slot: u32, - pub(crate) unk_44: u32, - pub(crate) event_seq: U64, - pub(crate) unk_50: u32, - pub(crate) unk_pointer: GpuWeakPointer, - pub(crate) unk_job_buf: GpuWeakPointer, - pub(crate) unk_64: u32, - pub(crate) unk_68: u32, - pub(crate) uuid: u32, - pub(crate) attachments: Attachments, - pub(crate) padding: u32, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - #[ver(V >= V13_0B4)] - pub(crate) notifier_buf: GpuWeakPointer>, - - pub(crate) unk_178: u32, -} - -#[versions(AGX_NEO)] -impl<'a> Operation for StartVertex::ver<'a> {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct FinalizeVertex { - pub(crate) header: op::FinalizeVertex, - pub(crate) scene: GpuWeakPointer, - pub(crate) buffer: GpuWeakPointer, - pub(crate) stats: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) vm_slot: u32, - pub(crate) unk_28: u32, - pub(crate) unk_pointer: GpuWeakPointer, - pub(crate) unk_34: u32, - pub(crate) uuid: u32, - pub(crate) fw_stamp: GpuWeakPointer, - pub(crate) stamp_value: EventValue, - pub(crate) unk_48: U64, - pub(crate) unk_50: u32, - pub(crate) unk_54: u32, - pub(crate) unk_58: U64, - pub(crate) unk_60: u32, - pub(crate) unk_64: u32, - pub(crate) unk_68: u32, - - #[ver(G >= G14 && V < V13_0B4)] - pub(crate) unk_68_g14: U64, - - pub(crate) restart_branch_offset: i32, - pub(crate) has_attachments: u32, // Check DCMP errors bits 2,3 1=ktrace 2=log 3=panic - - #[ver(V >= V13_0B4)] - pub(crate) unk_74: Array<0x10, u8>, -} - -#[versions(AGX_NEO)] -impl Operation for FinalizeVertex::ver {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct StartFragment<'a> { - pub(crate) header: op::StartFragment, - pub(crate) job_params2: Option>, - pub(crate) job_params1: Option>>, - #[ver(G >= G14X)] - pub(crate) registers: GpuWeakPointer, - pub(crate) scene: GpuPointer<'a, buffer::Scene::ver>, - pub(crate) stats: GpuWeakPointer, - pub(crate) busy_flag: GpuWeakPointer, - pub(crate) tvb_overflow_count: GpuWeakPointer, - pub(crate) unk_pointer: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) work_item: GpuWeakPointer, - pub(crate) vm_slot: u32, - pub(crate) unk_50: u32, - pub(crate) event_generation: u32, - pub(crate) buffer_slot: u32, - pub(crate) sync_grow: u32, - pub(crate) event_seq: U64, - pub(crate) unk_68: u32, - pub(crate) unk_758_flag: GpuWeakPointer, - pub(crate) unk_job_buf: GpuWeakPointer, - #[ver(V >= V13_3)] - pub(crate) unk_7c_0: U64, - pub(crate) unk_7c: u32, - pub(crate) unk_80: u32, - pub(crate) unk_84: u32, - pub(crate) uuid: u32, - pub(crate) attachments: Attachments, - pub(crate) padding: u32, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - #[ver(V >= V13_0B4)] - pub(crate) notifier_buf: GpuWeakPointer>, -} - -#[versions(AGX_NEO)] -impl<'a> Operation for StartFragment::ver<'a> {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct FinalizeFragment { - pub(crate) header: op::FinalizeFragment, - pub(crate) uuid: u32, - pub(crate) unk_8: u32, - pub(crate) fw_stamp: GpuWeakPointer, - pub(crate) stamp_value: EventValue, - pub(crate) unk_18: u32, - pub(crate) scene: GpuWeakPointer, - pub(crate) buffer: GpuWeakPointer, - pub(crate) unk_2c: U64, - pub(crate) stats: GpuWeakPointer, - pub(crate) unk_pointer: GpuWeakPointer, - pub(crate) busy_flag: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) work_item: GpuWeakPointer, - pub(crate) vm_slot: u32, - pub(crate) unk_60: u32, - pub(crate) unk_758_flag: GpuWeakPointer, - #[ver(V >= V13_3)] - pub(crate) unk_6c_0: U64, - pub(crate) unk_6c: U64, - pub(crate) unk_74: U64, - pub(crate) unk_7c: U64, - pub(crate) unk_84: U64, - pub(crate) unk_8c: U64, - - #[ver(G == G14 && V < V13_0B4)] - pub(crate) unk_8c_g14: U64, - - pub(crate) restart_branch_offset: i32, - pub(crate) has_attachments: u32, // Check DCMP errors bits 2,3 1=ktrace 2=log 3=panic - - #[ver(V >= V13_0B4)] - pub(crate) unk_9c: Array<0x10, u8>, -} - -#[versions(AGX_NEO)] -impl Operation for FinalizeFragment::ver {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct StartCompute<'a> { - pub(crate) header: op::StartCompute, - pub(crate) unk_pointer: GpuWeakPointer, - pub(crate) job_params1: Option>>, - #[ver(G >= G14X)] - pub(crate) registers: GpuWeakPointer, - pub(crate) stats: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) vm_slot: u32, - pub(crate) unk_28: u32, - pub(crate) event_generation: u32, - pub(crate) event_seq: U64, - pub(crate) unk_38: u32, - pub(crate) job_params2: GpuWeakPointer>, - pub(crate) unk_44: u32, - pub(crate) uuid: u32, - pub(crate) attachments: Attachments, - pub(crate) padding: u32, - - #[ver(V >= V13_0B4)] - pub(crate) unk_flag: GpuWeakPointer, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - #[ver(V >= V13_0B4)] - pub(crate) notifier_buf: GpuWeakPointer>, -} - -#[versions(AGX_NEO)] -impl<'a> Operation for StartCompute::ver<'a> {} - -#[versions(AGX_NEO)] -#[derive(Debug)] -#[repr(C)] -pub(crate) struct FinalizeCompute<'a> { - pub(crate) header: op::FinalizeCompute, - pub(crate) stats: GpuWeakPointer, - pub(crate) work_queue: GpuWeakPointer, - pub(crate) vm_slot: u32, - #[ver(V < V13_0B4)] - pub(crate) unk_18: u32, - pub(crate) job_params2: GpuWeakPointer>, - pub(crate) unk_24: u32, - pub(crate) uuid: u32, - pub(crate) fw_stamp: GpuWeakPointer, - pub(crate) stamp_value: EventValue, - pub(crate) unk_38: u32, - pub(crate) unk_3c: u32, - pub(crate) unk_40: u32, - pub(crate) unk_44: u32, - pub(crate) unk_48: u32, - pub(crate) unk_4c: u32, - pub(crate) unk_50: u32, - pub(crate) unk_54: u32, - pub(crate) unk_58: u32, - - #[ver(G == G14 && V < V13_0B4)] - pub(crate) unk_5c_g14: U64, - - pub(crate) restart_branch_offset: i32, - pub(crate) has_attachments: u32, // Check DCMP errors bits 2,3 1=ktrace 2=log 3=panic - - #[ver(V >= V13_0B4)] - pub(crate) unk_64: Array<0xd, u8>, - - #[ver(V >= V13_0B4)] - pub(crate) unk_flag: GpuWeakPointer, - - #[ver(V >= V13_0B4)] - pub(crate) unk_79: Array<0x7, u8>, -} - -#[versions(AGX_NEO)] -impl<'a> Operation for FinalizeCompute::ver<'a> {} diff --git a/drivers/gpu/drm/asahi-neo/fw/mod.rs b/drivers/gpu/drm/asahi-neo/fw/mod.rs deleted file mode 100644 index a5649aa20d3a8e..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/mod.rs +++ /dev/null @@ -1,15 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Firmware structures for Apple AGX GPUs - -pub(crate) mod buffer; -pub(crate) mod channels; -pub(crate) mod compute; -pub(crate) mod event; -pub(crate) mod fragment; -pub(crate) mod initdata; -pub(crate) mod job; -pub(crate) mod microseq; -pub(crate) mod types; -pub(crate) mod vertex; -pub(crate) mod workqueue; diff --git a/drivers/gpu/drm/asahi-neo/fw/types.rs b/drivers/gpu/drm/asahi-neo/fw/types.rs deleted file mode 100644 index f55d6cec6b8ca3..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/types.rs +++ /dev/null @@ -1,224 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Common types for firmware structure definitions - -use crate::{alloc, object}; -use core::fmt; -use core::ops::{Deref, DerefMut, Index, IndexMut}; - -pub(crate) use crate::event::EventValue; -pub(crate) use crate::object::{ - GpuPointer, - GpuStruct, - GpuWeakPointer, // -}; -pub(crate) use crate::{ - f32, - float::F32, // -}; - -pub(crate) use core::fmt::Debug; -pub(crate) use core::marker::PhantomData; -pub(crate) use core::sync::atomic::{ - AtomicI32, - AtomicU32, - AtomicU64, // -}; -pub(crate) use kernel::macros::versions; -pub(crate) use kernel::prelude::Zeroable; - -// Make the trait visible -pub(crate) use crate::alloc::Allocator as _Allocator; - -/// General allocator type used for the driver -pub(crate) type Allocator = alloc::DefaultAllocator; - -/// General GpuObject type used for the driver -pub(crate) type GpuObject = - object::GpuObject>; - -/// General GpuArray type used for the driver -pub(crate) type GpuArray = object::GpuArray>; - -/// General GpuOnlyArray type used for the driver -pub(crate) type GpuOnlyArray = - object::GpuOnlyArray>; - -/// A stamp slot that is shared between firmware and the driver. -#[derive(Debug, Default)] -#[repr(transparent)] -pub(crate) struct Stamp(pub(crate) AtomicU32); - -/// A stamp slot that is for private firmware use. -/// -/// This is a separate type to guard against pointer type confusion. -#[derive(Debug, Default)] -#[repr(transparent)] -pub(crate) struct FwStamp(pub(crate) AtomicU32); - -/// An unaligned u64 type. -/// -/// This is useful to avoid having to pack firmware structures entirely, since that is incompatible -/// with `#[derive(Debug)]` and atomics. -#[derive(Copy, Clone, Default)] -#[repr(C, packed(1))] -pub(crate) struct U64(pub(crate) u64); - -// SAFETY: U64 is zeroable just like u64 -unsafe impl Zeroable for U64 {} - -impl fmt::Debug for U64 { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let v = self.0; - f.write_fmt(format_args!("{:#x}", v)) - } -} - -/// An unaligned u32 type. -/// -/// This is useful to avoid having to pack firmware structures entirely, since that is incompatible -/// with `#[derive(Debug)]` and atomics. -#[derive(Copy, Clone, Default)] -#[repr(C, packed(1))] -pub(crate) struct U32(pub(crate) u32); - -// SAFETY: U32 is zeroable just like u32 -unsafe impl Zeroable for U32 {} - -impl fmt::Debug for U32 { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let v = self.0; - f.write_fmt(format_args!("{:#x}", v)) - } -} - -/// Create a dummy `Debug` implementation, for when we need it but it's too painful to write by -/// hand or not very useful. -#[macro_export] -macro_rules! no_debug { - ($type:ty) => { - impl ::core::fmt::Debug for $type { - fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { - write!(f, "...") - } - } - }; -} - -/// Implement Zeroable for a given type (and Default along with it). -/// -/// # Safety -/// -/// This macro must only be used if a type only contains primitive types which can be -/// zero-initialized, FFI structs intended to be zero-initialized, or other types which -/// impl Zeroable. -#[macro_export] -macro_rules! default_zeroed { - (<$($lt:lifetime),*>, $type:ty) => { - impl<$($lt),*> Default for $type { - fn default() -> $type { - unsafe { core::mem::zeroed() } - } - } - // SAFETY: The user is responsible for ensuring this is safe. - unsafe impl<$($lt),*> ::pin_init::Zeroable for $type {} - }; - ($type:ty) => { - impl Default for $type { - fn default() -> $type { - unsafe { core::mem::zeroed() } - } - } - // SAFETY: The user is responsible for ensuring this is safe. - unsafe impl ::pin_init::Zeroable for $type {} - }; -} - -/// A convenience type for a number of padding bytes. Hidden from Debug formatting. -#[derive(Copy, Clone)] -#[repr(C, packed)] -pub(crate) struct Pad([u8; N]); - -/// SAFETY: Primitive type, safe to zero-init. -unsafe impl Zeroable for Pad {} - -impl Default for Pad { - fn default() -> Self { - unsafe { core::mem::zeroed() } - } -} - -impl fmt::Debug for Pad { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_fmt(format_args!("")) - } -} - -/// A convenience type for a fixed-sized array with Default/Zeroable impls. -#[derive(Copy, Clone)] -#[repr(C)] -pub(crate) struct Array([T; N]); - -impl Array { - pub(crate) fn new(data: [T; N]) -> Self { - Self(data) - } -} - -// SAFETY: Arrays of Zeroable values can be safely Zeroable. -unsafe impl Zeroable for Array {} - -impl Default for Array { - fn default() -> Self { - unsafe { core::mem::zeroed() } - } -} - -impl Index for Array { - type Output = T; - - fn index(&self, index: usize) -> &Self::Output { - &self.0[index] - } -} - -impl IndexMut for Array { - fn index_mut(&mut self, index: usize) -> &mut Self::Output { - &mut self.0[index] - } -} - -impl Deref for Array { - type Target = [T; N]; - - fn deref(&self) -> &Self::Target { - &self.0 - } -} - -impl DerefMut for Array { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.0 - } -} - -impl fmt::Debug for Array { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - self.0.fmt(f) - } -} - -/// Convenience macro to define an identically-named trivial GpuStruct with no inner fields for a -/// given raw type name. -#[macro_export] -macro_rules! trivial_gpustruct { - ($type:ident) => { - #[derive(Debug)] - pub(crate) struct $type {} - - impl GpuStruct for $type { - type Raw<'a> = raw::$type; - } - $crate::default_zeroed!($type); - }; -} diff --git a/drivers/gpu/drm/asahi-neo/fw/vertex.rs b/drivers/gpu/drm/asahi-neo/fw/vertex.rs deleted file mode 100644 index 9f5aedc7875b04..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/vertex.rs +++ /dev/null @@ -1,193 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU vertex job firmware structures - -use super::types::*; -use super::{ - event, - job, - workqueue, // -}; -use crate::{ - buffer, - fw, - microseq, - mmu, // -}; -use kernel::sync::Arc; - -pub(crate) mod raw { - use super::*; - - #[derive(Debug, Default, Copy, Clone)] - #[repr(C)] - pub(crate) struct TilingParameters { - pub(crate) rgn_size: u32, - pub(crate) unk_4: u32, - pub(crate) ppp_ctrl: u32, - pub(crate) x_max: u16, - pub(crate) y_max: u16, - pub(crate) te_screen: u32, - pub(crate) te_mtile1: u32, - pub(crate) te_mtile2: u32, - pub(crate) tiles_per_mtile: u32, - pub(crate) tpc_stride: u32, - pub(crate) unk_24: u32, - pub(crate) unk_28: u32, - pub(crate) helper_cfg: u32, - pub(crate) __pad: Pad<0x70>, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters1<'a> { - pub(crate) unk_0: U64, - pub(crate) unk_8: F32, - pub(crate) unk_c: F32, - pub(crate) tvb_tilemap: GpuPointer<'a, &'a [u8]>, - #[ver(G < G14)] - pub(crate) tvb_cluster_tilemaps: Option>, - pub(crate) tpc: GpuPointer<'a, &'a [u8]>, - pub(crate) tvb_heapmeta: GpuPointer<'a, &'a [u8]>, - pub(crate) iogpu_unk_54: U64, - pub(crate) iogpu_unk_56: U64, - #[ver(G < G14)] - pub(crate) tvb_cluster_meta1: Option>, - pub(crate) utile_config: u32, - pub(crate) unk_4c: u32, - pub(crate) ppp_multisamplectl: U64, - pub(crate) tvb_layermeta: GpuPointer<'a, &'a [u8]>, - #[ver(G < G14)] - pub(crate) tvb_cluster_layermeta: Option>, - #[ver(G < G14)] - pub(crate) core_mask: Array<2, u32>, - pub(crate) preempt_buf1: GpuPointer<'a, &'a [u8]>, - pub(crate) preempt_buf2: GpuPointer<'a, &'a [u8]>, - pub(crate) unk_80: U64, - pub(crate) preempt_buf3: GpuPointer<'a, &'a [u8]>, - pub(crate) vdm_ctrl_stream_base: U64, - #[ver(G < G14)] - pub(crate) tvb_cluster_meta2: Option>, - #[ver(G < G14)] - pub(crate) tvb_cluster_meta3: Option>, - #[ver(G < G14)] - pub(crate) tiling_control: u32, - #[ver(G < G14)] - pub(crate) unk_ac: u32, - pub(crate) unk_b0: Array<6, U64>, - pub(crate) usc_exec_base_ta: U64, - #[ver(G < G14)] - pub(crate) tvb_cluster_meta4: Option>, - #[ver(G < G14)] - pub(crate) unk_f0: U64, - pub(crate) unk_f8: U64, - pub(crate) helper_program: u32, - pub(crate) unk_104: u32, - pub(crate) helper_arg: U64, - pub(crate) unk_110: U64, - pub(crate) unk_118: u32, - #[ver(G >= G14)] - pub(crate) __pad: Pad<{ 8 * 9 + 0x268 }>, - #[ver(G < G14)] - pub(crate) __pad: Pad<0x268>, - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct JobParameters2<'a> { - pub(crate) unk_480: Array<4, u32>, - pub(crate) unk_498: U64, - pub(crate) unk_4a0: u32, - pub(crate) preempt_buf1: GpuPointer<'a, &'a [u8]>, - pub(crate) unk_4ac: u32, - pub(crate) unk_4b0: U64, - pub(crate) unk_4b8: u32, - pub(crate) unk_4bc: U64, - pub(crate) unk_4c4_padding: Array<0x48, u8>, - pub(crate) unk_50c: u32, - pub(crate) unk_510: U64, - pub(crate) unk_518: U64, - pub(crate) unk_520: U64, - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RunVertex<'a> { - pub(crate) tag: workqueue::CommandType, - - #[ver(V >= V13_0B4)] - pub(crate) counter: U64, - - pub(crate) vm_slot: u32, - pub(crate) unk_8: u32, - pub(crate) notifier: GpuPointer<'a, event::Notifier::ver>, - pub(crate) buffer_slot: u32, - pub(crate) unk_1c: u32, - pub(crate) buffer: GpuPointer<'a, fw::buffer::Info::ver>, - pub(crate) scene: GpuPointer<'a, fw::buffer::Scene::ver>, - pub(crate) unk_buffer_buf: GpuWeakPointer<[u8]>, - pub(crate) unk_34: u32, - - #[ver(G < G14X)] - pub(crate) job_params1: JobParameters1::ver<'a>, - #[ver(G < G14X)] - pub(crate) tiling_params: TilingParameters, - #[ver(G >= G14X)] - pub(crate) registers: job::raw::RegisterArray, - - pub(crate) tpc: GpuPointer<'a, &'a [u8]>, - pub(crate) tpc_size: U64, - pub(crate) microsequence: GpuPointer<'a, &'a [u8]>, - pub(crate) microsequence_size: u32, - pub(crate) fragment_stamp_slot: u32, - pub(crate) fragment_stamp_value: EventValue, - pub(crate) unk_pointee: u32, - pub(crate) unk_pad: u32, - pub(crate) job_params2: JobParameters2<'a>, - pub(crate) encoder_params: job::raw::EncoderParams, - pub(crate) unk_55c: u32, - pub(crate) unk_560: u32, - pub(crate) sync_grow: u32, - pub(crate) unk_568: u32, - pub(crate) uses_scratch: u32, - pub(crate) meta: job::raw::JobMeta, - pub(crate) unk_after_meta: u32, - pub(crate) unk_buf_0: U64, - pub(crate) unk_buf_8: U64, - pub(crate) unk_buf_10: U64, - pub(crate) command_time: U64, - pub(crate) timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) user_timestamp_pointers: job::raw::TimestampPointers<'a>, - pub(crate) client_sequence: u8, - pub(crate) pad_5d5: Array<3, u8>, - pub(crate) unk_5d8: u32, - pub(crate) unk_5dc: u8, - - #[ver(V >= V13_0B4)] - pub(crate) unk_ts: U64, - - #[ver(V >= V13_0B4)] - pub(crate) unk_5dd_8: Array<0x1b, u8>, - } -} - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct RunVertex { - pub(crate) notifier: Arc>, - pub(crate) scene: Arc, - pub(crate) micro_seq: microseq::MicroSequence, - pub(crate) vm_bind: mmu::VmBind, - pub(crate) timestamps: Arc>, - pub(crate) user_timestamps: job::UserTimestamps, -} - -#[versions(AGX_NEO)] -impl GpuStruct for RunVertex::ver { - type Raw<'a> = raw::RunVertex::ver<'a>; -} - -#[versions(AGX_NEO)] -impl workqueue::Command for RunVertex::ver {} diff --git a/drivers/gpu/drm/asahi-neo/fw/workqueue.rs b/drivers/gpu/drm/asahi-neo/fw/workqueue.rs deleted file mode 100644 index 414c3c4e1a6887..00000000000000 --- a/drivers/gpu/drm/asahi-neo/fw/workqueue.rs +++ /dev/null @@ -1,187 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU work queue firmware structes - -use super::event; -use super::types::*; -use crate::event::EventValue; -use crate::{ - default_zeroed, - trivial_gpustruct, // -}; -use kernel::sync::Arc; - -#[derive(Debug)] -#[repr(u32)] -pub(crate) enum CommandType { - RunVertex = 0, - RunFragment = 1, - #[allow(dead_code)] - RunBlitter = 2, - RunCompute = 3, - Barrier = 4, - InitBuffer = 6, -} - -pub(crate) trait Command: GpuStruct + Send + Sync {} - -pub(crate) mod raw { - use super::*; - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct Barrier { - pub(crate) tag: CommandType, - pub(crate) wait_stamp: GpuWeakPointer, - pub(crate) wait_value: EventValue, - pub(crate) wait_slot: u32, - pub(crate) stamp_self: EventValue, - pub(crate) uuid: u32, - pub(crate) external_barrier: u32, - // G14X addition - pub(crate) internal_barrier_type: u32, - pub(crate) padding: Pad<0x1c>, - } - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct GpuContextData { - pub(crate) unk_0: u8, - pub(crate) unk_1: u8, - unk_2: Array<0x2, u8>, - pub(crate) unk_4: u8, - pub(crate) unk_5: u8, - unk_6: Array<0x18, u8>, - pub(crate) unk_1e: u8, - pub(crate) unk_1f: u8, - unk_20: Array<0x3, u8>, - pub(crate) unk_23: u8, - unk_24: Array<0x1c, u8>, - } - - impl Default for GpuContextData { - fn default() -> Self { - Self { - unk_0: 0xff, - unk_1: 0xff, - unk_2: Default::default(), - unk_4: 0, - unk_5: 1, - unk_6: Default::default(), - unk_1e: 0xff, - unk_1f: 0, - unk_20: Default::default(), - unk_23: 2, - unk_24: Default::default(), - } - } - } - - #[derive(Debug)] - #[repr(C)] - pub(crate) struct RingState { - pub(crate) gpu_doneptr: AtomicU32, - __pad0: Pad<0xc>, - pub(crate) unk_10: AtomicU32, - __pad1: Pad<0xc>, - pub(crate) unk_20: AtomicU32, - __pad2: Pad<0xc>, - pub(crate) gpu_rptr: AtomicU32, - __pad3: Pad<0xc>, - pub(crate) cpu_wptr: AtomicU32, - __pad4: Pad<0xc>, - pub(crate) rb_size: u32, - __pad5: Pad<0xc>, - // This isn't part of the structure, but it's here as a - // debugging hack so we can inspect what ring position - // the driver considered complete and freeable. - pub(crate) cpu_freeptr: AtomicU32, - __pad6: Pad<0xc>, - } - default_zeroed!(RingState); - - #[derive(Debug, Clone, Copy)] - #[repr(C)] - pub(crate) struct Priority( - pub(crate) u32, - pub(crate) u32, - pub(crate) U64, - pub(crate) u32, - pub(crate) u32, - pub(crate) u32, - ); - - pub(crate) const PRIORITY: [Priority; 4] = [ - Priority(0, 0, U64(0xffff_ffff_ffff_0000), 1, 0, 1), - Priority(1, 1, U64(0xffff_ffff_0000_0000), 0, 0, 0), - Priority(2, 2, U64(0xffff_0000_0000_0000), 0, 0, 2), - Priority(3, 3, U64(0x0000_0000_0000_0000), 0, 0, 3), - ]; - - impl Default for Priority { - fn default() -> Priority { - PRIORITY[2] - } - } - - #[versions(AGX_NEO)] - #[derive(Debug)] - #[repr(C)] - pub(crate) struct QueueInfo<'a> { - pub(crate) state: GpuPointer<'a, super::RingState>, - pub(crate) ring: GpuPointer<'a, &'a [u64]>, - pub(crate) notifier_list: GpuPointer<'a, event::NotifierList>, - pub(crate) gpu_buf: GpuPointer<'a, &'a [u8]>, - pub(crate) gpu_rptr1: AtomicU32, - pub(crate) gpu_rptr2: AtomicU32, - pub(crate) gpu_rptr3: AtomicU32, - pub(crate) event_id: AtomicI32, - pub(crate) priority: Priority, - pub(crate) unk_4c: i32, - pub(crate) uuid: u32, - pub(crate) unk_54: i32, - pub(crate) unk_58: U64, - pub(crate) busy: AtomicU32, - pub(crate) __pad: Pad<0x20>, - #[ver(V >= V13_2 && G < G14X)] - pub(crate) unk_84_0: u32, - pub(crate) unk_84_state: AtomicU32, - pub(crate) error_count: AtomicU32, - pub(crate) unk_8c: u32, - pub(crate) unk_90: u32, - pub(crate) unk_94: u32, - pub(crate) pending: AtomicU32, - pub(crate) unk_9c: u32, - pub(crate) gpu_context: GpuPointer<'a, super::GpuContextData>, - pub(crate) unk_a8: U64, - #[ver(V >= V13_2 && G < G14X)] - pub(crate) unk_b0: u32, - } -} - -trivial_gpustruct!(Barrier); -trivial_gpustruct!(RingState); - -impl Command for Barrier {} - -pub(crate) struct GpuContextData { - pub(crate) _buffer: Arc, -} -impl GpuStruct for GpuContextData { - type Raw<'a> = raw::GpuContextData; -} - -#[versions(AGX_NEO)] -#[derive(Debug)] -pub(crate) struct QueueInfo { - pub(crate) state: GpuObject, - pub(crate) ring: GpuArray, - pub(crate) gpu_buf: GpuArray, - pub(crate) notifier_list: Arc>, - pub(crate) gpu_context: Arc, -} - -#[versions(AGX_NEO)] -impl GpuStruct for QueueInfo::ver { - type Raw<'a> = raw::QueueInfo::ver<'a>; -} diff --git a/drivers/gpu/drm/asahi-neo/gem.rs b/drivers/gpu/drm/asahi-neo/gem.rs deleted file mode 100644 index 1edd3a16ebddaa..00000000000000 --- a/drivers/gpu/drm/asahi-neo/gem.rs +++ /dev/null @@ -1,315 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Asahi driver GEM object implementation -//! -//! Basic wrappers and adaptations between generic GEM shmem objects and this driver's -//! view of what a GPU buffer object is. It is in charge of keeping track of all mappings for -//! each GEM object so we can remove them when a client (File) or a Vm are destroyed, as well as -//! implementing RTKit buffers on top of GEM objects for firmware use. - -use kernel::{ - drm_neo, - drm_neo::gem::{ - shmem, - shmem::VMap, - BaseObject, - DriverObject, // - }, - error::Result, - prelude::*, - sync::aref::ARef, - uapi, // -}; - -use core::ops::Range; -use core::sync::atomic::{ - AtomicU64, - Ordering, // -}; - -use crate::{ - debug::*, - driver::{ - AsahiDevice, - AsahiDriver, // - }, - file, - mmu, - util::*, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Gem; - -/// Represents the inner data of a GEM object for this driver. -#[pin_data] -pub(crate) struct AsahiObject { - /// ID for debug - id: u64, - /// Object size retained for live-object accounting. - size: usize, - /// Object creation flags. - flags: u32, - /// Whether this object can be exported. - exportable: bool, - /// Whether this is a kernel-created object. - kernel: bool, - /// Driver aliases outside the user GPUVA tree; they pin the backing. - kernel_mappings: AtomicU64, -} - -/// Type alias for the shmem GEM object type for this driver. -pub(crate) type Object = shmem::Object; - -/// VM-private BOs share their assigned VM's reservation object. Validate this -/// before acquiring mapping state; generic GPUVM accepts external shareable BOs. -pub(crate) fn validate_vm_binding(gem: &Object, vm: &mmu::Vm) -> Result { - if gem.flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE != 0 - && vm.is_extobj(gem) - { - return Err(EINVAL); - } - Ok(()) -} - -unsafe impl Send for AsahiObject {} -unsafe impl Sync for AsahiObject {} - -// /// Type alias for the SGTable type for this driver. -// pub(crate) type SGTable = shmem::SGTable; - -/// A shared reference to a GEM object for this driver. -pub(crate) struct ObjectRef { - /// The underlying GEM object reference - pub(crate) gem: ARef, - /// The kernel-side VMap of this object, if needed - vmap: Option>, -} - -crate::no_debug!(ObjectRef); - -static GEM_ID: AtomicU64 = AtomicU64::new(0); - -impl ObjectRef { - /// Create a new wrapper for a raw GEM object reference. - pub(crate) fn new(gem: ARef) -> ObjectRef { - ObjectRef { gem, vmap: None } - } - - /// Return the `VMap` for this object, creating it if necessary. - pub(crate) fn vmap(&mut self) -> Result> { - if self.vmap.is_none() { - self.vmap = Some(self.gem.owned_vmap()?); - } - self.gem.vmap() - } - - /// Returns the size of an object in bytes - pub(crate) fn size(&self) -> usize { - self.gem.size() - } - - /// Maps an object into a given `Vm` at any free address within a given range. - pub(crate) fn map_into_range( - &mut self, - vm: &crate::mmu::Vm, - range: Range, - alignment: u64, - prot: mmu::Prot, - guard: bool, - ) -> Result { - // Only used for kernel objects now - if !self.gem.kernel { - return Err(EINVAL); - } - vm.map_in_range(&self.gem, 0..self.gem.size(), alignment, range, prot, guard) - } - - /// Maps a range within an object into a given `Vm` at any free address within a given range. - pub(crate) fn map_range_into_range( - &mut self, - vm: &crate::mmu::Vm, - obj_range: Range, - range: Range, - alignment: u64, - prot: mmu::Prot, - guard: bool, - ) -> Result { - if obj_range.end > self.gem.size() { - return Err(EINVAL); - } - validate_vm_binding(&self.gem, vm)?; - vm.map_in_range(&self.gem, obj_range, alignment, range, prot, guard) - } - - /// Maps an object into a given `Vm` at a specific address. - /// - /// Returns Err(ENOSPC) if the requested address is already busy. - pub(crate) fn map_at( - &mut self, - vm: &crate::mmu::Vm, - addr: u64, - prot: mmu::Prot, - guard: bool, - ) -> Result { - validate_vm_binding(&self.gem, vm)?; - - vm.map_at(addr, self.gem.size(), self.gem.clone(), prot, guard) - } -} - -pub(crate) struct AsahiObjConfig { - flags: u32, - exportable: bool, - kernel: bool, -} - -/// Create a new kernel-owned GEM object. -pub(crate) fn new_kernel_object(dev: &AsahiDevice, size: usize) -> Result { - new_kernel_object_mapped(dev, size, false) -} - -/// Create a new kernel-owned GEM object whose kernel mapping is write-combined. -/// -/// Used for memory that a non-coherent agent reads as soon as the host has stored to it and -/// issued a barrier, without cache maintenance. -pub(crate) fn new_kernel_object_wc(dev: &AsahiDevice, size: usize) -> Result { - new_kernel_object_mapped(dev, size, true) -} - -fn new_kernel_object_mapped(dev: &AsahiDevice, size: usize, map_wc: bool) -> Result { - let gem = shmem::Object::::new( - dev, - align(size, mmu::UAT_PGSZ), - shmem::ObjectConfig:: { - reclaimable_cpu_mappings: false, - map_wc, - parent_resv_obj: None, - }, - AsahiObjConfig { - flags: 0, - exportable: false, - kernel: true, - }, - )?; - - mod_pr_debug!("AsahiObject new kernel object id={}\n", gem.id); - Ok(ObjectRef::new(gem)) -} - -/// Create a new user-owned GEM object with the given flags. -pub(crate) fn new_object( - dev: &AsahiDevice, - size: usize, - flags: u32, - parent_object: Option<&shmem::Object>, - reclaim_cpu: bool, -) -> Result> { - if (flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE != 0) != parent_object.is_some() - { - return Err(EINVAL); - } - - let gem = shmem::Object::::new( - dev, - align(size, mmu::UAT_PGSZ), - shmem::ObjectConfig:: { - reclaimable_cpu_mappings: parent_object.is_some() && reclaim_cpu, - map_wc: flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_WRITEBACK == 0, - parent_resv_obj: parent_object, - }, - AsahiObjConfig { - flags, - exportable: parent_object.is_none(), - kernel: false, - }, - )?; - - mod_pr_debug!("AsahiObject new user object: id={}\n", gem.id); - Ok(gem) -} - -#[vtable] -impl DriverObject for AsahiObject { - type Driver = AsahiDriver; - type Args = AsahiObjConfig; - - const HAS_EXPORT: bool = true; - - /// Callback to create the inner data of a GEM object - fn new(_dev: &AsahiDevice, size: usize, args: Self::Args) -> impl PinInit { - let id = GEM_ID.fetch_add(1, Ordering::Relaxed); - mod_pr_debug!("AsahiObject::new id={}\n", id); - try_pin_init!(AsahiObject { - id, - size, - flags: args.flags, - exportable: args.exportable, - kernel: args.kernel, - kernel_mappings: AtomicU64::new(0), - }) - } - - /// Callback to drop all mappings for a GEM object owned by a given `File` - fn close(obj: &::Object, file: &drm_neo::gem::DriverFile) { - // fn close(obj: &Object, file: &DrmFile) { - mod_pr_debug!("AsahiObject::close id={}\n", obj.id); - if file::File::unbind_gem_object(file, obj).is_err() { - pr_err!("AsahiObject::close: Failed to unbind GEM object\n"); - } - } - - /// Optional handle for exporting a gem object. - fn export( - obj: &::Object, - flags: u32, - ) -> Result> { - if !obj.exportable { - return Err(EINVAL); - } - - obj.prime_export(flags) - } -} - -/// A driver-created alias of a GEM object, possibly in a root the user GPUVA -/// tree does not describe. It keeps the object out of idle reclaim. -pub(crate) struct KernelMappingPin { - object: ARef, -} - -impl KernelMappingPin { - pub(crate) fn new(object: &Object) -> Self { - object.kernel_mappings.fetch_add(1, Ordering::AcqRel); - Self { - object: object.into(), - } - } -} - -impl core::ops::Deref for KernelMappingPin { - type Target = Object; - fn deref(&self) -> &Object { - &self.object - } -} - -impl Drop for KernelMappingPin { - fn drop(&mut self) { - self.object.kernel_mappings.fetch_sub(1, Ordering::Release); - } -} - -impl AsahiObject { - pub(crate) fn idle_reclaim_candidate(&self) -> bool { - !self.kernel && !self.exportable && self.kernel_mappings.load(Ordering::Acquire) == 0 - } -} - -/// Purge advice for a VM-private, non-exportable object. Returns whether the -/// contents are retained (0 once purged). -pub(crate) fn madvise(gem: &Object, dontneed: bool) -> Result { - if gem.flags & uapi::drm_asahi_neo_gem_flags_DRM_ASAHI_NEO_GEM_VM_PRIVATE == 0 || gem.exportable { - return Err(EINVAL); - } - Ok(u32::from(gem.madvise(i32::from(dontneed)))) -} diff --git a/drivers/gpu/drm/asahi-neo/gpu.rs b/drivers/gpu/drm/asahi-neo/gpu.rs deleted file mode 100644 index 7ca3c505e30d37..00000000000000 --- a/drivers/gpu/drm/asahi-neo/gpu.rs +++ /dev/null @@ -1,1733 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Top-level GPU manager -//! -//! This module is the root of all GPU firmware management for a given driver instance. It is -//! responsible for initialization, owning the top-level managers (events, UAT, etc.), and -//! communicating with the raw RtKit endpoints to send and receive messages to/from the GPU -//! firmware. -//! -//! It is also the point where diverging driver firmware/GPU variants (using the versions macro) -//! are unified, so that the top level of the driver itself (in `driver`) does not have to concern -//! itself with version dependence. - -use core::any::Any; -use core::ops::Range; -use core::slice; -use core::sync::atomic::{ - AtomicBool, - AtomicU64, - Ordering, // -}; - -use kernel::{ - c_str, - drm_neo::gem::shmem, - error::code::*, - io::mem::{ - Mem, - MemFlag, // - }, - iosys_map::IoSysMapRef, - macros::versions, - new_mutex, - prelude::*, - soc::apple::rtkit, - sync::{ - lock::{ - mutex::MutexBackend, - Guard, // - }, - Arc, - Mutex, - UniqueArc, // - }, - time::{ - Delta, - Instant, - Monotonic, // - }, - types::ForeignOwnable, - uapi, // -}; -#[cfg(CONFIG_DEV_COREDUMP)] -use kernel::{ - devcoredump, - time::msecs_to_jiffies, // -}; - -use crate::alloc::Allocator; -use crate::debug::*; -use crate::driver::{ - AsahiDevRef, - AsahiDevice, // -}; -use crate::fw::channels::{ - ChannelErrorType, - PipeType, // -}; -use crate::fw::types::{ - U32, - U64, // -}; -use crate::{ - alloc, - buffer, - channel, - event, - fw, - gem, - hw, - initdata, - mem, - mmu, - queue, - regs, - stats, - workqueue, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Gpu; - -/// Firmware endpoint for init & incoming notifications. -const EP_FIRMWARE: u8 = 0x20; - -/// Doorbell endpoint for work/message submissions. -const EP_DOORBELL: u8 = 0x21; - -/// Initialize the GPU firmware. -const MSG_INIT: u64 = 0x81 << 48; -const INIT_DATA_MASK: u64 = (1 << 44) - 1; - -/// TX channel doorbell. -const MSG_TX_DOORBELL: u64 = 0x83 << 48; -/// Firmware control channel doorbell. -const MSG_FWCTL: u64 = 0x84 << 48; -// /// Halt the firmware (?). -// const MSG_HALT: u64 = 0x85 << 48; - -/// Receive channel doorbell notification. -const MSG_RX_DOORBELL: u64 = 0x42 << 48; - -/// Doorbell number for firmware kicks/wakeups. -const DOORBELL_KICKFW: u64 = 0x10; -/// Doorbell number for device control channel kicks. -const DOORBELL_DEVCTRL: u64 = 0x11; - -// Upper kernel half VA address ranges. -/// Private (cached) firmware structure VA range base. -const IOVA_KERN_PRIV_RANGE: Range = 0xffffffa000000000..0xffffffa600000000; -/// Private (cached) GPU-RO firmware structure VA range base. -const IOVA_KERN_GPU_RO_RANGE: Range = 0xffffffa600000000..0xffffffa800000000; -/// Shared (uncached) firmware structure VA range base. -const IOVA_KERN_SHARED_RANGE: Range = 0xffffffa800000000..0xffffffaa00000000; -/// Shared (uncached) read-only firmware structure VA range base. -const IOVA_KERN_SHARED_RO_RANGE: Range = 0xffffffaa00000000..0xffffffac00000000; -/// GPU/FW shared structure VA range base. -const IOVA_KERN_GPU_RANGE: Range = 0xffffffac00000000..0xffffffae00000000; -/// GPU/FW shared structure VA range base. -const IOVA_KERN_RTKIT_RANGE: Range = 0xffffffae00000000..0xffffffae10000000; -/// Shared (uncached) timestamp region. -pub(crate) const IOVA_KERN_TIMESTAMP_RANGE: Range = 0xffffffae10000000..0xffffffae14000000; -/// FW MMIO VA range base. -const IOVA_KERN_MMIO_RANGE: Range = 0xffffffaf00000000..0xffffffb000000000; - -/// GPU/FW buffer manager control address (context 0 low) -pub(crate) const IOVA_KERN_GPU_BUFMGR_LOW: u64 = 0x20_0000_0000; -/// GPU/FW buffer manager control address (context 0 high) -pub(crate) const IOVA_KERN_GPU_BUFMGR_HIGH: u64 = 0xffffffaeffff0000; - -/// Timeout for entering the halt state after a fault or request. -const HALT_ENTER_TIMEOUT: Delta = Delta::from_millis(100); - -/// Maximum amount of firmware-private memory garbage allowed before collection. -/// Collection flushes the FW cache and is expensive, so this needs to be -/// reasonably high. -const MAX_FW_ALLOC_GARBAGE_BYTES: usize = 16 * 1024 * 1024; -/// Maximum count of firmware-private memory garbage objects allowed before collection. -/// This works out to 16K of memory in the garbage list (8 bytes each), which keeps us -/// within the safe range for kmalloc (on 16K page systems). -const MAX_FW_ALLOC_GARBAGE_OBJECTS: usize = 2048; - -/// Global allocators used for kernel-half structures. -pub(crate) struct KernelAllocators { - pub(crate) private: alloc::DefaultAllocator, - pub(crate) shared: alloc::DefaultAllocator, - pub(crate) shared_ro: alloc::DefaultAllocator, - #[allow(dead_code)] - pub(crate) gpu: alloc::DefaultAllocator, - pub(crate) gpu_ro: alloc::DefaultAllocator, -} - -/// Receive (GPU->driver) ring buffer channels. -#[versions(AGX_NEO)] -#[pin_data] -struct RxChannels { - event: channel::EventChannel::ver, - fw_log: channel::FwLogChannel, - ktrace: channel::KTraceChannel, - stats: channel::StatsChannel::ver, -} - -/// GPU work submission pipe channels (driver->GPU). -#[versions(AGX_NEO)] -struct PipeChannels { - pub(crate) vtx: KVec>>>, - pub(crate) frag: KVec>>>, - pub(crate) comp: KVec>>>, -} - -/// Misc command transmit (driver->GPU) channels. -#[versions(AGX_NEO)] -#[pin_data] -struct TxChannels { - pub(crate) device_control: channel::DeviceControlChannel::ver, -} - -/// Number of work submission pipes per type, one for each priority level. -const NUM_PIPES: usize = 4; - -/// A generic monotonically incrementing ID used to uniquely identify object instances within the -/// driver. -pub(crate) struct ID(AtomicU64); - -impl ID { - /// Create a new ID counter with a given value. - fn new(val: u64) -> ID { - ID(AtomicU64::new(val)) - } - - /// Fetch the next unique ID. - pub(crate) fn next(&self) -> u64 { - self.0.fetch_add(1, Ordering::Relaxed) - } -} - -impl Default for ID { - /// IDs default to starting at 2, as 0/1 are considered reserved for the system. - fn default() -> Self { - Self::new(2) - } -} - -/// A guard representing one active submission on the GPU. When dropped, decrements the active -/// submission count. -pub(crate) struct OpGuard(Arc); - -impl Drop for OpGuard { - fn drop(&mut self) { - self.0.end_op(); - } -} - -/// Set of global sequence IDs used in the driver. -#[derive(Default)] -pub(crate) struct SequenceIDs { - /// `File` instance ID. - pub(crate) file: ID, - /// `Vm` instance ID. - pub(crate) vm: ID, - /// Submission instance ID. - pub(crate) submission: ID, - /// `Queue` instance ID. - pub(crate) queue: ID, -} - -/// Top-level GPU manager that owns all the global state relevant to the driver instance. -#[versions(AGX_NEO)] -#[pin_data] -pub(crate) struct GpuManager { - dev: AsahiDevRef, - cfg: &'static hw::HwConfig, - dyncfg: hw::DynConfig, - pub(crate) initdata: fw::types::GpuObject, - uat: mmu::Uat, - crashed: AtomicBool, - #[pin] - alloc: Mutex, - io_mappings: KVec, - next_mmio_iova: u64, - #[pin] - rtkit: Mutex>>, - #[pin] - rx_channels: Mutex, - #[pin] - tx_channels: Mutex, - #[pin] - fwctl_channel: Mutex, - pipes: PipeChannels::ver, - event_manager: Arc, - stats: Arc, - buffer_mgr: buffer::BufferManager::ver, - ids: SequenceIDs, - #[allow(clippy::vec_box)] - #[pin] - garbage_contexts: Mutex>>>, -} - -/// Operations the DRM driver performs on a GPU, independent of its firmware interface. -pub(crate) trait Gpu: Send + Sync { - /// Return a snapshot only for firmware families with a supported decoder. - fn stats_snapshot(&self) -> Option> { - None - } - - /// Cast as an Any type. - fn as_any(&self) -> &dyn Any; - /// Cast Arc as an Any type. - fn arc_as_any(self: Arc) -> Arc; - /// Return the G13/G14 firmware manager, if this GPU uses that firmware interface. - fn manager(&self) -> Option<&dyn GpuManager>; - /// Initialize the GPU. - fn init(&self) -> Result; - /// Return a reference to the global `SequenceIDs` instance. - fn ids(&self) -> &SequenceIDs; - /// Check whether the GPU is crashed - fn is_crashed(&self) -> bool; - /// Fill in the hardware description reported to userspace. - fn get_params(&self, params: &mut uapi::drm_asahi_neo_params_global) -> Result; - /// Whether user VMs retain a sticky error and support a userspace mirror. - fn supports_vm_status(&self) -> bool { - false - } - /// Limits for independently executing queues, if supported by this GPU. - fn queue_limits(&self) -> Option { - None - } - /// Temporary capacity hint and validity in milliseconds for subsequent syncobj waits. - fn syncobj_wait_hint(&self) -> Option<(u32, u32)> { - None - } - /// Frequency of the CPU timer that GPU timestamps are derived from. - fn base_clock_hz(&self) -> u32; - /// Return the GPU address space geometry. - fn uat_geometry(&self) -> mmu::UatGeometry; - /// Create a new `Vm` with the given range reserved for kernel-managed objects. - fn new_vm(&self, kernel_range: Range) -> Result; - /// Create a new user command queue. - fn new_queue( - &self, - vm: mmu::Vm, - ualloc: Arc>, - ualloc_priv: Arc>, - priority: u32, - usc_exec_base: u64, - ) -> Result>; - /// Map a BO as a timestamp buffer - fn map_timestamp_buffer( - &self, - bo: gem::ObjectRef, - range: Range, - ) -> Result; -} - -/// Trait used to abstract the firmware/GPU-dependent variants of the GpuManager. -pub(crate) trait GpuManager: Send + Sync { - /// Update the GPU globals from global info - /// - /// TODO: Unclear what can and cannot be updated like this. - fn update_globals(&self); - /// Get a reference to the KernelAllocators. - fn alloc(&self) -> Guard<'_, KernelAllocators, MutexBackend>; - /// Bind a `Vm` to an available slot and return the `VmBind`. - fn bind_vm(&self, vm: &mmu::Vm) -> Result; - /// Kick the firmware (wake it up if asleep). - /// - /// This should be useful to reduce latency on work submission, so we can ask the firmware to - /// wake up while we do some preparatory work for the work submission. - fn kick_firmware(&self) -> Result; - /// Flush the entire firmware cache. - /// - /// TODO: Does this actually work? - fn flush_fw_cache(&self) -> Result; - /// Handle a GPU work timeout event. - fn handle_timeout(&self, counter: u32, event_slot: i32, unk: u32); - /// Handle a GPU fault event. - fn handle_fault(&self); - /// Handle a channel error event. - fn handle_channel_error( - &self, - error_type: ChannelErrorType, - pipe_type: u32, - event_slot: u32, - event_value: u32, - ); - /// Acknowledge a Buffer grow op. - fn ack_grow(&self, buffer_slot: u32, vm_slot: u32, counter: u32); - /// Send a firmware control command (secure cache flush). - fn fwctl(&self, msg: fw::channels::FwCtlMsg) -> Result; - /// Get the static GPU configuration for this SoC. - fn get_cfg(&self) -> &'static hw::HwConfig; - /// Get the dynamic GPU configuration for this SoC. - fn get_dyncfg(&self) -> &hw::DynConfig; - /// Register an unused context as garbage - fn free_context(&self, data: KBox>); -} - -/// Private generic trait for functions that don't need to escape this module. -trait GpuManagerPriv { - /// Decrement the pending submission counter. - fn end_op(&self); -} - -pub(crate) struct RtkitObject { - vmap: shmem::VMap, - mapping: mmu::KernelMapping, -} - -impl rtkit::Buffer for RtkitObject { - fn iova(&self) -> Result { - Ok(self.mapping.iova() as usize) - } - fn buf(&mut self) -> Result> { - Ok(self.vmap.get()) - } -} - -#[versions(AGX_NEO)] -#[vtable] -impl rtkit::Operations for GpuManager::ver { - type Data = Arc; - type Buffer = RtkitObject; - - fn recv_message(data: ::Borrowed<'_>, ep: u8, msg: u64) { - let dev = &data.dev; - //dev_info!(dev.as_ref(), "RtKit message: {:#x}:{:#x}\n", ep, msg); - - if ep != EP_FIRMWARE || msg != MSG_RX_DOORBELL { - dev_err!(dev.as_ref(), "Unknown message: {:#x}:{:#x}\n", ep, msg); - return; - } - - let mut ch = data.rx_channels.lock(); - - ch.fw_log.poll(); - ch.ktrace.poll(); - ch.stats.poll(); - ch.event.poll(); - } - - fn crashed(data: ::Borrowed<'_>, crashlog: Option<&[u8]>) { - let dev = &data.dev; - - data.crashed.store(true, Ordering::Relaxed); - - #[cfg(CONFIG_DEV_COREDUMP)] - if let Err(e) = data.generate_crashdump(crashlog) { - dev_err!(dev.as_ref(), "Could not generate crashdump: {:?}\n", e); - } - #[cfg(not(CONFIG_DEV_COREDUMP))] - let _ = crashlog; - - if debug_enabled(DebugFlags::OopsOnGpuCrash) { - panic!("GPU firmware crashed"); - } else { - dev_err!(dev.as_ref(), "GPU firmware crashed, failing all jobs\n"); - data.event_manager.fail_all(workqueue::WorkError::NoDevice); - } - } - - fn shmem_alloc( - data: ::Borrowed<'_>, - size: usize, - ) -> Result { - let dev = &data.dev; - mod_dev_dbg!(dev, "shmem_alloc() {:#x} bytes\n", size); - - let mut obj = gem::new_kernel_object(dev, size)?; - let vmap = obj.gem.owned_vmap()?; - let mapping = obj.map_into_range( - data.uat.kernel_vm(), - IOVA_KERN_RTKIT_RANGE, - mmu::UAT_PGSZ as u64, - mmu::PROT_FW_SHARED_RW, - true, - )?; - mod_dev_dbg!(dev, "shmem_alloc() -> VA {:#x}\n", mapping.iova()); - Ok(RtkitObject { vmap, mapping }) - } -} - -#[versions(AGX_NEO)] -impl GpuManager::ver { - /// Create a new GpuManager of this version/GPU combination. - #[inline(never)] - pub(crate) fn new( - dev: &AsahiDevice, - res: ®s::Resources, - cfg: &'static hw::HwConfig, - ) -> Result> { - let uat = Self::make_uat(dev, cfg)?; - let dyncfg = Self::make_dyncfg(dev, res, cfg, &uat)?; - - let mut alloc = KernelAllocators { - private: alloc::DefaultAllocator::new( - dev, - uat.kernel_vm(), - IOVA_KERN_PRIV_RANGE, - 0x80, - mmu::PROT_FW_PRIV_RW, - 1024 * 1024, - true, - fmt!("Kernel Private"), - true, - )?, - shared: alloc::DefaultAllocator::new( - dev, - uat.kernel_vm(), - IOVA_KERN_SHARED_RANGE, - 0x80, - mmu::PROT_FW_SHARED_RW, - 1024 * 1024, - true, - fmt!("Kernel Shared"), - false, - )?, - shared_ro: alloc::DefaultAllocator::new( - dev, - uat.kernel_vm(), - IOVA_KERN_SHARED_RO_RANGE, - 0x80, - mmu::PROT_FW_SHARED_RO, - 64 * 1024, - true, - fmt!("Kernel RO Shared"), - false, - )?, - gpu: alloc::DefaultAllocator::new( - dev, - uat.kernel_vm(), - IOVA_KERN_GPU_RANGE, - 0x80, - mmu::PROT_GPU_FW_SHARED_RW, - 64 * 1024, - true, - fmt!("Kernel GPU Shared"), - false, - )?, - gpu_ro: alloc::DefaultAllocator::new( - dev, - uat.kernel_vm(), - IOVA_KERN_GPU_RO_RANGE, - 0x80, - mmu::PROT_GPU_RO_FW_PRIV_RW, - 1024 * 1024, - true, - fmt!("Kernel GPU RO Shared"), - true, - )?, - }; - - let event_manager = Self::make_event_manager(&mut alloc)?; - let mut initdata = Self::make_initdata(dev, cfg, &dyncfg, &mut alloc)?; - - initdata.runtime_pointers.buffer_mgr_ctl_low_mapping = - Some(initdata.runtime_pointers.buffer_mgr_ctl.map_at( - uat.kernel_lower_vm(), - IOVA_KERN_GPU_BUFMGR_LOW, - mmu::PROT_GPU_SHARED_RW, - false, - )?); - initdata.runtime_pointers.buffer_mgr_ctl_high_mapping = - Some(initdata.runtime_pointers.buffer_mgr_ctl.map_at( - uat.kernel_vm(), - IOVA_KERN_GPU_BUFMGR_HIGH, - mmu::PROT_FW_SHARED_RW, - false, - )?); - - let mut mgr = Self::make_mgr(dev, cfg, dyncfg, uat, alloc, event_manager, initdata)?; - - { - let fwctl = mgr.fwctl_channel.lock(); - let p_fwctl = fwctl.to_raw(); - core::mem::drop(fwctl); - - mgr.as_mut() - .initdata_mut() - .fw_status - .with_mut(|raw, _inner| { - raw.fwctl_channel = p_fwctl; - }); - } - - { - let txc = mgr.tx_channels.lock(); - let p_device_control = txc.device_control.to_raw(); - core::mem::drop(txc); - - let rxc = mgr.rx_channels.lock(); - let p_event = rxc.event.to_raw(); - let p_fw_log = rxc.fw_log.to_raw(); - let p_ktrace = rxc.ktrace.to_raw(); - let p_stats = rxc.stats.to_raw(); - let p_fwlog_buf = rxc.fw_log.get_buf(); - core::mem::drop(rxc); - - mgr.as_mut() - .initdata_mut() - .runtime_pointers - .with_mut(|raw, _inner| { - raw.device_control = p_device_control; - raw.event = p_event; - raw.fw_log = p_fw_log; - raw.ktrace = p_ktrace; - raw.stats = p_stats; - raw.fwlog_buf = Some(p_fwlog_buf); - }); - } - - let mut p_pipes: KVec = KVec::new(); - - for ((v, f), c) in mgr - .pipes - .vtx - .iter() - .zip(&mgr.pipes.frag) - .zip(&mgr.pipes.comp) - { - p_pipes.push( - fw::initdata::raw::PipeChannels::ver { - vtx: v.lock().to_raw(), - frag: f.lock().to_raw(), - comp: c.lock().to_raw(), - }, - GFP_KERNEL, - )?; - } - - mgr.as_mut() - .initdata_mut() - .runtime_pointers - .with_mut(|raw, _inner| { - for (i, p) in p_pipes.into_iter().enumerate() { - raw.pipes[i].vtx = p.vtx; - raw.pipes[i].frag = p.frag; - raw.pipes[i].comp = p.comp; - } - }); - - for (i, map) in cfg.io_mappings.iter().enumerate() { - if let Some(map) = map.as_ref() { - Self::iomap(&mut mgr, cfg, i, map)?; - } - } - - #[ver(V >= V13_0B4)] - if let Some(base) = cfg.sram_base { - let size = cfg.sram_size.unwrap(); - let iova = mgr.as_mut().alloc_mmio_iova(size); - - let mapping = mgr - .uat - .kernel_vm() - .map_io(iova, base, size, mmu::PROT_FW_SHARED_RW)?; - - mgr.as_mut() - .initdata_mut() - .runtime_pointers - .hwdata_b - .with_mut(|raw, _| { - raw.sgx_sram_ptr = U64(mapping.iova()); - }); - - mgr.as_mut().io_mappings_mut().push(mapping, GFP_KERNEL)?; - } - - let mgr = Arc::from(mgr); - - let rtkit = rtkit::RtKit::::new(dev.as_ref(), None, 0, mgr.clone())?; - - *mgr.rtkit.lock() = Some(rtkit); - - { - let mut rxc = mgr.rx_channels.lock(); - rxc.event.set_manager(mgr.clone()); - } - - Ok(mgr) - } - - /// Return a mutable reference to the initdata member - fn initdata_mut( - self: Pin<&mut Self>, - ) -> &mut fw::types::GpuObject { - // SAFETY: initdata does not require structural pinning. - unsafe { &mut self.get_unchecked_mut().initdata } - } - - /// Return a mutable reference to the io_mappings member - fn io_mappings_mut(self: Pin<&mut Self>) -> &mut KVec { - // SAFETY: io_mappings does not require structural pinning. - unsafe { &mut self.get_unchecked_mut().io_mappings } - } - - /// Allocate an MMIO iova range - fn alloc_mmio_iova(self: Pin<&mut Self>, size: usize) -> u64 { - // SAFETY: next_mmio_iova does not require structural pinning. - let next_ref = unsafe { &mut self.get_unchecked_mut().next_mmio_iova }; - - let addr = *next_ref; - let next = addr + (size + mmu::UAT_PGSZ) as u64; - - assert!(next <= IOVA_KERN_MMIO_RANGE.end); - - *next_ref = next; - - addr - } - - /// Build the entire GPU InitData structure tree and return it as a boxed GpuObject. - fn make_initdata( - dev: &AsahiDevice, - cfg: &'static hw::HwConfig, - dyncfg: &hw::DynConfig, - alloc: &mut KernelAllocators, - ) -> Result>> { - let mut builder = initdata::InitDataBuilder::ver::new(dev, alloc, cfg, dyncfg); - builder.build() - } - - /// Create a fresh boxed Uat instance. - /// - /// Force disable inlining to avoid blowing up the stack. - #[inline(never)] - fn make_uat(dev: &AsahiDevice, cfg: &'static hw::HwConfig) -> Result> { - // G14X has a new thing in the Scene structure that unfortunately requires - // write access from user contexts. Hopefully it's not security-sensitive. - #[ver(G >= G14X)] - let map_kernel_to_user = true; - #[ver(G < G14X)] - let map_kernel_to_user = false; - - Ok(KBox::new( - mmu::Uat::new( - dev, - mmu::UatGeometry { - ias: cfg.uat_ias, - oas: cfg.uat_oas, - }, - map_kernel_to_user, - mmu::UatFirmware::Handoff, - )?, - GFP_KERNEL, - )?) - } - - /// Actually create the final GpuManager instance, as a UniqueArc. - /// - /// Force disable inlining to avoid blowing up the stack. - #[inline(never)] - fn make_mgr( - dev: &AsahiDevice, - cfg: &'static hw::HwConfig, - dyncfg: KBox, - uat: KBox, - mut alloc: KernelAllocators, - event_manager: Arc, - initdata: KBox>, - ) -> Result>> { - let mut pipes = PipeChannels::ver { - vtx: KVec::new(), - frag: KVec::new(), - comp: KVec::new(), - }; - - for _i in 0..=NUM_PIPES - 1 { - pipes.vtx.push( - KBox::pin_init( - new_mutex!(channel::PipeChannel::ver::new(dev, &mut alloc)?, "pipe_vtx",), - GFP_KERNEL, - )?, - GFP_KERNEL, - )?; - pipes.frag.push( - KBox::pin_init( - new_mutex!( - channel::PipeChannel::ver::new(dev, &mut alloc)?, - "pipe_frag", - ), - GFP_KERNEL, - )?, - GFP_KERNEL, - )?; - pipes.comp.push( - KBox::pin_init( - new_mutex!( - channel::PipeChannel::ver::new(dev, &mut alloc)?, - "pipe_comp", - ), - GFP_KERNEL, - )?, - GFP_KERNEL, - )?; - } - - let fwctl_channel = channel::FwCtlChannel::new(dev, &mut alloc)?; - - let buffer_mgr = buffer::BufferManager::ver::new()?; - let event_manager_clone = event_manager.clone(); - let buffer_mgr_clone = buffer_mgr.clone(); - let stats_snap = Arc::new(stats::StatsSnapshot::default(), GFP_KERNEL)?; - let stats_snap_clone = stats_snap.clone(); - let alloc_ref = &mut alloc; - let rx_channels = KBox::init( - try_init!(RxChannels::ver { - event: channel::EventChannel::ver::new( - dev, - alloc_ref, - event_manager_clone, - buffer_mgr_clone, - )?, - fw_log: channel::FwLogChannel::new(dev, alloc_ref)?, - ktrace: channel::KTraceChannel::new(dev, alloc_ref)?, - stats: channel::StatsChannel::ver::new( - dev, - alloc_ref, - stats_snap_clone, - u64::from(cfg.base_clock_hz), - )?, - }), - GFP_KERNEL, - )?; - - let alloc_ref = &mut alloc; - let tx_channels = KBox::init( - try_init!(TxChannels::ver { - device_control: channel::DeviceControlChannel::ver::new(dev, alloc_ref)?, - }), - GFP_KERNEL, - )?; - - let x = UniqueArc::pin_init( - try_pin_init!(GpuManager::ver { - dev: dev.into(), - cfg, - dyncfg: KBox::::into_inner(dyncfg), - initdata: KBox::>::into_inner(initdata), - uat: KBox::::into_inner(uat), - io_mappings: KVec::new(), - next_mmio_iova: IOVA_KERN_MMIO_RANGE.start, - rtkit <- new_mutex!(None, "rtkit"), - crashed: AtomicBool::new(false), - event_manager, - stats: stats_snap.clone(), - alloc <- new_mutex!(alloc, "alloc"), - fwctl_channel <- new_mutex!(fwctl_channel, "fwctl_channel"), - rx_channels <- new_mutex!(KBox::::into_inner(rx_channels), "rx_channels"), - tx_channels <- new_mutex!(KBox::::into_inner(tx_channels), "tx_channels"), - pipes, - buffer_mgr, - ids: Default::default(), - garbage_contexts <- new_mutex!(KVec::new(), "garbage_contexts"), - }), - GFP_KERNEL, - )?; - - Ok(x) - } - - fn load_hwdata_blob(dev: &AsahiDevice, name: &CStr, size_name: &CStr) -> Result> { - let of_node = dev.as_ref().of_node().ok_or(EINVAL)?; - let size: usize = dev - .as_ref() - .fwnode() - .ok_or(ENOENT)? - .property_read::(size_name) - .or(0) - .try_into()?; - let res = of_node.reserved_mem_region_to_resource_byname(name)?; - // SAFETY: No dma here, just loading init data. - let mem = unsafe { Mem::try_new(res, (MemFlag::WB).into())? }; - if size > mem.size() { - return Err(ENOENT); - } - // SAFETY: trusting the bootloader to fill it out correctly - let blob_sl = unsafe { slice::from_raw_parts(mem.ptr(), size) }; - let mut blob = KVVec::new(); - blob.extend_from_slice(blob_sl, GFP_KERNEL)?; - Ok(blob) - } - - /// Fetch and validate the GPU dynamic configuration from the device tree and hardware. - /// - /// Force disable inlining to avoid blowing up the stack. - #[inline(never)] - fn make_dyncfg( - dev: &AsahiDevice, - res: ®s::Resources, - cfg: &'static hw::HwConfig, - uat: &mmu::Uat, - ) -> Result> { - let gpu_id = res.get_gpu_id()?; - - dev_info!(dev.as_ref(), "GPU Information:\n"); - dev_info!( - dev.as_ref(), - " Type: {:?}{:?}\n", - gpu_id.gpu_gen, - gpu_id.gpu_variant - ); - dev_info!(dev.as_ref(), " Clusters: {}\n", gpu_id.num_clusters); - dev_info!( - dev.as_ref(), - " Cores: {} ({})\n", - gpu_id.num_cores, - gpu_id.num_cores * gpu_id.num_clusters - ); - dev_info!( - dev.as_ref(), - " Frags: {} ({})\n", - gpu_id.num_frags, - gpu_id.num_frags * gpu_id.num_clusters - ); - dev_info!( - dev.as_ref(), - " GPs: {} ({})\n", - gpu_id.num_gps, - gpu_id.num_gps * gpu_id.num_clusters - ); - dev_info!(dev.as_ref(), " Core masks: {:#x?}\n", gpu_id.core_masks); - dev_info!( - dev.as_ref(), - " Active cores: {}\n", - gpu_id.total_active_cores - ); - - dev_info!(dev.as_ref(), "Getting configuration from device tree...\n"); - let pwr_cfg = hw::PwrConfig::load(dev, cfg)?; - dev_info!(dev.as_ref(), "Dynamic configuration fetched\n"); - - if gpu_id.gpu_gen != cfg.gpu_gen || gpu_id.gpu_variant != cfg.gpu_variant { - dev_err!( - dev.as_ref(), - "GPU type mismatch (expected {:?}{:?}, found {:?}{:?})\n", - cfg.gpu_gen, - cfg.gpu_variant, - gpu_id.gpu_gen, - gpu_id.gpu_variant - ); - return Err(EIO); - } - if gpu_id.num_clusters > cfg.max_num_clusters { - dev_err!( - dev.as_ref(), - "Too many clusters ({} > {})\n", - gpu_id.num_clusters, - cfg.max_num_clusters - ); - return Err(EIO); - } - if gpu_id.num_cores > cfg.max_num_cores { - dev_err!( - dev.as_ref(), - "Too many cores ({} > {})\n", - gpu_id.num_cores, - cfg.max_num_cores - ); - return Err(EIO); - } - if gpu_id.num_frags > cfg.max_num_frags { - dev_err!( - dev.as_ref(), - "Too many frags ({} > {})\n", - gpu_id.num_frags, - cfg.max_num_frags - ); - return Err(EIO); - } - if gpu_id.num_gps > cfg.max_num_gps { - dev_err!( - dev.as_ref(), - "Too many GPs ({} > {})\n", - gpu_id.num_gps, - cfg.max_num_gps - ); - return Err(EIO); - } - - let fwnode = dev.as_ref().fwnode().ok_or(ENOENT)?; - - Ok(KBox::new( - hw::DynConfig { - pwr: pwr_cfg, - uat_ttb_base: uat.ttb_base(), - id: gpu_id, - firmware_version: fwnode - .property_read_array_vec(c_str!("apple,firmware-version"), 3)? - .or(kernel::kvec![0; 3]?), - - hw_data_a: Self::load_hwdata_blob( - dev, - c_str!("hw-cal-a"), - c_str!("debug,hw-cal-a-size"), - ) - .unwrap_or(KVVec::new()), - hw_data_b: Self::load_hwdata_blob( - dev, - c_str!("hw-cal-b"), - c_str!("debug,hw-cal-b-size"), - ) - .unwrap_or(KVVec::new()), - hw_globals: Self::load_hwdata_blob( - dev, - c_str!("globals"), - c_str!("debug,globals-size"), - ) - .unwrap_or(KVVec::new()), - }, - GFP_KERNEL, - )?) - } - - /// Create the global GPU event manager, and return an `Arc<>` to it. - fn make_event_manager(alloc: &mut KernelAllocators) -> Result> { - Ok(Arc::new(event::EventManager::new(alloc)?, GFP_KERNEL)?) - } - - /// Create a new MMIO mapping and add it to the mappings list in initdata at the specified - /// index. - fn iomap( - this: &mut Pin>, - cfg: &'static hw::HwConfig, - index: usize, - map: &hw::IOMapping, - ) -> Result { - let dies = if map.per_die { - cfg.num_dies as usize - } else { - 1 - }; - - let off = map.base & mmu::UAT_PGMSK; - let base = map.base - off; - let end = (map.base + map.size + mmu::UAT_PGMSK) & !mmu::UAT_PGMSK; - let map_size = end - base; - - // Array mappings must be aligned - assert!((off == 0 && map_size == map.size) || (map.count == 1 && !map.per_die)); - assert!(map.count > 0); - - let iova = this.as_mut().alloc_mmio_iova(map_size * map.count * dies); - let mut cur_iova = iova; - - for die in 0..dies { - for i in 0..map.count { - let phys_off = die * 0x20_0000_0000 + i * map.stride; - - let mapping = this.uat.kernel_vm().map_io( - cur_iova, - base + phys_off, - map_size, - if map.writable { - mmu::PROT_FW_MMIO_RW - } else { - mmu::PROT_FW_MMIO_RO - }, - )?; - - this.as_mut().io_mappings_mut().push(mapping, GFP_KERNEL)?; - cur_iova += map_size as u64; - } - } - - this.as_mut() - .initdata_mut() - .runtime_pointers - .hwdata_b - .with_mut(|raw, _| { - raw.io_mappings[index] = fw::initdata::raw::IOMapping { - phys_addr: U64(map.base as u64), - virt_addr: U64(iova + off as u64), - total_size: (map.size * map.count * dies) as u32, - element_size: map.size as u32, - readwrite: U64(map.writable as u64), - }; - }); - - Ok(()) - } - - /// Mark work associated with currently in-progress event slots as failed, after a fault or - /// timeout. - fn mark_pending_events(&self, culprit_slot: Option, error: workqueue::WorkError) { - dev_err!(self.dev.as_ref(), " Pending events:\n"); - - self.initdata.globals.with(|raw, _inner| { - for (index, i) in raw.pending_stamps.iter().enumerate() { - let info = i.info.load(Ordering::Relaxed); - let wait_value = i.wait_value.load(Ordering::Relaxed); - - if info & 1 != 0 { - #[ver(V >= V13_5)] - let slot = (info >> 4) & 0x7f; - #[ver(V < V13_5)] - let slot = (info >> 3) & 0x7f; - #[ver(V >= V13_5)] - let flags = info & 0xf; - #[ver(V < V13_5)] - let flags = info & 0x7; - dev_err!( - self.dev.as_ref(), - " [{}:{}] flags={} value={:#x}\n", - index, - slot, - flags, - wait_value - ); - let error = if culprit_slot.is_some() && culprit_slot != Some(slot) { - workqueue::WorkError::Killed - } else { - error - }; - self.event_manager.mark_error(slot, wait_value, error); - i.info.store(0, Ordering::Relaxed); - i.wait_value.store(0, Ordering::Relaxed); - } - } - }); - } - - /// Fetch the GPU MMU fault information from the hardware registers. - fn get_fault_info(&self) -> Option { - let res = &(*self.dev).resources; - - let info = res.get_fault_info(self.cfg); - if info.is_some() { - dev_err!( - self.dev.as_ref(), - " Fault info: {:#x?}\n", - info.as_ref().unwrap() - ); - } - info - } - - /// Resume the GPU firmware after it halts (due to a timeout, fault, or request). - fn recover(&self) { - self.initdata.fw_status.with(|raw, _inner| { - let halt_count = raw.flags.halt_count.load(Ordering::Relaxed); - let mut halted = raw.flags.halted.load(Ordering::Relaxed); - dev_err!(self.dev.as_ref(), " Halt count: {}\n", halt_count); - dev_err!(self.dev.as_ref(), " Halted: {}\n", halted); - - if halted == 0 { - let start = Instant::::now(); - while start.elapsed() < HALT_ENTER_TIMEOUT { - halted = raw.flags.halted.load(Ordering::Relaxed); - if halted != 0 { - break; - } - mem::sync(); - } - halted = raw.flags.halted.load(Ordering::Relaxed); - } - - if debug_enabled(DebugFlags::NoGpuRecovery) { - dev_crit!( - self.dev.as_ref(), - " GPU recovery is disabled, wedging forever!\n" - ); - } else if halted != 0 { - dev_err!(self.dev.as_ref(), " Attempting recovery...\n"); - raw.flags.halted.store(0, Ordering::SeqCst); - raw.flags.resume.store(1, Ordering::SeqCst); - } else { - dev_err!(self.dev.as_ref(), " Cannot recover.\n"); - } - }); - } - - /// Return the packed GPU enabled core masks. - // Only used for some versions - #[allow(dead_code)] - pub(crate) fn core_masks_packed(&self) -> &[u32] { - self.dyncfg.id.core_masks_packed.as_slice() - } - - /// Kick a submission pipe for a submitted job to tell the firmware to start processing it. - pub(crate) fn run_job(&self, job: workqueue::JobSubmission::ver<'_>) -> Result { - mod_dev_dbg!(self.dev, "GPU: run_job\n"); - - let pipe_type = job.pipe_type(); - mod_dev_dbg!(self.dev, "GPU: run_job: pipe_type={:?}\n", pipe_type); - - let pipes = match pipe_type { - PipeType::Vertex => &self.pipes.vtx, - PipeType::Fragment => &self.pipes.frag, - PipeType::Compute => &self.pipes.comp, - }; - - let index: usize = job.priority() as usize; - let mut pipe = pipes.get(index).ok_or(EIO)?.lock(); - - mod_dev_dbg!(self.dev, "GPU: run_job: run()\n"); - job.run(&mut pipe); - mod_dev_dbg!(self.dev, "GPU: run_job: ring doorbell\n"); - - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - rtk.send_message( - EP_DOORBELL, - MSG_TX_DOORBELL | pipe_type as u64 | ((index as u64) << 2), - )?; - mod_dev_dbg!(self.dev, "GPU: run_job: done\n"); - - Ok(()) - } - - pub(crate) fn start_op(self: &Arc) -> Result { - if self.is_crashed() { - return Err(ENODEV); - } - - let val = self - .initdata - .globals - .with(|raw, _inner| raw.pending_submissions.fetch_add(1, Ordering::Acquire)); - - mod_dev_dbg!(self.dev, "OP start (pending: {})\n", val + 1); - let guard = OpGuard(self.clone()); - self.kick_firmware()?; - Ok(guard) - } - - fn invalidate_context( - &self, - context: &fw::types::GpuObject, - ) -> Result { - mod_dev_dbg!( - self.dev, - "Invalidating GPU context @ {:?}\n", - context.weak_pointer() - ); - - if self.is_crashed() { - return Err(ENODEV); - } - - let mut guard = self.alloc.lock(); - let (garbage_count, _) = guard.private.garbage(); - let (garbage_count_gpuro, _) = guard.gpu_ro.garbage(); - - let dc = context.with( - |raw, _inner| fw::channels::DeviceControlMsg::ver::DestroyContext { - unk_4: 0, - ctx_23: raw.unk_23, - #[ver(V < V13_3)] - __pad0: Default::default(), - unk_c: U32(0), - unk_10: U32(0), - ctx_0: raw.unk_0, - ctx_1: raw.unk_1, - ctx_4: raw.unk_4, - #[ver(V < V13_3)] - __pad1: Default::default(), - #[ver(V < V13_3)] - unk_18: 0, - gpu_context: Some(context.weak_pointer()), - __pad2: Default::default(), - }, - ); - - mod_dev_dbg!(self.dev, "Context invalidation command: {:?}\n", &dc); - - let mut txch = self.tx_channels.lock(); - - let token = txch.device_control.send(&dc); - - { - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - rtk.send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_DEVCTRL)?; - } - - txch.device_control.wait_for(token)?; - - mod_dev_dbg!( - self.dev, - "GPU context invalidated: {:?}\n", - context.weak_pointer() - ); - - // The invalidation does a cache flush, so it is okay to collect garbage - guard.private.collect_garbage(garbage_count); - guard.gpu_ro.collect_garbage(garbage_count_gpuro); - - Ok(()) - } - - #[cfg(CONFIG_DEV_COREDUMP)] - fn generate_crashdump(&self, crashlog: Option<&[u8]>) -> Result { - // Lock the allocators, to block kernel/FW memory mutations (mostly) - let kalloc = self.alloc(); - let pages = self.uat.dump_kernel_pages()?; - core::mem::drop(kalloc); - - let mut crashdump = crate::crashdump::CrashDumpBuilder::new(pages)?; - let initdata_addr = self.initdata.gpu_va().get(); - crashdump.add_agx_info(self.cfg, &self.dyncfg, initdata_addr)?; - if let Some(crashlog) = crashlog { - crashdump.add_crashlog(crashlog)?; - } - let crashdump = KBox::new(crashdump.finalize()?, GFP_KERNEL)?; - - devcoredump::dev_coredump( - self.dev.as_ref(), - &crate::THIS_MODULE, - crashdump, - GFP_KERNEL, - msecs_to_jiffies(60 * 60 * 1000), - ); - - Ok(()) - } -} - -#[versions(AGX_NEO)] -impl Gpu for GpuManager::ver { - fn stats_snapshot(&self) -> Option> { - Some(self.stats.clone()) - } - - fn as_any(&self) -> &dyn Any { - self - } - - fn arc_as_any(self: Arc) -> Arc { - self as Arc - } - - fn manager(&self) -> Option<&dyn GpuManager> { - Some(self) - } - - fn init(&self) -> Result { - self.tx_channels.lock().device_control.send( - &fw::channels::DeviceControlMsg::ver::Initialize(Default::default()), - ); - - let initdata = self.initdata.gpu_va().get(); - let mut guard = self.rtkit.lock(); - let mut rtk = guard.as_mut().as_pin_mut().unwrap(); - - rtk.as_mut().boot()?; - rtk.as_mut().start_endpoint(EP_FIRMWARE)?; - rtk.as_mut().start_endpoint(EP_DOORBELL)?; - rtk.as_mut() - .send_message(EP_FIRMWARE, MSG_INIT | (initdata & INIT_DATA_MASK))?; - rtk.as_mut() - .send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_DEVCTRL)?; - core::mem::drop(guard); - - self.kick_firmware()?; - Ok(()) - } - - fn ids(&self) -> &SequenceIDs { - &self.ids - } - - fn is_crashed(&self) -> bool { - self.crashed.load(Ordering::Relaxed) - } - - fn get_params(&self, params: &mut uapi::drm_asahi_neo_params_global) -> Result { - params.gpu_generation = self.dyncfg.id.gpu_gen as u32; - params.gpu_variant = self.dyncfg.id.gpu_variant as u32; - params.gpu_revision = self.dyncfg.id.gpu_rev as u32; - params.chip_id = self.cfg.chip_id; - params.num_dies = self.cfg.num_dies; - params.num_clusters_total = self.dyncfg.id.num_clusters; - params.num_cores_per_cluster = self.dyncfg.id.num_cores; - params.max_frequency_khz = self.dyncfg.pwr.max_frequency_khz(); - // G13 and G14 share the second shader ISA generation and predate the numbered command - // stream generations. - params.usc_generation = 2; - params.gpu_hal_generation = uapi::drm_asahi_neo_gpu_hal_generation_DRM_ASAHI_NEO_GPU_HAL_LEGACY; - - for (i, mask) in self.dyncfg.id.core_masks.iter().enumerate() { - *(params.core_masks.get_mut(i).ok_or(EIO)?) = (*mask).into(); - } - - Ok(()) - } - - fn base_clock_hz(&self) -> u32 { - self.cfg.base_clock_hz - } - - fn uat_geometry(&self) -> mmu::UatGeometry { - self.uat.geometry() - } - - fn new_vm(&self, kernel_range: Range) -> Result { - self.uat.new_vm(self.ids.vm.next(), kernel_range) - } - - fn new_queue( - &self, - vm: mmu::Vm, - ualloc: Arc>, - ualloc_priv: Arc>, - priority: u32, - usc_exec_base: u64, - ) -> Result> { - let mut kalloc = self.alloc(); - let id = self.ids.queue.next(); - Ok(KBox::new( - queue::Queue::ver::new( - &self.dev, - self, - vm, - &mut kalloc, - ualloc, - ualloc_priv, - self.event_manager.clone(), - self.stats.clone(), - &self.buffer_mgr, - id, - priority, - usc_exec_base, - )?, - GFP_KERNEL, - )?) - } - - fn map_timestamp_buffer( - &self, - mut bo: gem::ObjectRef, - range: Range, - ) -> Result { - bo.map_range_into_range( - self.uat.kernel_vm(), - range, - IOVA_KERN_TIMESTAMP_RANGE, - mmu::UAT_PGSZ as u64, - mmu::PROT_FW_SHARED_RW, - false, - ) - } -} - -#[versions(AGX_NEO)] -impl GpuManager for GpuManager::ver { - fn update_globals(&self) { - let mut timeout: u32 = 2; - if debug_enabled(DebugFlags::WaitForPowerOff) { - timeout = 0; - } else if debug_enabled(DebugFlags::KeepGpuPowered) { - timeout = 5000; - } - - self.initdata.globals.with(|raw, _inner| { - raw.idle_off_delay_ms.store(timeout, Ordering::Relaxed); - }); - } - - fn alloc(&self) -> Guard<'_, KernelAllocators, MutexBackend> { - /* Clean up idle contexts */ - let mut garbage_ctx = KVec::new(); - core::mem::swap(&mut *self.garbage_contexts.lock(), &mut garbage_ctx); - - for ctx in garbage_ctx { - if self.invalidate_context(&ctx).is_err() { - dev_err!( - self.dev.as_ref(), - "GpuContext: Failed to invalidate GPU context!\n" - ); - if debug_enabled(DebugFlags::OopsOnGpuCrash) { - panic!("GPU firmware timed out"); - } - } - } - - let mut guard = self.alloc.lock(); - let (garbage_count, garbage_bytes) = guard.private.garbage(); - let (ro_garbage_count, ro_garbage_bytes) = guard.gpu_ro.garbage(); - - if garbage_bytes > MAX_FW_ALLOC_GARBAGE_BYTES - || ro_garbage_bytes > MAX_FW_ALLOC_GARBAGE_BYTES - || garbage_count > MAX_FW_ALLOC_GARBAGE_OBJECTS - || ro_garbage_count > MAX_FW_ALLOC_GARBAGE_OBJECTS - { - mod_dev_dbg!( - self.dev, - "Collecting kalloc garbage (private: {} objects, {} bytes, gpuro: {} objects, {} bytes)\n", - garbage_count, - garbage_bytes, - ro_garbage_count, - ro_garbage_bytes - ); - if self.flush_fw_cache().is_err() { - dev_err!(self.dev.as_ref(), "Failed to flush FW cache\n"); - } else { - guard.private.collect_garbage(garbage_count); - guard.gpu_ro.collect_garbage(ro_garbage_count); - } - } - - guard - } - - fn bind_vm(&self, vm: &mmu::Vm) -> Result { - self.uat.bind(vm) - } - - fn kick_firmware(&self) -> Result { - if self.is_crashed() { - return Err(ENODEV); - } - - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - rtk.send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_KICKFW)?; - - Ok(()) - } - - fn flush_fw_cache(&self) -> Result { - mod_dev_dbg!(self.dev, "Flushing coprocessor data cache\n"); - - if self.is_crashed() { - return Err(ENODEV); - } - - // ctx_0 == 0xff or ctx_1 == 0xff cause no effect on context, - // but this command does a full cache flush too, so abuse it - // for that. - - let dc = fw::channels::DeviceControlMsg::ver::DestroyContext { - unk_4: 0, - - ctx_23: 0, - #[ver(V < V13_3)] - __pad0: Default::default(), - unk_c: U32(0), - unk_10: U32(0), - ctx_0: 0xff, - ctx_1: 0xff, - ctx_4: 0, - #[ver(V < V13_3)] - __pad1: Default::default(), - #[ver(V < V13_3)] - unk_18: 0, - gpu_context: None, - __pad2: Default::default(), - }; - - let mut txch = self.tx_channels.lock(); - - let token = txch.device_control.send(&dc); - { - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - rtk.send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_DEVCTRL)?; - } - - txch.device_control.wait_for(token)?; - Ok(()) - } - - fn handle_timeout(&self, counter: u32, event_slot: i32, unk: u32) { - dev_err!(self.dev.as_ref(), " (\\________/) \n"); - dev_err!(self.dev.as_ref(), " | | \n"); - dev_err!(self.dev.as_ref(), "'.| \\ , / |.'\n"); - dev_err!(self.dev.as_ref(), "--| / (( \\ |--\n"); - dev_err!(self.dev.as_ref(), ".'| _-_- |'.\n"); - dev_err!(self.dev.as_ref(), " |________| \n"); - dev_err!(self.dev.as_ref(), "** GPU timeout nya~!!!!! **\n"); - dev_err!(self.dev.as_ref(), " Event slot: {}\n", event_slot); - dev_err!(self.dev.as_ref(), " Timeout count: {}\n", counter); - dev_err!(self.dev.as_ref(), " Unk: {}\n", unk); - - // If we have fault info, consider it a fault. - let error = match self.get_fault_info() { - Some(info) => workqueue::WorkError::Fault(info), - None => workqueue::WorkError::Timeout, - }; - self.mark_pending_events(event_slot.try_into().ok(), error); - self.recover(); - } - - fn handle_fault(&self) { - dev_err!(self.dev.as_ref(), " (\\________/) \n"); - dev_err!(self.dev.as_ref(), " | | \n"); - dev_err!(self.dev.as_ref(), "'.| \\ , / |.'\n"); - dev_err!(self.dev.as_ref(), "--| / (( \\ |--\n"); - dev_err!(self.dev.as_ref(), ".'| _-_- |'.\n"); - dev_err!(self.dev.as_ref(), " |________| \n"); - dev_err!(self.dev.as_ref(), "GPU fault nya~!!!!!\n"); - let error = match self.get_fault_info() { - Some(info) => workqueue::WorkError::Fault(info), - None => workqueue::WorkError::Unknown, - }; - self.mark_pending_events(None, error); - self.recover(); - } - - fn handle_channel_error( - &self, - error_type: ChannelErrorType, - pipe_type: u32, - event_slot: u32, - event_value: u32, - ) { - dev_err!(self.dev.as_ref(), " (\\________/) \n"); - dev_err!(self.dev.as_ref(), " | | \n"); - dev_err!(self.dev.as_ref(), "'.| \\ , / |.'\n"); - dev_err!(self.dev.as_ref(), "--| / (( \\ |--\n"); - dev_err!(self.dev.as_ref(), ".'| _-_- |'.\n"); - dev_err!(self.dev.as_ref(), " |________| \n"); - dev_err!(self.dev.as_ref(), "GPU channel error nya~!!!!!\n"); - dev_err!(self.dev.as_ref(), " Error type: {:?}\n", error_type); - dev_err!(self.dev.as_ref(), " Pipe type: {}\n", pipe_type); - dev_err!(self.dev.as_ref(), " Event slot: {}\n", event_slot); - dev_err!(self.dev.as_ref(), " Event value: {:#x?}\n", event_value); - - self.event_manager.mark_error( - event_slot, - event_value, - workqueue::WorkError::ChannelError(error_type), - ); - - let wq = match self.event_manager.get_owner(event_slot) { - Some(wq) => wq, - None => { - dev_err!( - self.dev.as_ref(), - "Workqueue not found for this event slot!\n" - ); - return; - } - }; - - let wq = match wq.as_any().downcast_ref::() { - Some(wq) => wq, - None => { - dev_crit!(self.dev.as_ref(), "GpuManager mismatched with WorkQueue!\n"); - return; - } - }; - - if debug_enabled(DebugFlags::VerboseFaults) { - wq.dump_info(); - } - - let dc = fw::channels::DeviceControlMsg::ver::RecoverChannel { - pipe_type, - work_queue: wq.info_pointer(), - event_value, - __pad: Default::default(), - }; - - mod_dev_dbg!(self.dev, "Recover Channel command: {:?}\n", &dc); - let mut txch = self.tx_channels.lock(); - - let token = txch.device_control.send(&dc); - { - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - if rtk - .send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_DEVCTRL) - .is_err() - { - dev_err!( - self.dev.as_ref(), - "Failed to send Recover Channel command\n" - ); - } - } - - if txch.device_control.wait_for(token).is_err() { - dev_err!( - self.dev.as_ref(), - "Timed out waiting for Recover Channel command\n" - ); - } - - if debug_enabled(DebugFlags::VerboseFaults) { - wq.dump_info(); - } - } - - fn ack_grow(&self, buffer_slot: u32, vm_slot: u32, counter: u32) { - let halt_count = self - .initdata - .fw_status - .with(|raw, _inner| raw.flags.halt_count.load(Ordering::Relaxed)); - - let dc = fw::channels::DeviceControlMsg::ver::GrowTVBAck { - unk_4: 1, - buffer_slot, - vm_slot, - counter, - subpipe: 0, // TODO - halt_count: U64(halt_count), - __pad: Default::default(), - }; - - mod_dev_dbg!(self.dev, "TVB Grow Ack command: {:?}\n", &dc); - - let mut txch = self.tx_channels.lock(); - - txch.device_control.send(&dc); - { - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - if rtk - .send_message(EP_DOORBELL, MSG_TX_DOORBELL | DOORBELL_DEVCTRL) - .is_err() - { - dev_err!(self.dev.as_ref(), "Failed to send TVB Grow Ack command\n"); - } - } - } - - fn fwctl(&self, msg: fw::channels::FwCtlMsg) -> Result { - if self.is_crashed() { - return Err(ENODEV); - } - - let mut fwctl = self.fwctl_channel.lock(); - let token = fwctl.send(&msg); - { - let mut guard = self.rtkit.lock(); - let rtk = guard.as_mut().as_pin_mut().unwrap(); - rtk.send_message(EP_DOORBELL, MSG_FWCTL)?; - } - fwctl.wait_for(token)?; - Ok(()) - } - - fn get_cfg(&self) -> &'static hw::HwConfig { - self.cfg - } - - fn get_dyncfg(&self) -> &hw::DynConfig { - &self.dyncfg - } - - fn free_context(&self, ctx: KBox>) { - let mut garbage = self.garbage_contexts.lock(); - - if garbage.push(ctx, GFP_KERNEL).is_err() { - dev_err!( - self.dev.as_ref(), - "Failed to reserve space for freed context, deadlock possible.\n" - ); - } - } -} - -#[versions(AGX_NEO)] -impl GpuManagerPriv for GpuManager::ver { - fn end_op(&self) { - let val = self - .initdata - .globals - .with(|raw, _inner| raw.pending_submissions.fetch_sub(1, Ordering::Release)); - - mod_dev_dbg!(self.dev, "OP end (pending: {})\n", val - 1); - } -} diff --git a/drivers/gpu/drm/asahi-neo/hw/mod.rs b/drivers/gpu/drm/asahi-neo/hw/mod.rs deleted file mode 100644 index 8ed9c8625243a8..00000000000000 --- a/drivers/gpu/drm/asahi-neo/hw/mod.rs +++ /dev/null @@ -1,660 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Per-SoC hardware configuration structures -//! -//! This module contains the definitions used to store per-GPU and per-SoC configuration data. - -use crate::driver::AsahiDevice; -use crate::fw::types::*; -use kernel::c_str; -use kernel::prelude::*; - -const MAX_POWERZONES: usize = 5; - -pub(crate) mod t600x; -pub(crate) mod t602x; -pub(crate) mod t8103; -pub(crate) mod t8112; -pub(crate) mod t8140; - -/// GPU generation enumeration. Note: Part of the UABI. -#[derive(Debug, PartialEq, Copy, Clone)] -#[repr(u32)] -pub(crate) enum GpuGen { - G13 = 13, - G14 = 14, - G17 = 17, -} - -/// GPU variant enumeration. Note: Part of the UABI. -#[derive(Debug, PartialEq, Copy, Clone)] -#[repr(u32)] -pub(crate) enum GpuVariant { - P = 'P' as u32, - G = 'G' as u32, - S = 'S' as u32, - C = 'C' as u32, - D = 'D' as u32, -} - -/// GPU revision enumeration. Note: Part of the UABI. -#[derive(Debug, PartialEq, Copy, Clone)] -#[repr(u32)] -pub(crate) enum GpuRevision { - A0 = 0x00, - A1 = 0x01, - B0 = 0x10, - B1 = 0x11, - C0 = 0x20, - C1 = 0x21, -} - -/// GPU core type enumeration. Note: Part of the firmware ABI. -#[derive(Debug, Copy, Clone)] -#[repr(u32)] -pub(crate) enum GpuCore { - // Unknown = 0, - // G5P = 1, - // G5G = 2, - // G9P = 3, - // G9G = 4, - // G10P = 5, - // G11P = 6, - // G11M = 7, - // G11G = 8, - // G12P = 9, - // G13P = 10, - G13G = 11, - G13S = 12, - G13C = 13, - // G14P = 14, - G14G = 15, - G14S = 16, - G14C = 17, - G14D = 18, // Split out, unlike G13D -} - -/// GPU revision ID. Note: Part of the firmware ABI. -#[derive(Debug, PartialEq, Copy, Clone)] -#[repr(u32)] -pub(crate) enum GpuRevisionID { - // Unknown = 0, - A0 = 1, - A1 = 2, - B0 = 3, - B1 = 4, - C0 = 5, - C1 = 6, -} - -/// A single performance state of the GPU. -#[derive(Debug)] -pub(crate) struct PState { - /// Voltage in millivolts, per GPU cluster. - pub(crate) volt_mv: KVec, - /// Frequency in hertz. - pub(crate) freq_hz: u32, - /// Maximum power consumption of the GPU at this pstate, in milliwatts. - pub(crate) pwr_mw: u32, -} - -impl PState { - pub(crate) fn max_volt_mv(&self) -> u32 { - *self.volt_mv.iter().max().expect("No voltages") - } -} - -/// A power zone definition (we have no idea what this is but Apple puts them in the DT). -#[allow(missing_docs)] -#[derive(Debug, Copy, Clone)] -pub(crate) struct PowerZone { - pub(crate) target: u32, - pub(crate) target_offset: u32, - pub(crate) filter_tc: u32, -} - -/// An MMIO mapping used by the firmware. -#[derive(Debug, Copy, Clone)] -pub(crate) struct IOMapping { - /// Base physical address of the mapping. - pub(crate) base: usize, - /// Whether this mapping should be replicated to all dies - pub(crate) per_die: bool, - /// Number of mappings. - pub(crate) count: usize, - /// Size of one mapping. - pub(crate) size: usize, - /// Stride between mappings. - pub(crate) stride: usize, - /// Whether the mapping should be writable. - pub(crate) writable: bool, -} - -impl IOMapping { - /// Convenience constructor for a new IOMapping. - pub(crate) const fn new( - base: usize, - per_die: bool, - count: usize, - size: usize, - stride: usize, - writable: bool, - ) -> IOMapping { - IOMapping { - base, - per_die, - count, - size, - stride, - writable, - } - } -} - -/// Unknown HwConfigA fields that vary from SoC to SoC. -#[allow(missing_docs)] -#[derive(Debug, Copy, Clone)] -pub(crate) struct HwConfigA { - pub(crate) unk_87c: i32, - pub(crate) unk_8cc: u32, - pub(crate) unk_e24: u32, -} - -/// Unknown HwConfigB fields that vary from SoC to SoC. -#[allow(missing_docs)] -#[derive(Debug, Copy, Clone)] -pub(crate) struct HwConfigB { - pub(crate) unk_454: u32, - pub(crate) unk_4e0: u64, - pub(crate) unk_534: u32, - pub(crate) unk_ab8: u32, - pub(crate) unk_abc: u32, - pub(crate) unk_b30: u32, -} - -/// Render command configs that vary from SoC to SoC. -#[derive(Debug, Copy, Clone)] -pub(crate) struct HwRenderConfig { - /// Vertex/tiling-related configuration register (lsb: disable clustering) - pub(crate) tiling_control: u32, -} - -#[derive(Debug)] -pub(crate) struct HwConfigShared2Curves { - pub(crate) t1_coef: u32, - pub(crate) t2: &'static [i16], - pub(crate) t3_coefs: &'static [u32], - pub(crate) t3_scales: &'static [u32], -} - -/// Static hardware clustering configuration for multi-cluster SoCs. -#[derive(Debug)] -pub(crate) struct HwClusteringConfig { - pub(crate) meta1_blocksize: usize, - pub(crate) meta2_size: usize, - pub(crate) meta3_size: usize, - pub(crate) meta4_size: usize, - pub(crate) max_splits: usize, -} - -/// Static hardware configuration for a given SoC model. -#[derive(Debug)] -pub(crate) struct HwConfig { - /// Chip ID in hex format (e.g. 0x8103 for t8103). - pub(crate) chip_id: u32, - /// GPU generation. - pub(crate) gpu_gen: GpuGen, - /// GPU variant type. - pub(crate) gpu_variant: GpuVariant, - /// GPU core type ID (as known by the firmware). - pub(crate) gpu_core: GpuCore, - - /// Base clock used used for timekeeping. - pub(crate) base_clock_hz: u32, - /// Input address bits translated by each UAT page table root on this SoC. - pub(crate) uat_ias: u32, - /// Output address space for the UAT on this SoC. - pub(crate) uat_oas: u32, - /// Number of dies on this SoC. - pub(crate) num_dies: u32, - /// Maximum number of clusters on this SoC. - pub(crate) max_num_clusters: u32, - /// Maximum number of cores per cluster for this GPU. - pub(crate) max_num_cores: u32, - /// Maximum number of frags per cluster for this GPU. - pub(crate) max_num_frags: u32, - /// Maximum number of GPs per cluster for this GPU. - pub(crate) max_num_gps: u32, - - /// Required size of the first preemption buffer. - pub(crate) preempt1_size: usize, - /// Required size of the second preemption buffer. - pub(crate) preempt2_size: usize, - /// Required size of the third preemption buffer. - pub(crate) preempt3_size: usize, - - /// Required size of the compute preemption buffer. - pub(crate) compute_preempt1_size: usize, - - pub(crate) clustering: Option, - - /// Rendering-relevant configuration. - pub(crate) render: HwRenderConfig, - - /// Misc HWDataA field values. - pub(crate) da: HwConfigA, - /// Misc HWDataB field values. - pub(crate) db: HwConfigB, - /// HwDataShared1.table. - pub(crate) shared1_tab: &'static [i32], - /// HwDataShared1.unk_a4. - pub(crate) shared1_a4: u32, - /// HwDataShared2.table. - pub(crate) shared2_tab: &'static [i32], - /// HwDataShared2.unk_508. - pub(crate) shared2_unk_508: u32, - /// HwDataShared2.unk_508. - pub(crate) shared2_curves: Option, - - /// HwDataShared3.unk_8. - pub(crate) shared3_unk: u32, - /// HwDataShared3.table. - pub(crate) shared3_tab: &'static [u32], - - /// Globals.idle_off_standby_timer. - pub(crate) idle_off_standby_timer_default: u32, - /// Globals.unk_hws2_4. - pub(crate) unk_hws2_4: Option<[F32; 8]>, - /// Globals.unk_hws2_24. - pub(crate) unk_hws2_24: u32, - /// Globals.unk_54 - pub(crate) global_unk_54: u16, - - /// Constant related to SRAM voltages. - pub(crate) sram_k: F32, - /// Unknown per-cluster coefficients 1. - pub(crate) unk_coef_a: &'static [&'static [F32]], - /// Unknown per-cluster coefficients 2. - pub(crate) unk_coef_b: &'static [&'static [F32]], - /// Unknown table in Global struct. - pub(crate) global_tab: Option<&'static [u8]>, - /// Whether this GPU has CS/AFR performance states - pub(crate) has_csafr: bool, - - /// Temperature sensor list (8 bits per sensor). - pub(crate) fast_sensor_mask: [u64; 2], - /// Temperature sensor list (alternate). - pub(crate) fast_sensor_mask_alt: [u64; 2], - /// Temperature sensor present bitmask. - pub(crate) fast_die0_sensor_present: u32, - /// Required MMIO mappings for this GPU/firmware. - pub(crate) io_mappings: &'static [Option], - /// SRAM base - pub(crate) sram_base: Option, - /// SRAM size - pub(crate) sram_size: Option, -} - -/// Dynamic (fetched from hardware/DT) configuration. -#[derive(Debug)] -pub(crate) struct DynConfig { - /// Base physical address of the UAT TTB (from DT reserved memory region). - pub(crate) uat_ttb_base: u64, - /// GPU ID configuration read from hardware. - pub(crate) id: GpuIdConfig, - /// Power calibration configuration for this specific chip/device. - pub(crate) pwr: PwrConfig, - /// Firmware version. - #[allow(dead_code)] - pub(crate) firmware_version: KVec, - - pub(crate) hw_data_a: KVVec, - pub(crate) hw_data_b: KVVec, - pub(crate) hw_globals: KVVec, -} - -/// Specific GPU ID configuration fetched from SGX MMIO registers. -#[derive(Debug)] -pub(crate) struct GpuIdConfig { - /// GPU generation (should match static config). - pub(crate) gpu_gen: GpuGen, - /// GPU variant type (should match static config). - pub(crate) gpu_variant: GpuVariant, - /// GPU silicon revision. - pub(crate) gpu_rev: GpuRevision, - /// GPU silicon revision ID (firmware enum). - pub(crate) gpu_rev_id: GpuRevisionID, - /// Total number of GPU clusters. - pub(crate) num_clusters: u32, - /// Maximum number of GPU cores per cluster. - pub(crate) num_cores: u32, - /// Number of frags per cluster. - pub(crate) num_frags: u32, - /// Number of GPs per cluster. - pub(crate) num_gps: u32, - /// Total number of active cores for the whole GPU. - pub(crate) total_active_cores: u32, - /// Mask of active cores per cluster. - pub(crate) core_masks: KVec, - /// Packed mask of all active cores. - pub(crate) core_masks_packed: KVec, -} - -/// Configurable CS/AFR GPU power settings from the device tree. -#[derive(Debug)] -pub(crate) struct CsAfrPwrConfig { - /// GPU CS performance state list. - pub(crate) perf_states_cs: KVec, - /// GPU AFR performance state list. - pub(crate) perf_states_afr: KVec, - - /// CS leakage coefficient per die. - pub(crate) leak_coef_cs: KVec, - /// AFR leakage coefficient per die. - pub(crate) leak_coef_afr: KVec, - - /// Minimum voltage for the CS/AFR SRAM power domain in microvolts. - pub(crate) min_sram_microvolt: u32, -} - -/// Configurable GPU power settings from the device tree. -#[derive(Debug)] -pub(crate) struct PwrConfig { - /// GPU performance state list. - pub(crate) perf_states: KVec, - /// GPU power zone list. - pub(crate) power_zones: KVec, - - /// Core leakage coefficient per cluster. - pub(crate) core_leak_coef: KVec, - /// SRAM leakage coefficient per cluster. - pub(crate) sram_leak_coef: KVec, - - pub(crate) csafr: Option, - - /// Maximum total power of the GPU in milliwatts. - pub(crate) max_power_mw: u32, - /// Maximum frequency of the GPU in megahertz. - pub(crate) max_freq_mhz: u32, - - /// Minimum performance state to start at. - pub(crate) perf_base_pstate: u32, - /// Maximum enabled performance state. - pub(crate) perf_max_pstate: u32, - - /// Minimum voltage for the SRAM power domain in microvolts. - pub(crate) min_sram_microvolt: u32, - - // Most of these fields are just named after Apple ADT property names and we don't fully - // understand them. They configure various power-related PID loops and filters. - /// Average power filter time constant in milliseconds. - pub(crate) avg_power_filter_tc_ms: u32, - /// Average power filter PID integral gain? - pub(crate) avg_power_ki_only: F32, - /// Average power filter PID proportional gain? - pub(crate) avg_power_kp: F32, - pub(crate) avg_power_min_duty_cycle: u32, - /// Average power target filter time constant in periods. - pub(crate) avg_power_target_filter_tc: u32, - /// "Fast die0" (temperature?) PID integral gain. - pub(crate) fast_die0_integral_gain: F32, - /// "Fast die0" (temperature?) PID proportional gain. - pub(crate) fast_die0_proportional_gain: F32, - pub(crate) fast_die0_prop_tgt_delta: u32, - pub(crate) fast_die0_release_temp: u32, - /// Delay from the fender (?) becoming idle to powerdown - pub(crate) fender_idle_off_delay_ms: u32, - /// Timeout from firmware early wake to sleep if no work was submitted (?) - pub(crate) fw_early_wake_timeout_ms: u32, - /// Delay from the GPU becoming idle to powerdown - pub(crate) idle_off_delay_ms: u32, - /// Related to the above? - pub(crate) idle_off_standby_timer: u32, - /// Percent? - pub(crate) perf_boost_ce_step: u32, - /// Minimum utilization before performance state is increased in %. - pub(crate) perf_boost_min_util: u32, - pub(crate) perf_filter_drop_threshold: u32, - /// Performance PID filter time constant? (periods?) - pub(crate) perf_filter_time_constant: u32, - /// Performance PID filter time constant 2? (periods?) - pub(crate) perf_filter_time_constant2: u32, - /// Performance PID integral gain. - pub(crate) perf_integral_gain: F32, - /// Performance PID integral gain 2 (?). - pub(crate) perf_integral_gain2: F32, - pub(crate) perf_integral_min_clamp: u32, - /// Performance PID proportional gain. - pub(crate) perf_proportional_gain: F32, - /// Performance PID proportional gain 2 (?). - pub(crate) perf_proportional_gain2: F32, - pub(crate) perf_reset_iters: u32, - /// Target GPU utilization for the performance controller in %. - pub(crate) perf_tgt_utilization: u32, - /// Power sampling period in milliseconds. - pub(crate) power_sample_period: u32, - /// PPM (?) filter time constant in milliseconds. - pub(crate) ppm_filter_time_constant_ms: u32, - /// PPM (?) filter PID integral gain. - pub(crate) ppm_ki: F32, - /// PPM (?) filter PID proportional gain. - pub(crate) ppm_kp: F32, - /// Power consumption filter time constant (periods?) - pub(crate) pwr_filter_time_constant: u32, - /// Power consumption filter PID integral gain. - pub(crate) pwr_integral_gain: F32, - pub(crate) pwr_integral_min_clamp: u32, - pub(crate) pwr_min_duty_cycle: u32, - pub(crate) pwr_proportional_gain: F32, - /// Power sample period in base clocks, used when not an integer number of ms - pub(crate) pwr_sample_period_aic_clks: u32, - - pub(crate) se_engagement_criteria: i32, - pub(crate) se_filter_time_constant: u32, - pub(crate) se_filter_time_constant_1: u32, - pub(crate) se_inactive_threshold: u32, - pub(crate) se_ki: F32, - pub(crate) se_ki_1: F32, - pub(crate) se_kp: F32, - pub(crate) se_kp_1: F32, - pub(crate) se_reset_criteria: u32, -} - -impl PwrConfig { - fn load_opp( - dev: &AsahiDevice, - name: &CStr, - cfg: &HwConfig, - is_main: bool, - ) -> Result> { - let mut perf_states = KVec::new(); - - let node = dev.as_ref().of_node().ok_or(EIO)?; - let opps = node.parse_phandle(name, 0).ok_or(EIO)?; - - for opp in opps.children() { - let freq_hz: u64 = opp.get_property(c_str!("opp-hz"))?; - let mut volt_uv: KVec = opp.get_property(c_str!("opp-microvolt"))?; - let pwr_uw: u32 = if is_main { - opp.get_property(c_str!("opp-microwatt"))? - } else { - 0 - }; - - let voltage_count = if is_main { - cfg.max_num_clusters - } else { - cfg.num_dies - }; - - if volt_uv.len() != voltage_count as usize { - dev_err!( - dev.as_ref(), - "Invalid opp-microvolt length (expected {}, got {})\n", - voltage_count, - volt_uv.len() - ); - return Err(EINVAL); - } - - volt_uv.iter_mut().for_each(|a| *a /= 1000); - let volt_mv = volt_uv; - - let pwr_mw = pwr_uw / 1000; - - perf_states.push( - PState { - freq_hz: freq_hz.try_into()?, - volt_mv, - pwr_mw, - }, - GFP_KERNEL, - )?; - } - - if perf_states.is_empty() { - Err(EINVAL) - } else { - Ok(perf_states) - } - } - - /// Load the GPU power configuration from the device tree. - pub(crate) fn load(dev: &AsahiDevice, cfg: &HwConfig) -> Result { - let perf_states = Self::load_opp(dev, c_str!("operating-points-v2"), cfg, true)?; - let node = dev.as_ref().of_node().ok_or(EIO)?; - - macro_rules! prop { - ($prop:expr, $default:expr) => {{ - node.get_opt_property(c_str!($prop)) - .map_err(|e| { - dev_err!(dev.as_ref(), "Error reading property {}: {:?}\n", $prop, e); - e - })? - .unwrap_or($default) - }}; - ($prop:expr) => {{ - node.get_property(c_str!($prop)).map_err(|e| { - dev_err!(dev.as_ref(), "Error reading property {}: {:?}\n", $prop, e); - e - })? - }}; - } - - let pz_data = prop!("apple,power-zones", KVec::new()); - - if pz_data.len() > 3 * MAX_POWERZONES || pz_data.len() % 3 != 0 { - dev_err!(dev.as_ref(), "Invalid apple,power-zones value\n"); - return Err(EINVAL); - } - - let mut power_zones = KVec::new(); - for zone in pz_data.chunks_exact(3) { - power_zones.push( - PowerZone { - target: zone[0], - target_offset: zone[1], - filter_tc: zone[2], - }, - GFP_KERNEL, - )?; - } - - let core_leak_coef: KVec = prop!("apple,core-leak-coef"); - let sram_leak_coef: KVec = prop!("apple,sram-leak-coef"); - - if core_leak_coef.len() != cfg.max_num_clusters as usize { - dev_err!(dev.as_ref(), "Invalid apple,core-leak-coef\n"); - return Err(EINVAL); - } - if sram_leak_coef.len() != cfg.max_num_clusters as usize { - dev_err!(dev.as_ref(), "Invalid apple,sram_leak_coef\n"); - return Err(EINVAL); - } - - let csafr = if cfg.has_csafr { - Some(CsAfrPwrConfig { - perf_states_cs: Self::load_opp(dev, c_str!("apple,cs-opp"), cfg, false)?, - perf_states_afr: Self::load_opp(dev, c_str!("apple,afr-opp"), cfg, false)?, - leak_coef_cs: prop!("apple,cs-leak-coef"), - leak_coef_afr: prop!("apple,afr-leak-coef"), - min_sram_microvolt: prop!("apple,csafr-min-sram-microvolt"), - }) - } else { - None - }; - - let power_sample_period: u32 = prop!("apple,power-sample-period"); - - Ok(PwrConfig { - core_leak_coef, - sram_leak_coef, - - max_power_mw: perf_states.iter().map(|a| a.pwr_mw).max().unwrap(), - max_freq_mhz: perf_states.iter().map(|a| a.freq_hz).max().unwrap() / 1_000_000, - - perf_base_pstate: prop!("apple,perf-base-pstate", 1), - perf_max_pstate: perf_states.len() as u32 - 1, - min_sram_microvolt: prop!("apple,min-sram-microvolt"), - - avg_power_filter_tc_ms: prop!("apple,avg-power-filter-tc-ms"), - avg_power_ki_only: prop!("apple,avg-power-ki-only"), - avg_power_kp: prop!("apple,avg-power-kp"), - avg_power_min_duty_cycle: prop!("apple,avg-power-min-duty-cycle"), - avg_power_target_filter_tc: prop!("apple,avg-power-target-filter-tc"), - fast_die0_integral_gain: prop!("apple,fast-die0-integral-gain"), - fast_die0_proportional_gain: prop!("apple,fast-die0-proportional-gain"), - fast_die0_prop_tgt_delta: prop!("apple,fast-die0-prop-tgt-delta", 0), - fast_die0_release_temp: prop!("apple,fast-die0-release-temp", 80), - fender_idle_off_delay_ms: prop!("apple,fender-idle-off-delay-ms", 40), - fw_early_wake_timeout_ms: prop!("apple,fw-early-wake-timeout-ms", 5), - idle_off_delay_ms: prop!("apple,idle-off-delay-ms", 2), - idle_off_standby_timer: prop!( - "apple,idleoff-standby-timer", - cfg.idle_off_standby_timer_default - ), - perf_boost_ce_step: prop!("apple,perf-boost-ce-step", 25), - perf_boost_min_util: prop!("apple,perf-boost-min-util", 100), - perf_filter_drop_threshold: prop!("apple,perf-filter-drop-threshold"), - perf_filter_time_constant2: prop!("apple,perf-filter-time-constant2"), - perf_filter_time_constant: prop!("apple,perf-filter-time-constant"), - perf_integral_gain2: prop!("apple,perf-integral-gain2"), - perf_integral_gain: prop!("apple,perf-integral-gain", f32!(7.8956833)), - perf_integral_min_clamp: prop!("apple,perf-integral-min-clamp"), - perf_proportional_gain2: prop!("apple,perf-proportional-gain2"), - perf_proportional_gain: prop!("apple,perf-proportional-gain", f32!(14.707963)), - perf_reset_iters: prop!("apple,perf-reset-iters", 6), - perf_tgt_utilization: prop!("apple,perf-tgt-utilization"), - power_sample_period, - ppm_filter_time_constant_ms: prop!("apple,ppm-filter-time-constant-ms"), - ppm_ki: prop!("apple,ppm-ki"), - ppm_kp: prop!("apple,ppm-kp"), - pwr_filter_time_constant: prop!("apple,pwr-filter-time-constant", 313), - pwr_integral_gain: prop!("apple,pwr-integral-gain", f32!(0.0202129)), - pwr_integral_min_clamp: prop!("apple,pwr-integral-min-clamp", 0), - pwr_min_duty_cycle: prop!("apple,pwr-min-duty-cycle"), - pwr_proportional_gain: prop!("apple,pwr-proportional-gain", f32!(5.2831855)), - pwr_sample_period_aic_clks: prop!( - "apple,pwr-sample-period-aic-clks", - cfg.base_clock_hz / 1000 * power_sample_period - ), - se_engagement_criteria: prop!("apple,se-engagement-criteria", -1), - se_filter_time_constant: prop!("apple,se-filter-time-constant", 9), - se_filter_time_constant_1: prop!("apple,se-filter-time-constant-1", 3), - se_inactive_threshold: prop!("apple,se-inactive-threshold", 2500), - se_ki: prop!("apple,se-ki", f32!(-50.0)), - se_ki_1: prop!("apple,se-ki-1", f32!(-100.0)), - se_kp: prop!("apple,se-kp", f32!(-5.0)), - se_kp_1: prop!("apple,se-kp-1", f32!(-10.0)), - se_reset_criteria: prop!("apple,se-reset-criteria", 50), - - perf_states, - power_zones, - csafr, - }) - } - - pub(crate) fn max_frequency_khz(&self) -> u32 { - self.perf_states[self.perf_max_pstate as usize].freq_hz / 1000 - } -} diff --git a/drivers/gpu/drm/asahi-neo/hw/t600x.rs b/drivers/gpu/drm/asahi-neo/hw/t600x.rs deleted file mode 100644 index 85cf88054e44c5..00000000000000 --- a/drivers/gpu/drm/asahi-neo/hw/t600x.rs +++ /dev/null @@ -1,162 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Hardware configuration for t600x (M1 Pro/Max/Ultra) platforms. - -use crate::f32; - -use super::*; - -const fn iomaps(mcc_count: usize, has_die1: bool) -> [Option; 20] { - [ - Some(IOMapping::new(0x404d00000, false, 1, 0x1c000, 0, true)), // Fender - Some(IOMapping::new(0x20e100000, false, 1, 0x4000, 0, false)), // AICTimer - Some(IOMapping::new(0x28e104000, false, 1, 0x4000, 0, true)), // AICSWInt - Some(IOMapping::new(0x404000000, false, 1, 0x20000, 0, true)), // RGX - None, // UVD - None, // unused - None, // DisplayUnderrunWA - Some(IOMapping::new(0x28e494000, true, 1, 0x4000, 0, false)), // AnalogTempSensorControllerRegs - None, // PMPDoorbell - Some(IOMapping::new(0x404d80000, false, 1, 0x8000, 0, true)), // MetrologySensorRegs - Some(IOMapping::new(0x204d61000, false, 1, 0x1000, 0, true)), // GMGIFAFRegs - Some(IOMapping::new( - 0x200000000, - true, - mcc_count, - 0xd8000, - 0x1000000, - true, - )), // MCache registers - None, // AICBankedRegisters - None, // PMGRScratch - Some(IOMapping::new(0x2643c4000, false, 1, 0x1000, 0, true)), // NIA Special agent idle register die 0 - if has_die1 { - // NIA Special agent idle register die 1 - Some(IOMapping::new(0x22643c4000, false, 1, 0x1000, 0, true)) - } else { - None - }, - None, // CRE registers - None, // Streaming codec registers - Some(IOMapping::new(0x28e3d0000, false, 1, 0x1000, 0, true)), // ? - Some(IOMapping::new(0x28e3c0000, false, 1, 0x2000, 0, false)), // ? - ] -} - -pub(crate) const HWCONFIG_T6002: super::HwConfig = HwConfig { - chip_id: 0x6002, - gpu_gen: GpuGen::G13, - gpu_variant: GpuVariant::D, - gpu_core: GpuCore::G13C, - - base_clock_hz: 24_000_000, - uat_ias: 39, - uat_oas: 42, - num_dies: 2, - max_num_clusters: 8, - max_num_cores: 8, - max_num_frags: 8, - max_num_gps: 4, - - preempt1_size: 0x540, - preempt2_size: 0x280, - preempt3_size: 0x20, - compute_preempt1_size: 0x3bd00, - clustering: Some(HwClusteringConfig { - meta1_blocksize: 0x44, - meta2_size: 0xc0 * 8, - meta3_size: 0x280 * 8, - meta4_size: 0x30 * 16, - max_splits: 16, - }), - - render: HwRenderConfig { - tiling_control: 0xa540, - }, - - da: HwConfigA { - unk_87c: 900, - unk_8cc: 11000, - unk_e24: 125, - }, - db: HwConfigB { - unk_454: 1, - unk_4e0: 4, - unk_534: 1, - unk_ab8: 0x2084, - unk_abc: 0x80, - unk_b30: 0, - }, - shared1_tab: &[ - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - ], - shared1_a4: 0xffff, - shared2_tab: &[-1, -1, -1, -1, 0x2aa, 0xaaa, -1, -1, 0, 0], - shared2_unk_508: 0xcc00001, - shared2_curves: None, - shared3_unk: 0, - shared3_tab: &[], - idle_off_standby_timer_default: 0, - unk_hws2_4: None, - unk_hws2_24: 0, - global_unk_54: 0xffff, - sram_k: f32!(1.02), - unk_coef_a: &[ - &f32!([9.838]), - &f32!([9.819]), - &f32!([9.826]), - &f32!([9.799]), - &f32!([9.799]), - &f32!([9.826]), - &f32!([9.819]), - &f32!([9.838]), - ], - unk_coef_b: &[ - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - &f32!([13.0]), - ], - global_tab: Some(&[ - 0, 1, 2, 1, 1, 90, 75, 1, 1, 1, 2, 90, 75, 1, 1, 1, 1, 90, 75, 1, 1, - ]), - has_csafr: false, - fast_sensor_mask: [0x8080808080808080, 0], - fast_sensor_mask_alt: [0x9090909090909090, 0], - fast_die0_sensor_present: 0xff, - io_mappings: &iomaps(8, true), - sram_base: None, - sram_size: None, -}; - -pub(crate) const HWCONFIG_T6001: super::HwConfig = HwConfig { - chip_id: 0x6001, - gpu_variant: GpuVariant::C, - gpu_core: GpuCore::G13C, - - num_dies: 1, - max_num_clusters: 4, - fast_sensor_mask: [0x80808080, 0], - fast_sensor_mask_alt: [0x90909090, 0], - fast_die0_sensor_present: 0x0f, - io_mappings: &iomaps(8, false), - ..HWCONFIG_T6002 -}; - -pub(crate) const HWCONFIG_T6000: super::HwConfig = HwConfig { - chip_id: 0x6000, - gpu_variant: GpuVariant::S, - gpu_core: GpuCore::G13S, - - max_num_clusters: 2, - fast_sensor_mask: [0x8080, 0], - fast_sensor_mask_alt: [0x9090, 0], - fast_die0_sensor_present: 0x03, - io_mappings: &iomaps(4, false), - ..HWCONFIG_T6001 -}; diff --git a/drivers/gpu/drm/asahi-neo/hw/t602x.rs b/drivers/gpu/drm/asahi-neo/hw/t602x.rs deleted file mode 100644 index 6e2033ecb50ddd..00000000000000 --- a/drivers/gpu/drm/asahi-neo/hw/t602x.rs +++ /dev/null @@ -1,180 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Hardware configuration for t600x (M1 Pro/Max/Ultra) platforms. - -use crate::f32; - -use super::*; - -const fn iomaps(chip_id: u32, mcc_count: usize) -> [Option; 24] { - [ - Some(IOMapping::new(0x404d00000, false, 1, 0x144000, 0, true)), // Fender - Some(IOMapping::new(0x20e100000, false, 1, 0x4000, 0, false)), // AICTimer - Some(IOMapping::new(0x28e106000, false, 1, 0x4000, 0, true)), // AICSWInt - Some(IOMapping::new(0x404000000, false, 1, 0x20000, 0, true)), // RGX - None, // UVD - None, // unused - None, // DisplayUnderrunWA - Some(match chip_id { - 0x6020 => IOMapping::new(0x28e460000, true, 1, 0x4000, 0, false), - _ => IOMapping::new(0x28e478000, true, 1, 0x4000, 0, false), - }), // AnalogTempSensorControllerRegs - None, // PMPDoorbell - Some(IOMapping::new(0x404e08000, false, 1, 0x8000, 0, true)), // MetrologySensorRegs - None, // GMGIFAFRegs - Some(IOMapping::new( - 0x200000000, - true, - mcc_count, - 0xd8000, - 0x1000000, - true, - )), // MCache registers - Some(IOMapping::new(0x28e118000, false, 1, 0x4000, 0, false)), // AICBankedRegisters - None, // PMGRScratch - None, // NIA Special agent idle register die 0 - None, // NIA Special agent idle register die 1 - None, // CRE registers - None, // Streaming codec registers - Some(IOMapping::new(0x28e3d0000, false, 1, 0x4000, 0, true)), // ? - Some(IOMapping::new(0x28e3c0000, false, 1, 0x4000, 0, false)), // ? - Some(IOMapping::new(0x28e3d8000, false, 1, 0x4000, 0, true)), // ? - Some(IOMapping::new(0x404eac000, true, 1, 0x4000, 0, true)), // ? - None, - None, - ] -} - -// TODO: Tentative -pub(crate) const HWCONFIG_T6022: super::HwConfig = HwConfig { - chip_id: 0x6022, - gpu_gen: GpuGen::G14, - gpu_variant: GpuVariant::D, - gpu_core: GpuCore::G14D, - - base_clock_hz: 24_000_000, - uat_ias: 39, - uat_oas: 42, - num_dies: 2, - max_num_clusters: 8, - max_num_cores: 10, - max_num_frags: 10, - max_num_gps: 4, - - preempt1_size: 0x540, - preempt2_size: 0x280, - preempt3_size: 0x40, - compute_preempt1_size: 0x25980 * 2, // Conservative guess - clustering: Some(HwClusteringConfig { - meta1_blocksize: 0x44, - meta2_size: 0xc0 * 16, - meta3_size: 0x280 * 16, - meta4_size: 0x10 * 128, - max_splits: 64, - }), - - render: HwRenderConfig { - tiling_control: 0x180340, - }, - - da: HwConfigA { - unk_87c: 500, - unk_8cc: 11000, - unk_e24: 125, - }, - db: HwConfigB { - unk_454: 1, - unk_4e0: 4, - unk_534: 0, - unk_ab8: 0, // Unused - unk_abc: 0, // Unused - unk_b30: 0, - }, - shared1_tab: &[ - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - ], - shared1_a4: 0, - shared2_tab: &[0x800, 0x1555, -1, -1, -1, -1, -1, -1, 0xaaaaa, 0], - shared2_unk_508: 0xc00007, - shared2_curves: Some(HwConfigShared2Curves { - t1_coef: 11000, - t2: &[ - 0xf07, 0x4c0, 0x680, 0x8c0, 0xa80, 0xc40, 0xd80, 0xec0, 0xf40, - ], - t3_coefs: &[0, 20, 27, 36, 43, 50, 55, 60, 62], - t3_scales: &[9, 3209, 10400], - }), - shared3_unk: 8, - shared3_tab: &[ - 125, 125, 125, 125, 125, 125, 125, 125, 7500, 125, 125, 125, 125, 125, 125, 125, - ], - idle_off_standby_timer_default: 700, - unk_hws2_4: Some(f32!([1.0, 0.8, 0.2, 0.9, 0.1, 0.25, 0.5, 0.9])), - unk_hws2_24: 6, - global_unk_54: 4000, - sram_k: f32!(1.02), - unk_coef_a: &[ - &f32!([0.0, 8.2, 0.0, 6.9, 6.9]), - &f32!([0.0, 0.0, 0.0, 6.9, 6.9]), - &f32!([0.0, 8.2, 0.0, 6.9, 0.0]), - &f32!([0.0, 0.0, 0.0, 6.9, 0.0]), - &f32!([0.0, 0.0, 0.0, 6.9, 0.0]), - &f32!([0.0, 8.2, 0.0, 6.9, 0.0]), - &f32!([0.0, 0.0, 0.0, 6.9, 6.9]), - &f32!([0.0, 8.2, 0.0, 6.9, 6.9]), - ], - unk_coef_b: &[ - &f32!([0.0, 9.0, 0.0, 8.0, 8.0]), - &f32!([0.0, 0.0, 0.0, 8.0, 8.0]), - &f32!([0.0, 9.0, 0.0, 8.0, 0.0]), - &f32!([0.0, 0.0, 0.0, 8.0, 0.0]), - &f32!([0.0, 0.0, 0.0, 8.0, 0.0]), - &f32!([0.0, 9.0, 0.0, 8.0, 0.0]), - &f32!([0.0, 0.0, 0.0, 8.0, 8.0]), - &f32!([0.0, 9.0, 0.0, 8.0, 8.0]), - ], - global_tab: Some(&[ - 0, 2, 2, 1, 1, 90, 75, 1, 1, 1, 2, 90, 75, 1, 1, 1, 2, 90, 75, 1, 1, 1, 1, 90, 75, 1, 1, - ]), - has_csafr: true, - fast_sensor_mask: [0x40005000c000d00, 0xd000c0005000400], - // Apple typo? Should probably be 0x140015001c001d00 - fast_sensor_mask_alt: [0x140015001d001d00, 0x1d001c0015001400], - fast_die0_sensor_present: 0, // Unused - io_mappings: &iomaps(0x6022, 8), - sram_base: Some(0x404d60000), - sram_size: Some(0x20000), -}; - -pub(crate) const HWCONFIG_T6021: super::HwConfig = HwConfig { - chip_id: 0x6021, - gpu_variant: GpuVariant::C, - gpu_core: GpuCore::G14C, - - num_dies: 1, - max_num_clusters: 4, - compute_preempt1_size: 0x25980, - unk_hws2_4: Some(f32!([1.0, 0.8, 0.2, 0.9, 0.1, 0.25, 0.7, 0.9])), - fast_sensor_mask: [0x40005000c000d00, 0], - fast_sensor_mask_alt: [0x140015001d001d00, 0], - io_mappings: &iomaps(0x6021, 8), - ..HWCONFIG_T6022 -}; - -pub(crate) const HWCONFIG_T6020: super::HwConfig = HwConfig { - chip_id: 0x6020, - gpu_variant: GpuVariant::S, - gpu_core: GpuCore::G14S, - - db: HwConfigB { - unk_454: 0, - ..HWCONFIG_T6021.db - }, - - max_num_clusters: 2, - fast_sensor_mask: [0xc000d00, 0], - fast_sensor_mask_alt: [0x1d001d00, 0], - io_mappings: &iomaps(0x6020, 4), - ..HWCONFIG_T6021 -}; diff --git a/drivers/gpu/drm/asahi-neo/hw/t8103.rs b/drivers/gpu/drm/asahi-neo/hw/t8103.rs deleted file mode 100644 index a2a76600c0a3bb..00000000000000 --- a/drivers/gpu/drm/asahi-neo/hw/t8103.rs +++ /dev/null @@ -1,93 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Hardware configuration for t8103 platforms (M1). - -use crate::f32; - -use super::*; - -pub(crate) const HWCONFIG: super::HwConfig = HwConfig { - chip_id: 0x8103, - gpu_gen: GpuGen::G13, - gpu_variant: GpuVariant::G, - gpu_core: GpuCore::G13G, - - base_clock_hz: 24_000_000, - uat_ias: 39, - uat_oas: 40, - num_dies: 1, - max_num_clusters: 1, - max_num_cores: 8, - max_num_frags: 8, - max_num_gps: 4, - - preempt1_size: 0x540, - preempt2_size: 0x280, - preempt3_size: 0x20, - compute_preempt1_size: 0x7f80, - clustering: None, - - render: HwRenderConfig { - // bit 0: disable clustering (always) - tiling_control: 0xa041, - }, - - da: HwConfigA { - unk_87c: -220, - unk_8cc: 9880, - unk_e24: 112, - }, - db: HwConfigB { - unk_454: 1, - unk_4e0: 0, - unk_534: 0, - unk_ab8: 0x48, - unk_abc: 0x8, - unk_b30: 0, - }, - shared1_tab: &[ - -1, 0x7282, 0x50ea, 0x370a, 0x25be, 0x1c1f, 0x16fb, -1, -1, -1, -1, -1, -1, -1, -1, -1, - ], - shared1_a4: 0xffff, - shared2_tab: &[0x800, 0x1555, -1, -1, -1, -1, -1, -1, 0, 0], - shared2_unk_508: 0xc00007, - shared2_curves: None, - shared3_unk: 0, - shared3_tab: &[], - idle_off_standby_timer_default: 0, - unk_hws2_4: None, - unk_hws2_24: 0, - global_unk_54: 0xffff, - sram_k: f32!(1.02), - unk_coef_a: &[], - unk_coef_b: &[], - global_tab: None, - has_csafr: false, - fast_sensor_mask: [0x12, 0], - fast_sensor_mask_alt: [0x12, 0], - fast_die0_sensor_present: 0x01, - io_mappings: &[ - Some(IOMapping::new(0x204d00000, false, 1, 0x1c000, 0, true)), // Fender - Some(IOMapping::new(0x20e100000, false, 1, 0x4000, 0, false)), // AICTimer - Some(IOMapping::new(0x23b104000, false, 1, 0x4000, 0, true)), // AICSWInt - Some(IOMapping::new(0x204000000, false, 1, 0x20000, 0, true)), // RGX - None, // UVD - None, // unused - None, // DisplayUnderrunWA - Some(IOMapping::new(0x23b2e8000, false, 1, 0x1000, 0, false)), // AnalogTempSensorControllerRegs - Some(IOMapping::new(0x23bc00000, false, 1, 0x1000, 0, true)), // PMPDoorbell - Some(IOMapping::new(0x204d80000, false, 1, 0x5000, 0, true)), // MetrologySensorRegs - Some(IOMapping::new(0x204d61000, false, 1, 0x1000, 0, true)), // GMGIFAFRegs - Some(IOMapping::new(0x200000000, false, 1, 0xd6400, 0, true)), // MCache registers - None, // AICBankedRegisters - Some(IOMapping::new(0x23b738000, false, 1, 0x1000, 0, true)), // PMGRScratch - None, // NIA Special agent idle register die 0 - None, // NIA Special agent idle register die 1 - None, // CRE registers - None, // Streaming codec registers - None, // - None, // - ], - sram_base: None, - sram_size: None, -}; diff --git a/drivers/gpu/drm/asahi-neo/hw/t8112.rs b/drivers/gpu/drm/asahi-neo/hw/t8112.rs deleted file mode 100644 index bac2757175f2c2..00000000000000 --- a/drivers/gpu/drm/asahi-neo/hw/t8112.rs +++ /dev/null @@ -1,106 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Hardware configuration for t8112 platforms (M2). - -use crate::f32; - -use super::*; - -pub(crate) const HWCONFIG: super::HwConfig = HwConfig { - chip_id: 0x8112, - gpu_gen: GpuGen::G14, - gpu_variant: GpuVariant::G, - gpu_core: GpuCore::G14G, - - base_clock_hz: 24_000_000, - uat_ias: 39, - uat_oas: 40, - num_dies: 1, - max_num_clusters: 1, - max_num_cores: 10, - max_num_frags: 10, - max_num_gps: 4, - - preempt1_size: 0x540, - preempt2_size: 0x280, - preempt3_size: 0x20, - compute_preempt1_size: 0x10000, // TODO: Check - clustering: None, - - render: HwRenderConfig { - // TODO: this is unused here, may be present in newer FW - tiling_control: 0xa041, - }, - - da: HwConfigA { - unk_87c: 900, - unk_8cc: 11000, - unk_e24: 125, - }, - db: HwConfigB { - unk_454: 1, - unk_4e0: 4, - unk_534: 0, - unk_ab8: 0x2048, - unk_abc: 0x4000, - unk_b30: 1, - }, - shared1_tab: &[ - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, - ], - shared1_a4: 0, - shared2_tab: &[-1, -1, -1, -1, -1, -1, -1, -1, 0xaa5aa, 0], - shared2_unk_508: 0xc00000, - shared2_curves: Some(HwConfigShared2Curves { - t1_coef: 7200, - t2: &[ - 0xf07, 0x4c0, 0x6c0, 0x8c0, 0xac0, 0xc40, 0xdc0, 0xec0, 0xf80, - ], - t3_coefs: &[0, 20, 28, 36, 44, 50, 56, 60, 63], - t3_scales: &[9, 3209, 10400], - }), - shared3_unk: 5, - shared3_tab: &[ - 10700, 10700, 10700, 10700, 10700, 6000, 1000, 1000, 1000, 10700, 10700, 10700, 10700, - 10700, 10700, 10700, - ], - idle_off_standby_timer_default: 0, - unk_hws2_4: None, - unk_hws2_24: 0, - global_unk_54: 0xffff, - - sram_k: f32!(1.02), - // 13.2: last coef changed from 6.6 to 5.3, assuming that was a fix we can backport - unk_coef_a: &[&f32!([0.0, 0.0, 0.0, 0.0, 5.3, 0.0, 5.3, /*6.6*/ 5.3])], - unk_coef_b: &[&f32!([0.0, 0.0, 0.0, 0.0, 5.3, 0.0, 5.3, /*6.6*/ 5.3])], - global_tab: None, - has_csafr: false, - fast_sensor_mask: [0x6800, 0], - fast_sensor_mask_alt: [0x6800, 0], - fast_die0_sensor_present: 0x02, - io_mappings: &[ - Some(IOMapping::new(0x204d00000, false, 1, 0x14000, 0, true)), // Fender - Some(IOMapping::new(0x20e100000, false, 1, 0x4000, 0, false)), // AICTimer - Some(IOMapping::new(0x23b0c4000, false, 1, 0x4000, 0, true)), // AICSWInt - Some(IOMapping::new(0x204000000, false, 1, 0x20000, 0, true)), // RGX - None, // UVD - None, // unused - None, // DisplayUnderrunWA - Some(IOMapping::new(0x23b2c0000, false, 1, 0x1000, 0, false)), // AnalogTempSensorControllerRegs - None, // PMPDoorbell - Some(IOMapping::new(0x204d80000, false, 1, 0x8000, 0, true)), // MetrologySensorRegs - Some(IOMapping::new(0x204d61000, false, 1, 0x1000, 0, true)), // GMGIFAFRegs - Some(IOMapping::new(0x200000000, false, 1, 0xd6400, 0, true)), // MCache registers - None, // AICBankedRegisters - None, // PMGRScratch - None, // NIA Special agent idle register die 0 - None, // NIA Special agent idle register die 1 - Some(IOMapping::new(0x204e00000, false, 1, 0x10000, 0, true)), // CRE registers - Some(IOMapping::new(0x27d050000, false, 1, 0x4000, 0, true)), // Streaming codec registers - Some(IOMapping::new(0x23b3d0000, false, 1, 0x1000, 0, true)), // - Some(IOMapping::new(0x23b3c0000, false, 1, 0x1000, 0, false)), // - ], - sram_base: None, - sram_size: None, -}; diff --git a/drivers/gpu/drm/asahi-neo/initdata.rs b/drivers/gpu/drm/asahi-neo/initdata.rs deleted file mode 100644 index 91a5bd73d97e9c..00000000000000 --- a/drivers/gpu/drm/asahi-neo/initdata.rs +++ /dev/null @@ -1,1025 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![allow(clippy::unusual_byte_groupings)] - -//! GPU initialization data builder. -//! -//! The root of all interaction between the GPU firmware and the host driver is a complex set of -//! nested structures that we call InitData. This includes both GPU hardware/firmware configuration -//! and the pointers to the ring buffers and global data fields that are used for communication at -//! runtime. -//! -//! Many of these structures are poorly understood, so there are lots of hardcoded unknown values -//! derived from observing the InitData structures that macOS generates. - -use crate::f32; -use crate::fw::initdata::*; -use crate::fw::types::*; -use crate::module_parameters; -use crate::{ - driver::AsahiDevice, - gem, - gpu, - hw, - mmu, // -}; -use kernel::error::{ - Error, - Result, // -}; -use kernel::macros::versions; -use kernel::prelude::*; -use kernel::try_init; - -use ::pin_init; -use ::pin_init::Init; - -/// Builder helper for the global GPU InitData. -#[versions(AGX_NEO)] -pub(crate) struct InitDataBuilder<'a> { - dev: &'a AsahiDevice, - alloc: &'a mut gpu::KernelAllocators, - cfg: &'static hw::HwConfig, - dyncfg: &'a hw::DynConfig, -} - -#[versions(AGX_NEO)] -impl<'a> InitDataBuilder::ver<'a> { - /// Create a new InitData builder - pub(crate) fn new( - dev: &'a AsahiDevice, - alloc: &'a mut gpu::KernelAllocators, - cfg: &'static hw::HwConfig, - dyncfg: &'a hw::DynConfig, - ) -> InitDataBuilder::ver<'a> { - InitDataBuilder::ver { - dev, - alloc, - cfg, - dyncfg, - } - } - - /// Create the HwDataShared1 structure, which is used in two places in InitData. - fn hw_shared1(cfg: &'static hw::HwConfig) -> impl Init { - init!(raw::HwDataShared1 { - unk_a4: cfg.shared1_a4, - ..Zeroable::init_zeroed() - }) - .chain(|ret| { - for (i, val) in cfg.shared1_tab.iter().enumerate() { - ret.table[i] = *val; - } - Ok(()) - }) - } - - fn init_curve( - curve: &mut raw::HwDataShared2Curve, - unk_0: u32, - unk_4: u32, - t1: &[u16], - t2: &[i16], - t3: &[KVec], - ) { - curve.unk_0 = unk_0; - curve.unk_4 = unk_4; - (*curve.t1)[..t1.len()].copy_from_slice(t1); - (*curve.t1)[t1.len()..].fill(t1[0]); - (*curve.t2)[..t2.len()].copy_from_slice(t2); - (*curve.t2)[t2.len()..].fill(t2[0]); - for (i, a) in curve.t3.iter_mut().enumerate() { - a.fill(0x3ffffff); - if i < t3.len() { - let b = &t3[i]; - (**a)[..b.len()].copy_from_slice(b); - } - } - } - - /// Create the HwDataShared2 structure, which is used in two places in InitData. - fn hw_shared2( - cfg: &'static hw::HwConfig, - dyncfg: &'a hw::DynConfig, - ) -> impl Init + 'a { - try_init!(raw::HwDataShared2 { - unk_28: Array::new([0xff; 16]), - g14: Default::default(), - unk_508: cfg.shared2_unk_508, - ..Zeroable::init_zeroed() - }) - .chain(|ret| { - for (i, val) in cfg.shared2_tab.iter().enumerate() { - ret.table[i] = *val; - } - - let curve_cfg = match cfg.shared2_curves.as_ref() { - None => return Ok(()), - Some(a) => a, - }; - - let mut t1 = KVec::new(); - let mut t3 = KVec::new(); - - for _ in 0..curve_cfg.t3_scales.len() { - t3.push(KVec::new(), GFP_KERNEL)?; - } - - for (i, ps) in dyncfg.pwr.perf_states.iter().enumerate() { - let t3_coef = curve_cfg.t3_coefs[i]; - if t3_coef == 0 { - t1.push(0xffff, GFP_KERNEL)?; - for j in t3.iter_mut() { - j.push(0x3ffffff, GFP_KERNEL)?; - } - continue; - } - - let f_khz = (ps.freq_hz / 1000) as u64; - let v_max = ps.max_volt_mv() as u64; - - t1.push( - (1000000000 * (curve_cfg.t1_coef as u64) / (f_khz * v_max)) - .try_into() - .unwrap(), - GFP_KERNEL, - )?; - - for (j, scale) in curve_cfg.t3_scales.iter().enumerate() { - t3[j].push( - (t3_coef as u64 * 1000000100 * *scale as u64 / (f_khz * v_max * 6)) - .try_into() - .unwrap(), - GFP_KERNEL, - )?; - } - } - - ret.g14.unk_14 = 0x6000000; - Self::init_curve( - &mut ret.g14.curve1, - 0, - 0x20000000, - &[0xffff], - &[0x0f07], - &[], - ); - Self::init_curve(&mut ret.g14.curve2, 7, 0x80000000, &t1, curve_cfg.t2, &t3); - - Ok(()) - }) - } - - /// Create the HwDataShared3 structure, which is used in two places in InitData. - fn hw_shared3(cfg: &'static hw::HwConfig) -> impl Init { - pin_init::init_zeroed::().chain(|ret| { - if !cfg.shared3_tab.is_empty() { - ret.unk_0 = 1; - ret.unk_4 = 500; - ret.unk_8 = cfg.shared3_unk; - ret.table.copy_from_slice(cfg.shared3_tab); - ret.unk_4c = 1; - } - Ok(()) - }) - } - - /// Create an unknown T81xx-specific data structure. - fn t81xx_data( - cfg: &'static hw::HwConfig, - dyncfg: &'a hw::DynConfig, - ) -> impl Init { - let _perf_max_pstate = dyncfg.pwr.perf_max_pstate; - - pin_init::init_zeroed::().chain(move |_ret| { - match cfg.chip_id { - 0x8103 | 0x8112 => { - #[ver(V < V13_3)] - { - _ret.unk_d8c = 0x80000000; - _ret.unk_d90 = 4; - _ret.unk_d9c = f32!(0.6); - _ret.unk_da4 = f32!(0.4); - _ret.unk_dac = f32!(0.38552); - _ret.unk_db8 = f32!(65536.0); - _ret.unk_dbc = f32!(13.56); - _ret.max_pstate_scaled = 100 * _perf_max_pstate; - } - } - _ => (), - } - Ok(()) - }) - } - - /// Create the HwDataA structure. This mostly contains power-related configuration. - fn hwdata_a(&mut self) -> Result> { - let pwr = &self.dyncfg.pwr; - let period_ms = pwr.power_sample_period; - let period_s = F32::from(period_ms) / f32!(1000.0); - let ppm_filter_tc_periods = pwr.ppm_filter_time_constant_ms / period_ms; - #[ver(V >= V13_0B4)] - let ppm_filter_tc_ms_rounded = ppm_filter_tc_periods * period_ms; - let ppm_filter_a = f32!(1.0) / ppm_filter_tc_periods.into(); - let perf_filter_a = f32!(1.0) / pwr.perf_filter_time_constant.into(); - let perf_filter_a2 = f32!(1.0) / pwr.perf_filter_time_constant2.into(); - let avg_power_target_filter_a = f32!(1.0) / pwr.avg_power_target_filter_tc.into(); - let avg_power_filter_tc_periods = pwr.avg_power_filter_tc_ms / period_ms; - #[ver(V >= V13_0B4)] - let avg_power_filter_tc_ms_rounded = avg_power_filter_tc_periods * period_ms; - let avg_power_filter_a = f32!(1.0) / avg_power_filter_tc_periods.into(); - let pwr_filter_a = f32!(1.0) / pwr.pwr_filter_time_constant.into(); - - let base_ps = pwr.perf_base_pstate; - let base_ps_scaled = 100 * base_ps; - let max_ps = pwr.perf_max_pstate; - let max_ps_scaled = 100 * max_ps; - let boost_ps_count = max_ps - base_ps; - - #[allow(unused_variables)] - let base_clock_khz = self.cfg.base_clock_hz / 1000; - let v_clocks_per_period = pwr.pwr_sample_period_aic_clks; - - #[allow(unused_variables)] - let clocks_per_period_coarse = self.cfg.base_clock_hz / 1000 * pwr.power_sample_period; - - self.alloc - .private - .new_init(pin_init::init_zeroed(), |_inner, _ptr| { - let cfg = &self.cfg; - let dyncfg = &self.dyncfg; - try_init!(raw::HwDataA::ver { - clocks_per_period: v_clocks_per_period, - #[ver(V >= V13_0B4)] - clocks_per_period_2: v_clocks_per_period, - pwr_status: AtomicU32::new(4), - unk_10: f32!(1.0), - actual_pstate: 1, - tgt_pstate: 1, - base_pstate_scaled: base_ps_scaled, - unk_40: 1, - max_pstate_scaled: max_ps_scaled, - min_pstate_scaled: 100, - unk_64c: 625, - pwr_filter_a_neg: f32!(1.0) - pwr_filter_a, - pwr_filter_a: pwr_filter_a, - pwr_integral_gain: pwr.pwr_integral_gain, - pwr_integral_min_clamp: pwr.pwr_integral_min_clamp.into(), - max_power_1: pwr.max_power_mw.into(), - pwr_proportional_gain: pwr.pwr_proportional_gain, - pwr_pstate_related_k: -F32::from(max_ps_scaled) / pwr.max_power_mw.into(), - pwr_pstate_max_dc_offset: pwr.pwr_min_duty_cycle as i32 - max_ps_scaled as i32, - max_pstate_scaled_2: max_ps_scaled, - max_power_2: pwr.max_power_mw, - max_pstate_scaled_3: max_ps_scaled, - ppm_filter_tc_periods_x4: ppm_filter_tc_periods * 4, - ppm_filter_a_neg: f32!(1.0) - ppm_filter_a, - ppm_filter_a: ppm_filter_a, - ppm_ki_dt: pwr.ppm_ki * period_s, - unk_6fc: f32!(65536.0), - ppm_kp: pwr.ppm_kp, - pwr_min_duty_cycle: pwr.pwr_min_duty_cycle, - max_pstate_scaled_4: max_ps_scaled, - unk_71c: f32!(0.0), - max_power_3: pwr.max_power_mw, - cur_power_mw_2: 0x0, - ppm_filter_tc_ms: pwr.ppm_filter_time_constant_ms, - #[ver(V >= V13_0B4)] - ppm_filter_tc_clks: ppm_filter_tc_ms_rounded * base_clock_khz, - perf_tgt_utilization: pwr.perf_tgt_utilization, - perf_boost_min_util: pwr.perf_boost_min_util, - perf_boost_ce_step: pwr.perf_boost_ce_step, - perf_reset_iters: pwr.perf_reset_iters, - unk_774: 6, - unk_778: 1, - perf_filter_drop_threshold: pwr.perf_filter_drop_threshold, - perf_filter_a_neg: f32!(1.0) - perf_filter_a, - perf_filter_a2_neg: f32!(1.0) - perf_filter_a2, - perf_filter_a: perf_filter_a, - perf_filter_a2: perf_filter_a2, - perf_ki: pwr.perf_integral_gain, - perf_ki2: pwr.perf_integral_gain2, - perf_integral_min_clamp: pwr.perf_integral_min_clamp.into(), - unk_79c: f32!(95.0), - perf_kp: pwr.perf_proportional_gain, - perf_kp2: pwr.perf_proportional_gain2, - boost_state_unk_k: F32::from(boost_ps_count) / f32!(0.95), - base_pstate_scaled_2: base_ps_scaled, - max_pstate_scaled_5: max_ps_scaled, - base_pstate_scaled_3: base_ps_scaled, - perf_tgt_utilization_2: pwr.perf_tgt_utilization, - base_pstate_scaled_4: base_ps_scaled, - unk_7fc: f32!(65536.0), - pwr_min_duty_cycle_2: pwr.pwr_min_duty_cycle.into(), - max_pstate_scaled_6: max_ps_scaled.into(), - max_freq_mhz: pwr.max_freq_mhz, - pwr_min_duty_cycle_3: pwr.pwr_min_duty_cycle, - min_pstate_scaled_4: f32!(100.0), - max_pstate_scaled_7: max_ps_scaled, - unk_alpha_neg: f32!(0.8), - unk_alpha: f32!(0.2), - fast_die0_sensor_mask: U64(cfg.fast_sensor_mask[0]), - #[ver(G >= G14X)] - fast_die1_sensor_mask: U64(cfg.fast_sensor_mask[1]), - fast_die0_release_temp_cc: 100 * pwr.fast_die0_release_temp, - unk_87c: cfg.da.unk_87c, - unk_880: 0x4, - unk_894: f32!(1.0), - - fast_die0_ki_dt: pwr.fast_die0_integral_gain * period_s, - unk_8a8: f32!(65536.0), - fast_die0_kp: pwr.fast_die0_proportional_gain, - pwr_min_duty_cycle_4: pwr.pwr_min_duty_cycle, - max_pstate_scaled_8: max_ps_scaled, - max_pstate_scaled_9: max_ps_scaled, - fast_die0_prop_tgt_delta: 100 * pwr.fast_die0_prop_tgt_delta, - unk_8cc: cfg.da.unk_8cc, - max_pstate_scaled_10: max_ps_scaled, - max_pstate_scaled_11: max_ps_scaled, - unk_c2c: 1, - power_zone_count: pwr.power_zones.len() as u32, - max_power_4: pwr.max_power_mw, - max_power_5: pwr.max_power_mw, - max_power_6: pwr.max_power_mw, - avg_power_target_filter_a_neg: f32!(1.0) - avg_power_target_filter_a, - avg_power_target_filter_a: avg_power_target_filter_a, - avg_power_target_filter_tc_x4: 4 * pwr.avg_power_target_filter_tc, - avg_power_target_filter_tc_xperiod: period_ms * pwr.avg_power_target_filter_tc, - #[ver(V >= V13_0B4)] - avg_power_target_filter_tc_clks: period_ms - * pwr.avg_power_target_filter_tc - * base_clock_khz, - avg_power_filter_tc_periods_x4: 4 * avg_power_filter_tc_periods, - avg_power_filter_a_neg: f32!(1.0) - avg_power_filter_a, - avg_power_filter_a: avg_power_filter_a, - avg_power_ki_dt: pwr.avg_power_ki_only * period_s, - unk_d20: f32!(65536.0), - avg_power_kp: pwr.avg_power_kp, - avg_power_min_duty_cycle: pwr.avg_power_min_duty_cycle, - max_pstate_scaled_12: max_ps_scaled, - max_pstate_scaled_13: max_ps_scaled, - max_power_7: pwr.max_power_mw.into(), - max_power_8: pwr.max_power_mw, - avg_power_filter_tc_ms: pwr.avg_power_filter_tc_ms, - #[ver(V >= V13_0B4)] - avg_power_filter_tc_clks: avg_power_filter_tc_ms_rounded * base_clock_khz, - max_pstate_scaled_14: max_ps_scaled, - t81xx_data <- Self::t81xx_data(cfg, dyncfg), - #[ver(V >= V13_0B4)] - unk_e10_0 <- { - let filter_a = f32!(1.0) / pwr.se_filter_time_constant.into(); - let filter_1_a = f32!(1.0) / pwr.se_filter_time_constant_1.into(); - try_init!(raw::HwDataA130Extra { - unk_38: 4, - unk_3c: 8000, - gpu_se_inactive_threshold: pwr.se_inactive_threshold, - gpu_se_engagement_criteria: pwr.se_engagement_criteria, - gpu_se_reset_criteria: pwr.se_reset_criteria, - unk_54: 50, - unk_58: 0x1, - gpu_se_filter_a_neg: f32!(1.0) - filter_a, - gpu_se_filter_1_a_neg: f32!(1.0) - filter_1_a, - gpu_se_filter_a: filter_a, - gpu_se_filter_1_a: filter_1_a, - gpu_se_ki_dt: pwr.se_ki * period_s, - gpu_se_ki_1_dt: pwr.se_ki_1 * period_s, - unk_7c: f32!(65536.0), - gpu_se_kp: pwr.se_kp, - gpu_se_kp_1: pwr.se_kp_1, - - #[ver(V >= V13_3)] - unk_8c: 100, - #[ver(V < V13_3)] - unk_8c: 40, - - max_pstate_scaled_1: max_ps_scaled, - unk_9c: f32!(8000.0), - unk_a0: 1400, - gpu_se_filter_time_constant_ms: pwr.se_filter_time_constant * period_ms, - gpu_se_filter_time_constant_1_ms: pwr.se_filter_time_constant_1 - * period_ms, - gpu_se_filter_time_constant_clks: U64((pwr.se_filter_time_constant - * clocks_per_period_coarse) - .into()), - gpu_se_filter_time_constant_1_clks: U64((pwr - .se_filter_time_constant_1 - * clocks_per_period_coarse) - .into()), - unk_c4: f32!(65536.0), - unk_114: f32!(65536.0), - unk_124: 40, - max_pstate_scaled_2: max_ps_scaled, - ..Zeroable::init_zeroed() - }) - }, - fast_die0_sensor_mask_2: U64(cfg.fast_sensor_mask[0]), - #[ver(G >= G14X)] - fast_die1_sensor_mask_2: U64(cfg.fast_sensor_mask[1]), - unk_e24: cfg.da.unk_e24, - unk_e28: 1, - fast_die0_sensor_mask_alt: U64(cfg.fast_sensor_mask_alt[0]), - #[ver(G >= G14X)] - fast_die1_sensor_mask_alt: U64(cfg.fast_sensor_mask_alt[1]), - #[ver(V < V13_0B4)] - fast_die0_sensor_present: U64(cfg.fast_die0_sensor_present as u64), - unk_163c: 1, - unk_3644: 0, - hws1 <- Self::hw_shared1(cfg), - hws2 <- Self::hw_shared2(cfg, dyncfg), - hws3 <- Self::hw_shared3(cfg), - unk_3ce8: 1, - ..Zeroable::init_zeroed() - }) - .chain(|raw| { - for i in 0..self.dyncfg.pwr.perf_states.len() { - raw.sram_k[i] = self.cfg.sram_k; - } - - for (i, coef) in pwr.core_leak_coef.iter().enumerate() { - raw.core_leak_coef[i] = *coef; - } - - for (i, coef) in pwr.sram_leak_coef.iter().enumerate() { - raw.sram_leak_coef[i] = *coef; - } - - #[ver(V >= V13_0B4)] - if let Some(csafr) = pwr.csafr.as_ref() { - for (i, coef) in csafr.leak_coef_afr.iter().enumerate() { - raw.aux_leak_coef.cs_1[i] = *coef; - raw.aux_leak_coef.cs_2[i] = *coef; - } - - for (i, coef) in csafr.leak_coef_cs.iter().enumerate() { - raw.aux_leak_coef.afr_1[i] = *coef; - raw.aux_leak_coef.afr_2[i] = *coef; - } - } - - for i in 0..self.dyncfg.id.num_clusters as usize { - if let Some(coef_a) = self.cfg.unk_coef_a.get(i) { - (*raw.unk_coef_a1[i])[..coef_a.len()].copy_from_slice(coef_a); - (*raw.unk_coef_a2[i])[..coef_a.len()].copy_from_slice(coef_a); - } - if let Some(coef_b) = self.cfg.unk_coef_b.get(i) { - (*raw.unk_coef_b1[i])[..coef_b.len()].copy_from_slice(coef_b); - (*raw.unk_coef_b2[i])[..coef_b.len()].copy_from_slice(coef_b); - } - } - - for (i, pz) in pwr.power_zones.iter().enumerate() { - raw.power_zones[i].target = pz.target; - raw.power_zones[i].target_off = pz.target - pz.target_offset; - raw.power_zones[i].filter_tc_x4 = 4 * pz.filter_tc; - raw.power_zones[i].filter_tc_xperiod = period_ms * pz.filter_tc; - let filter_a = f32!(1.0) / pz.filter_tc.into(); - raw.power_zones[i].filter_a = filter_a; - raw.power_zones[i].filter_a_neg = f32!(1.0) - filter_a; - #[ver(V >= V13_0B4)] - raw.power_zones[i].unk_10 = 1320000000; - } - - #[ver(V >= V13_0B4 && G >= G14X)] - for (i, j) in raw.hws2.g14.curve2.t1.iter().enumerate() { - raw.unk_hws2[i] = if *j == 0xffff { 0 } else { j / 2 }; - } - - if !dyncfg.hw_data_b.is_empty() { - unsafe { - let mut matches: bool = true; - let sla = core::slice::from_raw_parts( - raw as *const raw::HwDataA::ver as *const u8, - core::mem::size_of::(), - ); - if sla.len() != dyncfg.hw_data_a.len() { - matches = false; - dev_err!( - self.dev.as_ref(), - "!!! Hwdata A size mismatch: {} {}", - sla.len(), - dyncfg.hw_data_a.len(), - ); - } - for i in 0..core::cmp::min(sla.len(), dyncfg.hw_data_a.len()) { - if sla[i] != dyncfg.hw_data_a[i] { - matches = false; - dev_err!(self.dev.as_ref(), "!!! Hwdata A first mismatch: {i}"); - break; - } - } - if matches { - dev_info!(self.dev.as_ref(), "!!! Hwdata A match"); - } - } - } - - Ok(()) - }) - }) - } - - /// Create the HwDataB structure. This mostly contains GPU-related configuration. - fn hwdata_b(&mut self) -> Result> { - self.alloc - .private - .new_init(pin_init::init_zeroed(), |_inner, _ptr| { - let cfg = &self.cfg; - let dyncfg = &self.dyncfg; - try_init!(raw::HwDataB::ver { - // Userspace VA map related - #[ver(V < V13_0B4)] - unk_0: U64(0x13_00000000), - unk_8: U64(0x14_00000000), - #[ver(V < V13_0B4)] - unk_10: U64(0x1_00000000), - unk_18: U64(0xffc00000), - // USC start - unk_20: U64(0), // U64(0x11_00000000), - unk_28: U64(0), // U64(0x11_00000000), - // Unknown page - //unk_30: U64(0x6f_ffff8000), - unk_30: U64(mmu::UatGeometry { - ias: cfg.uat_ias, - oas: cfg.uat_oas, - } - .unk_page()), - timestamp_area_base: U64(gpu::IOVA_KERN_TIMESTAMP_RANGE.start), - // TODO: yuv matrices - chip_id: cfg.chip_id, - unk_454: cfg.db.unk_454, - unk_458: 0x1, - unk_460: 0x1, - unk_464: 0x1, - unk_468: 0x1, - unk_47c: 0x1, - unk_484: 0x1, - unk_48c: 0x1, - base_clock_khz: cfg.base_clock_hz / 1000, - power_sample_period: dyncfg.pwr.power_sample_period, - unk_49c: 0x1, - unk_4a0: 0x1, - unk_4a4: 0x1, - unk_4c0: 0x1f, - unk_4e0: U64(cfg.db.unk_4e0), - unk_4f0: 0x1, - unk_4f4: 0x1, - unk_504: 0x31, - unk_524: 0x1, // use_secure_cache_flush - unk_534: cfg.db.unk_534, - num_frags: dyncfg.id.num_frags * dyncfg.id.num_clusters, - unk_554: 0x1, - uat_ttb_base: U64(dyncfg.uat_ttb_base), - gpu_core_id: cfg.gpu_core as u32, - gpu_rev_id: dyncfg.id.gpu_rev_id as u32, - num_cores: dyncfg.id.num_cores * dyncfg.id.num_clusters, - max_pstate: dyncfg.pwr.perf_states.len() as u32 - 1, - #[ver(V < V13_0B4)] - num_pstates: dyncfg.pwr.perf_states.len() as u32, - #[ver(V < V13_0B4)] - min_sram_volt: dyncfg.pwr.min_sram_microvolt / 1000, - #[ver(V < V13_0B4)] - unk_ab8: cfg.db.unk_ab8, - #[ver(V < V13_0B4)] - unk_abc: cfg.db.unk_abc, - #[ver(V < V13_0B4)] - unk_ac0: 0x1020, - - #[ver(V >= V13_0B4)] - unk_ae4: Array::new([0x0, 0x3, 0x7, 0x7]), - #[ver(V < V13_0B4)] - unk_ae4: Array::new([0x0, 0xf, 0x3f, 0x3f]), - unk_b10: 0x1, - timer_offset: U64(0), - unk_b24: 0x1, - unk_b28: 0x1, - unk_b2c: 0x1, - unk_b30: cfg.db.unk_b30, - #[ver(V >= V13_0B4)] - unk_b38_0: 1, - #[ver(V >= V13_0B4)] - unk_b38_4: 1, - unk_b38: Array::new([0xffffffff; 12]), - #[ver(V >= V13_0B4 && V < V13_3)] - unk_c3c: 0x19, - #[ver(V >= V13_3)] - unk_c3c: 0x1a, - ..Zeroable::init_zeroed() - }) - .chain(|raw| { - #[ver(V >= V13_3)] - for i in 0..16 { - raw.unk_arr_0[i] = i as u32; - } - - let base_ps = self.dyncfg.pwr.perf_base_pstate as usize; - let max_ps = self.dyncfg.pwr.perf_max_pstate as usize; - let base_freq = self.dyncfg.pwr.perf_states[base_ps].freq_hz; - let max_freq = self.dyncfg.pwr.perf_states[max_ps].freq_hz; - - for (i, ps) in self.dyncfg.pwr.perf_states.iter().enumerate() { - raw.frequencies[i] = ps.freq_hz / 1000000; - for (j, mv) in ps.volt_mv.iter().enumerate() { - let sram_mv = (*mv).max(self.dyncfg.pwr.min_sram_microvolt / 1000); - raw.voltages[i][j] = *mv; - raw.voltages_sram[i][j] = sram_mv; - } - for j in ps.volt_mv.len()..raw.voltages[i].len() { - raw.voltages[i][j] = raw.voltages[i][0]; - raw.voltages_sram[i][j] = raw.voltages_sram[i][0]; - } - raw.sram_k[i] = self.cfg.sram_k; - raw.rel_max_powers[i] = ps.pwr_mw * 100 / self.dyncfg.pwr.max_power_mw; - raw.rel_boost_freqs[i] = if i > base_ps { - (ps.freq_hz - base_freq) / ((max_freq - base_freq) / 100) - } else { - 0 - }; - } - - #[ver(V >= V13_0B4)] - if let Some(csafr) = self.dyncfg.pwr.csafr.as_ref() { - let aux = &mut raw.aux_ps; - aux.cs_max_pstate = (csafr.perf_states_cs.len() - 1).try_into()?; - aux.afr_max_pstate = (csafr.perf_states_afr.len() - 1).try_into()?; - - for (i, ps) in csafr.perf_states_cs.iter().enumerate() { - aux.cs_frequencies[i] = ps.freq_hz / 1000000; - for (j, mv) in ps.volt_mv.iter().enumerate() { - let sram_mv = (*mv).max(csafr.min_sram_microvolt / 1000); - aux.cs_voltages[i][j] = *mv; - aux.cs_voltages_sram[i][j] = sram_mv; - } - } - - for (i, ps) in csafr.perf_states_afr.iter().enumerate() { - aux.afr_frequencies[i] = ps.freq_hz / 1000000; - for (j, mv) in ps.volt_mv.iter().enumerate() { - let sram_mv = (*mv).max(csafr.min_sram_microvolt / 1000); - aux.afr_voltages[i][j] = *mv; - aux.afr_voltages_sram[i][j] = sram_mv; - } - } - } - - // Special case override for T602x - #[ver(G == G14X)] - if dyncfg.id.gpu_rev_id == hw::GpuRevisionID::B1 { - raw.gpu_rev_id = hw::GpuRevisionID::B0 as u32; - } - - if !dyncfg.hw_data_b.is_empty() { - unsafe { - let mut matches: bool = true; - let sla = core::slice::from_raw_parts( - raw as *const raw::HwDataB::ver as *const u8, - core::mem::size_of::(), - ); - if sla.len() != dyncfg.hw_data_b.len() { - matches = false; - dev_err!( - self.dev.as_ref(), - "!!! Hwdata B size mismatch: {} {}", - sla.len(), - dyncfg.hw_data_b.len(), - ); - } - for i in 0..core::cmp::min(sla.len(), dyncfg.hw_data_b.len()) { - if sla[i] != dyncfg.hw_data_b[i] { - matches = false; - dev_err!(self.dev.as_ref(), "!!! Hwdata B first mismatch: {i}"); - break; - } - } - if matches { - dev_info!(self.dev.as_ref(), "!!! Hwdata B match"); - } - } - } - - Ok(()) - }) - }) - } - - /// Create the Globals structure, which contains global firmware config including more power - /// configuration data and globals used to exchange state between the firmware and driver. - fn globals(&mut self) -> Result> { - self.alloc - .private - .new_init(pin_init::init_zeroed(), |_inner, _ptr| { - let cfg = &self.cfg; - let dyncfg = &self.dyncfg; - let pwr = &dyncfg.pwr; - let period_ms = pwr.power_sample_period; - let period_s = F32::from(period_ms) / f32!(1000.0); - let avg_power_filter_tc_periods = pwr.avg_power_filter_tc_ms / period_ms; - - let max_ps = pwr.perf_max_pstate; - let max_ps_scaled = 100 * max_ps; - - try_init!(raw::Globals::ver { - //ktrace_enable: 0xffffffff, - ktrace_enable: 0, - #[ver(V >= V13_2)] - unk_24_0: 3000, - unk_24: 0, - #[ver(V >= V13_0B4)] - debug: 0, - unk_28: 1, - #[ver(G >= G14X)] - unk_2c_0: 1, - #[ver(V >= V13_0B4 && G < G14X)] - unk_2c_0: 0, - unk_2c: 1, - unk_30: 0, - unk_34: 120, - // sub <- try_init!(raw::GlobalsSub::ver { - unk_54: cfg.global_unk_54, - unk_56: 40, - unk_58: 0xffff, - unk_5e: U32(1), - unk_66: U32(1), - // ..Zeroable::init_zeroed() - // }), - unk_8900: 1, - pending_submissions: AtomicU32::new(0), - max_power: pwr.max_power_mw, - max_pstate_scaled: max_ps_scaled, - max_pstate_scaled_2: max_ps_scaled, - max_pstate_scaled_3: max_ps_scaled, - power_zone_count: pwr.power_zones.len() as u32, - avg_power_filter_tc_periods: avg_power_filter_tc_periods, - avg_power_ki_dt: pwr.avg_power_ki_only * period_s, - avg_power_kp: pwr.avg_power_kp, - avg_power_min_duty_cycle: pwr.avg_power_min_duty_cycle, - avg_power_target_filter_tc: pwr.avg_power_target_filter_tc, - unk_89bc: cfg.da.unk_8cc, - fast_die0_release_temp: 100 * pwr.fast_die0_release_temp, - unk_89c4: cfg.da.unk_87c, - fast_die0_prop_tgt_delta: 100 * pwr.fast_die0_prop_tgt_delta, - fast_die0_kp: pwr.fast_die0_proportional_gain, - fast_die0_ki_dt: pwr.fast_die0_integral_gain * period_s, - unk_89e0: 1, - max_power_2: pwr.max_power_mw, - ppm_kp: pwr.ppm_kp, - ppm_ki_dt: pwr.ppm_ki * period_s, - #[ver(V >= V13_0B4)] - unk_89f4_8: 1, - unk_89f4: 0, - hws1 <- Self::hw_shared1(cfg), - hws2 <- Self::hw_shared2(cfg, dyncfg), - hws3 <- Self::hw_shared3(cfg), - #[ver(V >= V13_0B4)] - idle_off_standby_timer: pwr.idle_off_standby_timer, - #[ver(V >= V13_0B4)] - unk_hws2_4: cfg.unk_hws2_4.map(Array::new).unwrap_or_default(), - #[ver(V >= V13_0B4)] - unk_hws2_24: cfg.unk_hws2_24, - unk_900c: 1, - #[ver(V >= V13_0B4)] - unk_9010_0: 1, - #[ver(V >= V13_0B4)] - unk_903c: 1, - #[ver(V < V13_0B4)] - unk_903c: 0, - fault_control: *module_parameters::fault_control.value(), - do_init: 1, - progress_check_interval_3d: 40, - progress_check_interval_ta: 10, - progress_check_interval_cl: 250, - #[ver(V >= V13_0B4)] - unk_1102c_0: 1, - #[ver(V >= V13_0B4)] - unk_1102c_4: 1, - #[ver(V >= V13_0B4)] - unk_1102c_8: 100, - #[ver(V >= V13_0B4)] - unk_1102c_c: 1, - idle_off_delay_ms: AtomicU32::new(pwr.idle_off_delay_ms), - fender_idle_off_delay_ms: pwr.fender_idle_off_delay_ms, - fw_early_wake_timeout_ms: pwr.fw_early_wake_timeout_ms, - cl_context_switch_timeout_ms: 40, - #[ver(V >= V13_0B4)] - cl_kill_timeout_ms: 50, - #[ver(V >= V13_0B4)] - unk_11edc: 0, - #[ver(V >= V13_0B4)] - unk_11efc: 0, - ..Zeroable::init_zeroed() - }) - .chain(|raw| { - for (i, pz) in self.dyncfg.pwr.power_zones.iter().enumerate() { - raw.power_zones[i].target = pz.target; - raw.power_zones[i].target_off = pz.target - pz.target_offset; - raw.power_zones[i].filter_tc = pz.filter_tc; - } - - if let Some(tab) = self.cfg.global_tab.as_ref() { - for (i, x) in tab.iter().enumerate() { - raw.unk_118ec[i] = *x; - } - raw.unk_118e8 = 1; - } - - if !dyncfg.hw_globals.is_empty() { - unsafe { - let mut matches: bool = true; - let sla = core::slice::from_raw_parts( - raw as *const raw::Globals::ver as *const u8, - core::mem::size_of::(), - ); - if sla.len() != dyncfg.hw_globals.len() { - matches = false; - dev_err!( - self.dev.as_ref(), - "!!! Globals size mismatch: {} {}", - sla.len(), - dyncfg.hw_globals.len(), - ); - } - for i in 0..core::cmp::min(sla.len(), dyncfg.hw_globals.len()) { - if sla[i] != dyncfg.hw_globals[i] { - matches = false; - dev_err!(self.dev.as_ref(), "!!! Globals first mismatch: {i}"); - break; - } - } - if matches { - dev_info!(self.dev.as_ref(), "!!! Globals match"); - } - } - } - - Ok(()) - }) - }) - } - - /// Create the RuntimePointers structure, which contains pointers to most of the other - /// structures including the ring buffer channels, statistics structures, and HwDataA/HwDataB. - fn runtime_pointers(&mut self) -> Result> { - let hwa = self.hwdata_a()?; - let hwb = self.hwdata_b()?; - - let mut buffer_mgr_ctl = gem::new_kernel_object(self.dev, 0x4000)?; - buffer_mgr_ctl.vmap()?.memset(0); - - GpuObject::new_init_prealloc( - self.alloc.private.alloc_object()?, - |_ptr| { - let alloc = &mut *self.alloc; - try_init!(RuntimePointers::ver { - stats <- { - let alloc = &mut *alloc; - try_init!(Stats::ver { - vtx: alloc.private.new_default::()?, - frag: alloc.private.new_init( - pin_init::init_zeroed::(), - |_inner, _ptr| { - try_init!(raw::GpuGlobalStatsFrag::ver { - total_cmds: 0, - unk_4: 0, - stats: Default::default(), - }) - } - )?, - comp: alloc.private.new_default::()?, - }) - }, - - hwdata_a: hwa, - unkptr_190: alloc.private.array_empty_tagged(0x80, b"I190")?, - unkptr_198: alloc.private.array_empty_tagged(0xc0, b"I198")?, - hwdata_b: hwb, - - unkptr_1b8: alloc.private.array_empty_tagged(0x1000, b"I1B8")?, - unkptr_1c0: alloc.private.array_empty_tagged(0x300, b"I1C0")?, - unkptr_1c8: alloc.private.array_empty_tagged(0x1000, b"I1C8")?, - - buffer_mgr_ctl, - buffer_mgr_ctl_low_mapping: None, - buffer_mgr_ctl_high_mapping: None, - }) - }, - |inner, _ptr| { - try_init!(raw::RuntimePointers::ver { - pipes: Default::default(), - device_control: Default::default(), - event: Default::default(), - fw_log: Default::default(), - ktrace: Default::default(), - stats: Default::default(), - - stats_vtx: inner.stats.vtx.gpu_pointer(), - stats_frag: inner.stats.frag.gpu_pointer(), - stats_comp: inner.stats.comp.gpu_pointer(), - - hwdata_a: inner.hwdata_a.gpu_pointer(), - unkptr_190: inner.unkptr_190.gpu_pointer(), - unkptr_198: inner.unkptr_198.gpu_pointer(), - hwdata_b: inner.hwdata_b.gpu_pointer(), - hwdata_b_2: inner.hwdata_b.gpu_pointer(), - - fwlog_buf: None, - - unkptr_1b8: inner.unkptr_1b8.gpu_pointer(), - - #[ver(G < G14X)] - unkptr_1c0: inner.unkptr_1c0.gpu_pointer(), - #[ver(G < G14X)] - unkptr_1c8: inner.unkptr_1c8.gpu_pointer(), - - buffer_mgr_ctl_gpu_addr: U64(gpu::IOVA_KERN_GPU_BUFMGR_LOW), - buffer_mgr_ctl_fw_addr: U64(gpu::IOVA_KERN_GPU_BUFMGR_HIGH), - - __pad0: Default::default(), - unk_160: U64(0), - unk_168: U64(0), - unk_1d0: 0, - unk_1d4: 0, - unk_1d8: Default::default(), - - __pad1: Default::default(), - gpu_scratch: raw::RuntimeScratch::ver { - unk_6b38: 0xff, - ..Default::default() - }, - }) - }, - ) - } - - /// Create the FwStatus structure, which is used to coordinate the firmware halt state between - /// the firmware and the driver. - fn fw_status(&mut self) -> Result> { - self.alloc - .shared - .new_object(Default::default(), |_inner| Default::default()) - } - - /// Create one UatLevelInfo structure, which describes one level of translation for the UAT MMU. - fn uat_level_info( - cfg: &'static hw::HwConfig, - index_shift: usize, - num_entries: usize, - ) -> raw::UatLevelInfo { - raw::UatLevelInfo { - index_shift: index_shift as _, - unk_1: 14, - unk_2: 14, - unk_3: 8, - unk_4: 0x4000, - num_entries: num_entries as _, - unk_8: U64(1), - unk_10: U64(((1u64 << cfg.uat_oas) - 1) & !(mmu::UAT_PGMSK as u64)), - index_mask: U64(((num_entries - 1) << index_shift) as u64), - } - } - - /// Build the top-level InitData object. - #[inline(never)] - pub(crate) fn build(&mut self) -> Result>> { - let runtime_pointers = self.runtime_pointers()?; - let globals = self.globals()?; - let fw_status = self.fw_status()?; - let shared_ro = &mut self.alloc.shared_ro; - - let obj = self.alloc.private.new_init( - try_init!(InitData::ver { - unk_buf: shared_ro.array_empty_tagged(0x4000, b"IDTA")?, - runtime_pointers, - globals, - fw_status, - }), - |inner, _ptr| { - let cfg = &self.cfg; - try_init!(raw::InitData::ver { - #[ver(V == V13_5 && G != G14X)] - ver_info: Array::new([0x6ba0, 0x1f28, 0x601, 0xb0]), - #[ver(V == V13_5 && G == G14X)] - ver_info: Array::new([0xb390, 0x70f8, 0x601, 0xb0]), - unk_buf: inner.unk_buf.gpu_pointer(), - unk_8: 0, - unk_c: 0, - runtime_pointers: inner.runtime_pointers.gpu_pointer(), - globals: inner.globals.gpu_pointer(), - fw_status: inner.fw_status.gpu_pointer(), - uat_page_size: 0x4000, - uat_page_bits: 14, - uat_num_levels: 3, - uat_level_info: Array::new([ - Self::uat_level_info(cfg, 36, 8), - Self::uat_level_info(cfg, 25, 2048), - Self::uat_level_info(cfg, 14, 2048), - ]), - __pad0: Default::default(), - host_mapped_fw_allocations: 1, - unk_ac: 0, - unk_b0: 0, - unk_b4: 0, - unk_b8: 0, - }) - }, - )?; - Ok(KBox::new(obj, GFP_KERNEL)?) - } -} diff --git a/drivers/gpu/drm/asahi-neo/mem.rs b/drivers/gpu/drm/asahi-neo/mem.rs deleted file mode 100644 index 0e99ab33fd3fc4..00000000000000 --- a/drivers/gpu/drm/asahi-neo/mem.rs +++ /dev/null @@ -1,211 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! ARM64 low level memory operations. -//! -//! This GPU uses CPU-side `tlbi` outer-shareable instructions to manage its TLBs. -//! Yes, really. Even though the VA address spaces are unrelated. -//! -//! Right now we pick our own ASIDs and don't coordinate with the CPU. This might result -//! in needless TLB shootdowns on the CPU side... TODO: fix this. - -use core::arch::asm; -use core::cmp::min; - -use kernel::page::PAGE_SIZE; - -use crate::debug::*; -use crate::mmu; - -type Asid = u8; - -/// Invalidate the entire GPU TLB. -#[inline(always)] -pub(crate) fn tlbi_all() { - // SAFETY: tlbi is always safe by definition - unsafe { - asm!(".arch armv8.4-a", "tlbi vmalle1os",); - } -} - -/// Invalidate all TLB entries for a given ASID. -#[inline(always)] -pub(crate) fn tlbi_asid(asid: Asid) { - if debug_enabled(DebugFlags::ConservativeTlbi) { - tlbi_all(); - sync(); - return; - } - - // SAFETY: tlbi is always safe by definition - unsafe { - asm!( - ".arch armv8.4-a", - "tlbi aside1os, {x}", - x = in(reg) ((asid as u64) << 48) - ); - } -} - -/// Invalidate a single page for a given ASID. -#[inline(always)] -pub(crate) fn tlbi_page(asid: Asid, va: usize) { - if debug_enabled(DebugFlags::ConservativeTlbi) { - tlbi_all(); - sync(); - return; - } - - let val: u64 = ((asid as u64) << 48) | ((va as u64 >> 12) & 0xffffffffffc); - // SAFETY: tlbi is always safe by definition - unsafe { - asm!( - ".arch armv8.4-a", - "tlbi vae1os, {x}", - x = in(reg) val - ); - } -} - -/// Invalidate a range of pages for a given ASID. -#[inline(always)] -pub(crate) fn tlbi_range(asid: Asid, va: usize, len: usize) { - if debug_enabled(DebugFlags::ConservativeTlbi) { - tlbi_all(); - sync(); - return; - } - - if len == 0 { - return; - } - - let start_pg = va >> mmu::UAT_PGBIT; - let end_pg = (va + len + mmu::UAT_PGMSK) >> mmu::UAT_PGBIT; - - const BADDR_MASK: u64 = (1 << 37) - 1; - let mut val: u64 = ((asid as u64) << 48) | (2 << 46) | (start_pg as u64 & BADDR_MASK); - let pages = end_pg - start_pg; - - // Guess? It's possible that the page count is in terms of 4K pages - // when the CPU is in 4K mode... - #[cfg(CONFIG_ARM64_4K_PAGES)] - let pages = 4 * pages; - - if pages == 1 { - tlbi_page(asid, va); - return; - } - - // Page count is always in units of 2 - let num = ((pages + 1) >> 1) as u64; - // base: 5 bits - // exp: 2 bits - // pages = (base + 1) << (5 * exp + 1) - // 0:00000 -> 2 pages = 2 << 0 - // 0:11111 -> 32 * 2 pages = 2 << 5 - // 1:00000 -> 1 * 32 * 2 pages = 2 << 5 - // 1:11111 -> 32 * 32 * 2 pages = 2 << 10 - // 2:00000 -> 1 * 32 * 32 * 2 pages = 2 << 10 - // 2:11111 -> 32 * 32 * 32 * 2 pages = 2 << 15 - // 3:00000 -> 1 * 32 * 32 * 32 * 2 pages = 2 << 15 - // 3:11111 -> 32 * 32 * 32 * 32 * 2 pages = 2 << 20 - let exp = min(3, (64 - num.leading_zeros()) / 5); - let bits = 5 * exp; - let mut base = (num + (1 << bits) - 1) >> bits; - - val |= (exp as u64) << 44; - - while base > 32 { - // SAFETY: tlbi is always safe by definition - unsafe { - asm!( - ".arch armv8.4-a", - "tlbi rvae1os, {x}", - x = in(reg) val | (31 << 39) - ); - } - base -= 32; - // Each full operation covers 32 units of 2 << bits GPU granules. - // Advance only the base-address field, preserving the ASID and scale. - let next = (val & BADDR_MASK) + (32u64 << (bits + 1)); - val = (val & !BADDR_MASK) | (next & BADDR_MASK); - } - - // SAFETY: tlbi is always safe by definition - unsafe { - asm!( - ".arch armv8.4-a", - "tlbi rvae1os, {x}", - x = in(reg) val | ((base - 1) << 39) - ); - } -} - -/// Range length from which [`tlbi_range_or_asid`] invalidates the whole ASID. -const TLBI_RANGE_MAX: usize = 0x200_0000; -/// Range length from which [`tlbi_range_or_asid`] counts in 1 MiB units (scale 1) instead of -/// 32 KiB units (scale 0). -const TLBI_RANGE_SCALE1_MIN: usize = 0x10_0000; -/// Translation granule field value for 16 KiB pages. -const TLBI_TG_16K: u64 = 2; -/// Width of the base address field of a range TLBI operand. -const TLBI_RANGE_BADDR_MASK: u64 = (1 << 38) - 1; - -/// Invalidate a range of pages for a given ASID with a single range operation. -/// -/// Unlike [`tlbi_range`], which issues as many range operations as needed, this invalidates -/// ranges that do not fit one operation (and ranges starting at VA 0) by ASID. Single pages use a -/// page operation, at the CPU page size. -#[inline(always)] -pub(crate) fn tlbi_range_or_asid(asid: Asid, va: usize, len: usize) { - if debug_enabled(DebugFlags::ConservativeTlbi) { - tlbi_all(); - sync(); - return; - } - - if len == 0 { - return; - } - - if va == 0 || len >= TLBI_RANGE_MAX { - tlbi_asid(asid); - return; - } - if len <= PAGE_SIZE { - tlbi_page(asid, va); - return; - } - - // The range covers (NUM + 1) << (5 * SCALE + 1) granules of 16 KiB. - let (scale, unit_shift) = if len >= TLBI_RANGE_SCALE1_MIN { - (1u64, 20) - } else { - (0u64, 15) - }; - let num = ((len - 1) >> unit_shift) as u64; - - let val = ((asid as u64) << 48) - | (TLBI_TG_16K << 46) - | (scale << 44) - | (num << 39) - | (((va >> mmu::UAT_PGBIT) as u64) & TLBI_RANGE_BADDR_MASK); - - // SAFETY: tlbi is always safe by definition - unsafe { - asm!( - ".arch armv8.4-a", - "tlbi rvae1os, {x}", - x = in(reg) val - ); - } -} - -/// Issue a memory barrier (`dsb sy`). -#[inline(always)] -pub(crate) fn sync() { - // SAFETY: Barriers are always safe - unsafe { - asm!("dsb sy"); - } -} diff --git a/drivers/gpu/drm/asahi-neo/microseq.rs b/drivers/gpu/drm/asahi-neo/microseq.rs deleted file mode 100644 index cbdb5de62e9218..00000000000000 --- a/drivers/gpu/drm/asahi-neo/microseq.rs +++ /dev/null @@ -1,63 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU Micro operation sequence builder -//! -//! As part of a single job submisssion to the GPU, the GPU firmware interprets a sequence of -//! commands that we call a "microsequence". These are responsible for setting up the job execution, -//! timestamping the process, waiting for completion, tearing up any resources, and signaling -//! completion to the driver via the event stamp mechanism. -//! -//! Although the microsequences used by the macOS driver are usually quite uniform and simple, the -//! firmware actually implements enough operations to make this interpreter Turing-complete (!). -//! Most of those aren't implemented yet, since we don't need them, but they could come in handy in -//! the future to do strange things or work around firmware bugs... -//! -//! This module simply implements a collection of microsequence operations that can be appended to -//! and later concatenated into one buffer, ready for firmware execution. - -use crate::fw::microseq; -pub(crate) use crate::fw::microseq::*; -use crate::fw::types::*; -use kernel::prelude::*; - -/// MicroSequence object type, which is just an opaque byte array. -pub(crate) type MicroSequence = GpuArray; - -/// MicroSequence builder. -pub(crate) struct Builder { - ops: KVec, -} - -impl Builder { - /// Create a new Builder object - pub(crate) fn new() -> Builder { - Builder { ops: KVec::new() } - } - - /// Get the relative offset from the current pointer to a given target offset. - /// - /// Used for relative jumps. - pub(crate) fn offset_to(&self, target: i32) -> i32 { - target - self.ops.len() as i32 - } - - /// Add an operation to the end of the sequence. - pub(crate) fn add(&mut self, op: T) -> Result { - let off = self.ops.len(); - let p: *const T = &op; - let p: *const u8 = p as *const u8; - // SAFETY: Microseq operations always have no padding bytes, so it is safe to - // access them as a byte slice. - let s: &[u8] = unsafe { core::slice::from_raw_parts(p, core::mem::size_of::()) }; - self.ops.extend_from_slice(s, GFP_KERNEL)?; - Ok(off as i32) - } - - /// Collect all submitted operations into a finalized GPU object. - pub(crate) fn build(self, alloc: &mut Allocator) -> Result { - let mut array = alloc.array_empty::(self.ops.len())?; - - array.as_mut_slice().clone_from_slice(self.ops.as_slice()); - Ok(array) - } -} diff --git a/drivers/gpu/drm/asahi-neo/mmu.rs b/drivers/gpu/drm/asahi-neo/mmu.rs deleted file mode 100644 index 86576e4d4dc15b..00000000000000 --- a/drivers/gpu/drm/asahi-neo/mmu.rs +++ /dev/null @@ -1,2675 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU UAT (MMU) management -//! -//! AGX GPUs use an MMU called the UAT, which is largely compatible with the ARM64 page table -//! format. This module manages the global MMU structures, including a shared handoff structure -//! that is used to coordinate VM management operations with the firmware, the TTBAT which points -//! to currently active GPU VM contexts, as well as the individual `Vm` operations to map and -//! unmap buffer objects into a single user or kernel address space. -//! -//! The actual page table management is in the `pt` module. -//! -//! G17 firmware does not take part in the handoff protocol. Instead, each of its two coprocessor -//! instances publishes the top-level entries it owns into its own kernel root while it boots, and -//! TTBAT contexts other than a VM's bind slot may name a VM's root. See [`UatFirmware`]. - -use core::fmt::Debug; -use core::mem::size_of; -use core::num::NonZeroUsize; -use core::ops::Range; -use core::sync::atomic::{ - fence, - AtomicBool, - AtomicU32, - AtomicU64, - AtomicU8, - Ordering, // -}; - -use core::cmp; - -use kernel::{ - addr::PhysicalAddr, - bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE, - c_str, - device, - drm_neo::{ - gem::shmem, - gpuvm, - mm, // - }, - error::Result, - io, - new_mutex, - prelude::*, - static_lock_class, - sync::{ - aref::ARef, - lock::{ - mutex::MutexBackend, - Guard, // - }, - Arc, - Mutex, // - }, - time::{ - delay::fsleep, - Delta, - Instant, - Monotonic, // - }, // -}; - -use crate::debug::*; -use crate::module_parameters; -use crate::no_debug; -use crate::{ - driver, - fw, - gem, - mem, - pgtable, - slotalloc, - util::RangeExt, // -}; - -// KernelMapping protection types -pub(crate) use crate::pgtable::Prot; -pub(crate) use pgtable::prot::*; -pub(crate) use pgtable::{ - UatPageTable, - UAT_PGBIT, - UAT_PGMSK, - UAT_PGSZ, - UAT_TOP_LEVEL_SHIFT, // -}; - -use pin_init; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Mmu; - -mod bind; -mod context; -mod shared; -mod lifetime; -mod residency; -pub(crate) use residency::{Gate as ResidencyGate, VmShrinker}; -pub(crate) use bind::PreparedUserBindBatch; -pub(crate) use lifetime::VmJobGuard; - -/// PPL magic number for the handoff region -const PPL_MAGIC: u64 = 0x4b1d000000000002; - -/// Number of supported context entries in the TTBAT -const UAT_NUM_CTX: usize = 64; - -/// Application contexts owned by independent logical queues. Contexts 0..4 -/// and 63 remain reserved for the firmware and device-global resources. -const EXECUTION_CONTEXT_MASK: u64 = ((1u64 << 63) - 1) & !((1u64 << 5) - 1); -/// Maximum simultaneously retained logical queue contexts. -pub(crate) const MAX_EXECUTION_CONTEXTS: u32 = EXECUTION_CONTEXT_MASK.count_ones(); -/// First context available for users -const UAT_USER_CTX_START: usize = 1; -/// Number of available user contexts -const UAT_USER_CTX: usize = UAT_NUM_CTX - UAT_USER_CTX_START; - -/// Lower/user base VA -pub(crate) const IOVA_USER_BASE: u64 = UAT_PGSZ as u64; - -/// Top-level page table entry of the upper half that holds the driver-managed kernel VA range. -const IOVA_KERN_TOP_LEVEL_INDEX: u64 = 2; - -/// TTBAT context that starts out as an alias of the kernel context (with its own ASID) with -/// [`UatFirmware::PublishedRoots`]. -const KERNEL_ALIAS_CTX: usize = 1; - -/// How the GPU firmware shares UAT management with the driver. -#[derive(Debug, Copy, Clone, PartialEq, Eq)] -pub(crate) enum UatFirmware { - /// One coprocessor that publishes its magic number in the handoff region, arbitrates the - /// handoff lock with the driver, and acknowledges cache flush requests (G13/G14). - Handoff, - /// Two coprocessor instances that never touch the handoff region (G17). - /// - /// Each instance has its own kernel root and writes the top-level entries below and including - /// the driver-managed kernel window into it while it boots: the lower ones are private to the - /// instance, and the kernel window entry points at a table that the bootloader reserved. The - /// primary instance uses the `pagetables` root, the secondary one the root at - /// `secondary_root_offset` in the same region. TTBAT context 1 starts as an alias of the kernel - /// context. - PublishedRoots { - /// Offset of the secondary instance's kernel root in the `pagetables` region. - secondary_root_offset: u64, - }, -} - -const TTBR_VALID: u64 = 0x1; // BIT(0) -const TTBR_ASID_SHIFT: usize = 48; - -/// UAT address space geometry of a given SoC. -#[derive(Debug, Copy, Clone)] -pub(crate) struct UatGeometry { - /// Input address bits translated by each page table root. - pub(crate) ias: u32, - /// Output (physical) address bits. - pub(crate) oas: u32, -} - -impl UatGeometry { - /// Lower/user top VA - const fn user_top(&self) -> u64 { - 1 << self.ias - } - - /// Lower/user VA range - pub(crate) const fn user_range(&self) -> Range { - IOVA_USER_BASE..self.user_top() - } - - /// Address of a special dummy page? - pub(crate) const fn unk_page(&self) -> u64 { - self.user_top() - 2 * UAT_PGSZ as u64 - } - - /// User VA range excluding the unk page - pub(crate) const fn user_usable_range(&self) -> Range { - IOVA_USER_BASE..self.unk_page() - } - - /// Upper/kernel base VA - const fn upper_base(&self) -> u64 { - !(self.user_top() - 1) - } - - /// Driver-managed kernel VA range - pub(crate) const fn kernel_range(&self) -> Range { - let start = self.upper_base() + (IOVA_KERN_TOP_LEVEL_INDEX << UAT_TOP_LEVEL_SHIFT); - start..(start + (1 << UAT_TOP_LEVEL_SHIFT)) - } - - /// Full kernel VA range - #[cfg(CONFIG_DEV_COREDUMP)] - const fn upper_range(&self) -> Range { - self.upper_base()..(!UAT_PGMSK as u64) - } -} - -/// A pre-allocated memory region for UAT management -struct UatRegion { - base: PhysicalAddr, - map: io::mem::Mem, -} - -/// SAFETY: It's safe to share UAT region records across threads. -unsafe impl Send for UatRegion {} -/// SAFETY: It's safe to share UAT region records across threads. -unsafe impl Sync for UatRegion {} - -/// Handoff region flush info structure -#[repr(C)] -struct FlushInfo { - state: AtomicU64, - addr: AtomicU64, - size: AtomicU64, -} - -/// UAT Handoff region layout -#[repr(C)] -struct Handoff { - magic_ap: AtomicU64, - magic_fw: AtomicU64, - - lock_ap: AtomicU8, - lock_fw: AtomicU8, - // Implicit padding: 2 bytes - turn: AtomicU32, - cur_slot: AtomicU32, - // Implicit padding: 4 bytes - flush: [FlushInfo; UAT_NUM_CTX + 1], - - unk2: AtomicU8, - // Implicit padding: 7 bytes - unk3: AtomicU64, -} - -const HANDOFF_SIZE: usize = size_of::(); - -/// One VM slot in the TTBAT -#[repr(C)] -struct SlotTTBS { - ttb0: AtomicU64, - ttb1: AtomicU64, -} - -const SLOTS_SIZE: usize = UAT_NUM_CTX * size_of::(); - -// We need at least page 0 (ttb0) -const PAGETABLES_SIZE: usize = UAT_PGSZ; - -/// Lock-free visibility of the mapping epoch. Writers run under the GPUVM -/// exec lock; nested mutations keep cached validation unavailable until the -/// outermost mutation has finished its leaf writes and invalidations. -struct MappingEpoch { - visible: AtomicU64, - pending: AtomicU64, - writers: AtomicU32, -} - -impl MappingEpoch { - fn new() -> Self { - Self { - visible: AtomicU64::new(0), - pending: AtomicU64::new(0), - writers: AtomicU32::new(0), - } - } - fn snapshot(&self) -> Option { - let value = self.visible.load(Ordering::Acquire); - (value != u64::MAX).then_some(value) - } -} - -#[must_use = "retain until the mapping mutation and its invalidations finish"] -struct MappingMutation(Option>); - -impl Drop for MappingMutation { - fn drop(&mut self) { - if let Some(epoch) = self.0.as_ref() { - if epoch.writers.fetch_sub(1, Ordering::Relaxed) == 1 { - epoch - .visible - .store(epoch.pending.load(Ordering::Relaxed), Ordering::Release); - } - } - } -} - -/// Inner data for a Vm instance. This is reference-counted by the outer Vm object. -struct VmInner { - dev: driver::AsahiDevRef, - is_kernel: bool, - va_range: Range, - page_table: UatPageTable, - mapping_epoch: Option>, - epoch: Option, - mm: mm::Allocator<(), KernelMappingInner>, - uat_inner: Arc, - binding: Arc>, - id: u64, -} - -/// Slot binding-related inner data for a Vm instance. -struct VmBinding { - active_users: usize, - binding: Option>, - bind_token: Option, - ttb: u64, -} - -struct VmBoInner { - sgt: Option>, - sg_vec: Option)>>, -} - -/// Data associated with a VM <=> BO pairing -#[pin_data] -struct VmBo { - #[pin] - inner: Mutex, -} - -impl gpuvm::DriverGpuVmBo for VmBo { - fn new() -> impl PinInit { - pin_init!(VmBo { - inner <- new_mutex!(VmBoInner { - sgt: None, - sg_vec: None, - }, "VmBinding"), - }) - } -} - -/// Per-mapping metadata that survives backing eviction and GPUVA splits. -#[derive(Default, Clone)] -struct VmGpuVa { - prot: Option, -} -impl gpuvm::DriverGpuVa for VmGpuVa {} - -#[derive(Default)] -struct StepContext { - new_va: Option>>>, - prev_va: Option>>>, - next_va: Option>>>, - vm_bo: Option>>, - prot: Prot, -} - -impl gpuvm::DriverGpuVm for VmInner { - type Driver = driver::AsahiDriver; - type GpuVmBo = VmBo; - type GpuVa = VmGpuVa; - type StepContext = StepContext; - - fn step_map( - self: &mut gpuvm::UpdatingGpuVm<'_, Self>, - op: &mut gpuvm::OpMap, - ctx: &mut Self::StepContext, - ) -> Result { - let _mutation = self.mapping_mutation(); - let mut iova = op.addr(); - let mut left = op.range() as usize; - let mut offset = op.offset() as usize; - - let bo = ctx.vm_bo.as_ref().expect("step_map with no BO"); - - let one_page = op.flags().contains(gpuvm::GpuVaFlags::REPEAT); - - let mut do_map = |mut addr: usize, mut len: usize, offset: &mut usize| -> Result { - if left == 0 { - return Ok(false); - } - - if *offset > 0 { - let skip = len.min(*offset); - addr += skip; - len -= skip; - *offset -= skip; - } - if len == 0 { - return Ok(true); - } - assert!(*offset == 0); - - if one_page { - len = left; - } else { - len = len.min(left); - } - - mod_dev_dbg!( - self.dev, - "MMU: map: {:#x}:{:#x} -> {:#x} [OP={}]\n", - addr, - len, - iova, - one_page - ); - - if let Err(error) = self.page_table.map_pages( - iova..(iova + len as u64), - addr as PhysicalAddr, - ctx.prot, - one_page, - ) { - self.rollback_failed_map(op.addr(), op.range()); - return Err(error); - } - - left -= len; - iova += len as u64; - Ok(true) - }; - - let guard = bo.inner().inner.lock(); - if let Some(sg_vec) = guard.sg_vec.as_ref() { - let start_idx = sg_vec - .binary_search_by(|range| { - if range.0 > offset { - cmp::Ordering::Greater - } else if (range.0 + range.1.len()) <= offset { - cmp::Ordering::Less - } else { - cmp::Ordering::Equal - } - }) - .expect("sg_vec does not contain offset???"); - - offset -= sg_vec[start_idx].0 as usize; - - for cur in start_idx..sg_vec.len() { - let addr = sg_vec[cur].1.start as usize; - let len: usize = sg_vec[cur].1.len() as usize; - if do_map(addr, len, &mut offset)? == false { - break; - } - } - } else { - for range in guard.sgt.as_ref().expect("step_map with no SGT").iter() { - // TODO: proper DMA address/length handling - let addr = range.dma_address() as usize; - let len: usize = range.dma_len() as usize; - if do_map(addr, len, &mut offset)? == false { - break; - } - } - } - - let mut gpuva = ctx.new_va.take().expect("Multiple step_map calls"); - gpuva.as_mut().inner_mut().get_mut().prot = Some(ctx.prot); - - if op - .map_and_link_va( - self, - gpuva, - ctx.vm_bo.as_ref().expect("step_map with no BO"), - ) - .is_err() - { - dev_err!( - self.dev.as_ref(), - "map_and_link_va failed: {:#x} [{:#x}] -> {:#x}\n", - op.offset(), - op.range(), - op.addr() - ); - self.rollback_failed_map(op.addr(), op.range()); - return Err(EINVAL); - } - self.tlbi_contexts_naming_root(op.addr(), op.range() as usize); - Ok(()) - } - fn step_unmap( - self: &mut gpuvm::UpdatingGpuVm<'_, Self>, - op: &mut gpuvm::OpUnMap, - _ctx: &mut Self::StepContext, - ) -> Result { - let va = op.va().expect("step_unmap: missing VA"); - - self.unmap_gpuva(va)?; - - if op.unmap_and_unlink_va_defer().is_none() { - dev_err!(self.dev.as_ref(), "step_unmap: could not unlink gpuva"); - } - Ok(()) - } - fn step_remap( - self: &mut gpuvm::UpdatingGpuVm<'_, Self>, - op: &mut gpuvm::OpReMap, - vm_bo: &gpuvm::GpuVmBo, - ctx: &mut Self::StepContext, - ) -> Result { - let va = op.unmap().va().expect("No previous VA"); - let orig_addr = va.addr(); - let orig_range = va.range(); - let orig_prot = va.inner().prot.ok_or(EINVAL)?; - let orig_invalidated = va.flags().contains(gpuvm::GpuVaFlags::INVALIDATED); - - // Only unmap the hole between prev/next, if they exist - let unmap_start = if let Some(op) = op.prev_map() { - op.addr() + op.range() - } else { - orig_addr - }; - - let unmap_end = if let Some(op) = op.next_map() { - op.addr() - } else { - orig_addr + orig_range - }; - - mod_dev_dbg!( - self.dev, - "MMU: unmap for remap: {:#x}..{:#x} (from {:#x}:{:#x})\n", - unmap_start, - unmap_end, - orig_addr, - orig_range - ); - - let unmap_range = unmap_end - unmap_start; - - let _mutation = self.mapping_mutation(); - if orig_invalidated { - self.page_table.discard_partial_map(unmap_start..unmap_end)?; - } else { - self.page_table.unmap_pages(unmap_start..unmap_end)?; - } - - if let Some(asid) = self.slot() { - fence(Ordering::SeqCst); - self.tlbi_range(asid as u8, unmap_start, unmap_range as usize); - mod_dev_dbg!( - self.dev, - "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", - asid, - unmap_start, - unmap_range, - ); - mem::sync(); - } - - self.tlbi_contexts_naming_root(unmap_start, unmap_range as usize); - - if op.unmap().unmap_and_unlink_va_defer().is_none() { - dev_err!(self.dev.as_ref(), "step_unmap: could not unlink gpuva"); - } - - if let Some(prev_op) = op.prev_map() { - let mut prev_gpuva = ctx - .prev_va - .take() - .expect("Multiple step_remap calls with prev_op"); - prev_gpuva.as_mut().inner_mut().get_mut().prot = Some(orig_prot); - if prev_op.map_and_link_va(self, prev_gpuva, vm_bo).is_err() { - dev_err!(self.dev.as_ref(), "step_remap: could not relink prev gpuva"); - return Err(EINVAL); - } - } - - if let Some(next_op) = op.next_map() { - let mut next_gpuva = ctx - .next_va - .take() - .expect("Multiple step_remap calls with next_op"); - next_gpuva.as_mut().inner_mut().get_mut().prot = Some(orig_prot); - if next_op.map_and_link_va(self, next_gpuva, vm_bo).is_err() { - dev_err!(self.dev.as_ref(), "step_remap: could not relink next gpuva"); - return Err(EINVAL); - } - } - - Ok(()) - } -} - -impl VmInner { - /// Remove hardware leaves for one complete user GPUVA without allocating. - fn unmap_gpuva(&mut self, va: &gpuvm::GpuVa) -> Result { - mod_dev_dbg!(self.dev, "MMU: unmap: {:#x}:{:#x}\n", va.addr(), va.range()); - - let _mutation = self.mapping_mutation(); - if va.flags().contains(gpuvm::GpuVaFlags::INVALIDATED) { - self.page_table.discard_partial_map(va.addr()..(va.addr() + va.range()))?; - } else { - self.page_table.unmap_pages(va.addr()..(va.addr() + va.range()))?; - } - - if let Some(asid) = self.slot() { - fence(Ordering::SeqCst); - self.tlbi_range(asid as u8, va.addr(), va.range() as usize); - mod_dev_dbg!( - self.dev, - "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", - asid, - va.addr(), - va.range(), - ); - mem::sync(); - } - - self.tlbi_contexts_naming_root(va.addr(), va.range() as usize); - - Ok(()) - } - - /// A failed map has no GPUVA owner to retain its backing or remove its leaves. - /// GPUVM has already split or removed overlaps throughout this operation's range. - fn rollback_failed_map(&mut self, addr: u64, size: u64) { - // Include the current segment: map_pages can fail after writing some of its leaves. - // The checked GPUVM operation supplies an aligned range without overflow. - let result = self.page_table.discard_partial_map(addr..addr + size); - if self.uat_inner.firmware == UatFirmware::Handoff { - if let Some(asid) = self.slot() { - fence(Ordering::SeqCst); - self.tlbi_range(asid as u8, addr, size as usize); - mem::sync(); - } - } - self.tlbi_contexts_naming_root(addr, size as usize); - if let Err(error) = result { - dev_err!( - self.dev.as_ref(), - "Could not discard failed mapping: {:?}\n", - error - ); - } - } - - fn mapping_mutation(&mut self) -> MappingMutation { - let state = self.mapping_epoch.clone(); - if let Some(state) = state.as_ref() { - state.writers.fetch_add(1, Ordering::Relaxed); - state.visible.swap(u64::MAX, Ordering::AcqRel); - self.epoch = self.epoch.and_then(|epoch| epoch.checked_add(1)); - state - .pending - .store(self.epoch.unwrap_or(u64::MAX), Ordering::Relaxed); - } - MappingMutation(state) - } - - /// Returns the slot index, if this VM is bound. - fn slot(&self) -> Option { - if self.is_kernel { - // The GFX ASC does not care about the ASID. Pick an arbitrary one. - // TODO: This needs to be a persistently reserved ASID once we integrate - // with the ARM64 kernel ASID machinery to avoid overlap. - Some(0) - } else { - // We don't check whether we lost the slot, which could cause unnecessary - // invalidations against another Vm. However, this situation should be very - // rare (e.g. a Vm lost its slot, which means 63 other Vms bound in the - // interim, and then it gets killed / drops its mappings without doing any - // final rendering). Anything doing active maps/unmaps is probably also - // rendering and therefore likely bound. - self.binding - .lock() - .bind_token - .as_ref() - .map(|token| token.last_slot() + UAT_USER_CTX_START as u32) - } - } - - /// Returns the translation table base for this Vm - fn ttb(&self) -> u64 { - self.page_table.ttb() - } - - /// Invalidate a VA range of this Vm's address space in the TLB of one ASID. - fn tlbi_range(&self, asid: u8, iova: u64, size: usize) { - match self.uat_inner.firmware { - UatFirmware::Handoff => mem::tlbi_range(asid, iova as usize, size), - UatFirmware::PublishedRoots { .. } => { - mem::tlbi_range_or_asid(asid, iova as usize, size) - } - } - } - - /// Invalidate a VA range in every TTBAT context that names this Vm's root as its lower half. - /// - /// With [`UatFirmware::PublishedRoots`], contexts other than a Vm's bind slot name its root - /// (the kernel lower root sits in contexts 0 and 1 and has no slot at all), so a mapping - /// change must reach all of them. Other firmware only uses the bind slot, which the callers - /// already invalidate. - fn tlbi_contexts_naming_root(&self, iova: u64, size: usize) { - if self.uat_inner.firmware == UatFirmware::Handoff { - return; - } - - let contexts = self.uat_inner.lock().contexts_naming_root(self.ttb()); - self.tlbi_context_mask(iova, size, contexts); - } - - fn tlbi_context_mask(&self, iova: u64, size: usize, contexts: u64) { - if contexts == 0 { - return; - } - - fence(Ordering::SeqCst); - for ctx in 0..UAT_NUM_CTX { - if contexts & (1 << ctx) != 0 { - mem::tlbi_range_or_asid(ctx as u8, iova as usize, size); - } - } - mem::sync(); - } - - /// Map an `mm::Node` representing an mapping in VA space. - fn map_node(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { - let end = node - .start() - .checked_add(node.mapped_size as u64) - .ok_or(EOVERFLOW)?; - let _mutation = self.mapping_mutation(); - self.page_table.prepare_map(node.start()..end)?; - let result = self.map_node_prepared(node, prot); - if result.is_err() { - self.rollback_failed_map(node.start(), node.mapped_size as u64); - } - result - } - - /// Install only into the empty range reserved by `map_node`. - fn map_node_prepared(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { - let mut iova = node.start(); - let guard = node.bo.as_ref().ok_or(EINVAL)?.get()?.inner().inner.lock(); - let sgt = guard.sgt.as_ref().ok_or(EINVAL)?; - let mut offset = node.offset; - let mut left = node.mapped_size; - - for range in sgt.iter() { - if left == 0 { - break; - } - - // TODO: proper DMA address/length handling - let mut addr = range.dma_address() as usize; - let mut len: usize = range.dma_len() as usize; - - if (offset | addr | len | iova as usize) & UAT_PGMSK != 0 { - dev_err!( - self.dev.as_ref(), - "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", - addr, - len, - iova - ); - return Err(EINVAL); - } - - if offset > 0 { - let skip = len.min(offset); - addr += skip; - len -= skip; - offset -= skip; - } - - len = len.min(left); - - if len == 0 { - continue; - } - - mod_dev_dbg!( - self.dev, - "MMU: map: {:#x}:{:#x} -> {:#x}\n", - addr, - len, - iova - ); - - self.page_table.map_pages( - iova..(iova + len as u64), - addr as PhysicalAddr, - prot, - false, - )?; - - iova += len as u64; - left -= len; - } - - // A context naming this root may be live and hold a negative translation for the range. - self.tlbi_contexts_naming_root(node.start(), node.mapped_size); - Ok(()) - } -} - -/// Shared reference to a virtual memory address space ([`Vm`]). -#[derive(Clone)] -pub(crate) struct Vm { - id: u64, - mapping_epoch: Option>, - lifetime: Option>, - residency: Option>, - context_bindings: Option>, - shared_bindings: Option>, - status: Option>, - inner: ARef>, - dummy_obj: ARef, - binding: Arc>, -} -no_debug!(Vm); - -/// An application TTBAT identity retained by a logical queue and its jobs. -/// The root lease is dropped before the VM, so no page table can be freed -/// while an active context still names it. Installed firmware queues retain -/// this object through replacement retirement or processor stop; a consumed -/// closed-scheduler release may return its root identity while keeping storage. -pub(crate) struct ExecutionContext { - root: ExecutionRoot, - vm: Vm, -} - -struct ExecutionRoot { - released: AtomicBool, - inner: Arc, - id: u8, - generation: u8, - low: u64, - high: u64, -} - -impl ExecutionContext { - pub(crate) fn id(&self) -> u32 { - self.root.id.into() - } - pub(crate) fn generation(&self) -> u8 { - self.root.generation - } - pub(crate) fn vm(&self) -> &Vm { - &self.vm - } - - /// Called only after the closed scheduler's release was consumed and all - /// its commands retired. Installed storage may outlive this root lease. - pub(crate) fn release(&self) { - self.root.release(); - } - - pub(crate) fn is_current(&self) -> bool { - let inner = self.root.inner.lock(); - if self.root.released.load(Ordering::Acquire) { - return false; - } - let roots = &inner.ttbs()[usize::from(self.root.id)]; - roots.ttb0.load(Ordering::Acquire) == self.root.low - && roots.ttb1.load(Ordering::Acquire) == self.root.high - } -} - -impl ExecutionRoot { - fn release(&self) { - if self.released.swap(true, Ordering::AcqRel) { - return; - } - let id = usize::from(self.id); - let release = { - let inner = self.inner.lock(); - inner.handoff().lock(); - let roots = &inner.ttbs()[id]; - let low = roots.ttb0.load(Ordering::Acquire); - let high = roots.ttb1.load(Ordering::Acquire); - let owned = low == self.low && high == self.high; - if owned { - roots.ttb0.store(0, Ordering::Release); - roots.ttb1.store(0, Ordering::Release); - } - inner.handoff().unlock(); - owned || (low == 0 && high == 0) - }; - if release { - fence(Ordering::SeqCst); - mem::tlbi_asid(self.id); - mem::sync(); - self.inner.lock().execution_contexts &= !(1u64 << id); - } else { - pr_err!( - "MMU: application context {} changed owners while leased\n", - id - ); - } - } -} - -impl Drop for ExecutionRoot { - fn drop(&mut self) { - self.release(); - } -} - -/// Slot data for a [`Vm`] slot (nothing, we only care about the indices). -pub(crate) struct SlotInner(); - -impl slotalloc::SlotItem for SlotInner { - type Data = (); -} - -/// Represents a single user of a binding of a [`Vm`] to a slot. -/// -/// The number of users is counted, and the slot will be freed when it drops to 0. -#[derive(Debug)] -pub(crate) struct VmBind(Vm, u32); - -impl VmBind { - /// Returns the slot that this `Vm` is bound to. - pub(crate) fn slot(&self) -> u32 { - self.1 - } -} - -impl Drop for VmBind { - fn drop(&mut self) { - let mut binding = self.0.binding.lock(); - - assert_ne!(binding.active_users, 0); - binding.active_users -= 1; - mod_pr_debug!( - "MMU: slot {} active users {}\n", - self.1, - binding.active_users - ); - if binding.active_users == 0 { - binding.binding = None; - } - } -} - -impl Clone for VmBind { - fn clone(&self) -> VmBind { - let mut binding = self.0.binding.lock(); - - binding.active_users += 1; - mod_pr_debug!( - "MMU: slot {} active users {}\n", - self.1, - binding.active_users - ); - VmBind(self.0.clone(), self.1) - } -} - -/// A mapping BO reference released under the immediate-mode GPUVA lock. -/// Retain the owner until deferred cleanup completes, including its final BO. -struct MappingBo { - bo: Option>>, - owner: ARef>, -} -impl MappingBo { - fn new( - bo: ARef>, - owner: ARef>, - ) -> Self { - Self { - bo: Some(bo), - owner, - } - } - fn get(&self) -> Result<&gpuvm::GpuVmBo> { - self.bo.as_deref().ok_or(EINVAL) - } -} -impl Drop for MappingBo { - fn drop(&mut self) { - let Some(bo) = self.bo.take() else { - return; - }; - let raw = ARef::into_raw(bo); - // SAFETY: GpuVmBo is repr(C) with drm_gpuvm_bo first. into_raw - // transfers one live reference. Every Vm here uses IMMEDIATE_MODE; - // deferred put selects its GPUVA lock and retains cleanup ownership. - unsafe { - kernel::bindings::drm_gpuvm_bo_put_deferred( - raw.as_ptr().cast::(), - ) - }; - self.owner.bo_deferred_cleanup(); - } -} - -/// Inner data required for an object mapping into a [`Vm`]. -pub(crate) struct KernelMappingInner { - // Drop order matters: - // - Deferred-put the GpuVmBo under its immediate-mode GPUVA lock - // - Drop the GEM BO next, since BO free can take the resv lock itself - // - Drop the owner GpuVm last, since that again can take resv locks when the refcount drops to 0 - bo: Option, - _gem: Option, - owner: ARef>, - uat_inner: Arc, - prot: Prot, - offset: usize, - mapped_size: usize, -} - -/// Prepared CPU timestamp access; destruction happens after completion signalling. -pub(crate) struct TimestampMapping { - mapping: shmem::VMap, - offset: usize, -} - -impl TimestampMapping { - pub(crate) fn write(&self, value: u64) { - let map = self.mapping.get(); - for (index, byte) in value.to_le_bytes().into_iter().enumerate() { - // SAFETY: timestamp_mapping checked the complete eight-byte extent. - // The owned vmap pins its backing; byte atomics permit unaligned and - // overlapping timestamp destinations without mutable slice aliases. - let destination = unsafe { - &*map.as_mut_ptr().add(self.offset + index).cast::() - }; - destination.store(byte, Ordering::Relaxed); - } - } -} - -/// An object mapping into a [`Vm`], which reserves the address range from use by other mappings. -pub(crate) struct KernelMapping(mm::Node<(), KernelMappingInner>); - -impl KernelMapping { - /// Returns the IOVA base of this mapping - pub(crate) fn iova(&self) -> u64 { - self.0.start() - } - - /// Returns the size of this mapping in bytes - pub(crate) fn size(&self) -> usize { - self.0.mapped_size - } - - /// Returns the IOVA base of this mapping - pub(crate) fn iova_range(&self) -> Range { - self.0.start()..(self.0.start() + self.0.mapped_size as u64) - } - - /// Resolves a byte in this retained mapping without creating page tables. - pub(crate) fn translate_offset(&self, offset: usize) -> Result { - if offset >= self.size() { - return Err(EINVAL); - } - let address = self.iova().checked_add(offset as u64).ok_or(EOVERFLOW)?; - self.0.owner.exec_lock(None, false)?.page_table.translate_iova(address) - } - - /// Writes a userspace timestamp through the retained GEM mapping. Byte - /// stores support unaligned destinations and concurrent completion writes - /// without creating mutable aliases. The complete timestamp is not atomic. - /// Retain CPU access before this destination's completion fence is published. - pub(crate) fn timestamp_mapping(&self, offset: usize) -> Result { - use kernel::drm_neo::gem::BaseObject; - let gem = self.0._gem.as_ref().ok_or(EINVAL)?; - if offset.checked_add(size_of::()).ok_or(EOVERFLOW)? > self.size() { - return Err(ERANGE); - } - let start = self.0.offset.checked_add(offset).ok_or(EOVERFLOW)?; - let end = start.checked_add(size_of::()).ok_or(EOVERFLOW)?; - if end > gem.size() { - return Err(ERANGE); - } - Ok(TimestampMapping { mapping: gem.owned_vmap::()?, offset: start }) - } - - /// Gives a fresh kernel mapping a firmware-cached prefix, preserving its tail. - /// The result includes whether any page-table entry may have changed; after - /// mutation the caller must retain the mapping until processor stop even if - /// the readback fails. The original protection is retained for teardown. - /// - /// # Safety - /// The mapping must be unpublished. After mutation it must not be unmapped - /// while a firmware instance can retain a cached translation or contents. - pub(crate) unsafe fn set_fw_cached_prefix(&self, prefix: usize) -> (Result<[u64; 2]>, bool) { - const MAX_PAGES: usize = 32; - const ATTRIBUTE_SHIFT: u32 = 2; - const ATTRIBUTE_MASK: u64 = 7; - const UNCACHED: u64 = 2; - const CACHE_ATTRIBUTE_CHANGE: u64 = UNCACHED << ATTRIBUTE_SHIFT; - let mut mutated = false; - let result = (|| { - if self.0.uat_inner.firmware == UatFirmware::Handoff - || prefix == 0 - || prefix > self.size() - || (prefix | self.size()) & UAT_PGMSK != 0 - || self.size() > MAX_PAGES * UAT_PGSZ - { - return Err(EINVAL); - } - let mut owner = self.0.owner.exec_lock(None, false)?; - if !owner.is_kernel { - return Err(EINVAL); - } - let pages = self.size() / UAT_PGSZ; - let cached_pages = prefix / UAT_PGSZ; - let mut before = [0; MAX_PAGES]; - for (index, value) in before[..pages].iter_mut().enumerate() { - *value = owner - .page_table - .leaf_entry(self.iova() + (index * UAT_PGSZ) as u64)?; - if (*value >> ATTRIBUTE_SHIFT) & ATTRIBUTE_MASK != UNCACHED { - return Err(EINVAL); - } - } - let _mutation = owner.mapping_mutation(); - mutated = true; - owner.page_table.reprot_pages( - self.iova()..self.iova() + prefix as u64, - PROT_GPU_FW_PRIV_RW, - )?; - fence(Ordering::SeqCst); - mem::tlbi_all(); - mem::sync(); - let mut after = [0; MAX_PAGES]; - for (index, value) in after[..pages].iter_mut().enumerate() { - *value = owner - .page_table - .leaf_entry(self.iova() + (index * UAT_PGSZ) as u64)?; - let change = if index < cached_pages { - CACHE_ATTRIBUTE_CHANGE - } else { - 0 - }; - if before[index] ^ *value != change { - return Err(EFAULT); - } - } - Ok([after[0], after[cached_pages.min(pages - 1)]]) - })(); - (result, mutated) - } - - /// Remap a cached mapping as uncached, then synchronously flush that range of VAs from the - /// coprocessor cache. This is required to safely unmap cached/private mappings. - fn remap_uncached_and_flush(&mut self) { - let mut owner = self.0.owner.lock_inner(); - - mod_dev_dbg!( - owner.dev, - "MMU: remap as uncached {:#x}:{:#x}\n", - self.iova(), - self.size() - ); - - // Remap in-place as uncached. - // Do not try to unmap the guard page (-1) - let prot = self.0.prot.as_uncached(); - let _mutation = owner.mapping_mutation(); - if owner - .page_table - .reprot_pages(self.iova_range(), prot) - .is_err() - { - dev_err!( - owner.dev.as_ref(), - "MMU: remap {:#x}:{:#x} failed\n", - self.iova(), - self.size() - ); - } - fence(Ordering::SeqCst); - - // If we don't have (and have never had) a VM slot, just return - let slot = match owner.slot() { - None => return, - Some(slot) => slot, - }; - - let flush_slot = if owner.is_kernel { - // If this is a kernel mapping, always flush on index 64 - UAT_NUM_CTX as u32 - } else { - // Otherwise, check if this slot is the active one, otherwise return - // Also check that we actually own this slot - let ttb = owner.ttb() | TTBR_VALID | (slot as u64) << TTBR_ASID_SHIFT; - - let uat_inner = self.0.uat_inner.lock(); - uat_inner.handoff().lock(); - let cur_slot = uat_inner.handoff().current_slot(); - let ttb_cur = uat_inner.ttbs()[slot as usize].ttb0.load(Ordering::Relaxed); - uat_inner.handoff().unlock(); - if cur_slot == Some(slot) && ttb_cur == ttb { - slot - } else { - return; - } - }; - - // FIXME: There is a race here, though it'll probably never happen in practice. - // In theory, it's possible for the ASC to finish using our slot, whatever command - // it was processing to complete, the slot to be lost to another context, and the ASC - // to begin using it again with a different page table, thus faulting when it gets a - // flush request here. In practice, the chance of this happening is probably vanishingly - // small, as all 62 other slots would have to be recycled or in use before that slot can - // be reused, and the ASC using user contexts at all is very rare. - - // Still, the locking around UAT/Handoff/TTBs should probably be redesigned to better - // model the interactions with the firmware and avoid these races. - // Possibly TTB changes should be tied to slot locks: - - // Flush: - // - Can early check handoff here (no need to lock). - // If user slot and it doesn't match the active ASC slot, - // we can elide the flush as the ASC guarantees it flushes - // TLBs/caches when it switches context. We just need a - // barrier to ensure ordering. - // - Lock TTB slot - // - If user ctx: - // - Lock handoff AP-side - // - Lock handoff dekker - // - Check TTB & handoff cur ctx - // - Perform flush if necessary - // - This implies taking the fwring lock - // - // TTB change: - // - lock TTB slot - // - lock handoff AP-side - // - lock handoff dekker - // change TTB - - // Lock this flush slot, and write the range to it - let flush = self.0.uat_inner.lock_flush(flush_slot); - let pages = self.size() >> UAT_PGBIT; - flush.begin_flush(self.iova(), self.size() as u64); - if pages >= 0x10000 { - dev_err!( - owner.dev.as_ref(), - "MMU: Flush too big ({:#x} pages))\n", - pages - ); - } - - let cmd = fw::channels::FwCtlMsg { - addr: fw::types::U64(self.iova()), - unk_8: 0, - slot: flush_slot, - page_count: pages as u16, - unk_12: 2, // ? - }; - - // Tell the firmware to do a cache flush - let ret = match (*owner.dev).gpu.manager() { - Some(gpu) => gpu.fwctl(cmd), - None => Err(ENODEV), - }; - if let Err(e) = ret { - dev_err!( - owner.dev.as_ref(), - "MMU: ASC cache flush {:#x}:{:#x} failed (err: {:?})\n", - self.iova(), - self.size(), - e - ); - } - - // Finish the flush - flush.end_flush(); - - // Slot is unlocked here - } -} -no_debug!(KernelMapping); - -impl Drop for KernelMapping { - fn drop(&mut self) { - // This is the main unmap function for UAT mappings. - // The sequence of operations here is finicky, due to the interaction - // between cached GFX ASC mappings and the page tables. These mappings - // always have to be flushed from the cache before being unmapped. - - // For uncached mappings, just unmapping and flushing the TLB is sufficient. - - // For cached mappings, this is the required sequence: - // 1. Remap it as uncached - // 2. Flush the TLB range - // 3. If kernel VA mapping OR user VA mapping and handoff.current_slot() == slot: - // a. Take a lock for this slot - // b. Write the flush range to the right context slot in handoff area - // c. Issue a cache invalidation request via FwCtl queue - // d. Poll for completion via queue - // e. Check for completion flag in the handoff area - // f. Drop the lock - // 4. Unmap - // 5. Flush the TLB range again - - // Firmware that does not use the handoff protocol cannot be asked to flush its cache. - // Such firmware only caches kernel mappings that live until its coprocessors stop. - if self.0.prot.is_cached_noncoherent() && self.0.uat_inner.firmware == UatFirmware::Handoff - { - mod_pr_debug!( - "MMU: remap as uncached {:#x}:{:#x}\n", - self.iova(), - self.size() - ); - self.remap_uncached_and_flush(); - } - - // Only VM page tables/binding state are touched here. The shared - // reservation is sufficient and avoids fallible exec allocation in Drop. - let mut owner = self.0.owner.lock_inner(); - mod_dev_dbg!( - owner.dev, - "MMU: unmap {:#x}:{:#x}\n", - self.iova(), - self.size() - ); - - let _mutation = owner.mapping_mutation(); - if owner.page_table.unmap_pages(self.iova_range()).is_err() { - dev_err!( - owner.dev.as_ref(), - "MMU: unmap {:#x}:{:#x} failed\n", - self.iova(), - self.size() - ); - } - - if let Some(asid) = owner.slot() { - fence(Ordering::SeqCst); - owner.tlbi_range(asid as u8, self.iova(), self.size()); - mod_dev_dbg!( - owner.dev, - "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", - asid, - self.iova(), - self.size() - ); - mem::sync(); - } - - // The range returns to the VA allocator once this mapping is gone, so no context that - // names this root may keep a stale translation for it. - owner.tlbi_contexts_naming_root(self.iova(), self.size()); - } -} - -/// Shared UAT global data structures -struct UatShared { - kernel_ttb1: u64, - execution_contexts: u64, - context_generations: [u8; UAT_NUM_CTX], - /// Kernel root of the secondary firmware instance, if there is one. - secondary_ttb1: Option, - map_kernel_to_user: bool, - handoff_rgn: UatRegion, - ttbs_rgn: UatRegion, -} - -impl UatShared { - /// Returns the handoff region area - fn handoff(&self) -> &Handoff { - // SAFETY: pointer is non-null per the type invariant - unsafe { (self.handoff_rgn.map.ptr() as *mut Handoff).as_ref() }.unwrap() - } - - /// Returns the TTBAT area - fn ttbs(&self) -> &[SlotTTBS; UAT_NUM_CTX] { - // SAFETY: pointer is non-null per the type invariant - unsafe { (self.ttbs_rgn.map.ptr() as *mut [SlotTTBS; UAT_NUM_CTX]).as_ref() }.unwrap() - } - - /// Returns the mask of TTBAT contexts whose lower half is the root `ttb`. - fn contexts_naming_root(&self, ttb: u64) -> u64 { - self.ttbs() - .iter() - .enumerate() - .filter(|(ctx, slot)| slot.ttb0.load(Ordering::Acquire) == tagged_root(ttb, *ctx)) - .fold(0, |mask, (ctx, _)| mask | (1 << ctx)) - } -} - -/// Returns the TTBAT value naming root `ttb` in context `ctx`. -const fn tagged_root(ttb: u64, ctx: usize) -> u64 { - ttb | TTBR_VALID | (ctx as u64) << TTBR_ASID_SHIFT -} - -// SAFETY: Nothing here is unsafe to send across threads. -unsafe impl Send for UatShared {} - -/// Inner data for the top-level UAT instance. -#[pin_data] -struct UatInner { - firmware: UatFirmware, - #[pin] - shared: Mutex, - #[pin] - handoff_flush: [Mutex; UAT_NUM_CTX + 1], -} - -impl UatInner { - /// Take the lock on the shared data and return the guard. - fn lock(&self) -> Guard<'_, UatShared, MutexBackend> { - self.shared.lock() - } - - /// Take a lock on a handoff flush slot and return the guard. - fn lock_flush(&self, slot: u32) -> Guard<'_, HandoffFlush, MutexBackend> { - self.handoff_flush[slot as usize].lock() - } -} - -/// Top-level UAT manager object -pub(crate) struct Uat { - dev: driver::AsahiDevRef, - geometry: UatGeometry, - - inner: Arc, - slots: slotalloc::SlotAllocator, - - kernel_vm: Vm, - kernel_lower_vm: Vm, -} - -impl Handoff { - /// Lock the handoff region from firmware access - fn lock(&self) { - self.lock_ap.store(1, Ordering::Relaxed); - fence(Ordering::SeqCst); - - while self.lock_fw.load(Ordering::Relaxed) != 0 { - if self.turn.load(Ordering::Relaxed) != 0 { - self.lock_ap.store(0, Ordering::Relaxed); - while self.turn.load(Ordering::Relaxed) != 0 {} - self.lock_ap.store(1, Ordering::Relaxed); - fence(Ordering::SeqCst); - } - } - fence(Ordering::Acquire); - } - - /// Unlock the handoff region, allowing firmware access - fn unlock(&self) { - self.turn.store(1, Ordering::Relaxed); - self.lock_ap.store(0, Ordering::Release); - } - - /// Returns the current Vm slot mapped by the firmware for lower/unprivileged access, if any. - fn current_slot(&self) -> Option { - let slot = self.cur_slot.load(Ordering::Relaxed); - if slot == 0 || slot == u32::MAX { - None - } else { - Some(slot) - } - } - - /// Initialize the handoff region - fn init(&self, firmware: UatFirmware) -> Result { - self.magic_ap.store(PPL_MAGIC, Ordering::Relaxed); - self.cur_slot.store(0, Ordering::Relaxed); - self.unk3.store(0, Ordering::Relaxed); - fence(Ordering::SeqCst); - - if firmware == UatFirmware::Handoff { - self.wait_for_firmware()?; - } - - for i in 0..=UAT_NUM_CTX { - self.flush[i].state.store(0, Ordering::Relaxed); - self.flush[i].addr.store(0, Ordering::Relaxed); - self.flush[i].size.store(0, Ordering::Relaxed); - } - fence(Ordering::SeqCst); - Ok(()) - } - - /// Wait for the firmware to publish its half of the handoff region. - fn wait_for_firmware(&self) -> Result { - let start = Instant::::now(); - const TIMEOUT: Delta = Delta::from_millis(1000); - - self.lock(); - while start.elapsed() < TIMEOUT { - if self.magic_fw.load(Ordering::Relaxed) == PPL_MAGIC { - break; - } else { - self.unlock(); - fsleep(Delta::from_millis(10)); - self.lock(); - } - } - - if self.magic_fw.load(Ordering::Relaxed) != PPL_MAGIC { - self.unlock(); - pr_err!("Handoff: Failed to initialize (firmware not running?)\n"); - return Err(EIO); - } - - self.unlock(); - Ok(()) - } -} - -/// Represents a single flush info slot in the handoff region. -/// -/// # Invariants -/// The pointer is valid and there is no aliasing HandoffFlush instance. -struct HandoffFlush(*const FlushInfo); - -// SAFETY: These pointers are safe to send across threads. -unsafe impl Send for HandoffFlush {} - -impl HandoffFlush { - /// Set up a flush operation for the coprocessor - fn begin_flush(&self, start: u64, size: u64) { - // SAFETY: Per the type invariant, this is safe - let flush = unsafe { self.0.as_ref().unwrap() }; - - let state = flush.state.load(Ordering::Relaxed); - if state != 0 { - pr_err!("Handoff: expected flush state 0, got {}\n", state); - } - flush.addr.store(start, Ordering::Relaxed); - flush.size.store(size, Ordering::Relaxed); - flush.state.store(1, Ordering::Relaxed); - } - - /// Complete a flush operation for the coprocessor - fn end_flush(&self) { - // SAFETY: Per the type invariant, this is safe - let flush = unsafe { self.0.as_ref().unwrap() }; - let state = flush.state.load(Ordering::Relaxed); - if state != 2 { - pr_err!("Handoff: expected flush state 2, got {}\n", state); - } - flush.state.store(0, Ordering::Relaxed); - } -} - -impl Vm { - /// Create a new virtual memory address space - fn new( - dev: &driver::AsahiDevice, - uat_inner: Arc, - kernel_range: Range, - geometry: UatGeometry, - ttb: Option, - id: u64, - ) -> Result { - let dummy_obj = gem::new_kernel_object(dev, UAT_PGSZ)?; - let is_kernel = ttb.is_some(); - - let page_table = if let Some(ttb) = ttb { - UatPageTable::new_with_ttb( - ttb, - geometry.kernel_range(), - geometry.ias, - geometry.oas, - uat_inner.firmware != UatFirmware::Handoff, - )? - } else { - UatPageTable::new(geometry.ias, geometry.oas)? - }; - - let (va_range, gpuvm_range) = if is_kernel { - (geometry.kernel_range(), kernel_range.clone()) - } else { - let mut range = geometry.user_range(); - if uat_inner.firmware != UatFirmware::Handoff { - range.start = 0; - } - (range, geometry.user_usable_range()) - }; - - let mm = mm::Allocator::new(va_range.start, va_range.range(), ())?; - - let binding = Arc::pin_init( - new_mutex!( - VmBinding { - binding: None, - bind_token: None, - active_users: 0, - ttb: page_table.ttb(), - }, - "VmBinding", - ), - GFP_KERNEL, - )?; - - let mapping_epoch = if uat_inner.firmware != UatFirmware::Handoff { - Some(Arc::new(MappingEpoch::new(), GFP_KERNEL)?) - } else { - None - }; - let binding_clone = binding.clone(); - Ok(Vm { - id, - mapping_epoch: mapping_epoch.clone(), - lifetime: None, - residency: None, - context_bindings: None, - shared_bindings: None, - status: None, - dummy_obj: dummy_obj.gem.clone(), - inner: gpuvm::GpuVm::new( - c_str!("Asahi::GpuVm"), - // TODO: should we using DRM_GPUVM_RESV_PROTECTED as well? - drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE, - dev, - dummy_obj.gem.clone(), - gpuvm_range, - kernel_range, - init!(VmInner { - dev: dev.into(), - va_range, - is_kernel, - page_table, - mapping_epoch, - epoch: Some(0), - mm, - uat_inner, - binding: binding_clone, - id, - }), - )?, - binding, - }) - } - - /// Attach a status before sharing a newly created user VM with queues. - pub(crate) fn with_status(mut self) -> Result { - self.status = Some(Arc::pin_init(crate::g17::status::VmStatus::new(), GFP_KERNEL)?); - self.lifetime = Some(lifetime::VmLifetime::new()?); - self.residency = Some(residency::Gate::new()?); - self.context_bindings = Some(context::ContextBindings::new()?); - self.shared_bindings = Some(shared::SharedBindings::new()?); - Ok(self) - } - - /// Status shared by every queue using this VM, if enabled by its GPU. - pub(crate) fn status(&self) -> Option<&Arc> { - self.status.as_ref() - } - - fn covers_range_locked( - inner: &mut VmInner, - address: u64, - size: u64, - read: bool, - write: bool, - ) -> bool { - if size == 0 { - return false; - } - let Some(end) = address.checked_add(size) else { - return false; - }; - let mask = UAT_PGMSK as u64; - let start = address & !mask; - let end = if end & mask == 0 { - end - } else { - let Some(end) = (end | mask).checked_add(1) else { - return false; - }; - end - }; - if start < inner.va_range.start || end > inner.va_range.end { - return false; - } - inner - .page_table - .covers_range(start..end, read, write) - .unwrap_or(false) - } - - pub(crate) fn covers_range(&self, address: u64, size: u64, read: bool, write: bool) -> bool { - let Ok(mut inner) = self.inner.exec_lock(None, false) else { - return false; - }; - Self::covers_range_locked(&mut inner, address, size, read, write) - } - - /// Checks all writable sparse blocks and their guard coverage under one - /// GPUVM lock. A fully mapped guard is not a valid sparse allocation. - pub(crate) fn covers_sparse_blocks_with_guards( - &self, - addresses: &[u64], - block_size: u64, - stride: u64, - ) -> bool { - if stride <= block_size { - return false; - } - let Ok(mut inner) = self.inner.exec_lock(None, false) else { - return false; - }; - addresses.iter().all(|&address| { - let Some(guard) = address.checked_add(block_size) else { - return false; - }; - Self::covers_range_locked(&mut inner, address, block_size, true, true) - && !Self::covers_range_locked( - &mut inner, guard, stride - block_size, false, false, - ) - }) - } - - /// Checks all exact request tuples under one GPUVM exec lock. Access bit0 - /// requests reading and bit1 writing. The epoch names this same snapshot. - pub(crate) fn covers_ranges_batch(&self, ranges: &[(u64, u64, u8)]) -> (bool, Option) { - let Ok(mut inner) = self.inner.exec_lock(None, false) else { - return (false, None); - }; - let epoch = inner.epoch; - let covered = ranges.iter().all(|&(address, size, access)| { - Self::covers_range_locked(&mut inner, address, size, access & 1 != 0, access & 2 != 0) - }); - (covered, epoch) - } - - /// Returns None while a leaf mutation is in progress or after epoch overflow. - pub(crate) fn mapping_validation_epoch(&self) -> Option { - self.mapping_epoch.as_ref()?.snapshot() - } - - /// Get the translation table base for this Vm - fn ttb(&self) -> u64 { - self.binding.lock().ttb - } - - /// Map a GEM object (using its `SGTable`) into this Vm at a free address in a given range. - #[allow(clippy::too_many_arguments)] - pub(crate) fn map_in_range( - &self, - gem: &gem::Object, - object_range: Range, - alignment: u64, - range: Range, - prot: Prot, - guard: bool, - ) -> Result { - self.map_in_range_with_guard_size( - gem, - object_range, - alignment, - range, - prot, - if guard { UAT_PGSZ } else { 0 }, - ) - } - - /// Reserves a whole-page unmapped guard without adding it to the mapped backing. - pub(crate) fn map_in_range_with_guard_size( - &self, - gem: &gem::Object, - object_range: Range, - alignment: u64, - range: Range, - prot: Prot, - guard_size: usize, - ) -> Result { - use kernel::drm_neo::gem::BaseObject; - if guard_size % UAT_PGSZ != 0 { - return Err(EINVAL); - } - let size = object_range - .end - .checked_sub(object_range.start) - .ok_or(EINVAL)?; - if size == 0 || object_range.end > gem.size() { - return Err(EINVAL); - } - let reserved_size = size.checked_add(guard_size).ok_or(EOVERFLOW)?; - let _residency = self.enter_residency()?; - let sgt = gem.owned_sg_table()?; - let mut inner = self.inner.exec_lock(Some(gem), false)?; - let vm_bo = self.inner.obtain_bo(gem)?; - - let mut vm_bo_guard = vm_bo.inner().inner.lock(); - if vm_bo_guard.sgt.is_none() { - vm_bo_guard.sgt.replace(sgt); - } - core::mem::drop(vm_bo_guard); - - let mut vm_bo = Some(MappingBo::new(vm_bo, self.inner.clone())); - let uat_inner = inner.uat_inner.clone(); - // A failed reservation drops its payload while exec still owns GEM - // reservations. Attach BO/GEM references only after success, since - // their destructors may acquire the same reservation locks. - let result = inner.mm.insert_node_in_range( - KernelMappingInner { - owner: self.inner.clone(), - uat_inner, - prot, - bo: None, - _gem: None, - offset: object_range.start, - mapped_size: size, - }, - reserved_size as u64, - alignment, - 0, - range.start, - range.end, - mm::InsertMode::Best, - ); - let mut node = match result { - Ok(node) => node, - Err(error) => { - drop(inner); - return Err(error); - } - }; - { - let payload = node.as_mut().inner_mut(); - payload.bo = vm_bo.take(); - payload._gem = Some(gem::KernelMappingPin::new(gem)); - } - - let ret = inner.map_node(&node, prot); - // Drop the exec_lock first, so that if map_node failed the - // KernelMappingInner destructur does not deadlock. - core::mem::drop(inner); - ret?; - Ok(KernelMapping(node)) - } - - /// Map a GEM object into this Vm at a specific address. - #[allow(clippy::too_many_arguments)] - pub(crate) fn map_at( - &self, - addr: u64, - size: usize, - gem: ARef, - prot: Prot, - guard: bool, - ) -> Result { - let reserved_size = size - .checked_add(if guard { UAT_PGSZ } else { 0 }) - .ok_or(EOVERFLOW)?; - let _residency = self.enter_residency()?; - let sgt = gem.owned_sg_table()?; - let mut inner = self.inner.exec_lock(Some(&gem), false)?; - - let vm_bo = self.inner.obtain_bo(&gem)?; - - let mut vm_bo_guard = vm_bo.inner().inner.lock(); - if vm_bo_guard.sgt.is_none() { - vm_bo_guard.sgt.replace(sgt); - } - core::mem::drop(vm_bo_guard); - - let mut vm_bo = Some(MappingBo::new(vm_bo, self.inner.clone())); - let uat_inner = inner.uat_inner.clone(); - // Keep reservation-locking destructors out of the allocator payload - // until reservation succeeds. - let result = inner.mm.reserve_node( - KernelMappingInner { - owner: self.inner.clone(), - uat_inner, - prot, - bo: None, - _gem: None, - offset: 0, - mapped_size: size, - }, - addr, - reserved_size as u64, - 0, - ); - let mut node = match result { - Ok(node) => node, - Err(error) => { - drop(inner); - return Err(error); - } - }; - { - let payload = node.as_mut().inner_mut(); - payload.bo = vm_bo.take(); - payload._gem = Some(gem::KernelMappingPin::new(&gem)); - } - - let ret = inner.map_node(&node, prot); - // Drop the exec_lock first, so that if map_node failed the - // KernelMappingInner destructur does not deadlock. - core::mem::drop(inner); - ret?; - Ok(KernelMapping(node)) - } - - /// Map a range of a GEM object into this Vm using GPUVM. - #[allow(clippy::too_many_arguments)] - pub(crate) fn bind_object( - &self, - gem: &gem::Object, - addr: u64, - size: u64, - offset: u64, - prot: Prot, - single_page: bool, - ) -> Result { - let _residency = self.enter_residency_metadata()?; - gem::validate_vm_binding(gem, self)?; - self.wait_for_user_map_admission()?; - // Mapping needs a complete context - let mut ctx = StepContext { - new_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), - prev_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), - next_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), - prot, - ..Default::default() - }; - - let vm_bo = self.inner.obtain_bo(gem)?; - { - let needs_sgt = vm_bo.inner().inner.lock().sgt.is_none(); - if needs_sgt { - // Pin without the BO mutex: alias construction can hold the GEM - // reservation while acquiring this mutex. - let sgt = gem.owned_sg_table()?; - let mut vm_bo_guard = vm_bo.inner().inner.lock(); - if vm_bo_guard.sgt.is_none() { - if vm_bo_guard.sg_vec.is_none() { - let mut sg_vec = KVVec::new(); - let mut offset = 0; - for range in sgt.iter() { - let addr = range.dma_address() as usize; - let len = range.dma_len() as usize; - sg_vec.push((offset, addr..(addr + len)), GFP_KERNEL)?; - offset += len; - } - vm_bo_guard.sg_vec.replace(sg_vec); - } - vm_bo_guard.sgt.replace(sgt); - } - // Release the mutex before a losing initializer drops its SG - // owner. The declaration order also preserves this on errors. - drop(vm_bo_guard); - } - } - - let mut inner = self.inner.exec_lock(Some(gem), true)?; - - // Preallocate the page tables, to fail early if we ENOMEM - inner.page_table.alloc_pages(addr..(addr + size))?; - - ctx.vm_bo = Some(vm_bo); - - if (addr | size | offset) & (UAT_PGMSK as u64) != 0 { - dev_err!( - inner.dev.as_ref(), - "MMU: Map step {:#x} [{:#x}] -> {:#x} is not page-aligned\n", - offset, - size, - addr - ); - return Err(EINVAL); - } - - let (flags, gem_range) = if single_page { - (gpuvm::GpuVaFlags::REPEAT, UAT_PGSZ as u32) - } else { - (gpuvm::GpuVaFlags::NONE, 0u32) - }; - - mod_dev_dbg!( - inner.dev, - "MMU: sm_map: {:#x} [{:#x}] -> {:#x}\n", - offset, - size, - addr - ); - let _commit = self.mapping_commit()?; - inner.sm_map(&mut ctx, addr, size, offset, gem_range, flags) - } - - /// Add a direct MMIO mapping to this Vm at a free address. - pub(crate) fn map_io( - &self, - iova: u64, - phys: usize, - size: usize, - prot: Prot, - ) -> Result { - let mut inner = self.inner.exec_lock(None, false)?; - - if (iova as usize | phys | size) & UAT_PGMSK != 0 { - dev_err!( - inner.dev.as_ref(), - "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", - phys, - size, - iova - ); - return Err(EINVAL); - } - - dev_info!( - inner.dev.as_ref(), - "MMU: IO map: {:#x}:{:#x} -> {:#x}\n", - phys, - size, - iova - ); - - let uat_inner = inner.uat_inner.clone(); - let node = inner.mm.reserve_node( - KernelMappingInner { - owner: self.inner.clone(), - uat_inner, - prot, - bo: None, - _gem: None, - offset: 0, - mapped_size: size, - }, - iova, - size as u64, - 0, - )?; - - let end = iova.checked_add(size as u64).ok_or(EOVERFLOW)?; - inner.page_table.prepare_map(iova..end)?; - let mutation = inner.mapping_mutation(); - let ret = inner.page_table.map_pages( - iova..end, - phys as PhysicalAddr, - prot, - false, - ); - if ret.is_err() { - inner.rollback_failed_map(iova, size as u64); - } - // Drop the exec_lock first, so that if map_node failed the - // KernelMappingInner destructur does not deadlock. - drop(mutation); - core::mem::drop(inner); - ret?; - Ok(KernelMapping(node)) - } - - /// Unmap everything in an address range. - pub(crate) fn unmap_range(&self, iova: u64, size: u64) -> Result { - if self.defer_unmap(iova, size)? { - return Ok(()); - } - self.unmap_range_commit(iova, size, true) - } - - fn unmap_prepared_cleanup(&self, unmap: &mut bind::PreparedUserUnmap) -> Result { - let mut inner = self.inner.lock_inner(); - // SAFETY: step_unmap/step_remap modify this VM's translations and - // immediate GPUVA metadata only, using the retained split nodes. They - // do not access external BO backing and defer all BO reference puts. - let result = unsafe { inner.sm_unmap_inner(&mut unmap.ctx, unmap.iova, unmap.size) }; - if result.is_ok() { - self.untrack_shared_range(unmap.iova, unmap.size); - } - drop(inner); - self.bo_deferred_cleanup(); - result - } - - fn unmap_range_commit(&self, iova: u64, size: u64, user: bool) -> Result { - let _residency = self.enter_retirement_residency(); - // Unmapping a range can only do a single split, so just preallocate - // the prev and next GpuVas - let mut ctx = StepContext { - prev_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), - next_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), - ..Default::default() - }; - - let mut inner = self.inner.exec_lock(None, false)?; - - mod_dev_dbg!(inner.dev, "MMU: sm_unmap: {:#x}:{:#x}\n", iova, size); - let _commit = if user { - match self.mapping_commit() { - Ok(guard) => guard, - Err(error) => { - drop(inner); - if self.defer_unmap(iova, size)? { - return Ok(()); - } - return Err(error); - } - } - } else { - None - }; - inner.sm_unmap(&mut ctx, iova, size)?; - self.untrack_shared_range(iova, size); - Ok(()) - } - - fn drop_mappings_now(&self, gem: &gem::Object) -> Result { - let _residency = self.enter_retirement_residency(); - // Removing whole mappings only does unmaps, so no preallocated VAs - let mut ctx = Default::default(); - - let inner = match self.inner.exec_lock(Some(gem), false) { - Err(error) if error == ENOMEM => return self.drop_mappings_low_memory(gem), - result => result?, - }; - - if let Some(bo) = self.inner.find_bo(gem) { - mod_dev_dbg!(inner.dev, "MMU: bo_unmap\n"); - let result = self.inner.bo_unmap(&mut ctx, &bo); - if result.is_ok() { - // The close callback or deferred object list retains this GEM, - // so clearing its candidate cannot release its final reference. - self.untrack_context_object(gem); - self.untrack_shared_object(gem); - } - mod_dev_dbg!(inner.dev, "MMU: bo_unmap done\n"); - // BO release can drain unrelated deferred GEMs, including on an - // unmap error. Always release reservations before dropping it. - core::mem::drop(inner); - core::mem::drop(bo); - if result == Err(ENOMEM) { - return self.drop_mappings_low_memory(gem); - } - result?; - } - - Ok(()) - } - - /// Preserve the indexed fast path normally; memory pressure must not lose - /// accepted object cleanup just because an exec or operation list failed. - fn drop_mappings_low_memory(&self, gem: &gem::Object) -> Result { - let mut inner = self.inner.lock_inner(); - let result = inner.unmap_object(gem, VmInner::unmap_gpuva); - if result.is_ok() { - self.untrack_context_object(gem); - self.untrack_shared_object(gem); - } - drop(inner); - self.bo_deferred_cleanup(); - result - } - - /// Returns the dummy GEM object used to hold the shared DMA reservation locks - pub(crate) fn get_resv_obj(&self) -> ARef { - self.dummy_obj.clone() - } - - /// Check whether an object is external to this GpuVm - pub(crate) fn is_extobj(&self, gem: &gem::Object) -> bool { - self.inner.is_extobj(gem) - } - - /// Check whether an object is external to this GpuVm - pub(crate) fn bo_deferred_cleanup(&self) { - self.inner.bo_deferred_cleanup() - } -} - -impl Drop for VmInner { - fn drop(&mut self) { - let mut binding = self.binding.lock(); - assert_eq!(binding.active_users, 0); - - mod_pr_debug!( - "VmInner::Drop [{}]: bind_token={:?}\n", - self.id, - binding.bind_token - ); - - // Make sure this VM is not mapped to a TTB if it was - if let Some(token) = binding.bind_token.take() { - let idx = (token.last_slot() as usize) + UAT_USER_CTX_START; - let ttb = self.ttb() | TTBR_VALID | (idx as u64) << TTBR_ASID_SHIFT; - - let uat_inner = self.uat_inner.lock(); - uat_inner.handoff().lock(); - let handoff_cur = uat_inner.handoff().current_slot(); - let ttb_cur = uat_inner.ttbs()[idx].ttb0.load(Ordering::SeqCst); - let inval = ttb_cur == ttb; - if inval { - if handoff_cur == Some(idx as u32) { - pr_err!( - "VmInner::drop owning slot {}, but it is currently in use by the ASC?\n", - idx - ); - } - uat_inner.ttbs()[idx].ttb0.store(0, Ordering::SeqCst); - uat_inner.ttbs()[idx].ttb1.store(0, Ordering::SeqCst); - } - uat_inner.handoff().unlock(); - core::mem::drop(uat_inner); - - // In principle we dropped all the KernelMappings already, but we might as - // well play it safe and invalidate the whole ASID. - if inval { - mod_pr_debug!( - "VmInner::Drop [{}]: need inval for ASID {:#x}\n", - self.id, - idx - ); - mem::tlbi_asid(idx as u8); - mem::sync(); - } - } - } -} - -impl Uat { - /// Map a bootloader-preallocated memory region - fn map_region( - dev: &device::Device, - name: &CStr, - size: usize, - cached: bool, - ) -> Result { - let of_node = dev.of_node().ok_or(EINVAL)?; - let res = of_node.reserved_mem_region_to_resource_byname(name)?; - let base = res.start(); - let res_size = res.size().try_into()?; - - if size > res_size { - dev_err!( - dev, - "Region {} is too small (expected {}, got {})\n", - name, - size, - res_size - ); - return Err(ENOMEM); - } - - let flags = if cached { - io::mem::MemFlag::WB - } else { - io::mem::MemFlag::WC - }; - - // SAFETY: The safety of this operation hinges on the correctness of - // much of this file and also the `pgtable` module, so it is difficult - // to prove in a single safety comment. Such is life with raw GPU - // page table management... - let map = unsafe { io::mem::Mem::try_new(res, flags.into()) }.inspect_err(|_| { - dev_err!(dev, "Failed to remap {} mem resource\n", name); - })?; - - Ok(UatRegion { base, map }) - } - - /// Returns a reference to the global kernel (upper half) `Vm` - pub(crate) fn kernel_vm(&self) -> &Vm { - &self.kernel_vm - } - - /// Returns a reference to the local kernel (lower half) `Vm` - pub(crate) fn kernel_lower_vm(&self) -> &Vm { - &self.kernel_lower_vm - } - - /// Installed descriptor and bind contexts rooted in this user VM. A graph - /// constructor checks this before changing a retained pool's mappings. - pub(crate) fn vm_context_mask(&self, vm: &Vm) -> Result { - if !matches!(self.inner.firmware, UatFirmware::PublishedRoots { .. }) { - return Err(EINVAL); - } - let root = { - let owner = vm.inner.exec_lock(None, false)?; - if owner.is_kernel || !Arc::ptr_eq(&owner.uat_inner, &self.inner) { - return Err(EINVAL); - } - owner.ttb() - }; - Ok(self.inner.lock().contexts_naming_root(root)) - } - - /// Publish a completed alias set before its descriptors become visible. - /// Match actual roots, including independently allocated execution contexts. - pub(crate) fn flush_vm_contexts(&self, vm: &Vm) -> Result { - let contexts = self.vm_context_mask(vm)?; - if contexts == 0 { - return Err(ENOENT); - } - fence(Ordering::SeqCst); - for context in 0..UAT_NUM_CTX { - if contexts & (1u64 << context) != 0 { - mem::tlbi_asid(context as u8); - } - } - mem::sync(); - Ok(contexts) - } - - #[cfg(CONFIG_DEV_COREDUMP)] - pub(crate) fn dump_kernel_pages(&self) -> Result> { - let mut inner = self.kernel_vm.inner.exec_lock(None, false)?; - inner.page_table.dump_pages(self.geometry.upper_range()) - } - - /// Returns the base physical address of the TTBAT region. - pub(crate) fn ttb_base(&self) -> u64 { - let inner = self.inner.lock(); - - inner.ttbs_rgn.base - } - - /// Binds a `Vm` to a slot, preferring the last used one. - pub(crate) fn bind(&self, vm: &Vm) -> Result { - let mut binding = vm.binding.lock(); - - if binding.binding.is_none() { - assert_eq!(binding.active_users, 0); - - let isolation = *module_parameters::robust_isolation.value() != 0; - - self.slots.set_limit(if isolation { - NonZeroUsize::new(1) - } else { - None - }); - - let slot = self.slots.get(binding.bind_token)?; - if slot.changed() { - mod_pr_debug!("Vm Bind [{}]: bind_token={:?}\n", vm.id, slot.token(),); - let idx = (slot.slot() as usize) + UAT_USER_CTX_START; - let ttb = binding.ttb | TTBR_VALID | (idx as u64) << TTBR_ASID_SHIFT; - - let uat_inner = self.inner.lock(); - - let ttb1 = if uat_inner.map_kernel_to_user { - uat_inner.kernel_ttb1 | TTBR_VALID | (idx as u64) << TTBR_ASID_SHIFT - } else { - 0 - }; - - let ttbs = uat_inner.ttbs(); - uat_inner.handoff().lock(); - if uat_inner.handoff().current_slot() == Some(idx as u32) { - pr_err!( - "Vm::bind to slot {}, but it is currently in use by the ASC?\n", - idx - ); - } - ttbs[idx].ttb0.store(ttb, Ordering::Release); - ttbs[idx].ttb1.store(ttb1, Ordering::Release); - uat_inner.handoff().unlock(); - core::mem::drop(uat_inner); - - // Make sure all TLB entries from the previous owner of this ASID are gone - mem::tlbi_asid(idx as u8); - mem::sync(); - } - - binding.bind_token = Some(slot.token()); - binding.binding = Some(slot); - } - - binding.active_users += 1; - - let slot = binding.binding.as_ref().unwrap().slot() + UAT_USER_CTX_START as u32; - mod_pr_debug!("MMU: slot {} active users {}\n", slot, binding.active_users); - Ok(VmBind(vm.clone(), slot)) - } - - /// Allocates a stable application context without changing the VM bind slot. - /// Both roots and the generation remain unchanged for the lease's lifetime. - pub(crate) fn new_execution_context(&self, vm: &Vm) -> Result> { - if self.inner.firmware == UatFirmware::Handoff { - return Err(EINVAL); - } - { - let owner = vm.inner.exec_lock(None, false)?; - if owner.is_kernel - || !core::ptr::eq(Arc::as_ptr(&owner.uat_inner), Arc::as_ptr(&self.inner)) - { - return Err(EINVAL); - } - } - let (id, generation) = { - let mut inner = self.inner.lock(); - let free = !inner.execution_contexts & EXECUTION_CONTEXT_MASK; - if free == 0 { - return Err(ENOSPC); - } - let id = free.trailing_zeros() as usize; - inner.execution_contexts |= 1u64 << id; - inner.context_generations[id] = inner.context_generations[id].wrapping_add(1); - (id, inner.context_generations[id]) - }; - let low = tagged_root(vm.ttb(), id); - let high = tagged_root(self.kernel_vm.ttb(), id); - // Allocate the complete host owner before either TTBAT half is published. - // ExecutionRoot releases an empty claimed slot if allocation fails. - let context = Arc::new( - ExecutionContext { - root: ExecutionRoot { - released: AtomicBool::new(false), - inner: self.inner.clone(), - id: id as u8, - generation, - low, - high, - }, - vm: vm.clone(), - }, - GFP_KERNEL, - )?; - { - let inner = self.inner.lock(); - let roots = &inner.ttbs()[id]; - if roots.ttb0.load(Ordering::Acquire) != 0 || roots.ttb1.load(Ordering::Acquire) != 0 { - return Err(EBUSY); - } - roots.ttb0.store(low, Ordering::Release); - roots.ttb1.store(high, Ordering::Release); - } - fence(Ordering::SeqCst); - mem::tlbi_asid(id as u8); - mem::sync(); - Ok(context) - } - - /// Creates a new `Vm` linked to this UAT. - pub(crate) fn new_vm(&self, id: u64, kernel_range: Range) -> Result { - Vm::new( - &self.dev, - self.inner.clone(), - kernel_range, - self.geometry, - None, - id, - ) - } - - /// Returns the address space geometry of this UAT. - pub(crate) fn geometry(&self) -> UatGeometry { - self.geometry - } - - /// Adopts the top-level entries that the primary firmware instance published in the kernel - /// root while booting ([`UatFirmware::PublishedRoots`]). - /// - /// The entries below the kernel window belong to the firmware, and the kernel window entry - /// points at a table that the bootloader reserved, so the driver maps kernel objects beneath - /// it and never frees it. Entries above the kernel window do not belong to anyone and are - /// cleared. Must run after the primary instance booted and before anything is mapped into the - /// kernel `Vm`. - pub(crate) fn adopt_firmware_root_entries(&self) -> Result<[u64; 3]> { - let owned = IOVA_KERN_TOP_LEVEL_INDEX as usize + 1; - let mut snapshot = [0; 3]; - // SAFETY: The kernel root is the bootloader-reserved `pagetables` region, which outlives - // this `Uat`. - unsafe { - pgtable::with_root_entries(self.kernel_vm.ttb(), UAT_PGSZ / size_of::(), |ptes| { - for (value, pte) in snapshot.iter_mut().zip(ptes) { - *value = pte.load(Ordering::Relaxed); - } - for pte in &ptes[owned..] { - pte.store(0, Ordering::Relaxed); - } - }) - }?; - - fence(Ordering::SeqCst); - mem::tlbi_all(); - mem::sync(); - Ok(snapshot) - } - - /// Checks that allocation under the shared subtree did not replace a firmware-owned entry. - pub(crate) fn confirm_firmware_root_entries(&self, snapshot: [u64; 3]) -> Result { - // SAFETY: The root is a bootloader-reserved page retained for this UAT's lifetime. - let matches = unsafe { - pgtable::with_root_entries(self.kernel_vm.ttb(), snapshot.len(), |ptes| { - ptes.iter() - .zip(snapshot) - .all(|(pte, expected)| pte.load(Ordering::Relaxed) == expected) - }) - }?; - if !matches { - return Err(EIO); - } - Ok(()) - } - - /// Copies the kernel root's top-level entries from the kernel window up into the root of the - /// secondary firmware instance ([`UatFirmware::PublishedRoots`]), whose lower entries are - /// private to that instance. - pub(crate) fn mirror_secondary_root(&self) -> Result { - let secondary = self.inner.lock().secondary_ttb1.ok_or(EINVAL)?; - let count = UAT_PGSZ / size_of::(); - let first = IOVA_KERN_TOP_LEVEL_INDEX as usize; - - // SAFETY: Both roots are in the bootloader-reserved `pagetables` region, which outlives - // this `Uat`. - unsafe { - pgtable::with_root_entries(self.kernel_vm.ttb(), count, |primary| { - pgtable::with_root_entries(secondary, count, |secondary| { - for (src, dst) in primary.iter().zip(secondary).skip(first) { - let pte = src.load(Ordering::Relaxed); - if pte != 0 && dst.load(Ordering::Relaxed) != pte { - dst.store(pte, Ordering::Relaxed); - } - } - }) - }) - }??; - - fence(Ordering::SeqCst); - mem::tlbi_all(); - mem::sync(); - Ok(()) - } - - /// Activates the device-global render root in context 1 after the boot records retire. - /// Both contexts must still contain their original kernel roots. - pub(crate) fn activate_render_context(&self, vm: &Vm) -> Result { - let inner = self.inner.lock(); - inner.handoff().lock(); - let ttbs = inner.ttbs(); - let ready = [0, KERNEL_ALIAS_CTX].into_iter().all(|ctx| { - ttbs[ctx].ttb0.load(Ordering::Acquire) == tagged_root(self.kernel_lower_vm.ttb(), ctx) - && ttbs[ctx].ttb1.load(Ordering::Acquire) == tagged_root(self.kernel_vm.ttb(), ctx) - }); - if !ready { - inner.handoff().unlock(); - return Err(EIO); - } - ttbs[KERNEL_ALIAS_CTX] - .ttb0 - .store(tagged_root(vm.ttb(), KERNEL_ALIAS_CTX), Ordering::Release); - inner.handoff().unlock(); - core::mem::drop(inner); - fence(Ordering::SeqCst); - mem::tlbi_asid(KERNEL_ALIAS_CTX as u8); - mem::sync(); - Ok(()) - } - - /// Creates the reference-counted inner data for a new `Uat` instance. - #[inline(never)] - fn make_inner(dev: &driver::AsahiDevice, firmware: UatFirmware) -> Result> { - let handoff_rgn = Self::map_region(dev.as_ref(), c_str!("handoff"), HANDOFF_SIZE, true)?; - let ttbs_rgn = Self::map_region(dev.as_ref(), c_str!("ttbs"), SLOTS_SIZE, true)?; - - // SAFETY: The Handoff struct layout matches the firmware's view of memory at this address, - // and the region is at least large enough per the size specified above. - let handoff = unsafe { &(handoff_rgn.map.ptr() as *mut Handoff).as_ref().unwrap() }; - - dev_info!(dev.as_ref(), "MMU: Initializing kernel page table\n"); - - Arc::pin_init( - try_pin_init!(UatInner { - firmware, - handoff_flush <- pin_init::pin_init_array_from_fn(|i| { - new_mutex!(HandoffFlush(&handoff.flush[i]), "handoff_flush") - }), - shared <- new_mutex!( - UatShared { - kernel_ttb1: 0, - execution_contexts: !EXECUTION_CONTEXT_MASK, - context_generations: [0; UAT_NUM_CTX], - secondary_ttb1: None, - map_kernel_to_user: false, - handoff_rgn, - ttbs_rgn, - }, - "uat_shared" - ), - }), - GFP_KERNEL, - ) - } - - /// Creates a new `Uat` instance for the given address space geometry and firmware. - #[inline(never)] - pub(crate) fn new( - dev: &driver::AsahiDevice, - geometry: UatGeometry, - map_kernel_to_user: bool, - firmware: UatFirmware, - ) -> Result { - dev_info!(dev.as_ref(), "MMU: Initializing...\n"); - - let inner = Self::make_inner(dev, firmware)?; - - let of_node = dev.as_ref().of_node().ok_or(EINVAL)?; - let res = of_node.reserved_mem_region_to_resource_byname(c_str!("pagetables"))?; - let ttb1 = res.start(); - let ttb1size: usize = res.size().try_into()?; - - let (pagetables_size, secondary_ttb1) = match firmware { - UatFirmware::Handoff => (PAGETABLES_SIZE, None), - UatFirmware::PublishedRoots { - secondary_root_offset, - } => ( - secondary_root_offset as usize + UAT_PGSZ, - Some(ttb1 + secondary_root_offset), - ), - }; - if ttb1size < pagetables_size { - dev_err!(dev.as_ref(), "MMU: Pagetables region is too small\n"); - return Err(ENOMEM); - } - - dev_info!(dev.as_ref(), "MMU: Creating kernel page tables\n"); - let kernel_lower_vm = - Vm::new(dev, inner.clone(), geometry.user_range(), geometry, None, 1)?; - let kernel_vm = Vm::new( - dev, - inner.clone(), - geometry.kernel_range(), - geometry, - Some(ttb1), - 0, - )?; - - dev_info!(dev.as_ref(), "MMU: Kernel page tables created\n"); - - let ttb0 = kernel_lower_vm.ttb(); - - let uat = Self { - dev: dev.into(), - geometry, - kernel_vm, - kernel_lower_vm, - inner, - slots: slotalloc::SlotAllocator::new( - UAT_USER_CTX as u32, - (), - |_inner, _slot| Some(SlotInner()), - c_str!("Uat::SlotAllocator"), - static_lock_class!(), - static_lock_class!(), - )?, - }; - - let mut inner = uat.inner.lock(); - - inner.map_kernel_to_user = map_kernel_to_user; - inner.kernel_ttb1 = ttb1; - inner.secondary_ttb1 = secondary_ttb1; - - inner.handoff().init(firmware)?; - - dev_info!(dev.as_ref(), "MMU: Initializing TTBs\n"); - - inner.handoff().lock(); - - let ttbs = inner.ttbs(); - - ttbs[0].ttb0.store(ttb0 | TTBR_VALID, Ordering::SeqCst); - ttbs[0].ttb1.store(ttb1 | TTBR_VALID, Ordering::SeqCst); - - for ctx in &ttbs[1..] { - ctx.ttb0.store(0, Ordering::Relaxed); - ctx.ttb1.store(0, Ordering::Relaxed); - } - - if secondary_ttb1.is_some() { - let alias = &ttbs[KERNEL_ALIAS_CTX]; - alias - .ttb0 - .store(tagged_root(ttb0, KERNEL_ALIAS_CTX), Ordering::SeqCst); - alias - .ttb1 - .store(tagged_root(ttb1, KERNEL_ALIAS_CTX), Ordering::SeqCst); - } - - inner.handoff().unlock(); - - core::mem::drop(inner); - - dev_info!(dev.as_ref(), "MMU: initialized\n"); - - Ok(uat) - } -} - -impl Drop for Uat { - fn drop(&mut self) { - // Make sure we flush the TLBs - fence(Ordering::SeqCst); - mem::tlbi_all(); - mem::sync(); - } -} diff --git a/drivers/gpu/drm/asahi-neo/object.rs b/drivers/gpu/drm/asahi-neo/object.rs deleted file mode 100644 index 38a2268137effb..00000000000000 --- a/drivers/gpu/drm/asahi-neo/object.rs +++ /dev/null @@ -1,733 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Asahi GPU object model -//! -//! The AGX GPU includes a coprocessor that uses a large number of shared memory structures to -//! communicate with the driver. These structures contain GPU VA pointers to each other, which are -//! directly dereferenced by the firmware and are expected to always be valid for the usage -//! lifetime of the containing struct (which is an implicit contract, not explicitly managed). -//! Any faults cause an unrecoverable firmware crash, requiring a full system reboot. -//! -//! In order to manage this complexity safely, we implement a GPU object model using Rust's type -//! system to enforce GPU object lifetime relationships. GPU objects represent an allocated piece -//! of memory of a given type, mapped to the GPU (and usually also the CPU). On the CPU side, -//! these objects are associated with a pure Rust structure that contains the objects it depends -//! on (or references to them). This allows us to map Rust lifetimes into the GPU object model -//! system. Then, GPU VA pointers also inherit those lifetimes, which means the Rust borrow checker -//! can ensure that all pointers are assigned an address that is guaranteed to outlive the GPU -//! object it points to. -//! -//! Since the firmware object model does have self-referencing pointers (and there is of course no -//! underlying revocability mechanism to make it safe), we must have an escape hatch. GPU pointers -//! can be weak pointers, which do not enforce lifetimes. In those cases, it is the user's -//! responsibility to ensure that lifetime requirements are met. -//! -//! In other words, the model is necessarily leaky and there is no way to fully map Rust safety to -//! GPU firmware object safety. The goal of the model is to make it easy to model the lifetimes of -//! GPU objects and have the compiler help in avoiding mistakes, rather than to guarantee safety -//! 100% of the time as would be the case for CPU-side Rust code. - -// TODO: There is a fundamental soundness issue with sharing memory with the GPU (that even affects -// C code too). Since the GPU is free to mutate that memory at any time, normal reference invariants -// cannot be enforced on the CPU side. For example, the compiler could perform an optimization that -// assumes that a given memory location does not change between two reads, and causes UB otherwise, -// and then the GPU could mutate that memory out from under the CPU. -// -// For cases where we *expect* this to happen, we use atomic types, which avoid this issue. However, -// doing so for every single field of every type is a non-starter. Right now, there seems to be no -// good solution for this that does not come with significant performance or ergonomics downsides. -// -// In *practice* we are almost always only writing GPU memory, and only reading from atomics, so the -// chances of this actually triggering UB (e.g. a security issue that can be triggered from the GPU -// side) due to a compiler optimization are very slim. -// -// Further discussion: https://github.com/rust-lang/unsafe-code-guidelines/issues/152 - -use kernel::{ - error::code::*, - prelude::*, - sync::Arc, // -}; - -use core::fmt; -use core::fmt::Debug; -use core::fmt::Formatter; -use core::marker::PhantomData; -use core::mem::MaybeUninit; -use core::num::NonZeroU64; -use core::ops::{ - Deref, - DerefMut, - Index, - IndexMut, // -}; -use core::{mem, ptr, slice}; - -use crate::alloc::Allocation; -use crate::debug::*; -use crate::fw::types::Zeroable; -use crate::mmu; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Object; - -/// A GPU-side strong pointer, which is a 64-bit non-zero VA with an associated lifetime. -/// -/// In rare cases these pointers are not aligned, so this is `packed(1)`. -#[repr(C, packed(1))] -pub(crate) struct GpuPointer<'a, T: ?Sized>(NonZeroU64, PhantomData<&'a T>); - -impl<'a, T: ?Sized> GpuPointer<'a, T> { - /// Logical OR the pointer with an arbitrary `u64`. This is used when GPU struct fields contain - /// misc flag fields in the upper bits. The lifetime is retained. This is GPU-unsafe in - /// principle, but we assert that only non-implemented address bits are touched, which is safe - /// for pointers used by the GPU (not by firmware). - pub(crate) fn or(&self, other: u64) -> GpuPointer<'a, T> { - // This will fail for kernel-half pointers, which should not be ORed. - assert_eq!(self.0.get() & other, 0); - // Assert that we only touch the high bits. - assert_eq!(other & 0xffffffffff, 0); - GpuPointer(self.0 | other, PhantomData) - } - - /// Add an arbitrary offset to the pointer. This is not safe (from the GPU perspective), and - /// should only be used via the `inner_ptr` macro to get pointers to inner fields, hence we mark - /// it `unsafe` to discourage direct use. - /// - /// # Safety - /// Do not use directly, only via `inner_ptr`. - // NOTE: The third argument is a type inference hack. - pub(crate) unsafe fn offset(&self, off: usize, _: *const U) -> GpuPointer<'a, U> { - GpuPointer::<'a, U>( - NonZeroU64::new(self.0.get() + (off as u64)).unwrap(), - PhantomData, - ) - } -} - -impl<'a, T> GpuPointer<'a, T> { - /// Create a GPU pointer from a KernelMapping and an offset. - /// TODO: Change all GPU pointers to point to the raw types so size_of here is GPU-sound. - pub(crate) fn from_mapping( - mapping: &'a Arc, - offset: usize, - ) -> Result> { - let addr = mapping.iova().checked_add(offset as u64).ok_or(EINVAL)?; - let end = offset - .checked_add(core::mem::size_of::()) - .ok_or(EINVAL)?; - if end > mapping.size() { - Err(ERANGE) - } else { - Ok(Self(addr.try_into().unwrap(), PhantomData)) - } - } -} - -impl<'a, T: ?Sized> Debug for GpuPointer<'a, T> { - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - let val = self.0; - f.write_fmt(format_args!("{:#x} ({})", val, core::any::type_name::())) - } -} - -impl<'a, T: ?Sized> From> for u64 { - fn from(value: GpuPointer<'a, T>) -> Self { - value.0.get() - } -} - -/// Take a pointer to a sub-field within a structure pointed to by a GpuPointer, keeping the -/// lifetime. -#[macro_export] -macro_rules! inner_ptr { - ($gpuva:expr, $($f:tt)*) => ({ - // This mirrors kernel::offset_of(), except we use type inference to avoid having to know - // the type of the pointer explicitly. - fn uninit_from(_: GpuPointer<'_, T>) -> core::mem::MaybeUninit> { - core::mem::MaybeUninit::uninit() - } - let tmp = uninit_from($gpuva); - let outer = tmp.as_ptr(); - // SAFETY: The pointer is valid and aligned, just not initialised; `addr_of` ensures that - // we don't actually read from `outer` (which would be UB) nor create an intermediate - // reference. - let p: *const _ = unsafe { core::ptr::addr_of!((*outer).$($f)*) }; - let inner = p as *const u8; - // SAFETY: The two pointers are within the same allocation block. - let off = unsafe { inner.offset_from(outer as *const u8) }; - // SAFETY: The resulting pointer is guaranteed to point to valid memory within the outer - // object. - unsafe { $gpuva.offset(off.try_into().unwrap(), p) } - }) -} - -/// A GPU-side weak pointer, which is a 64-bit non-zero VA with no lifetime. -/// -/// In rare cases these pointers are not aligned, so this is `packed(1)`. -#[repr(C, packed(1))] -pub(crate) struct GpuWeakPointer(NonZeroU64, PhantomData<*const T>); - -/// SAFETY: GPU weak pointers are always safe to share between threads. -unsafe impl Send for GpuWeakPointer {} -/// SAFETY: GPU weak pointers are always safe to share between threads. -unsafe impl Sync for GpuWeakPointer {} - -// Weak pointers can be copied/cloned regardless of their target type. -impl Copy for GpuWeakPointer {} - -impl Clone for GpuWeakPointer { - fn clone(&self) -> Self { - *self - } -} - -impl GpuWeakPointer { - /// Add an arbitrary offset to the pointer. This is not safe (from the GPU perspective), and - /// should only be used via the `inner_weak_ptr` macro to get pointers to inner fields, hence we - /// mark it `unsafe` to discourage direct use. - /// - /// # Safety - /// Do not use directly, only via `inner_weak_ptr`. - // NOTE: The third argument is a type inference hack. - pub(crate) unsafe fn offset(&self, off: usize, _: *const U) -> GpuWeakPointer { - GpuWeakPointer::( - NonZeroU64::new(self.0.get() + (off as u64)).unwrap(), - PhantomData, - ) - } - - /// Upgrade a weak pointer into a strong pointer. This is not considered safe from the GPU - /// perspective. - /// - /// # Safety - /// The caller must ensure tht the data pointed to lives in the GPU at least as long as the - /// returned lifetime. - pub(crate) unsafe fn upgrade<'a>(&self) -> GpuPointer<'a, T> { - GpuPointer(self.0, PhantomData) - } -} - -impl From> for u64 { - fn from(value: GpuWeakPointer) -> Self { - value.0.get() - } -} - -impl Debug for GpuWeakPointer { - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - let val = self.0; - f.write_fmt(format_args!("{:#x} ({})", val, core::any::type_name::())) - } -} - -/// Take a pointer to a sub-field within a structure pointed to by a GpuWeakPointer. -#[macro_export] -macro_rules! inner_weak_ptr { - ($gpuva:expr, $($f:tt)*) => ({ - // See inner_ptr() - fn uninit_from(_: GpuWeakPointer) -> core::mem::MaybeUninit> { - core::mem::MaybeUninit::uninit() - } - let tmp = uninit_from($gpuva); - let outer = tmp.as_ptr(); - // SAFETY: The pointer is valid and aligned, just not initialised; `addr_of` ensures that - // we don't actually read from `outer` (which would be UB) nor create an intermediate - // reference. - let p: *const _ = unsafe { core::ptr::addr_of!((*outer).$($f)*) }; - let inner = p as *const u8; - // SAFETY: The two pointers are within the same allocation block. - let off = unsafe { inner.offset_from(outer as *const u8) }; - // SAFETY: The resulting pointer is guaranteed to point to valid memory within the outer - // object. - unsafe { $gpuva.offset(off.try_into().unwrap(), p) } - }) -} - -/// Types that implement this trait represent a GPU structure from the CPU side. -/// -/// The `Raw` type represents the actual raw structure definition on the GPU side. -/// -/// Types implementing [`GpuStruct`] must have fields owning any objects (or strong references -/// to them) that GPU pointers in the `Raw` structure point to. This mechanism is used to enforce -/// lifetimes. -pub(crate) trait GpuStruct: 'static { - /// The type of the GPU-side structure definition representing the firmware struct layout. - type Raw<'a>; -} - -/// An instance of a GPU object in memory. -/// -/// # Invariants -/// `raw` must point to a valid mapping of the `T::Raw` type associated with the `alloc` allocation. -/// `gpu_ptr` must be the GPU address of the same object. -pub(crate) struct GpuObject> { - raw: *mut T::Raw<'static>, - alloc: U, - gpu_ptr: GpuWeakPointer, - inner: KBox, -} - -impl> GpuObject { - /// Create a new GpuObject given an allocator and the inner data (a type implementing - /// GpuStruct). - /// - /// The caller passes a closure that constructs the `T::Raw` type given a reference to the - /// `GpuStruct`. This is the mechanism used to enforce lifetimes. - pub(crate) fn new( - alloc: U, - inner: T, - callback: impl for<'a> FnOnce(&'a T) -> T::Raw<'a>, - ) -> Result { - let size = mem::size_of::>(); - if size > 0x1000 { - dev_crit!( - alloc.device().as_ref(), - "Allocating {} of size {:#x}, with new, please use new_boxed!\n", - core::any::type_name::(), - size - ); - } - if alloc.size() < size { - return Err(ENOMEM); - } - let gpu_ptr = - GpuWeakPointer::(NonZeroU64::new(alloc.gpu_ptr()).ok_or(EINVAL)?, PhantomData); - mod_dev_dbg!( - alloc.device(), - "Allocating {} @ {:#x}\n", - core::any::type_name::(), - alloc.gpu_ptr() - ); - let p = alloc.ptr().ok_or(EINVAL)?.as_ptr() as *mut T::Raw<'static>; - let mut raw = callback(&inner); - // SAFETY: `p` is guaranteed to be valid per the Allocation invariant, and the type is - // identical to the type of `raw` other than the lifetime. - unsafe { p.copy_from(&mut raw as *mut _ as *mut u8 as *mut _, 1) }; - mem::forget(raw); - Ok(Self { - raw: p, - gpu_ptr, - alloc, - inner: KBox::new(inner, GFP_KERNEL)?, - }) - } - - /// Create a new GpuObject given an allocator and the boxed inner data (a type implementing - /// GpuStruct). - /// - /// The caller passes a closure that initializes the `T::Raw` type given a reference to the - /// `GpuStruct` and a `MaybeUninit`. This is intended to be used with the place!() - /// macro to avoid constructing the whole `T::Raw` object on the stack. - pub(crate) fn new_boxed( - alloc: U, - inner: KBox, - callback: impl for<'a> FnOnce( - &'a T, - &'a mut MaybeUninit>, - ) -> Result<&'a mut T::Raw<'a>>, - ) -> Result { - if alloc.size() < mem::size_of::>() { - return Err(ENOMEM); - } - let gpu_ptr = - GpuWeakPointer::(NonZeroU64::new(alloc.gpu_ptr()).ok_or(EINVAL)?, PhantomData); - mod_dev_dbg!( - alloc.device(), - "Allocating {} @ {:#x}\n", - core::any::type_name::(), - alloc.gpu_ptr() - ); - let p = alloc.ptr().ok_or(EINVAL)?.as_ptr() as *mut MaybeUninit>; - // SAFETY: `p` is guaranteed to be valid per the Allocation invariant. - let raw = callback(&inner, unsafe { &mut *p })?; - if p as *mut T::Raw<'_> != raw as *mut _ { - dev_err!( - alloc.device().as_ref(), - "Allocation callback returned a mismatched reference ({})\n", - core::any::type_name::(), - ); - return Err(EINVAL); - } - Ok(Self { - raw: p as *mut u8 as *mut T::Raw<'static>, - gpu_ptr, - alloc, - inner, - }) - } - - /// Create a new GpuObject given an allocator and the inner data (a type implementing - /// GpuStruct). - /// - /// The caller passes a closure that initializes the `T::Raw` type given a reference to the - /// `GpuStruct` and a `MaybeUninit`. This is intended to be used with the place!() - /// macro to avoid constructing the whole `T::Raw` object on the stack. - pub(crate) fn new_inplace( - alloc: U, - inner: T, - callback: impl for<'a> FnOnce( - &'a T, - &'a mut MaybeUninit>, - ) -> Result<&'a mut T::Raw<'a>>, - ) -> Result { - GpuObject::::new_boxed(alloc, KBox::new(inner, GFP_KERNEL)?, callback) - } - - /// Create a new GpuObject given an allocator and the boxed inner data (a type implementing - /// GpuStruct). - /// - /// The caller passes a closure that initializes the `T::Raw` type given a reference to the - /// `GpuStruct` and a `MaybeUninit`. This is intended to be used with the place!() - /// macro to avoid constructing the whole `T::Raw` object on the stack. - pub(crate) fn new_init_prealloc<'a, I: Init, R: PinInit, F>, E, F>( - alloc: U, - inner_init: impl FnOnce(GpuWeakPointer) -> I, - raw_init: impl FnOnce(&'a T, GpuWeakPointer) -> R, - ) -> Result - where - kernel::error::Error: core::convert::From, - kernel::error::Error: core::convert::From, - { - if alloc.size() < mem::size_of::>() { - return Err(ENOMEM); - } - let gpu_ptr = - GpuWeakPointer::(NonZeroU64::new(alloc.gpu_ptr()).ok_or(EINVAL)?, PhantomData); - mod_dev_dbg!( - alloc.device(), - "Allocating {} @ {:#x}\n", - core::any::type_name::(), - alloc.gpu_ptr() - ); - let inner = inner_init(gpu_ptr); - let p = alloc.ptr().ok_or(EINVAL)?.as_ptr() as *mut T::Raw<'_>; - let ret = Self { - raw: p as *mut u8 as *mut T::Raw<'static>, - gpu_ptr, - alloc, - inner: KBox::init(inner, GFP_KERNEL)?, - }; - let q = &*ret.inner as *const T; - // SAFETY: `p` is guaranteed to be valid per the Allocation invariant. - unsafe { raw_init(&*q, gpu_ptr).__pinned_init(p) }?; - Ok(ret) - } - - /// Returns the GPU VA of this object (as a raw [`NonZeroU64`]) - pub(crate) fn gpu_va(&self) -> NonZeroU64 { - self.gpu_ptr.0 - } - - /// Returns a strong GPU pointer to this object, with a lifetime. - pub(crate) fn gpu_pointer(&self) -> GpuPointer<'_, T> { - GpuPointer(self.gpu_ptr.0, PhantomData) - } - - /// Returns a weak GPU pointer to this object, with no lifetime. - pub(crate) fn weak_pointer(&self) -> GpuWeakPointer { - GpuWeakPointer(self.gpu_ptr.0, PhantomData) - } - - /// Perform a mutation to the inner `Raw` data given a user-supplied callback. - /// - /// The callback gets a mutable reference to the `GpuStruct` type. - pub(crate) fn with_mut( - &mut self, - callback: impl for<'a> FnOnce(&'a mut ::Raw<'a>, &'a mut T) -> RetVal, - ) -> RetVal { - // SAFETY: `self.raw` is valid per the type invariant, and the second half is just - // converting lifetimes. - unsafe { callback(&mut *self.raw, &mut *(&mut *self.inner as *mut _)) } - } - - /// Access the inner `Raw` data given a user-supplied callback. - /// - /// The callback gets a reference to the `GpuStruct` type. - pub(crate) fn with( - &self, - callback: impl for<'a> FnOnce(&'a ::Raw<'a>, &'a T) -> RetVal, - ) -> RetVal { - // SAFETY: `self.raw` is valid per the type invariant, and the second half is just - // converting lifetimes. - unsafe { callback(&*self.raw, &*(&*self.inner as *const _)) } - } -} - -impl> Deref for GpuObject { - type Target = T; - - fn deref(&self) -> &Self::Target { - &self.inner - } -} - -impl> DerefMut for GpuObject { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.inner - } -} - -impl> Debug for GpuObject -where - ::Raw<'static>: Debug, -{ - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - f.debug_struct(core::any::type_name::()) - // SAFETY: `self.raw` is valid per the type invariant. - .field("raw", &format_args!("{:#X?}", unsafe { &*self.raw })) - .field("inner", &format_args!("{:#X?}", &self.inner)) - .field("alloc", &format_args!("{:?}", &self.alloc)) - .finish() - } -} - -impl> GpuObject -where - for<'a> ::Raw<'a>: Default + Zeroable, -{ - /// Create a new GpuObject with default data. `T` must implement `Default` and `T::Raw` must - /// implement `Zeroable`, since the GPU-side memory is initialized by zeroing. - pub(crate) fn new_default(alloc: U) -> Result { - GpuObject::::new_inplace(alloc, Default::default(), |_inner, raw| { - // SAFETY: `raw` is valid here, and `T::Raw` implements `Zeroable`. - Ok(unsafe { - ptr::write_bytes(raw, 0, 1); - (*raw).assume_init_mut() - }) - }) - } -} - -impl> Drop for GpuObject { - fn drop(&mut self) { - mod_dev_dbg!( - self.alloc.device(), - "Dropping {} @ {:?}\n", - core::any::type_name::(), - self.gpu_pointer() - ); - } -} - -// SAFETY: GpuObjects are Send as long as the GpuStruct itself is Send -unsafe impl> Send for GpuObject {} -// SAFETY: GpuObjects are Send as long as the GpuStruct itself is Send -unsafe impl> Sync for GpuObject {} - -/// Trait used to erase the type of a GpuObject, used when we need to keep a list of heterogenous -/// objects around. -pub(crate) trait OpaqueGpuObject: Send + Sync { - fn gpu_va(&self) -> NonZeroU64; -} - -impl> OpaqueGpuObject for GpuObject { - fn gpu_va(&self) -> NonZeroU64 { - Self::gpu_va(self) - } -} - -/// An array of raw GPU objects that is only accessible to the GPU (no CPU-side mapping required). -/// -/// This must necessarily be uninitialized as far as the GPU is concerned, so it cannot be used -/// when initialization is required. -/// -/// # Invariants -/// -/// `alloc` is valid and at least as large as `len` times the size of one `T`. -/// `gpu_ptr` is valid and points to the allocation start. -pub(crate) struct GpuOnlyArray> { - len: usize, - alloc: U, - gpu_ptr: NonZeroU64, - _p: PhantomData, -} - -impl> GpuOnlyArray { - /// Allocate a new GPU-only array with the given length. - pub(crate) fn new(alloc: U, count: usize) -> Result> { - let bytes = count * mem::size_of::(); - let gpu_ptr = NonZeroU64::new(alloc.gpu_ptr()).ok_or(EINVAL)?; - if alloc.size() < bytes { - return Err(ENOMEM); - } - Ok(Self { - len: count, - alloc, - gpu_ptr, - _p: PhantomData, - }) - } - - /// Returns the GPU VA of this arraw (as a raw [`NonZeroU64`]) - pub(crate) fn gpu_va(&self) -> NonZeroU64 { - self.gpu_ptr - } - - /// Returns a strong GPU pointer to this array, with a lifetime. - pub(crate) fn gpu_pointer(&self) -> GpuPointer<'_, &'_ [T]> { - GpuPointer(self.gpu_ptr, PhantomData) - } - - /// Returns a weak GPU pointer to this array, with no lifetime. - pub(crate) fn weak_pointer(&self) -> GpuWeakPointer<[T]> { - GpuWeakPointer(self.gpu_ptr, PhantomData) - } - - /// Returns a pointer to an offset within the array (as a subslice). - pub(crate) fn gpu_offset_pointer(&self, offset: usize) -> GpuPointer<'_, &'_ [T]> { - if offset > self.len { - panic!("Index {} out of bounds (len: {})", offset, self.len); - } - GpuPointer( - NonZeroU64::new(self.gpu_ptr.get() + (offset * mem::size_of::()) as u64).unwrap(), - PhantomData, - ) - } - - /* Not used yet - /// Returns a weak pointer to an offset within the array (as a subslice). - pub(crate) fn weak_offset_pointer(&self, offset: usize) -> GpuWeakPointer<[T]> { - if offset > self.len { - panic!("Index {} out of bounds (len: {})", offset, self.len); - } - GpuWeakPointer( - NonZeroU64::new(self.gpu_ptr.get() + (offset * mem::size_of::()) as u64).unwrap(), - PhantomData, - ) - } - - /// Returns a pointer to an element within the array. - pub(crate) fn gpu_item_pointer(&self, index: usize) -> GpuPointer<'_, &'_ T> { - if index >= self.len { - panic!("Index {} out of bounds (len: {})", index, self.len); - } - GpuPointer( - NonZeroU64::new(self.gpu_ptr.get() + (index * mem::size_of::()) as u64).unwrap(), - PhantomData, - ) - } - */ - - /// Returns a weak pointer to an element within the array. - pub(crate) fn weak_item_pointer(&self, index: usize) -> GpuWeakPointer { - if index >= self.len { - panic!("Index {} out of bounds (len: {})", index, self.len); - } - GpuWeakPointer( - NonZeroU64::new(self.gpu_ptr.get() + (index * mem::size_of::()) as u64).unwrap(), - PhantomData, - ) - } - - /// Returns the length of the array. - pub(crate) fn len(&self) -> usize { - self.len - } -} - -impl> Debug for GpuOnlyArray { - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - f.debug_struct(core::any::type_name::()) - .field("len", &format_args!("{:#X?}", self.len())) - .finish() - } -} - -impl> Drop for GpuOnlyArray { - fn drop(&mut self) { - mod_dev_dbg!( - self.alloc.device(), - "Dropping {} @ {:?}\n", - core::any::type_name::(), - self.gpu_pointer() - ); - } -} - -/// An array of raw GPU objects that is also CPU-accessible. -/// -/// # Invariants -/// -/// `raw` is valid and points to the CPU-side view of the array (which must have one). -pub(crate) struct GpuArray> { - raw: *mut T, - array: GpuOnlyArray, -} - -impl> GpuArray { - /// Allocate a new GPU array, initializing each element to its default. - pub(crate) fn empty(alloc: U, count: usize) -> Result> { - let p = alloc.ptr().ok_or(EINVAL)?.as_ptr(); - let inner = GpuOnlyArray::new(alloc, count)?; - let mut pi = p; - for _i in 0..count { - // SAFETY: `pi` is valid per the Allocation type invariant, and GpuOnlyArray guarantees - // that it can never iterate beyond the buffer length. - unsafe { - pi.write(Default::default()); - pi = pi.add(1); - } - } - Ok(Self { - raw: p, - array: inner, - }) - } -} - -impl> GpuArray { - /// Get a slice view of the array contents. - pub(crate) fn as_slice(&self) -> &[T] { - // SAFETY: self.raw / self.len are valid per the type invariant - unsafe { slice::from_raw_parts(self.raw, self.len) } - } - - /// Get a mutable slice view of the array contents. - pub(crate) fn as_mut_slice(&mut self) -> &mut [T] { - // SAFETY: self.raw / self.len are valid per the type invariant - unsafe { slice::from_raw_parts_mut(self.raw, self.len) } - } -} - -impl> Deref for GpuArray { - type Target = GpuOnlyArray; - - fn deref(&self) -> &GpuOnlyArray { - &self.array - } -} - -impl> Index for GpuArray { - type Output = T; - - fn index(&self, index: usize) -> &T { - if index >= self.len { - panic!("Index {} out of bounds (len: {})", index, self.len); - } - // SAFETY: This is bounds checked above - unsafe { &*(self.raw.add(index)) } - } -} - -impl> IndexMut for GpuArray { - fn index_mut(&mut self, index: usize) -> &mut T { - if index >= self.len { - panic!("Index {} out of bounds (len: {})", index, self.len); - } - // SAFETY: This is bounds checked above - unsafe { &mut *(self.raw.add(index)) } - } -} - -// SAFETY: GpuArray are Send as long as the contained type itself is Send -unsafe impl> Send for GpuArray {} -// SAFETY: GpuArray are Sync as long as the contained type itself is Sync -unsafe impl> Sync for GpuArray {} - -impl> Debug for GpuArray { - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - f.debug_struct(core::any::type_name::()) - .field("array", &format_args!("{:#X?}", self.as_slice())) - .finish() - } -} diff --git a/drivers/gpu/drm/asahi-neo/pgtable.rs b/drivers/gpu/drm/asahi-neo/pgtable.rs deleted file mode 100644 index 03001892d98bb4..00000000000000 --- a/drivers/gpu/drm/asahi-neo/pgtable.rs +++ /dev/null @@ -1,892 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! UAT Page Table management -//! -//! AGX GPUs use an MMU called the UAT, which is largely compatible with the ARM64 page table -//! format. This module manages the actual page tables by allocating raw memory pages from -//! the kernel page allocator. - -use core::fmt::Debug; -use core::mem::size_of; -use core::ops::Range; -use core::sync::atomic::{ - AtomicU64, - Ordering, // -}; - -use kernel::{ - addr::PhysicalAddr, - error::Result, - page::Page, - prelude::*, // -}; -#[cfg(CONFIG_DEV_COREDUMP)] -use kernel::{ - types::Owned, - uapi::{ - PF_R, - PF_W, - PF_X, // - }, -}; - -use crate::debug::*; -use crate::util::align; - -const DEBUG_CLASS: DebugFlags = DebugFlags::PgTable; - -/// Number of bits in a page offset. -pub(crate) const UAT_PGBIT: usize = 14; -/// UAT page size. -pub(crate) const UAT_PGSZ: usize = 1 << UAT_PGBIT; -/// UAT page offset mask. -pub(crate) const UAT_PGMSK: usize = UAT_PGSZ - 1; - -type Pte = AtomicU64; - -const PTE_BIT: usize = 3; // log2(sizeof(Pte)) -const PTE_SIZE: usize = 1 << PTE_BIT; - -/// Number of PTEs per page. -const UAT_NPTE: usize = UAT_PGSZ / size_of::(); - -/// Number of address bits to address a level -const UAT_LVBIT: usize = UAT_PGBIT - PTE_BIT; -/// Number of entries per level -const UAT_LVSZ: usize = UAT_NPTE; -/// Mask of level bits -const UAT_LVMSK: u64 = (UAT_LVSZ - 1) as u64; - -const UAT_LEVELS: usize = 3; - -/// Number of address bits below the index of a top-level page table entry. -pub(crate) const UAT_TOP_LEVEL_SHIFT: usize = UAT_PGBIT + (UAT_LEVELS - 1) * UAT_LVBIT; - -/// Runs `f` on the first `count` top-level entries of the root page table at `ttb`. -/// -/// # Safety -/// -/// `ttb` must be the physical address of a page table root in RAM that stays allocated while `f` -/// runs. Other agents (the GPU firmware) may access the entries concurrently, which is why they -/// are only exposed as atomics. -pub(crate) unsafe fn with_root_entries( - ttb: PhysicalAddr, - count: usize, - f: impl FnOnce(&[AtomicU64]) -> R, -) -> Result { - if count > UAT_NPTE { - return Err(EINVAL); - } - // SAFETY: The caller guarantees that `ttb` is an allocated page in RAM. - let page = unsafe { Page::borrow_phys(&ttb) }.ok_or(EIO)?; - page.with_pointer_into_page(0, count * PTE_SIZE, |p| { - // SAFETY: with_pointer_into_page() ensures the pointer is valid for `count` PTEs, and page - // table entries are naturally aligned. - Ok(f(unsafe { - core::slice::from_raw_parts(p as *const AtomicU64, count) - })) - }) -} - -const PTE_TYPE_BITS: u64 = 3; -const PTE_TYPE_LEAF_TABLE: u64 = 3; - -const UAT_NON_GLOBAL: u64 = 1 << 11; -const UAT_AP_SHIFT: u32 = 6; -const UAT_AP_BITS: u64 = 3 << UAT_AP_SHIFT; -const UAT_HIGH_BITS_SHIFT: u32 = 52; -const UAT_HIGH_BITS: u64 = 0xfff << UAT_HIGH_BITS_SHIFT; -const UAT_MEMATTR_SHIFT: u32 = 2; -const UAT_MEMATTR_BITS: u64 = 7 << UAT_MEMATTR_SHIFT; - -const UAT_PROT_BITS: u64 = UAT_AP_BITS | UAT_MEMATTR_BITS | UAT_HIGH_BITS; - -const UAT_AF: u64 = 1 << 10; - -const MEMATTR_CACHED: u8 = 0; -const MEMATTR_DEV: u8 = 1; -const MEMATTR_UNCACHED: u8 = 2; - -const AP_FW_GPU: u8 = 0; -const AP_FW: u8 = 1; -const AP_GPU: u8 = 2; - -const HIGH_BITS_PXN: u16 = 1 << 1; -const HIGH_BITS_UXN: u16 = 1 << 2; -const HIGH_BITS_GPU_ACCESS: u16 = 1 << 3; - -#[cfg(CONFIG_DEV_COREDUMP)] -pub(crate) const PTE_ADDR_BITS: u64 = (!UAT_PGMSK as u64) & (!UAT_HIGH_BITS); - -#[derive(Debug, Copy, Clone)] -pub(crate) struct Prot { - memattr: u8, - ap: u8, - high_bits: u16, -} - -// Firmware + GPU access -const PROT_FW_GPU_NA: Prot = Prot::from_bits(AP_FW_GPU, 0, 0); -const _PROT_FW_GPU_RO: Prot = Prot::from_bits(AP_FW_GPU, 0, 1); -const _PROT_FW_GPU_WO: Prot = Prot::from_bits(AP_FW_GPU, 1, 0); -const PROT_FW_GPU_RW: Prot = Prot::from_bits(AP_FW_GPU, 1, 1); - -// Firmware only access -const PROT_FW_RO: Prot = Prot::from_bits(AP_FW, 0, 0); -const _PROT_FW_NA: Prot = Prot::from_bits(AP_FW, 0, 1); -const PROT_FW_RW: Prot = Prot::from_bits(AP_FW, 1, 0); -const PROT_FW_RW_GPU_RO: Prot = Prot::from_bits(AP_FW, 1, 1); - -// GPU only access -const PROT_GPU_RO: Prot = Prot::from_bits(AP_GPU, 0, 0); -const PROT_GPU_WO: Prot = Prot::from_bits(AP_GPU, 0, 1); -const PROT_GPU_RW: Prot = Prot::from_bits(AP_GPU, 1, 0); -const _PROT_GPU_NA: Prot = Prot::from_bits(AP_GPU, 1, 1); - -#[cfg(CONFIG_DEV_COREDUMP)] -const PF_RW: u32 = PF_R | PF_W; -#[cfg(CONFIG_DEV_COREDUMP)] -const PF_RX: u32 = PF_R | PF_X; - -// For crash dumps -#[cfg(CONFIG_DEV_COREDUMP)] -const PROT_TO_PERMS_FW: [[u32; 4]; 4] = [ - [0, 0, 0, PF_RW], - [0, PF_RW, 0, PF_RW], - [PF_RX, PF_RX, 0, PF_R], - [PF_RX, PF_RW, 0, PF_R], -]; -#[cfg(CONFIG_DEV_COREDUMP)] -const PROT_TO_PERMS_OS: [[u32; 4]; 4] = [ - [0, PF_R, PF_W, PF_RW], - [PF_R, 0, PF_RW, PF_RW], - [0, 0, 0, 0], - [0, 0, 0, 0], -]; - -pub(crate) mod prot { - pub(crate) use super::Prot; - use super::*; - - /// Firmware MMIO R/W - pub(crate) const PROT_FW_MMIO_RW: Prot = PROT_FW_RW.memattr(MEMATTR_DEV); - /// Firmware MMIO R/O - pub(crate) const PROT_FW_MMIO_RO: Prot = PROT_FW_RO.memattr(MEMATTR_DEV); - /// Firmware shared (uncached) RW - pub(crate) const PROT_FW_SHARED_RW: Prot = PROT_FW_RW.memattr(MEMATTR_UNCACHED); - /// Firmware shared (uncached) RO - pub(crate) const PROT_FW_SHARED_RO: Prot = PROT_FW_RO.memattr(MEMATTR_UNCACHED); - /// Firmware private (cached) RW - pub(crate) const PROT_FW_PRIV_RW: Prot = PROT_FW_RW.memattr(MEMATTR_CACHED); - /// Firmware/GPU shared (uncached) RW - pub(crate) const PROT_GPU_FW_SHARED_RW: Prot = PROT_FW_GPU_RW.memattr(MEMATTR_UNCACHED); - /// Firmware/GPU shared (private) RW - pub(crate) const PROT_GPU_FW_PRIV_RW: Prot = PROT_FW_GPU_RW.memattr(MEMATTR_CACHED); - /// Firmware-RW/GPU-RO shared (private) RW - pub(crate) const PROT_GPU_RO_FW_PRIV_RW: Prot = PROT_FW_RW_GPU_RO.memattr(MEMATTR_CACHED); - /// GPU shared/coherent RW - pub(crate) const PROT_GPU_SHARED_RW: Prot = PROT_GPU_RW.memattr(MEMATTR_UNCACHED); - /// GPU shared/coherent RO - pub(crate) const PROT_GPU_SHARED_RO: Prot = PROT_GPU_RO.memattr(MEMATTR_UNCACHED); - /// GPU shared/coherent WO - pub(crate) const PROT_GPU_SHARED_WO: Prot = PROT_GPU_WO.memattr(MEMATTR_UNCACHED); -} - -impl Prot { - const fn from_bits(ap: u8, uxn: u16, pxn: u16) -> Self { - assert!(uxn <= 1); - assert!(pxn <= 1); - assert!(ap <= 3); - - Prot { - high_bits: HIGH_BITS_GPU_ACCESS | (pxn * HIGH_BITS_PXN) | (uxn * HIGH_BITS_UXN), - memattr: 0, - ap, - } - } - - pub(crate) const fn from_pte(pte: u64) -> Self { - Prot { - high_bits: (pte >> UAT_HIGH_BITS_SHIFT) as u16, - ap: ((pte & UAT_AP_BITS) >> UAT_AP_SHIFT) as u8, - memattr: ((pte & UAT_MEMATTR_BITS) >> UAT_MEMATTR_SHIFT) as u8, - } - } - - #[cfg(CONFIG_DEV_COREDUMP)] - pub(crate) const fn elf_flags(&self) -> u32 { - let ap = (self.ap & 3) as usize; - let uxn = if self.high_bits & HIGH_BITS_UXN != 0 { - 1 - } else { - 0 - }; - let pxn = if self.high_bits & HIGH_BITS_PXN != 0 { - 1 - } else { - 0 - }; - let gpu = self.high_bits & HIGH_BITS_GPU_ACCESS != 0; - - // Format: - // [12 top bits of PTE] [12 bottom bits of PTE] [5 bits pad] [ELF RWX] - let mut perms = if gpu { - PROT_TO_PERMS_OS[ap][(uxn << 1) | pxn] - } else { - PROT_TO_PERMS_FW[ap][(uxn << 1) | pxn] - }; - - perms |= ((self.as_pte() >> 52) << 20) as u32; - perms |= ((self.as_pte() & 0xfff) << 8) as u32; - - perms - } - - const fn memattr(&self, memattr: u8) -> Self { - Self { memattr, ..*self } - } - - const fn as_pte(&self) -> u64 { - (self.ap as u64) << UAT_AP_SHIFT - | (self.high_bits as u64) << UAT_HIGH_BITS_SHIFT - | (self.memattr as u64) << UAT_MEMATTR_SHIFT - | UAT_AF - } - - /// Whether this leaf allows the requested GPU accesses. - pub(crate) const fn allows_gpu(&self, need_read: bool, need_write: bool) -> bool { - let uxn = self.high_bits & HIGH_BITS_UXN != 0; - let pxn = self.high_bits & HIGH_BITS_PXN != 0; - let (readable, writable) = match self.ap { - AP_FW_GPU => (pxn, uxn), - AP_FW => (uxn && pxn, false), - AP_GPU => (!pxn, uxn != pxn), - _ => (false, false), - }; - (!need_read || readable) && (!need_write || writable) - } - - pub(crate) const fn is_cached_noncoherent(&self) -> bool { - self.ap != AP_GPU && self.memattr == MEMATTR_CACHED - } - - pub(crate) const fn as_uncached(&self) -> Self { - self.memattr(MEMATTR_UNCACHED) - } -} - -impl Default for Prot { - fn default() -> Self { - PROT_FW_GPU_NA - } -} - -#[cfg(CONFIG_DEV_COREDUMP)] -pub(crate) struct DumpedPage { - pub(crate) iova: u64, - pub(crate) pte: u64, - pub(crate) data: Option>, -} - -pub(crate) struct UatPageTable { - ttb: PhysicalAddr, - ttb_owned: bool, - /// Tables below a root the driver did not allocate may belong to the bootloader or the - /// firmware, so only the tables recorded here are ever freed. `None` if the driver owns the - /// tables below the root, as with the handoff firmware interface. - driver_tables: Option>, - va_range: Range, - ias_mask: u64, - oas_mask: u64, -} - -impl UatPageTable { - pub(crate) fn new(ias: u32, oas: u32) -> Result { - mod_pr_debug!("UATPageTable::new: ias={} oas={}\n", ias, oas); - let ttb_page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?; - let ttb = Page::into_phys(ttb_page); - Ok(UatPageTable { - ttb, - ttb_owned: true, - driver_tables: None, - va_range: 0..(1u64 << ias), - ias_mask: (1u64 << ias) - 1, - oas_mask: (1u64 << oas) - 1, - }) - } - - /// Borrow a root; track individual child ownership only when other agents - /// can install tables below it. Handoff roots have driver-owned children. - pub(crate) fn new_with_ttb( - ttb: PhysicalAddr, - va_range: Range, - ias: u32, - oas: u32, - shared_tables: bool, - ) -> Result { - mod_pr_debug!( - "UATPageTable::new_with_ttb: ttb={:#x} range={:#x?} ias={} oas={}\n", - ttb, - va_range, - ias, - oas - ); - if ttb & (UAT_PGMSK as PhysicalAddr) != 0 { - return Err(EINVAL); - } - if (va_range.start | va_range.end) & (UAT_PGMSK as u64) != 0 { - return Err(EINVAL); - } - // SAFETY: The bootloader-managed root stays valid while this page table exists. - if unsafe { Page::borrow_phys(&ttb) }.is_none() { - pr_err!( - "UATPageTable::new_with_ttb: ttb at {:#x} is not mapped (DT using no-map?)\n", - ttb - ); - return Err(EIO); - } - - Ok(UatPageTable { - ttb, - ttb_owned: false, - driver_tables: shared_tables.then(KVec::new), - va_range, - ias_mask: (1u64 << ias) - 1, - oas_mask: (1u64 << oas) - 1, - }) - } - - pub(crate) fn ttb(&self) -> PhysicalAddr { - self.ttb - } - - fn with_pages( - &mut self, - iova_range: Range, - alloc: bool, - free: bool, - mut cb: F, - ) -> Result - where - F: FnMut(u64, &[Pte]) -> Result, - { - mod_pr_debug!( - "UATPageTable::with_pages: {:#x?} alloc={} free={}\n", - iova_range, - alloc, - free - ); - if (iova_range.start | iova_range.end) & (UAT_PGMSK as u64) != 0 { - pr_err!( - "UATPageTable::with_pages: iova range not aligned: {:#x?}\n", - iova_range - ); - return Err(EINVAL); - } - - if iova_range.is_empty() { - return Ok(()); - } - - let mut iova = iova_range.start & self.ias_mask; - let mut last_iova = iova; - // Handle the case where iova_range.end is just at the top boundary of the IAS - let end = ((iova_range.end - 1) & self.ias_mask) + 1; - - let mut pt_addr: [Option; UAT_LEVELS] = Default::default(); - // For every child table, retain the parent page and entry that names it. - // Children are detached before their parents are freed, so these borrowed - // parent pages remain valid while the descriptor is cleared. - let mut pt_parent: [Option<(PhysicalAddr, usize)>; UAT_LEVELS - 1] = Default::default(); - pt_addr[UAT_LEVELS - 1] = Some(self.ttb); - - 'outer: while iova < end { - mod_pr_debug!("UATPageTable::with_pages: iova={:#x}\n", iova); - let addr_diff = last_iova ^ iova; - // Detach leaves before intermediate tables. The old order freed a - // parent first and could leave its descriptor naming a freed child. - for level in 0..UAT_LEVELS - 1 { - // If the iova has changed at this level or above, invalidate the physaddr - if addr_diff & !((1 << (UAT_PGBIT + (level + 1) * UAT_LVBIT)) - 1) != 0 { - if let Some(phys) = pt_addr[level].take() { - if free { - mod_pr_debug!( - "UATPageTable::with_pages: free level {} {:#x?}\n", - level, - phys - ); - let parent = pt_parent[level].take(); - self.free_table(phys, parent)?; - } else { - pt_parent[level] = None; - } - mod_pr_debug!("UATPageTable::with_pages: invalidate level {}\n", level); - } - } - } - last_iova = iova; - for level in (0..UAT_LEVELS - 1).rev() { - // Fetch the page table base address for this level - if pt_addr[level].is_none() { - let phys = pt_addr[level + 1].unwrap(); - mod_pr_debug!( - "UATPageTable::with_pages: need level {}, parent phys {:#x}\n", - level, - phys - ); - let upidx = ((iova >> (UAT_PGBIT + (level + 1) * UAT_LVBIT) as u64) & UAT_LVMSK) - as usize; - // SAFETY: Page table addresses are either allocated by us, or - // firmware-managed and safe to borrow a struct page from. - let upt = unsafe { Page::borrow_phys_unchecked(&phys) }; - mod_pr_debug!("UATPageTable::with_pages: borrowed phys {:#x}\n", phys); - pt_addr[level] = - upt.with_pointer_into_page(upidx * PTE_SIZE, PTE_SIZE, |p| { - let uptep = p as *const _ as *const Pte; - // SAFETY: with_pointer_into_page() ensures the pointer is valid, - // and our index is aligned so it is safe to deref as an AtomicU64. - let upte = unsafe { &*uptep }; - let mut upte_val = upte.load(Ordering::Relaxed); - // Allocate if requested - if upte_val == 0 && alloc { - let pt_page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?; - mod_pr_debug!("UATPageTable::with_pages: alloc PT at {:#x}\n", pt_page.phys()); - if let Some(tables) = self.driver_tables.as_mut() { - tables.push(pt_page.phys(), GFP_KERNEL)?; - } - let pt_paddr = Page::into_phys(pt_page); - upte_val = pt_paddr | PTE_TYPE_LEAF_TABLE; - // Complete child-table initialization before a - // concurrent device walk can follow its parent. - // A later mapping/TLBI barrier cannot order an - // already-visible parent against earlier clears. - crate::mem::sync(); - upte.store(upte_val, Ordering::Relaxed); - } - if upte_val & PTE_TYPE_BITS == PTE_TYPE_LEAF_TABLE { - Ok(Some(upte_val & self.oas_mask & (!UAT_PGMSK as u64))) - } else if upte_val == 0 || (!alloc && !free) { - mod_pr_debug!("UATPageTable::with_pages: no level {}\n", level); - Ok(None) - } else { - pr_err!("UATPageTable::with_pages: Unexpected Table PTE value {:#x} at iova {:#x} index {} phys {:#x}\n", upte_val, - iova, level + 1, phys + ((upidx * PTE_SIZE) as PhysicalAddr)); - Ok(None) - } - })?; - if pt_addr[level].is_some() { - pt_parent[level] = Some((phys, upidx)); - } - mod_pr_debug!( - "UATPageTable::with_pages: level {} PT {:#x?}\n", - level, - pt_addr[level] - ); - } - // If we don't have a page table, skip this entire level - if pt_addr[level].is_none() { - let block = 1 << (UAT_PGBIT + UAT_LVBIT * (level + 1)); - let old = iova; - iova = align(iova + 1, block); - mod_pr_debug!( - "UATPageTable::with_pages: skip {:#x} {:#x} -> {:#x}\n", - block, - old, - iova - ); - continue 'outer; - } - } - - let idx = ((iova >> UAT_PGBIT as u64) & UAT_LVMSK) as usize; - let max_count = UAT_NPTE - idx; - let count = (((end - iova) >> UAT_PGBIT) as usize).min(max_count); - let phys = pt_addr[0].unwrap(); - mod_pr_debug!( - "UATPageTable::with_pages: leaf PT at {:#x} idx {:#x} count {:#x} iova {:#x}\n", - phys, - idx, - count, - iova - ); - // SAFETY: Page table addresses are either allocated by us, or - // firmware-managed and safe to borrow a struct page from. - let pt = unsafe { Page::borrow_phys_unchecked(&phys) }; - pt.with_pointer_into_page(idx * PTE_SIZE, count * PTE_SIZE, |p| { - let ptep = p as *const _ as *const Pte; - // SAFETY: We know this is a valid pointer to PTEs and the range is valid and - // checked by with_pointer_into_page(). - let ptes = unsafe { core::slice::from_raw_parts(ptep, count) }; - cb(iova, ptes)?; - Ok(()) - })?; - - let block = 1 << (UAT_PGBIT + UAT_LVBIT); - iova = align(iova + 1, block); - } - - if free { - // The final path is detached bottom-up for the same reason as the - // boundary transitions above. - for level in 0..UAT_LEVELS - 1 { - if let Some(phys) = pt_addr[level] { - mod_pr_debug!( - "UATPageTable::with_pages: free level {} {:#x?}\n", - level, - phys - ); - self.free_table(phys, pt_parent[level].take())?; - } - } - } - - Ok(()) - } - - /// Free a page table, unless it was not allocated by the driver. - fn free_table( - &mut self, - phys: PhysicalAddr, - parent: Option<(PhysicalAddr, usize)>, - ) -> Result { - let owned = if let Some(tables) = self.driver_tables.as_ref() { - match tables.iter().position(|&table| table == phys) { - Some(_) => true, - None => false, - } - } else { - true - }; - if !owned { - return Ok(()); - } - if let Some((parent_phys, parent_index)) = parent { - // SAFETY: The bottom-up traversal keeps the parent allocated while - // its child descriptor is detached. A changed descriptor indicates - // an ownership violation; fail closed instead of freeing a table - // that may still be reachable. - let page = unsafe { Page::borrow_phys(&parent_phys) }.ok_or(EIO)?; - page.with_pointer_into_page(parent_index * PTE_SIZE, PTE_SIZE, |p| { - let pte = unsafe { &*(p as *const AtomicU64) }; - let current = pte.load(Ordering::Acquire); - let expected = phys & self.oas_mask & (!UAT_PGMSK as u64); - if current & self.oas_mask & (!UAT_PGMSK as u64) != expected { - return Err(EIO); - } - pte.store(0, Ordering::Release); - Ok(()) - })?; - } - if let Some(tables) = self.driver_tables.as_mut() { - let index = tables - .iter() - .position(|&table| table == phys) - .ok_or(EIO)?; - tables.swap_remove(index); - } - // SAFETY: Page tables allocated by the driver always come from Page::into_phys(). Without - // `driver_tables`, every child table belongs to the driver. - unsafe { Page::from_phys(phys) }; - Ok(()) - } - - /// Checks every page, including holes skipped by the page-table walker. - pub(crate) fn covers_range( - &mut self, - range: Range, - read: bool, - write: bool, - ) -> Result { - if range.is_empty() || (range.start | range.end) & UAT_PGMSK as u64 != 0 { - return Ok(false); - } - let expected = (range.end - range.start) >> UAT_PGBIT; - let mut visited = 0u64; - let mut permitted = true; - self.with_pages(range, false, false, |_, ptes| { - visited = visited.checked_add(ptes.len() as u64).ok_or(EOVERFLOW)?; - for pte in ptes { - let value = pte.load(Ordering::Acquire); - if value & PTE_TYPE_BITS != PTE_TYPE_LEAF_TABLE - || !Prot::from_pte(value).allows_gpu(read, write) - { - permitted = false; - } - } - Ok(()) - })?; - Ok(expected == visited && permitted) - } - - /// Resolves an existing leaf without creating page tables. - pub(crate) fn translate_iova(&mut self, iova: u64) -> Result { - let entry = self.leaf_entry(iova & !(UAT_PGMSK as u64))?; - Ok((entry & self.oas_mask & !(UAT_PGMSK as u64)) | (iova & UAT_PGMSK as u64)) - } - - /// Reads one mapped leaf entry without allocating missing tables. - pub(crate) fn leaf_entry(&mut self, iova: u64) -> Result { - if iova & UAT_PGMSK as u64 != 0 { - return Err(EINVAL); - } - let end = iova.checked_add(UAT_PGSZ as u64).ok_or(EINVAL)?; - let mut value = 0; - self.with_pages(iova..end, false, false, |_, ptes| { - if let Some(pte) = ptes.first() { - value = pte.load(Ordering::Relaxed); - } - Ok(()) - })?; - if value & PTE_TYPE_BITS != PTE_TYPE_LEAF_TABLE { - return Err(EFAULT); - } - Ok(value) - } - - pub(crate) fn alloc_pages(&mut self, iova_range: Range) -> Result { - mod_pr_debug!("UATPageTable::alloc_pages: {:#x?}\n", iova_range); - self.with_pages(iova_range, true, false, |_, _| Ok(())) - } - - /// Reserve all tables for a fixed mapping before publishing any leaves. - /// GPUVM and the fixed-address allocator may share a page table, so check - /// that neither owns a destination leaf before allowing failure rollback. - pub(crate) fn prepare_map(&mut self, range: Range) -> Result { - if range.is_empty() || (range.start | range.end) & UAT_PGMSK as u64 != 0 { - return Err(EINVAL); - } - self.with_pages(range.clone(), false, false, |_, ptes| { - if ptes.iter().any(|pte| pte.load(Ordering::Acquire) != 0) { - return Err(EBUSY); - } - Ok(()) - })?; - self.alloc_pages(range) - } - - fn pte_bits(&self) -> u64 { - if self.ttb_owned { - // Owned page tables are userspace, so non-global - PTE_TYPE_LEAF_TABLE | UAT_NON_GLOBAL - } else { - // The sole non-owned page table is kernelspace, so global - PTE_TYPE_LEAF_TABLE - } - } - - pub(crate) fn map_pages( - &mut self, - iova_range: Range, - mut phys: PhysicalAddr, - prot: Prot, - one_page: bool, - ) -> Result { - mod_pr_debug!( - "UATPageTable::map_pages: {:#x?} {:#x?} {:?}\n", - iova_range, - phys, - prot - ); - if phys & (UAT_PGMSK as PhysicalAddr) != 0 { - pr_err!("UATPageTable::map_pages: phys not aligned: {:#x?}\n", phys); - return Err(EINVAL); - } - - let pte_bits = self.pte_bits(); - - self.with_pages(iova_range, true, false, |iova, ptes| { - for (idx, pte) in ptes.iter().enumerate() { - let ptev = pte.load(Ordering::Relaxed); - if ptev != 0 { - pr_err!( - "UATPageTable::map_pages: Page at IOVA {:#x} is mapped (PTE: {:#x})\n", - iova + (idx * UAT_PGSZ) as u64, - ptev - ); - } - pte.store(phys | prot.as_pte() | pte_bits, Ordering::Relaxed); - if !one_page { - phys += UAT_PGSZ as PhysicalAddr; - } - } - Ok(()) - }) - } - - pub(crate) fn reprot_pages(&mut self, iova_range: Range, prot: Prot) -> Result { - mod_pr_debug!( - "UATPageTable::reprot_pages: {:#x?} {:?}\n", - iova_range, - prot - ); - self.with_pages(iova_range, true, false, |iova, ptes| { - for (idx, pte) in ptes.iter().enumerate() { - let ptev = pte.load(Ordering::Relaxed); - if ptev & PTE_TYPE_BITS != PTE_TYPE_LEAF_TABLE { - pr_err!( - "UATPageTable::reprot_pages: Page at IOVA {:#x} is unmapped (PTE: {:#x})\n", - iova + (idx * UAT_PGSZ) as u64, - ptev - ); - continue; - } - pte.store((ptev & !UAT_PROT_BITS) | prot.as_pte(), Ordering::Relaxed); - } - Ok(()) - }) - } - - pub(crate) fn unmap_pages(&mut self, iova_range: Range) -> Result { - mod_pr_debug!("UATPageTable::unmap_pages: {:#x?}\n", iova_range); - self.with_pages(iova_range, false, false, |iova, ptes| { - for (idx, pte) in ptes.iter().enumerate() { - if pte.load(Ordering::Relaxed) & PTE_TYPE_LEAF_TABLE == 0 { - pr_err!( - "UATPageTable::unmap_pages: Page at IOVA {:#x} already unmapped\n", - iova + (idx * UAT_PGSZ) as u64 - ); - } - pte.store(0, Ordering::Relaxed); - } - Ok(()) - }) - } - - /// Discards an aligned map operation that failed before acquiring a GPUVA owner. - /// Missing tables and leaves are expected; existing table storage stays allocated. - pub(crate) fn discard_partial_map(&mut self, iova_range: Range) -> Result { - self.with_pages(iova_range, false, false, |_, ptes| { - for pte in ptes { - pte.store(0, Ordering::Relaxed); - } - Ok(()) - }) - } - - #[cfg(CONFIG_DEV_COREDUMP)] - pub(crate) fn dump_pages(&mut self, iova_range: Range) -> Result> { - let mut pages = KVVec::new(); - let oas_mask = self.oas_mask; - let iova_base = self.va_range.start & !self.ias_mask; - self.with_pages(iova_range, false, false, |iova, ptes| { - let iova = iova | iova_base; - for (idx, ppte) in ptes.iter().enumerate() { - let pte = ppte.load(Ordering::Relaxed); - if (pte & PTE_TYPE_LEAF_TABLE) != PTE_TYPE_LEAF_TABLE { - continue; - } - let memattr = ((pte & UAT_MEMATTR_BITS) >> UAT_MEMATTR_SHIFT) as u8; - - if !(memattr == MEMATTR_CACHED || memattr == MEMATTR_UNCACHED) { - pages.push( - DumpedPage { - iova: iova + (idx * UAT_PGSZ) as u64, - pte, - data: None, - }, - GFP_KERNEL, - )?; - continue; - } - let phys = pte & oas_mask & (!UAT_PGMSK as u64); - // SAFETY: GPU pages are either firmware/preallocated pages - // (which the kernel isn't concerned with and are either in - // the page map or not, and if they aren't, borrow_phys() - // will fail), or GPU page table pages (which we own), - // or GEM buffer pages (which are locked while they are - // mapped in the page table), so they should be safe to - // borrow. - // - // This does trust the firmware not to have any weird - // mappings in its own internal page tables, but since - // those are managed by the uPPL which is privileged anyway, - // this trust does not actually extend any trust boundary. - let src_page = match unsafe { Page::borrow_phys(&phys) } { - Some(page) => page, - None => { - pages.push( - DumpedPage { - iova: iova + (idx * UAT_PGSZ) as u64, - pte, - data: None, - }, - GFP_KERNEL, - )?; - continue; - } - }; - let dst_page = Page::alloc_page(GFP_KERNEL)?; - src_page.with_page_mapped(|psrc| -> Result { - // SAFETY: This could technically still have a data race with the firmware - // or other driver code (or even userspace with timestamp buffers), but while - // the Rust language technically says this is UB, in the real world, using - // atomic reads for this is guaranteed to never cause any harmful effects - // other than possibly reading torn/unreliable data. At least on ARM64 anyway. - // - // (Yes, I checked with Rust people about this. ~~ Lina) - // - let src_items = unsafe { - core::slice::from_raw_parts( - psrc as *const AtomicU64, - UAT_PGSZ / core::mem::size_of::(), - ) - }; - dst_page.with_page_mapped(|pdst| -> Result { - // SAFETY: We own the destination page, so it is safe to view its contents - // as a u64 slice. - let dst_items = unsafe { - core::slice::from_raw_parts_mut( - pdst as *mut u64, - UAT_PGSZ / core::mem::size_of::(), - ) - }; - for (si, di) in src_items.iter().zip(dst_items.iter_mut()) { - *di = si.load(Ordering::Relaxed); - } - Ok(()) - })?; - Ok(()) - })?; - pages.push( - DumpedPage { - iova: iova + (idx * UAT_PGSZ) as u64, - pte, - data: Some(dst_page), - }, - GFP_KERNEL, - )?; - } - Ok(()) - })?; - Ok(pages) - } -} - -impl Drop for UatPageTable { - fn drop(&mut self) { - mod_pr_debug!("UATPageTable::drop range: {:#x?}\n", &self.va_range); - if self - .with_pages(self.va_range.clone(), false, true, |iova, ptes| { - for (idx, pte) in ptes.iter().enumerate() { - if pte.load(Ordering::Relaxed) != 0 { - pr_err!( - "UATPageTable::drop: Leaked page at IOVA {:#x}\n", - iova + (idx * UAT_PGSZ) as u64 - ); - } - } - Ok(()) - }) - .is_err() - { - pr_err!("UATPageTable::drop failed to free page tables\n",); - } - if self.ttb_owned { - mod_pr_debug!("UATPageTable::drop: Free TTB {:#x}\n", self.ttb); - // SAFETY: If we own the ttb, it was allocated with Page::into_phys(). - unsafe { - Page::from_phys(self.ttb); - } - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/queue/common.rs b/drivers/gpu/drm/asahi-neo/queue/common.rs deleted file mode 100644 index 067624cd66734f..00000000000000 --- a/drivers/gpu/drm/asahi-neo/queue/common.rs +++ /dev/null @@ -1,42 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Common queue functionality. -//! -//! Shared helpers used by the submission logic for multiple command types. - -use crate::file; -use crate::fw::job::UserTimestamp; - -use kernel::prelude::*; -use kernel::uapi; -use kernel::xarray; - -pub(super) fn get_timestamp_object( - objects: Pin<&xarray::XArray>>, - timestamp: uapi::drm_asahi_neo_timestamp, -) -> Result> { - if timestamp.handle == 0 { - return Ok(None); - } - - let guard = objects.lock(); - let object = guard - .get(timestamp.handle.try_into()?) - .ok_or(ENOENT)? - .clone(); - core::mem::drop(guard); - - #[allow(irrefutable_let_patterns)] - if let file::Object::TimestampBuffer(mapping) = object { - let offset = timestamp.offset; - if (offset.checked_add(8).ok_or(EINVAL)?) as usize > mapping.size() { - return Err(ERANGE); - } - Ok(Some(UserTimestamp { - mapping: mapping.clone(), - offset: offset as usize, - })) - } else { - Err(EINVAL) - } -} diff --git a/drivers/gpu/drm/asahi-neo/queue/compute.rs b/drivers/gpu/drm/asahi-neo/queue/compute.rs deleted file mode 100644 index df9e4c97e32480..00000000000000 --- a/drivers/gpu/drm/asahi-neo/queue/compute.rs +++ /dev/null @@ -1,385 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![allow(clippy::unusual_byte_groupings)] - -//! Compute work queue. -//! -//! A compute queue consists of one underlying WorkQueue. -//! This module is in charge of creating all of the firmware structures required to submit compute -//! work to the GPU, based on the userspace command buffer. - -use super::common; -use crate::alloc::Allocator; -use crate::debug::*; -use crate::fw::types::*; -use crate::gpu::GpuManager; -use crate::{ - file, - fw, - gpu, - microseq, // -}; -use crate::{ - inner_ptr, - inner_weak_ptr, // -}; -use core::sync::atomic::Ordering; -use kernel::dma_fence::RawDmaFence; -use kernel::drm_neo::sched::Job; -use kernel::prelude::*; -use kernel::sync::Arc; -use kernel::uapi; -use kernel::xarray; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Compute; - -#[versions(AGX_NEO)] -impl super::QueueInner::ver { - /// Submit work to a compute queue. - pub(super) fn submit_compute( - &self, - job: &mut Job, - cmdbuf: &uapi::drm_asahi_neo_cmd_compute, - attachments: µseq::Attachments, - objects: Pin<&xarray::XArray>>, - id: u64, - flush_stamps: bool, - ) -> Result { - let gpu = match (*self.dev) - .gpu - .as_any() - .downcast_ref::() - { - Some(gpu) => gpu, - None => { - dev_crit!(self.dev.as_ref(), "GpuManager mismatched with Queue!\n"); - return Err(EIO); - } - }; - - let mut alloc = gpu.alloc(); - let kalloc = &mut *alloc; - - mod_dev_dbg!(self.dev, "[Submission {}] Compute!\n", id); - - if cmdbuf.flags != 0 { - return Err(EINVAL); - } - - let mut user_timestamps: fw::job::UserTimestamps = Default::default(); - user_timestamps.start = common::get_timestamp_object(objects, cmdbuf.ts.start)?; - user_timestamps.end = common::get_timestamp_object(objects, cmdbuf.ts.end)?; - - // This sequence number increases per new client/VM? assigned to some slot, - // but it's unclear *which* slot... - let slot_client_seq: u8 = (self.id & 0xff) as u8; - - let vm_bind = job.vm_bind.clone(); - - mod_dev_dbg!( - self.dev, - "[Submission {}] VM slot = {}\n", - id, - vm_bind.slot() - ); - - let notifier = self.notifier.clone(); - - let fence = job.fence.clone(); - let comp_job = job.get_comp()?; - let ev_comp = comp_job.event_info(); - - let preempt2_off = gpu.get_cfg().compute_preempt1_size; - let preempt3_off = preempt2_off + 8; - let preempt4_off = preempt3_off + 8; - let preempt5_off = preempt4_off + 8; - let preempt_size = preempt5_off + 8; - - let preempt_buf = self - .ualloc - .lock() - .array_empty_tagged(preempt_size, b"CPMT")?; - - mod_dev_dbg!( - self.dev, - "[Submission {}] Event #{} {:#x?} -> {:#x?}\n", - id, - ev_comp.slot, - ev_comp.value, - ev_comp.value.next(), - ); - - let timestamps = Arc::new( - kalloc.shared.new_default::()?, - GFP_KERNEL, - )?; - - let uuid = 0; - mod_dev_dbg!(self.dev, "[Submission {}] UUID = {:#x?}\n", id, uuid); - - // TODO: check - #[ver(V >= V13_0B4)] - let count = self.counter.fetch_add(1, Ordering::Relaxed); - - let comp = GpuObject::new_init_prealloc( - kalloc.gpu_ro.alloc_object()?, - |ptr: GpuWeakPointer| { - let notifier = notifier.clone(); - let vm_bind = vm_bind.clone(); - try_init!(fw::compute::RunCompute::ver { - preempt_buf: preempt_buf, - micro_seq: { - let mut builder = microseq::Builder::new(); - - let stats = gpu.initdata.runtime_pointers.stats.comp.weak_pointer(); - - let start_comp = builder.add(microseq::StartCompute::ver { - header: microseq::op::StartCompute::HEADER, - unk_pointer: inner_weak_ptr!(ptr, unk_pointee), - #[ver(G < G14X)] - job_params1: Some(inner_weak_ptr!(ptr, job_params1)), - #[ver(G >= G14X)] - job_params1: None, - #[ver(G >= G14X)] - registers: inner_weak_ptr!(ptr, registers), - stats, - work_queue: ev_comp.info_ptr, - vm_slot: vm_bind.slot(), - unk_28: 0x1, - event_generation: self.id as u32, - event_seq: U64(ev_comp.event_seq), - unk_38: 0x0, - job_params2: inner_weak_ptr!(ptr, job_params2), - unk_44: 0x0, - uuid, - attachments: *attachments, - padding: Default::default(), - #[ver(V >= V13_0B4)] - unk_flag: inner_weak_ptr!(ptr, unk_flag), - #[ver(V >= V13_0B4)] - counter: U64(count), - #[ver(V >= V13_0B4)] - notifier_buf: inner_weak_ptr!(notifier.weak_pointer(), state.unk_buf), - })?; - - if user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(true), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.start_addr), - work_queue: ev_comp.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, context_store_req), - uuid, - unk_30_padding: 0, - })?; - } - - #[ver(G < G14X)] - builder.add(microseq::WaitForIdle { - header: microseq::op::WaitForIdle::new(microseq::Pipe::Compute), - })?; - #[ver(G >= G14X)] - builder.add(microseq::WaitForIdle2 { - header: microseq::op::WaitForIdle2::HEADER, - })?; - - if user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(false), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.end_addr), - work_queue: ev_comp.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, context_store_req), - uuid, - unk_30_padding: 0, - })?; - } - - let off = builder.offset_to(start_comp); - builder.add(microseq::FinalizeCompute::ver { - header: microseq::op::FinalizeCompute::HEADER, - stats, - work_queue: ev_comp.info_ptr, - vm_slot: vm_bind.slot(), - #[ver(V < V13_0B4)] - unk_18: 0, - job_params2: inner_weak_ptr!(ptr, job_params2), - unk_24: 0, - uuid, - fw_stamp: ev_comp.fw_stamp_pointer, - stamp_value: ev_comp.value.next(), - unk_38: 0, - unk_3c: 0, - unk_40: 0, - unk_44: 0, - unk_48: 0, - unk_4c: 0, - unk_50: 0, - unk_54: 0, - unk_58: 0, - #[ver(G == G14 && V < V13_0B4)] - unk_5c_g14: U64(0), - restart_branch_offset: off, - has_attachments: (attachments.count > 0) as u32, - #[ver(V >= V13_0B4)] - unk_64: Default::default(), - #[ver(V >= V13_0B4)] - unk_flag: inner_weak_ptr!(ptr, unk_flag), - #[ver(V >= V13_0B4)] - unk_79: Default::default(), - })?; - - builder.add(microseq::RetireStamp { - header: microseq::op::RetireStamp::HEADER, - })?; - builder.build(&mut kalloc.private)? - }, - notifier, - vm_bind, - timestamps, - user_timestamps, - }) - }, - |inner, _ptr| { - let vm_slot = vm_bind.slot(); - try_init!(fw::compute::raw::RunCompute::ver { - tag: fw::workqueue::CommandType::RunCompute, - #[ver(V >= V13_0B4)] - counter: U64(count), - unk_4: 0, - vm_slot, - notifier: inner.notifier.gpu_pointer(), - unk_pointee: Default::default(), - #[ver(G < G14X)] - __pad0: Default::default(), - #[ver(G < G14X)] - job_params1 <- try_init!(fw::compute::raw::JobParameters1 { - preempt_buf1: inner.preempt_buf.gpu_pointer(), - cdm_ctrl_stream_base: U64(cmdbuf.cdm_ctrl_stream_base), - // buf2-5 Only if internal program is used - preempt_buf2: inner.preempt_buf.gpu_offset_pointer(preempt2_off), - preempt_buf3: inner.preempt_buf.gpu_offset_pointer(preempt3_off), - preempt_buf4: inner.preempt_buf.gpu_offset_pointer(preempt4_off), - preempt_buf5: inner.preempt_buf.gpu_offset_pointer(preempt5_off), - usc_exec_base_cp: U64(self.usc_exec_base), - unk_38: U64(0x8c60), - helper_program: cmdbuf.helper.binary, // Internal program addr | 1 - unk_44: 0, - helper_arg: U64(cmdbuf.helper.data), // Only if internal program used - helper_cfg: cmdbuf.helper.cfg, // 0x40 if internal program used - unk_54: 0, - unk_58: 1, - unk_5c: 0, - iogpu_unk_40: 0, // 0x1c if internal program used - __pad: Default::default(), - }), - #[ver(G >= G14X)] - registers: fw::job::raw::RegisterArray::new( - inner_weak_ptr!(_ptr, registers.registers), - |r| { - r.add(0x1a510, inner.preempt_buf.gpu_pointer().into()); - r.add(0x1a420, cmdbuf.cdm_ctrl_stream_base); - // buf2-5 Only if internal program is used - r.add(0x1a4d0, inner.preempt_buf.gpu_offset_pointer(preempt2_off).into()); - r.add(0x1a4d8, inner.preempt_buf.gpu_offset_pointer(preempt3_off).into()); - r.add(0x1a4e0, inner.preempt_buf.gpu_offset_pointer(preempt4_off).into()); - r.add(0x1a4e8, inner.preempt_buf.gpu_offset_pointer(preempt5_off).into()); - r.add(0x10071, self.usc_exec_base); // USC_EXEC_BASE_CP - r.add(0x11841, cmdbuf.helper.binary.into()); - r.add(0x11849, cmdbuf.helper.data); - r.add(0x11f81, cmdbuf.helper.cfg.into()); - r.add(0x1a440, 0x24201); - r.add(0x12091, 0 /* iogpu_unk_40 */); - /* - r.add(0x10201, 0x100); // Some kind of counter?? Does this matter? - r.add(0x10428, 0x100); // Some kind of counter?? Does this matter? - */ - } - ), - __pad1: Default::default(), - microsequence: inner.micro_seq.gpu_pointer(), - microsequence_size: inner.micro_seq.len() as u32, - job_params2 <- try_init!(fw::compute::raw::JobParameters2::ver { - #[ver(V >= V13_0B4)] - unk_0_0: 0, - unk_0: Default::default(), - preempt_buf1: inner.preempt_buf.gpu_pointer(), - cdm_ctrl_stream_end: U64(cmdbuf.cdm_ctrl_stream_end), - unk_34: Default::default(), - #[ver(G < G14X)] - unk_g14x: 0, - #[ver(G >= G14X)] - unk_g14x: 0x24201, - unk_58: 0, - #[ver(V < V13_0B4)] - unk_5c: 0, - }), - encoder_params <- try_init!(fw::job::raw::EncoderParams { - unk_8: 0x0, // fixed - sync_grow: 0x0, // check! - unk_10: 0x0, // fixed - encoder_id: 0, - unk_18: 0x0, // fixed - unk_mask: 0xffffffff, - sampler_array: U64(cmdbuf.sampler_heap), - sampler_count: cmdbuf.sampler_count as u32, - sampler_max: (cmdbuf.sampler_count as u32) + 1, - }), - meta <- try_init!(fw::job::raw::JobMeta { - unk_0: 0, - unk_2: 0, - no_preemption: 0, - stamp: ev_comp.stamp_pointer, - fw_stamp: ev_comp.fw_stamp_pointer, - stamp_value: ev_comp.value.next(), - stamp_slot: ev_comp.slot, - evctl_index: 0, // fixed - flush_stamps: flush_stamps as u32, - uuid, - event_seq: ev_comp.event_seq as u32, - }), - command_time: U64(0), - timestamp_pointers <- try_init!(fw::job::raw::TimestampPointers { - start_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), start)), - end_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), end)), - }), - user_timestamp_pointers: inner.user_timestamps.pointers()?, - client_sequence: slot_client_seq, - pad_2d1: Default::default(), - unk_2d4: 0, - unk_2d8: 0, - #[ver(V >= V13_0B4)] - context_store_req: U64(0), - #[ver(V >= V13_0B4)] - context_store_compl: U64(0), - #[ver(V >= V13_0B4)] - unk_2e9: Default::default(), - #[ver(V >= V13_0B4)] - unk_flag: U32(0), - #[ver(V >= V13_0B4)] - unk_pad: Default::default(), - }) - }, - )?; - - core::mem::drop(alloc); - - fence.add_command(); - comp_job.add_cb(comp, vm_bind.slot(), move |error| { - if let Some(err) = error { - fence.set_error(err.into()) - } - - fence.command_complete(); - })?; - - comp_job.next_seq(); - - Ok(()) - } -} diff --git a/drivers/gpu/drm/asahi-neo/queue/mod.rs b/drivers/gpu/drm/asahi-neo/queue/mod.rs deleted file mode 100644 index 03926179c44db3..00000000000000 --- a/drivers/gpu/drm/asahi-neo/queue/mod.rs +++ /dev/null @@ -1,958 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Submission queue management -//! -//! This module implements the userspace view of submission queues and the logic to map userspace -//! submissions to firmware queues. - -use kernel::dma_fence::*; -use kernel::prelude::*; -use kernel::{ - c_str, - dma_fence, - drm_neo::sched, - macros::versions, - sync::{ - Arc, - LockClassKey, - Mutex, // - }, - uapi, - xarray, // -}; - -use crate::alloc::Allocator; -use crate::debug::*; -use crate::driver::{AsahiDevRef, AsahiDevice}; -use crate::file::MAX_COMMANDS_PER_SUBMISSION; -use crate::fw::types::*; -use crate::gpu::{ - Gpu, - GpuManager, // -}; -use crate::inner_weak_ptr; -use crate::microseq; -use crate::module_parameters; -use crate::stats; -use crate::util::{ - AnyBitPattern, - Reader, // -}; -use crate::{ - alloc, - buffer, - channel, - event, - file, - fw, - gpu, - mmu, - workqueue, // -}; - -use core::sync::atomic::{ - AtomicU64, - Ordering, // -}; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Queue; - -const WQ_SIZE: u32 = 0x500; - -mod common; -mod compute; -mod render; - -/// Trait implemented by all versioned queues. -pub(crate) trait Queue: Send + Sync { - fn submit( - &mut self, - id: u64, - syncs: KVec, - in_sync_count: usize, - cmdbuf_raw: &[u8], - objects: Pin<&xarray::XArray>>, - ) -> Result; -} - -#[versions(AGX_NEO)] -struct SubQueue { - wq: Arc, -} - -#[versions(AGX_NEO)] -impl SubQueue::ver { - fn new_job(&mut self, fence: dma_fence::Fence) -> SubQueueJob::ver { - SubQueueJob::ver { - wq: self.wq.clone(), - fence: Some(fence), - job: None, - } - } -} - -#[versions(AGX_NEO)] -struct SubQueueJob { - wq: Arc, - job: Option, - fence: Option, -} - -#[versions(AGX_NEO)] -impl SubQueueJob::ver { - fn get(&mut self) -> Result<&mut workqueue::Job::ver> { - if self.job.is_none() { - mod_pr_debug!("SubQueueJob: Creating {:?} job\n", self.wq.pipe_type()); - self.job - .replace(self.wq.new_job(self.fence.take().unwrap())?); - } - Ok(self.job.as_mut().expect("expected a Job")) - } - - fn commit(&mut self) -> Result { - match self.job.as_mut() { - Some(job) => job.commit(), - None => Ok(()), - } - } - - fn can_submit(&self) -> Option { - self.job.as_ref().and_then(|job| job.can_submit()) - } -} - -#[versions(AGX_NEO)] -pub(crate) struct Queue { - dev: AsahiDevRef, - _sched: sched::Scheduler, - entity: sched::Entity, - vm: mmu::Vm, - q_vtx: Option, - q_frag: Option, - q_comp: Option, - fence_ctx: FenceContexts, - inner: QueueInner::ver, - stats: Arc, -} - -#[versions(AGX_NEO)] -pub(crate) struct QueueInner { - dev: AsahiDevRef, - ualloc: Arc>, - buffer: buffer::Buffer::ver, - gpu_context: Arc, - notifier_list: Arc>, - notifier: Arc>, - usc_exec_base: u64, - id: u64, - #[ver(V >= V13_0B4)] - counter: AtomicU64, -} - -#[versions(AGX_NEO)] -#[derive(Default)] -pub(crate) struct JobFence { - id: u64, - pending: AtomicU64, - stats: Option>, -} - -#[versions(AGX_NEO)] -impl JobFence::ver { - fn add_command(self: &FenceObject) { - self.pending.fetch_add(1, Ordering::Relaxed); - } - - fn command_complete(self: &FenceObject) { - let remain = self.pending.fetch_sub(1, Ordering::Relaxed) - 1; - mod_pr_debug!( - "JobFence[{}]: Command complete (remain: {})\n", - self.id, - remain - ); - if remain == 0 { - if let Some(stats) = &self.stats { - stats.note_job(); - } - mod_pr_debug!("JobFence[{}]: Signaling\n", self.id); - self.signal(); - } - } -} - -#[versions(AGX_NEO)] -#[vtable] -impl dma_fence::FenceOps for JobFence::ver { - fn get_driver_name<'a>(self: &'a FenceObject) -> &'a CStr { - c_str!("asahi") - } - fn get_timeline_name<'a>(self: &'a FenceObject) -> &'a CStr { - c_str!("queue") - } -} - -#[versions(AGX_NEO)] -pub(crate) struct QueueJob { - dev: AsahiDevRef, - vm_bind: mmu::VmBind, - op_guard: Option, - sj_vtx: Option, - sj_frag: Option, - sj_comp: Option, - fence: UserFence, - notifier: Arc>, - notification_count: u32, - id: u64, -} - -pub(crate) struct QueueTimeout { - dev: AsahiDevRef, - id: u64, -} - -#[versions(AGX_NEO)] -impl QueueJob::ver { - fn get_vtx(&mut self) -> Result<&mut workqueue::Job::ver> { - self.sj_vtx - .as_mut() - .ok_or_else(|| { - cls_pr_debug!(Errors, "No vertex queue\n"); - EINVAL - })? - .get() - } - fn get_frag(&mut self) -> Result<&mut workqueue::Job::ver> { - self.sj_frag - .as_mut() - .ok_or_else(|| { - cls_pr_debug!(Errors, "No fragment queue\n"); - EINVAL - })? - .get() - } - fn get_comp(&mut self) -> Result<&mut workqueue::Job::ver> { - self.sj_comp - .as_mut() - .ok_or_else(|| { - cls_pr_debug!(Errors, "No compute queue\n"); - EINVAL - })? - .get() - } - - fn commit(&mut self) -> Result { - mod_dev_dbg!(self.dev, "QueueJob {}: Committing\n", self.id); - - self.sj_vtx.as_mut().map(|a| a.commit()).unwrap_or(Ok(()))?; - self.sj_frag - .as_mut() - .map(|a| a.commit()) - .unwrap_or(Ok(()))?; - self.sj_comp.as_mut().map(|a| a.commit()).unwrap_or(Ok(())) - } -} - -#[versions(AGX_NEO)] -impl sched::JobImpl for QueueJob::ver { - type TimeoutData = QueueTimeout; - const MODULE: Option<&'static kernel::ThisModule> = Some(&crate::THIS_MODULE); - - fn timeout_data(&self) -> QueueTimeout { - QueueTimeout { dev: self.dev.clone(), id: self.id } - } - - fn prepare(job: &mut Self) -> Option { - mod_dev_dbg!(job.dev, "QueueJob {}: Checking runnability\n", job.id); - - if let Some(sj) = job.sj_vtx.as_ref() { - if let Some(fence) = sj.can_submit() { - mod_dev_dbg!( - job.dev, - "QueueJob {}: Blocking due to vertex queue full\n", - job.id - ); - return Some(fence); - } - } - if let Some(sj) = job.sj_frag.as_ref() { - if let Some(fence) = sj.can_submit() { - mod_dev_dbg!( - job.dev, - "QueueJob {}: Blocking due to fragment queue full\n", - job.id - ); - return Some(fence); - } - } - if let Some(sj) = job.sj_comp.as_ref() { - if let Some(fence) = sj.can_submit() { - mod_dev_dbg!( - job.dev, - "QueueJob {}: Blocking due to compute queue full\n", - job.id - ); - return Some(fence); - } - } - None - } - - #[allow(unused_assignments)] - fn run(job: &mut Self) -> Result> { - mod_dev_dbg!(job.dev, "QueueJob {}: Running Job\n", job.id); - - // We can only increase the notifier threshold here, now that we are - // actually running the job. We cannot increase it while queueing the - // job without introducing subtle race conditions. Suppose we did, as - // early versions of drm_neo/asahi did: - // - // 1. When processing the ioctl submit, a job is queued to drm_sched. - // Incorrectly, the notifier threshold is increased, gating firmware - // events. - // 2. When DRM schedules an event, the hardware is kicked. - // 3. When the number of processed jobs equals the threshold, the - // firmware signals the complete event to the kernel - // 4. When the kernel gets a complete event, we signal the out-syncs. - // - // Does that work? There are a few scenarios. - // - // 1. There is nothing else ioctl submitted before the job completes. - // The job is scheduled, completes, and signals immediately. - // Everything works. - // 2. There is nontrivial sync across different queues. Since each queue - // has a separate own notifier threshold, submitting one does not - // block scheduling of the other. Everything works the way you'd - // expect. drm_neo/sched handles the wait/signal ordering. - // 3. Two ioctls are submitted back-to-back. The first signals a fence - // that the second waits on. Due to the notifier threshold increment, - // the first job's completion event is deferred. But in good - // conditions, drm_neo/sched will schedule the second submit anyway - // because it kills the pointless intra-queue sync. Then both - // commands execute and are signalled together. - // 4. Two ioctls are submitted back-to-back as above, but conditions are - // bad. Reporting completion of the first job is still masked by the - // notifier threshold, but the intra-queue fences are not optimized - // out in drm_neo/sched... drm_neo/sched doesn't schedule the second job - // until the first is signalled, but the first isn't signalled until - // the second is completed, but the second can't complete until it's - // scheduled. We hang! - // - // In good conditions, everything works properly and/or we win the race - // to mask the issue. So the issue here is challenging to hit. - // Nevertheless, we do need to get it right. - // - // The intention with drm_neo/sched is that jobs that are not yet scheduled - // are "invisible" to the firmware. Incrementing the notifier threshold - // earlier than this violates that which leads to circles like the - // above. Deferring the increment to submit solves the race. - let gpu = match (*job.dev) - .gpu - .clone() - .arc_as_any() - .downcast::() - { - Ok(gpu) => gpu, - Err(_) => { - dev_crit!(job.dev.as_ref(), "GpuManager mismatched with QueueJob!\n"); - return Err(EIO); - } - }; - - if job.op_guard.is_none() { - job.op_guard = Some(gpu.start_op()?); - } - - // First submit all the commands for each queue. This can fail. - - let mut frag_job = None; - let mut frag_sub = None; - if let Some(sj) = job.sj_frag.as_mut() { - frag_job = sj.job.take(); - if let Some(wqjob) = frag_job.as_mut() { - mod_dev_dbg!(job.dev, "QueueJob {}: Submit fragment\n", job.id); - frag_sub = Some(wqjob.submit()?); - } - } - - let mut vtx_job = None; - let mut vtx_sub = None; - if let Some(sj) = job.sj_vtx.as_mut() { - vtx_job = sj.job.take(); - if let Some(wqjob) = vtx_job.as_mut() { - mod_dev_dbg!(job.dev, "QueueJob {}: Submit vertex\n", job.id); - vtx_sub = Some(wqjob.submit()?); - } - } - - let mut comp_job = None; - let mut comp_sub = None; - if let Some(sj) = job.sj_comp.as_mut() { - comp_job = sj.job.take(); - if let Some(wqjob) = comp_job.as_mut() { - mod_dev_dbg!(job.dev, "QueueJob {}: Submit compute\n", job.id); - comp_sub = Some(wqjob.submit()?); - } - } - - // Prepared submissions still hold their workqueue locks and have not - // published cpu_wptr. Only count them after every fallible submit() - // succeeded, before the first run_job exposes commands to firmware. - job.notifier.threshold.with(|raw, _inner| { - raw.increase(job.notification_count); - }); - - // Now we fully commit to running the job - mod_dev_dbg!(job.dev, "QueueJob {}: Run fragment\n", job.id); - frag_sub.map(|a| gpu.run_job(a)).transpose()?; - - mod_dev_dbg!(job.dev, "QueueJob {}: Run vertex\n", job.id); - vtx_sub.map(|a| gpu.run_job(a)).transpose()?; - - mod_dev_dbg!(job.dev, "QueueJob {}: Run compute\n", job.id); - comp_sub.map(|a| gpu.run_job(a)).transpose()?; - - mod_dev_dbg!(job.dev, "QueueJob {}: Drop compute job\n", job.id); - core::mem::drop(comp_job); - mod_dev_dbg!(job.dev, "QueueJob {}: Drop vertex job\n", job.id); - core::mem::drop(vtx_job); - mod_dev_dbg!(job.dev, "QueueJob {}: Drop fragment job\n", job.id); - core::mem::drop(frag_job); - - Ok(Some(Fence::from_fence(&job.fence))) - } - - fn timed_out(job: &mut QueueTimeout, _finished: bool) -> sched::Status { - // FIXME: Handle timeouts properly - dev_err!( - job.dev.as_ref(), - "QueueJob {}: Job timed out on the DRM scheduler, things will probably break\n", - job.id - ); - sched::Status::NoDevice - } - - fn cancel(job: &mut Self) { - dev_info!( - job.dev.as_ref(), - "QueueJob {}: Job canceled on DRM scheduler teardown\n", - job.id - ); - } -} - -#[versions(AGX_NEO)] -impl Drop for QueueJob::ver { - fn drop(&mut self) { - mod_dev_dbg!(self.dev, "QueueJob {}: Dropping\n", self.id); - } -} - -static QUEUE_NAME: &CStr = c_str!("asahi_fence"); -static QUEUE_CLASS_KEY: Pin<&LockClassKey> = kernel::static_lock_class!(); - -#[versions(AGX_NEO)] -impl Queue::ver { - /// Create a new user queue. - #[allow(clippy::too_many_arguments)] - pub(crate) fn new( - dev: &AsahiDevice, - gpu: &dyn GpuManager, - vm: mmu::Vm, - alloc: &mut gpu::KernelAllocators, - ualloc: Arc>, - ualloc_priv: Arc>, - event_manager: Arc, - stats: Arc, - mgr: &buffer::BufferManager::ver, - id: u64, - priority: u32, - usc_exec_base: u64, - ) -> Result { - mod_dev_dbg!(dev, "[Queue {}] Creating queue\n", id); - - // Must be shared, no cache management on this one! - let mut notifier_list = alloc.shared.new_default::()?; - - let self_ptr = notifier_list.weak_pointer(); - notifier_list.with_mut(|raw, _inner| { - raw.list_head.next = Some(inner_weak_ptr!(self_ptr, list_head)); - }); - - let threshold = alloc.shared.new_default::()?; - - let notifier: Arc> = Arc::new( - alloc.private.new_init( - /*try_*/ init!(fw::event::Notifier::ver { threshold }), - |inner, _p| { - try_init!(fw::event::raw::Notifier::ver { - threshold: inner.threshold.gpu_pointer(), - generation: AtomicU32::new(id as u32), - cur_count: AtomicU32::new(0), - unk_10: AtomicU32::new(0x50), - state: Default::default() - }) - }, - )?, - GFP_KERNEL, - )?; - - // Priorities are handled by the AGX scheduler, there is no meaning within a - // per-queue scheduler. Use a single run queue wth Kernel priority. - let sched = - sched::Scheduler::new(dev.as_ref(), 1, WQ_SIZE, 0, 100000, c_str!("asahi_sched"))?; - let entity = sched::Entity::new(&sched, sched::Priority::Kernel)?; - - let buffer = buffer::Buffer::ver::new(gpu, alloc, ualloc.clone(), ualloc_priv, mgr)?; - - let mut ret = Queue::ver { - dev: dev.into(), - _sched: sched, - entity, - vm, - q_vtx: None, - q_frag: None, - q_comp: None, - fence_ctx: FenceContexts::new(1, QUEUE_NAME, QUEUE_CLASS_KEY)?, - stats, - inner: QueueInner::ver { - dev: dev.into(), - ualloc, - gpu_context: Arc::new( - workqueue::GpuContext::new(dev, alloc, buffer.any_ref())?, - GFP_KERNEL, - )?, - - buffer, - notifier_list: Arc::new(notifier_list, GFP_KERNEL)?, - notifier, - usc_exec_base, - id, - #[ver(V >= V13_0B4)] - counter: AtomicU64::new(0), - }, - }; - - // Rendering structures - let tvb_blocks = *module_parameters::initial_tvb_size.value(); - - ret.inner.buffer.ensure_blocks(tvb_blocks)?; - - ret.q_vtx = Some(SubQueue::ver { - wq: workqueue::WorkQueue::ver::new( - dev, - alloc, - event_manager.clone(), - ret.inner.gpu_context.clone(), - ret.inner.notifier_list.clone(), - channel::PipeType::Vertex, - id, - priority, - WQ_SIZE, - )?, - }); - - ret.q_frag = Some(SubQueue::ver { - wq: workqueue::WorkQueue::ver::new( - dev, - alloc, - event_manager.clone(), - ret.inner.gpu_context.clone(), - ret.inner.notifier_list.clone(), - channel::PipeType::Fragment, - id, - priority, - WQ_SIZE, - )?, - }); - - // Compute structures - ret.q_comp = Some(SubQueue::ver { - wq: workqueue::WorkQueue::ver::new( - dev, - alloc, - event_manager, - ret.inner.gpu_context.clone(), - ret.inner.notifier_list.clone(), - channel::PipeType::Compute, - id, - priority, - WQ_SIZE, - )?, - }); - - mod_dev_dbg!(dev, "[Queue {}] Queue created\n", id); - Ok(ret) - } -} - -const SQ_RENDER: usize = 0; -const SQ_COMPUTE: usize = 1; -const SQ_COUNT: usize = 2; - -// SAFETY: All bit patterns are valid by construction. -unsafe impl AnyBitPattern for uapi::drm_asahi_neo_cmd_header {} -unsafe impl AnyBitPattern for uapi::drm_asahi_neo_cmd_render {} -unsafe impl AnyBitPattern for uapi::drm_asahi_neo_cmd_compute {} -unsafe impl AnyBitPattern for uapi::drm_asahi_neo_attachment {} - -fn build_attachments(reader: &mut Reader<'_>, size: usize) -> Result { - const STRIDE: usize = core::mem::size_of::(); - let count = size / STRIDE; - - if count > microseq::MAX_ATTACHMENTS { - return Err(EINVAL); - } - - let mut attachments: microseq::Attachments = Default::default(); - attachments.count = count as u32; - - for i in 0..count { - let att: uapi::drm_asahi_neo_attachment = reader.read()?; - - if att.flags != 0 || att.pad != 0 { - return Err(EINVAL); - } - - // Some kind of power-of-2 exponent related to attachment size, in - // bounds [1, 6]? We don't know what this is exactly yet. - let unk_e = 1; - - let cache_lines = (att.size + 127) >> 7; - attachments.list[i as usize] = microseq::Attachment { - address: U64(att.pointer), - size: cache_lines.try_into()?, - unk_c: 0x17, - unk_e: unk_e as u16, - }; - } - - Ok(attachments) -} - -#[versions(AGX_NEO)] -impl Queue for Queue::ver { - fn submit( - &mut self, - id: u64, - mut syncs: KVec, - in_sync_count: usize, - cmdbuf_raw: &[u8], - objects: Pin<&xarray::XArray>>, - ) -> Result { - let gpu = match (*self.dev) - .gpu - .clone() - .arc_as_any() - .downcast::() - { - Ok(gpu) => gpu, - Err(_) => { - dev_crit!(self.dev.as_ref(), "GpuManager mismatched with JobImpl!\n"); - return Err(EIO); - } - }; - - mod_dev_dbg!(self.dev, "[Submission {}] Submit job\n", id); - - if gpu.is_crashed() { - dev_err!( - self.dev.as_ref(), - "[Submission {}] GPU is crashed, cannot submit\n", - id - ); - return Err(ENODEV); - } - - let op_guard = if in_sync_count > 0 { - Some(gpu.start_op()?) - } else { - None - }; - - let mut events: [KVec>; SQ_COUNT] = - Default::default(); - - events[SQ_RENDER].push( - self.q_frag.as_ref().and_then(|a| a.wq.event_info()), - GFP_KERNEL, - )?; - events[SQ_COMPUTE].push( - self.q_comp.as_ref().and_then(|a| a.wq.event_info()), - GFP_KERNEL, - )?; - - let vm_bind = gpu.bind_vm(&self.vm)?; - let vm_slot = vm_bind.slot(); - - mod_dev_dbg!(self.dev, "[Submission {}] Creating job\n", id); - - // FIXME: I think this can violate the fence seqno ordering contract. - // If we have e.g. a render submission with no barriers and then a compute submission - // with no barriers, it's possible for the compute submission to complete first, and - // therefore its fence. Maybe we should have separate fence contexts for render - // and compute, and then do a ? (Vert+frag should be fine since there is no vert - // without frag, and frag always serializes.) - let fence: UserFence = self - .fence_ctx - .new_fence::( - 0, - JobFence::ver { - id, - pending: Default::default(), - stats: Some(self.stats.clone()), - }, - )? - .into(); - - let mut cmdbuf = Reader::new(cmdbuf_raw); - - // First, parse the headers to determine the number of compute/render - // commands. This will be used to determine when to flush stamps. - // - // We also use it to determine how many notifications the job will - // generate. We could calculate that in the second pass since we don't - // need until much later, but it's convenient to gather everything at - // the same time. - let mut nr_commands = 0; - let mut last_compute = 0; - let mut last_render = 0; - let mut nr_render = 0; - let mut nr_compute = 0; - - while !cmdbuf.is_empty() { - let header: uapi::drm_asahi_neo_cmd_header = cmdbuf.read()?; - cmdbuf.skip(header.size as usize); - nr_commands += 1; - - match header.cmd_type as u32 { - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_RENDER => { - last_render = nr_commands; - nr_render += 1; - } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_COMPUTE => { - last_compute = nr_commands; - nr_compute += 1; - } - _ => {} - } - } - - let mut job = self.entity.new_job( - 1, - QueueJob::ver { - dev: self.dev.clone(), - vm_bind, - op_guard, - sj_vtx: self - .q_vtx - .as_mut() - .map(|a| a.new_job(Fence::from_fence(&fence))), - sj_frag: self - .q_frag - .as_mut() - .map(|a| a.new_job(Fence::from_fence(&fence))), - sj_comp: self - .q_comp - .as_mut() - .map(|a| a.new_job(Fence::from_fence(&fence))), - fence, - notifier: self.inner.notifier.clone(), - - // Each render command generates 2 notifications: 1 for the - // vertex part, 1 for the fragment part. Each compute command - // generates 1 notification. Sum up to calculate the total - // notification count for the job. - notification_count: (2 * nr_render) + nr_compute, - - id, - }, - )?; - - mod_dev_dbg!( - self.dev, - "[Submission {}] Adding {} in_syncs\n", - id, - in_sync_count - ); - for sync in syncs.drain(0..in_sync_count) { - if let Some(fence) = sync.fence { - job.add_dependency(fence)?; - } - } - - // Validate the number of hardware commands, ignoring software commands - let nr_hw_commands = nr_render + nr_compute; - if nr_hw_commands == 0 || nr_hw_commands > MAX_COMMANDS_PER_SUBMISSION { - cls_pr_debug!( - Errors, - "submit: Command count {} out of valid range [1, {}]\n", - nr_hw_commands, - MAX_COMMANDS_PER_SUBMISSION - 1 - ); - return Err(EINVAL); - } - - cmdbuf.rewind(); - - let mut command_index = 0; - let mut vertex_attachments: microseq::Attachments = Default::default(); - let mut fragment_attachments: microseq::Attachments = Default::default(); - let mut compute_attachments: microseq::Attachments = Default::default(); - - // Parse the full command buffer submitting as we go - while !cmdbuf.is_empty() { - let header: uapi::drm_asahi_neo_cmd_header = cmdbuf.read()?; - let header_size = header.size as usize; - - // Pre-increment command index to match last_compute/last_render - command_index += 1; - - for (queue_idx, index) in [header.vdm_barrier, header.cdm_barrier].iter().enumerate() { - if *index == uapi::DRM_ASAHI_NEO_BARRIER_NONE as u16 { - continue; - } - if let Some(event) = events[queue_idx].get(*index as usize).ok_or_else(|| { - cls_pr_debug!(Errors, "Invalid barrier #{}: {}\n", queue_idx, index); - EINVAL - })? { - let mut alloc = gpu.alloc(); - let queue_job = match header.cmd_type as u32 { - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_RENDER => job.get_vtx()?, - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_COMPUTE => job.get_comp()?, - _ => return Err(EINVAL), - }; - mod_dev_dbg!(self.dev, "[Submission {}] Create Explicit Barrier\n", id); - let barrier = alloc.private.new_init( - pin_init::zeroed::(), - |_inner, _p| { - let queue_job = &queue_job; - try_init!(fw::workqueue::raw::Barrier { - tag: fw::workqueue::CommandType::Barrier, - wait_stamp: event.fw_stamp_pointer, - wait_value: event.value, - wait_slot: event.slot, - stamp_self: queue_job.event_info().value.next(), - uuid: 0xffffbbbb, - external_barrier: 0, - internal_barrier_type: 1, - padding: Default::default(), - }) - }, - )?; - mod_dev_dbg!(self.dev, "[Submission {}] Add Explicit Barrier\n", id); - queue_job.add(barrier, vm_slot)?; - } else { - assert!(*index == 0); - } - } - - match header.cmd_type as u32 { - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_RENDER => { - let render: uapi::drm_asahi_neo_cmd_render = cmdbuf.read_up_to(header_size)?; - - self.inner.submit_render( - &mut job, - &render, - &vertex_attachments, - &fragment_attachments, - objects, - id, - command_index == last_render, - )?; - events[SQ_RENDER].push( - Some( - job.sj_frag - .as_ref() - .expect("No frag queue?") - .job - .as_ref() - .expect("No frag job?") - .event_info(), - ), - GFP_KERNEL, - )?; - } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_COMPUTE => { - let compute: uapi::drm_asahi_neo_cmd_compute = cmdbuf.read_up_to(header_size)?; - - self.inner.submit_compute( - &mut job, - &compute, - &compute_attachments, - objects, - id, - command_index == last_compute, - )?; - events[SQ_COMPUTE].push( - Some( - job.sj_comp - .as_ref() - .expect("No comp queue?") - .job - .as_ref() - .expect("No comp job?") - .event_info(), - ), - GFP_KERNEL, - )?; - } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_VERTEX_ATTACHMENTS => { - vertex_attachments = build_attachments(&mut cmdbuf, header_size)?; - } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_FRAGMENT_ATTACHMENTS => { - fragment_attachments = build_attachments(&mut cmdbuf, header_size)?; - } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_COMPUTE_ATTACHMENTS => { - compute_attachments = build_attachments(&mut cmdbuf, header_size)?; - } - _ => { - cls_pr_debug!(Errors, "Unknown command type {}\n", header.cmd_type); - return Err(EINVAL); - } - } - } - - mod_dev_dbg!( - self.dev, - "Queue {}: Committing job {}\n", - self.inner.id, - job.id - ); - job.commit()?; - - mod_dev_dbg!(self.dev, "Queue {}: Arming job {}\n", self.inner.id, job.id); - let mut job = job.arm(); - let out_fence = job.fences().finished(); - mod_dev_dbg!( - self.dev, - "Queue {}: Pushing job {}\n", - self.inner.id, - job.id - ); - job.push(); - - mod_dev_dbg!( - self.dev, - "Queue {}: Adding {} out_syncs\n", - self.inner.id, - syncs.len() - ); - for mut sync in syncs { - if let Some(chain) = sync.chain_fence.take() { - sync.syncobj - .add_point(chain, &out_fence, sync.timeline_value); - } else { - sync.syncobj.replace_fence(Some(&out_fence)); - } - } - - Ok(()) - } -} - -#[versions(AGX_NEO)] -impl Drop for Queue::ver { - fn drop(&mut self) { - mod_dev_dbg!(self.dev, "[Queue {}] Dropping queue\n", self.inner.id); - } -} diff --git a/drivers/gpu/drm/asahi-neo/queue/render.rs b/drivers/gpu/drm/asahi-neo/queue/render.rs deleted file mode 100644 index 586a11f4708a4e..00000000000000 --- a/drivers/gpu/drm/asahi-neo/queue/render.rs +++ /dev/null @@ -1,1400 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -#![allow(clippy::unusual_byte_groupings)] - -//! Render work queue. -//! -//! A render queue consists of two underlying WorkQueues, one for vertex and one for fragment work. -//! This module is in charge of creating all of the firmware structures required to submit 3D -//! rendering work to the GPU, based on the userspace command buffer. - -use super::common; -use crate::alloc::Allocator; -use crate::debug::*; -use crate::fw::types::*; -use crate::gpu::GpuManager; -use crate::util::*; -use crate::{ - buffer, - file, - fw, - gpu, - microseq, // -}; -use crate::{ - inner_ptr, - inner_weak_ptr, // -}; -use core::sync::atomic::Ordering; -use kernel::dma_fence::RawDmaFence; -use kernel::drm_neo::sched::Job; -use kernel::prelude::*; -use kernel::sync::Arc; -use kernel::uapi; -use kernel::xarray; - -const DEBUG_CLASS: DebugFlags = DebugFlags::Render; - -/// Tiling/Vertex control bit to disable using more than one GPU cluster. This results in decreased -/// throughput but also less latency, which is probably desirable for light vertex loads where the -/// overhead of clustering/merging would exceed the time it takes to just run the job on one -/// cluster. -const TILECTL_DISABLE_CLUSTERING: u32 = 1u32 << 0; - -#[versions(AGX_NEO)] -impl super::QueueInner::ver { - /// Get the appropriate tiling parameters for a given userspace command buffer. - fn get_tiling_params( - cmdbuf: &uapi::drm_asahi_neo_cmd_render, - num_clusters: u32, - ) -> Result { - let width: u32 = cmdbuf.width_px as u32; - let height: u32 = cmdbuf.height_px as u32; - let layers: u32 = cmdbuf.layers as u32; - - if layers == 0 || layers > 2048 { - cls_pr_debug!(Errors, "Layer count invalid ({})\n", layers); - return Err(EINVAL); - } - - // This is overflow safe: all these calculations are done in u32. - // At 64Kx64K max dimensions above, this is 2**32 pixels max. - // In terms of tiles that are always larger than one pixel, - // this can never overflow. Note that real actual dimensions - // are limited to 16K * 16K below anyway. - // - // Once we multiply by the layer count, then we need to check - // for overflow or use u64. - - let tile_width = 32u32; - let tile_height = 32u32; - - let utile_width = cmdbuf.utile_width_px as u32; - let utile_height = cmdbuf.utile_height_px as u32; - - match (utile_width, utile_height) { - (32, 32) | (32, 16) | (16, 16) => (), - _ => { - cls_pr_debug!( - Errors, - "uTile size invalid ({} x {})\n", - utile_width, - utile_height - ); - return Err(EINVAL); - } - }; - - let utiles_per_tile_x = tile_width / utile_width; - let utiles_per_tile_y = tile_height / utile_height; - - let utiles_per_tile = utiles_per_tile_x * utiles_per_tile_y; - - let tiles_x = width.div_ceil(tile_width); - let tiles_y = height.div_ceil(tile_height); - let tiles = tiles_x * tiles_y; - - let mtiles_x = 4u32; - let mtiles_y = 4u32; - let mtiles = mtiles_x * mtiles_y; - - let tiles_per_mtile_x = align(tiles_x.div_ceil(mtiles_x), 4); - let tiles_per_mtile_y = align(tiles_y.div_ceil(mtiles_y), 4); - let tiles_per_mtile = tiles_per_mtile_x * tiles_per_mtile_y; - - let mtile_x1 = tiles_per_mtile_x; - let mtile_x2 = 2 * tiles_per_mtile_x; - let mtile_x3 = 3 * tiles_per_mtile_x; - - let mtile_y1 = tiles_per_mtile_y; - let mtile_y2 = 2 * tiles_per_mtile_y; - let mtile_y3 = 3 * tiles_per_mtile_y; - - let rgn_entry_size = 5; - // Macrotile stride in 32-bit words - let rgn_size = align(rgn_entry_size * tiles_per_mtile * utiles_per_tile, 4) / 4; - let tilemap_size = (4 * rgn_size * mtiles) as usize * layers as usize; - - let tpc_entry_size = 8; - // TPC stride in 32-bit words - let tpc_mtile_stride = tpc_entry_size * utiles_per_tile * tiles_per_mtile / 4; - let tpc_size = - (4 * tpc_mtile_stride * mtiles) as usize * layers as usize * num_clusters as usize; - - // No idea where this comes from, but it fits what macOS does... - // GUESS: Number of 32K heap blocks to fit a 5-byte region header/pointer per tile? - // That would make a ton of sense... - let meta1_layer_stride = if num_clusters > 1 { - (align(tiles_x, 2) * align(tiles_y, 4) * utiles_per_tile).div_ceil(0x1980) - } else { - 0 - }; - - let mut min_tvb_blocks = align((tiles_x * tiles_y).div_ceil(128), 8); - - if num_clusters > 1 { - min_tvb_blocks = min_tvb_blocks.max(7 + 2 * layers); - } - - Ok(buffer::TileInfo { - tiles_x, - tiles_y, - tiles, - utile_width, - utile_height, - //mtiles_x, - //mtiles_y, - tiles_per_mtile_x, - tiles_per_mtile_y, - //tiles_per_mtile, - utiles_per_mtile_x: tiles_per_mtile_x * utiles_per_tile_x, - utiles_per_mtile_y: tiles_per_mtile_y * utiles_per_tile_y, - //utiles_per_mtile: tiles_per_mtile * utiles_per_tile, - tilemap_size, - tpc_size, - meta1_layer_stride, - #[ver(G < G14X)] - meta1_blocks: meta1_layer_stride * (cmdbuf.layers as u32), - #[ver(G >= G14X)] - meta1_blocks: meta1_layer_stride, - layermeta_size: if layers > 1 { 0x100 } else { 0 }, - min_tvb_blocks: min_tvb_blocks as usize, - params: fw::vertex::raw::TilingParameters { - rgn_size, - unk_4: 0x88, - ppp_ctrl: cmdbuf.ppp_ctrl, - x_max: (width - 1) as u16, - y_max: (height - 1) as u16, - te_screen: ((tiles_y - 1) << 12) | (tiles_x - 1), - te_mtile1: mtile_x3 | (mtile_x2 << 9) | (mtile_x1 << 18), - te_mtile2: mtile_y3 | (mtile_y2 << 9) | (mtile_y1 << 18), - tiles_per_mtile, - tpc_stride: tpc_mtile_stride, - unk_24: 0x100, - unk_28: if layers > 1 { - 0xe000 | (layers - 1) - } else { - 0x8000 - }, - helper_cfg: cmdbuf.vertex_helper.cfg, - __pad: Default::default(), - }, - }) - } - - /// Submit work to a render queue. - pub(super) fn submit_render( - &self, - job: &mut Job, - cmdbuf: &uapi::drm_asahi_neo_cmd_render, - vertex_attachments: µseq::Attachments, - fragment_attachments: µseq::Attachments, - objects: Pin<&xarray::XArray>>, - id: u64, - flush_stamps: bool, - ) -> Result { - mod_dev_dbg!(self.dev, "[Submission {}] Render!\n", id); - - if cmdbuf.flags - & !(uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_VERTEX_SCRATCH - | uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES - | uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_NO_VERTEX_CLUSTERING - | uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT) as u32 - != 0 - { - cls_pr_debug!(Errors, "Invalid flags ({:#x})\n", cmdbuf.flags); - return Err(EINVAL); - } - - if cmdbuf.width_px == 0 - || cmdbuf.height_px == 0 - || cmdbuf.width_px > 16384 - || cmdbuf.height_px > 16384 - { - cls_pr_debug!( - Errors, - "Invalid dimensions ({}x{})\n", - cmdbuf.width_px, - cmdbuf.height_px - ); - return Err(EINVAL); - } - - let mut vtx_user_timestamps: fw::job::UserTimestamps = Default::default(); - let mut frg_user_timestamps: fw::job::UserTimestamps = Default::default(); - - vtx_user_timestamps.start = common::get_timestamp_object(objects, cmdbuf.ts_vtx.start)?; - vtx_user_timestamps.end = common::get_timestamp_object(objects, cmdbuf.ts_vtx.end)?; - frg_user_timestamps.start = common::get_timestamp_object(objects, cmdbuf.ts_frag.start)?; - frg_user_timestamps.end = common::get_timestamp_object(objects, cmdbuf.ts_frag.end)?; - - let gpu = match (*self.dev) - .gpu - .as_any() - .downcast_ref::() - { - Some(gpu) => gpu, - None => { - dev_crit!(self.dev.as_ref(), "GpuManager mismatched with Queue!\n"); - return Err(EIO); - } - }; - - let nclusters = gpu.get_dyncfg().id.num_clusters; - - // Can be set to false to disable clustering (for simpler jobs), but then the - // core masks below should be adjusted to cover a single rolling cluster. - let mut clustering = nclusters > 1; - - if debug_enabled(debug::DebugFlags::DisableClustering) - || cmdbuf.flags - & uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_NO_VERTEX_CLUSTERING as u32 - != 0 - { - clustering = false; - } - - #[ver(G != G14)] - let tiling_control = { - let render_cfg = gpu.get_cfg().render; - let mut tiling_control = render_cfg.tiling_control; - - if !clustering { - tiling_control |= TILECTL_DISABLE_CLUSTERING; - } - tiling_control - }; - - let mut alloc = gpu.alloc(); - let kalloc = &mut *alloc; - - // This sequence number increases per new client/VM? assigned to some slot, - // but it's unclear *which* slot... - let slot_client_seq: u8 = (self.id & 0xff) as u8; - - let tile_info = Self::get_tiling_params(&cmdbuf, if clustering { nclusters } else { 1 })?; - - let buffer = &self.buffer; - let notifier = self.notifier.clone(); - - let tvb_autogrown = buffer.auto_grow()?; - if tvb_autogrown { - let new_size = buffer.block_count() as usize; - cls_dev_dbg!( - TVBStats, - &self.dev, - "[Submission {}] TVB grew to {} bytes ({} blocks) due to overflows\n", - id, - new_size * buffer::BLOCK_SIZE, - new_size, - ); - } - - let tvb_grown = buffer.ensure_blocks(tile_info.min_tvb_blocks)?; - if tvb_grown { - cls_dev_dbg!( - TVBStats, - &self.dev, - "[Submission {}] TVB grew to {} bytes ({} blocks) due to dimensions ({}x{})\n", - id, - tile_info.min_tvb_blocks * buffer::BLOCK_SIZE, - tile_info.min_tvb_blocks, - cmdbuf.width_px, - cmdbuf.height_px - ); - } - - let scene = Arc::new(buffer.new_scene(kalloc, &tile_info)?, GFP_KERNEL)?; - - let vm_bind = job.vm_bind.clone(); - - mod_dev_dbg!( - self.dev, - "[Submission {}] VM slot = {}\n", - id, - vm_bind.slot() - ); - - let ev_vtx = job.get_vtx()?.event_info(); - let ev_frag = job.get_frag()?.event_info(); - - mod_dev_dbg!( - self.dev, - "[Submission {}] Vert event #{} -> {:#x?}\n", - id, - ev_vtx.slot, - ev_vtx.value.next(), - ); - mod_dev_dbg!( - self.dev, - "[Submission {}] Frag event #{} -> {:#x?}\n", - id, - ev_frag.slot, - ev_frag.value.next(), - ); - - let uuid_3d = 0; - let uuid_ta = 0; - - mod_dev_dbg!( - self.dev, - "[Submission {}] Vert UUID = {:#x?}\n", - id, - uuid_ta - ); - mod_dev_dbg!( - self.dev, - "[Submission {}] Frag UUID = {:#x?}\n", - id, - uuid_3d - ); - - let fence = job.fence.clone(); - let frag_job = job.get_frag()?; - - mod_dev_dbg!(self.dev, "[Submission {}] Create Barrier\n", id); - let barrier = kalloc.private.new_init( - pin_init::zeroed::(), - |_inner, _p| { - try_init!(fw::workqueue::raw::Barrier { - tag: fw::workqueue::CommandType::Barrier, - wait_stamp: ev_vtx.fw_stamp_pointer, - wait_value: ev_vtx.value.next(), - wait_slot: ev_vtx.slot, - stamp_self: ev_frag.value.next(), - uuid: uuid_3d, - external_barrier: 0, - internal_barrier_type: 0, - padding: Default::default(), - }) - }, - )?; - - mod_dev_dbg!(self.dev, "[Submission {}] Add Barrier\n", id); - frag_job.add(barrier, vm_bind.slot())?; - - let timestamps = Arc::new( - kalloc.shared.new_default::()?, - GFP_KERNEL, - )?; - - let unk1 = false; - - let mut tile_config: u64 = 0; - if !unk1 { - tile_config |= 0x280; - } - if cmdbuf.layers > 1 { - tile_config |= 1; - } - if cmdbuf.flags & uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES as u32 - != 0 - { - tile_config |= 0x10000; - } - - let samples_log2 = match cmdbuf.samples { - 1 => 0, - 2 => 1, - 4 => 2, - _ => { - cls_pr_debug!(Errors, "Invalid sample count {}\n", cmdbuf.samples); - return Err(EINVAL); - } - }; - - let utile_config = ((tile_info.utile_width / 16) << 12) - | ((tile_info.utile_height / 16) << 14) - | samples_log2; - - // Calculate the number of 2KiB blocks to allocate per utile. This is - // just a bit of dimensional analysis. - let pixels_per_utile: u32 = - (cmdbuf.utile_width_px as u32) * (cmdbuf.utile_height_px as u32); - let samples_per_utile: u32 = pixels_per_utile << samples_log2; - let utile_size_bytes: u32 = (cmdbuf.sample_size_B as u32) * samples_per_utile; - let block_size_bytes: u32 = 2048; - let blocks_per_utile: u32 = utile_size_bytes.div_ceil(block_size_bytes); - - #[ver(G >= G14X)] - let frg_tilecfg = 0x0000000_00036011 - | (((tile_info.tiles_x - 1) as u64) << 44) - | (((tile_info.tiles_y - 1) as u64) << 53) - | (if unk1 { 0 } else { 0x20_00000000 }) - | (if cmdbuf.layers > 1 { 0x1_00000000 } else { 0 }) - | ((utile_config as u64 & 0xf000) << 28); - - // TODO: check - #[ver(V >= V13_0B4)] - let count_frag = self.counter.fetch_add(2, Ordering::Relaxed); - #[ver(V >= V13_0B4)] - let count_vtx = count_frag + 1; - - // Unknowns handling - - #[ver(G >= G14)] - let g14_unk = 0x4040404; - #[ver(G < G14)] - let g14_unk = 0; - #[ver(G < G14X)] - let frg_unk_140 = 0x8c60; - let frg_unk_158 = 0x1c; - #[ver(G >= G14)] - let load_bgobjvals = cmdbuf.isp_bgobjvals as u64; - #[ver(G < G14)] - let load_bgobjvals = cmdbuf.isp_bgobjvals as u64 | 0x400; - let reload_zlsctrl = cmdbuf.zls_ctrl; - let iogpu_unk54 = 0x3a0012006b0003; - let iogpu_unk56 = 1; - #[ver(G < G14)] - let tiling_control_2 = 0; - #[ver(G >= G14X)] - let tiling_control_2 = 4; - #[ver(G >= G14X)] - let vtx_unk_f0 = 0x1c; - #[ver(G < G14)] - let vtx_unk_f0 = 0x1c + (align(tile_info.meta1_blocks, 4) as u64); - let vtx_unk_118 = 0x1c; - - // DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT chosen to match hardware bit. - let isp_ctl = 0xc000u32 - | (cmdbuf.flags & uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT as u32); - - // Always allow preemption at the UAPI level - let no_preemption = false; - - mod_dev_dbg!(self.dev, "[Submission {}] Create Frag\n", id); - let frag = GpuObject::new_init_prealloc( - kalloc.gpu_ro.alloc_object()?, - |ptr: GpuWeakPointer| { - let scene = scene.clone(); - let notifier = notifier.clone(); - let vm_bind = vm_bind.clone(); - let timestamps = timestamps.clone(); - let private = &mut kalloc.private; - try_init!(fw::fragment::RunFragment::ver { - micro_seq: { - let mut builder = microseq::Builder::new(); - - let stats = inner_weak_ptr!( - gpu.initdata.runtime_pointers.stats.frag.weak_pointer(), - stats - ); - - let start_frag = builder.add(microseq::StartFragment::ver { - header: microseq::op::StartFragment::HEADER, - #[ver(G < G14X)] - job_params2: Some(inner_weak_ptr!(ptr, job_params2)), - #[ver(G < G14X)] - job_params1: Some(inner_weak_ptr!(ptr, job_params1)), - #[ver(G >= G14X)] - job_params1: None, - #[ver(G >= G14X)] - job_params2: None, - #[ver(G >= G14X)] - registers: inner_weak_ptr!(ptr, registers), - scene: scene.gpu_pointer(), - stats, - busy_flag: inner_weak_ptr!(ptr, busy_flag), - tvb_overflow_count: inner_weak_ptr!(ptr, tvb_overflow_count), - unk_pointer: inner_weak_ptr!(ptr, unk_pointee), - work_queue: ev_frag.info_ptr, - work_item: ptr, - vm_slot: vm_bind.slot(), - unk_50: 0x1, // fixed - event_generation: self.id as u32, - buffer_slot: scene.slot(), - sync_grow: 0, - event_seq: U64(ev_frag.event_seq), - unk_68: 0, - unk_758_flag: inner_weak_ptr!(ptr, unk_758_flag), - unk_job_buf: inner_weak_ptr!(ptr, unk_buf_0), - #[ver(V >= V13_3)] - unk_7c_0: U64(0), - unk_7c: 0, - unk_80: 0, - unk_84: unk1.into(), - uuid: uuid_3d, - attachments: *fragment_attachments, - padding: 0, - #[ver(V >= V13_0B4)] - counter: U64(count_frag), - #[ver(V >= V13_0B4)] - notifier_buf: inner_weak_ptr!(notifier.weak_pointer(), state.unk_buf), - })?; - - if frg_user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(true), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.start_addr), - work_queue: ev_frag.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, unk_ts), - uuid: uuid_3d, - unk_30_padding: 0, - })?; - } - - #[ver(G < G14X)] - builder.add(microseq::WaitForIdle { - header: microseq::op::WaitForIdle::new(microseq::Pipe::Fragment), - })?; - #[ver(G >= G14X)] - builder.add(microseq::WaitForIdle2 { - header: microseq::op::WaitForIdle2::HEADER, - })?; - - if frg_user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(false), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.end_addr), - work_queue: ev_frag.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, unk_ts), - uuid: uuid_3d, - unk_30_padding: 0, - })?; - } - - let off = builder.offset_to(start_frag); - builder.add(microseq::FinalizeFragment::ver { - header: microseq::op::FinalizeFragment::HEADER, - uuid: uuid_3d, - unk_8: 0, - fw_stamp: ev_frag.fw_stamp_pointer, - stamp_value: ev_frag.value.next(), - unk_18: 0, - scene: scene.weak_pointer(), - buffer: scene.weak_buffer_pointer(), - unk_2c: U64(1), - stats, - unk_pointer: inner_weak_ptr!(ptr, unk_pointee), - busy_flag: inner_weak_ptr!(ptr, busy_flag), - work_queue: ev_frag.info_ptr, - work_item: ptr, - vm_slot: vm_bind.slot(), - unk_60: 0, - unk_758_flag: inner_weak_ptr!(ptr, unk_758_flag), - #[ver(V >= V13_3)] - unk_6c_0: U64(0), - unk_6c: U64(0), - unk_74: U64(0), - unk_7c: U64(0), - unk_84: U64(0), - unk_8c: U64(0), - #[ver(G == G14 && V < V13_0B4)] - unk_8c_g14: U64(0), - restart_branch_offset: off, - has_attachments: (fragment_attachments.count > 0) as u32, - #[ver(V >= V13_0B4)] - unk_9c: Default::default(), - })?; - - builder.add(microseq::RetireStamp { - header: microseq::op::RetireStamp::HEADER, - })?; - - builder.build(private)? - }, - notifier, - scene, - vm_bind, - aux_fb: self.ualloc.lock().array_empty_tagged(0x8000, b"AXFB")?, - timestamps, - user_timestamps: frg_user_timestamps, - }) - }, - |inner, _ptr| { - let vm_slot = vm_bind.slot(); - let aux_fb_info = fw::fragment::raw::AuxFBInfo::ver { - isp_ctl: isp_ctl, - unk2: 0, - width: cmdbuf.width_px as u32, - height: cmdbuf.height_px as u32, - #[ver(V >= V13_0B4)] - unk3: U64(0x100000), - }; - - try_init!(fw::fragment::raw::RunFragment::ver { - tag: fw::workqueue::CommandType::RunFragment, - #[ver(V >= V13_0B4)] - counter: U64(count_frag), - vm_slot, - unk_8: 0, - microsequence: inner.micro_seq.gpu_pointer(), - microsequence_size: inner.micro_seq.len() as u32, - notifier: inner.notifier.gpu_pointer(), - buffer: inner.scene.buffer_pointer(), - scene: inner.scene.gpu_pointer(), - unk_buffer_buf: inner.scene.kernel_buffer_pointer(), - tvb_tilemap: inner.scene.tvb_tilemap_pointer(), - ppp_multisamplectl: U64(cmdbuf.ppp_multisamplectl), - samples: cmdbuf.samples as u32, - tiles_per_mtile_y: tile_info.tiles_per_mtile_y as u16, - tiles_per_mtile_x: tile_info.tiles_per_mtile_x as u16, - unk_50: U64(0), - unk_58: U64(0), - isp_merge_upper_x: F32::from_bits(cmdbuf.isp_merge_upper_x), - isp_merge_upper_y: F32::from_bits(cmdbuf.isp_merge_upper_y), - unk_68: U64(0), - tile_count: U64(tile_info.tiles as u64), - #[ver(G < G14X)] - job_params1 <- try_init!(fw::fragment::raw::JobParameters1::ver { - utile_config, - unk_4: 0, - bg: fw::fragment::raw::BackgroundProgram { - rsrc_spec: U64(cmdbuf.bg.rsrc_spec as u64), - address: U64(cmdbuf.bg.usc as u64), - }, - ppp_multisamplectl: U64(cmdbuf.ppp_multisamplectl), - isp_scissor_base: U64(cmdbuf.isp_scissor_base), - isp_dbias_base: U64(cmdbuf.isp_dbias_base), - isp_oclqry_base: U64(cmdbuf.isp_oclqry_base), - aux_fb_info, - isp_zls_pixels: U64(cmdbuf.isp_zls_pixels as u64), - zls_ctrl: U64(cmdbuf.zls_ctrl), - #[ver(G >= G14)] - unk_58_g14_0: U64(g14_unk), - #[ver(G >= G14)] - unk_58_g14_8: U64(0), - z_load: U64(cmdbuf.depth.base), - z_store: U64(cmdbuf.depth.base), - s_load: U64(cmdbuf.stencil.base), - s_store: U64(cmdbuf.stencil.base), - #[ver(G >= G14)] - unk_68_g14_0: Default::default(), - z_load_stride: U64(cmdbuf.depth.stride as u64), - z_store_stride: U64(cmdbuf.depth.stride as u64), - s_load_stride: U64(cmdbuf.stencil.stride as u64), - s_store_stride: U64(cmdbuf.stencil.stride as u64), - z_load_comp: U64(cmdbuf.depth.comp_base), - z_load_comp_stride: U64(cmdbuf.depth.comp_stride as u64), - z_store_comp: U64(cmdbuf.depth.comp_base), - z_store_comp_stride: U64(cmdbuf.depth.comp_stride as u64), - s_load_comp: U64(cmdbuf.stencil.comp_base), - s_load_comp_stride: U64(cmdbuf.stencil.comp_stride as u64), - s_store_comp: U64(cmdbuf.stencil.comp_base), - s_store_comp_stride: U64(cmdbuf.stencil.comp_stride as u64), - tvb_tilemap: inner.scene.tvb_tilemap_pointer(), - tvb_layermeta: inner.scene.tvb_layermeta_pointer(), - mtile_stride_dwords: U64((4 * tile_info.params.rgn_size as u64) << 24), - tvb_heapmeta: inner.scene.tvb_heapmeta_pointer(), - tile_config: U64(tile_config), - aux_fb: inner.aux_fb.gpu_pointer(), - unk_108: Default::default(), - usc_exec_base_isp: U64(self.usc_exec_base), - unk_140: U64(frg_unk_140), - helper_program: cmdbuf.fragment_helper.binary, - unk_14c: 0, - helper_arg: U64(cmdbuf.fragment_helper.data), - unk_158: U64(frg_unk_158), - unk_160: U64(0), - __pad: Default::default(), - #[ver(V < V13_0B4)] - __pad1: Default::default(), - }), - #[ver(G < G14X)] - job_params2 <- try_init!(fw::fragment::raw::JobParameters2 { - eot_rsrc_spec: cmdbuf.eot.rsrc_spec, - eot_usc: cmdbuf.eot.usc, - unk_8: 0x0, - unk_c: 0x0, - isp_merge_upper_x: F32::from_bits(cmdbuf.isp_merge_upper_x), - isp_merge_upper_y: F32::from_bits(cmdbuf.isp_merge_upper_y), - unk_18: U64(0x0), - utiles_per_mtile_y: tile_info.utiles_per_mtile_y as u16, - utiles_per_mtile_x: tile_info.utiles_per_mtile_x as u16, - unk_24: 0x0, - tile_counts: ((tile_info.tiles_y - 1) << 12) | (tile_info.tiles_x - 1), - tib_blocks: blocks_per_utile, - isp_bgobjdepth: cmdbuf.isp_bgobjdepth, - // TODO: does this flag need to be exposed to userspace? - isp_bgobjvals: load_bgobjvals as u32, - unk_38: 0x0, - unk_3c: 0x1, - helper_cfg: cmdbuf.fragment_helper.cfg, - __pad: Default::default(), - }), - #[ver(G >= G14X)] - registers: fw::job::raw::RegisterArray::new( - inner_weak_ptr!(_ptr, registers.registers), - |r| { - r.add(0x1739, 1); - r.add(0x10009, utile_config.into()); - r.add(0x15379, cmdbuf.eot.rsrc_spec.into()); - r.add(0x15381, cmdbuf.eot.usc.into()); - r.add(0x15369, cmdbuf.bg.rsrc_spec.into()); - r.add(0x15371, cmdbuf.bg.usc.into()); - r.add(0x15131, cmdbuf.isp_merge_upper_x.into()); - r.add(0x15139, cmdbuf.isp_merge_upper_y.into()); - r.add(0x100a1, 0); - r.add(0x15069, 0); - r.add(0x15071, 0); // pointer - r.add(0x16058, 0); - r.add(0x10019, cmdbuf.ppp_multisamplectl); - let isp_mtile_size = (tile_info.utiles_per_mtile_y - | (tile_info.utiles_per_mtile_x << 16)) - .into(); - r.add(0x100b1, isp_mtile_size); // ISP_MTILE_SIZE - r.add(0x16030, isp_mtile_size); // ISP_MTILE_SIZE - r.add( - 0x100d9, - (((tile_info.tiles_y - 1) << 12) | (tile_info.tiles_x - 1)).into(), - ); // TE_SCREEN - r.add(0x16098, inner.scene.tvb_heapmeta_pointer().into()); - r.add(0x15109, cmdbuf.isp_scissor_base); // ISP_SCISSOR_BASE - r.add(0x15101, cmdbuf.isp_dbias_base); // ISP_DBIAS_BASE - r.add(0x15021, isp_ctl.into()); // aux_fb_info.unk_1 - r.add( - 0x15211, - ((cmdbuf.height_px as u64) << 32) | cmdbuf.width_px as u64, - ); // aux_fb_info.{width, heigh - r.add(0x15049, 0x100000); // s2.aux_fb_info.unk3 - r.add(0x10051, blocks_per_utile.into()); // s1.unk_2c - r.add(0x15321, cmdbuf.isp_zls_pixels.into()); // ISP_ZLS_PIXELS - r.add(0x15301, cmdbuf.isp_bgobjdepth.into()); // ISP_BGOBJDEPTH - r.add(0x15309, load_bgobjvals); // ISP_BGOBJVALS - r.add(0x15311, cmdbuf.isp_oclqry_base); // ISP_OCLQRY_BASE - r.add(0x15319, cmdbuf.zls_ctrl); // ISP_ZLSCTL - r.add(0x15349, g14_unk); // s2.unk_58_g14_0 - r.add(0x15351, 0); // s2.unk_58_g14_8 - r.add(0x15329, cmdbuf.depth.base); // ISP_ZLOAD_BASE - r.add(0x15331, cmdbuf.depth.base); // ISP_ZSTORE_BASE - r.add(0x15339, cmdbuf.stencil.base); // ISP_STENCIL_LOAD_BASE - r.add(0x15341, cmdbuf.stencil.base); // ISP_STENCIL_STORE_BASE - r.add(0x15231, 0); - r.add(0x15221, 0); - r.add(0x15239, 0); - r.add(0x15229, 0); - r.add(0x15401, cmdbuf.depth.stride as u64); // load - r.add(0x15421, cmdbuf.depth.stride as u64); // store - r.add(0x15409, cmdbuf.stencil.stride as u64); // load - r.add(0x15429, cmdbuf.stencil.stride as u64); - r.add(0x153c1, cmdbuf.depth.comp_base); // load - r.add(0x15411, cmdbuf.depth.comp_stride as u64); // load - r.add(0x153c9, cmdbuf.depth.comp_base); // store - r.add(0x15431, cmdbuf.depth.comp_stride as u64); // store - r.add(0x153d1, cmdbuf.stencil.comp_base); // load - r.add(0x15419, cmdbuf.stencil.comp_stride as u64); // load - r.add(0x153d9, cmdbuf.stencil.comp_base); // store - r.add(0x15439, cmdbuf.stencil.comp_stride as u64); // store - r.add(0x16429, inner.scene.tvb_tilemap_pointer().into()); - r.add(0x16060, inner.scene.tvb_layermeta_pointer().into()); - r.add(0x16431, (4 * tile_info.params.rgn_size as u64) << 24); // ISP_RGN? - r.add(0x10039, tile_config); // tile_config ISP_CTL? - r.add(0x16451, 0x0); // ISP_RENDER_ORIGIN - r.add(0x11821, cmdbuf.fragment_helper.binary.into()); - r.add(0x11829, cmdbuf.fragment_helper.data); - r.add(0x11f79, cmdbuf.fragment_helper.cfg.into()); - r.add(0x15359, 0); - r.add(0x10069, self.usc_exec_base); // frag; USC_EXEC_BASE_ISP - r.add(0x16020, 0); - r.add(0x16461, inner.aux_fb.gpu_pointer().into()); - r.add(0x16090, inner.aux_fb.gpu_pointer().into()); - r.add(0x120a1, frg_unk_158); - r.add(0x160a8, 0); - r.add(0x16068, frg_tilecfg); - r.add(0x160b8, 0x0); - /* - r.add(0x10201, 0x100); // Some kind of counter?? Does this matter? - r.add(0x10428, 0x100); // Some kind of counter?? Does this matter? - r.add(0x1c838, 1); // ? - r.add(0x1ca28, 0x1502960f00); // ?? - r.add(0x1731, 0x1); // ?? - */ - } - ), - job_params3 <- try_init!(fw::fragment::raw::JobParameters3::ver { - isp_dbias_base: fw::fragment::raw::ArrayAddr { - ptr: U64(cmdbuf.isp_dbias_base), - unk_padding: U64(0), - }, - isp_scissor_base: fw::fragment::raw::ArrayAddr { - ptr: U64(cmdbuf.isp_scissor_base), - unk_padding: U64(0), - }, - isp_oclqry_base: U64(cmdbuf.isp_oclqry_base), - unk_118: U64(0x0), - unk_120: Default::default(), - unk_partial_bg: fw::fragment::raw::BackgroundProgram { - rsrc_spec: U64(cmdbuf.partial_bg.rsrc_spec as u64), - address: U64(cmdbuf.partial_bg.usc as u64), - }, - unk_258: U64(0), - unk_260: U64(0), - unk_268: U64(0), - unk_270: U64(0), - partial_bg: fw::fragment::raw::BackgroundProgram { - rsrc_spec: U64(cmdbuf.partial_bg.rsrc_spec as u64), - address: U64(cmdbuf.partial_bg.usc as u64), - }, - zls_ctrl: U64(reload_zlsctrl), - unk_290: U64(g14_unk), - z_load: U64(cmdbuf.depth.base), - z_partial_stride: U64(cmdbuf.depth.stride as u64), - z_partial_comp_stride: U64(cmdbuf.depth.comp_stride as u64), - z_store: U64(cmdbuf.depth.base), - z_partial: U64(cmdbuf.depth.base), - z_partial_comp: U64(cmdbuf.depth.comp_base), - s_load: U64(cmdbuf.stencil.base), - s_partial_stride: U64(cmdbuf.stencil.stride as u64), - s_partial_comp_stride: U64(cmdbuf.stencil.comp_stride as u64), - s_store: U64(cmdbuf.stencil.base), - s_partial: U64(cmdbuf.stencil.base), - s_partial_comp: U64(cmdbuf.stencil.comp_base), - unk_2f8: Default::default(), - tib_blocks: blocks_per_utile, - unk_30c: 0x0, - aux_fb_info, - tile_config: U64(tile_config), - unk_328_padding: Default::default(), - unk_partial_eot: fw::fragment::raw::EotProgram::new( - cmdbuf.partial_eot.rsrc_spec, - cmdbuf.partial_eot.usc - ), - partial_eot: fw::fragment::raw::EotProgram::new( - cmdbuf.partial_eot.rsrc_spec, - cmdbuf.partial_eot.usc - ), - isp_bgobjdepth: cmdbuf.isp_bgobjdepth, - isp_bgobjvals: cmdbuf.isp_bgobjvals, - sample_size: cmdbuf.sample_size_B as u32, - unk_37c: 0x0, - unk_380: U64(0x0), - unk_388: U64(0x0), - #[ver(V >= V13_0B4)] - unk_390_0: U64(0x0), - isp_zls_pixels: U64(cmdbuf.isp_zls_pixels as u64), - }), - unk_758_flag: 0, - unk_75c_flag: 0, - unk_buf: Default::default(), - busy_flag: 0, - tvb_overflow_count: 0, - unk_878: 0, - encoder_params <- try_init!(fw::job::raw::EncoderParams { - // Maybe set when reloading z/s? - unk_8: 0, - sync_grow: 0, - unk_10: 0x0, // fixed - encoder_id: 0, - unk_18: 0x0, // fixed - unk_mask: 0xffffffffu32, - sampler_array: U64(cmdbuf.sampler_heap), - sampler_count: cmdbuf.sampler_count as u32, - sampler_max: (cmdbuf.sampler_count as u32) + 1, - }), - process_empty_tiles: (cmdbuf.flags - & uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES as u32 - != 0) as u32, - // TODO: needs to be investigated - no_clear_pipeline_textures: 1, - // TODO: needs to be investigated - msaa_zs: 0, - unk_pointee: 0, - #[ver(V >= V13_3)] - unk_v13_3: 0, - meta <- try_init!(fw::job::raw::JobMeta { - unk_0: 0, - unk_2: 0, - no_preemption: no_preemption as u8, - stamp: ev_frag.stamp_pointer, - fw_stamp: ev_frag.fw_stamp_pointer, - stamp_value: ev_frag.value.next(), - stamp_slot: ev_frag.slot, - evctl_index: 0, // fixed - flush_stamps: flush_stamps as u32, - uuid: uuid_3d, - event_seq: ev_frag.event_seq as u32, - }), - unk_after_meta: unk1.into(), - unk_buf_0: U64(0), - unk_buf_8: U64(0), - #[ver(G < G14X)] - unk_buf_10: U64(1), - #[ver(G >= G14X)] - unk_buf_10: U64(0), - command_time: U64(0), - timestamp_pointers <- try_init!(fw::job::raw::TimestampPointers { - start_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), frag.start)), - end_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), frag.end)), - }), - user_timestamp_pointers: inner.user_timestamps.pointers()?, - client_sequence: slot_client_seq, - pad_925: Default::default(), - unk_928: 0, - unk_92c: 0, - #[ver(V >= V13_0B4)] - unk_ts: U64(0), - #[ver(V >= V13_0B4)] - unk_92d_8: Default::default(), - }) - }, - )?; - - mod_dev_dbg!(self.dev, "[Submission {}] Add Frag\n", id); - fence.add_command(); - - frag_job.add_cb(frag, vm_bind.slot(), move |error| { - if let Some(err) = error { - fence.set_error(err.into()); - } - - fence.command_complete(); - })?; - - let fence = job.fence.clone(); - let vtx_job = job.get_vtx()?; - - if scene.rebind() || tvb_grown || tvb_autogrown { - mod_dev_dbg!(self.dev, "[Submission {}] Create Bind Buffer\n", id); - let bind_buffer = kalloc.private.new_init( - { - let scene = scene.clone(); - try_init!(fw::buffer::InitBuffer::ver { scene }) - }, - |inner, _ptr| { - let vm_slot = vm_bind.slot(); - try_init!(fw::buffer::raw::InitBuffer::ver { - tag: fw::workqueue::CommandType::InitBuffer, - vm_slot, - buffer_slot: inner.scene.slot(), - unk_c: 0, - block_count: buffer.block_count(), - buffer: inner.scene.buffer_pointer(), - stamp_value: ev_vtx.value.next(), - }) - }, - )?; - - mod_dev_dbg!(self.dev, "[Submission {}] Add Bind Buffer\n", id); - vtx_job.add(bind_buffer, vm_bind.slot())?; - } - - mod_dev_dbg!(self.dev, "[Submission {}] Create Vertex\n", id); - let vtx = GpuObject::new_init_prealloc( - kalloc.gpu_ro.alloc_object()?, - |ptr: GpuWeakPointer| { - let scene = scene.clone(); - let vm_bind = vm_bind.clone(); - let timestamps = timestamps.clone(); - let private = &mut kalloc.private; - try_init!(fw::vertex::RunVertex::ver { - micro_seq: { - let mut builder = microseq::Builder::new(); - - let stats = inner_weak_ptr!( - gpu.initdata.runtime_pointers.stats.vtx.weak_pointer(), - stats - ); - - let start_vtx = builder.add(microseq::StartVertex::ver { - header: microseq::op::StartVertex::HEADER, - #[ver(G < G14X)] - tiling_params: Some(inner_weak_ptr!(ptr, tiling_params)), - #[ver(G < G14X)] - job_params1: Some(inner_weak_ptr!(ptr, job_params1)), - #[ver(G >= G14X)] - tiling_params: None, - #[ver(G >= G14X)] - job_params1: None, - #[ver(G >= G14X)] - registers: inner_weak_ptr!(ptr, registers), - buffer: scene.weak_buffer_pointer(), - scene: scene.weak_pointer(), - stats, - work_queue: ev_vtx.info_ptr, - vm_slot: vm_bind.slot(), - unk_38: 1, // fixed - event_generation: self.id as u32, - buffer_slot: scene.slot(), - unk_44: 0, - event_seq: U64(ev_vtx.event_seq), - unk_50: 0, - unk_pointer: inner_weak_ptr!(ptr, unk_pointee), - unk_job_buf: inner_weak_ptr!(ptr, unk_buf_0), - unk_64: 0x0, // fixed - unk_68: unk1.into(), - uuid: uuid_ta, - attachments: *vertex_attachments, - padding: 0, - #[ver(V >= V13_0B4)] - counter: U64(count_vtx), - #[ver(V >= V13_0B4)] - notifier_buf: inner_weak_ptr!(notifier.weak_pointer(), state.unk_buf), - #[ver(V < V13_0B4)] - unk_178: 0x0, // padding? - #[ver(V >= V13_0B4)] - unk_178: (!clustering) as u32, - })?; - - if vtx_user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(true), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.start_addr), - work_queue: ev_vtx.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, unk_ts), - uuid: uuid_ta, - unk_30_padding: 0, - })?; - } - - #[ver(G < G14X)] - builder.add(microseq::WaitForIdle { - header: microseq::op::WaitForIdle::new(microseq::Pipe::Vertex), - })?; - #[ver(G >= G14X)] - builder.add(microseq::WaitForIdle2 { - header: microseq::op::WaitForIdle2::HEADER, - })?; - - if vtx_user_timestamps.any() { - builder.add(microseq::Timestamp::ver { - header: microseq::op::Timestamp::new(false), - command_time: inner_weak_ptr!(ptr, command_time), - ts_pointers: inner_weak_ptr!(ptr, timestamp_pointers), - update_ts: inner_weak_ptr!(ptr, timestamp_pointers.end_addr), - work_queue: ev_vtx.info_ptr, - user_ts_pointers: inner_weak_ptr!(ptr, user_timestamp_pointers), - #[ver(V >= V13_0B4)] - unk_ts: inner_weak_ptr!(ptr, unk_ts), - uuid: uuid_ta, - unk_30_padding: 0, - })?; - } - - let off = builder.offset_to(start_vtx); - builder.add(microseq::FinalizeVertex::ver { - header: microseq::op::FinalizeVertex::HEADER, - scene: scene.weak_pointer(), - buffer: scene.weak_buffer_pointer(), - stats, - work_queue: ev_vtx.info_ptr, - vm_slot: vm_bind.slot(), - unk_28: 0x0, // fixed - unk_pointer: inner_weak_ptr!(ptr, unk_pointee), - unk_34: 0x0, // fixed - uuid: uuid_ta, - fw_stamp: ev_vtx.fw_stamp_pointer, - stamp_value: ev_vtx.value.next(), - unk_48: U64(0x0), // fixed - unk_50: 0x0, // fixed - unk_54: 0x0, // fixed - unk_58: U64(0x0), // fixed - unk_60: 0x0, // fixed - unk_64: 0x0, // fixed - unk_68: 0x0, // fixed - #[ver(G >= G14 && V < V13_0B4)] - unk_68_g14: U64(0), - restart_branch_offset: off, - has_attachments: (vertex_attachments.count > 0) as u32, - #[ver(V >= V13_0B4)] - unk_74: Default::default(), // Ventura - })?; - - builder.add(microseq::RetireStamp { - header: microseq::op::RetireStamp::HEADER, - })?; - builder.build(private)? - }, - notifier, - scene, - vm_bind, - timestamps, - user_timestamps: vtx_user_timestamps, - }) - }, - |inner, _ptr| { - let vm_slot = vm_bind.slot(); - #[ver(G < G14)] - let core_masks = gpu.core_masks_packed(); - - try_init!(fw::vertex::raw::RunVertex::ver { - tag: fw::workqueue::CommandType::RunVertex, - #[ver(V >= V13_0B4)] - counter: U64(count_vtx), - vm_slot, - unk_8: 0, - notifier: inner.notifier.gpu_pointer(), - buffer_slot: inner.scene.slot(), - unk_1c: 0, - buffer: inner.scene.buffer_pointer(), - scene: inner.scene.gpu_pointer(), - unk_buffer_buf: inner.scene.kernel_buffer_pointer(), - unk_34: 0, - #[ver(G < G14X)] - job_params1 <- try_init!(fw::vertex::raw::JobParameters1::ver { - unk_0: U64(if unk1 { 0 } else { 0x200 }), // sometimes 0 - unk_8: f32!(1e-20), // fixed - unk_c: f32!(1e-20), // fixed - tvb_tilemap: inner.scene.tvb_tilemap_pointer(), - #[ver(G < G14)] - tvb_cluster_tilemaps: inner.scene.cluster_tilemaps_pointer(), - tpc: inner.scene.tpc_pointer(), - tvb_heapmeta: inner.scene.tvb_heapmeta_pointer().or(0x8000_0000_0000_0000), - iogpu_unk_54: U64(iogpu_unk54), // fixed - iogpu_unk_56: U64(iogpu_unk56), // fixed - #[ver(G < G14)] - tvb_cluster_meta1: inner - .scene - .meta_1_pointer() - .map(|x| x.or((tile_info.meta1_layer_stride as u64) << 50)), - utile_config, - unk_4c: 0, - ppp_multisamplectl: U64(cmdbuf.ppp_multisamplectl), // fixed - tvb_layermeta: inner.scene.tvb_layermeta_pointer(), - #[ver(G < G14)] - tvb_cluster_layermeta: inner.scene.tvb_cluster_layermeta_pointer(), - #[ver(G < G14)] - core_mask: Array::new([ - *core_masks.first().unwrap_or(&0), - *core_masks.get(1).unwrap_or(&0), - ]), - preempt_buf1: inner.scene.preempt_buf_1_pointer(), - preempt_buf2: inner.scene.preempt_buf_2_pointer(), - unk_80: U64(0x1), // fixed - preempt_buf3: inner.scene.preempt_buf_3_pointer().or(0x4_0000_0000_0000), // check - vdm_ctrl_stream_base: U64(cmdbuf.vdm_ctrl_stream_base), - #[ver(G < G14)] - tvb_cluster_meta2: inner.scene.meta_2_pointer(), - #[ver(G < G14)] - tvb_cluster_meta3: inner.scene.meta_3_pointer(), - #[ver(G < G14)] - tiling_control, - #[ver(G < G14)] - unk_ac: tiling_control_2 as u32, // fixed - unk_b0: Default::default(), // fixed - usc_exec_base_ta: U64(self.usc_exec_base), - #[ver(G < G14)] - tvb_cluster_meta4: inner - .scene - .meta_4_pointer() - .map(|x| x.or(0x3000_0000_0000_0000)), - #[ver(G < G14)] - unk_f0: U64(vtx_unk_f0), - unk_f8: U64(0x8c60), // fixed - helper_program: cmdbuf.vertex_helper.binary, - unk_104: 0, - helper_arg: U64(cmdbuf.vertex_helper.data), - unk_110: Default::default(), // fixed - unk_118: vtx_unk_118 as u32, // fixed - __pad: Default::default(), - }), - #[ver(G < G14X)] - tiling_params: tile_info.params, - #[ver(G >= G14X)] - registers: fw::job::raw::RegisterArray::new( - inner_weak_ptr!(_ptr, registers.registers), - |r| { - r.add(0x10141, if unk1 { 0 } else { 0x200 }); // s2.unk_0 - r.add(0x1c039, inner.scene.tvb_tilemap_pointer().into()); - r.add(0x1c9c8, inner.scene.tvb_tilemap_pointer().into()); - - let cl_tilemaps_ptr = inner - .scene - .cluster_tilemaps_pointer() - .map_or(0, |a| a.into()); - r.add(0x1c041, cl_tilemaps_ptr); - r.add(0x1c9d0, cl_tilemaps_ptr); - r.add(0x1c0a1, inner.scene.tpc_pointer().into()); // TE_TPC_ADDR - - let tvb_heapmeta_ptr = inner - .scene - .tvb_heapmeta_pointer() - .or(0x8000_0000_0000_0000) - .into(); - r.add(0x1c031, tvb_heapmeta_ptr); - r.add(0x1c9c0, tvb_heapmeta_ptr); - r.add(0x1c051, iogpu_unk54); // iogpu_unk_54/55 - r.add(0x1c061, iogpu_unk56); // iogpu_unk_56 - r.add(0x10149, utile_config.into()); // s2.unk_48 utile_config - r.add(0x10139, cmdbuf.ppp_multisamplectl); // PPP_MULTISAMPLECTL - r.add(0x10111, inner.scene.preempt_buf_1_pointer().into()); - r.add(0x1c9b0, inner.scene.preempt_buf_1_pointer().into()); - r.add(0x10119, inner.scene.preempt_buf_2_pointer().into()); - r.add(0x1c9b8, inner.scene.preempt_buf_2_pointer().into()); - r.add(0x1c958, 1); // s2.unk_80 - r.add( - 0x1c950, - inner - .scene - .preempt_buf_3_pointer() - .or(0x4_0000_0000_0000) - .into(), - ); - r.add(0x1c930, 0); // VCE related addr, lsb to enable - r.add(0x1c880, cmdbuf.vdm_ctrl_stream_base); // VDM_CTRL_STREAM_BASE - r.add(0x1c898, 0x0); // if lsb set, faults in UL1C0, possibly missing addr. - r.add( - 0x1c948, - inner.scene.meta_2_pointer().map_or(0, |a| a.into()), - ); // tvb_cluster_meta2 - r.add( - 0x1c888, - inner.scene.meta_3_pointer().map_or(0, |a| a.into()), - ); // tvb_cluster_meta3 - r.add(0x1c890, tiling_control.into()); // tvb_tiling_control - r.add(0x1c918, tiling_control_2); - r.add(0x1c079, inner.scene.tvb_layermeta_pointer().into()); - r.add(0x1c9d8, inner.scene.tvb_layermeta_pointer().into()); - let cl_layermeta_pointer = - inner.scene.tvb_cluster_layermeta_pointer().map_or(0, |a| a.into()); - r.add(0x1c089, cl_layermeta_pointer); - r.add(0x1c9e0, cl_layermeta_pointer); - let cl_meta_4_pointer = - inner.scene.meta_4_pointer().map_or(0, |a| a.into()); - r.add(0x16c41, cl_meta_4_pointer); // tvb_cluster_meta4 - r.add(0x1ca40, cl_meta_4_pointer); // tvb_cluster_meta4 - r.add(0x1c9a8, vtx_unk_f0); // + meta1_blocks? min_free_tvb_pages? - r.add( - 0x1c920, - inner.scene.meta_1_pointer().map_or(0, |a| a.into()), - ); // ??? | meta1_blocks? - r.add(0x10151, 0); - r.add(0x1c199, 0); - r.add(0x1c1a1, 0); - r.add(0x1c1a9, 0); // 0x10151 bit 1 enables - r.add(0x1c1b1, 0); - r.add(0x1c1b9, 0); - r.add(0x10061, self.usc_exec_base); // USC_EXEC_BASE_TA - r.add(0x11801, cmdbuf.vertex_helper.binary.into()); - r.add(0x11809, cmdbuf.vertex_helper.data); - r.add(0x11f71, cmdbuf.vertex_helper.cfg.into()); - r.add(0x1c0b1, tile_info.params.rgn_size.into()); // TE_PSG - r.add(0x1c850, tile_info.params.rgn_size.into()); - r.add(0x10131, tile_info.params.unk_4.into()); - r.add(0x10121, tile_info.params.ppp_ctrl.into()); // PPP_CTRL - r.add( - 0x10129, - tile_info.params.x_max as u64 - | ((tile_info.params.y_max as u64) << 16), - ); // PPP_SCREEN - r.add(0x101b9, tile_info.params.te_screen.into()); // TE_SCREEN - r.add(0x1c069, tile_info.params.te_mtile1.into()); // TE_MTILE1 - r.add(0x1c071, tile_info.params.te_mtile2.into()); // TE_MTILE2 - r.add(0x1c081, tile_info.params.tiles_per_mtile.into()); // TE_MTILE - r.add(0x1c0a9, tile_info.params.tpc_stride.into()); // TE_TPC - r.add(0x10171, tile_info.params.unk_24.into()); - r.add(0x10169, tile_info.params.unk_28.into()); // TA_RENDER_TARGET_MAX - r.add(0x12099, vtx_unk_118); - r.add(0x1c9e8, (tile_info.params.unk_28 & 0x4fff).into()); - /* - r.add(0x10209, 0x100); // Some kind of counter?? Does this matter? - r.add(0x1c9f0, 0x100); // Some kind of counter?? Does this matter? - r.add(0x1c830, 1); // ? - r.add(0x1ca30, 0x1502960e60); // ? - r.add(0x16c39, 0x1502960e60); // ? - r.add(0x1c910, 0xa0000b011d); // ? - r.add(0x1c8e0, 0xff); // cluster mask - r.add(0x1c8e8, 0); // ? - */ - } - ), - tpc: inner.scene.tpc_pointer(), - tpc_size: U64(tile_info.tpc_size as u64), - microsequence: inner.micro_seq.gpu_pointer(), - microsequence_size: inner.micro_seq.len() as u32, - fragment_stamp_slot: ev_frag.slot, - fragment_stamp_value: ev_frag.value.next(), - unk_pointee: 0, - unk_pad: 0, - job_params2 <- try_init!(fw::vertex::raw::JobParameters2 { - unk_480: Default::default(), // fixed - unk_498: U64(0x0), // fixed - unk_4a0: 0x0, // fixed - preempt_buf1: inner.scene.preempt_buf_1_pointer(), - unk_4ac: 0x0, // fixed - unk_4b0: U64(0x0), // fixed - unk_4b8: 0x0, // fixed - unk_4bc: U64(0x0), // fixed - unk_4c4_padding: Default::default(), - unk_50c: 0x0, // fixed - unk_510: U64(0x0), // fixed - unk_518: U64(0x0), // fixed - unk_520: U64(0x0), // fixed - }), - encoder_params <- try_init!(fw::job::raw::EncoderParams { - unk_8: 0x0, // fixed - sync_grow: 0x0, // fixed - unk_10: 0x0, // fixed - encoder_id: 0, - unk_18: 0x0, // fixed - unk_mask: 0xffffffffu32, - sampler_array: U64(cmdbuf.sampler_heap), - sampler_count: cmdbuf.sampler_count as u32, - sampler_max: (cmdbuf.sampler_count as u32) + 1, - }), - unk_55c: 0, - unk_560: 0, - sync_grow: 0, - unk_568: 0, - uses_scratch: (cmdbuf.flags - & uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_VERTEX_SCRATCH as u32 - != 0) as u32, - meta <- try_init!(fw::job::raw::JobMeta { - unk_0: 0, - unk_2: 0, - no_preemption: no_preemption as u8, - stamp: ev_vtx.stamp_pointer, - fw_stamp: ev_vtx.fw_stamp_pointer, - stamp_value: ev_vtx.value.next(), - stamp_slot: ev_vtx.slot, - evctl_index: 0, // fixed - flush_stamps: flush_stamps as u32, - uuid: uuid_ta, - event_seq: ev_vtx.event_seq as u32, - }), - unk_after_meta: unk1.into(), - unk_buf_0: U64(0), - unk_buf_8: U64(0), - unk_buf_10: U64(0), - command_time: U64(0), - timestamp_pointers <- try_init!(fw::job::raw::TimestampPointers { - start_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), vtx.start)), - end_addr: Some(inner_ptr!(inner.timestamps.gpu_pointer(), vtx.end)), - }), - user_timestamp_pointers: inner.user_timestamps.pointers()?, - client_sequence: slot_client_seq, - pad_5d5: Default::default(), - unk_5d8: 0, - unk_5dc: 0, - #[ver(V >= V13_0B4)] - unk_ts: U64(0), - #[ver(V >= V13_0B4)] - unk_5dd_8: Default::default(), - }) - }, - )?; - - core::mem::drop(alloc); - - mod_dev_dbg!(self.dev, "[Submission {}] Add Vertex\n", id); - fence.add_command(); - vtx_job.add_cb(vtx, vm_bind.slot(), move |error| { - if let Some(err) = error { - fence.set_error(err.into()) - } - - fence.command_complete(); - })?; - - mod_dev_dbg!(self.dev, "[Submission {}] Increment counters\n", id); - - // TODO: handle rollbacks, move to job submit? - buffer.increment(); - - job.get_vtx()?.next_seq(); - job.get_frag()?.next_seq(); - - Ok(()) - } -} diff --git a/drivers/gpu/drm/asahi-neo/regs.rs b/drivers/gpu/drm/asahi-neo/regs.rs deleted file mode 100644 index 2bfd754864358e..00000000000000 --- a/drivers/gpu/drm/asahi-neo/regs.rs +++ /dev/null @@ -1,742 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU MMIO register abstraction -//! -//! Since the vast majority of the interactions with the GPU are brokered through the firmware, -//! there is very little need to interact directly with GPU MMIO register. This module abstracts -//! the few operations that require that, mainly reading the MMU fault status, reading GPU ID -//! information, and starting the GPU firmware coprocessor. -//! -//! On G17, the driver also enables the bridge between the GPU and the fabric before the -//! coprocessors start, and enables dynamic power gating before the firmware is initialized. - -use crate::hw; -use core::mem::MaybeUninit; -use kernel::{ - bindings, c_str, - device::Core, - devres::Devres, - error::{from_err_ptr, to_result}, - io::{ - mem::IoMem, // - Io, - }, - platform, - prelude::*, - sync::{aref::ARef, Arc}, // -}; - -/// Size of the ASC control MMIO region. -pub(crate) const ASC_CTL_SIZE: usize = 0x4000; - -/// Size of the SGX MMIO region. -pub(crate) const SGX_SIZE: usize = 0x1000000; - -const CPU_CONTROL: usize = 0x44; -const CPU_RUN: u32 = 0x1 << 4; // BIT(4) - -const FAULT_INFO: usize = 0x17030; - -/// A nonzero low nibble permits writes through the G17 queue configuration port. -const G17_HOST_IRQ_SUMMARY: usize = 0xe01000; -const G17_HOST_POWERED: u32 = 0xf; - -const ID_VERSION: usize = 0xd04000; -const ID_UNK08: usize = 0xd04008; -const ID_COUNTS_1: usize = 0xd04010; -const ID_COUNTS_2: usize = 0xd04014; -const ID_UNK18: usize = 0xd04018; -const ID_CLUSTERS: usize = 0xd0401c; - -const CORE_MASK_0: usize = 0xd01500; -const CORE_MASK_1: usize = 0xd01514; - -const CORE_MASKS_G14X: usize = 0xe01500; -const FAULT_INFO_G14X: usize = 0xd8c0; -const FAULT_ADDR_G14X: usize = 0xd8c8; - -/// G17 GPU-to-fabric bridge control words. Bit 0 enables the bridge; the other bits are status. -const G17_BRIDGE_CONTROL: [usize; 2] = [0x1000104, 0x1000108]; -const G17_BRIDGE_ENABLE: u32 = 0x1; // BIT(0) - -/// G17 dynamic power gating control. Clearing these bits enables gating of idle GPU cores. -const G17_GATING_CONTROL: usize = 0xd06030; -const G17_GATING_DISABLE: u32 = 0x6; // BIT(1) | BIT(2) - -/// These performance selectors must be sampled before firmware can gate the core registers. -const G17_PERF_CONTROL: usize = 0xe0141c; -const G17_PERF_MAP: usize = 0xe01480; -const G17_CORE_POWER: usize = 0xe01000; - -/// Enum representing the unit that caused an MMU fault. -#[allow(non_camel_case_types)] -#[allow(clippy::upper_case_acronyms)] -#[derive(Copy, Clone, Debug, Eq, PartialEq)] -pub(crate) enum FaultUnit { - /// Decompress / pixel fetch - DCMP(u8), - /// USC L1 Cache (device loads/stores) - UL1C(u8), - /// Compress / pixel store - CMP(u8), - GSL1(u8), - IAP(u8), - VCE(u8), - /// Tiling Engine - TE(u8), - RAS(u8), - /// Vertex Data Master - VDM(u8), - PPP(u8), - /// ISP Parameter Fetch - IPF(u8), - IPF_CPF(u8), - VF(u8), - VF_CPF(u8), - /// Depth/Stencil load/store - ZLS(u8), - - /// Parameter Management - dPM, - /// Compute Data Master - dCDM_KS(u8), - dIPP, - dIPP_CS, - // Vertex Data Master - dVDM_CSD, - dVDM_SSD, - dVDM_ILF, - dVDM_ILD, - dRDE(u8), - FC, - GSL2, - - /// Graphics L2 Cache Control? - GL2CC_META(u8), - GL2CC_MB, - - /// Parameter Management - gPM_SP(u8), - /// Vertex Data Master - CSD - gVDM_CSD_SP(u8), - gVDM_SSD_SP(u8), - gVDM_ILF_SP(u8), - gVDM_TFP_SP(u8), - gVDM_MMB_SP(u8), - /// Compute Data Master - gCDM_CS_KS0_SP(u8), - gCDM_CS_KS1_SP(u8), - gCDM_CS_KS2_SP(u8), - gCDM_KS0_SP(u8), - gCDM_KS1_SP(u8), - gCDM_KS2_SP(u8), - gIPP_SP(u8), - gIPP_CS_SP(u8), - gRDE0_SP(u8), - gRDE1_SP(u8), - - gCDM_CS, - gCDM_ID, - gCDM_CSR, - gCDM_CSW, - gCDM_CTXR, - gCDM_CTXW, - gIPP, - gIPP_CS, - gKSM_RCE, - - Unknown(u8), -} - -/// Reason for an MMU fault. -#[derive(Copy, Clone, Debug, Eq, PartialEq)] -pub(crate) enum FaultReason { - Unmapped, - AfFault, - WriteOnly, - ReadOnly, - NoAccess, - Unknown(u8), -} - -/// Collection of information about an MMU fault. -#[derive(Copy, Clone, Debug, Eq, PartialEq)] -pub(crate) struct FaultInfo { - pub(crate) address: u64, - pub(crate) sideband: u8, - pub(crate) vm_slot: u32, - pub(crate) unit_code: u8, - pub(crate) unit: FaultUnit, - pub(crate) level: u8, - pub(crate) unk_5: u8, - pub(crate) read: bool, - pub(crate) reason: FaultReason, -} - -/// Identification of a G17 GPU. -pub(crate) struct G17Id { - /// GPU family field of the ID register. - pub(crate) family: u8, - /// GPU variant field of the ID register. - pub(crate) variant: u8, - /// Silicon revision. - pub(crate) gpu_rev: hw::GpuRevision, - /// Number of dies. - pub(crate) num_dies: u32, - /// Number of clusters, over all dies. - pub(crate) num_clusters: u32, - /// Number of cores per cluster. - pub(crate) num_cores: u32, - /// Enabled cores of each cluster. - pub(crate) core_masks: KVec, - /// Performance selector control, sampled before core power gating starts. - pub(crate) perf_control: u32, - /// Performance selector table indices, low and high nibbles. - pub(crate) perf_map: [u32; 2], -} - -/// Provider-owned lifecycle of one G17 firmware coprocessor. -/// -/// The mailbox provider owns the wrapper mappings and runtime-PM references. -/// The consumer device link keeps it alive until this GPU has stopped using it. -pub(crate) struct CpuControl { - dev: ARef, - mbox: *mut bindings::apple_mbox, -} - -// SAFETY: The live provider serializes CPU transitions with its lifecycle mutex; -// the device link orders provider removal after this consumer's teardown. -unsafe impl Send for CpuControl {} -// SAFETY: All accesses use the same provider-serialized lifecycle API. -unsafe impl Sync for CpuControl {} - -impl CpuControl { - /// Resolve the mailbox phandle and validate its firmware role before use. - pub(crate) fn new(pdev: &platform::Device, index: usize) -> Result { - let expected = match index { - 0 => bindings::apple_mbox_ascwrap_v6_role_APPLE_MBOX_ASCWRAP_V6_ROLE_GFX, - 1 => bindings::apple_mbox_ascwrap_v6_role_APPLE_MBOX_ASCWRAP_V6_ROLE_GFX1, - _ => return Err(EINVAL), - }; - // SAFETY: pdev is live; the provider establishes the consumer device link. - let mbox = unsafe { - from_err_ptr(bindings::apple_mbox_get(pdev.as_ref().as_raw(), index.try_into()?))? - }; - let mut lifecycle = MaybeUninit::uninit(); - // SAFETY: mbox is live, and lifecycle is writable storage for the C result. - to_result(unsafe { - bindings::apple_mbox_ascwrap_v6_get_lifecycle(mbox, lifecycle.as_mut_ptr()) - })?; - // SAFETY: A successful provider query initialized every field. - if unsafe { lifecycle.assume_init() }.role != expected { - return Err(EINVAL); - } - // SAFETY: This only checks the provider's matching start/stop capability. - to_result(unsafe { bindings::apple_mbox_ascwrap_v6_require_safe_cpu_lifecycle(mbox) })?; - Ok(Self { dev: pdev.into(), mbox }) - } - - /// Start through the owner of the CPU registers and runtime-PM reference. - pub(crate) fn start(&self) -> Result { - // SAFETY: The constructor established the live, role-checked provider. - to_result(unsafe { bindings::apple_mbox_ascwrap_v6_start_cpu(self.mbox) }) - } - - /// Stop before releasing the transport or any firmware-visible memory. - pub(crate) fn stop(&self) { - // SAFETY: The provider remains live during consumer teardown. Removal - // stops the CPU before setting the fence that can return ENODEV here. - let result = to_result(unsafe { bindings::apple_mbox_ascwrap_v6_stop_cpu(self.mbox) }); - if let Err(error) = result { - if error != ENODEV { - dev_err!(self.dev.as_ref(), "Could not stop GPU coprocessor: {:?}\n", error); - } - } - } -} - -/// Device resources for this GPU instance. -#[derive(Clone)] -pub(crate) struct Resources { - dev: ARef, - sgx: Arc>>, -} - -impl Resources { - /// Map the required resources given our platform device. - pub(crate) fn new(pdev: &platform::Device) -> Result { - let sgx_req = pdev.io_request_by_name(c_str!("sgx")).ok_or(EINVAL)?; - let sgx_iomem = Arc::pin_init(sgx_req.iomap_sized::(), GFP_KERNEL)?; - - Ok(Resources { - // SAFETY: This device does DMA via the UAT IOMMU. - dev: pdev.into(), - sgx: sgx_iomem, - }) - } - - /// Installs both render queues while the caller holds the device mutex. - /// No other port publisher may interleave between this power witness and - /// the final validity strobe. False means no write was attempted. - pub(crate) fn install_render_pair( - &self, - registration: &crate::g17::fw::kick::RenderRegistration, - ) -> Result { - let sgx = self.sgx.try_access().ok_or(ENODEV)?; - if sgx.relaxed().read32(G17_HOST_IRQ_SUMMARY) & G17_HOST_POWERED == 0 { - return Ok(false); - } - for &(offset, value) in registration.writes() { - sgx.try_write64(value, offset)?; - } - Ok(true) - } - - fn sgx_read32(&self) -> u32 { - if let Some(sgx) = self.sgx.try_access() { - sgx.relaxed().read32(OFF) - } else { - 0 - } - } - - /* Not yet used - fn sgx_write32(&self, val: u32) { - if let Some(sgx) = self.sgx.try_access() { - sgx.write32_relaxed(val, OFF) - } - } - */ - - fn sgx_read64(&self) -> u64 { - if let Some(sgx) = self.sgx.try_access() { - sgx.relaxed().read64(OFF) - } else { - 0 - } - } - - /* Not yet used - fn sgx_write64(&self, val: u64) { - if let Some(sgx) = self.sgx.try_access() { - sgx.write64_relaxed(val, OFF) - } - } - */ - - /// Initialize the MMIO registers for the GPU. - pub(crate) fn init_mmio(&self) -> Result { - // Nothing to do for now... - - Ok(()) - } - - /// Start the ASC coprocessor CPU. - pub(crate) fn start_cpu(pdev: &platform::Device) -> Result { - let asc_req = pdev.io_request_by_name(c_str!("asc")).ok_or(EINVAL)?; - let asc_iomem = KBox::pin_init(asc_req.iomap_sized::(), GFP_KERNEL)?; - let res = asc_iomem.access(pdev.as_ref())?.relaxed(); - - let val = res.read32(CPU_CONTROL); - res.write32(val | CPU_RUN, CPU_CONTROL); - Ok(()) - } - - /// Get the GPU identification info from registers. - /// - /// See [`hw::GpuIdConfig`] for the result. - pub(crate) fn get_gpu_id(&self) -> Result { - let id_version = self.sgx_read32::(); - let id_unk08 = self.sgx_read32::(); - let id_counts_1 = self.sgx_read32::(); - let id_counts_2 = self.sgx_read32::(); - let id_unk18 = self.sgx_read32::(); - let id_clusters = self.sgx_read32::(); - - dev_info!( - self.dev.as_ref(), - "GPU ID registers: {:#x} {:#x} {:#x} {:#x} {:#x} {:#x}\n", - id_version, - id_unk08, - id_counts_1, - id_counts_2, - id_unk18, - id_clusters - ); - - let gpu_gen = (id_version >> 24) & 0xff; - - let mut core_mask_regs = KVec::new(); - - let num_clusters = match gpu_gen { - 4 | 5 => { - // G13 | G14G - core_mask_regs.push(self.sgx_read32::(), GFP_KERNEL)?; - core_mask_regs.push(self.sgx_read32::(), GFP_KERNEL)?; - (id_clusters >> 12) & 0xff - } - 6 => { - // G14X - core_mask_regs.push(self.sgx_read32::(), GFP_KERNEL)?; - core_mask_regs.push(self.sgx_read32::<{ CORE_MASKS_G14X + 4 }>(), GFP_KERNEL)?; - core_mask_regs.push(self.sgx_read32::<{ CORE_MASKS_G14X + 8 }>(), GFP_KERNEL)?; - // Clusters per die * num dies - ((id_counts_1 >> 8) & 0xff) * ((id_counts_1 >> 16) & 0xf) - } - a => { - dev_err!(self.dev.as_ref(), "Unknown GPU generation {}\n", a); - return Err(ENODEV); - } - }; - - let mut core_masks_packed = KVec::new(); - core_masks_packed.extend_from_slice(&core_mask_regs, GFP_KERNEL)?; - - dev_info!(self.dev.as_ref(), "Core masks: {:#x?}\n", core_masks_packed); - - let num_cores = id_counts_1 & 0xff; - - if num_cores > 32 { - dev_err!( - self.dev.as_ref(), - "Too many cores per cluster ({} > 32)\n", - num_cores - ); - return Err(ENODEV); - } - - if num_cores * num_clusters > (core_mask_regs.len() * 32) as u32 { - dev_err!( - self.dev.as_ref(), - "Too many total cores ({} x {} > {})\n", - num_clusters, - num_cores, - core_mask_regs.len() * 32 - ); - return Err(ENODEV); - } - - let (core_masks, total_active_cores) = - self.split_core_masks(&mut core_mask_regs, num_clusters, num_cores)?; - let (gpu_rev, gpu_rev_id) = self.gpu_revision(id_version)?; - - Ok(hw::GpuIdConfig { - gpu_gen: match (id_version >> 24) & 0xff { - 4 => hw::GpuGen::G13, - 5 => hw::GpuGen::G14, - 6 => hw::GpuGen::G14, // G14X has a separate ID - a => { - dev_err!(self.dev.as_ref(), "Unknown GPU generation {}\n", a); - return Err(ENODEV); - } - }, - gpu_variant: match (id_version >> 16) & 0xff { - 1 => hw::GpuVariant::P, // Guess - 2 => hw::GpuVariant::G, - 3 => hw::GpuVariant::S, - 4 => { - if num_clusters > 4 { - hw::GpuVariant::D - } else { - hw::GpuVariant::C - } - } - a => { - dev_err!(self.dev.as_ref(), "Unknown GPU variant {}\n", a); - return Err(ENODEV); - } - }, - gpu_rev, - gpu_rev_id, - num_clusters, - num_cores, - num_frags: num_cores, // Used to be id_counts_1[15:8] but does not work for G14X - num_gps: (id_counts_2 >> 16) & 0xff, - total_active_cores, - core_masks, - core_masks_packed, - }) - } - - /// Get the identification of a G17 GPU from its registers. - pub(crate) fn get_g17_id(&self) -> Result { - let id_version = self.sgx_read32::(); - let id_unk08 = self.sgx_read32::(); - let id_counts_1 = self.sgx_read32::(); - let id_counts_2 = self.sgx_read32::(); - let id_unk18 = self.sgx_read32::(); - let id_clusters = self.sgx_read32::(); - - dev_info!( - self.dev.as_ref(), - "GPU ID registers: {:#x} {:#x} {:#x} {:#x} {:#x} {:#x}\n", - id_version, - id_unk08, - id_counts_1, - id_counts_2, - id_unk18, - id_clusters - ); - - // Later reads can fault while the firmware has gated the cores. Submission uses only - // these saved values and never re-reads the registers. - let perf_control = self.sgx_read32::() & 1; - let perf_map = self.sgx_read32::(); - let mut core_mask_regs = [ - self.sgx_read32::(), - self.sgx_read32::<{ CORE_MASKS_G14X + 4 }>(), - self.sgx_read32::<{ CORE_MASKS_G14X + 8 }>(), - ]; - dev_info!(self.dev.as_ref(), "Core masks: {:#x?}\n", core_mask_regs); - - let num_dies = (id_counts_1 >> 16) & 0xf; - let num_clusters = ((id_counts_1 >> 8) & 0xff) * num_dies; - let num_cores = id_counts_1 & 0xff; - if num_cores == 0 || num_cores > 32 || num_cores * num_clusters > 3 * 32 { - dev_err!( - self.dev.as_ref(), - "Invalid core counts ({} x {})\n", - num_clusters, - num_cores - ); - return Err(ENODEV); - } - let (core_masks, _) = - self.split_core_masks(&mut core_mask_regs, num_clusters, num_cores)?; - let (gpu_rev, _) = self.gpu_revision(id_version)?; - - Ok(G17Id { - family: ((id_version >> 24) & 0xff) as u8, - variant: ((id_version >> 16) & 0xff) as u8, - gpu_rev, - num_dies, - num_clusters, - num_cores, - core_masks, - perf_control, - perf_map: [perf_map & 0xf, (perf_map >> 16) & 0xf], - }) - } - - /// Enable the G17 GPU-to-fabric bridge and check that it took effect. - /// - /// The GPU power domain must be on. The coprocessors cannot reach memory until this is done. - pub(crate) fn enable_g17_bridge(&self) -> Result { - let sgx = self.sgx.try_access().ok_or(ENODEV)?; - for offset in G17_BRIDGE_CONTROL { - let val = sgx.try_read32(offset)?; - sgx.try_write32(val | G17_BRIDGE_ENABLE, offset)?; - if sgx.try_read32(offset)? & G17_BRIDGE_ENABLE == 0 { - dev_err!( - self.dev.as_ref(), - "GPU bridge {:#x} did not enable\n", - offset - ); - return Err(EIO); - } - } - Ok(()) - } - - /// Disable the G17 GPU-to-fabric bridge. Both coprocessors must be stopped. - pub(crate) fn disable_g17_bridge(&self) { - if let Some(sgx) = self.sgx.try_access() { - for offset in G17_BRIDGE_CONTROL { - if let Ok(val) = sgx.try_read32(offset) { - // Cannot fail: the offset was read just before. - let _ = sgx.try_write32(val & !G17_BRIDGE_ENABLE, offset); - } - } - } - } - - /// Power witness for the queue-configuration port, accessible while the cores are gated. - pub(crate) fn g17_configuration_powered(&self) -> bool { - self.sgx_read32::() & 0xf != 0 - } - - /// Enable dynamic power gating of idle G17 GPU cores. The firmware expects this before it is - /// initialized. - pub(crate) fn enable_g17_gating(&self) -> Result { - let sgx = self.sgx.try_access().ok_or(ENODEV)?; - let val = sgx.try_read32(G17_GATING_CONTROL)?; - sgx.try_write32(val & !G17_GATING_DISABLE, G17_GATING_CONTROL) - } - - /// Split the packed per-cluster core masks into one mask per cluster. - /// - /// Returns the masks and the total number of enabled cores. - fn split_core_masks( - &self, - core_mask_regs: &mut [u32], - num_clusters: u32, - num_cores: u32, - ) -> Result<(KVec, u32)> { - let mut core_masks = KVec::new(); - let mut total_active_cores: u32 = 0; - - let max_core_mask = ((1u64 << num_cores) - 1) as u32; - for _ in 0..num_clusters { - let mask = core_mask_regs[0] & max_core_mask; - core_masks.push(mask, GFP_KERNEL)?; - for i in 0..core_mask_regs.len() { - core_mask_regs[i] = core_mask_regs[i].checked_shr(num_cores).unwrap_or(0); - if i < (core_mask_regs.len() - 1) { - core_mask_regs[i] |= core_mask_regs[i + 1] - .checked_shl(32 - num_cores) - .unwrap_or(0); - } - } - total_active_cores += mask.count_ones(); - } - - if core_mask_regs.iter().any(|a| *a != 0) { - dev_err!( - self.dev.as_ref(), - "Leftover core mask: {:#x?}\n", - core_mask_regs - ); - return Err(EIO); - } - - Ok((core_masks, total_active_cores)) - } - - /// Decode the silicon revision from the ID version register. - fn gpu_revision(&self, id_version: u32) -> Result<(hw::GpuRevision, hw::GpuRevisionID)> { - Ok(match (id_version >> 8) & 0xff { - 0x00 => (hw::GpuRevision::A0, hw::GpuRevisionID::A0), - 0x01 => (hw::GpuRevision::A1, hw::GpuRevisionID::A1), - 0x10 => (hw::GpuRevision::B0, hw::GpuRevisionID::B0), - 0x11 => (hw::GpuRevision::B1, hw::GpuRevisionID::B1), - 0x20 => (hw::GpuRevision::C0, hw::GpuRevisionID::C0), - 0x21 => (hw::GpuRevision::C1, hw::GpuRevisionID::C1), - a => { - dev_err!(self.dev.as_ref(), "Unknown GPU revision {}\n", a); - return Err(ENODEV); - } - }) - } - - /// Get the fault information from the MMU status register, if one occurred. - pub(crate) fn get_fault_info(&self, cfg: &'static hw::HwConfig) -> Option { - let g14x = cfg.gpu_core as u32 >= hw::GpuCore::G14S as u32; - - let fault_info = if g14x { - self.sgx_read64::() - } else { - self.sgx_read64::() - }; - - if fault_info & 1 == 0 { - return None; - } - - let fault_addr = if g14x { - self.sgx_read64::() - } else { - fault_info >> 30 - }; - - let unit_code = ((fault_info >> 9) & 0xff) as u8; - let unit = match unit_code { - 0x00..=0x9f => match unit_code & 0xf { - 0x0 => FaultUnit::DCMP(unit_code >> 4), - 0x1 => FaultUnit::UL1C(unit_code >> 4), - 0x2 => FaultUnit::CMP(unit_code >> 4), - 0x3 => FaultUnit::GSL1(unit_code >> 4), - 0x4 => FaultUnit::IAP(unit_code >> 4), - 0x5 => FaultUnit::VCE(unit_code >> 4), - 0x6 => FaultUnit::TE(unit_code >> 4), - 0x7 => FaultUnit::RAS(unit_code >> 4), - 0x8 => FaultUnit::VDM(unit_code >> 4), - 0x9 => FaultUnit::PPP(unit_code >> 4), - 0xa => FaultUnit::IPF(unit_code >> 4), - 0xb => FaultUnit::IPF_CPF(unit_code >> 4), - 0xc => FaultUnit::VF(unit_code >> 4), - 0xd => FaultUnit::VF_CPF(unit_code >> 4), - 0xe => FaultUnit::ZLS(unit_code >> 4), - _ => FaultUnit::Unknown(unit_code), - }, - 0xa1 => FaultUnit::dPM, - 0xa2 => FaultUnit::dCDM_KS(0), - 0xa3 => FaultUnit::dCDM_KS(1), - 0xa4 => FaultUnit::dCDM_KS(2), - 0xa5 => FaultUnit::dIPP, - 0xa6 => FaultUnit::dIPP_CS, - 0xa7 => FaultUnit::dVDM_CSD, - 0xa8 => FaultUnit::dVDM_SSD, - 0xa9 => FaultUnit::dVDM_ILF, - 0xaa => FaultUnit::dVDM_ILD, - 0xab => FaultUnit::dRDE(0), - 0xac => FaultUnit::dRDE(1), - 0xad => FaultUnit::FC, - 0xae => FaultUnit::GSL2, - 0xb0..=0xb7 => FaultUnit::GL2CC_META(unit_code & 0xf), - 0xb8 => FaultUnit::GL2CC_MB, - 0xd0..=0xdf if g14x => match unit_code & 0xf { - 0x0 => FaultUnit::gCDM_CS, - 0x1 => FaultUnit::gCDM_ID, - 0x2 => FaultUnit::gCDM_CSR, - 0x3 => FaultUnit::gCDM_CSW, - 0x4 => FaultUnit::gCDM_CTXR, - 0x5 => FaultUnit::gCDM_CTXW, - 0x6 => FaultUnit::gIPP, - 0x7 => FaultUnit::gIPP_CS, - 0x8 => FaultUnit::gKSM_RCE, - _ => FaultUnit::Unknown(unit_code), - }, - 0xe0..=0xff if g14x => match unit_code & 0xf { - 0x0 => FaultUnit::gPM_SP((unit_code >> 4) & 1), - 0x1 => FaultUnit::gVDM_CSD_SP((unit_code >> 4) & 1), - 0x2 => FaultUnit::gVDM_SSD_SP((unit_code >> 4) & 1), - 0x3 => FaultUnit::gVDM_ILF_SP((unit_code >> 4) & 1), - 0x4 => FaultUnit::gVDM_TFP_SP((unit_code >> 4) & 1), - 0x5 => FaultUnit::gVDM_MMB_SP((unit_code >> 4) & 1), - 0x6 => FaultUnit::gRDE0_SP((unit_code >> 4) & 1), - _ => FaultUnit::Unknown(unit_code), - }, - 0xe0..=0xff if !g14x => match unit_code & 0xf { - 0x0 => FaultUnit::gPM_SP((unit_code >> 4) & 1), - 0x1 => FaultUnit::gVDM_CSD_SP((unit_code >> 4) & 1), - 0x2 => FaultUnit::gVDM_SSD_SP((unit_code >> 4) & 1), - 0x3 => FaultUnit::gVDM_ILF_SP((unit_code >> 4) & 1), - 0x4 => FaultUnit::gVDM_TFP_SP((unit_code >> 4) & 1), - 0x5 => FaultUnit::gVDM_MMB_SP((unit_code >> 4) & 1), - 0x6 => FaultUnit::gCDM_CS_KS0_SP((unit_code >> 4) & 1), - 0x7 => FaultUnit::gCDM_CS_KS1_SP((unit_code >> 4) & 1), - 0x8 => FaultUnit::gCDM_CS_KS2_SP((unit_code >> 4) & 1), - 0x9 => FaultUnit::gCDM_KS0_SP((unit_code >> 4) & 1), - 0xa => FaultUnit::gCDM_KS1_SP((unit_code >> 4) & 1), - 0xb => FaultUnit::gCDM_KS2_SP((unit_code >> 4) & 1), - 0xc => FaultUnit::gIPP_SP((unit_code >> 4) & 1), - 0xd => FaultUnit::gIPP_CS_SP((unit_code >> 4) & 1), - 0xe => FaultUnit::gRDE0_SP((unit_code >> 4) & 1), - 0xf => FaultUnit::gRDE1_SP((unit_code >> 4) & 1), - _ => FaultUnit::Unknown(unit_code), - }, - _ => FaultUnit::Unknown(unit_code), - }; - - let reason = match (fault_info >> 1) & 0x7 { - 0 => FaultReason::Unmapped, - 1 => FaultReason::AfFault, - 2 => FaultReason::WriteOnly, - 3 => FaultReason::ReadOnly, - 4 => FaultReason::NoAccess, - a => FaultReason::Unknown(a as u8), - }; - - Some(FaultInfo { - address: fault_addr << 6, - sideband: ((fault_info >> 23) & 0x7f) as u8, - vm_slot: ((fault_info >> 17) & 0x3f) as u32, - unit_code, - unit, - level: ((fault_info >> 7) & 3) as u8, - unk_5: ((fault_info >> 5) & 3) as u8, - read: (fault_info & (1 << 4)) != 0, - reason, - }) - } -} diff --git a/drivers/gpu/drm/asahi-neo/slotalloc.rs b/drivers/gpu/drm/asahi-neo/slotalloc.rs deleted file mode 100644 index fde7470fe57791..00000000000000 --- a/drivers/gpu/drm/asahi-neo/slotalloc.rs +++ /dev/null @@ -1,323 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Generic slot allocator -//! -//! This is a simple allocator to manage fixed-size pools of GPU resources that are transiently -//! required during command execution. Each item resides in a "slot" at a given index. Users borrow -//! and return free items from the available pool. -//! -//! Allocations are "sticky", and return a token that callers can use to request the same slot -//! again later. This allows slots to be lazily invalidated, so that multiple uses by the same user -//! avoid any actual cleanup work. -//! -//! The allocation policy is currently a simple LRU mechanism, doing a full linear scan over the -//! slots when no token was previously provided. This is probably good enough, since in the absence -//! of serious system contention most allocation requests will be immediately fulfilled from the -//! previous slot without doing an LRU scan. - -use core::num::NonZeroUsize; -use core::ops::{ - Deref, - DerefMut, // -}; -use kernel::{ - error::{ - code::*, - Result, // - }, - prelude::*, - str::CStr, - sync::{ - Arc, - CondVar, - LockClassKey, - Mutex, // - }, -}; - -/// Trait representing a single item within a slot. -pub(crate) trait SlotItem { - /// Arbitrary user data associated with the SlotAllocator. - type Data; - - /// Called eagerly when this item is released back into the available pool. - fn release(&mut self, _data: &mut Self::Data, _slot: u32) {} -} - -/// Trivial implementation for users which do not require any slot data nor any allocator data. -impl SlotItem for () { - type Data = (); -} - -/// Represents a current or previous allocation of an item from a slot. Users keep `SlotToken`s -/// around across allocations to request that, if possible, the same slot be reused. -#[derive(Copy, Clone, Debug)] -pub(crate) struct SlotToken { - time: u64, - slot: u32, -} - -impl SlotToken { - /// Returns the slot index that this token represents a past assignment to. - pub(crate) fn last_slot(&self) -> u32 { - self.slot - } -} - -/// A guard representing active ownership of a slot. -pub(crate) struct Guard { - item: Option, - changed: bool, - token: SlotToken, - alloc: Arc>, -} - -impl Guard { - /// Returns the active slot owned by this `Guard`. - pub(crate) fn slot(&self) -> u32 { - self.token.slot - } - - /// Returns `true` if the slot changed since the last allocation (or no `SlotToken` was - /// provided), or `false` if the previously allocated slot was successfully re-acquired with - /// no other users in the interim. - pub(crate) fn changed(&self) -> bool { - self.changed - } - - /// Returns a `SlotToken` that can be used to re-request the same slot at a later time, after - /// this `Guard` is dropped. - pub(crate) fn token(&self) -> SlotToken { - self.token - } -} - -impl Deref for Guard { - type Target = T; - - fn deref(&self) -> &Self::Target { - self.item.as_ref().expect("SlotItem Guard lost our item!") - } -} - -impl DerefMut for Guard { - fn deref_mut(&mut self) -> &mut Self::Target { - self.item.as_mut().expect("SlotItem Guard lost our item!") - } -} - -/// A slot item that is currently free. -struct Entry { - item: T, - get_time: u64, - drop_time: u64, -} - -/// Inner data for the `SlotAllocator`, protected by a `Mutex`. -struct SlotAllocatorInner { - data: T::Data, - slots: KVec>>, - get_count: u64, - drop_count: u64, - slot_limit: usize, -} - -/// A single slot allocator instance. -#[pin_data] -struct SlotAllocatorOuter { - #[pin] - inner: Mutex>, - #[pin] - cond: CondVar, -} - -/// A shared reference to a slot allocator instance. -pub(crate) struct SlotAllocator(Arc>); - -impl SlotAllocator { - /// Creates a new `SlotAllocator`, with a fixed number of slots and arbitrary associated data. - /// - /// The caller provides a constructor callback which takes a reference to the `T::Data` and - /// creates a single slot. This is called during construction to create all the initial - /// items, which then live the lifetime of the `SlotAllocator`. - pub(crate) fn new( - num_slots: u32, - mut data: T::Data, - mut constructor: impl FnMut(&mut T::Data, u32) -> Option, - name: &'static CStr, - lock_key1: Pin<&'static LockClassKey>, - lock_key2: Pin<&'static LockClassKey>, - ) -> Result> { - let mut slots = KVec::with_capacity(num_slots as usize, GFP_KERNEL)?; - - for i in 0..num_slots { - slots - .push( - constructor(&mut data, i).map(|item| Entry { - item, - get_time: 0, - drop_time: 0, - }), - GFP_KERNEL, - ) - .expect("try_push() failed after reservation"); - } - - let inner = SlotAllocatorInner { - data, - slots, - get_count: 0, - drop_count: 0, - slot_limit: usize::MAX, - }; - - let alloc = Arc::pin_init( - pin_init!(SlotAllocatorOuter { - // SAFETY: `mutex_init!` is called below. - inner <- Mutex::new(inner, name, lock_key1), - // SAFETY: `condvar_init!` is called below. - cond <- CondVar::new(name, lock_key2), - }), - GFP_KERNEL, - )?; - - Ok(SlotAllocator(alloc)) - } - - /// Calls a callback on the inner data associated with this allocator, taking the lock. - pub(crate) fn with_inner(&self, cb: impl FnOnce(&mut T::Data) -> RetVal) -> RetVal { - let mut inner = self.0.inner.lock(); - cb(&mut inner.data) - } - - /// Set the slot limit for this allocator. New bindings will not use slots above - /// this threshold. - pub(crate) fn set_limit(&self, limit: Option) { - let mut inner = self.0.inner.lock(); - inner.slot_limit = limit.unwrap_or(NonZeroUsize::MAX).get(); - } - - /// Gets a fresh slot, optionally reusing a previous allocation if a `SlotToken` is provided. - /// - /// Blocks if no slots are free. - pub(crate) fn get(&self, token: Option) -> Result> { - self.get_inner(token, |_a, _b| Ok(())) - } - - /// Gets a fresh slot, optionally reusing a previous allocation if a `SlotToken` is provided. - /// - /// Blocks if no slots are free. - /// - /// This version allows the caller to pass in a callback that gets a mutable reference to the - /// user data for the allocator and the freshly acquired slot, which is called before the - /// allocator lock is released. This can be used to perform bookkeeping associated with - /// specific slots (such as tracking their current owner). - pub(crate) fn get_inner( - &self, - token: Option, - cb: impl FnOnce(&mut T::Data, &mut Guard) -> Result<()>, - ) -> Result> { - let mut inner = self.0.inner.lock(); - - if let Some(token) = token { - if (token.slot as usize) < inner.slot_limit { - let slot = &mut inner.slots[token.slot as usize]; - if slot.is_some() { - let count = slot.as_ref().unwrap().get_time; - if count == token.time { - let mut guard = Guard { - item: Some(slot.take().unwrap().item), - token, - changed: false, - alloc: self.0.clone(), - }; - cb(&mut inner.data, &mut guard)?; - return Ok(guard); - } - } - } - } - - let mut first = true; - let slot = loop { - let mut oldest_time = u64::MAX; - let mut oldest_slot = 0u32; - - for (i, slot) in inner.slots.iter().enumerate() { - if i >= inner.slot_limit { - break; - } - if let Some(slot) = slot.as_ref() { - if slot.drop_time < oldest_time { - oldest_slot = i as u32; - oldest_time = slot.drop_time; - } - } - } - - if oldest_time == u64::MAX { - if first && inner.slot_limit == usize::MAX { - pr_warn!( - "{}: out of slots, blocking\n", - core::any::type_name::() - ); - } - first = false; - if self.0.cond.wait_interruptible(&mut inner) { - return Err(ERESTARTSYS); - } - } else { - break oldest_slot; - } - }; - - inner.get_count += 1; - - let item = inner.slots[slot as usize] - .take() - .expect("Someone stole our slot?") - .item; - - let mut guard = Guard { - item: Some(item), - changed: true, - token: SlotToken { - time: inner.get_count, - slot, - }, - alloc: self.0.clone(), - }; - - cb(&mut inner.data, &mut guard)?; - Ok(guard) - } -} - -impl Clone for SlotAllocator { - fn clone(&self) -> Self { - SlotAllocator(self.0.clone()) - } -} - -impl Drop for Guard { - fn drop(&mut self) { - let mut inner = self.alloc.inner.lock(); - if inner.slots[self.token.slot as usize].is_some() { - pr_crit!( - "{}: tried to return an item into a full slot ({})\n", - core::any::type_name::(), - self.token.slot - ); - } else { - inner.drop_count += 1; - let mut item = self.item.take().expect("Guard lost its item"); - item.release(&mut inner.data, self.token.slot); - inner.slots[self.token.slot as usize] = Some(Entry { - item, - get_time: self.token.time, - drop_time: inner.drop_count, - }); - self.alloc.cond.notify_one(); - } - } -} diff --git a/drivers/gpu/drm/asahi-neo/stats.rs b/drivers/gpu/drm/asahi-neo/stats.rs deleted file mode 100644 index 772d969d9baff9..00000000000000 --- a/drivers/gpu/drm/asahi-neo/stats.rs +++ /dev/null @@ -1,78 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Snapshot of AGX firmware statistics. -//! -//! [`StatsSnapshot`] accumulates the values decoded by `StatsChannel::poll` -//! (see `fw::channels.rs::StatsMsg`) and exposes them to the sysfs file -//! `agx_stats` (registered from `sysfs.c`). -//! -//! The snapshot belongs to one GPU manager. The firmware-stat -//! decoder that feeds it lives in `StatsChannel::poll` (`channel.rs`), inside -//! the versioned context where `StatsMsg` is nameable; `note_job()` is called -//! from the `recv_message` rtkit callback (so the firmware mailbox IRQ -//! context); the sysfs `show` callback runs in arbitrary process context. All -//! fields use `AtomicU*` so neither side takes a lock. -//! -//! Field semantics (firmware units, late-validated): -//! - `util1..util4`: raw `Utilization` u32s from the firmware. Meaning -//! (firmware tick or %?) is validated against a controlled MLX load; we -//! publish them unscaled per the producer contract. -//! - `pstate`: current performance state index from `PowerState`. -//! - `avg_power_mw`: average power in milliwatts from `AvgPower` (firmware -//! field is u32 — units not validated for every chip rev; treat as raw). -//! - `temperature`: raw `Temperature` value, scale, tmin, tmax. -//! - `busy_ns`: cumulative utilization-weighted busy time in nanoseconds, -//! derived from successive `Utilization` windows (busiest subqueue -//! percentage x window duration; the firmware timestamps are base-clock -//! ticks, 24 MHz, converted with `HwConfig::base_clock_hz`). Monotonic and -//! bounded by elapsed time. Saturates at u64::MAX. -//! - `jobs`: completed submissions since boot, counted at submission -//! completion (`JobFence::command_complete`). -//! -//! This module is read-only with respect to scheduling or power behaviour: -//! the firmware already sends the messages, the channel already polls them, -//! we only retain the latest values. - -use core::sync::atomic::{AtomicU32, AtomicU64, Ordering}; - -/// Per-device snapshot of AGX firmware stats. -/// -/// The manager and bound driver retain Arcs. The driver's Arc outlives the -/// per-device C sysfs registration; repr(C) establishes the shared layout. -#[derive(Default)] -#[repr(C)] -pub(crate) struct StatsSnapshot { - pub(crate) util1: AtomicU32, - pub(crate) util2: AtomicU32, - pub(crate) util3: AtomicU32, - pub(crate) util4: AtomicU32, - pub(crate) pstate: AtomicU32, - pub(crate) avg_power_mw: AtomicU32, - pub(crate) temperature_raw: AtomicU32, - pub(crate) temperature_scale: AtomicU32, - pub(crate) temperature_tmin: AtomicU32, - pub(crate) temperature_tmax: AtomicU32, - /// Cumulative utilization-weighted busy time. - pub(crate) busy_ns: AtomicU64, - /// Completed submissions since boot, counted at fence signal in - /// `JobFence::command_complete` when the last command of a submission - /// completes. - pub(crate) jobs: AtomicU64, -} - -// Keep the Rust half of the C sysfs layout contract checked as well. -const _: () = { - assert!(core::mem::offset_of!(StatsSnapshot, pstate) == 16); - assert!(core::mem::offset_of!(StatsSnapshot, busy_ns) == 40); - assert!(core::mem::offset_of!(StatsSnapshot, jobs) == 48); - assert!(core::mem::size_of::() == 56); -}; - -impl StatsSnapshot { - /// Bump the completed-submission counter (called from the queue - /// completion path in `JobFence::command_complete`, NOT from the stats - /// channel). - pub(crate) fn note_job(&self) { - self.jobs.fetch_add(1, Ordering::Relaxed); - } -} diff --git a/drivers/gpu/drm/asahi-neo/sysfs.c b/drivers/gpu/drm/asahi-neo/sysfs.c deleted file mode 100644 index 505cd8e1ac4a4b..00000000000000 --- a/drivers/gpu/drm/asahi-neo/sysfs.c +++ /dev/null @@ -1,131 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -// -// Per-device sysfs shim. Rust owns the snapshot through an Arc in the bound -// driver. The attribute wrapper is removed and its readers drained before -// that Arc is dropped. Atomic fields may reflect different sample instants. - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "sysfs.h" - -/* Mirror of repr(C) Rust atomics; READ_ONCE accesses the current values. */ -struct asahi_stats_snapshot { - u32 util1; - u32 util2; - u32 util3; - u32 util4; - u32 pstate; - u32 avg_power_mw; - u32 temperature_raw; - u32 temperature_scale; - u32 temperature_tmin; - u32 temperature_tmax; - u64 busy_ns; - u64 jobs; -}; - -struct asahi_stats_attribute { - struct device_attribute attr; - const struct asahi_stats_snapshot *snapshot; - bool export_enabled; -}; - -static ssize_t agx_stats_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct asahi_stats_attribute *stats = - container_of(attr, struct asahi_stats_attribute, attr); - const struct asahi_stats_snapshot *snap = stats->snapshot; - ssize_t n = 0; - - if (!stats->export_enabled) - return sysfs_emit(buf, "unsupported\n"); - n += scnprintf(buf + n, PAGE_SIZE - n, "busy_ns %llu\n", - (unsigned long long)READ_ONCE(snap->busy_ns)); - n += scnprintf(buf + n, PAGE_SIZE - n, "jobs %llu\n", - (unsigned long long)READ_ONCE(snap->jobs)); - n += scnprintf(buf + n, PAGE_SIZE - n, "pstate %u\n", - (u32)READ_ONCE(snap->pstate)); - n += scnprintf(buf + n, PAGE_SIZE - n, "power_mw %u\n", - (u32)READ_ONCE(snap->avg_power_mw)); - n += scnprintf(buf + n, PAGE_SIZE - n, "util1 %u\n", - (u32)READ_ONCE(snap->util1)); - n += scnprintf(buf + n, PAGE_SIZE - n, "util2 %u\n", - (u32)READ_ONCE(snap->util2)); - n += scnprintf(buf + n, PAGE_SIZE - n, "util3 %u\n", - (u32)READ_ONCE(snap->util3)); - n += scnprintf(buf + n, PAGE_SIZE - n, "util4 %u\n", - (u32)READ_ONCE(snap->util4)); - n += scnprintf(buf + n, PAGE_SIZE - n, "temperature_raw %u\n", - (u32)READ_ONCE(snap->temperature_raw)); - n += scnprintf(buf + n, PAGE_SIZE - n, "temperature_scale %u\n", - (u32)READ_ONCE(snap->temperature_scale)); - - return n; -} - - -/* - * Called from Rust's `AsahiDriver::probe` after the DRM device is - * registered. Returns 0 on success, or a negative errno. - */ -int asahi_neo_sysfs_register(struct device *dev, const void *snapshot, - int export_enabled, void **handle) -{ - struct asahi_stats_attribute *stats; - int ret; - - if (!dev || !handle || (export_enabled && !snapshot)) - return -ENODEV; - - BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, pstate) != 16); - BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, busy_ns) != 40); - BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, jobs) != 48); - BUILD_BUG_ON(sizeof(struct asahi_stats_snapshot) != 56); - - stats = kzalloc_obj(*stats); - if (!stats) - return -ENOMEM; - - sysfs_attr_init(&stats->attr.attr); - stats->attr.attr.name = "agx_stats"; - stats->attr.attr.mode = 0444; - stats->attr.show = agx_stats_show; - stats->snapshot = snapshot; - stats->export_enabled = export_enabled; - - ret = device_create_file(dev, &stats->attr); - if (ret) { - kfree(stats); - return ret; - } - - *handle = stats; - return 0; -} -EXPORT_SYMBOL_GPL(asahi_neo_sysfs_register); - -void asahi_neo_sysfs_unregister(struct device *dev, void *handle) -{ - struct asahi_stats_attribute *stats = handle; - - if (!dev || !stats) - return; - - /* Removal drains active show callbacks before either allocation dies. */ - device_remove_file(dev, &stats->attr); - kfree(stats); -} -EXPORT_SYMBOL_GPL(asahi_neo_sysfs_unregister); - -MODULE_AUTHOR("AGX driver maintainer"); -MODULE_LICENSE("Dual MIT/GPL"); -MODULE_DESCRIPTION("Asahi AGX firmware stats sysfs shim"); diff --git a/drivers/gpu/drm/asahi-neo/sysfs.h b/drivers/gpu/drm/asahi-neo/sysfs.h deleted file mode 100644 index 2693d05a84f23f..00000000000000 --- a/drivers/gpu/drm/asahi-neo/sysfs.h +++ /dev/null @@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ - -#ifndef _ASAHI_SYSFS_H -#define _ASAHI_SYSFS_H - -struct device; - -int asahi_neo_sysfs_register(struct device *dev, const void *snapshot, - int export_enabled, void **handle); -void asahi_neo_sysfs_unregister(struct device *dev, void *handle); - -#endif /* _ASAHI_SYSFS_H */ diff --git a/drivers/gpu/drm/asahi-neo/sysfs_exports.rs b/drivers/gpu/drm/asahi-neo/sysfs_exports.rs deleted file mode 100644 index 895d4c75eb2003..00000000000000 --- a/drivers/gpu/drm/asahi-neo/sysfs_exports.rs +++ /dev/null @@ -1,37 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Per-device sysfs registration. The caller retains the snapshot Arc until -//! unregister has drained active readers and freed the C attribute wrapper. - -use core::ffi::{c_int, c_void}; -use crate::stats::StatsSnapshot; - -extern "C" { - fn asahi_neo_sysfs_register( - dev: *mut c_void, - snapshot: *const StatsSnapshot, - export_enabled: c_int, - handle: *mut *mut c_void, - ) -> c_int; - fn asahi_neo_sysfs_unregister(dev: *mut c_void, handle: *mut c_void); -} - -/// Register only after initialization succeeds, keeping the snapshot Arc alive. -pub(crate) fn register( - dev: *mut kernel::bindings::device, - snapshot: *const StatsSnapshot, - export_enabled: bool, -) -> kernel::error::Result<*mut c_void> { - let mut handle = core::ptr::null_mut(); - let ret = unsafe { asahi_neo_sysfs_register(dev.cast(), snapshot, export_enabled as c_int, &mut handle) }; - if ret < 0 { - Err(kernel::error::Error::from_errno(ret)) - } else { - Ok(handle) - } -} - -/// Drain readers before the caller drops its snapshot Arc. -pub(crate) fn unregister(dev: *mut kernel::bindings::device, handle: *mut c_void) { - unsafe { asahi_neo_sysfs_unregister(dev.cast(), handle) }; -} diff --git a/drivers/gpu/drm/asahi-neo/util.rs b/drivers/gpu/drm/asahi-neo/util.rs deleted file mode 100644 index 00246e1f64ab78..00000000000000 --- a/drivers/gpu/drm/asahi-neo/util.rs +++ /dev/null @@ -1,142 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Miscellaneous utility functions - -use core::ops::{ - Add, - BitAnd, - Div, - Not, - Sub, // -}; -use kernel::prelude::*; - -/// Aligns an integer type to a power of two. -pub(crate) fn align(a: T, b: T) -> T -where - T: Copy - + Default - + BitAnd - + Not - + Add - + Sub - + Div - + core::cmp::PartialEq, -{ - let def: T = Default::default(); - #[allow(clippy::eq_op)] - let one: T = !def / !def; - - assert!((b & (b - one)) == def); - - (a + b - one) & !(b - one) -} - -/// Aligns an integer type down to a power of two. -pub(crate) fn align_down(a: T, b: T) -> T -where - T: Copy - + Default - + BitAnd - + Not - + Sub - + Div - + core::cmp::PartialEq, -{ - let def: T = Default::default(); - #[allow(clippy::eq_op)] - let one: T = !def / !def; - - assert!((b & (b - one)) == def); - - a & !(b - one) -} - -pub(crate) trait RangeExt { - fn overlaps(&self, other: Self) -> bool; - fn is_superset(&self, other: Self) -> bool; - // fn len(&self) -> usize; - fn range(&self) -> T; -} - -impl + Default + Copy + Sub> RangeExt for core::ops::Range -where - usize: core::convert::TryFrom, - >::Error: core::fmt::Debug, -{ - fn overlaps(&self, other: Self) -> bool { - !(self.is_empty() || other.is_empty() || self.end <= other.start || other.end <= self.start) - } - fn is_superset(&self, other: Self) -> bool { - !self.is_empty() - && (other.is_empty() || (other.start >= self.start && other.end <= self.end)) - } - fn range(&self) -> T { - if self.is_empty() { - Default::default() - } else { - self.end - self.start - } - } - // fn len(&self) -> usize { - // self.range().try_into().unwrap() - // } -} - -pub(crate) fn gcd(in_n: u64, in_m: u64) -> u64 { - let mut n = in_n; - let mut m = in_m; - - while n != 0 { - let remainder = m % n; - m = n; - n = remainder; - } - - m -} - -pub(crate) unsafe trait AnyBitPattern: Default + Sized + Copy + 'static {} - -pub(crate) struct Reader<'a> { - buffer: &'a [u8], - offset: usize, -} - -impl<'a> Reader<'a> { - pub(crate) fn new(buffer: &'a [u8]) -> Self { - Reader { buffer, offset: 0 } - } - - pub(crate) fn read_up_to(&mut self, declared_size: usize) -> Result { - let mut obj: T = Default::default(); - let size: usize = core::mem::size_of::().min(declared_size); - let end = self.offset.checked_add(declared_size).ok_or(EINVAL)?; - let range = self.offset..end; - let src = self.buffer.get(range).ok_or(EINVAL)?; - - // SAFETY: The output pointer is valid, and the size does not exceed - // the type size, and all bit patterns are valid. - let dst = unsafe { core::slice::from_raw_parts_mut(&mut obj as *mut _ as *mut u8, size) }; - - dst.copy_from_slice(&src[..size]); - self.offset = end; - Ok(obj) - } - - pub(crate) fn read(&mut self) -> Result { - self.read_up_to(core::mem::size_of::()) - } - - pub(crate) fn is_empty(&self) -> bool { - self.offset >= self.buffer.len() - } - - pub(crate) fn skip(&mut self, size: usize) { - self.offset += size - } - - pub(crate) fn rewind(&mut self) { - self.offset = 0 - } -} diff --git a/drivers/gpu/drm/asahi-neo/vm/mod.rs b/drivers/gpu/drm/asahi-neo/vm/mod.rs deleted file mode 100644 index 63cf8a76cd5ce4..00000000000000 --- a/drivers/gpu/drm/asahi-neo/vm/mod.rs +++ /dev/null @@ -1,5 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! Virtual address space management - -mod range; diff --git a/drivers/gpu/drm/asahi-neo/vm/range.rs b/drivers/gpu/drm/asahi-neo/vm/range.rs deleted file mode 100644 index 727d29a5cccf02..00000000000000 --- a/drivers/gpu/drm/asahi-neo/vm/range.rs +++ /dev/null @@ -1,125 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 or MIT - -// Copied from tyr's mmu/vm/range.rs - -//! Range allocator. -//! -//! This module allows you to search for unused ranges to store GEM objects. - -use kernel::alloc::Flags; -use kernel::maple_tree::MapleTreeAlloc; -use kernel::prelude::*; -use kernel::sync::Arc; - -use core::ops::Range; - -/// The actual storage for the ranges. -/// -/// All ranges must fit within the `range` field. -/// -/// The implementation is different on 32-bit and 64-bit cpus. On 64-bit, the 64-bit addresses are -/// stored directly in the maple tree, but on 32-bit, the maple tree stores the ranges translated -/// in the range zero until `range.end-range.start`. This is done because the maple tree uses -/// unsigned long as its address type, which is too small to store the 64-bit address directly on -/// 32-bit machines. -#[pin_data] -struct RangeAllocInner { - #[pin] - maple: MapleTreeAlloc<()>, - range: Range, -} - -/// This object allows you to allocate ranges on the inner maple tree. -pub(crate) struct RangeAlloc { - inner: Arc, -} - -/// Represents a live range in the maple tree. -/// -/// The destructor removes the range from the maple tree, allowing others to allocate it in the -/// future. -pub(crate) struct LiveRange { - inner: Arc, - offset: u64, - size: usize, -} - -impl RangeAlloc { - pub(crate) fn new(start: u64, end: u64, gfp: Flags) -> Result { - if end < start { - return Err(EINVAL); - } - - let inner = Arc::pin_init( - try_pin_init!(RangeAllocInner { - maple <- MapleTreeAlloc::new(), - range: start..end, - }), - gfp, - )?; - - Ok(RangeAlloc { inner }) - } - - pub(crate) fn allocate(&self, size: usize, gfp: Flags) -> Result { - let maple_start = self.inner.range.start as usize; - let maple_end = self.inner.range.end as usize; - - let offset = self - .inner - .maple - .alloc_range(size, (), maple_start..maple_end, gfp)?; - - Ok(LiveRange { - inner: self.inner.clone(), - offset: offset as u64, - size, - }) - } - - pub(crate) fn insert(&self, start: u64, end: u64, gfp: Flags) -> Result { - if end < start { - return Err(EINVAL); - } - if start < self.inner.range.start { - return Err(EINVAL); - } - if end > self.inner.range.end { - return Err(EINVAL); - } - - self.inner - .maple - .insert_range(start as usize..end as usize, (), gfp)?; - - Ok(LiveRange { - inner: self.inner.clone(), - offset: start, - size: (end - start) as usize, - }) - } -} - -impl LiveRange { - pub(crate) fn size(&self) -> usize { - self.size - } - - pub(crate) fn start(&self) -> u64 { - self.offset - } - - pub(crate) fn end(&self) -> u64 { - self.offset + self.size as u64 - } - - pub(crate) fn range(&self) -> Range { - self.start()..self.end() - } -} - -impl Drop for LiveRange { - fn drop(&mut self) { - self.inner.maple.erase(self.offset as usize); - } -} diff --git a/drivers/gpu/drm/asahi-neo/workqueue.rs b/drivers/gpu/drm/asahi-neo/workqueue.rs deleted file mode 100644 index 0eabec8d410c77..00000000000000 --- a/drivers/gpu/drm/asahi-neo/workqueue.rs +++ /dev/null @@ -1,1019 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT - -//! GPU command execution queues -//! -//! The AGX GPU firmware schedules GPU work commands out of work queues, which are ring buffers of -//! pointers to work commands. There can be an arbitrary number of work queues. Work queues have an -//! associated type (vertex, fragment, or compute) and may only contain generic commands or commands -//! specific to that type. -//! -//! This module manages queueing work commands into a work queue and submitting them for execution -//! by the firmware. An active work queue needs an event to signal completion of its work, which is -//! owned by what we call a batch. This event then notifies the work queue when work is completed, -//! and that triggers freeing of all resources associated with that work. An idle work queue gives -//! up its associated event. - -use crate::debug::*; -use crate::fw::channels::{ - ChannelErrorType, - PipeType, // -}; -use crate::fw::types::*; -use crate::fw::workqueue::*; -use crate::no_debug; -use crate::object::OpaqueGpuObject; -use crate::{ - channel, - driver, - event, - fw, - gpu, - regs, // -}; -use core::any::Any; -use core::num::NonZeroU64; -use core::sync::atomic::Ordering; -use kernel::{ - dma_fence, - error::code::*, - new_mutex, - prelude::*, - sync::{ - lock::{ - mutex::MutexBackend, - Guard, // - }, - Arc, - Mutex, // - }, - workqueue::{ - self, - impl_has_work, - new_work, - Work, - WorkItem, // - }, // -}; - -pub(crate) trait OpaqueCommandObject: OpaqueGpuObject {} - -impl OpaqueCommandObject for GpuObject where T: Command {} - -const DEBUG_CLASS: DebugFlags = DebugFlags::WorkQueue; - -const MAX_JOB_SLOTS: u32 = 127; - -/// An enum of possible errors that might cause a piece of work to fail execution. -#[derive(Copy, Clone, Debug, PartialEq, Eq)] -pub(crate) enum WorkError { - /// GPU timeout (command execution took too long). - Timeout, - /// GPU MMU fault (invalid access). - Fault(regs::FaultInfo), - /// Work failed due to an error caused by other concurrent GPU work. - Killed, - /// Channel error - ChannelError(ChannelErrorType), - /// The GPU crashed. - NoDevice, - /// Unknown reason. - Unknown, -} - -impl From for kernel::error::Error { - fn from(err: WorkError) -> Self { - match err { - WorkError::Timeout => ETIMEDOUT, - // Not EFAULT because that's for userspace faults - WorkError::Fault(_) => EIO, - WorkError::Unknown => ENODATA, - WorkError::Killed => ECANCELED, - WorkError::NoDevice => ENODEV, - WorkError::ChannelError(_) => EIO, - } - } -} - -/// A GPU context tracking structure, which must be explicitly invalidated when dropped. -pub(crate) struct GpuContext { - dev: driver::AsahiDevRef, - data: Option>>, -} -no_debug!(GpuContext); - -impl GpuContext { - /// Allocate a new GPU context. - pub(crate) fn new( - dev: &driver::AsahiDevice, - alloc: &mut gpu::KernelAllocators, - buffer: Arc, - ) -> Result { - Ok(GpuContext { - dev: dev.into(), - data: Some(KBox::new( - alloc.shared.new_object( - fw::workqueue::GpuContextData { _buffer: buffer }, - |_inner| Default::default(), - )?, - GFP_KERNEL, - )?), - }) - } - - /// Returns the GPU pointer to the inner GPU context data structure. - pub(crate) fn gpu_pointer(&self) -> GpuPointer<'_, fw::workqueue::GpuContextData> { - self.data.as_ref().unwrap().gpu_pointer() - } -} - -impl Drop for GpuContext { - fn drop(&mut self) { - mod_dev_dbg!(self.dev, "GpuContext: Freeing GPU context\n"); - let data = self.data.take().unwrap(); - // A context only exists for the manager whose allocators created it, so the manager is - // always present. - if let Some(gpu) = (*self.dev).gpu.manager() { - gpu.free_context(data); - } - } -} - -struct SubmittedWork -where - O: OpaqueCommandObject, - C: FnOnce(Option) + Send + Sync + 'static, -{ - object: O, - value: EventValue, - error: Option, - wptr: u32, - vm_slot: u32, - callback: Option, - fence: dma_fence::Fence, -} - -pub(crate) trait GenSubmittedWork: Send + Sync { - fn gpu_va(&self) -> NonZeroU64; - fn value(&self) -> event::EventValue; - fn wptr(&self) -> u32; - fn set_wptr(&mut self, wptr: u32); - fn mark_error(&mut self, error: WorkError); - fn complete(&mut self); - fn get_fence(&self) -> dma_fence::Fence; -} - -#[pin_data] -struct SubmittedWorkContainer { - #[pin] - work: Work, - inner: KBox, -} - -impl_has_work! { - impl HasWork for SubmittedWorkContainer { self.work } -} - -impl WorkItem for SubmittedWorkContainer { - type Pointer = Pin>; - - fn run(this: Pin>) { - mod_pr_debug!("WorkQueue: Freeing command @ {:?}\n", this.inner.gpu_va()); - } -} - -impl SubmittedWorkContainer { - fn inner_mut(self: Pin<&mut Self>) -> &mut KBox { - // SAFETY: inner does not require structural pinning. - unsafe { &mut self.get_unchecked_mut().inner } - } -} - -impl) + Send + Sync> GenSubmittedWork - for SubmittedWork -{ - fn gpu_va(&self) -> NonZeroU64 { - self.object.gpu_va() - } - - fn value(&self) -> event::EventValue { - self.value - } - - fn wptr(&self) -> u32 { - self.wptr - } - - fn set_wptr(&mut self, wptr: u32) { - self.wptr = wptr; - } - - fn complete(&mut self) { - if let Some(cb) = self.callback.take() { - cb(self.error); - } - } - - fn mark_error(&mut self, error: WorkError) { - mod_pr_debug!("WorkQueue: Command at value {:#x?} failed\n", self.value); - self.error = Some(match error { - WorkError::Fault(info) if info.vm_slot != self.vm_slot => WorkError::Killed, - err => err, - }); - } - - fn get_fence(&self) -> dma_fence::Fence { - self.fence.clone() - } -} - -/// Inner data for managing a single work queue. -#[versions(AGX_NEO)] -struct WorkQueueInner { - dev: driver::AsahiDevRef, - event_manager: Arc, - info: GpuObject, - new: bool, - pipe_type: PipeType, - size: u32, - wptr: u32, - pending: KVec>>, - last_token: Option, - pending_jobs: usize, - last_submitted: Option, - last_completed: Option, - event: Option<(event::Event, event::EventValue)>, - priority: u32, - commit_seq: u64, - submit_seq: u64, - event_seq: u64, -} - -/// An instance of a work queue. -#[versions(AGX_NEO)] -#[pin_data] -pub(crate) struct WorkQueue { - info_pointer: GpuWeakPointer, - #[pin] - inner: Mutex, -} - -#[versions(AGX_NEO)] -impl WorkQueueInner::ver { - /// Return the GPU done pointer, representing how many work items have been completed by the - /// GPU. - fn doneptr(&self) -> u32 { - self.info - .state - .with(|raw, _inner| raw.gpu_doneptr.load(Ordering::Acquire)) - } -} - -#[versions(AGX_NEO)] -#[derive(Copy, Clone)] -pub(crate) struct QueueEventInfo { - pub(crate) stamp_pointer: GpuWeakPointer, - pub(crate) fw_stamp_pointer: GpuWeakPointer, - pub(crate) slot: u32, - pub(crate) value: event::EventValue, - pub(crate) cmd_seq: u64, - pub(crate) event_seq: u64, - pub(crate) info_ptr: GpuWeakPointer, -} - -#[versions(AGX_NEO)] -pub(crate) struct Job { - wq: Arc, - event_info: QueueEventInfo::ver, - start_value: EventValue, - pending: KVec>>, - committed: bool, - submitted: bool, - event_count: usize, - fence: dma_fence::Fence, -} - -#[versions(AGX_NEO)] -pub(crate) struct JobSubmission<'a> { - inner: Option>, - wptr: u32, - event_count: usize, - command_count: usize, -} - -#[versions(AGX_NEO)] -impl Job::ver { - pub(crate) fn event_info(&self) -> QueueEventInfo::ver { - let mut info = self.event_info; - info.cmd_seq += self.pending.len() as u64; - info.event_seq += self.event_count as u64; - - info - } - - pub(crate) fn next_seq(&mut self) { - self.event_count += 1; - self.event_info.value.increment(); - } - - pub(crate) fn add( - &mut self, - command: O, - vm_slot: u32, - ) -> Result { - self.add_cb(command, vm_slot, |_| {}) - } - - pub(crate) fn add_cb( - &mut self, - command: O, - vm_slot: u32, - callback: impl FnOnce(Option) + Sync + Send + 'static, - ) -> Result { - if self.committed { - pr_err!("WorkQueue: Tried to mutate committed Job\n"); - return Err(EINVAL); - } - - let fence = self.fence.clone(); - let value = self.event_info.value.next(); - - self.pending.push( - KBox::try_pin_init( - try_pin_init!(SubmittedWorkContainer { - work <- new_work!("SubmittedWorkWrapper::work"), - inner: KBox::new(SubmittedWork::<_, _> { - object: command, - value, - error: None, - callback: Some(callback), - wptr: 0, - vm_slot, - fence, - }, GFP_KERNEL)? - }), - GFP_KERNEL, - )?, - GFP_KERNEL, - )?; - - Ok(()) - } - - pub(crate) fn commit(&mut self) -> Result { - if self.committed { - pr_err!("WorkQueue: Tried to commit committed Job\n"); - return Err(EINVAL); - } - - if self.pending.is_empty() { - pr_err!("WorkQueue: Job::commit() with no commands\n"); - return Err(EINVAL); - } - - let mut inner = self.wq.inner.lock(); - - let ev = inner.event.as_mut().expect("WorkQueue: Job lost its event"); - - if ev.1 != self.start_value { - pr_err!( - "WorkQueue: Job::commit() out of order (event slot {} {:?} != {:?}\n", - ev.0.slot(), - ev.1, - self.start_value - ); - return Err(EINVAL); - } - - ev.1 = self.event_info.value; - inner.commit_seq += self.pending.len() as u64; - inner.event_seq += self.event_count as u64; - self.committed = true; - - Ok(()) - } - - pub(crate) fn can_submit(&self) -> Option { - let inner = self.wq.inner.lock(); - if inner.free_slots() > self.event_count && inner.free_space() > self.pending.len() { - None - } else if let Some(work) = inner.pending.first() { - Some(work.inner.get_fence()) - } else { - pr_err!( - "WorkQueue: Cannot submit, but queue is empty? {} > {}, {} > {} (pend={} ls={:#x?} lc={:#x?}) ev={:#x?} cur={:#x?} slot {:?}\n", - inner.free_slots(), - self.event_count, - inner.free_space(), - self.pending.len(), - inner.pending.len(), - inner.last_submitted, - inner.last_completed, - inner.event.as_ref().map(|a| a.1), - inner.event.as_ref().map(|a| a.0.current()), - inner.event.as_ref().map(|a| a.0.slot()), - ); - None - } - } - - pub(crate) fn submit(&mut self) -> Result> { - if !self.committed { - pr_err!("WorkQueue: Tried to submit uncommitted Job\n"); - return Err(EINVAL); - } - - if self.submitted { - pr_err!("WorkQueue: Tried to submit Job twice\n"); - return Err(EINVAL); - } - - if self.pending.is_empty() { - pr_err!("WorkQueue: Job::submit() with no commands\n"); - return Err(EINVAL); - } - - let mut inner = self.wq.inner.lock(); - - if inner.submit_seq != self.event_info.cmd_seq { - pr_err!( - "WorkQueue: Job::submit() out of order (submit_seq {} != {})\n", - inner.submit_seq, - self.event_info.cmd_seq - ); - return Err(EINVAL); - } - - if inner.commit_seq < (self.event_info.cmd_seq + self.pending.len() as u64) { - pr_err!( - "WorkQueue: Job::submit() out of order (commit_seq {} != {})\n", - inner.commit_seq, - (self.event_info.cmd_seq + self.pending.len() as u64) - ); - return Err(EINVAL); - } - - let mut wptr = inner.wptr; - let command_count = self.pending.len(); - - if inner.free_space() <= command_count { - pr_err!("WorkQueue: Job does not fit in ring buffer\n"); - return Err(EBUSY); - } - - inner.pending.reserve(command_count, GFP_KERNEL)?; - - inner.last_submitted = Some(self.event_info.value); - mod_dev_dbg!( - inner.dev, - "WorkQueue: submitting {} cmds at {:#x?}, lc {:#x?}, cur {:#x?}, pending {}, events {}\n", - self.pending.len(), - inner.last_submitted, - inner.last_completed, - inner.event.as_ref().map(|a| a.0.current()), - inner.pending.len(), - self.event_count, - ); - - for mut command in self.pending.drain(..) { - command.as_mut().inner_mut().set_wptr(wptr); - - let next_wptr = (wptr + 1) % inner.size; - assert!(inner.doneptr() != next_wptr); - inner.info.ring[wptr as usize] = command.inner.gpu_va().get(); - wptr = next_wptr; - - // Cannot fail, since we did a reserve(1) above - inner - .pending - .push(command, GFP_KERNEL) - .expect("push() failed after reserve()"); - } - - self.submitted = true; - - Ok(JobSubmission::ver { - inner: Some(inner), - wptr, - command_count, - event_count: self.event_count, - }) - } -} - -#[versions(AGX_NEO)] -impl<'a> JobSubmission::ver<'a> { - pub(crate) fn run(mut self, channel: &mut channel::PipeChannel::ver) { - let command_count = self.command_count; - let mut inner = self.inner.take().expect("No inner?"); - let wptr = self.wptr; - core::mem::forget(self); - - inner - .info - .state - .with(|raw, _inner| raw.cpu_wptr.store(wptr, Ordering::Release)); - - inner.wptr = wptr; - - let event = inner.event.as_mut().expect("JobSubmission lost its event"); - - let event_slot = event.0.slot(); - - let msg = fw::channels::RunWorkQueueMsg::ver { - pipe_type: inner.pipe_type, - work_queue: Some(inner.info.weak_pointer()), - wptr: inner.wptr, - event_slot, - is_new: inner.new, - __pad: Default::default(), - }; - channel.send(&msg); - inner.new = false; - - inner.submit_seq += command_count as u64; - } - - pub(crate) fn pipe_type(&self) -> PipeType { - self.inner.as_ref().expect("No inner?").pipe_type - } - - pub(crate) fn priority(&self) -> u32 { - self.inner.as_ref().expect("No inner?").priority - } -} - -#[versions(AGX_NEO)] -impl Drop for Job::ver { - fn drop(&mut self) { - mod_pr_debug!("WorkQueue: Dropping Job\n"); - let mut inner = self.wq.inner.lock(); - - if !self.committed { - pr_info!( - "WorkQueue: Dropping uncommitted job with {} events\n", - self.event_count - ); - } - - if self.committed && !self.submitted { - let pipe_type = inner.pipe_type; - let event = inner.event.as_mut().expect("Job lost its event"); - pr_info!( - "WorkQueue({:?}): Roll back {} events (slot {} val {:#x?}) and {} commands\n", - pipe_type, - self.event_count, - event.0.slot(), - event.1, - self.pending.len() - ); - event.1.sub(self.event_count as u32); - inner.commit_seq -= self.pending.len() as u64; - inner.event_seq -= self.event_count as u64; - } - - inner.pending_jobs -= 1; - - if inner.pending.is_empty() && inner.pending_jobs == 0 { - mod_pr_debug!("WorkQueue({:?}): Dropping event\n", inner.pipe_type); - inner.event = None; - inner.last_submitted = None; - inner.last_completed = None; - } - mod_pr_debug!("WorkQueue({:?}): Dropped Job\n", inner.pipe_type); - } -} - -#[versions(AGX_NEO)] -impl<'a> Drop for JobSubmission::ver<'a> { - fn drop(&mut self) { - let inner = self.inner.as_mut().expect("No inner?"); - mod_pr_debug!("WorkQueue({:?}): Dropping JobSubmission\n", inner.pipe_type); - - let new_len = inner.pending.len() - self.command_count; - inner.pending.truncate(new_len); - - let pipe_type = inner.pipe_type; - let event = inner.event.as_mut().expect("JobSubmission lost its event"); - pr_info!( - "WorkQueue({:?}): JobSubmission: Roll back {} events (slot {} val {:#x?}) and {} commands\n", - pipe_type, - self.event_count, - event.0.slot(), - event.1, - self.command_count - ); - event.1.sub(self.event_count as u32); - let val = event.1; - inner.commit_seq -= self.command_count as u64; - inner.event_seq -= self.event_count as u64; - inner.last_submitted = Some(val); - mod_pr_debug!("WorkQueue({:?}): Dropped JobSubmission\n", inner.pipe_type); - } -} - -#[versions(AGX_NEO)] -impl WorkQueueInner::ver { - /// Return the number of free entries in the workqueue - pub(crate) fn free_space(&self) -> usize { - self.size as usize - self.pending.len() - 1 - } - - pub(crate) fn free_slots(&self) -> usize { - let busy_slots = if let Some(ls) = self.last_submitted { - let lc = self - .last_completed - .expect("last_submitted but not completed?"); - ls.delta(&lc) - } else { - 0 - }; - - ((MAX_JOB_SLOTS as i32) - busy_slots).max(0) as usize - } -} - -#[versions(AGX_NEO)] -impl WorkQueue::ver { - /// Create a new WorkQueue of a given type and priority. - #[allow(clippy::too_many_arguments)] - pub(crate) fn new( - dev: &driver::AsahiDevice, - alloc: &mut gpu::KernelAllocators, - event_manager: Arc, - gpu_context: Arc, - notifier_list: Arc>, - pipe_type: PipeType, - id: u64, - priority: u32, - size: u32, - ) -> Result> { - let gpu_buf = alloc.private.array_empty_tagged(0x2c18, b"GPBF")?; - let mut state = alloc.shared.new_default::()?; - let ring = alloc.shared.array_empty(size as usize)?; - let mut prio = *raw::PRIORITY.get(priority as usize).ok_or(EINVAL)?; - - if pipe_type == PipeType::Compute && !debug_enabled(DebugFlags::Debug0) { - // Hack to disable compute preemption until we fix it - prio.0 = 0; - prio.5 = 1; - } - - let inner = WorkQueueInner::ver { - dev: dev.into(), - event_manager, - // Use shared (coherent) state with verbose faults so we can dump state correctly - info: if debug_enabled(DebugFlags::VerboseFaults) { - &mut alloc.shared - } else { - &mut alloc.private - } - .new_init( - try_init!(QueueInfo::ver { - state: { - state.with_mut(|raw, _inner| { - raw.rb_size = size; - }); - state - }, - ring, - gpu_buf, - notifier_list: notifier_list, - gpu_context: gpu_context, - }), - |inner, _p| { - try_init!(raw::QueueInfo::ver { - state: inner.state.gpu_pointer(), - ring: inner.ring.gpu_pointer(), - notifier_list: inner.notifier_list.gpu_pointer(), - gpu_buf: inner.gpu_buf.gpu_pointer(), - gpu_rptr1: Default::default(), - gpu_rptr2: Default::default(), - gpu_rptr3: Default::default(), - event_id: AtomicI32::new(-1), - priority: prio, - unk_4c: -1, - uuid: id as u32, - unk_54: -1, - unk_58: Default::default(), - busy: Default::default(), - __pad: Default::default(), - #[ver(V >= V13_2 && G < G14X)] - unk_84_0: 0, - unk_84_state: Default::default(), - error_count: Default::default(), - unk_8c: 0, - unk_90: 0, - unk_94: 0, - pending: Default::default(), - unk_9c: 0, - gpu_context: inner.gpu_context.gpu_pointer(), - unk_a8: Default::default(), - #[ver(V >= V13_2 && G < G14X)] - unk_b0: 0, - }) - }, - )?, - new: true, - pipe_type, - size, - wptr: 0, - pending: KVec::new(), - last_token: None, - event: None, - priority, - pending_jobs: 0, - commit_seq: 0, - submit_seq: 0, - event_seq: 0, - last_completed: None, - last_submitted: None, - }; - - let info_pointer = inner.info.weak_pointer(); - - Arc::pin_init( - pin_init!(Self { - info_pointer, - inner <- match pipe_type { - PipeType::Vertex => new_mutex!(inner, "WorkQueue::inner (Vertex)"), - PipeType::Fragment => new_mutex!(inner, "WorkQueue::inner (Fragment)"), - PipeType::Compute => new_mutex!(inner, "WorkQueue::inner (Compute)"), - }, - }), - GFP_KERNEL, - ) - } - - pub(crate) fn event_info(&self) -> Option { - let inner = self.inner.lock(); - - inner.event.as_ref().map(|ev| QueueEventInfo::ver { - stamp_pointer: ev.0.stamp_pointer(), - fw_stamp_pointer: ev.0.fw_stamp_pointer(), - slot: ev.0.slot(), - value: ev.1, - cmd_seq: inner.commit_seq, - event_seq: inner.event_seq, - info_ptr: self.info_pointer, - }) - } - - pub(crate) fn new_job(self: &Arc, fence: dma_fence::Fence) -> Result { - let mut inner = self.inner.lock(); - - if inner.event.is_none() { - mod_pr_debug!("WorkQueue({:?}): Grabbing event\n", inner.pipe_type); - let event = inner.event_manager.get(inner.last_token, self.clone())?; - let cur = event.current(); - inner.last_token = Some(event.token()); - mod_pr_debug!( - "WorkQueue({:?}): Grabbed event slot {}: {:#x?}\n", - inner.pipe_type, - event.slot(), - cur - ); - inner.event = Some((event, cur)); - inner.last_submitted = Some(cur); - inner.last_completed = Some(cur); - } - - inner.pending_jobs += 1; - - let ev = &inner.event.as_ref().unwrap(); - - mod_pr_debug!( - "WorkQueue({:?}): New job at value {:#x?} slot {}\n", - inner.pipe_type, - ev.1, - ev.0.slot() - ); - Ok(Job::ver { - wq: self.clone(), - event_info: QueueEventInfo::ver { - stamp_pointer: ev.0.stamp_pointer(), - fw_stamp_pointer: ev.0.fw_stamp_pointer(), - slot: ev.0.slot(), - value: ev.1, - cmd_seq: inner.commit_seq, - event_seq: inner.event_seq, - info_ptr: self.info_pointer, - }, - start_value: ev.1, - pending: KVec::new(), - event_count: 0, - committed: false, - submitted: false, - fence, - }) - } - - pub(crate) fn pipe_type(&self) -> PipeType { - self.inner.lock().pipe_type - } - - pub(crate) fn dump_info(&self) { - pr_info!("WorkQueue @ {:?}:", self.info_pointer); - self.inner.lock().info.with(|raw, _inner| { - pr_info!(" GPU rptr1: {:#x}", raw.gpu_rptr1.load(Ordering::Relaxed)); - pr_info!(" GPU rptr1: {:#x}", raw.gpu_rptr2.load(Ordering::Relaxed)); - pr_info!(" GPU rptr1: {:#x}", raw.gpu_rptr3.load(Ordering::Relaxed)); - pr_info!(" Event ID: {:#x}", raw.event_id.load(Ordering::Relaxed)); - pr_info!(" Busy: {:#x}", raw.busy.load(Ordering::Relaxed)); - pr_info!(" Unk 84: {:#x}", raw.unk_84_state.load(Ordering::Relaxed)); - pr_info!( - " Error count: {:#x}", - raw.error_count.load(Ordering::Relaxed) - ); - pr_info!(" Pending: {:#x}", raw.pending.load(Ordering::Relaxed)); - }); - } - - pub(crate) fn info_pointer(&self) -> GpuWeakPointer { - self.info_pointer - } -} - -/// Trait used to erase the version-specific type of WorkQueues, to avoid leaking -/// version-specificity into the event module. -pub(crate) trait WorkQueue { - /// Cast as an Any type. - fn as_any(&self) -> &dyn Any; - - fn signal(&self) -> bool; - fn mark_error(&self, value: event::EventValue, error: WorkError); - fn fail_all(&self, error: WorkError); -} - -#[versions(AGX_NEO)] -impl WorkQueue for WorkQueue::ver { - fn as_any(&self) -> &dyn Any { - self - } - - /// Signal a workqueue that some work was completed. - /// - /// This will check the event stamp value to find out exactly how many commands were processed. - fn signal(&self) -> bool { - let mut inner = self.inner.lock(); - let event = inner.event.as_ref(); - let value = match event { - None => { - mod_pr_debug!("WorkQueue: signal() called but no event?\n"); - - if inner.pending_jobs > 0 || !inner.pending.is_empty() { - pr_crit!("WorkQueue: signal() called with no event and pending jobs.\n"); - } - return true; - } - Some(event) => event.0.current(), - }; - - if let Some(lc) = inner.last_completed { - if value < lc { - pr_err!( - "WorkQueue: event rolled back? cur {:#x?}, lc {:#x?}, ls {:#x?}", - value, - inner.last_completed, - inner.last_submitted - ); - } - } else { - pr_crit!("WorkQueue: signal() called with no last_completed.\n"); - } - inner.last_completed = Some(value); - - mod_pr_debug!( - "WorkQueue({:?}): Signaling event {:?} value {:#x?}\n", - inner.pipe_type, - inner.last_token, - value - ); - - let mut completed_commands: usize = 0; - - for cmd in inner.pending.iter() { - if cmd.inner.value() <= value { - mod_pr_debug!( - "WorkQueue({:?}): Command at value {:#x?} complete\n", - inner.pipe_type, - cmd.inner.value() - ); - completed_commands += 1; - } else { - break; - } - } - - if completed_commands == 0 { - return inner.pending.is_empty(); - } - - let last_wptr = inner.pending[completed_commands - 1].inner.wptr(); - let pipe_type = inner.pipe_type; - - for mut cmd in inner.pending.drain(..completed_commands) { - mod_pr_debug!( - "WorkQueue({:?}): Queueing command @ {:?} for cleanup\n", - pipe_type, - cmd.inner.gpu_va() - ); - cmd.as_mut().inner_mut().complete(); - workqueue::system().enqueue(cmd); - } - - mod_pr_debug!( - "WorkQueue({:?}): Completed {} commands, left pending {}, ls {:#x?}, lc {:#x?}\n", - inner.pipe_type, - completed_commands, - inner.pending.len(), - inner.last_submitted, - inner.last_completed, - ); - - inner - .info - .state - .with(|raw, _inner| raw.cpu_freeptr.store(last_wptr, Ordering::Release)); - - let empty = inner.pending.is_empty(); - if empty && inner.pending_jobs == 0 { - inner.event = None; - inner.last_submitted = None; - inner.last_completed = None; - } - - empty - } - - /// Mark this queue's work up to a certain stamp value as having failed. - fn mark_error(&self, value: event::EventValue, error: WorkError) { - // If anything is marked completed, we can consider it successful - // at this point, even if we didn't get the signal event yet. - self.signal(); - - let mut inner = self.inner.lock(); - - if inner.event.is_none() { - mod_pr_debug!("WorkQueue: signal_fault() called but no event?\n"); - - if inner.pending_jobs > 0 || !inner.pending.is_empty() { - pr_crit!("WorkQueue: signal_fault() called with no event and pending jobs.\n"); - } - return; - } - - mod_pr_debug!( - "WorkQueue({:?}): Signaling fault for event {:?} at value {:#x?}\n", - inner.pipe_type, - inner.last_token, - value - ); - - for cmd in inner.pending.iter_mut() { - if cmd.inner.value() <= value { - cmd.as_mut().inner_mut().mark_error(error); - } else { - break; - } - } - } - - /// Mark all of this queue's work as having failed, and complete it. - fn fail_all(&self, error: WorkError) { - // If anything is marked completed, we can consider it successful - // at this point, even if we didn't get the signal event yet. - self.signal(); - - let mut inner = self.inner.lock(); - - if inner.event.is_none() { - mod_pr_debug!("WorkQueue: fail_all() called but no event?\n"); - - if inner.pending_jobs > 0 || !inner.pending.is_empty() { - pr_crit!("WorkQueue: fail_all() called with no event and pending jobs.\n"); - } - return; - } - - mod_pr_debug!( - "WorkQueue({:?}): Failing all jobs {:?}\n", - inner.pipe_type, - error - ); - - let mut cmds = KVec::new(); - - core::mem::swap(&mut inner.pending, &mut cmds); - - if inner.pending_jobs == 0 { - inner.event = None; - } - - core::mem::drop(inner); - - for mut cmd in cmds { - cmd.as_mut().inner_mut().mark_error(error); - cmd.as_mut().inner_mut().complete(); - } - } -} diff --git a/drivers/gpu/drm/asahi/asahi.rs b/drivers/gpu/drm/asahi/asahi.rs index a01ae008af44cb..46d73120a918cd 100644 --- a/drivers/gpu/drm/asahi/asahi.rs +++ b/drivers/gpu/drm/asahi/asahi.rs @@ -6,6 +6,7 @@ mod alloc; mod buffer; mod channel; +mod cleanup; #[cfg(CONFIG_DEV_COREDUMP)] mod crashdump; mod debug; @@ -110,6 +111,7 @@ mod agx_resources; mod agx_status; mod agx_uapi; mod gem; +mod g17; mod gpu; mod hw; mod identity; @@ -130,8 +132,32 @@ mod util; mod vm; mod workqueue; -kernel::module_platform_driver! { - type: driver::AsahiDriver, +use kernel::prelude::*; + +#[pin_data] +struct DriverModule { + // Field drop order unregisters/joins driver producers before queue teardown. + #[pin] + _driver: kernel::driver::Registration>, + // Deferred G17P cleanup work; allocated at module load on every platform. + _cleanup: cleanup::QueueOwner, +} + +impl kernel::InPlaceModule for DriverModule { + fn init(module: &'static kernel::ThisModule) -> impl PinInit { + try_pin_init!(Self { + // Initialize the queue first, before registration can invoke probe. + _cleanup: cleanup::QueueOwner::new()?, + _driver <- kernel::driver::Registration::new( + ::NAME, + module, + ), + }) + } +} + +module! { + type: DriverModule, name: "asahi", description: "AGX GPU driver for Apple silicon SoCs", license: "Dual MIT/GPL", diff --git a/drivers/gpu/drm/asahi-neo/cleanup.rs b/drivers/gpu/drm/asahi/cleanup.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/cleanup.rs rename to drivers/gpu/drm/asahi/cleanup.rs diff --git a/drivers/gpu/drm/asahi/driver.rs b/drivers/gpu/drm/asahi/driver.rs index 517a10c7a2f91e..b7a02abb7a0a15 100644 --- a/drivers/gpu/drm/asahi/driver.rs +++ b/drivers/gpu/drm/asahi/driver.rs @@ -26,6 +26,7 @@ use crate::{ drm_gpu, file, g15_boot, + g17, gem::AsahiObject, gpu, hw, @@ -44,6 +45,9 @@ pub(crate) struct AsahiData { pub(crate) completion: SetOnce>, pub(crate) pdev: ARef, pub(crate) resources: Option, + /// Kernel GEM objects are marked held for the device (`gem::set_mapping_held`). Only the + /// G17P runtime sets this. + pub(crate) hold_kernel_pages: bool, #[pin] g16_completion_work: Work, @@ -58,10 +62,33 @@ impl AsahiData { completion: SetOnce::new(), pdev, resources, + hold_kernel_pages: false, g16_completion_work <- new_work!("AsahiData::g16_completion_work"), }) } + /// Driver data of a G17P device. + fn new_g17(pdev: &platform::Device, resources: regs::Resources) -> impl PinInit { + let pdev: ARef = pdev.into(); + try_pin_init!(Self { + gpu: SetOnce::new(), + is_m3: false, + completion: SetOnce::new(), + pdev, + resources: Some(resources), + hold_kernel_pages: true, + g16_completion_work <- new_work!("AsahiData::g16_completion_work"), + }) + } + + /// The G17P runtime, if this device runs it. + pub(crate) fn g17(&self) -> Option<&Arc> { + match self.gpu.as_ref() { + Some(drm_gpu::Backend::G17(gpu)) => Some(gpu), + _ => None, + } + } + /// Runtime construction can use GEM before the GPU backend exists. All /// DRM operations fail with ENODEV until probe publishes the backend once. pub(crate) fn gpu(&self) -> Result<&dyn drm_gpu::DrmGpu> { @@ -89,6 +116,7 @@ enum AsahiRuntime { M3(crate::m3_drm::Registered), G16(crate::g16_drm::Registered), G15(crate::g15_probe::G15Manager), + G17(ARef>, Arc), } pub(crate) struct AsahiDriver { @@ -103,6 +131,8 @@ pub(crate) enum ProbeConfig { /// topology decode run at probe time, then the probe fails closed before /// DMA setup, ASC start, or any MMIO write. Agx3Diagnostic(&'static hw::agx3::SocConfig), + /// G17 (T8140): the G17P firmware runtime. + G17(&'static g17::Config), } unsafe impl Send for AsahiDriver {} @@ -132,6 +162,8 @@ impl drm::driver::Driver for AsahiDriver { type Object = drm::gem::shmem::Object; const INFO: drm::driver::DriverInfo = INFO; + /// Open files and exported buffers pin the module while G17P deferred cleanup can run. + const MODULE: Option<&'static kernel::ThisModule> = Some(&crate::THIS_MODULE); const FEATURES: u32 = drm::driver::FEAT_GEM | drm::driver::FEAT_RENDER | drm::driver::FEAT_SYNCOBJ @@ -161,6 +193,8 @@ impl drm::driver::Driver for AsahiDriver { ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::queue_destroy), (ASAHI_SUBMIT, drm_asahi_submit, ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::submit), + (ASAHI_GEM_MADVISE, drm_asahi_gem_madvise, + ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_madvise), } } @@ -209,8 +243,8 @@ kernel::of_device_table!( // while its table is incomplete; T6031 starts that runtime only with // asahi.t6031_start=1 and otherwise fails closed the same way; the // others fail closed before any firmware handoff. - // The M4 (G16) and M5/A18 Pro (G17) bring-up runtimes are not bound - // in this kernel. + // The M4 (G16) and M5 bring-up runtimes are not bound in this kernel. + // The A18 Pro (T8140, G17P) runtime is bound below. ( of::DeviceId::new(c_str!("apple,agx-t8122")), ProbeConfig::Agx3Diagnostic(&hw::agx3::T8122) @@ -227,6 +261,10 @@ kernel::of_device_table!( of::DeviceId::new(c_str!("apple,agx-t6032")), ProbeConfig::Agx3Diagnostic(&hw::agx3::T6032) ), + ( + of::DeviceId::new(c_str!("apple,agx-t8140")), + ProbeConfig::G17(&hw::t8140::CONFIG) + ), ] ); @@ -417,6 +455,8 @@ impl platform::Driver for AsahiDriver { match &this.runtime { AsahiRuntime::M3(runtime) => runtime.stop(), AsahiRuntime::G16(runtime) => runtime.stop(), + // Driver data is dropped after devres, so CPU control must happen in unbind. + AsahiRuntime::G17(_, gpu) => gpu.shutdown(), _ => {}, } } @@ -483,6 +523,7 @@ impl platform::Driver for AsahiDriver { } } ProbeConfig::Agx3Diagnostic(soc) => return Err(refuse_agx3_probe(pdev, soc)), + ProbeConfig::G17(cfg) => return Self::probe_g17(pdev, cfg), }; unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(cfg.uat_oas)?)? }; @@ -544,3 +585,35 @@ impl platform::Driver for AsahiDriver { }) } } + +impl AsahiDriver { + fn probe_g17(pdev: &platform::Device, cfg: &'static g17::Config) -> Result { + // SAFETY: The GPU performs DMA through a UAT whose output width is part of the SoC config. + unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(cfg.uat_oas)?)? }; + let res = regs::Resources::new(pdev)?; + // No userspace or GPU backend is published before the runtime is constructed. + let drm: ARef = + drm::device::Device::new(pdev.as_ref(), AsahiData::new_g17(pdev, res.clone()))?; + let gpu = g17::Gpu::new(pdev, &drm, cfg, res)?; + if !drm.gpu.populate(drm_gpu::Backend::G17(gpu.clone())) { + gpu.shutdown(); + return Err(EBUSY); + } + if let Err(error) = drm::driver::Registration::new_foreign_owned(&drm, pdev.as_ref(), 0) { + gpu.shutdown(); + return Err(error); + } + Ok(Self { + runtime: AsahiRuntime::G17(drm, gpu), + }) + } +} + +impl Drop for AsahiDriver { + fn drop(&mut self) { + if let AsahiRuntime::G17(_, gpu) = &self.runtime { + // Also covers failure to allocate platform driver data after the probe body returned. + gpu.shutdown(); + } + } +} diff --git a/drivers/gpu/drm/asahi/drm_gpu.rs b/drivers/gpu/drm/asahi/drm_gpu.rs index c9d997186d12a6..cd1b720b55f6b7 100644 --- a/drivers/gpu/drm/asahi/drm_gpu.rs +++ b/drivers/gpu/drm/asahi/drm_gpu.rs @@ -4,7 +4,8 @@ //! //! The legacy manager remains unchanged behind [`LegacyDrmGpu`]. M3 uses //! the same file and ioctl layer through its own adapter without pretending to -//! be an AGX2 firmware manager. +//! be an AGX2 firmware manager. The G17P runtime reaches this layer through its +//! [`gpu::Gpu`] implementation, adapted below. use core::ops::Range; @@ -14,7 +15,7 @@ use kernel::{ uapi, }; -use crate::{alloc, file, fw, gem, gpu, mmu, queue}; +use crate::{alloc, file, fw, g17, gem, gpu, mmu, queue}; /// Hardware facts returned through `DRM_ASAHI_GET_PARAMS`. pub(crate) struct DrmGpuParams { @@ -71,6 +72,16 @@ pub(crate) trait DrmGpu: Send + Sync { fn fwctl(&self, _msg: fw::channels::FwCtlMsg) -> Result { Err(ENODEV) } + + /// Independent-queue limits; only the G17P runtime has them. + fn queue_limits(&self) -> Option { + None + } + /// Temporary capacity hint and validity in milliseconds for subsequent syncobj waits; + /// only the G17P runtime gives one. + fn syncobj_wait_hint(&self) -> Option<(u32, u32)> { + None + } } /// Publish an existing manager without allocating another reference-counted adapter. @@ -78,6 +89,7 @@ pub(crate) enum Backend { Legacy(LegacyDrmGpu), M3(Arc), G16(Arc), + G17(Arc), } impl Backend { @@ -85,7 +97,110 @@ impl Backend { match self { Self::Legacy(manager) => manager, Self::M3(runtime) | Self::G16(runtime) => &**runtime, + Self::G17(runtime) => &**runtime, + } + } +} + +/// The G17P runtime's `gpu::Gpu` operations, as the file layer sees them. +impl DrmGpu for g17::Gpu { + fn init(&self) -> Result { + gpu::Gpu::init(self) + } + + fn ids(&self) -> &gpu::SequenceIDs { + gpu::Gpu::ids(self) + } + + fn is_crashed(&self) -> bool { + gpu::Gpu::is_crashed(self) + } + + fn supports_vm_status(&self) -> bool { + gpu::Gpu::supports_vm_status(self) + } + + /// G17P has no G13/G14 globals. + fn update_globals(&self) {} + + fn extra_features(&self) -> u64 { + // The out-of-line bindgen `Default` is not exported to modules; zero-fill instead. + let mut params: uapi::drm_asahi_params_global = pin_init::zeroed(); + if gpu::Gpu::get_params(self, &mut params).is_err() { + return 0; + } + let mut features = params.features; + if gpu::Gpu::supports_vm_status(self) { + features |= uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_VM_STATUS_MIRROR as u64; + } + features + } + + fn params(&self) -> Result { + // The out-of-line bindgen `Default` is not exported to modules; zero-fill instead. + let mut params: uapi::drm_asahi_params_global = pin_init::zeroed(); + gpu::Gpu::get_params(self, &mut params)?; + let mut core_masks = [0u32; uapi::DRM_ASAHI_MAX_CLUSTERS as usize]; + for (mask, reported) in core_masks.iter_mut().zip(params.core_masks.iter()) { + *mask = u32::try_from(*reported).map_err(|_| EIO)?; } + Ok(DrmGpuParams { + gpu_generation: params.gpu_generation, + gpu_variant: params.gpu_variant, + gpu_revision: params.gpu_revision, + chip_id: params.chip_id, + num_dies: params.num_dies, + num_clusters_total: params.num_clusters_total, + num_cores_per_cluster: params.num_cores_per_cluster, + core_masks, + max_frequency_khz: params.max_frequency_khz, + usc_generation: params.usc_generation, + gpu_hal_generation: params.gpu_hal_generation, + max_commands_per_submission: file::MAX_COMMANDS_PER_SUBMISSION, + }) + } + + fn user_range(&self) -> Result> { + Ok(gpu::Gpu::uat_geometry(self).user_usable_range()) + } + + fn unknown_page(&self) -> Result { + Ok(gpu::Gpu::uat_geometry(self).unk_page()) + } + + fn base_clock_hz(&self) -> u32 { + gpu::Gpu::base_clock_hz(self) + } + + fn new_vm(&self, kernel_range: Range) -> Result { + gpu::Gpu::new_vm(self, kernel_range) + } + + fn new_queue( + &self, + vm: mmu::Vm, + ualloc: Arc>, + ualloc_priv: Arc>, + priority: u32, + usc_exec_base: u64, + ) -> Result> { + gpu::Gpu::new_queue(self, vm, ualloc, ualloc_priv, priority, usc_exec_base) + } + + fn map_timestamp_buffer( + &self, + bo: gem::ObjectRef, + range: Range, + ) -> Result { + gpu::Gpu::map_timestamp_buffer(self, bo, range) + } + + fn queue_limits(&self) -> Option { + gpu::Gpu::queue_limits(self) + } + + fn syncobj_wait_hint(&self) -> Option<(u32, u32)> { + gpu::Gpu::syncobj_wait_hint(self) } } diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 4738a825fa83e6..e977f63617bd3f 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -63,6 +63,8 @@ const LEGACY_VM_USER_RANGE: Range = mmu::IOVA_USER_BASE..mmu::IOVA_UNK_PAGE /// A client instance of an `mmu::Vm` address space. struct Vm { + /// Residency reclaim of a G17 VM; None elsewhere. + shrinker: Option>, ualloc: Arc>, ualloc_priv: Arc>, vm: mmu::Vm, @@ -73,6 +75,27 @@ struct Vm { impl Drop for Vm { fn drop(&mut self) { + if self.vm.status().is_some() { + // G17 VM. + self.shrinker = None; + self.vm.close_idle_reclaim(); + if let Some(status) = self.vm.status() { + // Retained graphs may keep this status alive; the client's mirror + // object must not outlive the client's VM handle. + status.release_mirror(); + } + // Tracked jobs can outlive their file's VM handle. Their last guard performs + // these unmaps once no accepted work can still dereference the mappings. + if let Err(error) = self + .vm + .unmap_user_ranges(self.user_range.clone(), self.kernel_range.clone()) + { + pr_err!("Vm::Drop: deferred user unmap failed: {:?}\n", error); + } + self.vm.bo_deferred_cleanup(); + return; + } + if !self.vm.is_m3() { // When the user Vm is dropped, unmap everything in the user range @@ -185,11 +208,11 @@ impl SyncItem { in_count: u32, out_count: u32, ) -> Result> { - let count = in_count + out_count; - let mut vec = KVec::with_capacity(count as usize, GFP_KERNEL)?; + let count = in_count.checked_add(out_count).ok_or(EOVERFLOW)?; const STRIDE: usize = core::mem::size_of::(); - let size = STRIDE * count as usize; + let size = STRIDE.checked_mul(count as usize).ok_or(EOVERFLOW)?; + let mut vec = KVec::with_capacity(count as usize, GFP_KERNEL)?; // SAFETY: We only read this once, so there are no TOCTOU issues. let mut reader = UserSlice::new(UserPtr::from_addr(ptr as _), size).reader(); @@ -314,8 +337,30 @@ impl File { let gpu = device.gpu()?; - if data.param_group == crate::agx_status::PARAM_GROUP_VM_STATUS { - return Self::get_vm_status(device, data, file); + match data.param_group { + crate::agx_status::PARAM_GROUP_VM_STATUS => { + return Self::get_vm_status(device, data, file); + } + uapi::DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR if device.g17().is_some() => { + return Self::register_vm_status_mirror(data, file); + } + uapi::DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS => { + if data.pad != 0 + || data.size < core::mem::size_of::() as u64 + { + return Err(EINVAL); + } + let limits = gpu.queue_limits().ok_or(EINVAL)?; + let words = [limits.max_queues, limits.max_in_flight_per_queue, 0, 0]; + UserSlice::new( + UserPtr::from_addr(data.pointer as _), + core::mem::size_of_val(&words), + ) + .writer() + .write(&words)?; + return Ok(0); + } + _ => {} } if data.param_group != 0 || data.pad != 0 { @@ -359,6 +404,10 @@ impl File { params.features |= crate::agx_status::FEATURE_VM_STATUS; } + if gpu.queue_limits().is_some() { + params.features |= uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES as u64; + } + if gpu.supports_scheduled_queues() { params.features |= uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_SCHEDULED_QUEUES as u64; } @@ -383,6 +432,10 @@ impl File { { params.features |= uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_SOFT_FAULTS as u64; } + // The G17P runtime keeps its own soft-fault rule. + if device.g17().is_some() && *module_parameters::fault_control.value() == 0xb { + params.features |= uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_SOFT_FAULTS as u64; + } let size = core::mem::size_of::().min(data.size.try_into()?); @@ -410,27 +463,76 @@ impl File { let mut bytes = [0u8; VM_STATUS_SIZE]; UserSlice::new(user, VM_STATUS_SIZE).reader().read_slice(&mut bytes)?; let vm_id = decode_request(&bytes).ok_or(EINVAL)?; - let status = { + let (g17_status, status) = { let vms = file.inner().vms(); let guard = vms.lock(); let vm = guard.get(vm_id as usize).ok_or(ENOENT)?; - vm.vm.status()?.clone() + match vm.vm.status() { + // G17 VMs keep their own sticky status object. + Some(status) => (Some(status.clone()), None), + None => (None, Some(vm.vm.agx_status()?.clone())), + } + }; + let error = match (g17_status, status) { + (Some(status), _) => status.reported_error(), + (None, Some(status)) => { + status.record(device.gpu()?.submission_error()); + status.get() + } + (None, None) => return Err(EINVAL), }; - status.record(device.gpu()?.submission_error()); - let bytes = encode_response(vm_id, status.get()); + let bytes = encode_response(vm_id, error); UserSlice::new(user, VM_STATUS_SIZE).writer().write_slice(&bytes)?; Ok(0) } + /// Register a file-owned GEM word as the mirror of a file-owned VM's status. + fn register_vm_status_mirror(data: &uapi::drm_asahi_get_params, file: &DrmFile) -> Result { + let size = core::mem::size_of::(); + if data.pad != 0 || data.size < size as u64 { + return Err(EINVAL); + } + let [vm_id, handle, offset, flags] = + UserSlice::new(UserPtr::from_addr(data.pointer as _), size) + .reader() + .read::<[u32; 4]>()?; + if flags != 0 || handle == 0 || offset % core::mem::align_of::() as u32 != 0 { + return Err(EINVAL); + } + let status = Self::vm_status(file, vm_id)?; + let owner = gem::ObjectRef::new(gem::Object::lookup_handle(file, handle)?); + status.attach_mirror(owner, offset as usize)?; + Ok(0) + } + + fn vm_status(file: &DrmFile, vm_id: u32) -> Result> { + let vms = file.inner().vms(); + let guard = vms.lock(); + Ok(guard + .get(vm_id as usize) + .ok_or(ENOENT)? + .vm + .status() + .ok_or(EINVAL)? + .clone()) + } + /// IOCTL: vm_create: Create a new `Vm`. pub(crate) fn vm_create( device: &AsahiDevice, data: &mut uapi::drm_asahi_vm_create, file: &DrmFile, ) -> Result { - let hal = device.gpu()?.params()?.gpu_hal_generation; - let expected = if hal == 200 { uapi::drm_asahi_vm_create_flags_DRM_ASAHI_VM_CREATE_HAL200 } else { 0 }; - if data.flags != expected { return Err(EINVAL); } + if device.g17().is_some() { + // G17P reports HAL200; it accepts the acknowledgement but does not require it. + if data.flags & !uapi::drm_asahi_vm_create_flags_DRM_ASAHI_VM_CREATE_HAL200 != 0 { + return Err(EINVAL); + } + } else { + let hal = device.gpu()?.params()?.gpu_hal_generation; + let expected = if hal == 200 { uapi::drm_asahi_vm_create_flags_DRM_ASAHI_VM_CREATE_HAL200 } else { 0 }; + if data.flags != expected { return Err(EINVAL); } + } if let Some(gate) = &file.inner().m3_client { gate.admit_vm(device, data.flags)?; } let kernel_range = data.kernel_start..data.kernel_end; let user_range = device.gpu()?.user_range()?; @@ -516,6 +618,11 @@ impl File { let profile_vm_id = vm.id(); resv.fill(KBox::new( Vm { + shrinker: if vm.residency_reclaim_enabled() { + Some(mmu::VmShrinker::new(&vm)?) + } else { + None + }, ualloc, ualloc_priv, vm, @@ -574,21 +681,20 @@ impl File { } let resv_gem; + let mut reclaim_cpu = false; let resv_obj = if data.flags & uapi::drm_asahi_gem_flags_DRM_ASAHI_GEM_VM_PRIVATE != 0 { - resv_gem = file - .inner() - .vms() - .lock() - .get(data.vm_id.try_into()?) - .ok_or(ENOENT)? - .vm - .get_resv_obj(); + let vms = file.inner().vms(); + let vms = vms.lock(); + let vm = &vms.get(data.vm_id.try_into()?).ok_or(ENOENT)?.vm; + // Only G17 VMs enable residency reclaim. + reclaim_cpu = vm.residency_reclaim_enabled(); + resv_gem = vm.get_resv_obj(); Some(resv_gem.deref()) } else { None }; - let gem = gem::new_object(device, data.size.try_into()?, data.flags, resv_obj)?; + let gem = gem::new_object(device, data.size.try_into()?, data.flags, resv_obj, reclaim_cpu)?; let handle = gem.create_handle(file)?; data.handle = handle; @@ -627,6 +733,27 @@ impl File { Ok(0) } + /// IOCTL: gem_madvise: Mark a VM-private object purgeable or needed. + pub(crate) fn gem_madvise( + device: &AsahiDevice, + data: &mut uapi::drm_asahi_gem_madvise, + file: &DrmFile, + ) -> Result { + // Only the G17P runtime implements purgeable objects. + if device.g17().is_none() { + return Err(EINVAL); + } + if data.pad != 0 || data.madv > uapi::drm_asahi_gem_madv_DRM_ASAHI_GEM_MADV_DONTNEED { + return Err(EINVAL); + } + let gem = gem::Object::lookup_handle(file, data.handle)?; + data.retained = gem::madvise( + &gem, + data.madv == uapi::drm_asahi_gem_madv_DRM_ASAHI_GEM_MADV_DONTNEED, + )?; + Ok(0) + } + /// IOCTL: vm_bind: Map or unmap memory into a Vm. pub(crate) fn vm_bind( device: &AsahiDevice, @@ -645,6 +772,10 @@ impl File { return Err(EINVAL); } + if device.g17().is_some() { + return Self::vm_bind_g17(data, file); + } + let vm_id = data.vm_id.try_into()?; let mut vec = KVec::new(); @@ -672,6 +803,94 @@ impl File { Ok(0) } + /// Validate all operations and retain their objects before changing a tracked VM. + fn vm_bind_g17(data: &uapi::drm_asahi_vm_bind, file: &DrmFile) -> Result { + let count = usize::try_from(data.num_binds)?; + let stride = usize::try_from(data.stride)?; + let size = stride.checked_mul(count).ok_or(EOVERFLOW)?; + let mut bytes = KVec::new(); + UserSlice::new(UserPtr::from_addr(data.userptr as _), size) + .reader() + .read_all(&mut bytes, GFP_KERNEL)?; + let (vm, user_range, kernel_range) = { + let vms = file.inner().vms(); + let guard = vms.lock(); + let entry = guard.get(data.vm_id as usize).ok_or(ENOENT)?; + ( + entry.vm.clone(), + entry.user_range.clone(), + entry.kernel_range.clone(), + ) + }; + let mut batch = mmu::PreparedUserBindBatch::new(count)?; + let mut unmaps = 0; + // A fresh reader per declared stride also skips extension bytes in longer records. + for bytes in bytes.chunks_exact(stride) { + let op: uapi::drm_asahi_gem_bind_op = Reader::new(bytes).read_up_to(stride)?; + if op.range == 0 || (op.addr | op.range | op.offset) & mmu::UAT_PGMSK as u64 != 0 { + return Err(EINVAL); + } + let range = op.addr..op.addr.checked_add(op.range).ok_or(EINVAL)?; + if !user_range.is_superset(range.clone()) + || kernel_range.overlaps(range.clone()) + || vm.driver_range_overlaps(range.clone()) + { + return Err(EINVAL); + } + if op.flags & uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_UNBIND != 0 { + if op.offset != 0 + || op.flags != uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_UNBIND + || op.handle != 0 + { + return Err(EINVAL); + } + batch.push_unmap(vm.prepare_user_unmap(op.addr, op.range)?)?; + unmaps += 1; + continue; + } + vm.wait_for_user_map_admission()?; + if op.flags + & !(uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_READ + | uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_WRITE + | uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_SINGLE_PAGE) + != 0 + { + return Err(EINVAL); + } + let read = op.flags & uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_READ != 0; + let write = op.flags & uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_WRITE != 0; + let prot = match (read, write) { + (true, true) => mmu::PROT_GPU_SHARED_RW, + (true, false) => mmu::PROT_GPU_SHARED_RO, + (false, true) => mmu::PROT_GPU_SHARED_WO, + (false, false) => return Err(EINVAL), + }; + let single_page = op.flags & uapi::drm_asahi_bind_flags_DRM_ASAHI_BIND_SINGLE_PAGE != 0; + let object = gem::Object::lookup_handle(file, op.handle)?; + let accessed = if single_page { + mmu::UAT_PGSZ as u64 + } else { + op.range + }; + if op.offset.checked_add(accessed).ok_or(EINVAL)? > u64::try_from(object.size())? { + return Err(EINVAL); + } + vm.validate_context_binding(op.addr, op.range, single_page)?; + batch.push_map(vm.prepare_bind_object( + &object, + op.addr, + op.range, + op.offset, + prot, + single_page, + )?)?; + } + vm.reserve_deferred_user_unmaps(unmaps)?; + vm.commit_prepared_user_bind_batch(&mut batch)?; + vm.bo_deferred_cleanup(); + Ok(0) + } + pub(crate) fn do_gem_bind_unbind( vm_id: usize, data: &uapi::drm_asahi_gem_bind_op, @@ -994,7 +1213,10 @@ impl File { .inner() .objects() .lock() - .insert_limit(1..=u32::MAX, obj, GFP_KERNEL)? as u64; + .insert_limit(1..=u32::MAX, obj, GFP_KERNEL); + // StoreError owns the rejected object. Convert it only after the + // temporary XArray guard is gone: dropping a mapping may sleep. + let handle = handle? as u64; data.object_handle = handle as u32; Ok(0) @@ -1081,7 +1303,7 @@ impl File { let _ = file_vm; core::mem::drop(guard); - let queue = device.gpu()?.new_queue( + let mut queue = device.gpu()?.new_queue( vm, ualloc, ualloc_priv, @@ -1090,6 +1312,9 @@ impl File { data.usc_exec_base, )?; + // Only G17P queues accept a nonzero render ray query scratch layout. + queue.reserve_render_scs(data.render_scs_layout)?; + data.queue_id = resv.index().try_into()?; resv.fill(Arc::pin_init(new_mutex!(queue), GFP_KERNEL)?)?; @@ -1152,6 +1377,21 @@ impl File { data.queue_id, id ); + let mut vec = KVec::new(); + let copy_commands = |bytes: &mut KVec| { + UserSlice::new( + UserPtr::from_addr(data.cmdbuf as _), + data.cmdbuf_size as usize, + ) + .reader() + .read_all(bytes, GFP_KERNEL) + }; + let guarded = device.g17().is_some(); + // Once G17 output syncobjs are resolved, its queue must install a fence + // on every failure. Take the fallible userspace snapshot before that point. + if guarded { + copy_commands(&mut vec)?; + } let syncs = SyncItem::parse_array(file, data.syncs, data.in_sync_count, data.out_sync_count)?; @@ -1163,23 +1403,24 @@ impl File { id ); - let mut vec = KVec::new(); - // Copy the command buffer into the kernel. Because we need to iterate // the command buffer twice, we do this in one big copy_from_user to // avoid TOCTOU issues. - let reader = UserSlice::new( - UserPtr::from_addr(data.cmdbuf as _), - data.cmdbuf_size as usize, - ) - .reader(); - reader.read_all(&mut vec, GFP_KERNEL)?; + if !guarded { + copy_commands(&mut vec)?; + } let objects = file.inner().objects(); let ret = queue .lock() .submit(id, syncs, data.in_sync_count as usize, &vec, objects); + if ret.is_ok() { + if let Some((util_min, duration_ms)) = gpu.syncobj_wait_hint() { + file.set_syncobj_wait_hint(util_min, duration_ms); + } + } + match ret { Err(ERESTARTSYS) => Err(ERESTARTSYS), Err(e) => { diff --git a/drivers/gpu/drm/asahi/g16_submit.rs b/drivers/gpu/drm/asahi/g16_submit.rs index 00b35afb9b699a..e5913424398894 100644 --- a/drivers/gpu/drm/asahi/g16_submit.rs +++ b/drivers/gpu/drm/asahi/g16_submit.rs @@ -115,7 +115,7 @@ impl Packet { if let Err(error) = result { let _ = self.failure.compare_exchange(0, error.to_errno(), Ordering::Relaxed, Ordering::Relaxed); - if let Ok(status) = self.vm.status() { status.record(error.to_errno()); } + if let Ok(status) = self.vm.agx_status() { status.record(error.to_errno()); } self.completion.set_error(error); } if result.is_ok() { @@ -134,8 +134,14 @@ impl Packet { } pub(crate) struct Job { shared: Shared, packet: Arc, dev: driver::AsahiDevRef } +/// Timeout state of a [`Job`], owned by the scheduler's timeout worker. +pub(crate) struct JobTimeout { shared: Shared, packet: Arc, dev: driver::AsahiDevRef } impl sched::JobImpl for Job { - fn run(job: &mut sched::Job) -> Result> { + type TimeoutData = JobTimeout; + fn timeout_data(&self) -> JobTimeout { + JobTimeout { shared: self.shared.clone(), packet: self.packet.clone(), dev: self.dev.clone() } + } + fn run(job: &mut Self) -> Result> { let failure = job.packet.failure.load(Ordering::Relaxed); if failure != 0 { let error = Error::from_errno(failure); @@ -153,7 +159,7 @@ impl sched::JobImpl for Job { driver::queue_g16_completion_worker(job.dev.clone()); Ok(Some(Fence::from_fence(&job.packet.completion))) } - fn timed_out(job: &mut sched::Job) -> sched::Status { + fn timed_out(job: &mut JobTimeout, _finished: bool) -> sched::Status { let mut guard = job.shared.lock(); let Some(runtime) = Option::as_mut(&mut *guard) else { job.packet.finish(Err(ENODEV)); @@ -169,7 +175,7 @@ impl sched::JobImpl for Job { runtime.fail_job(job.packet.id, ETIMEDOUT); sched::Status::NoDevice } - fn cancel(job: &mut sched::Job) { + fn cancel(job: &mut Self) { let mut guard = job.shared.lock(); if let Some(runtime) = Option::as_mut(&mut *guard) { runtime.fail_job(job.packet.id, ECANCELED); diff --git a/drivers/gpu/drm/asahi-neo/g17/buffer.rs b/drivers/gpu/drm/asahi/g17/buffer.rs similarity index 72% rename from drivers/gpu/drm/asahi-neo/g17/buffer.rs rename to drivers/gpu/drm/asahi/g17/buffer.rs index 90ab8ae21aa14f..015245f061ccae 100644 --- a/drivers/gpu/drm/asahi-neo/g17/buffer.rs +++ b/drivers/gpu/drm/asahi/g17/buffer.rs @@ -331,8 +331,15 @@ pub(crate) struct TvbGrowth { extension: KernelObject, } -/// Sparse 128-KiB data blocks at 160-KiB backing strides. Firmware owns their -/// contents; CPU zeroing occurs only on fresh backing, never on pool reuse. +/// Aliases and backing of growth extensions a shrink removed from an idle +/// heap. Field order unmaps the aliases before freeing their backing. +pub(crate) struct TvbRelease { + _mappings: KVec, + _extensions: KVec, +} + +/// Packed 128-KiB data blocks mapped at guarded 160-KiB virtual strides. +/// Firmware owns their contents; CPU zeroing occurs only on fresh backing. pub(crate) struct Tvb { addresses: KVec, // Retain sparse-block backing until the pool and its aliases retire. @@ -340,21 +347,17 @@ pub(crate) struct Tvb { extensions: KVec, } impl Tvb { + /// Blocks are packed in the backing; only their VA placement keeps the + /// stride and its trailing guard. fn bytes(blocks: usize) -> Result { - blocks - .checked_sub(1) - .and_then(|v| v.checked_mul(cfg::TVB_BLOCK_STRIDE as usize)) - .and_then(|v| v.checked_add(cfg::TVB_BLOCK_SIZE)) - .ok_or(EOVERFLOW) + blocks.checked_mul(cfg::TVB_BLOCK_SIZE).ok_or(EOVERFLOW) } fn map_block( object: &KernelObject, vm: &mmu::Vm, relative: usize, ) -> Result { - let start = relative - .checked_mul(cfg::TVB_BLOCK_STRIDE as usize) - .ok_or(EOVERFLOW)?; + let start = relative.checked_mul(cfg::TVB_BLOCK_SIZE).ok_or(EOVERFLOW)?; let source = start..start.checked_add(cfg::TVB_BLOCK_SIZE).ok_or(EOVERFLOW)?; let mapping = compact_alias( object, @@ -398,6 +401,18 @@ impl Tvb { pub(crate) fn addresses(&self) -> &[u64] { &self.addresses } + /// Blocks of each growth extension; extensions are packed like the primary. + fn extension_blocks(extension: &KernelObject) -> usize { + extension.size() / cfg::TVB_BLOCK_SIZE + } + /// Blocks of the primary backing, which a heap never gives back. + fn primary_blocks(&self) -> usize { + self.extensions + .iter() + .fold(self.blocks(), |blocks, extension| { + blocks.saturating_sub(Self::extension_blocks(extension)) + }) + } pub(crate) fn prepare_growth( &mut self, @@ -540,6 +555,44 @@ pub(crate) struct Manager { scratch: u64, buffer_id: u8, generation: u64, + shrink: Shrink, +} + +/// Native tiler-heap shrink policy, matching the heap behaviour seen +/// under macOS. Once per +/// period the page high-water folds into an average, +/// `max((sample + 4 * average) / 5, sample)`. Each period whose average +/// stays below a quarter of the capacity earns one 512 KiB block of shrink, +/// and a heap idle for ten seconds returns to its minimum. Only an idle heap +/// is rebuilt here, by whole growth extensions. +/// +/// Unlike macOS, the minimum keeps what the recent work needs: never below +/// the size at which the automatic growth check would grow the heap again +/// for the largest high-water sample of the last ten active periods. A +/// shrink then never makes the next pass at the recent load grow back. +const SHRINK_PERIOD_NS: u64 = 1_000_000_000; +const SHRINK_IDLE_NS: u64 = 10_000_000_000; +/// Blocks in one native 512 KiB parameter-buffer block. +const SHRINK_STEP_BLOCKS: usize = 0x8_0000 / cfg::TVB_BLOCK_SIZE; +/// Active periods in the recent-load window. +const SHRINK_WINDOW: usize = (SHRINK_IDLE_NS / SHRINK_PERIOD_NS) as usize; + +#[derive(Default)] +struct Shrink { + last_busy: u64, + period_start: u64, + /// High-water samples since the last period and the last growth check. + period_max: u32, + growth_max: u32, + average: u32, + credit: usize, + /// Blocks the most recent pass layout required. + required: usize, + /// High-water samples of the last active periods (with a sample or a + /// pass) and their largest value. Idle periods leave both unchanged. + window: [u32; SHRINK_WINDOW], + window_next: usize, + busy_peak: u32, } pub(crate) struct Submitted<'a> { @@ -603,6 +656,7 @@ impl Manager { scratch: args.scratch, buffer_id: args.buffer_id, generation: args.generation, + shrink: Shrink::default(), }) } pub(crate) fn buffer_id(&self) -> u8 { @@ -662,7 +716,9 @@ impl Manager { next: word.load(Ordering::Relaxed).wrapping_add(1), }) } - pub(crate) fn automatic_target(&self) -> Result> { + /// Reads the firmware page high-water and requests its reset. Both the + /// growth check and the shrink period see every sample. + fn sample_max_pages(&mut self) -> Result { let max_pages = self .graph .word(fw::CONTROL + core::mem::offset_of!(fw::Control, max_pages))?; @@ -673,8 +729,83 @@ impl Manager { let pages = max_pages.load(Ordering::Relaxed); fence(Ordering::Release); reset.store(1, Ordering::Relaxed); + self.shrink.period_max = self.shrink.period_max.max(pages); + self.shrink.growth_max = self.shrink.growth_max.max(pages); + Ok(pages) + } + pub(crate) fn automatic_target(&mut self) -> Result> { + self.sample_max_pages()?; + let pages = core::mem::take(&mut self.shrink.growth_max); Ok(automatic_growth_target(pages as usize, self.blocks())) } + /// Records the blocks a pass layout requires, the floor of any shrink. + /// A pass is being prepared at `now`; the heap counts as busy. + pub(crate) fn note_required(&mut self, blocks: usize, now: u64) { + self.shrink.required = blocks; + self.shrink.last_busy = now; + } + /// The smallest heap a shrink may leave: the primary backing, the blocks + /// of the last pass layout, and the size below which the automatic growth + /// check would grow again for the recent peak. + fn shrink_floor(&self) -> usize { + let recent = (self.shrink.busy_peak as usize) + .saturating_mul(2) + .div_ceil(cfg::TVB_PAGES_PER_BLOCK); + self.tvb.primary_blocks().max(self.shrink.required).max(recent) + } + /// Whether the heap holds growth beyond its primary backing. + pub(crate) fn grown(&self) -> bool { + !self.tvb.extensions.is_empty() + } + /// Whether the heap holds growth that a shrink could give back. + pub(crate) fn shrinkable(&self) -> bool { + self.tvb.extensions.last().is_some_and(|extension| { + self.blocks() - Tvb::extension_blocks(extension) >= self.shrink_floor() + }) + } + /// Advances the shrink policy at `now` and returns the block count an idle + /// heap should shrink to. `busy` is true while the pair has work in flight + /// or is borrowed; a busy heap only updates its average and recent peak. + /// Call it for every grown heap, so the recent peak can also fall. + pub(crate) fn shrink_target(&mut self, now: u64, busy: bool) -> Result> { + if busy || self.shrink.last_busy == 0 { + self.shrink.last_busy = now; + } + if self.shrink.period_start == 0 { + self.shrink.period_start = now; + } + if now.saturating_sub(self.shrink.period_start) >= SHRINK_PERIOD_NS { + self.sample_max_pages()?; + let sample = core::mem::take(&mut self.shrink.period_max); + let average = (u64::from(sample) + 4 * u64::from(self.shrink.average)) / 5; + self.shrink.average = (average as u32).max(sample); + if sample != 0 || self.shrink.last_busy >= self.shrink.period_start { + let next = self.shrink.window_next; + self.shrink.window[next] = sample; + self.shrink.window_next = (next + 1) % SHRINK_WINDOW; + self.shrink.busy_peak = self.shrink.window.iter().copied().max().unwrap_or(0); + } + self.shrink.period_start = now; + let capacity = self.blocks() * cfg::TVB_PAGES_PER_BLOCK; + if (self.shrink.average as usize).saturating_mul(4) < capacity { + self.shrink.credit = self.shrink.credit.saturating_add(SHRINK_STEP_BLOCKS); + } else { + self.shrink.credit = 0; + } + } + if !self.shrinkable() { + self.shrink.credit = 0; + return Ok(None); + } + if busy { + return Ok(None); + } + if now.saturating_sub(self.shrink.last_busy) >= SHRINK_IDLE_NS { + return Ok(Some(self.shrink_floor())); + } + let last = self.tvb.extensions.last().map_or(0, Tvb::extension_blocks); + Ok((last != 0 && self.shrink.credit >= last).then(|| self.blocks() - last)) + } fn cursors(&self) -> Result { Ok(Cursors { total: self.graph.word(fw::CONTROL)?.load(Ordering::Relaxed), @@ -876,6 +1007,111 @@ impl Manager { } } +impl Manager { + /// Rebuilds an idle heap with whole trailing growth extensions removed, + /// keeping at least `target` blocks. The kept blocks are laid out as + /// `Graph::init` lays out a fresh heap, starting at the firmware read + /// cursor. This runs only while every block is back in the inventory, as + /// an idle-host growth requires the complete snapshot unchanged. + /// + /// `truncate(blocks, kept, out)` moves the removed blocks' client aliases + /// out of the VM cache. Returns the reloaded PB row and the released + /// aliases and backing, which the caller frees after the device mutex. + /// None leaves the heap, the firmware records and the VM unchanged. + pub(crate) fn shrink( + &mut self, + target: usize, + descriptor: [u32; 4], + truncate: impl FnOnce(usize, usize, &mut KVec) -> Result, + ) -> Result> { + let blocks = self.blocks(); + let mut kept = blocks; + let mut dropped = 0; + for extension in self.tvb.extensions.iter().rev() { + let size = Tvb::extension_blocks(extension); + if kept - size < target.max(self.tvb.primary_blocks()) { + break; + } + kept -= size; + dropped += 1; + } + if dropped == 0 { + return Ok(None); + } + let before = self.cursors()?; + let retained = (before.total as usize + cfg::TVB_BLOCK_SLOTS - before.read as usize) + % cfg::TVB_BLOCK_SLOTS; + if !before.valid() || before.total != before.committed || retained != blocks { + return Ok(None); + } + // Allocation failure only postpones the shrink to a later pass. + let (Ok(mappings), Ok(extensions)) = ( + KVec::with_capacity(blocks - kept, GFP_NOWAIT), + KVec::with_capacity(dropped, GFP_NOWAIT), + ) else { + return Ok(None); + }; + let mut release = TvbRelease { + _mappings: mappings, + _extensions: extensions, + }; + for &address in &self.tvb.addresses[..kept] { + fw::compact_page(address)? + .checked_add(cfg::TVB_PAGES_PER_BLOCK as u32 - 1) + .ok_or(EOVERFLOW)?; + } + let base = self.graph.pointer(0, fw::SIZE)?.cast::(); + self.graph.word(0)?; + // The callback moves aliases only after checking the cached owner, + // its block count and the spare capacity. + truncate(blocks, kept, &mut release._mappings)?; + + // From here every index, address and CPU view has been checked. + let write = |offset: usize, value: u32| { + // SAFETY: The full graph and base alignment were checked above. + // Every offset is an aligned field or a checked array index of + // Graph. No mutable reference aliases firmware-owned fields. + unsafe { AtomicU32::from_ptr(base.add(offset / 4)).store(value, Ordering::Relaxed) }; + }; + for (index, &address) in self.tvb.addresses[..kept].iter().enumerate() { + // The same pure conversion succeeded for every kept block above. + let Ok(first) = fw::compact_page(address) else { + continue; + }; + let slot = (before.read as usize + index) % cfg::TVB_BLOCK_SLOTS; + let at = fw::BLOCK_TABLE + slot * size_of::(); + write(at, first); + write(at + 4, 0); + for page in 0..cfg::TVB_PAGES_PER_BLOCK { + let entry = index * cfg::TVB_PAGES_PER_BLOCK + page; + write(fw::PAGE_LIST + entry * 4, first + page as u32); + } + } + let total = ((before.read as usize + kept) % cfg::TVB_BLOCK_SLOTS) as u32; + let pages = (kept * cfg::TVB_PAGES_PER_BLOCK) as u32; + write(core::mem::offset_of!(fw::State, pages), pages); + write(core::mem::offset_of!(fw::State, producer), total); + write(core::mem::offset_of!(fw::State, consumer), before.read); + write(core::mem::offset_of!(fw::State, last_page), pages - 1); + write(fw::CONTROL, total); + fence(Ordering::SeqCst); + write(fw::CONTROL + 4, total); + fence(Ordering::SeqCst); + + self.tvb.addresses.truncate(kept); + for _ in 0..dropped { + if let Some(extension) = self.tvb.extensions.pop() { + append_reserved(&mut release._extensions, extension); + } + } + self.shrink.credit = self.shrink.credit.saturating_sub(blocks - kept); + Ok(Some(( + fw::reload_descriptor(descriptor, self.page_list_client, pages), + release, + ))) + } +} + /// Caller reserves all needed entries before any firmware-visible mutation. fn append_reserved(values: &mut KVec, value: T) { let length = values.len(); diff --git a/drivers/gpu/drm/asahi-neo/g17/buffer/target.rs b/drivers/gpu/drm/asahi/g17/buffer/target.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/buffer/target.rs rename to drivers/gpu/drm/asahi/g17/buffer/target.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/channel.rs b/drivers/gpu/drm/asahi/g17/channel.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/channel.rs rename to drivers/gpu/drm/asahi/g17/channel.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/command.rs b/drivers/gpu/drm/asahi/g17/command.rs similarity index 75% rename from drivers/gpu/drm/asahi-neo/g17/command.rs rename to drivers/gpu/drm/asahi/g17/command.rs index aa7d140d177af2..3dfa760969224e 100644 --- a/drivers/gpu/drm/asahi-neo/g17/command.rs +++ b/drivers/gpu/drm/asahi/g17/command.rs @@ -3,9 +3,10 @@ //! Decode a submission and validate its user resources before publishing any work. use super::fw::{ - compute::ScratchRequest, + compute::{IntersectionTables, ScratchRequest}, kick::WriteRange, render::{RenderPass, UscProgram, ZlsSurface}, + render_rt::RayQueryStage, }; use crate::util::{AnyBitPattern, Reader}; use core::ops::Range; @@ -14,17 +15,24 @@ use kernel::{prelude::*, uapi}; const MAX_ATTACHMENTS: usize = crate::microseq::MAX_ATTACHMENTS; const USC_WINDOW_SIZE: u64 = 1 << 32; const MAX_SAMPLERS: u32 = 1024; +const RENDER_BASE_SIZE: usize = core::mem::offset_of!(uapi::drm_asahi_cmd_render, prior_cdm) + size_of::(); +const COMPUTE_BASE_SIZE: usize = core::mem::offset_of!(uapi::drm_asahi_cmd_compute, prior_cdm) + size_of::(); +/// The timestamp-copy arrays of the common command layout, which G17P does not implement. +const RENDER_TIMESTAMP_COPIES: Range = core::mem::offset_of!(uapi::drm_asahi_cmd_render, ts_frag_end_copies) + ..core::mem::offset_of!(uapi::drm_asahi_cmd_render, prior_vdm); +const COMPUTE_TIMESTAMP_COPIES: Range = core::mem::offset_of!(uapi::drm_asahi_cmd_compute, ts_end_copies) + ..core::mem::offset_of!(uapi::drm_asahi_cmd_compute, scs_layout); const SAMPLER_SIZE: u64 = 8; const PAGE_SIZE: u64 = 0x4000; const ZLS_METADATA_STRIDE: u64 = 0x80; const MAX_DIMENSION: u16 = 16384; const MAX_LAYERS: u16 = 2048; const MAX_TILE_BUFFER: u64 = 32768; -const PROCESS_EMPTY: u32 = uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES; -const RSRC_SPEC_HI: u32 = uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_RSRC_SPEC_HI; -const DBIAS_INT: u32 = uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT; -const FEW_PRIMITIVES: u32 = uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_FEW_PRIMITIVES; -const FRAGMENT_BARRIERS: u32 = uapi::drm_asahi_neo_render_flags_DRM_ASAHI_NEO_RENDER_FRAGMENT_BARRIERS; +const PROCESS_EMPTY: u32 = uapi::drm_asahi_render_flags_DRM_ASAHI_RENDER_PROCESS_EMPTY_TILES; +const RSRC_SPEC_HI: u32 = uapi::drm_asahi_render_flags_DRM_ASAHI_RENDER_RSRC_SPEC_HI; +const DBIAS_INT: u32 = uapi::drm_asahi_render_flags_DRM_ASAHI_RENDER_DBIAS_IS_INT; +const FEW_PRIMITIVES: u32 = uapi::drm_asahi_render_flags_DRM_ASAHI_RENDER_FEW_PRIMITIVES; +const FRAGMENT_BARRIERS: u32 = uapi::drm_asahi_render_flags_DRM_ASAHI_RENDER_FRAGMENT_BARRIERS; const RENDER_FLAGS: u32 = PROCESS_EMPTY | RSRC_SPEC_HI | DBIAS_INT | FEW_PRIMITIVES | FRAGMENT_BARRIERS; @@ -46,14 +54,14 @@ impl Attachments { } fn parse(bytes: &[u8]) -> Result { - let size = size_of::(); + let size = size_of::(); if bytes.len() % size != 0 || bytes.len() / size > MAX_ATTACHMENTS { return Err(EINVAL); } let mut list = Self::EMPTY; let mut reader = Reader::new(bytes); for range in &mut list.ranges[..bytes.len() / size] { - let item: uapi::drm_asahi_neo_attachment = reader.read()?; + let item: uapi::drm_asahi_attachment = reader.read()?; if item.flags != 0 || item.pad != 0 { return Err(EINVAL); } @@ -77,8 +85,8 @@ impl Attachments { // Commands stay inline in the submission plan, avoiding an allocation per render pass. #[allow(clippy::large_enum_variant)] pub(crate) enum Payload { - Render(uapi::drm_asahi_neo_cmd_render, [Attachments; 2]), - Compute(uapi::drm_asahi_neo_cmd_compute, Attachments), + Render(uapi::drm_asahi_cmd_render, [Attachments; 2], [RayQueryStage; 2]), + Compute(uapi::drm_asahi_cmd_compute, Attachments, IntersectionTables), } pub(crate) struct Command { @@ -107,41 +115,51 @@ impl<'a> Parser<'a> { pub(crate) fn next(&mut self) -> Result> { while !self.bytes.is_empty() { let mut reader = Reader::new(self.bytes); - let header: uapi::drm_asahi_neo_cmd_header = reader.read()?; - let start = size_of::(); + let header: uapi::drm_asahi_cmd_header = reader.read()?; + let start = size_of::(); let end = start + header.size as usize; let bytes = self.bytes.get(start..end).ok_or(EINVAL)?; self.bytes = &self.bytes[end..]; let barriers = [header.vdm_barrier, header.cdm_barrier]; match header.cmd_type as u32 { - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_RENDER - | uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_COMPUTE => { + uapi::drm_asahi_cmd_type_DRM_ASAHI_CMD_RENDER + | uapi::drm_asahi_cmd_type_DRM_ASAHI_CMD_COMPUTE => { if barriers.iter().zip(self.counts).any(|(&barrier, count)| { - barrier != uapi::DRM_ASAHI_NEO_BARRIER_NONE as u16 && barrier > count + barrier != uapi::DRM_ASAHI_BARRIER_NONE as u16 && barrier > count }) || u32::from(self.counts[0]) + u32::from(self.counts[1]) >= crate::file::MAX_COMMANDS_PER_SUBMISSION { return Err(EINVAL); } let payload = if header.cmd_type as u32 - == uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_CMD_RENDER + == uapi::drm_asahi_cmd_type_DRM_ASAHI_CMD_RENDER { self.counts[0] += 1; - Payload::Render(payload(bytes)?, [self.attachments[0], self.attachments[1]]) + no_timestamp_copies(bytes, RENDER_TIMESTAMP_COPIES)?; + Payload::Render( + payload(&bytes[..bytes.len().min(RENDER_BASE_SIZE)])?, + [self.attachments[0], self.attachments[1]], + RayQueryStage::parse_pair(bytes.get(RENDER_BASE_SIZE..).unwrap_or(&[]))?, + ) } else { self.counts[1] += 1; - Payload::Compute(payload(bytes)?, self.attachments[2]) + no_timestamp_copies(bytes, COMPUTE_TIMESTAMP_COPIES)?; + Payload::Compute( + payload(&bytes[..bytes.len().min(COMPUTE_BASE_SIZE)])?, + self.attachments[2], + IntersectionTables::parse(bytes.get(COMPUTE_BASE_SIZE..).unwrap_or(&[]))?, + ) }; return Ok(Some(Command { barriers, payload })); } - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_VERTEX_ATTACHMENTS - | uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_FRAGMENT_ATTACHMENTS - | uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_COMPUTE_ATTACHMENTS => { - if barriers != [uapi::DRM_ASAHI_NEO_BARRIER_NONE as u16; 2] { + uapi::drm_asahi_cmd_type_DRM_ASAHI_SET_VERTEX_ATTACHMENTS + | uapi::drm_asahi_cmd_type_DRM_ASAHI_SET_FRAGMENT_ATTACHMENTS + | uapi::drm_asahi_cmd_type_DRM_ASAHI_SET_COMPUTE_ATTACHMENTS => { + if barriers != [uapi::DRM_ASAHI_BARRIER_NONE as u16; 2] { return Err(EINVAL); } let slot = header.cmd_type as usize - - uapi::drm_asahi_neo_cmd_type_DRM_ASAHI_NEO_SET_VERTEX_ATTACHMENTS as usize; + - uapi::drm_asahi_cmd_type_DRM_ASAHI_SET_VERTEX_ATTACHMENTS as usize; self.attachments[slot] = Attachments::parse(bytes)?; } _ => return Err(EINVAL), @@ -158,6 +176,16 @@ impl<'a> Parser<'a> { } } +/// Reject a command that asks for timestamp copies. +fn no_timestamp_copies(bytes: &[u8], copies: Range) -> Result { + let end = copies.end.min(bytes.len()); + let start = copies.start.min(end); + if bytes[start..end].iter().any(|&b| b != 0) { + return Err(EINVAL); + } + Ok(()) +} + fn payload(bytes: &[u8]) -> Result { if bytes .get(size_of::()..) @@ -227,13 +255,13 @@ fn sampler(space: &impl AddressSpace, heap: u64, count: u32) -> Result { Ok(()) } -fn helper_is_empty(helper: &uapi::drm_asahi_neo_helper_program) -> bool { +fn helper_is_empty(helper: &uapi::drm_asahi_helper_program) -> bool { helper.binary == 0 && helper.cfg == 0 && helper.data == 0 } fn surface( space: &impl AddressSpace, - zls: &uapi::drm_asahi_neo_zls_buffer, + zls: &uapi::drm_asahi_zls_buffer, layers: u16, ) -> Result { if (zls.base == 0 && (zls.comp_base != 0 || zls.stride != 0 || zls.comp_stride != 0)) @@ -279,25 +307,27 @@ fn surface( pub(crate) enum Validated { Render { pass: RenderPass, + ray_query: [RayQueryStage; 2], attachments: [Attachments; 2], - timestamps: [uapi::drm_asahi_neo_timestamps; 2], + timestamps: [uapi::drm_asahi_timestamps; 2], }, Compute { + intersection: IntersectionTables, scratch: ScratchRequest, usc_base: u64, cdm_va: u64, cdm_end_va: u64, sampler_heap_va: u64, sampler_count: u32, - timestamps: uapi::drm_asahi_neo_timestamps, + timestamps: uapi::drm_asahi_timestamps, }, } impl Payload { pub(crate) fn prior(&self) -> [u64; 2] { match self { - Self::Render(cmd, _) => [cmd.prior_vdm, cmd.prior_cdm], - Self::Compute(cmd, _) => [cmd.prior_vdm, cmd.prior_cdm], + Self::Render(cmd, _, _) => [cmd.prior_vdm, cmd.prior_cdm], + Self::Compute(cmd, _, _) => [cmd.prior_vdm, cmd.prior_cdm], } } @@ -307,7 +337,7 @@ impl Payload { space: &impl AddressSpace, ) -> Result { match self { - Self::Render(cmd, attachments) => { + Self::Render(cmd, attachments, ray_query) => { if cmd.flags & !RENDER_FLAGS != 0 || !(1..=MAX_DIMENSION).contains(&cmd.width_px) || !(1..=MAX_DIMENSION).contains(&cmd.height_px) @@ -409,13 +439,21 @@ impl Payload { | (u64::from(cmd.bg_eot_partial_rsrc_spec_hi) << 32), rsrc_spec_hi: cmd.flags & RSRC_SPEC_HI != 0, }; + for stage in ray_query { + if stage.intersection.enabled() { + require_range(space, stage.intersection.pool, stage.intersection.pool_bytes, Access::Read)?; + let code = stage.intersection.default_program & 0x0000_ffff_ffff_ffff; + if code != 0 { require_range(space, code, 64, Access::Read)?; } + } + } Ok(Validated::Render { pass, + ray_query: *ray_query, attachments: *attachments, timestamps: [cmd.ts_vtx, cmd.ts_frag], }) } - Self::Compute(cmd, attachments) => { + Self::Compute(cmd, attachments, intersection) => { if cmd.flags != 0 || cmd.cdm_ctrl_stream_base & 3 != 0 || cmd.cdm_ctrl_stream_end & 3 != 0 @@ -432,7 +470,13 @@ impl Payload { )?; sampler(space, cmd.sampler_heap, cmd.sampler_count)?; attachments.validate(space)?; + if intersection.enabled() { + require_range(space, intersection.pool, intersection.pool_bytes, Access::Read)?; + let code = intersection.default_program & 0x0000_ffff_ffff_ffff; + if code != 0 { require_range(space, code, 64, Access::Read)?; } + } Ok(Validated::Compute { + intersection: *intersection, scratch: ScratchRequest::new(cmd.scs_layout)?, usc_base: window.base, cdm_va: cmd.cdm_ctrl_stream_base, diff --git a/drivers/gpu/drm/asahi-neo/g17/completion.rs b/drivers/gpu/drm/asahi/g17/completion.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/completion.rs rename to drivers/gpu/drm/asahi/g17/completion.rs index 570da5d23af0ca..8338f8984d2d1e 100644 --- a/drivers/gpu/drm/asahi-neo/g17/completion.rs +++ b/drivers/gpu/drm/asahi/g17/completion.rs @@ -35,7 +35,7 @@ struct Destination { impl Destination { fn resolve( objects: Pin<&xarray::XArray>>, - timestamp: uapi::drm_asahi_neo_timestamp, + timestamp: uapi::drm_asahi_timestamp, ) -> Result> { if timestamp.handle == 0 { return Ok(None); @@ -72,7 +72,7 @@ pub(crate) struct Destinations(Option>>, - timestamps: &[uapi::drm_asahi_neo_timestamps], + timestamps: &[uapi::drm_asahi_timestamps], ) -> Result { if timestamps.is_empty() || timestamps.len() > 2 { return Err(EINVAL); diff --git a/drivers/gpu/drm/asahi-neo/g17/completion/publication.rs b/drivers/gpu/drm/asahi/g17/completion/publication.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/completion/publication.rs rename to drivers/gpu/drm/asahi/g17/completion/publication.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/context.rs b/drivers/gpu/drm/asahi/g17/context.rs similarity index 95% rename from drivers/gpu/drm/asahi-neo/g17/context.rs rename to drivers/gpu/drm/asahi/g17/context.rs index afd0d1d22f5bbb..6bc62a7d44beef 100644 --- a/drivers/gpu/drm/asahi-neo/g17/context.rs +++ b/drivers/gpu/drm/asahi/g17/context.rs @@ -14,7 +14,7 @@ use super::{ status::VmStatus, }; use crate::mmu::{self, ExecutionContext, Vm, UAT_PGSZ}; -use core::sync::atomic::{fence, AtomicBool, AtomicU32, Ordering}; +use core::sync::atomic::{fence, AtomicBool, AtomicU32, AtomicUsize, Ordering}; use identity::{RenderSlots, WorkStates, FREE, NODE_STRIDE, STATE_COUNT}; use kernel::{ new_mutex, @@ -120,6 +120,7 @@ pub(crate) struct Context { work: Arc>, work_base: u64, render: Arc>>>, + render_blocks: AtomicUsize, /// Fixed for the context's life. Physical queues bound to it carry this /// profile in every record, ring, doorbell and registration. policy: Policy, @@ -192,6 +193,7 @@ impl Context { work, work_base, render, + render_blocks: AtomicUsize::new(super::fw::freelist::RENDER_BLOCKS), policy, }) } @@ -277,6 +279,14 @@ impl Context { } Ok(result) } + pub(crate) fn reserve_render_blocks(&self, blocks: usize) -> Result { + if self.is_published() || self.render.lock().is_some() { return Err(EBUSY); } + self.render_blocks.store(blocks, Ordering::Release); + Ok(()) + } + pub(crate) fn render_blocks(&self) -> usize { + self.render_blocks.load(Ordering::Acquire) + } pub(crate) fn render_pool( &self, alloc: &Allocator<'_>, @@ -293,7 +303,7 @@ impl Context { return Ok(pool.clone()); } let pool = Arc::new( - RenderPool::new(alloc, ids, render_global, self.vm())?, + RenderPool::new(alloc, ids, render_global, self.vm(), self.render_blocks())?, GFP_KERNEL, )?; *render = Some(pool.clone()); diff --git a/drivers/gpu/drm/asahi-neo/g17/context/identity.rs b/drivers/gpu/drm/asahi/g17/context/identity.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/context/identity.rs rename to drivers/gpu/drm/asahi/g17/context/identity.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/dependency.rs b/drivers/gpu/drm/asahi/g17/dependency.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/dependency.rs rename to drivers/gpu/drm/asahi/g17/dependency.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/event.rs b/drivers/gpu/drm/asahi/g17/event.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/event.rs rename to drivers/gpu/drm/asahi/g17/event.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/feed.rs b/drivers/gpu/drm/asahi/g17/feed.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/feed.rs rename to drivers/gpu/drm/asahi/g17/feed.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fence.rs b/drivers/gpu/drm/asahi/g17/fence.rs similarity index 85% rename from drivers/gpu/drm/asahi-neo/g17/fence.rs rename to drivers/gpu/drm/asahi/g17/fence.rs index 9291b310369140..3414e36db9d365 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fence.rs +++ b/drivers/gpu/drm/asahi/g17/fence.rs @@ -7,7 +7,7 @@ use super::context::{Context, WorkStateLease}; use crate::file::SyncItem; -use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU32, AtomicUsize, Ordering}; +use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU32, Ordering}; use kernel::{ bindings, c_str, dma_fence::{Fence, FenceContexts, FenceObject, FenceOps, RawDmaFence, UserFence}, @@ -30,26 +30,6 @@ impl FenceOps for CompletionFence { } } -/// Status of an explicitly signalled `CompletionFence`, without taking its lock. -/// This is only for our own fences; foreign dependencies still use the DMA-fence API. -pub(crate) fn completion_status(fence: &Fence) -> i32 { - let raw = fence.raw(); - // SAFETY: the owned reference retains the naturally aligned unsigned-long - // flags. DMA-fence updates published flags with atomic bitops; usize matches - // unsigned long on the supported architectures. Acquire observes the error, - // VM status and host timestamps published before the ordered signal bit. - let flags = unsafe { AtomicUsize::from_ptr(core::ptr::addr_of_mut!((*raw).flags).cast()) } - .load(Ordering::Acquire); - if flags & (1usize << bindings::dma_fence_flag_bits_DMA_FENCE_FLAG_SIGNALED_BIT) == 0 { - return 0; - } - // SAFETY: CompletionFence has no signaled callback. Its error is set before - // signalling and is immutable afterwards. The acquired signal bit makes that - // write visible; read once, as the DMA-fence status API does under its lock. - let error = unsafe { core::ptr::read_volatile(core::ptr::addr_of!((*raw).error)) }; - if error == 0 { 1 } else { error } -} - /// Give independently retired work a timeline that fence merging cannot subsume. pub(crate) fn independent(contexts: &FenceContexts) -> Result> { let fence = contexts.new_fence(0, CompletionFence)?; diff --git a/drivers/gpu/drm/asahi-neo/g17/freelist.rs b/drivers/gpu/drm/asahi/g17/freelist.rs similarity index 94% rename from drivers/gpu/drm/asahi-neo/g17/freelist.rs rename to drivers/gpu/drm/asahi/g17/freelist.rs index c2c9470d10aa3e..c9c9ebda7f6bc0 100644 --- a/drivers/gpu/drm/asahi-neo/g17/freelist.rs +++ b/drivers/gpu/drm/asahi/g17/freelist.rs @@ -8,7 +8,7 @@ //! reference and are released only after the owner's teardown is acknowledged. use super::fw::freelist::*; -use super::fw::initdata::{FreeListArgs, FreeListClass, FreeListControl, FREE_LIST_BLOCK_SIZE}; +use super::fw::initdata::{FreeListArgs, FreeListClass, FreeListControl, FREE_LIST_BLOCK_PAGES, FREE_LIST_BLOCK_SIZE}; use super::object::{Allocator, CpuMap, KernelObject}; use crate::mmu; use core::sync::atomic::{fence, AtomicBool, AtomicU64, Ordering}; @@ -90,9 +90,15 @@ impl Pages { layout, }) } - /// Fresh zeroed compute backing, built without the device mutex. - pub(crate) fn compute(alloc: &Allocator<'_>) -> Result { - Self::new(alloc, COMPUTE_BLOCKS, RunLayout::Compute) + /// Fresh compute backing at the admitted capacity, built off-lock. + pub(crate) fn compute_sized(alloc: &Allocator<'_>, blocks: usize) -> Result { + if blocks < COMPUTE_BLOCKS || blocks > PAGE_LIST_SIZE / (FREE_LIST_BLOCK_PAGES as usize * 8) { + return Err(EINVAL); + } + Self::new(alloc, blocks, RunLayout::Compute) + } + pub(crate) fn map_client(&self, vm: &mmu::Vm) -> Result { + self.backing.map_alias(vm, self.page_list_va(), mmu::PROT_GPU_SHARED_RW) } fn page_list_va(&self) -> u64 { self.backing.gpu_va() @@ -133,8 +139,12 @@ impl RenderBacking { alloc: &Allocator<'_>, render_global: &mmu::Vm, client: &mmu::Vm, + blocks: usize, ) -> Result { - let pages = Pages::new(alloc, RENDER_BLOCKS, RunLayout::Render)?; + if blocks < RENDER_BLOCKS || blocks > PAGE_LIST_SIZE / (FREE_LIST_BLOCK_PAGES as usize * 8) { + return Err(EINVAL); + } + let pages = Pages::new(alloc, blocks, RunLayout::Render)?; let address = pages.page_list_va(); let global = pages .backing @@ -155,6 +165,7 @@ impl RenderBacking { /// consumed and the client VM is evicted; a fresh backing is installed before /// the pool is staged again. pub(crate) struct RenderPool { + blocks: usize, backing: Arc>>, /// Page-list VA of the installed backing; zero while vacant. page_list: AtomicU64, @@ -176,9 +187,10 @@ impl RenderPool { ids: &Arc, render_global: &mmu::Vm, client: &mmu::Vm, + blocks: usize, ) -> Result { let id = RenderId::new(ids)?; - let backing = RenderBacking::new(alloc, render_global, client)?; + let backing = RenderBacking::new(alloc, render_global, client, blocks)?; let address = backing.pages.page_list_va(); let run_list_va = backing.pages.run_list_va(); let state = alloc.kernel( @@ -201,7 +213,7 @@ impl RenderPool { class: FreeListClass::Two, page_list_va: address, run_list_va, - blocks: RENDER_BLOCKS as u32, + blocks: blocks as u32, state_va: state.gpu_va(), }, 0, @@ -216,6 +228,7 @@ impl RenderPool { GFP_KERNEL, )?; Ok(Self { + blocks, backing, page_list: AtomicU64::new(address), epoch: AtomicU64::new(0), @@ -273,7 +286,7 @@ impl RenderPool { if !self.vacant() { return Ok(()); } - self.rearm(RenderBacking::new(alloc, render_global, client)?) + self.rearm(RenderBacking::new(alloc, render_global, client, self.blocks)?) } pub(crate) fn serial(&self) -> u64 { @@ -297,7 +310,7 @@ impl RenderPool { } pub(crate) fn end_va(&self) -> u64 { self.run_list_va() - + (RENDER_BLOCKS * size_of::()) as u64 + + (self.blocks * size_of::()) as u64 } /// Fresh pools are already populated. Reuse requires the previous release @@ -313,7 +326,7 @@ impl RenderPool { class: FreeListClass::Two, page_list_va: self.page_list_va(), run_list_va: self.run_list_va(), - blocks: RENDER_BLOCKS as u32, + blocks: self.blocks as u32, state_va: self.state.gpu_va(), }, submitted, @@ -437,12 +450,16 @@ pub(crate) struct ComputePool { id: u16, } impl ComputePool { - pub(crate) fn new(alloc: &Allocator<'_>, qid: u8) -> Result { + pub(crate) fn fits(&self, blocks: usize) -> bool { + self.pages.as_ref().is_some_and(|pages| pages.blocks >= blocks) + } + + pub(crate) fn new(alloc: &Allocator<'_>, qid: u8, blocks: usize) -> Result { if qid > super::fw::kick::QID_MAX { return Err(EINVAL); } let id = u16::from(qid) + 1; - let pages = Pages::new(alloc, COMPUTE_BLOCKS, RunLayout::Compute)?; + let pages = Pages::compute_sized(alloc, blocks)?; let mut control = alloc.kernel( CONTROL_SIZE, mmu::UAT_PGSZ as u64, @@ -462,7 +479,7 @@ impl ComputePool { class: FreeListClass::One, page_list_va: pages.page_list_va(), run_list_va: pages.run_list_va(), - blocks: COMPUTE_BLOCKS as u32, + blocks: blocks as u32, state_va: state.gpu_va(), }), )?; @@ -517,7 +534,7 @@ impl ComputePool { class: FreeListClass::One, page_list_va: pages.page_list_va(), run_list_va: pages.run_list_va(), - blocks: COMPUTE_BLOCKS as u32, + blocks: pages.blocks as u32, state_va: self.state.gpu_va(), }); header.unk_54[..8].copy_from_slice(&bytes); diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/buffer.rs b/drivers/gpu/drm/asahi/g17/fw/buffer.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/buffer.rs rename to drivers/gpu/drm/asahi/g17/fw/buffer.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/channels.rs b/drivers/gpu/drm/asahi/g17/fw/channels.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/channels.rs rename to drivers/gpu/drm/asahi/g17/fw/channels.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/compute.rs b/drivers/gpu/drm/asahi/g17/fw/compute.rs similarity index 70% rename from drivers/gpu/drm/asahi-neo/g17/fw/compute.rs rename to drivers/gpu/drm/asahi/g17/fw/compute.rs index 5b7489cfe93f4b..0633da7f435411 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fw/compute.rs +++ b/drivers/gpu/drm/asahi/g17/fw/compute.rs @@ -23,57 +23,114 @@ pub(crate) const COMPUTE_QOS_CLASS: u8 = 0x08; /// Event mask of compute kicks. pub(crate) const COMPUTE_KICK_EVENT_MASK: [u64; 4] = [0, 0, 4, 0]; -const COMPUTE_REGISTERS: usize = 40; +const COMPUTE_REGISTERS: usize = 41; const COMPUTE_REGISTERS_MAX: usize = COMPUTE_REGISTERS + 1; /// Validated per-command shader-context scratch request. +/// +/// The layout word is Apple's packed SCS layout: depth (bits 0:15), levels +/// (16:23), cliques (24:31), user (32:47) and driver (48:63). The region the +/// hardware needs is derived with G17P's Gen1 section model: fixed prefixes +/// totalling [`Self::FIXED_BYTES`], then per-thread stack, per-thread driver +/// and per-thread user bytes. Gen1 has no per-clique level payload. The scratch +/// registers the program writes are the native `generateRegisterList` +/// formulas use `com.apple.AGXG17P`; the +/// disabled program keeps the native disabled constants. #[derive(Clone, Copy)] pub(crate) struct ScratchRequest { pages: u16, + window: u64, + stack: u64, + driver: u64, + tail: u64, } impl ScratchRequest { + /// Native ScsLayoutGen1::getSectionEnd(4). const FIXED_BYTES: u64 = 0x58000; const THREADS_PER_CLIQUE: u64 = 32; const BYTES_PER_STACK_ENTRY: u64 = 8; const PAGE_SHIFT: u32 = 12; const PAGE_SIZE: u64 = 1 << Self::PAGE_SHIFT; const PAGES_PER_MIB: u64 = 1 << (20 - Self::PAGE_SHIFT); + /// Window word when enabled: bit 24 plus the clique count at 16:23. const WINDOW_ENABLED: u64 = 1 << 24; + /// Native disabled window word, [`UNK_10791`]. + const WINDOW_DISABLED: u64 = UNK_10791; + /// Native disabled stack word, [`UNK_1A0E9`]. + const STACK_DISABLED: u64 = UNK_1A0E9; + /// Native tail reservation when enabled. + const TAIL_ENABLED: u64 = 8; /// Payload-free layouts request no scratch even if depth/cliques are set. - /// Otherwise round the fixed and per-thread sections and add a guard page. + /// Otherwise derive the Gen1 end and round it up with the native two-page + /// guard; the per-register words are the native formulas, so they are + /// checked here against Apple's own values before they reach hardware. pub(crate) fn new(layout: u64) -> Result { let depth = layout & u16::MAX as u64; let cliques = (layout >> 24) & u8::MAX as u64; let user = (layout >> 32) & u16::MAX as u64; let driver = (layout >> 48) & u16::MAX as u64; if user == 0 && driver == 0 { - return Ok(Self { pages: 0 }); + return Ok(Self { + pages: 0, + window: Self::WINDOW_DISABLED, + stack: Self::STACK_DISABLED, + driver: 0, + tail: 0, + }); } let threads = cliques.checked_mul(Self::THREADS_PER_CLIQUE).ok_or(EINVAL)?; - let per_thread = depth.checked_mul(Self::BYTES_PER_STACK_ENTRY) - .and_then(|bytes| bytes.checked_add(user)) - .and_then(|bytes| bytes.checked_add(driver)).ok_or(EINVAL)?; - let bytes = threads.checked_mul(per_thread) - .and_then(|bytes| bytes.checked_add(Self::FIXED_BYTES)) + let stack_bytes = threads + .checked_mul(depth.checked_mul(Self::BYTES_PER_STACK_ENTRY).ok_or(EINVAL)?) + .ok_or(EINVAL)?; + let driver_bytes = threads.checked_mul(driver).ok_or(EINVAL)?; + let user_bytes = threads.checked_mul(user).ok_or(EINVAL)?; + let end10 = Self::FIXED_BYTES + .checked_add(stack_bytes) + .and_then(|bytes| bytes.checked_add(driver_bytes)) + .ok_or(EINVAL)?; + let end = end10 + .checked_add(user_bytes) .and_then(|bytes| bytes.checked_add(2 * Self::PAGE_SIZE - 1)) .ok_or(EINVAL)?; - let pages = bytes >> Self::PAGE_SHIFT; - Ok(Self { pages: u16::try_from(pages).map_err(|_| EINVAL)? }) + let pages = end >> Self::PAGE_SHIFT; + Ok(Self { + pages: u16::try_from(pages).map_err(|_| EINVAL)?, + window: Self::WINDOW_ENABLED | (cliques << 16), + stack: depth & !7, + // G17P's native register list does not publish the Gen2 level + // region register. Preserve the historical zero slot. + driver: 0, + tail: Self::TAIL_ENABLED, + }) } pub(crate) fn enabled(self) -> bool { self.pages != 0 } - fn size_word(self) -> u64 { + pub(crate) fn pages(self) -> u16 { self.pages } + + pub(crate) fn size_word(self) -> u64 { let pages = u64::from(self.pages); pages | (pages.div_ceil(Self::PAGES_PER_MIB) << 16) } - fn window_word(self) -> u64 { - if self.enabled() { Self::WINDOW_ENABLED } else { UNK_10791 } + pub(crate) fn window_word(self) -> u64 { + self.window + } + + pub(crate) fn stack_word(self) -> u64 { + self.stack + } + + fn driver_word(self) -> u64 { + self.driver + } + + pub(crate) fn tail_word(self) -> u64 { + self.tail } fn checkpoints(self) -> [u16; 4] { @@ -106,10 +163,94 @@ const OPERAND_STATE_0A5C9: u64 = 0x9000; /// Checkpoints into the main register array, as entry counts: after the /// first 0x0a5c9 write, after the final 0x1a440 write (twice), and the full /// count. -const CHECKPOINTS: [u16; 4] = [38, 39, 39, COMPUTE_REGISTERS as u16]; +const CHECKPOINTS: [u16; 4] = [39, 40, 40, COMPUTE_REGISTERS as u16]; + +#[derive(Clone, Copy)] +pub(crate) struct IntersectionTables { + pub(crate) pool: u64, + pub(crate) pool_bytes: u64, + auxiliary: u32, + first: u32, + count: u32, + pub(crate) default_program: u64, + default_control: u64, +} + +impl IntersectionTables { + pub(crate) const NONE: Self = Self { + pool: 0, pool_bytes: 0, auxiliary: 0, first: 0, count: 0, + default_program: 0, default_control: 0, + }; + + pub(crate) fn new(pool: u64, pool_bytes: u64, first: u32, count: u32, + default_program: u64, default_control: u64) -> Result { + if pool == 0 { + if pool_bytes != 0 || first != 0 || count != 0 + || default_program != 0 || default_control != 0 { + return Err(EINVAL); + } + return Ok(Self::NONE); + } + // G17P HAL200 Gen1 publishes a byte-sized table length (25 bits), + // with a plain aligned address. Unlike G17X HAL300, its address has + // no bit 51 and its two pool registers use fixed native values. + // Keep the shared UAPI's 128-byte blocks and 16 KiB allocation units. + let end = pool.checked_add(pool_bytes).ok_or(EINVAL)?; + let blocks_end = u64::from(first) + u64::from(count); + let code = default_program & 0x0000_ffff_ffff_ffff; + if pool & 0x3fff != 0 || pool_bytes == 0 || pool_bytes & 0x3fff != 0 + || end > (1u64 << 48) || (pool_bytes >> 14) >= (1u64 << 25) + || count == 0 || count >= (1 << 18) || blocks_end * 128 > pool_bytes + || code & 63 != 0 + || default_control & !0x0000_7fff_0000_3fff != 0 + || (code == 0 && (default_program != 0 || default_control != 0)) + || (code != 0 && default_control & (1u64 << 40) == 0) { + return Err(EINVAL); + } + Ok(Self { pool, pool_bytes, auxiliary: 0, first, count, default_program, default_control }) + } + + pub(crate) fn parse(bytes: &[u8]) -> Result { + if bytes.get(48..).is_some_and(|tail| tail.iter().any(|&b| b != 0)) { + return Err(EINVAL); + } + let mut tail = [0u8; 48]; + let count = bytes.len().min(tail.len()); + tail[..count].copy_from_slice(&bytes[..count]); + let auxiliary = u32::from_le_bytes(tail[..4].try_into().unwrap()); + if tail[4..8].iter().any(|&b| b != 0) + || (auxiliary != 0 && !(8..=32).contains(&auxiliary)) { + return Err(EINVAL); + } + let mut tables = Self::new( + u64::from_le_bytes(tail[8..16].try_into().unwrap()), + u64::from_le_bytes(tail[16..24].try_into().unwrap()), + u32::from_le_bytes(tail[24..28].try_into().unwrap()), + u32::from_le_bytes(tail[28..32].try_into().unwrap()), + u64::from_le_bytes(tail[32..40].try_into().unwrap()), + u64::from_le_bytes(tail[40..48].try_into().unwrap()), + )?; + if auxiliary != 0 && !tables.enabled() { return Err(EINVAL); } + tables.auxiliary = auxiliary; + Ok(tables) + } + + pub(crate) fn enabled(self) -> bool { self.pool != 0 } + + pub(crate) fn words(self) -> [u64; 6] { + if !self.enabled() { return [0; 6]; } + // AGXMetalG17P 26A428 endComputePass, both command encodings; + // pinned AGXG17P processComputeSetup copies these six words as-is. + [self.pool + u64::from(self.first) * 128, + 0xc0000000 | (u64::from(self.count) << 7), 0x6f00000000, + 0x3feff, self.default_program, self.default_control] + } +} + /// Inputs of one compute descriptor. pub(crate) struct ComputeArgs { + pub(crate) intersection: IntersectionTables, /// Scratch request validated with the userspace command. pub(crate) scratch: ScratchRequest, /// Base of the queue's USC window. @@ -166,6 +307,7 @@ fn compute_registers(args: &ComputeArgs) -> Result<[RegisterWrite; COMPUTE_REGIS let operand = args.operand_state_va; let key = work_key(args.context_id, args.work_state); let ids = OBJECT_IDS.word(); + let intersection = args.intersection.words(); Ok([ RegisterWrite::new(0x017d9, 1), @@ -183,15 +325,20 @@ fn compute_registers(args: &ComputeArgs) -> Result<[RegisterWrite; COMPUTE_REGIS RegisterWrite::new(0x1a440, UNK_1A440), RegisterWrite::new(0x1a458, UNK_1A458), RegisterWrite::new(0x101d9, UNK_101D9), - RegisterWrite::new(0x1a089, 0), - RegisterWrite::new(0x1a091, 0), - RegisterWrite::new(0x1a059, 0), - RegisterWrite::new(0x1a061, 0), - RegisterWrite::new(0x1a0b9, 0), - RegisterWrite::new(0x1a0c1, 0), + RegisterWrite::new(0x1a089, intersection[0]), + RegisterWrite::new(0x1a091, intersection[1]), + RegisterWrite::new(0x1a059, intersection[2]), + RegisterWrite::new(0x1a061, intersection[3]), + RegisterWrite::new(0x1a0b9, intersection[4]), + RegisterWrite::new(0x1a0c1, intersection[5]), RegisterWrite::new(0x101d1, args.scratch.size_word()), - RegisterWrite::new(0x0d479, 0), - RegisterWrite::new(0x1a0e9, UNK_1A0E9), + RegisterWrite::new(0x0d479, if args.intersection.auxiliary == 0 { + args.scratch.tail_word() + } else { + u64::from(args.intersection.auxiliary) + }), + RegisterWrite::new(0x1a0e9, args.scratch.stack_word()), + RegisterWrite::new(0x1a141, args.scratch.driver_word()), RegisterWrite::new(0x107a1, args.scratch.window_word()), RegisterWrite::new(0x0a599, UNK_0A599), RegisterWrite::new(0x0d411, UNK_0D411), diff --git a/drivers/gpu/drm/asahi/g17/fw/compute_pool.rs b/drivers/gpu/drm/asahi/g17/fw/compute_pool.rs new file mode 100644 index 00000000000000..0f0614f6155a17 --- /dev/null +++ b/drivers/gpu/drm/asahi/g17/fw/compute_pool.rs @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +//! G17P compute pool capacity for a validated Gen1 SCS request. + +use kernel::prelude::*; +use super::initdata::{FREE_LIST_BLOCK_PAGES, FREE_LIST_BLOCK_SIZE}; +use super::freelist::{COMPUTE_BLOCKS, PAGE_LIST_SIZE}; + +pub(crate) fn extra_bytes(active_cores: u32, scs_pages: u16) -> Result { + if active_cores == 0 { return Err(EINVAL); } + if scs_pages == 0 { return Ok(0); } + // Register0x101d1 counts 4-KiB pages and their 1-MiB large-page inventory. + // Register0x0d479 reserves eight further pages for each enabled USC. + let small = u64::from(scs_pages); + let per_core = small.checked_add(small.div_ceil(256)) + .and_then(|pages| pages.checked_add(8)).ok_or(EOVERFLOW)?; + let extra = per_core.checked_mul(u64::from(active_cores)) + .and_then(|pages| pages.checked_mul(4096)).ok_or(EOVERFLOW)?; + Ok(extra) +} + +pub(crate) fn required_blocks(active_cores: u32, scs_pages: u16) -> Result { + let extra = extra_bytes(active_cores, scs_pages)?; + let bytes = extra.checked_add((COMPUTE_BLOCKS * FREE_LIST_BLOCK_SIZE) as u64) + .ok_or(EOVERFLOW)?; + let blocks = usize::try_from(bytes.div_ceil(FREE_LIST_BLOCK_SIZE as u64)) + .map_err(|_| EOVERFLOW)?; + let maximum = PAGE_LIST_SIZE / (FREE_LIST_BLOCK_PAGES as usize * 8); + if blocks > maximum { return Err(E2BIG); } + Ok(blocks) +} diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/context.rs b/drivers/gpu/drm/asahi/g17/fw/context.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/context.rs rename to drivers/gpu/drm/asahi/g17/fw/context.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/freelist.rs b/drivers/gpu/drm/asahi/g17/fw/freelist.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/freelist.rs rename to drivers/gpu/drm/asahi/g17/fw/freelist.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/initdata.rs b/drivers/gpu/drm/asahi/g17/fw/initdata.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/fw/initdata.rs rename to drivers/gpu/drm/asahi/g17/fw/initdata.rs index 94b03ebe284dba..1f25bd0fc1c0dc 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fw/initdata.rs +++ b/drivers/gpu/drm/asahi/g17/fw/initdata.rs @@ -2239,7 +2239,11 @@ static_assert!(size_of::() == 0x20); impl PagePoolDescriptor { /// Complete row for a physical compute queue's independent 21-block pool. pub(crate) const fn compute(page_list_va: u64) -> Self { - let pages = 21 * FREE_LIST_BLOCK_PAGES as u64; + Self::compute_blocks(page_list_va, 21) + } + + pub(crate) const fn compute_blocks(page_list_va: u64, blocks: usize) -> Self { + let pages = blocks as u64 * FREE_LIST_BLOCK_PAGES as u64; Self::new(page_list_va, pages << PAGE_POOL_FREE_PAGES_SHIFT, pages) } diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/kick.rs b/drivers/gpu/drm/asahi/g17/fw/kick.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/fw/kick.rs rename to drivers/gpu/drm/asahi/g17/fw/kick.rs index 47cdce3ca09e5c..e07fbfb485a10a 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fw/kick.rs +++ b/drivers/gpu/drm/asahi/g17/fw/kick.rs @@ -370,7 +370,7 @@ pub(crate) struct KickArgs<'a> { /// Register arrays bound to the kick. pub(crate) register_arrays: [Option; 4], /// The compute register program includes a shader-context scratch request. - pub(crate) compute_scratch: bool, + pub(crate) scs_enabled: bool, /// Priority class of the queue. pub(crate) priority: u8, } @@ -423,7 +423,7 @@ impl KickEntry { const QID_SHIFT: u32 = 42; const MCACHE_PRESENT: u64 = 1 << 58; const PRIORITY_SHIFT: u32 = 59; - const COMPUTE_SCRATCH: u64 = 1 << 61; + const SCS_ENABLED: u64 = 1 << 61; const QOS_SLOT_SHIFT: u32 = 32; const QOS_CLASS_SHIFT: u32 = 40; const QOS_CLASS_MAX: u8 = 0x1f; @@ -446,8 +446,8 @@ impl KickEntry { let mut header = args.timestamp.get() << Self::TIMESTAMP_SHIFT | (args.qid as u64) << Self::QID_SHIFT | (args.priority as u64) << Self::PRIORITY_SHIFT; - if args.compute_scratch { - header |= Self::COMPUTE_SCRATCH; + if args.scs_enabled { + header |= Self::SCS_ENABLED; } let mcache = match args.mcache { Some(table) => { diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/mod.rs b/drivers/gpu/drm/asahi/g17/fw/mod.rs similarity index 98% rename from drivers/gpu/drm/asahi-neo/g17/fw/mod.rs rename to drivers/gpu/drm/asahi/g17/fw/mod.rs index 4f0ae8aa253ed0..b6bb6aaea860ff 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fw/mod.rs +++ b/drivers/gpu/drm/asahi/g17/fw/mod.rs @@ -14,12 +14,14 @@ use kernel::prelude::*; pub(crate) mod buffer; pub(crate) mod channels; pub(crate) mod compute; +pub(crate) mod compute_pool; pub(crate) mod context; pub(crate) mod freelist; pub(crate) mod initdata; pub(crate) mod kick; pub(crate) mod queue; pub(crate) mod render; +pub(crate) mod render_rt; pub(crate) mod qos; /// Clears a firmware structure in place. diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/qos.rs b/drivers/gpu/drm/asahi/g17/fw/qos.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/qos.rs rename to drivers/gpu/drm/asahi/g17/fw/qos.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/queue.rs b/drivers/gpu/drm/asahi/g17/fw/queue.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/queue.rs rename to drivers/gpu/drm/asahi/g17/fw/queue.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/render.rs b/drivers/gpu/drm/asahi/g17/fw/render.rs similarity index 94% rename from drivers/gpu/drm/asahi-neo/g17/fw/render.rs rename to drivers/gpu/drm/asahi/g17/fw/render.rs index 23eaa7e7dfba9e..85a2385b421b81 100644 --- a/drivers/gpu/drm/asahi-neo/g17/fw/render.rs +++ b/drivers/gpu/drm/asahi/g17/fw/render.rs @@ -23,7 +23,7 @@ use super::kick::{ use super::queue::CommandTag; use super::{ clear, offset_va, sampler_max, stamp_value, timestamp_end_va, work_key, ObjectIds, UNK_0A599, - UNK_0D411, UNK_101D9, UNK_10791, UNK_1A0E9, UNK_1C8F8, + UNK_0D411, UNK_101D9, UNK_1C8F8, }; /// Size of a tiling descriptor. @@ -54,7 +54,11 @@ const DEFLAKE_20: u64 = 0x20; const DEFLAKE_2A0: u64 = 0x2a0; const TA_REGISTERS: usize = 73; +const TA_REGISTERS_MAX: usize = TA_REGISTERS + 1; +const TA_SCS_ENABLE_INDEX: usize = 1; const FRAGMENT_REGISTERS: usize = 89; +const FRAGMENT_REGISTERS_MAX: usize = FRAGMENT_REGISTERS + 1; +const FRAGMENT_SCS_ENABLE_INDEX: usize = 1; const PARTIAL_STORE_REGISTERS: usize = 16; const PARTIAL_RESUME_REGISTERS: usize = 23; const PARTIAL_LOAD_REGISTERS: usize = 10; @@ -122,8 +126,6 @@ const UNK_15359_PARTIAL: u64 = 0x20; /// Required value of register 0x15049 and of fragment descriptor field /// +0x1f50. const UNK_15049: u64 = 0x10_0000; -/// Value of the fragment register 0x10791. -const UNK_10791_FRAGMENT: u64 = UNK_10791 | 1 << 9; /// USC program offsets carry tag bits that are not part of the address. const USC_OFFSET_TAG_MASK: u32 = 7; @@ -303,6 +305,7 @@ pub(crate) struct RenderStage { pub(crate) struct RenderArgs<'a> { /// Render-pass state. pub(crate) pass: &'a RenderPass, + pub(crate) ray_query: [super::render_rt::RayQueryStage; 2], /// Kernel-owned objects. pub(crate) resources: &'a RenderResources, /// Submission ordinal of the render queue pair, from zero. @@ -481,7 +484,7 @@ fn ta_registers( args: &RenderArgs<'_>, g: &Geometry, w: &PassWords, -) -> Result<[RegisterWrite; TA_REGISTERS]> { +) -> Result<[RegisterWrite; TA_REGISTERS_MAX]> { let p = args.pass; let r = args.resources; let tilemap = user_offset(r.tilemap_va)?; @@ -497,9 +500,12 @@ fn ta_registers( } let key = work_key(r.context_id, args.ta.state_word); let ids = r.ta_ids.word(); + let rt = args.ray_query[0]; + let ift = rt.intersection.words(); Ok([ RegisterWrite::new(0x01748, 1), + RegisterWrite::new(0x017d9, 1), RegisterWrite::new(0x10141, 0x200), RegisterWrite::new(0x1c039, tilemap), RegisterWrite::new(0x1c9c8, tilemap), @@ -545,16 +551,16 @@ fn ta_registers( RegisterWrite::new(0x1c898, 0), RegisterWrite::new(0x101e1, UNK_101D9), RegisterWrite::new(0x1c9e8, w.layer_mode & LAYER_MODE_PIPE_MASK), - RegisterWrite::new(0x1a099, 0), - RegisterWrite::new(0x1a0a1, 0), - RegisterWrite::new(0x1a069, 0), - RegisterWrite::new(0x1a071, 0), - RegisterWrite::new(0x1a0c9, 0), - RegisterWrite::new(0x1a0d1, 0), - RegisterWrite::new(0x101c9, 0), - RegisterWrite::new(0x0d471, 0), - RegisterWrite::new(0x1a0f1, UNK_1A0E9), - RegisterWrite::new(0x10799, UNK_10791), + RegisterWrite::new(0x1a099, ift[0]), + RegisterWrite::new(0x1a0a1, ift[1]), + RegisterWrite::new(0x1a069, ift[2]), + RegisterWrite::new(0x1a071, ift[3]), + RegisterWrite::new(0x1a0c9, ift[4]), + RegisterWrite::new(0x1a0d1, ift[5]), + RegisterWrite::new(0x101c9, rt.scratch.size_word()), + RegisterWrite::new(0x0d471, rt.scratch.tail_word()), + RegisterWrite::new(0x1a0f1, rt.scratch.stack_word()), + RegisterWrite::new(0x10799, rt.scratch.window_word()), RegisterWrite::new(0x1c830, r.pb_slot as u64), RegisterWrite::new(0x1ca30, r.pm_scratch), RegisterWrite::new(0x16c39, r.pm_scratch), @@ -579,7 +585,7 @@ fn fragment_registers( args: &RenderArgs<'_>, g: &Geometry, w: &PassWords, -) -> [RegisterWrite; FRAGMENT_REGISTERS] { +) -> [RegisterWrite; FRAGMENT_REGISTERS_MAX] { let p = args.pass; let r = args.resources; let mut tile_state = TILE_STATE_BASE @@ -591,9 +597,12 @@ fn fragment_registers( } let key = work_key(r.context_id, args.fragment.state_word); let ids = r.fragment_ids.word(); + let rt = args.ray_query[1]; + let ift = rt.intersection.words(); [ RegisterWrite::new(0x01739, 1), + RegisterWrite::new(0x017d9, 1), RegisterWrite::new(0x10009, w.utile_config), // End-of-tile resource specifier. RegisterWrite::new(0x15379, w.eot_rsrc_spec), @@ -611,7 +620,7 @@ fn fragment_registers( RegisterWrite::new(0x16030, g.macro_size), RegisterWrite::new(0x100d9, g.screen), RegisterWrite::new(0x0a301, 0), - RegisterWrite::new(0x10791, UNK_10791_FRAGMENT), + RegisterWrite::new(0x10791, rt.scratch.window_word() | 1 << 9), RegisterWrite::new(0x16098, r.heapmeta_va), RegisterWrite::new(0x15109, p.scissor_va), RegisterWrite::new(0x15101, p.depth_bias_va), @@ -659,15 +668,15 @@ fn fragment_registers( RegisterWrite::new(0x101e9, UNK_101D9), RegisterWrite::new(0x160a8, 0), RegisterWrite::new(0x16068, tile_state), - RegisterWrite::new(0x1a0a9, 0), - RegisterWrite::new(0x1a0b1, 0), - RegisterWrite::new(0x1a079, 0), - RegisterWrite::new(0x1a081, 0), - RegisterWrite::new(0x1a0d9, 0), - RegisterWrite::new(0x1a0e1, 0), - RegisterWrite::new(0x101c1, 0), - RegisterWrite::new(0x0d469, 0), - RegisterWrite::new(0x1a0f9, UNK_1A0E9), + RegisterWrite::new(0x1a0a9, ift[0]), + RegisterWrite::new(0x1a0b1, ift[1]), + RegisterWrite::new(0x1a079, ift[2]), + RegisterWrite::new(0x1a081, ift[3]), + RegisterWrite::new(0x1a0d9, ift[4]), + RegisterWrite::new(0x1a0e1, ift[5]), + RegisterWrite::new(0x101c1, rt.scratch.size_word()), + RegisterWrite::new(0x0d469, rt.scratch.tail_word()), + RegisterWrite::new(0x1a0f9, rt.scratch.stack_word()), RegisterWrite::new(0x0a5a9, UNK_0A599), RegisterWrite::new(0x0d429, UNK_0D411), RegisterWrite::new(0x160e0, ids), @@ -907,9 +916,16 @@ impl TaDescriptor { self.pb_va = resources.pb_va; self.scene_va = resources.scene_va; self.scene_end_va = resources.scene_end_va; - self.checkpoints = TA_CHECKPOINTS; + let scs_register = args.ray_query[0].scratch.enabled(); + self.checkpoints = TA_CHECKPOINTS.map(|count| count + u16::from(scs_register)); self.unk_48 = (args.ordinal as u32).wrapping_add((args.ordinal / 2) as u32); - self.registers.set(array_va, ®isters)?; + if scs_register { + self.registers.set(array_va, ®isters)?; + } else { + let mut trimmed = [registers[0]; TA_REGISTERS]; + trimmed[TA_SCS_ENABLE_INDEX..].copy_from_slice(®isters[TA_SCS_ENABLE_INDEX + 1..]); + self.registers.set(array_va, &trimmed)?; + } self.tpc_va = resources.tpc_va; self.tpc_size = resources.tpc_size; self.fragment_qid = args.fragment.qid.into(); @@ -963,10 +979,11 @@ impl TaDescriptor { /// descriptor's address as seen by the pass's VM. pub(crate) fn register_bindings( descriptor_va: u64, + scs_register: bool, ) -> Result<[Option; 4]> { let array = RegisterArrayBinding::new( offset_va(descriptor_va, Self::ARRAY_OFFSET)?, - TA_CHECKPOINTS[0] as u8, + TA_CHECKPOINTS[0] as u8 + u8::from(scs_register), )?; Ok([Some(array), None, None, None]) } @@ -1153,13 +1170,14 @@ impl FragmentDescriptor { /// the descriptor's address as seen by the pass's VM. pub(crate) fn register_bindings( descriptor_va: u64, + scs_register: bool, ) -> Result<[Option; 4]> { let array = |index, checkpoint: usize| -> Result> { let va = offset_va(descriptor_va, Self::array_offset(index))?; Ok(Some(RegisterArrayBinding::new(va, checkpoint as u8)?)) }; Ok([ - array(0, FRAGMENT_CHECKPOINTS[0] as usize)?, + array(0, FRAGMENT_CHECKPOINTS[0] as usize + usize::from(scs_register))?, array(1, PARTIAL_STORE_REGISTERS)?, array(2, PARTIAL_RESUME_REGISTERS)?, array(3, PARTIAL_LOAD_REGISTERS)?, @@ -1226,9 +1244,19 @@ impl FragmentSlot { desc.merge_upper_x = pass.merge_upper_x; desc.merge_upper_y = pass.merge_upper_y; desc.tiles = geometry.tiles_x * geometry.tiles_y; - desc.checkpoints = FRAGMENT_CHECKPOINTS; - desc.entries = FRAGMENT_REGISTERS as u16; - desc.arrays[0].set(arrays[0], &fragment_registers(args, &geometry, &words))?; + let scs_register = args.ray_query[1].scratch.enabled(); + let extra = u16::from(scs_register); + desc.checkpoints = FRAGMENT_CHECKPOINTS.map(|count| count + extra); + desc.entries = FRAGMENT_REGISTERS as u16 + extra; + let registers = fragment_registers(args, &geometry, &words); + if scs_register { + desc.arrays[0].set(arrays[0], ®isters)?; + } else { + let mut trimmed = [registers[0]; FRAGMENT_REGISTERS]; + trimmed[FRAGMENT_SCS_ENABLE_INDEX..] + .copy_from_slice(®isters[FRAGMENT_SCS_ENABLE_INDEX + 1..]); + desc.arrays[0].set(arrays[0], &trimmed)?; + } desc.arrays[1].set(arrays[1], &partial_store_registers(pass, &words))?; desc.arrays[2].set(arrays[2], &partial_resume_registers(pass, &words))?; desc.arrays[3].set(arrays[3], &partial_load_registers(pass, &words))?; diff --git a/drivers/gpu/drm/asahi-neo/g17/fw/render/storage.rs b/drivers/gpu/drm/asahi/g17/fw/render/storage.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/fw/render/storage.rs rename to drivers/gpu/drm/asahi/g17/fw/render/storage.rs diff --git a/drivers/gpu/drm/asahi/g17/fw/render_rt.rs b/drivers/gpu/drm/asahi/g17/fw/render_rt.rs new file mode 100644 index 00000000000000..5ae23d84400eb0 --- /dev/null +++ b/drivers/gpu/drm/asahi/g17/fw/render_rt.rs @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT + +use super::{compute::{IntersectionTables, ScratchRequest}, compute_pool, + freelist::{PAGE_LIST_SIZE, RENDER_BLOCKS}, + initdata::{FREE_LIST_BLOCK_PAGES, FREE_LIST_BLOCK_SIZE}}; +use kernel::prelude::*; + +#[derive(Clone, Copy)] +pub(crate) struct RayQueryStage { + pub(crate) intersection: IntersectionTables, + pub(crate) scratch: ScratchRequest, +} + +impl RayQueryStage { + fn parse(bytes: &[u8]) -> Result { + let mut wire = [0u8; 56]; + wire[..bytes.len()].copy_from_slice(bytes); + let layout = u64::from_le_bytes(wire[..8].try_into().unwrap()); + let intersection = IntersectionTables::parse(&wire[8..])?; + let scratch = ScratchRequest::new(layout)?; + if scratch.enabled() != intersection.enabled() { return Err(EINVAL); } + Ok(Self { intersection, scratch }) + } + + pub(crate) fn parse_pair(bytes: &[u8]) -> Result<[Self; 2]> { + if bytes.get(112..).is_some_and(|tail| tail.iter().any(|&b| b != 0)) { + return Err(EINVAL); + } + let mut wire = [0u8; 112]; + let count = bytes.len().min(wire.len()); + wire[..count].copy_from_slice(&bytes[..count]); + Ok([Self::parse(&wire[..56])?, Self::parse(&wire[56..])?]) + } +} + +pub(crate) fn required_blocks(active_cores: u32, pages: [u16; 2]) -> Result { + let extra = compute_pool::extra_bytes(active_cores, pages[0])? + .checked_add(compute_pool::extra_bytes(active_cores, pages[1])?) + .and_then(|bytes| bytes.checked_mul(crate::hw::t8140::queues::RENDER_PAIRS_PER_OWNER as u64)) + .ok_or(EOVERFLOW)?; + let bytes = extra.checked_add((RENDER_BLOCKS * FREE_LIST_BLOCK_SIZE) as u64) + .ok_or(EOVERFLOW)?; + let blocks = usize::try_from(bytes.div_ceil(FREE_LIST_BLOCK_SIZE as u64)) + .map_err(|_| EOVERFLOW)?; + let maximum = PAGE_LIST_SIZE / (FREE_LIST_BLOCK_PAGES as usize * 8); + if blocks > maximum { return Err(E2BIG); } + Ok(blocks) +} diff --git a/drivers/gpu/drm/asahi-neo/g17/hazard.rs b/drivers/gpu/drm/asahi/g17/hazard.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/hazard.rs rename to drivers/gpu/drm/asahi/g17/hazard.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/ids.rs b/drivers/gpu/drm/asahi/g17/ids.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/ids.rs rename to drivers/gpu/drm/asahi/g17/ids.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/initdata.rs b/drivers/gpu/drm/asahi/g17/initdata.rs similarity index 98% rename from drivers/gpu/drm/asahi-neo/g17/initdata.rs rename to drivers/gpu/drm/asahi/g17/initdata.rs index 5993be786874e7..4df78cca06d461 100644 --- a/drivers/gpu/drm/asahi-neo/g17/initdata.rs +++ b/drivers/gpu/drm/asahi/g17/initdata.rs @@ -23,7 +23,6 @@ const CONTEXT_SIZE: usize = 0x20000; const REGION_A_SIZE: usize = core::mem::size_of::(); const REGION_C_SIZE: usize = 0x1000; -const COMPUTE_BLOCKS_ALLOCATED: usize = 23; const COMPUTE_BLOCKS_MAPPED: usize = 22; const COMPUTE_BLOCKS_POPULATED: usize = 21; const COMPUTE_CONTROL_BLOCKS: u32 = 25; @@ -65,8 +64,8 @@ impl ComputeShared { KernelObject::backing(alloc.dev, low::RENDER_PAGE_LIST_SIZE, CpuMap::WriteCombined)?; let runs = KernelObject::backing(alloc.dev, low::RENDER_RUN_LIST_SIZE, CpuMap::WriteCombined)?; - let mut blocks = KVec::with_capacity(COMPUTE_BLOCKS_ALLOCATED, GFP_KERNEL)?; - for _ in 0..COMPUTE_BLOCKS_ALLOCATED { + let mut blocks = KVec::with_capacity(COMPUTE_BLOCKS_MAPPED, GFP_KERNEL)?; + for _ in 0..COMPUTE_BLOCKS_MAPPED { blocks.push( KernelObject::backing(alloc.dev, abi::FREE_LIST_BLOCK_SIZE, CpuMap::WriteCombined)?, GFP_KERNEL, @@ -286,11 +285,12 @@ impl InitData { /// Initialize a fresh physical compute QID's row before exposing its queue. An installed /// or rebound queue keeps the row's firmware-owned counters and must never call this again. - pub(super) fn initialize_compute_pool(&self, slot: u16, page_list_va: u64) -> Result { + /// Called for a fresh row or after its free-list release was consumed. + pub(super) fn initialize_compute_pool_blocks(&self, slot: u16, page_list_va: u64, blocks: usize) -> Result { if slot == 0 || slot > u16::from(super::fw::kick::QID_MAX) + 1 { return Err(EINVAL); } - let entry = abi::PagePoolDescriptor::compute(page_list_va); + let entry = abi::PagePoolDescriptor::compute_blocks(page_list_va, blocks); let size = core::mem::size_of::(); let offset = usize::from(slot) * size; let object = self.object(self.page_pool)?; @@ -300,8 +300,8 @@ impl InitData { object.dword(offset + core::mem::offset_of!(abi::PagePoolDescriptor, counters))?; let count = object.dword(offset + core::mem::offset_of!(abi::PagePoolDescriptor, page_count))?; - // SAFETY: the whole fresh row was bounds checked, the QID has never been published, - // and the device mutex excludes another host publisher. No other row is touched. + // SAFETY: the whole fresh row was bounds checked, this is a fresh QID or its release + // was consumed, and the device mutex excludes another host publisher. No other row is touched. unsafe { pointer.write_bytes(0, size); } @@ -867,9 +867,6 @@ impl InitData { let compute_state = self.add(KernelObject::backing(env.dev, mmu::UAT_PGSZ, WB)?)?; self.object_mut(compute_state)? .write(0, abi::FreeListState::COMPUTE)?; - for _ in &low::COMPUTE_ZERO_BLOCKS { - self.add(KernelObject::backing(env.dev, abi::FREE_LIST_BLOCK_SIZE, WB)?)?; - } let compute_runs = self.add(KernelObject::new( env.dev, env.lower, diff --git a/drivers/gpu/drm/asahi-neo/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs similarity index 95% rename from drivers/gpu/drm/asahi-neo/g17/job.rs rename to drivers/gpu/drm/asahi/g17/job.rs index 9e897ea0c59400..07db5d32e92c1e 100644 --- a/drivers/gpu/drm/asahi-neo/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -8,7 +8,7 @@ use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU8, Ordering}; use kernel::{ bindings, c_str, dma_fence::{Fence, RawDmaFence}, - drm_neo::sched, + drm::sched, prelude::*, sync::Arc, }; @@ -185,6 +185,11 @@ impl Packet { pub(crate) fn is_published(&self) -> bool { self.publication.load(Ordering::Acquire) == 2 } + /// A published group retired by firmware without executing is published + /// again from the same packet. Publication state returns to "begun". + pub(crate) fn reset_publication_for_replay(&self) { + self.publication.store(1, Ordering::Release); + } pub(crate) fn cancel_requested(&self) -> bool { self.cancelled.load(Ordering::Acquire) } @@ -238,7 +243,11 @@ pub(crate) trait Backend: Send + Sync { } /// Lazily create or rebind this logical queue's physical compute graph. - fn ensure_compute(&self) -> Result; + fn ensure_compute(&self, scs_pages: u16) -> Result; + fn render_scs_blocks(&self, pages: [u16; 2]) -> Result { + if pages != [0; 2] { return Err(EINVAL); } + Ok(super::fw::freelist::RENDER_BLOCKS) + } /// Settle outstanding work of a terminal VM before an ioctl error is published. fn fail_vm(&self, error: Error); /// Called after both scheduler entities have cancelled their queued jobs. @@ -279,6 +288,7 @@ impl Drop for Job { impl sched::JobImpl for Job { type TimeoutData = Timeout; const MODULE: Option<&'static kernel::ThisModule> = Some(&crate::THIS_MODULE); + const SYNCHRONIZED_CANCEL: bool = true; fn timeout_data(&self) -> Timeout { Timeout { diff --git a/drivers/gpu/drm/asahi-neo/g17/kick.rs b/drivers/gpu/drm/asahi/g17/kick.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/kick.rs rename to drivers/gpu/drm/asahi/g17/kick.rs index f109c8261476e3..3b78b9443a86de 100644 --- a/drivers/gpu/drm/asahi-neo/g17/kick.rs +++ b/drivers/gpu/drm/asahi/g17/kick.rs @@ -308,7 +308,7 @@ impl Queue { if self.id.engine != DataMaster::Compute { return Err(EINVAL); } - fence(Ordering::SeqCst); + mem::sync(); for asid in 0..4 { mem::tlbi_range_or_asid(asid, self.low_va() as usize, KICK_RING_SIZE); mem::tlbi_range_or_asid(asid, self.firmware_va() as usize, KICK_RING_SIZE); diff --git a/drivers/gpu/drm/asahi-neo/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs similarity index 97% rename from drivers/gpu/drm/asahi-neo/g17/mod.rs rename to drivers/gpu/drm/asahi/g17/mod.rs index e2f25bf96ffc68..0ffa568d6d4b3d 100644 --- a/drivers/gpu/drm/asahi-neo/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -56,7 +56,7 @@ use core::sync::atomic::{ use kernel::{ c_str, device::{self, Core}, - drm_neo::gem::shmem, + drm::gem::shmem, impl_has_delayed_work, impl_has_work, iosys_map::IoSysMapRef, new_condvar, new_delayed_work, new_mutex, new_work, platform, @@ -601,8 +601,9 @@ struct Shared { /// Allocation users take independent references before reserving resources. /// Shutdown removes this owning backedge after joining device work. #[pin] - drm_neo: Mutex>, + drm: Mutex>, clusters: u32, + active_cores: u32, descriptor_flags: [u32; 2], crashed: AtomicBool, feed: Arc, @@ -709,14 +710,14 @@ impl WorkItem for Shared { } impl Shared { - fn drm_neo(&self) -> Result { - self.drm_neo.lock().as_ref().cloned().ok_or(ENODEV) + fn drm(&self) -> Result { + self.drm.lock().as_ref().cloned().ok_or(ENODEV) } fn release_drm(&self) { // The final put may call device-release callbacks. Never run those with // the reference-slot mutex held. - let retired = self.drm_neo.lock().take(); + let retired = self.drm.lock().take(); drop(retired); } @@ -880,11 +881,13 @@ impl Gpu { let preparations = preparation::Gate::new()?; let reclaim = runtime::teardown::ReclaimBatch::new()?; let feed = feed::Feed::new()?; + let active_cores = id.core_masks.iter().map(|mask| mask.count_ones()).sum(); let shared = Arc::pin_init( pin_init!(Shared { - drm_neo <- new_mutex!(Some(dev.into()), "g17::Shared::drm_neo"), + drm <- new_mutex!(Some(dev.into()), "g17::Shared::drm"), dev: dev.as_ref().into(), clusters: id.num_clusters, + active_cores, descriptor_flags: [id.perf_control, id.perf_map[0]], crashed: AtomicBool::new(false), feed, @@ -1094,12 +1097,13 @@ impl gpu::Gpu for Gpu { self.shared.crashed.load(Ordering::Acquire) } - fn get_params(&self, params: &mut uapi::drm_asahi_neo_params_global) -> Result { + fn get_params(&self, params: &mut uapi::drm_asahi_params_global) -> Result { params.features |= - uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_PER_COMMAND_INPUT_SYNCS as u64 - | uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_EXACT_PRIOR_BARRIERS as u64 - | uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_FEW_PRIMITIVES as u64 - | uapi::drm_asahi_neo_feature_DRM_ASAHI_NEO_FEATURE_FRAGMENT_BARRIERS as u64; + uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS as u64 + | uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS as u64 + | uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_FEW_PRIMITIVES as u64 + | uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS as u64 + | uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_RENDER_RAY_QUERY as u64; params.gpu_generation = hw::GpuGen::G17 as u32; params.gpu_variant = self.cfg.identity.gpu_variant as u32; params.gpu_revision = self.id.gpu_rev as u32; @@ -1129,8 +1133,8 @@ impl gpu::Gpu for Gpu { true } - fn queue_limits(&self) -> Option { - Some(uapi::drm_asahi_neo_queue_limits { + fn queue_limits(&self) -> Option { + Some(uapi::drm_asahi_queue_limits { max_queues: mmu::MAX_EXECUTION_CONTEXTS, max_in_flight_per_queue: hw::t8140::queues::RENDER_PAIRS_PER_OWNER as u32, ..Default::default() @@ -1168,7 +1172,7 @@ impl gpu::Gpu for Gpu { priority: u32, usc_exec_base: u64, ) -> Result> { - let dev = self.shared.drm_neo()?; + let dev = self.shared.drm()?; let owner = self.ids.queue.next(); let frontend = queue::Frontend::new(priority, usc_exec_base, self.geometry.user_usable_range())?; @@ -1216,7 +1220,7 @@ impl gpu::Gpu for Gpu { } /// Scheduling profile of a new logical queue. `file.rs` passes -/// `DRM_ASAHI_NEO_PRIORITY_REALTIME - priority` and refuses HIGH and REALTIME, so +/// `DRM_ASAHI_PRIORITY_REALTIME - priority` and refuses HIGH and REALTIME, so /// MEDIUM arrives as 2 and LOW as 3. fn scheduling_policy(priority: u32) -> Result { match priority { diff --git a/drivers/gpu/drm/asahi-neo/g17/object.rs b/drivers/gpu/drm/asahi/g17/object.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/object.rs rename to drivers/gpu/drm/asahi/g17/object.rs index 4076cd91bff163..cf1d49fb03a6bb 100644 --- a/drivers/gpu/drm/asahi-neo/g17/object.rs +++ b/drivers/gpu/drm/asahi/g17/object.rs @@ -17,7 +17,7 @@ use core::sync::atomic::{ }; use kernel::{ - drm_neo::gem::shmem, + drm::gem::shmem, prelude::*, // }; diff --git a/drivers/gpu/drm/asahi-neo/g17/power.rs b/drivers/gpu/drm/asahi/g17/power.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/power.rs rename to drivers/gpu/drm/asahi/g17/power.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/preparation.rs b/drivers/gpu/drm/asahi/g17/preparation.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/preparation.rs rename to drivers/gpu/drm/asahi/g17/preparation.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/qos.rs b/drivers/gpu/drm/asahi/g17/qos.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/qos.rs rename to drivers/gpu/drm/asahi/g17/qos.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/admission.rs b/drivers/gpu/drm/asahi/g17/queue/admission.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/admission.rs rename to drivers/gpu/drm/asahi/g17/queue/admission.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/compute.rs b/drivers/gpu/drm/asahi/g17/queue/compute.rs similarity index 93% rename from drivers/gpu/drm/asahi-neo/g17/queue/compute.rs rename to drivers/gpu/drm/asahi/g17/queue/compute.rs index 8eb31de5b98b64..b78598cfbb7609 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/compute.rs +++ b/drivers/gpu/drm/asahi/g17/queue/compute.rs @@ -416,8 +416,8 @@ pub(crate) struct PoolBacking { } impl PoolBacking { - pub(crate) fn new(alloc: &Allocator<'_>, id: kick::Id) -> Result { - let pool = ComputePool::new(alloc, id.qid())?; + pub(crate) fn new(alloc: &Allocator<'_>, id: kick::Id, blocks: usize) -> Result { + let pool = ComputePool::new(alloc, id.qid(), blocks)?; Ok(Self { pool, id }) } @@ -459,6 +459,8 @@ pub(crate) struct Queue { /// the installed queue record still names them until the next owner binds. retained_owner: Option, pool_phase: PoolPhase, + growing: bool, + growth_blocks: usize, /// A killed queue was recycled: its pool/ring must be vacated and re-armed /// fresh before any owner binds it (set by `recycle_killed`, cleared by `rearm`). needs_fresh_backing: bool, @@ -518,6 +520,8 @@ impl Queue { owner: Some(owner), retained_owner: None, pool_phase: PoolPhase::Populated, + growing: false, + growth_blocks: 0, needs_fresh_backing: false, released_at: 0, free_list_generation: 0, @@ -925,25 +929,24 @@ impl Queue { self.retirement_proved = false; self.replay_retirement = false; self.spared_deferred = false; - // The complete retirement witness also covers guilty work. Keep its - // VM status failed, but let logical close return the retained physical - // queue to service instead of consuming another QID for its successor. - self.spared_quarantine = false; - self.quarantine_error = None; - self.submitted = 0; - self.kick.clear_parent_after_recovery(); - self.quarantined = false; - self.retire_pending = false; - self.retirement_ready = true; - if self - .previous - .as_ref() - .is_some_and(|p| p.publication.is_some()) - { - drop(self.previous.take()); - } - if self.released { - self.owner = None; + if self.spared_quarantine { + self.spared_quarantine = false; + self.quarantine_error = None; + self.submitted = 0; + self.kick.clear_parent_after_recovery(); + self.quarantined = false; + self.retire_pending = false; + self.retirement_ready = true; + if self + .previous + .as_ref() + .is_some_and(|p| p.publication.is_some()) + { + drop(self.previous.take()); + } + if self.released { + self.owner = None; + } } Ok(()) } @@ -1043,6 +1046,7 @@ impl Queue { Ok(true) } pub(crate) fn reusable(&self) -> bool { + if self.growing { return false; } self.installed // A pool release in flight must be witnessed before any successor. && !matches!(self.pool_phase, PoolPhase::ReleaseSent(_)) @@ -1188,6 +1192,33 @@ impl Queue { { return Ok(false); } + self.recycle_settled()?; + Ok(true) + } + /// A quarantined queue whose owner closed and whose every retained command was + /// already settled, by the exact retirement witness or by an acknowledged kill, + /// holds no pending command: the firmware consumed its item ring exactly as after + /// a kill. Neither `reusable` nor `exit_candidate` matches it, so without this it + /// keeps its QID and USC pool backing until processor stop, and repeated client + /// failures exhaust the compute pool VA range. Return it to service as + /// `recycle_killed` does. + pub(crate) fn recycle_quarantined_idle(&mut self) -> Result { + if !self.installed + || !self.quarantined + || !self.released + || self.growing + || self.exit.is_some() + || self.awaiting_witness + || self.active.len != 0 + || self.replays.len != 0 + || self.previous.is_some() + { + return Ok(false); + } + self.recycle_settled()?; + Ok(true) + } + fn recycle_settled(&mut self) -> Result { self.graph.reset_items()?; self.exit = None; self.quarantined = false; @@ -1209,7 +1240,7 @@ impl Queue { self.retirement_ready = true; self.released_at = now_ns(); self.needs_fresh_backing = true; - Ok(true) + Ok(()) } pub(crate) fn released_at(&self) -> u64 { self.released_at @@ -1239,6 +1270,7 @@ impl Queue { consumer: u32, producer: u32, ) -> Result> { + if self.growing { return Ok(None); } let PoolPhase::ReleaseSent(cursor) = self.pool_phase else { return Ok(None); }; @@ -1275,6 +1307,55 @@ impl Queue { self.needs_fresh_backing = false; Ok((self.pool.id(), self.pool.page_list_va()?)) } + pub(crate) fn pool_fits(&self, required: usize) -> bool { + !self.growing && self.pool_phase == PoolPhase::Populated && self.pool.fits(required) + } + + /// Only a capacity increase waits for this queue's previously published work. + /// Other queues and ordinary submissions keep their existing concurrency. + pub(crate) fn begin_growth(&mut self, init: &crate::g17::initdata::InitData, blocks: usize) -> Result { + if self.growing { + if blocks < self.growth_blocks { return Err(EAGAIN); } + self.growth_blocks = blocks; + return Ok(false); + } + if self.quarantined || self.released || self.exit.is_some() + || self.active.len != 0 || self.replays.len != 0 || self.previous.is_some() + || (self.installed && !self.retirement_ready) + || self.pool_phase != PoolPhase::Populated { return Err(EAGAIN); } + if self.installed { + let record = ControlRecord::FreeListRelease(self.pool.release_record(self.free_list_generation)); + let cursor = Rings::publish_control(init, &record)?; + self.pool_phase = PoolPhase::ReleaseSent(cursor); + } else { self.pool_phase = PoolPhase::Vacant; } + self.growing = true; + self.growth_blocks = blocks; + Ok(self.installed) + } + + pub(crate) fn finish_growth(&mut self, init: &crate::g17::initdata::InitData, + pages: &mut Option, + alias: &mut Option, blocks: usize) -> Result { + if !self.growing || blocks < self.growth_blocks { return Err(EAGAIN); } + if let PoolPhase::ReleaseSent(cursor) = self.pool_phase { + let consumer = init.control_consumer()?.load(Ordering::Acquire); + let producer = init.control_producer()?.load(Ordering::Acquire); + if !crate::g17::freelist::control_consumed(consumer, producer, cursor)? { + return Err(EAGAIN); + } + } + let binding = self.binding.as_mut().ok_or(EFAULT)?; + let old_alias = binding.pool_alias.take(); + let old_pages = self.pool.take_pages(); + self.pool.rearm(pages.take().ok_or(EINVAL)?)?; + init.initialize_compute_pool_blocks(self.pool.id(), self.pool.page_list_va()?, blocks)?; + binding.pool_alias = alias.take(); + self.pool_phase = PoolPhase::Populated; + self.growing = false; + self.growth_blocks = 0; + Ok(Vacated { _alias: old_alias, _pages: old_pages, _descriptors: None, _ring: None }) + } + /// Exact host identity remains valid even when several commands share an ioctl ID. pub(crate) fn owns_packet(&self, packet: &Arc) -> bool { self.active @@ -1487,28 +1568,6 @@ pub(crate) trait Host { } impl Queue { - /// Snapshot only while publication is closed and firmware is halted. - pub(crate) fn recovery_diagnostics(&self) -> impl core::fmt::Debug + '_ { - let command = self.active.front().and_then(|active| { - if let crate::g17::command::Validated::Compute { - cdm_va, cdm_end_va, sampler_count, scratch, .. - } = &active.packet.command { - Some((*cdm_va, *cdm_end_va, *sampler_count, scratch.enabled())) - } else { - None - } - }); - ( - self.owner, - self.context().map(|context| (context.id(), context.generation())), - self.ordinal, - self.active.len, - self.graph.cursors(), - self.graph.stamp(), - command, - ) - } - pub(crate) fn spared_pending(&self) -> bool { self.quarantined && self.spared_quarantine && self.awaiting_witness } @@ -1529,6 +1588,7 @@ impl Queue { }; let (epoch, addresses) = self.publication_preflight(&packet, host)?; let crate::g17::command::Validated::Compute { + intersection, scratch, usc_base, cdm_va, @@ -1563,6 +1623,7 @@ impl Queue { self.graph.ensure_record_slot(slot)?; } let args = ComputeArgs { + intersection: *intersection, scratch: *scratch, usc_base: *usc_base, cdm_va: *cdm_va, @@ -1617,7 +1678,7 @@ impl Queue { mcache: None, event_mask: COMPUTE_KICK_EVENT_MASK, register_arrays: ComputeDescriptor::register_bindings(descriptor_low, *scratch)?, - compute_scratch: scratch.enabled(), + scs_enabled: scratch.enabled(), priority: policy.priority(), }; let published = host.next_compute_publication()?; diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/compute/exit.rs b/drivers/gpu/drm/asahi/g17/queue/compute/exit.rs similarity index 85% rename from drivers/gpu/drm/asahi-neo/g17/queue/compute/exit.rs rename to drivers/gpu/drm/asahi/g17/queue/compute/exit.rs index 10d2a372748002..187fd27f8e78e6 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/compute/exit.rs +++ b/drivers/gpu/drm/asahi/g17/queue/compute/exit.rs @@ -83,11 +83,25 @@ impl Transaction { } impl Queue { + /// A command with ray traversal scratch is never killed. On Neo, killing a + /// traversal that stopped making progress either crashed the GPU firmware + /// when the queue's pool was then released, or stalled every later render + /// when it was retained; left alone, the rest of the GPU kept working. Such + /// a queue stays quarantined and retained until its commands retire. + fn active_traversal(&self) -> bool { + self.active.iter().any(|active| { + matches!( + &active.packet.command, + crate::g17::command::Validated::Compute { scratch, .. } if scratch.enabled() + ) + }) + } pub(crate) fn exit_candidate(&self, context: &Arc, age: u64) -> bool { self.context().is_some_and(|bound| Arc::ptr_eq(bound, context)) && self.released && self.active.len != 0 && self.exit.is_none() + && !self.active_traversal() && (self.quarantined || self.retired_by_teardown) && age >= if self.quarantined { @@ -156,10 +170,16 @@ impl Queue { complete_qos: &mut impl FnMut(crate::g17::qos::Owner) -> Result, defer: &mut impl FnMut(Deferred) -> Result, ) -> Result { - if !self.exit.as_ref().is_some_and(|exit| exit.phase == Phase::Acknowledged) { + let Some(phase) = self.exit.as_ref().map(|exit| exit.phase) else { + return Ok(()); + }; + if !matches!(phase, Phase::Acknowledged | Phase::Unproven) { return Ok(()); } - if self.active.len != 0 { + // An unproven kill (acknowledged before the scheduler page showed the killed + // state) is re-read on later passes and settles only once the page proves it, + // even after its commands were retired by other means. + if self.active.len != 0 || phase == Phase::Unproven { if !recovery_closed { return Ok(()); } diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/compute/retirement.rs b/drivers/gpu/drm/asahi/g17/queue/compute/retirement.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/compute/retirement.rs rename to drivers/gpu/drm/asahi/g17/queue/compute/retirement.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/mod.rs b/drivers/gpu/drm/asahi/g17/queue/mod.rs similarity index 92% rename from drivers/gpu/drm/asahi-neo/g17/queue/mod.rs rename to drivers/gpu/drm/asahi/g17/queue/mod.rs index 370f6f5f791488..326405f05fdf90 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/mod.rs +++ b/drivers/gpu/drm/asahi/g17/queue/mod.rs @@ -10,8 +10,8 @@ use super::{ command::{Parser, UscWindow, Validated}, completion::{Completion, Destinations, PreparedPublication}, context::Context, - fence::{self, Outputs, Submission}, - job::{Backend, Engine, Fences, Order, Packet, Scheduler}, + fence::{Outputs, Submission}, + job::{self, Backend, Engine, Fences, Order, Packet, Scheduler}, validation::Proof, }; use crate::{ @@ -127,9 +127,9 @@ impl Queue { }) } - fn ensure_compute(&mut self) -> Result { + fn ensure_compute(&mut self, scs_pages: u16) -> Result { // Admission is checked even when this queue already owns a scheduler. - self.backend.ensure_compute()?; + self.backend.ensure_compute(scs_pages)?; if self.compute.is_none() { self.compute = Some(Scheduler::new( &self.dev, @@ -143,7 +143,7 @@ impl Queue { fn prune_frontier(&mut self, index: usize) { let failure = &mut self.failures[index]; self.frontiers[index].retain(|(sequence, fences)| { - let status = fence::completion_status(&fences.completed); + let status = job::fence_status(&fences.completed); if status == 0 { return true; } @@ -171,10 +171,15 @@ impl Queue { Validated::Compute { .. } => Engine::Compute, }; let index = engine.index(); + if let Validated::Render { ray_query, .. } = &command { + let blocks = self.backend.render_scs_blocks(ray_query.map(|stage| stage.scratch.pages()))?; + if blocks > self.context.render_blocks() { return Err(EINVAL); } + } self.prune_frontier(index); self.frontiers[index].reserve(1, GFP_KERNEL)?; if engine == Engine::Compute { - self.ensure_compute()?; + let Validated::Compute { scratch, .. } = &command else { return Err(EINVAL); }; + self.ensure_compute(scratch.pages())?; } // Only earlier render ordinals covered by the explicit logical prefix // can delay host publication. These links never become GPU dependencies. @@ -255,7 +260,7 @@ impl Queue { // Successful completion includes host timestamp publication. Its // ready fence adds no ordering and would prevent shallow inline // publication. Failed and pending producers still need checking. - if fence::completion_status(&fences.completed) > 0 { + if job::fence_status(&fences.completed) > 0 { continue; } dependencies.push(fences.ready.clone(), GFP_KERNEL)?; @@ -353,6 +358,13 @@ impl Queue { } impl crate::queue::Queue for Queue { + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { + let pages = [super::fw::compute::ScratchRequest::new(layouts[0])?.pages(), + super::fw::compute::ScratchRequest::new(layouts[1])?.pages()]; + let blocks = self.backend.render_scs_blocks(pages)?; + self.context.reserve_render_blocks(blocks) + } + fn submit( &mut self, id: u64, diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render.rs b/drivers/gpu/drm/asahi/g17/queue/render.rs similarity index 89% rename from drivers/gpu/drm/asahi-neo/g17/queue/render.rs rename to drivers/gpu/drm/asahi/g17/queue/render.rs index d82ee84c1283ff..99be455b48e402 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/render.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render.rs @@ -43,6 +43,8 @@ use retirement::{Ticket, Tracker}; pub(crate) struct RenderDependencies { pub(crate) render: [KickDependency; 2], pub(crate) render_count: usize, + pub(crate) tiling: [KickDependency; 2], + pub(crate) tiling_count: usize, pub(crate) compute: Option, } @@ -50,6 +52,8 @@ impl RenderDependencies { pub(crate) const EMPTY: Self = Self { render: [KickDependency::ZERO; 2], render_count: 0, + tiling: [KickDependency::ZERO; 2], + tiling_count: 0, compute: None, }; pub(crate) fn render(&self) -> &[KickDependency] { @@ -126,8 +130,14 @@ struct Active { context_update: bool, failed: bool, drained_result: Option>, + /// First observation of this group retired and unlinked without execution. + abandoned_since: Option, } +/// A firmware-retired, unexecuted group must hold on two observations this far +/// apart before it is republished; a single sample may straddle an update. +const ABANDON_GRACE_NS: u64 = 2_000_000; + /// Installed kick backing remains owned even when this pair has no logical /// owner. Commands retain their exact descriptor and allocation generations. pub(crate) struct Pair { @@ -158,6 +168,8 @@ pub(crate) struct Pair { previous: Option, displaced: Option, context_update: bool, + /// A retired, never-executed group awaiting republication on this pair. + replay: Option>, } impl Pair { #[inline(never)] @@ -239,6 +251,7 @@ impl Pair { terminal: None, previous: None, displaced: None, + replay: None, context_update: true, })) } @@ -267,7 +280,21 @@ impl Pair { &mut self.memory.manager } pub(crate) fn in_flight(&self) -> bool { - self.active.len() != 0 + self.active.len() != 0 || self.replay.is_some() + } + /// A replay or an abandonment grace period needs the worker to poll. + pub(crate) fn replay_polling(&self) -> bool { + self.replay.is_some() || self.active.iter().any(|a| a.abandoned_since.is_some()) + } + pub(crate) fn replay_front(&self) -> Option> { + self.replay.clone() + } + pub(crate) fn finish_replay(&mut self, packet: &Arc) -> Result { + if !self.replay.as_ref().is_some_and(|front| Arc::ptr_eq(front, packet)) { + return Err(EIO); + } + self.replay = None; + Ok(()) } pub(crate) fn published_in_flight(&self) -> bool { !self.quarantined && self.active.iter().any(|active| active.outer_published) @@ -294,7 +321,7 @@ impl Pair { return Err(EBUSY); } let Validated::Render { - pass, attachments, .. + pass, attachments, ray_query, .. } = &packet.command else { return Err(EINVAL); @@ -358,6 +385,7 @@ impl Pair { let qids = self.qids(); let args = abi::RenderArgs { pass, + ray_query: *ray_query, resources: &resources, ordinal, ta: abi::RenderStage { @@ -403,8 +431,8 @@ impl Pair { /// Kick barriers of one render. The fragment kick always waits for its own tiling kick; /// with the fragment-barriers flag the resolved prefix orders the fragment kick too and the -/// tiling kick keeps only its implicit parent, otherwise the prefix orders the tiling kick. -/// A compute dependency always orders the tiling kick. +/// tiling kick preserves the TA prefix across physical pairs. Otherwise the render prefix +/// orders the tiling kick. A compute dependency always orders the tiling kick. struct RenderBarriers { tiling: [KickDependency; 3], tiling_count: usize, @@ -430,6 +458,17 @@ fn render_barriers( tiling_count += 1; } } + if fragment_stage { + // The implicit parent covers only this physical ring. Preserve the logical + // TA prefix when a predecessor was published on another pair, without + // making this TA kick wait for that predecessor's fragment completion. + for dependency in &dependencies.tiling[..dependencies.tiling_count] { + if dependency.qid() != own_tiling.qid() { + tiling_barriers[tiling_count] = *dependency; + tiling_count += 1; + } + } + } if let Some(compute) = dependencies.compute { tiling_barriers[tiling_count] = compute; tiling_count += 1; @@ -547,7 +586,7 @@ impl Pair { self.memory.graph.queues.gpu_va() + u64::from(qid) * size_of::() as u64 }); - let Validated::Render { pass, .. } = &packet.command else { + let Validated::Render { pass, ray_query, .. } = &packet.command else { return Err(EINVAL); }; let own_tiling = KickDependency::new(qids[0], kicks[0]).ok_or(EINVAL)?; @@ -571,13 +610,16 @@ impl Pair { mcache, event_mask: abi::RENDER_KICK_EVENT_MASK, register_arrays: arrays, - compute_scratch: false, + scs_enabled: ray_query[stage].scratch.enabled(), priority: policy.priority(), }; let ta_args = args( 0, tiling_barriers, - abi::TaDescriptor::register_bindings(self.memory.graph.descriptor_client(0, ordinal))?, + abi::TaDescriptor::register_bindings( + self.memory.graph.descriptor_client(0, ordinal), + ray_query[0].scratch.enabled(), + )?, None, ); let fragment_args = args( @@ -585,6 +627,7 @@ impl Pair { &barriers.fragment[..barriers.fragment_count], abi::FragmentDescriptor::register_bindings( self.memory.graph.descriptor_client(1, ordinal), + ray_query[1].scratch.enabled(), )?, mcache, ); @@ -667,6 +710,7 @@ impl Pair { context_update: self.context_update, failed: false, drained_result: None, + abandoned_since: None, }) .map_err(|_| EBUSY)?; self.prepared = None; @@ -889,6 +933,9 @@ impl Pair { .as_ref() .is_some_and(|prepared| Arc::ptr_eq(&prepared.packet, packet)) } + pub(crate) fn replay_contains(&self, packet: &Arc) -> bool { + self.replay.as_ref().is_some_and(|replay| Arc::ptr_eq(replay, packet)) + } pub(crate) fn take_deferred_timeout(&self) -> Option<(Arc, Error)> { self.active.iter().find_map(|active| { active @@ -909,6 +956,7 @@ impl Pair { return Err(EINVAL); }; let required = target::Layout::new(pass, self.clusters)?.tvb_blocks; + self.memory.manager.note_required(required, crate::g17::now_ns()); if self.in_flight() { return Ok(None); } @@ -1063,7 +1111,8 @@ impl Pair { }; let timestamps = match result { Ok(retirement::Poll::Complete(timestamps)) => timestamps, - Ok(retirement::Poll::Retry | retirement::Poll::Pending) => return Ok(false), + Ok(retirement::Poll::Pending) => return self.replay_abandoned(host, defer), + Ok(retirement::Poll::Retry) => return Ok(false), Err(error) => { self.quarantine(error, defer)?; return Ok(false); @@ -1093,6 +1142,105 @@ impl Pair { } } + /// The front group was retired and unlinked by firmware without executing + /// (job list empty, stamps cover it, no recovery, fragment end never + /// written). Nothing of this group reached its attachments, so the same + /// command is published again instead of failing the owner's VM. The + /// proof is the strict timeout witness, taken twice ABANDON_GRACE_NS apart, + /// and only for a lone group whose completion still accepts a replay. + fn replay_abandoned( + &mut self, + host: &mut impl Host, + defer: &mut impl FnMut(Deferred) -> Result, + ) -> Result { + use retirement::Memory; + if self.quarantined || self.active.len() != 1 || self.replay.is_some() { + return Ok(false); + } + let front = self.active.front().ok_or(EIO)?; + if !front.outer_published || front.failed { + return Ok(false); + } + let mut observer = Observer { + memory: &self.memory, + qids: self.qids(), + host, + }; + core::sync::atomic::fence(core::sync::atomic::Ordering::Acquire); + let snapshot = observer.snapshot(front.ticket.ordinal)?; + let stamps = observer.stamps()?; + let (_, state) = observer.host.epoch()?; + let abandoned = + retirement::abandoned(&front.ticket, snapshot, stamps, state, false)?; + let now = crate::g17::now_ns(); + let front = self.active.front_mut().ok_or(EIO)?; + if !abandoned { + front.abandoned_since = None; + return Ok(false); + } + let Some(since) = front.abandoned_since else { + front.abandoned_since = Some(now); + return Ok(false); + }; + if now.saturating_sub(since) < ABANDON_GRACE_NS { + return Ok(false); + } + if crate::debug::debug_enabled(crate::debug::DebugFlags::VerboseFaults) { + pr_info!( + "render slot {} retired without execution (pass {:?}); replaying\n", + self.slot, + snapshot.pass + ); + } + let qos = self.qos_owners(); + let buffer = self.buffer_id(); + let pool = self.free_list.id(); + let front = self.active.front_mut().ok_or(EIO)?; + if !front.flist_completed { + host.complete_render( + (!front.qos_completed).then_some(qos), + pool, + front.generation, + buffer, + )?; + front.flist_completed = true; + front.qos_completed = true; + } + let ordinal = front.ticket.ordinal; + let packet = front.packet.clone(); + if !packet.completion.begin_replay() { + if !packet.completion.spared() { + packet.completion.status().record(EIO); + if self.terminal.is_none() { + self.terminal = Some((packet.completion.status().clone(), EIO)); + } + } + defer(Deferred::Retired(packet.completion.clone(), Err(EIO)))?; + self.memory.retire(ordinal)?; + self.active.pop().ok_or(EIO)?; + return Ok(true); + } + self.memory.retire(ordinal)?; + self.active.pop().ok_or(EIO)?; + packet.reset_publication_for_replay(); + self.replay = Some(packet); + Ok(true) + } + + /// The pending replay cannot proceed (closed owner, cancellation or a + /// publication refusal). ECANCELED leaves the VM usable; any other error + /// is the ordinary device-loss outcome the replay tried to avoid. + pub(crate) fn fail_replay( + &mut self, + error: Error, + defer: &mut impl FnMut(Deferred) -> Result, + ) -> Result { + let Some(packet) = self.replay.take() else { + return Ok(()); + }; + defer(Deferred::Retired(packet.completion.clone(), Err(error))) + } + /// A closed preparation epoch invalidates only this unpublished descriptor. /// Its completion remains available for a retry under the next epoch. pub(crate) fn discard_prepared(&mut self, packet: &Arc) -> Result { @@ -1276,6 +1424,9 @@ impl Pair { self.memory.retire(ordinal)?; self.active.pop().ok_or(EIO)?; } + if let Some(packet) = self.replay.take() { + defer(Deferred::Drained(packet.completion.clone(), Err(EIO)))?; + } if let Some(previous) = self.previous.as_mut() { previous.ready = true; } diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/dependencies.rs b/drivers/gpu/drm/asahi/g17/queue/render/dependencies.rs similarity index 72% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/dependencies.rs rename to drivers/gpu/drm/asahi/g17/queue/render/dependencies.rs index be1d9fc9c902ce..53a0fa6e449ed7 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/render/dependencies.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render/dependencies.rs @@ -15,6 +15,7 @@ use kernel::{prelude::*, sync::Arc}; struct Predecessor { packet: Arc, word: u64, + ta_word: u64, } /// A loan of the graph cannot hide work already published on that pair. Fence @@ -68,6 +69,21 @@ impl Dependencies { } Ok(()) } + pub(crate) fn collect_tiling( + &self, + owner: u64, + status: &Arc, + prefix: Option, + frontier: &mut Frontier, + ) -> Result { + if self.owner != Some(owner) { + return Ok(()); + } + for entry in self.entries.iter().flatten() { + include(frontier, status, prefix, &entry.packet, entry.ta_word)?; + } + Ok(()) + } } impl Pair { @@ -81,6 +97,8 @@ impl Pair { entries: core::array::from_fn(|_| { active.next().map(|entry| Predecessor { packet: entry.packet.clone(), + ta_word: KickDependency::new(self.qids()[0], entry.kicks[0]) + .map_or(u64::MAX, KickDependency::word), word: KickDependency::new(qid, entry.kicks[1]) .map_or(u64::MAX, KickDependency::word), }) @@ -106,4 +124,22 @@ impl Pair { } Ok(()) } + pub(crate) fn collect_tiling_dependencies( + &self, + owner: u64, + status: &Arc, + prefix: Option, + frontier: &mut Frontier, + ) -> Result { + if self.owner != Some(owner) { + return Ok(()); + } + let qid = self.qids()[0]; + for entry in self.active.iter() { + let word = + KickDependency::new(qid, entry.kicks[0]).map_or(u64::MAX, KickDependency::word); + include(frontier, status, prefix, &entry.packet, word)?; + } + Ok(()) + } } diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/memory.rs b/drivers/gpu/drm/asahi/g17/queue/render/memory.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/memory.rs rename to drivers/gpu/drm/asahi/g17/queue/render/memory.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/observation.rs b/drivers/gpu/drm/asahi/g17/queue/render/observation.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/observation.rs rename to drivers/gpu/drm/asahi/g17/queue/render/observation.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/parked.rs b/drivers/gpu/drm/asahi/g17/queue/render/parked.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/parked.rs rename to drivers/gpu/drm/asahi/g17/queue/render/parked.rs index f712bb7b1b87dd..39941de815e077 100644 --- a/drivers/gpu/drm/asahi-neo/g17/queue/render/parked.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render/parked.rs @@ -213,6 +213,7 @@ impl ParkedPair { terminal: None, previous: self.previous.take(), displaced: None, + replay: None, context_update: true, })) } diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/publication.rs b/drivers/gpu/drm/asahi/g17/queue/render/publication.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/publication.rs rename to drivers/gpu/drm/asahi/g17/queue/render/publication.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/retirement.rs b/drivers/gpu/drm/asahi/g17/queue/render/retirement.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/retirement.rs rename to drivers/gpu/drm/asahi/g17/queue/render/retirement.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/queue/render/warm.rs b/drivers/gpu/drm/asahi/g17/queue/render/warm.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/queue/render/warm.rs rename to drivers/gpu/drm/asahi/g17/queue/render/warm.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/recovery.rs b/drivers/gpu/drm/asahi/g17/recovery.rs similarity index 98% rename from drivers/gpu/drm/asahi-neo/g17/recovery.rs rename to drivers/gpu/drm/asahi/g17/recovery.rs index 74bb9ca7567eac..ba368341d84d26 100644 --- a/drivers/gpu/drm/asahi-neo/g17/recovery.rs +++ b/drivers/gpu/drm/asahi/g17/recovery.rs @@ -155,7 +155,6 @@ impl State { pub(super) trait Host: Memory + Queues { fn recovery(&self) -> &State; fn recovery_mut(&mut self) -> &mut State; - fn report_recovery(&self, _generation: u64, _blamed: Option, _sources: &Sources) {} } /// Performs one pending restart. The caller has closed preparation admission and joined all @@ -197,7 +196,6 @@ pub(super) fn service( host.clear_timestamps()?; } let mut sources = host.fault_sources()?; - host.report_recovery(request.generation, request.blamed, &sources); let target_error = host.publish_recovery_targets(); host.transition(1, 2)?; let mut recovered = false; diff --git a/drivers/gpu/drm/asahi-neo/g17/recovery/attribution.rs b/drivers/gpu/drm/asahi/g17/recovery/attribution.rs similarity index 92% rename from drivers/gpu/drm/asahi-neo/g17/recovery/attribution.rs rename to drivers/gpu/drm/asahi/g17/recovery/attribution.rs index a580a860ee3c46..1755484219187e 100644 --- a/drivers/gpu/drm/asahi-neo/g17/recovery/attribution.rs +++ b/drivers/gpu/drm/asahi/g17/recovery/attribution.rs @@ -6,7 +6,7 @@ pub(super) const SLOTS: usize = 16; const STAMP_MASK: u64 = (1 << 40) - 1; const PAGE_FAULT: u32 = 3; -#[derive(Clone, Copy, Debug, Default)] +#[derive(Clone, Copy, Default)] pub(super) struct Source { slot: u8, qid: Option, @@ -30,11 +30,6 @@ pub(super) enum Attribution { } impl Sources { - /// Selected slot identities only; unused records carry no diagnostic evidence. - pub(crate) fn diagnostics(&self) -> impl core::fmt::Debug + '_ { - (self.mask, &self.records[..self.count]) - } - /// Firmware is halted for every selected key and progress-record read. /// Failed reads stay unknown; incomplete evidence cannot override firmware blame. pub(crate) fn sample(mask: u32, mut read: impl FnMut(usize) -> Option<(u64, u32)>) -> Self { diff --git a/drivers/gpu/drm/asahi-neo/g17/recovery/slot.rs b/drivers/gpu/drm/asahi/g17/recovery/slot.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/recovery/slot.rs rename to drivers/gpu/drm/asahi/g17/recovery/slot.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime.rs b/drivers/gpu/drm/asahi/g17/runtime.rs similarity index 87% rename from drivers/gpu/drm/asahi-neo/g17/runtime.rs rename to drivers/gpu/drm/asahi/g17/runtime.rs index 029dd590148544..ffaff6d80dbf9f 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime.rs +++ b/drivers/gpu/drm/asahi/g17/runtime.rs @@ -121,10 +121,10 @@ impl Registry { /// replacement context must succeed before its old binding is displaced. /// A vacant queue (USC backing given back) is offered for re-arming only /// when no populated idle queue exists. - pub(super) fn rebind_compute(&mut self, owner: u64, context: &Arc) -> Result { + pub(super) fn rebind_compute(&mut self, owner: u64, context: &Arc, required: usize) -> Result { if let Some(queue) = self.compute(owner) { return Ok(if queue.matches(owner, queue.qid(), context)? { - Rebind::Done + if queue.pool_fits(required) { Rebind::Done } else { Rebind::Grow } } else { Rebind::Fresh }); @@ -141,6 +141,7 @@ impl Registry { vacant.get_or_insert(queue.qid()); continue; } + if !queue.pool_fits(required) { continue; } queue.bind_owner(owner, context.clone())?; return Ok(Rebind::Done); } @@ -176,30 +177,42 @@ impl Registry { queue.bind_owner(owner, context.clone()) } - /// Publishes releases for the oldest idle retained pools beyond `cap`, at - /// most four per pass. Returns the number published. + /// Delay before retrying pools a turn could not release. + const POOL_RETRY_NS: u64 = 100_000_000; + /// Publishes releases for the oldest idle retained pools beyond `cap` and + /// for every pool released at least `expiry` before `now`, at most four + /// per pass. Returns the number published and the delay until the next + /// retained pool expires or a retry is due. pub(super) fn vacate_idle_pools( &mut self, init: &super::initdata::InitData, cap: usize, - ) -> Result { + now: u64, + expiry: u64, + ) -> Result<(usize, Option)> { // Count first: event turns are frequent and the steady state has no excess. - let forced = self - .compute - .iter() - .flatten() - .filter_map(|entry| entry.queue.as_deref()) - .filter(|queue| queue.vacate_candidate() && queue.needs_fresh_backing()) - .count(); - let count = self + let mut forced = 0; + let mut count = 0; + let mut next = None::; + for queue in self .compute .iter() .flatten() .filter_map(|entry| entry.queue.as_deref()) .filter(|queue| queue.vacate_candidate()) - .count(); + { + count += 1; + let released = queue.released_at(); + let age = now.saturating_sub(released); + if queue.needs_fresh_backing() || (released != 0 && age >= expiry) { + forced += 1; + } else if released != 0 { + let left = expiry - age; + next = Some(next.map_or(left, |next| next.min(left))); + } + } if count <= cap && forced == 0 { - return Ok(0); + return Ok((0, next)); } let mut candidates: KVec<(u64, u8)> = KVec::with_capacity(count, GFP_KERNEL)?; for queue in self @@ -213,6 +226,7 @@ impl Registry { let key = if queue.needs_fresh_backing() { 0 } else { queue.released_at() }; candidates.push((key, queue.qid()), GFP_KERNEL)?; } + // Recycled and expired pools are the oldest keys and sort first. candidates.sort_unstable(); let excess = candidates.len().saturating_sub(cap).max(forced); let mut published = 0; @@ -230,7 +244,11 @@ impl Registry { } published += 1; } - Ok(published) + if published < excess { + // Beyond this turn's bound, or not yet releasable: retry soon. + next = Some(next.map_or(Self::POOL_RETRY_NS, |next| next.min(Self::POOL_RETRY_NS))); + } + Ok((published, next)) } /// Frees the backing of every pool whose release the firmware consumed. @@ -348,6 +366,7 @@ impl Registry { pub(super) enum Rebind { /// The owner now holds an installed, populated queue. Done, + Grow, /// Only a vacant queue is available; re-arm `qid` with fresh backing. Rearm(u8), /// No retained queue suits the owner: reserve a fresh QID. @@ -371,7 +390,7 @@ impl Backend { policy: super::fw::queue::Policy, ) -> Result<(Arc, admission::QueueSlot)> { let lease = shared.preparations.enter(true)?; - let dev = shared.drm_neo()?; + let dev = shared.drm()?; let (uat, pool, qos, execution, slot) = { let mut state = shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; @@ -422,7 +441,7 @@ impl Backend { fn shared_compute(&self) -> Result { loop { let lease = self.shared.preparations.enter(false)?; - let dev = self.shared.drm_neo()?; + let dev = self.shared.drm()?; let uat = { let mut state = self.shared.state.lock(); loop { @@ -532,7 +551,52 @@ impl Backend { result } - fn build_compute(&self) -> Result { + fn grow_compute(&self, required: usize) -> Result { + use kernel::time::{delay::fsleep, Delta}; + { + let state = self.shared.state.lock(); + let firmware = state.as_ref().ok_or(ENODEV)?; + if firmware.queues.compute(self.owner).is_some_and(|queue| queue.pool_fits(required)) { + return Ok(()); + } + } + let lease = self.shared.preparations.enter(false)?; + let dev = self.shared.drm()?; + let uat = self.shared.state.lock().as_ref().ok_or(ENODEV)?.uat.clone(); + let alloc = super::object::Allocator { dev: &dev, uat: &uat }; + let mut pages = Some(super::freelist::Pages::compute_sized(&alloc, required)?); + let mut alias = Some(pages.as_ref().ok_or(EIO)?.map_client(self.context.vm())?); + let mut deferred = DeferredBatch::publication()?; + let started = now_ns(); + loop { + let result = (|| { + let mut state = self.shared.state.lock(); + let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; + if !lease.is_current() { return Err(EAGAIN); } + if self.shared.crashed.load(Ordering::Acquire) { return Err(EIO); } + firmware.scan_compute(None, &mut deferred)?; + let queue = firmware.queues.compute_mut(self.owner).ok_or(EIO)?; + if queue.pool_fits(required) { return Ok(None); } + if queue.begin_growth(&firmware.init, required)? { + fence(Ordering::SeqCst); + firmware.primary.notify(crate::g17::MSG_CONTROL_NOTIFY)?; + } + queue.finish_growth(&firmware.init, &mut pages, &mut alias, required).map(Some) + })(); + deferred.finish(); + match result { + Ok(old) => { drop(old); return Ok(()); } + Err(error) if error == EAGAIN && lease.is_current() + && now_ns().saturating_sub(started) < WORK_RING_WAIT_NS => { + self.shared.queue_events(); + fsleep(Delta::from_millis(1)); + } + Err(error) => return Err(error), + } + } + } + + fn build_compute(&self, required: usize) -> Result { loop { match self.shared_compute() { Err(error) if error == EAGAIN => continue, @@ -544,7 +608,7 @@ impl Backend { } loop { let lease = self.shared.preparations.enter(false)?; - let dev = self.shared.drm_neo()?; + let dev = self.shared.drm()?; let (uat, reservation) = { let mut state = self.shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; @@ -555,8 +619,9 @@ impl Backend { { return Err(EIO); } - match firmware.queues.rebind_compute(self.owner, &self.context)? { + match firmware.queues.rebind_compute(self.owner, &self.context, required)? { Rebind::Done => return Ok(()), + Rebind::Grow => { drop(state); return self.grow_compute(required); } Rebind::Rearm(qid) => (firmware.uat.clone(), Err(qid)), Rebind::Fresh => { let reservation = firmware.queues.reserve_compute(self.owner)?; @@ -573,7 +638,7 @@ impl Backend { Err(qid) => { // Fresh zeroed backing is built without the device mutex; the // queue may have been taken or re-armed by another owner since. - let pages = crate::g17::freelist::Pages::compute(&alloc)?; + let pages = crate::g17::freelist::Pages::compute_sized(&alloc, required)?; let ring = compute::DescriptorRing::new(&alloc)?; let mut state = self.shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; @@ -586,13 +651,13 @@ impl Backend { }; // The firmware consumed this slot's release before the backing // was dropped; the row is rewritten before the queue is kicked. - firmware.init.initialize_compute_pool(slot, page_list)?; + firmware.init.initialize_compute_pool_blocks(slot, page_list, required)?; firmware.queues.bind_rearmed(qid, self.owner, &self.context)?; return Ok(()); } }; let built = (|| { - let pool = compute::PoolBacking::new(&alloc, reservation.id)?; + let pool = compute::PoolBacking::new(&alloc, reservation.id, required)?; { let mut state = self.shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; @@ -601,7 +666,7 @@ impl Backend { } firmware.queues.reserved(reservation)?; let (slot, page_list) = pool.pool_descriptor()?; - firmware.init.initialize_compute_pool(slot, page_list)?; + firmware.init.initialize_compute_pool_blocks(slot, page_list, required)?; } let backing = compute::Backing::new(&alloc, pool)?; // The global scheduler is created after pool/kick backing and @@ -656,6 +721,8 @@ pub(super) struct DeferredBatch { vacated: KVVec, /// Backing of render pools vacated for evicted idle clients. render: KVVec, + /// Tiler-heap extensions removed from idle heaps, with their aliases. + tvb: KVVec, } impl DeferredBatch { const DEPTH: usize = if crate::hw::t8140::queues::RENDER_DEPTH as usize @@ -676,14 +743,17 @@ impl DeferredBatch { Self::with_capacity(2 * Self::DEPTH, 0) } fn with_capacity(capacity: usize, detached: usize) -> Result { + let render_slots = if detached != 0 { + crate::hw::t8140::queues::RENDER_SLOTS + } else { + 0 + }; Ok(Self { values: KVVec::with_capacity(capacity, GFP_KERNEL)?, bindings: KVVec::with_capacity(detached, GFP_KERNEL)?, vacated: KVVec::with_capacity(detached, GFP_KERNEL)?, - render: KVVec::with_capacity( - if detached != 0 { crate::hw::t8140::queues::RENDER_SLOTS } else { 0 }, - GFP_KERNEL, - )?, + render: KVVec::with_capacity(render_slots, GFP_KERNEL)?, + tvb: KVVec::with_capacity(render_slots, GFP_KERNEL)?, }) } /// Keeps the detached owner's unmapping and VM teardown out of the device @@ -702,6 +772,14 @@ impl DeferredBatch { drop(backing); } } + pub(super) fn tvb_room(&self) -> bool { + self.tvb.len() < self.tvb.capacity() + } + pub(super) fn defer_tvb(&mut self, release: super::buffer::TvbRelease) { + if let Err(release) = self.tvb.push_within_capacity(release) { + drop(release); + } + } pub(super) fn defer_vacated(&mut self, vacated: super::queue::compute::Vacated) { if let Err(vacated) = self.vacated.push_within_capacity(vacated) { drop(vacated); @@ -753,6 +831,9 @@ impl DeferredBatch { for backing in self.render.drain_all() { drop(backing); } + for release in self.tvb.drain_all() { + drop(release); + } } } @@ -972,7 +1053,8 @@ impl super::Firmware { } impl Backend { - fn ensure_compute(&self) -> Result { + fn ensure_compute(&self, scs_pages: u16) -> Result { + let required = super::fw::compute_pool::required_blocks(self.shared.active_cores, scs_pages)?; use kernel::time::{delay::fsleep, Delta}; const SPARED_WAIT_MS: usize = 200; let mut deferred = DeferredBatch::publication()?; @@ -988,7 +1070,7 @@ impl Backend { deferred.finish(); let pending = pending?; if !pending { - return self.build_compute(); + return self.build_compute(required); } // Other owners continue submitting while this owner gives its // queue-local drain witness time to arrive. @@ -1145,16 +1227,24 @@ impl super::Shared { /// Idle retained compute queues that keep their USC pool populated; older idle /// pools give their backing back. const COMPUTE_WARM_CAP: usize = 4; +/// A retained pool idle this long gives its backing back regardless of the +/// cap: a burst of short-lived clients still finds warm pools, but the pools +/// of clients that are gone do not stay resident. The macOS driver frees a +/// client's USC pools with the client and has an idle shrink; ten seconds +/// matches its idle parameter-buffer shrink. +const COMPUTE_WARM_EXPIRY_NS: u64 = 10_000_000_000; impl super::Firmware { /// Event-worker service step: witness consumed pool releases, then release the /// oldest idle retained pools beyond the warm cap. /// Detached owner bindings and vacated pool backing go to `deferred`, which /// the worker finishes after releasing the device mutex. + /// Returns the delay until a retained pool expires, for the worker's wake. pub(in crate::g17) fn service_compute_pools( &mut self, deferred: &mut DeferredBatch, - ) -> Result { + now: u64, + ) -> Result> { for queue in self .queues .compute @@ -1163,19 +1253,26 @@ impl super::Firmware { .filter_map(|entry| entry.queue.as_deref_mut()) { queue.recycle_killed()?; + queue.recycle_quarantined_idle()?; if let Some(binding) = queue.detach_exited_owner() { deferred.defer_binding(binding); } } self.queues.observe_pool_releases(&self.init, deferred)?; if self.render_control_backpressured() { - return Ok(()); + return Ok(None); } - let published = self.queues.vacate_idle_pools(&self.init, COMPUTE_WARM_CAP)?; + let (published, next) = self.queues.vacate_idle_pools( + &self.init, + COMPUTE_WARM_CAP, + now, + COMPUTE_WARM_EXPIRY_NS, + )?; if published != 0 { fence(Ordering::SeqCst); self.primary.notify(crate::g17::MSG_CONTROL_NOTIFY)?; } - Ok(()) + Ok(next) } } + diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/backend.rs b/drivers/gpu/drm/asahi/g17/runtime/backend.rs similarity index 94% rename from drivers/gpu/drm/asahi-neo/g17/runtime/backend.rs rename to drivers/gpu/drm/asahi/g17/runtime/backend.rs index 63a8a0db669c12..3d5f441d8428f1 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/backend.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/backend.rs @@ -8,7 +8,7 @@ use crate::g17::{ job::{self, Engine, Packet}, }; use core::sync::atomic::Ordering; -use kernel::{dma_fence::Fence, drm_neo::sched, prelude::*, sync::Arc}; +use kernel::{dma_fence::Fence, drm::sched, prelude::*, sync::Arc}; impl Backend { pub(super) fn release_backlog(&self, packet: &Packet) { @@ -70,8 +70,12 @@ impl job::Backend for Backend { Some(self.shared.feed.clone()) } - fn ensure_compute(&self) -> Result { - Backend::ensure_compute(self) + fn render_scs_blocks(&self, pages: [u16; 2]) -> Result { + crate::g17::fw::render_rt::required_blocks(self.shared.active_cores, pages) + } + + fn ensure_compute(&self, scs_pages: u16) -> Result { + Backend::ensure_compute(self, scs_pages) } fn fail_vm(&self, error: Error) { diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/compute_control.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/compute_control.rs rename to drivers/gpu/drm/asahi/g17/runtime/compute_control.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/compute_exit.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_exit.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/compute_exit.rs rename to drivers/gpu/drm/asahi/g17/runtime/compute_exit.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/compute_replay.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_replay.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/compute_replay.rs rename to drivers/gpu/drm/asahi/g17/runtime/compute_replay.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/events.rs b/drivers/gpu/drm/asahi/g17/runtime/events.rs similarity index 91% rename from drivers/gpu/drm/asahi-neo/g17/runtime/events.rs rename to drivers/gpu/drm/asahi/g17/runtime/events.rs index 8944327b84c5c3..3ae1d0a609a335 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/events.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/events.rs @@ -13,6 +13,9 @@ use crate::g17::{ }; use kernel::{prelude::*, sync::Arc}; +/// Wake interval while a tiler heap may still shrink: the native check period. +const SHRINK_WAKE_NS: u64 = 1_000_000_000; + struct Service<'a> { firmware: &'a mut Firmware, deferred: &'a mut DeferredBatch, @@ -106,42 +109,6 @@ impl recovery::Host for Service<'_> { fn recovery_mut(&mut self) -> &mut recovery::State { &mut self.firmware.recovery } - fn report_recovery(&self, generation: u64, blamed: Option, sources: &recovery::Sources) { - dev_warn!( - self.firmware.primary.state.shared.dev.as_ref(), - "G17 firmware recovery: generation={} qid={:?} reason={:?} slots={:?}\n", - generation, - blamed, - sources.reason, - sources.diagnostics() - ); - let Some(qid) = blamed else { - return; - }; - for queue in self.firmware.queues.compute.iter().flatten() - .filter_map(|entry| entry.queue.as_deref()) - .filter(|queue| queue.qid() == qid) - { - dev_warn!( - self.firmware.primary.state.shared.dev.as_ref(), - "G17 recovery compute: qid={} (owner, context/generation, ordinal, active, cursors, stamp, CDM/samplers/scratch)={:?}\n", - qid, - queue.recovery_diagnostics() - ); - } - if let Some((qids, owner, context, generation)) = - self.firmware.queues.render_recovery_diagnostics(qid) - { - dev_warn!( - self.firmware.primary.state.shared.dev.as_ref(), - "G17 recovery render: qids={:?} owner={:?} context={} generation={}\n", - qids, - owner, - context, - generation - ); - } - } } impl recovery::Memory for Service<'_> { @@ -401,6 +368,7 @@ impl crate::g17::Shared { // Replay only at this outer boundary, after recovery has settled. // Publication may service events but must never recurse into replay. firmware.replay_compute(deferred)?; + firmware.replay_render(deferred)?; drop(state); self.observe_renders(masks, deferred)?; let mut state = self.state.lock(); @@ -422,11 +390,15 @@ impl crate::g17::Shared { let idle = firmware.idle_check(now, in_flight)?; firmware.reclaim_drained_renders(deferred)?; firmware.service_compute_exit(deferred)?; - firmware.service_compute_pools(deferred)?; + let pools = firmware.service_compute_pools(deferred, now)?; + let mut shrinking = false; if !firmware.recovery.pending() { firmware .queues .vacate_evicted_render_pools(&firmware.init, deferred)?; + shrinking = firmware + .queues + .shrink_render_heaps(&firmware.init, now, deferred); } { let mut reclaim = self.reclaim.lock(); @@ -438,6 +410,7 @@ impl crate::g17::Shared { let backpressured = firmware.render_control_backpressured(); let polling = backpressured || firmware.queues.replay_outer_wait.is_some() + || firmware.queues.render.replay_polling() || firmware.queues.teardown.polling(now) || firmware.compute_exit_polling(now) || firmware @@ -450,6 +423,18 @@ impl crate::g17::Shared { if polling { self.queue_poll(); } + // A heap with growth to give back is re-evaluated at least once per + // shrink period, also while the device is otherwise idle. + let idle = if shrinking { + Some(idle.map_or(SHRINK_WAKE_NS, |delay| delay.min(SHRINK_WAKE_NS))) + } else { + idle + }; + // A retained compute pool gives its backing back when it expires. + let idle = match (idle, pools) { + (Some(delay), Some(expiry)) => Some(delay.min(expiry)), + (delay, expiry) => delay.or(expiry), + }; if let Some(delay) = idle { self.queue_idle(delay); } diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/feed.rs b/drivers/gpu/drm/asahi/g17/runtime/feed.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/feed.rs rename to drivers/gpu/drm/asahi/g17/runtime/feed.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render.rs b/drivers/gpu/drm/asahi/g17/runtime/render.rs similarity index 89% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render.rs rename to drivers/gpu/drm/asahi/g17/runtime/render.rs index 71b21f98479a6c..a47abf20455261 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render.rs @@ -11,6 +11,7 @@ mod observe; mod pools; mod prepare; mod recovery; +mod replay; mod settle; mod submit; pub(super) mod teardown; @@ -647,6 +648,80 @@ impl Registry { || !self.render.retired.is_empty() || !self.teardown.is_empty() } + /// Applies the native tiler-heap shrink policy (`buffer::Manager:: + /// shrink_target`) to every grown heap at `now` and rebuilds the heaps of + /// idle pairs that are due, removing whole growth extensions. The removed + /// aliases and backing go to the worker's deferred batch, which frees them + /// after the device mutex. A shrink is best effort: any failure leaves the + /// heap as it was. Returns whether a heap still holds growth it could give + /// back, so the worker keeps a periodic wake. + pub(in crate::g17) fn shrink_render_heaps( + &mut self, + init: &crate::g17::initdata::InitData, + now: u64, + deferred: &mut super::DeferredBatch, + ) -> bool { + let mut pending = false; + for (slot, entry) in self.render.entries.iter_mut().enumerate() { + let Some(entry) = entry.as_mut() else { + continue; + }; + let borrowed = entry.constructing || entry.growing || entry.preparing.is_some(); + let Some(pair) = entry.pair.as_mut() else { + continue; + }; + if !pair.manager().grown() { + continue; + } + let busy = borrowed || !pair.idle_for_context(pair.context()).unwrap_or(false); + if let Ok(Some(target)) = pair.manager().shrink_target(now, busy) { + if deferred.tvb_room() { + if let Ok(slot) = u8::try_from(slot) { + // Ignoring the result keeps the heap unchanged on failure. + let _ = Self::shrink_pair(init, slot, pair, target, deferred); + } + } + } + pending |= pair.manager().shrinkable(); + } + pending + } + + fn shrink_pair( + init: &crate::g17::initdata::InitData, + slot: u8, + pair: &mut Pair, + target: usize, + deferred: &mut super::DeferredBatch, + ) -> Result { + use core::sync::atomic::{fence, Ordering}; + let offset = usize::from(pair.buffer_id()) * 16; + let table = init.pb_descriptor_table()?; + let words = [ + table.word(offset)?, + table.word(offset + 4)?, + table.word(offset + 8)?, + table.word(offset + 12)?, + ]; + fence(Ordering::Acquire); + let descriptor = words.each_ref().map(|word| word.load(Ordering::Relaxed)); + let vm = pair.context().vm().clone(); + let pool = pair.pool_id(); + let shrunk = pair + .manager() + .shrink(target, descriptor, |blocks, kept, out| { + vm.truncate_render_pool_mappings(slot, pool, blocks, kept, out) + })?; + if let Some((reloaded, release)) = shrunk { + for (word, value) in words.into_iter().zip(reloaded) { + word.store(value, Ordering::Relaxed); + } + fence(Ordering::SeqCst); + deferred.defer_tvb(release); + } + Ok(()) + } + /// Gives back the USC backing of render pools whose client VM pressure /// reclaim evicted (an idle VM) and whose final free-list release was /// consumed. The vacated backings go to the worker's deferred batch, which diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/dependencies.rs b/drivers/gpu/drm/asahi/g17/runtime/render/dependencies.rs similarity index 56% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/dependencies.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/dependencies.rs index 09e4322bf42cef..2b14b4548e0db8 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/dependencies.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/dependencies.rs @@ -31,4 +31,29 @@ impl Registry { } Ok(frontier) } + pub(in crate::g17::runtime) fn tiling_prefix( + &self, + owner: u64, + packet: &Arc, + ) -> Result { + let mut frontier = Frontier::new(); + for entry in self.render.entries.iter().flatten() { + if let Some(pair) = entry.pair.as_ref() { + pair.collect_tiling_dependencies( + owner, + packet.completion.status(), + packet.order.wait_through[0], + &mut frontier, + )?; + } else if let Some(dependencies) = entry.borrowed_dependencies.as_ref() { + dependencies.collect_tiling( + owner, + packet.completion.status(), + packet.order.wait_through[0], + &mut frontier, + )?; + } + } + Ok(frontier) + } } diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/exit.rs b/drivers/gpu/drm/asahi/g17/runtime/render/exit.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/exit.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/exit.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/grow.rs b/drivers/gpu/drm/asahi/g17/runtime/render/grow.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/grow.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/grow.rs index 0ab673b79be265..b283fb8e7d55ca 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/grow.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/grow.rs @@ -194,7 +194,7 @@ impl crate::g17::Shared { pub(in crate::g17) fn grow_render(&self) -> Result { loop { let lease = self.preparations.enter(false)?; - let dev = self.drm_neo()?; + let dev = self.drm()?; let (request, slot, mut pair) = { let mut state = self.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/observe.rs b/drivers/gpu/drm/asahi/g17/runtime/render/observe.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/observe.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/observe.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/pools.rs b/drivers/gpu/drm/asahi/g17/runtime/render/pools.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/pools.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/pools.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/prepare.rs b/drivers/gpu/drm/asahi/g17/runtime/render/prepare.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/prepare.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/prepare.rs index 0d1a3ec72093bc..06bea3c84fa7ad 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/prepare.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/prepare.rs @@ -74,7 +74,7 @@ struct BuildContext<'a> { impl super::super::Backend { fn build_render_pair(&self, slot: u8, first: &Arc, optional: bool) -> Result { let lease = self.shared.preparations.enter(false)?; - let dev = self.shared.drm_neo()?; + let dev = self.shared.drm()?; let (resources, build) = { let mut state = self.shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; @@ -585,7 +585,7 @@ impl super::super::Backend { deferred: &mut super::super::DeferredBatch, ) -> Result { let lease = self.shared.preparations.enter(false)?; - let dev = self.shared.drm_neo()?; + let dev = self.shared.drm()?; let (uat, mut pair, addresses, growth_target) = { let mut state = self.shared.state.lock(); let firmware = (*state).as_deref_mut().ok_or(ENODEV)?; diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/recovery.rs b/drivers/gpu/drm/asahi/g17/runtime/render/recovery.rs similarity index 68% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/recovery.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/recovery.rs index 7a897a7afca939..3a5b042fff6834 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/recovery.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/recovery.rs @@ -6,16 +6,6 @@ use crate::g17::runtime::Registry; use kernel::prelude::*; impl Registry { - pub(in crate::g17::runtime) fn render_recovery_diagnostics( - &self, - qid: u8, - ) -> Option<([u8; 2], Option, u32, u8)> { - let pair = self.render.entries.iter().flatten() - .filter_map(|entry| entry.pair.as_ref()) - .find(|pair| pair.qids().contains(&qid))?; - Some((pair.qids(), pair.owner(), pair.context().id(), pair.context().generation())) - } - pub(in crate::g17::runtime) fn render_pass_started(&self, qid: u8, stamp: u64) -> Option { self.render.entries.iter().flatten() .filter_map(|entry| entry.pair.as_ref()) diff --git a/drivers/gpu/drm/asahi/g17/runtime/render/replay.rs b/drivers/gpu/drm/asahi/g17/runtime/render/replay.rs new file mode 100644 index 00000000000000..ce9771778acd4d --- /dev/null +++ b/drivers/gpu/drm/asahi/g17/runtime/render/replay.rs @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT + +//! Republication of render groups that firmware retired without executing. +//! The pair proves the abandonment (retired, unlinked, stamps covering it, no +//! recovery, fragment end never written) and parks the packet; this pass +//! prepares and publishes the same command again under the device mutex. + +use super::super::DeferredBatch; +use crate::g17::{object::Allocator, queue::render::RenderAddresses, Firmware}; +use kernel::prelude::*; + +impl super::RenderState { + pub(in crate::g17) fn replay_polling(&self) -> bool { + self.entries + .iter() + .flatten() + .filter_map(|entry| entry.pair.as_ref()) + .any(|pair| pair.replay_polling()) + } +} + +impl Firmware { + pub(in crate::g17) fn replay_render(&mut self, deferred: &mut DeferredBatch) -> Result { + if self.recovery.pending() + || crate::g17::recovery::Memory::recovery_state(&self.init)? != 0 + { + return Ok(()); + } + for slot in 0..self.queues.render.entries.len() { + let Some((packet, owner, qids, closed, busy)) = + self.queues.render.entries[slot].as_ref().and_then(|entry| { + let pair = entry.pair.as_ref()?; + let packet = pair.replay_front()?; + Some(( + packet, + entry.reservation.owner, + pair.qids(), + entry.closed, + entry.preparing.is_some() + || entry.growing + || entry.pending_timeout + || entry.pending_error.is_some(), + )) + }) + else { + continue; + }; + if busy { + continue; + } + if closed || packet.cancel_requested() || !packet.context.is_current() { + self.with_render(slot, |pair, _| { + pair.fail_replay(ECANCELED, &mut |event| deferred.push(event)) + })?; + continue; + } + if packet.completion.status().get() != 0 { + self.with_render(slot, |pair, _| { + pair.fail_replay(EIO, &mut |event| deferred.push(event)) + })?; + continue; + } + let addresses = RenderAddresses { + stamps: [ + self.init.stamp_addresses(qids[0])?, + self.init.stamp_addresses(qids[1])?, + ], + }; + let dev = self.primary.state.shared.drm()?; + let uat = self.uat.clone(); + let prepared = self.with_render(slot, |pair, _| { + if !pair.room() { + return Err(EBUSY); + } + let alloc = Allocator { + dev: &dev, + uat: &uat, + }; + pair.prepare(&alloc, &packet, addresses) + }); + match prepared { + Ok(()) => {} + Err(error) if error == EBUSY || error == EAGAIN => continue, + Err(error) => { + self.with_render(slot, |pair, _| { + pair.fail_replay(error, &mut |event| deferred.push(event)) + })?; + continue; + } + } + match self.publish_render(owner, slot as u8, &packet, deferred) { + Ok(()) => { + self.with_render(slot, |pair, _| pair.finish_replay(&packet))?; + } + Err(error) if error == EBUSY || error == EAGAIN => continue, + Err(error) => { + self.with_render(slot, |pair, _| { + pair.fail_replay(error, &mut |event| deferred.push(event)) + })?; + } + } + } + Ok(()) + } +} diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/settle.rs b/drivers/gpu/drm/asahi/g17/runtime/render/settle.rs similarity index 98% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/settle.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/settle.rs index 415d1f391f601b..2e88848b932161 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/settle.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/settle.rs @@ -70,7 +70,7 @@ impl crate::g17::Firmware { .and_then(Option::as_ref) .and_then(|entry| entry.pair.as_ref()) .ok_or(EIO)?; - if !pair.contains_packet(packet) { + if pair.replay_contains(packet) || !pair.contains_packet(packet) { return Ok(()); } let generation = self.recovery.generation(); diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/submit.rs b/drivers/gpu/drm/asahi/g17/runtime/render/submit.rs similarity index 95% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/submit.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/submit.rs index dbf7175c5cd4bd..13381b62da86c5 100644 --- a/drivers/gpu/drm/asahi-neo/g17/runtime/render/submit.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render/submit.rs @@ -187,6 +187,19 @@ impl Registry { .ok_or(EOVERFLOW)? = *dependency; dependencies.render_count += 1; } + if dependencies.render_count != 0 + && matches!(&packet.command, crate::g17::command::Validated::Render { pass, .. } + if pass.fragment_barriers) + { + let tiling = self.tiling_prefix(owner, packet)?; + for dependency in tiling.as_slice() { + *dependencies + .tiling + .get_mut(dependencies.tiling_count) + .ok_or(EOVERFLOW)? = *dependency; + dependencies.tiling_count += 1; + } + } if let Some(queue) = self.compute(owner) { dependencies.compute = queue.dependency( owner, diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/render/teardown.rs b/drivers/gpu/drm/asahi/g17/runtime/render/teardown.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/render/teardown.rs rename to drivers/gpu/drm/asahi/g17/runtime/render/teardown.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/runtime/teardown.rs b/drivers/gpu/drm/asahi/g17/runtime/teardown.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/runtime/teardown.rs rename to drivers/gpu/drm/asahi/g17/runtime/teardown.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/status.rs b/drivers/gpu/drm/asahi/g17/status.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/status.rs rename to drivers/gpu/drm/asahi/g17/status.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/teardown.rs b/drivers/gpu/drm/asahi/g17/teardown.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/teardown.rs rename to drivers/gpu/drm/asahi/g17/teardown.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/timeout.rs rename to drivers/gpu/drm/asahi/g17/timeout.rs diff --git a/drivers/gpu/drm/asahi-neo/g17/validation.rs b/drivers/gpu/drm/asahi/g17/validation.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/g17/validation.rs rename to drivers/gpu/drm/asahi/g17/validation.rs diff --git a/drivers/gpu/drm/asahi/gem.rs b/drivers/gpu/drm/asahi/gem.rs index 2b8f312b28d46e..58eed45bf43c77 100644 --- a/drivers/gpu/drm/asahi/gem.rs +++ b/drivers/gpu/drm/asahi/gem.rs @@ -13,7 +13,8 @@ use kernel::{ shmem, shmem::VMap, BaseObject, - DriverObject, // + DriverObject, + IntoGEMObject, // }, error::Result, prelude::*, @@ -24,6 +25,7 @@ use kernel::{ use core::ops::Range; use core::sync::atomic::{ AtomicU64, + AtomicUsize, Ordering, // }; @@ -44,6 +46,8 @@ const DEBUG_CLASS: DebugFlags = DebugFlags::Gem; pub(crate) struct AsahiObject { /// ID for debug id: u64, + /// Object size retained for live-object accounting. + size: usize, /// Object creation flags. flags: u32, /// Whether this object can be exported. @@ -51,11 +55,69 @@ pub(crate) struct AsahiObject { /// Whether this is a kernel-created object. kernel: bool, _allocation: Option, + /// Driver aliases outside the user GPUVA tree; they pin the backing. + kernel_mappings: AtomicU64, } /// Type alias for the shmem GEM object type for this driver. pub(crate) type Object = shmem::Object; +/// Marks whether the pages of `gem` are held for the device. +/// +/// Populated asahi objects keep their pages referenced (device SG table, kernel vmap or a PFN +/// CPU mapping), so the pages cannot migrate. Unmarked, the compaction migrate scanner still +/// isolates them from the unevictable LRU and every migration fails. The shmem mapping of a +/// held object is therefore marked inaccessible, which compaction and migration check and which +/// comes with unevictable, as `mapping_set_inaccessible()` sets it; nothing reads or writes a GEM +/// shmem file through its mapping. Releasing idle backing or purging clears the mark, so the +/// released pages can be swapped and migrated again. +pub(crate) fn set_mapping_held(gem: &Object, held: bool) { + let raw = gem.as_raw(); + // SAFETY: `gem` keeps the GEM object alive. A shmem-backed object's `filp` and its + // `f_mapping` are set at initialization and never change; imported objects have none. + let mapping = unsafe { + let filp = (*raw).filp; + if filp.is_null() { + return; + } + (*filp).f_mapping + }; + if mapping.is_null() { + return; + } + // SAFETY: `flags` is the aligned flag word of the live mapping. The kernel updates it only + // with atomic bit operations on the same word. + let flags = + unsafe { AtomicUsize::from_ptr(core::ptr::addr_of_mut!((*mapping).flags).cast::()) }; + let inaccessible = 1usize << kernel::bindings::mapping_flags_AS_INACCESSIBLE; + let unevictable = 1usize << kernel::bindings::mapping_flags_AS_UNEVICTABLE; + if held { + flags.fetch_or(inaccessible | unevictable, Ordering::Relaxed); + } else { + flags.fetch_and(!inaccessible, Ordering::Relaxed); + } +} + +/// Takes an owned SG table of `gem`, which holds its pages for the device, and marks the +/// object held (see [`set_mapping_held`]). +pub(crate) fn held_sg_table(gem: &Object) -> Result> { + let sgt = gem.owned_sg_table()?; + set_mapping_held(gem, true); + Ok(sgt) +} + +/// VM-private BOs share their assigned VM's reservation object. Validate this +/// before acquiring mapping state; generic GPUVM accepts external shareable BOs. +pub(crate) fn validate_vm_binding(gem: &Object, vm: &mmu::Vm) -> Result { + if gem.flags & uapi::drm_asahi_gem_flags_DRM_ASAHI_GEM_VM_PRIVATE != 0 + && vm.is_extobj(gem) + { + return Err(EINVAL); + } + Ok(()) +} + + unsafe impl Send for AsahiObject {} unsafe impl Sync for AsahiObject {} @@ -167,6 +229,10 @@ pub(crate) struct AsahiObjConfig { flags: u32, exportable: bool, kernel: bool, + /// Created by this driver rather than by the DRM core for a dma-buf import or a dumb + /// buffer. Only driver-created objects enter the M3 memory accounting, as before core + /// objects were allocated as asahi objects. + driver_created: bool, } /// Validate before rounding: wrapping a user size can create a much smaller @@ -189,6 +255,7 @@ fn new_kernel_object_with_cpu_mapping( checked_object_size(size)?, shmem::ObjectConfig:: { map_wc, + reclaimable_cpu_mappings: false, parent_resv_obj: None, synchronous_reclaim: dev.is_m3, }, @@ -196,9 +263,15 @@ fn new_kernel_object_with_cpu_mapping( flags: 0, exportable: false, kernel: true, + driver_created: true, }, )?; + // Only the G17P runtime marks its kernel objects held (see `set_mapping_held`). + if dev.hold_kernel_pages { + set_mapping_held(&gem, true); + } + mod_pr_debug!("AsahiObject new kernel object id={}\n", gem.id); Ok(ObjectRef::new(gem)) } @@ -218,6 +291,7 @@ pub(crate) fn new_object( size: usize, flags: u32, parent_object: Option<&shmem::Object>, + reclaim_cpu: bool, ) -> Result> { if (flags & uapi::drm_asahi_gem_flags_DRM_ASAHI_GEM_VM_PRIVATE != 0) != parent_object.is_some() { @@ -228,6 +302,7 @@ pub(crate) fn new_object( dev, checked_object_size(size)?, shmem::ObjectConfig:: { + reclaimable_cpu_mappings: parent_object.is_some() && reclaim_cpu, map_wc: flags & uapi::drm_asahi_gem_flags_DRM_ASAHI_GEM_WRITEBACK == 0, parent_resv_obj: parent_object, synchronous_reclaim: dev.is_m3, @@ -236,6 +311,7 @@ pub(crate) fn new_object( flags, exportable: parent_object.is_none(), kernel: false, + driver_created: true, }, )?; @@ -256,12 +332,25 @@ impl DriverObject for AsahiObject { mod_pr_debug!("AsahiObject::new id={}\n", id); try_pin_init!(AsahiObject { id, + _allocation: if dev.is_m3 && args.driver_created { + Some(crate::agx_memory_stats::Allocation::gem(size, args.kernel)?) + } else { None }, + size, flags: args.flags, exportable: args.exportable, kernel: args.kernel, - _allocation: if dev.is_m3 { - Some(crate::agx_memory_stats::Allocation::gem(size, args.kernel)?) - } else { None }, + kernel_mappings: AtomicU64::new(0), + }) + } + + /// Objects the DRM core allocates for us (dma-bufs imported from other devices and dumb + /// buffers) are shared user objects, like a GEM_CREATE object without flags. + fn core_object_args(_dev: &AsahiDevice, _size: usize) -> Result { + Ok(AsahiObjConfig { + flags: 0, + exportable: true, + kernel: false, + driver_created: false, }) } @@ -286,3 +375,46 @@ impl DriverObject for AsahiObject { obj.prime_export(flags) } } + +/// A driver-created alias of a GEM object, possibly in a root the user GPUVA +/// tree does not describe. It keeps the object out of idle reclaim. +pub(crate) struct KernelMappingPin { + object: ARef, +} + +impl KernelMappingPin { + pub(crate) fn new(object: &Object) -> Self { + object.kernel_mappings.fetch_add(1, Ordering::AcqRel); + Self { + object: object.into(), + } + } +} + +impl core::ops::Deref for KernelMappingPin { + type Target = Object; + fn deref(&self) -> &Object { + &self.object + } +} + +impl Drop for KernelMappingPin { + fn drop(&mut self) { + self.object.kernel_mappings.fetch_sub(1, Ordering::Release); + } +} + +impl AsahiObject { + pub(crate) fn idle_reclaim_candidate(&self) -> bool { + !self.kernel && !self.exportable && self.kernel_mappings.load(Ordering::Acquire) == 0 + } +} + +/// Purge advice for a VM-private, non-exportable object. Returns whether the +/// contents are retained (0 once purged). +pub(crate) fn madvise(gem: &Object, dontneed: bool) -> Result { + if gem.flags & uapi::drm_asahi_gem_flags_DRM_ASAHI_GEM_VM_PRIVATE == 0 || gem.exportable { + return Err(EINVAL); + } + Ok(u32::from(gem.madvise(i32::from(dontneed)))) +} diff --git a/drivers/gpu/drm/asahi/gpu.rs b/drivers/gpu/drm/asahi/gpu.rs index bb9492270b7a90..1f87dfa479cab1 100644 --- a/drivers/gpu/drm/asahi/gpu.rs +++ b/drivers/gpu/drm/asahi/gpu.rs @@ -552,6 +552,65 @@ pub(crate) struct GpuManager { garbage_contexts: Mutex>>>, } +/// Operations the G17P runtime (`g17::Gpu`) provides to the DRM driver. +/// +/// `drm_gpu` adapts it to [`crate::drm_gpu::DrmGpu`]; the G13/G14 and M3 runtimes use that +/// trait directly. +pub(crate) trait Gpu: Send + Sync { + // `g17::Gpu` implements these three; the DRM layer selects that runtime through + // `drm_gpu::Backend::G17` instead of downcasting, so nothing calls them. + /// Cast as an Any type. + #[allow(dead_code)] + fn as_any(&self) -> &dyn Any; + /// Cast Arc as an Any type. + #[allow(dead_code)] + fn arc_as_any(self: Arc) -> Arc; + /// Return the G13/G14 firmware manager, if this GPU uses that firmware interface. + #[allow(dead_code)] + fn manager(&self) -> Option<&dyn GpuManager>; + /// Initialize the GPU. + fn init(&self) -> Result; + /// Return a reference to the global `SequenceIDs` instance. + fn ids(&self) -> &SequenceIDs; + /// Check whether the GPU is crashed + fn is_crashed(&self) -> bool; + /// Fill in the hardware description reported to userspace. + fn get_params(&self, params: &mut kernel::uapi::drm_asahi_params_global) -> Result; + /// Whether user VMs retain a sticky error and support a userspace mirror. + fn supports_vm_status(&self) -> bool { + false + } + /// Limits for independently executing queues, if supported by this GPU. + fn queue_limits(&self) -> Option { + None + } + /// Temporary capacity hint and validity in milliseconds for subsequent syncobj waits. + fn syncobj_wait_hint(&self) -> Option<(u32, u32)> { + None + } + /// Frequency of the CPU timer that GPU timestamps are derived from. + fn base_clock_hz(&self) -> u32; + /// Return the GPU address space geometry. + fn uat_geometry(&self) -> mmu::UatGeometry; + /// Create a new `Vm` with the given range reserved for kernel-managed objects. + fn new_vm(&self, kernel_range: Range) -> Result; + /// Create a new user command queue. + fn new_queue( + &self, + vm: mmu::Vm, + ualloc: Arc>, + ualloc_priv: Arc>, + priority: u32, + usc_exec_base: u64, + ) -> Result>; + /// Map a BO as a timestamp buffer + fn map_timestamp_buffer( + &self, + bo: gem::ObjectRef, + range: Range, + ) -> Result; +} + /// Trait used to abstract the firmware/GPU-dependent variants of the GpuManager. pub(crate) trait GpuManager: Send + Sync { /// Cast as an Any type. @@ -1460,7 +1519,7 @@ impl GpuManager::ver { let map_kernel_to_user = false; Ok(KBox::new( - mmu::Uat::new(dev, cfg, map_kernel_to_user)?, + mmu::Uat::new_for_config(dev, cfg, map_kernel_to_user)?, GFP_KERNEL, )?) } diff --git a/drivers/gpu/drm/asahi/hw/mod.rs b/drivers/gpu/drm/asahi/hw/mod.rs index dab1274d80dfce..c90156bd1244bd 100644 --- a/drivers/gpu/drm/asahi/hw/mod.rs +++ b/drivers/gpu/drm/asahi/hw/mod.rs @@ -19,6 +19,7 @@ pub(crate) mod t6031; pub(crate) mod t8103; pub(crate) mod t8112; pub(crate) mod t8122; +pub(crate) mod t8140; /// GPU revision enumeration. Note: Part of the UABI. #[derive(Debug, PartialEq, Copy, Clone)] diff --git a/drivers/gpu/drm/asahi-neo/hw/t8140.rs b/drivers/gpu/drm/asahi/hw/t8140.rs similarity index 98% rename from drivers/gpu/drm/asahi-neo/hw/t8140.rs rename to drivers/gpu/drm/asahi/hw/t8140.rs index 9772d318ea819a..143ba720d06f22 100644 --- a/drivers/gpu/drm/asahi-neo/hw/t8140.rs +++ b/drivers/gpu/drm/asahi/hw/t8140.rs @@ -196,8 +196,9 @@ pub(crate) mod low { pub(crate) const RENDER_BLOCKS: u64 = 0x70_0022_0000; /// Distance between consecutive blocks. pub(crate) const BLOCK_STRIDE: u64 = 0x10_8000; - /// Render blocks allocated at boot. - pub(crate) const RENDER_BLOCKS_ALLOCATED: usize = 28; + /// Render blocks allocated at boot: the mapped blocks and the block that + /// holds the compute page list. + pub(crate) const RENDER_BLOCKS_ALLOCATED: usize = COMPUTE_PAGE_LIST_BLOCK + 1; /// Render blocks mapped at boot. pub(crate) const RENDER_BLOCKS_MAPPED: usize = 22; /// Render blocks in the free list at boot. @@ -383,9 +384,9 @@ pub(crate) mod qos { pub(crate) mod scheduling { use crate::g17::fw::queue::Policy; - /// `DRM_ASAHI_NEO_PRIORITY_MEDIUM`. + /// `DRM_ASAHI_PRIORITY_MEDIUM`. pub(crate) const MEDIUM: Policy = Policy::new(2, 2, super::qos::CLASS, super::qos::SHARE); - /// `DRM_ASAHI_NEO_PRIORITY_LOW`. + /// `DRM_ASAHI_PRIORITY_LOW`. pub(crate) const LOW: Policy = Policy::new(3, 2, super::qos::CLASS, super::qos::SHARE); } diff --git a/drivers/gpu/drm/asahi/identity.rs b/drivers/gpu/drm/asahi/identity.rs index eb62a32da07dc8..f468272c5680ed 100644 --- a/drivers/gpu/drm/asahi/identity.rs +++ b/drivers/gpu/drm/asahi/identity.rs @@ -8,6 +8,7 @@ pub(crate) enum GpuGen { G13 = 13, G14 = 14, G15 = 15, + G17 = 17, } /// GPU variant enumeration. Note: Part of the UABI. diff --git a/drivers/gpu/drm/asahi/m3_submit.rs b/drivers/gpu/drm/asahi/m3_submit.rs index f1c6e561b9aeb8..9bd08d0257b0c9 100644 --- a/drivers/gpu/drm/asahi/m3_submit.rs +++ b/drivers/gpu/drm/asahi/m3_submit.rs @@ -84,7 +84,7 @@ impl Packet { // Claim error/signal ownership once; this flag is NOT proof that the // fence is signaled or that hardware DMA has stopped. if self.finish_claimed.swap(true, Ordering::Relaxed) { return; } - if let Err(e)=result {if let Ok(status) = self.vm.status() { status.record(e.to_errno()); }self.completion.set_error(e);} + if let Err(e)=result {if let Ok(status) = self.vm.agx_status() { status.record(e.to_errno()); }self.completion.set_error(e);} // A scheduler job/fence and the runtime's last packet can outlive // hardware retirement. They must not keep the VM's active-job count // nonzero: GEM closes otherwise accumulate across completed frames. @@ -103,8 +103,14 @@ impl Packet { } } pub(crate) struct Job {shared:Shared,packet:Arc} +/// Timeout state of a [`Job`], owned by the scheduler's timeout worker. +pub(crate) struct JobTimeout {shared:Shared,packet:Arc} impl sched::JobImpl for Job { - fn run(job:&mut sched::Job)->Result> { + type TimeoutData = JobTimeout; + fn timeout_data(&self)->JobTimeout { + JobTimeout {shared:self.shared.clone(),packet:self.packet.clone()} + } + fn run(job:&mut Self)->Result> { // Queue the packet and return its fence; the completion worker publishes and retires // it, so the scheduler can hand over further ready jobs meanwhile. if let Err(e)=crate::m3_runtime::Runtime::submit(&job.shared,job.packet.clone()) { @@ -114,7 +120,7 @@ impl sched::JobImpl for Job { } Ok(Some(Fence::from_fence(&job.packet.completion))) } - fn timed_out(job:&mut sched::Job)->sched::Status { + fn timed_out(job:&mut JobTimeout,_finished:bool)->sched::Status { // Retain the publishing branch's explicit diagnostic override. if !crate::m3_params::timeout_nohang() { if let Some(r)=Option::as_mut(&mut *job.shared.lock()) {r.health().mark_failed();} @@ -136,7 +142,7 @@ impl sched::JobImpl for Job { job.packet.finish(Err(EIO)); sched::Status::NoDevice } - fn cancel(job:&mut sched::Job) { + fn cancel(job:&mut Self) { crate::m3_runtime::Runtime::withdraw(&job.shared,&job.packet); job.packet.finish(Err(ECANCELED)); } diff --git a/drivers/gpu/drm/asahi/mem.rs b/drivers/gpu/drm/asahi/mem.rs index 60a64e23a161c5..f354e4bb1f71ab 100644 --- a/drivers/gpu/drm/asahi/mem.rs +++ b/drivers/gpu/drm/asahi/mem.rs @@ -11,6 +11,8 @@ use core::arch::asm; use core::cmp::min; +use kernel::page::PAGE_SIZE; + use crate::debug::*; use crate::mmu; @@ -134,6 +136,66 @@ pub(crate) fn tlbi_range(asid: Asid, va: usize, len: usize) { } } +/// Range length from which [`tlbi_range_or_asid`] invalidates the whole ASID. +const TLBI_RANGE_MAX: usize = 0x200_0000; +/// Range length from which [`tlbi_range_or_asid`] counts in 1 MiB units (scale 1) instead of +/// 32 KiB units (scale 0). +const TLBI_RANGE_SCALE1_MIN: usize = 0x10_0000; +/// Translation granule field value for 16 KiB pages. +const TLBI_TG_16K: u64 = 2; +/// Width of the base address field of a range TLBI operand. +const TLBI_RANGE_BADDR_MASK: u64 = (1 << 38) - 1; + +/// Invalidate a range of pages for a given ASID with a single range operation. +/// +/// Unlike [`tlbi_range`], which issues as many range operations as needed, this invalidates +/// ranges that do not fit one operation (and ranges starting at VA 0) by ASID. Single pages use a +/// page operation, at the CPU page size. +#[inline(always)] +pub(crate) fn tlbi_range_or_asid(asid: Asid, va: usize, len: usize) { + if debug_enabled(DebugFlags::ConservativeTlbi) { + tlbi_all(); + sync(); + return; + } + + if len == 0 { + return; + } + + if va == 0 || len >= TLBI_RANGE_MAX { + tlbi_asid(asid); + return; + } + if len <= PAGE_SIZE { + tlbi_page(asid, va); + return; + } + + // The range covers (NUM + 1) << (5 * SCALE + 1) granules of 16 KiB. + let (scale, unit_shift) = if len >= TLBI_RANGE_SCALE1_MIN { + (1u64, 20) + } else { + (0u64, 15) + }; + let num = ((len - 1) >> unit_shift) as u64; + + let val = ((asid as u64) << 48) + | (TLBI_TG_16K << 46) + | (scale << 44) + | (num << 39) + | (((va >> mmu::UAT_PGBIT) as u64) & TLBI_RANGE_BADDR_MASK); + + // SAFETY: tlbi is always safe by definition + unsafe { + asm!( + ".arch armv8.4-a", + "tlbi rvae1os, {x}", + x = in(reg) val + ); + } +} + /// Issue a memory barrier (`dsb sy`). #[inline(always)] pub(crate) fn sync() { diff --git a/drivers/gpu/drm/asahi/mmu.rs b/drivers/gpu/drm/asahi/mmu.rs index e39a328f5394d4..06acbcef391ce6 100644 --- a/drivers/gpu/drm/asahi/mmu.rs +++ b/drivers/gpu/drm/asahi/mmu.rs @@ -9,6 +9,12 @@ //! unmap buffer objects into a single user or kernel address space. //! //! The actual page table management is in the `pt` module. +//! +//! G17 firmware does not take part in the handoff protocol. Instead, each of its two coprocessor +//! instances publishes the top-level entries it owns into its own kernel root while it boots, and +//! TTBAT contexts other than a VM's bind slot may name a VM's root. See [`UatFirmware`]. G17 UATs +//! and their VMs ("published-roots" UATs) run the G17 paths of this module; every other UAT keeps +//! the handoff/M3/J613 paths. use core::fmt::Debug; use core::mem::{size_of, ManuallyDrop}; @@ -69,7 +75,7 @@ use crate::{ mem, pgtable, slotalloc, - uat::UatGeometry, + uat::UatGeometry as RootGeometry, util::RangeExt, // }; @@ -83,18 +89,34 @@ pub(crate) use pgtable::{ UatPageTable, UAT_PGBIT, UAT_PGMSK, - UAT_PGSZ, // + UAT_PGSZ, + UAT_TOP_LEVEL_SHIFT, // }; use pin_init; const DEBUG_CLASS: DebugFlags = DebugFlags::Mmu; +mod bind; +mod context; +mod shared; +mod lifetime; +mod residency; +pub(crate) use residency::{Gate as ResidencyGate, VmShrinker}; +pub(crate) use bind::PreparedUserBindBatch; +pub(crate) use lifetime::VmJobGuard; + /// PPL magic number for the handoff region const PPL_MAGIC: u64 = 0x4b1d000000000002; /// Number of supported context entries in the TTBAT const UAT_NUM_CTX: usize = 64; + +/// Application contexts owned by independent logical queues. Contexts 0..4 +/// and 63 remain reserved for the firmware and device-global resources. +const EXECUTION_CONTEXT_MASK: u64 = ((1u64 << 63) - 1) & !((1u64 << 5) - 1); +/// Maximum simultaneously retained logical queue contexts. +pub(crate) const MAX_EXECUTION_CONTEXTS: u32 = EXECUTION_CONTEXT_MASK.count_ones(); /// First context available for users const UAT_USER_CTX_START: usize = 1; /// Number of available user contexts @@ -166,6 +188,18 @@ impl UatConfig { }) } + /// The UAT of a published-roots (G17) GPU. No chip-specific owner applies to it. + fn published(geometry: UatGeometry) -> Result { + let ias = u8::try_from(geometry.ias)?; + RootGeometry::new(ias).ok_or(ENODEV)?; + Ok(Self { + chip_id: 0, + ias, + oas: geometry.oas, + reserved_tables: false, + }) + } + /// Whether an M3 owner holds this UAT: the M3 runtime (reserved tables) on any SoC, or the /// G15 manager backend, which runs on T6030 only. These owners look their reserved regions /// up through `m3_resources`, track faults and VM status, and keep the AP side of the @@ -175,10 +209,10 @@ impl UatConfig { } } -fn uat_geometry(cfg: UatConfig) -> Result { +fn uat_geometry(cfg: UatConfig) -> Result { // 39-bit (G13/G14) and 42-bit (G15+) roots are the two recovered // geometries; anything else is undefined. - UatGeometry::new(cfg.ias).ok_or(ENODEV) + RootGeometry::new(cfg.ias).ok_or(ENODEV) } /// Lower/user VA range for this SoC's root geometry (one page up to the @@ -208,7 +242,7 @@ const IOVA_TTBR1_BASE_39: u64 = 0xffffff8000000000; /// Returns the upper/kernel (TTBR1) base VA for this GPU's root geometry. pub(crate) fn iova_ttbr1_base(cfg: &hw::HwConfig) -> u64 { - UatGeometry::new(cfg.uat_ias) + RootGeometry::new(cfg.uat_ias) .map(|g| g.upper_canonical_base()) .unwrap_or(IOVA_TTBR1_BASE_39) } @@ -235,6 +269,101 @@ const TTBR_ASID_SHIFT: usize = 48; //const IOVA_UNK_PAGE: u64 = 0x6f_ffff8000; pub(crate) const IOVA_UNK_PAGE: u64 = IOVA_USER_TOP - 2 * UAT_PGSZ as u64; +/// Top-level page table entry of the upper half that holds the driver-managed kernel VA range. +const IOVA_KERN_TOP_LEVEL_INDEX: u64 = 2; + +/// TTBAT context that starts out as an alias of the kernel context (with its own ASID) with +/// [`UatFirmware::PublishedRoots`]. +const KERNEL_ALIAS_CTX: usize = 1; + +/// How the GPU firmware shares UAT management with the driver. +#[derive(Debug, Copy, Clone, PartialEq, Eq)] +pub(crate) enum UatFirmware { + /// One coprocessor that publishes its magic number in the handoff region, arbitrates the + /// handoff lock with the driver, and acknowledges cache flush requests (G13/G14). + Handoff, + /// Two coprocessor instances that never touch the handoff region (G17). + /// + /// Each instance has its own kernel root and writes the top-level entries below and including + /// the driver-managed kernel window into it while it boots: the lower ones are private to the + /// instance, and the kernel window entry points at a table that the bootloader reserved. The + /// primary instance uses the `pagetables` root, the secondary one the root at + /// `secondary_root_offset` in the same region. TTBAT context 1 starts as an alias of the kernel + /// context. + PublishedRoots { + /// Offset of the secondary instance's kernel root in the `pagetables` region. + secondary_root_offset: u64, + }, +} + +/// UAT address space geometry of a given SoC. +#[derive(Debug, Copy, Clone)] +pub(crate) struct UatGeometry { + /// Input address bits translated by each page table root. + pub(crate) ias: u32, + /// Output (physical) address bits. + pub(crate) oas: u32, +} + +impl UatGeometry { + /// Lower/user top VA + const fn user_top(&self) -> u64 { + 1 << self.ias + } + + /// Lower/user VA range + pub(crate) const fn user_range(&self) -> Range { + IOVA_USER_BASE..self.user_top() + } + + /// Address of a special dummy page? + pub(crate) const fn unk_page(&self) -> u64 { + self.user_top() - 2 * UAT_PGSZ as u64 + } + + /// User VA range excluding the unk page + pub(crate) const fn user_usable_range(&self) -> Range { + IOVA_USER_BASE..self.unk_page() + } + + /// Upper/kernel base VA + const fn upper_base(&self) -> u64 { + !(self.user_top() - 1) + } + + /// Driver-managed kernel VA range + pub(crate) const fn kernel_range(&self) -> Range { + let start = self.upper_base() + (IOVA_KERN_TOP_LEVEL_INDEX << UAT_TOP_LEVEL_SHIFT); + start..(start + (1 << UAT_TOP_LEVEL_SHIFT)) + } + + /// Full kernel VA range + #[cfg(CONFIG_DEV_COREDUMP)] + const fn upper_range(&self) -> Range { + self.upper_base()..(!UAT_PGMSK as u64) + } +} + +// The G17 geometry (`UatGeometry`) and the root geometry used by the other UATs +// (`RootGeometry`) describe the same windows for the two supported root widths. +const _: () = { + let ias = [39u32, 42]; + let mut i = 0; + while i < ias.len() { + let g = UatGeometry { ias: ias[i], oas: 42 }; + let r = match RootGeometry::new(ias[i] as u8) { + Some(r) => r, + None => panic!("unsupported root geometry"), + }; + assert!(g.user_top() == r.user_va_top()); + assert!(g.upper_base() == r.upper_canonical_base()); + assert!(g.kernel_range().start == r.kernel_va_base()); + assert!(g.kernel_range().end == r.kernel_va_top()); + i += 1; + } + assert!(UatGeometry { ias: 39, oas: 40 }.unk_page() == IOVA_UNK_PAGE); +}; + /// A pre-allocated memory region for UAT management struct UatRegion { base: PhysicalAddr, @@ -304,6 +433,44 @@ const SLOTS_SIZE: usize = UAT_NUM_CTX * size_of::(); // We need at least page 0 (ttb0) const PAGETABLES_SIZE: usize = UAT_PGSZ; +/// Lock-free visibility of the mapping epoch. Writers run under the GPUVM +/// exec lock; nested mutations keep cached validation unavailable until the +/// outermost mutation has finished its leaf writes and invalidations. +struct MappingEpoch { + visible: AtomicU64, + pending: AtomicU64, + writers: AtomicU32, +} + +impl MappingEpoch { + fn new() -> Self { + Self { + visible: AtomicU64::new(0), + pending: AtomicU64::new(0), + writers: AtomicU32::new(0), + } + } + fn snapshot(&self) -> Option { + let value = self.visible.load(Ordering::Acquire); + (value != u64::MAX).then_some(value) + } +} + +#[must_use = "retain until the mapping mutation and its invalidations finish"] +struct MappingMutation(Option>); + +impl Drop for MappingMutation { + fn drop(&mut self) { + if let Some(epoch) = self.0.as_ref() { + if epoch.writers.fetch_sub(1, Ordering::Relaxed) == 1 { + epoch + .visible + .store(epoch.pending.load(Ordering::Relaxed), Ordering::Release); + } + } + } +} + /// Inner data for a Vm instance. This is reference-counted by the outer Vm object. struct VmInner { dev: driver::AsahiDevRef, @@ -312,6 +479,9 @@ struct VmInner { fixed_slot: Option, va_range: Range, page_table: UatPageTable, + /// Mapping epoch of a published-roots (G17) VM; `None` on every other VM. + mapping_epoch: Option>, + epoch: Option, mm: mm::Allocator<(), KernelMappingInner>, uat_inner: Arc, binding: Arc>, @@ -349,6 +519,13 @@ impl gpuvm::DriverGpuVmBo for VmBo { } } +/// Per-mapping metadata that survives backing eviction and GPUVA splits. +#[derive(Default, Clone)] +struct VmGpuVa { + prot: Option, +} +impl gpuvm::DriverGpuVa for VmGpuVa {} + #[derive(Default)] struct StepContext { new_va: Option>>>, @@ -361,6 +538,7 @@ struct StepContext { impl gpuvm::DriverGpuVm for VmInner { type Driver = driver::AsahiDriver; type GpuVmBo = VmBo; + type GpuVa = VmGpuVa; type StepContext = StepContext; fn step_map( @@ -368,6 +546,9 @@ impl gpuvm::DriverGpuVm for VmInner { op: &mut gpuvm::OpMap, ctx: &mut Self::StepContext, ) -> Result { + if self.published() { + return Self::step_map_published(self, op, ctx); + } if self.uat_inner.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { return Err(EIO); } let mut iova = op.addr(); let mut left = op.range() as usize; @@ -488,6 +669,9 @@ impl gpuvm::DriverGpuVm for VmInner { op: &mut gpuvm::OpUnMap, _ctx: &mut Self::StepContext, ) -> Result { + if self.published() { + return Self::step_unmap_published(self, op, _ctx); + } if self.uat_inner.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { return Err(EIO); } let va = op.va().expect("step_unmap: missing VA"); @@ -520,6 +704,9 @@ impl gpuvm::DriverGpuVm for VmInner { vm_bo: &gpuvm::GpuVmBo, ctx: &mut Self::StepContext, ) -> Result { + if self.published() { + return Self::step_remap_published(self, op, vm_bo, ctx); + } if self.uat_inner.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { return Err(EIO); } let va = op.unmap().va().expect("No previous VA"); let orig_addr = va.addr(); @@ -595,202 +782,629 @@ impl gpuvm::DriverGpuVm for VmInner { } impl VmInner { - /// Returns the slot index, if this VM is bound. - fn slot(&self) -> Option { - if let Some(slot) = self.fixed_slot { return Some(slot); } - if self.is_kernel { - // The GFX ASC does not care about the ASID. Pick an arbitrary one. - // TODO: This needs to be a persistently reserved ASID once we integrate - // with the ARM64 kernel ASID machinery to avoid overlap. - Some(0) - } else { - // We don't check whether we lost the slot, which could cause unnecessary - // invalidations against another Vm. However, this situation should be very - // rare (e.g. a Vm lost its slot, which means 63 other Vms bound in the - // interim, and then it gets killed / drops its mappings without doing any - // final rendering). Anything doing active maps/unmaps is probably also - // rendering and therefore likely bound. - self.binding - .lock() - .bind_token - .as_ref() - .map(|token| token.last_slot() + UAT_USER_CTX_START as u32) - } - } - - /// Returns the translation table base for this Vm - fn ttb(&self) -> u64 { - self.page_table.ttb() - } - - /// Check that a byte range is fully mapped with the requested GPU access. - /// The caller holds the VM execution lock. - fn covers_mapped_range( - &mut self, - address: u64, - size: u64, - need_read: bool, - need_write: bool, - ) -> bool { - if size == 0 { - return false; - } - let Some(end) = address.checked_add(size) else { - return false; - }; - let page_mask = UAT_PGMSK as u64; - let start = address & !page_mask; - let aligned_end = if end & page_mask == 0 { - end - } else { - match (end | page_mask).checked_add(1) { - Some(end) => end, - None => return false, - } - }; - if start < self.va_range.start || aligned_end > self.va_range.end { - return false; - } - self.page_table - .covers_range(start..aligned_end, need_read, need_write) - .unwrap_or(false) + /// Whether this VM belongs to a published-roots (G17) UAT. + fn published(&self) -> bool { + self.uat_inner.firmware != UatFirmware::Handoff } - /// Copy `out.len()` bytes at `iova` from the pages this VM maps, through - /// the kernel's linear map. `before_copy` receives the kernel address and - /// length of each chunk, which never crosses a page, before it is copied. - /// - /// The caller holds the VM execution lock for the whole call. Every PTE - /// change takes that lock, and a page leaves the page table before the GEM - /// reference that keeps it allocated is released, so a concurrent unbind - /// or GEM close cannot free a page while it is being read. - fn copy_from_mapped_pages( - &mut self, - iova: u64, - out: &mut [u8], - mut before_copy: impl FnMut(*mut u8, usize), + /// `DriverGpuVm::step_map` of a published-roots (G17) VM. + fn step_map_published( + self: &mut gpuvm::UpdatingGpuVm<'_, Self>, + op: &mut gpuvm::OpMap, + ctx: &mut StepContext, ) -> Result { - let total = out.len(); - let mut done = 0usize; - while done < total { - let addr = iova.checked_add(done as u64).ok_or(EOVERFLOW)?; - let phys = self.page_table.translate_iova(addr)?; - let page_mask = (kernel::page::PAGE_SIZE as PhysicalAddr) - 1; - let page_phys = phys & !page_mask; - let offset = (phys - page_phys) as usize; - let chunk = cmp::min(kernel::page::PAGE_SIZE - offset, total - done); - // SAFETY: `translate_iova` resolved this address through a leaf - // PTE of this VM while the VM execution lock is held. The lock - // stays held until the copy below finishes, so the PTE cannot be - // removed and the GEM object it maps keeps the page allocated. - // `borrow_phys` additionally rejects anything without a struct - // page. - let page = unsafe { Page::borrow_phys(&page_phys) }.ok_or(EFAULT)?; - let dst = &mut out[done..done + chunk]; - page.with_pointer_into_page(offset, chunk, |ptr| { - before_copy(ptr, chunk); - // SAFETY: `with_pointer_into_page` bounds-checked - // `offset..offset + chunk` inside the page. - let src = unsafe { core::slice::from_raw_parts(ptr as *const u8, chunk) }; - dst.copy_from_slice(src); - Ok(()) - })?; - done += chunk; - } - Ok(()) - } - - /// Map an `mm::Node` representing an mapping in VA space. - fn map_node(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { - let mut iova = node.start(); - let mapping_end = if self.uat_inner.fault.is_some() { - iova.checked_add(node.mapped_size as u64).ok_or(EOVERFLOW)? - } else { iova + node.mapped_size as u64 }; - // The private fixed-mapping allocator cannot see GPUVM-owned leaves. - // Reserve the complete page-table walk before installing any SG run. - if self.uat_inner.fault.is_some() { self.page_table.prepare_map(iova..mapping_end)?; } - let guard = node.bo.as_ref().ok_or(EINVAL)?.inner().inner.lock(); - let sgt = guard.sgt.as_ref().ok_or(EINVAL)?; - let mut offset = node.offset; - let mut left = node.mapped_size; + let _mutation = self.mapping_mutation(); + let mut iova = op.addr(); + let mut left = op.range() as usize; + let mut offset = op.offset() as usize; - for range in sgt.iter() { - if left == 0 { - break; - } + let bo = ctx.vm_bo.as_ref().expect("step_map with no BO"); - // TODO: proper DMA address/length handling - let mut addr = range.dma_address() as usize; - let mut len: usize = range.dma_len() as usize; + let one_page = op.flags().contains(gpuvm::GpuVaFlags::REPEAT); - if (offset | addr | len | iova as usize) & UAT_PGMSK != 0 { - dev_err!( - self.dev.as_ref(), - "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", - addr, - len, - iova - ); - return Err(EINVAL); + let mut do_map = |mut addr: usize, mut len: usize, offset: &mut usize| -> Result { + if left == 0 { + return Ok(false); } - if offset > 0 { - let skip = len.min(offset); + if *offset > 0 { + let skip = len.min(*offset); addr += skip; len -= skip; - offset -= skip; + *offset -= skip; } - - len = len.min(left); - if len == 0 { - continue; + return Ok(true); + } + assert!(*offset == 0); + + if one_page { + len = left; + } else { + len = len.min(left); } mod_dev_dbg!( self.dev, - "MMU: map: {:#x}:{:#x} -> {:#x}\n", + "MMU: map: {:#x}:{:#x} -> {:#x} [OP={}]\n", addr, len, - iova + iova, + one_page ); - self.page_table.map_pages( + if let Err(error) = self.page_table.map_pages( iova..(iova + len as u64), addr as PhysicalAddr, - prot, - false, - )?; + ctx.prot, + one_page, + ) { + self.rollback_failed_map(op.addr(), op.range()); + return Err(error); + } - iova += len as u64; left -= len; - } - Ok(()) - } -} + iova += len as u64; + Ok(true) + }; -/// Shared reference to a virtual memory address space ([`Vm`]). -#[derive(Clone)] -pub(crate) struct Vm { - fault: Option>, - id: u64, - inner: ARef>, - dummy_obj: ARef, - binding: Arc>, - driver_mappings: Option>>>, - status: Option>, - job_lifetime: Option>>, -} -impl Drop for Vm { - fn drop(&mut self) { - if self.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { - // Keep the GPUVM and its user BO bindings until a future reset - // implementation can prove that firmware no longer references it. - core::mem::forget(self.inner.clone()); - } - } -} -no_debug!(Vm); + let guard = bo.inner().inner.lock(); + if let Some(sg_vec) = guard.sg_vec.as_ref() { + let start_idx = sg_vec + .binary_search_by(|range| { + if range.0 > offset { + cmp::Ordering::Greater + } else if (range.0 + range.1.len()) <= offset { + cmp::Ordering::Less + } else { + cmp::Ordering::Equal + } + }) + .expect("sg_vec does not contain offset???"); + + offset -= sg_vec[start_idx].0 as usize; + + for cur in start_idx..sg_vec.len() { + let addr = sg_vec[cur].1.start as usize; + let len: usize = sg_vec[cur].1.len() as usize; + if do_map(addr, len, &mut offset)? == false { + break; + } + } + } else { + for range in guard.sgt.as_ref().expect("step_map with no SGT").iter() { + // TODO: proper DMA address/length handling + let addr = range.dma_address() as usize; + let len: usize = range.dma_len() as usize; + if do_map(addr, len, &mut offset)? == false { + break; + } + } + } + + let mut gpuva = ctx.new_va.take().expect("Multiple step_map calls"); + gpuva.as_mut().inner_mut().get_mut().prot = Some(ctx.prot); + + if op + .map_and_link_va( + self, + gpuva, + ctx.vm_bo.as_ref().expect("step_map with no BO"), + ) + .is_err() + { + dev_err!( + self.dev.as_ref(), + "map_and_link_va failed: {:#x} [{:#x}] -> {:#x}\n", + op.offset(), + op.range(), + op.addr() + ); + self.rollback_failed_map(op.addr(), op.range()); + return Err(EINVAL); + } + self.tlbi_contexts_naming_root(op.addr(), op.range() as usize); + Ok(()) + } + /// `DriverGpuVm::step_unmap` of a published-roots (G17) VM. + fn step_unmap_published( + self: &mut gpuvm::UpdatingGpuVm<'_, Self>, + op: &mut gpuvm::OpUnMap, + _ctx: &mut StepContext, + ) -> Result { + let va = op.va().expect("step_unmap: missing VA"); + + self.unmap_gpuva(va)?; + + if op.unmap_and_unlink_va_defer().is_none() { + dev_err!(self.dev.as_ref(), "step_unmap: could not unlink gpuva"); + } + Ok(()) + } + /// `DriverGpuVm::step_remap` of a published-roots (G17) VM. + fn step_remap_published( + self: &mut gpuvm::UpdatingGpuVm<'_, Self>, + op: &mut gpuvm::OpReMap, + vm_bo: &gpuvm::GpuVmBo, + ctx: &mut StepContext, + ) -> Result { + let va = op.unmap().va().expect("No previous VA"); + let orig_addr = va.addr(); + let orig_range = va.range(); + let orig_prot = va.inner().prot.ok_or(EINVAL)?; + let orig_invalidated = va.flags().contains(gpuvm::GpuVaFlags::INVALIDATED); + + // Only unmap the hole between prev/next, if they exist + let unmap_start = if let Some(op) = op.prev_map() { + op.addr() + op.range() + } else { + orig_addr + }; + + let unmap_end = if let Some(op) = op.next_map() { + op.addr() + } else { + orig_addr + orig_range + }; + + mod_dev_dbg!( + self.dev, + "MMU: unmap for remap: {:#x}..{:#x} (from {:#x}:{:#x})\n", + unmap_start, + unmap_end, + orig_addr, + orig_range + ); + + let unmap_range = unmap_end - unmap_start; + + let _mutation = self.mapping_mutation(); + if orig_invalidated { + self.page_table.discard_partial_map(unmap_start..unmap_end)?; + } else { + self.page_table.unmap_pages(unmap_start..unmap_end)?; + } + + if let Some(asid) = self.slot() { + mem::sync(); + self.tlbi_range(asid as u8, unmap_start, unmap_range as usize); + mod_dev_dbg!( + self.dev, + "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", + asid, + unmap_start, + unmap_range, + ); + mem::sync(); + } + + self.tlbi_contexts_naming_root(unmap_start, unmap_range as usize); + + if op.unmap().unmap_and_unlink_va_defer().is_none() { + dev_err!(self.dev.as_ref(), "step_unmap: could not unlink gpuva"); + } + + if let Some(prev_op) = op.prev_map() { + let mut prev_gpuva = ctx + .prev_va + .take() + .expect("Multiple step_remap calls with prev_op"); + prev_gpuva.as_mut().inner_mut().get_mut().prot = Some(orig_prot); + if prev_op.map_and_link_va(self, prev_gpuva, vm_bo).is_err() { + dev_err!(self.dev.as_ref(), "step_remap: could not relink prev gpuva"); + return Err(EINVAL); + } + } + + if let Some(next_op) = op.next_map() { + let mut next_gpuva = ctx + .next_va + .take() + .expect("Multiple step_remap calls with next_op"); + next_gpuva.as_mut().inner_mut().get_mut().prot = Some(orig_prot); + if next_op.map_and_link_va(self, next_gpuva, vm_bo).is_err() { + dev_err!(self.dev.as_ref(), "step_remap: could not relink next gpuva"); + return Err(EINVAL); + } + } + + Ok(()) + } + + /// Remove hardware leaves for one complete user GPUVA without allocating. + fn unmap_gpuva(&mut self, va: &gpuvm::GpuVa) -> Result { + mod_dev_dbg!(self.dev, "MMU: unmap: {:#x}:{:#x}\n", va.addr(), va.range()); + + let _mutation = self.mapping_mutation(); + if va.flags().contains(gpuvm::GpuVaFlags::INVALIDATED) { + self.page_table.discard_partial_map(va.addr()..(va.addr() + va.range()))?; + } else { + self.page_table.unmap_pages(va.addr()..(va.addr() + va.range()))?; + } + + if let Some(asid) = self.slot() { + mem::sync(); + self.tlbi_range(asid as u8, va.addr(), va.range() as usize); + mod_dev_dbg!( + self.dev, + "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", + asid, + va.addr(), + va.range(), + ); + mem::sync(); + } + + self.tlbi_contexts_naming_root(va.addr(), va.range() as usize); + + Ok(()) + } + + /// A failed map has no GPUVA owner to retain its backing or remove its leaves. + /// GPUVM has already split or removed overlaps throughout this operation's range. + fn rollback_failed_map(&mut self, addr: u64, size: u64) { + // Include the current segment: map_pages can fail after writing some of its leaves. + // The checked GPUVM operation supplies an aligned range without overflow. + let result = self.page_table.discard_partial_map(addr..addr + size); + if self.uat_inner.firmware == UatFirmware::Handoff { + if let Some(asid) = self.slot() { + mem::sync(); + self.tlbi_range(asid as u8, addr, size as usize); + mem::sync(); + } + } + self.tlbi_contexts_naming_root(addr, size as usize); + if let Err(error) = result { + dev_err!( + self.dev.as_ref(), + "Could not discard failed mapping: {:?}\n", + error + ); + } + } + + fn mapping_mutation(&mut self) -> MappingMutation { + let state = self.mapping_epoch.clone(); + if let Some(state) = state.as_ref() { + state.writers.fetch_add(1, Ordering::Relaxed); + state.visible.swap(u64::MAX, Ordering::AcqRel); + self.epoch = self.epoch.and_then(|epoch| epoch.checked_add(1)); + state + .pending + .store(self.epoch.unwrap_or(u64::MAX), Ordering::Relaxed); + } + MappingMutation(state) + } + + /// Invalidate a VA range of this Vm's address space in the TLB of one ASID. + fn tlbi_range(&self, asid: u8, iova: u64, size: usize) { + match self.uat_inner.firmware { + UatFirmware::Handoff => mem::tlbi_range(asid, iova as usize, size), + UatFirmware::PublishedRoots { .. } => { + mem::tlbi_range_or_asid(asid, iova as usize, size) + } + } + } + + /// Invalidate a VA range in every TTBAT context that names this Vm's root as its lower half. + /// + /// With [`UatFirmware::PublishedRoots`], contexts other than a Vm's bind slot name its root + /// (the kernel lower root sits in contexts 0 and 1 and has no slot at all), so a mapping + /// change must reach all of them. Other firmware only uses the bind slot, which the callers + /// already invalidate. + fn tlbi_contexts_naming_root(&self, iova: u64, size: usize) { + if self.uat_inner.firmware == UatFirmware::Handoff { + mem::sync(); + return; + } + + let contexts = self.uat_inner.lock().contexts_naming_root(self.ttb()); + self.tlbi_context_mask(iova, size, contexts); + } + + fn tlbi_context_mask(&self, iova: u64, size: usize, contexts: u64) { + // Complete new leaves even if no context currently names this root: + // a later context can publish it without repeating the map operation. + // DMB orders accesses but does not complete stores before TLBI; the + // GPU uses outer-shareable invalidation and the barrier covers it. + mem::sync(); + if contexts == 0 { + return; + } + for ctx in 0..UAT_NUM_CTX { + if contexts & (1 << ctx) != 0 { + mem::tlbi_range_or_asid(ctx as u8, iova as usize, size); + } + } + mem::sync(); + } + + /// `map_node` of a published-roots (G17) VM. + fn map_node_published(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { + let end = node + .start() + .checked_add(node.mapped_size as u64) + .ok_or(EOVERFLOW)?; + let _mutation = self.mapping_mutation(); + self.page_table.prepare_map(node.start()..end)?; + let result = self.map_node_prepared(node, prot); + if result.is_err() { + self.rollback_failed_map(node.start(), node.mapped_size as u64); + } + result + } + + /// Install only into the empty range reserved by `map_node`. + fn map_node_prepared(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { + let mut iova = node.start(); + let guard = node.bo.as_ref().ok_or(EINVAL)?.get()?.inner().inner.lock(); + let sgt = guard.sgt.as_ref().ok_or(EINVAL)?; + let mut offset = node.offset; + let mut left = node.mapped_size; + + for range in sgt.iter() { + if left == 0 { + break; + } + + // TODO: proper DMA address/length handling + let mut addr = range.dma_address() as usize; + let mut len: usize = range.dma_len() as usize; + + if (offset | addr | len | iova as usize) & UAT_PGMSK != 0 { + dev_err!( + self.dev.as_ref(), + "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", + addr, + len, + iova + ); + return Err(EINVAL); + } + + if offset > 0 { + let skip = len.min(offset); + addr += skip; + len -= skip; + offset -= skip; + } + + len = len.min(left); + + if len == 0 { + continue; + } + + mod_dev_dbg!( + self.dev, + "MMU: map: {:#x}:{:#x} -> {:#x}\n", + addr, + len, + iova + ); + + self.page_table.map_pages( + iova..(iova + len as u64), + addr as PhysicalAddr, + prot, + false, + )?; + + iova += len as u64; + left -= len; + } + + // A context naming this root may be live and hold a negative translation for the range. + self.tlbi_contexts_naming_root(node.start(), node.mapped_size); + Ok(()) + } + + /// Returns the slot index, if this VM is bound. + fn slot(&self) -> Option { + if let Some(slot) = self.fixed_slot { return Some(slot); } + if self.is_kernel { + // The GFX ASC does not care about the ASID. Pick an arbitrary one. + // TODO: This needs to be a persistently reserved ASID once we integrate + // with the ARM64 kernel ASID machinery to avoid overlap. + Some(0) + } else { + // We don't check whether we lost the slot, which could cause unnecessary + // invalidations against another Vm. However, this situation should be very + // rare (e.g. a Vm lost its slot, which means 63 other Vms bound in the + // interim, and then it gets killed / drops its mappings without doing any + // final rendering). Anything doing active maps/unmaps is probably also + // rendering and therefore likely bound. + self.binding + .lock() + .bind_token + .as_ref() + .map(|token| token.last_slot() + UAT_USER_CTX_START as u32) + } + } + + /// Returns the translation table base for this Vm + fn ttb(&self) -> u64 { + self.page_table.ttb() + } + + /// Check that a byte range is fully mapped with the requested GPU access. + /// The caller holds the VM execution lock. + fn covers_mapped_range( + &mut self, + address: u64, + size: u64, + need_read: bool, + need_write: bool, + ) -> bool { + if size == 0 { + return false; + } + let Some(end) = address.checked_add(size) else { + return false; + }; + let page_mask = UAT_PGMSK as u64; + let start = address & !page_mask; + let aligned_end = if end & page_mask == 0 { + end + } else { + match (end | page_mask).checked_add(1) { + Some(end) => end, + None => return false, + } + }; + if start < self.va_range.start || aligned_end > self.va_range.end { + return false; + } + self.page_table + .covers_range(start..aligned_end, need_read, need_write) + .unwrap_or(false) + } + + /// Copy `out.len()` bytes at `iova` from the pages this VM maps, through + /// the kernel's linear map. `before_copy` receives the kernel address and + /// length of each chunk, which never crosses a page, before it is copied. + /// + /// The caller holds the VM execution lock for the whole call. Every PTE + /// change takes that lock, and a page leaves the page table before the GEM + /// reference that keeps it allocated is released, so a concurrent unbind + /// or GEM close cannot free a page while it is being read. + fn copy_from_mapped_pages( + &mut self, + iova: u64, + out: &mut [u8], + mut before_copy: impl FnMut(*mut u8, usize), + ) -> Result { + let total = out.len(); + let mut done = 0usize; + while done < total { + let addr = iova.checked_add(done as u64).ok_or(EOVERFLOW)?; + let phys = self.page_table.translate_iova(addr)?; + let page_mask = (kernel::page::PAGE_SIZE as PhysicalAddr) - 1; + let page_phys = phys & !page_mask; + let offset = (phys - page_phys) as usize; + let chunk = cmp::min(kernel::page::PAGE_SIZE - offset, total - done); + // SAFETY: `translate_iova` resolved this address through a leaf + // PTE of this VM while the VM execution lock is held. The lock + // stays held until the copy below finishes, so the PTE cannot be + // removed and the GEM object it maps keeps the page allocated. + // `borrow_phys` additionally rejects anything without a struct + // page. + let page = unsafe { Page::borrow_phys(&page_phys) }.ok_or(EFAULT)?; + let dst = &mut out[done..done + chunk]; + page.with_pointer_into_page(offset, chunk, |ptr| { + before_copy(ptr, chunk); + // SAFETY: `with_pointer_into_page` bounds-checked + // `offset..offset + chunk` inside the page. + let src = unsafe { core::slice::from_raw_parts(ptr as *const u8, chunk) }; + dst.copy_from_slice(src); + Ok(()) + })?; + done += chunk; + } + Ok(()) + } + + /// Map an `mm::Node` representing an mapping in VA space. + fn map_node(&mut self, node: &mm::Node<(), KernelMappingInner>, prot: Prot) -> Result { + if self.published() { + return self.map_node_published(node, prot); + } + let mut iova = node.start(); + let mapping_end = if self.uat_inner.fault.is_some() { + iova.checked_add(node.mapped_size as u64).ok_or(EOVERFLOW)? + } else { iova + node.mapped_size as u64 }; + // The private fixed-mapping allocator cannot see GPUVM-owned leaves. + // Reserve the complete page-table walk before installing any SG run. + if self.uat_inner.fault.is_some() { self.page_table.prepare_map(iova..mapping_end)?; } + let guard = node.bo.as_ref().ok_or(EINVAL)?.get()?.inner().inner.lock(); + let sgt = guard.sgt.as_ref().ok_or(EINVAL)?; + let mut offset = node.offset; + let mut left = node.mapped_size; + + for range in sgt.iter() { + if left == 0 { + break; + } + + // TODO: proper DMA address/length handling + let mut addr = range.dma_address() as usize; + let mut len: usize = range.dma_len() as usize; + + if (offset | addr | len | iova as usize) & UAT_PGMSK != 0 { + dev_err!( + self.dev.as_ref(), + "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", + addr, + len, + iova + ); + return Err(EINVAL); + } + + if offset > 0 { + let skip = len.min(offset); + addr += skip; + len -= skip; + offset -= skip; + } + + len = len.min(left); + + if len == 0 { + continue; + } + + mod_dev_dbg!( + self.dev, + "MMU: map: {:#x}:{:#x} -> {:#x}\n", + addr, + len, + iova + ); + + self.page_table.map_pages( + iova..(iova + len as u64), + addr as PhysicalAddr, + prot, + false, + )?; + + iova += len as u64; + left -= len; + } + Ok(()) + } +} + +/// Shared reference to a virtual memory address space ([`Vm`]). +#[derive(Clone)] +pub(crate) struct Vm { + fault: Option>, + id: u64, + inner: ARef>, + dummy_obj: ARef, + binding: Arc>, + driver_mappings: Option>>>, + agx_status: Option>, + job_lifetime: Option>>, + // Published-roots (G17) VM state; `None` on every other VM. + mapping_epoch: Option>, + lifetime: Option>, + residency: Option>, + context_bindings: Option>, + shared_bindings: Option>, + status: Option>, +} +impl Drop for Vm { + fn drop(&mut self) { + if self.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { + // Keep the GPUVM and its user BO bindings until a future reset + // implementation can prove that firmware no longer references it. + core::mem::forget(self.inner.clone()); + } + } +} +no_debug!(Vm); struct M3VmJobLifetime { active: usize, @@ -918,14 +1532,164 @@ impl Clone for VmBind { } } +/// An application TTBAT identity retained by a logical queue and its jobs. +/// The root lease is dropped before the VM, so no page table can be freed +/// while an active context still names it. Installed firmware queues retain +/// this object through replacement retirement or processor stop; a consumed +/// closed-scheduler release may return its root identity while keeping storage. +pub(crate) struct ExecutionContext { + root: ExecutionRoot, + vm: Vm, +} + +struct ExecutionRoot { + released: AtomicBool, + inner: Arc, + id: u8, + generation: u8, + low: u64, + high: u64, +} + +impl ExecutionContext { + pub(crate) fn id(&self) -> u32 { + self.root.id.into() + } + pub(crate) fn generation(&self) -> u8 { + self.root.generation + } + pub(crate) fn vm(&self) -> &Vm { + &self.vm + } + + /// Called only after the closed scheduler's release was consumed and all + /// its commands retired. Installed storage may outlive this root lease. + pub(crate) fn release(&self) { + self.root.release(); + } + + pub(crate) fn is_current(&self) -> bool { + let inner = self.root.inner.lock(); + if self.root.released.load(Ordering::Acquire) { + return false; + } + let roots = &inner.ttbs()[usize::from(self.root.id)]; + roots.ttb0.load(Ordering::Acquire) == self.root.low + && roots.ttb1.load(Ordering::Acquire) == self.root.high + } +} + +impl ExecutionRoot { + fn release(&self) { + if self.released.swap(true, Ordering::AcqRel) { + return; + } + let id = usize::from(self.id); + let release = { + let inner = self.inner.lock(); + let handoff = inner.handoff().lock_legacy(); + let roots = &inner.ttbs()[id]; + let low = roots.ttb0.load(Ordering::Acquire); + let high = roots.ttb1.load(Ordering::Acquire); + let owned = low == self.low && high == self.high; + if owned { + roots.ttb0.store(0, Ordering::Release); + roots.ttb1.store(0, Ordering::Release); + } + drop(handoff); + owned || (low == 0 && high == 0) + }; + if release { + mem::sync(); + mem::tlbi_asid(self.id); + mem::sync(); + self.inner.lock().execution_contexts &= !(1u64 << id); + } else { + pr_err!( + "MMU: application context {} changed owners while leased\n", + id + ); + } + } +} + +impl Drop for ExecutionRoot { + fn drop(&mut self) { + self.release(); + } +} + +/// A mapping BO reference. On published-roots (G17) VMs it is released under the +/// immediate-mode GPUVA lock, retaining the owner until deferred cleanup completes, including +/// its final BO. Other VMs drop the reference synchronously. +struct MappingBo { + bo: Option>>, + owner: Option>>, +} +impl MappingBo { + fn new( + bo: ARef>, + owner: ARef>, + ) -> Self { + Self { + bo: Some(bo), + owner: Some(owner), + } + } + fn synchronous(bo: ARef>) -> Self { + Self { bo: Some(bo), owner: None } + } + fn get(&self) -> Result<&gpuvm::GpuVmBo> { + self.bo.as_deref().ok_or(EINVAL) + } +} +impl Drop for MappingBo { + fn drop(&mut self) { + let Some(owner) = self.owner.as_ref() else { + // The `bo` field drops normally. + return; + }; + let Some(bo) = self.bo.take() else { + return; + }; + let raw = ARef::into_raw(bo); + // SAFETY: GpuVmBo is repr(C) with drm_gpuvm_bo first. into_raw + // transfers one live reference. Every Vm here uses IMMEDIATE_MODE; + // deferred put selects its GPUVA lock and retains cleanup ownership. + unsafe { + kernel::bindings::drm_gpuvm_bo_put_deferred( + raw.as_ptr().cast::(), + ) + }; + owner.bo_deferred_cleanup(); + } +} + +/// The GEM object of a mapping: published-roots (G17) mappings pin it against idle reclaim. +enum MappingObject { + Shared(ARef), + Pinned(gem::KernelMappingPin), +} + +impl core::ops::Deref for MappingObject { + type Target = gem::Object; + fn deref(&self) -> &gem::Object { + match self { + Self::Shared(object) => object, + Self::Pinned(pin) => pin, + } + } +} + /// Inner data required for an object mapping into a [`Vm`]. pub(crate) struct KernelMappingInner { // Drop order matters: // - Drop the GpuVmBo first, which locks its BO GPUVA list and drops a GpuVm reference + // (published-roots VMs: deferred-put it under its immediate-mode GPUVA lock) // - Drop the GEM BO next, since BO free can take the resv lock itself // - Drop the owner GpuVm last, since that again can take resv locks when the refcount drops to 0 - bo: Option>>, - _gem: Option>, + bo: Option, + _gem: Option, owner: ARef>, uat_inner: Arc, prot: Prot, @@ -933,6 +1697,27 @@ pub(crate) struct KernelMappingInner { mapped_size: usize, } +/// Prepared CPU timestamp access; destruction happens after completion signalling. +pub(crate) struct TimestampMapping { + mapping: shmem::VMap, + offset: usize, +} + +impl TimestampMapping { + pub(crate) fn write(&self, value: u64) { + let map = self.mapping.get(); + for (index, byte) in value.to_le_bytes().into_iter().enumerate() { + // SAFETY: timestamp_mapping checked the complete eight-byte extent. + // The owned vmap pins its backing; byte atomics permit unaligned and + // overlapping timestamp destinations without mutable slice aliases. + let destination = unsafe { + &*map.as_mut_ptr().add(self.offset + index).cast::() + }; + destination.store(byte, Ordering::Relaxed); + } + } +} + /// An object mapping into a [`Vm`], which reserves the address range from use by other mappings. pub(crate) struct KernelMapping(ManuallyDrop>); @@ -952,6 +1737,147 @@ impl KernelMapping { self.0.start()..(self.0.start() + self.0.mapped_size as u64) } + /// Resolves a byte in this retained mapping without creating page tables. + pub(crate) fn translate_offset(&self, offset: usize) -> Result { + if offset >= self.size() { + return Err(EINVAL); + } + let address = self.iova().checked_add(offset as u64).ok_or(EOVERFLOW)?; + self.0.owner.exec_lock(None, false)?.page_table.translate_iova(address) + } + + /// Writes a userspace timestamp through the retained GEM mapping. Byte + /// stores support unaligned destinations and concurrent completion writes + /// without creating mutable aliases. The complete timestamp is not atomic. + /// Retain CPU access before this destination's completion fence is published. + pub(crate) fn timestamp_mapping(&self, offset: usize) -> Result { + use kernel::drm::gem::BaseObject; + let gem = self.0._gem.as_ref().ok_or(EINVAL)?; + if offset.checked_add(size_of::()).ok_or(EOVERFLOW)? > self.size() { + return Err(ERANGE); + } + let start = self.0.offset.checked_add(offset).ok_or(EOVERFLOW)?; + let end = start.checked_add(size_of::()).ok_or(EOVERFLOW)?; + if end > gem.size() { + return Err(ERANGE); + } + Ok(TimestampMapping { mapping: gem.owned_vmap::()?, offset: start }) + } + + /// Gives a fresh kernel mapping a firmware-cached prefix, preserving its tail. + /// The result includes whether any page-table entry may have changed; after + /// mutation the caller must retain the mapping until processor stop even if + /// the readback fails. The original protection is retained for teardown. + /// + /// # Safety + /// The mapping must be unpublished. After mutation it must not be unmapped + /// while a firmware instance can retain a cached translation or contents. + pub(crate) unsafe fn set_fw_cached_prefix(&self, prefix: usize) -> (Result<[u64; 2]>, bool) { + const MAX_PAGES: usize = 32; + const ATTRIBUTE_SHIFT: u32 = 2; + const ATTRIBUTE_MASK: u64 = 7; + const UNCACHED: u64 = 2; + const CACHE_ATTRIBUTE_CHANGE: u64 = UNCACHED << ATTRIBUTE_SHIFT; + let mut mutated = false; + let result = (|| { + if self.0.uat_inner.firmware == UatFirmware::Handoff + || prefix == 0 + || prefix > self.size() + || (prefix | self.size()) & UAT_PGMSK != 0 + || self.size() > MAX_PAGES * UAT_PGSZ + { + return Err(EINVAL); + } + let mut owner = self.0.owner.exec_lock(None, false)?; + if !owner.is_kernel { + return Err(EINVAL); + } + let pages = self.size() / UAT_PGSZ; + let cached_pages = prefix / UAT_PGSZ; + let mut before = [0; MAX_PAGES]; + for (index, value) in before[..pages].iter_mut().enumerate() { + *value = owner + .page_table + .leaf_entry(self.iova() + (index * UAT_PGSZ) as u64)?; + if (*value >> ATTRIBUTE_SHIFT) & ATTRIBUTE_MASK != UNCACHED { + return Err(EINVAL); + } + } + let _mutation = owner.mapping_mutation(); + mutated = true; + owner.page_table.reprot_pages( + self.iova()..self.iova() + prefix as u64, + PROT_GPU_FW_PRIV_RW, + )?; + mem::sync(); + mem::tlbi_all(); + mem::sync(); + let mut after = [0; MAX_PAGES]; + for (index, value) in after[..pages].iter_mut().enumerate() { + *value = owner + .page_table + .leaf_entry(self.iova() + (index * UAT_PGSZ) as u64)?; + let change = if index < cached_pages { + CACHE_ATTRIBUTE_CHANGE + } else { + 0 + }; + if before[index] ^ *value != change { + return Err(EFAULT); + } + } + Ok([after[0], after[cached_pages.min(pages - 1)]]) + })(); + (result, mutated) + } + + /// `Drop` of a published-roots (G17) mapping. That firmware does not use the handoff + /// protocol and cannot be asked to flush its cache; it only caches kernel mappings that live + /// until its coprocessors stop. + fn drop_published(&mut self) { + // Only VM page tables/binding state are touched here. The shared + // reservation is sufficient and avoids fallible exec allocation in Drop. + let mut owner = self.0.owner.lock_inner(); + mod_dev_dbg!( + owner.dev, + "MMU: unmap {:#x}:{:#x}\n", + self.iova(), + self.size() + ); + + let mutation = owner.mapping_mutation(); + if owner.page_table.unmap_pages(self.iova_range()).is_err() { + dev_err!( + owner.dev.as_ref(), + "MMU: unmap {:#x}:{:#x} failed\n", + self.iova(), + self.size() + ); + } + + if let Some(asid) = owner.slot() { + mem::sync(); + owner.tlbi_range(asid as u8, self.iova(), self.size()); + mod_dev_dbg!( + owner.dev, + "MMU: flush range: asid={:#x} start={:#x} len={:#x}\n", + asid, + self.iova(), + self.size() + ); + mem::sync(); + } + + // The range returns to the VA allocator once this mapping is gone, so no context that + // names this root may keep a stale translation for it. + owner.tlbi_contexts_naming_root(self.iova(), self.size()); + drop(mutation); + drop(owner); + // SAFETY: The PTEs are gone and invalidated, and the VM reservation is released before + // the node drops its BO references. + unsafe { ManuallyDrop::drop(&mut self.0) }; + } + fn remap_uncached_and_flush_legacy(&mut self) { let mut owner = self .0 @@ -1292,6 +2218,10 @@ no_debug!(KernelMapping); impl Drop for KernelMapping { fn drop(&mut self) { + if self.0.uat_inner.firmware != UatFirmware::Handoff { + self.drop_published(); + return; + } if self.0.uat_inner.fault.is_none() { self.drop_legacy(); return; @@ -1379,6 +2309,11 @@ impl Drop for KernelMapping { struct UatShared { fault: Option>, kernel_ttb1: u64, + /// Execution contexts leased to logical queues (published-roots UATs). + execution_contexts: u64, + context_generations: [u8; UAT_NUM_CTX], + /// Kernel root of the secondary firmware instance, if there is one. + secondary_ttb1: Option, map_kernel_to_user: bool, handoff_rgn: UatRegion, ttbs_rgn: UatRegion, @@ -1405,6 +2340,20 @@ impl UatShared { // SAFETY: pointer is non-null per the type invariant unsafe { (self.ttbs_rgn.map.ptr() as *mut [SlotTTBS; UAT_NUM_CTX]).as_ref() }.unwrap() } + + /// Returns the mask of TTBAT contexts whose lower half is the root `ttb`. + fn contexts_naming_root(&self, ttb: u64) -> u64 { + self.ttbs() + .iter() + .enumerate() + .filter(|(ctx, slot)| slot.ttb0.load(Ordering::Acquire) == tagged_root(ttb, *ctx)) + .fold(0, |mask, (ctx, _)| mask | (1 << ctx)) + } +} + +/// Returns the TTBAT value naming root `ttb` in context `ctx`. +const fn tagged_root(ttb: u64, ctx: usize) -> u64 { + ttb | TTBR_VALID | (ctx as u64) << TTBR_ASID_SHIFT } // SAFETY: Nothing here is unsafe to send across threads. @@ -1413,6 +2362,7 @@ unsafe impl Send for UatShared {} /// Inner data for the top-level UAT instance. #[pin_data] struct UatInner { + firmware: UatFirmware, m3_running: bool, fault: Option>, firmware_cache_flush_ready: AtomicBool, @@ -1442,6 +2392,7 @@ impl UatInner { pub(crate) struct Uat { dev: driver::AsahiDevRef, cfg: UatConfig, + geometry: UatGeometry, inner: Arc, slots: slotalloc::SlotAllocator, @@ -1521,7 +2472,7 @@ impl Handoff { /// In [`HandoffMode::FirmwareDekker`] this blocks (bounded) until the /// firmware publishes `magic_fw`. The stopped M3 owner initializes /// its AP state before firmware starts. - fn init(&self, mode: HandoffMode, m3: bool) -> Result { + fn init(&self, mode: HandoffMode, m3: bool, firmware: UatFirmware) -> Result { self.magic_ap.store(PPL_MAGIC, Ordering::Relaxed); self.cur_slot.store(if m3 && mode != HandoffMode::FirmwareDekker { u32::MAX } else { 0 }, Ordering::Relaxed); self.unk3.store(0, Ordering::Relaxed); @@ -1537,7 +2488,10 @@ impl Handoff { } fence(Ordering::SeqCst); - if matches!(mode, HandoffMode::FirmwareDekker | HandoffMode::FirmwareM3) { + // Published-roots (G17) firmware never touches the handoff region. + if matches!(mode, HandoffMode::FirmwareDekker | HandoffMode::FirmwareM3) + && firmware == UatFirmware::Handoff + { let start = Instant::::now(); const TIMEOUT: Delta = Delta::from_millis(1000); @@ -1674,8 +2628,194 @@ impl Vm { pub(crate) fn same(&self, other: &Vm) -> bool { self.id == other.id } /// The DRM VM's id. pub(crate) fn id(&self) -> u64 { self.id } - pub(crate) fn status(&self) -> Result<&Arc> { self.status.as_ref().ok_or(EINVAL) } + pub(crate) fn agx_status(&self) -> Result<&Arc> { self.agx_status.as_ref().ok_or(EINVAL) } pub(crate) fn is_m3(&self) -> bool { self.fault.is_some() } + /// Whether this VM belongs to a published-roots (G17) UAT. + fn published(&self) -> bool { self.mapping_epoch.is_some() } + + /// Create a new virtual memory address space of a published-roots (G17) UAT. + fn new_published( + dev: &driver::AsahiDevice, + uat_inner: Arc, + kernel_range: Range, + geometry: UatGeometry, + ttb: Option, + id: u64, + ) -> Result { + let dummy_obj = gem::new_kernel_object(dev, UAT_PGSZ)?; + let is_kernel = ttb.is_some(); + + let page_table = if let Some(ttb) = ttb { + UatPageTable::new_with_ttb( + ttb, + geometry.kernel_range(), + geometry.ias, + geometry.oas, + uat_inner.firmware != UatFirmware::Handoff, + )? + } else { + UatPageTable::new(geometry.ias, geometry.oas)? + }; + + let (va_range, gpuvm_range) = if is_kernel { + (geometry.kernel_range(), kernel_range.clone()) + } else { + let mut range = geometry.user_range(); + if uat_inner.firmware != UatFirmware::Handoff { + range.start = 0; + } + (range, geometry.user_usable_range()) + }; + + let mm = mm::Allocator::new(va_range.start, va_range.range(), ())?; + + let binding = Arc::pin_init( + new_mutex!( + VmBinding { + binding: None, + bind_token: None, + active_users: 0, + ttb: page_table.ttb(), + }, + "VmBinding", + ), + GFP_KERNEL, + )?; + + let mapping_epoch = if uat_inner.firmware != UatFirmware::Handoff { + Some(Arc::new(MappingEpoch::new(), GFP_KERNEL)?) + } else { + None + }; + let binding_clone = binding.clone(); + Ok(Vm { + fault: uat_inner.fault.clone(), + driver_mappings: None, + agx_status: None, + job_lifetime: None, + id, + mapping_epoch: mapping_epoch.clone(), + lifetime: None, + residency: None, + context_bindings: None, + shared_bindings: None, + status: None, + dummy_obj: dummy_obj.gem.clone(), + inner: gpuvm::GpuVm::new_deferred_bo_put( + c_str!("Asahi::GpuVm"), + // TODO: should we using DRM_GPUVM_RESV_PROTECTED as well? + drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE, + dev, + dummy_obj.gem.clone(), + gpuvm_range, + kernel_range, + init!(VmInner { + dev: dev.into(), + va_range, + is_kernel, + fixed_slot: None, + page_table, + mapping_epoch, + epoch: Some(0), + mm, + uat_inner, + binding: binding_clone, + id, + }), + )?, + binding, + }) + } + + /// Attach a status before sharing a newly created user VM with queues. + pub(crate) fn with_status(mut self) -> Result { + self.status = Some(Arc::pin_init(crate::g17::status::VmStatus::new(), GFP_KERNEL)?); + self.lifetime = Some(lifetime::VmLifetime::new()?); + self.residency = Some(residency::Gate::new()?); + self.context_bindings = Some(context::ContextBindings::new()?); + self.shared_bindings = Some(shared::SharedBindings::new()?); + Ok(self) + } + + /// Status shared by every queue using this VM, if enabled by its GPU. + pub(crate) fn status(&self) -> Option<&Arc> { + self.status.as_ref() + } + + fn covers_range_locked( + inner: &mut VmInner, + address: u64, + size: u64, + read: bool, + write: bool, + ) -> bool { + if size == 0 { + return false; + } + let Some(end) = address.checked_add(size) else { + return false; + }; + let mask = UAT_PGMSK as u64; + let start = address & !mask; + let end = if end & mask == 0 { + end + } else { + let Some(end) = (end | mask).checked_add(1) else { + return false; + }; + end + }; + if start < inner.va_range.start || end > inner.va_range.end { + return false; + } + inner + .page_table + .covers_range(start..end, read, write) + .unwrap_or(false) + } + + /// Checks all writable sparse blocks and their guard coverage under one + /// GPUVM lock. A fully mapped guard is not a valid sparse allocation. + pub(crate) fn covers_sparse_blocks_with_guards( + &self, + addresses: &[u64], + block_size: u64, + stride: u64, + ) -> bool { + if stride <= block_size { + return false; + } + let Ok(mut inner) = self.inner.exec_lock(None, false) else { + return false; + }; + addresses.iter().all(|&address| { + let Some(guard) = address.checked_add(block_size) else { + return false; + }; + Self::covers_range_locked(&mut inner, address, block_size, true, true) + && !Self::covers_range_locked( + &mut inner, guard, stride - block_size, false, false, + ) + }) + } + + /// Checks all exact request tuples under one GPUVM exec lock. Access bit0 + /// requests reading and bit1 writing. The epoch names this same snapshot. + pub(crate) fn covers_ranges_batch(&self, ranges: &[(u64, u64, u8)]) -> (bool, Option) { + let Ok(mut inner) = self.inner.exec_lock(None, false) else { + return (false, None); + }; + let epoch = inner.epoch; + let covered = ranges.iter().all(|&(address, size, access)| { + Self::covers_range_locked(&mut inner, address, size, access & 1 != 0, access & 2 != 0) + }); + (covered, epoch) + } + + /// Returns None while a leaf mutation is in progress or after epoch overflow. + pub(crate) fn mapping_validation_epoch(&self) -> Option { + self.mapping_epoch.as_ref()?.snapshot() + } /// Create a new virtual memory address space fn new( @@ -1716,13 +2856,13 @@ impl Vm { table } } else { - UatPageTable::new_with_ttb(ttb, iova_kern_range.clone(), cfg.ias, cfg.oas)? + UatPageTable::new_with_ttb_by_geometry(ttb, iova_kern_range.clone(), cfg.ias, cfg.oas)? } } else if cfg.reserved_tables { UatPageTable::new_m3_coherent(cfg.ias,cfg.oas,dev)? } else { - UatPageTable::new(cfg.ias, cfg.oas)? + UatPageTable::new_by_geometry(cfg.ias, cfg.oas)? }; let lower_start = IOVA_USER_BASE; @@ -1770,6 +2910,8 @@ impl Vm { is_kernel, fixed_slot: None, page_table, + mapping_epoch: None, + epoch: Some(0), mm, uat_inner, binding: binding_clone, @@ -1777,10 +2919,16 @@ impl Vm { }), )?, binding, + mapping_epoch: None, + lifetime: None, + residency: None, + context_bindings: None, + shared_bindings: None, + status: None, driver_mappings: if cfg.m3_owner() { Some(Arc::pin_init(new_mutex!(None, "VmDriverMappings"), GFP_KERNEL)?) } else { None }, - status: if cfg.m3_owner() { + agx_status: if cfg.m3_owner() { Some(Arc::new(crate::agx_status::VmStatus::new(), GFP_KERNEL)?) } else { None }, job_lifetime: if cfg.m3_owner() { @@ -1800,7 +2948,9 @@ impl Vm { /// Retain hardware-required fixed aliases for this VM's complete /// lifetime. Installing them before the VM reaches userspace also makes /// the GPUVM allocator reserve the addresses against user mappings. - pub(crate) fn install_driver_mappings( + /// + /// `install_driver_mappings` of every VM except published-roots (G17) ones. + fn m3_install_driver_mappings( &self, mappings: KVec, reserved_ranges: KVec>, @@ -1816,7 +2966,8 @@ impl Vm { Ok(()) } - pub(crate) fn driver_range_overlaps(&self, range: Range) -> bool { + /// `driver_range_overlaps` of every VM except published-roots (G17) ones. + fn m3_driver_range_overlaps(&self, range: Range) -> bool { let Some(driver_mappings) = &self.driver_mappings else { return false; }; driver_mappings.lock().as_ref().is_some_and(|driver| { any_range_overlaps( @@ -1834,7 +2985,9 @@ impl Vm { } /// Remove userspace mappings while preserving VM-lifetime driver aliases. - pub(crate) fn unmap_user_ranges( + /// + /// `unmap_user_ranges` of every VM except published-roots (G17) ones. + fn m3_unmap_user_ranges( &self, user_range: Range, kernel_range: Range, @@ -1915,12 +3068,7 @@ impl Vm { inner.covers_mapped_range(address, size, need_read, need_write) } - /// Resolve an IOVA through this VM's current page table. Callers use this - /// for mapping-identity checks only; it does not read the mapped page. - pub(crate) fn translate_iova(&self, address: u64) -> Result { - let mut inner = self.inner.exec_lock(None, false)?; - inner.page_table.translate_iova(address) - } + // `translate_iova` (the same exec-locked page-table lookup) is defined in `context.rs`. /// Read GPU-visible bytes out of this VM. /// @@ -2000,6 +3148,16 @@ impl Vm { prot: Prot, guard: bool, ) -> Result { + if self.published() { + return self.map_in_range_with_guard_size( + gem, + object_range, + alignment, + range, + prot, + if guard { UAT_PGSZ } else { 0 }, + ); + } let size = object_range.range(); let sgt = gem.owned_sg_table()?; let mut inner = self.inner.exec_lock(Some(gem), false)?; @@ -2041,8 +3199,8 @@ impl Vm { }; { let node_inner = node.as_mut().inner_mut(); - node_inner.bo = Some(vm_bo); - node_inner._gem = Some(gem.into()); + node_inner.bo = Some(MappingBo::synchronous(vm_bo)); + node_inner._gem = Some(MappingObject::Shared(gem.into())); } let ret = inner.map_node(&node, prot); @@ -2063,6 +3221,9 @@ impl Vm { prot: Prot, guard: bool, ) -> Result { + if self.published() { + return self.map_at_published(addr, size, gem, prot, guard); + } let sgt = gem.owned_sg_table()?; let mut inner = self.inner.exec_lock(Some(&gem), false)?; @@ -2100,8 +3261,8 @@ impl Vm { }; { let node_inner = node.as_mut().inner_mut(); - node_inner.bo = Some(vm_bo); - node_inner._gem = Some(gem.clone()); + node_inner.bo = Some(MappingBo::synchronous(vm_bo)); + node_inner._gem = Some(MappingObject::Shared(gem.clone())); } let ret = inner.map_node(&node, prot); @@ -2123,6 +3284,9 @@ impl Vm { prot: Prot, single_page: bool, ) -> Result { + if self.published() { + return self.bind_object_published(gem, addr, size, offset, prot, single_page); + } // Mapping needs a complete context let mut ctx = StepContext { new_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), @@ -2198,6 +3362,9 @@ impl Vm { size: usize, prot: Prot, ) -> Result { + if self.published() { + return self.map_io_published(iova, phys, size, prot); + } self.map_io_internal(Some(iova), 0..0, phys, size, prot) } @@ -2249,29 +3416,387 @@ impl Vm { mod_dev_dbg!(inner.dev, "MMU: IO map: {:#x}:{:#x} -> {:#x}\n", phys, size, iova); } - let prepared = if self.fault.is_some() { - inner.page_table.prepare_map(iova..iova.checked_add(size as u64).ok_or(EOVERFLOW)?) - } else { Ok(()) }; - if let Err(error) = prepared { - // Drop the VM lock before the reserved node and its owner refs. - core::mem::drop(inner); - return Err(error); - } + let prepared = if self.fault.is_some() { + inner.page_table.prepare_map(iova..iova.checked_add(size as u64).ok_or(EOVERFLOW)?) + } else { Ok(()) }; + if let Err(error) = prepared { + // Drop the VM lock before the reserved node and its owner refs. + core::mem::drop(inner); + return Err(error); + } + let ret = inner.page_table.map_pages( + iova..(iova + size as u64), + phys as PhysicalAddr, + prot, + false, + ); + // Drop the exec_lock first, so that if map_node failed the + // KernelMappingInner destructur does not deadlock. + core::mem::drop(inner); + ret?; + Ok(KernelMapping(ManuallyDrop::new(node))) + } + + /// Unmap everything in an address range. + pub(crate) fn unmap_range(&self, iova: u64, size: u64) -> Result { + if self.published() { + if self.defer_unmap(iova, size)? { + return Ok(()); + } + return self.unmap_range_commit(iova, size, true); + } + // Unmapping a range can only do a single split, so just preallocate + // the prev and next GpuVas + let mut ctx = StepContext { + prev_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), + next_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), + ..Default::default() + }; + + let mut inner = self.inner.exec_lock(None, false)?; + + mod_dev_dbg!(inner.dev, "MMU: sm_unmap: {:#x}:{:#x}\n", iova, size); + inner.sm_unmap(&mut ctx, iova, size) + } + + /// Drop mappings for a given bo. + /// + /// `drop_mappings` of every VM except published-roots (G17) ones. + fn m3_drop_mappings(&self, gem: &gem::Object) -> Result { + if let Some(job_lifetime) = &self.job_lifetime { + let mut state = job_lifetime.lock(); + if state.active != 0 { + if !state.closed_objects.iter().any(|object| core::ptr::eq(&**object, gem)) { + state.closed_objects.push(gem.into(), GFP_KERNEL)?; + } + return Ok(()); + } + } + + // Removing whole mappings only does unmaps, so no preallocated VAs + let mut ctx = Default::default(); + + let inner = self.inner.exec_lock(Some(gem), false)?; + + if let Some(bo) = self.inner.find_bo(gem) { + mod_dev_dbg!(inner.dev, "MMU: bo_unmap\n"); + self.inner.bo_unmap(&mut ctx, &bo)?; + mod_dev_dbg!(inner.dev, "MMU: bo_unmap done\n"); + // We need to drop the exec_lock first, then the GpuVmBo since that will take the lock itself. + core::mem::drop(inner); + core::mem::drop(bo); + } + + Ok(()) + } + + /// Reserves a whole-page unmapped guard without adding it to the mapped backing. + pub(crate) fn map_in_range_with_guard_size( + &self, + gem: &gem::Object, + object_range: Range, + alignment: u64, + range: Range, + prot: Prot, + guard_size: usize, + ) -> Result { + use kernel::drm::gem::BaseObject; + if guard_size % UAT_PGSZ != 0 { + return Err(EINVAL); + } + let size = object_range + .end + .checked_sub(object_range.start) + .ok_or(EINVAL)?; + if size == 0 || object_range.end > gem.size() { + return Err(EINVAL); + } + let reserved_size = size.checked_add(guard_size).ok_or(EOVERFLOW)?; + let _residency = self.enter_residency()?; + let sgt = crate::gem::held_sg_table(&gem)?; + let mut inner = self.inner.exec_lock(Some(gem), false)?; + let vm_bo = self.inner.obtain_bo(gem)?; + + let mut vm_bo_guard = vm_bo.inner().inner.lock(); + if vm_bo_guard.sgt.is_none() { + vm_bo_guard.sgt.replace(sgt); + } + core::mem::drop(vm_bo_guard); + + let mut vm_bo = Some(MappingBo::new(vm_bo, self.inner.clone())); + let uat_inner = inner.uat_inner.clone(); + // A failed reservation drops its payload while exec still owns GEM + // reservations. Attach BO/GEM references only after success, since + // their destructors may acquire the same reservation locks. + let result = inner.mm.insert_node_in_range( + KernelMappingInner { + owner: self.inner.clone(), + uat_inner, + prot, + bo: None, + _gem: None, + offset: object_range.start, + mapped_size: size, + }, + reserved_size as u64, + alignment, + 0, + range.start, + range.end, + mm::InsertMode::Best, + ); + let mut node = match result { + Ok(node) => node, + Err(error) => { + drop(inner); + return Err(error); + } + }; + { + let payload = node.as_mut().inner_mut(); + payload.bo = vm_bo.take(); + payload._gem = Some(MappingObject::Pinned(gem::KernelMappingPin::new(gem))); + } + + let ret = inner.map_node(&node, prot); + // Drop the exec_lock first, so that if map_node failed the + // KernelMappingInner destructur does not deadlock. + core::mem::drop(inner); + ret?; + Ok(KernelMapping(ManuallyDrop::new(node))) + } + + /// `map_at` of a published-roots (G17) VM. + #[allow(clippy::too_many_arguments)] + fn map_at_published( + &self, + addr: u64, + size: usize, + gem: ARef, + prot: Prot, + guard: bool, + ) -> Result { + let reserved_size = size + .checked_add(if guard { UAT_PGSZ } else { 0 }) + .ok_or(EOVERFLOW)?; + let _residency = self.enter_residency()?; + let sgt = crate::gem::held_sg_table(&gem)?; + let mut inner = self.inner.exec_lock(Some(&gem), false)?; + + let vm_bo = self.inner.obtain_bo(&gem)?; + + let mut vm_bo_guard = vm_bo.inner().inner.lock(); + if vm_bo_guard.sgt.is_none() { + vm_bo_guard.sgt.replace(sgt); + } + core::mem::drop(vm_bo_guard); + + let mut vm_bo = Some(MappingBo::new(vm_bo, self.inner.clone())); + let uat_inner = inner.uat_inner.clone(); + // Keep reservation-locking destructors out of the allocator payload + // until reservation succeeds. + let result = inner.mm.reserve_node( + KernelMappingInner { + owner: self.inner.clone(), + uat_inner, + prot, + bo: None, + _gem: None, + offset: 0, + mapped_size: size, + }, + addr, + reserved_size as u64, + 0, + ); + let mut node = match result { + Ok(node) => node, + Err(error) => { + drop(inner); + return Err(error); + } + }; + { + let payload = node.as_mut().inner_mut(); + payload.bo = vm_bo.take(); + payload._gem = Some(MappingObject::Pinned(gem::KernelMappingPin::new(&gem))); + } + + let ret = inner.map_node(&node, prot); + // Drop the exec_lock first, so that if map_node failed the + // KernelMappingInner destructur does not deadlock. + core::mem::drop(inner); + ret?; + Ok(KernelMapping(ManuallyDrop::new(node))) + } + + /// `bind_object` of a published-roots (G17) VM. + #[allow(clippy::too_many_arguments)] + fn bind_object_published( + &self, + gem: &gem::Object, + addr: u64, + size: u64, + offset: u64, + prot: Prot, + single_page: bool, + ) -> Result { + let _residency = self.enter_residency_metadata()?; + gem::validate_vm_binding(gem, self)?; + self.wait_for_user_map_admission()?; + // Mapping needs a complete context + let mut ctx = StepContext { + new_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), + prev_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), + next_va: Some(gpuvm::GpuVa::::new(pin_init::default())?), + prot, + ..Default::default() + }; + + let vm_bo = self.inner.obtain_bo(gem)?; + { + let needs_sgt = vm_bo.inner().inner.lock().sgt.is_none(); + if needs_sgt { + // Pin without the BO mutex: alias construction can hold the GEM + // reservation while acquiring this mutex. + let sgt = crate::gem::held_sg_table(&gem)?; + let mut vm_bo_guard = vm_bo.inner().inner.lock(); + if vm_bo_guard.sgt.is_none() { + if vm_bo_guard.sg_vec.is_none() { + let mut sg_vec = KVVec::new(); + let mut offset = 0; + for range in sgt.iter() { + let addr = range.dma_address() as usize; + let len = range.dma_len() as usize; + sg_vec.push((offset, addr..(addr + len)), GFP_KERNEL)?; + offset += len; + } + vm_bo_guard.sg_vec.replace(sg_vec); + } + vm_bo_guard.sgt.replace(sgt); + } + // Release the mutex before a losing initializer drops its SG + // owner. The declaration order also preserves this on errors. + drop(vm_bo_guard); + } + } + + let mut inner = self.inner.exec_lock(Some(gem), true)?; + + // Preallocate the page tables, to fail early if we ENOMEM + inner.page_table.alloc_pages(addr..(addr + size))?; + + ctx.vm_bo = Some(vm_bo); + + if (addr | size | offset) & (UAT_PGMSK as u64) != 0 { + dev_err!( + inner.dev.as_ref(), + "MMU: Map step {:#x} [{:#x}] -> {:#x} is not page-aligned\n", + offset, + size, + addr + ); + return Err(EINVAL); + } + + let (flags, gem_range) = if single_page { + (gpuvm::GpuVaFlags::REPEAT, UAT_PGSZ as u32) + } else { + (gpuvm::GpuVaFlags::NONE, 0u32) + }; + + mod_dev_dbg!( + inner.dev, + "MMU: sm_map: {:#x} [{:#x}] -> {:#x}\n", + offset, + size, + addr + ); + let _commit = self.mapping_commit()?; + inner.sm_map(&mut ctx, addr, size, offset, gem_range, flags) + } + + /// `map_io` of a published-roots (G17) VM. + fn map_io_published( + &self, + iova: u64, + phys: usize, + size: usize, + prot: Prot, + ) -> Result { + let mut inner = self.inner.exec_lock(None, false)?; + + if (iova as usize | phys | size) & UAT_PGMSK != 0 { + dev_err!( + inner.dev.as_ref(), + "MMU: KernelMapping {:#x}:{:#x} -> {:#x} is not page-aligned\n", + phys, + size, + iova + ); + return Err(EINVAL); + } + + dev_info!( + inner.dev.as_ref(), + "MMU: IO map: {:#x}:{:#x} -> {:#x}\n", + phys, + size, + iova + ); + + let uat_inner = inner.uat_inner.clone(); + let node = inner.mm.reserve_node( + KernelMappingInner { + owner: self.inner.clone(), + uat_inner, + prot, + bo: None, + _gem: None, + offset: 0, + mapped_size: size, + }, + iova, + size as u64, + 0, + )?; + + let end = iova.checked_add(size as u64).ok_or(EOVERFLOW)?; + inner.page_table.prepare_map(iova..end)?; + let mutation = inner.mapping_mutation(); let ret = inner.page_table.map_pages( - iova..(iova + size as u64), + iova..end, phys as PhysicalAddr, prot, false, ); + if ret.is_err() { + inner.rollback_failed_map(iova, size as u64); + } // Drop the exec_lock first, so that if map_node failed the // KernelMappingInner destructur does not deadlock. + drop(mutation); core::mem::drop(inner); ret?; Ok(KernelMapping(ManuallyDrop::new(node))) } - /// Unmap everything in an address range. - pub(crate) fn unmap_range(&self, iova: u64, size: u64) -> Result { + fn unmap_prepared_cleanup(&self, unmap: &mut bind::PreparedUserUnmap) -> Result { + let mut inner = self.inner.lock_inner(); + // SAFETY: step_unmap/step_remap modify this VM's translations and + // immediate GPUVA metadata only, using the retained split nodes. They + // do not access external BO backing and defer all BO reference puts. + let result = unsafe { inner.sm_unmap_inner(&mut unmap.ctx, unmap.iova, unmap.size) }; + if result.is_ok() { + self.untrack_shared_range(unmap.iova, unmap.size); + } + drop(inner); + self.bo_deferred_cleanup(); + result + } + + fn unmap_range_commit(&self, iova: u64, size: u64, user: bool) -> Result { + let _residency = self.enter_retirement_residency(); // Unmapping a range can only do a single split, so just preallocate // the prev and next GpuVas let mut ctx = StepContext { @@ -2283,38 +3808,75 @@ impl Vm { let mut inner = self.inner.exec_lock(None, false)?; mod_dev_dbg!(inner.dev, "MMU: sm_unmap: {:#x}:{:#x}\n", iova, size); - inner.sm_unmap(&mut ctx, iova, size) - } - - /// Drop mappings for a given bo. - pub(crate) fn drop_mappings(&self, gem: &gem::Object) -> Result { - if let Some(job_lifetime) = &self.job_lifetime { - let mut state = job_lifetime.lock(); - if state.active != 0 { - if !state.closed_objects.iter().any(|object| core::ptr::eq(&**object, gem)) { - state.closed_objects.push(gem.into(), GFP_KERNEL)?; + let _commit = if user { + match self.mapping_commit() { + Ok(guard) => guard, + Err(error) => { + drop(inner); + if self.defer_unmap(iova, size)? { + return Ok(()); + } + return Err(error); } - return Ok(()); } - } + } else { + None + }; + inner.sm_unmap(&mut ctx, iova, size)?; + self.untrack_shared_range(iova, size); + Ok(()) + } + fn drop_mappings_now(&self, gem: &gem::Object) -> Result { + if !self.published() { + return self.m3_drop_mappings(gem); + } + let _residency = self.enter_retirement_residency(); // Removing whole mappings only does unmaps, so no preallocated VAs let mut ctx = Default::default(); - let inner = self.inner.exec_lock(Some(gem), false)?; + let inner = match self.inner.exec_lock(Some(gem), false) { + Err(error) if error == ENOMEM => return self.drop_mappings_low_memory(gem), + result => result?, + }; if let Some(bo) = self.inner.find_bo(gem) { mod_dev_dbg!(inner.dev, "MMU: bo_unmap\n"); - self.inner.bo_unmap(&mut ctx, &bo)?; + let result = self.inner.bo_unmap(&mut ctx, &bo); + if result.is_ok() { + // The close callback or deferred object list retains this GEM, + // so clearing its candidate cannot release its final reference. + self.untrack_context_object(gem); + self.untrack_shared_object(gem); + } mod_dev_dbg!(inner.dev, "MMU: bo_unmap done\n"); - // We need to drop the exec_lock first, then the GpuVmBo since that will take the lock itself. + // BO release can drain unrelated deferred GEMs, including on an + // unmap error. Always release reservations before dropping it. core::mem::drop(inner); core::mem::drop(bo); + if result == Err(ENOMEM) { + return self.drop_mappings_low_memory(gem); + } + result?; } Ok(()) } + /// Preserve the indexed fast path normally; memory pressure must not lose + /// accepted object cleanup just because an exec or operation list failed. + fn drop_mappings_low_memory(&self, gem: &gem::Object) -> Result { + let mut inner = self.inner.lock_inner(); + let result = inner.unmap_object(gem, VmInner::unmap_gpuva); + if result.is_ok() { + self.untrack_context_object(gem); + self.untrack_shared_object(gem); + } + drop(inner); + self.bo_deferred_cleanup(); + result + } + /// Returns the dummy GEM object used to hold the shared DMA reservation locks pub(crate) fn get_resv_obj(&self) -> ARef { self.dummy_obj.clone() @@ -2331,8 +3893,63 @@ impl Vm { } } +impl VmInner { + /// `Drop` of a published-roots (G17) VM. + fn drop_published(&mut self) { + let mut binding = self.binding.lock(); + assert_eq!(binding.active_users, 0); + + mod_pr_debug!( + "VmInner::Drop [{}]: bind_token={:?}\n", + self.id, + binding.bind_token + ); + + // Make sure this VM is not mapped to a TTB if it was + if let Some(token) = binding.bind_token.take() { + let idx = (token.last_slot() as usize) + UAT_USER_CTX_START; + let ttb = self.ttb() | TTBR_VALID | (idx as u64) << TTBR_ASID_SHIFT; + + let uat_inner = self.uat_inner.lock(); + let handoff = uat_inner.handoff().lock_legacy(); + let handoff_cur = uat_inner.handoff().current_slot(); + let ttb_cur = uat_inner.ttbs()[idx].ttb0.load(Ordering::SeqCst); + let inval = ttb_cur == ttb; + if inval { + if handoff_cur == Some(idx as u32) { + pr_err!( + "VmInner::drop owning slot {}, but it is currently in use by the ASC?\n", + idx + ); + } + uat_inner.ttbs()[idx].ttb0.store(0, Ordering::SeqCst); + uat_inner.ttbs()[idx].ttb1.store(0, Ordering::SeqCst); + } + drop(handoff); + core::mem::drop(uat_inner); + + // In principle we dropped all the KernelMappings already, but we might as + // well play it safe and invalidate the whole ASID. + if inval { + mod_pr_debug!( + "VmInner::Drop [{}]: need inval for ASID {:#x}\n", + self.id, + idx + ); + mem::sync(); + mem::tlbi_asid(idx as u8); + mem::sync(); + } + } + } +} + impl Drop for VmInner { fn drop(&mut self) { + if self.published() { + self.drop_published(); + return; + } if self.uat_inner.fault.as_ref().is_some_and(|fault| fault.load(Ordering::Acquire)) { self.page_table.quarantine(); core::mem::forget(self.uat_inner.clone()); @@ -2452,10 +4069,47 @@ impl Uat { &self.kernel_lower_vm } + /// Installed descriptor and bind contexts rooted in this user VM. A graph + /// constructor checks this before changing a retained pool's mappings. + pub(crate) fn vm_context_mask(&self, vm: &Vm) -> Result { + if !matches!(self.inner.firmware, UatFirmware::PublishedRoots { .. }) { + return Err(EINVAL); + } + let root = { + let owner = vm.inner.exec_lock(None, false)?; + if owner.is_kernel || !Arc::ptr_eq(&owner.uat_inner, &self.inner) { + return Err(EINVAL); + } + owner.ttb() + }; + Ok(self.inner.lock().contexts_naming_root(root)) + } + + /// Publish a completed alias set before its descriptors become visible. + /// Match actual roots, including independently allocated execution contexts. + pub(crate) fn flush_vm_contexts(&self, vm: &Vm) -> Result { + let contexts = self.vm_context_mask(vm)?; + if contexts == 0 { + return Err(ENOENT); + } + mem::sync(); + for context in 0..UAT_NUM_CTX { + if contexts & (1u64 << context) != 0 { + mem::tlbi_asid(context as u8); + } + } + mem::sync(); + Ok(contexts) + } + #[cfg(CONFIG_DEV_COREDUMP)] pub(crate) fn dump_kernel_pages(&self) -> Result> { // Full upper-half range for this SoC's root geometry. - let full_range = uat_geometry(self.cfg)?.upper_canonical_base()..(!UAT_PGMSK as u64); + let full_range = if self.inner.firmware != UatFirmware::Handoff { + self.geometry.upper_range() + } else { + uat_geometry(self.cfg)?.upper_canonical_base()..(!UAT_PGMSK as u64) + }; let mut inner = self.kernel_vm.inner.exec_lock(None, false)?; inner.page_table.dump_pages(full_range) } @@ -2574,8 +4228,181 @@ impl Uat { Ok(VmBind(vm.clone(), slot, binding.binding.as_ref().unwrap().generation)) } + /// Allocates a stable application context without changing the VM bind slot. + /// Both roots and the generation remain unchanged for the lease's lifetime. + pub(crate) fn new_execution_context(&self, vm: &Vm) -> Result> { + if self.inner.firmware == UatFirmware::Handoff { + return Err(EINVAL); + } + { + let owner = vm.inner.exec_lock(None, false)?; + if owner.is_kernel + || !core::ptr::eq(Arc::as_ptr(&owner.uat_inner), Arc::as_ptr(&self.inner)) + { + return Err(EINVAL); + } + } + let (id, generation) = { + let mut inner = self.inner.lock(); + let free = !inner.execution_contexts & EXECUTION_CONTEXT_MASK; + if free == 0 { + return Err(ENOSPC); + } + let id = free.trailing_zeros() as usize; + inner.execution_contexts |= 1u64 << id; + inner.context_generations[id] = inner.context_generations[id].wrapping_add(1); + (id, inner.context_generations[id]) + }; + let low = tagged_root(vm.ttb(), id); + let high = tagged_root(self.kernel_vm.ttb(), id); + // Allocate the complete host owner before either TTBAT half is published. + // ExecutionRoot releases an empty claimed slot if allocation fails. + let context = Arc::new( + ExecutionContext { + root: ExecutionRoot { + released: AtomicBool::new(false), + inner: self.inner.clone(), + id: id as u8, + generation, + low, + high, + }, + vm: vm.clone(), + }, + GFP_KERNEL, + )?; + { + let inner = self.inner.lock(); + let roots = &inner.ttbs()[id]; + if roots.ttb0.load(Ordering::Acquire) != 0 || roots.ttb1.load(Ordering::Acquire) != 0 { + return Err(EBUSY); + } + roots.ttb0.store(low, Ordering::Release); + roots.ttb1.store(high, Ordering::Release); + } + mem::sync(); + mem::tlbi_asid(id as u8); + mem::sync(); + Ok(context) + } + + /// Returns the address space geometry of this UAT. + pub(crate) fn geometry(&self) -> UatGeometry { + self.geometry + } + + /// Adopts the top-level entries that the primary firmware instance published in the kernel + /// root while booting ([`UatFirmware::PublishedRoots`]). + /// + /// The entries below the kernel window belong to the firmware, and the kernel window entry + /// points at a table that the bootloader reserved, so the driver maps kernel objects beneath + /// it and never frees it. Entries above the kernel window do not belong to anyone and are + /// cleared. Must run after the primary instance booted and before anything is mapped into the + /// kernel `Vm`. + pub(crate) fn adopt_firmware_root_entries(&self) -> Result<[u64; 3]> { + let owned = IOVA_KERN_TOP_LEVEL_INDEX as usize + 1; + let mut snapshot = [0; 3]; + // SAFETY: The kernel root is the bootloader-reserved `pagetables` region, which outlives + // this `Uat`. + unsafe { + pgtable::with_root_entries(self.kernel_vm.ttb(), UAT_PGSZ / size_of::(), |ptes| { + for (value, pte) in snapshot.iter_mut().zip(ptes) { + *value = pte.load(Ordering::Relaxed); + } + for pte in &ptes[owned..] { + pte.store(0, Ordering::Relaxed); + } + }) + }?; + + mem::sync(); + mem::tlbi_all(); + mem::sync(); + Ok(snapshot) + } + + /// Checks that allocation under the shared subtree did not replace a firmware-owned entry. + pub(crate) fn confirm_firmware_root_entries(&self, snapshot: [u64; 3]) -> Result { + // SAFETY: The root is a bootloader-reserved page retained for this UAT's lifetime. + let matches = unsafe { + pgtable::with_root_entries(self.kernel_vm.ttb(), snapshot.len(), |ptes| { + ptes.iter() + .zip(snapshot) + .all(|(pte, expected)| pte.load(Ordering::Relaxed) == expected) + }) + }?; + if !matches { + return Err(EIO); + } + Ok(()) + } + + /// Copies the kernel root's top-level entries from the kernel window up into the root of the + /// secondary firmware instance ([`UatFirmware::PublishedRoots`]), whose lower entries are + /// private to that instance. + pub(crate) fn mirror_secondary_root(&self) -> Result { + let secondary = self.inner.lock().secondary_ttb1.ok_or(EINVAL)?; + let count = UAT_PGSZ / size_of::(); + let first = IOVA_KERN_TOP_LEVEL_INDEX as usize; + + // SAFETY: Both roots are in the bootloader-reserved `pagetables` region, which outlives + // this `Uat`. + unsafe { + pgtable::with_root_entries(self.kernel_vm.ttb(), count, |primary| { + pgtable::with_root_entries(secondary, count, |secondary| { + for (src, dst) in primary.iter().zip(secondary).skip(first) { + let pte = src.load(Ordering::Relaxed); + if pte != 0 && dst.load(Ordering::Relaxed) != pte { + dst.store(pte, Ordering::Relaxed); + } + } + }) + }) + }??; + + mem::sync(); + mem::tlbi_all(); + mem::sync(); + Ok(()) + } + + /// Activates the device-global render root in context 1 after the boot records retire. + /// Both contexts must still contain their original kernel roots. + pub(crate) fn activate_render_context(&self, vm: &Vm) -> Result { + let inner = self.inner.lock(); + let handoff = inner.handoff().lock_legacy(); + let ttbs = inner.ttbs(); + let ready = [0, KERNEL_ALIAS_CTX].into_iter().all(|ctx| { + ttbs[ctx].ttb0.load(Ordering::Acquire) == tagged_root(self.kernel_lower_vm.ttb(), ctx) + && ttbs[ctx].ttb1.load(Ordering::Acquire) == tagged_root(self.kernel_vm.ttb(), ctx) + }); + if !ready { + drop(handoff); + return Err(EIO); + } + ttbs[KERNEL_ALIAS_CTX] + .ttb0 + .store(tagged_root(vm.ttb(), KERNEL_ALIAS_CTX), Ordering::Release); + drop(handoff); + core::mem::drop(inner); + mem::sync(); + mem::tlbi_asid(KERNEL_ALIAS_CTX as u8); + mem::sync(); + Ok(()) + } + /// Creates a new `Vm` linked to this UAT. pub(crate) fn new_vm(&self, id: u64, kernel_range: Range) -> Result { + if self.inner.firmware != UatFirmware::Handoff { + return Vm::new_published( + &self.dev, + self.inner.clone(), + kernel_range, + self.geometry, + None, + id, + ); + } Vm::new( &self.dev, self.inner.clone(), @@ -2588,7 +4415,12 @@ impl Uat { /// Creates the reference-counted inner data for a new `Uat` instance. #[inline(never)] - fn make_inner(dev: &driver::AsahiDevice, handoff_mode: HandoffMode, m3: bool) -> Result> { + fn make_inner( + dev: &driver::AsahiDevice, + handoff_mode: HandoffMode, + m3: bool, + firmware: UatFirmware, + ) -> Result> { let cached = !matches!( handoff_mode, HandoffMode::StoppedFirmwareT6030 | HandoffMode::FirmwareM3 | HandoffMode::FirmwareG16 ); @@ -2610,6 +4442,7 @@ impl Uat { let shared_fault = fault.clone(); Arc::pin_init( try_pin_init!(UatInner { + firmware, m3_running: handoff_mode == HandoffMode::FirmwareM3, fault, firmware_cache_flush_ready: AtomicBool::new( @@ -2622,6 +4455,9 @@ impl Uat { UatShared { fault: shared_fault, kernel_ttb1: 0, + execution_contexts: !EXECUTION_CONTEXT_MASK, + context_generations: [0; UAT_NUM_CTX], + secondary_ttb1: None, map_kernel_to_user: false, handoff_rgn, ttbs_rgn, @@ -2681,7 +4517,7 @@ impl Uat { } #[inline(never)] - pub(crate) fn new( + pub(crate) fn new_for_config( dev: &driver::AsahiDevice, cfg: &'static hw::HwConfig, map_kernel_to_user: bool, @@ -2748,7 +4584,7 @@ impl Uat { } - let inner = Self::make_inner(dev, handoff_mode, cfg.m3_owner())?; + let inner = Self::make_inner(dev, handoff_mode, cfg.m3_owner(), UatFirmware::Handoff)?; let of_node = dev.as_ref().of_node().ok_or(EINVAL)?; let res = if cfg.m3_owner() { @@ -2791,6 +4627,10 @@ impl Uat { let uat = Self { dev: dev.into(), cfg, + geometry: UatGeometry { + ias: u32::from(cfg.ias), + oas: cfg.oas, + }, kernel_vm, kernel_lower_vm, inner, @@ -2809,7 +4649,7 @@ impl Uat { inner.map_kernel_to_user = map_kernel_to_user; inner.kernel_ttb1 = ttb1; - inner.handoff().init(handoff_mode, cfg.m3_owner())?; + inner.handoff().init(handoff_mode, cfg.m3_owner(), UatFirmware::Handoff)?; dev_info!(dev.as_ref(), "MMU: Initializing TTBs\n"); @@ -2833,10 +4673,127 @@ impl Uat { Ok(uat) } + + /// Creates a new published-roots (G17) `Uat` for the given address space geometry. + #[inline(never)] + pub(crate) fn new( + dev: &driver::AsahiDevice, + geometry: UatGeometry, + map_kernel_to_user: bool, + firmware: UatFirmware, + ) -> Result { + dev_info!(dev.as_ref(), "MMU: Initializing...\n"); + + if firmware == UatFirmware::Handoff { + // Handoff UATs are built by `new_for_config` and its siblings. + return Err(EINVAL); + } + let cfg = UatConfig::published(geometry)?; + let inner = Self::make_inner(dev, HandoffMode::FirmwareDekker, false, firmware)?; + + let of_node = dev.as_ref().of_node().ok_or(EINVAL)?; + let res = of_node.reserved_mem_region_to_resource_byname(c_str!("pagetables"))?; + let ttb1 = res.start(); + let ttb1size: usize = res.size().try_into()?; + + let (pagetables_size, secondary_ttb1) = match firmware { + UatFirmware::Handoff => (PAGETABLES_SIZE, None), + UatFirmware::PublishedRoots { + secondary_root_offset, + } => ( + secondary_root_offset as usize + UAT_PGSZ, + Some(ttb1 + secondary_root_offset), + ), + }; + if ttb1size < pagetables_size { + dev_err!(dev.as_ref(), "MMU: Pagetables region is too small\n"); + return Err(ENOMEM); + } + + dev_info!(dev.as_ref(), "MMU: Creating kernel page tables\n"); + let kernel_lower_vm = + Vm::new_published(dev, inner.clone(), geometry.user_range(), geometry, None, 1)?; + let kernel_vm = Vm::new_published( + dev, + inner.clone(), + geometry.kernel_range(), + geometry, + Some(ttb1), + 0, + )?; + + dev_info!(dev.as_ref(), "MMU: Kernel page tables created\n"); + + let ttb0 = kernel_lower_vm.ttb(); + + let uat = Self { + dev: dev.into(), + cfg, + geometry, + kernel_vm, + kernel_lower_vm, + inner, + slots: slotalloc::SlotAllocator::new( + UAT_USER_CTX as u32, + (), + |_inner, _slot| Some(SlotInner { generation: 0 }), + c_str!("Uat::SlotAllocator"), + static_lock_class!(), + static_lock_class!(), + )?, + }; + + let mut inner = uat.inner.lock(); + + inner.map_kernel_to_user = map_kernel_to_user; + inner.kernel_ttb1 = ttb1; + inner.secondary_ttb1 = secondary_ttb1; + + inner.handoff().init(HandoffMode::FirmwareDekker, false, firmware)?; + + dev_info!(dev.as_ref(), "MMU: Initializing TTBs\n"); + + let handoff = inner.handoff().lock_legacy(); + + let ttbs = inner.ttbs(); + + ttbs[0].ttb0.store(ttb0 | TTBR_VALID, Ordering::SeqCst); + ttbs[0].ttb1.store(ttb1 | TTBR_VALID, Ordering::SeqCst); + + for ctx in &ttbs[1..] { + ctx.ttb0.store(0, Ordering::Relaxed); + ctx.ttb1.store(0, Ordering::Relaxed); + } + + if secondary_ttb1.is_some() { + let alias = &ttbs[KERNEL_ALIAS_CTX]; + alias + .ttb0 + .store(tagged_root(ttb0, KERNEL_ALIAS_CTX), Ordering::SeqCst); + alias + .ttb1 + .store(tagged_root(ttb1, KERNEL_ALIAS_CTX), Ordering::SeqCst); + } + + drop(handoff); + + core::mem::drop(inner); + + dev_info!(dev.as_ref(), "MMU: initialized\n"); + + Ok(uat) + } } impl Drop for Uat { fn drop(&mut self) { + if self.inner.firmware != UatFirmware::Handoff { + // Make sure we flush the TLBs + mem::sync(); + mem::tlbi_all(); + mem::sync(); + return; + } if self.cfg.reserved_tables { self.inner.firmware_cache_flush_ready.store(false, Ordering::Release); let inner = self.inner.lock(); diff --git a/drivers/gpu/drm/asahi-neo/mmu/bind.rs b/drivers/gpu/drm/asahi/mmu/bind.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/mmu/bind.rs rename to drivers/gpu/drm/asahi/mmu/bind.rs index adbb609d5a88ce..824a960f6530b5 100644 --- a/drivers/gpu/drm/asahi-neo/mmu/bind.rs +++ b/drivers/gpu/drm/asahi/mmu/bind.rs @@ -81,7 +81,7 @@ impl Vm { if needs_sgt { // Pin without the BO mutex: alias construction can hold the GEM // reservation while acquiring this mutex. - let sgt = gem.owned_sg_table()?; + let sgt = crate::gem::held_sg_table(&gem)?; let mut bo = vm_bo.inner().inner.lock(); if bo.sgt.is_none() { if bo.sg_vec.is_none() { diff --git a/drivers/gpu/drm/asahi-neo/mmu/context.rs b/drivers/gpu/drm/asahi/mmu/context.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/mmu/context.rs rename to drivers/gpu/drm/asahi/mmu/context.rs index e41c73ce295af7..efebcc6a5a84fe 100644 --- a/drivers/gpu/drm/asahi-neo/mmu/context.rs +++ b/drivers/gpu/drm/asahi/mmu/context.rs @@ -9,7 +9,7 @@ use super::*; use crate::hw::t8140::CONTEXT_VIEWS; -use kernel::drm_neo::gem::BaseObject; +use kernel::drm::gem::BaseObject; struct Binding { gem: ARef, diff --git a/drivers/gpu/drm/asahi-neo/mmu/lifetime.rs b/drivers/gpu/drm/asahi/mmu/lifetime.rs similarity index 93% rename from drivers/gpu/drm/asahi-neo/mmu/lifetime.rs rename to drivers/gpu/drm/asahi/mmu/lifetime.rs index d53c19b93bfdcd..42ac32bdae1302 100644 --- a/drivers/gpu/drm/asahi-neo/mmu/lifetime.rs +++ b/drivers/gpu/drm/asahi/mmu/lifetime.rs @@ -134,6 +134,9 @@ impl Vm { mappings: KVec, reserved: KVec>, ) -> Result { + if !self.published() { + return self.m3_install_driver_mappings(mappings, reserved); + } let lifetime = self.lifetime.as_ref().ok_or(EINVAL)?; let mut driver = lifetime.driver.lock(); if driver.is_some() { @@ -238,6 +241,40 @@ impl Vm { Ok(()) } + /// Moves the aliases of the last `blocks - kept` heap blocks of the exact + /// cached owner into `out`, which the caller drops after its locks. The + /// growth path appends one alias per added block, so these are the + /// blocks a shrink removed. Nothing moves unless all of them fit in `out`. + pub(crate) fn truncate_render_pool_mappings( + &self, + slot: u8, + id: u64, + blocks: usize, + kept: usize, + out: &mut KVec, + ) -> Result { + let count = blocks.checked_sub(kept).ok_or(EINVAL)?; + if id == 0 || count == 0 || out.capacity() - out.len() < count { + return Err(EINVAL); + } + let lifetime = self.lifetime.as_ref().ok_or(EINVAL)?; + let mut driver = lifetime.driver.lock(); + let pool = Option::as_mut(&mut *driver) + .and_then(|driver| driver.render.get_mut(usize::from(slot))) + .and_then(Option::as_mut) + .filter(|pool| pool.id == id) + .ok_or(EINVAL)?; + if pool.blocks != blocks || pool.mappings.len() < count { + return Err(EBUSY); + } + for _ in 0..count { + let mapping = pool.mappings.pop().ok_or(EIO)?; + out.push_within_capacity(mapping).map_err(|_| EIO)?; + } + pool.blocks = kept; + Ok(()) + } + /// A delayed teardown must not remove a successor generation. Remove /// under the cache lock, then unmap after dropping that lock. pub(crate) fn clear_render_pool_mappings_if_owner(&self, slot: u8, id: u64) { @@ -301,6 +338,9 @@ impl Vm { /// Tests mappings and reservations which userspace must never replace. pub(crate) fn driver_range_overlaps(&self, range: Range) -> bool { + if !self.published() { + return self.m3_driver_range_overlaps(range); + } let Some(lifetime) = self.lifetime.as_ref() else { return false; }; @@ -527,6 +567,9 @@ impl Vm { /// Closes admission permanently and preserves mappings referenced by accepted jobs. pub(crate) fn unmap_user_ranges(&self, user: Range, kernel: Range) -> Result { + if !self.published() { + return self.m3_unmap_user_ranges(user, kernel); + } if let Some(lifetime) = self.lifetime.as_ref() { let mut state = lifetime.state.lock(); state.blocked = true; diff --git a/drivers/gpu/drm/asahi-neo/mmu/residency.rs b/drivers/gpu/drm/asahi/mmu/residency.rs similarity index 99% rename from drivers/gpu/drm/asahi-neo/mmu/residency.rs rename to drivers/gpu/drm/asahi/mmu/residency.rs index 3a76f07577d2b6..afc9255100fa2b 100644 --- a/drivers/gpu/drm/asahi-neo/mmu/residency.rs +++ b/drivers/gpu/drm/asahi/mmu/residency.rs @@ -6,7 +6,7 @@ //! Its writer never waits for active readers: reclaim skips the VM instead. use super::*; -use kernel::drm_neo::gem::BaseObject; +use kernel::drm::gem::BaseObject; use kernel::{new_condvar, sync::CondVar}; /// Re-evicting backing that a job just had to restore only repeats the @@ -440,6 +440,7 @@ impl Vm { bo_inner.sgt = None; } if object.purge_locked().is_ok() { + crate::gem::set_mapping_held(object, false); purged = purged.saturating_add(object.size() / kernel::page::PAGE_SIZE); } else { gate.note_evicted(); @@ -519,6 +520,7 @@ impl Vm { // tables above. Kernel mappings and other SG owners are excluded; // the C/Rust backing helper rechecks exclusion before release. if object.release_idle_pages_locked().is_ok() { + crate::gem::set_mapping_held(object, false); unpinned = unpinned.saturating_add(object.size() / kernel::page::PAGE_SIZE); } Ok(unpinned < target) @@ -566,7 +568,7 @@ impl Vm { if needs_sgt { // Preserve review1's reservation/BO lock order: the getter // may acquire the object reservation; never hold BO here. - let sgt = map.object.owned_sg_table()?; + let sgt = crate::gem::held_sg_table(&map.object)?; let mut bo = bo_owner.inner().inner.lock(); if bo.sgt.is_none() { bo.sgt = Some(sgt); } drop(bo); diff --git a/drivers/gpu/drm/asahi-neo/mmu/shared.rs b/drivers/gpu/drm/asahi/mmu/shared.rs similarity index 100% rename from drivers/gpu/drm/asahi-neo/mmu/shared.rs rename to drivers/gpu/drm/asahi/mmu/shared.rs diff --git a/drivers/gpu/drm/asahi/pgtable.rs b/drivers/gpu/drm/asahi/pgtable.rs index 8ac13d5bd2578a..534af29461baf6 100644 --- a/drivers/gpu/drm/asahi/pgtable.rs +++ b/drivers/gpu/drm/asahi/pgtable.rs @@ -55,6 +55,35 @@ const UAT_LVMSK: u64 = (UAT_LVSZ - 1) as u64; const UAT_LEVELS: usize = crate::uat::UAT_LEVELS; +/// Number of address bits below the index of a top-level page table entry. +pub(crate) const UAT_TOP_LEVEL_SHIFT: usize = UAT_PGBIT + (UAT_LEVELS - 1) * UAT_LVBIT; + +/// Runs `f` on the first `count` top-level entries of the root page table at `ttb`. +/// +/// # Safety +/// +/// `ttb` must be the physical address of a page table root in RAM that stays allocated while `f` +/// runs. Other agents (the GPU firmware) may access the entries concurrently, which is why they +/// are only exposed as atomics. +pub(crate) unsafe fn with_root_entries( + ttb: PhysicalAddr, + count: usize, + f: impl FnOnce(&[AtomicU64]) -> R, +) -> Result { + if count > UAT_NPTE { + return Err(EINVAL); + } + // SAFETY: The caller guarantees that `ttb` is an allocated page in RAM. + let page = unsafe { Page::borrow_phys(&ttb) }.ok_or(EIO)?; + page.with_pointer_into_page(0, count * PTE_SIZE, |p| { + // SAFETY: with_pointer_into_page() ensures the pointer is valid for `count` PTEs, and page + // table entries are naturally aligned. + Ok(f(unsafe { + core::slice::from_raw_parts(p as *const AtomicU64, count) + })) + }) +} + const PTE_TYPE_BITS: u64 = 3; const PTE_TYPE_LEAF_TABLE: u64 = 3; @@ -365,10 +394,39 @@ pub(crate) struct UatPageTable { geometry: UatGeometry, oas_mask: u64, coverage: Option>, + /// G17 tables (see [`Self::new`]): every walk, including those of 42-bit roots, uses the + /// ordered walk with the leaf-level semantics of the 39-bit walk. + ordered_walk: bool, + /// Tables below a root the driver did not allocate may belong to the bootloader or the + /// firmware, so only the tables recorded here are ever freed. `None` if the driver owns the + /// tables below the root. Only used by the ordered walk. + driver_tables: Option>, } impl UatPageTable { - pub(crate) fn new(ias: u8, oas: u32) -> Result { + /// Allocate a root for a G17 user VM. Its tables use the ordered walk. + pub(crate) fn new(ias: u32, oas: u32) -> Result { + mod_pr_debug!("UATPageTable::new: ias={} oas={}\n", ias, oas); + let geometry = UatGeometry::new(u8::try_from(ias)?).ok_or(EINVAL)?; + let ttb_page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?; + let ttb = Page::into_phys(ttb_page); + Ok(UatPageTable { + ttb, + ttb_owned: true, + quarantined: false, + noncoherent: false, + dma_tables: None, + reserved_tables: None, + va_range: 0..(1u64 << ias), + geometry, + oas_mask: (1u64 << oas) - 1, + coverage: None, + ordered_walk: true, + driver_tables: None, + }) + } + + pub(crate) fn new_by_geometry(ias: u8, oas: u32) -> Result { mod_pr_debug!("UATPageTable::new: ias={} oas={}\n", ias, oas); let geometry = UatGeometry::new(ias).ok_or(EINVAL)?; let ttb_page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?; @@ -384,11 +442,13 @@ impl UatPageTable { geometry, oas_mask: (1u64 << oas) - 1, coverage: None, + ordered_walk: false, + driver_tables: None, }) } pub(crate) fn new_m3_coherent(ias:u8,oas:u32,dev:&crate::driver::AsahiDevice)->Result { - let mut table=Self::new(ias,oas)?; + let mut table=Self::new_by_geometry(ias,oas)?; let dma=DmaTables::new(dev);let root=dma.alloc()?; // SAFETY: this newly allocated, unpublished root is still empty. unsafe {Page::from_phys(table.ttb)}; @@ -400,6 +460,10 @@ impl UatPageTable { /// mapped memory. This is used to prove that separately named aliases /// refer to the same backing page before firmware can consume them. pub(crate) fn translate_iova(&mut self, iova: u64) -> Result { + if self.ordered_walk { + let entry = self.leaf_entry(iova & !(UAT_PGMSK as u64))?; + return Ok((entry & self.oas_mask & !(UAT_PGMSK as u64)) | (iova & UAT_PGMSK as u64)); + } let page_mask = UAT_PGMSK as u64; let page = iova & !page_mask; let end = page.checked_add(UAT_PGSZ as u64).ok_or(EOVERFLOW)?; @@ -417,7 +481,55 @@ impl UatPageTable { translated.ok_or(EFAULT) } + /// Borrow a root; track individual child ownership only when other agents + /// can install tables below it. Used for G17 kernel roots (ordered walk). pub(crate) fn new_with_ttb( + ttb: PhysicalAddr, + va_range: Range, + ias: u32, + oas: u32, + shared_tables: bool, + ) -> Result { + mod_pr_debug!( + "UATPageTable::new_with_ttb: ttb={:#x} range={:#x?} ias={} oas={}\n", + ttb, + va_range, + ias, + oas + ); + let geometry = UatGeometry::new(u8::try_from(ias)?).ok_or(EINVAL)?; + if ttb & (UAT_PGMSK as PhysicalAddr) != 0 { + return Err(EINVAL); + } + if (va_range.start | va_range.end) & (UAT_PGMSK as u64) != 0 { + return Err(EINVAL); + } + // SAFETY: The bootloader-managed root stays valid while this page table exists. + if unsafe { Page::borrow_phys(&ttb) }.is_none() { + pr_err!( + "UATPageTable::new_with_ttb: ttb at {:#x} is not mapped (DT using no-map?)\n", + ttb + ); + return Err(EIO); + } + + Ok(UatPageTable { + ttb, + ttb_owned: false, + quarantined: false, + noncoherent: false, + dma_tables: None, + reserved_tables: None, + va_range, + geometry, + oas_mask: (1u64 << oas) - 1, + coverage: None, + ordered_walk: true, + driver_tables: shared_tables.then(KVec::new), + }) + } + + pub(crate) fn new_with_ttb_by_geometry( ttb: PhysicalAddr, va_range: Range, ias: u8, @@ -457,6 +569,8 @@ impl UatPageTable { geometry, oas_mask: (1u64 << oas) - 1, coverage: None, + ordered_walk: false, + driver_tables: None, }) } @@ -491,6 +605,8 @@ impl UatPageTable { geometry, oas_mask: (1u64 << oas) - 1, coverage: None, + ordered_walk: false, + driver_tables: None, }) } @@ -508,7 +624,8 @@ impl UatPageTable { })?; crate::mem::tlbi_all();crate::mem::sync(); Ok(Self {ttb,ttb_owned:false,quarantined:false,noncoherent:true,dma_tables:Some(DmaTables::new(dev)), - reserved_tables:Some(tables),va_range,geometry,oas_mask:(1u64<<42)-1,coverage:None}) + reserved_tables:Some(tables),va_range,geometry,oas_mask:(1u64<<42)-1,coverage:None, + ordered_walk:false,driver_tables:None}) } /// Adopt the loader-designated shared middle table before ASC starts. @@ -629,6 +746,244 @@ impl UatPageTable { self.ttb } + /// INT's 42-bit (AGX3) walk with prepared, non-allocating leaf writes. 39-bit roots and G17 + /// tables use the leaf semantics of the 39-bit walk instead. + fn agx3_walk(&self) -> bool { + !self.ordered_walk && self.geometry.root_size() != 1u64 << 39 + } + + /// The G17 walk (`ordered_walk`): records driver-allocated tables below shared roots, + /// completes child-table initialization before linking it, and detaches tables bottom-up. + fn with_pages_ordered( + &mut self, + iova_range: Range, + alloc: bool, + free: bool, + mut cb: F, + ) -> Result + where + F: FnMut(u64, &[Pte]) -> Result, + { + mod_pr_debug!( + "UATPageTable::with_pages: {:#x?} alloc={} free={}\n", + iova_range, + alloc, + free + ); + if (iova_range.start | iova_range.end) & (UAT_PGMSK as u64) != 0 { + pr_err!( + "UATPageTable::with_pages: iova range not aligned: {:#x?}\n", + iova_range + ); + return Err(EINVAL); + } + + if iova_range.is_empty() { + return Ok(()); + } + + let mut iova = iova_range.start & self.geometry.root_mask(); + let mut last_iova = iova; + // Handle the case where iova_range.end is just at the top boundary of the IAS + let end = ((iova_range.end - 1) & self.geometry.root_mask()) + 1; + + let mut pt_addr: [Option; UAT_LEVELS] = Default::default(); + // For every child table, retain the parent page and entry that names it. + // Children are detached before their parents are freed, so these borrowed + // parent pages remain valid while the descriptor is cleared. + let mut pt_parent: [Option<(PhysicalAddr, usize)>; UAT_LEVELS - 1] = Default::default(); + pt_addr[UAT_LEVELS - 1] = Some(self.ttb); + + 'outer: while iova < end { + mod_pr_debug!("UATPageTable::with_pages: iova={:#x}\n", iova); + let addr_diff = last_iova ^ iova; + // Detach leaves before intermediate tables. The old order freed a + // parent first and could leave its descriptor naming a freed child. + for level in 0..UAT_LEVELS - 1 { + // If the iova has changed at this level or above, invalidate the physaddr + if addr_diff & !((1 << (UAT_PGBIT + (level + 1) * UAT_LVBIT)) - 1) != 0 { + if let Some(phys) = pt_addr[level].take() { + if free { + mod_pr_debug!( + "UATPageTable::with_pages: free level {} {:#x?}\n", + level, + phys + ); + let parent = pt_parent[level].take(); + self.free_detached_table(phys, parent)?; + } else { + pt_parent[level] = None; + } + mod_pr_debug!("UATPageTable::with_pages: invalidate level {}\n", level); + } + } + } + last_iova = iova; + for level in (0..UAT_LEVELS - 1).rev() { + // Fetch the page table base address for this level + if pt_addr[level].is_none() { + let phys = pt_addr[level + 1].unwrap(); + mod_pr_debug!( + "UATPageTable::with_pages: need level {}, parent phys {:#x}\n", + level, + phys + ); + let upidx = ((iova >> (UAT_PGBIT + (level + 1) * UAT_LVBIT) as u64) & UAT_LVMSK) + as usize; + // SAFETY: Page table addresses are either allocated by us, or + // firmware-managed and safe to borrow a struct page from. + let upt = unsafe { Page::borrow_phys_unchecked(&phys) }; + mod_pr_debug!("UATPageTable::with_pages: borrowed phys {:#x}\n", phys); + pt_addr[level] = + upt.with_pointer_into_page(upidx * PTE_SIZE, PTE_SIZE, |p| { + let uptep = p as *const _ as *const Pte; + // SAFETY: with_pointer_into_page() ensures the pointer is valid, + // and our index is aligned so it is safe to deref as an AtomicU64. + let upte = unsafe { &*uptep }; + let mut upte_val = upte.load(Ordering::Relaxed); + // Allocate if requested + if upte_val == 0 && alloc { + let pt_page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?; + mod_pr_debug!("UATPageTable::with_pages: alloc PT at {:#x}\n", pt_page.phys()); + if let Some(tables) = self.driver_tables.as_mut() { + tables.push(pt_page.phys(), GFP_KERNEL)?; + } + let pt_paddr = Page::into_phys(pt_page); + upte_val = pt_paddr | PTE_TYPE_LEAF_TABLE; + // Complete child-table initialization before a + // concurrent device walk can follow its parent. + // A later mapping/TLBI barrier cannot order an + // already-visible parent against earlier clears. + crate::mem::sync(); + upte.store(upte_val, Ordering::Relaxed); + } + if upte_val & PTE_TYPE_BITS == PTE_TYPE_LEAF_TABLE { + Ok(Some(upte_val & self.oas_mask & (!UAT_PGMSK as u64))) + } else if upte_val == 0 || (!alloc && !free) { + mod_pr_debug!("UATPageTable::with_pages: no level {}\n", level); + Ok(None) + } else { + pr_err!("UATPageTable::with_pages: Unexpected Table PTE value {:#x} at iova {:#x} index {} phys {:#x}\n", upte_val, + iova, level + 1, phys + ((upidx * PTE_SIZE) as PhysicalAddr)); + Ok(None) + } + })?; + if pt_addr[level].is_some() { + pt_parent[level] = Some((phys, upidx)); + } + mod_pr_debug!( + "UATPageTable::with_pages: level {} PT {:#x?}\n", + level, + pt_addr[level] + ); + } + // If we don't have a page table, skip this entire level + if pt_addr[level].is_none() { + let block = 1 << (UAT_PGBIT + UAT_LVBIT * (level + 1)); + let old = iova; + iova = align(iova + 1, block); + mod_pr_debug!( + "UATPageTable::with_pages: skip {:#x} {:#x} -> {:#x}\n", + block, + old, + iova + ); + continue 'outer; + } + } + + let idx = ((iova >> UAT_PGBIT as u64) & UAT_LVMSK) as usize; + let max_count = UAT_NPTE - idx; + let count = (((end - iova) >> UAT_PGBIT) as usize).min(max_count); + let phys = pt_addr[0].unwrap(); + mod_pr_debug!( + "UATPageTable::with_pages: leaf PT at {:#x} idx {:#x} count {:#x} iova {:#x}\n", + phys, + idx, + count, + iova + ); + // SAFETY: Page table addresses are either allocated by us, or + // firmware-managed and safe to borrow a struct page from. + let pt = unsafe { Page::borrow_phys_unchecked(&phys) }; + pt.with_pointer_into_page(idx * PTE_SIZE, count * PTE_SIZE, |p| { + let ptep = p as *const _ as *const Pte; + // SAFETY: We know this is a valid pointer to PTEs and the range is valid and + // checked by with_pointer_into_page(). + let ptes = unsafe { core::slice::from_raw_parts(ptep, count) }; + cb(iova, ptes)?; + Ok(()) + })?; + + let block = 1 << (UAT_PGBIT + UAT_LVBIT); + iova = align(iova + 1, block); + } + + if free { + // The final path is detached bottom-up for the same reason as the + // boundary transitions above. + for level in 0..UAT_LEVELS - 1 { + if let Some(phys) = pt_addr[level] { + mod_pr_debug!( + "UATPageTable::with_pages: free level {} {:#x?}\n", + level, + phys + ); + self.free_detached_table(phys, pt_parent[level].take())?; + } + } + } + + Ok(()) + } + + /// Free a page table, unless it was not allocated by the driver. + fn free_detached_table( + &mut self, + phys: PhysicalAddr, + parent: Option<(PhysicalAddr, usize)>, + ) -> Result { + let owned = if let Some(tables) = self.driver_tables.as_ref() { + match tables.iter().position(|&table| table == phys) { + Some(_) => true, + None => false, + } + } else { + true + }; + if !owned { + return Ok(()); + } + if let Some((parent_phys, parent_index)) = parent { + // SAFETY: The bottom-up traversal keeps the parent allocated while + // its child descriptor is detached. A changed descriptor indicates + // an ownership violation; fail closed instead of freeing a table + // that may still be reachable. + let page = unsafe { Page::borrow_phys(&parent_phys) }.ok_or(EIO)?; + page.with_pointer_into_page(parent_index * PTE_SIZE, PTE_SIZE, |p| { + let pte = unsafe { &*(p as *const AtomicU64) }; + let current = pte.load(Ordering::Acquire); + let expected = phys & self.oas_mask & (!UAT_PGMSK as u64); + if current & self.oas_mask & (!UAT_PGMSK as u64) != expected { + return Err(EIO); + } + pte.store(0, Ordering::Release); + Ok(()) + })?; + } + if let Some(tables) = self.driver_tables.as_mut() { + let index = tables + .iter() + .position(|&table| table == phys) + .ok_or(EIO)?; + tables.swap_remove(index); + } + // SAFETY: Page tables allocated by the driver always come from Page::into_phys(). Without + // `driver_tables`, every child table belongs to the driver. + unsafe { Page::from_phys(phys) }; + Ok(()) + } + fn with_pages_legacy( &mut self, iova_range: Range, @@ -639,6 +994,9 @@ impl UatPageTable { where F: FnMut(u64, &[Pte]) -> Result, { + if self.ordered_walk { + return self.with_pages_ordered(iova_range, alloc, free, cb); + } mod_pr_debug!( "UATPageTable::with_pages: {:#x?} alloc={} free={}\n", iova_range, @@ -803,7 +1161,7 @@ impl UatPageTable { where F: FnMut(u64, &[Pte]) -> Result, { - if self.geometry.root_size() == 1u64 << 39 { + if !self.agx3_walk() { return self.with_pages_legacy(iova_range, alloc, free, cb); } @@ -972,6 +1330,20 @@ impl UatPageTable { /// can detect ownership held by the other one. The page table is the /// authoritative final arbiter. pub(crate) fn prepare_map(&mut self, iova_range: Range) -> Result { + if self.ordered_walk { + // G17: GPUVM and the fixed-address allocator may share a page table, so check + // that neither owns a destination leaf before allowing failure rollback. + if iova_range.is_empty() || (iova_range.start | iova_range.end) & UAT_PGMSK as u64 != 0 { + return Err(EINVAL); + } + self.with_pages(iova_range.clone(), false, false, false, |_, ptes| { + if ptes.iter().any(|pte| pte.load(Ordering::Acquire) != 0) { + return Err(EBUSY); + } + Ok(()) + })?; + return self.alloc_pages(iova_range); + } if self.geometry.root_size() == 1u64 << 39 { return self.alloc_pages(iova_range); } if iova_range.is_empty() @@ -1104,7 +1476,7 @@ impl UatPageTable { prot: Prot, one_page: bool, ) -> Result { - if self.geometry.root_size() == 1u64 << 39 { + if !self.agx3_walk() { return self.map_pages_legacy(iova_range, phys, prot, one_page); } @@ -1190,7 +1562,7 @@ impl UatPageTable { } pub(crate) fn reprot_pages(&mut self, iova_range: Range, prot: Prot) -> Result { - if self.geometry.root_size() == 1u64 << 39 { + if !self.agx3_walk() { return self.reprot_pages_legacy(iova_range, prot); } @@ -1246,7 +1618,7 @@ impl UatPageTable { } pub(crate) fn unmap_pages(&mut self, iova_range: Range) -> Result { - if self.geometry.root_size() == 1u64 << 39 { + if !self.agx3_walk() { return self.unmap_pages_legacy(iova_range); } @@ -1276,6 +1648,36 @@ impl UatPageTable { }) } + /// Reads one mapped leaf entry without allocating missing tables. + pub(crate) fn leaf_entry(&mut self, iova: u64) -> Result { + if iova & UAT_PGMSK as u64 != 0 { + return Err(EINVAL); + } + let end = iova.checked_add(UAT_PGSZ as u64).ok_or(EINVAL)?; + let mut value = 0; + self.with_pages(iova..end, false, false, false, |_, ptes| { + if let Some(pte) = ptes.first() { + value = pte.load(Ordering::Relaxed); + } + Ok(()) + })?; + if value & PTE_TYPE_BITS != PTE_TYPE_LEAF_TABLE { + return Err(EFAULT); + } + Ok(value) + } + + /// Discards an aligned map operation that failed before acquiring a GPUVA owner. + /// Missing tables and leaves are expected; existing table storage stays allocated. + pub(crate) fn discard_partial_map(&mut self, iova_range: Range) -> Result { + self.with_pages(iova_range, false, false, true, |_, ptes| { + for pte in ptes { + pte.store(0, Ordering::Relaxed); + } + Ok(()) + }) + } + /// Read one PTE from a page table page, without ever allocating or modifying anything. /// /// Fails with an error (instead of oopsing) if the table page is not mapped. diff --git a/drivers/gpu/drm/asahi/queue/mod.rs b/drivers/gpu/drm/asahi/queue/mod.rs index a1153334665f4e..862e6ad92e752f 100644 --- a/drivers/gpu/drm/asahi/queue/mod.rs +++ b/drivers/gpu/drm/asahi/queue/mod.rs @@ -61,6 +61,13 @@ mod render; /// Trait implemented by all versioned queues. pub(crate) trait Queue: Send + Sync { + /// Reserve render ray query scratch for the given vertex and fragment layouts. Only the + /// G17P runtime supports render ray queries; every other queue accepts only zero. + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { + if layouts != [0; 2] { return Err(EINVAL); } + Ok(()) + } + fn submit( &mut self, id: u64, @@ -193,7 +200,12 @@ pub(crate) struct QueueJob { fence: UserFence, notifier: Arc>, notification_count: u32, - did_run: bool, + id: u64, +} + +/// Timeout state of a [`QueueJob`], owned by the scheduler's timeout worker. +pub(crate) struct QueueTimeout { + dev: AsahiDevRef, id: u64, } @@ -241,7 +253,13 @@ impl QueueJob::ver { #[versions(AGX)] impl sched::JobImpl for QueueJob::ver { - fn prepare(job: &mut sched::Job) -> Option { + type TimeoutData = QueueTimeout; + + fn timeout_data(&self) -> QueueTimeout { + QueueTimeout { dev: self.dev.clone(), id: self.id } + } + + fn prepare(job: &mut Self) -> Option { mod_dev_dbg!(job.dev, "QueueJob {}: Checking runnability\n", job.id); if let Some(sj) = job.sj_vtx.as_ref() { @@ -278,7 +296,7 @@ impl sched::JobImpl for QueueJob::ver { } #[allow(unused_assignments)] - fn run(job: &mut sched::Job) -> Result> { + fn run(job: &mut Self) -> Result> { mod_dev_dbg!(job.dev, "QueueJob {}: Running Job\n", job.id); // We can only increase the notifier threshold here, now that we are @@ -397,22 +415,22 @@ impl sched::JobImpl for QueueJob::ver { mod_dev_dbg!(job.dev, "QueueJob {}: Drop fragment job\n", job.id); core::mem::drop(frag_job); - job.did_run = true; - Ok(Some(Fence::from_fence(&job.fence))) } - fn timed_out(job: &mut sched::Job) -> sched::Status { + fn timed_out(job: &mut QueueTimeout, _finished: bool) -> sched::Status { // FIXME: Handle timeouts properly + // The timeout worker no longer sees the job itself, so it cannot tell whether run() + // completed. dev_err!( job.dev.as_ref(), - "QueueJob {}: Job timed out on the DRM scheduler, things will probably break (ran: {})\n", - job.id, job.did_run + "QueueJob {}: Job timed out on the DRM scheduler, things will probably break\n", + job.id ); sched::Status::NoDevice } - fn cancel(job: &mut sched::Job) { + fn cancel(job: &mut Self) { dev_info!( job.dev.as_ref(), "QueueJob {}: Job canceled on DRM scheduler teardown\n", @@ -748,7 +766,6 @@ impl Queue for Queue::ver { // notification count for the job. notification_count: (2 * nr_render) + nr_compute, - did_run: false, id, }, )?; diff --git a/drivers/gpu/drm/asahi/regs.rs b/drivers/gpu/drm/asahi/regs.rs index 0219d8f803b73f..7b1bf698c054a4 100644 --- a/drivers/gpu/drm/asahi/regs.rs +++ b/drivers/gpu/drm/asahi/regs.rs @@ -6,12 +6,17 @@ //! there is very little need to interact directly with GPU MMIO register. This module abstracts //! the few operations that require that, mainly reading the MMU fault status, reading GPU ID //! information, and starting the GPU firmware coprocessor. +//! +//! On G17, the driver also enables the bridge between the GPU and the fabric before the +//! coprocessors start, and enables dynamic power gating before the firmware is initialized. use crate::{hw, identity}; +use core::mem::MaybeUninit; use kernel::{ - c_str, + bindings, c_str, device::Core, devres::Devres, + error::{from_err_ptr, to_result}, io::{ mem::IoMem, // poll::read_poll_timeout, @@ -19,7 +24,7 @@ use kernel::{ }, platform, prelude::*, - sync::aref::ARef, // + sync::{aref::ARef, Arc}, // time::Delta, }; @@ -34,6 +39,10 @@ const CPU_RUN: u32 = 0x1 << 4; // BIT(4) const FAULT_INFO: usize = 0x17030; +/// A nonzero low nibble permits writes through the G17 queue configuration port. +const G17_HOST_IRQ_SUMMARY: usize = 0xe01000; +const G17_HOST_POWERED: u32 = 0xf; + const ID_VERSION: usize = 0xd04000; const ID_UNK08: usize = 0xd04008; const ID_COUNTS_1: usize = 0xd04010; @@ -48,6 +57,19 @@ const CORE_MASKS_G14X: usize = 0xe01500; const FAULT_INFO_G14X: usize = 0xd8c0; const FAULT_ADDR_G14X: usize = 0xd8c8; +/// G17 GPU-to-fabric bridge control words. Bit 0 enables the bridge; the other bits are status. +const G17_BRIDGE_CONTROL: [usize; 2] = [0x1000104, 0x1000108]; +const G17_BRIDGE_ENABLE: u32 = 0x1; // BIT(0) + +/// G17 dynamic power gating control. Clearing these bits enables gating of idle GPU cores. +const G17_GATING_CONTROL: usize = 0xd06030; +const G17_GATING_DISABLE: u32 = 0x6; // BIT(1) | BIT(2) + +/// These performance selectors must be sampled before firmware can gate the core registers. +const G17_PERF_CONTROL: usize = 0xe0141c; +const G17_PERF_MAP: usize = 0xe01480; +const G17_CORE_POWER: usize = 0xe01000; + // G15 (t6030) Fender registers. Fender = SGX + 0xd00000 on every AGX checked, so the ID block // above is Fender+0x4000 and the G14X core masks are Fender+0x101500. Everything // here is inside the 16 MiB `sgx` mapping. @@ -204,10 +226,93 @@ pub(crate) struct FaultInfo { pub(crate) reason: FaultReason, } +/// Identification of a G17 GPU. +pub(crate) struct G17Id { + /// GPU family field of the ID register. + pub(crate) family: u8, + /// GPU variant field of the ID register. + pub(crate) variant: u8, + /// Silicon revision. + pub(crate) gpu_rev: hw::GpuRevision, + /// Number of dies. + pub(crate) num_dies: u32, + /// Number of clusters, over all dies. + pub(crate) num_clusters: u32, + /// Number of cores per cluster. + pub(crate) num_cores: u32, + /// Enabled cores of each cluster. + pub(crate) core_masks: KVec, + /// Performance selector control, sampled before core power gating starts. + pub(crate) perf_control: u32, + /// Performance selector table indices, low and high nibbles. + pub(crate) perf_map: [u32; 2], +} + +/// Provider-owned lifecycle of one G17 firmware coprocessor. +/// +/// The mailbox provider owns the wrapper mappings and runtime-PM references. +/// The consumer device link keeps it alive until this GPU has stopped using it. +pub(crate) struct CpuControl { + dev: ARef, + mbox: *mut bindings::apple_mbox, +} + +// SAFETY: The live provider serializes CPU transitions with its lifecycle mutex; +// the device link orders provider removal after this consumer's teardown. +unsafe impl Send for CpuControl {} +// SAFETY: All accesses use the same provider-serialized lifecycle API. +unsafe impl Sync for CpuControl {} + +impl CpuControl { + /// Resolve the mailbox phandle and validate its firmware role before use. + pub(crate) fn new(pdev: &platform::Device, index: usize) -> Result { + let expected = match index { + 0 => bindings::apple_mbox_ascwrap_v6_role_APPLE_MBOX_ASCWRAP_V6_ROLE_GFX, + 1 => bindings::apple_mbox_ascwrap_v6_role_APPLE_MBOX_ASCWRAP_V6_ROLE_GFX1, + _ => return Err(EINVAL), + }; + // SAFETY: pdev is live; the provider establishes the consumer device link. + let mbox = unsafe { + from_err_ptr(bindings::apple_mbox_get(pdev.as_ref().as_raw(), index.try_into()?))? + }; + let mut lifecycle = MaybeUninit::uninit(); + // SAFETY: mbox is live, and lifecycle is writable storage for the C result. + to_result(unsafe { + bindings::apple_mbox_ascwrap_v6_get_lifecycle(mbox, lifecycle.as_mut_ptr()) + })?; + // SAFETY: A successful provider query initialized every field. + if unsafe { lifecycle.assume_init() }.role != expected { + return Err(EINVAL); + } + // SAFETY: This only checks the provider's matching start/stop capability. + to_result(unsafe { bindings::apple_mbox_ascwrap_v6_require_safe_cpu_lifecycle(mbox) })?; + Ok(Self { dev: pdev.into(), mbox }) + } + + /// Start through the owner of the CPU registers and runtime-PM reference. + pub(crate) fn start(&self) -> Result { + // SAFETY: The constructor established the live, role-checked provider. + to_result(unsafe { bindings::apple_mbox_ascwrap_v6_start_cpu(self.mbox) }) + } + + /// Stop before releasing the transport or any firmware-visible memory. + pub(crate) fn stop(&self) { + // SAFETY: The provider remains live during consumer teardown. Removal + // stops the CPU before setting the fence that can return ENODEV here. + let result = to_result(unsafe { bindings::apple_mbox_ascwrap_v6_stop_cpu(self.mbox) }); + if let Err(error) = result { + if error != ENODEV { + dev_err!(self.dev.as_ref(), "Could not stop GPU coprocessor: {:?}\n", error); + } + } + } +} + /// Device resources for this GPU instance. +#[derive(Clone)] pub(crate) struct Resources { dev: ARef, - sgx: Pin>>>, + sgx: Arc>>, } impl Resources { @@ -400,7 +505,7 @@ impl Resources { /// Map the required resources given our platform device. pub(crate) fn new(pdev: &platform::Device) -> Result { let sgx_req = pdev.io_request_by_name(c_str!("sgx")).ok_or(EINVAL)?; - let sgx_iomem = KBox::pin_init(sgx_req.iomap_sized::(), GFP_KERNEL)?; + let sgx_iomem = Arc::pin_init(sgx_req.iomap_sized::(), GFP_KERNEL)?; Ok(Resources { // SAFETY: This device does DMA via the UAT IOMMU. @@ -409,6 +514,23 @@ impl Resources { }) } + /// Installs both render queues while the caller holds the device mutex. + /// No other port publisher may interleave between this power witness and + /// the final validity strobe. False means no write was attempted. + pub(crate) fn install_render_pair( + &self, + registration: &crate::g17::fw::kick::RenderRegistration, + ) -> Result { + let sgx = self.sgx.try_access().ok_or(ENODEV)?; + if sgx.relaxed().read32(G17_HOST_IRQ_SUMMARY) & G17_HOST_POWERED == 0 { + return Ok(false); + } + for &(offset, value) in registration.writes() { + sgx.try_write64(value, offset)?; + } + Ok(true) + } + fn sgx_read32(&self) -> u32 { if let Some(sgx) = self.sgx.try_access() { sgx.relaxed().read32(OFF) @@ -623,6 +745,166 @@ impl Resources { }) } + /// Get the identification of a G17 GPU from its registers. + pub(crate) fn get_g17_id(&self) -> Result { + let id_version = self.sgx_read32::(); + let id_unk08 = self.sgx_read32::(); + let id_counts_1 = self.sgx_read32::(); + let id_counts_2 = self.sgx_read32::(); + let id_unk18 = self.sgx_read32::(); + let id_clusters = self.sgx_read32::(); + + dev_info!( + self.dev.as_ref(), + "GPU ID registers: {:#x} {:#x} {:#x} {:#x} {:#x} {:#x}\n", + id_version, + id_unk08, + id_counts_1, + id_counts_2, + id_unk18, + id_clusters + ); + + // Later reads can fault while the firmware has gated the cores. Submission uses only + // these saved values and never re-reads the registers. + let perf_control = self.sgx_read32::() & 1; + let perf_map = self.sgx_read32::(); + let mut core_mask_regs = [ + self.sgx_read32::(), + self.sgx_read32::<{ CORE_MASKS_G14X + 4 }>(), + self.sgx_read32::<{ CORE_MASKS_G14X + 8 }>(), + ]; + dev_info!(self.dev.as_ref(), "Core masks: {:#x?}\n", core_mask_regs); + + let num_dies = (id_counts_1 >> 16) & 0xf; + let num_clusters = ((id_counts_1 >> 8) & 0xff) * num_dies; + let num_cores = id_counts_1 & 0xff; + if num_cores == 0 || num_cores > 32 || num_cores * num_clusters > 3 * 32 { + dev_err!( + self.dev.as_ref(), + "Invalid core counts ({} x {})\n", + num_clusters, + num_cores + ); + return Err(ENODEV); + } + let (core_masks, _) = + self.split_core_masks(&mut core_mask_regs, num_clusters, num_cores)?; + let (gpu_rev, _) = self.gpu_revision(id_version)?; + + Ok(G17Id { + family: ((id_version >> 24) & 0xff) as u8, + variant: ((id_version >> 16) & 0xff) as u8, + gpu_rev, + num_dies, + num_clusters, + num_cores, + core_masks, + perf_control, + perf_map: [perf_map & 0xf, (perf_map >> 16) & 0xf], + }) + } + + /// Enable the G17 GPU-to-fabric bridge and check that it took effect. + /// + /// The GPU power domain must be on. The coprocessors cannot reach memory until this is done. + pub(crate) fn enable_g17_bridge(&self) -> Result { + let sgx = self.sgx.try_access().ok_or(ENODEV)?; + for offset in G17_BRIDGE_CONTROL { + let val = sgx.try_read32(offset)?; + sgx.try_write32(val | G17_BRIDGE_ENABLE, offset)?; + if sgx.try_read32(offset)? & G17_BRIDGE_ENABLE == 0 { + dev_err!( + self.dev.as_ref(), + "GPU bridge {:#x} did not enable\n", + offset + ); + return Err(EIO); + } + } + Ok(()) + } + + /// Disable the G17 GPU-to-fabric bridge. Both coprocessors must be stopped. + pub(crate) fn disable_g17_bridge(&self) { + if let Some(sgx) = self.sgx.try_access() { + for offset in G17_BRIDGE_CONTROL { + if let Ok(val) = sgx.try_read32(offset) { + // Cannot fail: the offset was read just before. + let _ = sgx.try_write32(val & !G17_BRIDGE_ENABLE, offset); + } + } + } + } + + /// Power witness for the queue-configuration port, accessible while the cores are gated. + pub(crate) fn g17_configuration_powered(&self) -> bool { + self.sgx_read32::() & 0xf != 0 + } + + /// Enable dynamic power gating of idle G17 GPU cores. The firmware expects this before it is + /// initialized. + pub(crate) fn enable_g17_gating(&self) -> Result { + let sgx = self.sgx.try_access().ok_or(ENODEV)?; + let val = sgx.try_read32(G17_GATING_CONTROL)?; + sgx.try_write32(val & !G17_GATING_DISABLE, G17_GATING_CONTROL) + } + + /// Split the packed per-cluster core masks into one mask per cluster. + /// + /// Returns the masks and the total number of enabled cores. + fn split_core_masks( + &self, + core_mask_regs: &mut [u32], + num_clusters: u32, + num_cores: u32, + ) -> Result<(KVec, u32)> { + let mut core_masks = KVec::new(); + let mut total_active_cores: u32 = 0; + + let max_core_mask = ((1u64 << num_cores) - 1) as u32; + for _ in 0..num_clusters { + let mask = core_mask_regs[0] & max_core_mask; + core_masks.push(mask, GFP_KERNEL)?; + for i in 0..core_mask_regs.len() { + core_mask_regs[i] = core_mask_regs[i].checked_shr(num_cores).unwrap_or(0); + if i < (core_mask_regs.len() - 1) { + core_mask_regs[i] |= core_mask_regs[i + 1] + .checked_shl(32 - num_cores) + .unwrap_or(0); + } + } + total_active_cores += mask.count_ones(); + } + + if core_mask_regs.iter().any(|a| *a != 0) { + dev_err!( + self.dev.as_ref(), + "Leftover core mask: {:#x?}\n", + core_mask_regs + ); + return Err(EIO); + } + + Ok((core_masks, total_active_cores)) + } + + /// Decode the silicon revision from the ID version register. + fn gpu_revision(&self, id_version: u32) -> Result<(hw::GpuRevision, hw::GpuRevisionID)> { + Ok(match (id_version >> 8) & 0xff { + 0x00 => (hw::GpuRevision::A0, hw::GpuRevisionID::A0), + 0x01 => (hw::GpuRevision::A1, hw::GpuRevisionID::A1), + 0x10 => (hw::GpuRevision::B0, hw::GpuRevisionID::B0), + 0x11 => (hw::GpuRevision::B1, hw::GpuRevisionID::B1), + 0x20 => (hw::GpuRevision::C0, hw::GpuRevisionID::C0), + 0x21 => (hw::GpuRevision::C1, hw::GpuRevisionID::C1), + a => { + dev_err!(self.dev.as_ref(), "Unknown GPU revision {}\n", a); + return Err(ENODEV); + } + }) + } + /// Get the fault information from the MMU status register, if one occurred. pub(crate) fn get_fault_info(&self, cfg: &'static hw::HwConfig) -> Option { let g14x = cfg diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index 3b2448a3a9de85..2176ab547bfb6a 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -1343,7 +1343,7 @@ unsigned long drm_gem_get_unmapped_area(struct file *filp, unsigned long uaddr, struct drm_gem_object *obj; unsigned long ret; - obj = drm_gem_object_lookup_at_offset(filp, pgoff, len >> PAGE_SHIFT); + obj = drm_gem_object_lookup_at_offset(filp, pgoff, DIV_ROUND_UP(len, PAGE_SIZE)); if (IS_ERR(obj)) obj = NULL; @@ -1379,8 +1379,11 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma) struct drm_gem_object *obj; int ret; + /* GEM fake offsets identify an exact object start, not an interior byte. */ + if (vma_subpage_offset(vma)) + return -EINVAL; obj = drm_gem_object_lookup_at_offset(filp, vma->vm_pgoff, - vma_pages(vma)); + DIV_ROUND_UP(vma->vm_end - vma->vm_start, PAGE_SIZE)); if (IS_ERR(obj)) return PTR_ERR(obj); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index fc05581d47772b..eab3abedcc8741 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -763,7 +763,10 @@ static vm_fault_t try_insert_pfn(struct vm_fault *vmf, unsigned int order, vm_fault_t ret = VM_FAULT_FALLBACK; if (!order) { - ret = vmf_insert_pfn_mkwrite(vmf->vma, vmf->address, pfn, write); + struct vm_page_offset pos = vma_page_offset_at(vmf->vma, vmf->address); + + ret = vmf_insert_pfn_prot_mkwrite_offset(vmf->vma, vmf->address, + pfn, pos.offset, vmf->vma->vm_page_prot, write); #ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP } else if (order == PMD_ORDER) { unsigned long paddr = pfn << PAGE_SHIFT; @@ -800,7 +803,7 @@ static vm_fault_t drm_gem_shmem_any_fault(struct vm_fault *vmf, unsigned int ord struct folio *folio; unsigned long pfn; - if (order && order != PMD_ORDER) + if (order && (order != PMD_ORDER || mm_page_size(vma->vm_mm) != PAGE_SIZE)) return VM_FAULT_FALLBACK; dma_resv_lock(obj->resv, NULL); @@ -956,6 +959,17 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct if (is_cow_mapping(vma->vm_flags)) return -EINVAL; + /* Validate bytes, including a partial native-page offset from PRIME. */ + { + pgoff_t index = drm_gem_shmem_object_offset(obj, vma->vm_pgoff); + unsigned long offset, length = vma->vm_end - vma->vm_start; + + if (check_mul_overflow(index, PAGE_SIZE, &offset) || + check_add_overflow(offset, (unsigned long)vma_subpage_offset(vma), &offset) || + offset > obj->size || length > obj->size - offset) + return -EINVAL; + } + /* Revocation zaps the device anon mapping only. */ if (shmem->reclaimable_cpu_mappings && (!vma->vm_file || vma->vm_file->f_mapping != obj->dev->anon_inode->i_mapping)) diff --git a/drivers/gpu/drm/drm_gpuvm.c b/drivers/gpu/drm/drm_gpuvm.c index 91f7861e36b84d..92ab5271e523e9 100644 --- a/drivers/gpu/drm/drm_gpuvm.c +++ b/drivers/gpu/drm/drm_gpuvm.c @@ -1396,9 +1396,23 @@ fn_lock_array(struct drm_gpuvm_exec *vm_exec) struct drm_gem_object **objs; unsigned int num_objs; } *args = vm_exec->extra.priv; + unsigned int i; + int ret; - return drm_exec_prepare_array(&vm_exec->exec, args->objs, - args->num_objs, vm_exec->num_fences); + /* + * Keep the zero-fence contract of every other drm_gpuvm_exec helper. + * drm_exec_prepare_array() unconditionally calls + * dma_resv_reserve_fences(), where zero is invalid; exec_prepare_obj() + * deliberately selects drm_exec_lock_obj() for that case instead. + */ + for (i = 0; i < args->num_objs; i++) { + ret = exec_prepare_obj(&vm_exec->exec, args->objs[i], + vm_exec->num_fences); + if (ret) + return ret; + } + + return 0; } /** @@ -2368,6 +2382,7 @@ op_map_cb(const struct drm_gpuvm_ops *fn, void *priv, op.map.va.range = req->map.va.range; op.map.gem.obj = req->map.gem.obj; op.map.gem.offset = req->map.gem.offset; + op.map.gem.range = req->map.gem.range; op.map.flags = req->map.flags; return fn->sm_step_map(&op, priv); @@ -2424,7 +2439,7 @@ static bool __can_merge(struct drm_gpuvm *gpuvm, const struct drm_gpuva_op_map * if (a->gem.obj != b->gem.obj || !a->gem.obj) return false; - if (can_merge_flags(gpuvm, a->flags, b->flags)) + if (!can_merge_flags(gpuvm, a->flags, b->flags)) return false; /* Order VAs for the rest of the checks. */ @@ -2443,7 +2458,7 @@ static bool __can_merge(struct drm_gpuvm *gpuvm, const struct drm_gpuva_op_map * /* If this is a repeated mapping, both the GEM range * and offset must match. */ - if (a->gem.range != b->gem.range || + if (!a->gem.range || a->gem.range != b->gem.range || a->gem.offset != b->gem.offset) return false; @@ -2471,6 +2486,7 @@ static bool can_merge(struct drm_gpuvm *gpuvm, const struct drm_gpuva *a, .va.range = a->va.range, .gem.offset = a->gem.offset, .gem.obj = a->gem.obj, + .gem.range = a->gem.range, .flags = a->flags, }; @@ -2724,7 +2740,7 @@ __drm_gpuvm_sm_unmap(struct drm_gpuvm *gpuvm, next.va.addr = req_end; next.va.range = end - req_end; next.gem.obj = obj; - prev.gem.range = va->gem.range; + next.gem.range = va->gem.range; next.gem.offset = offset + (va->flags & DRM_GPUVA_REPEAT ? 0 : req_end - addr); next.flags = va->flags; diff --git a/drivers/gpu/drm/pl111/pl111_versatile.c b/drivers/gpu/drm/pl111/pl111_versatile.c index 0d8331a3909f4d..93ef58ffc91d38 100644 --- a/drivers/gpu/drm/pl111/pl111_versatile.c +++ b/drivers/gpu/drm/pl111/pl111_versatile.c @@ -486,7 +486,7 @@ int pl111_versatile_init(struct drm_device *dev, struct pl111_drm_dev_private *p return 0; } - versatile_clcd_type = (enum versatile_clcd)clcd_id->data; + versatile_clcd_type = (enum versatile_clcd)(uintptr_t)clcd_id->data; /* Versatile Express special handling */ if (versatile_clcd_type == VEXPRESS_CLCD_V2M) { @@ -506,7 +506,7 @@ int pl111_versatile_init(struct drm_device *dev, struct pl111_drm_dev_private *p np = of_find_matching_node_and_match(NULL, impd1_clcd_of_match, &clcd_id); if (np) - versatile_clcd_type = (enum versatile_clcd)clcd_id->data; + versatile_clcd_type = (enum versatile_clcd)(uintptr_t)clcd_id->data; } map = syscon_node_to_regmap(np); diff --git a/drivers/gpu/drm/vc4/vc4_drv.c b/drivers/gpu/drm/vc4/vc4_drv.c index a14ecb7694613f..2454ce0b0bc479 100644 --- a/drivers/gpu/drm/vc4/vc4_drv.c +++ b/drivers/gpu/drm/vc4/vc4_drv.c @@ -299,7 +299,7 @@ static int vc4_drm_bind(struct device *dev) dev->coherent_dma_mask = DMA_BIT_MASK(32); - gen = (enum vc4_gen)of_device_get_match_data(dev); + gen = (enum vc4_gen)(uintptr_t)of_device_get_match_data(dev); if (gen > VC4_GEN_4) driver = &vc5_drm_driver; diff --git a/drivers/hid/dockchannel-hid/dockchannel-hid.c b/drivers/hid/dockchannel-hid/dockchannel-hid.c index 527c00d26c46ca..6a5e412887082a 100644 --- a/drivers/hid/dockchannel-hid/dockchannel-hid.c +++ b/drivers/hid/dockchannel-hid/dockchannel-hid.c @@ -177,6 +177,7 @@ struct dchid_iface { u32 keyboard_layout_id; u32 power_method; + bool firmware_managed_reset; }; struct dockchannel_hid { @@ -217,6 +218,7 @@ static struct dchid_iface * dchid_get_interface(struct dockchannel_hid *dchid, int index, const char *name) { u32 power_method = DCHID_POWER_METHOD_1; + bool firmware_managed_reset = false; struct device_node *of_node = NULL; struct dchid_iface *iface; @@ -240,13 +242,19 @@ dchid_get_interface(struct dockchannel_hid *dchid, int index, const char *name) } of_property_read_u32(of_node, "apple,power-method", &power_method); + firmware_managed_reset = + of_property_read_bool(of_node, "apple,firmware-managed-reset"); + if (firmware_managed_reset && + (strcmp(name, "multi-touch") || power_method != DCHID_POWER_METHOD_2)) + goto err_put_node; if (power_method != DCHID_POWER_METHOD_1 && power_method != DCHID_POWER_METHOD_2) { dev_err(dchid->dev, "Unsupported power method %u for %s\n", power_method, name); goto err_put_node; } - if (power_method == DCHID_POWER_METHOD_2 && !dchid->afe_reset) { + if (power_method == DCHID_POWER_METHOD_2 && + !firmware_managed_reset && !dchid->afe_reset) { dev_err(dchid->dev, "Power method 2 for %s needs the AFE reset line\n", name); goto err_put_node; @@ -264,6 +272,7 @@ dchid_get_interface(struct dockchannel_hid *dchid, int index, const char *name) iface->dchid = dchid; iface->of_node = of_node; iface->power_method = power_method; + iface->firmware_managed_reset = firmware_managed_reset; iface->out_report = -1; init_completion(&iface->out_complete); init_completion(&iface->ready); @@ -425,12 +434,18 @@ static int dchid_pm2_set_power(struct dchid_iface *iface, u8 state) if (ret < 0) return ret; - ret = gpiod_set_value_cansleep(dchid->afe_reset, - state == DCHID_POWER_STATE_OFF); - if (ret < 0) - return ret; - if (state == DCHID_POWER_STATE_OFF) - fsleep(50 * USEC_PER_MSEC); + if (iface->firmware_managed_reset) { + /* Firmware owns the reset; allow its power-on transition to settle. */ + if (state == DCHID_POWER_STATE_ON) + fsleep(50 * USEC_PER_MSEC); + } else { + ret = gpiod_set_value_cansleep(dchid->afe_reset, + state == DCHID_POWER_STATE_OFF); + if (ret < 0) + return ret; + if (state == DCHID_POWER_STATE_OFF) + fsleep(50 * USEC_PER_MSEC); + } dchid_pm2_command(iface->index, state, true, msg); return dchid_comm_cmd(dchid, msg, sizeof(msg)); @@ -527,6 +542,9 @@ static int dchid_request_gpio(struct dchid_iface *iface) char prop_name[MAX_GPIO_NAME + 16]; int ret; + if (iface->firmware_managed_reset) + return -EINVAL; + if (iface->gpio) return 0; @@ -562,7 +580,7 @@ static int dchid_start_interface(struct dchid_iface *iface) int ret; if (iface->starting) { - dev_warn(iface->dchid->dev, "Interface %s is already starting", iface->name); + dev_warn(iface->dchid->dev, "Interface %s is already starting\n", iface->name); return -EINPROGRESS; } @@ -589,7 +607,7 @@ static int dchid_start_interface(struct dchid_iface *iface) dev_info(iface->dchid->dev, "Sending firmware for %s\n", iface->name); ret = dchid_send_firmware(iface, fw, size); if (ret < 0) { - dev_err(iface->dchid->dev, "Failed to send %s firmwareS", iface->name); + dev_err(iface->dchid->dev, "Failed to send %s firmware\n", iface->name); goto err; } @@ -622,7 +640,7 @@ static int dchid_start(struct hid_device *hdev) if (iface->keyboard_layout_id) { int ret = device_create_file(&hdev->dev, &dev_attr_apple_layout_id); if (ret) { - dev_warn(iface->dchid->dev, "Failed to create apple_layout_id: %d", ret); + dev_warn(iface->dchid->dev, "Failed to create apple_layout_id: %d\n", ret); iface->keyboard_layout_id = 0; } } @@ -657,7 +675,8 @@ static int dchid_open(struct hid_device *hdev) } } - iface->open = true; + /* Pairs with the acquire in dchid_handle_report(); publishes iface->hid. */ + smp_store_release(&iface->open, true); return 0; } @@ -665,7 +684,7 @@ static void dchid_close(struct hid_device *hdev) { struct dchid_iface *iface = hdev->driver_data; - iface->open = false; + WRITE_ONCE(iface->open, false); } static int dchid_parse(struct hid_device *hdev) @@ -783,7 +802,7 @@ static void dchid_create_interface_work(struct work_struct *ws) if (ret < 0) { iface->hid = NULL; hid_destroy_device(hid); - dev_warn(iface->dchid->dev, "Failed to register hid device %s", iface->name); + dev_warn(iface->dchid->dev, "Failed to register hid device %s\n", iface->name); } } @@ -1045,14 +1064,12 @@ static void dchid_handle_event(struct dockchannel_hid *dchid, void *data, size_t static void dchid_handle_report(struct dchid_iface *iface, void *data, size_t length) { - struct dockchannel_hid *dchid = iface->dchid; - - if (!iface->hid) { - dev_warn(dchid->dev, "Report received but %s is not initialized!\n", iface->name); - return; - } - - if (!iface->open) + /* + * An interface starts reporting as soon as it is enabled, which is + * before its HID device has been registered and opened. Until then + * there is nobody to deliver the reports to. + */ + if (!smp_load_acquire(&iface->open)) return; hid_input_report(iface->hid, HID_INPUT_REPORT, data, length, 1); @@ -1116,7 +1133,7 @@ static void dchid_handle_ack(struct dchid_iface *iface, struct dchid_hdr *hdr, v } if (shdr->flags != iface->out_flags) { dev_err(iface->dchid->dev, - "Received unexpected flags 0x%x on ACK channel (expFected 0x%x)\n", + "Received unexpected flags 0x%x on ACK channel (expected 0x%x)\n", shdr->flags, iface->out_flags); return; } @@ -1302,10 +1319,17 @@ static int dockchannel_hid_probe(struct platform_device *pdev) child = of_get_child_by_name(dev->of_node, "multi-touch"); if (child) { u32 power_method = DCHID_POWER_METHOD_1; + bool firmware_managed_reset; of_property_read_u32(child, "apple,power-method", &power_method); + firmware_managed_reset = + of_property_read_bool(child, "apple,firmware-managed-reset"); of_node_put(child); - if (power_method == DCHID_POWER_METHOD_2) { + if (firmware_managed_reset && + (power_method != DCHID_POWER_METHOD_2 || + of_property_present(dev->of_node, "apple,afe-reset-gpios"))) + return dev_err_probe(dev, -EINVAL, "Conflicting touch reset ownership\n"); + if (power_method == DCHID_POWER_METHOD_2 && !firmware_managed_reset) { dchid->afe_reset = devm_gpiod_get(dev, "apple,afe-reset", GPIOD_ASIS); if (IS_ERR(dchid->afe_reset)) @@ -1320,14 +1344,14 @@ static int dockchannel_hid_probe(struct platform_device *pdev) */ helper = of_parse_phandle(dev->of_node, "apple,helper-cpu", 0); if (!helper) { - dev_err(dev, "Missing apple,helper-cpu property"); + dev_err(dev, "Missing apple,helper-cpu property\n"); return -EINVAL; } helper_pdev = of_find_device_by_node(helper); of_node_put(helper); if (!helper_pdev) { - dev_err(dev, "Failed to find helper device"); + dev_err(dev, "Failed to find helper device\n"); return -EINVAL; } @@ -1335,7 +1359,7 @@ static int dockchannel_hid_probe(struct platform_device *pdev) DL_FLAG_AUTOREMOVE_CONSUMER); put_device(&helper_pdev->dev); if (!dchid->helper_link) { - dev_err(dev, "Failed to link to helper device"); + dev_err(dev, "Failed to link to helper device\n"); return -EINVAL; } @@ -1364,7 +1388,7 @@ static int dockchannel_hid_probe(struct platform_device *pdev) dchid->comm = dchid_get_interface(dchid, IFACE_COMM, "comm"); if (!dchid->comm) { - dev_err(dchid->dev, "Failed to initialize comm interface"); + dev_err(dchid->dev, "Failed to initialize comm interface\n"); destroy_workqueue(dchid->new_iface_wq); return -EIO; } diff --git a/drivers/hwmon/Kconfig b/drivers/hwmon/Kconfig index a2bd15114d3f77..0e60c5e4c8de73 100644 --- a/drivers/hwmon/Kconfig +++ b/drivers/hwmon/Kconfig @@ -1189,6 +1189,16 @@ config SENSORS_MACSMC_HWMON on Apple Silicon devices. Say Y here if you have an Apple Silicon device. +config SENSORS_MACSMC_HWMON_KUNIT_TEST + bool "KUnit tests for the Apple SMC hwmon driver" if !KUNIT_ALL_TESTS + depends on SENSORS_MACSMC_HWMON && KUNIT=y + default KUNIT_ALL_TESTS + help + Unit tests for the channel numbering of the Apple SMC hwmon driver. + Thermal zones refer to temperature channels by index, so a sensor + that cannot be set up must keep its channel; the tests check that. + No SMC is needed. Only useful for kernel developers. + This driver can also be built as a module. If so, the module will be called macsmc-hwmon. diff --git a/drivers/hwmon/macsmc-hwmon-test.c b/drivers/hwmon/macsmc-hwmon-test.c new file mode 100644 index 00000000000000..e7239986ce3c9e --- /dev/null +++ b/drivers/hwmon/macsmc-hwmon-test.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * KUnit tests for the Apple SMC hwmon driver: channel numbering of the + * temperature sensors, which thermal zones depend on. + * + * Included by macsmc-hwmon.c. + */ + +#include + +/* What macsmc_hwmon_create_sensor() leaves behind for one node. */ +static void macsmc_hwmon_test_add(struct macsmc_hwmon_sensors *temp, + const char *key, int create_ret) +{ + struct macsmc_hwmon_sensor *sensor = &temp->sensors[temp->count]; + + sensor->macsmc_key = _SMC_KEY(key); + strscpy_pad(sensor->label, key, sizeof(sensor->label)); + macsmc_hwmon_add_temp_channel(temp, create_ret); +} + +/* A node that fails to set up keeps its channel; later channels do not move. */ +static void macsmc_hwmon_test_failed_temp_keeps_slot(struct kunit *test) +{ + struct macsmc_hwmon_sensor sensors[3] = {}; + struct macsmc_hwmon hwmon = { + .temp = { .sensors = sensors }, + }; + u32 configs[ARRAY_SIZE(sensors) + 1] = {}; + + macsmc_hwmon_test_add(&hwmon.temp, "Te05", 0); + macsmc_hwmon_test_add(&hwmon.temp, "Te0S", -EINVAL); + macsmc_hwmon_test_add(&hwmon.temp, "Tp01", 0); + + KUNIT_ASSERT_EQ(test, hwmon.temp.count, 3); + KUNIT_EXPECT_EQ(test, sensors[0].macsmc_key, _SMC_KEY("Te05")); + KUNIT_EXPECT_EQ(test, sensors[2].macsmc_key, _SMC_KEY("Tp01")); + + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_input, 0), 0444); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_input, 1), 0); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_label, 1), 0); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_input, 2), 0444); + + /* A zero entry would end the channel list at the failed sensor. */ + macsmc_hwmon_populate_configs(configs, &hwmon.temp); + KUNIT_EXPECT_NE(test, configs[0], 0); + KUNIT_EXPECT_NE(test, configs[1], 0); + KUNIT_EXPECT_NE(test, configs[2], 0); + KUNIT_EXPECT_EQ(test, configs[3], 0); +} + +static void macsmc_hwmon_test_temp_label(struct kunit *test) +{ + struct macsmc_hwmon_sensor sensors[2] = {}; + struct macsmc_hwmon hwmon = { + .temp = { .sensors = sensors }, + }; + + macsmc_hwmon_test_add(&hwmon.temp, "Tp05", 0); + sensors[1].macsmc_key = _SMC_KEY("Tp09"); + macsmc_hwmon_add_temp_channel(&hwmon.temp, 0); + + KUNIT_ASSERT_EQ(test, hwmon.temp.count, 2); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_label, 0), 0444); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_label, 1), 0); + KUNIT_EXPECT_EQ(test, macsmc_hwmon_is_visible(&hwmon, hwmon_temp, + hwmon_temp_input, 1), 0444); +} + +static struct kunit_case macsmc_hwmon_test_cases[] = { + KUNIT_CASE(macsmc_hwmon_test_failed_temp_keeps_slot), + KUNIT_CASE(macsmc_hwmon_test_temp_label), + {} +}; + +static struct kunit_suite macsmc_hwmon_test_suite = { + .name = "macsmc-hwmon", + .test_cases = macsmc_hwmon_test_cases, +}; + +kunit_test_suite(macsmc_hwmon_test_suite); diff --git a/drivers/hwmon/macsmc-hwmon.c b/drivers/hwmon/macsmc-hwmon.c index c85430ad14e66f..06d2c6e2d0ac38 100644 --- a/drivers/hwmon/macsmc-hwmon.c +++ b/drivers/hwmon/macsmc-hwmon.c @@ -602,6 +602,10 @@ static umode_t macsmc_hwmon_is_visible(const void *data, return 0; } + /* A sensor without a key could not be set up; see macsmc_hwmon_add_temp_channel(). */ + if (!sensor->macsmc_key) + return 0; + /* Sensors only register ro attributes */ if (sensor->attrs & BIT(attr)) return 0444; @@ -753,6 +757,89 @@ static int macsmc_hwmon_create_pressure(struct macsmc_hwmon *hwmon, return 0; } +/* + * Account for the temperature sensor just set up at the end of @temp. + * + * Thermal zones refer to a temperature channel by the index of its node + * among the temperature-* nodes, so every node takes a channel. A sensor + * that could not be set up keeps its channel with no key, which hides it: + * hwmon creates no attributes and no thermal sensor for it, and only the + * zones that use it fail to register, instead of every later channel + * moving onto the wrong sensor. + */ +static void macsmc_hwmon_add_temp_channel(struct macsmc_hwmon_sensors *temp, + int create_ret) +{ + struct macsmc_hwmon_sensor *sensor = &temp->sensors[temp->count++]; + + if (create_ret) + memset(sensor, 0, sizeof(*sensor)); + + /* A zero entry would end the list of channels here. */ + sensor->attrs = HWMON_T_INPUT; + if (*sensor->label) + sensor->attrs |= HWMON_T_LABEL; +} + +/* Discover reporting-only temperatures without a board-specific key table. */ +static int macsmc_hwmon_discover_temperatures(struct macsmc_hwmon *hwmon) +{ + struct apple_smc *smc = hwmon->smc; + struct macsmc_hwmon_sensor *sensor; + struct apple_smc_key_info info; + smc_key key, previous = 0; + u32 i; + int ret; + + if (!of_property_read_bool(hwmon->dev->of_node, "apple,discover-temperatures")) + return 0; + + /* Dynamic channel numbers must not replace DT-described channels. */ + if (hwmon->temp.count) + return -EINVAL; + + /* Bound both allocation and mailbox transactions for invalid firmware data. */ + if (!smc->key_count || smc->key_count > U16_MAX) + return -EINVAL; + + hwmon->temp.sensors = devm_kcalloc(hwmon->dev, smc->key_count, + sizeof(*sensor), GFP_KERNEL); + if (!hwmon->temp.sensors) + return -ENOMEM; + + for (i = 0; i < smc->key_count; i++) { + ret = apple_smc_get_key_by_index(smc, i, &key); + if (ret < 0) + return ret; + /* SMC key tables are strictly ordered; reject duplicates as well. */ + if (i && key <= previous) + return -EINVAL; + previous = key; + if ((key >> 24) < 'T') + continue; + if ((key >> 24) > 'T') + break; + + ret = apple_smc_get_key_info(smc, key, &info); + if (ret < 0) + return ret; + if (!(info.flags & APPLE_SMC_READABLE) || + (info.flags & APPLE_SMC_FUNCTION) || + info.type_code != __SMC_KEY('f', 'l', 't', ' ') || info.size != 4) + continue; + + sensor = &hwmon->temp.sensors[hwmon->temp.count++]; + sensor->macsmc_key = key; + sensor->info = info; + sensor->attrs = HWMON_T_INPUT | HWMON_T_LABEL; + /* Key identity only: do not invent locations or calibration. */ + snprintf(sensor->label, sizeof(sensor->label), "%c%c%c%c", + key >> 24, (key >> 16) & 0xff, (key >> 8) & 0xff, key & 0xff); + } + + return 0; +} + /* * Fan data is exposed by the SMC as multiple sensors. * @@ -832,6 +919,7 @@ static int macsmc_hwmon_populate_sensors(struct macsmc_hwmon *hwmon, u32 n_current = 0, n_fan = 0, n_power = 0, n_temperature = 0, n_voltage = 0; const char *die_keys = macsmc_hwmon_die_key_list(), *list; char key[5]; + bool indexed; int ret; for_each_child_of_node_with_prefix(hwmon_node, key_node, "current-") { @@ -909,16 +997,22 @@ static int macsmc_hwmon_populate_sensors(struct macsmc_hwmon *hwmon, if (!hwmon->temp.sensors) return -ENOMEM; + /* + * Only a node that is a thermal sensor provider (T8140) has zones + * referring to its temperature channels by index: there a node + * that fails to set up keeps its channel. Elsewhere it is skipped. + */ + indexed = of_property_present(hwmon_node, "#thermal-sensor-cells"); for_each_child_of_node_with_prefix(hwmon_node, key_node, "temperature-") { sensor = &hwmon->temp.sensors[hwmon->temp.count]; - if (!macsmc_hwmon_create_sensor(hwmon->dev, hwmon->smc, key_node, sensor)) { - sensor->attrs = HWMON_T_INPUT; - - if (*sensor->label) - sensor->attrs |= HWMON_T_LABEL; - - hwmon->temp.count++; - } + ret = macsmc_hwmon_create_sensor(hwmon->dev, hwmon->smc, + key_node, sensor); + if (ret && !indexed) + continue; + if (ret) + dev_warn(hwmon->dev, "%pOFn unavailable (%d), temp%u hidden\n", + key_node, ret, hwmon->temp.count + 1); + macsmc_hwmon_add_temp_channel(&hwmon->temp, ret); } for (list = die_keys; list && (ret = macsmc_hwmon_next_key(&list, key));) { @@ -1262,6 +1356,10 @@ static int macsmc_hwmon_probe(struct platform_device *pdev) return ret; } + ret = macsmc_hwmon_discover_temperatures(hwmon); + if (ret) + return dev_err_probe(hwmon->dev, ret, "Temperature discovery failed\n"); + if (!hwmon->curr.count && !hwmon->fan.count && !hwmon->power.count && !hwmon->temp.count && !hwmon->volt.count && !hwmon->pressure.macsmc_key) { @@ -1318,6 +1416,10 @@ static struct platform_driver macsmc_hwmon_driver = { }; module_platform_driver(macsmc_hwmon_driver); +#if IS_ENABLED(CONFIG_SENSORS_MACSMC_HWMON_KUNIT_TEST) +#include "macsmc-hwmon-test.c" +#endif + MODULE_DESCRIPTION("Apple Silicon SMC hwmon driver"); MODULE_AUTHOR("James Calligeros "); MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/input/misc/macsmc-input.c b/drivers/input/misc/macsmc-input.c index 2cead3b7f45fed..e149b043fef820 100644 --- a/drivers/input/misc/macsmc-input.c +++ b/drivers/input/misc/macsmc-input.c @@ -106,6 +106,13 @@ static int macsmc_input_event(struct notifier_block *nb, unsigned long event, vo } } +static void macsmc_input_unregister_notifier(void *data) +{ + struct macsmc_input *smcin = data; + + blocking_notifier_chain_unregister(&smcin->smc->event_handlers, &smcin->nb); +} + static int macsmc_input_probe(struct platform_device *pdev) { struct apple_smc *smc = dev_get_drvdata(pdev->dev.parent); @@ -168,7 +175,19 @@ static int macsmc_input_probe(struct platform_device *pdev) input_sync(smcin->input); smcin->nb.notifier_call = macsmc_input_event; - blocking_notifier_chain_register(&smc->event_handlers, &smcin->nb); + error = blocking_notifier_chain_register(&smc->event_handlers, &smcin->nb); + if (error) + return error; + + /* + * The notifier block lives in devm-allocated memory and the callback + * reports to a devm-managed input device: take it off the SMC's chain + * before either goes away on unbind. + */ + error = devm_add_action_or_reset(&pdev->dev, + macsmc_input_unregister_notifier, smcin); + if (error) + return error; device_init_wakeup(&pdev->dev, true); diff --git a/drivers/iommu/apple-dart-test.c b/drivers/iommu/apple-dart-test.c index 2998c873747222..86ed79c8d0c1d4 100644 --- a/drivers/iommu/apple-dart-test.c +++ b/drivers/iommu/apple-dart-test.c @@ -47,9 +47,12 @@ static void apple_dart_test_before_attach(struct kunit *test) KUNIT_EXPECT_EQ(test, atomic_long_read(&domain->stream_maps[0].sidmap[0]), 0L); KUNIT_EXPECT_EQ(test, readl(dart->regs + DART_T8020_STREAM_COMMAND), 0U); - /* Once attached, missing root mappings must still report an error. */ + /* + * A stream whose locked roots are not mapped yet is left to attach, + * which publishes the complete page table. + */ KUNIT_EXPECT_EQ(test, apple_dart_domain_add_streams(domain, &master), 0); - KUNIT_EXPECT_EQ(test, apple_dart_iotlb_sync_map(&domain->domain, 0, PAGE_SIZE), -EIO); + KUNIT_EXPECT_EQ(test, apple_dart_iotlb_sync_map(&domain->domain, 0, PAGE_SIZE), 0); pm_runtime_disable(dart->dev); free_io_pgtable_ops(ops); @@ -93,6 +96,37 @@ static void apple_dart_test_locked_handoff(struct kunit *test) KUNIT_EXPECT_MEMEQ(test, live, ours, sizeof(live)); } +static void apple_dart_test_sync_unknown_roots(struct kunit *test) +{ + struct apple_dart_stream_map stream = {}; + struct io_pgtable_cfg cfg = {}; + struct apple_dart *dart; + u64 live[] = { 0x1001, 0, 0, 0 }; + u64 ours[] = { 0, 0x2001, 0, 0 }; + + dart = kunit_kzalloc(test, sizeof(*dart), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, dart); + dart->hw = &apple_dart_hw_t8110; + dart->version = 0x0202; + dart->num_streams = 1; + dart->pgsize = sizeof(live); + spin_lock_init(&dart->lock); + /* The live root is mapped, its ownership shadow is not yet. */ + dart->locked_ttbr[0][0] = live; + stream.dart = dart; + __set_bit(0, stream.sidmap); + cfg.apple_dart_cfg.n_ttbrs = 1; + cfg.apple_dart_cfg.ttbr[0] = ours; + + /* An early TLB sync skips the stream and leaves the root alone... */ + KUNIT_EXPECT_EQ(test, apple_dart_hw_sync_locked(&cfg, &stream, true), 0); + KUNIT_EXPECT_EQ(test, live[0], 0x1001ULL); + KUNIT_EXPECT_EQ(test, live[1], 0ULL); + /* ...but publishing at attach still requires both. */ + KUNIT_EXPECT_EQ(test, apple_dart_hw_sync_locked(&cfg, &stream, false), -EIO); + KUNIT_EXPECT_EQ(test, live[1], 0ULL); +} + static void apple_dart_test_firmware_roots(struct kunit *test) { struct apple_dart_stream_map stream = {}; @@ -377,6 +411,7 @@ static void apple_dart_test_gated_irq(struct kunit *test) static struct kunit_case apple_dart_test_cases[] = { KUNIT_CASE(apple_dart_test_before_attach), KUNIT_CASE(apple_dart_test_locked_handoff), + KUNIT_CASE(apple_dart_test_sync_unknown_roots), KUNIT_CASE(apple_dart_test_firmware_roots), KUNIT_CASE(apple_dart_test_gated_commands), KUNIT_CASE(apple_dart_test_resume_commands_failure), diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c index 0ee29b60992909..db90c8d3877150 100644 --- a/drivers/iommu/apple-dart.c +++ b/drivers/iommu/apple-dart.c @@ -235,6 +235,8 @@ struct apple_dart_hw { struct apple_dart_fw_root { u64 *entry; u64 **leaf; + /* Linux root entry already verified against each firmware slot. */ + u64 *accepted; }; /* @@ -635,7 +637,10 @@ static bool apple_dart_fw_slot_mirrors(const struct apple_dart *dart, * A locked root belongs to firmware. Publish only into vacant slots and * remember exactly what we installed, so teardown cannot erase firmware * mappings added after Linux attached. A slot firmware already holds is - * never written; its descendants must still mirror Linux's mappings, and + * never written. It is accepted once if it mirrors Linux's mappings, and + * apple_dart_check_fw_map() checks later mappings into it page by page. + * With the reserved-table handoff (fw_handoff) its descendants must still + * mirror Linux's mappings at every publication, and * apple_dart_check_fw_map() rejects new mappings anywhere in that slot. */ static int apple_dart_publish_root(const struct apple_dart *dart, @@ -655,7 +660,8 @@ static int apple_dart_publish_root(const struct apple_dart *dart, continue; if (previous || !apple_dart_fw_slot_owned(fw, live, i)) return -EBUSY; - if (!apple_dart_fw_slot_mirrors(dart, fw, next, i, entries)) + if ((dart->fw_handoff || next != fw->accepted[i]) && + !apple_dart_fw_slot_mirrors(dart, fw, next, i, entries)) return -EBUSY; } @@ -667,6 +673,7 @@ static int apple_dart_publish_root(const struct apple_dart *dart, if (next == previous) continue; if (!previous && apple_dart_fw_slot_owned(fw, live, i)) { + fw->accepted[i] = next; continue; } if (cmpxchg(&live[i], previous, next) != previous) @@ -703,6 +710,7 @@ static void apple_dart_free_fw_root(struct apple_dart *dart, for (i = 0; i < dart->pgsize / sizeof(u64); i++) if (fw->leaf[i]) memunmap(fw->leaf[i]); + kfree(fw->accepted); kfree(fw->leaf); kfree(fw->entry); kfree(fw); @@ -711,17 +719,23 @@ static void apple_dart_free_fw_root(struct apple_dart *dart, /* * Record the leaf tables behind the slots firmware holds in a three-level * T8110-format locked root, on DARTs whose loader re-exports firmware - * translations (fw_handoff). Other roots are not snapshotted; their firmware + * translations (fw_mirror). Other roots are not snapshotted; their firmware * slots keep rejecting Linux mappings as before. + * + * With the reserved-table handoff (fw_handoff) every table page must be + * reserved RAM that does not change while it is recorded, and any failure + * fails the attach. Without it, a leaf table that cannot be mapped is left + * out and a failed allocation leaves the root unrecorded. */ static struct apple_dart_fw_root * apple_dart_snapshot_fw_root(struct apple_dart *dart, int sid, const u64 *live) { size_t i, entries = dart->pgsize / sizeof(u64); struct apple_dart_fw_root *fw; + struct apple_dart_fw_root *nomem = dart->fw_handoff ? ERR_PTR(-ENOMEM) : NULL; /* Only the T8110-style table layout this walker decodes. */ - if (!dart->fw_handoff || dart->hw->fmt != APPLE_DART2) + if (!dart->fw_mirror || dart->hw->fmt != APPLE_DART2) return NULL; if (dart->hw->tcr_4level && (apple_dart_readl(dart, DART_TCR(dart, sid)) & dart->hw->tcr_4level)) @@ -729,12 +743,13 @@ apple_dart_snapshot_fw_root(struct apple_dart *dart, int sid, const u64 *live) fw = kzalloc(sizeof(*fw), GFP_KERNEL); if (!fw) - return ERR_PTR(-ENOMEM); + return nomem; fw->entry = kcalloc(entries, sizeof(*fw->entry), GFP_KERNEL); fw->leaf = kcalloc(entries, sizeof(*fw->leaf), GFP_KERNEL); - if (!fw->entry || !fw->leaf) { + fw->accepted = kcalloc(entries, sizeof(*fw->accepted), GFP_KERNEL); + if (!fw->entry || !fw->leaf || !fw->accepted) { apple_dart_free_fw_root(dart, fw); - return ERR_PTR(-ENOMEM); + return nomem; } for (i = 0; i < entries; i++) { @@ -743,6 +758,14 @@ apple_dart_snapshot_fw_root(struct apple_dart *dart, int sid, const u64 *live) if (!(pte & APPLE_DART_PTE_VALID)) continue; + if (!dart->fw_handoff) { + fw->leaf[i] = memremap(apple_dart_pte_to_paddr(dart, pte), + dart->pgsize, MEMREMAP_WB); + if (!fw->leaf[i]) + continue; + fw->entry[i] = pte; + continue; + } if (!apple_dart_fw_table_decode(dart, pte, &phys)) { apple_dart_free_fw_root(dart, fw); return ERR_PTR(-EINVAL); @@ -908,7 +931,8 @@ apple_dart_hw_sync_locked(struct io_pgtable_cfg *cfg, * before attachment, so replace the old root entries. * T8110 v2.2+ instead retains firmware-owned entries. */ - if (dart->hw->type != DART_T8110 || dart->version < 0x0202) { + if (!(dart->fw_mirror && !dart->fw_handoff) && + (dart->hw->type != DART_T8110 || dart->version < 0x0202)) { size_t entry; dma_wmb(); @@ -1041,7 +1065,7 @@ static bool apple_dart_fw_slot_foreign(const u64 *live, const u64 *owned, } /* - * On T8140 DARTs that keep firmware roots, the hardware keeps + * On DARTs that keep firmware roots (T8140 and T8142), the hardware keeps * using the translations behind the firmware-owned root slots, e.g. the * display controller scanning out the boot framebuffer while Linux attaches, * or the AOP firmware's own DMA on the AOP DART streams that Linux shares. @@ -1069,6 +1093,7 @@ static int apple_dart_fw_invalidate(struct apple_dart_stream_map *stream_map, return 0; for_each_set_bit(sid, stream_map->sidmap, dart->num_streams) { + const struct apple_dart_fw_root *fw; const u64 *live, *owned; u64 base, span, start_dva, end_dva; bool four_level; @@ -1107,8 +1132,16 @@ static int apple_dart_fw_invalidate(struct apple_dart_stream_map *stream_map, spin_lock_irqsave(&dart->lock, flags); live = dart->locked_ttbr[sid][0]; owned = dart->locked_owned[sid][0]; + fw = dart->locked_fw[sid][0]; + /* + * Four-level roots are not snapshotted, and with the reserved-table + * handoff firmware may replace its slots' descendants: there every + * valid slot Linux did not publish is firmware's. + */ for (i = first; live && i <= last; i++) - if (apple_dart_fw_slot_foreign(live, owned, i)) + if (four_level || dart->fw_handoff ? + apple_dart_fw_slot_foreign(live, owned, i) : + apple_dart_fw_slot_owned(fw, live, i)) __set_bit(i, fw_slots); spin_unlock_irqrestore(&dart->lock, flags); if (!live) @@ -1389,8 +1422,48 @@ static bool apple_dart_domain_uses_fw(struct apple_dart_domain *domain, int i, } /* - * Hardware continues to use every firmware-owned slot. Its entire IOVA - * span is excluded from Linux allocation, including currently unmapped holes. + * With the reserved-table handoff (fw_handoff), hardware continues to use + * every firmware-owned slot. Its entire IOVA span is excluded from Linux + * allocation, including currently unmapped holes. Called with dart->lock held. + */ +static bool apple_dart_fw_handoff_page_busy(struct apple_dart *dart, + struct apple_dart_domain *domain, + int i, + struct apple_dart_atomic_stream_map *map, + u64 addr) +{ + u32 entries = dart->pgsize / sizeof(u64); + u32 pgshift = ilog2(dart->pgsize), bits = ilog2(entries); + int sid; + + for (sid = 0; sid < dart->num_streams; sid++) { + bool four; + u64 idx, slot; + const u64 *live, *owned; + + if (!apple_dart_domain_uses_fw(domain, i, map, sid)) + continue; + four = apple_dart_readl(dart, DART_TCR(dart, sid)) & + dart->hw->tcr_4level; + idx = four ? 0 : addr >> (pgshift + 2 * bits); + slot = (addr >> (pgshift + (four ? 2 : 1) * bits)) & + (entries - 1); + if (idx >= DART_MAX_TTBR) + continue; + live = dart->locked_ttbr[sid][idx]; + owned = dart->locked_owned[sid][idx]; + /* All pages of an inherited slot remain firmware's. */ + if (live && apple_dart_fw_slot_foreign(live, owned, slot)) + return true; + } + return false; +} + +/* + * Hardware continues to use every firmware-owned slot. Accept a private + * software mapping only when every stream using such a slot agrees on the + * full-page translation and its protection/cache attributes. With the + * reserved-table handoff, refuse any mapping into such a slot instead. */ static int apple_dart_check_fw_map(struct apple_dart_domain *domain, u64 dva, phys_addr_t paddr, size_t size, int prot) @@ -1405,30 +1478,43 @@ static int apple_dart_check_fw_map(struct apple_dart_domain *domain, u64 dva, unsigned long flags; size_t off; - if (!dart->fw_handoff) + if (!dart->fw_mirror) continue; spin_lock_irqsave(&dart->lock, flags); for (off = 0; off < size; off += dart->pgsize) { u64 addr = dva + off; + u64 leaf = (addr >> pgshift) & (entries - 1); + u64 slot = (addr >> (pgshift + bits)) & (entries - 1); + u64 idx = addr >> (pgshift + 2 * bits); + if (dart->fw_handoff) { + if (apple_dart_fw_handoff_page_busy(dart, domain, i, + map, addr)) { + ret = -EBUSY; + goto unlock; + } + continue; + } + if (idx >= DART_MAX_TTBR) + continue; for (sid = 0; sid < dart->num_streams; sid++) { - bool four; - u64 idx, slot; - const u64 *live, *owned; + const struct apple_dart_fw_root *fw; + phys_addr_t phys; + u64 pte; + int fw_prot; if (!apple_dart_domain_uses_fw(domain, i, map, sid)) continue; - four = apple_dart_readl(dart, DART_TCR(dart, sid)) & - dart->hw->tcr_4level; - idx = four ? 0 : addr >> (pgshift + 2 * bits); - slot = (addr >> (pgshift + (four ? 2 : 1) * bits)) & - (entries - 1); - if (idx >= DART_MAX_TTBR) + fw = dart->locked_fw[sid][idx]; + if (!dart->locked_ttbr[sid][idx] || + !apple_dart_fw_slot_owned(fw, + dart->locked_ttbr[sid][idx], slot)) continue; - live = dart->locked_ttbr[sid][idx]; - owned = dart->locked_owned[sid][idx]; - /* All pages of an inherited slot remain firmware's. */ - if (live && apple_dart_fw_slot_foreign(live, owned, slot)) { + pte = READ_ONCE(fw->leaf[slot][leaf]); + if (!apple_dart_fw_leaf_decode(pte, addr, + &phys, &fw_prot) || + !apple_dart_fw_mapping_matches(phys, fw_prot, + paddr + off, prot)) { ret = -EBUSY; goto unlock; } @@ -1456,7 +1542,7 @@ apple_dart_inherited_phys(struct apple_dart_domain *domain, dma_addr_t address) size_t page = (address / dart->pgsize) % entries; size_t slot = address / dart->pgsize / entries; - if (!dart->locked || !dart->fw_handoff || slot >= entries) + if (!dart->locked || !dart->fw_mirror || slot >= entries) return 0; spin_lock_irqsave(&dart->lock, flags); for (sid = 0; sid < dart->num_streams; sid++) { @@ -1726,7 +1812,7 @@ static int apple_dart_finalize_domain(struct apple_dart_domain *dart_domain, /* Make inherited translations visible before core reservation setup. */ for_each_stream_map(i, cfg, stream) { - if (!stream->dart->locked || !stream->dart->fw_handoff) + if (!stream->dart->locked || !stream->dart->fw_mirror) continue; apple_dart_set_locked_window(dart_domain, stream); for (index = 0; index < pgtbl_cfg.apple_dart_cfg.n_ttbrs; index++) { @@ -2846,8 +2932,10 @@ static int apple_dart_probe(struct platform_device *pdev) dart->locked = false; else dart->locked = apple_dart_is_locked(dart); + /* These loaders re-export the firmware's own display translations. */ dart->fw_mirror = dart->locked && - of_device_is_compatible(dev->of_node, "apple,t8140-dart"); + (of_device_is_compatible(pdev->dev.of_node, "apple,t8140-dart") || + of_device_is_compatible(pdev->dev.of_node, "apple,t8142-dart")); /* Reserved table-page lifetime requires the matched three-level loader. */ dart->fw_handoff = apple_dart_fw_handoff_enabled(dart); if (!dart->locked) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index f1f8e01a7e9142..57069672545172 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -36,13 +36,13 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } -static u64 page_size_to_cd(void) +static u64 page_size_to_cd(unsigned long page_size) { static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || PAGE_SIZE == SZ_64K); - if (PAGE_SIZE == SZ_64K) + if (page_size == SZ_64K) return ARM_LPAE_TCR_TG0_64K; - if (PAGE_SIZE == SZ_16K) + if (page_size == SZ_16K) return ARM_LPAE_TCR_TG0_16K; return ARM_LPAE_TCR_TG0_4K; } @@ -83,7 +83,7 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, target->data[0] |= cpu_to_le64( FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd(mm_page_size(mm))) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, @@ -138,7 +138,7 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, */ size = end - start; - arm_smmu_domain_inv_range(smmu_domain, start, size, PAGE_SIZE, false); + arm_smmu_domain_inv_range(smmu_domain, start, size, mm_page_size(mm), false); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) @@ -256,6 +256,8 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) return -EOPNOTSUPP; + if (!(master->smmu->pgsize_bitmap & mm_page_size(domain->mm))) + return -EOPNOTSUPP; /* Prevent arm_smmu_mm_release from being called while we are attaching */ if (!mmget_not_zero(domain->mm)) @@ -312,6 +314,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) return ERR_PTR(-EOPNOTSUPP); + if (!(smmu->pgsize_bitmap & mm_page_size(mm))) + return ERR_PTR(-EOPNOTSUPP); smmu_domain = arm_smmu_domain_alloc(); if (IS_ERR(smmu_domain)) @@ -323,7 +327,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, * Choose page_size as the leaf page size for invalidation when * ARM_SMMU_FEAT_RANGE_INV is present */ - smmu_domain->domain.pgsize_bitmap = PAGE_SIZE; + smmu_domain->domain.pgsize_bitmap = mm_page_size(mm); smmu_domain->stage = ARM_SMMU_DOMAIN_SVA; smmu_domain->smmu = smmu; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c index add671363c828c..fa1c7b03394d68 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c @@ -616,12 +616,25 @@ static void arm_smmu_v3_write_cd_test_sva_clear(struct kunit *test) { struct arm_smmu_cd cd = {}; struct arm_smmu_cd cd_2; + const unsigned long flags[] = { 0, MMCF_USER_4K, MMCF_USER_16K }; + /* TG0 encodings: 4K = 0, 64K = 1, 16K = 2. */ + const unsigned int tg[] = { PAGE_SHIFT == 16 ? 1 : PAGE_SHIFT == 14 ? 2 : 0, 0, 2 }; + unsigned int i; - arm_smmu_test_make_sva_cd(&cd_2, 1997); - arm_smmu_v3_test_cd_expect_non_hitless_transition( - test, &cd, &cd_2, NUM_EXPECTED_SYNCS(2)); - arm_smmu_v3_test_cd_expect_non_hitless_transition( - test, &cd_2, &cd, NUM_EXPECTED_SYNCS(2)); + for (i = 0; i < ARRAY_SIZE(flags); i++) { + if (i && (!IS_ENABLED(CONFIG_ARM64_USER4K_EXPERIMENTAL) || + (i == 2 && PAGE_SHIFT <= 14))) + continue; + sva_mm.context.flags = flags[i]; + arm_smmu_test_make_sva_cd(&cd_2, 1997); + KUNIT_EXPECT_EQ(test, FIELD_GET(CTXDESC_CD_0_TCR_TG0, + le64_to_cpu(cd_2.data[0])), tg[i]); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd, &cd_2, NUM_EXPECTED_SYNCS(2)); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd_2, &cd, NUM_EXPECTED_SYNCS(2)); + } + sva_mm.context.flags = 0; } static void arm_smmu_v3_write_cd_test_sva_release(struct kunit *test) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 87581d6778009d..c8ca796ee542c3 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3350,10 +3350,12 @@ static ssize_t iommu_group_store_type(struct iommu_group *group, * * The device driver about to bind @dev wants to do DMA through the kernel * DMA API. Return 0 if it is allowed, otherwise an error. + * Drivers with driver_managed_dma may manage this usage themselves. The + * device lock must serialize probe and removal while holding the usage. */ int iommu_device_use_default_domain(struct device *dev) { - /* Caller is the driver core during the pre-probe path */ + /* Called with the device lock held during probe. */ struct iommu_group *group = dev->iommu_group; int ret = 0; @@ -3380,6 +3382,7 @@ int iommu_device_use_default_domain(struct device *dev) mutex_unlock(&group->mutex); return ret; } +EXPORT_SYMBOL_GPL(iommu_device_use_default_domain); /** * iommu_device_unuse_default_domain() - Device driver stops handling device @@ -3391,7 +3394,7 @@ int iommu_device_use_default_domain(struct device *dev) */ void iommu_device_unuse_default_domain(struct device *dev) { - /* Caller is the driver core during the post-probe path */ + /* Called with the device lock held during removal. */ struct iommu_group *group = dev->iommu_group; if (!group) @@ -3403,6 +3406,7 @@ void iommu_device_unuse_default_domain(struct device *dev) mutex_unlock(&group->mutex); } +EXPORT_SYMBOL_GPL(iommu_device_unuse_default_domain); static int __iommu_group_alloc_blocking_domain(struct iommu_group *group) { diff --git a/drivers/iommu/iommufd/Makefile b/drivers/iommu/iommufd/Makefile index 71d692c9a8f49d..a868155e208d53 100644 --- a/drivers/iommu/iommufd/Makefile +++ b/drivers/iommu/iommufd/Makefile @@ -10,6 +10,8 @@ iommufd-y := \ vfio_compat.o \ viommu.o +iommufd-$(CONFIG_MM_SUBPAGE) += fragments.o + iommufd-$(CONFIG_IOMMUFD_TEST) += selftest.o obj-$(CONFIG_IOMMUFD) += iommufd.o diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 2895e5370910fe..3ccb352bb3e506 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -1171,6 +1171,18 @@ struct iommufd_access *iommufd_access_create_internal(struct iommufd_ctx *ictx) return access; } +/* Physical byte-span consumers do not impose native-page IOVA alignment. */ +struct iommufd_access *iommufd_access_create_internal_phys(struct iommufd_ctx *ictx) +{ + struct iommufd_access *access = __iommufd_access_create(ictx); + + if (IS_ERR(access)) + return access; + access->iova_alignment = 1; + iommufd_object_finalize(ictx, &access->obj); + return access; +} + /** * iommufd_access_create - Create an iommufd_access * @ictx: iommufd file descriptor @@ -1404,6 +1416,12 @@ static bool check_area_prot(struct iopt_area *area, unsigned int flags) * ioas alignment is >= PAGE_SIZE and the iova is PAGE_SIZE aligned. However * smaller alignments have corner cases where this API can fail on otherwise * aligned iova. + * + * Alternative user granules are accepted only if each native output page has + * an aligned physical base and consecutive backing bytes within the requested + * range. The driver must access only @length bytes: a partial final page does + * not grant access to its remaining bytes. Fragment ownership stays in the IOAS + * until unpin; incompatible physical layouts return -EINVAL. */ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, struct page **out_pages, @@ -1456,7 +1474,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, flags, internal); if (rc) goto err_remove; - out_pages += last_index - index + 1; + out_pages += (last - iter.cur_iova) / PAGE_SIZE + 1; } if (!iopt_area_contig_done(&iter)) { rc = -ENOENT; @@ -1485,6 +1503,88 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, } EXPORT_SYMBOL_NS_GPL(iommufd_access_pin_pages, "IOMMUFD"); +/** + * iommufd_access_pin_phys - Pin an internal contiguous physical byte span + * @access: Internal access created for byte-span pinning + * @iova: First IOVA byte + * @length: Exact byte length + * @out_phys: Physical address of the first byte, set only on success + * + * Retains native or fragment pin ownership and prevents IOAS unmap until the + * matching iommufd_access_unpin_pages(). No surrounding IOVA bytes are required. + * The caller must stop hardware access before unpinning or destroying access. + */ +int iommufd_access_pin_phys(struct iommufd_access *access, unsigned long iova, + unsigned long length, phys_addr_t *out_phys) +{ + struct iopt_area_contig_iter iter; + struct io_pagetable *iopt; + struct iopt_area *area; + unsigned long last_iova; + phys_addr_t base = 0; + int rc; + + if (!iommufd_access_is_internal(access) || !out_phys || !length) + return -EINVAL; + if (check_add_overflow(iova, length - 1, &last_iova)) + return -EOVERFLOW; + + mutex_lock(&access->ioas_lock); + if (!access->ioas) { + rc = -ENOENT; + goto out_unlock; + } + iopt = &access->ioas->iopt; + down_read(&iopt->iova_rwsem); + iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova) { + unsigned long last = min(last_iova, iopt_area_last_iova(area)); + phys_addr_t phys; + + if (area->prevent_access || !(area->iommu_prot & IOMMU_READ)) { + rc = -EPERM; + goto out_remove; + } + rc = iopt_area_add_phys_access(area, + iopt_area_start_byte(area, iter.cur_iova), + last - iter.cur_iova + 1, &phys); + if (rc) + goto out_remove; + if (iter.cur_iova == iova) { + base = phys; + } else if (phys != base + (iter.cur_iova - iova)) { + iopt_area_remove_access(area, + iopt_area_iova_to_index(area, iter.cur_iova), + iopt_area_iova_to_index(area, last), true); + rc = -EFAULT; + goto out_remove; + } + } + if (!iopt_area_contig_done(&iter)) { + rc = -ENOENT; + goto out_remove; + } + *out_phys = base; + up_read(&iopt->iova_rwsem); + rc = 0; + goto out_unlock; + +out_remove: + /* iter.cur_iova is the first byte whose access was not retained. */ + if (iter.cur_iova > iova) { + unsigned long end = iter.cur_iova - 1; + + iopt_for_each_contig_area(&iter, area, iopt, iova, end) + iopt_area_remove_access(area, + iopt_area_iova_to_index(area, iter.cur_iova), + iopt_area_iova_to_index(area, min(end, iopt_area_last_iova(area))), + true); + } + up_read(&iopt->iova_rwsem); +out_unlock: + mutex_unlock(&access->ioas_lock); + return rc; +} + /** * iommufd_access_rw - Read or write data under the iova * @access: IOAS access to act on diff --git a/drivers/iommu/iommufd/fragments.c b/drivers/iommu/iommufd/fragments.c new file mode 100644 index 00000000000000..5229605852133d --- /dev/null +++ b/drivers/iommu/iommufd/fragments.c @@ -0,0 +1,421 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Alternative user granules need an ownership record in addition to a physical + * address. A device translation cannot retain a subpage allocation identity. + * Keep those records until the final domain interval has gone away. The native + * PFN reader remains in pages.c; file and dma-buf suppliers retain its geometry. + */ +#include +#include +#include +#include + +#include "iommufd_private.h" +#include "io_pagetable.h" + +static phys_addr_t fragment_phys(const struct user_page_fragment *fragment) +{ + return PFN_PHYS(folio_pfn(fragment->folio)) + fragment->offset; +} + +/* Charge each native physical page once, even when several slots pin it. */ +static int fragment_store(struct iopt_pages *pages, unsigned long index, + struct user_page_fragment *fragment) +{ + unsigned long pfn = PHYS_PFN(fragment_phys(fragment)); + unsigned long count = xa_to_value(xa_load(&pages->fragment_pages, pfn)); + struct user_page_fragment *record; + int rc; + + record = kmemdup(fragment, sizeof(*record), GFP_KERNEL_ACCOUNT); + if (!record) + return -ENOMEM; + rc = xa_reserve(&pages->fragments, index, GFP_KERNEL_ACCOUNT); + if (rc) + goto out_free; + rc = xa_err(xa_store(&pages->fragment_pages, pfn, xa_mk_value(count + 1), + GFP_KERNEL_ACCOUNT)); + if (rc) + goto out_release; + /* xa_reserve() made this infallible. Ownership transfers here. */ + xa_store(&pages->fragments, index, record, GFP_NOWAIT); + if (!count) + pages->npinned++; + *fragment = (struct user_page_fragment) {}; + return 0; + +out_release: + xa_release(&pages->fragments, index); +out_free: + kfree(record); + return rc; +} + +/* No allocation on teardown, and callers have already unmapped all devices. */ +static void fragments_release_unused(struct iopt_pages *pages, + unsigned long first, unsigned long last) +{ + struct user_page_fragment *record; + unsigned long index = first; + + lockdep_assert_held(&pages->mutex); + xa_for_each_range(&pages->fragments, index, record, first, last) { + unsigned long pfn, count; + + if (interval_tree_iter_first(&pages->domains_itree, index, index) || + interval_tree_iter_first(&pages->access_itree, index, index)) + continue; + pfn = PHYS_PFN(fragment_phys(record)); + count = xa_to_value(xa_load(&pages->fragment_pages, pfn)); + if (WARN_ON(!count)) + continue; + if (count == 1) { + xa_erase(&pages->fragment_pages, pfn); + pages->npinned--; + } else { + xa_store(&pages->fragment_pages, pfn, xa_mk_value(count - 1), + GFP_NOWAIT); + } + xa_erase(&pages->fragments, index); + release_user_fragments(record, 1, pages->writable); + kfree(record); + cond_resched(); + } + if (pages->npinned < pages->last_npinned) + iopt_pages_update_pinned(pages, pages->last_npinned - pages->npinned, + false, NULL); +} + +/* Reuse existing records even if the original VA was unmapped or replaced. */ +static int fragments_acquire(struct iopt_pages *pages, unsigned long first, + unsigned long last) +{ + unsigned long size = 1UL << pages->page_shift, index = first; + unsigned long capacity = PAGE_SIZE / sizeof(struct user_page_fragment); + struct user_page_fragment *batch = NULL; + bool mm_live = false; + int rc = 0; + + lockdep_assert_held(&pages->mutex); + while (index <= last) { + unsigned long next = index, end, i; + long count; + + if (xa_load(&pages->fragments, index)) { + index++; + continue; + } + if (!mm_live) { + if (!mmget_not_zero(pages->source_mm)) { + rc = -EFAULT; + goto out; + } + mm_live = true; + batch = kmalloc(PAGE_SIZE, GFP_KERNEL_ACCOUNT); + if (!batch) { + rc = -ENOMEM; + goto out; + } + } + /* Stop before any cached pin; never replace its ownership record. */ + end = xa_find(&pages->fragments, &next, last, XA_PRESENT) ? next : last + 1; + end = min(end, index + capacity); + count = pin_user_fragments_remote(pages->source_mm, + (unsigned long)pages->uptr + (index << pages->page_shift), + (end - index) << pages->page_shift, + FOLL_LONGTERM | (pages->writable ? FOLL_WRITE : 0), batch, capacity); + if (count <= 0) { + rc = count ?: -EFAULT; + goto out; + } + for (i = 0; i < count; i++) { + /* Each aligned logical index must have exactly one full pin. */ + if (WARN_ON(batch[i].length != size || + !IS_ALIGNED(fragment_phys(&batch[i]), size))) + rc = -EFAULT; + else + rc = fragment_store(pages, index, &batch[i]); + if (rc) { + release_user_fragments(batch + i, count - i, false); + goto out; + } + index++; + } + cond_resched(); + } + if (pages->npinned != pages->last_npinned) { + if (iommufd_should_fail()) + rc = -ENOMEM; + else + rc = iopt_pages_update_pinned(pages, + pages->npinned - pages->last_npinned, true, NULL); + } +out: + kfree(batch); + if (mm_live) + mmput(pages->source_mm); + if (rc) + fragments_release_unused(pages, first, last); + return rc; +} + +static void fragments_unmap(struct iommu_domain *domain, unsigned long iova, + unsigned long length) +{ + if (length) + WARN_ON(iommu_unmap(domain, iova, length) != length); +} + +/* All pins are present. Merge physically consecutive fragments before mapping. */ +static int fragments_map(struct iopt_area *area, struct iopt_pages *pages, + struct iommu_domain *domain) +{ + unsigned long size = 1UL << pages->page_shift; + unsigned long index = iopt_area_index(area), last = iopt_area_last_index(area); + unsigned long remaining = iopt_area_length(area), done = 0; + unsigned long offset = area->page_offset; + unsigned long min_size = 1UL << __ffs(domain->pgsize_bitmap); + int rc; + + while (remaining) { + struct user_page_fragment *record = xa_load(&pages->fragments, index); + phys_addr_t phys = fragment_phys(record) + offset; + unsigned long bytes = min(size - offset, remaining); + + index++; + /* Legacy splitting forbids mappings larger than a native page. */ + while (index <= last && bytes < remaining && + (!area->iopt->disable_large_pages || bytes < PAGE_SIZE)) { + record = xa_load(&pages->fragments, index); + if (fragment_phys(record) != phys + bytes) + break; + bytes += min(size, remaining - bytes); + index++; + } + /* Never widen a device mapping to include an unowned neighbour. */ + if (!IS_ALIGNED(phys | (iopt_area_iova(area) + done) | bytes, min_size)) { + rc = -EINVAL; + goto out_unmap; + } + rc = iommu_map(domain, iopt_area_iova(area) + done, phys, bytes, + area->iommu_prot, GFP_KERNEL_ACCOUNT); + if (rc) + goto out_unmap; + done += bytes; + remaining -= bytes; + offset = 0; + cond_resched(); + } + return 0; + +out_unmap: + fragments_unmap(domain, iopt_area_iova(area), done); + return rc; +} + +int iopt_fragments_fill_domain(struct iopt_area *area, struct iopt_pages *pages, + struct iommu_domain *domain) +{ + int rc; + + lockdep_assert_held(&pages->mutex); + rc = fragments_acquire(pages, iopt_area_index(area), iopt_area_last_index(area)); + if (rc) + return rc; + rc = fragments_map(area, pages, domain); + if (rc) + fragments_release_unused(pages, iopt_area_index(area), iopt_area_last_index(area)); + return rc; +} + +int iopt_fragments_fill_domains(struct iopt_area *area, struct iopt_pages *pages) +{ + struct iommu_domain *domain; + unsigned long index, failed; + int rc; + + guard(mutex)(&pages->mutex); + rc = fragments_acquire(pages, iopt_area_index(area), iopt_area_last_index(area)); + if (rc) + return rc; + xa_for_each(&area->iopt->domains, index, domain) { + rc = fragments_map(area, pages, domain); + if (rc) + goto out_unmap; + } + area->storage_domain = xa_load(&area->iopt->domains, 0); + interval_tree_insert(&area->pages_node, &pages->domains_itree); + return 0; + +out_unmap: + failed = index; + xa_for_each(&area->iopt->domains, index, domain) { + if (index == failed) + break; + fragments_unmap(domain, iopt_area_iova(area), iopt_area_length(area)); + } + fragments_release_unused(pages, iopt_area_index(area), iopt_area_last_index(area)); + return rc; +} + +void iopt_fragments_unfill_domain(struct iopt_area *area, struct iopt_pages *pages, + struct iommu_domain *domain) +{ + lockdep_assert_held(&pages->mutex); + fragments_unmap(domain, iopt_area_iova(area), iopt_area_length(area)); + fragments_release_unused(pages, iopt_area_index(area), iopt_area_last_index(area)); +} + +int iopt_fragments_rw(struct iopt_pages *pages, unsigned long start_byte, + void *data, unsigned long length, unsigned int flags) +{ + unsigned long size = 1UL << pages->page_shift; + unsigned long first = start_byte >> pages->page_shift; + unsigned long last = (start_byte + length - 1) >> pages->page_shift; + unsigned long index, offset = start_byte & (size - 1); + int rc; + + guard(mutex)(&pages->mutex); + rc = fragments_acquire(pages, first, last); + if (rc) + return rc; + for (index = first; index <= last; index++) { + struct user_page_fragment *record = xa_load(&pages->fragments, index); + unsigned long bytes = min(size - offset, length); + void *addr = kmap_local_folio(record->folio, record->offset + offset); + + if (flags & IOMMUFD_ACCESS_RW_WRITE) + memcpy(addr, data, bytes); + else + memcpy(data, addr, bytes); + kunmap_local(addr); + data += bytes; + length -= bytes; + offset = 0; + } + fragments_release_unused(pages, first, last); + return 0; +} + +/* The retained typed records, not the output page array, own the pins. */ +static int fragments_register_access(struct iopt_area *area, unsigned long first, + unsigned long last, bool lock_area) +{ + struct iopt_pages *pages = area->pages; + struct iopt_pages_access *access; + struct interval_tree_node *node; + + lockdep_assert_held(&pages->mutex); + for (node = interval_tree_iter_first(&pages->access_itree, first, last); + node; node = interval_tree_iter_next(node, first, last)) { + if (node->start == first && node->last == last) { + access = container_of(node, struct iopt_pages_access, node); + access->users++; + goto out_count; + } + } + access = kzalloc_obj(*access, GFP_KERNEL_ACCOUNT); + if (!access) + return -ENOMEM; + access->node.start = first; + access->node.last = last; + access->users = 1; + interval_tree_insert(&access->node, &pages->access_itree); +out_count: + area->num_accesses++; + if (lock_area) + area->num_locks++; + return 0; +} + +/* Register a physical byte span without discarding fragment ownership. */ +int iopt_fragments_add_phys_access(struct iopt_area *area, unsigned long start_byte, + unsigned long length, phys_addr_t *out_phys) +{ + struct iopt_pages *pages = area->pages; + unsigned long size = 1UL << pages->page_shift; + unsigned long first = start_byte >> pages->page_shift; + unsigned long last = (start_byte + length - 1) >> pages->page_shift; + struct user_page_fragment *record; + unsigned long index; + phys_addr_t base; + int rc; + + guard(mutex)(&pages->mutex); + rc = fragments_acquire(pages, first, last); + if (rc) + return rc; + record = xa_load(&pages->fragments, first); + base = fragment_phys(record); + for (index = first + 1; index <= last; index++) { + record = xa_load(&pages->fragments, index); + if (fragment_phys(record) != + base + ((index - first) << pages->page_shift)) { + rc = -EFAULT; + goto out_release; + } + cond_resched(); + } + rc = fragments_register_access(area, first, last, true); + if (rc) + goto out_release; + *out_phys = base + (start_byte & (size - 1)); + return 0; + +out_release: + fragments_release_unused(pages, first, last); + return rc; +} + +/* + * A native page pointer has no fragment offset. Require an aligned physical + * base and consecutive fragments within each output page; different output + * pages may be scattered. Only the requested logical interval is retained. + */ +int iopt_fragments_add_access(struct iopt_area *area, unsigned long first, + unsigned long last, struct page **out_pages, + bool lock_area) +{ + struct iopt_pages *pages = area->pages; + unsigned long per_page = 1UL << (PAGE_SHIFT - pages->page_shift); + unsigned long index; + phys_addr_t base = 0; + int rc; + + if (first & (per_page - 1)) + return -EINVAL; + guard(mutex)(&pages->mutex); + rc = fragments_acquire(pages, first, last); + if (rc) + return rc; + for (index = first; index <= last; index++) { + struct user_page_fragment *record = xa_load(&pages->fragments, index); + unsigned long offset = (index - first) & (per_page - 1); + phys_addr_t phys = fragment_phys(record); + + if (!offset) { + base = phys; + if (!IS_ALIGNED(base, PAGE_SIZE)) { + rc = -EINVAL; + goto out_release; + } + *out_pages++ = pfn_to_page(PHYS_PFN(base)); + } else if (phys != base + (offset << pages->page_shift)) { + rc = -EINVAL; + goto out_release; + } + cond_resched(); + } + rc = fragments_register_access(area, first, last, lock_area); + if (!rc) + return 0; +out_release: + fragments_release_unused(pages, first, last); + return rc; +} + +void iopt_fragments_unfill_access(struct iopt_pages *pages, unsigned long first, + unsigned long last) +{ + fragments_release_unused(pages, first, last); +} diff --git a/drivers/iommu/iommufd/io_pagetable.c b/drivers/iommu/iommufd/io_pagetable.c index 24d4917105d942..bf8cabee53fb6c 100644 --- a/drivers/iommu/iommufd/io_pagetable.c +++ b/drivers/iommu/iommufd/io_pagetable.c @@ -217,7 +217,8 @@ static int iopt_insert_area(struct io_pagetable *iopt, struct iopt_area *area, return -EPERM; area->iommu_prot = iommu_prot; - area->page_offset = start_byte % PAGE_SIZE; + area->page_shift = pages->page_shift; + area->page_offset = start_byte & ((1UL << area->page_shift) - 1); if (area->page_offset & (iopt->iova_alignment - 1)) return -EINVAL; @@ -225,10 +226,10 @@ static int iopt_insert_area(struct io_pagetable *iopt, struct iopt_area *area, if (check_add_overflow(iova, length - 1, &area->node.last)) return -EOVERFLOW; - area->pages_node.start = start_byte / PAGE_SIZE; + area->pages_node.start = start_byte >> area->page_shift; if (check_add_overflow(start_byte, length - 1, &area->pages_node.last)) return -EOVERFLOW; - area->pages_node.last = area->pages_node.last / PAGE_SIZE; + area->pages_node.last >>= area->page_shift; if (WARN_ON(area->pages_node.last >= pages->npages)) return -EOVERFLOW; @@ -572,7 +573,7 @@ iommu_read_and_clear_dirty(struct iommu_domain *domain, struct iommu_dirty_bitmap dirty; struct iova_bitmap_fn_arg arg; struct iova_bitmap *iter; - int ret = 0; + int ret; if (!ops || !ops->read_and_clear_dirty) return -EOPNOTSUPP; @@ -581,7 +582,7 @@ iommu_read_and_clear_dirty(struct iommu_domain *domain, bitmap->page_size, u64_to_user_ptr(bitmap->data)); if (IS_ERR(iter)) - return -ENOMEM; + return PTR_ERR(iter); iommu_dirty_bitmap_init(&dirty, iter, &gather); @@ -589,7 +590,7 @@ iommu_read_and_clear_dirty(struct iommu_domain *domain, arg.iopt = iopt; arg.domain = domain; arg.dirty = &dirty; - iova_bitmap_for_each(iter, &arg, __iommu_read_and_clear_dirty); + ret = iova_bitmap_for_each(iter, &arg, __iommu_read_and_clear_dirty); if (!(flags & IOMMU_DIRTY_NO_CLEAR)) iommu_iotlb_sync(domain, &gather); @@ -615,7 +616,7 @@ int iommufd_check_iova_range(struct io_pagetable *iopt, ((last_iova + 1) & (iommu_pgsize - 1))) return -EINVAL; - if (!bitmap->page_size) + if (!is_power_of_2(bitmap->page_size)) return -EINVAL; if ((bitmap->iova & (bitmap->page_size - 1)) || diff --git a/drivers/iommu/iommufd/io_pagetable.h b/drivers/iommu/iommufd/io_pagetable.h index 27e3e311d395b2..d15e51fee0c107 100644 --- a/drivers/iommu/iommufd/io_pagetable.h +++ b/drivers/iommu/iommufd/io_pagetable.h @@ -5,6 +5,7 @@ #ifndef __IO_PAGETABLE_H #define __IO_PAGETABLE_H +#include #include #include #include @@ -45,6 +46,7 @@ struct iopt_area { struct iommu_domain *storage_domain; /* How many bytes into the first page the area starts */ unsigned int page_offset; + u8 page_shift; /* IOMMU_READ, IOMMU_WRITE, etc */ int iommu_prot; bool prevent_access : 1; @@ -107,8 +109,8 @@ static inline size_t iopt_area_length(struct iopt_area *area) /* * Number of bytes from the start of the iopt_pages that the iova begins. - * iopt_area_start_byte() / PAGE_SIZE encodes the starting page index - * iopt_area_start_byte() % PAGE_SIZE encodes the offset within that page + * iopt_area_start_byte() >> area->page_shift encodes the starting page index + * iopt_area_start_byte() % (1UL << area->page_shift) encodes the offset within that page */ static inline unsigned long iopt_area_start_byte(struct iopt_area *area, unsigned long iova) @@ -117,13 +119,13 @@ static inline unsigned long iopt_area_start_byte(struct iopt_area *area, WARN_ON(iova < iopt_area_iova(area) || iova > iopt_area_last_iova(area)); return (iova - iopt_area_iova(area)) + area->page_offset + - iopt_area_index(area) * PAGE_SIZE; + (iopt_area_index(area) << area->page_shift); } static inline unsigned long iopt_area_iova_to_index(struct iopt_area *area, unsigned long iova) { - return iopt_area_start_byte(area, iova) / PAGE_SIZE; + return iopt_area_start_byte(area, iova) >> area->page_shift; } #define __make_iopt_iter(name) \ @@ -214,7 +216,7 @@ struct iopt_pages_dmabuf { * io_pagetable's, through their iopt_area's, can share a single iopt_pages * which avoids multi-pinning and double accounting of page consumption. * - * indexes in this structure are measured in PAGE_SIZE units, are 0 based from + * Indexes use (1UL << page_shift) units and are 0 based from * the start of the uptr and extend to npages. pages are pinned dynamically * according to the intervals in the access_itree and domains_itree, npinned * records the current number of pages pinned. @@ -222,6 +224,8 @@ struct iopt_pages_dmabuf { struct iopt_pages { struct kref kref; struct mutex mutex; + /* Logical indices use page_shift; pin accounting remains native pages. */ + u8 page_shift; size_t npages; size_t npinned; size_t last_npinned; @@ -242,12 +246,73 @@ struct iopt_pages { u8 account_mode; struct xarray pinned_pfns; +#ifdef CONFIG_MM_SUBPAGE + /* Exact fragment pins, indexed independently of device translations. */ + struct xarray fragments; + /* Native physical PFN -> fragment count, for pin accounting. */ + struct xarray fragment_pages; +#endif /* Of iopt_pages_access::node */ struct rb_root_cached access_itree; /* Of iopt_area::pages_node */ struct rb_root_cached domains_itree; }; +static inline bool iopt_pages_use_fragments(struct iopt_pages *pages) +{ + return IS_ENABLED(CONFIG_MM_SUBPAGE) && pages->type == IOPT_ADDRESS_USER && + mm_page_size(pages->source_mm) != PAGE_SIZE; +} + +#ifdef CONFIG_MM_SUBPAGE +int iopt_fragments_fill_domain(struct iopt_area *area, struct iopt_pages *pages, + struct iommu_domain *domain); +int iopt_fragments_fill_domains(struct iopt_area *area, struct iopt_pages *pages); +void iopt_fragments_unfill_domain(struct iopt_area *area, struct iopt_pages *pages, + struct iommu_domain *domain); +int iopt_fragments_add_access(struct iopt_area *area, unsigned long first, + unsigned long last, struct page **out_pages, + bool lock_area); +int iopt_fragments_add_phys_access(struct iopt_area *area, unsigned long start_byte, + unsigned long length, phys_addr_t *out_phys); +void iopt_fragments_unfill_access(struct iopt_pages *pages, unsigned long first, + unsigned long last); +int iopt_fragments_rw(struct iopt_pages *pages, unsigned long start_byte, + void *data, unsigned long length, unsigned int flags); +#else +static inline int iopt_fragments_fill_domain(struct iopt_area *area, + struct iopt_pages *pages, struct iommu_domain *domain) +{ + return -EOPNOTSUPP; +} +static inline int iopt_fragments_fill_domains(struct iopt_area *area, + struct iopt_pages *pages) +{ + return -EOPNOTSUPP; +} +static inline void iopt_fragments_unfill_domain(struct iopt_area *area, + struct iopt_pages *pages, struct iommu_domain *domain) { } +static inline int iopt_fragments_add_access(struct iopt_area *area, + unsigned long first, unsigned long last, struct page **out_pages, + bool lock_area) +{ + return -EOPNOTSUPP; +} +static inline int iopt_fragments_add_phys_access(struct iopt_area *area, + unsigned long start_byte, unsigned long length, phys_addr_t *out_phys) +{ + return -EOPNOTSUPP; +} +static inline void iopt_fragments_unfill_access(struct iopt_pages *pages, + unsigned long first, unsigned long last) { } +static inline int iopt_fragments_rw(struct iopt_pages *pages, + unsigned long start_byte, void *data, unsigned long length, + unsigned int flags) +{ + return -EOPNOTSUPP; +} +#endif + static inline bool iopt_is_dmabuf(struct iopt_pages *pages) { if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) @@ -290,6 +355,8 @@ void iopt_pages_unfill_xarray(struct iopt_pages *pages, unsigned long start, int iopt_area_add_access(struct iopt_area *area, unsigned long start, unsigned long last, struct page **out_pages, unsigned int flags, bool lock_area); +int iopt_area_add_phys_access(struct iopt_area *area, unsigned long start_byte, + unsigned long length, phys_addr_t *out_phys); void iopt_area_remove_access(struct iopt_area *area, unsigned long start, unsigned long last, bool unlock_area); int iopt_pages_rw_access(struct iopt_pages *pages, unsigned long start_byte, diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommufd/iommufd_private.h index 43fbc5bed8de3f..d1a14e51df55e5 100644 --- a/drivers/iommu/iommufd/iommufd_private.h +++ b/drivers/iommu/iommufd/iommufd_private.h @@ -531,6 +531,10 @@ static inline bool iommufd_access_is_internal(struct iommufd_access *access) } struct iommufd_access *iommufd_access_create_internal(struct iommufd_ctx *ictx); +struct iommufd_access *iommufd_access_create_internal_phys(struct iommufd_ctx *ictx); +int iommufd_access_pin_phys(struct iommufd_access *access, unsigned long iova, + unsigned long length, phys_addr_t *out_phys); + static inline void iommufd_access_destroy_internal(struct iommufd_ctx *ictx, diff --git a/drivers/iommu/iommufd/iova_bitmap.c b/drivers/iommu/iommufd/iova_bitmap.c index dac3e657d498da..12e1dd035871a7 100644 --- a/drivers/iommu/iommufd/iova_bitmap.c +++ b/drivers/iommu/iommufd/iova_bitmap.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #define BITS_PER_PAGE (PAGE_SIZE * BITS_PER_BYTE) @@ -44,11 +45,16 @@ struct iova_bitmap_map { /* page offset of the first user page pinned */ unsigned long pgoff; - /* number of pages pinned */ + /* number of native pages or typed fragments pinned */ unsigned long npages; /* pinned pages representing the bitmap data */ struct page **pages; +#ifdef CONFIG_MM_SUBPAGE + /* Exact byte ranges, including offsets unrelated to the user VA. */ + struct user_page_fragment *fragments; + unsigned long pinned_bytes; +#endif }; /* @@ -63,7 +69,10 @@ struct iova_bitmap_map { * The bitmap object uses one base page to store all the pinned pages * pointers related to the bitmap. For sizeof(struct page*) == 8 it stores * 512 struct page pointers which, if the base page size is 4K, it means - * 2M of bitmap data is pinned at a time. If the iova_bitmap page size is + * 2M of bitmap data is pinned at a time. Alternative-granule processes use + * one native page of fragment descriptors instead, retaining each physical + * byte offset and bounding the window by the sum of fragment lengths. + * If the iova_bitmap page size is * also 4K then the range window to iterate is 64G. * * For example iterating on a total IOVA range of 4G..128G, it will walk @@ -116,6 +125,9 @@ struct iova_bitmap { /* length of the IOVA range for the whole bitmap */ size_t length; + + /* First lazy pin/write error during an iteration. */ + int error; }; /* @@ -192,6 +204,26 @@ static int iova_bitmap_get(struct iova_bitmap *bitmap) npages = min(npages + !!offset_in_page(addr), PAGE_SIZE / sizeof(struct page *)); +#ifdef CONFIG_MM_SUBPAGE + if (mapped->fragments) { + unsigned long bytes = bitmap->mapped_total_index - bitmap->mapped_base_index; + unsigned long i; + + ret = pin_user_fragments_remote(current->mm, (unsigned long)addr, + bytes, FOLL_WRITE, mapped->fragments, + PAGE_SIZE / sizeof(*mapped->fragments)); + if (ret <= 0) + return -EFAULT; + mapped->npages = ret; + mapped->pinned_bytes = 0; + for (i = 0; i < mapped->npages; i++) + mapped->pinned_bytes += mapped->fragments[i].length; + mapped->pgoff = 0; + mapped->iova = iova_bitmap_mapped_iova(bitmap); + mapped->length = iova_bitmap_mapped_length(bitmap); + return 0; + } +#endif ret = pin_user_pages_fast((unsigned long)addr, npages, FOLL_WRITE, mapped->pages); if (ret <= 0) @@ -221,7 +253,13 @@ static void iova_bitmap_put(struct iova_bitmap *bitmap) struct iova_bitmap_map *mapped = &bitmap->mapped; if (mapped->npages) { - unpin_user_pages(mapped->pages, mapped->npages); +#ifdef CONFIG_MM_SUBPAGE + if (mapped->fragments) { + release_user_fragments(mapped->fragments, mapped->npages, true); + mapped->pinned_bytes = 0; + } else +#endif + unpin_user_pages_dirty_lock(mapped->pages, mapped->npages, true); mapped->npages = 0; } } @@ -259,6 +297,16 @@ struct iova_bitmap *iova_bitmap_alloc(unsigned long iova, size_t length, bitmap->iova = iova; bitmap->length = length; mapped->iova = iova; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) { + mapped->fragments = (void *)__get_free_page(GFP_KERNEL); + if (!mapped->fragments) { + rc = -ENOMEM; + goto err; + } + return bitmap; + } +#endif mapped->pages = (struct page **)__get_free_page(GFP_KERNEL); if (!mapped->pages) { rc = -ENOMEM; @@ -291,6 +339,10 @@ void iova_bitmap_free(struct iova_bitmap *bitmap) mapped->pages = NULL; } +#ifdef CONFIG_MM_SUBPAGE + if (mapped->fragments) + free_page((unsigned long)mapped->fragments); +#endif kfree(bitmap); } EXPORT_SYMBOL_NS_GPL(iova_bitmap_free, "IOMMUFD"); @@ -303,7 +355,12 @@ static unsigned long iova_bitmap_mapped_remaining(struct iova_bitmap *bitmap) { unsigned long remaining, bytes; - bytes = (bitmap->mapped.npages << PAGE_SHIFT) - bitmap->mapped.pgoff; +#ifdef CONFIG_MM_SUBPAGE + if (bitmap->mapped.fragments) + bytes = bitmap->mapped.pinned_bytes; + else +#endif + bytes = (bitmap->mapped.npages << PAGE_SHIFT) - bitmap->mapped.pgoff; remaining = bitmap->mapped_total_index - bitmap->mapped_base_index; remaining = min_t(unsigned long, remaining, @@ -356,16 +413,21 @@ static int iova_bitmap_advance_to(struct iova_bitmap *bitmap, unsigned long iova) { unsigned long index; + int ret = -EINVAL; index = iova_bitmap_offset_to_index(bitmap, iova - bitmap->iova); if (index >= bitmap->mapped_total_index) - return -EINVAL; + goto out; bitmap->mapped_base_index = index; iova_bitmap_put(bitmap); /* Pin the next set of bitmap pages */ - return iova_bitmap_get(bitmap); + ret = iova_bitmap_get(bitmap); +out: + if (ret && !bitmap->error) + bitmap->error = ret; + return ret; } /** @@ -379,15 +441,58 @@ static int iova_bitmap_advance_to(struct iova_bitmap *bitmap, * mapped bitmap user pages into IOVA ranges to process. * * Return: 0 on success, and an error on failure either upon - * iteration or when the callback returns an error. + * iteration, a lazy bitmap write, or when the callback returns an error. */ int iova_bitmap_for_each(struct iova_bitmap *bitmap, void *opaque, iova_bitmap_fn_t fn) { - return fn(bitmap, bitmap->iova, bitmap->length, opaque); + int ret; + + bitmap->error = 0; + ret = fn(bitmap, bitmap->iova, bitmap->length, opaque); + return ret ?: bitmap->error; } EXPORT_SYMBOL_NS_GPL(iova_bitmap_for_each, "IOMMUFD"); +#ifdef CONFIG_MM_SUBPAGE +static void iova_bitmap_set_fragments(struct iova_bitmap *bitmap, + unsigned long iova, size_t length) +{ + struct iova_bitmap_map *mapped = &bitmap->mapped; + unsigned long bit = (iova - bitmap->iova) >> mapped->pgshift; + unsigned long end = ((iova + length - 1 - bitmap->iova) >> mapped->pgshift) + 1; + + while (bit < end) { + unsigned long base, relative, i; + + iova = bitmap->iova + (bit << mapped->pgshift); + if (!iova_bitmap_mapped_range(mapped, iova, 1) && + iova_bitmap_advance_to(bitmap, iova)) + return; + base = bitmap->mapped_base_index * BITS_PER_BYTE; + relative = bit - base; + for (i = 0; i < mapped->npages && bit < end; i++) { + struct user_page_fragment *fragment = &mapped->fragments[i]; + unsigned long bits = fragment->length * BITS_PER_BYTE; + unsigned long count; + void *addr; + + if (relative >= bits) { + relative -= bits; + continue; + } + count = min(end - bit, bits - relative); + addr = kmap_local_folio(fragment->folio, fragment->offset & PAGE_MASK); + bitmap_set(addr, offset_in_page(fragment->offset) * BITS_PER_BYTE + + relative, count); + kunmap_local(addr); + bit += count; + relative = 0; + } + } +} +#endif + /** * iova_bitmap_set() - Records an IOVA range in bitmap * @bitmap: IOVA bitmap @@ -395,7 +500,8 @@ EXPORT_SYMBOL_NS_GPL(iova_bitmap_for_each, "IOMMUFD"); * @length: IOVA range length * * Set the bits corresponding to the range [iova .. iova+length-1] in - * the user bitmap. + * the user bitmap. A lazy pin failure is retained for the enclosing + * iova_bitmap_for_each() to report; successful prefix writes are not undone. * */ void iova_bitmap_set(struct iova_bitmap *bitmap, @@ -404,6 +510,13 @@ void iova_bitmap_set(struct iova_bitmap *bitmap, struct iova_bitmap_map *mapped = &bitmap->mapped; unsigned long cur_bit, last_bit, last_page_idx; +#ifdef CONFIG_MM_SUBPAGE + if (mapped->fragments) { + iova_bitmap_set_fragments(bitmap, iova, length); + return; + } +#endif + update_indexes: if (unlikely(!iova_bitmap_mapped_range(mapped, iova, length))) { /* diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c index 404f31d8f72916..0839878bfb67c5 100644 --- a/drivers/iommu/iommufd/pages.c +++ b/drivers/iommu/iommufd/pages.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -207,7 +208,7 @@ static unsigned long iopt_area_index_to_iova(struct iopt_area *area, index -= iopt_area_index(area); if (index == 0) return iopt_area_iova(area); - return iopt_area_iova(area) - area->page_offset + index * PAGE_SIZE; + return iopt_area_iova(area) - area->page_offset + (index << area->page_shift); } static unsigned long iopt_area_index_to_iova_last(struct iopt_area *area, @@ -219,7 +220,7 @@ static unsigned long iopt_area_index_to_iova_last(struct iopt_area *area, if (index == iopt_area_last_index(area)) return iopt_area_last_iova(area); return iopt_area_iova(area) - area->page_offset + - (index - iopt_area_index(area) + 1) * PAGE_SIZE - 1; + ((index - iopt_area_index(area) + 1) << area->page_shift) - 1; } static void iommu_unmap_nofail(struct iommu_domain *domain, unsigned long iova, @@ -993,9 +994,11 @@ static int update_mm_locked_vm(struct iopt_pages *pages, unsigned long npages, } else if ((!user || (!user->upages && !user->ufolios)) && pages->source_mm != current->mm) { - if (!mmget_not_zero(pages->source_mm)) + if (mmget_not_zero(pages->source_mm)) + do_put = true; + else if (inc) return -EINVAL; - do_put = true; + /* mmgrab() keeps the lock and counters alive for final uncharge. */ } mmap_write_lock(pages->source_mm); @@ -1378,10 +1381,15 @@ static struct iopt_pages *iopt_alloc_pages(unsigned long start_byte, kref_init(&pages->kref); xa_init_flags(&pages->pinned_pfns, XA_FLAGS_ACCOUNT); +#ifdef CONFIG_MM_SUBPAGE + xa_init_flags(&pages->fragments, XA_FLAGS_ACCOUNT); + xa_init_flags(&pages->fragment_pages, XA_FLAGS_ACCOUNT); +#endif + pages->page_shift = PAGE_SHIFT; mutex_init(&pages->mutex); pages->source_mm = current->mm; mmgrab(pages->source_mm); - pages->npages = DIV_ROUND_UP(length + start_byte, PAGE_SIZE); + pages->npages = ((length - 1 + start_byte) >> PAGE_SHIFT) + 1; pages->access_itree = RB_ROOT_CACHED; pages->domains_itree = RB_ROOT_CACHED; pages->writable = writable; @@ -1400,8 +1408,10 @@ struct iopt_pages *iopt_alloc_user_pages(void __user *uptr, { struct iopt_pages *pages; unsigned long end; + unsigned int shift = min_t(unsigned int, mm_page_shift(current->mm), PAGE_SHIFT); + unsigned long granule = 1UL << shift; void __user *uptr_down = - (void __user *)ALIGN_DOWN((uintptr_t)uptr, PAGE_SIZE); + (void __user *)ALIGN_DOWN((uintptr_t)uptr, granule); if (check_add_overflow((unsigned long)uptr, length, &end)) return ERR_PTR(-EOVERFLOW); @@ -1409,6 +1419,8 @@ struct iopt_pages *iopt_alloc_user_pages(void __user *uptr, pages = iopt_alloc_pages(uptr - uptr_down, length, writable); if (IS_ERR(pages)) return pages; + pages->page_shift = shift; + pages->npages = ((length - 1 + (uptr - uptr_down)) >> shift) + 1; pages->uptr = uptr_down; pages->type = IOPT_ADDRESS_USER; return pages; @@ -1656,6 +1668,12 @@ void iopt_release_pages(struct kref *kref) WARN_ON(!RB_EMPTY_ROOT(&pages->domains_itree.rb_root)); WARN_ON(pages->npinned); WARN_ON(!xa_empty(&pages->pinned_pfns)); +#ifdef CONFIG_MM_SUBPAGE + WARN_ON(!xa_empty(&pages->fragments)); + WARN_ON(!xa_empty(&pages->fragment_pages)); + xa_destroy(&pages->fragments); + xa_destroy(&pages->fragment_pages); +#endif if (iopt_is_dmabuf(pages) && pages->dmabuf.attach) { struct dma_buf *dmabuf = pages->dmabuf.attach->dmabuf; @@ -1832,6 +1850,9 @@ void iopt_area_unmap_domain(struct iopt_area *area, struct iommu_domain *domain) void iopt_area_unfill_domain(struct iopt_area *area, struct iopt_pages *pages, struct iommu_domain *domain) { + if (iopt_pages_use_fragments(pages)) + return iopt_fragments_unfill_domain(area, pages, domain); + if (iopt_dmabuf_revoked(pages)) return; @@ -1855,6 +1876,9 @@ int iopt_area_fill_domain(struct iopt_area *area, struct iommu_domain *domain) lockdep_assert_held(&area->pages->mutex); + if (iopt_pages_use_fragments(area->pages)) + return iopt_fragments_fill_domain(area, area->pages, domain); + if (iopt_dmabuf_revoked(area->pages)) return 0; @@ -1916,6 +1940,9 @@ int iopt_area_fill_domains(struct iopt_area *area, struct iopt_pages *pages) if (xa_empty(&area->iopt->domains)) return 0; + if (iopt_pages_use_fragments(pages)) + return iopt_fragments_fill_domains(area, pages); + mutex_lock(&pages->mutex); if (iopt_is_dmabuf(pages)) { rc = iopt_dmabuf_track_all_domains(area, pages); @@ -2365,6 +2392,10 @@ int iopt_pages_rw_access(struct iopt_pages *pages, unsigned long start_byte, if (iopt_is_dmabuf(pages)) return -EINVAL; + if (iopt_pages_use_fragments(pages) && change_mm && + !(flags & IOMMUFD_ACCESS_RW_KTHREAD)) + return iopt_fragments_rw(pages, start_byte, data, length, flags); + if (pages->type != IOPT_ADDRESS_USER) return iopt_pages_rw_slow(pages, start_index, last_index, start_byte % PAGE_SIZE, data, length, @@ -2385,11 +2416,14 @@ int iopt_pages_rw_access(struct iopt_pages *pages, unsigned long start_byte, * ignore any pinning inconsistencies, unlike a real DMA path. */ if (change_mm) { - if (!mmget_not_zero(pages->source_mm)) + if (!mmget_not_zero(pages->source_mm)) { + if (iopt_pages_use_fragments(pages)) + return iopt_fragments_rw(pages, start_byte, data, length, flags); return iopt_pages_rw_slow(pages, start_index, last_index, start_byte % PAGE_SIZE, data, length, flags); + } kthread_use_mm(pages->source_mm); } @@ -2454,6 +2488,11 @@ int iopt_area_add_access(struct iopt_area *area, unsigned long start_index, if (iopt_is_dmabuf(pages)) return -EINVAL; + /* Preserve typed ownership and validate the native output geometry. */ + if (iopt_pages_use_fragments(pages)) + return iopt_fragments_add_access(area, start_index, last_index, + out_pages, lock_area); + mutex_lock(&pages->mutex); access = iopt_pages_get_exact_access(pages, start_index, last_index); if (access) { @@ -2494,6 +2533,47 @@ int iopt_area_add_access(struct iopt_area *area, unsigned long start_index, return rc; } +/* Internal hardware queues consume a contiguous physical byte range. */ +int iopt_area_add_phys_access(struct iopt_area *area, unsigned long start_byte, + unsigned long length, phys_addr_t *out_phys) +{ + struct iopt_pages *pages = area->pages; + unsigned long first = start_byte >> pages->page_shift; + unsigned long last = (start_byte + length - 1) >> pages->page_shift; + unsigned long count = last - first + 1, i; + struct page **list; + phys_addr_t base; + int rc; + + if (iopt_is_dmabuf(pages)) + return -EINVAL; + if (iopt_pages_use_fragments(pages)) + return iopt_fragments_add_phys_access(area, start_byte, length, out_phys); + + /* Retain the native page supplier and its accounting/batching rules. */ + list = kvcalloc(count, sizeof(*list), GFP_KERNEL_ACCOUNT); + if (!list) + return -ENOMEM; + rc = iopt_area_add_access(area, first, last, list, 0, true); + if (rc) + goto out_free; + base = PFN_PHYS(page_to_pfn(list[0])); + for (i = 1; i < count; i++) { + if (page_to_pfn(list[i]) != page_to_pfn(list[0]) + i) { + rc = -EFAULT; + goto out_unpin; + } + } + *out_phys = base + offset_in_page(start_byte); + goto out_free; + +out_unpin: + iopt_area_remove_access(area, first, last, true); +out_free: + kvfree(list); + return rc; +} + /** * iopt_area_remove_access() - Release an in-kernel access for PFNs * @area: The source of PFNs @@ -2526,7 +2606,10 @@ void iopt_area_remove_access(struct iopt_area *area, unsigned long start_index, goto out_unlock; interval_tree_remove(&access->node, &pages->access_itree); - iopt_pages_unfill_xarray(pages, start_index, last_index); + if (iopt_pages_use_fragments(pages)) + iopt_fragments_unfill_access(pages, start_index, last_index); + else + iopt_pages_unfill_xarray(pages, start_index, last_index); kfree(access); out_unlock: mutex_unlock(&pages->mutex); diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c index af07c642a5260e..146b1fefa0475f 100644 --- a/drivers/iommu/iommufd/selftest.c +++ b/drivers/iommu/iommufd/selftest.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -1201,6 +1202,28 @@ static int iommufd_test_md_check_pa(struct iommufd_ucmd *ucmd, unsigned long pfn; long npages; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) { + struct user_page_fragment fragment; + phys_addr_t user_phys; + + npages = get_user_fragments_remote(current->mm, + (uintptr_t)uptr, page_size, 0, &fragment, 1); + if (npages != 1) { + rc = npages < 0 ? npages : -EFAULT; + goto out_put; + } + user_phys = PFN_PHYS(folio_pfn(fragment.folio)) + fragment.offset; + io_phys = iommu_iova_to_phys(&mock->domain, iova); + rc = fragment.length == page_size && io_phys == user_phys ? 0 : -EINVAL; + release_user_fragments(&fragment, 1, false); + if (rc) + goto out_put; + iova += page_size; + uptr += page_size; + continue; + } +#endif npages = get_user_pages_fast((uintptr_t)uptr & PAGE_MASK, 1, 0, pages); if (npages < 0) { @@ -1506,9 +1529,39 @@ static int iommufd_test_access_replace_ioas(struct iommufd_ucmd *ucmd, /* Check that the pages in a page array match the pages in the user VA */ static int iommufd_test_check_pages(void __user *uptr, struct page **pages, - size_t npages) -{ - for (; npages; npages--) { + size_t length) +{ +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) { + unsigned long offset = 0; + + while (offset < length) { + struct user_page_fragment fragment; + unsigned long bytes = min_t(unsigned long, length - offset, + mm_user_fragment_size(current->mm) - + (((unsigned long)uptr + offset) & + (mm_user_fragment_size(current->mm) - 1))); + phys_addr_t actual, expected; + long rc; + + rc = get_user_fragments_remote(current->mm, + (unsigned long)uptr + offset, bytes, 0, &fragment, 1); + if (rc <= 0) + return rc ?: -EFAULT; + actual = PFN_PHYS(folio_pfn(fragment.folio)) + fragment.offset; + expected = PFN_PHYS(page_to_pfn(pages[offset / PAGE_SIZE])) + + offset_in_page(offset); + bytes = fragment.length; + release_user_fragments(&fragment, 1, false); + if (actual != expected || !bytes || + bytes > PAGE_SIZE - offset_in_page(offset)) + return -EBADE; + offset += bytes; + } + return 0; + } +#endif + for (size_t npages = DIV_ROUND_UP(length, PAGE_SIZE); npages; npages--) { struct page *tmp_pages[1]; long rc; @@ -1586,7 +1639,7 @@ static int iommufd_test_access_pages(struct iommufd_ucmd *ucmd, if (uptr) { rc = iommufd_test_check_pages( uptr - (iova - ALIGN_DOWN(iova, PAGE_SIZE)), pages, - npages); + length + offset_in_page(iova)); if (rc) goto out_unaccess; } diff --git a/drivers/iommu/iommufd/viommu.c b/drivers/iommu/iommufd/viommu.c index fc13cf4737ea3f..fa8296b15ef596 100644 --- a/drivers/iommu/iommufd/viommu.c +++ b/drivers/iommu/iommufd/viommu.c @@ -263,11 +263,7 @@ static void iommufd_hw_queue_destroy_access(struct iommufd_ctx *ictx, struct iommufd_access *access, u64 base_iova, size_t length) { - u64 aligned_iova = PAGE_ALIGN_DOWN(base_iova); - u64 offset = base_iova - aligned_iova; - - iommufd_access_unpin_pages(access, aligned_iova, - PAGE_ALIGN(length + offset)); + iommufd_access_unpin_pages(access, base_iova, length); iommufd_access_detach_internal(access); iommufd_access_destroy_internal(ictx, access); } @@ -300,66 +296,28 @@ static struct iommufd_access * iommufd_hw_queue_alloc_phys(struct iommu_hw_queue_alloc *cmd, struct iommufd_viommu *viommu, phys_addr_t *base_pa) { - u64 aligned_iova = PAGE_ALIGN_DOWN(cmd->nesting_parent_iova); - u64 offset = cmd->nesting_parent_iova - aligned_iova; struct iommufd_access *access; - struct page **pages; - size_t max_npages; - size_t length; - size_t i; int rc; - /* max_npages = DIV_ROUND_UP(offset + cmd->length, PAGE_SIZE) */ - if (check_add_overflow(offset, cmd->length, &length)) - return ERR_PTR(-ERANGE); - if (check_add_overflow(length, PAGE_SIZE - 1, &length)) + if (cmd->nesting_parent_iova > ULONG_MAX || cmd->length > ULONG_MAX) return ERR_PTR(-ERANGE); - max_npages = length / PAGE_SIZE; - /* length needs to be page aligned too */ - length = max_npages * PAGE_SIZE; - - /* - * Use kvcalloc() to avoid memory fragmentation for a large page array. - * Set __GFP_NOWARN to avoid syzkaller blowups - */ - pages = kvzalloc_objs(*pages, max_npages, GFP_KERNEL | __GFP_NOWARN); - if (!pages) - return ERR_PTR(-ENOMEM); - - access = iommufd_access_create_internal(viommu->ictx); - if (IS_ERR(access)) { - rc = PTR_ERR(access); - goto out_free; - } + access = iommufd_access_create_internal_phys(viommu->ictx); + if (IS_ERR(access)) + return access; rc = iommufd_access_attach_internal(access, viommu->hwpt->ioas); if (rc) goto out_destroy; - - rc = iommufd_access_pin_pages(access, aligned_iova, length, pages, 0); + rc = iommufd_access_pin_phys(access, cmd->nesting_parent_iova, + cmd->length, base_pa); if (rc) goto out_detach; - - /* Validate if the underlying physical pages are contiguous */ - for (i = 1; i < max_npages; i++) { - if (page_to_pfn(pages[i]) == page_to_pfn(pages[i - 1]) + 1) - continue; - rc = -EFAULT; - goto out_unpin; - } - - *base_pa = (page_to_pfn(pages[0]) << PAGE_SHIFT) + offset; - kvfree(pages); return access; -out_unpin: - iommufd_access_unpin_pages(access, aligned_iova, length); out_detach: iommufd_access_detach_internal(access); out_destroy: iommufd_access_destroy_internal(viommu->ictx, access); -out_free: - kvfree(pages); return ERR_PTR(rc); } diff --git a/drivers/media/platform/apple/Kconfig b/drivers/media/platform/apple/Kconfig index 77588c8f4e5a58..f656926099b3e3 100644 --- a/drivers/media/platform/apple/Kconfig +++ b/drivers/media/platform/apple/Kconfig @@ -4,3 +4,4 @@ comment "Apple media platform drivers" source "drivers/media/platform/apple/isp/Kconfig" source "drivers/media/platform/apple/avd/Kconfig" +source "drivers/media/platform/apple/ave/Kconfig" diff --git a/drivers/media/platform/apple/Makefile b/drivers/media/platform/apple/Makefile index 72f8bb54b376df..dac50fc70e277f 100644 --- a/drivers/media/platform/apple/Makefile +++ b/drivers/media/platform/apple/Makefile @@ -2,3 +2,4 @@ obj-y += isp/ obj-y += avd/ +obj-y += ave/ diff --git a/drivers/media/platform/apple/avd/avd-drv.c b/drivers/media/platform/apple/avd/avd-drv.c index 8f7a9d99322b5f..270ffb69965960 100644 --- a/drivers/media/platform/apple/avd/avd-drv.c +++ b/drivers/media/platform/apple/avd/avd-drv.c @@ -429,6 +429,17 @@ static void avd_device_run(void *priv) avd_job_finish(ctx, result); } +/* + * Capture buffers are found by timestamp and handed to the hardware by DMA + * address, so nothing limits them to VB2_MAX_FRAME. A larger pool lets + * userspace decode ahead to ride out bitrate spikes the hardware can't + * decode in real time (e.g. 150+ Mbit/s 4K60 H.264). + */ +static unsigned int max_capture_buffers = 128; +module_param(max_capture_buffers, uint, 0444); +MODULE_PARM_DESC(max_capture_buffers, + "Maximum number of capture buffers per context (default 128)"); + static int avd_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq) { @@ -466,6 +477,7 @@ static int avd_queue_init(void *priv, struct vb2_queue *src_vq, dst_vq->lock = &ctx->dev->vdev_lock; dst_vq->dev = ctx->dev->v4l2_dev.dev; dst_vq->allow_cache_hints = true; + dst_vq->max_num_buffers = max(max_capture_buffers, VB2_MAX_FRAME); return vb2_queue_init(dst_vq); } diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig new file mode 100644 index 00000000000000..629ce9244039ab --- /dev/null +++ b/drivers/media/platform/apple/ave/Kconfig @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config VIDEO_APPLE_AVE_EXPERIMENTAL + tristate "Apple T8140 experimental video encoder transport" + depends on ARCH_APPLE && ARM64_16K_PAGES + depends on PM_GENERIC_DOMAINS && IOMMU_DMA && DEBUG_FS + depends on !SUSPEND && !HIBERNATION && !KEXEC_CORE + select CRC32 + help + Experimental root-only AVE firmware and command transport for the + MacBook Neo. This does not register a V4L2 or FFmpeg encoder. + + It requires the bootloader's reserved, preloaded firmware and an + explicit enable=1 module parameter. After firmware startup, memory + and power references remain pinned until a hardware reset. Do not + enable this on a general-purpose kernel requiring suspend or kexec. diff --git a/drivers/media/platform/apple/ave/Makefile b/drivers/media/platform/apple/ave/Makefile new file mode 100644 index 00000000000000..c8e673e68f0807 --- /dev/null +++ b/drivers/media/platform/apple/ave/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only + +obj-$(CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL) += apple-ave.o diff --git a/drivers/media/platform/apple/ave/apple-ave-ipc.h b/drivers/media/platform/apple/ave/apple-ave-ipc.h new file mode 100644 index 00000000000000..d317af1dad72b8 --- /dev/null +++ b/drivers/media/platform/apple/ave/apple-ave-ipc.h @@ -0,0 +1,446 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Experimental 25G76/64-bit AVE mailbox and bidirectional ring transport. + * Included after private driver types. All callers hold ave->lock. + */ +#define AVE_CHANNEL_OFF 0x4000 +#define AVE_HANDSHAKE_OFF 0x24000 +#define AVE_SHARED_OFF 0x28000 +#define AVE_COMMAND_OFF 0x40000 +#define AVE_COMMAND_BYTES SZ_128K +#define AVE_ARENA_OFF 0x80000 +#define AVE_START_MAGIC 0x08042006 + +static int ave_fw_offset(struct apple_ave *ave, u64 address, size_t bytes, u32 *offset) +{ + u64 delta = address - ave->ipc_fw_base; + + if (address < ave->ipc_fw_base || delta > ave->ipc.size || bytes > ave->ipc.size - delta) + return -ERANGE; + *offset = delta; + return 0; +} + +static void ave_mailbox_send(struct apple_ave *ave, u32 a, u32 b, u32 c, u32 d) +{ + writel(a, ave->mailbox + 0x18); + writel(b, ave->mailbox + 0x1c); + writel(c, ave->mailbox + 0x20); + writel(d, ave->mailbox + 0x24); + wmb(); + writel(1, ave->mailbox + 0x0c); +} + +static int ave_mailbox_receive(struct apple_ave *ave) +{ + u32 status; + int ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); + + if (ret) + return ret; + /* Acknowledge the interrupt before reading the four scratch words. */ + writel(1, ave->mailbox + 0x10); + for (unsigned int i = 0; i < 4; i++) + ave->ipc_reply[i] = readl(ave->mailbox + 0x18 + i * 4); + dma_rmb(); + dev_info(ave->dev, "AVE_IPC_MAILBOX phase=%s words=%#x,%#x,%#x,%#x\n", + ave->phase, ave->ipc_reply[0], ave->ipc_reply[1], ave->ipc_reply[2], ave->ipc_reply[3]); + return 0; +} + +static int ave_create_channels(struct apple_ave *ave) +{ + bool found[2] = { false, false }; + + for (unsigned int i = 0; i < 2; i++) { + u8 *desc = ave->ipc.cpu + AVE_CHANNEL_OFF + i * 0x100; + u32 direction = get_unaligned_le32(desc + 0x40); + u32 interrupt = get_unaligned_le32(desc + 0x44); + u32 slots = get_unaligned_le32(desc + 0x48), offset; + u64 address = get_unaligned_le64(desc + 0x4c); + struct ave_channel *c; + unsigned int id; + + if (!memchr(desc, 0, 64)) + return -EPROTO; + if (!strcmp(desc, "IO")) + id = 0; + else if (!strcmp(desc, "IO_T2H")) + id = 1; + else + return -EPROTO; + if (found[id] || direction != id || !interrupt || interrupt >= 32 || + !slots || slots > ave->channel_bytes / 64 || !IS_ALIGNED(address, 64) || + ave_fw_offset(ave, address, (size_t)slots * 64, &offset) || + offset < AVE_CHANNEL_OFF + 0x200 || + offset + slots * 64 > AVE_CHANNEL_OFF + ave->channel_bytes) + return -EPROTO; + found[id] = true; + c = &ave->channel[id]; + strscpy(c->name, desc, sizeof(c->name)); + c->mode = direction ^ 1; + c->interrupt = interrupt; + c->slots = slots; + c->offset = offset; + c->consumer = c->mode ? -1 : 0; + c->producer = c->mode ? 0 : -1; + } + if (!found[0] || !found[1] || + (ave->channel[0].offset < ave->channel[1].offset + ave->channel[1].slots * 64 && + ave->channel[1].offset < ave->channel[0].offset + ave->channel[0].slots * 64)) + return -EPROTO; + /* Initialize only host-originated slots; leave all padding and + * firmware-originated slots unchanged. Ownership is bit zero of address. + */ + for (unsigned int i = 0; i < ave->channel[0].slots; i++) { + __le64 *slot = ave->ipc.cpu + ave->channel[0].offset + i * 64; + slot[0] = cpu_to_le64(1); + slot[1] = 0; + slot[2] = 0; + } + dma_wmb(); + return 0; +} + +static int ave_negotiate_ipc(struct apple_ave *ave) +{ + u64 ticks, time_delta, handshake_address, channel_address; + u32 frequency, status, offset; + u8 *handshake = ave->ipc.cpu + AVE_HANDSHAKE_OFF; + int ret; + + ave->phase = "validate-offer"; + if (ave->hello[0] != 2 || ave->hello[2] != 0x100 || + ave->hello[1] < 0x200 || ave->hello[1] > AVE_HANDSHAKE_OFF - AVE_CHANNEL_OFF || + !ave->hello[3] || ave->hello[3] > 4 * SZ_1M) + return -EPROTO; + ave->channel_bytes = ave->hello[1]; + ret = ave_alloc_buffer(ave, &ave->heap, PAGE_ALIGN(ave->hello[3])); + if (ret) + return ret; + writel(1, ave->mailbox + 0x10); + ticks = readq(ave->coproc + 0x378000); + frequency = readl(ave->coproc + 0x360020); + if (frequency && frequency < 1000000) + return -EPROTO; + ticks = div64_u64(ticks, frequency ? frequency / 1000000 : 24); + time_delta = div_u64(ktime_get_ns(), 1000) - ticks; + writel(lower_32_bits(time_delta), ave->mailbox + 0x28); + writel(upper_32_bits(time_delta), ave->mailbox + 0x2c); + ave->phase = "ipc-base"; + ave_mailbox_send(ave, lower_32_bits(ave->ipc.dma), upper_32_bits(ave->ipc.dma), ave->ipc.size, 0); + ret = ave_mailbox_receive(ave); + if (ret) + return ret; + ave->ipc_fw_base = (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0]; + if (!ave->ipc_fw_base || !IS_ALIGNED(ave->ipc_fw_base, PAGE_SIZE) || + ave->ipc_fw_base > U64_MAX - ave->ipc.size) + return -EPROTO; + memset(ave->ipc.cpu + AVE_CHANNEL_OFF, 0, ave->channel_bytes); + memset(handshake, 0, 0x50); + memset(ave->ipc.cpu + AVE_SHARED_OFF, 0, SZ_64K); + channel_address = ave->ipc_fw_base + AVE_CHANNEL_OFF; + handshake_address = ave->ipc_fw_base + AVE_HANDSHAKE_OFF; + put_unaligned_le64(channel_address, handshake + 8); + put_unaligned_le64(ave->ipc_fw_base + AVE_SHARED_OFF, handshake + 0x10); + put_unaligned_le32(SZ_64K, handshake + 0x18); + put_unaligned_le64(ave->heap.dma, handshake + 0x1c); + put_unaligned_le32(ave->hello[3], handshake + 0x24); + put_unaligned_le32(27, handshake + 0x28); + dma_wmb(); + ave->phase = "channel-offer"; + ave_mailbox_send(ave, lower_32_bits(handshake_address), upper_32_bits(handshake_address), 0, 0); + ret = ave_mailbox_receive(ave); + if (ret) + return ret; + ave->extra_bytes = ave->ipc_reply[2]; + if (ave_fw_offset(ave, (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0], + ave->channel_bytes, &offset) || offset != AVE_CHANNEL_OFF || + ave->extra_bytes > 0x13c000) + return -EPROTO; + ave->phase = "channel-layout"; + ret = ave_create_channels(ave); + if (ret) + return ret; + ave->phase = "ready-flag"; + writel(AVE_START_MAGIC, ave->mailbox + 0x24); + ret = readl_poll_timeout(ave->mailbox + 0x24, status, !status, 50, 5000000); + if (ret) + return ret; + ave->ipc_ready = true; + ave->phase = "ready"; + return 0; +} + +static int ave_ring_send(struct apple_ave *ave, struct ave_channel *c, u64 addr, u32 len, u32 flags) +{ + __le64 *slot; + int next; + + if (c->producer < 0) + return -ENOSPC; + slot = ave->ipc.cpu + c->offset + c->producer * 64; + WRITE_ONCE(slot[1], cpu_to_le64(len)); + WRITE_ONCE(slot[2], cpu_to_le64(flags)); + dma_wmb(); + WRITE_ONCE(slot[0], cpu_to_le64(addr | (c->mode ^ 1))); + dma_wmb(); + if (c->consumer < 0) + c->consumer = c->producer; + next = (c->producer + 1) % c->slots; + c->producer = next == c->consumer ? -1 : next; + writel(BIT(c->interrupt), ave->mailbox + 0x0c); + return 0; +} + +static int ave_ring_receive(struct apple_ave *ave, struct ave_channel *c, u64 *addr, u32 *len, u32 *flags) +{ + __le64 *slot; + u64 word; + int next; + + if (c->consumer < 0) + return -EAGAIN; + slot = ave->ipc.cpu + c->offset + c->consumer * 64; + word = le64_to_cpu(READ_ONCE(slot[0])); + if ((word & 1) != c->mode) + return -EAGAIN; + dma_rmb(); + *addr = word & ~3ULL; + *len = le64_to_cpu(READ_ONCE(slot[1])); + *flags = le64_to_cpu(READ_ONCE(slot[2])); + if (c->producer < 0) + c->producer = c->consumer; + next = (c->consumer + 1) % c->slots; + c->consumer = next == c->producer ? -1 : next; + return 0; +} + +/* Root-only experimental packet transport. The packet is copied into owned + * coherent memory; timeout pins the command too and forbids reuse this boot. + * Firmware command success is established by decoding replies, not this API. + */ +static ssize_t ave_command_write(struct file *file, const char __user *data, size_t len, loff_t *pos) +{ + struct apple_ave *ave = file->private_data; + u8 *cmd = ave->ipc.cpu + AVE_COMMAND_OFF; + u64 deadline, addr; + u32 bytes, flags, offset; + bool ack = false; + int ret = 0; + + if (!capable(CAP_SYS_RAWIO)) + return -EPERM; + static_assert(AVE_COMMAND_OFF + AVE_COMMAND_BYTES <= AVE_ARENA_OFF); + if (len < 0x48 || len > AVE_COMMAND_BYTES) + return -EINVAL; + mutex_lock(&ave->lock); + if (!ave->ipc_ready || ave->command_failed) { + ret = -EBUSY; + goto out; + } + if (copy_from_user(cmd, data, len)) { + ret = -EFAULT; + goto out; + } + /* The common command timeout is {duration_us, creation_time_us}. + * Stamp it in the same Linux monotonic clock synchronized at startup; + * captured macOS absolute times cannot be replayed on this boot. + */ + put_unaligned_le64(3000000, cmd + 0x30); + put_unaligned_le64(div_u64(ktime_get_ns(), 1000), cmd + 0x38); + ave->response_bytes = 0; + ret = ave_ring_send(ave, &ave->channel[0], ave->ipc_fw_base + AVE_COMMAND_OFF, len, 0); + if (ret) + goto out; + ave->commands++; + deadline = ktime_get_ns() + 5ULL * NSEC_PER_SEC; + do { + u32 pending = readl(ave->mailbox + 0x10); + if (pending) + writel(pending, ave->mailbox + 0x10); + for (unsigned int ch = 0; ch < 2; ch++) { + ret = ave_ring_receive(ave, &ave->channel[ch], &addr, &bytes, &flags); + if (ret == -EAGAIN) + continue; + /* Channel 1 returns the original address with zero payload + * bytes (verified native AVC captures). Only channel 2 is + * copied to response, independent of submitted packet size. + */ + if (ret || (!ch && bytes != 0) || + (ch && bytes > sizeof(ave->response) - 24) || + ave_fw_offset(ave, addr, bytes, &offset)) { + ret = -EPROTO; + goto failed; + } + dev_info(ave->dev, "AVE_COMMAND_REPLY channel=%u address=%#llx bytes=%u flags=%#x\n", + ch + 1, addr, bytes, flags); + if (!ch) { + if (addr != ave->ipc_fw_base + AVE_COMMAND_OFF) { + ret = -EPROTO; + goto failed; + } + ack = true; + ave->transport_acks++; + } else { + put_unaligned_le32(ch + 1, ave->response); + put_unaligned_le32(bytes, ave->response + 4); + put_unaligned_le32(flags, ave->response + 8); + put_unaligned_le32(0, ave->response + 12); + put_unaligned_le64(addr, ave->response + 16); + memcpy(ave->response + 24, ave->ipc.cpu + offset, bytes); + ave->response_bytes = bytes + 24; + ave->replies++; + /* Reply echoes use zero flags on the observed transport. */ + ret = ave_ring_send(ave, &ave->channel[ch], addr, bytes, 0); + if (ret) + goto failed; + } + } + if (ack && ave->response_bytes) { + ret = 0; + goto out; + } + usleep_range(100, 200); + } while (ktime_get_ns() < deadline); + ret = -ETIMEDOUT; +failed: + ave->command_failed = true; +out: + mutex_unlock(&ave->lock); + return ret ?: len; +} + +static ssize_t ave_command_read(struct file *file, char __user *data, size_t len, loff_t *pos) +{ + struct apple_ave *ave = file->private_data; + ssize_t ret; + + mutex_lock(&ave->lock); + ret = simple_read_from_buffer(data, len, pos, ave->response, ave->response_bytes); + mutex_unlock(&ave->lock); + return ret; +} + +static const struct file_operations ave_command_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = ave_command_write, + .read = ave_command_read, +}; + +/* Drain later firmware events without publishing another command. Results + * replace the response buffer and consist of one or more 24-byte-header + * records, each followed by its declared payload. Never discard silently. + */ +static ssize_t ave_poll_write(struct file *file, const char __user *data, size_t len, loff_t *pos) +{ + struct apple_ave *ave = file->private_data; + bool requested; + u64 deadline, addr; + u32 bytes, flags, offset; + int ret; + + if (!capable(CAP_SYS_RAWIO)) + return -EPERM; + ret = kstrtobool_from_user(data, len, &requested); + if (ret || !requested) + return ret ?: -EINVAL; + mutex_lock(&ave->lock); + if (!ave->ipc_ready || ave->command_failed) { + ret = -EBUSY; + goto out; + } + ave->response_bytes = 0; + deadline = ktime_get_ns() + NSEC_PER_SEC; + do { + u32 pending = readl(ave->mailbox + 0x10); + u8 *dst; + + if (pending) + writel(pending, ave->mailbox + 0x10); + ret = ave_ring_receive(ave, &ave->channel[1], &addr, &bytes, &flags); + if (ret == -EAGAIN) { + usleep_range(100, 200); + continue; + } + if (ret || ave_fw_offset(ave, addr, bytes, &offset) || + bytes > sizeof(ave->response) - 24 || + ave->response_bytes > sizeof(ave->response) - 24 - bytes) { + ret = -EPROTO; + goto failed; + } + dst = ave->response + ave->response_bytes; + put_unaligned_le32(2, dst); + put_unaligned_le32(bytes, dst + 4); + put_unaligned_le32(flags, dst + 8); + put_unaligned_le32(0, dst + 12); + put_unaligned_le64(addr, dst + 16); + memcpy(dst + 24, ave->ipc.cpu + offset, bytes); + ave->response_bytes += 24 + bytes; + ave->replies++; + ret = ave_ring_send(ave, &ave->channel[1], addr, bytes, 0); + if (ret) + goto failed; + } while (ktime_get_ns() < deadline); + ret = 0; + goto out; +failed: + ave->command_failed = true; +out: + mutex_unlock(&ave->lock); + return ret ?: len; +} + +static const struct file_operations ave_poll_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = ave_poll_write, +}; + +/* Bounded staging area for image/reference/output surfaces. Userspace offsets + * start at zero here, beyond all handshake, ring and command allocations. + * The status file supplies both DART and firmware addresses for this arena. + */ +static ssize_t ave_arena_read(struct file *file, char __user *data, size_t len, loff_t *pos) +{ + struct apple_ave *ave = file->private_data; + ssize_t ret; + + mutex_lock(&ave->lock); + dma_rmb(); + ret = simple_read_from_buffer(data, len, pos, ave->ipc.cpu + AVE_ARENA_OFF, + ave->ipc.size - AVE_ARENA_OFF); + mutex_unlock(&ave->lock); + return ret; +} + +static ssize_t ave_arena_write(struct file *file, const char __user *data, size_t len, loff_t *pos) +{ + struct apple_ave *ave = file->private_data; + ssize_t ret; + + if (!capable(CAP_SYS_RAWIO)) + return -EPERM; + mutex_lock(&ave->lock); + if (!ave->ipc_ready || ave->command_failed) + ret = -EBUSY; + else + ret = simple_write_to_buffer(ave->ipc.cpu + AVE_ARENA_OFF, + ave->ipc.size - AVE_ARENA_OFF, pos, data, len); + dma_wmb(); + mutex_unlock(&ave->lock); + return ret; +} + +static loff_t ave_arena_seek(struct file *file, loff_t offset, int whence) +{ + return fixed_size_llseek(file, offset, whence, AVE_IPC_BYTES - AVE_ARENA_OFF); +} + +static const struct file_operations ave_arena_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .read = ave_arena_read, + .write = ave_arena_write, + .llseek = ave_arena_seek, +}; diff --git a/drivers/media/platform/apple/ave/apple-ave.c b/drivers/media/platform/apple/ave/apple-ave.c new file mode 100644 index 00000000000000..d2cf7abadcadf2 --- /dev/null +++ b/drivers/media/platform/apple/ave/apple-ave.c @@ -0,0 +1,532 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* T8140 AVE bring-up: resources, firmware negotiation and IPC transport. + * Experimental root-only command interface. No registered encoder yet. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define AVE_IPC_BYTES (20 * SZ_1M) +#define AVE_QUEUE_BYTES SZ_64K +#define AVE_FW_IOVA 0x10000000000ULL +#define AVE_LOG_BYTES 0x24000 +#define AVE_LOG_USED 0x20200 + +static bool enable; +module_param(enable, bool, 0400); +MODULE_PARM_DESC(enable, "Opt in to the experimental AVE transport"); + +/* Read the known Uranus registers once after PM resume. The separate hello + * command pins all resources because reset/quiescence is not yet qualified. + */ +static bool snapshot; +module_param(snapshot, bool, 0400); +MODULE_PARM_DESC(snapshot, "Capture powered AVE registers once without writing them"); + +struct ave_firmware { + phys_addr_t physical; + u64 iova; + size_t size; + u32 crc; + unsigned int verified_pages; +}; + +struct ave_buffer { + void *cpu; + dma_addr_t dma; + size_t size; + unsigned int verified_pages; +}; + +struct ave_channel { + char name[64]; + u32 mode, interrupt, slots; + u32 offset; + int consumer, producer; +}; + +struct apple_ave { + struct device *dev; + struct device *power[5]; + unsigned int num_power; + struct iommu_domain *domain; + struct ave_buffer ipc; + struct ave_buffer queue; + struct ave_buffer log; + struct ave_buffer heap; + struct ave_firmware fw[2]; + bool firmware_verified; + bool start_requested; + int start_result; + struct mutex lock; + void __iomem *coproc, *mailbox; + u32 hello[4]; + u64 ipc_fw_base; + u32 ipc_reply[4], channel_bytes, extra_bytes; + const char *phase; + bool ipc_ready, command_failed; + struct ave_channel channel[2]; + u32 commands, replies, transport_acks; + u8 response[SZ_64K]; + size_t response_bytes; + struct debugfs_blob_wrapper ipc_blob; + struct debugfs_blob_wrapper log_blob; + struct debugfs_blob_wrapper config_blob; + u64 vcbar; + u32 core_control, core_status, mailbox_status, scratch[8]; + struct dentry *debugfs; +}; + +static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size); +#include "apple-ave-ipc.h" + +/* Cold-boot hello only. A successful response still needs the remaining IPC + * negotiation. Keep the module, PM reference and DMA buffers pinned until a + * target reboot, on both success and timeout; no unproven stop/free sequence. + */ +static ssize_t ave_start_write(struct file *file, const char __user *data, + size_t len, loff_t *ppos) +{ + struct apple_ave *ave = file->private_data; + __le32 *cfg = ave->queue.cpu, *log = ave->log.cpu; + bool requested; + u32 status; + int ret; + + if (!capable(CAP_SYS_RAWIO)) + return -EPERM; + ret = kstrtobool_from_user(data, len, &requested); + if (ret || !requested) + return ret ?: -EINVAL; + mutex_lock(&ave->lock); + if (ave->start_requested) { + ret = -EALREADY; + goto out; + } + if (!ave->firmware_verified || readq(ave->coproc + 0x50000) != (AVE_FW_IOVA | 1) || + readl(ave->coproc + 0x600044) || readl(ave->mailbox + 0x10)) { + ret = -EBUSY; + goto out; + } + for (unsigned int i = 0; i < 8; i++) { + if (readl(ave->mailbox + 0x18 + i * 4)) { + ret = -EBUSY; + goto out; + } + } + /* Startup values observed with the qualified 25G76 firmware: device + * enum 29, one active unit/group and sub-ID mask 1. Firmware maps the + * full page-aligned log allocation; the ring header advertises usage. + */ + memset(cfg, 0, 0x40); + cfg[1] = cpu_to_le32(29); + cfg[2] = cpu_to_le32(1); + cfg[3] = cpu_to_le32(1); + cfg[4] = cpu_to_le32(1); + cfg[8] = cpu_to_le32(lower_32_bits(ave->log.dma)); + cfg[9] = cpu_to_le32(upper_32_bits(ave->log.dma)); + cfg[10] = cpu_to_le32(AVE_LOG_BYTES); + cfg[12] = cpu_to_le32(0x77); + /* Log header: 128 KiB ring after its 512-byte header. */ + log[0] = cpu_to_le32(AVE_LOG_USED); + log[1] = cpu_to_le32(0x200); + log[2] = cpu_to_le32(0x20000); + dma_wmb(); + __module_get(THIS_MODULE); + ave->start_requested = true; + ave->phase = "hello"; + ave->start_result = -EINPROGRESS; + dev_info(ave->dev, "AVE_START_ONCE cfg=%pad log=%pad resources_pinned_until_reboot=1\n", + &ave->queue.dma, &ave->log.dma); + writel(0x08042006, ave->mailbox + 0x18); + writel(lower_32_bits(ave->queue.dma), ave->mailbox + 0x1c); + writel(upper_32_bits(ave->queue.dma), ave->mailbox + 0x20); + /* Preserve the protected firmware mappings inherited from iBoot. */ + writel(1, ave->coproc + 0x600808); + writel(0, ave->coproc + 0x600044); + writel(0x10000, ave->coproc + 0x600400); + writel(0x10, ave->coproc + 0x600044); + ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); + ave->start_result = ret; + ave->core_control = readl(ave->coproc + 0x600044); + ave->core_status = readl(ave->coproc + 0x600048); + ave->mailbox_status = status; + for (unsigned int i = 0; i < 8; i++) + ave->scratch[i] = readl(ave->mailbox + 0x18 + i * 4); + for (unsigned int i = 0; i < 4; i++) + ave->hello[i] = ave->scratch[i]; + dma_rmb(); + dev_info(ave->dev, "AVE_HELLO_RESULT result=%d status=%#x words=%#x,%#x,%#x,%#x core=%#x\n", + ret, status, ave->hello[0], ave->hello[1], ave->hello[2], ave->hello[3], ave->core_status); + if (!ret) + ret = ave_negotiate_ipc(ave); + ave->start_result = ret; + dev_info(ave->dev, "AVE_IPC_RESULT result=%d phase=%s ready=%u\n", + ret, ave->phase, ave->ipc_ready); +out: + mutex_unlock(&ave->lock); + return ret ?: len; +} + +static const struct file_operations ave_start_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = ave_start_write, +}; + +static int ave_verify_firmware(struct apple_ave *ave) +{ + static const char * const crc_names[] = { + "aurora,ave-text-crc32", "aurora,ave-data-crc32", + }; + static const size_t sizes[] = { 0x14c000, 0x130000 }; + struct device_node *chosen, *node; + unsigned int i; + int ret = 0; + + if (of_count_phandle_with_args(ave->dev->of_node, "memory-region", NULL) != 2) + return dev_err_probe(ave->dev, -EINVAL, "Expected two live AVE firmware reservations\n"); + chosen = of_find_node_by_path("/chosen"); + if (!chosen) + return -ENOENT; + for (i = 0; i < ARRAY_SIZE(ave->fw); i++) { + struct ave_firmware *fw = &ave->fw[i]; + struct resource res; + const char *crc_string; + u32 maps[5], expected_crc; + void *cpu; + size_t offset; + + node = of_parse_phandle(ave->dev->of_node, "memory-region", i); + ret = of_address_to_resource(node, 0, &res); + if (!ret) + ret = of_property_read_u32_array(node, "iommu-addresses", maps, ARRAY_SIZE(maps)); + if (!ret && (!of_property_read_bool(node, "no-map") || + !of_device_is_compatible(node, "apple,asc-mem"))) + ret = -EINVAL; + of_node_put(node); + if (ret) + break; + fw->physical = res.start; + fw->size = resource_size(&res); + fw->iova = (u64)maps[1] << 32 | maps[2]; + if (fw->size != sizes[i] || !IS_ALIGNED(fw->physical, PAGE_SIZE) || + maps[0] != ave->dev->of_node->phandle || maps[3] || maps[4] != sizes[i] || + fw->iova != AVE_FW_IOVA + (i ? sizes[0] : 0)) { + ret = -EINVAL; + break; + } + /* The IOMMU core owns these translated reservations across module + * unloads. DMA allocation must never reuse their IOVAs. + */ + for (offset = 0; offset < fw->size; offset += PAGE_SIZE) { + if (iommu_iova_to_phys(ave->domain, fw->iova + offset) != fw->physical + offset) { + dev_err(ave->dev, "Firmware %u mapping mismatch at %#zx\n", i, offset); + ret = -EIO; + break; + } + fw->verified_pages++; + } + if (ret) + break; + ret = of_property_read_string(chosen, crc_names[i], &crc_string); + if (!ret) + ret = kstrtou32(crc_string, 16, &expected_crc); + if (ret) + break; + cpu = memremap(fw->physical, fw->size, MEMREMAP_WB); + if (!cpu) { + ret = -ENOMEM; + break; + } + fw->crc = crc32_le(~0U, cpu, fw->size) ^ ~0U; + memunmap(cpu); + if (fw->crc != expected_crc) { + dev_err(ave->dev, "Firmware %u changed since the boot preflight\n", i); + ret = -EILSEQ; + break; + } + /* Only the two inspected 9003.78.0 text images are supported. + * CRCs detect build drift; they are not firmware authentication. + */ + if (!i && fw->crc != 0x2ce7359e && fw->crc != 0x69c54454) { + dev_err(ave->dev, "Unsupported AVE firmware text CRC %#x\n", fw->crc); + ret = -ENODEV; + break; + } + } + of_node_put(chosen); + if (!ret) + ave->firmware_verified = true; + return ret; +} + +/* Check the real IOMMU page table against the CPU mapping of each allocation. + * This proves mappings, not that firmware has performed a DMA transaction. + */ +static int ave_verify_mapping(struct apple_ave *ave, struct ave_buffer *buf) +{ + size_t offset; + + if (!IS_ALIGNED(buf->dma, PAGE_SIZE) || !IS_ALIGNED(buf->size, PAGE_SIZE)) + return -EINVAL; + for (offset = 0; offset < buf->size; offset += PAGE_SIZE) { + void *cpu = buf->cpu + offset; + struct page *page = is_vmalloc_addr(cpu) ? vmalloc_to_page(cpu) : virt_to_page(cpu); + phys_addr_t physical = iommu_iova_to_phys(ave->domain, buf->dma + offset); + + if (!page || physical != page_to_phys(page)) { + dev_err(ave->dev, "DMA mapping mismatch at offset %#zx\n", offset); + return -EIO; + } + buf->verified_pages++; + } + return 0; +} + +static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size) +{ + buf->size = size; + buf->cpu = dma_alloc_coherent(ave->dev, size, &buf->dma, GFP_KERNEL); + if (!buf->cpu) + return -ENOMEM; + memset(buf->cpu, 0, size); + return ave_verify_mapping(ave, buf); +} + +static void ave_free_buffer(struct apple_ave *ave, struct ave_buffer *buf) +{ + if (buf->cpu) + dma_free_coherent(ave->dev, buf->size, buf->cpu, buf->dma); + buf->cpu = NULL; +} + +static void ave_power_off(struct apple_ave *ave) +{ + while (ave->num_power) { + struct device *pd = ave->power[--ave->num_power]; + + pm_runtime_put_sync(pd); + dev_pm_domain_detach(pd, true); + } +} + +static int ave_power_on(struct apple_ave *ave) +{ + static const char * const names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; + int ret; + + for (unsigned int i = 0; i < ARRAY_SIZE(names); i++) { + struct device *pd = dev_pm_domain_attach_by_name(ave->dev, names[i]); + + if (IS_ERR(pd)) + return PTR_ERR(pd); + if (!pd) + return -ENODEV; + ret = pm_runtime_resume_and_get(pd); + if (ret < 0) { + dev_pm_domain_detach(pd, true); + return ret; + } + ave->power[ave->num_power++] = pd; + } + return 0; +} + +static int ave_status_show(struct seq_file *s, void *unused) +{ + struct apple_ave *ave = s->private; + + mutex_lock(&ave->lock); + seq_printf(s, "state=%s\nfirmware_start_requested=%u\nstart_result=%d\n", + ave->start_requested ? (ave->start_result ? "startup-failed" : "ipc-ready") : + "firmware-mapped", ave->start_requested, ave->start_result); + seq_printf(s, "hardware_dma_tested=%u\nencoder_registered=0\nphase=%s\n", + ave->ipc_ready, ave->phase ?: "not-started"); + seq_printf(s, "ipc_ready=%u\nipc_fw_base=%#llx\nchannel_bytes=%#x\nextra_bytes=%#x\n", + ave->ipc_ready, ave->ipc_fw_base, ave->channel_bytes, ave->extra_bytes); + seq_printf(s, "commands=%u\nreplies=%u\ntransport_acks=%u\ncommand_failed=%u\nresponse_bytes=%zu\n", + ave->commands, ave->replies, ave->transport_acks, ave->command_failed, ave->response_bytes); + seq_printf(s, "arena_bytes=%zu\narena_ipc_offset=%#x\narena_dma=%#llx\narena_fw=%#llx\n", + ave->ipc.size - AVE_ARENA_OFF, AVE_ARENA_OFF, + (u64)ave->ipc.dma + AVE_ARENA_OFF, ave->ipc_fw_base + AVE_ARENA_OFF); + for (unsigned int i = 0; i < 2; i++) { + struct ave_channel *c = &ave->channel[i]; + seq_printf(s, "channel%u_name=%s\nchannel%u_mode=%u\nchannel%u_interrupt=%u\n" + "channel%u_slots=%u\nchannel%u_offset=%#x\nchannel%u_consumer=%d\nchannel%u_producer=%d\n", + i, c->name, i, c->mode, i, c->interrupt, i, c->slots, + i, c->offset, i, c->consumer, i, c->producer); + } + seq_printf(s, "page_size=%lu\n", PAGE_SIZE); + seq_printf(s, "powered_domains=%u\n", ave->num_power); + seq_printf(s, "ipc_bytes=%zu\nipc_dma=%pad\nipc_verified_pages=%u\n", + ave->ipc.size, &ave->ipc.dma, ave->ipc.verified_pages); + seq_printf(s, "queue_bytes=%zu\nqueue_dma=%pad\nqueue_verified_pages=%u\n", + ave->queue.size, &ave->queue.dma, ave->queue.verified_pages); + seq_printf(s, "log_bytes=%zu\nlog_dma=%pad\nlog_verified_pages=%u\n", + ave->log.size, &ave->log.dma, ave->log.verified_pages); + seq_printf(s, "heap_bytes=%zu\nheap_dma=%pad\nheap_verified_pages=%u\n", + ave->heap.size, &ave->heap.dma, ave->heap.verified_pages); + for (unsigned int i = 0; i < ARRAY_SIZE(ave->fw); i++) + seq_printf(s, "fw%u_physical=%pa\nfw%u_iova=%#llx\nfw%u_bytes=%zu\n" + "fw%u_verified_pages=%u\nfw%u_crc32=%08x\n", + i, &ave->fw[i].physical, i, ave->fw[i].iova, i, ave->fw[i].size, + i, ave->fw[i].verified_pages, i, ave->fw[i].crc); + seq_printf(s, "register_snapshot=%u\n", snapshot); + if (snapshot) { + seq_printf(s, "vcbar=%#llx\ncore_control=%#x\ncore_status=%#x\nmailbox_status=%#x\n", + ave->vcbar, ave->core_control, ave->core_status, ave->mailbox_status); + for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) + seq_printf(s, "scratch%u=%#x\n", i, ave->scratch[i]); + } + seq_printf(s, "resources_pinned_until_reboot=%u\n", ave->start_requested); + mutex_unlock(&ave->lock); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(ave_status); + +static int apple_ave_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct apple_ave *ave; + void __iomem *regs, *mailbox, *coproc; + int ret; + + if (!enable) + return -ENODEV; + if (!of_machine_is_compatible("apple,j700") || PAGE_SIZE != SZ_16K) + return -ENODEV; + ave = devm_kzalloc(dev, sizeof(*ave), GFP_KERNEL); + if (!ave) + return -ENOMEM; + ave->dev = dev; + mutex_init(&ave->lock); + /* Claim only dedicated AVE ranges, never the shared PMGR/SRAM windows. */ + regs = devm_platform_ioremap_resource_byname(pdev, "codec"); + if (IS_ERR(regs)) + return PTR_ERR(regs); + mailbox = devm_platform_ioremap_resource_byname(pdev, "mailbox"); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); + if (IS_ERR(coproc)) + return PTR_ERR(coproc); + ave->coproc = coproc; + ave->mailbox = mailbox; + ave->domain = iommu_get_domain_for_dev(dev); + if (!ave->domain || !(ave->domain->type & __IOMMU_DOMAIN_PAGING)) + return dev_err_probe(dev, -ENODEV, "A translated IOMMU domain is required\n"); + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); + if (ret) + return ret; + ret = ave_power_on(ave); + if (ret) + goto release_power; + ret = devm_pm_runtime_enable(dev); + if (ret) + goto release_power; + ret = pm_runtime_resume_and_get(dev); + if (ret < 0) + goto release_power; + ret = ave_alloc_buffer(ave, &ave->ipc, AVE_IPC_BYTES); + if (ret) + goto free_buffers; + ret = ave_alloc_buffer(ave, &ave->queue, AVE_QUEUE_BYTES); + if (ret) + goto free_buffers; + ret = ave_alloc_buffer(ave, &ave->log, AVE_LOG_BYTES); + if (ret) + goto free_buffers; + ret = ave_verify_firmware(ave); + if (ret) + goto free_buffers; + if (snapshot) { + ave->vcbar = readq(coproc + 0x50000); + ave->core_control = readl(coproc + 0x600044); + ave->core_status = readl(coproc + 0x600048); + ave->mailbox_status = readl(mailbox + 0x10); + for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) + ave->scratch[i] = readl(mailbox + 0x18 + i * 4); + } + platform_set_drvdata(pdev, ave); + ave->debugfs = debugfs_create_dir(dev_name(dev), NULL); + debugfs_create_file("ave-status", 0400, ave->debugfs, ave, &ave_status_fops); + debugfs_create_file("ave-start-once", 0200, ave->debugfs, ave, &ave_start_fops); + debugfs_create_file("ave-command", 0600, ave->debugfs, ave, &ave_command_fops); + debugfs_create_file("ave-poll", 0200, ave->debugfs, ave, &ave_poll_fops); + debugfs_create_file("ave-arena", 0600, ave->debugfs, ave, &ave_arena_fops); + ave->ipc_blob.data = ave->ipc.cpu; + ave->ipc_blob.size = ave->ipc.size; + debugfs_create_blob("ave-ipc-memory", 0400, ave->debugfs, &ave->ipc_blob); + ave->log_blob.data = ave->log.cpu; + ave->log_blob.size = ave->log.size; + debugfs_create_blob("ave-firmware-log", 0400, ave->debugfs, &ave->log_blob); + ave->config_blob.data = ave->queue.cpu; + ave->config_blob.size = 0x40; + debugfs_create_blob("ave-boot-config", 0400, ave->debugfs, &ave->config_blob); + dev_info(dev, "AVE_DMA_READY ipc=%zu queue=%zu verified_pages=%u firmware_start=0\n", + ave->ipc.size, ave->queue.size, + ave->ipc.verified_pages + ave->queue.verified_pages); + dev_info(dev, "AVE_FIRMWARE_MAPPED pages=%u contents_match_boot=1 register_snapshot=%u\n", + ave->fw[0].verified_pages + ave->fw[1].verified_pages, snapshot); + return 0; + +free_buffers: + ave_free_buffer(ave, &ave->heap); + ave_free_buffer(ave, &ave->log); + ave_free_buffer(ave, &ave->queue); + ave_free_buffer(ave, &ave->ipc); + pm_runtime_put_sync(dev); +release_power: + ave_power_off(ave); + return dev_err_probe(dev, ret, "AVE DMA allocation/verification failed\n"); +} + +static void apple_ave_remove(struct platform_device *pdev) +{ + struct apple_ave *ave = platform_get_drvdata(pdev); + + debugfs_remove_recursive(ave->debugfs); + ave_free_buffer(ave, &ave->heap); + ave_free_buffer(ave, &ave->log); + ave_free_buffer(ave, &ave->queue); + ave_free_buffer(ave, &ave->ipc); + pm_runtime_put_sync(ave->dev); + ave_power_off(ave); + dev_info(ave->dev, "AVE_DMA_RELEASED\n"); +} + +static const struct of_device_id apple_ave_match[] = { + { .compatible = "apple,t8140-ave" }, + { } +}; +MODULE_DEVICE_TABLE(of, apple_ave_match); + +static struct platform_driver apple_ave_driver = { + .probe = apple_ave_probe, + .remove = apple_ave_remove, + .driver = { + .name = "apple-ave", + .of_match_table = apple_ave_match, + .suppress_bind_attrs = true, + }, +}; +module_platform_driver(apple_ave_driver); +MODULE_DESCRIPTION("Apple T8140 experimental video encoder transport"); +MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/isp/Kconfig b/drivers/media/platform/apple/isp/Kconfig index 942509978fab8c..2b026dba15f08c 100644 --- a/drivers/media/platform/apple/isp/Kconfig +++ b/drivers/media/platform/apple/isp/Kconfig @@ -27,3 +27,15 @@ config VIDEO_APPLE_ISP_KUNIT_TEST overflow, alignment and overlap with the channel table or other rings. These tests run without starting or accessing the ISP and do not require a supported Apple device or the ISP driver itself. + +config VIDEO_APPLE_ISP_V4L2_KUNIT_TEST + bool "KUnit tests for the Apple ISP driver" if !KUNIT_ALL_TESTS + depends on VIDEO_APPLE_ISP && KUNIT + depends on KUNIT=y || VIDEO_APPLE_ISP=m + default KUNIT_ALL_TESTS + help + Builds unit tests for the videobuf2 queue set-up of the Apple ISP + driver into the driver. They need no hardware. Only useful for + kernel developers running KUnit. + + If unsure, say N. diff --git a/drivers/media/platform/apple/isp/isp-drv.c b/drivers/media/platform/apple/isp/isp-drv.c index 9f5f8f58b3748e..d11ce3eff54ba0 100644 --- a/drivers/media/platform/apple/isp/isp-drv.c +++ b/drivers/media/platform/apple/isp/isp-drv.c @@ -312,9 +312,17 @@ static enum isp_firmware_version isp_read_fw_version(struct device *dev, const char *name) { u32 ver[ISP_FW_VERSION_MAX_LEN]; - int len = of_property_read_variable_u32_array(dev->of_node, name, ver, - ISP_FW_VERSION_MIN_LEN, - ISP_FW_VERSION_MAX_LEN); + int size, len; + + /* The array reader truncates trailing bytes; require complete cells. */ + if (of_find_property(dev->of_node, name, &size) && size % sizeof(u32)) { + dev_warn(dev, "could not parse %s: invalid length %d\n", name, size); + return ISP_FIRMWARE_V_UNKNOWN; + } + + len = of_property_read_variable_u32_array(dev->of_node, name, ver, + ISP_FW_VERSION_MIN_LEN, + ISP_FW_VERSION_MAX_LEN); switch (len) { case -EINVAL: @@ -354,6 +362,10 @@ isp_check_firmware_version(struct apple_isp *isp) struct device *dev = isp->dev; enum isp_firmware_version version, compat; + /* The H17 interface is selected by match data, not by version. */ + if (isp->hw->fw_abi == ISP_FW_ABI_H17) + return ISP_FIRMWARE_V_UNKNOWN; + /* firmware version is just informative */ version = isp_read_fw_version(dev, "apple,firmware-version"); compat = isp_read_fw_version(dev, "apple,firmware-compat"); @@ -361,9 +373,7 @@ isp_check_firmware_version(struct apple_isp *isp) dev_dbg(dev, "ISP firmware-compat: %s (FW: %s)\n", isp_fw2str(compat), isp_fw2str(version)); - /* The H17 interface does not depend on it. */ - if (isp->hw->fw_abi == ISP_FW_ABI_LEGACY && - !of_property_present(dev->of_node, "apple,firmware-compat")) + if (!of_property_present(dev->of_node, "apple,firmware-compat")) dev_warn(dev, "firmware compatibility version not provided, assuming 12.x\n"); return compat; @@ -380,7 +390,11 @@ static int apple_isp_probe(struct platform_device *pdev) if (err) return err; - isp = devm_kzalloc(dev, sizeof(*isp), GFP_KERNEL); + /* + * Not devres: an open video device file keeps this until it is + * closed, also after unbind. See apple_isp_release(). + */ + isp = kzalloc_obj(*isp); if (!isp) return -ENOMEM; @@ -399,7 +413,7 @@ static int apple_isp_probe(struct platform_device *pdev) if (err) { dev_err(dev, "failed to get 'apple,platform-id' property: %d\n", err); - return err; + goto free_isp; } err = of_property_read_u32(dev->of_node, "apple,temporal-filter", @@ -410,13 +424,13 @@ static int apple_isp_probe(struct platform_device *pdev) err = apple_isp_init_presets(isp); if (err) { dev_err(dev, "failed to initialize presets\n"); - return err; + goto free_isp; } err = apple_isp_attach_genpd(isp); if (err) { dev_err(dev, "failed to attatch power domains\n"); - return err; + goto free_isp; } isp->coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); @@ -529,6 +543,9 @@ static int apple_isp_probe(struct platform_device *pdev) destroy_workqueue(isp->wq); detach_genpd: apple_isp_detach_genpd(isp); +free_isp: + /* Nothing else can reach it before the video device is registered. */ + kfree(isp); return err; } @@ -545,6 +562,8 @@ static void apple_isp_remove(struct platform_device *pdev) apple_isp_free_iommu(isp); destroy_workqueue(isp->wq); apple_isp_detach_genpd(isp); + /* Freed now, or when the last open file is closed. */ + v4l2_device_put(&isp->v4l2_dev); } static const struct apple_isp_hw apple_isp_hw_t8103 = { diff --git a/drivers/media/platform/apple/isp/isp-fw.c b/drivers/media/platform/apple/isp/isp-fw.c index 7f6dc9d8cca2d7..6d8f4b91416b4c 100644 --- a/drivers/media/platform/apple/isp/isp-fw.c +++ b/drivers/media/platform/apple/isp/isp-fw.c @@ -96,7 +96,7 @@ static void apple_isp_power_down_domains(struct apple_isp *isp) ret = pm_runtime_put_sync(isp->pd_dev[i]); if (ret < 0) dev_err(isp->dev, - "Failed to power up power domain %d: %d\n", i, ret); + "Failed to power down power domain %d: %d\n", i, ret); } isp->pds_active = false; @@ -456,7 +456,11 @@ void apple_isp_free_firmware_surface(struct apple_isp *isp) static void isp_firmware_shutdown_stage2(struct apple_isp *isp) { + if (!isp->extra_surf) + return; + isp_free_surface(isp, isp->extra_surf); + isp->extra_surf = NULL; } static void isp_write_bootargs(struct apple_isp *isp, void *virt, @@ -499,13 +503,12 @@ static void isp_write_bootargs_h16(struct apple_isp *isp, void *virt, static int isp_firmware_boot_stage2(struct apple_isp *isp) { - bool h16 = isp->hw->fw_abi == ISP_FW_ABI_H17; - size_t args_size = h16 ? sizeof(struct isp_firmware_bootargs_h16) : + bool h17 = isp->hw->fw_abi == ISP_FW_ABI_H17; + size_t args_size = h17 ? sizeof(struct isp_firmware_bootargs_h16) : sizeof(struct isp_firmware_bootargs); size_t cmd_size = ISP_CMD_AREA_SIZE(isp_num_capmeta(isp)); dma_addr_t args_iova, cmd_iova; void *args_virt, *cmd_virt; - int err; u32 num_ipc_chans = isp_gpio_read32(isp, ISP_GPIO_0); u32 args_offset = isp_gpio_read32(isp, ISP_GPIO_1); @@ -524,7 +527,7 @@ static int isp_firmware_boot_stage2(struct apple_isp *isp) num_ipc_chans); /* Only hand the H16 descriptor to firmware that asks for it. */ - if (h16 && !(desc_flags & ISP_GPIO2_H16_DESCRIPTOR)) { + if (h17 && !(desc_flags & ISP_GPIO2_H16_DESCRIPTOR)) { dev_err(isp->dev, "firmware did not request the H16 boot descriptor (0x%x)\n", desc_flags); @@ -562,7 +565,7 @@ static int isp_firmware_boot_stage2(struct apple_isp *isp) isp->ipc_boot_size = args_size + 0x40 + cmd_size; isp->cmd_virt = cmd_virt; - if (h16) + if (h17) isp_write_bootargs_h16(isp, args_virt, args_offset); else isp_write_bootargs(isp, args_virt, args_iova); @@ -578,19 +581,18 @@ static int isp_firmware_boot_stage2(struct apple_isp *isp) /* Wait for ISP_GPIO_7 to 0xf7fbdff9 -> 0x8042006 */ isp_gpio_write32(isp, ISP_GPIO_7, 0xf7fbdff9); + /* + * The firmware may already use the extra heap: it is freed only once + * the coprocessor has stopped, see isp_firmware_boot(). + */ if (isp_gpio_wait(isp, ISP_GPIO_7, 0x8042006)) { isp_err(isp, "never received second magic number from firmware\n"); - err = -ENODEV; - goto free_extra; + return -ENODEV; } isp_dbg(isp, "got second magic number from firmware\n"); return 0; - -free_extra: - isp_free_surface(isp, isp->extra_surf); - return err; } static inline struct isp_channel *isp_get_chan_index(struct apple_isp *isp, @@ -893,32 +895,49 @@ static void isp_collect_gc_surface(struct apple_isp *isp) } } +/* + * Stop the coprocessor and power its domains down, then release the memory + * the firmware may still be using: its channel table, extra heap and the + * surfaces it requested. + */ +static void isp_firmware_stop(struct apple_isp *isp) +{ + isp_firmware_shutdown_stage1(isp); + + isp_firmware_shutdown_stage3(isp); + isp_firmware_shutdown_stage2(isp); + isp_collect_gc_surface(isp); +} + static int isp_firmware_boot(struct apple_isp *isp) { int err; err = isp_firmware_boot_stage1(isp); if (err < 0) { + /* Stage 1 stops the coprocessor and powers it down itself. */ isp_err(isp, "failed firmware boot stage 1: %d\n", err); - goto garbage_collect; + isp_collect_gc_surface(isp); + return err; } + /* From here on the firmware runs, and may use what it was given. */ err = isp_firmware_boot_stage2(isp); if (err < 0) { isp_err(isp, "failed firmware boot stage 2: %d\n", err); - goto shutdown_stage1; + goto stop; } err = isp_firmware_boot_stage3(isp); if (err < 0) { isp_err(isp, "failed firmware boot stage 3: %d\n", err); - goto shutdown_stage2; + goto stop; } err = isp_enable_irq(isp); if (err < 0) { isp_err(isp, "failed to enable interrupts: %d\n", err); - goto shutdown_stage3; + goto stop; } err = isp_start_command_processor(isp); @@ -933,14 +952,8 @@ static int isp_firmware_boot(struct apple_isp *isp) disable_irqs: isp_disable_irq(isp); -shutdown_stage3: - isp_firmware_shutdown_stage3(isp); -shutdown_stage2: - isp_firmware_shutdown_stage2(isp); -shutdown_stage1: - isp_firmware_shutdown_stage1(isp); -garbage_collect: - isp_collect_gc_surface(isp); +stop: + isp_firmware_stop(isp); return err; } @@ -955,11 +968,7 @@ static void isp_firmware_shutdown(struct apple_isp *isp) if (isp_stop_command_processor(isp)) dev_warn(isp->dev, "firmware did not suspend, stopping it\n"); isp_disable_irq(isp); - isp_firmware_shutdown_stage1(isp); - - isp_firmware_shutdown_stage3(isp); - isp_firmware_shutdown_stage2(isp); - isp_collect_gc_surface(isp); + isp_firmware_stop(isp); } /* diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c new file mode 100644 index 00000000000000..895a6d1299c220 --- /dev/null +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * KUnit tests for the videobuf2 queue of the Apple ISP driver, built into + * isp-v4l2.c. They check the queue parameters vb2 accepts and the buffer + * limit queue_setup() enforces, without any hardware. + */ + +#include +#include +#include + +#define ISP_TEST_FRAME_SIZE 0x1000 + +static struct apple_isp *isp_test_alloc(struct kunit *test) +{ + struct apple_isp *isp; + + isp = kunit_kzalloc(test, sizeof(*isp), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, isp); + + mutex_init(&isp->video_lock); + isp_get_current_format(isp)->total_size = ISP_TEST_FRAME_SIZE; + + return isp; +} + +/* The queue must initialise with the parameters the driver probes with. */ +static void isp_test_queue_init(struct kunit *test) +{ + struct apple_isp *isp = isp_test_alloc(test); + + isp->dev = kunit_device_register(test, "apple-isp-test"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, isp->dev); + + KUNIT_ASSERT_EQ(test, isp_vb2_queue_init(isp), 0); + KUNIT_EXPECT_GE(test, isp->vbq.max_num_buffers, VB2_MAX_FRAME); + vb2_queue_release(&isp->vbq); +} + +/* Pretend the queue already has @num buffers, as vb2_get_num_buffers() sees. */ +static void isp_test_set_num_buffers(struct kunit *test, struct vb2_queue *q, + unsigned int num) +{ + q->max_num_buffers = VB2_MAX_FRAME; + q->bufs_bitmap = kunit_kzalloc(test, + BITS_TO_LONGS(VB2_MAX_FRAME) * + sizeof(unsigned long), + GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, q->bufs_bitmap); + bitmap_set(q->bufs_bitmap, 0, num); +} + +static int isp_test_setup(struct kunit *test, unsigned int existing, + unsigned int *nbuffers, unsigned int *num_planes, + unsigned int sizes[]) +{ + struct apple_isp *isp = isp_test_alloc(test); + + isp->vbq.drv_priv = isp; + isp_test_set_num_buffers(test, &isp->vbq, existing); + + return isp_vb2_queue_setup(&isp->vbq, nbuffers, num_planes, sizes, + NULL); +} + +/* REQBUFS: an empty queue gets at most one pool of buffers. */ +static void isp_test_reqbufs_limit(struct kunit *test) +{ + unsigned int sizes[VB2_MAX_PLANES] = {}; + unsigned int nbuffers = VB2_MAX_FRAME, num_planes = 0; + + KUNIT_ASSERT_EQ(test, isp_test_setup(test, 0, &nbuffers, &num_planes, + sizes), 0); + KUNIT_EXPECT_EQ(test, nbuffers, ISP_MAX_BUFFERS); + KUNIT_EXPECT_EQ(test, num_planes, 1); + KUNIT_EXPECT_EQ(test, sizes[0], ISP_TEST_FRAME_SIZE); + + nbuffers = 4; + num_planes = 0; + KUNIT_ASSERT_EQ(test, isp_test_setup(test, 0, &nbuffers, &num_planes, + sizes), 0); + KUNIT_EXPECT_EQ(test, nbuffers, 4); +} + +/* CREATE_BUFS: the buffers the queue has count against the pool. */ +static void isp_test_create_bufs_limit(struct kunit *test) +{ + unsigned int sizes[VB2_MAX_PLANES] = { ISP_TEST_FRAME_SIZE }; + unsigned int nbuffers, num_planes; + + nbuffers = 10; + num_planes = 1; + KUNIT_ASSERT_EQ(test, isp_test_setup(test, 10, &nbuffers, &num_planes, + sizes), 0); + KUNIT_EXPECT_EQ(test, nbuffers, ISP_MAX_BUFFERS - 10); + + nbuffers = 1; + num_planes = 1; + KUNIT_EXPECT_EQ(test, isp_test_setup(test, ISP_MAX_BUFFERS, &nbuffers, + &num_planes, sizes), -ENOBUFS); + + /* A plane smaller than a frame is still refused. */ + nbuffers = 1; + num_planes = 1; + sizes[0] = ISP_TEST_FRAME_SIZE - 1; + KUNIT_EXPECT_EQ(test, isp_test_setup(test, 0, &nbuffers, &num_planes, + sizes), -EINVAL); +} + +static struct kunit_case isp_v4l2_test_cases[] = { + KUNIT_CASE(isp_test_queue_init), + KUNIT_CASE(isp_test_reqbufs_limit), + KUNIT_CASE(isp_test_create_bufs_limit), + {} +}; + +static struct kunit_suite isp_v4l2_test_suite = { + .name = "apple-isp-v4l2", + .test_cases = isp_v4l2_test_cases, +}; + +kunit_test_suite(isp_v4l2_test_suite); diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index 299177b9075cff..01cd82122201a0 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -292,13 +292,20 @@ static int isp_vb2_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers, { struct apple_isp *isp = vb2_get_drv_priv(vq); struct isp_format *fmt = isp_get_current_format(isp); + unsigned int q_num_bufs = vb2_get_num_buffers(vq); /* This is not strictly neccessary but makes it easy to enforce that * at most 16 buffers are submitted at once. ISP on t6001 (FW 12.3) * times out if more buffers are submitted than set in the buffer pool * config before streaming is started. + * + * REQBUFS frees the old buffers first, CREATE_BUFS adds to them, so + * count the buffers the queue already has. */ - *nbuffers = min_t(unsigned int, *nbuffers, ISP_MAX_BUFFERS); + if (q_num_bufs >= ISP_MAX_BUFFERS) + return -ENOBUFS; + *nbuffers = min_t(unsigned int, *nbuffers, + ISP_MAX_BUFFERS - q_num_bufs); if (*num_planes) { if (sizes[0] < fmt->total_size) @@ -757,6 +764,10 @@ static int isp_vidioc_set_format(struct file *file, void *fh, struct isp_preset *preset; int err; + /* The buffers and the queue type depend on the format. */ + if (vb2_is_busy(&isp->vbq)) + return -EBUSY; + preset = isp_select_preset(isp, f->fmt.pix.width, f->fmt.pix.height); err = isp_set_preset(isp, fmt, preset); if (err) @@ -812,6 +823,9 @@ static int isp_vidioc_set_format_mplane(struct file *file, void *fh, if (!isp->multiplanar) return -ENOTTY; + if (vb2_is_busy(&isp->vbq)) + return -EBUSY; + preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height); err = isp_set_preset(isp, fmt, preset); @@ -969,6 +983,22 @@ static const struct media_device_ops isp_media_device_ops = { .link_notify = v4l2_pipeline_link_notify, }; +/* + * The video device's file operations can run until its last open file is + * closed, after the driver has been unbound, so the state they reach is freed + * only with the last reference to the v4l2_device: the one remove drops, or + * the one the video device holds while it is open. + */ +static void apple_isp_release(struct v4l2_device *v4l2_dev) +{ + struct apple_isp *isp = container_of(v4l2_dev, struct apple_isp, + v4l2_dev); + + v4l2_ctrl_handler_free(&isp->ctrl_handler); + media_device_cleanup(&isp->mdev); + kfree(isp); +} + static void isp_free_meta_surfaces(struct apple_isp *isp) { for (int i = 0; i < ARRAY_SIZE(isp->capmeta_surfs); i++) { @@ -984,6 +1014,24 @@ static void isp_free_meta_surfaces(struct apple_isp *isp) } } +static int isp_vb2_queue_init(struct apple_isp *isp) +{ + struct vb2_queue *vbq = &isp->vbq; + + vbq->drv_priv = isp; + vbq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + vbq->io_modes = VB2_MMAP; + vbq->dev = isp->dev; + vbq->ops = &isp_vb2_ops; + vbq->mem_ops = &vb2_dma_sg_memops; + vbq->buf_struct_size = sizeof(struct isp_buffer); + vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC; + vbq->min_queued_buffers = ISP_MIN_FRAMES; + vbq->lock = &isp->video_lock; + + return vb2_queue_init(vbq); +} + int apple_isp_setup_video(struct apple_isp *isp) { struct video_device *vdev = &isp->vdev; @@ -1056,18 +1104,7 @@ int apple_isp_setup_video(struct apple_isp *isp) goto ctrl_free; } - vbq->drv_priv = isp; - vbq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE; - vbq->io_modes = VB2_MMAP; - vbq->dev = isp->dev; - vbq->ops = &isp_vb2_ops; - vbq->mem_ops = &vb2_dma_sg_memops; - vbq->buf_struct_size = sizeof(struct isp_buffer); - vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC; - vbq->min_queued_buffers = ISP_MIN_FRAMES; - vbq->lock = &isp->video_lock; - - err = vb2_queue_init(vbq); + err = isp_vb2_queue_init(isp); if (err) { dev_err(isp->dev, "failed to init vb2 queue: %d\n", err); goto v4l2_unregister; @@ -1076,7 +1113,7 @@ int apple_isp_setup_video(struct apple_isp *isp) vdev->queue = vbq; vdev->fops = &isp_v4l2_fops; vdev->ioctl_ops = &isp_v4l2_ioctl_ops; - vdev->device_caps = V4L2_BUF_TYPE_VIDEO_CAPTURE | V4L2_CAP_STREAMING; + vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING; if (isp->multiplanar) vdev->device_caps |= V4L2_CAP_VIDEO_CAPTURE_MPLANE; vdev->v4l2_dev = &isp->v4l2_dev; @@ -1093,6 +1130,9 @@ int apple_isp_setup_video(struct apple_isp *isp) goto v4l2_unregister; } + /* From here on, the last v4l2_device reference frees the state. */ + isp->v4l2_dev.release = apple_isp_release; + return 0; v4l2_unregister: @@ -1108,13 +1148,15 @@ int apple_isp_setup_video(struct apple_isp *isp) return err; } +/* + * Unregister the device nodes. An open file keeps the video device, and + * with it the driver state, until it is closed: see apple_isp_release(). + */ void apple_isp_remove_video(struct apple_isp *isp) { vb2_video_unregister_device(&isp->vdev); v4l2_device_unregister(&isp->v4l2_dev); - v4l2_ctrl_handler_free(&isp->ctrl_handler); media_device_unregister(&isp->mdev); - media_device_cleanup(&isp->mdev); } /* After apple_isp_remove_video() and with the firmware stopped */ @@ -1122,3 +1164,7 @@ void apple_isp_free_video(struct apple_isp *isp) { isp_free_meta_surfaces(isp); } + +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_V4L2_KUNIT_TEST) +#include "isp-v4l2-test.c" +#endif diff --git a/drivers/mux/apple-display-crossbar.c b/drivers/mux/apple-display-crossbar.c index 560ec53aee828c..d08b8abe6358ce 100644 --- a/drivers/mux/apple-display-crossbar.c +++ b/drivers/mux/apple-display-crossbar.c @@ -337,6 +337,8 @@ static int apple_dpxbar_set_t602x(struct mux_control *mux, int state) unsigned int dispext_bit_en; u32 atc_bit, mux_mask, mux_val; bool enable; + /* J700 physical DP retains the original T602x clock-gate sequence. */ + bool j700_physical = index == MUX_DPPHY && of_machine_is_compatible("apple,j700"); int ret = 0; if (index >= MUX_MAX) @@ -401,11 +403,14 @@ static int apple_dpxbar_set_t602x(struct mux_control *mux, int state) dpxbar_set32(dpxbar, T602X_REG_014, 0x4); /* Native T602x DPIN0 teardown restores the read reset at +0x24. */ - dpxbar_set32(dpxbar, T602X_FIFO_RD_PCLK2_EN, atc_bit); + dpxbar_set32(dpxbar, j700_physical ? FIFO_RD_PCLK1_EN : + T602X_FIFO_RD_PCLK2_EN, atc_bit); dpxbar_clear32(dpxbar, T602X_REG_034, atc_bit); - dpxbar_clear32(dpxbar, CROSSBAR_ATC_EN, atc_bit); - dpxbar_clear32(dpxbar, CROSSBAR_DISPEXT_EN, prev_dispext_bit); + if (!j700_physical) { + dpxbar_clear32(dpxbar, CROSSBAR_ATC_EN, atc_bit); + dpxbar_clear32(dpxbar, CROSSBAR_DISPEXT_EN, prev_dispext_bit); + } dpxbar_mask32(dpxbar, T602X_REG_030, mux_mask, 0); dpxbar->selected_dispext[index] = -1; @@ -450,13 +455,15 @@ static int apple_dpxbar_set_t602x(struct mux_control *mux, int state) index == MUX_DPIN1 ? FIFO_RD_PCLK1_EN : T602X_REG_01C, atc_bit); dpxbar_set32(dpxbar, T602X_REG_034, atc_bit); - dpxbar_set32(dpxbar, CROSSBAR_ATC_EN, atc_bit); + if (!j700_physical) + dpxbar_set32(dpxbar, CROSSBAR_ATC_EN, atc_bit); /* * t8103 enables the dispext source at 0x050. T602x left * that register at 0, so dpin0 was selected with no * source clock into the ACIO analog PHY. */ - dpxbar_set32(dpxbar, CROSSBAR_DISPEXT_EN, dispext_bit); + if (!j700_physical) + dpxbar_set32(dpxbar, CROSSBAR_DISPEXT_EN, dispext_bit); dpxbar_set32(dpxbar, T602X_FIFO_RD_UNK_EN, dispext_bit); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c index 9ff0c6829d1efb..d1b2aa550ee5a0 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c @@ -605,6 +605,267 @@ brcmf_cfg80211_update_proto_addr_mode(struct wireless_dev *wdev) * * Return: pointer to new vif on success, ERR_PTR(-errno) if not */ +static void brcmf_awdl_cleanup_locked(struct brcmf_cfg80211_info *cfg) +{ + struct brcmf_cfg80211_vif *vif; + struct brcmf_if *ifp; + + spin_lock(&cfg->vif_event.vif_event_lock); + ifp = cfg->awdl_failed_ifp; + vif = cfg->awdl_vif; + cfg->awdl_failed_ifp = NULL; + cfg->awdl_vif = NULL; + WRITE_ONCE(cfg->awdl_pending, false); + cfg->awdl_attaching = false; + if (cfg->vif_event.vif == vif) { + cfg->vif_event.vif = NULL; + cfg->vif_event.action = 0; + } + spin_unlock(&cfg->vif_event.vif_event_lock); + + if (ifp) { + brcmf_fil_iovar_data_set(ifp, "interface_remove", NULL, 0); + brcmf_proto_del_if(cfg->pub, ifp); + ifp->vif = NULL; + } + if (vif) + brcmf_free_vif(vif); + if (ifp) { + if (ifp->ndev) { + ifp->ndev->ieee80211_ptr = NULL; + free_netdev(ifp->ndev); + } else { + kfree(ifp); + } + } +} + +static void brcmf_awdl_cleanup(struct wiphy *wiphy, struct wiphy_work *work) +{ + struct brcmf_cfg80211_info *cfg = + container_of(work, struct brcmf_cfg80211_info, awdl_cleanup); + + brcmf_awdl_cleanup_locked(cfg); +} + +void brcmf_cfg80211_awdl_stop(struct brcmf_cfg80211_info *cfg) +{ + wiphy_lock(cfg->wiphy); + wiphy_delayed_work_cancel(cfg->wiphy, &cfg->awdl_timeout); + wiphy_work_cancel(cfg->wiphy, &cfg->awdl_cleanup); + brcmf_awdl_cleanup_locked(cfg); + wiphy_unlock(cfg->wiphy); +} + +static void brcmf_awdl_timeout(struct wiphy *wiphy, struct wiphy_work *work) +{ + struct brcmf_cfg80211_info *cfg = + container_of(work, struct brcmf_cfg80211_info, awdl_timeout.work); + struct brcmf_cfg80211_vif *vif; + + spin_lock(&cfg->vif_event.vif_event_lock); + vif = cfg->awdl_vif; + /* IF_ADD has already claimed the vif; its worker owns completion. */ + if (!cfg->awdl_pending || cfg->awdl_attaching || + (vif && vif->ifp)) { + spin_unlock(&cfg->vif_event.vif_event_lock); + return; + } + cfg->awdl_vif = NULL; + WRITE_ONCE(cfg->awdl_pending, false); + if (cfg->vif_event.vif == vif) { + cfg->vif_event.vif = NULL; + cfg->vif_event.action = 0; + } + spin_unlock(&cfg->vif_event.vif_event_lock); + + brcmf_err("AWDL interface creation timed out\n"); + if (vif) + brcmf_free_vif(vif); +} + +/** + * brcmf_awdl_add_vif() - create an AWDL interface and give it a netdev. + * + * @wiphy: wiphy device of the new interface. + * @name: name of the new interface. + * + * AWDL is Apple Wireless Direct Link, the link layer AirDrop and AirPlay run + * over. BCM4387 firmware implements it and accepts BRCMF_INTERFACE_TYPE_AWDL + * through interface_create, but there is no nl80211 interface type for it, so + * this cannot go through add_virtual_intf(). It is driven from the vendor + * command instead. + * + * The vif is allocated as NL80211_IFTYPE_STATION because cfg80211 needs some + * valid type for its bookkeeping; the firmware side is what actually decides + * the interface behaviour, from the iftype passed to interface_create. + * + * Return: 0 when requested, or a negative error code. + */ +int brcmf_awdl_add_vif(struct wiphy *wiphy, const char *name) +{ + struct brcmf_cfg80211_info *cfg = wiphy_to_cfg(wiphy); + struct brcmf_if *ifp = netdev_priv(cfg_to_ndev(cfg)); + struct brcmf_cfg80211_vif *vif; + int err; + + if (cfg->pub->bus_if->state != BRCMF_BUS_UP) + return -ENODEV; + if (brcmf_cfg80211_vif_event_armed(cfg) || + READ_ONCE(cfg->awdl_pending)) + return -EBUSY; + + vif = brcmf_alloc_vif(cfg, NL80211_IFTYPE_STATION); + if (IS_ERR(vif)) + return PTR_ERR(vif); + + /* Arm before asking firmware: the BRCMF_E_IF event is what links the + * brcmf_if to this vif (brcmf_notify_vif_event() bails with -EBADF if + * nothing is armed, leaving ndev->ieee80211_ptr unset). + */ + brcmf_cfg80211_arm_vif_event(cfg, vif); + + /* Pass NULL rather than a derived address. Supplying a MAC address + * at interface_create time kills the firmware control channel about + * two seconds after create, so the address is set afterwards via + * cur_etheraddr instead. + * + */ + strscpy(cfg->awdl_ifname, name, sizeof(cfg->awdl_ifname)); + spin_lock(&cfg->vif_event.vif_event_lock); + cfg->awdl_vif = vif; + WRITE_ONCE(cfg->awdl_pending, true); + spin_unlock(&cfg->vif_event.vif_event_lock); + wiphy_delayed_work_queue(wiphy, &cfg->awdl_timeout, + BRCMF_VIF_EVENT_TIMEOUT); + + err = brcmf_cfg80211_request_awdl_if(ifp, NULL); + if (err) { + /* An IF_ADD may already be in flight: retire under the same lock. */ + wiphy_delayed_work_queue(wiphy, &cfg->awdl_timeout, 0); + return err; + } + + /* Deliberately do NOT wait for the interface here, and do NOT attach + * the netdev. nl80211 calls vendor commands holding the wiphy mutex + * (NL80211_CMD_VENDOR sets NEED_WIPHY without NO_WIPHY_MTX), and + * register_netdev() re-enters that mutex via the netdev notifier + * chain -> cfg80211_netdev_notifier_call(). Doing it here self- + * deadlocks and wedges the RTNL for the whole system. The attach is + * done from brcmf_cfg80211_awdl_attach_pending(), which the fweh + * event worker calls while holding no locks at all. + */ + return 0; +} + +/** + * brcmf_cfg80211_awdl_attach_pending() - finish AWDL interface creation. + * + * @ifp: interface the firmware just reported via BRCMF_E_IF_ADD. + * + * Called from the fweh event worker (brcmf_fweh_handle_if_event()), which + * runs on the system workqueue holding neither the RTNL nor the wiphy mutex. + * That is the only context in which register_netdev() is safe for an + * interface created outside cfg80211's own add_virtual_intf() path. + */ +void brcmf_cfg80211_awdl_attach_pending(struct brcmf_if *ifp) +{ + struct brcmf_cfg80211_info *cfg = ifp->drvr->config; + struct brcmf_cfg80211_vif *vif; + int err; + + if (!cfg) + return; + spin_lock(&cfg->vif_event.vif_event_lock); + vif = cfg->awdl_vif; + if (!cfg->awdl_pending || cfg->awdl_attaching || !vif || + ifp->vif != vif) { + spin_unlock(&cfg->vif_event.vif_event_lock); + return; + } + cfg->awdl_attaching = true; + spin_unlock(&cfg->vif_event.vif_event_lock); + + if (!ifp->ndev) { + bphy_err(ifp->drvr, "AWDL if event without netdev\n"); + goto fail; + } + + ifp->is_awdl = true; + strscpy(ifp->ndev->name, cfg->awdl_ifname, sizeof(ifp->ndev->name)); + err = brcmf_net_attach(ifp, false); + if (err) { + bphy_err(ifp->drvr, "registering AWDL netdevice failed\n"); + goto fail; + } + + spin_lock(&cfg->vif_event.vif_event_lock); + cfg->awdl_vif = NULL; + WRITE_ONCE(cfg->awdl_pending, false); + cfg->awdl_attaching = false; + if (cfg->vif_event.vif == vif) { + cfg->vif_event.vif = NULL; + cfg->vif_event.action = 0; + } + spin_unlock(&cfg->vif_event.vif_event_lock); + brcmf_info("AWDL interface %s created (bsscfgidx %d)\n", + ifp->ndev->name, ifp->bsscfgidx); + return; + +fail: + /* Do not free a vif while an event or wiphy operation can still see it. */ + spin_lock(&cfg->vif_event.vif_event_lock); + if (cfg->pub->iflist[ifp->bsscfgidx] == ifp) + cfg->pub->iflist[ifp->bsscfgidx] = NULL; + cfg->awdl_failed_ifp = ifp; + cfg->vif_event.vif = NULL; + cfg->vif_event.action = 0; + spin_unlock(&cfg->vif_event.vif_event_lock); + wiphy_work_queue(cfg->wiphy, &cfg->awdl_cleanup); +} + +/** + * brcmf_awdl_del_vif() - tear down an interface made by brcmf_awdl_add_vif(). + * + * @wiphy: wiphy device of the interface. + * @wdev: wireless device to remove. + * + * Return: 0 on success, negative errno on failure. + */ +int brcmf_awdl_del_vif(struct wiphy *wiphy, struct wireless_dev *wdev) +{ + struct brcmf_cfg80211_vif *vif = container_of(wdev, + struct brcmf_cfg80211_vif, + wdev); + struct brcmf_if *ifp = vif->ifp; + int err; + + if (wdev->wiphy != wiphy || !ifp || !ifp->is_awdl) + return -ENODEV; + + /* Asymmetric with the rest of brcmfmac on purpose, for the same reason + * brcmf_awdl_add_vif() does not call brcmf_net_attach(): nl80211 calls + * vendor commands holding the wiphy mutex and NOT the RTNL + * (NL80211_CMD_VENDOR sets NEED_WIPHY without NO_WIPHY_MTX). Tearing + * the netdev down here reaches cfg80211_close_dependents(), which + * asserts the RTNL, and then deadlocks the whole netdev layer against + * any concurrent RTNL holder. Do NOT call brcmf_remove_interface() or + * arm the vif event from this context. + * + * Just ask the firmware to drop the BSS and return. The BRCMF_E_IF_DEL + * event lands in brcmf_fweh_event_worker(), which is a plain work item + * holding neither lock, and brcmf_fweh_handle_if_event() removes the + * interface from there. + */ + err = brcmf_fil_iovar_data_set(ifp, "interface_remove", NULL, 0); + if (err) + err = brcmf_fil_bsscfg_data_set(ifp, "interface_remove", NULL, 0); + if (err) + bphy_err(ifp->drvr, "interface_remove failed %d\n", err); + + return err; +} + static struct wireless_dev *brcmf_apsta_add_vif(struct wiphy *wiphy, const char *name, struct vif_params *params, @@ -8664,6 +8925,8 @@ struct brcmf_cfg80211_info *brcmf_cfg80211_attach(struct brcmf_pub *drvr, cfg->wiphy = wiphy; cfg->pub = drvr; init_vif_event(&cfg->vif_event); + wiphy_delayed_work_init(&cfg->awdl_timeout, brcmf_awdl_timeout); + wiphy_work_init(&cfg->awdl_cleanup, brcmf_awdl_cleanup); INIT_LIST_HEAD(&cfg->vif_list); cfg->force_band_setup = true; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.h index e15e5b689df956..d548ca995e319d 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.h @@ -396,6 +396,14 @@ struct brcmf_cfg80211_info { struct work_struct escan_timeout_work; struct list_head vif_list; struct brcmf_cfg80211_vif_event vif_event; + /* Protected by vif_event.vif_event_lock. */ + bool awdl_pending; + bool awdl_attaching; + struct brcmf_cfg80211_vif *awdl_vif; + struct brcmf_if *awdl_failed_ifp; + struct wiphy_delayed_work awdl_timeout; + struct wiphy_work awdl_cleanup; + char awdl_ifname[IFNAMSIZ]; struct completion vif_disabled; struct brcmu_d11inf d11inf; struct brcmf_assoclist_le assoclist; @@ -507,6 +515,10 @@ bool brcmf_get_vif_state_any(struct brcmf_cfg80211_info *cfg, void brcmf_cfg80211_arm_vif_event(struct brcmf_cfg80211_info *cfg, struct brcmf_cfg80211_vif *vif); bool brcmf_cfg80211_vif_event_armed(struct brcmf_cfg80211_info *cfg); +void brcmf_cfg80211_awdl_stop(struct brcmf_cfg80211_info *cfg); +int brcmf_awdl_add_vif(struct wiphy *wiphy, const char *name); +void brcmf_cfg80211_awdl_attach_pending(struct brcmf_if *ifp); +int brcmf_awdl_del_vif(struct wiphy *wiphy, struct wireless_dev *wdev); int brcmf_cfg80211_wait_vif_event(struct brcmf_cfg80211_info *cfg, u8 action, ulong timeout); s32 brcmf_notify_escan_complete(struct brcmf_cfg80211_info *cfg, diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c index e3a9ca5526c459..2c99a990315682 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c @@ -43,6 +43,15 @@ int brcmf_msg_level; module_param_named(debug, brcmf_msg_level, int, 0600); MODULE_PARM_DESC(debug, "Level of debug output"); +/* AWDL tx completions are logged per frame. Userspace counts them to tell a + * radio that is transmitting from one whose firmware has parked the AWDL data + * path: a parked radio completes nothing, and it looks healthy by every other + * measure. Off by default, writable at runtime so no reload is needed. + */ +int brcmf_awdl_trace; +module_param_named(awdl_trace, brcmf_awdl_trace, int, 0644); +MODULE_PARM_DESC(awdl_trace, "Log AWDL tx completions"); + static int brcmf_p2p_enable; module_param_named(p2pon, brcmf_p2p_enable, int, 0); MODULE_PARM_DESC(p2pon, "Enable legacy p2p management functionality"); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c index 3c4125c3363552..a4460f06c1301c 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -38,6 +39,17 @@ #define D11_PHY_HDR_LEN 6 +/* AWDL data frames on the air are 802.3 + LLC/SNAP (OUI 00:17:f2, protocol + * 0x0800) + a 4-byte data header, then the real ethertype. The kernel only + * speaks Ethernet II, so the driver translates at the awdl0 boundary in + * both directions. + */ +#define BRCMF_AWDL_ENCAP_LEN 16 +#define BRCMF_AWDL_DATA_MIN_LEN (ETH_HLEN + BRCMF_AWDL_ENCAP_LEN) + +static const u8 brcmf_awdl_snap[] = { 0xaa, 0xaa, 0x03, 0x00, 0x17, 0xf2, + 0x08, 0x00, 0x03, 0x04 }; + struct d11rxhdr_le { __le16 RxFrameSize; u16 PAD; @@ -290,6 +302,58 @@ static bool brcmf_skb_is_iapp(struct sk_buff *skb) #endif } +/* Unwrap SNAP + AWDL data header into Ethernet II, in place. skb->data is + * the 802.3 header. Frames that are not AWDL-encapsulated pass unchanged. + */ +static void brcmf_awdl_decap(struct sk_buff *skb) +{ + const struct ethhdr *eh = (const struct ethhdr *)skb->data; + + if (skb->len < BRCMF_AWDL_DATA_MIN_LEN || + ntohs(eh->h_proto) > ETH_DATA_LEN || + memcmp(skb->data + ETH_HLEN, brcmf_awdl_snap, + sizeof(brcmf_awdl_snap))) + return; + + memmove(skb->data + BRCMF_AWDL_ENCAP_LEN, skb->data, 2 * ETH_ALEN); + skb_pull(skb, BRCMF_AWDL_ENCAP_LEN); +} + +/* Every received data frame passes through here, on the BCDC and the msgbuf + * (PCIe) paths alike, before the kernel classifies it. + */ +__be16 brcmf_rx_eth_type_trans(struct brcmf_if *ifp, struct sk_buff *skb) +{ + if (ifp->is_awdl) + brcmf_awdl_decap(skb); + return eth_type_trans(skb, ifp->ndev); +} + +/* Wrap Ethernet II into 802.3 + SNAP + AWDL data header, in place. Frames + * already carrying a length field (raw AF_PACKET senders that build the + * encapsulation themselves) pass unchanged. Caller guarantees headroom. + */ +static int brcmf_awdl_encap(struct brcmf_if *ifp, struct sk_buff *skb) +{ + const struct ethhdr *eh = (const struct ethhdr *)skb->data; + u8 *hdr; + + if (ntohs(eh->h_proto) < ETH_P_802_3_MIN) + return 0; + if (skb->len - ETH_HLEN + BRCMF_AWDL_ENCAP_LEN > ETH_DATA_LEN) + return -EMSGSIZE; + + hdr = skb_push(skb, BRCMF_AWDL_ENCAP_LEN); + memmove(hdr, hdr + BRCMF_AWDL_ENCAP_LEN, 2 * ETH_ALEN); + put_unaligned_be16(skb->len - ETH_HLEN, hdr + 2 * ETH_ALEN); + memcpy(hdr + ETH_HLEN, brcmf_awdl_snap, sizeof(brcmf_awdl_snap)); + put_unaligned_le16(ifp->awdl_tx_seq++, + hdr + ETH_HLEN + sizeof(brcmf_awdl_snap)); + hdr[ETH_HLEN + sizeof(brcmf_awdl_snap) + 2] = 0; + hdr[ETH_HLEN + sizeof(brcmf_awdl_snap) + 3] = 0; + return 0; +} + static netdev_tx_t brcmf_netdev_start_xmit(struct sk_buff *skb, struct net_device *ndev) { @@ -298,6 +362,7 @@ static netdev_tx_t brcmf_netdev_start_xmit(struct sk_buff *skb, struct brcmf_pub *drvr = ifp->drvr; struct ethhdr *eh; int head_delta; + unsigned int hdrlen = drvr->hdrlen; unsigned int tx_bytes = skb->len; brcmf_dbg(DATA, "Enter, bsscfgidx=%d\n", ifp->bsscfgidx); @@ -328,9 +393,12 @@ static netdev_tx_t brcmf_netdev_start_xmit(struct sk_buff *skb, goto done; } + if (ifp->is_awdl) + hdrlen += BRCMF_AWDL_ENCAP_LEN; + /* Make sure there's enough writeable headroom */ - if (skb_headroom(skb) < drvr->hdrlen || skb_header_cloned(skb)) { - head_delta = max_t(int, drvr->hdrlen - skb_headroom(skb), 0); + if (skb_headroom(skb) < hdrlen || skb_header_cloned(skb)) { + head_delta = max_t(int, hdrlen - skb_headroom(skb), 0); brcmf_dbg(INFO, "%s: %s headroom\n", brcmf_ifname(ifp), head_delta ? "insufficient" : "unmodifiable"); @@ -365,6 +433,14 @@ static netdev_tx_t brcmf_netdev_start_xmit(struct sk_buff *skb, /* set pacing shift for packet aggregation */ sk_pacing_shift_update(skb->sk, 8); + if (ifp->is_awdl) { + ret = brcmf_awdl_encap(ifp, skb); + if (ret) { + brcmf_txfinalize(ifp, skb, false); + goto done; + } + } + ret = brcmf_proto_tx_queue_data(drvr, ifp->ifidx, skb); if (ret < 0) brcmf_txfinalize(ifp, skb, false); @@ -495,7 +571,7 @@ static int brcmf_rx_hdrpull(struct brcmf_pub *drvr, struct sk_buff *skb, return -ENODATA; } - skb->protocol = eth_type_trans(skb, (*ifp)->ndev); + skb->protocol = brcmf_rx_eth_type_trans(*ifp, skb); return 0; } @@ -632,6 +708,55 @@ static int brcmf_netdev_open(struct net_device *ndev) return 0; } +/* AWDL interfaces are not STAs. brcmf_netdev_open() runs the standard station + * bring-up -- a toe_ol query, brcmf_cfg80211_up(), then the multicast and + * promisc configuration driven by ndo_set_rx_mode -- and issuing those against + * an AWDL bsscfg stops the firmware answering dcmds at all: every subsequent + * command times out and only a brcmfmac reload recovers it. Bring the netdev + * up without touching firmware, and deliberately omit ndo_set_rx_mode so the + * multicast worker never runs for these interfaces. + */ +static int brcmf_netdev_open_awdl(struct net_device *ndev) +{ + struct brcmf_if *ifp = netdev_priv(ndev); + struct brcmf_pub *drvr = ifp->drvr; + struct brcmf_bus *bus_if = drvr->bus_if; + + brcmf_dbg(TRACE, "Enter, bsscfgidx=%d (AWDL)\n", ifp->bsscfgidx); + + if (bus_if->state != BRCMF_BUS_UP) { + bphy_err(drvr, "failed bus is not ready\n"); + return -EAGAIN; + } + + atomic_set(&ifp->pend_8021x_cnt, 0); + + /* Carrier stays off until AWDL is actually running. */ + netif_carrier_off(ndev); + return 0; +} + +static int brcmf_netdev_stop_awdl(struct net_device *ndev) +{ + struct brcmf_if *ifp = netdev_priv(ndev); + + brcmf_dbg(TRACE, "Enter, bsscfgidx=%d (AWDL)\n", ifp->bsscfgidx); + + /* No brcmf_cfg80211_down(): this vif was never brought up through + * cfg80211, and tearing down the STA state would hit the same + * unresponsive-firmware path as the bring-up. + */ + brcmf_net_setcarrier(ifp, false); + return 0; +} + +static const struct net_device_ops brcmf_netdev_ops_awdl = { + .ndo_open = brcmf_netdev_open_awdl, + .ndo_stop = brcmf_netdev_stop_awdl, + .ndo_start_xmit = brcmf_netdev_start_xmit, + .ndo_set_mac_address = brcmf_netdev_set_mac_address, +}; + static const struct net_device_ops brcmf_netdev_ops_pri = { .ndo_open = brcmf_netdev_open, .ndo_stop = brcmf_netdev_stop, @@ -651,9 +776,16 @@ int brcmf_net_attach(struct brcmf_if *ifp, bool locked) ndev = ifp->ndev; /* set appropriate operations */ - ndev->netdev_ops = &brcmf_netdev_ops_pri; + ndev->netdev_ops = ifp->is_awdl ? &brcmf_netdev_ops_awdl + : &brcmf_netdev_ops_pri; ndev->needed_headroom += drvr->hdrlen; + if (ifp->is_awdl) { + ndev->needed_headroom += BRCMF_AWDL_ENCAP_LEN; + /* The 802.3 length includes the AWDL envelope. */ + ndev->max_mtu = ETH_DATA_LEN - BRCMF_AWDL_ENCAP_LEN; + ndev->mtu = ndev->max_mtu; + } ndev->ethtool_ops = &brcmf_ethtool_ops; /* set the mac address & netns */ @@ -1492,14 +1624,18 @@ void brcmf_detach(struct device *dev) #endif brcmf_bus_change_state(bus_if, BRCMF_BUS_DOWN); + /* Stop event producers before draining their worker. */ + brcmf_bus_stop(drvr->bus_if); + /* Drain IF events before their interfaces and protocol state go away. */ + brcmf_fweh_detach(drvr); + if (drvr->config) + brcmf_cfg80211_awdl_stop(drvr->config); /* make sure primary interface removed last */ for (i = BRCMF_MAX_IFS - 1; i > -1; i--) { if (drvr->iflist[i]) brcmf_remove_interface(drvr->iflist[i], false); } - brcmf_bus_stop(drvr->bus_if); - brcmf_fweh_detach(drvr); brcmf_proto_detach(drvr); if (drvr->mon_if) { diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h index 4b52a3aa855de8..86c9ef09c90909 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h @@ -250,6 +250,8 @@ enum brcmf_netif_stop_reason { * @pend_8021x_cnt: tracks outstanding number of 802.1x frames. * @pend_8021x_wait: used for signalling change in count. * @fwil_fwerr: flag indicating fwil layer should return firmware error codes. + * @is_awdl: interface carries AWDL data frames (SNAP + AWDL data header). + * @awdl_tx_seq: sequence number written into the AWDL data header on TX. */ struct brcmf_if { struct brcmf_pub *drvr; @@ -268,6 +270,8 @@ struct brcmf_if { struct in6_addr ipv6_addr_tbl[NDOL_MAX_ENTRIES]; u8 ipv6addr_idx; bool fwil_fwerr; + bool is_awdl; + u16 awdl_tx_seq; }; int brcmf_netdev_wait_pend8021x(struct brcmf_if *ifp); @@ -284,6 +288,7 @@ void brcmf_txflowblock_if(struct brcmf_if *ifp, enum brcmf_netif_stop_reason reason, bool state); void brcmf_txfinalize(struct brcmf_if *ifp, struct sk_buff *txp, bool success); void brcmf_netif_rx(struct brcmf_if *ifp, struct sk_buff *skb); +__be16 brcmf_rx_eth_type_trans(struct brcmf_if *ifp, struct sk_buff *skb); void brcmf_netif_mon_rx(struct brcmf_if *ifp, struct sk_buff *skb); void brcmf_net_detach(struct net_device *ndev, bool locked); int brcmf_net_mon_attach(struct brcmf_if *ifp); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h index 432d93ae8fb854..4c7bd41c704c0c 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h @@ -117,6 +117,7 @@ do { \ } while (0) extern int brcmf_msg_level; +extern int brcmf_awdl_trace; struct brcmf_pub; #ifdef DEBUG diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c index eec4b10e1f6741..3e06c23defcb3f 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c @@ -183,6 +183,13 @@ static void brcmf_fweh_handle_if_event(struct brcmf_pub *drvr, brcmf_fweh_call_event_handler(drvr, ifp, emsg->event_code, emsg, data); + /* The handler above establishes the vif <-> ifp linkage. Only then can + * the AWDL netdev be registered, and only from here: this worker holds + * no locks, unlike the vendor command that requested the interface. + */ + if (ifp && ifevent->action == BRCMF_E_IF_ADD) + brcmf_cfg80211_awdl_attach_pending(ifp); + if (ifp && ifevent->action == BRCMF_E_IF_DEL) { bool armed = brcmf_cfg80211_vif_event_armed(drvr->config); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.c index 1f40ff8d632c25..3a4f6b3be44708 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -22,6 +23,24 @@ #include "common.h" #include "interface_create.h" +/* Creating a usable AWDL interface requires interface_create flags + * MAC_USE | IF_INDEX_USE | BSSID_USE together with an explicit BSSID and + * an interface index; brcmfmac otherwise sends MAC_USE alone, which the + * firmware accepts but leaves the AWDL iovars unusable. These parameters + * exist so the three values can be swept without rebuilding the module. + */ +static uint brcmf_awdl_create_flags = 0x1a; +module_param_named(awdl_create_flags, brcmf_awdl_create_flags, uint, 0644); +MODULE_PARM_DESC(awdl_create_flags, "interface_create flags for AWDL"); + +static uint brcmf_awdl_if_index = 2; +module_param_named(awdl_if_index, brcmf_awdl_if_index, uint, 0644); +MODULE_PARM_DESC(awdl_if_index, "interface_create if_index for AWDL"); + +static char *brcmf_awdl_bssid = "00:25:00:ff:94:73"; +module_param_named(awdl_bssid, brcmf_awdl_bssid, charp, 0644); +MODULE_PARM_DESC(awdl_bssid, "interface_create BSSID for AWDL"); + #define BRCMF_INTERFACE_CREATE_VER_1 1 #define BRCMF_INTERFACE_CREATE_VER_2 2 #define BRCMF_INTERFACE_CREATE_VER_3 3 @@ -52,7 +71,7 @@ enum brcmf_interface_type { BRCMF_INTERFACE_TYPE_STA = 0, BRCMF_INTERFACE_TYPE_AP = 1, - /* The missing number here is deliberate */ + BRCMF_INTERFACE_TYPE_AWDL = 2, BRCMF_INTERFACE_TYPE_NAN = 3, BRCMF_INTERFACE_TYPE_P2P_GO = 4, BRCMF_INTERFACE_TYPE_P2P_GC = 5, @@ -130,6 +149,8 @@ static int brcmf_cfg80211_request_if_internal(struct brcmf_if *ifp, u32 version, enum brcmf_interface_type if_type, u8 *macaddr) { + bool awdl = if_type == BRCMF_INTERFACE_TYPE_AWDL; + switch (version) { case BRCMF_INTERFACE_CREATE_VER_1: { struct brcmf_interface_create_v1 iface_v1 = {}; @@ -182,6 +203,27 @@ static int brcmf_cfg80211_request_if_internal(struct brcmf_if *ifp, u32 version, brcmf_set_vif_sta_macaddr(ifp, iface_v3.mac_addr); } + if (awdl) { + flags = brcmf_awdl_create_flags; + /* brcmf_awdl_add_vif() passes NULL, so derive a MAC + * here rather than send MAC_USE with an all-zero + * address. + */ + if ((flags & BRCMF_INTERFACE_MAC_USE) && + is_zero_ether_addr(iface_v3.mac_addr)) + brcmf_set_vif_sta_macaddr(ifp, + iface_v3.mac_addr); + iface_v3.if_index = brcmf_awdl_if_index; + if (mac_pton(brcmf_awdl_bssid, iface_v3.bssid)) { + brcmf_info("awdl interface_create: flags 0x%x, if_index %u, bssid %pM, mac %pM\n", + flags, iface_v3.if_index, + iface_v3.bssid, iface_v3.mac_addr); + } else { + bphy_err(ifp->drvr, "bad awdl_bssid \"%s\"\n", + brcmf_awdl_bssid); + return -EINVAL; + } + } iface_v3.flags = cpu_to_le32(flags); return brcmf_fil_iovar_data_get(ifp, "interface_create", &iface_v3, sizeof(iface_v3)); @@ -240,6 +282,12 @@ int brcmf_cfg80211_request_sta_if(struct brcmf_if *ifp, u8 *macaddr) macaddr); } +int brcmf_cfg80211_request_awdl_if(struct brcmf_if *ifp, u8 *macaddr) +{ + return brcmf_cfg80211_request_if(ifp, BRCMF_INTERFACE_TYPE_AWDL, + macaddr); +} + int brcmf_cfg80211_request_ap_if(struct brcmf_if *ifp) { int err; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.h index 669fa1508b67f6..7fadbea590e28b 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/interface_create.h @@ -9,5 +9,6 @@ int brcmf_cfg80211_request_sta_if(struct brcmf_if *ifp, u8 *macaddr); int brcmf_cfg80211_request_ap_if(struct brcmf_if *ifp); +int brcmf_cfg80211_request_awdl_if(struct brcmf_if *ifp, u8 *macaddr); #endif /* _BRCMF_INTERFACE_CREATE_H_ */ diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c index f100ce8dcbfd37..d841d9e80637d2 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -929,14 +930,26 @@ brcmf_msgbuf_process_ioctl_complete(struct brcmf_msgbuf *msgbuf, void *buf) } +/* Firmware-reported completion status of AWDL data frames. brcmfmac never + * looks at tx_status; it is the WLFC_CTL_PKTFLAG_* fate of the frame + * (0 acked, 4 sent-no-ack i.e. multicast on air, 7 dropped by the firmware). + * Observed: awdl0 multicast completes ~1:3 sent:dropped, template or not. + * Logged for every frame queued on a non-primary flowring. 2000/s so a + * 200-frame burst at 1000/s is logged in full and in order -- the sequence + * of fates is the measurement. + */ +static DEFINE_RATELIMIT_STATE(brcmf_awdl_txs_rs, HZ, 2000); + static void brcmf_msgbuf_process_txstatus(struct brcmf_msgbuf *msgbuf, void *buf) { struct brcmf_commonring *commonring; struct msgbuf_tx_status *tx_status; + struct brcmf_if *ifp; u32 idx; struct sk_buff *skb; u16 flowid; + u8 ring_ifidx; tx_status = (struct msgbuf_tx_status *)buf; idx = le32_to_cpu(tx_status->msg.request_id) - 1; @@ -947,12 +960,42 @@ brcmf_msgbuf_process_txstatus(struct brcmf_msgbuf *msgbuf, void *buf) if (!skb) return; + if (flowid >= msgbuf->max_flowrings) { + bphy_err(msgbuf->drvr, "txstatus for flowring %u out of range\n", + flowid); + brcmu_pkt_buf_free_skb(skb); + return; + } + set_bit(flowid, msgbuf->txstatus_done_map); commonring = msgbuf->flowrings[flowid]; atomic_dec(&commonring->outstanding_tx); - brcmf_txfinalize(brcmf_get_ifp(msgbuf->drvr, tx_status->msg.ifidx), - skb, true); + ifp = brcmf_get_ifp(msgbuf->drvr, tx_status->msg.ifidx); + /* Key on the flowring's ifidx, which the host assigned when it queued + * the frame, not on what the firmware echoes: repeated runs + * logged nothing for 1227 awdl0 frames each, and a completion + * echoed under ifidx 0 would have been invisible to both filters. + * + * The ring may already be gone: brcmf_msgbuf_delete_flowring() gives up + * on outstanding frames after ~75 ms and the firmware completes them + * (0x0006 EXPIRED) after the ring was freed -- awdl0's VO multicast + * never completes while the ring is open (observed as a NULL deref + * here on `awdl-if destroy`). The skb and its pktid are unaffected. + */ + ring_ifidx = msgbuf->flow->rings[flowid] ? + brcmf_flowring_ifidx_get(msgbuf->flow, flowid) : 0; + if (ring_ifidx && READ_ONCE(brcmf_awdl_trace) && + __ratelimit(&brcmf_awdl_txs_rs)) + pr_info("brcmfmac: awdl txstatus ring_ifidx=%u msg_ifidx=%u awdl=%d flow=%u status=%d tx_status=0x%04x meta=%u dst=%pM\n", + ring_ifidx, tx_status->msg.ifidx, + ifp ? ifp->is_awdl : -1, flowid, + (int)(s16)le16_to_cpu(tx_status->compl_hdr.status), + le16_to_cpu(tx_status->tx_status), + le16_to_cpu(tx_status->metadata_len), + skb->len >= ETH_ALEN ? skb->data : (u8 *)"\0\0\0\0\0\0"); + + brcmf_txfinalize(ifp, skb, true); } @@ -1253,7 +1296,7 @@ brcmf_msgbuf_process_rx_complete(struct brcmf_msgbuf *msgbuf, void *buf) return; } - skb->protocol = eth_type_trans(skb, ifp->ndev); + skb->protocol = brcmf_rx_eth_type_trans(ifp, skb); brcmf_netif_rx(ifp, skb); } @@ -1601,14 +1644,25 @@ static int brcmf_msgbuf_stats_read(struct seq_file *seq, void *data) if (!msgbuf->flow->rings[i]) continue; ring = msgbuf->flow->rings[i]; - if (ring->status != RING_OPEN) + hash = &msgbuf->flow->hash[ring->hash_id]; + if (ring->status != RING_OPEN) { + /* A ring the firmware has not acknowledged holds every + * frame queued to it, invisibly, while tx_packets + * advances. Show it rather than skip it. + */ + seq_printf(seq, "id %3u: status %d (not open), qlen %4u\n" + " ifidx %u, fifo %u, da %pM\n", + i, ring->status, + skb_queue_len(&ring->skblist), + hash->ifidx, hash->fifo, hash->mac); continue; + } commonring = msgbuf->flowrings[i]; - hash = &msgbuf->flow->hash[ring->hash_id]; - seq_printf(seq, "id %3u: rp %4u, wp %4u, qlen %4u, blocked %u\n" + seq_printf(seq, "id %3u: rp %4u, wp %4u, qlen %4u, blocked %u, outstanding_tx %d\n" " ifidx %u, fifo %u, da %pM\n", i, commonring->r_ptr, commonring->w_ptr, skb_queue_len(&ring->skblist), ring->blocked, + atomic_read(&commonring->outstanding_tx), hash->ifidx, hash->fifo, hash->mac); } diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c index d07e7c7355d9d2..25623533effa7f 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c @@ -104,6 +104,50 @@ static int brcmf_cfg80211_vndr_cmds_dcmd_handler(struct wiphy *wiphy, return ret; } +#define BRCMF_AWDL_IFNAME "awdl0" + +static int brcmf_cfg80211_vndr_cmds_awdl_handler(struct wiphy *wiphy, + struct wireless_dev *wdev, + const void *data, int len) +{ + struct net_device *ndev; + struct wireless_dev *awdl_wdev; + u32 op; + int err; + + if (len < sizeof(op)) + return -EINVAL; + op = *(const u32 *)data; + + switch (op) { + case BRCMF_VNDR_AWDL_OP_CREATE: + ndev = dev_get_by_name(wiphy_net(wiphy), BRCMF_AWDL_IFNAME); + if (ndev) { + dev_put(ndev); + return -EEXIST; + } + /* Asynchronous: returns as soon as firmware has been asked. + * The netdev appears once the fweh worker registers it, so + * userspace must wait for awdl0 rather than assume it exists. + */ + return brcmf_awdl_add_vif(wiphy, BRCMF_AWDL_IFNAME); + case BRCMF_VNDR_AWDL_OP_DESTROY: + ndev = dev_get_by_name(wiphy_net(wiphy), BRCMF_AWDL_IFNAME); + if (!ndev) + return -ENODEV; + awdl_wdev = ndev->ieee80211_ptr; + if (!awdl_wdev || awdl_wdev->wiphy != wiphy) { + dev_put(ndev); + return -ENODEV; + } + err = brcmf_awdl_del_vif(wiphy, awdl_wdev); + dev_put(ndev); + return err; + default: + return -EINVAL; + } +} + const struct wiphy_vendor_command brcmf_vendor_cmds[] = { { { @@ -115,4 +159,14 @@ const struct wiphy_vendor_command brcmf_vendor_cmds[] = { .policy = VENDOR_CMD_RAW_DATA, .doit = brcmf_cfg80211_vndr_cmds_dcmd_handler }, + { + { + .vendor_id = BROADCOM_OUI, + .subcmd = BRCMF_VNDR_CMDS_AWDL + }, + .flags = WIPHY_VENDOR_CMD_NEED_WDEV | + WIPHY_VENDOR_CMD_NEED_NETDEV, + .policy = VENDOR_CMD_RAW_DATA, + .doit = brcmf_cfg80211_vndr_cmds_awdl_handler + }, }; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.h index 418f33ea6fd3f9..74886de2ba78de 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.h @@ -11,9 +11,21 @@ enum brcmf_vndr_cmds { BRCMF_VNDR_CMDS_UNSPEC, BRCMF_VNDR_CMDS_DCMD, + BRCMF_VNDR_CMDS_AWDL, BRCMF_VNDR_CMDS_LAST }; +/** + * enum brcmf_vndr_awdl_op - operations for BRCMF_VNDR_CMDS_AWDL + * + * @BRCMF_VNDR_AWDL_OP_CREATE: create the AWDL interface + * @BRCMF_VNDR_AWDL_OP_DESTROY: remove the AWDL interface + */ +enum brcmf_vndr_awdl_op { + BRCMF_VNDR_AWDL_OP_CREATE, + BRCMF_VNDR_AWDL_OP_DESTROY, +}; + /** * enum brcmf_nlattrs - nl80211 message attributes * diff --git a/drivers/net/wireless/mediatek/mt7932/Kconfig b/drivers/net/wireless/mediatek/mt7932/Kconfig index bed6279895a48e..49a6fbad0e67e4 100644 --- a/drivers/net/wireless/mediatek/mt7932/Kconfig +++ b/drivers/net/wireless/mediatek/mt7932/Kconfig @@ -1,24 +1,26 @@ # SPDX-License-Identifier: GPL-2.0-only config MT7932_FULLMAC - tristate "Experimental J700 MT7932 fullmac wireless support" - depends on PCI && OF && ARM64 && ARCH_APPLE && IOMMU_DMA + tristate "MediaTek MT7932 Wi-Fi on the Apple MacBook Neo (J700)" + depends on PCI && OF && ARM64 && IOMMU_DMA depends on PCIE_APPLE_PIODMA_DIAG && CFG80211 select FW_LOADER help - Native support for the built-in J700 Wi-Fi endpoint: managed - open-network or WPA2-CCMP station operation on 2.4 GHz at 20 MHz and channels 36..48 - at 20/40/80 MHz, with a software bandwidth limit of 80 MHz. - Requires original firmware, own calibration and original-derived - country policy packages in addition to the cfg80211 regulatory database. - Requires a trained retained PCI link and a translated SID1 DMA domain. - Firmware, runtime DMA and calibration negotiation start automatically; - cfg80211 and userspace select the country and network. No test boot - parameters are required for ordinary station operation. - Once DMA pointers are published the module is pinned until hardware - reset. Normal unload is refused, including with active Bluetooth. - A failed probe reset retains the binding and DMA; a successful checked - reset retires ownership before releasing probe resources. - Other MediaTek devices and other machines are not matched. + Support for the Wi-Fi function of the MediaTek MT7932 radio in the + Apple MacBook Neo (J700). The firmware runs association and the + WPA2-PSK handshake; the driver is a cfg80211 station on 2.4 GHz and + on the 5 GHz UNII-1, -2, -2e and -3 channels that the regulatory + domain permits. Radar (DFS) channels are scanned passively but not + joined. Other MediaTek devices and other machines are not matched. + + The driver needs the original firmware, the unit's own calibration + and the country policy packages under mediatek/mt7932/, besides the + cfg80211 regulatory database. The device is probed before the root + filesystem is mounted, so say M unless those files are in the + initramfs. + + Once the device has been given DMA addresses, the module cannot be + unloaded. A function that fails to initialize and cannot be reset + stays bound and keeps its DMA memory. config MT7932_RF_KUNIT_TEST bool "KUnit tests for MT7932 terminal RF failure" if !KUNIT_ALL_TESTS @@ -29,3 +31,14 @@ config MT7932_RF_KUNIT_TEST admission paths with synthetic devices, without radio hardware. Check association wakeups, retained peer ownership and refusal to reopen RF admission after a terminal calibration error. + +config MT7932_KUNIT_TEST + bool "KUnit tests for MT7932 firmware input validation" if !KUNIT_ALL_TESTS + depends on KUNIT=y && MT7932_FULLMAC + default KUNIT_ALL_TESTS + help + Exercise the validation of what the firmware and the device hand + to the driver, with synthetic, truncated and corrupted input and + without radio hardware: receive descriptors, including frames the + hardware did not decrypt, command replies, regulatory snapshots + and country power packages. diff --git a/drivers/net/wireless/mediatek/mt7932/Makefile b/drivers/net/wireless/mediatek/mt7932/Makefile index 9e76679c46a8a6..781d4530fbee16 100644 --- a/drivers/net/wireless/mediatek/mt7932/Makefile +++ b/drivers/net/wireless/mediatek/mt7932/Makefile @@ -1,4 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_MT7932_FULLMAC) += mt7932-fullmac.o -mt7932-fullmac-y := mt7932.o recovery.o calibration_transport.o data.o frontend.o link.o startup.o reset.o lifecycle.o station.o power.o station_query.o +mt7932-fullmac-y := mt7932.o recovery.o calibration_transport.o data.o frontend.o link.o startup.o reset.o lifecycle.o station.o power.o mt7932-fullmac-$(CONFIG_MT7932_RF_KUNIT_TEST) += rf-test.o +mt7932-fullmac-$(CONFIG_MT7932_KUNIT_TEST) += input-test.o +mt7932-fullmac-$(CONFIG_MT7932_KUNIT_TEST) += regulatory-test.o diff --git a/drivers/net/wireless/mediatek/mt7932/association.h b/drivers/net/wireless/mediatek/mt7932/association.h index eb207921906e14..ba733a885e2617 100644 --- a/drivers/net/wireless/mediatek/mt7932/association.h +++ b/drivers/net/wireless/mediatek/mt7932/association.h @@ -91,8 +91,8 @@ static inline int mt7932_assoc_capabilities(u8 *out, size_t capacity, } /* Original25G83 pure WPA2 binary-key result: structured type6, not a string. - * NEO_TYPE6_BINARY_PMK_NATIVE_CONTRACT: standard nl80211 PMK maps directly; - * no passphrase API, key derivation, duplicate independent PMK or extra flags. + * The standard nl80211 PMK maps directly; no passphrase API, key derivation, + * duplicate independent PMK or extra flags. */ static inline int mt7932_connect_binary_pmk(u8 *body, size_t capacity, const u8 *pmk, size_t length) @@ -111,8 +111,8 @@ static inline int mt7932_connect_binary_pmk(u8 *body, size_t capacity, } /* Explicit OPEN or PMK-only WPA2/CCMP; never an automatic security fallback. - * Caller owns identities/resources and validated BSS state. - * NEO_WPA2_FULLMAC_ASSOCIATION_AND_ETHERNET_CONTRACT. Never log this body. + * Caller owns identities/resources and validated BSS state. Never log this + * body: it carries the PMK. */ static inline int mt7932_connect_body(u8 *body, size_t capacity, const u8 *ssid, size_t ssid_length, const u8 pmk[32], const u8 bssid[6], @@ -141,7 +141,6 @@ static inline int mt7932_connect_body(u8 *body, size_t capacity, const u8 *ssid, /* NORMAL25G83 original strict keyless OPEN branch, without optional * automatic-authentication policy. Keep this coherent version/auth/ * policy tuple together; no dummy keys or stale security state. - * NEO_NORMAL_STRICT_OPEN_PRODUCER_CONTRACT. */ if (open) { body[1] = 1; diff --git a/drivers/net/wireless/mediatek/mt7932/bandwidth.h b/drivers/net/wireless/mediatek/mt7932/bandwidth.h index c5efe6599e0a46..07c281e0fc5893 100644 --- a/drivers/net/wireless/mediatek/mt7932/bandwidth.h +++ b/drivers/net/wireless/mediatek/mt7932/bandwidth.h @@ -13,13 +13,14 @@ static inline u8 mt7932_domain_5g_bw(void) } /* EE4d width 0 combines 20/40; primary and extension select the center. - * Keep all operational primaries inside 36..48, and 2 GHz at 20 MHz. + * Extension 1 places the secondary above the primary, 3 below it. Keep 2 GHz + * at 20 MHz, and 165, which has no 40 MHz pair, at 20 MHz too. */ static inline bool mt7932_channel_layout(u8 primary, u8 width, u8 extension, u8 maximum, u8 *center, u8 *bw) { u8 band = mt7932_channel_band(primary); - u8 expected = primary == 36 || primary == 44 ? 1 : 3; + u8 pair = mt7932_channel_ht40_centre(primary); if (!band || maximum > 2 || width > 1) return false; @@ -28,12 +29,12 @@ static inline bool mt7932_channel_layout(u8 primary, u8 width, u8 extension, *bw = 0; return true; } - if (band != 2 || extension != expected) + if (band != 2 || !pair || extension != (pair > primary ? 1 : 3)) return false; *bw = width ? 2 : 1; if (*bw > maximum) return false; - *center = width ? 42 : extension == 1 ? primary + 2 : primary - 2; + *center = width ? mt7932_channel_vht80_centre(primary) : pair; return true; } diff --git a/drivers/net/wireless/mediatek/mt7932/calibration.h b/drivers/net/wireless/mediatek/mt7932/calibration.h index cd53e0be435096..f13ee11866bade 100644 --- a/drivers/net/wireless/mediatek/mt7932/calibration.h +++ b/drivers/net/wireless/mediatek/mt7932/calibration.h @@ -2,6 +2,7 @@ #ifndef MT7932_CALIBRATION_H #define MT7932_CALIBRATION_H +#include "channels.h" #include "protocol.h" /* Protected by the RX/procedure lock. One active logical procedure only; @@ -79,7 +80,7 @@ struct mt7932_cal_segment { }; /* Validate the entire original container before selecting any data to send. - * NEO_MT7932_STOCK_ARTIFACTS_AND_CAL_CONVERSION: raw BLOB, not a transcript. + * The input is the raw BLOB container, not a converted transcript. */ static inline int mt7932_cal_validate(const u8 *data, size_t size) { @@ -159,7 +160,7 @@ static inline int mt7932_cal_association(const u8 *data, size_t size, u8 channel *input_tag = 0; if (channel >= 1 && channel <= 13) group = 0; - else if (channel >= 36 && channel <= 48 && !(channel % 2)) + else if (mt7932_channel_5g_cal(channel)) group = 1; else return -EOPNOTSUPP; @@ -245,7 +246,7 @@ static inline int mt7932_cal_requested_5g(const u8 *data, size_t size, *input_tag = 0; if (upper != 0x1000 || version != 12 || module != 0x89 || - channel < 36 || channel > 48 || channel % 2) + !mt7932_channel_5g_cal(channel)) return -EOPNOTSUPP; ret = mt7932_cal_validate(data, size); if (ret) @@ -260,7 +261,7 @@ static inline int mt7932_cal_requested_5g(const u8 *data, size_t size, if (segment.data[i]) return -EOPNOTSUPP; /* Own row610f07 does not remap center selectors1/2; only primaries. */ - selected = channel % 4 ? channel : channel <= 40 ? 38 : 46; + selected = mt7932_channel_band(channel) ? mt7932_channel_cal_centre(channel) : channel; params[0] = 0x10000001; params[1] = 0x10001000 | channel; params[2] = 0x10001000 | (channel << 17) | selected; diff --git a/drivers/net/wireless/mediatek/mt7932/calibration_transport.c b/drivers/net/wireless/mediatek/mt7932/calibration_transport.c index 1d6d4d6985a45b..4efd5bed774798 100644 --- a/drivers/net/wireless/mediatek/mt7932/calibration_transport.c +++ b/drivers/net/wireless/mediatek/mt7932/calibration_transport.c @@ -1,8 +1,6 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include "mt7932.h" - - /* Called only by the serialized command owner, never from the RX consumer. */ int mt_cal_procedure(struct mt7932 *m, struct mt7932_cal_piece *pieces, unsigned int count, unsigned int logical, u8 context_version) @@ -124,12 +122,10 @@ int mt_calibration_gate(struct mt7932 *m) get_unaligned_le32(m->reply + 40)) { dev_err(&m->pdev->dev, "WCAL_REPLY_REJECTED: length=%zu eid=%02x ext=%02x\n", m->reply_length, m->reply[28], m->reply[32]); - print_hex_dump(KERN_INFO, "WCAL_RESULT: ", DUMP_PREFIX_OFFSET, 16, 1, - m->reply, min_t(size_t, m->reply_length, 64), false); ret = -EPROTO; goto out; } - dev_info(&m->pdev->dev, "OWN_WCAL_ACCEPTED: %zu bytes\n", wcal->size); + dev_dbg(&m->pdev->dev, "OWN_WCAL_ACCEPTED: %zu bytes\n", wcal->size); phase = "power-on D6 command/completion"; for (at = 0, group = 0; at < planned; at += count, group++) { count = pieces[at].fragment >> 4; @@ -140,10 +136,10 @@ int mt_calibration_gate(struct mt7932 *m) ret = mt_cal_procedure(m, pieces + at, count, 1, m->preload_version); if (ret) goto out; - dev_info(&m->pdev->dev, "POWER_ON_CAL_GROUP_OK: %u type=%u parameter=%08x fragments=%u\n", - group, pieces[at].type, pieces[at].parameter, count); + dev_dbg(&m->pdev->dev, "POWER_ON_CAL_GROUP_OK: %u type=%u parameter=%08x fragments=%u\n", + group, pieces[at].type, pieces[at].parameter, count); } - dev_info(&m->pdev->dev, "POWER_ON_CAL_COMPLETE: %u groups, %d fragments\n", group, planned); + dev_dbg(&m->pdev->dev, "POWER_ON_CAL_COMPLETE: %u groups, %d fragments\n", group, planned); phase = "queued D7 planning/command/completion"; ret = mt_cal_drain_requests(m, oca); out: @@ -187,7 +183,8 @@ int mt_cal_drain_requests(struct mt7932 *m, const struct firmware *oca) mt7932_cal_request_replies(request), m->preload_version); if (ret) return ret; - dev_info(&m->pdev->dev, "CAL_REQUEST_SERVICED: channel=%u\n", get_unaligned_le32(request + 12)); + dev_dbg(&m->pdev->dev, "CAL_REQUEST_SERVICED: channel=%u\n", + get_unaligned_le32(request + 12)); } return 0; } diff --git a/drivers/net/wireless/mediatek/mt7932/channels.h b/drivers/net/wireless/mediatek/mt7932/channels.h index 8c4b61e4a3ec75..bae04c245d72a5 100644 --- a/drivers/net/wireless/mediatek/mt7932/channels.h +++ b/drivers/net/wireless/mediatek/mt7932/channels.h @@ -4,14 +4,78 @@ /* Bounded operational whitelist, not the full calibration channel namespace. * Firmware RF bands1/2 are distinct from single-interface MAC index0. + * 5 GHz covers the 20 MHz primaries of UNII-1/2 (36..64), UNII-2e (100..144) + * and UNII-3 (149..165); regulatory flags decide which of them are usable. */ +#define MT7932_CHANNELS_2G 13 +#define MT7932_CHANNELS_5G 25 +#define MT7932_CHANNELS (MT7932_CHANNELS_2G + MT7932_CHANNELS_5G) + static inline unsigned int mt7932_channel_band(unsigned int channel) { if (channel >= 1 && channel <= 13) return 1; - if (channel >= 36 && channel <= 48 && !(channel % 4)) + if (((channel >= 36 && channel <= 64) || (channel >= 100 && channel <= 144)) && + !(channel % 4)) + return 2; + if (channel >= 149 && channel <= 165 && !((channel - 149) % 4)) return 2; return 0; } +/* The nth 5 GHz primary, in the order the wiphy advertises them. */ +static inline unsigned int mt7932_channel_5g(unsigned int index) +{ + if (index < 8) + return 36 + 4 * index; + if (index < 20) + return 100 + 4 * (index - 8); + return 149 + 4 * (index - 20); +} + +/* Offset of a 5 GHz primary within its 80 MHz block numbering. */ +static inline unsigned int mt7932_channel_slot(unsigned int primary, unsigned int *base) +{ + *base = primary >= 149 ? 149 : primary >= 100 ? 100 : 36; + return (primary - *base) / 4; +} + +/* 40 MHz pair centre of a 5 GHz primary; 0 where no pair exists (165). */ +static inline unsigned int mt7932_channel_ht40_centre(unsigned int primary) +{ + unsigned int base, slot; + + if (mt7932_channel_band(primary) != 2 || primary == 165) + return 0; + slot = mt7932_channel_slot(primary, &base); + return base + 4 * (slot & ~1U) + 2; +} + +/* 80 MHz block centre of a 5 GHz primary; 0 where no block exists (165). */ +static inline unsigned int mt7932_channel_vht80_centre(unsigned int primary) +{ + unsigned int base, slot; + + if (mt7932_channel_band(primary) != 2 || primary == 165) + return 0; + slot = mt7932_channel_slot(primary, &base); + return base + 4 * (slot & ~3U) + 6; +} + +/* A 5 GHz primary or a 40/80 MHz centre that calibration can be asked for. */ +static inline bool mt7932_channel_5g_cal(unsigned int channel) +{ + if ((channel >= 36 && channel <= 64) || (channel >= 100 && channel <= 144)) + return !(channel % 2); + return channel >= 149 && channel <= 167 && (channel % 2); +} + +/* Calibration centre for a 5 GHz primary. 165 has no 40 MHz pair; its own + * calibration package carries the record for the 165/169 centre, 167. + */ +static inline unsigned int mt7932_channel_cal_centre(unsigned int primary) +{ + return primary == 165 ? 167 : mt7932_channel_ht40_centre(primary); +} + #endif diff --git a/drivers/net/wireless/mediatek/mt7932/data.c b/drivers/net/wireless/mediatek/mt7932/data.c index 4d89f8925f86d0..ba06db134414ea 100644 --- a/drivers/net/wireless/mediatek/mt7932/data.c +++ b/drivers/net/wireless/mediatek/mt7932/data.c @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include "mt7932.h" static bool mt_bss_tx_allowed(struct mt7932 *m); @@ -35,6 +35,25 @@ void mt_data_clean_locked(struct mt7932 *m) netif_wake_queue(m->netdev); } +/* + * The stack marks a packet that has another one right behind it with + * netdev_xmit_more(), so the TX0 doorbell is rung once per burst instead of + * once per packet, and at least every MT7932_TX_BATCH packets so that the + * device starts on a long burst early. + */ +#define MT7932_TX_BATCH 64 + +/* data_lock held: hand every published TX0 descriptor to the device. */ +void mt_data_kick_locked(struct mt7932 *m) +{ + struct mt7932_ring *q = &m->tx[0]; + + if (!m->data_unkicked) + return; + m->data_unkicked = 0; + mt_write(m, q->reg + 8, q->head); +} + void mt_data_clean(struct mt7932 *m) { unsigned long flags; @@ -42,6 +61,10 @@ void mt_data_clean(struct mt7932 *m) if (!READ_ONCE(m->data_ready)) return; spin_lock_irqsave(&m->data_lock, flags); + /* The queue can be stopped from outside ndo_start_xmit() in the middle + * of a burst; no descriptor waits longer than the next interrupt. + */ + mt_data_kick_locked(m); mt_data_clean_locked(m); spin_unlock_irqrestore(&m->data_lock, flags); } @@ -59,10 +82,12 @@ void mt_bss_presence(struct mt7932 *m, const struct mt7932_event *event) spin_lock_irqsave(&m->data_lock, flags); m->bss_absent = !!body[1]; m->bss_quota = body[2]; - if (!mt_bss_tx_allowed(m)) + if (!mt_bss_tx_allowed(m)) { netif_stop_queue(m->netdev); - else if (READ_ONCE(m->data_ready)) + mt_data_kick_locked(m); + } else if (READ_ONCE(m->data_ready)) { mt_data_clean_locked(m); + } spin_unlock_irqrestore(&m->data_lock, flags); dev_dbg_ratelimited(&m->pdev->dev, "BSS_PRESENCE: absent=%u quota=%u\n", !!body[1], body[2]); @@ -122,6 +147,7 @@ netdev_tx_t mt_net_xmit(struct sk_buff *skb, struct net_device *netdev) slot = q->head; if (!mt_bss_tx_allowed(m) || m->data_live[slot] || q->queued >= q->count - 1) { netif_stop_queue(netdev); + mt_data_kick_locked(m); spin_unlock_irqrestore(&m->data_lock, flags); return NETDEV_TX_BUSY; } @@ -147,23 +173,49 @@ netdev_tx_t mt_net_xmit(struct sk_buff *skb, struct net_device *netdev) q->head = (slot + 1) % q->count; q->queued++; dma_wmb(); - mt_write(m, q->reg + 8, q->head); + m->data_unkicked++; mt_bss_tx_account(m); m->station_tx_packets++; m->station_tx_bytes += skb->len; if (!mt_bss_tx_allowed(m) || m->data_live[q->head] || q->queued >= q->count - 1) netif_stop_queue(netdev); + if (!netdev_xmit_more() || netif_queue_stopped(netdev) || + m->data_unkicked >= MT7932_TX_BATCH) + mt_data_kick_locked(m); spin_unlock_irqrestore(&m->data_lock, flags); netdev->stats.tx_packets++; netdev->stats.tx_bytes += skb->len; dev_kfree_skb_any(skb); return NETDEV_TX_OK; drop: + /* The burst ends here: publish what earlier packets queued. */ + if (!netdev_xmit_more() && READ_ONCE(m->data_ready)) { + spin_lock_irqsave(&m->data_lock, flags); + mt_data_kick_locked(m); + spin_unlock_irqrestore(&m->data_lock, flags); + } netdev->stats.tx_dropped++; dev_kfree_skb_any(skb); return NETDEV_TX_OK; } +/* Width in MHz of a rate decoded by mt7932_rx_rate(). */ +static int mt_rate_width(enum rate_info_bw bw) +{ + switch (bw) { + case RATE_INFO_BW_20: + return 20; + case RATE_INFO_BW_40: + return 40; + case RATE_INFO_BW_80: + return 80; + case RATE_INFO_BW_160: + return 160; + default: + return INT_MAX; + } +} + void mt_data_receive(struct mt7932 *m, const struct mt7932_rx_frame *frame) { struct sk_buff *skb; @@ -182,6 +234,13 @@ void mt_data_receive(struct mt7932 *m, const struct mt7932_rx_frame *frame) m->netdev->stats.rx_dropped++; goto unlock; } + if (!mt7932_rx_protection_ok(frame, m->connect_open)) { + m->netdev->stats.rx_dropped++; + dev_dbg_ratelimited(&m->pdev->dev, + "RX_UNPROTECTED_DROPPED: cipher=%u mismatch=%u\n", + frame->sec_mode, frame->cipher_mismatch); + goto unlock; + } skb = netdev_alloc_skb_ip_align(m->netdev, frame->length); if (!skb) { m->netdev->stats.rx_dropped++; @@ -200,19 +259,15 @@ void mt_data_receive(struct mt7932 *m, const struct mt7932_rx_frame *frame) if (frame->signal_valid && signal > -128) { m->station_signal = signal; m->station_signal_valid = true; + mt_cqm_signal(m, signal); } if (frame->unicast && frame->rate_valid) { - m->station_rx_rate_valid = mt7932_rx_rate(frame->rxv, m->phy_cap[4], - &m->station_rx_rate); - /* Report only RX widths admitted by the negotiated channel. - * The qualified decoder already bounds mode/MCS/NSS/GI; the - * association remains 20 MHz on 2 GHz and at most 80 MHz on 5 GHz. - */ - if (((frame->rxv >> 12) & 7) > - (m->connect_chandef.width == NL80211_CHAN_WIDTH_80 ? 2 : - m->connect_chandef.width == NL80211_CHAN_WIDTH_40 ? 1 : 0) || - (frame->channel <= 14 && ((frame->rxv >> 12) & 7)) || - (frame->channel > 14 && !((frame->rxv >> 24) & 15))) + struct rate_info *rate = &m->station_rx_rate; + + m->station_rx_rate_valid = mt7932_rx_rate(frame->rxv, m->phy_cap[4], rate); + /* No CCK on 5 GHz, and no rate wider than the associated channel. */ + if ((frame->channel > 14 && !((frame->rxv >> 24) & 15)) || + mt_rate_width(rate->bw) > cfg80211_chandef_get_width(&m->connect_chandef)) m->station_rx_rate_valid = false; m->station_rx_rate_time = jiffies; } @@ -265,10 +320,10 @@ int mt_data_prepare(struct mt7932 *m) /* The selected response parser establishes sequence completion only, * not a command-specific firmware status. No payload-status invention. */ - dev_info(&m->pdev->dev, "MAC_RTS_RESPONSE: sequence matched, EID=%02x length=%zu\n", - m->reply[28], m->reply_length); + dev_dbg(&m->pdev->dev, "MAC_RTS_RESPONSE: sequence matched, EID=%02x length=%zu\n", + m->reply[28], m->reply_length); /* Publish the initialized data_lock and DMA state to failure handlers. */ smp_store_release(&m->data_ready, true); - dev_info(&m->pdev->dev, "ETHERNET_PREPARED: TX0 retained coherent slots=512, RX translation enabled\n"); + dev_dbg(&m->pdev->dev, "ETHERNET_PREPARED: TX0 retained coherent slots=512, RX translation enabled\n"); return 0; } diff --git a/drivers/net/wireless/mediatek/mt7932/frontend.c b/drivers/net/wireless/mediatek/mt7932/frontend.c index f303afe2246f4d..84170c8f613265 100644 --- a/drivers/net/wireless/mediatek/mt7932/frontend.c +++ b/drivers/net/wireless/mediatek/mt7932/frontend.c @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include "mt7932.h" static void mt_scan_finish_work(struct work_struct *work); @@ -15,9 +15,6 @@ struct mt7932 *mt_from_wiphy(struct wiphy *wiphy) return *(struct mt7932 **)wiphy_priv(wiphy); } - - - static void mt_scan_finish_work(struct work_struct *work) { struct mt7932 *m = container_of(work, struct mt7932, scan_finish_work); @@ -64,8 +61,8 @@ static void mt_scan_finish_work(struct work_struct *work) } mutex_unlock(&m->command_mutex); if (request) { - dev_info(&m->pdev->dev, "SCAN_FINISHED: seq=%u aborted=%u received-BSS=%llu\n", - cancel[0], info.aborted, m->beacons); + dev_dbg(&m->pdev->dev, "SCAN_FINISHED: seq=%u aborted=%u received-BSS=%llu\n", + cancel[0], info.aborted, m->beacons); cfg80211_scan_done(request, &info); } } @@ -89,7 +86,7 @@ void mt_scan_event(struct mt7932 *m, const struct mt7932_event *event) if (m->stopping) return; /* Abort completes userspace locally, not by an invented cancel ACK. - * Keep the sequence reserved until an actual old-campaign done arrives. + * Keep the sequence reserved until the old scan's own done event arrives. */ if (event->length >= 37 && m->retired_scan_seq && event->packet[36] == m->retired_scan_seq) { @@ -126,15 +123,35 @@ void mt_abort_scan(struct wiphy *wiphy, struct wireless_dev *wdev) spin_unlock_irqrestore(&m->response_lock, flags); } +/* Wait at most @timeout in total for a cancelled scan to retire. Callers + * may hold RTNL, so this never waits longer than asked. A stopping driver + * completes scan_done for good, which would turn the wait into a spin. + */ +static void mt_scan_retire_wait(struct mt7932 *m, unsigned long timeout) +{ + unsigned long deadline = jiffies + timeout; + long left; + + mt_abort_scan(m->wiphy, &m->wdev); + flush_work(&m->scan_finish_work); + while (READ_ONCE(m->retired_scan_seq) && !READ_ONCE(m->stopping)) { + left = (long)(deadline - jiffies); + if (left <= 0 || !wait_for_completion_timeout(&m->scan_done, left)) + break; + } +} + /* No command/response lock held. The caller closes new scan admission first. - * Natural EID0d completion fences the campaign; a local cancel alone does not. + * Natural EID0d completion fences the scan; a local cancel alone does not. + * The batch awaited here may already have been followed by the next one, + * which the abort then cancels: give the firmware time to end that one too, + * rather than declaring the link unrecoverable while its cancel is in flight. */ int mt_scan_quiesce(struct mt7932 *m) { if (READ_ONCE(m->scan_request) || READ_ONCE(m->retired_scan_seq)) wait_for_completion_timeout(&m->scan_done, msecs_to_jiffies(5000)); - mt_abort_scan(m->wiphy, &m->wdev); - flush_work(&m->scan_finish_work); + mt_scan_retire_wait(m, msecs_to_jiffies(2000)); return READ_ONCE(m->retired_scan_seq) ? -EBUSY : 0; } @@ -146,7 +163,7 @@ static int mt_scan_submit_next(struct mt7932 *m) { struct cfg80211_scan_request *request = m->scan_request; const struct cfg80211_ssid *ssid = NULL; - u8 body[0x4d4], channels[13]; + u8 body[0x4d4], channels[MT7932_CHANNELS_5G]; unsigned int count = 0, i; unsigned long flags; bool passive = false; @@ -160,11 +177,15 @@ static int mt_scan_submit_next(struct mt7932 *m) passive = !(m->scan_batch & 1); for (i = 0; i < request->n_channels; i++) { struct ieee80211_channel *channel = request->channels[i]; - bool listen = !request->n_ssids || + bool radar = channel->flags & IEEE80211_CHAN_RADAR; + bool listen = !request->n_ssids || radar || (channel->flags & IEEE80211_CHAN_NO_IR); + /* Radar channels are never joined; while associated, + * a passive dwell there only delays the home channel. + */ if (mt7932_channel_band(channel->hw_value) != m->scan_band || - listen != passive) + listen != passive || (radar && m->scan_home_channel)) continue; if (channel->flags & (IEEE80211_CHAN_DISABLED | IEEE80211_CHAN_NO_20MHZ) || count == ARRAY_SIZE(channels)) @@ -204,8 +225,8 @@ static int mt_scan_submit_next(struct mt7932 *m) ret = mt_request(m, 3, true, true, false, body, sizeof(body)); } if (!ret) - dev_info(&m->pdev->dev, "SCAN_SUBMITTED: seq=%u band=%u channels=%u passive=%u\n", - m->scan_seq, m->scan_band, count, passive); + dev_dbg(&m->pdev->dev, "SCAN_SUBMITTED: seq=%u band=%u channels=%u passive=%u\n", + m->scan_seq, m->scan_band, count, passive); return ret; } @@ -216,7 +237,7 @@ static int mt_scan_submit_next(struct mt7932 *m) */ int mt_connect_discover(struct mt7932 *m) { - u8 body[0x4d4], channels[13], cancel[4] = {}; + u8 body[0x4d4], channels[MT7932_CHANNELS_5G], cancel[4] = {}; unsigned long flags; unsigned int attempt, band, i, count; int ret = 0; @@ -237,7 +258,8 @@ int mt_connect_discover(struct mt7932 *m) if (m->connect_channel_req && channel != m->connect_channel_req) continue; - if (channel->flags & (IEEE80211_CHAN_DISABLED | IEEE80211_CHAN_NO_20MHZ)) + if (channel->flags & (IEEE80211_CHAN_DISABLED | IEEE80211_CHAN_NO_20MHZ | + IEEE80211_CHAN_RADAR)) continue; passive |= !!(channel->flags & IEEE80211_CHAN_NO_IR); channels[count++] = channel->hw_value; @@ -327,10 +349,14 @@ int mt_scan(struct wiphy *wiphy, struct cfg80211_scan_request *request) if (request->wdev != &m->wdev) return -ENODEV; mt_retry_missing_policy(m); - dev_info_ratelimited(&m->pdev->dev, "SCAN_REQUEST: channels=%u ssids=%d ies=%zu flags=%08x duration=%u/%u\n", - request->n_channels, request->n_ssids, request->ie_len, request->flags, - request->duration, request->duration_mandatory); - if (!request->n_channels || request->n_channels > 17 || request->ie_len > 600 || + dev_dbg(&m->pdev->dev, + "SCAN_REQUEST: channels=%u ssids=%d ies=%zu flags=%08x duration=%u/%u\n", + request->n_channels, request->n_ssids, request->ie_len, request->flags, + request->duration, request->duration_mandatory); + /* cfg80211 lists each advertised channel at most once. */ + if (request->n_channels > MT7932_CHANNELS) + return -EOPNOTSUPP; + if (!request->n_channels || request->ie_len > 600 || request->duration_mandatory || (request->flags & ~(NL80211_SCAN_FLAG_FLUSH | NL80211_SCAN_FLAG_COLOCATED_6GHZ)) || request->n_ssids > 1 || (request->n_ssids && request->ssids[0].ssid_len > 32)) @@ -364,6 +390,9 @@ int mt_scan(struct wiphy *wiphy, struct cfg80211_scan_request *request) spin_unlock_irqrestore(&m->response_lock, flags); cancel_delayed_work(&m->scan_timeout_work); } else { + /* Nothing eligible to submit: complete the request. */ + if (ret) + schedule_work(&m->scan_finish_work); ret = 0; } out: @@ -450,12 +479,20 @@ static int mt_set_mac_address(struct net_device *netdev, void *address) /* No live-MAC change: the next BSS activation and connect commands use * this address together. A failed retirement must not change identity. + * NetworkManager takes the interface down before changing the address, + * which cancels any scan; let that scan retire instead of failing. */ + if (!netif_running(netdev)) + mt_scan_retire_wait(m, msecs_to_jiffies(2000)); mutex_lock(&m->command_mutex); spin_lock_irqsave(&m->response_lock, flags); if (m->stopping || m->link_failed || m->bss_active || m->connecting || m->connected || m->disconnecting || m->peer_valid || m->scan_request || m->retired_scan_seq || m->discovering) { + dev_dbg(&m->pdev->dev, + "address change refused: bss=%u connecting=%u connected=%u peer=%u scan=%u retired=%u\n", + m->bss_active, m->connecting, m->connected, m->peer_valid, + !!m->scan_request, m->retired_scan_seq); ret = -EBUSY; } else { ret = eth_mac_addr(netdev, address); @@ -510,7 +547,8 @@ static void mt_cal_work(struct work_struct *work) m->preload_version); if (ret) break; - dev_info(&m->pdev->dev, "D7_CALIBRATION_COMPLETE: channel=%u\n", get_unaligned_le32(request + 12)); + dev_dbg(&m->pdev->dev, "D7_CALIBRATION_COMPLETE: channel=%u\n", + get_unaligned_le32(request + 12)); } release_firmware(oca); if (!ret) @@ -564,7 +602,8 @@ int mt_register_interface(struct mt7932 *m) free_netdev(netdev); return ret; } - dev_info(&m->pdev->dev, "CFG80211_INTERFACE_READY: %s, RF awaits country policy\n", netdev->name); + dev_dbg(&m->pdev->dev, "CFG80211_INTERFACE_READY: %s, RF awaits country policy\n", + netdev->name); return 0; } @@ -600,7 +639,8 @@ int mt_enable_scan(struct mt7932 *m) return ret; } WRITE_ONCE(m->rf_ready, true); - dev_info(&m->pdev->dev, "CFG80211_SCAN_READY: %s, carrier off until WPA2 authorization\n", m->netdev->name); + dev_dbg(&m->pdev->dev, "CFG80211_SCAN_READY: %s, carrier off until WPA2 authorization\n", + m->netdev->name); schedule_work(&m->cal_work); spin_unlock_irqrestore(&m->response_lock, flags); return 0; @@ -632,11 +672,14 @@ void mt_packet_receive(struct mt7932 *m, const u8 *packet, size_t length) unsigned long flags; spin_lock_irqsave(&m->response_lock, flags); - if (tim && tim->datalen >= 4 && m->connected && !m->disconnecting && - !m->stopping && !m->power_tim && frame.channel == m->connect_channel && + if (m->connected && !m->disconnecting && !m->stopping && + frame.channel == m->connect_channel && ether_addr_equal(frame.data + 16, m->connect_bssid)) { - m->power_tim = true; - schedule_work(&m->power_work); + mt_cqm_signal(m, mt7932_rx_signal(&frame, m->antenna_mask)); + if (tim && tim->datalen >= 4 && !m->power_tim) { + m->power_tim = true; + schedule_work(&m->power_work); + } } spin_unlock_irqrestore(&m->response_lock, flags); } diff --git a/drivers/net/wireless/mediatek/mt7932/input-test.c b/drivers/net/wireless/mediatek/mt7932/input-test.c new file mode 100644 index 00000000000000..9ee9253f34a307 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt7932/input-test.c @@ -0,0 +1,190 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * KUnit tests for the validation of firmware input: RX descriptors and + * command replies. The firmware and the device are treated as untrusted. + */ +#include + +#include "mt7932.h" + +#define RX_HEADER 24 +#define RX_ETHERNET 14 +#define RX_PAYLOAD 40 +#define RX_SIZE (RX_HEADER + RX_ETHERNET + RX_PAYLOAD) + +/* A translated (Ethernet) unicast data frame on channel 36 for WTBL 5. */ +static void rx_build(u8 *packet, u32 extra_w1) +{ + memset(packet, 0, RX_SIZE); + put_unaligned_le32(RX_SIZE | (2U << 27), packet); + put_unaligned_le32(5 | extra_w1, packet + 4); + put_unaligned_le32(BIT(13), packet + 8); + put_unaligned_le32((36U << 8) | (1U << 16), packet + 12); + memset(packet + RX_HEADER, 0xa5, RX_ETHERNET + RX_PAYLOAD); + put_unaligned_be16(ETH_P_IP, packet + RX_HEADER + 12); +} + +static u32 rx_cipher(unsigned int mode) +{ + return mode << MT7932_RXD1_SEC_MODE_SHIFT; +} + +static void mt7932_rx_decrypted_test(struct kunit *test) +{ + struct mt7932_rx_frame frame; + u8 packet[RX_SIZE]; + + rx_build(packet, rx_cipher(4)); + KUNIT_ASSERT_EQ(test, mt7932_rx_frame(packet, sizeof(packet), &frame), 0); + KUNIT_EXPECT_TRUE(test, frame.translated); + KUNIT_EXPECT_EQ(test, frame.sec_mode, 4); + KUNIT_EXPECT_EQ(test, frame.wtbl, 5); + KUNIT_EXPECT_EQ(test, frame.length, (size_t)(RX_ETHERNET + RX_PAYLOAD)); + KUNIT_EXPECT_TRUE(test, mt7932_rx_protection_ok(&frame, false)); + KUNIT_EXPECT_TRUE(test, mt7932_rx_protection_ok(&frame, true)); +} + +/* A plaintext frame on a protected connection is never delivered. */ +static void mt7932_rx_plaintext_test(struct kunit *test) +{ + struct mt7932_rx_frame frame; + u8 packet[RX_SIZE]; + + rx_build(packet, 0); + KUNIT_ASSERT_EQ(test, mt7932_rx_frame(packet, sizeof(packet), &frame), 0); + KUNIT_EXPECT_EQ(test, frame.sec_mode, 0); + KUNIT_EXPECT_FALSE(test, mt7932_rx_protection_ok(&frame, false)); + /* An open connection has nothing to decrypt. */ + KUNIT_EXPECT_TRUE(test, mt7932_rx_protection_ok(&frame, true)); +} + +/* Decrypted with the wrong cipher or key length: refused on WPA2. */ +static void mt7932_rx_mismatch_test(struct kunit *test) +{ + struct mt7932_rx_frame frame; + u8 packet[RX_SIZE]; + + rx_build(packet, rx_cipher(4) | MT7932_RXD1_CIPHER_LEN_MISMATCH); + KUNIT_ASSERT_EQ(test, mt7932_rx_frame(packet, sizeof(packet), &frame), 0); + KUNIT_EXPECT_TRUE(test, frame.cipher_mismatch); + KUNIT_EXPECT_FALSE(test, mt7932_rx_protection_ok(&frame, false)); + /* A translated frame with a cipher mismatch is rejected by the parser. */ + rx_build(packet, rx_cipher(4) | MT7932_RXD1_CIPHER_MISMATCH); + KUNIT_EXPECT_EQ(test, mt7932_rx_frame(packet, sizeof(packet), &frame), -EBADMSG); +} + +/* ICV, TKIP MIC and FCS errors are rejected before any field is used. */ +static void mt7932_rx_integrity_test(struct kunit *test) +{ + struct mt7932_rx_frame frame; + u8 packet[RX_SIZE]; + unsigned int bit; + + for (bit = 25; bit <= 27; bit++) { + rx_build(packet, rx_cipher(4) | BIT(bit)); + KUNIT_EXPECT_EQ(test, mt7932_rx_frame(packet, sizeof(packet), &frame), + -EBADMSG); + } +} + +/* Short buffers and length fields beyond the buffer never parse. */ +static void mt7932_rx_short_test(struct kunit *test) +{ + struct mt7932_rx_frame frame; + u8 packet[RX_SIZE], *copy; + size_t cut; + + rx_build(packet, rx_cipher(4)); + for (cut = 0; cut < RX_HEADER + RX_ETHERNET; cut++) { + /* An exact-size copy lets KASAN catch any read past the end. */ + copy = kmemdup(packet, cut, GFP_KERNEL); + KUNIT_ASSERT_TRUE(test, copy || !cut); + if (cut >= 4) + put_unaligned_le32(cut | (2U << 27), copy); + KUNIT_EXPECT_LT_MSG(test, mt7932_rx_frame(copy, cut, &frame), 0, + "length %zu", cut); + kfree(copy); + } + /* The descriptor claims more than the DMA buffer holds. */ + copy = kmemdup(packet, RX_SIZE - 1, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, copy); + KUNIT_EXPECT_EQ(test, mt7932_rx_frame(copy, RX_SIZE - 1, &frame), -EMSGSIZE); + kfree(copy); + /* Optional groups (3 and 5) that do not fit. */ + rx_build(packet, rx_cipher(4) | BIT(13) | BIT(15)); + put_unaligned_le32((RX_HEADER + 20) | (2U << 27), packet); + KUNIT_EXPECT_EQ(test, mt7932_rx_frame(packet, RX_HEADER + 20, &frame), -EMSGSIZE); +} + +static void event_build(u8 *packet, u8 eid, u8 seq, u8 option) +{ + memset(packet, 0, 64); + put_unaligned_le32(64 | (7U << 27), packet); + put_unaligned_le16(40, packet + 24); + packet[28] = eid; + packet[29] = seq; + packet[30] = option; +} + +/* Only the event that answers a command completes it. */ +static void mt7932_event_reply_test(struct kunit *test) +{ + struct mt7932_event event; + u8 packet[64]; + int eid = mt7932_reply_eid(0xed, true); + + event_build(packet, 0xed, 9, 0); + KUNIT_ASSERT_EQ(test, mt7932_event_parse(packet, sizeof(packet), &event), 0); + KUNIT_EXPECT_TRUE(test, mt7932_event_matches(&event, 9, eid)); + /* Another sequence, the no-reply option bit, or nothing waiting. */ + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 10, eid)); + event.option = 4; + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, eid)); + event.option = 0; + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 0, eid)); + /* An unrelated event that happens to carry the waiting sequence. */ + event_build(packet, 0x99, 9, 0); + KUNIT_ASSERT_EQ(test, mt7932_event_parse(packet, sizeof(packet), &event), 0); + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, eid)); + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, mt7932_reply_eid(0x8a, true))); + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, mt7932_reply_eid(0x03, false))); + /* The ROM's patch semaphore answers with its own event only. */ + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, mt7932_reply_eid(0x10, false))); + event_build(packet, 0x04, 9, 0); + KUNIT_ASSERT_EQ(test, mt7932_event_parse(packet, sizeof(packet), &event), 0); + KUNIT_EXPECT_TRUE(test, mt7932_event_matches(&event, 9, mt7932_reply_eid(0x10, false))); + event_build(packet, 0x01, 9, 0); + KUNIT_ASSERT_EQ(test, mt7932_event_parse(packet, sizeof(packet), &event), 0); + KUNIT_EXPECT_FALSE(test, mt7932_event_matches(&event, 9, mt7932_reply_eid(0x10, false))); +} + +static void mt7932_event_reply_eid_test(struct kunit *test) +{ + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0xed, false), 0xed); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0xed, true), 0xed); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x8a, true), 0xec); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0xbf, true), 0xbf); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x03, false), 0x02); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x01, false), 0x01); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x05, false), 0x01); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x10, false), 0x04); + KUNIT_EXPECT_EQ(test, mt7932_reply_eid(0x50, false), -1); +} + +static struct kunit_case mt7932_input_test_cases[] = { + KUNIT_CASE(mt7932_rx_decrypted_test), + KUNIT_CASE(mt7932_rx_plaintext_test), + KUNIT_CASE(mt7932_rx_mismatch_test), + KUNIT_CASE(mt7932_rx_integrity_test), + KUNIT_CASE(mt7932_rx_short_test), + KUNIT_CASE(mt7932_event_reply_test), + KUNIT_CASE(mt7932_event_reply_eid_test), + {} +}; + +static struct kunit_suite mt7932_input_test_suite = { + .name = "mt7932-input", + .test_cases = mt7932_input_test_cases, +}; + +kunit_test_suite(mt7932_input_test_suite); diff --git a/drivers/net/wireless/mediatek/mt7932/link.c b/drivers/net/wireless/mediatek/mt7932/link.c index 3360e8c18c0cbe..1d6ef0596de367 100644 --- a/drivers/net/wireless/mediatek/mt7932/link.c +++ b/drivers/net/wireless/mediatek/mt7932/link.c @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include "mt7932.h" static int mt_peer_reserve(struct mt7932 *m); @@ -71,11 +71,17 @@ void mt_rf_fail_locked(struct mt7932 *m, int error) complete_all(&m->assoc_done); complete_all(&m->discovery_done); } else if (m->connected) { - /* Hardware ownership is uncertain: report failure, retain the peer. */ + /* Hardware ownership is uncertain: report failure, retain the peer. + * A connect worker still publishing the connection reports this + * after it, so that cfg80211 sees the two in order. + */ m->connected = false; m->disconnecting = true; - cfg80211_disconnected(m->netdev, WLAN_REASON_UNSPECIFIED, - NULL, 0, true, GFP_ATOMIC); + if (m->connect_publishing) + m->connect_lost = true; + else + cfg80211_disconnected(m->netdev, WLAN_REASON_UNSPECIFIED, + NULL, 0, true, GFP_ATOMIC); } if (m->scan_request) { m->scan_aborted = true; @@ -113,11 +119,11 @@ void mt_link_event(struct mt7932 *m, const struct mt7932_event *event) selector = get_unaligned_le32(event->packet + 40); body = event->packet + 48; length = event->length - 48; - dev_info(&m->pdev->dev, "LINK_EVENT: selector=%02x length=%zu\n", selector, length); + dev_dbg(&m->pdev->dev, "LINK_EVENT: selector=%02x length=%zu\n", selector, length); if (selector == 0x44) { if (length < 5 || (!m->connecting && !m->connected && !m->disconnecting)) return; - dev_info(&m->pdev->dev, "DISCONNECT_EVENT: metadata=%u deferred=%u reason-index=%u\n", + dev_dbg(&m->pdev->dev, "DISCONNECT_EVENT: metadata=%u deferred=%u reason-index=%u\n", body[0], body[3], body[4]); if (m->disconnect_seen) return; @@ -179,18 +185,18 @@ void mt_link_event(struct mt7932 *m, const struct mt7932_event *event) * rate-control setting. Bounds were checked by the exporter. */ if (he) - dev_info(&m->pdev->dev, - "ASSOC_HE_CAPABILITIES: epoch=%u rx-map=%04x tx-map=%04x\n", - m->connection_generation, - get_unaligned_le16(he->data + 18), - get_unaligned_le16(he->data + 20)); + dev_dbg(&m->pdev->dev, + "ASSOC_HE_CAPABILITIES: epoch=%u rx-map=%04x tx-map=%04x\n", + m->connection_generation, + get_unaligned_le16(he->data + 18), + get_unaligned_le16(he->data + 20)); } break; } case 0x4d: if (length >= 20) - dev_info(&m->pdev->dev, "ASSOC_START: band=%u width=%u channel=%u extension=%u\n", - get_unaligned_le32(body + 12), body[16], body[17], body[19]); + dev_dbg(&m->pdev->dev, "ASSOC_START: band=%u width=%u channel=%u extension=%u\n", + get_unaligned_le32(body + 12), body[16], body[17], body[19]); if (!mt_channel_event(m, body, length)) { m->connect_error = -EPROTO; complete(&m->assoc_done); @@ -199,9 +205,10 @@ void mt_link_event(struct mt7932 *m, const struct mt7932_event *event) break; case 0x40: if (length >= 52) - dev_info(&m->pdev->dev, "ASSOC_REPORT: status=%u band=%u channel=%u bss=%u bmc=%u station=%u peer=%u cipher=%08x\n", - get_unaligned_le16(body), body[11], body[12], body[16], body[17], body[20], body[21], - get_unaligned_le32(body + 48)); + dev_dbg(&m->pdev->dev, + "ASSOC_REPORT: status=%u band=%u channel=%u bss=%u bmc=%u station=%u peer=%u cipher=%08x\n", + get_unaligned_le16(body), body[11], body[12], body[16], body[17], + body[20], body[21], get_unaligned_le32(body + 48)); /* OPEN reports retain cipher as numeric metadata only; it neither * installs a key nor changes the immutable requested security mode. */ @@ -215,12 +222,13 @@ void mt_link_event(struct mt7932 *m, const struct mt7932_event *event) m->peer_valid = true; /* HOST decoder confirms the RSN suite namespace, not request enum10. */ m->event_cipher = get_unaligned_le32(body + 48); - dev_info(&m->pdev->dev, "ASSOCIATED: peer=%u station=%u cipher-raw=%u\n", - m->peer_wtbl, body[20], m->event_cipher); + dev_dbg(&m->pdev->dev, "ASSOCIATED: peer=%u station=%u cipher-raw=%u\n", + m->peer_wtbl, body[20], m->event_cipher); break; case 0x42: if (length >= 2) - dev_info(&m->pdev->dev, "CONNECT_REPORT: status=%u peer-owned=%u\n", get_unaligned_le16(body), m->peer_valid); + dev_dbg(&m->pdev->dev, "CONNECT_REPORT: status=%u peer-owned=%u\n", + get_unaligned_le16(body), m->peer_valid); if (length < 2 || get_unaligned_le16(body) || !m->peer_valid) m->connect_error = -ECONNREFUSED; complete(&m->assoc_done); @@ -283,8 +291,12 @@ static int mt_retire_connection(struct mt7932 *m) return -ESHUTDOWN; netif_carrier_off(m->netdev); netif_tx_disable(m->netdev); - /* ndo_start_xmit rechecks the closed admission gate under data_lock. */ + /* ndo_start_xmit rechecks the closed admission gate under data_lock. + * The queue may have stopped in the middle of a burst; hand the device + * what it already holds so that the drain below can complete. + */ spin_lock_irqsave(&m->data_lock, flags); + mt_data_kick_locked(m); spin_unlock_irqrestore(&m->data_lock, flags); flush_work(&m->cal_work); mutex_lock(&m->command_mutex); @@ -350,8 +362,9 @@ static int mt_retire_connection(struct mt7932 *m) m->link_failed = true; /* uncertain ownership: recovery required */ spin_unlock_irqrestore(&m->response_lock, flags); mutex_unlock(&m->command_mutex); - dev_info(&m->pdev->dev, "CONNECTION_RETIRED: status=%d%s\n", ret, - ret ? "; recovery required, DMA ownership retained" : "; old TX drained, six IDs released"); + dev_dbg(&m->pdev->dev, "CONNECTION_RETIRED: status=%d%s\n", ret, + ret ? "; recovery required, DMA ownership retained" : + "; old TX drained, six IDs released"); return ret; } @@ -383,7 +396,8 @@ void mt_connect_work(struct work_struct *work) struct cfg80211_connect_resp_params response = {}; u8 *body, mac[8] = {}, queue[32]; unsigned long flags; - bool submitted = false, peer_owned = false; + bool submitted = false, peer_owned = false, cancelled; + u16 reason; int ret; mutex_lock(&m->command_mutex); @@ -396,13 +410,13 @@ void mt_connect_work(struct work_struct *work) goto free; } if (!m->connect_bss) { - dev_info(&m->pdev->dev, "CONNECT_DISCOVERY: refreshing missing scan candidate\n"); + dev_dbg(&m->pdev->dev, "CONNECT_DISCOVERY: refreshing missing scan candidate\n"); ret = mt_connect_discover(m); if (ret) goto free; } if (m->connect_bss->channel->flags & (IEEE80211_CHAN_DISABLED | IEEE80211_CHAN_NO_IR | - IEEE80211_CHAN_NO_20MHZ)) { + IEEE80211_CHAN_NO_20MHZ | IEEE80211_CHAN_RADAR)) { ret = -EINVAL; goto free; } @@ -418,7 +432,7 @@ void mt_connect_work(struct work_struct *work) goto free; /* Initial station policy: framework M=0 branch and no optional flags. * This yields flag2 independently of credentials; no scan-result count - * belongs in this field. The recovered framework uses listen interval20. + * belongs in this field. The original driver uses listen interval 20. * Other platform flag policies remain unsupported, not guessed mappings. */ ret = mt7932_connect_body(body, 856, m->connect_ssid, m->connect_ssid_length, @@ -442,7 +456,8 @@ void mt_connect_work(struct work_struct *work) memzero_explicit(m->connect_pmk, sizeof(m->connect_pmk)); if (ret) goto report; - dev_info(&m->pdev->dev, "CONNECT_SUBMITTED: BSSID=%pM channel=%u\n", m->connect_bssid, m->connect_channel); + dev_dbg(&m->pdev->dev, "CONNECT_SUBMITTED: BSSID=%pM channel=%u\n", + m->connect_bssid, m->connect_channel); /* Genuine D7 requests must remain serviceable during asynchronous waits. */ mutex_unlock(&m->command_mutex); ret = wait_for_completion_timeout(&m->assoc_start, msecs_to_jiffies(5000)) ? 0 : -ETIMEDOUT; @@ -458,8 +473,8 @@ void mt_connect_work(struct work_struct *work) ret = ret ?: READ_ONCE(m->connect_error); if (ret) goto report; - dev_info(&m->pdev->dev, "ASSOCIATION_CALIBRATED: primary=%u center=%u\n", - m->connect_channel, m->connect_center); + dev_dbg(&m->pdev->dev, "ASSOCIATION_CALIBRATED: primary=%u center=%u\n", + m->connect_channel, m->connect_center); mt7932_peer_queue(queue, m->peer_ids.station[0], true); ret = mt_request(m, 0x6b, true, true, false, queue, sizeof(queue)); if (ret) @@ -498,7 +513,10 @@ void mt_connect_work(struct work_struct *work) } /* Serialize the final outcome with cancellation. In particular, a late * firmware success must not publish carrier/authorization after .deauth. - * cfg80211 queues these notifications; GFP_ATOMIC keeps this IRQ-safe. + * The cfg80211 calls may take bottom-half locks, so they are made after + * response_lock is dropped. command_mutex keeps the disconnect worker, + * which retires the connection before reporting its end, behind them; + * an RF failure in the meantime leaves its report to this worker. */ spin_lock_irqsave(&m->response_lock, flags); if (!ret && (m->connect_error || !mt_rf_allowed(m))) { @@ -533,34 +551,54 @@ void mt_connect_work(struct work_struct *work) m->station_connected = jiffies; m->station_signal_valid = false; m->station_rx_rate_valid = false; - m->station_tx_rate_valid = false; spin_unlock(&m->data_lock); + mt_cqm_reset(m); } m->connected = !ret; m->disconnecting = false; - if (m->connect_cancelled) { - if (response.links[0].bss) - cfg80211_put_bss(m->wiphy, response.links[0].bss); - cfg80211_disconnected(m->netdev, m->disconnect_reason, NULL, 0, true, GFP_ATOMIC); - } else { - cfg80211_connect_done(m->netdev, &response, GFP_ATOMIC); - } + cancelled = m->connect_cancelled; + reason = m->disconnect_reason; + m->connect_publishing = true; if (!ret) { schedule_work(&m->power_work); if (READ_ONCE(m->data_ready)) { netif_carrier_on(m->netdev); netif_wake_queue(m->netdev); } - if (m->connect_open) { - dev_info(&m->pdev->dev, "OPEN_CONNECTED: owned peer, firmware completion; Ethernet=%u\n", m->data_ready); - } else { - cfg80211_port_authorized(m->netdev, m->connect_bssid, NULL, 0, GFP_ATOMIC); - dev_info(&m->pdev->dev, "WPA2_AUTHORIZED: firmware handshake complete; Ethernet=%u\n", m->data_ready); - } + } + spin_unlock_irqrestore(&m->response_lock, flags); + + if (cancelled) { + if (response.links[0].bss) + cfg80211_put_bss(m->wiphy, response.links[0].bss); + cfg80211_disconnected(m->netdev, reason, NULL, 0, true, GFP_KERNEL); } else { - dev_err(&m->pdev->dev, "CONNECT_FAILED: %d; recovery-required=%u\n", ret, m->link_failed); + cfg80211_connect_done(m->netdev, &response, GFP_KERNEL); + } + if (!ret && !m->connect_open) + cfg80211_port_authorized(m->netdev, m->connect_bssid, NULL, 0, GFP_KERNEL); + + spin_lock_irqsave(&m->response_lock, flags); + m->connect_publishing = false; + if (m->connect_lost) { + m->connect_lost = false; + cfg80211_disconnected(m->netdev, WLAN_REASON_UNSPECIFIED, + NULL, 0, true, GFP_ATOMIC); } spin_unlock_irqrestore(&m->response_lock, flags); + if (!ret && m->connect_open) + dev_dbg(&m->pdev->dev, + "OPEN_CONNECTED: owned peer, firmware completion; Ethernet=%u\n", + READ_ONCE(m->data_ready)); + else if (!ret) + dev_dbg(&m->pdev->dev, + "WPA2_AUTHORIZED: firmware handshake complete; Ethernet=%u\n", + READ_ONCE(m->data_ready)); + else if (cancelled) + dev_dbg(&m->pdev->dev, "CONNECT_CANCELLED: %d\n", ret); + else + dev_err(&m->pdev->dev, "CONNECT_FAILED: %d; recovery-required=%u\n", + ret, READ_ONCE(m->link_failed)); mutex_unlock(&m->command_mutex); } @@ -597,9 +635,11 @@ int mt_connect(struct wiphy *wiphy, struct net_device *netdev, mt_retry_missing_policy(m); if (!mt_rf_allowed(m) || !m->bss_active || netdev != m->netdev) return -EAGAIN; - dev_info_ratelimited(&m->pdev->dev, "CONNECT_REQUEST: WPA=%u pairwise-count=%d group=%08x AKM-count=%d PMK-present=%u MFP=%u IE-bytes=%zu\n", - params->crypto.wpa_versions, params->crypto.n_ciphers_pairwise, params->crypto.cipher_group, - params->crypto.n_akm_suites, !!params->crypto.psk, params->mfp, params->ie_len); + dev_dbg(&m->pdev->dev, + "CONNECT_REQUEST: WPA=%u pairwise-count=%d group=%08x AKM-count=%d PMK-present=%u MFP=%u IE-bytes=%zu\n", + params->crypto.wpa_versions, params->crypto.n_ciphers_pairwise, + params->crypto.cipher_group, params->crypto.n_akm_suites, + !!params->crypto.psk, params->mfp, params->ie_len); if (!params->ssid_len || params->ssid_len > 32) return -EINVAL; if (!open && (params->crypto.wpa_versions != NL80211_WPA_VERSION_2 || @@ -643,8 +683,6 @@ int mt_connect(struct wiphy *wiphy, struct net_device *netdev, m->connect_center = 0; memset(&m->connect_chandef, 0, sizeof(m->connect_chandef)); m->connection_generation++; - m->station_rate_query_failed = false; - m->station_rate_query_time = jiffies - HZ; m->connect_cancelled = false; m->peer_valid = false; m->connecting = true; @@ -677,8 +715,9 @@ void mt_disconnect_work(struct work_struct *work) spin_lock_irqsave(&m->response_lock, flags); m->disconnecting = false; cfg80211_disconnected(m->netdev, reason, NULL, 0, local, GFP_ATOMIC); + mt_reg_apply_deferred_locked(m); spin_unlock_irqrestore(&m->response_lock, flags); - dev_info(&m->pdev->dev, "DISCONNECTED: teardown=%d local=%u\n", ret, local); + dev_dbg(&m->pdev->dev, "DISCONNECTED: teardown=%d local=%u\n", ret, local); } int mt_disconnect(struct wiphy *wiphy, struct net_device *netdev, u16 reason) diff --git a/drivers/net/wireless/mediatek/mt7932/mt7932.c b/drivers/net/wireless/mediatek/mt7932/mt7932.c index 3e783ec85dd916..b90c262a92177b 100644 --- a/drivers/net/wireless/mediatek/mt7932/mt7932.c +++ b/drivers/net/wireless/mediatek/mt7932/mt7932.c @@ -1,12 +1,16 @@ // SPDX-License-Identifier: GPL-2.0-only /* J700 firmware-owned Wi-Fi: native transport and cfg80211 station support. + * Covers firmware download and the NIC capability exchange. * All addresses here are BAR0 offsets, never chip/CPU physical addresses. */ -#include #include +#include #include "mt7932.h" +#include +#include + static int mt_ring_alloc(struct mt7932 *m, struct mt7932_ring *q, unsigned int n, bool rx, unsigned int count); static void mt_rx_post(struct mt7932_ring *q, unsigned int index); @@ -37,22 +41,6 @@ static int mt_initialize(struct mt7932 *m); static int mt_probe(struct pci_dev *pdev, const struct pci_device_id *id); static void mt_shutdown(struct pci_dev *pdev); -static bool legacy_irq_test; -module_param(legacy_irq_test, bool, 0444); -MODULE_PARM_DESC(legacy_irq_test, "Explicit diagnostic use of standard PCI INTx instead of MSI"); -static bool poll_test; -module_param(poll_test, bool, 0444); -MODULE_PARM_DESC(poll_test, "Diagnostic ROM/firmware polling with endpoint IRQ generation masked"); -static bool reset_test; -module_param(reset_test, bool, 0444); -MODULE_PARM_DESC(reset_test, "Qualify one FLR and fresh firmware epoch before publishing an interface, retaining old DMA"); - -bool mt_transport_polled(void) -{ - return poll_test; -} - - bool mt_rf_allowed(struct mt7932 *m) { return !READ_ONCE(m->stopping) && READ_ONCE(m->rf_ready) && !READ_ONCE(m->reg_pending) && @@ -181,8 +169,8 @@ static void mt_receive(struct mt7932 *m, const u8 *packet, size_t length) ret = mt7932_cal_null(&m->cal_state, packet + 36, event.length >= 36 ? event.length - 36 : 0, m->smart_version); - dev_info(&m->pdev->dev, "CAL_EVENT: seq=%u length=%zu result=%d credits=%u/%u\n", - event.seq, event.length, ret, m->cal_state.received, m->cal_state.expected); + dev_dbg(&m->pdev->dev, "CAL_EVENT: seq=%u length=%zu result=%d credits=%u/%u\n", + event.seq, event.length, ret, m->cal_state.received, m->cal_state.expected); if (ret < 0) mt_rf_fail_locked(m, ret); if (ret) @@ -208,15 +196,16 @@ static void mt_receive(struct mt7932 *m, const u8 *packet, size_t length) memcpy(m->cal_requests[at], packet + 36, 16); m->cal_request_seq[at] = event.seq; m->cal_request_count++; - dev_info(&m->pdev->dev, "CAL_REQUEST: seq=%u words=%08x/%08x/%08x/%08x queued=%u\n", - event.seq, get_unaligned_le32(packet + 36), get_unaligned_le32(packet + 40), - get_unaligned_le32(packet + 44), get_unaligned_le32(packet + 48), m->cal_request_count); + dev_dbg(&m->pdev->dev, "CAL_REQUEST: seq=%u words=%08x/%08x/%08x/%08x queued=%u\n", + event.seq, get_unaligned_le32(packet + 36), + get_unaligned_le32(packet + 40), get_unaligned_le32(packet + 44), + get_unaligned_le32(packet + 48), m->cal_request_count); if (READ_ONCE(m->rf_ready)) schedule_work(&m->cal_work); } goto unlock; } - if (mt7932_event_matches(&event, m->waiting) && !m->reply_length) { + if (mt7932_event_matches(&event, m->waiting, m->waiting_eid) && !m->reply_length) { memcpy(m->reply, packet, event.length); m->reply_length = event.length; complete(&m->response); @@ -247,12 +236,17 @@ static void mt_rx_drain(struct mt7932 *m, struct mt7932_ring *q) if (READ_ONCE(m->ram_config) && length >= 33 && length <= MT7932_RX_CAPACITY) { u8 *packet = q->buffers + q->tail * RX_STRIDE; - /* Source-specific RAM-config acknowledgment before RX0 refill. */ + /* + * Source-specific RAM-config acknowledgment before RX0 + * refill. mt_service() has already acknowledged this + * pass's causes, RX0 (bit 0) included, and restores the + * mask once every ring is drained. Leave bit 0 alone, so + * that an RX0 completion arriving now keeps its status, + * and leave the mask alone while the rings are serviced. + */ if (q == &m->rx[0] && packet[28] == 1 && packet[29] == m->waiting && !packet[30] && packet[32] == 1) { - mt_write(m, W + 0x204, 0); - mt_write(m, W + 0x200, 0x0c000001); - mt_write(m, W + 0x204, poll_test ? 0 : IRQ_MASK); + mt_write(m, W + 0x200, 0x0c000000); mt_write(m, W + 0x1f4, 0xffff); } } @@ -283,17 +277,28 @@ static void mt_service(struct mt7932 *m) u32 causes = mt_read(m, W + 0x200) & m->irq_mask; unsigned int i; - mt_data_clean(m); - for (i = 0; i < ARRAY_SIZE(m->rx); i++) - if (m->rx[i].count) - mt_rx_drain(m, &m->rx[i]); + /* + * Acknowledge before servicing the rings. A completion that lands + * while they are being drained then sets its status bit again and + * raises a new interrupt once the mask is restored, instead of being + * cleared by a late acknowledgment and left waiting for an unrelated + * interrupt. Read the status back so that the acknowledgment has + * reached the device, and every descriptor it covers is visible, + * before the rings are read. + */ if (causes & BIT(29)) { u32 sw = mt_read(m, W + 0x1f0); mt_write(m, W + 0x1f0, sw); } mt_write(m, W + 0x200, causes); - if (READ_ONCE(m->running) && !poll_test) + mt_read(m, W + 0x200); + + mt_data_clean(m); + for (i = 0; i < ARRAY_SIZE(m->rx); i++) + if (m->rx[i].count) + mt_rx_drain(m, &m->rx[i]); + if (READ_ONCE(m->running)) mt_write(m, W + 0x204, m->irq_mask); } @@ -372,60 +377,28 @@ int mt_request_ext(struct mt7932 *m, u8 cid, u8 ext, bool runtime, bool set, boo reinit_completion(&m->response); spin_lock_irqsave(&m->response_lock, flags); m->waiting = wait ? m->seq : 0; + m->waiting_eid = mt7932_reply_eid(cid, runtime); m->reply_length = 0; spin_unlock_irqrestore(&m->response_lock, flags); ret = mt_submit(m, q, ret); - if (!ret && wait) { - if (poll_test) { - unsigned long deadline = jiffies + msecs_to_jiffies(5000); - - do { - mt_service(m); - if (completion_done(&m->response)) - break; - usleep_range(100, 200); - } while (time_before(jiffies, deadline)); - if (!completion_done(&m->response)) - ret = -ETIMEDOUT; - } else if (!wait_for_completion_timeout(&m->response, msecs_to_jiffies(5000))) { - ret = -ETIMEDOUT; - } - } + if (!ret && wait && + !wait_for_completion_timeout(&m->response, msecs_to_jiffies(5000))) + ret = -ETIMEDOUT; if (READ_ONCE(m->stopping)) ret = -ESHUTDOWN; if (ret == -ETIMEDOUT) { - /* Diagnostic only: never turn a polled late reply into IRQ success. */ unsigned int i; - u16 command; - u32 mask, pending; - unsigned int cap = pci_find_capability(m->pdev, PCI_CAP_ID_MSI); - - pci_read_config_word(m->pdev, PCI_COMMAND, &command); - pci_read_config_dword(m->pdev, cap + PCI_MSI_MASK_64, &mask); - pci_read_config_dword(m->pdev, cap + PCI_MSI_PENDING_64, &pending); - dev_info(&m->pdev->dev, "timeout PCI command=%04x MSI mask=%08x pending=%08x\n", - command, mask, pending); - - dev_info(&m->pdev->dev, "timeout gates MAC=%08x WFDMA=%08x SW=%08x\n", - mt_read(m, 0x10188), mt_read(m, W + 0x204), mt_read(m, W + 0x1f4)); + + /* + * Collect a reply that arrived without its interrupt, so that the + * error below reports it. It does not turn the timeout into success. + */ disable_irq(pci_irq_vector(m->pdev, 0)); - for (i = 0; i < ARRAY_SIZE(m->rx); i++) { - struct mt7932_ring *rx = &m->rx[i]; - - if (!rx->count) - continue; - dev_info(&m->pdev->dev, "timeout RX%u ctrl=%08x CPU=%u DMA=%u\n", i, - le32_to_cpu(rx->desc[rx->tail].control), - mt_read(m, rx->reg + 8), mt_read(m, rx->reg + 12)); - if (le32_to_cpu(rx->desc[rx->tail].control) & DONE) { - dma_rmb(); - print_hex_dump(KERN_INFO, "ROM RX prefix: ", DUMP_PREFIX_OFFSET, - 16, 1, rx->buffers + rx->tail * RX_STRIDE, 40, false); - } - mt_rx_drain(m, rx); - } - dev_info(&m->pdev->dev, "late polled reply length=%zu eid=%02x (not IRQ success)\n", - m->reply_length, m->reply_length > 28 ? m->reply[28] : 0); + for (i = 0; i < ARRAY_SIZE(m->rx); i++) + if (m->rx[i].count) + mt_rx_drain(m, &m->rx[i]); + dev_dbg(&m->pdev->dev, "late reply length=%zu eid=%02x\n", + m->reply_length, m->reply_length > 28 ? m->reply[28] : 0); enable_irq(pci_irq_vector(m->pdev, 0)); } spin_lock_irqsave(&m->response_lock, flags); @@ -468,7 +441,7 @@ static int mt_ownership(struct mt7932 *m) udelay(2); if ((mt_read(m, 0xfe250) >> 16) != 0x7000) return -EIO; - dev_info(&m->pdev->dev, "DRIVER_OWN_OK power=%08x\n", mt_read(m, 0xe00f0)); + dev_dbg(&m->pdev->dev, "DRIVER_OWN_OK power=%08x\n", mt_read(m, 0xe00f0)); return 0; } @@ -484,7 +457,7 @@ static int mt_dma_setup(struct mt7932 *m) u32 glo; mt_write(m, W + 0x204, 0); - mt_write(m, 0x10188, poll_test ? 0 : 0xff); + mt_write(m, 0x10188, 0xff); mt_rmw(m, W + 0x208, BIT(0) | BIT(2) | BIT(15) | BIT(21) | BIT(27) | BIT(28), 0); ret = mt_poll(m, W + 0x208, BIT(1) | BIT(3), 0, 100000); if (ret) @@ -578,9 +551,9 @@ static int mt_dma_setup(struct mt7932 *m) mt_write(m, W + 0x2f0, 0x8032800a); mt_rmw(m, 0x2120, 0, BIT(1)); mt_write(m, W + 0x1f4, 0xffff); - mt_write(m, W + 0x204, poll_test ? 0 : IRQ_MASK); - dev_info(&m->pdev->dev, "DMA_PUBLISHED glo=%08x TX17=%pad RX0=%pad IPC=%pad\n", - mt_read(m, W + 0x208), &m->tx[17].dma, &m->rx[0].dma, &m->ipc_dma); + mt_write(m, W + 0x204, IRQ_MASK); + dev_dbg(&m->pdev->dev, "DMA_PUBLISHED glo=%08x TX17=%pad RX0=%pad IPC=%pad\n", + mt_read(m, W + 0x208), &m->tx[17].dma, &m->rx[0].dma, &m->ipc_dma); return 0; } @@ -613,11 +586,11 @@ static int mt_rom_gate(struct mt7932 *m) if (m->reply_length < 40 || m->reply[28] != 2 || get_unaligned_le32(m->reply + 32) != addresses[i]) return -EPROTO; - dev_info(&m->pdev->dev, "ROM_READ_OK %08x=%08x seq=%u IRQ=%llu\n", - addresses[i], get_unaligned_le32(m->reply + 36), m->seq, m->interrupts); + dev_dbg(&m->pdev->dev, "ROM_READ_OK %08x=%08x seq=%u IRQ=%llu\n", + addresses[i], get_unaligned_le32(m->reply + 36), m->seq, m->interrupts); } - dev_info(&m->pdev->dev, "ROM_TRANSPORT_OK mode=%s IRQ=%llu (no firmware download or RF yet)\n", - poll_test ? "polled-test" : "interrupt", m->interrupts); + dev_dbg(&m->pdev->dev, "ROM_TRANSPORT_OK IRQ=%llu (no firmware download or RF yet)\n", + m->interrupts); return 0; } @@ -672,8 +645,8 @@ static int mt_download(struct mt7932 *m, const u8 *data, struct mt7932_region *r if (ret) return ret; ret = mt_scatter(m, data + r->offset, r->length); - dev_info(&m->pdev->dev, "%s region target=%08x length=%u scatter=%d\n", - ram ? "RAM" : "PATCH", r->target, r->length, ret); + dev_dbg(&m->pdev->dev, "%s region target=%08x length=%u scatter=%d\n", + ram ? "RAM" : "PATCH", r->target, r->length, ret); return ret; } @@ -707,6 +680,8 @@ static int mt_otp_byte(struct mt7932 *m, u32 address, u8 *value) if (m->reply_length < 40) return -EPROTO; *value = m->reply[36]; + dev_dbg(&m->pdev->dev, "OTP_REPLY: eid=%02x word=%08x address=%08x length=%zu\n", + m->reply[28], get_unaligned_le32(m->reply + 32), address, m->reply_length); return 0; } @@ -737,8 +712,8 @@ static int mt_capability_gate(struct mt7932 *m) count = mt7932_capabilities(body, m->reply_length - 36); if (count < 0) { dev_err(&m->pdev->dev, "invalid capability reply len=%zu parse=%d\n", m->reply_length, count); - print_hex_dump(KERN_INFO, "NIC_CAP_REJECTED: ", DUMP_PREFIX_OFFSET, - 16, 1, m->reply, m->reply_length, false); + print_hex_dump_debug("NIC_CAP_REJECTED: ", DUMP_PREFIX_OFFSET, + 16, 1, m->reply, m->reply_length, false); return count; } for (i = 0; i < count; i++) { @@ -746,9 +721,9 @@ static int mt_capability_gate(struct mt7932 *m) u32 length = get_unaligned_le32(body + offset + 4); const u8 *value = body + offset + 8; - dev_info(&m->pdev->dev, "capability TLV type=%x bytes=%u\n", type, length); + dev_dbg(&m->pdev->dev, "capability TLV type=%x bytes=%u\n", type, length); if (type == 7) { - dev_info(&m->pdev->dev, "firmware MAC %pM\n", value); + dev_dbg(&m->pdev->dev, "firmware MAC %pM\n", value); mac = true; } else if (type == 8) { if (!value[4] || value[4] > 4) { @@ -757,13 +732,13 @@ static int mt_capability_gate(struct mt7932 *m) } memcpy(m->phy_cap, value, 12); m->antenna_mask = (1U << value[4]) - 1; - dev_info(&m->pdev->dev, "firmware PHY NSS=%u paths=%02x HT=%u VHT=%u HE=%u\n", - value[4], value[10], value[0], value[1], value[11]); + dev_dbg(&m->pdev->dev, "firmware PHY NSS=%u paths=%02x HT=%u VHT=%u HE=%u\n", + value[4], value[10], value[0], value[1], value[11]); phy = true; } else if (type == 0x38 && length) { m->runtime_cal = value[0]; m->runtime_cal_valid = true; - dev_info(&m->pdev->dev, "RUNTIME_CAL_CAPABILITY: %u\n", value[0]); + dev_dbg(&m->pdev->dev, "RUNTIME_CAL_CAPABILITY: %u\n", value[0]); } else if (type == 0x18 && length >= 4) { m->six_ghz = value[0]; m->six_ghz_valid = value[0] <= 1; @@ -776,10 +751,10 @@ static int mt_capability_gate(struct mt7932 *m) dev_err(&m->pdev->dev, "CAPABILITY_MISSING: MAC=%u PHY=%u\n", mac, phy); return -ENODATA; } - print_hex_dump(KERN_INFO, "NIC_CAP: ", DUMP_PREFIX_OFFSET, 16, 1, - m->reply, m->reply_length, false); - dev_info(&m->pdev->dev, "MCU_CAPABILITY_OK eid=%02x seq=%u (no calibration or RF yet)\n", - m->reply[28], m->seq); + print_hex_dump_debug("NIC_CAP: ", DUMP_PREFIX_OFFSET, 16, 1, + m->reply, m->reply_length, false); + dev_dbg(&m->pdev->dev, "MCU_CAPABILITY_OK eid=%02x seq=%u (no calibration or RF yet)\n", + m->reply[28], m->seq); return 0; } @@ -801,7 +776,7 @@ static int mt_config_gate(struct mt7932 *m) do { mt_tx_clean(&m->tx[17]); if (!m->tx[17].queued) { - dev_info(&m->pdev->dev, "BASIC_MLME_SUBMITTED: TX retired, no semantic ACK or RF startup\n"); + dev_dbg(&m->pdev->dev, "BASIC_MLME_SUBMITTED: TX retired, no semantic ACK or RF startup\n"); return 0; } usleep_range(100, 200); @@ -809,7 +784,6 @@ static int mt_config_gate(struct mt7932 *m) return -ETIMEDOUT; } - static int mt_board_gate(struct mt7932 *m) { const u32 addresses[] = {0x70, 0x160}; @@ -825,7 +799,7 @@ static int mt_board_gate(struct mt7932 *m) return ret; if (m->reply_length < 44 || m->reply[28] != 0xed || m->reply[32] != 0xc1) return -EPROTO; - dev_info(&m->pdev->dev, "CHIP_UID_QUERY_OK: 8 opaque bytes received, not published\n"); + dev_dbg(&m->pdev->dev, "CHIP_UID_QUERY_OK: 8 opaque bytes received, not published\n"); for (i = 0; i < ARRAY_SIZE(addresses); i++) { put_unaligned_le32(addresses[i], request); ret = mt_request_ext(m, 0xed, 1, true, false, true, request, sizeof(request)); @@ -843,8 +817,8 @@ static int mt_board_gate(struct mt7932 *m) candidate = !(module & 15) ? 2 : !board_byte ? 3 : 1; m->board_type = candidate; m->module_byte = module; - dev_info(&m->pdev->dev, "BOARD_CLASSIFICATION_CAPTURED: candidate=%u module=%02x board-byte=%02x status=%08x/%08x; no buffer branch selected\n", - candidate, module, board_byte, status[0], status[1]); + dev_dbg(&m->pdev->dev, "BOARD_CLASSIFICATION_CAPTURED: candidate=%u module=%02x board-byte=%02x status=%08x/%08x; no buffer branch selected\n", + candidate, module, board_byte, status[0], status[1]); return 0; } @@ -883,7 +857,7 @@ static int mt_precal_gate(struct mt7932 *m) return -EPROTO; } status = get_unaligned_le32(m->reply + 40); - dev_info(&m->pdev->dev, "PPR_SOURCE2_RESULT: status=%08x\n", status); + dev_dbg(&m->pdev->dev, "PPR_SOURCE2_RESULT: status=%08x\n", status); if (status) return -EREMOTEIO; memset(request, 0, sizeof(request)); @@ -893,17 +867,17 @@ static int mt_precal_gate(struct mt7932 *m) if (m->reply_length < 40 || m->reply[28] != 0xed || m->reply[32] != 0xaa) return -EPROTO; m->smart_version = m->reply[38] & 15; - dev_info(&m->pdev->dev, "SMART_VERSION_REPLY: eid=%02x ext=%02x version=%u\n", - m->reply[28], m->reply[32], m->smart_version); + dev_dbg(&m->pdev->dev, "SMART_VERSION_REPLY: eid=%02x ext=%02x version=%u\n", + m->reply[28], m->reply[32], m->smart_version); request[3] = 9; ret = mt_request(m, 0xbf, true, false, true, request, 20); if (ret) return ret; if (m->reply_length < 56) return -EPROTO; - dev_info(&m->pdev->dev, "CALTYPE_REPLY: eid=%02x ext=%02x H=%u P=%u O=%u V=%u marker=%08x\n", - m->reply[28], m->reply[32], m->runtime_cal, m->reply[37], - m->reply[44], m->reply[38], get_unaligned_le32(m->reply + 48)); + dev_dbg(&m->pdev->dev, "CALTYPE_REPLY: eid=%02x ext=%02x H=%u P=%u O=%u V=%u marker=%08x\n", + m->reply[28], m->reply[32], m->runtime_cal, m->reply[37], + m->reply[44], m->reply[38], get_unaligned_le32(m->reply + 48)); /* M=0/T=3 exact host predicate. Stage metadata only after acceptance; * do not select the legacy absent-marker D6 compatibility fallback. */ @@ -911,7 +885,7 @@ static int mt_precal_gate(struct mt7932 *m) m->reply[37] != 1 || m->reply[44] != 1) return -EPROTO; m->preload_version = m->reply[38]; - dev_info(&m->pdev->dev, "PRECAL_HANDSHAKE_OK: own calibration and RF policy still required\n"); + dev_dbg(&m->pdev->dev, "PRECAL_HANDSHAKE_OK: own calibration and RF policy still required\n"); return 0; } @@ -967,8 +941,8 @@ static int mt_firmware_gate(struct mt7932 *m) dev_err(&m->pdev->dev, "local input mediatek/mt7932/IZUBA_W7932_2.bin has invalid region %d\n", i); goto out; } - dev_info(&m->pdev->dev, "stock containers patch=%zu/%d regions RAM=%zu/%d regions\n", - patch->size, patches, ram->size, regions); + dev_dbg(&m->pdev->dev, "stock containers patch=%zu/%d regions RAM=%zu/%d regions\n", + patch->size, patches, ram->size, regions); phase = "patch semaphore command/reply"; put_unaligned_le32(2, request); ret = mt_command(m, 0x10, request, 4); @@ -995,7 +969,7 @@ static int mt_firmware_gate(struct mt7932 *m) goto out; mt_secure_release(m); } - dev_info(&m->pdev->dev, "PATCH_READY\n"); + dev_dbg(&m->pdev->dev, "PATCH_READY\n"); phase = "OTP command/profile"; ret = mt_otp_byte(m, 0x7a, &value); if (!ret) @@ -1050,7 +1024,7 @@ static int mt_firmware_gate(struct mt7932 *m) ret = mt_poll(m, 0xe00f0, 3, 3, 5000000); if (ret) goto out; - dev_info(&m->pdev->dev, "FIRMWARE_READY entry=%08x\n", entry); + dev_dbg(&m->pdev->dev, "FIRMWARE_READY entry=%08x\n", entry); phase = "runtime capability/transport gates"; ret = mt_runtime_start(m); out: @@ -1067,20 +1041,19 @@ void mt_transport_snapshot(struct mt7932 *m) unsigned int i; struct mt7932_ring *tx = &m->tx[17]; - dev_info(&m->pdev->dev, "TRANSPORT: interrupts=%llu packets=%llu pending=%08x mask=%08x TX17 head=%u tail=%u queued=%u DMA=%u\n", - m->interrupts, m->packets, mt_read(m, W + 0x200), mt_read(m, W + 0x204), - tx->head, tx->tail, tx->queued, mt_read(m, tx->reg + 12)); + dev_dbg(&m->pdev->dev, "TRANSPORT: interrupts=%llu packets=%llu pending=%08x mask=%08x TX17 head=%u tail=%u queued=%u DMA=%u\n", + m->interrupts, m->packets, mt_read(m, W + 0x200), mt_read(m, W + 0x204), + tx->head, tx->tail, tx->queued, mt_read(m, tx->reg + 12)); for (i = 0; i < ARRAY_SIZE(m->rx); i++) { struct mt7932_ring *rx = &m->rx[i]; if (rx->count) - dev_info(&m->pdev->dev, "TRANSPORT_RX%u: tail=%u producer=%u DMA=%u control=%08x\n", - i, rx->tail, mt_read(m, rx->reg + 8), mt_read(m, rx->reg + 12), - le32_to_cpu(READ_ONCE(rx->desc[rx->tail].control))); + dev_dbg(&m->pdev->dev, "TRANSPORT_RX%u: tail=%u producer=%u DMA=%u control=%08x\n", + i, rx->tail, mt_read(m, rx->reg + 8), mt_read(m, rx->reg + 12), + le32_to_cpu(READ_ONCE(rx->desc[rx->tail].control))); } } - static int mt_initialize(struct mt7932 *m) { int ret; @@ -1093,63 +1066,6 @@ static int mt_initialize(struct mt7932 *m) return ret; } -static int mt_reset_retained(struct mt7932 **epoch) -{ - struct mt7932 *old = *epoch, *fresh; - struct pci_dev *pdev = old->pdev; - int irq = pci_irq_vector(pdev, 0), ret; - - /* No interface/work producers have been published at this test gate. */ - WRITE_ONCE(old->running, false); - mt_write(old, W + 0x204, 0); - devm_free_irq(&pdev->dev, irq, old); - old->irq_requested = false; - ret = mt_ownership(old); - if (!ret) - ret = mt_dma_stop(old); - if (!ret) - ret = mt_function_reset(old); - if (ret) - return ret; - old->dma_owned = false; - if (old->module_pinned) { - old->module_pinned = false; - module_put(THIS_MODULE); - } - dev_info(&pdev->dev, "RESET_TEST: function reset complete; retaining old DMA\n"); - fresh = devm_kzalloc(&pdev->dev, sizeof(*fresh), GFP_KERNEL); - if (!fresh) - return -ENOMEM; - fresh->pdev = pdev; - fresh->bar = old->bar; - fresh->irq_mask = IRQ_MASK; - fresh->vectors_allocated = true; - ether_addr_copy(fresh->mac, old->mac); - init_completion(&fresh->response); - init_completion(&fresh->cal_response); - init_completion(&fresh->reset_retry); - mutex_init(&fresh->command_mutex); - spin_lock_init(&fresh->response_lock); - *epoch = fresh; - pci_set_drvdata(pdev, fresh); - ret = mt_ownership(fresh); - if (ret) - return ret; - ret = mt_poll(fresh, 0xf0140, BIT(4), BIT(4), 200000); - if (ret) - return ret; - mt_write(fresh, W + 0x204, 0); - ret = devm_request_threaded_irq(&pdev->dev, irq, mt_irq, mt_irq_thread, - IRQF_ONESHOT | IRQF_SHARED, "mt7932", fresh); - if (ret) - return ret; - fresh->irq_requested = true; - ret = mt_initialize(fresh); - if (!ret) - dev_info(&pdev->dev, "RESET_TEST: fresh ROM and firmware gates passed\n"); - return ret; -} - /* A failed PCI reset cannot be turned into successful removal: devres would * free still-owned DMA. Keep the remove callback (and therefore its device, * mappings and module text) alive. This explicit retry does not run hardware @@ -1221,7 +1137,7 @@ static void mt_remove(struct pci_dev *pdev) wiphy_free(m->wiphy); if (m->vectors_allocated) pci_free_irq_vectors(pdev); - dev_info(&pdev->dev, "REMOVE_COMPLETE: host users joined, PCI function reset, DMA retired\n"); + dev_dbg(&pdev->dev, "REMOVE_COMPLETE: host users joined, PCI function reset, DMA retired\n"); /* All coherent allocations/BAR mappings are devres-owned and now safe * to release. No default-domain ownership manipulation is necessary. */ @@ -1245,7 +1161,10 @@ static int mt_probe(struct pci_dev *pdev, const struct pci_device_id *id) ret = apple_piodma_radio_check(pdev); if (ret) return ret; - /* A remove/rescan may have recreated link state under the global policy. */ + /* + * Station operation was tested with ASPM and clock PM off on this link, + * and a remove/rescan may have restored them under the global policy. + */ ret = pci_disable_link_state(pdev, PCIE_LINK_STATE_ALL); if (ret) return ret; @@ -1283,7 +1202,7 @@ static int mt_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (ret) goto retain; mt_write(m, W + 0x204, 0); - ret = pci_alloc_irq_vectors(pdev, 1, 1, legacy_irq_test ? PCI_IRQ_INTX : PCI_IRQ_ALL_TYPES); + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES); if (ret < 0) goto retain; m->vectors_allocated = true; @@ -1292,30 +1211,7 @@ static int mt_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (ret) goto retain; m->irq_requested = true; - { - struct irq_data *irq = irq_get_irq_data(pci_irq_vector(pdev, 0)); - unsigned int cap = pci_find_capability(pdev, PCI_CAP_ID_MSI); - u16 control, command; - u32 lo, hi, data, mask; - - pci_read_config_word(pdev, PCI_COMMAND, &command); - pci_read_config_word(pdev, cap + PCI_MSI_FLAGS, &control); - pci_read_config_dword(pdev, cap + PCI_MSI_ADDRESS_LO, &lo); - pci_read_config_dword(pdev, cap + PCI_MSI_ADDRESS_HI, &hi); - pci_read_config_dword(pdev, cap + PCI_MSI_DATA_64, &data); - pci_read_config_dword(pdev, cap + PCI_MSI_MASK_64, &mask); - dev_dbg(&pdev->dev, "PCI command=%04x MSI cap=%x control=%04x addr=%08x:%08x data=%08x\n", - command, cap, control, hi, lo, data); - dev_dbg(&pdev->dev, "MSI mask=%08x MSI-X cap=%x chosen=%s\n", mask, - pci_find_capability(pdev, PCI_CAP_ID_MSIX), - pdev->msix_enabled ? "MSI-X" : pdev->msi_enabled ? "MSI" : "INTx"); - for (; irq; irq = irq->parent_data) - dev_dbg(&pdev->dev, "IRQ %u chip=%s hwirq=%lu\n", irq->irq, - irq->chip->name, irq->hwirq); - } ret = mt_initialize(m); - if (!ret && reset_test) - ret = mt_reset_retained(&m); if (!ret) ret = mt_register_regulatory_gate(m); if (!ret) @@ -1348,6 +1244,41 @@ static void mt_shutdown(struct pci_dev *pdev) /* Storage remains allocated through reset, even if the engine is stuck. */ } +/* + * Stopping the firmware for sleep would take the whole remove and probe + * sequence, so leave it running across suspend-to-idle. A saved state keeps + * the PCI core from changing the function's power state or command register, + * and the awake path keeps the PCIe controller powered until resume. + */ +static int mt_suspend(struct device *dev) +{ + if (!pm_suspend_no_platform()) + return -EOPNOTSUPP; + pci_save_state(to_pci_dev(dev)); + device_set_awake_path(dev); + return 0; +} + +/* + * Hibernation is refused. Its freeze phase does not call .suspend, so without + * this the firmware would keep writing to memory while the image is saved, + * and the restored kernel would resume with ring state the firmware of the + * booting kernel does not have. Supporting it needs DMA retirement and a full + * reinitialisation on restore. Returning an error here has no side effect to + * undo: the PM core aborts hibernation before this device changes state. + */ +static int mt_freeze(struct device *dev) +{ + dev_err(dev, "hibernation is not supported\n"); + return -EOPNOTSUPP; +} + +static const struct dev_pm_ops mt_pm_ops = { + .suspend = mt_suspend, + .freeze = mt_freeze, + .poweroff = mt_freeze, +}; + static const struct pci_device_id mt_ids[] = { { PCI_DEVICE(PCI_VENDOR_ID_MEDIATEK, 0x7932) }, {} @@ -1361,7 +1292,9 @@ static struct pci_driver mt_driver = { .remove = mt_remove, .shutdown = mt_shutdown, .driver.suppress_bind_attrs = true, + .driver.pm = pm_sleep_ptr(&mt_pm_ops), }; + /** * mt7932_fullmac_ready() - check that the Wi-Fi function is up * @pdev: function 0 of the radio diff --git a/drivers/net/wireless/mediatek/mt7932/mt7932.h b/drivers/net/wireless/mediatek/mt7932/mt7932.h index 89a58bd9c1b5d2..aae173b97cc134 100644 --- a/drivers/net/wireless/mediatek/mt7932/mt7932.h +++ b/drivers/net/wireless/mediatek/mt7932/mt7932.h @@ -2,13 +2,12 @@ #ifndef MT7932_H #define MT7932_H - +#include #include #include #include #include #include -#include #include #include #include @@ -31,7 +30,6 @@ #include "rate.h" #include "capabilities.h" - #define W 0xd4000 #define H 0xd6000 #define C 0xd7000 @@ -40,6 +38,8 @@ #define STRIDE 8192 #define RX_STRIDE ALIGN(MT7932_RX_CAPACITY, 64) +DECLARE_EWMA(mt7932_signal, 4, 4) + struct mt7932; int mt_dma_stop(struct mt7932 *m); int mt_function_reset(struct mt7932 *m); @@ -75,12 +75,16 @@ struct mt7932 { unsigned int cal_request_head, cal_request_count; spinlock_t response_lock; u8 seq, waiting; + /* Event that answers the waiting command; see mt7932_reply_eid(). */ + int waiting_eid; u8 mac[ETH_ALEN]; u8 reply[MT7932_RX_CAPACITY]; size_t reply_length; u64 interrupts, packets; bool running; bool stopping; + /* Probe finished initialization; the Bluetooth function may start. */ + bool ready; bool irq_requested, vectors_allocated, wiphy_registered; bool dma_owned, module_pinned; struct completion reset_retry; @@ -99,20 +103,24 @@ struct mt7932 { struct ieee80211_supported_band band2; struct cfg80211_chan_def connect_chandef; u8 connect_center; - struct ieee80211_channel channels[13]; - struct ieee80211_channel channels5[4]; + struct ieee80211_channel channels[MT7932_CHANNELS_2G]; + struct ieee80211_channel channels5[MT7932_CHANNELS_5G]; struct ieee80211_supported_band band5; u8 scan_band, scan_batch; struct ieee80211_rate rates[12]; struct work_struct startup_work; struct mutex command_mutex; bool startup_started, rf_ready, reg_pending, policy_failed, reg_retryable; - bool ready; bool interface_registered, interface_up; u32 reg_generation, reg_attempted; struct mt7932_reg_snapshot reg_desired; - /* Applied package exclusions; protected by RTNL. */ - DECLARE_BITMAP(policy_disabled, 17); + /* A relaxation kept from the firmware while associated; applied by the + * next pass once the connection has ended. Under response_lock. + */ + struct mt7932_reg_snapshot reg_deferred; + bool reg_deferral; + /* Channels the applied country package leaves out; under RTNL. */ + DECLARE_BITMAP(policy_disabled, MT7932_CHANNELS); struct net_device *netdev; struct wireless_dev wdev; struct cfg80211_scan_request *scan_request; @@ -144,6 +152,10 @@ struct mt7932 { u8 connect_ssid_length, connect_channel, peer_wtbl; bool connecting, connected, peer_valid, link_failed, connect_open; bool disconnecting, disconnect_seen, disconnect_event_received, disconnect_local; + /* The connect worker is reporting its outcome; an RF failure meanwhile + * sets connect_lost for the worker to report. Under response_lock. + */ + bool connect_publishing, connect_lost; u16 disconnect_reason; int connect_error; u32 event_cipher; @@ -157,14 +169,19 @@ struct mt7932 { struct rate_info station_rx_rate; unsigned long station_rx_rate_time; bool station_rx_rate_valid; - struct rate_info station_tx_rate; - unsigned long station_tx_rate_time, station_rate_query_time; - bool station_tx_rate_valid, station_rate_query_failed; + /* Connection quality monitor, protected by response_lock. */ + struct ewma_mt7932_signal cqm_signal; + unsigned long cqm_sample_time; + s32 cqm_threshold, cqm_last_event; + u32 cqm_hysteresis; + u8 cqm_samples; u8 *data_headers, *data_payloads; dma_addr_t data_headers_dma, data_payloads_dma; bool data_live[512], data_token[512], data_retired[512]; struct mt7932_duplicate duplicates[17]; u64 tx_released, rx_ethernet; + /* TX0 descriptors published since the last doorbell; data_lock. */ + unsigned int data_unkicked; bool data_ready; bool bss_absent; u8 bss_quota; @@ -175,6 +192,7 @@ struct mt7932 { bool mt_rf_allowed(struct mt7932 *m); void mt_retry_missing_policy(struct mt7932 *m); +void mt_reg_apply_deferred_locked(struct mt7932 *m); int mt_net_open(struct net_device *netdev); int mt_request_input(struct mt7932 *m, const struct firmware **fw, const char *name); /* response_lock held; closes RF admission and reports terminal host state. */ @@ -197,6 +215,7 @@ int mt_calibration_gate(struct mt7932 *m); int mt_cal_drain_requests(struct mt7932 *m, const struct firmware *oca); void mt_cal_input_error(struct mt7932 *m, int error, unsigned int tag, const char *stage); void mt_data_clean_locked(struct mt7932 *m); +void mt_data_kick_locked(struct mt7932 *m); void mt_data_clean(struct mt7932 *m); void mt_bss_presence(struct mt7932 *m, const struct mt7932_event *event); void mt_data_complete(struct mt7932 *m, const u8 *packet, size_t length); @@ -230,12 +249,13 @@ int mt_get_station(struct wiphy *wiphy, struct wireless_dev *wdev, const u8 *mac, struct station_info *sinfo); int mt_dump_station(struct wiphy *wiphy, struct wireless_dev *wdev, int idx, u8 *mac, struct station_info *sinfo); +int mt_set_cqm_rssi_config(struct wiphy *wiphy, struct net_device *netdev, + s32 rssi_thold, u32 rssi_hyst); +void mt_cqm_reset(struct mt7932 *m); +void mt_cqm_signal(struct mt7932 *m, int signal); int mt_set_power_mgmt(struct wiphy *wiphy, struct net_device *netdev, bool enabled, int timeout); void mt_power_work(struct work_struct *work); int mt_power_init(struct mt7932 *m); -void mt_station_update_rate(struct mt7932 *m, const u8 *mac); - -bool mt_transport_polled(void); #endif diff --git a/drivers/net/wireless/mediatek/mt7932/packet.h b/drivers/net/wireless/mediatek/mt7932/packet.h index 303cc87fef8b04..d7480c81872b26 100644 --- a/drivers/net/wireless/mediatek/mt7932/packet.h +++ b/drivers/net/wireless/mediatek/mt7932/packet.h @@ -2,6 +2,8 @@ #ifndef MT7932_PACKET_H #define MT7932_PACKET_H +#include + #include "protocol.h" #include "channels.h" @@ -10,12 +12,22 @@ struct mt7932_rx_frame { size_t length; u32 timestamp, rcpi, rxv; u16 frame_control, sequence_control, qos_control, wtbl; - u8 channel, amsdu; + u8 channel, amsdu, sec_mode; bool translated, group4, signal_valid, rate_valid, unicast; + bool cipher_mismatch; }; -/* NEO_PACKET_RX_GROUPS_AND_DATA_TXP_CONTRACT. Strict non-monitor subset; - * descriptor groups are ordered4,1,2,3,5, not by numerical group identifier. +/* RX descriptor word1: the cipher the hardware decrypted the frame with (0 + * when it did not decrypt it), and the cipher and cipher-length mismatch + * flags. The layout is the connac2 one. + */ +#define MT7932_RXD1_SEC_MODE_SHIFT 16 +#define MT7932_RXD1_SEC_MODE_MASK 0x1f +#define MT7932_RXD1_CIPHER_MISMATCH BIT(23) +#define MT7932_RXD1_CIPHER_LEN_MISMATCH BIT(24) + +/* Strict non-monitor subset of the RX descriptor; its groups are ordered + * 4, 1, 2, 3, 5, not by numerical group identifier. */ static inline int mt7932_rx_frame(const u8 *packet, size_t size, struct mt7932_rx_frame *out) { @@ -54,6 +66,9 @@ static inline int mt7932_rx_frame(const u8 *packet, size_t size, struct mt7932_r out->wtbl = w1 & 0x3ff; out->unicast = ((w3 >> 16) & 3) == 1; out->amsdu = w4 & 3; + out->sec_mode = (w1 >> MT7932_RXD1_SEC_MODE_SHIFT) & MT7932_RXD1_SEC_MODE_MASK; + out->cipher_mismatch = w1 & (MT7932_RXD1_CIPHER_MISMATCH | + MT7932_RXD1_CIPHER_LEN_MISMATCH); if (!mt7932_channel_band(out->channel)) return -EOPNOTSUPP; if ((out->translated && (w2 & (1U << 27))) || (!out->translated && out->amsdu)) @@ -103,6 +118,18 @@ static inline int mt7932_rx_frame(const u8 *packet, size_t size, struct mt7932_r return 0; } +/* + * On a protected connection, deliver only frames the hardware decrypted with + * the cipher and key length it expected. The firmware runs the WPA2 handshake + * itself, so the host has no unprotected EAPOL to accept. Nothing is + * decrypted on an open connection. + */ +static inline bool mt7932_rx_protection_ok(const struct mt7932_rx_frame *frame, + bool open) +{ + return open || (frame->sec_mode && !frame->cipher_mismatch); +} + static inline int mt7932_rx_signal(const struct mt7932_rx_frame *frame, u8 antenna_mask) { int signal = -128; diff --git a/drivers/net/wireless/mediatek/mt7932/protocol.h b/drivers/net/wireless/mediatek/mt7932/protocol.h index 00751e130ef44a..e7f8820a9ff8c6 100644 --- a/drivers/net/wireless/mediatek/mt7932/protocol.h +++ b/drivers/net/wireless/mediatek/mt7932/protocol.h @@ -65,9 +65,37 @@ static inline int mt7932_event_parse(const u8 *packet, size_t length, return 0; } -static inline bool mt7932_event_matches(const struct mt7932_event *e, u8 seq) +/* + * The event that answers a command waiting for a reply, or -1 for any event. + * Extended commands answer with the extended event, the capability query + * with its own event, and the boot ROM's status commands with the generic + * one. The ROM's register read answers with the access-register event and + * its patch semaphore with the patch-semaphore event, as on other connac + * devices. Other runtime commands answer with an event of their own ID. + * The ROM's OTP read alone still matches on the sequence: its reply event + * is not documented, and mt_otp_byte() records the one it receives. + */ +static inline int mt7932_reply_eid(u8 cid, bool runtime) { - return seq && e->seq == seq && !(e->option & 4); + if (cid == 0xed) + return 0xed; + if (runtime) + return cid == 0x8a ? 0xec : cid; + switch (cid) { + case 0x03: + return 0x02; + case 0x10: + return 0x04; + case 0x50: + return -1; + default: + return 0x01; + } +} + +static inline bool mt7932_event_matches(const struct mt7932_event *e, u8 seq, int eid) +{ + return seq && e->seq == seq && !(e->option & 4) && (eid < 0 || e->eid == eid); } #endif diff --git a/drivers/net/wireless/mediatek/mt7932/rate.h b/drivers/net/wireless/mediatek/mt7932/rate.h index d570a69f81f441..7e7c5685d40fdb 100644 --- a/drivers/net/wireless/mediatek/mt7932/rate.h +++ b/drivers/net/wireless/mediatek/mt7932/rate.h @@ -65,33 +65,4 @@ static inline bool mt7932_rx_rate(u32 rxv, u8 max_nss, struct rate_info *rate) return true; } -/* Stock CID85 reply: exact version1 body, owned BSS/MAC and pending sequence. - * Byte8's station identity and byte0a remain opaque; no invented WTBL test. - */ -static inline bool mt7932_tx_rate_reply(const struct mt7932_event *event, - u8 sequence, const u8 *peer, u8 max_nss, - struct rate_info *rate) -{ - const u8 *body; - u32 a, b, c, rxv; - - if (!mt7932_event_matches(event, sequence) || event->eid != 0x21 || - event->length != 344) - return false; - body = event->packet + 36; - if (body[0] != 1 || - !(get_unaligned_le32(body + 4) & 1) || body[9] || - memcmp(body + 12, peer, 6)) - return false; - a = get_unaligned_le32(body + 0x9c); - b = get_unaligned_le32(body + 0xa0); - c = get_unaligned_le32(body + 0xa4); - if (((a >> 12) & 15) >= 5) - return false; /* TX HE GI/DCM/allocation fields not qualified by this reply. */ - /* Normalize the independently qualified TX words to the shared fields. */ - rxv = (c & 127) | (((c >> 8) & 7) << 7) | - (((a >> 8) & 7) << 12) | (((b >> 26) & 3) << 15) | - (((a >> 6) & 3) << 22) | (((a >> 12) & 15) << 24); - return mt7932_rx_rate(rxv, max_nss, rate); -} #endif diff --git a/drivers/net/wireless/mediatek/mt7932/recovery.c b/drivers/net/wireless/mediatek/mt7932/recovery.c index f0b7cc9714c241..31168bfa2b489a 100644 --- a/drivers/net/wireless/mediatek/mt7932/recovery.c +++ b/drivers/net/wireless/mediatek/mt7932/recovery.c @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include "mt7932.h" static int mt_tx_retire(struct mt7932 *m); @@ -6,9 +6,6 @@ static int mt_runtime_scheduler(struct mt7932 *m, bool publish); static int mt_runtime_rebuild(struct mt7932 *m); static void mt_runtime_publish(struct mt7932 *m); static int mt_runtime_ack(struct mt7932 *m, u32 bit); -/* First runtime epoch only. Contracts: NEO_IRQ_SELECTION_AND_RUNTIME_HANDSHAKE - * and NEO_RUNTIME_RING_REBUILD_AND_OBSERVED_PRECAL. - */ static int mt_tx_retire(struct mt7932 *m) { @@ -179,7 +176,7 @@ static int mt_runtime_ack(struct mt7932 *m, u32 bit) int mt_recovery_gate(struct mt7932 *m) { /* Original observed 25F84 CID6b bodies, not PL country tables. - * Explicit experiment; first two action2 field meanings remain opaque. + * The meaning of the first two action2 fields is unknown. */ static const u8 queue_config[3][32] = { {0x10,0x11,2,0,3,0,7,0,0x2f,0,2}, @@ -191,8 +188,11 @@ int mt_recovery_gate(struct mt7932 *m) unsigned int i, phase = 0; int ret; - /* mt_runtime_start() completed configuration before entering this stage. */ - if (mt_transport_polled() || m->runtime_epoch) + /* + * mt_runtime_start() completed configuration before entering this stage, + * which switches to the runtime ring layout once per firmware boot. + */ + if (m->runtime_epoch) return -EINVAL; for (i = 0; i < ARRAY_SIZE(queue_config); i++) { ret = mt_request(m, 0x6b, true, true, false, queue_config[i], 32); @@ -274,7 +274,7 @@ int mt_recovery_gate(struct mt7932 *m) mt_read(m, m->tx[17].reg + 12)); } else { WRITE_ONCE(m->running, true); - dev_info(&m->pdev->dev, "RUNTIME_DMA_READY: full RX epoch, TX17 reset, firmware NORMAL_STATE\n"); + dev_dbg(&m->pdev->dev, "RUNTIME_DMA_READY: full RX epoch, TX17 reset, firmware NORMAL_STATE\n"); } enable_irq(pci_irq_vector(m->pdev, 0)); if (!ret) { @@ -284,8 +284,8 @@ int mt_recovery_gate(struct mt7932 *m) mt7932_capabilities(m->reply + 36, m->reply_length - 36) < 0)) ret = -EPROTO; if (!ret) - dev_info(&m->pdev->dev, "RUNTIME_QUERY_OK: TX17 capability response, IRQ=%llu\n", - m->interrupts); + dev_dbg(&m->pdev->dev, "RUNTIME_QUERY_OK: TX17 capability response, IRQ=%llu\n", + m->interrupts); } return ret; } diff --git a/drivers/net/wireless/mediatek/mt7932/regulatory-test.c b/drivers/net/wireless/mediatek/mt7932/regulatory-test.c new file mode 100644 index 00000000000000..225f3b8f35a0c0 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt7932/regulatory-test.c @@ -0,0 +1,429 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * KUnit tests for the regulatory snapshot logic and the power package + * parser and channel filter. The packages are local files, so + * the parser treats them as untrusted. + */ +#include +#include + +#include "mt7932.h" + +/* A world-domain snapshot: channels 1, 6, 36 and 149, the 5 GHz ones NO_IR. */ +static void reg_build(struct mt7932_reg_snapshot *reg) +{ + static const u16 channels[] = {1, 6, 36, 149}; + unsigned int i; + + memset(reg, 0, sizeof(*reg)); + memcpy(reg->alpha2, "00", 2); + memcpy(reg->domain, "XZ", 2); + reg->domain[4] = 1; + reg->domain[5] = 2; + reg->domain[8] = 2; + reg->domain[9] = 2; + for (i = 0; i < ARRAY_SIZE(channels); i++) { + put_unaligned_le16(channels[i], reg->domain + 12 + i * 8); + put_unaligned_le32(channels[i] > 14 ? IEEE80211_CHAN_NO_IR : 0, + reg->domain + 16 + i * 8); + reg->power[i] = 20; + } + reg->length = 12 + ARRAY_SIZE(channels) * 8; +} + +static void reg_set_flags(struct mt7932_reg_snapshot *reg, unsigned int entry, u32 flags) +{ + put_unaligned_le32(flags, reg->domain + 16 + entry * 8); +} + +/* A beacon hint lifting NO_IR only relaxes; nothing else does. */ +static void mt7932_reg_relaxation_test(struct kunit *test) +{ + struct mt7932_reg_snapshot *prev, *next; + + prev = kunit_kzalloc(test, sizeof(*prev), GFP_KERNEL); + next = kunit_kzalloc(test, sizeof(*next), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, prev); + KUNIT_ASSERT_NOT_NULL(test, next); + reg_build(prev); + + reg_build(next); + reg_set_flags(next, 2, 0); + KUNIT_EXPECT_TRUE(test, mt7932_reg_only_relaxes(prev, next)); + /* An identical snapshot is not treated as a change by the caller, + * but it is trivially within the applied domain. + */ + KUNIT_EXPECT_TRUE(test, mt7932_reg_only_relaxes(prev, prev)); + + /* NO_IR added: a restriction. */ + reg_build(next); + reg_set_flags(next, 0, IEEE80211_CHAN_NO_IR); + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + /* Radar or a bandwidth limit appears, or another flag goes away. */ + reg_build(next); + reg_set_flags(next, 2, IEEE80211_CHAN_NO_IR | IEEE80211_CHAN_RADAR); + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_set_flags(prev, 3, IEEE80211_CHAN_NO_IR | IEEE80211_CHAN_NO_HT40); + reg_build(next); + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_build(prev); + /* A new country, power limit, channel or channel order. */ + reg_build(next); + memcpy(next->alpha2, "AU", 2); + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_build(next); + next->power[1] = 19; + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_build(next); + put_unaligned_le16(40, next->domain + 12 + 3 * 8); + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_build(next); + put_unaligned_le16(44, next->domain + 12 + 4 * 8); + next->domain[9] = 3; + next->length += 8; + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + /* A failed snapshot never relaxes, and a corrupt length is refused. */ + reg_build(next); + next->error = -EOPNOTSUPP; + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); + reg_build(next); + prev->length = sizeof(prev->domain) + 8; + next->length = prev->length; + KUNIT_EXPECT_FALSE(test, mt7932_reg_only_relaxes(prev, next)); +} + +#define PKG_MODE_SIZE 328 +#define PKG_ENTRY_SIZE 122 + +/* Channels of the synthetic 2.4 and 5 GHz tables 1..9, in table order. */ +static const u8 pkg_channels[] = { + 1, 2, 3, 4, 5, 6, 7, 8, + 9, 10, 11, 12, 13, 14, + 36, 40, 44, 48, 52, 56, 60, 64, + 100, 104, 108, 112, 116, 120, 124, 128, + 132, 136, 140, 144, 149, 153, 157, 161, + 38, 46, 54, 62, 102, 110, 118, 126, + 134, 142, 151, 159, 42, 58, 106, 122, + 138, 155, 50, 114, 130, 146, 163, 167, + 165, 169, 171, 173, 175, +}; + +static unsigned int pkg_count(unsigned int table, bool world) +{ + if (table == 2) + return world ? 6 : 5; + return table == 9 ? 5 : 8; +} + +static size_t pkg_size(bool world) +{ + static const u16 others[] = {532, 144, 58, 265}; + size_t size = 16 + (world ? 2 : 3) * PKG_MODE_SIZE; + unsigned int i; + + for (i = 1; i <= 9; i++) + size += 44 + pkg_count(i, world) * PKG_ENTRY_SIZE; + for (i = 0; i < ARRAY_SIZE(others); i++) + size += others[i]; + return size; +} + +/* Build a structurally valid package. Channel 14 is only in the world one. */ +static u8 *pkg_build(struct kunit *test, const char *alpha2, size_t *size) +{ + static const u16 lengths[] = {532, 144, 58, 265}; + static const u8 types[] = {5, 1, 4, 2}; + bool world = !memcmp(alpha2, "XZ", 2); + unsigned int i, j, modes = world ? 2 : 3, channel = 0; + u8 *data, *p; + + *size = pkg_size(world); + data = kunit_kzalloc(test, *size, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, data); + memcpy(data, "J7RP", 4); + put_unaligned_le16(1, data + 4); + memcpy(data + 6, alpha2, 2); + data[8] = modes; + data[9] = MT7932_POLICY_TABLES; + put_unaligned_le32(*size, data + 12); + p = data + 16; + for (i = 0; i < modes; i++, p += PKG_MODE_SIZE) + put_unaligned_le32(17 + i, p + 4); + for (i = 0; i < MT7932_POLICY_TABLES; i++) { + if (i >= 1 && i <= 9) { + unsigned int count = pkg_count(i, world); + + p[0] = 3; + p[1] = 0x0a; + put_unaligned_le16(44 + count * PKG_ENTRY_SIZE, p + 2); + p[4] = count; + p[5] = i <= 2 ? 1 : 2; + p[6] = i == 2 || i == 9; + memcpy(p + 8, alpha2, 2); + for (j = 0; j < count; j++) { + u8 *entry = p + 44 + j * PKG_ENTRY_SIZE; + + /* A country package has no channel 14 entry. */ + if (!world && pkg_channels[channel] == 14) + channel++; + entry[0] = pkg_channels[channel++]; + memset(entry + 1, 30, PKG_ENTRY_SIZE - 1); + } + p += 44 + count * PKG_ENTRY_SIZE; + } else { + unsigned int k = i ? i - 9 : 0; + + p[1] = 2; + put_unaligned_le16(lengths[k], p + 2); + p[7] = types[k]; + if (i == 10) + memcpy(p + 8, alpha2, 2); + p += lengths[k]; + } + } + KUNIT_ASSERT_PTR_EQ(test, p, data + *size); + return data; +} + +/* The entry of @channel in a parsed package, or NULL. */ +static u8 *pkg_entry(const struct mt7932_policy *policy, unsigned int channel) +{ + unsigned int i, j; + + for (i = 1; i <= 9; i++) + for (j = 0; j < policy->table[i][4]; j++) + if (policy->table[i][44 + j * PKG_ENTRY_SIZE] == channel) + return (u8 *)policy->table[i] + 44 + j * PKG_ENTRY_SIZE; + return NULL; +} + +static void mt7932_policy_valid_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data; + + data = pkg_build(test, "AU", &size); + KUNIT_EXPECT_EQ(test, size, 10447); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "AU"), 0); + KUNIT_EXPECT_EQ(test, policy.modes, 3); + KUNIT_EXPECT_PTR_EQ(test, policy.table[0], data + 16 + 3 * PKG_MODE_SIZE); + KUNIT_EXPECT_EQ(test, policy.length[12], 265); + KUNIT_EXPECT_TRUE(test, mt7932_policy_permits(&policy, 36)); + KUNIT_EXPECT_FALSE(test, mt7932_policy_permits(&policy, 14)); + + data = pkg_build(test, "XZ", &size); + KUNIT_EXPECT_EQ(test, size, 10241); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "XZ"), 0); + KUNIT_EXPECT_EQ(test, policy.modes, 2); + KUNIT_EXPECT_TRUE(test, mt7932_policy_permits(&policy, 14)); +} + +/* A package serves only the country it names, in the header and table 10. */ +static void mt7932_policy_country_mismatch_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data; + + data = pkg_build(test, "AU", &size); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "XZ"), -EINVAL); + /* A country package renamed to the world one is not the world one. */ + memcpy(data + 6, "XZ", 2); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "XZ"), -EINVAL); +} + +/* Every prefix of a valid package, with a consistent header, is rejected. */ +static void mt7932_policy_truncation_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size, cut; + u8 *data, *copy; + + data = pkg_build(test, "PL", &size); + for (cut = 0; cut < size; cut++) { + /* An exact-size copy lets KASAN catch any read past the end. */ + copy = kmemdup(data, cut, GFP_KERNEL); + KUNIT_ASSERT_TRUE(test, copy || !cut); + if (cut >= 16) + put_unaligned_le32(cut, copy + 12); + KUNIT_EXPECT_EQ_MSG(test, mt7932_policy_parse(&policy, copy, cut, "PL"), + -EINVAL, "length %zu", cut); + kfree(copy); + } +} + +static void mt7932_policy_header_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data, *longer; + + data = pkg_build(test, "PL", &size); + /* Trailing bytes after the last table. */ + longer = kunit_kzalloc(test, size + 4, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, longer); + memcpy(longer, data, size); + put_unaligned_le32(size + 4, longer + 12); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, longer, size + 4, "PL"), -EINVAL); + + data[4] = 2; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + data[4] = 1; + data[8] = 4; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + data[8] = 3; + data[9] = 12; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + data[9] = MT7932_POLICY_TABLES; + put_unaligned_le32(size - 1, data + 12); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + put_unaligned_le32(size, data + 12); + /* Mode command lengths are bounded by the 320-byte body. */ + put_unaligned_le32(321, data + 16 + 4); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + put_unaligned_le32(0, data + 16 + 4); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + put_unaligned_le32(17, data + 16 + 4); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), 0); +} + +/* Table lengths and entry counts must agree with the table type. */ +static void mt7932_policy_table_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data, *table; + + data = pkg_build(test, "PL", &size); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), 0); + table = (u8 *)policy.table[3]; + /* An entry count that would run past the table. */ + table[4] = 9; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + table[4] = 8; + /* A table length that claims the rest of the file. */ + put_unaligned_le16(0xffff, table + 2); + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); + put_unaligned_le16(44 + 8 * PKG_ENTRY_SIZE, table + 2); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), 0); + table = (u8 *)policy.table[10]; + table[7] = 2; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), -EINVAL); +} + +/* The world package's 2.4 GHz tables must list channels 1..14 in order. */ +static void mt7932_policy_world_order_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data; + + data = pkg_build(test, "XZ", &size); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "XZ"), 0); + pkg_entry(&policy, 9)[0] = 10; + KUNIT_EXPECT_EQ(test, mt7932_policy_parse(&policy, data, size, "XZ"), -EINVAL); +} + +static void mt7932_policy_permits_test(struct kunit *test) +{ + struct mt7932_policy policy; + size_t size; + u8 *data, *entry; + + data = pkg_build(test, "PL", &size); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), 0); + entry = pkg_entry(&policy, 144); + KUNIT_ASSERT_NOT_NULL(test, entry); + memset(entry + 1, MT7932_POLICY_NOT_PERMITTED, PKG_ENTRY_SIZE - 1); + KUNIT_EXPECT_FALSE(test, mt7932_policy_permits(&policy, 144)); + /* One rate with a limit is enough to permit the channel. */ + entry[PKG_ENTRY_SIZE - 1] = 8; + KUNIT_EXPECT_TRUE(test, mt7932_policy_permits(&policy, 144)); + /* A channel the package does not list is not permitted. */ + KUNIT_EXPECT_FALSE(test, mt7932_policy_permits(&policy, 177)); +} + +static void mt7932_policy_filter_test(struct kunit *test) +{ + static const u16 channels[] = {1, 6, 13, 36, 144, 149, 165}; + struct mt7932_reg_snapshot *reg; + struct mt7932_policy policy; + unsigned int i; + size_t size; + u8 *data; + + reg = kunit_kzalloc(test, sizeof(*reg), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, reg); + data = pkg_build(test, "PL", &size); + KUNIT_ASSERT_EQ(test, mt7932_policy_parse(&policy, data, size, "PL"), 0); + memset(pkg_entry(&policy, 144) + 1, MT7932_POLICY_NOT_PERMITTED, + PKG_ENTRY_SIZE - 1); + memset(pkg_entry(&policy, 6) + 1, MT7932_POLICY_NOT_PERMITTED, + PKG_ENTRY_SIZE - 1); + memcpy(reg->domain, "PL", 2); + for (i = 0; i < ARRAY_SIZE(channels); i++) { + put_unaligned_le16(channels[i], reg->domain + 12 + i * 8); + put_unaligned_le32(i, reg->domain + 16 + i * 8); + } + reg->domain[8] = 3; + reg->domain[9] = 4; + reg->length = 12 + ARRAY_SIZE(channels) * 8; + + mt7932_policy_filter(reg, &policy); + KUNIT_EXPECT_EQ(test, reg->length, 12 + 5 * 8); + KUNIT_EXPECT_EQ(test, reg->domain[8], 2); + KUNIT_EXPECT_EQ(test, reg->domain[9], 3); + KUNIT_EXPECT_EQ(test, get_unaligned_le16(reg->domain + 12 + 8), 13); + KUNIT_EXPECT_EQ(test, get_unaligned_le16(reg->domain + 12 + 3 * 8), 149); + /* Each entry keeps its own channel flags. */ + KUNIT_EXPECT_EQ(test, get_unaligned_le32(reg->domain + 16 + 3 * 8), 5); + /* The vacated tail is cleared. */ + KUNIT_EXPECT_TRUE(test, !memchr_inv(reg->domain + 12 + 5 * 8, 0, 2 * 8)); +} + +/* Random corruption must never be accepted in a way that reads out of bounds. */ +static void mt7932_policy_corruption_test(struct kunit *test) +{ + struct rnd_state rnd; + struct mt7932_policy policy; + unsigned int round, flips, channel; + size_t size; + u8 *data, *copy; + + prandom_seed_state(&rnd, 7932); + data = pkg_build(test, "AU", &size); + copy = kunit_kzalloc(test, size, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, copy); + for (round = 0; round < 2000; round++) { + memcpy(copy, data, size); + for (flips = 1 + prandom_u32_state(&rnd) % 4; flips; flips--) + copy[prandom_u32_state(&rnd) % size] ^= 1 << (prandom_u32_state(&rnd) % 8); + if (mt7932_policy_parse(&policy, copy, size, "AU")) + continue; + for (channel = 0; channel < 256; channel++) + mt7932_policy_permits(&policy, channel); + } +} + +static struct kunit_case mt7932_policy_test_cases[] = { + KUNIT_CASE(mt7932_reg_relaxation_test), + KUNIT_CASE(mt7932_policy_valid_test), + KUNIT_CASE(mt7932_policy_country_mismatch_test), + KUNIT_CASE(mt7932_policy_truncation_test), + KUNIT_CASE(mt7932_policy_header_test), + KUNIT_CASE(mt7932_policy_table_test), + KUNIT_CASE(mt7932_policy_world_order_test), + KUNIT_CASE(mt7932_policy_permits_test), + KUNIT_CASE(mt7932_policy_filter_test), + KUNIT_CASE(mt7932_policy_corruption_test), + {} +}; + +static struct kunit_suite mt7932_policy_test_suite = { + .name = "mt7932-regulatory", + .test_cases = mt7932_policy_test_cases, +}; + +kunit_test_suite(mt7932_policy_test_suite); diff --git a/drivers/net/wireless/mediatek/mt7932/regulatory.h b/drivers/net/wireless/mediatek/mt7932/regulatory.h index 45cb3d6ce91f0d..3a8c1ffc740151 100644 --- a/drivers/net/wireless/mediatek/mt7932/regulatory.h +++ b/drivers/net/wireless/mediatek/mt7932/regulatory.h @@ -5,6 +5,10 @@ /* Private, versioned packaging of original-derived country policy. This is * not firmware code and does not replace cfg80211's regulatory database. */ +#include + +#include "channels.h" + #define MT7932_POLICY_TABLES 13 #define MT7932_POLICY_MODES 3 @@ -20,12 +24,12 @@ struct mt7932_reg_snapshot { * profile. Stock unknown-country startup uses XZ while Linux stays 00. */ u8 alpha2[2]; - u8 domain[148]; + u8 domain[12 + 8 * MT7932_CHANNELS]; u16 length; /* Keep numeric limits in equality checks even though CID0f contains only * flags. An unsupported reduction must not be mistaken for no change. */ - s32 power[17]; + s32 power[MT7932_CHANNELS]; int error; }; @@ -100,6 +104,36 @@ static inline int mt7932_policy_parse(struct mt7932_policy *out, return at == size ? 0 : -EINVAL; } +/* + * Whether @next differs from the applied @prev only by NO_IR lifted on some + * channels, as a beacon hint does in the world domain: same country, firmware + * domain header, channel list and limits, and every channel's flags equal to + * the applied ones less, at most, NO_IR. Such a change only permits more; the + * firmware may keep the stricter domain until it is idle. + */ +static inline bool mt7932_reg_only_relaxes(const struct mt7932_reg_snapshot *prev, + const struct mt7932_reg_snapshot *next) +{ + unsigned int i, count; + + if (prev->error || next->error || prev->length != next->length || + prev->length < 12 || prev->length > sizeof(prev->domain) || + memcmp(prev->alpha2, next->alpha2, 2) || + memcmp(prev->domain, next->domain, 12) || + memcmp(prev->power, next->power, sizeof(prev->power))) + return false; + count = (prev->length - 12) / 8; + for (i = 0; i < count; i++) { + const u8 *a = prev->domain + 12 + i * 8, *b = next->domain + 12 + i * 8; + u32 was = get_unaligned_le32(a + 4), now = get_unaligned_le32(b + 4); + + if (memcmp(a, b, 4) || (now & ~was) || + ((was & ~now) & ~IEEE80211_CHAN_NO_IR)) + return false; + } + return true; +} + /* Every entry of a validated package starts with its channel number and is * followed by per-rate power limits. A country that does not permit a * channel sets all of that channel's limits to this marker. diff --git a/drivers/net/wireless/mediatek/mt7932/reset.c b/drivers/net/wireless/mediatek/mt7932/reset.c index 488a3ba5414b9b..d93080513c77f4 100644 --- a/drivers/net/wireless/mediatek/mt7932/reset.c +++ b/drivers/net/wireless/mediatek/mt7932/reset.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only /* J700 MT7932 mapped preparation and function-scoped reset. - * Clean-room contracts: NEO_MT7932_DEVICE_RESET_UNLOAD_CONTRACT.md and - * NEO_MT7932_FLR_PREPARATION_RESOLVED.md. No secondary-bus reset fallback. + * Covers device reset, firmware unload and FLR preparation. + * No secondary-bus reset fallback. */ #include "mt7932.h" @@ -125,7 +125,7 @@ int mt_function_reset(struct mt7932 *m) if (!peer) return mt_function_reset_locked(m); /* The radio functions may share state beyond PCI's advertised FLR scope. - * No qualified contract permits resetting Wi-Fi while Bluetooth is live. + * Resetting Wi-Fi while Bluetooth is live has never been tested. * Trylock avoids inversion with a concurrent sibling probe/remove; hold * it across FLR to prevent Bluetooth starting after the admission check. */ diff --git a/drivers/net/wireless/mediatek/mt7932/rf-test.c b/drivers/net/wireless/mediatek/mt7932/rf-test.c index ede549842f7b2a..5f9a34825cc231 100644 --- a/drivers/net/wireless/mediatek/mt7932/rf-test.c +++ b/drivers/net/wireless/mediatek/mt7932/rf-test.c @@ -412,6 +412,120 @@ static void mt_late_association_channel_preserves_terminal_test(struct kunit *te } } +struct mt_scan_retire_test_context { + struct delayed_work retire; + struct mt7932 *m; + bool late; +}; + +static struct mt_scan_retire_test_context *mt_scan_retire_ctx; + +/* The firmware reports the end of the cancelled batch a little later. */ +static void mt_scan_retire_test_late(struct work_struct *work) +{ + struct mt_scan_retire_test_context *ctx = + container_of(to_delayed_work(work), struct mt_scan_retire_test_context, retire); + struct mt7932 *m = ctx->m; + unsigned long flags; + + spin_lock_irqsave(&m->response_lock, flags); + m->retired_scan_seq = 0; + complete(&m->scan_done); + spin_unlock_irqrestore(&m->response_lock, flags); +} + +/* + * The scan completion work had already submitted the next batch when the + * abort arrived: the abort cancels it, and its end is still to come. + */ +static void mt_scan_retire_test_finish(struct work_struct *work) +{ + struct mt_scan_retire_test_context *ctx = mt_scan_retire_ctx; + struct mt7932 *m = ctx->m; + unsigned long flags; + + spin_lock_irqsave(&m->response_lock, flags); + if (m->scan_request && m->scan_aborted) { + m->scan_request = NULL; + m->retired_scan_seq = 8; + } + spin_unlock_irqrestore(&m->response_lock, flags); + if (ctx->late) + schedule_delayed_work(&ctx->retire, msecs_to_jiffies(300)); +} + +static void mt_scan_quiesce_retirement_test(struct kunit *test) +{ + static const struct cfg80211_ops ops = {}; + struct mt_scan_retire_test_context ctx = {}; + struct cfg80211_scan_request *scan; + struct mt7932 *m = test->priv; + unsigned long start; + unsigned int round; + + scan = kunit_kzalloc(test, sizeof(*scan), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scan); + m->wiphy = wiphy_new(&ops, sizeof(m)); + KUNIT_ASSERT_NOT_NULL(test, m->wiphy); + *(struct mt7932 **)wiphy_priv(m->wiphy) = m; + init_completion(&m->scan_done); + INIT_WORK(&m->scan_finish_work, mt_scan_retire_test_finish); + INIT_DELAYED_WORK(&ctx.retire, mt_scan_retire_test_late); + ctx.m = m; + mt_scan_retire_ctx = &ctx; + for (round = 0; round < 2; round++) { + ctx.late = !round; + reinit_completion(&m->scan_done); + m->scan_request = scan; + scan->wdev = &m->wdev; + m->scan_aborted = false; + m->retired_scan_seq = 0; + /* The batch the quiesce waits for has just finished. */ + complete(&m->scan_done); + start = jiffies; + if (ctx.late) { + /* A late but genuine end retires the scan: no failure. */ + KUNIT_EXPECT_EQ(test, mt_scan_quiesce(m), 0); + KUNIT_EXPECT_EQ(test, m->retired_scan_seq, 0); + } else { + /* A batch the firmware never ends still fails, after the bound. */ + KUNIT_EXPECT_EQ(test, mt_scan_quiesce(m), -EBUSY); + KUNIT_EXPECT_EQ(test, m->retired_scan_seq, 8); + KUNIT_EXPECT_GE(test, jiffies - start, msecs_to_jiffies(1900)); + KUNIT_EXPECT_LE(test, jiffies - start, msecs_to_jiffies(3000)); + } + cancel_delayed_work_sync(&ctx.retire); + flush_work(&m->scan_finish_work); + } + mt_scan_retire_ctx = NULL; + wiphy_free(m->wiphy); + m->wiphy = NULL; +} + +/* + * An RF failure while the connect worker is publishing a new connection + * must not report the disconnect ahead of the connect: it leaves the report + * to the worker. (cfg80211 is not called here: the synthetic netdev has no + * wireless device, so a direct report would crash the test.) + */ +static void mt_rf_failure_during_publication_test(struct kunit *test) +{ + struct mt7932 *m = test->priv; + + m->connecting = false; + m->connected = true; + m->connect_publishing = true; + mt_rf_test_fail(m, -EIO); + KUNIT_EXPECT_TRUE(test, m->connect_lost); + KUNIT_EXPECT_FALSE(test, m->connected); + KUNIT_EXPECT_TRUE(test, m->disconnecting); + KUNIT_EXPECT_FALSE(test, netif_carrier_ok(m->netdev)); + /* Ownership is uncertain after an RF failure: keep the peer. */ + KUNIT_EXPECT_TRUE(test, m->peer_valid); + m->connect_lost = false; + m->connect_publishing = false; +} + static struct kunit_case mt_rf_test_cases[] = { KUNIT_CASE(mt_invalid_association_channel_wakes_both_test), KUNIT_CASE(mt_valid_repeated_association_start_test), @@ -422,6 +536,8 @@ static struct kunit_case mt_rf_test_cases[] = { KUNIT_CASE(mt_rf_failure_wakes_association_test), KUNIT_CASE(mt_rf_failure_blocks_startup_test), KUNIT_CASE(mt_rf_failure_preserves_first_error_test), + KUNIT_CASE(mt_scan_quiesce_retirement_test), + KUNIT_CASE(mt_rf_failure_during_publication_test), {} }; diff --git a/drivers/net/wireless/mediatek/mt7932/scan.h b/drivers/net/wireless/mediatek/mt7932/scan.h index 4899cbb6c1654f..217fc5c9d1e96c 100644 --- a/drivers/net/wireless/mediatek/mt7932/scan.h +++ b/drivers/net/wireless/mediatek/mt7932/scan.h @@ -5,10 +5,13 @@ #include "protocol.h" #include "channels.h" +/* CID03 carries this many band/channel slots, from 0xa0 up to 0xe0. */ +#define MT7932_SCAN_CHANNELS 32 + /* Selected ordinary TX17/fullmac wildcard scan, not a pre-RF gate. * Caller must own BSS0 and filter cfg80211 DISABLED channels first. NO_IR - * channels require mt7932_scan_passive() before submission. - * NEO_FIRST_CFG80211_SCAN_CONTRACT: 20ms dwell and meaningful MT7932 tail. + * channels require mt7932_scan_passive() before submission. The body uses + * a 20 ms dwell and fills the MT7932-specific tail. */ static inline int mt7932_scan_body(u8 *body, size_t capacity, u8 seq, const u8 *channels, unsigned int count) @@ -17,7 +20,7 @@ static inline int mt7932_scan_body(u8 *body, size_t capacity, u8 seq, 0x6b,7,0x0f,0xff,0xff,0xff,0xff,0xff,0xff}; unsigned int i, j; - if (capacity < 0x4d4 || !seq || seq > 127 || !count || count > 13) + if (capacity < 0x4d4 || !seq || seq > 127 || !count || count > MT7932_SCAN_CHANNELS) return -EINVAL; for (i = 0; i < count; i++) { if (!mt7932_channel_band(channels[i]) || @@ -84,7 +87,7 @@ static inline void mt7932_scan_disconnected_passive(u8 body[0x4d4]) } /* This firmware requires its native wildcard template. Directed scans use - * no probe IEs. Forwarding the normal supplicant extras stalled the campaign + * no probe IEs. Forwarding the normal supplicant extras stalled the scan * on J700; arbitrary caller probe-IE carriage remains unsupported here. */ static inline int mt7932_scan_options(u8 body[0x4d4], const u8 *ssid, diff --git a/drivers/net/wireless/mediatek/mt7932/startup.c b/drivers/net/wireless/mediatek/mt7932/startup.c index 3dd160694c57f6..679c9193f85bc5 100644 --- a/drivers/net/wireless/mediatek/mt7932/startup.c +++ b/drivers/net/wireless/mediatek/mt7932/startup.c @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only #include #include "mt7932.h" @@ -123,7 +123,7 @@ static int mt_stock_config(struct mt7932 *m) if (ret) goto free; } - dev_info(&m->pdev->dev, "STOCK_CONFIG_SUBMITTED: %u original/PHY-derived entries\n", count); + dev_dbg(&m->pdev->dev, "STOCK_CONFIG_SUBMITTED: %u original/PHY-derived entries\n", count); free: kfree(records); out: @@ -165,9 +165,9 @@ static int mt_publish_policy(struct mt7932 *m, const struct mt7932_reg_snapshot /* Native SETs have no response/atomic policy-commit event. Command * submission is not misrepresented as firmware readback or an ACK. */ - dev_info(&m->pdev->dev, "REGULATORY_POLICY_SUBMITTED: %c%c firmware=%c%c channels=%u (native no-response SETs)\n", - reg->alpha2[0], reg->alpha2[1], reg->domain[0], reg->domain[1], - (reg->length - 12) / 8); + dev_dbg(&m->pdev->dev, "REGULATORY_POLICY_SUBMITTED: %c%c firmware=%c%c channels=%u (native no-response SETs)\n", + reg->alpha2[0], reg->alpha2[1], reg->domain[0], reg->domain[1], + (reg->length - 12) / 8); return 0; } @@ -226,7 +226,7 @@ static int mt_startup_once(struct mt7932 *m) ret = mt_calibration_gate(m); if (!ret) ret = mt_request(m, 0x3e, true, true, false, roam_lock, sizeof(roam_lock)); - dev_info(&m->pdev->dev, "OWN_CALIBRATION_STARTUP_RESULT: %d\n", ret); + dev_dbg(&m->pdev->dev, "OWN_CALIBRATION_STARTUP_RESULT: %d\n", ret); if (!ret) ret = mt_enable_scan(m); if (ret) @@ -236,10 +236,15 @@ static int mt_startup_once(struct mt7932 *m) static struct ieee80211_channel *mt_channel(struct mt7932 *m, unsigned int i) { - return i < 13 ? &m->channels[i] : - &m->channels5[i - 13]; + return i < MT7932_CHANNELS_2G ? &m->channels[i] : + &m->channels5[i - MT7932_CHANNELS_2G]; } +/* Keep cfg80211's channel list in step with the channels the country + * package leaves out of the firmware domain. cfg80211 rebuilds the flags + * on every regulatory update; the notifier applies the recorded set again + * when the update keeps this package, or starts a new pass otherwise. + */ static void mt_policy_disable(struct mt7932 *m, const struct mt7932_policy *policy, u32 generation) { @@ -258,7 +263,7 @@ static void mt_policy_disable(struct mt7932 *m, const struct mt7932_policy *poli return; } wiphy_lock(m->wiphy); - for (i = 0; i < 17; i++) { + for (i = 0; i < MT7932_CHANNELS; i++) { struct ieee80211_channel *channel = mt_channel(m, i); bool forbidden = !mt7932_policy_permits(policy, channel->hw_value); @@ -274,6 +279,20 @@ static void mt_policy_disable(struct mt7932 *m, const struct mt7932_policy *poli * Consume the flag under the same lock as the worker publication, so repeated * userspace requests cannot continually invalidate an in-flight attempt. */ +/* response_lock held: start the pass a deferred relaxation is waiting for. */ +void mt_reg_apply_deferred_locked(struct mt7932 *m) +{ + lockdep_assert_held(&m->response_lock); + if (!m->reg_deferral || m->stopping) + return; + m->reg_deferral = false; + m->reg_desired = m->reg_deferred; + m->reg_generation++; + WRITE_ONCE(m->reg_pending, true); + if (READ_ONCE(m->interface_registered)) + schedule_work(&m->startup_work); +} + void mt_retry_missing_policy(struct mt7932 *m) { unsigned long flags; @@ -345,12 +364,12 @@ static void mt_startup_work(struct work_struct *work) } if (!ret) { ret = mt7932_policy_parse(&policy, file->data, file->size, reg.domain); - if (ret) + if (!ret) { + mt7932_policy_filter(®, &policy); + mt_policy_disable(m, &policy, generation); + } else { dev_err_ratelimited(&m->pdev->dev, "local input %s has invalid policy: %d\n", path, ret); - } - if (!ret) { - mt7932_policy_filter(®, &policy); - mt_policy_disable(m, &policy, generation); + } } mutex_lock(&m->command_mutex); if (READ_ONCE(m->stopping) || generation != READ_ONCE(m->reg_generation)) @@ -380,9 +399,9 @@ static void mt_startup_work(struct work_struct *work) reg.alpha2[0], reg.alpha2[1], generation, ret, m->policy_failed || m->link_failed); else - dev_info(&m->pdev->dev, "REGULATORY_READY: %c%c firmware=%c%c generation=%u current=%u\n", - reg.alpha2[0], reg.alpha2[1], reg.domain[0], reg.domain[1], - generation, !m->reg_pending); + dev_dbg(&m->pdev->dev, "REGULATORY_READY: %c%c firmware=%c%c generation=%u current=%u\n", + reg.alpha2[0], reg.alpha2[1], reg.domain[0], reg.domain[1], + generation, !m->reg_pending); spin_unlock_irqrestore(&m->response_lock, flags); next: mutex_unlock(&m->command_mutex); @@ -418,22 +437,28 @@ static void mt_regulatory_notify(struct wiphy *wiphy, struct regulatory_request memcpy(reg.domain, "XZ", 2); reg.domain[4] = 1; /* 2 GHz remains 20 MHz. */ reg.domain[5] = mt7932_domain_5g_bw(); - for (i = 0; i < 17; i++) { + for (i = 0; i < MT7932_CHANNELS; i++) { struct ieee80211_channel *channel = mt_channel(m, i); u32 flags = channel->flags; reg.power[i] = channel->max_power; if (flags & IEEE80211_CHAN_DISABLED) continue; - if (flags & (IEEE80211_CHAN_RADAR | IEEE80211_CHAN_NO_OFDM | - IEEE80211_CHAN_NO_20MHZ | IEEE80211_CHAN_PSD | IEEE80211_CHAN_IR_CONCURRENT)) - reg.error = -EOPNOTSUPP; /* Native per-rate units are not a proven dBm conversion. Do not * silently ignore a numeric reduction below the qualified 20 dBm * channel envelope or reinterpret sentinel bytes as power caps. + * A channel with a restriction the driver cannot honour is left + * out of the firmware domain and disabled for cfg80211 instead. + */ + if (flags & (IEEE80211_CHAN_NO_OFDM | IEEE80211_CHAN_NO_20MHZ | + IEEE80211_CHAN_PSD | IEEE80211_CHAN_IR_CONCURRENT) || + channel->max_power < 20) { + channel->flags |= IEEE80211_CHAN_DISABLED; + continue; + } + /* Radar channels carry cfg80211's RADAR flag: the firmware may + * listen there, but the station never probes or joins them. */ - if (channel->max_power < 20) - reg.error = -EOPNOTSUPP; put_unaligned_le16(channel->hw_value, reg.domain + 12 + count * 8); /* Driver bandwidth limits are already in channel->flags. Adding * the old blanket 20 MHz mask here would contradict the wiphy @@ -441,7 +466,7 @@ static void mt_regulatory_notify(struct wiphy *wiphy, struct regulatory_request */ put_unaligned_le32(flags, reg.domain + 16 + count * 8); count++; - reg.domain[i < 13 ? 8 : 9]++; + reg.domain[i < MT7932_CHANNELS_2G ? 8 : 9]++; } reg.length = 12 + count * 8; if (!count) @@ -451,18 +476,48 @@ static void mt_regulatory_notify(struct wiphy *wiphy, struct regulatory_request spin_unlock_irqrestore(&m->response_lock, irqflags); return; } + if (!m->reg_pending && m->connected && !m->disconnecting && + mt7932_reg_only_relaxes(&m->reg_desired, ®) && + memcmp(®, &m->reg_desired, sizeof(reg))) { + /* A pass would end the connection only to permit more than the + * firmware now allows. Keep the stricter applied domain and run + * the pass once the connection has ended. + */ + m->reg_deferred = reg; + m->reg_deferral = true; + spin_unlock_irqrestore(&m->response_lock, irqflags); + for_each_set_bit(i, m->policy_disabled, MT7932_CHANNELS) + mt_channel(m, i)->flags |= IEEE80211_CHAN_DISABLED; + dev_dbg(&m->pdev->dev, "REGULATORY_RELAXATION_DEFERRED: %c%c until disconnected\n", + reg.alpha2[0], reg.alpha2[1]); + return; + } if (!memcmp(®, &m->reg_desired, sizeof(reg)) && !m->reg_pending) { + m->reg_deferral = false; spin_unlock_irqrestore(&m->response_lock, irqflags); - for_each_set_bit(i, m->policy_disabled, 17) + /* The applied package still holds, but cfg80211 has just + * rebuilt the channel flags without its disables. + */ + for_each_set_bit(i, m->policy_disabled, MT7932_CHANNELS) mt_channel(m, i)->flags |= IEEE80211_CHAN_DISABLED; return; } m->reg_desired = reg; + m->reg_deferral = false; m->reg_generation++; - bitmap_zero(m->policy_disabled, 17); + /* The next package pass records its own set. */ + bitmap_zero(m->policy_disabled, MT7932_CHANNELS); WRITE_ONCE(m->reg_pending, true); if (m->netdev) { netif_stop_queue(m->netdev); + /* The stop can land in the middle of a burst: hand the device the + * TX0 descriptors already published behind the doorbell. + */ + if (READ_ONCE(m->data_ready)) { + spin_lock(&m->data_lock); + mt_data_kick_locked(m); + spin_unlock(&m->data_lock); + } if (m->connecting) { m->connect_error = -ECANCELED; complete(&m->assoc_start); @@ -473,8 +528,8 @@ static void mt_regulatory_notify(struct wiphy *wiphy, struct regulatory_request if (READ_ONCE(m->interface_registered)) schedule_work(&m->startup_work); spin_unlock_irqrestore(&m->response_lock, irqflags); - dev_info_ratelimited(&m->pdev->dev, "REGULATORY_REQUEST: %c%c initiator=%u generation=%u\n", - reg.alpha2[0], reg.alpha2[1], request->initiator, m->reg_generation); + dev_dbg(&m->pdev->dev, "REGULATORY_REQUEST: %c%c initiator=%u generation=%u\n", + reg.alpha2[0], reg.alpha2[1], request->initiator, m->reg_generation); } static const struct cfg80211_ops mt_cfg_ops = { @@ -486,6 +541,7 @@ static const struct cfg80211_ops mt_cfg_ops = { .get_channel = mt_get_channel, .dump_station = mt_dump_station, .set_power_mgmt = mt_set_power_mgmt, + .set_cqm_rssi_config = mt_set_cqm_rssi_config, }; int mt_register_regulatory_gate(struct mt7932 *m) @@ -526,7 +582,7 @@ int mt_register_regulatory_gate(struct mt7932 *m) * Global hints from other radios still reach the notifier and are handled. */ m->wiphy->regulatory_flags |= REGULATORY_COUNTRY_IE_IGNORE; - for (i = 0; i < 13; i++) { + for (i = 0; i < MT7932_CHANNELS_2G; i++) { m->channels[i].band = NL80211_BAND_2GHZ; m->channels[i].center_freq = 2412 + 5 * i; m->channels[i].hw_value = i + 1; @@ -544,12 +600,22 @@ int mt_register_regulatory_gate(struct mt7932 *m) m->wiphy->bands[NL80211_BAND_2GHZ] = &m->band2; { for (i = 0; i < ARRAY_SIZE(m->channels5); i++) { + unsigned int primary = mt7932_channel_5g(i); + unsigned int pair = mt7932_channel_ht40_centre(primary); + u32 flags = IEEE80211_CHAN_NO_160MHZ; + + /* The secondary channel sits on the far side of the pair centre. */ + if (!pair) + flags |= IEEE80211_CHAN_NO_HT40 | IEEE80211_CHAN_NO_80MHZ; + else if (pair < primary) + flags |= IEEE80211_CHAN_NO_HT40PLUS; + else + flags |= IEEE80211_CHAN_NO_HT40MINUS; m->channels5[i].band = NL80211_BAND_5GHZ; - m->channels5[i].center_freq = 5180 + 20 * i; - m->channels5[i].hw_value = 36 + 4 * i; + m->channels5[i].center_freq = 5000 + 5 * primary; + m->channels5[i].hw_value = primary; m->channels5[i].max_power = 20; - m->channels5[i].flags = IEEE80211_CHAN_NO_160MHZ | - (i % 2 ? IEEE80211_CHAN_NO_HT40PLUS : IEEE80211_CHAN_NO_HT40MINUS); + m->channels5[i].flags = flags; if (MT7932_MAX_5G_BW < 2) m->channels5[i].flags |= IEEE80211_CHAN_NO_80MHZ; if (!MT7932_MAX_5G_BW) @@ -577,6 +643,6 @@ int mt_register_regulatory_gate(struct mt7932 *m) return ret; WRITE_ONCE(m->interface_registered, true); schedule_work(&m->startup_work); - dev_info(&m->pdev->dev, "REGULATORY_GATE_REGISTERED: cfg80211 country policy required\n"); + dev_dbg(&m->pdev->dev, "REGULATORY_GATE_REGISTERED: cfg80211 country policy required\n"); return 0; } diff --git a/drivers/net/wireless/mediatek/mt7932/station.c b/drivers/net/wireless/mediatek/mt7932/station.c index c9d388e50d8d52..83cee4b650926e 100644 --- a/drivers/net/wireless/mediatek/mt7932/station.c +++ b/drivers/net/wireless/mediatek/mt7932/station.c @@ -103,3 +103,69 @@ int mt_dump_station(struct wiphy *wiphy, struct wireless_dev *wdev, spin_unlock_irqrestore(&m->response_lock, flags); return ret; } + +/* + * The firmware reports no signal threshold events, so the connection quality + * monitor runs on the host, as mac80211 runs it for drivers without one: + * average the signal of the frames the associated access point sends and + * report when the average crosses the threshold. Sample at most once per + * beacon interval, so that the average follows time rather than traffic. + */ +#define MT7932_CQM_INTERVAL (HZ / 10) +#define MT7932_CQM_MIN_SAMPLES 4 + +/* Caller holds response_lock; a new association starts a new average. */ +void mt_cqm_reset(struct mt7932 *m) +{ + ewma_mt7932_signal_init(&m->cqm_signal); + m->cqm_samples = 0; + m->cqm_last_event = 0; +} + +/* Caller holds response_lock; @signal is a frame from the associated AP. */ +void mt_cqm_signal(struct mt7932 *m, int signal) +{ + enum nl80211_cqm_rssi_threshold_event event; + int average, last = m->cqm_last_event; + int hysteresis = m->cqm_hysteresis; + + if (signal >= 0 || signal <= -128) + return; + if (m->cqm_samples && + time_before(jiffies, m->cqm_sample_time + MT7932_CQM_INTERVAL)) + return; + m->cqm_sample_time = jiffies; + ewma_mt7932_signal_add(&m->cqm_signal, -signal); + if (m->cqm_samples < MT7932_CQM_MIN_SAMPLES) + m->cqm_samples++; + if (!m->cqm_threshold || m->cqm_samples < MT7932_CQM_MIN_SAMPLES) + return; + + /* The first event after (re)configuration reports the current side. */ + average = -(int)ewma_mt7932_signal_read(&m->cqm_signal); + if (average < m->cqm_threshold && (!last || average < last - hysteresis)) + event = NL80211_CQM_RSSI_THRESHOLD_EVENT_LOW; + else if (average > m->cqm_threshold && (!last || average > last + hysteresis)) + event = NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH; + else + return; + m->cqm_last_event = average; + cfg80211_cqm_rssi_notify(m->netdev, event, average, GFP_ATOMIC); +} + +int mt_set_cqm_rssi_config(struct wiphy *wiphy, struct net_device *netdev, + s32 rssi_thold, u32 rssi_hyst) +{ + struct mt7932 *m = mt_from_wiphy(wiphy); + unsigned long flags; + + if (netdev != m->netdev) + return -ENODEV; + spin_lock_irqsave(&m->response_lock, flags); + /* A zero threshold disables the monitor. */ + m->cqm_threshold = rssi_thold; + m->cqm_hysteresis = min_t(u32, rssi_hyst, S8_MAX); + m->cqm_last_event = 0; + spin_unlock_irqrestore(&m->response_lock, flags); + return 0; +} diff --git a/drivers/net/wireless/mediatek/mt7932/station_query.c b/drivers/net/wireless/mediatek/mt7932/station_query.c deleted file mode 100644 index 8382768898172b..00000000000000 --- a/drivers/net/wireless/mediatek/mt7932/station_query.c +++ /dev/null @@ -1,65 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -#include "mt7932.h" - -/* Research helper, deliberately not called by production station reporting: - * the reply identity is per-peer, but its vector is shared and unattributed. - * Retained while qualifying a real per-peer rate source, not a public metric. - * One serialized query for the owned peer. Keep the command owner until the - * reply has been checked; disconnect may close admission in IRQ context but - * cannot reuse BSS/station resources across this critical section. - */ -void mt_station_update_rate(struct mt7932 *m, const u8 *mac) -{ - struct mt7932_event event; - struct rate_info rate; - u8 body[28] = {}, peer[ETH_ALEN]; - u32 generation; - unsigned long flags; - bool valid = false; - int ret; - - mutex_lock(&m->command_mutex); - spin_lock_irqsave(&m->response_lock, flags); - if (!m->connected || m->disconnecting || !m->peer_valid || !mt_rf_allowed(m) || - m->station_rate_query_failed || (mac && !ether_addr_equal(mac, m->connect_bssid)) || - time_before(jiffies, m->station_rate_query_time + HZ / 2)) { - spin_unlock_irqrestore(&m->response_lock, flags); - goto unlock; - } - generation = m->connection_generation; - ether_addr_copy(peer, m->connect_bssid); - body[0] = m->peer_ids.station[0]; - memcpy(body + 4, peer, ETH_ALEN); - m->station_rate_query_time = jiffies; - spin_unlock_irqrestore(&m->response_lock, flags); - - ret = mt_request(m, 0x85, true, false, true, body, sizeof(body)); - if (!ret) - ret = mt7932_event_parse(m->reply, m->reply_length, &event); - if (!ret) - valid = mt7932_tx_rate_reply(&event, m->seq, peer, m->phy_cap[4], &rate); - if (valid && (rate.bw != RATE_INFO_BW_20 || - (m->connect_channel > 14 && !rate.flags && - (rate.legacy == 10 || rate.legacy == 20 || rate.legacy == 55 || rate.legacy == 110)))) - valid = false; - - spin_lock_irqsave(&m->response_lock, flags); - if (generation == m->connection_generation && m->connected && - !m->disconnecting && m->peer_valid && ether_addr_equal(peer, m->connect_bssid)) { - /* A timed-out command must not turn every station dump into another - * five-second wait. A new connection/module epoch may try again. - */ - if (ret) - m->station_rate_query_failed = true; - spin_lock(&m->data_lock); - m->station_tx_rate_valid = valid; - if (valid) { - m->station_tx_rate = rate; - m->station_tx_rate_time = jiffies; - } - spin_unlock(&m->data_lock); - } - spin_unlock_irqrestore(&m->response_lock, flags); -unlock: - mutex_unlock(&m->command_mutex); -} diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index 71490e5989cb23..12d88a8d75672e 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -116,6 +116,7 @@ struct apple_nvmmu_tcb { #define APPLE_ANS_TCB_DMA_FROM_DEVICE BIT(0) #define APPLE_ANS_TCB_DMA_TO_DEVICE BIT(1) +#define APPLE_ANS_TCB_AES BIT(2) u8 dma_flags; u8 command_id; @@ -209,7 +210,6 @@ struct apple_nvme { /* Protects reset_ready against RTKit crash callbacks. */ spinlock_t event_lock; bool reset_ready; - bool handoff_disabled; struct list_head quarantine_node; resource_size_t controller_start; bool quarantined; @@ -336,33 +336,15 @@ static bool apple_nvme_can_adopt_rtkit(struct apple_nvme *anv) return false; cc = readl(anv->mmio_nvme + NVME_REG_CC); csts = readl(anv->mmio_nvme + NVME_REG_CSTS); - if (csts & NVME_CSTS_CFS) - return false; /* - * CPU_RUN/BOOT_STATUS alone survive inactive sessions. A disabled - * controller requires the producer's explicit live-session contract. - * Borrowed buffers still require reserved RAM and protected SART ranges. + * CPU_RUN and BOOT_STATUS alone survive a session that has gone + * inactive. Only a controller that the previous owner left enabled + * and ready identifies a live session. Borrowed buffers still require + * reserved RAM and protected SART ranges. */ - if (anv->handoff_disabled) - return !(cc & NVME_CC_ENABLE) && !(csts & NVME_CSTS_RDY); - return (cc & NVME_CC_ENABLE) && (csts & NVME_CSTS_RDY); -} - -static int apple_nvme_read_handoff(struct apple_nvme *anv) -{ - const char *handoff; - int ret; - - ret = of_property_read_string(anv->dev->of_node, "apple,rtkit-handoff", - &handoff); - if (ret == -EINVAL) - return 0; - if (ret || !anv->hw->needs_ioq_registers || - strcmp(handoff, "nvme-disabled-v1")) - return -EINVAL; - anv->handoff_disabled = true; - return apple_nvme_can_adopt_rtkit(anv) ? 0 : -EIO; + return (cc & NVME_CC_ENABLE) && (csts & NVME_CSTS_RDY) && + !(csts & NVME_CSTS_CFS); } static inline struct apple_nvme *queue_to_apple_nvme(struct apple_nvme_queue *q) @@ -585,8 +567,6 @@ static void apple_nvme_submit_cmd_t8103(struct apple_nvme_queue *q, struct request *req) { struct apple_nvme *anv = queue_to_apple_nvme(q); - const u8 *key = NULL; - u64 dun = 0; u32 tag = nvme_tag_from_cid(cmd->common.command_id); struct apple_nvmmu_tcb *tcb = &q->tcbs[tag]; @@ -604,16 +584,10 @@ static void apple_nvme_submit_cmd_t8103(struct apple_nvme_queue *q, tcb->dma_flags = APPLE_ANS_TCB_DMA_FROM_DEVICE; if (unlikely(req->crypt_ctx)) { - const struct blk_crypto_key *blk_key = req->crypt_ctx->bc_key; - - key = blk_key->bytes; - dun = req->crypt_ctx->bc_dun[0]; - } - - if (key) { - __le64 dun_le = cpu_to_le64(dun); + const u8 *key = req->crypt_ctx->bc_key->bytes; + __le64 dun_le = cpu_to_le64(req->crypt_ctx->bc_dun[0]); - tcb->dma_flags |= BIT(2); + tcb->dma_flags |= APPLE_ANS_TCB_AES; memcpy(tcb->aes_iv, &dun_le, sizeof(dun_le)); memset(tcb->_aes_unk, 0, sizeof(tcb->_aes_unk)); memcpy(tcb->_aes_unk, key, 4); @@ -876,12 +850,15 @@ static __always_inline void apple_nvme_unmap_rq(struct request *req) if (blk_rq_nr_phys_segments(req)) apple_nvme_unmap_data(anv, req); + if (!anv->hw->has_lsq_nvmmu) + return; + tag = nvme_tag_from_cid(iod->cmd.common.command_id); tcb = &iod->q->tcbs[tag]; - if (unlikely(tcb->dma_flags & BIT(2))) { + if (unlikely(tcb->dma_flags & APPLE_ANS_TCB_AES)) { memzero_explicit(tcb->aes_iv, sizeof(tcb->aes_iv)); memzero_explicit(tcb->_aes_unk, sizeof(tcb->_aes_unk)); - tcb->dma_flags &= ~BIT(2); + tcb->dma_flags &= ~APPLE_ANS_TCB_AES; } } @@ -2314,12 +2291,6 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev) anv->mmio_nvmmu = anv->mmio_nvme; } - ret = apple_nvme_read_handoff(anv); - if (ret) { - dev_err_probe(dev, ret, "Invalid live firmware handoff\n"); - goto out_detach_genpd; - } - if (anv->hw->has_lsq_nvmmu) { anv->adminq.sq_db = anv->mmio_nvme + APPLE_ANS_LINEAR_ASQ_DB; anv->adminq.cq_db = anv->mmio_nvme + APPLE_ANS_ACQ_DB; @@ -2444,6 +2415,7 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev) /* Released by apple_nvme_free_ctrl(), including asynchronous users. */ kref_get(&anv->ref); + if (anv->hw->has_lsq_nvmmu) anv->ctrl.crypto_profile = &anv->crypto_profile; @@ -2578,6 +2550,28 @@ static int apple_nvme_resume(struct device *dev) return nvme_reset_ctrl(&anv->ctrl); } +/* + * The ANS stays up across system sleep on post-M4 hardware, and it keeps + * using its PCIe link while it runs. genpd, however, switches a domain off + * in the noirq phase once every device attached to it has suspended, no + * matter what runtime PM says. Put the devices that hold the controller's + * domains on the awake path so that genpd leaves those domains powered + * until resume. + */ +static void apple_nvme_keep_link_powered(struct apple_nvme *anv) +{ + int i; + + /* A single domain is attached to the controller device itself. */ + if (anv->pd_count <= 1) { + device_set_awake_path(anv->dev); + return; + } + + for (i = 0; i < anv->pd_count; i++) + device_set_awake_path(anv->pd_dev[i]); +} + static int apple_nvme_suspend(struct device *dev) { struct apple_nvme *anv = dev_get_drvdata(dev); @@ -2597,6 +2591,7 @@ static int apple_nvme_suspend(struct device *dev) */ if (anv->hw->needs_ioq_registers) { cancel_work_sync(&anv->recovery_work); + apple_nvme_keep_link_powered(anv); return 0; } diff --git a/drivers/nvmem/apple-spmi-nvmem.c b/drivers/nvmem/apple-spmi-nvmem.c index 7acb0c07d6abe9..338e2e72f803df 100644 --- a/drivers/nvmem/apple-spmi-nvmem.c +++ b/drivers/nvmem/apple-spmi-nvmem.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -37,6 +38,7 @@ static int apple_spmi_nvmem_write(void *priv, unsigned int offset, void *val, static int apple_spmi_nvmem_probe(struct spmi_device *sdev) { struct regmap *regmap; + struct nvmem_device *provider; struct nvmem_config nvmem_cfg = { .dev = &sdev->dev, .name = "spmi_nvmem", @@ -54,7 +56,13 @@ static int apple_spmi_nvmem_probe(struct spmi_device *sdev) nvmem_cfg.priv = regmap; - return PTR_ERR_OR_ZERO(devm_nvmem_register(&sdev->dev, &nvmem_cfg)); + provider = devm_nvmem_register(&sdev->dev, &nvmem_cfg); + if (IS_ERR(provider)) + return PTR_ERR(provider); + if (!of_device_is_compatible(sdev->dev.of_node, "apple,baku-pmic")) + return 0; + + return devm_of_platform_populate(&sdev->dev); } static const struct of_device_id apple_spmi_nvmem_id_table[] = { diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig index 830c2c24a68ec0..7e2f0198da0d7d 100644 --- a/drivers/pci/controller/Kconfig +++ b/drivers/pci/controller/Kconfig @@ -59,17 +59,19 @@ config PCIE_APPLE config PCIE_APPLE_PIODMA_DIAG bool "J700 PIODMA diagnostic (requires full reset after use)" - depends on PCIE_APPLE != n && ARCH_APPLE && IOMMU_DMA + depends on PCIE_APPLE=y && ARCH_APPLE && IOMMU_DMA depends on PCIEASPM help Build the experimental SID17 retained bootstrap provider. It submits - one fixed request, validates typed ECAM and permits enumeration of - the two Neo functions by default on J700/T8140 only. The parameter - pcie_apple_piodma_diag.enumerate=0 disables the bootstrap. - Its arena remains owned until a full reset. While that memory is - retained, suspend, hibernate and kexec are refused at runtime on - the Neo; other machines retain their normal lifecycle support. - The provider is built-in even when the Apple PCIe host is a module. + one fixed request, validates typed ECAM and then enumerates the two + Neo radio functions when the board device tree links the PCIe host + to this engine. The enumerate=0 parameter keeps the host on its + root-port-only path instead. + The host and supplier must both be built in: supplier removal cannot + release a published DMA arena safely. Suspend-to-idle sleep retains that + arena and restores the configuration without repeating a request. + Hibernation and deeper suspend are rejected after activation. + This does not qualify memory reuse or arbitrary downstream devices. config PCI_VERSATILE bool "ARM Versatile PB PCI controller" diff --git a/drivers/pci/controller/pcie-apple-piodma-diag.c b/drivers/pci/controller/pcie-apple-piodma-diag.c index 62cd323a080e04..6f5d9bac0e64e4 100644 --- a/drivers/pci/controller/pcie-apple-piodma-diag.c +++ b/drivers/pci/controller/pcie-apple-piodma-diag.c @@ -55,9 +55,14 @@ #define PIODMA_FIFO_MASK GENMASK(5, 0) #define PIODMA_IRQ_LIMIT 16 +/* + * The board DT opts in by pointing the host's "apple,piodma" at this engine; + * a host without that property stays root-port-only. enumerate=0 keeps every + * host on its root-port-only path for diagnosis. + */ static bool enumerate = true; -module_param(enumerate, bool, 0444); -MODULE_PARM_DESC(enumerate, "Boot-time radio gate (read-only, default on; no effect off J700/T8140)"); +module_param(enumerate, bool, 0400); +MODULE_PARM_DESC(enumerate, "Prime once, validate native ECAM, and enumerate the two Neo functions (default: on)"); struct apple_piodma_request { u16 offset; @@ -99,6 +104,7 @@ struct apple_piodma_diag { u64 pointer_prefix; u8 secondary_bus; bool retained; + bool ready; bool guarded; bool radio_admitted; struct notifier_block pm_notifier; @@ -162,6 +168,14 @@ static int apple_piodma_pm_notify(struct notifier_block *nb, switch (action) { case PM_SUSPEND_PREPARE: + /* + * The suspend core marks suspend-to-idle only after these + * notifiers have run, so pm_suspend_no_platform() is still + * false here for every suspend. apple_piodma_pm_prepare() + * makes the same check once the target state is known, and + * still vetoes everything but suspend-to-idle. + */ + return NOTIFY_DONE; case PM_HIBERNATION_PREPARE: case PM_RESTORE_PREPARE: dev_warn_ratelimited(diag->dev, @@ -179,6 +193,9 @@ static int apple_piodma_pm_prepare(struct device *dev) /* Acquire admission before either PCI children or DMA pointers escape. */ if (!diag || !smp_load_acquire(&diag->guarded)) return 0; + if (pm_suspend_no_platform() && READ_ONCE(diag->ready) && + smp_load_acquire(&diag->radio_admitted)) + return 0; /* * A /dev/snapshot file may have sent its PREPARE notifier before this * supplier probed. Its later image/restore/suspend ioctls go straight @@ -189,10 +206,6 @@ static int apple_piodma_pm_prepare(struct device *dev) return -EOPNOTSUPP; } -static const struct dev_pm_ops apple_piodma_pm_ops = { - .prepare = apple_piodma_pm_prepare, -}; - static int apple_piodma_diag_root(struct apple_piodma_diag *diag, struct pci_dev *root) { int ret; @@ -733,12 +746,13 @@ int apple_piodma_bootstrap_prime(struct device *supplier, struct pci_dev *root) ret = apple_piodma_diag_initialize(diag); if (!ret) ret = apple_piodma_diag_submit(diag, 0); - if (ret) - dev_err(supplier, "bootstrap failure=%d; %s; no retry\n", ret, - diag->retained ? "retained ownership" : "no hardware publication"); - else + if (ret) { + dev_err(supplier, "bootstrap failure=%d; retained ownership; no retry\n", ret); + } else { + WRITE_ONCE(diag->ready, true); dev_info(supplier, "BOOTSTRAP_VALIDATED word=%#x; one retained slot\n", diag->slots[0].result); + } return ret; } EXPORT_SYMBOL_GPL(apple_piodma_bootstrap_prime); @@ -857,6 +871,60 @@ static void apple_piodma_diag_shutdown(struct platform_device *pdev) dev_warn(&pdev->dev, "diagnostic arena retained; require full hardware reset\n"); } +/* No new request is permitted once the host has admitted its endpoints. */ +static int apple_piodma_diag_sleep_check(struct apple_piodma_diag *diag) +{ + if (!READ_ONCE(diag->ready) || READ_ONCE(diag->irq_fault)) + return -EIO; + if (readl(diag->engine + 0x54) & BIT(0)) + return -EBUSY; + if ((readl(diag->engine + 0x0c) & PIODMA_FIFO_MASK) != 16) + return -EBUSY; + return apple_piodma_diag_check_slots(diag); +} + +static int apple_piodma_diag_suspend_noirq(struct device *dev) +{ + struct apple_piodma_diag *diag = dev_get_drvdata(dev); + + if (!diag->retained) + return 0; + if (pm_suspend_target_state != PM_SUSPEND_TO_IDLE) + return -EOPNOTSUPP; + return apple_piodma_diag_sleep_check(diag); +} + +static int apple_piodma_diag_resume_noirq(struct device *dev) +{ + struct apple_piodma_diag *diag = dev_get_drvdata(dev); + int ret; + + if (!diag->retained) + return 0; + if (!READ_ONCE(diag->ready) || READ_ONCE(diag->irq_fault)) + return -EIO; + /* Genpd may gate this island; restore configuration, never a command. */ + ret = apple_piodma_diag_initialize(diag); + if (ret) + return ret; + return apple_piodma_diag_sleep_check(diag); +} + +static int apple_piodma_diag_freeze(struct device *dev) +{ + struct apple_piodma_diag *diag = dev_get_drvdata(dev); + + return diag->retained ? -EOPNOTSUPP : 0; +} + +static const struct dev_pm_ops apple_piodma_diag_pm_ops = { + .prepare = apple_piodma_pm_prepare, + .suspend_noirq = apple_piodma_diag_suspend_noirq, + .resume_noirq = apple_piodma_diag_resume_noirq, + .freeze = apple_piodma_diag_freeze, + .poweroff = apple_piodma_diag_freeze, +}; + static const struct of_device_id apple_piodma_diag_match[] = { { .compatible = "apple,t8140-piodma-diagnostic" }, { } @@ -868,7 +936,7 @@ static struct platform_driver apple_piodma_diag_driver = { .driver = { .name = "apple-piodma-diag", .of_match_table = apple_piodma_diag_match, - .pm = pm_sleep_ptr(&apple_piodma_pm_ops), + .pm = pm_sleep_ptr(&apple_piodma_diag_pm_ops), .suppress_bind_attrs = true, }, }; diff --git a/drivers/pci/controller/pcie-apple.c b/drivers/pci/controller/pcie-apple.c index 3d75bcfee59110..621739615d7da5 100644 --- a/drivers/pci/controller/pcie-apple.c +++ b/drivers/pci/controller/pcie-apple.c @@ -1787,10 +1787,10 @@ static int apple_pcie_setup_port(struct apple_pcie *pcie, if (pcie->hw->tunneled) WRITE_ONCE(port->started, true); if (pcie->hw->root_bus_only) - dev_info(pcie->dev, - "port %pOF RUN=%#x link=%#x; downstream config blocked\n", + dev_info(pcie->dev, "port %pOF RUN=%#x link=%#x; downstream config %s\n", np, apple_pcie_port_readl(port, PORT_STATUS), - apple_pcie_port_readl(port, PORT_LINKSTS)); + apple_pcie_port_readl(port, PORT_LINKSTS), + pcie->piodma_supplier ? "waits for the bootstrap" : "blocked"); return 0; @@ -2519,12 +2519,56 @@ static int apple_pcie_neo_check_memory(struct pci_dev *root) return 0; } +/* + * Both functions must come out of the bootstrap cold. Check that before they + * are published, because publication lets their drivers bind and take over. + */ +static int apple_pcie_neo_check_cold(struct apple_pcie *pcie, struct pci_dev *root) +{ + struct pci_dev *endpoint; + u16 command; + int function, ret = 0; + + /* Check and quiesce both functions; keep the first error. */ + for (function = 0; function < 2; function++) { + endpoint = pci_get_slot(root->subordinate, PCI_DEVFN(0, function)); + if (!endpoint) { + ret = ret ?: -ENODEV; + continue; + } + if (pci_read_config_word(endpoint, PCI_COMMAND, &command)) { + ret = ret ?: -EIO; + } else { + dev_info(pcie->dev, "enumerated %s id=%04x:%04x command=%#x\n", + pci_name(endpoint), endpoint->vendor, endpoint->device, command); + if (command & PCI_COMMAND_MASTER) { + pci_clear_master(endpoint); + ret = ret ?: -EIO; + } + } + pci_dev_put(endpoint); + } + return ret; +} + +/* + * Rescan lock held, downstream config still open. Remove functions that were + * scanned but will not be published, so that a later rescan cannot publish + * them after config access has been closed again. + */ +static void apple_pcie_neo_unscan(struct pci_dev *root) +{ + struct pci_dev *dev, *tmp; + + list_for_each_entry_safe_reverse(dev, tmp, &root->subordinate->devices, bus_list) + pci_stop_and_remove_bus_device(dev); +} + static int apple_pcie_neo_enumerate(struct pci_host_bridge *bridge) { struct apple_pcie *pcie = pci_host_bridge_priv(bridge); - struct pci_dev *root, *endpoint; - u16 command; - int function, ret; + struct pci_dev *root; + int ret; root = pci_get_slot(bridge->bus, PCI_DEVFN(0, 0)); if (!root) @@ -2555,55 +2599,16 @@ static int apple_pcie_neo_enumerate(struct pci_host_bridge *bridge) dev_err(pcie->dev, "root memory forwarding unavailable after assignment: %d\n", ret); goto close_config; } - for (function = 0; function < 2; function++) { - endpoint = pci_get_slot(root->subordinate, PCI_DEVFN(0, function)); - if (!endpoint) { - ret = -ENODEV; - break; - } - if (pci_read_config_word(endpoint, PCI_COMMAND, &command)) { - ret = -EIO; - } else if (endpoint->vendor != PCI_VENDOR_ID_MEDIATEK || - endpoint->device != (function ? 0x793b : 0x7932)) { - ret = -ENODEV; - } else { - dev_info(pcie->dev, "enumerated %s id=%04x:%04x command=%#x driver=%s\n", - pci_name(endpoint), endpoint->vendor, endpoint->device, command, - endpoint->driver ? endpoint->driver->name : "unbound"); - if (command & PCI_COMMAND_MASTER) { - /* Admission requires cold, unbound endpoints. */ - pci_clear_master(endpoint); - ret = -EIO; - } - if (endpoint->driver) - ret = -EBUSY; - if (!ret) { - /* - * The radios require the equivalent of ASPM performance - * policy before their drivers start. Keep this restriction - * on their shared link, including L1 substates and Clock - * PM, without changing policy on other PCIe links. The - * PCI core tracks the restriction across policy changes. - */ - ret = pci_disable_link_state(endpoint, - PCIE_LINK_STATE_ALL); - if (ret) - pci_err(endpoint, - "cannot disable radio link power states: %d\n", - ret); - } - } - pci_dev_put(endpoint); - if (ret) - break; - } - if (!ret) { - apple_piodma_bootstrap_admitted(pcie->piodma_supplier); - pci_bus_add_devices(root->subordinate); - } -close_config: + ret = apple_pcie_neo_check_cold(pcie, root); if (ret) + goto close_config; + apple_piodma_bootstrap_admitted(pcie->piodma_supplier); + pci_bus_add_devices(root->subordinate); +close_config: + if (ret) { + apple_pcie_neo_unscan(root); WRITE_ONCE(pcie->neo_config_ready, false); + } pci_unlock_rescan_remove(); if (!ret) dev_info(pcie->dev, "NEO_RADIO_READY; functions0/1 admitted with link power states disabled\n"); @@ -2644,9 +2649,9 @@ static int apple_pcie_probe(struct platform_device *pdev) } } - if (hw->root_bus_only && - of_device_is_compatible(dev->of_node, "apple,t8140-pcie") && - apple_piodma_bootstrap_enabled()) { + /* Only a host that the board links to a bootstrap engine opens bus 1. */ + if (hw->root_bus_only && apple_piodma_bootstrap_enabled() && + of_property_present(dev->of_node, "apple,piodma")) { ret = apple_piodma_bootstrap_get(dev, &piodma_supplier); if (ret) return dev_err_probe(dev, ret, "PIODMA supplier unavailable\n"); @@ -2670,7 +2675,7 @@ static int apple_pcie_probe(struct platform_device *pdev) bridge->native_dpc = false; dev_info_once(dev, "Neo radio host: native AER/DPC recovery disabled for retained-DMA bootstrap\n"); } - if (hw->root_bus_only) + if (hw->root_bus_only && !piodma_supplier) dev_info(dev, "root-port probe only; downstream config is blocked\n"); pcie->base = devm_platform_ioremap_resource(pdev, 1); if (IS_ERR(pcie->base)) diff --git a/drivers/pci/doe.c b/drivers/pci/doe.c index 7b41da4ec11a39..44be0d513a8935 100644 --- a/drivers/pci/doe.c +++ b/drivers/pci/doe.c @@ -891,3 +891,4 @@ void pci_doe_disconnected(struct pci_dev *pdev) xa_for_each(&pdev->doe_mbs, index, doe_mb) pci_doe_cancel_tasks(doe_mb); } +EXPORT_SYMBOL_GPL(pci_doe_disconnected); diff --git a/drivers/phy/apple/Kconfig b/drivers/phy/apple/Kconfig index de01c11160ca35..0e4ae9056e35b2 100644 --- a/drivers/phy/apple/Kconfig +++ b/drivers/phy/apple/Kconfig @@ -3,6 +3,7 @@ config PHY_APPLE_ATC tristate "Apple Type-C PHY" depends on (ARM64 && ARCH_APPLE) || (COMPILE_TEST && !GENERIC_ATOMIC64) depends on TYPEC + depends on USB || !USB select GENERIC_PHY select APPLE_TUNABLE help diff --git a/drivers/phy/apple/atc-dp-tunables.h b/drivers/phy/apple/atc-dp-tunables.h new file mode 100644 index 00000000000000..6d0cb00158c71f --- /dev/null +++ b/drivers/phy/apple/atc-dp-tunables.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* Copyright 2026 Aurora Silicon */ +#ifndef APPLE_ATC_DP_TUNABLES_H +#define APPLE_ATC_DP_TUNABLES_H + +#include + +static inline bool atc_dp_list_length_valid(int length) +{ + return length >= 0 && length <= 65536 && length % 12 == 0; +} + +#endif diff --git a/drivers/phy/apple/atc-j700-dp.h b/drivers/phy/apple/atc-j700-dp.h new file mode 100644 index 00000000000000..a07f1125c2c43f --- /dev/null +++ b/drivers/phy/apple/atc-j700-dp.h @@ -0,0 +1,536 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* Copyright 2026 Aurora Silicon */ +#ifndef APPLE_ATC_J700_DP_H +#define APPLE_ATC_J700_DP_H + +/* Masks are the union of writes in the public J700 DP start/voltage/stop paths. */ +struct j700_usb_field { + u32 offset, mask; +}; + +static const struct j700_usb_field j700_usb_fields[] = { + { 0x0008, 0x00015540 }, + { 0x000c, 0x00000055 }, + { 0x010c, 0x00000555 }, + { 0x0200, 0x0000000f }, + { 0x9160, 0x00000001 }, + { 0x91d4, 0x00000003 }, + { 0x91dc, 0x00018000 }, + { 0x9210, 0x0000c00f }, + { 0x9244, 0x0007ffff }, + { 0x9248, 0x10000000 }, + { 0xb030, 0x00000c00 }, + { 0xb064, 0x0000fc0c }, + { 0xb068, 0x000001e0 }, + { 0xb06c, 0x0001ee00 }, + { 0xb0a4, 0x00000003 }, + { 0xb0a8, 0x00ff8000 }, + { 0xb0d4, 0x001fff80 }, + { 0xb0d8, 0x00000007 }, + { 0xb0dc, 0x00000018 }, + { 0xc050, 0x0000c00f }, + { 0xc0d0, 0x0017ffff }, + { 0xc0d4, 0x10000000 }, + { 0xd004, 0x33000000 }, + { 0xd00c, 0x07000000 }, + { 0xd010, 0x00000600 }, + { 0xd01c, 0x00c00000 }, + { 0xd044, 0xffffffff }, +}; + +static_assert(ARRAY_SIZE(j700_usb_fields) == 27, "USB context storage size"); + +static void j700_usb_field(unsigned int index, unsigned int lane, u32 *offset, u32 *mask) +{ + *offset = j700_usb_fields[index].offset; + *mask = j700_usb_fields[index].mask; + if (index < 4) + *mask <<= index == 3 ? 4 * lane : lane; + else + *offset += 0x7000 * lane; +} + +/* Capture before the first DP tunable or idle write; never replace a retained context. */ +static int j700_usb_capture(struct apple_atcphy *p, enum atcphy_mode mode) +{ + const struct atcphy_mode_configuration *cfg = atcphy_get_mode_config(p, mode); + + if (!p->hw->has_t8140_dp || !p->fixed_usb2) + return 0; + if (p->usb_off_pending || + p->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY || p->pipe_pending) + return -EBUSY; + for (unsigned int lane = 0; lane < 2; lane++) { + if (!cfg->dp_lane[lane] || (p->dp_usb_context_valid & BIT(lane))) + continue; + for (unsigned int i = 0; i < ARRAY_SIZE(j700_usb_fields); i++) { + u32 offset, mask; + + j700_usb_field(i, lane, &offset, &mask); + p->dp_usb_context[lane][i] = core_read32(p, offset) & mask; + } + p->dp_usb_context_valid |= BIT(lane); + } + return 0; +} + +/* Same control write/delay/poll sequence, retaining its final sample for diagnostics. */ +static int j700_usb_field_update(struct apple_atcphy *p, u32 offset, u32 mask, + u32 value, u32 *observed) +{ + core_mask32(p, offset, mask, value); + usleep_range(10, 20); + return readl_poll_timeout_atomic(p->regs.core + offset, *observed, + (*observed & mask) == value, 1, 100); +} + +/* Restore only DP-owned bits, with the PIPE parked and the DP PMA idle. */ +static int j700_usb_restore(struct apple_atcphy *p, enum atcphy_mode mode) +{ + const struct atcphy_mode_configuration *cfg = atcphy_get_mode_config(p, mode); + const unsigned int power_order[] = { 2, 1, 0, 3 }; + const unsigned int receiver_order[] = { 4, 0, 6, 2 }; + int ret; + + if (!p->hw->has_t8140_dp || !p->fixed_usb2) + return 0; + if (p->usb_off_pending && mode != APPLE_ATCPHY_MODE_OFF) + return -EBUSY; + if (p->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY || p->pipe_pending) + return -EBUSY; + if (p->dp_owned_pmas || p->dp_idle_pmas) + return -EBUSY; + if (mode == APPLE_ATCPHY_MODE_OFF && p->usb_off_drained) + return 0; + for (unsigned int lane = 0; lane < 2; lane++) { + if ((mode != APPLE_ATCPHY_MODE_OFF && + cfg->lane_mode[lane] != ACIOPHY_LANE_MODE_USB3) || + !(p->dp_usb_context_valid & BIT(lane))) + continue; + if (p->dp_usb_context_dirty & BIT(lane)) { + for (unsigned int i = 0; i < ARRAY_SIZE(j700_usb_fields); i++) { + u32 offset, mask, observed, value = p->dp_usb_context[lane][i]; + + if (i == 1 || i == 3) + continue; + j700_usb_field(i, lane, &offset, &mask); + core_mask32(p, offset, mask, value); + observed = core_read32(p, offset) & mask; + if (observed != value) { + dev_dbg(p->dev, "USB restore stage=static pma=%u reg=%#x mask=%#x expected=%#x observed=%#x\n", + lane, offset, mask, value, observed); + return -EIO; + } + } + } + /* Preserve the public PMA power bit order, using saved values. */ + for (unsigned int i = 0; i < ARRAY_SIZE(power_order); i++) { + u32 mask = BIT(4 * lane + power_order[i]); + u32 value = p->dp_usb_context[lane][3] & mask, observed; + + ret = j700_usb_field_update(p, 0x200, mask, value, &observed); + if (ret) { + dev_dbg(p->dev, "USB restore stage=power pma=%u reg=0x200 mask=%#x expected=%#x observed=%#x ret=%d\n", + lane, mask, value, observed & mask, ret); + return ret; + } + } + /* Restore receiver enables, then release its two idle controls. */ + for (unsigned int i = 0; i < ARRAY_SIZE(receiver_order); i++) { + u32 mask = BIT(lane + receiver_order[i]); + u32 value = p->dp_usb_context[lane][1] & mask, observed; + + ret = j700_usb_field_update(p, 0xc, mask, value, &observed); + if (ret) { + dev_dbg(p->dev, "USB restore stage=receiver pma=%u reg=0xc mask=%#x expected=%#x observed=%#x ret=%d\n", + lane, mask, value, observed & mask, ret); + return ret; + } + } + /* OFF keeps the shadow until domain shutdown succeeds. */ + if (mode != APPLE_ATCPHY_MODE_OFF) { + p->dp_usb_context_dirty &= ~BIT(lane); + p->dp_usb_context_valid &= ~BIT(lane); + } + } + return 0; +} + +/* Retry only the idle operations claimed by an incomplete route change. */ +static int j700_dp_idle_pending(struct apple_atcphy *p) +{ + int ret; + + for (unsigned int lane = 0; lane < 2; lane++) { + if (!(p->dp_idle_pmas & BIT(lane))) + continue; + ret = j700_pma_idle(p, lane); + if (ret) + return ret; + p->dp_idle_pmas &= ~BIT(lane); + } + return 0; +} + +/* Preserve the upper routing state established by the platform firmware. */ +static int j700_dp_route(struct apple_atcphy *p, bool four_lanes) +{ + u32 protocols = four_lanes ? 0x492 : (p->swap_lanes ? 0x252 : 0x489); + + core_mask32(p, 0x60, 0xfff, protocols); + if (four_lanes) { + core_mask32(p, 0x64, 0x1f, 0x14); + core_mask32(p, 0x64, 0x3ffe0, p->swap_lanes ? 0x100 : 0x22000); + } else { + core_mask32(p, 0x64, 0x1ffff, 0x110 | p->swap_lanes); + } + for (unsigned int lane = 0; lane < 2; lane++) { + if (!four_lanes && lane != !p->swap_lanes) + continue; + p->dp_idle_pmas |= BIT(lane); + } + return j700_dp_idle_pending(p); +} + +/* Register facts from the authorized J700 hardware audit. */ +struct j700_dp_step { + u32 offset, mask, value; + u16 delay_ns; +}; + +static void j700_dp_steps(struct apple_atcphy *p, u32 base, + const struct j700_dp_step *steps, size_t count) +{ + for (size_t i = 0; i < count; i++) { + core_mask32(p, base + steps[i].offset, steps[i].mask, steps[i].value); + if (steps[i].delay_ns) + ndelay(steps[i].delay_ns); + } +} + +static int j700_dp_wait(struct apple_atcphy *p, u32 offset, u32 mask, bool set) +{ + u32 value; + + return readl_poll_timeout_atomic(p->regs.core + offset, value, + !!(value & mask) == set, 1, 100); +} + +static void j700_dp_power(struct apple_atcphy *p, unsigned int lane) +{ + unsigned int shift = 4 * lane; + const u32 bits[] = { BIT(2), BIT(1), BIT(0), BIT(3) }; + + for (unsigned int i = 0; i < ARRAY_SIZE(bits); i++) { + u32 mask = bits[i] << shift; + + core_mask32(p, 0x200, mask, i == 2 ? 0 : mask); + udelay(1); + } + for (unsigned int bank = 0; bank < 2; bank++) { + core_clear32(p, 0xc, BIT(6 - 4 * bank + lane)); + udelay(1); + } +} + +static void j700_dp_output_power(struct apple_atcphy *p, unsigned int lane, bool tx) +{ + u32 reg = tx ? 0x10c : 0xc; + + for (unsigned int pair = 0; pair < 2; pair++) { + core_set32(p, reg, BIT(4 - 4 * pair + lane)); + core_set32(p, reg, BIT(6 - 4 * pair + lane)); + udelay(1); + } + if (!tx) { + for (unsigned int pair = 0; pair < 2; pair++) { + core_set32(p, 0x8, BIT(10 - 4 * pair + lane)); + core_set32(p, 0x8, BIT(12 - 4 * pair + lane)); + udelay(1); + } + } + reg = tx ? 0x10c : 0x8; + core_clear32(p, reg, BIT(tx ? 8 : 14) << lane); + core_set32(p, reg, BIT(tx ? 10 : 16) << lane); + udelay(1); +} + +static void j700_dp_equalize(struct apple_atcphy *p, unsigned int lane, + bool tx, u32 preset) +{ + u32 base = (tx ? 0xc000 : 0x9000) + lane * 0x7000; + u32 swing = ((preset & 3) << 14) | ((preset >> 2) & 15); + u32 taps = ((preset >> 6) & 0x3ffff) * 2 + 1; + + if (tx) + core_set32(p, base + 0xd0, BIT(20)); + core_mask32(p, base + (tx ? 0x50 : 0x210), 0xc00f, swing); + core_set32(p, base + (tx ? 0xd4 : 0x248), BIT(28)); + core_mask32(p, base + (tx ? 0xd0 : 0x244), 0x7ffff, taps); +} + +static void j700_dp_transmitter(struct apple_atcphy *p, unsigned int lane, int rate) +{ + static const struct j700_dp_step start[] = { + { 0x44, ~0U, 0x0aafb800, 500 }, + { 0x44, 0x1000000, 0x1000000, 250 }, + { 0x44, 0x4000000, 0x4000000, 250 }, + { 0x44, 0x7c0, 0x300, 250 }, + { 0x44, 0x6c000, 0, 250 }, + }; + static const struct j700_dp_step divide[] = { + { 0x44, 0x100000, 0x100000, 250 }, + { 0x44, 0x400000, 0x400000, 0 }, + }; + static const struct j700_dp_step finish[] = { + { 0x04, 0x03000000, 0x02000000, 0 }, + { 0x04, 0x30000000, 0x30000000, 0 }, + { 0x10, 0x600, 0x600, 0 }, + { 0x1c, 0xc00000, 0x800000, 0 }, + }; + u32 base = 0xd000 + lane * 0x7000; + + j700_dp_output_power(p, lane, true); + j700_dp_steps(p, base, start, ARRAY_SIZE(start)); + j700_dp_equalize(p, lane, true, 0); + if (rate != ATCPHY_DP_LINK_RATE_HBR3) + j700_dp_steps(p, base, divide, ARRAY_SIZE(divide)); + j700_dp_steps(p, base, finish, 2); + core_mask32(p, base + 0xc, 0x7000000, + rate == ATCPHY_DP_LINK_RATE_RBR ? 0x5000000 : 0x4000000); + j700_dp_steps(p, base, finish + 2, ARRAY_SIZE(finish) - 2); +} + +static void j700_dp_receiver(struct apple_atcphy *p, unsigned int lane, int rate) +{ + static const struct j700_dp_step setup[] = { + { 0xa4, 3, 2, 0 }, { 0xa8, 0x18000, 0x10000, 0 }, + { 0x6c, 0x18000, 0x10000, 0 }, + { 0xd4, 0x1f1f80, 0x121180, 0 }, { 0x64, 0xfc00, 0xb800, 0 }, + { 0xd8, 7, 4, 0 }, { 0xdc, 0x18, 0, 0 }, + { 0xd4, 0xe000, 0xc000, 100 }, + }; + static const struct j700_dp_step settle[] = { + { 0x6c, 0x600, 0x400, 100 }, { 0xd4, 0xd0000, 0x50000, 0 }, + { 0xd8, 3, 1, 500 }, { 0xd4, 0xc0000, 0, 200 }, + { 0x68, 0x180, 0x180, 200 }, { 0x68, 0x60, 0x60, 200 }, + { 0xd8, 3, 3, 1000 }, { 0xd8, 3, 2, 1000 }, + { 0xd8, 3, 0, 750 }, { 0xa8, 0x60000, 0x60000, 1000 }, + { 0x6c, 0x6000, 0x4000, 0 }, + }; + u32 top = 0x9000 + lane * 0x7000; + u32 base = top + 0x2000; + + core_set32(p, top + 0x160, 1); + core_set32(p, top + 0x1dc, 0x18000); + j700_dp_steps(p, base, setup, ARRAY_SIZE(setup)); + j700_dp_output_power(p, lane, false); + core_set32(p, base + 0x6c, BIT(11)); + core_mask32(p, top + 0x1d4, 3, 1); + ndelay(10); + core_set32(p, top + 0x1d4, 2); + j700_dp_steps(p, base, settle, ARRAY_SIZE(settle)); + j700_dp_equalize(p, lane, false, 0x3f000); + core_mask32(p, base + 0xa8, 0x380000, + rate == ATCPHY_DP_LINK_RATE_RBR ? 0x280000 : 0x200000); + core_mask32(p, base + 0xa8, 0xc00000, 0x800000); + ndelay(50); + core_set32(p, base + 0x30, 0xc00); + ndelay(100); +} + +static void j700_dp_outputs_off(struct apple_atcphy *p, unsigned int lane) +{ + static const struct j700_dp_step tx[] = { + { 0x1c, 0xc00000, 0xc00000, 0 }, { 0x10, 0x600, 0x400, 0 }, + { 0x04, 0x30000000, 0x20000000, 250 }, + { 0x44, 0x100000, 0, 250 }, { 0x44, 0x400000, 0, 250 }, + { 0x44, 0x4000000, 0, 250 }, { 0x44, 0x1000000, 0, 250 }, + { 0x44, 0x1000, 0, 0 }, + }; + static const struct j700_dp_step rx[] = { + { 0x30, 0xc00, 0x800, 50 }, { 0xa8, 0xc00000, 0xc00000, 500 }, + { 0xa8, 0x60000, 0x40000, 0 }, { 0x68, 0x60, 0x40, 200 }, + { 0x68, 0x180, 0x100, 200 }, { 0xd4, 0x10000, 0, 100 }, + { 0x6c, 0x600, 0x600, 0 }, + }; + const u32 tx_bits[] = { BIT(8), BIT(0), BIT(4) }; + const u32 rx_bits[] = { BIT(14), BIT(6), BIT(10) }; + + j700_dp_steps(p, 0xd000 + lane * 0x7000, tx, ARRAY_SIZE(tx)); + for (unsigned int i = 0; i < ARRAY_SIZE(tx_bits); i++) { + core_mask32(p, 0x10c, tx_bits[i] << lane, i ? 0 : tx_bits[i] << lane); + udelay(1); + } + j700_dp_steps(p, 0xb000 + lane * 0x7000, rx, ARRAY_SIZE(rx)); + core_mask32(p, 0x91d4 + lane * 0x7000, 3, 1); + core_clear32(p, 0xb06c + lane * 0x7000, BIT(11)); + for (unsigned int i = 0; i < ARRAY_SIZE(rx_bits); i++) { + core_mask32(p, 0x8, rx_bits[i] << lane, i ? 0 : rx_bits[i] << lane); + udelay(1); + } + for (unsigned int i = 0; i < 2; i++) { + core_clear32(p, 0xc, BIT(4 * i + lane)); + udelay(1); + } +} + +static int j700_dp_stop(struct apple_atcphy *p) +{ + int ret; + + if (!p->dp_owned_pmas) + return 0; + core_set32(p, 0x7000, BIT(0)); + for (unsigned int i = 0; i < 2; i++) + if (p->dp_owned_pmas & BIT(i)) + j700_dp_outputs_off(p, i); + core_mask32(p, 0x7000, 0xc, 8); + core_mask32(p, 0x2000, 0x1ffffff9, 0x10000001); + ret = j700_dp_wait(p, 0x2000, BIT(1), true); + if (ret) + return ret; + core_mask32(p, 0x2000, 0x1ffffff9, 0x10000018); + ret = j700_dp_wait(p, 0x2000, BIT(1), false); + if (!ret) + ret = j700_dp_wait(p, 0x7034, BIT(3), false); + if (ret) + return ret; + core_clear32(p, 0x2200, BIT(3)); + for (unsigned int i = 0; i < 2; i++) { + if (!(p->dp_owned_pmas & BIT(i))) + continue; + ret = j700_pma_idle(p, i); + if (ret) + return ret; + } + p->dp_owned_pmas = 0; + p->dp_link_rate = -1; + dev_dbg(p->dev, "J700 DP main stopped\n"); + return 0; +} + +static int j700_dp_start(struct apple_atcphy *p, unsigned int pmas, int rate, bool ssc) +{ + static const u32 parameters[4][4] = { + { 0x1e0e021c, 0, 0x156600, 0x15671b }, + { 0x1e0e01c2, 0x07fffffe, 0x155200, 0x155316 }, + { 0x1e0e01c2, 0x07fffffe, 0x554800, 0x554916 }, + { 0x1e0e02a3, 0x0bff7ffc, 0x554810, 0x564922 }, + }; + int ret; + + if (!pmas || (pmas & ~3U) || rate < 0 || (unsigned int)rate >= ARRAY_SIZE(parameters)) + return -EINVAL; + if (p->hw->has_t8140_dp && p->fixed_usb2) { + if ((p->dp_usb_context_valid & pmas) != pmas) + return -EIO; + p->dp_usb_context_dirty |= pmas; + } + /* Claim before the first write: failed setup must still be cleaned up. */ + p->dp_owned_pmas = pmas; + p->dp_link_rate = -1; + for (unsigned int i = 0; i < 2; i++) + if (pmas & BIT(i)) + j700_dp_power(p, i); + core_clear32(p, 0x2234, 3); + core_mask32(p, 0x7000, 0x1ff0, 0x490); + for (unsigned int i = 0; i < 3; i++) + writel(parameters[rate][i == 2 ? 2 + ssc : i], p->regs.core + 0x2080 + 4 * i); + core_mask32(p, 0x2208, 0x1f0000, 0x70000); + core_set32(p, 0x2218, 1); + core_set32(p, 0x2200, BIT(3)); + core_mask32(p, 0x2000, 0x1ffffff9, 0x10000001); + ret = j700_dp_wait(p, 0x2000, BIT(1), true); + if (!ret) + ret = j700_dp_wait(p, 0x7034, BIT(3), true); + if (ret) + return ret; + core_mask32(p, 0x2000, 0x1ffffff9, 0x10014000); + ret = j700_dp_wait(p, 0x2000, BIT(1), false); + if (ret) + return ret; + for (unsigned int i = 0; i < 2; i++) + if (pmas & BIT(i)) + core_mask32(p, 0xb064 + i * 0x7000, 0xc, 8); + core_set32(p, 0x7000, 0xc); + udelay(1); + for (unsigned int i = 0; i < 2; i++) { + if (!(pmas & BIT(i))) + continue; + j700_dp_transmitter(p, i, rate); + j700_dp_receiver(p, i, rate); + } + core_clear32(p, 0x7000, 1); + p->dp_link_rate = rate; + p->dp_spread_spectrum = ssc; + dev_dbg(p->dev, "J700 DP main ready: PMAs=%u rate=%d ssc=%u\n", pmas, rate, ssc); + return 0; +} + +static int j700_dp_configure(struct apple_atcphy *p, const struct phy_configure_opts_dp *o) +{ + const struct atcphy_mode_configuration *mode; + unsigned int pmas, capacity, first, table; + int rate = -1, ret; + + guard(mutex)(&p->lock); + if ((o->set_rate && !o->link_rate) || (o->set_lanes && !o->lanes)) + return o->set_voltages ? -EINVAL : j700_dp_stop(p); + /* The last completed mode cannot describe a partially changed lane route. */ + if (p->mode_dirty || p->pipe_pending || p->usb_off_pending) + return -EBUSY; + if (p->mode != APPLE_ATCPHY_MODE_DP && p->mode != APPLE_ATCPHY_MODE_USB3_DP) + return -EINVAL; + mode = atcphy_get_mode_config(p, p->mode); + pmas = !!mode->dp_lane[0] | (!!mode->dp_lane[1] << 1); + capacity = 2 * hweight32(pmas); + if (o->lanes > capacity || (o->set_lanes && o->lanes != 1 && + o->lanes != 2 && o->lanes != 4)) + return -EINVAL; + if (!p->dp_presets_valid[0]) + return -ENOENT; + if (o->set_voltages) { + if (!o->lanes) + return -EINVAL; + for (unsigned int i = 0; i < o->lanes; i++) + if (o->voltage[i] > 3 || o->pre[i] > 3) + return -EINVAL; + } + if (o->set_rate) { + const unsigned int rates[] = { 1620, 2700, 5400, 8100 }; + + for (unsigned int i = 0; i < ARRAY_SIZE(rates); i++) + if (o->link_rate == rates[i]) + rate = i; + if (rate < 0) + return -EINVAL; + if (p->dp_link_rate != rate || p->dp_owned_pmas != pmas || + p->dp_spread_spectrum != o->ssc) { + ret = j700_dp_stop(p); + if (ret) + return ret; + ret = j700_dp_start(p, pmas, rate, o->ssc); + if (ret) { + int cleanup = j700_dp_stop(p); + + dev_err(p->dev, "DP setup failed: %d, cleanup: %d\n", ret, cleanup); + return ret; + } + } + } + if (!o->set_voltages) + return 0; + if (!p->dp_owned_pmas || p->dp_link_rate < 0) + return -EINVAL; + table = p->dp_link_rate + 1; + if (!p->dp_presets_valid[table]) + table = 0; + first = pmas == 3 ? !p->swap_lanes : (pmas == 2); + for (unsigned int i = 0; i < o->lanes; i++) + j700_dp_equalize(p, first ^ (i / 2), i % 2, + p->dp_presets[table][4 * o->voltage[i] + o->pre[i]]); + return 0; +} + +#endif diff --git a/drivers/phy/apple/atc.c b/drivers/phy/apple/atc.c index 4211e4baabb8dd..1b6f5df70bf577 100644 --- a/drivers/phy/apple/atc.c +++ b/drivers/phy/apple/atc.c @@ -28,14 +28,18 @@ #include #include #include +#include #include #include #include #include +#include #include #include #include #include +#include +#include #include #include #include @@ -43,6 +47,7 @@ #include #include +#include "atc-dp-tunables.h" #include "atc-t8122-dp.h" #include "atc-tunnel.h" @@ -488,8 +493,10 @@ enum atcphy_lane_mode { #define PIPEHANDLER_OVERRIDE_VAL_PHY_STATUS BIT(4) #define PIPEHANDLER_MUX_CTRL 0x0c -#define PIPEHANDLER_MUX_CTRL_CLK GENMASK(5, 3) -#define PIPEHANDLER_MUX_CTRL_DATA GENMASK(2, 0) +#define PIPEHANDLER_MUX_CTRL_CLK_T8103 GENMASK(5, 3) +#define PIPEHANDLER_MUX_CTRL_DATA_T8103 GENMASK(2, 0) +#define PIPEHANDLER_MUX_CTRL_CLK_T8142 GENMASK(6, 4) +#define PIPEHANDLER_MUX_CTRL_DATA_T8142 GENMASK(1, 0) #define PIPEHANDLER_MUX_CTRL_CLK_OFF 0 #define PIPEHANDLER_MUX_CTRL_CLK_USB3 1 #define PIPEHANDLER_MUX_CTRL_CLK_USB4 2 @@ -621,7 +628,12 @@ struct atcphy_mode_configuration { * @aciophy_lane_mode: Lane mode register offset * @aciophy_crossbar: Crossbar register offset * @has_usb4: A USB4/Thunderbolt controller sits behind the PHY + * @pipe_mux_clk: PIPE clock selection field mask + * @pipe_mux_data: PIPE data selection field mask * @has_usb2phy_reg: The PHY has the secondary eUSB2 register bank (T8140) + * @has_t8140_dp: Uses the T8140 DisplayPort tunable bundle, AUX, lane routing + * and link training, and retains failed PIPE and mode transitions + * for a retry (the T8140 transition state below is used only here) * @optional_tunables: The bootloader may leave out the common-a tunables (this * generation has none) and the SuperSpeed tunables; USB2 * still works without the latter @@ -640,8 +652,11 @@ struct atcphy_hw { int aciophy_lane_mode; int aciophy_crossbar; bool has_usb4; + u32 pipe_mux_clk; + u32 pipe_mux_data; bool has_usb2phy_reg; bool optional_tunables; + bool has_t8140_dp; bool dp_t8122; bool park_pipe_unlocked; bool park_dummy_phy; @@ -658,6 +673,9 @@ struct atcphy_hw { * @tunables.lane_usb3: USB3 lane-specific tunables * @tunables.lane_dp: DisplayPort lane-specific tunables * @tunables.lane_usb4: USB4 lane-specific tunables + * @tunables.dp_common_pre: T8140 DisplayPort common setup + * @tunables.dp_lpdptx: T8140 DisplayPort LPDPTX setup + * @tunables.dp_common_post: T8140 DisplayPort common completion * @tunables.usb2phy_reg_dflt: Defaults for the secondary eUSB2 register bank * @hw: SoC-specific PHY description * @ss_tunables: The complete SuperSpeed tunable set was supplied @@ -665,6 +683,20 @@ struct atcphy_hw { * mode is then allowed without the USB lane tunables * @dp_only: A DisplayPort-only instance without a USB side, such as the PHY * behind the T6030 HDMI port; it maps only its core window + * @t8140_dp_tunables: The complete T8140 DisplayPort bundle was supplied + * @aux_power_requested: T8140 AUX power-up began and needs matching shutdown + * @mode_dirty: T8140 mode transition needs cleanup before it can be retried + * @dp_idle_pmas: J700 PMAs with an incomplete routing idle sequence + * @dp_owned_pmas: J700 PMAs requiring main-link cleanup + * @dp_usb_context_valid: Physical PMAs retaining their pre-DP USB field values + * @dp_usb_context_dirty: PMAs whose static fields were programmed by DP + * @usb_off_pending: Failed full OFF must finish before a new powered epoch + * @usb_off_drained: Masked recovery data was discharged before OFF began + * @pipe_pending: T8140 PIPE handoff still requires acknowledged dummy parking + * @dp_usb_context: Saved bits from the public per-PMA DP write masks + * @dp_spread_spectrum: J700 main-link spread-spectrum state + * @dp_presets: Firmware-derived per-rate drive presets + * @dp_presets_valid: Firmware supplied each drive-preset table * @fixed_usb2: The USB2 pairs go to a fixed hub on the USB controller, which * stays in host mode; the PHY provides USB2 from probe on * @typec_mode: Mode the Type-C mux last asked for; a fixed-hub port returns to @@ -702,6 +734,7 @@ struct atcphy_hw { * @regs.lpdptx: DisplayPort registers * @res: Resources for memory-mapped registers, used to verify that tunables aren't out of bounds * @res.core: Core register resource + * @res.lpdptx: DisplayPort register resource * @res.axi2af: AXI to Apple Fabric interface resource * @res.usb2phy_reg: Secondary eUSB2 register bank resource * @phys: PHY instances @@ -725,12 +758,27 @@ struct apple_atcphy { struct apple_tunable *lane_dp[2]; struct apple_tunable *lane_usb4[2]; struct apple_tunable *usb2phy_reg_dflt; + struct apple_tunable *dp_common_pre; + struct apple_tunable *dp_lpdptx; + struct apple_tunable *dp_common_post; } tunables; const struct atcphy_hw *hw; bool ss_tunables; bool dp_tunables; bool dp_only; + /* T8140 only: no other PHY ever sets the transition state below. */ + bool t8140_dp_tunables; + bool aux_power_requested; + bool mode_dirty; + u8 dp_idle_pmas; + u8 dp_owned_pmas; + u8 dp_usb_context_valid, dp_usb_context_dirty; + u32 dp_usb_context[2][27]; + bool usb_off_pending, usb_off_drained, pipe_pending; + bool dp_spread_spectrum; + u32 dp_presets[5][16]; + bool dp_presets_valid[5]; bool fixed_usb2; enum atcphy_mode typec_mode; bool host_active; @@ -770,6 +818,7 @@ struct apple_atcphy { struct { struct resource *core; + struct resource *lpdptx; struct resource *axi2af; struct resource *usb2phy_reg; } res; @@ -1029,6 +1078,21 @@ static const struct atcphy_mode_configuration *atcphy_get_mode_config(struct app return &atcphy_modes[mode].normal; } +/* Readback only; callers hold the PHY mutex and keep the fixed host live. */ +static inline u32 core_read32(struct apple_atcphy *atcphy, u32 reg) +{ + return readl(atcphy->regs.core + reg); +} + +/* Static configuration fields already read/modified by the public DP path. */ +static void j700_dp_tunables(struct apple_atcphy *atcphy) +{ + apple_tunable_apply(atcphy->regs.axi2af, atcphy->tunables.axi2af); + apple_tunable_apply(atcphy->regs.core, atcphy->tunables.dp_common_pre); + apple_tunable_apply(atcphy->regs.lpdptx, atcphy->tunables.dp_lpdptx); + apple_tunable_apply(atcphy->regs.core, atcphy->tunables.dp_common_post); +} + static void atcphy_apply_tunables(struct apple_atcphy *atcphy, enum atcphy_mode mode) { const int lane0 = atcphy->swap_lanes ? 1 : 0; @@ -1043,6 +1107,11 @@ static void atcphy_apply_tunables(struct apple_atcphy *atcphy, enum atcphy_mode if (mode == APPLE_ATCPHY_MODE_USB2 && atcphy->fixed_usb2) return; + if (atcphy->hw->has_t8140_dp && mode == APPLE_ATCPHY_MODE_DP) { + j700_dp_tunables(atcphy); + return; + } + apple_tunable_apply(atcphy->regs.core, atcphy->tunables.common[0]); apple_tunable_apply(atcphy->regs.axi2af, atcphy->tunables.axi2af); apple_tunable_apply(atcphy->regs.core, atcphy->tunables.common[1]); @@ -1079,6 +1148,8 @@ static void atcphy_apply_tunables(struct apple_atcphy *atcphy, enum atcphy_mode case APPLE_ATCPHY_MODE_USB2: break; } + if (atcphy->hw->has_t8140_dp && mode == APPLE_ATCPHY_MODE_USB3_DP) + j700_dp_tunables(atcphy); } static int atcphy_pipehandler_lock(struct apple_atcphy *atcphy) @@ -1087,12 +1158,15 @@ static int atcphy_pipehandler_lock(struct apple_atcphy *atcphy) u32 reg; if (readl(atcphy->regs.pipehandler + PIPEHANDLER_LOCK_REQ) & PIPEHANDLER_LOCK_EN) { - dev_warn(atcphy->dev, "Pipehandler already locked\n"); - return 0; + /* T8140: a request left behind does not prove the ACK; wait for it. */ + if (!atcphy->hw->has_t8140_dp) { + dev_warn(atcphy->dev, "Pipehandler already locked\n"); + return 0; + } + } else { + set32(atcphy->regs.pipehandler + PIPEHANDLER_LOCK_REQ, PIPEHANDLER_LOCK_EN); } - set32(atcphy->regs.pipehandler + PIPEHANDLER_LOCK_REQ, PIPEHANDLER_LOCK_EN); - ret = readl_poll_timeout(atcphy->regs.pipehandler + PIPEHANDLER_LOCK_ACK, reg, reg & PIPEHANDLER_LOCK_EN, 10, PIPEHANDLER_LOCK_ACK_TIMEOUT_US); if (ret) { @@ -1138,14 +1212,14 @@ static int atcphy_pipehandler_check(struct apple_atcphy *atcphy) static void atcphy_pipehandler_set_mux(struct apple_atcphy *atcphy, u32 data, u32 clk) { - mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, PIPEHANDLER_MUX_CTRL_CLK, - FIELD_PREP(PIPEHANDLER_MUX_CTRL_CLK, PIPEHANDLER_MUX_CTRL_CLK_OFF)); + mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, atcphy->hw->pipe_mux_clk, + field_prep(atcphy->hw->pipe_mux_clk, PIPEHANDLER_MUX_CTRL_CLK_OFF)); udelay(10); - mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, PIPEHANDLER_MUX_CTRL_DATA, - FIELD_PREP(PIPEHANDLER_MUX_CTRL_DATA, data)); + mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, atcphy->hw->pipe_mux_data, + field_prep(atcphy->hw->pipe_mux_data, data)); udelay(10); - mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, PIPEHANDLER_MUX_CTRL_CLK, - FIELD_PREP(PIPEHANDLER_MUX_CTRL_CLK, clk)); + mask32(atcphy->regs.pipehandler + PIPEHANDLER_MUX_CTRL, atcphy->hw->pipe_mux_clk, + field_prep(atcphy->hw->pipe_mux_clk, clk)); udelay(10); } @@ -1167,9 +1241,9 @@ static void atcphy_usb3_bist_step_dbg(struct apple_atcphy *atcphy, int step) readl(atcphy->regs.core + ACIOPHY_BIST_CFG1)); } -static void atcphy_configure_pipehandler_usb3_host_t8122(struct apple_atcphy *atcphy) +static int atcphy_configure_pipehandler_usb3_host_t8122(struct apple_atcphy *atcphy) { - int ret; + int ret, error = 0; u32 reg; core_set32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_CLOCK_EN); @@ -1182,12 +1256,16 @@ static void atcphy_configure_pipehandler_usb3_host_t8122(struct apple_atcphy *at core_set32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_BIST_CFG1, reg, (reg & ACIOPHY_BIST_CFG1_USB_EN), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_BIST_CFG1_USB_EN\n"); core_clear32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_BIST_MAC_USBPHY_STATUS2, reg, !(reg & ACIOPHY_BIST_MAC_USBPHY_STATUS2_LN0_PHY_STATUS2), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_BIST_MAC_USBPHY_STATUS2_LN0_PHY_STATUS2\n"); @@ -1204,12 +1282,16 @@ static void atcphy_configure_pipehandler_usb3_host_t8122(struct apple_atcphy *at core_set32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_BIST_CFG1, reg, (reg & ACIOPHY_BIST_CFG1_USB_EN), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_BIST_CFG1_USB_EN 2nd toggle\n"); core_clear32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_BIST_MAC_USBPHY_STATUS2, reg, !(reg & ACIOPHY_BIST_MAC_USBPHY_STATUS2_LN0_PHY_STATUS2), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_BIST_MAC_USBPHY_STATUS2_LN0_PHY_STATUS2 2nd time\n"); @@ -1225,17 +1307,20 @@ static void atcphy_configure_pipehandler_usb3_host_t8122(struct apple_atcphy *at core_set32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_BIST_CFG1, reg, (reg & ACIOPHY_BIST_CFG1_USB_EN), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_BIST_CFG1_USB_EN 3rd toggle\n"); core_clear32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_USB_EN); atcphy_usb3_bist_step_dbg(atcphy, 3); core_clear32(atcphy, ACIOPHY_BIST_CFG1, ACIOPHY_BIST_CFG1_CLOCK_EN); + return error; } -static void atcphy_configure_pipehandler_usb3_host_t8103(struct apple_atcphy *atcphy) +static int atcphy_configure_pipehandler_usb3_host_t8103(struct apple_atcphy *atcphy) { - int ret; + int ret, error = 0; u32 reg; core_set32(atcphy, ACIOPHY_TOP_BIST_PHY_CFG0, @@ -1243,6 +1328,8 @@ static void atcphy_configure_pipehandler_usb3_host_t8103(struct apple_atcphy *at core_set32(atcphy, ACIOPHY_TOP_BIST_OV_CFG, ACIOPHY_TOP_BIST_OV_CFG_LN0_RESET_N_OV); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_TOP_PHY_STAT, reg, !(reg & ACIOPHY_TOP_PHY_STAT_LN0_UNK23), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "Timed out waiting for ACIOPHY_TOP_PHY_STAT_LN0_UNK23\n"); @@ -1266,12 +1353,16 @@ static void atcphy_configure_pipehandler_usb3_host_t8103(struct apple_atcphy *at ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_TOP_PHY_STAT, reg, (reg & ACIOPHY_TOP_PHY_STAT_LN0_UNK0), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "timed out waiting for ACIOPHY_TOP_PHY_STAT_LN0_UNK0\n"); ret = readl_poll_timeout(atcphy->regs.core + ACIOPHY_TOP_PHY_STAT, reg, !(reg & ACIOPHY_TOP_PHY_STAT_LN0_UNK23), 10, 10000); + if (ret && !error) + error = ret; if (ret) dev_warn(atcphy->dev, "timed out waiting for ACIOPHY_TOP_PHY_STAT_LN0_UNK23\n"); @@ -1284,6 +1375,7 @@ static void atcphy_configure_pipehandler_usb3_host_t8103(struct apple_atcphy *at ACIOPHY_TOP_BIST_CIOPHY_CFG1_CLK_EN); core_set32(atcphy, ACIOPHY_TOP_BIST_CIOPHY_CFG1, ACIOPHY_TOP_BIST_CIOPHY_CFG1_BIST_EN); + return error; } static int atcphy_configure_pipehandler_usb3(struct apple_atcphy *atcphy, bool host) @@ -1315,19 +1407,34 @@ static int atcphy_configure_pipehandler_usb3(struct apple_atcphy *atcphy, bool h /* BIST dance */ if (atcphy->hw->gen == ATCPHY_GENERATION_T8103) - atcphy_configure_pipehandler_usb3_host_t8103(atcphy); + ret = atcphy_configure_pipehandler_usb3_host_t8103(atcphy); else - atcphy_configure_pipehandler_usb3_host_t8122(atcphy); + ret = atcphy_configure_pipehandler_usb3_host_t8122(atcphy); if (atcphy->hw->dp_t8122) dev_dbg(atcphy->dev, "USB3 host BIST done: status2=%08x ov1=%08x ov2=%08x\n", readl(atcphy->regs.core + ACIOPHY_BIST_MAC_USBPHY_STATUS2), readl(atcphy->regs.core + ACIOPHY_BIST_MAC_USBPHY_CFG_OV1), readl(atcphy->regs.core + ACIOPHY_BIST_MAC_USBPHY_CFG_OV2)); + /* T8140 retries a failed BIST; other PHYs carry on as before. */ + if (ret && atcphy->hw->has_t8140_dp) { + int cleanup; + + clear32(atcphy->regs.pipehandler + PIPEHANDLER_OVERRIDE, + PIPEHANDLER_OVERRIDE_RXVALID | PIPEHANDLER_OVERRIDE_RXDETECT); + cleanup = atcphy_pipehandler_unlock(atcphy); + if (cleanup) + dev_warn(atcphy->dev, "PIPE cleanup after BIST: %d\n", cleanup); + /* Native BIST changed state; a retry must rebuild after parking. */ + atcphy->mode_dirty = true; + return ret; + } } /* Configure PIPE mux to USB3 PHY */ atcphy_pipehandler_set_mux(atcphy, PIPEHANDLER_MUX_CTRL_DATA_USB3, PIPEHANDLER_MUX_CTRL_CLK_USB3); + /* Track the selected backend even if its completion ACK fails. */ + atcphy->pipe_state = ATCPHY_PIPEHANDLER_STATE_USB3; /* Remove link detection override */ clear32(atcphy->regs.pipehandler + PIPEHANDLER_OVERRIDE, PIPEHANDLER_OVERRIDE_RXVALID); @@ -1336,8 +1443,13 @@ static int atcphy_configure_pipehandler_usb3(struct apple_atcphy *atcphy, bool h /* Pipehandler was only locked when the BIST sequence was applied for host mode */ if (host) { ret = atcphy_pipehandler_unlock(atcphy); - if (ret) + if (ret) { dev_warn(atcphy->dev, "Failed to unlock pipehandler"); + if (atcphy->hw->has_t8140_dp) { + atcphy->mode_dirty = true; + return ret; + } + } } return 0; @@ -1409,11 +1521,19 @@ static bool atcphy_needs_dummy_phy(struct apple_atcphy *atcphy) static int atcphy_configure_pipehandler_dummy(struct apple_atcphy *atcphy, bool lock) { + /* pipe_pending is only ever set on T8140, see atcphy_configure_pipehandler(). */ + bool live_usb3 = atcphy->pipe_pending || + (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2 && atcphy->host_active && + atcphy->pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3); int ret; ret = atcphy_pipehandler_check(atcphy); - if (ret) + if (ret) { + if (live_usb3) + if (atcphy->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY) + atcphy->mode_dirty = true; return ret; + } /* Force disable link detection */ clear32(atcphy->regs.pipehandler + PIPEHANDLER_OVERRIDE_VALUES, @@ -1423,8 +1543,15 @@ static int atcphy_configure_pipehandler_dummy(struct apple_atcphy *atcphy, bool if (lock) { ret = atcphy_pipehandler_lock(atcphy); - if (ret) + if (ret) { + /* A live controller must acknowledge before its backend changes. */ + if (live_usb3) { + if (atcphy->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY) + atcphy->mode_dirty = true; + return ret; + } dev_warn(atcphy->dev, "Failed to lock pipehandler"); + } } /* Switch to dummy PHY */ @@ -1433,8 +1560,18 @@ static int atcphy_configure_pipehandler_dummy(struct apple_atcphy *atcphy, bool if (lock) { ret = atcphy_pipehandler_unlock(atcphy); - if (ret) + if (ret) { + /* + * The mux changed, but the handoff has not completed. + * Retain ownership. + */ + if (live_usb3) { + if (atcphy->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY) + atcphy->mode_dirty = true; + return ret; + } dev_warn(atcphy->dev, "Failed to unlock pipehandler"); + } } mask32(atcphy->regs.pipehandler + PIPEHANDLER_NONSELECTED_OVERRIDE, @@ -1461,6 +1598,8 @@ static int atcphy_configure_pipehandler(struct apple_atcphy *atcphy, bool host) switch (state) { case ATCPHY_PIPEHANDLER_STATE_USB3: + if (atcphy->hw->has_t8140_dp) + atcphy->pipe_pending = true; ret = atcphy_configure_pipehandler_usb3(atcphy, host); break; case ATCPHY_PIPEHANDLER_STATE_USB4: @@ -1482,7 +1621,11 @@ static int atcphy_configure_pipehandler(struct apple_atcphy *atcphy, bool host) readl(atcphy->regs.pipehandler + PIPEHANDLER_LOCK_ACK), readl(atcphy->regs.pipehandler + PIPEHANDLER_AON_GEN), readl(atcphy->regs.pipehandler + PIPEHANDLER_NONSELECTED_OVERRIDE)); - atcphy->pipe_state = state; + /* T8140 keeps a failed attempt retryable; other PHYs configure once per mode. */ + if (!ret || !atcphy->hw->has_t8140_dp) { + atcphy->pipe_state = state; + atcphy->pipe_pending = false; + } return ret; } @@ -1495,22 +1638,28 @@ static int atcphy_configure_pipehandler(struct apple_atcphy *atcphy, bool host) * controller no longer runs, so the request times out. On PHYs with * park_pipe_unlocked the mux is then switched without it. */ -static void atcphy_park_pipehandler(struct apple_atcphy *atcphy, bool stopped) +static int atcphy_park_pipehandler(struct apple_atcphy *atcphy, bool stopped) { bool lock = !(stopped && atcphy->hw->park_pipe_unlocked); int ret; lockdep_assert_held(&atcphy->lock); - if (atcphy->pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY) - return; + if (atcphy->pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && !atcphy->pipe_pending) + return 0; if (!lock) dev_dbg(atcphy->dev, "Parking the PIPE of a stopped controller\n"); ret = atcphy_configure_pipehandler_dummy(atcphy, lock); - if (ret) + if (ret) { dev_warn(atcphy->dev, "Failed to switch PIPE to dummy: %d\n", ret); + /* T8140 retains a failed handoff; other PHYs park regardless. */ + if (atcphy->hw->has_t8140_dp) + return ret; + } atcphy->pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + atcphy->pipe_pending = false; + return 0; } static void atcphy_setup_pipehandler(struct apple_atcphy *atcphy) @@ -1520,14 +1669,65 @@ static void atcphy_setup_pipehandler(struct apple_atcphy *atcphy) atcphy_pipehandler_set_mux(atcphy, PIPEHANDLER_MUX_CTRL_DATA_DUMMY, PIPEHANDLER_MUX_CTRL_CLK_DUMMY); atcphy->pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + atcphy->pipe_pending = false; if (atcphy_needs_dummy_phy(atcphy)) atcphy_enable_dummy_phy(atcphy); } -static void atcphy_configure_lanes(struct apple_atcphy *atcphy, enum atcphy_mode mode) +#define J700_PMA_RECEIVER_CONTROL 0x000c +#define J700_PMA_POWER_CONTROL 0x0200 + +static int j700_field_update(struct apple_atcphy *atcphy, u32 offset, u32 mask, u32 value) +{ + u32 observed; + + core_mask32(atcphy, offset, mask, value); + usleep_range(10, 20); + return readl_poll_timeout_atomic(atcphy->regs.core + offset, observed, + (observed & mask) == value, 1, 100); +} + +static int j700_pma_idle(struct apple_atcphy *atcphy, unsigned int lane) +{ + u32 field; + int ret; + + if (lane >= 2) + return -EINVAL; + for (unsigned int supply = 0; supply < 2; supply++) { + field = BIT(2 + lane + 4 * supply); + ret = j700_field_update(atcphy, J700_PMA_RECEIVER_CONTROL, field, field); + if (ret) + return ret; + } + field = BIT(4 * lane + 3); + ret = j700_field_update(atcphy, J700_PMA_POWER_CONTROL, field, 0); + if (ret) + return ret; + field = BIT(4 * lane); + ret = j700_field_update(atcphy, J700_PMA_POWER_CONTROL, field, field); + if (ret) + return ret; + for (unsigned int control = 1; control <= 2; control++) { + field = BIT(4 * lane + control); + ret = j700_field_update(atcphy, J700_PMA_POWER_CONTROL, field, 0); + if (ret) + return ret; + } + return 0; +} + +#include "atc-j700-dp.h" + +static int atcphy_configure_lanes(struct apple_atcphy *atcphy, enum atcphy_mode mode) { const struct atcphy_mode_configuration *mode_cfg = atcphy_get_mode_config(atcphy, mode); + if (atcphy->hw->has_t8140_dp && + (mode == APPLE_ATCPHY_MODE_USB3 || mode == APPLE_ATCPHY_MODE_USB3_DP || + mode == APPLE_ATCPHY_MODE_DP)) + return j700_dp_route(atcphy, mode == APPLE_ATCPHY_MODE_DP); + core_mask32(atcphy, atcphy->hw->aciophy_lane_mode, ACIOPHY_LANE_MODE_RX0, FIELD_PREP(ACIOPHY_LANE_MODE_RX0, mode_cfg->lane_mode[0])); core_mask32(atcphy, atcphy->hw->aciophy_lane_mode, ACIOPHY_LANE_MODE_TX0, @@ -1556,7 +1756,7 @@ static void atcphy_configure_lanes(struct apple_atcphy *atcphy, enum atcphy_mode * are started by the link rate sequence, see atcphy_dp_set_rate_t8122(). */ if (atcphy->hw->gen == ATCPHY_GENERATION_T8122) - return; + return 0; if (mode_cfg->dp_lane[0]) { core_set32(atcphy, LN0_AUSPMA_RX_TOP + LN_AUSPMA_RX_TOP_PMAFSM, @@ -1581,13 +1781,69 @@ static void atcphy_configure_lanes(struct apple_atcphy *atcphy, enum atcphy_mode LN_AUSPMA_RX_TOP_PMAFSM_PCS_OV); udelay(10); } + return 0; +} + +#define J700_AUX_ANALOG_CONTROL 0x16000 +#define J700_AUX_SUPPLY_CONTROL 0x16400 +#define J700_AUX_ANALOG_OFF BIT(0) +#define J700_AUX_CLAMP_SELECT BIT(8) +#define J700_AUX_CLAMP_ENABLE BIT(9) + +static int j700_aux_enable(struct apple_atcphy *atcphy) +{ + u32 clamp = J700_AUX_CLAMP_SELECT | J700_AUX_CLAMP_ENABLE; + int ret; + + /* Even an incomplete power-up must be unwound before the domain is stopped. */ + atcphy->aux_power_requested = true; + for (unsigned int stage = 0; stage < 2; stage++) { + u32 fields = 3U << (2 * stage); + + ret = j700_field_update(atcphy, J700_AUX_SUPPLY_CONTROL, fields, fields); + if (ret) + return ret; + } + ret = j700_field_update(atcphy, J700_AUX_SUPPLY_CONTROL, clamp, J700_AUX_CLAMP_SELECT); + if (ret) + return ret; + + return j700_field_update(atcphy, J700_AUX_ANALOG_CONTROL, J700_AUX_ANALOG_OFF, 0); +} + +static int j700_aux_disable(struct apple_atcphy *atcphy) +{ + u32 clamp = J700_AUX_CLAMP_SELECT | J700_AUX_CLAMP_ENABLE; + int ret; + + if (!atcphy->aux_power_requested) + return 0; + ret = j700_field_update(atcphy, J700_AUX_ANALOG_CONTROL, + J700_AUX_ANALOG_OFF, J700_AUX_ANALOG_OFF); + if (ret) + return ret; + ret = j700_field_update(atcphy, J700_AUX_SUPPLY_CONTROL, clamp, clamp); + if (ret) + return ret; + for (unsigned int remaining = 2; remaining; remaining--) { + u32 override = BIT(2 * remaining - 1); + + ret = j700_field_update(atcphy, J700_AUX_SUPPLY_CONTROL, override, 0); + if (ret) + return ret; + } + atcphy->aux_power_requested = false; + return 0; } -static void atcphy_enable_dp_aux(struct apple_atcphy *atcphy) +static int atcphy_enable_dp_aux(struct apple_atcphy *atcphy) { + if (atcphy->hw->has_t8140_dp) + return j700_aux_enable(atcphy); + /* T8122 PHYs with DP support enable AUX in atcphy_dp_aux_on_t8122() */ if (atcphy->hw->gen == ATCPHY_GENERATION_T8122) - return; + return 0; core_set32(atcphy, ACIOPHY_LANE_DP_CFG_BLK_TX_DP_CTRL0, DPTXPHY_PMA_LANE_RESET_N); core_set32(atcphy, ACIOPHY_LANE_DP_CFG_BLK_TX_DP_CTRL0, DPTXPHY_PMA_LANE_RESET_N_OV); @@ -1637,10 +1893,17 @@ static void atcphy_enable_dp_aux(struct apple_atcphy *atcphy) FIELD_PREP(LPDPTX_AUX_MARGIN_RCAL_TXSWING, 12)); atcphy->dp_link_rate = -1; + return 0; } -static void atcphy_disable_dp_aux(struct apple_atcphy *atcphy) +static int atcphy_disable_dp_aux(struct apple_atcphy *atcphy) { + if (atcphy->hw->has_t8140_dp) { + int ret = j700_dp_stop(atcphy); + + return ret ? ret : j700_aux_disable(atcphy); + } + /* T8122 PHYs with DP support stop AUX in atcphy_dp_stop_t8122() */ if (atcphy->hw->gen != ATCPHY_GENERATION_T8122) { set32(atcphy->regs.lpdptx + LPDPTX_AUX_CONTROL, LPDPTX_AUX_PWN_DOWN); @@ -1655,6 +1918,7 @@ static void atcphy_disable_dp_aux(struct apple_atcphy *atcphy) core_clear32(atcphy, ACIOPHY_LANE_DP_CFG_BLK_TX_DP_CTRL0, DPRX_PCLK_ENABLE); core_clear32(atcphy, ACIOPHY_LANE_DP_CFG_BLK_TX_DP_CTRL0, DPTX_PCLK1_ENABLE); core_clear32(atcphy, ACIOPHY_LANE_DP_CFG_BLK_TX_DP_CTRL0, DPTX_PCLK2_ENABLE); + return 0; } static int atcphy_dp_configure_lane(struct apple_atcphy *atcphy, enum atcphy_lane lane, @@ -2021,7 +2285,9 @@ static int atcphy_power_off(struct apple_atcphy *atcphy) u32 reg; int ret; - atcphy_disable_dp_aux(atcphy); + ret = atcphy_disable_dp_aux(atcphy); + if (ret) + return ret; /* Enable all reset lines */ core_clear32(atcphy, ATCPHY_POWER_CTRL, ATCPHY_POWER_PHY_RESET_N); @@ -2995,6 +3261,40 @@ static int atcphy_configure(struct apple_atcphy *atcphy, enum atcphy_mode mode) u32 reg; lockdep_assert_held(&atcphy->lock); + /* The T8140 transition state is never set on other PHYs. */ + if (atcphy->pipe_pending) + return -EBUSY; + if (atcphy->usb_off_pending && mode != APPLE_ATCPHY_MODE_OFF) + return -EBUSY; + if (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2 && + atcphy->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY) + return -EBUSY; + if (atcphy->hw->has_t8140_dp && atcphy->dp_owned_pmas && + (mode != atcphy->mode || atcphy->mode_dirty)) { + atcphy->mode_dirty = true; + ret = j700_dp_stop(atcphy); + if (ret) + return ret; + } + if (atcphy->hw->has_t8140_dp && atcphy->mode_dirty) { + ret = j700_dp_idle_pending(atcphy); + if (ret) + return ret; + } + if (atcphy->hw->has_t8140_dp && atcphy_modes[mode].enable_dp_aux && + !atcphy->t8140_dp_tunables) + return -EOPNOTSUPP; + if (atcphy->hw->has_t8140_dp && + (!atcphy_modes[mode].enable_dp_aux || atcphy->mode_dirty) && + atcphy->aux_power_requested) { + atcphy->mode_dirty = true; + ret = j700_aux_disable(atcphy); + if (ret) + return ret; + } + /* T8140 keeps failed transitions retryable even when the mode is unchanged. */ + if (atcphy->hw->has_t8140_dp) + atcphy->mode_dirty = true; /* the mode change below may power the PHY down: stop the tunnel clock first */ if (apple_atc_tunnel_is_t6030(atcphy)) { ret = atc_t8122_tunnel_stop(atcphy); @@ -3008,15 +3308,49 @@ static int atcphy_configure(struct apple_atcphy *atcphy, enum atcphy_mode mode) atcphy_dp_stop_t8122(atcphy); if (mode == APPLE_ATCPHY_MODE_OFF) { + if (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2 && + (atcphy->dp_usb_context_valid || atcphy->usb_off_pending)) { + atcphy->usb_off_pending = true; + if (!atcphy->usb_off_drained) { + ret = j700_usb_restore(atcphy, APPLE_ATCPHY_MODE_OFF); + if (ret) + return ret; + atcphy->usb_off_drained = true; + } + } ret = atcphy_power_off(atcphy); + if (ret && atcphy->hw->has_t8140_dp) + return ret; + /* Recovery data was discharged before OFF, not discarded while dirty. */ + atcphy->dp_usb_context_valid = 0; + atcphy->dp_usb_context_dirty = 0; + atcphy->usb_off_pending = false; + atcphy->usb_off_drained = false; atcphy->mode = mode; + atcphy->mode_dirty = false; return ret; } + /* + * Fixed-hub SAFE keeps USB2 up rather than taking the block OFF. + * With PIPE parked and DP owners released, reinitialize USB3 before + * programming its lanes; the existing sequence releases this reset. + */ + if (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2) + core_clear32(atcphy, ATCPHY_POWER_CTRL, ATCPHY_POWER_PHY_RESET_N); + ret = atcphy_power_on(atcphy); if (ret) return ret; + if (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2) { + ret = j700_usb_capture(atcphy, mode); + if (!ret) + ret = j700_usb_restore(atcphy, mode); + if (ret) + return ret; + } + atcphy_apply_tunables(atcphy, mode); if (atcphy->hw->gen == ATCPHY_GENERATION_T8103) { @@ -3067,15 +3401,20 @@ static int atcphy_configure(struct apple_atcphy *atcphy, enum atcphy_mode mode) } /* Setup AUX channel if DP altmode is requested */ - if (atcphy_modes[mode].enable_dp_aux) - atcphy_enable_dp_aux(atcphy); + if (atcphy_modes[mode].enable_dp_aux) { + ret = atcphy_enable_dp_aux(atcphy); + if (ret) + return ret; + } /* Enable clocks and configure lanes */ if (atcphy->hw->gen == ATCPHY_GENERATION_T8103) { core_set32(atcphy, CIO3PLL_CLK_CTRL, CIO3PLL_CLK_PCLK_EN); core_set32(atcphy, CIO3PLL_CLK_CTRL, CIO3PLL_CLK_REFCLK_EN); } - atcphy_configure_lanes(atcphy, mode); + ret = atcphy_configure_lanes(atcphy, mode); + if (ret) + return ret; /* Take the USB3 PHY out of reset */ core_set32(atcphy, ATCPHY_POWER_CTRL, ATCPHY_POWER_PHY_RESET_N); @@ -3089,6 +3428,7 @@ static int atcphy_configure(struct apple_atcphy *atcphy, enum atcphy_mode mode) } atcphy->mode = mode; + atcphy->mode_dirty = false; /* T8122: the AUX block needs the PHY out of reset and calibrated */ if (atcphy->hw->dp_t8122 && atcphy_modes[mode].enable_dp_aux) @@ -3216,11 +3556,27 @@ static int atcphy_usb2_init(struct phy *phy) { struct apple_atcphy *atcphy = phy_get_drvdata(phy); - if (!atcphy->fixed_usb2) + if (!atcphy->fixed_usb2 && !atcphy->hw->has_t8140_dp) return 0; guard(mutex)(&atcphy->lock); atcphy_restore_after_pd_off(atcphy); + if (atcphy->usb_off_pending) + return -EBUSY; + if (atcphy->pipe_pending && + (atcphy->mode == APPLE_ATCPHY_MODE_OFF || atcphy->mode_dirty)) { + int ret = atcphy_park_pipehandler(atcphy, false); + + if (ret) + return ret; + } + + /* T8140: a delayed shutdown may have completed after the last Type-C event. */ + if (!atcphy->fixed_usb2) { + if (atcphy->mode == APPLE_ATCPHY_MODE_OFF || atcphy->mode_dirty) + return atcphy_configure(atcphy, atcphy->typec_mode); + return 0; + } /* * dwc3 initialises its PHYs after releasing its reset, and asserting that @@ -3231,7 +3587,7 @@ static int atcphy_usb2_init(struct phy *phy) * the Type-C mux last asked for, USB2 until a cable event says otherwise. */ set32(atcphy->regs.usb2phy + USB2PHY_SIG, USB2PHY_SIG_HOST); - if (atcphy->mode == APPLE_ATCPHY_MODE_OFF) + if (atcphy->mode == APPLE_ATCPHY_MODE_OFF || atcphy->mode_dirty) return atcphy_configure(atcphy, atcphy->typec_mode); atcphy_usb2_power_on(atcphy); @@ -3248,11 +3604,15 @@ static const struct phy_ops apple_atc_usb2_phy_ops = { static int atcphy_usb3_power_off(struct phy *phy) { struct apple_atcphy *atcphy = phy_get_drvdata(phy); + int ret; guard(mutex)(&atcphy->lock); atcphy_restore_after_pd_off(atcphy); - atcphy_park_pipehandler(atcphy, true); + /* Only T8140 reports a failed park, and then keeps the controller's PIPE. */ + ret = atcphy_park_pipehandler(atcphy, true); + if (ret) + return ret; atcphy->host_active = false; /* @@ -3260,13 +3620,17 @@ static int atcphy_usb3_power_off(struct phy *phy) * e.g. when a monitor's built-in hub re-enumerates after a hot-plug. * Powering the PHY off then cuts the DP lanes under DCP, which reports a * DPTX FIFO error and drops the display. The Type-C mux turns the PHY off - * when the alt mode actually ends, so leave a DP mode alone here. + * when the alt mode actually ends, so leave a DP mode alone here, unless + * a failed T8140 transition left it to be cleaned up. */ - if (atcphy_modes[atcphy->mode].enable_dp_aux) + if (atcphy_modes[atcphy->mode].enable_dp_aux && !atcphy->mode_dirty) return 0; - if (atcphy->mode != APPLE_ATCPHY_MODE_OFF) - atcphy_configure(atcphy, APPLE_ATCPHY_MODE_OFF); + if (atcphy->mode != APPLE_ATCPHY_MODE_OFF || atcphy->mode_dirty) { + ret = atcphy_configure(atcphy, APPLE_ATCPHY_MODE_OFF); + if (ret && atcphy->hw->has_t8140_dp) + return ret; + } return 0; } @@ -3278,6 +3642,9 @@ static int atcphy_usb3_set_mode(struct phy *phy, enum phy_mode mode, int submode guard(mutex)(&atcphy->lock); atcphy_restore_after_pd_off(atcphy); + if (atcphy->usb_off_pending) + return -EBUSY; + switch (mode) { case PHY_MODE_USB_HOST: atcphy->host_active = true; @@ -3294,7 +3661,18 @@ static int atcphy_usb3_set_mode(struct phy *phy, enum phy_mode mode, int submode * driver and then another one from the generic xhci code) but must only configure the * PIPE handler once. Nothing needs to be done either when the PIPE is already routed to * the backend the current mode uses, which includes the dummy backend of the USB2 modes. + * + * T8140: a failed lane/domain transition must be restored, and a failed + * handoff parked, before the PIPE is routed again. */ + if (atcphy->mode_dirty) + return -EBUSY; + if (atcphy->pipe_pending) { + int ret = atcphy_park_pipehandler(atcphy, false); + + if (ret) + return ret; + } if (atcphy->pipe_state == atcphy_modes[atcphy->mode].pipehandler_state) return 0; @@ -3366,6 +3744,8 @@ static int atcphy_dpphy_configure(struct phy *phy, union phy_configure_opts *opt struct apple_atcphy *atcphy = phy_get_drvdata(phy); enum atcphy_dp_link_rate link_rate; + if (atcphy->hw->has_t8140_dp) + return j700_dp_configure(atcphy, opts); if (atcphy->hw->dp_t8122) return atcphy_dpphy_configure_t8122(atcphy, opts); @@ -3588,10 +3968,18 @@ static void _atcphy_dwc3_reset_assert(struct apple_atcphy *atcphy) static int atcphy_dwc3_reset_assert(struct reset_controller_dev *rcdev, unsigned long id) { struct apple_atcphy *atcphy = container_of(rcdev, struct apple_atcphy, rcdev); + int ret; guard(mutex)(&atcphy->lock); atcphy_restore_after_pd_off(atcphy); + /* T8140: do not reset the controller or USB2 PHY if PIPE parking fails. */ + if (atcphy->hw->has_t8140_dp) { + ret = atcphy_park_pipehandler(atcphy, true); + if (ret) + return ret; + } + _atcphy_dwc3_reset_assert(atcphy); atcphy_park_pipehandler(atcphy, true); atcphy->host_active = false; @@ -3647,9 +4035,14 @@ static int atcphy_sw_set(struct typec_switch_dev *sw, enum typec_orientation ori case TYPEC_ORIENTATION_NONE: break; case TYPEC_ORIENTATION_NORMAL: + /* The mode alone cannot describe a T8140 route after its orientation changes. */ + if (atcphy->hw->has_t8140_dp && atcphy->swap_lanes) + atcphy->mode_dirty = true; atcphy->swap_lanes = false; break; case TYPEC_ORIENTATION_REVERSE: + if (atcphy->hw->has_t8140_dp && !atcphy->swap_lanes) + atcphy->mode_dirty = true; atcphy->swap_lanes = true; break; } @@ -3680,7 +4073,208 @@ static int atcphy_probe_switch(struct apple_atcphy *atcphy) return devm_add_action_or_reset(atcphy->dev, atcphy_typec_switch_unregister, sw); } -static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *state) +/* + * The fixed USB2 hub keeps the host controller active, but its independent + * SuperSpeed root hub may autosuspend while PIPE still needs it awake. + * Own a bounded root-hub resume reference across a Type-C handoff. Resume + * precedes the ATC lock: USB PM can call back into the PHY on recovery. + */ +struct atcphy_usb3_pm { + struct platform_device *controller; + struct device *host; + struct usb_device *root; + struct usb_device *link; +}; + +#if IS_REACHABLE(CONFIG_USB) +/* The root's runtime state does not cover selective suspend of its port. */ +static int atcphy_usb3_link_get(struct atcphy_usb3_pm *pm) +{ + struct usb_device *link; + int ret; + + /* The graph identifies the controller, not a port in a multiport root. */ + if (pm->root->maxchild != 1) + return pm->root->maxchild ? -ENODEV : -EAGAIN; + link = usb_hub_find_child(pm->root, 1); + if (!link) + return 0; + pm->link = usb_get_dev(link); + usb_lock_device(pm->link); + if (READ_ONCE(pm->link->state) == USB_STATE_NOTATTACHED) { + ret = 0; + goto unlock_link; + } + ret = pm_runtime_resume_and_get(&pm->link->dev); + if (ret < 0) + goto unlock_link; + if (READ_ONCE(pm->link->state) == USB_STATE_NOTATTACHED) { + pm_runtime_put_noidle(&pm->link->dev); + ret = -ENODEV; + goto unlock_link; + } + return 0; + +unlock_link: + usb_unlock_device(pm->link); + usb_put_dev(pm->link); + pm->link = NULL; + return ret; +} + +static void atcphy_usb3_link_put(struct atcphy_usb3_pm *pm) +{ + if (!pm->link) + return; + usb_mark_last_busy(pm->link); + pm_runtime_put_autosuspend(&pm->link->dev); + usb_unlock_device(pm->link); + usb_put_dev(pm->link); + pm->link = NULL; +} + +static int atcphy_usb3_pm_find(struct usb_device *udev, void *data) +{ + struct atcphy_usb3_pm *pm = data; + + if (udev->parent || udev->speed < USB_SPEED_SUPER || + udev->bus->sysdev != &pm->controller->dev) + return 0; + + pm->root = usb_get_dev(udev); + return 1; +} +#endif + +static int atcphy_usb3_pm_get(struct apple_atcphy *atcphy, struct atcphy_usb3_pm *pm) +{ +#if IS_REACHABLE(CONFIG_USB) + struct device_node *ep __free(device_node) = NULL; + struct device_node *ctrl __free(device_node) = NULL; + struct usb_hcd *hcd; + int ret; +#endif + + if (!atcphy->fixed_usb2 || !atcphy->hw->has_t8140_dp) + return 0; + +#if IS_REACHABLE(CONFIG_USB) + ep = of_graph_get_endpoint_by_regs(atcphy->np, 1, -1); + ctrl = ep ? of_graph_get_remote_port_parent(ep) : NULL; + if (!ctrl) + return -ENODEV; + pm->controller = of_find_device_by_node(ctrl); + if (!pm->controller) + return -EAGAIN; + + /* Driver core takes this lock before DWC3 removal touches SUSPHY. */ + device_lock(&pm->controller->dev); + if (!pm->controller->dev.driver) { + ret = -EAGAIN; + goto put_controller; + } + if (strcmp(pm->controller->dev.driver->name, "dwc3-apple")) { + ret = -ENODEV; + goto put_controller; + } + + /* Firmware identity, not a bus number, binds this PHY to its root hub. */ + usb_for_each_dev(pm, atcphy_usb3_pm_find); + if (!pm->root) { + ret = -EAGAIN; + goto put_controller; + } + + /* Also serialize direct xHCI unbind, before it marks the HCD removing. */ + pm->host = get_device(pm->root->bus->controller); + device_lock(pm->host); + if (!pm->host->driver) { + ret = -EAGAIN; + goto put_root; + } + if (pm->host->parent != &pm->controller->dev || + strcmp(pm->host->driver->name, "xhci-hcd")) { + ret = -ENODEV; + goto put_root; + } + + /* USB destruction follows the platform lifecycle locks, not vice versa. */ + usb_lock_device(pm->root); + hcd = bus_to_hcd(pm->root->bus); + if (READ_ONCE(pm->root->state) == USB_STATE_NOTATTACHED || HCD_DEAD(hcd)) { + ret = -ENODEV; + goto unlock_root; + } + ret = pm_runtime_resume_and_get(&pm->root->dev); + if (ret < 0) + goto unlock_root; + if (READ_ONCE(pm->root->state) == USB_STATE_NOTATTACHED || + HCD_DEAD(hcd) || !HCD_HW_ACCESSIBLE(hcd)) { + pm_runtime_put_noidle(&pm->root->dev); + ret = -ENODEV; + goto unlock_root; + } + /* Resume the attached upstream link before taking the ATC mutex. */ + ret = atcphy_usb3_link_get(pm); + if (ret) { + usb_mark_last_busy(pm->root); + pm_runtime_put_autosuspend(&pm->root->dev); + goto unlock_root; + } + if (READ_ONCE(pm->root->state) == USB_STATE_NOTATTACHED || + READ_ONCE(hcd->state) != HC_STATE_RUNNING || + HCD_DEAD(hcd) || !HCD_HW_ACCESSIBLE(hcd)) { + atcphy_usb3_link_put(pm); + pm_runtime_put_noidle(&pm->root->dev); + ret = -ENODEV; + goto unlock_root; + } + return 0; + +unlock_root: + usb_unlock_device(pm->root); +put_root: + device_unlock(pm->host); + put_device(pm->host); + pm->host = NULL; + usb_put_dev(pm->root); + pm->root = NULL; +put_controller: + device_unlock(&pm->controller->dev); + put_device(&pm->controller->dev); + pm->controller = NULL; + return ret; +#else + /* typec_mux_set() skips EOPNOTSUPP, which would report a false success. */ + return -ENODEV; +#endif +} + +static void atcphy_usb3_pm_put(struct atcphy_usb3_pm *pm) +{ +#if IS_REACHABLE(CONFIG_USB) + if (!pm->root) + return; + + /* Release child ownership before the root: keep the lock order hierarchical. */ + atcphy_usb3_link_put(pm); + + /* Restore normal runtime policy after both success and failed parking. */ + usb_mark_last_busy(pm->root); + pm_runtime_put_autosuspend(&pm->root->dev); + usb_unlock_device(pm->root); + usb_put_dev(pm->root); + device_unlock(pm->host); + put_device(pm->host); + device_unlock(&pm->controller->dev); + put_device(&pm->controller->dev); + pm->host = NULL; + pm->root = NULL; + pm->controller = NULL; +#endif +} + +static int atcphy_mux_set_locked(struct typec_mux_dev *mux, struct typec_mux_state *state) { struct apple_atcphy *atcphy = typec_mux_get_drvdata(mux); enum atcphy_mode target_mode; @@ -3753,8 +4347,20 @@ static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *sta target_mode != APPLE_ATCPHY_MODE_DP) return -EOPNOTSUPP; - if (atcphy->mode == target_mode) + if (atcphy->mode == target_mode && !atcphy->mode_dirty) { + /* T8140: retry a failed PIPE setup without disturbing the configured lanes. */ + if (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2 && atcphy->host_active && + (atcphy->pipe_state != atcphy_modes[target_mode].pipehandler_state || + atcphy->pipe_pending)) { + if (atcphy->pipe_pending) { + ret = atcphy_park_pipehandler(atcphy, false); + if (ret) + return ret; + } + return atcphy_configure_pipehandler(atcphy, true); + } return 0; + } switch (target_mode) { case APPLE_ATCPHY_MODE_TBT: @@ -3765,6 +4371,9 @@ static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *sta case APPLE_ATCPHY_MODE_USB3: case APPLE_ATCPHY_MODE_USB3_DP: case APPLE_ATCPHY_MODE_DP: + if (atcphy->hw->has_t8140_dp && atcphy_modes[target_mode].enable_dp_aux && + !atcphy->t8140_dp_tunables) + return -EOPNOTSUPP; /* * Without the lane tunables the SuperSpeed lanes are not calibrated. * T8122 four-lane DP programs its lanes itself and needs only the @@ -3791,8 +4400,15 @@ static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *sta * mode without a USB3 backend, and route it to the new backend * afterwards while dwc3 is up in host mode. */ - if (pipe_state != ATCPHY_PIPEHANDLER_STATE_USB3) - atcphy_park_pipehandler(atcphy, false); + /* + * Even another USB3 mode can move a T8140 backend to the other PMA + * when orientation changes, so T8140 parks before every rebuild. + */ + if (pipe_state != ATCPHY_PIPEHANDLER_STATE_USB3 || atcphy->hw->has_t8140_dp) { + ret = atcphy_park_pipehandler(atcphy, false); + if (ret) + return ret; + } ret = atcphy_configure(atcphy, target_mode); if (ret) @@ -3804,6 +4420,13 @@ static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *sta return ret; } + /* A failed T8140 USB3 handoff must finish parking before ordinary-port lane work. */ + if (atcphy->pipe_pending) { + ret = atcphy_park_pipehandler(atcphy, false); + if (ret) + return ret; + } + /* * If the pipehandler is still/already up here there's a bug somewhere so make sure to * complain loudly. We can still try to switch modes and hope for the best though, @@ -3813,6 +4436,20 @@ static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *sta return atcphy_configure(atcphy, target_mode); } +static int atcphy_mux_set(struct typec_mux_dev *mux, struct typec_mux_state *state) +{ + struct apple_atcphy *atcphy = typec_mux_get_drvdata(mux); + struct atcphy_usb3_pm pm = {}; + int ret; + + ret = atcphy_usb3_pm_get(atcphy, &pm); + if (ret) + return ret; + ret = atcphy_mux_set_locked(mux, state); + atcphy_usb3_pm_put(&pm); + return ret; +} + static void atcphy_typec_mux_unregister(void *data) { typec_mux_unregister(data); @@ -3834,6 +4471,57 @@ static int atcphy_probe_mux(struct apple_atcphy *atcphy) return devm_add_action_or_reset(atcphy->dev, atcphy_typec_mux_unregister, mux); } +static int atcphy_load_t8140_dp(struct apple_atcphy *atcphy) +{ + struct { + const char *name; + struct resource *resource; + struct apple_tunable **out; + } lists[] = { + { "apple,tunable-dp-common-pre", atcphy->res.core, + &atcphy->tunables.dp_common_pre }, + { "apple,tunable-dp-lpdptx", atcphy->res.lpdptx, + &atcphy->tunables.dp_lpdptx }, + { "apple,tunable-dp-common-post", atcphy->res.core, + &atcphy->tunables.dp_common_post }, + }; + if (!of_machine_is_compatible("apple,j700")) + return -EOPNOTSUPP; + for (unsigned int i = 0; i < ARRAY_SIZE(lists); i++) { + struct property *property = of_find_property(atcphy->np, lists[i].name, NULL); + + if (!property || !atc_dp_list_length_valid(property->length)) + return -EINVAL; + *lists[i].out = devm_apple_tunable_parse(atcphy->dev, atcphy->np, + lists[i].name, lists[i].resource); + if (IS_ERR(*lists[i].out)) + return PTR_ERR(*lists[i].out); + } + + for (unsigned int i = 0; i < ARRAY_SIZE(atcphy->dp_presets); i++) { + static const char * const properties[] = { + "apple,dp-training-table", "apple,dp-training-table-rbr", + "apple,dp-training-table-hbr", "apple,dp-training-table-hbr2", + "apple,dp-training-table-hbr3", + }; + int count = of_property_count_u32_elems(atcphy->np, properties[i]); + int ret; + + if (i && !of_find_property(atcphy->np, properties[i], NULL)) + continue; + if (count != ARRAY_SIZE(atcphy->dp_presets[i])) + return -EINVAL; + ret = of_property_read_u32_array(atcphy->np, properties[i], + atcphy->dp_presets[i], count); + if (ret) + return ret; + atcphy->dp_presets_valid[i] = true; + } + + atcphy->t8140_dp_tunables = true; + return 0; +} + static int atcphy_load_tunables(struct apple_atcphy *atcphy) { size_t tunable_count; @@ -3926,6 +4614,13 @@ static int atcphy_load_tunables(struct apple_atcphy *atcphy) atcphy->tunables.usb2phy_reg_dflt = tunable; } + if (atcphy->hw->has_t8140_dp) { + int ret = atcphy_load_t8140_dp(atcphy); + + if (ret) + dev_warn(atcphy->dev, "DisplayPort tunable bundle unavailable: %d\n", ret); + } + atcphy->ss_tunables = !ss_missing; atcphy->dp_tunables = atcphy->hw->dp_t8122 && !common_missing; if (atcphy->dp_only && !atcphy->dp_tunables) @@ -3947,7 +4642,7 @@ static int atcphy_map_resources(struct platform_device *pdev, struct apple_atcph struct resource **res; } resources[] = { { "core", &atcphy->regs.core, &atcphy->res.core }, - { "lpdptx", &atcphy->regs.lpdptx, NULL }, + { "lpdptx", &atcphy->regs.lpdptx, &atcphy->res.lpdptx }, { "axi2af", &atcphy->regs.axi2af, &atcphy->res.axi2af }, { "usb2phy", &atcphy->regs.usb2phy, NULL }, { "pipehandler", &atcphy->regs.pipehandler, NULL }, @@ -4217,6 +4912,8 @@ static const struct atcphy_hw atcphy_hw_t8103 = { .gen = ATCPHY_GENERATION_T8103, .aciophy_lane_mode = ACIOPHY_LANE_MODE_T8103, .aciophy_crossbar = ACIOPHY_CROSSBAR_T8103, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, .has_usb4 = true, }; @@ -4224,6 +4921,8 @@ static const struct atcphy_hw atcphy_hw_t8122 = { .gen = ATCPHY_GENERATION_T8122, .aciophy_lane_mode = ACIOPHY_LANE_MODE_T8122, .aciophy_crossbar = ACIOPHY_CROSSBAR_T8122, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, .has_usb4 = true, .dp_t8122 = true, .park_pipe_unlocked = true, @@ -4235,13 +4934,26 @@ static const struct atcphy_hw atcphy_hw_t8140 = { .gen = ATCPHY_GENERATION_T8122, .aciophy_lane_mode = ACIOPHY_LANE_MODE_T8122, .aciophy_crossbar = ACIOPHY_CROSSBAR_T8122, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, .has_usb2phy_reg = true, + .has_t8140_dp = true, .optional_tunables = true, }; +static const struct atcphy_hw atcphy_hw_t8142 = { + .gen = ATCPHY_GENERATION_T8122, + .aciophy_lane_mode = ACIOPHY_LANE_MODE_T8122, + .aciophy_crossbar = ACIOPHY_CROSSBAR_T8122, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8142, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8142, + .has_usb4 = true, +}; + static const struct of_device_id atcphy_match[] = { { .compatible = "apple,t8103-atcphy", .data = &atcphy_hw_t8103 }, { .compatible = "apple,t8122-atcphy", .data = &atcphy_hw_t8122 }, + { .compatible = "apple,t8142-atcphy", .data = &atcphy_hw_t8142 }, { .compatible = "apple,t8140-atcphy", .data = &atcphy_hw_t8140 }, {}, }; diff --git a/drivers/pmdomain/apple/Kconfig b/drivers/pmdomain/apple/Kconfig index 2fc24c49ddaf01..0e8f833861880d 100644 --- a/drivers/pmdomain/apple/Kconfig +++ b/drivers/pmdomain/apple/Kconfig @@ -20,19 +20,6 @@ config APPLE_PMP_REPORT depends on PM select PM_GENERIC_DOMAINS -endif - -config APPLE_PMP_REPORT_KUNIT_TEST - tristate "Apple PMP report validation tests" if !KUNIT_ALL_TESTS - depends on KUNIT && PM - select PM_GENERIC_DOMAINS - default KUNIT_ALL_TESTS - help - Tests the PMP report layout, request acknowledgment flags and boot - argument validation without accessing hardware. - Includes malformed tables, address arithmetic and device ordering. - The PMP report driver does not need to be enabled to run this suite. - config APPLE_PMC bool "Apple PMC voter interface" depends on OF @@ -47,9 +34,25 @@ config APPLE_PMC GPU, while its device is powered on. It can also limit the DRAM controller rail as a thermal cooling device. +config APPLE_PMC_KUNIT_TEST + bool "KUnit tests for the Apple PMC voter interface" if !KUNIT_ALL_TESTS + depends on APPLE_PMC && KUNIT=y + default KUNIT_ALL_TESTS + help + Unit tests for the Apple PMC driver. They check that enabling and + disabling a voter agent only changes that agent on each rail, and + that lifting a DCS limit gives the AP agent its own vote back. The + registers are simulated in memory. Only useful for kernel developers. + +endif -config APPLE_PMP_REPORT_NEO - bool "Apple T8140 PMP report and fast die-temperature limits" - depends on ARCH_APPLE && OF && PM +config APPLE_PMP_REPORT_KUNIT_TEST + tristate "Apple PMP report validation tests" if !KUNIT_ALL_TESTS + depends on KUNIT && PM select PM_GENERIC_DOMAINS - default ARCH_APPLE + default KUNIT_ALL_TESTS + help + Tests the PMP report layout, request acknowledgment flags and boot + argument validation without accessing hardware. + Includes malformed tables, address arithmetic and device ordering. + The PMP report driver does not need to be enabled to run this suite. diff --git a/drivers/pmdomain/apple/Makefile b/drivers/pmdomain/apple/Makefile index 99f2c3dcef9bd9..fdd5e4f1987d74 100644 --- a/drivers/pmdomain/apple/Makefile +++ b/drivers/pmdomain/apple/Makefile @@ -5,4 +5,3 @@ obj-$(CONFIG_APPLE_PMP_REPORT) += pmp-report.o obj-$(CONFIG_APPLE_PMP_REPORT_KUNIT_TEST) += pmp-report-test.o obj-$(CONFIG_APPLE_PMC) += pmc.o -obj-$(CONFIG_APPLE_PMP_REPORT_NEO) += pmp-report-neo.o diff --git a/drivers/pmdomain/apple/pmc-test.c b/drivers/pmdomain/apple/pmc-test.c new file mode 100644 index 00000000000000..3f0e277b212551 --- /dev/null +++ b/drivers/pmdomain/apple/pmc-test.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * KUnit tests for the Apple PMC voter interface: agent enables and the + * AP's DCS vote, with memory standing in for the registers. + * + * Included by pmc.c. + */ + +#include + +#define APPLE_PMC_TEST_RAILS_SIZE 0x4000 + +static struct apple_pmc *apple_pmc_test_alloc(struct kunit *test) +{ + const struct apple_pmc_hw *hw = &apple_pmc_hw_t8140; + struct apple_pmc *pmc; + + pmc = kunit_kzalloc(test, sizeof(*pmc), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pmc); + pmc->hw = hw; + raw_spin_lock_init(&pmc->lock); + pmc->voters = (void __iomem *)kunit_kzalloc(test, + hw->agents * hw->agent_stride, + GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, (void __force *)pmc->voters); + pmc->rails = (void __iomem *)kunit_kzalloc(test, + APPLE_PMC_TEST_RAILS_SIZE, + GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, (void __force *)pmc->rails); + + return pmc; +} + +static void apple_pmc_test_set_dcs_vote(struct apple_pmc *pmc, + unsigned int agent, unsigned int state) +{ + void __iomem *reg = pmc->voters + agent * pmc->hw->agent_stride; + + writel(state << APPLE_PMC_RAIL_SHIFT(pmc->hw->dcs_rail), reg); +} + +/* Enabling and disabling an agent changes its bit on every rail only. */ +static void apple_pmc_test_agent_enable(struct kunit *test) +{ + struct apple_pmc *pmc = apple_pmc_test_alloc(test); + unsigned int rail; + + pmc->enable[0] = BIT(0); + writel(BIT(0), pmc->rails + pmc->hw->rail_enable); + + KUNIT_ASSERT_EQ(test, apple_pmc_agent_enable(pmc, 1, true), 0); + for (rail = 0; rail < pmc->hw->rails; rail++) + KUNIT_EXPECT_EQ(test, apple_pmc_enable_read(pmc, rail), + BIT(1) | (rail ? 0 : BIT(0))); + + KUNIT_ASSERT_EQ(test, apple_pmc_agent_enable(pmc, 1, false), 0); + for (rail = 0; rail < pmc->hw->rails; rail++) + KUNIT_EXPECT_EQ(test, apple_pmc_enable_read(pmc, rail), + rail ? 0 : BIT(0)); +} + +/* Lifting a DCS limit gives the AP's agent back its own vote, not 0. */ +static void apple_pmc_test_dcs_limit_restores_ap_vote(struct kunit *test) +{ + struct apple_pmc *pmc = apple_pmc_test_alloc(test); + const struct apple_pmc_hw *hw = pmc->hw; + + apple_pmc_test_set_dcs_vote(pmc, hw->ap_agent, 2); + apple_pmc_test_set_dcs_vote(pmc, 0, 4); + pmc->enable[hw->dcs_rail] = BIT(0) | BIT(hw->ap_agent); + pmc->ap_dcs_vote = apple_pmc_ap_dcs_read(pmc); + KUNIT_ASSERT_EQ(test, pmc->ap_dcs_vote, 2); + + /* A limit of 3 lets the AP vote at most DCS 1 for the PMP's 4. */ + pmc->dcs_state = 3; + apple_pmc_dcs_vote(pmc); + KUNIT_EXPECT_EQ(test, apple_pmc_ap_dcs_read(pmc), 1); + + pmc->dcs_state = 0; + apple_pmc_dcs_vote(pmc); + KUNIT_EXPECT_EQ(test, apple_pmc_ap_dcs_read(pmc), 2); +} + +static struct kunit_case apple_pmc_test_cases[] = { + KUNIT_CASE(apple_pmc_test_agent_enable), + KUNIT_CASE(apple_pmc_test_dcs_limit_restores_ap_vote), + {} +}; + +static struct kunit_suite apple_pmc_test_suite = { + .name = "apple-pmc", + .test_cases = apple_pmc_test_cases, +}; + +kunit_test_suite(apple_pmc_test_suite); diff --git a/drivers/pmdomain/apple/pmc.c b/drivers/pmdomain/apple/pmc.c index 397fce7fc8ad26..d1d37a9917ff40 100644 --- a/drivers/pmdomain/apple/pmc.c +++ b/drivers/pmdomain/apple/pmc.c @@ -92,6 +92,8 @@ struct apple_pmc { struct dentry *debugfs; /* Cooling state of the DCS limit: 0 is none, dcs_max holds DCS at 0. */ unsigned long dcs_state; + /* DCS state the AP's agent voted for before probe, kept without a limit. */ + unsigned int ap_dcs_vote; /* Power cost of each DCS state in mW, if a model is described. */ u32 dcs_power[APPLE_PMC_MAX_STATES]; struct delayed_work dcs_work; @@ -151,18 +153,46 @@ static int apple_pmc_enable_update(struct apple_pmc *pmc, unsigned int rail, return apple_pmc_enable_write(pmc, rail); } +/* + * Caller holds pmc->lock. Put the rails up to and including @last back to + * the enables in @prev, after a change failed on @last. + */ +static void apple_pmc_enable_restore(struct apple_pmc *pmc, const u32 *prev, + unsigned int last) +{ + unsigned int rail; + + for (rail = 0; rail <= last; rail++) { + pmc->enable[rail] = prev[rail]; + apple_pmc_enable_write(pmc, rail); + } +} + +/* + * Enable or disable @agent on every rail. If a rail does not take the + * change, undo it on the rails changed so far: genpd keeps a domain whose + * power_on() fails off, and never calls its power_off(), so an agent left + * enabled on some rails would stay enabled for good. + */ static int apple_pmc_agent_enable(struct apple_pmc *pmc, unsigned int agent, bool enable) { + u32 prev[APPLE_PMC_MAX_RAILS]; unsigned long flags; unsigned int rail; int ret = 0; raw_spin_lock_irqsave(&pmc->lock, flags); - for (rail = 0; rail < pmc->hw->rails && !ret; rail++) + memcpy(prev, pmc->enable, sizeof(prev)); + for (rail = 0; rail < pmc->hw->rails; rail++) { ret = apple_pmc_enable_update(pmc, rail, enable ? BIT(agent) : 0, enable ? 0 : BIT(agent)); + if (ret) { + apple_pmc_enable_restore(pmc, prev, rail); + break; + } + } raw_spin_unlock_irqrestore(&pmc->lock, flags); return ret; @@ -287,16 +317,27 @@ static unsigned int apple_pmc_dcs_demand(struct apple_pmc *pmc) return min(demand, hw->dcs_max); } +/* The DCS state the AP's agent votes for. */ +static unsigned int apple_pmc_ap_dcs_read(struct apple_pmc *pmc) +{ + const struct apple_pmc_hw *hw = pmc->hw; + u32 floor = readl(pmc->voters + hw->ap_agent * hw->agent_stride); + + return (floor & APPLE_PMC_RAIL_MASK(hw->dcs_rail)) >> + APPLE_PMC_RAIL_SHIFT(hw->dcs_rail); +} + /* * Caller holds pmc->lock. Vote through the AP's agent for what the other - * agents ask for, capped by the limit; vote nothing without a limit. + * agents ask for, capped by the limit; without a limit, put back the vote + * the AP's agent had before probe. */ static void apple_pmc_dcs_vote(struct apple_pmc *pmc) { const struct apple_pmc_hw *hw = pmc->hw; void __iomem *reg = pmc->voters + hw->ap_agent * hw->agent_stride; u32 mask = APPLE_PMC_RAIL_MASK(hw->dcs_rail); - unsigned int state = 0; + unsigned int state = pmc->ap_dcs_vote; u32 old, val; if (pmc->dcs_state) @@ -675,6 +716,7 @@ static int apple_pmc_probe(struct platform_device *pdev) "PMC is not responding\n"); pmc->enable[rail] = pmc->saved_enable[rail]; } + pmc->ap_dcs_vote = apple_pmc_ap_dcs_read(pmc); count = of_property_count_u32_elems(np, "apple,static-voters"); if (count > 0) { @@ -769,3 +811,7 @@ module_init(apple_pmc_init); MODULE_DESCRIPTION("Apple SoC PMC voter interface driver"); MODULE_LICENSE("Dual MIT/GPL"); + +#if IS_ENABLED(CONFIG_APPLE_PMC_KUNIT_TEST) +#include "pmc-test.c" +#endif diff --git a/drivers/pmdomain/apple/pmgr-pwrstate.c b/drivers/pmdomain/apple/pmgr-pwrstate.c index 4d5c584d1152b7..fcafa198656bce 100644 --- a/drivers/pmdomain/apple/pmgr-pwrstate.c +++ b/drivers/pmdomain/apple/pmgr-pwrstate.c @@ -49,6 +49,10 @@ struct apple_pmgr_ps { bool force_reset; bool externally_clocked; bool retained_25g83; + bool firmware_owned; + /* T8140: follow a transition that times out with the opposite one. */ + bool undo_timeout; + bool system_sleep; }; #define genpd_to_apple_pmgr_ps(_genpd) container_of(_genpd, struct apple_pmgr_ps, genpd) @@ -147,14 +151,86 @@ static bool apple_pmgr_ps_is_active(struct apple_pmgr_ps *ps) reg & APPLE_PMGR_AUTO_ENABLE)); } +/* + * With undo_timeout (T8140, T8142): genpd keeps a domain off when ->power_on() fails + * and on when ->power_off() fails. Follow a failed transition with the + * opposite one, so the domain is not left requesting a state genpd does not + * know it is in. + * + * The exception is a failed power-on in the noirq and syscore phases of a + * system sleep transition: genpd ignores the error there and records the + * domain as on. The PM core runs those phases after the suspend_late callback + * of every device and before any resume_early callback, so keep requesting + * the domain on between this device's two callbacks. + */ static int apple_pmgr_ps_power_on(struct generic_pm_domain *genpd) { - return apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_ACTIVE, true); + struct apple_pmgr_ps *ps = genpd_to_apple_pmgr_ps(genpd); + int ret; + + /* Its coprocessor's firmware owns the state; Linux only holds a vote. */ + if (ps->firmware_owned) + return 0; + + ret = apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_ACTIVE, true); + if (ret && ps->undo_timeout && !ps->system_sleep) + apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_PWRGATE, false); + + return ret; } static int apple_pmgr_ps_power_off(struct generic_pm_domain *genpd) { - return apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_PWRGATE, false); + struct apple_pmgr_ps *ps = genpd_to_apple_pmgr_ps(genpd); + int ret; + + if (ps->firmware_owned) + return 0; + + ret = apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_PWRGATE, false); + if (ret && ps->undo_timeout) + apple_pmgr_ps_set(genpd, APPLE_PMGR_PS_ACTIVE, true); + + return ret; +} + +static void apple_pmgr_ps_set_system_sleep(struct device *dev, bool sleep) +{ + struct apple_pmgr_ps *ps = dev_get_drvdata(dev); + unsigned long flags; + + /* Only read by domains that undo a failed transition. */ + if (!ps->undo_timeout) + return; + + spin_lock_irqsave(&ps->genpd.slock, flags); + ps->system_sleep = sleep; + spin_unlock_irqrestore(&ps->genpd.slock, flags); +} + +/* + * Domains that do not undo a failed transition need no sleep window: let the + * PM core treat them as it treats a device without PM callbacks. + */ +static int apple_pmgr_ps_prepare(struct device *dev) +{ + struct apple_pmgr_ps *ps = dev_get_drvdata(dev); + + return !ps->undo_timeout; +} + +static int apple_pmgr_ps_suspend_late(struct device *dev) +{ + apple_pmgr_ps_set_system_sleep(dev, true); + + return 0; +} + +static int apple_pmgr_ps_resume_early(struct device *dev) +{ + apple_pmgr_ps_set_system_sleep(dev, false); + + return 0; } static int apple_pmgr_reset_assert(struct reset_controller_dev *rcdev, unsigned long id) @@ -164,6 +240,8 @@ static int apple_pmgr_reset_assert(struct reset_controller_dev *rcdev, unsigned if (ps->retained_25g83) return -EBUSY; + if (ps->firmware_owned) + return -EPERM; spin_lock_irqsave(&ps->genpd.slock, flags); @@ -190,6 +268,8 @@ static int apple_pmgr_reset_deassert(struct reset_controller_dev *rcdev, unsigne if (ps->retained_25g83) return -EBUSY; + if (ps->firmware_owned) + return -EPERM; spin_lock_irqsave(&ps->genpd.slock, flags); @@ -285,6 +365,7 @@ static int apple_pmgr_ps_probe(struct platform_device *pdev) ps->dev = dev; ps->regmap = regmap; + platform_set_drvdata(pdev, ps); ret = of_property_read_string(node, "label", &name); if (ret < 0) { @@ -299,13 +380,23 @@ static int apple_pmgr_ps_probe(struct platform_device *pdev) } ps->retained_25g83 = apple_pmgr_retained_25g83(node, ps->offset); + ps->undo_timeout = of_device_is_compatible(node, "apple,t8140-pmgr-pwrstate") || + of_device_is_compatible(node, "apple,t8142-pmgr-pwrstate"); ps->genpd.flags |= GENPD_FLAG_IRQ_SAFE; ps->genpd.name = name; ps->genpd.power_on = apple_pmgr_ps_power_on; ps->genpd.power_off = apple_pmgr_ps_power_off; + /* + * A domain whose coprocessor the boot loader hands over running (the DCP) is + * owned by that coprocessor's firmware: writing its state faults. Only read + * it, keep it on and never change its target or automatic gating. + */ + ps->firmware_owned = of_property_read_bool(node, "apple,firmware-owned"); + ret = of_property_read_u32(node, "apple,min-state", &ps->min_state); - if (!ps->retained_25g83 && ret == 0 && ps->min_state <= APPLE_PMGR_PS_ACTIVE) + if (!ps->retained_25g83 && ret == 0 && ps->min_state <= APPLE_PMGR_PS_ACTIVE && + !ps->firmware_owned) regmap_update_bits(regmap, ps->offset, APPLE_PMGR_FLAGS | APPLE_PMGR_PS_MIN, FIELD_PREP(APPLE_PMGR_PS_MIN, ps->min_state)); @@ -324,19 +415,30 @@ static int apple_pmgr_ps_probe(struct platform_device *pdev) return dev_err_probe(dev, -EBUSY, "inherited domain %s is not active\n", name); ps->genpd.flags |= GENPD_FLAG_ALWAYS_ON; } - if (of_property_read_bool(node, "apple,always-on")) { + if (ps->firmware_owned) { + /* Consumers (the boot framebuffer) need it; never try to turn it on. */ + if (!active) + dev_warn(dev, "firmware-owned domain %s does not read as on\n", name); + ps->genpd.flags |= GENPD_FLAG_ALWAYS_ON; + active = true; + } else if (of_property_read_bool(node, "apple,always-on")) { ps->genpd.flags |= GENPD_FLAG_ALWAYS_ON; if (!active) { dev_warn(dev, "always-on domain %s is not on at boot\n", name); - /* Turn it on so pm_genpd_init does not fail */ - active = apple_pmgr_ps_power_on(&ps->genpd) == 0; + /* + * Turn it on so pm_genpd_init does not fail. genpd never + * powers an always-on domain, so keep requesting it on + * even if it does not get there in time. + */ + active = apple_pmgr_ps_set(&ps->genpd, APPLE_PMGR_PS_ACTIVE, + true) == 0; } } else if (active) { ps->genpd.flags |= GENPD_FLAG_DEFER_OFF | GENPD_FLAG_ACTIVE_WAKEUP; } /* Turn on auto-PM if the domain is already on */ - if (active && !ps->retained_25g83) + if (active && !ps->retained_25g83 && !ps->firmware_owned) regmap_update_bits(regmap, ps->offset, APPLE_PMGR_FLAGS | APPLE_PMGR_AUTO_ENABLE, APPLE_PMGR_AUTO_ENABLE); @@ -404,11 +506,17 @@ static const struct of_device_id apple_pmgr_ps_of_match[] = { MODULE_DEVICE_TABLE(of, apple_pmgr_ps_of_match); +static const struct dev_pm_ops apple_pmgr_ps_pm_ops = { + .prepare = pm_sleep_ptr(apple_pmgr_ps_prepare), + LATE_SYSTEM_SLEEP_PM_OPS(apple_pmgr_ps_suspend_late, apple_pmgr_ps_resume_early) +}; + static struct platform_driver apple_pmgr_ps_driver = { .probe = apple_pmgr_ps_probe, .driver = { .name = "apple-pmgr-pwrstate", .of_match_table = apple_pmgr_ps_of_match, + .pm = pm_sleep_ptr(&apple_pmgr_ps_pm_ops), }, }; diff --git a/drivers/pmdomain/apple/pmp-report-neo.c b/drivers/pmdomain/apple/pmp-report-neo.c deleted file mode 100644 index d0c27cb8250789..00000000000000 --- a/drivers/pmdomain/apple/pmp-report-neo.c +++ /dev/null @@ -1,529 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only OR MIT -/* - * Apple SoC PMP power state reporting driver - * - * Copyright The Asahi Linux Contributors - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#define PMP_REPORT_READY 0x1 - -/** - * struct apple_neo_pmp_report_offsets - per-SoC layout of the report region - * @tgt_read: power state request, read side - * @tgt_write: power state request, write side - * @actual: power state acknowledgment - * @status: PMP status; bit 0 is set while the PMP is running - * @fast_die: 32-bit word holding the fast die-temperature effort of each - * CPU cluster, one byte per lane; 0 if the SoC has none - * @fast_die_lanes: number of lanes at @fast_die, at most 4 - */ -struct apple_neo_pmp_report_offsets { - u32 tgt_read; - u32 tgt_write; - u32 actual; - u32 status; - u32 fast_die; - u32 fast_die_lanes; -}; - -struct apple_neo_pmp_report; - -static int apple_neo_pmp_dvfs_create(struct apple_neo_pmp_report *report); - -struct apple_neo_pmp_report { - struct device *dev; - const struct apple_neo_pmp_report_offsets *offsets; - void __iomem *base; - spinlock_t lock; - struct list_head list; - struct dentry *debugfs; -}; - -/* Bound report regions, protected by RCU until managed teardown drains readers. */ -static LIST_HEAD(apple_neo_pmp_reports); -static DEFINE_MUTEX(apple_neo_pmp_reports_lock); -static struct dentry *apple_neo_pmp_report_debugfs; - -/** - * apple_neo_pmp_report_fast_die_effort() - read the fast die-temperature effort - * of a CPU cluster - * @np: device tree node of the report region - * @lane: lane of the cluster in the fast die-temperature report - * @effort: set to the effort on success - * - * The PMP of some SoCs runs a control loop on its own die-temperature sensors - * and publishes an 8-bit effort for each CPU cluster: 0 requests no limit, - * and the effort rises towards 255 as the die exceeds the firmware's target. - * The effort is only valid while the PMP reports that it is running; a - * stopped PMP reads 0. - * - * Context: Any context. - * Return: 0 on success, -EPROBE_DEFER if the region is not bound (yet), - * -EOPNOTSUPP if the SoC's PMP publishes no such effort, -EINVAL if @lane is - * out of range, or -EAGAIN if the PMP does not report that it is running. - */ -int apple_neo_pmp_report_fast_die_effort(const struct device_node *np, - unsigned int lane, u8 *effort) -{ - const struct apple_neo_pmp_report_offsets *offs; - const struct apple_neo_pmp_report *rep; - int ret = -EPROBE_DEFER; - - rcu_read_lock(); - list_for_each_entry_rcu(rep, &apple_neo_pmp_reports, list) { - if (rep->dev->of_node != np) - continue; - - offs = rep->offsets; - if (!offs->fast_die) { - ret = -EOPNOTSUPP; - } else if (lane >= offs->fast_die_lanes) { - ret = -EINVAL; - } else if (!(readl(rep->base + offs->status) & PMP_REPORT_READY)) { - ret = -EAGAIN; - } else { - *effort = readl(rep->base + offs->fast_die) >> - (lane * BITS_PER_BYTE); - ret = 0; - } - break; - } - rcu_read_unlock(); - - return ret; -} -EXPORT_SYMBOL_GPL(apple_neo_pmp_report_fast_die_effort); - -/* Read only the documented status and fast-die entries, never the aperture. */ -static int apple_neo_pmp_report_status_show(struct seq_file *s, void *unused) -{ - struct apple_neo_pmp_report *rep = s->private; - const struct apple_neo_pmp_report_offsets *offs = rep->offsets; - u32 status = readl(rep->base + offs->status); - unsigned int lane; - u32 effort; - - seq_printf(s, "running: %u\n", !!(status & PMP_REPORT_READY)); - if (!(status & PMP_REPORT_READY) || !offs->fast_die) - return 0; - - effort = readl(rep->base + offs->fast_die); - for (lane = 0; lane < offs->fast_die_lanes; lane++) - seq_printf(s, "lane%u: %u\n", lane, - (effort >> (lane * BITS_PER_BYTE)) & 0xff); - - return 0; -} -DEFINE_SHOW_ATTRIBUTE(apple_neo_pmp_report_status); - -static void apple_neo_pmp_report_release(void *data) -{ - struct apple_neo_pmp_report *rep = data; - - /* Withdraw child providers while their power callbacks can still use MMIO. */ - of_platform_depopulate(rep->dev); - mutex_lock(&apple_neo_pmp_reports_lock); - if (!list_empty(&rep->list)) - list_del_rcu(&rep->list); - mutex_unlock(&apple_neo_pmp_reports_lock); - synchronize_rcu(); - /* The debugfs proxy drains active reads and rejects subsequent accesses. */ - debugfs_remove(rep->debugfs); -} - -static int apple_neo_pmp_report_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct device_node *np = dev->of_node; - struct apple_neo_pmp_report *rep; - int ret; - - rep = devm_kzalloc(dev, sizeof(*rep), GFP_KERNEL); - if (!rep) - return -ENOMEM; - - rep->dev = dev; - spin_lock_init(&rep->lock); - rep->base = devm_platform_ioremap_resource(pdev, 0); - if (IS_ERR(rep->base)) - return PTR_ERR(rep->base); - rep->offsets = of_device_get_match_data(dev); - dev_set_drvdata(dev, rep); - ret = apple_neo_pmp_dvfs_create(rep); - if (ret) - return ret; - - INIT_LIST_HEAD(&rep->list); - ret = devm_add_action_or_reset(dev, apple_neo_pmp_report_release, rep); - if (ret) - return ret; - - mutex_lock(&apple_neo_pmp_reports_lock); - list_add_tail_rcu(&rep->list, &apple_neo_pmp_reports); - mutex_unlock(&apple_neo_pmp_reports_lock); - - if (rep->offsets->fast_die) - rep->debugfs = debugfs_create_file(dev_name(dev), 0400, - apple_neo_pmp_report_debugfs, rep, - &apple_neo_pmp_report_status_fops); - - /* Publish the region before child power callbacks start looking it up. */ - ret = of_platform_populate(np, NULL, NULL, dev); - if (ret) - return dev_err_probe(dev, ret, "failed to create child devices\n"); - - return 0; -} - -static const struct apple_neo_pmp_report_offsets apple_neo_pmp_offsets_t600x = { - .tgt_read = 0xf80, - .tgt_write = 0x107c0, - .actual = 0x1000, - .status = 0x10, -}; - -static const struct apple_neo_pmp_report_offsets apple_neo_pmp_offsets_t602x = { - .tgt_read = 0x2000, - .tgt_write = 0x11000, - .actual = 0x2080, - .status = 0x10, -}; - -static const struct apple_neo_pmp_report_offsets apple_neo_pmp_offsets_t8112 = { - .tgt_read = 0xa00, - .tgt_write = 0x10500, - .actual = 0xa40, - .status = 0x10, -}; - -/* The fast die-temperature report is the low two bytes of entry 0x1c1. */ -static const struct apple_neo_pmp_report_offsets apple_neo_pmp_offsets_t8140 = { - .tgt_read = 0x1880, - .tgt_write = 0x10c40, - .actual = 0x18c0, - .status = 0x10, - .fast_die = 0x1c10, - .fast_die_lanes = 2, -}; - -static const struct of_device_id apple_neo_pmp_report_of_match[] = { - { .compatible = "apple,t8140-pmp-v2-report", .data = &apple_neo_pmp_offsets_t8140 }, - {} -}; - -static struct platform_driver apple_neo_pmp_report_driver = { - .probe = apple_neo_pmp_report_probe, - .driver = { - .name = "apple-pmp-report-neo", - .of_match_table = apple_neo_pmp_report_of_match, - /* Power-domain providers are not intended for manual unbinding. */ - .suppress_bind_attrs = true, - }, -}; - -struct apple_neo_pmp_report_entry { - struct device_node *node; - struct device_node *report_node; - struct generic_pm_domain genpd; - u32 id; - bool removed; -}; - -#define genpd_to_apple_neo_pmp_report_entry(_genpd) \ - container_of(_genpd, struct apple_neo_pmp_report_entry, genpd) - -static int apple_neo_pmp_report_set_state(struct apple_neo_pmp_report *rep, u32 id, - bool enable) -{ - u64 bit_val = BIT_ULL(id); - u64 val; - unsigned long flags; - - spin_lock_irqsave(&rep->lock, flags); - val = readq(rep->base + rep->offsets->tgt_read); - val &= ~bit_val; - if (enable) - val |= bit_val; - writeq(val, rep->base + rep->offsets->tgt_write); - spin_unlock_irqrestore(&rep->lock, flags); - val = readq(rep->base + rep->offsets->status); - if ((val & PMP_REPORT_READY) == 0) - return 0; - return readq_poll_timeout_atomic( - rep->base + rep->offsets->actual, - val, - !!(val & bit_val) == !!enable, - 100, - 50000); -} - -static int apple_neo_pmp_report_entry_set_state(struct generic_pm_domain *genpd, bool enable) -{ - struct apple_neo_pmp_report_entry *ent = genpd_to_apple_neo_pmp_report_entry(genpd); - struct apple_neo_pmp_report *rep; - int ret = -ENODEV; - - if (READ_ONCE(ent->removed)) - return ret; - - rcu_read_lock(); - list_for_each_entry_rcu(rep, &apple_neo_pmp_reports, list) { - if (rep->dev->of_node == ent->report_node) { - ret = apple_neo_pmp_report_set_state(rep, ent->id, enable); - break; - } - } - rcu_read_unlock(); - return ret; -} - -static int apple_neo_pmp_report_entry_power_on(struct generic_pm_domain *genpd) -{ - return apple_neo_pmp_report_entry_set_state(genpd, true); -} - -static int apple_neo_pmp_report_entry_power_off(struct generic_pm_domain *genpd) -{ - return apple_neo_pmp_report_entry_set_state(genpd, false); -} - -/* ADT DVFS-STATE: four 16-byte entries beginning at entry 8. */ -#define APPLE_PMP_DVFS_OFFSET 0x80 -#define APPLE_PMP_DVFS_STRIDE 16 -#define APPLE_PMP_DVFS_COUNT 4 - -static int apple_neo_pmp_dvfs_show(struct seq_file *seq, void *unused) -{ - struct apple_neo_pmp_report *report = seq->private; - u64 values[APPLE_PMP_DVFS_COUNT][2]; - unsigned int index; - - if (!(readl(report->base + report->offsets->status) & PMP_REPORT_READY)) - return -EAGAIN; - - for (index = 0; index < ARRAY_SIZE(values); index++) { - void __iomem *entry = report->base + APPLE_PMP_DVFS_OFFSET + - index * APPLE_PMP_DVFS_STRIDE; - - values[index][0] = readq(entry); - values[index][1] = readq(entry + sizeof(values[index][0])); - } - - if (!(readl(report->base + report->offsets->status) & PMP_REPORT_READY)) - return -EAGAIN; - - for (index = 0; index < ARRAY_SIZE(values); index++) - seq_printf(seq, "entry%u: 0x%016llx 0x%016llx\n", index + 8, - values[index][0], values[index][1]); - - return 0; -} -DEFINE_SHOW_ATTRIBUTE(apple_neo_pmp_dvfs); - -static void apple_neo_pmp_dvfs_remove(void *directory) -{ - debugfs_remove_recursive(directory); -} - -static int apple_neo_pmp_dvfs_create(struct apple_neo_pmp_report *report) -{ - struct device *dev = report->dev; - struct resource *resource; - struct dentry *directory; - char *name; - int ret; - - if (!of_device_is_compatible(dev->of_node, "apple,t8140-pmp-v2-report")) - return 0; - - resource = platform_get_resource(to_platform_device(dev), IORESOURCE_MEM, 0); - if (resource_size(resource) < APPLE_PMP_DVFS_OFFSET + - APPLE_PMP_DVFS_COUNT * APPLE_PMP_DVFS_STRIDE) - return dev_err_probe(dev, -EINVAL, "DVFS report exceeds resource\n"); - - name = devm_kasprintf(dev, GFP_KERNEL, "apple-pmp-%s", dev_name(dev)); - if (!name) - return -ENOMEM; - - directory = debugfs_create_dir(name, NULL); - if (IS_ERR(directory)) - return 0; - - ret = devm_add_action_or_reset(dev, apple_neo_pmp_dvfs_remove, directory); - if (ret) - return ret; - - debugfs_create_file("dvfs", 0400, directory, report, &apple_neo_pmp_dvfs_fops); - return 0; -} - -static void apple_neo_pmp_report_entry_free(struct apple_neo_pmp_report_entry *ent) -{ - of_node_put(ent->report_node); - of_node_put(ent->node); - kfree(ent->genpd.name); - kfree(ent); -} - -static void apple_neo_pmp_report_entry_remove(struct platform_device *pdev) -{ - struct apple_neo_pmp_report_entry *ent = platform_get_drvdata(pdev); - - WRITE_ONCE(ent->removed, true); - of_genpd_del_provider(ent->node); - if (pm_genpd_remove(&ent->genpd)) { - /* The core still owns this domain. Keep its now inert storage alive. */ - dev_warn(&pdev->dev, "retaining busy power domain %s\n", ent->genpd.name); - return; - } - apple_neo_pmp_report_entry_free(ent); -} - -static int apple_neo_pmp_report_entry_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct device_node *node = dev->of_node; - struct apple_neo_pmp_report_entry *ent; - int ret; - const char *name; - struct of_phandle_iterator it; - - ent = kzalloc_obj(*ent); - if (!ent) - return -ENOMEM; - - ent->node = of_node_get(node); - ent->report_node = of_node_get(dev->parent->of_node); - - ret = of_property_read_u32(node, "reg", &ent->id); - if (ret) { - ret = dev_err_probe(dev, ret, "missing reg property\n"); - goto err_free; - } - if (ent->id >= 64) { - ret = -EINVAL; - goto err_free; - } - - ret = of_property_read_string(node, "label", &name); - if (ret < 0) { - ret = dev_err_probe(dev, ret, "missing label property\n"); - goto err_free; - } - ent->genpd.name = kstrdup(name, GFP_KERNEL); - if (!ent->genpd.name) { - ret = -ENOMEM; - goto err_free; - } - - if (of_property_read_bool(node, "apple,always-on")) { - ent->genpd.flags |= GENPD_FLAG_ACTIVE_WAKEUP; - apple_neo_pmp_report_entry_set_state(&ent->genpd, true); - } - - ent->genpd.power_on = apple_neo_pmp_report_entry_power_on; - ent->genpd.power_off = apple_neo_pmp_report_entry_power_off; - - ret = pm_genpd_init(&ent->genpd, NULL, true); - if (ret) { - ret = dev_err_probe(dev, ret, "pm_genpd_init failed\n"); - goto err_free; - } - - ret = of_genpd_add_provider_simple(node, &ent->genpd); - if (ret) { - ret = dev_err_probe(dev, ret, "of_genpd_add_provider_simple failed\n"); - goto err_remove_domain; - } - - of_for_each_phandle(&it, ret, node, "power-domains", "#power-domain-cells", -1) { - struct of_phandle_args parent, child; - - parent.np = it.node; - parent.args_count = of_phandle_iterator_args(&it, parent.args, MAX_PHANDLE_ARGS); - child.np = node; - child.args_count = 0; - ret = of_genpd_add_subdomain(&parent, &child); - - if (ret == -EPROBE_DEFER) { - of_node_put(parent.np); - goto err_remove; - } else if (ret < 0) { - dev_err(dev, "failed to add to parent domain: %d (%s -> %s)\n", - ret, it.node->name, node->name); - of_node_put(parent.np); - goto err_remove; - } - } - - pm_genpd_remove_device(dev); - platform_set_drvdata(pdev, ent); - - return 0; -err_remove: - of_genpd_del_provider(node); -err_remove_domain: - WRITE_ONCE(ent->removed, true); - if (pm_genpd_remove(&ent->genpd)) { - dev_warn(dev, "retaining busy power domain %s\n", ent->genpd.name); - return ret; - } -err_free: - apple_neo_pmp_report_entry_free(ent); - return ret; -} - -static const struct of_device_id apple_neo_pmp_report_entry_of_match[] = { - { .compatible = "apple,t8140-pmp-v2-report-entry" }, - {} -}; - -static struct platform_driver apple_neo_pmp_report_entry_driver = { - .probe = apple_neo_pmp_report_entry_probe, - .remove = apple_neo_pmp_report_entry_remove, - .driver = { - .name = "apple-pmp-report-neo-entry", - .of_match_table = apple_neo_pmp_report_entry_of_match, - /* Power-domain providers are not intended for manual unbinding. */ - .suppress_bind_attrs = true, - }, -}; - -MODULE_DEVICE_TABLE(of, apple_neo_pmp_report_of_match); -MODULE_DEVICE_TABLE(of, apple_neo_pmp_report_entry_of_match); - -static int __init apple_neo_pmp_report_init(void) -{ - apple_neo_pmp_report_debugfs = debugfs_create_dir("apple-pmp-report-neo", NULL); - platform_driver_register(&apple_neo_pmp_report_entry_driver); - platform_driver_register(&apple_neo_pmp_report_driver); - return 0; -} - -static void __exit apple_neo_pmp_report_exit(void) -{ - debugfs_remove_recursive(apple_neo_pmp_report_debugfs); - platform_driver_unregister(&apple_neo_pmp_report_entry_driver); - platform_driver_unregister(&apple_neo_pmp_report_driver); -} - -module_init(apple_neo_pmp_report_init); -module_exit(apple_neo_pmp_report_exit); - -MODULE_DESCRIPTION("PMP power state reporting driver for Apple SoCs"); -MODULE_LICENSE("Dual MIT/GPL"); diff --git a/drivers/pmdomain/apple/pmp-report.c b/drivers/pmdomain/apple/pmp-report.c index c9acf48bdcb0dc..9d585f5a0f712d 100644 --- a/drivers/pmdomain/apple/pmp-report.c +++ b/drivers/pmdomain/apple/pmp-report.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -20,7 +21,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -48,6 +51,19 @@ struct apple_pmp_report_offsets { bool temps; /* With starts_pmp: soc-device record i holds device id i + 1. */ bool devices_ordered; + /* + * The report is published in apple_pmp_reports, where the fast + * die-temperature reader and the power callbacks of its entries look + * it up by node, and its teardown withdraws the entries first (T8140). + */ + bool published; + /* + * With published: 32-bit word holding the fast die-temperature effort + * of each CPU cluster, one byte per lane; 0 if the SoC has none. + */ + u32 fast_die; + /* With fast_die: number of lanes at fast_die, at most 4. */ + u32 fast_die_lanes; }; struct apple_pmp_report { @@ -57,6 +73,10 @@ struct apple_pmp_report { void __iomem *temp_sram; spinlock_t lock; + /* Only with offsets->published */ + struct list_head list; + struct dentry *debugfs; + /* Only with offsets->starts_pmp */ struct device_node *pmp; u64 seed; @@ -1086,6 +1106,128 @@ int apple_pmp_report_wait_supplier_ready(struct device *supplier, unsigned int t } EXPORT_SYMBOL_GPL(apple_pmp_report_wait_supplier_ready); +/* Bound published report regions, protected by RCU until managed teardown drains readers. */ +static LIST_HEAD(apple_pmp_reports); +static DEFINE_MUTEX(apple_pmp_reports_lock); +static struct dentry *apple_pmp_report_debugfs; + +/** + * apple_pmp_report_fast_die_effort() - read the fast die-temperature effort + * of a CPU cluster + * @np: device tree node of the report region + * @lane: lane of the cluster in the fast die-temperature report + * @effort: set to the effort on success + * + * The PMP of some SoCs runs a control loop on its own die-temperature sensors + * and publishes an 8-bit effort for each CPU cluster: 0 requests no limit, + * and the effort rises towards 255 as the die exceeds the firmware's target. + * The effort is only valid while the PMP reports that it is running; a + * stopped PMP reads 0. + * + * Context: Any context. + * Return: 0 on success, -EPROBE_DEFER if the region is not bound (yet) or + * is not a published region (only T8140 regions are), -EOPNOTSUPP if the + * SoC's PMP publishes no such effort, -EINVAL if @lane is out of range, or + * -EAGAIN if the PMP does not report that it is running. + */ +int apple_pmp_report_fast_die_effort(const struct device_node *np, + unsigned int lane, u8 *effort) +{ + const struct apple_pmp_report_offsets *offs; + const struct apple_pmp_report *rep; + int ret = -EPROBE_DEFER; + + rcu_read_lock(); + list_for_each_entry_rcu(rep, &apple_pmp_reports, list) { + if (rep->dev->of_node != np) + continue; + + offs = rep->offsets; + if (!offs->fast_die) { + ret = -EOPNOTSUPP; + } else if (lane >= offs->fast_die_lanes) { + ret = -EINVAL; + } else if (!(readl(rep->base + offs->status) & PMP_REPORT_READY)) { + ret = -EAGAIN; + } else { + *effort = readl(rep->base + offs->fast_die) >> + (lane * BITS_PER_BYTE); + ret = 0; + } + break; + } + rcu_read_unlock(); + + return ret; +} +EXPORT_SYMBOL_GPL(apple_pmp_report_fast_die_effort); + +/* Read only the documented status and fast-die entries, never the aperture. */ +static int apple_pmp_report_status_show(struct seq_file *s, void *unused) +{ + struct apple_pmp_report *rep = s->private; + const struct apple_pmp_report_offsets *offs = rep->offsets; + u32 status = readl(rep->base + offs->status); + unsigned int lane; + u32 effort; + + seq_printf(s, "running: %u\n", !!(status & PMP_REPORT_READY)); + if (!(status & PMP_REPORT_READY) || !offs->fast_die) + return 0; + + effort = readl(rep->base + offs->fast_die); + for (lane = 0; lane < offs->fast_die_lanes; lane++) + seq_printf(s, "lane%u: %u\n", lane, + (effort >> (lane * BITS_PER_BYTE)) & 0xff); + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(apple_pmp_report_status); + +static void apple_pmp_report_release(void *data) +{ + struct apple_pmp_report *rep = data; + + /* Withdraw child providers while their power callbacks can still use MMIO. */ + of_platform_depopulate(rep->dev); + mutex_lock(&apple_pmp_reports_lock); + if (!list_empty(&rep->list)) + list_del_rcu(&rep->list); + mutex_unlock(&apple_pmp_reports_lock); + synchronize_rcu(); + /* The debugfs proxy drains active reads and rejects subsequent accesses. */ + debugfs_remove(rep->debugfs); +} + +/* + * With offsets->published: publish the region before its children are + * populated, and withdraw it again on teardown, children first. + */ +static int apple_pmp_report_publish(struct apple_pmp_report *rep) +{ + struct device *dev = rep->dev; + int ret; + + INIT_LIST_HEAD(&rep->list); + ret = devm_add_action_or_reset(dev, apple_pmp_report_release, rep); + if (ret) + return ret; + + mutex_lock(&apple_pmp_reports_lock); + list_add_tail_rcu(&rep->list, &apple_pmp_reports); + /* Only SoCs with a fast die-temperature report have the directory. */ + if (rep->offsets->fast_die && !apple_pmp_report_debugfs) + apple_pmp_report_debugfs = debugfs_create_dir("apple-pmp-report", NULL); + mutex_unlock(&apple_pmp_reports_lock); + + if (rep->offsets->fast_die) + rep->debugfs = debugfs_create_file(dev_name(dev), 0400, + apple_pmp_report_debugfs, rep, + &apple_pmp_report_status_fops); + + return 0; +} + static int apple_pmp_report_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; @@ -1135,6 +1277,12 @@ static int apple_pmp_report_probe(struct platform_device *pdev) if (ret) return ret; } + if (rep->offsets->published) { + /* Publish the region before child power callbacks start looking it up. */ + ret = apple_pmp_report_publish(rep); + if (ret) + return ret; + } ret = of_platform_populate(np, NULL, NULL, dev); if (ret && !rep->offsets->starts_pmp) return dev_err_probe(dev, ret, "failed to create child devices\n"); @@ -1234,6 +1382,17 @@ static const struct apple_pmp_report_offsets apple_pmp_offsets_t6031 = { .devices_ordered = true, }; +/* The fast die-temperature report is the low two bytes of entry 0x1c1. */ +static const struct apple_pmp_report_offsets apple_pmp_offsets_t8140 = { + .tgt_read = 0x1880, + .tgt_write = 0x10c40, + .actual = 0x18c0, + .status = 0x10, + .published = true, + .fast_die = 0x1c10, + .fast_die_lanes = 2, +}; + /* Same PTD apertures; startup belongs to the native PMP supplier. */ static const struct apple_pmp_report_offsets apple_pmp_offsets_j613_25g83 = { .tgt_read = 0x1000, @@ -1251,6 +1410,7 @@ static const struct of_device_id apple_pmp_report_of_match[] = { { .compatible = "apple,t6030-pmp-v2-report", .data = &apple_pmp_offsets_t6030 }, { .compatible = "apple,t8122-pmp-v2-report", .data = &apple_pmp_offsets_t8122 }, { .compatible = "apple,t6031-pmp-v2-report", .data = &apple_pmp_offsets_t6031 }, + { .compatible = "apple,t8140-pmp-v2-report", .data = &apple_pmp_offsets_t8140 }, {} }; @@ -1269,16 +1429,23 @@ struct apple_pmp_report_entry { u32 id; /* The PMP does not acknowledge this request. */ bool no_ack; + + /* + * Only for an entry of a published report (T8140), which owns this + * storage and looks its report up by node: see apple_pmp_t8140_entry_probe(). + */ + struct device_node *node; + struct device_node *report_node; + bool removed; }; #define genpd_to_apple_pmp_report_entry(_genpd) \ container_of(_genpd, struct apple_pmp_report_entry, genpd) -static int apple_pmp_report_set_state(struct generic_pm_domain *genpd, bool enable) +static int apple_pmp_report_request(struct apple_pmp_report *rep, u32 id, + bool enable, bool no_ack) { - struct apple_pmp_report_entry *ent = genpd_to_apple_pmp_report_entry(genpd); - struct apple_pmp_report *rep = dev_get_drvdata(ent->dev->parent); - u64 bit_val = BIT_ULL(ent->id); + u64 bit_val = BIT_ULL(id); u64 val; unsigned long flags; @@ -1290,7 +1457,7 @@ static int apple_pmp_report_set_state(struct generic_pm_domain *genpd, bool enab writeq(val, rep->base + rep->offsets->tgt_write); spin_unlock_irqrestore(&rep->lock, flags); val = readq(rep->base + rep->offsets->status); - if ((val & PMP_REPORT_READY) == 0 || ent->no_ack) + if ((val & PMP_REPORT_READY) == 0 || no_ack) return 0; return readq_poll_timeout_atomic( rep->base + rep->offsets->actual, @@ -1300,6 +1467,14 @@ static int apple_pmp_report_set_state(struct generic_pm_domain *genpd, bool enab 50000); } +static int apple_pmp_report_set_state(struct generic_pm_domain *genpd, bool enable) +{ + struct apple_pmp_report_entry *ent = genpd_to_apple_pmp_report_entry(genpd); + struct apple_pmp_report *rep = dev_get_drvdata(ent->dev->parent); + + return apple_pmp_report_request(rep, ent->id, enable, ent->no_ack); +} + static int apple_pmp_report_entry_power_on(struct generic_pm_domain *genpd) { return apple_pmp_report_set_state(genpd, true); @@ -1310,16 +1485,176 @@ static int apple_pmp_report_entry_power_off(struct generic_pm_domain *genpd) return apple_pmp_report_set_state(genpd, false); } +static int apple_pmp_t8140_entry_set_state(struct generic_pm_domain *genpd, bool enable) +{ + struct apple_pmp_report_entry *ent = genpd_to_apple_pmp_report_entry(genpd); + struct apple_pmp_report *rep; + int ret = -ENODEV; + + if (READ_ONCE(ent->removed)) + return ret; + + rcu_read_lock(); + list_for_each_entry_rcu(rep, &apple_pmp_reports, list) { + if (rep->dev->of_node == ent->report_node) { + ret = apple_pmp_report_request(rep, ent->id, enable, false); + break; + } + } + rcu_read_unlock(); + return ret; +} + +static int apple_pmp_t8140_entry_power_on(struct generic_pm_domain *genpd) +{ + return apple_pmp_t8140_entry_set_state(genpd, true); +} + +static int apple_pmp_t8140_entry_power_off(struct generic_pm_domain *genpd) +{ + return apple_pmp_t8140_entry_set_state(genpd, false); +} + +static void apple_pmp_report_entry_free(struct apple_pmp_report_entry *ent) +{ + of_node_put(ent->report_node); + of_node_put(ent->node); + kfree(ent->genpd.name); + kfree(ent); +} + +static void apple_pmp_report_entry_remove(struct platform_device *pdev) +{ + struct apple_pmp_report_entry *ent = platform_get_drvdata(pdev); + + /* Only entries of a published report (T8140) are withdrawn. */ + if (!ent) + return; + + WRITE_ONCE(ent->removed, true); + of_genpd_del_provider(ent->node); + if (pm_genpd_remove(&ent->genpd)) { + /* The core still owns this domain. Keep its now inert storage alive. */ + dev_warn(&pdev->dev, "retaining busy power domain %s\n", ent->genpd.name); + return; + } + apple_pmp_report_entry_free(ent); +} + +/* + * An entry of a published report (T8140). Its storage, name and node + * references are its own, not the device's: if genpd refuses to remove a + * busy domain, they stay alive with callbacks that return -ENODEV. + */ +static int apple_pmp_t8140_entry_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct device_node *node = dev->of_node; + struct apple_pmp_report_entry *ent; + int ret; + const char *name; + struct of_phandle_iterator it; + + ent = kzalloc_obj(*ent); + if (!ent) + return -ENOMEM; + + ent->node = of_node_get(node); + ent->report_node = of_node_get(dev->parent->of_node); + + ret = of_property_read_u32(node, "reg", &ent->id); + if (ret) { + ret = dev_err_probe(dev, ret, "missing reg property\n"); + goto err_free; + } + if (ent->id >= 64) { + ret = -EINVAL; + goto err_free; + } + + ret = of_property_read_string(node, "label", &name); + if (ret < 0) { + ret = dev_err_probe(dev, ret, "missing label property\n"); + goto err_free; + } + ent->genpd.name = kstrdup(name, GFP_KERNEL); + if (!ent->genpd.name) { + ret = -ENOMEM; + goto err_free; + } + + if (of_property_read_bool(node, "apple,always-on")) { + ent->genpd.flags |= GENPD_FLAG_ACTIVE_WAKEUP; + apple_pmp_t8140_entry_set_state(&ent->genpd, true); + } + + ent->genpd.power_on = apple_pmp_t8140_entry_power_on; + ent->genpd.power_off = apple_pmp_t8140_entry_power_off; + + ret = pm_genpd_init(&ent->genpd, NULL, true); + if (ret) { + ret = dev_err_probe(dev, ret, "pm_genpd_init failed\n"); + goto err_free; + } + + ret = of_genpd_add_provider_simple(node, &ent->genpd); + if (ret) { + ret = dev_err_probe(dev, ret, "of_genpd_add_provider_simple failed\n"); + goto err_remove_domain; + } + + of_for_each_phandle(&it, ret, node, "power-domains", "#power-domain-cells", -1) { + struct of_phandle_args parent, child; + + parent.np = it.node; + parent.args_count = of_phandle_iterator_args(&it, parent.args, MAX_PHANDLE_ARGS); + child.np = node; + child.args_count = 0; + ret = of_genpd_add_subdomain(&parent, &child); + + if (ret == -EPROBE_DEFER) { + of_node_put(parent.np); + goto err_remove; + } else if (ret < 0) { + dev_err(dev, "failed to add to parent domain: %d (%s -> %s)\n", + ret, it.node->name, node->name); + of_node_put(parent.np); + goto err_remove; + } + } + + pm_genpd_remove_device(dev); + platform_set_drvdata(pdev, ent); + + return 0; +err_remove: + of_genpd_del_provider(node); +err_remove_domain: + WRITE_ONCE(ent->removed, true); + if (pm_genpd_remove(&ent->genpd)) { + dev_warn(dev, "retaining busy power domain %s\n", ent->genpd.name); + return ret; + } +err_free: + apple_pmp_report_entry_free(ent); + return ret; +} + static int apple_pmp_report_entry_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct device_node *node = dev->of_node; + const struct apple_pmp_report *rep = dev_get_drvdata(dev->parent); struct apple_pmp_report_entry *ent; int ret; bool record; const char *name; struct of_phandle_iterator it; + /* Entries of a published report (T8140) look it up by node. */ + if (rep->offsets->published) + return apple_pmp_t8140_entry_probe(pdev); + ent = devm_kzalloc(dev, sizeof(*ent), GFP_KERNEL); if (!ent) return -ENOMEM; @@ -1389,6 +1724,7 @@ static const struct of_device_id apple_pmp_report_entry_of_match[] = { static struct platform_driver apple_pmp_report_entry_driver = { .probe = apple_pmp_report_entry_probe, + .remove = apple_pmp_report_entry_remove, .driver = { .name = "apple-pmp-report-entry", .of_match_table = apple_pmp_report_entry_of_match, @@ -1409,6 +1745,7 @@ static void __exit apple_pmp_report_exit(void) { platform_driver_unregister(&apple_pmp_report_entry_driver); platform_driver_unregister(&apple_pmp_report_driver); + debugfs_remove_recursive(apple_pmp_report_debugfs); } module_init(apple_pmp_report_init); diff --git a/drivers/reset/reset-apple-cio.c b/drivers/reset/reset-apple-cio.c index 5e818246ef32f2..1154272a19c07b 100644 --- a/drivers/reset/reset-apple-cio.c +++ b/drivers/reset/reset-apple-cio.c @@ -18,6 +18,7 @@ * t8103 0 0x84000 + 0x4000 * port BIT(0), rmw BIT(1) * t8112 0x63 0x8000 / 0x4000 BIT(0), rmw BIT(1) * t8122 0x62 0x8 BIT(slot) BIT(16 + slot) + * t8142 0x62 0x30 BIT(2 * port) BIT(2 * port + 1) * t6000 0 0x1a034 BIT(port) BIT(16 + port) * t6020 0 0xa02c BIT(port) BIT(16 + port) * t6030 0x4c 0x8 BIT(slot) BIT(16 + slot) @@ -36,10 +37,12 @@ * separate 16 KiB page at 0x350780000, so the register is 0x350780008. The * device tree describes that page as its own syscon. t8122 is unresolved. * - * t8132 and t8142 use a different, per-port layout and are not supported. + * On t8142, selector 0x62 is a separate MMIO region at 0x380780000, using a + * per-port layout. The t8132 layout has not been decoded. * - * Every implementation polls the busy bits, stores the request, then polls - * again, with a 192 ms budget. The first poll's result is discarded. + * Reference firmware polls the busy bits, stores the request, then polls + * again, with a 192 ms budget. It discards the first poll's result. The + * t8142 implementation below instead refuses to issue a request while busy. */ #include @@ -62,6 +65,11 @@ #define T6000_CIO_CTRL_INIT_REQ(port) BIT(port) #define T6000_CIO_CTRL_INIT_BUSY(port) BIT(16 + (port)) +#define T8142_CIO_CTRL 0x30 +#define T8142_CIO_CTRL_INIT_REQ(port) BIT(2 * (port)) +#define T8142_CIO_CTRL_INIT_BUSY(port) BIT(2 * (port) + 1) +#define T8142_CIO_CTRL_BUSY_MASK 0xaa + struct apple_cio_reset; struct apple_cio_reset_variant { @@ -143,6 +151,37 @@ static int t6000_cio_deassert(struct apple_cio_reset *priv, unsigned long id) APPLE_CIO_RESET_TIMEOUT_US); } +static int t8142_cio_deassert(struct apple_cio_reset *priv, unsigned long id) +{ + u32 val; + int ret; + + guard(mutex)(&priv->lock); + + /* Serialize all four ports sharing the request register. */ + ret = regmap_read_poll_timeout(priv->regmap, T8142_CIO_CTRL, val, + !(val & T8142_CIO_CTRL_BUSY_MASK), + APPLE_CIO_RESET_POLL_US, + APPLE_CIO_RESET_TIMEOUT_US); + if (ret) + return ret; + + ret = regmap_write(priv->regmap, T8142_CIO_CTRL, + T8142_CIO_CTRL_INIT_REQ(id)); + if (ret) + return ret; + + return regmap_read_poll_timeout(priv->regmap, T8142_CIO_CTRL, val, + !(val & T8142_CIO_CTRL_INIT_BUSY(id)), + APPLE_CIO_RESET_POLL_US, + APPLE_CIO_RESET_TIMEOUT_US); +} + +static const struct apple_cio_reset_variant apple_t8142_cio_reset = { + .nr_resets = 4, + .deassert = t8142_cio_deassert, +}; + static const struct apple_cio_reset_variant apple_t8103_cio_reset = { .nr_resets = 2, .deassert = t8103_cio_deassert, @@ -221,6 +260,10 @@ static int apple_cio_reset_probe(struct platform_device *pdev) } static const struct of_device_id apple_cio_reset_match[] = { + { + .compatible = "apple,t8142-cio-reset", + .data = &apple_t8142_cio_reset, + }, { .compatible = "apple,t8103-cio-reset", .data = &apple_t8103_cio_reset, diff --git a/drivers/rtc/Kconfig b/drivers/rtc/Kconfig index 364afc73f8abd2..e6077d109afb40 100644 --- a/drivers/rtc/Kconfig +++ b/drivers/rtc/Kconfig @@ -2100,6 +2100,17 @@ config RTC_DRV_WILCO_EC This can also be built as a module. If so, the module will be named "rtc_wilco_ec". +config RTC_DRV_APPLE_PMU + tristate "Apple J700 Baku PMU RTC alarm" + depends on ARCH_APPLE && NVMEM_APPLE_SPMI && PM_SLEEP + help + Read the Baku PMU clock and use its alarm to wake the system + from suspend-to-idle. Alarms are limited to one hour; deeper + suspend states are not supported. Continue setting the shared + clock through the SMC RTC. + + This driver can be built as the module rtc-apple-pmu. + config RTC_DRV_MACSMC tristate "Apple Mac System Management Controller RTC" depends on MFD_MACSMC diff --git a/drivers/rtc/Makefile b/drivers/rtc/Makefile index 6cf7e066314e13..729ff56c871d3d 100644 --- a/drivers/rtc/Makefile +++ b/drivers/rtc/Makefile @@ -94,6 +94,7 @@ obj-$(CONFIG_RTC_DRV_M48T35) += rtc-m48t35.o obj-$(CONFIG_RTC_DRV_M48T59) += rtc-m48t59.o obj-$(CONFIG_RTC_DRV_M48T86) += rtc-m48t86.o obj-$(CONFIG_RTC_DRV_MA35D1) += rtc-ma35d1.o +obj-$(CONFIG_RTC_DRV_APPLE_PMU) += rtc-apple-pmu.o obj-$(CONFIG_RTC_DRV_MACSMC) += rtc-macsmc.o obj-$(CONFIG_RTC_DRV_MAX31335) += rtc-max31335.o obj-$(CONFIG_RTC_DRV_MAX6900) += rtc-max6900.o diff --git a/drivers/rtc/rtc-apple-pmu.c b/drivers/rtc/rtc-apple-pmu.c new file mode 100644 index 00000000000000..2e3161a914c8a2 --- /dev/null +++ b/drivers/rtc/rtc-apple-pmu.c @@ -0,0 +1,382 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Apple Baku PMU clock and retained-power alarm. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define BAKU_RTC_CONTROL 0xf800 +#define BAKU_RTC_COUNTER 0xf802 +#define BAKU_RTC_COMPARE 0xf808 +#define BAKU_RTC_EVENT 0xf80c +#define BAKU_RTC_MASK 0xf80e +#define BAKU_RTC_ARM BIT(6) +#define BAKU_RTC_ALARM BIT(0) +#define BAKU_RTC_HZ_SHIFT 16 +#define BAKU_RTC_MAX_DELAY 3600 + +struct baku_rtc { + struct regmap *map; + struct nvmem_cell *offset; + struct rtc_device *clock; + /* Serialize comparator updates and interrupt acknowledgment. */ + struct mutex transaction; + int interrupt; + bool io_failed; +}; + +static int baku_rtc_seconds(struct baku_rtc *baku, u64 *seconds) +{ + u8 count[8] = {}; + int error; + + error = regmap_bulk_read(baku->map, BAKU_RTC_COUNTER, count, 6); + if (error) + return error; + *seconds = get_unaligned_le64(count) >> BAKU_RTC_HZ_SHIFT; + return 0; +} + +static int baku_rtc_offset(struct baku_rtc *baku, s64 *seconds) +{ + size_t bytes; + void *value; + + /* The SMC RTC can update this cell, so do not cache it at probe. */ + value = nvmem_cell_read(baku->offset, &bytes); + if (IS_ERR(value)) + return PTR_ERR(value); + if (bytes != sizeof(__le64)) { + kfree(value); + return -EINVAL; + } + *seconds = get_unaligned_le64(value); + kfree(value); + if (*seconds < 0 || *seconds > RTC_TIMESTAMP_END_9999 - U32_MAX) + return -ERANGE; + return 0; +} + +static int baku_rtc_read_time(struct device *dev, struct rtc_time *time) +{ + struct baku_rtc *baku = dev_get_drvdata(dev); + u64 counter; + s64 epoch; + int error; + + guard(mutex)(&baku->transaction); + error = baku_rtc_offset(baku, &epoch); + if (error) + return error; + error = baku_rtc_seconds(baku, &counter); + if (error) + return error; + rtc_time64_to_tm(epoch + counter, time); + return 0; +} + +/* All comparator/control accesses are serialized with the IRQ thread. */ +static int baku_rtc_stop_alarm(struct baku_rtc *baku) +{ + int error; + + error = regmap_update_bits(baku->map, BAKU_RTC_MASK, + BAKU_RTC_ALARM, BAKU_RTC_ALARM); + if (error) + return error; + error = regmap_update_bits(baku->map, BAKU_RTC_CONTROL, BAKU_RTC_ARM, 0); + if (error) + return error; + return regmap_write(baku->map, BAKU_RTC_EVENT, BAKU_RTC_ALARM); +} + +static int baku_rtc_start_alarm(struct baku_rtc *baku) +{ + int error; + + error = regmap_write(baku->map, BAKU_RTC_EVENT, BAKU_RTC_ALARM); + if (!error) + error = regmap_update_bits(baku->map, BAKU_RTC_MASK, BAKU_RTC_ALARM, 0); + if (!error) + error = regmap_update_bits(baku->map, BAKU_RTC_CONTROL, + BAKU_RTC_ARM, BAKU_RTC_ARM); + if (error) + baku_rtc_stop_alarm(baku); + return error; +} + +static int baku_rtc_validate_alarm(struct baku_rtc *baku, u64 comparator) +{ + u64 counter; + int error; + + error = baku_rtc_seconds(baku, &counter); + if (error) + return error; + if (comparator <= counter) + return -ETIME; + if (comparator - counter > BAKU_RTC_MAX_DELAY) + return -ERANGE; + return 0; +} + +static int baku_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alarm) +{ + struct baku_rtc *baku = dev_get_drvdata(dev); + unsigned int control, event, mask; + __le32 comparator; + s64 epoch; + int error; + + guard(mutex)(&baku->transaction); + error = baku_rtc_offset(baku, &epoch); + if (error) + return error; + error = regmap_bulk_read(baku->map, BAKU_RTC_COMPARE, + &comparator, sizeof(comparator)); + if (!error) + error = regmap_read(baku->map, BAKU_RTC_CONTROL, &control); + if (!error) + error = regmap_read(baku->map, BAKU_RTC_EVENT, &event); + if (!error) + error = regmap_read(baku->map, BAKU_RTC_MASK, &mask); + if (error) + return error; + rtc_time64_to_tm(epoch + le32_to_cpu(comparator), &alarm->time); + alarm->enabled = (control & BAKU_RTC_ARM) && !(mask & BAKU_RTC_ALARM); + alarm->pending = !!(event & BAKU_RTC_ALARM); + return 0; +} + +static int baku_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alarm) +{ + struct baku_rtc *baku = dev_get_drvdata(dev); + __le32 comparator, readback; + s64 epoch, deadline; + int error; + + guard(mutex)(&baku->transaction); + if (baku->io_failed) + return -EIO; + error = baku_rtc_offset(baku, &epoch); + if (error) + return error; + deadline = rtc_tm_to_time64(&alarm->time) - epoch; + if (deadline < 0 || deadline > U32_MAX) + return -ERANGE; + error = baku_rtc_validate_alarm(baku, deadline); + if (error) + return error; + error = baku_rtc_stop_alarm(baku); + if (error) + return error; + comparator = cpu_to_le32(deadline); + error = regmap_bulk_write(baku->map, BAKU_RTC_COMPARE, + &comparator, sizeof(comparator)); + if (error) + return error; + error = regmap_bulk_read(baku->map, BAKU_RTC_COMPARE, + &readback, sizeof(readback)); + if (error) + return error; + if (readback != comparator) + return -EIO; + if (alarm->enabled) + return baku_rtc_start_alarm(baku); + return 0; +} + +static int baku_rtc_alarm_enable(struct device *dev, unsigned int enable) +{ + struct baku_rtc *baku = dev_get_drvdata(dev); + __le32 comparator; + int error; + + guard(mutex)(&baku->transaction); + if (!enable) + return baku_rtc_stop_alarm(baku); + if (baku->io_failed) + return -EIO; + error = regmap_bulk_read(baku->map, BAKU_RTC_COMPARE, + &comparator, sizeof(comparator)); + if (error) + return error; + error = baku_rtc_validate_alarm(baku, le32_to_cpu(comparator)); + return error ?: baku_rtc_start_alarm(baku); +} + +static irqreturn_t baku_rtc_interrupt(int irq, void *context) +{ + struct device *dev = context; + struct baku_rtc *baku = dev_get_drvdata(dev); + unsigned int event; + int error; + + mutex_lock(&baku->transaction); + error = regmap_read(baku->map, BAKU_RTC_EVENT, &event); + if (!error && (event & BAKU_RTC_ALARM)) + error = baku_rtc_stop_alarm(baku); + if (error) { + baku->io_failed = true; + disable_irq_nosync(irq); + } + mutex_unlock(&baku->transaction); + if (error) { + dev_err(dev, "cannot acknowledge RTC interrupt: %d\n", error); + pm_wakeup_dev_event(dev, 0, true); + return IRQ_HANDLED; + } + if (!(event & BAKU_RTC_ALARM)) + return IRQ_NONE; + if (device_may_wakeup(dev)) + pm_wakeup_dev_event(dev, 0, true); + rtc_update_irq(baku->clock, 1, RTC_IRQF | RTC_AF); + return IRQ_HANDLED; +} + +/* + * Firmware leaves the alarm armed even when it has no wake scheduled, with + * the comparator set close to the end of 32-bit time. Keep an alarm that is + * still pending within the supported range, so that the RTC core picks it + * up at registration, and disarm anything else. + */ +static int baku_rtc_init_alarm(struct baku_rtc *baku) +{ + unsigned int control, mask; + __le32 comparator; + int error; + + error = regmap_read(baku->map, BAKU_RTC_CONTROL, &control); + if (!error) + error = regmap_read(baku->map, BAKU_RTC_MASK, &mask); + if (!error) + error = regmap_bulk_read(baku->map, BAKU_RTC_COMPARE, + &comparator, sizeof(comparator)); + if (error) + return error; + if ((control & BAKU_RTC_ARM) && !(mask & BAKU_RTC_ALARM) && + !baku_rtc_validate_alarm(baku, le32_to_cpu(comparator))) + return 0; + return baku_rtc_stop_alarm(baku); +} + +static const struct rtc_class_ops baku_rtc_ops = { + .read_time = baku_rtc_read_time, + .read_alarm = baku_rtc_read_alarm, + .set_alarm = baku_rtc_set_alarm, + .alarm_irq_enable = baku_rtc_alarm_enable, +}; + +static int baku_rtc_prepare(struct device *dev) +{ + struct baku_rtc *baku = dev_get_drvdata(dev); + + if (pm_suspend_target_state != PM_SUSPEND_TO_IDLE || baku->io_failed) + return -EBUSY; + return 0; +} + +static const struct dev_pm_ops baku_rtc_pm = { + .prepare = baku_rtc_prepare, +}; + +static void baku_rtc_release(void *context) +{ + struct device *dev = context; + struct baku_rtc *baku = dev_get_drvdata(dev); + + disable_irq(baku->interrupt); + mutex_lock(&baku->transaction); + if (baku_rtc_stop_alarm(baku)) + dev_err(dev, "cannot disable RTC alarm on removal\n"); + mutex_unlock(&baku->transaction); + device_init_wakeup(dev, false); +} + +static int baku_rtc_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct baku_rtc *baku; + int error; + + baku = devm_kzalloc(dev, sizeof(*baku), GFP_KERNEL); + if (!baku) + return -ENOMEM; + baku->map = dev_get_regmap(dev->parent, NULL); + if (!baku->map) + return dev_err_probe(dev, -EPROBE_DEFER, "PMIC regmap unavailable\n"); + baku->offset = devm_nvmem_cell_get(dev, "rtc_offset"); + if (IS_ERR(baku->offset)) + return dev_err_probe(dev, PTR_ERR(baku->offset), "RTC offset unavailable\n"); + baku->interrupt = platform_get_irq(pdev, 0); + if (baku->interrupt < 0) + return baku->interrupt; + mutex_init(&baku->transaction); + platform_set_drvdata(pdev, baku); + baku->clock = devm_rtc_allocate_device(dev); + if (IS_ERR(baku->clock)) + return PTR_ERR(baku->clock); + baku->clock->ops = &baku_rtc_ops; + baku->clock->range_min = 0; + baku->clock->range_max = RTC_TIMESTAMP_END_9999; + baku->clock->alarm_offset_max = BAKU_RTC_MAX_DELAY; + clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, baku->clock->features); + error = device_init_wakeup(dev, true); + if (error) + return error; + /* + * This interrupt remains live during retained-power s2idle. The + * thread uses the parent PMIC's SPMI regmap to acknowledge the alarm. + * Off-domain wake routing is not established by this driver. + */ + error = devm_request_threaded_irq(dev, baku->interrupt, NULL, + baku_rtc_interrupt, + IRQF_NO_AUTOEN | IRQF_NO_SUSPEND | IRQF_ONESHOT, + dev_name(dev), dev); + if (error) + goto disable_wakeup; + error = baku_rtc_init_alarm(baku); + if (error) + goto disable_wakeup; + error = devm_rtc_register_device(baku->clock); + if (error) + goto disable_wakeup; + error = devm_add_action_or_reset(dev, baku_rtc_release, dev); + if (error) + return error; + enable_irq(baku->interrupt); + return 0; + +disable_wakeup: + device_init_wakeup(dev, false); + return error; +} + +static const struct of_device_id baku_rtc_match[] = { + { .compatible = "apple,j700-pmu-rtc" }, + { } +}; +MODULE_DEVICE_TABLE(of, baku_rtc_match); + +static struct platform_driver baku_rtc_driver = { + .probe = baku_rtc_probe, + .driver = { + .name = "apple-pmu-rtc", + .of_match_table = baku_rtc_match, + .pm = &baku_rtc_pm, + }, +}; +module_platform_driver(baku_rtc_driver); + +MODULE_AUTHOR("Eryk Wieliczko "); +MODULE_DESCRIPTION("Apple Baku PMU RTC alarm with retained-power s2idle"); +MODULE_LICENSE("GPL"); diff --git a/drivers/rtc/rtc-macsmc.c b/drivers/rtc/rtc-macsmc.c index 25d5af5cfbbcd3..1c1c529df82966 100644 --- a/drivers/rtc/rtc-macsmc.c +++ b/drivers/rtc/rtc-macsmc.c @@ -113,7 +113,10 @@ static int macsmc_rtc_set_time(struct device *dev, struct rtc_time *tm) if (rtc->offset_in_seconds) { __le64 off_secs; - /* The counter's fraction of a second carries into the new time */ + /* + * The offset holds whole seconds, so the counter's fraction of a + * second is lost: the clock may run up to a second ahead. + */ off_secs = cpu_to_le64(rtc_tm_to_time64(tm) - (ctr >> RTC_SEC_SHIFT)); ret = nvmem_cell_write(rtc->rtc_offset, &off_secs, sizeof(off_secs)); return ret < 0 ? ret : 0; diff --git a/drivers/soc/apple/Kconfig b/drivers/soc/apple/Kconfig index 5cec2b22a5de87..938cfc26b03d68 100644 --- a/drivers/soc/apple/Kconfig +++ b/drivers/soc/apple/Kconfig @@ -25,6 +25,18 @@ config APPLE_MAILBOX Say Y here if you have an Apple SoC. +config APPLE_MCC + tristate "Apple T8140 system level cache" + depends on ARCH_APPLE || COMPILE_TEST + depends on OF + default ARCH_APPLE + help + The boot firmware of the Apple T8140 SoC enables the system level + cache without letting it allocate. This driver allows allocation + once the devices that took over DMA from the bootloader are bound. + + Say 'y' or 'm' here if you have an Apple T8140 SoC. + config APPLE_PMGR_MISC bool "Apple SoC PMGR miscellaneous support" depends on PM @@ -104,6 +116,7 @@ config APPLE_SEP_FV_INTERFACE config APPLE_SEP tristate "Apple SEP (Secure Enclave Processor)" + default m if ARCH_APPLE depends on ARCH_APPLE depends on PM depends on RUST @@ -129,9 +142,11 @@ config APPLE_SEP the iBoot system container, locating its .gl extent through read-only, checksum-validated APFS metadata. It validates the complete store and both root records before registering with SEP. - Shared xART writes and identity-keybag creation default on. A write - still runs only when APFS proves a single unsnapshotted .gl extent. - The first boot creates the Linux identity keybag when it is absent. + xART writes are enabled by default, but accepted only when the driver + validates the physical extent ownership invariants. Unsupported APFS + layouts fail closed. A Linux identity keybag is created automatically + when none exists; existing identities and reference keys are retained. + Set xart_writes=0 provision_keybag=0 for read-only operation. With writes off the driver still attaches, but refuses Touch ID enrol and verify: the enclave saves anti-replay state after every match. @@ -199,23 +214,6 @@ config APPLE_DISPLAY_GATE_KUNIT_TEST nothing changed when a check fails. On T6030: on by default, as before, whatever the T6031 option says. -endmenu - -endif - -config APPLE_MCC - tristate "Apple T8140 system level cache" - depends on ARCH_APPLE || COMPILE_TEST - depends on OF - default ARCH_APPLE - help - The boot firmware of the Apple T8140 SoC enables the system level - cache without letting it allocate. This driver allows allocation - once the devices that took over DMA from the bootloader are bound. - - Say 'y' or 'm' here if you have an Apple T8140 SoC. - - config APPLE_CLPC tristate "Apple host performance controller floors" depends on ARCH_APPLE || COMPILE_TEST @@ -228,3 +226,7 @@ config APPLE_CLPC are busy, and releases them at idle. Say 'y' or 'm' here if you have an Apple SoC with a PMC. + +endmenu + +endif diff --git a/drivers/soc/apple/bio.rs b/drivers/soc/apple/bio.rs index c2bbf63364f3ae..6b4e1edbe29664 100644 --- a/drivers/soc/apple/bio.rs +++ b/drivers/soc/apple/bio.rs @@ -152,7 +152,7 @@ pub(crate) struct Attest { /// in: challenge to sign. pub(crate) challenge: [u8; ATTEST_CHALLENGE_LEN], /// out: ref-key public point. - pub(crate) public: [u8; ATTEST_PUB_LEN], + pub(crate) public_key: [u8; ATTEST_PUB_LEN], /// out: DER signature, zero-padded to `ATTEST_SIG_MAX`. pub(crate) signature: [u8; ATTEST_SIG_MAX], pub(crate) reserved: [u8; 3], @@ -420,6 +420,8 @@ impl Op { } pub(crate) struct Session { + generation: u64, + worker_epoch: Option, open: bool, op: Op, unseen: bool, @@ -430,6 +432,8 @@ pub(crate) struct Session { impl Session { pub(crate) fn new() -> Session { Session { + generation: 0, + worker_epoch: None, open: false, op: Op::Idle, unseen: false, @@ -439,6 +443,7 @@ impl Session { } fn reset(&mut self) { + self.generation = self.generation.saturating_add(1); self.open = false; self.op = Op::Idle; self.unseen = false; @@ -446,6 +451,15 @@ impl Session { self.token = None; } + fn start_worker(&mut self) -> Result<()> { + if self.worker_epoch.is_some() { + return Err(EBUSY); + } + self.generation = self.generation.checked_add(1).ok_or(EOVERFLOW)?; + self.worker_epoch = Some(self.generation); + Ok(()) + } + pub(crate) fn ready(&self) -> bool { self.unseen } @@ -563,7 +577,7 @@ fn enrol_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { } require_sensor(ctx.sensor_present)?; - if !ctx.session.op.may_start() { + if ctx.session.worker_epoch.is_some() || !ctx.session.op.may_start() { return Err(EBUSY); } if matches!(ctx.live_identity_count, Some(count) if count >= SEP_USER_CAPACITY) @@ -572,6 +586,7 @@ fn enrol_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { return Err(ENOSPC); } + ctx.session.start_worker()?; ctx.session.op = Op::Enrol { label: request.label, stage: 0, @@ -593,10 +608,14 @@ fn enrol_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { pub(crate) fn enrol_advance( session: &mut Session, + epoch: u64, completed: u32, percent_now: u32, guide: Guidance, ) -> bool { + if !worker_is_current(session, epoch) { + return false; + } if let Op::Enrol { stage, terminal, @@ -617,7 +636,10 @@ pub(crate) fn enrol_advance( false } -pub(crate) fn capture_guide(session: &mut Session, guide: Guidance) -> bool { +pub(crate) fn capture_guide(session: &mut Session, epoch: u64, guide: Guidance) -> bool { + if !worker_is_current(session, epoch) { + return false; + } let changed = match &mut session.op { Op::Enrol { terminal, guidance, .. } => { if terminal.is_none() && *guidance != guide { @@ -645,9 +667,13 @@ pub(crate) fn capture_guide(session: &mut Session, guide: Guidance) -> bool { pub(crate) fn enrol_finish( session: &mut Session, + epoch: u64, index: &mut IdentityIndex, outcome: core::result::Result<[u8; UUID_LEN], u32>, ) -> bool { + if !worker_is_current(session, epoch) { + return false; + } let Op::Enrol { label, terminal, .. } = &mut session.op @@ -673,8 +699,8 @@ pub(crate) fn enrol_finish( true } -pub(crate) fn enrol_is_live(session: &Session) -> bool { - matches!(&session.op, Op::Enrol { terminal: None, .. }) && session.open +pub(crate) fn enrol_is_live(session: &Session, epoch: u64) -> bool { + worker_is_current(session, epoch) && matches!(&session.op, Op::Enrol { terminal: None, .. }) } fn enrol_poll(ctx: &mut Context<'_>, user: UserPtr) -> Result { @@ -735,7 +761,7 @@ fn verify_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { } require_sensor(ctx.sensor_present)?; - if !ctx.session.op.may_start() { + if ctx.session.worker_epoch.is_some() || !ctx.session.op.may_start() { return Err(EBUSY); } @@ -757,6 +783,7 @@ fn verify_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { }); } + ctx.session.start_worker()?; ctx.session.token = None; ctx.session.op = Op::Verify { nonce: request.nonce, @@ -775,9 +802,13 @@ fn verify_start(ctx: &mut Context<'_>, user: UserPtr) -> Result { pub(crate) fn verify_finish( session: &mut Session, + epoch: u64, outcome: VerifyOutcome, token_bytes: [u8; TOKEN_LEN], ) -> bool { + if !worker_is_current(session, epoch) { + return false; + } let Op::Verify { nonce, terminal, .. } = &mut session.op else { return false; }; @@ -796,12 +827,33 @@ pub(crate) fn verify_finish( true } -pub(crate) fn verify_is_live(session: &Session) -> bool { - matches!(&session.op, Op::Verify { terminal: None, .. }) && session.open +pub(crate) fn verify_is_live(session: &Session, epoch: u64) -> bool { + worker_is_current(session, epoch) && matches!(&session.op, Op::Verify { terminal: None, .. }) +} + +pub(crate) fn capture_is_live(session: &Session, epoch: u64) -> bool { + enrol_is_live(session, epoch) || verify_is_live(session, epoch) +} + +pub(crate) fn queued_epoch(session: &Session) -> Option { + session.worker_epoch +} + +fn worker_is_current(session: &Session, epoch: u64) -> bool { + session.open && session.generation == epoch && session.worker_epoch == Some(epoch) +} + +pub(crate) fn worker_stopped(session: &mut Session, epoch: u64) { + if session.worker_epoch == Some(epoch) { + session.worker_epoch = None; + } } -pub(crate) fn capture_is_live(session: &Session) -> bool { - enrol_is_live(session) || verify_is_live(session) +/// Whether an open (`opening`) or a capture start by the current opener would +/// be refused only because the last enrol or verify work item has not stopped +/// yet, although its operation has ended or its opener has gone. +pub(crate) fn waiting_on_worker(session: &Session, opening: bool) -> bool { + session.worker_epoch.is_some() && session.op.may_start() && session.open != opening } fn verify_poll(ctx: &mut Context<'_>, user: UserPtr) -> Result { @@ -871,6 +923,7 @@ fn cancel(ctx: &mut Context<'_>) -> Result { Op::Verify { terminal, .. } => *terminal = Some(VerifyOutcome::Failed(STATUS_LOCAL)), } + ctx.session.generation = ctx.session.generation.saturating_add(1); ctx.session.token = None; ctx.session.unseen = true; @@ -972,7 +1025,7 @@ fn delete_all(ctx: &mut Context<'_>) -> Result { } pub(crate) fn open(session: &mut Session) -> Result<()> { - if session.open { + if session.open || session.worker_epoch.is_some() { return Err(EBUSY); } session.reset(); diff --git a/drivers/soc/apple/bio_shim.c b/drivers/soc/apple/bio_shim.c index 50f7e7ddf61ea7..476c9e86cb6c99 100644 --- a/drivers/soc/apple/bio_shim.c +++ b/drivers/soc/apple/bio_shim.c @@ -168,9 +168,9 @@ void sep_bio_unregister(void *dev) mutex_lock(&d->lock); d->ctx = NULL; free_now = (d->open_count == 0); - mutex_unlock(&d->lock); - + /* The last close may free d as soon as the lifetime lock is dropped. */ wake_up_interruptible(&d->wq); + mutex_unlock(&d->lock); if (free_now) { mutex_destroy(&d->lock); diff --git a/drivers/soc/apple/dt.rs b/drivers/soc/apple/dt.rs index 48a4732008967e..51c83d1249cdf8 100644 --- a/drivers/soc/apple/dt.rs +++ b/drivers/soc/apple/dt.rs @@ -400,7 +400,14 @@ unsafe fn add_u32( } } -pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { +// The public J700 DTS enables SPI2. A disabled controller at handoff is a +// firmware hardware-presence decision; older platforms still need activation. +fn sensor_controller_refused(controller: &DtNode) -> bool { + crate::profile::machine_has(b"apple,j700") && !controller.is_available() +} + +/// Returns false when the bootloader explicitly withheld J700's controller. +pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result { // A machine that describes the sensor in its own device tree (compatible // apple,mesa-fingerprint) is handled machine-agnostically: enable that node // and its SPI controller wherever they sit, with the power/IRQ GPIOs coming @@ -411,6 +418,9 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { pr_err!("apple_sep: sensor node has no parent SPI controller\n"); ENODEV })?; + if sensor_controller_refused(&controller) { + return Ok(false); + } pr_info!( "apple_sep: sensor node present in the device tree; enabling it and its SPI bus\n" ); @@ -425,7 +435,7 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { queued = true; } Ok(queued) - }); + }).map(|()| true); } let controller = node_at_address(base).ok_or_else(|| { @@ -436,6 +446,9 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { ENODEV })?; + if sensor_controller_refused(&controller) { + return Ok(false); + } let controller_ok = controller.is_available(); let existing = controller.child_with_reg(cs); pr_info!( @@ -447,7 +460,7 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { ); if controller_ok && existing.is_some() { - return Ok(()); + return Ok(true); } if existing.is_some() && !controller_ok { @@ -456,7 +469,7 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { return with_changeset(|cs_handle| { queue_status_okay(cs_handle, &controller)?; Ok(true) - }); + }).map(|()| true); } pr_info!( @@ -470,7 +483,7 @@ pub(crate) fn enable_spi_sensor(base: u64, cs: u32) -> Result<()> { } queue_sensor_node(cs_handle, &controller)?; Ok(true) - }) + }).map(|()| true) } /// Resolve a reserved-memory region by name on the SEP node's device. diff --git a/drivers/soc/apple/mailbox.c b/drivers/soc/apple/mailbox.c index 13c8c5594305e1..7804cf7922820c 100644 --- a/drivers/soc/apple/mailbox.c +++ b/drivers/soc/apple/mailbox.c @@ -512,6 +512,18 @@ void apple_mbox_stop(struct apple_mbox *mbox) } EXPORT_SYMBOL(apple_mbox_stop); +void apple_mbox_hold_rx(struct apple_mbox *mbox) +{ + disable_irq(mbox->irq_recv_not_empty); +} +EXPORT_SYMBOL(apple_mbox_hold_rx); + +void apple_mbox_release_rx(struct apple_mbox *mbox) +{ + enable_irq(mbox->irq_recv_not_empty); +} +EXPORT_SYMBOL(apple_mbox_release_rx); + /* * Stop the provider-owned CPU, then quiesce every transport edge before * devres releases IRQs or MMIO mappings. Device links order consumer removal diff --git a/drivers/soc/apple/mcc.c b/drivers/soc/apple/mcc.c index 10f5e6e01e8cd1..7b8c2e2220c8ba 100644 --- a/drivers/soc/apple/mcc.c +++ b/drivers/soc/apple/mcc.c @@ -72,6 +72,24 @@ static bool apple_mcc_plane_ready(struct apple_mcc *mcc, unsigned int plane) return false; } +/* Stop allocating on the first @count planes, then write back and invalidate them. */ +static void apple_mcc_disable_planes(struct apple_mcc *mcc, unsigned int count) +{ + unsigned int i; + u32 val; + + for (i = 0; i < count; i++) { + writel(1, mcc->planes[i] + MCC_ALLOC_DISABLE); + writel(1, mcc->planes[i] + MCC_CACHE_FLUSH); + } + + for (i = 0; i < count; i++) + if (readl_poll_timeout_atomic(mcc->planes[i] + MCC_CACHE_FLUSH, val, + !(val & MCC_CACHE_FLUSH_BUSY), 10, + MCC_FLUSH_TIMEOUT_US)) + dev_err(mcc->dev, "plane %u flush timed out\n", i); +} + static int apple_mcc_enable(struct apple_mcc *mcc) { unsigned int i; @@ -84,6 +102,12 @@ static int apple_mcc_enable(struct apple_mcc *mcc) writel(0, mcc->planes[i] + MCC_ALLOC_DISABLE); if (readl(mcc->planes[i] + MCC_ALLOC_DISABLE)) { dev_err(mcc->dev, "plane %u did not enable allocation\n", i); + /* + * The driver will not bind, so nothing disables these + * planes on shutdown: disable the ones enabled so far, + * and this one, whose state is unknown, now. + */ + apple_mcc_disable_planes(mcc, i + 1); return -EIO; } } @@ -91,26 +115,12 @@ static int apple_mcc_enable(struct apple_mcc *mcc) return 0; } -/* Stop allocating, then write back and invalidate the cache. */ static void apple_mcc_quiesce(struct apple_mcc *mcc) { - unsigned int i; - u32 val; - if (!mcc->enabled) return; - for (i = 0; i < mcc->nr_planes; i++) { - writel(1, mcc->planes[i] + MCC_ALLOC_DISABLE); - writel(1, mcc->planes[i] + MCC_CACHE_FLUSH); - } - - for (i = 0; i < mcc->nr_planes; i++) - if (readl_poll_timeout_atomic(mcc->planes[i] + MCC_CACHE_FLUSH, val, - !(val & MCC_CACHE_FLUSH_BUSY), 10, - MCC_FLUSH_TIMEOUT_US)) - dev_err(mcc->dev, "plane %u flush timed out\n", i); - + apple_mcc_disable_planes(mcc, mcc->nr_planes); mcc->enabled = false; } diff --git a/drivers/soc/apple/pm.rs b/drivers/soc/apple/pm.rs index c2f59228a76e04..a517e6e448732b 100644 --- a/drivers/soc/apple/pm.rs +++ b/drivers/soc/apple/pm.rs @@ -21,7 +21,7 @@ use crate::SepData; static SEP: AtomicPtr = AtomicPtr::new(core::ptr::null_mut()); extern "C" { - fn sep_pm_register(event: unsafe extern "C" fn(bool)) -> c_int; + fn sep_pm_register(event: unsafe extern "C" fn(bool) -> c_int) -> c_int; fn sep_pm_unregister(); fn sep_pm_keep_domains(dev: *mut c_void) -> c_int; } @@ -41,20 +41,25 @@ pub(crate) fn keep_domains(dev: &kernel::device::Device) -> Result<()> { /// # Safety /// Called by `pm_shim.c` from the PM notifier chain, in process context. -unsafe extern "C" fn pm_event(entering: bool) { +unsafe extern "C" fn pm_event(entering: bool) -> c_int { let ptr = SEP.load(Ordering::Acquire); if ptr.is_null() { - return; + return 0; } // SAFETY: produced by `into_foreign` in `register`, and reclaimed by // `unregister` only after `sep_pm_unregister` has waited out any running // notifier call, so it is live for this borrow. let sep = unsafe { as ForeignOwnable>::borrow(ptr) }; if entering { - sep.sleep_prepare(); + if let Err(error) = sep.sleep_prepare() { + // A failing notifier is not included in the PM chain's rollback. + sep.sleep_finished(); + return error.to_errno(); + } } else { sep.sleep_finished(); } + 0 } pub(crate) fn register(sep: Arc) -> Result<()> { diff --git a/drivers/soc/apple/pm_shim.c b/drivers/soc/apple/pm_shim.c index 884ecbb354a062..2ef96c5fd96ebe 100644 --- a/drivers/soc/apple/pm_shim.c +++ b/drivers/soc/apple/pm_shim.c @@ -14,12 +14,12 @@ #include "shim.h" -static void (*sep_pm_event)(bool entering); +static int (*sep_pm_event)(bool entering); static int sep_pm_notify(struct notifier_block *nb, unsigned long action, void *data) { - void (*event)(bool entering) = READ_ONCE(sep_pm_event); + int (*event)(bool entering) = READ_ONCE(sep_pm_event); if (!event) return NOTIFY_DONE; @@ -28,8 +28,7 @@ static int sep_pm_notify(struct notifier_block *nb, unsigned long action, case PM_SUSPEND_PREPARE: case PM_HIBERNATION_PREPARE: case PM_RESTORE_PREPARE: - event(true); - return NOTIFY_OK; + return notifier_from_errno(event(true)); case PM_POST_SUSPEND: case PM_POST_HIBERNATION: case PM_POST_RESTORE: @@ -44,7 +43,7 @@ static struct notifier_block sep_pm_nb = { .notifier_call = sep_pm_notify, }; -int sep_pm_register(void (*event)(bool entering)) +int sep_pm_register(int (*event)(bool entering)) { int ret; diff --git a/drivers/soc/apple/pmp.rs b/drivers/soc/apple/pmp.rs index be248317cdc783..0533d7a3b5acfe 100644 --- a/drivers/soc/apple/pmp.rs +++ b/drivers/soc/apple/pmp.rs @@ -130,6 +130,9 @@ impl PmpState { #[pin_data] struct PmpData { dev: ARef, + /// T8140 firmware is not sent the UPDATE_IOREG confirmation, and its + /// messages are not taken as device-power replies. + t8140: bool, pmp_mmio: Pin>>>, asc_mmio: Pin>>>, #[pin] @@ -143,7 +146,7 @@ struct PmpData { } impl PmpData { - fn new(dev: &platform::Device) -> Result> { + fn new(dev: &platform::Device, t8140: bool) -> Result> { let pmp_req = dev.io_request_by_name(c"pmp").ok_or(EINVAL)?; let pmp_mmio = KBox::pin_init(pmp_req.iomap_sized::(), GFP_KERNEL)?; let asc_req = dev.io_request_by_name(c"asc").ok_or(EINVAL)?; @@ -152,6 +155,7 @@ impl PmpData { try_pin_init!( PmpData { dev: dev.as_ref().into(), + t8140, pmp_mmio, asc_mmio, rtkit <- new_mutex!(None), @@ -339,7 +343,9 @@ impl PmpData { } size = len as u32; } else { - dev_info!(self.dev, "unknown property {:?}", name_str); + // Not every boot loader provides every tunable the firmware + // asks for, and the firmware starts without them. + dev_dbg!(self.dev, "unknown property {:?}", name_str); } } state.ioreg_entries.push(size, GFP_KERNEL)?; @@ -359,7 +365,8 @@ impl PmpData { } fn recv_message(&self, msg: u64) -> Result<()> { let pm_command = (msg >> PM_COMMAND_SHIFT) & 0xff; - if msg & PM_COMMAND_MARKER != 0 && (pm_command == 0x0f || pm_command == 0x10) { + if !self.t8140 && msg & PM_COMMAND_MARKER != 0 && (pm_command == 0x0f || pm_command == 0x10) + { let mut state = self.power_command.lock(); state.reply(msg); if *state == Command::Failed { @@ -378,7 +385,7 @@ impl PmpData { OPC_SET_IOREG => self.set_ioreg(msg & SET_IOREG_INDEX_MASK)?, // Firmware waits for this confirmation before publishing reports. // Do not echo the registry index/payload or add a generic ACK bit. - OPC_UPDATE_IOREG => OPC_CONFIRM_IOREG << OPC_SHIFT, + OPC_UPDATE_IOREG if !self.t8140 => OPC_CONFIRM_IOREG << OPC_SHIFT, _ => { dev_err!(self.dev, "Got unknown message {}", msg); return Err(EIO); @@ -508,11 +515,12 @@ impl platform::Driver for PmpDriver { fn probe(pdev: &platform::Device, info: Option<&bool>) -> impl PinInit { let dev: ARef = pdev.as_ref().into(); - if info.copied().unwrap_or(false) { + let t8140 = info.copied().unwrap_or(false); + if t8140 { // SAFETY: The T8140 DART accepts 42-bit IOVAs; no DMA allocations exist yet. unsafe { pdev.dma_set_mask_and_coherent(DmaMask::new::<42>())? }; } - let data = PmpData::new(pdev)?; + let data = PmpData::new(pdev, t8140)?; let node = dev.fwnode().ok_or(EIO)?; let dvid = node .property_read(c"apple,dram-vendor-id") diff --git a/drivers/soc/apple/profile.rs b/drivers/soc/apple/profile.rs index 646f7862e41040..af803b13b7ca3b 100644 --- a/drivers/soc/apple/profile.rs +++ b/drivers/soc/apple/profile.rs @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT // Copyright 2026 Dj +// Copyright 2026 Ryan Murray //! Typed platform profiles for the SEP driver. //! @@ -146,6 +147,8 @@ pub(crate) struct PlatformProfile { pub(crate) firmware_region: &'static CStr, /// Identity keybag CREATE_KEYBAG field encoding (per-SoC; see [`KeybagCreate`]). pub(crate) keybag_create: KeybagCreate, + /// Require nonce authentication and signed sensor material before capture. + pub(crate) nonce_session: bool, /// Key-store protocol generation (see [`KeyStore`]). pub(crate) key_store: KeyStore, /// Platforms with reboot-tested owner export and user-before-master saves. @@ -183,6 +186,7 @@ const T8103: PlatformProfile = PlatformProfile { cpx_encryption_mode: 2, }, persistent_enrol: true, + nonce_session: false, }; /// M1 Pro (J316s). The M1 family cold-boots its SEP like T8103, so the boot @@ -219,6 +223,7 @@ const T6000: PlatformProfile = PlatformProfile { cpx_encryption_mode: 2, }, persistent_enrol: true, + nonce_session: false, }; const T6020: PlatformProfile = PlatformProfile { @@ -248,6 +253,7 @@ const T6020: PlatformProfile = PlatformProfile { }, key_store: KeyStore::Variant5, persistent_enrol: true, + nonce_session: false, }; /// M2 Pro/Max on system firmware before 26.3. Their enclave refuses the @@ -299,6 +305,7 @@ const T8140: PlatformProfile = PlatformProfile { bag_type: 0, arg: -1, }, + nonce_session: true, }; /// MacBook Air M2 (J413, J415). iBoot leaves the SEP running and the Mesa @@ -340,6 +347,7 @@ const T8112: PlatformProfile = PlatformProfile { }, key_store: KeyStore::Variant5, persistent_enrol: true, + nonce_session: false, }; /// MacBook Air M3 (J613, J615). The J613 and J615 ADTs (26.6.2, 25G83) are @@ -385,7 +393,7 @@ static_assert!(T8122.wide_dma_mask); /// /// The root property is a NUL-separated list, so whole entries are compared /// instead of substrings: `apple,t8103` cannot match a longer unrelated value. -fn machine_has(compatible: &[u8]) -> bool { +pub(crate) fn machine_has(compatible: &[u8]) -> bool { let Some(root) = of::root() else { return false; }; diff --git a/drivers/soc/apple/registration.rs b/drivers/soc/apple/registration.rs new file mode 100644 index 00000000000000..95edd6a0b65b59 --- /dev/null +++ b/drivers/soc/apple/registration.rs @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +// Copyright 2026 Ryan Murray + +//! What a Touch ID sensor registration has to send to the enclave. +//! +//! A registration identifies the sensor (REGISTER_SENSOR), registers its +//! serial and, on profiles with a nonce session, authenticates it. Any of +//! those steps can fail after the enclave has already accepted an earlier +//! one, and then the enclave holds a partial registration. On boot +//! d1962b3a every retry that registered the sensor again without clearing +//! it failed, 42 of them over 800 s. Re-registration that identifies the +//! sensor and then clears the enclave's sensor state, the order every +//! verify uses, has not failed on an idle machine. So the next attempt +//! clears whenever the enclave may hold any registration, complete or not. + +/// Nothing is registered with the enclave yet. +pub(crate) const FRESH: u32 = 0; +/// The sensor is identified; profiles without a nonce session register the +/// serial next. +pub(crate) const IDENTIFIED: u32 = 1; +/// The registration completed. +pub(crate) const ESTABLISHED: u32 = 2; + +/// The enclave requests one registration attempt makes, in order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Plan { + /// PLATFORM_IDENTITY and REGISTER_SENSOR, before anything else. A + /// nonce-session profile identifies the sensor on every reset. + pub(crate) identify_first: bool, + /// CLEAR_STATE, after any identification. + pub(crate) clear_state: bool, + /// REGISTER_SENSOR after the clear, on a profile without a nonce session. + pub(crate) register_sensor: bool, +} + +/// Plans one registration attempt. +/// +/// `stage` is the bring-up stage and `holds_sensor` whether an +/// identification of the sensor was submitted to the enclave since it +/// booted, answered or not: a lost reply does not mean the enclave did not +/// act on it. `sepos13` says whether the profile starts from the serial, as +/// the 13.5 key store does. +pub(crate) fn plan(stage: u32, holds_sensor: bool, nonce_session: bool, sepos13: bool) -> Plan { + Plan { + identify_first: nonce_session, + clear_state: stage >= ESTABLISHED || (nonce_session && holds_sensor), + register_sensor: stage == FRESH && !nonce_session && !sepos13, + } +} + +#[kernel::macros::kunit_tests(apple_sep_registration)] +mod tests { + use super::*; + + #[test] + fn first_registration_after_boot_does_not_clear() { + assert_eq!( + plan(FRESH, false, true, false), + Plan { + identify_first: true, + clear_state: false, + register_sensor: false + } + ); + } + + #[test] + fn retry_after_a_partial_registration_clears() { + // REGISTER_SENSOR was accepted, then the serial, nonce or session + // step failed: the bring-up stage is still FRESH. + let retry = plan(FRESH, true, true, false); + assert!(retry.identify_first); + assert!(retry.clear_state); + } + + #[test] + fn re_registration_after_a_complete_one_clears() { + assert!(plan(ESTABLISHED, true, true, false).clear_state); + assert!(plan(ESTABLISHED, false, false, false).clear_state); + } + + #[test] + fn a_fresh_restart_still_clears_what_the_enclave_holds() { + // Restarting the bring-up from FRESH does not erase the enclave's + // registration; only CLEAR_STATE does. + assert!(plan(FRESH, true, true, false).clear_state); + } + + #[test] + fn profiles_without_a_nonce_session_are_unchanged() { + assert_eq!( + plan(FRESH, true, false, false), + Plan { + identify_first: false, + clear_state: false, + register_sensor: true + } + ); + assert_eq!( + plan(IDENTIFIED, true, false, false), + Plan { + identify_first: false, + clear_state: false, + register_sensor: false + } + ); + assert_eq!( + plan(FRESH, false, false, true), + Plan { + identify_first: false, + clear_state: false, + register_sensor: false + } + ); + } +} diff --git a/drivers/soc/apple/rtkit-helper.c b/drivers/soc/apple/rtkit-helper.c index ef0400e07d49cd..313281372a5f82 100644 --- a/drivers/soc/apple/rtkit-helper.c +++ b/drivers/soc/apple/rtkit-helper.c @@ -12,13 +12,19 @@ #include #include #include +#include #define APPLE_ASC_CPU_CONTROL 0x44 #define APPLE_ASC_CPU_CONTROL_RUN BIT(4) +/* A running T8140 MTP session described by the boot stage that started it. */ +#define APPLE_RTKIT_HELPER_HANDOFF "mtp-live-v1" + struct apple_rtkit_helper { struct device *dev; struct apple_rtkit *rtk; + /* Inherited from the boot stage; it cannot be restarted. */ + bool adopted; void __iomem *asc_base; @@ -73,6 +79,39 @@ static const struct apple_rtkit_ops apple_rtkit_helper_ops = { .shmem_destroy = apple_rtkit_helper_shmem_destroy, }; +static void apple_rtkit_helper_release(void *rtk) +{ + apple_rtkit_free_retaining_buffers(rtk); +} + +/* + * T8140 MTP firmware cannot be restarted once it has been taken to QUIESCED + * or SLEEP, so a boot stage that used it hands the session over running. It + * reserves the buffers the firmware holds and maps them 1:1 through the + * DART. Only adopt a session it described whose CPU still runs. + */ +static bool apple_rtkit_helper_live(struct apple_rtkit_helper *helper) +{ + const char *handoff; + + if (of_property_read_string(helper->dev->of_node, "apple,rtkit-handoff", + &handoff)) + return false; + + if (strcmp(handoff, APPLE_RTKIT_HELPER_HANDOFF)) { + dev_warn(helper->dev, "unknown RTKit handoff \"%s\"\n", handoff); + return false; + } + + if (!(readl_relaxed(helper->asc_base + APPLE_ASC_CPU_CONTROL) & + APPLE_ASC_CPU_CONTROL_RUN)) { + dev_warn(helper->dev, "RTKit handoff without a running CPU\n"); + return false; + } + + return true; +} + static int apple_rtkit_helper_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; @@ -103,6 +142,23 @@ static int apple_rtkit_helper_probe(struct platform_device *pdev) "Failed to map SRAM region"); } + if (apple_rtkit_helper_live(helper)) { + helper->rtk = apple_rtkit_init_adopted(dev, helper, NULL, 0, + &apple_rtkit_helper_ops); + if (IS_ERR(helper->rtk)) + return dev_err_probe(dev, PTR_ERR(helper->rtk), + "Failed to adopt RTKit"); + + ret = devm_add_action_or_reset(dev, apple_rtkit_helper_release, + helper->rtk); + if (ret) + return ret; + + helper->adopted = true; + dev_info(dev, "adopted the running RTKit session\n"); + return 0; + } + helper->rtk = devm_apple_rtkit_init(dev, helper, NULL, 0, &apple_rtkit_helper_ops); if (IS_ERR(helper->rtk)) @@ -113,7 +169,10 @@ static int apple_rtkit_helper_probe(struct platform_device *pdev) helper->asc_base + APPLE_ASC_CPU_CONTROL); /* Works for both wake and boot */ - ret = apple_rtkit_wake(helper->rtk); + if (of_device_is_compatible(dev->of_node, "apple,t8142-mtp")) + ret = apple_rtkit_wake_early_ap(helper->rtk); + else + ret = apple_rtkit_wake(helper->rtk); if (ret != 0) return dev_err_probe(dev, ret, "Failed to wake up coprocessor"); @@ -124,6 +183,10 @@ static void apple_rtkit_helper_remove(struct platform_device *pdev) { struct apple_rtkit_helper *helper = platform_get_drvdata(pdev); + /* An inherited session would not come back from QUIESCED. */ + if (helper->adopted) + return; + if (apple_rtkit_is_running(helper->rtk)) apple_rtkit_quiesce(helper->rtk); diff --git a/drivers/soc/apple/rtkit.c b/drivers/soc/apple/rtkit.c index 43528fef6d199f..cce23cd167e0bb 100644 --- a/drivers/soc/apple/rtkit.c +++ b/drivers/soc/apple/rtkit.c @@ -754,6 +754,17 @@ static void apple_rtkit_oslog_rx(struct apple_rtkit *rtk, u64 msg) u8 type = FIELD_GET(APPLE_RTKIT_OSLOG_TYPE, msg); int err; + /* + * T8140 and T8142 co-processors send notifications 2 to 5 during + * normal operation; they expect no reply and keep running without one. + */ + if (type >= APPLE_RTKIT_OSLOG_LOG && type <= 5 && + (of_machine_is_compatible("apple,t8140") || + of_machine_is_compatible("apple,t8142"))) { + dev_dbg(rtk->dev, "RTKit: oslog notification: %016llx\n", msg); + return; + } + switch (type) { case APPLE_RTKIT_OSLOG_BUFFER_REQUEST: apple_rtkit_common_rx_get_buffer(rtk, &rtk->oslog_buffer, @@ -983,6 +994,18 @@ void apple_rtkit_flush_rx(struct apple_rtkit *rtk) } EXPORT_SYMBOL_GPL(apple_rtkit_flush_rx); +void apple_rtkit_hold_rx(struct apple_rtkit *rtk) +{ + apple_mbox_hold_rx(rtk->mbox); +} +EXPORT_SYMBOL_GPL(apple_rtkit_hold_rx); + +void apple_rtkit_resume_rx(struct apple_rtkit *rtk) +{ + apple_mbox_release_rx(rtk->mbox); +} +EXPORT_SYMBOL_GPL(apple_rtkit_resume_rx); + bool apple_rtkit_has_endpoint(struct apple_rtkit *rtk, u8 ep) { return test_bit(ep, rtk->endpoints); diff --git a/drivers/soc/apple/sbio.rs b/drivers/soc/apple/sbio.rs index 155381756e8138..1b2f821d6f96a3 100644 --- a/drivers/soc/apple/sbio.rs +++ b/drivers/soc/apple/sbio.rs @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT // Copyright 2026 Dj +// Copyright 2026 Ryan Murray //! Touch ID: the biometric endpoint (SBIO) and its `/dev/sep-bio` interface. //! The enclave matches; no biometric image ever crosses to userspace. @@ -263,7 +264,9 @@ impl SepData { let filed = { let mut session = self.bio_session.lock(); let mut index = self.bio_index.lock(); - bio::enrol_finish(&mut session, &mut index, outcome) + bio::enrol_finish( + &mut session, self.capture_epoch.load(Relaxed), &mut index, outcome, + ) }; if !filed { return; @@ -301,14 +304,24 @@ impl SepData { return; } - if let Err(e) = dt::enable_spi_sensor(sensor::CONTROLLER_BASE, sensor::CHIP_SELECT) { - dev_warn!( - self.dev, - "sensor: could not enable the SPI bus at 0x{:x} or create the sensor node ({:?}); no sensor\n", - sensor::CONTROLLER_BASE, - e - ); - return; + match dt::enable_spi_sensor(sensor::CONTROLLER_BASE, sensor::CHIP_SELECT) { + Ok(true) => {}, + Ok(false) => { + dev_info!( + self.dev, + "sensor: SPI controller withheld by the J700 firmware handoff; Touch ID unavailable\n" + ); + return; + } + Err(e) => { + dev_warn!( + self.dev, + "sensor: could not enable the SPI bus at 0x{:x} or create the sensor node ({:?}); no sensor\n", + sensor::CONTROLLER_BASE, + e + ); + return; + } } let bound = sensor::is_bound(); @@ -1033,16 +1046,31 @@ impl SepData { self.register_ool(&self.ool_sbio)?; self.sbio_ready.store(true, Relaxed); - if matches!(self.profile.key_store, profile::KeyStore::Sepos13 { .. }) { - // 22G74 AppleMesaSEPDriver::initSbioCommunication sends 0x73 - // with u32(1) before its first ordinary SBIO transaction. - // Without it, J313 can complete sensor setup and authenticate - // its SCRD context but rejects BEGIN_ENROL. Buffer readiness is - // required by this transaction; revoke it if initialization fails. - let result = self.sbio_transfer_raw(0x73, c"INIT_SBIO_COMMUNICATION", &1u32.to_le_bytes()); + // 22G74 AppleMesaSEPDriver::initSbioCommunication sends 0x73 + // with u32(1) before its first ordinary SBIO transaction. + // Without it, J313 can complete sensor setup and authenticate + // its SCRD context but rejects BEGIN_ENROL. Buffer readiness is + // required by this transaction; revoke it if initialization fails. + // The J700 nonce-session protocol sends the same request with u32(6). + let init_word = if self.profile.nonce_session { + Some(6u32) + } else if matches!(self.profile.key_store, profile::KeyStore::Sepos13 { .. }) { + Some(1u32) + } else { + None + }; + if let Some(word) = init_word { + let result = self.sbio_transfer_raw( + 0x73, c"INIT_SBIO_COMMUNICATION", &word.to_le_bytes(), + ); match result { - Ok(done) if done.status.is_ok() && done.payload.is_empty() => { - dev_info!(self.dev, "sbio: native 13.5 communication initialization accepted\n"); + Ok(done) if done.status.is_ok() + && (self.profile.nonce_session || done.payload.is_empty()) => { + if self.profile.nonce_session { + dev_info!(self.dev, "sbio: communication initialization accepted\n"); + } else { + dev_info!(self.dev, "sbio: native 13.5 communication initialization accepted\n"); + } } Ok(done) => { self.sbio_ready.store(false, Relaxed); @@ -1061,6 +1089,31 @@ impl SepData { } fn attach_bringup(&self) { + if self.profile.nonce_session { + // A bring-up that fails on the bus leaves nothing a new attempt + // cannot undo: each attempt power-cycles the sensor and, after a + // failed one, clears the enclave's partial registration first, + // as every verify does. + const ATTEMPTS: u32 = 3; + for attempt in 1..=ATTEMPTS { + if self.bring_sensor_online() { + return; + } + if attempt < ATTEMPTS { + dev_warn!( + self.dev, + "sensor: attach-time bring-up attempt {} of {} failed; starting again\n", + attempt, + ATTEMPTS + ); + } + } + dev_warn!( + self.dev, + "sensor: authenticated attach incomplete; capture unavailable\n" + ); + return; + } if !sensor::is_bound() { dev_warn!( self.dev, @@ -1094,20 +1147,36 @@ impl SepData { return false; }; + // Newer sensors require the encrypted parameters and transparent + // channel before the calibration challenge can be serviced. + let parameters = if self.profile.nonce_session { + match self.apply_sensor_parameters() { + Some(parameters) => Some(parameters), + None => return false, + } + } else { + None + }; + if !self.sensor_calibrated.load(Relaxed) { if !self.calibrate_sensor() { diag::sensor_brought_up(bind, false); return false; } self.sensor_calibrated.store(true, Relaxed); - let Some(reloaded_patch) = self.wake_sensor() else { - diag::sensor_brought_up(bind, false); - return false; - }; - patch = reloaded_patch; + if parameters.is_none() { + let Some(reloaded_patch) = self.wake_sensor() else { + diag::sensor_brought_up(bind, false); + return false; + }; + patch = reloaded_patch; + } } - let ok = self.complete_bringup(patch); + let ok = match parameters { + Some(parameters) => self.finish_sensor_setup(patch, parameters), + None => self.complete_bringup(patch), + }; diag::sensor_brought_up(bind, ok); // The sensor is patched and idle here -- the only safe moment to // configure the data-ready interrupt, which is then left alone for the @@ -1431,7 +1500,9 @@ impl SepData { } CaptureWait::Abandon => { // A cancelled verify also abandons; only a live one is a fault. - if bio::verify_is_live(&self.bio_session.lock()) { + if bio::verify_is_live( + &self.bio_session.lock(), self.capture_epoch.load(Relaxed), + ) { dev_err!( self.dev, "verify: the sensor status became unreadable during the capture\n" @@ -1580,7 +1651,7 @@ impl SepData { let mut last_percent: u32 = 0; let outcome = loop { - if !bio::enrol_is_live(&self.bio_session.lock()) { + if !bio::enrol_is_live(&self.bio_session.lock(), self.capture_epoch.load(Relaxed)) { break None; } if counter >= ENROL_MAX_CAPTURES { @@ -1607,6 +1678,7 @@ impl SepData { last_percent = last_percent.max(percent); let woke = bio::enrol_advance( &mut self.bio_session.lock(), + self.capture_epoch.load(Relaxed), stage, percent, bio::Guidance::LiftAndMove, @@ -1635,7 +1707,10 @@ impl SepData { // The capture did not take (partial/unusable contact). Nudge // the person to hold still and press again rather than // silently re-capturing with no feedback. - if bio::capture_guide(&mut self.bio_session.lock(), bio::Guidance::HoldStill) { + if bio::capture_guide( + &mut self.bio_session.lock(), self.capture_epoch.load(Relaxed), + bio::Guidance::HoldStill, + ) { self.bio_wake(); } } @@ -1691,14 +1766,123 @@ impl SepData { } } + fn sensor_ms(&self) -> u64 { + shim::boottime_ns().saturating_sub(self.sensor_power_ns.load(Relaxed)) / 1_000_000 + } + + fn sensor_trace(&self, step: &CStr) { + if *module_parameters::sensor_trace.value() != 0 { + dev_info!(self.dev, "sensor: {} done at {} ms\n", step, self.sensor_ms()); + } + } + + /// Reports frames the sensor exchange had to read again or discard. + fn note_sensor_recoveries(&self) { + let (reread, drained) = sensor::take_recoveries(); + if reread != 0 || drained != 0 { + dev_warn!( + self.dev, + "sensor: re-read {} frame(s) the sensor still offered after a garbled read, discarded {} stale frame(s)\n", + reread, + drained + ); + } + } + + /// Reports a failed registration step with what the sensor shows + /// afterwards: its state, patch acknowledgment, whether it presents an + /// identifier and the length of any frame it holds. Never the + /// identifier, the serial or any key material. + fn sensor_step_failed(&self, step: &CStr) { + self.note_sensor_recoveries(); + let ms = self.sensor_ms(); + match sensor::status() { + Ok(st) => dev_warn!( + self.dev, + "sensor: {} failed {} ms after power-on; sensor now state {} ack {:#04x} identifier {} holding {} bytes{}\n", + step, + ms, + st.state, + st.patch_ack(), + st.identifier_to_register().is_some(), + st.available().unwrap_or(0), + if st.is_silent() { c" (silent)" } else { c"" } + ), + Err(e) => dev_warn!( + self.dev, + "sensor: {} failed {} ms after power-on; its status read failed too ({:?})\n", + step, + ms, + e + ), + } + } + + /// One enclave request of the registration. A refusal is named. + fn sensor_enclave_step(&self, op: &crate::sbio::SbioOp) -> Option> { + match self.sbio_transfer(op) { + Ok(done) if done.status.is_ok() => { + self.sensor_trace(op.name()); + Some(done.payload) + } + Ok(done) => { + dev_warn!(self.dev, "sensor: {} refused with status {}\n", op.name(), done.status); + self.sensor_step_failed(op.name()); + None + } + Err(e) => { + dev_warn!(self.dev, "sensor: {} transfer failed: {:?}\n", op.name(), e); + self.sensor_step_failed(op.name()); + None + } + } + } + fn register_sensor(&self, id: &sensor::Identifier) -> bool { let stage = self.bringup.load(Relaxed); - dev_info!(self.dev, "sbio: registering sensor at bring-up stage {}\n", stage); + let plan = registration::plan( + stage, + self.sensor_held.load(Relaxed), + self.profile.nonce_session, + matches!(self.profile.key_store, profile::KeyStore::Sepos13 { .. }), + ); + dev_info!( + self.dev, + "sbio: registering sensor at bring-up stage {}{}\n", + stage, + if plan.clear_state && stage < BRINGUP_ESTABLISHED { + c", clearing the enclave's partial registration first" + } else { + c"" + } + ); + + if plan.identify_first { + // J700 needs identification before CLEAR_STATE on every reset. + let platform = match Self::sensor_platform_identity() { + Ok(platform) => platform, + Err(error) => { + dev_err!(self.dev, "sensor: platform identity unavailable: {:?}\n", error); + return false; + } + }; + if !self.sensor_session_request(0x48, c"PLATFORM_IDENTITY", &platform) { + self.sensor_step_failed(c"PLATFORM_IDENTITY"); + return false; + } + self.sensor_trace(c"PLATFORM_IDENTITY"); + // Counted as held from submission on: a reply that is lost or + // late does not mean the enclave did not act on the request. + self.sensor_held.store(true, Relaxed); + if self.sensor_enclave_step(&crate::sbio::sbio_register_sensor(id)).is_none() { + return false; + } + } - if stage >= BRINGUP_ESTABLISHED { + if plan.clear_state { let _ = self.log_identity_count(c"before CLEAR_STATE for sensor re-registration"); if self - .sbio_expect_ok(&crate::sbio::sbio_clear_state()) + .sensor_enclave_step(&crate::sbio::sbio_clear_state()) .is_none() { return false; @@ -1712,7 +1896,7 @@ impl SepData { // The 13.5 J313 flow begins with REGISTER_SENSOR_SERIAL (0x17). // The newer REGISTER_SENSOR (0x80) returned 0x16 on this firmware. self.bringup.store(BRINGUP_IDENTIFIED, Relaxed); - } else if stage == BRINGUP_FRESH { + } else if plan.register_sensor { let op = crate::sbio::sbio_register_sensor(id); match self.sbio_call(&op) { SbioOutcome::Ok(_payload) => { @@ -1726,25 +1910,107 @@ impl SepData { let serial = match sensor::read_sensor_serial() { Ok(serial) => serial, - Err(_) => { + Err(e) => { + dev_warn!(self.dev, "sensor: serial read failed: {:?}\n", e); + self.sensor_step_failed(c"serial read"); return false; } }; + self.sensor_trace(c"serial read"); let op = crate::sbio::sbio_register_sensor_serial(&serial); - match self.sbio_call(&op) { - SbioOutcome::Ok(_payload) => { - self.bringup.store(BRINGUP_ESTABLISHED, Relaxed); - let _ = self.log_identity_count(c"after sensor serial registration"); - true + if self.sensor_enclave_step(&op).is_none() { + return false; + } + if self.profile.nonce_session && !self.authenticate_sensor_session() { + return false; + } + self.bringup.store(BRINGUP_ESTABLISHED, Relaxed); + self.note_sensor_recoveries(); + let _ = self.log_identity_count(c"after sensor serial registration"); + true + } + + fn sensor_platform_identity() -> Result<[u8; 12]> { + let node = kernel::of::root().ok_or(ENODEV)?; + let property: KVec = node.get_property(c"serial-number")?; + let length = property.iter().position(|byte| *byte == 0).ok_or(EINVAL)?; + let mut identity = [0; 12]; + if length == 0 || length > identity.len() || property.len() != length + 1 { + return Err(EINVAL); + } + identity[..length].copy_from_slice(&property[..length]); + Ok(identity) + } + + fn sensor_session_request(&self, opcode: u16, name: &'static CStr, bytes: &[u8]) -> bool { + match self.sbio_transfer_raw(opcode, name, bytes) { + Ok(reply) if reply.status.is_ok() => true, + Ok(reply) => { + dev_err!(self.dev, "sensor: {} refused with status {}\n", name, reply.status); + false + } + Err(error) => { + dev_err!(self.dev, "sensor: {} transfer failed: {:?}\n", name, error); + false } - _ => false, } } + fn authenticate_sensor_session(&self) -> bool { + let nonce = match sensor::session_nonce() { + Ok(nonce) => nonce, + Err(error) => { + dev_err!(self.dev, "sensor: nonce read failed: {:?}\n", error); + self.sensor_step_failed(c"nonce read"); + return false; + } + }; + self.sensor_trace(c"nonce read"); + if !self.sensor_session_request(0x81, c"SENSOR_NONCE", &nonce) { + self.sensor_step_failed(c"SENSOR_NONCE"); + return false; + } + self.sensor_trace(c"SENSOR_NONCE"); + + if !self.sensor_key_loaded.load(Relaxed) { + let firmware = match kernel::firmware::Firmware::request( + c"apple/msrk-j700.bin", &self.dev, + ) { + Ok(firmware) => firmware, + Err(error) => { + dev_err!( + self.dev, + "sensor: signed root-key container unavailable: {:?}\n", + error + ); + return false; + } + }; + if firmware.data().is_empty() + || !self.sensor_session_request(0x86, c"SENSOR_ROOT_KEY", firmware.data()) + { + self.sensor_step_failed(c"SENSOR_ROOT_KEY"); + return false; + } + self.sensor_key_loaded.store(true, Relaxed); + self.sensor_trace(c"SENSOR_ROOT_KEY"); + } + // Each identification starts a new sensor-side exchange, even though + // the signed key remains loaded for the lifetime of the enclave. + self.establish_session() + } + fn wake_sensor(&self) -> Option { let _source = sensor::power_source(); + self.sensor_power_ns.store(shim::boottime_ns(), Relaxed); let cycled = sensor::power_cycle(); + if !cycled && self.profile.nonce_session { + // A sensor that was not reset may still hold the session of an + // earlier attempt; registering it again cannot be trusted. + dev_err!(self.dev, "sensor: the power cycle failed; not registering the sensor\n"); + return None; + } if !sensor::cs_timing().is_hardware() { dev_warn!( self.dev, @@ -1784,11 +2050,36 @@ impl SepData { answered_without_identifier = true; continue; }; + if *module_parameters::sensor_trace.value() != 0 { + dev_info!( + self.dev, + "sensor: identifier present {} ms after power-on\n", + self.sensor_ms() + ); + } if !self.register_sensor(&id) { dev_warn!(self.dev, "sensor: the enclave did not register the sensor\n"); return None; } + if self.profile.nonce_session { + // Registration can reset the sensor session. Do not reuse the + // acknowledgement sampled before CLEAR_STATE and authentication. + let st = match sensor::status() { + Ok(st) if !st.is_silent() => st, + _ => { + dev_warn!(self.dev, "sensor: status unavailable after registration\n"); + return None; + } + }; + if st.patch_ack() == sensor::PATCH_ACCEPTED + && st.state != sensor::STATE_NEEDS_PATCH + { + return Some(PatchLoaded(())); + } + return self.apply_sensor_patch(st.patch_ack()); + } + // The patch decision is made on status[8], not on state 9. if st.patch_ack() == sensor::PATCH_ACCEPTED { return Some(PatchLoaded(())); @@ -1853,13 +2144,20 @@ impl SepData { return false; }; - let request = match crate::sbio::SbioCalibration::new(&blob) { + let source = if self.profile.nonce_session { 6 } else { SBIO_CALIBRATION_SOURCE }; + let request = match crate::sbio::SbioCalibration::new(&blob, source) { Ok(request) => request, Err(_) => { return false; } }; - if self + if self.profile.nonce_session { + if !self.sensor_session_request(request.opcode(), request.name(), request.payload()) { + dev_err!(self.dev, "sensor: calibration refused (source {}, {} bytes)\n", + source, request.payload().len()); + return false; + } + } else if self .sbio_transfer_raw(request.opcode(), request.name(), request.payload()) .ok() .filter(|done| done.status.is_ok()) @@ -1882,6 +2180,10 @@ impl SepData { return false; }; + self.finish_sensor_setup(patch, params) + } + + fn finish_sensor_setup(&self, patch: PatchLoaded, params: ParametersApplied) -> bool { let op = crate::sbio::sbio_complete_init(patch, params); if self.sbio_expect_ok(&op).is_none() { dev_err!( @@ -1989,43 +2291,37 @@ impl SepData { } fn establish_session(&self) -> bool { - let share = match self.sbio_call(&crate::sbio::sbio_request_session_share()) { - SbioOutcome::Ok(sh) => sh, - SbioOutcome::Status16 => { - return false; - } - SbioOutcome::PrerequisiteMissing => { - return false; - } - SbioOutcome::Other => { - return false; - } + let Some(share) = self.sensor_enclave_step(&crate::sbio::sbio_request_session_share()) + else { + return false; }; // 0x30 reserved for this reply; length is a floor, take the first 40 if share.len() < sensor::SESSION_SHARE_LEN { + dev_warn!(self.dev, "sensor: short session share ({} bytes)\n", share.len()); return false; } let mut out = [0u8; sensor::SESSION_SHARE_LEN]; out.copy_from_slice(&share[..sensor::SESSION_SHARE_LEN]); // relay it: class 0x72, no CRC, no padding - if sensor::send_session_share(&out).is_err() { + if let Err(e) = sensor::send_session_share(&out) { + dev_warn!(self.dev, "sensor: sending the session share failed: {:?}\n", e); + self.sensor_step_failed(c"session share send"); return false; } let reply = match sensor::read_session_reply() { Ok(r) => r, - Err(_) => { + Err(e) => { + dev_warn!(self.dev, "sensor: session reply read failed: {:?}\n", e); + self.sensor_step_failed(c"session reply read"); return false; } }; - match self.sbio_transfer(&crate::sbio::sbio_commit_session_share(&reply)) { - Ok(done) if done.status.is_ok() => true, - Ok(_) => false, - Err(_) => false, - } + self.sensor_enclave_step(&crate::sbio::sbio_commit_session_share(&reply)) + .is_some() } // Each failure names its step: a report that only says "sequence counter @@ -2079,6 +2375,7 @@ impl SepData { Ok(done) if done.status.is_ok() => true, Ok(done) => { dev_warn!(self.dev, "sequence counter: the enclave refused the sensor's reply, status {}\n", done.status); + self.sensor_step_failed(c"challenge commit"); false } Err(e) => { @@ -2217,7 +2514,7 @@ impl SepData { } CaptureWait::Abandon => { // A cancelled enrolment also abandons; only a live one is a fault. - if bio::enrol_is_live(&self.bio_session.lock()) { + if bio::enrol_is_live(&self.bio_session.lock(), self.capture_epoch.load(Relaxed)) { dev_err!( self.dev, "enrol: the sensor status became unreadable during capture {}\n", @@ -2358,7 +2655,7 @@ impl SepData { ); let mut attempt: u32 = 0; loop { - if !bio::capture_is_live(&self.bio_session.lock()) { + if !bio::capture_is_live(&self.bio_session.lock(), self.capture_epoch.load(Relaxed)) { return CaptureWait::Abandon; } @@ -2383,7 +2680,9 @@ impl SepData { _ => None, }; if let Some(guide) = guide { - if bio::capture_guide(&mut self.bio_session.lock(), guide) { + if bio::capture_guide( + &mut self.bio_session.lock(), self.capture_epoch.load(Relaxed), guide, + ) { self.bio_wake(); } } @@ -2425,6 +2724,9 @@ impl SepData { if !self.sbio_ready.load(Relaxed) { return; } + // Before reading the reply: a requester that times out meanwhile and + // begins a new exchange must not receive it. + let generation = self.sbio_rx.lock().generation(); // notification and the enclave's DMA are not ordered; buffer may still be poison if !self.ool_await_written(&self.ool_sbio, 0, transfer::HEADER_LEN) { @@ -2463,7 +2765,7 @@ impl SepData { } }; - let progress = self.sbio_rx.lock().on_chunk(marker, &packet, &payload); + let progress = self.sbio_rx.lock().on_chunk(generation, marker, &packet, &payload); match progress { transfer::Progress::NeedMore(cont) => { @@ -2521,7 +2823,12 @@ impl SepData { name: &'static CStr, payload: &[u8], ) -> Result { - if !self.sbio_ready.load(Relaxed) { + // The capture work item and the ioctl path both reach this endpoint. + // Without this, the second caller is refused with EBUSY, or clears a + // reply the first has not collected yet and takes the next one. Taken + // before the shutdown check, so a waiter never starts after remove(). + let _exchange = self.sbio_exchange_lock.lock(); + if self.shutting_down.load(Relaxed) || !self.sbio_ready.load(Relaxed) { return Err(ENODEV); } @@ -2537,6 +2844,11 @@ impl SepData { let mut remaining = time::msecs_to_jiffies(SBIO_TIMEOUT_MS); let mut guard = self.sbio_rx.lock(); loop { + if self.shutting_down.load(Relaxed) { + guard.abort(); + let _ = guard.take_done(); + return Err(ENODEV); + } if let Some(done) = guard.take_done() { return Ok(done); } @@ -2631,6 +2943,9 @@ impl SepData { let mut remaining = time::msecs_to_jiffies(SBIO_TIMEOUT_MS); let mut guard = self.sbio_rx.lock(); loop { + if self.shutting_down.load(Relaxed) { + return Err(ENODEV); + } if guard.take_grant() { return Ok(()); } @@ -2680,7 +2995,11 @@ impl SepData { } pub(crate) fn bio_open(&self) -> Result<()> { + if self.shutting_down.load(Relaxed) { + return Err(ENODEV); + } let mut session = self.bio_session.lock(); + self.wait_for_worker_exit(&mut session, true)?; bio::open(&mut session)?; drop(session); if let Err(e) = self.prepare_bio_open() { @@ -2720,10 +3039,49 @@ impl SepData { self.captures_running.fetch_sub(1, Relaxed); } + /// The enrol or verify work item `epoch` has stopped. + pub(crate) fn capture_worker_stopped(&self, epoch: u64) { + bio::worker_stopped(&mut self.bio_session.lock(), epoch); + self.worker_gone.notify_all(); + } + + /// Holds an open or a capture start while the last enrol or verify work + /// item is still on its way out. It reports its result before it stops, + /// and after a cancel or a close it stops only once the capture notices, + /// so an open or a start straight after either would be refused with + /// EBUSY, which the caller cannot tell from a device in use. If it has + /// not stopped in time, the caller refuses as before. + fn wait_for_worker_exit( + &self, + session: &mut MutexGuard<'_, bio::Session>, + opening: bool, + ) -> Result<()> { + let mut remaining = time::msecs_to_jiffies(WORKER_EXIT_WAIT_MS); + while bio::waiting_on_worker(session, opening) { + if self.shutting_down.load(Relaxed) { + return Err(ENODEV); + } + if remaining == 0 { + dev_warn!( + self.dev, + "Touch ID: the last capture had not stopped after {} ms\n", + WORKER_EXIT_WAIT_MS + ); + return Ok(()); + } + match self.worker_gone.wait_interruptible_timeout(session, remaining) { + CondVarTimeoutResult::Woken { jiffies } => remaining = jiffies, + CondVarTimeoutResult::Timeout => remaining = 0, + CondVarTimeoutResult::Signal { .. } => return Err(ERESTARTSYS), + } + } + Ok(()) + } + /// The system is about to suspend. End any capture in progress, hold new /// ones, and wait for the enrol or verify work to stop touching the sensor /// before devices go down. - pub(crate) fn sleep_prepare(&self) { + pub(crate) fn sleep_prepare(&self) -> Result<()> { let woke = { let mut session = self.bio_session.lock(); self.suspending.store(true, Relaxed); @@ -2742,14 +3100,15 @@ impl SepData { if waited >= SLEEP_DRAIN_MS { dev_warn!( self.dev, - "Touch ID: a capture was still running {} ms after the system started to sleep; continuing\n", + "Touch ID: capture did not drain within {} ms; aborting suspend\n", SLEEP_DRAIN_MS ); - return; + return Err(EBUSY); } kernel::time::delay::fsleep(kernel::time::Delta::from_millis(10)); waited += 10; } + Ok(()) } /// The system has resumed, or the suspend was aborted. @@ -2774,15 +3133,24 @@ impl SepData { } pub(crate) fn queue_enrolment(this: Arc) { + let Some(epoch) = bio::queued_epoch(&this.bio_session.lock()) else { + return; + }; + this.capture_epoch.store(epoch, Relaxed); if workqueue::system() .enqueue::, ENROL_WORK_ID>(this.clone()) .is_err() { this.finish_enrolment(Err(ENROL_STATUS_SENSOR)); + this.capture_worker_stopped(epoch); } } pub(crate) fn queue_verify(this: Arc) { + let Some(epoch) = bio::queued_epoch(&this.bio_session.lock()) else { + return; + }; + this.capture_epoch.store(epoch, Relaxed); if workqueue::system() .enqueue::, VERIFY_WORK_ID>(this.clone()) .is_err() @@ -2791,11 +3159,14 @@ impl SepData { bio::VerifyOutcome::Failed(ENROL_STATUS_SENSOR), [0u8; bio::TOKEN_LEN], ); + this.capture_worker_stopped(epoch); } } fn finish_verify(&self, outcome: bio::VerifyOutcome, token_bytes: [u8; bio::TOKEN_LEN]) { - let filed = bio::verify_finish(&mut self.bio_session.lock(), outcome, token_bytes); + let filed = bio::verify_finish( + &mut self.bio_session.lock(), self.capture_epoch.load(Relaxed), outcome, token_bytes, + ); if filed { self.bio_wake(); } @@ -2815,6 +3186,9 @@ impl SepData { } pub(crate) fn bio_ioctl(&self, cmd: u32, arg: usize) -> Result { + if self.shutting_down.load(Relaxed) { + return Err(ENODEV); + } if cmd == bio::IOC_ATTEST { return self.bio_attest(arg); } @@ -2851,6 +3225,9 @@ impl SepData { let sensor_present = sensor::is_bound(); let handled = { let mut session = self.bio_session.lock(); + if bio::starts_capture(cmd) { + self.wait_for_worker_exit(&mut session, false)?; + } let index = self.bio_index.lock(); let mut ctx = bio::Context { session: &mut session, @@ -2909,11 +3286,11 @@ impl SepData { let mut out = bio::Attest { sig_len: sig.len() as u32, challenge: req.challenge, - public: [0u8; bio::ATTEST_PUB_LEN], + public_key: [0u8; bio::ATTEST_PUB_LEN], signature: [0u8; bio::ATTEST_SIG_MAX], reserved: [0u8; 3], }; - out.public.copy_from_slice(&pubk); + out.public_key.copy_from_slice(&pubk); out.signature[..sig.len()].copy_from_slice(&sig); kernel::uaccess::UserSlice::new(user, core::mem::size_of::()) .writer() @@ -2959,6 +3336,17 @@ impl SepData { let restored = self.restore_all_components(); dev_info!(self.dev, "sbio: cold-match preparation {}, component restore {}\n", prepared, restored); + // Restore can replace the credential and the device's identity view. + // Refresh both after loading J700's persistent components, before + // admitting a verify operation. + let match_context = if self.profile.nonce_session { + let synced = restored && self.sync_device_view(); + self.device_view_synced.store(synced, Relaxed); + synced && crate::sbio::UserId::new(SBIO_PROBE_USER_ID) + .is_some_and(|user| self.establish_scrd_match_context(user)) + } else { + match_context + }; let identities = self.log_identity_count(c"after Catacomb restore with sensor registered"); let identity_ready = matches!(identities, Some(n) if n > 0); let candidate = prepared && match_context && restored && identity_ready; @@ -2967,6 +3355,11 @@ impl SepData { // verification available on the preliminary Catacomb proof alone; // complete_bringup finishes the proof and sets templates_restored. self.templates_restored.store(false, Relaxed); + // The nonce path synced the view before COMPLETE_INIT to refresh SCRD. + // Re-arm the final proof and its bounded retry on every cold restore. + if self.profile.nonce_session { + self.device_view_synced.store(false, Relaxed); + } if !candidate { dev_err!( self.dev, @@ -3521,10 +3914,10 @@ pub(crate) const SBIO_CALIBRATION_SOURCE: u32 = 3; pub(crate) struct SbioCalibration(KVec); impl SbioCalibration { - pub(crate) fn new(blob: &crate::CalibrationBlob) -> Result { + pub(crate) fn new(blob: &crate::CalibrationBlob, source: u32) -> Result { let bytes = blob.bytes(); let mut body = KVec::new(); - body.extend_from_slice(&SBIO_CALIBRATION_SOURCE.to_le_bytes(), GFP_KERNEL)?; + body.extend_from_slice(&source.to_le_bytes(), GFP_KERNEL)?; body.extend_from_slice(bytes, GFP_KERNEL)?; Ok(SbioCalibration(body)) } diff --git a/drivers/soc/apple/sensor.rs b/drivers/soc/apple/sensor.rs index 8fec6b71c8715f..c0f3a6159f74c9 100644 --- a/drivers/soc/apple/sensor.rs +++ b/drivers/soc/apple/sensor.rs @@ -1,9 +1,11 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT // Copyright 2026 Dj +// Copyright 2026 Ryan Murray //! The fingerprint sensor over SPI. use kernel::prelude::*; +use kernel::sync::atomic::{Atomic, Relaxed}; use kernel::time::delay::fsleep; use kernel::time::Delta; @@ -117,6 +119,7 @@ const CMD_START_CAPTURE: [u8; CMD_LEN] = [0x80, 0x40, 0x00, 0x07, 0x00, 0x00, 0x // Precedes the patch blob. const CMD_SETUP_PATCH_ENABLE: [u8; CMD_LEN] = [0x80, 0x60, 0x00, 0x07, 0x00, 0x00, 0x00]; const CMD_GET_SENSOR_SERIAL: [u8; CMD_LEN] = [0x80, 0x70, 0x00, 0x07, 0x00, 0x00, 0x00]; +const CMD_GET_NONCE: [u8; CMD_LEN] = [0x80, 0x35, 0x00, 0x07, 0x00, 0x00, 0x00]; const READ_CMD_LEN: usize = 11; const CMD_READ_PREFIX: [u8; CMD_LEN] = [0x80, 0x13, 0x00, 0x0b, 0x00, 0x00, 0x00]; @@ -474,26 +477,18 @@ fn poll_until_ready_any(accepted: &[usize]) -> Result { } pub(crate) fn send_session_share(share: &[u8; SESSION_SHARE_LEN]) -> Result<()> { + drain_stale_frames()?; send_framed(TYPE_SESSION, share) } pub(crate) fn send_challenge(challenge: &[u8; CHALLENGE_LEN]) -> Result<()> { + drain_stale_frames()?; send_framed(TYPE_SEQUENCE, challenge) } static_assert!(FRAME_HEADER + SESSION_SHARE_LEN == 0x2f); static_assert!(FRAME_HEADER + CHALLENGE_LEN == 0x47); -fn read_framed(total: usize, at: usize, out: &mut [u8]) -> Result<()> { - let mut buf = [0xffu8; MAX_REPLY]; - read_reply(total, &mut buf)?; - if at + out.len() > total { - return Err(EINVAL); - } - out.copy_from_slice(&buf[at..at + out.len()]); - Ok(()) -} - fn poll_until_ready(expected: usize) -> Result<()> { for _ in 0..READ_POLL_ATTEMPTS { let st = status()?; @@ -510,11 +505,6 @@ fn poll_until_ready(expected: usize) -> Result<()> { const READ_POLL_MS: i64 = 5; const READ_POLL_ATTEMPTS: usize = 200; -fn read_reply(total: usize, buf: &mut [u8; MAX_REPLY]) -> Result<()> { - poll_until_ready(total)?; - read_reply_ready(total, buf) -} - fn read_reply_ready(total: usize, buf: &mut [u8; MAX_REPLY]) -> Result<()> { if total > MAX_REPLY { return Err(EINVAL); @@ -537,22 +527,99 @@ fn read_reply_ready(total: usize, buf: &mut [u8; MAX_REPLY]) -> Result<()> { }) } -fn read_framed_verified(total: usize, at: usize, out: &mut [u8]) -> Result<()> { - if total < CRC_LEN || at + out.len() > total - CRC_LEN { - return Err(EINVAL); +/// Frames the sensor still offered after the read that should have taken +/// them, and stale frames found on offer before a command. Both mean an +/// exchange went wrong on the bus; the caller reports them. +static REREADS: Atomic = Atomic::new(0); +static DRAINED: Atomic = Atomic::new(0); + +/// Takes the counts of re-read and drained frames since the last call. +pub(crate) fn take_recoveries() -> (u32, u32) { + (REREADS.xchg(0, Relaxed), DRAINED.xchg(0, Relaxed)) +} + +// A garbled read the sensor did not take is read once more, twice at most. +const FRAME_READS: usize = 3; +// Stale frames taken before a command; the sensor queues at most a couple. +const DRAIN_FRAMES: usize = 2; + +/// A reply buffer, wiped however the read that fills it ends. +struct FrameBuf([u8; MAX_REPLY]); + +impl FrameBuf { + fn new() -> Self { + FrameBuf([0xff; MAX_REPLY]) } - let mut buf = [0xffu8; MAX_REPLY]; - read_reply(total, &mut buf)?; +} - let split = total - CRC_LEN; - let expected = u16::from_le_bytes([buf[split], buf[split + 1]]); - let computed = crc16_ansi(&buf[..split]); - if expected != computed { - return Err(EIO); +impl Drop for FrameBuf { + fn drop(&mut self) { + for b in self.0.iter_mut() { + // SAFETY: a valid, uniquely borrowed byte. + unsafe { core::ptr::write_volatile(b, 0) }; + } } +} - out.copy_from_slice(&buf[at..at + out.len()]); - Ok(()) +fn frame_crc_valid(frame: &[u8]) -> bool { + let split = frame.len() - CRC_LEN; + u16::from_le_bytes([frame[split], frame[split + 1]]) == crc16_ansi(&frame[..split]) +} + +/// Takes any frame the sensor still offers before a new command, so the +/// reply to that command is not mistaken for, or queued behind, a stale one. +/// Fails if the sensor still offers a frame once the budget is spent: a +/// read is no proof the sensor took the frame. +fn drain_stale_frames() -> Result<()> { + for _ in 0..DRAIN_FRAMES { + let Some(count) = status()?.available().filter(|n| *n != 0) else { + return Ok(()); + }; + let count = count as usize; + if count > MAX_REPLY { + return Err(EIO); + } + let mut buf = FrameBuf::new(); + read_reply_ready(count, &mut buf.0)?; + DRAINED.fetch_add(1, Relaxed); + } + match status()?.available() { + None | Some(0) => Ok(()), + Some(_) => Err(EBUSY), + } +} + +/// Sends `cmd` and reads its `total`-byte reply, checking the reply's CRC. +fn command_frame(cmd: &[u8; CMD_LEN], total: usize, at: usize, out: &mut [u8]) -> Result<()> { + drain_stale_frames()?; + command(cmd)?; + read_frame(total, at, out) +} + +/// Reads a `total`-byte reply once the sensor offers it and checks its CRC. +/// A read the sensor did not take leaves the frame on offer at the same +/// length: only then is it read again. +fn read_frame(total: usize, at: usize, out: &mut [u8]) -> Result<()> { + if total < CRC_LEN || total > MAX_REPLY || at + out.len() > total - CRC_LEN { + return Err(EINVAL); + } + poll_until_ready(total)?; + + let mut buf = FrameBuf::new(); + for attempt in 0..FRAME_READS { + if attempt > 0 { + if status()?.available() != Some(total as u32) { + break; + } + REREADS.fetch_add(1, Relaxed); + } + read_reply_ready(total, &mut buf.0)?; + if frame_crc_valid(&buf.0[..total]) { + out.copy_from_slice(&buf.0[at..at + out.len()]); + return Ok(()); + } + } + Err(EIO) } const MAX_REPLY: usize = 0x4f; @@ -575,21 +642,30 @@ impl SensorSerial { } pub(crate) fn read_sensor_serial() -> Result { - command(&CMD_GET_SENSOR_SERIAL)?; let mut serial = [0u8; SENSOR_SERIAL_LEN]; - read_framed_verified(SENSOR_SERIAL_REPLY_LEN, SENSOR_SERIAL_AT, &mut serial)?; + command_frame(&CMD_GET_SENSOR_SERIAL, SENSOR_SERIAL_REPLY_LEN, SENSOR_SERIAL_AT, &mut serial)?; Ok(SensorSerial(serial)) } +/// Read the nonce as one frame; a separate acknowledgment would consume it. +pub(crate) fn session_nonce() -> Result<[u8; 32]> { + let mut value = [0; 32]; + command_frame(&CMD_GET_NONCE, 9 + value.len() + CRC_LEN, 9, &mut value)?; + if value.iter().all(|byte| *byte == 0) { + return Err(EIO); + } + Ok(value) +} + pub(crate) fn read_session_reply() -> Result<[u8; SESSION_SHARE_LEN]> { let mut share = [0u8; SESSION_SHARE_LEN]; - read_framed(SESSION_REPLY_LEN, SESSION_REPLY_AT, &mut share)?; + read_frame(SESSION_REPLY_LEN, SESSION_REPLY_AT, &mut share)?; Ok(share) } pub(crate) fn read_challenge_reply() -> Result<[u8; CHALLENGE_LEN]> { let mut reply = [0u8; CHALLENGE_LEN]; - read_framed(CHALLENGE_REPLY_LEN, CHALLENGE_REPLY_AT, &mut reply)?; + read_frame(CHALLENGE_REPLY_LEN, CHALLENGE_REPLY_AT, &mut reply)?; Ok(reply) } @@ -623,6 +699,14 @@ impl Identifier { } impl Status { + /// The length of the frame the sensor offers, if it offers one. + pub(crate) fn available(&self) -> Option { + match self.offset12() { + Offset12::Available(n) => Some(n), + _ => None, + } + } + pub(crate) fn offset12(&self) -> Offset12 { match self.state { STATE_IDLE => Offset12::Identifier(u16::from_le_bytes([ diff --git a/drivers/soc/apple/sensor_shim.c b/drivers/soc/apple/sensor_shim.c index ab6eb3ddb69b2e..76ac37b308e003 100644 --- a/drivers/soc/apple/sensor_shim.c +++ b/drivers/soc/apple/sensor_shim.c @@ -19,6 +19,8 @@ #include #include +#include + #include "shim.h" /* Use the board's device-tree mode (J313: 1, J414s: 2). */ @@ -35,6 +37,28 @@ MODULE_PARM_DESC(sensor_spi_mode, #define SEP_SENSOR_OFF_DELAY_MS 10 #define SEP_SENSOR_ON_DELAY_MS 7 +struct sep_sensor_lifetime { + /* Serializes sensor operations with this device's resource teardown. */ + struct mutex lock; +}; + +static DEFINE_MUTEX(sep_sensor_registry_lock); +static struct sep_sensor_lifetime *sep_active_sensor; + +static struct sep_sensor_lifetime *sep_lock_sensor(void) +{ + struct sep_sensor_lifetime *sensor; + + mutex_lock(&sep_sensor_registry_lock); + sensor = sep_active_sensor; + if (sensor) + mutex_lock(&sensor->lock); + mutex_unlock(&sep_sensor_registry_lock); + return sensor; +} + +static void sep_sensor_irq_teardown_locked(void); + static struct spi_device *sep_spi; static struct gpio_desc *sep_power; static bool sep_registered; @@ -216,8 +240,12 @@ static int sep_acquire_power(struct spi_device *spi) sep_power = gpiod_get(&spi->dev, "enable", GPIOD_OUT_LOW); if (IS_ERR(sep_power) && PTR_ERR(sep_power) == -ENOENT) sep_power = gpiod_get_index(&spi->dev, NULL, 0, GPIOD_OUT_LOW); - if (IS_ERR(sep_power) && PTR_ERR(sep_power) == -EPROBE_DEFER) - return -EPROBE_DEFER; + if (IS_ERR(sep_power) && PTR_ERR(sep_power) != -ENOENT) { + int error = PTR_ERR(sep_power); + + sep_power = NULL; + return error; + } if (!IS_ERR(sep_power)) { sep_power_source = SEP_POWER_NODE_PROPERTY; dev_info(&spi->dev, @@ -229,12 +257,12 @@ static int sep_acquire_power(struct spi_device *spi) /* * The fallback below is the j414s power line; a sibling board's sensor sits * on a different pin, so only j414s may use it. Every other machine must - * describe the power GPIO in its device node (the gpiod_get_index path). + * describe enable-gpios in its device node. */ if (!of_machine_is_compatible("apple,j414s")) { dev_warn(&spi->dev, - "sep sensor: no power GPIO in the device node; describe gpios in DT\n"); - return 0; + "sep sensor: no power GPIO in the device node; describe enable-gpios in DT\n"); + return -ENODEV; } np = of_find_node_by_path(SEP_SENSOR_GPIO_NODE); @@ -242,7 +270,7 @@ static int sep_acquire_power(struct spi_device *spi) dev_warn(&spi->dev, "sep sensor: no DT node at %s, no power line\n", SEP_SENSOR_GPIO_NODE); - return 0; + return -ENODEV; } gdev = gpio_device_find_by_fwnode(of_fwnode_handle(np)); @@ -251,7 +279,7 @@ static int sep_acquire_power(struct spi_device *spi) dev_warn(&spi->dev, "sep sensor: %s has no registered GPIO device\n", SEP_SENSOR_GPIO_NODE); - return 0; + return -EPROBE_DEFER; } gc = gpio_device_get_chip(gdev); @@ -268,7 +296,7 @@ static int sep_acquire_power(struct spi_device *spi) SEP_SENSOR_GPIO_LINE, SEP_SENSOR_GPIO_NODE, gpio_device_get_label(gdev)); gpio_device_put(gdev); - return 0; + return -ENODEV; } sep_power_source = SEP_POWER_CHIP_LINE; @@ -331,11 +359,14 @@ static int sep_apply_cs_timing(struct spi_device *spi) * select. spi_setup() takes the controller io_mutex, not the bus lock, * so no deadlock. */ + if (sep_sensor_mode != ~0U && sep_sensor_mode > SPI_MODE_3) + return -EINVAL; + rc = spi_bus_lock(ctlr); if (rc) return rc; - if (sep_sensor_mode <= SPI_MODE_3) + if (sep_sensor_mode != ~0U) spi->mode = sep_sensor_mode; spi->bits_per_word = SEP_SENSOR_BITS; spi->max_speed_hz = SEP_SENSOR_HZ; @@ -370,7 +401,7 @@ static int sep_apply_cs_timing(struct spi_device *spi) return 0; } -static int sep_sensor_probe(struct spi_device *spi) +static int sep_sensor_probe_locked(struct spi_device *spi) { int rc; @@ -393,12 +424,42 @@ static int sep_sensor_probe(struct spi_device *spi) return 0; } +static int sep_sensor_probe(struct spi_device *spi) +{ + struct sep_sensor_lifetime *sensor; + int error; + + sensor = devm_kzalloc(&spi->dev, sizeof(*sensor), GFP_KERNEL); + if (!sensor) + return -ENOMEM; + mutex_init(&sensor->lock); + mutex_lock(&sep_sensor_registry_lock); + if (sep_active_sensor) { + error = -EBUSY; + } else { + error = sep_sensor_probe_locked(spi); + if (!error) { + spi_set_drvdata(spi, sensor); + sep_active_sensor = sensor; + } + } + mutex_unlock(&sep_sensor_registry_lock); + return error; +} + static void sep_sensor_remove(struct spi_device *spi) { - sep_sensor_irq_teardown(); + struct sep_sensor_lifetime *sensor = spi_get_drvdata(spi); + + mutex_lock(&sep_sensor_registry_lock); + mutex_lock(&sensor->lock); + sep_active_sensor = NULL; + sep_sensor_irq_teardown_locked(); sep_release_power(); sep_spi = NULL; sep_diag_set_sensor_bound(false); + mutex_unlock(&sensor->lock); + mutex_unlock(&sep_sensor_registry_lock); } /* @@ -455,19 +516,19 @@ void sep_sensor_unregister(void) sep_registered = false; } -int sep_sensor_bound(void) +static int sep_sensor_bound_locked(void) { return sep_spi != NULL; } /* Whether those delays reach hardware, are emulated, or are silently bypassed. */ -int sep_sensor_cs_timing_mode(void) +static int sep_sensor_cs_timing_mode_locked(void) { return sep_cs_timing_mode; } /* Power cycle: off, 10 ms, on, 7 ms. -ENODEV if there is no power line. */ -int sep_sensor_power_cycle(void) +static int sep_sensor_power_cycle_locked(void) { if (sep_power_source == SEP_POWER_SMC_RAILS) { int rc = sep_smc_set_power(0); @@ -492,12 +553,12 @@ int sep_sensor_power_cycle(void) } /* How the power line was obtained: none, the node's property, or chip+line. */ -int sep_sensor_power_source(void) +static int sep_sensor_power_source_locked(void) { return sep_power_source; } -int sep_sensor_power_line(void) +static int sep_sensor_power_line_locked(void) { if (sep_power_source == SEP_POWER_SMC_RAILS) return -ENODEV; @@ -511,7 +572,7 @@ int sep_sensor_power_line(void) * the sensor node's "firmware-name" property. NULL when absent, so the caller * can fall back to a default. */ -const char *sep_sensor_firmware_name(void) +static const char *sep_sensor_firmware_name_locked(void) { const char *name = NULL; @@ -523,7 +584,7 @@ const char *sep_sensor_firmware_name(void) } /* Powers the sensor on/off, holding the hardware settling delay. */ -int sep_sensor_power(int on) +static int sep_sensor_power_locked(int on) { if (sep_power_source == SEP_POWER_SMC_RAILS) { int rc = sep_smc_set_power(on ? 1 : 0); @@ -562,7 +623,7 @@ static irqreturn_t sep_drdy_isr(int irq, void *dev_id) * firmware patch. Returns 0 when an interrupt is available, -errno otherwise * (the caller then falls back to polling). */ -int sep_sensor_irq_setup(void) +static int sep_sensor_irq_setup_locked(void) { struct device_node *np; struct gpio_device *gdev; @@ -635,7 +696,7 @@ int sep_sensor_irq_setup(void) return 0; } -void sep_sensor_irq_teardown(void) +static void sep_sensor_irq_teardown_locked(void) { if (sep_drdy_irq >= 0) { free_irq(sep_drdy_irq, sep_spi); @@ -647,14 +708,14 @@ void sep_sensor_irq_teardown(void) } } -int sep_sensor_irq_available(void) +static int sep_sensor_irq_available_locked(void) { return sep_drdy_irq >= 0; } /* Clear any stale signal before a capture so the next wait reflects a fresh * data-ready edge, not a leftover from the previous frame. */ -void sep_sensor_irq_arm(void) +static void sep_sensor_irq_arm_locked(void) { if (sep_drdy_irq >= 0) reinit_completion(&sep_drdy_done); @@ -665,7 +726,7 @@ void sep_sensor_irq_arm(void) * -ETIMEDOUT = no signal (the caller re-polls status over SPI regardless, so a * missed interrupt only costs one poll interval), -ENODEV = no interrupt. */ -int sep_sensor_irq_wait(unsigned int timeout_ms) +static int sep_sensor_irq_wait_locked(unsigned int timeout_ms) { if (sep_drdy_irq < 0) return -ENODEV; @@ -680,7 +741,7 @@ int sep_sensor_irq_wait(unsigned int timeout_ms) * `len` transmit bytes (trailing ones as 0xff) rather than relying on the * controller running its transmit buffer dry. */ -int sep_sensor_xfer(const void *tx, void *rx, size_t len) +static int sep_sensor_xfer_locked(const void *tx, void *rx, size_t len) { struct spi_transfer xfer = { .tx_buf = tx, @@ -703,7 +764,7 @@ int sep_sensor_xfer(const void *tx, void *rx, size_t len) * A simultaneous receive changes the long transfer's pacing and the sensor * rejects the patch blob, so the receive must be absent entirely. */ -int sep_sensor_xfer_tx(const void *tx, size_t len) +static int sep_sensor_xfer_tx_locked(const void *tx, size_t len) { struct spi_transfer xfer = { .tx_buf = tx, @@ -725,7 +786,7 @@ int sep_sensor_xfer_tx(const void *tx, size_t len) * Two transfers in one chip-select assertion: command out, then read in. * Chip-select stays asserted because neither transfer sets cs_change. */ -int sep_sensor_xfer2(const void *tx, size_t tx_len, void *rx, size_t rx_len) +static int sep_sensor_xfer2_locked(const void *tx, size_t tx_len, void *rx, size_t rx_len) { struct spi_transfer xfers[2] = { { @@ -749,3 +810,179 @@ int sep_sensor_xfer2(const void *tx, size_t tx_len, void *rx, size_t rx_len) return spi_sync_transfer(sep_spi, xfers, 2); } + +int sep_sensor_bound(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return 0; + result = sep_sensor_bound_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_cs_timing_mode(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return 0; + result = sep_sensor_cs_timing_mode_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_power_cycle(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_power_cycle_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_power_source(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return 0; + result = sep_sensor_power_source_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_power_line(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_power_line_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +const char *sep_sensor_firmware_name(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + const char *result; + + if (!sensor) + return NULL; + result = sep_sensor_firmware_name_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_power(int on) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_power_locked(on); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_irq_setup(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_irq_setup_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +void sep_sensor_irq_teardown(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + + if (!sensor) + return; + sep_sensor_irq_teardown_locked(); + mutex_unlock(&sensor->lock); +} + +int sep_sensor_irq_available(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return 0; + result = sep_sensor_irq_available_locked(); + mutex_unlock(&sensor->lock); + return result; +} + +void sep_sensor_irq_arm(void) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + + if (!sensor) + return; + sep_sensor_irq_arm_locked(); + mutex_unlock(&sensor->lock); +} + +int sep_sensor_irq_wait(unsigned int timeout_ms) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_irq_wait_locked(timeout_ms); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_xfer(const void *tx, void *rx, size_t len) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_xfer_locked(tx, rx, len); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_xfer_tx(const void *tx, size_t len) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_xfer_tx_locked(tx, len); + mutex_unlock(&sensor->lock); + return result; +} + +int sep_sensor_xfer2(const void *tx, size_t tx_len, void *rx, size_t rx_len) +{ + struct sep_sensor_lifetime *sensor = sep_lock_sensor(); + int result; + + if (!sensor) + return -ENODEV; + result = sep_sensor_xfer2_locked(tx, tx_len, rx, rx_len); + mutex_unlock(&sensor->lock); + return result; +} diff --git a/drivers/soc/apple/sep.rs b/drivers/soc/apple/sep.rs index 14e410216ecee9..8ff12dfefab8ec 100644 --- a/drivers/soc/apple/sep.rs +++ b/drivers/soc/apple/sep.rs @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT // Copyright 2026 Dj +// Copyright 2026 Ryan Murray //! Driver for the Apple SEP (Secure Enclave Processor): attaches to a running //! SEP over the AP mailbox, drives Touch ID enrol/verify via `/dev/sep-bio`, //! and registers a hwrng and a SEP-backed `trusted` key source. @@ -24,6 +25,7 @@ mod profile; mod proto; mod refkey; mod refkey_seal; +mod registration; mod rxring; mod sbio; mod scrd; @@ -63,7 +65,8 @@ use kernel::{ Arc, CondVar, CondVarTimeoutResult, - Mutex, // + Mutex, + MutexGuard, // }, time, types::ForeignOwnable, @@ -113,6 +116,11 @@ const SBIO_TIMEOUT_MS: time::Msecs = 5000; const SLEEP_DRAIN_MS: u32 = 6000; static_assert!(SLEEP_DRAIN_MS as u64 > SBIO_TIMEOUT_MS as u64); +// How long an open or a capture start waits for the last enrol or verify work +// item to stop once its operation has ended. It is stopped in the same way, +// so it gets the same bound as a capture draining before sleep. +const WORKER_EXIT_WAIT_MS: time::Msecs = SLEEP_DRAIN_MS; + const SKS_ALLOC: usize = 0x8000; const SKS_SECRET_LEN: usize = 32; @@ -244,9 +252,9 @@ enum SbioOutcome { Other, } -const BRINGUP_FRESH: u32 = 0; -const BRINGUP_IDENTIFIED: u32 = 1; -const BRINGUP_ESTABLISHED: u32 = 2; +const BRINGUP_FRESH: u32 = registration::FRESH; +const BRINGUP_IDENTIFIED: u32 = registration::IDENTIFIED; +const BRINGUP_ESTABLISHED: u32 = registration::ESTABLISHED; pub(crate) struct PatchLoaded(()); @@ -586,6 +594,11 @@ struct SepData { #[pin] dma_ring: Mutex>, + // One SBIO request/reply exchange at a time: the endpoint has a single + // out-of-line buffer and `sbio_rx` tracks a single transfer. + #[pin] + sbio_exchange_lock: Mutex<()>, + #[pin] sbio_rx: Mutex, @@ -599,6 +612,7 @@ struct SepData { cold_restore_candidate: Atomic, sensor_calibrated: Atomic, + sensor_key_loaded: Atomic, enrol_open: Atomic, @@ -615,6 +629,10 @@ struct SepData { last_capture_end_ns: Atomic, + // Boot-time clock at the start of the last sensor power cycle, for + // reporting how far into a bring-up a step failed. + sensor_power_ns: Atomic, + device_view_synced: Atomic, keybag_designated: Atomic, @@ -624,6 +642,9 @@ struct SepData { touchid_failed: Atomic, bringup: Atomic, + // The enclave accepted an identification of the sensor since it booted, + // even if the registration it began then failed. + sensor_held: Atomic, #[pin] ool_sks: Mutex>, @@ -686,6 +707,7 @@ struct SepData { registered: Atomic, shutting_down: Atomic, + capture_epoch: Atomic, // The system is going to sleep: no capture may start. Changed under the // bio session lock; `sleep_over` is signalled when it clears. @@ -709,6 +731,10 @@ struct SepData { // only under `catacomb_writer`. deletion_unsaved: Atomic, + // Signalled when an enrol or verify work item has stopped. + #[pin] + worker_gone: CondVar, + #[pin] rx_work: Work, @@ -831,6 +857,7 @@ impl SepData { sks_initialized: Atomic::new(false), sks_wedged: Atomic::new(0), bringup: Atomic::new(BRINGUP_FRESH), + sensor_held: Atomic::new(false), keybag_designated: Atomic::new(false), bringup_started: Atomic::new(false), touchid_started: Atomic::new(false), @@ -839,11 +866,13 @@ impl SepData { xart_refusal_logged: Atomic::new(false), capture_refusal_logged: Atomic::new(false), sensor_calibrated: Atomic::new(false), + sensor_key_loaded: Atomic::new(false), templates_restored: Atomic::new(false), cold_restore_candidate: Atomic::new(false), enrol_material <- new_mutex!(None), enrol_identity_candidates <- new_mutex!(KVec::new()), last_capture_end_ns: Atomic::new(0), + sensor_power_ns: Atomic::new(0), device_view_synced: Atomic::new(false), phase: Atomic::new(PHASE_ATTACH), ool_xarm <- new_mutex!(Some(ool_xarm)), @@ -856,6 +885,7 @@ impl SepData { scrd_probe <- new_mutex!(ScrdProbe::new()), scrd_wq <- new_condvar!("SepData::scrd_wq"), dma_ring <- new_mutex!(Some(dma_ring)), + sbio_exchange_lock <- new_mutex!(()), sbio_rx <- new_mutex!(transfer::Reassembly::new()), sbio_wq <- new_condvar!("SepData::sbio_wq"), sbio_ready: Atomic::new(false), @@ -878,11 +908,13 @@ impl SepData { settle_idle_ticks: Atomic::new(0), registered: Atomic::new(false), shutting_down: Atomic::new(false), + capture_epoch: Atomic::new(0), suspending: Atomic::new(false), sleep_over <- new_condvar!("SepData::sleep_over"), captures_running: Atomic::new(0), catacomb_writer <- new_mutex!(()), deletion_unsaved: Atomic::new(false), + worker_gone <- new_condvar!("SepData::worker_gone"), rx_work <- new_work!("SepData::rx_work"), enrol_work <- new_work!("SepData::enrol_work"), verify_work <- new_work!("SepData::verify_work"), @@ -1076,6 +1108,9 @@ impl SepData { } fn send(&self, msg: Message) -> Result<()> { + if self.shutting_down.load(Relaxed) { + return Err(ENODEV); + } self.mbox.lock().as_ref().ok_or(ENODEV)?.send(msg, false) } @@ -1092,6 +1127,10 @@ impl SepData { let mut remaining = time::msecs_to_jiffies(op.timeout_ms()); let mut guard = self.control.lock(); loop { + if self.shutting_down.load(Relaxed) { + guard.abandon(idx); + return Err(ENODEV); + } if let Some(reply) = guard.take_reply(idx) { guard.release(idx); drop(guard); @@ -1546,7 +1585,8 @@ impl SepData { let mut remaining = time::msecs_to_jiffies(ms); let mut guard = self.scrd_probe.lock(); loop { - if guard.captured.len() >= until || remaining == 0 { + if self.shutting_down.load(Relaxed) + || guard.captured.len() >= until || remaining == 0 { return; } match self @@ -2053,20 +2093,21 @@ impl SepData { fn remove(&self) { self.shutting_down.store(true, Relaxed); + bio::release(&mut self.bio_session.lock()); + self.worker_gone.notify_all(); + self.rng_shutdown.store(true, Relaxed); + self.control_wq.notify_all(); + self.sbio_wq.notify_all(); + self.sks_wq.notify_all(); + self.scrd_wq.notify_all(); pm::unregister(); // Release any capture start still held for a resume that will not be // reported now. self.sleep_finished(); - self.release_firmware_mapping(); trusted::unregister(); self.unregister_fv_kernel(); let dev = self.bio_dev.lock().take(); - if dev.is_some() { - bio::release(&mut self.bio_session.lock()); - sensor::power(false); - sensor::unregister_driver(); - } drop(dev); self.rng_shutdown.store(true, Relaxed); @@ -2091,6 +2132,10 @@ impl SepData { ); } + sensor::power(false); + sensor::unregister_driver(); + self.release_firmware_mapping(); + let _ = self.store.lock().take(); let _ = self.host_store.lock().take(); @@ -2243,14 +2288,17 @@ impl WorkItem for SepData { type Pointer = Arc; fn run(this: Arc) { - // Taken before `capture_begin`, so the sleep drain never counts the - // wait for it. - let _writer = this.catacomb_writer.lock(); - if this.shutting_down.load(Relaxed) || !this.capture_begin() { - return; + let epoch = this.capture_epoch.load(Relaxed); + { + // Taken before `capture_begin`, so the sleep drain never counts the + // wait for it. + let _writer = this.catacomb_writer.lock(); + if !this.shutting_down.load(Relaxed) && this.capture_begin() { + this.run_enrolment(); + this.capture_end(); + } } - this.run_enrolment(); - this.capture_end(); + this.capture_worker_stopped(epoch); } } @@ -2258,20 +2306,27 @@ impl WorkItem for SepData { type Pointer = Arc; fn run(this: Arc) { - let _writer = this.catacomb_writer.lock(); - if this.shutting_down.load(Relaxed) || !this.capture_begin() { - return; + let epoch = this.capture_epoch.load(Relaxed); + { + let _writer = this.catacomb_writer.lock(); + if !this.shutting_down.load(Relaxed) && this.capture_begin() { + this.run_verify(); + this.capture_end(); + } } - this.run_verify(); - this.capture_end(); + this.capture_worker_stopped(epoch); } } struct SepDriver(Arc); -const OF_TABLE: kernel::device_id::IdArray = - kernel::device_id::IdArray::new([(of::DeviceId::new(c"apple,sep"), ())]); +kernel::of_device_table!( + OF_TABLE, + MODULE_OF_TABLE, + (), + [(of::DeviceId::new(c"apple,sep"), ())] +); impl platform::Driver for SepDriver { type IdInfo = (); @@ -2405,7 +2460,7 @@ module! { }, provision_keybag: u8 { default: 1, - description: "Create the Linux identity keybag when none exists; requires xart_writes=1", + description: "Create the Linux identity keybag when none exists; set to 0 to disable first-boot provisioning; requires xart_writes=1", }, os_uuid_hi: u64 { default: 0, @@ -2415,6 +2470,10 @@ module! { default: 0, description: "Low 64 bits of an explicit xART OS UUID", }, + sensor_trace: u8 { + default: 0, + description: "Trace Touch ID sensor registration steps and timing, never sensor data", + }, xarm_trace: u8 { default: 0, description: "Opt-in xART opcode/status trace without payloads or key material", diff --git a/drivers/soc/apple/shim.h b/drivers/soc/apple/shim.h index 47e63602756041..83839b9353691e 100644 --- a/drivers/soc/apple/shim.h +++ b/drivers/soc/apple/shim.h @@ -25,7 +25,7 @@ void sep_cancel_delayed_work_sync(void *work); /* -- pm_shim.c ---------------------------------------------------------- */ -int sep_pm_register(void (*event)(bool entering)); +int sep_pm_register(int (*event)(bool entering)); void sep_pm_unregister(void); int sep_pm_keep_domains(void *dev); @@ -42,6 +42,7 @@ void sep_hwrng_unregister(void *mem); /* -- store_shim.c ------------------------------------------------------- */ void *sep_store_open(const char *path); +void *sep_store_open_host(const char *path, bool *created); void *sep_store_open_trunc(const char *path); void *sep_store_open_ro(const char *path); void *sep_store_open_block(const char *path, int writable); diff --git a/drivers/soc/apple/shim.rs b/drivers/soc/apple/shim.rs index 4e25060d15d25a..7e7f489e594818 100644 --- a/drivers/soc/apple/shim.rs +++ b/drivers/soc/apple/shim.rs @@ -7,6 +7,7 @@ use kernel::prelude::*; extern "C" { fn sep_store_open(path: *const c_char) -> *mut c_void; + fn sep_store_open_host(path: *const c_char, created: *mut bool) -> *mut c_void; fn sep_store_open_trunc(path: *const c_char) -> *mut c_void; fn sep_store_open_ro(path: *const c_char) -> *mut c_void; fn sep_store_open_block(path: *const c_char, writable: c_int) -> *mut c_void; @@ -29,7 +30,7 @@ pub(crate) fn random_bytes(buf: &mut [u8]) -> Result<()> { /// Backing-store file handle. /// /// # Invariants -/// `handle` is a live `struct file *` from `sep_store_open`. +/// `handle` is a live `struct file *` from one of the store-open shims. pub(crate) struct StoreFile { handle: *mut c_void, } @@ -56,6 +57,19 @@ impl StoreFile { Ok(StoreFile { handle }) } + /// Opens the file like [`StoreFile::open`], and also reports whether this + /// call created it. + pub(crate) fn open_host(path: &CStr) -> Result<(StoreFile, bool)> { + let mut created = false; + // SAFETY: `path` is NUL-terminated and `created` is valid for writes + // for the duration of the call. The shim returns a live file or an + // error pointer. + let handle = unsafe { + kernel::error::from_err_ptr(sep_store_open_host(path.as_char_ptr(), &mut created))? + }; + Ok((StoreFile { handle }, created)) + } + pub(crate) fn open_trunc(path: &CStr) -> Result { // SAFETY: `path` is NUL-terminated; the shim returns NULL on failure. let handle = unsafe { sep_store_open_trunc(path.as_char_ptr()) }; diff --git a/drivers/soc/apple/sks.rs b/drivers/soc/apple/sks.rs index f882bf2e92e41c..4020d089eb0019 100644 --- a/drivers/soc/apple/sks.rs +++ b/drivers/soc/apple/sks.rs @@ -1127,7 +1127,8 @@ impl SepData { let mut remaining = time::msecs_to_jiffies(ms); let mut guard = self.sks_probe.lock(); loop { - if guard.captured.len() >= until || remaining == 0 { + if self.shutting_down.load(Relaxed) + || guard.captured.len() >= until || remaining == 0 { return; } match self diff --git a/drivers/soc/apple/store.rs b/drivers/soc/apple/store.rs index dc37025ce3b8ad..03bd6f1bf0a01d 100644 --- a/drivers/soc/apple/store.rs +++ b/drivers/soc/apple/store.rs @@ -73,6 +73,20 @@ const SLOT_ENTRY_SIZE: usize = 24; const INTENT_NONE: u8 = 0; const INTENT_WRITE: u8 = 1; +/// Why an existing file is refused. It is logged, so it never carries record +/// contents. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Damage { + /// The file is shorter than the store. + Size, + /// The magic, version or geometry is not this format's. + Header, + /// The slot's record is longer than its block. + Slot(usize), + /// The pending write intent names a block that holds a committed record. + Intent, +} + // Not internally locked; the caller holds a mutex around all access. pub(crate) struct Store { file: shim::StoreFile, @@ -92,9 +106,34 @@ fn le64(buf: &[u8], off: usize) -> u64 { u64::from_le_bytes(v) } +/// Describes an existing file that holds no store as "empty" or "all zero". +/// That is all an interrupted creation leaves, and such a file holds nothing +/// to keep; anything else may be what is left of a store. Reads at most a +/// store's worth of the file. +fn describe(file: &shim::StoreFile, size: u64) -> Option<&'static str> { + if size == 0 { + return Some("empty"); + } + if size > STORE_SIZE as u64 { + return None; + } + let mut buf = KVec::new(); + buf.resize(BLOCK_SIZE, 0, GFP_KERNEL).ok()?; + let mut off = 0; + while off < size { + let len = (size - off).min(BLOCK_SIZE as u64) as usize; + file.read_exact(off, &mut buf[..len]).ok()?; + if buf[..len].iter().any(|&b| b != 0) { + return None; + } + off += len as u64; + } + Some("all zero") +} + impl Store { pub(crate) fn open() -> Result { - let file = shim::StoreFile::open(STORE_PATH)?; + let (file, created) = shim::StoreFile::open_host(STORE_PATH)?; let mut store = Store { file, @@ -104,11 +143,18 @@ impl Store { fresh: false, }; + // Only a new, empty file is initialised. An existing file may hold the + // only copy of the host state, so one that this driver cannot use is + // refused and left as it is. let size = store.file.size()?; - if size < STORE_SIZE as u64 { + if created && size == 0 { store.create()?; + pr_info!("created a new host-state store\n"); return Ok(store); } + if size < STORE_SIZE as u64 { + return Err(store.refuse(Damage::Size, size)); + } let mut sb = KVec::with_capacity(BLOCK_SIZE, GFP_KERNEL)?; sb.resize(BLOCK_SIZE, 0, GFP_KERNEL)?; @@ -119,8 +165,7 @@ impl Store { || le32(&sb, SB_SLOT_COUNT) as usize != SLOT_COUNT || le32(&sb, SB_BLOCK_SIZE) as usize != BLOCK_SIZE { - store.create()?; - return Ok(store); + return Err(store.refuse(Damage::Header, size)); } store.generation = le64(&sb, SB_GENERATION); @@ -134,6 +179,10 @@ impl Store { uuid, len: le32(&sb, off + 18), }; + // A longer record would be read from the blocks of the next slots. + if store.slots[i].used && store.slots[i].len as usize > MAX_VALUE { + return Err(store.refuse(Damage::Slot(i), size)); + } } // Crash recovery: writes are copy-on-write, so discard the intent, never complete it. @@ -142,6 +191,12 @@ impl Store { store.recovered = true; let slot = u16::from_le_bytes([sb[SB_INTENT_SLOT], sb[SB_INTENT_SLOT + 1]]) as usize; if intent_kind == INTENT_WRITE && slot < SLOT_COUNT { + // The intent names the free block that was being filled. One + // that names a committed record is damage, and freeing that + // block would discard the record. + if store.slots[slot].used { + return Err(store.refuse(Damage::Intent, size)); + } store.slots[slot] = Slot::FREE; } store.commit()?; @@ -150,6 +205,27 @@ impl Store { Ok(store) } + /// Logs why the existing file is refused, and returns the error. Nothing + /// is written to the file. + fn refuse(&self, damage: Damage, size: u64) -> Error { + let what = match damage { + Damage::Size | Damage::Header => describe(&self.file, size), + _ => None, + }; + match what { + Some(what) => pr_err!( + "host-state store failed validation ({:?}, {}), refusing it\n", + damage, + what + ), + None => pr_err!( + "host-state store failed validation ({:?}), refusing it\n", + damage + ), + } + EIO + } + fn create(&mut self) -> Result<()> { self.slots = [Slot::FREE; SLOT_COUNT]; self.generation = 1; diff --git a/drivers/soc/apple/store_shim.c b/drivers/soc/apple/store_shim.c index bdfd5dfbb6391d..4c89f92eb5f13b 100644 --- a/drivers/soc/apple/store_shim.c +++ b/drivers/soc/apple/store_shim.c @@ -51,6 +51,23 @@ void *sep_store_open(const char *path) return IS_ERR(f) ? NULL : f; } +/* + * As sep_store_open(), but reports whether this call created the file, so that + * only a new file is ever initialised. Returns an error pointer on failure. + */ +void *sep_store_open_host(const char *path, bool *created) +{ + struct file *f; + + *created = false; + f = open_as_kernel(path, O_RDWR | O_CREAT | O_EXCL | O_LARGEFILE, 0600); + if (PTR_ERR_OR_ZERO(f) == -EEXIST) + return open_as_kernel(path, O_RDWR | O_LARGEFILE, 0); + if (!IS_ERR(f)) + *created = true; + return f; +} + /* * Opens truncated to empty, for callers that rewrite the whole file each time, * so a shorter record cannot leave stale trailing bytes from a larger one. diff --git a/drivers/soc/apple/transfer.rs b/drivers/soc/apple/transfer.rs index 1f77f451625760..a4f5f0df3e308b 100644 --- a/drivers/soc/apple/transfer.rs +++ b/drivers/soc/apple/transfer.rs @@ -177,6 +177,10 @@ pub(crate) struct Reassembly { done: Option, sending: Option, grants: u32, + // Counts exchanges begun. A reply is matched only by its opcode, so a + // late reply to an exchange that timed out would otherwise complete the + // next exchange with the same opcode. + generation: u64, } impl Reassembly { @@ -186,14 +190,22 @@ impl Reassembly { done: None, sending: None, grants: 0, + generation: 0, } } + /// The exchange a message arriving now can belong to. Sampled before the + /// message is read and passed back to [`Self::on_chunk`]. + pub(crate) fn generation(&self) -> u64 { + self.generation + } + pub(crate) fn begin(&mut self, opcode: u32) -> Result<()> { if self.active.is_some() { return Err(EBUSY); } self.done = None; + self.generation = self.generation.wrapping_add(1); self.active = Some(Active { opcode, total: 0, @@ -242,7 +254,16 @@ impl Reassembly { self.abort(); } - pub(crate) fn on_chunk(&mut self, marker: u8, packet: &Packet, payload: &[u8]) -> Progress { + /// Takes one chunk of a reply. `generation` is [`Self::generation`] as + /// sampled when the message arrived: a chunk that arrived before the + /// current exchange began belongs to an earlier one and is dropped. + pub(crate) fn on_chunk( + &mut self, + generation: u64, + marker: u8, + packet: &Packet, + payload: &[u8], + ) -> Progress { if marker < MARKER_FIRST { return Progress::Notification; } @@ -262,6 +283,9 @@ impl Reassembly { let Some(active_opcode) = self.active.as_ref().map(|a| a.opcode) else { return Progress::Ignored; }; + if generation != self.generation { + return Progress::Ignored; + } // Checked before the opcode test below, deliberately. if marker == MARKER_ERROR { @@ -371,3 +395,60 @@ impl Reassembly { self.done = Some(Completed { status, payload }); } } + +#[kernel::macros::kunit_tests(apple_sep_transfer)] +mod tests { + use super::*; + + const OPCODE: u32 = 0x48; + + fn reply(err: u32) -> Packet { + Packet { + version: VERSION, + total: 0, + offset: 0, + flags: 0, + err, + opcode: OPCODE, + chunk: 0, + } + } + + #[test] + fn a_reply_to_the_current_exchange_completes_it() { + let mut rx = Reassembly::new(); + assert!(rx.begin(OPCODE).is_ok()); + let generation = rx.generation(); + assert!(matches!( + rx.on_chunk(generation, MARKER_FIRST, &reply(0), &[]), + Progress::Complete + )); + assert!(rx.take_done().is_some_and(|done| done.status.is_ok())); + } + + #[test] + fn a_late_reply_does_not_complete_the_next_exchange() { + let mut rx = Reassembly::new(); + assert!(rx.begin(OPCODE).is_ok()); + // The reply to the first exchange is being read when it times out. + let late = rx.generation(); + rx.abort(); + assert!(rx.take_done().is_some()); + // A retry begins the same opcode before the late reply is delivered. + assert!(rx.begin(OPCODE).is_ok()); + assert!(matches!( + rx.on_chunk(late, MARKER_FIRST, &reply(0), &[]), + Progress::Ignored + )); + assert!(matches!( + rx.on_chunk(late, MARKER_ERROR, &reply(0x16), &[]), + Progress::Ignored + )); + assert!(rx.take_done().is_none()); + let current = rx.generation(); + assert!(matches!( + rx.on_chunk(current, MARKER_FIRST, &reply(0), &[]), + Progress::Complete + )); + } +} diff --git a/drivers/spi/Kconfig b/drivers/spi/Kconfig index 67df03c7952975..f700acbcf8a168 100644 --- a/drivers/spi/Kconfig +++ b/drivers/spi/Kconfig @@ -137,9 +137,10 @@ config SPI_APPLE_QSPI This enables support for Apple's command-FIFO QSPIMC controller. Unlike the ordinary Apple SPIMC block, QSPIMC accepts packed SPI memory descriptors and is used for the boot NOR (and its NVRAM) - on T8122 (J613 25G83), T8132 (M4) and T8140 systems. The driver - defaults read-only. Only the pinned J613 NVRAM layout can enable - bounded writes through an explicit root runtime opt-in. + on T8122 (J613 25G83), T8132 (M4) and T8140 systems. On T8122 and + T8132 the driver defaults read-only, and only the pinned J613 NVRAM + layout can enable bounded writes through an explicit root runtime + opt-in. T8140 is not restricted. config SPI_AR934X tristate "Qualcomm Atheros AR934X/QCA95XX SPI controller driver" diff --git a/drivers/spi/spi-apple-qspi.c b/drivers/spi/spi-apple-qspi.c index 45b3566ae5a9f7..0367b63ae8aaee 100644 --- a/drivers/spi/spi-apple-qspi.c +++ b/drivers/spi/spi-apple-qspi.c @@ -7,11 +7,15 @@ * command FIFO. This driver intentionally implements the PIO spi-mem path * only; that is sufficient for SPI NOR discovery and access. * - * The boot NOR also holds iBoot and the system configuration, so the driver - * starts read-only: only identification, status and read opcodes reach the - * controller unless root explicitly clears read_only at runtime. Even then, only - * bounded J613 NVRAM page programs and 4 KiB erases are admitted. - * A power loss can interrupt a NOR update within the admitted range. + * On T8122 and T8132 the boot NOR also holds iBoot and the system + * configuration, so those controllers start read-only: only identification, + * status and read opcodes reach the controller unless root explicitly clears + * read_only at runtime. Even then, only bounded J613 NVRAM page programs and + * 4 KiB erases are admitted. A power loss can interrupt a NOR update within + * the admitted range. + * + * T8140 keeps unrestricted spi-mem access with dual and quad transfers; the + * read_only policy does not apply to it. */ #include @@ -25,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -128,7 +133,7 @@ static const struct kernel_param_ops apple_qspi_read_only_ops = { .get = apple_qspi_get_read_only, }; module_param_cb(read_only, &apple_qspi_read_only_ops, &read_only, 0644); -MODULE_PARM_DESC(read_only, "Default true; root runtime opt-in for bounded J613 NVRAM writes"); +MODULE_PARM_DESC(read_only, "Default true; root runtime opt-in for bounded J613 NVRAM writes (T8122/T8132 only)"); static bool apple_qspi_opcode_is_read(u8 opcode) { @@ -151,7 +156,47 @@ static bool apple_qspi_opcode_is_read(u8 opcode) } } +/* Per-compatible controller behaviour. */ +struct apple_qspi_hw { + /* + * Gate requests through the read_only/J613 NVRAM write policy. Without + * it, every supported spi-mem operation reaches the controller. + */ + bool nvram_policy; + /* + * Send TX-only operations as one payload stream: zero header counts and + * a length word that covers opcode, address and data. Otherwise the + * length word carries only the data length and the descriptor counts the + * opcode and address bytes. + */ + bool tx_single_stream; + /* Advertise dual and quad transfers to spi-mem. */ + bool multi_io; + /* + * Service the FIFOs by microsecond polling instead of once per jiffy + * while waiting for a transfer to complete. + */ + bool us_poll; + /* Refuse a requested clock below the divider range instead of clamping. */ + bool exact_clock; + /* Report the controller at probe at debug level only. */ + bool quiet_probe; +}; + +static const struct apple_qspi_hw apple_qspi_t8122_hw = { + .nvram_policy = true, + .tx_single_stream = true, +}; + +static const struct apple_qspi_hw apple_qspi_t8140_hw = { + .multi_io = true, + .us_poll = true, + .exact_clock = true, + .quiet_probe = true, +}; + struct apple_qspi { + const struct apple_qspi_hw *hw; /* Only the pinned J613 controller/partition may opt in. */ bool nvram_write_candidate; bool write_fault; @@ -424,12 +469,39 @@ static int apple_qspi_set_frequency(struct apple_qspi *qspi, return -EINVAL; divisor = DIV_ROUND_UP(input, requested); - divisor = clamp_t(u32, divisor, 2, 256); + if (qspi->hw->exact_clock) { + /* Never exceed the requested clock when it is below our range. */ + if (divisor > 256) + return -EINVAL; + divisor = max_t(u32, divisor, 2); + } else { + divisor = clamp_t(u32, divisor, 2, 256); + } apple_qspi_c_write(qspi, APPLE_QSPI_C_CLKDIV, divisor - 1); return 0; } -static bool apple_qspi_wait_complete(struct apple_qspi *qspi) +static bool apple_qspi_poll_complete(struct apple_qspi *qspi) +{ + unsigned long flags; + u32 queue_irqs, command_irqs; + + spin_lock_irqsave(&qspi->lock, flags); + queue_irqs = apple_qspi_p_read(qspi, APPLE_QSPI_P_IRQ_STATUS); + command_irqs = apple_qspi_c_read(qspi, APPLE_QSPI_C_IRQ_STATUS); + apple_qspi_p_write(qspi, APPLE_QSPI_P_IRQ_STATUS, queue_irqs); + apple_qspi_c_write(qspi, APPLE_QSPI_C_IRQ_STATUS, command_irqs); + if (qspi->active) { + apple_qspi_drain_rx(qspi); + apple_qspi_fill_tx(qspi); + } + spin_unlock_irqrestore(&qspi->lock, flags); + + return (command_irqs & APPLE_QSPI_C_IRQ_COMPLETE) || + try_wait_for_completion(&qspi->done); +} + +static bool apple_qspi_wait_complete_tick(struct apple_qspi *qspi) { unsigned long deadline = jiffies + msecs_to_jiffies(APPLE_QSPI_TIMEOUT_MS); @@ -472,6 +544,25 @@ static bool apple_qspi_wait_complete(struct apple_qspi *qspi) return false; } +static bool apple_qspi_wait_complete(struct apple_qspi *qspi) +{ + bool finished; + + if (!qspi->hw->us_poll) + return apple_qspi_wait_complete_tick(qspi); + + /* + * T8140 can latch completion without delivering the wired IRQ, and + * the RX threshold IRQ does not reliably drain the FIFO. A jiffy + * between polls throttles reads to one FIFO per scheduler tick. + * Service the FIFOs at microsecond intervals while retaining the + * bounded transfer deadline and IRQ completion path. + */ + return !read_poll_timeout(apple_qspi_poll_complete, finished, finished, + 20, APPLE_QSPI_TIMEOUT_MS * USEC_PER_MSEC, + false, qspi); +} + static int apple_qspi_transfer_op(struct spi_mem *mem, const struct spi_mem_op *op) { @@ -484,7 +575,7 @@ static int apple_qspi_transfer_op(struct spi_mem *mem, u32 descriptor, mode; u32 p_status, c_status; int ret, retries; - bool complete; + bool complete, tx_stream; ret = apple_qspi_shape_op(op, &shape); if (ret) @@ -521,10 +612,13 @@ static int apple_qspi_transfer_op(struct spi_mem *mem, mode = apple_qspi_width_mode(shape.width); if (op->data.dir == SPI_MEM_DATA_IN) mode |= APPLE_QSPI_CMD_RX; - /* TX-only transfers contain the opcode, address and payload in one - * byte stream. RX transfers keep the read header separate from data. + /* On tx_single_stream controllers, TX-only transfers contain the + * opcode, address and payload in one byte stream. RX transfers keep the + * read header separate from data. */ - if (op->data.dir != SPI_MEM_DATA_IN) { + tx_stream = qspi->hw->tx_single_stream && + op->data.dir != SPI_MEM_DATA_IN; + if (tx_stream) { shape.shared_count = 0; shape.multi_count = 0; } @@ -552,7 +646,7 @@ static int apple_qspi_transfer_op(struct spi_mem *mem, qspi->active = true; apple_qspi_p_write(qspi, APPLE_QSPI_P_TXFIFO, descriptor); apple_qspi_p_write(qspi, APPLE_QSPI_P_TXFIFO, - op->data.dir == SPI_MEM_DATA_IN ? op->data.nbytes : tx_len); + tx_stream ? tx_len : op->data.nbytes); apple_qspi_fill_tx(qspi); spin_unlock_irqrestore(&qspi->lock, flags); @@ -806,10 +900,20 @@ static int apple_qspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *op) return ret; } +/* Controllers without the NVRAM policy pass every supported request through. */ +static int apple_qspi_mem_exec_op(struct spi_mem *mem, const struct spi_mem_op *op) +{ + struct apple_qspi *qspi = spi_controller_get_devdata(mem->spi->controller); + + if (!qspi->hw->nvram_policy) + return apple_qspi_transfer_op(mem, op); + return apple_qspi_exec_op(mem, op); +} + static const struct spi_controller_mem_ops apple_qspi_mem_ops = { .adjust_op_size = apple_qspi_adjust_op_size, .supports_op = apple_qspi_supports_op, - .exec_op = apple_qspi_exec_op, + .exec_op = apple_qspi_mem_exec_op, }; /* Require the pinned partition, aperture and single physical chip select. */ @@ -862,6 +966,9 @@ static int apple_qspi_probe(struct platform_device *pdev) return -ENOMEM; qspi = spi_controller_get_devdata(ctlr); + qspi->hw = of_device_get_match_data(dev); + if (!qspi->hw) + return -ENODEV; qspi->nvram_write_candidate = of_device_is_compatible(dev->of_node, "apple,t8122-qspi"); if (qspi->nvram_write_candidate && !of_machine_is_compatible("apple,j613")) return -ENODEV; @@ -875,7 +982,9 @@ static int apple_qspi_probe(struct platform_device *pdev) qspi->nvram_write_candidate &= res->start == 0x2a1118000ULL && apple_qspi_nvram_layout(dev->of_node); - dev_info(dev, "NVRAM layout candidate: %u\n", qspi->nvram_write_candidate); + if (qspi->hw->nvram_policy) + dev_info(dev, "NVRAM layout candidate: %u\n", + qspi->nvram_write_candidate); qspi->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(qspi->base)) return PTR_ERR(qspi->base); @@ -909,10 +1018,16 @@ static int apple_qspi_probe(struct platform_device *pdev) qspi->tx_depth, qspi->rx_depth); apple_qspi_init_config(qspi); - /* Keep QE and every other flash configuration bit untouched. */ + /* + * Without multi_io, single-wire modes keep QE and every other flash + * configuration bit untouched. + */ ctlr->num_chipselect = 1; ctlr->max_native_cs = 1; ctlr->mode_bits = SPI_CPHA | SPI_CPOL; + if (qspi->hw->multi_io) + ctlr->mode_bits |= SPI_RX_DUAL | SPI_RX_QUAD | + SPI_TX_DUAL | SPI_TX_QUAD; ctlr->bits_per_word_mask = SPI_BPW_MASK(8); ctlr->mem_ops = &apple_qspi_mem_ops; @@ -926,9 +1041,14 @@ static int apple_qspi_probe(struct platform_device *pdev) runtime_ready = true; mutex_unlock(&apple_qspi_policy_lock); version = apple_qspi_p_read(qspi, APPLE_QSPI_P_VERSION); - dev_info(dev, "QSPIMC version %#x, %lu Hz clock, txq=%u rxq=%u%s\n", - version, clk_get_rate(qspi->clk), qspi->tx_depth, - qspi->rx_depth, read_only ? ", read-only" : ""); + if (qspi->hw->quiet_probe) + dev_dbg(dev, "QSPIMC version %#x, %lu Hz clock, txq=%u rxq=%u\n", + version, clk_get_rate(qspi->clk), qspi->tx_depth, + qspi->rx_depth); + else + dev_info(dev, "QSPIMC version %#x, %lu Hz clock, txq=%u rxq=%u%s\n", + version, clk_get_rate(qspi->clk), qspi->tx_depth, + qspi->rx_depth, read_only ? ", read-only" : ""); return 0; } @@ -988,9 +1108,9 @@ static struct attribute *apple_qspi_attrs[] = { ATTRIBUTE_GROUPS(apple_qspi); static const struct of_device_id apple_qspi_of_match[] = { - { .compatible = "apple,t8122-qspi" }, - { .compatible = "apple,t8132-qspi" }, - { .compatible = "apple,t8140-qspi" }, + { .compatible = "apple,t8122-qspi", .data = &apple_qspi_t8122_hw }, + { .compatible = "apple,t8132-qspi", .data = &apple_qspi_t8122_hw }, + { .compatible = "apple,t8140-qspi", .data = &apple_qspi_t8140_hw }, {} }; MODULE_DEVICE_TABLE(of, apple_qspi_of_match); diff --git a/drivers/spi/spi-apple.c b/drivers/spi/spi-apple.c index 21937bb50ba08d..f18cc6e1b8d0ee 100644 --- a/drivers/spi/spi-apple.c +++ b/drivers/spi/spi-apple.c @@ -112,10 +112,9 @@ #define APPLE_SPI_DELAY_CYCLES GENMASK(31, 16) /* - * These are the values used by AppleSPIMCController's hardware-delay path. - * The pre-delay asserts active-low CS and the post-delay releases it. The - * otherwise undocumented bit 11 is named for its observed role rather than - * assigning it a speculative electrical meaning. + * Hardware CS delays: the pre-delay asserts the active-low CS and the + * post-delay releases it. Bit 11 is undocumented and named for its observed + * role only. */ #define APPLE_SPI_DELAY_PRE_FLAGS (APPLE_SPI_DELAY_PRE_ARM | \ APPLE_SPI_DELAY_SET_CS | \ diff --git a/drivers/spmi/spmi-apple-controller.c b/drivers/spmi/spmi-apple-controller.c index ec49d412a4b0e8..d31bcf33a8a359 100644 --- a/drivers/spmi/spmi-apple-controller.c +++ b/drivers/spmi/spmi-apple-controller.c @@ -52,8 +52,32 @@ #define REG_POLL_INTERVAL_US 10000 #define REG_POLL_TIMEOUT_US (REG_POLL_INTERVAL_US * 5) +struct apple_spmi_hw { + u32 status; + u32 command; + u32 reply; + u32 rx_empty; + bool legacy_irqs; +}; + +static const struct apple_spmi_hw apple_spmi_gen1 = { + .status = SPMI_STATUS_REG, + .command = SPMI_CMD_REG, + .reply = SPMI_RSP_REG, + .rx_empty = SPMI_RX_FIFO_EMPTY, + .legacy_irqs = true, +}; + +static const struct apple_spmi_hw apple_spmi_gen4 = { + .status = 0x200, + .command = 0x210, + .reply = 0x220, + .rx_empty = BIT(30), +}; + struct apple_spmi { void __iomem *regs; + const struct apple_spmi_hw *hw; struct mutex fifo_lock; struct completion fifo_rx; struct irq_domain *irqd; @@ -64,10 +88,6 @@ struct apple_spmi { bool prev_fail; }; -#define poll_reg(spmi, reg, val, cond) \ - readl_poll_timeout((spmi)->regs + (reg), (val), (cond), \ - REG_POLL_INTERVAL_US, REG_POLL_TIMEOUT_US) - static void apple_spmi_irq_ack_raw(struct apple_spmi *spmi, u32 irq) { u32 __iomem *reg = spmi->regs + SPMI_IRQ_ACK_BASE + (irq / 32) * 4; @@ -135,12 +155,14 @@ static int apple_spmi_wait_rx_not_empty(struct spmi_controller *ctrl) usecs_to_jiffies(REG_POLL_TIMEOUT_US)); if (!ret) ret = -ETIMEDOUT; - else if (readl(spmi->regs + SPMI_STATUS_REG) & SPMI_RX_FIFO_EMPTY) + else if (readl(spmi->regs + spmi->hw->status) & spmi->hw->rx_empty) ret = -EIO; else ret = 0; } else { - ret = poll_reg(spmi, SPMI_STATUS_REG, status, !(status & SPMI_RX_FIFO_EMPTY)); + ret = readl_poll_timeout(spmi->regs + spmi->hw->status, status, + !(status & spmi->hw->rx_empty), 10, + REG_POLL_TIMEOUT_US); } if (ret) { @@ -166,19 +188,22 @@ static int spmi_raw_cmd(struct spmi_controller *ctrl, u8 opc, u8 sid, guard(mutex)(&spmi->fifo_lock); if (spmi->prev_fail) { - writel(SPMI_ACT_FIFO_FLUSH, spmi->regs + SPMI_RSP_REG); + /* Do not consume a late reply as the reply to a new command. */ + if (!spmi->hw->legacy_irqs) + return -EIO; + writel(SPMI_ACT_FIFO_FLUSH, spmi->regs + SPMI_ACT_REG); apple_spmi_irq_ack_raw(spmi, SPMI_IRQ_NOTIFY); spmi->prev_fail = false; } reinit_completion(&spmi->fifo_rx); - writel(spmi_cmd, spmi->regs + SPMI_CMD_REG); + writel(spmi_cmd, spmi->regs + spmi->hw->command); while (i < len) { j = min_t(size_t, sizeof(spmi_cmd), len - i); spmi_cmd = 0; memcpy(&spmi_cmd, buf + i, j); - writel(spmi_cmd, spmi->regs + SPMI_CMD_REG); + writel(spmi_cmd, spmi->regs + spmi->hw->command); i += j; } @@ -186,23 +211,23 @@ static int spmi_raw_cmd(struct spmi_controller *ctrl, u8 opc, u8 sid, if (ret) return ret; - reply = readl(spmi->regs + SPMI_RSP_REG); + reply = readl(spmi->regs + spmi->hw->reply); /* Read SPMI data reply */ while (len_read < ilen) { - if (readl(spmi->regs + SPMI_STATUS_REG) & SPMI_RX_FIFO_EMPTY) { + if (readl(spmi->regs + spmi->hw->status) & spmi->hw->rx_empty) { spmi->prev_fail = true; dev_err_ratelimited(&ctrl->dev, "FIFO lacks reply data, controller stuck?\n"); return -EIO; } - rsp = readl(spmi->regs + SPMI_RSP_REG); + rsp = readl(spmi->regs + spmi->hw->reply); i = min_t(size_t, sizeof(spmi_cmd), ilen - len_read); memcpy(ibuf + len_read, &rsp, i); len_read += i; } - if (!(readl(spmi->regs + SPMI_STATUS_REG) & SPMI_RX_FIFO_EMPTY)) { + if (!(readl(spmi->regs + spmi->hw->status) & spmi->hw->rx_empty)) { dev_warn(&ctrl->dev, "FIFO has extra data\n"); spmi->prev_fail = true; } @@ -441,6 +466,9 @@ static int apple_spmi_probe(struct platform_device *pdev) return -ENOMEM; spmi = spmi_controller_get_drvdata(ctrl); + spmi->hw = device_get_match_data(&pdev->dev); + if (!spmi->hw) + return -EINVAL; mutex_init(&spmi->fifo_lock); init_completion(&spmi->fifo_rx); platform_set_drvdata(pdev, spmi); @@ -459,6 +487,9 @@ static int apple_spmi_probe(struct platform_device *pdev) if (spmi->irq < 0 && spmi->irq != -ENXIO) return spmi->irq; if (spmi->irq >= 0) { + if (!spmi->hw->legacy_irqs) + return dev_err_probe(&pdev->dev, -EOPNOTSUPP, + "Generation 4 interrupt layout is not supported\n"); ret = apple_spmi_init_irq(pdev, spmi, spmi->irq); if (ret) return ret; @@ -473,8 +504,9 @@ static int apple_spmi_probe(struct platform_device *pdev) } static const struct of_device_id apple_spmi_match_table[] = { - { .compatible = "apple,t8103-spmi", }, - { .compatible = "apple,spmi", }, + { .compatible = "apple,t8142-spmi", .data = &apple_spmi_gen4 }, + { .compatible = "apple,t8103-spmi", .data = &apple_spmi_gen1 }, + { .compatible = "apple,spmi", .data = &apple_spmi_gen1 }, {} }; MODULE_DEVICE_TABLE(of, apple_spmi_match_table); diff --git a/drivers/thunderbolt/apple.c b/drivers/thunderbolt/apple.c index 1f3b1ba568b86d..a099e5a8101509 100644 --- a/drivers/thunderbolt/apple.c +++ b/drivers/thunderbolt/apple.c @@ -125,6 +125,14 @@ static const struct apple_cio_hw apple_cio_t6030_hw = { .dp = &apple_dpin_m3, }; +/* + * T8142 (M5) Type 7 ACIO: ready is checked by acio->type7. Thunderbolt DP + * tunnels are not driven on this SoC yet. + */ +static const struct apple_cio_hw apple_cio_t8142_hw = { + .dp = &apple_dpin_disabled, +}; + #define APPLE_CIO_NHI_HOP_COUNT 0x0 #define APPLE_CIO_NHI_HOP_COUNT_MASK GENMASK(9, 0) @@ -194,6 +202,7 @@ struct apple_dpin_binding; struct apple_cio { struct device *dev; + bool type7; struct device_node *np; const struct apple_cio_hw *hw; const struct apple_dpin_policy *dp; @@ -1151,7 +1160,9 @@ static void apple_nhi_ring_configure(struct tb_ring *ring, u32 flags, u32 e2e_fl sof_eof_mask = ring->sof_mask << 16 | ring->eof_mask; writel(sof_eof_mask, options + 4); - writel(sof_eof_mask, anhi->pdf_base + ring->hop * APPLE_CIO_NHI_PDF_STRIDE); + if (anhi->pdf_base) + writel(sof_eof_mask, + anhi->pdf_base + ring->hop * APPLE_CIO_NHI_PDF_STRIDE); } /* @@ -1906,6 +1917,12 @@ static int apple_nhi_probe(struct platform_device *pdev) int cap_apple; int ret = 0; + if (acio->type7 != + of_device_is_compatible(pdev->dev.of_node, "apple,t8142-usb4-nhi")) { + ret = dev_err_probe(&pdev->dev, -EINVAL, "ACIO/NHI generation mismatch\n"); + goto err; + } + anhi = devm_kzalloc(&pdev->dev, sizeof(*anhi), GFP_KERNEL); if (!anhi) { ret = -ENOMEM; @@ -1921,6 +1938,13 @@ static int apple_nhi_probe(struct platform_device *pdev) INIT_DELAYED_WORK(&anhi->dp_aux_work, apple_dp_aux_work); platform_set_drvdata(pdev, anhi); + if (acio->type7 && + of_property_count_u32_elems(anhi->np, "apple,tunable-nhi") <= 0) { + ret = dev_err_probe(anhi->dev, -EINVAL, + "Missing bootloader NHI tuning\n"); + goto err; + } + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "nhi"); anhi->nhi_base = devm_ioremap_resource(&pdev->dev, res); if (IS_ERR(anhi->nhi_base)) { @@ -1935,12 +1959,15 @@ static int apple_nhi_probe(struct platform_device *pdev) } apple_tunable_apply(anhi->nhi_base, tunable); - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "pdf"); - anhi->pdf_base = devm_ioremap_resource(&pdev->dev, res); - if (IS_ERR(anhi->pdf_base)) { - ret = dev_err_probe(&pdev->dev, PTR_ERR(anhi->pdf_base), - "Unable to map PDF regs\n"); - goto err; + /* Type 7 keeps the PDF masks in the receive ring registers only. */ + if (!acio->type7) { + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "pdf"); + anhi->pdf_base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(anhi->pdf_base)) { + ret = dev_err_probe(&pdev->dev, PTR_ERR(anhi->pdf_base), + "Unable to map PDF regs\n"); + goto err; + } } ret = apple_nhi_probe_irqs(anhi); @@ -2283,6 +2310,9 @@ static const struct dev_pm_ops apple_nhi_pm_ops = { }; static const struct of_device_id apple_nhi_match[] = { + { + .compatible = "apple,t8142-usb4-nhi", + }, { .compatible = "apple,t8103-usb4-nhi", }, @@ -2529,6 +2559,12 @@ static int apple_cio_start(struct apple_cio *acio) FIELD_GET(APPLE_CIO_M3_STAT_STATE, state) == APPLE_CIO_M3_STAT_STATE_READY, 100, 500000); + else if (acio->type7) + /* Type 7 reports ready as state 1; other nonzero states are not ready. */ + ret = readl_poll_timeout(acio->rc_base + APPLE_CIO_M3_STAT, state, + FIELD_GET(APPLE_CIO_M3_STAT_STATE, state) == + APPLE_CIO_M3_STAT_STATE_READY, + 100, 500000); else ret = readl_poll_timeout(acio->rc_base + APPLE_CIO_M3_STAT, state, state & APPLE_CIO_M3_STAT_STATE, 100, 500000); @@ -3007,6 +3043,11 @@ static int apple_cio_probe(struct platform_device *pdev) return dev_err_probe(dev, PTR_ERR(acio->cpu_base), "Unable to map MxWrap cpu regs\n"); } + acio->type7 = of_device_is_compatible(acio->np, "apple,t8142-usb4-acio"); + if (acio->type7 && + of_property_count_u32_elems(acio->np, "apple,tunable-rc") <= 0) + return dev_err_probe(dev, -EINVAL, "Missing bootloader RC tuning\n"); + acio->pcie_tunnel_np = of_parse_phandle(dev->of_node, "apple,pcie-tunnel", 0); if (acio->pcie_tunnel_np) { @@ -3180,6 +3221,10 @@ static void apple_cio_remove(struct platform_device *pdev) } static const struct of_device_id apple_acio_match[] = { + { + .compatible = "apple,t8142-usb4-acio", + .data = &apple_cio_t8142_hw, + }, { .compatible = "apple,t6030-usb4-acio", .data = &apple_cio_t6030_hw, diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c index 6f3c00b089371c..498325fa1001da 100644 --- a/drivers/usb/dwc3/core.c +++ b/drivers/usb/dwc3/core.c @@ -610,6 +610,7 @@ void dwc3_event_buffers_cleanup(struct dwc3 *dwc) reg = dwc3_readl(dwc, DWC3_GEVNTCOUNT(0)); dwc3_writel(dwc, DWC3_GEVNTCOUNT(0), reg); } +EXPORT_SYMBOL_GPL(dwc3_event_buffers_cleanup); static void dwc3_core_num_eps(struct dwc3 *dwc) { @@ -995,14 +996,20 @@ static void dwc3_clk_disable(struct dwc3 *dwc) clk_disable_unprepare(dwc->bus_clk); } -void dwc3_core_exit(struct dwc3 *dwc) +void dwc3_core_exit_after_phy_power_off(struct dwc3 *dwc) { - dwc3_event_buffers_cleanup(dwc); - dwc3_phy_power_off(dwc); dwc3_phy_exit(dwc); dwc3_clk_disable(dwc); reset_control_assert(dwc->reset); } +EXPORT_SYMBOL_GPL(dwc3_core_exit_after_phy_power_off); + +void dwc3_core_exit(struct dwc3 *dwc) +{ + dwc3_event_buffers_cleanup(dwc); + dwc3_phy_power_off(dwc); + dwc3_core_exit_after_phy_power_off(dwc); +} EXPORT_SYMBOL_GPL(dwc3_core_exit); static bool dwc3_core_is_valid(struct dwc3 *dwc) @@ -2410,14 +2417,16 @@ static int dwc3_probe(struct platform_device *pdev) return dwc3_core_probe(&probe_data); } -void dwc3_core_remove(struct dwc3 *dwc) +static void dwc3_core_remove_common(struct dwc3 *dwc, bool core_exited) { pm_runtime_get_sync(dwc->dev); - dwc3_core_exit_mode(dwc); + if (!core_exited) + dwc3_core_exit_mode(dwc); dwc3_debugfs_exit(dwc); - dwc3_core_exit(dwc); + if (!core_exited) + dwc3_core_exit(dwc); dwc3_ulpi_exit(dwc); pm_runtime_allow(dwc->dev); @@ -2438,8 +2447,19 @@ void dwc3_core_remove(struct dwc3 *dwc) power_supply_put(dwc->usb_psy); } } + +void dwc3_core_remove(struct dwc3 *dwc) +{ + dwc3_core_remove_common(dwc, false); +} EXPORT_SYMBOL_GPL(dwc3_core_remove); +void dwc3_core_remove_no_shutdown(struct dwc3 *dwc) +{ + dwc3_core_remove_common(dwc, true); +} +EXPORT_SYMBOL_GPL(dwc3_core_remove_no_shutdown); + static void dwc3_remove(struct platform_device *pdev) { dwc3_core_remove(platform_get_drvdata(pdev)); diff --git a/drivers/usb/dwc3/dwc3-apple.c b/drivers/usb/dwc3/dwc3-apple.c index 232a4de95bb75a..c8fc03421051f0 100644 --- a/drivers/usb/dwc3/dwc3-apple.c +++ b/drivers/usb/dwc3/dwc3-apple.c @@ -596,9 +596,12 @@ static int dwc3_apple_probe(struct platform_device *pdev) scoped_guard(mutex, &appledwc->lock) ret = dwc3_apple_init(appledwc, DWC3_APPLE_HOST); if (ret) { - /* dwc3_apple_init() has already undone the core initialisation */ + /* + * dwc3_apple_init() has already undone the core initialisation: + * release the core's resources without a second shutdown. + */ if (appledwc->state != DWC3_APPLE_PROBE_PENDING) - dwc3_core_remove(&appledwc->dwc); + dwc3_core_remove_no_shutdown(&appledwc->dwc); usb_role_switch_unregister(appledwc->role_sw); return dev_err_probe(dev, ret, "Failed to start host mode\n"); } @@ -621,7 +624,15 @@ static void dwc3_apple_remove(struct platform_device *pdev) * dwc3_apple_exit can be called unconditionally because it checks the state itself. */ dwc3_apple_exit(appledwc); - if (appledwc->state != DWC3_APPLE_PROBE_PENDING) + if (appledwc->state == DWC3_APPLE_PROBE_PENDING) + return; + /* + * dwc3_apple_exit() has stopped the host and the core of a fixed-hub + * controller: release its resources without running either exit twice. + */ + if (appledwc->fixed_hub) + dwc3_core_remove_no_shutdown(&appledwc->dwc); + else dwc3_core_remove(&appledwc->dwc); } diff --git a/drivers/usb/dwc3/glue.h b/drivers/usb/dwc3/glue.h index d738e1739ae015..db5f32df712bf8 100644 --- a/drivers/usb/dwc3/glue.h +++ b/drivers/usb/dwc3/glue.h @@ -108,6 +108,26 @@ int dwc3_core_init(struct dwc3 *dwc); */ void dwc3_core_exit(struct dwc3 *dwc); +/** + * dwc3_core_exit_after_phy_power_off - Finish a glue-owned core shutdown + * @dwc: Pointer to DWC3 controller context + * + * The glue must first stop host/gadget operation, clean up event buffers, + * release every core-owned generic PHY power reference and suspend legacy + * USB PHYs. This finishes PHY exit, clock disable and reset exactly once. + */ +void dwc3_core_exit_after_phy_power_off(struct dwc3 *dwc); + +/** + * dwc3_core_remove_no_shutdown - Release resources without a second shutdown + * @dwc: Pointer to DWC3 controller context + * + * The glue owns host/gadget and core shutdown, including reporting any + * failure to finish it. This releases software resources without repeating + * those operations or discarding another generic PHY reference. + */ +void dwc3_core_remove_no_shutdown(struct dwc3 *dwc); + /** * dwc3_host_init - Initialize host mode operation * @dwc: Pointer to DWC3 controller context diff --git a/drivers/usb/typec/mux.c b/drivers/usb/typec/mux.c index a7cf8c46c541e8..70023d7342f892 100644 --- a/drivers/usb/typec/mux.c +++ b/drivers/usb/typec/mux.c @@ -398,24 +398,83 @@ void typec_mux_put(struct typec_mux *mux) } EXPORT_SYMBOL_GPL(typec_mux_put); +/* Abort, in reverse, the providers whose prepare stage was reached. */ +static void typec_mux_abort(struct typec_mux *mux, unsigned int prepared, + struct typec_mux_state *state) +{ + struct typec_mux_dev *mux_dev; + + while (prepared--) { + mux_dev = mux->mux_devs[prepared]; + if (mux_dev->abort) + mux_dev->abort(mux_dev, state); + } +} + +/** + * typec_mux_set - Set the state of the muxes of a connector + * @mux: USB Type-C Connector Multiplexer/DeMultiplexer + * @state: New state + * + * Sets every mux of the connector. Providers that register transaction + * phases (struct typec_mux_desc) take part in one transaction: every + * provider's prepare runs before any provider's set, and every finish, then + * every commit, after all of them. A failure in prepare, set or finish calls + * abort, in reverse order, on the providers whose prepare stage was reached. + * A provider that registers only set is just set, in connection order, as + * before. + */ int typec_mux_set(struct typec_mux *mux, struct typec_mux_state *state) { struct typec_mux_dev *mux_dev; + unsigned int prepared = 0; unsigned int i; int ret; if (IS_ERR_OR_NULL(mux)) return 0; + /* Quiesce every consumer before any provider changes the route. */ + for (i = 0; i < mux->num_mux_devs; i++) { + mux_dev = mux->mux_devs[i]; + prepared = i + 1; + if (!mux_dev->prepare) + continue; + ret = mux_dev->prepare(mux_dev, state); + if (ret) + goto abort; + } + for (i = 0; i < mux->num_mux_devs; i++) { mux_dev = mux->mux_devs[i]; ret = mux_dev->set(mux_dev, state); if (ret && ret != -EOPNOTSUPP) - return ret; + goto abort; + } + + /* Finalizers can fail, but must not release their reservation. */ + for (i = 0; i < mux->num_mux_devs; i++) { + mux_dev = mux->mux_devs[i]; + if (!mux_dev->finish) + continue; + ret = mux_dev->finish(mux_dev, state); + if (ret) + goto abort; + } + + /* Only this phase, which cannot fail, may release a reservation. */ + for (i = 0; i < mux->num_mux_devs; i++) { + mux_dev = mux->mux_devs[i]; + if (mux_dev->commit) + mux_dev->commit(mux_dev, state); } return 0; + +abort: + typec_mux_abort(mux, prepared, state); + return ret; } EXPORT_SYMBOL_GPL(typec_mux_set); @@ -453,6 +512,10 @@ typec_mux_register(struct device *parent, const struct typec_mux_desc *desc) return ERR_PTR(-ENOMEM); mux_dev->set = desc->set; + mux_dev->prepare = desc->prepare; + mux_dev->finish = desc->finish; + mux_dev->commit = desc->commit; + mux_dev->abort = desc->abort; device_initialize(&mux_dev->dev); mux_dev->dev.parent = parent; diff --git a/drivers/usb/typec/mux.h b/drivers/usb/typec/mux.h index 0f1547fdbd11ae..b055705a03ffad 100644 --- a/drivers/usb/typec/mux.h +++ b/drivers/usb/typec/mux.h @@ -15,6 +15,10 @@ struct typec_switch_dev { struct typec_mux_dev { struct device dev; typec_mux_set_fn_t set; + typec_mux_set_fn_t prepare; + typec_mux_set_fn_t finish; + typec_mux_notify_fn_t commit; + typec_mux_notify_fn_t abort; }; #define to_typec_switch_dev(_dev_) container_of(_dev_, struct typec_switch_dev, dev) diff --git a/drivers/usb/typec/tipd/Kconfig b/drivers/usb/typec/tipd/Kconfig index 2f73bb17ed352b..a447a4622ea765 100644 --- a/drivers/usb/typec/tipd/Kconfig +++ b/drivers/usb/typec/tipd/Kconfig @@ -2,6 +2,7 @@ config TYPEC_TPS6598X_CORE tristate select USB_ROLE_SWITCH select POWER_SUPPLY + select USB_COMMON if DEBUG_FS config TYPEC_TPS6598X tristate "TI TPS6598x USB Power Delivery controller driver" diff --git a/drivers/usb/typec/tipd/Makefile b/drivers/usb/typec/tipd/Makefile index c0e850a56f2896..1a7605e7e57ecc 100644 --- a/drivers/usb/typec/tipd/Makefile +++ b/drivers/usb/typec/tipd/Makefile @@ -4,6 +4,7 @@ CFLAGS_trace.o := -I$(src) obj-$(CONFIG_TYPEC_TPS6598X_CORE) += tps6598x-core.o tps6598x-core-y := core.o tps6598x-core-$(CONFIG_TRACING) += trace.o +tps6598x-core-$(CONFIG_DEBUG_FS) += debugfs.o obj-$(CONFIG_TYPEC_TPS6598X) += tps6598x.o tps6598x-y := i2c.o diff --git a/drivers/usb/typec/tipd/cd321x-pm-test.c b/drivers/usb/typec/tipd/cd321x-pm-test.c index 92b2b9cbeed2cc..0e86ff1f42bd13 100644 --- a/drivers/usb/typec/tipd/cd321x-pm-test.c +++ b/drivers/usb/typec/tipd/cd321x-pm-test.c @@ -754,6 +754,22 @@ static void cd321x_restore_ignores_fault_budget_test(struct kunit *test) KUNIT_EXPECT_EQ(test, cd321x_pm_notifications(&pm, false, false, true), 0U); } +/* Only a pipeline owned by another port skips the retry. */ +static void cd321x_display_taken_test(struct kunit *test) +{ + /* T8140/T8142: a PHY mid-transition or a pending SAFE state is retried. */ + KUNIT_EXPECT_TRUE(test, cd321x_display_taken(-ENOSPC, true)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(-EBUSY, true)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(-EAGAIN, true)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(-ETIMEDOUT, true)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(0, true)); + /* Other SoCs keep -EBUSY as the taken display. */ + KUNIT_EXPECT_TRUE(test, cd321x_display_taken(-EBUSY, false)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(-ENOSPC, false)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(-EAGAIN, false)); + KUNIT_EXPECT_FALSE(test, cd321x_display_taken(0, false)); +} + static struct kunit_case cd321x_pm_cases[] = { KUNIT_CASE(cd321x_reset_first_irq_before_ready_test), KUNIT_CASE(cd321x_reset_first_irq_during_prepare_test), @@ -790,6 +806,7 @@ static struct kunit_case cd321x_pm_cases[] = { KUNIT_CASE(cd321x_status_event_after_exhaustion_test), KUNIT_CASE(cd321x_runtime_failure_revalidates_test), KUNIT_CASE(cd321x_restore_ignores_fault_budget_test), + KUNIT_CASE(cd321x_display_taken_test), {} }; diff --git a/drivers/usb/typec/tipd/cd321x-pm.h b/drivers/usb/typec/tipd/cd321x-pm.h index de35c14b7b6b75..7f7cae8166e776 100644 --- a/drivers/usb/typec/tipd/cd321x-pm.h +++ b/drivers/usb/typec/tipd/cd321x-pm.h @@ -2,6 +2,7 @@ #ifndef __CD321X_PM_H__ #define __CD321X_PM_H__ +#include #include #define CD321X_RESUME_ATTEMPTS 3 @@ -211,4 +212,17 @@ static inline void cd321x_pm_remove(struct cd321x_pm_state *pm) pm->attempts_left = 0; } +/* + * Whether a display mux error means that every display pipeline drives + * another port. That lasts until the other display goes away, so the port's + * USB path is set up without the display and nothing is retried. T8140 and + * T8142 display routes report it as -ENOSPC (@enospc), and their PHY returns + * -EBUSY while a transition is pending, which must be retried. Elsewhere the + * display route reports it as -EBUSY. + */ +static inline bool cd321x_display_taken(int ret, bool enospc) +{ + return ret == (enospc ? -ENOSPC : -EBUSY); +} + #endif /* __CD321X_PM_H__ */ diff --git a/drivers/usb/typec/tipd/core.c b/drivers/usb/typec/tipd/core.c index b228a7b363a084..6c90eb3221adf0 100644 --- a/drivers/usb/typec/tipd/core.c +++ b/drivers/usb/typec/tipd/core.c @@ -29,7 +29,6 @@ /* Register offsets */ #define TPS_REG_VID 0x00 -#define TPS_REG_MODE 0x03 #define TPS_REG_CMD1 0x08 #define TPS_REG_DATA1 0x09 #define TPS_REG_VERSION 0x0F @@ -93,10 +92,6 @@ struct tps6598x_rx_identity_reg { struct usb_pd_identity identity; } __packed; -/* Standard Task return codes */ -#define TPS_TASK_TIMEOUT 1 -#define TPS_TASK_REJECTED 3 - /* Debounce delay for mode changes, in milliseconds */ #define CD321X_DEBOUNCE_DELAY_MS 500 @@ -106,6 +101,7 @@ enum { TPS_MODE_BIST, TPS_MODE_DISC, TPS_MODE_PTCH, + TPS_MODE_DBMA, }; static const char *const modes[] = { @@ -114,6 +110,7 @@ static const char *const modes[] = { [TPS_MODE_BIST] = "BIST", [TPS_MODE_DISC] = "DISC", [TPS_MODE_PTCH] = "PTCH", + [TPS_MODE_DBMA] = "DBMa", }; /* Unrecognized commands will be replaced with "!CMD" */ @@ -126,14 +123,7 @@ static enum power_supply_property tps6598x_psy_props[] = { static const char *tps6598x_psy_name_prefix = "tps6598x-source-psy-"; -/* - * Max data bytes for Data1, Data2, and other registers. See ch 1.3.2: - * https://www.ti.com/lit/ug/slvuan1a/slvuan1a.pdf - */ -#define TPS_MAX_LEN 64 - -static int -tps6598x_block_read(struct tps6598x *tps, u8 reg, void *val, size_t len) +int tps6598x_block_read(struct tps6598x *tps, u8 reg, void *val, size_t len) { u8 data[TPS_MAX_LEN + 1]; int ret; @@ -299,10 +289,10 @@ static void tps6598x_disconnect(struct tps6598x *tps, u32 status) power_supply_changed(tps->psy); } -static int tps6598x_exec_cmd_tmo(struct tps6598x *tps, const char *cmd, - size_t in_len, const u8 *in_data, - size_t out_len, u8 *out_data, - u32 cmd_timeout_ms, u32 res_delay_ms) +int tps6598x_exec_cmd_tmo(struct tps6598x *tps, const char *cmd, + size_t in_len, const u8 *in_data, + size_t out_len, u8 *out_data, + u32 cmd_timeout_ms, u32 res_delay_ms) { unsigned long timeout; u32 val; @@ -552,6 +542,20 @@ static void tps6598x_handle_plug_event(struct tps6598x *tps, u32 status) } } +/* + * The interrupt mask of this controller. T8140 and T8142 leave the SN201202x + * DP status event masked: the display handling these machines were brought + * up with never used it (see h17_display). + */ +static u64 tipd_irq_mask1(const struct tps6598x *tps) +{ + u64 mask = tps->data->irq_mask1; + + if (tps->h17_display) + mask &= ~SN201202X_INT_DP_SID_UPDATE; + return mask; +} + /* * The HPD level of a DisplayPort partner. DATA_STATUS reports it. An * SN201202x also signals every new DP status of the partner, and a DP-to-HDMI @@ -700,7 +704,19 @@ static int cd321x_typec_update_mode(struct tps6598x *tps, else dp_data.status &= ~DP_STATUS_HPD_STATE; + /* + * The sink dropped HPD and raised it again within the debounce + * window, as a monitor does after losing the link. The coalesced + * state looks unchanged, so the display side would never hear of + * it: report the pulse as IRQ_HPD. + */ + if ((st->data_status_changed & CD321X_DATA_STATUS_HPD_LEVEL) && + (dp_data.status & DP_STATUS_HPD_STATE) && + (cd321x->dp_status & DP_STATUS_HPD_STATE)) + dp_data.status |= DP_STATUS_IRQ_HPD; + if (cd321x->state_valid && cd321x->state.alt == cd321x->port_altmode_dp && + !(dp_data.status & DP_STATUS_IRQ_HPD) && cd321x->state.mode == mode && cd321x->dp_status == dp_data.status && cd321x->dp_conf == dp_data.conf) @@ -713,7 +729,7 @@ static int cd321x_typec_update_mode(struct tps6598x *tps, if (ret) goto out; ret = typec_mux_set(cd321x->mux, &cd321x->state); - *display_busy = ret == -EBUSY; + *display_busy = cd321x_display_taken(ret, tps->h17_display); if (!ret) { cd321x->dp_status = dp_data.status; cd321x->dp_conf = dp_data.conf; @@ -791,7 +807,13 @@ static int cd321x_dp_release_first(struct cd321x *cd321x) struct typec_mux_state state; int ret; - if (cd321x->tps.data != &tipd_sn201202x_data || !cd321x->display_route_active || + /* + * T8140/T8142 keep the teardown they were brought up with: there the + * display route releases the display itself, in the prepare phase of + * the SAFE transaction, before the PHY changes. + */ + if (cd321x->tps.data != &tipd_sn201202x_data || cd321x->tps.h17_display || + !cd321x->display_route_active || cd321x->state.alt != cd321x->port_altmode_dp || !(cd321x->dp_status & DP_STATUS_HPD_STATE)) return 0; @@ -935,10 +957,11 @@ static void cd321x_update_work(struct work_struct *work) * Every Type-C port references the same external DCP connector. Only the * port which successfully acquired the shared display route may report a * disconnect. A second DP sink can negotiate Alt Mode even though its mux - * request is rejected with -EBUSY; treating that sink as the connector - * owner would tear down the display carried by the first port. + * request is rejected with -EBUSY (-ENOSPC on T8140/T8142); treating that + * sink as the connector owner would tear down the display carried by the + * first port. */ - if (cd321x->connector_fwnode && + if (IS_REACHABLE(CONFIG_DRM) && cd321x->connector_fwnode && dp_route_was_active && (!new_connected || was_disconnected || !dp_connected || !dp_hpd || dp_hpd_changed || dp_mode_changed)) { @@ -1033,7 +1056,8 @@ static void cd321x_update_work(struct work_struct *work) if (ret) goto retry; - if (cd321x->connector_fwnode && cd321x->display_route_active && dp_hpd) + if (IS_REACHABLE(CONFIG_DRM) && cd321x->connector_fwnode && + cd321x->display_route_active && dp_hpd) drm_connector_oob_hotplug_event(cd321x->connector_fwnode, connector_status_connected); power_supply_changed(tps->psy); @@ -1237,9 +1261,9 @@ static int cd321x_check_setup(struct cd321x *cd321x, u64 event) lockdep_assert_held(&tps->lock); if (cd321x->pm.phase == CD321X_PM_REMOVED || - (!cd321x->pm.setup_pending && !(event & ~tps->data->irq_mask1))) + (!cd321x->pm.setup_pending && !(event & ~tipd_irq_mask1(tps)))) return 0; - ret = cd321x_setup_run(&cd321x->pm, event, tps->data->irq_mask1, + ret = cd321x_setup_run(&cd321x->pm, event, tipd_irq_mask1(tps), &cd321x_setup_ops, cd321x); if (cd321x->pm.phase == CD321X_PM_PREPARED || cd321x->pm.phase == CD321X_PM_INITIALIZING) @@ -1302,10 +1326,10 @@ static irqreturn_t cd321x_interrupt(int irq, void *data) } if (event & (APPLE_CD_REG_INT_DATA_STATUS_UPDATE | - (tps->data->irq_mask1 & SN201202X_INT_DP_SID_UPDATE))) + (tipd_irq_mask1(tps) & SN201202X_INT_DP_SID_UPDATE))) if (!tps->data->read_data_status(tps)) goto err_unlock; - if (event & tps->data->irq_mask1 & SN201202X_INT_DP_SID_UPDATE) + if (event & tipd_irq_mask1(tps) & SN201202X_INT_DP_SID_UPDATE) cd321x_dp_sid_event(container_of(tps, struct cd321x, tps)); /* Can be called uncondtionally since it will check for any changes itself */ @@ -1473,6 +1497,14 @@ static int tps6598x_check_mode(struct tps6598x *tps) case TPS_MODE_BOOT: dev_warn(tps->dev, "dead-battery condition\n"); return ret; + case TPS_MODE_DBMA: + /* + * Apple firmware only: the application firmware runs in its + * debug mode (entered with LOCK + "DBMa"), in which it still + * handles the port and additionally accepts VDMs/DVEn. A boot + * loader arming Apple Debug USB on this port leaves it here. + */ + return ret; case TPS_MODE_BIST: case TPS_MODE_DISC: default: @@ -1754,7 +1786,18 @@ cd321x_register_port(struct tps6598x *tps, struct fwnode_handle *fwnode) cd321x->state.alt = NULL; cd321x->state.mode = TYPEC_STATE_SAFE; cd321x->state.data = NULL; - cd321x->state_valid = !typec_set_mode(tps->port, TYPEC_STATE_SAFE); + ret = typec_set_mode(tps->port, TYPEC_STATE_SAFE); + cd321x->state_valid = !ret; + /* + * A fixed host can register its SuperSpeed root after the mux provider + * (the T8140 ATC PHY reports this as -EAGAIN). Wait for that consumer at + * probe instead of exhausting the cable retry budget before it exists. + * Host startup does not depend on this port. + */ + if (ret == -EAGAIN) { + ret = -EPROBE_DEFER; + goto err_put_switch; + } cd321x->tbt_notifier.notifier_call = cd321x_tbt_notify; ret = typec_thunderbolt_switch_register_notifier(cd321x->tbt_switch, @@ -1769,6 +1812,9 @@ cd321x_register_port(struct tps6598x *tps, struct fwnode_handle *fwnode) typec_thunderbolt_switch_put(cd321x->tbt_switch); cd321x->tbt_switch = NULL; err_unregister_mux: + /* A deferred probe must not leak the connector reference on every retry. */ + fwnode_handle_put(cd321x->connector_fwnode); + cd321x->connector_fwnode = NULL; typec_mux_put(cd321x->mux); cd321x->mux = NULL; err_unregister_altmodes: @@ -1793,6 +1839,8 @@ cd321x_unregister_port(struct tps6598x *tps) struct cd321x *cd321x = container_of(tps, struct cd321x, tps); cd321x_stop_work(cd321x); + fwnode_handle_put(cd321x->connector_fwnode); + cd321x->connector_fwnode = NULL; typec_thunderbolt_switch_put(cd321x->tbt_switch); cd321x->tbt_switch = NULL; typec_mux_put(cd321x->mux); @@ -2253,6 +2301,14 @@ int tipd_init(struct tps6598x *tps) if (tps->reset) msleep(TPS_SETUP_MS); + /* + * The T8140 and T8142 display routes report a display taken by another + * port as -ENOSPC, and these machines were brought up without the + * SN201202x DP status event; keep that display handling there. + */ + tps->h17_display = of_machine_is_compatible("apple,t8140") || + of_machine_is_compatible("apple,t8142"); + if (!device_is_compatible(tps->dev, "ti,tps25750")) { ret = tps6598x_read32(tps, TPS_REG_VID, &vid); if (ret < 0 || !vid) @@ -2276,7 +2332,18 @@ int tipd_init(struct tps6598x *tps) return ret; } - ret = tps6598x_write64(tps, TPS_REG_INT_MASK1, tps->data->irq_mask1); + /* + * A controller found in "DBMa" is used as it is. Nothing below issues a + * 4CC command (the S0 power state switch above only does if the + * controller is not in S0 already), so probing neither leaves the debug + * mode nor disturbs a Debug USB or serial route set up before Linux. + * Remember it, so that the debugfs interface never tears that mode down. + */ + tps->dbma_at_probe = ret == TPS_MODE_DBMA; + if (tps->dbma_at_probe) + dev_info(tps->dev, "controller in debug mode \"DBMa\", leaving it there\n"); + + ret = tps6598x_write64(tps, TPS_REG_INT_MASK1, tipd_irq_mask1(tps)); if (ret) goto err_reset_controller; @@ -2344,6 +2411,7 @@ int tipd_init(struct tps6598x *tps) enable_irq_wake(tps->irq); } + tipd_debugfs_register(tps); if (tps->data->ready) tps->data->ready(tps); @@ -2369,6 +2437,8 @@ EXPORT_SYMBOL_GPL(tipd_init); void tipd_remove(struct tps6598x *tps) { + tipd_debugfs_unregister(tps); + if (!tps->irq) cancel_delayed_work_sync(&tps->wq_poll); else @@ -2542,6 +2612,12 @@ const struct tipd_data tipd_sn201202x_data = { }; EXPORT_SYMBOL_GPL(tipd_sn201202x_data); +static void __exit tipd_core_exit(void) +{ + tipd_debugfs_exit(); +} +module_exit(tipd_core_exit); + MODULE_AUTHOR("Heikki Krogerus "); MODULE_LICENSE("GPL v2"); MODULE_DESCRIPTION("TI TPS6598x USB Power Delivery Controller Core Functions"); diff --git a/drivers/usb/typec/tipd/debugfs.c b/drivers/usb/typec/tipd/debugfs.c new file mode 100644 index 00000000000000..8f35394624fe69 --- /dev/null +++ b/drivers/usb/typec/tipd/debugfs.c @@ -0,0 +1,300 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * TI TPS6598x / Apple CD321x / SN201202x debugfs interface + * + * /sys/kernel/debug/usb/tipd//command + * write: 4CC command followed by 0..64 bytes for DATA1. The command is + * executed as a whole under the driver's lock. Only a small set + * of commands used to talk to a connected Apple device (as + * tuxvdmtool does) is accepted. + * read: the 64 bytes of DATA1 that the last completed command of this + * open file left behind. + * + * /sys/kernel/debug/usb/tipd//regs + * read-only register access: pread() with the register as offset + * (0x00..0x7f) and the length (1..64 bytes) as count. + * + * An open "command" file that put the controller into "DBMa" leaves that + * mode (DBMa 0, then LOCK 0000) when it is released, unless the controller + * was in "DBMa" already when the driver probed: that mode belongs to + * whoever set it up before Linux (e.g. a boot loader arming Debug USB). + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "tps6598x.h" + +#define TPS_REG_MAX 0x7f + +struct tipd_debugfs { + struct dentry *dentry; + /* Open "command" files that entered "DBMa"; protected by tps->lock */ + unsigned int dbma_holders; +}; + +struct tipd_debugfs_cmd { + char cmd[4]; + u8 min_len; + u8 max_len; + u32 timeout_ms; +}; + +static const struct tipd_debugfs_cmd tipd_debugfs_cmds[] = { + { "LOCK", 4, 4, 1000 }, /* unlock key, 0000 locks again */ + { "DBMa", 1, 1, 1000 }, /* 1: enter, 0: leave debug mode */ + { "VDMs", 5, 29, 200 }, /* SOP type/count + 1..7 VDOs */ + { "DVEn", 4, 28, 1000 }, /* local end of a serial/debug route */ + { "DISC", 0, 1, 1000 }, /* simulated disconnect */ +}; + +struct tipd_debugfs_file { + struct tps6598x *tps; + bool holds_dbma; + bool have_result; + u8 result[TPS_MAX_LEN]; +}; + +static struct dentry *tipd_debugfs_root; +static DEFINE_MUTEX(tipd_debugfs_root_lock); + +static const struct tipd_debugfs_cmd *tipd_debugfs_find_cmd(const u8 *cmd) +{ + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(tipd_debugfs_cmds); i++) + if (!memcmp(tipd_debugfs_cmds[i].cmd, cmd, 4)) + return &tipd_debugfs_cmds[i]; + + return NULL; +} + +/* Leave the debug mode entered through debugfs. Called with tps->lock held. */ +static void tipd_debugfs_leave_dbma(struct tps6598x *tps) +{ + static const u8 lock_key[4]; + u8 dbma = 0; + int ret; + + lockdep_assert_held(&tps->lock); + + if (tps->dbma_at_probe) + return; + + ret = tps6598x_exec_cmd_tmo(tps, "DBMa", sizeof(dbma), &dbma, 0, NULL, + 1000, 0); + if (ret) + dev_warn(tps->dev, "debugfs: failed to leave \"DBMa\": %d\n", ret); + + ret = tps6598x_exec_cmd_tmo(tps, "LOCK", sizeof(lock_key), lock_key, + 0, NULL, 1000, 0); + if (ret) + dev_warn(tps->dev, "debugfs: failed to lock: %d\n", ret); +} + +static int tipd_debugfs_command_open(struct inode *inode, struct file *file) +{ + struct tipd_debugfs_file *f; + + f = kzalloc_obj(*f); + if (!f) + return -ENOMEM; + + f->tps = inode->i_private; + file->private_data = f; + + return 0; +} + +static int tipd_debugfs_command_release(struct inode *inode, struct file *file) +{ + struct tipd_debugfs_file *f = file->private_data; + struct dentry *dentry = file->f_path.dentry; + struct tps6598x *tps = f->tps; + + /* + * debugfs calls ->release even after the file was removed, when tps + * may be gone. tipd_debugfs_unregister() has then left "DBMa" itself. + */ + if (f->holds_dbma && !debugfs_file_get(dentry)) { + scoped_guard(mutex, &tps->lock) { + if (!--tps->debugfs->dbma_holders) + tipd_debugfs_leave_dbma(tps); + } + debugfs_file_put(dentry); + } + + kfree(f); + + return 0; +} + +static ssize_t tipd_debugfs_command_write(struct file *file, + const char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct tipd_debugfs_file *f = file->private_data; + struct tps6598x *tps = f->tps; + const struct tipd_debugfs_cmd *cmd; + u8 buf[4 + TPS_MAX_LEN]; + u8 out[TPS_MAX_LEN] = { }; + size_t len; + int ret; + + if (count < 4 || count > sizeof(buf)) + return -EINVAL; + + if (copy_from_user(buf, ubuf, count)) + return -EFAULT; + + cmd = tipd_debugfs_find_cmd(buf); + if (!cmd) + return -EOPNOTSUPP; + + len = count - 4; + if (len < cmd->min_len || len > cmd->max_len) + return -EINVAL; + if (!memcmp(cmd->cmd, "VDMs", 4) && (len - 1) % 4) + return -EINVAL; + if (!memcmp(cmd->cmd, "DVEn", 4) && len % 4) + return -EINVAL; + if (!memcmp(cmd->cmd, "DBMa", 4) && buf[4] > 1) + return -EINVAL; + + guard(mutex)(&tps->lock); + + /* Preserve inherited mode and the leases held by other open files. */ + if ((!memcmp(cmd->cmd, "DBMa", 4) && !buf[4] && + (tps->dbma_at_probe || + tps->debugfs->dbma_holders > (unsigned int)f->holds_dbma)) || + (!memcmp(cmd->cmd, "LOCK", 4) && !memchr_inv(buf + 4, 0, 4) && + (tps->dbma_at_probe || tps->debugfs->dbma_holders))) + return -EBUSY; + + ret = tps6598x_exec_cmd_tmo(tps, cmd->cmd, len, len ? &buf[4] : NULL, + sizeof(out), out, cmd->timeout_ms, 0); + + /* DATA1 was read back if the command ran, even if its task failed */ + f->have_result = !ret || + (ret == -ETIMEDOUT && out[0] == TPS_TASK_TIMEOUT) || + (ret == -EPERM && out[0] == TPS_TASK_REJECTED); + if (f->have_result) + memcpy(f->result, out, sizeof(out)); + + if (!ret && !memcmp(cmd->cmd, "DBMa", 4)) { + if (buf[4] && !(out[0] & 0xf) && !f->holds_dbma) { + f->holds_dbma = true; + tps->debugfs->dbma_holders++; + } else if (!buf[4] && f->holds_dbma) { + f->holds_dbma = false; + tps->debugfs->dbma_holders--; + } + } + + return ret ?: count; +} + +static ssize_t tipd_debugfs_command_read(struct file *file, char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct tipd_debugfs_file *f = file->private_data; + + guard(mutex)(&f->tps->lock); + if (!f->have_result) + return -ENODATA; + + return simple_read_from_buffer(ubuf, count, ppos, f->result, + sizeof(f->result)); +} + +static const struct file_operations tipd_debugfs_command_fops = { + .owner = THIS_MODULE, + .open = tipd_debugfs_command_open, + .release = tipd_debugfs_command_release, + .write = tipd_debugfs_command_write, + .read = tipd_debugfs_command_read, + .llseek = default_llseek, +}; + +static ssize_t tipd_debugfs_regs_read(struct file *file, char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct tps6598x *tps = file->private_data; + u8 buf[TPS_MAX_LEN]; + int ret; + + if (*ppos < 0 || *ppos > TPS_REG_MAX) + return -EINVAL; + if (!count) + return 0; + if (count > sizeof(buf)) + return -EINVAL; + + scoped_guard(mutex, &tps->lock) + ret = tps6598x_block_read(tps, *ppos, buf, count); + if (ret) + return ret; + + /* The offset selects a register; it does not advance. */ + if (copy_to_user(ubuf, buf, count)) + return -EFAULT; + + return count; +} + +static const struct file_operations tipd_debugfs_regs_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .read = tipd_debugfs_regs_read, + .llseek = default_llseek, +}; + +void tipd_debugfs_register(struct tps6598x *tps) +{ + struct tipd_debugfs *dbg; + + scoped_guard(mutex, &tipd_debugfs_root_lock) { + if (!tipd_debugfs_root) + tipd_debugfs_root = debugfs_create_dir("tipd", usb_debug_root); + } + + dbg = kzalloc_obj(*dbg); + if (!dbg) + return; + + dbg->dentry = debugfs_create_dir(dev_name(tps->dev), tipd_debugfs_root); + tps->debugfs = dbg; + debugfs_create_file("command", 0600, dbg->dentry, tps, + &tipd_debugfs_command_fops); + debugfs_create_file("regs", 0400, dbg->dentry, tps, + &tipd_debugfs_regs_fops); +} + +void tipd_debugfs_unregister(struct tps6598x *tps) +{ + struct tipd_debugfs *dbg = tps->debugfs; + + if (!dbg) + return; + + /* Waits for running file operations, later ones fail with -EIO */ + debugfs_remove_recursive(dbg->dentry); + + scoped_guard(mutex, &tps->lock) { + if (dbg->dbma_holders) + tipd_debugfs_leave_dbma(tps); + tps->debugfs = NULL; + } + + kfree(dbg); +} + +void tipd_debugfs_exit(void) +{ + debugfs_remove(tipd_debugfs_root); +} diff --git a/drivers/usb/typec/tipd/spmi.c b/drivers/usb/typec/tipd/spmi.c index dc57ef88ca7fe6..3ff9f602c23b6d 100644 --- a/drivers/usb/typec/tipd/spmi.c +++ b/drivers/usb/typec/tipd/spmi.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include #include @@ -24,6 +25,15 @@ static int regmap_sn201202x_select_reg(struct spmi_device *sdev, u8 reg) if (err) return err; + if (sn->polled) { + int poll_err; + + poll_err = read_poll_timeout(spmi_register_read, err, + err || val == reg, 100, 100000, + false, sdev, 0, &val); + return err ?: poll_err; + } + if (!wait_for_completion_timeout(&sn->select_completion, msecs_to_jiffies(100))) return -ETIMEDOUT; @@ -157,6 +167,39 @@ static const struct of_device_id sn201202x_of_match[] = { {} }; +static int sn201202x_wake(struct sn201202x *sn) +{ + unsigned long deadline; + u8 size; + int ret; + + reinit_completion(&sn->wake_completion); + ret = spmi_command_wakeup(sn->sdev); + if (ret) + return ret; + if (!sn->polled) + return wait_for_completion_timeout(&sn->wake_completion, + msecs_to_jiffies(100)) ? 0 : -ETIMEDOUT; + + /* A sleeping ACE3 ACKs commands but returns zeros until it is awake. */ + deadline = jiffies + msecs_to_jiffies(500); + do { + ret = regmap_sn201202x_select_reg(sn->sdev, TPS_REG_MODE); + if (ret == -ETIMEDOUT) + continue; + if (ret) + return ret; + ret = spmi_register_read(sn->sdev, 0x1f, &size); + if (ret) + return ret; + if (size) + return 0; + usleep_range(1000, 2000); + } while (time_before(jiffies, deadline)); + + return -ETIMEDOUT; +} + static int sn201202x_probe(struct spmi_device *device) { const struct of_device_id *match; @@ -181,46 +224,48 @@ static int sn201202x_probe(struct spmi_device *device) tps->dev = &device->dev; tps->data = data; - tps->irq = of_irq_get_byname(device->dev.of_node, "irq"); - if (tps->irq < 0) - return tps->irq; - irq_select = of_irq_get_byname(device->dev.of_node, "select"); - if (irq_select < 0) - return irq_select; - irq_sleep = of_irq_get_byname(device->dev.of_node, "sleep"); - if (irq_sleep < 0) - return irq_sleep; - irq_wake = of_irq_get_byname(device->dev.of_node, "wake"); - if (irq_wake < 0) - return irq_wake; - init_completion(&sn->select_completion); init_completion(&sn->sleep_completion); init_completion(&sn->wake_completion); - ret = devm_request_irq(&device->dev, irq_select, sn201202x_irq, - 0, NULL, &sn->select_completion); - if (ret) - return ret; - ret = devm_request_irq(&device->dev, irq_sleep, sn201202x_irq, - 0, NULL, &sn->sleep_completion); - if (ret) - return ret; - ret = devm_request_irq(&device->dev, irq_wake, sn201202x_irq, - 0, NULL, &sn->wake_completion); - if (ret) - return ret; + sn->polled = !of_property_present(device->dev.of_node, "interrupts") && + !of_property_present(device->dev.of_node, "interrupts-extended"); + if (!sn->polled) { + tps->irq = of_irq_get_byname(device->dev.of_node, "irq"); + if (tps->irq < 0) + return tps->irq; + irq_select = of_irq_get_byname(device->dev.of_node, "select"); + if (irq_select < 0) + return irq_select; + irq_sleep = of_irq_get_byname(device->dev.of_node, "sleep"); + if (irq_sleep < 0) + return irq_sleep; + irq_wake = of_irq_get_byname(device->dev.of_node, "wake"); + if (irq_wake < 0) + return irq_wake; + + ret = devm_request_irq(&device->dev, irq_select, sn201202x_irq, + 0, NULL, &sn->select_completion); + if (ret) + return ret; + ret = devm_request_irq(&device->dev, irq_sleep, sn201202x_irq, + 0, NULL, &sn->sleep_completion); + if (ret) + return ret; + ret = devm_request_irq(&device->dev, irq_wake, sn201202x_irq, + 0, NULL, &sn->wake_completion); + if (ret) + return ret; + } spmi_device_set_drvdata(device, tps); tps->regmap = devm_regmap_init_sn201202x(device, &tps6598x_regmap_config); if (IS_ERR(tps->regmap)) return PTR_ERR(tps->regmap); - ret = spmi_command_wakeup(device); + ret = sn201202x_wake(sn); if (ret) return ret; - if (!wait_for_completion_timeout(&sn->wake_completion, msecs_to_jiffies(100))) - return -ETIMEDOUT; ret = tipd_init(tps); if (ret) @@ -243,12 +288,9 @@ static int __maybe_unused sn201202x_resume(struct device *dev) struct sn201202x *sn = tps_to_sn(tps); int err; - reinit_completion(&sn->wake_completion); - err = spmi_command_wakeup(sn->sdev); + err = sn201202x_wake(sn); if (err) return err; - if (!wait_for_completion_timeout(&sn->wake_completion, msecs_to_jiffies(100))) - return -ETIMEDOUT; return tipd_resume(tps); } @@ -265,7 +307,8 @@ static int __maybe_unused sn201202x_suspend(struct device *dev) err = spmi_command_sleep(sn->sdev); if (err) goto out_resume; - if (!wait_for_completion_timeout(&sn->sleep_completion, msecs_to_jiffies(100))) { + if (!sn->polled && + !wait_for_completion_timeout(&sn->sleep_completion, msecs_to_jiffies(100))) { err = -ETIMEDOUT; goto out_resume; } diff --git a/drivers/usb/typec/tipd/tps6598x.h b/drivers/usb/typec/tipd/tps6598x.h index 1239cd971f2499..de74a55c3f8b1f 100644 --- a/drivers/usb/typec/tipd/tps6598x.h +++ b/drivers/usb/typec/tipd/tps6598x.h @@ -20,6 +20,9 @@ #include "cd321x-pm.h" +/* Also read by the SPMI glue to tell when the controller has woken up */ +#define TPS_REG_MODE 0x03 + #define TPS_FIELD_GET(_mask, _reg) ((typeof(_mask))(((_reg) & (_mask)) >> __bf_shf(_mask))) /* TPS_REG_STATUS bits */ @@ -284,6 +287,16 @@ #define TPS_65981_2_6_INTEVENT_LEN 8 #define TPS_65987_8_INTEVENT_LEN 11 +/* + * Max data bytes for Data1, Data2, and other registers. See ch 1.3.2: + * https://www.ti.com/lit/ug/slvuan1a/slvuan1a.pdf + */ +#define TPS_MAX_LEN 64 + +/* Standard Task return codes */ +#define TPS_TASK_TIMEOUT 1 +#define TPS_TASK_REJECTED 3 + struct tps6598x; struct tipd_data { @@ -313,6 +326,14 @@ struct tps6598x { struct mutex lock; /* device lock */ int irq; u8 i2c_protocol:1; + u8 dbma_at_probe:1; /* Apple "DBMa" debug mode was active at probe */ + /* + * T8140 (H17P) and T8142 (H17G) Type-C display handling: the display + * mux reports a display taken by another port as -ENOSPC (-EBUSY is a + * transient PHY state and is retried), and the SN201202x DP status + * event and its HPD rules stay off. + */ + u8 h17_display:1; struct gpio_desc *reset; struct typec_port *port; @@ -332,6 +353,8 @@ struct tps6598x { struct delayed_work wq_poll; const struct tipd_data *data; + + struct tipd_debugfs *debugfs; }; /* TPS_REG_USB4_STATUS */ @@ -411,6 +434,7 @@ struct cd321x { struct sn201202x { struct cd321x cd; + bool polled; struct completion select_completion; struct completion sleep_completion; struct completion wake_completion; @@ -430,4 +454,21 @@ void tipd_complete(struct device *dev); int tipd_suspend(struct tps6598x *tps); int tipd_resume(struct tps6598x *tps); +/* Internal to tps6598x-core */ +int tps6598x_block_read(struct tps6598x *tps, u8 reg, void *val, size_t len); +int tps6598x_exec_cmd_tmo(struct tps6598x *tps, const char *cmd, + size_t in_len, const u8 *in_data, + size_t out_len, u8 *out_data, + u32 cmd_timeout_ms, u32 res_delay_ms); + +#ifdef CONFIG_DEBUG_FS +void tipd_debugfs_register(struct tps6598x *tps); +void tipd_debugfs_unregister(struct tps6598x *tps); +void tipd_debugfs_exit(void); +#else +static inline void tipd_debugfs_register(struct tps6598x *tps) { } +static inline void tipd_debugfs_unregister(struct tps6598x *tps) { } +static inline void tipd_debugfs_exit(void) { } +#endif + #endif /* __TPS6598X_H__ */ diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig index ceae52fd7586d0..33f6c7e9608d5b 100644 --- a/drivers/vfio/Kconfig +++ b/drivers/vfio/Kconfig @@ -54,6 +54,18 @@ config VFIO_IOMMU_TYPE1 tristate default n +config VFIO_TYPE1_ACCOUNT_KUNIT_TEST + bool "Test VFIO type1 pin accounting and retained fragments" if !KUNIT_ALL_TESTS + depends on KUNIT=y && VFIO_IOMMU_TYPE1=y + default KUNIT_ALL_TESTS + help + Exercise locked-memory refunds after source-mm exit and DMA owner + changes. Arm64 mixed-granule configurations also test accounting + conversions between user and native page sizes. + Mixed-granule tests exercise retained pins, mappings, replay and + external page arrays with recorded IOMMU translations. No physical + device DMA is performed. + config VFIO_IOMMU_SPAPR_TCE tristate depends on SPAPR_TCE_IOMMU diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index e6d3a2311a99b2..df079de9ee5de5 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -28,11 +28,14 @@ #include #include #include +#include +#include #include #include #include #include #include +#include #include #include #include @@ -84,6 +87,8 @@ struct vfio_domain { bool enforce_cache_coherency : 1; }; +struct vfio_fragment_provider; + struct vfio_dma { struct rb_node node; dma_addr_t iova; /* Device address */ @@ -99,6 +104,7 @@ struct vfio_dma { unsigned long *bitmap; struct mm_struct *mm; size_t locked_vm; + struct vfio_fragment_provider *fragments; }; struct vfio_batch { @@ -128,7 +134,9 @@ struct vfio_pfn { struct rb_node node; dma_addr_t iova; /* Device address */ unsigned long pfn; /* Host pfn */ + unsigned long length; /* Bytes retained in this DMA */ unsigned int ref_count; + bool native_pin; /* separately adopted legacy external PIN */ }; struct vfio_regions { @@ -152,6 +160,9 @@ struct vfio_regions { #define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) static int put_pfn(unsigned long pfn, int prot); +static void vfio_fragments_put_native_pin(struct vfio_dma *dma, unsigned long pfn); +static void vfio_fragments_release_range(struct vfio_dma *dma, unsigned long start, + unsigned long length); static struct vfio_iommu_group* vfio_iommu_find_iommu_group(struct vfio_iommu *iommu, @@ -238,6 +249,12 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old) } +/* Software dirty granularity must survive domain attach/detach unchanged. */ +static size_t vfio_dirty_pgsize(struct vfio_iommu *iommu) +{ + return IS_ENABLED(CONFIG_MM_SUBPAGE) ? SZ_4K : 1UL << __ffs(iommu->pgsize_bitmap); +} + static int vfio_dma_bitmap_alloc(struct vfio_dma *dma, size_t pgsize) { uint64_t npages = dma->size / pgsize; @@ -272,14 +289,15 @@ static void vfio_dma_populate_bitmap(struct vfio_dma *dma, size_t pgsize) for (p = rb_first(&dma->pfn_list); p; p = rb_next(p)) { struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); - bitmap_set(dma->bitmap, (vpfn->iova - dma->iova) >> pgshift, 1); + bitmap_set(dma->bitmap, (vpfn->iova - dma->iova) >> pgshift, + vpfn->length >> pgshift); } } static void vfio_iommu_populate_bitmap_full(struct vfio_iommu *iommu) { struct rb_node *n; - unsigned long pgshift = __ffs(iommu->pgsize_bitmap); + unsigned long pgshift = __ffs(vfio_dirty_pgsize(iommu)); for (n = rb_first(&iommu->dma_list); n; n = rb_next(n)) { struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); @@ -383,7 +401,7 @@ static void vfio_unlink_pfn(struct vfio_dma *dma, struct vfio_pfn *old) } static int vfio_add_to_pfn_list(struct vfio_dma *dma, dma_addr_t iova, - unsigned long pfn) + unsigned long pfn, unsigned long length) { struct vfio_pfn *vpfn; @@ -393,6 +411,7 @@ static int vfio_add_to_pfn_list(struct vfio_dma *dma, dma_addr_t iova, vpfn->iova = iova; vpfn->pfn = pfn; + vpfn->length = length; vpfn->ref_count = 1; vfio_link_pfn(dma, vpfn); return 0; @@ -421,8 +440,21 @@ static int vfio_iova_put_vfio_pfn(struct vfio_dma *dma, struct vfio_pfn *vpfn) vpfn->ref_count--; if (!vpfn->ref_count) { - ret = put_pfn(vpfn->pfn, dma->prot); - vfio_remove_from_pfn_list(dma, vpfn); + if (dma->fragments) { + unsigned long offset = vpfn->iova - dma->iova; + unsigned long length = vpfn->length; + + unsigned long pfn = vpfn->pfn; + bool native_pin = vpfn->native_pin; + + vfio_remove_from_pfn_list(dma, vpfn); + if (native_pin) + vfio_fragments_put_native_pin(dma, pfn); + vfio_fragments_release_range(dma, offset, length); + } else { + ret = put_pfn(vpfn->pfn, dma->prot); + vfio_remove_from_pfn_list(dma, vpfn); + } } return ret; } @@ -430,10 +462,19 @@ static int vfio_iova_put_vfio_pfn(struct vfio_dma *dma, struct vfio_pfn *vpfn) static int mm_lock_acct(struct task_struct *task, struct mm_struct *mm, bool lock_cap, long npage) { - int ret = mmap_write_lock_killable(mm); + int ret; - if (ret) - return ret; + if (!npage) + return 0; + + /* A released pin must be refunded even if the caller is interrupted. */ + if (npage < 0) { + mmap_write_lock(mm); + } else { + ret = mmap_write_lock_killable(mm); + if (ret) + return ret; + } ret = __account_locked_vm(mm, abs(npage), npage > 0, task, lock_cap); mmap_write_unlock(mm); @@ -443,25 +484,28 @@ static int mm_lock_acct(struct task_struct *task, struct mm_struct *mm, static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async) { struct mm_struct *mm; + bool get_mm = async && npage > 0; int ret; if (!npage) return 0; mm = dma->mm; - if (async && !mmget_not_zero(mm)) + if (get_mm && !mmget_not_zero(mm)) return -ESRCH; /* process exited */ ret = mm_lock_acct(dma->task, mm, dma->lock_cap, npage); if (!ret) dma->locked_vm += npage; - if (async) + if (get_mm) mmput(mm); return ret; } +#include "vfio_iommu_type1_fragments.c" + /* * Some mappings aren't backed by a struct page, for example an mmap'd * MMIO range for our own or another device. These use a different @@ -765,7 +809,8 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, */ if (acct_pages) { if (!dma->lock_cap && - mm->locked_vm + lock_acct + acct_pages > limit) { + mm_locked_vm_native_pages(mm) + + lock_acct + acct_pages > limit) { pr_warn("%s: RLIMIT_MEMLOCK (%ld) exceeded\n", __func__, limit << PAGE_SHIFT); ret = -ENOMEM; @@ -889,6 +934,78 @@ static int vfio_unpin_page_external(struct vfio_dma *dma, dma_addr_t iova, return unlocked; } +/* A native page may contain independently owned fine-grained DMA areas. */ +static unsigned long vfio_unpin_external_range(struct vfio_iommu *iommu, + dma_addr_t start, unsigned long length, + bool do_accounting) +{ + unsigned long done = 0; + + while (done < length) { + dma_addr_t iova = start + done; + struct vfio_dma *dma = vfio_find_dma(iommu, iova, 1); + unsigned long bytes; + + if (!dma || iova < dma->iova) + break; + bytes = min_t(unsigned long, length - done, + PAGE_SIZE - offset_in_page(iova)); + bytes = min(bytes, dma->size - (iova - dma->iova)); + vfio_unpin_page_external(dma, iova, do_accounting); + done += bytes; + } + return done; +} + +static int vfio_pin_external_segment(struct vfio_iommu *iommu, + struct vfio_dma *dma, dma_addr_t iova, unsigned long length, + int prot, bool do_accounting, unsigned long *pfn) +{ + struct vfio_pfn *vpfn; + int ret; + + if ((dma->prot & prot) != prot) + return -EPERM; + vpfn = vfio_iova_get_vfio_pfn(dma, iova); + if (vpfn) { + if (vpfn->length != length) { + vfio_unpin_page_external(dma, iova, do_accounting); + return -EINVAL; + } + *pfn = vpfn->pfn; + } else if (dma->fragments) { + ret = vfio_fragments_external(dma, iova, length, pfn); + if (ret) + return ret; + } else { + /* Ordinary native mappings still use one native GUP per page. */ + if (length != PAGE_SIZE) + return -EINVAL; + ret = vfio_pin_page_external(dma, dma->vaddr + iova - dma->iova, + pfn, do_accounting); + if (ret) + return ret; + if (!pfn_valid(*pfn)) { + ret = -EINVAL; + goto put_native; + } + ret = vfio_add_to_pfn_list(dma, iova, *pfn, length); + if (ret) + goto put_native; + } + if (iommu->dirty_page_tracking) { + unsigned long pgshift = __ffs(vfio_dirty_pgsize(iommu)); + + bitmap_set(dma->bitmap, (iova - dma->iova) >> pgshift, + length >> pgshift); + } + return 0; +put_native: + if (put_pfn(*pfn, dma->prot) && do_accounting) + vfio_lock_acct(dma, -1, true); + return ret; +} + static int vfio_iommu_type1_pin_pages(void *iommu_data, struct iommu_group *iommu_group, dma_addr_t user_iova, @@ -897,119 +1014,72 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data, { struct vfio_iommu *iommu = iommu_data; struct vfio_iommu_group *group; - int i, j, ret; - unsigned long remote_vaddr; - struct vfio_dma *dma; + unsigned long pinned = 0; bool do_accounting; dma_addr_t iova_end; size_t iova_size; + int i, ret; if (!iommu || !pages || npage <= 0) return -EINVAL; - - /* Supported for v2 version only */ + /* Callers recover the original byte offset from the returned page. */ + user_iova = ALIGN_DOWN(user_iova, PAGE_SIZE); if (!iommu->v2) return -EACCES; - if (check_mul_overflow(npage, PAGE_SIZE, &iova_size) || check_add_overflow(user_iova, iova_size - 1, &iova_end)) return -EOVERFLOW; mutex_lock(&iommu->lock); - if (WARN_ONCE(iommu->vaddr_invalid_count, "vfio_pin_pages not allowed with VFIO_UPDATE_VADDR\n")) { ret = -EBUSY; goto pin_done; } - - /* Fail if no dma_umap notifier is registered */ if (list_empty(&iommu->device_list)) { ret = -EINVAL; goto pin_done; } - - /* - * If iommu capable domain exist in the container then all pages are - * already pinned and accounted. Accounting should be done if there is no - * iommu capable domain in the container. - */ do_accounting = list_empty(&iommu->domain_list); - for (i = 0; i < npage; i++) { - unsigned long phys_pfn; - dma_addr_t iova; - struct vfio_pfn *vpfn; - - iova = user_iova + PAGE_SIZE * i; - dma = vfio_find_dma(iommu, iova, PAGE_SIZE); - if (!dma) { - ret = -EINVAL; - goto pin_unwind; - } - - if ((dma->prot & prot) != prot) { - ret = -EPERM; - goto pin_unwind; - } - - vpfn = vfio_iova_get_vfio_pfn(dma, iova); - if (vpfn) { - pages[i] = pfn_to_page(vpfn->pfn); - continue; - } - - remote_vaddr = dma->vaddr + (iova - dma->iova); - ret = vfio_pin_page_external(dma, remote_vaddr, &phys_pfn, - do_accounting); - if (ret) - goto pin_unwind; - - if (!pfn_valid(phys_pfn)) { - ret = -EINVAL; - goto pin_unwind; - } - - ret = vfio_add_to_pfn_list(dma, iova, phys_pfn); - if (ret) { - if (put_pfn(phys_pfn, dma->prot) && do_accounting) - vfio_lock_acct(dma, -1, true); - goto pin_unwind; - } + unsigned long done = 0; - pages[i] = pfn_to_page(phys_pfn); + pages[i] = NULL; + while (done < PAGE_SIZE) { + dma_addr_t iova = user_iova + PAGE_SIZE * i + done; + struct vfio_dma *dma = vfio_find_dma(iommu, iova, 1); + unsigned long length, pfn; - if (iommu->dirty_page_tracking) { - unsigned long pgshift = __ffs(iommu->pgsize_bitmap); - - /* - * Bitmap populated with the smallest supported page - * size - */ - bitmap_set(dma->bitmap, - (iova - dma->iova) >> pgshift, 1); + if (!dma || iova < dma->iova) { + ret = -EINVAL; + goto pin_unwind; + } + length = min_t(unsigned long, PAGE_SIZE - done, + dma->size - (iova - dma->iova)); + ret = vfio_pin_external_segment(iommu, dma, iova, length, + prot, do_accounting, &pfn); + if (ret) + goto pin_unwind; + pinned += length; + if (pages[i] && page_to_pfn(pages[i]) != pfn) { + ret = -EINVAL; + goto pin_unwind; + } + pages[i] = pfn_to_page(pfn); + done += length; } } - ret = i; - + ret = npage; group = vfio_iommu_find_iommu_group(iommu, iommu_group); if (!group->pinned_page_dirty_scope) { group->pinned_page_dirty_scope = true; iommu->num_non_pinned_groups--; } - goto pin_done; - pin_unwind: - pages[i] = NULL; - for (j = 0; j < i; j++) { - dma_addr_t iova; - - iova = user_iova + PAGE_SIZE * j; - dma = vfio_find_dma(iommu, iova, PAGE_SIZE); - vfio_unpin_page_external(dma, iova, do_accounting); - pages[j] = NULL; - } + vfio_unpin_external_range(iommu, user_iova, pinned, do_accounting); + while (i >= 0) + pages[i--] = NULL; pin_done: mutex_unlock(&iommu->lock); return ret; @@ -1019,39 +1089,21 @@ static void vfio_iommu_type1_unpin_pages(void *iommu_data, dma_addr_t user_iova, int npage) { struct vfio_iommu *iommu = iommu_data; - bool do_accounting; dma_addr_t iova_end; size_t iova_size; - int i; - - /* Supported for v2 version only */ - if (WARN_ON(!iommu->v2)) - return; + unsigned long unpinned; - if (WARN_ON(npage <= 0)) + if (WARN_ON(!iommu->v2) || WARN_ON(npage <= 0)) return; - + user_iova = ALIGN_DOWN(user_iova, PAGE_SIZE); if (WARN_ON(check_mul_overflow(npage, PAGE_SIZE, &iova_size) || check_add_overflow(user_iova, iova_size - 1, &iova_end))) return; - mutex_lock(&iommu->lock); - - do_accounting = list_empty(&iommu->domain_list); - for (i = 0; i < npage; i++) { - dma_addr_t iova = user_iova + PAGE_SIZE * i; - struct vfio_dma *dma; - - dma = vfio_find_dma(iommu, iova, PAGE_SIZE); - if (!dma) - break; - - vfio_unpin_page_external(dma, iova, do_accounting); - } - + unpinned = vfio_unpin_external_range(iommu, user_iova, iova_size, + list_empty(&iommu->domain_list)); mutex_unlock(&iommu->lock); - - WARN_ON(i != npage); + WARN_ON(unpinned != iova_size); } static long vfio_sync_unpin(struct vfio_dma *dma, struct vfio_domain *domain, @@ -1155,6 +1207,10 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, if (!dma->size) return 0; + if (dma->fragments) { + vfio_fragments_unmap(iommu, dma); + return 0; + } if (list_empty(&iommu->domain_list)) return 0; @@ -1233,6 +1289,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma) { WARN_ON(!RB_EMPTY_ROOT(&dma->pfn_list)); vfio_unmap_unpin(iommu, dma, true); + vfio_fragments_destroy(dma); vfio_unlink_dma(iommu, dma); put_task_struct(dma->task); mmdrop(dma->mm); @@ -1246,21 +1303,18 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma) static void vfio_update_pgsize_bitmap(struct vfio_iommu *iommu) { struct vfio_domain *domain; + unsigned long minimum = SZ_4K; iommu->pgsize_bitmap = ULONG_MAX; - - list_for_each_entry(domain, &iommu->domain_list, next) + list_for_each_entry(domain, &iommu->domain_list, next) { iommu->pgsize_bitmap &= domain->domain->pgsize_bitmap; - - /* - * In case the IOMMU supports page sizes smaller than PAGE_SIZE - * we pretend PAGE_SIZE is supported and hide sub-PAGE_SIZE sizes. - * That way the user will be able to map/unmap buffers whose size/ - * start address is aligned with PAGE_SIZE. Pinning code uses that - * granularity while iommu driver can use the sub-PAGE_SIZE size - * to map the buffer. - */ - if (iommu->pgsize_bitmap & ~PAGE_MASK) { + minimum = max(minimum, 1UL << __ffs(domain->domain->pgsize_bitmap)); + } + if (IS_ENABLED(CONFIG_MM_SUBPAGE)) { + /* Every domain can compose its minimum pages into this common unit. */ + iommu->pgsize_bitmap &= ~(minimum - 1); + iommu->pgsize_bitmap |= minimum; + } else if (iommu->pgsize_bitmap & ~PAGE_MASK) { iommu->pgsize_bitmap &= PAGE_MASK; iommu->pgsize_bitmap |= PAGE_SIZE; } @@ -1367,6 +1421,8 @@ static void vfio_notify_dma_unmap(struct vfio_iommu *iommu, struct vfio_dma *dma) { struct vfio_device *device; + dma_addr_t iova = dma->iova; + size_t size = dma->size; if (list_empty(&iommu->device_list)) return; @@ -1377,11 +1433,12 @@ static void vfio_notify_dma_unmap(struct vfio_iommu *iommu, * call back down to this code and try to obtain the iommu->lock we must * drop it. */ - mutex_lock(&iommu->device_list_lock); + /* The range is a snapshot: another unmap may free dma while unlocked. */ mutex_unlock(&iommu->lock); + mutex_lock(&iommu->device_list_lock); list_for_each_entry(device, &iommu->device_list, iommu_entry) - device->ops->dma_unmap(device, dma->iova, dma->size); + device->ops->dma_unmap(device, iova, size); mutex_unlock(&iommu->device_list_lock); mutex_lock(&iommu->lock); @@ -1437,11 +1494,10 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, /* When dirty tracking is enabled, allow only min supported pgsize */ if ((unmap->flags & VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP) && - (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + (!iommu->dirty_page_tracking || (bitmap->pgsize != vfio_dirty_pgsize(iommu)))) { goto unlock; } - WARN_ON((pgsize - 1) & PAGE_MASK); again: /* * vfio-iommu-type1 (v1) - User mappings were coalesced together to @@ -1530,7 +1586,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, if (unmap->flags & VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP) { ret = update_user_bitmap(bitmap->data, iommu, dma, - iova, pgsize); + iova, vfio_dirty_pgsize(iommu)); if (ret) break; } @@ -1587,6 +1643,13 @@ static int vfio_pin_map_dma(struct vfio_iommu *iommu, struct vfio_dma *dma, unsigned long pfn, limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; int ret = 0; + if (dma->fragments) { + dma->size = map_size; + ret = vfio_fragments_map_all(iommu, dma); + if (ret) + vfio_remove_dma(iommu, dma); + return ret; + } vfio_batch_init(&batch); while (size) { @@ -1644,27 +1707,30 @@ static bool vfio_iommu_iova_dma_valid(struct vfio_iommu *iommu, return list_empty(iova); } -static int vfio_change_dma_owner(struct vfio_dma *dma) +static int vfio_change_dma_mm_pages(struct vfio_dma *dma, struct task_struct *task, + struct mm_struct *mm, bool lock_cap, unsigned long pages) { - struct task_struct *task = current->group_leader; - struct mm_struct *mm = current->mm; - long npage = dma->locked_vm; - bool lock_cap; + long old_pages = dma->locked_vm; int ret; - if (mm == dma->mm) - return 0; - - lock_cap = capable(CAP_IPC_LOCK); - ret = mm_lock_acct(task, mm, lock_cap, npage); + if (mm == dma->mm) { + if (pages == old_pages) + return 0; + ret = mm_lock_acct(task, mm, lock_cap, (long)pages - old_pages); + if (!ret) + dma->locked_vm = pages; + return ret; + } + ret = mm_lock_acct(task, mm, lock_cap, pages); if (ret) return ret; - if (mmget_not_zero(dma->mm)) { - mm_lock_acct(dma->task, dma->mm, dma->lock_cap, -npage); - mmput(dma->mm); + /* Metadata adoption can deduplicate charges, but not original PIN refs. */ + ret = mm_lock_acct(dma->task, dma->mm, dma->lock_cap, -old_pages); + if (ret) { + mm_lock_acct(task, mm, lock_cap, -(long)pages); + return ret; } - if (dma->task != task) { put_task_struct(dma->task); dma->task = get_task_struct(task); @@ -1673,9 +1739,40 @@ static int vfio_change_dma_owner(struct vfio_dma *dma) dma->mm = mm; mmgrab(dma->mm); dma->lock_cap = lock_cap; + dma->locked_vm = pages; return 0; } +static int vfio_change_dma_mm(struct vfio_dma *dma, struct task_struct *task, + struct mm_struct *mm, bool lock_cap) +{ + return vfio_change_dma_mm_pages(dma, task, mm, lock_cap, dma->locked_vm); +} + +static int vfio_change_dma_owner(struct vfio_iommu *iommu, struct vfio_dma *dma, + unsigned long vaddr) +{ + struct vfio_fragment_provider *provider; + int ret; + + if (dma->fragments || (mm_page_size(current->mm) == PAGE_SIZE && + IS_ALIGNED(vaddr, PAGE_SIZE))) + return vfio_change_dma_mm(dma, current->group_leader, current->mm, + capable(CAP_IPC_LOCK)); + + /* Prepare only metadata: failure must preserve every legacy pin/charge. */ + provider = vfio_fragments_prepare_native(iommu, dma); + if (IS_ERR(provider)) + return PTR_ERR(provider); + ret = vfio_change_dma_mm_pages(dma, current->group_leader, current->mm, + capable(CAP_IPC_LOCK), vfio_fragments_native_pages(provider)); + if (ret) + vfio_fragments_abort_native(provider); + else + vfio_fragments_commit_native(dma, provider); + return ret; +} + static int vfio_dma_do_map(struct vfio_iommu *iommu, struct vfio_iommu_type1_dma_map *map) { @@ -1713,8 +1810,6 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu, pgsize = (size_t)1 << __ffs(iommu->pgsize_bitmap); - WARN_ON((pgsize - 1) & PAGE_MASK); - if ((size | iova | vaddr) & (pgsize - 1)) { ret = -EINVAL; goto out_unlock; @@ -1728,7 +1823,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu, dma->size != size) { ret = -EINVAL; } else { - ret = vfio_change_dma_owner(dma); + ret = vfio_change_dma_owner(iommu, dma, vaddr); if (ret) goto out_unlock; dma->vaddr = vaddr; @@ -1781,6 +1876,11 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu, /* Insert zero-sized and grow as we map chunks of it */ vfio_link_dma(iommu, dma); + ret = vfio_fragments_init(dma, size); + if (ret) { + vfio_remove_dma(iommu, dma); + goto out_unlock; + } /* Don't pin and map if container doesn't contain IOMMU capable domain*/ if (list_empty(&iommu->domain_list)) @@ -1789,7 +1889,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu, ret = vfio_pin_map_dma(iommu, dma, size); if (!ret && iommu->dirty_page_tracking) { - ret = vfio_dma_bitmap_alloc(dma, pgsize); + ret = vfio_dma_bitmap_alloc(dma, vfio_dirty_pgsize(iommu)); if (ret) vfio_remove_dma(iommu, dma); } @@ -1822,6 +1922,14 @@ static int vfio_iommu_replay(struct vfio_iommu *iommu, size_t pos = 0; dma = rb_entry(n, struct vfio_dma, node); + if (dma->fragments) { + ret = vfio_fragments_acquire(dma, 0, dma->size); + if (!ret) + ret = vfio_fragments_map_domain(dma, domain->domain); + if (ret) + goto unwind; + continue; + } while (pos < dma->size) { dma_addr_t iova = dma->iova + pos; @@ -1906,6 +2014,11 @@ static int vfio_iommu_replay(struct vfio_iommu *iommu, struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); size_t pos = 0; + if (dma->fragments) { + iommu_unmap(domain->domain, dma->iova, dma->size); + vfio_fragments_release_unused(dma); + continue; + } if (dma->iommu_mapped) { iommu_unmap(domain->domain, dma->iova, dma->size); continue; @@ -2442,6 +2555,10 @@ static void vfio_iommu_unmap_unpin_reaccount(struct vfio_iommu *iommu) long locked = 0, unlocked = 0; dma = rb_entry(n, struct vfio_dma, node); + if (dma->fragments) { + vfio_unmap_unpin(iommu, dma, true); + continue; + } unlocked += vfio_unmap_unpin(iommu, dma, false); p = rb_first(&dma->pfn_list); for (; p; p = rb_next(p)) { @@ -2646,7 +2763,8 @@ static void *vfio_iommu_type1_open(unsigned long arg) mutex_init(&iommu->lock); mutex_init(&iommu->device_list_lock); INIT_LIST_HEAD(&iommu->device_list); - iommu->pgsize_bitmap = PAGE_MASK; + iommu->pgsize_bitmap = IS_ENABLED(CONFIG_MM_SUBPAGE) ? + GENMASK(BITS_PER_LONG - 1, 12) : PAGE_MASK; INIT_LIST_HEAD(&iommu->emulated_iommu_groups); return iommu; @@ -2812,7 +2930,7 @@ static int vfio_iommu_migration_build_caps(struct vfio_iommu *iommu, cap_mig.flags = 0; /* support minimum pgsize */ - cap_mig.pgsize_bitmap = (size_t)1 << __ffs(iommu->pgsize_bitmap); + cap_mig.pgsize_bitmap = vfio_dirty_pgsize(iommu); cap_mig.max_dirty_bitmap_size = DIRTY_BITMAP_SIZE_MAX; return vfio_info_add_capability(caps, &cap_mig.header, sizeof(cap_mig)); @@ -2910,6 +3028,63 @@ static int vfio_iommu_type1_map_dma(struct vfio_iommu *iommu, return vfio_dma_do_map(iommu, &map); } +#ifdef CONFIG_COMPAT +struct compat_vfio_bitmap { + compat_u64 pgsize; + compat_u64 size; + compat_uptr_t data; +}; +#endif + +static size_t vfio_bitmap_size(void) +{ +#ifdef CONFIG_COMPAT + /* IA32 differs from AArch32/x32 in its u64 alignment and tail padding. */ + if (in_compat_syscall() && compat_need_64bit_alignment_fixup()) + return sizeof(struct compat_vfio_bitmap); +#endif + return sizeof(struct vfio_bitmap); +} + +static size_t vfio_dirty_bitmap_get_size(void) +{ + return offsetof(struct vfio_iommu_type1_dirty_bitmap_get, bitmap) + + vfio_bitmap_size(); +} + +static int vfio_bitmap_from_user(struct vfio_bitmap *bitmap, void __user *user) +{ +#ifdef CONFIG_COMPAT + if (in_compat_syscall()) { + compat_uptr_t data; + unsigned long offset = offsetof(struct vfio_bitmap, data); + + /* Read only the pointer field, never its trailing compat padding. */ + if (copy_from_user(bitmap, user, offset) || + get_user(data, (compat_uptr_t __user *)(user + offset))) + return -EFAULT; + bitmap->data = compat_ptr(data); + return 0; + } +#endif + return copy_from_user(bitmap, user, sizeof(*bitmap)) ? -EFAULT : 0; +} + +static int vfio_dirty_bitmap_get_from_user( + struct vfio_iommu_type1_dirty_bitmap_get *range, void __user *user) +{ +#ifdef CONFIG_COMPAT + if (in_compat_syscall()) { + unsigned long offset = offsetof(typeof(*range), bitmap); + + if (copy_from_user(range, user, offset)) + return -EFAULT; + return vfio_bitmap_from_user(&range->bitmap, user + offset); + } +#endif + return copy_from_user(range, user, sizeof(*range)) ? -EFAULT : 0; +} + static int vfio_iommu_type1_unmap_dma(struct vfio_iommu *iommu, unsigned long arg) { @@ -2937,13 +3112,12 @@ static int vfio_iommu_type1_unmap_dma(struct vfio_iommu *iommu, if (unmap.flags & VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP) { unsigned long pgshift; - if (unmap.argsz < (minsz + sizeof(bitmap))) + if (unmap.argsz < minsz + vfio_bitmap_size()) return -EINVAL; - if (copy_from_user(&bitmap, - (void __user *)(arg + minsz), - sizeof(bitmap))) - return -EFAULT; + ret = vfio_bitmap_from_user(&bitmap, (void __user *)(arg + minsz)); + if (ret) + return ret; if (!access_ok((void __user *)bitmap.data, bitmap.size)) return -EINVAL; @@ -2992,7 +3166,7 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu, size_t pgsize; mutex_lock(&iommu->lock); - pgsize = 1 << __ffs(iommu->pgsize_bitmap); + pgsize = vfio_dirty_pgsize(iommu); if (!iommu->dirty_page_tracking) { ret = vfio_dma_bitmap_alloc_all(iommu, pgsize); if (!ret) @@ -3015,12 +3189,12 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu, size_t size, iommu_pgsize; dma_addr_t iova, iova_end; - if (!data_size || data_size < sizeof(range)) + if (data_size < vfio_dirty_bitmap_get_size()) return -EINVAL; - if (copy_from_user(&range, (void __user *)(arg + minsz), - sizeof(range))) - return -EFAULT; + ret = vfio_dirty_bitmap_get_from_user(&range, (void __user *)(arg + minsz)); + if (ret) + return ret; iova = range.iova; size = range.size; @@ -3046,7 +3220,7 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu, mutex_lock(&iommu->lock); - iommu_pgsize = (size_t)1 << __ffs(iommu->pgsize_bitmap); + iommu_pgsize = vfio_dirty_pgsize(iommu); /* allow only smallest supported pgsize */ if (range.bitmap.pgsize != iommu_pgsize) { @@ -3112,11 +3286,12 @@ static void vfio_iommu_type1_register_device(void *iommu_data, * Holding both locks here allows avoiding the device_list_lock in * several fast paths. See vfio_notify_dma_unmap() */ - mutex_lock(&iommu->lock); + /* Unmap callbacks hold device_list_lock while reentering iommu->lock. */ mutex_lock(&iommu->device_list_lock); + mutex_lock(&iommu->lock); list_add(&vdev->iommu_entry, &iommu->device_list); - mutex_unlock(&iommu->device_list_lock); mutex_unlock(&iommu->lock); + mutex_unlock(&iommu->device_list_lock); } static void vfio_iommu_type1_unregister_device(void *iommu_data, @@ -3127,11 +3302,12 @@ static void vfio_iommu_type1_unregister_device(void *iommu_data, if (!vdev->ops->dma_unmap) return; - mutex_lock(&iommu->lock); + /* Unmap callbacks hold device_list_lock while reentering iommu->lock. */ mutex_lock(&iommu->device_list_lock); + mutex_lock(&iommu->lock); list_del(&vdev->iommu_entry); - mutex_unlock(&iommu->device_list_lock); mutex_unlock(&iommu->lock); + mutex_unlock(&iommu->device_list_lock); } static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, @@ -3156,6 +3332,22 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, return -EPERM; mm = dma->mm; + if (dma->fragments) { + int ret; + + if (!kthread && current->mm != mm) + return -EFAULT; + offset = user_iova - dma->iova; + count = min(count, dma->size - offset); + ret = vfio_fragments_rw(dma, offset, data, count, write, copied); + if (*copied && write && iommu->dirty_page_tracking) { + unsigned long shift = __ffs(vfio_dirty_pgsize(iommu)); + + bitmap_set(dma->bitmap, offset >> shift, + ((offset + *copied - 1) >> shift) - (offset >> shift) + 1); + } + return ret; + } if (!mmget_not_zero(mm)) return -EPERM; @@ -3175,7 +3367,7 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, *copied = copy_to_user((void __user *)vaddr, data, count) ? 0 : count; if (*copied && iommu->dirty_page_tracking) { - unsigned long pgshift = __ffs(iommu->pgsize_bitmap); + unsigned long pgshift = __ffs(vfio_dirty_pgsize(iommu)); /* * Bitmap populated with the smallest supported page * size @@ -3281,3 +3473,13 @@ MODULE_VERSION(DRIVER_VERSION); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR(DRIVER_AUTHOR); MODULE_DESCRIPTION(DRIVER_DESC); + +#ifdef CONFIG_VFIO_TYPE1_ACCOUNT_KUNIT_TEST +#include "vfio_iommu_type1_account_test.c" +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_ARM64_USER4K_EXPERIMENTAL) +#include "vfio_iommu_type1_fragments_test.c" +#ifdef CONFIG_ARM64_USER4K_VFIO_IOCTL_TEST +#include "vfio_iommu_type1_ioctl_test.c" +#endif +#endif +#endif diff --git a/drivers/vfio/vfio_iommu_type1_account_test.c b/drivers/vfio/vfio_iommu_type1_account_test.c new file mode 100644 index 00000000000000..5447258ce49b8c --- /dev/null +++ b/drivers/vfio/vfio_iommu_type1_account_test.c @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Included by type1 so the tests exercise the actual accounting helpers. */ +#include +#include + +struct vfio_account_mm { + struct mm_struct *mm; + bool live; +}; + +static void vfio_account_mm_free(void *data) +{ + struct vfio_account_mm *ctx = data; + + if (ctx->live) + mmput(ctx->mm); + mmdrop(ctx->mm); +} + +static struct vfio_account_mm *vfio_account_mm_new(struct kunit *test, + unsigned int shift) +{ + struct vfio_account_mm *ctx; + + ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return NULL; + ctx->mm = mm_alloc(); + if (!ctx->mm) + return NULL; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + if (shift != PAGE_SHIFT) { + pgd_t *old = ctx->mm->pgd, *pgd; + unsigned long flag = shift == 12 ? MMCF_USER_4K : + shift == 14 ? MMCF_USER_16K : MMCF_USER_64K; + + ctx->mm->context.flags |= flag; + pgd = pgd_alloc(ctx->mm); + ctx->mm->context.flags &= ~flag; + if (!pgd) { + mmput(ctx->mm); + return NULL; + } + pgd_free(ctx->mm, old); + ctx->mm->pgd = pgd; + ctx->mm->context.flags |= flag; + } +#endif + ctx->live = true; + mmgrab(ctx->mm); + if (kunit_add_action_or_reset(test, vfio_account_mm_free, ctx)) + return NULL; + return ctx; +} + +static void vfio_account_dma_free(void *data) +{ + struct vfio_dma *dma = data; + + /* Cleanup must also work with the negative-control implementation. */ + if (dma->locked_vm) + mm_lock_acct(dma->task, dma->mm, true, -dma->locked_vm); + mmdrop(dma->mm); + put_task_struct(dma->task); +} + +static struct vfio_dma *vfio_account_dma_new(struct kunit *test, + struct vfio_account_mm *ctx) +{ + struct vfio_dma *dma = kunit_kzalloc(test, sizeof(*dma), GFP_KERNEL); + + if (!dma) + return NULL; + dma->mm = ctx->mm; + mmgrab(dma->mm); + dma->task = get_task_struct(current->group_leader); + dma->lock_cap = true; + if (kunit_add_action_or_reset(test, vfio_account_dma_free, dma)) + return NULL; + return dma; +} + +static void vfio_account_exit_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + 12, 14, 16, +#endif + }; + + for (unsigned int i = 0; i < ARRAY_SIZE(shifts); i++) { + struct vfio_account_mm *ctx; + struct vfio_dma *dma; + + if (i && shifts[i] == PAGE_SHIFT) + continue; + ctx = vfio_account_mm_new(test, shifts[i]); + KUNIT_ASSERT_NOT_NULL(test, ctx); + KUNIT_ASSERT_EQ(test, mm_page_shift(ctx->mm), shifts[i]); + dma = vfio_account_dma_new(test, ctx); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_EQ(test, vfio_lock_acct(dma, 2, true), 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(ctx->mm), 2UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 2UL * PAGE_SIZE); + ctx->live = false; + mmput(ctx->mm); + KUNIT_ASSERT_EQ(test, atomic_read(&ctx->mm->mm_users), 0); + KUNIT_EXPECT_EQ(test, vfio_lock_acct(dma, 1, true), -ESRCH); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + KUNIT_EXPECT_EQ(test, vfio_lock_acct(dma, -1, true), 0); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(ctx->mm), 1UL); + KUNIT_EXPECT_EQ(test, vfio_lock_acct(dma, -1, true), 0); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 0UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + KUNIT_EXPECT_EQ(test, atomic_read(&ctx->mm->mm_users), 0); + kunit_info(test, "VFIO %uK user/%luK native dead-mm refunds checked", + 1U << (shifts[i] - 10), PAGE_SIZE >> 10); + } +} + +static void vfio_account_owner_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + 12, 14, 16, +#endif + }; + + for (unsigned int reverse = 0; reverse < 2; reverse++) { + for (unsigned int i = 0; i < ARRAY_SIZE(shifts); i++) { + struct vfio_account_mm *old, *next; + struct vfio_dma *dma; + + if ((i || reverse) && shifts[i] == PAGE_SHIFT) + continue; + old = vfio_account_mm_new(test, reverse ? shifts[i] : PAGE_SHIFT); + next = vfio_account_mm_new(test, reverse ? PAGE_SHIFT : shifts[i]); + KUNIT_ASSERT_NOT_NULL(test, old); + KUNIT_ASSERT_NOT_NULL(test, next); + dma = vfio_account_dma_new(test, old); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_EQ(test, vfio_lock_acct(dma, 3, true), 0); + old->live = false; + mmput(old->mm); + KUNIT_ASSERT_EQ(test, atomic_read(&old->mm->mm_users), 0); + if (task_rlimit(dma->task, RLIMIT_MEMLOCK) != RLIM_INFINITY) { + unsigned long budget = task_rlimit(dma->task, RLIMIT_MEMLOCK) >> + MM_ACCOUNT_SHIFT; + + /* Model a recipient whose existing pins consume its budget. */ + next->mm->locked_vm = budget; + KUNIT_EXPECT_EQ(test, + vfio_change_dma_mm(dma, dma->task, next->mm, false), + -ENOMEM); + KUNIT_EXPECT_PTR_EQ(test, dma->mm, old->mm); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(old->mm), 3UL); + KUNIT_EXPECT_EQ(test, next->mm->locked_vm, budget); + next->mm->locked_vm = 0; + } + KUNIT_ASSERT_EQ(test, + vfio_change_dma_mm(dma, dma->task, next->mm, true), 0); + KUNIT_EXPECT_PTR_EQ(test, dma->mm, next->mm); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(old->mm), 0UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(next->mm), 3UL); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 3UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(next->mm), 3UL * PAGE_SIZE); + /* Same-mm ownership is idempotent; it must not double the charge. */ + KUNIT_ASSERT_EQ(test, + vfio_change_dma_mm(dma, dma->task, next->mm, true), 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(next->mm), 3UL); + KUNIT_ASSERT_EQ(test, vfio_lock_acct(dma, -3, true), 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(next->mm), 0UL); + kunit_info(test, "VFIO dead %uK mm -> %uK mm ownership refund checked", + 1U << (mm_page_shift(old->mm) - 10), + 1U << (mm_page_shift(next->mm) - 10)); + } + } +} + +static struct kunit_case vfio_account_cases[] = { + KUNIT_CASE(vfio_account_exit_test), + KUNIT_CASE(vfio_account_owner_test), + {} +}; + +static struct kunit_suite vfio_account_suite = { + .name = "vfio-type1-account", + .test_cases = vfio_account_cases, +}; + +kunit_test_suite(vfio_account_suite); diff --git a/drivers/vfio/vfio_iommu_type1_fragments.c b/drivers/vfio/vfio_iommu_type1_fragments.c new file mode 100644 index 00000000000000..22d46a1b805a7b --- /dev/null +++ b/drivers/vfio/vfio_iommu_type1_fragments.c @@ -0,0 +1,609 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Included by type1: retained byte ranges, rather than IOMMU PFNs, own pins. */ +#ifdef CONFIG_MM_SUBPAGE +struct vfio_fragment { + struct rb_node node; + unsigned long start; + struct user_page_fragment pin; +}; + +struct vfio_fragment_provider { + struct rb_root records; + struct xarray physical_pages; + unsigned long npinned; +}; + +static phys_addr_t vfio_fragment_phys(const struct user_page_fragment *pin) +{ + return PFN_PHYS(folio_pfn(pin->folio)) + pin->offset; +} + +/* Returns a containing range, or the first range after the requested byte. */ +static struct vfio_fragment *vfio_fragment_find(struct vfio_dma *dma, + unsigned long offset) +{ + struct rb_node *node = dma->fragments->records.rb_node; + struct vfio_fragment *next = NULL; + + while (node) { + struct vfio_fragment *record = rb_entry(node, struct vfio_fragment, node); + + if (offset < record->start) { + next = record; + node = node->rb_left; + } else if (offset - record->start >= record->pin.length) { + node = node->rb_right; + } else { + return record; + } + } + return next; +} + +static int vfio_fragments_init(struct vfio_dma *dma, size_t length) +{ + struct vfio_fragment_provider *provider; + + if (mm_page_size(dma->mm) == PAGE_SIZE && + IS_ALIGNED(dma->vaddr | dma->iova | length, PAGE_SIZE)) + return 0; + provider = kzalloc_obj(*provider, GFP_KERNEL_ACCOUNT); + if (!provider) + return -ENOMEM; + provider->records = RB_ROOT; + xa_init(&provider->physical_pages); + dma->fragments = provider; + return 0; +} + +static int vfio_fragment_store(struct vfio_dma *dma, unsigned long start, + struct user_page_fragment *pin) +{ + struct vfio_fragment_provider *provider = dma->fragments; + struct rb_node **link = &provider->records.rb_node, *parent = NULL; + unsigned long pfn = PHYS_PFN(vfio_fragment_phys(pin)); + unsigned long count = xa_to_value(xa_load(&provider->physical_pages, pfn)); + struct vfio_fragment *record; + int ret; + + record = kmalloc_obj(*record, GFP_KERNEL_ACCOUNT); + if (!record) + return -ENOMEM; + ret = xa_err(xa_store(&provider->physical_pages, pfn, xa_mk_value(count + 1), + GFP_KERNEL_ACCOUNT)); + if (ret) { + kfree(record); + return ret; + } + while (*link) { + struct vfio_fragment *cur = rb_entry(*link, struct vfio_fragment, node); + + parent = *link; + link = start < cur->start ? &(*link)->rb_left : &(*link)->rb_right; + } + record->start = start; + record->pin = *pin; + rb_link_node(&record->node, parent, link); + rb_insert_color(&record->node, &provider->records); + if (!count) + provider->npinned++; + *pin = (struct user_page_fragment) {}; + return 0; +} + +/* These descriptors borrow existing native GUP pins until owner commit. */ +static int vfio_fragment_native_pin(unsigned long pfn, struct user_page_fragment *pin) +{ + struct page *page; + + if (!pfn_valid(pfn)) + return -EOPNOTSUPP; + page = pfn_to_page(pfn); + if (PageReserved(page)) + return -EOPNOTSUPP; + *pin = (struct user_page_fragment) { + .folio = page_folio(page), + .offset = page_to_phys(page) - PFN_PHYS(folio_pfn(page_folio(page))), + .length = PAGE_SIZE, + .pinned = true, + }; + return 0; +} + +/* No PIN, GET or account reference was acquired while preparing metadata. */ +static void vfio_fragments_abort_native(struct vfio_fragment_provider *provider) +{ + struct rb_node *node; + + while ((node = rb_first(&provider->records))) { + struct vfio_fragment *record = rb_entry(node, struct vfio_fragment, node); + + rb_erase(node, &provider->records); + kfree(record); + } + xa_destroy(&provider->physical_pages); + kfree(provider); +} + +static struct vfio_fragment_provider * +vfio_fragments_prepare_native(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ + struct vfio_fragment_provider *provider; + struct vfio_dma stage = {}; + struct vfio_domain *domain = NULL; + struct rb_node *node; + int ret; + + if (mm_page_size(dma->mm) != PAGE_SIZE || dma->has_rsvd) + return ERR_PTR(-EOPNOTSUPP); + if (dma->iommu_mapped) { + if (list_empty(&iommu->domain_list)) + return ERR_PTR(-EFAULT); + domain = list_first_entry(&iommu->domain_list, struct vfio_domain, next); + } + provider = kzalloc_obj(*provider, GFP_KERNEL_ACCOUNT); + if (!provider) + return ERR_PTR(-ENOMEM); + provider->records = RB_ROOT; + xa_init(&provider->physical_pages); + stage.fragments = provider; + /* One descriptor owns each original native GUP pin, including aliases. */ + if (domain) { + for (unsigned long offset = 0; offset < dma->size; offset += PAGE_SIZE) { + phys_addr_t phys = iommu_iova_to_phys(domain->domain, dma->iova + offset); + struct user_page_fragment pin; + + if (!phys || !IS_ALIGNED(phys, PAGE_SIZE)) { + ret = -EFAULT; + goto out; + } + ret = vfio_fragment_native_pin(PHYS_PFN(phys), &pin); + if (!ret) + ret = vfio_fragment_store(&stage, offset, &pin); + if (ret) + goto out; + cond_resched(); + } + } + for (node = rb_first(&dma->pfn_list); node; node = rb_next(node)) { + struct vfio_pfn *vpfn = rb_entry(node, struct vfio_pfn, node); + struct user_page_fragment pin; + + ret = vfio_fragment_native_pin(vpfn->pfn, &pin); + if (ret) + goto out; + if (!domain) { + /* Without a domain, the external pin is the cached buffer. */ + ret = vfio_fragment_store(&stage, vpfn->iova - dma->iova, &pin); + } else { + unsigned long count = xa_to_value(xa_load(&provider->physical_pages, + vpfn->pfn)); + + /* The external GUP reference may identify a different page. */ + ret = xa_err(xa_store(&provider->physical_pages, vpfn->pfn, + xa_mk_value(count + 1), GFP_KERNEL_ACCOUNT)); + if (!ret && !count) + provider->npinned++; + } + if (ret) + goto out; + } + return provider; +out: + vfio_fragments_abort_native(provider); + return ERR_PTR(ret); +} + +static unsigned long vfio_fragments_native_pages(struct vfio_fragment_provider *provider) +{ + return provider->npinned; +} + +static void vfio_fragments_commit_native(struct vfio_dma *dma, + struct vfio_fragment_provider *provider) +{ + struct rb_node *node; + + /* Mapped legacy external consumers own separate native GUP references. */ + if (dma->iommu_mapped) { + for (node = rb_first(&dma->pfn_list); node; node = rb_next(node)) + rb_entry(node, struct vfio_pfn, node)->native_pin = true; + } + dma->fragments = provider; +} + +static void vfio_fragments_put_native_pin(struct vfio_dma *dma, unsigned long pfn) +{ + struct vfio_fragment_provider *provider = dma->fragments; + unsigned long count = xa_to_value(xa_load(&provider->physical_pages, pfn)); + struct user_page_fragment pin; + + if (WARN_ON(!count || vfio_fragment_native_pin(pfn, &pin))) + return; + if (count == 1) { + xa_erase(&provider->physical_pages, pfn); + provider->npinned--; + } else { + xa_store(&provider->physical_pages, pfn, xa_mk_value(count - 1), GFP_NOWAIT); + } + release_user_fragments(&pin, 1, dma->prot & IOMMU_WRITE); + if (provider->npinned < dma->locked_vm) + vfio_lock_acct(dma, -(long)(dma->locked_vm - provider->npinned), true); +} + +static bool vfio_fragments_external_overlap(struct vfio_dma *dma, + dma_addr_t start, unsigned long length) +{ + struct rb_node *node = dma->pfn_list.rb_node; + + while (node) { + struct vfio_pfn *vpfn = rb_entry(node, struct vfio_pfn, node); + + if (start + length <= vpfn->iova) + node = node->rb_left; + else if (start >= vpfn->iova + vpfn->length) + node = node->rb_right; + else + return true; + } + return false; +} + +/* Device unmaps and their IOTLB synchronization must precede this call. */ +static void vfio_fragments_release_range(struct vfio_dma *dma, unsigned long start, + unsigned long length) +{ + struct vfio_fragment_provider *provider = dma->fragments; + struct vfio_fragment *first = vfio_fragment_find(dma, start); + struct rb_node *node = first ? &first->node : NULL; + unsigned long end = start + length; + + if (dma->iommu_mapped) + return; + while (node) { + struct vfio_fragment *record = rb_entry(node, struct vfio_fragment, node); + dma_addr_t iova = dma->iova + record->start; + unsigned long pfn, count; + + if (record->start >= end) + break; + node = rb_next(node); + /* Native-page consumers retain all records overlapping their page. */ + if (vfio_fragments_external_overlap(dma, iova, record->pin.length)) + continue; + pfn = PHYS_PFN(vfio_fragment_phys(&record->pin)); + count = xa_to_value(xa_load(&provider->physical_pages, pfn)); + if (WARN_ON(!count)) + continue; + if (count == 1) { + xa_erase(&provider->physical_pages, pfn); + provider->npinned--; + } else { + xa_store(&provider->physical_pages, pfn, xa_mk_value(count - 1), + GFP_NOWAIT); + } + rb_erase(&record->node, &provider->records); + release_user_fragments(&record->pin, 1, dma->prot & IOMMU_WRITE); + kfree(record); + cond_resched(); + } + if (provider->npinned < dma->locked_vm) + vfio_lock_acct(dma, -(long)(dma->locked_vm - provider->npinned), true); +} + +static void vfio_fragments_release_unused(struct vfio_dma *dma) +{ + vfio_fragments_release_range(dma, 0, dma->size); +} + +/* Cached records survive source VA replacement and the last source mm user. */ +static int vfio_fragments_acquire(struct vfio_dma *dma, unsigned long start, + unsigned long length) +{ + unsigned long end = start + length, original = start; + unsigned long capacity = PAGE_SIZE / sizeof(struct user_page_fragment); + struct user_page_fragment *batch = NULL; + bool live = false; + int ret = 0; + + while (start < end) { + struct vfio_fragment *record = vfio_fragment_find(dma, start); + unsigned long stop = end; + long count; + + if (record && record->start <= start) { + start = record->start + record->pin.length; + continue; + } + if (record) + stop = min(stop, record->start); + if (!live) { + if (!mmget_not_zero(dma->mm)) { + ret = -EFAULT; + goto out; + } + live = true; + batch = kmalloc(PAGE_SIZE, GFP_KERNEL_ACCOUNT); + if (!batch) { + ret = -ENOMEM; + goto out; + } + } + count = pin_user_fragments_remote(dma->mm, dma->vaddr + start, + stop - start, FOLL_LONGTERM | (dma->prot & IOMMU_WRITE ? FOLL_WRITE : 0), + batch, capacity); + if (count <= 0) { + ret = count ?: -EFAULT; + goto out; + } + for (unsigned long i = 0; i < count; i++) { + unsigned long bytes = batch[i].length; + + if (WARN_ON(!bytes || bytes > stop - start || + offset_in_page(vfio_fragment_phys(&batch[i])) > + PAGE_SIZE - bytes)) + ret = -EFAULT; + else + ret = vfio_fragment_store(dma, start, &batch[i]); + if (ret) { + release_user_fragments(batch + i, count - i, false); + goto out; + } + start += bytes; + } + cond_resched(); + } + if (dma->fragments->npinned > dma->locked_vm) + ret = vfio_lock_acct(dma, dma->fragments->npinned - dma->locked_vm, false); +out: + kfree(batch); + if (live) + mmput(dma->mm); + if (ret) + vfio_fragments_release_range(dma, original, length); + return ret; +} + +/* Merge physical byte runs while retaining each original pin identity. */ +static int vfio_fragments_map_domain(struct vfio_dma *dma, struct iommu_domain *domain) +{ + unsigned long done = 0, min_size = 1UL << __ffs(domain->pgsize_bitmap); + int ret = 0; + + while (done < dma->size) { + struct vfio_fragment *record = vfio_fragment_find(dma, done); + unsigned long bytes, limit = dma->size - done; + phys_addr_t phys; + + if (WARN_ON(!record || record->start > done)) { + ret = -EFAULT; + goto out; + } + if (disable_hugepages) + limit = min(limit, PAGE_SIZE); + phys = vfio_fragment_phys(&record->pin) + done - record->start; + bytes = min(record->pin.length - (done - record->start), limit); + while (bytes < limit) { + record = vfio_fragment_find(dma, done + bytes); + if (!record || record->start != done + bytes || + vfio_fragment_phys(&record->pin) != phys + bytes) + break; + bytes += min_t(unsigned long, record->pin.length, limit - bytes); + } + if (!IS_ALIGNED((dma->iova + done) | phys | bytes, min_size)) { + ret = -EINVAL; + goto out; + } + ret = iommu_map(domain, dma->iova + done, phys, bytes, + dma->prot | IOMMU_CACHE, GFP_KERNEL_ACCOUNT); + if (ret) + goto out; + done += bytes; + cond_resched(); + } +out: + if (ret && done) + iommu_unmap(domain, dma->iova, done); + return ret; +} + +static int vfio_fragments_map_all(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ + struct vfio_domain *domain; + int ret; + + ret = vfio_fragments_acquire(dma, 0, dma->size); + if (ret) + return ret; + list_for_each_entry(domain, &iommu->domain_list, next) { + ret = vfio_fragments_map_domain(dma, domain->domain); + if (ret) + goto unwind; + } + dma->iommu_mapped = true; + return 0; +unwind: + list_for_each_entry_continue_reverse(domain, &iommu->domain_list, next) + iommu_unmap(domain->domain, dma->iova, dma->size); + vfio_fragments_release_unused(dma); + return ret; +} + +static void vfio_fragments_unmap(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ + struct vfio_domain *domain; + + if (dma->iommu_mapped) { + list_for_each_entry(domain, &iommu->domain_list, next) + WARN_ON(iommu_unmap(domain->domain, dma->iova, dma->size) != dma->size); + dma->iommu_mapped = false; + } + vfio_fragments_release_unused(dma); +} + +static void vfio_fragments_destroy(struct vfio_dma *dma) +{ + if (!dma->fragments) + return; + vfio_fragments_release_unused(dma); + if (WARN_ON(!RB_EMPTY_ROOT(&dma->fragments->records))) + return; + xa_destroy(&dma->fragments->physical_pages); + kfree(dma->fragments); + dma->fragments = NULL; +} + +/* The output page has no offset field: validate every byte and its native offset. */ +static int vfio_fragments_external(struct vfio_dma *dma, dma_addr_t iova, + unsigned long length, unsigned long *pfn) +{ + unsigned long first = iova - dma->iova, done = 0; + phys_addr_t phys = 0; + int ret; + + ret = vfio_fragments_acquire(dma, first, length); + if (ret) + return ret; + while (done < length) { + struct vfio_fragment *record = vfio_fragment_find(dma, first + done); + phys_addr_t next; + unsigned long offset; + + if (WARN_ON(!record || record->start > first + done)) { + ret = -EFAULT; + goto out; + } + offset = first + done - record->start; + next = vfio_fragment_phys(&record->pin) + offset; + if (!done) + phys = next; + if (offset_in_page(phys) != offset_in_page(iova) || next != phys + done) { + ret = -EINVAL; + goto out; + } + done += min_t(unsigned long, record->pin.length - offset, length - done); + } + *pfn = PHYS_PFN(phys); + ret = vfio_add_to_pfn_list(dma, iova, *pfn, length); +out: + if (ret) + vfio_fragments_release_range(dma, first, length); + return ret; +} + +/* CPU-only missing ranges take GET references, not new long-term DMA pins. */ +static int vfio_fragments_rw(struct vfio_dma *dma, unsigned long offset, void *data, + unsigned long length, bool write, size_t *copied) +{ + bool live = false; + int ret = 0; + + while (length) { + struct vfio_fragment *record = vfio_fragment_find(dma, offset); + struct user_page_fragment temp = {}, *pin; + unsigned long in_record = 0, bytes = length; + void *addr; + + if (record && record->start <= offset) { + pin = &record->pin; + in_record = offset - record->start; + } else { + long count; + + if (record) + bytes = min(bytes, record->start - offset); + if (!live) { + if (!mmget_not_zero(dma->mm)) { + ret = -EFAULT; + break; + } + live = true; + } + count = get_user_fragments_remote(dma->mm, dma->vaddr + offset, bytes, + write ? FOLL_WRITE : 0, &temp, 1); + if (count <= 0) { + ret = count ?: -EFAULT; + break; + } + pin = &temp; + } + bytes = min_t(unsigned long, bytes, pin->length - in_record); + addr = kmap_local_folio(pin->folio, pin->offset + in_record); + if (write) + memcpy(addr, data, bytes); + else + memcpy(data, addr, bytes); + kunmap_local(addr); + if (write && record && record->start <= offset) + mark_user_fragment_dirty(pin); + release_user_fragments(&temp, 1, write); + data += bytes; + offset += bytes; + length -= bytes; + *copied += bytes; + } + if (live) + mmput(dma->mm); + return ret; +} +#else +static struct vfio_fragment_provider * +vfio_fragments_prepare_native(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ + return ERR_PTR(-EOPNOTSUPP); +} +static void vfio_fragments_abort_native(struct vfio_fragment_provider *provider) +{ +} +static unsigned long vfio_fragments_native_pages(struct vfio_fragment_provider *provider) +{ + return 0; +} +static void vfio_fragments_commit_native(struct vfio_dma *dma, + struct vfio_fragment_provider *provider) +{ +} +static void vfio_fragments_put_native_pin(struct vfio_dma *dma, unsigned long pfn) +{ +} +static int vfio_fragments_init(struct vfio_dma *dma, size_t length) +{ + return 0; +} +static void vfio_fragments_release_range(struct vfio_dma *dma, unsigned long start, + unsigned long length) +{ +} +static void vfio_fragments_release_unused(struct vfio_dma *dma) +{ +} +static int vfio_fragments_acquire(struct vfio_dma *dma, unsigned long start, + unsigned long length) +{ + return -EOPNOTSUPP; +} +static int vfio_fragments_map_domain(struct vfio_dma *dma, struct iommu_domain *domain) +{ + return -EOPNOTSUPP; +} +static int vfio_fragments_map_all(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ + return -EOPNOTSUPP; +} +static void vfio_fragments_unmap(struct vfio_iommu *iommu, struct vfio_dma *dma) +{ +} +static void vfio_fragments_destroy(struct vfio_dma *dma) +{ +} +static int vfio_fragments_external(struct vfio_dma *dma, dma_addr_t iova, + unsigned long length, unsigned long *pfn) +{ + return -EOPNOTSUPP; +} +static int vfio_fragments_rw(struct vfio_dma *dma, unsigned long offset, void *data, + unsigned long length, bool write, size_t *copied) +{ + return -EOPNOTSUPP; +} +#endif diff --git a/drivers/vfio/vfio_iommu_type1_fragments_test.c b/drivers/vfio/vfio_iommu_type1_fragments_test.c new file mode 100644 index 00000000000000..60fffff10c6bee --- /dev/null +++ b/drivers/vfio/vfio_iommu_type1_fragments_test.c @@ -0,0 +1,1177 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Actual type1 lifecycle with recorded translations; no physical device DMA. */ +#include +#include +#include +#include + +#define VFIO_TEST_VA SZ_64M +#define VFIO_TEST_IOVA SZ_256M + +struct vfio_test_domain { + struct iommu_domain core; + struct vfio_domain type1; + struct xarray translations; + unsigned long mapped; + unsigned long fail_after; + unsigned long largest_map; + unsigned int syncs; + struct folio *observed; + struct kunit *test; +}; + +struct vfio_fragment_fixture { + struct vfio_iommu *iommu; + struct vfio_test_domain domains[2]; + struct vfio_iommu_group group; + struct vfio_device device; + struct kunit *test; + unsigned int external_refs; + unsigned int notifications; +}; + +static int vfio_test_domain_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t phys, size_t pgsize, size_t pgcount, int prot, + gfp_t gfp, size_t *mapped) +{ + struct vfio_test_domain *mock = container_of(domain, typeof(*mock), core); + unsigned long length = pgsize * pgcount; + + mock->largest_map = max(mock->largest_map, length); + for (unsigned long done = 0; done < length; done += SZ_4K) { + int ret; + + if (mock->fail_after && mock->mapped >= mock->fail_after) + return -ENOMEM; + ret = xa_insert(&mock->translations, (iova + done) >> 12, + xa_mk_value((phys + done) >> 12), gfp); + if (ret) + return ret; + mock->mapped += SZ_4K; + *mapped += SZ_4K; + } + return 0; +} + +static size_t vfio_test_domain_unmap(struct iommu_domain *domain, unsigned long iova, + size_t pgsize, size_t pgcount, struct iommu_iotlb_gather *gather) +{ + struct vfio_test_domain *mock = container_of(domain, typeof(*mock), core); + unsigned long length = pgsize * pgcount, done; + + for (done = 0; done < length; done += SZ_4K) { + if (!xa_erase(&mock->translations, (iova + done) >> 12)) + break; + mock->mapped -= SZ_4K; + } + return done; +} + +static phys_addr_t vfio_test_domain_translate(struct iommu_domain *domain, dma_addr_t iova) +{ + struct vfio_test_domain *mock = container_of(domain, typeof(*mock), core); + void *entry = xa_load(&mock->translations, iova >> 12); + + return entry ? ((phys_addr_t)xa_to_value(entry) << 12) + (iova & (SZ_4K - 1)) : 0; +} + +static void vfio_test_domain_sync(struct iommu_domain *domain, + struct iommu_iotlb_gather *gather) +{ + struct vfio_test_domain *mock = container_of(domain, typeof(*mock), core); + + mock->syncs++; + if (mock->observed) + KUNIT_EXPECT_TRUE(mock->test, folio_maybe_dma_pinned(mock->observed)); +} + +static const struct iommu_domain_ops vfio_test_domain_ops = { + .map_pages = vfio_test_domain_map, + .unmap_pages = vfio_test_domain_unmap, + .iova_to_phys = vfio_test_domain_translate, + .iotlb_sync = vfio_test_domain_sync, +}; + +static void vfio_test_notify(struct vfio_device *device, u64 iova, u64 length) +{ + struct vfio_fragment_fixture *f = container_of(device, typeof(*f), device); + + f->notifications++; + while (f->external_refs) { + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 1); + f->external_refs--; + } +} + +static const struct vfio_device_ops vfio_test_device_ops = { + .dma_unmap = vfio_test_notify, +}; + +static void vfio_fragment_fixture_free(void *ptr) +{ + struct vfio_fragment_fixture *f = ptr; + struct rb_node *node; + + while ((node = rb_first(&f->iommu->dma_list))) { + struct vfio_dma *dma = rb_entry(node, struct vfio_dma, node); + struct rb_node *pin; + + while ((pin = rb_first(&dma->pfn_list))) { + struct vfio_pfn *vpfn = rb_entry(pin, struct vfio_pfn, node); + + vfio_unpin_page_external(dma, vpfn->iova, + list_empty(&f->iommu->domain_list)); + } + vfio_remove_dma(f->iommu, dma); + } + for (unsigned int i = 0; i < ARRAY_SIZE(f->domains); i++) { + KUNIT_EXPECT_EQ(f->test, f->domains[i].mapped, 0UL); + KUNIT_EXPECT_TRUE(f->test, xa_empty(&f->domains[i].translations)); + xa_destroy(&f->domains[i].translations); + } + kfree(f->iommu); +} + +static struct vfio_fragment_fixture *vfio_fragment_fixture_new(struct kunit *test) +{ + struct vfio_fragment_fixture *f = kunit_kzalloc(test, sizeof(*f), GFP_KERNEL); + + if (!f) + return NULL; + f->test = test; + f->iommu = vfio_iommu_type1_open(VFIO_TYPE1v2_IOMMU); + if (IS_ERR(f->iommu)) + return NULL; + for (unsigned int i = 0; i < ARRAY_SIZE(f->domains); i++) { + struct vfio_test_domain *mock = &f->domains[i]; + + xa_init(&mock->translations); + mock->test = test; + mock->core.ops = &vfio_test_domain_ops; + mock->core.type = IOMMU_DOMAIN_UNMANAGED; + mock->core.pgsize_bitmap = SZ_4K | PAGE_SIZE; + mock->type1.domain = &mock->core; + INIT_LIST_HEAD(&mock->type1.next); + INIT_LIST_HEAD(&mock->type1.group_list); + } + f->group.iommu_group = (struct iommu_group *)f; + list_add(&f->group.next, &f->iommu->emulated_iommu_groups); + f->iommu->num_non_pinned_groups = 1; + f->device.ops = &vfio_test_device_ops; + vfio_iommu_type1_register_device(f->iommu, &f->device); + if (kunit_add_action_or_reset(test, vfio_fragment_fixture_free, f)) + return NULL; + return f; +} + +static void vfio_test_file_put(void *ptr) +{ + fput(ptr); +} + +static struct file *vfio_fragment_file(struct kunit *test, unsigned long length, + unsigned char value) +{ + struct file *file = shmem_file_setup("vfio-fragments", length, EMPTY_VMA_FLAGS); + unsigned char *data; + loff_t pos = 0; + + if (IS_ERR(file)) + return file; + if (kunit_add_action_or_reset(test, vfio_test_file_put, file)) + return ERR_PTR(-ENOMEM); + data = kunit_kmalloc(test, length, GFP_KERNEL); + if (!data) + return ERR_PTR(-ENOMEM); + memset(data, value, length); + if (kernel_write(file, data, length, &pos) != length) + return ERR_PTR(-EIO); + return file; +} + +static unsigned long vfio_fragment_source(struct vfio_account_mm *ctx, + struct file *file, unsigned long length) +{ + unsigned long addr; + + kthread_use_mm(ctx->mm); + addr = vm_mmap(file, VFIO_TEST_VA, length, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, 0); + kthread_unuse_mm(ctx->mm); + return addr; +} + +static int vfio_fragment_test_map(struct vfio_fragment_fixture *f, + struct vfio_account_mm *ctx, + unsigned long offset, unsigned long length) +{ + struct vfio_iommu_type1_dma_map map = { + .argsz = sizeof(map), + .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE, + .vaddr = VFIO_TEST_VA + offset, + .iova = VFIO_TEST_IOVA, + .size = length, + }; + int ret; + + kthread_use_mm(ctx->mm); + ret = vfio_dma_do_map(f->iommu, &map); + kthread_unuse_mm(ctx->mm); + return ret; +} + +static void vfio_fragments_lifetime_test(struct kunit *test) +{ + unsigned int shifts[] = { 12, 14 }; + + for (unsigned int i = 0; i < ARRAY_SIZE(shifts); i++) { + struct vfio_account_mm *ctx; + struct vfio_fragment_fixture *f; + struct vfio_dma *dma; + struct file *file; + struct page *page; + unsigned char data[64]; + struct vfio_iommu_type1_dma_unmap unmap = { + .argsz = sizeof(unmap), .iova = VFIO_TEST_IOVA, .size = 2 * PAGE_SIZE, + }; + struct vfio_bitmap bitmap = {}; + + if (shifts[i] == PAGE_SHIFT) + continue; + ctx = vfio_account_mm_new(test, shifts[i]); + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x71); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, 0, 2 * PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_NOT_NULL(test, dma->fragments); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + for (unsigned int d = 0; d < 2; d++) + f->domains[d].observed = vfio_fragment_find(dma, 0)->pin.folio; + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, 2 * PAGE_SIZE); + KUNIT_EXPECT_GE(test, f->domains[0].largest_map, PAGE_SIZE); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + 17, 1, IOMMU_READ | IOMMU_WRITE, &page), 1); + f->external_refs++; + KUNIT_EXPECT_EQ(test, page_to_phys(page), + iommu_iova_to_phys(&f->domains[0].core, VFIO_TEST_IOVA)); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + /* The user VA goes away; retained pins are still the DMA buffer. */ + ctx->live = false; + mmput(ctx->mm); + KUNIT_ASSERT_EQ(test, atomic_read(&ctx->mm->mm_users), 0); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, VFIO_TEST_IOVA, + data, sizeof(data), false), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(data, 0x71, sizeof(data)), NULL); + KUNIT_ASSERT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[1].type1), 0); + list_add_tail(&f->domains[1].type1.next, &f->iommu->domain_list); + KUNIT_EXPECT_EQ(test, f->domains[1].mapped, 2 * PAGE_SIZE); + KUNIT_EXPECT_EQ(test, iommu_iova_to_phys(&f->domains[0].core, VFIO_TEST_IOVA), + iommu_iova_to_phys(&f->domains[1].core, VFIO_TEST_IOVA)); + KUNIT_ASSERT_EQ(test, vfio_dma_do_unmap(f->iommu, &unmap, &bitmap), 0); + KUNIT_EXPECT_EQ(test, unmap.size, 2 * PAGE_SIZE); + KUNIT_EXPECT_EQ(test, f->notifications, 1U); + KUNIT_EXPECT_GT(test, f->domains[0].syncs, 0U); + KUNIT_EXPECT_GT(test, f->domains[1].syncs, 0U); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + kunit_release_action(test, vfio_fragment_fixture_free, f); + kunit_info(test, "type1 retained %uK user/%luK native pins after mm exit and replay", + 1U << (shifts[i] - 10), PAGE_SIZE >> 10); + } +} + +static void vfio_fragments_lazy_test(struct kunit *test) +{ + struct vfio_account_mm *ctx = vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f; + struct file *file; + struct vfio_dma *dma; + struct page *page; + unsigned char data[64]; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x52); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, 0, 2 * PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_NOT_NULL(test, dma->fragments); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 0UL); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, VFIO_TEST_IOVA, + data, sizeof(data), false), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(data, 0x52, sizeof(data)), NULL); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 0UL); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + 37, 1, IOMMU_READ, &page), 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA, 1, IOMMU_READ, &page), 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + ctx->live = false; + mmput(ctx->mm); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA, 1, IOMMU_READ, &page), 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + PAGE_SIZE, 1, IOMMU_READ, &page), -EFAULT); + for (unsigned int i = 0; i < 3; i++) + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 37, 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 0UL); + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&dma->fragments->records)); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + kunit_release_action(test, vfio_fragment_fixture_free, f); +} + +static void vfio_fragments_map_failure_test(struct kunit *test) +{ + struct vfio_account_mm *ctx = vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f; + struct file *file; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x25); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + list_add_tail(&f->domains[1].type1.next, &f->iommu->domain_list); + f->domains[1].fail_after = SZ_4K; + KUNIT_EXPECT_EQ(test, vfio_fragment_test_map(f, ctx, 0, 2 * PAGE_SIZE), -ENOMEM); + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&f->iommu->dma_list)); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, 0UL); + KUNIT_EXPECT_EQ(test, f->domains[1].mapped, 0UL); + f->domains[1].fail_after = 0; + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, 0, 2 * PAGE_SIZE), 0); + kunit_release_action(test, vfio_fragment_fixture_free, f); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); +} + +static void vfio_fragments_fine_test(struct kunit *test) +{ + unsigned int shifts[] = { PAGE_SHIFT, 12, 14 }; + + for (unsigned int i = 0; i < ARRAY_SIZE(shifts); i++) { + struct vfio_account_mm *ctx; + struct vfio_fragment_fixture *f; + struct vfio_dma *dma; + struct file *file; + struct page *page = NULL; + unsigned char data[64]; + unsigned long length = PAGE_SIZE + SZ_4K; + size_t copied = 0; + + if (i && shifts[i] == PAGE_SHIFT) + continue; + ctx = vfio_account_mm_new(test, shifts[i]); + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x64); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, SZ_4K, length), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_NOT_NULL(test, dma->fragments); + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, length); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + /* The physical base is offset inside a native page, not representable. */ + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA, 1, IOMMU_READ, &page), -EINVAL); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + PAGE_SIZE, 1, IOMMU_READ, &page), -EINVAL); + KUNIT_ASSERT_EQ(test, vfio_dma_bitmap_alloc_all(f->iommu, + vfio_dirty_pgsize(f->iommu)), 0); + f->iommu->dirty_page_tracking = true; + memset(data, 0x97, sizeof(data)); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw_chunk(f->iommu, + VFIO_TEST_IOVA + SZ_4K - 17, data, sizeof(data), true, &copied), 0); + KUNIT_EXPECT_EQ(test, copied, sizeof(data)); + KUNIT_EXPECT_EQ(test, bitmap_weight(dma->bitmap, length / SZ_4K), 2U); + KUNIT_EXPECT_TRUE(test, test_bit(0, dma->bitmap)); + KUNIT_EXPECT_TRUE(test, test_bit(1, dma->bitmap)); + memset(data, 0, sizeof(data)); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, + VFIO_TEST_IOVA + SZ_4K - 17, data, sizeof(data), false), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(data, 0x97, sizeof(data)), NULL); + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, + VFIO_TEST_IOVA + length, data, sizeof(data), false), -EINVAL); + kunit_release_action(test, vfio_fragment_fixture_free, f); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + kunit_info(test, "%uK userspace fine IOVA/physical offsets and dirty bytes checked", + 1U << (shifts[i] - 10)); + } +} + +static void vfio_fragments_replay_failure_test(struct kunit *test) +{ + struct vfio_account_mm *ctx = vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f; + struct vfio_dma *dma; + struct file *file; + struct page *page; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x58); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, 0, 2 * PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA, 1, IOMMU_READ, &page), 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + f->domains[0].fail_after = SZ_4K; + KUNIT_EXPECT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[0].type1), -ENOMEM); + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, 0UL); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + KUNIT_EXPECT_FALSE(test, dma->iommu_mapped); + f->domains[0].fail_after = 0; + KUNIT_ASSERT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[0].type1), 0); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + /* Detaching the last hardware domain keeps only the external page. */ + vfio_iommu_unmap_unpin_reaccount(f->iommu); + list_del_init(&f->domains[0].type1.next); + vfio_update_pgsize_bitmap(f->iommu); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 1UL); + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, 0UL); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 0UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + kunit_release_action(test, vfio_fragment_fixture_free, f); +} + +static void vfio_fragments_anon_owner_test(struct kunit *test) +{ + struct vfio_account_mm *old = vfio_account_mm_new(test, 12); + struct vfio_account_mm *next = vfio_account_mm_new(test, PAGE_SHIFT); + struct vfio_fragment_fixture *f; + struct vfio_dma *dma; + unsigned char *data = kunit_kmalloc(test, PAGE_SIZE, GFP_KERNEL); + unsigned long addr, copied, charge; + unsigned char check[64]; + + KUNIT_ASSERT_NOT_NULL(test, old); + KUNIT_ASSERT_NOT_NULL(test, next); + KUNIT_ASSERT_NOT_NULL(test, data); + memset(data, 0x49, PAGE_SIZE); + kthread_use_mm(old->mm); + addr = vm_mmap(NULL, VFIO_TEST_VA, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, 0); + copied = addr == VFIO_TEST_VA ? + copy_to_user((void __user *)addr, data, PAGE_SIZE) : PAGE_SIZE; + kthread_unuse_mm(old->mm); + KUNIT_ASSERT_EQ(test, addr, VFIO_TEST_VA); + KUNIT_ASSERT_EQ(test, copied, 0UL); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, old, 0, PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_NOT_NULL(test, vfio_fragment_find(dma, 0)->pin.subpage); + charge = dma->locked_vm; + KUNIT_ASSERT_GT(test, charge, 0UL); + old->live = false; + mmput(old->mm); + KUNIT_ASSERT_EQ(test, vfio_change_dma_mm(dma, dma->task, next->mm, true), 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(old->mm), 0UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(next->mm), charge); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, VFIO_TEST_IOVA, + check, sizeof(check), false), 0); + KUNIT_EXPECT_PTR_EQ(test, memchr_inv(check, 0x49, sizeof(check)), NULL); + KUNIT_ASSERT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[1].type1), 0); + list_add_tail(&f->domains[1].type1.next, &f->iommu->domain_list); + kunit_release_action(test, vfio_fragment_fixture_free, f); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(next->mm), 0UL); + kunit_info(test, "anonymous 4K slot pins survive source exit and native-owner transfer"); +} + +static void vfio_fragments_domain_granule_test(struct kunit *test) +{ + struct vfio_account_mm *ctx = vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f; + struct vfio_dma *dma; + struct file *file; + struct page *page; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + file = vfio_fragment_file(test, PAGE_SIZE, 0x73); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, PAGE_SIZE), VFIO_TEST_VA); + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, ctx, 0, PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_EQ(test, vfio_dma_bitmap_alloc_all(f->iommu, + vfio_dirty_pgsize(f->iommu)), 0); + f->iommu->dirty_page_tracking = true; + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + 31, 1, IOMMU_READ, &page), 1); + KUNIT_EXPECT_EQ(test, bitmap_weight(dma->bitmap, PAGE_SIZE / SZ_4K), + (unsigned int)(PAGE_SIZE / SZ_4K)); + f->domains[0].core.pgsize_bitmap = SZ_16K; + KUNIT_ASSERT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[0].type1), 0); + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + KUNIT_EXPECT_EQ(test, 1UL << __ffs(f->iommu->pgsize_bitmap), SZ_16K); + KUNIT_EXPECT_EQ(test, vfio_dirty_pgsize(f->iommu), SZ_4K); + vfio_iommu_unmap_unpin_reaccount(f->iommu); + list_del_init(&f->domains[0].type1.next); + vfio_update_pgsize_bitmap(f->iommu); + KUNIT_EXPECT_EQ(test, 1UL << __ffs(f->iommu->pgsize_bitmap), SZ_4K); + KUNIT_EXPECT_EQ(test, vfio_dirty_pgsize(f->iommu), SZ_4K); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 31, 1); + kunit_release_action(test, vfio_fragment_fixture_free, f); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); +} + +static void vfio_fragments_native_owner_test(struct kunit *test) +{ + for (unsigned int variant = 0; variant < 9; variant++) { + bool same_mm = variant == 8; + bool mapped = same_mm || (variant & 1); + bool alias = same_mm || (variant & 2), different = variant & 4; + struct vfio_account_mm *old = vfio_account_mm_new(test, PAGE_SHIFT); + struct vfio_account_mm *next = same_mm ? old : vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f; + struct vfio_fragment_provider *prepared; + struct file *file, *replacement; + struct vfio_dma *dma; + struct page *pages[2]; + phys_addr_t external[2]; + unsigned long unique, addr; + unsigned char data[32]; + int ret; + + KUNIT_ASSERT_NOT_NULL(test, old); + KUNIT_ASSERT_NOT_NULL(test, next); + file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x63); + replacement = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x44); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, replacement); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(old, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + if (alias) { + kthread_use_mm(old->mm); + addr = vm_mmap(file, VFIO_TEST_VA + PAGE_SIZE, PAGE_SIZE, + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, 0); + kthread_unuse_mm(old->mm); + KUNIT_ASSERT_EQ(test, addr, VFIO_TEST_VA + PAGE_SIZE); + } + f = vfio_fragment_fixture_new(test); + KUNIT_ASSERT_NOT_NULL(test, f); + if (mapped) { + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + } + KUNIT_ASSERT_EQ(test, vfio_fragment_test_map(f, old, 0, 2 * PAGE_SIZE), 0); + dma = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + KUNIT_ASSERT_NOT_NULL(test, dma); + KUNIT_ASSERT_NULL(test, dma->fragments); + if (different) { + kthread_use_mm(old->mm); + addr = vm_mmap(replacement, VFIO_TEST_VA, 2 * PAGE_SIZE, + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, 0); + kthread_unuse_mm(old->mm); + KUNIT_ASSERT_EQ(test, addr, VFIO_TEST_VA); + } + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA + 17, 2, + IOMMU_READ | IOMMU_WRITE, pages), 2); + for (unsigned int i = 0; i < 2; i++) + external[i] = page_to_phys(pages[i]); + if (mapped && different) + KUNIT_EXPECT_NE(test, external[0], + iommu_iova_to_phys(&f->domains[0].core, VFIO_TEST_IOVA)); + KUNIT_ASSERT_EQ(test, dma->locked_vm, 2UL); + if (!same_mm) { + old->live = false; + mmput(old->mm); + KUNIT_ASSERT_EQ(test, atomic_read(&old->mm->mm_users), 0); + } + + prepared = vfio_fragments_prepare_native(f->iommu, dma); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, prepared); + unique = mapped ? (alias ? 1UL : 2UL) : 0; + if (!mapped || different) + unique += different || !alias ? 2UL : 1UL; + KUNIT_EXPECT_EQ(test, vfio_fragments_native_pages(prepared), unique); + /* Failed new-owner credit must not publish metadata or release pins. */ + if (!same_mm && task_rlimit(dma->task, RLIMIT_MEMLOCK) != RLIM_INFINITY) { + unsigned long budget = task_rlimit(dma->task, RLIMIT_MEMLOCK) >> + mm_page_shift(next->mm); + + next->mm->locked_vm = budget; + KUNIT_EXPECT_EQ(test, vfio_change_dma_mm_pages(dma, dma->task, + next->mm, false, unique), -ENOMEM); + KUNIT_EXPECT_PTR_EQ(test, dma->mm, old->mm); + KUNIT_EXPECT_EQ(test, dma->locked_vm, 2UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(old->mm), 2UL); + KUNIT_EXPECT_EQ(test, next->mm->locked_vm, budget); + next->mm->locked_vm = 0; + } + vfio_fragments_abort_native(prepared); + KUNIT_EXPECT_PTR_EQ(test, dma->fragments, NULL); + KUNIT_EXPECT_TRUE(test, folio_maybe_dma_pinned(page_folio(pages[0]))); + kthread_use_mm(next->mm); + ret = vfio_change_dma_owner(f->iommu, dma, + VFIO_TEST_VA + (same_mm ? SZ_4K : 0)); + kthread_unuse_mm(next->mm); + KUNIT_ASSERT_EQ(test, ret, 0); + KUNIT_ASSERT_NOT_NULL(test, dma->fragments); + KUNIT_EXPECT_EQ(test, dma->locked_vm, unique); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(old->mm), same_mm ? unique : 0UL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(next->mm), unique); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_dma_rw(f->iommu, VFIO_TEST_IOVA, + data, sizeof(data), false), 0); + KUNIT_EXPECT_PTR_EQ(test, + memchr_inv(data, !mapped && different ? 0x44 : 0x63, sizeof(data)), NULL); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA + 31, 2, + IOMMU_READ | IOMMU_WRITE, pages), 2); + for (unsigned int i = 0; i < 2; i++) + KUNIT_EXPECT_EQ(test, page_to_phys(pages[i]), external[i]); + KUNIT_EXPECT_EQ(test, dma->locked_vm, unique); + KUNIT_ASSERT_EQ(test, vfio_iommu_replay(f->iommu, &f->domains[1].type1), 0); + list_add_tail(&f->domains[1].type1.next, &f->iommu->domain_list); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 31, 2); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 17, 2); + KUNIT_EXPECT_EQ(test, dma->locked_vm, + mapped ? (alias ? 1UL : 2UL) : unique); + kunit_release_action(test, vfio_fragment_fixture_free, f); + KUNIT_EXPECT_EQ(test, mm_locked_vm_native_pages(next->mm), 0UL); + kunit_info(test, "native ownership variant=%u preserves PIN identity and unique charge", + variant); + } +} + +static int vfio_fragment_map_area(struct vfio_fragment_fixture *f, + struct vfio_account_mm *ctx, unsigned long iova_offset, + unsigned long va_offset, unsigned long length, unsigned int flags) +{ + struct vfio_iommu_type1_dma_map map = { + .argsz = sizeof(map), + .flags = flags, + .vaddr = VFIO_TEST_VA + va_offset, + .iova = VFIO_TEST_IOVA + iova_offset, + .size = length, + }; + int ret; + + kthread_use_mm(ctx->mm); + ret = vfio_dma_do_map(f->iommu, &map); + kthread_unuse_mm(ctx->mm); + return ret; +} + +static void vfio_fragments_external_span_test(struct kunit *test) +{ + unsigned int rw = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; + + for (unsigned int mapped = 0; mapped < 2; mapped++) { + struct vfio_account_mm *a = vfio_account_mm_new(test, 12); + struct vfio_account_mm *b = vfio_account_mm_new(test, PAGE_SHIFT); + struct vfio_fragment_fixture *f = vfio_fragment_fixture_new(test); + struct file *file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x6b); + struct vfio_iommu_type1_dma_unmap unmap = { + .argsz = sizeof(unmap), .iova = VFIO_TEST_IOVA, .size = SZ_4K, + }; + struct vfio_bitmap bitmap = {}; + struct vfio_dma *first, *rest; + struct page *pages[3]; + unsigned long bits = 2 * PAGE_SIZE / SZ_4K - 1; + + KUNIT_ASSERT_NOT_NULL(test, a); + KUNIT_ASSERT_NOT_NULL(test, b); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(a, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(b, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + if (mapped) { + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + } + KUNIT_ASSERT_EQ(test, vfio_fragment_map_area(f, a, 0, 0, SZ_4K, rw), 0); + KUNIT_ASSERT_EQ(test, vfio_fragment_map_area(f, b, SZ_4K, SZ_4K, + 2 * PAGE_SIZE - SZ_4K, rw), 0); + first = vfio_find_dma(f->iommu, VFIO_TEST_IOVA, 1); + rest = vfio_find_dma(f->iommu, VFIO_TEST_IOVA + SZ_4K, 1); + KUNIT_ASSERT_NOT_NULL(test, first); + KUNIT_ASSERT_NOT_NULL(test, rest); + f->iommu->dirty_page_tracking = true; + KUNIT_ASSERT_EQ(test, vfio_dma_bitmap_alloc_all(f->iommu, SZ_4K), 0); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA + 37, 2, + IOMMU_READ | IOMMU_WRITE, pages), 2); + KUNIT_EXPECT_EQ(test, first->locked_vm, 1UL); + KUNIT_EXPECT_EQ(test, rest->locked_vm, 2UL); + KUNIT_EXPECT_EQ(test, bitmap_weight(first->bitmap, 1), 1); + KUNIT_EXPECT_EQ(test, bitmap_weight(rest->bitmap, bits), (int)bits); + bitmap_clear(rest->bitmap, 0, bits); + vfio_dma_populate_bitmap(rest, SZ_4K); + KUNIT_EXPECT_EQ(test, bitmap_weight(rest->bitmap, bits), (int)bits); + if (mapped) + KUNIT_EXPECT_EQ(test, page_to_phys(pages[0]), + iommu_iova_to_phys(&f->domains[0].core, VFIO_TEST_IOVA)); + /* A failed larger request must leave prior caller references intact. */ + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA, 3, IOMMU_READ, pages), -EINVAL); + KUNIT_EXPECT_PTR_EQ(test, pages[0], NULL); + KUNIT_EXPECT_PTR_EQ(test, pages[1], NULL); + KUNIT_EXPECT_PTR_EQ(test, pages[2], NULL); + a->live = b->live = false; + mmput(a->mm); + mmput(b->mm); + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA, 2, IOMMU_READ, pages), 2); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 2); + /* Release the second page first: it shares the second DMA owner. */ + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + PAGE_SIZE + 7, 1); + KUNIT_EXPECT_EQ(test, rest->locked_vm, mapped ? 2UL : 1UL); + f->external_refs = 1; + KUNIT_ASSERT_EQ(test, vfio_dma_do_unmap(f->iommu, &unmap, &bitmap), 0); + KUNIT_EXPECT_EQ(test, f->notifications, 1U); + KUNIT_EXPECT_EQ(test, f->external_refs, 0U); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(a->mm), 0UL); + KUNIT_EXPECT_EQ(test, rest->locked_vm, mapped ? 2UL : 0UL); + unmap.iova += SZ_4K; + unmap.size = 2 * PAGE_SIZE - SZ_4K; + KUNIT_ASSERT_EQ(test, vfio_dma_do_unmap(f->iommu, &unmap, &bitmap), 0); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(b->mm), 0UL); + kunit_release_action(test, vfio_fragment_fixture_free, f); + } + for (unsigned int failure = 0; failure < 4; failure++) { + struct vfio_account_mm *ctx = vfio_account_mm_new(test, 12); + struct vfio_fragment_fixture *f = vfio_fragment_fixture_new(test); + struct file *file = vfio_fragment_file(test, 2 * PAGE_SIZE, 0x3a); + struct page *page = NULL; + struct rb_node *node; + unsigned long va = failure == 2 ? PAGE_SIZE + SZ_4K : SZ_4K; + int expected = failure == 1 ? -EPERM : -EINVAL; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, 2 * PAGE_SIZE), VFIO_TEST_VA); + KUNIT_ASSERT_EQ(test, vfio_fragment_map_area(f, ctx, 0, 0, SZ_4K, rw), 0); + if (failure == 3) + va += SZ_4K; + if (failure != 0) + KUNIT_ASSERT_EQ(test, vfio_fragment_map_area(f, ctx, SZ_4K, va, + SZ_4K, failure == 1 ? VFIO_DMA_MAP_FLAG_READ : rw), 0); + KUNIT_EXPECT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA, 1, + IOMMU_READ | IOMMU_WRITE, &page), expected); + KUNIT_EXPECT_PTR_EQ(test, page, NULL); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + for (node = rb_first(&f->iommu->dma_list); node; node = rb_next(node)) { + struct vfio_dma *dma = rb_entry(node, struct vfio_dma, node); + + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&dma->pfn_list)); + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&dma->fragments->records)); + } + kunit_release_action(test, vfio_fragment_fixture_free, f); + } +} + +#define VFIO_STRESS_CYCLES 1024 + +struct vfio_pin_stress { + struct vfio_fragment_fixture *fixture; + struct vfio_device consumer; + struct vfio_device registrar; + struct vfio_device observer; + struct task_struct *competing_thread; + struct task_struct *pin_thread; + struct task_struct *register_thread; + struct mutex consumer_lock; + struct completion pin_go; + struct completion pin_ready; + struct completion notify_entered; + struct completion pin_done; + struct completion register_go; + struct completion register_entered; + struct completion register_done; + struct completion competing_go; + struct completion competing_done; + unsigned long competing_size; + int competing_ret; + bool competing; + unsigned int competing_unmaps; + unsigned int observations; + atomic_t error; + unsigned int pins; + unsigned int registrations; + bool stopping; +}; + +static void vfio_stress_error(struct vfio_pin_stress *s, int ret) +{ + atomic_cmpxchg(&s->error, 0, ret < 0 ? ret : -EIO); +} + +static void vfio_stress_notify(struct vfio_device *device, u64 iova, u64 length) +{ + struct vfio_pin_stress *s = container_of(device, typeof(*s), consumer); + struct vfio_fragment_fixture *f = s->fixture; + + /* The callback holds the production device-list lock, not the DMA lock. */ + complete(&s->register_go); + if (!wait_for_completion_timeout(&s->register_entered, msecs_to_jiffies(5000))) + vfio_stress_error(s, -ETIMEDOUT); + complete(&s->notify_entered); + mutex_lock(&s->consumer_lock); + f->notifications++; + while (f->external_refs) { + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 1); + f->external_refs--; + } + mutex_unlock(&s->consumer_lock); + if (s->competing) { + /* A second unmap may free the DMA while this callback still runs. */ + complete(&s->competing_go); + if (!wait_for_completion_timeout(&s->competing_done, msecs_to_jiffies(5000))) + vfio_stress_error(s, -ETIMEDOUT); + } +} + +static void vfio_stress_observe(struct vfio_device *device, u64 iova, u64 length) +{ + struct vfio_pin_stress *s = container_of(device, typeof(*s), observer); + + s->observations++; + if (iova != VFIO_TEST_IOVA || length != SZ_4K) + vfio_stress_error(s, -ESTALE); +} + +static void vfio_stress_empty_notify(struct vfio_device *device, u64 iova, u64 length) +{ +} + +static const struct vfio_device_ops vfio_stress_consumer_ops = { + .dma_unmap = vfio_stress_notify, +}; + +static const struct vfio_device_ops vfio_stress_registrar_ops = { + .dma_unmap = vfio_stress_empty_notify, +}; + +static const struct vfio_device_ops vfio_stress_observer_ops = { + .dma_unmap = vfio_stress_observe, +}; + +static int vfio_stress_competing_unmaps(void *data) +{ + struct vfio_pin_stress *s = data; + + for (;;) { + struct vfio_iommu_type1_dma_unmap unmap = { + .argsz = sizeof(unmap), .iova = VFIO_TEST_IOVA, + .size = SZ_4K, + }; + struct vfio_bitmap bitmap = {}; + + wait_for_completion(&s->competing_go); + if (READ_ONCE(s->stopping) || kthread_should_stop()) + break; + s->competing_ret = vfio_dma_do_unmap(s->fixture->iommu, &unmap, &bitmap); + s->competing_size = unmap.size; + s->competing_unmaps++; + complete(&s->competing_done); + } + return 0; +} + +static int vfio_stress_pinner(void *data) +{ + struct vfio_pin_stress *s = data; + struct vfio_fragment_fixture *f = s->fixture; + + for (;;) { + struct page *page = NULL, *original; + void *bytes; + int ret; + + wait_for_completion(&s->pin_go); + if (READ_ONCE(s->stopping) || kthread_should_stop()) + break; + mutex_lock(&s->consumer_lock); + ret = vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + 37, 1, IOMMU_READ | IOMMU_WRITE, &page); + if (ret != 1) { + vfio_stress_error(s, ret); + mutex_unlock(&s->consumer_lock); + complete(&s->pin_ready); + complete(&s->pin_done); + continue; + } + f->external_refs++; + s->pins++; + original = page; + complete(&s->pin_ready); + /* Unmap cannot revoke a reference before the consumer publishes it. */ + if (!wait_for_completion_timeout(&s->notify_entered, msecs_to_jiffies(5000))) + vfio_stress_error(s, -ETIMEDOUT); + if (!READ_ONCE(s->stopping)) { + ret = vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + VFIO_TEST_IOVA + 17, 1, IOMMU_READ, &page); + if (ret == 1) { + f->external_refs++; + s->pins++; + if (page != original) + vfio_stress_error(s, -ESTALE); + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 17, 1); + f->external_refs--; + } else { + vfio_stress_error(s, ret); + } + bytes = kmap_local_page(original); + if (memchr_inv(bytes, 0x6b, PAGE_SIZE)) + vfio_stress_error(s, -EIO); + kunmap_local(bytes); + } + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA + 37, 1); + f->external_refs--; + mutex_unlock(&s->consumer_lock); + complete(&s->pin_done); + } + return 0; +} + +static int vfio_stress_registrations(void *data) +{ + struct vfio_pin_stress *s = data; + + for (;;) { + wait_for_completion(&s->register_go); + if (READ_ONCE(s->stopping) || kthread_should_stop()) + break; + complete(&s->register_entered); + vfio_iommu_type1_register_device(s->fixture->iommu, &s->registrar); + vfio_iommu_type1_unregister_device(s->fixture->iommu, &s->registrar); + s->registrations++; + complete(&s->register_done); + } + return 0; +} + +static void vfio_stress_free(void *data) +{ + struct vfio_pin_stress *s = data; + struct vfio_fragment_fixture *f = s->fixture; + + WRITE_ONCE(s->stopping, true); + complete_all(&s->pin_go); + complete_all(&s->notify_entered); + complete_all(&s->register_go); + complete_all(&s->competing_go); + if (s->competing_thread) { + kthread_stop(s->competing_thread); + put_task_struct(s->competing_thread); + } + if (s->pin_thread) { + kthread_stop(s->pin_thread); + put_task_struct(s->pin_thread); + } + if (s->register_thread) { + kthread_stop(s->register_thread); + put_task_struct(s->register_thread); + } + mutex_lock(&s->consumer_lock); + while (f->external_refs) { + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 1); + f->external_refs--; + } + mutex_unlock(&s->consumer_lock); + vfio_iommu_type1_unregister_device(f->iommu, &s->consumer); + vfio_iommu_type1_unregister_device(f->iommu, &s->observer); +} + +static struct vfio_pin_stress *vfio_stress_new(struct kunit *test, + struct vfio_fragment_fixture *f) +{ + struct vfio_pin_stress *s = kunit_kzalloc(test, sizeof(*s), GFP_KERNEL); + + if (!s) + return NULL; + s->fixture = f; + mutex_init(&s->consumer_lock); + init_completion(&s->pin_go); + init_completion(&s->pin_ready); + init_completion(&s->notify_entered); + init_completion(&s->pin_done); + init_completion(&s->register_go); + init_completion(&s->register_entered); + init_completion(&s->register_done); + init_completion(&s->competing_go); + init_completion(&s->competing_done); + atomic_set(&s->error, 0); + s->consumer.ops = &vfio_stress_consumer_ops; + s->registrar.ops = &vfio_stress_registrar_ops; + s->observer.ops = &vfio_stress_observer_ops; + vfio_iommu_type1_unregister_device(f->iommu, &f->device); + /* list_add puts the consumer before the observer in callback order. */ + vfio_iommu_type1_register_device(f->iommu, &s->observer); + vfio_iommu_type1_register_device(f->iommu, &s->consumer); + if (kunit_add_action_or_reset(test, vfio_stress_free, s)) + return NULL; + s->pin_thread = kthread_run(vfio_stress_pinner, s, "vfio-pin-stress"); + if (IS_ERR(s->pin_thread)) { + s->pin_thread = NULL; + return NULL; + } + get_task_struct(s->pin_thread); + s->register_thread = kthread_run(vfio_stress_registrations, s, "vfio-reg-stress"); + if (IS_ERR(s->register_thread)) { + s->register_thread = NULL; + return NULL; + } + get_task_struct(s->register_thread); + s->competing_thread = kthread_run(vfio_stress_competing_unmaps, s, "vfio-unmap-stress"); + if (IS_ERR(s->competing_thread)) { + s->competing_thread = NULL; + return NULL; + } + get_task_struct(s->competing_thread); + return s; +} + +static void vfio_fragments_concurrency_test(struct kunit *test) +{ + unsigned int shifts[] = { 12, 14, 16 }; + unsigned int rw = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; + + for (unsigned int shift = 0; shift < ARRAY_SIZE(shifts); shift++) { + if (shifts[shift] > PAGE_SHIFT) + continue; + for (unsigned int mapped = 0; mapped < 2; mapped++) { + struct vfio_account_mm *ctx = vfio_account_mm_new(test, shifts[shift]); + struct vfio_fragment_fixture *f = vfio_fragment_fixture_new(test); + struct file *file = vfio_fragment_file(test, PAGE_SIZE, 0x6b); + struct vfio_pin_stress *s; + + KUNIT_ASSERT_NOT_NULL(test, ctx); + KUNIT_ASSERT_NOT_NULL(test, f); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, file); + KUNIT_ASSERT_EQ(test, vfio_fragment_source(ctx, file, PAGE_SIZE), + VFIO_TEST_VA); + s = vfio_stress_new(test, f); + KUNIT_ASSERT_NOT_NULL(test, s); + if (mapped) { + list_add_tail(&f->domains[0].type1.next, &f->iommu->domain_list); + vfio_update_pgsize_bitmap(f->iommu); + } + for (unsigned int cycle = 0; cycle < VFIO_STRESS_CYCLES; cycle++) { + struct vfio_iommu_type1_dma_unmap unmap = { + .argsz = sizeof(unmap), .iova = VFIO_TEST_IOVA, + .size = SZ_4K, + }; + struct vfio_bitmap bitmap = {}; + bool fine = cycle & 1; + unsigned long remaining = PAGE_SIZE - SZ_4K; + + KUNIT_ASSERT_EQ(test, + vfio_fragment_map_area(f, ctx, 0, 0, SZ_4K, rw), 0); + for (unsigned long offset = SZ_4K; offset < PAGE_SIZE;) { + unsigned long length = fine ? SZ_4K : remaining; + + KUNIT_ASSERT_EQ(test, vfio_fragment_map_area(f, ctx, offset, + offset, length, rw), 0); + offset += length; + } + if (cycle == VFIO_STRESS_CYCLES - 1) { + struct page *page; + + KUNIT_ASSERT_EQ(test, vfio_iommu_type1_pin_pages(f->iommu, + f->group.iommu_group, VFIO_TEST_IOVA, 1, + IOMMU_READ, &page), 1); + f->external_refs = 1; + ctx->live = false; + mmput(ctx->mm); + KUNIT_ASSERT_EQ(test, atomic_read(&ctx->mm->mm_users), 0); + } + s->competing = cycle & 2; + complete(&s->pin_go); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&s->pin_ready, + msecs_to_jiffies(5000)), 0UL); + KUNIT_ASSERT_EQ(test, atomic_read(&s->error), 0); + KUNIT_ASSERT_EQ(test, + vfio_dma_do_unmap(f->iommu, &unmap, &bitmap), 0); + KUNIT_EXPECT_EQ(test, unmap.size, s->competing ? 0UL : SZ_4K); + if (s->competing) { + KUNIT_EXPECT_EQ(test, s->competing_ret, 0); + KUNIT_EXPECT_EQ(test, s->competing_size, SZ_4K); + } + unmap.iova += SZ_4K; + unmap.size = remaining; + KUNIT_ASSERT_EQ(test, + vfio_dma_do_unmap(f->iommu, &unmap, &bitmap), 0); + KUNIT_EXPECT_EQ(test, unmap.size, remaining); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&s->pin_done, + msecs_to_jiffies(5000)), 0UL); + KUNIT_ASSERT_NE(test, wait_for_completion_timeout(&s->register_done, + msecs_to_jiffies(5000)), 0UL); + KUNIT_ASSERT_EQ(test, atomic_read(&s->error), 0); + KUNIT_EXPECT_EQ(test, f->external_refs, 0U); + KUNIT_EXPECT_EQ(test, mm_locked_vm_bytes(ctx->mm), 0UL); + KUNIT_EXPECT_TRUE(test, RB_EMPTY_ROOT(&f->iommu->dma_list)); + KUNIT_EXPECT_EQ(test, f->domains[0].mapped, 0UL); + reinit_completion(&s->pin_ready); + reinit_completion(&s->notify_entered); + reinit_completion(&s->pin_done); + reinit_completion(&s->register_entered); + reinit_completion(&s->register_done); + reinit_completion(&s->competing_done); + } + KUNIT_EXPECT_EQ(test, s->pins, 2U * VFIO_STRESS_CYCLES); + KUNIT_EXPECT_EQ(test, s->registrations, (unsigned int)VFIO_STRESS_CYCLES); + KUNIT_EXPECT_EQ(test, f->notifications, (unsigned int)VFIO_STRESS_CYCLES); + KUNIT_EXPECT_EQ(test, s->observations, (unsigned int)VFIO_STRESS_CYCLES); + KUNIT_EXPECT_EQ(test, s->competing_unmaps, VFIO_STRESS_CYCLES / 2U); + kunit_info(test, "%uK user/%luK native mapped=%u: %u forced pin/unmap/register cycles, half with competing unmap", + 1U << (shifts[shift] - 10), PAGE_SIZE >> 10, + mapped, VFIO_STRESS_CYCLES); + kunit_release_action(test, vfio_stress_free, s); + kunit_release_action(test, vfio_fragment_fixture_free, f); + } + } +} + +static struct kunit_case vfio_fragments_cases[] = { + KUNIT_CASE(vfio_fragments_lifetime_test), + KUNIT_CASE(vfio_fragments_lazy_test), + KUNIT_CASE(vfio_fragments_map_failure_test), + KUNIT_CASE(vfio_fragments_fine_test), + KUNIT_CASE(vfio_fragments_replay_failure_test), + KUNIT_CASE(vfio_fragments_anon_owner_test), + KUNIT_CASE(vfio_fragments_domain_granule_test), + KUNIT_CASE(vfio_fragments_native_owner_test), + KUNIT_CASE(vfio_fragments_external_span_test), + KUNIT_CASE(vfio_fragments_concurrency_test), + {} +}; + +static struct kunit_suite vfio_fragments_suite = { + .name = "vfio-type1-fragments", + .test_cases = vfio_fragments_cases, +}; + +kunit_test_suite(vfio_fragments_suite); diff --git a/drivers/vfio/vfio_iommu_type1_ioctl_test.c b/drivers/vfio/vfio_iommu_type1_ioctl_test.c new file mode 100644 index 00000000000000..33e981bcdf74d1 --- /dev/null +++ b/drivers/vfio/vfio_iommu_type1_ioctl_test.c @@ -0,0 +1,305 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Included after the KUnit harness; real EL0 arguments, recorded domains only. */ +#include +#include +#include +#include "vfio_iommu_type1_test.h" + +struct vfio_ioctl_fixture { + struct vfio_fragment_fixture pins; + struct mutex request_lock; +}; + +struct vfio_ioctl_rw { + struct work_struct work; + struct vfio_iommu *iommu; + dma_addr_t iova; + void *data; + size_t size; + bool write; + int ret; +}; + +static void vfio_ioctl_rw_work(struct work_struct *work) +{ + struct vfio_ioctl_rw *rw = container_of(work, typeof(*rw), work); + + rw->ret = vfio_iommu_type1_dma_rw(rw->iommu, rw->iova, rw->data, + rw->size, rw->write); +} + +static int vfio_ioctl_test_open(struct inode *inode, struct file *file) +{ + struct vfio_ioctl_fixture *ctx; + struct vfio_fragment_fixture *f; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + ctx = kzalloc_obj(*ctx); + if (!ctx) + return -ENOMEM; + f = &ctx->pins; + f->iommu = vfio_iommu_type1_open(VFIO_TYPE1v2_IOMMU); + if (IS_ERR(f->iommu)) { + int ret = PTR_ERR(f->iommu); + + kfree(ctx); + return ret; + } + mutex_init(&ctx->request_lock); + for (unsigned int i = 0; i < ARRAY_SIZE(f->domains); i++) { + struct vfio_test_domain *mock = &f->domains[i]; + + xa_init(&mock->translations); + mock->core.ops = &vfio_test_domain_ops; + mock->core.type = IOMMU_DOMAIN_UNMANAGED; + mock->core.pgsize_bitmap = SZ_4K | PAGE_SIZE; + mock->type1.domain = &mock->core; + INIT_LIST_HEAD(&mock->type1.next); + INIT_LIST_HEAD(&mock->type1.group_list); + } + f->group.iommu_group = (struct iommu_group *)ctx; + list_add(&f->group.next, &f->iommu->emulated_iommu_groups); + f->iommu->num_non_pinned_groups = 1; + f->device.ops = &vfio_test_device_ops; + vfio_iommu_type1_register_device(f->iommu, &f->device); + file->private_data = ctx; + return 0; +} + +static int vfio_ioctl_test_release(struct inode *inode, struct file *file) +{ + struct vfio_ioctl_fixture *ctx = file->private_data; + struct vfio_fragment_fixture *f = &ctx->pins; + + while (f->external_refs) { + vfio_iommu_type1_unpin_pages(f->iommu, VFIO_TEST_IOVA, 1); + f->external_refs--; + } + vfio_iommu_type1_unregister_device(f->iommu, &f->device); + vfio_iommu_unmap_unpin_all(f->iommu); + for (unsigned int i = 0; i < ARRAY_SIZE(f->domains); i++) { + list_del_init(&f->domains[i].type1.next); + xa_destroy(&f->domains[i].translations); + } + list_del(&f->group.next); + vfio_iommu_type1_release(f->iommu); + kfree(ctx); + return 0; +} + +static int vfio_ioctl_test_domain(struct vfio_fragment_fixture *f, + struct vfio_type1_test_cmd *cmd, bool add) +{ + struct vfio_test_domain *mock; + struct rb_node *node; + int ret = 0; + + if (cmd->domain >= ARRAY_SIZE(f->domains) || cmd->flags) + return -EINVAL; + mock = &f->domains[cmd->domain]; + mutex_lock(&f->iommu->lock); + if (add) { + if (!list_empty(&mock->type1.next)) { + ret = -EEXIST; + goto out; + } + if (cmd->size != SZ_4K && cmd->size != SZ_16K && cmd->size != SZ_64K) { + ret = -EINVAL; + goto out; + } + mock->core.pgsize_bitmap = cmd->size; + mock->fail_after = cmd->fail_after; + ret = vfio_iommu_replay(f->iommu, &mock->type1); + if (ret) + goto out; + list_add_tail(&mock->type1.next, &f->iommu->domain_list); + } else { + if (list_empty(&mock->type1.next)) { + ret = -ENOENT; + goto out; + } + if (list_is_singular(&f->iommu->domain_list)) { + vfio_iommu_unmap_unpin_reaccount(f->iommu); + } else { + for (node = rb_first(&f->iommu->dma_list); node; node = rb_next(node)) { + struct vfio_dma *dma = rb_entry(node, struct vfio_dma, node); + + iommu_unmap(&mock->core, dma->iova, dma->size); + } + } + list_del_init(&mock->type1.next); + } + vfio_update_pgsize_bitmap(f->iommu); +out: + mutex_unlock(&f->iommu->lock); + return ret; +} + +static int vfio_ioctl_test_rw(struct vfio_fragment_fixture *f, + struct vfio_type1_test_cmd *cmd) +{ + struct vfio_ioctl_rw rw = { + .iommu = f->iommu, .iova = cmd->iova, .size = cmd->size, + .write = cmd->flags & VFIO_TYPE1_TEST_WRITE, + }; + void __user *buffer = u64_to_user_ptr(cmd->buffer); + int ret; + + if (!cmd->size || cmd->size > 256 || + cmd->flags & ~(VFIO_TYPE1_TEST_WRITE | VFIO_TYPE1_TEST_WORKER)) + return -EINVAL; + rw.data = kzalloc(cmd->size, GFP_KERNEL); + if (!rw.data) + return -ENOMEM; + if (rw.write && copy_from_user(rw.data, buffer, cmd->size)) { + ret = -EFAULT; + goto out; + } + if (cmd->flags & VFIO_TYPE1_TEST_WORKER) { + INIT_WORK(&rw.work, vfio_ioctl_rw_work); + queue_work(system_unbound_wq, &rw.work); + flush_work(&rw.work); + ret = rw.ret; + } else { + ret = vfio_iommu_type1_dma_rw(f->iommu, rw.iova, rw.data, rw.size, rw.write); + } + if (!ret && !rw.write && copy_to_user(buffer, rw.data, cmd->size)) + ret = -EFAULT; +out: + kfree(rw.data); + return ret; +} + +static long vfio_ioctl_test_ioctl(struct file *file, unsigned int request, + unsigned long arg) +{ + struct vfio_ioctl_fixture *ctx = file->private_data; + struct vfio_fragment_fixture *f = &ctx->pins; + struct vfio_type1_test_cmd cmd; + void __user *user = (void __user *)arg; + struct page *page; + int ret = 0; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + /* Request serialization belongs to this fixture, not production type1. */ + mutex_lock(&ctx->request_lock); + switch (request) { + case VFIO_CHECK_EXTENSION: + case VFIO_IOMMU_GET_INFO: + case VFIO_IOMMU_MAP_DMA: + case VFIO_IOMMU_UNMAP_DMA: + case VFIO_IOMMU_DIRTY_PAGES: + ret = vfio_iommu_type1_ioctl(f->iommu, request, arg); + goto out; + default: + break; + } + if (copy_from_user(&cmd, user, sizeof(cmd))) { + ret = -EFAULT; + goto out; + } + if (cmd.reserved) { + ret = -EINVAL; + goto out; + } + switch (request) { + case VFIO_TYPE1_TEST_DOMAIN_ADD: + case VFIO_TYPE1_TEST_DOMAIN_DEL: + ret = vfio_ioctl_test_domain(f, &cmd, request == VFIO_TYPE1_TEST_DOMAIN_ADD); + break; + case VFIO_TYPE1_TEST_RW: + ret = vfio_ioctl_test_rw(f, &cmd); + break; + case VFIO_TYPE1_TEST_PIN: + case VFIO_TYPE1_TEST_UNPIN: + if (ALIGN_DOWN(cmd.iova, PAGE_SIZE) != VFIO_TEST_IOVA || cmd.flags) { + ret = -EINVAL; + break; + } + if (request == VFIO_TYPE1_TEST_PIN) { + ret = vfio_iommu_type1_pin_pages(f->iommu, f->group.iommu_group, + cmd.iova, 1, IOMMU_READ | IOMMU_WRITE, &page); + if (ret == 1) { + f->external_refs++; + ret = 0; + } + } else if (f->external_refs) { + vfio_iommu_type1_unpin_pages(f->iommu, cmd.iova, 1); + f->external_refs--; + } else { + ret = -ENOENT; + } + break; + case VFIO_TYPE1_TEST_EMULATED: + if (cmd.flags > 1 || f->external_refs) { + ret = -EBUSY; + break; + } + mutex_lock(&f->iommu->lock); + if (cmd.flags && list_empty(&f->group.next)) { + list_add(&f->group.next, &f->iommu->emulated_iommu_groups); + if (!f->group.pinned_page_dirty_scope) + f->iommu->num_non_pinned_groups++; + } else if (!cmd.flags && !list_empty(&f->group.next)) { + list_del_init(&f->group.next); + if (!f->group.pinned_page_dirty_scope) + f->iommu->num_non_pinned_groups--; + } + mutex_unlock(&f->iommu->lock); + break; + case VFIO_TYPE1_TEST_STATE: + cmd.locked_bytes = 0; + cmd.owner_locked_bytes = mm_locked_vm_bytes(current->mm); + cmd.flags = 0; + cmd.owner_users = 0; + mutex_lock(&f->iommu->lock); + for (struct rb_node *node = rb_first(&f->iommu->dma_list); + node; node = rb_next(node)) { + struct vfio_dma *dma = rb_entry(node, struct vfio_dma, node); + + cmd.locked_bytes += dma->locked_vm * PAGE_SIZE; + cmd.owner_locked_bytes = mm_locked_vm_bytes(dma->mm); + cmd.owner_users = atomic_read(&dma->mm->mm_users); + if (dma->fragments) + cmd.flags |= VFIO_TYPE1_TEST_STATE_FRAGMENTS; + } + cmd.native_page_size = PAGE_SIZE; + cmd.minimum_page_size = 1UL << __ffs(f->iommu->pgsize_bitmap); + for (unsigned int i = 0; i < ARRAY_SIZE(f->domains); i++) + cmd.mapped_bytes[i] = f->domains[i].mapped; + mutex_unlock(&f->iommu->lock); + cmd.notifications = f->notifications; + cmd.external_refs = f->external_refs; + break; + default: + ret = -ENOTTY; + } + if (!ret && copy_to_user(user, &cmd, sizeof(cmd))) + ret = -EFAULT; +out: + mutex_unlock(&ctx->request_lock); + return ret; +} + +static const struct file_operations vfio_ioctl_test_fops = { + .owner = THIS_MODULE, + .open = vfio_ioctl_test_open, + .release = vfio_ioctl_test_release, + .unlocked_ioctl = vfio_ioctl_test_ioctl, + .compat_ioctl = compat_ptr_ioctl, +}; + +static struct miscdevice vfio_ioctl_test_device = { + .minor = MISC_DYNAMIC_MINOR, + .name = "aurora-vfio-type1-test", + .fops = &vfio_ioctl_test_fops, + .mode = 0600, +}; + +static int __init vfio_ioctl_test_init(void) +{ + return misc_register(&vfio_ioctl_test_device); +} +late_initcall(vfio_ioctl_test_init); diff --git a/drivers/vfio/vfio_iommu_type1_test.h b/drivers/vfio/vfio_iommu_type1_test.h new file mode 100644 index 00000000000000..36c09030bcf0ef --- /dev/null +++ b/drivers/vfio/vfio_iommu_type1_test.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* Private disposable-guest test protocol, not a supported VFIO ABI. */ +#ifndef VFIO_IOMMU_TYPE1_TEST_H +#define VFIO_IOMMU_TYPE1_TEST_H +#include +#include + +struct vfio_type1_test_cmd { + __u64 iova; + __u64 size; + __u64 buffer; + __u64 fail_after; + __u64 locked_bytes; + __u64 owner_locked_bytes; + __u64 mapped_bytes[2]; + __u64 native_page_size; + __u64 minimum_page_size; + __u32 domain; + __u32 flags; + __u32 owner_users; + __u32 notifications; + __u32 external_refs; + __u32 reserved; +}; + +#define VFIO_TYPE1_TEST_DOMAIN_ADD _IOWR('V', 1, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_DOMAIN_DEL _IOWR('V', 2, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_RW _IOWR('V', 3, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_PIN _IOWR('V', 4, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_UNPIN _IOWR('V', 5, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_STATE _IOWR('V', 6, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_EMULATED _IOWR('V', 7, struct vfio_type1_test_cmd) +#define VFIO_TYPE1_TEST_STATE_FRAGMENTS 1U +#define VFIO_TYPE1_TEST_WRITE 1U +#define VFIO_TYPE1_TEST_WORKER 2U +#endif diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index e4c44f2ea413ac..4c3577ce7b85b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -2436,6 +2437,26 @@ static int set_bit_to_user(int nr, void __user *addr) int bit = nr + (log % PAGE_SIZE) * 8; int r; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) { + struct user_page_fragment fragment; + unsigned long word = log & ~(sizeof(unsigned long) - 1); + + /* Pin the complete atomic word, which stays inside one leaf. + * A relocated leaf's physical offset need not match its VA. + */ + r = pin_user_fragments_remote(current->mm, word, + sizeof(unsigned long), FOLL_WRITE, &fragment, 1); + if (r != 1) + return r < 0 ? r : -EFAULT; + base = kmap_local_folio(fragment.folio, fragment.offset); + set_bit(nr + (log - word) * 8, base); + kunmap_local(base); + release_user_fragments(&fragment, 1, true); + return 0; + } +#endif + r = pin_user_pages_fast(log, 1, FOLL_WRITE, &page); if (r < 0) return r; diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c index 1dcc4675580ed1..c35d81d26e981f 100644 --- a/drivers/xen/gntdev.c +++ b/drivers/xen/gntdev.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include @@ -64,7 +65,11 @@ struct gntdev_copy_batch { s16 __user *status[GNTDEV_COPY_BATCH]; unsigned int nr_ops; unsigned int nr_pages; - bool writeable; + unsigned long writeable; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment *fragments; + bool use_fragments; +#endif struct gntdev_copy_batch *next; }; @@ -631,6 +636,9 @@ static int gntdev_release(struct inode *inode, struct file *flip) while (priv->batch) { batch = priv->batch; priv->batch = batch->next; +#ifdef CONFIG_MM_SUBPAGE + kfree(batch->fragments); +#endif kfree(batch); } mutex_unlock(&priv->batch_lock); @@ -805,20 +813,45 @@ static long gntdev_ioctl_notify(struct gntdev_priv *priv, void __user *u) } static int gntdev_get_page(struct gntdev_copy_batch *batch, void __user *virt, - unsigned long *gfn) + size_t len, bool writeable, unsigned long *gfn) { unsigned long addr = (unsigned long)virt; struct page *page; unsigned long xen_pfn; int ret; - ret = pin_user_pages_fast(addr, 1, batch->writeable ? FOLL_WRITE : 0, &page); - if (ret < 0) - return ret; - - batch->pages[batch->nr_pages++] = page; +#ifdef CONFIG_MM_SUBPAGE + if (batch->use_fragments) { + struct user_page_fragment *fragment; - xen_pfn = page_to_xen_pfn(page) + XEN_PFN_DOWN(addr & ~PAGE_MASK); + if (!batch->fragments) { + batch->fragments = kcalloc(GNTDEV_COPY_BATCH, + sizeof(*fragment), GFP_KERNEL); + if (!batch->fragments) + return -ENOMEM; + } + fragment = &batch->fragments[batch->nr_pages]; + ret = pin_user_fragments_remote(current->mm, addr, len, + writeable ? FOLL_WRITE : 0, fragment, 1); + if (ret != 1) + return ret < 0 ? ret : -EFAULT; + if (fragment->length != len) { + release_user_fragments(fragment, 1, false); + return -EFAULT; + } + xen_pfn = XEN_PFN_DOWN(PFN_PHYS(folio_pfn(fragment->folio)) + fragment->offset); + } else +#endif + { + ret = pin_user_pages_fast(addr, 1, writeable ? FOLL_WRITE : 0, &page); + if (ret != 1) + return ret < 0 ? ret : -EFAULT; + batch->pages[batch->nr_pages] = page; + xen_pfn = page_to_xen_pfn(page) + XEN_PFN_DOWN(addr & ~PAGE_MASK); + } + if (writeable) + __set_bit(batch->nr_pages, &batch->writeable); + batch->nr_pages++; *gfn = pfn_to_gfn(xen_pfn); return 0; @@ -826,9 +859,24 @@ static int gntdev_get_page(struct gntdev_copy_batch *batch, void __user *virt, static void gntdev_put_pages(struct gntdev_copy_batch *batch) { - unpin_user_pages_dirty_lock(batch->pages, batch->nr_pages, batch->writeable); + unsigned int first = 0; + + while (first < batch->nr_pages) { + bool dirty = test_bit(first, &batch->writeable); + unsigned int next = dirty ? + find_next_zero_bit(&batch->writeable, batch->nr_pages, first) : + find_next_bit(&batch->writeable, batch->nr_pages, first); + +#ifdef CONFIG_MM_SUBPAGE + if (batch->use_fragments) + release_user_fragments(batch->fragments + first, next - first, dirty); + else +#endif + unpin_user_pages_dirty_lock(batch->pages + first, next - first, dirty); + first = next; + } batch->nr_pages = 0; - batch->writeable = false; + batch->writeable = 0; } static int gntdev_copy(struct gntdev_copy_batch *batch) @@ -917,9 +965,8 @@ static int gntdev_grant_copy_seg(struct gntdev_copy_batch *batch, virt = seg->source.virt + copied; off = (unsigned long)virt & ~XEN_PAGE_MASK; len = min(len, (size_t)XEN_PAGE_SIZE - off); - batch->writeable = false; - ret = gntdev_get_page(batch, virt, &gfn); + ret = gntdev_get_page(batch, virt, len, false, &gfn); if (ret < 0) return ret; @@ -937,9 +984,8 @@ static int gntdev_grant_copy_seg(struct gntdev_copy_batch *batch, virt = seg->dest.virt + copied; off = (unsigned long)virt & ~XEN_PAGE_MASK; len = min(len, (size_t)XEN_PAGE_SIZE - off); - batch->writeable = true; - ret = gntdev_get_page(batch, virt, &gfn); + ret = gntdev_get_page(batch, virt, len, true, &gfn); if (ret < 0) return ret; @@ -971,6 +1017,10 @@ static long gntdev_ioctl_grant_copy(struct gntdev_priv *priv, void __user *u) mutex_lock(&priv->batch_lock); if (!priv->batch) { batch = kmalloc_obj(*batch); +#ifdef CONFIG_MM_SUBPAGE + if (batch) + batch->fragments = NULL; +#endif } else { batch = priv->batch; priv->batch = batch->next; @@ -981,6 +1031,10 @@ static long gntdev_ioctl_grant_copy(struct gntdev_priv *priv, void __user *u) batch->nr_ops = 0; batch->nr_pages = 0; + batch->writeable = 0; +#ifdef CONFIG_MM_SUBPAGE + batch->use_fragments = mm_page_size(current->mm) < PAGE_SIZE; +#endif for (i = 0; i < copy.count; i++) { struct gntdev_grant_copy_segment seg; diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 16a56b6b3f6ca1..826235fd977c25 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -79,11 +79,11 @@ static int elf_core_dump(struct coredump_params *cprm); #define elf_core_dump NULL #endif -#if ELF_EXEC_PAGESIZE > PAGE_SIZE -#define ELF_MIN_ALIGN ELF_EXEC_PAGESIZE -#else -#define ELF_MIN_ALIGN PAGE_SIZE +#ifndef ELF_EXEC_PAGESIZE_MM +#define ELF_EXEC_PAGESIZE_MM(mm) ELF_EXEC_PAGESIZE #endif +#define ELF_MIN_ALIGN max_t(unsigned long, mm_page_size(current->mm), \ + ELF_EXEC_PAGESIZE_MM(current->mm)) #ifndef ELF_CORE_EFLAGS #define ELF_CORE_EFLAGS 0 @@ -93,12 +93,20 @@ static int elf_core_dump(struct coredump_params *cprm); #define ELF_PAGEOFFSET(_v) ((_v) & (ELF_MIN_ALIGN-1)) #define ELF_PAGEALIGN(_v) (((_v) + ELF_MIN_ALIGN - 1) & ~(ELF_MIN_ALIGN - 1)) +#ifdef CONFIG_COREDUMP +static unsigned long elf_min_coredump_size(void) +{ + return ELF_EXEC_PAGESIZE_MM(current->mm); +} +#endif + static struct linux_binfmt elf_format = { .module = THIS_MODULE, .load_binary = load_elf_binary, #ifdef CONFIG_COREDUMP .core_dump = elf_core_dump, .min_coredump = ELF_EXEC_PAGESIZE, + .min_coredump_size = elf_min_coredump_size, #endif }; @@ -248,7 +256,7 @@ create_elf_tables(struct linux_binprm *bprm, const struct elfhdr *exec, ARCH_DLINFO; #endif NEW_AUX_ENT(AT_HWCAP, ELF_HWCAP); - NEW_AUX_ENT(AT_PAGESZ, ELF_EXEC_PAGESIZE); + NEW_AUX_ENT(AT_PAGESZ, ELF_EXEC_PAGESIZE_MM(mm)); NEW_AUX_ENT(AT_CLKTCK, CLOCKS_PER_SEC); NEW_AUX_ENT(AT_PHDR, phdr_addr); NEW_AUX_ENT(AT_PHENT, sizeof(struct elf_phdr)); @@ -622,6 +630,13 @@ static inline int arch_check_elf(struct elfhdr *ehdr, bool has_interp, #endif /* !CONFIG_ARCH_BINFMT_ELF_STATE */ +#ifndef arch_bprm_prepare_mm +static inline int arch_bprm_prepare_mm(struct linux_binprm *bprm, bool compat) +{ + return 0; +} +#endif + static inline int make_prot(u32 p_flags, struct arch_elf_state *arch_state, bool has_interp, bool is_interp) { @@ -1006,6 +1021,11 @@ static int load_elf_binary(struct linux_binprm *bprm) if (retval) goto out_free_dentry; + retval = arch_bprm_prepare_mm(bprm, + elf_ex->e_ident[EI_CLASS] == ELFCLASS32); + if (retval) + goto out_free_dentry; + /* Flush all traces of the currently running executable */ retval = begin_new_exec(bprm); if (retval) @@ -1335,7 +1355,7 @@ static int load_elf_binary(struct linux_binprm *bprm) * leave a gap between .bss and brk. */ if (!brk_moved) - mm->brk = mm->start_brk = mm->brk + PAGE_SIZE; + mm->brk = mm->start_brk = mm->brk + mm_page_size(mm); mm->brk = mm->start_brk = arch_randomize_brk(mm); brk_moved = true; @@ -1351,10 +1371,10 @@ static int load_elf_binary(struct linux_binprm *bprm) and some applications "depend" upon this behavior. Since we do not have the power to recompile these, we emulate the SVr4 behavior. Sigh. */ - error = vm_mmap(NULL, 0, PAGE_SIZE, PROT_READ | PROT_EXEC, + error = vm_mmap(NULL, 0, mm_page_size(mm), PROT_READ | PROT_EXEC, MAP_FIXED | MAP_PRIVATE, 0); - retval = do_mseal(0, PAGE_SIZE, 0); + retval = do_mseal(0, mm_page_size(mm), 0); if (retval) pr_warn_ratelimited("pid=%d, couldn't seal address 0, ret=%d.\n", task_pid_nr(current), retval); @@ -1658,7 +1678,7 @@ static int fill_files_note(struct memelfnote *note, struct coredump_params *cprm /* Now we know exact count of files, can store it */ data[0] = count; - data[1] = PAGE_SIZE; + data[1] = mm_page_size(current->mm); /* * Count usually is less than mm->map_count, * we need to move filenames down. @@ -2043,7 +2063,7 @@ static int elf_core_dump(struct coredump_params *cprm) offset += sz; } - dataoff = offset = roundup(offset, ELF_EXEC_PAGESIZE); + dataoff = offset = roundup(offset, ELF_EXEC_PAGESIZE_MM(current->mm)); offset += cprm->vma_data_size; offset += elf_core_extra_data_size(cprm); @@ -2083,7 +2103,7 @@ static int elf_core_dump(struct coredump_params *cprm) phdr.p_flags |= PF_W; if (meta->flags & VM_EXEC) phdr.p_flags |= PF_X; - phdr.p_align = ELF_EXEC_PAGESIZE; + phdr.p_align = ELF_EXEC_PAGESIZE_MM(current->mm); if (!dump_emit(cprm, &phdr, sizeof(phdr))) goto end_coredump; diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index 2ceb5661e0718a..96ae95c3029bc3 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -540,11 +540,10 @@ static noinline int add_ra_bio_pages(struct inode *inode, * After the compressed pages are read, we copy the bytes into the * bio we were passed and then call the bio end_io calls */ -void btrfs_submit_compressed_read(struct btrfs_bio *bbio) +void btrfs_submit_compressed_read(struct btrfs_bio *bbio, struct extent_map *em) { struct btrfs_inode *inode = bbio->inode; struct btrfs_fs_info *fs_info = inode->root->fs_info; - struct extent_map_tree *em_tree = &inode->extent_tree; struct compressed_bio *cb; unsigned int compressed_len; const u32 min_folio_size = btrfs_min_folio_size(fs_info); @@ -552,7 +551,6 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) gfp_t gfp; u64 em_len; u64 em_start; - struct extent_map *em; unsigned long pflags; int memstall = 0; int ret; @@ -568,16 +566,18 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) else gfp = GFP_NOFS; - /* we need the actual starting offset of this extent in the file */ - read_lock(&em_tree->lock); - em = btrfs_lookup_extent_mapping(em_tree, file_offset, fs_info->sectorsize); - read_unlock(&em_tree->lock); - if (!em) { + /* + * We need the actual starting offset of this extent in the file. The + * caller holds a reference on the extent map the bio was built from; + * it may no longer be in the inode's extent map tree. + */ + if (WARN_ON_ONCE(!em)) { ret = -EIO; goto out; } ASSERT(btrfs_extent_map_is_compressed(em)); + ASSERT(em->start <= file_offset && file_offset < btrfs_extent_map_end(em)); compressed_len = em->disk_num_bytes; cb = alloc_compressed_bio(inode, file_offset, REQ_OP_READ, @@ -592,8 +592,6 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) cb->orig_bbio = bbio; cb->bbio.csum_search_commit_root = bbio->csum_search_commit_root; - btrfs_free_extent_map(em); - for (int i = 0; i * min_folio_size < compressed_len; i++) { struct folio *folio; u32 cur_len = min(compressed_len - i * min_folio_size, min_folio_size); diff --git a/fs/btrfs/compression.h b/fs/btrfs/compression.h index 1022dc53ec51ea..6ccbcfaf3280e9 100644 --- a/fs/btrfs/compression.h +++ b/fs/btrfs/compression.h @@ -20,6 +20,7 @@ struct address_space; struct inode; struct btrfs_inode; struct btrfs_ordered_extent; +struct extent_map; /* * We want to make sure that amount of RAM required to uncompress an extent is @@ -94,7 +95,7 @@ struct compressed_bio *btrfs_alloc_compressed_write(struct btrfs_inode *inode, u64 start, u64 len); void btrfs_submit_compressed_write(struct btrfs_ordered_extent *ordered, struct compressed_bio *cb); -void btrfs_submit_compressed_read(struct btrfs_bio *bbio); +void btrfs_submit_compressed_read(struct btrfs_bio *bbio, struct extent_map *em); int btrfs_compress_str2level(unsigned int type, const char *str, int *level_ret); diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 741fd1d4ab2ead..eddc6a834f214b 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -150,6 +150,16 @@ struct btrfs_bio_ctrl { * inside the same inode. */ u64 last_em_start; + + /* + * For a compressed read, the extent map of the compressed extent that + * @bbio reads, referenced until the bio is submitted. + * + * btrfs_submit_compressed_read() needs it, but must not look it up in + * the inode's extent map tree again: by then the extent map shrinker + * may have removed it from the tree. + */ + struct extent_map *compressed_em; }; /* @@ -176,8 +186,10 @@ static void submit_one_bio(struct btrfs_bio_ctrl *bio_ctrl) { struct btrfs_bio *bbio = bio_ctrl->bbio; - if (!bbio) + if (!bbio) { + ASSERT(!bio_ctrl->compressed_em); return; + } /* Caller should ensure the bio has at least some range added */ ASSERT(bbio->bio.bi_iter.bi_size); @@ -186,12 +198,14 @@ static void submit_one_bio(struct btrfs_bio_ctrl *bio_ctrl) if (btrfs_op(&bbio->bio) == BTRFS_MAP_READ && bio_ctrl->compress_type != BTRFS_COMPRESS_NONE) - btrfs_submit_compressed_read(bbio); + btrfs_submit_compressed_read(bbio, bio_ctrl->compressed_em); else btrfs_submit_bbio(bbio, 0); /* The bbio is owned by the end_io handler now */ bio_ctrl->bbio = NULL; + btrfs_free_extent_map(bio_ctrl->compressed_em); + bio_ctrl->compressed_em = NULL; /* * We used the generation to decide whether to lookup csums in the * commit_root or not when we called bio_set_csum_search_commit_root() @@ -963,6 +977,26 @@ static struct extent_map *get_extent_map(struct btrfs_inode *inode, return em; } +/* + * A block of the compressed extent mapped by @em has just been added to + * @bio_ctrl->bbio, which only ever reads blocks of one compressed extent. + * Take over the caller's reference on @em: bio_ctrl keeps one reference on + * the extent map of its bio until submit_one_bio() passes it on. + */ +static void bio_ctrl_hold_compressed_em(struct btrfs_bio_ctrl *bio_ctrl, + struct extent_map *em) +{ + struct extent_map *old = bio_ctrl->compressed_em; + + /* + * Any reference held so far is for an earlier block of the same bio, + * on the same extent map or on a copy of it loaded again after the + * shrinker dropped it: one is enough. + */ + bio_ctrl->compressed_em = em; + btrfs_free_extent_map(old); +} + static void btrfs_readahead_expand(struct readahead_control *ractl, const struct extent_map *em) { @@ -1121,17 +1155,17 @@ static int btrfs_do_readpage(struct folio *folio, struct extent_map **em_cached, bio_ctrl->last_em_start = em->start; em_gen = em->generation; - btrfs_free_extent_map(em); - em = NULL; /* we've found a hole, just zero and go on */ if (block_start == EXTENT_MAP_HOLE) { + btrfs_free_extent_map(em); folio_zero_range(folio, pg_offset, blocksize); end_folio_read(vi, folio, true, cur, blocksize); continue; } /* the get_extent function already copied into the folio */ if (block_start == EXTENT_MAP_INLINE) { + btrfs_free_extent_map(em); end_folio_read(vi, folio, true, cur, blocksize); continue; } @@ -1145,6 +1179,11 @@ static int btrfs_do_readpage(struct folio *folio, struct extent_map **em_cached, submit_one_bio(bio_ctrl); submit_extent_folio(bio_ctrl, disk_bytenr, folio, blocksize, pg_offset, em_gen); + + if (compress_type != BTRFS_COMPRESS_NONE) + bio_ctrl_hold_compressed_em(bio_ctrl, em); + else + btrfs_free_extent_map(em); } return 0; } diff --git a/fs/coredump.c b/fs/coredump.c index bb6fdb1f458e9b..6ab5f2e076698d 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -884,7 +885,8 @@ static bool coredump_file(struct core_name *cn, struct coredump_params *cprm, struct file *file __free(fput) = NULL; int open_flags = O_CREAT | O_WRONLY | O_NOFOLLOW | O_LARGEFILE | O_EXCL; - if (cprm->limit < binfmt->min_coredump) + if (cprm->limit < (binfmt->min_coredump_size ? + binfmt->min_coredump_size() : binfmt->min_coredump)) return false; if (coredump_force_suid_safe(cprm) && cn->corename[0] != '/') { @@ -1278,7 +1280,8 @@ void dump_skip(struct coredump_params *cprm, size_t nr) EXPORT_SYMBOL(dump_skip); #ifdef CONFIG_ELF_CORE -static int dump_emit_page(struct coredump_params *cprm, struct page *page) +static int dump_emit_page(struct coredump_params *cprm, struct page *page, + unsigned int offset, unsigned int size) { struct bio_vec bvec; struct iov_iter iter; @@ -1294,19 +1297,19 @@ static int dump_emit_page(struct coredump_params *cprm, struct page *page) return 0; cprm->to_skip = 0; } - if (cprm->written + PAGE_SIZE > cprm->limit) + if (cprm->written + size > cprm->limit) return 0; if (dump_interrupted()) return 0; pos = file->f_pos; - bvec_set_page(&bvec, page, PAGE_SIZE, 0); - iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, PAGE_SIZE); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, size); n = __kernel_write_iter(cprm->file, &iter, &pos); - if (n != PAGE_SIZE) + if (n != size) return 0; file->f_pos = pos; - cprm->written += PAGE_SIZE; - cprm->pos += PAGE_SIZE; + cprm->written += size; + cprm->pos += size; return 1; } @@ -1321,10 +1324,11 @@ static int dump_emit_page(struct coredump_params *cprm, struct page *page) #define dump_page_alloc() alloc_page(GFP_KERNEL) #define dump_page_free(x) __free_page(x) -static struct page *dump_page_copy(struct page *src, struct page *dst) +static struct page *dump_page_copy(struct page *src, struct page *dst, + unsigned int offset, unsigned int size) { void *buf = kmap_local_page(src); - size_t left = copy_mc_to_kernel(page_address(dst), buf, PAGE_SIZE); + size_t left = copy_mc_to_kernel(page_address(dst) + offset, buf + offset, size); kunmap_local(buf); return left ? NULL : dst; } @@ -1334,12 +1338,51 @@ static struct page *dump_page_copy(struct page *src, struct page *dst) /* We just want to return non-NULL; it's never used. */ #define dump_page_alloc() ERR_PTR(-EINVAL) #define dump_page_free(x) ((void)(x)) -static inline struct page *dump_page_copy(struct page *src, struct page *dst) +static inline struct page *dump_page_copy(struct page *src, struct page *dst, + unsigned int offset, unsigned int size) { return src; } #endif +#ifdef CONFIG_MM_SUBPAGE +static int dump_subpage_range(struct coredump_params *cprm, unsigned long start, + unsigned long len) +{ + unsigned long size = mm_user_fragment_size(current->mm), addr; + struct page *scratch = dump_page_alloc(); + int ret = 0; + + if (!scratch) + return 0; + for (addr = start; addr < start + len; addr += size) { + struct user_page_fragment fragment; + + if (get_dump_fragment(addr, &fragment) == 1) { + struct page *page = folio_page(fragment.folio, + fragment.offset >> PAGE_SHIFT); + unsigned int offset = offset_in_page(fragment.offset); + int emitted; + + emitted = dump_emit_page(cprm, + dump_page_copy(page, scratch, offset, size), offset, size); + release_user_fragments(&fragment, 1, false); + if (!emitted) + goto out; + } else { + dump_skip(cprm, size); + } + if (dump_interrupted()) + goto out; + cond_resched(); + } + ret = 1; +out: + dump_page_free(scratch); + return ret; +} +#endif + int dump_user_range(struct coredump_params *cprm, unsigned long start, unsigned long len) { @@ -1347,6 +1390,10 @@ int dump_user_range(struct coredump_params *cprm, unsigned long start, struct page *dump_page; int locked, ret; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return dump_subpage_range(cprm, start, len); +#endif dump_page = dump_page_alloc(); if (!dump_page) return 0; @@ -1375,7 +1422,8 @@ int dump_user_range(struct coredump_params *cprm, unsigned long start, mmap_read_unlock(current->mm); locked = 0; } - int stop = !dump_emit_page(cprm, dump_page_copy(page, dump_page)); + int stop = !dump_emit_page(cprm, + dump_page_copy(page, dump_page, 0, PAGE_SIZE), 0, PAGE_SIZE); put_page(page); if (stop) goto out; @@ -1652,9 +1700,10 @@ static unsigned long vma_dump_size(struct vm_area_struct *vma, * dump the first page to aid in determining what was mapped here. */ if (FILTER(ELF_HEADERS) && - vma->vm_pgoff == 0 && (vma->vm_flags & VM_READ)) { + vma->vm_pgoff == 0 && !vma_subpage_offset(vma) && + (vma->vm_flags & VM_READ)) { if ((READ_ONCE(file_inode(vma->vm_file)->i_mode) & 0111) != 0) - return PAGE_SIZE; + return min(mm_page_size(vma->vm_mm), vma->vm_end - vma->vm_start); /* * ELF libraries aren't always executable. @@ -1754,7 +1803,7 @@ static bool dump_vma_snapshot(struct coredump_params *cprm) m->end = vma->vm_end; m->flags = vma->vm_flags; m->dump_size = vma_dump_size(vma, cprm->mm_flags); - m->pgoff = vma->vm_pgoff; + m->pgoff = core_vma_pgoff(vma); m->file = vma->vm_file; if (m->file) get_file(m->file); @@ -1773,7 +1822,7 @@ static bool dump_vma_snapshot(struct coredump_params *cprm) memcmp(elfmag, ELFMAG, SELFMAG) != 0) { m->dump_size = 0; } else { - m->dump_size = PAGE_SIZE; + m->dump_size = min(mm_page_size(mm), m->end - m->start); } } diff --git a/fs/exec.c b/fs/exec.c index 2eacc1441c59fb..c9e6ecf187b7b6 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -122,6 +123,54 @@ bool path_noexec(const struct path *path) (path->mnt->mnt_sb->s_iflags & SB_I_NOEXEC); } +/* A temporary argument mapping owns one native page or an exact subpage slot. */ +struct arg_page { + struct page *page; +#ifdef CONFIG_MM_SUBPAGE + struct mm_subpage *subpage; + unsigned int offset; +#endif +}; + +static unsigned long bprm_page_size(const struct linux_binprm *bprm) +{ +#ifdef CONFIG_MMU + return mm_page_size(bprm->mm); +#else + return PAGE_SIZE; +#endif +} + +/* kmap_local_page() maps one native page, even for a larger user leaf. */ +static unsigned long bprm_copy_size(const struct linux_binprm *bprm) +{ + return min(bprm_page_size(bprm), PAGE_SIZE); +} + +static unsigned long bprm_copy_mask(const struct linux_binprm *bprm) +{ + return ~(bprm_copy_size(bprm) - 1); +} + +static unsigned long bprm_max_arg_strlen(const struct linux_binprm *bprm) +{ + return (MAX_ARG_STRLEN / PAGE_SIZE) * bprm_page_size(bprm); +} + +static unsigned int arg_page_offset(struct arg_page page) +{ +#ifdef CONFIG_MM_SUBPAGE + return page.offset; +#else + return 0; +#endif +} + +static void *map_arg_page(struct arg_page page) +{ + return kmap_local_page(page.page) + arg_page_offset(page); +} + #ifdef CONFIG_MMU /* * The nascent bprm->mm is not visible until exec_mmap() but it can @@ -141,10 +190,10 @@ static void acct_arg_size(struct linux_binprm *bprm, unsigned long pages) add_mm_counter(mm, MM_ANONPAGES, diff); } -static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos, +static struct arg_page get_arg_page(struct linux_binprm *bprm, unsigned long pos, int write) { - struct page *page; + struct arg_page page = {}; struct vm_area_struct *vma = bprm->vma; struct mm_struct *mm = bprm->mm; int ret; @@ -155,28 +204,47 @@ static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos, * ahead of time. */ if (!mmap_read_lock_maybe_expand(mm, vma, pos, write)) - return NULL; + return page; /* * We are doing an exec(). 'current' is the process * doing the exec and 'mm' is the new process's mm. */ - ret = get_user_pages_remote(mm, pos, 1, - write ? FOLL_WRITE : 0, - &page, NULL); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) != PAGE_SIZE) { + struct user_page_fragment fragment; + struct vm_area_struct *area; + + ret = get_user_fragment_vma_remote(mm, pos, 1, write ? FOLL_WRITE : 0, + &fragment, &area); + if (ret > 0) { + page.page = folio_page(fragment.folio, fragment.offset >> PAGE_SHIFT); + page.subpage = fragment.subpage; + page.offset = offset_in_page(fragment.offset) & mm_page_mask(mm); + } + } else +#endif + ret = get_user_pages_remote(mm, pos, 1, + write ? FOLL_WRITE : 0, &page.page, NULL); mmap_read_unlock(mm); if (ret <= 0) - return NULL; + return page; if (write) - acct_arg_size(bprm, vma_pages(vma)); + acct_arg_size(bprm, mm_pages_for_reader(mm, current->mm, vma_user_pages(vma))); return page; } -static void put_arg_page(struct page *page) +static void put_arg_page(struct arg_page page) { - put_page(page); +#ifdef CONFIG_MM_SUBPAGE + if (page.subpage) { + mm_subpage_put(page.subpage); + return; + } +#endif + put_page(page.page); } static void free_arg_pages(struct linux_binprm *bprm) @@ -184,14 +252,14 @@ static void free_arg_pages(struct linux_binprm *bprm) } static void flush_arg_page(struct linux_binprm *bprm, unsigned long pos, - struct page *page) + struct arg_page page) { - flush_cache_page(bprm->vma, pos, page_to_pfn(page)); + flush_cache_page(bprm->vma, pos, page_to_pfn(page.page)); } static bool valid_arg_len(struct linux_binprm *bprm, long len) { - return len <= MAX_ARG_STRLEN; + return len <= bprm_max_arg_strlen(bprm); } #else @@ -200,7 +268,7 @@ static inline void acct_arg_size(struct linux_binprm *bprm, unsigned long pages) { } -static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos, +static struct arg_page get_arg_page(struct linux_binprm *bprm, unsigned long pos, int write) { struct page *page; @@ -209,14 +277,14 @@ static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos, if (!page && write) { page = alloc_page(GFP_HIGHUSER|__GFP_ZERO); if (!page) - return NULL; + return (struct arg_page) {}; bprm->page[pos / PAGE_SIZE] = page; } - return page; + return (struct arg_page) { .page = page }; } -static void put_arg_page(struct page *page) +static void put_arg_page(struct arg_page page) { } @@ -237,7 +305,7 @@ static void free_arg_pages(struct linux_binprm *bprm) } static void flush_arg_page(struct linux_binprm *bprm, unsigned long pos, - struct page *page) + struct arg_page page) { } @@ -259,7 +327,7 @@ static int bprm_mm_init(struct linux_binprm *bprm) int err; struct mm_struct *mm = NULL; - bprm->mm = mm = mm_alloc(); + bprm->mm = mm = mm_alloc_exec(); err = -ENOMEM; if (!mm) goto err; @@ -288,6 +356,104 @@ static int bprm_mm_init(struct linux_binprm *bprm) return err; } +#ifdef CONFIG_MM_SUBPAGE +/* + * Final ELF selection may change the granule after argv, envp and script + * arguments have been staged. Table geometry is immutable: build a second mm + * and transfer bytes at the same temporary stack addresses before committing + * exec. Until success, the original bprm and its argument charge remain intact. + */ +int bprm_set_page_shift(struct linux_binprm *bprm, unsigned int page_shift) +{ + struct linux_binprm replacement = {}; + struct mm_struct *old_mm; + unsigned long pos, end, new_top, string_len = 0; + unsigned long old_size = mm_user_fragment_size(bprm->mm), new_size; + int ret; + + if (mm_page_shift(bprm->mm) == page_shift) + return 0; + if (WARN_ON_ONCE(bprm->point_of_no_return)) + return -EINVAL; + + replacement.mm = mm_alloc_exec_page_shift(page_shift); + if (!replacement.mm) + return -ENOMEM; + ret = -EINVAL; + if (mm_page_shift(replacement.mm) != page_shift) + goto out; + new_size = mm_user_fragment_size(replacement.mm); + ret = create_init_stack_vma(replacement.mm, &replacement.vma, &new_top); + if (ret) + goto out; + end = bprm->vma->vm_end - sizeof(void *); + ret = -EINVAL; + if (new_top != end || bprm->p > end) + goto out; + + for (pos = bprm->p; pos < end;) { + struct arg_page src, dst; + unsigned long src_off = pos & (old_size - 1); + unsigned long dst_off = pos & (new_size - 1); + unsigned long len = min3(end - pos, old_size - src_off, + new_size - dst_off); + const char *from; + char *to; + + ret = -ERESTARTNOHAND; + if (fatal_signal_pending(current)) + goto out; + src = get_arg_page(bprm, pos, 0); + ret = -EFAULT; + if (!src.page) + goto out; + dst = get_arg_page(&replacement, pos, 1); + if (!dst.page) { + put_arg_page(src); + ret = -ENOMEM; + goto out; + } + from = map_arg_page(src); + to = map_arg_page(dst); + /* Recheck per-string limits if the selected granule shrank. */ + ret = 0; + for (unsigned long i = 0; i < len; i++) { + if (++string_len > bprm_max_arg_strlen(&replacement)) { + ret = -E2BIG; + break; + } + if (!from[src_off + i]) + string_len = 0; + } + if (!ret) + memcpy(to + dst_off, from + src_off, len); + kunmap_local(to); + kunmap_local(from); + flush_arg_page(&replacement, pos, dst); + put_arg_page(dst); + put_arg_page(src); + if (ret) + goto out; + pos += len; + cond_resched(); + } + + old_mm = bprm->mm; + put_user_ns(replacement.mm->user_ns); + replacement.mm->user_ns = get_user_ns(old_mm->user_ns); + acct_arg_size(bprm, 0); + bprm->mm = replacement.mm; + bprm->vma = replacement.vma; + bprm->vma_pages = replacement.vma_pages; + mmput(old_mm); + return 0; +out: + acct_arg_size(&replacement, 0); + mmput(replacement.mm); + return ret; +} +#endif + struct user_arg_ptr { #ifdef CONFIG_COMPAT bool is_compat; @@ -449,7 +615,7 @@ static int bprm_stack_limits(struct linux_binprm *bprm) static int copy_strings(int argc, struct user_arg_ptr argv, struct linux_binprm *bprm) { - struct page *kmapped_page = NULL; + struct arg_page kmapped_page = {}; char *kaddr = NULL; unsigned long kpos = 0; int ret; @@ -464,7 +630,7 @@ static int copy_strings(int argc, struct user_arg_ptr argv, if (IS_ERR(str)) goto out; - len = strnlen_user(str, MAX_ARG_STRLEN); + len = strnlen_user(str, bprm_max_arg_strlen(bprm)); if (!len) goto out; @@ -488,9 +654,9 @@ static int copy_strings(int argc, struct user_arg_ptr argv, } cond_resched(); - offset = pos % PAGE_SIZE; + offset = pos & ~bprm_copy_mask(bprm); if (offset == 0) - offset = PAGE_SIZE; + offset = bprm_copy_size(bprm); bytes_to_copy = offset; if (bytes_to_copy > len) @@ -501,23 +667,23 @@ static int copy_strings(int argc, struct user_arg_ptr argv, str -= bytes_to_copy; len -= bytes_to_copy; - if (!kmapped_page || kpos != (pos & PAGE_MASK)) { - struct page *page; + if (!kmapped_page.page || kpos != (pos & bprm_copy_mask(bprm))) { + struct arg_page page; page = get_arg_page(bprm, pos, 1); - if (!page) { + if (!page.page) { ret = -E2BIG; goto out; } - if (kmapped_page) { - flush_dcache_page(kmapped_page); + if (kmapped_page.page) { + flush_dcache_page(kmapped_page.page); kunmap_local(kaddr); put_arg_page(kmapped_page); } kmapped_page = page; - kaddr = kmap_local_page(kmapped_page); - kpos = pos & PAGE_MASK; + kaddr = map_arg_page(kmapped_page); + kpos = pos & bprm_copy_mask(bprm); flush_arg_page(bprm, kpos, kmapped_page); } if (copy_from_user(kaddr+offset, str, bytes_to_copy)) { @@ -528,8 +694,8 @@ static int copy_strings(int argc, struct user_arg_ptr argv, } ret = 0; out: - if (kmapped_page) { - flush_dcache_page(kmapped_page); + if (kmapped_page.page) { + flush_dcache_page(kmapped_page.page); kunmap_local(kaddr); put_arg_page(kmapped_page); } @@ -541,7 +707,7 @@ static int copy_strings(int argc, struct user_arg_ptr argv, */ int copy_string_kernel(const char *arg, struct linux_binprm *bprm) { - int len = strnlen(arg, MAX_ARG_STRLEN) + 1 /* terminating NUL */; + int len = strnlen(arg, bprm_max_arg_strlen(bprm)) + 1 /* terminating NUL */; unsigned long pos = bprm->p; if (len == 0) @@ -557,18 +723,19 @@ int copy_string_kernel(const char *arg, struct linux_binprm *bprm) while (len > 0) { unsigned int bytes_to_copy = min(len, - min_not_zero(offset_in_page(pos), PAGE_SIZE)); - struct page *page; + min_not_zero(pos & ~bprm_copy_mask(bprm), bprm_copy_size(bprm))); + struct arg_page page; pos -= bytes_to_copy; arg -= bytes_to_copy; len -= bytes_to_copy; page = get_arg_page(bprm, pos, 1); - if (!page) + if (!page.page) return -E2BIG; - flush_arg_page(bprm, pos & PAGE_MASK, page); - memcpy_to_page(page, offset_in_page(pos), arg, bytes_to_copy); + flush_arg_page(bprm, pos & bprm_copy_mask(bprm), page); + memcpy_to_page(page.page, arg_page_offset(page) + + (pos & ~bprm_copy_mask(bprm)), arg, bytes_to_copy); put_arg_page(page); } @@ -621,20 +788,20 @@ int setup_arg_pages(struct linux_binprm *bprm, /* Add space for stack randomization. */ if (current->flags & PF_RANDOMIZE) - stack_base += (STACK_RND_MASK << PAGE_SHIFT); + stack_base += (STACK_RND_MASK << mm_page_shift(mm)); /* Make sure we didn't let the argument array grow too large. */ if (vma->vm_end - vma->vm_start > stack_base) return -ENOMEM; - stack_base = PAGE_ALIGN(stack_top - stack_base); + stack_base = mm_page_align(mm, stack_top - stack_base); stack_shift = vma->vm_start - stack_base; mm->arg_start = bprm->p - stack_shift; bprm->p = vma->vm_end - stack_shift; #else stack_top = arch_align_stack(stack_top); - stack_top = PAGE_ALIGN(stack_top); + stack_top = mm_page_align(mm, stack_top); if (unlikely(stack_top < mmap_min_addr) || unlikely(vma->vm_end - vma->vm_start >= stack_top - mmap_min_addr)) @@ -702,7 +869,7 @@ int setup_arg_pages(struct linux_binprm *bprm, * Align this down to a page boundary as expand_stack * will align it up. */ - rlim_stack = bprm->rlim_stack.rlim_cur & PAGE_MASK; + rlim_stack = bprm->rlim_stack.rlim_cur & mm_page_mask(mm); stack_expand = min(rlim_stack, stack_size + stack_expand); @@ -1323,6 +1490,10 @@ void setup_new_exec(struct linux_binprm * bprm) arch_pick_mmap_layout(me->mm, &bprm->rlim_stack); + /* Consume the one-shot request only after committing to the new image. */ +#ifdef CONFIG_ARCH_HAS_USER_PAGE_SIZE + me->exec_page_shift = 0; +#endif arch_setup_new_exec(); /* Set the new mm task size. We have to do that late because it may @@ -1616,25 +1787,25 @@ int remove_arg_zero(struct linux_binprm *bprm) { unsigned long offset; char *kaddr; - struct page *page; + struct arg_page page; if (!bprm->argc) return 0; do { - offset = bprm->p & ~PAGE_MASK; + offset = bprm->p & ~bprm_copy_mask(bprm); page = get_arg_page(bprm, bprm->p, 0); - if (!page) + if (!page.page) return -EFAULT; - kaddr = kmap_local_page(page); + kaddr = map_arg_page(page); - for (; offset < PAGE_SIZE && kaddr[offset]; + for (; offset < bprm_copy_size(bprm) && kaddr[offset]; offset++, bprm->p++) ; kunmap_local(kaddr); put_arg_page(page); - } while (offset == PAGE_SIZE); + } while (offset == bprm_copy_size(bprm)); bprm->p++; bprm->argc--; diff --git a/fs/ocfs2/file.c b/fs/ocfs2/file.c index 7df9921c1a3899..21153bd53f66ea 100644 --- a/fs/ocfs2/file.c +++ b/fs/ocfs2/file.c @@ -504,8 +504,7 @@ int ocfs2_truncate_file(struct inode *inode, goto bail_unlock_sem; } - unmap_mapping_range(inode->i_mapping, new_i_size + PAGE_SIZE - 1, 0, 1); - truncate_inode_pages(inode->i_mapping, new_i_size); + truncate_pagecache(inode, new_i_size); status = ocfs2_commit_truncate(osb, inode, di_bh); if (status < 0) { diff --git a/fs/proc/array.c b/fs/proc/array.c index 479ea8cb4ef43d..5b45506942991e 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -610,7 +610,8 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns, seq_put_decimal_ull(m, " ", 0); seq_put_decimal_ull(m, " ", start_time); seq_put_decimal_ull(m, " ", vsize); - seq_put_decimal_ull(m, " ", mm ? get_mm_rss(mm) : 0); + seq_put_decimal_ull(m, " ", mm ? + mm_pages_for_reader(mm, current->mm, get_mm_rss(mm)) : 0); seq_put_decimal_ull(m, " ", rsslim); seq_put_decimal_ull(m, " ", mm ? (permitted ? mm->start_code : 1) : 0); seq_put_decimal_ull(m, " ", mm ? (permitted ? mm->end_code : 1) : 0); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 436e1c1d11f41e..8938ae67581001 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include #include @@ -33,7 +34,7 @@ #define SENTINEL_VMA_GATE -2 #define SEQ_PUT_DEC(str, val) \ - seq_put_decimal_ull_width(m, str, (val) << (PAGE_SHIFT-10), 8) + seq_put_decimal_ull_width(m, str, mm_pages_to_kb(mm, val), 8) void task_mem(struct seq_file *m, struct mm_struct *mm) { unsigned long text, lib, swap, anon, file, shmem; @@ -58,15 +59,18 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) hiwater_rss = mm->hiwater_rss; /* split executable areas between text and lib */ - text = PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK); - text = min(text, mm->exec_vm << PAGE_SHIFT); - lib = (mm->exec_vm << PAGE_SHIFT) - text; + text = mm_page_align(mm, mm->end_code) - + (mm->start_code & mm_page_mask(mm)); + text = min(text, mm_pages_to_bytes(mm, mm->exec_vm)); + lib = (mm_pages_to_bytes(mm, mm->exec_vm)) - text; swap = get_mm_counter_sum(mm, MM_SWAPENTS); SEQ_PUT_DEC("VmPeak:\t", hiwater_vm); SEQ_PUT_DEC(" kB\nVmSize:\t", total_vm); - SEQ_PUT_DEC(" kB\nVmLck:\t", mm->locked_vm); - SEQ_PUT_DEC(" kB\nVmPin:\t", atomic64_read(&mm->pinned_vm)); + seq_put_decimal_ull_width(m, " kB\nVmLck:\t", + mm_locked_vm_bytes(mm) >> 10, 8); + seq_put_decimal_ull_width(m, " kB\nVmPin:\t", + atomic64_read(&mm->pinned_vm) << (PAGE_SHIFT - 10), 8); SEQ_PUT_DEC(" kB\nVmHWM:\t", hiwater_rss); SEQ_PUT_DEC(" kB\nVmRSS:\t", total_rss); SEQ_PUT_DEC(" kB\nRssAnon:\t", anon); @@ -88,20 +92,25 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) unsigned long task_vsize(struct mm_struct *mm) { - return PAGE_SIZE * mm->total_vm; + return mm_pages_to_bytes(mm, mm->total_vm); } unsigned long task_statm(struct mm_struct *mm, unsigned long *shared, unsigned long *text, unsigned long *data, unsigned long *resident) { - *shared = get_mm_counter_sum(mm, MM_FILEPAGES) + + struct mm_struct *reader = current->mm; + unsigned long file = get_mm_counter_sum(mm, MM_FILEPAGES) + get_mm_counter_sum(mm, MM_SHMEMPAGES); - *text = (PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK)) - >> PAGE_SHIFT; - *data = mm->data_vm + mm->stack_vm; - *resident = *shared + get_mm_counter_sum(mm, MM_ANONPAGES); - return mm->total_vm; + unsigned long code = (mm_page_align(mm, mm->end_code) - + (mm->start_code & mm_page_mask(mm))) >> mm_page_shift(mm); + + *shared = mm_pages_for_reader(mm, reader, file); + *text = mm_pages_for_reader(mm, reader, code); + *data = mm_pages_for_reader(mm, reader, mm->data_vm + mm->stack_vm); + *resident = mm_pages_for_reader(mm, reader, + file + get_mm_counter_sum(mm, MM_ANONPAGES)); + return mm_pages_for_reader(mm, reader, mm->total_vm); } #ifdef CONFIG_NUMA @@ -475,7 +484,8 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) dev = inode->i_sb->s_dev; ino = inode->i_ino; - pgoff = ((loff_t)vma->vm_pgoff) << PAGE_SHIFT; + pgoff = (((loff_t)vma->vm_pgoff) << PAGE_SHIFT) + + vma_subpage_offset(vma); } start = vma->vm_start; @@ -717,7 +727,8 @@ static int do_procmap_query(struct mm_struct *mm, void __user *uarg) if (vma->vm_file) { const struct inode *inode = file_user_inode(vma->vm_file); - karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT; + karg.vma_offset = (((__u64)vma->vm_pgoff) << PAGE_SHIFT) + + vma_subpage_offset(vma); karg.dev_major = MAJOR(inode->i_sb->s_dev); karg.dev_minor = MINOR(inode->i_sb->s_dev); karg.inode = inode->i_ino; @@ -857,6 +868,50 @@ const struct file_operations proc_pid_maps_operations = { */ #define PSS_SHIFT 12 +#if defined(CONFIG_MM_SUBPAGE) && (defined(CONFIG_PROC_PAGE_MONITOR) || defined(CONFIG_NUMA)) +/* Snapshot only: rmap and quarter counters can change in other address spaces. */ +static void file_quarter_mapcounts(struct page *page, struct mm_subpage_mapcounts *counts) +{ + int maps = folio_precise_page_mapcount(page_folio(page), page); + unsigned int small = mm_subpage_file_mapcounts(page, counts); + unsigned int native = maps > small ? maps - small : 0; + unsigned int i; + + for (i = 0; i < MM_SUBPAGES_PER_PAGE; i++) + counts->slots[i] += native; +} + +static unsigned int proc_leaf_mapcount(struct vm_area_struct *vma, pte_t pte, + struct page *page) +{ + struct folio *folio = page_folio(page); + struct mm_subpage_mapcounts counts; + unsigned int i, count = 0; + + if (!folio_test_anon(folio)) { + unsigned int first = mm_page_size(vma->vm_mm) < PAGE_SIZE ? + offset_in_page(pte_phys_mm(vma->vm_mm, pte)) / SZ_4K : 0; + unsigned int nr = mm_page_size(vma->vm_mm) / SZ_4K; + + file_quarter_mapcounts(page, &counts); + for (i = first; i < first + nr; i++) + count = max(count, counts.slots[i]); + return count; + } + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + struct mm_subpage *slot = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, pte)); + + if (slot) { + count = mm_subpage_mapcount(slot); + mm_subpage_put(slot); + } + return count; + } + return folio_precise_page_mapcount(folio, page); +} + +#endif + #ifdef CONFIG_PROC_PAGE_MONITOR struct mem_size_stats { unsigned long resident; @@ -913,6 +968,73 @@ static void smaps_page_accumulate(struct mem_size_stats *mss, } } +#ifdef CONFIG_MM_SUBPAGE +static void smaps_account_file_extent(struct mem_size_stats *mss, struct page *page, + unsigned int offset, unsigned long size, bool dirty, bool locked, bool present) +{ + struct folio *folio = page_folio(page); + struct mm_subpage_mapcounts counts = {}; + unsigned int i; + + if (present) + file_quarter_mapcounts(page, &counts); + for (i = offset / SZ_4K; i < (offset + size) / SZ_4K; i++) { + unsigned long pss = SZ_4K << PSS_SHIFT; + + if (counts.slots[i] > 1) + pss /= counts.slots[i]; + smaps_page_accumulate(mss, folio, SZ_4K, pss, dirty, locked, + present && counts.slots[i] < 2); + } +} + +static void smaps_account_small(struct mem_size_stats *mss, struct vm_area_struct *vma, + struct page *page, pte_t pte, bool young, + bool dirty, bool locked, bool present) +{ + struct folio *folio = page_folio(page); + unsigned long size = mm_page_size(vma->vm_mm), pss = size << PSS_SHIFT; + unsigned int count = present && folio_test_anon(folio) ? + proc_leaf_mapcount(vma, pte, page) : 0; + + mss->resident += size; + if (folio_test_anon(folio)) { + mss->anonymous += size; + if (present && !dirty) { + struct mm_subpage *slot = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, pte)); + + if (slot) { + if (mm_subpage_is_lazyfree(slot)) + mss->lazyfree += size; + mm_subpage_put(slot); + } + } + } + if (folio_test_ksm(folio)) + mss->ksm += size; + if (young || folio_test_young(folio) || folio_test_referenced(folio)) + mss->referenced += size; + if (present && !folio_test_anon(folio)) { + smaps_account_file_extent(mss, page, offset_in_page(pte_phys_mm(vma->vm_mm, pte)), + size, dirty, locked, present); + return; + } + if (count > 1) + pss /= count; + smaps_page_accumulate(mss, folio, size, pss, dirty, locked, present && count < 2); +} + +static bool smaps_account_file_quarters(struct mem_size_stats *mss, struct page *page, + bool dirty, bool locked) +{ + if (folio_test_anon(page_folio(page))) + return false; + smaps_account_file_extent(mss, page, 0, PAGE_SIZE, dirty, locked, true); + return true; +} + +#endif + static void smaps_account(struct mem_size_stats *mss, struct page *page, bool compound, bool young, bool dirty, bool locked, bool present) @@ -976,6 +1098,11 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, for (i = 0; i < nr; i++, page++) { unsigned long pss = PAGE_SIZE << PSS_SHIFT; +#ifdef CONFIG_MM_SUBPAGE + if (smaps_account_file_quarters(mss, page, dirty, locked)) + continue; +#endif + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { mapcount = folio_precise_page_mapcount(folio, page); exclusive = mapcount < 2; @@ -995,6 +1122,14 @@ static int smaps_pte_hole(unsigned long addr, unsigned long end, struct mem_size_stats *mss = walk->private; struct vm_area_struct *vma = walk->vma; + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + struct vm_page_offset pos = vma_page_offset_at(vma, addr); + u64 start = ((u64)pos.index << PAGE_SHIFT) + pos.offset; + + mss->swap += shmem_partial_swap_usage_bytes(vma->vm_file->f_mapping, + start, start + end - addr); + return 0; + } mss->swap += shmem_partial_swap_usage(walk->vma->vm_file->f_mapping, linear_page_index(vma, addr), linear_page_index(vma, end)); @@ -1010,7 +1145,7 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk) #ifdef CONFIG_SHMEM if (walk->ops->pte_hole) { /* depth is not used */ - smaps_pte_hole(addr, addr + PAGE_SIZE, 0, walk); + smaps_pte_hole(addr, addr + mm_page_size(walk->mm), 0, walk); } #endif } @@ -1038,15 +1173,20 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, if (softleaf_is_swap(entry)) { int mapcount; - mss->swap += PAGE_SIZE; - mapcount = swp_swapcount(entry); + mss->swap += mm_page_size(vma->vm_mm); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + mapcount = swap_subpage_count(entry, pte_swp_subpage_offset(ptent)); + else +#endif + mapcount = swp_swapcount(entry); if (mapcount >= 2) { - u64 pss_delta = (u64)PAGE_SIZE << PSS_SHIFT; + u64 pss_delta = (u64)mm_page_size(vma->vm_mm) << PSS_SHIFT; do_div(pss_delta, mapcount); mss->swap_pss += pss_delta; } else { - mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT; + mss->swap_pss += (u64)mm_page_size(vma->vm_mm) << PSS_SHIFT; } } else if (softleaf_has_pfn(entry)) { if (softleaf_is_device_private(entry)) @@ -1058,6 +1198,12 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, if (!page) return; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + smaps_account_small(mss, vma, page, ptent, young, dirty, locked, present); + return; + } +#endif smaps_account(mss, page, false, young, dirty, locked, present); } @@ -1124,7 +1270,7 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, walk->action = ACTION_AGAIN; return 0; } - for (; addr != end; pte++, addr += PAGE_SIZE) + for (; addr != end; pte++, addr += mm_page_size(vma->vm_mm)) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); out: @@ -1712,7 +1858,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } - for (; addr != end; pte++, addr += PAGE_SIZE) { + for (; addr != end; pte++, addr += mm_page_size(vma->vm_mm)) { ptent = ptep_get(pte); if (cp->type == CLEAR_REFS_SOFT_DIRTY) { @@ -1855,9 +2001,6 @@ struct pagemapread { bool show_pfn; }; -#define PAGEMAP_WALK_SIZE (PMD_SIZE) -#define PAGEMAP_WALK_MASK (PMD_MASK) - #define PM_ENTRY_BYTES sizeof(pagemap_entry_t) #define PM_PFRAME_BITS 55 #define PM_PFRAME_MASK GENMASK_ULL(PM_PFRAME_BITS - 1, 0) @@ -1884,7 +2027,7 @@ static int add_to_pagemap(pagemap_entry_t *pme, struct pagemapread *pm) return 0; } -static bool __folio_page_mapped_exclusively(struct folio *folio, struct page *page) +static inline bool __folio_page_mapped_exclusively(struct folio *folio, struct page *page) { if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) return folio_precise_page_mapcount(folio, page) == 1; @@ -1909,7 +2052,7 @@ static int pagemap_pte_hole(unsigned long start, unsigned long end, else hole_end = end; - for (; addr < hole_end; addr += PAGE_SIZE) { + for (; addr < hole_end; addr += mm_page_size(walk->mm)) { err = add_to_pagemap(&pme, pm); if (err) goto out; @@ -1921,7 +2064,7 @@ static int pagemap_pte_hole(unsigned long start, unsigned long end, /* Addresses in the VMA. */ if (vma->vm_flags & VM_SOFTDIRTY) pme = make_pme(0, PM_SOFT_DIRTY); - for (; addr < min(end, vma->vm_end); addr += PAGE_SIZE) { + for (; addr < min(end, vma->vm_end); addr += mm_page_size(walk->mm)) { err = add_to_pagemap(&pme, pm); if (err) goto out; @@ -1931,6 +2074,16 @@ static int pagemap_pte_hole(unsigned long start, unsigned long end, return err; } +static bool pagemap_page_exclusive(struct vm_area_struct *vma, pte_t pte, + struct folio *folio, struct page *page) +{ +#ifdef CONFIG_MM_SUBPAGE + return proc_leaf_mapcount(vma, pte, page) == 1; +#else + return __folio_page_mapped_exclusively(folio, page); +#endif +} + static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, struct vm_area_struct *vma, unsigned long addr, pte_t pte) { @@ -1943,7 +2096,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, if (pte_present(pte)) { if (pm->show_pfn) - frame = pte_pfn(pte); + frame = pte_phys_mm(vma->vm_mm, pte) >> mm_page_shift(vma->vm_mm); flags |= PM_PRESENT; page = vm_normal_page(vma, addr, pte); if (pte_soft_dirty(pte)) @@ -1969,6 +2122,12 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, offset = softleaf_to_pfn(entry); else offset = swp_offset(entry); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE && + (softleaf_is_swap(entry) || softleaf_has_pfn(entry))) + offset = (offset << (PAGE_SHIFT - mm_page_shift(vma->vm_mm))) + + (pte_swp_subpage_offset(pte) >> mm_page_shift(vma->vm_mm)); +#endif frame = swp_type(entry) | (offset << MAX_SWAPFILES_SHIFT); } @@ -1985,8 +2144,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, folio = page_folio(page); if (!folio_test_anon(folio)) flags |= PM_FILE; - if ((flags & PM_PRESENT) && - __folio_page_mapped_exclusively(folio, page)) + if ((flags & PM_PRESENT) && pagemap_page_exclusive(vma, pte, folio, page)) flags |= PM_MMAP_EXCLUSIVE; } @@ -2102,7 +2260,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, walk->action = ACTION_AGAIN; return err; } - for (; addr < end; pte++, addr += PAGE_SIZE) { + for (; addr < end; pte++, addr += mm_page_size(walk->mm)) { pagemap_entry_t pme; pme = pte_to_pagemap_entry(pm, vma, addr, ptep_get(pte)); @@ -2236,7 +2394,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* do not disclose physical addresses: attack vector */ pm.show_pfn = file_ns_capable(file, &init_user_ns, CAP_SYS_ADMIN); - pm.len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT); + pm.len = (pmd_size_mm(mm) >> mm_page_shift(mm)); pm.buffer = kmalloc_array(pm.len, PM_ENTRY_BYTES, GFP_KERNEL); ret = -ENOMEM; if (!pm.buffer) @@ -2248,16 +2406,16 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; - if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) { + if (svpfn <= (ULONG_MAX >> mm_page_shift(mm))) { unsigned long end; ret = mmap_read_lock_killable(mm); if (ret) goto out_free; - start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); + start_vaddr = untagged_addr_remote(mm, svpfn << mm_page_shift(mm)); mmap_read_unlock(mm); - end = start_vaddr + ((count / PM_ENTRY_BYTES) << PAGE_SHIFT); + end = start_vaddr + ((count / PM_ENTRY_BYTES) << mm_page_shift(mm)); if (end >= start_vaddr && end < mm->task_size) end_vaddr = end; } @@ -2272,7 +2430,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, unsigned long end; pm.pos = 0; - end = (start_vaddr + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK; + end = (start_vaddr + pmd_size_mm(mm)) & pmd_mask_mm(mm); /* overflow ? */ if (end < start_vaddr || end > end_vaddr) end = end_vaddr; @@ -2333,6 +2491,7 @@ static int pagemap_release(struct inode *inode, struct file *file) struct pagemap_scan_private { struct pm_scan_arg arg; + unsigned long page_size; unsigned long masks_of_interest, cur_vma_category; struct page_region *vec_buf; unsigned long vec_buf_len, vec_buf_index, found_pages; @@ -2559,7 +2718,7 @@ static void pagemap_scan_backout_range(struct pagemap_scan_private *p, else cur_buf->start = cur_buf->end = 0; - p->found_pages -= (end - addr) / PAGE_SIZE; + p->found_pages -= (end - addr) / p->page_size; } #endif @@ -2671,11 +2830,11 @@ static int pagemap_scan_output(unsigned long categories, categories &= p->arg.return_mask; - n_pages = (*end - addr) / PAGE_SIZE; + n_pages = (*end - addr) / p->page_size; if (check_add_overflow(p->found_pages, n_pages, &total_pages) || total_pages > p->arg.max_pages) { size_t n_too_much = total_pages - p->arg.max_pages; - *end -= n_too_much * PAGE_SIZE; + *end -= n_too_much * p->page_size; n_pages -= n_too_much; ret = -ENOSPC; } @@ -2769,7 +2928,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) { /* Fast path for performing exclusive WP */ - for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { + for (addr = start; addr != end; pte++, addr += p->page_size) { pte_t ptent = ptep_get(pte); if ((pte_present(ptent) && pte_uffd_wp(ptent)) || @@ -2778,7 +2937,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, make_uffd_wp_pte(vma, addr, pte, ptent); if (!flush_end) start = addr; - flush_end = addr + PAGE_SIZE; + flush_end = addr + p->page_size; } goto flush_and_return; } @@ -2786,8 +2945,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, if (!p->arg.category_anyof_mask && !p->arg.category_inverted && p->arg.category_mask == PAGE_IS_WRITTEN && p->arg.return_mask == PAGE_IS_WRITTEN) { - for (addr = start; addr < end; pte++, addr += PAGE_SIZE) { - unsigned long next = addr + PAGE_SIZE; + for (addr = start; addr < end; pte++, addr += p->page_size) { + unsigned long next = addr + p->page_size; pte_t ptent = ptep_get(pte); if ((pte_present(ptent) && pte_uffd_wp(ptent)) || @@ -2807,11 +2966,11 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, goto flush_and_return; } - for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { + for (addr = start; addr != end; pte++, addr += p->page_size) { pte_t ptent = ptep_get(pte); unsigned long categories = p->cur_vma_category | pagemap_page_category(p, vma, addr, ptent); - unsigned long next = addr + PAGE_SIZE; + unsigned long next = addr + p->page_size; if (!pagemap_scan_is_interesting_page(categories, p)) continue; @@ -3007,7 +3166,7 @@ static const struct mm_walk_ops pagemap_scan_ops = { .hugetlb_entry = pagemap_scan_hugetlb_entry, }; -static int pagemap_scan_get_args(struct pm_scan_arg *arg, +static int pagemap_scan_get_args(struct mm_struct *mm, struct pm_scan_arg *arg, unsigned long uarg) { if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg))) @@ -3028,7 +3187,7 @@ static int pagemap_scan_get_args(struct pm_scan_arg *arg, arg->vec = untagged_addr((unsigned long)arg->vec); /* Validate memory pointers */ - if (!IS_ALIGNED(arg->start, PAGE_SIZE)) + if (!IS_ALIGNED(arg->start, mm_page_size(mm))) return -EINVAL; if (!access_ok((void __user *)(long)arg->start, arg->end - arg->start)) return -EFAULT; @@ -3041,7 +3200,7 @@ static int pagemap_scan_get_args(struct pm_scan_arg *arg, return -EFAULT; /* Fixup default values */ - arg->end = ALIGN(arg->end, PAGE_SIZE); + arg->end = ALIGN(arg->end, mm_page_size(mm)); arg->walk_end = 0; if (!arg->max_pages) arg->max_pages = ULONG_MAX; @@ -3060,12 +3219,13 @@ static int pagemap_scan_writeback_args(struct pm_scan_arg *arg, return 0; } -static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p) +static int pagemap_scan_init_bounce_buffer(struct mm_struct *mm, + struct pagemap_scan_private *p) { if (!p->arg.vec_len) return 0; - p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT, + p->vec_buf_len = min_t(size_t, pmd_size_mm(mm) >> mm_page_shift(mm), p->arg.vec_len); p->vec_buf = kmalloc_objs(*p->vec_buf, p->vec_buf_len); if (!p->vec_buf) @@ -3106,18 +3266,18 @@ static long pagemap_scan_flush_buffer(struct pagemap_scan_private *p) static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) { - struct pagemap_scan_private p = {0}; + struct pagemap_scan_private p = { .page_size = mm_page_size(mm) }; unsigned long walk_start; size_t n_ranges_out = 0; int ret; - ret = pagemap_scan_get_args(&p.arg, uarg); + ret = pagemap_scan_get_args(mm, &p.arg, uarg); if (ret) return ret; p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask | p.arg.return_mask; - ret = pagemap_scan_init_bounce_buffer(&p); + ret = pagemap_scan_init_bounce_buffer(mm, &p); if (ret) return ret; @@ -3219,16 +3379,17 @@ struct numa_maps_private { struct numa_maps md; }; +static int numa_page_mapcount(struct page *page) +{ + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + return folio_precise_page_mapcount(page_folio(page), page); + return folio_average_page_mapcount(page_folio(page)); +} + static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, - unsigned long nr_pages) + unsigned long nr_pages, int count) { struct folio *folio = page_folio(page); - int count; - - if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) - count = folio_precise_page_mapcount(folio, page); - else - count = folio_average_page_mapcount(folio); md->pages += nr_pages; if (pte_dirty || folio_test_dirty(folio)) @@ -3318,7 +3479,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, page = can_gather_numa_stats_pmd(*pmd, vma, addr); if (page) gather_stats(page, md, pmd_dirty(*pmd), - HPAGE_PMD_SIZE/PAGE_SIZE); + HPAGE_PMD_SIZE/PAGE_SIZE, numa_page_mapcount(page)); spin_unlock(ptl); return 0; } @@ -3333,9 +3494,13 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, struct page *page = can_gather_numa_stats(ptent, vma, addr); if (!page) continue; - gather_stats(page, md, pte_dirty(ptent), 1); +#ifdef CONFIG_MM_SUBPAGE + gather_stats(page, md, pte_dirty(ptent), 1, proc_leaf_mapcount(vma, ptent, page)); +#else + gather_stats(page, md, pte_dirty(ptent), 1, numa_page_mapcount(page)); +#endif - } while (pte++, addr += PAGE_SIZE, addr != end); + } while (pte++, addr += mm_page_size(vma->vm_mm), addr != end); pte_unmap_unlock(orig_pte, ptl); cond_resched(); return 0; @@ -3357,7 +3522,7 @@ static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, page = pte_page(huge_pte); md = walk->private; - gather_stats(page, md, pte_dirty(huge_pte), 1); + gather_stats(page, md, pte_dirty(huge_pte), 1, numa_page_mapcount(page)); out: spin_unlock(ptl); return 0; diff --git a/fs/splice.c b/fs/splice.c index 9d8f63e2fd1abe..4b707ce00e86dd 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -25,6 +25,8 @@ #include #include #include +#include +#include #include #include #include @@ -183,6 +185,79 @@ static void wakeup_pipe_readers(struct pipe_inode_info *pipe) kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN); } +#ifdef CONFIG_MM_SUBPAGE +/* A shared lifetime for all user leaves coalesced into one native pipe page. */ +struct subpage_pipe_refs { + refcount_t refs; + unsigned int count; + struct user_page_fragment fragments[]; +}; + +static void subpage_pipe_buf_release(struct pipe_inode_info *pipe, struct pipe_buffer *buf) +{ + struct subpage_pipe_refs *refs = (void *)buf->private; + + if (refcount_dec_and_test(&refs->refs)) { + release_user_fragments(refs->fragments, refs->count, false); + kfree(refs); + } +} + +static bool subpage_pipe_buf_get(struct pipe_inode_info *pipe, struct pipe_buffer *buf) +{ + struct subpage_pipe_refs *refs = (void *)buf->private; + + return refcount_inc_not_zero(&refs->refs); +} + +static const struct pipe_buf_operations subpage_pipe_buf_ops = { + .release = subpage_pipe_buf_release, + .get = subpage_pipe_buf_get, + /* A gift of user leaves is not ownership of the containing native page. */ +}; + +static int subpage_pipe_buf_retain(struct pipe_buffer *buf) +{ + struct subpage_pipe_refs *refs; + unsigned int pos, count = 0; + + if (!buf->len || buf->ops != &nosteal_pipe_buf_ops || + !folio_test_anon(page_folio(buf->page))) + return 0; + refs = kmalloc(struct_size(refs, fragments, PAGE_SIZE / MM_SUBPAGE_SIZE), GFP_KERNEL); + if (!refs) + return -ENOMEM; + for (pos = round_down(buf->offset, MM_SUBPAGE_SIZE); + pos < buf->offset + buf->len; pos += MM_SUBPAGE_SIZE) { + struct mm_subpage *slot = mm_subpage_get_from_phys(page_to_phys(buf->page) + pos); + + if (!slot) { + release_user_fragments(refs->fragments, count, false); + kfree(refs); + return count ? -EFAULT : 0; + } + refs->fragments[count++] = (struct user_page_fragment) { + .folio = page_folio(buf->page), + .subpage = slot, + .offset = pos, + .length = MM_SUBPAGE_SIZE, + }; + } + refcount_set(&refs->refs, 1); + refs->count = count; + /* Replace the incoming native reference with the typed references. */ + put_page(buf->page); + buf->ops = &subpage_pipe_buf_ops; + buf->private = (unsigned long)refs; + return 0; +} +#else +static int subpage_pipe_buf_retain(struct pipe_buffer *buf) +{ + return 0; +} +#endif + /** * splice_to_pipe - fill passed data into a pipe * @pipe: pipe to fill @@ -221,6 +296,16 @@ ssize_t splice_to_pipe(struct pipe_inode_info *pipe, buf->private = spd->partial[page_nr].private; buf->ops = spd->ops; buf->flags = 0; + { + int err = subpage_pipe_buf_retain(buf); + + if (err) { + buf->ops = NULL; + if (!ret) + ret = err; + break; + } + } head++; pipe->head = head; @@ -852,6 +937,16 @@ ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, continue; } +#ifdef CONFIG_MM_SUBPAGE + /* Other transports still need their downstream lifetime audit. */ + if (buf->ops == &subpage_pipe_buf_ops && + sock->ops->family != AF_UNIX && + sock->ops->family != AF_INET && + sock->ops->family != AF_INET6) { + ret = -EOPNOTSUPP; + break; + } +#endif seg = min_t(size_t, remain, buf->len); ret = pipe_buf_confirm(pipe, buf); @@ -1440,6 +1535,79 @@ static ssize_t __do_splice(struct file *in, loff_t __user *off_in, return ret; } +#ifdef CONFIG_MM_SUBPAGE +static ssize_t subpage_iter_to_pipe(struct iov_iter *from, + struct pipe_inode_info *pipe, unsigned int flags) +{ + struct user_page_fragment fragments[16]; + size_t total = 0; + long ret = 0; + + if (WARN_ON_ONCE(!user_backed_iter(from))) + return -EINVAL; + while (iov_iter_count(from)) { + unsigned int i = 0; + long count; + + /* Normalize empty iovecs before taking the next address. */ + iov_iter_advance(from, 0); + count = get_user_fragments_remote(current->mm, + (unsigned long)iter_iov_addr(from), + min(iov_iter_count(from), iter_iov_len(from)), 0, + fragments, ARRAY_SIZE(fragments)); + if (count <= 0) { + ret = count; + break; + } + while (i < count) { + struct user_page_fragment *first = &fragments[i]; + struct page *page = folio_page(first->folio, first->offset >> PAGE_SHIFT); + unsigned int offset = offset_in_page(first->offset); + unsigned int length = first->length, nr = 1; + struct subpage_pipe_refs *refs; + struct pipe_buffer buf; + + /* Preserve native pipe capacity for contiguous packed quarters. */ + while (i + nr < count && length < PAGE_SIZE - offset) { + struct user_page_fragment *next = &fragments[i + nr]; + + if (folio_page(next->folio, next->offset >> PAGE_SHIFT) != page || + offset_in_page(next->offset) != offset + length) + break; + length += next->length; + nr++; + } + refs = kmalloc(struct_size(refs, fragments, nr), GFP_KERNEL); + if (!refs) { + ret = -ENOMEM; + break; + } + refcount_set(&refs->refs, 1); + refs->count = nr; + memcpy(refs->fragments, first, sizeof(*first) * nr); + buf = (struct pipe_buffer) { + .page = page, + .offset = offset, + .len = length, + .ops = &subpage_pipe_buf_ops, + .flags = flags, + .private = (unsigned long)refs, + }; + i += nr; /* add_to_pipe consumes these references even on failure. */ + ret = add_to_pipe(pipe, &buf); + if (ret < 0) + break; + iov_iter_advance(from, ret); + total += ret; + } + release_user_fragments(fragments + i, count - i, false); + if (ret < 0) + break; + } + return total ? total : ret; +} +#endif + static ssize_t iter_to_pipe(struct iov_iter *from, struct pipe_inode_info *pipe, unsigned int flags) @@ -1451,6 +1619,10 @@ static ssize_t iter_to_pipe(struct iov_iter *from, size_t total = 0; ssize_t ret = 0; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return subpage_iter_to_pipe(from, pipe, flags); +#endif while (iov_iter_count(from)) { struct page *pages[16]; ssize_t left; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index dba1172436b745..3174e3b6edb414 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -301,6 +301,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, pmd_t *pmd, _pmd; pte_t *pte; pte_t ptent; + spinlock_t *ptl; bool ret; assert_fault_locked(vmf); @@ -308,13 +309,13 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, pgd = pgd_offset(mm, address); if (!pgd_present(*pgd)) return true; - p4d = p4d_offset(pgd, address); - if (!p4d_present(*p4d)) + p4d = p4d_offset_mm(mm, pgd, address); + if (!p4d_present_mm(mm, *p4d)) return true; - pud = pud_offset(p4d, address); + pud = pud_offset_mm(mm, p4d, address); if (!pud_present(*pud)) return true; - pmd = pmd_offset(pud, address); + pmd = pmd_offset_mm(mm, pud, address); again: _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) @@ -331,7 +332,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (pmd_trans_huge(_pmd)) return !pmd_write(_pmd) && (reason & VM_UFFD_WP); - pte = pte_offset_map(pmd, address); + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) goto again; @@ -1251,13 +1252,11 @@ static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx, __wake_userfault(ctx, range); } -static __always_inline int validate_unaligned_range( +static __always_inline int validate_byte_range( struct mm_struct *mm, __u64 start, __u64 len) { __u64 task_size = mm->task_size; - if (len & ~PAGE_MASK) - return -EINVAL; if (!len) return -EINVAL; if (start >= task_size) @@ -1269,10 +1268,19 @@ static __always_inline int validate_unaligned_range( return 0; } +static __always_inline int validate_unaligned_range( + struct mm_struct *mm, __u64 start, __u64 len) +{ + if (len & ~mm_page_mask(mm)) + return -EINVAL; + + return validate_byte_range(mm, start, len); +} + static __always_inline int validate_range(struct mm_struct *mm, __u64 start, __u64 len) { - if (start & ~PAGE_MASK) + if (start & ~mm_page_mask(mm)) return -EINVAL; return validate_unaligned_range(mm, start, len); @@ -1428,6 +1436,13 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, ioctls_out = basic_ioctls ? UFFD_API_RANGE_IOCTLS_BASIC : UFFD_API_RANGE_IOCTLS; + if (mm_page_size(mm) != PAGE_SIZE) { + ioctls_out &= (1ULL << _UFFDIO_WAKE) | (1ULL << _UFFDIO_WRITEPROTECT) | + (1ULL << _UFFDIO_CONTINUE) | (1ULL << _UFFDIO_COPY) | + (1ULL << _UFFDIO_ZEROPAGE) | (1ULL << _UFFDIO_POISON); + } + if (mm_page_size(mm) < PAGE_SIZE) + ioctls_out |= 1ULL << _UFFDIO_MOVE; /* * Declare the WP ioctl only if the WP mode is @@ -1645,8 +1660,12 @@ static int userfaultfd_copy(struct userfaultfd_ctx *ctx, sizeof(uffdio_copy)-sizeof(__s64))) goto out; - ret = validate_unaligned_range(ctx->mm, uffdio_copy.src, - uffdio_copy.len); + /* + * COPY reads the pager's address space, which can have a different + * task size and granule from the registered target. Only the target + * range below requires a page-aligned length. + */ + ret = validate_byte_range(current->mm, uffdio_copy.src, uffdio_copy.len); if (ret) goto out; ret = validate_range(ctx->mm, uffdio_copy.dst, uffdio_copy.len); @@ -2015,6 +2034,18 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, /* report all available features and ioctls to userland */ uffdio_api.features = UFFD_API_FEATURES; + /* Base-page anonymous and shared shmem operations support alternative leaves. */ + if (mm_page_size(ctx->mm) != PAGE_SIZE) + uffdio_api.features &= UFFD_FEATURE_PAGEFAULT_FLAG_WP | + UFFD_FEATURE_MISSING_SHMEM | UFFD_FEATURE_MINOR_SHMEM | + UFFD_FEATURE_EVENT_FORK | UFFD_FEATURE_EVENT_REMAP | + UFFD_FEATURE_EVENT_REMOVE | UFFD_FEATURE_EVENT_UNMAP | + UFFD_FEATURE_SIGBUS | UFFD_FEATURE_THREAD_ID | UFFD_FEATURE_EXACT_ADDRESS | + UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC | UFFD_FEATURE_POISON; + + if (mm_page_size(ctx->mm) < PAGE_SIZE) + uffdio_api.features |= UFFD_FEATURE_MOVE; + #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM); @@ -2062,6 +2093,10 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, if (cmd != UFFDIO_API && !userfaultfd_is_initialized(ctx)) return -EINVAL; + /* Larger-than-native MOVE still requires contiguous-owner support. */ + if (cmd == UFFDIO_MOVE && mm_page_size(ctx->mm) > PAGE_SIZE) + return -EOPNOTSUPP; + switch(cmd) { case UFFDIO_API: ret = userfaultfd_api(ctx, arg); diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 57137d3ac15925..619fd3ed406340 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -81,7 +81,7 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp) return NULL; } - return ptdesc_page(ptdesc); + return pgtable_from_page(ptdesc_page(ptdesc)); } #define __pte_alloc_one(...) alloc_hooks(__pte_alloc_one_noprof(__VA_ARGS__)) @@ -111,17 +111,18 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm) * @mm: the mm_struct of the current context * @pte_page: the `struct page` referencing the ptdesc */ -static inline void pte_free(struct mm_struct *mm, struct page *pte_page) +#ifndef __HAVE_ARCH_PTE_FREE +static inline void pte_free(struct mm_struct *mm, pgtable_t pte_page) { - struct ptdesc *ptdesc = page_ptdesc(pte_page); + struct ptdesc *ptdesc = page_ptdesc(pgtable_to_page(pte_page)); pagetable_dtor_free(ptdesc); } +#endif #if CONFIG_PGTABLE_LEVELS > 2 -#ifndef __HAVE_ARCH_PMD_ALLOC_ONE /** * pmd_alloc_one - allocate memory for a PMD-level page table * @mm: the mm_struct of the current context @@ -133,7 +134,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) * * Return: pointer to the allocated memory or %NULL on error */ -static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +static inline pmd_t *__pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) { struct ptdesc *ptdesc; gfp_t gfp = GFP_PGTABLE_USER; @@ -153,6 +154,12 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad return ptdesc_address(ptdesc); } + +#ifndef __HAVE_ARCH_PMD_ALLOC_ONE +static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __pmd_alloc_one_noprof(mm, addr); +} #define pmd_alloc_one(...) alloc_hooks(pmd_alloc_one_noprof(__VA_ARGS__)) #endif diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index bdcc2778ac64f4..900bee13b40cb8 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,6 +222,15 @@ static inline void __tlb_remove_table(void *table) } #endif +/* Architectures may pass opaque table tokens instead of descriptors. */ +#ifndef tlb_table_is_ptdesc +static inline bool tlb_table_is_ptdesc(void *table) +{ + (void)table; + return true; +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_TABLE_FREE */ @@ -295,6 +304,12 @@ extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, in bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, unsigned int nr_pages, bool delay_rmap); +#ifdef CONFIG_MM_SUBPAGE +struct mm_subpage; +/* Consumes one slot reference; true requires flushing the full batch. */ +bool __tlb_remove_subpage(struct mmu_gather *tlb, struct mm_subpage *subpage); +#endif + #ifdef CONFIG_SMP /* * This both sets 'delayed_rmap', and returns true. It would be an inline @@ -543,15 +558,15 @@ static inline void tlb_change_page_size(struct mmu_gather *tlb, static inline unsigned long tlb_get_unmap_shift(struct mmu_gather *tlb) { if (tlb->cleared_ptes) - return PAGE_SHIFT; + return mm_page_shift(tlb->mm); if (tlb->cleared_pmds) - return PMD_SHIFT; + return pmd_shift_mm(tlb->mm); if (tlb->cleared_puds) - return PUD_SHIFT; + return pud_shift_mm(tlb->mm); if (tlb->cleared_p4ds) - return P4D_SHIFT; + return p4d_shift_mm(tlb->mm); - return PAGE_SHIFT; + return mm_page_shift(tlb->mm); } static inline unsigned long tlb_get_unmap_size(struct mmu_gather *tlb) @@ -658,7 +673,7 @@ static inline void __tlb_remove_tlb_entry(struct mmu_gather *tlb, pte_t *ptep, u */ #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ - tlb_flush_pte_range(tlb, address, PAGE_SIZE); \ + tlb_flush_pte_range(tlb, address, mm_page_size((tlb)->mm)); \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) @@ -672,13 +687,15 @@ static inline void __tlb_remove_tlb_entry(struct mmu_gather *tlb, pte_t *ptep, u static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, pte_t *ptep, unsigned int nr, unsigned long address) { - tlb_flush_pte_range(tlb, address, PAGE_SIZE * nr); + unsigned long page_size = mm_page_size(tlb->mm); + + tlb_flush_pte_range(tlb, address, page_size * nr); for (;;) { __tlb_remove_tlb_entry(tlb, ptep, address); if (--nr == 0) break; ptep++; - address += PAGE_SIZE; + address += page_size; } } @@ -739,13 +756,13 @@ static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, * architecture to do its own odd thing, not cause pain for others * http://lkml.kernel.org/r/CA+55aFzBggoXtNXQeng5d_mRoDnaMBE5Y+URs+PHR67nUpMtaw@mail.gmail.com * - * For now w.r.t page table cache, mark the range_size as PAGE_SIZE + * For now w.r.t page table cache, mark the range_size as one mm granule. */ #ifndef pte_free_tlb #define pte_free_tlb(tlb, ptep, address) \ do { \ - tlb_flush_pmd_range(tlb, address, PAGE_SIZE); \ + tlb_flush_pmd_range(tlb, address, mm_page_size((tlb)->mm)); \ tlb->freed_tables = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) @@ -754,7 +771,7 @@ static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, #ifndef pmd_free_tlb #define pmd_free_tlb(tlb, pmdp, address) \ do { \ - tlb_flush_pud_range(tlb, address, PAGE_SIZE); \ + tlb_flush_pud_range(tlb, address, mm_page_size((tlb)->mm)); \ tlb->freed_tables = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) @@ -763,7 +780,7 @@ static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, #ifndef pud_free_tlb #define pud_free_tlb(tlb, pudp, address) \ do { \ - tlb_flush_p4d_range(tlb, address, PAGE_SIZE); \ + tlb_flush_p4d_range(tlb, address, mm_page_size((tlb)->mm)); \ tlb->freed_tables = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) @@ -772,7 +789,7 @@ static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, #ifndef p4d_free_tlb #define p4d_free_tlb(tlb, pudp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, mm_page_size((tlb)->mm)); \ tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) diff --git a/include/drm/drm_gpuvm.h b/include/drm/drm_gpuvm.h index aca218734dcdec..367444b74e8bab 100644 --- a/include/drm/drm_gpuvm.h +++ b/include/drm/drm_gpuvm.h @@ -1165,6 +1165,7 @@ static inline void drm_gpuva_init_from_op(struct drm_gpuva *va, va->va.range = op->va.range; va->gem.obj = op->gem.obj; va->gem.offset = op->gem.offset; + va->gem.range = op->gem.range; } /** diff --git a/include/linux/binfmts.h b/include/linux/binfmts.h index 65abd5ab8836cc..c242a5e9cc758f 100644 --- a/include/linux/binfmts.h +++ b/include/linux/binfmts.h @@ -93,6 +93,8 @@ struct linux_binfmt { #ifdef CONFIG_COREDUMP int (*core_dump)(struct coredump_params *cprm); unsigned long min_coredump; /* minimal dump size */ + /* Optional per-mm replacement for formats with a variable page ABI. */ + unsigned long (*min_coredump_size)(void); #endif } __randomize_layout; @@ -121,6 +123,10 @@ static inline void insert_binfmt(struct linux_binfmt *fmt) extern void unregister_binfmt(struct linux_binfmt *); +#ifdef CONFIG_MM_SUBPAGE +int bprm_set_page_shift(struct linux_binprm *bprm, unsigned int page_shift); +#endif + extern int __must_check remove_arg_zero(struct linux_binprm *); extern int begin_new_exec(struct linux_binprm * bprm); extern void setup_new_exec(struct linux_binprm * bprm); diff --git a/include/linux/bio.h b/include/linux/bio.h index dc17780d6c1e36..714e44044373aa 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -389,11 +389,19 @@ static inline void bio_wouldblock_error(struct bio *bio) * pointed by @iter. If @iter is backed by bvec it's going to be reused * instead of allocating a new one. */ +#ifdef CONFIG_MM_SUBPAGE +int bio_iov_user_vecs(const struct iov_iter *iter, int max_vecs); +#endif + static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) { if (iov_iter_is_bvec(iter)) return 0; +#ifdef CONFIG_MM_SUBPAGE + return bio_iov_user_vecs(iter, max_segs); +#else return iov_iter_npages(iter, max_segs); +#endif } /** diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 8808ee76e73c09..cc9b1eeddd415b 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -267,6 +267,10 @@ struct bio { struct bio_integrity_payload *bi_integrity; /* data integrity */ #endif +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment *bi_user_fragments; + unsigned short bi_nr_user_fragments; +#endif unsigned short bi_vcnt; /* how many bio_vec's */ /* diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a2cf24d1688020..cfec2e3b66979a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -263,6 +263,11 @@ struct btf_record { int wq_off; int refcount_off; int task_work_off; +#ifdef CONFIG_MM_SUBPAGE + /* Private ownership pointer appended to task-storage values with UPTRs. */ + u32 uptr_pins_off; + u32 uptr_extra_size; +#endif struct btf_field fields[]; }; @@ -563,6 +568,15 @@ static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src bpf_obj_memcpy(map->record, dst, src, map->value_size, true); } +static inline u32 bpf_obj_uptr_extra_size(const struct btf_record *rec) +{ +#ifdef CONFIG_MM_SUBPAGE + return IS_ERR_OR_NULL(rec) ? 0 : rec->uptr_extra_size; +#else + return 0; +#endif +} + static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src) { unsigned long *src_uptr, *dst_uptr; @@ -580,6 +594,14 @@ static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, v dst_uptr = dst + field->offset; swap(*src_uptr, *dst_uptr); } +#ifdef CONFIG_MM_SUBPAGE + if (rec->uptr_extra_size) { + void **src_pins = src + rec->uptr_pins_off; + void **dst_pins = dst + rec->uptr_pins_off; + + swap(*src_pins, *dst_pins); + } +#endif } static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size) diff --git a/include/linux/coredump.h b/include/linux/coredump.h index 68861da4cf7c2f..5f85a12c7f9985 100644 --- a/include/linux/coredump.h +++ b/include/linux/coredump.h @@ -12,10 +12,21 @@ struct core_vma_metadata { unsigned long start, end; vm_flags_t flags; unsigned long dump_size; - unsigned long pgoff; + unsigned long pgoff; /* File offset in the dumped mm's page units. */ struct file *file; }; +/* NT_FILE offsets are expressed in the dumped process's page units. */ +static inline unsigned long core_vma_pgoff(const struct vm_area_struct *vma) +{ + unsigned int shift = mm_page_shift(vma->vm_mm); + + if (shift > PAGE_SHIFT) + return vma->vm_pgoff >> (shift - PAGE_SHIFT); + return (vma->vm_pgoff << (PAGE_SHIFT - shift)) + + (vma_subpage_offset(vma) >> shift); +} + struct coredump_params { const kernel_siginfo_t *siginfo; struct file *file; diff --git a/include/linux/highmem.h b/include/linux/highmem.h index d7aac9de1c8a4e..37812575512be3 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -330,6 +330,21 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, } #endif +#ifndef vma_alloc_zeroed_movable_folio_order +static inline struct folio *vma_alloc_zeroed_movable_folio_order( + struct vm_area_struct *vma, unsigned long addr, unsigned int order) +{ + struct folio *folio; + + if (!order) + return vma_alloc_zeroed_movable_folio(vma, addr); + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, order, vma, addr); + if (folio && user_alloc_needs_zeroing()) + clear_user_highpages(&folio->page, addr, folio_nr_pages(folio)); + return folio; +} +#endif + static inline void clear_highpage(struct page *page) { void *kaddr = kmap_local_page(page); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 2949e5acff351b..1f7f729b87868c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -3,6 +3,7 @@ #define _LINUX_HUGE_MM_H #include +#include #include /* only for vma_is_dax() */ #include @@ -331,6 +332,9 @@ struct thpsize { static inline bool vma_thp_disabled(struct vm_area_struct *vma, vm_flags_t vm_flags, bool forced_collapse) { + /* Native THP geometry and rmap cannot describe alternative user leaves. */ + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return true; /* Are THPs disabled for this VMA? */ if (vm_flags & VM_NOHUGEPAGE) return true; diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f61b6cad229b4a..e29fcd83060eb6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -512,6 +512,9 @@ struct io_ring_ctx { struct io_mapped_region ring_region; /* used for optimised request parameter and wait argument passing */ struct io_mapped_region param_region; +#ifdef CONFIG_MM_SUBPAGE + struct io_fragment_account *fragment_account; +#endif }; /* diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index d7a9053ff4fed1..1510fdb0480107 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -8,6 +8,8 @@ extern unsigned int khugepaged_max_ptes_none __read_mostly; #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern struct attribute_group khugepaged_attr_group; +int collapse_file_user_page(struct mm_struct *mm, struct file *file, pgoff_t start); + extern int khugepaged_init(void); extern void khugepaged_destroy(void); extern int start_stop_khugepaged(void); @@ -32,6 +34,12 @@ static inline void khugepaged_exit(struct mm_struct *mm) __khugepaged_exit(mm); } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline int collapse_file_user_page(struct mm_struct *mm, struct file *file, + pgoff_t start) +{ + return -EOPNOTSUPP; +} + static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmm) { } diff --git a/include/linux/mm.h b/include/linux/mm.h index f4481c338b36cb..0536bb678c2e99 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -750,6 +751,9 @@ struct vm_fault { }; struct page *cow_page; /* Page handler may use for COW fault */ +#ifdef CONFIG_MM_SUBPAGE + struct mm_subpage *cow_subpage; /* Owns cow_page's selected fragment */ +#endif struct page *page; /* ->fault handlers should return a * page here, unless VM_FAULT_NOPAGE * is set (which is also implied by @@ -1615,7 +1619,7 @@ static inline unsigned long vma_kernel_pagesize(struct vm_area_struct *vma) { if (unlikely(vma->vm_ops && vma->vm_ops->pagesize)) return vma->vm_ops->pagesize(vma); - return PAGE_SIZE; + return mm_page_size(vma->vm_mm); } unsigned long vma_mmu_pagesize(struct vm_area_struct *vma); @@ -2010,6 +2014,10 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa void set_pte_range(struct vm_fault *vmf, struct folio *folio, struct page *page, unsigned int nr, unsigned long addr); +#ifdef CONFIG_MM_SUBPAGE +vm_fault_t map_file_user_pages(struct vm_fault *vmf, struct folio *folio, + pgoff_t first, pgoff_t last, bool fault); +#endif vm_fault_t finish_fault(struct vm_fault *vmf); #endif @@ -3143,10 +3151,13 @@ struct follow_pfnmap_args { * @address: the virtual address to walk * @write: if true, fail with -EFAULT unless the mapping is * writable + * @allow_subpage: caller handles exact byte offsets and user-leaf bounds; + * otherwise a non-native user granule is rejected */ struct vm_area_struct *vma; unsigned long address; bool write; + bool allow_subpage; /** * Internals: * @@ -3157,13 +3168,15 @@ struct follow_pfnmap_args { /** * Outputs: * - * @pfn: the PFN of the address - * @addr_mask: address mask covering pfn + * @pfn: native PFN containing the requested physical byte + * @offset: byte offset of that byte within the native physical page + * @addr_mask: virtual address mask of the leaf mapping * @pgprot: the pgprot_t of the mapping * @writable: whether the mapping is writable * @special: whether the mapping is a special mapping (real PFN maps) */ unsigned long pfn; + unsigned int offset; unsigned long addr_mask; pgprot_t pgprot; bool writable; @@ -3215,6 +3228,13 @@ static inline void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows) { } #endif +#ifdef CONFIG_MM_SUBPAGE +void unmap_mapping_subpage_tail(struct address_space *mapping, loff_t newsize); +void unmap_mapping_subpage_hole(struct address_space *mapping, loff_t start, u64 end); +#else +static inline void unmap_mapping_subpage_tail(struct address_space *mapping, loff_t newsize) { } +#endif + static inline void unmap_shared_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen) { @@ -3443,7 +3463,8 @@ static inline void reset_mm_hiwater_rss(struct mm_struct *mm) static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, struct mm_struct *mm) { - unsigned long hiwater_rss = get_mm_hiwater_rss(mm); + /* signal->maxrss/cmaxrss survive exec and child reaping: store kB. */ + unsigned long hiwater_rss = mm_pages_to_kb(mm, get_mm_hiwater_rss(mm)); if (*maxrss < hiwater_rss) *maxrss = hiwater_rss; @@ -3514,14 +3535,14 @@ static inline void mm_inc_nr_puds(struct mm_struct *mm) { if (mm_pud_folded(mm)) return; - atomic_long_add(PTRS_PER_PUD * sizeof(pud_t), &mm->pgtables_bytes); + atomic_long_add(pud_table_bytes_mm(mm), &mm->pgtables_bytes); } static inline void mm_dec_nr_puds(struct mm_struct *mm) { if (mm_pud_folded(mm)) return; - atomic_long_sub(PTRS_PER_PUD * sizeof(pud_t), &mm->pgtables_bytes); + atomic_long_sub(pud_table_bytes_mm(mm), &mm->pgtables_bytes); } #endif @@ -3542,14 +3563,14 @@ static inline void mm_inc_nr_pmds(struct mm_struct *mm) { if (mm_pmd_folded(mm)) return; - atomic_long_add(PTRS_PER_PMD * sizeof(pmd_t), &mm->pgtables_bytes); + atomic_long_add(pmd_table_bytes_mm(mm), &mm->pgtables_bytes); } static inline void mm_dec_nr_pmds(struct mm_struct *mm) { if (mm_pmd_folded(mm)) return; - atomic_long_sub(PTRS_PER_PMD * sizeof(pmd_t), &mm->pgtables_bytes); + atomic_long_sub(pmd_table_bytes_mm(mm), &mm->pgtables_bytes); } #endif @@ -3566,12 +3587,12 @@ static inline unsigned long mm_pgtables_bytes(const struct mm_struct *mm) static inline void mm_inc_nr_ptes(struct mm_struct *mm) { - atomic_long_add(PTRS_PER_PTE * sizeof(pte_t), &mm->pgtables_bytes); + atomic_long_add(pte_table_bytes_mm(mm), &mm->pgtables_bytes); } static inline void mm_dec_nr_ptes(struct mm_struct *mm) { - atomic_long_sub(PTRS_PER_PTE * sizeof(pte_t), &mm->pgtables_bytes); + atomic_long_sub(pte_table_bytes_mm(mm), &mm->pgtables_bytes); } #else @@ -3594,32 +3615,40 @@ static inline p4d_t *p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address) { return (unlikely(pgd_none(*pgd)) && __p4d_alloc(mm, pgd, address)) ? - NULL : p4d_offset(pgd, address); + NULL : p4d_offset_mm(mm, pgd, address); } static inline pud_t *pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address) { - return (unlikely(p4d_none(*p4d)) && __pud_alloc(mm, p4d, address)) ? - NULL : pud_offset(p4d, address); + return (unlikely(p4d_none_mm(mm, *p4d)) && __pud_alloc(mm, p4d, address)) ? + NULL : pud_offset_mm(mm, p4d, address); } static inline pmd_t *pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) { return (unlikely(pud_none(*pud)) && __pmd_alloc(mm, pud, address))? - NULL: pmd_offset(pud, address); + NULL : pmd_offset_mm(mm, pud, address); } #endif /* CONFIG_MMU */ enum pt_flags { PT_kernel = PG_referenced, PT_reserved = PG_reserved, + PT_fragmented = PG_private, + PT_frag_defer = PG_active, /* High bits are used for zone/node/section */ }; static inline struct ptdesc *virt_to_ptdesc(const void *x) { - return page_ptdesc(virt_to_page(x)); + struct page *page = virt_to_page(x); + +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + /* One hardware table may span multiple native allocator pages. */ + page = compound_head(page); +#endif + return page_ptdesc(page); } /** @@ -3836,12 +3865,25 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm, } pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); +#ifdef CONFIG_ARCH_HAS_MM_PGTABLE +pte_t *__pte_offset_map_mm(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp); +#else +#define __pte_offset_map_mm(mm, pmd, addr, pmdvalp) \ + __pte_offset_map(pmd, addr, pmdvalp) +#endif static inline pte_t *pte_offset_map(pmd_t *pmd, unsigned long addr) { return __pte_offset_map(pmd, addr, NULL); } +static inline pte_t *pte_offset_map_mm(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr) +{ + return __pte_offset_map_mm(mm, pmd, addr, NULL); +} + pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); @@ -3859,7 +3901,7 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, #define pte_alloc(mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd)) #define pte_alloc_map(mm, pmd, address) \ - (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address)) + (pte_alloc(mm, pmd) ? NULL : pte_offset_map_mm(mm, pmd, address)) #define pte_alloc_map_lock(mm, pmd, address, ptlp) \ (pte_alloc(mm, pmd) ? \ @@ -3874,7 +3916,7 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, static inline struct page *pmd_pgtable_page(pmd_t *pmd) { unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1); - return virt_to_page((void *)((unsigned long) pmd & mask)); + return ptdesc_page(virt_to_ptdesc((void *)((unsigned long)pmd & mask))); } static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd) @@ -4160,6 +4202,12 @@ extern unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf); +#ifdef CONFIG_MM_SUBPAGE +unsigned long do_mmap_offset(struct file *file, unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, struct vm_page_offset pos, unsigned long *populate, + struct list_head *uf); +#endif extern int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool unlock); @@ -4288,7 +4336,11 @@ static inline unsigned long vma_pages(const struct vm_area_struct *vma) static inline unsigned long vma_last_pgoff(struct vm_area_struct *vma) { +#ifdef CONFIG_MM_SUBPAGE + return vma_page_offset_at(vma, vma->vm_end - 1).index; +#else return vma->vm_pgoff + vma_pages(vma) - 1; +#endif } static inline unsigned long vma_desc_size(const struct vm_area_desc *desc) @@ -4301,6 +4353,15 @@ static inline unsigned long vma_desc_pages(const struct vm_area_desc *desc) return vma_desc_size(desc) >> PAGE_SHIFT; } +static inline unsigned int mmap_action_remap_offset(const struct mmap_action *action) +{ +#ifdef CONFIG_MM_SUBPAGE + return action->remap.offset; +#else + return 0; +#endif +} + /** * mmap_action_remap - helper for mmap_prepare hook to specify that a pure PFN * remap is required. @@ -4326,6 +4387,9 @@ static inline void mmap_action_remap(struct vm_area_desc *desc, action->remap.start_pfn = start_pfn; action->remap.size = size; action->remap.pgprot = desc->page_prot; +#ifdef CONFIG_MM_SUBPAGE + action->remap.offset = 0; +#endif } /** @@ -4539,8 +4603,23 @@ struct vm_area_struct *find_extend_vma_locked(struct mm_struct *, unsigned long addr); int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, unsigned long size, pgprot_t pgprot); +#ifdef CONFIG_MMU +int remap_pfn_range_offset(struct vm_area_struct *vma, unsigned long addr, + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot); +#else +static inline int remap_pfn_range_offset(struct vm_area_struct *vma, unsigned long addr, + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) +{ + if (offset) + return -EINVAL; + return remap_pfn_range(vma, addr, pfn, size, prot); +} +#endif int vm_insert_page(struct vm_area_struct *, unsigned long addr, struct page *); +int vm_insert_pages_range(struct vm_area_struct *vma, unsigned long addr, + struct page **pages, unsigned long nr_pages, + unsigned long offset, unsigned long length); int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num); int map_kernel_pages_prepare(struct vm_area_desc *desc); @@ -4552,6 +4631,9 @@ int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, unsigned long num); vm_fault_t vmf_insert_page_mkwrite(struct vm_fault *vmf, struct page *page, bool write); +vm_fault_t vmf_insert_pfn_prot_mkwrite_offset(struct vm_area_struct *vma, + unsigned long addr, unsigned long pfn, unsigned int offset, + pgprot_t pgprot, bool mkwrite); vm_fault_t vmf_insert_pfn_prot_mkwrite(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, pgprot_t pgprot, bool mkwrite); vm_fault_t vmf_insert_mixed(struct vm_area_struct *vma, unsigned long addr, @@ -4648,6 +4730,18 @@ static inline vm_fault_t vmf_insert_page(struct vm_area_struct *vma, return VM_FAULT_NOPAGE; } +/* Insert one user granule from an explicit offset within a native page. */ +#ifdef CONFIG_MM_SUBPAGE +vm_fault_t vmf_insert_page_offset(struct vm_area_struct *vma, unsigned long addr, + struct page *page, unsigned int offset); +#else +static inline vm_fault_t vmf_insert_page_offset(struct vm_area_struct *vma, + unsigned long addr, struct page *page, unsigned int offset) +{ + return offset ? VM_FAULT_SIGBUS : vmf_insert_page(vma, addr, page); +} +#endif + #ifndef io_remap_pfn_range_pfn static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn, unsigned long size) @@ -4656,14 +4750,24 @@ static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn, } #endif +static inline int io_remap_pfn_range_offset(struct vm_area_struct *vma, + unsigned long addr, unsigned long orig_pfn, unsigned int offset, + unsigned long size, pgprot_t orig_prot) +{ + const pgprot_t prot = pgprot_decrypted(orig_prot); + unsigned long pfn; + + if (size > ULONG_MAX - offset) + return -EINVAL; + pfn = io_remap_pfn_range_pfn(orig_pfn, size + offset); + return remap_pfn_range_offset(vma, addr, pfn, offset, size, prot); +} + static inline int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, unsigned long orig_pfn, unsigned long size, pgprot_t orig_prot) { - const unsigned long pfn = io_remap_pfn_range_pfn(orig_pfn, size); - const pgprot_t prot = pgprot_decrypted(orig_prot); - - return remap_pfn_range(vma, addr, pfn, size, prot); + return io_remap_pfn_range_offset(vma, addr, orig_pfn, 0, size, orig_prot); } static inline vm_fault_t vmf_error(int err) @@ -5186,6 +5290,8 @@ static inline void setup_nr_node_ids(void) {} #endif extern int memcmp_pages(struct page *page1, struct page *page2); +int memcmp_pages_range(struct page *page1, unsigned int offset1, + struct page *page2, unsigned int offset2, size_t len); static inline int pages_identical(struct page *page1, struct page *page2) { diff --git a/include/linux/mm_granule.h b/include/linux/mm_granule.h new file mode 100644 index 00000000000000..6e73216edcf96d --- /dev/null +++ b/include/linux/mm_granule.h @@ -0,0 +1,245 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_MM_GRANULE_H +#define _LINUX_MM_GRANULE_H + +#include +#include +#include + +/* + * User virtual mapping granule. Never use these for physical PFNs, struct + * page/folio sizes, page-cache indices, or physical memory accounting. + * The granule is immutable once an mm has mappings or is visible to tasks. + */ +#ifndef arch_mm_page_shift +static inline unsigned int arch_mm_page_shift(const struct mm_struct *mm) +{ + return PAGE_SHIFT; +} +#endif + +static inline unsigned int mm_page_shift(const struct mm_struct *mm) +{ + return arch_mm_page_shift(mm); +} + +static inline unsigned long mm_page_size(const struct mm_struct *mm) +{ + return 1UL << mm_page_shift(mm); +} + +/* A byte-range reference must fit both the user leaf and a native page. */ +static inline unsigned long mm_user_fragment_size(const struct mm_struct *mm) +{ + return mm_page_shift(mm) < PAGE_SHIFT ? mm_page_size(mm) : PAGE_SIZE; +} + +/* Native physical pages covered by one base user PTE, including fragments. */ +static inline unsigned long mm_pte_native_pages(const struct mm_struct *mm) +{ + unsigned long pages = mm_page_size(mm) >> PAGE_SHIFT; + + return pages ?: 1; +} + +static inline unsigned long mm_page_mask(const struct mm_struct *mm) +{ + return ~(mm_page_size(mm) - 1); +} + +/* + * ASLR tunables count native-page bits. Retain their byte window when a + * process uses another granule, including the extra low user-page bits. + * native_mask must be a contiguous low-bit mask whose byte span fits ulong. + */ +static inline unsigned long mm_aslr_mask(const struct mm_struct *mm, + unsigned long native_mask) +{ + unsigned int shift = mm_page_shift(mm); + + if (shift < PAGE_SHIFT) { + unsigned int extra = PAGE_SHIFT - shift; + + return (native_mask << extra) | ((1UL << extra) - 1); + } + return native_mask >> (shift - PAGE_SHIFT); +} + +/* Virtual-range accounting. Physical pages and cache indices stay native. */ +static inline unsigned long vma_user_pages(const struct vm_area_struct *vma) +{ + return (vma->vm_end - vma->vm_start) >> mm_page_shift(vma->vm_mm); +} + +static inline unsigned long mm_page_align(const struct mm_struct *mm, unsigned long bytes) +{ + return (bytes + mm_page_size(mm) - 1) & mm_page_mask(mm); +} + +/* + * RSS and virtual counters use the target mm's user granule. Consumers + * comparing different mms or physical memory must convert explicitly. + */ +static inline unsigned long mm_pages_to_bytes(const struct mm_struct *mm, + unsigned long pages) +{ + return pages << mm_page_shift(mm); +} + +static inline unsigned long mm_pages_to_kb(const struct mm_struct *mm, + unsigned long pages) +{ + return pages << (mm_page_shift(mm) - 10); +} + +/* Round up only when reducing precision; do not hide a partial user page. */ +static inline unsigned long mm_pages_to_shift(const struct mm_struct *mm, + unsigned long pages, unsigned int shift) +{ + unsigned int source = mm_page_shift(mm); + + if (source >= shift) + return pages << (source - shift); + return (pages >> (shift - source)) + + !!(pages & ((1UL << (shift - source)) - 1)); +} + +static inline unsigned long mm_pages_to_native(const struct mm_struct *mm, + unsigned long pages) +{ + return mm_pages_to_shift(mm, pages, PAGE_SHIFT); +} + +/* A page-count proc ABI uses the reader's granule, including remote reads. */ +static inline unsigned long mm_pages_for_reader(const struct mm_struct *mm, + const struct mm_struct *reader, unsigned long pages) +{ + return mm_pages_to_shift(mm, pages, + reader ? mm_page_shift(reader) : PAGE_SHIFT); +} + +/* + * Time integrals need one unit across exec. Locked VM also uses this unit: + * it must represent both logical mappings and native physical charges exactly, + * including a single native-page pin in a larger-granule userspace mm. + */ +#ifdef CONFIG_MM_SUBPAGE +#define MM_ACCOUNT_SHIFT 12 +#else +#define MM_ACCOUNT_SHIFT PAGE_SHIFT +#endif + +static inline unsigned long mm_pages_to_account(const struct mm_struct *mm, + unsigned long pages) +{ + return pages << (mm_page_shift(mm) - MM_ACCOUNT_SHIFT); +} + +static inline unsigned long mm_locked_vm_bytes(const struct mm_struct *mm) +{ + return READ_ONCE(mm->locked_vm) << MM_ACCOUNT_SHIFT; +} + +/* Physical-page limits must include any partial native page of locked VM. */ +static inline unsigned long mm_locked_vm_native_pages(const struct mm_struct *mm) +{ + unsigned long units = READ_ONCE(mm->locked_vm); + unsigned int shift = PAGE_SHIFT - MM_ACCOUNT_SHIFT; + + return (units >> shift) + !!(units & ((1UL << shift) - 1)); +} + +/* File-cache indices remain native-page units even for a 4K VMA. */ +struct vm_page_offset { + pgoff_t index; + unsigned int offset; +}; + +static inline struct vm_page_offset vm_page_offset_add(struct vm_page_offset pos, + unsigned long bytes) +{ + unsigned long low = pos.offset + (bytes & ~PAGE_MASK); + + pos.index += (bytes >> PAGE_SHIFT) + (low >> PAGE_SHIFT); + pos.offset = low & ~PAGE_MASK; + return pos; +} + +static inline struct vm_page_offset vm_page_offset_sub(struct vm_page_offset pos, + unsigned long bytes) +{ + unsigned long low = bytes & ~PAGE_MASK; + + pos.index -= (bytes >> PAGE_SHIFT) + (low > pos.offset); + pos.offset = (pos.offset - low) & ~PAGE_MASK; + return pos; +} + +static inline bool vm_page_offset_equal(struct vm_page_offset a, struct vm_page_offset b) +{ + return a.index == b.index && a.offset == b.offset; +} + +static inline unsigned int vma_subpage_offset(const struct vm_area_struct *vma) +{ +#ifdef CONFIG_MM_SUBPAGE + return vma->vm_subpage_offset; +#else + return 0; +#endif +} + +/* address must be >= vm_start; vm_end computes the next contiguous offset. */ +static inline struct vm_page_offset vma_page_offset_at(const struct vm_area_struct *vma, + unsigned long address) +{ + unsigned long delta = address - vma->vm_start; + unsigned long low = (delta & ~PAGE_MASK) + vma_subpage_offset(vma); + + return (struct vm_page_offset) { + .index = vma->vm_pgoff + (delta >> PAGE_SHIFT) + (low >> PAGE_SHIFT), + .offset = low & ~PAGE_MASK, + }; +} + +/* The start byte of a faulting granule must precede EOF. */ +static inline bool vma_file_offset_valid(const struct vm_area_struct *vma, + unsigned long address, loff_t size) +{ + struct vm_page_offset pos = vma_page_offset_at(vma, address); + pgoff_t end = size >> PAGE_SHIFT; + + return pos.index < end || + (pos.index == end && pos.offset < (size & ~PAGE_MASK)); +} + +static inline void vma_set_page_offset(struct vm_area_struct *vma, struct vm_page_offset pos) +{ + vma->vm_pgoff = pos.index; +#ifdef CONFIG_MM_SUBPAGE + vma->vm_subpage_offset = pos.offset; +#endif +} + +/* Inverse lookup of a single byte; an offset outside the VMA is not clamped. */ +static inline unsigned long vma_address_at_offset(const struct vm_area_struct *vma, + struct vm_page_offset pos) +{ + unsigned long len = vma->vm_end - vma->vm_start; + unsigned long index, delta, last; + unsigned int offset = vma_subpage_offset(vma); + + if (!len || pos.index < vma->vm_pgoff || pos.offset >= PAGE_SIZE) + return -EFAULT; + index = pos.index - vma->vm_pgoff; + last = ((len - 1) >> PAGE_SHIFT) + + ((((len - 1) & ~PAGE_MASK) + offset) >> PAGE_SHIFT); + if (index > last) + return -EFAULT; + delta = (index << PAGE_SHIFT) + pos.offset; + if (delta < offset || delta - offset >= len) + return -EFAULT; + return vma->vm_start + delta - offset; +} + +#endif diff --git a/include/linux/mm_subpage.h b/include/linux/mm_subpage.h new file mode 100644 index 00000000000000..81f70bed1f2c89 --- /dev/null +++ b/include/linux/mm_subpage.h @@ -0,0 +1,322 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_MM_SUBPAGE_H +#define _LINUX_MM_SUBPAGE_H + +#include +#include +#include + +#define MM_SUBPAGE_SHIFT 12 +#define MM_SUBPAGE_SIZE (1UL << MM_SUBPAGE_SHIFT) +#define MM_SUBPAGES_PER_PAGE (PAGE_SIZE / MM_SUBPAGE_SIZE) + +struct mm_subpage_mapcounts { + unsigned int slots[MM_SUBPAGES_PER_PAGE]; +}; + +struct folio; +struct page; +struct mm_struct; +struct mm_subpage; +struct mm_subpage_pool; +struct page_ext_operations; +struct vm_area_struct; + +extern struct page_ext_operations mm_subpage_ext_ops; + +#ifdef CONFIG_MM_SUBPAGE +/* Per-native-page shmem validity; caller holds the containing folio lock. */ +unsigned long mm_subpage_shmem_missing(struct page *page); +void mm_subpage_shmem_set_missing(struct page *page, unsigned long missing); +#endif + +/* + * A pool is one caller-defined allocation/policy/charge domain. The caller + * supplies fresh, unmapped native folios using its normal NUMA and memcg + * allocation path. No mm, VMA, rmap, swap or folio-private field is repurposed. + * Fault paths transfer slot references to PTEs through the rmap helpers below. + * + * Each caller must hold a pool reference through pool operations. close() + * stops allocation and releases unused backing; it does not consume that + * reference. Existing slot references outlive close() and the originating mm. + */ +struct mm_subpage_pool *mm_subpage_pool_create(gfp_t gfp); +struct mm_subpage_pool *mm_subpage_pool_create_granule(gfp_t gfp, unsigned int shift); +void mm_subpage_pool_get(struct mm_subpage_pool *pool); +void mm_subpage_pool_put(struct mm_subpage_pool *pool); +void mm_subpage_pool_close(struct mm_subpage_pool *pool); +#ifdef CONFIG_MM_SUBPAGE +/* Optional per-mm pool: default policy, one memory node, matching memcg. */ +struct mm_subpage_pool *mm_subpage_cow_pool_get(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long address); +/* Fork's child owns the pool before it joins a cgroup; charge the source mm. */ +struct mm_subpage_pool *mm_subpage_cow_pool_get_charged(struct mm_struct *mm, + struct mm_struct *charge_mm, struct vm_area_struct *vma, + unsigned long address); +void mm_subpage_cow_pool_exit(struct mm_struct *mm); +bool mm_subpage_cow_folio_matches(struct mm_subpage_pool *pool, struct folio *folio); +bool mm_subpage_cow_wait_busy_at(struct mm_subpage_pool *pool, unsigned int offset); +/* COW slots may use any physical quarter; rmap retains the virtual position. */ +bool mm_subpage_cow_wait_busy(struct mm_subpage_pool *pool); +struct mm_subpage *mm_subpage_alloc_locked(struct mm_subpage_pool *pool); +struct mm_subpage *mm_subpage_copy_locked(struct mm_subpage_pool *pool, + const struct mm_subpage *source); +int mm_subpage_prepare_anon_rmap(struct mm_subpage *slot, + struct vm_area_struct *vma, unsigned long address); +/* Exact physical-slot mask, one owner and folio lock for the entire batch. + * slots is indexed by physical offset >> pool shift; only mask bits are set. + * Success returns initialized owned slots; errors leave slots untouched. + */ +int mm_subpage_alloc_mask_locked(struct mm_subpage_pool *pool, unsigned long mask, + struct mm_subpage **slots); +int mm_subpage_pool_add_folio_mask(struct mm_subpage_pool *pool, struct folio *folio, + unsigned long mask, gfp_t gfp); +bool mm_subpage_cow_wait_busy_mask(struct mm_subpage_pool *pool, unsigned long mask); +/* Arbitrary physical offsets, one owner/folio lock for the whole logical mask. + * Output slots are indexed by logical mask bits; use each slot's actual offset. + * Count-aware supply/wait must accompany this allocation to guarantee progress. + */ +int mm_subpage_alloc_any_mask_locked(struct mm_subpage_pool *pool, unsigned long mask, + struct mm_subpage **slots); +int mm_subpage_pool_add_folio_count(struct mm_subpage_pool *pool, struct folio *folio, + unsigned int count, gfp_t gfp); +bool mm_subpage_cow_wait_busy_count(struct mm_subpage_pool *pool, unsigned int count); +/* Return an initialized slot with its folio locked; may wait without a pool lock. */ +struct mm_subpage *mm_subpage_alloc_at_locked(struct mm_subpage_pool *pool, + unsigned int offset); +struct mm_subpage *mm_subpage_copy_at_locked(struct mm_subpage_pool *pool, + const struct mm_subpage *source, unsigned int offset); +#else +static inline void mm_subpage_cow_pool_exit(struct mm_struct *mm) { } +#endif + +/* Success consumes the folio reference. On error the caller retains it. */ +int mm_subpage_pool_add_folio(struct mm_subpage_pool *pool, struct folio *folio, + gfp_t gfp); +int mm_subpage_pool_add_folio_at(struct mm_subpage_pool *pool, struct folio *folio, + unsigned int offset, gfp_t gfp); +unsigned long mm_subpage_pool_backing_pages(struct mm_subpage_pool *pool); + +/* -EAGAIN requires more compatible backing; -ESHUTDOWN means pool closed. */ +struct mm_subpage *mm_subpage_alloc(struct mm_subpage_pool *pool); +struct mm_subpage *mm_subpage_copy(struct mm_subpage_pool *pool, + const struct mm_subpage *source); +/* Exact physical byte offset within a native folio; no other slot is substituted. */ +struct mm_subpage *mm_subpage_alloc_at(struct mm_subpage_pool *pool, unsigned int offset); +struct mm_subpage *mm_subpage_copy_at(struct mm_subpage_pool *pool, + const struct mm_subpage *source, unsigned int offset); + +/* + * Reattach a slot to an uptodate, locked order-0 swap-cache folio, without + * zeroing or copying its bytes. Caller holds a native folio reference and + * retains it. The returned slot owns an additional reference. -EAGAIN means + * ownership changed concurrently; -ENOMEM means metadata allocation failed. + */ +struct mm_subpage *mm_subpage_restore(struct folio *folio, unsigned int offset, gfp_t gfp); +struct mm_subpage *mm_subpage_restore_granule(struct folio *folio, unsigned int offset, + unsigned int shift, gfp_t gfp); + +/* PTE lock held, one file-backed alternative leaf per call. Native mms are no-ops. */ +#ifdef CONFIG_MM_SUBPAGE +void __mm_subpage_file_map_add(phys_addr_t phys, unsigned long size); +void __mm_subpage_file_map_del(phys_addr_t phys, unsigned long size); +static inline void mm_subpage_file_map_add(struct vm_area_struct *vma, phys_addr_t phys) +{ + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + __mm_subpage_file_map_add(phys, mm_page_size(vma->vm_mm)); +} +static inline void mm_subpage_file_map_del(struct vm_area_struct *vma, phys_addr_t phys) +{ + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + __mm_subpage_file_map_del(phys, mm_page_size(vma->vm_mm)); +} +/* Fill 4K coverage counts; return alternative leaf count. Statistics only. */ +unsigned int mm_subpage_file_mapcounts(struct page *page, + struct mm_subpage_mapcounts *counts); +#else +static inline void mm_subpage_file_map_add(struct vm_area_struct *vma, phys_addr_t phys) { } +static inline void mm_subpage_file_map_del(struct vm_area_struct *vma, phys_addr_t phys) { } +#endif + +/* Both folios locked; preallocate destination metadata before removing PTEs. */ +#ifdef CONFIG_MM_SUBPAGE +int mm_subpage_migrate_prepare(struct folio *src, struct folio *dst); +bool mm_subpage_migrating(struct folio *folio); +void mm_subpage_migrate_save_lazyfree(struct mm_subpage *subpage); +/* Infallible metadata reuse after prepare; caller still holds the folio lock. */ +struct mm_subpage *mm_subpage_migrate_restore(struct folio *folio, unsigned int offset); +void mm_subpage_migrate_finish(struct folio *folio); +#else +static inline int mm_subpage_migrate_prepare(struct folio *src, struct folio *dst) +{ + return 0; +} +static inline bool mm_subpage_migrating(struct folio *folio) +{ + return false; +} +static inline void mm_subpage_migrate_finish(struct folio *folio) { } +#endif + +/* + * get() requires a live reference. A mapping reference must not be put until + * its PTE is removed and TLB invalidation completes. Every pin/alias holds + * its own reference: the slot becomes reusable only after the final put. + * Every live slot reference also holds one reference to the native folio. + * copy() requires the caller to stabilise the source contents (as for COW). + * A failed data copy returns -EHWPOISON without publishing the destination. + */ +void mm_subpage_get(struct mm_subpage *subpage); +void mm_subpage_put(struct mm_subpage *subpage); +/* + * Acquire the live identity containing a 4K-aligned physical address, or NULL. + * Interior quarters of a larger identity return that same identity. + * The caller must stabilise the physical allocation, e.g. with a live PTE + * under its lock or a folio reference. A lockless walker additionally needs + * GUP's TLB/free ordering; RCU protects metadata, not arbitrary physical PFNs. + * This is not a mapping identity check: a lockless PTE reader must revalidate + * its PTE after acquisition, just as for speculative folio references. + */ +struct mm_subpage *mm_subpage_get_from_phys(phys_addr_t phys); +/* + * Optional shared sets for asynchronous consumers of native byte spans. + * add() requires the source slot identities to remain live for the call. + * The pointer being modified is exclusively owned by the caller; get() shares + * a read-only set, and mutation uses copy-on-write. Duplicate ranges add no + * references. prune() requires a stable, side-effect-free membership predicate. + * Allocation failure leaves the original set intact. put() accepts NULL. + */ +struct mm_subpage_refs; +#ifdef CONFIG_MM_SUBPAGE +struct mm_subpage_refs *mm_subpage_refs_get(struct mm_subpage_refs *refs); +void mm_subpage_refs_put(struct mm_subpage_refs *refs); +int mm_subpage_refs_add(struct mm_subpage_refs **refs, struct page *page, + unsigned int offset, unsigned int len, gfp_t gfp); +int mm_subpage_refs_prune(struct mm_subpage_refs **refs, + bool (*keep)(const struct mm_subpage *, void *), + void *context, gfp_t gfp); +#else +static inline struct mm_subpage_refs *mm_subpage_refs_get(struct mm_subpage_refs *refs) +{ + return refs; +} +static inline void mm_subpage_refs_put(struct mm_subpage_refs *refs) { } +static inline int mm_subpage_refs_add(struct mm_subpage_refs **refs, struct page *page, + unsigned int offset, unsigned int len, gfp_t gfp) +{ + return 0; +} +static inline int mm_subpage_refs_prune(struct mm_subpage_refs **refs, + bool (*keep)(const struct mm_subpage *, void *), void *context, gfp_t gfp) +{ + return 0; +} +#endif + +/* + * Anonymous rmap attachment: caller holds the folio lock and PTE lock, has + * prepared vma->anon_vma, and transfers an existing slot reference to its PTE. + * A folio may cover only one native linear index and anon_vma root; its slot + * offset must match the VMA's linear byte offset. Removal needs the PTE lock. + * These helpers do not install PTEs, update RSS, enqueue on LRU, or drop refs. + */ +int mm_subpage_add_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma, + unsigned long address); +int mm_subpage_add_new_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma, + unsigned long address); +void mm_subpage_remove_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma); +/* Both PTE locks and source write_protect_seq held; both mappings become R/O. */ +int mm_subpage_dup_anon_rmap(struct mm_subpage *subpage, + struct vm_area_struct *dst, struct vm_area_struct *src, unsigned long address); +/* + * Per-slot anonymous exclusivity. Setting requires the folio and PTE locks, + * and proof of a fresh private slot or an exclusive migration/swap entry. + * Sharing requires the PTE lock and fails while a typed pin exists. + */ +void mm_subpage_set_exclusive(struct mm_subpage *subpage); +bool mm_subpage_is_exclusive(const struct mm_subpage *subpage); +/* + * Folio and PTE locked; caller proves one remaining swap PTE for this slot. + * Rmap has been attached, but its PTE is not published yet. Reject old GET + * references or present aliases before recovering lost exclusive ownership. + */ +bool mm_subpage_try_reuse_swap(struct mm_subpage *subpage); +/* Present PTE plus caller lookup ref; caller proves no swapped aliases. */ +bool mm_subpage_try_reuse_anon(struct mm_subpage *subpage); +/* PTE locked, caller holds one GET; retire the PTE/TLB before dropping its ref. */ +bool mm_subpage_can_zap(struct mm_subpage *subpage); +bool mm_subpage_try_share(struct mm_subpage *subpage); +/* Folio/PTE locked, caller owns a lookup reference. */ +bool mm_subpage_mark_lazyfree(struct mm_subpage *subpage); +bool mm_subpage_all_lazyfree(struct mm_subpage *subpage); +bool mm_subpage_discardable(struct mm_subpage *subpage); +bool mm_subpage_is_lazyfree(const struct mm_subpage *subpage); +/* Hold a slot reference through cancellation; serializes with discard checks. */ +void mm_subpage_cancel_lazyfree(struct mm_subpage *subpage); +/* + * Caller holds a live slot reference and its PTE lock. Success takes an + * additional slot reference plus a native FOLL_PIN reference. The pin keeps + * the exact fragment occupied after unmap; release only with subpage_unpin. + * This is not a GUP walker and does not validate user addresses/permissions. + */ +int mm_subpage_pin(struct mm_subpage *subpage); +void mm_subpage_unpin(struct mm_subpage *subpage); +unsigned int mm_subpage_pincount(const struct mm_subpage *subpage); +#ifdef CONFIG_MM_SUBPAGE +/* A referenced/pinned folio range; never infer offset from the user VA. */ +struct user_page_fragment { + struct folio *folio; + struct mm_subpage *subpage; + unsigned long offset; + unsigned int length; + bool pinned; +}; + +/* + * Takes/releases mmap_lock internally. Returns the number of records filled, + * or -errno if none. A positive partial result is valid; sum record lengths + * to advance the user address. Each record stops at the target mm's granule + * or a native physical-page boundary, whichever comes first. + * Caller owns the target mm lifetime. Release either kind of record with + * release_user_fragments(); copying a record does not acquire a reference. + * FOLL_LONGTERM pins migrate unpinnable backing before returning success. + * Device/PFN mappings are not accepted by this interface. + */ +long pin_user_fragments_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragments, unsigned long capacity); +long get_user_fragments_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragments, unsigned long capacity); +/* + * Single-record GET with a stable VMA. Caller holds mmap_lock throughout; + * this function never releases it. Returns 1 or -errno (0 for zero length). + * FOLL_NOWAIT is not supported. The returned VMA is valid until unlock. + */ +int get_user_fragment_vma_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragment, struct vm_area_struct **vma); +/* Single-record PIN under the caller's mmap_lock, retained throughout. + * Same stable-VMA/partial-record contract as GET above. FOLL_LONGTERM and + * FOLL_NOWAIT are rejected: long-term migration can require dropping locks. + */ +int pin_user_fragment_vma_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragment, struct vm_area_struct **vma); +/* Current mm, one aligned fragment; sparse zero pages return a negative errno. */ +int get_dump_fragment(unsigned long addr, struct user_page_fragment *fragment); +/* Dirty a retained reference without releasing its allocation identity. */ +void mark_user_fragment_dirty(struct user_page_fragment *fragment); +void release_user_fragments(struct user_page_fragment *fragments, + unsigned long count, bool dirty); +#endif + +int mm_subpage_mapcount(const struct mm_subpage *subpage); +/* Borrowed folio: a raw folio reference alone does not keep a slot occupied. */ +struct folio *mm_subpage_folio(const struct mm_subpage *subpage); +unsigned int mm_subpage_shift(const struct mm_subpage *subpage); +unsigned int mm_subpage_size(const struct mm_subpage *subpage); +unsigned int mm_subpage_offset(const struct mm_subpage *subpage); +phys_addr_t mm_subpage_phys(const struct mm_subpage *subpage); + +#endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a308e2c23b829f..180207618dbaea 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -558,7 +558,8 @@ FOLIO_MATCH(compound_info, _head_3); * @__page_mapping: Aliases with page->mapping. Unused for page tables. * @pt_index: Used for s390 gmap. * @pt_mm: Used for x86 pgds. - * @pt_frag_refcount: For fragmented page table tracking. Powerpc only. + * @pt_frag_refcount: For fragmented page table tracking (powerpc, arm64). + * @pt_frag_shift: Immutable arm64 fragment granule, valid through RCU release. * @pt_share_count: Used for HugeTLB PMD page table share count. * @_pt_pad_2: Padding to ensure proper alignment. * @ptl: Lock for the page table. @@ -585,7 +586,15 @@ struct ptdesc { union { pgoff_t pt_index; struct mm_struct *pt_mm; +#ifdef CONFIG_ARM64_USER4K_EXPERIMENTAL + /* arm64 is 64-bit: use the spare half of this existing word. */ + struct { + atomic_t pt_frag_refcount; + u8 pt_frag_shift; + }; +#else atomic_t pt_frag_refcount; +#endif #ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING atomic_t pt_share_count; #endif @@ -829,6 +838,9 @@ struct mmap_action { unsigned long start_pfn; unsigned long size; pgprot_t pgprot; +#ifdef CONFIG_MM_SUBPAGE + unsigned int offset; +#endif } remap; struct { phys_addr_t start_phys_addr; @@ -905,6 +917,9 @@ struct vm_area_desc { /* Mutable fields. Populated with initial state. */ pgoff_t pgoff; +#ifdef CONFIG_MM_SUBPAGE + unsigned int subpage_offset; +#endif struct file *vm_file; vma_flags_t vma_flags; pgprot_t page_prot; @@ -992,6 +1007,9 @@ struct vm_area_struct { /* Information about our backing store: */ unsigned long vm_pgoff; /* Offset (within vm_file) in PAGE_SIZE units */ +#ifdef CONFIG_MM_SUBPAGE + unsigned int vm_subpage_offset; /* Byte offset within vm_pgoff. */ +#endif struct file * vm_file; /* File we map to (can be NULL). */ void * vm_private_data; /* was vm_pte (shared mem) */ @@ -1281,12 +1299,18 @@ struct mm_struct { unsigned int __percpu *futex_ref; #endif - unsigned long hiwater_rss; /* High-watermark of RSS usage */ - unsigned long hiwater_vm; /* High-water virtual memory usage */ + unsigned long hiwater_rss; /* Peak RSS in user granules */ + unsigned long hiwater_vm; /* Peak VM in user granules */ - unsigned long total_vm; /* Total pages mapped */ - unsigned long locked_vm; /* Pages that have PG_mlocked set */ - atomic64_t pinned_vm; /* Refcount permanently increased */ + unsigned long total_vm; /* Mapped user granules */ +#ifdef CONFIG_MM_SUBPAGE + /* VM_ACCOUNT granules; global commitment rounds this sum once. */ + unsigned long committed_user_pages; + /* Shared COW slots; never inherited by a forked mm. */ + struct mm_subpage_cow_pools *cow_subpage_pool; +#endif + unsigned long locked_vm; /* Locked bytes >> MM_ACCOUNT_SHIFT */ + atomic64_t pinned_vm; /* Pinned native physical pages */ unsigned long data_vm; /* VM_WRITE & ~VM_SHARED & ~VM_STACK */ unsigned long exec_vm; /* VM_EXEC & ~VM_WRITE & ~VM_STACK */ unsigned long stack_vm; /* VM_STACK */ diff --git a/include/linux/mman.h b/include/linux/mman.h index 389521594c69c2..7b80c19f6aa5da 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -83,6 +83,17 @@ static inline void vm_unacct_memory(long pages) vm_acct_memory(-pages); } +/* VMA reservations use user granules; shmem/physical reservations use pages. */ +#ifdef CONFIG_MM_SUBPAGE +int mm_account_memory(struct mm_struct *mm, struct mm_struct *policy_mm, long pages); +void mm_acct_memory(struct mm_struct *mm, long pages); +void mm_unacct_memory(struct mm_struct *mm, unsigned long pages); +#else +#define mm_account_memory(mm, policy_mm, pages) security_vm_enough_memory_mm(policy_mm, pages) +#define mm_acct_memory(mm, pages) vm_acct_memory(pages) +#define mm_unacct_memory(mm, pages) vm_unacct_memory(pages) +#endif + /* * Allow architectures to handle additional protection and flag bits. The * overriding macros must be defined in the arch-specific asm/mman.h file. diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index ed3dd0f3fe1946..a4260bc898205c 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -5,6 +5,11 @@ #include #include +/* Select immutable exec geometry before allocating any page tables. */ +#ifndef arch_mm_init_exec +static inline void arch_mm_init_exec(struct mm_struct *mm, unsigned int page_shift) { } +#endif + /* Architectures that care about IRQ state in switch_mm can override this. */ #ifndef switch_mm_irqs_off # define switch_mm_irqs_off switch_mm diff --git a/include/linux/page_table_check.h b/include/linux/page_table_check.h index 12268a32e8be15..3ee1096fb0e53f 100644 --- a/include/linux/page_table_check.h +++ b/include/linux/page_table_check.h @@ -12,6 +12,10 @@ extern struct static_key_true page_table_check_disabled; extern struct page_ext_operations page_table_check_ops; +#if IS_ENABLED(CONFIG_KUNIT) +int page_table_check_get_counts(struct page *page, unsigned int offset, + int *anon, int *file); +#endif void __page_table_check_zero(struct page *page, unsigned int order); void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 31a848485ad9d9..9cde417795f002 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1068,10 +1068,7 @@ static inline pgoff_t folio_pgoff(const struct folio *folio) static inline pgoff_t linear_page_index(const struct vm_area_struct *vma, const unsigned long address) { - pgoff_t pgoff; - pgoff = (address - vma->vm_start) >> PAGE_SHIFT; - pgoff += vma->vm_pgoff; - return pgoff; + return vma_page_offset_at(vma, address).index; } struct wait_page_key { diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 48d851fbd8ea58..14a21f187e9f4d 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -1209,6 +1209,11 @@ perf_cgroup_from_task(struct task_struct *task, struct perf_event_context *ctx) extern struct perf_event_context *perf_cpu_task_ctx(void); +#if IS_ENABLED(CONFIG_KUNIT) +/* The caller must disable IRQs, as for the sampling path. */ +u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr); +#endif + extern void *perf_aux_output_begin(struct perf_output_handle *handle, struct perf_event *event); extern void perf_aux_output_end(struct perf_output_handle *handle, diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index cdd68ed3ae1a90..5c67d9b5d2f8df 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -12,11 +12,18 @@ #ifdef CONFIG_MMU #include +#include #include #include #include #include +/* pgtable_t is an opaque allocation token, not necessarily struct page *. */ +#ifndef pgtable_to_page +#define pgtable_to_page(table) (table) +#define pgtable_from_page(page) (page) +#endif + #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) != CONFIG_PGTABLE_LEVELS #error CONFIG_PGTABLE_LEVELS is not consistent with __PAGETABLE_{P4D,PUD,PMD}_FOLDED @@ -155,6 +162,59 @@ static inline pgd_t *pgd_offset_pgd(pgd_t *pgd, unsigned long address) */ #define pgd_offset_k(address) pgd_offset(&init_mm, (address)) +/* Explicit target-mm accessors; legacy accessors keep native geometry. */ +#ifndef pte_phys_mm +#define pte_phys_mm(mm, pte) PFN_PHYS(pte_pfn(pte)) +#endif +#ifndef phys_pte_mm +#define phys_pte_mm(mm, phys, prot) pfn_pte((phys) >> PAGE_SHIFT, prot) +#endif +#ifndef pte_pgprot_mm +#define pte_pgprot_mm(mm, pte) pte_pgprot(pte) +#endif +#ifndef pgd_offset_pgd_mm +#define pgd_offset_pgd_mm(mm, pgd, addr) pgd_offset_pgd(pgd, addr) +#endif +#ifndef p4d_offset_mm +#define p4d_offset_mm(mm, pgd, addr) p4d_offset(pgd, addr) +#endif +#ifndef pud_offset_mm +#define pud_offset_mm(mm, p4d, addr) pud_offset(p4d, addr) +#endif +#ifndef pmd_offset_mm +#define pmd_offset_mm(mm, pud, addr) pmd_offset(pud, addr) +#endif +#ifndef __pte_map_mm +#define __pte_map_mm(mm, pmd, addr) __pte_map(pmd, addr) +#endif +#ifndef pte_offset_kernel_mm +#define pte_offset_kernel_mm(mm, pmd, addr) pte_offset_kernel(pmd, addr) +#endif + +#ifndef pud_pgtable_mm +#define pud_pgtable_mm(mm, pud) pud_pgtable(pud) +#endif + +#ifndef pmd_size_mm +#define pmd_shift_mm(mm) PMD_SHIFT +#define pud_shift_mm(mm) PUD_SHIFT +#define p4d_shift_mm(mm) P4D_SHIFT +#define pmd_size_mm(mm) PMD_SIZE +#define pud_size_mm(mm) PUD_SIZE +#define p4d_size_mm(mm) P4D_SIZE +#define pgd_size_mm(mm) PGDIR_SIZE +#define pmd_mask_mm(mm) PMD_MASK +#define pud_mask_mm(mm) PUD_MASK +#define p4d_mask_mm(mm) P4D_MASK +#define pgd_mask_mm(mm) PGDIR_MASK +#endif + +#ifndef pte_table_bytes_mm +#define pte_table_bytes_mm(mm) (PTRS_PER_PTE * sizeof(pte_t)) +#define pmd_table_bytes_mm(mm) (PTRS_PER_PMD * sizeof(pmd_t)) +#define pud_table_bytes_mm(mm) (PTRS_PER_PUD * sizeof(pud_t)) +#endif + /* * In many cases it is known that a virtual address is mapped at PMD or PTE * level, so instead of traversing all the page table levels, we can get a @@ -388,6 +448,10 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) #define pte_next_pfn(pte) pte_advance_pfn(pte, 1) +#ifndef pte_advance_pfn_mm +#define pte_advance_pfn_mm(mm, pte, nr) pte_advance_pfn(pte, nr) +#endif + #ifndef set_ptes /** * set_ptes - Map consecutive pages to a contiguous range of addresses. @@ -1465,6 +1529,13 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) * and any p?d_bad entries - reporting the error before resetting to none. * Do the tests inline, but report and clear the bad entry in mm/memory.c. */ +#ifndef pgd_addr_end_mm +#define pgd_addr_end_mm(mm, addr, end) pgd_addr_end(addr, end) +#define p4d_addr_end_mm(mm, addr, end) p4d_addr_end(addr, end) +#define pud_addr_end_mm(mm, addr, end) pud_addr_end(addr, end) +#define pmd_addr_end_mm(mm, addr, end) pmd_addr_end(addr, end) +#endif + void pgd_clear_bad(pgd_t *); #ifndef __PAGETABLE_P4D_FOLDED @@ -1492,6 +1563,25 @@ static inline int pgd_none_or_clear_bad(pgd_t *pgd) return 0; } +#ifndef p4d_none_mm +#define p4d_none_mm(mm, p4d) p4d_none(p4d) +#define p4d_bad_mm(mm, p4d) p4d_bad(p4d) +#define p4d_present_mm(mm, p4d) p4d_present(p4d) +#define p4d_clear_mm(mm, p4d) p4d_clear(p4d) +#endif + +static inline int p4d_none_or_clear_bad_mm(struct mm_struct *mm, p4d_t *p4d) +{ + if (p4d_none_mm(mm, *p4d)) + return 1; + if (unlikely(p4d_bad_mm(mm, *p4d))) { + p4d_ERROR(*p4d); + p4d_clear_mm(mm, p4d); + return 1; + } + return 0; +} + static inline int p4d_none_or_clear_bad(p4d_t *p4d) { if (p4d_none(*p4d)) @@ -1979,11 +2069,15 @@ static inline int is_zero_pfn(unsigned long pfn) #define zero_pfn(addr) page_to_pfn(ZERO_PAGE(addr)) #else +#ifndef ARCH_ZERO_PAGE_SIZE +#define ARCH_ZERO_PAGE_SIZE PAGE_SIZE +#endif + static inline int is_zero_pfn(unsigned long pfn) { extern unsigned long zero_page_pfn; - return pfn == zero_page_pfn; + return pfn - zero_page_pfn < (ARCH_ZERO_PAGE_SIZE >> PAGE_SHIFT); } static inline unsigned long zero_pfn(unsigned long addr) @@ -1993,7 +2087,7 @@ static inline unsigned long zero_pfn(unsigned long addr) return zero_page_pfn; } -extern uint8_t empty_zero_page[PAGE_SIZE]; +extern uint8_t empty_zero_page[ARCH_ZERO_PAGE_SIZE]; extern struct page *__zero_page; static inline struct page *_zero_page(unsigned long addr) @@ -2291,6 +2385,19 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level) #define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) #endif +#ifndef p4d_offset_lockless_mm +#define p4d_offset_lockless_mm(mm, pgdp, pgd, address) \ + p4d_offset_lockless(pgdp, pgd, address) +#endif +#ifndef pud_offset_lockless_mm +#define pud_offset_lockless_mm(mm, p4dp, p4d, address) \ + pud_offset_lockless(p4dp, p4d, address) +#endif +#ifndef pmd_offset_lockless_mm +#define pmd_offset_lockless_mm(mm, pudp, pud, address) \ + pmd_offset_lockless(pudp, pud, address) +#endif + /* * pXd_leaf() is the API to check whether a pgtable entry is a huge page * mapping. It should work globally across all archs, without any @@ -2344,6 +2451,22 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level) #define __pte_leaf_size(x,y) pte_leaf_size(y) #endif +#ifndef pgd_leaf_size_mm +#define pgd_leaf_size_mm(mm, pgd) pgd_leaf_size(pgd) +#endif +#ifndef p4d_leaf_size_mm +#define p4d_leaf_size_mm(mm, p4d) p4d_leaf_size(p4d) +#endif +#ifndef pud_leaf_size_mm +#define pud_leaf_size_mm(mm, pud) pud_leaf_size(pud) +#endif +#ifndef pmd_leaf_size_mm +#define pmd_leaf_size_mm(mm, pmd) pmd_leaf_size(pmd) +#endif +#ifndef __pte_leaf_size_mm +#define __pte_leaf_size_mm(mm, pmd, pte) __pte_leaf_size(pmd, pte) +#endif + /* * We always define pmd_pfn for all archs as it's used in lots of generic * code. Now it happens too for pud_pfn (and can happen for larger diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8dc0871e5f0011..2bc2c986eb76e8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -40,6 +40,10 @@ struct anon_vma { * anon_vma if they are the last user on release */ atomic_t refcount; +#ifdef CONFIG_MM_SUBPAGE + /* Immutable base-page requirement shared by this anonymous lineage. */ + unsigned int min_folio_order; +#endif /* * Count of child anon_vmas. Equals to the count of all anon_vmas that @@ -398,29 +402,40 @@ static __always_inline void __folio_rmap_sanity_checks(const struct folio *folio } /* - * rmap interfaces called when adding or removing pte of page + * Single-PTE interfaces cover the complete native-page span of a user leaf. + * Multi-page interfaces below retain explicit native physical-page counts. */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *, unsigned long address, rmap_t flags); -#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ - folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +static inline void folio_add_anon_rmap_pte(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ + folio_add_anon_rmap_ptes(folio, page, mm_pte_native_pages(vma->vm_mm), + vma, address, flags); +} void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); -#define folio_add_file_rmap_pte(folio, page, vma) \ - folio_add_file_rmap_ptes(folio, page, 1, vma) +static inline void folio_add_file_rmap_pte(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ + folio_add_file_rmap_ptes(folio, page, mm_pte_native_pages(vma->vm_mm), vma); +} void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void folio_add_file_rmap_pud(struct folio *, struct page *, struct vm_area_struct *); void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); -#define folio_remove_rmap_pte(folio, page, vma) \ - folio_remove_rmap_ptes(folio, page, 1, vma) +static inline void folio_remove_rmap_pte(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ + folio_remove_rmap_ptes(folio, page, mm_pte_native_pages(vma->vm_mm), vma); +} void folio_remove_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void folio_remove_rmap_pud(struct folio *, struct page *, @@ -541,7 +556,8 @@ static inline void folio_dup_file_rmap_ptes(struct folio *folio, static __always_inline void folio_dup_file_rmap_pte(struct folio *folio, struct page *page, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, 1, dst_vma, PGTABLE_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, mm_pte_native_pages(dst_vma->vm_mm), + dst_vma, PGTABLE_LEVEL_PTE); } /** @@ -665,8 +681,8 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, struct page *page, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) { - return __folio_try_dup_anon_rmap(folio, page, 1, dst_vma, src_vma, - PGTABLE_LEVEL_PTE); + return __folio_try_dup_anon_rmap(folio, page, mm_pte_native_pages(src_vma->vm_mm), + dst_vma, src_vma, PGTABLE_LEVEL_PTE); } /** @@ -708,13 +724,16 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, struct page *page, int nr_pages, enum pgtable_level level) { + int i, exclusive_pages = level == PGTABLE_LEVEL_PTE ? nr_pages : 1; + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); __folio_rmap_sanity_checks(folio, page, nr_pages, level); /* device private folios cannot get pinned via GUP. */ if (unlikely(folio_is_device_private(folio))) { - ClearPageAnonExclusive(page); + for (i = 0; i < exclusive_pages; i++) + ClearPageAnonExclusive(page + i); return 0; } @@ -766,7 +785,8 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; - ClearPageAnonExclusive(page); + for (i = 0; i < exclusive_pages; i++) + ClearPageAnonExclusive(page + i); /* * This is conceptually a smp_wmb() paired with the smp_rmb() in @@ -782,7 +802,8 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, * mapped by a PTE possibly shared to prepare * for KSM or temporary unmapping * @folio: The folio to share a mapping of - * @page: The mapped exclusive page + * @page: The first native page covered by the exclusive user PTE + * @vma: The VMA defining the base user PTE granule * * The caller needs to hold the page table lock and has to have the page table * entries cleared/invalidated. @@ -799,9 +820,10 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, * -EBUSY otherwise. */ static inline int folio_try_share_anon_rmap_pte(struct folio *folio, - struct page *page) + struct page *page, struct vm_area_struct *vma) { - return __folio_try_share_anon_rmap(folio, page, 1, PGTABLE_LEVEL_PTE); + return __folio_try_share_anon_rmap(folio, page, mm_pte_native_pages(vma->vm_mm), + PGTABLE_LEVEL_PTE); } /** @@ -855,12 +877,23 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, #define PVMW_SYNC (1 << 0) /* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) +/* Match one present subpage PTE at address, including its physical offset. */ +#define PVMW_SUBPAGE (1 << 2) /* Result flags */ /* The page is mapped across page table boundary */ #define PVMW_PGTABLE_CROSSED (1 << 16) +/* One reverse-map callback, optionally restricted to one physical user leaf. */ +struct rmap_walk_range { + unsigned long address; +#ifdef CONFIG_MM_SUBPAGE + unsigned int subpage_offset; + bool subpage; +#endif +}; + struct page_vma_mapped_walk { unsigned long pfn; unsigned long nr_pages; @@ -871,6 +904,10 @@ struct page_vma_mapped_walk { pte_t *pte; spinlock_t *ptl; unsigned int flags; +#ifdef CONFIG_MM_SUBPAGE + /* PVMW_SUBPAGE: byte offset within pfn; nr_pages must be one. */ + unsigned int subpage_offset; +#endif }; #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \ @@ -883,6 +920,32 @@ struct page_vma_mapped_walk { .flags = _flags, \ } +static inline struct page_vma_mapped_walk folio_rmap_walk_init( + struct folio *folio, struct vm_area_struct *vma, + struct rmap_walk_range range, unsigned int flags) +{ + struct page_vma_mapped_walk walk = { + .pfn = folio_pfn(folio), + .nr_pages = folio_nr_pages(folio), + .pgoff = folio_pgoff(folio), + .vma = vma, + .address = range.address, + .flags = flags, + }; + +#ifdef CONFIG_MM_SUBPAGE + if (range.subpage) { + walk.flags |= PVMW_SUBPAGE; + walk.subpage_offset = range.subpage_offset; + } +#endif + return walk; +} + +#define DEFINE_FOLIO_RMAP_WALK(name, _folio, _vma, _range, _flags) \ + struct page_vma_mapped_walk name = \ + folio_rmap_walk_init(_folio, _vma, _range, _flags) + static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) { /* HugeTLB pte is set to the relevant page table entry without pte_mapped. */ @@ -943,7 +1006,7 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, * arg: passed to rmap_one() and invalid_vma() * try_lock: bail out if the rmap lock is contended * contended: indicate the rmap traversal bailed out due to lock contention - * rmap_one: executed on each vma where page is mapped + * rmap_one: executed for each mapped VMA range (possibly one user leaf) * done: for checking traversing termination condition * anon_lock: for getting anon_lock by optimized way rather than default * invalid_vma: for skipping uninterested vma @@ -957,7 +1020,7 @@ struct rmap_walk_control { * Otherwise, return true. */ bool (*rmap_one)(struct folio *folio, struct vm_area_struct *vma, - unsigned long addr, void *arg); + struct rmap_walk_range range, void *arg); int (*done)(struct folio *folio); struct anon_vma *(*anon_lock)(const struct folio *folio, struct rmap_walk_control *rwc); @@ -966,6 +1029,15 @@ struct rmap_walk_control { void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc); void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc); +#ifdef CONFIG_MM_SUBPAGE +unsigned int folio_anon_min_order(struct folio *folio); +#else +static inline unsigned int folio_anon_min_order(struct folio *folio) +{ + return 0; +} +#endif + struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, struct rmap_walk_control *rwc); diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 95d0040df58413..38d573026d7fb8 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -14,6 +14,10 @@ * Routines for handling mm_structs */ extern struct mm_struct *mm_alloc(void); +extern struct mm_struct *mm_alloc_exec(void); +extern struct mm_struct *mm_alloc_exec_page_shift(unsigned int page_shift); +/* Return zero for native, an alternative shift, or a negative errno. */ +int arch_exec_page_size_shift(unsigned long size); /** * mmgrab() - Pin a &struct mm_struct. diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h index 584ae88b435e9e..7af2aeb81e41d9 100644 --- a/include/linux/sched/signal.h +++ b/include/linux/sched/signal.h @@ -194,7 +194,7 @@ struct signal_struct { unsigned long nvcsw, nivcsw, cnvcsw, cnivcsw; unsigned long min_flt, maj_flt, cmin_flt, cmaj_flt; unsigned long inblock, oublock, cinblock, coublock; - unsigned long maxrss, cmaxrss; + unsigned long maxrss, cmaxrss; /* kB, independent of each mm granule */ struct task_io_accounting ioac; /* diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 69b0177da15648..bf3a0129547001 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -32,6 +32,8 @@ struct swap_iocb; * isn't directly visible to userspace. */ #define SHMEM_F_MAPPING_FROZEN BIT(2) +/* This inode has used sub-native shared-page validity. */ +#define SHMEM_F_UFFD_SUBPAGE BIT(3) struct shmem_inode_info { spinlock_t lock; @@ -52,6 +54,10 @@ struct shmem_inode_info { pgoff_t fallocend; /* highest fallocate endindex */ unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */ atomic_t stop_eviction; /* hold when working on inode */ +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + /* Swapped missing 4K units, keyed by native cache index. */ + struct xarray uffd_missing; +#endif #ifdef CONFIG_TMPFS_QUOTA struct dquot __rcu *i_dquot[MAXQUOTAS]; #endif @@ -113,13 +119,56 @@ extern unsigned long shmem_get_unmapped_area(struct file *, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); extern int shmem_lock(struct file *file, int lock, struct ucounts *ucounts); #ifdef CONFIG_SHMEM +struct folio *shmem_alloc_collapse_folio(struct mm_struct *mm, struct inode *inode, + pgoff_t index, unsigned int order); bool shmem_mapping(const struct address_space *mapping); #else +static inline struct folio *shmem_alloc_collapse_folio(struct mm_struct *mm, + struct inode *inode, pgoff_t index, unsigned int order) +{ + return ERR_PTR(-EOPNOTSUPP); +} static inline bool shmem_mapping(const struct address_space *mapping) { return false; } #endif /* CONFIG_SHMEM */ +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) +/* Return a locked cache folio and caller reference; new backing is all missing. */ +struct folio *shmem_uffd_prepare_folio(struct vm_area_struct *vma, + unsigned long address, bool *new); +void shmem_uffd_abort_folio(struct folio *folio, struct vm_area_struct *vma, bool new); +bool shmem_uffd_has_missing_native_page(struct folio *folio); +bool shmem_uffd_range_missing(struct folio *folio, pgoff_t index, + unsigned int offset, unsigned long length); +/* Resident masks require no allocation; inode metadata persists only in swap. */ +void shmem_uffd_copy_page(struct page *dst, struct page *src); +void shmem_uffd_migrate(struct folio *dst, struct folio *src); +/* Locked, uptodate cache folio. */ +unsigned long shmem_uffd_missing_mask(struct folio *folio, pgoff_t index); +int shmem_uffd_set_missing(struct folio *folio, pgoff_t index, + unsigned long missing); +/* Clear populated units; no allocation. Caller holds the folio lock. */ +void shmem_uffd_mark_present(struct folio *folio, pgoff_t index, + unsigned long present); +/* Mark exactly the byte range exposed by a mapping; folio lock held. */ +void shmem_uffd_mark_mapped(struct folio *folio, unsigned long offset, + unsigned long length); +/* i_pages lock held; deletion, not replacement by swap or migration. */ +void shmem_uffd_forget(struct address_space *mapping, pgoff_t start, + unsigned long nr); +#else +static inline void shmem_uffd_copy_page(struct page *dst, struct page *src) { } +static inline void shmem_uffd_migrate(struct folio *dst, struct folio *src) { } +static inline bool shmem_uffd_has_missing_native_page(struct folio *folio) { return false; } +static inline bool shmem_uffd_range_missing(struct folio *folio, pgoff_t index, + unsigned int offset, unsigned long length) { return false; } +static inline void shmem_uffd_mark_mapped(struct folio *folio, + unsigned long offset, unsigned long length) { } +static inline void shmem_uffd_forget(struct address_space *mapping, + pgoff_t start, unsigned long nr) { } +#endif + void shmem_unlock_mapping(struct address_space *mapping); struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); @@ -160,6 +209,8 @@ static inline void shmem_uncharge(struct inode *inode, long pages) { } #endif +unsigned long shmem_partial_swap_usage_bytes(struct address_space *mapping, + u64 start, u64 end); extern unsigned long shmem_partial_swap_usage(struct address_space *mapping, pgoff_t start, pgoff_t end); diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index 937823856de59d..30406f8cee1167 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -41,6 +41,8 @@ struct sk_msg_sg { }; /* UAPI in filter.c depends on struct sk_msg_sg being first element. */ +struct mm_subpage_refs; + struct sk_msg { struct sk_msg_sg sg; void *data; @@ -49,6 +51,9 @@ struct sk_msg { u32 cork_bytes; u32 flags; struct sk_buff *skb; +#ifdef CONFIG_MM_SUBPAGE + struct mm_subpage_refs *subpage_refs; +#endif struct sock *sk_redir; struct sock *sk; struct list_head list; @@ -124,6 +129,33 @@ struct sk_psock { struct rcu_work rwork; }; +#ifdef CONFIG_MM_SUBPAGE +int sk_msg_subpages_add(struct sk_msg *msg, struct page *page, u32 offset, u32 len); +void sk_msg_subpages_get(struct sk_msg *msg); +void sk_msg_subpages_share(struct sk_msg *dst, const struct sk_msg *src); +void sk_msg_subpages_release(struct sk_msg *msg); +static inline void sk_msg_subpages_transfer(struct sk_msg *dst, struct sk_msg *src) +{ + dst->subpage_refs = src->subpage_refs; + src->subpage_refs = NULL; +} +#else +static inline int sk_msg_subpages_add(struct sk_msg *msg, struct page *page, u32 offset, u32 len) +{ + return 0; +} +static inline void sk_msg_subpages_get(struct sk_msg *msg) { } +static inline void sk_msg_subpages_share(struct sk_msg *dst, const struct sk_msg *src) { } +static inline void sk_msg_subpages_release(struct sk_msg *msg) { } +static inline void sk_msg_subpages_transfer(struct sk_msg *dst, struct sk_msg *src) { } +#endif + +static inline void sk_msg_subpages_drop_empty(struct sk_msg *msg) +{ + if (!msg->sg.size) + sk_msg_subpages_release(msg); +} + int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len, int elem_first_coalesce); int sk_msg_clone(struct sock *sk, struct sk_msg *dst, struct sk_msg *src, @@ -259,10 +291,14 @@ static inline void sk_msg_compute_data_pointers(struct sk_msg *msg) } } -static inline void sk_msg_page_add(struct sk_msg *msg, struct page *page, +static inline int sk_msg_page_add(struct sk_msg *msg, struct page *page, u32 len, u32 offset) { struct scatterlist *sge; + int ret = sk_msg_subpages_add(msg, page, offset, len); + + if (ret) + return ret; get_page(page); sge = sk_msg_elem(msg, msg->sg.end); @@ -272,6 +308,7 @@ static inline void sk_msg_page_add(struct sk_msg *msg, struct page *page, __set_bit(msg->sg.end, msg->sg.copy); msg->sg.size += len; sk_msg_iter_next(msg, end); + return 0; } static inline void sk_msg_sg_copy(struct sk_msg *msg, u32 i, bool copy_state) @@ -420,6 +457,7 @@ static inline bool sk_psock_queue_empty(const struct sk_psock *psock) static inline void kfree_sk_msg(struct sk_msg *msg) { + sk_msg_subpages_release(msg); if (msg->skb) consume_skb(msg->skb); kfree(msg); diff --git a/include/linux/soc/apple/mailbox.h b/include/linux/soc/apple/mailbox.h index e5697975a50b50..8813438326ffcf 100644 --- a/include/linux/soc/apple/mailbox.h +++ b/include/linux/soc/apple/mailbox.h @@ -93,6 +93,12 @@ struct apple_mbox *apple_mbox_get_byname(struct device *dev, const char *name); int apple_mbox_start(struct apple_mbox *mbox); void apple_mbox_stop(struct apple_mbox *mbox); int apple_mbox_poll(struct apple_mbox *mbox); +/* + * Leave received messages in the hardware FIFO, without an interrupt, until + * apple_mbox_release_rx(). For consumers that must not wake the system. + */ +void apple_mbox_hold_rx(struct apple_mbox *mbox); +void apple_mbox_release_rx(struct apple_mbox *mbox); int apple_mbox_send(struct apple_mbox *mbox, struct apple_mbox_msg msg, bool atomic); diff --git a/include/linux/soc/apple/pmp-report-neo.h b/include/linux/soc/apple/pmp-report-neo.h deleted file mode 100644 index 1d0db98281195c..00000000000000 --- a/include/linux/soc/apple/pmp-report-neo.h +++ /dev/null @@ -1,26 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ -/* - * Apple SoC PMP report region - * Copyright (C) The Asahi Linux Contributors - */ - -#ifndef _LINUX_SOC_APPLE_PMP_REPORT_NEO_H_ -#define _LINUX_SOC_APPLE_PMP_REPORT_NEO_H_ - -#include -#include - -struct device_node; - -#if IS_ENABLED(CONFIG_APPLE_PMP_REPORT_NEO) -int apple_neo_pmp_report_fast_die_effort(const struct device_node *np, - unsigned int lane, u8 *effort); -#else -static inline int apple_neo_pmp_report_fast_die_effort(const struct device_node *np, - unsigned int lane, u8 *effort) -{ - return -ENODEV; -} -#endif - -#endif /* _LINUX_SOC_APPLE_PMP_REPORT_NEO_H_ */ diff --git a/include/linux/soc/apple/pmp-report.h b/include/linux/soc/apple/pmp-report.h index d71a0d04d56fdd..62a6c05a52996a 100644 --- a/include/linux/soc/apple/pmp-report.h +++ b/include/linux/soc/apple/pmp-report.h @@ -9,6 +9,7 @@ #define _LINUX_SOC_APPLE_PMP_REPORT_H_ #include +#include struct device_node; struct device; @@ -16,12 +17,20 @@ struct device; #if IS_ENABLED(CONFIG_APPLE_PMP_REPORT) int apple_pmp_report_wait_ready(struct device_node *entry, unsigned long timeout); int apple_pmp_report_wait_supplier_ready(struct device *supplier, unsigned int timeout_ms); +int apple_pmp_report_fast_die_effort(const struct device_node *np, + unsigned int lane, u8 *effort); #else static inline int apple_pmp_report_wait_ready(struct device_node *entry, unsigned long timeout) { return -ENODEV; } + +static inline int apple_pmp_report_fast_die_effort(const struct device_node *np, + unsigned int lane, u8 *effort) +{ + return -ENODEV; +} #endif #endif /* _LINUX_SOC_APPLE_PMP_REPORT_H_ */ diff --git a/include/linux/soc/apple/rtkit.h b/include/linux/soc/apple/rtkit.h index c11f1585f3a4e4..11d13a4a9cd461 100644 --- a/include/linux/soc/apple/rtkit.h +++ b/include/linux/soc/apple/rtkit.h @@ -240,6 +240,13 @@ int apple_rtkit_send_message(struct apple_rtkit *rtk, u8 ep, u64 message, int apple_rtkit_poll(struct apple_rtkit *rtk); void apple_rtkit_flush_rx(struct apple_rtkit *rtk); +/* + * Stop taking interrupts for messages from the co-processor, which then + * stay queued in the mailbox until apple_rtkit_resume_rx(). + */ +void apple_rtkit_hold_rx(struct apple_rtkit *rtk); +void apple_rtkit_resume_rx(struct apple_rtkit *rtk); + /* * Checks if an endpoint with a given index exists * diff --git a/include/linux/swap.h b/include/linux/swap.h index 7a09df6977a5f3..4e9136d3ae3f68 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -228,7 +228,7 @@ enum { */ #define SWAP_ENTRY_INVALID 0 -#ifdef CONFIG_THP_SWAP +#if defined(CONFIG_THP_SWAP) || defined(CONFIG_MM_SUBPAGE) #define SWAP_NR_ORDERS (PMD_ORDER + 1) #else #define SWAP_NR_ORDERS 1 @@ -443,6 +443,7 @@ extern int swp_swapcount(swp_entry_t entry); struct backing_dev_info; extern struct swap_info_struct *get_swap_device(swp_entry_t entry); sector_t swap_folio_sector(struct folio *folio); +sector_t swap_extent_sector(swp_entry_t entry, unsigned long *nr_pages); /* * If there is an existing swap slot reference (swap entry) and the caller @@ -455,6 +456,13 @@ sector_t swap_folio_sector(struct folio *folio); * a swap count > 1. See comments of folio_*_swap helpers for more info. */ int swap_dup_entry_direct(swp_entry_t entry); +int swap_dup_entries_direct(swp_entry_t entry, unsigned int nr); +#ifdef CONFIG_MM_SUBPAGE +/* Offset is bytes within one native swap entry; caller owns a PTE/cache ref. */ +int swap_subpage_dup(swp_entry_t entry, unsigned int offset); +void swap_subpage_put(swp_entry_t entry, unsigned int offset, bool reclaim_cache); +unsigned int swap_subpage_count(swp_entry_t entry, unsigned int offset); +#endif void swap_put_entries_direct(swp_entry_t entry, int nr); /* @@ -498,11 +506,27 @@ static inline void free_swap_cache(struct folio *folio) { } +#ifdef CONFIG_MM_SUBPAGE +static inline int swap_subpage_dup(swp_entry_t entry, unsigned int offset) +{ + return -EINVAL; +} +static inline void swap_subpage_put(swp_entry_t entry, unsigned int offset, bool reclaim_cache) { } +static inline unsigned int swap_subpage_count(swp_entry_t entry, unsigned int offset) +{ + return 0; +} +#endif static inline int swap_dup_entry_direct(swp_entry_t ent) { return 0; } +static inline int swap_dup_entries_direct(swp_entry_t ent, unsigned int nr) +{ + return 0; +} + static inline void swap_put_entries_direct(swp_entry_t ent, int nr) { } diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index 4fb7291f54b62b..29beb44f370fe6 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -1261,6 +1261,11 @@ static inline int ksys_fadvise64_64(int fd, loff_t offset, loff_t len, unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, unsigned long fd, unsigned long pgoff); +#ifdef CONFIG_MM_SUBPAGE +unsigned long ksys_mmap_user(unsigned long addr, unsigned long len, + unsigned long prot, unsigned long flags, unsigned long fd, unsigned long offset); +#endif + ssize_t ksys_readahead(int fd, loff_t offset, size_t count); int ksys_ipc(unsigned int call, int first, unsigned long second, unsigned long third, void __user * ptr, long fifth); diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index f548fea2adec8f..237b530ed600fd 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -192,6 +192,7 @@ struct uprobes_state { }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr, + unsigned int page_offset, uprobe_opcode_t *insn, int nbytes, void *data); extern void __init uprobes_init(void); diff --git a/include/linux/usb/typec_mux.h b/include/linux/usb/typec_mux.h index b99701e512527a..0703ec219e08bd 100644 --- a/include/linux/usb/typec_mux.h +++ b/include/linux/usb/typec_mux.h @@ -86,6 +86,9 @@ struct typec_mux_state { void *data; }; +typedef void (*typec_mux_notify_fn_t)(struct typec_mux_dev *mux, + struct typec_mux_state *state); + typedef int (*typec_mux_set_fn_t)(struct typec_mux_dev *mux, struct typec_mux_state *state); @@ -94,6 +97,19 @@ struct typec_mux_desc { typec_mux_set_fn_t set; const char *name; void *drvdata; + /* + * Optional transaction phases, run by typec_mux_set() only for the + * providers that register them: every provider's prepare runs before + * any provider's set, and finish and then commit after all of them. + * A failure in any phase calls abort, in reverse order, on the + * providers whose prepare stage was reached. + */ + /* Quiesce consumers before any provider changes the shared route. */ + typec_mux_set_fn_t prepare; + /* Fallible finalization retains ownership; commit cannot fail. */ + typec_mux_set_fn_t finish; + typec_mux_notify_fn_t commit; + typec_mux_notify_fn_t abort; }; #if IS_ENABLED(CONFIG_TYPEC) diff --git a/include/net/tls.h b/include/net/tls.h index ebd2550280ae24..79d358a5fe8874 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -34,6 +34,7 @@ #ifndef _TLS_OFFLOAD_H #define _TLS_OFFLOAD_H +#include #include #include #include @@ -139,6 +140,9 @@ struct tls_sw_context_rx { struct tls_strparser strp; atomic_t decrypt_pending; +#ifdef CONFIG_MM_SUBPAGE + struct llist_head subpage_cleanup; +#endif struct sk_buff_head async_hold; struct wait_queue_head wq; }; @@ -148,6 +152,9 @@ struct tls_record_info { u32 end_seq; int len; int num_frags; +#ifdef CONFIG_MM_SUBPAGE + struct mm_subpage_refs *subpage_refs; +#endif skb_frag_t frags[MAX_SKB_FRAGS]; }; diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index cd7920c81f85ba..fefbf65d61441b 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -449,7 +449,7 @@ TRACE_EVENT(rss_stat, __entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD); __entry->member = member; __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member]) - << PAGE_SHIFT); + << mm_page_shift(mm)); ), TP_printk("mm_id=%u curr=%d type=%s size=%ldB", diff --git a/include/trace/events/mmap.h b/include/trace/events/mmap.h index ee2843a5daef10..46df97560969d2 100644 --- a/include/trace/events/mmap.h +++ b/include/trace/events/mmap.h @@ -26,7 +26,7 @@ TRACE_EVENT(vm_unmapped_area, TP_fast_assign( __entry->addr = addr; - __entry->total_vm = current->mm->total_vm; + __entry->total_vm = mm_pages_to_native(current->mm, current->mm->total_vm); __entry->flags = info->flags; __entry->length = info->length; __entry->low_limit = info->low_limit; diff --git a/include/trace/events/oom.h b/include/trace/events/oom.h index 9f0a5d1482c448..d5321b6f01ffa3 100644 --- a/include/trace/events/oom.h +++ b/include/trace/events/oom.h @@ -7,7 +7,6 @@ #include #include -#define PG_COUNT_TO_KB(x) ((x) << (PAGE_SHIFT - 10)) TRACE_EVENT(oom_score_adj_update, @@ -93,10 +92,10 @@ TRACE_EVENT(mark_victim, TP_fast_assign( __entry->pid = task->pid; __assign_str(comm); - __entry->total_vm = PG_COUNT_TO_KB(task->mm->total_vm); - __entry->anon_rss = PG_COUNT_TO_KB(get_mm_counter(task->mm, MM_ANONPAGES)); - __entry->file_rss = PG_COUNT_TO_KB(get_mm_counter(task->mm, MM_FILEPAGES)); - __entry->shmem_rss = PG_COUNT_TO_KB(get_mm_counter(task->mm, MM_SHMEMPAGES)); + __entry->total_vm = mm_pages_to_kb(task->mm, task->mm->total_vm); + __entry->anon_rss = mm_pages_to_kb(task->mm, get_mm_counter(task->mm, MM_ANONPAGES)); + __entry->file_rss = mm_pages_to_kb(task->mm, get_mm_counter(task->mm, MM_FILEPAGES)); + __entry->shmem_rss = mm_pages_to_kb(task->mm, get_mm_counter(task->mm, MM_SHMEMPAGES)); __entry->uid = uid; __entry->pgtables = mm_pgtables_bytes(task->mm) >> 10; __entry->oom_score_adj = task->signal->oom_score_adj; diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index b31844bd41a6c5..524e55d74849c9 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -93,6 +93,12 @@ enum drm_asahi_ioctl_id { /** @DRM_ASAHI_SUBMIT: Submit commands to a queue. */ DRM_ASAHI_SUBMIT, + + /** + * @DRM_ASAHI_GEM_MADVISE: Mark a VM-private object purgeable or needed. + * Only the G17P runtime accepts it; others fail with EINVAL. + */ + DRM_ASAHI_GEM_MADVISE, }; #define DRM_ASAHI_MAX_CLUSTERS 64 @@ -205,7 +211,8 @@ struct drm_asahi_params_global { * of enum drm_asahi_gpu_hal_generation. * * G13-G15 on 14.x report DRM_ASAHI_GPU_HAL_LEGACY. - * The exact J613 25G83 runtime reports DRM_ASAHI_GPU_HAL_200. + * The exact J613 25G83 runtime and the G17P runtime report + * DRM_ASAHI_GPU_HAL_200. */ __u32 gpu_hal_generation; }; @@ -217,7 +224,10 @@ enum drm_asahi_gpu_hal_generation { /** @DRM_ASAHI_GPU_HAL_LEGACY: GPU predates the numbered HAL generations. */ DRM_ASAHI_GPU_HAL_LEGACY = 0, - /** @DRM_ASAHI_GPU_HAL_200: HAL200 command stream, reported by the J613 25G83 runtime. */ + /** + * @DRM_ASAHI_GPU_HAL_200: HAL200 command stream, reported by the J613 + * 25G83 runtime and by the G17P runtime. + */ DRM_ASAHI_GPU_HAL_200 = 200, /** @DRM_ASAHI_GPU_HAL_300: HAL300 command stream. Never reported by this kernel. */ @@ -262,8 +272,9 @@ enum drm_asahi_feature { /** * @DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES: GET_PARAMS group - * DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS is available. This kernel never - * sets this bit. + * DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS is available, and each queue + * reserves its own hardware execution state when it is created. Only + * the G17P runtime sets this bit. */ DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES = (1UL) << 2, @@ -279,7 +290,7 @@ enum drm_asahi_feature { DRM_ASAHI_FEATURE_SCHEDULED_QUEUES = (1UL) << 3, /* - * Bits 4, 5 and 7 are reported only by the HAL200 runtime. + * Bits 4, 5 and 7 are reported only by the J613 25G83 HAL200 runtime. * Bit 9 remains a diagnostic and is not advertised. */ @@ -289,11 +300,69 @@ enum drm_asahi_feature { DRM_ASAHI_FEATURE_FRAGMENT_DEPENDENCY = (1UL) << 6, DRM_ASAHI_FEATURE_COMPUTE_WIDE_VISIBILITY = (1UL) << 8, + + /* + * Bits 10 to 15 are reported only by the G17P runtime. + */ + + /** + * @DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS: Every hardware command of + * a submission waits for all of the submission's input syncs, including + * their errors, so independent commands need no extra dependency on the + * first command. Userspace that splits a batch across several + * submissions and relies on this must pass its input syncs to each of + * them. + */ + DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS = (1UL) << 10, + + /** + * @DRM_ASAHI_FEATURE_VM_STATUS_MIRROR: GET_PARAMS supports + * DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR, which registers a userspace + * word that always holds the VM status that + * DRM_ASAHI_PARAM_GROUP_VM_STATUS would return. The kernel stores a + * failure there before it signals any fence of the failed job, so after + * a successful wait a load of the word replaces the VM_STATUS query. + * Registration may still be refused. + */ + DRM_ASAHI_FEATURE_VM_STATUS_MIRROR = (1UL) << 11, + + /** + * @DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS: Render and compute payloads + * support exact queue-local historical prefixes in prior_vdm/prior_cdm. + * These fields supplement the header barriers. + */ + DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS = (1UL) << 12, + + /** + * @DRM_ASAHI_FEATURE_FEW_PRIMITIVES: Render commands accept + * DRM_ASAHI_RENDER_FEW_PRIMITIVES. + */ + DRM_ASAHI_FEATURE_FEW_PRIMITIVES = (1UL) << 13, + + /** + * @DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS: Render commands accept + * DRM_ASAHI_RENDER_FRAGMENT_BARRIERS. + */ + DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS = (1UL) << 14, + + /** + * @DRM_ASAHI_FEATURE_RENDER_RAY_QUERY: Render commands accept per-stage + * ray query state (struct drm_asahi_render_ray_query_stage) and queues + * accept render_scs_layout. + */ + DRM_ASAHI_FEATURE_RENDER_RAY_QUERY = (1UL) << 15, }; #define DRM_ASAHI_PARAM_GROUP_VM_STATUS 1 #define DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS 2 +/** + * define DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR - GET_PARAMS group for + * struct drm_asahi_vm_status_mirror, available with + * DRM_ASAHI_FEATURE_VM_STATUS_MIRROR + */ +#define DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR 3 + /* GET_PARAMS group VM_STATUS is an in/out query on the calling file's VM. * Set pointer to this structure and size to sizeof(structure). * vm_id selects the VM; flags and pad must be zero. error is output only. @@ -329,8 +398,8 @@ struct drm_asahi_vm_status { * @pad: Reserved, zero. * * The query is available only when DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES is - * set, which this kernel never does; otherwise it fails with EINVAL. A query - * does not allocate or reset anything. + * set, which only the G17P runtime does; otherwise it fails with EINVAL. A + * query does not allocate or reset anything. */ struct drm_asahi_queue_limits { __u32 max_queues; @@ -339,6 +408,31 @@ struct drm_asahi_queue_limits { __u32 pad; }; +/** + * struct drm_asahi_vm_status_mirror - VM status mirror registration, GET_PARAMS + * group DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR + * + * Registers a 32-bit word in a GEM object of the calling file. The kernel + * keeps its own reference to the object for the lifetime of the VM status, + * zeroes the word, and from then on keeps it equal to the VM status error + * (see &drm_asahi_vm_status.error). Each VM accepts one registration. + * EOPNOTSUPP means that userspace must keep using + * DRM_ASAHI_PARAM_GROUP_VM_STATUS. + */ +struct drm_asahi_vm_status_mirror { + /** @vm_id: VM whose status is mirrored */ + __u32 vm_id; + + /** @handle: GEM object holding the word, at most 64 KiB in size */ + __u32 handle; + + /** @offset: Byte offset of the word in the object, 4-byte aligned */ + __u32 offset; + + /** @flags: MBZ */ + __u32 flags; +}; + /** * struct drm_asahi_get_params - Arguments passed to DRM_IOCTL_ASAHI_GET_PARAMS */ @@ -348,7 +442,9 @@ struct drm_asahi_get_params { * drm_asahi_params_global, DRM_ASAHI_PARAM_GROUP_VM_STATUS when * DRM_ASAHI_FEATURE_VM_STATUS is set, or * DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS when - * DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES is set. Any other group, or a + * DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES is set, or + * DRM_ASAHI_PARAM_GROUP_VM_STATUS_MIRROR when + * DRM_ASAHI_FEATURE_VM_STATUS_MIRROR is set. Any other group, or a * group whose feature bit is clear, fails with EINVAL. */ __u32 param_group; @@ -356,7 +452,10 @@ struct drm_asahi_get_params { /** @pad: MBZ */ __u32 pad; - /** @pointer: User pointer to parameter struct (VM status is in/out). */ + /** + * @pointer: User pointer to parameter struct. The VM status groups also + * read their input from it. + */ __u64 pointer; /** @@ -471,6 +570,41 @@ struct drm_asahi_gem_mmap_offset { __u64 offset; }; +/** + * enum drm_asahi_gem_madv - Advice for DRM_IOCTL_ASAHI_GEM_MADVISE + */ +enum drm_asahi_gem_madv { + /** @DRM_ASAHI_GEM_MADV_WILLNEED: The object's contents are needed. */ + DRM_ASAHI_GEM_MADV_WILLNEED = 0, + /** + * @DRM_ASAHI_GEM_MADV_DONTNEED: The object is not referenced by any + * pending or future GPU work until the next WILLNEED. The kernel may + * discard its contents under memory pressure. + */ + DRM_ASAHI_GEM_MADV_DONTNEED = 1, +}; + +/** + * struct drm_asahi_gem_madvise - Arguments passed to + * DRM_IOCTL_ASAHI_GEM_MADVISE + * + * Only VM-private objects accept advice. A purged object keeps its handle + * and mappings but has no contents; userspace must not use it again. + */ +struct drm_asahi_gem_madvise { + /** @handle: Handle of the object. */ + __u32 handle; + + /** @madv: One of enum drm_asahi_gem_madv. */ + __u32 madv; + + /** @retained: Set by the kernel: 1 if the contents are retained, 0 if purged. */ + __u32 retained; + + /** @pad: Must be zero. */ + __u32 pad; +}; + /** * enum drm_asahi_bind_flags - Flags for GEM binding */ @@ -719,6 +853,13 @@ struct drm_asahi_queue_create { * fix a 4GiB VA carveout for USC memory and pass its base address here. */ __u64 usc_exec_base; + + /** + * @render_scs_layout: Largest ray query scratch layouts of the vertex and + * fragment stages this queue will submit, used to size its render pool. + * Requires DRM_ASAHI_FEATURE_RENDER_RAY_QUERY; zero otherwise. + */ + __u64 render_scs_layout[2]; }; /** @@ -934,7 +1075,7 @@ enum drm_asahi_render_flags { * @DRM_ASAHI_RENDER_RSRC_SPEC_HI: The appended resource-specifier high * dwords are present. A kernel that cannot use them rejects the flag * instead of silently truncating the values. This kernel accepts it on - * T6030 only. + * T6030 only. The G17P runtime also accepts it. */ DRM_ASAHI_RENDER_RSRC_SPEC_HI = (1U << 4), @@ -950,8 +1091,8 @@ enum drm_asahi_render_flags { DRM_ASAHI_RENDER_FRAGMENT_DEPENDENCY = (1U << 5), /* - * Bits 6 and 7 are supported only by the HAL200 runtime. - * The legacy runtime rejects render commands that set them. + * Bits 6 and 7 are supported only by the J613 25G83 HAL200 runtime. + * The legacy and G17P runtimes reject render commands that set them. */ /** @@ -967,6 +1108,40 @@ enum drm_asahi_render_flags { * so we match that here for efficiency. */ DRM_ASAHI_RENDER_DBIAS_IS_INT = (1U << 18), + + /** + * @DRM_ASAHI_RENDER_FEW_PRIMITIVES: The render pass is known to rasterize + * at most four primitives in total. + * + * The count includes every primitive the pass generates: topology-derived + * primitive counts of all draws multiplied by their instance counts, plus + * internal work (clears, visibility, scissor/state generated work). Passes + * with indirect draws, mesh work or any work whose primitive count is not + * known at record time must not set this flag. The flag sets bit 0 of the + * auxiliary framebuffer control word for the fragment stage; the hardware + * effect is an optimization for small known passes and its omission is + * always safe. Requires DRM_ASAHI_FEATURE_FEW_PRIMITIVES. + */ + DRM_ASAHI_RENDER_FEW_PRIMITIVES = (1U << 19), + + /** + * @DRM_ASAHI_RENDER_FRAGMENT_BARRIERS: Only the fragment stage waits for + * this command's barriers. + * + * The render dependencies resolved from @vdm_barrier and @prior_vdm are + * attached to the fragment half of the render instead of its vertex + * (tiling) half, so vertex processing may start while the renders this + * command depends on are still executing their fragment work. The + * fragment half still waits for its own vertex half and for the previous + * fragment work of its physical queue; the compute dependencies from + * @cdm_barrier and @prior_cdm keep ordering the vertex half. Set this + * only when every render dependency into the command is consumed by + * fragment-stage or attachment accesses alone: nothing the vertex stage + * reads (vertex/index/indirect buffers, vertex, tessellation or geometry + * shader resources, transform feedback, conditional-rendering predicates) + * may depend on it. Requires DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS. + */ + DRM_ASAHI_RENDER_FRAGMENT_BARRIERS = (1U << 20), }; /** @@ -1105,6 +1280,42 @@ struct drm_asahi_bg_eot { __u32 rsrc_spec; }; +/** + * struct drm_asahi_render_ray_query_stage - Ray query state of one render stage + * + * Index 0 describes the vertex (tiling) stage and index 1 the fragment stage + * of a render command. An all-zero entry disables ray queries in that stage. + * Requires DRM_ASAHI_FEATURE_RENDER_RAY_QUERY. + */ +struct drm_asahi_render_ray_query_stage { + /** @scs_layout: Per-thread ray query scratch layout, as for compute. */ + __u64 scs_layout; + + /** @intersection_aux: Auxiliary intersection table word. */ + __u32 intersection_aux; + + /** @intersection_pad: MBZ */ + __u32 intersection_pad; + + /** @intersection_pool: GPU VA of the intersection program pool. */ + __u64 intersection_pool; + + /** @intersection_pool_bytes: Size of the intersection program pool. */ + __u64 intersection_pool_bytes; + + /** @intersection_first: First intersection table entry. */ + __u32 intersection_first; + + /** @intersection_count: Number of intersection table entries. */ + __u32 intersection_count; + + /** @intersection_default: GPU VA of the default intersection program. */ + __u64 intersection_default; + + /** @intersection_default_control: Control word of the default program. */ + __u64 intersection_default_control; +}; + /** * struct drm_asahi_cmd_render - Command to submit 3D * @@ -1287,6 +1498,30 @@ struct drm_asahi_cmd_render { /** @ts_frag_end_copies: End timestamp copies, requiring the corresponding feature bit. */ struct drm_asahi_timestamp ts_frag_end_copies[8]; + + /** + * @prior_vdm: Historical render prefix, or zero for no added dependency. + * + * Requires DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS. Counts hardware render + * commands accepted on this queue since creation, starting at one. A + * nonzero value must not exceed the render count at this ioctl's entry; + * it waits for every render command through that ordinal. The effective + * dependency is the maximum of this prefix and the header's resolved + * vdm_barrier. Header NONE does not suppress this field. + */ + __u64 prior_vdm; + + /** + * @prior_cdm: Historical compute prefix; see @prior_vdm, using the + * queue's compute command count and the header's cdm_barrier instead. + */ + __u64 prior_cdm; + + /** + * @ray_query: Ray query state of the vertex and fragment stages. Older + * userspace omits it; a shorter command is read with both stages zeroed. + */ + struct drm_asahi_render_ray_query_stage ray_query[2]; }; /** @@ -1327,6 +1562,48 @@ struct drm_asahi_cmd_compute { /** @ts_end_copies: End timestamp copies, requiring the corresponding feature bit. */ struct drm_asahi_timestamp ts_end_copies[8]; + + /** + * @scs_layout: Packed G17P Gen1 shader-context scratch layout, or zero. + * Bits 0:15 contain stack depth, 16:23 levels, 24:31 cliques, + * 32:47 user bytes and 48:63 driver bytes. Non-G17P paths require zero. + */ + __u64 scs_layout; + + /** + * @prior_vdm: Historical render prefix, or zero for no added dependency. + * + * Requires DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS. Counts hardware render + * commands accepted on this queue since creation, starting at one. A + * nonzero value must not exceed the render count at this ioctl's entry; + * it waits for every render command through that ordinal. The effective + * dependency is the maximum of this prefix and the header's resolved + * vdm_barrier. Header NONE does not suppress this field. + */ + __u64 prior_vdm; + + /** + * @prior_cdm: Historical compute prefix; see @prior_vdm, using the + * queue's compute command count and the header's cdm_barrier instead. + */ + __u64 prior_cdm; + + /** @cdm_auxiliary_allocation: G17P auxiliary allocation, zero or 8..32. */ + __u32 cdm_auxiliary_allocation; + /** @reserved: MBZ. */ + __u32 reserved; + /** @intersection_pool: 16-KiB aligned G17P intersection-table address. */ + __u64 intersection_pool; + /** @intersection_pool_bytes: Mapped pool size in bytes, a multiple of 16 KiB. */ + __u64 intersection_pool_bytes; + /** @intersection_first: First 128-byte entry in @intersection_pool. */ + __u32 intersection_first; + /** @intersection_count: Number of 128-byte entries, less than 2^18. */ + __u32 intersection_count; + /** @intersection_default_program: Encoded default G17P program address. */ + __u64 intersection_default_program; + /** @intersection_default_control: Default G17P program control word. */ + __u64 intersection_default_control; }; /** @@ -1381,6 +1658,7 @@ enum { DRM_IOCTL_ASAHI_QUEUE_CREATE = DRM_IOCTL_ASAHI(WR, QUEUE_CREATE, queue_create), DRM_IOCTL_ASAHI_QUEUE_DESTROY = DRM_IOCTL_ASAHI(W, QUEUE_DESTROY, queue_destroy), DRM_IOCTL_ASAHI_SUBMIT = DRM_IOCTL_ASAHI(W, SUBMIT, submit), + DRM_IOCTL_ASAHI_GEM_MADVISE = DRM_IOCTL_ASAHI(WR, GEM_MADVISE, gem_madvise), }; #if defined(__cplusplus) diff --git a/include/uapi/drm/asahi_neo_drm.h b/include/uapi/drm/asahi_neo_drm.h deleted file mode 100644 index 8af13919eae333..00000000000000 --- a/include/uapi/drm/asahi_neo_drm.h +++ /dev/null @@ -1,1524 +0,0 @@ -/* SPDX-License-Identifier: MIT */ -/* - * Copyright (C) The Asahi Linux Contributors - * Copyright (C) 2018-2023 Collabora Ltd. - * Copyright (C) 2014-2018 Broadcom - */ -#ifndef _ASAHI_NEO_DRM_H_ -#define _ASAHI_NEO_DRM_H_ - -#include "drm.h" - -#if defined(__cplusplus) -extern "C" { -#endif - -/** - * DOC: Introduction to the Asahi UAPI - * - * This documentation describes the Asahi IOCTLs. - * - * Just a few generic rules about the data passed to the Asahi IOCTLs (cribbed - * from Panthor): - * - * - Structures must be aligned on 64-bit/8-byte. If the object is not - * naturally aligned, a padding field must be added. - * - Fields must be explicitly aligned to their natural type alignment with - * pad[0..N] fields. - * - All padding fields will be checked by the driver to make sure they are - * zeroed. - * - Flags can be added, but not removed/replaced. - * - New fields can be added to the main structures (the structures - * directly passed to the ioctl). Those fields can be added at the end of - * the structure, or replace existing padding fields. Any new field being - * added must preserve the behavior that existed before those fields were - * added when a value of zero is passed. - * - New fields can be added to indirect objects (objects pointed by the - * main structure), iff those objects are passed a size to reflect the - * size known by the userspace driver (see - * drm_asahi_neo_cmd_header::size). - * - If the kernel driver is too old to know some fields, those will be - * ignored if zero, and otherwise rejected (and so will be zero on output). - * - If userspace is too old to know some fields, those will be zeroed - * (input) before the structure is parsed by the kernel driver. - * - Each new flag/field addition must come with a driver version update so - * the userspace driver doesn't have to guess which flags are supported. - * - Structures should not contain unions, as this would defeat the - * extensibility of such structures. - * - IOCTLs can't be removed or replaced. New IOCTL IDs should be placed - * at the end of the drm_asahi_neo_ioctl_id enum. - */ - -/** - * enum drm_asahi_neo_ioctl_id - IOCTL IDs - * - * Place new ioctls at the end, don't re-order, don't replace or remove entries. - * - * These IDs are not meant to be used directly. Use the DRM_IOCTL_ASAHI_NEO_xxx - * definitions instead. - */ -enum drm_asahi_neo_ioctl_id { - /** @DRM_ASAHI_NEO_GET_PARAMS: Query device properties. */ - DRM_ASAHI_NEO_GET_PARAMS = 0, - - /** @DRM_ASAHI_NEO_GET_TIME: Query device time. */ - DRM_ASAHI_NEO_GET_TIME, - - /** @DRM_ASAHI_NEO_VM_CREATE: Create a GPU VM address space. */ - DRM_ASAHI_NEO_VM_CREATE, - - /** @DRM_ASAHI_NEO_VM_DESTROY: Destroy a VM. */ - DRM_ASAHI_NEO_VM_DESTROY, - - /** @DRM_ASAHI_NEO_VM_BIND: Bind/unbind memory to a VM. */ - DRM_ASAHI_NEO_VM_BIND, - - /** @DRM_ASAHI_NEO_GEM_CREATE: Create a buffer object. */ - DRM_ASAHI_NEO_GEM_CREATE, - - /** - * @DRM_ASAHI_NEO_GEM_MMAP_OFFSET: Get offset to pass to mmap() to map a - * given GEM handle. - */ - DRM_ASAHI_NEO_GEM_MMAP_OFFSET, - - /** @DRM_ASAHI_NEO_GEM_BIND_OBJECT: Bind memory as a special object */ - DRM_ASAHI_NEO_GEM_BIND_OBJECT, - - /** @DRM_ASAHI_NEO_QUEUE_CREATE: Create a scheduling queue. */ - DRM_ASAHI_NEO_QUEUE_CREATE, - - /** @DRM_ASAHI_NEO_QUEUE_DESTROY: Destroy a scheduling queue. */ - DRM_ASAHI_NEO_QUEUE_DESTROY, - - /** @DRM_ASAHI_NEO_SUBMIT: Submit commands to a queue. */ - DRM_ASAHI_NEO_SUBMIT, - - /** @DRM_ASAHI_NEO_GEM_MADVISE: Mark a VM-private object purgeable or needed. */ - DRM_ASAHI_NEO_GEM_MADVISE, -}; - -#define DRM_ASAHI_NEO_MAX_CLUSTERS 64 - -/** - * struct drm_asahi_neo_params_global - Global parameters. - * - * This struct may be queried by drm_asahi_neo_get_params. - */ -struct drm_asahi_neo_params_global { - /** @features: Feature bits from drm_asahi_neo_feature */ - __u64 features; - - /** @gpu_generation: GPU generation, e.g. 13 for G13G */ - __u32 gpu_generation; - - /** @gpu_variant: GPU variant as a character, e.g. 'C' for G13C */ - __u32 gpu_variant; - - /** - * @gpu_revision: GPU revision in BCD, e.g. 0x00 for 'A0' or - * 0x21 for 'C1' - */ - __u32 gpu_revision; - - /** @chip_id: Chip ID in BCD, e.g. 0x8103 for T8103 */ - __u32 chip_id; - - /** @num_dies: Number of dies in the SoC */ - __u32 num_dies; - - /** @num_clusters_total: Number of GPU clusters (across all dies) */ - __u32 num_clusters_total; - - /** - * @num_cores_per_cluster: Number of logical cores per cluster - * (including inactive/nonexistent) - */ - __u32 num_cores_per_cluster; - - /** @max_frequency_khz: Maximum GPU core clock frequency */ - __u32 max_frequency_khz; - - /** @core_masks: Bitmask of present/enabled cores per cluster */ - __u64 core_masks[DRM_ASAHI_NEO_MAX_CLUSTERS]; - - /** - * @vm_start: VM range start VMA. Together with @vm_end, this defines - * the window of valid GPU VAs. Userspace is expected to subdivide VAs - * out of this window. - * - * This window contains all virtual addresses that userspace needs to - * know about. There may be kernel-internal GPU VAs outside this range, - * but that detail is not relevant here. - */ - __u64 vm_start; - - /** @vm_end: VM range end VMA */ - __u64 vm_end; - - /** - * @vm_kernel_min_size: Minimum kernel VMA window size. - * - * When creating a VM, userspace is required to carve out a section of - * virtual addresses (within the range given by @vm_start and - * @vm_end). The kernel will allocate various internal structures - * within the specified VA range. - * - * Allowing userspace to choose the VA range for the kernel, rather than - * the kernel reserving VAs and requiring userspace to cope, can assist - * in implementing SVM. - */ - __u64 vm_kernel_min_size; - - /** - * @max_commands_per_submission: Maximum number of supported commands - * per submission. This mirrors firmware limits. Userspace must split up - * larger command buffers, which may require inserting additional - * synchronization. - */ - __u32 max_commands_per_submission; - - /** - * @max_attachments: Maximum number of drm_asahi_neo_attachment's per - * command - */ - __u32 max_attachments; - - /** - * @command_timestamp_frequency_hz: Timebase frequency for timestamps - * written during command execution, specified via drm_asahi_neo_timestamp - * structures. As this rate is controlled by the firmware, it is a - * queryable parameter. - * - * Userspace must divide by this frequency to convert timestamps to - * seconds, rather than hardcoding a particular firmware's rate. - */ - __u64 command_timestamp_frequency_hz; - - /** - * @usc_generation: Instruction set generation of the unified shader - * cores: 2 for G13 and G14, 3 for G15 and newer. - */ - __u32 usc_generation; - - /** - * @gpu_hal_generation: Command stream generation, from - * enum drm_asahi_neo_gpu_hal_generation. - */ - __u32 gpu_hal_generation; -}; - -/** - * enum drm_asahi_neo_gpu_hal_generation - GPU command stream generation - * - * The command stream generation determines the layout of the control streams - * and descriptors that userspace builds for the GPU. - */ -enum drm_asahi_neo_gpu_hal_generation { - /** @DRM_ASAHI_NEO_GPU_HAL_LEGACY: Unnumbered generation of G13 to G15. */ - DRM_ASAHI_NEO_GPU_HAL_LEGACY = 0, - - /** @DRM_ASAHI_NEO_GPU_HAL_200: Generation 200, used by G17P. */ - DRM_ASAHI_NEO_GPU_HAL_200 = 200, - - /** @DRM_ASAHI_NEO_GPU_HAL_300: Generation 300. */ - DRM_ASAHI_NEO_GPU_HAL_300 = 300, -}; - -/** - * enum drm_asahi_neo_feature - Feature bits - * - * This covers only features that userspace cannot infer from the architecture - * version. Most features don't need to be here. - */ -enum drm_asahi_neo_feature { - /** - * @DRM_ASAHI_NEO_FEATURE_SOFT_FAULTS: GPU has "soft fault" enabled. Shader - * loads of unmapped memory will return zero. Shader stores to unmapped - * memory will be silently discarded. Note that only shader load/store - * is affected. Other hardware units are not affected, notably including - * texture sampling. - * - * Soft fault is set when initializing the GPU and cannot be runtime - * toggled. Therefore, it is exposed as a feature bit and not a - * userspace-settable flag on the VM. When soft fault is enabled, - * userspace can speculate memory accesses more aggressively. - */ - DRM_ASAHI_NEO_FEATURE_SOFT_FAULTS = (1UL) << 0, - - /** - * @DRM_ASAHI_NEO_FEATURE_VM_STATUS: GET_PARAMS supports - * DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS. - */ - DRM_ASAHI_NEO_FEATURE_VM_STATUS = (1UL) << 1, - - /** - * @DRM_ASAHI_NEO_FEATURE_INDEPENDENT_QUEUES: Each queue reserves its own - * hardware execution state when it is created, and GET_PARAMS - * supports DRM_ASAHI_NEO_PARAM_GROUP_QUEUE_LIMITS. - */ - DRM_ASAHI_NEO_FEATURE_INDEPENDENT_QUEUES = (1UL) << 2, - - /** - * @DRM_ASAHI_NEO_FEATURE_PER_COMMAND_INPUT_SYNCS: Every hardware command of - * a submission waits for all of the submission's input syncs, including - * their errors, so independent commands need no extra dependency on the - * first command. Userspace that splits a batch across several - * submissions and relies on this must pass its input syncs to each of - * them. - */ - DRM_ASAHI_NEO_FEATURE_PER_COMMAND_INPUT_SYNCS = (1UL) << 3, - - /** - * @DRM_ASAHI_NEO_FEATURE_VM_STATUS_MIRROR: GET_PARAMS supports - * DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS_MIRROR, which registers a userspace - * word that always holds the VM status that - * DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS would return. The kernel stores a - * failure there before it signals any fence of the failed job, so after - * a successful wait a load of the word replaces the VM_STATUS query. - * Registration may still be refused. - */ - DRM_ASAHI_NEO_FEATURE_VM_STATUS_MIRROR = (1UL) << 4, - - /** - * @DRM_ASAHI_NEO_FEATURE_EXACT_PRIOR_BARRIERS: Render and compute payloads - * support exact queue-local historical prefixes in prior_vdm/prior_cdm. - * These fields supplement the header barriers. - */ - DRM_ASAHI_NEO_FEATURE_EXACT_PRIOR_BARRIERS = (1UL) << 5, - - /** - * @DRM_ASAHI_NEO_FEATURE_FEW_PRIMITIVES: Render commands accept - * DRM_ASAHI_NEO_RENDER_FEW_PRIMITIVES. - */ - DRM_ASAHI_NEO_FEATURE_FEW_PRIMITIVES = (1UL) << 6, - - /** - * @DRM_ASAHI_NEO_FEATURE_FRAGMENT_BARRIERS: Render commands accept - * DRM_ASAHI_NEO_RENDER_FRAGMENT_BARRIERS. - */ - DRM_ASAHI_NEO_FEATURE_FRAGMENT_BARRIERS = (1UL) << 7, -}; - -/** - * define DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS - GET_PARAMS group for - * struct drm_asahi_neo_vm_status - */ -#define DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS 1 - -/** - * define DRM_ASAHI_NEO_PARAM_GROUP_QUEUE_LIMITS - GET_PARAMS group for - * struct drm_asahi_neo_queue_limits - */ -#define DRM_ASAHI_NEO_PARAM_GROUP_QUEUE_LIMITS 2 - -/** - * define DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS_MIRROR - GET_PARAMS group for - * struct drm_asahi_neo_vm_status_mirror - */ -#define DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS_MIRROR 3 - -/** - * struct drm_asahi_neo_vm_status - VM status, GET_PARAMS group - * DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS - * - * This is an in/out query on a VM of the calling file. Userspace sets - * &drm_asahi_neo_get_params.pointer to this structure and - * &drm_asahi_neo_get_params.size to its size. - * - * The kernel records the failure of an accepted job before it signals the - * job's fences, so userspace must query the status after a successful syncobj - * wait before it treats the work as successfully executed. A signalled - * syncobj alone does not indicate successful execution. An execution failure - * also marks other VMs whose work can no longer make progress. A query never - * resets or recovers the GPU. - * - * Invalid requests fail with EINVAL (non-zero @flags or @pad, wrong size), - * ENOENT (@vm_id unknown to the calling file) or EFAULT, and write nothing. - */ -struct drm_asahi_neo_vm_status { - /** @vm_id: VM to query */ - __u32 vm_id; - - /** @flags: MBZ */ - __u32 flags; - - /** - * @error: Output: 0 if no submission error was observed, otherwise the - * first error as a negative errno. The error is sticky for the lifetime - * of the VM: neither creating queues nor waiting on other fences clears - * it. - */ - __s32 error; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * struct drm_asahi_neo_queue_limits - Queue limits, GET_PARAMS group - * DRM_ASAHI_NEO_PARAM_GROUP_QUEUE_LIMITS - * - * The query neither allocates nor resets anything. - */ -struct drm_asahi_neo_queue_limits { - /** - * @max_queues: Maximum number of queues on the device, shared by all - * files and VMs. This is a hardware bound, not a promise that as many - * are currently available: QUEUE_CREATE reserves a queue's hardware - * state and fails once it runs out. - */ - __u32 max_queues; - - /** - * @max_in_flight_per_queue: Number of jobs of a queue that the - * hardware may run at once. Further jobs wait behind the preceding work - * and their dependencies. - */ - __u32 max_in_flight_per_queue; - - /** @flags: Reserved, zero */ - __u32 flags; - - /** @pad: Reserved, zero */ - __u32 pad; -}; - -/** - * struct drm_asahi_neo_vm_status_mirror - VM status mirror registration, GET_PARAMS - * group DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS_MIRROR - * - * Registers a 32-bit word in a GEM object of the calling file. The kernel - * keeps its own reference to the object for the lifetime of the VM status, - * zeroes the word, and from then on keeps it equal to the VM status error - * (see &drm_asahi_neo_vm_status.error). Each VM accepts one registration. - * EOPNOTSUPP means that userspace must keep using - * DRM_ASAHI_NEO_PARAM_GROUP_VM_STATUS. - */ -struct drm_asahi_neo_vm_status_mirror { - /** @vm_id: VM whose status is mirrored */ - __u32 vm_id; - - /** @handle: GEM object holding the word, at most 64 KiB in size */ - __u32 handle; - - /** @offset: Byte offset of the word in the object, 4-byte aligned */ - __u32 offset; - - /** @flags: MBZ */ - __u32 flags; -}; - -/** - * struct drm_asahi_neo_get_params - Arguments passed to DRM_IOCTL_ASAHI_NEO_GET_PARAMS - */ -struct drm_asahi_neo_get_params { - /** - * @param_group: Parameter group: 0 for struct drm_asahi_neo_params_global, - * otherwise one of DRM_ASAHI_NEO_PARAM_GROUP_*. - */ - __u32 param_group; - - /** @pad: MBZ */ - __u32 pad; - - /** - * @pointer: User pointer to the parameter struct. The VM status groups - * also read their input from it. - */ - __u64 pointer; - - /** - * @size: Size of the user buffer. In case of older userspace, this may - * be less than sizeof(struct drm_asahi_neo_params_global). The kernel will - * not write past the length specified here, allowing extensibility. - */ - __u64 size; -}; - -/** - * struct drm_asahi_neo_vm_create - Arguments passed to DRM_IOCTL_ASAHI_NEO_VM_CREATE - */ -struct drm_asahi_neo_vm_create { - /** - * @kernel_start: Start of the kernel-reserved address range. See - * drm_asahi_neo_params_global::vm_kernel_min_size. - * - * Both @kernel_start and @kernel_end must be within the range of - * valid VAs given by drm_asahi_neo_params_global::vm_start and - * drm_asahi_neo_params_global::vm_end. The size of the kernel range - * (@kernel_end - @kernel_start) must be at least - * drm_asahi_neo_params_global::vm_kernel_min_size. - * - * Userspace must not bind any memory on this VM into this reserved - * range, it is for kernel use only. - */ - __u64 kernel_start; - - /** - * @kernel_end: End of the kernel-reserved address range. See - * @kernel_start. - */ - __u64 kernel_end; - - /** @vm_id: Returned VM ID */ - __u32 vm_id; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * struct drm_asahi_neo_vm_destroy - Arguments passed to DRM_IOCTL_ASAHI_NEO_VM_DESTROY - */ -struct drm_asahi_neo_vm_destroy { - /** @vm_id: VM ID to be destroyed */ - __u32 vm_id; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * enum drm_asahi_neo_gem_flags - Flags for GEM creation - */ -enum drm_asahi_neo_gem_flags { - /** - * @DRM_ASAHI_NEO_GEM_WRITEBACK: BO should be CPU-mapped as writeback. - * - * Map as writeback instead of write-combine. This optimizes for CPU - * reads. - */ - DRM_ASAHI_NEO_GEM_WRITEBACK = (1L << 0), - - /** - * @DRM_ASAHI_NEO_GEM_VM_PRIVATE: BO is private to this GPU VM (no exports). - */ - DRM_ASAHI_NEO_GEM_VM_PRIVATE = (1L << 1), -}; - -/** - * struct drm_asahi_neo_gem_create - Arguments passed to DRM_IOCTL_ASAHI_NEO_GEM_CREATE - */ -struct drm_asahi_neo_gem_create { - /** @size: Size of the BO */ - __u64 size; - - /** @flags: Combination of drm_asahi_neo_gem_flags flags. */ - __u32 flags; - - /** - * @vm_id: VM ID to assign to the BO, if DRM_ASAHI_NEO_GEM_VM_PRIVATE is set - */ - __u32 vm_id; - - /** @handle: Returned GEM handle for the BO */ - __u32 handle; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * struct drm_asahi_neo_gem_mmap_offset - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_GEM_MMAP_OFFSET - */ -struct drm_asahi_neo_gem_mmap_offset { - /** @handle: Handle for the object being mapped. */ - __u32 handle; - - /** @flags: Must be zero */ - __u32 flags; - - /** @offset: The fake offset to use for subsequent mmap call */ - __u64 offset; -}; - -/** - * enum drm_asahi_neo_gem_madv - Advice for DRM_IOCTL_ASAHI_NEO_GEM_MADVISE - */ -enum drm_asahi_neo_gem_madv { - /** @DRM_ASAHI_NEO_GEM_MADV_WILLNEED: The object's contents are needed. */ - DRM_ASAHI_NEO_GEM_MADV_WILLNEED = 0, - /** - * @DRM_ASAHI_NEO_GEM_MADV_DONTNEED: The object is not referenced by any - * pending or future GPU work until the next WILLNEED. The kernel may - * discard its contents under memory pressure. - */ - DRM_ASAHI_NEO_GEM_MADV_DONTNEED = 1, -}; - -/** - * struct drm_asahi_neo_gem_madvise - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_GEM_MADVISE - * - * Only VM-private objects accept advice. A purged object keeps its handle - * and mappings but has no contents; userspace must not use it again. - */ -struct drm_asahi_neo_gem_madvise { - /** @handle: Handle of the object. */ - __u32 handle; - - /** @madv: One of enum drm_asahi_neo_gem_madv. */ - __u32 madv; - - /** @retained: Set by the kernel: 1 if the contents are retained, 0 if purged. */ - __u32 retained; - - /** @pad: Must be zero. */ - __u32 pad; -}; - -/** - * enum drm_asahi_neo_bind_flags - Flags for GEM binding - */ -enum drm_asahi_neo_bind_flags { - /** - * @DRM_ASAHI_NEO_BIND_UNBIND: Instead of binding a GEM object to the range, - * simply unbind the GPU VMA range. - */ - DRM_ASAHI_NEO_BIND_UNBIND = (1L << 0), - - /** @DRM_ASAHI_NEO_BIND_READ: Map BO with GPU read permission */ - DRM_ASAHI_NEO_BIND_READ = (1L << 1), - - /** @DRM_ASAHI_NEO_BIND_WRITE: Map BO with GPU write permission */ - DRM_ASAHI_NEO_BIND_WRITE = (1L << 2), - - /** - * @DRM_ASAHI_NEO_BIND_SINGLE_PAGE: Map a single page of the BO repeatedly - * across the VA range. - * - * This is useful to fill a VA range with scratch pages or zero pages. - * It is intended as a mechanism to accelerate sparse. - */ - DRM_ASAHI_NEO_BIND_SINGLE_PAGE = (1L << 3), -}; - -/** - * struct drm_asahi_neo_gem_bind_op - Description of a single GEM bind operation. - */ -struct drm_asahi_neo_gem_bind_op { - /** @flags: Combination of drm_asahi_neo_bind_flags flags. */ - __u32 flags; - - /** @handle: GEM object to bind (except for UNBIND) */ - __u32 handle; - - /** - * @offset: Offset into the object (except for UNBIND). - * - * For a regular bind, this is the beginning of the region of the GEM - * object to bind. - * - * For a single-page bind, this is the offset to the single page that - * will be repeatedly bound. - * - * Must be page-size aligned. - */ - __u64 offset; - - /** - * @range: Number of bytes to bind/unbind to @addr. - * - * Must be page-size aligned. - */ - __u64 range; - - /** - * @addr: Address to bind to. - * - * Must be page-size aligned. - */ - __u64 addr; -}; - -/** - * struct drm_asahi_neo_vm_bind - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_VM_BIND - */ -struct drm_asahi_neo_vm_bind { - /** @vm_id: The ID of the VM to bind to */ - __u32 vm_id; - - /** @num_binds: number of binds in this IOCTL. */ - __u32 num_binds; - - /** - * @stride: Stride in bytes between consecutive binds. This allows - * extensibility of drm_asahi_neo_gem_bind_op. - */ - __u32 stride; - - /** @pad: MBZ */ - __u32 pad; - - /** - * @userptr: User pointer to an array of @num_binds structures of type - * @drm_asahi_neo_gem_bind_op and size @stride bytes. - */ - __u64 userptr; -}; - -/** - * enum drm_asahi_neo_bind_object_op - Special object bind operation - */ -enum drm_asahi_neo_bind_object_op { - /** @DRM_ASAHI_NEO_BIND_OBJECT_OP_BIND: Bind a BO as a special GPU object */ - DRM_ASAHI_NEO_BIND_OBJECT_OP_BIND = 0, - - /** @DRM_ASAHI_NEO_BIND_OBJECT_OP_UNBIND: Unbind a special GPU object */ - DRM_ASAHI_NEO_BIND_OBJECT_OP_UNBIND = 1, -}; - -/** - * enum drm_asahi_neo_bind_object_flags - Special object bind flags - */ -enum drm_asahi_neo_bind_object_flags { - /** - * @DRM_ASAHI_NEO_BIND_OBJECT_USAGE_TIMESTAMPS: Map a BO as a timestamp - * buffer. - */ - DRM_ASAHI_NEO_BIND_OBJECT_USAGE_TIMESTAMPS = (1L << 0), -}; - -/** - * struct drm_asahi_neo_gem_bind_object - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_GEM_BIND_OBJECT - */ -struct drm_asahi_neo_gem_bind_object { - /** @op: Bind operation (enum drm_asahi_neo_bind_object_op) */ - __u32 op; - - /** @flags: Combination of drm_asahi_neo_bind_object_flags flags. */ - __u32 flags; - - /** @handle: GEM object to bind/unbind (BIND) */ - __u32 handle; - - /** @vm_id: The ID of the VM to operate on (MBZ currently) */ - __u32 vm_id; - - /** @offset: Offset into the object (BIND only) */ - __u64 offset; - - /** @range: Number of bytes to bind/unbind (BIND only) */ - __u64 range; - - /** @object_handle: Object handle (out for BIND, in for UNBIND) */ - __u32 object_handle; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * enum drm_asahi_neo_cmd_type - Command type - */ -enum drm_asahi_neo_cmd_type { - /** - * @DRM_ASAHI_NEO_CMD_RENDER: Render command, executing on the render - * subqueue. Combined vertex and fragment operation. - * - * Followed by a @drm_asahi_neo_cmd_render payload. - */ - DRM_ASAHI_NEO_CMD_RENDER = 0, - - /** - * @DRM_ASAHI_NEO_CMD_COMPUTE: Compute command on the compute subqueue. - * - * Followed by a @drm_asahi_neo_cmd_compute payload. - */ - DRM_ASAHI_NEO_CMD_COMPUTE = 1, - - /** - * @DRM_ASAHI_NEO_SET_VERTEX_ATTACHMENTS: Software command to set - * attachments for subsequent vertex shaders in the same submit. - * - * Followed by (possibly multiple) @drm_asahi_neo_attachment payloads. - */ - DRM_ASAHI_NEO_SET_VERTEX_ATTACHMENTS = 2, - - /** - * @DRM_ASAHI_NEO_SET_FRAGMENT_ATTACHMENTS: Software command to set - * attachments for subsequent fragment shaders in the same submit. - * - * Followed by (possibly multiple) @drm_asahi_neo_attachment payloads. - */ - DRM_ASAHI_NEO_SET_FRAGMENT_ATTACHMENTS = 3, - - /** - * @DRM_ASAHI_NEO_SET_COMPUTE_ATTACHMENTS: Software command to set - * attachments for subsequent compute shaders in the same submit. - * - * Followed by (possibly multiple) @drm_asahi_neo_attachment payloads. - */ - DRM_ASAHI_NEO_SET_COMPUTE_ATTACHMENTS = 4, -}; - -/** - * enum drm_asahi_neo_priority - Scheduling queue priority. - * - * These priorities are forwarded to the firmware to influence firmware - * scheduling. The exact policy is ultimately decided by firmware, but - * these enums allow userspace to communicate the intentions. - */ -enum drm_asahi_neo_priority { - /** @DRM_ASAHI_NEO_PRIORITY_LOW: Low priority queue. */ - DRM_ASAHI_NEO_PRIORITY_LOW = 0, - - /** @DRM_ASAHI_NEO_PRIORITY_MEDIUM: Medium priority queue. */ - DRM_ASAHI_NEO_PRIORITY_MEDIUM = 1, - - /** - * @DRM_ASAHI_NEO_PRIORITY_HIGH: High priority queue. - * - * Reserved for future extension. - */ - DRM_ASAHI_NEO_PRIORITY_HIGH = 2, - - /** - * @DRM_ASAHI_NEO_PRIORITY_REALTIME: Real-time priority queue. - * - * Reserved for future extension. - */ - DRM_ASAHI_NEO_PRIORITY_REALTIME = 3, -}; - -/** - * struct drm_asahi_neo_queue_create - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_QUEUE_CREATE - */ -struct drm_asahi_neo_queue_create { - /** @flags: MBZ */ - __u32 flags; - - /** @vm_id: The ID of the VM this queue is bound to */ - __u32 vm_id; - - /** @priority: One of drm_asahi_neo_priority */ - __u32 priority; - - /** @queue_id: The returned queue ID */ - __u32 queue_id; - - /** - * @usc_exec_base: GPU base address for all USC binaries (shaders) on - * this queue. USC addresses are 32-bit relative to this 64-bit base. - * - * This sets the following registers on all queue commands: - * - * USC_EXEC_BASE_TA (vertex) - * USC_EXEC_BASE_ISP (fragment) - * USC_EXEC_BASE_CP (compute) - * - * While the hardware lets us configure these independently per command, - * we do not have a use case for this. Instead, we expect userspace to - * fix a 4GiB VA carveout for USC memory and pass its base address here. - */ - __u64 usc_exec_base; -}; - -/** - * struct drm_asahi_neo_queue_destroy - Arguments passed to - * DRM_IOCTL_ASAHI_NEO_QUEUE_DESTROY - */ -struct drm_asahi_neo_queue_destroy { - /** @queue_id: The queue ID to be destroyed */ - __u32 queue_id; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * enum drm_asahi_neo_sync_type - Sync item type - */ -enum drm_asahi_neo_sync_type { - /** @DRM_ASAHI_NEO_SYNC_SYNCOBJ: Binary sync object */ - DRM_ASAHI_NEO_SYNC_SYNCOBJ = 0, - - /** @DRM_ASAHI_NEO_SYNC_TIMELINE_SYNCOBJ: Timeline sync object */ - DRM_ASAHI_NEO_SYNC_TIMELINE_SYNCOBJ = 1, -}; - -/** - * struct drm_asahi_neo_sync - Sync item - */ -struct drm_asahi_neo_sync { - /** @sync_type: One of drm_asahi_neo_sync_type */ - __u32 sync_type; - - /** @handle: The sync object handle */ - __u32 handle; - - /** @timeline_value: Timeline value for timeline sync objects */ - __u64 timeline_value; -}; - -/** - * define DRM_ASAHI_NEO_BARRIER_NONE - Command index for no barrier - * - * This special value may be passed in to drm_asahi_neo_command::vdm_barrier or - * drm_asahi_neo_command::cdm_barrier to indicate that the respective subqueue - * should not wait on any previous work. - */ -#define DRM_ASAHI_NEO_BARRIER_NONE (0xFFFFu) - -/** - * struct drm_asahi_neo_cmd_header - Top level command structure - * - * This struct is core to the command buffer definition and therefore is not - * extensible. - */ -struct drm_asahi_neo_cmd_header { - /** @cmd_type: One of drm_asahi_neo_cmd_type */ - __u16 cmd_type; - - /** - * @size: Size of this command, not including this header. - * - * For hardware commands, this enables extensibility of commands without - * requiring extra command types. Passing a command that is shorter - * than expected is explicitly allowed for backwards-compatibility. - * Truncated fields will be zeroed. - * - * For the synthetic attachment setting commands, this implicitly - * encodes the number of attachments. These commands take multiple - * fixed-size @drm_asahi_neo_attachment structures as their payload, so size - * equals number of attachments * sizeof(struct drm_asahi_neo_attachment). - */ - __u16 size; - - /** - * @vdm_barrier: VDM (render) command index to wait on. - * - * Barriers are indices relative to the beginning of a given submit. A - * barrier of 0 waits on commands submitted to the respective subqueue - * in previous submit ioctls. A barrier of N waits on N previous - * commands on the subqueue within the current submit ioctl. As a - * special case, passing @DRM_ASAHI_NEO_BARRIER_NONE avoids waiting on any - * commands in the subqueue. - * - * Examples: - * - * 0: This waits on all previous work. - * - * NONE: This does not wait for anything on this subqueue. - * - * 1: This waits on the first render command in the submit. - * This is valid only if there are multiple render commands in the - * same submit. - * - * Barriers are valid only for hardware commands. Synthetic software - * commands to set attachments must pass NONE here. - */ - __u16 vdm_barrier; - - /** - * @cdm_barrier: CDM (compute) command index to wait on. - * - * See @vdm_barrier, and replace VDM/render with CDM/compute. - */ - __u16 cdm_barrier; -}; - -/** - * struct drm_asahi_neo_submit - Arguments passed to DRM_IOCTL_ASAHI_NEO_SUBMIT - */ -struct drm_asahi_neo_submit { - /** - * @syncs: An optional pointer to an array of drm_asahi_neo_sync. The first - * @in_sync_count elements are in-syncs, then the remaining - * @out_sync_count elements are out-syncs. Using a single array with - * explicit partitioning simplifies handling. - */ - __u64 syncs; - - /** - * @cmdbuf: Pointer to the command buffer to submit. - * - * This is a flat command buffer. By design, it contains no CPU - * pointers, which makes it suitable for a virtgpu wire protocol without - * requiring any serializing/deserializing step. - * - * It consists of a series of commands. Each command begins with a - * fixed-size @drm_asahi_neo_cmd_header header and is followed by a - * variable-length payload according to the type and size in the header. - * - * The combined count of "real" hardware commands must be nonzero and at - * most drm_asahi_neo_params_global::max_commands_per_submission. - */ - __u64 cmdbuf; - - /** @flags: Flags for command submission (MBZ) */ - __u32 flags; - - /** @queue_id: The queue ID to be submitted to */ - __u32 queue_id; - - /** - * @in_sync_count: Number of sync objects to wait on before starting - * this job. - */ - __u32 in_sync_count; - - /** - * @out_sync_count: Number of sync objects to signal upon completion of - * this job. - */ - __u32 out_sync_count; - - /** @cmdbuf_size: Command buffer size in bytes */ - __u32 cmdbuf_size; - - /** @pad: MBZ */ - __u32 pad; -}; - -/** - * struct drm_asahi_neo_attachment - Describe an "attachment". - * - * Attachments are any memory written by shaders, notably including render - * target attachments written by the end-of-tile program. This is purely a hint - * about the accessed memory regions. It is optional to specify, which is - * fortunate as it cannot be specified precisely with bindless access anyway. - * But where possible, it's probably a good idea for userspace to include these - * hints, forwarded to the firmware. - * - * This struct is implicitly sized and therefore is not extensible. - */ -struct drm_asahi_neo_attachment { - /** @pointer: Base address of the attachment */ - __u64 pointer; - - /** @size: Size of the attachment in bytes */ - __u64 size; - - /** @pad: MBZ */ - __u32 pad; - - /** @flags: MBZ */ - __u32 flags; -}; - -enum drm_asahi_neo_render_flags { - /** - * @DRM_ASAHI_NEO_RENDER_VERTEX_SCRATCH: A vertex stage shader uses scratch - * memory. - */ - DRM_ASAHI_NEO_RENDER_VERTEX_SCRATCH = (1U << 0), - - /** - * @DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES: Process even empty tiles. - * This must be set when clearing render targets. - */ - DRM_ASAHI_NEO_RENDER_PROCESS_EMPTY_TILES = (1U << 1), - - /** - * @DRM_ASAHI_NEO_RENDER_NO_VERTEX_CLUSTERING: Run vertex stage on a single - * cluster (on multi-cluster GPUs) - * - * This harms performance but can workaround certain sync/coherency - * bugs, and therefore is useful for debugging. - */ - DRM_ASAHI_NEO_RENDER_NO_VERTEX_CLUSTERING = (1U << 2), - - /** - * @DRM_ASAHI_NEO_RENDER_RSRC_SPEC_HI: The command carries the high halves - * of the background and end-of-tile resource specifiers - * (&drm_asahi_neo_cmd_render.bg_rsrc_spec_hi and the following fields). - * Kernels that do not know the flag reject the command instead of - * silently truncating the specifiers. - */ - DRM_ASAHI_NEO_RENDER_RSRC_SPEC_HI = (1U << 4), - - /** - * @DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT: Use integer depth bias formula. - * - * Graphics specifications contain two alternate formulas for depth - * bias, a float formula used with floating-point depth buffers and an - * integer formula using with unorm depth buffers. This flag specifies - * that the integer formula should be used. If omitted, the float - * formula is used instead. - * - * This corresponds to bit 18 of the relevant hardware control register, - * so we match that here for efficiency. - */ - DRM_ASAHI_NEO_RENDER_DBIAS_IS_INT = (1U << 18), - - /** - * @DRM_ASAHI_NEO_RENDER_FEW_PRIMITIVES: The render pass is known to rasterize - * at most four primitives in total. - * - * The count includes every primitive the pass generates: topology-derived - * primitive counts of all draws multiplied by their instance counts, plus - * internal work (clears, visibility, scissor/state generated work). Passes - * with indirect draws, mesh work or any work whose primitive count is not - * known at record time must not set this flag. The flag sets bit 0 of the - * auxiliary framebuffer control word for the fragment stage; the hardware - * effect is an optimization for small known passes and its omission is - * always safe. Requires DRM_ASAHI_NEO_FEATURE_FEW_PRIMITIVES. - */ - DRM_ASAHI_NEO_RENDER_FEW_PRIMITIVES = (1U << 19), - - /** - * @DRM_ASAHI_NEO_RENDER_FRAGMENT_BARRIERS: Only the fragment stage waits for - * this command's barriers. - * - * The render dependencies resolved from @vdm_barrier and @prior_vdm are - * attached to the fragment half of the render instead of its vertex - * (tiling) half, so vertex processing may start while the renders this - * command depends on are still executing their fragment work. The - * fragment half still waits for its own vertex half and for the previous - * fragment work of its physical queue; the compute dependencies from - * @cdm_barrier and @prior_cdm keep ordering the vertex half. Set this - * only when every render dependency into the command is consumed by - * fragment-stage or attachment accesses alone: nothing the vertex stage - * reads (vertex/index/indirect buffers, vertex, tessellation or geometry - * shader resources, transform feedback, conditional-rendering predicates) - * may depend on it. Requires DRM_ASAHI_NEO_FEATURE_FRAGMENT_BARRIERS. - */ - DRM_ASAHI_NEO_RENDER_FRAGMENT_BARRIERS = (1U << 20), -}; - -/** - * struct drm_asahi_neo_zls_buffer - Describe a depth or stencil buffer. - * - * These fields correspond to hardware registers in the ZLS (Z Load/Store) unit. - * There are three hardware registers for each field respectively for loads, - * stores, and partial renders. In practice, it makes sense to set all to the - * same values, except in exceptional cases not yet implemented in userspace, so - * we do not duplicate here for simplicity/efficiency. - * - * This struct is embedded in other structs and therefore is not extensible. - */ -struct drm_asahi_neo_zls_buffer { - /** @base: Base address of the buffer */ - __u64 base; - - /** - * @comp_base: If the load buffer is compressed, address of the - * compression metadata section. - */ - __u64 comp_base; - - /** - * @stride: If layered rendering is enabled, the number of bytes - * between each layer of the buffer. - */ - __u32 stride; - - /** - * @comp_stride: If layered rendering is enabled, the number of bytes - * between each layer of the compression metadata. - */ - __u32 comp_stride; -}; - -/** - * struct drm_asahi_neo_timestamp - Describe a timestamp write. - * - * The firmware can optionally write the GPU timestamp at render pass - * granularities, but it needs to be mapped specially via - * DRM_IOCTL_ASAHI_NEO_GEM_BIND_OBJECT. This structure therefore describes where to - * write as a handle-offset pair, rather than a GPU address like normal. - * - * This struct is embedded in other structs and therefore is not extensible. - */ -struct drm_asahi_neo_timestamp { - /** - * @handle: Handle of the timestamp buffer, or 0 to skip this - * timestamp. If nonzero, this must equal the value returned in - * drm_asahi_neo_gem_bind_object::object_handle. - */ - __u32 handle; - - /** @offset: Offset to write into the timestamp buffer */ - __u32 offset; -}; - -/** - * struct drm_asahi_neo_timestamps - Describe timestamp writes. - * - * Each operation that can be timestamped, can be timestamped at the start and - * end. Therefore, drm_asahi_neo_timestamp structs always come in pairs, bundled - * together into drm_asahi_neo_timestamps. - * - * This struct is embedded in other structs and therefore is not extensible. - */ -struct drm_asahi_neo_timestamps { - /** @start: Timestamp recorded at the start of the operation */ - struct drm_asahi_neo_timestamp start; - - /** @end: Timestamp recorded at the end of the operation */ - struct drm_asahi_neo_timestamp end; -}; - -/** - * struct drm_asahi_neo_helper_program - Describe helper program configuration. - * - * The helper program is a compute-like kernel required for various hardware - * functionality. Its most important role is dynamically allocating - * scratch/stack memory for individual subgroups, by partitioning a static - * allocation shared for the whole device. It is supplied by userspace via - * drm_asahi_neo_helper_program and internally dispatched by the hardware as needed. - * - * This struct is embedded in other structs and therefore is not extensible. - */ -struct drm_asahi_neo_helper_program { - /** - * @binary: USC address to the helper program binary. This is a tagged - * pointer with configuration in the bottom bits. - */ - __u32 binary; - - /** @cfg: Additional configuration bits for the helper program. */ - __u32 cfg; - - /** - * @data: Data passed to the helper program. This value is not - * interpreted by the kernel, firmware, or hardware in any way. It is - * simply a sideband for userspace, set with the submit ioctl and read - * via special registers inside the helper program. - * - * In practice, userspace will pass a 64-bit GPU VA here pointing to the - * actual arguments, which presumably don't fit in 64-bits. - */ - __u64 data; -}; - -/** - * struct drm_asahi_neo_bg_eot - Describe a background or end-of-tile program. - * - * The background and end-of-tile programs are dispatched by the hardware at the - * beginning and end of rendering. As the hardware "tilebuffer" is simply local - * memory, these programs are necessary to implement API-level render targets. - * The fragment-like background program is responsible for loading either the - * clear colour or the existing render target contents, while the compute-like - * end-of-tile program stores the tilebuffer contents to memory. - * - * This struct is embedded in other structs and therefore is not extensible. - */ -struct drm_asahi_neo_bg_eot { - /** - * @usc: USC address of the hardware USC words binding resources - * (including images and uniforms) and the program itself. Note this is - * an additional layer of indirection compared to the helper program, - * avoiding the need for a sideband for data. This is a tagged pointer - * with additional configuration in the bottom bits. - */ - __u32 usc; - - /** - * @rsrc_spec: Resource specifier for the program. This is a packed - * hardware data structure describing the required number of registers, - * uniforms, bound textures, and bound samplers. - */ - __u32 rsrc_spec; -}; - -/** - * struct drm_asahi_neo_cmd_render - Command to submit 3D - * - * This command submits a single render pass. The hardware control stream may - * include many draws and subpasses, but within the command, the framebuffer - * dimensions and attachments are fixed. - * - * The hardware requires the firmware to set a large number of Control Registers - * setting up state at render pass granularity before each command rendering 3D. - * The firmware bundles this state into data structures. Unfortunately, we - * cannot expose either any of that directly to userspace, because the - * kernel-firmware ABI is not stable. Although we can guarantee the firmware - * updates in tandem with the kernel, we cannot break old userspace when - * upgrading the firmware and kernel. Therefore, we need to abstract well the - * data structures to avoid tying our hands with future firmwares. - * - * The bulk of drm_asahi_neo_cmd_render therefore consists of values of hardware - * control registers, marshalled via the firmware interface. - * - * The framebuffer/tilebuffer dimensions are also specified here. In addition to - * being passed to the firmware/hardware, the kernel requires these dimensions - * to calculate various essential tiling-related data structures. It is - * unfortunate that our submits are heavier than on vendors with saner - * hardware-software interfaces. The upshot is all of this information is - * readily available to userspace with all current APIs. - * - * It looks odd - but it's not overly burdensome and it ensures we can remain - * compatible with old userspace. - */ -struct drm_asahi_neo_cmd_render { - /** @flags: Combination of drm_asahi_neo_render_flags flags. */ - __u32 flags; - - /** - * @isp_zls_pixels: ISP_ZLS_PIXELS register value. This contains the - * depth/stencil width/height, which may differ from the framebuffer - * width/height. - */ - __u32 isp_zls_pixels; - - /** - * @vdm_ctrl_stream_base: VDM_CTRL_STREAM_BASE register value. GPU - * address to the beginning of the VDM control stream. - */ - __u64 vdm_ctrl_stream_base; - - /** @vertex_helper: Helper program used for the vertex shader */ - struct drm_asahi_neo_helper_program vertex_helper; - - /** @fragment_helper: Helper program used for the fragment shader */ - struct drm_asahi_neo_helper_program fragment_helper; - - /** - * @isp_scissor_base: ISP_SCISSOR_BASE register value. GPU address of an - * array of scissor descriptors indexed in the render pass. - */ - __u64 isp_scissor_base; - - /** - * @isp_dbias_base: ISP_DBIAS_BASE register value. GPU address of an - * array of depth bias values indexed in the render pass. - */ - __u64 isp_dbias_base; - - /** - * @isp_oclqry_base: ISP_OCLQRY_BASE register value. GPU address of an - * array of occlusion query results written by the render pass. - */ - __u64 isp_oclqry_base; - - /** @depth: Depth buffer */ - struct drm_asahi_neo_zls_buffer depth; - - /** @stencil: Stencil buffer */ - struct drm_asahi_neo_zls_buffer stencil; - - /** @zls_ctrl: ZLS_CTRL register value */ - __u64 zls_ctrl; - - /** @ppp_multisamplectl: PPP_MULTISAMPLECTL register value */ - __u64 ppp_multisamplectl; - - /** - * @sampler_heap: Base address of the sampler heap. This heap is used - * for both vertex shaders and fragment shaders. The registers are - * per-stage, but there is no known use case for separate heaps. - */ - __u64 sampler_heap; - - /** @ppp_ctrl: PPP_CTRL register value */ - __u32 ppp_ctrl; - - /** @width_px: Framebuffer width in pixels */ - __u16 width_px; - - /** @height_px: Framebuffer height in pixels */ - __u16 height_px; - - /** @layers: Number of layers in the framebuffer */ - __u16 layers; - - /** @sampler_count: Number of samplers in the sampler heap. */ - __u16 sampler_count; - - /** @utile_width_px: Width of a logical tilebuffer tile in pixels */ - __u8 utile_width_px; - - /** @utile_height_px: Height of a logical tilebuffer tile in pixels */ - __u8 utile_height_px; - - /** @samples: # of samples in the framebuffer. Must be 1, 2, or 4. */ - __u8 samples; - - /** @sample_size_B: # of bytes in the tilebuffer required per sample. */ - __u8 sample_size_B; - - /** - * @isp_merge_upper_x: 32-bit float used in the hardware triangle - * merging. Calculate as: tan(60 deg) * width. - * - * Making these values UAPI avoids requiring floating-point calculations - * in the kernel in the hot path. - */ - __u32 isp_merge_upper_x; - - /** - * @isp_merge_upper_y: 32-bit float. Calculate as: tan(60 deg) * height. - * See @isp_merge_upper_x. - */ - __u32 isp_merge_upper_y; - - /** @bg: Background program run for each tile at the start */ - struct drm_asahi_neo_bg_eot bg; - - /** @eot: End-of-tile program ran for each tile at the end */ - struct drm_asahi_neo_bg_eot eot; - - /** - * @partial_bg: Background program ran at the start of each tile when - * resuming the render pass during a partial render. - */ - struct drm_asahi_neo_bg_eot partial_bg; - - /** - * @partial_eot: End-of-tile program ran at the end of each tile when - * pausing the render pass during a partial render. - */ - struct drm_asahi_neo_bg_eot partial_eot; - - /** - * @isp_bgobjdepth: ISP_BGOBJDEPTH register value. This is the depth - * buffer clear value, encoded in the depth buffer's format: either a - * 32-bit float or a 16-bit unorm (with upper bits zeroed). - */ - __u32 isp_bgobjdepth; - - /** - * @isp_bgobjvals: ISP_BGOBJVALS register value. The bottom 8-bits - * contain the stencil buffer clear value. - */ - __u32 isp_bgobjvals; - - /** @ts_vtx: Timestamps for the vertex portion of the render */ - struct drm_asahi_neo_timestamps ts_vtx; - - /** @ts_frag: Timestamps for the fragment portion of the render */ - struct drm_asahi_neo_timestamps ts_frag; - - /** @bg_eot_rsrc_spec_hi: High half of @eot.rsrc_spec */ - __u32 bg_eot_rsrc_spec_hi; - - /** @bg_eot_partial_rsrc_spec_hi: High half of @partial_eot.rsrc_spec */ - __u32 bg_eot_partial_rsrc_spec_hi; - - /** @bg_rsrc_spec_hi: High half of @bg.rsrc_spec */ - __u32 bg_rsrc_spec_hi; - - /** @bg_partial_rsrc_spec_hi: High half of @partial_bg.rsrc_spec */ - __u32 bg_partial_rsrc_spec_hi; - - /** - * @prior_vdm: Historical render prefix, or zero for no added dependency. - * - * Requires DRM_ASAHI_NEO_FEATURE_EXACT_PRIOR_BARRIERS. Counts hardware render - * commands accepted on this queue since creation, starting at one. A - * nonzero value must not exceed the render count at this ioctl's entry; - * it waits for every render command through that ordinal. The effective - * dependency is the maximum of this prefix and the header's resolved - * vdm_barrier. Header NONE does not suppress this field. - */ - __u64 prior_vdm; - - /** - * @prior_cdm: Historical compute prefix; see @prior_vdm, using the - * queue's compute command count and the header's cdm_barrier instead. - */ - __u64 prior_cdm; -}; - -/** - * struct drm_asahi_neo_cmd_compute - Command to submit compute - * - * This command submits a control stream consisting of compute dispatches. There - * is essentially no limit on how many compute dispatches may be included in a - * single compute command, although timestamps are at command granularity. - */ -struct drm_asahi_neo_cmd_compute { - /** @flags: MBZ */ - __u32 flags; - - /** @sampler_count: Number of samplers in the sampler heap. */ - __u32 sampler_count; - - /** - * @cdm_ctrl_stream_base: CDM_CTRL_STREAM_BASE register value. GPU - * address to the beginning of the CDM control stream. - */ - __u64 cdm_ctrl_stream_base; - - /** - * @cdm_ctrl_stream_end: GPU base address to the end of the hardware - * control stream. Note this only considers the first contiguous segment - * of the control stream, as the stream might jump elsewhere. - */ - __u64 cdm_ctrl_stream_end; - - /** @sampler_heap: Base address of the sampler heap. */ - __u64 sampler_heap; - - /** @helper: Helper program used for this compute command */ - struct drm_asahi_neo_helper_program helper; - - /** @ts: Timestamps for the compute command */ - struct drm_asahi_neo_timestamps ts; - - /** @scs_layout: Reserved; must be zero. */ - __u64 scs_layout; - - /** - * @prior_vdm: Historical render prefix, or zero for no added dependency. - * - * Requires DRM_ASAHI_NEO_FEATURE_EXACT_PRIOR_BARRIERS. Counts hardware render - * commands accepted on this queue since creation, starting at one. A - * nonzero value must not exceed the render count at this ioctl's entry; - * it waits for every render command through that ordinal. The effective - * dependency is the maximum of this prefix and the header's resolved - * vdm_barrier. Header NONE does not suppress this field. - */ - __u64 prior_vdm; - - /** - * @prior_cdm: Historical compute prefix; see @prior_vdm, using the - * queue's compute command count and the header's cdm_barrier instead. - */ - __u64 prior_cdm; -}; - -/** - * struct drm_asahi_neo_get_time - Arguments passed to DRM_IOCTL_ASAHI_NEO_GET_TIME - */ -struct drm_asahi_neo_get_time { - /** @flags: MBZ. */ - __u64 flags; - - /** @gpu_timestamp: On return, the GPU timestamp in nanoseconds. */ - __u64 gpu_timestamp; -}; - -/** - * DRM_IOCTL_ASAHI_NEO() - Build an Asahi IOCTL number - * @__access: Access type. Must be R, W or RW. - * @__id: One of the DRM_ASAHI_NEO_xxx id. - * @__type: Suffix of the type being passed to the IOCTL. - * - * Don't use this macro directly, use the DRM_IOCTL_ASAHI_NEO_xxx - * values instead. - * - * Return: An IOCTL number to be passed to ioctl() from userspace. - */ -#define DRM_IOCTL_ASAHI_NEO(__access, __id, __type) \ - DRM_IO ## __access(DRM_COMMAND_BASE + DRM_ASAHI_NEO_ ## __id, \ - struct drm_asahi_neo_ ## __type) - -/* Note: this is an enum so that it can be resolved by Rust bindgen. */ -enum { - DRM_IOCTL_ASAHI_NEO_GET_PARAMS = DRM_IOCTL_ASAHI_NEO(W, GET_PARAMS, get_params), - DRM_IOCTL_ASAHI_NEO_GET_TIME = DRM_IOCTL_ASAHI_NEO(WR, GET_TIME, get_time), - DRM_IOCTL_ASAHI_NEO_VM_CREATE = DRM_IOCTL_ASAHI_NEO(WR, VM_CREATE, vm_create), - DRM_IOCTL_ASAHI_NEO_VM_DESTROY = DRM_IOCTL_ASAHI_NEO(W, VM_DESTROY, vm_destroy), - DRM_IOCTL_ASAHI_NEO_VM_BIND = DRM_IOCTL_ASAHI_NEO(W, VM_BIND, vm_bind), - DRM_IOCTL_ASAHI_NEO_GEM_CREATE = DRM_IOCTL_ASAHI_NEO(WR, GEM_CREATE, gem_create), - DRM_IOCTL_ASAHI_NEO_GEM_MMAP_OFFSET = DRM_IOCTL_ASAHI_NEO(WR, GEM_MMAP_OFFSET, gem_mmap_offset), - DRM_IOCTL_ASAHI_NEO_GEM_BIND_OBJECT = DRM_IOCTL_ASAHI_NEO(WR, GEM_BIND_OBJECT, gem_bind_object), - DRM_IOCTL_ASAHI_NEO_QUEUE_CREATE = DRM_IOCTL_ASAHI_NEO(WR, QUEUE_CREATE, queue_create), - DRM_IOCTL_ASAHI_NEO_QUEUE_DESTROY = DRM_IOCTL_ASAHI_NEO(W, QUEUE_DESTROY, queue_destroy), - DRM_IOCTL_ASAHI_NEO_SUBMIT = DRM_IOCTL_ASAHI_NEO(W, SUBMIT, submit), - DRM_IOCTL_ASAHI_NEO_GEM_MADVISE = DRM_IOCTL_ASAHI_NEO(WR, GEM_MADVISE, gem_madvise), -}; - -#if defined(__cplusplus) -} -#endif - -#endif /* _ASAHI_NEO_DRM_H_ */ diff --git a/include/uapi/linux/apple-sep.h b/include/uapi/linux/apple-sep.h new file mode 100644 index 00000000000000..394b497ff7a59c --- /dev/null +++ b/include/uapi/linux/apple-sep.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note OR MIT */ +/* Copyright 2026 Dj */ +/* + * Userspace interface for the SEP biometric device. + * + * The enclave does the matching. Nothing biometric crosses this interface: only + * an operation, a stage, a status, an opaque identity UUID, and an opaque host + * label userspace chooses. + */ + +#ifndef _UAPI_LINUX_APPLE_SEP_H +#define _UAPI_LINUX_APPLE_SEP_H + +#include +#include + +#define SEP_BIO_IFACE_VERSION 4 + +#define SEP_BIO_UUID_LEN 16 +#define SEP_BIO_LABEL_LEN 128 +#define SEP_BIO_NONCE_LEN 32 +#define SEP_BIO_TOKEN_LEN 32 +#define SEP_BIO_MAX_IDENTITIES 32 +#define SEP_BIO_CHALLENGE_LEN 32 +#define SEP_BIO_ATTEST_PUB_LEN 65 +#define SEP_BIO_ATTEST_SIG_MAX 72 + +enum { + SEP_BIO_STATE_IDLE = 0, + SEP_BIO_STATE_PENDING = 1, + SEP_BIO_STATE_PROGRESS = 2, + SEP_BIO_STATE_DONE = 3, + SEP_BIO_STATE_FAILED = 4, +}; + +enum { + SEP_BIO_NO_MATCH = 0, + SEP_BIO_MATCH = 1, + SEP_BIO_NOT_COMPARED = 2, +}; + +struct sep_bio_identity { + __u8 uuid[SEP_BIO_UUID_LEN]; + __u8 label[SEP_BIO_LABEL_LEN]; +}; + +struct sep_bio_info { + __u32 version; + __u32 sensor_present; + __u32 enrolled; + __u32 capacity; /* usable slots in this SEP user context, not ABI list size */ + __u32 enroll_stages; + __u32 reserved[3]; +}; + +/* reserved[0] flags; reserved[1] is a live SEP identity count only when valid. */ +#define SEP_BIO_INFO_LIVE_COUNT_VALID 1u + +struct sep_bio_list { + __u32 count; + __u32 reserved; + struct sep_bio_identity id[SEP_BIO_MAX_IDENTITIES]; +}; + +struct sep_bio_enrol_start { + __u32 flags; + __u32 reserved; + __u8 label[SEP_BIO_LABEL_LEN]; +}; + +#define SEP_BIO_GUIDANCE_NONE 0 +#define SEP_BIO_GUIDANCE_PLACE 1 +#define SEP_BIO_GUIDANCE_LIFT_AND_MOVE 2 +#define SEP_BIO_GUIDANCE_HOLD_STILL 3 + +struct sep_bio_enrol_poll { + __u32 state; + __u32 stage; + __u32 stages_total; + __u32 status; + __u8 uuid[SEP_BIO_UUID_LEN]; + __u32 guidance; + __u32 progress_percent; +}; + +struct sep_bio_verify_start { + __u32 flags; + __u32 reserved; + __u8 nonce[SEP_BIO_NONCE_LEN]; +}; + +struct sep_bio_verify_poll { + __u32 state; + __u32 result; + __u32 status; + __u32 guidance; /* SEP_BIO_GUIDANCE_*; status only, no capture side effect */ + __u8 uuid[SEP_BIO_UUID_LEN]; + __u8 token[SEP_BIO_TOKEN_LEN]; /* single use, bound to the nonce */ + __u64 deadline_ns; /* CLOCK_MONOTONIC; past this the token is void */ +}; + +struct sep_bio_delete { + __u8 uuid[SEP_BIO_UUID_LEN]; +}; + +/* + * Device attestation of key possession: the enclave signs 'challenge' with the + * machine ref-key (ECDSA-P256 over the challenge as the pre-computed digest) and + * returns the DER signature and public point. The private key never leaves the + * enclave. + */ +struct sep_bio_attest { + __u32 sig_len; /* out: DER signature length */ + __u8 challenge[SEP_BIO_CHALLENGE_LEN]; /* in */ + __u8 public_key[SEP_BIO_ATTEST_PUB_LEN]; /* out: P-256 point, 04||X||Y */ + __u8 signature[SEP_BIO_ATTEST_SIG_MAX]; /* out: DER SEQUENCE{r,s} */ + __u8 reserved[3]; +}; + +#define SEP_BIO_IOC_MAGIC 0xB1 + +#define SEP_BIO_GET_INFO _IOR(SEP_BIO_IOC_MAGIC, 0x01, struct sep_bio_info) +#define SEP_BIO_LIST _IOR(SEP_BIO_IOC_MAGIC, 0x02, struct sep_bio_list) +#define SEP_BIO_ENROL_START _IOW(SEP_BIO_IOC_MAGIC, 0x03, struct sep_bio_enrol_start) +#define SEP_BIO_ENROL_POLL _IOR(SEP_BIO_IOC_MAGIC, 0x04, struct sep_bio_enrol_poll) +#define SEP_BIO_VERIFY_START _IOW(SEP_BIO_IOC_MAGIC, 0x05, struct sep_bio_verify_start) +#define SEP_BIO_VERIFY_POLL _IOR(SEP_BIO_IOC_MAGIC, 0x06, struct sep_bio_verify_poll) +#define SEP_BIO_CANCEL _IO(SEP_BIO_IOC_MAGIC, 0x07) +#define SEP_BIO_DELETE _IOW(SEP_BIO_IOC_MAGIC, 0x08, struct sep_bio_delete) +#define SEP_BIO_DELETE_ALL _IO(SEP_BIO_IOC_MAGIC, 0x09) +#define SEP_BIO_ATTEST _IOWR(SEP_BIO_IOC_MAGIC, 0x0a, struct sep_bio_attest) + +#endif /* _UAPI_LINUX_APPLE_SEP_H */ diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 1a06e7feb7dc9b..6e08471fbe822f 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -421,4 +421,22 @@ struct prctl_mm_map { # define PR_CFI_DISABLE _BITUL(1) # define PR_CFI_LOCK _BITUL(2) +/* + * Aurora experimental ABI: private numbers, not upstream assignments. + * SET_EXEC selects the calling thread's next successful exec granule. + * Zero requests native pages; with an alternative DEFAULT in effect this + * installs an explicit one-shot native override. GET_EXEC returns zero when + * no override remains, or its size in bytes. Successful exec consumes it. + * SET_DEFAULT selects the fallback for subsequent execs; zero restores native. + * GET_DEFAULT returns the alternative size in bytes, or zero for native. + * Fork inherits both policies; exec preserves DEFAULT. Failed exec preserves + * both. Neither setter changes the live address space or other threads. + * AArch32 always selects 4K when supported, otherwise the native granule, + * independently of these preferences. Sizes are capability-checked. + */ +#define PR_AURORA_SET_EXEC_PAGE_SIZE 0x41555001 +#define PR_AURORA_GET_EXEC_PAGE_SIZE 0x41555002 +#define PR_AURORA_SET_DEFAULT_PAGE_SIZE 0x41555003 +#define PR_AURORA_GET_DEFAULT_PAGE_SIZE 0x41555004 + #endif /* _LINUX_PRCTL_H */ diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 4f9b439319c466..72e123757bea4f 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -289,7 +289,10 @@ static int io_region_mmap(struct io_ring_ctx *ctx, unsigned long nr_pages = min(mr->nr_pages, max_pages); vm_flags_set(vma, VM_DONTEXPAND); - return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); + if (mm_page_size(vma->vm_mm) == PAGE_SIZE) + return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); + return vm_insert_pages_range(vma, vma->vm_start, mr->pages, nr_pages, 0, + min_t(unsigned long, vma->vm_end - vma->vm_start, nr_pages << PAGE_SHIFT)); } __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) @@ -301,6 +304,9 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) struct io_mapped_region *region; void *ptr; + if (vma_subpage_offset(vma)) + return -EINVAL; + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(file, vma->vm_pgoff); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 2dfc26233e2865..b442d5cf6c7f5d 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -4,6 +4,8 @@ #include #include #include +#include +#include #include #include #include @@ -30,6 +32,10 @@ struct io_rsrc_update { static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct page **last_hpage); +#ifdef CONFIG_MM_SUBPAGE +static void io_fragment_account_put(struct io_fragment_account *account); +#endif + /* only define max */ #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) @@ -167,6 +173,10 @@ bool io_rsrc_cache_init(struct io_ring_ctx *ctx) void io_rsrc_cache_free(struct io_ring_ctx *ctx) { +#ifdef CONFIG_MM_SUBPAGE + io_fragment_account_put(ctx->fragment_account); + ctx->fragment_account = NULL; +#endif io_alloc_cache_free(&ctx->node_cache, kfree); io_alloc_cache_free(&ctx->imu_cache, kvfree); } @@ -634,6 +644,9 @@ static bool headpage_already_acct(struct io_ring_ctx *ctx, struct page **pages, if (!node) continue; imu = node->buf; + /* Other backing owners have independent charge lifetimes. */ + if (imu->release != io_release_ubuf) + continue; for (j = 0; j < imu->nr_bvecs; j++) { if (!PageCompound(imu->bvec[j].bv_page)) continue; @@ -762,6 +775,218 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, return true; } +#ifdef CONFIG_MM_SUBPAGE +/* + * Shared by rings charging the same counters, including cloned buffers. + * uring_lock nests outside lock; the registry lock is never held with lock. + * Long-term pins keep each indexed folio and its size stable until release. + */ +struct io_fragment_account { + struct hlist_node node; + refcount_t refs; + struct mutex lock; + struct xarray folios; + struct user_struct *user; + struct mm_struct *mm; +}; + +static DEFINE_HASHTABLE(io_fragment_accounts, 6); +static DEFINE_MUTEX(io_fragment_accounts_lock); + +static void io_fragment_account_put(struct io_fragment_account *account) +{ + if (!account || !refcount_dec_and_mutex_lock(&account->refs, + &io_fragment_accounts_lock)) + return; + hash_del(&account->node); + mutex_unlock(&io_fragment_accounts_lock); + WARN_ON_ONCE(!xa_empty(&account->folios)); + xa_destroy(&account->folios); + if (account->mm) + mmdrop(account->mm); + if (account->user) + free_uid(account->user); + kfree(account); +} + +static struct io_fragment_account *io_fragment_account_get(struct io_ring_ctx *ctx) +{ + unsigned long key = (unsigned long)ctx->mm_account ^ (unsigned long)ctx->user; + struct io_fragment_account *account; + + lockdep_assert_held(&ctx->uring_lock); + account = ctx->fragment_account; + if (account) + goto found; + + mutex_lock(&io_fragment_accounts_lock); + hash_for_each_possible(io_fragment_accounts, account, node, key) { + if (account->user == ctx->user && account->mm == ctx->mm_account) { + refcount_inc(&account->refs); + goto install; + } + } + account = kzalloc(sizeof(*account), GFP_KERNEL_ACCOUNT); + if (!account) { + mutex_unlock(&io_fragment_accounts_lock); + return NULL; + } + refcount_set(&account->refs, 1); + mutex_init(&account->lock); + xa_init(&account->folios); + account->user = ctx->user ? get_uid(ctx->user) : NULL; + account->mm = ctx->mm_account; + if (account->mm) + mmgrab(account->mm); + hash_add(io_fragment_accounts, &account->node, key); +install: + ctx->fragment_account = account; + mutex_unlock(&io_fragment_accounts_lock); +found: + /* The buffer retains its account after the originating ring exits. */ + refcount_inc(&account->refs); + return account; +} + +struct io_fragment_buffer { + unsigned int count; + unsigned int accounted; + struct io_fragment_account *account; + struct user_page_fragment fragments[]; +}; + +static int io_account_fragment_buffer(struct io_fragment_buffer *buffer) +{ + struct io_fragment_account *account = buffer->account; + unsigned int i; + int ret = 0; + + mutex_lock(&account->lock); + for (i = 0; i < buffer->count; i++) { + struct folio *folio = buffer->fragments[i].folio; + unsigned long pfn = folio_pfn(folio); + void *entry = xa_load(&account->folios, pfn); + unsigned long refs = entry ? xa_to_value(entry) : 0; + + if (refs == LONG_MAX) { + ret = -EOVERFLOW; + break; + } + ret = xa_err(xa_store(&account->folios, pfn, xa_mk_value(refs + 1), + GFP_KERNEL_ACCOUNT)); + if (ret) + break; + if (!refs) { + ret = io_account_mem(account->user, account->mm, + folio_nr_pages(folio)); + if (ret) { + xa_erase(&account->folios, pfn); + break; + } + } + buffer->accounted++; + } + mutex_unlock(&account->lock); + return ret; +} + +static void io_release_fragment_buffer(void *priv) +{ + struct io_fragment_buffer *buffer = priv; + struct io_fragment_account *account = buffer->account; + unsigned int i; + + if (account) { + mutex_lock(&account->lock); + for (i = 0; i < buffer->accounted; i++) { + struct folio *folio = buffer->fragments[i].folio; + unsigned long pfn = folio_pfn(folio); + unsigned long refs = xa_to_value(xa_load(&account->folios, pfn)); + + if (refs == 1) { + xa_erase(&account->folios, pfn); + io_unaccount_mem(account->user, account->mm, + folio_nr_pages(folio)); + } else { + xa_store(&account->folios, pfn, xa_mk_value(refs - 1), + GFP_NOWAIT); + } + } + mutex_unlock(&account->lock); + io_fragment_account_put(account); + } + + release_user_fragments(buffer->fragments, buffer->count, false); + kvfree(buffer); +} + +static struct io_rsrc_node *io_register_fragment_buffer(struct io_ring_ctx *ctx, + struct iovec *iov) +{ + unsigned long size = mm_page_size(current->mm); + unsigned long first = (unsigned long)iov->iov_base & (size - 1); + unsigned int count = DIV_ROUND_UP(first + iov->iov_len, size); + struct io_fragment_buffer *buffer; + struct io_mapped_ubuf *imu = NULL; + struct io_rsrc_node *node; + unsigned int i; + long nr; + int ret = -ENOMEM; + + node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); + if (!node) + return ERR_PTR(-ENOMEM); + buffer = kvmalloc(struct_size(buffer, fragments, count), GFP_KERNEL_ACCOUNT); + if (!buffer) + goto free_node; + buffer->count = 0; + buffer->accounted = 0; + buffer->account = NULL; + nr = pin_user_fragments_remote(current->mm, (unsigned long)iov->iov_base, + iov->iov_len, FOLL_WRITE | FOLL_LONGTERM, buffer->fragments, count); + if (nr > 0) + buffer->count = nr; + if (nr != count) { + ret = nr < 0 ? nr : -EFAULT; + goto free_buffer; + } + imu = io_alloc_imu(ctx, count); + if (!imu) + goto free_buffer; + imu->nr_bvecs = count; + imu->acct_pages = 0; + for (i = 0; i < count; i++) { + struct user_page_fragment *f = &buffer->fragments[i]; + + bvec_set_page(&imu->bvec[i], folio_page(f->folio, f->offset >> PAGE_SHIFT), + f->length, offset_in_page(f->offset)); + } + buffer->account = io_fragment_account_get(ctx); + if (!buffer->account) + goto free_imu; + ret = io_account_fragment_buffer(buffer); + if (ret) + goto free_imu; + imu->ubuf = (unsigned long)iov->iov_base; + imu->len = iov->iov_len; + imu->folio_shift = mm_page_shift(current->mm); + imu->release = io_release_fragment_buffer; + imu->priv = buffer; + imu->flags = 0; + imu->dir = IO_IMU_DEST | IO_IMU_SOURCE; + refcount_set(&imu->refs, 1); + node->buf = imu; + return node; +free_imu: + io_free_imu(ctx, imu); +free_buffer: + io_release_fragment_buffer(buffer); +free_node: + io_cache_free(&ctx->node_cache, node); + return ERR_PTR(ret); +} +#endif + static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct page **last_hpage) @@ -787,6 +1012,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, if (ret) return ERR_PTR(ret); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) < PAGE_SIZE) + return io_register_fragment_buffer(ctx, iov); +#endif + node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); if (!node) return ERR_PTR(-ENOMEM); @@ -1095,6 +1325,14 @@ static int io_import_fixed(int ddir, struct iov_iter *iter, bvec += seg_skip; offset &= folio_mask; } +#ifdef CONFIG_MM_SUBPAGE + if (imu->folio_shift < PAGE_SHIFT) { + /* Physical fragment offsets are independent of the user address. */ + size_t first = min_t(size_t, len, bvec->bv_len - offset); + + nr_segs = 1 + DIV_ROUND_UP(len - first, 1UL << imu->folio_shift); + } else +#endif nr_segs = (offset + len + bvec->bv_offset + folio_mask) >> imu->folio_shift; iov_iter_bvec(iter, ddir, bvec, nr_segs, len); iter->iov_offset = offset; @@ -1356,6 +1594,27 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, return -EOVERFLOW; offset = buf_addr - imu->ubuf; +#ifdef CONFIG_MM_SUBPAGE + if (imu->folio_shift < PAGE_SHIFT) { + unsigned int index; + unsigned long first = imu->ubuf & folio_mask; + + offset += first; + index = offset >> imu->folio_shift; + offset &= folio_mask; + if (!index) + offset -= first; + src_bvec = imu->bvec + index; + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t bytes = min_t(size_t, iov_len, src_bvec->bv_len - offset); + + bvec_set_page(&res_bvec[bvec_idx], src_bvec->bv_page, + bytes, src_bvec->bv_offset + offset); + iov_len -= bytes; + } + continue; + } +#endif /* * Only the first bvec can have non zero bv_offset, account it * here and work with full folios below. diff --git a/ipc/shm.c b/ipc/shm.c index b3e8a58e177d3b..2929b9243f8e89 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -51,6 +51,14 @@ #include "util.h" +#ifndef arch_shm_attach_align +static unsigned long arch_shm_attach_align(const struct mm_struct *mm, + unsigned long align) +{ + return align; +} +#endif + struct shmid_kernel /* private to the kernel */ { struct kern_ipc_perm shm_perm; @@ -1538,6 +1546,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, if (shmid < 0) goto out; + shmlba = arch_shm_attach_align(current->mm, shmlba); if (addr) { if (addr & (shmlba - 1)) { if (shmflg & SHM_RND) { @@ -1552,7 +1561,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto out; } else #ifndef __ARCH_FORCE_SHMLBA - if (addr & ~PAGE_MASK) + if (addr & ~mm_page_mask(current->mm)) #endif goto out; } @@ -1723,6 +1732,20 @@ COMPAT_SYSCALL_DEFINE3(shmat, int, shmid, compat_uptr_t, shmaddr, int, shmflg) } #endif +/* VMA splits retain native file indices plus a subpage byte offset. */ +#ifdef CONFIG_MMU +static bool shm_vma_matches_addr(struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long offset; + + if (vma->vm_ops != &shm_vm_ops || vma->vm_start < addr) + return false; + offset = vma->vm_start - addr; + return (offset >> PAGE_SHIFT) == vma->vm_pgoff && + (offset & ~PAGE_MASK) == vma_subpage_offset(vma); +} +#endif + /* * detach and kill segment if marked destroyed. * The work is done in shm_close. @@ -1739,7 +1762,7 @@ long ksys_shmdt(char __user *shmaddr) VMA_ITERATOR(vmi, mm, addr); #endif - if (addr & ~PAGE_MASK) + if (addr & ~mm_page_mask(mm)) return retval; if (mmap_write_lock_killable(mm)) @@ -1774,8 +1797,7 @@ long ksys_shmdt(char __user *shmaddr) * a fragment created by mprotect() and/or munmap(), or it * otherwise it starts at this address with no hassles. */ - if ((vma->vm_ops == &shm_vm_ops) && - (vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) { + if (shm_vma_matches_addr(vma, addr)) { /* * Record the file of the shm segment being @@ -1804,12 +1826,10 @@ long ksys_shmdt(char __user *shmaddr) * could possibly have landed at. Also cast things to loff_t to * prevent overflows and make comparisons vs. equal-width types. */ - size = PAGE_ALIGN(size); + size = ALIGN(size, mm_page_size(mm)); while (vma && (loff_t)(vma->vm_end - addr) <= size) { /* finding a matching vma now does not alter retval */ - if ((vma->vm_ops == &shm_vm_ops) && - ((vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) && - (vma->vm_file == file)) { + if (shm_vma_matches_addr(vma, addr) && vma->vm_file == file) { do_vmi_align_munmap(&vmi, vma, mm, vma->vm_start, vma->vm_end, NULL, false); } diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 6fc6a4b672b556..9abd4972966ffe 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -24,6 +24,11 @@ select_bucket(struct bpf_local_storage_map *smap, return &smap->buckets[hash_ptr(local_storage, smap->bucket_log)]; } +static u32 bpf_selem_size(const struct bpf_local_storage_map *smap) +{ + return smap->elem_size + bpf_obj_uptr_extra_size(smap->map.record); +} + static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size) { struct bpf_map *map = &smap->map; @@ -72,10 +77,10 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, { struct bpf_local_storage_elem *selem; - if (mem_charge(smap, owner, smap->elem_size)) + if (mem_charge(smap, owner, bpf_selem_size(smap))) return NULL; - selem = bpf_map_kmalloc_nolock(&smap->map, smap->elem_size, + selem = bpf_map_kmalloc_nolock(&smap->map, bpf_selem_size(smap), __GFP_ZERO, NUMA_NO_NODE); if (selem) { @@ -91,7 +96,7 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, return selem; } - mem_uncharge(smap, owner, smap->elem_size); + mem_uncharge(smap, owner, bpf_selem_size(smap)); return NULL; } @@ -178,7 +183,7 @@ static void bpf_selem_unlink_storage_nolock_misc(struct bpf_local_storage_elem * bool free_local_storage, bool pin_owner) { void *owner = local_storage->owner; - u32 uncharge = smap->elem_size; + u32 uncharge = bpf_selem_size(smap); if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) == SDATA(selem)) @@ -204,7 +209,7 @@ static void bpf_selem_unlink_storage_nolock_misc(struct bpf_local_storage_elem * /* local_storage->lock must be held and selem->local_storage == local_storage. * The caller must ensure selem->smap is still valid to be - * dereferenced for its smap->elem_size and smap->cache_idx. + * dereferenced for its element size and smap->cache_idx. */ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage, struct bpf_local_storage_elem *selem, @@ -234,7 +239,7 @@ void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage, struct bpf_local_storage_map *smap; smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held()); - local_storage->mem_charge += smap->elem_size; + local_storage->mem_charge += bpf_selem_size(smap); RCU_INIT_POINTER(selem->local_storage, local_storage); hlist_add_head_rcu(&selem->snode, &local_storage->list); @@ -577,7 +582,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, err = bpf_local_storage_alloc(owner, smap, selem); if (err) { bpf_selem_free(selem, true); - mem_uncharge(smap, owner, smap->elem_size); + mem_uncharge(smap, owner, bpf_selem_size(smap)); return ERR_PTR(err); } @@ -661,7 +666,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, free_selem: bpf_selem_free_list(&old_selem_free_list, false); if (alloc_selem) { - mem_uncharge(smap, owner, smap->elem_size); + mem_uncharge(smap, owner, bpf_selem_size(smap)); bpf_selem_free(alloc_selem, true); } return err ? ERR_PTR(err) : SDATA(selem); @@ -862,7 +867,7 @@ void bpf_local_storage_map_free(struct bpf_map *map, * exit immediately. * * However, while freeing the storage one still needs to access the - * smap->elem_size to do the uncharging in + * bpf_selem_size(smap) to do the uncharging in * bpf_selem_unlink_storage_nolock(). * * Hence, wait another rcu grace period for the storage to be freed. diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 96d1b797e1523c..c084c1cad03d85 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -15,6 +15,8 @@ #include #include #include +#include +#include #include #include #include @@ -164,6 +166,43 @@ static void maybe_wait_bpf_programs(struct bpf_map *map) synchronize_rcu_expedited(); } +#ifdef CONFIG_MM_SUBPAGE +struct bpf_uptr_pins { + struct work_struct work; + u32 count; + struct user_page_fragment fragments[]; +}; + +static void bpf_uptr_release_work(struct work_struct *work) +{ + struct bpf_uptr_pins *pins = container_of(work, struct bpf_uptr_pins, work); + + release_user_fragments(pins->fragments, pins->count, true); + kfree(pins); +} +#endif + +/* May run from an RCU callback. The bundle owns its pins independently of + * the map, task and value, so dirty release can wait for folio locks in work. + */ +static bool bpf_obj_release_uptr_pins(const struct btf_record *rec, void *obj) +{ +#ifdef CONFIG_MM_SUBPAGE + struct bpf_uptr_pins **owner, *pins; + + if (!bpf_obj_uptr_extra_size(rec)) + return false; + owner = obj + rec->uptr_pins_off; + pins = *owner; + if (pins) { + *owner = NULL; + queue_work(system_unbound_wq, &pins->work); + return true; + } +#endif + return false; +} + static void unpin_uptr_kaddr(void *kaddr) { if (kaddr) @@ -176,6 +215,9 @@ static void __bpf_obj_unpin_uptrs(struct btf_record *rec, u32 cnt, void *obj) void **uptr_addr; int i; + if (bpf_obj_release_uptr_pins(rec, obj)) + return; + for (i = 0, field = rec->fields; i < cnt; i++, field++) { if (field->type != BPF_UPTR) continue; @@ -193,8 +235,12 @@ static void bpf_obj_unpin_uptrs(struct btf_record *rec, void *obj) __bpf_obj_unpin_uptrs(rec, rec->cnt, obj); } -static int bpf_obj_pin_uptrs(struct btf_record *rec, void *obj) +static int bpf_obj_pin_uptrs(struct bpf_map *map, void *obj) { + struct btf_record *rec = map->record; +#ifdef CONFIG_MM_SUBPAGE + struct bpf_uptr_pins *pins = NULL; +#endif const struct btf_field *field; const struct btf_type *t; unsigned long start, end; @@ -205,6 +251,18 @@ static int bpf_obj_pin_uptrs(struct btf_record *rec, void *obj) if (!btf_record_has_field(rec, BPF_UPTR)) return 0; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) { + pins = bpf_map_kzalloc(map, struct_size(pins, fragments, rec->cnt), + GFP_KERNEL); + if (!pins) + return -ENOMEM; + INIT_WORK(&pins->work, bpf_uptr_release_work); + pins->count = rec->cnt; + *(struct bpf_uptr_pins **)(obj + rec->uptr_pins_off) = pins; + } +#endif + for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) { if (field->type != BPF_UPTR) continue; @@ -221,12 +279,35 @@ static int bpf_obj_pin_uptrs(struct btf_record *rec, void *obj) goto unpin_all; } - /* The uptr's struct cannot span across two pages */ - if ((start & PAGE_MASK) != (end & PAGE_MASK)) { + /* The object must fit in one page of the supplying process. */ + if ((start & mm_page_mask(current->mm)) != + (end & mm_page_mask(current->mm))) { err = -EOPNOTSUPP; goto unpin_all; } +#ifdef CONFIG_MM_SUBPAGE + if (pins) { + struct user_page_fragment *fragment = &pins->fragments[i]; + + err = pin_user_fragments_remote(current->mm, start, t->size, + FOLL_LONGTERM | FOLL_WRITE, fragment, 1); + if (err != 1) { + if (!err) + err = -EFAULT; + goto unpin_all; + } + if (fragment->length != t->size || + PageHighMem(folio_page(fragment->folio, + fragment->offset >> PAGE_SHIFT))) { + err = -EOPNOTSUPP; + goto unpin_all; + } + *uptr_addr = folio_address(fragment->folio) + fragment->offset; + continue; + } +#endif + err = pin_user_pages_fast(start, 1, FOLL_LONGTERM | FOLL_WRITE, &page); if (err != 1) goto unpin_all; @@ -291,7 +372,7 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, map->map_type == BPF_MAP_TYPE_BLOOM_FILTER) { err = map->ops->map_push_elem(map, value, flags); } else { - err = bpf_obj_pin_uptrs(map->record, value); + err = bpf_obj_pin_uptrs(map, value); if (!err) { rcu_read_lock(); err = map->ops->map_update_elem(map, key, value, flags); @@ -815,10 +896,12 @@ void bpf_obj_cancel_fields(struct bpf_map *map, void *obj) void bpf_obj_free_fields(const struct btf_record *rec, void *obj) { const struct btf_field *fields; + bool fragment_pins; int i; if (IS_ERR_OR_NULL(rec)) return; + fragment_pins = bpf_obj_release_uptr_pins(rec, obj); fields = rec->fields; for (i = 0; i < rec->cnt; i++) { struct btf_struct_meta *pointee_struct_meta; @@ -859,8 +942,9 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) } break; case BPF_UPTR: - /* The caller ensured that no one is using the uptr */ - unpin_uptr_kaddr(*(void **)field_ptr); + /* The caller ensured that no one is using the uptr. */ + if (!fragment_pins) + unpin_uptr_kaddr(*(void **)field_ptr); break; case BPF_LIST_HEAD: if (WARN_ON_ONCE(rec->spin_lock_off < 0)) @@ -1350,6 +1434,14 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, if (ret < 0) goto free_map_tab; +#ifdef CONFIG_MM_SUBPAGE + if (btf_record_has_field(map->record, BPF_UPTR)) { + map->record->uptr_pins_off = ALIGN(map->value_size, sizeof(void *)); + map->record->uptr_extra_size = map->record->uptr_pins_off + + sizeof(void *) - map->value_size; + } +#endif + if (map->ops->map_check_btf) { ret = map->ops->map_check_btf(map, btf, key_type, value_type); if (ret < 0) @@ -1775,6 +1867,24 @@ static int map_lookup_elem(union bpf_attr *attr) #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags +static void *bpf_map_memdup_value(struct bpf_map *map, bpfptr_t src, u32 size) +{ + u32 extra = bpf_obj_uptr_extra_size(map->record); + void *value; + + if (!extra) + return kvmemdup_bpfptr(src, size); + value = kvmalloc(size + extra, GFP_USER); + if (!value) + return ERR_PTR(-ENOMEM); + memset(value + size, 0, extra); + if (copy_from_bpfptr(value, src, size)) { + kvfree(value); + return ERR_PTR(-EFAULT); + } + return value; +} + static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) { bpfptr_t ukey = make_bpfptr(attr->key, uattr.is_kernel); @@ -1808,7 +1918,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) } value_size = bpf_map_value_size(map, attr->flags); - value = kvmemdup_bpfptr(uvalue, value_size); + value = bpf_map_memdup_value(map, uvalue, value_size); if (IS_ERR(value)) { err = PTR_ERR(value); goto free_key; @@ -2021,7 +2131,9 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, if (!key) return -ENOMEM; - value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); + value = kvmalloc(value_size + bpf_obj_uptr_extra_size(map->record), + GFP_USER | __GFP_NOWARN | + (bpf_obj_uptr_extra_size(map->record) ? __GFP_ZERO : 0)); if (!value) { kvfree(key); return -ENOMEM; diff --git a/kernel/events/core.c b/kernel/events/core.c index 1a23ae67c3cc8f..4b1b0f8b15cf29 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -8,6 +8,7 @@ * Copyright © 2009 Paul Mackerras, IBM Corp. */ +#include #include #include #include @@ -6812,7 +6813,7 @@ static void perf_event_init_userpage(struct perf_event *event) /* Allow new userspace to detect that bit 0 is deprecated */ userpg->cap_bit0_is_deprecated = 1; userpg->size = offsetof(struct perf_event_mmap_page, __reserved); - userpg->data_offset = PAGE_SIZE; + userpg->data_offset = perf_metadata_size(rb); userpg->data_size = perf_data_size(rb); unlock: @@ -6997,6 +6998,11 @@ typedef void (*mapped_f)(struct perf_event *event, struct mm_struct *mm); f; \ }) +static u64 perf_mmap_offset(struct vm_area_struct *vma) +{ + return ((u64)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma); +} + static void perf_mmap_open(struct vm_area_struct *vma) { struct perf_event *event = vma->vm_file->private_data; @@ -7005,7 +7011,7 @@ static void perf_mmap_open(struct vm_area_struct *vma) refcount_inc(&event->mmap_count); refcount_inc(&event->rb->mmap_count); - if (vma->vm_pgoff) + if (perf_mmap_offset(vma)) refcount_inc(&event->rb->aux_mmap_count); if (mapped) @@ -7039,7 +7045,7 @@ static void perf_mmap_close(struct vm_area_struct *vma) * The AUX buffer is strictly a sub-buffer, serialize using aux_mutex * to avoid complications. */ - if (rb_has_aux(rb) && vma->vm_pgoff == rb->aux_pgoff && + if (rb_has_aux(rb) && perf_mmap_offset(vma) == rb->aux_offset && refcount_dec_and_mutex_lock(&rb->aux_mmap_count, &rb->aux_mutex)) { /* * Stop all AUX events that are writing to this buffer, @@ -7132,7 +7138,9 @@ static void perf_mmap_close(struct vm_area_struct *vma) static vm_fault_t perf_mmap_pfn_mkwrite(struct vm_fault *vmf) { /* The first page is the user control page, others are read-only. */ - return vmf->pgoff == 0 ? 0 : VM_FAULT_SIGBUS; + return !perf_mmap_offset(vmf->vma) && + vmf->address - vmf->vma->vm_start < mm_page_size(vmf->vma->vm_mm) ? + 0 : VM_FAULT_SIGBUS; } static int perf_mmap_may_split(struct vm_area_struct *vma, unsigned long addr) @@ -7153,7 +7161,8 @@ static const struct vm_operations_struct perf_mmap_vmops = { static int map_range(struct perf_buffer *rb, struct vm_area_struct *vma) { - unsigned long nr_pages = vma_pages(vma); + unsigned long va = vma->vm_start; + bool aux = perf_mmap_offset(vma) != 0; int err = 0; unsigned long pagenum; @@ -7197,9 +7206,21 @@ static int map_range(struct perf_buffer *rb, struct vm_area_struct *vma) * perf_mmap_open() and perf_mmap_close()) so we ensure the lifetime of * this mapping is maintained correctly. */ - for (pagenum = 0; pagenum < nr_pages; pagenum++) { - unsigned long va = vma->vm_start + PAGE_SIZE * pagenum; - struct page *page = perf_mmap_to_page(rb, vma->vm_pgoff + pagenum); + for (pagenum = 0; va < vma->vm_end; pagenum++) { + unsigned long size = min(PAGE_SIZE, vma->vm_end - va); + struct page *page; + + if (aux) { + if (pagenum >= rb->aux_nr_pages) { + err = -EINVAL; + break; + } + page = virt_to_page(rb->aux_pages[pagenum]); + } else { + page = perf_mmap_to_page(rb, pagenum); + if (!pagenum) + size = perf_metadata_size(rb); + } if (page == NULL) { err = -EINVAL; @@ -7207,16 +7228,17 @@ static int map_range(struct perf_buffer *rb, struct vm_area_struct *vma) } /* Map readonly, perf_mmap_pfn_mkwrite() called on write fault. */ - err = remap_pfn_range(vma, va, page_to_pfn(page), PAGE_SIZE, + err = remap_pfn_range(vma, va, page_to_pfn(page), size, vm_get_page_prot(vma->vm_flags & ~VM_SHARED)); if (err) break; + va += size; } #ifdef CONFIG_MMU /* Clear any partial mappings on error. */ if (err) - zap_vma_range(vma, vma->vm_start, nr_pages * PAGE_SIZE); + zap_vma_range(vma, vma->vm_start, vma->vm_end - vma->vm_start); #endif return err; @@ -7269,7 +7291,7 @@ static void perf_mmap_unaccount(struct vm_area_struct *vma, struct perf_buffer * { struct user_struct *user = rb->mmap_user; - atomic_long_sub((perf_data_size(rb) >> PAGE_SHIFT) + 1 - rb->mmap_locked, + atomic_long_sub(data_page_nr(rb) + 1 - rb->mmap_locked, &user->locked_vm); atomic64_sub(rb->mmap_locked, &vma->vm_mm->pinned_vm); } @@ -7277,7 +7299,9 @@ static void perf_mmap_unaccount(struct vm_area_struct *vma, struct perf_buffer * static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, unsigned long nr_pages) { - long extra = 0, user_extra = nr_pages; + unsigned int page_shift = mm_page_shift(vma->vm_mm); + unsigned long data_size; + long extra = 0, user_extra; struct perf_buffer *rb; int rb_flags = 0; @@ -7289,11 +7313,14 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, */ if (nr_pages != 0 && !is_power_of_2(nr_pages)) return -EINVAL; + data_size = nr_pages << page_shift; + /* Physical allocation and locked-memory accounting stay in native pages. */ + user_extra = DIV_ROUND_UP(data_size, PAGE_SIZE) + 1; WARN_ON_ONCE(event->ctx->parent_ctx); if (event->rb) { - if (data_page_nr(event->rb) != nr_pages) + if (perf_data_size(event->rb) != data_size) return -EINVAL; /* @@ -7329,7 +7356,7 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, if (vma->vm_flags & VM_WRITE) rb_flags |= RING_BUFFER_WRITABLE; - rb = rb_alloc(nr_pages, + rb = rb_alloc(data_size, page_shift, event->attr.watermark ? event->attr.wakeup_watermark : 0, event->cpu, rb_flags); @@ -7350,9 +7377,10 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, return 0; } -static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event, - unsigned long nr_pages) +static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event) { + unsigned long size = vma->vm_end - vma->vm_start; + unsigned long nr_pages = size >> PAGE_SHIFT; long extra = 0, user_extra = nr_pages; u64 aux_offset, aux_size; struct perf_buffer *rb; @@ -7361,6 +7389,9 @@ static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event, rb = event->rb; if (!rb) return -EINVAL; + /* PMU setup_aux() consumes whole native pages, independent of mmap layout. */ + if (!IS_ALIGNED(size, PAGE_SIZE)) + return -EINVAL; guard(mutex)(&rb->aux_mutex); @@ -7372,14 +7403,14 @@ static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event, aux_offset = READ_ONCE(rb->user_page->aux_offset); aux_size = READ_ONCE(rb->user_page->aux_size); - if (aux_offset < perf_data_size(rb) + PAGE_SIZE) + if (aux_offset < perf_data_size(rb) + perf_metadata_size(rb)) return -EINVAL; - if (aux_offset != vma->vm_pgoff << PAGE_SHIFT) + if (aux_offset != perf_mmap_offset(vma)) return -EINVAL; /* already mapped with a different offset */ - if (rb_has_aux(rb) && rb->aux_pgoff != vma->vm_pgoff) + if (rb_has_aux(rb) && rb->aux_offset != aux_offset) return -EINVAL; if (aux_size != nr_pages * PAGE_SIZE) @@ -7409,7 +7440,7 @@ static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event, if (vma->vm_flags & VM_WRITE) rb_flags |= RING_BUFFER_WRITABLE; - ret = rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, + ret = rb_alloc_aux(rb, event, aux_offset, nr_pages, event->attr.aux_watermark, rb_flags); if (ret) { refcount_dec(&rb->mmap_count); @@ -7449,12 +7480,13 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) return ret; vma_size = vma->vm_end - vma->vm_start; - nr_pages = vma_size / PAGE_SIZE; + nr_pages = vma_size >> mm_page_shift(vma->vm_mm); if (nr_pages > INT_MAX) return -ENOMEM; - if (vma_size != PAGE_SIZE * nr_pages) + if (vma_size != mm_page_size(vma->vm_mm) * nr_pages || + mm_page_shift(vma->vm_mm) > PAGE_SHIFT) return -EINVAL; scoped_guard (mutex, &event->mmap_mutex) { @@ -7466,10 +7498,14 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) if (event->state <= PERF_EVENT_STATE_REVOKED) return -ENODEV; - if (vma->vm_pgoff == 0) + /* One ring has one immutable metadata/data boundary for all aliases. */ + if (event->rb && event->rb->mmap_page_shift != mm_page_shift(vma->vm_mm)) + return -EINVAL; + + if (!perf_mmap_offset(vma)) ret = perf_mmap_rb(vma, event, nr_pages); else - ret = perf_mmap_aux(vma, event, nr_pages); + ret = perf_mmap_aux(vma, event); if (ret) return ret; @@ -8456,7 +8492,7 @@ static u64 perf_virt_to_phys(u64 virt) /* * Return the pagetable size of a given virtual address. */ -static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) +VISIBLE_IF_KUNIT u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) { u64 size = 0; @@ -8473,45 +8509,46 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return 0; if (pgd_leaf(pgd)) - return pgd_leaf_size(pgd); + return pgd_leaf_size_mm(mm, pgd); - p4dp = p4d_offset_lockless(pgdp, pgd, addr); + p4dp = p4d_offset_lockless_mm(mm, pgdp, pgd, addr); p4d = p4dp_get(p4dp); - if (!p4d_present(p4d)) + if (!p4d_present_mm(mm, p4d)) return 0; if (p4d_leaf(p4d)) - return p4d_leaf_size(p4d); + return p4d_leaf_size_mm(mm, p4d); - pudp = pud_offset_lockless(p4dp, p4d, addr); + pudp = pud_offset_lockless_mm(mm, p4dp, p4d, addr); pud = pudp_get(pudp); if (!pud_present(pud)) return 0; if (pud_leaf(pud)) - return pud_leaf_size(pud); + return pud_leaf_size_mm(mm, pud); - pmdp = pmd_offset_lockless(pudp, pud, addr); + pmdp = pmd_offset_lockless_mm(mm, pudp, pud, addr); again: pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) return 0; if (pmd_leaf(pmd)) - return pmd_leaf_size(pmd); + return pmd_leaf_size_mm(mm, pmd); - ptep = pte_offset_map(&pmd, addr); + ptep = pte_offset_map_mm(mm, &pmd, addr); if (!ptep) goto again; pte = ptep_get_lockless(ptep); if (pte_present(pte)) - size = __pte_leaf_size(pmd, pte); + size = __pte_leaf_size_mm(mm, pmd, pte); pte_unmap(ptep); #endif /* CONFIG_HAVE_GUP_FAST */ return size; } +EXPORT_SYMBOL_IF_KUNIT(perf_get_pgtable_size); static u64 perf_get_page_size(unsigned long addr) { diff --git a/kernel/events/internal.h b/kernel/events/internal.h index c03c4f2eea5718..5238cfa3399b1a 100644 --- a/kernel/events/internal.h +++ b/kernel/events/internal.h @@ -15,9 +15,10 @@ struct perf_buffer { struct rcu_head rcu_head; #ifdef CONFIG_PERF_USE_VMALLOC struct work_struct work; - int page_order; /* allocation order */ #endif int nr_pages; /* nr of data pages */ + unsigned int data_page_shift; /* bytes per output chunk */ + unsigned int mmap_page_shift; /* userspace metadata size */ int overwrite; /* can overwrite itself */ int paused; /* can write into ring buffer */ @@ -44,7 +45,7 @@ struct perf_buffer { long aux_head; unsigned int aux_nest; long aux_wakeup; /* last aux_watermark boundary crossed by aux_head */ - unsigned long aux_pgoff; + u64 aux_offset; int aux_nr_pages; int aux_overwrite; refcount_t aux_mmap_count; @@ -80,10 +81,11 @@ static inline void rb_toggle_paused(struct perf_buffer *rb, bool pause) } extern struct perf_buffer * -rb_alloc(int nr_pages, long watermark, int cpu, int flags); +rb_alloc(unsigned long data_size, unsigned int mmap_page_shift, + long watermark, int cpu, int flags); extern void perf_event_wakeup(struct perf_event *event); extern int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event, - pgoff_t pgoff, int nr_pages, long watermark, int flags); + u64 offset, int nr_pages, long watermark, int flags); extern void rb_free_aux(struct perf_buffer *rb); extern struct perf_buffer *ring_buffer_get(struct perf_event *event); extern void ring_buffer_put(struct perf_buffer *rb); @@ -99,34 +101,19 @@ void perf_event_aux_event(struct perf_event *event, unsigned long head, extern struct page * perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff); -#ifdef CONFIG_PERF_USE_VMALLOC -/* - * Back perf_mmap() with vmalloc memory. - * - * Required for architectures that have d-cache aliasing issues. - */ - -static inline int page_order(struct perf_buffer *rb) -{ - return rb->page_order; -} - -#else - -static inline int page_order(struct perf_buffer *rb) +static inline unsigned long perf_data_size(struct perf_buffer *rb) { - return 0; + return (unsigned long)rb->nr_pages << rb->data_page_shift; } -#endif static inline int data_page_nr(struct perf_buffer *rb) { - return rb->nr_pages << page_order(rb); + return DIV_ROUND_UP(perf_data_size(rb), PAGE_SIZE); } -static inline unsigned long perf_data_size(struct perf_buffer *rb) +static inline unsigned long perf_metadata_size(struct perf_buffer *rb) { - return rb->nr_pages << (PAGE_SHIFT + page_order(rb)); + return 1UL << rb->mmap_page_shift; } static inline unsigned long perf_aux_size(struct perf_buffer *rb) @@ -154,7 +141,7 @@ static inline unsigned long perf_aux_size(struct perf_buffer *rb) handle->page++; \ handle->page &= rb->nr_pages - 1; \ handle->addr = rb->data_pages[handle->page]; \ - handle->size = PAGE_SIZE << page_order(rb); \ + handle->size = 1UL << rb->data_page_shift; \ } \ } while (len && written == size); \ \ diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index 9fe92161715e09..6c5de8fba83f23 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -238,7 +238,7 @@ __perf_output_begin(struct perf_output_handle *handle, if (unlikely(head - local_read(&rb->wakeup) > rb->watermark)) local_add(rb->watermark, &rb->wakeup); - page_shift = PAGE_SHIFT + page_order(rb); + page_shift = rb->data_page_shift; handle->page = (offset >> page_shift) & (rb->nr_pages - 1); offset &= (1UL << page_shift) - 1; @@ -677,7 +677,7 @@ static void __rb_free_aux(struct perf_buffer *rb) } int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event, - pgoff_t pgoff, int nr_pages, long watermark, int flags) + u64 offset, int nr_pages, long watermark, int flags) { bool overwrite = !(flags & RING_BUFFER_WRITABLE); int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu); @@ -784,7 +784,7 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event, out: if (!ret) - rb->aux_pgoff = pgoff; + rb->aux_offset = offset; else __rb_free_aux(rb); @@ -803,8 +803,8 @@ void rb_free_aux(struct perf_buffer *rb) * Back perf_mmap() with regular GFP_KERNEL-0 pages. */ -static struct page * -__perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) +struct page * +perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) { if (pgoff > rb->nr_pages) return NULL; @@ -835,10 +835,12 @@ static void perf_mmap_free_page(void *addr) __free_page(page); } -struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) +struct perf_buffer *rb_alloc(unsigned long data_size, unsigned int mmap_page_shift, + long watermark, int cpu, int flags) { struct perf_buffer *rb; unsigned long size; + int nr_pages = DIV_ROUND_UP(data_size, PAGE_SIZE); int i, node; size = sizeof(struct perf_buffer); @@ -863,6 +865,10 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) } rb->nr_pages = nr_pages; + rb->mmap_page_shift = mmap_page_shift; + /* A ring smaller than a native page still wraps at its requested size. */ + rb->data_page_shift = data_size && data_size < PAGE_SIZE ? + ilog2(data_size) : PAGE_SHIFT; ring_buffer_init(rb, watermark, flags); @@ -892,8 +898,8 @@ void rb_free(struct perf_buffer *rb) } #else -static struct page * -__perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) +struct page * +perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) { /* The '>' counts in the user page. */ if (pgoff > data_page_nr(rb)) @@ -917,10 +923,12 @@ void rb_free(struct perf_buffer *rb) schedule_work(&rb->work); } -struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) +struct perf_buffer *rb_alloc(unsigned long data_size, unsigned int mmap_page_shift, + long watermark, int cpu, int flags) { struct perf_buffer *rb; unsigned long size; + int nr_pages = DIV_ROUND_UP(data_size, PAGE_SIZE); void *all_buf; int node; @@ -940,9 +948,10 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) rb->user_page = all_buf; rb->data_pages[0] = all_buf + PAGE_SIZE; + rb->mmap_page_shift = mmap_page_shift; if (nr_pages) { rb->nr_pages = 1; - rb->page_order = ilog2(nr_pages); + rb->data_page_shift = ilog2(data_size); } ring_buffer_init(rb, watermark, flags); @@ -957,21 +966,3 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) } #endif - -struct page * -perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) -{ - if (rb->aux_nr_pages) { - /* above AUX space */ - if (pgoff > rb->aux_pgoff + rb->aux_nr_pages) - return NULL; - - /* AUX space */ - if (pgoff >= rb->aux_pgoff) { - int aux_pgoff = array_index_nospec(pgoff - rb->aux_pgoff, rb->aux_nr_pages); - return virt_to_page(rb->aux_pages[aux_pgoff]); - } - } - - return __perf_mmap_to_page(rb, pgoff); -} diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 6300b216012cbe..0d8dc2ae9fdd57 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -30,6 +30,7 @@ #include #include /* check_stable_address_space */ #include +#include #include @@ -144,12 +145,14 @@ static bool valid_vma(struct vm_area_struct *vma, bool is_register) static unsigned long offset_to_vaddr(struct vm_area_struct *vma, loff_t offset) { - return vma->vm_start + offset - ((loff_t)vma->vm_pgoff << PAGE_SHIFT); + return vma->vm_start + offset - ((loff_t)vma->vm_pgoff << PAGE_SHIFT) - + vma_subpage_offset(vma); } static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) { - return ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + (vaddr - vma->vm_start); + return ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma) + + (vaddr - vma->vm_start); } /** @@ -191,7 +194,8 @@ static void copy_to_page(struct page *page, unsigned long vaddr, const void *src kunmap_local(kaddr); } -static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t *insn, +static int verify_opcode(struct page *page, unsigned long vaddr, unsigned int page_offset, + uprobe_opcode_t *insn, int nbytes, void *data) { uprobe_opcode_t old_opcode; @@ -206,7 +210,7 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t * is a trap variant; uprobes always wins over any other (gdb) * breakpoint. */ - uprobe_copy_from_page(page, vaddr, &old_opcode, UPROBE_SWBP_INSN_SIZE); + uprobe_copy_from_page(page, page_offset, &old_opcode, UPROBE_SWBP_INSN_SIZE); is_swbp = is_swbp_insn(&old_opcode); if (is_swbp_insn(insn)) { @@ -307,14 +311,24 @@ __update_ref_ctr(struct mm_struct *mm, unsigned long vaddr, short d) { void *kaddr; struct page *page; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment; + struct vm_area_struct *vma; + bool alternative = mm_page_size(mm) != PAGE_SIZE; +#endif int ret; short *ptr; if (!vaddr || !d) return -EINVAL; - ret = get_user_pages_remote(mm, vaddr, 1, - FOLL_WRITE, &page, NULL); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) + ret = get_user_fragment_vma_remote(mm, vaddr, sizeof(short), + FOLL_WRITE, &fragment, &vma); + else +#endif + ret = get_user_pages_remote(mm, vaddr, 1, FOLL_WRITE, &page, NULL); if (unlikely(ret <= 0)) { /* * We are asking for 1 page. If get_user_pages_remote() fails, @@ -323,8 +337,16 @@ __update_ref_ctr(struct mm_struct *mm, unsigned long vaddr, short d) return ret == 0 ? -EBUSY : ret; } - kaddr = kmap_local_page(page); - ptr = kaddr + (vaddr & ~PAGE_MASK); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) { + kaddr = kmap_local_folio(fragment.folio, fragment.offset); + ptr = kaddr; + } else +#endif + { + kaddr = kmap_local_page(page); + ptr = kaddr + (vaddr & ~PAGE_MASK); + } if (unlikely(*ptr + d < 0)) { pr_warn("ref_ctr going negative. vaddr: 0x%lx, " @@ -337,7 +359,12 @@ __update_ref_ctr(struct mm_struct *mm, unsigned long vaddr, short d) ret = 0; out: kunmap_local(kaddr); - put_page(page); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) + release_user_fragments(&fragment, 1, !ret); + else +#endif + put_page(page); return ret; } @@ -381,7 +408,8 @@ static int update_ref_ctr(struct uprobe *uprobe, struct mm_struct *mm, } static bool orig_page_is_identical(struct vm_area_struct *vma, - unsigned long vaddr, struct page *page, bool *pmd_mappable) + unsigned long vaddr, struct page *page, unsigned int page_offset, + bool *pmd_mappable) { const pgoff_t index = vaddr_to_offset(vma, vaddr) >> PAGE_SHIFT; struct folio *orig_folio = filemap_get_folio(vma->vm_file->f_mapping, @@ -393,19 +421,28 @@ static bool orig_page_is_identical(struct vm_area_struct *vma, return false; orig_page = folio_file_page(orig_folio, index); - *pmd_mappable = folio_test_pmd_mappable(orig_folio); - identical = folio_test_uptodate(orig_folio) && - pages_identical(page, orig_page); + *pmd_mappable = mm_page_size(vma->vm_mm) == PAGE_SIZE && + folio_test_pmd_mappable(orig_folio); + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + unsigned int offset = vma_page_offset_at(vma, vaddr).offset; + + identical = folio_test_uptodate(orig_folio) && + !memcmp_pages_range(page, page_offset, orig_page, offset, + mm_page_size(vma->vm_mm)); + } else { + identical = folio_test_uptodate(orig_folio) && + pages_identical(page, orig_page); + } folio_put(orig_folio); return identical; } static int __uprobe_write(struct vm_area_struct *vma, - struct folio_walk *fw, struct folio *folio, - unsigned long insn_vaddr, uprobe_opcode_t *insn, int nbytes, + struct folio_walk *fw, struct folio *folio, struct mm_subpage *slot, + unsigned int page_offset, unsigned long insn_vaddr, uprobe_opcode_t *insn, int nbytes, bool is_register) { - const unsigned long vaddr = insn_vaddr & PAGE_MASK; + const unsigned long vaddr = insn_vaddr & mm_page_mask(vma->vm_mm); bool pmd_mappable; /* For now, we'll only handle PTE-mapped folios. */ @@ -417,6 +454,12 @@ static int __uprobe_write(struct vm_area_struct *vma, * but the VMA might not be writable. */ if (!pte_write(fw->pte)) { +#ifdef CONFIG_MM_SUBPAGE + if (slot) { + if (!mm_subpage_is_exclusive(slot)) + return -EFAULT; + } else +#endif if (!PageAnonExclusive(fw->page)) return -EFAULT; if (unlikely(userfaultfd_pte_wp(vma, fw->pte))) @@ -430,31 +473,49 @@ static int __uprobe_write(struct vm_area_struct *vma, */ flush_cache_page(vma, vaddr, pte_pfn(fw->pte)); fw->pte = ptep_clear_flush(vma, vaddr, fw->ptep); - copy_to_page(fw->page, insn_vaddr, insn, nbytes); + copy_to_page(fw->page, page_offset, insn, nbytes); /* * When unregistering, we may only zap a PTE if uffd is disabled and * there are no unexpected folio references ... */ - if (is_register || userfaultfd_missing(vma) || - (folio_ref_count(folio) != folio_expected_ref_count(folio) + 1)) + if (is_register || userfaultfd_missing(vma)) + goto remap; +#ifdef CONFIG_MM_SUBPAGE + if (slot) { + if (!mm_subpage_can_zap(slot)) + goto remap; + } else +#endif + if (folio_ref_count(folio) != folio_expected_ref_count(folio) + 1) goto remap; /* * ... and the mapped page is identical to the original page that * would get faulted in on next access. */ - if (!orig_page_is_identical(vma, vaddr, fw->page, &pmd_mappable)) + if (!orig_page_is_identical(vma, vaddr, fw->page, + page_offset - (insn_vaddr - vaddr), &pmd_mappable)) goto remap; dec_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_remove_rmap_pte(folio, fw->page, vma); +#ifdef CONFIG_MM_SUBPAGE + if (slot) + mm_subpage_remove_anon_rmap(slot, vma); + else +#endif + folio_remove_rmap_pte(folio, fw->page, vma); if (!folio_mapped(folio) && folio_test_swapcache(folio) && folio_trylock(folio)) { folio_free_swap(folio); folio_unlock(folio); } - folio_put(folio); +#ifdef CONFIG_MM_SUBPAGE + if (slot) + mm_subpage_put(slot); /* Retired PTE reference; caller still has its GET. */ + else +#endif + folio_put(folio); return pmd_mappable; remap: @@ -494,13 +555,24 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, verify_opcode, is_register, true /* do_update_ref_ctr */, NULL); } +static void uprobe_put_folio(struct folio *folio, struct mm_subpage *slot) +{ +#ifdef CONFIG_MM_SUBPAGE + if (slot) { + mm_subpage_put(slot); + return; + } +#endif + folio_put(folio); +} + int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, const unsigned long insn_vaddr, uprobe_opcode_t *insn, int nbytes, uprobe_write_verify_t verify, bool is_register, bool do_update_ref_ctr, void *data) { - const unsigned long vaddr = insn_vaddr & PAGE_MASK; struct mm_struct *mm = vma->vm_mm; + const unsigned long vaddr = insn_vaddr & mm_page_mask(mm); struct uprobe *uprobe; int ret, ref_ctr_updated = 0; unsigned int gup_flags = FOLL_FORCE; @@ -508,7 +580,16 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, struct folio_walk fw; struct folio *folio; struct page *page; - + struct mm_subpage *slot = NULL; + unsigned int page_offset; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment; + struct vm_area_struct *found; +#endif + + /* Larger-than-native public mms are not enabled by the mixed-page ABI yet. */ + if (mm_page_size(mm) > PAGE_SIZE) + return -EOPNOTSUPP; uprobe = container_of(auprobe, struct uprobe, arch); if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags))) @@ -526,14 +607,29 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, gup_flags |= FOLL_WRITE | FOLL_SPLIT_PMD; retry: - ret = get_user_pages_remote(mm, vaddr, 1, gup_flags, &page, NULL); - if (ret <= 0) - goto out; - folio = page_folio(page); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + ret = get_user_fragment_vma_remote(mm, vaddr, mm_page_size(mm), + gup_flags & ~FOLL_SPLIT_PMD, &fragment, &found); + if (ret <= 0) + goto out; + folio = fragment.folio; + slot = fragment.subpage; + page = folio_page(folio, fragment.offset >> PAGE_SHIFT); + page_offset = offset_in_page(fragment.offset) + insn_vaddr - vaddr; + } else +#endif + { + ret = get_user_pages_remote(mm, vaddr, 1, gup_flags, &page, NULL); + if (ret <= 0) + goto out; + folio = page_folio(page); + page_offset = offset_in_page(insn_vaddr); + } - ret = verify(page, insn_vaddr, insn, nbytes, data); + ret = verify(page, insn_vaddr, page_offset, insn, nbytes, data); if (ret <= 0) { - folio_put(folio); + uprobe_put_folio(folio, slot); goto out; } @@ -541,7 +637,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, if (do_update_ref_ctr && !ref_ctr_updated && uprobe->ref_ctr_offset) { ret = update_ref_ctr(uprobe, mm, is_register ? 1 : -1); if (ret) { - folio_put(folio); + uprobe_put_folio(folio, slot); goto out; } @@ -551,7 +647,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, ret = 0; if (unlikely(!folio_test_anon(folio) || folio_is_zone_device(folio))) { VM_WARN_ON_ONCE(is_register); - folio_put(folio); + uprobe_put_folio(folio, slot); goto out; } @@ -562,22 +658,29 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, * be able to do it under PTL. */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, - vaddr, vaddr + PAGE_SIZE); + vaddr, vaddr + mm_page_size(mm)); mmu_notifier_invalidate_range_start(&range); } ret = -EAGAIN; /* Walk the page tables again, to perform the actual update. */ if (folio_walk_start(&fw, vma, vaddr, 0)) { - if (fw.page == page) - ret = __uprobe_write(vma, &fw, folio, insn_vaddr, insn, nbytes, is_register); + bool same = fw.page == page; +#ifdef CONFIG_MM_SUBPAGE + if (slot) + same &= fw.level == FW_LEVEL_PTE && + pte_phys_mm(mm, fw.pte) == mm_subpage_phys(slot); +#endif + if (same) + ret = __uprobe_write(vma, &fw, folio, slot, page_offset, + insn_vaddr, insn, nbytes, is_register); folio_walk_end(&fw, vma); } if (!is_register) mmu_notifier_invalidate_range_end(&range); - folio_put(folio); + uprobe_put_folio(folio, slot); switch (ret) { case -EFAULT: gup_flags |= FOLL_WRITE | FOLL_SPLIT_PMD; @@ -1211,7 +1314,9 @@ build_map_info(struct address_space *mapping, loff_t offset, bool is_register) again: i_mmap_lock_read(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) { - if (!valid_vma(vma, is_register)) + if (!valid_vma(vma, is_register) || + offset < vaddr_to_offset(vma, vma->vm_start) || + offset >= vaddr_to_offset(vma, vma->vm_end)) continue; if (!prev && !more) { @@ -1482,7 +1587,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) file_inode(vma->vm_file) != uprobe->inode) continue; - offset = (loff_t)vma->vm_pgoff << PAGE_SHIFT; + offset = vaddr_to_offset(vma, vma->vm_start); if (uprobe->offset < offset || uprobe->offset >= offset + vma->vm_end - vma->vm_start) continue; @@ -1696,8 +1801,14 @@ static const struct vm_special_mapping xol_mapping = { unsigned long __weak arch_uprobe_get_xol_area(void) { - /* Try to map as high as possible, this is only a hint. */ - return get_unmapped_area(NULL, TASK_SIZE - PAGE_SIZE, PAGE_SIZE, 0, 0); + unsigned long size = PAGE_SIZE; + unsigned long addr; + + /* XOL arch copy hooks use native offsets into the one backing page. */ + if (mm_page_size(current->mm) < PAGE_SIZE) + size += PAGE_SIZE - mm_page_size(current->mm); + addr = get_unmapped_area(NULL, TASK_SIZE - size, size, 0, 0); + return IS_ERR_VALUE(addr) ? addr : ALIGN(addr, PAGE_SIZE); } /* Slot allocation for XOL */ @@ -2413,6 +2524,23 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) if (likely(result == 0)) goto out; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + struct user_page_fragment fragment; + struct vm_area_struct *vma; + void *kaddr; + + result = get_user_fragment_vma_remote(mm, vaddr, UPROBE_SWBP_INSN_SIZE, + FOLL_FORCE, &fragment, &vma); + if (result != 1) + return result < 0 ? result : -EFAULT; + kaddr = kmap_local_folio(fragment.folio, fragment.offset); + memcpy(&opcode, kaddr, UPROBE_SWBP_INSN_SIZE); + kunmap_local(kaddr); + release_user_fragments(&fragment, 1, false); + goto out; + } +#endif result = get_user_pages(vaddr, 1, FOLL_FORCE, &page); if (result < 0) return result; @@ -2453,7 +2581,7 @@ static struct uprobe *find_active_uprobe_speculative(unsigned long bp_vaddr) if (!vm_file) return NULL; - offset = (loff_t)(vma->vm_pgoff << PAGE_SHIFT) + (bp_vaddr - vma->vm_start); + offset = vaddr_to_offset(vma, bp_vaddr); uprobe = find_uprobe_rcu(vm_file->f_inode, offset); if (!uprobe) return NULL; diff --git a/kernel/fork.c b/kernel/fork.c index 5a4623b058a20b..a9fedd8b8abacb 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -112,7 +113,7 @@ #include #include -#include +#include #include #include @@ -639,6 +640,12 @@ static void check_mm(struct mm_struct *mm) } } +#ifdef CONFIG_MM_SUBPAGE + if (mm->committed_user_pages) + pr_alert("BUG: non-zero user commitment on freeing mm: %lu\n", + mm->committed_user_pages); +#endif + if (mm_pgtables_bytes(mm)) pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", mm_pgtables_bytes(mm)); @@ -1089,6 +1096,10 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); mm_pgtables_bytes_init(mm); mm->map_count = 0; +#ifdef CONFIG_MM_SUBPAGE + mm->committed_user_pages = 0; + mm->cow_subpage_pool = NULL; +#endif mm->locked_vm = 0; atomic64_set(&mm->pinned_vm, 0); memset(&mm->rss_stat, 0, sizeof(mm->rss_stat)); @@ -1159,7 +1170,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, /* * Allocate and initialize an mm_struct. */ -struct mm_struct *mm_alloc(void) +static struct mm_struct *__mm_alloc(bool for_exec, unsigned int page_shift) { struct mm_struct *mm; @@ -1168,10 +1179,32 @@ struct mm_struct *mm_alloc(void) return NULL; memset(mm, 0, sizeof(*mm)); + if (for_exec) + arch_mm_init_exec(mm, page_shift); return mm_init(mm, current, current_user_ns()); } + +struct mm_struct *mm_alloc(void) +{ + return __mm_alloc(false, 0); +} EXPORT_SYMBOL_IF_KUNIT(mm_alloc); +struct mm_struct *mm_alloc_exec(void) +{ +#ifdef CONFIG_ARCH_HAS_USER_PAGE_SIZE + return __mm_alloc(true, current->exec_page_shift); +#else + return __mm_alloc(true, 0); +#endif +} + +/* The caller has validated the requested exec granule. */ +struct mm_struct *mm_alloc_exec_page_shift(unsigned int page_shift) +{ + return __mm_alloc(true, page_shift); +} + static inline void __mmput(struct mm_struct *mm) { VM_BUG_ON(atomic_read(&mm->mm_users)); @@ -1181,6 +1214,7 @@ static inline void __mmput(struct mm_struct *mm) ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ exit_mmap(mm); + mm_subpage_cow_pool_exit(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); if (!list_empty(&mm->mmlist)) { diff --git a/kernel/futex/core.c b/kernel/futex/core.c index 7ffe6dc79b28e8..771c12983cd2a7 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include "futex.h" @@ -512,6 +513,44 @@ static u64 get_inode_sequence_number(struct inode *inode) } } +/* Keep a typed slot reference alive until the mapping identity is sampled. */ +struct futex_page_ref { + struct page *page; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment; +#endif +}; + +static int futex_get_page_ref(struct mm_struct *mm, unsigned long address, + unsigned long uaddr, unsigned int size, unsigned int flags, + struct futex_page_ref *ref) +{ +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) != PAGE_SIZE) { + int ret = get_user_fragments_remote(mm, uaddr, size, flags, + &ref->fragment, 1); + + if (ret != 1) + return ret ?: -EFAULT; + ref->page = folio_page(ref->fragment.folio, + ref->fragment.offset >> PAGE_SHIFT); + return 1; + } +#endif + return get_user_pages_fast(address, 1, flags, &ref->page); +} + +static void futex_put_page_ref(struct futex_page_ref *ref) +{ +#ifdef CONFIG_MM_SUBPAGE + if (ref->fragment.folio) { + release_user_fragments(&ref->fragment, 1, false); + return; + } +#endif + put_page(ref->page); +} + /** * get_futex_key() - Get parameters which are the keys for a futex * @uaddr: virtual address of the futex @@ -545,6 +584,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, unsigned long address = (unsigned long)uaddr; struct mm_struct *mm = current->mm; struct page *page; + struct futex_page_ref ref = {}; struct folio *folio; struct address_space *mapping; int node, err, size, ro = 0; @@ -629,13 +669,13 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, if (unlikely(should_fail_futex(true))) return -EFAULT; - err = get_user_pages_fast(address, 1, FOLL_WRITE, &page); + err = futex_get_page_ref(mm, address, (unsigned long)uaddr, size, FOLL_WRITE, &ref); /* * If write access is not required (eg. FUTEX_WAIT), try * and get read-only access. */ if (err == -EFAULT && rw == FUTEX_READ) { - err = get_user_pages_fast(address, 1, 0, &page); + err = futex_get_page_ref(mm, address, (unsigned long)uaddr, size, 0, &ref); ro = 1; } if (err < 0) @@ -660,6 +700,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, * filesystem-backed pages, the precise page is required as the * index of the page determines the key. */ + page = ref.page; folio = page_folio(page); mapping = READ_ONCE(folio->mapping); @@ -689,7 +730,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, folio_lock(folio); shmem_swizzled = folio_test_swapcache(folio) || folio->mapping; folio_unlock(folio); - folio_put(folio); + futex_put_page_ref(&ref); if (shmem_swizzled) goto again; @@ -739,7 +780,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, if (READ_ONCE(folio->mapping) != mapping) { rcu_read_unlock(); - folio_put(folio); + futex_put_page_ref(&ref); goto again; } @@ -747,11 +788,15 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, inode = READ_ONCE(mapping->host); if (!inode) { rcu_read_unlock(); - folio_put(folio); + futex_put_page_ref(&ref); goto again; } +#ifdef CONFIG_MM_SUBPAGE + if (ref.fragment.folio) + key->both.offset = offset_in_page(ref.fragment.offset); +#endif key->both.offset |= FUT_OFF_INODE; /* inode-based key */ key->shared.i_seq = get_inode_sequence_number(inode); key->shared.pgoff = page_pgoff(folio, page); @@ -759,7 +804,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key, } out: - folio_put(folio); + futex_put_page_ref(&ref); return err; } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index abb76775aa7287..e06b05a08f7f4b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1694,7 +1694,7 @@ static unsigned int task_nr_scan_windows(struct task_struct *p) * on resident pages */ nr_scan_pages = MB_TO_PAGES(sysctl_numa_balancing_scan_size); - rss = get_mm_rss(p->mm); + rss = mm_pages_to_native(p->mm, get_mm_rss(p->mm)); if (!rss) rss = nr_scan_pages; @@ -3682,8 +3682,8 @@ static void task_numa_work(struct callback_head *work) * areas faster. */ if (nr_pte_updates) - pages -= (end - start) >> PAGE_SHIFT; - virtpages -= (end - start) >> PAGE_SHIFT; + pages -= DIV_ROUND_UP(end - start, PAGE_SIZE); + virtpages -= DIV_ROUND_UP(end - start, PAGE_SIZE); start = end; if (pages <= 0 || virtpages <= 0) diff --git a/kernel/sys.c b/kernel/sys.c index ec432b89e85406..09702e8f773ba0 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -1927,7 +1927,7 @@ void getrusage(struct task_struct *p, int who, struct rusage *r) } out_children: - r->ru_maxrss = maxrss * (PAGE_SIZE / 1024); /* convert pages to KBs */ + r->ru_maxrss = maxrss; /* signal and mm high-water maxima are in kB */ r->ru_utime = ns_to_kernel_old_timeval(utime); r->ru_stime = ns_to_kernel_old_timeval(stime); } @@ -2532,6 +2532,40 @@ static int prctl_set_thp_disable(bool thp_disable, unsigned long flags, return 0; } +static int prctl_page_size_get(struct task_struct *task, bool persistent) +{ +#ifdef CONFIG_ARCH_HAS_USER_PAGE_SIZE + unsigned int shift = persistent ? task->default_exec_page_shift : + task->exec_page_shift; + + return shift ? 1U << shift : 0; +#else + return -EINVAL; +#endif +} + +static int prctl_page_size_set(struct task_struct *task, unsigned long size, + bool persistent) +{ +#ifdef CONFIG_ARCH_HAS_USER_PAGE_SIZE + int shift = arch_exec_page_size_shift(size); + + if (shift < 0) + return shift; + if (persistent) { + task->default_exec_page_shift = shift; + } else { + /* A native request overrides an inherited default for one exec. */ + if (!shift && task->default_exec_page_shift) + shift = PAGE_SHIFT; + task->exec_page_shift = shift; + } + return 0; +#else + return -EINVAL; +#endif +} + int __weak arch_prctl_mem_model_get(struct task_struct *t) { return -EINVAL; @@ -2555,6 +2589,22 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = 0; switch (option) { + case PR_AURORA_GET_EXEC_PAGE_SIZE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + return prctl_page_size_get(me, false); + case PR_AURORA_SET_EXEC_PAGE_SIZE: + if (arg3 || arg4 || arg5) + return -EINVAL; + return prctl_page_size_set(me, arg2, false); + case PR_AURORA_GET_DEFAULT_PAGE_SIZE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + return prctl_page_size_get(me, true); + case PR_AURORA_SET_DEFAULT_PAGE_SIZE: + if (arg3 || arg4 || arg5) + return -EINVAL; + return prctl_page_size_set(me, arg2, true); case PR_GET_MEM_MODEL: if (arg2 || arg3 || arg4 || arg5) return -EINVAL; diff --git a/kernel/trace/trace_events_user.c b/kernel/trace/trace_events_user.c index 8c82ecb735f415..e8b622a8178cb7 100644 --- a/kernel/trace/trace_events_user.c +++ b/kernel/trace/trace_events_user.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include "trace_dynevent.h" @@ -554,6 +555,11 @@ static int user_event_enabler_write(struct user_event_mm *mm, unsigned long uaddr = enabler->addr; unsigned long *ptr; struct page *page; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment; + struct vm_area_struct *vma; + bool alternative = mm_page_size(mm->mm) != PAGE_SIZE; +#endif void *kaddr; int bit = ENABLE_BIT(enabler); int ret; @@ -573,8 +579,14 @@ static int user_event_enabler_write(struct user_event_mm *mm, align_addr_bit(&uaddr, &bit, ENABLE_BITOPS(enabler)); - ret = pin_user_pages_remote(mm->mm, uaddr, 1, FOLL_WRITE | FOLL_NOFAULT, - &page, NULL); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) + ret = pin_user_fragment_vma_remote(mm->mm, uaddr, sizeof(long), + FOLL_WRITE | FOLL_NOFAULT, &fragment, &vma); + else +#endif + ret = pin_user_pages_remote(mm->mm, uaddr, 1, FOLL_WRITE | FOLL_NOFAULT, + &page, NULL); if (unlikely(ret <= 0)) { if (!fixup_fault) @@ -586,8 +598,17 @@ static int user_event_enabler_write(struct user_event_mm *mm, return -EFAULT; } - kaddr = kmap_local_page(page); - ptr = kaddr + (uaddr & ~PAGE_MASK); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) { + /* align_addr_bit() keeps the complete atomic word in this fragment. */ + kaddr = kmap_local_folio(fragment.folio, fragment.offset); + ptr = kaddr; + } else +#endif + { + kaddr = kmap_local_page(page); + ptr = kaddr + (uaddr & ~PAGE_MASK); + } /* Update bit atomically, user tracers must be atomic as well */ if (enabler->event && enabler->event->status) @@ -596,7 +617,12 @@ static int user_event_enabler_write(struct user_event_mm *mm, clear_bit(bit, ptr); kunmap_local(kaddr); - unpin_user_pages_dirty_lock(&page, 1, true); +#ifdef CONFIG_MM_SUBPAGE + if (alternative) + release_user_fragments(&fragment, 1, true); + else +#endif + unpin_user_pages_dirty_lock(&page, 1, true); return 0; } diff --git a/kernel/tsacct.c b/kernel/tsacct.c index 5c153106e64271..5f59d071ab5655 100644 --- a/kernel/tsacct.c +++ b/kernel/tsacct.c @@ -91,16 +91,16 @@ void xacct_add_tsk(struct taskstats *stats, struct task_struct *p) { struct mm_struct *mm; - /* convert pages-nsec/1024 to Mbyte-usec, see __acct_update_integrals */ - stats->coremem = p->acct_rss_mem1 * PAGE_SIZE; + /* convert account-units-nsec/1024 to Mbyte-usec, see __acct_update_integrals */ + stats->coremem = p->acct_rss_mem1 * (1UL << MM_ACCOUNT_SHIFT); do_div(stats->coremem, 1000 * KB); - stats->virtmem = p->acct_vm_mem1 * PAGE_SIZE; + stats->virtmem = p->acct_vm_mem1 * (1UL << MM_ACCOUNT_SHIFT); do_div(stats->virtmem, 1000 * KB); mm = get_task_mm(p); if (mm) { /* adjust to KB unit */ - stats->hiwater_rss = get_mm_hiwater_rss(mm) * PAGE_SIZE / KB; - stats->hiwater_vm = get_mm_hiwater_vm(mm) * PAGE_SIZE / KB; + stats->hiwater_rss = mm_pages_to_kb(mm, get_mm_hiwater_rss(mm)); + stats->hiwater_vm = mm_pages_to_kb(mm, get_mm_hiwater_vm(mm)); mmput(mm); } stats->read_char = p->ioac.rchar & KB_MASK; @@ -140,8 +140,10 @@ static void __acct_update_integrals(struct task_struct *tsk, * The final unit reported to userspace is Mbyte-usecs, * the rest of the math is done in xacct_add_tsk. */ - tsk->acct_rss_mem1 += delta * get_mm_rss(tsk->mm) >> 10; - tsk->acct_vm_mem1 += delta * READ_ONCE(tsk->mm->total_vm) >> 10; + tsk->acct_rss_mem1 += delta * mm_pages_to_shift(tsk->mm, get_mm_rss(tsk->mm), + MM_ACCOUNT_SHIFT) >> 10; + tsk->acct_vm_mem1 += delta * mm_pages_to_shift(tsk->mm, READ_ONCE(tsk->mm->total_vm), + MM_ACCOUNT_SHIFT) >> 10; } /** diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c index 17d37b82ebc685..d808ddcb94b01a 100644 --- a/lib/vdso/datastore.c +++ b/lib/vdso/datastore.c @@ -83,7 +83,8 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm, vm_fault_t err; addr = vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE; - err = vmf_insert_page(vma, addr, page); + err = vmf_insert_page_offset(vma, addr, page, + vma_page_offset_at(vma, vmf->address).offset); if (unlikely(err & VM_FAULT_ERROR)) return err; page = timens_page; diff --git a/mm/Kconfig b/mm/Kconfig index e8bf1e9e6ad907..b743b2290b3091 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1164,6 +1164,28 @@ config IDLE_PAGE_TRACKING See Documentation/admin-guide/mm/idle_page_tracking.rst for more details. +config ARCH_HAS_MM_PGTABLE + bool + help + Architecture supplies target-mm page-table geometry and accessors. + This does not change the native physical allocation granule. + +config ARCH_HAS_USER_PAGE_SIZE + bool + help + Architecture supports selecting the user mapping granule at exec. + It supplies arch_exec_page_size_shift() to validate requests, and + integrates the selected granule into mm allocation and binary loading. + +config MM_SUBPAGE + bool + select PAGE_EXTENSION + select XARRAY_MULTI + help + Reference-counted user data slots smaller than a native folio. + Architectures must integrate rmap, faults, reclaim and pins before + exposing these backing primitives to userspace. + # Architectures which implement cpu_dcache_is_aliasing() to query # whether the data caches are aliased (VIVT or VIPT with dcache # aliasing) need to select this. @@ -1439,9 +1461,15 @@ config ARCH_HAS_USER_SHADOW_STACK config HAVE_ARCH_TLB_REMOVE_TABLE def_bool n +config ARCH_SUPPORTS_PT_RECLAIM + bool + help + The architecture's custom table-free callback supports PTE reclamation. + config PT_RECLAIM def_bool y - depends on MMU_GATHER_RCU_TABLE_FREE && !HAVE_ARCH_TLB_REMOVE_TABLE + depends on MMU_GATHER_RCU_TABLE_FREE + depends on !HAVE_ARCH_TLB_REMOVE_TABLE || ARCH_SUPPORTS_PT_RECLAIM help Try to reclaim empty user page table pages in paths other than munmap and exit_mmap path. diff --git a/mm/Makefile b/mm/Makefile index 8ad2ab08244ebe..f531742956b8e5 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o +obj-$(CONFIG_MM_SUBPAGE) += user-subpage.o diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index cca7014ab5f137..3d618828c2c8dd 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -155,9 +155,10 @@ int damon_cold_score(struct damon_ctx *c, struct damon_region *r, } static bool damon_folio_mkold_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr, void *arg) + struct vm_area_struct *vma, struct rmap_walk_range rmap, void *arg) { - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0); + unsigned long addr = rmap.address; + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); while (page_vma_mapped_walk(&pvmw)) { addr = pvmw.address; @@ -190,10 +191,11 @@ void damon_folio_mkold(struct folio *folio) } static bool damon_folio_young_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr, void *arg) + struct vm_area_struct *vma, struct rmap_walk_range rmap, void *arg) { + unsigned long addr = rmap.address; bool *accessed = arg; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0); + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); pte_t pte; *accessed = false; diff --git a/mm/filemap.c b/mm/filemap.c index bbd400d893aa5e..14e2656b6a9ccc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -225,6 +225,7 @@ void __filemap_remove_folio(struct folio *folio, void *shadow) trace_mm_filemap_delete_from_page_cache(folio); filemap_unaccount_folio(mapping, folio); + shmem_uffd_forget(mapping, folio->index, folio_nr_pages(folio)); page_cache_delete(mapping, folio, shadow); } @@ -308,6 +309,7 @@ static void page_cache_delete_batch(struct address_space *mapping, WARN_ON_ONCE(!folio_test_locked(folio)); + shmem_uffd_forget(mapping, folio->index, folio_nr_pages(folio)); folio->mapping = NULL; /* Leave folio->index set: truncation lookup relies on it */ @@ -827,6 +829,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) mem_cgroup_replace_folio(old, new); xas_lock_irq(&xas); + shmem_uffd_migrate(new, old); xas_store(&xas, new); old->mapping = NULL; @@ -3387,7 +3390,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) */ struct vm_area_struct *vma = vmf->vma; unsigned long start = vma->vm_pgoff; - unsigned long end = start + vma_pages(vma); + unsigned long end = vma_last_pgoff(vma) + 1; unsigned long ra_end; ra->order = exec_folio_order(); @@ -3529,7 +3532,12 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) bool mapping_locked = false; max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(index >= max_idx)) + if (unlikely(index >= max_idx +#ifdef CONFIG_MM_SUBPAGE + || (mm_page_size(vmf->vma->vm_mm) < PAGE_SIZE && + !vma_file_offset_valid(vmf->vma, vmf->address, i_size_read(inode))) +#endif + )) return VM_FAULT_SIGBUS; trace_mm_filemap_fault(mapping, index); @@ -3633,7 +3641,12 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * We must recheck i_size under page lock. */ max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(index >= max_idx)) { + if (unlikely(index >= max_idx +#ifdef CONFIG_MM_SUBPAGE + || (mm_page_size(vmf->vma->vm_mm) < PAGE_SIZE && + !vma_file_offset_valid(vmf->vma, vmf->address, i_size_read(inode))) +#endif + )) { folio_unlock(folio); folio_put(folio); return VM_FAULT_SIGBUS; @@ -3783,6 +3796,10 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, } do { + if (userfaultfd_missing(vmf->vma) && + shmem_uffd_range_missing(folio, folio->index + + folio_page_idx(folio, page + count), 0, PAGE_SIZE)) + goto skip; if (PageHWPoison(page + count)) goto skip; @@ -3847,6 +3864,9 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, vm_fault_t ret = 0; struct page *page = &folio->page; + if (userfaultfd_missing(vmf->vma) && + shmem_uffd_range_missing(folio, folio->index, 0, PAGE_SIZE)) + goto out; if (PageHWPoison(page)) goto out; @@ -3875,6 +3895,48 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, return ret; } +#ifdef CONFIG_MM_SUBPAGE +static vm_fault_t filemap_map_user_pages(struct vm_fault *vmf, + pgoff_t first, pgoff_t last) +{ + struct vm_area_struct *vma = vmf->vma; + struct address_space *mapping = vma->vm_file->f_mapping; + XA_STATE(xas, &mapping->i_pages, first); + struct folio *folio; + vm_fault_t ret = 0; + unsigned short miss = 0, saved; + + rcu_read_lock(); + folio = next_uptodate_folio(&xas, mapping, last); + if (!folio) + goto out; + if (pmd_none(*vmf->pmd) && vmf->prealloc_pte) + pmd_install(vma->vm_mm, vmf->pmd, &vmf->prealloc_pte); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (!vmf->pte) { + folio_unlock(folio); + folio_put(folio); + goto out; + } + do { + ret |= map_file_user_pages(vmf, folio, first, last, false); + if (!folio_test_workingset(folio)) + miss += min(folio_next_index(folio) - 1, last) - + max(folio->index, first) + 1; + folio_unlock(folio); + folio_put(folio); + } while ((folio = next_uptodate_folio(&xas, mapping, last)) != NULL); + pte_unmap_unlock(vmf->pte, vmf->ptl); + trace_mm_filemap_map_pages(mapping, first, last); +out: + rcu_read_unlock(); + saved = READ_ONCE(vma->vm_file->f_ra.mmap_miss); + WRITE_ONCE(vma->vm_file->f_ra.mmap_miss, saved > miss ? saved - miss : 0); + return ret; +} +#endif + vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff) { @@ -3890,6 +3952,11 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, unsigned int nr_pages = 0, folio_type; unsigned short mmap_miss = 0, mmap_miss_saved; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return filemap_map_user_pages(vmf, start_pgoff, end_pgoff); +#endif + /* * Recalculate end_pgoff based on file_end before calling * next_uptodate_folio() to avoid races with concurrent diff --git a/mm/gup.c b/mm/gup.c index ad9ded39609cb6..b7738d87d55b83 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -1297,6 +1298,11 @@ static struct vm_area_struct *gup_vma_lookup(struct mm_struct *mm, #endif } +#ifdef CONFIG_MM_SUBPAGE +static long prefault_subpage_range(struct mm_struct *mm, unsigned long start, + unsigned long nr_pages, unsigned int flags, int *locked); +#endif + /** * __get_user_pages() - pin user pages in memory * @mm: mm_struct of target mm @@ -1363,6 +1369,15 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; +#ifdef CONFIG_MM_SUBPAGE + /* A prefault has no page-array ownership or physical-offset ambiguity. */ + if (mm_page_size(mm) != PAGE_SIZE) { + if (pages || (gup_flags & (FOLL_GET | FOLL_PIN))) + return -EOPNOTSUPP; + return prefault_subpage_range(mm, start, nr_pages, gup_flags, locked); + } +#endif + start = untagged_addr_remote(mm, start); VM_WARN_ON_ONCE(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1721,7 +1736,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, */ if (likely(pages)) pages += ret; - start += ret << PAGE_SHIFT; + start += ret << mm_page_shift(mm); /* The lock was temporarily dropped, so we must unlock later */ must_unlock = true; @@ -1768,7 +1783,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, break; if (likely(pages)) pages++; - start += PAGE_SIZE; + start += mm_page_size(mm); } if (must_unlock && *locked) { /* @@ -1814,13 +1829,13 @@ long populate_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, int *locked) { struct mm_struct *mm = vma->vm_mm; - unsigned long nr_pages = (end - start) / PAGE_SIZE; + unsigned long nr_pages = (end - start) / mm_page_size(mm); int local_locked = 1; int gup_flags; long ret; - VM_WARN_ON_ONCE(!PAGE_ALIGNED(start)); - VM_WARN_ON_ONCE(!PAGE_ALIGNED(end)); + VM_WARN_ON_ONCE(!IS_ALIGNED(start, mm_page_size(mm))); + VM_WARN_ON_ONCE(!IS_ALIGNED(end, mm_page_size(mm))); VM_WARN_ON_ONCE_VMA(start < vma->vm_start, vma); VM_WARN_ON_ONCE_VMA(end > vma->vm_end, vma); mmap_assert_locked(mm); @@ -1887,12 +1902,12 @@ long populate_vma_page_range(struct vm_area_struct *vma, long faultin_page_range(struct mm_struct *mm, unsigned long start, unsigned long end, bool write, int *locked) { - unsigned long nr_pages = (end - start) / PAGE_SIZE; + unsigned long nr_pages = (end - start) / mm_page_size(mm); int gup_flags; long ret; - VM_WARN_ON_ONCE(!PAGE_ALIGNED(start)); - VM_WARN_ON_ONCE(!PAGE_ALIGNED(end)); + VM_WARN_ON_ONCE(!IS_ALIGNED(start, mm_page_size(mm))); + VM_WARN_ON_ONCE(!IS_ALIGNED(end, mm_page_size(mm))); mmap_assert_locked(mm); /* @@ -1968,7 +1983,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) } break; } - nend = nstart + ret * PAGE_SIZE; + nend = nstart + ret * mm_page_size(mm); ret = 0; } if (locked) @@ -2047,6 +2062,7 @@ size_t fault_in_writeable(char __user *uaddr, size_t size) { const unsigned long start = (unsigned long)uaddr; const unsigned long end = start + size; + const unsigned long page_size = current->mm ? mm_page_size(current->mm) : PAGE_SIZE; unsigned long cur; if (unlikely(size == 0)) @@ -2055,7 +2071,7 @@ size_t fault_in_writeable(char __user *uaddr, size_t size) return size; /* Stop once we overflow to 0. */ - for (cur = start; cur && cur < end; cur = PAGE_ALIGN_DOWN(cur + PAGE_SIZE)) + for (cur = start; cur && cur < end; cur = round_down(cur + page_size, page_size)) unsafe_put_user(0, (char __user *)cur, out); out: user_write_access_end(); @@ -2116,6 +2132,7 @@ size_t fault_in_safe_writeable(const char __user *uaddr, size_t size) { const unsigned long start = (unsigned long)uaddr; const unsigned long end = start + size; + const unsigned long page_size = current->mm ? mm_page_size(current->mm) : PAGE_SIZE; unsigned long cur; struct mm_struct *mm = current->mm; bool unlocked = false; @@ -2125,7 +2142,7 @@ size_t fault_in_safe_writeable(const char __user *uaddr, size_t size) mmap_read_lock(mm); /* Stop once we overflow to 0. */ - for (cur = start; cur && cur < end; cur = PAGE_ALIGN_DOWN(cur + PAGE_SIZE)) + for (cur = start; cur && cur < end; cur = round_down(cur + page_size, page_size)) if (fixup_user_fault(mm, cur, FAULT_FLAG_WRITE, &unlocked)) break; mmap_read_unlock(mm); @@ -2148,6 +2165,7 @@ size_t fault_in_readable(const char __user *uaddr, size_t size) { const unsigned long start = (unsigned long)uaddr; const unsigned long end = start + size; + const unsigned long page_size = current->mm ? mm_page_size(current->mm) : PAGE_SIZE; unsigned long cur; volatile char c; @@ -2157,7 +2175,7 @@ size_t fault_in_readable(const char __user *uaddr, size_t size) return size; /* Stop once we overflow to 0. */ - for (cur = start; cur && cur < end; cur = PAGE_ALIGN_DOWN(cur + PAGE_SIZE)) + for (cur = start; cur && cur < end; cur = round_down(cur + page_size, page_size)) unsafe_get_user(c, (const char __user *)cur, out); out: user_read_access_end(); @@ -2211,11 +2229,18 @@ struct pages_or_folios { void **entries; }; bool has_folios; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment *fragments; +#endif long nr_entries; }; static struct folio *pofs_get_folio(struct pages_or_folios *pofs, long i) { +#ifdef CONFIG_MM_SUBPAGE + if (pofs->fragments) + return pofs->fragments[i].folio; +#endif if (pofs->has_folios) return pofs->folios[i]; return page_folio(pofs->pages[i]); @@ -2223,22 +2248,56 @@ static struct folio *pofs_get_folio(struct pages_or_folios *pofs, long i) static void pofs_clear_entry(struct pages_or_folios *pofs, long i) { +#ifdef CONFIG_MM_SUBPAGE + if (pofs->fragments) { + pofs->fragments[i] = (struct user_page_fragment) {}; + return; + } +#endif pofs->entries[i] = NULL; } static void pofs_unpin(struct pages_or_folios *pofs) { +#ifdef CONFIG_MM_SUBPAGE + if (pofs->fragments) { + release_user_fragments(pofs->fragments, pofs->nr_entries, false); + return; + } +#endif if (pofs->has_folios) unpin_folios(pofs->folios, pofs->nr_entries); else unpin_user_pages(pofs->pages, pofs->nr_entries); } +static void pofs_unpin_entry(struct pages_or_folios *pofs, long i) +{ +#ifdef CONFIG_MM_SUBPAGE + if (pofs->fragments) { + release_user_fragments(&pofs->fragments[i], 1, false); + return; + } +#endif + unpin_folio(pofs_get_folio(pofs, i)); +} + static struct folio *pofs_next_folio(struct folio *folio, struct pages_or_folios *pofs, long *index_ptr) { long i = *index_ptr + 1; +#ifdef CONFIG_MM_SUBPAGE + /* Several adjacent user quarters can pin the same native folio. */ + if (pofs->fragments) { + while (i < pofs->nr_entries && pofs->fragments[i].folio == folio) + i++; + if (i == pofs->nr_entries) + return NULL; + *index_ptr = i; + return pofs->fragments[i].folio; + } +#endif if (!pofs->has_folios && folio_test_large(folio)) { const unsigned long start_pfn = folio_pfn(folio); const unsigned long end_pfn = start_pfn + folio_nr_pages(folio); @@ -2333,9 +2392,9 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list, * convert the pin on the source folio to a normal * reference. */ - pofs_clear_entry(pofs, i); folio_get(folio); - gup_put_folio(folio, 1, FOLL_PIN); + pofs_unpin_entry(pofs, i); + pofs_clear_entry(pofs, i); if (migrate_device_coherent_folio(folio)) { ret = -EBUSY; @@ -2352,7 +2411,7 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list, * calling folio_isolate_lru() which takes a reference so the * folio won't be freed if it's migrating. */ - unpin_folio(folio); + pofs_unpin_entry(pofs, i); pofs_clear_entry(pofs, i); } @@ -2458,6 +2517,23 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios, } #endif /* CONFIG_MIGRATION */ +#ifdef CONFIG_MM_SUBPAGE +static long check_and_migrate_movable_fragments(unsigned long count, + struct user_page_fragment *fragments) +{ +#ifdef CONFIG_MIGRATION + struct pages_or_folios pofs = { + .fragments = fragments, + .nr_entries = count, + }; + + return check_and_migrate_movable_pages_or_folios(&pofs); +#else + return 0; +#endif +} +#endif + /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which * allows us to process the FOLL_LONGTERM flag. @@ -3133,6 +3209,11 @@ static unsigned long gup_fast(unsigned long start, unsigned long end, int nr_pinned = 0; unsigned seq; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(current->mm) != PAGE_SIZE) + return 0; +#endif + if (!IS_ENABLED(CONFIG_HAVE_GUP_FAST) || !gup_fast_permitted(start, end)) return 0; @@ -3355,6 +3436,460 @@ long pin_user_pages_remote(struct mm_struct *mm, } EXPORT_SYMBOL(pin_user_pages_remote); +#ifdef CONFIG_MM_SUBPAGE +/* 1: acquired, 0: fault required, -EMLINK: read pin needs private backing. */ +static int follow_subpage_fragment(struct vm_area_struct *vma, unsigned long addr, + unsigned int flags, struct user_page_fragment *fragment) +{ + struct mm_struct *mm = vma->vm_mm; + struct mm_subpage *slot = NULL; + struct folio *folio; + struct page *page; + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *ptep, pte; + spinlock_t *ptl; + phys_addr_t phys; + int ret = 0; + bool file_ref = false; + + if (pgd_none(*pgd) || pgd_bad(*pgd)) + return PTR_ERR_OR_ZERO(no_page_table(vma, flags, addr)); + p4d = p4d_offset_mm(mm, pgd, addr); + if (p4d_none_mm(mm, *p4d) || p4d_bad_mm(mm, *p4d)) + return PTR_ERR_OR_ZERO(no_page_table(vma, flags, addr)); + pud = pud_offset_mm(mm, p4d, addr); + if (pud_none(*pud)) + return PTR_ERR_OR_ZERO(no_page_table(vma, flags, addr)); + if (pud_leaf(*pud) || pud_bad(*pud)) + return -EOPNOTSUPP; + pmd = pmd_offset_mm(mm, pud, addr); + if (pmd_none(*pmd)) + return PTR_ERR_OR_ZERO(no_page_table(vma, flags, addr)); + if (pmd_leaf(*pmd) || pmd_bad(*pmd)) + return -EOPNOTSUPP; + ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!ptep) + return 0; + pte = ptep_get(ptep); + if (pte_none(pte)) { + ret = PTR_ERR_OR_ZERO(no_page_table(vma, flags, addr)); + goto out; + } + if (!pte_present(pte) || (pte_protnone(pte) && !gup_can_follow_protnone(vma, flags))) + goto out; + phys = pte_phys_mm(mm, pte); + page = vm_normal_page(vma, addr, pte); + if (mm_page_size(mm) > PAGE_SIZE) { + unsigned long offset = (addr & (mm_page_size(mm) - 1)) & PAGE_MASK; + + phys += offset; + if (page) + page += offset >> PAGE_SHIFT; + } + if (!page) { + if (flags & FOLL_DUMP) { + ret = -EFAULT; + goto out; + } + if (!is_zero_pfn(pte_pfn(pte))) { + ret = -EOPNOTSUPP; + goto out; + } + if (flags & FOLL_WRITE) + goto out; + folio = page_folio(pfn_to_page(PHYS_PFN(phys))); + if (flags & FOLL_GET) + folio_get(folio); + } else { + folio = page_folio(page); + /* Coarse leaves use ordinary native-page ownership, not slots. */ + if (mm_page_size(mm) > PAGE_SIZE) { + if ((flags & FOLL_WRITE) && + !can_follow_write_pte(pte, page, vma, flags)) + goto out; + if (!pte_write(pte) && gup_must_unshare(vma, flags, page)) { + ret = -EMLINK; + goto out; + } + ret = try_grab_folio(folio, 1, flags); + if (ret) + goto out; + file_ref = true; + goto acquired; + } + if (!folio_test_anon(folio)) { + if ((flags & FOLL_WRITE) && !pte_write(pte)) + goto out; + /* Private non-anonymous read pins must first acquire COW ownership. */ + if ((flags & FOLL_PIN) && is_cow_mapping(vma->vm_flags)) { + ret = -EMLINK; + goto out; + } + ret = try_grab_folio(folio, 1, flags); + if (ret) + goto out; + file_ref = true; + goto acquired; + } + if (folio_test_ksm(folio)) { + ret = -EOPNOTSUPP; + goto out; + } + slot = mm_subpage_get_from_phys(phys); + if (!slot) { + ret = -EFAULT; + goto out; + } + if ((flags & FOLL_WRITE) && !pte_write(pte)) { + bool force = (flags & FOLL_FORCE) && + !(vma->vm_flags & (VM_SHARED | VM_MAYSHARE | VM_WRITE)) && + (vma->vm_flags & VM_MAYWRITE) && mm_subpage_is_exclusive(slot) && + !pte_needs_soft_dirty_wp(vma, pte) && !userfaultfd_pte_wp(vma, pte); + + if (!force) + goto out; + } + if ((flags & FOLL_PIN) && !mm_subpage_is_exclusive(slot)) { + ret = flags & FOLL_WRITE ? 0 : -EMLINK; + goto out; + } + if (flags & FOLL_PIN) { + ret = mm_subpage_pin(slot); + if (ret) { + if (ret == -EAGAIN) + ret = 0; + goto out; + } + } + } +acquired: + if (slot && (flags & FOLL_WRITE)) + mm_subpage_cancel_lazyfree(slot); + ret = flags & FOLL_PIN ? arch_make_folio_accessible(folio) : 0; + if (ret) { + if (slot) + mm_subpage_unpin(slot); + else if (file_ref) + gup_put_folio(folio, 1, flags); + goto out; + } + if ((flags & FOLL_TOUCH) && (flags & FOLL_WRITE) && + !pte_dirty(pte) && !folio_test_dirty(folio)) + folio_mark_dirty(folio); + folio_mark_accessed(folio); + flush_anon_page(vma, &folio->page, addr); + flush_dcache_folio(folio); + fragment->folio = folio; + fragment->subpage = slot; + fragment->pinned = flags & FOLL_PIN; + if (flags & FOLL_GET) + slot = NULL; /* The returned record consumes the lookup reference. */ + fragment->offset = phys - PFN_PHYS(folio_pfn(folio)) + + (addr & (mm_user_fragment_size(mm) - 1)); + ret = 1; +out: + if (slot) + mm_subpage_put(slot); /* Lookup reference; success retains a separate pin. */ + pte_unmap_unlock(ptep, ptl); + return ret; +} + +/* No references escape: only ensure each user leaf can be followed. */ +static long prefault_subpage_range(struct mm_struct *mm, unsigned long start, + unsigned long nr_pages, unsigned int flags, int *locked) +{ + unsigned long done = 0; + long ret = 0; + + mmap_assert_locked(mm); + start = untagged_addr_remote(mm, start); + while (done < nr_pages) { + struct user_page_fragment unused; + struct vm_area_struct *vma = vma_lookup(mm, start); + + if (!vma) { + ret = flags & FOLL_MADV_POPULATE ? -ENOMEM : -EFAULT; + break; + } + ret = check_vma_flags(vma, flags); + if (ret) { + if (flags & FOLL_MADV_POPULATE) + ret = -EINVAL; + break; + } + if (is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + ret = flags & FOLL_MADV_POPULATE ? -EINVAL : -EOPNOTSUPP; + break; + } + if (gup_signal_pending(flags)) { + ret = -EINTR; + break; + } + cond_resched(); + ret = follow_subpage_fragment(vma, start, flags, &unused); + if (!ret) { + ret = faultin_page(vma, start, flags, false, locked); + if (!ret) + continue; + if (ret == -EBUSY || ret == -EAGAIN) + ret = 0; /* Preserve GUP's caller-managed retry contract. */ + break; + } + if (ret < 0) + break; + done++; + start += mm_page_size(mm); + } + if (ret == -EOPNOTSUPP && (flags & FOLL_MADV_POPULATE)) + ret = -EINVAL; + return done ? (long)done : ret; +} + +static long user_fragments_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragments, unsigned long capacity, bool pin, + struct vm_area_struct **vma_out, bool dump) +{ + const unsigned int allowed = FOLL_WRITE | FOLL_FORCE | FOLL_NOFAULT | + FOLL_NOWAIT | FOLL_INTERRUPTIBLE | FOLL_LONGTERM | FOLL_ANON; + unsigned int internal = flags | FOLL_TOUCH | FOLL_REMOTE | + (pin ? FOLL_PIN : FOLL_GET) | (dump ? FOLL_DUMP : 0); + unsigned long end, count = 0; + int locked = 0; + long ret; + + if (!length) + return 0; + if ((flags & FOLL_LONGTERM) && !pin) + return -EINVAL; + if (!fragments || !capacity || capacity > LONG_MAX || (flags & ~allowed)) + return -EINVAL; + if (vma_out) { + mmap_assert_locked(mm); + *vma_out = NULL; + } else { + internal |= FOLL_UNLOCKABLE; + ret = mmap_read_lock_killable(mm); + if (ret) + return ret; + } + locked = 1; + start = untagged_addr_remote(mm, start); + if (check_add_overflow(start, length, &end)) { + ret = -EFAULT; + goto out; + } + if (pin) + mm_set_has_pinned_flag(mm); + while (start < end && count < capacity) { + unsigned long bytes = min(end - start, + mm_user_fragment_size(mm) - (start & (mm_user_fragment_size(mm) - 1))); + struct vm_area_struct *vma; + + if (!locked) { + ret = mmap_read_lock_killable(mm); + if (ret) + break; + locked = 1; + } + if (gup_signal_pending(internal)) { + ret = -EINTR; + break; + } + cond_resched(); + if (mm_page_size(mm) == PAGE_SIZE) { + struct page *page; + + if (dump) + ret = __get_user_pages_locked(mm, start, 1, &page, &locked, internal); + else if (pin) + ret = pin_user_pages_remote(mm, start, 1, flags, &page, + vma_out ? NULL : &locked); + else + ret = get_user_pages_remote(mm, start, 1, flags, &page, + vma_out ? NULL : &locked); + if (ret != 1) { + if (!ret) + ret = -EAGAIN; + break; + } + fragments[count].folio = page_folio(page); + fragments[count].subpage = NULL; + fragments[count].pinned = pin; + fragments[count].offset = folio_page_idx(page_folio(page), page) * PAGE_SIZE + + offset_in_page(start); + } else { + vma = gup_vma_lookup(mm, start); + ret = vma ? check_vma_flags(vma, internal) : -EFAULT; + if (ret) + break; + /* Huge and special/device mappings need separate integration. */ + if (is_vm_hugetlb_page(vma) || + (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP | VM_IO))) { + ret = -EOPNOTSUPP; + break; + } + ret = follow_subpage_fragment(vma, start, internal, &fragments[count]); + if (!ret || ret == -EMLINK) { + ret = faultin_page(vma, start, internal, ret == -EMLINK, &locked); + if (!ret) + continue; + if (!locked && (ret == -EBUSY || ret == -EAGAIN)) { + internal |= FOLL_TRIED; + continue; + } + if (ret == -EBUSY) + ret = -EAGAIN; + break; + } + if (ret < 0) + break; + } + if (vma_out) + *vma_out = vma_lookup(mm, start); + fragments[count++].length = bytes; + start += bytes; + internal &= ~FOLL_TRIED; + } +out: + if (locked && !vma_out) + mmap_read_unlock(mm); + return count ? (long)count : ret; +} + +long pin_user_fragments_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragments, unsigned long capacity) +{ + unsigned int saved; + long ret, count; + + if (!(flags & FOLL_LONGTERM) || mm_page_size(mm) == PAGE_SIZE) + return user_fragments_remote(mm, start, length, flags, fragments, + capacity, true, NULL, false); + /* Match native long-term GUP: fault outside movable zones, migrate any + * existing unpinnable backing, then retry with fresh typed references. + */ + saved = memalloc_pin_save(); + do { + count = user_fragments_remote(mm, start, length, flags, fragments, + capacity, true, NULL, false); + if (count <= 0) { + ret = count; + break; + } + ret = check_and_migrate_movable_fragments(count, fragments); + } while (ret == -EAGAIN); + memalloc_pin_restore(saved); + return ret ? ret : count; +} +EXPORT_SYMBOL_GPL(pin_user_fragments_remote); + +long get_user_fragments_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragments, unsigned long capacity) +{ + return user_fragments_remote(mm, start, length, flags, fragments, capacity, + false, NULL, false); +} +EXPORT_SYMBOL_GPL(get_user_fragments_remote); + +int get_user_fragment_vma_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragment, struct vm_area_struct **vma) +{ + int ret; + + /* As with get_user_page_vma_remote(), preserve the caller's VMA lock. */ + if (!vma) + return -EINVAL; + *vma = NULL; + if (flags & FOLL_NOWAIT) + return -EINVAL; + ret = user_fragments_remote(mm, start, length, flags, fragment, 1, false, vma, false); + if (ret > 0 && !*vma) { + release_user_fragments(fragment, 1, false); + return -EFAULT; + } + return ret; +} + +int pin_user_fragment_vma_remote(struct mm_struct *mm, unsigned long start, + unsigned long length, unsigned int flags, + struct user_page_fragment *fragment, struct vm_area_struct **vma) +{ + int ret; + + if (!vma) + return -EINVAL; + *vma = NULL; + if (flags & (FOLL_NOWAIT | FOLL_LONGTERM)) + return -EINVAL; + ret = user_fragments_remote(mm, start, length, flags, fragment, 1, true, vma, false); + if (ret > 0 && !*vma) { + release_user_fragments(fragment, 1, false); + return -EFAULT; + } + return ret; +} + +#ifdef CONFIG_ELF_CORE +/* Like get_dump_page(), but preserves the selected user-page ownership. */ +int get_dump_fragment(unsigned long addr, struct user_page_fragment *fragment) +{ + return user_fragments_remote(current->mm, addr, mm_user_fragment_size(current->mm), + FOLL_FORCE, fragment, 1, false, NULL, true); +} +#endif + +void mark_user_fragment_dirty(struct user_page_fragment *fragment) +{ + struct folio *folio = fragment->folio; + + if (!folio) + return; + if (!is_zero_folio(folio) && !folio_test_dirty(folio)) { + folio_lock(folio); + folio_mark_dirty(folio); + folio_unlock(folio); + } + if (fragment->subpage) + mm_subpage_cancel_lazyfree(fragment->subpage); +} +EXPORT_SYMBOL_GPL(mark_user_fragment_dirty); + +void release_user_fragments(struct user_page_fragment *fragments, + unsigned long count, bool dirty) +{ + unsigned long i; + + for (i = 0; i < count; i++) { + struct user_page_fragment *fragment = &fragments[i]; + struct folio *folio = fragment->folio; + + if (!folio) + continue; + if (dirty) + mark_user_fragment_dirty(fragment); + if (fragment->subpage) { + if (fragment->pinned) + mm_subpage_unpin(fragment->subpage); + else + mm_subpage_put(fragment->subpage); + } else if (fragment->pinned) { + unpin_folio(folio); + } else { + folio_put(folio); + } + *fragment = (struct user_page_fragment) {}; + } +} +EXPORT_SYMBOL_GPL(release_user_fragments); +#endif + /** * pin_user_pages() - pin user pages in memory for use by other devices * @@ -3407,6 +3942,25 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, } EXPORT_SYMBOL(pin_user_pages_unlocked); +/* Direct cache pins have no PTE fault to publish fine shmem exposure. */ +static int memfd_pin_folio(struct file *memfd, struct folio *folio) +{ + bool lock = IS_ENABLED(CONFIG_MM_SUBPAGE) && + IS_ENABLED(CONFIG_USERFAULTFD) && shmem_file(memfd); + int ret; + + /* Serialize both acquisition and exposure against partial hole punch. */ + if (lock) + folio_lock(folio); + ret = try_grab_folio(folio, 1, FOLL_PIN); + if (lock) { + if (!ret) + shmem_uffd_mark_mapped(folio, 0, folio_size(folio)); + folio_unlock(folio); + } + return ret; +} + /** * memfd_pin_folios() - pin folios associated with a memfd * @memfd: the memfd whose folios are to be pinned @@ -3492,7 +4046,7 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, for (i = 0; i < nr_found; i++) { folio = fbatch.folios[i]; - if (try_grab_folio(folio, 1, FOLL_PIN)) { + if (memfd_pin_folio(memfd, folio)) { folio_batch_release(&fbatch); ret = -EINVAL; goto err; diff --git a/mm/hmm.c b/mm/hmm.c index 5955f2f0c83db1..c4b96191ce09f9 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -652,6 +652,7 @@ static const struct mm_walk_ops hmm_walk_ops = { * the invalidation to finish. * -EFAULT: A page was requested to be valid and could not be made valid * ie it has no backing VMA or it is illegal to access + * -EOPNOTSUPP: The MM uses a non-native mapping granule. * * This is similar to get_user_pages(), except that it can read the page tables * without mutating them (ie causing faults). @@ -666,6 +667,9 @@ int hmm_range_fault(struct hmm_range *range) int ret; mmap_assert_locked(mm); + /* Each output PFN describes a whole native page, without a byte offset. */ + if (mm_page_size(mm) != PAGE_SIZE) + return -EOPNOTSUPP; do { /* If range is no longer valid force retry. */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dd93f1b0c44a20..7ab369477fcc74 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2715,7 +2715,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, goto unlock; if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma, - vma_is_single_threaded_private(vma))) + vma_is_single_threaded_private(vma), + folio_maybe_mapped_shared(pmd_folio(*pmd)))) goto unlock; } /* @@ -4118,6 +4119,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } anon_vma_lock_write(anon_vma); mapping = NULL; +#ifdef CONFIG_MM_SUBPAGE + if (new_order < anon_vma->min_folio_order) { + ret = -EBUSY; + goto out_unlock; + } +#endif } else { unsigned int min_order; gfp_t gfp; @@ -4394,7 +4401,7 @@ int folio_split(struct folio *folio, unsigned int new_order, unsigned int min_order_for_split(struct folio *folio) { if (folio_test_anon(folio)) - return 0; + return folio_anon_min_order(folio); /* * If the folio got truncated, we don't know the previous mapping and diff --git a/mm/internal.h b/mm/internal.h index 5a2ddcf68e0b6d..bd953128ed6f1e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -305,6 +305,50 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) return pte_mkold(pte); } +#ifdef CONFIG_MM_SUBPAGE +/* Batch contiguous user leaves without confusing their offsets with PFNs. */ +static inline unsigned int folio_subpage_pte_batch_flags(struct vm_area_struct *vma, + struct folio *folio, pte_t *ptep, pte_t *ptentp, unsigned int max_nr, fpb_t flags) +{ + struct mm_struct *mm = vma->vm_mm; + pte_t pte = *ptentp; + phys_addr_t offset = pte_phys_mm(mm, pte) - PFN_PHYS(folio_pfn(folio)); + pte_t expected; + unsigned int nr; + + VM_BUG_ON_FOLIO(offset >= folio_size(folio), folio); + max_nr = min_t(unsigned long, max_nr, (folio_size(folio) - offset) >> mm_page_shift(mm)); + VM_BUG_ON_FOLIO(!max_nr, folio); + nr = min(max_nr, pte_batch_hint(ptep, pte)); + expected = pte_advance_pfn_mm(mm, __pte_batch_clear_ignored(pte, flags), nr); + while (nr < max_nr) { + unsigned int batch; + + pte = ptep_get(ptep + nr); + if (!pte_same(__pte_batch_clear_ignored(pte, flags), expected)) + break; + if ((flags & FPB_MERGE_WRITE) && pte_write(pte)) + *ptentp = pte_mkwrite(*ptentp, vma); + if (flags & FPB_MERGE_YOUNG_DIRTY) { + if (pte_young(pte)) + *ptentp = pte_mkyoung(*ptentp); + if (pte_dirty(pte)) + *ptentp = pte_mkdirty(*ptentp); + } + batch = min(max_nr - nr, pte_batch_hint(ptep + nr, pte)); + nr += batch; + expected = pte_advance_pfn_mm(mm, expected, batch); + } + return nr; +} + +static inline unsigned int folio_subpage_pte_batch(struct vm_area_struct *vma, + struct folio *folio, pte_t *ptep, pte_t pte, unsigned int max_nr, fpb_t flags) +{ + return folio_subpage_pte_batch_flags(vma, folio, ptep, &pte, max_nr, flags); +} +#endif + /** * folio_pte_batch_flags - detect a PTE batch for a large folio * @folio: The large folio to detect a PTE batch for. @@ -416,6 +460,10 @@ static inline pte_t pte_move_swp_offset(pte_t pte, long delta) if (pte_swp_uffd_wp(pte)) new = pte_swp_mkuffd_wp(new); +#ifdef CONFIG_MM_SUBPAGE + new = pte_swp_set_subpage_offset(new, pte_swp_subpage_offset(pte)); +#endif + return new; } @@ -1158,6 +1206,11 @@ folio_within_range(struct folio *folio, struct vm_area_struct *vma, end = vma->vm_end; pgoff = folio_pgoff(folio); + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) { + addr = vma_address_at_offset(vma, (struct vm_page_offset) { .index = pgoff }); + return !IS_ERR_VALUE(addr) && addr >= start && addr <= end && + end - addr >= folio_size(folio); + } /* if folio start address is not in vma range */ if (!in_range(pgoff, vma->vm_pgoff, vma_pglen)) @@ -1240,6 +1293,8 @@ static inline unsigned long vma_address(const struct vm_area_struct *vma, if (pgoff >= vma->vm_pgoff) { address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + if (pgoff != vma->vm_pgoff) + address -= vma_subpage_offset(vma); /* Check for address beyond vma (or wrapped through 0?) */ if (address < vma->vm_start || address >= vma->vm_end) address = -EFAULT; @@ -1262,15 +1317,24 @@ static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw) pgoff_t pgoff; unsigned long address; +#ifdef CONFIG_MM_SUBPAGE + if (pvmw->flags & PVMW_SUBPAGE) + return min(pvmw->address + mm_page_size(vma->vm_mm), vma->vm_end); +#endif + /* Common case, plus ->pgoff is invalid for KSM */ - if (pvmw->nr_pages == 1) - return pvmw->address + PAGE_SIZE; + if (pvmw->nr_pages == 1 && mm_page_size(vma->vm_mm) >= PAGE_SIZE) + return min((pvmw->address & mm_page_mask(vma->vm_mm)) + + mm_page_size(vma->vm_mm), vma->vm_end); pgoff = pvmw->pgoff + pvmw->nr_pages; address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + address -= vma_subpage_offset(vma); /* Check for address beyond vma (or wrapped through 0?) */ if (address < vma->vm_start || address > vma->vm_end) address = vma->vm_end; + if (mm_page_size(vma->vm_mm) > PAGE_SIZE) + address = min(mm_page_align(vma->vm_mm, address), vma->vm_end); return address; } @@ -1568,11 +1632,11 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma) #ifdef CONFIG_NUMA_BALANCING bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, - bool is_private_single_threaded); + bool is_private_single_threaded, bool shared); #else static inline bool folio_can_map_prot_numa(struct folio *folio, - struct vm_area_struct *vma, bool is_private_single_threaded) + struct vm_area_struct *vma, bool is_private_single_threaded, bool shared) { return false; } @@ -1859,9 +1923,13 @@ static inline int io_remap_pfn_range_prepare(struct vm_area_desc *desc) const unsigned long orig_pfn = action->remap.start_pfn; const pgprot_t orig_pgprot = action->remap.pgprot; const unsigned long size = action->remap.size; - const unsigned long pfn = io_remap_pfn_range_pfn(orig_pfn, size); + const unsigned int offset = mmap_action_remap_offset(action); + unsigned long pfn; int err; + if (size > ULONG_MAX - offset) + return -EINVAL; + pfn = io_remap_pfn_range_pfn(orig_pfn, size + offset); action->remap.start_pfn = pfn; action->remap.pgprot = pgprot_decrypted(orig_pgprot); err = remap_pfn_range_prepare(desc); @@ -1900,7 +1968,7 @@ static inline bool clear_flush_young_ptes_notify(struct vm_area_struct *vma, young = clear_flush_young_ptes(vma, addr, ptep, nr); young |= mmu_notifier_clear_flush_young(vma->vm_mm, addr, - addr + nr * PAGE_SIZE); + addr + nr * mm_page_size(vma->vm_mm)); return young; } @@ -1920,7 +1988,8 @@ static inline bool test_and_clear_young_ptes_notify(struct vm_area_struct *vma, bool young; young = test_and_clear_young_ptes(vma, addr, ptep, nr); - young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + nr * PAGE_SIZE); + young |= mmu_notifier_clear_young(vma->vm_mm, addr, + addr + nr * mm_page_size(vma->vm_mm)); return young; } @@ -1952,4 +2021,14 @@ static inline int get_sysctl_max_map_count(void) bool may_expand_vm(struct mm_struct *mm, const vma_flags_t *vma_flags, unsigned long npages); +#ifdef CONFIG_MM_SUBPAGE +struct iov_iter; +int process_vm_rw_fragments(struct mm_struct *mm, unsigned long addr, + unsigned long len, struct iov_iter *iter, bool write); +#endif + +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SWAP) +vm_fault_t do_swap_coarse_page(struct vm_fault *vmf); +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d6e04041f5dc11..ad4832f64fc306 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1066,28 +1066,31 @@ static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm, } static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, - struct collapse_control *cc) + struct collapse_control *cc, unsigned int order) { gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : GFP_TRANSHUGE); int node = collapse_find_target_node(cc); struct folio *folio; - folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask); + folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask); if (!folio) { *foliop = NULL; - count_vm_event(THP_COLLAPSE_ALLOC_FAILED); + if (is_pmd_order(order)) + count_vm_event(THP_COLLAPSE_ALLOC_FAILED); return SCAN_ALLOC_HUGE_PAGE_FAIL; } - count_vm_event(THP_COLLAPSE_ALLOC); + if (is_pmd_order(order)) + count_vm_event(THP_COLLAPSE_ALLOC); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *foliop = NULL; return SCAN_CGROUP_CHARGE_FAIL; } - count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); + if (is_pmd_order(order)) + count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); *foliop = folio; return SCAN_SUCCEED; @@ -1116,7 +1119,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a */ mmap_read_unlock(mm); - result = alloc_charge_folio(&folio, mm, cc); + result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out_nolock; @@ -1880,22 +1883,33 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) * + unlock and free huge page; */ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, - struct file *file, pgoff_t start, struct collapse_control *cc) + struct file *file, pgoff_t start, struct collapse_control *cc, + unsigned int order) { struct address_space *mapping = file->f_mapping; struct page *dst; struct folio *folio, *tmp, *new_folio; - pgoff_t index = 0, end = start + HPAGE_PMD_NR; + unsigned long nr_pages = 1UL << order; + pgoff_t index = 0, end = start + nr_pages; LIST_HEAD(pagelist); - XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); + XA_STATE_ORDER(xas, &mapping->i_pages, start, order); enum scan_result result = SCAN_SUCCEED; int nr_none = 0; bool is_shmem = shmem_file(file); VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); - VM_BUG_ON(start & (HPAGE_PMD_NR - 1)); + VM_BUG_ON(start & (nr_pages - 1)); - result = alloc_charge_folio(&new_folio, mm, cc); + if (is_pmd_order(order)) { + result = alloc_charge_folio(&new_folio, mm, cc, order); + } else { + /* Sub-PMD user leaves must respect the shmem inode's allocation policy. */ + new_folio = shmem_alloc_collapse_folio(mm, mapping->host, start, order); + if (IS_ERR(new_folio)) { + new_folio = NULL; + result = SCAN_ALLOC_HUGE_PAGE_FAIL; + } + } if (result != SCAN_SUCCEED) goto out; @@ -2024,7 +2038,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, * we locked the first folio, then a THP might be there already. * This will be discovered on the first iteration. */ - if (is_pmd_order(folio_order(folio))) { + if (folio_order(folio) >= order) { result = SCAN_PTE_MAPPED_HUGEPAGE; goto out_unlock; } @@ -2149,6 +2163,8 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, while (index < folio->index) { clear_highpage(dst); + if (is_shmem) + shmem_uffd_copy_page(dst, NULL); index++; dst++; } @@ -2158,12 +2174,16 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, result = SCAN_COPY_MC; goto rollback; } + if (is_shmem) + shmem_uffd_copy_page(dst, folio_page(folio, i)); index++; dst++; } } while (index < end) { clear_highpage(dst); + if (is_shmem) + shmem_uffd_copy_page(dst, NULL); index++; dst++; } @@ -2205,8 +2225,13 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, * not be able to observe any missing pages due to the * previously inserted retry entries. */ + /* + * A required user leaf populates its whole granule, as a native + * fault does. Only optional PMD collapse must preserve cache holes + * merely because another VMA has missing-page registration. + */ vma_interval_tree_foreach(vma, &mapping->i_mmap, start, end) { - if (userfaultfd_missing(vma)) { + if (is_pmd_order(order) && userfaultfd_missing(vma)) { result = SCAN_EXCEED_NONE_PTE; goto immap_locked; } @@ -2229,12 +2254,13 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, } if (is_shmem) { - lruvec_stat_mod_folio(new_folio, NR_SHMEM, HPAGE_PMD_NR); - lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, HPAGE_PMD_NR); - } else { - lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, HPAGE_PMD_NR); + lruvec_stat_mod_folio(new_folio, NR_SHMEM, nr_pages); + if (is_pmd_order(order)) + lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, nr_pages); + } else if (is_pmd_order(order)) { + lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, nr_pages); } - lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, HPAGE_PMD_NR); + lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, nr_pages); /* * Mark new_folio as uptodate before inserting it into the @@ -2242,14 +2268,14 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, * unwritten page. */ folio_mark_uptodate(new_folio); - folio_ref_add(new_folio, HPAGE_PMD_NR - 1); + folio_ref_add(new_folio, nr_pages - 1); if (is_shmem) folio_mark_dirty(new_folio); folio_add_lru(new_folio); /* Join all the small entries into a single multi-index entry. */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_set_order(&xas, start, order); xas_store(&xas, new_folio); WARN_ON_ONCE(xas_error(&xas)); xas_unlock_irq(&xas); @@ -2258,9 +2284,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, * Remove pte page tables, so we can re-fault the page as huge. * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). */ - retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) - result = SCAN_PTE_MAPPED_HUGEPAGE; + if (is_pmd_order(order)) { + retract_page_tables(mapping, start); + if (cc && !cc->is_khugepaged) + result = SCAN_PTE_MAPPED_HUGEPAGE; + } folio_unlock(new_folio); /* @@ -2317,7 +2345,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, folio_put(new_folio); out: VM_BUG_ON(!list_empty(&pagelist)); - trace_mm_khugepaged_collapse_file(mm, new_folio, index, addr, is_shmem, file, HPAGE_PMD_NR, result); + trace_mm_khugepaged_collapse_file(mm, new_folio, index, addr, is_shmem, file, nr_pages, result); return result; } @@ -2420,7 +2448,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, result = SCAN_EXCEED_NONE_PTE; count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } else { - result = collapse_file(mm, addr, file, start, cc); + result = collapse_file(mm, addr, file, start, cc, HPAGE_PMD_ORDER); } } @@ -2831,6 +2859,47 @@ static int madvise_collapse_errno(enum scan_result r) } } +/* + * Promote the cache backing for one larger-than-native user leaf. The caller + * owns mm/file references; no folio or page-table lock may be held. Start is + * a native file index. This sub-PMD shmem path may run under the fault lock: + * it neither retracts page tables (which requires mmap write locking) nor + * invokes filesystem writeback. Source folios are trylocked and rolled back + * on contention. Ordinary PMD collapse retains its existing lock policy. + */ +int collapse_file_user_page(struct mm_struct *mm, struct file *file, pgoff_t start) +{ + struct collapse_control *cc; + unsigned int order; + enum scan_result result; + + if (mm_page_shift(mm) <= PAGE_SHIFT || !shmem_file(file)) + return -EOPNOTSUPP; + order = mm_page_shift(mm) - PAGE_SHIFT; + /* Shmem's order limit reflects THP policy, not an architectural leaf limit. */ + if (order >= HPAGE_PMD_ORDER || order > MAX_PAGECACHE_ORDER || + !IS_ALIGNED(start, 1UL << order) || start > ULONG_MAX - (1UL << order)) + return -EINVAL; + cc = kzalloc(sizeof(*cc), GFP_KERNEL); + if (!cc) + return -ENOMEM; + lru_add_drain_all(); + result = collapse_file(mm, 0, file, start, cc, order); + kfree(cc); + if (result == SCAN_SUCCEED || result == SCAN_PTE_MAPPED_HUGEPAGE) + return 0; + /* Retained references cannot be displaced by replacing their cache folio. */ + if (result == SCAN_PAGE_COUNT) + return -EBUSY; + if (result == SCAN_CGROUP_CHARGE_FAIL) + return -ENOMEM; + if (result == SCAN_COPY_MC) + return -EHWPOISON; + if (result == SCAN_TRUNCATED) + return -EAGAIN; + return madvise_collapse_errno(result); +} + int madvise_collapse(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool *lock_dropped) { diff --git a/mm/ksm.c b/mm/ksm.c index 7d5b76478f0b65..cd0398e9729410 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -804,6 +804,10 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item) struct folio_walk fw; struct folio *folio; + /* The KSM comparison and replacement paths still require native leaves. */ + if (mm_page_size(mm) != PAGE_SIZE) + return NULL; + mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (!vma) @@ -1338,7 +1342,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct folio *folio, /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, &folio->page)) { + folio_try_share_anon_rmap_pte(folio, &folio->page, vma)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } @@ -3206,7 +3210,8 @@ void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc) if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) continue; - if (!rwc->rmap_one(folio, vma, addr, rwc->arg)) { + if (!rwc->rmap_one(folio, vma, + (struct rmap_walk_range) { .address = addr }, rwc->arg)) { anon_vma_unlock_read(anon_vma); return; } diff --git a/mm/madvise.c b/mm/madvise.c index 5665785e5ac383..e746437c6b1f3f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -7,6 +7,7 @@ */ #include +#include #include #include #include @@ -193,7 +194,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, spinlock_t *ptl; unsigned long addr; - for (addr = start; addr < end; addr += PAGE_SIZE) { + for (addr = start; addr < end; addr += mm_page_size(vma->vm_mm)) { pte_t pte; softleaf_t entry; struct folio *folio; @@ -236,7 +237,7 @@ static void shmem_swapin_range(struct vm_area_struct *vma, struct address_space *mapping) { XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start)); - pgoff_t end_index = linear_page_index(vma, end) - 1; + pgoff_t end_index = linear_page_index(vma, end - 1); struct folio *folio; struct swap_iocb *splug = NULL; @@ -252,8 +253,11 @@ static void shmem_swapin_range(struct vm_area_struct *vma, if (!softleaf_is_swap(entry)) continue; - addr = vma->vm_start + - ((xas.xa_index - vma->vm_pgoff) << PAGE_SHIFT); + /* The first cache page may start before this VMA's byte offset. */ + addr = xas.xa_index == linear_page_index(vma, start) ? start : + vma_address_at_offset(vma, (struct vm_page_offset) { + .index = xas.xa_index, + }); xas_pause(&xas); rcu_read_unlock(); @@ -318,7 +322,7 @@ static long madvise_willneed(struct madvise_behavior *madv_behavior) mark_mmap_lock_dropped(madv_behavior); get_file(file); offset = (loff_t)(start - vma->vm_start) - + ((loff_t)vma->vm_pgoff << PAGE_SHIFT); + + ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma); mmap_read_unlock(mm); vfs_fadvise(file, offset, end - start, POSIX_FADV_WILLNEED); fput(file); @@ -340,12 +344,18 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } -static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, +static inline int madvise_folio_pte_batch(struct vm_area_struct *vma, + unsigned long addr, unsigned long end, struct folio *folio, pte_t *ptep, pte_t *ptentp) { - int max_nr = (end - addr) / PAGE_SIZE; + int max_nr = (end - addr) >> mm_page_shift(vma->vm_mm); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return folio_subpage_pte_batch_flags(vma, folio, ptep, ptentp, + max_nr, FPB_MERGE_YOUNG_DIRTY); +#endif return folio_pte_batch_flags(folio, NULL, ptep, ptentp, max_nr, FPB_MERGE_YOUNG_DIRTY); } @@ -376,7 +386,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, #ifdef CONFIG_TRANSPARENT_HUGEPAGE if (pmd_trans_huge(*pmd)) { pmd_t orig_pmd; - unsigned long next = pmd_addr_end(addr, end); + unsigned long next = pmd_addr_end_mm(mm, addr, end); tlb_change_page_size(tlb, HPAGE_PMD_SIZE); ptl = pmd_trans_huge_lock(pmd, vma); @@ -446,14 +456,14 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, regular_folio: #endif - tlb_change_page_size(tlb, PAGE_SIZE); + tlb_change_page_size(tlb, mm_page_size(mm)); restart: start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (!start_pte) return 0; flush_tlb_batched_pending(mm); lazy_mmu_mode_enable(); - for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) { + for (; addr < end; pte += nr, addr += nr * mm_page_size(mm)) { nr = 1; ptent = ptep_get(pte); @@ -484,9 +494,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * fail to split a folio, leave it in place and advance to the * next pte in the range. */ - if (folio_test_large(folio)) { - nr = madvise_folio_pte_batch(addr, end, folio, pte, &ptent); - if (nr < folio_nr_pages(folio)) { + if (folio_test_large(folio) || mm_page_size(mm) != PAGE_SIZE) { + nr = madvise_folio_pte_batch(vma, addr, end, folio, pte, &ptent); + if (folio_test_large(folio) && + nr < (folio_size(folio) >> mm_page_shift(mm))) { int err; if (folio_maybe_mapped_shared(folio)) @@ -521,7 +532,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * number of pages, it must be exclusive. */ if (!folio_test_lru(folio) || - folio_mapcount(folio) != folio_nr_pages(folio)) + folio_mapcount(folio) != + (mm_page_size(mm) != PAGE_SIZE ? nr : folio_nr_pages(folio))) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) @@ -649,6 +661,75 @@ static long madvise_pageout(struct madvise_behavior *madv_behavior) return 0; } +#ifdef CONFIG_MM_SUBPAGE +static int madvise_free_subpage_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + struct vm_area_struct *vma = walk->vma; + struct mm_struct *mm = vma->vm_mm; + struct mmu_gather *tlb = walk->private; + spinlock_t *ptl; + pte_t *base, *ptep; + long swapped = 0; + + base = ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!base) + return 0; + flush_tlb_batched_pending(mm); + for (; addr < end; addr += mm_page_size(mm), ptep++) { + pte_t pte = ptep_get(ptep); + struct mm_subpage *slot; + struct folio *folio; + struct page *page; + + if (!pte_present(pte)) { + softleaf_t entry = softleaf_from_pte(pte); + + if (softleaf_is_swap(entry)) { + swap_subpage_put(entry, + pte_swp_subpage_offset(pte), true); + pte_clear_not_present_full(mm, addr, ptep, tlb->fullmm); + swapped--; + } else if (softleaf_is_hwpoison(entry) || + softleaf_is_poison_marker(entry)) { + pte_clear_not_present_full(mm, addr, ptep, tlb->fullmm); + } + continue; + } + page = vm_normal_page(vma, addr, pte); + if (!page) + continue; + folio = page_folio(page); + if (!folio_test_anon(folio) || !folio_trylock(folio)) + continue; + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + if (!slot) + goto unlock; + if (folio_test_swapcache(folio) && !folio_free_swap(folio)) + goto put; + if (!mm_subpage_mark_lazyfree(slot)) + goto put; + clear_young_dirty_ptes(vma, addr, ptep, 1, + CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY); + tlb_remove_tlb_entry(tlb, ptep, addr); + /* Partial hints must never clear a neighbor's dirty state. */ + if (mm_subpage_all_lazyfree(slot)) { + folio_clear_dirty(folio); + folio_mark_lazyfree(folio); + } +put: + mm_subpage_put(slot); +unlock: + folio_unlock(folio); + } + if (swapped) + add_mm_counter(mm, MM_SWAPENTS, swapped); + pte_unmap_unlock(base, ptl); + cond_resched(); + return 0; +} +#endif + static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -664,6 +745,10 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long next; int nr, max_nr; + #ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) != PAGE_SIZE) + return madvise_free_subpage_range(pmd, addr, end, walk); +#endif next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) @@ -714,7 +799,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * next pte in the range. */ if (folio_test_large(folio)) { - nr = madvise_folio_pte_batch(addr, end, folio, pte, &ptent); + nr = madvise_folio_pte_batch(vma, addr, end, folio, pte, &ptent); if (nr < folio_nr_pages(folio)) { int err; @@ -988,7 +1073,7 @@ static long madvise_populate(struct madvise_behavior *madv_behavior) return -ENOMEM; } } - start += pages * PAGE_SIZE; + start += pages * mm_page_size(mm); } return 0; } @@ -1022,7 +1107,7 @@ static long madvise_remove(struct madvise_behavior *madv_behavior) return -EACCES; offset = (loff_t)(start - vma->vm_start) - + ((loff_t)vma->vm_pgoff << PAGE_SHIFT); + + ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma); /* * Filesystem's fallocate may need to take i_rwsem. We need to @@ -1182,7 +1267,7 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior) if (err == 0) { unsigned long nr_expected_pages = - PHYS_PFN(range->end - range->start); + (range->end - range->start) >> mm_page_shift(vma->vm_mm); VM_WARN_ON(nr_pages != nr_expected_pages); return 0; @@ -1833,16 +1918,17 @@ static void madvise_finish_tlb(struct madvise_behavior *madv_behavior) tlb_finish_mmu(madv_behavior->tlb); } -static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior) +static bool is_valid_madvise(struct mm_struct *mm, unsigned long start, + size_t len_in, int behavior) { size_t len; if (!madvise_behavior_valid(behavior)) return false; - if (!PAGE_ALIGNED(start)) + if (!IS_ALIGNED(start, mm_page_size(mm))) return false; - len = PAGE_ALIGN(len_in); + len = mm_page_align(mm, len_in); /* Check to see whether len was rounded up from small -ve to zero */ if (len_in && !len) @@ -1865,14 +1951,29 @@ static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior) * operation. This function returns true in the cases, otherwise false. In * the former case we store an error on @err. */ -static bool madvise_should_skip(unsigned long start, size_t len_in, +static bool madvise_should_skip(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior, int *err) { - if (!is_valid_madvise(start, len_in, behavior)) { + if (!is_valid_madvise(mm, start, len_in, behavior)) { *err = -EINVAL; return true; } - if (start + PAGE_ALIGN(len_in) == start) { +#ifdef CONFIG_MM_SUBPAGE + /* These engines still assume whole native folios/tables. */ + if (mm_page_size(mm) != PAGE_SIZE) { + switch (behavior) { + case MADV_MERGEABLE: + case MADV_UNMERGEABLE: + case MADV_HUGEPAGE: + case MADV_COLLAPSE: + case MADV_HWPOISON: + case MADV_SOFT_OFFLINE: + *err = -EOPNOTSUPP; + return true; + } + } +#endif + if (start + mm_page_align(mm, len_in) == start) { *err = 0; return true; } @@ -1919,7 +2020,7 @@ static int madvise_do_behavior(unsigned long start, size_t len_in, } range->start = get_untagged_addr(madv_behavior->mm, start); - range->end = range->start + PAGE_ALIGN(len_in); + range->end = range->start + mm_page_align(madv_behavior->mm, len_in); blk_start_plug(&plug); if (is_madvise_populate(madv_behavior)) @@ -2012,7 +2113,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh .tlb = &tlb, }; - if (madvise_should_skip(start, len_in, behavior, &error)) + if (madvise_should_skip(mm, start, len_in, behavior, &error)) return error; error = madvise_lock(&madv_behavior); if (error) @@ -2055,7 +2156,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, size_t len_in = iter_iov_len(iter); int error; - if (madvise_should_skip(start, len_in, behavior, &error)) + if (madvise_should_skip(mm, start, len_in, behavior, &error)) ret = error; else ret = madvise_do_behavior(start, len_in, &madv_behavior); @@ -2185,9 +2286,9 @@ static int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, .anon_name = anon_name, }; - if (start & ~PAGE_MASK) + if (start & ~mm_page_mask(mm)) return -EINVAL; - len = (len_in + ~PAGE_MASK) & PAGE_MASK; + len = mm_page_align(mm, len_in); /* Check to see whether len was rounded up from small -ve to zero */ if (len_in && !len) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index d47aef256a3249..964c002533131a 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -67,6 +67,9 @@ #include "swap.h" #include "internal.h" +#ifdef CONFIG_MM_SUBPAGE +#include "user-subpage-internal.h" +#endif static int sysctl_memory_failure_early_kill __read_mostly; @@ -539,6 +542,37 @@ struct task_struct *task_early_kill(struct task_struct *tsk, int force_early) return find_early_kill_thread(tsk); } +#ifdef CONFIG_MM_SUBPAGE +struct subpage_collect_procs { + const struct page *page; + struct list_head *to_kill; + int force_early; +}; + +static bool collect_procs_subpage(struct folio *folio, struct vm_area_struct *vma, + struct rmap_walk_range range, void *arg) +{ + struct subpage_collect_procs *args = arg; + DEFINE_FOLIO_RMAP_WALK(walk, folio, vma, range, PVMW_SYNC); + struct task_struct *tsk; + unsigned long address; + + if (!page_vma_mapped_walk(&walk)) + return true; + address = walk.address; + page_vma_mapped_walk_done(&walk); + rcu_read_lock(); + for_each_process(tsk) { + struct task_struct *t = task_early_kill(tsk, args->force_early); + + if (t && t->mm == vma->vm_mm) + add_to_kill_anon_file(t, args->page, vma, args->to_kill, address); + } + rcu_read_unlock(); + return true; +} +#endif + /* * Collect processes when the error hit an anonymous page. */ @@ -549,6 +583,17 @@ static void collect_procs_anon(const struct folio *folio, struct task_struct *tsk; struct anon_vma *av; pgoff_t pgoff; +#ifdef CONFIG_MM_SUBPAGE + struct subpage_collect_procs args = { page, to_kill, force_early }; + struct rmap_walk_control rwc = { + .rmap_one = collect_procs_subpage, + .arg = &args, + }; + + if (!folio_test_large(folio) && + rmap_walk_subpages((struct folio *)folio, &rwc, false)) + return; +#endif av = folio_lock_anon_vma_read(folio, NULL); if (av == NULL) /* Not actually mapped anymore */ diff --git a/mm/memory.c b/mm/memory.c index a903d5c1859cb6..8e58ea71b88c0f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -87,6 +87,9 @@ #include #include "pgalloc-track.h" +#include +#include "user-subpage-internal.h" + #include "internal.h" #include "swap.h" @@ -191,26 +194,26 @@ static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, unsigned long start; start = addr; - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(tlb->mm, pud, addr); do { - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(tlb->mm, addr, end); if (pmd_none_or_clear_bad(pmd)) continue; free_pte_range(tlb, pmd, addr); } while (pmd++, addr = next, addr != end); - start &= PUD_MASK; + start &= pud_mask_mm(tlb->mm); if (start < floor) return; if (ceiling) { - ceiling &= PUD_MASK; + ceiling &= pud_mask_mm(tlb->mm); if (!ceiling) return; } if (end - 1 > ceiling - 1) return; - pmd = pmd_offset(pud, start); + pmd = pmd_offset_mm(tlb->mm, pud, start); pud_clear(pud); pmd_free_tlb(tlb, pmd, start); mm_dec_nr_pmds(tlb->mm); @@ -225,27 +228,27 @@ static inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, unsigned long start; start = addr; - pud = pud_offset(p4d, addr); + pud = pud_offset_mm(tlb->mm, p4d, addr); do { - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(tlb->mm, addr, end); if (pud_none_or_clear_bad(pud)) continue; free_pmd_range(tlb, pud, addr, next, floor, ceiling); } while (pud++, addr = next, addr != end); - start &= P4D_MASK; + start &= p4d_mask_mm(tlb->mm); if (start < floor) return; if (ceiling) { - ceiling &= P4D_MASK; + ceiling &= p4d_mask_mm(tlb->mm); if (!ceiling) return; } if (end - 1 > ceiling - 1) return; - pud = pud_offset(p4d, start); - p4d_clear(p4d); + pud = pud_offset_mm(tlb->mm, p4d, start); + p4d_clear_mm(tlb->mm, p4d); pud_free_tlb(tlb, pud, start); mm_dec_nr_puds(tlb->mm); } @@ -259,26 +262,26 @@ static inline void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd, unsigned long start; start = addr; - p4d = p4d_offset(pgd, addr); + p4d = p4d_offset_mm(tlb->mm, pgd, addr); do { - next = p4d_addr_end(addr, end); - if (p4d_none_or_clear_bad(p4d)) + next = p4d_addr_end_mm(tlb->mm, addr, end); + if (p4d_none_or_clear_bad_mm(tlb->mm, p4d)) continue; free_pud_range(tlb, p4d, addr, next, floor, ceiling); } while (p4d++, addr = next, addr != end); - start &= PGDIR_MASK; + start &= pgd_mask_mm(tlb->mm); if (start < floor) return; if (ceiling) { - ceiling &= PGDIR_MASK; + ceiling &= pgd_mask_mm(tlb->mm); if (!ceiling) return; } if (end - 1 > ceiling - 1) return; - p4d = p4d_offset(pgd, start); + p4d = p4d_offset_mm(tlb->mm, pgd, start); pgd_clear(pgd); p4d_free_tlb(tlb, p4d, start); } @@ -320,7 +323,7 @@ void free_pgd_range(struct mmu_gather *tlb, * Wherever addr is brought up or ceiling brought down, we must * be careful to reject "the opposite 0" before it confuses the * subsequent tests. But what about where end is brought down - * by PMD_SIZE below? no, end can't go down to 0 there. + * by one PMD span below? no, end can't go down to 0 there. * * Whereas we round start (addr) and ceiling down, by different * masks at different levels, in order to test whether a table @@ -328,19 +331,19 @@ void free_pgd_range(struct mmu_gather *tlb, * bother to round floor or end up - the tests don't need that. */ - addr &= PMD_MASK; + addr &= pmd_mask_mm(tlb->mm); if (addr < floor) { - addr += PMD_SIZE; + addr += pmd_size_mm(tlb->mm); if (!addr) return; } if (ceiling) { - ceiling &= PMD_MASK; + ceiling &= pmd_mask_mm(tlb->mm); if (!ceiling) return; } if (end - 1 > ceiling - 1) - end -= PMD_SIZE; + end -= pmd_size_mm(tlb->mm); if (addr > end - 1) return; /* @@ -350,7 +353,7 @@ void free_pgd_range(struct mmu_gather *tlb, tlb_change_page_size(tlb, PAGE_SIZE); pgd = pgd_offset(tlb->mm, addr); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(tlb->mm, addr, end); if (pgd_none_or_clear_bad(pgd)) continue; free_p4d_range(tlb, pgd, addr, next, floor, ceiling); @@ -539,16 +542,16 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add return; } - p4dp = p4d_offset(pgdp, addr); + p4dp = p4d_offset_mm(mm, pgdp, addr); p4d = p4dp_get(p4dp); p4dv = p4d_val(p4d); - if (!p4d_present(p4d) || p4d_leaf(p4d)) { + if (!p4d_present_mm(mm, p4d) || p4d_leaf(p4d)) { pr_alert("pgd:%08llx p4d:%08llx\n", pgdv, p4dv); return; } - pudp = pud_offset(p4dp, addr); + pudp = pud_offset_mm(mm, p4dp, addr); pud = pudp_get(pudp); pudv = pud_val(pud); @@ -557,7 +560,7 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add return; } - pmdp = pmd_offset(pudp, addr); + pmdp = pmd_offset_mm(mm, pudp, addr); pmd = pmdp_get(pmdp); pmdv = pmd_val(pmd); @@ -950,7 +953,15 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, struct page *page; if (likely(softleaf_is_swap(entry))) { - if (swap_dup_entry_direct(entry) < 0) + int err; + +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(src_mm) < PAGE_SIZE) + err = swap_subpage_dup(entry, pte_swp_subpage_offset(orig_pte)); + else +#endif + err = swap_dup_entries_direct(entry, mm_pte_native_pages(src_mm)); + if (err < 0) return -EIO; /* make sure dst_mm is on swapoff's mmlist. */ @@ -982,6 +993,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, entry = make_readable_migration_entry( swp_offset(entry)); pte = softleaf_to_pte(entry); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(src_mm) < PAGE_SIZE) + pte = pte_swp_set_subpage_offset(pte, pte_swp_subpage_offset(orig_pte)); +#endif if (pte_swp_soft_dirty(orig_pte)) pte = pte_swp_mksoft_dirty(pte); if (pte_swp_uffd_wp(orig_pte)) @@ -1076,10 +1091,14 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma * over and copy the page & arm it. */ - if (copy_mc_user_highpage(&new_folio->page, page, addr, src_vma)) - return -EHWPOISON; + for (unsigned int i = 0; i < mm_pte_native_pages(src_vma->vm_mm); i++) { + if (copy_mc_user_highpage(folio_page(new_folio, i), page + i, + addr + i * PAGE_SIZE, src_vma)) + return -EHWPOISON; + } *prealloc = NULL; + folio_ref_add(new_folio, mm_pte_native_pages(dst_vma->vm_mm) - 1); __folio_mark_uptodate(new_folio); folio_add_new_anon_rmap(new_folio, dst_vma, addr, RMAP_EXCLUSIVE); folio_add_lru_vma(new_folio, dst_vma); @@ -1095,6 +1114,59 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return 0; } +#ifdef CONFIG_MM_SUBPAGE +/* One new native folio may publish several independently locked user leaves. */ +static void subpage_add_new_lru(struct folio *folio, struct vm_area_struct *vma, + unsigned int mapped) +{ + folio_add_lru_vma(folio, vma); + if ((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED) + while (--mapped) + mlock_folio(folio); +} + +/* Destination allocation, folio locking and rmap preflight precede both PTLs. */ +static int copy_pinned_subpage(struct vm_area_struct *dst_vma, pte_t *dst_pte, + pte_t source_pte, unsigned long addr, int *rss, + struct mm_subpage *source, struct mm_subpage **prealloc) +{ + struct mm_subpage *dest = *prealloc; + struct folio *new; + pte_t pte; + bool fresh; + int ret; + + if (!dest) + return -EAGAIN; + new = mm_subpage_folio(dest); + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); + ret = copy_user_subpage_range(&new->page, mm_subpage_offset(dest), + &mm_subpage_folio(source)->page, mm_subpage_offset(source), + mm_subpage_size(source)); + if (ret) + return ret; + fresh = !mm_subpage_anon_root(dest); + + mm_subpage_set_exclusive(dest); + ret = fresh ? mm_subpage_add_new_anon_rmap(dest, dst_vma, addr) : + mm_subpage_add_anon_rmap(dest, dst_vma, addr); + if (!ret) { + if (fresh) + subpage_add_new_lru(new, dst_vma, 1); + rss[MM_ANONPAGES]++; + pte = phys_pte_mm(dst_vma->vm_mm, mm_subpage_phys(dest), dst_vma->vm_page_prot); + pte = maybe_mkwrite(pte_mkdirty(pte), dst_vma); + if (userfaultfd_pte_wp(dst_vma, source_pte)) + pte = pte_mkuffd_wp(pte); + set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); + *prealloc = NULL; + folio_unlock(new); + } + /* An unconsumed reservation remains locked until the caller drops PTLs. */ + return ret; +} +#endif + static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, pte_t pte, unsigned long addr, int nr) @@ -1128,18 +1200,46 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, static inline int copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, pte_t pte, unsigned long addr, - int max_nr, int *rss, struct folio **prealloc) + int max_nr, int *rss, struct folio **prealloc, + struct mm_subpage **subprealloc) { fpb_t flags = FPB_MERGE_WRITE; struct page *page; struct folio *folio; - int err, nr; + int err, nr, nr_native; + unsigned int span = mm_pte_native_pages(src_vma->vm_mm); page = vm_normal_page(src_vma, addr, pte); if (unlikely(!page)) goto copy_pte; folio = page_folio(page); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(src_vma->vm_mm) < PAGE_SIZE) { + if (folio_test_anon(folio)) { + struct mm_subpage *slot; + + slot = mm_subpage_get_from_phys(pte_phys_mm(src_vma->vm_mm, pte)); + if (WARN_ON_ONCE(!slot)) + return -EINVAL; + err = mm_subpage_dup_anon_rmap(slot, dst_vma, src_vma, addr); + if (err == -EBUSY) { + err = copy_pinned_subpage(dst_vma, dst_pte, pte, addr, + rss, slot, subprealloc); + mm_subpage_put(slot); + return err ? err : 1; + } + if (err) { + mm_subpage_put(slot); + return err; + } + /* The lookup reference becomes the child's PTE reference. */ + rss[MM_ANONPAGES]++; + goto copy_pte; + } + max_nr = 1; + } +#endif /* * If we likely have to copy, just don't bother with batching. Make @@ -1152,18 +1252,25 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma if (vma_soft_dirty_enabled(src_vma)) flags |= FPB_RESPECT_SOFT_DIRTY; - nr = folio_pte_batch_flags(folio, src_vma, src_pte, &pte, max_nr, flags); - folio_ref_add(folio, nr); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(src_vma->vm_mm) > PAGE_SIZE) + nr = folio_subpage_pte_batch_flags(src_vma, folio, src_pte, + &pte, max_nr, flags); + else +#endif + nr = folio_pte_batch_flags(folio, src_vma, src_pte, &pte, max_nr, flags); + nr_native = nr * span; + folio_ref_add(folio, nr_native); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, - nr, dst_vma, src_vma))) { - folio_ref_sub(folio, nr); + nr_native, dst_vma, src_vma))) { + folio_ref_sub(folio, nr_native); return -EAGAIN; } rss[MM_ANONPAGES] += nr; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { - folio_dup_file_rmap_ptes(folio, page, nr, dst_vma); + folio_dup_file_rmap_ptes(folio, page, nr_native, dst_vma); rss[mm_counter_file(folio)] += nr; } __copy_present_ptes(dst_vma, src_vma, dst_pte, src_pte, pte, @@ -1171,7 +1278,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return nr; } - folio_get(folio); + folio_ref_add(folio, span); if (folio_test_anon(folio)) { /* * If this page may have been pinned by the parent process, @@ -1181,7 +1288,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma */ if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, dst_vma, src_vma))) { /* Page may be pinned, we have to copy. */ - folio_put(folio); + folio_put_refs(folio, span); err = copy_present_page(dst_vma, src_vma, dst_pte, src_pte, addr, rss, prealloc, page); return err ? err : 1; @@ -1189,6 +1296,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma rss[MM_ANONPAGES]++; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { + mm_subpage_file_map_add(dst_vma, pte_phys_mm(src_vma->vm_mm, pte)); folio_dup_file_rmap_pte(folio, page, dst_vma); rss[mm_counter_file(folio)]++; } @@ -1202,11 +1310,12 @@ static inline struct folio *folio_prealloc(struct mm_struct *src_mm, struct vm_area_struct *vma, unsigned long addr, bool need_zero) { struct folio *new_folio; + unsigned int order = max_t(unsigned int, mm_page_shift(src_mm), PAGE_SHIFT) - PAGE_SHIFT; if (need_zero) - new_folio = vma_alloc_zeroed_movable_folio(vma, addr); + new_folio = vma_alloc_zeroed_movable_folio_order(vma, addr, order); else - new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, addr); + new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, order, vma, addr); if (!new_folio) return NULL; @@ -1220,6 +1329,163 @@ static inline struct folio *folio_prealloc(struct mm_struct *src_mm, return new_folio; } +#ifdef CONFIG_MM_SUBPAGE +/* Skipping a busy owner must not trigger an avoidable global or memcg OOM. */ +static struct folio *cow_subpage_speculative_folio(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr) +{ + gfp_t alloc_gfp = GFP_HIGHUSER_MOVABLE | __GFP_NORETRY | __GFP_NOWARN; + gfp_t charge_gfp = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; + unsigned int order = max_t(unsigned int, mm_page_shift(mm), PAGE_SHIFT) - PAGE_SHIFT; + struct folio *folio = vma_alloc_folio(alloc_gfp, order, vma, addr); + + if (!folio) + return NULL; + if (mem_cgroup_charge(folio, mm, charge_gfp)) { + folio_put(folio); + return NULL; + } + folio_throttle_swaprate(folio, charge_gfp); + return folio; +} + +/* The returned initialized slot owns a reference and has its folio locked. */ +static struct mm_subpage *cow_subpage_prealloc(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr, + struct mm_subpage *source, bool force_private, bool *pooled, + struct mm_struct *charge_mm) +{ + struct mm_subpage_pool *pool; + struct mm_subpage *slot; + struct folio *folio; + unsigned int offset = vma_page_offset_at(vma, addr).offset; + unsigned int retired_retries = 0; + int err; + +retry_private: + pool = force_private ? NULL : + mm_subpage_cow_pool_get_charged(mm, charge_mm, vma, addr); + *pooled = pool != NULL; + if (!pool) + pool = mm_subpage_pool_create_granule(GFP_KERNEL, mm_page_shift(mm)); + if (!pool) + return ERR_PTR(-ENOMEM); + for (;;) { + if (*pooled) + slot = source ? mm_subpage_copy_locked(pool, source) : + mm_subpage_alloc_locked(pool); + else + slot = source ? mm_subpage_copy_at_locked(pool, source, offset) : + mm_subpage_alloc_at_locked(pool, offset); + if (!IS_ERR(slot) || PTR_ERR(slot) != -EAGAIN) + break; + folio = NULL; + if (*pooled) { + folio = cow_subpage_speculative_folio(charge_mm, vma, addr); + if (!folio && mm_subpage_cow_wait_busy(pool)) { + cond_resched(); + continue; + } + } + /* No reusable owner exists after speculative allocation failed. + * Ordinary allocation/charge semantics are now appropriate. + */ + if (!folio) + folio = folio_prealloc(charge_mm, vma, addr, false); + if (!folio) { + slot = ERR_PTR(-ENOMEM); + break; + } + /* A concurrent cgroup move may change the fresh folio's charge. */ + if (*pooled && !mm_subpage_cow_folio_matches(pool, folio)) { + folio_put(folio); + mm_subpage_pool_put(pool); + force_private = true; + goto retry_private; + } + err = *pooled ? mm_subpage_pool_add_folio(pool, folio, + GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN) : + mm_subpage_pool_add_folio_at(pool, folio, offset, GFP_KERNEL); + if (*pooled && err == -ENOMEM) { + if (mm_subpage_cow_wait_busy(pool)) { + folio_put(folio); + cond_resched(); + continue; + } + err = mm_subpage_pool_add_folio(pool, folio, GFP_KERNEL); + } + if (err) { + folio_put(folio); + if (err == -EEXIST) { + cond_resched(); + continue; + } + slot = ERR_PTR(err); + break; + } + } + if (!*pooled) + mm_subpage_pool_close(pool); + mm_subpage_pool_put(pool); + if (*pooled && IS_ERR(slot) && PTR_ERR(slot) == -ESHUTDOWN) { + if (++retired_retries > 1) + force_private = true; + goto retry_private; + } + return slot; +} + +/* The child owns the pool; fork retains the source mm allocation charge. */ +static struct mm_subpage *fork_subpage_prealloc(struct vm_area_struct *dst_vma, + struct mm_struct *src_mm, unsigned long addr) +{ + struct mm_subpage *slot; + bool pooled, force_private = false; + int err; + + for (;;) { + slot = cow_subpage_prealloc(dst_vma->vm_mm, dst_vma, addr, + NULL, force_private, &pooled, src_mm); + if (IS_ERR(slot)) + return slot; + err = mm_subpage_prepare_anon_rmap(slot, dst_vma, addr); + if (!err) + return slot; + folio_unlock(mm_subpage_folio(slot)); + mm_subpage_put(slot); + if (!pooled || err != -ENOMEM) + return ERR_PTR(err); + force_private = true; + } +} + +static struct mm_subpage *anon_subpage_prealloc(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr) +{ + struct mm_subpage_pool *pool = mm_subpage_pool_create_granule(GFP_KERNEL, mm_page_shift(mm)); + struct mm_subpage *slot = ERR_PTR(-ENOMEM); + struct folio *folio; + int ret; + + if (!pool) + return slot; + folio = folio_prealloc(mm, vma, addr, false); + if (!folio) + goto out; + ret = mm_subpage_pool_add_folio(pool, folio, GFP_KERNEL); + if (ret) { + folio_put(folio); + slot = ERR_PTR(ret); + goto out; + } + slot = mm_subpage_alloc_at(pool, vma_page_offset_at(vma, addr).offset); +out: + mm_subpage_pool_close(pool); + mm_subpage_pool_put(pool); + return slot; +} +#endif + static int copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, @@ -1236,6 +1502,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, int rss[NR_MM_COUNTERS]; softleaf_t entry = softleaf_mk_none(); struct folio *prealloc = NULL; + struct mm_subpage *subprealloc = NULL; + unsigned long subprealloc_addr __maybe_unused = 0; int nr; again: @@ -1276,6 +1544,14 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, do { nr = 1; +#ifdef CONFIG_MM_SUBPAGE + /* A vanished source PTE must not move a prepared slot to another VA. */ + if (subprealloc && addr != subprealloc_addr) { + folio_unlock(mm_subpage_folio(subprealloc)); + mm_subpage_put(subprealloc); + subprealloc = NULL; + } +#endif /* * We are holding two locks at this point - either of them @@ -1316,16 +1592,24 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, WARN_ON_ONCE(ret != -ENOENT); } /* copy_present_ptes() will clear `*prealloc' if consumed */ - max_nr = (end - addr) / PAGE_SIZE; + max_nr = (end - addr) >> mm_page_shift(src_mm); ret = copy_present_ptes(dst_vma, src_vma, dst_pte, src_pte, - ptent, addr, max_nr, rss, &prealloc); + ptent, addr, max_nr, rss, &prealloc, &subprealloc); /* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again. * If copy failed due to hwpoison in source page, break out. */ - if (unlikely(ret == -EAGAIN || ret == -EHWPOISON)) + if (unlikely(ret == -EAGAIN || ret == -EHWPOISON || + (IS_ENABLED(CONFIG_MM_SUBPAGE) && ret < 0))) break; +#ifdef CONFIG_MM_SUBPAGE + if (subprealloc) { + folio_unlock(mm_subpage_folio(subprealloc)); + mm_subpage_put(subprealloc); + subprealloc = NULL; + } +#endif if (unlikely(prealloc)) { /* * pre-alloc page cannot be reused by next time so as @@ -1338,7 +1622,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, } nr = ret; progress += 8 * nr; - } while (dst_pte += nr, src_pte += nr, addr += PAGE_SIZE * nr, + } while (dst_pte += nr, src_pte += nr, addr += mm_page_size(src_mm) * nr, addr != end); lazy_mmu_mode_disable(); @@ -1357,10 +1641,23 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, } else if (ret == -EBUSY || unlikely(ret == -EHWPOISON)) { goto out; } else if (ret == -EAGAIN) { +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(src_mm) < PAGE_SIZE) { + subprealloc_addr = addr; + subprealloc = fork_subpage_prealloc(dst_vma, src_mm, addr); + if (IS_ERR(subprealloc)) + return PTR_ERR(subprealloc); + ret = 0; + goto again; + } +#endif prealloc = folio_prealloc(src_mm, src_vma, addr, false); if (!prealloc) return -ENOMEM; } else if (ret < 0) { + /* Subpage ownership checks can return additional fatal errors. */ + if (IS_ENABLED(CONFIG_MM_SUBPAGE)) + goto out; VM_WARN_ON_ONCE(1); } @@ -1370,6 +1667,12 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, if (addr != end) goto again; out: +#ifdef CONFIG_MM_SUBPAGE + if (subprealloc) { + folio_unlock(mm_subpage_folio(subprealloc)); + mm_subpage_put(subprealloc); + } +#endif if (unlikely(prealloc)) folio_put(prealloc); return ret; @@ -1388,9 +1691,9 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, dst_pmd = pmd_alloc(dst_mm, dst_pud, addr); if (!dst_pmd) return -ENOMEM; - src_pmd = pmd_offset(src_pud, addr); + src_pmd = pmd_offset_mm(src_vma->vm_mm, src_pud, addr); do { - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(src_vma->vm_mm, addr, end); if (pmd_is_huge(*src_pmd)) { int err; @@ -1425,9 +1728,9 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, dst_pud = pud_alloc(dst_mm, dst_p4d, addr); if (!dst_pud) return -ENOMEM; - src_pud = pud_offset(src_p4d, addr); + src_pud = pud_offset_mm(src_vma->vm_mm, src_p4d, addr); do { - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(src_vma->vm_mm, addr, end); if (pud_trans_huge(*src_pud)) { int err; @@ -1461,10 +1764,10 @@ copy_p4d_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, dst_p4d = p4d_alloc(dst_mm, dst_pgd, addr); if (!dst_p4d) return -ENOMEM; - src_p4d = p4d_offset(src_pgd, addr); + src_p4d = p4d_offset_mm(src_vma->vm_mm, src_pgd, addr); do { - next = p4d_addr_end(addr, end); - if (p4d_none_or_clear_bad(src_p4d)) + next = p4d_addr_end_mm(src_vma->vm_mm, addr, end); + if (p4d_none_or_clear_bad_mm(src_vma->vm_mm, src_p4d)) continue; if (copy_pud_range(dst_vma, src_vma, dst_p4d, src_p4d, addr, next)) @@ -1516,6 +1819,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) bool is_cow; int ret; + if (mm_page_shift(dst_mm) != mm_page_shift(src_mm)) + return -EINVAL; + if (!vma_needs_copy(dst_vma, src_vma)) return 0; @@ -1549,7 +1855,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) dst_pgd = pgd_offset(dst_mm, addr); src_pgd = pgd_offset(src_mm, addr); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(src_vma->vm_mm, addr, end); if (pgd_none_or_clear_bad(src_pgd)) continue; if (unlikely(copy_p4d_range(dst_vma, src_vma, dst_pgd, src_pgd, @@ -1629,7 +1935,7 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, if (--nr == 0) break; pte++; - addr += PAGE_SIZE; + addr += mm_page_size(vma->vm_mm); } return was_installed; @@ -1642,10 +1948,12 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, bool *force_flush, bool *force_break, bool *any_skipped) { struct mm_struct *mm = tlb->mm; + unsigned int nr_native = nr * mm_pte_native_pages(mm); bool delay_rmap = false; if (!folio_test_anon(folio)) { ptent = get_and_clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); + mm_subpage_file_map_del(vma, pte_phys_mm(mm, ptent)); if (pte_dirty(ptent)) { folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { @@ -1669,16 +1977,44 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, nr, details, ptent); if (!delay_rmap) { - folio_remove_rmap_ptes(folio, page, nr, vma); + folio_remove_rmap_ptes(folio, page, nr_native, vma); if (unlikely(folio_mapcount(folio) < 0)) print_bad_pte(vma, addr, ptent, page); } - if (unlikely(__tlb_remove_folio_pages(tlb, page, nr, delay_rmap))) { + /* Gather payloads count native references, not logical PTEs. */ + if (unlikely(__tlb_remove_folio_pages(tlb, page, nr_native, delay_rmap))) { + *force_flush = true; + *force_break = true; + } +} + +#ifdef CONFIG_MM_SUBPAGE +static void zap_present_subpage(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned long addr, int *rss, bool *force_flush, + bool *force_break, bool *any_skipped) +{ + struct mm_subpage *slot = mm_subpage_get_from_phys(pte_phys_mm(tlb->mm, ptent)); + + /* All small-granule anonymous PTEs must carry slot ownership. */ + if (WARN_ON_ONCE(!slot)) { + *any_skipped = true; + return; + } + ptep_get_and_clear_full(tlb->mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + mm_subpage_remove_anon_rmap(slot, vma); + rss[MM_ANONPAGES]--; + /* Drop the lookup ref, then transfer the PTE ref to the TLB batch. */ + mm_subpage_put(slot); + if (__tlb_remove_subpage(tlb, slot)) { *force_flush = true; *force_break = true; } } +#endif /* * Zap or skip at least one present PTE, trying to batch-process subsequent @@ -1716,12 +2052,30 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, return 1; } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + if (folio_test_anon(folio)) { + zap_present_subpage(tlb, vma, pte, ptent, addr, rss, + force_flush, force_break, any_skipped); + return 1; + } + /* Native folio batching still expresses physical PAGE_SIZE steps. */ + max_nr = 1; + } +#endif + /* * Make sure that the common "small folio" case is as fast as possible * by keeping the batching logic separate. */ if (unlikely(folio_test_large(folio) && max_nr != 1)) { - nr = folio_pte_batch(folio, pte, ptent, max_nr); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) > PAGE_SIZE) + nr = folio_subpage_pte_batch(vma, folio, pte, ptent, + max_nr, 0); + else +#endif + nr = folio_pte_batch(folio, pte, ptent, max_nr); zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush, force_break, any_skipped); @@ -1764,9 +2118,20 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, if (!should_zap_cows(details)) return 1; - nr = swap_pte_batch(pte, max_nr, ptent); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + swap_subpage_put(entry, pte_swp_subpage_offset(ptent), true); + } else +#endif + { + unsigned int span = mm_pte_native_pages(vma->vm_mm); + + /* Adjacent coarse PTEs advance by a full group of swap slots. */ + if (span == 1) + nr = swap_pte_batch(pte, max_nr, ptent); + swap_put_entries_direct(entry, nr * span); + } rss[MM_SWAPENTS] -= nr; - swap_put_entries_direct(entry, nr); } else if (softleaf_is_migration(entry)) { struct folio *folio = softleaf_to_folio(entry); @@ -1811,7 +2176,7 @@ static inline int do_zap_pte_range(struct mmu_gather *tlb, bool *any_skipped) { pte_t ptent = ptep_get(pte); - int max_nr = (end - addr) / PAGE_SIZE; + int max_nr = (end - addr) >> mm_page_shift(vma->vm_mm); int nr = 0; /* Skip all consecutive none ptes */ @@ -1825,7 +2190,7 @@ static inline int do_zap_pte_range(struct mmu_gather *tlb, if (!max_nr) return nr; pte += nr; - addr += nr * PAGE_SIZE; + addr += nr * mm_page_size(vma->vm_mm); } if (pte_present(ptent)) @@ -1839,13 +2204,14 @@ static inline int do_zap_pte_range(struct mmu_gather *tlb, return nr; } -static bool pte_table_reclaim_possible(unsigned long start, unsigned long end, +static bool pte_table_reclaim_possible(struct mm_struct *mm, + unsigned long start, unsigned long end, struct zap_details *details) { if (!IS_ENABLED(CONFIG_PT_RECLAIM)) return false; /* Only zap if we are allowed to and cover the full page table. */ - return details && details->reclaim_pt && (end - start >= PMD_SIZE); + return details && details->reclaim_pt && (end - start >= pmd_size_mm(mm)); } static bool zap_empty_pte_table(struct mm_struct *mm, pmd_t *pmd, @@ -1877,7 +2243,7 @@ static bool zap_pte_table_if_empty(struct mm_struct *mm, pmd_t *pmd, if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); - for (i = 0, pte = start_pte; i < PTRS_PER_PTE; i++, pte++) { + for (i = 0, pte = start_pte; i < pte_table_bytes_mm(mm) / sizeof(pte_t); i++, pte++) { if (!pte_none(ptep_get(pte))) goto out_ptl; } @@ -1902,7 +2268,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, struct zap_details *details) { - bool can_reclaim_pt = pte_table_reclaim_possible(addr, end, details); + bool can_reclaim_pt = pte_table_reclaim_possible(tlb->mm, addr, end, details); bool force_flush = false, force_break = false; struct mm_struct *mm = tlb->mm; int rss[NR_MM_COUNTERS]; @@ -1915,7 +2281,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, int nr; retry: - tlb_change_page_size(tlb, PAGE_SIZE); + tlb_change_page_size(tlb, mm_page_size(mm)); init_rss_vec(rss); start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!pte) @@ -1936,11 +2302,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (any_skipped) can_reclaim_pt = false; if (unlikely(force_break)) { - addr += nr * PAGE_SIZE; + addr += nr * mm_page_size(vma->vm_mm); direct_reclaim = false; break; } - } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); + } while (pte += nr, addr += mm_page_size(mm) * nr, addr != end); /* * Fast path: try to hold the pmd lock and unmap the PTE page. @@ -1997,9 +2363,9 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(tlb->mm, pud, addr); do { - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(tlb->mm, addr, end); if (pmd_is_huge(*pmd)) { if (next - addr != HPAGE_PMD_SIZE) __split_huge_pmd(vma, pmd, addr, false); @@ -2033,9 +2399,9 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb, pud_t *pud; unsigned long next; - pud = pud_offset(p4d, addr); + pud = pud_offset_mm(tlb->mm, p4d, addr); do { - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(tlb->mm, addr, end); if (pud_trans_huge(*pud)) { if (next - addr != HPAGE_PUD_SIZE) split_huge_pud(vma, pud, addr); @@ -2061,10 +2427,10 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, p4d_t *p4d; unsigned long next; - p4d = p4d_offset(pgd, addr); + p4d = p4d_offset_mm(tlb->mm, pgd, addr); do { - next = p4d_addr_end(addr, end); - if (p4d_none_or_clear_bad(p4d)) + next = p4d_addr_end_mm(tlb->mm, addr, end); + if (p4d_none_or_clear_bad_mm(tlb->mm, p4d)) continue; next = zap_pud_range(tlb, vma, p4d, addr, next, details); } while (p4d++, addr = next, addr != end); @@ -2103,7 +2469,7 @@ static void __zap_vma_range(struct mmu_gather *tlb, struct vm_area_struct *vma, tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(tlb->mm, addr, end); if (pgd_none_or_clear_bad(pgd)) continue; next = zap_p4d_range(tlb, vma, pgd, addr, next, details); @@ -2350,7 +2716,7 @@ static int validate_page_before_insert(struct vm_area_struct *vma, static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, - pgprot_t prot, bool mkwrite) + pgprot_t prot, bool mkwrite, unsigned int offset) { struct folio *folio = page_folio(page); pte_t pteval = ptep_get(pte); @@ -2360,7 +2726,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, return -EBUSY; /* see insert_pfn(). */ - if (pte_pfn(pteval) != page_to_pfn(page)) { + if (pte_phys_mm(vma->vm_mm, pteval) != page_to_phys(page) + offset) { WARN_ON_ONCE(!is_zero_pfn(pte_pfn(pteval))); return -EFAULT; } @@ -2372,17 +2738,18 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, } /* Ok, finally just insert the thing.. */ - pteval = mk_pte(page, prot); + pteval = phys_pte_mm(vma->vm_mm, page_to_phys(page) + offset, prot); if (unlikely(is_zero_folio(folio))) { pteval = pte_mkspecial(pteval); } else { folio_get(folio); - pteval = mk_pte(page, prot); + pteval = phys_pte_mm(vma->vm_mm, page_to_phys(page) + offset, prot); if (mkwrite) { pteval = pte_mkyoung(pteval); pteval = maybe_mkwrite(pte_mkdirty(pteval), vma); } inc_mm_counter(vma->vm_mm, mm_counter_file(folio)); + mm_subpage_file_map_add(vma, pte_phys_mm(vma->vm_mm, pteval)); folio_add_file_rmap_pte(folio, page, vma); } set_pte_at(vma->vm_mm, addr, pte, pteval); @@ -2390,7 +2757,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, } static int insert_page(struct vm_area_struct *vma, unsigned long addr, - struct page *page, pgprot_t prot, bool mkwrite) + struct page *page, pgprot_t prot, bool mkwrite, unsigned int offset) { int retval; pte_t *pte; @@ -2404,83 +2771,120 @@ static int insert_page(struct vm_area_struct *vma, unsigned long addr, if (!pte) goto out; retval = insert_page_into_pte_locked(vma, pte, addr, page, prot, - mkwrite); + mkwrite, offset); pte_unmap_unlock(pte, ptl); out: return retval; } -static int insert_page_in_batch_locked(struct vm_area_struct *vma, pte_t *pte, - unsigned long addr, struct page *page, pgprot_t prot) +/* Keep lock batching in units of hardware leaves, backing in native pages. */ +static int insert_page_range(struct vm_area_struct *vma, unsigned long addr, + struct page **pages, unsigned long offset, + unsigned long *length, pgprot_t prot) { - int err; + struct mm_struct *mm = vma->vm_mm; + unsigned long remaining = *length; + unsigned long size = mm_page_size(mm); + int ret = 0; - err = validate_page_before_insert(vma, page); - if (err) - return err; - return insert_page_into_pte_locked(vma, pte, addr, page, prot, false); + while (remaining) { + unsigned long in_pmd = min(remaining, + pmd_size_mm(mm) - (addr & ~pmd_mask_mm(mm))); + pmd_t *pmd = walk_to_pmd(mm, addr); + + if (!pmd) { + ret = -EFAULT; + break; + } + if (pte_alloc(mm, pmd)) { + ret = -ENOMEM; + break; + } + while (in_pmd) { + unsigned int batch = min_t(unsigned long, in_pmd / size, 8); + unsigned int i; + spinlock_t *ptl; + pte_t *start = pte_offset_map_lock(mm, pmd, addr, &ptl); + + if (!start) { + ret = -EFAULT; + goto out; + } + for (i = 0; i < batch; i++) { + struct page *page = pages[offset >> PAGE_SHIFT]; + + ret = validate_page_before_insert(vma, page); + if (!ret) + ret = insert_page_into_pte_locked(vma, start + i, + addr, page, prot, false, offset_in_page(offset)); + if (ret) + break; + addr += size; + offset += size; + remaining -= size; + in_pmd -= size; + } + pte_unmap_unlock(start, ptl); + if (ret) + goto out; + } + } +out: + *length = remaining; + return ret; } -/* insert_pages() amortizes the cost of spinlock operations - * when inserting pages in a loop. - */ static int insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num, pgprot_t prot) { - pmd_t *pmd = NULL; - pte_t *start_pte, *pte; - spinlock_t *pte_lock; - struct mm_struct *const mm = vma->vm_mm; - unsigned long curr_page_idx = 0; - unsigned long remaining_pages_total = *num; - unsigned long pages_to_write_in_pmd; + unsigned long length; int ret; -more: - ret = -EFAULT; - pmd = walk_to_pmd(mm, addr); - if (!pmd) - goto out; - - pages_to_write_in_pmd = min_t(unsigned long, - remaining_pages_total, PTRS_PER_PTE - pte_index(addr)); - /* Allocate the PTE if necessary; takes PMD lock once only. */ - ret = -ENOMEM; - if (pte_alloc(mm, pmd)) - goto out; + if (check_mul_overflow(*num, PAGE_SIZE, &length)) + return -EOVERFLOW; + ret = insert_page_range(vma, addr, pages, 0, &length, prot); + /* A partially inserted native page is still an incomplete input page. */ + *num = DIV_ROUND_UP(length, PAGE_SIZE); + return ret; +} - while (pages_to_write_in_pmd) { - int pte_idx = 0; - const int batch_size = min_t(int, pages_to_write_in_pmd, 8); +/** + * vm_insert_pages_range - map a byte range from a native-page array + * @vma: target VMA, held under the mmap write lock + * @addr: first user address + * @pages: native backing pages + * @nr_pages: size of @pages + * @offset: byte offset within the backing array + * @length: bytes to map + * + * Address, offset and length must match the target mm's granule. A failed + * insertion can leave a prefix mapped, as with vm_insert_pages(); the caller + * must unmap that prefix when abandoning the mapping. Physical ownership + * remains in native pages. Each installed user leaf acquires its own ref. + */ +int vm_insert_pages_range(struct vm_area_struct *vma, unsigned long addr, + struct page **pages, unsigned long nr_pages, + unsigned long offset, unsigned long length) +{ + unsigned long bytes, end, size = mm_page_size(vma->vm_mm); - start_pte = pte_offset_map_lock(mm, pmd, addr, &pte_lock); - if (!start_pte) { - ret = -EFAULT; - goto out; - } - for (pte = start_pte; pte_idx < batch_size; ++pte, ++pte_idx) { - int err = insert_page_in_batch_locked(vma, pte, - addr, pages[curr_page_idx], prot); - if (unlikely(err)) { - pte_unmap_unlock(start_pte, pte_lock); - ret = err; - remaining_pages_total -= pte_idx; - goto out; - } - addr += PAGE_SIZE; - ++curr_page_idx; - } - pte_unmap_unlock(start_pte, pte_lock); - pages_to_write_in_pmd -= batch_size; - remaining_pages_total -= batch_size; + if (!IS_ALIGNED(addr | offset | length, size) || + check_mul_overflow(nr_pages, PAGE_SIZE, &bytes) || + offset > bytes || length > bytes - offset || + check_add_overflow(addr, length, &end) || + !range_in_vma(vma, addr, end)) + return -EINVAL; + if (!length) + return 0; + if (vma->vm_flags & VM_PFNMAP) + return -EINVAL; + if (!(vma->vm_flags & VM_MIXEDMAP)) { + mmap_assert_write_locked(vma->vm_mm); + vm_flags_set(vma, VM_MIXEDMAP); } - if (remaining_pages_total) - goto more; - ret = 0; -out: - *num = remaining_pages_total; - return ret; + return insert_page_range(vma, addr, pages, offset, &length, vma->vm_page_prot); } +EXPORT_SYMBOL_GPL(vm_insert_pages_range); /** * vm_insert_pages - insert multiple pages into user vma, batching the pmd lock. @@ -2501,9 +2905,12 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num) { const unsigned long nr_pages = *num; - const unsigned long end = addr + PAGE_SIZE * nr_pages; + unsigned long length, end; - if (!range_in_vma(vma, addr, end)) + if (!IS_ALIGNED(addr, mm_page_size(vma->vm_mm)) || + check_mul_overflow(nr_pages, PAGE_SIZE, &length) || + check_add_overflow(addr, length, &end) || + !range_in_vma(vma, addr, end)) return -EFAULT; if (!(vma->vm_flags & VM_MIXEDMAP)) { VM_WARN_ON_ONCE(mmap_read_trylock(vma->vm_mm)); @@ -2528,8 +2935,10 @@ int map_kernel_pages_prepare(struct vm_area_desc *desc) } nr_pages = action->map_kernel.nr_pages; - end = addr + PAGE_SIZE * nr_pages; - if (!range_in_vma_desc(desc, addr, end)) + if (!IS_ALIGNED(addr, mm_page_size(desc->mm)) || + check_mul_overflow(nr_pages, PAGE_SIZE, &end) || + check_add_overflow(addr, end, &end) || + !range_in_vma_desc(desc, addr, end)) return -EFAULT; return 0; @@ -2588,10 +2997,35 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, BUG_ON(vma->vm_flags & VM_PFNMAP); vm_flags_set(vma, VM_MIXEDMAP); } - return insert_page(vma, addr, page, vma->vm_page_prot, false); + return insert_page(vma, addr, page, vma->vm_page_prot, false, 0); } EXPORT_SYMBOL(vm_insert_page); +#ifdef CONFIG_MM_SUBPAGE +/* Explicit physical offset; the VMA and supplied page retain native ownership. */ +vm_fault_t vmf_insert_page_offset(struct vm_area_struct *vma, unsigned long addr, + struct page *page, unsigned int offset) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned long size = mm_page_size(mm); + int err; + + if (!(vma->vm_flags & VM_MIXEDMAP) || (vma->vm_flags & VM_PFNMAP) || + (addr & (size - 1)) || (offset & (size - 1)) || + offset > PAGE_SIZE - size || addr < vma->vm_start || + addr >= vma->vm_end || vma->vm_end - addr < size) + return VM_FAULT_SIGBUS; + err = insert_page(vma, addr, page, vma->vm_page_prot, false, offset); + if (err == -ENOMEM) + return VM_FAULT_OOM; + if (err < 0 && err != -EBUSY) + return VM_FAULT_SIGBUS; + return VM_FAULT_NOPAGE; +} +EXPORT_SYMBOL_GPL(vmf_insert_page_offset); +#endif + + /* * __vm_map_pages - maps range of kernel pages into user vma * @vma: user vma to map to @@ -2606,20 +3040,22 @@ EXPORT_SYMBOL(vm_insert_page); * Return: 0 on success and error code otherwise. */ static int __vm_map_pages(struct vm_area_struct *vma, struct page **pages, - unsigned long num, unsigned long offset) + unsigned long num, unsigned long offset, + unsigned int subpage_offset) { - unsigned long count = vma_pages(vma); - unsigned long uaddr = vma->vm_start; + unsigned long bytes, available; - /* Fail if the user requested offset is beyond the end of the object */ - if (offset >= num) + if (offset >= num || + check_mul_overflow(num - offset, PAGE_SIZE, &available) || + subpage_offset >= PAGE_SIZE || + check_mul_overflow(offset, PAGE_SIZE, &bytes) || + check_add_overflow(bytes, (unsigned long)subpage_offset, &bytes)) return -ENXIO; - - /* Fail if the user requested size exceeds available object size */ - if (count > num - offset) + if (vma->vm_end - vma->vm_start > + available - subpage_offset) return -ENXIO; - - return vm_insert_pages(vma, uaddr, pages + offset, &count); + return vm_insert_pages_range(vma, vma->vm_start, pages, num, bytes, + vma->vm_end - vma->vm_start); } /** @@ -2643,7 +3079,7 @@ static int __vm_map_pages(struct vm_area_struct *vma, struct page **pages, int vm_map_pages(struct vm_area_struct *vma, struct page **pages, unsigned long num) { - return __vm_map_pages(vma, pages, num, vma->vm_pgoff); + return __vm_map_pages(vma, pages, num, vma->vm_pgoff, vma_subpage_offset(vma)); } EXPORT_SYMBOL(vm_map_pages); @@ -2663,12 +3099,12 @@ EXPORT_SYMBOL(vm_map_pages); int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, unsigned long num) { - return __vm_map_pages(vma, pages, num, 0); + return __vm_map_pages(vma, pages, num, 0, 0); } EXPORT_SYMBOL(vm_map_pages_zero); static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, pgprot_t prot, bool mkwrite) + unsigned long pfn, pgprot_t prot, bool mkwrite, unsigned int offset) { struct mm_struct *mm = vma->vm_mm; pte_t *pte, entry; @@ -2690,7 +3126,7 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, * allocation and mapping invalidation so just skip the * update. */ - if (pte_pfn(entry) != pfn) { + if (pte_phys_mm(mm, entry) != PFN_PHYS(pfn) + offset) { WARN_ON_ONCE(!is_zero_pfn(pte_pfn(entry))); goto out_unlock; } @@ -2703,7 +3139,7 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, } /* Ok, finally just insert the thing.. */ - entry = pte_mkspecial(pfn_pte(pfn, prot)); + entry = pte_mkspecial(phys_pte_mm(mm, PFN_PHYS(pfn) + offset, prot)); if (mkwrite) { entry = pte_mkyoung(entry); @@ -2765,9 +3201,9 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, * Context: Process context. May allocate using %GFP_KERNEL. * Return: vm_fault_t value. */ -vm_fault_t vmf_insert_pfn_prot_mkwrite(struct vm_area_struct *vma, +static vm_fault_t __vmf_insert_pfn_prot_mkwrite(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, pgprot_t pgprot, - bool mkwrite) + bool mkwrite, unsigned int offset) { /* * Technically, architectures with pte_special can avoid all these @@ -2789,10 +3225,31 @@ vm_fault_t vmf_insert_pfn_prot_mkwrite(struct vm_area_struct *vma, pfnmap_setup_cachemode_pfn(pfn, &pgprot); - return insert_pfn(vma, addr, pfn, pgprot, mkwrite); + return insert_pfn(vma, addr, pfn, pgprot, mkwrite, offset); +} +vm_fault_t vmf_insert_pfn_prot_mkwrite(struct vm_area_struct *vma, + unsigned long addr, unsigned long pfn, pgprot_t pgprot, + bool mkwrite) +{ + return __vmf_insert_pfn_prot_mkwrite(vma, addr, pfn, pgprot, mkwrite, 0); } EXPORT_SYMBOL(vmf_insert_pfn_prot_mkwrite); +/* Explicit quarter offset; a native PFN alone cannot encode a 4K leaf. */ +vm_fault_t vmf_insert_pfn_prot_mkwrite_offset(struct vm_area_struct *vma, + unsigned long addr, unsigned long pfn, unsigned int offset, + pgprot_t pgprot, bool mkwrite) +{ + unsigned long size = mm_page_size(vma->vm_mm); + + if (!IS_ALIGNED(addr | offset, size) || offset > PAGE_SIZE - size || + addr < vma->vm_start || addr >= vma->vm_end || + vma->vm_end - addr < size) + return VM_FAULT_SIGBUS; + return __vmf_insert_pfn_prot_mkwrite(vma, addr, pfn, pgprot, mkwrite, offset); +} +EXPORT_SYMBOL_GPL(vmf_insert_pfn_prot_mkwrite_offset); + static bool vm_mixed_ok(struct vm_area_struct *vma, unsigned long pfn, bool mkwrite) { @@ -2840,9 +3297,9 @@ static vm_fault_t __vm_insert_mixed(struct vm_area_struct *vma, * result in pfn_t_has_page() == false. */ page = pfn_to_page(pfn); - err = insert_page(vma, addr, page, pgprot, mkwrite); + err = insert_page(vma, addr, page, pgprot, mkwrite, 0); } else { - return insert_pfn(vma, addr, pfn, pgprot, mkwrite); + return insert_pfn(vma, addr, pfn, pgprot, mkwrite, 0); } if (err == -ENOMEM) @@ -2863,7 +3320,7 @@ vm_fault_t vmf_insert_page_mkwrite(struct vm_fault *vmf, struct page *page, if (addr < vmf->vma->vm_start || addr >= vmf->vma->vm_end) return VM_FAULT_SIGBUS; - err = insert_page(vmf->vma, addr, page, pgprot, write); + err = insert_page(vmf->vma, addr, page, pgprot, write, 0); if (err == -ENOMEM) return VM_FAULT_OOM; if (err < 0 && err != -EBUSY) @@ -2898,7 +3355,7 @@ vm_fault_t vmf_insert_mixed_mkwrite(struct vm_area_struct *vma, */ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, unsigned long end, - unsigned long pfn, pgprot_t prot) + phys_addr_t phys, pgprot_t prot) { pte_t *pte, *mapped_pte; spinlock_t *ptl; @@ -2910,13 +3367,13 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, lazy_mmu_mode_enable(); do { BUG_ON(!pte_none(ptep_get(pte))); - if (!pfn_modify_allowed(pfn, prot)) { + if (!pfn_modify_allowed(PHYS_PFN(phys), prot)) { err = -EACCES; break; } - set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot))); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); + set_pte_at(mm, addr, pte, pte_mkspecial(phys_pte_mm(mm, phys, prot))); + phys += mm_page_size(mm); + } while (pte++, addr += mm_page_size(mm), addr != end); lazy_mmu_mode_disable(); pte_unmap_unlock(mapped_pte, ptl); return err; @@ -2924,67 +3381,67 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud, unsigned long addr, unsigned long end, - unsigned long pfn, pgprot_t prot) + phys_addr_t phys, pgprot_t prot) { pmd_t *pmd; unsigned long next; int err; - pfn -= addr >> PAGE_SHIFT; pmd = pmd_alloc(mm, pud, addr); if (!pmd) return -ENOMEM; VM_BUG_ON(pmd_trans_huge(*pmd)); do { - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(mm, addr, end); err = remap_pte_range(mm, pmd, addr, next, - pfn + (addr >> PAGE_SHIFT), prot); + phys, prot); if (err) return err; + phys += next - addr; } while (pmd++, addr = next, addr != end); return 0; } static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d, unsigned long addr, unsigned long end, - unsigned long pfn, pgprot_t prot) + phys_addr_t phys, pgprot_t prot) { pud_t *pud; unsigned long next; int err; - pfn -= addr >> PAGE_SHIFT; pud = pud_alloc(mm, p4d, addr); if (!pud) return -ENOMEM; do { - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(mm, addr, end); err = remap_pmd_range(mm, pud, addr, next, - pfn + (addr >> PAGE_SHIFT), prot); + phys, prot); if (err) return err; + phys += next - addr; } while (pud++, addr = next, addr != end); return 0; } static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd, unsigned long addr, unsigned long end, - unsigned long pfn, pgprot_t prot) + phys_addr_t phys, pgprot_t prot) { p4d_t *p4d; unsigned long next; int err; - pfn -= addr >> PAGE_SHIFT; p4d = p4d_alloc(mm, pgd, addr); if (!p4d) return -ENOMEM; do { - next = p4d_addr_end(addr, end); + next = p4d_addr_end_mm(mm, addr, end); err = remap_pud_range(mm, p4d, addr, next, - pfn + (addr >> PAGE_SHIFT), prot); + phys, prot); if (err) return err; + phys += next - addr; } while (p4d++, addr = next, addr != end); return 0; } @@ -3009,29 +3466,30 @@ static int get_remap_pgoff(bool is_cow, unsigned long addr, } static int remap_pfn_range_internal(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, unsigned long size, pgprot_t prot) + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) { pgd_t *pgd; unsigned long next; - unsigned long end = addr + PAGE_ALIGN(size); struct mm_struct *mm = vma->vm_mm; + unsigned long end = addr + mm_page_align(mm, size); + phys_addr_t phys = PFN_PHYS(pfn) + offset; int err; - if (WARN_ON_ONCE(!PAGE_ALIGNED(addr))) + if (WARN_ON_ONCE(!IS_ALIGNED(addr, mm_page_size(mm)))) return -EINVAL; VM_WARN_ON_ONCE(!vma_test_all_mask(vma, VMA_REMAP_FLAGS)); BUG_ON(addr >= end); - pfn -= addr >> PAGE_SHIFT; pgd = pgd_offset(mm, addr); flush_cache_range(vma, addr, end); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(mm, addr, end); err = remap_p4d_range(mm, pgd, addr, next, - pfn + (addr >> PAGE_SHIFT), prot); + phys, prot); if (err) return err; + phys += next - addr; } while (pgd++, addr = next, addr != end); return 0; @@ -3042,9 +3500,9 @@ static int remap_pfn_range_internal(struct vm_area_struct *vma, unsigned long ad * must have pre-validated the caching bits of the pgprot_t. */ static int remap_pfn_range_notrack(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, unsigned long size, pgprot_t prot) + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) { - int error = remap_pfn_range_internal(vma, addr, pfn, size, prot); + int error = remap_pfn_range_internal(vma, addr, pfn, offset, size, prot); if (!error) return 0; @@ -3088,12 +3546,17 @@ void pfnmap_track_ctx_release(struct kref *ref) } static int remap_pfn_range_track(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, unsigned long size, pgprot_t prot) + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) { struct pfnmap_track_ctx *ctx = NULL; + unsigned long track_size; int err; - size = PAGE_ALIGN(size); + size = mm_page_align(vma->vm_mm, size); + if (check_add_overflow(size, (unsigned long)offset, &track_size) || + track_size > ULONG_MAX - (PAGE_SIZE - 1)) + return -EINVAL; + track_size = PAGE_ALIGN(track_size); /* * If we cover the full VMA, we'll perform actual tracking, and @@ -3107,14 +3570,14 @@ static int remap_pfn_range_track(struct vm_area_struct *vma, unsigned long addr, if (addr == vma->vm_start && addr + size == vma->vm_end) { if (vma->pfnmap_track_ctx) return -EINVAL; - ctx = pfnmap_track_ctx_alloc(pfn, size, &prot); + ctx = pfnmap_track_ctx_alloc(pfn, track_size, &prot); if (IS_ERR(ctx)) return PTR_ERR(ctx); - } else if (pfnmap_setup_cachemode(pfn, size, &prot)) { + } else if (pfnmap_setup_cachemode(pfn, track_size, &prot)) { return -EINVAL; } - err = remap_pfn_range_notrack(vma, addr, pfn, size, prot); + err = remap_pfn_range_notrack(vma, addr, pfn, offset, size, prot); if (ctx) { if (err) kref_put(&ctx->kref, pfnmap_track_ctx_release); @@ -3125,15 +3588,15 @@ static int remap_pfn_range_track(struct vm_area_struct *vma, unsigned long addr, } static int do_remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, unsigned long size, pgprot_t prot) + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) { - return remap_pfn_range_track(vma, addr, pfn, size, prot); + return remap_pfn_range_track(vma, addr, pfn, offset, size, prot); } #else static int do_remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, - unsigned long pfn, unsigned long size, pgprot_t prot) + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) { - return remap_pfn_range_notrack(vma, addr, pfn, size, prot); + return remap_pfn_range_notrack(vma, addr, pfn, offset, size, prot); } #endif @@ -3143,16 +3606,23 @@ int remap_pfn_range_prepare(struct vm_area_desc *desc) const unsigned long start = action->remap.start; const unsigned long end = start + action->remap.size; const unsigned long pfn = action->remap.start_pfn; + const unsigned int offset = mmap_action_remap_offset(action); const bool is_cow = vma_desc_is_cow_mapping(desc); int err; if (!range_in_vma_desc(desc, start, end)) return -EFAULT; + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, mm_page_size(desc->mm))) + return -EINVAL; err = get_remap_pgoff(is_cow, start, end, desc->start, desc->end, pfn, &desc->pgoff); if (err) return err; +#ifdef CONFIG_MM_SUBPAGE + if (is_cow) + desc->subpage_offset = offset; +#endif vma_desc_set_flags_mask(desc, VMA_REMAP_FLAGS); return 0; @@ -3160,9 +3630,9 @@ int remap_pfn_range_prepare(struct vm_area_desc *desc) static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, - unsigned long size) + unsigned int offset, unsigned long size) { - const unsigned long end = addr + PAGE_ALIGN(size); + const unsigned long end = addr + mm_page_align(vma->vm_mm, size); const bool is_cow = is_cow_mapping(vma->vm_flags); int err; @@ -3171,10 +3641,41 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, if (err) return err; + if (is_cow) + vma_set_page_offset(vma, (struct vm_page_offset) { pfn, offset }); vma_set_flags_mask(vma, VMA_REMAP_FLAGS); return 0; } +/** + * remap_pfn_range_offset - remap a physical range starting within a native page + * @vma: user VMA to map to + * @addr: target address, aligned to the VMA's mapping granule + * @pfn: native page frame containing the first byte + * @offset: byte offset within @pfn, aligned to the mapping granule + * @size: size of the mapping, in bytes + * @prot: page protection flags + * + * The caller must hold the mmap lock. The physical offset is independent of + * the target virtual address. References and error cleanup match remap_pfn_range(). + * + * Return: 0 on success, a negative error code otherwise. + */ +int remap_pfn_range_offset(struct vm_area_struct *vma, unsigned long addr, + unsigned long pfn, unsigned int offset, unsigned long size, pgprot_t prot) +{ + int err; + + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, mm_page_size(vma->vm_mm))) + return -EINVAL; + err = remap_pfn_range_prepare_vma(vma, addr, pfn, offset, size); + if (err) + return err; + + return do_remap_pfn_range(vma, addr, pfn, offset, size, prot); +} +EXPORT_SYMBOL(remap_pfn_range_offset); + /** * remap_pfn_range - remap kernel memory to userspace * @vma: user vma to map to @@ -3190,13 +3691,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, unsigned long size, pgprot_t prot) { - int err; - - err = remap_pfn_range_prepare_vma(vma, addr, pfn, size); - if (err) - return err; - - return do_remap_pfn_range(vma, addr, pfn, size, prot); + return remap_pfn_range_offset(vma, addr, pfn, 0, size, prot); } EXPORT_SYMBOL(remap_pfn_range); @@ -3208,40 +3703,43 @@ int remap_pfn_range_complete(struct vm_area_struct *vma, const unsigned long size = action->remap.size; const pgprot_t prot = action->remap.pgprot; - return do_remap_pfn_range(vma, start, pfn, size, prot); + return do_remap_pfn_range(vma, start, pfn, mmap_action_remap_offset(action), size, prot); } -static int __simple_ioremap_prep(unsigned long vm_len, pgoff_t vm_pgoff, +static int __simple_ioremap_prep(struct mm_struct *mm, unsigned long vm_len, + struct vm_page_offset pos, phys_addr_t start_phys, unsigned long size, - unsigned long *pfnp) + struct vm_page_offset *physical) { - unsigned long pfn, pages; + unsigned long granule = mm_page_size(mm); + unsigned long offset, bias = start_phys & (granule - 1); /* Check that the physical memory area passed in looks valid */ if (start_phys + size < start_phys) return -EINVAL; /* - * You *really* shouldn't map things that aren't page-aligned, - * but we've historically allowed it because IO memory might - * just have smaller alignment. + * Preserve the historical allowance for unaligned I/O resources, using + * the target MM's granule for the prefix and final partial mapping. */ - size += start_phys & ~PAGE_MASK; - pfn = start_phys >> PAGE_SHIFT; - pages = (size + ~PAGE_MASK) >> PAGE_SHIFT; - if (pfn + pages < pfn) + if (check_add_overflow(size, bias, &size) || size > ULONG_MAX - (granule - 1)) return -EINVAL; - - /* We start the mapping 'vm_pgoff' pages into the area */ - if (vm_pgoff > pages) + size = ALIGN(size, granule); + start_phys -= bias; + if (start_phys + size < start_phys) + return -EINVAL; + if (pos.offset >= PAGE_SIZE || !IS_ALIGNED(pos.offset, granule) || + check_mul_overflow(pos.index, PAGE_SIZE, &offset) || + check_add_overflow(offset, (unsigned long)pos.offset, &offset)) return -EINVAL; - pfn += vm_pgoff; - pages -= vm_pgoff; - /* Can we fit all of the mapping? */ - if ((vm_len >> PAGE_SHIFT) > pages) + if (offset > size || vm_len > size - offset) return -EINVAL; - *pfnp = pfn; + start_phys += offset; + physical->index = PHYS_PFN(start_phys); + physical->offset = offset_in_page(start_phys); + if (PFN_PHYS(physical->index) + physical->offset != start_phys) + return -EINVAL; return 0; } @@ -3250,16 +3748,22 @@ int simple_ioremap_prepare(struct vm_area_desc *desc) struct mmap_action *action = &desc->action; const phys_addr_t start = action->simple_ioremap.start_phys_addr; const unsigned long size = action->simple_ioremap.size; - unsigned long pfn; + struct vm_page_offset physical, pos = { .index = desc->pgoff }; int err; - err = __simple_ioremap_prep(vma_desc_size(desc), desc->pgoff, - start, size, &pfn); +#ifdef CONFIG_MM_SUBPAGE + pos.offset = desc->subpage_offset; +#endif + err = __simple_ioremap_prep(desc->mm, vma_desc_size(desc), pos, + start, size, &physical); if (err) return err; /* The I/O remap logic does the heavy lifting. */ - mmap_action_ioremap_full(desc, pfn); + mmap_action_ioremap_full(desc, physical.index); +#ifdef CONFIG_MM_SUBPAGE + action->remap.offset = physical.offset; +#endif return io_remap_pfn_range_prepare(desc); } @@ -3283,15 +3787,17 @@ int vm_iomap_memory(struct vm_area_struct *vma, phys_addr_t start, unsigned long const unsigned long vm_start = vma->vm_start; const unsigned long vm_end = vma->vm_end; const unsigned long vm_len = vm_end - vm_start; - unsigned long pfn; + struct vm_page_offset physical; int err; - err = __simple_ioremap_prep(vm_len, vma->vm_pgoff, start, len, &pfn); + err = __simple_ioremap_prep(vma->vm_mm, vm_len, vma_page_offset_at(vma, vm_start), + start, len, &physical); if (err) return err; /* Ok, let it rip */ - return io_remap_pfn_range(vma, vma->vm_start, pfn, vm_len, vma->vm_page_prot); + return io_remap_pfn_range_offset(vma, vm_start, physical.index, physical.offset, + vm_len, vma->vm_page_prot); } EXPORT_SYMBOL(vm_iomap_memory); @@ -3540,15 +4046,14 @@ static inline int pte_unmap_same(struct vm_fault *vmf) * -EHWPOISON: copy failed due to hwpoison in source page * -EAGAIN: copied failed (some other reason) */ -static inline int __wp_page_copy_user(struct page *dst, struct page *src, - struct vm_fault *vmf) +static inline int __wp_native_page_copy_user(struct page *dst, struct page *src, + struct vm_fault *vmf, unsigned long addr) { int ret; void *kaddr; void __user *uaddr; struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; - unsigned long addr = vmf->address; if (likely(src)) { if (copy_mc_user_highpage(dst, src, addr, vma)) @@ -3638,15 +4143,31 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, return ret; } -static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma) +static int __wp_page_copy_user(struct page *dst, struct page *src, + struct vm_fault *vmf) { - struct file *vm_file = vma->vm_file; + unsigned int i, nr = mm_pte_native_pages(vmf->vma->vm_mm); - if (vm_file) - return mapping_gfp_mask(vm_file->f_mapping) | __GFP_FS | __GFP_IO; + for (i = 0; i < nr; i++) { + int ret = __wp_native_page_copy_user(dst + i, src ? src + i : NULL, + vmf, vmf->address + i * PAGE_SIZE); - /* - * Special mappings (e.g. VDSO) do not have any file so fake + if (ret) + return ret; + kmsan_copy_page_meta(dst + i, src ? src + i : NULL); + } + return 0; +} + +static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma) +{ + struct file *vm_file = vma->vm_file; + + if (vm_file) + return mapping_gfp_mask(vm_file->f_mapping) | __GFP_FS | __GFP_IO; + + /* + * Special mappings (e.g. VDSO) do not have any file so fake * a default GFP_KERNEL for them. */ return GFP_KERNEL; @@ -3883,14 +4404,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) delayacct_wpcopy_end(); return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } - kmsan_copy_page_meta(&new_folio->page, vmf->page); } __folio_mark_uptodate(new_folio); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, - vmf->address & PAGE_MASK, - (vmf->address & PAGE_MASK) + PAGE_SIZE); + vmf->address & mm_page_mask(mm), + (vmf->address & mm_page_mask(mm)) + mm_page_size(mm)); mmu_notifier_invalidate_range_start(&range); /* @@ -3927,6 +4447,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * some TLBs while the old PTE remains in others. */ ptep_clear_flush(vma, vmf->address, vmf->pte); + folio_ref_add(new_folio, mm_pte_native_pages(mm) - 1); folio_add_new_anon_rmap(new_folio, vma, vmf->address, RMAP_EXCLUSIVE); folio_add_lru_vma(new_folio, vma); BUG_ON(unshare && pte_write(entry)); @@ -3970,7 +4491,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) mmu_notifier_invalidate_range_end(&range); if (new_folio) - folio_put(new_folio); + folio_put_refs(new_folio, page_copied ? mm_pte_native_pages(mm) : 1); if (old_folio) { if (page_copied) free_swap_cache(old_folio); @@ -4092,7 +4613,7 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_MM_ID static bool __wp_can_reuse_large_anon_folio(struct folio *folio, struct vm_area_struct *vma) { @@ -4158,18 +4679,18 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio, folio_unlock_large_mapcount(folio); return exclusive; } -#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +#else /* !CONFIG_MM_ID */ static bool __wp_can_reuse_large_anon_folio(struct folio *folio, struct vm_area_struct *vma) { BUILD_BUG(); } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_MM_ID */ static bool wp_can_reuse_anon_folio(struct folio *folio, struct vm_area_struct *vma) { - if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio)) + if (IS_ENABLED(CONFIG_MM_ID) && folio_test_large(folio)) return __wp_can_reuse_large_anon_folio(folio, vma); /* @@ -4207,6 +4728,187 @@ static bool wp_can_reuse_anon_folio(struct folio *folio, return true; } +#ifdef CONFIG_MM_SUBPAGE +/* Enter with the original PTE locked; a slot, not its native folio, is copied. */ +static vm_fault_t wp_subpage_copy(struct vm_fault *vmf) + __releases(vmf->ptl) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct mm_subpage *source = NULL, *dest = NULL; + struct folio *old = vmf->page ? page_folio(vmf->page) : NULL; + struct folio *new = NULL; + struct mmu_notifier_range range; + bool copied = false, unshare = vmf->flags & FAULT_FLAG_UNSHARE; + bool force_private = false, pooled, fresh, retry_private = false; + vm_fault_t ret = VM_FAULT_SIGBUS; + pte_t entry; + int err; + + if (old && folio_test_anon(old)) { + source = mm_subpage_get_from_phys(pte_phys_mm(mm, vmf->orig_pte)); + if (WARN_ON_ONCE(!source)) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return VM_FAULT_SIGBUS; + } + if (!mm_subpage_is_exclusive(source) && mm_subpage_mapcount(source) == 1 && + folio_trylock(old)) { + bool reusable = true; + + if (folio_test_swapcache(old)) { + swp_entry_t entry = page_swap_entry(&old->page); + + reusable = !swap_subpage_count(entry, mm_subpage_offset(source)) && + !(folio_test_writeback(old) && + (folio_swap_flags(old) & SWP_STABLE_WRITES)); + } + if (reusable) + mm_subpage_try_reuse_anon(source); + folio_unlock(old); + } + if (mm_subpage_is_exclusive(source)) { + if (unshare) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + } else { + folio_xchg_last_cpupid(old, (1 << LAST_CPUPID_SHIFT) - 1); + /* Exclusivity belongs to the slot, not PageAnonExclusive. */ + wp_page_reuse(vmf, NULL); + } + mm_subpage_put(source); + return 0; + } + } else if (old) { + folio_get(old); + } else if (!is_zero_pfn(pte_pfn(vmf->orig_pte))) { + /* Private device/PFN mappings need their own subpage contract. */ + pte_unmap_unlock(vmf->pte, vmf->ptl); + return VM_FAULT_SIGBUS; + } + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; + delayacct_wpcopy_start(); + ret = vmf_anon_prepare(vmf); + if (ret) + goto out; + ret = VM_FAULT_OOM; +retry_copy: + dest = cow_subpage_prealloc(mm, vma, vmf->address, source, force_private, + &pooled, mm); + if (IS_ERR(dest)) { + if (PTR_ERR(dest) == -EHWPOISON) + ret = VM_FAULT_HWPOISON; + dest = NULL; + goto out; + } + new = mm_subpage_folio(dest); + if (old && !source) { + err = copy_user_subpage_range(&new->page, mm_subpage_offset(dest), vmf->page, + pte_phys_mm(mm, vmf->orig_pte) & ~PAGE_MASK, + mm_page_size(mm)); + if (err) { + folio_unlock(new); + ret = VM_FAULT_HWPOISON; + goto out; + } + } + /* Initialisation is complete; the slot reference prevents migration/free. + * Do not carry a folio lock through MMU notifier callbacks. + */ + folio_unlock(new); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, + vmf->address, vmf->address + mm_page_size(mm)); + mmu_notifier_invalidate_range_start(&range); + /* Keep the folio alive after a concurrent zap can consume its PTE ref. */ + folio_get(new); + folio_lock(new); + ret = 0; + /* A different fault may publish this owner's first PTE while we wait. + * Only the first publisher establishes folio rmap/LRU. Preflight nonlinear + * slot identity before taking PTL, with a fresh-folio fallback on ENOMEM. + */ + fresh = !mm_subpage_anon_root(dest); + err = mm_subpage_prepare_anon_rmap(dest, vma, vmf->address); + if (err) { + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + retry_private = pooled && err == -ENOMEM; + goto unlock_folio; + } + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto unlock_folio; + if (!pte_same(ptep_get(vmf->pte), vmf->orig_pte)) { + update_mmu_tlb(vma, vmf->address, vmf->pte); + goto unlock_pte; + } + mm_subpage_set_exclusive(dest); + err = fresh ? mm_subpage_add_new_anon_rmap(dest, vma, vmf->address) : + mm_subpage_add_anon_rmap(dest, vma, vmf->address); + if (WARN_ON_ONCE(err)) { + ret = VM_FAULT_SIGBUS; + goto unlock_pte; + } + entry = pte_sw_mkyoung(phys_pte_mm(mm, mm_subpage_phys(dest), vma->vm_page_prot)); + if (unshare) { + if (pte_soft_dirty(vmf->orig_pte)) + entry = pte_mksoft_dirty(entry); + if (pte_uffd_wp(vmf->orig_pte)) + entry = pte_mkuffd_wp(entry); + } else { + entry = maybe_mkwrite(pte_mkdirty(entry), vma); + } + BUG_ON(unshare && pte_write(entry)); + flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); + /* Break-before-make also orders removal of the old slot's rmap. */ + ptep_clear_flush(vma, vmf->address, vmf->pte); + if (fresh) + subpage_add_new_lru(new, vma, 1); + set_pte_at(mm, vmf->address, vmf->pte, entry); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + if (source) { + mm_subpage_remove_anon_rmap(source, vma); + } else if (old) { + mm_subpage_file_map_del(vma, pte_phys_mm(mm, vmf->orig_pte)); + folio_remove_rmap_pte(old, vmf->page, vma); + dec_mm_counter(mm, mm_counter_file(old)); + inc_mm_counter(mm, MM_ANONPAGES); + } else { + ksm_might_unmap_zero_page(mm, vmf->orig_pte); + inc_mm_counter(mm, MM_ANONPAGES); + } + dest = NULL; /* The new PTE owns the reference. */ + copied = true; +unlock_pte: + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; +unlock_folio: + folio_unlock(new); + folio_put(new); + mmu_notifier_invalidate_range_end(&range); + if (retry_private) { + mm_subpage_put(dest); + dest = NULL; + force_private = true; + retry_private = false; + ret = VM_FAULT_OOM; + goto retry_copy; + } +out: + if (dest) + mm_subpage_put(dest); + if (source) { + if (copied) + mm_subpage_put(source); /* Retired PTE reference, after TLBI. */ + mm_subpage_put(source); /* Source stabilisation reference. */ + } else if (old) { + if (copied) + folio_put(old); + folio_put(old); + } + delayacct_wpcopy_end(); + return ret; +} +#endif + /* * This routine handles present pages, when * * users try to write to a shared page (FAULT_FLAG_WRITE) @@ -4292,6 +4994,11 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return wp_page_shared(vmf, folio); } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return wp_subpage_copy(vmf); +#endif + /* * Private mapping: create an exclusive anonymous page copy if reuse * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. @@ -4301,8 +5008,12 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) */ if (folio && folio_test_anon(folio) && (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) { - if (!PageAnonExclusive(vmf->page)) - SetPageAnonExclusive(vmf->page); + if (!PageAnonExclusive(vmf->page)) { + unsigned int i, nr = mm_pte_native_pages(vma->vm_mm); + + for (i = 0; i < nr; i++) + SetPageAnonExclusive(vmf->page + i); + } if (unlikely(unshare)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; @@ -4340,12 +5051,116 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, start = vma->vm_start + ((start_idx - vma->vm_pgoff) << PAGE_SHIFT); size = (end_idx - start_idx) << PAGE_SHIFT; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + unsigned long end = vma->vm_end; + + start = vma_address(vma, first_index, last_index - first_index + 1); + /* The interval tree guarantees a nonempty intersection. */ + VM_BUG_ON_VMA(IS_ERR_VALUE(start), vma); + if (last_index < vma_last_pgoff(vma)) + end = vma_address_at_offset(vma, (struct vm_page_offset) { + .index = last_index + 1, + }); + size = end - start; + } else if (mm_page_size(vma->vm_mm) > PAGE_SIZE) { + unsigned long granule = mm_page_size(vma->vm_mm); + unsigned long end = min(start + size, vma->vm_end); + + /* A native-page invalidation removes every overlapping leaf. */ + start = max(round_down(start, granule), vma->vm_start); + end = min(round_up(end, granule), vma->vm_end); + size = end - start; + } +#endif + tlb_gather_mmu(&tlb, vma->vm_mm); zap_vma_range_batched(&tlb, vma, start, size, details); tlb_finish_mmu(&tlb); } } +#ifdef CONFIG_MM_SUBPAGE +/* Unmap affected user leaves, including coarse aliases; preserve private COW. */ +void unmap_mapping_subpage_hole(struct address_space *mapping, loff_t first, u64 last) +{ + struct vm_area_struct *vma; + struct zap_details details = { .skip_cows = true }; + + i_mmap_lock_read(mapping); + vma_interval_tree_foreach(vma, &mapping->i_mmap, + first >> PAGE_SHIFT, (last - 1) >> PAGE_SHIFT) { + unsigned long start, end, granule = mm_page_size(vma->vm_mm); + u64 low = round_down((u64)first, granule), high = round_up(last, granule); + struct vm_page_offset begin = { low >> PAGE_SHIFT, low & ~PAGE_MASK }; + struct vm_page_offset finish = { high >> PAGE_SHIFT, high & ~PAGE_MASK }; + struct vm_page_offset vbegin = vma_page_offset_at(vma, vma->vm_start); + struct vm_page_offset vend = vma_page_offset_at(vma, vma->vm_end); + struct mmu_gather tlb; + + if (begin.index < vbegin.index || + (begin.index == vbegin.index && begin.offset <= vbegin.offset)) + start = vma->vm_start; + else + start = vma_address_at_offset(vma, begin); + if (finish.index > vend.index || + (finish.index == vend.index && finish.offset >= vend.offset)) + end = vma->vm_end; + else + end = vma_address_at_offset(vma, finish); + if (IS_ERR_VALUE(start) || IS_ERR_VALUE(end) || start >= end) + continue; + tlb_gather_mmu(&tlb, vma->vm_mm); + zap_vma_range_batched(&tlb, vma, start, end - start, &details); + tlb_finish_mmu(&tlb); + } + i_mmap_unlock_read(mapping); +} + +/* + * Native truncation keeps its final partial page. Smaller-granule mms must + * additionally discard complete granules in that native page, including + * private COW copies. Call on both sides of cache truncation, as for the + * ordinary unmap_mapping_range() calls. + */ +void unmap_mapping_subpage_tail(struct address_space *mapping, loff_t newsize) +{ + struct vm_area_struct *vma; + struct zap_details details = { }; + pgoff_t index = newsize >> PAGE_SHIFT; + unsigned int offset = newsize & ~PAGE_MASK; + + if (!offset) + return; + i_mmap_lock_read(mapping); + vma_interval_tree_foreach(vma, &mapping->i_mmap, index, index) { + unsigned long start, end, granule = mm_page_size(vma->vm_mm); + struct vm_page_offset pos = { .index = index, .offset = round_up(offset, granule) }; + struct vm_page_offset vpos = vma_page_offset_at(vma, vma->vm_start); + struct mmu_gather tlb; + + if (pos.offset >= PAGE_SIZE) + continue; + if (pos.index < vpos.index || + (pos.index == vpos.index && pos.offset < vpos.offset)) + start = vma->vm_start; + else + start = vma_address_at_offset(vma, pos); + if (IS_ERR_VALUE(start)) + continue; + end = vma->vm_end; + if (index < vma_last_pgoff(vma)) + end = vma_address_at_offset(vma, (struct vm_page_offset) { + .index = index + 1, + }); + tlb_gather_mmu(&tlb, vma->vm_mm); + zap_vma_range_batched(&tlb, vma, start, end - start, &details); + tlb_finish_mmu(&tlb); + } + i_mmap_unlock_read(mapping); +} +#endif + /** * unmap_mapping_folio() - Unmap single folio from processes. * @folio: The locked folio to be unmapped. @@ -4772,6 +5587,139 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, } while (--nr_pages); } +#ifdef CONFIG_MM_SUBPAGE +/* The swap-cache index remains native-sized; the PTE retains its slot offset. */ +static vm_fault_t do_swap_subpage(struct vm_fault *vmf, swp_entry_t entry) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct swap_info_struct *si; + struct mm_subpage *slot = NULL; + struct folio *folio; + unsigned int offset = pte_swp_subpage_offset(vmf->orig_pte); + vm_fault_t ret; + pte_t pte; + int err; + + ret = vmf_anon_prepare(vmf); + if (ret) + return ret; + /* Swap encodes the physical slot, independent of its virtual position. */ + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, mm_page_size(mm))) + return VM_FAULT_SIGBUS; + si = get_swap_device(entry); + if (!si) + return 0; + folio = swap_cache_get_folio(entry); + if (folio) { + swap_update_readahead(folio, vma, vmf->address); + } else { + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); + if (!folio) { + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (vmf->pte) { + if (pte_same(ptep_get(vmf->pte), vmf->orig_pte)) + ret = VM_FAULT_OOM; + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; + } + goto out_device; + } + ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + } + ret |= folio_lock_or_retry(folio, vmf); + if (ret & VM_FAULT_RETRY) + goto out_folio; + if (!folio_matches_swap_entry(folio, entry)) + goto unlock_folio; + if (folio_order(folio) || !folio_test_uptodate(folio) || folio_test_ksm(folio)) { + ret = VM_FAULT_SIGBUS; + goto unlock_folio; + } + if (PageHWPoison(&folio->page)) { + ret = VM_FAULT_HWPOISON; + goto unlock_folio; + } + arch_swap_restore(folio_swap(entry, folio), folio); + slot = mm_subpage_restore_granule(folio, offset, mm_page_shift(mm), GFP_KERNEL); + if (IS_ERR(slot)) { + err = PTR_ERR(slot); + slot = NULL; + if (err != -EAGAIN) + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + goto unlock_folio; + } + err = mm_subpage_prepare_anon_rmap(slot, vma, vmf->address); + if (err) { + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + goto unlock_folio; + } + folio_throttle_swaprate(folio, GFP_KERNEL); + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto unlock_folio; + if (!pte_same(ptep_get(vmf->pte), vmf->orig_pte)) + goto unlock_pte; + err = mm_subpage_add_anon_rmap(slot, vma, vmf->address); + if (err) { + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + goto unlock_pte; + } + /* + * A carried marker proves exclusivity directly. After the last fork + * sharer exits, recover it only with one swap reference to this exact + * quarter and no other mapping or escaped slot reference. Dropping a + * nonpresent PTE in zap flushes pending reclaim TLB invalidations first. + * Stable-write backends must still finish writeback before reuse. + */ + if (!(folio_test_writeback(folio) && data_race(si->flags & SWP_STABLE_WRITES))) { + if (pte_swp_exclusive(vmf->orig_pte)) + mm_subpage_set_exclusive(slot); + else if (swap_subpage_count(entry, offset) == 1) + mm_subpage_try_reuse_swap(slot); + } + pte = pte_sw_mkyoung(phys_pte_mm(mm, mm_subpage_phys(slot), vma->vm_page_prot)); + if (pte_swp_soft_dirty(vmf->orig_pte)) + pte = pte_mksoft_dirty(pte); + if (pte_swp_uffd_wp(vmf->orig_pte)) + pte = pte_mkuffd_wp(pte); + inc_mm_counter(mm, MM_ANONPAGES); + dec_mm_counter(mm, MM_SWAPENTS); + swap_subpage_put(entry, offset, false); + set_pte_at(mm, vmf->address, vmf->pte, pte); + arch_do_swap_page_nr(mm, vma, vmf->address, pte, pte, 1); + slot = NULL; /* The restored PTE owns the slot reference. */ + vmf->orig_pte = pte; + if (should_try_to_free_swap(si, folio, vma, 2, vmf->flags)) + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); /* Drop the cache lookup, not the PTE reference. */ + put_swap_device(si); + if (vmf->flags & FAULT_FLAG_WRITE) { + ret |= do_wp_page(vmf); + return ret & VM_FAULT_ERROR ? ret & VM_FAULT_ERROR : ret; + } + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; + return ret; +unlock_pte: + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; +unlock_folio: + if (slot) + mm_subpage_put(slot); + folio_unlock(folio); +out_folio: + folio_put(folio); +out_device: + put_swap_device(si); + return ret; +} +#endif + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4780,6 +5728,209 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, * We return with the mmap_lock locked or unlocked in the same cases * as does filemap_fault(). */ +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SWAP) +/* One coarse PTE owns a naturally aligned group of native swap entries. */ +vm_fault_t do_swap_coarse_page(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + /* Arm64's largest base user page is 64K, its smallest native page 4K. */ + struct folio *cache[SZ_64K / PAGE_SIZE]; + unsigned int nr = mm_pte_native_pages(mm), covered = 0, count = 0, i; + swp_entry_t entry = softleaf_from_pte(vmf->orig_pte), next; + struct swap_info_struct *si; + struct folio *backing = NULL; + struct page *page; + unsigned int start; + bool copied = false, exclusive, mapped = false; + vm_fault_t ret = 0; + pte_t pte; + + vmf->pte = NULL; + if (WARN_ON_ONCE(nr <= 1 || nr > ARRAY_SIZE(cache) || + !IS_ALIGNED(swp_offset(entry), nr))) + return VM_FAULT_SIGBUS; + si = get_swap_device(entry); + if (!si) + return 0; + + while (covered < nr) { + struct folio *folio; + unsigned int take; + vm_fault_t wait; + + next = entry; + next.val += covered; + folio = swap_cache_get_folio(next); + if (!folio) { + folio = read_swap_cache_async(next, GFP_HIGHUSER_MOVABLE, vma, + vmf->address + covered * PAGE_SIZE, NULL); + if (!(ret & VM_FAULT_MAJOR)) { + ret |= VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + } + } + if (!folio) + goto oom; + if (!count) { + wait = folio_lock_or_retry(folio, vmf); + if (wait & VM_FAULT_RETRY) { + folio_put(folio); + ret |= wait; + goto out; + } + } else if (!folio_trylock(folio)) { + /* Drop earlier cache locks before waiting for this I/O or owner. */ + for (i = 0; i < count; i++) { + folio_unlock(cache[i]); + folio_put(cache[i]); + } + count = 0; + covered = 0; + wait = folio_lock_or_retry(folio, vmf); + if (wait & VM_FAULT_RETRY) { + folio_put(folio); + ret |= wait; + goto out; + } + folio_unlock(folio); + folio_put(folio); + cond_resched(); + continue; + } + cache[count++] = folio; + if (!folio_matches_swap_entry(folio, next)) + goto out; + if (!folio_test_uptodate(folio)) { + ret = VM_FAULT_SIGBUS; + goto out; + } + start = swp_offset(next) - swp_offset(folio->swap); + take = min(nr - covered, folio_nr_pages(folio) - start); + for (i = 0; i < take; i++) { + if (PageHWPoison(folio_page(folio, start + i))) { + ret = VM_FAULT_HWPOISON; + goto out; + } + } + arch_swap_restore(folio_swap(next, folio), folio); + covered += take; + } + + start = swp_offset(entry) - swp_offset(cache[0]->swap); + if (count == 1 && IS_ALIGNED(start, nr) && + (folio_test_anon(cache[0]) || folio_nr_pages(cache[0]) == nr)) { + backing = cache[0]; + page = folio_page(backing, start); + } else { + backing = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, ilog2(nr), vma, vmf->address); + if (!backing) + goto oom; + copied = true; + if (mem_cgroup_swapin_charge_folio(backing, mm, GFP_KERNEL, entry)) + goto oom; + covered = 0; + for (i = 0; i < count; i++) { + unsigned int j, take; + + start = swp_offset(entry) + covered - swp_offset(cache[i]->swap); + take = min(nr - covered, folio_nr_pages(cache[i]) - start); + for (j = 0; j < take; j++) { + struct page *dst = folio_page(backing, covered + j); + struct page *src = folio_page(cache[i], start + j); + + if (copy_mc_user_highpage(dst, src, + vmf->address + (covered + j) * PAGE_SIZE, vma)) { + ret = VM_FAULT_HWPOISON; + goto out; + } + kmsan_copy_page_meta(dst, src); + } + covered += take; + } + folio_set_dirty(backing); + __folio_mark_uptodate(backing); + page = &backing->page; + } + + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)) + goto out; + exclusive = copied || pte_swp_exclusive(vmf->orig_pte); + if (!copied && exclusive) { + check_swap_exclusive(backing, entry, nr); + if (folio_test_writeback(backing) && data_race(si->flags & SWP_STABLE_WRITES)) + exclusive = false; + } + pte = mk_pte(page, vma->vm_page_prot); + if (pte_swp_soft_dirty(vmf->orig_pte)) + pte = pte_mksoft_dirty(pte); + if (pte_swp_uffd_wp(vmf->orig_pte)) + pte = pte_mkuffd_wp(pte); + if (exclusive && (vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) && + !pte_needs_soft_dirty_wp(vma, pte)) { + pte = pte_mkwrite(pte, vma); + if (vmf->flags & FAULT_FLAG_WRITE) { + pte = pte_mkdirty(pte); + vmf->flags &= ~FAULT_FLAG_WRITE; + } + } + folio_ref_add(backing, nr - copied); + if (!folio_test_anon(backing)) + folio_add_new_anon_rmap(backing, vma, vmf->address, + exclusive ? RMAP_EXCLUSIVE : RMAP_NONE); + else + folio_add_anon_rmap_pte(backing, page, vma, vmf->address, + exclusive ? RMAP_EXCLUSIVE : RMAP_NONE); + if (copied) + folio_add_lru_vma(backing, vma); + inc_mm_counter(mm, MM_ANONPAGES); + dec_mm_counter(mm, MM_SWAPENTS); + flush_icache_pages(vma, page, nr); + vmf->orig_pte = pte; + set_pte_at(mm, vmf->address, vmf->pte, pte); + arch_do_swap_page_nr(mm, vma, vmf->address, pte, pte, 1); + mapped = true; + covered = 0; + for (i = 0; i < count; i++) { + unsigned int take; + + start = swp_offset(entry) + covered - swp_offset(cache[i]->swap); + take = min(nr - covered, folio_nr_pages(cache[i]) - start); + folio_put_swap_range(cache[i], folio_page(cache[i], start), take); + covered += take; + } + goto out; + +oom: + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (vmf->pte && pte_same(ptep_get(vmf->pte), vmf->orig_pte)) + ret |= VM_FAULT_OOM; +out: + if (copied && !mapped) + folio_put(backing); + for (i = 0; i < count; i++) { + if (mapped) + folio_free_swap(cache[i]); + folio_unlock(cache[i]); + folio_put(cache[i]); + } + if (mapped && (vmf->flags & FAULT_FLAG_WRITE)) { + ret |= do_wp_page(vmf); + vmf->pte = NULL; /* do_wp_page drops the PT lock. */ + } else if (mapped) { + update_mmu_cache(vma, vmf->address, vmf->pte); + } + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); + put_swap_device(si); + if (!mapped && !(ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))) + cond_resched(); + return ret; +} +#endif + vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -4854,6 +6005,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out; } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return do_swap_subpage(vmf, entry); +#ifdef CONFIG_SWAP + if (mm_page_size(vma->vm_mm) > PAGE_SIZE) + return do_swap_coarse_page(vmf); +#endif +#endif + /* Prevent swapoff from happening to us. */ si = get_swap_device(entry); if (unlikely(!si)) @@ -5276,47 +6436,285 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) folio_zero_user(folio, vmf->address); return folio; } -next: - count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); - order = next_order(&orders, order); +next: + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); + order = next_order(&orders, order); + } + +fallback: +#endif + return folio_prealloc(vma->vm_mm, vma, vmf->address, true); +} + +void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, + struct vm_area_struct *vma, unsigned long addr, + bool uffd_wp) +{ + const unsigned int nr_pages = folio_nr_pages(folio); + const unsigned int nr_ptes = folio_size(folio) >> mm_page_shift(vma->vm_mm); + pte_t entry = folio_mk_pte(folio, vma->vm_page_prot); + + entry = pte_sw_mkyoung(entry); + + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry), vma); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + + folio_ref_add(folio, nr_pages - 1); + folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + set_ptes(vma->vm_mm, addr, pte, entry, nr_ptes); + update_mmu_cache_range(NULL, vma, addr, pte, nr_ptes); +} + +static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte, + struct vm_area_struct *vma, unsigned long addr, bool uffd_wp) +{ + const unsigned int order = folio_order(folio); + + map_anon_folio_pte_nopf(folio, pte, vma, addr, uffd_wp); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, folio_size(folio) >> mm_page_shift(vma->vm_mm)); + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_ALLOC); +} + +#ifdef CONFIG_MM_SUBPAGE +/* One native-window batch, initialized and locked before PTE publication. */ +static int anon_subpage_prealloc_mask(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr, + unsigned long mask, bool use_pool, bool any_offset, + bool *pooled, struct mm_subpage **slots) +{ + struct mm_subpage_pool *pool; + struct folio *folio; + unsigned int count = hweight_long(mask); + unsigned int retired_retries = 0; + bool arbitrary; + int err; + +retry_private: + pool = use_pool ? mm_subpage_cow_pool_get(mm, vma, addr) : NULL; + *pooled = pool != NULL; + arbitrary = *pooled && any_offset; + if (!pool) + pool = mm_subpage_pool_create_granule(GFP_KERNEL, mm_page_shift(mm)); + if (!pool) + return -ENOMEM; + for (;;) { + err = arbitrary ? mm_subpage_alloc_any_mask_locked(pool, mask, slots) : + mm_subpage_alloc_mask_locked(pool, mask, slots); + if (err != -EAGAIN) + break; + folio = NULL; + if (*pooled) { + folio = cow_subpage_speculative_folio(mm, vma, addr); + if (!folio && (arbitrary ? mm_subpage_cow_wait_busy_count(pool, count) : + mm_subpage_cow_wait_busy_mask(pool, mask))) { + cond_resched(); + continue; + } + } + if (!folio) + folio = folio_prealloc(mm, vma, addr, false); + if (!folio) { + err = -ENOMEM; + break; + } + if (*pooled && !mm_subpage_cow_folio_matches(pool, folio)) { + folio_put(folio); + mm_subpage_pool_put(pool); + use_pool = false; + goto retry_private; + } + err = arbitrary ? mm_subpage_pool_add_folio_count(pool, folio, count, + GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN) : + mm_subpage_pool_add_folio_mask(pool, folio, mask, + *pooled ? GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN : GFP_KERNEL); + if (*pooled && err == -ENOMEM) { + if (arbitrary ? mm_subpage_cow_wait_busy_count(pool, count) : + mm_subpage_cow_wait_busy_mask(pool, mask)) { + folio_put(folio); + cond_resched(); + continue; + } + err = arbitrary ? + mm_subpage_pool_add_folio_count(pool, folio, count, GFP_KERNEL) : + mm_subpage_pool_add_folio_mask(pool, folio, mask, GFP_KERNEL); + } + if (err) { + folio_put(folio); + if (err == -EEXIST) { + cond_resched(); + continue; + } + break; + } } - -fallback: -#endif - return folio_prealloc(vma->vm_mm, vma, vmf->address, true); + if (!*pooled) + mm_subpage_pool_close(pool); + mm_subpage_pool_put(pool); + if (*pooled && err == -ESHUTDOWN) { + if (++retired_retries > 1) + use_pool = false; + goto retry_private; + } + return err; } -void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, - struct vm_area_struct *vma, unsigned long addr, - bool uffd_wp) +static unsigned long anon_subpage_missing_mask(pte_t *first, unsigned int nr, + unsigned int target, unsigned int offset) { - const unsigned int nr_pages = folio_nr_pages(folio); - pte_t entry = folio_mk_pte(folio, vma->vm_page_prot); - - entry = pte_sw_mkyoung(entry); - - if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry), vma); - if (uffd_wp) - entry = pte_mkuffd_wp(entry); + unsigned long mask = 0; + unsigned int i; - folio_ref_add(folio, nr_pages - 1); - folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE); - folio_add_lru_vma(folio, vma); - set_ptes(vma->vm_mm, addr, pte, entry, nr_pages); - update_mmu_cache_range(NULL, vma, addr, pte, nr_pages); + for (i = 0; i < nr; i++) + if (i == target || pte_none(ptep_get(first + i))) + mask |= BIT(offset + i); + return mask; } -static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte, - struct vm_area_struct *vma, unsigned long addr, bool uffd_wp) +static vm_fault_t do_anonymous_subpage(struct vm_fault *vmf) { - const unsigned int order = folio_order(folio); + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct mm_subpage *slots[PAGE_SIZE / MM_SUBPAGE_SIZE] = {}; + struct vm_page_offset pos = vma_page_offset_at(vma, vmf->address); + struct folio *folio; + unsigned long start, end, lower, mask, current_mask, window_mask; + unsigned int first_offset, nr, target, i, index, mapped; + bool pooled, fresh, force_private = false, retry_private; + vm_fault_t ret; + pte_t *first; + int err; - map_anon_folio_pte_nopf(folio, pte, vma, addr, uffd_wp); - add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1L << order); - count_mthp_stat(order, MTHP_STAT_ANON_FAULT_ALLOC); + /* One native linear index, clipped to the VMA and this PTE table. */ + lower = max(vma->vm_start, vmf->address & pmd_mask_mm(mm)); + start = vmf->address - min_t(unsigned long, pos.offset, vmf->address - lower); + end = vmf->address + min(PAGE_SIZE - pos.offset, + pmd_addr_end_mm(mm, vmf->address, vma->vm_end) - vmf->address); + first_offset = (pos.offset - (vmf->address - start)) >> mm_page_shift(mm); + nr = (end - start) >> mm_page_shift(mm); + target = (vmf->address - start) >> mm_page_shift(mm); + window_mask = GENMASK(first_offset + nr - 1, first_offset); +retry: + /* This optimistic snapshot only selects backing; PTL revalidates it. + * Initial dense faults retain one-folio, whole-window faultaround. + */ + vmf->pte = pte_offset_map_mm(mm, vmf->pmd, vmf->address); + if (!vmf->pte) + return 0; + mask = anon_subpage_missing_mask(vmf->pte - target, nr, target, first_offset); + pte_unmap(vmf->pte); + vmf->pte = NULL; + err = anon_subpage_prealloc_mask(mm, vma, vmf->address, mask, + (mask != window_mask || nr < (PAGE_SIZE >> mm_page_shift(mm))) && + !force_private && !userfaultfd_armed(vma), + nr < (PAGE_SIZE >> mm_page_shift(mm)), &pooled, slots); + if (err) + return VM_FAULT_OOM; + folio = mm_subpage_folio(slots[__ffs(mask)]); + /* PTE references may disappear as soon as we drop the PTL. */ + folio_get(folio); + fresh = !mm_subpage_anon_root(slots[__ffs(mask)]); + mapped = 0; + retry_private = false; + ret = 0; + /* Existing owners may need nonlinear identities. Allocate metadata before + * PTL and keep the established fresh-folio fallback on metadata ENOMEM. + */ + for_each_set_bit(index, &mask, ARRAY_SIZE(slots)) { + unsigned long address = start + + ((index - first_offset) << mm_page_shift(mm)); + + err = mm_subpage_prepare_anon_rmap(slots[index], vma, address); + if (err) { + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + retry_private = pooled && err == -ENOMEM; + goto unlock_folio; + } + } + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto unlock_folio; + if (vmf_pte_changed(vmf)) { + update_mmu_tlb(vma, vmf->address, vmf->pte); + goto unlock_pte; + } + ret = check_stable_address_space(mm); + if (ret) + goto unlock_pte; + if (userfaultfd_missing(vma)) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; + folio_unlock(folio); + folio_put(folio); + ret = handle_userfault(vmf, VM_UFFD_MISSING); + goto release; + } + + first = vmf->pte - target; + current_mask = anon_subpage_missing_mask(first, nr, target, first_offset); + /* Do not lose newly missing neighbors or publish sparse dense-path supply. + * Return to the fault handler on a racing mask change instead of spinning + * under repeated zaps. No mapping has been published yet. + */ + if ((current_mask & ~mask) || + (mask == window_mask && current_mask != mask && + !force_private && !userfaultfd_armed(vma))) + goto unlock_pte; + for (i = 0; i < nr; i++) { + unsigned long address = start + i * mm_page_size(mm); + pte_t entry; + + index = first_offset + i; + if (!(current_mask & BIT(index))) + continue; + mm_subpage_set_exclusive(slots[index]); + err = fresh ? mm_subpage_add_new_anon_rmap(slots[index], vma, address) : + mm_subpage_add_anon_rmap(slots[index], vma, address); + if (WARN_ON_ONCE(err)) { + ret = VM_FAULT_SIGBUS; + break; + } + entry = phys_pte_mm(mm, mm_subpage_phys(slots[index]), vma->vm_page_prot); + entry = pte_sw_mkyoung(entry); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry), vma); + if (i == target && vmf_orig_pte_uffd_wp(vmf)) + entry = pte_mkuffd_wp(entry); + set_pte_at(mm, address, first + i, entry); + update_mmu_cache(vma, address, first + i); + slots[index] = NULL; /* The PTE now owns the reference. */ + mapped++; + } + if (mapped) { + if (fresh) + subpage_add_new_lru(folio, vma, mapped); + add_mm_counter(mm, MM_ANONPAGES, mapped); + count_mthp_stat(0, MTHP_STAT_ANON_FAULT_ALLOC); + } +unlock_pte: + pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; +unlock_folio: + folio_unlock(folio); + folio_put(folio); +release: + for (i = 0; i < ARRAY_SIZE(slots); i++) { + if (slots[i]) { + mm_subpage_put(slots[i]); + slots[i] = NULL; + } + } + if (retry_private) { + force_private = true; + goto retry; + } + return ret; } +#endif + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -5376,6 +6774,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) ret = vmf_anon_prepare(vmf); if (ret) return ret; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return do_anonymous_subpage(vmf); +#endif /* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ folio = alloc_anon_folio(vmf); if (IS_ERR(folio)) @@ -5383,8 +6785,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (!folio) goto oom; - nr_pages = folio_nr_pages(folio); - addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); + nr_pages = folio_size(folio) >> mm_page_shift(vma->vm_mm); + addr = ALIGN_DOWN(vmf->address, folio_size(folio)); /* * The memory barrier inside __folio_mark_uptodate makes sure that @@ -5517,6 +6919,10 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa * PMD mappings if THPs are disabled. As we already have a THP, * behave as if we are forcing a collapse. */ + /* Match the native PTE fault policy: partial data exposes its zero subunits. */ + if (userfaultfd_missing(vma) && shmem_uffd_has_missing_native_page(folio)) + return ret; + if (thp_disabled_by_hw() || vma_thp_disabled(vma, vma->vm_flags, /* forced_collapse=*/ true)) return ret; @@ -5558,6 +6964,8 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(folio), HPAGE_PMD_NR); + shmem_uffd_mark_mapped(folio, folio_page_idx(folio, page) << PAGE_SHIFT, + HPAGE_PMD_SIZE); folio_add_file_rmap_pmd(folio, page, vma); /* @@ -5620,6 +7028,8 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE); folio_add_lru_vma(folio, vma); } else { + shmem_uffd_mark_mapped(folio, folio_page_idx(folio, page) << PAGE_SHIFT, + (unsigned long)nr << PAGE_SHIFT); folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); @@ -5636,6 +7046,265 @@ static bool vmf_pte_changed(struct vm_fault *vmf) return !pte_none(ptep_get(vmf->pte)); } +#ifdef CONFIG_MM_SUBPAGE +static bool folio_user_range_valid(struct folio *folio, unsigned long offset, + unsigned long size) +{ + return offset <= folio_size(folio) && size <= folio_size(folio) - offset && + IS_ALIGNED(page_to_phys(&folio->page) + offset, size); +} + +/* + * Map the intersection of a locked cache folio, file-index range, VMA and + * the fault's PTE table. vmf->pte points at vmf->address and its lock is held. + * Each PTE owns a reference per covered native page (one for a subpage). + * The caller's reference is untouched, including when no PTE is installed. + */ +vm_fault_t map_file_user_pages(struct vm_fault *vmf, struct folio *folio, + pgoff_t first, pgoff_t last, bool fault) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + unsigned long addr, end, table_start = vmf->address & pmd_mask_mm(mm); + unsigned long size = mm_page_size(mm); + loff_t file_size = i_size_read(vma->vm_file->f_mapping->host); + vm_fault_t ret = 0; + unsigned int mapped = 0; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + first = max(first, folio->index); + last = min(last, folio_next_index(folio) - 1); + if (first > last) + return 0; + addr = vma_address(vma, first, last - first + 1); + if (IS_ERR_VALUE(addr)) + return 0; + addr = ALIGN(max(addr, table_start), size); + end = pmd_addr_end_mm(mm, vmf->address, vma->vm_end); + if (userfaultfd_armed(vma)) { + addr = max(addr, vmf->address); + end = min(end, vmf->address + size); + } + for (; addr < end; addr += size) { + struct vm_page_offset pos = vma_page_offset_at(vma, addr); + struct page *page; + unsigned long offset; + unsigned int i, nr = mm_pte_native_pages(mm); + bool poisoned = false; + pte_t entry, *ptep; + + if (pos.index > last || !vma_file_offset_valid(vma, addr, file_size)) + break; + offset = ((pos.index - folio->index) << PAGE_SHIFT) + pos.offset; + /* + * A coarse leaf cannot span separate cache folios or a partial + * requested index range, even if neighboring PFNs happen to match. + */ + if (nr - 1 > last - pos.index || + !folio_user_range_valid(folio, offset, size)) + continue; + /* A native cache folio can contain independently missing leaves. */ + if (userfaultfd_missing(vma) && + shmem_uffd_range_missing(folio, pos.index, pos.offset, size)) + continue; + page = folio_file_page(folio, pos.index); + ptep = vmf->pte + ((long)(addr - vmf->address) >> mm_page_shift(mm)); + for (i = 0; i < nr; i++) + poisoned |= PageHWPoison(page + i); + if (poisoned) + continue; + if (!pte_none(ptep_get(ptep)) && + !(fault && addr == vmf->address && !vmf_pte_changed(vmf))) + continue; + flush_icache_pages(vma, page, nr); + entry = phys_pte_mm(mm, page_to_phys(page) + pos.offset, vma->vm_page_prot); + if (addr != vmf->address && arch_wants_old_prefaulted_pte()) + entry = pte_mkold(entry); + else + entry = pte_sw_mkyoung(entry); + if (vmf->flags & FAULT_FLAG_WRITE) + entry = maybe_mkwrite(pte_mkdirty(entry), vma); + else if (pte_write(entry) && folio_test_dirty(folio)) + entry = pte_mkdirty(entry); + if (fault && addr == vmf->address && vmf_orig_pte_uffd_wp(vmf)) + entry = pte_mkuffd_wp(entry); + folio_ref_add(folio, nr); + mm_subpage_file_map_add(vma, pte_phys_mm(vma->vm_mm, entry)); + shmem_uffd_mark_mapped(folio, + ((pos.index - folio->index) << PAGE_SHIFT) + pos.offset, size); + folio_add_file_rmap_pte(folio, page, vma); + set_pte_at(mm, addr, ptep, entry); + update_mmu_cache_range(vmf, vma, addr, ptep, 1); + mapped++; + if (addr == vmf->address) + ret = VM_FAULT_NOPAGE; + } + add_mm_counter(mm, mm_counter_file(folio), mapped); + return ret; +} + +/* + * A non-page-cache fault callback supplies one native page, not a cache-indexed + * folio. In particular, VVAR callbacks may reorder native pages. Only map + * quarters of this supplied page; never infer adjacent pages from its folio. + */ +static vm_fault_t map_special_subpages(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct page *page = vmf->page; + struct folio *folio = page_folio(page); + unsigned int offset = vma_page_offset_at(vma, vmf->address).offset; + unsigned long size = mm_page_size(mm), addr, end; + vm_fault_t ret = 0; + unsigned int mapped = 0; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (PageHWPoison(page)) + return 0; + addr = offset > vmf->address ? vma->vm_start : + max(vma->vm_start, vmf->address - offset); + addr = max(addr, vmf->address & pmd_mask_mm(mm)); + end = min(vmf->address + PAGE_SIZE - offset, + pmd_addr_end_mm(mm, vmf->address, vma->vm_end)); + if (userfaultfd_armed(vma)) { + addr = vmf->address; + end = vmf->address + size; + } + flush_icache_pages(vma, page, 1); + for (; addr < end; addr += size) { + unsigned int physical_offset = addr < vmf->address ? + offset - (vmf->address - addr) : offset + (addr - vmf->address); + pte_t *ptep = vmf->pte + ((long)(addr - vmf->address) >> mm_page_shift(mm)); + pte_t entry; + + if (!pte_none(ptep_get(ptep)) && + !(addr == vmf->address && !vmf_pte_changed(vmf))) + continue; + entry = phys_pte_mm(mm, page_to_phys(page) + physical_offset, vma->vm_page_prot); + entry = pte_sw_mkyoung(entry); + if (vmf->flags & FAULT_FLAG_WRITE) + entry = maybe_mkwrite(pte_mkdirty(entry), vma); + if (addr == vmf->address && vmf_orig_pte_uffd_wp(vmf)) + entry = pte_mkuffd_wp(entry); + folio_get(folio); + mm_subpage_file_map_add(vma, pte_phys_mm(vma->vm_mm, entry)); + folio_add_file_rmap_pte(folio, page, vma); + set_pte_at(mm, addr, ptep, entry); + update_mmu_cache_range(vmf, vma, addr, ptep, 1); + mapped++; + if (addr == vmf->address) + ret = VM_FAULT_NOPAGE; + } + add_mm_counter(mm, mm_counter_file(folio), mapped); + return ret; +} + +static vm_fault_t finish_user_page_fault(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct mm_subpage *slot = vmf->cow_subpage; + bool is_cow = (vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED); + struct folio *folio = page_folio(is_cow ? vmf->cow_page : vmf->page); + bool fresh = false; + vm_fault_t ret; + int err; + pte_t entry; + + if (!(vma->vm_flags & VM_SHARED)) { + ret = check_stable_address_space(mm); + if (ret) + return ret; + } + if (pmd_none(*vmf->pmd)) { + if (vmf->prealloc_pte) + pmd_install(mm, vmf->pmd, &vmf->prealloc_pte); + else if (pte_alloc(mm, vmf->pmd)) + return VM_FAULT_OOM; + } + /* A reserved destination slot is private; sibling slots may be published. */ + if (is_cow) { + folio_get(folio); + folio_lock(folio); + if (slot) { + fresh = !mm_subpage_anon_root(slot); + err = mm_subpage_prepare_anon_rmap(slot, vma, vmf->address); + if (err) { + ret = err == -ENOMEM ? VM_FAULT_OOM : VM_FAULT_SIGBUS; + goto unlock_folio; + } + } + } + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + ret = VM_FAULT_NOPAGE; + if (!vmf->pte) + goto unlock_folio; + if (vmf_pte_changed(vmf)) { + update_mmu_tlb(vma, vmf->address, vmf->pte); + goto unlock; + } + if (slot) { + mm_subpage_set_exclusive(slot); + err = fresh ? mm_subpage_add_new_anon_rmap(slot, vma, vmf->address) : + mm_subpage_add_anon_rmap(slot, vma, vmf->address); + if (WARN_ON_ONCE(err)) { + ret = VM_FAULT_SIGBUS; + goto unlock; + } + entry = phys_pte_mm(mm, mm_subpage_phys(slot), vma->vm_page_prot); + entry = maybe_mkwrite(pte_mkdirty(pte_sw_mkyoung(entry)), vma); + if (vmf_orig_pte_uffd_wp(vmf)) + entry = pte_mkuffd_wp(entry); + if (fresh) + subpage_add_new_lru(folio, vma, 1); + set_pte_at(mm, vmf->address, vmf->pte, entry); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + inc_mm_counter(mm, MM_ANONPAGES); + vmf->cow_subpage = NULL; + ret = 0; + } else if (is_cow) { + if (!folio_user_range_valid(folio, 0, mm_page_size(mm))) { + ret = VM_FAULT_SIGBUS; + goto unlock; + } + entry = phys_pte_mm(mm, page_to_phys(&folio->page), vma->vm_page_prot); + entry = maybe_mkwrite(pte_mkdirty(pte_sw_mkyoung(entry)), vma); + if (vmf_orig_pte_uffd_wp(vmf)) + entry = pte_mkuffd_wp(entry); + folio_ref_add(folio, mm_pte_native_pages(mm) - 1); + folio_add_new_anon_rmap(folio, vma, vmf->address, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + set_pte_at(mm, vmf->address, vmf->pte, entry); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + inc_mm_counter(mm, MM_ANONPAGES); + ret = 0; + } else { + if (vma->vm_file && folio_mapping(folio) == vma->vm_file->f_mapping) + ret = map_file_user_pages(vmf, folio, folio->index, + folio_next_index(folio) - 1, true); + else if (mm_page_size(mm) < PAGE_SIZE) + ret = map_special_subpages(vmf); + else + ret = VM_FAULT_SIGBUS; + if (ret & VM_FAULT_NOPAGE) { + folio_put(folio); /* Consume the fault callback's reference. */ + ret = 0; + } else { + ret = VM_FAULT_SIGBUS; + } + } +unlock: + pte_unmap_unlock(vmf->pte, vmf->ptl); +unlock_folio: + if (is_cow) { + folio_unlock(folio); + folio_put(folio); + } + return ret; +} +#endif + /** * finish_fault - finish page fault once we have prepared the page to fault * @@ -5663,6 +7332,11 @@ vm_fault_t finish_fault(struct vm_fault *vmf) unsigned long addr; bool needs_fallback = false; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return finish_user_page_fault(vmf); +#endif + fallback: addr = vmf->address; @@ -5768,13 +7442,12 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return ret; } -static unsigned long fault_around_pages __read_mostly = - 65536 >> PAGE_SHIFT; +static unsigned long fault_around_bytes __read_mostly = SZ_64K; #ifdef CONFIG_DEBUG_FS static int fault_around_bytes_get(void *data, u64 *val) { - *val = fault_around_pages << PAGE_SHIFT; + *val = READ_ONCE(fault_around_bytes); return 0; } @@ -5788,11 +7461,13 @@ static int fault_around_bytes_set(void *data, u64 val) return -EINVAL; /* - * The minimum value is 1 page, however this results in no fault-around - * at all. See should_fault_around(). + * Keep the policy in bytes: one native page can still contain several + * user leaves. The smallest supported leaf disables fault-around for + * every mm; larger values are compared with each mm's own granule. */ - val = max(val, PAGE_SIZE); - fault_around_pages = rounddown_pow_of_two(val) >> PAGE_SHIFT; + val = max_t(u64, val, IS_ENABLED(CONFIG_MM_SUBPAGE) ? + MM_SUBPAGE_SIZE : PAGE_SIZE); + WRITE_ONCE(fault_around_bytes, rounddown_pow_of_two(val)); return 0; } @@ -5819,24 +7494,42 @@ late_initcall(fault_around_debugfs); * This function doesn't cross VMA or page table boundaries, in order to call * map_pages() and acquire a PTE lock only once. * - * fault_around_pages defines how many pages we'll try to map. - * do_fault_around() expects it to be set to a power of two less than or equal - * to PTRS_PER_PTE. - * - * The virtual address of the area that we map is naturally aligned to - * fault_around_pages * PAGE_SIZE rounded down to the machine page size - * (and therefore to page order). This way it's easier to guarantee - * that we don't cross page table boundaries. + * fault_around_bytes is a power-of-two byte policy, independent of either + * physical or user granules. The native path converts it to native pages; + * the alternative path clamps it to the target mm's PTE-table coverage. + * Natural alignment keeps the window inside one page table. */ static vm_fault_t do_fault_around(struct vm_fault *vmf) { - pgoff_t nr_pages = READ_ONCE(fault_around_pages); + unsigned long configured_bytes = READ_ONCE(fault_around_bytes); + pgoff_t nr_pages = max(1UL, configured_bytes >> PAGE_SHIFT); pgoff_t pte_off = pte_index(vmf->address); /* The page offset of vmf->address within the VMA. */ pgoff_t vma_off = vmf->pgoff - vmf->vma->vm_pgoff; pgoff_t from_pte, to_pte; vm_fault_t ret; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vmf->vma->vm_mm) != PAGE_SIZE) { + struct vm_area_struct *vma = vmf->vma; + unsigned long bytes = min(configured_bytes, + pmd_size_mm(vma->vm_mm)); + unsigned long start = max(ALIGN_DOWN(vmf->address, bytes), vma->vm_start); + unsigned long end = min(ALIGN_DOWN(vmf->address, bytes) + bytes, vma->vm_end); + + if (pmd_none(*vmf->pmd)) { + vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); + if (!vmf->prealloc_pte) + return VM_FAULT_OOM; + } + rcu_read_lock(); + ret = vma->vm_ops->map_pages(vmf, vma_page_offset_at(vma, start).index, + vma_page_offset_at(vma, end - 1).index); + rcu_read_unlock(); + return ret; + } +#endif + /* The PTE offset of the start address, clamped to the VMA. */ from_pte = max(ALIGN_DOWN(pte_off, nr_pages), pte_off - min(pte_off, vma_off)); @@ -5871,7 +7564,7 @@ static inline bool should_fault_around(struct vm_fault *vmf) return false; /* A single page implies no faulting 'around' at all. */ - return fault_around_pages > 1; + return READ_ONCE(fault_around_bytes) > mm_page_size(vmf->vma->vm_mm); } static vm_fault_t do_read_fault(struct vm_fault *vmf) @@ -5911,6 +7604,10 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct folio *folio; vm_fault_t ret; +#ifdef CONFIG_MM_SUBPAGE + bool pooled = false, force_private = false; + vm_fault_t earlier = 0; +#endif ret = vmf_can_call_fault(vmf); if (!ret) @@ -5918,10 +7615,39 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) if (ret) return ret; +#ifdef CONFIG_MM_SUBPAGE +retry_subpage: + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + /* Known callbacks only supply a locked source page for the core. + * Custom/DAX callbacks may fill a native cow_page or finish COW. + */ + if (!force_private && !userfaultfd_armed(vma) && + (vma->vm_ops->fault == filemap_fault || vma_is_shmem(vma))) { + vmf->cow_subpage = cow_subpage_prealloc(vma->vm_mm, vma, + vmf->address, NULL, false, &pooled, vma->vm_mm); + if (!IS_ERR(vmf->cow_subpage)) + folio_unlock(mm_subpage_folio(vmf->cow_subpage)); + } else { + pooled = false; + vmf->cow_subpage = anon_subpage_prealloc(vma->vm_mm, vma, vmf->address); + } + if (IS_ERR(vmf->cow_subpage)) { + vmf->cow_subpage = NULL; + return earlier | VM_FAULT_OOM; + } + folio = mm_subpage_folio(vmf->cow_subpage); + goto allocated; + } +#endif + folio = folio_prealloc(vma->vm_mm, vma, vmf->address, false); if (!folio) return VM_FAULT_OOM; +#ifdef CONFIG_MM_SUBPAGE +allocated: +#endif + vmf->cow_page = &folio->page; ret = __do_fault(vmf); @@ -5930,20 +7656,80 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) if (ret & VM_FAULT_DONE_COW) return ret; - if (copy_mc_user_highpage(vmf->cow_page, vmf->page, vmf->address, vma)) { - ret = VM_FAULT_HWPOISON; +#ifdef CONFIG_MM_SUBPAGE + if (vmf->cow_subpage) { + int err; + + /* No destination lock crosses the source fault callback or retry. */ + if (pooled) + folio_lock(folio); + err = copy_user_subpage_range(vmf->cow_page, mm_subpage_offset(vmf->cow_subpage), + vmf->page, vma_page_offset_at(vma, vmf->address).offset, + mm_page_size(vma->vm_mm)); + if (pooled) + folio_unlock(folio); + if (err) { + ret = VM_FAULT_HWPOISON; + goto unlock; + } + /* Preserve concurrent sibling lock/LRU/waiter flag updates. */ + folio_mark_uptodate(folio); + goto copied; + } +#endif + +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) > PAGE_SIZE && + !folio_user_range_valid(page_folio(vmf->page), + folio_page_idx(page_folio(vmf->page), vmf->page) << PAGE_SHIFT, + mm_page_size(vma->vm_mm))) { + ret = VM_FAULT_SIGBUS; goto unlock; } +#endif + for (unsigned int i = 0; i < mm_pte_native_pages(vma->vm_mm); i++) { + if (copy_mc_user_highpage(vmf->cow_page + i, vmf->page + i, + vmf->address + i * PAGE_SIZE, vma)) { + ret = VM_FAULT_HWPOISON; + goto unlock; + } + } __folio_mark_uptodate(folio); +#ifdef CONFIG_MM_SUBPAGE +copied: +#endif ret |= finish_fault(vmf); unlock: unlock_page(vmf->page); put_page(vmf->page); +#ifdef CONFIG_MM_SUBPAGE + vmf->page = NULL; + /* Publication OOM left the PTE untouched. Retry with private metadata, + * outside the source lock; never loop internally on callback RETRY. + */ + if (pooled && (ret & VM_FAULT_OOM)) { + mm_subpage_put(vmf->cow_subpage); + vmf->cow_subpage = NULL; + vmf->cow_page = NULL; + earlier |= ret & VM_FAULT_MAJOR; + force_private = true; + goto retry_subpage; + } + ret |= earlier; +#endif if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) goto uncharge_out; return ret; uncharge_out: +#ifdef CONFIG_MM_SUBPAGE + ret |= earlier; + if (vmf->cow_subpage) { + mm_subpage_put(vmf->cow_subpage); + vmf->cow_subpage = NULL; + return ret; + } +#endif folio_put(folio); return ret; } @@ -6104,31 +7890,33 @@ static void numa_rebuild_single_mapping(struct vm_fault *vmf, struct vm_area_str update_mmu_cache_range(vmf, vma, fault_addr, fault_pte, 1); } -static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_struct *vma, +static void numa_rebuild_folio_mapping(struct vm_fault *vmf, struct vm_area_struct *vma, struct folio *folio, pte_t fault_pte, bool ignore_writable, bool pte_write_upgrade) { - int nr = pte_pfn(fault_pte) - folio_pfn(folio); + struct mm_struct *mm = vma->vm_mm; + unsigned int page_shift = mm_page_shift(mm); + unsigned long page_size = 1UL << page_shift; + unsigned long offset = pte_phys_mm(mm, fault_pte) - PFN_PHYS(folio_pfn(folio)); unsigned long start, end, addr = vmf->address; - unsigned long addr_start = addr - (nr << PAGE_SHIFT); - unsigned long pt_start = ALIGN_DOWN(addr, PMD_SIZE); + unsigned long lower = max(addr & pmd_mask_mm(mm), vma->vm_start); + unsigned long upper = pmd_addr_end_mm(mm, addr, vma->vm_end); pte_t *start_ptep; - /* Stay within the VMA and within the page table. */ - start = max3(addr_start, pt_start, vma->vm_start); - end = min3(addr_start + folio_size(folio), pt_start + PMD_SIZE, - vma->vm_end); - start_ptep = vmf->pte - ((addr - start) >> PAGE_SHIFT); + /* Stay within the VMA, page table and physical owner without wrapping. */ + start = addr - min(offset, addr - lower); + end = addr + min(folio_size(folio) - offset, upper - addr); + start_ptep = vmf->pte - ((addr - start) >> page_shift); - /* Restore all PTEs' mapping of the large folio */ - for (addr = start; addr != end; start_ptep++, addr += PAGE_SIZE) { + /* Restore the owner's PTEs, including smaller leaves in a native folio. */ + for (addr = start; addr != end; start_ptep++, addr += page_size) { pte_t ptent = ptep_get(start_ptep); bool writable = false; if (!pte_present(ptent) || !pte_protnone(ptent)) continue; - if (pfn_folio(pte_pfn(ptent)) != folio) + if (pfn_folio(PHYS_PFN(pte_phys_mm(mm, ptent))) != folio) continue; if (!ignore_writable) { @@ -6221,8 +8009,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * Make it present again, depending on how arch implements * non-accessible ptes, some can allow access by kernel mode. */ - if (folio && folio_test_large(folio)) - numa_rebuild_large_mapping(vmf, vma, folio, pte, ignore_writable, + if (folio && (folio_test_large(folio) || mm_page_shift(vma->vm_mm) < PAGE_SHIFT)) + numa_rebuild_folio_mapping(vmf, vma, folio, pte, ignore_writable, pte_write_upgrade); else numa_rebuild_single_mapping(vmf, vma, vmf->address, vmf->pte, @@ -6455,7 +8243,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, { struct vm_fault vmf = { .vma = vma, - .address = address & PAGE_MASK, + .address = address & mm_page_mask(vma->vm_mm), .real_address = address, .flags = flags, .pgoff = linear_page_index(vma, address), @@ -6786,7 +8574,7 @@ int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address) return -ENOMEM; spin_lock(&mm->page_table_lock); - if (!p4d_present(*p4d)) { + if (!p4d_present_mm(mm, *p4d)) { mm_inc_nr_puds(mm); smp_wmb(); /* See comment in pmd_install() */ p4d_populate(mm, p4d, new); @@ -6824,16 +8612,19 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) static inline int pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, - pgprot_t pgprot, unsigned long pfn_base, + pgprot_t pgprot, phys_addr_t phys_base, unsigned long addr_mask, bool writable, bool special) { + phys_addr_t phys = phys_base + (args->address & ~addr_mask); + if (!writable && args->write) return -EFAULT; args->lock = lock; args->ptep = ptep; - args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pfn = PHYS_PFN(phys); + args->offset = offset_in_page(phys); args->addr_mask = addr_mask; args->pgprot = pgprot; args->writable = writable; @@ -6885,8 +8676,14 @@ static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) * * This function must not be used to modify PTE content. * + * Legacy callers retain the native-page contract. A caller setting + * allow_subpage must use pfn plus offset to identify the requested physical + * byte and must limit access to the leaf described by addr_mask. This does + * not pin the native page or grant access to neighbouring user mappings. + * * Return: zero on success, -EFAULT if @args->write was set but the - * mapping is not writable, -EINVAL if there is no mapping at all. + * mapping is not writable, -EINVAL if there is no mapping at all, or + * -EOPNOTSUPP for a non-native granule without allow_subpage. */ int follow_pfnmap_start(struct follow_pfnmap_args *args) { @@ -6908,17 +8705,20 @@ int follow_pfnmap_start(struct follow_pfnmap_args *args) if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) goto out; + /* A legacy PFN consumer cannot represent partial native-page ownership. */ + if (mm_page_size(mm) != PAGE_SIZE && !args->allow_subpage) + return -EOPNOTSUPP; retry: pgdp = pgd_offset(mm, address); if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) goto out; - p4dp = p4d_offset(pgdp, address); + p4dp = p4d_offset_mm(mm, pgdp, address); p4d = p4dp_get(p4dp); - if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + if (p4d_none_mm(mm, p4d) || unlikely(p4d_bad_mm(mm, p4d))) goto out; - pudp = pud_offset(p4dp, address); + pudp = pud_offset_mm(mm, p4dp, address); pud = pudp_get(pudp); if (!pud_present(pud)) goto out; @@ -6934,14 +8734,14 @@ int follow_pfnmap_start(struct follow_pfnmap_args *args) goto retry; } r = pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), - pud_pfn(pud), PUD_MASK, pud_write(pud), + PFN_PHYS(pud_pfn(pud)), pud_mask_mm(mm), pud_write(pud), pud_special(pud)); if (r) spin_unlock(lock); return r; } - pmdp = pmd_offset(pudp, address); + pmdp = pmd_offset_mm(mm, pudp, address); pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) goto out; @@ -6957,7 +8757,7 @@ int follow_pfnmap_start(struct follow_pfnmap_args *args) goto retry; } r = pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), - pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + PFN_PHYS(pmd_pfn(pmd)), pmd_mask_mm(mm), pmd_write(pmd), pmd_special(pmd)); if (r) spin_unlock(lock); @@ -6970,8 +8770,8 @@ int follow_pfnmap_start(struct follow_pfnmap_args *args) pte = ptep_get(ptep); if (!pte_present(pte)) goto unlock; - r = pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), - pte_pfn(pte), PAGE_MASK, pte_write(pte), + r = pfnmap_args_setup(args, lock, ptep, pte_pgprot_mm(mm, pte), + pte_phys_mm(mm, pte), mm_page_mask(mm), pte_write(pte), pte_special(pte)); if (r) goto unlock; @@ -7018,20 +8818,30 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; pgprot_t prot = __pgprot(0); void __iomem *maddr; - int offset = offset_in_page(addr); + int offset; int ret = -EINVAL; bool writable; struct follow_pfnmap_args args = { .vma = vma, .address = addr, - .write = !!(write & FOLL_WRITE) + .write = !!(write & FOLL_WRITE), + .allow_subpage = true }; + if (len <= 0) + return 0; + if (addr < vma->vm_start || addr >= vma->vm_end) + return -EINVAL; + /* Each following leaf needs its own translation and permission check. */ + len = min_t(unsigned long, len, vma->vm_end - addr); + len = min_t(unsigned long, len, mm_page_size(vma->vm_mm) - + (addr & ~mm_page_mask(vma->vm_mm))); retry: if (follow_pfnmap_start(&args)) return -EINVAL; prot = args.pgprot; phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + offset = args.offset; writable = args.writable; follow_pfnmap_end(&args); @@ -7043,7 +8853,8 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, goto out_unmap; if ((pgprot_val(prot) != pgprot_val(args.pgprot)) || - (phys_addr != (args.pfn << PAGE_SHIFT)) || + (phys_addr != ((resource_size_t)args.pfn << PAGE_SHIFT)) || + (offset != args.offset) || (writable != args.writable)) { follow_pfnmap_end(&args); iounmap(maddr); @@ -7089,8 +8900,19 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *maddr; struct folio *folio; struct vm_area_struct *vma = NULL; - struct page *page = get_user_page_vma_remote(mm, addr, - gup_flags, &vma); + struct page *page; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment fragment = {}; + + if (mm_page_size(mm) != PAGE_SIZE) { + int got = get_user_fragment_vma_remote(mm, addr, len, + gup_flags, &fragment, &vma); + + page = got > 0 ? folio_page(fragment.folio, + fragment.offset >> PAGE_SHIFT) : ERR_PTR(got ?: -EFAULT); + } else +#endif + page = get_user_page_vma_remote(mm, addr, gup_flags, &vma); if (IS_ERR(page)) { /* We might need to expand the stack to access it */ @@ -7124,6 +8946,12 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, offset = addr & (PAGE_SIZE-1); if (bytes > PAGE_SIZE-offset) bytes = PAGE_SIZE-offset; +#ifdef CONFIG_MM_SUBPAGE + if (fragment.folio) { + offset = offset_in_page(fragment.offset); + bytes = fragment.length; + } +#endif maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE); if (write) { @@ -7134,7 +8962,13 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, copy_from_user_page(vma, page, addr, buf, maddr + offset, bytes); } - folio_release_kmap(folio, maddr); +#ifdef CONFIG_MM_SUBPAGE + if (fragment.folio) { + kunmap_local(maddr); + release_user_fragments(&fragment, 1, false); + } else +#endif + folio_release_kmap(folio, maddr); } len -= bytes; buf += bytes; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 4e4421b22b59f7..3b97c71c9d3cf5 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -78,6 +78,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include #include #include @@ -688,6 +689,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, struct folio *folio; struct queue_pages *qp = walk->private; unsigned long flags = qp->flags; + unsigned int page_shift = mm_page_shift(walk->mm); + unsigned long page_size = 1UL << page_shift; pte_t *pte, *mapped_pte; pte_t ptent; spinlock_t *ptl; @@ -705,8 +708,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } - for (; addr != end; pte += nr, addr += nr * PAGE_SIZE) { - max_nr = (end - addr) >> PAGE_SHIFT; + for (; addr != end; pte += nr, addr += nr * page_size) { + max_nr = (end - addr) >> page_shift; nr = 1; ptent = ptep_get(pte); if (pte_none(ptent)) @@ -721,7 +724,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio) && max_nr != 1) +#ifdef CONFIG_MM_SUBPAGE + if (page_shift < PAGE_SHIFT) + nr = folio_subpage_pte_batch(vma, folio, pte, ptent, max_nr, 0); +#endif + if (page_shift >= PAGE_SHIFT && folio_test_large(folio) && max_nr != 1) nr = folio_pte_batch(folio, pte, ptent, max_nr); /* * vm_normal_folio() filters out zero pages, but there might @@ -731,9 +738,10 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, continue; if (!queue_folio_required(folio, qp)) continue; - if (folio_test_large(folio)) { + if (folio_test_large(folio) || page_shift < PAGE_SHIFT) { /* - * A large folio can only be isolated from LRU once, + * A native folio backing smaller leaves, like a large + * folio, can only be isolated from LRU once, * but may be mapped by many PTEs (and Copy-On-Write may * intersperse PTEs of other, order 0, folios). This is * a common case, so don't mistake it for failure (but @@ -755,7 +763,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || !vma_migratable(vma) || !migrate_folio_add(folio, qp->pagelist, flags)) { - qp->nr_failed += nr; + qp->nr_failed += page_shift < PAGE_SHIFT ? 1 : nr; if (strictly_unmovable(flags)) break; } @@ -824,6 +832,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, * @folio: The folio whose mapping considered for being made NUMA hintable * @vma: The VMA that the folio belongs to. * @is_private_single_threaded: Is this a single-threaded private VMA or not + * @shared: Is the mapped memory potentially shared between address spaces * * This function checks to see if the folio actually indicates that * we need to make the mapping one which causes a NUMA hinting fault, @@ -833,7 +842,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, * Return: True if the mapping of the folio needs to be changed, false otherwise. */ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, - bool is_private_single_threaded) + bool is_private_single_threaded, bool shared) { int nid; @@ -841,7 +850,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, return false; /* Also skip shared copy-on-write folios */ - if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio)) + if (is_cow_mapping(vma->vm_flags) && shared) return false; /* Folios are pinned and can't be migrated */ @@ -1050,6 +1059,12 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, vmstart = vma->vm_start; } + /* Shared policy is indexed by native page-cache allocation units. */ + if (vma_is_shmem(vma) && + (vma_page_offset_at(vma, vmstart).offset || + ((vmend - vmstart) & ~PAGE_MASK))) + return -EINVAL; + if (mpol_equal(vma->vm_policy, new_pol)) { *prev = vma; return 0; @@ -1135,6 +1150,18 @@ static int lookup_node(struct mm_struct *mm, unsigned long addr) struct page *p = NULL; int ret; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_shift(mm) != PAGE_SHIFT) { + struct user_page_fragment fragment; + + ret = get_user_fragments_remote(mm, addr, 1, 0, &fragment, 1); + if (ret > 0) { + ret = folio_nid(fragment.folio); + release_user_fragments(&fragment, 1, false); + } + return ret; + } +#endif ret = get_user_pages_fast(addr & PAGE_MASK, 1, 0, &p); if (ret > 0) { ret = page_to_nid(p); @@ -1502,13 +1529,13 @@ static long do_mbind(unsigned long start, unsigned long len, if ((flags & MPOL_MF_MOVE_ALL) && !capable(CAP_SYS_NICE)) return -EPERM; - if (start & ~PAGE_MASK) + if (start & ~mm_page_mask(mm)) return -EINVAL; if (mode == MPOL_DEFAULT) flags &= ~MPOL_MF_STRICT; - len = PAGE_ALIGN(len); + len = mm_page_align(mm, len); end = start + len; if (end < start) @@ -1768,7 +1795,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le VMA_ITERATOR(vmi, mm, start); start = untagged_addr(start); - if (start & ~PAGE_MASK) + if (start & ~mm_page_mask(mm)) return -EINVAL; /* * flags is used for future extension if any. @@ -1783,7 +1810,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le if (home_node >= MAX_NUMNODES || !node_online(home_node)) return -EINVAL; - len = PAGE_ALIGN(len); + len = mm_page_align(mm, len); end = start + len; if (end < start) @@ -2049,7 +2076,7 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, if (pol->mode == MPOL_INTERLEAVE || pol->mode == MPOL_WEIGHTED_INTERLEAVE) { *ilx += vma->vm_pgoff >> order; - *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); + *ilx += (vma_page_offset_at(vma, addr).index - vma->vm_pgoff) >> order; } return pol; } @@ -3254,6 +3281,10 @@ int mpol_set_shared_policy(struct shared_policy *sp, struct sp_node *new = NULL; unsigned long sz = vma_pages(vma); + /* Never truncate a partial cache-page policy update to zero or a neighbour. */ + if (vma_subpage_offset(vma) || ((vma->vm_end - vma->vm_start) & ~PAGE_MASK)) + return -EINVAL; + if (pol) { new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, pol); if (!new) diff --git a/mm/migrate.c b/mm/migrate.c index 8a64291ab5b44c..795143e1b4cc44 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -14,6 +14,8 @@ */ #include +#include +#include #include #include #include @@ -340,14 +342,50 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; +#ifdef CONFIG_MM_SUBPAGE +static void remove_migration_subpage(struct folio *folio, + struct page_vma_mapped_walk *pvmw, pte_t old) +{ + struct vm_area_struct *vma = pvmw->vma; + softleaf_t entry = softleaf_from_pte(old); + struct mm_subpage *slot; + pte_t pte; + int ret; + + slot = mm_subpage_migrate_restore(folio, pte_swp_subpage_offset(old)); + pte = phys_pte_mm(vma->vm_mm, mm_subpage_phys(slot), READ_ONCE(vma->vm_page_prot)); + if (!softleaf_is_migration_young(entry)) + pte = pte_mkold(pte); + if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) + pte = pte_mkdirty(pte); + if (pte_swp_soft_dirty(old)) + pte = pte_mksoft_dirty(pte); + else + pte = pte_clear_soft_dirty(pte); + if (softleaf_is_migration_write(entry)) + pte = pte_mkwrite(pte, vma); + else if (pte_swp_uffd_wp(old)) + pte = pte_mkuffd_wp(pte); + if (!softleaf_is_migration_read(entry)) + mm_subpage_set_exclusive(slot); + ret = mm_subpage_add_anon_rmap(slot, vma, pvmw->address); + VM_BUG_ON_FOLIO(ret, folio); + set_pte_at(vma->vm_mm, pvmw->address, pvmw->pte, pte); + if (READ_ONCE(vma->vm_flags) & VM_LOCKED) + mlock_drain_local(); + trace_remove_migration_pte(pvmw->address, pte_val(pte), 0); + update_mmu_cache(vma, pvmw->address, pvmw->pte); +} +#endif + /* * Restore a potential migration pte to a working pte entry */ static bool remove_migration_pte(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr, void *arg) + struct vm_area_struct *vma, struct rmap_walk_range rmap, void *arg) { struct rmap_walk_arg *rmap_walk_arg = arg; - DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); + DEFINE_FOLIO_RMAP_WALK(pvmw, rmap_walk_arg->folio, vma, rmap, PVMW_SYNC | PVMW_MIGRATION); while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; @@ -372,13 +410,29 @@ static bool remove_migration_pte(struct folio *folio, } #endif old_pte = ptep_get(pvmw.pte); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE && folio_test_anon(folio)) { + remove_migration_subpage(folio, &pvmw, old_pte); + continue; + } +#endif if (rmap_walk_arg->map_unused_to_zeropage && try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx)) continue; - folio_get(folio); + /* Match the native-page references dropped by try_to_migrate_one(). */ + if (!folio_test_hugetlb(folio)) + folio_ref_add(folio, mm_pte_native_pages(vma->vm_mm)); + else + folio_get(folio); pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + pte = phys_pte_mm(vma->vm_mm, page_to_phys(new) + + pte_swp_subpage_offset(old_pte), READ_ONCE(vma->vm_page_prot)); +#endif + entry = softleaf_from_pte(old_pte); if (!softleaf_is_migration_young(entry)) pte = pte_mkold(pte); @@ -431,8 +485,10 @@ static bool remove_migration_pte(struct folio *folio, if (folio_test_anon(folio)) folio_add_anon_rmap_pte(folio, new, vma, pvmw.address, rmap_flags); - else + else { + mm_subpage_file_map_add(vma, pte_phys_mm(vma->vm_mm, pte)); folio_add_file_rmap_pte(folio, new, vma); + } set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (READ_ONCE(vma->vm_flags) & VM_LOCKED) @@ -640,10 +696,12 @@ static int __folio_migrate_mapping(struct address_space *mapping, folio_set_dirty(newfolio); } - if (folio_test_swapcache(folio)) + if (folio_test_swapcache(folio)) { __swap_cache_replace_folio(ci, folio, newfolio); - else + } else { + shmem_uffd_migrate(newfolio, folio); xas_store(&xas, newfolio); + } /* * Drop cache reference from old folio by unfreezing @@ -1170,8 +1228,10 @@ static void migrate_folio_undo_src(struct folio *src, /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - if (locked) + if (locked) { + mm_subpage_migrate_finish(src); folio_unlock(src); + } if (ret) list_move_tail(&src->lru, ret); } @@ -1180,8 +1240,10 @@ static void migrate_folio_undo_src(struct folio *src, static void migrate_folio_undo_dst(struct folio *dst, bool locked, free_folio_t put_new_folio, unsigned long private) { - if (locked) + if (locked) { + mm_subpage_migrate_finish(dst); folio_unlock(dst); + } if (put_new_folio) put_new_folio(dst, private); else @@ -1306,6 +1368,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, return 0; } + rc = mm_subpage_migrate_prepare(src, dst); + if (rc) + goto out; + rc = -EAGAIN; + /* * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU @@ -1411,6 +1478,8 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, remove_migration_ptes(src, dst, 0); out_unlock_both: + mm_subpage_migrate_finish(src); + mm_subpage_migrate_finish(dst); folio_unlock(dst); folio_set_owner_migrate_reason(dst, reason); /* diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 17074c2573b94c..4d72fb07810ab9 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -408,7 +408,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (folio_try_share_anon_rmap_pte(folio, page)) { + if (folio_try_share_anon_rmap_pte(folio, page, vma)) { set_pte_at(mm, addr, ptep, pte); if (fault_folio != folio) folio_unlock(folio); @@ -668,7 +668,7 @@ static void migrate_vma_unmap(struct migrate_vma *migrate) * @args: contains the vma, start, and pfns arrays for the migration * * Returns: negative errno on failures, 0 when 0 or more pages were migrated - * without an error. + * without an error. Non-native MM granules return -EOPNOTSUPP. * * Prepare to migrate a range of memory virtual address range by collecting all * the pages backing each virtual address in the range, saving them inside the @@ -731,6 +731,10 @@ int migrate_vma_setup(struct migrate_vma *args) { long nr_pages = (args->end - args->start) >> PAGE_SHIFT; + /* PFN arrays and migration ownership describe whole native pages. */ + if (args->vma && mm_page_size(args->vma->vm_mm) != PAGE_SIZE) + return -EOPNOTSUPP; + args->start &= PAGE_MASK; args->end &= PAGE_MASK; if (!args->vma || is_vm_hugetlb_page(args->vma) || diff --git a/mm/mincore.c b/mm/mincore.c index c8757c5085bf90..66458b6bb20915 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -48,7 +48,7 @@ static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr, present = 1; } - for (; addr != end; vec++, addr += PAGE_SIZE) + for (; addr != end; vec++, addr += mm_page_size(walk->mm)) *vec = present; walk->private = vec; spin_unlock(ptl); @@ -135,15 +135,22 @@ static unsigned char mincore_page(struct address_space *mapping, pgoff_t index) static int __mincore_unmapped_range(unsigned long addr, unsigned long end, struct vm_area_struct *vma, unsigned char *vec) { - unsigned long nr = (end - addr) >> PAGE_SHIFT; + unsigned long nr = (end - addr) >> mm_page_shift(vma->vm_mm); int i; if (vma->vm_file) { - pgoff_t pgoff; + unsigned int shift = mm_page_shift(vma->vm_mm); - pgoff = linear_page_index(vma, addr); - for (i = 0; i < nr; i++, pgoff++) - vec[i] = mincore_page(vma->vm_file->f_mapping, pgoff); + /* Cache indices are native; several user leaves can share one. */ + for (i = 0; i < nr; ) { + struct vm_page_offset pos = vma_page_offset_at(vma, addr); + unsigned long batch = min(nr - i, (PAGE_SIZE - pos.offset) >> shift); + unsigned char present = mincore_page(vma->vm_file->f_mapping, pos.index); + + memset(vec + i, present, batch); + i += batch; + addr += batch << shift; + } } else { for (i = 0; i < nr; i++) vec[i] = 0; @@ -167,7 +174,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct vm_area_struct *vma = walk->vma; pte_t *ptep; unsigned char *vec = walk->private; - int nr = (end - addr) >> PAGE_SHIFT; + int nr = (end - addr) >> mm_page_shift(walk->mm); int step, i; ptl = pmd_trans_huge_lock(pmd, vma); @@ -182,19 +189,19 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, walk->action = ACTION_AGAIN; return 0; } - for (; addr != end; ptep += step, addr += step * PAGE_SIZE) { + for (; addr != end; ptep += step, addr += step * mm_page_size(walk->mm)) { pte_t pte = ptep_get(ptep); step = 1; /* We need to do cache lookup too for markers */ if (pte_none(pte) || pte_is_marker(pte)) - __mincore_unmapped_range(addr, addr + PAGE_SIZE, + __mincore_unmapped_range(addr, addr + mm_page_size(walk->mm), vma, vec); else if (pte_present(pte)) { unsigned int batch = pte_batch_hint(ptep, pte); if (batch > 1) { - unsigned int max_nr = (end - addr) >> PAGE_SHIFT; + unsigned int max_nr = (end - addr) >> mm_page_shift(walk->mm); step = min_t(unsigned int, batch, max_nr); } @@ -252,16 +259,16 @@ static long do_mincore(unsigned long addr, unsigned long pages, unsigned char *v vma = vma_lookup(current->mm, addr); if (!vma) return -ENOMEM; - end = min(vma->vm_end, addr + (pages << PAGE_SHIFT)); + end = min(vma->vm_end, addr + (pages << mm_page_shift(vma->vm_mm))); if (!can_do_mincore(vma)) { - unsigned long pages = DIV_ROUND_UP(end - addr, PAGE_SIZE); + unsigned long pages = DIV_ROUND_UP(end - addr, mm_page_size(vma->vm_mm)); memset(vec, 1, pages); return pages; } err = walk_page_range(vma->vm_mm, addr, end, &mincore_walk_ops, vec); if (err < 0) return err; - return (end - addr) >> PAGE_SHIFT; + return (end - addr) >> mm_page_shift(vma->vm_mm); } /* @@ -281,7 +288,7 @@ static long do_mincore(unsigned long addr, unsigned long pages, unsigned char *v * return values: * zero - success * -EFAULT - vec points to an illegal address - * -EINVAL - addr is not a multiple of PAGE_SIZE + * -EINVAL - addr is not aligned to the process page size * -ENOMEM - Addresses in the range [addr, addr + len] are * invalid for the address space of this process, or * specify one or more pages which are not currently @@ -298,7 +305,7 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len, start = untagged_addr(start); /* Check the start address: needs to be page-aligned.. */ - if (unlikely(start & ~PAGE_MASK)) + if (unlikely(start & ~mm_page_mask(current->mm))) return -EINVAL; /* ..and we need to be passed a valid user-space range */ @@ -306,8 +313,8 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len, return -ENOMEM; /* This also avoids any overflows on PAGE_ALIGN */ - pages = len >> PAGE_SHIFT; - pages += (offset_in_page(len)) != 0; + pages = len >> mm_page_shift(current->mm); + pages += (len & ~mm_page_mask(current->mm)) != 0; if (!access_ok(vec, pages)) return -EFAULT; @@ -334,7 +341,7 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len, } pages -= retval; vec += retval; - start += retval << PAGE_SHIFT; + start += retval << mm_page_shift(current->mm); retval = 0; } free_page((unsigned long) tmp); diff --git a/mm/mlock.c b/mm/mlock.c index 8c227fefa2df0e..a6b67d6b20cac9 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -304,15 +304,19 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } -static inline unsigned int folio_mlock_step(struct folio *folio, +static inline unsigned int folio_mlock_step(struct vm_area_struct *vma, struct folio *folio, pte_t *pte, unsigned long addr, unsigned long end) { - unsigned int count = (end - addr) >> PAGE_SHIFT; + unsigned int count = (end - addr) >> mm_page_shift(vma->vm_mm); pte_t ptent = ptep_get(pte); if (!folio_test_large(folio)) return 1; + #ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return folio_subpage_pte_batch(vma, folio, pte, ptent, count, 0); +#endif return folio_pte_batch(folio, pte, ptent, count); } @@ -341,7 +345,7 @@ static inline bool allow_mlock_munlock(struct folio *folio, return false; /* folio is not fully mapped, skip mlock */ - if (step != folio_nr_pages(folio)) + if (step != (folio_size(folio) >> mm_page_shift(vma->vm_mm))) return false; return true; @@ -381,7 +385,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { + for (pte = start_pte; addr != end; pte++, addr += mm_page_size(vma->vm_mm)) { ptent = ptep_get(pte); if (!pte_present(ptent)) continue; @@ -389,7 +393,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, if (!folio || folio_is_zone_device(folio)) continue; - step = folio_mlock_step(folio, pte, addr, end); + step = folio_mlock_step(vma, folio, pte, addr, end); if (!allow_mlock_munlock(folio, vma, start, end, step)) goto next_entry; @@ -400,7 +404,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, next_entry: pte += step - 1; - addr += (step - 1) << PAGE_SHIFT; + addr += (step - 1) << mm_page_shift(vma->vm_mm); } pte_unmap(start_pte); out: @@ -471,7 +475,7 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); const vma_flags_t old_vma_flags = vma->flags; struct mm_struct *mm = vma->vm_mm; - int nr_pages; + long nr_pages; int ret = 0; if (vma_flags_same_pair(&old_vma_flags, &new_vma_flags) || @@ -492,7 +496,7 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, /* * Keep track of amount of locked VM. */ - nr_pages = (end - start) >> PAGE_SHIFT; + nr_pages = (end - start) >> MM_ACCOUNT_SHIFT; if (!vma_flags_test(&new_vma_flags, VMA_LOCKED_BIT)) nr_pages = -nr_pages; else if (vma_flags_test(&old_vma_flags, VMA_LOCKED_BIT)) @@ -524,8 +528,8 @@ static int apply_vma_lock_flags(unsigned long start, size_t len, struct vm_area_struct *vma, *prev; VMA_ITERATOR(vmi, current->mm, start); - VM_BUG_ON(offset_in_page(start)); - VM_BUG_ON(len != PAGE_ALIGN(len)); + VM_BUG_ON((start & ~mm_page_mask(current->mm))); + VM_BUG_ON(len != mm_page_align(current->mm, len)); end = start + len; if (end < start) return -EINVAL; @@ -572,7 +576,7 @@ static int apply_vma_lock_flags(unsigned long start, size_t len, * vma pages, as return value. * Note deferred memory locking case(mlock2(,,MLOCK_ONFAULT) * is also counted. - * Return value: previously mlocked page counts + * Return value: previously mlocked MM_ACCOUNT_SHIFT units */ static unsigned long count_mm_mlocked_page_nr(struct mm_struct *mm, unsigned long start, size_t len) @@ -600,7 +604,7 @@ static unsigned long count_mm_mlocked_page_nr(struct mm_struct *mm, } } - return count >> PAGE_SHIFT; + return count >> MM_ACCOUNT_SHIFT; } /* @@ -626,12 +630,12 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t fla if (!can_do_mlock()) return -EPERM; - len = PAGE_ALIGN(len + (offset_in_page(start))); - start &= PAGE_MASK; + len = mm_page_align(current->mm, len + (start & ~mm_page_mask(current->mm))); + start &= mm_page_mask(current->mm); lock_limit = rlimit(RLIMIT_MEMLOCK); - lock_limit >>= PAGE_SHIFT; - locked = len >> PAGE_SHIFT; + lock_limit >>= MM_ACCOUNT_SHIFT; + locked = len >> MM_ACCOUNT_SHIFT; if (mmap_write_lock_killable(current->mm)) return -EINTR; @@ -686,8 +690,8 @@ SYSCALL_DEFINE2(munlock, unsigned long, start, size_t, len) start = untagged_addr(start); - len = PAGE_ALIGN(len + (offset_in_page(start))); - start &= PAGE_MASK; + len = mm_page_align(current->mm, len + (start & ~mm_page_mask(current->mm))); + start &= mm_page_mask(current->mm); if (mmap_write_lock_killable(current->mm)) return -EINTR; @@ -761,7 +765,7 @@ SYSCALL_DEFINE1(mlockall, int, flags) return -EPERM; lock_limit = rlimit(RLIMIT_MEMLOCK); - lock_limit >>= PAGE_SHIFT; + lock_limit >>= mm_page_shift(current->mm); if (mmap_write_lock_killable(current->mm)) return -EINTR; diff --git a/mm/mm_init.c b/mm/mm_init.c index c7e498fc1c4c01..8d14ce83be36b9 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -57,7 +57,8 @@ unsigned long zero_page_pfn __ro_after_init; EXPORT_SYMBOL(zero_page_pfn); #ifndef __HAVE_COLOR_ZERO_PAGE -uint8_t empty_zero_page[PAGE_SIZE] __page_aligned_bss; +uint8_t empty_zero_page[ARCH_ZERO_PAGE_SIZE] __page_aligned_bss + __aligned(ARCH_ZERO_PAGE_SIZE); EXPORT_SYMBOL(empty_zero_page); struct page *__zero_page __ro_after_init; diff --git a/mm/mmap.c b/mm/mmap.c index 2311ae7c2ff45c..097161e3e61efd 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -151,8 +151,8 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) mm->end_data, mm->start_data)) goto out; - newbrk = PAGE_ALIGN(brk); - oldbrk = PAGE_ALIGN(mm->brk); + newbrk = mm_page_align(mm, brk); + oldbrk = mm_page_align(mm, mm->brk); if (oldbrk == newbrk) { mm->brk = brk; goto success; @@ -186,8 +186,8 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) * expansion area */ vma_iter_init(&vmi, mm, oldbrk); - next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) + next = vma_find(&vmi, newbrk + mm_page_size(mm) + stack_guard_gap); + if (next && newbrk + mm_page_size(mm) > vm_start_gap(next)) goto out; brkvma = vma_prev_limit(&vmi, mm->start_brk); @@ -218,12 +218,12 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) * If a hint addr is less than mmap_min_addr change hint to be as * low as possible but still greater than mmap_min_addr */ -static inline unsigned long round_hint_to_min(unsigned long hint) +static inline unsigned long round_hint_to_min(struct mm_struct *mm, unsigned long hint) { - hint &= PAGE_MASK; + hint &= mm_page_mask(mm); if (((void *)hint != NULL) && (hint < mmap_min_addr)) - return PAGE_ALIGN(mmap_min_addr); + return mm_page_align(mm, mmap_min_addr); return hint; } @@ -235,11 +235,11 @@ bool mlock_future_ok(const struct mm_struct *mm, bool is_vma_locked, if (!is_vma_locked || capable(CAP_IPC_LOCK)) return true; - locked_pages = bytes >> PAGE_SHIFT; + locked_pages = bytes >> MM_ACCOUNT_SHIFT; locked_pages += mm->locked_vm; limit_pages = rlimit(RLIMIT_MEMLOCK); - limit_pages >>= PAGE_SHIFT; + limit_pages >>= MM_ACCOUNT_SHIFT; return locked_pages <= limit_pages; } @@ -264,24 +264,25 @@ static inline u64 file_mmap_size_max(struct file *file, struct inode *inode) } static inline bool file_mmap_ok(struct file *file, struct inode *inode, - unsigned long pgoff, unsigned long len) + struct vm_page_offset pos, unsigned long len) { u64 maxsize = file_mmap_size_max(file, inode); if (maxsize && len > maxsize) return false; maxsize -= len; - if (pgoff > maxsize >> PAGE_SHIFT) + if (pos.index > maxsize >> PAGE_SHIFT || + (pos.index == maxsize >> PAGE_SHIFT && pos.offset > (maxsize & ~PAGE_MASK))) return false; return true; } /** - * do_mmap() - Perform a userland memory mapping into the current process + * do_mmap_internal() - Perform a userland memory mapping into the current process * address space of length @len with protection bits @prot, mmap flags @flags * (from which VMA flags will be inferred), and any additional VMA flags to * apply @vm_flags. If this is a file-backed mapping then the file is specified - * in @file and page offset into the file via @pgoff. + * in @file and its byte position via @pos. * * This function does not perform security checks on the file and assumes, if * @uf is non-NULL, the caller has provided a list head to track unmap events @@ -321,7 +322,7 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode, * @flags: Flags specifying how the mapping should be performed, see mmap (2) * for details. * @vm_flags: VMA flags which should be set by default, or 0 otherwise. - * @pgoff: Page offset into the @file if file-backed, should be 0 otherwise. + * @pos: Native cache index and byte offset; ignored for anonymous mappings. * @populate: A pointer to a value which will be set to 0 if no population of * the range is required, or the number of bytes to populate if it is. Must be * non-NULL. See mmap (2) for details as to under what circumstances population @@ -333,20 +334,22 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode, * Returns: Either an error, or the address at which the requested mapping has * been performed. */ -unsigned long do_mmap(struct file *file, unsigned long addr, +static unsigned long do_mmap_internal(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, vm_flags_t vm_flags, - unsigned long pgoff, unsigned long *populate, + struct vm_page_offset pos, unsigned long *populate, struct list_head *uf) { struct mm_struct *mm = current->mm; + unsigned long pgoff = pos.index; int pkey = 0; *populate = 0; mmap_assert_write_locked(mm); - if (!len) + if (!len || pos.offset >= PAGE_SIZE || + !IS_ALIGNED(pos.offset, mm_page_size(mm))) return -EINVAL; /* @@ -364,15 +367,15 @@ unsigned long do_mmap(struct file *file, unsigned long addr, flags |= MAP_FIXED; if (!(flags & MAP_FIXED)) - addr = round_hint_to_min(addr); + addr = round_hint_to_min(mm, addr); /* Careful about overflows.. */ - len = PAGE_ALIGN(len); + len = mm_page_align(mm, len); if (!len) return -ENOMEM; /* offset overflow? */ - if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) + if (vm_page_offset_add(pos, len).index < pgoff) return -EOVERFLOW; /* Too many mappings? */ @@ -426,7 +429,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long flags_mask; int err; - if (!file_mmap_ok(file, inode, pgoff, len)) + if (!file_mmap_ok(file, inode, pos, len)) return -EOVERFLOW; flags_mask = LEGACY_MAP_MASK; @@ -500,6 +503,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, * Ignore pgoff. */ pgoff = 0; + pos.offset = 0; vm_flags |= VM_SHARED | VM_MAYSHARE; break; case MAP_DROPPABLE: @@ -537,6 +541,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, * Set pgoff according to addr for anon_vma. */ pgoff = addr >> PAGE_SHIFT; + pos.offset = addr & ~PAGE_MASK; break; default: return -EINVAL; @@ -557,7 +562,11 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |= VM_NORESERVE; } - addr = mmap_region(file, addr, len, vm_flags, pgoff, uf); + addr = mmap_region(file, addr, len, vm_flags, + (struct vm_page_offset) { + .index = pgoff, + .offset = pos.offset, + }, uf); if (!IS_ERR_VALUE(addr) && ((vm_flags & VM_LOCKED) || (flags & (MAP_POPULATE | MAP_NONBLOCK)) == MAP_POPULATE)) @@ -565,9 +574,28 @@ unsigned long do_mmap(struct file *file, unsigned long addr, return addr; } -unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, +unsigned long do_mmap(struct file *file, unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, + struct list_head *uf) +{ + return do_mmap_internal(file, addr, len, prot, flags, vm_flags, + (struct vm_page_offset) { pgoff, 0 }, populate, uf); +} + +#ifdef CONFIG_MM_SUBPAGE +unsigned long do_mmap_offset(struct file *file, unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, struct vm_page_offset pos, unsigned long *populate, + struct list_head *uf) +{ + return do_mmap_internal(file, addr, len, prot, flags, vm_flags, pos, populate, uf); +} +#endif + +static unsigned long ksys_mmap_offset(unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, - unsigned long fd, unsigned long pgoff) + unsigned long fd, unsigned long pgoff, unsigned int suboffset) { struct file *file = NULL; unsigned long retval; @@ -603,13 +631,34 @@ unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, return PTR_ERR(file); } - retval = vm_mmap_pgoff(file, addr, len, prot, flags, pgoff); + if (suboffset) + retval = vm_mmap(file, addr, len, prot, flags, + (pgoff << PAGE_SHIFT) | suboffset); + else + retval = vm_mmap_pgoff(file, addr, len, prot, flags, pgoff); out_fput: if (file) fput(file); return retval; } +unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, + unsigned long prot, unsigned long flags, unsigned long fd, unsigned long pgoff) +{ + return ksys_mmap_offset(addr, len, prot, flags, fd, pgoff, 0); +} + +#ifdef CONFIG_MM_SUBPAGE +unsigned long ksys_mmap_user(unsigned long addr, unsigned long len, + unsigned long prot, unsigned long flags, unsigned long fd, unsigned long offset) +{ + if (!IS_ALIGNED(offset, mm_page_size(current->mm))) + return -EINVAL; + return ksys_mmap_offset(addr, len, prot, flags, fd, + offset >> PAGE_SHIFT, offset & ~PAGE_MASK); +} +#endif + SYSCALL_DEFINE6(mmap_pgoff, unsigned long, addr, unsigned long, len, unsigned long, prot, unsigned long, flags, unsigned long, fd, unsigned long, pgoff) @@ -703,7 +752,7 @@ generic_get_unmapped_area(struct file *filp, unsigned long addr, return addr; if (addr) { - addr = PAGE_ALIGN(addr); + addr = mm_page_align(mm, addr); vma = find_vma_prev(mm, addr, &prev); if (mmap_end - len >= addr && addr >= mmap_min_addr && (!vma || addr + len <= vm_start_gap(vma)) && @@ -754,7 +803,7 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, /* requesting a specific address */ if (addr) { - addr = PAGE_ALIGN(addr); + addr = mm_page_align(mm, addr); vma = find_vma_prev(mm, addr, &prev); if (mmap_end - len >= addr && addr >= mmap_min_addr && (!vma || addr + len <= vm_start_gap(vma)) && @@ -764,7 +813,7 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, info.flags = VM_UNMAPPED_AREA_TOPDOWN; info.length = len; - info.low_limit = PAGE_SIZE; + info.low_limit = mm_page_size(mm); info.high_limit = arch_get_mmap_base(addr, mm->mmap_base); info.start_gap = stack_guard_placement(vm_flags); if (filp && is_file_hugepages(filp)) @@ -777,7 +826,7 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, * can happen with large stack limits and large mmap() * allocations. */ - if (offset_in_page(addr)) { + if (addr & ~mm_page_mask(mm)) { VM_BUG_ON(addr != -ENOMEM); info.flags = 0; info.low_limit = TASK_UNMAPPED_BASE; @@ -857,7 +906,7 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (addr > TASK_SIZE - len) return -ENOMEM; - if (offset_in_page(addr)) + if (addr & ~mm_page_mask(current->mm)) return -EINVAL; error = security_mmap_addr(addr); @@ -962,7 +1011,7 @@ struct vm_area_struct *find_extend_vma_locked(struct mm_struct *mm, unsigned lon { struct vm_area_struct *vma, *prev; - addr &= PAGE_MASK; + addr &= mm_page_mask(mm); vma = find_vma_prev(mm, addr, &prev); if (vma && (vma->vm_start <= addr)) return vma; @@ -985,7 +1034,7 @@ struct vm_area_struct *find_extend_vma_locked(struct mm_struct *mm, unsigned lon struct vm_area_struct *vma; unsigned long start; - addr &= PAGE_MASK; + addr &= mm_page_mask(mm); vma = find_vma(mm, addr); if (!vma) return NULL; @@ -1214,7 +1263,7 @@ int vm_brk_flags(unsigned long addr, unsigned long request, bool is_exec) LIST_HEAD(uf); VMA_ITERATOR(vmi, mm, addr); - len = PAGE_ALIGN(request); + len = mm_page_align(mm, request); if (len < request) return -ENOMEM; if (!len) @@ -1257,7 +1306,7 @@ unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi, vma_iter_set(vmi, vma->vm_end); do { if (vma->vm_flags & VM_ACCOUNT) - nr_accounted += vma_pages(vma); + nr_accounted += vma_user_pages(vma); vma_mark_detached(vma); remove_vma(vma); count++; @@ -1322,8 +1371,8 @@ void exit_mmap(struct mm_struct *mm) destroy: __mt_destroy(&mm->mm_mt); trace_exit_mmap(mm); + mm_unacct_memory(mm, nr_accounted); mmap_write_unlock(mm); - vm_unacct_memory(nr_accounted); } /* @@ -1333,19 +1382,19 @@ void exit_mmap(struct mm_struct *mm) bool may_expand_vm(struct mm_struct *mm, const vma_flags_t *vma_flags, unsigned long npages) { - if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> PAGE_SHIFT) + if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> mm_page_shift(mm)) return false; if (is_data_mapping_vma_flags(vma_flags) && - mm->data_vm + npages > rlimit(RLIMIT_DATA) >> PAGE_SHIFT) { + mm->data_vm + npages > rlimit(RLIMIT_DATA) >> mm_page_shift(mm)) { /* Workaround for Valgrind */ if (rlimit(RLIMIT_DATA) == 0 && - mm->data_vm + npages <= rlimit_max(RLIMIT_DATA) >> PAGE_SHIFT) + mm->data_vm + npages <= rlimit_max(RLIMIT_DATA) >> mm_page_shift(mm)) return true; pr_warn_once("%s (%d): VmData %lu exceed data ulimit %lu. Update limits%s.\n", current->comm, current->pid, - (mm->data_vm + npages) << PAGE_SHIFT, + (mm->data_vm + npages) << mm_page_shift(mm), rlimit(RLIMIT_DATA), ignore_rlimit_data ? "" : " or use boot option ignore_rlimit_data"); @@ -1474,7 +1523,7 @@ static struct vm_area_struct *__install_special_mapping( if (ret) goto out; - vm_stat_account(mm, vma->vm_flags, len >> PAGE_SHIFT); + vm_stat_account(mm, vma->vm_flags, len >> mm_page_shift(mm)); perf_event_mmap(vma); @@ -1771,14 +1820,14 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) if (retval) goto loop_out; - vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); + vm_stat_account(mm, mpnt->vm_flags, -vma_user_pages(mpnt)); continue; } charge = 0; if (mpnt->vm_flags & VM_ACCOUNT) { - unsigned long len = vma_pages(mpnt); + unsigned long len = vma_user_pages(mpnt); - if (security_vm_enough_memory_mm(oldmm, len)) /* sic */ + if (mm_account_memory(mm, oldmm, len)) /* sic */ goto fail_nomem; charge = len; } @@ -1890,7 +1939,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) flush_cache_mm(mm); unmap_region(&unmap); charge = tear_down_vmas(mm, &vmi, tmp, end); - vm_unacct_memory(charge); + mm_unacct_memory(mm, charge); } __mt_destroy(&mm->mm_mt); /* @@ -1916,6 +1965,6 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) vm_area_free(tmp); fail_nomem: retval = -ENOMEM; - vm_unacct_memory(charge); + mm_unacct_memory(mm, charge); goto loop_out; } diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 3985d856de7f9b..42419a47a1880d 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -12,6 +12,9 @@ #include #include #include +#include + +#include "user-subpage-internal.h" #include @@ -100,6 +103,49 @@ void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma) */ #define MAX_NR_FOLIOS_PER_FREE 512 +#ifdef CONFIG_MM_SUBPAGE +/* Private to gather batches: never pass this token to encoded_page_ptr(). */ +#define TLB_SUBPAGE_TAG BIT(2) + +static void tlb_free_pages_and_subpages(struct encoded_page **pages, unsigned int nr) +{ + unsigned int start = 0, i = 0; + + while (i < nr) { + unsigned long token = (unsigned long)pages[i]; + + if (token & TLB_SUBPAGE_TAG) { + if (i != start) + free_pages_and_swap_cache(pages + start, i - start); + mm_subpage_put((struct mm_subpage *)(token & ~TLB_SUBPAGE_TAG)); + start = ++i; + } else { + /* The count payload is not a pointer and may have bit 2 set. */ + i += encoded_page_flags(pages[i]) & ENCODED_PAGE_BIT_NR_PAGES_NEXT ? 2 : 1; + } + } + if (i != start) + free_pages_and_swap_cache(pages + start, i - start); +} + +bool __tlb_remove_subpage(struct mmu_gather *tlb, struct mm_subpage *subpage) +{ + struct mmu_gather_batch *batch = tlb->active; + + BUILD_BUG_ON(__alignof__(struct page) < 8); + BUILD_BUG_ON(__alignof__(struct mm_subpage) < 8); + BUILD_BUG_ON(TLB_SUBPAGE_TAG & ENCODED_PAGE_BITS); + VM_BUG_ON(!tlb->end); + VM_BUG_ON(mm_page_shift(tlb->mm) != mm_subpage_shift(subpage)); + batch->encoded_pages[batch->nr++] = + (struct encoded_page *)((unsigned long)subpage | TLB_SUBPAGE_TAG); + /* Keep room for a following native page/count pair. */ + return batch->nr >= batch->max - 1 && !tlb_next_batch(tlb); +} +#else +#define tlb_free_pages_and_subpages free_pages_and_swap_cache +#endif + static void __tlb_batch_free_encoded_pages(struct mmu_gather_batch *batch) { struct encoded_page **pages = batch->encoded_pages; @@ -135,7 +181,7 @@ static void __tlb_batch_free_encoded_pages(struct mmu_gather_batch *batch) } } - free_pages_and_swap_cache(pages, nr); + tlb_free_pages_and_subpages(pages, nr); pages += nr; batch->nr -= nr; @@ -174,7 +220,15 @@ static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb, #ifdef CONFIG_MMU_GATHER_PAGE_SIZE VM_WARN_ON(tlb->page_size != page_size); - VM_WARN_ON_ONCE(nr_pages != 1 && page_size != PAGE_SIZE); + if (page_size > PAGE_SIZE && page_size == mm_page_size(tlb->mm)) { + /* A coarse base PTE owns one reference per covered native page. */ + unsigned long span = mm_pte_native_pages(tlb->mm); + + VM_WARN_ON_ONCE(!nr_pages || nr_pages % span || + !IS_ALIGNED(page_to_pfn(page), span)); + } else { + VM_WARN_ON_ONCE(nr_pages != 1 && page_size != PAGE_SIZE); + } VM_WARN_ON_ONCE(page_folio(page) != page_folio(page + nr_pages - 1)); #endif @@ -208,7 +262,7 @@ bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, unsigned int nr_pages, bool delay_rmap) { return __tlb_remove_folio_pages_size(tlb, page, nr_pages, delay_rmap, - PAGE_SIZE); + mm_page_size(tlb->mm)); } bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) @@ -352,6 +406,13 @@ static inline void __tlb_remove_table_one(void *table) { struct ptdesc *ptdesc; + /* Preserve opaque architecture tokens through the RCU grace period. */ + if (!tlb_table_is_ptdesc(table)) { + tlb_remove_table_sync_rcu(); + __tlb_remove_table(table); + return; + } + ptdesc = table; call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } diff --git a/mm/mprotect.c b/mm/mprotect.c index 9cbf932b028cf9..0f5ea22bf4c907 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -58,6 +59,31 @@ static bool maybe_change_pte_writable(struct vm_area_struct *vma, pte_t pte) return true; } +#ifdef CONFIG_MM_SUBPAGE +static bool subpage_pte_exclusive(struct vm_area_struct *vma, pte_t pte) +{ + struct mm_subpage *slot = mm_subpage_get_from_phys(pte_phys_mm(vma->vm_mm, pte)); + bool exclusive; + + if (!slot) + return false; + exclusive = mm_subpage_is_exclusive(slot); + mm_subpage_put(slot); + return exclusive; +} + +static int subpage_exclusive_pte_batch(struct vm_area_struct *vma, pte_t *ptep, + int nr_ptes, bool exclusive) +{ + int nr; + + for (nr = 1; nr < nr_ptes; nr++) + if (subpage_pte_exclusive(vma, ptep_get(ptep + nr)) != exclusive) + break; + return nr; +} +#endif + static bool can_change_private_pte_writable(struct vm_area_struct *vma, unsigned long addr, pte_t pte) { @@ -73,6 +99,10 @@ static bool can_change_private_pte_writable(struct vm_area_struct *vma, * any additional checks while holding the PT lock. */ page = vm_normal_page(vma, addr, pte); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return page && PageAnon(page) && subpage_pte_exclusive(vma, pte); +#endif return page && PageAnon(page) && PageAnonExclusive(page); } @@ -103,13 +133,20 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, return can_change_shared_pte_writable(vma, pte); } -static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep, - pte_t pte, int max_nr_ptes, fpb_t flags) + +static __always_inline int mprotect_folio_pte_batch(struct vm_area_struct *vma, + struct folio *folio, pte_t *ptep, pte_t pte, int max_nr_ptes, + fpb_t flags, unsigned int page_shift) { /* No underlying folio, so cannot batch */ if (!folio) return 1; +#ifdef CONFIG_MM_SUBPAGE + if (page_shift != PAGE_SHIFT) + return folio_subpage_pte_batch(vma, folio, ptep, pte, max_nr_ptes, flags); +#endif + if (!folio_test_large(folio)) return 1; @@ -125,17 +162,17 @@ static __always_inline void prot_commit_flush_ptes(struct vm_area_struct *vma, * Advance the position in the batch by idx; note that if idx > 0, * then the nr_ptes passed here is <= batch size - idx. */ - addr += idx * PAGE_SIZE; + addr += idx * mm_page_size(vma->vm_mm); ptep += idx; - oldpte = pte_advance_pfn(oldpte, idx); - ptent = pte_advance_pfn(ptent, idx); + oldpte = pte_advance_pfn_mm(vma->vm_mm, oldpte, idx); + ptent = pte_advance_pfn_mm(vma->vm_mm, ptent, idx); if (set_write) ptent = pte_mkwrite(ptent, vma); modify_prot_commit_ptes(vma, addr, ptep, oldpte, ptent, nr_ptes); if (pte_needs_flush(oldpte, ptent)) - tlb_flush_pte_range(tlb, addr, nr_ptes * PAGE_SIZE); + tlb_flush_pte_range(tlb, addr, nr_ptes * mm_page_size(vma->vm_mm)); } /* @@ -177,6 +214,25 @@ static __always_inline void commit_anon_folio_batch(struct vm_area_struct *vma, int sub_batch_idx = 0; int len; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + while (nr_ptes) { + expected_anon_exclusive = subpage_pte_exclusive(vma, + pte_advance_pfn_mm(vma->vm_mm, oldpte, sub_batch_idx)); + for (len = 1; len < nr_ptes; len++) + if (subpage_pte_exclusive(vma, + pte_advance_pfn_mm(vma->vm_mm, oldpte, sub_batch_idx + len)) != + expected_anon_exclusive) + break; + prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } + return; + } +#endif + while (nr_ptes) { expected_anon_exclusive = PageAnonExclusive(first_page + sub_batch_idx); len = page_anon_exclusive_sub_batch(sub_batch_idx, nr_ptes, @@ -265,6 +321,11 @@ static long change_softleaf_pte(struct vm_area_struct *vma, newpte = oldpte; } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + newpte = pte_swp_set_subpage_offset(newpte, pte_swp_subpage_offset(oldpte)); +#endif + if (uffd_wp) newpte = pte_swp_mkuffd_wp(newpte); else if (uffd_wp_resolve) @@ -320,6 +381,9 @@ static long change_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + /* The mm's mapping granule cannot change during the walk. */ + const unsigned int page_shift = mm_page_shift(vma->vm_mm); + const unsigned long page_size = 1UL << page_shift; pte_t *pte, oldpte; spinlock_t *ptl; long pages = 0; @@ -328,7 +392,7 @@ static long change_pte_range(struct mmu_gather *tlb, bool uffd_wp = cp_flags & MM_CP_UFFD_WP; int nr_ptes; - tlb_change_page_size(tlb, PAGE_SIZE); + tlb_change_page_size(tlb, page_size); pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (!pte) return -EAGAIN; @@ -343,7 +407,7 @@ static long change_pte_range(struct mmu_gather *tlb, oldpte = ptep_get(pte); if (pte_present(oldpte)) { const fpb_t flags = FPB_RESPECT_SOFT_DIRTY | FPB_RESPECT_WRITE; - int max_nr_ptes = (end - addr) >> PAGE_SHIFT; + int max_nr_ptes = (end - addr) >> page_shift; struct folio *folio = NULL; struct page *page; @@ -359,17 +423,32 @@ static long change_pte_range(struct mmu_gather *tlb, * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ - if (prot_numa && - !folio_can_map_prot_numa(folio, vma, - is_private_single_threaded)) { - - /* determine batch to skip */ - nr_ptes = mprotect_folio_pte_batch(folio, - pte, oldpte, max_nr_ptes, /* flags = */ 0); - continue; + if (prot_numa) { + bool shared = folio && folio_maybe_mapped_shared(folio); + +#ifdef CONFIG_MM_SUBPAGE + /* Multiple private leaves do not imply shared COW data. */ + if (folio && folio_test_anon(folio) && page_shift < PAGE_SHIFT) { + shared = !subpage_pte_exclusive(vma, oldpte); + max_nr_ptes = folio_subpage_pte_batch(vma, folio, + pte, oldpte, max_nr_ptes, flags); + /* COW sharing can differ within one physical owner. */ + max_nr_ptes = subpage_exclusive_pte_batch(vma, pte, + max_nr_ptes, !shared); + } +#endif + if (!folio_can_map_prot_numa(folio, vma, + is_private_single_threaded, shared)) { + /* determine batch to skip */ + nr_ptes = mprotect_folio_pte_batch(vma, folio, + pte, oldpte, max_nr_ptes, /* flags = */ 0, + page_shift); + continue; + } } - nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags); + nr_ptes = mprotect_folio_pte_batch(vma, folio, pte, oldpte, + max_nr_ptes, flags, page_shift); /* * Optimize for the small-folio common case by @@ -408,7 +487,7 @@ static long change_pte_range(struct mmu_gather *tlb, } else { pages += change_softleaf_pte(vma, addr, pte, oldpte, cp_flags); } - } while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end); + } while (pte += nr_ptes, addr += nr_ptes * page_size, addr != end); lazy_mmu_mode_disable(); pte_unmap_unlock(pte - 1, ptl); @@ -487,12 +566,12 @@ static inline long change_pmd_range(struct mmu_gather *tlb, long pages = 0; unsigned long nr_huge_updates = 0; - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(vma->vm_mm, pud, addr); do { long ret; pmd_t _pmd; again: - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(vma->vm_mm, addr, end); ret = change_pmd_prepare(vma, pmd, cp_flags); if (ret) { @@ -559,10 +638,10 @@ static inline long change_pud_range(struct mmu_gather *tlb, range.start = 0; - pudp = pud_offset(p4d, addr); + pudp = pud_offset_mm(vma->vm_mm, p4d, addr); do { again: - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(vma->vm_mm, addr, end); ret = change_prepare(vma, pudp, pmd, addr, cp_flags); if (ret) { pages = ret; @@ -615,13 +694,13 @@ static inline long change_p4d_range(struct mmu_gather *tlb, unsigned long next; long pages = 0, ret; - p4d = p4d_offset(pgd, addr); + p4d = p4d_offset_mm(vma->vm_mm, pgd, addr); do { - next = p4d_addr_end(addr, end); + next = p4d_addr_end_mm(vma->vm_mm, addr, end); ret = change_prepare(vma, p4d, pud, addr, cp_flags); if (ret) return ret; - if (p4d_none_or_clear_bad(p4d)) + if (p4d_none_or_clear_bad_mm(vma->vm_mm, p4d)) continue; pages += change_pud_range(tlb, vma, p4d, addr, next, newprot, cp_flags); @@ -643,7 +722,7 @@ static long change_protection_range(struct mmu_gather *tlb, pgd = pgd_offset(mm, addr); tlb_start_vma(tlb, vma); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(vma->vm_mm, addr, end); ret = change_prepare(vma, pgd, p4d, addr, cp_flags); if (ret) { pages = ret; @@ -729,7 +808,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, struct mm_struct *mm = vma->vm_mm; const vma_flags_t old_vma_flags = READ_ONCE(vma->flags); vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); - long nrpages = (end - start) >> PAGE_SHIFT; + long nrpages = (end - start) >> mm_page_shift(mm); unsigned int mm_cp_flags = 0; unsigned long charged = 0; int error; @@ -777,7 +856,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, VMA_ACCOUNT_BIT, VMA_WRITE_BIT, VMA_HUGETLB_BIT, VMA_SHARED_BIT, VMA_NORESERVE_BIT)) { charged = nrpages; - if (security_vm_enough_memory_mm(mm, charged)) + if (mm_account_memory(mm, mm, charged)) return -ENOMEM; vma_flags_set(&new_vma_flags, VMA_ACCOUNT_BIT); } @@ -808,7 +887,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, if (vma_flags_test(&old_vma_flags, VMA_ACCOUNT_BIT) && !vma_flags_test(&new_vma_flags, VMA_ACCOUNT_BIT)) - vm_unacct_memory(nrpages); + mm_unacct_memory(mm, nrpages); /* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major @@ -826,7 +905,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, return 0; fail: - vm_unacct_memory(charged); + mm_unacct_memory(mm, charged); return error; } @@ -851,11 +930,11 @@ static int do_mprotect_pkey(unsigned long start, size_t len, if (grows == (PROT_GROWSDOWN|PROT_GROWSUP)) /* can't be both */ return -EINVAL; - if (start & ~PAGE_MASK) + if (start & ~mm_page_mask(current->mm)) return -EINVAL; if (!len) return 0; - len = PAGE_ALIGN(len); + len = mm_page_align(current->mm, len); end = start + len; if (end <= start) return -ENOMEM; diff --git a/mm/mremap.c b/mm/mremap.c index e9c8b1d05832be..97d5ebf9452852 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -82,11 +82,11 @@ static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) if (pgd_none_or_clear_bad(pgd)) return NULL; - p4d = p4d_offset(pgd, addr); - if (p4d_none_or_clear_bad(p4d)) + p4d = p4d_offset_mm(mm, pgd, addr); + if (p4d_none_or_clear_bad_mm(mm, p4d)) return NULL; - pud = pud_offset(p4d, addr); + pud = pud_offset_mm(mm, p4d, addr); if (pud_none_or_clear_bad(pud)) return NULL; @@ -102,7 +102,7 @@ static pmd_t *get_old_pmd(struct mm_struct *mm, unsigned long addr) if (!pud) return NULL; - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(mm, pud, addr); if (pmd_none(*pmd)) return NULL; @@ -183,6 +183,14 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr if (max_nr == 1) return 1; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) { + folio = vm_normal_folio(vma, addr, pte); + return folio ? folio_subpage_pte_batch(vma, folio, ptep, pte, + max_nr, FPB_RESPECT_WRITE) : 1; + } +#endif + /* Avoid expensive folio lookup if we stand no chance of benefit. */ if (pte_batch_hint(ptep, pte) == 1) return 1; @@ -262,12 +270,12 @@ static int move_ptes(struct pagetable_move_control *pmc, flush_tlb_batched_pending(vma->vm_mm); lazy_mmu_mode_enable(); - for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE, - new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) { + for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * mm_page_size(mm), + new_ptep += nr_ptes, new_addr += nr_ptes * mm_page_size(mm)) { VM_WARN_ON_ONCE(!pte_none(*new_ptep)); nr_ptes = 1; - max_nr_ptes = (old_end - old_addr) >> PAGE_SHIFT; + max_nr_ptes = (old_end - old_addr) >> mm_page_shift(mm); old_pte = ptep_get(old_ptep); if (pte_none(old_pte)) continue; @@ -407,7 +415,7 @@ static bool move_normal_pmd(struct pagetable_move_control *pmc, VM_BUG_ON(!pmd_none(*new_pmd)); pmd_populate(mm, new_pmd, pmd_pgtable(pmd)); - flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PMD_SIZE); + flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + pmd_size_mm(mm)); out_unlock: if (new_ptl != old_ptl) spin_unlock(new_ptl); @@ -458,8 +466,8 @@ static bool move_normal_pud(struct pagetable_move_control *pmc, VM_BUG_ON(!pud_none(*new_pud)); - pud_populate(mm, new_pud, pud_pgtable(pud)); - flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PUD_SIZE); + pud_populate(mm, new_pud, pud_pgtable_mm(mm, pud)); + flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + pud_size_mm(mm)); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); @@ -549,13 +557,13 @@ static __always_inline unsigned long get_extent(enum pgt_entry entry, switch (entry) { case HPAGE_PMD: case NORMAL_PMD: - mask = PMD_MASK; - size = PMD_SIZE; + mask = pmd_mask_mm(pmc->old->vm_mm); + size = pmd_size_mm(pmc->old->vm_mm); break; case HPAGE_PUD: case NORMAL_PUD: - mask = PUD_MASK; - size = PUD_SIZE; + mask = pud_mask_mm(pmc->old->vm_mm); + size = pud_size_mm(pmc->old->vm_mm); break; default: BUILD_BUG(); @@ -811,7 +819,7 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) * If possible, realign addresses to PMD boundary for faster copy. * Only realign if the mremap copying hits a PMD boundary. */ - try_realign_addr(pmc, PMD_MASK); + try_realign_addr(pmc, pmd_mask_mm(mm)); flush_cache_range(pmc->old, pmc->old_addr, pmc->old_end); mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, mm, @@ -838,7 +846,7 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) /* We ignore and continue on error? */ continue; } - } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) { + } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == pud_size_mm(mm)) { if (move_pgt_entry(pmc, NORMAL_PUD, old_pud, new_pud)) continue; } @@ -857,7 +865,7 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) continue; split_huge_pmd(pmc->old, old_pmd, pmc->old_addr); } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) && - extent == PMD_SIZE) { + extent == pmd_size_mm(mm)) { /* * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible. @@ -947,9 +955,7 @@ static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) { struct vm_area_struct *vma = vrm->vma; unsigned long map_flags = 0; - /* Page Offset _into_ the VMA. */ - pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >> PAGE_SHIFT; - pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; + pgoff_t pgoff = vma_page_offset_at(vma, vrm->addr).index; unsigned long new_addr = vrm_implies_new_addr(vrm) ? vrm->new_addr : 0; unsigned long res; @@ -985,13 +991,13 @@ static bool vrm_calc_charge(struct vma_remap_struct *vrm) * the length of the new one. Otherwise it's just the delta in size. */ if (vrm->flags & MREMAP_DONTUNMAP) - charged = vrm->new_len >> PAGE_SHIFT; + charged = vrm->new_len >> mm_page_shift(current->mm); else - charged = vrm->delta >> PAGE_SHIFT; + charged = vrm->delta >> mm_page_shift(current->mm); /* This accounts 'charged' pages of memory. */ - if (security_vm_enough_memory_mm(current->mm, charged)) + if (mm_account_memory(current->mm, current->mm, charged)) return false; vrm->charged = charged; @@ -1007,7 +1013,7 @@ static void vrm_uncharge(struct vma_remap_struct *vrm) if (!(vrm->vma->vm_flags & VM_ACCOUNT)) return; - vm_unacct_memory(vrm->charged); + mm_unacct_memory(current->mm, vrm->charged); vrm->charged = 0; } @@ -1019,13 +1025,13 @@ static void vrm_uncharge(struct vma_remap_struct *vrm) static void vrm_stat_account(struct vma_remap_struct *vrm, unsigned long bytes) { - unsigned long pages = bytes >> PAGE_SHIFT; + unsigned long pages = bytes >> mm_page_shift(current->mm); struct mm_struct *mm = current->mm; struct vm_area_struct *vma = vrm->vma; vm_stat_account(mm, vma->vm_flags, pages); if (vma->vm_flags & VM_LOCKED) - mm->locked_vm += pages; + mm->locked_vm += bytes >> MM_ACCOUNT_SHIFT; } static bool __check_map_count_against_split(struct mm_struct *mm, @@ -1198,7 +1204,7 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) vrm->vmi_needs_invalidate = true; if (err) { /* OOM: unable to split vma, just get accounts right */ - vm_acct_memory(len >> PAGE_SHIFT); + mm_acct_memory(current->mm, len >> mm_page_shift(current->mm)); return; } @@ -1255,16 +1261,14 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) static int copy_vma_and_data(struct vma_remap_struct *vrm, struct vm_area_struct **new_vma_ptr) { - unsigned long internal_offset = vrm->addr - vrm->vma->vm_start; - unsigned long internal_pgoff = internal_offset >> PAGE_SHIFT; - unsigned long new_pgoff = vrm->vma->vm_pgoff + internal_pgoff; + struct vm_page_offset pos = vma_page_offset_at(vrm->vma, vrm->addr); unsigned long moved_len; struct vm_area_struct *vma = vrm->vma; struct vm_area_struct *new_vma; int err = 0; PAGETABLE_MOVE(pmc, NULL, NULL, vrm->addr, vrm->new_addr, vrm->old_len); - new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff, + new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, pos, &pmc.need_rmap_locks); if (!new_vma) { vrm_uncharge(vrm); @@ -1473,7 +1477,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm) /* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ if (vrm->flags & MREMAP_DONTUNMAP) { vma_flags_t vma_flags = vrm->vma->flags; - unsigned long pages = vrm->old_len >> PAGE_SHIFT; + unsigned long pages = vrm->old_len >> mm_page_shift(current->mm); if (!may_expand_vm(mm, &vma_flags, pages)) return -ENOMEM; @@ -1495,7 +1499,7 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) if (find_vma_intersection(vma->vm_mm, vma->vm_end, end)) return 0; if (get_unmapped_area(NULL, vma->vm_start, end - vma->vm_start, - 0, MAP_FIXED) & ~PAGE_MASK) + 0, MAP_FIXED) & ~mm_page_mask(vma->vm_mm)) return 0; return 1; } @@ -1727,7 +1731,8 @@ static int check_prep_vma(struct vma_remap_struct *vrm) struct vm_area_struct *vma = vrm->vma; struct mm_struct *mm = current->mm; unsigned long addr = vrm->addr; - unsigned long old_len, new_len, pgoff; + unsigned long old_len, new_len; + struct vm_page_offset pos, end; if (!vma) return -EFAULT; @@ -1802,9 +1807,9 @@ static int check_prep_vma(struct vma_remap_struct *vrm) vrm->populate_expand = true; /* Need to be careful about a growing mapping */ - pgoff = (addr - vma->vm_start) >> PAGE_SHIFT; - pgoff += vma->vm_pgoff; - if (pgoff + (new_len >> PAGE_SHIFT) < pgoff) + pos = vma_page_offset_at(vma, addr); + end = vm_page_offset_add(pos, new_len); + if (end.index < pos.index) return -EINVAL; if (vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP)) @@ -1813,7 +1818,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) if (!mlock_future_ok(mm, vma->vm_flags & VM_LOCKED, vrm->delta)) return -EAGAIN; - if (!may_expand_vm(mm, &vma->flags, vrm->delta >> PAGE_SHIFT)) + if (!may_expand_vm(mm, &vma->flags, vrm->delta >> mm_page_shift(current->mm))) return -ENOMEM; return 0; @@ -1834,7 +1839,7 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm) return -EINVAL; /* Start address must be page-aligned. */ - if (offset_in_page(addr)) + if (addr & ~mm_page_mask(current->mm)) return -EINVAL; /* @@ -1858,7 +1863,7 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm) return -EINVAL; /* The new address must be page-aligned. */ - if (offset_in_page(vrm->new_addr)) + if (vrm->new_addr & ~mm_page_mask(current->mm)) return -EINVAL; /* A fixed address implies a move. */ @@ -1971,8 +1976,8 @@ static unsigned long do_mremap(struct vma_remap_struct *vrm) unsigned long res; bool failed; - vrm->old_len = PAGE_ALIGN(vrm->old_len); - vrm->new_len = PAGE_ALIGN(vrm->new_len); + vrm->old_len = mm_page_align(mm, vrm->old_len); + vrm->new_len = mm_page_align(mm, vrm->new_len); res = check_mremap_params(vrm); if (res) diff --git a/mm/mseal.c b/mm/mseal.c index e2093ae3d25cdc..62df360b43545b 100644 --- a/mm/mseal.c +++ b/mm/mseal.c @@ -151,10 +151,10 @@ int do_mseal(unsigned long start, size_t len_in, unsigned long flags) return -EINVAL; start = untagged_addr(start); - if (!PAGE_ALIGNED(start)) + if (start & ~mm_page_mask(mm)) return -EINVAL; - len = PAGE_ALIGN(len_in); + len = mm_page_align(mm, len_in); /* Check to see whether len was rounded up from small -ve to zero. */ if (len_in && !len) return -EINVAL; diff --git a/mm/msync.c b/mm/msync.c index ac4c9bfea2e7fa..6d9027ee47246a 100644 --- a/mm/msync.c +++ b/mm/msync.c @@ -41,12 +41,12 @@ SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags) if (flags & ~(MS_ASYNC | MS_INVALIDATE | MS_SYNC)) goto out; - if (offset_in_page(start)) + if (start & ~mm_page_mask(mm)) goto out; if ((flags & MS_ASYNC) && (flags & MS_SYNC)) goto out; error = -ENOMEM; - len = (len + ~PAGE_MASK) & PAGE_MASK; + len = mm_page_align(mm, len); end = start + len; if (end < start) goto out; @@ -86,7 +86,7 @@ SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags) } file = vma->vm_file; fstart = (start - vma->vm_start) + - ((loff_t)vma->vm_pgoff << PAGE_SHIFT); + ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma); fend = fstart + (min(end, vma->vm_end) - start) - 1; start = vma->vm_end; if ((flags & MS_SYNC) && file && diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 5f372f6e26fa32..5345f43c10b6ff 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -225,7 +225,8 @@ long oom_badness(struct task_struct *p, unsigned long totalpages) * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ - points = get_mm_rss_sum(p->mm) + get_mm_counter_sum(p->mm, MM_SWAPENTS) + + points = mm_pages_to_native(p->mm, get_mm_rss_sum(p->mm) + + get_mm_counter_sum(p->mm, MM_SWAPENTS)) + mm_pgtables_bytes(p->mm) / PAGE_SIZE; task_unlock(p); @@ -399,10 +400,10 @@ static int dump_task(struct task_struct *p, void *arg) pr_info("[%7d] %5d %5d %8lu %8lu %8lu %8lu %9lu %8ld %8lu %5hd %s\n", task->pid, from_kuid(&init_user_ns, task_uid(task)), - task->tgid, task->mm->total_vm, get_mm_rss_sum(task->mm), - get_mm_counter_sum(task->mm, MM_ANONPAGES), get_mm_counter_sum(task->mm, MM_FILEPAGES), - get_mm_counter_sum(task->mm, MM_SHMEMPAGES), mm_pgtables_bytes(task->mm), - get_mm_counter_sum(task->mm, MM_SWAPENTS), + task->tgid, mm_pages_to_native(task->mm, task->mm->total_vm), mm_pages_to_native(task->mm, get_mm_rss_sum(task->mm)), + mm_pages_to_native(task->mm, get_mm_counter_sum(task->mm, MM_ANONPAGES)), mm_pages_to_native(task->mm, get_mm_counter_sum(task->mm, MM_FILEPAGES)), + mm_pages_to_native(task->mm, get_mm_counter_sum(task->mm, MM_SHMEMPAGES)), mm_pgtables_bytes(task->mm), + mm_pages_to_native(task->mm, get_mm_counter_sum(task->mm, MM_SWAPENTS)), task->signal->oom_score_adj, task->comm); task_unlock(task); @@ -588,9 +589,9 @@ static bool oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm) pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n", task_pid_nr(tsk), tsk->comm, - K(get_mm_counter_sum(mm, MM_ANONPAGES)), - K(get_mm_counter_sum(mm, MM_FILEPAGES)), - K(get_mm_counter_sum(mm, MM_SHMEMPAGES))); + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_ANONPAGES)), + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_FILEPAGES)), + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_SHMEMPAGES))); out_finish: trace_finish_task_reaping(tsk->pid); out_unlock: @@ -943,10 +944,10 @@ static void __oom_kill_process(struct task_struct *victim, const char *message) do_send_sig_info(SIGKILL, SEND_SIG_PRIV, victim, PIDTYPE_TGID); mark_oom_victim(victim); pr_err("%s: Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB, UID:%u pgtables:%lukB oom_score_adj:%d\n", - message, task_pid_nr(victim), victim->comm, K(mm->total_vm), - K(get_mm_counter_sum(mm, MM_ANONPAGES)), - K(get_mm_counter_sum(mm, MM_FILEPAGES)), - K(get_mm_counter_sum(mm, MM_SHMEMPAGES)), + message, task_pid_nr(victim), victim->comm, mm_pages_to_kb(mm, mm->total_vm), + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_ANONPAGES)), + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_FILEPAGES)), + mm_pages_to_kb(mm, get_mm_counter_sum(mm, MM_SHMEMPAGES)), from_kuid(&init_user_ns, task_uid(victim)), mm_pgtables_bytes(mm) >> 10, victim->signal->oom_score_adj); task_unlock(victim); diff --git a/mm/page_ext.c b/mm/page_ext.c index e2e92bd27ebdd0..a73aab719c0406 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include #include @@ -78,6 +79,9 @@ static struct page_ext_operations page_idle_ops __initdata = { #endif static struct page_ext_operations *page_ext_ops[] __initdata = { +#ifdef CONFIG_MM_SUBPAGE + &mm_subpage_ext_ops, +#endif #ifdef CONFIG_PAGE_OWNER &page_owner_ops, #endif diff --git a/mm/page_idle.c b/mm/page_idle.c index 9c67cbac296598..8af46b0d8f7460 100644 --- a/mm/page_idle.c +++ b/mm/page_idle.c @@ -51,9 +51,10 @@ static struct folio *page_idle_get_folio(unsigned long pfn) static bool page_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, - unsigned long addr, void *arg) + struct rmap_walk_range rmap, void *arg) { - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0); + unsigned long addr = rmap.address; + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); bool referenced = false; while (page_vma_mapped_walk(&pvmw)) { @@ -69,7 +70,7 @@ static bool page_idle_clear_pte_refs_one(struct folio *folio, */ if (likely(pte_present(ptep_get(pvmw.pte)))) referenced |= ptep_test_and_clear_young(vma, addr, pvmw.pte); - referenced |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + referenced |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + mm_page_size(vma->vm_mm)); } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { pmd_t pmdval = pmdp_get(pvmw.pmd); diff --git a/mm/page_io.c b/mm/page_io.c index 70cea9e24d2fd3..290706f931bdcd 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -25,6 +25,7 @@ #include #include #include +#include #include "swap.h" static void __end_swap_bio_write(struct bio *bio) @@ -411,6 +412,58 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) *swap_plug = sio; } +static DEFINE_MUTEX(swap_extent_io_mutex); +static struct bio_set swap_extent_bioset; + +int swap_extent_io_init(void) +{ + int ret = 0; + + /* Swapon prepares the reserve before any reclaim can use this file. */ + mutex_lock(&swap_extent_io_mutex); + if (!bioset_initialized(&swap_extent_bioset)) + ret = bioset_init(&swap_extent_bioset, SWAP_CLUSTER_MAX, 0, + BIOSET_NEED_RESCUER); + mutex_unlock(&swap_extent_io_mutex); + return ret; +} + +/* + * A mandatory large base page cannot split to fit a swap-file extent. Chain + * each physical run to the original bio: its folio completion must wait for + * every run, including on partial I/O or allocation failure. + */ +static void swap_submit_bio(struct bio *bio, void *private) +{ + struct folio *folio = private; + struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); + swp_entry_t entry = folio->swap; + + if (!(READ_ONCE(sis->flags) & SWP_BLKDEV) && folio_test_large(folio)) { + while (true) { + unsigned long nr = bio->bi_iter.bi_size >> PAGE_SHIFT; + unsigned int bytes; + struct bio *split; + + bio->bi_iter.bi_sector = swap_extent_sector(entry, &nr); + bytes = nr << PAGE_SHIFT; + if (bytes == bio->bi_iter.bi_size) + break; + /* The unsubmitted parent may own a bio from fs_bio_set. */ + split = bio_split(bio, bytes >> 9, GFP_NOIO, &swap_extent_bioset); + if (IS_ERR(split)) { + bio->bi_status = BLK_STS_RESOURCE; + bio_endio(bio); + return; + } + bio_chain(split, bio); + submit_bio(split); + entry.val += nr; + } + } + submit_bio(bio); +} + static void swap_writepage_bdev_sync(struct folio *folio, struct swap_info_struct *sis) { @@ -427,7 +480,7 @@ static void swap_writepage_bdev_sync(struct folio *folio, folio_start_writeback(folio); folio_unlock(folio); - submit_bio_wait(&bio); + bio_await(&bio, folio, swap_submit_bio); __end_swap_bio_write(&bio); } @@ -445,7 +498,7 @@ static void swap_writepage_bdev_async(struct folio *folio, count_swpout_vm_event(folio); folio_start_writeback(folio); folio_unlock(folio); - submit_bio(bio); + swap_submit_bio(bio, folio); } void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug) @@ -590,7 +643,7 @@ static void swap_read_folio_bdev_sync(struct folio *folio, count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); count_vm_events(PSWPIN, folio_nr_pages(folio)); - submit_bio_wait(&bio); + bio_await(&bio, folio, swap_submit_bio); __end_swap_bio_read(&bio); put_task_struct(current); } @@ -607,7 +660,7 @@ static void swap_read_folio_bdev_async(struct folio *folio, count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); count_vm_events(PSWPIN, folio_nr_pages(folio)); - submit_bio(bio); + swap_submit_bio(bio, folio); } void swap_read_folio(struct folio *folio, struct swap_iocb **plug) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 2403f5a1141058..dc6797a41daaa2 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -4,6 +4,7 @@ * Copyright (c) 2021, Google LLC. * Pasha Tatashin */ +#include #include #include #include @@ -13,11 +14,21 @@ #undef pr_fmt #define pr_fmt(fmt) "page_table_check: " fmt -struct page_table_check { +#ifdef CONFIG_MM_SUBPAGE +#define PAGE_TABLE_CHECK_SLOTS (PAGE_SIZE / SZ_4K) +#else +#define PAGE_TABLE_CHECK_SLOTS 1 +#endif + +struct page_table_check_slot { atomic_t anon_map_count; atomic_t file_map_count; }; +struct page_table_check { + struct page_table_check_slot slots[PAGE_TABLE_CHECK_SLOTS]; +}; + static bool __page_table_check_enabled __initdata = IS_ENABLED(CONFIG_PAGE_TABLE_CHECK_ENFORCED); @@ -56,6 +67,74 @@ static struct page_table_check *get_page_table_check(struct page_ext *page_ext) return page_ext_data(page_ext, &page_table_check_ops); } +#if IS_ENABLED(CONFIG_KUNIT) +/* Diagnostic snapshot for private test allocations, not a synchronization API. */ +int page_table_check_get_counts(struct page *page, unsigned int offset, + int *anon, int *file) +{ + struct page_ext *ext; + struct page_table_check *ptc; + unsigned int slot = 0; + + if (static_branch_likely(&page_table_check_disabled)) + return -EOPNOTSUPP; + if (offset >= PAGE_SIZE) + return -EINVAL; +#ifdef CONFIG_MM_SUBPAGE + slot = offset / SZ_4K; +#endif + ext = page_ext_get(page); + if (!ext) + return -ENOENT; + ptc = get_page_table_check(ext); + *anon = atomic_read(&ptc->slots[slot].anon_map_count); + *file = atomic_read(&ptc->slots[slot].file_map_count); + page_ext_put(ext); + return 0; +} +EXPORT_SYMBOL_IF_KUNIT(page_table_check_get_counts); +#endif + +static void page_table_check_update(struct page_table_check_slot *slot, + bool anon, bool clear, bool rw) +{ + if (anon) { + BUG_ON(atomic_read(&slot->file_map_count)); + if (clear) + BUG_ON(atomic_dec_return(&slot->anon_map_count) < 0); + else + BUG_ON(atomic_inc_return(&slot->anon_map_count) > 1 && rw); + } else { + BUG_ON(atomic_read(&slot->anon_map_count)); + if (clear) + BUG_ON(atomic_dec_return(&slot->file_map_count) < 0); + else + BUG_ON(atomic_inc_return(&slot->file_map_count) < 0); + } +} + +#ifdef CONFIG_MM_SUBPAGE +static void page_table_check_subpage(phys_addr_t phys, unsigned long size, + bool clear, bool rw) +{ + unsigned long pfn = PHYS_PFN(phys); + struct page_table_check *ptc; + struct page_ext *ext; + struct page *page; + unsigned int first = offset_in_page(phys) / SZ_4K, i; + + if (!pfn_valid(pfn)) + return; + page = pfn_to_page(pfn); + BUG_ON(PageSlab(page)); + ext = page_ext_get(page); + ptc = get_page_table_check(ext); + for (i = first; i < first + size / SZ_4K; i++) + page_table_check_update(&ptc->slots[i], PageAnon(page), clear, rw); + page_ext_put(ext); +} +#endif + /* * An entry is removed from the page table, decrement the counters for that page * verify that it is of correct type and counters do not become negative. @@ -78,13 +157,8 @@ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); - if (anon) { - BUG_ON(atomic_read(&ptc->file_map_count)); - BUG_ON(atomic_dec_return(&ptc->anon_map_count) < 0); - } else { - BUG_ON(atomic_read(&ptc->anon_map_count)); - BUG_ON(atomic_dec_return(&ptc->file_map_count) < 0); - } + for (unsigned int i = 0; i < PAGE_TABLE_CHECK_SLOTS; i++) + page_table_check_update(&ptc->slots[i], anon, true, false); } rcu_read_unlock(); } @@ -113,13 +187,8 @@ static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); - if (anon) { - BUG_ON(atomic_read(&ptc->file_map_count)); - BUG_ON(atomic_inc_return(&ptc->anon_map_count) > 1 && rw); - } else { - BUG_ON(atomic_read(&ptc->anon_map_count)); - BUG_ON(atomic_inc_return(&ptc->file_map_count) < 0); - } + for (unsigned int i = 0; i < PAGE_TABLE_CHECK_SLOTS; i++) + page_table_check_update(&ptc->slots[i], anon, false, rw); } rcu_read_unlock(); } @@ -139,8 +208,10 @@ void __page_table_check_zero(struct page *page, unsigned int order) for_each_page_ext(page, 1 << order, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); - BUG_ON(atomic_read(&ptc->anon_map_count)); - BUG_ON(atomic_read(&ptc->file_map_count)); + for (unsigned int i = 0; i < PAGE_TABLE_CHECK_SLOTS; i++) { + BUG_ON(atomic_read(&ptc->slots[i].anon_map_count)); + BUG_ON(atomic_read(&ptc->slots[i].file_map_count)); + } } rcu_read_unlock(); } @@ -151,8 +222,15 @@ void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, if (&init_mm == mm) return; - if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) - page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT); + if (!pte_user_accessible_page(mm, addr, pte) || pte_special(pte)) + return; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + page_table_check_subpage(pte_phys_mm(mm, pte), mm_page_size(mm), true, false); + return; + } +#endif + page_table_check_clear(pte_pfn(pte), mm_pte_native_pages(mm)); } EXPORT_SYMBOL(__page_table_check_pte_clear); @@ -218,9 +296,20 @@ void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, page_table_check_pte_flags(pte); for (i = 0; i < nr; i++) - __page_table_check_pte_clear(mm, addr + PAGE_SIZE * i, ptep_get(ptep + i)); - if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) - page_table_check_set(pte_pfn(pte), nr, pte_write(pte)); + __page_table_check_pte_clear(mm, addr + mm_page_size(mm) * i, ptep_get(ptep + i)); + if (!pte_user_accessible_page(mm, addr, pte) || pte_special(pte)) + return; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + phys_addr_t phys = pte_phys_mm(mm, pte); + + for (i = 0; i < nr; i++) + page_table_check_subpage(phys + i * mm_page_size(mm), mm_page_size(mm), false, + pte_write(pte)); + return; + } +#endif + page_table_check_set(pte_pfn(pte), nr * mm_pte_native_pages(mm), pte_write(pte)); } EXPORT_SYMBOL(__page_table_check_ptes_set); @@ -279,16 +368,17 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm, return; if (!pmd_bad(pmd) && !pmd_leaf(pmd)) { - pte_t *ptep = pte_offset_map(&pmd, addr); + pte_t *ptep = pte_offset_map_mm(mm, &pmd, addr); + unsigned long nr = pte_table_bytes_mm(mm) / sizeof(pte_t); unsigned long i; if (WARN_ON(!ptep)) return; - for (i = 0; i < PTRS_PER_PTE; i++) { + for (i = 0; i < nr; i++) { __page_table_check_pte_clear(mm, addr, ptep_get(ptep)); - addr += PAGE_SIZE; + addr += mm_page_size(mm); ptep++; } - pte_unmap(ptep - PTRS_PER_PTE); + pte_unmap(ptep - nr); } } diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ccd1c622c8b24f..9377be895563a8 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -87,7 +87,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, * mapped at the @pvmw->pte * @pvmw: page_vma_mapped_walk struct, includes a pair pte and pfn range * for checking - * @pte_nr: the number of small pages described by @pvmw->pte. + * @pte_nr: the number of native physical pages covered by @pvmw->pte. * * page_vma_mapped_walk() found a place where pfn range is *potentially* * mapped. check_pte() has to validate this. @@ -109,6 +109,18 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) unsigned long pfn; pte_t ptent = ptep_get(pvmw->pte); +#ifdef CONFIG_MM_SUBPAGE + if (pvmw->flags & PVMW_SUBPAGE) { + if (!(pvmw->flags & PVMW_MIGRATION)) + return pte_present(ptent) && + pte_phys_mm(pvmw->vma->vm_mm, ptent) == + PFN_PHYS(pvmw->pfn) + pvmw->subpage_offset; + if (pte_swp_subpage_offset(ptent) != pvmw->subpage_offset) + return false; + /* The common migration check below validates the native PFN. */ + } +#endif + if (pvmw->flags & PVMW_MIGRATION) { const softleaf_t entry = softleaf_from_pte(ptent); @@ -188,6 +200,18 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pud_t *pud; pmd_t pmde; +#ifdef CONFIG_MM_SUBPAGE + if (pvmw->flags & PVMW_SUBPAGE) { + if (WARN_ON_ONCE(pvmw->nr_pages != 1 || + mm_page_size(mm) >= PAGE_SIZE || + !IS_ALIGNED(pvmw->address, mm_page_size(mm)) || + pvmw->subpage_offset >= PAGE_SIZE || + !IS_ALIGNED(pvmw->subpage_offset, mm_page_size(mm)) || + is_vm_hugetlb_page(vma))) + return not_found(pvmw); + } +#endif + /* The only possible pmd mapping has been handled on last iteration */ if (pvmw->pmd && !pvmw->pte) return not_found(pvmw); @@ -213,6 +237,9 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return true; } + /* A query for an interior native page still visits one whole user leaf. */ + if (mm_page_size(mm) > PAGE_SIZE) + pvmw->address &= mm_page_mask(mm); end = vma_address_end(pvmw); if (pvmw->pte) goto next_pte; @@ -220,21 +247,21 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) do { pgd = pgd_offset(mm, pvmw->address); if (!pgd_present(*pgd)) { - step_forward(pvmw, PGDIR_SIZE); + step_forward(pvmw, pgd_size_mm(mm)); continue; } - p4d = p4d_offset(pgd, pvmw->address); - if (!p4d_present(*p4d)) { - step_forward(pvmw, P4D_SIZE); + p4d = p4d_offset_mm(mm, pgd, pvmw->address); + if (!p4d_present_mm(mm, *p4d)) { + step_forward(pvmw, p4d_size_mm(mm)); continue; } - pud = pud_offset(p4d, pvmw->address); + pud = pud_offset_mm(mm, p4d, pvmw->address); if (!pud_present(*pud)) { - step_forward(pvmw, PUD_SIZE); + step_forward(pvmw, pud_size_mm(mm)); continue; } - pvmw->pmd = pmd_offset(pud, pvmw->address); + pvmw->pmd = pmd_offset_mm(mm, pud, pvmw->address); /* * Make sure the pmd value isn't cached in a register by the * compiler and used as a stale value after we've observed a @@ -245,6 +272,9 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (pmd_trans_huge(pmde) || pmd_is_migration_entry(pmde) || pmd_is_device_private_entry(pmde))) { + /* Subpage PMD mappings need a distinct huge-page implementation. */ + if (mm_page_size(mm) != PAGE_SIZE) + return not_found(pvmw); pvmw->ptl = pmd_lock(mm, pvmw->pmd); pmde = *pvmw->pmd; if (pmd_is_migration_entry(pmde)) { @@ -285,7 +315,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) (pvmw->nr_pages >= HPAGE_PMD_NR)) sync_with_folio_pmd_zap(mm, pvmw->pmd); - step_forward(pvmw, PMD_SIZE); + step_forward(pvmw, pmd_size_mm(mm)); continue; } if (!map_pte(pvmw, &pmde, &ptl)) { @@ -294,15 +324,15 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) goto next_pte; } this_pte: - if (check_pte(pvmw, 1)) + if (check_pte(pvmw, mm_pte_native_pages(mm))) return true; next_pte: do { - pvmw->address += PAGE_SIZE; + pvmw->address += mm_page_size(mm); if (pvmw->address >= end) return not_found(pvmw); /* Did we cross page table boundary? */ - if ((pvmw->address & (PMD_SIZE - PAGE_SIZE)) == 0) { + if ((pvmw->address & (pmd_size_mm(mm) - mm_page_size(mm))) == 0) { if (pvmw->ptl) { spin_unlock(pvmw->ptl); pvmw->ptl = NULL; @@ -345,20 +375,27 @@ unsigned long page_mapped_in_vma(const struct page *page, struct vm_area_struct *vma) { const struct folio *folio = page_folio(page); + unsigned long address; struct page_vma_mapped_walk pvmw = { .pfn = page_to_pfn(page), + .pgoff = page_pgoff(folio, page), .nr_pages = 1, .vma = vma, .flags = PVMW_SYNC, }; - pvmw.address = vma_address(vma, page_pgoff(folio, page), 1); + address = vma_address(vma, pvmw.pgoff, 1); + pvmw.address = address; if (pvmw.address == -EFAULT) goto out; if (!page_vma_mapped_walk(&pvmw)) return -EFAULT; + /* A smaller leaf can map the page after an unmapped leading fragment. */ + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + address = pvmw.address; page_vma_mapped_walk_done(&pvmw); out: - return pvmw.address; + /* Coarser leaves retain this native page's byte address within the leaf. */ + return address; } #endif diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 5d87c632a25507..54bc7ac30befa5 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -30,13 +31,14 @@ static int walk_pte_range_inner(pte_t *pte, unsigned long addr, unsigned long end, struct mm_walk *walk) { const struct mm_walk_ops *ops = walk->ops; + const unsigned long page_size = mm_page_size(walk->mm); int err = 0; for (;;) { if (ops->install_pte && pte_none(ptep_get(pte))) { pte_t new_pte; - err = ops->install_pte(addr, addr + PAGE_SIZE, &new_pte, + err = ops->install_pte(addr, addr + page_size, &new_pte, walk); if (err) break; @@ -46,13 +48,13 @@ static int walk_pte_range_inner(pte_t *pte, unsigned long addr, if (!WARN_ON_ONCE(walk->no_vma)) update_mmu_cache(walk->vma, addr, pte); } else { - err = ops->pte_entry(pte, addr, addr + PAGE_SIZE, walk); + err = ops->pte_entry(pte, addr, addr + page_size, walk); if (err) break; } - if (addr >= end - PAGE_SIZE) + if (addr >= end - page_size) break; - addr += PAGE_SIZE; + addr += page_size; pte++; } return err; @@ -67,15 +69,15 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, if (walk->no_vma) { /* - * pte_offset_map() might apply user-specific validation. + * pte_offset_map_mm() might apply user-specific validation. * Indeed, on x86_64 the pmd entries set up by init_espfix_ap() * fit its pmd_bad() check (_PAGE_NX set and _PAGE_RW clear), * and CONFIG_EFI_PGT_DUMP efi_mm goes so far as to walk them. */ if (walk->mm == &init_mm || addr >= TASK_SIZE) - pte = pte_offset_kernel(pmd, addr); + pte = pte_offset_kernel_mm(walk->mm, pmd, addr); else - pte = pte_offset_map(pmd, addr); + pte = pte_offset_map_mm(walk->mm, pmd, addr); if (pte) { err = walk_pte_range_inner(pte, addr, end, walk); if (walk->mm != &init_mm && addr < TASK_SIZE) @@ -123,10 +125,10 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, return 0; } - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(walk->mm, pud, addr); do { again: - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(walk->mm, addr, end); if (pmd_none(*pmd)) { if (has_install) err = __pte_alloc(walk->mm, pmd); @@ -193,10 +195,10 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, int err = 0; int depth = real_depth(2); - pud = pud_offset(p4d, addr); + pud = pud_offset_mm(walk->mm, p4d, addr); do { again: - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(walk->mm, addr, end); if (pud_none(*pud)) { if (has_install) err = __pmd_alloc(walk->mm, pud, addr); @@ -258,10 +260,10 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, int err = 0; int depth = real_depth(1); - p4d = p4d_offset(pgd, addr); + p4d = p4d_offset_mm(walk->mm, pgd, addr); do { - next = p4d_addr_end(addr, end); - if (p4d_none_or_clear_bad(p4d)) { + next = p4d_addr_end_mm(walk->mm, addr, end); + if (p4d_none_or_clear_bad_mm(walk->mm, p4d)) { if (has_install) err = __pud_alloc(walk->mm, p4d, addr); else if (ops->pte_hole) @@ -297,11 +299,11 @@ static int walk_pgd_range(unsigned long addr, unsigned long end, int err = 0; if (walk->pgd) - pgd = walk->pgd + pgd_index(addr); + pgd = pgd_offset_pgd_mm(walk->mm, walk->pgd, addr); else pgd = pgd_offset(walk->mm, addr); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(walk->mm, addr, end); if (pgd_none_or_clear_bad(pgd)) { if (has_install) err = __p4d_alloc(walk->mm, pgd, addr); @@ -808,27 +810,26 @@ int walk_page_mapping(struct address_space *mapping, pgoff_t first_index, .private = private, }; struct vm_area_struct *vma; - pgoff_t vba, vea, cba, cea; + pgoff_t last_index; unsigned long start_addr, end_addr; int err = 0; if (!check_ops_safe(ops)) return -EINVAL; + if (!nr) + return 0; + if (check_add_overflow(first_index, nr - 1, &last_index)) + return -EINVAL; lockdep_assert_held(&mapping->i_mmap_rwsem); vma_interval_tree_foreach(vma, &mapping->i_mmap, first_index, - first_index + nr - 1) { - /* Clip to the vma */ - vba = vma->vm_pgoff; - vea = vba + vma_pages(vma); - cba = first_index; - cba = max(cba, vba); - cea = first_index + nr; - cea = min(cea, vea); - - start_addr = ((cba - vba) << PAGE_SHIFT) + vma->vm_start; - end_addr = ((cea - vba) << PAGE_SHIFT) + vma->vm_start; - if (start_addr >= end_addr) + last_index) { + /* Clip native file indices to the VMA's exact byte interval. */ + start_addr = first_index <= vma->vm_pgoff ? vma->vm_start : + vma_address_at_offset(vma, (struct vm_page_offset) { first_index, 0 }); + end_addr = last_index >= vma_last_pgoff(vma) ? vma->vm_end : + vma_address_at_offset(vma, (struct vm_page_offset) { last_index + 1, 0 }); + if (IS_ERR_VALUE(start_addr) || IS_ERR_VALUE(end_addr) || start_addr >= end_addr) continue; walk.vma = vma; @@ -924,11 +925,11 @@ struct folio *folio_walk_start(struct folio_walk *fw, if (pgd_none_or_clear_bad(pgdp)) goto not_found; - p4dp = p4d_offset(pgdp, addr); - if (p4d_none_or_clear_bad(p4dp)) + p4dp = p4d_offset_mm(vma->vm_mm, pgdp, addr); + if (p4d_none_or_clear_bad_mm(vma->vm_mm, p4dp)) goto not_found; - pudp = pud_offset(p4dp, addr); + pudp = pud_offset_mm(vma->vm_mm, p4dp, addr); pud = pudp_get(pudp); if (pud_none(pud)) goto not_found; @@ -937,7 +938,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, ptl = pud_lock(vma->vm_mm, pudp); pud = pudp_get(pudp); - entry_size = PUD_SIZE; + entry_size = pud_size_mm(vma->vm_mm); fw->level = FW_LEVEL_PUD; fw->pudp = pudp; fw->pud = pud; @@ -959,7 +960,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, pmd_table: VM_WARN_ON_ONCE(!pud_present(pud) || pud_leaf(pud)); - pmdp = pmd_offset(pudp, addr); + pmdp = pmd_offset_mm(vma->vm_mm, pudp, addr); pmd = pmdp_get_lockless(pmdp); if (pmd_none(pmd)) goto not_found; @@ -968,7 +969,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, ptl = pmd_lock(vma->vm_mm, pmdp); pmd = pmdp_get(pmdp); - entry_size = PMD_SIZE; + entry_size = pmd_size_mm(vma->vm_mm); fw->level = FW_LEVEL_PMD; fw->pmdp = pmdp; fw->pmd = pmd; @@ -1001,7 +1002,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, goto not_found; pte = ptep_get(ptep); - entry_size = PAGE_SIZE; + entry_size = mm_page_size(vma->vm_mm); fw->level = FW_LEVEL_PTE; fw->ptep = ptep; fw->pte = pte; @@ -1028,5 +1029,5 @@ struct folio *folio_walk_start(struct folio_walk *fw, else fw->page = NULL; fw->ptl = ptl; - return page_folio(page); + return page_folio(zeropage ? page : fw->page); } diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index b91b1a98029c7f..7a9eef9b4f5b7e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -168,13 +168,15 @@ pud_t pudp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address, void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable) { + struct page *page = pgtable_to_page(pgtable); + assert_spin_locked(pmd_lockptr(mm, pmdp)); /* FIFO */ if (!pmd_huge_pte(mm, pmdp)) - INIT_LIST_HEAD(&pgtable->lru); + INIT_LIST_HEAD(&page->lru); else - list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->lru); + list_add(&page->lru, &pgtable_to_page(pmd_huge_pte(mm, pmdp))->lru); pmd_huge_pte(mm, pmdp) = pgtable; } #endif @@ -184,15 +186,17 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) { pgtable_t pgtable; + struct page *page, *next; assert_spin_locked(pmd_lockptr(mm, pmdp)); /* FIFO */ pgtable = pmd_huge_pte(mm, pmdp); - pmd_huge_pte(mm, pmdp) = list_first_entry_or_null(&pgtable->lru, - struct page, lru); + page = pgtable_to_page(pgtable); + next = list_first_entry_or_null(&page->lru, struct page, lru); + pmd_huge_pte(mm, pmdp) = pgtable_from_page(next); if (pmd_huge_pte(mm, pmdp)) - list_del(&pgtable->lru); + list_del(&page->lru); return pgtable; } #endif @@ -282,7 +286,12 @@ static unsigned long pmdp_get_lockless_start(void) { return 0; } static void pmdp_get_lockless_end(unsigned long irqflags) { } #endif +#ifdef CONFIG_ARCH_HAS_MM_PGTABLE +pte_t *__pte_offset_map_mm(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp) +#else pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) +#endif { unsigned long irqflags; pmd_t pmdval; @@ -302,19 +311,27 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) pmd_clear_bad(pmd); goto nomap; } - return __pte_map(&pmdval, addr); + return __pte_map_mm(mm, &pmdval, addr); nomap: rcu_read_unlock(); return NULL; } +#ifdef CONFIG_ARCH_HAS_MM_PGTABLE +pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) +{ + /* NULL retains native geometry for callers not yet converted. */ + return __pte_offset_map_mm(NULL, pmd, addr, pmdvalp); +} +#endif + pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { pmd_t pmdval; pte_t *pte; - pte = __pte_offset_map(pmd, addr, &pmdval); + pte = __pte_offset_map_mm(mm, pmd, addr, &pmdval); if (likely(pte)) *ptlp = pte_lockptr(mm, &pmdval); return pte; @@ -327,7 +344,7 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte; VM_WARN_ON_ONCE(!pmdvalp); - pte = __pte_offset_map(pmd, addr, pmdvalp); + pte = __pte_offset_map_mm(mm, pmd, addr, pmdvalp); if (likely(pte)) *ptlp = pte_lockptr(mm, pmdvalp); return pte; @@ -397,7 +414,7 @@ pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pmd_t pmdval; pte_t *pte; again: - pte = __pte_offset_map(pmd, addr, &pmdval); + pte = __pte_offset_map_mm(mm, pmd, addr, &pmdval); if (unlikely(!pte)) return pte; ptl = pte_lockptr(mm, &pmdval); diff --git a/mm/process_vm_access.c b/mm/process_vm_access.c index 656d3e88755b6f..f178b5fdb65890 100644 --- a/mm/process_vm_access.c +++ b/mm/process_vm_access.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -15,6 +16,8 @@ #include #include +#include "internal.h" + /** * process_vm_rw_pages - read/write pages from task specified * @pages: array of pointers to pages we want to copy @@ -135,6 +138,63 @@ static int process_vm_rw_single_vec(unsigned long addr, which lives on stack */ #define PVM_MAX_PP_ARRAY_COUNT 16 +#ifdef CONFIG_MM_SUBPAGE +/* Same pin/copy/dirty lifetime as the page-array path, with explicit offsets. */ +int process_vm_rw_fragments(struct mm_struct *mm, unsigned long addr, + unsigned long len, struct iov_iter *iter, bool write) +{ + struct user_page_fragment stack[PVM_MAX_PP_ARRAY_COUNT]; + struct user_page_fragment *fragments = stack; + unsigned long capacity, nr; + int ret = 0; + + if (!len || !iov_iter_count(iter)) + return 0; + capacity = 1 + ((len - 1) >> MM_SUBPAGE_SHIFT) + + ((((len - 1) & (MM_SUBPAGE_SIZE - 1)) + + (addr & (MM_SUBPAGE_SIZE - 1))) >> MM_SUBPAGE_SHIFT); + capacity = min_t(unsigned long, capacity, + PVM_MAX_KMALLOC_PAGES * PAGE_SIZE / sizeof(*fragments)); + if (capacity > ARRAY_SIZE(stack)) { + fragments = kmalloc_array(capacity, sizeof(*fragments), GFP_KERNEL); + if (!fragments) + return -ENOMEM; + } + while (len && iov_iter_count(iter)) { + long got = pin_user_fragments_remote(mm, addr, len, + write ? FOLL_WRITE : 0, fragments, capacity); + + if (got <= 0) { + ret = -EFAULT; + break; + } + for (nr = 0; nr < got && iov_iter_count(iter); nr++) { + struct user_page_fragment *f = &fragments[nr]; + struct page *page = folio_page(f->folio, f->offset >> PAGE_SHIFT); + unsigned int offset = offset_in_page(f->offset); + size_t copied; + + if (write) + copied = copy_page_from_iter(page, offset, f->length, iter); + else + copied = copy_page_to_iter(page, offset, f->length, iter); + addr += copied; + len -= copied; + if (copied < f->length && iov_iter_count(iter)) { + ret = -EFAULT; + break; + } + } + release_user_fragments(fragments, got, write); + if (ret) + break; + } + if (fragments != stack) + kfree(fragments); + return ret; +} +#endif + /** * process_vm_rw_core - core of reading/writing pages from task specified * @pid: PID of process to read/write from/to @@ -182,17 +242,6 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter, if (nr_pages == 0) return 0; - if (nr_pages > PVM_MAX_PP_ARRAY_COUNT) { - /* For reliability don't try to kmalloc more than - 2 pages worth */ - process_pages = kmalloc(min_t(size_t, PVM_MAX_KMALLOC_PAGES * PAGE_SIZE, - sizeof(struct page *)*nr_pages), - GFP_KERNEL); - - if (!process_pages) - return -ENOMEM; - } - /* Get process information */ task = find_get_task_by_vpid(pid); if (!task) { @@ -212,11 +261,35 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter, goto put_task_struct; } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) != PAGE_SIZE) { + for (i = 0; i < riovcnt && iov_iter_count(iter) && !rc; i++) + rc = process_vm_rw_fragments(mm, (unsigned long)rvec[i].iov_base, + rvec[i].iov_len, iter, vm_write); + goto copied; + } +#endif + if (nr_pages > PVM_MAX_PP_ARRAY_COUNT) { + /* For reliability don't try to kmalloc more than + 2 pages worth */ + process_pages = kmalloc(min_t(size_t, PVM_MAX_KMALLOC_PAGES * PAGE_SIZE, + sizeof(struct page *)*nr_pages), + GFP_KERNEL); + + if (!process_pages) { + rc = -ENOMEM; + goto put_mm; + } + } + for (i = 0; i < riovcnt && iov_iter_count(iter) && !rc; i++) rc = process_vm_rw_single_vec( (unsigned long)rvec[i].iov_base, rvec[i].iov_len, iter, process_pages, mm, task, vm_write); +#ifdef CONFIG_MM_SUBPAGE +copied: +#endif /* copied = space before - space after */ total_len -= iov_iter_count(iter); @@ -226,6 +299,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter, if (total_len) rc = total_len; +put_mm: mmput(mm); put_task_struct: diff --git a/mm/rmap.c b/mm/rmap.c index 99e1b3dc390b7f..84aba8ec5fa418 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -75,6 +75,7 @@ #include #include #include +#include #include @@ -83,17 +84,22 @@ #include "internal.h" #include "swap.h" +#include "user-subpage-internal.h" static struct kmem_cache *anon_vma_cachep; static struct kmem_cache *anon_vma_chain_cachep; -static inline struct anon_vma *anon_vma_alloc(void) +static inline struct anon_vma *anon_vma_alloc(struct vm_area_struct *vma) { struct anon_vma *anon_vma; anon_vma = kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL); if (anon_vma) { atomic_set(&anon_vma->refcount, 1); +#ifdef CONFIG_MM_SUBPAGE + anon_vma->min_folio_order = + max_t(unsigned int, mm_page_shift(vma->vm_mm), PAGE_SHIFT) - PAGE_SHIFT; +#endif anon_vma->num_children = 0; anon_vma->num_active_vmas = 0; anon_vma->parent = anon_vma; @@ -198,7 +204,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma) anon_vma = find_mergeable_anon_vma(vma); allocated = NULL; if (!anon_vma) { - anon_vma = anon_vma_alloc(); + anon_vma = anon_vma_alloc(vma); if (unlikely(!anon_vma)) goto out_enomem_free_avc; anon_vma->num_children++; /* self-parent link for new root */ @@ -388,7 +394,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) /* Drop inherited anon_vma, we'll reuse existing or allocate new. */ vma->anon_vma = NULL; - anon_vma = anon_vma_alloc(); + anon_vma = anon_vma_alloc(vma); if (!anon_vma) return -ENOMEM; avc = anon_vma_chain_alloc(GFP_KERNEL); @@ -622,6 +628,38 @@ struct anon_vma *folio_get_anon_vma(const struct folio *folio) return anon_vma; } +#ifdef CONFIG_MM_SUBPAGE +/* + * The caller retains the folio; splitting rechecks the order under its lock. + * May sleep when dropping the temporary anon-vma reference. + */ +unsigned int folio_anon_min_order(struct folio *folio) +{ + struct anon_vma *anon_vma = NULL; + unsigned long mapping; + unsigned int order = 0; + + /* Also used by memory-failure callers before they lock the folio. */ + rcu_read_lock(); + mapping = (unsigned long)READ_ONCE(folio->mapping); + if ((mapping & FOLIO_MAPPING_FLAGS) != FOLIO_MAPPING_ANON || !folio_mapped(folio)) + goto out; + anon_vma = (struct anon_vma *)(mapping - FOLIO_MAPPING_ANON); + if (!atomic_inc_not_zero(&anon_vma->refcount)) { + anon_vma = NULL; + goto out; + } + /* Revalidate against unmap/reuse of the RCU-typesafe anon_vma slab. */ + if (folio_mapped(folio) && mapping == (unsigned long)READ_ONCE(folio->mapping)) + order = anon_vma->min_folio_order; +out: + rcu_read_unlock(); + if (anon_vma) + put_anon_vma(anon_vma); + return order; +} +#endif + /* * Similar to folio_get_anon_vma() except it locks the anon_vma. * @@ -856,6 +894,13 @@ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) unsigned long page_address_in_vma(const struct folio *folio, const struct page *page, const struct vm_area_struct *vma) { +#ifdef CONFIG_MM_SUBPAGE + unsigned long address; + + if (folio_test_anon(folio) && !folio_test_large(folio) && + mm_subpage_address_in_vma(folio, vma, &address)) + return address; +#endif if (folio_test_anon(folio)) { struct anon_vma *anon_vma = folio_anon_vma(folio); /* @@ -891,15 +936,15 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address) if (!pgd_present(*pgd)) goto out; - p4d = p4d_offset(pgd, address); - if (!p4d_present(*p4d)) + p4d = p4d_offset_mm(mm, pgd, address); + if (!p4d_present_mm(mm, *p4d)) goto out; - pud = pud_offset(p4d, address); + pud = pud_offset_mm(mm, p4d, address); if (!pud_present(*pud)) goto out; - pmd = pmd_offset(pud, address); + pmd = pmd_offset_mm(mm, pud, address); out: return pmd; } @@ -915,10 +960,11 @@ struct folio_referenced_arg { * arg: folio_referenced_arg will be passed */ static bool folio_referenced_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long address, void *arg) + struct vm_area_struct *vma, struct rmap_walk_range rmap, void *arg) { + unsigned long address = rmap.address; struct folio_referenced_arg *pra = arg; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); int ptes = 0, referenced = 0; unsigned int nr; @@ -928,10 +974,11 @@ static bool folio_referenced_one(struct folio *folio, if (vma->vm_flags & VM_LOCKED) { ptes++; - pra->mapcount--; + pra->mapcount -= pvmw.pte ? mm_pte_native_pages(vma->vm_mm) : 1; /* Only mlock fully mapped pages */ - if (pvmw.pte && ptes != pvmw.nr_pages) + if (pvmw.pte && ptes != (folio_test_large(folio) ? + ((unsigned long)pvmw.nr_pages << PAGE_SHIFT) >> mm_page_shift(vma->vm_mm) : 1)) continue; /* @@ -966,11 +1013,18 @@ static bool folio_referenced_one(struct folio *folio, } if (pvmw.pte && folio_test_large(folio)) { - const unsigned long end_addr = pmd_addr_end(address, vma->vm_end); - const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT; + const unsigned long end_addr = pmd_addr_end_mm(vma->vm_mm, + address, vma->vm_end); + const unsigned int max_nr = (end_addr - address) >> mm_page_shift(vma->vm_mm); pte_t pteval = ptep_get(pvmw.pte); - nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + nr = folio_subpage_pte_batch(vma, folio, pvmw.pte, + pteval, max_nr, 0); + else +#endif + nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr); } /* @@ -994,19 +1048,19 @@ static bool folio_referenced_one(struct folio *folio, } ptes += nr; - pra->mapcount -= nr; + pra->mapcount -= nr * (pvmw.pte ? mm_pte_native_pages(vma->vm_mm) : 1); /* * If we are sure that we batched the entire folio, * we can just optimize and stop right here. */ - if (ptes == pvmw.nr_pages) { + if (ptes == (((unsigned long)pvmw.nr_pages << PAGE_SHIFT) >> mm_page_shift(vma->vm_mm))) { page_vma_mapped_walk_done(&pvmw); break; } /* Skip the batched PTEs */ pvmw.pte += nr - 1; - pvmw.address += (nr - 1) * PAGE_SIZE; + pvmw.address += (nr - 1) * mm_page_size(vma->vm_mm); } if (referenced) @@ -1178,9 +1232,9 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) } static bool page_mkclean_one(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, void *arg) + struct rmap_walk_range rmap, void *arg) { - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, PVMW_SYNC); + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, PVMW_SYNC); int *cleaned = arg; *cleaned += page_vma_mkclean_one(&pvmw); @@ -1229,8 +1283,9 @@ struct wrprotect_file_state { }; static bool mapping_wrprotect_range_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long address, void *arg) + struct vm_area_struct *vma, struct rmap_walk_range rmap, void *arg) { + unsigned long address = rmap.address; struct wrprotect_file_state *state = (struct wrprotect_file_state *)arg; struct page_vma_mapped_walk pvmw = { .pfn = state->pfn, @@ -1515,7 +1570,7 @@ static void __page_check_anon_rmap(const struct folio *folio, static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, - unsigned long address, rmap_t flags, enum pgtable_level level) + unsigned long address, rmap_t flags, enum pgtable_level level, bool new) { int i; @@ -1572,7 +1627,7 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, * Partially mapped folios can be split on reclaim and part outside * of mlocked VMA can be evicted or freed. */ - if (folio_nr_pages(folio) == nr_pages) + if (!new && folio_nr_pages(folio) == nr_pages) mlock_vma_folio(folio, vma); } @@ -1597,7 +1652,7 @@ void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, rmap_t flags) { __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, - PGTABLE_LEVEL_PTE); + PGTABLE_LEVEL_PTE, false); } /** @@ -1618,7 +1673,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, { #ifdef CONFIG_TRANSPARENT_HUGEPAGE __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, - PGTABLE_LEVEL_PMD); + PGTABLE_LEVEL_PMD, false); #else WARN_ON_ONCE(true); #endif @@ -1697,6 +1752,112 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1); } +#ifdef CONFIG_MM_SUBPAGE +static int __mm_subpage_add_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma, + unsigned long address, bool new) +{ + struct folio *folio = mm_subpage_folio(subpage); + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (mm_page_shift(vma->vm_mm) != mm_subpage_shift(subpage) || !vma->anon_vma || + address < vma->vm_start || address >= vma->vm_end || + !IS_ALIGNED(address, mm_subpage_size(subpage))) + return -EINVAL; + if (mm_subpage_anon_root(subpage)) { + if (mm_subpage_is_droppable(subpage) != !!(vma->vm_flags & VM_DROPPABLE)) + return -EINVAL; + if (!folio_test_anon(folio) || folio_test_ksm(folio) || + PageAnonExclusive(&folio->page)) + return -EINVAL; + } else { + bool was_anon = folio_test_anon(folio); + + /* Swap-cache folios can retain a stale, unmapped anon pointer. */ + if (folio_mapped(folio) || folio_test_ksm(folio) || + (folio->mapping && !was_anon) || + (was_anon && PageAnonExclusive(&folio->page))) + return -EINVAL; + /* Slot references can survive the last VMA in this root. */ + mm_subpage_hold_anon_root(subpage, vma->anon_vma->root, + vma->vm_flags & VM_DROPPABLE); + /* + * A restored folio is already visible in the swap cache. Other + * faults may set PG_waiters while we hold its lock; preserve those + * concurrent flag updates even when PG_swapbacked is already set. + */ + if (!(vma->vm_flags & VM_DROPPABLE)) + folio_set_swapbacked(folio); + /* Neighboring slots may acquire different fork/COW relationships. */ + __folio_set_anon(folio, vma, address, false); + if (!was_anon) + mod_mthp_stat(0, MTHP_STAT_NR_ANON, 1); + } + ret = mm_subpage_bind_rmap(subpage, vma, address); + if (ret) + return ret; + atomic_inc(&subpage->mapcount); + /* Identity was checked per slot, not against the folio's linear anchor. */ + __folio_add_rmap(folio, &folio->page, 1, vma, PGTABLE_LEVEL_PTE); + if (!new) + mlock_vma_folio(folio, vma); + return 0; +} + +int mm_subpage_add_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma, + unsigned long address) +{ + return __mm_subpage_add_anon_rmap(subpage, vma, address, false); +} + +/* Caller adds this fresh folio to LRU and accounts its locked leaves together. */ +int mm_subpage_add_new_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma, + unsigned long address) +{ + return __mm_subpage_add_anon_rmap(subpage, vma, address, true); +} + +void mm_subpage_remove_anon_rmap(struct mm_subpage *subpage, struct vm_area_struct *vma) +{ + struct folio *folio = mm_subpage_folio(subpage); + + VM_BUG_ON_FOLIO(atomic_read(&subpage->mapcount) <= 0, folio); + atomic_dec(&subpage->mapcount); + folio_remove_rmap_pte(folio, &folio->page, vma); +} + +int mm_subpage_dup_anon_rmap(struct mm_subpage *subpage, + struct vm_area_struct *dst, struct vm_area_struct *src, unsigned long address) +{ + struct folio *folio = mm_subpage_folio(subpage); + struct vm_page_offset pos; + int ret; + + mmap_assert_write_locked(src->vm_mm); + mmap_assert_write_locked(dst->vm_mm); + if (mm_page_shift(src->vm_mm) != mm_subpage_shift(subpage) || + mm_page_shift(dst->vm_mm) != mm_subpage_shift(subpage) || + !src->anon_vma || !dst->anon_vma || !folio_test_anon(folio) || + folio_test_ksm(folio) || PageAnonExclusive(&folio->page) || + address < dst->vm_start || address >= dst->vm_end || + address < src->vm_start || address >= src->vm_end) + return -EINVAL; + pos = vma_page_offset_at(dst, address); + /* Both mmap locks and the source PTE stabilize this slot's identity. */ + if (!mm_subpage_match_rmap(subpage, dst, address) || + src->anon_vma->root != dst->anon_vma->root || + !vm_page_offset_equal(pos, vma_page_offset_at(src, address))) + return -EINVAL; + /* A pin in a neighboring physical slot does not share this slot. */ + if (!mm_subpage_try_share(subpage)) + return -EBUSY; + ret = folio_try_dup_anon_rmap_pte(folio, &folio->page, dst, src); + if (!ret) + atomic_inc(&subpage->mapcount); + return ret; +} +#endif + static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum pgtable_level level) @@ -1949,14 +2110,20 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio, struct vm_area_struct *vma = pvmw->vma; unsigned int max_nr; +#ifdef CONFIG_MM_SUBPAGE + /* Native PFN batching advances physical pages, not 4K file quarters. */ + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return 1; +#endif + if (flags & TTU_HWPOISON) return 1; if (!folio_test_large(folio)) return 1; /* We may only batch within a single VMA and a single page table. */ - end_addr = pmd_addr_end(addr, vma->vm_end); - max_nr = (end_addr - addr) >> PAGE_SHIFT; + end_addr = pmd_addr_end_mm(vma->vm_mm, addr, vma->vm_end); + max_nr = (end_addr - addr) >> mm_page_shift(vma->vm_mm); /* We only support lazyfree or file folios batching for now ... */ if (folio_test_anon(folio) && folio_test_swapbacked(folio)) @@ -1978,20 +2145,130 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio, FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY); } +#ifdef CONFIG_MM_SUBPAGE +/* Folio and PTE locks held. Each PTE takes one reference to a native swap entry. */ +static bool try_to_unmap_subpage(struct folio *folio, struct vm_area_struct *vma, + struct page_vma_mapped_walk *pvmw, enum ttu_flags flags) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned long address = pvmw->address, end = address + mm_page_size(mm); + struct mm_subpage *slot; + pte_t pte = ptep_get(pvmw->pte), swapped; + swp_entry_t entry; + bool exclusive, ret = false; + + if (!pte_present(pte)) + return false; + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + if (WARN_ON_ONCE(!slot)) + return false; + if (folio_test_hwpoison(folio) && (flags & TTU_HWPOISON)) { + flush_cache_range(vma, address, end); + pte = ptep_get_and_clear(mm, address, pvmw->pte); + /* Retained references keep their slot, but no CPU may keep its PTE. */ + flush_tlb_range(vma, address, end); + if (pte_dirty(pte)) + folio_mark_dirty(folio); + update_hiwater_rss(mm); + dec_mm_counter(mm, MM_ANONPAGES); + set_pte_at(mm, address, pvmw->pte, + swp_entry_to_pte(make_hwpoison_entry(&folio->page))); + mm_subpage_remove_anon_rmap(slot, vma); + mm_subpage_put(slot); /* Removed PTE reference, after invalidation. */ + ret = true; + goto out; + } + if ((vma->vm_flags & VM_DROPPABLE) || mm_subpage_is_lazyfree(slot)) { + flush_cache_range(vma, address, end); + pte = ptep_get_and_clear(mm, address, pvmw->pte); + /* A released quarter can be reused before its native folio is freed. */ + flush_tlb_range(vma, address, end); + smp_mb(); + if (((vma->vm_flags & VM_DROPPABLE) || !pte_dirty(pte)) && + mm_subpage_discardable(slot) && + arch_unmap_one(mm, vma, address, pte) >= 0) { + update_hiwater_rss(mm); + dec_mm_counter(mm, MM_ANONPAGES); + mm_subpage_remove_anon_rmap(slot, vma); + mm_subpage_put(slot); /* Removed PTE reference. */ + ret = true; + goto out; + } + set_pte_at(mm, address, pvmw->pte, pte); + /* A retained reference alone does not cancel clean lazyfree data. */ + if ((vma->vm_flags & VM_DROPPABLE) || !pte_dirty(pte)) + goto out; + mm_subpage_cancel_lazyfree(slot); + folio_mark_dirty(folio); + } + if (!folio_test_swapbacked(folio)) { + /* A write cancelled whole-folio lazy reclaim. */ + folio_set_swapbacked(folio); + goto out; + } + if (!folio_test_swapcache(folio)) + goto out; + entry = page_swap_entry(&folio->page); + if (swap_subpage_dup(entry, mm_subpage_offset(slot)) < 0) + goto out; + flush_cache_range(vma, address, end); + pte = ptep_get_and_clear(mm, address, pvmw->pte); + if (should_defer_flush(mm, flags)) + set_tlb_ubc_flush_pending(mm, pte, address, end); + else + flush_tlb_range(vma, address, end); + if (pte_dirty(pte)) + folio_mark_dirty(folio); + /* The PTE lock serializes fork and this slot's ownership transfer. */ + exclusive = mm_subpage_is_exclusive(slot); + if (arch_unmap_one(mm, vma, address, pte) < 0 || !mm_subpage_try_share(slot)) { + swap_subpage_put(entry, mm_subpage_offset(slot), false); + set_pte_at(mm, address, pvmw->pte, pte); + goto out; + } + if (list_empty(&mm->mmlist)) { + spin_lock(&mmlist_lock); + if (list_empty(&mm->mmlist)) + list_add(&mm->mmlist, &init_mm.mmlist); + spin_unlock(&mmlist_lock); + } + swapped = pte_swp_set_subpage_offset(swp_entry_to_pte(entry), mm_subpage_offset(slot)); + if (exclusive) + swapped = pte_swp_mkexclusive(swapped); + if (pte_soft_dirty(pte)) + swapped = pte_swp_mksoft_dirty(swapped); + if (pte_uffd_wp(pte)) + swapped = pte_swp_mkuffd_wp(swapped); + update_hiwater_rss(mm); + dec_mm_counter(mm, MM_ANONPAGES); + inc_mm_counter(mm, MM_SWAPENTS); + set_pte_at(mm, address, pvmw->pte, swapped); + mm_subpage_remove_anon_rmap(slot, vma); + /* Swap-cache backing cannot be recycled while a deferred TLBI is pending. */ + mm_subpage_put(slot); + ret = true; +out: + mm_subpage_put(slot); + return ret; +} +#endif + /* * @arg: enum ttu_flags will be passed to this argument */ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, void *arg) + struct rmap_walk_range rmap, void *arg) { + unsigned long address = rmap.address; struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); bool anon_exclusive, ret = true; pte_t pteval; + phys_addr_t file_phys = 0; struct page *subpage; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; - unsigned long nr_pages = 1, end_addr; + unsigned long nr_pages = 1, end_addr, nr_refs; unsigned long pfn; unsigned long hsz = 0; int ptes = 0; @@ -2003,7 +2280,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * if page table locking is skipped: use TTU_SYNC to wait for that. */ if (flags & TTU_SYNC) - pvmw.flags = PVMW_SYNC; + pvmw.flags |= PVMW_SYNC; /* * For THP, we have to assume the worse case ie pmd for invalidation. @@ -2049,7 +2326,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, ret = false; /* Only mlock fully mapped pages */ - if (pvmw.pte && ptes != pvmw.nr_pages) + if (pvmw.pte && ptes != (folio_test_large(folio) ? + ((unsigned long)pvmw.nr_pages << PAGE_SHIFT) >> mm_page_shift(vma->vm_mm) : 1)) continue; /* @@ -2100,6 +2378,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * actually map pages. */ pteval = ptep_get(pvmw.pte); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE && folio_test_anon(folio)) { + if (!try_to_unmap_subpage(folio, vma, &pvmw, flags)) + goto walk_abort; + /* One native folio may cover four independently mapped slots. */ + continue; + } +#endif + if (likely(pte_present(pteval))) { pfn = pte_pfn(pteval); } else { @@ -2110,6 +2397,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } subpage = folio_page(folio, pfn - folio_pfn(folio)); + if (!folio_test_anon(folio)) + file_phys = pte_phys_mm(mm, ptep_get(pvmw.pte)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(subpage); @@ -2167,7 +2456,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, folio_mark_dirty(folio); } else if (likely(pte_present(pteval))) { nr_pages = folio_unmap_pte_batch(folio, &pvmw, flags, pteval); - end_addr = address + nr_pages * PAGE_SIZE; + end_addr = address + nr_pages * mm_page_size(mm); flush_cache_range(vma, address, end_addr); /* Nuke the page table entry. */ @@ -2281,7 +2570,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto discard; } - if (folio_dup_swap(folio, subpage) < 0) { + if (folio_dup_swap_range(folio, subpage, mm_pte_native_pages(mm)) < 0) { set_pte_at(mm, address, pvmw.pte, pteval); goto walk_abort; } @@ -2292,15 +2581,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - folio_put_swap(folio, subpage); + folio_put_swap_range(folio, subpage, mm_pte_native_pages(mm)); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_abort; } /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { - folio_put_swap(folio, subpage); + folio_try_share_anon_rmap_pte(folio, subpage, vma)) { + folio_put_swap_range(folio, subpage, mm_pte_native_pages(mm)); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_abort; } @@ -2342,20 +2631,26 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, add_mm_counter(mm, mm_counter_file(folio), -nr_pages); } discard: + nr_refs = nr_pages; + /* A base user leaf can own references to several native pages. */ + if (!folio_test_hugetlb(folio)) + nr_refs *= mm_pte_native_pages(mm); + if (!folio_test_anon(folio)) + mm_subpage_file_map_del(vma, file_phys); if (unlikely(folio_test_hugetlb(folio))) { hugetlb_remove_rmap(folio); } else { - folio_remove_rmap_ptes(folio, subpage, nr_pages, vma); + folio_remove_rmap_ptes(folio, subpage, nr_refs, vma); } if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); - folio_put_refs(folio, nr_pages); + folio_put_refs(folio, nr_refs); /* * If we are sure that we batched the entire folio and cleared * all PTEs, we can just optimize and stop right here. */ - if (nr_pages == folio_nr_pages(folio)) + if (nr_pages == folio_nr_pages(folio) && mm_page_size(mm) == PAGE_SIZE) goto walk_done; continue; walk_abort: @@ -2412,11 +2707,72 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags) * If TTU_SPLIT_HUGE_PMD is specified any PMD mappings will be split into PTEs * containing migration entries. */ +#ifdef CONFIG_MM_SUBPAGE +static bool try_to_migrate_subpage(struct folio *folio, struct vm_area_struct *vma, + struct page_vma_mapped_walk *pvmw, enum ttu_flags flags) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned long address = pvmw->address, end = address + mm_page_size(mm); + struct mm_subpage *slot; + pte_t pte = ptep_get(pvmw->pte), migration; + swp_entry_t entry; + bool ret = false; + + /* Other migration drivers must prepare ownership before removing PTEs. */ + if (!pte_present(pte) || !mm_subpage_migrating(folio) || + folio_test_hwpoison(folio) || folio_is_zone_device(folio)) + return false; + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, pte)); + if (WARN_ON_ONCE(!slot)) + return false; + flush_cache_range(vma, address, end); + pte = ptep_get_and_clear(mm, address, pvmw->pte); + if (should_defer_flush(mm, flags)) + set_tlb_ubc_flush_pending(mm, pte, address, end); + else + flush_tlb_range(vma, address, end); + if (pte_dirty(pte)) + folio_mark_dirty(folio); + if (arch_unmap_one(mm, vma, address, pte) < 0) { + set_pte_at(mm, address, pvmw->pte, pte); + goto out; + } + if (pte_write(pte)) + entry = make_writable_migration_entry(folio_pfn(folio)); + else if (mm_subpage_is_exclusive(slot)) + entry = make_readable_exclusive_migration_entry(folio_pfn(folio)); + else + entry = make_readable_migration_entry(folio_pfn(folio)); + if (pte_young(pte)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry = make_migration_entry_dirty(entry); + migration = pte_swp_set_subpage_offset(swp_entry_to_pte(entry), mm_subpage_offset(slot)); + if (pte_soft_dirty(pte)) + migration = pte_swp_mksoft_dirty(migration); + if (pte_uffd_wp(pte)) + migration = pte_swp_mkuffd_wp(migration); + set_pte_at(mm, address, pvmw->pte, migration); + trace_set_migration_pte(address, pte_val(migration), 0); + mm_subpage_migrate_save_lazyfree(slot); + mm_subpage_remove_anon_rmap(slot, vma); + if (vma->vm_flags & VM_LOCKED) + mlock_drain_local(); + /* The driver holds backing until its batched invalidation completes. */ + mm_subpage_put(slot); + ret = true; +out: + mm_subpage_put(slot); + return ret; +} +#endif + static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, void *arg) + struct rmap_walk_range rmap, void *arg) { + unsigned long address = rmap.address; struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + DEFINE_FOLIO_RMAP_WALK(pvmw, folio, vma, rmap, 0); bool anon_exclusive, writable, ret = true; pte_t pteval; struct page *subpage; @@ -2424,6 +2780,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; unsigned long hsz = 0; + unsigned int nr_refs = 1; + + /* A base PTE owns one reference for each covered native page. */ + if (!folio_test_hugetlb(folio)) + nr_refs = mm_pte_native_pages(mm); /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2432,7 +2793,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * if page table locking is skipped: use TTU_SYNC to wait for that. */ if (flags & TTU_SYNC) - pvmw.flags = PVMW_SYNC; + pvmw.flags |= PVMW_SYNC; /* * For THP, we have to assume the worse case ie pmd for invalidation. @@ -2501,6 +2862,17 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); + +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE && folio_test_anon(folio)) { + if (!try_to_migrate_subpage(folio, vma, &pvmw, flags)) { + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + continue; + } +#endif /* * Handle PFN swap PTEs, such as device-exclusive ones, that * actually map pages. @@ -2586,7 +2958,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address, address + PAGE_SIZE); + set_tlb_ubc_flush_pending(mm, pteval, address, address + mm_page_size(mm)); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -2663,7 +3035,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage, vma)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); @@ -2701,6 +3073,13 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (pte_swp_uffd_wp(pteval)) swp_pte = pte_swp_mkuffd_wp(swp_pte); } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) + swp_pte = pte_swp_set_subpage_offset(swp_pte, + pte_present(pteval) ? pte_phys_mm(mm, pteval) & ~PAGE_MASK : + pte_swp_subpage_offset(pteval)); +#endif + if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); @@ -2714,13 +3093,15 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } + if (!folio_test_anon(folio)) + mm_subpage_file_map_del(vma, pte_phys_mm(mm, pteval)); if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); - folio_put(folio); + folio_put_refs(folio, nr_refs); } mmu_notifier_invalidate_range_end(&range); @@ -2951,6 +3332,61 @@ static struct anon_vma *rmap_walk_anon_lock(const struct folio *folio, return anon_vma; } +#ifdef CONFIG_MM_SUBPAGE +noinline bool rmap_walk_subpages(struct folio *folio, + struct rmap_walk_control *rwc, bool locked) +{ + struct mm_subpage_rmap maps[MM_SUBPAGES_PER_PAGE]; + int i, nr = mm_subpage_snapshot_rmap(folio, maps); + + if (nr < 0) + return false; + /* Existing locked anonymous walks operate on large folios. */ + VM_BUG_ON_FOLIO(locked, folio); + for (i = 0; i < nr; i++) { + struct mm_subpage_rmap *map = &maps[i]; + struct anon_vma_chain *avc; + bool stop = false; + + if (!anon_vma_trylock_read(map->root)) { + if (rwc->try_lock) { + rwc->contended = true; + break; + } + anon_vma_lock_read(map->root); + } + anon_vma_interval_tree_foreach(avc, &map->root->rb_root, + map->pos.index, map->pos.index) { + struct vm_area_struct *vma = avc->vma; + unsigned long address = vma_address_at_offset(vma, map->pos); + struct rmap_walk_range range = { + .address = address, + .subpage = true, + .subpage_offset = map->offset, + }; + + /* Native-index overlap need not include this exact leaf. */ + if (address == -EFAULT) + continue; + cond_resched(); + if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) + continue; + if (!rwc->rmap_one(folio, vma, range, rwc->arg) || + (rwc->done && rwc->done(folio))) { + stop = true; + break; + } + } + anon_vma_unlock_read(map->root); + if (stop) + break; + } + for (i = 0; i < nr; i++) + put_anon_vma(maps[i].root); + return true; +} +#endif + /* * rmap_walk_anon - do something to anonymous page using the object-based * rmap method @@ -2973,6 +3409,10 @@ static void rmap_walk_anon(struct folio *folio, * to an anon_vma with different root. */ VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); +#ifdef CONFIG_MM_SUBPAGE + if (!folio_test_large(folio) && rmap_walk_subpages(folio, rwc, locked)) + return; +#endif if (locked) { anon_vma = folio_anon_vma(folio); @@ -2998,7 +3438,8 @@ static void rmap_walk_anon(struct folio *folio, if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) continue; - if (!rwc->rmap_one(folio, vma, address, rwc->arg)) + if (!rwc->rmap_one(folio, vma, + (struct rmap_walk_range) { .address = address }, rwc->arg)) break; if (rwc->done && rwc->done(folio)) break; @@ -3061,7 +3502,8 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping, if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) continue; - if (!rwc->rmap_one(folio, vma, address, rwc->arg)) + if (!rwc->rmap_one(folio, vma, + (struct rmap_walk_range) { .address = address }, rwc->arg)) goto done; if (rwc->done && rwc->done(folio)) goto done; diff --git a/mm/shmem.c b/mm/shmem.c index 7b1ea9fb598f4a..705765dddf1d22 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -30,6 +30,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -875,6 +878,266 @@ static void shmem_update_stats(struct folio *folio, int nr_pages) lruvec_stat_mod_folio(folio, NR_SHMEM, nr_pages); } +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) +#define SHMEM_UFFD_ALL_UNITS GENMASK(PAGE_SIZE / SZ_4K - 1, 0) + +/* + * Resident masks live in page_ext, allowing partial truncate to update them + * without allocation. All cache insertion/replacement paths initialize or + * transfer them under the folio lock. The inode XArray persists masks only + * for swapped indices. Its publication/removal additionally holds i_pages, + * before taking uffd_missing's XArray lock. + */ +static void shmem_uffd_enable(struct inode *inode) +{ + struct shmem_inode_info *info = SHMEM_I(inode); + + spin_lock(&info->lock); + info->flags |= SHMEM_F_UFFD_SUBPAGE; + spin_unlock(&info->lock); +} + +static unsigned long shmem_uffd_mask(unsigned int offset, unsigned long length) +{ + return GENMASK((offset + length - 1) / SZ_4K, offset / SZ_4K); +} + +unsigned long shmem_uffd_missing_mask(struct folio *folio, pgoff_t index) +{ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); + VM_BUG_ON_FOLIO(!folio_mapping(folio) || !shmem_mapping(folio_mapping(folio)), folio); + return mm_subpage_shmem_missing(folio_page(folio, index - folio->index)); +} + +int shmem_uffd_set_missing(struct folio *folio, pgoff_t index, + unsigned long missing) +{ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (!folio_mapping(folio) || !shmem_mapping(folio_mapping(folio)) || + !folio_contains(folio, index) || + !folio_test_uptodate(folio) || (missing & ~SHMEM_UFFD_ALL_UNITS)) + return -EINVAL; + if (missing) + shmem_uffd_enable(folio->mapping->host); + mm_subpage_shmem_set_missing(folio_page(folio, index - folio->index), missing); + return 0; +} + +bool shmem_uffd_has_missing_native_page(struct folio *folio) +{ + struct address_space *mapping = folio_mapping(folio); + unsigned long i; + + if (!mapping || !shmem_mapping(mapping) || + !(READ_ONCE(SHMEM_I(mapping->host)->flags) & SHMEM_F_UFFD_SUBPAGE)) + return false; + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + /* Native PMD exposure is allowed when every native leaf has some data. */ + for (i = 0; i < folio_nr_pages(folio); i++) + if (mm_subpage_shmem_missing(folio_page(folio, i)) == SHMEM_UFFD_ALL_UNITS) + return true; + return false; +} + +bool shmem_uffd_range_missing(struct folio *folio, pgoff_t index, + unsigned int offset, unsigned long length) +{ + struct address_space *mapping = folio_mapping(folio); + unsigned long mask; + + if (!mapping || !shmem_mapping(mapping)) + return false; + mask = shmem_uffd_mask(offset, length); + return (shmem_uffd_missing_mask(folio, index) & mask) == mask; +} + +/* A coarse alias with any populated units preserves them and exposes zero holes. */ +static unsigned long shmem_uffd_reason(struct vm_fault *vmf, unsigned long missing) +{ + struct vm_area_struct *vma = vmf->vma; + struct vm_page_offset pos = vma_page_offset_at(vma, vmf->address); + unsigned long mask = shmem_uffd_mask(pos.offset, mm_page_size(vma->vm_mm)); + + if ((missing & mask) == mask) + return userfaultfd_missing(vma) ? VM_UFFD_MISSING : 0; + return userfaultfd_minor(vma) ? VM_UFFD_MINOR : 0; +} + +void shmem_uffd_mark_present(struct folio *folio, pgoff_t index, + unsigned long present) +{ + unsigned long missing = shmem_uffd_missing_mask(folio, index); + + if (missing & present) + mm_subpage_shmem_set_missing(folio_page(folio, index - folio->index), + missing & ~present); +} + +void shmem_uffd_mark_mapped(struct folio *folio, unsigned long offset, + unsigned long length) +{ + struct address_space *mapping = folio_mapping(folio); + + if (!mapping || !shmem_mapping(mapping) || + !(READ_ONCE(SHMEM_I(mapping->host)->flags) & SHMEM_F_UFFD_SUBPAGE)) + return; + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(offset + length > folio_size(folio), folio); + while (length) { + unsigned int low = offset & ~PAGE_MASK; + unsigned long size = min(length, PAGE_SIZE - low); + unsigned long mask = GENMASK((low + size - 1) / SZ_4K, low / SZ_4K); + + shmem_uffd_mark_present(folio, folio->index + (offset >> PAGE_SHIFT), mask); + offset += size; + length -= size; + } +} + +void shmem_uffd_forget(struct address_space *mapping, pgoff_t start, + unsigned long nr) +{ + struct xarray *xa; + unsigned long index; + void *entry; + + if (!shmem_mapping(mapping)) + return; + lockdep_assert_held(&mapping->i_pages.xa_lock); + xa = &SHMEM_I(mapping->host)->uffd_missing; + if (xa_empty(xa)) + return; + xa_lock(xa); + xa_for_each_range(xa, index, entry, start, start + nr - 1) + __xa_erase(xa, index); + xa_unlock(xa); +} + +/* New cache identity or swap-in: initialize every native page before unlock. */ +static void shmem_uffd_cache_init(struct folio *folio, bool from_swap) +{ + struct address_space *mapping = folio->mapping; + struct xarray *xa = &SHMEM_I(mapping->host)->uffd_missing; + unsigned long i; + + lockdep_assert_held(&mapping->i_pages.xa_lock); + for (i = 0; i < folio_nr_pages(folio); i++) { + void *entry = from_swap ? xa_load(xa, folio->index + i) : NULL; + + mm_subpage_shmem_set_missing(folio_page(folio, i), + entry ? xa_to_value(entry) : 0); + } + shmem_uffd_forget(mapping, folio->index, folio_nr_pages(folio)); +} + +/* + * Reserve persistence before committing swap. On failure the cache folio and + * its resident mask stay intact; reclaim may retry once memory is available. + */ +static int shmem_uffd_swap_prepare(struct folio *folio) +{ + struct address_space *mapping = folio->mapping; + struct xarray *xa = &SHMEM_I(mapping->host)->uffd_missing; + struct mem_cgroup *memcg, *saved; + unsigned long i; + int err = 0; + + if (!(READ_ONCE(SHMEM_I(mapping->host)->flags) & SHMEM_F_UFFD_SUBPAGE)) + return 0; + /* Reclaim may run outside the backing folio's charge domain. */ + rcu_read_lock(); + do { + memcg = folio_memcg(folio) ?: root_mem_cgroup; + } while (memcg && !mem_cgroup_tryget(memcg)); + rcu_read_unlock(); + saved = set_active_memcg(memcg); + for (i = 0; i < folio_nr_pages(folio); i++) { + unsigned long mask = mm_subpage_shmem_missing(folio_page(folio, i)); + + if (!mask) + continue; + err = xa_err(xa_store(xa, folio->index + i, xa_mk_value(mask), + GFP_NOWAIT | __GFP_ACCOUNT | __GFP_NOWARN)); + if (err) + break; + } + set_active_memcg(saved); + mem_cgroup_put(memcg); + if (err) { + xa_lock_irq(&mapping->i_pages); + shmem_uffd_forget(mapping, folio->index, folio_nr_pages(folio)); + xa_unlock_irq(&mapping->i_pages); + } + return err; +} + +void shmem_uffd_copy_page(struct page *dst, struct page *src) +{ + mm_subpage_shmem_set_missing(dst, src ? mm_subpage_shmem_missing(src) : 0); +} + +void shmem_uffd_migrate(struct folio *dst, struct folio *src) +{ + unsigned long i; + + if (!shmem_mapping(src->mapping)) + return; + VM_BUG_ON_FOLIO(!folio_test_locked(src), src); + VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); + for (i = 0; i < folio_nr_pages(src); i++) + shmem_uffd_copy_page(folio_page(dst, i), folio_page(src, i)); +} + +/* Only fully removed 4K units become holes; sub-unit edge bytes stay valid. */ +static void shmem_uffd_partial_hole(struct folio *folio, loff_t start, uoff_t end) +{ + loff_t first = max(start, folio_pos(folio)); + u64 last = min_t(u64, end, folio_next_pos(folio) - 1); + unsigned long offset, length; + bool pinned; + + if (first >= last || + (first == folio_pos(folio) && last == folio_next_pos(folio) - 1)) + return; + first = round_up(first, SZ_4K); + last = round_down(last + 1, SZ_4K); + if (first >= last) + return; + shmem_uffd_enable(folio->mapping->host); + /* Revoke covering coarse aliases before their holes can be refilled. */ + unmap_mapping_subpage_hole(folio->mapping, first, last); + /* Pair with fast GUP's pin-before-PTE-recheck barrier. */ + smp_mb(); + pinned = folio_maybe_dma_pinned(folio); + offset = first - folio_pos(folio); + length = last - first; + while (length) { + unsigned int low = offset & ~PAGE_MASK; + unsigned long nr = min(length, PAGE_SIZE - low); + struct page *page = folio_page(folio, offset >> PAGE_SHIFT); + unsigned long mask = shmem_uffd_mask(low, nr); + unsigned long missing = mm_subpage_shmem_missing(page); + + /* + * A pin survives PTE removal and can modify retained backing. + * Preserve native partial-hole semantics: zero it, but do not + * let a missing-page COPY overwrite a subsequent device write. + * Pins have folio granularity, so exposure is conservative when + * only a neighbouring range is pinned. Full-folio removal still + * detaches old pins from the file's new cache identity. + */ + mm_subpage_shmem_set_missing(page, pinned ? missing & ~mask : missing | mask); + offset += nr; + length -= nr; + } +} +#else +static void shmem_uffd_cache_init(struct folio *folio, bool from_swap) { } +static int shmem_uffd_swap_prepare(struct folio *folio) { return 0; } +static void shmem_uffd_partial_hole(struct folio *folio, loff_t start, uoff_t end) { } +#endif + /* * Somewhat like filemap_add_folio, but error if expected item has gone. */ @@ -922,6 +1185,7 @@ int shmem_add_to_page_cache(struct folio *folio, xas_store(&xas, folio); if (xas_error(&xas)) goto unlock; + shmem_uffd_cache_init(folio, expected != NULL); shmem_update_stats(folio, nr); mapping->nrpages += nr; unlock: @@ -976,8 +1240,10 @@ static long shmem_free_swap(struct address_space *mapping, base = round_down(xas.xa_index, nr_pages); if (base < index || base + nr_pages - 1 > end) nr_pages = 0; - else + else { + shmem_uffd_forget(mapping, base, nr_pages); xas_store(&xas, NULL); + } } xas_unlock_irq(&xas); @@ -1020,6 +1286,39 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping, return swapped << PAGE_SHIFT; } +/* Byte-clipped variant for mappings smaller than a native cache page. */ +unsigned long shmem_partial_swap_usage_bytes(struct address_space *mapping, + u64 start, u64 end) +{ + XA_STATE(xas, &mapping->i_pages, start >> PAGE_SHIFT); + void *entry; + unsigned long swapped = 0, last; + + if (start >= end) + return 0; + last = (end - 1) >> PAGE_SHIFT; + rcu_read_lock(); + xas_for_each(&xas, entry, last) { + if (xas_retry(&xas, entry)) + continue; + if (xa_is_value(entry)) { + unsigned int order = xas_get_order(&xas); + u64 first = ((u64)xas.xa_index & ~((1UL << order) - 1)) << PAGE_SHIFT; + u64 stop = first + ((u64)PAGE_SIZE << order); + + swapped += min(stop, end) - max(first, start); + } + if (xas.xa_index == last) + break; + if (need_resched()) { + xas_pause(&xas); + cond_resched_rcu(); + } + } + rcu_read_unlock(); + return swapped; +} + /* * Determine (in bytes) how many of the shmem object's pages mapped by the * given vma is swapped out. @@ -1044,6 +1343,12 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma) */ if (!swapped) return 0; + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) { + u64 start = ((u64)vma->vm_pgoff << PAGE_SHIFT) + vma_subpage_offset(vma); + + return shmem_partial_swap_usage_bytes(mapping, start, + start + vma->vm_end - vma->vm_start); + } if (!vma->vm_pgoff && vma->vm_end - vma->vm_start >= inode->i_size) return swapped << PAGE_SHIFT; @@ -1101,6 +1406,12 @@ static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) return folio; } +static bool shmem_truncate_partial_folio(struct folio *folio, loff_t start, uoff_t end) +{ + shmem_uffd_partial_hole(folio, start, end); + return truncate_inode_partial_folio(folio, start, end); +} + /* * Remove range of pages and swap entries from page cache, and free them. * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate. @@ -1164,7 +1475,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (folio) { same_folio = lend < folio_next_pos(folio); folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) { + if (!shmem_truncate_partial_folio(folio, lstart, lend)) { start = folio_next_index(folio); if (same_folio) end = folio->index; @@ -1178,7 +1489,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, folio = shmem_get_partial_folio(inode, lend >> PAGE_SHIFT); if (folio) { folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) + if (!shmem_truncate_partial_folio(folio, lstart, lend)) end = folio->index; folio_unlock(folio); folio_put(folio); @@ -1247,7 +1558,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (!folio_test_large(folio)) { truncate_inode_folio(mapping, folio); - } else if (truncate_inode_partial_folio(folio, lstart, lend)) { + } else if (shmem_truncate_partial_folio(folio, lstart, lend)) { /* * If we split a page, reset the loop so * that we pick up the new sub pages. @@ -1355,6 +1666,7 @@ static int shmem_setattr(struct mnt_idmap *idmap, } if (newsize <= oldsize) { loff_t holebegin = round_up(newsize, PAGE_SIZE); + unmap_mapping_subpage_tail(inode->i_mapping, newsize); if (oldsize > holebegin) unmap_mapping_range(inode->i_mapping, holebegin, 0, 1); @@ -1362,6 +1674,7 @@ static int shmem_setattr(struct mnt_idmap *idmap, shmem_truncate_range(inode, newsize, (loff_t)-1); /* unmap again to remove racily COWed private pages */ + unmap_mapping_subpage_tail(inode->i_mapping, newsize); if (oldsize > holebegin) unmap_mapping_range(inode->i_mapping, holebegin, 0, 1); @@ -1677,6 +1990,8 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug, folio_mark_uptodate(folio); } + if (shmem_uffd_swap_prepare(folio)) + goto redirty; if (!folio_alloc_swap(folio)) { bool first_swapped = shmem_recalc_inode(inode, 0, nr_pages); int error; @@ -1730,6 +2045,10 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug, swap_cache_del_folio(folio); goto redirty; } + /* Allocation failed: no swap entry owns the reserved inode metadata. */ + xa_lock_irq(&mapping->i_pages); + shmem_uffd_forget(mapping, folio->index, folio_nr_pages(folio)); + xa_unlock_irq(&mapping->i_pages); if (nr_pages > 1) goto try_split; redirty: @@ -1934,6 +2253,23 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, int order, return folio; } +/* Allocate unexposed promotion backing with the inode's zone/NUMA policy. */ +struct folio *shmem_alloc_collapse_folio(struct mm_struct *mm, struct inode *inode, + pgoff_t index, unsigned int order) +{ + gfp_t gfp = limit_gfp_mask(GFP_TRANSHUGE, mapping_gfp_mask(inode->i_mapping)); + struct folio *folio; + + folio = shmem_alloc_folio(gfp, order, SHMEM_I(inode), index); + if (!folio) + return ERR_PTR(-ENOMEM); + if (mem_cgroup_charge(folio, mm, gfp)) { + folio_put(folio); + return ERR_PTR(-ENOMEM); + } + return folio; +} + static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, gfp_t gfp, struct inode *inode, pgoff_t index, struct mm_struct *fault_mm, unsigned long orders) @@ -2484,13 +2820,36 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm : NULL; folio = filemap_get_entry(inode->i_mapping, index); - if (folio && vma && userfaultfd_minor(vma)) { + if (folio && vma && userfaultfd_minor(vma) && + !(READ_ONCE(SHMEM_I(inode)->flags) & SHMEM_F_UFFD_SUBPAGE)) { if (!xa_is_value(folio)) folio_put(folio); *fault_type = handle_userfault(vmf, VM_UFFD_MINOR); return 0; } +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + if (xa_is_value(folio) && vma && userfaultfd_armed(vma) && + (READ_ONCE(SHMEM_I(inode)->flags) & SHMEM_F_UFFD_SUBPAGE)) { + unsigned long reason; + void *partial; + + /* Cache identity and its persisted mask must describe the same entry. */ + xa_lock_irq(&inode->i_mapping->i_pages); + if (xa_load(&inode->i_mapping->i_pages, index) != folio) { + xa_unlock_irq(&inode->i_mapping->i_pages); + goto repeat; + } + partial = xa_load(&SHMEM_I(inode)->uffd_missing, index); + reason = shmem_uffd_reason(vmf, partial ? xa_to_value(partial) : 0); + xa_unlock_irq(&inode->i_mapping->i_pages); + if (reason) { + *fault_type = handle_userfault(vmf, reason); + return 0; + } + } +#endif + if (xa_is_value(folio)) { error = shmem_swapin_folio(inode, index, &folio, sgp, gfp, vma, fault_type); @@ -2510,6 +2869,20 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, folio_put(folio); goto repeat; } +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + if (vma && userfaultfd_armed(vma) && + (READ_ONCE(SHMEM_I(inode)->flags) & SHMEM_F_UFFD_SUBPAGE)) { + unsigned long reason = shmem_uffd_reason(vmf, + shmem_uffd_missing_mask(folio, index)); + + if (reason) { + folio_unlock(folio); + folio_put(folio); + *fault_type = handle_userfault(vmf, reason); + return 0; + } + } +#endif if (sgp == SGP_WRITE) folio_mark_accessed(folio); if (folio_test_uptodate(folio)) @@ -2748,15 +3121,33 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) struct folio *folio = NULL; vm_fault_t ret = 0; int err; +#ifdef CONFIG_MM_SUBPAGE + bool promoted = false; + unsigned long user_size = mm_page_size(vmf->vma->vm_mm); + + /* Missing coarse PTEs require a blocking caller to prepare backing. */ + if (user_size > PAGE_SIZE && (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) + return VM_FAULT_RETRY; +#endif +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vmf->vma->vm_mm) < PAGE_SIZE && + !vma_file_offset_valid(vmf->vma, vmf->address, i_size_read(inode))) + return VM_FAULT_SIGBUS; +#endif + +#ifdef CONFIG_MM_SUBPAGE +retry: +#endif /* * Trinity finds that probing a hole which tmpfs is punching can * prevent the hole-punch from ever completing: noted in i_private. */ if (unlikely(inode->i_private)) { - ret = shmem_falloc_wait(vmf, inode); - if (ret) - return ret; + vm_fault_t wait = shmem_falloc_wait(vmf, inode); + + if (wait) + return ret | wait; } WARN_ON_ONCE(vmf->page != NULL); @@ -2765,6 +3156,45 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) if (err) return vmf_error(err); if (folio) { +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vmf->vma->vm_mm) < PAGE_SIZE && + !vma_file_offset_valid(vmf->vma, vmf->address, i_size_read(inode))) { + folio_unlock(folio); + folio_put(folio); + return VM_FAULT_SIGBUS; + } + if (user_size > PAGE_SIZE) { + struct vm_page_offset pos = vma_page_offset_at(vmf->vma, vmf->address); + unsigned long nr = user_size >> PAGE_SHIFT; + + if (pos.offset || !IS_ALIGNED(pos.index, nr)) { + folio_unlock(folio); + folio_put(folio); + return VM_FAULT_SIGBUS; + } + if (folio_size(folio) < user_size) { + folio_unlock(folio); + folio_put(folio); + folio = NULL; + /* Bound work if splitting races a successful promotion. */ + if (promoted) { + cond_resched(); + return ret | VM_FAULT_NOPAGE; + } + err = collapse_file_user_page(vmf->vma->vm_mm, + vmf->vma->vm_file, pos.index); + if (err == -EAGAIN) { + cond_resched(); + return ret | VM_FAULT_NOPAGE; + } + if (err) + return vmf_error(err); + promoted = true; + goto retry; + } + } +#endif + vmf->page = folio_file_page(folio, vmf->pgoff); ret |= VM_FAULT_LOCKED; } @@ -2899,7 +3329,7 @@ static struct mempolicy *shmem_get_policy(struct vm_area_struct *vma, * by page order, as in shmem_get_pgoff_policy() and get_vma_policy()). */ *ilx = inode->i_ino; - index = ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff; + index = vma_page_offset_at(vma, addr).index; return mpol_shared_policy_lookup(&SHMEM_I(inode)->policy, index); } @@ -3086,6 +3516,9 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap, memset(info, 0, (char *)inode - (char *)info); INIT_LIST_HEAD_RCU(&info->xattrs); spin_lock_init(&info->lock); +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + xa_init_flags(&info->uffd_missing, XA_FLAGS_ACCOUNT); +#endif atomic_set(&info->stop_eviction, 0); info->seals = F_SEAL_SEAL; info->flags = vma_flags_test(&flags, VMA_NORESERVE_BIT) @@ -3245,6 +3678,57 @@ static void shmem_mfill_filemap_remove(struct folio *folio, folio_unlock(folio); } +#ifdef CONFIG_MM_SUBPAGE +struct folio *shmem_uffd_prepare_folio(struct vm_area_struct *vma, + unsigned long address, bool *new) +{ + struct inode *inode = file_inode(vma->vm_file); + pgoff_t index = vma_page_offset_at(vma, address).index; + struct folio *folio; + int err; + + *new = false; + /* Must precede cache insertion: minor faults must wait for initialization. */ + shmem_uffd_enable(inode); +retry: + err = shmem_get_folio(inode, index, 0, &folio, SGP_NOALLOC); + if (!err) + return folio; + if (err != -ENOENT) + return ERR_PTR(err); + folio = shmem_mfill_folio_alloc(vma, address); + if (!folio) + return ERR_PTR(-ENOMEM); + clear_highpage(&folio->page); + flush_dcache_folio(folio); + __folio_mark_uptodate(folio); + err = shmem_mfill_filemap_add(folio, vma, address); + if (err) { + folio_put(folio); + if (err == -EEXIST) + goto retry; + return ERR_PTR(err); + } + err = shmem_uffd_set_missing(folio, index, SHMEM_UFFD_ALL_UNITS); + if (err) { + shmem_mfill_filemap_remove(folio, vma); + folio_put(folio); + return ERR_PTR(err); + } + *new = true; + return folio; +} + +void shmem_uffd_abort_folio(struct folio *folio, struct vm_area_struct *vma, bool new) +{ + if (new) + shmem_mfill_filemap_remove(folio, vma); + else + folio_unlock(folio); + folio_put(folio); +} +#endif + static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) { struct folio *folio; @@ -3331,6 +3815,7 @@ shmem_write_end(const struct kiocb *iocb, struct address_space *mapping, } folio_mark_uptodate(folio); } + shmem_uffd_mark_mapped(folio, offset_in_folio(folio, pos), copied); folio_mark_dirty(folio); folio_unlock(folio); folio_put(folio); @@ -5143,6 +5628,10 @@ static struct inode *shmem_alloc_inode(struct super_block *sb) info = alloc_inode_sb(sb, shmem_inode_cachep, GFP_KERNEL); if (!info) return NULL; +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + /* Also cover inode-init failure before __shmem_get_inode() runs. */ + xa_init_flags(&info->uffd_missing, XA_FLAGS_ACCOUNT); +#endif return &info->vfs_inode; } @@ -5155,6 +5644,9 @@ static void shmem_free_in_core_inode(struct inode *inode) static void shmem_destroy_inode(struct inode *inode) { +#if defined(CONFIG_SHMEM) && defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_USERFAULTFD) + xa_destroy(&SHMEM_I(inode)->uffd_missing); +#endif if (S_ISREG(inode->i_mode)) mpol_free_shared_policy(&SHMEM_I(inode)->policy); if (S_ISDIR(inode->i_mode)) @@ -5972,6 +6464,8 @@ struct folio *shmem_read_folio_gfp(struct address_space *mapping, if (error) return ERR_PTR(error); + /* A kernel caller receives and may modify the complete native folio. */ + shmem_uffd_mark_mapped(folio, 0, folio_size(folio)); folio_unlock(folio); return folio; #else diff --git a/mm/swap.h b/mm/swap.h index a77016f2423b90..5d064cab36f4c5 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -10,9 +10,12 @@ extern int page_cluster; #ifdef CONFIG_THP_SWAP #define SWAPFILE_CLUSTER HPAGE_PMD_NR -#define swap_entry_order(order) (order) #else #define SWAPFILE_CLUSTER 256 +#endif +#if defined(CONFIG_THP_SWAP) || defined(CONFIG_MM_SUBPAGE) +#define swap_entry_order(order) (order) +#else #define swap_entry_order(order) 0 #endif @@ -38,6 +41,9 @@ struct swap_cluster_info { u8 order; atomic_long_t __rcu *table; /* Swap table entries, see mm/swap_table.h */ unsigned int *extend_table; /* For large swap count, protected by ci->lock */ +#ifdef CONFIG_MM_SUBPAGE + unsigned int *subpage_counts; /* Per-quarter PTE refs, protected by ci->lock */ +#endif struct list_head list; }; @@ -207,6 +213,8 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp); */ int folio_alloc_swap(struct folio *folio); int folio_dup_swap(struct folio *folio, struct page *subpage); +int folio_dup_swap_range(struct folio *folio, struct page *page, unsigned int nr); +void folio_put_swap_range(struct folio *folio, struct page *page, unsigned int nr); void folio_put_swap(struct folio *folio, struct page *subpage); /* For internal use */ @@ -216,6 +224,7 @@ extern void __swap_cluster_free_entries(struct swap_info_struct *si, /* linux/mm/page_io.c */ int sio_pool_init(void); +int swap_extent_io_init(void); struct swap_iocb; void swap_read_folio(struct folio *folio, struct swap_iocb **plug); void __swap_read_unplug(struct swap_iocb *plug); @@ -399,6 +408,17 @@ static inline void folio_put_swap(struct folio *folio, struct page *page) { } +static inline int folio_dup_swap_range(struct folio *folio, struct page *page, + unsigned int nr) +{ + return -EINVAL; +} + +static inline void folio_put_swap_range(struct folio *folio, struct page *page, + unsigned int nr) +{ +} + static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) { } diff --git a/mm/swap_state.c b/mm/swap_state.c index 1415a5c54a43b2..db1285bc1880dd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -427,6 +427,10 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_large(folio))) return; + /* Small-granule PTEs currently use native swap-cluster readahead. */ + if (vma && mm_page_size(vma->vm_mm) != PAGE_SIZE) + vma = NULL; + readahead = folio_test_clear_readahead(folio); if (vma && vma_ra) { unsigned long ra_val; @@ -896,7 +900,7 @@ struct folio *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, struct folio *folio; mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = swap_use_vma_readahead() ? + folio = swap_use_vma_readahead() && mm_page_size(vmf->vma->vm_mm) == PAGE_SIZE ? swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : swap_cluster_readahead(entry, gfp_mask, mpol, ilx); mpol_cond_put(mpol); diff --git a/mm/swapfile.c b/mm/swapfile.c index 74a1e324449dc6..ca11d95c2d0371 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -329,17 +330,22 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset) BUG(); } +/* The extent tree is immutable while the swap entry is owned. */ +sector_t swap_extent_sector(swp_entry_t entry, unsigned long *nr_pages) +{ + struct swap_info_struct *sis = __swap_entry_to_info(entry); + pgoff_t offset = swp_offset(entry); + struct swap_extent *se = offset_to_swap_extent(sis, offset); + + *nr_pages = min(*nr_pages, se->start_page + se->nr_pages - offset); + return (se->start_block + offset - se->start_page) << (PAGE_SHIFT - 9); +} + sector_t swap_folio_sector(struct folio *folio) { - struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); - struct swap_extent *se; - sector_t sector; - pgoff_t offset; + unsigned long nr = folio_nr_pages(folio); - offset = swp_offset(folio->swap); - se = offset_to_swap_extent(sis, offset); - sector = se->start_block + (offset - se->start_page); - return sector << (PAGE_SHIFT - 9); + return swap_extent_sector(folio->swap, &nr); } /* @@ -479,6 +485,16 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci) lockdep_assert_held(&ci->lock); table = (void *)rcu_dereference_protected(ci->table, true); rcu_assign_pointer(ci->table, NULL); +#ifdef CONFIG_MM_SUBPAGE + if (ci->subpage_counts) { + unsigned int i; + + for (i = 0; i < SWAPFILE_CLUSTER * (PAGE_SIZE / MM_SUBPAGE_SIZE); i++) + VM_WARN_ON_ONCE(ci->subpage_counts[i]); + kfree(ci->subpage_counts); + ci->subpage_counts = NULL; + } +#endif swap_table_free(table); } @@ -1067,22 +1083,32 @@ static void swap_reclaim_work(struct work_struct *work) swap_reclaim_full_clusters(si, true); } +static bool swap_folio_alloc_allowed(struct swap_info_struct *si, struct folio *folio, + bool mandatory) +{ + unsigned int flags; + + if (!folio || !folio_order(folio)) + return true; + flags = READ_ONCE(si->flags); + if (flags & SWP_BLKDEV) + return true; + /* Filesystem swap_rw implementations retain their existing contract. */ + return si->bdev && !(flags & SWP_FS_OPS) && mandatory; +} + /* * Try to allocate swap entries with specified order and try set a new * cluster for current CPU too. */ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, - struct folio *folio) + struct folio *folio, bool mandatory) { struct swap_cluster_info *ci; unsigned int order = likely(folio) ? folio_order(folio) : 0; unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; - /* - * Swapfile is not block device so unable - * to allocate large entries. - */ - if (order && !(si->flags & SWP_BLKDEV)) + if (!swap_folio_alloc_allowed(si, folio, mandatory)) return 0; if (!(si->flags & SWP_SOLIDSTATE)) { @@ -1346,7 +1372,7 @@ static bool get_swap_device_info(struct swap_info_struct *si) * Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static bool swap_alloc_fast(struct folio *folio) +static bool swap_alloc_fast(struct folio *folio, bool mandatory) { unsigned int order = folio_order(folio); struct swap_cluster_info *ci; @@ -1361,6 +1387,10 @@ static bool swap_alloc_fast(struct folio *folio) offset = this_cpu_read(percpu_swap_cluster.offset[order]); if (!si || !offset || !get_swap_device_info(si)) return false; + if (!swap_folio_alloc_allowed(si, folio, mandatory)) { + put_swap_device(si); + return false; + } ci = swap_cluster_lock(si, offset); if (cluster_is_usable(ci, order)) { @@ -1376,7 +1406,7 @@ static bool swap_alloc_fast(struct folio *folio) } /* Rotate the device and switch to a new cluster */ -static void swap_alloc_slow(struct folio *folio) +static void swap_alloc_slow(struct folio *folio, bool mandatory) { struct swap_info_struct *si, *next; @@ -1387,7 +1417,7 @@ static void swap_alloc_slow(struct folio *folio) plist_requeue(&si->avail_list, &swap_avail_head); spin_unlock(&swap_avail_lock); if (get_swap_device_info(si)) { - cluster_alloc_swap_entry(si, folio); + cluster_alloc_swap_entry(si, folio, mandatory); put_swap_device(si); if (folio_test_swapcache(folio)) return; @@ -1543,7 +1573,7 @@ static void __swap_cluster_put_entry(struct swap_cluster_info *ci, */ static void swap_put_entries_cluster(struct swap_info_struct *si, pgoff_t offset, int nr, - bool reclaim_cache) + bool reclaim_cache, int subpage) { struct swap_cluster_info *ci; unsigned int ci_off, ci_end; @@ -1556,6 +1586,17 @@ static void swap_put_entries_cluster(struct swap_info_struct *si, ci = swap_cluster_lock(si, offset); ci_off = offset % SWAPFILE_CLUSTER; ci_end = ci_off + nr; +#ifdef CONFIG_MM_SUBPAGE + if (subpage >= 0) { + unsigned int index = ci_off * (PAGE_SIZE / MM_SUBPAGE_SIZE) + subpage; + + if (WARN_ON_ONCE(nr != 1 || !ci->subpage_counts || !ci->subpage_counts[index])) { + swap_cluster_unlock(ci); + return; + } + ci->subpage_counts[index]--; + } +#endif do { swp_tb = __swap_table_get(ci, ci_off); if (swp_tb_get_count(swp_tb) == 1) { @@ -1649,7 +1690,7 @@ static int __swap_cluster_dup_entry(struct swap_cluster_info *ci, * and failed to allocate an extended table, -EINVAL if any entry is bad entry. */ static int swap_dup_entries_cluster(struct swap_info_struct *si, - pgoff_t offset, int nr) + pgoff_t offset, int nr, int subpage) { int err; struct swap_cluster_info *ci; @@ -1659,6 +1700,25 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si, ci_end = ci_start + nr; ci_off = ci_start; ci = swap_cluster_lock(si, offset); +#ifdef CONFIG_MM_SUBPAGE + if (subpage >= 0) { + unsigned int index = ci_start * (PAGE_SIZE / MM_SUBPAGE_SIZE) + subpage; + + VM_BUG_ON(nr != 1); + if (!ci->subpage_counts) + ci->subpage_counts = kcalloc(SWAPFILE_CLUSTER, + (PAGE_SIZE / MM_SUBPAGE_SIZE) * sizeof(*ci->subpage_counts), + GFP_ATOMIC | __GFP_NOWARN); + if (!ci->subpage_counts) { + err = -ENOMEM; + goto failed; + } + if (ci->subpage_counts[index] == UINT_MAX) { + err = -EOVERFLOW; + goto failed; + } + } +#endif restart: do { err = __swap_cluster_dup_entry(ci, ci_off); @@ -1673,6 +1733,10 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si, goto failed; } } while (++ci_off < ci_end); +#ifdef CONFIG_MM_SUBPAGE + if (subpage >= 0) + ci->subpage_counts[ci_start * (PAGE_SIZE / MM_SUBPAGE_SIZE) + subpage]++; +#endif swap_cluster_unlock(ci); return 0; failed: @@ -1683,6 +1747,132 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si, return err; } +#ifdef CONFIG_MM_SUBPAGE +static bool swap_subpage_offset_valid(unsigned int offset) +{ + return offset < PAGE_SIZE && IS_ALIGNED(offset, MM_SUBPAGE_SIZE); +} + +int swap_subpage_dup(swp_entry_t entry, unsigned int offset) +{ + struct swap_info_struct *si = swap_entry_to_info(entry); + + if (WARN_ON_ONCE(!si || !swap_subpage_offset_valid(offset))) + return -EINVAL; + /* A locked swap-cache folio also permits the initial zero-to-one ref. */ + return swap_dup_entries_cluster(si, swp_offset(entry), 1, offset >> MM_SUBPAGE_SHIFT); +} + +void swap_subpage_put(swp_entry_t entry, unsigned int offset, bool reclaim_cache) +{ + struct swap_info_struct *si; + + if (WARN_ON_ONCE(!swap_subpage_offset_valid(offset))) + return; + si = get_swap_device(entry); + if (WARN_ON_ONCE(!si)) + return; + swap_put_entries_cluster(si, swp_offset(entry), 1, reclaim_cache, + offset >> MM_SUBPAGE_SHIFT); + put_swap_device(si); +} + +unsigned int swap_subpage_count(swp_entry_t entry, unsigned int offset) +{ + struct swap_cluster_info *ci; + struct swap_info_struct *si; + unsigned int count = 0; + + if (!swap_subpage_offset_valid(offset)) + return 0; + si = get_swap_device(entry); + if (!si) + return 0; + ci = swap_cluster_lock(si, swp_offset(entry)); + if (ci->subpage_counts) + count = ci->subpage_counts[swp_cluster_offset(entry) * + (PAGE_SIZE / MM_SUBPAGE_SIZE) + (offset >> MM_SUBPAGE_SHIFT)]; + swap_cluster_unlock(ci); + put_swap_device(si); + return count; +} +#endif + +/* + * Optional THPs can split if their order-specific cluster lists have no room. + * A mandatory base user page cannot: find a free aligned group without keeping + * the per-CPU or device allocation lock held over a device-wide scan. + */ +static void swap_alloc_required_from_device(struct swap_info_struct *si, + struct folio *folio) +{ + unsigned int order = folio_order(folio), nr = folio_nr_pages(folio); + unsigned long start; + + if (!swap_folio_alloc_allowed(si, folio, true)) + return; + for (start = 0; start < si->max; start += SWAPFILE_CLUSTER) { + struct swap_cluster_info *ci = __swap_offset_to_cluster(si, start); + unsigned long offset, end = min(start + SWAPFILE_CLUSTER, si->max); + bool found = false; + + if (!(READ_ONCE(si->flags) & SWP_WRITEOK)) + break; + if (!spin_trylock(&ci->lock)) + goto next; + /* Order zero's usability test permits occupied clusters of any order. */ + if (!cluster_is_usable(ci, 0) || ci->count + nr > SWAPFILE_CLUSTER) + goto unlock; + for (offset = start; offset + nr <= end; offset += nr) { + bool need_reclaim = false; + + /* Only genuinely free slots qualify; retain every existing owner. */ + if (!cluster_scan_range(si, ci, offset, nr, &need_reclaim) || need_reclaim) + continue; + if (!__swap_cluster_alloc_entries(si, ci, folio, offset - start)) + break; + ci->order = order; + if (ci->count < SWAPFILE_CLUSTER && ci->flags == CLUSTER_FLAG_FRAG) { + /* Same flag, different order: relocate the list explicitly. */ + spin_lock(&si->lock); + list_move_tail(&ci->list, &si->frag_clusters[order]); + spin_unlock(&si->lock); + } else { + relocate_cluster(si, ci); + } + found = true; + break; + } +unlock: + swap_cluster_unlock(ci); + if (found) + return; +next: + cond_resched(); + } +} + +static void swap_alloc_required(struct folio *folio) +{ + struct swap_info_struct *devices[MAX_SWAPFILES], *si; + unsigned int count = 0, i; + + if (get_nr_swap_pages() < folio_nr_pages(folio)) + return; + /* Pin a bounded snapshot in the allocator's existing priority order. */ + spin_lock(&swap_avail_lock); + plist_for_each_entry(si, &swap_avail_head, avail_list) { + if (get_swap_device_info(si)) + devices[count++] = si; + } + spin_unlock(&swap_avail_lock); + for (i = 0; i < count; i++) { + if (!folio_test_swapcache(folio)) + swap_alloc_required_from_device(devices[i], folio); + put_swap_device(devices[i]); + } +} + /** * folio_alloc_swap - allocate swap space for a folio * @folio: folio we want to move to swap @@ -1697,16 +1887,19 @@ int folio_alloc_swap(struct folio *folio) { unsigned int order = folio_order(folio); unsigned int size = 1 << order; + bool mandatory; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); + /* The temporary anon-vma reference may require sleepable final release. */ + mandatory = order && folio_anon_min_order(folio); if (order) { /* - * Reject large allocation when THP_SWAP is disabled, - * the caller should split the folio and try again. + * Optional THP backing may split when THP_SWAP is disabled. + * A larger base user page must retain contiguous swap ownership. */ - if (!IS_ENABLED(CONFIG_THP_SWAP)) + if (!IS_ENABLED(CONFIG_THP_SWAP) && !mandatory) return -EAGAIN; /* @@ -1721,10 +1914,13 @@ int folio_alloc_swap(struct folio *folio) again: local_lock(&percpu_swap_cluster.lock); - if (!swap_alloc_fast(folio)) - swap_alloc_slow(folio); + if (!swap_alloc_fast(folio, mandatory)) + swap_alloc_slow(folio, mandatory); local_unlock(&percpu_swap_cluster.lock); + if (mandatory && unlikely(!folio_test_swapcache(folio))) + swap_alloc_required(folio); + if (!order && unlikely(!folio_test_swapcache(folio))) { if (swap_sync_discard()) goto again; @@ -1757,6 +1953,24 @@ int folio_alloc_swap(struct folio *folio) * swap_put_entries_direct on its swap entry before this helper returns, or * the swap count may underflow. */ +int folio_dup_swap_range(struct folio *folio, struct page *page, unsigned int nr) +{ + swp_entry_t entry = page_swap_entry(page); + + VM_BUG_ON_FOLIO(!folio_test_locked(folio) || !folio_test_swapcache(folio), folio); + VM_BUG_ON_FOLIO(!nr || folio_page_idx(folio, page) + nr > folio_nr_pages(folio), folio); + return swap_dup_entries_cluster(swap_entry_to_info(entry), swp_offset(entry), nr, -1); +} + +void folio_put_swap_range(struct folio *folio, struct page *page, unsigned int nr) +{ + swp_entry_t entry = page_swap_entry(page); + + VM_BUG_ON_FOLIO(!folio_test_locked(folio) || !folio_test_swapcache(folio), folio); + VM_BUG_ON_FOLIO(!nr || folio_page_idx(folio, page) + nr > folio_nr_pages(folio), folio); + swap_put_entries_cluster(__swap_entry_to_info(entry), swp_offset(entry), nr, false, -1); +} + int folio_dup_swap(struct folio *folio, struct page *subpage) { swp_entry_t entry = folio->swap; @@ -1771,7 +1985,7 @@ int folio_dup_swap(struct folio *folio, struct page *subpage) } return swap_dup_entries_cluster(swap_entry_to_info(entry), - swp_offset(entry), nr_pages); + swp_offset(entry), nr_pages, -1); } /** @@ -1797,7 +2011,7 @@ void folio_put_swap(struct folio *folio, struct page *subpage) nr_pages = 1; } - swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false); + swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false, -1); } /* @@ -2070,7 +2284,7 @@ void swap_put_entries_direct(swp_entry_t entry, int nr) offset = start_offset; do { cluster_end = min(round_up(offset + 1, SWAPFILE_CLUSTER), end_offset); - swap_put_entries_cluster(si, offset, cluster_end - offset, true); + swap_put_entries_cluster(si, offset, cluster_end - offset, true, -1); offset = cluster_end; } while (offset < end_offset); out: @@ -2107,7 +2321,7 @@ swp_entry_t swap_alloc_hibernation_slot(int type) swap_cluster_unlock(ci); } if (!offset) - offset = cluster_alloc_swap_entry(si, NULL); + offset = cluster_alloc_swap_entry(si, NULL, false); local_unlock(&percpu_swap_cluster.lock); if (offset) entry = swp_entry(si->type, offset); @@ -2239,14 +2453,97 @@ static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte) return pte_same(pte_swp_clear_flags(pte), swp_pte); } +#ifdef CONFIG_MM_SUBPAGE +/* Caller holds the swap-cache folio lock and the VMA's mmap read lock. */ +static int unuse_subpage(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, swp_entry_t entry, struct folio *folio) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned int offset; + struct mm_subpage *slot = NULL; + spinlock_t *ptl; + pte_t *ptep, old, snapshot, pte; + bool poisoned = PageHWPoison(&folio->page) || !folio_test_uptodate(folio); + int ret = 0; + + if (folio_order(folio) || folio_test_ksm(folio)) + return -EOPNOTSUPP; + /* A swap PTE records the physical quarter, independently of the VMA. + * Snapshot under PTL, then allocate metadata while only folio is locked. + */ + ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!ptep) + return 0; + snapshot = ptep_get(ptep); + if (!softleaf_is_swap(softleaf_from_pte(snapshot)) || + softleaf_from_pte(snapshot).val != entry.val) { + pte_unmap_unlock(ptep, ptl); + return 0; + } + offset = pte_swp_subpage_offset(snapshot); + pte_unmap_unlock(ptep, ptl); + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, mm_page_size(mm))) + return -EINVAL; + if (!poisoned) { + arch_swap_restore(folio_swap(entry, folio), folio); + /* Allocate metadata before taking the PTE spinlock. */ + slot = mm_subpage_restore_granule(folio, offset, mm_page_shift(mm), GFP_KERNEL); + if (IS_ERR(slot)) + return PTR_ERR(slot) == -EAGAIN ? 0 : PTR_ERR(slot); + ret = mm_subpage_prepare_anon_rmap(slot, vma, addr); + if (ret) + goto out; + } + ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!ptep) + goto out; + old = ptep_get(ptep); + /* Includes quarter and flag changes while metadata was prepared. */ + if (!pte_same(old, snapshot)) + goto unlock; + if (poisoned) { + pte = swp_entry_to_pte(PageHWPoison(&folio->page) ? + make_hwpoison_entry(&folio->page) : make_poisoned_swp_entry()); + } else { + ret = mm_subpage_add_anon_rmap(slot, vma, addr); + if (ret) + goto unlock; + /* unuse_pte_range() waited for writeback and holds the folio lock. */ + VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio); + if (pte_swp_exclusive(old)) + mm_subpage_set_exclusive(slot); + else if (swap_subpage_count(entry, offset) == 1) + mm_subpage_try_reuse_swap(slot); + pte = pte_mkold(phys_pte_mm(mm, mm_subpage_phys(slot), vma->vm_page_prot)); + if (pte_swp_soft_dirty(old)) + pte = pte_mksoft_dirty(pte); + if (pte_swp_uffd_wp(old)) + pte = pte_mkuffd_wp(pte); + inc_mm_counter(mm, MM_ANONPAGES); + slot = NULL; + ret = 1; + } + dec_mm_counter(mm, MM_SWAPENTS); + set_pte_at(mm, addr, ptep, pte); + swap_subpage_put(entry, offset, false); +unlock: + pte_unmap_unlock(ptep, ptl); +out: + if (slot) + mm_subpage_put(slot); + return ret; +} +#endif + /* * No need to decide whether this PTE shares the swap entry with others, * just let do_wp_page work it out if a write is requested later - to * force COW, vm_page_prot omits write permission from any private vma. */ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long addr, swp_entry_t entry, struct folio *folio) + unsigned long addr, pte_t orig_pte, struct folio *folio) { + swp_entry_t entry = softleaf_from_pte(orig_pte); struct page *page; struct folio *swapcache; spinlock_t *ptl; @@ -2261,6 +2558,11 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (!folio_matches_swap_entry(folio, entry)) return 0; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) < PAGE_SIZE) + return unuse_subpage(vma, pmd, addr, entry, folio); +#endif + swapcache = folio; folio = ksm_might_need_to_copy(folio, vma, addr); if (unlikely(!folio)) @@ -2366,7 +2668,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_t ptent; if (!pte++) { - pte = pte_offset_map(pmd, addr); + pte = pte_offset_map_mm(vma->vm_mm, pmd, addr); if (!pte) break; } @@ -2382,6 +2684,22 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) > PAGE_SIZE) { + struct vm_fault vmf = { + .vma = vma, .address = addr, .real_address = addr, + .pmd = pmd, .orig_pte = ptent, + }; + vm_fault_t fault = do_swap_coarse_page(&vmf); + + if (fault & VM_FAULT_OOM) + return -ENOMEM; + if (fault & VM_FAULT_ERROR) + return -EIO; + continue; + } +#endif + folio = swap_cache_get_folio(entry); if (!folio) { struct vm_fault vmf = { @@ -2404,7 +2722,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio_lock(folio); folio_wait_writeback(folio); - ret = unuse_pte(vma, pmd, addr, entry, folio); + ret = unuse_pte(vma, pmd, addr, ptent, folio); if (ret < 0) { folio_unlock(folio); folio_put(folio); @@ -2414,7 +2732,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio_free_swap(folio); folio_unlock(folio); folio_put(folio); - } while (addr += PAGE_SIZE, addr != end); + } while (addr += mm_page_size(vma->vm_mm), addr != end); if (pte) pte_unmap(pte); @@ -2429,10 +2747,10 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long next; int ret; - pmd = pmd_offset(pud, addr); + pmd = pmd_offset_mm(vma->vm_mm, pud, addr); do { cond_resched(); - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(vma->vm_mm, addr, end); ret = unuse_pte_range(vma, pmd, addr, next, type); if (ret) return ret; @@ -2448,9 +2766,9 @@ static inline int unuse_pud_range(struct vm_area_struct *vma, p4d_t *p4d, unsigned long next; int ret; - pud = pud_offset(p4d, addr); + pud = pud_offset_mm(vma->vm_mm, p4d, addr); do { - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(vma->vm_mm, addr, end); if (pud_none_or_clear_bad(pud)) continue; ret = unuse_pmd_range(vma, pud, addr, next, type); @@ -2468,10 +2786,10 @@ static inline int unuse_p4d_range(struct vm_area_struct *vma, pgd_t *pgd, unsigned long next; int ret; - p4d = p4d_offset(pgd, addr); + p4d = p4d_offset_mm(vma->vm_mm, pgd, addr); do { - next = p4d_addr_end(addr, end); - if (p4d_none_or_clear_bad(p4d)) + next = p4d_addr_end_mm(vma->vm_mm, addr, end); + if (p4d_none_or_clear_bad_mm(vma->vm_mm, p4d)) continue; ret = unuse_pud_range(vma, p4d, addr, next, type); if (ret) @@ -2491,7 +2809,7 @@ static int unuse_vma(struct vm_area_struct *vma, unsigned int type) pgd = pgd_offset(vma->vm_mm, addr); do { - next = pgd_addr_end(addr, end); + next = pgd_addr_end_mm(vma->vm_mm, addr, end); if (pgd_none_or_clear_bad(pgd)) continue; ret = unuse_p4d_range(vma, pgd, addr, next, type); @@ -2798,10 +3116,15 @@ static int setup_swap_extents(struct swap_info_struct *sis, destroy_swap_extents(sis, swap_file); return -ENOMEM; } - return ret; + } else { + ret = generic_swapfile_activate(sis, swap_file, span); } - - return generic_swapfile_activate(sis, swap_file, span); + if (ret >= 0 && sis->bdev && + !(sis->flags & SWP_FS_OPS) && swap_extent_io_init()) { + destroy_swap_extents(sis, swap_file); + return -ENOMEM; + } + return ret; } static void _enable_swap_info(struct swap_info_struct *si) @@ -3708,7 +4031,7 @@ void si_swapinfo(struct sysinfo *val) * Also the swap entry must have a count >= 1. Otherwise folio_dup_swap should * be used. */ -int swap_dup_entry_direct(swp_entry_t entry) +int swap_dup_entries_direct(swp_entry_t entry, unsigned int nr) { struct swap_info_struct *si; @@ -3725,7 +4048,14 @@ int swap_dup_entry_direct(swp_entry_t entry) */ VM_WARN_ON_ONCE(!swap_entry_swapped(si, entry)); - return swap_dup_entries_cluster(si, swp_offset(entry), 1); + if (WARN_ON_ONCE(!nr || swp_cluster_offset(entry) + nr > SWAPFILE_CLUSTER)) + return -EINVAL; + return swap_dup_entries_cluster(si, swp_offset(entry), nr, -1); +} + +int swap_dup_entry_direct(swp_entry_t entry) +{ + return swap_dup_entries_direct(entry, 1); } #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) diff --git a/mm/truncate.c b/mm/truncate.c index 12cc89f89afcf7..ca1830b647a975 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -794,8 +794,10 @@ void truncate_pagecache(struct inode *inode, loff_t newsize) * truncate_inode_pages finishes, hence the second * unmap_mapping_range call must be made for correctness. */ + unmap_mapping_subpage_tail(mapping, newsize); unmap_mapping_range(mapping, holebegin, 0, 1); truncate_inode_pages(mapping, newsize); + unmap_mapping_subpage_tail(mapping, newsize); unmap_mapping_range(mapping, holebegin, 0, 1); } EXPORT_SYMBOL(truncate_pagecache); diff --git a/mm/user-subpage-internal.h b/mm/user-subpage-internal.h new file mode 100644 index 00000000000000..bddb5eeb2ca7be --- /dev/null +++ b/mm/user-subpage-internal.h @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _MM_USER_SUBPAGE_INTERNAL_H +#define _MM_USER_SUBPAGE_INTERNAL_H + +#include +#include +#include +#include + +struct mm_subpage_owner; +struct rmap_walk_control; +struct anon_vma; + +struct mm_subpage { + struct mm_subpage_owner *owner; + refcount_t refs; + atomic_t mapcount; + unsigned int index; + u8 shift; + unsigned int pins; + bool exclusive; + bool lazyfree; + bool rmap_bound; +}; + +/* Logical identity is independent of the physical slot after relocation. */ +struct mm_subpage_rmap { + struct anon_vma *root; + struct vm_page_offset pos; + unsigned int offset; +}; + +struct subpage_retired_root; +struct mm_subpage_move { + struct mm_subpage *slot; + struct mm_subpage_rmap map; + struct subpage_retired_root *retired; + bool swap; + bool root_changed; + bool committed; +}; + +/* Folio and both PTE locks stay held across prepare, commit and finish. */ +int mm_subpage_move_prepare(struct mm_subpage_move *move, struct mm_subpage *slot, + struct vm_area_struct *src, unsigned long src_address, + struct vm_area_struct *dst, unsigned long dst_address, bool swap); +/* Present source PTE must be cleared and its TLB invalidated before commit. */ +int mm_subpage_move_commit(struct mm_subpage_move *move); +void mm_subpage_move_finish(struct mm_subpage_move *move); + +int mm_subpage_bind_rmap(struct mm_subpage *slot, struct vm_area_struct *vma, + unsigned long address); +bool mm_subpage_match_rmap(struct mm_subpage *slot, struct vm_area_struct *vma, + unsigned long address); +/* Return -1 for ordinary linear rmap; otherwise return referenced identities. */ +int mm_subpage_snapshot_rmap(struct folio *folio, struct mm_subpage_rmap *maps); +bool mm_subpage_address_in_vma(const struct folio *folio, + const struct vm_area_struct *vma, unsigned long *address); +bool rmap_walk_subpages(struct folio *folio, struct rmap_walk_control *rwc, + bool locked); + +struct anon_vma *mm_subpage_anon_root(const struct mm_subpage *subpage); + +/* Folio lock held; called once before publishing anonymous folio mapping. */ +void mm_subpage_hold_anon_root(struct mm_subpage *subpage, struct anon_vma *root, + bool droppable); +bool mm_subpage_is_droppable(const struct mm_subpage *subpage); + +#endif diff --git a/mm/user-subpage.c b/mm/user-subpage.c new file mode 100644 index 00000000000000..c9101990099df9 --- /dev/null +++ b/mm/user-subpage.c @@ -0,0 +1,2256 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Reference-counted user data slots in native folios. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "user-subpage-internal.h" +#include "internal.h" + +#define SUBPAGES_PER_FOLIO (PAGE_SIZE / MM_SUBPAGE_SIZE) +static bool subpage_shift_valid(unsigned int shift) +{ + return shift >= MM_SUBPAGE_SHIFT && shift < PAGE_SHIFT; +} + +static unsigned long subpage_mask(unsigned int shift) +{ + return GENMASK((PAGE_SIZE >> shift) - 1, 0); +} + +static unsigned long offset_mask(unsigned int shift, unsigned int offset) +{ + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, 1U << shift)) + return 0; + return BIT(offset >> shift); +} + +struct mm_subpage_owner { + struct rcu_head rcu; + struct work_struct free_work; + struct mm_subpage_pool *pool; + struct folio *folio; + struct anon_vma *anon_root; + struct mm_subpage_owner *migration_target; + /* Allocated only when slots need independent logical identities. */ + struct mm_subpage_rmap *rmap; + struct list_head all; + struct list_head available; + unsigned long used; + unsigned long zeroed; + u8 shift; + bool detached; + bool migrating; + bool droppable; + struct mm_subpage slots[SUBPAGES_PER_FOLIO]; +}; + +struct mm_subpage_ext { + struct mm_subpage_owner __rcu *owner; + atomic_t file_maps[SUBPAGES_PER_FOLIO]; + atomic_t file_leaves; + /* Folio lock protects validity; fits the 16K/64K layout padding. */ + u16 shmem_missing; +}; +static_assert(SUBPAGES_PER_FOLIO == 1 || sizeof(struct mm_subpage_ext) == + ALIGN(sizeof(void *) + sizeof(atomic_t) * (SUBPAGES_PER_FOLIO + 1), + sizeof(void *))); +static_assert(SUBPAGES_PER_FOLIO <= sizeof(u16) * BITS_PER_BYTE); + +unsigned long mm_subpage_shmem_missing(struct page *page) +{ + struct page_ext *ext = page_ext_get(page); + struct mm_subpage_ext *subext; + unsigned long missing; + + /* A referenced native page keeps its initialized section online. */ + VM_BUG_ON(!ext); + subext = page_ext_data(ext, &mm_subpage_ext_ops); + missing = subext->shmem_missing; + page_ext_put(ext); + return missing; +} + +void mm_subpage_shmem_set_missing(struct page *page, unsigned long missing) +{ + struct page_ext *ext = page_ext_get(page); + struct mm_subpage_ext *subext; + + VM_BUG_ON(!ext); + VM_BUG_ON(missing & ~GENMASK(SUBPAGES_PER_FOLIO - 1, 0)); + subext = page_ext_data(ext, &mm_subpage_ext_ops); + subext->shmem_missing = missing; + page_ext_put(ext); +} + +static bool __init subpage_ext_needed(void) +{ + return true; +} + +struct page_ext_operations mm_subpage_ext_ops = { + .size = sizeof(struct mm_subpage_ext), + .need = subpage_ext_needed, +}; + +/* Only alternative PTE mappings contribute; native rmap counts stay unchanged. */ +static void subpage_file_map_update(phys_addr_t phys, unsigned long size, int delta) +{ + struct page_ext *ext; + struct mm_subpage_ext *subext; + unsigned int index, nr, i; + int count; + + VM_BUG_ON(!is_power_of_2(size) || size < MM_SUBPAGE_SIZE || + size >= PAGE_SIZE || !IS_ALIGNED(phys, size)); + ext = page_ext_from_phys(phys); + if (WARN_ON_ONCE(!ext)) + return; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + index = offset_in_page(phys) >> MM_SUBPAGE_SHIFT; + nr = size >> MM_SUBPAGE_SHIFT; + for (i = index; i < index + nr; i++) { + count = atomic_add_return(delta, &subext->file_maps[i]); + VM_WARN_ON(count < 0); + } + /* Rmap counts one mapping, regardless of its coverage in 4K units. */ + count = atomic_add_return(delta, &subext->file_leaves); + VM_WARN_ON(count < 0); + page_ext_put(ext); +} + +void __mm_subpage_file_map_add(phys_addr_t phys, unsigned long size) +{ + subpage_file_map_update(phys, size, 1); +} + +void __mm_subpage_file_map_del(phys_addr_t phys, unsigned long size) +{ + subpage_file_map_update(phys, size, -1); +} + +/* Snapshot for statistics only, never an ownership or pinning decision. */ +unsigned int mm_subpage_file_mapcounts(struct page *page, struct mm_subpage_mapcounts *counts) +{ + struct page_ext *ext = page_ext_get(page); + struct mm_subpage_ext *subext; + unsigned int i, leaves; + + if (WARN_ON_ONCE(!ext)) { + memset(counts, 0, sizeof(*counts)); + return 0; + } + subext = page_ext_data(ext, &mm_subpage_ext_ops); + for (i = 0; i < SUBPAGES_PER_FOLIO; i++) + counts->slots[i] = max(atomic_read(&subext->file_maps[i]), 0); + leaves = max(atomic_read(&subext->file_leaves), 0); + page_ext_put(ext); + return leaves; +} + +static int owner_bind(struct mm_subpage_owner *owner) +{ + struct page_ext *ext = page_ext_get(&owner->folio->page); + struct mm_subpage_ext *subext; + int ret; + + if (!ext) + return -EOPNOTSUPP; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + ret = cmpxchg(&subext->owner, NULL, owner) ? -EEXIST : 0; + page_ext_put(ext); + return ret; +} + +static void owner_unbind(struct mm_subpage_owner *owner) +{ + struct page_ext *ext = page_ext_get(&owner->folio->page); + struct mm_subpage_ext *subext; + + /* The allocated, referenced folio prevents its section going offline. */ + VM_BUG_ON(!ext); + subext = page_ext_data(ext, &mm_subpage_ext_ops); + VM_BUG_ON(rcu_access_pointer(subext->owner) != owner); + rcu_assign_pointer(subext->owner, NULL); + page_ext_put(ext); +} + +struct mm_subpage *mm_subpage_get_from_phys(phys_addr_t phys) +{ + struct mm_subpage_owner *owner; + struct mm_subpage_ext *subext; + struct mm_subpage *slot = NULL; + struct page_ext *ext; + unsigned int index; + + if (!IS_ALIGNED(phys, MM_SUBPAGE_SIZE)) + return NULL; + ext = page_ext_from_phys(phys); + if (!ext) + return NULL; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + index = owner ? offset_in_page(phys) >> owner->shift : 0; + /* Keep the native allocation alive before publishing another slot ref. */ + if (owner && folio_try_get(owner->folio)) { + if (refcount_inc_not_zero_acquire(&owner->slots[index].refs)) + slot = &owner->slots[index]; + else + folio_put(owner->folio); + } + page_ext_put(ext); + return slot; +} + +struct cow_pool_key { + struct mem_cgroup *memcg; + int nid; + u32 bucket; +}; + +struct mm_subpage_pool { + struct rcu_head rcu; + spinlock_t lock; + refcount_t refs; + struct list_head owners; + struct list_head available; + struct mm_subpage_owner *unissued; + unsigned long backing_pages; + unsigned long all_slots; + u8 shift; + bool closed; + /* Only a per-mm COW pool owns this charge-domain reference. */ + struct mem_cgroup *cow_memcg; + int cow_nid; + bool cow_shared; + /* Allocated only for shared COW pools; protected by pool->lock. */ + unsigned long *cow_free_slots; + struct cow_pool_key cow_key; + struct rhash_head cow_hash; + struct list_head cow_domain; + /* Protected by this pool's lock, never used after registry detachment. */ + struct mm_subpage_cow_pools *cow_registry; +}; + +#define COW_POOL_BITS 4 +struct mm_subpage_cow_pools { + struct rhashtable domains; + spinlock_t lock; + struct list_head pools; + refcount_t refs; + struct rcu_head rcu; + struct work_struct free_work; +}; + +static const struct rhashtable_params cow_domain_params = { + .head_offset = offsetof(struct mm_subpage_pool, cow_hash), + .key_offset = offsetof(struct mm_subpage_pool, cow_key), + .key_len = sizeof(struct cow_pool_key), + .nelem_hint = 16, + .automatic_shrinking = true, +}; + +static void cow_registry_free_work(struct work_struct *work) +{ + struct mm_subpage_cow_pools *pools = + container_of(work, struct mm_subpage_cow_pools, free_work); + + WARN_ON_ONCE(!list_empty(&pools->pools)); + rhashtable_destroy(&pools->domains); + kfree(pools); +} + +static void cow_registry_free_rcu(struct rcu_head *rcu) +{ + struct mm_subpage_cow_pools *pools = + container_of(rcu, struct mm_subpage_cow_pools, rcu); + + queue_work(system_unbound_wq, &pools->free_work); +} + +static void cow_registry_put(struct mm_subpage_cow_pools *pools) +{ + if (refcount_dec_and_test(&pools->refs)) + call_rcu(&pools->rcu, cow_registry_free_rcu); +} + +static void pool_put_raw(struct mm_subpage_pool *pool); +static void owner_remove(struct mm_subpage_owner *owner); +static void owner_free(struct mm_subpage_owner *owner, bool put_folio); + +/* The caller retains a pool reference independently of registry membership. */ +static void cow_pool_detach_locked(struct mm_subpage_pool *pool) +{ + struct mm_subpage_cow_pools *pools = pool->cow_registry; + + lockdep_assert_held(&pool->lock); + if (!pools) + return; + pool->cow_registry = NULL; + /* A failed prospective insertion may not have entered the hash/list. */ + rhashtable_remove_fast(&pools->domains, &pool->cow_hash, cow_domain_params); + spin_lock(&pools->lock); + list_del_init(&pool->cow_domain); + spin_unlock(&pools->lock); + pool_put_raw(pool); /* Removed (or unpublished prospective) membership. */ + cow_registry_put(pools); +} + +static void cow_pool_retire_empty(struct mm_subpage_pool *pool) +{ + struct mm_subpage_owner *owner = NULL; + unsigned long flags; + + if (!pool->cow_shared) + return; + spin_lock_irqsave(&pool->lock, flags); + if (!pool->backing_pages || + (pool->backing_pages == 1 && pool->unissued)) { + pool->closed = true; + cow_pool_detach_locked(pool); + /* A failed allocation may leave wholly unissued supply, not a + * live domain. Retire it under the same reservation lock. + */ + owner = pool->unissued; + if (owner) + owner_remove(owner); + } + spin_unlock_irqrestore(&pool->lock, flags); + if (owner) + owner_free(owner, true); +} + +static void owner_init(struct mm_subpage_owner *owner, struct mm_subpage_pool *pool, + struct folio *folio) +{ + unsigned int i; + + owner->pool = pool; + owner->shift = pool->shift; + owner->folio = folio; + INIT_LIST_HEAD(&owner->available); + for (i = 0; i < (PAGE_SIZE >> pool->shift); i++) { + owner->slots[i].owner = owner; + owner->slots[i].index = i; + owner->slots[i].shift = pool->shift; + refcount_set(&owner->slots[i].refs, 0); + atomic_set(&owner->slots[i].mapcount, 0); + } +} + +struct mm_subpage_pool *mm_subpage_pool_create_granule(gfp_t gfp, unsigned int shift) +{ + struct mm_subpage_pool *pool; + + BUILD_BUG_ON(PAGE_SHIFT <= MM_SUBPAGE_SHIFT); + BUILD_BUG_ON(SUBPAGES_PER_FOLIO > BITS_PER_LONG); + if (!subpage_shift_valid(shift)) + return NULL; + pool = kzalloc(sizeof(*pool), gfp | __GFP_ACCOUNT); + if (!pool) + return NULL; + pool->shift = shift; + pool->all_slots = subpage_mask(shift); + spin_lock_init(&pool->lock); + refcount_set(&pool->refs, 1); + INIT_LIST_HEAD(&pool->owners); + INIT_LIST_HEAD(&pool->available); + return pool; +} + +struct mm_subpage_pool *mm_subpage_pool_create(gfp_t gfp) +{ + return mm_subpage_pool_create_granule(gfp, MM_SUBPAGE_SHIFT); +} + +void mm_subpage_pool_get(struct mm_subpage_pool *pool) +{ + refcount_inc(&pool->refs); +} + +static void pool_put_raw(struct mm_subpage_pool *pool) +{ + if (!refcount_dec_and_test(&pool->refs)) + return; + WARN_ON_ONCE(!pool->closed || !list_empty(&pool->owners)); + mem_cgroup_put(pool->cow_memcg); + kfree(pool->cow_free_slots); + kfree_rcu(pool, rcu); +} + +void mm_subpage_pool_put(struct mm_subpage_pool *pool) +{ + /* Failed allocations must not leave an empty per-domain cache behind. */ + cow_pool_retire_empty(pool); + pool_put_raw(pool); +} + +/* Per-mm pools retain each still-backed charge/node/shard domain separately. */ +struct mm_subpage_pool *mm_subpage_cow_pool_get_charged(struct mm_struct *mm, + struct mm_struct *charge_mm, struct vm_area_struct *vma, + unsigned long address) +{ + struct mm_subpage_cow_pools *pools, *new_pools; + struct mm_subpage_pool *pool, *new; + struct cow_pool_key key = {}; + struct mem_cgroup *memcg; + unsigned long flags; + unsigned int attempt; + void *result; + int nid; + + if ((vma->vm_flags & (VM_LOCKED | VM_DROPPABLE | VM_SPECIAL)) || + num_online_nodes() != 1 || num_node_state(N_MEMORY) != 1) + return NULL; +#ifdef CONFIG_NUMA + if (vma_policy(vma) || current->mempolicy) + return NULL; +#endif + nid = first_node(node_states[N_MEMORY]); + if (!node_isset(nid, cpuset_current_mems_allowed)) + return NULL; + /* Acquire initialization published by the mm's cmpxchg below. */ + pools = smp_load_acquire(&mm->cow_subpage_pool); + if (!pools) { + new_pools = kzalloc(sizeof(*new_pools), GFP_KERNEL | __GFP_ACCOUNT); + if (!new_pools) + return NULL; + spin_lock_init(&new_pools->lock); + INIT_LIST_HEAD(&new_pools->pools); + refcount_set(&new_pools->refs, 1); + INIT_WORK(&new_pools->free_work, cow_registry_free_work); + if (rhashtable_init(&new_pools->domains, &cow_domain_params)) { + kfree(new_pools); + return NULL; + } + pools = cmpxchg(&mm->cow_subpage_pool, NULL, new_pools); + if (!pools) + pools = new_pools; + else + cow_registry_put(new_pools); + } + memcg = get_mem_cgroup_from_mm(charge_mm); + key.memcg = memcg; + key.nid = nid; + key.bucket = hash_long(address >> PAGE_SHIFT, COW_POOL_BITS); + for (attempt = 0; attempt < 3; attempt++) { + rcu_read_lock(); + pool = rhashtable_lookup(&pools->domains, &key, cow_domain_params); + if (pool && !refcount_inc_not_zero(&pool->refs)) + pool = NULL; + rcu_read_unlock(); + if (pool) { + mem_cgroup_put(memcg); + return pool; + } + new = mm_subpage_pool_create_granule(GFP_KERNEL, mm_page_shift(mm)); + if (!new) + break; + new->cow_free_slots = kcalloc(SUBPAGES_PER_FOLIO, + sizeof(*new->cow_free_slots), + GFP_KERNEL | __GFP_ACCOUNT); + if (!new->cow_free_slots) { + mm_subpage_pool_close(new); + mm_subpage_pool_put(new); + break; + } +#ifdef CONFIG_MEMCG + if (memcg) + css_get(&memcg->css); +#endif + new->cow_memcg = memcg; + new->cow_nid = nid; + new->cow_shared = true; + new->cow_key = key; + INIT_LIST_HEAD(&new->cow_domain); + new->cow_registry = pools; + /* Hold caller, prospective membership and registry lifetime refs + * before a concurrent lookup can observe this pool. + */ + mm_subpage_pool_get(new); + refcount_inc(&pools->refs); + spin_lock_irqsave(&new->lock, flags); + result = rhashtable_lookup_get_insert_fast(&pools->domains, + &new->cow_hash, cow_domain_params); + if (!result) { + spin_lock(&pools->lock); + list_add_tail(&new->cow_domain, &pools->pools); + spin_unlock(&pools->lock); + } + spin_unlock_irqrestore(&new->lock, flags); + if (!result) { + mem_cgroup_put(memcg); + return new; + } + /* Do not dereference an unreferenced object returned by insert. */ + mm_subpage_pool_close(new); + mm_subpage_pool_put(new); + if (IS_ERR(result)) + break; + } + mem_cgroup_put(memcg); + return NULL; +} + +struct mm_subpage_pool *mm_subpage_cow_pool_get(struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long address) +{ + return mm_subpage_cow_pool_get_charged(mm, mm, vma, address); +} + +bool mm_subpage_cow_folio_matches(struct mm_subpage_pool *pool, struct folio *folio) +{ + bool matches; + + /* The folio charge may be reparented while this pool remains alive. */ + rcu_read_lock(); + matches = pool->cow_memcg == folio_memcg(folio) && + pool->cow_nid == folio_nid(folio); + rcu_read_unlock(); + return matches; +} + +void mm_subpage_cow_pool_exit(struct mm_struct *mm) +{ + struct mm_subpage_cow_pools *pools = xchg(&mm->cow_subpage_pool, NULL); + struct mm_subpage_pool *pool; + unsigned long flags; + + /* No faults remain; outstanding owners/callbacks never use this mm. */ + if (!pools) + return; + for (;;) { + spin_lock_irqsave(&pools->lock, flags); + pool = list_first_entry_or_null(&pools->pools, + struct mm_subpage_pool, cow_domain); + if (pool) + mm_subpage_pool_get(pool); + spin_unlock_irqrestore(&pools->lock, flags); + if (!pool) + break; + /* Lock order is pool -> registry list, never the reverse. */ + mm_subpage_pool_close(pool); + mm_subpage_pool_put(pool); + } + cow_registry_put(pools); +} + +/* Counts include unavailable (busy/migrating) owners, so zero is conclusive. + * A nonzero count still requires all the normal owner eligibility checks. + */ +static void cow_update_free(struct mm_subpage_pool *pool, unsigned long mask, int delta) +{ + unsigned int i; + + lockdep_assert_held(&pool->lock); + if (!pool->cow_shared) + return; + for_each_set_bit(i, &mask, SUBPAGES_PER_FOLIO) { + VM_BUG_ON(delta < 0 && !pool->cow_free_slots[i]); + pool->cow_free_slots[i] += delta; + } +} + +static bool cow_has_free(struct mm_subpage_pool *pool, unsigned long eligible) +{ + unsigned int i; + + lockdep_assert_held(&pool->lock); + if (!pool->cow_shared) + return true; + for_each_set_bit(i, &eligible, SUBPAGES_PER_FOLIO) + if (pool->cow_free_slots[i]) + return true; + return false; +} + +static void owner_remove(struct mm_subpage_owner *owner) +{ + lockdep_assert_held(&owner->pool->lock); + owner->detached = true; + cow_update_free(owner->pool, owner->pool->all_slots & ~owner->used, -1); + list_del(&owner->all); + list_del_init(&owner->available); + if (owner->pool->unissued == owner) + owner->pool->unissued = NULL; + owner->pool->backing_pages--; + if (!owner->pool->backing_pages && owner->pool->cow_shared) { + owner->pool->closed = true; + cow_pool_detach_locked(owner->pool); + } +} + +static void owner_free_rmap(struct mm_subpage_owner *owner) +{ + unsigned int i; + + if (!owner->rmap) + return; + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) + if (owner->rmap[i].root) + put_anon_vma(owner->rmap[i].root); + kfree(owner->rmap); +} + +static void owner_free_metadata(struct mm_subpage_owner *owner) +{ + struct mm_subpage_pool *pool = owner->pool; + + if (owner->anon_root) + put_anon_vma(owner->anon_root); + owner_free_rmap(owner); + kfree_rcu(owner, rcu); + mm_subpage_pool_put(pool); +} + +static void owner_free_work(struct work_struct *work) +{ + owner_free_metadata(container_of(work, struct mm_subpage_owner, free_work)); +} + +static void owner_free(struct mm_subpage_owner *owner, bool put_folio) +{ + owner_unbind(owner); + if (put_folio) { + if (owner->anon_root) + free_swap_cache(owner->folio); + /* Last slot reference, or wholly unissued supply. */ + folio_put(owner->folio); + } + if (owner->anon_root || owner->rmap) { + /* Any final anon-vma put can sleep, including a linear owner's. */ + INIT_WORK(&owner->free_work, owner_free_work); + queue_work(system_unbound_wq, &owner->free_work); + } else { + owner_free_metadata(owner); + } +} + +struct anon_vma *mm_subpage_anon_root(const struct mm_subpage *subpage) +{ + VM_BUG_ON_FOLIO(!folio_test_locked(subpage->owner->folio), subpage->owner->folio); + return subpage->owner->anon_root; +} + +void mm_subpage_hold_anon_root(struct mm_subpage *subpage, struct anon_vma *root, + bool droppable) +{ + struct mm_subpage_owner *owner = subpage->owner; + + VM_BUG_ON_FOLIO(!folio_test_locked(owner->folio), owner->folio); + VM_BUG_ON(owner->anon_root || root != root->root); + get_anon_vma(root); + owner->droppable = droppable; + owner->anon_root = root; +} + +static struct mm_subpage_rmap slot_rmap(struct mm_subpage *slot) +{ + struct mm_subpage_owner *owner = slot->owner; + + lockdep_assert_held(&owner->pool->lock); + if (owner->rmap) + return owner->rmap[slot->index]; + return (struct mm_subpage_rmap) { + .root = owner->anon_root, + .pos = { .index = owner->folio->index, + .offset = mm_subpage_offset(slot) }, + .offset = mm_subpage_offset(slot), + }; +} + +static bool rmap_matches(struct mm_subpage_rmap map, struct vm_area_struct *vma, + unsigned long address) +{ + return map.root == vma->anon_vma->root && + vm_page_offset_equal(map.pos, vma_page_offset_at(vma, address)); +} + +bool mm_subpage_match_rmap(struct mm_subpage *slot, struct vm_area_struct *vma, + unsigned long address) +{ + struct mm_subpage_pool *pool = slot->owner->pool; + unsigned long flags; + bool matches; + + spin_lock_irqsave(&pool->lock, flags); + matches = slot->rmap_bound && rmap_matches(slot_rmap(slot), vma, address); + spin_unlock_irqrestore(&pool->lock, flags); + return matches; +} + +static void *owner_kcalloc(struct mm_subpage_owner *owner, size_t nr, size_t size) +{ + struct mem_cgroup *memcg, *saved; + void *maps; + + rcu_read_lock(); + do { + memcg = mem_cgroup_from_virt(owner) ?: root_mem_cgroup; + } while (memcg && !mem_cgroup_tryget(memcg)); + rcu_read_unlock(); + saved = set_active_memcg(memcg); + maps = kcalloc(nr, size, GFP_NOWAIT | __GFP_NOWARN | __GFP_ACCOUNT); + set_active_memcg(saved); + mem_cgroup_put(memcg); + return maps; +} + +struct subpage_retired_root { + struct work_struct work; + struct anon_vma *root; +}; + +static void subpage_put_retired_root(struct work_struct *work) +{ + struct subpage_retired_root *retired = + container_of(work, struct subpage_retired_root, work); + + put_anon_vma(retired->root); + kfree(retired); +} + +/* The folio lock serializes binding and sidecar creation against migration. */ +int mm_subpage_bind_rmap(struct mm_subpage *slot, struct vm_area_struct *vma, + unsigned long address) +{ + struct mm_subpage_owner *owner = slot->owner; + struct mm_subpage_pool *pool = owner->pool; + struct mm_subpage_rmap *maps = NULL; + struct subpage_retired_root *retired = NULL; + unsigned long flags; + unsigned int i; + int ret = 0; + + VM_BUG_ON_FOLIO(!folio_test_locked(owner->folio), owner->folio); + spin_lock_irqsave(&pool->lock, flags); + if (slot->rmap_bound) { + ret = rmap_matches(slot_rmap(slot), vma, address) ? 0 : -EINVAL; + goto unlock; + } + if (!owner->rmap && !rmap_matches(slot_rmap(slot), vma, address)) { + spin_unlock_irqrestore(&pool->lock, flags); + /* The caller may also hold a PTE lock. Never enter reclaim here. */ + maps = owner_kcalloc(owner, PAGE_SIZE >> owner->shift, sizeof(*maps)); + if (!maps) + return -ENOMEM; + spin_lock_irqsave(&pool->lock, flags); + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) { + maps[i].offset = i << owner->shift; + if (!owner->slots[i].rmap_bound) + continue; + maps[i] = slot_rmap(&owner->slots[i]); + get_anon_vma(maps[i].root); + } + owner->rmap = maps; + } + if (owner->rmap) { + struct mm_subpage_rmap *map = &owner->rmap[slot->index]; + + if (map->root && map->root != vma->anon_vma->root) { + /* A final anon_vma put may sleep; the caller can hold a PTL. */ + spin_unlock_irqrestore(&pool->lock, flags); + retired = owner_kcalloc(owner, 1, sizeof(*retired)); + if (!retired) + return -ENOMEM; + INIT_WORK(&retired->work, subpage_put_retired_root); + spin_lock_irqsave(&pool->lock, flags); + retired->root = map->root; + } + if (map->root != vma->anon_vma->root) { + get_anon_vma(vma->anon_vma->root); + map->root = vma->anon_vma->root; + } + map->pos = vma_page_offset_at(vma, address); + map->offset = mm_subpage_offset(slot); + } + slot->rmap_bound = true; +unlock: + spin_unlock_irqrestore(&pool->lock, flags); + if (retired) + queue_work(system_unbound_wq, &retired->work); + return ret; +} + +/* Preflight publication without making an unexposed owner look published. + * A fresh linear slot needs no sidecar: add_new_anon_rmap will set its anchor. + * Otherwise bind the independent virtual position before any PTE lock. + */ +int mm_subpage_prepare_anon_rmap(struct mm_subpage *slot, + struct vm_area_struct *vma, unsigned long address) +{ + struct mm_subpage_owner *owner = slot->owner; + + VM_BUG_ON_FOLIO(!folio_test_locked(owner->folio), owner->folio); + if (!owner->anon_root && !owner->rmap && + mm_subpage_offset(slot) == vma_page_offset_at(vma, address).offset) + return 0; + return mm_subpage_bind_rmap(slot, vma, address); +} + +static bool subpage_move_allowed(struct mm_subpage *slot, bool swap) +{ + lockdep_assert_held(&slot->owner->pool->lock); + return !slot->owner->migrating && !slot->pins && + atomic_read(&slot->mapcount) == (swap ? 0 : 1) && + (swap || (slot->rmap_bound && slot->exclusive)); +} + +int mm_subpage_move_prepare(struct mm_subpage_move *move, struct mm_subpage *slot, + struct vm_area_struct *src, unsigned long src_address, + struct vm_area_struct *dst, unsigned long dst_address, bool swap) +{ + struct mm_subpage_owner *owner = slot->owner; + struct mm_subpage_rmap *maps, old; + unsigned long flags; + unsigned int i; + int ret = -EBUSY; + + VM_BUG_ON_FOLIO(!folio_test_locked(owner->folio), owner->folio); + memset(move, 0, sizeof(*move)); + if (!src->anon_vma || !dst->anon_vma || !owner->anon_root || + mm_page_shift(src->vm_mm) != slot->shift || + mm_page_shift(dst->vm_mm) != slot->shift || + owner->droppable != !!(dst->vm_flags & VM_DROPPABLE)) + return -EINVAL; + spin_lock_irqsave(&owner->pool->lock, flags); + if (!subpage_move_allowed(slot, swap) || + (slot->rmap_bound && !rmap_matches(slot_rmap(slot), src, src_address))) + goto unlock; + spin_unlock_irqrestore(&owner->pool->lock, flags); + + /* Allocate before touching either PTE. The folio lock serializes writers. */ + if (!owner->rmap) { + maps = owner_kcalloc(owner, PAGE_SIZE >> owner->shift, sizeof(*maps)); + if (!maps) + return -ENOMEM; + spin_lock_irqsave(&owner->pool->lock, flags); + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) { + maps[i].offset = i << owner->shift; + if (!owner->slots[i].rmap_bound) + continue; + maps[i] = slot_rmap(&owner->slots[i]); + get_anon_vma(maps[i].root); + } + owner->rmap = maps; + spin_unlock_irqrestore(&owner->pool->lock, flags); + } + old = owner->rmap[slot->index]; + move->root_changed = old.root != dst->anon_vma->root; + if (move->root_changed && old.root) { + move->retired = owner_kcalloc(owner, 1, sizeof(*move->retired)); + if (!move->retired) + return -ENOMEM; + INIT_WORK(&move->retired->work, subpage_put_retired_root); + move->retired->root = old.root; + } + if (move->root_changed) + get_anon_vma(dst->anon_vma->root); + move->slot = slot; + move->swap = swap; + move->map = (struct mm_subpage_rmap) { + .root = dst->anon_vma->root, + .pos = vma_page_offset_at(dst, dst_address), + .offset = mm_subpage_offset(slot), + }; + return 0; +unlock: + spin_unlock_irqrestore(&owner->pool->lock, flags); + return ret; +} + +int mm_subpage_move_commit(struct mm_subpage_move *move) +{ + struct mm_subpage *slot = move->slot; + struct mm_subpage_owner *owner = slot->owner; + unsigned long flags; + int ret = -EBUSY; + + VM_BUG_ON_FOLIO(!folio_test_locked(owner->folio), owner->folio); + spin_lock_irqsave(&owner->pool->lock, flags); + /* Recheck pins after source-PTE clearing and the matching TLB barrier. */ + if (subpage_move_allowed(slot, move->swap)) { + owner->rmap[slot->index] = move->map; + slot->rmap_bound = true; + move->committed = true; + ret = 0; + } + spin_unlock_irqrestore(&owner->pool->lock, flags); + return ret; +} + +void mm_subpage_move_finish(struct mm_subpage_move *move) +{ + if (move->committed) { + if (move->retired) + queue_work(system_unbound_wq, &move->retired->work); + } else { + /* The locked destination VMA still holds this root. */ + if (move->root_changed) + put_anon_vma(move->map.root); + kfree(move->retired); + } +} + +int mm_subpage_snapshot_rmap(struct folio *folio, struct mm_subpage_rmap *maps) +{ + struct page_ext *ext = page_ext_get(&folio->page); + struct mm_subpage_ext *subext; + struct mm_subpage_owner *owner; + unsigned long flags; + unsigned int i; + int nr = -1; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (!ext) + return nr; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (!owner) + goto out; + spin_lock_irqsave(&owner->pool->lock, flags); + if (!owner->detached && owner->rmap) { + nr = 0; + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) { + if (!owner->slots[i].rmap_bound) + continue; + maps[nr] = owner->rmap[i]; + get_anon_vma(maps[nr].root); + nr++; + } + } + spin_unlock_irqrestore(&owner->pool->lock, flags); +out: + page_ext_put(ext); + return nr; +} + +/* A referenced folio plus a stable VMA suffices; no folio lock required. */ +bool mm_subpage_address_in_vma(const struct folio *folio, + const struct vm_area_struct *vma, unsigned long *address) +{ + struct page_ext *ext = page_ext_get(&folio->page); + struct mm_subpage_ext *subext; + struct mm_subpage_owner *owner; + unsigned long flags; + unsigned int i; + bool nonlinear = false; + + if (!ext) + return false; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (!owner) + goto out; + spin_lock_irqsave(&owner->pool->lock, flags); + if (!owner->detached && owner->rmap) { + nonlinear = true; + *address = -EFAULT; + for (i = 0; vma->anon_vma && i < (PAGE_SIZE >> owner->shift); i++) { + if (!owner->slots[i].rmap_bound || + owner->rmap[i].root != vma->anon_vma->root) + continue; + *address = vma_address_at_offset(vma, owner->rmap[i].pos); + if (*address != -EFAULT) + break; + } + } + spin_unlock_irqrestore(&owner->pool->lock, flags); +out: + page_ext_put(ext); + return nonlinear; +} + +bool mm_subpage_is_droppable(const struct mm_subpage *subpage) +{ + return subpage->owner->droppable; +} + +void mm_subpage_pool_close(struct mm_subpage_pool *pool) +{ + struct mm_subpage_owner *owner; + unsigned long flags; + + spin_lock_irqsave(&pool->lock, flags); + pool->closed = true; + cow_pool_detach_locked(pool); + /* At most one newly supplied owner has no issued slots. Close is O(1). */ + owner = pool->unissued; + if (owner) + owner_remove(owner); + spin_unlock_irqrestore(&pool->lock, flags); + if (owner) + owner_free(owner, true); +} + +static bool cow_owner_eligible(struct mm_subpage_pool *pool, struct folio *folio) +{ + /* All allocation and retry paths must exclude poisoned backing. */ + if (folio_test_hwpoison(folio)) + return false; + if (!pool->cow_shared) + return true; + /* A previously unlocked owner can be mlocked or fully MADV_FREE'd + * after entering the pool. Never place a new live slot in such backing. + */ + return !folio_test_mlocked(folio) && !folio_test_unevictable(folio) && + !folio_test_ksm(folio) && + (!folio_test_anon(folio) || folio_test_swapbacked(folio)) && + mm_subpage_cow_folio_matches(pool, folio); +} + +/* Pressure-only fallback after a non-OOM speculative allocation failed. + * Return true when existing compatible backing can be retried. Unlike the + * normal bounded search, inspect all owners before allowing an OOM-capable + * allocation: a busy slot farther down the list may require no new charge. + */ +static bool cow_wait_busy(struct mm_subpage_pool *pool, unsigned long eligible, + bool require_all, unsigned int count) +{ + struct mm_subpage_owner *owner; + struct folio *wait = NULL; + unsigned long flags; + + spin_lock_irqsave(&pool->lock, flags); + if (!pool->closed && pool->cow_shared && cow_has_free(pool, eligible)) { + list_for_each_entry(owner, &pool->available, available) { + if (!owner->migrating && (~owner->used & eligible) && + (!require_all || !(owner->used & eligible)) && + (count == 1 || hweight_long(~owner->used & eligible) >= count) && + cow_owner_eligible(pool, owner->folio) && + !folio_test_swapcache(owner->folio) && + !folio_test_writeback(owner->folio)) { + wait = owner->folio; + folio_get(wait); + break; + } + } + } + spin_unlock_irqrestore(&pool->lock, flags); + if (wait) { + folio_wait_locked(wait); + folio_put(wait); + return true; + } + return false; +} + +bool mm_subpage_cow_wait_busy(struct mm_subpage_pool *pool) +{ + return cow_wait_busy(pool, pool->all_slots, false, 1); +} + +bool mm_subpage_cow_wait_busy_at(struct mm_subpage_pool *pool, unsigned int offset) +{ + return cow_wait_busy(pool, offset_mask(pool->shift, offset), false, 1); +} + +bool mm_subpage_cow_wait_busy_mask(struct mm_subpage_pool *pool, unsigned long mask) +{ + return cow_wait_busy(pool, mask, true, 1); +} + +bool mm_subpage_cow_wait_busy_count(struct mm_subpage_pool *pool, unsigned int count) +{ + if (!count || count > hweight_long(pool->all_slots)) + return false; + return cow_wait_busy(pool, pool->all_slots, false, count); +} + +static struct mm_subpage_owner *find_available(struct mm_subpage_pool *pool, + unsigned long eligible, bool require_all, + unsigned int count) +{ + struct mm_subpage_owner *owner; + unsigned int visited = 0; + + lockdep_assert_held(&pool->lock); + if (!cow_has_free(pool, eligible)) + return NULL; + list_for_each_entry(owner, &pool->available, available) { + /* A fixed-offset stream must not scan every partially used owner + * on each fault. New COW supply is inserted first, so a bounded + * miss can always make progress using a fresh folio. + */ + if (pool->cow_shared && visited++ == 32) { + /* The next search starts past this incompatible prefix. A + * phased offset stream must not strand older reusable slots. + */ + list_rotate_to_front(&owner->available, &pool->available); + break; + } + /* A nonpresent PTE may still need any of this folio's slot bytes. */ + if (!owner->migrating && (~owner->used & eligible) && + (!require_all || !(owner->used & eligible)) && + (count == 1 || hweight_long(~owner->used & eligible) >= count) && + (!pool->cow_shared || !folio_test_locked(owner->folio)) && + cow_owner_eligible(pool, owner->folio) && + !folio_test_swapcache(owner->folio) && + !folio_test_writeback(owner->folio)) + return owner; + } + return NULL; +} + +static int pool_add_folio(struct mm_subpage_pool *pool, struct folio *folio, + unsigned long eligible, bool require_all, + unsigned int count, gfp_t gfp) +{ + struct mm_subpage_owner *owner; + unsigned long flags; + int ret = 0; + + /* Only an unexposed order-0 allocation can become a slot owner. */ + if (folio_order(folio) || folio_ref_count(folio) != 1 || + folio->mapping || folio_mapped(folio) || folio_test_lru(folio) || + folio_test_private(folio) || folio_test_swapcache(folio) || + folio_test_hwpoison(folio)) + return -EINVAL; + owner = kzalloc(sizeof(*owner), gfp | __GFP_ACCOUNT); + if (!owner) + return -ENOMEM; + owner_init(owner, pool, folio); + /* Reject duplicate ownership before touching any existing data. */ + ret = owner_bind(owner); + if (ret) { + kfree(owner); + return ret; + } + /* Initialise even unissued bytes before any future whole-folio operation. */ + clear_user_subpage_range(&folio->page, 0, PAGE_SIZE); + __folio_mark_uptodate(folio); + owner->zeroed = pool->all_slots; + spin_lock_irqsave(&pool->lock, flags); + if (pool->closed) + ret = -ESHUTDOWN; + else if (pool->unissued || find_available(pool, eligible, require_all, count)) + /* Another allocator supplied backing while we allocated metadata. + * Preserve the single-unissued-owner invariant even when a shared + * COW lookup skips a concurrently locked or rotating list prefix. + */ + ret = -EEXIST; + else { + mm_subpage_pool_get(pool); + list_add_tail(&owner->all, &pool->owners); + cow_update_free(pool, pool->all_slots, 1); + if (pool->cow_shared) + list_add(&owner->available, &pool->available); + else + list_add_tail(&owner->available, &pool->available); + pool->backing_pages++; + pool->unissued = owner; + } + spin_unlock_irqrestore(&pool->lock, flags); + if (ret) { + owner_unbind(owner); + kfree_rcu(owner, rcu); + } + return ret; +} + +int mm_subpage_pool_add_folio(struct mm_subpage_pool *pool, struct folio *folio, + gfp_t gfp) +{ + return pool_add_folio(pool, folio, pool->all_slots, false, 1, gfp); +} + +int mm_subpage_pool_add_folio_at(struct mm_subpage_pool *pool, struct folio *folio, + unsigned int offset, gfp_t gfp) +{ + unsigned long eligible = offset_mask(pool->shift, offset); + + if (!eligible) + return -EINVAL; + return pool_add_folio(pool, folio, eligible, false, 1, gfp); +} + +int mm_subpage_pool_add_folio_mask(struct mm_subpage_pool *pool, struct folio *folio, + unsigned long mask, gfp_t gfp) +{ + if (!mask || (mask & ~pool->all_slots)) + return -EINVAL; + return pool_add_folio(pool, folio, mask, true, 1, gfp); +} + +int mm_subpage_pool_add_folio_count(struct mm_subpage_pool *pool, struct folio *folio, + unsigned int count, gfp_t gfp) +{ + if (!count || count > hweight_long(pool->all_slots)) + return -EINVAL; + return pool_add_folio(pool, folio, pool->all_slots, false, count, gfp); +} + +unsigned long mm_subpage_pool_backing_pages(struct mm_subpage_pool *pool) +{ + unsigned long pages, flags; + + spin_lock_irqsave(&pool->lock, flags); + pages = pool->backing_pages; + spin_unlock_irqrestore(&pool->lock, flags); + return pages; +} + +/* Caller holds pool lock; each issued slot owns one native folio reference. */ +static struct mm_subpage *reserve_owner_slot(struct mm_subpage_owner *owner, + unsigned int index, bool *needs_zero) +{ + struct mm_subpage_pool *pool = owner->pool; + struct mm_subpage *subpage; + + lockdep_assert_held(&pool->lock); + if (pool->unissued == owner) + /* Transfer the supplied reference to the first reservation. */ + pool->unissued = NULL; + else + folio_get(owner->folio); + __set_bit(index, &owner->used); + cow_update_free(pool, BIT(index), -1); + if (needs_zero) + *needs_zero = !test_bit(index, &owner->zeroed); + __clear_bit(index, &owner->zeroed); + if (owner->used == pool->all_slots) + list_del_init(&owner->available); + subpage = &owner->slots[index]; + WRITE_ONCE(subpage->lazyfree, false); + /* The used bit keeps backing alive while data is initialised. */ + VM_BUG_ON(refcount_read(&subpage->refs)); + VM_BUG_ON(atomic_read(&subpage->mapcount)); + return subpage; +} + +static struct mm_subpage *subpage_reserve(struct mm_subpage_pool *pool, + unsigned long eligible, bool *needs_zero, bool lock_folio) +{ + struct mm_subpage_owner *owner; + struct mm_subpage *subpage; + unsigned long flags; + unsigned int index; + +retry: + spin_lock_irqsave(&pool->lock, flags); + if (pool->closed) { + subpage = ERR_PTR(-ESHUTDOWN); + goto out; + } + owner = find_available(pool, eligible, false, 1); + if (!owner) { + subpage = ERR_PTR(-EAGAIN); + goto out; + } + if (lock_folio) { + if (!folio_trylock(owner->folio)) { + struct folio *wait; + + if (pool->cow_shared) { + /* Another COW fault won this folio after the lookup. + * Try other backing, or supply another folio, rather + * than making parallel faults queue on its lock. + */ + spin_unlock_irqrestore(&pool->lock, flags); + cond_resched(); + goto retry; + } + wait = owner->folio; + folio_get(wait); + spin_unlock_irqrestore(&pool->lock, flags); + folio_wait_locked(wait); + folio_put(wait); + goto retry; + } + /* Reclaim may have acquired the folio after find_available's check. */ + if (folio_test_swapcache(owner->folio) || folio_test_writeback(owner->folio) || + !cow_owner_eligible(pool, owner->folio)) { + folio_unlock(owner->folio); + spin_unlock_irqrestore(&pool->lock, flags); + return ERR_PTR(-EAGAIN); + } + } + index = __ffs(~owner->used & eligible); + subpage = reserve_owner_slot(owner, index, needs_zero); +out: + spin_unlock_irqrestore(&pool->lock, flags); + return subpage; +} + +/* One owner/folio lock; output remains indexed by the caller's logical mask. */ +static int subpage_alloc_mask_locked(struct mm_subpage_pool *pool, unsigned long mask, + bool any_offset, struct mm_subpage **slots) +{ + struct mm_subpage_owner *owner; + unsigned long flags, zero = 0, available; + unsigned int i, physical, count = hweight_long(mask); + bool needs_zero; + + if (!mask || (mask & ~pool->all_slots)) + return -EINVAL; +retry: + spin_lock_irqsave(&pool->lock, flags); + if (pool->closed) { + spin_unlock_irqrestore(&pool->lock, flags); + return -ESHUTDOWN; + } + owner = find_available(pool, any_offset ? pool->all_slots : mask, + !any_offset, any_offset ? count : 1); + if (!owner) { + spin_unlock_irqrestore(&pool->lock, flags); + return -EAGAIN; + } + if (!folio_trylock(owner->folio)) { + /* Parallel faults use another owner rather than queue behind it. */ + spin_unlock_irqrestore(&pool->lock, flags); + cond_resched(); + goto retry; + } + if (folio_test_swapcache(owner->folio) || folio_test_writeback(owner->folio) || + !cow_owner_eligible(pool, owner->folio)) { + folio_unlock(owner->folio); + spin_unlock_irqrestore(&pool->lock, flags); + return -EAGAIN; + } + available = ~owner->used & pool->all_slots; + for_each_set_bit(i, &mask, SUBPAGES_PER_FOLIO) { + physical = any_offset ? __ffs(available) : i; + available &= ~BIT(physical); + slots[i] = reserve_owner_slot(owner, physical, &needs_zero); + if (needs_zero) + zero |= BIT(i); + } + spin_unlock_irqrestore(&pool->lock, flags); + for_each_set_bit(i, &mask, SUBPAGES_PER_FOLIO) { + if (zero & BIT(i)) + clear_user_subpage_range(&owner->folio->page, + mm_subpage_offset(slots[i]), 1U << pool->shift); + /* Zeroing must precede publication, including reused slots. */ + refcount_set_release(&slots[i]->refs, 1); + } + return 0; +} + +int mm_subpage_alloc_mask_locked(struct mm_subpage_pool *pool, unsigned long mask, + struct mm_subpage **slots) +{ + return subpage_alloc_mask_locked(pool, mask, false, slots); +} + +int mm_subpage_alloc_any_mask_locked(struct mm_subpage_pool *pool, unsigned long mask, + struct mm_subpage **slots) +{ + return subpage_alloc_mask_locked(pool, mask, true, slots); +} + +static struct mm_subpage *subpage_alloc(struct mm_subpage_pool *pool, unsigned long eligible, bool lock_folio) +{ + bool needs_zero; + struct mm_subpage *subpage = subpage_reserve(pool, eligible, &needs_zero, lock_folio); + + if (IS_ERR(subpage)) + return subpage; + /* Fresh supply was already zeroed as a whole; reused slots must be cleared. */ + if (needs_zero) + clear_user_subpage_range(&subpage->owner->folio->page, + mm_subpage_offset(subpage), mm_subpage_size(subpage)); + /* Data stores must precede publication in a PTE or another CPU's reference. */ + refcount_set_release(&subpage->refs, 1); + return subpage; +} + +struct mm_subpage *mm_subpage_alloc(struct mm_subpage_pool *pool) +{ + return subpage_alloc(pool, pool->all_slots, false); +} + +struct mm_subpage *mm_subpage_alloc_at(struct mm_subpage_pool *pool, unsigned int offset) +{ + unsigned long eligible = offset_mask(pool->shift, offset); + + if (!eligible) + return ERR_PTR(-EINVAL); + return subpage_alloc(pool, eligible, false); +} + +static void subpage_release_locked(struct mm_subpage *subpage, unsigned long flags) +{ + struct mm_subpage_owner *owner = subpage->owner; + struct mm_subpage_pool *pool = owner->pool; + struct folio *folio = owner->folio; + bool release; + + lockdep_assert_held(&pool->lock); + VM_BUG_ON(refcount_read(&subpage->refs)); + VM_BUG_ON(atomic_read(&subpage->mapcount)); + VM_BUG_ON(subpage->pins); + WRITE_ONCE(subpage->exclusive, false); + if (!owner->migrating) { + WRITE_ONCE(subpage->lazyfree, false); + subpage->rmap_bound = false; + } + __clear_bit(subpage->index, &owner->used); + cow_update_free(pool, BIT(subpage->index), 1); + release = !owner->used && !owner->migrating; + if (release) + owner_remove(owner); + else if (!pool->closed && !owner->migrating && list_empty(&owner->available)) + list_add(&owner->available, &pool->available); + spin_unlock_irqrestore(&pool->lock, flags); + if (release) + owner_free(owner, true); + else + folio_put(folio); +} + +static void subpage_release(struct mm_subpage *subpage) +{ + unsigned long flags; + + spin_lock_irqsave(&subpage->owner->pool->lock, flags); + subpage_release_locked(subpage, flags); +} + +static struct mm_subpage *subpage_copy(struct mm_subpage_pool *pool, + const struct mm_subpage *source, unsigned long eligible, bool lock_folio) +{ + struct mm_subpage *dest; + int err; + + if (pool->shift != mm_subpage_shift(source)) + return ERR_PTR(-EINVAL); + dest = subpage_reserve(pool, eligible, NULL, lock_folio); + + if (IS_ERR(dest)) + return dest; + err = copy_user_subpage_range(&dest->owner->folio->page, mm_subpage_offset(dest), + &source->owner->folio->page, mm_subpage_offset(source), + mm_subpage_size(source)); + if (err) { + /* Never publish a reservation whose copy did not finish. */ + if (lock_folio) + folio_unlock(dest->owner->folio); + subpage_release(dest); + return ERR_PTR(err); + } + refcount_set_release(&dest->refs, 1); + return dest; +} + +struct mm_subpage *mm_subpage_copy(struct mm_subpage_pool *pool, + const struct mm_subpage *source) +{ + return subpage_copy(pool, source, pool->all_slots, false); +} + +struct mm_subpage *mm_subpage_copy_at(struct mm_subpage_pool *pool, + const struct mm_subpage *source, unsigned int offset) +{ + unsigned long eligible = offset_mask(pool->shift, offset); + + if (!eligible) + return ERR_PTR(-EINVAL); + return subpage_copy(pool, source, eligible, false); +} + +struct mm_subpage *mm_subpage_alloc_at_locked(struct mm_subpage_pool *pool, + unsigned int offset) +{ + unsigned long eligible = offset_mask(pool->shift, offset); + + return eligible ? subpage_alloc(pool, eligible, true) : ERR_PTR(-EINVAL); +} + +struct mm_subpage *mm_subpage_copy_at_locked(struct mm_subpage_pool *pool, + const struct mm_subpage *source, unsigned int offset) +{ + unsigned long eligible = offset_mask(pool->shift, offset); + + return eligible ? subpage_copy(pool, source, eligible, true) : ERR_PTR(-EINVAL); +} + +struct mm_subpage *mm_subpage_alloc_locked(struct mm_subpage_pool *pool) +{ + return subpage_alloc(pool, pool->all_slots, true); +} + +struct mm_subpage *mm_subpage_copy_locked(struct mm_subpage_pool *pool, + const struct mm_subpage *source) +{ + return subpage_copy(pool, source, pool->all_slots, true); +} + +/* The page-extension RCU section also protects the pool until its lock is held. */ +static struct mm_subpage *restore_bound_slot(struct folio *folio, unsigned int offset, + unsigned int shift) +{ + struct page_ext *ext = page_ext_get(&folio->page); + struct mm_subpage_owner *owner; + struct mm_subpage_pool *pool; + struct mm_subpage_ext *subext; + struct mm_subpage *slot = NULL; + unsigned int index; + unsigned long flags; + + if (!ext) + return ERR_PTR(-EOPNOTSUPP); + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (!owner) + goto out; + if ((shift && shift != owner->shift) || !offset_mask(owner->shift, offset)) { + slot = ERR_PTR(-EINVAL); + goto out; + } + index = offset >> owner->shift; + pool = owner->pool; + spin_lock_irqsave(&pool->lock, flags); + if (owner->detached) { + slot = ERR_PTR(-EAGAIN); + goto unlock; + } + slot = &owner->slots[index]; + if (test_bit(index, &owner->used)) { + folio_get(folio); + if (!refcount_inc_not_zero_acquire(&slot->refs)) { + /* Its final put or initial publication has not finished. */ + folio_put(folio); + slot = ERR_PTR(-EAGAIN); + } + goto unlock; + } + VM_BUG_ON(refcount_read(&slot->refs) || atomic_read(&slot->mapcount)); + if (pool->unissued == owner) + pool->unissued = NULL; + else + folio_get(folio); + __set_bit(index, &owner->used); + cow_update_free(pool, BIT(index), -1); + __clear_bit(index, &owner->zeroed); + if (owner->used == pool->all_slots) + list_del_init(&owner->available); + /* The folio lock and uptodate state stabilise the restored bytes. */ + refcount_set_release(&slot->refs, 1); +unlock: + spin_unlock_irqrestore(&pool->lock, flags); +out: + page_ext_put(ext); + return slot; +} + +struct mm_subpage *mm_subpage_restore_granule(struct folio *folio, unsigned int offset, + unsigned int shift, gfp_t gfp) +{ + struct mm_subpage_pool *pool; + struct mm_subpage_owner *owner; + struct mm_subpage *slot; + unsigned int index; + int err; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (!subpage_shift_valid(shift) || !offset_mask(shift, offset) || folio_order(folio) || !folio_test_swapcache(folio) || + !folio_test_uptodate(folio) || folio_test_ksm(folio) || + (folio->mapping && !folio_test_anon(folio)) || + (folio_test_anon(folio) && PageAnonExclusive(&folio->page))) + return ERR_PTR(-EINVAL); + index = offset >> shift; + slot = restore_bound_slot(folio, offset, shift); + if (slot) + return slot; + + /* A reclaimed folio has no surviving owner; do not initialise its data. */ + pool = mm_subpage_pool_create_granule(gfp, shift); + if (!pool) + return ERR_PTR(-ENOMEM); + pool->closed = true; + owner = kzalloc(sizeof(*owner), gfp | __GFP_ACCOUNT); + if (!owner) { + mm_subpage_pool_put(pool); + return ERR_PTR(-ENOMEM); + } + owner_init(owner, pool, folio); + owner->used = BIT(index); + refcount_set(&owner->slots[index].refs, 1); + list_add(&owner->all, &pool->owners); + pool->backing_pages = 1; + folio_get(folio); + err = owner_bind(owner); + if (err) { + folio_put(folio); + list_del(&owner->all); + pool->backing_pages = 0; + kfree(owner); + mm_subpage_pool_put(pool); + return ERR_PTR(err == -EEXIST ? -EAGAIN : err); + } + /* The private closed pool's initial reference belongs to this owner. */ + return &owner->slots[index]; +} + +struct mm_subpage *mm_subpage_restore(struct folio *folio, unsigned int offset, gfp_t gfp) +{ + return mm_subpage_restore_granule(folio, offset, MM_SUBPAGE_SHIFT, gfp); +} + +/* Preserve the metadata charge when migration runs outside its owning task. */ +static struct mm_subpage_owner *owner_alloc_migration(struct mm_subpage_owner *owner) +{ + struct mem_cgroup *memcg, *saved; + struct mm_subpage_owner *new; + + rcu_read_lock(); + do { + memcg = mem_cgroup_from_virt(owner) ?: root_mem_cgroup; + } while (memcg && !mem_cgroup_tryget(memcg)); + rcu_read_unlock(); + saved = set_active_memcg(memcg); + /* Folio locks are held: do not recurse into reclaim or trigger memcg OOM. */ + new = kzalloc(sizeof(*new), GFP_NOWAIT | __GFP_NOWARN | __GFP_ACCOUNT); + if (new && owner->rmap) { + new->rmap = kcalloc(PAGE_SIZE >> owner->shift, sizeof(*new->rmap), + GFP_NOWAIT | __GFP_NOWARN | __GFP_ACCOUNT); + if (!new->rmap) { + kfree(new); + new = NULL; + } + } + set_active_memcg(saved); + mem_cgroup_put(memcg); + return new; +} + +/* A migration hold pins metadata, not the native folio: the driver pins it. */ +int mm_subpage_migrate_prepare(struct folio *src, struct folio *dst) +{ + struct page_ext *ext; + struct mm_subpage_ext *subext; + struct mm_subpage_owner *owner, *new; + struct mm_subpage_pool *pool; + unsigned long flags; + int ret = 0; + unsigned int i; + + VM_BUG_ON_FOLIO(!folio_test_locked(src), src); + VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); + ext = page_ext_get(&src->page); + if (!ext) + return 0; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (!owner) { + page_ext_put(ext); + return 0; + } + pool = owner->pool; + spin_lock_irqsave(&pool->lock, flags); + if (owner->detached) { + owner = NULL; + } else if (src == dst || folio_order(src) || folio_order(dst) || + owner->migrating || pool->unissued == owner) { + ret = -EBUSY; + } else { + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) + if (owner->slots[i].pins) + ret = -EBUSY; + if (!ret) + owner->migrating = true; + } + spin_unlock_irqrestore(&pool->lock, flags); + page_ext_put(ext); + if (!owner || ret) + return ret; + + /* Async migration must not recurse into reclaim while holding folio locks. */ + new = owner_alloc_migration(owner); + if (!new) { + ret = -ENOMEM; + goto undo; + } + owner_init(new, pool, dst); + new->migrating = true; + new->zeroed = owner->zeroed; + spin_lock_irqsave(&pool->lock, flags); + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) { + new->slots[i].rmap_bound = owner->slots[i].rmap_bound; + if (new->rmap) { + new->rmap[i] = owner->rmap[i]; + if (new->rmap[i].root) + get_anon_vma(new->rmap[i].root); + } + } + spin_unlock_irqrestore(&pool->lock, flags); + + new->droppable = owner->droppable; + new->anon_root = owner->anon_root; + if (new->anon_root) + get_anon_vma(new->anon_root); + mm_subpage_pool_get(pool); + spin_lock_irqsave(&pool->lock, flags); + list_add_tail(&new->all, &pool->owners); + cow_update_free(pool, pool->all_slots, 1); + pool->backing_pages++; + ret = owner_bind(new); + if (ret) + owner_remove(new); + else + owner->migration_target = new; + spin_unlock_irqrestore(&pool->lock, flags); + if (!ret) + return 0; + if (new->anon_root) + put_anon_vma(new->anon_root); + owner_free_rmap(new); + kfree(new); + mm_subpage_pool_put(pool); +undo: + mm_subpage_migrate_finish(src); + return ret; +} + +bool mm_subpage_migrating(struct folio *folio) +{ + struct page_ext *ext = page_ext_get(&folio->page); + struct mm_subpage_owner *owner; + struct mm_subpage_ext *subext; + bool active = false; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (!ext) + return false; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (owner) + active = READ_ONCE(owner->migrating); + page_ext_put(ext); + return active; +} + +/* Source folio/PTE locked: writable GETs cannot race this final snapshot. */ +void mm_subpage_migrate_save_lazyfree(struct mm_subpage *subpage) +{ + struct mm_subpage_owner *owner = subpage->owner; + + VM_BUG_ON_FOLIO(!owner->migrating || !owner->migration_target, owner->folio); + WRITE_ONCE(owner->migration_target->slots[subpage->index].lazyfree, + READ_ONCE(subpage->lazyfree)); +} + +struct mm_subpage *mm_subpage_migrate_restore(struct folio *folio, unsigned int offset) +{ + struct mm_subpage *slot; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!offset_mask(MM_SUBPAGE_SHIFT, offset) || !mm_subpage_migrating(folio), folio); + /* No allocation or data initialisation can fail during migration recovery. */ + slot = restore_bound_slot(folio, offset, 0); + VM_BUG_ON_FOLIO(IS_ERR_OR_NULL(slot), folio); + return slot; +} + +void mm_subpage_migrate_finish(struct folio *folio) +{ + struct page_ext *ext = page_ext_get(&folio->page); + struct mm_subpage_owner *owner; + struct mm_subpage_ext *subext; + struct mm_subpage_pool *pool; + unsigned long flags; + unsigned int i; + bool release = false; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (!ext) + return; + subext = page_ext_data(ext, &mm_subpage_ext_ops); + owner = rcu_dereference(subext->owner); + if (!owner) + goto out; + pool = owner->pool; + spin_lock_irqsave(&pool->lock, flags); + if (owner->migrating) { + VM_BUG_ON(owner->detached); + owner->migrating = false; + owner->migration_target = NULL; + for (i = 0; i < (PAGE_SIZE >> owner->shift); i++) + if (!(owner->used & BIT(i))) + owner->slots[i].rmap_bound = false; + release = !owner->used; + if (release) + owner_remove(owner); + else if (!pool->closed && owner->used != pool->all_slots && + list_empty(&owner->available)) + list_add(&owner->available, &pool->available); + } + spin_unlock_irqrestore(&pool->lock, flags); +out: + page_ext_put(ext); + if (release) + /* No slot references remain; only the migration driver pins backing. */ + owner_free(owner, false); +} + +void mm_subpage_get(struct mm_subpage *subpage) +{ + folio_get(subpage->owner->folio); + refcount_inc(&subpage->refs); +} + +void mm_subpage_put(struct mm_subpage *subpage) +{ + struct mm_subpage_owner *owner = subpage->owner; + struct mm_subpage_pool *pool = owner->pool; + struct folio *folio = owner->folio; + unsigned long flags; + + /* + * Snapshot the backing while our slot reference still retains owner. + * A non-final decrement can race the final put and metadata reclamation; + * only our physical folio reference survives that decrement. + */ + if (refcount_dec_and_lock_irqsave(&subpage->refs, &pool->lock, &flags)) + subpage_release_locked(subpage, flags); + else + folio_put(folio); +} + +void mm_subpage_set_exclusive(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + + VM_BUG_ON_FOLIO(!folio_test_locked(subpage->owner->folio), subpage->owner->folio); + spin_lock_irqsave(&pool->lock, flags); + VM_BUG_ON(atomic_read(&subpage->mapcount) > 1 || !refcount_read(&subpage->refs)); + WRITE_ONCE(subpage->exclusive, true); + spin_unlock_irqrestore(&pool->lock, flags); +} + +bool mm_subpage_is_exclusive(const struct mm_subpage *subpage) +{ + return READ_ONCE(subpage->exclusive); +} + +static bool subpage_try_reuse(struct mm_subpage *subpage, unsigned int references) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + bool reusable; + + VM_BUG_ON_FOLIO(!folio_test_locked(subpage->owner->folio), subpage->owner->folio); + spin_lock_irqsave(&pool->lock, flags); + reusable = atomic_read(&subpage->mapcount) == 1 && + refcount_read(&subpage->refs) == references && !subpage->pins; + if (reusable) + WRITE_ONCE(subpage->exclusive, true); + spin_unlock_irqrestore(&pool->lock, flags); + return reusable; +} + +bool mm_subpage_try_reuse_swap(struct mm_subpage *subpage) +{ + return subpage_try_reuse(subpage, 1); +} + +bool mm_subpage_try_reuse_anon(struct mm_subpage *subpage) +{ + return subpage_try_reuse(subpage, 2); +} + +bool mm_subpage_can_zap(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + bool eligible; + + spin_lock_irqsave(&pool->lock, flags); + eligible = !subpage->owner->migrating && !subpage->pins && + atomic_read(&subpage->mapcount) == 1 && + refcount_read(&subpage->refs) == 2; + spin_unlock_irqrestore(&pool->lock, flags); + return eligible; +} + +bool mm_subpage_try_share(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + bool shared; + + spin_lock_irqsave(&pool->lock, flags); + shared = !subpage->pins; + if (shared) + WRITE_ONCE(subpage->exclusive, false); + spin_unlock_irqrestore(&pool->lock, flags); + return shared; +} + +bool mm_subpage_is_lazyfree(const struct mm_subpage *subpage) +{ + return READ_ONCE(subpage->lazyfree); +} + +void mm_subpage_cancel_lazyfree(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + + spin_lock_irqsave(&pool->lock, flags); + WRITE_ONCE(subpage->lazyfree, false); + spin_unlock_irqrestore(&pool->lock, flags); +} + +bool mm_subpage_mark_lazyfree(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + bool eligible; + + VM_BUG_ON_FOLIO(!folio_test_locked(subpage->owner->folio), subpage->owner->folio); + spin_lock_irqsave(&pool->lock, flags); + eligible = atomic_read(&subpage->mapcount) == 1 && + refcount_read(&subpage->refs) == 2 && !subpage->pins; + if (eligible) + WRITE_ONCE(subpage->lazyfree, true); + spin_unlock_irqrestore(&pool->lock, flags); + return eligible; +} + +bool mm_subpage_all_lazyfree(struct mm_subpage *subpage) +{ + struct mm_subpage_owner *owner = subpage->owner; + unsigned long flags; + unsigned int i; + bool all = true; + + spin_lock_irqsave(&owner->pool->lock, flags); + for (i = 0; i < SUBPAGES_PER_FOLIO; i++) { + if (!(owner->used & BIT(i))) + continue; + if (!READ_ONCE(owner->slots[i].lazyfree)) { + all = false; + break; + } + } + spin_unlock_irqrestore(&owner->pool->lock, flags); + return all; +} + +bool mm_subpage_discardable(struct mm_subpage *subpage) +{ + struct mm_subpage_pool *pool = subpage->owner->pool; + unsigned long flags; + bool eligible; + + spin_lock_irqsave(&pool->lock, flags); + eligible = (subpage->owner->droppable || READ_ONCE(subpage->lazyfree)) && + !subpage->pins && + refcount_read(&subpage->refs) == atomic_read(&subpage->mapcount) + 1; + spin_unlock_irqrestore(&pool->lock, flags); + return eligible; +} + +int mm_subpage_pin(struct mm_subpage *subpage) +{ + struct mm_subpage_owner *owner = subpage->owner; + unsigned long flags; + int ret = -EAGAIN; + + spin_lock_irqsave(&owner->pool->lock, flags); + if (!owner->anon_root || owner->migrating || !subpage->exclusive || + !atomic_read(&subpage->mapcount)) + goto out; + ret = try_grab_folio(owner->folio, 1, FOLL_PIN); + if (ret) + goto out; + mm_subpage_get(subpage); + subpage->pins++; +out: + spin_unlock_irqrestore(&owner->pool->lock, flags); + return ret; +} + +void mm_subpage_unpin(struct mm_subpage *subpage) +{ + struct mm_subpage_owner *owner = subpage->owner; + unsigned long flags; + + spin_lock_irqsave(&owner->pool->lock, flags); + VM_BUG_ON(!subpage->pins || !subpage->exclusive); + subpage->pins--; + spin_unlock_irqrestore(&owner->pool->lock, flags); + unpin_folio(owner->folio); + mm_subpage_put(subpage); +} + +unsigned int mm_subpage_pincount(const struct mm_subpage *subpage) +{ + return READ_ONCE(subpage->pins); +} + +int mm_subpage_mapcount(const struct mm_subpage *subpage) +{ + return atomic_read(&subpage->mapcount); +} + +struct folio *mm_subpage_folio(const struct mm_subpage *subpage) +{ + return subpage->owner->folio; +} + +unsigned int mm_subpage_shift(const struct mm_subpage *subpage) +{ + return subpage->shift; +} + +unsigned int mm_subpage_size(const struct mm_subpage *subpage) +{ + return 1U << subpage->shift; +} + +unsigned int mm_subpage_offset(const struct mm_subpage *subpage) +{ + return subpage->index << subpage->shift; +} + +phys_addr_t mm_subpage_phys(const struct mm_subpage *subpage) +{ + return PFN_PHYS(folio_pfn(subpage->owner->folio)) + mm_subpage_offset(subpage); +} + +/* + * Round the mm's aggregate reservation, never each VMA: splitting a VMA + * must not create extra charges or refunds. mmap writers serialize updates; + * munmap may retain the lock downgraded while finishing its refund. + */ +static int mm_commit_delta(struct mm_struct *mm, long pages, + unsigned long *next, long *delta) +{ + unsigned long old = mm->committed_user_pages; + unsigned int shift = mm_page_shift(mm); + + mmap_assert_locked(mm); + if (pages < 0 && WARN_ON_ONCE(pages < -(long)old)) + return -EINVAL; + if (pages > 0 && (unsigned long)pages > LONG_MAX - old) + return -ENOMEM; + *next = old + pages; + if (shift > PAGE_SHIFT) { + unsigned int extra = shift - PAGE_SHIFT; + + /* A logical reservation may span several native commit units. */ + if (*next > (LONG_MAX >> extra) || old > (LONG_MAX >> extra)) + return -ENOMEM; + *delta = (long)(*next << extra) - (long)(old << extra); + } else { + unsigned long per_native = 1UL << (PAGE_SHIFT - shift); + + *delta = DIV_ROUND_UP(*next, per_native) - DIV_ROUND_UP(old, per_native); + } + return 0; +} + +int mm_account_memory(struct mm_struct *mm, struct mm_struct *policy_mm, long pages) +{ + unsigned long next; + long delta; + int ret; + + if (mm_page_size(mm) == PAGE_SIZE) + return security_vm_enough_memory_mm(policy_mm, pages); + ret = mm_commit_delta(mm, pages, &next, &delta); + if (ret) + return ret; + ret = security_vm_enough_memory_mm(policy_mm, delta); + if (!ret) + mm->committed_user_pages = next; + return ret; +} + +void mm_acct_memory(struct mm_struct *mm, long pages) +{ + unsigned long next; + long delta; + + if (mm_page_size(mm) == PAGE_SIZE) { + vm_acct_memory(pages); + return; + } + if (WARN_ON_ONCE(mm_commit_delta(mm, pages, &next, &delta))) + return; + vm_acct_memory(delta); + mm->committed_user_pages = next; +} + +void mm_unacct_memory(struct mm_struct *mm, unsigned long pages) +{ + mm_acct_memory(mm, -(long)pages); +} + +/* Shared ownership for consumers whose native page spans contain user slots. */ +struct mm_subpage_refs { + refcount_t refs; + unsigned int count; + unsigned int capacity; + struct mm_subpage *slots[]; +}; + +struct mm_subpage_refs *mm_subpage_refs_get(struct mm_subpage_refs *refs) +{ + if (refs) + refcount_inc(&refs->refs); + return refs; +} +EXPORT_SYMBOL_GPL(mm_subpage_refs_get); + +void mm_subpage_refs_put(struct mm_subpage_refs *refs) +{ + unsigned int i; + + if (!refs || !refcount_dec_and_test(&refs->refs)) + return; + for (i = 0; i < refs->count; i++) + mm_subpage_put(refs->slots[i]); + kfree(refs); +} +EXPORT_SYMBOL_GPL(mm_subpage_refs_put); + +static struct mm_subpage_refs *subpage_refs_new(unsigned int capacity, gfp_t gfp) +{ + struct mm_subpage_refs *refs = kmalloc(struct_size(refs, slots, capacity), gfp); + + if (!refs) + return NULL; + refcount_set(&refs->refs, 1); + refs->count = 0; + refs->capacity = capacity; + return refs; +} + +static int subpage_refs_prepare(struct mm_subpage_refs **refsp, + unsigned int needed, gfp_t gfp) +{ + struct mm_subpage_refs *old = *refsp, *refs; + unsigned int i; + + if (old && refcount_read(&old->refs) == 1 && old->capacity >= needed) + return 0; + refs = subpage_refs_new(max(needed, 4U), gfp); + if (!refs) + return -ENOMEM; + if (old) { + for (i = 0; i < old->count; i++) { + mm_subpage_get(old->slots[i]); + refs->slots[refs->count++] = old->slots[i]; + } + } + *refsp = refs; + mm_subpage_refs_put(old); + return 0; +} + +int mm_subpage_refs_add(struct mm_subpage_refs **refsp, struct page *page, + unsigned int offset, unsigned int len, gfp_t gfp) +{ + struct mm_subpage *slots[PAGE_SIZE / MM_SUBPAGE_SIZE]; + struct mm_subpage_refs *refs = *refsp; + unsigned int pos, count = 0, i, j; + int ret = 0; + bool found = false; + + if (!len || folio_order(page_folio(page)) || !folio_test_anon(page_folio(page))) + return 0; + if (offset >= PAGE_SIZE || len > PAGE_SIZE - offset) + return -EINVAL; + for (pos = round_down(offset, MM_SUBPAGE_SIZE); pos < offset + len; + pos += MM_SUBPAGE_SIZE) { + struct mm_subpage *slot = mm_subpage_get_from_phys(page_to_phys(page) + pos); + + if (!slot) { + /* A native anonymous allocation has no quarter owner. */ + ret = found ? -EFAULT : 0; + goto out; + } + found = true; + for (j = 0; refs && j < refs->count; j++) + if (refs->slots[j] == slot) + break; + if (refs && j < refs->count) { + mm_subpage_put(slot); + continue; + } + for (j = 0; j < count; j++) + if (slots[j] == slot) + break; + if (j < count) + mm_subpage_put(slot); + else + slots[count++] = slot; + } + if (!count) + return 0; + if (refs && refs->count > UINT_MAX - count) { + ret = -ENOMEM; + goto out; + } + ret = subpage_refs_prepare(refsp, (refs ? refs->count : 0) + count, gfp); + if (ret) + goto out; + refs = *refsp; + for (i = 0; i < count; i++) + refs->slots[refs->count++] = slots[i]; + return 0; +out: + for (i = 0; i < count; i++) + mm_subpage_put(slots[i]); + return ret; +} +EXPORT_SYMBOL_GPL(mm_subpage_refs_add); + +int mm_subpage_refs_prune(struct mm_subpage_refs **refsp, + bool (*keep)(const struct mm_subpage *, void *), + void *context, gfp_t gfp) +{ + struct mm_subpage_refs *refs = *refsp, *new; + unsigned int i, count = 0; + + if (!refs) + return 0; + for (i = 0; i < refs->count; i++) + count += keep(refs->slots[i], context); + if (count == refs->count) + return 0; + if (!count) { + *refsp = NULL; + mm_subpage_refs_put(refs); + return 0; + } + if (refcount_read(&refs->refs) == 1) { + for (i = 0, count = 0; i < refs->count; i++) { + if (keep(refs->slots[i], context)) + refs->slots[count++] = refs->slots[i]; + else + mm_subpage_put(refs->slots[i]); + } + refs->count = count; + return 0; + } + new = subpage_refs_new(count, gfp); + if (!new) + return -ENOMEM; + for (i = 0; i < refs->count; i++) { + if (!keep(refs->slots[i], context)) + continue; + mm_subpage_get(refs->slots[i]); + new->slots[new->count++] = refs->slots[i]; + } + *refsp = new; + mm_subpage_refs_put(refs); + return 0; +} +EXPORT_SYMBOL_GPL(mm_subpage_refs_prune); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 01b151dca3085d..e7327a67499629 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -16,10 +16,19 @@ #include #include #include +#include +#include #include #include #include "internal.h" #include "swap.h" +#ifdef CONFIG_MM_SUBPAGE +#include "user-subpage-internal.h" +#endif + +struct mfill_shared_data { + struct folio *source; /* NULL requests a zero-fill. */ +}; struct mfill_state { struct userfaultfd_ctx *ctx; @@ -32,6 +41,10 @@ struct mfill_state { unsigned long src_addr; unsigned long dst_addr; pmd_t *pmd; + struct folio *copy_buffer; +#ifdef CONFIG_MM_SUBPAGE + struct mm_subpage *slots[MM_SUBPAGES_PER_PAGE]; +#endif }; static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags) @@ -194,8 +207,24 @@ static void uffd_mfill_unlock(struct vm_area_struct *vma) } #endif +/* Reservations keep the uninstalled leaves private to this operation. */ +static void mfill_free_subpages(struct mfill_state *state) +{ +#ifdef CONFIG_MM_SUBPAGE + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(state->slots); i++) { + if (state->slots[i]) { + mm_subpage_put(state->slots[i]); + state->slots[i] = NULL; + } + } +#endif +} + static void mfill_put_vma(struct mfill_state *state) { + mfill_free_subpages(state); if (!state->vma) return; @@ -248,6 +277,7 @@ static int mfill_get_vma(struct mfill_state *state) if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP)) goto out_unlock; + if (is_vm_hugetlb_page(dst_vma)) return 0; @@ -320,16 +350,11 @@ static int mfill_establish_pmd(struct mfill_state *state) static bool mfill_file_over_size(struct vm_area_struct *dst_vma, unsigned long dst_addr) { - struct inode *inode; - pgoff_t offset, max_off; - if (!dst_vma->vm_file) return false; - inode = dst_vma->vm_file->f_inode; - offset = linear_page_index(dst_vma, dst_addr); - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - return offset >= max_off; + return !vma_file_offset_valid(dst_vma, dst_addr, + i_size_read(file_inode(dst_vma->vm_file))); } /* @@ -341,7 +366,8 @@ static bool mfill_file_over_size(struct vm_area_struct *dst_vma, static int mfill_atomic_install_pte(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, struct page *page, - uffd_flags_t flags) + uffd_flags_t flags, + const struct mfill_shared_data *shared_data) { int ret; struct mm_struct *dst_mm = dst_vma->vm_mm; @@ -354,6 +380,11 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd, pte_t dst_ptep; _dst_pte = mk_pte(page, dst_vma->vm_page_prot); +#ifdef CONFIG_MM_SUBPAGE + if (page_in_cache && mm_page_size(dst_mm) < PAGE_SIZE) + _dst_pte = phys_pte_mm(dst_mm, page_to_phys(page) + + vma_page_offset_at(dst_vma, dst_addr).offset, dst_vma->vm_page_prot); +#endif _dst_pte = pte_mkdirty(_dst_pte); if (page_in_cache && !vm_shared) writable = false; @@ -384,7 +415,39 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd, if (!pte_none(dst_ptep) && !pte_is_uffd_marker(dst_ptep)) goto out_unlock; +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SHMEM) + if (shared_data) { + struct vm_page_offset pos = vma_page_offset_at(dst_vma, dst_addr); + unsigned long offset = ((pos.index - folio->index) << PAGE_SHIFT) + pos.offset; + unsigned long size = mm_page_size(dst_mm); + void *dst, *src; + + if (PageHWPoison(page)) { + ret = -EIO; + goto out_unlock; + } + if (!shmem_uffd_range_missing(folio, pos.index, pos.offset, size)) + goto out_unlock; + dst = kmap_local_folio(folio, offset); + if (shared_data->source) { + src = kmap_local_folio(shared_data->source, 0); + memcpy(dst, src, size); + kunmap_local(src); + } else { + memset(dst, 0, size); + } + kunmap_local(dst); + flush_dcache_folio(folio); + folio_mark_uptodate(folio); + folio_mark_dirty(folio); + } +#endif + if (page_in_cache) { + mm_subpage_file_map_add(dst_vma, pte_phys_mm(dst_mm, _dst_pte)); + shmem_uffd_mark_mapped(folio, + (folio_page_idx(folio, page) << PAGE_SHIFT) + + vma_page_offset_at(dst_vma, dst_addr).offset, mm_page_size(dst_mm)); folio_add_file_rmap_pte(folio, page, dst_vma); } else { folio_add_new_anon_rmap(folio, dst_vma, dst_addr, RMAP_EXCLUSIVE); @@ -411,12 +474,13 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd, return ret; } -static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) +static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr, + unsigned int offset, unsigned long len) { void *kaddr; int ret; - kaddr = kmap_local_folio(folio, 0); + kaddr = kmap_local_folio(folio, offset); /* * The read mmap_lock is held here. Despite the * mmap_lock being read recursive a deadlock is still @@ -433,8 +497,7 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr) * and retry the copy outside the mmap_lock. */ pagefault_disable(); - ret = copy_from_user(kaddr, (const void __user *) src_addr, - PAGE_SIZE); + ret = copy_from_user(kaddr, (const void __user *)src_addr, len); pagefault_enable(); kunmap_local(kaddr); @@ -507,8 +570,11 @@ DEFINE_FREE(retry_put, struct mfill_retry_state *, if (_T) mfill_retry_state_put(_T)); static int mfill_copy_folio_retry(struct mfill_state *mfill_state, - struct folio *folio) + struct folio *folio, unsigned int offset, + unsigned long len) { + struct vm_page_offset pos = vma_page_offset_at(mfill_state->vma, + mfill_state->dst_addr); struct mfill_retry_state retry_state = { 0 }; struct mfill_retry_state *for_free __free(retry_put) = &retry_state; unsigned long src_addr = mfill_state->src_addr; @@ -520,8 +586,8 @@ static int mfill_copy_folio_retry(struct mfill_state *mfill_state, /* retry copying with mm_lock dropped */ mfill_put_vma(mfill_state); - kaddr = kmap_local_folio(folio, 0); - err = copy_from_user(kaddr, (const void __user *) src_addr, PAGE_SIZE); + kaddr = kmap_local_folio(folio, offset); + err = copy_from_user(kaddr, (const void __user *) src_addr, len); kunmap_local(kaddr); if (unlikely(err)) return -EFAULT; @@ -533,7 +599,10 @@ static int mfill_copy_folio_retry(struct mfill_state *mfill_state, if (err) return err; - if (mfill_retry_state_changed(&retry_state, mfill_state->vma)) + if (mfill_retry_state_changed(&retry_state, mfill_state->vma) || + (mm_page_size(mfill_state->ctx->mm) < PAGE_SIZE && + !vm_page_offset_equal(pos, vma_page_offset_at(mfill_state->vma, + mfill_state->dst_addr)))) return -EAGAIN; err = mfill_establish_pmd(mfill_state); @@ -543,6 +612,28 @@ static int mfill_copy_folio_retry(struct mfill_state *mfill_state, return 0; } +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SHMEM) +/* Fill one missing leaf in new or retained shmem backing using staged data. */ +static int mfill_atomic_shmem_hole(struct mfill_state *state, + const struct mfill_shared_data *data) +{ + struct vm_page_offset pos; + struct folio *folio; + bool new; + int err; + + folio = shmem_uffd_prepare_folio(state->vma, state->dst_addr, &new); + if (IS_ERR(folio)) + return PTR_ERR(folio); + pos = vma_page_offset_at(state->vma, state->dst_addr); + err = mfill_atomic_install_pte(state->pmd, state->vma, state->dst_addr, + folio_file_page(folio, pos.index), state->flags, data); + if (err) + shmem_uffd_abort_folio(folio, state->vma, new); + return err; +} +#endif + static int __mfill_atomic_pte(struct mfill_state *state, const struct vm_uffd_ops *ops) { @@ -562,7 +653,7 @@ static int __mfill_atomic_pte(struct mfill_state *state, return -ENOMEM; if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY)) { - ret = mfill_copy_folio_locked(folio, src_addr); + ret = mfill_copy_folio_locked(folio, src_addr, 0, PAGE_SIZE); /* * Fallback to copy_from_user outside mmap_lock. * If retry is successful, mfill_copy_folio_locked() returns @@ -571,7 +662,7 @@ static int __mfill_atomic_pte(struct mfill_state *state, * will take care of unlocking if needed. */ if (unlikely(ret)) { - ret = mfill_copy_folio_retry(state, folio); + ret = mfill_copy_folio_retry(state, folio, 0, PAGE_SIZE); if (ret) goto err_folio_put; } @@ -590,12 +681,29 @@ static int __mfill_atomic_pte(struct mfill_state *state, if (ops->filemap_add) { ret = ops->filemap_add(folio, state->vma, state->dst_addr); +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SHMEM) + /* + * A pinned large shmem folio can survive a partial hole punch. + * Its cache entry exists while this native-sized range is missing. + * Reuse the prepared source and preserve all neighbouring bytes. + * Ordinary native insertion and non-shmem paths stay unchanged. + */ + if (ret == -EEXIST && (state->vma->vm_flags & VM_SHARED) && + shmem_file(state->vma->vm_file) && + (READ_ONCE(SHMEM_I(file_inode(state->vma->vm_file))->flags) & + SHMEM_F_UFFD_SUBPAGE)) { + struct mfill_shared_data data = { .source = folio }; + + ret = mfill_atomic_shmem_hole(state, &data); + goto err_folio_put; + } +#endif if (ret) goto err_folio_put; } ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr, - &folio->page, flags); + &folio->page, flags, NULL); if (ret) goto err_filemap_remove; @@ -609,10 +717,184 @@ static int __mfill_atomic_pte(struct mfill_state *state, return ret; } +#ifdef CONFIG_MM_SUBPAGE +/* Pack only this request's leaves in one native linear index. */ +static struct mm_subpage *mfill_alloc_subpage(struct mfill_state *state) +{ + struct vm_area_struct *vma = state->vma; + struct mm_struct *mm = vma->vm_mm; + struct vm_page_offset pos = vma_page_offset_at(vma, state->dst_addr); + unsigned long size = mm_page_size(mm); + unsigned int first = pos.offset >> mm_page_shift(mm), i, nr; + struct mm_subpage_pool *pool; + struct mm_subpage *slot; + struct folio *folio; + int err = -ENOMEM; + + if (state->slots[first]) + goto take; + + mfill_free_subpages(state); + pool = mm_subpage_pool_create_granule(GFP_KERNEL, mm_page_shift(mm)); + if (!pool) + return ERR_PTR(-ENOMEM); + folio = anon_alloc_folio(vma, state->dst_addr); + if (!folio) + goto out; + folio_throttle_swaprate(folio, GFP_KERNEL); + err = mm_subpage_pool_add_folio(pool, folio, GFP_KERNEL); + if (err) { + folio_put(folio); + goto out; + } + nr = min_t(unsigned long, PAGE_SIZE - pos.offset, + state->dst_start + state->len - state->dst_addr) / size; + for (i = first; i < first + nr; i++) { + slot = mm_subpage_alloc_at(pool, i * size); + if (IS_ERR(slot)) { + err = PTR_ERR(slot); + mfill_free_subpages(state); + goto out; + } + state->slots[i] = slot; + } +out: + mm_subpage_pool_close(pool); + mm_subpage_pool_put(pool); + if (err) + return ERR_PTR(err); +take: + slot = state->slots[first]; + state->slots[first] = NULL; + return slot; +} + +static int mfill_atomic_subpage(struct mfill_state *state) +{ + struct mm_struct *mm = state->ctx->mm; + struct mm_subpage *slot = mfill_alloc_subpage(state); + struct vm_area_struct *vma; + struct folio *folio; + pte_t entry, *pte; + spinlock_t *ptl; + bool fresh; + int err; + + if (IS_ERR(slot)) + return PTR_ERR(slot); + folio = mm_subpage_folio(slot); + folio_lock(folio); + /* A previous leaf may have entered reclaim between iterations. */ + err = -EAGAIN; + if (folio_test_swapcache(folio)) + goto unlock_folio; + if (uffd_flags_mode_is(state->flags, MFILL_ATOMIC_COPY)) { + err = mfill_copy_folio_locked(folio, state->src_addr, + mm_subpage_offset(slot), mm_page_size(mm)); + if (err) { + /* + * The faulting retry must not hold the folio lock, or write + * a folio whose already-visible neighbours can be reclaimed. + * Use fresh backing and discard the remaining reservations + * before dropping the VMA lock. + */ + folio_unlock(folio); + mm_subpage_put(slot); + mfill_free_subpages(state); + slot = mfill_alloc_subpage(state); + if (IS_ERR(slot)) + return PTR_ERR(slot); + folio = mm_subpage_folio(slot); + err = mfill_copy_folio_retry(state, folio, + mm_subpage_offset(slot), mm_page_size(mm)); + if (err) + goto put_slot; + folio_lock(folio); + } + } + /* The allocator zeroed every byte, including unissued neighbours. */ + folio_mark_uptodate(folio); + vma = state->vma; + pte = pte_offset_map_lock(mm, state->pmd, state->dst_addr, &ptl); + err = -EAGAIN; + if (!pte) + goto unlock_folio; + err = -EFAULT; + if (mfill_file_over_size(vma, state->dst_addr)) + goto unlock_pte; + err = -EEXIST; + if (!pte_none(ptep_get(pte)) && !pte_is_uffd_marker(ptep_get(pte))) + goto unlock_pte; + fresh = !folio_test_anon(folio); + mm_subpage_set_exclusive(slot); + err = mm_subpage_add_new_anon_rmap(slot, vma, state->dst_addr); + if (err) + goto unlock_pte; + if (fresh) + folio_add_lru_vma(folio, vma); + else if ((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED) + mlock_folio(folio); + inc_mm_counter(mm, MM_ANONPAGES); + entry = phys_pte_mm(mm, mm_subpage_phys(slot), vma->vm_page_prot); + entry = pte_mkdirty(entry); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(entry, vma); + if (state->flags & MFILL_ATOMIC_WP) + entry = pte_mkuffd_wp(entry); + set_pte_at(mm, state->dst_addr, pte, entry); + update_mmu_cache(vma, state->dst_addr, pte); + /* The PTE owns slot now; release the folio lock before dropping PTL. */ + folio_unlock(folio); + pte_unmap_unlock(pte, ptl); + return 0; +unlock_pte: + pte_unmap_unlock(pte, ptl); +unlock_folio: + folio_unlock(folio); +put_slot: + mm_subpage_put(slot); + return err; +} +#endif + +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SHMEM) +static int mfill_atomic_shared_subpage(struct mfill_state *state) +{ + struct mfill_shared_data data = {}; + int err; + + if (uffd_flags_mode_is(state->flags, MFILL_ATOMIC_COPY)) { + if (!state->copy_buffer) + state->copy_buffer = folio_alloc(GFP_KERNEL_ACCOUNT, 0); + data.source = state->copy_buffer; + if (!data.source) + return -ENOMEM; + err = mfill_copy_folio_locked(data.source, state->src_addr, 0, + mm_page_size(state->ctx->mm)); + if (err) + err = mfill_copy_folio_retry(state, data.source, 0, + mm_page_size(state->ctx->mm)); + if (err) + return err; + } + return mfill_atomic_shmem_hole(state, &data); +} +#endif + static int mfill_atomic_pte_copy(struct mfill_state *state) { const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(state->ctx->mm) < PAGE_SIZE) { +#ifdef CONFIG_SHMEM + if (state->vma->vm_flags & VM_SHARED) + return mfill_atomic_shared_subpage(state); +#endif + return mfill_atomic_subpage(state); + } +#endif + /* * The normal page fault path for a MAP_PRIVATE mapping in a * file-backed VMA will invoke the fault, fill the hole in the file and @@ -632,6 +914,16 @@ static int mfill_atomic_pte_zeroed_folio(struct mfill_state *state) { const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma); +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(state->ctx->mm) < PAGE_SIZE) { +#ifdef CONFIG_SHMEM + if (state->vma->vm_flags & VM_SHARED) + return mfill_atomic_shared_subpage(state); +#endif + return mfill_atomic_subpage(state); + } +#endif + return __mfill_atomic_pte(state, ops); } @@ -695,6 +987,13 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) if (IS_ERR_OR_NULL(folio)) return -EFAULT; +#if defined(CONFIG_MM_SUBPAGE) && defined(CONFIG_SHMEM) + if (vma_is_shmem(dst_vma) && shmem_uffd_range_missing(folio, pgoff, + vma_page_offset_at(dst_vma, dst_addr).offset, mm_page_size(dst_vma->vm_mm))) { + ret = -EFAULT; + goto out_release; + } +#endif page = folio_file_page(folio, pgoff); if (PageHWPoison(page)) { ret = -EIO; @@ -702,7 +1001,7 @@ static int mfill_atomic_pte_continue(struct mfill_state *state) } ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr, - page, flags); + page, flags, NULL); if (ret) goto out_release; @@ -964,8 +1263,8 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, /* * Sanitize the command parameters: */ - VM_WARN_ON_ONCE(dst_start & ~PAGE_MASK); - VM_WARN_ON_ONCE(len & ~PAGE_MASK); + VM_WARN_ON_ONCE(dst_start & ~mm_page_mask(ctx->mm)); + VM_WARN_ON_ONCE(len & ~mm_page_mask(ctx->mm)); /* Does the address range wrap, or is the span zero-sized? */ VM_WARN_ON_ONCE(src_start + len <= src_start); @@ -998,9 +1297,9 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, cond_resched(); if (!err) { - state.dst_addr += PAGE_SIZE; - state.src_addr += PAGE_SIZE; - copied += PAGE_SIZE; + state.dst_addr += mm_page_size(ctx->mm); + state.src_addr += mm_page_size(ctx->mm); + copied += mm_page_size(ctx->mm); if (fatal_signal_pending(current)) err = -EINTR; @@ -1011,6 +1310,8 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, mfill_put_vma(&state); out: + if (state.copy_buffer) + folio_put(state.copy_buffer); VM_WARN_ON_ONCE(copied < 0); VM_WARN_ON_ONCE(err > 0); VM_WARN_ON_ONCE(!copied && !err); @@ -1099,8 +1400,8 @@ int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, /* * Sanitize the command parameters: */ - VM_WARN_ON_ONCE(start & ~PAGE_MASK); - VM_WARN_ON_ONCE(len & ~PAGE_MASK); + VM_WARN_ON_ONCE(start & ~mm_page_mask(dst_mm)); + VM_WARN_ON_ONCE(len & ~mm_page_mask(dst_mm)); /* Does the address range wrap, or is the span zero-sized? */ VM_WARN_ON_ONCE(start + len <= start); @@ -1303,7 +1604,102 @@ static long move_present_ptes(struct mm_struct *mm, return src_addr > src_start ? src_addr - src_start : err; } +#ifdef CONFIG_MM_SUBPAGE +static long move_present_subpage(struct mm_struct *mm, + struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, + unsigned long dst_addr, unsigned long src_addr, + pte_t *dst_pte, pte_t *src_pte, pte_t orig_dst, pte_t orig_src, + pmd_t *dst_pmd, pmd_t dst_pmdval, + spinlock_t *dst_ptl, spinlock_t *src_ptl, struct folio *folio) +{ + struct mm_subpage_move move; + struct mm_subpage *slot = NULL; + pte_t pte; + long ret = -EAGAIN; + + flush_cache_range(src_vma, src_addr, src_addr + mm_page_size(mm)); + double_pt_lock(dst_ptl, src_ptl); + if (!is_pte_pages_stable(dst_pte, src_pte, orig_dst, orig_src, + dst_pmd, dst_pmdval)) + goto out; + ret = -EBUSY; + if (!folio_test_anon(folio) || folio_test_large(folio)) + goto out; + slot = mm_subpage_get_from_phys(pte_phys_mm(mm, orig_src)); + if (!slot || mm_subpage_folio(slot) != folio) + goto out; + ret = mm_subpage_move_prepare(&move, slot, src_vma, src_addr, + dst_vma, dst_addr, false); + if (ret) + goto out; + orig_src = ptep_clear_flush(src_vma, src_addr, src_pte); + ret = mm_subpage_move_commit(&move); + if (ret) { + set_pte_at(mm, src_addr, src_pte, orig_src); + } else { + pte = phys_pte_mm(mm, mm_subpage_phys(slot), dst_vma->vm_page_prot); + if (pgtable_supports_soft_dirty()) + pte = pte_mksoft_dirty(pte); + if (pte_dirty(orig_src)) + pte = pte_mkdirty(pte); + pte = pte_mkwrite(pte, dst_vma); + set_pte_at(mm, dst_addr, dst_pte, pte); + ret = mm_page_size(mm); + } + mm_subpage_move_finish(&move); +out: + if (slot) + mm_subpage_put(slot); + double_pt_unlock(dst_ptl, src_ptl); + return ret; +} + +/* Both PTLs and, if present, the swap-cache folio lock are held. */ +static int move_swap_subpage(struct mm_struct *mm, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, unsigned long dst_addr, + unsigned long src_addr, pte_t *dst_pte, pte_t *src_pte, + pte_t orig_src, struct folio *folio, swp_entry_t entry) +{ + unsigned int offset = pte_swp_subpage_offset(orig_src); + struct mm_subpage_move move; + struct mm_subpage *slot = NULL; + int ret = -EBUSY; + + if (offset >= PAGE_SIZE || !IS_ALIGNED(offset, mm_page_size(mm))) + return -EFAULT; + if (swap_subpage_count(entry, offset) != 1) + return -EBUSY; + if (folio) { + slot = mm_subpage_get_from_phys(PFN_PHYS(folio_pfn(folio)) + offset); + if (slot) { + ret = mm_subpage_move_prepare(&move, slot, src_vma, src_addr, + dst_vma, dst_addr, true); + if (ret) + goto out; + ret = mm_subpage_move_commit(&move); + if (ret) { + mm_subpage_move_finish(&move); + goto out; + } + } + } + /* No mapping count changes: the same-mm exclusive swap PTE is transferred. */ + orig_src = ptep_get_and_clear(mm, src_addr, src_pte); + if (pgtable_supports_soft_dirty()) + orig_src = pte_swp_mksoft_dirty(orig_src); + set_pte_at(mm, dst_addr, dst_pte, orig_src); + if (slot) + mm_subpage_move_finish(&move); + ret = mm_page_size(mm); +out: + if (slot) + mm_subpage_put(slot); + return ret; +} +#endif + static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, unsigned long dst_addr, unsigned long src_addr, pte_t *dst_pte, pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, @@ -1329,6 +1725,21 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, return -EAGAIN; } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + int ret; + + if (!src_folio && swap_cache_has_folio(entry)) { + double_pt_unlock(dst_ptl, src_ptl); + return -EAGAIN; + } + ret = move_swap_subpage(mm, dst_vma, src_vma, dst_addr, src_addr, + dst_pte, src_pte, orig_src_pte, src_folio, entry); + double_pt_unlock(dst_ptl, src_ptl); + return ret; + } +#endif + /* * The src_folio resides in the swapcache, requiring an update to its * index and mapping to align with the dst_vma, where a swap-in may @@ -1381,13 +1792,13 @@ static int move_zeropage_pte(struct mm_struct *mm, return -EAGAIN; } - zero_pte = pte_mkspecial(pfn_pte(zero_pfn(dst_addr), - dst_vma->vm_page_prot)); + zero_pte = pte_mkspecial(phys_pte_mm(mm, PFN_PHYS(zero_pfn(dst_addr)), + dst_vma->vm_page_prot)); ptep_clear_flush(src_vma, src_addr, src_pte); set_pte_at(mm, dst_addr, dst_pte, zero_pte); double_pt_unlock(dst_ptl, src_ptl); - return PAGE_SIZE; + return mm_page_size(mm); } @@ -1474,7 +1885,7 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) ret = -ENOENT; else /* nothing to do to move a hole */ - ret = PAGE_SIZE; + ret = mm_page_size(mm); goto out; } @@ -1514,7 +1925,8 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd } folio = vm_normal_folio(src_vma, src_addr, orig_src_pte); - if (!folio || !PageAnonExclusive(&folio->page)) { + if (!folio || (mm_page_size(mm) >= PAGE_SIZE && + !PageAnonExclusive(&folio->page))) { spin_unlock(src_ptl); ret = -EBUSY; goto out; @@ -1554,6 +1966,15 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd } } +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(mm) < PAGE_SIZE) { + ret = move_present_subpage(mm, dst_vma, src_vma, dst_addr, src_addr, + dst_pte, src_pte, orig_dst_pte, orig_src_pte, + dst_pmd, dst_pmdval, dst_ptl, src_ptl, src_folio); + goto out; + } +#endif + /* at this point we have src_folio locked */ if (folio_test_large(src_folio)) { /* split_folio() can block */ @@ -1592,7 +2013,8 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd goto out; } - if (!pte_swp_exclusive(orig_src_pte)) { + /* Smaller leaves validate per-slot swap and mapping counts under PTL. */ + if (mm_page_size(mm) >= PAGE_SIZE && !pte_swp_exclusive(orig_src_pte)) { ret = -EBUSY; goto out; } @@ -1635,7 +2057,7 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd goto retry; } } - ret = move_swap_pte(mm, dst_vma, dst_addr, src_addr, dst_pte, src_pte, + ret = move_swap_pte(mm, dst_vma, src_vma, dst_addr, src_addr, dst_pte, src_pte, orig_dst_pte, orig_src_pte, dst_pmd, dst_pmdval, dst_ptl, src_ptl, src_folio, si, entry); } @@ -1694,6 +2116,11 @@ static int validate_move_areas(struct userfaultfd_ctx *ctx, pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) return -EINVAL; + /* Native backing cannot mix droppable and swap-backed slot policies. */ + if (mm_page_size(src_vma->vm_mm) < PAGE_SIZE && + ((src_vma->vm_flags ^ dst_vma->vm_flags) & VM_DROPPABLE)) + return -EINVAL; + /* Only allow moving if both are mlocked or both aren't */ if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) return -EINVAL; @@ -1926,9 +2353,9 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, ssize_t moved = 0; /* Sanitize the command parameters. */ - VM_WARN_ON_ONCE(src_start & ~PAGE_MASK); - VM_WARN_ON_ONCE(dst_start & ~PAGE_MASK); - VM_WARN_ON_ONCE(len & ~PAGE_MASK); + VM_WARN_ON_ONCE(src_start & ~mm_page_mask(mm)); + VM_WARN_ON_ONCE(dst_start & ~mm_page_mask(mm)); + VM_WARN_ON_ONCE(len & ~mm_page_mask(mm)); /* Does the address range wrap, or is the span zero-sized? */ VM_WARN_ON_ONCE(src_start + len < src_start); @@ -2053,10 +2480,12 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, ret = move_pages_ptes(mm, dst_pmd, src_pmd, dst_vma, src_vma, dst_addr, src_addr, src_end - src_addr, mode); - if (ret < 0) + if (ret < 0) { err = ret; - else + } else { + err = 0; step_size = ret; + } } cond_resched(); @@ -2095,6 +2524,17 @@ bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, { const struct vm_uffd_ops *ops = vma_uffd_ops(vma); + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) { + if (mm_page_size(vma->vm_mm) > PAGE_SIZE) + return false; + if (vma_is_anonymous(vma)) { + if (vm_flags & VM_UFFD_MINOR) + return false; + } else if (!vma_is_shmem(vma)) { + return false; + } + } + if (vma->vm_flags & (VM_DROPPABLE | VM_SHADOW_STACK)) return false; diff --git a/mm/util.c b/mm/util.c index 3b26f5bf920121..2adb3d08660d68 100644 --- a/mm/util.c +++ b/mm/util.c @@ -344,21 +344,22 @@ unsigned long randomize_stack_top(unsigned long stack_top) if (current->flags & PF_RANDOMIZE) { random_variable = get_random_long(); - random_variable &= STACK_RND_MASK; - random_variable <<= PAGE_SHIFT; + random_variable &= mm_aslr_mask(current->mm, STACK_RND_MASK); + random_variable <<= mm_page_shift(current->mm); } #ifdef CONFIG_STACK_GROWSUP - return PAGE_ALIGN(stack_top) + random_variable; + return mm_page_align(current->mm, stack_top) + random_variable; #else - return PAGE_ALIGN(stack_top) - random_variable; + return mm_page_align(current->mm, stack_top) - random_variable; #endif } /** - * randomize_page - Generate a random, page aligned address + * randomize_page_shift - Generate a random, granule aligned address * @start: The smallest acceptable address the caller will take. * @range: The size of the area, starting at @start, within which the * random address must fall. + * @shift: Log2 of the required address alignment. * * If @start + @range would overflow, @range is capped. * @@ -368,22 +369,31 @@ unsigned long randomize_stack_top(unsigned long stack_top) * Return: A page aligned address within [start, start + range). On error, * @start is returned. */ -unsigned long randomize_page(unsigned long start, unsigned long range) +static unsigned long randomize_page_shift(unsigned long start, + unsigned long range, unsigned int shift) { - if (!PAGE_ALIGNED(start)) { - range -= PAGE_ALIGN(start) - start; - start = PAGE_ALIGN(start); + unsigned long page_size = 1UL << shift; + + if (!IS_ALIGNED(start, page_size)) { + range -= ALIGN(start, page_size) - start; + start = ALIGN(start, page_size); } if (start > ULONG_MAX - range) range = ULONG_MAX - start; - range >>= PAGE_SHIFT; + range >>= shift; if (range == 0) return start; - return start + (get_random_long() % range << PAGE_SHIFT); + return start + (get_random_long() % range << shift); +} + +/* Existing callers retain native-page alignment. */ +unsigned long randomize_page(unsigned long start, unsigned long range) +{ + return randomize_page_shift(start, range, PAGE_SHIFT); } #ifdef CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT @@ -391,23 +401,25 @@ unsigned long __weak arch_randomize_brk(struct mm_struct *mm) { /* Is the current task 32bit ? */ if (!IS_ENABLED(CONFIG_64BIT) || is_compat_task()) - return randomize_page(mm->brk, SZ_32M); + return randomize_page_shift(mm->brk, SZ_32M, mm_page_shift(mm)); - return randomize_page(mm->brk, SZ_1G); + return randomize_page_shift(mm->brk, SZ_1G, mm_page_shift(mm)); } unsigned long arch_mmap_rnd(void) { - unsigned long rnd; + unsigned int bits; + unsigned long mask; #ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS if (is_compat_task()) - rnd = get_random_long() & ((1UL << mmap_rnd_compat_bits) - 1); + bits = mmap_rnd_compat_bits; else #endif /* CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS */ - rnd = get_random_long() & ((1UL << mmap_rnd_bits) - 1); + bits = mmap_rnd_bits; - return rnd << PAGE_SHIFT; + mask = mm_aslr_mask(current->mm, (1UL << bits) - 1); + return (get_random_long() & mask) << mm_page_shift(current->mm); } static int mmap_is_legacy(const struct rlimit *rlim_stack) @@ -440,14 +452,15 @@ static unsigned long mmap_base(const unsigned long rnd, const struct rlimit *rli * task. mmap_base starts directly below the stack and grows * downwards. */ - return PAGE_ALIGN_DOWN(mmap_upper_limit(rlim_stack) - rnd); + return (mmap_upper_limit(rlim_stack) - rnd) & mm_page_mask(current->mm); #else unsigned long gap = rlim_stack->rlim_cur; unsigned long pad = stack_guard_gap; /* Account for stack randomization if necessary */ if (current->flags & PF_RANDOMIZE) - pad += (STACK_RND_MASK << PAGE_SHIFT); + pad += mm_aslr_mask(current->mm, STACK_RND_MASK) << + mm_page_shift(current->mm); /* Values close to RLIM_INFINITY can overflow. */ if (gap + pad > gap) @@ -458,7 +471,7 @@ static unsigned long mmap_base(const unsigned long rnd, const struct rlimit *rli else if (gap > MAX_GAP) gap = MAX_GAP; - return PAGE_ALIGN(STACK_TOP - gap - rnd); + return mm_page_align(current->mm, STACK_TOP - gap - rnd); #endif } @@ -491,7 +504,7 @@ EXPORT_SYMBOL_IF_KUNIT(arch_pick_mmap_layout); /** * __account_locked_vm - account locked pages to an mm's locked_vm * @mm: mm to account against - * @pages: number of pages to account + * @pages: number of native physical pages to account * @inc: %true if @pages should be considered positive, %false if not * @task: task used to check RLIMIT_MEMLOCK * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped @@ -507,27 +520,35 @@ int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, const struct task_struct *task, bool bypass_rlim) { unsigned long locked_vm, limit; + unsigned int shift = PAGE_SHIFT - MM_ACCOUNT_SHIFT; int ret = 0; mmap_assert_write_locked(mm); + /* This exported API receives native physical-page counts. */ + if (pages > ULONG_MAX >> shift) + return -ENOMEM; + pages <<= shift; locked_vm = mm->locked_vm; if (inc) { if (!bypass_rlim) { - limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT; - if (locked_vm + pages > limit) + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> MM_ACCOUNT_SHIFT; + if (pages > limit || locked_vm > limit - pages) ret = -ENOMEM; } + if (!ret && pages > ULONG_MAX - locked_vm) + ret = -ENOMEM; if (!ret) mm->locked_vm = locked_vm + pages; } else { - WARN_ON_ONCE(pages > locked_vm); + if (WARN_ON_ONCE(pages > locked_vm)) + return -EINVAL; mm->locked_vm = locked_vm - pages; } pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid, - (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT, - locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK), + (void *)_RET_IP_, (inc) ? '+' : '-', pages << MM_ACCOUNT_SHIFT, + locked_vm << MM_ACCOUNT_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK), ret ? " - exceeded" : ""); return ret; @@ -562,11 +583,11 @@ int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc) } EXPORT_SYMBOL_GPL(account_locked_vm); -unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, +static unsigned long vm_mmap_offset_internal(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, - unsigned long flag, unsigned long pgoff) + unsigned long flag, struct vm_page_offset pos) { - loff_t off = (loff_t)pgoff << PAGE_SHIFT; + loff_t off = ((loff_t)pos.index << PAGE_SHIFT) | pos.offset; unsigned long ret; struct mm_struct *mm = current->mm; unsigned long populate; @@ -578,8 +599,11 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, if (!ret) { if (mmap_write_lock_killable(mm)) return -EINTR; - ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate, - &uf); +#ifdef CONFIG_MM_SUBPAGE + ret = do_mmap_offset(file, addr, len, prot, flag, 0, pos, &populate, &uf); +#else + ret = do_mmap(file, addr, len, prot, flag, 0, pos.index, &populate, &uf); +#endif mmap_write_unlock(mm); userfaultfd_unmap_complete(mm, &uf); if (populate) @@ -588,6 +612,13 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, return ret; } +unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flag, unsigned long pgoff) +{ + return vm_mmap_offset_internal(file, addr, len, prot, flag, + (struct vm_page_offset) { pgoff, 0 }); +} + /* * Perform a userland memory mapping into the current process address space. See * the comment for do_mmap() for more details on this operation in general. @@ -609,12 +640,13 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flag, unsigned long offset) { - if (unlikely(offset + PAGE_ALIGN(len) < offset)) + if (unlikely(offset + mm_page_align(current->mm, len) < offset)) return -EINVAL; - if (unlikely(offset_in_page(offset))) + if (unlikely(offset & ~mm_page_mask(current->mm))) return -EINVAL; - return vm_mmap_pgoff(file, addr, len, prot, flag, offset >> PAGE_SHIFT); + return vm_mmap_offset_internal(file, addr, len, prot, flag, + (struct vm_page_offset) { offset >> PAGE_SHIFT, offset & ~PAGE_MASK }); } EXPORT_SYMBOL(vm_mmap); @@ -988,7 +1020,8 @@ int __vm_enough_memory(const struct mm_struct *mm, long pages, int cap_sys_admin if (mm) { long reserve = sysctl_user_reserve_kbytes >> (PAGE_SHIFT - 10); - allowed -= min_t(long, mm->total_vm / 32, reserve); + allowed -= min_t(long, + mm_pages_to_bytes(mm, mm->total_vm) >> (PAGE_SHIFT + 5), reserve); } if (percpu_counter_read_positive(&vm_committed_as) < allowed) @@ -1061,19 +1094,26 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) return res; } -int __weak memcmp_pages(struct page *page1, struct page *page2) +/* Both byte ranges must fit within their respective native pages. */ +int __weak memcmp_pages_range(struct page *page1, unsigned int offset1, + struct page *page2, unsigned int offset2, size_t len) { char *addr1, *addr2; int ret; addr1 = kmap_local_page(page1); addr2 = kmap_local_page(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); + ret = memcmp(addr1 + offset1, addr2 + offset2, len); kunmap_local(addr2); kunmap_local(addr1); return ret; } +int __weak memcmp_pages(struct page *page1, struct page *page2) +{ + return memcmp_pages_range(page1, 0, page2, 0, PAGE_SIZE); +} + #ifdef CONFIG_PRINTK /** * mem_dump_obj - Print available provenance information @@ -1189,6 +1229,9 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, desc->end = vma->vm_end; desc->pgoff = vma->vm_pgoff; +#ifdef CONFIG_MM_SUBPAGE + desc->subpage_offset = vma->vm_subpage_offset; +#endif desc->vm_file = vma->vm_file; desc->vma_flags = vma->flags; desc->page_prot = vma->vm_page_prot; diff --git a/mm/vma.c b/mm/vma.c index d90791b00a7b81..4161f7739a10e6 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -13,7 +13,7 @@ struct mmap_state { unsigned long addr; unsigned long end; - pgoff_t pgoff; + struct vm_page_offset offset; unsigned long pglen; union { vm_flags_t vm_flags; @@ -42,19 +42,25 @@ struct mmap_state { bool file_doesnt_need_get :1; }; -#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, vma_flags_, file_) \ +#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pos_, vma_flags_, file_) \ struct mmap_state name = { \ .mm = mm_, \ .vmi = vmi_, \ .addr = addr_, \ .end = (addr_) + (len_), \ - .pgoff = pgoff_, \ - .pglen = PHYS_PFN(len_), \ + .offset = pos_, \ + .pglen = (len_) >> mm_page_shift(mm_), \ .vma_flags = vma_flags_, \ .file = file_, \ .page_prot = vma_get_page_prot(vma_flags_), \ } +#ifdef CONFIG_MM_SUBPAGE +#define VMG_MMAP_SUBPAGE_STATE(map_) .subpage_offset = (map_)->offset.offset, +#else +#define VMG_MMAP_SUBPAGE_STATE(map_) +#endif + #define VMG_MMAP_STATE(name, map_, vma_) \ struct vma_merge_struct name = { \ .mm = (map_)->mm, \ @@ -62,7 +68,8 @@ struct mmap_state { .start = (map_)->addr, \ .end = (map_)->end, \ .vma_flags = (map_)->vma_flags, \ - .pgoff = (map_)->pgoff, \ + .pgoff = (map_)->offset.index, \ + VMG_MMAP_SUBPAGE_STATE(map_) \ .file = (map_)->file, \ .prev = (map_)->prev, \ .middle = vma_, \ @@ -197,11 +204,12 @@ static void init_multi_vma_prep(struct vma_prepare *vp, */ static bool can_vma_merge_before(struct vma_merge_struct *vmg) { - pgoff_t pglen = PHYS_PFN(vmg->end - vmg->start); + struct vm_page_offset end = vm_page_offset_add(vmg_page_offset(vmg), + vmg->end - vmg->start); if (is_mergeable_vma(vmg, /* merge_next = */ true) && is_mergeable_anon_vma(vmg, /* merge_next = */ true)) { - if (vmg->next->vm_pgoff == vmg->pgoff + pglen) + if (vm_page_offset_equal(vma_page_offset_at(vmg->next, vmg->next->vm_start), end)) return true; } @@ -221,7 +229,8 @@ static bool can_vma_merge_after(struct vma_merge_struct *vmg) { if (is_mergeable_vma(vmg, /* merge_next = */ false) && is_mergeable_anon_vma(vmg, /* merge_next = */ false)) { - if (vmg->prev->vm_pgoff + vma_pages(vmg->prev) == vmg->pgoff) + if (vm_page_offset_equal(vma_page_offset_at(vmg->prev, vmg->prev->vm_end), + vmg_page_offset(vmg))) return true; } return false; @@ -521,7 +530,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, new->vm_end = addr; } else { new->vm_start = addr; - new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT); + vma_set_page_offset(new, vma_page_offset_at(vma, addr)); } err = -ENOMEM; @@ -560,7 +569,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, if (new_below) { vma->vm_start = addr; - vma->vm_pgoff += (addr - new->vm_start) >> PAGE_SHIFT; + vma_set_page_offset(vma, vma_page_offset_at(new, addr)); } else { vma->vm_end = addr; } @@ -705,19 +714,21 @@ void validate_mm(struct mm_struct *mm) static void vmg_adjust_set_range(struct vma_merge_struct *vmg) { struct vm_area_struct *adjust; - pgoff_t pgoff; + struct vm_page_offset pos; if (vmg->__adjust_middle_start) { adjust = vmg->middle; - pgoff = adjust->vm_pgoff + PHYS_PFN(vmg->end - adjust->vm_start); + pos = vma_page_offset_at(adjust, vmg->end); } else if (vmg->__adjust_next_start) { adjust = vmg->next; - pgoff = adjust->vm_pgoff - PHYS_PFN(adjust->vm_start - vmg->end); + pos = vm_page_offset_sub(vma_page_offset_at(adjust, adjust->vm_start), + adjust->vm_start - vmg->end); } else { return; } - vma_set_range(adjust, vmg->end, adjust->vm_end, pgoff); + vma_set_range(adjust, vmg->end, adjust->vm_end, pos.index); + vma_set_page_offset(adjust, pos); } /* @@ -762,6 +773,7 @@ static int commit_merge(struct vma_merge_struct *vmg) vma_adjust_trans_huge(vma, vmg->start, vmg->end, vmg->__adjust_middle_start ? vmg->middle : NULL); vma_set_range(vma, vmg->start, vmg->end, vmg->pgoff); + vma_set_page_offset(vma, vmg_page_offset(vmg)); vmg_adjust_set_range(vmg); vma_iter_store_overwrite(vmg->vmi, vmg->target); @@ -931,7 +943,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->start = prev->vm_start; vmg->end = next->vm_end; - vmg->pgoff = prev->vm_pgoff; + vmg_set_page_offset(vmg, vma_page_offset_at(prev, prev->vm_start)); /* * We already ensured anon_vma compatibility above, so now it's @@ -950,7 +962,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( */ vmg->start = prev->vm_start; - vmg->pgoff = prev->vm_pgoff; + vmg_set_page_offset(vmg, vma_page_offset_at(prev, prev->vm_start)); if (!vmg->__remove_middle) vmg->__adjust_middle_start = true; @@ -965,7 +977,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( * shrink/delete extend */ - pgoff_t pglen = PHYS_PFN(vmg->end - vmg->start); + unsigned long len = vmg->end - vmg->start; VM_WARN_ON_VMG(!merge_right, vmg); /* If we are offset into a VMA, then prev must be middle. */ @@ -973,13 +985,14 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( if (vmg->__remove_middle) { vmg->end = next->vm_end; - vmg->pgoff = next->vm_pgoff - pglen; + vmg_set_page_offset(vmg, + vm_page_offset_sub(vma_page_offset_at(next, next->vm_start), len)); } else { /* We shrink middle and expand next. */ vmg->__adjust_next_start = true; vmg->start = middle->vm_start; vmg->end = start; - vmg->pgoff = middle->vm_pgoff; + vmg_set_page_offset(vmg, vma_page_offset_at(middle, middle->vm_start)); } err = dup_anon_vma(next, middle, &anon_dup); @@ -1088,7 +1101,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg) if (can_merge_left) { vmg->start = prev->vm_start; vmg->target = prev; - vmg->pgoff = prev->vm_pgoff; + vmg_set_page_offset(vmg, vma_page_offset_at(prev, prev->vm_start)); /* * If this merge would result in removal of the next VMA but we @@ -1245,7 +1258,7 @@ int vma_expand(struct vma_merge_struct *vmg) * Returns: 0 on success, -ENOMEM otherwise */ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, - unsigned long start, unsigned long end, pgoff_t pgoff) + unsigned long start, unsigned long end, struct vm_page_offset pos) { struct vma_prepare vp; @@ -1266,7 +1279,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_adjust_trans_huge(vma, start, end, NULL); vma_iter_clear(vmi); - vma_set_range(vma, start, end, pgoff); + vma_set_range(vma, start, end, pos.index); + vma_set_page_offset(vma, pos); vma_complete(&vp, vmi, vma->vm_mm); validate_mm(vma->vm_mm); return 0; @@ -1335,7 +1349,7 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, mm = current->mm; mm->map_count -= vms->vma_count; - mm->locked_vm -= vms->locked_vm; + mm->locked_vm -= mm_pages_to_account(mm, vms->locked_vm); if (vms->unlock) mmap_write_downgrade(mm); @@ -1360,7 +1374,7 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, mas_for_each(mas_detach, vma, ULONG_MAX) remove_vma(vma); - vm_unacct_memory(vms->nr_accounted); + mm_unacct_memory(mm, vms->nr_accounted); validate_mm(mm); if (vms->unlock) mmap_read_unlock(mm); @@ -1456,7 +1470,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, goto munmap_gather_failed; vma_mark_detached(next); - nrpages = vma_pages(next); + nrpages = vma_user_pages(next); vms->nr_pages += nrpages; if (vma_test(next, VMA_LOCKED_BIT)) @@ -1634,10 +1648,10 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long end; struct vm_area_struct *vma; - if ((offset_in_page(start)) || start > TASK_SIZE || len > TASK_SIZE-start) + if ((start & ~mm_page_mask(mm)) || start > TASK_SIZE || len > TASK_SIZE-start) return -EINVAL; - end = start + PAGE_ALIGN(len); + end = start + mm_page_align(mm, len); if (end == start) return -EINVAL; @@ -1862,7 +1876,7 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) * prior to moving page table entries, to effect an mremap move. */ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, - unsigned long addr, unsigned long len, pgoff_t pgoff, + unsigned long addr, unsigned long len, struct vm_page_offset pos, bool *need_rmap_locks) { struct vm_area_struct *vma = *vmap; @@ -1878,7 +1892,10 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, * to match new location, to increase its chance of merging. */ if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) { - pgoff = addr >> PAGE_SHIFT; + pos = (struct vm_page_offset) { + .index = addr >> PAGE_SHIFT, + .offset = addr & ~PAGE_MASK, + }; faulted_in_anon_vma = false; } @@ -1894,7 +1911,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, if (new_vma && new_vma->vm_start < addr + len) return NULL; /* should never get here */ - vmg.pgoff = pgoff; + vmg_set_page_offset(&vmg, pos); vmg.next = vma_iter_next_rewind(&vmi, NULL); new_vma = vma_merge_copied_range(&vmg); @@ -1924,7 +1941,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma = vm_area_dup(vma); if (!new_vma) goto out; - vma_set_range(new_vma, addr, addr + len, pgoff); + vma_set_range(new_vma, addr, addr + len, pos.index); + vma_set_page_offset(new_vma, pos); if (vma_dup_policy(vma, new_vma)) goto out_free_vma; if (anon_vma_clone(new_vma, vma, VMA_OP_REMAP)) @@ -1979,7 +1997,8 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * mpol_equal(vma_policy(a), vma_policy(b)) && a->vm_file == b->vm_file && vma_flags_empty(&diff) && - b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT); + vm_page_offset_equal(vma_page_offset_at(a, b->vm_start), + vma_page_offset_at(b, b->vm_start)); } /* @@ -2398,7 +2417,10 @@ static void set_desc_from_map(struct vm_area_desc *desc, desc->start = map->addr; desc->end = map->end; - desc->pgoff = map->pgoff; + desc->pgoff = map->offset.index; +#ifdef CONFIG_MM_SUBPAGE + desc->subpage_offset = map->offset.offset; +#endif desc->vm_file = map->file; desc->vma_flags = map->vma_flags; desc->page_prot = map->page_prot; @@ -2458,7 +2480,7 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, map->charged = map->pglen; map->charged -= vms->nr_accounted; if (map->charged) { - error = security_vm_enough_memory_mm(map->mm, map->charged); + error = mm_account_memory(map->mm, map->mm, map->charged); if (error) return error; } @@ -2549,7 +2571,8 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, return -ENOMEM; vma_iter_config(vmi, map->addr, map->end); - vma_set_range(vma, map->addr, map->end, map->pgoff); + vma_set_range(vma, map->addr, map->end, map->offset.index); + vma_set_page_offset(vma, map->offset); vma->flags = map->vma_flags; vma->vm_page_prot = map->page_prot; @@ -2621,7 +2644,7 @@ static void __mmap_complete(struct mmap_state *map, struct vm_area_struct *vma) if (!vma_supports_mlock(vma)) vma_clear_flags_mask(vma, VMA_LOCKED_MASK); else - mm->locked_vm += map->pglen; + mm->locked_vm += mm_pages_to_account(mm, map->pglen); } if (vma->vm_file) @@ -2678,7 +2701,10 @@ static int call_mmap_prepare(struct mmap_state *map, return err; /* Update fields permitted to be changed. */ - map->pgoff = desc->pgoff; + map->offset.index = desc->pgoff; +#ifdef CONFIG_MM_SUBPAGE + map->offset.offset = desc->subpage_offset; +#endif if (desc->vm_file != map->file) { map->file_doesnt_need_get = true; map->file = desc->vm_file; @@ -2689,6 +2715,12 @@ static int call_mmap_prepare(struct mmap_state *map, map->vm_ops = desc->vm_ops; map->vm_private_data = desc->private_data; +#ifdef CONFIG_MM_SUBPAGE + if (map->offset.offset >= PAGE_SIZE || + !IS_ALIGNED(map->offset.offset, mm_page_size(map->mm))) + return -EINVAL; +#endif + return 0; } @@ -2731,13 +2763,13 @@ static bool can_set_ksm_flags_early(struct mmap_state *map) static unsigned long __mmap_region(struct file *file, unsigned long addr, unsigned long len, vma_flags_t vma_flags, - unsigned long pgoff, struct list_head *uf) + struct vm_page_offset pos, struct list_head *uf) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma = NULL; bool have_mmap_prepare = file && file->f_op->mmap_prepare; VMA_ITERATOR(vmi, mm, addr); - MMAP_STATE(map, mm, &vmi, addr, len, pgoff, vma_flags, file); + MMAP_STATE(map, mm, &vmi, addr, len, pos, vma_flags, file); struct vm_area_desc desc = { .mm = mm, .file = file, @@ -2791,7 +2823,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, /* Accounting was done by __mmap_setup(). */ unacct_error: if (map.charged) - vm_unacct_memory(map.charged); + mm_unacct_memory(mm, map.charged); abort_munmap: /* * This indicates that .mmap_prepare has set a new file, differing from @@ -2807,7 +2839,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, /** * mmap_region() - Actually perform the userland mapping of a VMA into * current->mm with known, aligned and overflow-checked @addr and @len, and - * correctly determined VMA flags @vm_flags and page offset @pgoff. + * correctly determined VMA flags @vm_flags and file byte position @pos. * * This is an internal memory management function, and should not be used * directly. @@ -2819,8 +2851,8 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, * @addr: The page-aligned address at which to perform the mapping. * @len: The page-aligned, non-zero, length of the mapping. * @vm_flags: The VMA flags which should be applied to the mapping. - * @pgoff: If @file is specified, the page offset into the file, if not then - * the virtual page offset in memory of the anonymous mapping. + * @pos: Native page-cache index and byte offset within it. For anonymous + * mappings, the position is derived from the virtual address. * @uf: Optionally, a pointer to a list head used for tracking userfaultfd unmap * events. * @@ -2829,13 +2861,15 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, */ unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long len, vm_flags_t vm_flags, - unsigned long pgoff, struct list_head *uf) + struct vm_page_offset pos, struct list_head *uf) { unsigned long ret; bool writable_file_mapping = false; const vma_flags_t vma_flags = legacy_to_vma_flags(vm_flags); mmap_assert_write_locked(current->mm); + if (pos.offset >= PAGE_SIZE || !IS_ALIGNED(pos.offset, mm_page_size(current->mm))) + return -EINVAL; /* Check to see if MDWE is applicable. */ if (map_deny_write_exec(&vma_flags, &vma_flags)) @@ -2854,7 +2888,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, writable_file_mapping = true; } - ret = __mmap_region(file, addr, len, vma_flags, pgoff, uf); + ret = __mmap_region(file, addr, len, vma_flags, pos, uf); /* Clear our write mapping regardless of error. */ if (writable_file_mapping) @@ -2892,13 +2926,13 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_flags_set_mask(&vma_flags, mm->def_vma_flags); vma_flags = ksm_vma_flags(mm, NULL, vma_flags); - if (!may_expand_vm(mm, &vma_flags, len >> PAGE_SHIFT)) + if (!may_expand_vm(mm, &vma_flags, len >> mm_page_shift(mm))) return -ENOMEM; if (mm->map_count > get_sysctl_max_map_count()) return -ENOMEM; - if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) + if (mm_account_memory(mm, mm, len >> mm_page_shift(mm))) return -ENOMEM; /* @@ -2908,6 +2942,10 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma && vma->vm_end == addr) { VMG_STATE(vmg, mm, vmi, addr, addr + len, vma_flags, PHYS_PFN(addr)); + vmg_set_page_offset(&vmg, (struct vm_page_offset) { + .index = addr >> PAGE_SHIFT, .offset = addr & ~PAGE_MASK, + }); + vmg.prev = vma; /* vmi is positioned at prev, which this mode expects. */ vmg.just_expand = true; @@ -2927,6 +2965,9 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_set_anonymous(vma); vma_set_range(vma, addr, addr + len, addr >> PAGE_SHIFT); + vma_set_page_offset(vma, (struct vm_page_offset) { + .index = addr >> PAGE_SHIFT, .offset = addr & ~PAGE_MASK, + }); vma->flags = vma_flags; vma->vm_page_prot = vm_get_page_prot(vma_flags_to_legacy(vma_flags)); vma_start_write(vma); @@ -2937,10 +2978,10 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, validate_mm(mm); out: perf_event_mmap(vma); - mm->total_vm += len >> PAGE_SHIFT; - mm->data_vm += len >> PAGE_SHIFT; + mm->total_vm += len >> mm_page_shift(mm); + mm->data_vm += len >> mm_page_shift(mm); if (vma_flags_test(&vma_flags, VMA_LOCKED_BIT)) - mm->locked_vm += (len >> PAGE_SHIFT); + mm->locked_vm += len >> MM_ACCOUNT_SHIFT; if (pgtable_supports_soft_dirty()) vma_set_flags(vma, VMA_SOFTDIRTY_BIT); return 0; @@ -2948,7 +2989,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, mas_store_fail: vm_area_free(vma); unacct_fail: - vm_unacct_memory(len >> PAGE_SHIFT); + mm_unacct_memory(mm, len >> mm_page_shift(mm)); return -ENOMEM; } @@ -3084,7 +3125,7 @@ static int acct_stack_growth(struct vm_area_struct *vma, /* mlock limit tests */ if (!mlock_future_ok(mm, vma_test(vma, VMA_LOCKED_BIT), - grow << PAGE_SHIFT)) + grow << mm_page_shift(mm))) return -ENOMEM; /* Check to ensure the stack will not grow into a hugetlb-only region */ @@ -3100,7 +3141,7 @@ static int acct_stack_growth(struct vm_area_struct *vma, * Overcommit.. This must be the final test, as it will * update security statistics. */ - if (security_vm_enough_memory_mm(mm, grow)) + if (mm_account_memory(mm, mm, grow)) return -ENOMEM; return 0; @@ -3125,10 +3166,10 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) mmap_assert_write_locked(mm); /* Guard against exceeding limits of the address space. */ - address &= PAGE_MASK; - if (address >= (TASK_SIZE & PAGE_MASK)) + address &= mm_page_mask(mm); + if (address >= (TASK_SIZE & mm_page_mask(mm))) return -ENOMEM; - address += PAGE_SIZE; + address += mm_page_size(mm); /* Enforce stack_guard_gap */ gap_addr = address + stack_guard_gap; @@ -3167,14 +3208,14 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) unsigned long size, grow; size = address - vma->vm_start; - grow = (address - vma->vm_end) >> PAGE_SHIFT; + grow = (address - vma->vm_end) >> mm_page_shift(mm); error = -ENOMEM; if (vma->vm_pgoff + (size >> PAGE_SHIFT) >= vma->vm_pgoff) { error = acct_stack_growth(vma, size, grow); if (!error) { if (vma_test(vma, VMA_LOCKED_BIT)) - mm->locked_vm += grow; + mm->locked_vm += mm_pages_to_account(mm, grow); vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); vma->vm_end = address; @@ -3209,7 +3250,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) mmap_assert_write_locked(mm); - address &= PAGE_MASK; + address &= mm_page_mask(mm); if (address < mmap_min_addr || address < FIRST_USER_ADDRESS) return -EPERM; @@ -3244,20 +3285,22 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) /* Somebody else might have raced and expanded it already */ if (address < vma->vm_start) { unsigned long size, grow; + struct vm_page_offset pos = vm_page_offset_sub( + vma_page_offset_at(vma, vma->vm_start), vma->vm_start - address); size = vma->vm_end - address; - grow = (vma->vm_start - address) >> PAGE_SHIFT; + grow = (vma->vm_start - address) >> mm_page_shift(mm); error = -ENOMEM; - if (grow <= vma->vm_pgoff) { + if (pos.index <= vma->vm_pgoff) { error = acct_stack_growth(vma, size, grow); if (!error) { if (vma_test(vma, VMA_LOCKED_BIT)) - mm->locked_vm += grow; + mm->locked_vm += mm_pages_to_account(mm, grow); vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); vma->vm_start = address; - vma->vm_pgoff -= grow; + vma_set_page_offset(vma, pos); /* Overwrite old entry in mtree. */ vma_iter_store_overwrite(&vmi, vma); anon_vma_interval_tree_post_update_vma(vma); @@ -3296,13 +3339,13 @@ int __vm_munmap(unsigned long start, size_t len, bool unlock) */ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) { - unsigned long charged = vma_pages(vma); + unsigned long charged = vma_user_pages(vma); if (find_vma_intersection(mm, vma->vm_start, vma->vm_end)) return -ENOMEM; if (vma_test(vma, VMA_ACCOUNT_BIT) && - security_vm_enough_memory_mm(mm, charged)) + mm_account_memory(mm, mm, charged)) return -ENOMEM; /* @@ -3319,12 +3362,15 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) */ if (vma_is_anonymous(vma)) { BUG_ON(vma->anon_vma); - vma->vm_pgoff = vma->vm_start >> PAGE_SHIFT; + vma_set_page_offset(vma, (struct vm_page_offset) { + .index = vma->vm_start >> PAGE_SHIFT, + .offset = vma->vm_start & ~PAGE_MASK, + }); } if (vma_link(mm, vma)) { if (vma_test(vma, VMA_ACCOUNT_BIT)) - vm_unacct_memory(charged); + mm_unacct_memory(mm, charged); return -ENOMEM; } diff --git a/mm/vma.h b/mm/vma.h index 8e4b61a7304c68..3d9c8ee4be05d1 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -97,6 +97,9 @@ struct vma_merge_struct { unsigned long start; unsigned long end; pgoff_t pgoff; +#ifdef CONFIG_MM_SUBPAGE + unsigned int subpage_offset; +#endif union { /* Temporary while VMA flags are being converted. */ @@ -234,9 +237,34 @@ static inline bool vmg_nomem(struct vma_merge_struct *vmg) static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma, unsigned long addr) { - return vma->vm_pgoff + PHYS_PFN(addr - vma->vm_start); + return vma_page_offset_at(vma, addr).index; +} + +static inline struct vm_page_offset vmg_page_offset(const struct vma_merge_struct *vmg) +{ + return (struct vm_page_offset) { + .index = vmg->pgoff, +#ifdef CONFIG_MM_SUBPAGE + .offset = vmg->subpage_offset, +#endif + }; +} + +static inline void vmg_set_page_offset(struct vma_merge_struct *vmg, struct vm_page_offset pos) +{ + vmg->pgoff = pos.index; +#ifdef CONFIG_MM_SUBPAGE + vmg->subpage_offset = pos.offset; +#endif } +#ifdef CONFIG_MM_SUBPAGE +#define VMG_SUBPAGE_STATE(vma, start) \ + .subpage_offset = vma_page_offset_at(vma, start).offset, +#else +#define VMG_SUBPAGE_STATE(vma, start) +#endif + #define VMG_STATE(name, mm_, vmi_, start_, end_, vma_flags_, pgoff_) \ struct vma_merge_struct name = { \ .mm = mm_, \ @@ -259,6 +287,7 @@ static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma, .end = end_, \ .vm_flags = vma_->vm_flags, \ .pgoff = vma_pgoff_offset(vma_, start_), \ + VMG_SUBPAGE_STATE(vma_, start_) \ .file = vma_->vm_file, \ .anon_vma = vma_->anon_vma, \ .policy = vma_policy(vma_), \ @@ -276,7 +305,7 @@ void validate_mm(struct mm_struct *mm); __must_check int vma_expand(struct vma_merge_struct *vmg); __must_check int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, - unsigned long start, unsigned long end, pgoff_t pgoff); + unsigned long start, unsigned long end, struct vm_page_offset pos); static inline int vma_iter_store_gfp(struct vma_iterator *vmi, struct vm_area_struct *vma, gfp_t gfp) @@ -311,6 +340,9 @@ static inline void compat_set_vma_from_desc(struct vm_area_struct *vma, /* Mutable fields. Populated with initial state. */ vma->vm_pgoff = desc->pgoff; +#ifdef CONFIG_MM_SUBPAGE + vma->vm_subpage_offset = desc->subpage_offset; +#endif if (desc->vm_file != vma->vm_file) vma_set_file(vma, desc->vm_file); vma->flags = desc->vma_flags; @@ -448,7 +480,7 @@ void unlink_file_vma_batch_add(struct unlink_vma_file_batch *vb, struct vm_area_struct *vma); struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, - unsigned long addr, unsigned long len, pgoff_t pgoff, + unsigned long addr, unsigned long len, struct vm_page_offset pos, bool *need_rmap_locks); struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma); @@ -460,7 +492,7 @@ int mm_take_all_locks(struct mm_struct *mm); void mm_drop_all_locks(struct mm_struct *mm); unsigned long mmap_region(struct file *file, unsigned long addr, - unsigned long len, vm_flags_t vm_flags, unsigned long pgoff, + unsigned long len, vm_flags_t vm_flags, struct vm_page_offset pos, struct list_head *uf); int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *brkvma, diff --git a/mm/vma_exec.c b/mm/vma_exec.c index 5cee8b7efa0f21..8394f45a911133 100644 --- a/mm/vma_exec.c +++ b/mm/vma_exec.c @@ -42,6 +42,8 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) struct mmu_gather tlb; PAGETABLE_MOVE(pmc, vma, vma, old_start, new_start, length); + vmg_set_page_offset(&vmg, vma_page_offset_at(vma, old_start)); + BUG_ON(new_start > new_end); /* @@ -89,7 +91,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) vma_prev(&vmi); /* Shrink the vma to just the new range */ - return vma_shrink(&vmi, vma, new_start, new_end, vma->vm_pgoff); + return vma_shrink(&vmi, vma, new_start, new_end, vma_page_offset_at(vma, vma->vm_start)); } /* @@ -138,7 +140,7 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap, */ VM_WARN_ON_ONCE(VM_STACK_FLAGS & VM_STACK_INCOMPLETE_SETUP); vma->vm_end = STACK_TOP_MAX; - vma->vm_start = vma->vm_end - PAGE_SIZE; + vma->vm_start = vma->vm_end - mm_page_size(mm); if (pgtable_supports_soft_dirty()) flags |= VM_SOFTDIRTY; vm_flags_init(vma, flags); diff --git a/mm/vma_init.c b/mm/vma_init.c index 3c0b65950510e0..a41ba35de73203 100644 --- a/mm/vma_init.c +++ b/mm/vma_init.c @@ -46,7 +46,10 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_start = src->vm_start; dest->vm_end = src->vm_end; dest->anon_vma = src->anon_vma; - dest->vm_pgoff = src->vm_pgoff; + vma_set_page_offset(dest, (struct vm_page_offset) { + .index = src->vm_pgoff, + .offset = vma_subpage_offset(src), + }); dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data; vm_flags_init(dest, src->vm_flags); diff --git a/mm/vmscan.c b/mm/vmscan.c index 1af6c96d1bc14b..fe28f70fd51b17 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3462,6 +3462,46 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio, } } +static unsigned long lru_pte_index(struct mm_struct *mm, unsigned long addr) +{ + if (mm_page_size(mm) == PAGE_SIZE) + return pte_index(addr); + return (addr >> mm_page_shift(mm)) & (pte_table_bytes_mm(mm) / sizeof(pte_t) - 1); +} + +static unsigned long lru_pmd_index(struct mm_struct *mm, unsigned long addr) +{ + if (mm_page_size(mm) == PAGE_SIZE) + return pmd_index(addr); + return (addr >> pmd_shift_mm(mm)) & (pmd_table_bytes_mm(mm) / sizeof(pmd_t) - 1); +} + +static unsigned long lru_pud_entries(struct mm_struct *mm) +{ + return mm_pud_folded(mm) ? 1 : pud_table_bytes_mm(mm) / sizeof(pud_t); +} + +static unsigned long lru_pud_index(struct mm_struct *mm, unsigned long addr) +{ + if (mm_pud_folded(mm)) + return 0; + if (mm_page_size(mm) == PAGE_SIZE) + return pud_index(addr); + return (addr >> pud_shift_mm(mm)) & (lru_pud_entries(mm) - 1); +} + +static unsigned int lru_folio_pte_batch(struct vm_area_struct *vma, + struct folio *folio, pte_t *ptep, pte_t *entry, unsigned int max_nr) +{ +#ifdef CONFIG_MM_SUBPAGE + if (mm_page_size(vma->vm_mm) != PAGE_SIZE) + return folio_subpage_pte_batch_flags(vma, folio, ptep, entry, + max_nr, FPB_MERGE_YOUNG_DIRTY); +#endif + return folio_pte_batch_flags(folio, NULL, ptep, entry, max_nr, + FPB_MERGE_YOUNG_DIRTY); +} + static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *args) { @@ -3481,7 +3521,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, unsigned int nr; pmd_t pmdval; - pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & pmd_mask_mm(args->mm), &pmdval, &ptl); if (!pte) return false; @@ -3497,7 +3537,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, lazy_mmu_mode_enable(); restart: - for (i = pte_index(start), addr = start; addr != end; i += nr, addr += nr * PAGE_SIZE) { + for (i = lru_pte_index(args->mm, start), addr = start; addr != end; i += nr, addr += nr * mm_page_size(args->mm)) { unsigned long pfn; struct folio *folio; pte_t *cur_pte = pte + i; @@ -3516,10 +3556,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, continue; if (folio_test_large(folio)) { - const unsigned int max_nr = (end - addr) >> PAGE_SHIFT; + const unsigned int max_nr = (end - addr) >> mm_page_shift(args->mm); - nr = folio_pte_batch_flags(folio, NULL, cur_pte, &ptent, - max_nr, FPB_MERGE_YOUNG_DIRTY); + nr = lru_folio_pte_batch(args->vma, folio, cur_pte, &ptent, max_nr); total += nr - 1; walk->mm_stats[MM_LEAF_TOTAL] += nr - 1; } @@ -3544,7 +3583,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, walk_update_folio(walk, last, gen, dirty); last = NULL; - if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) + if (i < pte_table_bytes_mm(args->mm) / sizeof(pte_t) && + get_next_vma(pmd_mask_mm(args->mm), mm_page_size(args->mm), args, &start, &end)) goto restart; lazy_mmu_mode_disable(); @@ -3576,13 +3616,13 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area return; } - i = addr == -1 ? 0 : pmd_index(addr) - pmd_index(*first); + i = addr == -1 ? 0 : lru_pmd_index(args->mm, addr) - lru_pmd_index(args->mm, *first); if (i && i <= MIN_LRU_BATCH) { __set_bit(i - 1, bitmap); return; } - pmd = pmd_offset(pud, *first); + pmd = pmd_offset_mm(args->mm, pud, *first); ptl = pmd_lockptr(args->mm, pmd); if (!spin_trylock(ptl)) @@ -3595,7 +3635,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area struct folio *folio; /* don't round down the first address */ - addr = i ? (*first & PMD_MASK) + i * PMD_SIZE : *first; + addr = i ? (*first & pmd_mask_mm(args->mm)) + i * pmd_size_mm(args->mm) : *first; if (!pmd_present(pmd[i])) goto next; @@ -3661,14 +3701,14 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, * tables to avoid taking the PMD lock; the second, if necessary, takes * the PMD lock to clear the accessed bit in PMD entries. */ - pmd = pmd_offset(pud, start & PUD_MASK); + pmd = pmd_offset_mm(args->mm, pud, start & pud_mask_mm(args->mm)); restart: /* walk_pte_range() may call get_next_vma() */ vma = args->vma; - for (i = pmd_index(start), addr = start; addr != end; i++, addr = next) { + for (i = lru_pmd_index(args->mm, start), addr = start; addr != end; i++, addr = next) { pmd_t val = pmdp_get_lockless(pmd + i); - next = pmd_addr_end(addr, end); + next = pmd_addr_end_mm(args->mm, addr, end); if (!pmd_present(val) || is_huge_zero_pmd(val)) { walk->mm_stats[MM_LEAF_TOTAL]++; @@ -3710,7 +3750,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first); - if (i < PTRS_PER_PMD && get_next_vma(PUD_MASK, PMD_SIZE, args, &start, &end)) + if (i < pmd_table_bytes_mm(args->mm) / sizeof(pmd_t) && + get_next_vma(pud_mask_mm(args->mm), pmd_size_mm(args->mm), args, &start, &end)) goto restart; } @@ -3725,12 +3766,12 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, VM_WARN_ON_ONCE(p4d_leaf(*p4d)); - pud = pud_offset(p4d, start & P4D_MASK); + pud = pud_offset_mm(args->mm, p4d, start & p4d_mask_mm(args->mm)); restart: - for (i = pud_index(start), addr = start; addr != end; i++, addr = next) { + for (i = lru_pud_index(args->mm, start), addr = start; addr != end; i++, addr = next) { pud_t val = pudp_get(pud + i); - next = pud_addr_end(addr, end); + next = pud_addr_end_mm(args->mm, addr, end); if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val))) continue; @@ -3738,15 +3779,16 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, walk_pmd_range(&val, addr, next, args); if (need_resched() || walk->batched >= MAX_LRU_BATCH) { - end = (addr | ~PUD_MASK) + 1; + end = (addr | ~pud_mask_mm(args->mm)) + 1; goto done; } } - if (i < PTRS_PER_PUD && get_next_vma(P4D_MASK, PUD_SIZE, args, &start, &end)) + if (i < lru_pud_entries(args->mm) && + get_next_vma(p4d_mask_mm(args->mm), pud_size_mm(args->mm), args, &start, &end)) goto restart; - end = round_up(end, P4D_SIZE); + end = round_up(end, p4d_size_mm(args->mm)); done: if (!end || !args->vma) return 1; @@ -4215,20 +4257,20 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) /* avoid taking the LRU lock under the PTL when possible */ walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; - start = max(addr & PMD_MASK, vma->vm_start); - end = min(addr | ~PMD_MASK, vma->vm_end - 1) + 1; + start = max(addr & pmd_mask_mm(vma->vm_mm), vma->vm_start); + end = min(addr | ~pmd_mask_mm(vma->vm_mm), vma->vm_end - 1) + 1; - if (end - start == PAGE_SIZE) + if (end - start == mm_page_size(vma->vm_mm)) return true; - if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { - if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) - end = start + MIN_LRU_BATCH * PAGE_SIZE; - else if (end - addr < MIN_LRU_BATCH * PAGE_SIZE / 2) - start = end - MIN_LRU_BATCH * PAGE_SIZE; + if (end - start > MIN_LRU_BATCH * mm_page_size(vma->vm_mm)) { + if (addr - start < MIN_LRU_BATCH * mm_page_size(vma->vm_mm) / 2) + end = start + MIN_LRU_BATCH * mm_page_size(vma->vm_mm); + else if (end - addr < MIN_LRU_BATCH * mm_page_size(vma->vm_mm) / 2) + start = end - MIN_LRU_BATCH * mm_page_size(vma->vm_mm); else { - start = addr - MIN_LRU_BATCH * PAGE_SIZE / 2; - end = addr + MIN_LRU_BATCH * PAGE_SIZE / 2; + start = addr - MIN_LRU_BATCH * mm_page_size(vma->vm_mm) / 2; + end = addr + MIN_LRU_BATCH * mm_page_size(vma->vm_mm) / 2; } } @@ -4240,10 +4282,10 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) lazy_mmu_mode_enable(); - pte -= (addr - start) / PAGE_SIZE; + pte -= (addr - start) / mm_page_size(vma->vm_mm); for (i = 0, addr = start; addr != end; - i += nr, pte += nr, addr += nr * PAGE_SIZE) { + i += nr, pte += nr, addr += nr * mm_page_size(vma->vm_mm)) { unsigned long pfn; pte_t ptent = ptep_get(pte); @@ -4257,10 +4299,9 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) continue; if (folio_test_large(folio)) { - const unsigned int max_nr = (end - addr) >> PAGE_SHIFT; + const unsigned int max_nr = (end - addr) >> mm_page_shift(vma->vm_mm); - nr = folio_pte_batch_flags(folio, NULL, pte, &ptent, - max_nr, FPB_MERGE_YOUNG_DIRTY); + nr = lru_folio_pte_batch(vma, folio, pte, &ptent, max_nr); } if (!test_and_clear_young_ptes_notify(vma, addr, pte, nr)) @@ -4634,6 +4675,16 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c sc->nr.unqueued_dirty += delta; } + /* + * Legacy memcg reclaim relies on shrink_folio_list() waiting for + * writeback when reclaim-marked folios cycle through the LRU. Moving + * all of them to a younger generation can exhaust the charge limit + * before I/O completes. Let the common path apply its GFP and mapping + * deadlock checks instead of hiding those folios from it. + */ + if (writeback && !writeback_throttling_sane(sc)) + return false; + /* waiting for writeback */ if (writeback || (type == LRU_GEN_FILE && dirty)) { gen = folio_inc_gen(lruvec, folio, true); @@ -4668,8 +4719,9 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca if (!folio_test_referenced(folio)) set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0); - /* for shrink_folio_list() */ - folio_clear_reclaim(folio); + /* Preserve legacy memcg's repeated-writeback marker for shrink_folio_list(). */ + if (writeback_throttling_sane(sc) || !folio_test_writeback(folio)) + folio_clear_reclaim(folio); success = lru_gen_del_folio(lruvec, folio, true); VM_WARN_ON_ONCE_FOLIO(!success, folio); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a5cdcc0fc55e1c..090a26ed9f2db2 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -1766,8 +1767,11 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, u32 bytelen, next; /* there might be non MSG_ZEROCOPY users */ - if (uarg->ops != &msg_zerocopy_ubuf_ops) + if (uarg->ops != &msg_zerocopy_ubuf_ops) { + if (sk->sk_type == SOCK_STREAM) + goto new_alloc; return NULL; + } /* realloc only when socket is locked (TCP, UDP cork), * so uarg->len and sk_zckey access is serialized @@ -7347,8 +7351,113 @@ nodefer: kfree_skb_napi_cache(skb); kick_defer_list_purge(cpu); } +#ifdef CONFIG_MM_SUBPAGE +/* The ordinary frag page ref protects allocation, not an anonymous quarter. */ +struct skb_subpage_refs { + struct ubuf_info ubuf; + struct mm_subpage_refs *owner; +}; + +static void skb_subpage_complete(struct sk_buff *skb, struct ubuf_info *ubuf, + bool success) +{ + struct skb_subpage_refs *refs = container_of(ubuf, struct skb_subpage_refs, ubuf); + + if (!refcount_dec_and_test(&ubuf->refcnt)) + return; + mm_subpage_refs_put(refs->owner); + kfree(refs); +} + +static const struct ubuf_info_ops skb_subpage_ops = { + .complete = skb_subpage_complete, +}; + +static bool skb_contains_subpage(const struct sk_buff *skb, phys_addr_t phys, + unsigned int size, unsigned int depth) +{ + struct sk_buff *child; + unsigned int i; + + /* Match the networking recursion bound, conservatively keeping unknowns. */ + if (depth >= 24) + return true; + if (skb->head_frag && skb_headlen(skb)) { + phys_addr_t start = page_to_phys(virt_to_page(skb->data)) + offset_in_page(skb->data); + + if (start < phys + size && phys < start + skb_headlen(skb)) + return true; + } + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { + const skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; + struct page *page = skb_frag_page(frag); + phys_addr_t start; + + if (!page) + continue; + start = page_to_phys(page) + skb_frag_off(frag); + if (start < phys + size && phys < start + skb_frag_size(frag)) + return true; + } + skb_walk_frags(skb, child) + if (skb_contains_subpage(child, phys, size, depth + 1)) + return true; + return false; +} + +static bool skb_keeps_subpage(const struct mm_subpage *slot, void *context) +{ + return skb_contains_subpage(context, mm_subpage_phys(slot), mm_subpage_size(slot), 0); +} + +/* Caller owns the writable skb and keeps the source slot identity alive. */ +static int skb_splice_hold_subpages(struct sk_buff *skb, struct page *page, + unsigned int off, unsigned int len) +{ + struct skb_subpage_refs *refs, *new; + struct ubuf_info *old = skb_zcopy(skb); + struct mm_subpage *probe; + int ret; + + if (!len || folio_order(page_folio(page)) || !folio_test_anon(page_folio(page))) + return 0; + /* Preserve the allocation-free native anonymous path. */ + probe = mm_subpage_get_from_phys(page_to_phys(page) + round_down(off, MM_SUBPAGE_SIZE)); + if (!probe) + return 0; + mm_subpage_put(probe); + if (old && (skb_zcopy_is_nouarg(skb) || old->ops != &skb_subpage_ops)) + return -EEXIST; + refs = old ? container_of(old, struct skb_subpage_refs, ubuf) : NULL; + if (!refs || refcount_read(&old->refcnt) != 1) { + new = kzalloc(sizeof(*new), GFP_ATOMIC); + if (!new) + return -ENOMEM; + new->ubuf.ops = &skb_subpage_ops; + new->ubuf.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + refcount_set(&new->ubuf.refcnt, 1); + if (refs) + new->owner = mm_subpage_refs_get(refs->owner); + skb_zcopy_init(skb, &new->ubuf); + net_zcopy_put(old); + refs = new; + } + /* Repeated trim/append must not accumulate references to discarded data. */ + ret = mm_subpage_refs_prune(&refs->owner, skb_keeps_subpage, skb, GFP_ATOMIC); + if (ret) + return ret; + return mm_subpage_refs_add(&refs->owner, page, off, len, GFP_ATOMIC); +} +#else +static int skb_splice_hold_subpages(struct sk_buff *skb, struct page *page, + unsigned int off, unsigned int len) +{ + return 0; +} +#endif + static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, - size_t offset, size_t len) + size_t offset, size_t len, size_t progress) { const char *kaddr; __wsum csum; @@ -7356,7 +7465,8 @@ static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, kaddr = kmap_local_page(page); csum = csum_partial(kaddr + offset, len, 0); kunmap_local(kaddr); - skb->csum = csum_block_add(skb->csum, csum, skb->len); + /* skb->len is updated only after the entire iterator batch. */ + skb->csum = csum_block_add(skb->csum, csum, skb->len + progress); } /** @@ -7371,7 +7481,8 @@ static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, * pages). * * Returns the amount of data spliced/copied or -EMSGSIZE if there's - * insufficient space in the buffer to transfer anything. + * insufficient space in the buffer to transfer anything, or -EEXIST if the + * buffer already has an incompatible fragment owner. */ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, ssize_t maxsize) @@ -7408,6 +7519,12 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, if (WARN_ON_ONCE(!sendpage_ok(page))) goto out; + ret = skb_splice_hold_subpages(skb, page, off, part); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + ret = skb_append_pagefrags(skb, page, off, part, frag_limit); if (ret < 0) { @@ -7416,7 +7533,7 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, } if (skb->ip_summed == CHECKSUM_NONE) - skb_splice_csum_page(skb, page, off, part); + skb_splice_csum_page(skb, page, off, part, spliced); off = 0; spliced += part; diff --git a/net/core/skmsg.c b/net/core/skmsg.c index 30c3b9a2681c44..10eb255be0ffa4 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -1,6 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2017 - 2018 Covalent IO, Inc. http://covalent.io */ +#include +#include #include #include #include @@ -24,6 +26,65 @@ static bool sk_msg_try_coalesce_ok(struct sk_msg *msg, int elem_first_coalesce) return false; } +#ifdef CONFIG_MM_SUBPAGE +static bool sk_msg_keeps_subpage(const struct mm_subpage *slot, void *context) +{ + struct sk_msg *msg = context; + phys_addr_t phys = mm_subpage_phys(slot); + phys_addr_t end = phys + mm_subpage_size(slot); + u32 i = msg->sg.start; + + if (!msg->sg.size) + return false; + while (i != msg->sg.end) { + struct scatterlist *sg = &msg->sg.data[i]; + + if (sg->length) { + phys_addr_t start = page_to_phys(sg_page(sg)) + sg->offset; + + if (start < end && phys < start + sg->length) + return true; + } + sk_msg_iter_var_next(i); + } + return false; +} + +int sk_msg_subpages_add(struct sk_msg *msg, struct page *page, u32 offset, u32 len) +{ + int ret; + + ret = mm_subpage_refs_prune(&msg->subpage_refs, sk_msg_keeps_subpage, msg, GFP_ATOMIC); + if (ret) + return ret; + return mm_subpage_refs_add(&msg->subpage_refs, page, offset, len, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(sk_msg_subpages_add); + +/* Called after a borrowed descriptor copy whose source remains live. */ +void sk_msg_subpages_get(struct sk_msg *msg) +{ + mm_subpage_refs_get(msg->subpage_refs); +} +EXPORT_SYMBOL_GPL(sk_msg_subpages_get); + +void sk_msg_subpages_share(struct sk_msg *dst, const struct sk_msg *src) +{ + WARN_ON_ONCE(dst->subpage_refs); + dst->subpage_refs = mm_subpage_refs_get(src->subpage_refs); +} +EXPORT_SYMBOL_GPL(sk_msg_subpages_share); + +void sk_msg_subpages_release(struct sk_msg *msg) +{ + struct mm_subpage_refs *refs = msg->subpage_refs; + + msg->subpage_refs = NULL; + mm_subpage_refs_put(refs); +} +EXPORT_SYMBOL_GPL(sk_msg_subpages_release); +#endif + int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len, int elem_first_coalesce) { @@ -88,7 +149,7 @@ EXPORT_SYMBOL_GPL(sk_msg_alloc); int sk_msg_clone(struct sock *sk, struct sk_msg *dst, struct sk_msg *src, u32 off, u32 len) { - int i = src->sg.start; + int i = src->sg.start, ret; struct scatterlist *sge = sk_msg_elem(src, i); struct scatterlist *sgd = NULL; u32 sge_len, sge_off; @@ -114,11 +175,16 @@ int sk_msg_clone(struct sock *sk, struct sk_msg *dst, struct sk_msg *src, if (sgd && (sg_page(sge) == sg_page(sgd)) && (sg_virt(sge) + off == sg_virt(sgd) + sgd->length)) { + ret = sk_msg_subpages_add(dst, sg_page(sge), sge->offset + off, sge_len); + if (ret) + return ret; sgd->length += sge_len; dst->sg.size += sge_len; } else if (!sk_msg_full(dst)) { sge_off = sge->offset + off; - sk_msg_page_add(dst, sg_page(sge), sge_len, sge_off); + ret = sk_msg_page_add(dst, sg_page(sge), sge_len, sge_off); + if (ret) + return ret; } else { return -ENOSPC; } @@ -205,6 +271,7 @@ static int __sk_msg_free(struct sock *sk, struct sk_msg *msg, u32 i, sk_msg_check_to_free(msg, i, msg->sg.size); sge = sk_msg_elem(msg, i); } + sk_msg_subpages_release(msg); consume_skb(msg->skb); sk_msg_init(msg); return freed; @@ -248,6 +315,7 @@ static void __sk_msg_free_partial(struct sock *sk, struct sk_msg *msg, sk_msg_check_to_free(msg, i, bytes); } msg->sg.start = i; + sk_msg_subpages_drop_empty(msg); } void sk_msg_free_partial(struct sock *sk, struct sk_msg *msg, u32 bytes) @@ -267,6 +335,7 @@ void sk_msg_trim(struct sock *sk, struct sk_msg *msg, int len) int trim = msg->sg.size - len; u32 i = msg->sg.end; + sk_msg_subpages_drop_empty(msg); if (trim <= 0) { WARN_ON(trim < 0); return; @@ -299,6 +368,7 @@ void sk_msg_trim(struct sock *sk, struct sk_msg *msg, int len) * does not require an update. */ if (!msg->sg.size) { + sk_msg_subpages_release(msg); msg->sg.curr = msg->sg.start; msg->sg.copybreak = 0; } else if (sk_msg_iter_dist(msg->sg.start, msg->sg.curr) >= @@ -334,11 +404,19 @@ int sk_msg_zerocopy_from_iter(struct sock *sk, struct iov_iter *from, goto out; } - bytes -= copied; - msg->sg.size += copied; - while (copied) { use = min_t(int, copied, PAGE_SIZE - offset); + ret = sk_msg_subpages_add(msg, pages[i], offset, use); + if (ret) { + unsigned int nr = DIV_ROUND_UP(offset + copied, PAGE_SIZE); + + while (nr--) + put_page(pages[i++]); + iov_iter_revert(from, copied); + goto out; + } + bytes -= use; + msg->sg.size += use; sg_set_page(&msg->sg.data[msg->sg.end], pages[i], use, offset); sg_unmark_end(&msg->sg.data[msg->sg.end]); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 389a7cc17110da..05c8c9504bb3c0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1356,7 +1356,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) err = skb_splice_from_iter(skb, &msg->msg_iter, copy); if (err < 0) { - if (err == -EMSGSIZE) { + if (err == -EMSGSIZE || err == -EEXIST) { tcp_mark_push(tp, skb); goto new_segment; } diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 18b45950d1accc..d082be17d76fc6 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -44,6 +44,7 @@ static int bpf_tcp_ingress(struct sock *sk, struct sk_psock *psock, return -ENOMEM; lock_sock(sk); + sk_msg_subpages_share(tmp, msg); tmp->sg.start = msg->sg.start; i = msg->sg.start; do { @@ -76,6 +77,7 @@ static int bpf_tcp_ingress(struct sock *sk, struct sk_psock *psock, if (!ret) { msg->sg.start = i; + sk_msg_subpages_drop_empty(msg); if (!sk_psock_queue_msg(psock, tmp)) atomic_sub(copied, &sk->sk_rmem_alloc); sk_psock_data_ready(sk, psock); @@ -147,6 +149,7 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, break; } + sk_msg_subpages_drop_empty(msg); return 0; } @@ -434,6 +437,8 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, } } memcpy(psock->cork, msg, sizeof(*msg)); + if (psock->cork != msg) + sk_msg_subpages_transfer(psock->cork, msg); return 0; } diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 33c42f65364a22..10da3de523b93c 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -32,6 +32,8 @@ #include #include #include +#include +#include #include #include #include @@ -144,6 +146,10 @@ static void destroy_record(struct tls_record_info *record) for (i = 0; i < record->num_frags; i++) __skb_frag_unref(&record->frags[i], false); + +#ifdef CONFIG_MM_SUBPAGE + mm_subpage_refs_put(record->subpage_refs); +#endif kfree(record); } @@ -349,6 +355,9 @@ static int tls_create_new_record(struct tls_offload_context_tx *offload_ctx, record = kmalloc_obj(*record); if (!record) return -ENOMEM; +#ifdef CONFIG_MM_SUBPAGE + record->subpage_refs = NULL; +#endif frag = &record->frags[0]; skb_frag_fill_page_desc(frag, pfrag->page, pfrag->offset, @@ -513,6 +522,15 @@ static int tls_push_data(struct sock *sk, goto handle_error; } +#ifdef CONFIG_MM_SUBPAGE + rc = mm_subpage_refs_add(&record->subpage_refs, zc_pfrag.page, + off, copy, GFP_KERNEL); + if (rc) { + iov_iter_revert(iter, copy); + goto handle_error; + } +#endif + zc_pfrag.offset = off; zc_pfrag.size = copy; tls_append_frag(record, &zc_pfrag, copy); @@ -1118,6 +1136,10 @@ int tls_set_device_offload(struct sock *sk) goto release_netdev; } + +#ifdef CONFIG_MM_SUBPAGE + start_marker_record->subpage_refs = NULL; +#endif offload_ctx = alloc_offload_ctx_tx(ctx); if (!offload_ctx) { rc = -ENOMEM; diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index ca0f657121a3b9..f4f708533f79b9 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -35,8 +35,11 @@ * SOFTWARE. */ +#include #include #include +#include +#include #include #include #include @@ -65,9 +68,36 @@ struct tls_decrypt_ctx { u8 aad[TLS_MAX_AAD_SIZE]; u8 tail; bool free_sgout; +#ifdef CONFIG_MM_SUBPAGE + struct user_page_fragment *fragments; + unsigned int nr_fragments; + bool fragments_dirty; + struct llist_node cleanup; + void *allocation; +#endif struct scatterlist sg[]; }; +#if IS_ENABLED(CONFIG_KUNIT) && IS_ENABLED(CONFIG_MM_SUBPAGE) +/* Explicit test-only async provider selection; normal cipher selection stays intact. */ +static bool user4k_test_async; +module_param_named(user4k_test_async, user4k_test_async, bool, 0400); +MODULE_PARM_DESC(user4k_test_async, "KUnit: select cryptd for TLS receive testing"); +static atomic_long_t user4k_rx_deferred = ATOMIC_LONG_INIT(0); +static atomic_long_t user4k_rx_released = ATOMIC_LONG_INIT(0); + +static int user4k_read_count(char *buffer, const struct kernel_param *param) +{ + return scnprintf(buffer, PAGE_SIZE, "%ld\n", atomic_long_read(param->arg)); +} + +static const struct kernel_param_ops user4k_count_ops = { + .get = user4k_read_count, +}; +module_param_cb(user4k_rx_deferred, &user4k_count_ops, &user4k_rx_deferred, 0444); +module_param_cb(user4k_rx_released, &user4k_count_ops, &user4k_rx_released, 0444); +#endif + noinline void tls_err_abort(struct sock *sk, int err) { WARN_ON_ONCE(err >= 0); @@ -184,6 +214,35 @@ static int tls_padding_length(struct tls_prot_info *prot, struct sk_buff *skb, return sub; } +#ifdef CONFIG_MM_SUBPAGE +static void tls_release_user_fragments(struct tls_decrypt_ctx *dctx) +{ + release_user_fragments(dctx->fragments, dctx->nr_fragments, dctx->fragments_dirty); + kfree(dctx->fragments); + dctx->fragments = NULL; + dctx->nr_fragments = 0; +} +#endif + +/* Dirty pin release can take folio locks, so it belongs to the waiting task. */ +static void tls_finish_subpage_decrypts(struct tls_sw_context_rx *ctx) +{ +#ifdef CONFIG_MM_SUBPAGE + struct tls_decrypt_ctx *dctx, *next; + struct llist_node *head = llist_del_all(&ctx->subpage_cleanup); + + llist_for_each_entry_safe(dctx, next, head, cleanup) { + void *allocation = dctx->allocation; + + tls_release_user_fragments(dctx); +#if IS_ENABLED(CONFIG_KUNIT) + atomic_long_inc(&user4k_rx_released); +#endif + kfree(allocation); + } +#endif +} + static void tls_decrypt_done(void *data, int err) { struct aead_request *aead_req = data; @@ -196,6 +255,7 @@ static void tls_decrypt_done(void *data, int err) unsigned int pages; struct sock *sk; int aead_size; + bool deferred = false; /* If requests get too backlogged crypto API returns -EBUSY and calls * ->complete(-EINPROGRESS) immediately followed by ->complete(0) @@ -224,17 +284,29 @@ static void tls_decrypt_done(void *data, int err) tls_err_abort(sk, err); } - /* Free the destination pages if skb was not decrypted inplace */ - if (dctx->free_sgout) { - /* Skip the first S/G entry as it points to AAD */ - for_each_sg(sg_next(sgout), sg, UINT_MAX, pages) { - if (!sg) - break; - put_page(sg_page(sg)); +#ifdef CONFIG_MM_SUBPAGE + if (dctx->fragments) { + dctx->allocation = aead_req; +#if IS_ENABLED(CONFIG_KUNIT) + atomic_long_inc(&user4k_rx_deferred); +#endif + llist_add(&dctx->cleanup, &ctx->subpage_cleanup); + deferred = true; + } +#endif + if (!deferred) { + /* Free the destination pages if skb was not decrypted inplace */ + if (dctx->free_sgout) { + /* Skip the first S/G entry as it points to AAD */ + for_each_sg(sg_next(sgout), sg, UINT_MAX, pages) { + if (!sg) + break; + put_page(sg_page(sg)); + } } - } - kfree(aead_req); + kfree(aead_req); + } if (atomic_dec_and_test(&ctx->decrypt_pending)) complete(&ctx->async_wait.completion); @@ -246,6 +318,7 @@ static int tls_decrypt_async_wait(struct tls_sw_context_rx *ctx) crypto_wait_req(-EINPROGRESS, &ctx->async_wait); atomic_inc(&ctx->decrypt_pending); + tls_finish_subpage_decrypts(ctx); __skb_queue_purge(&ctx->async_hold); return ctx->async_wait.err; } @@ -670,6 +743,7 @@ static int tls_split_open_record(struct sock *sk, struct tls_rec *from, msg_opl->sg.size = bytes; msg_npl = &new->msg_plaintext; + sk_msg_subpages_share(msg_npl, msg_opl); msg_npl->apply_bytes = apply; msg_npl->sg.size = orig_size - bytes; @@ -733,6 +807,7 @@ static void tls_merge_open_record(struct sock *sk, struct tls_rec *to, sk_msg_free(sk, &to->msg_encrypted); sk_msg_xfer_full(&to->msg_encrypted, &from->msg_encrypted); + sk_msg_subpages_release(msg_npl); kfree(from); } @@ -935,12 +1010,14 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk, redir_ingress = psock->redir_ingress; sk_redir = psock->sk_redir; memcpy(&msg_redir, msg, sizeof(*msg)); + sk_msg_subpages_get(&msg_redir); if (msg->apply_bytes < send) msg->apply_bytes = 0; else msg->apply_bytes -= send; sk_msg_return_zero(sk, msg, send); msg->sg.size -= send; + sk_msg_subpages_drop_empty(msg); release_sock(sk); err = tcp_bpf_sendmsg_redir(sk_redir, redir_ingress, &msg_redir, send, flags); @@ -956,6 +1033,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk, *copied -= sk_msg_free_nocharge(sk, &msg_redir); msg->sg.size = 0; } + sk_msg_subpages_release(&msg_redir); if (msg->sg.size == 0) tls_free_open_rec(sk); break; @@ -1036,7 +1114,14 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return -EIO; } - sk_msg_page_add(msg_pl, page, part, off); + { + int ret = sk_msg_page_add(msg_pl, page, part, off); + + if (ret) { + iov_iter_revert(&msg->msg_iter, part); + return ret; + } + } msg_pl->sg.copybreak = 0; msg_pl->sg.curr = msg_pl->sg.end; sk_mem_charge(sk, part); @@ -1454,10 +1539,96 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock, return 1; } +#ifdef CONFIG_MM_SUBPAGE +static bool tls_subpage_iter(const struct iov_iter *iter) +{ + return user_backed_iter(iter) && current->mm && mm_page_size(current->mm) < PAGE_SIZE; +} +#endif + +static int tls_iter_sg_capacity(struct iov_iter *iter, size_t length) +{ +#ifdef CONFIG_MM_SUBPAGE + if (tls_subpage_iter(iter)) { + struct iov_iter walk = *iter; + unsigned int count = 0; + + length = min(length, iov_iter_count(iter)); + while (length) { + size_t bytes, offset; + + iov_iter_advance(&walk, 0); + bytes = min(length, iter_iov_len(&walk)); + if (WARN_ON_ONCE(!bytes)) + return INT_MAX; + offset = (unsigned long)iter_iov_addr(&walk) & (MM_SUBPAGE_SIZE - 1); + count += DIV_ROUND_UP(offset + bytes, MM_SUBPAGE_SIZE); + iov_iter_advance(&walk, bytes); + length -= bytes; + } + return count; + } +#endif + return iov_iter_npages_cap(iter, INT_MAX, length); +} + +#ifdef CONFIG_MM_SUBPAGE +static int tls_setup_subpage_iter(struct iov_iter *from, int length, int *pages_used, + struct scatterlist *to, int capacity, + struct tls_decrypt_ctx *dctx) +{ + unsigned int used = 0, total = 0; + int ret = 0; + + if (!length) + return 0; + if (WARN_ON_ONCE(*pages_used || capacity <= 0)) + return -EINVAL; + dctx->fragments = kmalloc_array(capacity, sizeof(*dctx->fragments), GFP_KERNEL); + if (!dctx->fragments) + return -ENOMEM; + while (length) { + long nr; + unsigned int i, bytes = 0; + + iov_iter_advance(from, 0); + nr = pin_user_fragments_remote(current->mm, + (unsigned long)iter_iov_addr(from), + min_t(size_t, length, iter_iov_len(from)), FOLL_WRITE, + dctx->fragments + used, capacity - used); + if (nr <= 0) { + ret = nr ?: -EFAULT; + goto out; + } + for (i = 0; i < nr; i++) { + struct user_page_fragment *fragment = &dctx->fragments[used]; + + sg_set_page(&to[used], + folio_page(fragment->folio, fragment->offset >> PAGE_SHIFT), + fragment->length, offset_in_page(fragment->offset)); + sg_unmark_end(&to[used]); + bytes += fragment->length; + used++; + } + dctx->nr_fragments = used; + iov_iter_advance(from, bytes); + length -= bytes; + total += bytes; + } + if (used) + sg_mark_end(&to[used - 1]); +out: + if (ret) + iov_iter_revert(from, total); + *pages_used = used; + return ret; +} +#endif + static int tls_setup_from_iter(struct iov_iter *from, int length, int *pages_used, struct scatterlist *to, - int to_max_pages) + int to_max_pages, struct tls_decrypt_ctx *dctx) { int rc = 0, i = 0, num_elem = *pages_used, maxpages; struct page *pages[MAX_SKB_FRAGS]; @@ -1465,6 +1636,10 @@ static int tls_setup_from_iter(struct iov_iter *from, ssize_t copied, use; size_t offset; +#ifdef CONFIG_MM_SUBPAGE + if (tls_subpage_iter(from)) + return tls_setup_subpage_iter(from, length, pages_used, to, to_max_pages, dctx); +#endif while (length > 0) { i = 0; maxpages = to_max_pages - num_elem; @@ -1582,7 +1757,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, if (out_iov) n_sgout = 1 + tail_pages + - iov_iter_npages_cap(out_iov, INT_MAX, data_len); + tls_iter_sg_capacity(out_iov, data_len); else n_sgout = sg_nents(out_sg); } else { @@ -1615,6 +1790,11 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, aead_req = (struct aead_request *)mem; dctx = (struct tls_decrypt_ctx *)(mem + aead_size); dctx->sk = sk; +#ifdef CONFIG_MM_SUBPAGE + dctx->fragments = NULL; + dctx->nr_fragments = 0; + dctx->fragments_dirty = false; +#endif sgin = &dctx->sg[0]; sgout = &dctx->sg[n_sgin]; @@ -1672,7 +1852,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, sg_set_buf(&sgout[0], dctx->aad, prot->aad_size); err = tls_setup_from_iter(out_iov, data_len, &pages, &sgout[1], - (n_sgout - 1 - tail_pages)); + (n_sgout - 1 - tail_pages), dctx); if (err < 0) goto exit_free_pages; @@ -1687,6 +1867,10 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, } dctx->free_sgout = !!pages; +#ifdef CONFIG_MM_SUBPAGE + /* An error from crypto may still leave writes in the destination. */ + dctx->fragments_dirty = true; +#endif /* Prepare and submit AEAD request */ err = tls_do_decryption(sk, sgin, sgout, dctx->iv, data_len + prot->tail_size, aead_req, darg); @@ -1715,6 +1899,12 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, darg->tail = dctx->tail; exit_free_pages: +#ifdef CONFIG_MM_SUBPAGE + if (dctx->fragments) { + tls_release_user_fragments(dctx); + pages = 0; + } +#endif /* Release the pages in case iov was mapped to pages */ for (; pages > 0; pages--) put_page(sg_page(&sgout[pages])); @@ -2795,6 +2985,9 @@ static struct tls_sw_context_rx *init_ctx_rx(struct tls_context *ctx) crypto_init_wait(&sw_ctx_rx->async_wait); atomic_set(&sw_ctx_rx->decrypt_pending, 1); +#ifdef CONFIG_MM_SUBPAGE + init_llist_head(&sw_ctx_rx->subpage_cleanup); +#endif init_waitqueue_head(&sw_ctx_rx->wq); skb_queue_head_init(&sw_ctx_rx->rx_list); skb_queue_head_init(&sw_ctx_rx->async_hold); @@ -2903,7 +3096,24 @@ int tls_set_sw_offload(struct sock *sk, int tx, rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc); if (!*aead) { - *aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0); +#if IS_ENABLED(CONFIG_KUNIT) && IS_ENABLED(CONFIG_MM_SUBPAGE) + if (!tx && user4k_test_async) { + struct crypto_aead *sync; + char name[CRYPTO_MAX_ALG_NAME]; + + /* cryptd instances are named for the concrete child driver. */ + sync = crypto_alloc_aead(cipher_desc->cipher_name, 0, CRYPTO_ALG_ASYNC); + if (IS_ERR(sync)) { + *aead = sync; + } else { + snprintf(name, sizeof(name), "cryptd(%s)", + crypto_tfm_alg_driver_name(crypto_aead_tfm(sync))); + *aead = crypto_alloc_aead(name, CRYPTO_ALG_ASYNC, CRYPTO_ALG_ASYNC); + crypto_free_aead(sync); + } + } else +#endif + *aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0); if (IS_ERR(*aead)) { rc = PTR_ERR(*aead); *aead = NULL; diff --git a/rust/Makefile b/rust/Makefile index bec9726f256ce9..2fee22bdb0e146 100644 --- a/rust/Makefile +++ b/rust/Makefile @@ -444,10 +444,13 @@ endif bindgen_c_flags_final = $(bindgen_c_flags_lto) -fno-builtin -D__BINDGEN__ # `--rust-target` points to our minimum supported Rust version. +# Keep bindgen layout assertions enabled: a lost alignment can change field +# offsets without changing the containing type size. See Documentation/rust/ +# bindgen-layout-checks.rst for generator requirements. quiet_cmd_bindgen = BINDGEN $@ cmd_bindgen = \ $(BINDGEN) $< $(bindgen_target_flags) --rust-target 1.85 \ - --use-core --with-derive-default --ctypes-prefix ffi --no-layout-tests \ + --use-core --with-derive-default --ctypes-prefix ffi \ --no-debug '.*' --enable-function-attribute-detection \ -o $@ -- $(bindgen_c_flags_final) -DMODULE \ $(bindgen_target_cflags) $(bindgen_target_extra) diff --git a/rust/kernel/drm/device.rs b/rust/kernel/drm/device.rs index 11f9f4e7bea4af..045e6444f88366 100644 --- a/rust/kernel/drm/device.rs +++ b/rust/kernel/drm/device.rs @@ -111,7 +111,7 @@ impl Device { fops: &Self::GEM_FOPS, }; - const GEM_FOPS: bindings::file_operations = drm::gem::create_fops(); + const GEM_FOPS: bindings::file_operations = drm::gem::create_fops(T::MODULE); /// Create a new `drm::Device` for a `drm::Driver`. pub fn new(dev: &device::Device, data: impl PinInit) -> Result> { diff --git a/rust/kernel/drm/driver.rs b/rust/kernel/drm/driver.rs index 260ed453e7f5b4..ca7b519d3c5991 100644 --- a/rust/kernel/drm/driver.rs +++ b/rust/kernel/drm/driver.rs @@ -122,6 +122,10 @@ pub trait Driver { /// Driver metadata const INFO: DriverInfo; + /// Module implementing the driver callbacks. Loadable drivers should provide + /// their `THIS_MODULE` so open files and exported DMA buffers retain the code. + const MODULE: Option<&'static crate::ThisModule> = None; + /// Feature flags const FEATURES: u32; diff --git a/rust/kernel/drm/file.rs b/rust/kernel/drm/file.rs index 3603e473c4e0f6..c3263018880286 100644 --- a/rust/kernel/drm/file.rs +++ b/rust/kernel/drm/file.rs @@ -48,6 +48,14 @@ impl File { self.0.get() } + /// Tag recent feeder work for temporary capacity hints during syncobj waits. + /// User scheduling attributes and CPU affinity remain unchanged. + pub fn set_syncobj_wait_hint(&self, util_min: u32, duration_ms: u32) { + // SAFETY: this File keeps the opened drm_file alive; the C helper + // bounds both arguments and serializes the per-file hint internally. + unsafe { bindings::drm_syncobj_set_wait_hint(self.as_raw(), util_min, duration_ms) }; + } + fn driver_priv(&self) -> *mut T { // SAFETY: By the type invariants of `Self`, `self.as_raw()` is always valid. unsafe { (*self.as_raw()).driver_priv }.cast() diff --git a/rust/kernel/drm/gem/mod.rs b/rust/kernel/drm/gem/mod.rs index f766e90d2dd7f3..42892c71f84d01 100644 --- a/rust/kernel/drm/gem/mod.rs +++ b/rust/kernel/drm/gem/mod.rs @@ -109,6 +109,15 @@ pub trait DriverObject: Sync + Send + Sized { ) -> Result::Object>> { unimplemented!() } + + /// Arguments for an object that the DRM core allocates on the driver's behalf, such as a + /// dma-buf imported from another device or a dumb buffer. + /// + /// The shmem helpers build the driver data of such an object from these arguments. If the + /// core then fails to initialize the object, it frees it without dropping the driver data. + fn core_object_args(_dev: &drm::Device, _size: usize) -> Result { + build_error!(crate::error::VTABLE_DEFAULT_ERROR) + } } /// Trait that represents a GEM object subtype @@ -469,12 +478,17 @@ impl DmaBuf { } } -pub(super) const fn create_fops() -> bindings::file_operations { +pub(super) const fn create_fops( + module: Option<&'static crate::ThisModule>, +) -> bindings::file_operations { // SAFETY: As by the type invariant, it is safe to initialize `bindings::file_operations` // zeroed. let mut fops: bindings::file_operations = unsafe { core::mem::zeroed() }; - fops.owner = core::ptr::null_mut(); + fops.owner = match module { + Some(module) => module.as_ptr(), + None => core::ptr::null_mut(), + }; fops.open = Some(bindings::drm_open); fops.release = Some(bindings::drm_release); fops.unlocked_ioctl = Some(bindings::drm_ioctl); diff --git a/rust/kernel/drm/gem/shmem.rs b/rust/kernel/drm/gem/shmem.rs index 6536c9be946849..01be8d1c032622 100644 --- a/rust/kernel/drm/gem/shmem.rs +++ b/rust/kernel/drm/gem/shmem.rs @@ -38,6 +38,7 @@ use core::{ DerefMut, // }, ptr::NonNull, + sync::atomic::{AtomicUsize, Ordering}, }; use gem::{ BaseObject, @@ -55,6 +56,9 @@ pub struct ObjectConfig<'a, T: DriverObject> { /// Whether to set the write-combine map flag. pub map_wc: bool, + /// Opt in to revocable CPU VMAs, with driver-controlled idle eviction. + pub reclaimable_cpu_mappings: bool, + /// Reclaim private backing pages before returning from the final free callback. /// The default retains the usual deferred file-release policy. pub synchronous_reclaim: bool, @@ -65,6 +69,10 @@ pub struct ObjectConfig<'a, T: DriverObject> { pub parent_resv_obj: Option<&'a Object>, } +const SGT_PERMANENT: usize = 1 << (usize::BITS - 1); +const SGT_RECLAIMING: usize = 1 << (usize::BITS - 2); +const SGT_USERS: usize = SGT_RECLAIMING - 1; + /// A shmem-backed GEM object. /// /// # Invariants @@ -78,6 +86,8 @@ pub struct Object { obj: Opaque, /// Parent object that owns this object's DMA reservation object. parent_resv_obj: Option>>, + /// Owned SGT readers, or permanent borrowed access, exclude backing eviction. + sgt_access: AtomicUsize, synchronous_reclaim: bool, #[pin] inner: T, @@ -135,6 +145,7 @@ impl Object { try_pin_init!(Self { obj <- Opaque::init_zeroed(), parent_resv_obj: config.parent_resv_obj.map(|p| p.into()), + sgt_access: AtomicUsize::new(0), synchronous_reclaim: config.synchronous_reclaim, inner <- T::new(dev, size, args), }), @@ -164,6 +175,7 @@ impl Object { // to have exclusive access - thus making this safe to hold a mutable reference to. let shmem = unsafe { &mut *obj.as_raw_shmem() }; shmem.set_map_wc(config.map_wc); + shmem.reclaimable_cpu_mappings = config.reclaimable_cpu_mappings; Ok(obj) } @@ -224,12 +236,84 @@ impl Object { let _ = unsafe { KBox::from_raw(this) }; } + /// `gem_create_object` callback, for the objects the DRM core allocates on the driver's + /// behalf (dma-buf imports and dumb buffers) before it initializes their shmem part. + /// + /// Every GEM object of the driver is taken to be an `Object`, these included, so build one + /// with the driver data from [`DriverObject::core_object_args`]. + extern "C" fn create_object_callback( + raw_dev: *mut bindings::drm_device, + size: usize, + ) -> *mut bindings::drm_gem_object { + // SAFETY: The DRM core only calls this for a device of `T::Driver`, which outlives the + // call. + let dev = unsafe { device::Device::::from_raw(raw_dev) }; + + let new = T::core_object_args(dev, size).and_then(|args| { + KBox::try_pin_init( + try_pin_init!(Self { + obj <- Opaque::init_zeroed(), + parent_resv_obj: None, + sgt_access: AtomicUsize::new(0), + // Imports have no private shmem file; dumb buffers keep the + // default deferred file release. + synchronous_reclaim: false, + inner <- T::new(dev, size, args), + }), + GFP_KERNEL, + ) + }); + let new = match new { + Ok(new) => new, + Err(e) => return e.to_ptr(), + }; + + // SAFETY: We never move out of `self`. The DRM core owns the object from here on and + // releases it through `free_callback()`. An object the core then fails to initialize is + // freed with `kfree()`, which the `KBox` allocator allows, without dropping `inner`. + let new = KBox::into_raw(unsafe { Pin::into_inner_unchecked(new) }); + + // SAFETY: `new` is a valid object that nothing else references yet. + unsafe { + let obj = (*new).obj.get(); + (*obj).base.funcs = &Self::VTABLE; + &raw mut (*obj).base + } + } + /// Creates (if necessary) and returns an immutable reference to a scatter-gather table of DMA /// pages for this object. /// /// This will pin the object in memory. #[inline] pub fn sg_table(&self) -> Result<&scatterlist::SGTable> { + // The borrowed reference can outlive any lock, so pin permanently. + self.acquire_sgt_access(true)?; + self.sg_table_resident() + } + + fn acquire_sgt_access(&self, permanent: bool) -> Result { + let mut state = self.sgt_access.load(Ordering::Acquire); + loop { + if state & SGT_RECLAIMING != 0 { + return Err(EBUSY); + } + let next = if permanent { + state | SGT_PERMANENT + } else { + if state & SGT_USERS == SGT_USERS { return Err(EOVERFLOW); } + state + 1 + }; + match self.sgt_access.compare_exchange_weak( + state, next, Ordering::AcqRel, Ordering::Acquire, + ) { + Ok(_) => return Ok(()), + Err(current) => state = current, + } + } + } + + fn sg_table_resident(&self) -> Result<&scatterlist::SGTable> { // SAFETY: // - drm_gem_shmem_get_pages_sgt is thread-safe. // - drm_gem_shmem_get_pages_sgt returns either a valid pointer to a scatterlist, or an @@ -252,12 +336,87 @@ impl Object { /// /// [`shmem::SGTable`]: SGTable pub fn owned_sg_table(&self) -> Result> { - Ok(SGTable { - sgt: self.sg_table()?.into(), - // INVARIANT: We take an owned refcount to `self` here, ensuring that `sgt` remains - // valid for as long as this `SGTable`. - _owner: self.into(), - }) + self.acquire_sgt_access(false)?; + let sgt = match self.sg_table_resident() { + Ok(sgt) => sgt.into(), + Err(error) => { + self.sgt_access.fetch_sub(1, Ordering::Release); + return Err(error); + } + }; + Ok(SGTable { sgt, _owner: self.into() }) + } + + /// Whether the caller's sole SG lease is the last user of the backing. + /// + /// # Safety + /// The caller holds the object's DMA reservation lock. + pub unsafe fn sole_sg_lease_releasable_locked(&self) -> bool { + self.sgt_access.load(Ordering::Acquire) == 1 && + // SAFETY: The caller establishes the reservation-lock contract. + unsafe { bindings::drm_gem_shmem_idle_pages_releasable_locked(self.as_raw_shmem()) } + } + + /// Release idle backing without discarding contents. A later + /// `owned_sg_table()` may return different DMA addresses. + /// + /// # Safety + /// + /// The caller holds this object's DMA reservation lock, has removed and + /// invalidated every device mapping, and excludes new device access. + pub unsafe fn release_idle_pages_locked(&self) -> Result { + self.sgt_access.compare_exchange( + 0, SGT_RECLAIMING, Ordering::AcqRel, Ordering::Acquire, + ).map_err(|_| EBUSY)?; + // SAFETY: The caller's contract plus the exclusive state above. + let result = to_result(unsafe { + bindings::drm_gem_shmem_release_idle_pages_locked(self.as_raw_shmem()) + }); + self.sgt_access.store(0, Ordering::Release); + result + } + + /// Apply purge advice (0 = needed, >0 = may be purged) under the + /// object's reservation lock. Returns false if the object was already + /// purged; the advice is then ignored. + pub fn madvise(&self, madv: i32) -> bool { + // SAFETY: The object retains its reservation; the C helper requires it held. + unsafe { + bindings::dma_resv_lock(self.raw_dma_resv(), core::ptr::null_mut()); + let retained = bindings::drm_gem_shmem_madvise_locked(self.as_raw_shmem(), madv) != 0; + bindings::dma_resv_unlock(self.raw_dma_resv()); + retained + } + } + + /// Current purge advice: negative once purged. + /// + /// # Safety + /// The caller holds the object's DMA reservation lock. + pub unsafe fn madv_locked(&self) -> i32 { + // SAFETY: Reservation ownership serializes the field. + unsafe { (*self.as_raw_shmem()).madv } + } + + /// Discard the backing of a purgeable object. The handle and logical + /// mappings remain; the contents are gone and `madv_locked()` turns negative. + /// + /// # Safety + /// + /// The caller holds this object's DMA reservation lock, has removed and + /// invalidated every device mapping, and no Rust SG lease remains. + pub unsafe fn purge_locked(&self) -> Result { + // SAFETY: The caller's contract; the exclusive state below excludes new leases. + if !unsafe { bindings::drm_gem_shmem_is_purgeable(self.as_raw_shmem()) } { + return Err(EBUSY); + } + self.sgt_access.compare_exchange( + 0, SGT_RECLAIMING, Ordering::AcqRel, Ordering::Acquire, + ).map_err(|_| EBUSY)?; + // SAFETY: See above. + unsafe { bindings::drm_gem_shmem_purge_locked(self.as_raw_shmem()) }; + self.sgt_access.store(0, Ordering::Release); + Ok(()) } /// Attempt to create a [`RawIoSysMap`] from the gem object. @@ -366,7 +525,11 @@ impl driver::AllocImpl for Object { type Driver = T::Driver; const ALLOC_OPS: driver::AllocOps = driver::AllocOps { - gem_create_object: None, + gem_create_object: if T::HAS_CORE_OBJECT_ARGS { + Some(Self::create_object_callback) + } else { + None + }, prime_handle_to_fd: None, prime_fd_to_handle: None, gem_prime_import: None, @@ -472,7 +635,8 @@ unsafe impl Sync for VMap {} /// /// # Invariants /// -/// - `sgt` is kept alive by `_owner`, ensuring it remains valid for as long as `Self`. +/// - `_owner` retains both the GEM object and one counted SG-table lease; eviction +/// cannot begin until every lease has been released. /// - `sgt` corresponds to the owned object in `_owner`. /// - This object is only exposed in situations where we know the underlying `SGTable` will not be /// modified for the lifetime of this object. Thus, it is safe to send/access this type across @@ -487,6 +651,12 @@ unsafe impl Send for SGTable {} // SAFETY: This object is thread-safe via our type invariants. unsafe impl Sync for SGTable {} +impl Drop for SGTable { + fn drop(&mut self) { + self._owner.sgt_access.fetch_sub(1, Ordering::Release); + } +} + impl Deref for SGTable { type Target = scatterlist::SGTable; diff --git a/rust/kernel/drm/gpuvm.rs b/rust/kernel/drm/gpuvm.rs index fbfab4bc7cf31a..7293df41f2dee1 100644 --- a/rust/kernel/drm/gpuvm.rs +++ b/rust/kernel/drm/gpuvm.rs @@ -21,6 +21,7 @@ use crate::{ EINVAL, ENOMEM, // }, + from_err_ptr, from_result, to_result, Error, @@ -31,12 +32,12 @@ use crate::{ ARef, AlwaysRefCounted, // }, - types::Opaque, // + types::{NotThreadSafe, Opaque}, // }; use core::cell::UnsafeCell; use core::marker::{PhantomData, PhantomPinned}; -use core::mem::ManuallyDrop; +use core::mem::MaybeUninit; use core::ops::{Deref, DerefMut, Range}; use core::ptr::NonNull; use pin_init; @@ -284,6 +285,15 @@ impl GpuVa { ) } + /// Driver metadata retained independently of the resident page tables. + pub fn inner(&self) -> &T::GpuVa { &self.inner } + + /// Update metadata before linking a newly allocated GPUVA. + pub fn inner_mut(self: Pin<&mut Self>) -> Pin<&mut T::GpuVa> { + // SAFETY: inner is structurally pinned with its enclosing GPUVA. + unsafe { self.map_unchecked_mut(|va| &mut va.inner) } + } + pub fn addr(&self) -> u64 { self.gpuva.va.addr } @@ -311,6 +321,13 @@ pub struct GpuVmBo { } impl GpuVmBo { + /// The GEM object retained by this VM/BO association. + pub fn object(&self) -> &Object { + // SAFETY: drm_gpuvm_bo owns this GEM reference until its destructor; + // DriverGpuVm fixes the GEM type for every association in this VM. + unsafe { &* as IntoGEMObject>::from_raw(self.bo.obj) } + } + /// Return a reference to the inner driver data for this GpuVmBo pub fn inner(&self) -> &T::GpuVmBo { &self.inner @@ -347,6 +364,23 @@ unsafe impl AlwaysRefCounted for GpuVmBo { // reference until after unlocking: the final vm_bo put can release // its GEM reference and otherwise free the mutex we still hold. unsafe { + let vm = (*obj.as_ptr()).bo.vm; + // SAFETY: Every drm_gpuvm with Rust GpuVmBo objects is embedded in a GpuVm. + let gpuvm = crate::container_of!( + vm.cast::>(), GpuVm, gpuvm + ); + if (*gpuvm).deferred_bo_put { + // The VM was created by `GpuVm::new_deferred_bo_put`, which requires + // immediate mode. The deferred put releases the GEM's GPUVA mutex before + // final GEM destruction. Keep the GPUVM alive across draining, which can + // drop its last BO-owned reference. + let bo = &raw mut (*obj.as_ptr()).bo; + bindings::drm_gpuvm_get(vm); + bindings::drm_gpuvm_bo_put_deferred(bo); + bindings::drm_gpuvm_bo_deferred_cleanup(vm); + bindings::drm_gpuvm_put(vm); + return; + } let bo = &mut obj.as_mut().bo; let gem = bo.obj; bindings::drm_gem_object_get(gem); @@ -378,6 +412,8 @@ pub struct GpuVm { gpuvm: Opaque, #[pin] inner: UnsafeCell, + /// Final VM/BO association puts are deferred (see [`GpuVm::new_deferred_bo_put`]). + deferred_bo_put: bool, #[pin] _p: PhantomPinned, } @@ -402,7 +438,10 @@ pub(super) unsafe extern "C" fn vm_bo_alloc_callback() -> *mut b { let obj: Result>>> = KBox::try_pin_init( try_pin_init!(GpuVmBo:: { - bo <- pin_init::default(), + // The bindgen `Default` implementation is exactly a zero fill, but its + // out-of-line symbol is not exported to loadable modules. `drm_gpuvm_bo` + // is `Zeroable`, so initialize it directly without a cross-crate call. + bo <- pin_init::zeroed::(), inner <- T::GpuVmBo::new(), _p: PhantomPinned }), @@ -529,6 +568,47 @@ impl GpuVm { reserve_range: Range, inner: impl PinInit, ) -> Result>> + where + Error: From, + { + Self::new_inner(name, flags, dev, r_obj, range, reserve_range, inner, false) + } + + /// Create an immediate-mode GPUVM whose final VM/BO association puts are deferred. + /// + /// Dropping the last [`GpuVmBo`] reference puts the association with + /// `drm_gpuvm_bo_put_deferred()` and then drains the VM's deferred list, instead of + /// putting it synchronously under the GEM's GPUVA lock. The GEM GPUVA mutex is + /// released before final GEM destruction. + pub fn new_deferred_bo_put( + name: &'static CStr, + flags: bindings::drm_gpuvm_flags, + dev: &device::Device, + r_obj: ARef>, + range: Range, + reserve_range: Range, + inner: impl PinInit, + ) -> Result>> + where + Error: From, + { + if flags & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { + return Err(EINVAL); + } + Self::new_inner(name, flags, dev, r_obj, range, reserve_range, inner, true) + } + + #[allow(clippy::too_many_arguments)] + fn new_inner( + name: &'static CStr, + flags: bindings::drm_gpuvm_flags, + dev: &device::Device, + r_obj: ARef>, + range: Range, + reserve_range: Range, + inner: impl PinInit, + deferred_bo_put: bool, + ) -> Result>> where Error: From, { @@ -560,6 +640,7 @@ impl GpuVm { inner.__pinned_init(slot as *mut _) }) }, + deferred_bo_put, _p: PhantomPinned }), GFP_KERNEL, @@ -575,6 +656,51 @@ impl GpuVm { Ok(vm_ref) } + /// Lock only the VM's driver data, without allocating an exec context. + /// + /// This takes the same shared reservation as `exec_lock`, but does not lock + /// external BO reservations. The returned guard exposes only the inner + /// driver data, not GPUVA operations requiring those other reservations. + /// Callers must not already hold a reservation lock. + pub fn lock_inner(&self) -> GpuVmInnerGuard<'_, T> { + // SAFETY: The borrowed GPUVM retains r_obj and its reservation for the + // guard's lifetime. A non-interruptible single-reservation lock with no + // ww context cannot fail or allocate. No other reservation is acquired. + unsafe { + bindings::dma_resv_lock((*(*self.gpuvm()).r_obj).resv, core::ptr::null_mut()); + } + GpuVmInnerGuard { + gpuvm: self, + _not_send: NotThreadSafe, + } + } + + // The caller owns the VM reservation and keeps the exact logical VA linked. + unsafe fn set_invalidated_locked(&self, addr: u64, range: u64, invalidated: bool) -> Result { + if range == 0 || addr.checked_add(range).is_none() { return Err(EINVAL); } + // SAFETY: The caller owns the VM reservation and keeps the VA linked. + let raw = unsafe { bindings::drm_gpuva_find(self.gpuvm() as *mut _, addr, range) }; + if raw.is_null() { return Err(EINVAL); } + // SAFETY: Reservation ownership excludes all concurrent VA mutation. + unsafe { + if (*raw).va.addr != addr || (*raw).va.range != range { return Err(EINVAL); } + if invalidated { (*raw).flags |= GpuVaFlags::INVALIDATED.as_raw(); } + else { (*raw).flags &= !GpuVaFlags::INVALIDATED.as_raw(); } + } + Ok(()) + } + + /// Try to lock only the VM reservation, without allocation or waiting. + /// The guard protects VM metadata and private BOs, not external backing. + /// Callers must not already hold a reservation lock. + pub fn try_lock_private(&self) -> Option> { + // SAFETY: r_obj and its reservation are retained by this GPUVM. + let resv = unsafe { (*(*self.gpuvm()).r_obj).resv }; + // SAFETY: The reservation outlives the returned borrow and guard. + if !unsafe { bindings::dma_resv_trylock(resv) } { return None; } + Some(PrivateGpuVmGuard { gpuvm: self, resv, _not_send: NotThreadSafe }) + } + pub fn exec_lock<'a, 'b>( &'a self, obj: Option<&'b Object>, @@ -583,36 +709,44 @@ impl GpuVm { // Do not try to lock the object if it is internal (since it is already locked). let is_ext = obj.map(|a| self.is_extobj(a)).unwrap_or(false); - let mut guard = ManuallyDrop::new(LockedGpuVm { - gpuvm: self, - // vm_exec needs to be pinned, so stick it in a Box. - vm_exec: KBox::init( - init!(bindings::drm_gpuvm_exec { - vm: self.gpuvm() as *mut _, - flags: if interruptible { - bindings::DRM_EXEC_INTERRUPTIBLE_WAIT - } else { - 0 + // vm_exec needs a stable address while locking. Construct the unlock + // guard only after success; C already finalizes the exec on error. + let mut vm_exec = KBox::init( + init!(bindings::drm_gpuvm_exec { + vm: self.gpuvm() as *mut _, + flags: if interruptible { + bindings::DRM_EXEC_INTERRUPTIBLE_WAIT + } else { + 0 + }, + // SAFETY: bindgen's `Default` for this C structure is exactly + // `MaybeUninit::zeroed().assume_init()`. Spell that out here so + // external modules do not import its unexported trait method. + exec: unsafe { MaybeUninit::::zeroed().assume_init() }, + extra: match (is_ext, obj) { + (true, Some(obj)) => bindings::drm_gpuvm_exec__bindgen_ty_1 { + fn_: Some(exec_lock_gem_object), + priv_: obj.as_raw() as *const _ as *mut _, }, - exec: Default::default(), - extra: match (is_ext, obj) { - (true, Some(obj)) => bindings::drm_gpuvm_exec__bindgen_ty_1 { - fn_: Some(exec_lock_gem_object), - priv_: obj.as_raw() as *const _ as *mut _, - }, - _ => Default::default(), + // SAFETY: as above, this bindgen C structure is zero-valid. + _ => unsafe { + MaybeUninit::::zeroed() + .assume_init() }, - num_fences: 0, - }), - GFP_KERNEL, - )?, - obj, - }); + }, + num_fences: 0, + }), + GFP_KERNEL, + )?; // SAFETY: The object is valid and was initialized above - to_result(unsafe { bindings::drm_gpuvm_exec_lock(&mut *guard.vm_exec) })?; + to_result(unsafe { bindings::drm_gpuvm_exec_lock(&mut *vm_exec) })?; - Ok(ManuallyDrop::into_inner(guard)) + Ok(LockedGpuVm { + gpuvm: self, + vm_exec, + obj, + }) } /// Returns true if the given object is external to the GPUVM @@ -623,6 +757,12 @@ impl GpuVm { unsafe { bindings::drm_gpuvm_is_extobj(self.gpuvm() as *mut _, gem) } } + /// Validate against the immutable address-space bounds and kernel cutout. + pub fn range_valid(&self, addr: u64, range: u64) -> bool { + // SAFETY: the borrowed VM retains its immutable geometry. + unsafe { bindings::drm_gpuvm_range_valid(self.gpuvm() as *mut _, addr, range) } + } + pub fn bo_deferred_cleanup(&self) { unsafe { bindings::drm_gpuvm_bo_deferred_cleanup(self.gpuvm() as *mut _) } } @@ -655,6 +795,7 @@ impl GpuVm { ) }; obj.unlock_gpuva(); + let p = from_err_ptr(p)?; if p.is_null() { Err(ENOMEM) } else { @@ -689,6 +830,307 @@ unsafe impl AlwaysRefCounted for GpuVm { } } +pub struct GpuVmInnerGuard<'a, T: DriverGpuVm> { + gpuvm: &'a GpuVm, + // Reservation locks must be released by the task that acquired them. + _not_send: NotThreadSafe, +} + +impl GpuVmInnerGuard<'_, T> { + /// Run immediate-mode unmap steps without allocating an exec context. + /// + /// # Safety + /// The driver's unmap/remap callbacks must touch only VM metadata and + /// translations protected by this reservation. They must not access + /// external BO backing that requires its own reservation. Any split-node + /// storage must already be owned by `ctx`; callbacks must defer BO puts. + pub unsafe fn sm_unmap_inner( + &mut self, ctx: &mut T::StepContext, addr: u64, range: u64, + ) -> Result { + let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; + // SAFETY: this guard retains the VM and its shared reservation. + if unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { + return Err(EINVAL); + } + let mut ctx = StepContext { gpuvm: self.gpuvm, ctx }; + // SAFETY: caller supplies the callback contract above. Immediate-mode + // link/unlink acquire the separate GEM GPUVA-list mutex themselves. + to_result(unsafe { + bindings::drm_gpuvm_sm_unmap(vm, &mut ctx as *mut _ as *mut _, addr, range) + }) + } + + /// Remove every driver GPUVA in an immediate-mode VM, preserving its kernel cutout. + /// + /// `unmap` must remove the GPU translation for each complete GPUVA before + /// returning success. Only the VM's shared reservation is held. No external + /// BO reservations are acquired and no split nodes or operation lists are + /// allocated. The callback cannot retain the borrowed GPUVA. + /// + /// Call `GpuVm::bo_deferred_cleanup` after releasing this guard, including + /// when a callback fails after earlier mappings have already been removed. + pub fn unmap_all( + &mut self, + unmap: impl FnMut(&mut T, &GpuVa) -> Result, + ) -> Result { + self.unmap_matching(None, unmap) + } + + /// Allocation-free whole-node removal for one GEM in an immediate-mode VM. + /// Uses the same translation-removal and deferred-put contract as unmap_all. + /// This scans the VM list and is intended for low-memory cleanup fallback. + pub fn unmap_object( + &mut self, + object: &Object, + unmap: impl FnMut(&mut T, &GpuVa) -> Result, + ) -> Result { + self.unmap_matching(Some(object), unmap) + } + + fn unmap_matching( + &mut self, + object: Option<&Object>, + mut unmap: impl FnMut(&mut T, &GpuVa) -> Result, + ) -> Result { + let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; + // SAFETY: The guard owns this live VM's shared reservation. Immediate + // mode lets unlink take the GEM GPUVA mutex independently of its resv. + unsafe { + if (*vm).flags & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { + return Err(EINVAL); + } + let head = &raw mut (*vm).rb.list; + let mut entry = (*head).next; + while entry != head { + let va = crate::container_of!(entry, bindings::drm_gpuva, rb.entry) as *mut _; + // Save the successor before removing and freeing this GPUVA. + entry = (*entry).next; + // This node is embedded in the C GPUVM, not a driver GpuVa. + if core::ptr::eq(va, &raw const (*vm).kernel_alloc_node) { + continue; + } + if object.is_some_and(|object| (*va).gem.obj != object.as_raw()) { + continue; + } + let driver_va = crate::container_of!(va, GpuVa, gpuva); + unmap(&mut *self.gpuvm.inner.get(), &*driver_va)?; + let mut op = OpUnMap( + bindings::drm_gpuva_op_unmap { va, keep: false }, + PhantomData::, + ); + drop(op.unmap_and_unlink_va_defer()); + } + } + Ok(()) + } +} + +impl Deref for GpuVmInnerGuard<'_, T> { + type Target = T; + + fn deref(&self) -> &T { + // SAFETY: The shared reservation excludes every other inner-data guard, + // including LockedGpuVm and its UpdatingGpuVm callbacks. + unsafe { &*self.gpuvm.inner.get() } + } +} + +impl DerefMut for GpuVmInnerGuard<'_, T> { + fn deref_mut(&mut self) -> &mut T { + // SAFETY: As above, with exclusive access to this guard. + unsafe { &mut *self.gpuvm.inner.get() } + } +} + +impl Drop for GpuVmInnerGuard<'_, T> { + fn drop(&mut self) { + // SAFETY: This non-Send guard owns the reservation on this task, and + // its GPUVM borrow keeps r_obj alive until after the unlock. + unsafe { bindings::dma_resv_unlock((*(*self.gpuvm.gpuvm()).r_obj).resv) }; + } +} + +/// Logical binding retained independently of hardware page-table residency. +/// Owns a GEM reference and copied metadata, never a GpuVmBo reference. +/// Driver metadata cloning must not introduce a reservation-taking destructor. +pub struct MappingSnapshot { + pub object: ARef>, + pub addr: u64, + pub range: u64, + pub offset: u64, + pub flags: GpuVaFlags, + pub inner: T::GpuVa, +} + +/// Nonblocking reservation ownership for pressure scans of private objects. +pub struct PrivateGpuVmGuard<'a, T: DriverGpuVm> { + gpuvm: &'a GpuVm, + resv: *mut bindings::dma_resv, + // DMA reservation ownership must remain on the acquiring task. + _not_send: NotThreadSafe, +} + +impl PrivateGpuVmGuard<'_, T> { + /// Visit each private BO with linked logical mappings once, without allocating or retaining + /// references. Returning false stops the walk. Contended GEM lists are skipped. + /// + /// # Safety + /// The callback must not insert/remove logical mappings or modify either GPUVA list. This + /// reservation keeps their linked BO/GEM references alive for the complete callback. + /// The callback must not retain a borrowed BO or acquire its GEM GPUVA mutex recursively. + pub unsafe fn for_each_private_bo( + &mut self, + mut visit: impl FnMut(&mut Self, &GpuVmBo) -> Result, + ) -> Result { + let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; + if unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { + return Err(EINVAL); + } + let head = unsafe { &raw const (*vm).rb.list }; + // SAFETY: The VM reservation protects this ordered mapping list. The callback does not + // mutate its links, and no borrowed or owned reference outlives the guard. + let mut node = unsafe { (*head).next }; + while node != head.cast_mut() { + let (next, raw) = unsafe { + ((*node).next, crate::container_of!(node, bindings::drm_gpuva, rb.entry)) + }; + let va = unsafe { &*raw }; + if !va.gem.obj.is_null() && !va.vm_bo.is_null() { + let object = unsafe { &*Object::::from_raw(va.gem.obj) }; + if !self.gpuvm.is_extobj(object) { + let lock = unsafe { &raw mut (*va.gem.obj).gpuva.lock }; + // Never block direct reclaim on a GEM-list owner. The canonical alias is + // the list's first-linked node, not its lowest address or current residency. + if unsafe { bindings::mutex_trylock(lock) } != 0 { + let canonical = unsafe { + (*va.vm_bo).list.gpuva.next == (&raw const va.gem.entry).cast_mut() + }; + unsafe { bindings::mutex_unlock(lock) }; + if canonical { + let bo = unsafe { + &*crate::container_of!(va.vm_bo, GpuVmBo, bo) + }; + if !visit(self, bo)? { break; } + } + } + } + } + node = next; + } + Ok(()) + } + + /// Mark and visit every logical alias of this private BO without a snapshot allocation. + /// GEM-list locking is nonblocking and held only while reading a cursor, never during the + /// callback. Failure (including contention) can leave a prefix marked INVALIDATED; callers + /// must retain backing unless this method succeeds for the entire association. + /// + /// # Safety + /// The callback must not insert/remove logical mappings, alter list links, or release the + /// BO's SG/backing lease. The BO must remain linked throughout the walk. Exclude new jobs and + /// binding edits for the whole invalidation and backing-release transaction. + pub unsafe fn invalidate_private_bo_mappings( + &mut self, + bo: &GpuVmBo, + mut visit: impl FnMut(&mut Self, u64, u64) -> Result, + ) -> Result { + let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; + if bo.bo.vm != vm || self.gpuvm.is_extobj(bo.object()) + || unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 + { return Err(EINVAL); } + let head = &raw const bo.bo.list.gpuva; + let lock = unsafe { &raw mut (*bo.bo.obj).gpuva.lock }; + if unsafe { bindings::mutex_trylock(lock) } == 0 { return Err(EAGAIN); } + let mut node = unsafe { (*head).next }; + unsafe { bindings::mutex_unlock(lock) }; + while node != head.cast_mut() { + if unsafe { bindings::mutex_trylock(lock) } == 0 { return Err(EAGAIN); } + // SAFETY: The list mutex protects link reads; this reservation and the caller's + // no-unlink contract keep current/next and their association alive after unlocking. + let (next, raw) = unsafe { + ((*node).next, crate::container_of!(node, bindings::drm_gpuva, gem.entry)) + }; + unsafe { bindings::mutex_unlock(lock) }; + let va = unsafe { &mut *raw }; + if va.vm != vm || va.vm_bo != (&raw const bo.bo).cast_mut() + || va.va.range == 0 || va.va.addr.checked_add(va.va.range).is_none() + { return Err(EINVAL); } + // Flags are serialized by the VM reservation, independently of GEM's list mutex. + va.flags |= GpuVaFlags::INVALIDATED.as_raw(); + visit(self, va.va.addr, va.va.range)?; + node = next; + } + Ok(()) + } + + /// Retain the next private mapping containing all `required_flags`. + /// + /// `cursor` is an exclusive end address and advances past skipped mappings + /// as well. No collection is allocated; only the returned object is retained. + /// Callers must drop this reservation guard before releasing that reference + /// or acquiring backing. To cover a stable set across calls, separately + /// exclude logical binding edits (including retirement) for the whole walk. + pub fn next_private_mapping( + &self, + cursor: &mut u64, + required_flags: GpuVaFlags, + ) -> Result>> + where T::GpuVa: Clone { + while *cursor != u64::MAX { + // SAFETY: The VM reservation serializes mutation of its VA tree. + let raw = unsafe { + bindings::drm_gpuva_find_first( + self.gpuvm.gpuvm() as *mut _, *cursor, u64::MAX - *cursor, + ) + }; + if raw.is_null() { return Ok(None); } + // SAFETY: raw stays linked and alive while this guard is held. + let va = unsafe { &*raw }; + let end = va.va.addr.checked_add(va.va.range).ok_or(EINVAL)?; + if end <= *cursor { return Err(EINVAL); } + *cursor = end; + if !GpuVaFlags(va.flags).contains(required_flags) + || va.gem.obj.is_null() || va.vm_bo.is_null() { + continue; + } + // SAFETY: The linked VA retains this driver's object. + let obj = unsafe { &*Object::::from_raw(va.gem.obj) }; + if self.gpuvm.is_extobj(obj) { continue; } + // SAFETY: Driver-owned mappings have this allocation layout. The + // special kernel cutout has no GEM object and was skipped above. + let typed = unsafe { &*crate::container_of!(raw, GpuVa, gpuva) }; + return Ok(Some(MappingSnapshot { + object: obj.into(), addr: va.va.addr, + range: va.va.range, offset: va.gem.offset, + flags: GpuVaFlags(va.flags), inner: typed.inner.clone(), + })); + } + Ok(None) + } + + +} + +impl Deref for PrivateGpuVmGuard<'_, T> { + type Target = T; + fn deref(&self) -> &T { + // SAFETY: The reservation serializes all access to the VM's inner data. + unsafe { &*self.gpuvm.inner.get() } + } +} +impl DerefMut for PrivateGpuVmGuard<'_, T> { + fn deref_mut(&mut self) -> &mut T { + // SAFETY: This guard has exclusive reservation ownership. + unsafe { &mut *self.gpuvm.inner.get() } + } +} +impl Drop for PrivateGpuVmGuard<'_, T> { + fn drop(&mut self) { + // SAFETY: try_lock_private acquired exactly this reservation. + unsafe { bindings::dma_resv_unlock(self.resv) }; + } +} + pub struct LockedGpuVm<'a, 'b, T: DriverGpuVm> { gpuvm: &'a GpuVm, vm_exec: KBox, @@ -696,6 +1138,12 @@ pub struct LockedGpuVm<'a, 'b, T: DriverGpuVm> { } impl LockedGpuVm<'_, '_, T> { + /// Mark the exact retained logical mapping after page-table restoration. + pub fn set_invalidated(&mut self, addr: u64, range: u64, invalidated: bool) -> Result { + // SAFETY: Both guard types own the VM reservation. + unsafe { self.gpuvm.set_invalidated_locked(addr, range, invalidated) } + } + pub fn find_bo(&mut self) -> Option>> { let obj = self.obj?; // SAFETY: LockedGpuVm implies the right locks are held. @@ -725,6 +1173,7 @@ impl LockedGpuVm<'_, '_, T> { obj.as_raw() as *const _ as *mut _, ) }; + let p = from_err_ptr(p)?; if p.is_null() { Err(ENOMEM) } else { diff --git a/rust/kernel/drm/sched.rs b/rust/kernel/drm/sched.rs index 7275fd254e7e8e..a76bf624237c83 100644 --- a/rust/kernel/drm/sched.rs +++ b/rust/kernel/drm/sched.rs @@ -44,25 +44,51 @@ pub enum Priority { /// Trait to be implemented by driver job objects. pub trait JobImpl: Sized { + /// State owned exclusively by the timeout worker. It must not borrow the + /// mutable submission state: timeout can run before `run` returns. + type TimeoutData; + + /// Snapshot or independently retain timeout state before scheduler ownership. + fn timeout_data(&self) -> Self::TimeoutData; + + /// Module implementing the callbacks. Its caller must retain the module + /// through entity destruction; the core pins detached kill-job callbacks. + const MODULE: Option<&'static crate::ThisModule> = None; + + /// Opt in to synchronized cancellation and teardown. + /// + /// When set, scheduler teardown calls `drm_sched_fini()` without a preceding + /// `drm_sched_stop()`, and the cancel callback removes the job's parent fence + /// callback under the parent's lock before it settles the scheduler's credit and + /// score accounting exactly once. Backends that leave this unset keep the + /// stop-then-fini teardown and the unconditional cancel signal. + const SYNCHRONIZED_CANCEL: bool = false; + + /// Reconsider a timeout before stopping the scheduler or settling fences. + /// Returning true asks the core to reinsert the retained job and rearm. + /// The backend must not stop/start the scheduler from this hook. + fn false_timeout(_data: &mut Self::TimeoutData, _finished: bool) -> bool { false } + /// Called when the scheduler is considering scheduling this job next, to get another Fence /// for this job to block on. Once it returns None, run() may be called. - fn prepare(_job: &mut Job) -> Option { + fn prepare(_job: &mut Self) -> Option { None // Equivalent to NULL function pointer } /// Called to execute the job once all of the dependencies have been resolved. This may be /// called multiple times, if timed_out() has happened and drm_sched_job_recovery() decides /// to try it again. - fn run(job: &mut Job) -> Result>; + fn run(job: &mut Self) -> Result>; /// Called when a job has taken too long to execute, to trigger GPU recovery. /// - /// This method is called in a workqueue context. - fn timed_out(job: &mut Job) -> Status; + /// This method is called in a workqueue context. `finished` reports whether the job's + /// scheduler fence had already signalled. + fn timed_out(data: &mut Self::TimeoutData, finished: bool) -> Status; /// Called for remaining jobs in drm_sched_fini() to ensure the job's fences /// get signalled before the scheduler is torn down. - fn cancel(job: &mut Job); + fn cancel(job: &mut Self); } unsafe extern "C" fn prepare_job_cb( @@ -72,8 +98,9 @@ unsafe extern "C" fn prepare_job_cb( // SAFETY: All of our jobs are Job. let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - // SAFETY: All of our jobs are Job. - match T::prepare(unsafe { &mut *p }) { + // SAFETY: prepare/run are serialized by submit work. Only borrow inner; + // the timeout worker independently owns timeout_data and C owns job. + match T::prepare(unsafe { &mut *addr_of_mut!((*p).inner) }) { None => core::ptr::null_mut(), Some(fence) => fence.into_raw(), } @@ -85,8 +112,9 @@ unsafe extern "C" fn run_job_cb( // SAFETY: All of our jobs are Job. let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - // SAFETY: All of our jobs are Job. - match T::run(unsafe { &mut *p }) { + // SAFETY: submit work owns inner until return, even while timeout work + // accesses the disjoint timeout_data field. Do not borrow the whole Job. + match T::run(unsafe { &mut *addr_of_mut!((*p).inner) }) { Err(e) => e.to_ptr(), Ok(None) => core::ptr::null_mut(), Ok(Some(fence)) => fence.into_raw(), @@ -99,8 +127,19 @@ unsafe extern "C" fn timedout_job_cb( // SAFETY: All of our jobs are Job. let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - // SAFETY: All of our jobs are Job. - T::timed_out(unsafe { &mut *p }) as bindings::drm_gpu_sched_stat + // The C timeout worker owns this job outside pending_list until return. + // Free work cannot release it even if its parent signals concurrently. + // SAFETY: An armed job has a valid scheduler fence for its lifetime. + let finished = unsafe { addr_of_mut!((*(*sched_job).s_fence).finished) }; + // SAFETY: `finished` stays valid while the timeout worker owns the job. + let is_finished = || unsafe { bindings::dma_fence_get_status(finished) != 0 }; + // SAFETY: the C core retains the detached job and serializes its timeout + // worker. This field is disjoint from inner, which run may still mutate. + let data = unsafe { &mut *addr_of_mut!((*p).timeout_data) }; + if T::false_timeout(data, is_finished()) { + return bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_NO_HANG; + } + T::timed_out(data, is_finished()) as bindings::drm_gpu_sched_stat } unsafe extern "C" fn free_job_cb(sched_job: *mut bindings::drm_sched_job) { @@ -116,18 +155,77 @@ unsafe extern "C" fn cancel_job_cb(sched_job: *mut bindings::drm_sch // SAFETY: All of our jobs are Job. let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - // SAFETY: All of our jobs are Job. - T::cancel(unsafe { &mut *p }); + if !T::SYNCHRONIZED_CANCEL { + // SAFETY: All of our jobs are Job; fini has stopped the scheduler workers. + T::cancel(unsafe { &mut *addr_of_mut!((*p).inner) }); + + // SAFETY: s_fence is valid for an armed job. + let fence = unsafe { Fence::get_raw(&mut (*(*sched_job).s_fence).finished) }; + fence.set_error(ECANCELED); + let _ = fence.signal(); + return; + } - let fence = unsafe { Fence::get_raw(&mut (*(*sched_job).s_fence).finished) }; - fence.set_error(ECANCELED); - let _ = fence.signal(); + // fini has stopped/synchronized submission and timeout workers. A parent + // callback can still run on the device completion thread. The scheduler + // fence retains the parent reference while we remove its callback under + // the parent's lock; that also waits for an already-running callback. + // SAFETY: s_fence is valid for an armed job. + let s_fence = unsafe { (*sched_job).s_fence }; + // SAFETY: s_fence is valid; parent is read once under fini's exclusion. + let parent = unsafe { (*s_fence).parent }; + let removed_parent_callback = if !parent.is_null() { + // SAFETY: run_job installed (or attempted to install on an already + // signaled parent) this callback before fini synchronized submit work. + unsafe { bindings::dma_fence_remove_callback(parent, addr_of_mut!((*sched_job).cb)) } + } else { false }; + let finish_here = removed_parent_callback || (parent.is_null() + // SAFETY: the finished fence is valid for an armed job. + && unsafe { bindings::dma_fence_get_status(addr_of_mut!((*s_fence).finished)) == 0 }); + // Null/error run_job calls job_done before submit work returns, so its + // finished fence is already signaled. NULL+unfinished can instead survive + // a legacy backend's prior reset stop: C stop dropped the parent and its + // credits, but not its score. Do not subtract those credits twice. + + // SAFETY: cancellation cannot race a scheduler callback using this job + // after the removal/parent-lock synchronization above. The backend must + // retain any still-hardware-owned resources independently of Job storage. + T::cancel(unsafe { &mut *addr_of_mut!((*p).inner) }); + + if finish_here { + // Match drm_sched_job_done's accounting exactly once, only when its + // callback did not do it. Submission/free work is stopped, so fini + // itself unlinks and frees this job; no workqueue rearm is appropriate. + // SAFETY: the job's scheduler outlives fini's cancellation of its jobs. + let sched = unsafe { (*sched_job).sched }; + // SAFETY: credit_count and score belong to the live scheduler. + unsafe { + if removed_parent_callback { + bindings::atomic_sub((*sched_job).credits as i32, + addr_of_mut!((*sched).credit_count)); + } + bindings::atomic_sub(1, (*sched).score); + } + // SAFETY: the finished fence is valid for an armed job. + let fence = unsafe { Fence::get_raw(addr_of_mut!((*s_fence).finished)) }; + fence.set_error(ECANCELED); + let _ = fence.signal(); + } + // Do not clear/put parent here: s_fence still owns that reference and its + // normal release path drops it. A completed callback's result is retained. } /// A DRM scheduler job. pub struct Job { job: bindings::drm_sched_job, inner: T, + timeout_data: T::TimeoutData, + // Entity destruction can leave kill_jobs_work queued on the system + // workqueue or waiting for a dependency. That work still dereferences + // job->sched->ops after drm_sched_fini has stopped the scheduler workers. + // Retain only the allocation: retaining SchedulerOwner here would keep + // fini from running while pending jobs still need its cancellation. + _scheduler: Arc>, } impl Deref for Job { @@ -241,14 +339,15 @@ impl<'a> JobFences<'a> { struct EntityInner { entity: bindings::drm_sched_entity, // TODO: Allow users to share guilty flag between entities - sched: Arc>, + sched: Arc>, guilty: bindings::atomic_t, _p: PhantomData, } impl Drop for EntityInner { fn drop(&mut self) { - // SAFETY: The EntityInner is initialized. This will cancel/free all jobs. + // SAFETY: The EntityInner is initialized. Jobs removed from the entity + // may still await kill_jobs_work; each retains the scheduler allocation. unsafe { bindings::drm_sched_entity_destroy(&mut self.entity) }; } } @@ -267,7 +366,7 @@ impl Entity { let mut entity: KBox>> = KBox::new_uninit(GFP_KERNEL | __GFP_ZERO)?; - let mut sched_ptr = &sched.0.sched as *const _ as *mut _; + let mut sched_ptr = &sched.0.allocation.sched as *const _ as *mut _; // SAFETY: The Box is allocated above and valid. unsafe { @@ -329,6 +428,7 @@ impl Entity { // The exclusive entity borrow prevents allocation/push interleaving. fn prepare_job(&self, credits: u32, inner: T) -> Result> { let mut job: KBox>> = Box::new_uninit(GFP_KERNEL | __GFP_ZERO)?; + let timeout_data = inner.timeout_data(); // SAFETY: We hold a reference to the entity (which is a valid pointer), // and the job object was just allocated above. @@ -344,24 +444,49 @@ impl Entity { // SAFETY: The Box pointer is valid, and this initializes the inner member. unsafe { addr_of_mut!((*job.as_mut_ptr()).inner).write(inner) }; + // SAFETY: initialize the independent timeout worker's owned state. + unsafe { addr_of_mut!((*job.as_mut_ptr()).timeout_data).write(timeout_data) }; + + // SAFETY: The job allocation is valid and this initializes its last + // field. Arc::clone is infallible and the entity still owns the live + // scheduler, so every successfully initialized job retains its C + // callback target before it can be armed or submitted. + unsafe { + addr_of_mut!((*job.as_mut_ptr())._scheduler) + .write(self.0.as_ref().get_ref().sched.allocation.clone()) + }; // SAFETY: All fields of the Job are now initialized. Ok(PendingJob(unsafe { job.assume_init() }, PhantomData)) } } -/// DRM scheduler inner data +/// DRM scheduler allocation, retained by live owners and deferred job callbacks. pub struct SchedulerInner { sched: bindings::drm_gpu_scheduler, _p: PhantomData, } -impl Drop for SchedulerInner { +// Only public scheduler/entity handles keep this owner alive. Jobs retain +// SchedulerInner instead, so outstanding work cannot prevent finalization. +struct SchedulerOwner { + allocation: Arc>, +} + +impl Drop for SchedulerOwner { fn drop(&mut self) { - // SAFETY: The scheduler is valid. This assumes drm_sched_fini() will take care of - // freeing all in-progress jobs. - unsafe { bindings::drm_sched_stop(&mut self.sched, core::ptr::null_mut()) }; - unsafe { bindings::drm_sched_fini(&mut self.sched) }; + let sched = addr_of!(self.allocation.sched).cast_mut(); + if !T::SYNCHRONIZED_CANCEL { + // SAFETY: The scheduler is valid. This assumes drm_sched_fini() will take care of + // freeing all in-progress jobs. + unsafe { bindings::drm_sched_stop(sched, core::ptr::null_mut()) }; + } + // SAFETY: fini stops submission, synchronizes the timeout worker and + // cancels/frees remaining jobs through the cancel callback. With + // SYNCHRONIZED_CANCEL, the C contract prohibits drm_sched_stop before fini. + // Deferred entity-kill callbacks are on a different workqueue and may + // survive this call. Their job guards retain allocation through cleanup. + unsafe { bindings::drm_sched_fini(sched) }; } } @@ -371,7 +496,7 @@ unsafe impl Sync for SchedulerInner {} unsafe impl Send for SchedulerInner {} /// A DRM Scheduler -pub struct Scheduler(Arc>); +pub struct Scheduler(Arc>); impl Scheduler { const OPS: bindings::drm_sched_backend_ops = bindings::drm_sched_backend_ops { @@ -380,7 +505,10 @@ impl Scheduler { timedout_job: Some(timedout_job_cb::), free_job: Some(free_job_cb::), cancel_job: Some(cancel_job_cb::), - owner: core::ptr::null_mut(), + owner: match T::MODULE { + Some(module) => module.as_ptr(), + None => core::ptr::null_mut(), + }, retain_job_on_enodev: false, }; /// Creates a new DRM Scheduler object @@ -395,6 +523,10 @@ impl Scheduler { ) -> Result> { let mut sched: UniqueArc>> = UniqueArc::new_uninit(GFP_KERNEL)?; + // Allocate the owner before starting C workers: no fallible allocation + // may abandon an initialized scheduler without running drm_sched_fini. + let owner: UniqueArc>> = + UniqueArc::new_uninit(GFP_KERNEL)?; // SAFETY: zero sched->sched_rq as drm_sched_init() uses it to exit early withoput initialisation // TODO: allocate sched zzeroed instead @@ -424,7 +556,9 @@ impl Scheduler { ) })?; - // SAFETY: All fields of SchedulerInner are now initialized. - Ok(Scheduler(unsafe { sched.assume_init() }.into())) + // SAFETY: C initialized sched and PhantomData has no stored state. + // No fallible operation follows successful initialization. + let allocation = unsafe { sched.assume_init() }.into(); + Ok(Scheduler(owner.write(SchedulerOwner { allocation }).into())) } } diff --git a/rust/kernel/drm_neo/device.rs b/rust/kernel/drm_neo/device.rs deleted file mode 100644 index 8786b91f2aad6c..00000000000000 --- a/rust/kernel/drm_neo/device.rs +++ /dev/null @@ -1,338 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM device. -//! -//! C header: [`include/drm_neo/drm_device.h`](srctree/include/drm_neo/drm_device.h) - -use crate::{ - alloc::allocator::Kmalloc, - bindings, device, - drm_neo::{ - self, - driver::AllocImpl, // - }, - error::from_err_ptr, - prelude::*, - sync::aref::{ - ARef, - AlwaysRefCounted, // - }, - types::Opaque, - workqueue::{ - HasDelayedWork, - HasWork, - Work, - WorkItem, // - }, -}; -use core::{ - alloc::Layout, - cell::UnsafeCell, - mem::{self, MaybeUninit}, - ops::Deref, - ptr::{ - self, - NonNull, // - }, - sync::atomic::{AtomicBool, Ordering}, -}; - -#[cfg(CONFIG_DRM_LEGACY)] -macro_rules! drm_legacy_fields { - ( $($field:ident: $val:expr),* $(,)? ) => { - bindings::drm_driver { - $( $field: $val ),*, - firstopen: None, - preclose: None, - dma_ioctl: None, - dma_quiescent: None, - context_dtor: None, - irq_handler: None, - irq_preinstall: None, - irq_postinstall: None, - irq_uninstall: None, - get_vblank_counter: None, - enable_vblank: None, - disable_vblank: None, - dev_priv_size: 0, - } - } -} - -#[cfg(not(CONFIG_DRM_LEGACY))] -macro_rules! drm_legacy_fields { - ( $($field:ident: $val:expr),* $(,)? ) => { - bindings::drm_driver { - $( $field: $val ),* - } - } -} - -/// A typed DRM device with a specific `drm_neo::Driver` implementation. -/// -/// The device is always reference-counted. -/// -/// # Invariants -/// -/// `self.dev` is a valid instance of a `struct device`. -/// When `data_initialized` is true, `data` owns initialized, pinned driver data. -/// Otherwise, driver data must not be accessed. -#[repr(C)] -pub struct Device { - dev: Opaque, - data: UnsafeCell>, - data_initialized: AtomicBool, -} - -impl Device { - const VTABLE: bindings::drm_driver = drm_legacy_fields! { - load: None, - open: Some(drm_neo::File::::open_callback), - postclose: Some(drm_neo::File::::postclose_callback), - unload: None, - release: Some(Self::release), - master_set: None, - master_drop: None, - debugfs_init: None, - gem_create_object: T::Object::ALLOC_OPS.gem_create_object, - prime_handle_to_fd: T::Object::ALLOC_OPS.prime_handle_to_fd, - prime_fd_to_handle: T::Object::ALLOC_OPS.prime_fd_to_handle, - gem_prime_import: T::Object::ALLOC_OPS.gem_prime_import, - gem_prime_import_sg_table: T::Object::ALLOC_OPS.gem_prime_import_sg_table, - dumb_create: T::Object::ALLOC_OPS.dumb_create, - dumb_map_offset: T::Object::ALLOC_OPS.dumb_map_offset, - show_fdinfo: None, - fbdev_probe: None, - - major: T::INFO.major, - minor: T::INFO.minor, - patchlevel: T::INFO.patchlevel, - name: crate::str::as_char_ptr_in_const_context(T::INFO.name).cast_mut(), - desc: crate::str::as_char_ptr_in_const_context(T::INFO.desc).cast_mut(), - - driver_features: T::FEATURES, - ioctls: T::IOCTLS.as_ptr(), - num_ioctls: T::IOCTLS.len() as i32, - fops: &Self::GEM_FOPS, - }; - - const ALLOC_VTABLE: bindings::drm_driver = bindings::drm_driver { - release: None, - ..Self::VTABLE - }; - - const GEM_FOPS: bindings::file_operations = drm_neo::gem::create_fops(T::MODULE); - - /// Create a new `drm_neo::Device` for a `drm_neo::Driver`. - pub fn new(dev: &device::Device, data: impl PinInit) -> Result> { - // SAFETY: The device remains private until its data is initialized below. - let drm_neo = unsafe { Self::new_uninit(dev)? }; - // SAFETY: This is the only initializer, and no references to the data exist yet. - unsafe { drm_neo.init_data(data)? }; - Ok(drm_neo) - } - - /// Allocate a DRM device whose driver data will be initialized separately. - /// - /// # Safety - /// - /// The caller must not access driver data, publish the device to userspace, or - /// schedule work that accesses driver data before calling [`Self::init_data`]. - pub unsafe fn new_uninit(dev: &device::Device) -> Result> { - let layout = Kmalloc::aligned_layout(Layout::new::()); - // Allocation failure must not inspect fields that we have not initialized yet. - let raw_drm: *mut Self = unsafe { - bindings::__drm_dev_alloc( - dev.as_raw(), - const { &Self::ALLOC_VTABLE }, - layout.size(), - mem::offset_of!(Self, dev), - ) - } - .cast(); - let raw_drm = NonNull::new(from_err_ptr(raw_drm)?).ok_or(ENOMEM)?; - // SAFETY: Allocation succeeded, and these fields have no previous value. - unsafe { - ptr::addr_of_mut!((*raw_drm.as_ptr()).data) - .write(UnsafeCell::new(MaybeUninit::uninit())); - ptr::addr_of_mut!((*raw_drm.as_ptr()).data_initialized).write(AtomicBool::new(false)); - } - // SAFETY: The allocation is private and its release state is now initialized. - let drm_dev = unsafe { Self::into_drm_device(raw_drm) }; - unsafe { (*drm_dev).driver = const { &Self::VTABLE } }; - // SAFETY: We own the initial DRM reference; uninitialized data is behind UnsafeCell. - Ok(unsafe { ARef::from_raw(raw_drm) }) - } - - /// Initialize the pinned driver data of a device allocated by [`Self::new_uninit`]. - /// - /// # Safety - /// - /// The caller must be the only initializer, the data must be uninitialized, and - /// no driver-data users may run until this method returns successfully. - pub unsafe fn init_data(&self, data: impl PinInit) -> Result { - let raw_data = self.data.get().cast::(); - // SAFETY: The caller guarantees exclusive initialization of this pinned slot. - unsafe { data.__pinned_init(raw_data) }?; - // Publish destructor ownership only after the real initializer succeeded. - self.data_initialized.store(true, Ordering::Release); - Ok(()) - } - - pub(crate) fn as_raw(&self) -> *mut bindings::drm_device { - self.dev.get() - } - - /// # Safety - /// - /// `ptr` must be a valid pointer to a `struct device` embedded in `Self`. - unsafe fn from_drm_device(ptr: *const bindings::drm_device) -> *mut Self { - // SAFETY: By the safety requirements of this function `ptr` is a valid pointer to a - // `struct drm_device` embedded in `Self`. - unsafe { crate::container_of!(Opaque::cast_from(ptr), Self, dev) }.cast_mut() - } - - /// # Safety - /// - /// `ptr` must be a valid pointer to `Self`. - unsafe fn into_drm_device(ptr: NonNull) -> *mut bindings::drm_device { - // SAFETY: By the safety requirements of this function, `ptr` is a valid pointer to `Self`. - unsafe { &raw mut (*ptr.as_ptr()).dev }.cast() - } - - /// Not intended to be called externally, except via declare_drm_neo_ioctls!() - /// - /// # Safety - /// - /// Callers must ensure that `ptr` is valid, non-null, and has a non-zero reference count, - /// i.e. it must be ensured that the reference count of the C `struct drm_device` `ptr` points - /// to can't drop to zero, for the duration of this function call and the entire duration when - /// the returned reference exists. - /// - /// Additionally, callers must ensure that the `struct device`, `ptr` is pointing to, is - /// embedded in `Self`. - #[doc(hidden)] - pub unsafe fn from_raw<'a>(ptr: *const bindings::drm_device) -> &'a Self { - // SAFETY: By the safety requirements of this function `ptr` is a valid pointer to a - // `struct drm_device` embedded in `Self`. - let ptr = unsafe { Self::from_drm_device(ptr) }; - - // SAFETY: `ptr` is valid by the safety requirements of this function. - unsafe { &*ptr.cast() } - } - - extern "C" fn release(ptr: *mut bindings::drm_device) { - // SAFETY: `ptr` is a valid pointer to a `struct drm_device` and embedded in `Self`. - let this = unsafe { Self::from_drm_device(ptr) }; - - // SAFETY: new_uninit() initialized this flag before enabling this callback. - // A failed initializer unwinds its own partial fields and never sets the flag. - if unsafe { (*this).data_initialized.load(Ordering::Acquire) } { - // SAFETY: Successful initialization published ownership of pinned driver data. - // DRM still owns the embedded C device and its final managed cleanup/kfree. - unsafe { ptr::drop_in_place((*this).data.get().cast::()) }; - } - } -} - -impl Deref for Device { - type Target = T::Data; - - fn deref(&self) -> &Self::Target { - debug_assert!(self.data_initialized.load(Ordering::Acquire)); - // SAFETY: Safe constructors initialize data; new_uninit() requires callers - // to prevent data access until init_data() succeeds. - unsafe { &*self.data.get().cast::() } - } -} - -// SAFETY: DRM device objects are always reference counted and the get/put functions -// satisfy the requirements. -unsafe impl AlwaysRefCounted for Device { - fn inc_ref(&self) { - // SAFETY: The existence of a shared reference guarantees that the refcount is non-zero. - unsafe { bindings::drm_dev_get(self.as_raw()) }; - } - - unsafe fn dec_ref(obj: NonNull) { - // SAFETY: `obj` is a valid pointer to `Self`. - let drm_dev = unsafe { Self::into_drm_device(obj) }; - - // SAFETY: The safety requirements guarantee that the refcount is non-zero. - unsafe { bindings::drm_dev_put(drm_dev) }; - } -} - -impl AsRef for Device { - fn as_ref(&self) -> &device::Device { - // SAFETY: `bindings::drm_device::dev` is valid as long as the DRM device itself is valid, - // which is guaranteed by the type invariant. - unsafe { device::Device::from_raw((*self.as_raw()).dev) } - } -} - -// SAFETY: A `drm_neo::Device` can be released from any thread. -unsafe impl Send for Device {} - -// SAFETY: A `drm_neo::Device` can be shared among threads because all immutable methods are protected -// by the synchronization in `struct drm_device`. -unsafe impl Sync for Device {} - -impl WorkItem for Device -where - T: drm_neo::Driver, - T::Data: WorkItem>>, - T::Data: HasWork, ID>, -{ - type Pointer = ARef>; - - fn run(ptr: ARef>) { - T::Data::run(ptr); - } -} - -// SAFETY: -// -// - `raw_get_work` and `work_container_of` return valid pointers by relying on -// `T::Data::raw_get_work` and `container_of`. In particular, `T::Data` is -// stored inline in `drm_neo::Device`, so the `container_of` call is valid. -// -// - The two methods are true inverses of each other: given `ptr: *mut -// Device`, `raw_get_work` will return a `*mut Work, ID>` through -// `T::Data::raw_get_work` and given a `ptr: *mut Work, ID>`, -// `work_container_of` will return a `*mut Device` through `container_of`. -unsafe impl HasWork, ID> for Device -where - T: drm_neo::Driver, - T::Data: HasWork, ID>, -{ - unsafe fn raw_get_work(ptr: *mut Self) -> *mut Work, ID> { - // SAFETY: The caller promises that `ptr` points to a valid `Device`. - let data_ptr = unsafe { (&raw mut (*ptr).data).cast::() }; - - // SAFETY: `data_ptr` is a valid pointer to `T::Data`. - unsafe { T::Data::raw_get_work(data_ptr) } - } - - unsafe fn work_container_of(ptr: *mut Work, ID>) -> *mut Self { - // SAFETY: The caller promises that `ptr` points at a `Work` field in - // `T::Data`. - let data_ptr = unsafe { T::Data::work_container_of(ptr) } - .cast::>>(); - - // SAFETY: `T::Data` is stored as the `data` field in `Device`. - unsafe { crate::container_of!(data_ptr, Self, data) } - } -} - -// SAFETY: Our `HasWork` implementation returns a `work_struct` that is -// stored in the `work` field of a `delayed_work` with the same access rules as -// the `work_struct` owing to the bound on `T::Data: HasDelayedWork, -// ID>`, which requires that `T::Data::raw_get_work` return a `work_struct` that -// is inside a `delayed_work`. -unsafe impl HasDelayedWork, ID> for Device -where - T: drm_neo::Driver, - T::Data: HasDelayedWork, ID>, -{ -} diff --git a/rust/kernel/drm_neo/driver.rs b/rust/kernel/drm_neo/driver.rs deleted file mode 100644 index 1e0838f78ea7d9..00000000000000 --- a/rust/kernel/drm_neo/driver.rs +++ /dev/null @@ -1,188 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM driver core. -//! -//! C header: [`include/drm_neo/drm_drv.h`](srctree/include/drm_neo/drm_drv.h) - -use crate::{ - bindings, - device, - devres, - drm_neo, - error::to_result, - prelude::*, - sync::aref::ARef, // -}; - -/// Driver use the GEM memory manager. This should be set for all modern drivers. -pub const FEAT_GEM: u32 = bindings::drm_driver_feature_DRIVER_GEM; -/// Driver supports dedicated render nodes. -pub const FEAT_RENDER: u32 = bindings::drm_driver_feature_DRIVER_RENDER; -/// Driver supports DRM sync objects for explicit synchronization of command submission. -pub const FEAT_SYNCOBJ: u32 = bindings::drm_driver_feature_DRIVER_SYNCOBJ; -/// Driver supports the timeline flavor of DRM sync objects for explicit synchronization of command -/// submission. -pub const FEAT_SYNCOBJ_TIMELINE: u32 = bindings::drm_driver_feature_DRIVER_SYNCOBJ_TIMELINE; -/// Driver supports user defined GPU VA bindings for GEM objects. -pub const FEAT_GEM_GPUVA: u32 = bindings::drm_driver_feature_DRIVER_GEM_GPUVA; - -/// Information data for a DRM Driver. -pub struct DriverInfo { - /// Driver major version. - pub major: i32, - /// Driver minor version. - pub minor: i32, - /// Driver patchlevel version. - pub patchlevel: i32, - /// Driver name. - pub name: &'static CStr, - /// Driver description. - pub desc: &'static CStr, -} - -/// Internal memory management operation set, normally created by memory managers (e.g. GEM). -pub struct AllocOps { - pub(crate) gem_create_object: Option< - unsafe extern "C" fn( - dev: *mut bindings::drm_device, - size: usize, - ) -> *mut bindings::drm_gem_object, - >, - pub(crate) prime_handle_to_fd: Option< - unsafe extern "C" fn( - dev: *mut bindings::drm_device, - file_priv: *mut bindings::drm_file, - handle: u32, - flags: u32, - prime_fd: *mut core::ffi::c_int, - ) -> core::ffi::c_int, - >, - pub(crate) prime_fd_to_handle: Option< - unsafe extern "C" fn( - dev: *mut bindings::drm_device, - file_priv: *mut bindings::drm_file, - prime_fd: core::ffi::c_int, - handle: *mut u32, - ) -> core::ffi::c_int, - >, - pub(crate) gem_prime_import: Option< - unsafe extern "C" fn( - dev: *mut bindings::drm_device, - dma_buf: *mut bindings::dma_buf, - ) -> *mut bindings::drm_gem_object, - >, - pub(crate) gem_prime_import_sg_table: Option< - unsafe extern "C" fn( - dev: *mut bindings::drm_device, - attach: *mut bindings::dma_buf_attachment, - sgt: *mut bindings::sg_table, - ) -> *mut bindings::drm_gem_object, - >, - pub(crate) dumb_create: Option< - unsafe extern "C" fn( - file_priv: *mut bindings::drm_file, - dev: *mut bindings::drm_device, - args: *mut bindings::drm_mode_create_dumb, - ) -> core::ffi::c_int, - >, - pub(crate) dumb_map_offset: Option< - unsafe extern "C" fn( - file_priv: *mut bindings::drm_file, - dev: *mut bindings::drm_device, - handle: u32, - offset: *mut u64, - ) -> core::ffi::c_int, - >, -} - -/// Trait for memory manager implementations. Implemented internally. -pub trait AllocImpl: super::private::Sealed + drm_neo::gem::IntoGEMObject { - /// The [`Driver`] implementation for this [`AllocImpl`]. - type Driver: drm_neo::Driver; - - /// The C callback operations for this memory manager. - const ALLOC_OPS: AllocOps; -} - -/// The DRM `Driver` trait. -/// -/// This trait must be implemented by drivers in order to create a `struct drm_device` and `struct -/// drm_driver` to be registered in the DRM subsystem. -#[vtable] -pub trait Driver { - /// Context data associated with the DRM driver - type Data: Sync + Send; - - /// The type used to manage memory for this driver. - type Object: AllocImpl; - - /// The type used to represent a DRM File (client) - type File: drm_neo::file::DriverFile; - - /// Driver metadata - const INFO: DriverInfo; - - /// Module implementing the driver callbacks. Loadable drivers should provide - /// their `THIS_MODULE` so open files and exported DMA buffers retain the code. - const MODULE: Option<&'static crate::ThisModule> = None; - - /// Feature flags - const FEATURES: u32; - - /// IOCTL list. See `kernel::drm_neo::ioctl::declare_drm_neo_ioctls!{}`. - const IOCTLS: &'static [drm_neo::ioctl::DrmIoctlDescriptor]; -} - -/// The registration type of a `drm_neo::Device`. -/// -/// Once the `Registration` structure is dropped, the device is unregistered. -pub struct Registration(ARef>); - -impl Registration { - fn new(drm_neo: &drm_neo::Device, flags: usize) -> Result { - // SAFETY: `drm_neo.as_raw()` is valid by the invariants of `drm_neo::Device`. - to_result(unsafe { bindings::drm_dev_register(drm_neo.as_raw(), flags) })?; - - Ok(Self(drm_neo.into())) - } - - /// Registers a new [`Device`](drm_neo::Device) with userspace. - /// - /// Ownership of the [`Registration`] object is passed to [`devres::register`]. - pub fn new_foreign_owned( - drm_neo: &drm_neo::Device, - dev: &device::Device, - flags: usize, - ) -> Result - where - T: 'static, - { - if drm_neo.as_ref().as_raw() != dev.as_raw() { - return Err(EINVAL); - } - - let reg = Registration::::new(drm_neo, flags)?; - - devres::register(dev, reg, GFP_KERNEL) - } - - /// Returns a reference to the `Device` instance for this registration. - pub fn device(&self) -> &drm_neo::Device { - &self.0 - } -} - -// SAFETY: `Registration` doesn't offer any methods or access to fields when shared between -// threads, hence it's safe to share it. -unsafe impl Sync for Registration {} - -// SAFETY: Registration with and unregistration from the DRM subsystem can happen from any thread. -unsafe impl Send for Registration {} - -impl Drop for Registration { - fn drop(&mut self) { - // SAFETY: Safe by the invariant of `ARef>`. The existence of this - // `Registration` also guarantees the this `drm_neo::Device` is actually registered. - unsafe { bindings::drm_dev_unregister(self.0.as_raw()) }; - } -} diff --git a/rust/kernel/drm_neo/file.rs b/rust/kernel/drm_neo/file.rs deleted file mode 100644 index 9bb81f21556f0b..00000000000000 --- a/rust/kernel/drm_neo/file.rs +++ /dev/null @@ -1,114 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM File objects. -//! -//! C header: [`include/drm_neo/drm_file.h`](srctree/include/drm_neo/drm_file.h) - -use crate::{ - bindings, - drm_neo, - prelude::*, - types::Opaque, // -}; -use core::marker::PhantomData; - -/// Trait that must be implemented by DRM drivers to represent a DRM File (a client instance). -pub trait DriverFile { - /// The parent `Driver` implementation for this `DriverFile`. - type Driver: drm_neo::Driver; - - /// Open a new file (called when a client opens the DRM device). - fn open(device: &drm_neo::Device) -> Result>>; - - /// Get raw drm_file pointer - fn as_raw(&self) -> *mut bindings::drm_file; -} - -/// An open DRM File. -/// -/// # Invariants -/// -/// `self.0` is a valid instance of a `struct drm_file`. -#[repr(transparent)] -pub struct File(Opaque, PhantomData); - -impl File { - #[doc(hidden)] - /// Not intended to be called externally, except via declare_drm_neo_ioctls!() - /// - /// # Safety - /// - /// `raw_file` must be a valid pointer to an open `struct drm_file`, opened through `T::open`. - pub unsafe fn from_raw<'a>(ptr: *mut bindings::drm_file) -> &'a File { - // SAFETY: `raw_file` is valid by the safety requirements of this function. - unsafe { &*ptr.cast() } - } - - pub(super) fn as_raw(&self) -> *mut bindings::drm_file { - self.0.get() - } - - /// Tag recent feeder work for temporary capacity hints during syncobj waits. - /// User scheduling attributes and CPU affinity remain unchanged. - pub fn set_syncobj_wait_hint(&self, util_min: u32, duration_ms: u32) { - // SAFETY: this File keeps the opened drm_file alive; the C helper - // bounds both arguments and serializes the per-file hint internally. - unsafe { bindings::drm_syncobj_set_wait_hint(self.as_raw(), util_min, duration_ms) }; - } - - fn driver_priv(&self) -> *mut T { - // SAFETY: By the type invariants of `Self`, `self.as_raw()` is always valid. - unsafe { (*self.as_raw()).driver_priv }.cast() - } - - /// Return a pinned reference to the driver file structure. - pub fn inner(&self) -> Pin<&T> { - // SAFETY: By the type invariant the pointer `self.as_raw()` points to a valid and opened - // `struct drm_file`, hence `driver_priv` has been properly initialized by `open_callback`. - unsafe { Pin::new_unchecked(&*(self.driver_priv())) } - } - - /// The open callback of a `struct drm_file`. - pub(crate) extern "C" fn open_callback( - raw_dev: *mut bindings::drm_device, - raw_file: *mut bindings::drm_file, - ) -> core::ffi::c_int { - // SAFETY: A callback from `struct drm_driver::open` guarantees that - // - `raw_dev` is valid pointer to a `struct drm_device`, - // - the corresponding `struct drm_device` has been registered. - let drm_neo = unsafe { drm_neo::Device::from_raw(raw_dev) }; - - // SAFETY: `raw_file` is a valid pointer to a `struct drm_file`. - let file = unsafe { File::::from_raw(raw_file) }; - - let inner = match T::open(drm_neo) { - Err(e) => { - return e.to_errno(); - } - Ok(i) => i, - }; - - // SAFETY: This pointer is treated as pinned, and the Drop guarantee is upheld in - // `postclose_callback()`. - let driver_priv = KBox::into_raw(unsafe { Pin::into_inner_unchecked(inner) }); - - // SAFETY: By the type invariants of `Self`, `self.as_raw()` is always valid. - unsafe { (*file.as_raw()).driver_priv = driver_priv.cast() }; - - 0 - } - - /// The postclose callback of a `struct drm_file`. - pub(crate) extern "C" fn postclose_callback( - _raw_dev: *mut bindings::drm_device, - raw_file: *mut bindings::drm_file, - ) { - // SAFETY: This reference won't escape this function - let file = unsafe { File::::from_raw(raw_file) }; - - // SAFETY: `file.driver_priv` has been created in `open_callback` through `KBox::into_raw`. - let _ = unsafe { KBox::from_raw(file.driver_priv()) }; - } -} - -impl super::private::Sealed for File {} diff --git a/rust/kernel/drm_neo/gem/mod.rs b/rust/kernel/drm_neo/gem/mod.rs deleted file mode 100644 index bf986fe3ece6f0..00000000000000 --- a/rust/kernel/drm_neo/gem/mod.rs +++ /dev/null @@ -1,497 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM GEM API -//! -//! C header: [`include/drm_neo/drm_gem.h`](srctree/include/drm_neo/drm_gem.h) - -use crate::{ - bindings, - dma_buf, - drm_neo::{ - self, - driver::{ - AllocImpl, - AllocOps, // - }, - }, - error::{ - from_err_ptr, - to_result, // - }, - prelude::*, - sync::aref::{ - ARef, - AlwaysRefCounted, // - }, - types::Opaque, // -}; -use core::{ - marker::PhantomData, - ops::Deref, - ptr::NonNull, // -}; - -#[cfg(CONFIG_RUST_DRM_GEM_SHMEM_HELPER)] -pub mod shmem; - -/// A macro for implementing [`AlwaysRefCounted`] for any GEM object type. -/// -/// Since all GEM objects use the same refcounting scheme. -#[macro_export] -macro_rules! impl_aref_for_neo_gem_obj { - ( - impl $( <$( $tparam_id:ident ),+> )? for $type:ty - $( - where - $( $bind_param:path : $bind_trait:path ),+ - )? - ) => { - // SAFETY: All GEM objects are refcounted. - unsafe impl $( <$( $tparam_id ),+> )? $crate::sync::aref::AlwaysRefCounted for $type - where - Self: IntoGEMObject, - $( $( $bind_param : $bind_trait ),+ )? - { - fn inc_ref(&self) { - // SAFETY: The existence of a shared reference guarantees that the refcount is - // non-zero. - unsafe { bindings::drm_gem_object_get(self.as_raw()) }; - } - - unsafe fn dec_ref(obj: core::ptr::NonNull) { - // SAFETY: `obj` is a valid pointer to an `Object`. - let obj = unsafe { obj.as_ref() }.as_raw(); - - // SAFETY: The safety requirements guarantee that the refcount is non-zero. - unsafe { bindings::drm_gem_object_put(obj) }; - } - } - }; -} -#[cfg_attr(not(CONFIG_RUST_DRM_GEM_SHMEM_HELPER), allow(unused))] -pub(crate) use impl_aref_for_neo_gem_obj; - -/// A type alias for retrieving a [`Driver`]s [`DriverFile`] implementation from its -/// [`DriverObject`] implementation. -/// -/// [`Driver`]: drm_neo::Driver -/// [`DriverFile`]: drm_neo::file::DriverFile -pub type DriverFile = drm_neo::File<<::Driver as drm_neo::Driver>::File>; - -/// GEM object functions, which must be implemented by drivers. -#[vtable] -pub trait DriverObject: Sync + Send + Sized { - /// Parent `Driver` for this object. - type Driver: drm_neo::Driver; - - /// The data type to use for passing arguments to [`DriverObject::new`]. - type Args; - - /// Create a new driver data object for a GEM object of a given size. - fn new( - dev: &drm_neo::Device, - size: usize, - args: Self::Args, - ) -> impl PinInit; - - /// Open a new handle to an existing object, associated with a File. - fn open(_obj: &::Object, _file: &DriverFile) -> Result { - Ok(()) - } - - /// Close a handle to an existing object, associated with a File. - fn close(_obj: &::Object, _file: &DriverFile) {} - - /// Optional handle for exporting a gem object. - fn export( - _obj: &::Object, - _flags: u32, - ) -> Result::Object>> { - unimplemented!() - } -} - -/// Trait that represents a GEM object subtype -pub trait IntoGEMObject: Sized + super::private::Sealed + AlwaysRefCounted { - /// Returns a reference to the raw `drm_gem_object` structure, which must be valid as long as - /// this owning object is valid. - fn as_raw(&self) -> *mut bindings::drm_gem_object; - - /// Converts a pointer to a `struct drm_gem_object` into a reference to `Self`. - /// - /// # Safety - /// - /// - `self_ptr` must be a valid pointer to `Self`. - /// - The caller promises that holding the immutable reference returned by this function does - /// not violate rust's data aliasing rules and remains valid throughout the lifetime of `'a`. - unsafe fn from_raw<'a>(self_ptr: *mut bindings::drm_gem_object) -> &'a Self; -} - -extern "C" fn open_callback( - raw_obj: *mut bindings::drm_gem_object, - raw_file: *mut bindings::drm_file, -) -> core::ffi::c_int { - // SAFETY: `open_callback` is only ever called with a valid pointer to a `struct drm_file`. - let file = unsafe { DriverFile::::from_raw(raw_file) }; - - // SAFETY: `open_callback` is specified in the AllocOps structure for `DriverObject`, - // ensuring that `raw_obj` is contained within a `DriverObject` - let obj = unsafe { <::Object as IntoGEMObject>::from_raw(raw_obj) }; - - match T::open(obj, file) { - Err(e) => e.to_errno(), - Ok(()) => 0, - } -} - -extern "C" fn close_callback( - raw_obj: *mut bindings::drm_gem_object, - raw_file: *mut bindings::drm_file, -) { - // SAFETY: `open_callback` is only ever called with a valid pointer to a `struct drm_file`. - let file = unsafe { DriverFile::::from_raw(raw_file) }; - - // SAFETY: `close_callback` is specified in the AllocOps structure for `Object`, ensuring - // that `raw_obj` is indeed contained within a `Object`. - let obj = unsafe { <::Object as IntoGEMObject>::from_raw(raw_obj) }; - - T::close(obj, file); -} - -extern "C" fn export_callback( - raw_obj: *mut bindings::drm_gem_object, - flags: i32, -) -> *mut bindings::dma_buf { - // SAFETY: `export_callback` is specified in the AllocOps structure for `Object`, ensuring - // that `raw_obj` is contained within a `Object`. - let obj = unsafe { <::Object as IntoGEMObject>::from_raw(raw_obj) }; - - match T::export(obj, flags as _) { - // DRM takes a hold of the reference - Ok(buf) => buf.into_raw(), - Err(e) => e.to_ptr(), - } -} - -impl IntoGEMObject for Object { - fn as_raw(&self) -> *mut bindings::drm_gem_object { - self.obj.get() - } - - unsafe fn from_raw<'a>(self_ptr: *mut bindings::drm_gem_object) -> &'a Self { - // SAFETY: `obj` is guaranteed to be in an `Object` via the safety contract of this - // function - unsafe { &*crate::container_of!(Opaque::cast_from(self_ptr), Object, obj) } - } -} - -/// Base operations shared by all GEM object classes -pub trait BaseObject: IntoGEMObject { - /// Returns the size of the object in bytes. - fn size(&self) -> usize { - // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `struct drm_gem_object`. - unsafe { (*self.as_raw()).size } - } - - /// Creates a new handle for the object associated with a given `File` - /// (or returns an existing one). - fn create_handle(&self, file: &drm_neo::File) -> Result - where - Self: AllocImpl, - D: drm_neo::Driver, - F: drm_neo::file::DriverFile, - { - let mut handle: u32 = 0; - // SAFETY: The arguments are all valid per the type invariants. - to_result(unsafe { - bindings::drm_gem_handle_create(file.as_raw().cast(), self.as_raw(), &mut handle) - })?; - Ok(handle) - } - - /// Looks up an object by its handle for a given `File`. - fn lookup_handle(file: &drm_neo::File, handle: u32) -> Result> - where - Self: AllocImpl, - D: drm_neo::Driver, - F: drm_neo::file::DriverFile, - { - // SAFETY: The arguments are all valid per the type invariants. - let ptr = unsafe { bindings::drm_gem_object_lookup(file.as_raw().cast(), handle) }; - if ptr.is_null() { - return Err(ENOENT); - } - - // SAFETY: - // - A `drm_neo::Driver` can only have a single `File` implementation. - // - `file` uses the same `drm_neo::Driver` as `Self`. - // - Therefore, we're guaranteed that `ptr` must be a gem object embedded within `Self`. - // - And we check if the pointer is null befoe calling from_raw(), ensuring that `ptr` is a - // valid pointer to an initialized `Self`. - let obj = unsafe { Self::from_raw(ptr) }; - - // SAFETY: - // - We take ownership of the reference of `drm_gem_object_lookup()`. - // - Our `NonNull` comes from an immutable reference, thus ensuring it is a valid pointer to - // `Self`. - Ok(unsafe { ARef::from_raw(obj.into()) }) - } - - /// Export a [`DmaBuf`] for this GEM object using the DRM prime helper library. - /// - /// `flags` should be a set of flags from [`fs::file::flags`](kernel::fs::file::flags). - fn prime_export(&self, flags: u32) -> Result> { - // SAFETY: - // - `as_raw()` always returns a valid pointer to a `drm_gem_object`. - // - `drm_gem_prime_export()` returns either an error pointer, or a valid pointer to an - // initialized `dma_buf` on success. - let dma_ptr = - from_err_ptr(unsafe { bindings::drm_gem_prime_export(self.as_raw(), flags as _) })?; - - // SAFETY: - // - We checked that dma_ptr is not an error, so it must point to an initialized dma_buf - // - We used drm_gem_prime_export(), so `dma_ptr` will remain valid until a call to - // `drm_gem_prime_release()` which we don't call here. - let dma_buf = unsafe { dma_buf::DmaBuf::as_ref(dma_ptr) }; - - // INVARIANT: We used drm_gem_prime_export() to create this dma_buf, fulfilling the - // invariant that this dma_buf came from a GEM object of type `Self`. - Ok(DmaBuf(dma_buf.into(), PhantomData)) - } - - /// Creates an mmap offset to map the object from userspace. - fn create_mmap_offset(&self) -> Result { - // SAFETY: The arguments are valid per the type invariant. - to_result(unsafe { bindings::drm_gem_create_mmap_offset(self.as_raw()) })?; - - // SAFETY: The arguments are valid per the type invariant. - Ok(unsafe { bindings::drm_vma_node_offset_addr(&raw mut (*self.as_raw()).vma_node) }) - } - - /// Lock the gpuva lock - fn lock_gpuva(&self) { - unsafe { - bindings::mutex_lock(&raw mut (*self.as_raw()).gpuva.lock); - } - } - - /// Lock the gpuva lock - fn unlock_gpuva(&self) { - unsafe { - bindings::mutex_unlock(&raw mut (*self.as_raw()).gpuva.lock); - } - } -} - -impl BaseObject for T {} - -/// Crate-private base operations shared by all GEM object classes. -#[cfg_attr(not(CONFIG_RUST_DRM_GEM_SHMEM_HELPER), expect(unused))] -pub(crate) trait BaseObjectPrivate: IntoGEMObject { - /// Return a pointer to this object's dma_resv. - fn raw_dma_resv(&self) -> *mut bindings::dma_resv { - // SAFETY: `self.as_raw()` always returns a valid pointer to the base DRM GEM object. - unsafe { (*self.as_raw()).resv } - } -} - -impl BaseObjectPrivate for T {} - -/// A base GEM object. -/// -/// # Invariants -/// -/// - `self.obj` is a valid instance of a `struct drm_gem_object`. -#[repr(C)] -#[pin_data] -pub struct Object { - obj: Opaque, - #[pin] - data: T, -} - -impl Object { - const OBJECT_FUNCS: bindings::drm_gem_object_funcs = bindings::drm_gem_object_funcs { - free: Some(Self::free_callback), - open: Some(open_callback::), - close: Some(close_callback::), - print_info: None, - export: if T::HAS_EXPORT { - Some(export_callback::) - } else { - None - }, - pin: None, - unpin: None, - get_sg_table: None, - vmap: None, - vunmap: None, - mmap: None, - status: None, - vm_ops: core::ptr::null_mut(), - evict: None, - rss: None, - }; - - /// Create a new GEM object. - pub fn new(dev: &drm_neo::Device, size: usize, args: T::Args) -> Result> { - let obj: Pin> = KBox::pin_init( - try_pin_init!(Self { - obj: Opaque::new(bindings::drm_gem_object::default()), - data <- T::new(dev, size, args), - }), - GFP_KERNEL, - )?; - - // SAFETY: `obj.as_raw()` is guaranteed to be valid by the initialization above. - unsafe { (*obj.as_raw()).funcs = &Self::OBJECT_FUNCS }; - - if let Err(err) = - // SAFETY: The arguments are all valid per the type invariants. - to_result(unsafe { - bindings::drm_gem_object_init(dev.as_raw(), obj.obj.get(), size) - }) - { - // SAFETY: `drm_gem_object_init()` initializes the private GEM object state before - // failing, so `drm_gem_private_object_fini()` is the matching cleanup. - unsafe { bindings::drm_gem_private_object_fini(obj.obj.get()) }; - return Err(err); - } - - // SAFETY: We will never move out of `Self` as `ARef` is always treated as pinned. - let ptr = KBox::into_raw(unsafe { Pin::into_inner_unchecked(obj) }); - - // SAFETY: `ptr` comes from `KBox::into_raw` and hence can't be NULL. - let ptr = unsafe { NonNull::new_unchecked(ptr) }; - - // SAFETY: We take over the initial reference count from `drm_gem_object_init()`. - Ok(unsafe { ARef::from_raw(ptr) }) - } - - /// Returns the `Device` that owns this GEM object. - pub fn dev(&self) -> &drm_neo::Device { - // SAFETY: - // - `struct drm_gem_object.dev` is initialized and valid for as long as the GEM - // object lives. - // - The device we used for creating the gem object is passed as &drm_neo::Device to - // Object::::new(), so we know that `T::Driver` is the right generic parameter to use - // here. - unsafe { drm_neo::Device::from_raw((*self.as_raw()).dev) } - } - - fn as_raw(&self) -> *mut bindings::drm_gem_object { - self.obj.get() - } - - extern "C" fn free_callback(obj: *mut bindings::drm_gem_object) { - let ptr: *mut Opaque = obj.cast(); - - // SAFETY: All of our objects are of type `Object`. - let this = unsafe { crate::container_of!(ptr, Self, obj) }; - - // SAFETY: The C code only ever calls this callback with a valid pointer to a `struct - // drm_gem_object`. - unsafe { bindings::drm_gem_object_release(obj) }; - - // SAFETY: All of our objects are allocated via `KBox`, and we're in the - // free callback which guarantees this object has zero remaining references, - // so we can drop it. - let _ = unsafe { KBox::from_raw(this) }; - } -} - -impl_aref_for_neo_gem_obj!(impl for Object where T: DriverObject); - -impl super::private::Sealed for Object {} - -impl Deref for Object { - type Target = T; - - fn deref(&self) -> &Self::Target { - &self.data - } -} - -impl AllocImpl for Object { - type Driver = T::Driver; - - const ALLOC_OPS: AllocOps = AllocOps { - gem_create_object: None, - prime_handle_to_fd: None, - prime_fd_to_handle: None, - gem_prime_import: None, - gem_prime_import_sg_table: None, - dumb_create: None, - dumb_map_offset: None, - }; -} - -/// A [`dma_buf::DmaBuf`] which has been exported from a GEM object. -/// -/// The [`dma_buf::DmaBuf`] will be released when this type is dropped. -/// -/// # Invariants -/// -/// - `self.0` points to a valid initialized [`dma_buf::DmaBuf`] for the lifetime of this object. -/// - The GEM object from which this [`dma_buf::DmaBuf`] was exported from is guaranteed to be of -/// type `T`. -pub struct DmaBuf(NonNull, PhantomData); - -impl Deref for DmaBuf { - type Target = dma_buf::DmaBuf; - - #[inline] - fn deref(&self) -> &Self::Target { - // SAFETY: This pointer is guaranteed to be valid by our type invariants. - unsafe { self.0.as_ref() } - } -} - -impl Drop for DmaBuf { - #[inline] - fn drop(&mut self) { - // SAFETY: - // - `dma_buf::DmaBuf` is guaranteed to have an identical layout to `struct dma_buf` - // by its type invariants. - // - We hold the last reference to this `DmaBuf`, making it safe to destroy. - unsafe { bindings::drm_gem_dmabuf_release(self.0.cast().as_ptr()) } - } -} - -impl DmaBuf { - /// Leak the reference for this [`DmaBuf`] and return a raw pointer to it. - #[inline] - pub(crate) fn into_raw(self) -> *mut bindings::dma_buf { - let dma_ptr = self.as_raw(); - - core::mem::forget(self); - dma_ptr - } -} - -pub(super) const fn create_fops( - module: Option<&'static crate::ThisModule>, -) -> bindings::file_operations { - // SAFETY: As by the type invariant, it is safe to initialize `bindings::file_operations` - // zeroed. - let mut fops: bindings::file_operations = unsafe { core::mem::zeroed() }; - - fops.owner = match module { - Some(module) => module.as_ptr(), - None => core::ptr::null_mut(), - }; - fops.open = Some(bindings::drm_open); - fops.release = Some(bindings::drm_release); - fops.unlocked_ioctl = Some(bindings::drm_ioctl); - #[cfg(CONFIG_COMPAT)] - { - fops.compat_ioctl = Some(bindings::drm_compat_ioctl); - } - fops.poll = Some(bindings::drm_poll); - fops.read = Some(bindings::drm_read); - fops.llseek = Some(bindings::noop_llseek); - fops.mmap = Some(bindings::drm_gem_mmap); - fops.fop_flags = bindings::FOP_UNSIGNED_OFFSET; - - fops -} diff --git a/rust/kernel/drm_neo/gem/shmem.rs b/rust/kernel/drm_neo/gem/shmem.rs deleted file mode 100644 index 91e75ae396d9b6..00000000000000 --- a/rust/kernel/drm_neo/gem/shmem.rs +++ /dev/null @@ -1,584 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 - -//! DRM GEM shmem helper objects -//! -//! C header: [`include/linux/drm_neo/drm_gem_shmem_helper.h`](srctree/include/drm_neo/drm_gem_shmem_helper.h) - -// TODO: -// - There are a number of spots here that manually acquire/release the DMA reservation lock using -// dma_resv_(un)lock(). In the future we should add support for ww mutex, expose a method to -// acquire a reference to the WwMutex, and then use that directly instead of the C functions here. - -use crate::{ - container_of, - drm_neo::{ - device, - driver, - gem, - private::Sealed, // - }, - error::{ - from_err_ptr, - to_result, // - }, - iosys_map::*, - prelude::*, - scatterlist, - sync::aref::ARef, - transmute::*, - types::Opaque, // -}; -use core::{ - mem::{ - self, - MaybeUninit, // - }, - ops::{ - Deref, - DerefMut, // - }, - ptr::NonNull, - sync::atomic::{AtomicUsize, Ordering}, -}; -use gem::{ - BaseObject, - BaseObjectPrivate, - DriverObject, - IntoGEMObject, // -}; - -/// A struct for controlling the creation of shmem-backed GEM objects. -/// -/// This is used with [`Object::new()`] to control various properties that can only be set when -/// initially creating a shmem-backed GEM object. -#[derive(Default)] -pub struct ObjectConfig<'a, T: DriverObject> { - /// Whether to set the write-combine map flag. - pub map_wc: bool, - - /// Opt in to revocable CPU VMAs, with driver-controlled idle eviction. - pub reclaimable_cpu_mappings: bool, - - /// Reuse the DMA reservation from another GEM object. - /// - /// The newly created [`Object`] will hold an owned refcount to `parent_resv_obj` if specified. - pub parent_resv_obj: Option<&'a Object>, -} - -const SGT_PERMANENT: usize = 1 << (usize::BITS - 1); -const SGT_RECLAIMING: usize = 1 << (usize::BITS - 2); -const SGT_USERS: usize = SGT_RECLAIMING - 1; - -/// A shmem-backed GEM object. -/// -/// # Invariants -/// -/// `obj` contains a valid initialized `struct drm_gem_shmem_object` for the lifetime of this -/// object. -#[repr(C)] -#[pin_data] -pub struct Object { - #[pin] - obj: Opaque, - /// Parent object that owns this object's DMA reservation object. - parent_resv_obj: Option>>, - /// Owned SGT readers, or permanent borrowed access, exclude backing eviction. - sgt_access: AtomicUsize, - #[pin] - inner: T, -} - -super::impl_aref_for_neo_gem_obj!(impl for Object where T: DriverObject); - -// SAFETY: All GEM objects are thread-safe. -unsafe impl Send for Object {} - -// SAFETY: All GEM objects are thread-safe. -unsafe impl Sync for Object {} - -impl Object { - /// `drm_gem_object_funcs` vtable suitable for GEM shmem objects. - const VTABLE: bindings::drm_gem_object_funcs = bindings::drm_gem_object_funcs { - free: Some(Self::free_callback), - open: Some(super::open_callback::), - close: Some(super::close_callback::), - print_info: Some(bindings::drm_gem_shmem_object_print_info), - export: if T::HAS_EXPORT { - Some(super::export_callback::) - } else { - None - }, - pin: Some(bindings::drm_gem_shmem_object_pin), - unpin: Some(bindings::drm_gem_shmem_object_unpin), - get_sg_table: Some(bindings::drm_gem_shmem_object_get_sg_table), - vmap: Some(bindings::drm_gem_shmem_object_vmap), - vunmap: Some(bindings::drm_gem_shmem_object_vunmap), - mmap: Some(bindings::drm_gem_shmem_object_mmap), - status: None, - rss: None, - #[allow(unused_unsafe, reason = "Safe since Rust 1.82.0")] - // SAFETY: `drm_gem_shmem_vm_ops` is a valid, static const on the C side. - vm_ops: unsafe { &raw const bindings::drm_gem_shmem_vm_ops }, - evict: None, - }; - - /// Return a raw pointer to the embedded drm_gem_shmem_object. - fn as_raw_shmem(&self) -> *mut bindings::drm_gem_shmem_object { - self.obj.get() - } - - /// Create a new shmem-backed DRM object of the given size. - /// - /// Additional config options can be specified using `config`. - pub fn new( - dev: &device::Device, - size: usize, - config: ObjectConfig<'_, T>, - args: T::Args, - ) -> Result> { - let new: Pin> = KBox::try_pin_init( - try_pin_init!(Self { - obj <- Opaque::init_zeroed(), - parent_resv_obj: config.parent_resv_obj.map(|p| p.into()), - sgt_access: AtomicUsize::new(0), - inner <- T::new(dev, size, args), - }), - GFP_KERNEL, - )?; - - // SAFETY: `obj.as_raw()` is guaranteed to be valid by the initialization above. - unsafe { (*new.as_raw()).funcs = &Self::VTABLE }; - - // SAFETY: The arguments are all valid via the type invariants. - to_result(unsafe { bindings::drm_gem_shmem_init(dev.as_raw(), new.as_raw_shmem(), size) })?; - - // SAFETY: We never move out of `self`. - let new = KBox::into_raw(unsafe { Pin::into_inner_unchecked(new) }); - - // SAFETY: We're taking over the owned refcount from `drm_gem_shmem_init`. - let obj = unsafe { ARef::from_raw(NonNull::new_unchecked(new)) }; - - // Start filling out values from `config` - if let Some(parent_resv) = config.parent_resv_obj { - // SAFETY: We have yet to expose the new gem object outside of this function, so it is - // safe to modify this field. - unsafe { (*obj.obj.get()).base.resv = parent_resv.raw_dma_resv() }; - } - - // SAFETY: We have yet to expose this object outside of this function, so we're guaranteed - // to have exclusive access - thus making this safe to hold a mutable reference to. - let shmem = unsafe { &mut *obj.as_raw_shmem() }; - shmem.set_map_wc(config.map_wc); - shmem.reclaimable_cpu_mappings = config.reclaimable_cpu_mappings; - - Ok(obj) - } - - /// Returns the `Device` that owns this GEM object. - pub fn dev(&self) -> &device::Device { - // SAFETY: `dev` will have been initialized in `Self::new()` by `drm_gem_shmem_init()`. - unsafe { device::Device::from_raw((*self.as_raw()).dev) } - } - - extern "C" fn free_callback(obj: *mut bindings::drm_gem_object) { - // SAFETY: - // - DRM always passes a valid gem object here - // - We used drm_gem_shmem_create() in our create_gem_object callback, so we know that - // `obj` is contained within a drm_gem_shmem_object - let this = unsafe { container_of!(obj, bindings::drm_gem_shmem_object, base) }; - - // SAFETY: - // - We're in free_callback - so this function is safe to call. - // - We won't be using the gem resources on `this` after this call. - unsafe { bindings::drm_gem_shmem_release(this) }; - - // SAFETY: - // - We verified above that `obj` is valid, which makes `this` valid - // - This function is set in AllocOps, so we know that `this` is contained within a - // `Object` - let this = unsafe { container_of!(Opaque::cast_from(this), Self, obj) }.cast_mut(); - - // SAFETY: We're recovering the Kbox<> we created in gem_create_object() - let _ = unsafe { KBox::from_raw(this) }; - } - - /// Creates (if necessary) and returns an immutable reference to a scatter-gather table of DMA - /// pages for this object. - /// - /// This will pin the object in memory. - #[inline] - pub fn sg_table(&self) -> Result<&scatterlist::SGTable> { - // The borrowed reference can outlive any lock, so pin permanently. - self.acquire_sgt_access(true)?; - self.sg_table_resident() - } - - fn acquire_sgt_access(&self, permanent: bool) -> Result { - let mut state = self.sgt_access.load(Ordering::Acquire); - loop { - if state & SGT_RECLAIMING != 0 { - return Err(EBUSY); - } - let next = if permanent { - state | SGT_PERMANENT - } else { - if state & SGT_USERS == SGT_USERS { return Err(EOVERFLOW); } - state + 1 - }; - match self.sgt_access.compare_exchange_weak( - state, next, Ordering::AcqRel, Ordering::Acquire, - ) { - Ok(_) => return Ok(()), - Err(current) => state = current, - } - } - } - - fn sg_table_resident(&self) -> Result<&scatterlist::SGTable> { - // SAFETY: - // - drm_gem_shmem_get_pages_sgt is thread-safe. - // - drm_gem_shmem_get_pages_sgt returns either a valid pointer to a scatterlist, or an - // error pointer. - let sgt = from_err_ptr(unsafe { bindings::drm_gem_shmem_get_pages_sgt(self.as_raw_shmem()) })?; - - // SAFETY: We checked above that `sgt` is not an error pointer, so it must be a valid - // pointer to a scatterlist - Ok(unsafe { scatterlist::SGTable::from_raw(sgt) }) - } - - /// Creates (if necessary) and returns an owned reference to a scatter-gather table of DMA pages - /// for this object. - /// - /// This is the same as [`sg_table`](Self::sg_table), except that it instead returns an - /// [`shmem::SGTable`] which holds a reference to the associated gem object, instead of a - /// reference to an [`scatterlist::SGTable`]. - /// - /// This will pin the object in memory. - /// - /// [`shmem::SGTable`]: SGTable - pub fn owned_sg_table(&self) -> Result> { - self.acquire_sgt_access(false)?; - let sgt = match self.sg_table_resident() { - Ok(sgt) => sgt.into(), - Err(error) => { - self.sgt_access.fetch_sub(1, Ordering::Release); - return Err(error); - } - }; - Ok(SGTable { sgt, _owner: self.into() }) - } - - /// Whether the caller's sole SG lease is the last user of the backing. - /// - /// # Safety - /// The caller holds the object's DMA reservation lock. - pub unsafe fn sole_sg_lease_releasable_locked(&self) -> bool { - self.sgt_access.load(Ordering::Acquire) == 1 && - // SAFETY: The caller establishes the reservation-lock contract. - unsafe { bindings::drm_gem_shmem_idle_pages_releasable_locked(self.as_raw_shmem()) } - } - - /// Release idle backing without discarding contents. A later - /// `owned_sg_table()` may return different DMA addresses. - /// - /// # Safety - /// - /// The caller holds this object's DMA reservation lock, has removed and - /// invalidated every device mapping, and excludes new device access. - pub unsafe fn release_idle_pages_locked(&self) -> Result { - self.sgt_access.compare_exchange( - 0, SGT_RECLAIMING, Ordering::AcqRel, Ordering::Acquire, - ).map_err(|_| EBUSY)?; - // SAFETY: The caller's contract plus the exclusive state above. - let result = to_result(unsafe { - bindings::drm_gem_shmem_release_idle_pages_locked(self.as_raw_shmem()) - }); - self.sgt_access.store(0, Ordering::Release); - result - } - - /// Apply purge advice (0 = needed, >0 = may be purged) under the - /// object's reservation lock. Returns false if the object was already - /// purged; the advice is then ignored. - pub fn madvise(&self, madv: i32) -> bool { - // SAFETY: The object retains its reservation; the C helper requires it held. - unsafe { - bindings::dma_resv_lock(self.raw_dma_resv(), core::ptr::null_mut()); - let retained = bindings::drm_gem_shmem_madvise_locked(self.as_raw_shmem(), madv) != 0; - bindings::dma_resv_unlock(self.raw_dma_resv()); - retained - } - } - - /// Current purge advice: negative once purged. - /// - /// # Safety - /// The caller holds the object's DMA reservation lock. - pub unsafe fn madv_locked(&self) -> i32 { - // SAFETY: Reservation ownership serializes the field. - unsafe { (*self.as_raw_shmem()).madv } - } - - /// Discard the backing of a purgeable object. The handle and logical - /// mappings remain; the contents are gone and `madv_locked()` turns negative. - /// - /// # Safety - /// - /// The caller holds this object's DMA reservation lock, has removed and - /// invalidated every device mapping, and no Rust SG lease remains. - pub unsafe fn purge_locked(&self) -> Result { - // SAFETY: The caller's contract; the exclusive state below excludes new leases. - if !unsafe { bindings::drm_gem_shmem_is_purgeable(self.as_raw_shmem()) } { - return Err(EBUSY); - } - self.sgt_access.compare_exchange( - 0, SGT_RECLAIMING, Ordering::AcqRel, Ordering::Acquire, - ).map_err(|_| EBUSY)?; - // SAFETY: See above. - unsafe { bindings::drm_gem_shmem_purge_locked(self.as_raw_shmem()) }; - self.sgt_access.store(0, Ordering::Release); - Ok(()) - } - - /// Attempt to create a [`RawIoSysMap`] from the gem object. - fn raw_vmap(&self) -> Result> { - build_assert!( - mem::size_of::() > 0, - "It doesn't make sense for the mapping type to be a ZST" - ); - - let mut map: MaybeUninit = MaybeUninit::uninit(); - - // SAFETY: drm_gem_shmem_vmap can be called with the DMA reservation lock held - to_result(unsafe { - // TODO: see top of file - bindings::dma_resv_lock(self.raw_dma_resv(), core::ptr::null_mut()); - let ret = bindings::drm_gem_shmem_vmap_locked(self.as_raw_shmem(), map.as_mut_ptr()); - bindings::dma_resv_unlock(self.raw_dma_resv()); - ret - })?; - - // SAFETY: if drm_gem_shmem_vmap did not fail, map is initialized now - Ok(unsafe { RawIoSysMap::from_raw(map.assume_init()) }) - } - - /// Unmap a [`RawIoSysMap`] from the gem object. - /// - /// # Safety - /// - /// - The caller promises that `map` came from a prior call to [`Self::raw_vmap`] on this gem - /// object. - /// - The caller promises that the memory pointed to by `map` will no longer be accesed through - /// this instance. - unsafe fn raw_vunmap(&self, map: &mut RawIoSysMap) { - let resv = self.raw_dma_resv(); - - // SAFETY: - // - This function is safe to call with the DMA reservation lock held - // - Our `ARef` is proof that the underlying gem object here is initialized and thus safe to - // dereference. - unsafe { - // TODO: see top of file - bindings::dma_resv_lock(resv, core::ptr::null_mut()); - bindings::drm_gem_shmem_vunmap_locked(self.as_raw_shmem(), map.as_raw_mut()); - bindings::dma_resv_unlock(resv); - } - } - - /// Creates and returns a virtual kernel memory mapping for this object. - pub fn vmap(&self) -> Result> { - let map = self.raw_vmap()?; - - Ok(VMapRef { - // SAFETY: - // - The size of the vmap is the same as the size of the gem - // - The vmap will remain alive until this object is dropped. - map: unsafe { IoSysMapRef::new(map, self.size()) }, - owner: self, - }) - } - - /// Creates and returns an owned reference to a virtual kernel memory mapping for this object. - pub fn owned_vmap(&self) -> Result> { - Ok(VMap { - map: self.raw_vmap()?, - owner: self.into(), - }) - } -} - -impl Deref for Object { - type Target = T; - - fn deref(&self) -> &Self::Target { - &self.inner - } -} - -impl DerefMut for Object { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.inner - } -} - -impl Sealed for Object {} - -impl gem::IntoGEMObject for Object { - fn as_raw(&self) -> *mut bindings::drm_gem_object { - // SAFETY: - // - Our immutable reference is proof that this is safe to dereference. - // - `obj` is always a valid drm_gem_shmem_object via our type invariants. - unsafe { &raw mut (*self.obj.get()).base } - } - - unsafe fn from_raw<'a>(obj: *mut bindings::drm_gem_object) -> &'a Object { - // SAFETY: The safety contract of from_gem_obj() guarantees that `obj` is contained within - // `Self` - unsafe { - let obj = Opaque::cast_from(container_of!(obj, bindings::drm_gem_shmem_object, base)); - - &*container_of!(obj, Object, obj) - } - } -} - -impl driver::AllocImpl for Object { - type Driver = T::Driver; - - const ALLOC_OPS: driver::AllocOps = driver::AllocOps { - gem_create_object: None, - prime_handle_to_fd: None, - prime_fd_to_handle: None, - gem_prime_import: None, - gem_prime_import_sg_table: Some(bindings::drm_gem_shmem_prime_import_sg_table), - dumb_create: Some(bindings::drm_gem_shmem_dumb_create), - dumb_map_offset: None, - }; -} - -/// A borrowed reference to a virtual mapping for a shmem-based GEM object in kernel address space. -pub struct VMapRef<'a, D: DriverObject, T: AsBytes + FromBytes> { - map: IoSysMapRef<'a, T>, - owner: &'a Object, -} - -impl<'a, D: DriverObject, T: AsBytes + FromBytes> Clone for VMapRef<'a, D, T> { - fn clone(&self) -> Self { - // SAFETY: We have a successful vmap already, so this can't fail - unsafe { self.owner.vmap().unwrap_unchecked() } - } -} - -impl<'a, D: DriverObject, T: AsBytes + FromBytes> Deref for VMapRef<'a, D, T> { - type Target = IoSysMapRef<'a, T>; - - fn deref(&self) -> &Self::Target { - &self.map - } -} - -impl<'a, D: DriverObject, T: AsBytes + FromBytes> DerefMut for VMapRef<'a, D, T> { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.map - } -} - -impl<'a, D: DriverObject, T: AsBytes + FromBytes> Drop for VMapRef<'a, D, T> { - fn drop(&mut self) { - // SAFETY: Our existence is proof that this map was previously created using self.owner. - unsafe { self.owner.raw_vunmap(&mut self.map) }; - } -} - -/// An owned reference to a virtual mapping for a shmem-based GEM object in kernel address space. -/// -/// # Invariants -/// -/// - The memory pointed to by `map` is at least as large as `T`. -/// - The memory pointed to by `map` remains valid at least until this object is dropped. -pub struct VMap { - map: RawIoSysMap, - owner: ARef>, -} - -impl Clone for VMap { - fn clone(&self) -> Self { - // SAFETY: We have a successful vmap already, so this can't fail - unsafe { self.owner.owned_vmap().unwrap_unchecked() } - } -} - -impl<'a, D: DriverObject, T: AsBytes + FromBytes> From> for VMap { - fn from(value: VMapRef<'a, D, T>) -> Self { - let this = Self { - map: value.map.clone(), - owner: value.owner.into(), - }; - - mem::forget(value); - this - } -} - -impl VMap { - /// Return a reference to the iosys map for this `VMap`. - pub fn get(&self) -> IoSysMapRef<'_, T> { - // SAFETY: The size of the iosys_map is equivalent to the size of the gem object. - unsafe { IoSysMapRef::new(self.map.clone(), self.owner.size()) } - } - - /// Borrows a reference to the object that owns this virtual mapping. - pub fn owner(&self) -> &Object { - &self.owner - } -} - -impl Drop for VMap { - fn drop(&mut self) { - // SAFETY: Our existence is proof that this map was previously created using self.owner - unsafe { self.owner.raw_vunmap(&mut self.map) }; - } -} - -/// SAFETY: `iosys_map` objects are safe to send across threads. -unsafe impl Send for VMap {} -/// SAFETY: `iosys_map` objects are safe to send across threads. -unsafe impl Sync for VMap {} - -/// An owned reference to a scatter-gather table of DMA address spans for a GEM shmem object. -/// -/// This object holds an owned reference to the underlying GEM shmem object, ensuring that the -/// [`scatterlist::SGTable`] referenced by this type remains valid for the lifetime of this object. -/// -/// # Invariants -/// -/// - `_owner` retains both the GEM object and one counted SG-table lease; eviction -/// cannot begin until every lease has been released. -/// - `sgt` corresponds to the owned object in `_owner`. -/// - This object is only exposed in situations where we know the underlying `SGTable` will not be -/// modified for the lifetime of this object. Thus, it is safe to send/access this type across -/// threads. -pub struct SGTable { - sgt: NonNull, - _owner: ARef>, -} - -// SAFETY: This object is thread-safe via our type invariants. -unsafe impl Send for SGTable {} -// SAFETY: This object is thread-safe via our type invariants. -unsafe impl Sync for SGTable {} - -impl Drop for SGTable { - fn drop(&mut self) { - self._owner.sgt_access.fetch_sub(1, Ordering::Release); - } -} - -impl Deref for SGTable { - type Target = scatterlist::SGTable; - - fn deref(&self) -> &Self::Target { - // SAFETY: Creating an immutable reference to this is safe via our type invariants. - unsafe { self.sgt.as_ref() } - } -} diff --git a/rust/kernel/drm_neo/gpuvm.rs b/rust/kernel/drm_neo/gpuvm.rs deleted file mode 100644 index ac82eaf1955d3e..00000000000000 --- a/rust/kernel/drm_neo/gpuvm.rs +++ /dev/null @@ -1,1375 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM Sync Objects -//! -//! C header: [`include/drm_neo/drm_gpuvm.h`](../../../../include/drm_neo/drm_gpuvm.h) - -#![allow(missing_docs)] - -use crate::{ - bindings, - drm_neo::{ - self, - device, - gem::{ - BaseObject, - IntoGEMObject, // - }, // - }, - error::{ - code::{ - EINVAL, - ENOMEM, // - }, - from_err_ptr, - from_result, - to_result, - Error, - Result, // - }, - prelude::*, - sync::aref::{ - ARef, - AlwaysRefCounted, // - }, - types::{NotThreadSafe, Opaque}, // -}; - -use core::cell::UnsafeCell; -use core::marker::{PhantomData, PhantomPinned}; -use core::mem::MaybeUninit; -use core::ops::{Deref, DerefMut, Range}; -use core::ptr::NonNull; -use pin_init; - -/// GpuVaFlags to be used for a GpuVa. -/// -/// They can be combined with the operators `|`, `&`, and `!`. -#[derive(Clone, Copy, PartialEq, Default)] -pub struct GpuVaFlags(u32); - -impl GpuVaFlags { - /// No GpuVaFlags (zero) - pub const NONE: GpuVaFlags = GpuVaFlags(0); - - /// The backing GEM is invalidated. - pub const INVALIDATED: GpuVaFlags = GpuVaFlags(bindings::drm_gpuva_flags_DRM_GPUVA_INVALIDATED); - - /// The GpuVa is a sparse mapping. - pub const SPARSE: GpuVaFlags = GpuVaFlags(bindings::drm_gpuva_flags_DRM_GPUVA_SPARSE); - - /// The GpuVa is a repeat mapping. - pub const REPEAT: GpuVaFlags = GpuVaFlags(bindings::drm_gpuva_flags_DRM_GPUVA_REPEAT); - - /// Construct a driver-specific GpuVaFlag. - /// - /// The argument must be a flag index in the range [0..28]. - pub const fn user_flag(index: u32) -> GpuVaFlags { - let flags = bindings::drm_gpuva_flags_DRM_GPUVA_USERBITS << index; - assert!(flags != 0); - GpuVaFlags(flags) - } - - /// Get the raw representation of this flag. - pub(crate) fn as_raw(self) -> u32 { - self.0 - } - - /// Check whether `flags` is contained in `self`. - pub fn contains(self, flags: GpuVaFlags) -> bool { - (self & flags) == flags - } -} - -impl core::ops::BitOr for GpuVaFlags { - type Output = Self; - fn bitor(self, rhs: Self) -> Self::Output { - Self(self.0 | rhs.0) - } -} - -impl core::ops::BitAnd for GpuVaFlags { - type Output = Self; - fn bitand(self, rhs: Self) -> Self::Output { - Self(self.0 & rhs.0) - } -} - -impl core::ops::Not for GpuVaFlags { - type Output = Self; - fn not(self) -> Self::Output { - Self(!self.0) - } -} - -/// Trait that must be implemented by DRM drivers to represent a DRM GpuVm (a GPU address space). -pub trait DriverGpuVm: Sized { - /// The parent `Driver` implementation for this `DriverGpuVm`. - type Driver: drm_neo::Driver; - type GpuVa: DriverGpuVa = (); - type GpuVmBo: DriverGpuVmBo = (); - type StepContext = (); - - fn step_map( - self: &mut UpdatingGpuVm<'_, Self>, - op: &mut OpMap, - ctx: &mut Self::StepContext, - ) -> Result; - fn step_unmap( - self: &mut UpdatingGpuVm<'_, Self>, - op: &mut OpUnMap, - ctx: &mut Self::StepContext, - ) -> Result; - fn step_remap( - self: &mut UpdatingGpuVm<'_, Self>, - op: &mut OpReMap, - vm_bo: &GpuVmBo, - ctx: &mut Self::StepContext, - ) -> Result; -} - -struct StepContext<'a, T: DriverGpuVm> { - gpuvm: &'a GpuVm, - ctx: &'a mut T::StepContext, -} - -/// Trait that must be implemented by DRM drivers to represent a DRM GpuVa (a mapping in GPU address space). -pub trait DriverGpuVa: Sized {} - -impl DriverGpuVa for () {} - -/// Trait that must be implemented by DRM drivers to represent a DRM GpuVmBo (a connection between a BO and a VM). -pub trait DriverGpuVmBo: Sized { - fn new() -> impl PinInit; -} - -/// Provide a default implementation for trivial types -impl DriverGpuVmBo for T { - fn new() -> impl PinInit { - pin_init::default() - } -} - -/// A convenience type for the driver's GEM object. -type Object = <::Driver as drm_neo::driver::Driver>::Object; - -#[repr(transparent)] -pub struct OpMap(bindings::drm_gpuva_op_map, PhantomData); -#[repr(transparent)] -pub struct OpUnMap(bindings::drm_gpuva_op_unmap, PhantomData); -#[repr(transparent)] -pub struct OpReMap(bindings::drm_gpuva_op_remap, PhantomData); - -impl OpMap { - pub fn addr(&self) -> u64 { - self.0.va.addr - } - pub fn range(&self) -> u64 { - self.0.va.range - } - pub fn offset(&self) -> u64 { - self.0.gem.offset - } - pub fn flags(&self) -> GpuVaFlags { - GpuVaFlags(self.0.flags) - } - pub fn object(&self) -> &Object { - let p = unsafe { as IntoGEMObject>::from_raw(self.0.gem.obj) }; - // SAFETY: The GEM object has an active reference for the lifetime of this op - &*p - } - pub fn map_and_link_va( - &mut self, - gpuvm: &mut UpdatingGpuVm<'_, T>, - gpuva: Pin>>, - gpuvmbo: &GpuVmBo, - ) -> Result<(), Pin>>> { - // SAFETY: We are handing off the GpuVa ownership and it will not be moved. - let p = KBox::leak(unsafe { Pin::into_inner_unchecked(gpuva) }); - // SAFETY: These C functions are called with the correct invariants - unsafe { - bindings::drm_gpuva_init_from_op(&mut p.gpuva, &mut self.0); - if bindings::drm_gpuva_insert(gpuvm.0.gpuvm() as *mut _, &mut p.gpuva) != 0 { - // EEXIST, return the GpuVa to the caller as an error - return Err(Pin::new_unchecked(KBox::from_raw(p))); - }; - // SAFETY: This takes a new reference to the gpuvmbo. - gpuvmbo.lock_gpuva(); - bindings::drm_gpuva_link(&mut p.gpuva, &gpuvmbo.bo as *const _ as *mut _); - gpuvmbo.unlock_gpuva(); - } - Ok(()) - } -} - -impl OpUnMap { - pub fn va(&self) -> Option<&GpuVa> { - if self.0.va.is_null() { - return None; - } - // SAFETY: Container invariant is guaranteed for ops structs created for our types. - let p = unsafe { crate::container_of!(self.0.va, GpuVa, gpuva) as *mut GpuVa }; - // SAFETY: The GpuVa object reference is valid per the op_unmap contract - Some(unsafe { &*p }) - } - pub fn unmap_and_unlink_va(&mut self) -> Option>>> { - self.do_unmap_and_unlink_va(false) - } - pub fn unmap_and_unlink_va_defer(&mut self) -> Option>>> { - self.do_unmap_and_unlink_va(true) - } - fn do_unmap_and_unlink_va(&mut self, defer: bool) -> Option>>> { - if self.0.va.is_null() { - return None; - } - // SAFETY: Container invariant is guaranteed for ops structs created for our types. - let p = unsafe { crate::container_of!(self.0.va, GpuVa, gpuva) as *mut GpuVa }; - - // SAFETY: The GpuVa object reference is valid per the op_unmap contract - unsafe { - bindings::drm_gpuva_unmap(&mut self.0); - if defer { - bindings::drm_gpuva_unlink_defer(self.0.va); - } else { - bindings::drm_gpuva_unlink(self.0.va); - } - } - - // Unlinking/unmapping relinquishes ownership of the GpuVa object, - // so clear the pointer - self.0.va = core::ptr::null_mut(); - // SAFETY: The GpuVa object reference is valid per the op_unmap contract - Some(unsafe { Pin::new_unchecked(KBox::from_raw(p)) }) - } -} - -impl OpReMap { - pub fn prev_map(&mut self) -> Option<&mut OpMap> { - // SAFETY: The prev pointer must be valid if not-NULL per the op_remap contract - unsafe { (self.0.prev as *mut OpMap).as_mut() } - } - pub fn next_map(&mut self) -> Option<&mut OpMap> { - // SAFETY: The next pointer must be valid if not-NULL per the op_remap contract - unsafe { (self.0.next as *mut OpMap).as_mut() } - } - pub fn unmap(&mut self) -> &mut OpUnMap { - // SAFETY: The unmap pointer is always valid per the op_remap contract - unsafe { (self.0.unmap as *mut OpUnMap).as_mut().unwrap() } - } -} - -/// A base GPU VA. -#[repr(C)] -#[pin_data] -pub struct GpuVa { - #[pin] - gpuva: bindings::drm_gpuva, - #[pin] - inner: T::GpuVa, - #[pin] - _p: PhantomPinned, -} - -impl GpuVa { - pub fn new(inner: impl PinInit) -> Result>>> - where - Error: From, - { - KBox::try_pin_init( - try_pin_init!(Self { - gpuva <- pin_init::init_zeroed(), - inner <- inner, - _p: PhantomPinned - }), - GFP_KERNEL, - ) - } - - /// Driver metadata retained independently of the resident page tables. - pub fn inner(&self) -> &T::GpuVa { &self.inner } - - /// Update metadata before linking a newly allocated GPUVA. - pub fn inner_mut(self: Pin<&mut Self>) -> Pin<&mut T::GpuVa> { - // SAFETY: inner is structurally pinned with its enclosing GPUVA. - unsafe { self.map_unchecked_mut(|va| &mut va.inner) } - } - - pub fn addr(&self) -> u64 { - self.gpuva.va.addr - } - pub fn range(&self) -> u64 { - self.gpuva.va.range - } - pub fn offset(&self) -> u64 { - self.gpuva.gem.offset - } - pub fn flags(&self) -> GpuVaFlags { - GpuVaFlags(self.gpuva.flags) - } -} - -/// A base GpuVm BO. -#[repr(C)] -#[pin_data] -pub struct GpuVmBo { - #[pin] - bo: bindings::drm_gpuvm_bo, - #[pin] - inner: T::GpuVmBo, - #[pin] - _p: PhantomPinned, -} - -impl GpuVmBo { - /// The GEM object retained by this VM/BO association. - pub fn object(&self) -> &Object { - // SAFETY: drm_gpuvm_bo owns this GEM reference until its destructor; - // DriverGpuVm fixes the GEM type for every association in this VM. - unsafe { &* as IntoGEMObject>::from_raw(self.bo.obj) } - } - - /// Return a reference to the inner driver data for this GpuVmBo - pub fn inner(&self) -> &T::GpuVmBo { - &self.inner - } - /// Lock the GpuVmBo's gem boject gpuva lock - pub fn lock_gpuva(&self) { - unsafe { - let lock = &raw mut (*self.bo.obj).gpuva.lock; - bindings::mutex_lock(lock); - } - } - /// Unlock the GpuVmBo's gem boject gpuva lock - pub fn unlock_gpuva(&self) { - unsafe { - let lock = &raw mut (*self.bo.obj).gpuva.lock; - bindings::mutex_unlock(lock); - } - } -} - -// SAFETY: DRM GpuVmBo objects are always reference counted and the get/put functions -// satisfy the requirements. -unsafe impl AlwaysRefCounted for GpuVmBo { - fn inc_ref(&self) { - // SAFETY: The drm_gpuvm_get function satisfies the requirements for inc_ref(). - unsafe { bindings::drm_gpuvm_bo_get(&self.bo as *const _ as *mut _) }; - } - - unsafe fn dec_ref(obj: NonNull) { - // SAFETY: This owns a live BO reference. Immediate-mode GPUVA lists - // use the GEM's gpuva mutex, not its reservation. The deferred put - // releases that mutex before final GEM destruction. Keep the GPUVM - // alive across draining, which can drop its last BO-owned reference. - unsafe { - let bo = &raw mut (*obj.as_ptr()).bo; - let vm = (*bo).vm; - if (*vm).flags & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE != 0 { - bindings::drm_gpuvm_get(vm); - bindings::drm_gpuvm_bo_put_deferred(bo); - bindings::drm_gpuvm_bo_deferred_cleanup(vm); - bindings::drm_gpuvm_put(vm); - return; - } - - // A final BO put also puts its GEM. Retain the GEM until after - // unlocking its reservation, which may be embedded in that GEM. - let gem = (*bo).obj; - bindings::drm_gem_object_get(gem); - let resv = (*gem).resv; - bindings::dma_resv_lock(resv, core::ptr::null_mut()); - bindings::drm_gpuvm_bo_put(bo); - bindings::dma_resv_unlock(resv); - bindings::drm_gem_object_put(gem); - } - } -} - -/// A base GPU VM. -#[repr(C)] -#[pin_data] -pub struct GpuVm { - #[pin] - gpuvm: Opaque, - #[pin] - inner: UnsafeCell, - #[pin] - _p: PhantomPinned, -} - -pub(super) unsafe extern "C" fn vm_free_callback( - raw_gpuvm: *mut bindings::drm_gpuvm, -) { - // SAFETY: Container invariant is guaranteed for objects using our callback. - let p = unsafe { - crate::container_of!( - raw_gpuvm as *mut Opaque, - GpuVm, - gpuvm - ) as *mut GpuVm - }; - - // SAFETY: p is guaranteed to be valid for drm_gpuvm objects using this callback. - unsafe { drop(KBox::from_raw(p)) }; -} - -pub(super) unsafe extern "C" fn vm_bo_alloc_callback() -> *mut bindings::drm_gpuvm_bo -{ - let obj: Result>>> = KBox::try_pin_init( - try_pin_init!(GpuVmBo:: { - // The bindgen `Default` implementation is exactly a zero fill, but its - // out-of-line symbol is not exported to loadable modules. `drm_gpuvm_bo` - // is `Zeroable`, so initialize it directly without a cross-crate call. - bo <- pin_init::zeroed::(), - inner <- T::GpuVmBo::new(), - _p: PhantomPinned - }), - GFP_KERNEL, - ); - - match obj { - Ok(obj) => - // SAFETY: The DRM core will keep this object pinned - unsafe { - let p = KBox::leak(Pin::into_inner_unchecked(obj)); - &mut p.bo - }, - Err(_) => core::ptr::null_mut(), - } -} - -pub(super) unsafe extern "C" fn vm_bo_free_callback( - raw_vm_bo: *mut bindings::drm_gpuvm_bo, -) { - // SAFETY: Container invariant is guaranteed for objects using this callback. - let p = unsafe { crate::container_of!(raw_vm_bo, GpuVmBo, bo) as *mut GpuVmBo }; - - // SAFETY: p is guaranteed to be valid for drm_gpuvm_bo objects using this callback. - unsafe { drop(KBox::from_raw(p)) }; -} - -pub(super) unsafe extern "C" fn step_map_callback( - op: *mut bindings::drm_gpuva_op, - _priv: *mut core::ffi::c_void, -) -> core::ffi::c_int { - // SAFETY: We know this is a map op, and OpMap is a transparent wrapper. - let map = unsafe { &mut *((&mut (*op).__bindgen_anon_1.map) as *mut _ as *mut OpMap) }; - // SAFETY: This is a pointer to a StepContext created inline in sm_map(), which is - // guaranteed to outlive this function. - let ctx = unsafe { &mut *(_priv as *mut StepContext<'_, T>) }; - - from_result(|| { - UpdatingGpuVm(ctx.gpuvm).step_map(map, ctx.ctx)?; - Ok(0) - }) -} - -pub(super) unsafe extern "C" fn step_remap_callback( - op: *mut bindings::drm_gpuva_op, - _priv: *mut core::ffi::c_void, -) -> core::ffi::c_int { - // SAFETY: We know this is a map op, and OpReMap is a transparent wrapper. - let remap = unsafe { &mut *((&mut (*op).__bindgen_anon_1.remap) as *mut _ as *mut OpReMap) }; - // SAFETY: This is a pointer to a StepContext created inline in sm_map(), which is - // guaranteed to outlive this function. - let ctx = unsafe { &mut *(_priv as *mut StepContext<'_, T>) }; - - let p_vm_bo = remap.unmap().va().unwrap().gpuva.vm_bo; - - let res = { - // SAFETY: vm_bo pointer must be valid and non-null by the step_remap invariants. - // Since we grab a ref, this reference's lifetime is until the decref. - let vm_bo_ref = unsafe { - bindings::drm_gpuvm_bo_get(p_vm_bo); - &*(crate::container_of!(p_vm_bo, GpuVmBo, bo) as *mut GpuVmBo) - }; - - from_result(|| { - UpdatingGpuVm(ctx.gpuvm).step_remap(remap, vm_bo_ref, ctx.ctx)?; - Ok(0) - }) - }; - - // SAFETY: We incremented the refcount above, and the Rust reference we took is - // no longer in scope. - unsafe { bindings::drm_gpuvm_bo_put_deferred(p_vm_bo) }; - - res -} -pub(super) unsafe extern "C" fn step_unmap_callback( - op: *mut bindings::drm_gpuva_op, - _priv: *mut core::ffi::c_void, -) -> core::ffi::c_int { - // SAFETY: We know this is a map op, and OpUnMap is a transparent wrapper. - let unmap = unsafe { &mut *((&mut (*op).__bindgen_anon_1.unmap) as *mut _ as *mut OpUnMap) }; - // SAFETY: This is a pointer to a StepContext created inline in sm_map(), which is - // guaranteed to outlive this function. - let ctx = unsafe { &mut *(_priv as *mut StepContext<'_, T>) }; - - from_result(|| { - UpdatingGpuVm(ctx.gpuvm).step_unmap(unmap, ctx.ctx)?; - Ok(0) - }) -} - -pub(super) unsafe extern "C" fn exec_lock_gem_object( - vm_exec: *mut bindings::drm_gpuvm_exec, -) -> core::ffi::c_int { - // SAFETY: The gpuvm_exec object is valid and priv_ is a GEM object pointer - // when this callback is used - unsafe { bindings::drm_exec_lock_obj(&mut (*vm_exec).exec, (*vm_exec).extra.priv_ as *mut _) } -} - -impl GpuVm { - const OPS: bindings::drm_gpuvm_ops = bindings::drm_gpuvm_ops { - vm_free: Some(vm_free_callback::), - op_alloc: None, - op_free: None, - vm_bo_alloc: Some(vm_bo_alloc_callback::), - vm_bo_free: Some(vm_bo_free_callback::), - vm_bo_validate: None, - sm_step_map: Some(step_map_callback::), - sm_step_remap: Some(step_remap_callback::), - sm_step_unmap: Some(step_unmap_callback::), - sm_can_merge_flags: None, - }; - - fn gpuvm(&self) -> *const bindings::drm_gpuvm { - self.gpuvm.get() - } - - pub fn new( - name: &'static CStr, - flags: bindings::drm_gpuvm_flags, - dev: &device::Device, - r_obj: ARef>, - range: Range, - reserve_range: Range, - inner: impl PinInit, - ) -> Result>> - where - Error: From, - { - let obj: Pin> = KBox::try_pin_init( - try_pin_init!(Self { - // SAFETY: drm_gpuvm_init cannot fail and always initializes the member - gpuvm <- unsafe { - pin_init::pin_init_from_closure(move |slot: *mut Opaque | { - // Zero-init required by drm_gpuvm_init - *slot = Opaque::zeroed(); - bindings::drm_gpuvm_init( - Opaque::cast_into(slot), - name.as_char_ptr(), - flags, - dev.as_raw(), - r_obj.as_raw() as *const _ as *mut _, - range.start, - range.end - range.start, - reserve_range.start, - reserve_range.end - reserve_range.start, - &Self::OPS - ); - Ok(()) - }) - }, - // SAFETY: Just passing through to the initializer argument - inner <- unsafe { - pin_init::pin_init_from_closure(move |slot: *mut UnsafeCell | { - inner.__pinned_init(slot as *mut _) - }) - }, - _p: PhantomPinned - }), - GFP_KERNEL, - )?; - - // SAFETY: We never move out of the object - let vm_ref = unsafe { - ARef::from_raw(NonNull::new_unchecked(KBox::leak( - Pin::into_inner_unchecked(obj), - ))) - }; - - Ok(vm_ref) - } - - /// Lock only the VM's driver data, without allocating an exec context. - /// - /// This takes the same shared reservation as `exec_lock`, but does not lock - /// external BO reservations. The returned guard exposes only the inner - /// driver data, not GPUVA operations requiring those other reservations. - /// Callers must not already hold a reservation lock. - pub fn lock_inner(&self) -> GpuVmInnerGuard<'_, T> { - // SAFETY: The borrowed GPUVM retains r_obj and its reservation for the - // guard's lifetime. A non-interruptible single-reservation lock with no - // ww context cannot fail or allocate. No other reservation is acquired. - unsafe { - bindings::dma_resv_lock((*(*self.gpuvm()).r_obj).resv, core::ptr::null_mut()); - } - GpuVmInnerGuard { - gpuvm: self, - _not_send: NotThreadSafe, - } - } - - // The caller owns the VM reservation and keeps the exact logical VA linked. - unsafe fn set_invalidated_locked(&self, addr: u64, range: u64, invalidated: bool) -> Result { - if range == 0 || addr.checked_add(range).is_none() { return Err(EINVAL); } - // SAFETY: The caller owns the VM reservation and keeps the VA linked. - let raw = unsafe { bindings::drm_gpuva_find(self.gpuvm() as *mut _, addr, range) }; - if raw.is_null() { return Err(EINVAL); } - // SAFETY: Reservation ownership excludes all concurrent VA mutation. - unsafe { - if (*raw).va.addr != addr || (*raw).va.range != range { return Err(EINVAL); } - if invalidated { (*raw).flags |= GpuVaFlags::INVALIDATED.as_raw(); } - else { (*raw).flags &= !GpuVaFlags::INVALIDATED.as_raw(); } - } - Ok(()) - } - - /// Try to lock only the VM reservation, without allocation or waiting. - /// The guard protects VM metadata and private BOs, not external backing. - /// Callers must not already hold a reservation lock. - pub fn try_lock_private(&self) -> Option> { - // SAFETY: r_obj and its reservation are retained by this GPUVM. - let resv = unsafe { (*(*self.gpuvm()).r_obj).resv }; - // SAFETY: The reservation outlives the returned borrow and guard. - if !unsafe { bindings::dma_resv_trylock(resv) } { return None; } - Some(PrivateGpuVmGuard { gpuvm: self, resv, _not_send: NotThreadSafe }) - } - - pub fn exec_lock<'a, 'b>( - &'a self, - obj: Option<&'b Object>, - interruptible: bool, - ) -> Result> { - // Do not try to lock the object if it is internal (since it is already locked). - let is_ext = obj.map(|a| self.is_extobj(a)).unwrap_or(false); - - // vm_exec needs a stable address while locking. Construct the unlock - // guard only after success; C already finalizes the exec on error. - let mut vm_exec = KBox::init( - init!(bindings::drm_gpuvm_exec { - vm: self.gpuvm() as *mut _, - flags: if interruptible { - bindings::DRM_EXEC_INTERRUPTIBLE_WAIT - } else { - 0 - }, - // SAFETY: bindgen's `Default` for this C structure is exactly - // `MaybeUninit::zeroed().assume_init()`. Spell that out here so - // external modules do not import its unexported trait method. - exec: unsafe { MaybeUninit::::zeroed().assume_init() }, - extra: match (is_ext, obj) { - (true, Some(obj)) => bindings::drm_gpuvm_exec__bindgen_ty_1 { - fn_: Some(exec_lock_gem_object), - priv_: obj.as_raw() as *const _ as *mut _, - }, - // SAFETY: as above, this bindgen C structure is zero-valid. - _ => unsafe { - MaybeUninit::::zeroed() - .assume_init() - }, - }, - num_fences: 0, - }), - GFP_KERNEL, - )?; - - // SAFETY: The object is valid and was initialized above - to_result(unsafe { bindings::drm_gpuvm_exec_lock(&mut *vm_exec) })?; - - Ok(LockedGpuVm { - gpuvm: self, - vm_exec, - objects: LockedObjects::Single(obj), - }) - } - - /// Lock the GPUVM and an array of additional GEM objects in one - /// wound/wait transaction. - pub fn exec_lock_array<'a, 'b>( - &'a self, - objects: &'b [ARef>], - interruptible: bool, - ) -> Result> { - let count = u32::try_from(objects.len()).map_err(|_| EINVAL)?; - let mut raw_objects = KVec::with_capacity(objects.len(), GFP_KERNEL)?; - for object in objects { - raw_objects.push(object.as_raw(), GFP_KERNEL)?; - } - - // vm_exec needs a stable address while locking. Construct the unlock - // guard only after success; C already finalizes the exec on error. - let mut vm_exec = KBox::init( - init!(bindings::drm_gpuvm_exec { - vm: self.gpuvm() as *mut _, - flags: (if interruptible { - bindings::DRM_EXEC_INTERRUPTIBLE_WAIT - } else { - 0 - }) | bindings::DRM_EXEC_IGNORE_DUPLICATES, - // SAFETY: both bindgen C structures are zero-valid; their generated - // `Default` methods perform the same zero initialization but are not - // exported from the kernel crate to loadable modules. - exec: unsafe { MaybeUninit::::zeroed().assume_init() }, - extra: unsafe { - MaybeUninit::::zeroed() - .assume_init() - }, - num_fences: 0, - }), - GFP_KERNEL, - )?; - - // SAFETY: Every raw pointer is backed by an ARef in `objects`, which - // outlives the returned guard. The helper consumes the temporary - // pointer array before returning and keeps its own object references. - to_result(unsafe { - bindings::drm_gpuvm_exec_lock_array( - &mut *vm_exec, - raw_objects.as_mut_ptr(), - count, - ) - })?; - - Ok(LockedGpuVm { - gpuvm: self, - vm_exec, - objects: LockedObjects::Array(objects), - }) - } - - /// Returns true if the given object is external to the GPUVM - /// (that is, if it does not share the DMA reservation object of the GPUVM). - pub fn is_extobj(&self, obj: &impl IntoGEMObject) -> bool { - let gem = obj.as_raw() as *const _ as *mut _; - // SAFETY: This is safe to call as long as the arguments are valid pointers. - unsafe { bindings::drm_gpuvm_is_extobj(self.gpuvm() as *mut _, gem) } - } - - /// Validate against the immutable address-space bounds and kernel cutout. - pub fn range_valid(&self, addr: u64, range: u64) -> bool { - // SAFETY: the borrowed VM retains its immutable geometry. - unsafe { bindings::drm_gpuvm_range_valid(self.gpuvm() as *mut _, addr, range) } - } - - pub fn bo_deferred_cleanup(&self) { - unsafe { bindings::drm_gpuvm_bo_deferred_cleanup(self.gpuvm() as *mut _) } - } - - pub fn find_bo(&self, obj: &Object) -> Option>> { - obj.lock_gpuva(); - // SAFETY: drm_gem_object.gpuva.lock was just locked. - let p = unsafe { - bindings::drm_gpuvm_bo_find(self.gpuvm() as *mut _, obj.as_raw() as *const _ as *mut _) - }; - obj.unlock_gpuva(); - if p.is_null() { - None - } else { - // SAFETY: All the drm_gpuvm_bo objects in this GpuVm are always allocated by us as GpuVmBo. - let p = unsafe { crate::container_of!(p, GpuVmBo, bo) as *mut GpuVmBo }; - // SAFETY: We checked for NULL above, and the types ensure that - // this object was created by vm_bo_alloc_callback. - Some(unsafe { ARef::from_raw(NonNull::new_unchecked(p)) }) - } - } - - pub fn obtain_bo(&self, obj: &Object) -> Result>> { - obj.lock_gpuva(); - // SAFETY: drm_gem_object.gpuva.lock was just locked. - let p = unsafe { - bindings::drm_gpuvm_bo_obtain_locked( - self.gpuvm() as *mut _, - obj.as_raw() as *const _ as *mut _, - ) - }; - obj.unlock_gpuva(); - let p = from_err_ptr(p)?; - if p.is_null() { - Err(ENOMEM) - } else { - // SAFETY: Container invariant is guaranteed for GpuVmBo objects for this GpuVm. - let p = unsafe { crate::container_of!(p, GpuVmBo, bo) as *mut GpuVmBo }; - // SAFETY: We checked for NULL above, and the types ensure that - // this object was created by vm_bo_alloc_callback. - Ok(unsafe { ARef::from_raw(NonNull::new_unchecked(p)) }) - } - } - - pub fn bo_unmap(&self, ctx: &mut T::StepContext, bo: &GpuVmBo) -> Result { - let mut ctx = StepContext { ctx, gpuvm: self }; - // SAFETY: LockedGpuVm implies the right locks are held. - to_result(unsafe { - bindings::drm_gpuvm_bo_unmap(&bo.bo as *const _ as *mut _, &mut ctx as *mut _ as *mut _) - }) - } -} - -// SAFETY: DRM GpuVm objects are always reference counted and the get/put functions -// satisfy the requirements. -unsafe impl AlwaysRefCounted for GpuVm { - fn inc_ref(&self) { - // SAFETY: The drm_gpuvm_get function satisfies the requirements for inc_ref(). - unsafe { bindings::drm_gpuvm_get(&self.gpuvm as *const _ as *mut _) }; - } - - unsafe fn dec_ref(obj: NonNull) { - // SAFETY: The drm_gpuvm_put function satisfies the requirements for dec_ref(). - unsafe { bindings::drm_gpuvm_put(Opaque::cast_into(&(*obj.as_ptr()).gpuvm)) }; - } -} - -enum LockedObjects<'a, T: DriverGpuVm> { - Single(Option<&'a Object>), - Array(&'a [ARef>]), -} - -impl LockedObjects<'_, T> { - fn single(&self) -> Option<&Object> { - match self { - Self::Single(object) => *object, - Self::Array(_) => None, - } - } - - fn indexed(&self, index: usize) -> Option<&Object> { - match self { - Self::Array(objects) => objects.get(index).map(|object| &**object), - Self::Single(_) => None, - } - } -} - -/// Access to VM driver data under its shared reservation, without external BO locks. -pub struct GpuVmInnerGuard<'a, T: DriverGpuVm> { - gpuvm: &'a GpuVm, - // Reservation locks must be released by the task that acquired them. - _not_send: NotThreadSafe, -} - -impl GpuVmInnerGuard<'_, T> { - /// Run immediate-mode unmap steps without allocating an exec context. - /// - /// # Safety - /// The driver's unmap/remap callbacks must touch only VM metadata and - /// translations protected by this reservation. They must not access - /// external BO backing that requires its own reservation. Any split-node - /// storage must already be owned by `ctx`; callbacks must defer BO puts. - pub unsafe fn sm_unmap_inner( - &mut self, ctx: &mut T::StepContext, addr: u64, range: u64, - ) -> Result { - let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; - // SAFETY: this guard retains the VM and its shared reservation. - if unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { - return Err(EINVAL); - } - let mut ctx = StepContext { gpuvm: self.gpuvm, ctx }; - // SAFETY: caller supplies the callback contract above. Immediate-mode - // link/unlink acquire the separate GEM GPUVA-list mutex themselves. - to_result(unsafe { - bindings::drm_gpuvm_sm_unmap(vm, &mut ctx as *mut _ as *mut _, addr, range) - }) - } - - /// Remove every driver GPUVA in an immediate-mode VM, preserving its kernel cutout. - /// - /// `unmap` must remove the GPU translation for each complete GPUVA before - /// returning success. Only the VM's shared reservation is held. No external - /// BO reservations are acquired and no split nodes or operation lists are - /// allocated. The callback cannot retain the borrowed GPUVA. - /// - /// Call `GpuVm::bo_deferred_cleanup` after releasing this guard, including - /// when a callback fails after earlier mappings have already been removed. - pub fn unmap_all( - &mut self, - unmap: impl FnMut(&mut T, &GpuVa) -> Result, - ) -> Result { - self.unmap_matching(None, unmap) - } - - /// Allocation-free whole-node removal for one GEM in an immediate-mode VM. - /// Uses the same translation-removal and deferred-put contract as unmap_all. - /// This scans the VM list and is intended for low-memory cleanup fallback. - pub fn unmap_object( - &mut self, - object: &Object, - unmap: impl FnMut(&mut T, &GpuVa) -> Result, - ) -> Result { - self.unmap_matching(Some(object), unmap) - } - - fn unmap_matching( - &mut self, - object: Option<&Object>, - mut unmap: impl FnMut(&mut T, &GpuVa) -> Result, - ) -> Result { - let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; - // SAFETY: The guard owns this live VM's shared reservation. Immediate - // mode lets unlink take the GEM GPUVA mutex independently of its resv. - unsafe { - if (*vm).flags & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { - return Err(EINVAL); - } - let head = &raw mut (*vm).rb.list; - let mut entry = (*head).next; - while entry != head { - let va = crate::container_of!(entry, bindings::drm_gpuva, rb.entry) as *mut _; - // Save the successor before removing and freeing this GPUVA. - entry = (*entry).next; - // This node is embedded in the C GPUVM, not a driver GpuVa. - if core::ptr::eq(va, &raw const (*vm).kernel_alloc_node) { - continue; - } - if object.is_some_and(|object| (*va).gem.obj != object.as_raw()) { - continue; - } - let driver_va = crate::container_of!(va, GpuVa, gpuva); - unmap(&mut *self.gpuvm.inner.get(), &*driver_va)?; - let mut op = OpUnMap( - bindings::drm_gpuva_op_unmap { va, keep: false }, - PhantomData::, - ); - drop(op.unmap_and_unlink_va_defer()); - } - } - Ok(()) - } -} - -impl Deref for GpuVmInnerGuard<'_, T> { - type Target = T; - - fn deref(&self) -> &T { - // SAFETY: The shared reservation excludes every other inner-data guard, - // including LockedGpuVm and its UpdatingGpuVm callbacks. - unsafe { &*self.gpuvm.inner.get() } - } -} - -impl DerefMut for GpuVmInnerGuard<'_, T> { - fn deref_mut(&mut self) -> &mut T { - // SAFETY: As above, with exclusive access to this guard. - unsafe { &mut *self.gpuvm.inner.get() } - } -} - -impl Drop for GpuVmInnerGuard<'_, T> { - fn drop(&mut self) { - // SAFETY: This non-Send guard owns the reservation on this task, and - // its GPUVM borrow keeps r_obj alive until after the unlock. - unsafe { bindings::dma_resv_unlock((*(*self.gpuvm.gpuvm()).r_obj).resv) }; - } -} - -/// Logical binding retained independently of hardware page-table residency. -/// Owns a GEM reference and copied metadata, never a GpuVmBo reference. -/// Driver metadata cloning must not introduce a reservation-taking destructor. -pub struct MappingSnapshot { - pub object: ARef>, - pub addr: u64, - pub range: u64, - pub offset: u64, - pub flags: GpuVaFlags, - pub inner: T::GpuVa, -} - -/// Nonblocking reservation ownership for pressure scans of private objects. -pub struct PrivateGpuVmGuard<'a, T: DriverGpuVm> { - gpuvm: &'a GpuVm, - resv: *mut bindings::dma_resv, - // DMA reservation ownership must remain on the acquiring task. - _not_send: NotThreadSafe, -} - -impl PrivateGpuVmGuard<'_, T> { - /// Visit each private BO with linked logical mappings once, without allocating or retaining - /// references. Returning false stops the walk. Contended GEM lists are skipped. - /// - /// # Safety - /// The callback must not insert/remove logical mappings or modify either GPUVA list. This - /// reservation keeps their linked BO/GEM references alive for the complete callback. - /// The callback must not retain a borrowed BO or acquire its GEM GPUVA mutex recursively. - pub unsafe fn for_each_private_bo( - &mut self, - mut visit: impl FnMut(&mut Self, &GpuVmBo) -> Result, - ) -> Result { - let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; - if unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 { - return Err(EINVAL); - } - let head = unsafe { &raw const (*vm).rb.list }; - // SAFETY: The VM reservation protects this ordered mapping list. The callback does not - // mutate its links, and no borrowed or owned reference outlives the guard. - let mut node = unsafe { (*head).next }; - while node != head.cast_mut() { - let (next, raw) = unsafe { - ((*node).next, crate::container_of!(node, bindings::drm_gpuva, rb.entry)) - }; - let va = unsafe { &*raw }; - if !va.gem.obj.is_null() && !va.vm_bo.is_null() { - let object = unsafe { &*Object::::from_raw(va.gem.obj) }; - if !self.gpuvm.is_extobj(object) { - let lock = unsafe { &raw mut (*va.gem.obj).gpuva.lock }; - // Never block direct reclaim on a GEM-list owner. The canonical alias is - // the list's first-linked node, not its lowest address or current residency. - if unsafe { bindings::mutex_trylock(lock) } != 0 { - let canonical = unsafe { - (*va.vm_bo).list.gpuva.next == (&raw const va.gem.entry).cast_mut() - }; - unsafe { bindings::mutex_unlock(lock) }; - if canonical { - let bo = unsafe { - &*crate::container_of!(va.vm_bo, GpuVmBo, bo) - }; - if !visit(self, bo)? { break; } - } - } - } - } - node = next; - } - Ok(()) - } - - /// Mark and visit every logical alias of this private BO without a snapshot allocation. - /// GEM-list locking is nonblocking and held only while reading a cursor, never during the - /// callback. Failure (including contention) can leave a prefix marked INVALIDATED; callers - /// must retain backing unless this method succeeds for the entire association. - /// - /// # Safety - /// The callback must not insert/remove logical mappings, alter list links, or release the - /// BO's SG/backing lease. The BO must remain linked throughout the walk. Exclude new jobs and - /// binding edits for the whole invalidation and backing-release transaction. - pub unsafe fn invalidate_private_bo_mappings( - &mut self, - bo: &GpuVmBo, - mut visit: impl FnMut(&mut Self, u64, u64) -> Result, - ) -> Result { - let vm = self.gpuvm.gpuvm() as *mut bindings::drm_gpuvm; - if bo.bo.vm != vm || self.gpuvm.is_extobj(bo.object()) - || unsafe { (*vm).flags } & bindings::drm_gpuvm_flags_DRM_GPUVM_IMMEDIATE_MODE == 0 - { return Err(EINVAL); } - let head = &raw const bo.bo.list.gpuva; - let lock = unsafe { &raw mut (*bo.bo.obj).gpuva.lock }; - if unsafe { bindings::mutex_trylock(lock) } == 0 { return Err(EAGAIN); } - let mut node = unsafe { (*head).next }; - unsafe { bindings::mutex_unlock(lock) }; - while node != head.cast_mut() { - if unsafe { bindings::mutex_trylock(lock) } == 0 { return Err(EAGAIN); } - // SAFETY: The list mutex protects link reads; this reservation and the caller's - // no-unlink contract keep current/next and their association alive after unlocking. - let (next, raw) = unsafe { - ((*node).next, crate::container_of!(node, bindings::drm_gpuva, gem.entry)) - }; - unsafe { bindings::mutex_unlock(lock) }; - let va = unsafe { &mut *raw }; - if va.vm != vm || va.vm_bo != (&raw const bo.bo).cast_mut() - || va.va.range == 0 || va.va.addr.checked_add(va.va.range).is_none() - { return Err(EINVAL); } - // Flags are serialized by the VM reservation, independently of GEM's list mutex. - va.flags |= GpuVaFlags::INVALIDATED.as_raw(); - visit(self, va.va.addr, va.va.range)?; - node = next; - } - Ok(()) - } - - /// Retain the next private mapping containing all `required_flags`. - /// - /// `cursor` is an exclusive end address and advances past skipped mappings - /// as well. No collection is allocated; only the returned object is retained. - /// Callers must drop this reservation guard before releasing that reference - /// or acquiring backing. To cover a stable set across calls, separately - /// exclude logical binding edits (including retirement) for the whole walk. - pub fn next_private_mapping( - &self, - cursor: &mut u64, - required_flags: GpuVaFlags, - ) -> Result>> - where T::GpuVa: Clone { - while *cursor != u64::MAX { - // SAFETY: The VM reservation serializes mutation of its VA tree. - let raw = unsafe { - bindings::drm_gpuva_find_first( - self.gpuvm.gpuvm() as *mut _, *cursor, u64::MAX - *cursor, - ) - }; - if raw.is_null() { return Ok(None); } - // SAFETY: raw stays linked and alive while this guard is held. - let va = unsafe { &*raw }; - let end = va.va.addr.checked_add(va.va.range).ok_or(EINVAL)?; - if end <= *cursor { return Err(EINVAL); } - *cursor = end; - if !GpuVaFlags(va.flags).contains(required_flags) - || va.gem.obj.is_null() || va.vm_bo.is_null() { - continue; - } - // SAFETY: The linked VA retains this driver's object. - let obj = unsafe { &*Object::::from_raw(va.gem.obj) }; - if self.gpuvm.is_extobj(obj) { continue; } - // SAFETY: Driver-owned mappings have this allocation layout. The - // special kernel cutout has no GEM object and was skipped above. - let typed = unsafe { &*crate::container_of!(raw, GpuVa, gpuva) }; - return Ok(Some(MappingSnapshot { - object: obj.into(), addr: va.va.addr, - range: va.va.range, offset: va.gem.offset, - flags: GpuVaFlags(va.flags), inner: typed.inner.clone(), - })); - } - Ok(None) - } - - -} - -impl Deref for PrivateGpuVmGuard<'_, T> { - type Target = T; - fn deref(&self) -> &T { - // SAFETY: The reservation serializes all access to the VM's inner data. - unsafe { &*self.gpuvm.inner.get() } - } -} -impl DerefMut for PrivateGpuVmGuard<'_, T> { - fn deref_mut(&mut self) -> &mut T { - // SAFETY: This guard has exclusive reservation ownership. - unsafe { &mut *self.gpuvm.inner.get() } - } -} -impl Drop for PrivateGpuVmGuard<'_, T> { - fn drop(&mut self) { - // SAFETY: try_lock_private acquired exactly this reservation. - unsafe { bindings::dma_resv_unlock(self.resv) }; - } -} - -pub struct LockedGpuVm<'a, 'b, T: DriverGpuVm> { - gpuvm: &'a GpuVm, - vm_exec: KBox, - objects: LockedObjects<'b, T>, -} - -impl LockedGpuVm<'_, '_, T> { - /// Mark the exact retained logical mapping after page-table restoration. - pub fn set_invalidated(&mut self, addr: u64, range: u64, invalidated: bool) -> Result { - // SAFETY: Both guard types own the VM reservation. - unsafe { self.gpuvm.set_invalidated_locked(addr, range, invalidated) } - } - - pub fn find_bo(&mut self) -> Option>> { - let obj = self.objects.single()?; - // SAFETY: LockedGpuVm implies the right locks are held. - let p = unsafe { - bindings::drm_gpuvm_bo_find( - self.gpuvm.gpuvm() as *mut _, - obj.as_raw() as *const _ as *mut _, - ) - }; - if p.is_null() { - None - } else { - // SAFETY: All the drm_gpuvm_bo objects in this GpuVm are always allocated by us as GpuVmBo. - let p = unsafe { crate::container_of!(p, GpuVmBo, bo) as *mut GpuVmBo }; - // SAFETY: We checked for NULL above, and the types ensure that - // this object was created by vm_bo_alloc_callback. - Some(unsafe { ARef::from_raw(NonNull::new_unchecked(p)) }) - } - } - - pub fn obtain_bo(&mut self) -> Result>> { - let obj = self.objects.single().ok_or(EINVAL)?; - // SAFETY: LockedGpuVm implies the right locks are held. - let p = unsafe { - bindings::drm_gpuvm_bo_obtain_locked( - self.gpuvm.gpuvm() as *mut _, - obj.as_raw() as *const _ as *mut _, - ) - }; - let p = from_err_ptr(p)?; - if p.is_null() { - Err(ENOMEM) - } else { - // SAFETY: Container invariant is guaranteed for GpuVmBo objects for this GpuVm. - let p = unsafe { crate::container_of!(p, GpuVmBo, bo) as *mut GpuVmBo }; - // SAFETY: We checked for NULL above, and the types ensure that - // this object was created by vm_bo_alloc_callback. - Ok(unsafe { ARef::from_raw(NonNull::new_unchecked(p)) }) - } - } - - pub fn sm_map( - &mut self, - ctx: &mut T::StepContext, - req_addr: u64, - req_range: u64, - req_offset: u64, - req_gem_range: u32, - flags: GpuVaFlags, - ) -> Result { - let obj = self.objects.single().ok_or(EINVAL)?.as_raw(); - self.sm_map_raw( - obj, - ctx, - req_addr, - req_range, - req_offset, - req_gem_range, - flags, - ) - } - - /// Map with an object acquired by [`GpuVm::exec_lock_array`]. - pub fn sm_map_indexed( - &mut self, - object_index: usize, - ctx: &mut T::StepContext, - req_addr: u64, - req_range: u64, - req_offset: u64, - req_gem_range: u32, - flags: GpuVaFlags, - ) -> Result { - let obj = self - .objects - .indexed(object_index) - .ok_or(EINVAL)? - .as_raw(); - self.sm_map_raw( - obj, - ctx, - req_addr, - req_range, - req_offset, - req_gem_range, - flags, - ) - } - - fn sm_map_raw( - &mut self, - obj: *mut bindings::drm_gem_object, - ctx: &mut T::StepContext, - req_addr: u64, - req_range: u64, - req_offset: u64, - req_gem_range: u32, - flags: GpuVaFlags, - ) -> Result { - let mut ctx = StepContext { - ctx, - gpuvm: self.gpuvm, - }; - - let req = bindings::drm_gpuvm_map_req { - map: bindings::drm_gpuva_op_map { - va: bindings::drm_gpuva_op_map__bindgen_ty_1 { - addr: req_addr, - range: req_range, - }, - gem: bindings::drm_gpuva_op_map__bindgen_ty_2 { - offset: req_offset, - range: req_gem_range, - obj, - }, - flags: flags.as_raw(), - }, - }; - - // SAFETY: The object pointer came from the reference set retained by - // this guard, and LockedGpuVm implies the required locks are held. - to_result(unsafe { - bindings::drm_gpuvm_sm_map( - self.gpuvm.gpuvm() as *mut _, - &mut ctx as *mut _ as *mut _, - &raw const req, - ) - }) - } - - pub fn sm_unmap(&mut self, ctx: &mut T::StepContext, req_addr: u64, req_range: u64) -> Result { - let mut ctx = StepContext { - ctx, - gpuvm: self.gpuvm, - }; - // SAFETY: LockedGpuVm implies the right locks are held. - to_result(unsafe { - bindings::drm_gpuvm_sm_unmap( - self.gpuvm.gpuvm() as *mut _, - &mut ctx as *mut _ as *mut _, - req_addr, - req_range, - ) - }) - } - - pub fn bo_unmap(&mut self, ctx: &mut T::StepContext, bo: &GpuVmBo) -> Result { - let mut ctx = StepContext { - ctx, - gpuvm: self.gpuvm, - }; - // SAFETY: LockedGpuVm implies the right locks are held. - to_result(unsafe { - bindings::drm_gpuvm_bo_unmap(&bo.bo as *const _ as *mut _, &mut ctx as *mut _ as *mut _) - }) - } -} - -impl Deref for LockedGpuVm<'_, '_, T> { - type Target = T; - - fn deref(&self) -> &T { - // SAFETY: The existence of this LockedGpuVm implies the lock is held, - // so this is the only reference - unsafe { &*self.gpuvm.inner.get() } - } -} - -impl DerefMut for LockedGpuVm<'_, '_, T> { - fn deref_mut(&mut self) -> &mut T { - // SAFETY: The existence of this UpdatingGpuVm implies the lock is held, - // so this is the only reference - unsafe { &mut *self.gpuvm.inner.get() } - } -} - -impl Drop for LockedGpuVm<'_, '_, T> { - fn drop(&mut self) { - // SAFETY: We hold the lock, so it's safe to unlock - unsafe { - bindings::drm_gpuvm_exec_unlock(&mut *self.vm_exec); - } - } -} - -pub struct UpdatingGpuVm<'a, T: DriverGpuVm>(&'a GpuVm); - -impl UpdatingGpuVm<'_, T> {} - -impl Deref for UpdatingGpuVm<'_, T> { - type Target = T; - - fn deref(&self) -> &T { - // SAFETY: The existence of this UpdatingGpuVm implies the lock is held, - // so this is the only reference - unsafe { &*self.0.inner.get() } - } -} - -impl DerefMut for UpdatingGpuVm<'_, T> { - fn deref_mut(&mut self) -> &mut T { - // SAFETY: The existence of this UpdatingGpuVm implies the lock is held, - // so this is the only reference - unsafe { &mut *self.0.inner.get() } - } -} - -// SAFETY: All our trait methods take locks -unsafe impl Sync for GpuVm {} -// SAFETY: All our trait methods take locks -unsafe impl Send for GpuVm {} - -// SAFETY: All our trait methods take locks -unsafe impl Sync for GpuVmBo {} -// SAFETY: All our trait methods take locks -unsafe impl Send for GpuVmBo {} diff --git a/rust/kernel/drm_neo/ioctl.rs b/rust/kernel/drm_neo/ioctl.rs deleted file mode 100644 index a7c7bf586e87c6..00000000000000 --- a/rust/kernel/drm_neo/ioctl.rs +++ /dev/null @@ -1,167 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM IOCTL definitions. -//! -//! C header: [`include/drm_neo/drm_ioctl.h`](srctree/include/drm_neo/drm_ioctl.h) - -use crate::ioctl; - -const BASE: u32 = uapi::DRM_IOCTL_BASE as u32; - -/// Construct a DRM ioctl number with no argument. -#[allow(non_snake_case)] -#[inline(always)] -pub const fn IO(nr: u32) -> u32 { - ioctl::_IO(BASE, nr) -} - -/// Construct a DRM ioctl number with a read-only argument. -#[allow(non_snake_case)] -#[inline(always)] -pub const fn IOR(nr: u32) -> u32 { - ioctl::_IOR::(BASE, nr) -} - -/// Construct a DRM ioctl number with a write-only argument. -#[allow(non_snake_case)] -#[inline(always)] -pub const fn IOW(nr: u32) -> u32 { - ioctl::_IOW::(BASE, nr) -} - -/// Construct a DRM ioctl number with a read-write argument. -#[allow(non_snake_case)] -#[inline(always)] -pub const fn IOWR(nr: u32) -> u32 { - ioctl::_IOWR::(BASE, nr) -} - -/// Descriptor type for DRM ioctls. Use the `declare_drm_neo_ioctls!{}` macro to construct them. -pub type DrmIoctlDescriptor = bindings::drm_ioctl_desc; - -/// This is for ioctl which are used for rendering, and require that the file descriptor is either -/// for a render node, or if it’s a legacy/primary node, then it must be authenticated. -pub const AUTH: u32 = bindings::drm_ioctl_flags_DRM_AUTH; - -/// This must be set for any ioctl which can change the modeset or display state. Userspace must -/// call the ioctl through a primary node, while it is the active master. -/// -/// Note that read-only modeset ioctl can also be called by unauthenticated clients, or when a -/// master is not the currently active one. -pub const MASTER: u32 = bindings::drm_ioctl_flags_DRM_MASTER; - -/// Anything that could potentially wreak a master file descriptor needs to have this flag set. -/// -/// Current that’s only for the SETMASTER and DROPMASTER ioctl, which e.g. logind can call to -/// force a non-behaving master (display compositor) into compliance. -/// -/// This is equivalent to callers with the SYSADMIN capability. -pub const ROOT_ONLY: u32 = bindings::drm_ioctl_flags_DRM_ROOT_ONLY; - -/// This is used for all ioctl needed for rendering only, for drivers which support render nodes. -/// This should be all new render drivers, and hence it should be always set for any ioctl with -/// `AUTH` set. Note though that read-only query ioctl might have this set, but have not set -/// DRM_AUTH because they do not require authentication. -pub const RENDER_ALLOW: u32 = bindings::drm_ioctl_flags_DRM_RENDER_ALLOW; - -/// Internal structures used by the `declare_drm_neo_ioctls!{}` macro. Do not use directly. -#[doc(hidden)] -pub mod internal { - pub use bindings::drm_device; - pub use bindings::drm_file; - pub use bindings::drm_ioctl_desc; -} - -/// Declare the DRM ioctls for a driver. -/// -/// Each entry in the list should have the form: -/// -/// `(ioctl_number, argument_type, flags, user_callback),` -/// -/// `argument_type` is the type name within the `bindings` crate. -/// `user_callback` should have the following prototype: -/// -/// ```ignore -/// fn foo(device: &kernel::drm_neo::Device, -/// data: &mut uapi::argument_type, -/// file: &kernel::drm_neo::File, -/// ) -> Result -/// ``` -/// where `Self` is the drm_neo::drv::Driver implementation these ioctls are being declared within. -/// -/// # Examples -/// -/// ```ignore -/// kernel::declare_drm_neo_ioctls! { -/// (FOO_GET_PARAM, drm_foo_get_param, ioctl::RENDER_ALLOW, my_get_param_handler), -/// } -/// ``` -/// -#[macro_export] -macro_rules! declare_drm_neo_ioctls { - ( $(($cmd:ident, $struct:ident, $flags:expr, $func:expr)),* $(,)? ) => { - const IOCTLS: &'static [$crate::drm_neo::ioctl::DrmIoctlDescriptor] = { - use $crate::uapi::*; - const _:() = { - let i: u32 = $crate::uapi::DRM_COMMAND_BASE; - // Assert that all the IOCTLs are in the right order and there are no gaps, - // and that the size of the specified type is correct. - $( - let cmd: u32 = $crate::macros::concat_idents!(DRM_IOCTL_, $cmd); - ::core::assert!(i == $crate::ioctl::_IOC_NR(cmd)); - ::core::assert!(core::mem::size_of::<$crate::uapi::$struct>() == - $crate::ioctl::_IOC_SIZE(cmd)); - let i: u32 = i + 1; - )* - }; - - let ioctls = &[$( - $crate::drm_neo::ioctl::internal::drm_ioctl_desc { - cmd: $crate::macros::concat_idents!(DRM_IOCTL_, $cmd) as u32, - func: { - #[allow(non_snake_case)] - unsafe extern "C" fn $cmd( - raw_dev: *mut $crate::drm_neo::ioctl::internal::drm_device, - raw_data: *mut ::core::ffi::c_void, - raw_file: *mut $crate::drm_neo::ioctl::internal::drm_file, - ) -> core::ffi::c_int { - // SAFETY: - // - The DRM core ensures the device lives while callbacks are being - // called. - // - The DRM device must have been registered when we're called through - // an IOCTL. - // - // FIXME: Currently there is nothing enforcing that the types of the - // dev/file match the current driver these ioctls are being declared - // for, and it's not clear how to enforce this within the type system. - let dev = $crate::drm_neo::device::Device::from_raw(raw_dev); - // SAFETY: The ioctl argument has size `_IOC_SIZE(cmd)`, which we - // asserted above matches the size of this type, and all bit patterns of - // UAPI structs must be valid. - // The `ioctl` argument is exclusively owned by the handler - // and guaranteed by the C implementation (`drm_ioctl()`) to remain - // valid for the entire lifetime of the reference taken here. - // There is no concurrent access or aliasing; no other references - // to this object exist during this call. - let data = unsafe { &mut *(raw_data.cast::<$crate::uapi::$struct>()) }; - // SAFETY: This is just the DRM file structure - let file = unsafe { $crate::drm_neo::File::from_raw(raw_file) }; - - match $func(dev, data, file) { - Err(e) => e.to_errno(), - Ok(i) => i.try_into() - .unwrap_or($crate::error::code::ERANGE.to_errno()), - } - } - Some($cmd) - }, - flags: $flags, - name: $crate::str::as_char_ptr_in_const_context( - $crate::c_str!(::core::stringify!($cmd)), - ), - } - ),*]; - ioctls - }; - }; -} diff --git a/rust/kernel/drm_neo/mm.rs b/rust/kernel/drm_neo/mm.rs deleted file mode 100644 index daa05abbad1580..00000000000000 --- a/rust/kernel/drm_neo/mm.rs +++ /dev/null @@ -1,344 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM MM range allocator -//! -//! C header: [`include/drm_neo/drm_mm.h`](../../../../include/drm_neo/drm_mm.h) - -use crate::{ - alloc::flags::*, - bindings, - error::{code::EINVAL, to_result, Result}, - sync::{new_mutex, Arc, Mutex, UniqueArc}, - types::Opaque, -}; - -use crate::init::InPlaceInit; -use crate::prelude::KBox; - -use core::{ - marker::{PhantomData, PhantomPinned}, - ops::{Deref, Range}, - pin::Pin, -}; - -/// Type alias representing a DRM MM node. -pub type Node = Pin>>; - -/// Trait which must be implemented by the inner allocator state type provided by the user. -pub trait AllocInner { - /// Notification that a node was dropped from the allocator. - fn drop_object(&mut self, _start: u64, _size: u64, _color: usize, _object: &mut T) {} -} - -impl AllocInner for () {} - -/// Wrapper type for a `struct drm_mm` plus user AllocInner object. -/// -/// # Invariants -/// The `drm_mm` struct is valid and initialized. -struct MmInner, T>(Opaque, A, PhantomData); - -/// Represents a single allocated node in the MM allocator -pub struct NodeData, T> { - node: bindings::drm_mm_node, - mm: Arc>>, - valid: bool, - /// A drm_mm_node needs to be pinned because nodes reference each other in a linked list. - _pin: PhantomPinned, - inner: T, -} - -// SAFETY: Allocator ops take the mutex, and there are no mutable actions on the node. -unsafe impl, T: Send> Send for NodeData {} -// SAFETY: Allocator ops take the mutex, and there are no mutable actions on the node. -unsafe impl, T: Sync> Sync for NodeData {} - -/// Available MM node insertion modes -#[repr(u32)] -pub enum InsertMode { - /// Search for the smallest hole (within the search range) that fits the desired node. - /// - /// Allocates the node from the bottom of the found hole. - Best = bindings::drm_mm_insert_mode_DRM_MM_INSERT_BEST, - - /// Search for the lowest hole (address closest to 0, within the search range) that fits the - /// desired node. - /// - /// Allocates the node from the bottom of the found hole. - Low = bindings::drm_mm_insert_mode_DRM_MM_INSERT_LOW, - - /// Search for the highest hole (address closest to U64_MAX, within the search range) that fits - /// the desired node. - /// - /// Allocates the node from the top of the found hole. The specified alignment for the node is - /// applied to the base of the node (`Node.start()`). - High = bindings::drm_mm_insert_mode_DRM_MM_INSERT_HIGH, - - /// Search for the most recently evicted hole (within the search range) that fits the desired - /// node. This is appropriate for use immediately after performing an eviction scan and removing - /// the selected nodes to form a hole. - /// - /// Allocates the node from the bottom of the found hole. - Evict = bindings::drm_mm_insert_mode_DRM_MM_INSERT_EVICT, -} - -/// A clonable, interlocked reference to the allocator state. -/// -/// This is useful to perform actions on the user-supplied `AllocInner` type given just a Node, -/// without immediately taking the lock. -#[derive(Clone)] -pub struct InnerRef, T>(Arc>>); - -impl, T> InnerRef { - /// Operate on the user `AllocInner` implementation, taking the lock. - pub fn with(&self, cb: impl FnOnce(&mut A) -> RetVal) -> RetVal { - let mut l = self.0.lock(); - cb(&mut l.1) - } -} - -impl, T> NodeData { - /// Returns the color of the node (an opaque value) - pub fn color(&self) -> usize { - self.node.color as usize - } - - /// Returns the start address of the node - pub fn start(&self) -> u64 { - self.node.start - } - - /// Returns the size of the node in bytes - pub fn size(&self) -> u64 { - self.node.size - } - - /// Operate on the user `AllocInner` implementation associated with this node's allocator. - pub fn with_inner(&self, cb: impl FnOnce(&mut A) -> RetVal) -> RetVal { - let mut l = self.mm.lock(); - cb(&mut l.1) - } - - /// Return a clonable, detached reference to the allocator inner data. - pub fn alloc_ref(&self) -> InnerRef { - InnerRef(self.mm.clone()) - } - - /// Return a mutable reference to the inner data. - pub fn inner_mut(self: Pin<&mut Self>) -> &mut T { - // SAFETY: This is okay because inner is not structural - unsafe { &mut self.get_unchecked_mut().inner } - } -} - -impl, T> Deref for NodeData { - type Target = T; - - fn deref(&self) -> &Self::Target { - &self.inner - } -} - -impl, T> Drop for NodeData { - fn drop(&mut self) { - if self.valid { - let mut guard = self.mm.lock(); - - // Inform the user allocator that a node is being dropped. - guard - .1 - .drop_object(self.start(), self.size(), self.color(), &mut self.inner); - // SAFETY: The MM lock is still taken, so we can safely remove the node. - unsafe { bindings::drm_mm_remove_node(&mut self.node) }; - } - } -} - -/// An instance of a DRM MM range allocator. -pub struct Allocator, T> { - mm: Arc>>, - _p: PhantomData, -} - -impl, T> Allocator { - /// Return the first allocator reservation intersecting an address range. - /// Unlike a page-table walk, this includes unmapped guard space retained - /// inside a node. The interval query and its result are protected by the - /// allocator mutex; no probe node is inserted into the live VA space. - pub fn first_overlap(&self, range: Range) -> Result>> { - if range.start >= range.end { - return Err(EINVAL); - } - let guard = self.mm.lock(); - // The DRM interval query requires both endpoints to lie inside its - // allocator. head_node is the end sentinel; its wrapping size is the - // negated allocator span. - let head = unsafe { &(*guard.0.get()).head_node }; - let allocator_start = head.start.wrapping_add(head.size); - if range.start < allocator_start || range.end > head.start { - return Err(EINVAL); - } - // SAFETY: The allocator is initialized for the lifetime of `self`. - // Its mutex prevents node removal while the returned pointer is read. - // drm_mm returns its end sentinel when no interval intersects. - let node = unsafe { - bindings::__drm_mm_interval_first( - guard.0.get(), range.start, range.end - 1, - ) - }; - // SAFETY: drm_mm_interval_first returns a valid node or the allocator - // end sentinel, both of which remain live while `guard` is held. - let node = unsafe { node.as_ref().ok_or(EINVAL)? }; - if node.start >= range.end { - return Ok(None); - } - Ok(Some(node.start..node.start.saturating_add(node.size))) - } - - /// Create a new range allocator for the given start and size range of addresses. - /// - /// The user may optionally provide an inner object representing allocator state, which will - /// be protected by the same lock. If not required, `()` can be used. - #[track_caller] - pub fn new(start: u64, size: u64, inner: A) -> Result> { - // SAFETY: We call `Mutex::init_lock` below. - let mm = UniqueArc::pin_init( - new_mutex!(MmInner(Opaque::uninit(), inner, PhantomData)), - GFP_KERNEL, - )?; - - // SAFETY: The Opaque instance provides a valid pointer, and it is initialized after - // this call. - unsafe { - bindings::drm_mm_init(mm.lock().0.get(), start, size); - } - - Ok(Allocator { - mm: mm.into(), - _p: PhantomData, - }) - } - - /// Insert a new node into the allocator of a given size. - /// - /// `node` is the user `T` type data to store into the node. - pub fn insert_node(&mut self, node: T, size: u64) -> Result> { - self.insert_node_generic(node, size, 0, 0, InsertMode::Best) - } - - /// Insert a new node into the allocator of a given size, with configurable alignment, - /// color, and insertion mode. - /// - /// `node` is the user `T` type data to store into the node. - pub fn insert_node_generic( - &mut self, - node: T, - size: u64, - alignment: u64, - color: usize, - mode: InsertMode, - ) -> Result> { - self.insert_node_in_range(node, size, alignment, color, 0, u64::MAX, mode) - } - - /// Insert a new node into the allocator of a given size, with configurable alignment, - /// color, insertion mode, and sub-range to allocate from. - /// - /// `node` is the user `T` type data to store into the node. - #[allow(clippy::too_many_arguments)] - pub fn insert_node_in_range( - &mut self, - node: T, - size: u64, - alignment: u64, - color: usize, - start: u64, - end: u64, - mode: InsertMode, - ) -> Result> { - let mut mm_node = KBox::new( - NodeData { - // SAFETY: This C struct should be zero-initialized. - node: unsafe { core::mem::zeroed() }, - valid: false, - inner: node, - mm: self.mm.clone(), - _pin: PhantomPinned, - }, - GFP_KERNEL, - )?; - - let guard = self.mm.lock(); - // SAFETY: We hold the lock and all pointers are valid. - to_result(unsafe { - bindings::drm_mm_insert_node_in_range( - guard.0.get(), - &mut mm_node.node, - size, - alignment, - color, - start, - end, - mode as u32, - ) - })?; - - mm_node.valid = true; - - Ok(Pin::from(mm_node)) - } - - /// Insert a node into the allocator at a fixed start address. - /// - /// `node` is the user `T` type data to store into the node. - pub fn reserve_node( - &mut self, - node: T, - start: u64, - size: u64, - color: usize, - ) -> Result> { - let mut mm_node = KBox::new( - NodeData { - // SAFETY: This C struct should be zero-initialized. - node: unsafe { core::mem::zeroed() }, - valid: false, - inner: node, - mm: self.mm.clone(), - _pin: PhantomPinned, - }, - GFP_KERNEL, - )?; - - mm_node.node.start = start; - mm_node.node.size = size; - mm_node.node.color = color as crate::ffi::c_ulong; - - let guard = self.mm.lock(); - // SAFETY: We hold the lock and all pointers are valid. - to_result(unsafe { bindings::drm_mm_reserve_node(guard.0.get(), &mut mm_node.node) })?; - - mm_node.valid = true; - - Ok(Pin::from(mm_node)) - } - - /// Operate on the inner user type `A`, taking the allocator lock - pub fn with_inner(&self, cb: impl FnOnce(&mut A) -> RetVal) -> RetVal { - let mut guard = self.mm.lock(); - cb(&mut guard.1) - } -} - -impl, T> Drop for MmInner { - fn drop(&mut self) { - // SAFETY: If the MmInner is dropped then all nodes are gone (since they hold references), - // so it is safe to tear down the allocator. - unsafe { - bindings::drm_mm_takedown(self.0.get()); - } - } -} - -// SAFETY: MmInner is safely Send if the AllocInner user type is Send. -unsafe impl, T> Send for MmInner {} diff --git a/rust/kernel/drm_neo/mod.rs b/rust/kernel/drm_neo/mod.rs deleted file mode 100644 index 882841415aa414..00000000000000 --- a/rust/kernel/drm_neo/mod.rs +++ /dev/null @@ -1,24 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM subsystem abstractions. - -pub mod device; -pub mod driver; -pub mod file; -pub mod gem; -#[cfg(CONFIG_DRM_GPUVM = "y")] -pub mod gpuvm; -pub mod ioctl; -pub mod mm; -pub mod sched; -pub mod syncobj; - -pub use self::device::Device; -pub use self::driver::Driver; -pub use self::driver::DriverInfo; -pub use self::driver::Registration; -pub use self::file::File; - -pub(crate) mod private { - pub trait Sealed {} -} diff --git a/rust/kernel/drm_neo/sched.rs b/rust/kernel/drm_neo/sched.rs deleted file mode 100644 index 910385f16cb556..00000000000000 --- a/rust/kernel/drm_neo/sched.rs +++ /dev/null @@ -1,549 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM Scheduler -//! -//! C header: [`include/drm_neo/gpu_scheduler.h`](../../../../include/drm_neo/gpu_scheduler.h) - -use crate::{ - bindings, device, - dma_fence::*, - error::{to_result, Result}, - prelude::*, - sync::{Arc, UniqueArc}, - time::{self, msecs_to_jiffies}, -}; -use core::marker::PhantomData; -use core::mem::MaybeUninit; -use core::ops::{Deref, DerefMut}; -use core::ptr::{addr_of, addr_of_mut}; - -/// Scheduler status after timeout recovery -#[repr(u32)] -pub enum Status { - /// Device recovered from the timeout and can execute jobs again - Nominal = bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_RESET, - /// Device is no longer available - NoDevice = bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_ENODEV, - /// Reinsert the job and rearm its timeout without resetting the scheduler. - NoHang = bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_NO_HANG, -} - -/// Scheduler priorities -#[repr(u32)] -pub enum Priority { - /// Low userspace priority - Low = bindings::drm_sched_priority_DRM_SCHED_PRIORITY_LOW, - /// Normal userspace priority - Normal = bindings::drm_sched_priority_DRM_SCHED_PRIORITY_NORMAL, - /// High userspace priority - High = bindings::drm_sched_priority_DRM_SCHED_PRIORITY_HIGH, - /// Kernel priority (highest) - Kernel = bindings::drm_sched_priority_DRM_SCHED_PRIORITY_KERNEL, -} - -/// Trait to be implemented by driver job objects. -pub trait JobImpl: Sized { - /// State owned exclusively by the timeout worker. It must not borrow the - /// mutable submission state: timeout can run before `run` returns. - type TimeoutData; - - /// Snapshot or independently retain timeout state before scheduler ownership. - fn timeout_data(&self) -> Self::TimeoutData; - - /// Module implementing the callbacks. Its caller must retain the module - /// through entity destruction; the core pins detached kill-job callbacks. - const MODULE: Option<&'static crate::ThisModule> = None; - - /// Handle a scoped timeout without resetting the C scheduler. - /// - /// The backend must settle/quarantine its exact hardware work, or retain - /// that owner until an in-progress host transaction can classify it. It - /// must not invoke drm_sched_stop/start or replay scheduler jobs. The core - /// reinserts the detached guilty job through NO_HANG, preserving every - /// parent callback and the backend's actual success/error fence result. - const MANAGED_TIMEOUT_RECOVERY: bool = false; - - /// Reconsider a timeout before stopping the scheduler or settling fences. - /// Returning true asks the core to reinsert the retained job and rearm. - /// The backend must not stop/start the scheduler from this hook. - fn false_timeout(_data: &mut Self::TimeoutData, _finished: bool) -> bool { false } - - - /// Called when the scheduler is considering scheduling this job next, to get another Fence - /// for this job to block on. Once it returns None, run() may be called. - fn prepare(_job: &mut Self) -> Option { - None // Equivalent to NULL function pointer - } - - /// Called to execute the job once all of the dependencies have been resolved. This may be - /// called multiple times, if timed_out() has happened and drm_sched_job_recovery() decides - /// to try it again. - fn run(job: &mut Self) -> Result>; - - /// Called when a job has taken too long to execute, to trigger GPU recovery. - /// - /// This method is called in a workqueue context. - fn timed_out(data: &mut Self::TimeoutData, finished: bool) -> Status; - - /// Called for remaining jobs in drm_sched_fini() to ensure the job's fences - /// get signalled before the scheduler is torn down. - fn cancel(job: &mut Self); -} - -unsafe extern "C" fn prepare_job_cb( - sched_job: *mut bindings::drm_sched_job, - _s_entity: *mut bindings::drm_sched_entity, -) -> *mut bindings::dma_fence { - // SAFETY: All of our jobs are Job. - let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - - // SAFETY: prepare/run are serialized by submit work. Only borrow inner; - // the timeout worker independently owns timeout_data and C owns job. - match T::prepare(unsafe { &mut *addr_of_mut!((*p).inner) }) { - None => core::ptr::null_mut(), - Some(fence) => fence.into_raw(), - } -} - -unsafe extern "C" fn run_job_cb( - sched_job: *mut bindings::drm_sched_job, -) -> *mut bindings::dma_fence { - // SAFETY: All of our jobs are Job. - let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - - // SAFETY: submit work owns inner until return, even while timeout work - // accesses the disjoint timeout_data field. Do not borrow the whole Job. - match T::run(unsafe { &mut *addr_of_mut!((*p).inner) }) { - Err(e) => e.to_ptr(), - Ok(None) => core::ptr::null_mut(), - Ok(Some(fence)) => fence.into_raw(), - } -} - -unsafe extern "C" fn timedout_job_cb( - sched_job: *mut bindings::drm_sched_job, -) -> bindings::drm_gpu_sched_stat { - // SAFETY: All of our jobs are Job. - let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - - // The C timeout worker owns this job outside pending_list until return. - // Free work cannot release it even if its parent signals concurrently. - let finished = unsafe { addr_of_mut!((*(*sched_job).s_fence).finished) }; - let is_finished = || unsafe { bindings::dma_fence_get_status(finished) != 0 }; - if T::MANAGED_TIMEOUT_RECOVERY { - if is_finished() { - return bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_NO_HANG; - } - } - // SAFETY: the C core retains the detached job and serializes its timeout - // worker. This field is disjoint from inner, which run may still mutate. - let data = unsafe { &mut *addr_of_mut!((*p).timeout_data) }; - if T::false_timeout(data, is_finished()) { - return bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_NO_HANG; - } - let status = T::timed_out(data, is_finished()); - if T::MANAGED_TIMEOUT_RECOVERY { - // This is scoped fence handling, not C reset recovery. stop would - // detach and discard healthy parents; start would cancel those jobs. - // NO_HANG reinserts the exact job and queues free work if it completed - // during the callback. It does not change the parent's error or replay - // work. A deferred exact owner remains eligible for its real callback. - bindings::drm_gpu_sched_stat_DRM_GPU_SCHED_STAT_NO_HANG - } else { - status as bindings::drm_gpu_sched_stat - } -} - -unsafe extern "C" fn free_job_cb(sched_job: *mut bindings::drm_sched_job) { - // SAFETY: All of our jobs are Job. - let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - - // Fault-only ownership witness: this callback is where the core returns - // ownership of an errored job. A signalled fence alone is not that proof. - // SAFETY: A job reaching free_job has a live scheduler fence; its error - // was fixed before signal and the core's completion callback has finished. - if T::MANAGED_TIMEOUT_RECOVERY - && unsafe { (*(*sched_job).s_fence).finished.error } == ETIMEDOUT.to_errno() - { - // Use numeric identity: the fmt adapter borrows arguments, and - // Pointer formatting of that reference can print a stack address. - pr_info!("DRM Rust scheduler: released timed-out job={:#x} context={} seqno={}\n", - sched_job as usize, - unsafe { (*(*sched_job).s_fence).finished.context }, - unsafe { (*(*sched_job).s_fence).finished.seqno }); - } - // Convert the job back to a Box and drop it - // SAFETY: All of our Jobs are created inside a box. - unsafe { drop(KBox::from_raw(p)) }; -} - -unsafe extern "C" fn cancel_job_cb(sched_job: *mut bindings::drm_sched_job) { - // SAFETY: All of our jobs are Job. - let p = unsafe { crate::container_of!(sched_job, Job, job) as *mut Job }; - - // fini has stopped/synchronized submission and timeout workers. A parent - // callback can still run on the device completion thread. The scheduler - // fence retains the parent reference while we remove its callback under - // the parent's lock; that also waits for an already-running callback. - let s_fence = unsafe { (*sched_job).s_fence }; - let parent = unsafe { (*s_fence).parent }; - let removed_parent_callback = if !parent.is_null() { - // SAFETY: run_job installed (or attempted to install on an already - // signaled parent) this callback before fini synchronized submit work. - unsafe { bindings::dma_fence_remove_callback(parent, addr_of_mut!((*sched_job).cb)) } - } else { false }; - let finish_here = removed_parent_callback || (parent.is_null() - && unsafe { bindings::dma_fence_get_status(addr_of_mut!((*s_fence).finished)) == 0 }); - // Null/error run_job calls job_done before submit work returns, so its - // finished fence is already signaled. NULL+unfinished can instead survive - // a legacy backend's prior reset stop: C stop dropped the parent and its - // credits, but not its score. Do not subtract those credits twice. - - // SAFETY: cancellation cannot race a scheduler callback using this job - // after the removal/parent-lock synchronization above. The backend must - // retain any still-hardware-owned resources independently of Job storage. - T::cancel(unsafe { &mut *addr_of_mut!((*p).inner) }); - - if finish_here { - // Match drm_sched_job_done's accounting exactly once, only when its - // callback did not do it. Submission/free work is stopped, so fini - // itself unlinks and frees this job; no workqueue rearm is appropriate. - let sched = unsafe { (*sched_job).sched }; - unsafe { - if removed_parent_callback { - bindings::atomic_sub((*sched_job).credits as i32, - addr_of_mut!((*sched).credit_count)); - } - bindings::atomic_sub(1, (*sched).score); - } - let fence = unsafe { Fence::get_raw(addr_of_mut!((*s_fence).finished)) }; - fence.set_error(ECANCELED); - let _ = fence.signal(); - } - // Do not clear/put parent here: s_fence still owns that reference and its - // normal release path drops it. A completed callback's result is retained. -} - -/// A DRM scheduler job. -pub struct Job { - job: bindings::drm_sched_job, - inner: T, - timeout_data: T::TimeoutData, - // Entity destruction can leave kill_jobs_work queued on the system - // workqueue or waiting for a dependency. That work still dereferences - // job->sched->ops after drm_sched_fini has stopped the scheduler workers. - // Retain only the allocation: retaining SchedulerOwner here would keep - // fini from running while pending jobs still need its cancellation. - _scheduler: Arc>, -} - -impl Job { - /// Returns whether an armed job has finished, successfully or not. - pub fn is_finished(&self) -> bool { - if self.job.sched.is_null() { - return false; - } - // SAFETY: An armed job has an initialized scheduler fence, retained - // for its lifetime, including while the timeout worker owns the job. - unsafe { bindings::dma_fence_get_status(addr_of_mut!((*self.job.s_fence).finished)) != 0 } - } -} - -impl Deref for Job { - type Target = T; - - fn deref(&self) -> &Self::Target { - &self.inner - } -} - -impl DerefMut for Job { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.inner - } -} - -impl Drop for Job { - fn drop(&mut self) { - // SAFETY: At this point the job has either been submitted and this is being called from - // `free_job_cb` above, or it hasn't and it is safe to call `drm_sched_job_cleanup`. - unsafe { bindings::drm_sched_job_cleanup(&mut self.job) }; - } -} - -/// A pending DRM scheduler job (not yet armed) -pub struct PendingJob<'a, T: JobImpl>(KBox>, PhantomData<&'a T>); - -impl<'a, T: JobImpl> PendingJob<'a, T> { - /// Add a fence as a dependency to the job - pub fn add_dependency(&mut self, fence: Fence) -> Result { - // SAFETY: C call with correct arguments - to_result(unsafe { - bindings::drm_sched_job_add_dependency(&mut self.0.job, fence.into_raw()) - }) - } - - /// Arm the job to make it ready for execution - pub fn arm(mut self) -> ArmedJob<'a, T> { - // SAFETY: C call with correct arguments - unsafe { bindings::drm_sched_job_arm(&mut self.0.job) }; - ArmedJob(self.0, PhantomData) - } -} - -impl<'a, T: JobImpl> Deref for PendingJob<'a, T> { - type Target = Job; - - fn deref(&self) -> &Self::Target { - &self.0 - } -} - -impl<'a, T: JobImpl> DerefMut for PendingJob<'a, T> { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.0 - } -} - -/// An armed DRM scheduler job (not yet submitted) -pub struct ArmedJob<'a, T: JobImpl>(KBox>, PhantomData<&'a T>); - -impl<'a, T: JobImpl> ArmedJob<'a, T> { - /// Returns the job fences - pub fn fences(&mut self) -> JobFences<'_> { - // SAFETY: s_fence is always a valid drm_sched_fence pointer - JobFences(unsafe { &mut *self.0.job.s_fence }) - } - - /// Push the job for execution into the scheduler - pub fn push(self) { - // After this point, the job is submitted and owned by the scheduler - let ptr = match self { - ArmedJob(job, _) => KBox::>::into_raw(job), - }; - - // SAFETY: We are passing in ownership of a valid Box raw pointer. - unsafe { bindings::drm_sched_entity_push_job(addr_of_mut!((*ptr).job)) }; - } -} -impl<'a, T: JobImpl> Deref for ArmedJob<'a, T> { - type Target = Job; - - fn deref(&self) -> &Self::Target { - &self.0 - } -} - -impl<'a, T: JobImpl> DerefMut for ArmedJob<'a, T> { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.0 - } -} - -/// Reference to the bundle of fences attached to a DRM scheduler job -pub struct JobFences<'a>(&'a mut bindings::drm_sched_fence); - -impl<'a> JobFences<'a> { - /// Returns a new reference to the job scheduled fence. - pub fn scheduled(&mut self) -> Fence { - // SAFETY: self.0.scheduled is always a valid fence - unsafe { Fence::get_raw(&mut self.0.scheduled) } - } - - /// Returns a new reference to the job finished fence. - pub fn finished(&mut self) -> Fence { - // SAFETY: self.0.finished is always a valid fence - unsafe { Fence::get_raw(&mut self.0.finished) } - } -} - -struct EntityInner { - entity: bindings::drm_sched_entity, - // TODO: Allow users to share guilty flag between entities - sched: Arc>, - guilty: bindings::atomic_t, - _p: PhantomData, -} - -impl Drop for EntityInner { - fn drop(&mut self) { - // SAFETY: The EntityInner is initialized. Jobs removed from the entity - // may still await kill_jobs_work; each retains the scheduler allocation. - unsafe { bindings::drm_sched_entity_destroy(&mut self.entity) }; - } -} - -// SAFETY: TODO -unsafe impl Sync for EntityInner {} -// SAFETY: TODO -unsafe impl Send for EntityInner {} - -/// A DRM scheduler entity. -pub struct Entity(Pin>>); - -impl Entity { - /// Create a new scheduler entity. - pub fn new(sched: &Scheduler, priority: Priority) -> Result { - let mut entity: KBox>> = - KBox::new_uninit(GFP_KERNEL | __GFP_ZERO)?; - - let mut sched_ptr = &sched.0.allocation.sched as *const _ as *mut _; - - // SAFETY: The Box is allocated above and valid. - unsafe { - bindings::drm_sched_entity_init( - addr_of_mut!((*entity.as_mut_ptr()).entity), - priority as _, - &mut sched_ptr, - 1, - addr_of_mut!((*entity.as_mut_ptr()).guilty), - ) - }; - - // SAFETY: The Box is allocated above and valid. - unsafe { addr_of_mut!((*entity.as_mut_ptr()).sched).write(sched.0.clone()) }; - - // SAFETY: entity is now initialized. - Ok(Self(Pin::from(unsafe { entity.assume_init() }))) - } - - /// Create a new job on this entity. - /// - /// The entity must outlive the pending job until it transitions into the submitted state, - /// after which the scheduler owns it. Since jobs must be submitted in creation order, - /// this requires a mutable reference to the entity, ensuring that only one new job can be - /// in flight at once. - pub fn new_job(&mut self, credits: u32, inner: T) -> Result> { - let mut job: KBox>> = Box::new_uninit(GFP_KERNEL | __GFP_ZERO)?; - let timeout_data = inner.timeout_data(); - - // SAFETY: We hold a reference to the entity (which is a valid pointer), - // and the job object was just allocated above. - to_result(unsafe { - bindings::drm_sched_job_init( - addr_of_mut!((*job.as_mut_ptr()).job), - &self.0.as_ref().get_ref().entity as *const _ as *mut _, - credits, - core::ptr::null_mut(), - 0, - ) - })?; - - // SAFETY: The Box pointer is valid, and this initializes the inner member. - unsafe { addr_of_mut!((*job.as_mut_ptr()).inner).write(inner) }; - // SAFETY: initialize the independent timeout worker's owned state. - unsafe { addr_of_mut!((*job.as_mut_ptr()).timeout_data).write(timeout_data) }; - - // SAFETY: The job allocation is valid and this initializes its last - // field. Arc::clone is infallible and the entity still owns the live - // scheduler, so every successfully initialized job retains its C - // callback target before it can be armed or submitted. - unsafe { - addr_of_mut!((*job.as_mut_ptr())._scheduler) - .write(self.0.as_ref().get_ref().sched.allocation.clone()) - }; - - // SAFETY: All fields of the Job are now initialized. - Ok(PendingJob(unsafe { job.assume_init() }, PhantomData)) - } -} - -/// DRM scheduler allocation, retained by live owners and deferred job callbacks. -pub struct SchedulerInner { - sched: bindings::drm_gpu_scheduler, - _p: PhantomData, -} - -// Only public scheduler/entity handles keep this owner alive. Jobs retain -// SchedulerInner instead, so outstanding work cannot prevent finalization. -struct SchedulerOwner { - allocation: Arc>, -} - -impl Drop for SchedulerOwner { - fn drop(&mut self) { - // SAFETY: fini stops submission, synchronizes the timeout worker and - // cancels/frees remaining jobs through our synchronized cancel callback. - // The C contract explicitly prohibits drm_sched_stop before fini. - // Deferred entity-kill callbacks are on a different workqueue and may - // survive this call. Their job guards retain allocation through cleanup. - unsafe { - bindings::drm_sched_fini(addr_of!(self.allocation.sched).cast_mut()) - }; - } -} - -// SAFETY: TODO -unsafe impl Sync for SchedulerInner {} -// SAFETY: TODO -unsafe impl Send for SchedulerInner {} - -/// A DRM Scheduler -pub struct Scheduler(Arc>); - -impl Scheduler { - const OPS: bindings::drm_sched_backend_ops = bindings::drm_sched_backend_ops { - prepare_job: Some(prepare_job_cb::), - run_job: Some(run_job_cb::), - timedout_job: Some(timedout_job_cb::), - free_job: Some(free_job_cb::), - cancel_job: Some(cancel_job_cb::), - // A terminal backend result must not orphan the detached Job. The - // core restores ownership after all Rust callback borrows end. - retain_job_on_enodev: true, - owner: match T::MODULE { - Some(module) => module.as_ptr(), - None => core::ptr::null_mut(), - }, - }; - /// Creates a new DRM Scheduler object - // TODO: Shared timeout workqueues & scores - pub fn new( - device: &device::Device, - num_rqs: u32, - credit_limit: u32, - hang_limit: u32, - timeout_ms: time::Msecs, - name: &'static CStr, - ) -> Result> { - let mut sched: UniqueArc>> = - UniqueArc::new_uninit(GFP_KERNEL)?; - // Allocate the owner before starting C workers: no fallible allocation - // may abandon an initialized scheduler without running drm_sched_fini. - let owner: UniqueArc>> = - UniqueArc::new_uninit(GFP_KERNEL)?; - - // SAFETY: zero sched->sched_rq as drm_sched_init() uses it to exit early withoput initialisation - // TODO: allocate sched zzeroed instead - unsafe { - (*sched.as_mut_ptr()).sched.sched_rq = core::ptr::null_mut(); - }; - - let init_ops = bindings::drm_sched_init_args { - ops: &Self::OPS, - submit_wq: core::ptr::null_mut(), - timeout_wq: core::ptr::null_mut(), - num_rqs, - credit_limit, - hang_limit, - timeout: msecs_to_jiffies(timeout_ms).try_into()?, - score: core::ptr::null_mut(), - name: name.as_char_ptr(), - dev: device.as_raw(), - }; - - // SAFETY: The drm_sched pointer is valid and pinned as it was just allocated above. - // `device` is valid by its type invarants - to_result(unsafe { - bindings::drm_sched_init( - addr_of_mut!((*sched.as_mut_ptr()).sched), - addr_of!(init_ops), - ) - })?; - - // SAFETY: C initialized sched and PhantomData has no stored state. - // No fallible operation follows successful initialization. - let allocation = unsafe { sched.assume_init() }.into(); - Ok(Scheduler(owner.write(SchedulerOwner { allocation }).into())) - } -} diff --git a/rust/kernel/drm_neo/syncobj.rs b/rust/kernel/drm_neo/syncobj.rs deleted file mode 100644 index 1aff064f43ecc5..00000000000000 --- a/rust/kernel/drm_neo/syncobj.rs +++ /dev/null @@ -1,83 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR MIT - -//! DRM Sync Objects -//! -//! C header: [`include/drm_neo/drm_syncobj.h`](../../../../include/drm_neo/drm_syncobj.h) - -use crate::{bindings, dma_fence::*, drm_neo, error::Result, prelude::*}; - -/// A DRM Sync Object -/// -/// # Invariants -/// ptr is a valid pointer to a drm_syncobj and we own a reference to it. -pub struct SyncObj { - ptr: *mut bindings::drm_syncobj, -} - -impl SyncObj { - /// Looks up a sync object by its handle for a given `File`. - pub fn lookup_handle( - file: &drm_neo::File, - handle: u32, - ) -> Result { - // SAFETY: The arguments are all valid per the type invariants. - let ptr = unsafe { bindings::drm_syncobj_find(file.as_raw() as *mut _, handle) }; - - if ptr.is_null() { - Err(ENOENT) - } else { - Ok(SyncObj { ptr }) - } - } - - /// Returns the DMA fence associated with this sync object, if any. - pub fn fence_get(&self) -> Option { - // SAFETY: self.ptr is always valid - let fence = unsafe { bindings::drm_syncobj_fence_get(self.ptr) }; - if fence.is_null() { - None - } else { - // SAFETY: The pointer is non-NULL and drm_syncobj_fence_get acquired an - // additional reference. - Some(unsafe { Fence::from_raw(fence) }) - } - } - - /// Replaces the DMA fence with a new one, or removes it if fence is None. - pub fn replace_fence(&self, fence: Option<&Fence>) { - // SAFETY: All arguments should be valid per the respective type invariants. - unsafe { - bindings::drm_syncobj_replace_fence( - self.ptr, - fence.map_or(core::ptr::null_mut(), |a| a.raw()), - ) - }; - } - - /// Adds a new timeline point to the syncobj. - pub fn add_point(&self, chain: FenceChain, fence: &Fence, point: u64) { - // SAFETY: All arguments should be valid per the respective type invariants. - // This takes over the FenceChain ownership. - unsafe { bindings::drm_syncobj_add_point(self.ptr, chain.into_raw(), fence.raw(), point) }; - } -} - -impl Drop for SyncObj { - fn drop(&mut self) { - // SAFETY: We own a reference to this syncobj. - unsafe { bindings::drm_syncobj_put(self.ptr) }; - } -} - -impl Clone for SyncObj { - fn clone(&self) -> Self { - // SAFETY: `ptr` is valid per the type invariant and we own a reference to it. - unsafe { bindings::drm_syncobj_get(self.ptr) }; - SyncObj { ptr: self.ptr } - } -} - -// SAFETY: drm_syncobj operations are internally locked. -unsafe impl Sync for SyncObj {} -// SAFETY: drm_syncobj operations are internally locked. -unsafe impl Send for SyncObj {} diff --git a/rust/kernel/iosys_map.rs b/rust/kernel/iosys_map.rs index 3fad1c1aebbbc8..629f40de3dbf2c 100644 --- a/rust/kernel/iosys_map.rs +++ b/rust/kernel/iosys_map.rs @@ -536,7 +536,7 @@ pub mod tests { use super::*; /// A helper struct for managed IoSysMapRef structs which point to a [`Vec`]. - pub struct VecIoSysMap { + pub struct VecIoSysMap { map: RawIoSysMap, vec: KVec, } diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs index 20b12c1497d988..a40585a222baab 100644 --- a/rust/kernel/lib.rs +++ b/rust/kernel/lib.rs @@ -86,8 +86,6 @@ pub mod dma_fence; pub mod driver; #[cfg(CONFIG_DRM = "y")] pub mod drm; -#[cfg(CONFIG_DRM_ASAHI_NEO)] -pub mod drm_neo; pub mod error; pub mod faux; #[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)] diff --git a/rust/uapi/uapi_helper.h b/rust/uapi/uapi_helper.h index 3f1bf3477224dc..512ac0aea08fde 100644 --- a/rust/uapi/uapi_helper.h +++ b/rust/uapi/uapi_helper.h @@ -8,7 +8,6 @@ #include #include -#include #include #include #include diff --git a/security/tomoyo/common.h b/security/tomoyo/common.h index d098cf8aae6109..b859f099ae9eac 100644 --- a/security/tomoyo/common.h +++ b/security/tomoyo/common.h @@ -575,8 +575,8 @@ struct tomoyo_mini_stat { /* Structure for dumping argv[] and envp[] of "struct linux_binprm". */ struct tomoyo_page_dump { - struct page *page; /* Previously dumped page. */ - char *data; /* Contents of "page". Size is PAGE_SIZE. */ + unsigned long pos; /* First cached virtual byte, or ULONG_MAX. */ + char *data; /* Buffer for one native page of virtual bytes. */ }; /* Structure for attribute checks in addition to pathname checks. */ diff --git a/security/tomoyo/domain.c b/security/tomoyo/domain.c index eeaad421f15baa..2ff8d783f54cc2 100644 --- a/security/tomoyo/domain.c +++ b/security/tomoyo/domain.c @@ -906,45 +906,37 @@ int tomoyo_find_next_domain(struct linux_binprm *bprm) bool tomoyo_dump_page(struct linux_binprm *bprm, unsigned long pos, struct tomoyo_page_dump *dump) { - struct page *page; -#ifdef CONFIG_MMU - int ret; -#endif + const unsigned int offset = offset_in_page(pos); + const unsigned int length = PAGE_SIZE - offset; /* dump->data is released by tomoyo_find_next_domain(). */ if (!dump->data) { dump->data = kzalloc(PAGE_SIZE, GFP_NOFS); if (!dump->data) return false; + dump->pos = ULONG_MAX; } - /* Same with get_arg_page(bprm, pos, 0) in fs/exec.c */ + if (pos >= dump->pos && (pos & PAGE_MASK) == (dump->pos & PAGE_MASK)) + return true; + /* A failed partial copy must not leave the previous cache valid. */ + dump->pos = ULONG_MAX; #ifdef CONFIG_MMU /* - * This is called at execve() time in order to dig around - * in the argv/environment of the new process - * (represented by bprm). + * Inspect the nascent MM's virtual bytes. Smaller userspace mappings + * can use several independently retained fragments within this range. */ - mmap_read_lock(bprm->mm); - ret = get_user_pages_remote(bprm->mm, pos, 1, - FOLL_FORCE, &page, NULL); - mmap_read_unlock(bprm->mm); - if (ret <= 0) + if (access_remote_vm(bprm->mm, pos, dump->data + offset, length, + FOLL_FORCE) != length) return false; #else - page = bprm->page[pos / PAGE_SIZE]; -#endif - if (page != dump->page) { - const unsigned int offset = pos % PAGE_SIZE; + { + struct page *page = bprm->page[pos / PAGE_SIZE]; char *kaddr = kmap_local_page(page); - dump->page = page; - memcpy(dump->data + offset, kaddr + offset, - PAGE_SIZE - offset); + memcpy(dump->data + offset, kaddr + offset, length); kunmap_local(kaddr); } - /* Same with put_arg_page(page) in fs/exec.c */ -#ifdef CONFIG_MMU - put_page(page); #endif + dump->pos = pos; return true; } diff --git a/sound/soc/apple/t8140_aop_audio.rs b/sound/soc/apple/t8140_aop_audio.rs index 069656e5e6d98b..b63c6c8baa0849 100644 --- a/sound/soc/apple/t8140_aop_audio.rs +++ b/sound/soc/apple/t8140_aop_audio.rs @@ -52,6 +52,7 @@ use kernel::{ }, error::from_err_ptr, module_platform_driver, + new_mutex, new_spinlock, of, platform, @@ -73,6 +74,7 @@ use kernel::{ }, Arc, ArcBorrow, + Mutex, SpinLock, // }, types::{ForeignOwnable, Opaque}, @@ -432,6 +434,14 @@ struct StreamState { // sets it and close clears it under the same lock. unsafe impl Send for StreamState {} +/// The holders of the low-power microphone's `runn` state: its own PCM +/// while prepared, and every service startup that raises the fabric. +struct LpaiRun { + users: u32, + /// `runn` is the last state the firmware confirmed. + running: bool, +} + #[pin_data] struct SndSocT8140AopData { dev: ARef, @@ -440,7 +450,7 @@ struct SndSocT8140AopData { ring: Arc, ring_bytes: Atomic, sequence: Atomic, - /// lpai is in `runn`. + /// The low-power microphone PCM holds a reference on `lpai_run`. powered: AtomicFlag, /// A low-power microphone report arrived since the last run request. reported: AtomicFlag, @@ -478,6 +488,10 @@ struct SndSocT8140AopData { hpai_start_work: Work, #[pin] stream: SpinLock, + /// Every lpai power request goes through here, see + /// [`SndSocT8140AopData::lpai_run_get`]. + #[pin] + lpai_run: Mutex, } const SPKR_START_WORK: u64 = 0; @@ -680,6 +694,39 @@ impl SndSocT8140AopData { Ok(()) } + /// Takes a reference on the low-power microphone's `runn` state, + /// requesting it for the first holder. + /// + /// The PCM and the fabric raises share the stream, and every power + /// request is checked against the state the firmware reports back: a + /// raise that returns lpai to `idle` while another service starts up + /// makes that one read `idle` after requesting `runn`, and idles a + /// capture the PCM just started. So they count their holds here and + /// only the first and last holder change the state. + fn lpai_run_get(&self) -> Result<()> { + let mut run = self.lpai_run.lock(); + if !run.running { + self.reported.store(false, Relaxed); + self.set_power(POWER_STATE_RUN)?; + run.running = true; + } + run.users += 1; + Ok(()) + } + + /// Drops a reference taken by [`Self::lpai_run_get`], returning lpai to + /// `idle` after the last holder. If that fails, the next holder finds + /// it still running. + fn lpai_run_put(&self) -> Result<()> { + let mut run = self.lpai_run.lock(); + run.users -= 1; + if run.users == 0 && run.running { + self.set_power(POWER_STATE_IDLE)?; + run.running = false; + } + Ok(()) + } + // ---- the back-end services ---- /// Property 202 with the service's sequence and a 24 MHz timestamp; @@ -785,14 +832,10 @@ impl SndSocT8140AopData { /// does not do that (measured: `audio_p` stays gated for the whole leaf /// retry window), only a running stream does, after which a service in /// `pw0 ` keeps it up only while its stream runs. The low-power - /// microphone is attached for the firmware's lifetime, so run it briefly - /// around `f` when it is idle, once its first report has arrived. + /// microphone is attached for the firmware's lifetime, so hold it in + /// `runn` around `f`, once its first report has arrived. fn with_fabric_raised(&self, f: impl FnOnce() -> Result) -> Result { - if self.powered.load(Relaxed) { - return f(); - } - self.reported.store(false, Relaxed); - self.set_power(POWER_STATE_RUN)?; + self.lpai_run_get()?; let mut polls = 0; while !self.reported.load(Relaxed) && polls < FABRIC_RAISE_POLLS { kernel::time::delay::fsleep(kernel::time::Delta::from_millis(FABRIC_RAISE_POLL_MS)); @@ -802,8 +845,12 @@ impl SndSocT8140AopData { dev_warn!(self.dev, "lpai: no report while raising the fabric\n"); } let ret = f(); - if let Err(e) = self.set_power(POWER_STATE_IDLE) { - dev_warn!(self.dev, "lpai: unable to idle after raising the fabric: {:?}\n", e); + if let Err(e) = self.lpai_run_put() { + dev_warn!( + self.dev, + "lpai: unable to idle after raising the fabric: {:?}\n", + e + ); } ret } @@ -1192,12 +1239,11 @@ unsafe extern "C" fn lpai_pcm_close(substream: *mut bindings::snd_pcm_substream) let mut st = data.stream.lock(); st.substream = ptr::null_mut(); } - if data.powered.load(Relaxed) { - if let Err(e) = data.set_power(POWER_STATE_IDLE) { + if data.powered.xchg(false, Relaxed) { + if let Err(e) = data.lpai_run_put() { dev_err!(data.dev, "unable to return lpai to idle\n"); return e.to_errno(); } - data.powered.store(false, Relaxed); } 0 } @@ -1221,7 +1267,7 @@ unsafe extern "C" fn lpai_pcm_prepare(substream: *mut bindings::snd_pcm_substrea data.hw_ptr_frames.store(0, Relaxed); } if !data.powered.load(Relaxed) { - if let Err(e) = data.set_power(POWER_STATE_RUN) { + if let Err(e) = data.lpai_run_get() { dev_err!(data.dev, "unable to run lpai\n"); return e.to_errno(); } @@ -1256,6 +1302,24 @@ unsafe extern "C" fn lpai_pcm_trigger( } } +/// Called after a stop, without the ALSA stream lock, before the buffer is +/// reallocated or freed (hw_params, hw_free) or the counters are reset +/// (prepare). The trigger only clears the run flag, so a report that saw +/// the stream running may still be copying into the buffer: it holds the +/// stream lock for the whole copy, so taking that lock waits for it, and +/// every report after it sees the flag clear. +/// +/// # Safety +/// +/// Called by ALSA with a live substream of this card's PCM, whose private +/// data is the driver's `Arc`. +unsafe extern "C" fn lpai_pcm_sync_stop(substream: *mut bindings::snd_pcm_substream) -> i32 { + // SAFETY: ALSA calls this op with a live substream of the LPAI PCM. + let data = unsafe { pcm_data(substream) }; + drop(data.stream.lock()); + 0 +} + /// # Safety /// /// Called by ALSA with a live substream of this card's PCM, whose private @@ -1374,6 +1438,19 @@ unsafe extern "C" fn hpai_pcm_trigger( 0 } +/// The trigger stops the DMA asynchronously: wait for the channel to stop +/// and for its completion callbacks to finish before ALSA reuses or frees +/// the buffer. +/// +/// # Safety +/// +/// Called by ALSA with a live substream of this card's PCM. +unsafe extern "C" fn hpai_pcm_sync_stop(substream: *mut bindings::snd_pcm_substream) -> i32 { + // SAFETY: the substream is live and its dmaengine runtime was set up by + // `snd_dmaengine_pcm_open` in `hpai_pcm_open`. + unsafe { bindings::snd_dmaengine_pcm_sync_stop(substream) } +} + /// # Safety /// /// Called once by ALSA when the PCM goes away, with the `Arc` leaked into @@ -1834,8 +1911,8 @@ fn f32_bits_of_q31(v: i32) -> u32 { } let sign = if v < 0 { 1u32 << 31 } else { 0 }; let m = v.unsigned_abs(); - let p = 31 - m.leading_zeros(); // position of the leading one, 0..=31 - // value = m * 2^-31 = 1.f * 2^(p - 31) + // The position of the leading one, 0..=31: |v| / 2^31 = 1.f * 2^(p - 31). + let p = 31 - m.leading_zeros(); let exp = (p as i32 - 31 + 127) as u32; let mant = if p >= 23 { (m >> (p - 23)) & 0x7f_ffff @@ -1953,6 +2030,28 @@ unsafe extern "C" fn asoc_pcm_trigger( unsafe { bindings::snd_dmaengine_pcm_trigger(substream, cmd) } } +/// Front-end stop synchronization, as for the microphone array: the trigger +/// stops the DMA asynchronously, so wait for the channel to stop and for its +/// completion callbacks to finish before ALSA reuses or frees the buffer. +/// +/// # Safety +/// +/// Called by ASoC on this driver's registered component, with live +/// arguments belonging to one of its links. +unsafe extern "C" fn asoc_pcm_sync_stop( + _component: *mut bindings::snd_soc_component, + substream: *mut bindings::snd_pcm_substream, +) -> i32 { + // SAFETY: ASoC hands its ops a live substream whose private data is the + // `snd_soc_pcm_runtime` of one of this component's links. + if unsafe { rtd_is_be(substream_rtd(substream)) } { + return 0; + } + // SAFETY: the substream is live and its dmaengine runtime was set up by + // `snd_dmaengine_pcm_open` in `asoc_pcm_open`. + unsafe { bindings::snd_dmaengine_pcm_sync_stop(substream) } +} + /// # Safety /// /// Called by ASoC on this driver's registered component, with live @@ -2389,6 +2488,7 @@ impl AsocContext { hw_params: Some(asoc_pcm_hw_params), prepare: Some(asoc_pcm_prepare), trigger: Some(asoc_pcm_trigger), + sync_stop: Some(asoc_pcm_sync_stop), pointer: Some(asoc_pcm_pointer), copy: Some(asoc_pcm_copy), ..Default::default() @@ -2432,7 +2532,7 @@ impl SndSocT8140AopDriver { ioctl: None, hw_params: None, hw_free: None, - sync_stop: None, + sync_stop: Some(lpai_pcm_sync_stop), get_time_info: None, fill_silence: None, copy: None, @@ -2450,7 +2550,7 @@ impl SndSocT8140AopDriver { ioctl: None, hw_params: Some(hpai_pcm_hw_params), hw_free: None, - sync_stop: None, + sync_stop: Some(hpai_pcm_sync_stop), get_time_info: None, fill_silence: None, copy: None, @@ -2620,8 +2720,10 @@ impl SndSocT8140AopDriver { // SAFETY: the card is freed once, while the binding remains live. unsafe { bindings::snd_card_free(self.card) }; } - if self.data.powered.load(Relaxed) && self.data.set_power(POWER_STATE_IDLE).is_ok() { - self.data.powered.store(false, Relaxed); + // Every stream is closed now; retry an idle request that failed then. + let mut run = self.data.lpai_run.lock(); + if run.running && self.data.set_power(POWER_STATE_IDLE).is_ok() { + run.running = false; } } } @@ -2750,6 +2852,10 @@ impl platform::Driver for SndSocT8140AopDriver { period_acc: 0, buffer_bytes: 0, }), + lpai_run <- new_mutex!(LpaiRun { + users: 0, + running: false, + }), }), GFP_KERNEL, )?; @@ -2767,3 +2873,90 @@ module_platform_driver! { description: "Apple T8140 AOP audio: microphones, headphone jack and speakers", license: "Dual MIT/GPL", } + +/// The speaker ceiling and the copy-path conversion are the whole digital +/// protection of the J700 speakers below the daemon: check them where they +/// are built, without the hardware. +#[kunit_tests(snd_soc_t8140_aop_audio_speaker)] +mod tests { + use super::*; + + /// macaudio's level for the J700 while no daemon holds the lock: + /// MAX98360A_DB(safe_vol = -20). + const LOCKED_VOLUME: usize = 87; + /// 10^(0.5 / 20) in parts per 10^9. + const HALF_DB_PPB: u128 = 1_059_253_725; + /// 0.5 as a float32: the amplifier's full scale on the wire. + const F32_HALF: u32 = 0x3f00_0000; + + #[test] + fn the_top_of_the_volume_range_is_the_amplifier_full_scale() { + assert_eq!( + SPKR_VOLUME_Q31[SPKR_VOLUME_MAX as usize] as i64, + SPKR_CEILING_Q31 + ); + assert_eq!(SPKR_CEILING_Q31, 1 << 30); + assert_eq!(f32_bits_of_q31(SPKR_CEILING_Q31 as i32), F32_HALF); + } + + #[test] + fn the_volume_table_falls_in_half_db_steps_to_mute() { + assert_eq!(SPKR_VOLUME_Q31[0], 0); + for i in 2..=SPKR_VOLUME_MAX as usize { + let (lo, hi) = (SPKR_VOLUME_Q31[i - 1] as u128, SPKR_VOLUME_Q31[i] as u128); + assert!(hi > lo); + // hi / lo == 10^(0.5/20) to within the table's rounding + let expect = lo * HALF_DB_PPB / 1_000_000_000; + assert!(hi.abs_diff(expect) <= 2); + } + } + + #[test] + fn the_locked_level_is_20_db_below_full_scale() { + let locked = SPKR_VOLUME_Q31[LOCKED_VOLUME] as i64; + // 10 * locked == ceiling to within the table's rounding (0.1 ppm) + assert!((10 * locked - SPKR_CEILING_Q31).abs() <= 10); + } + + #[test] + fn no_sample_at_any_volume_exceeds_the_full_scale() { + for &gain in SPKR_VOLUME_Q31.iter() { + for sample in [ + i32::MAX, + i32::MIN, + i32::MAX - 1, + i32::MIN + 1, + 1 << 30, + -(1 << 30), + 1, + -1, + 0, + ] { + let bits = spkr_f32_bits(sample, gain); + assert!(bits & 0x7fff_ffff <= F32_HALF); + } + } + // the extremes at full volume: just under and exactly at full scale + assert_eq!( + spkr_f32_bits(i32::MAX, SPKR_CEILING_Q31 as u32), + 0x3eff_ffff + ); + assert_eq!( + spkr_f32_bits(i32::MIN, SPKR_CEILING_Q31 as u32), + 0x8000_0000 | F32_HALF + ); + // and 20 dB down at the locked level + let locked = spkr_f32_bits(i32::MIN, SPKR_VOLUME_Q31[LOCKED_VOLUME]); + // 0.05 as a float32 is 0x3d4c_cccd; the conversion truncates + assert!(locked & 0x7fff_ffff <= 0x3d4c_cccd && locked & 0x7fff_ffff > 0x3d4c_cccd - 64); + } + + #[test] + fn the_float_conversion_is_exact_on_powers_of_two() { + assert_eq!(f32_bits_of_q31(0), 0); + assert_eq!(f32_bits_of_q31(1 << 29), 0x3e80_0000); // 0.25 + assert_eq!(f32_bits_of_q31(-(1 << 30)), 0xbf00_0000); // -0.5 + assert_eq!(f32_bits_of_q31(1), 0x3000_0000); // 2^-31 + assert_eq!(f32_bits_of_q31(i32::MIN), 0xbf80_0000); // -1.0 + } +} diff --git a/tools/asahi/neo-auxiliary-tests.py b/tools/asahi/neo-auxiliary-tests.py index 276e4960e7490e..899fd9d8a78dc2 100644 --- a/tools/asahi/neo-auxiliary-tests.py +++ b/tools/asahi/neo-auxiliary-tests.py @@ -9,7 +9,7 @@ import subprocess ROOT = Path(__file__).resolve().parents[2] -PATH = 'drivers/gpu/drm/asahi-neo/g17/queue/render/memory.rs' +PATH = 'drivers/gpu/drm/asahi/g17/queue/render/memory.rs' PREFIX = r''' #![allow(dead_code)] use std::ops::{Deref,Range}; @@ -82,7 +82,7 @@ def main(): structure = item(source, 'pub(super) struct Generation').replace('pub(super)', '') constructor = item(source[source.index('impl Generation'):], ' fn new(') addresses = item(source, ' pub(super) fn addresses').replace('pub(super)', '') - storage=(ROOT/'drivers/gpu/drm/asahi-neo/g17/fw/render/storage.rs').read_text() + storage=(ROOT/'drivers/gpu/drm/asahi/g17/fw/render/storage.rs').read_text() constants=[] for name in ['STATE_SIZE','DEFLAKE','TA_STATUS','AUXILIARY','AUXILIARY_SIZE']: line=next(x for x in storage.splitlines() if 'const '+name+':' in x) diff --git a/tools/asahi/neo-registration-tests.py b/tools/asahi/neo-registration-tests.py deleted file mode 100644 index e5e276640d8f4e..00000000000000 --- a/tools/asahi/neo-registration-tests.py +++ /dev/null @@ -1,121 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: GPL-2.0-only OR MIT -"""Exercise display registration, SoC admission and registration failure cleanup.""" -import argparse -from pathlib import Path -import re -import subprocess -import tempfile - -ROOT = Path(__file__).resolve().parents[2] - - -def function(source, name): - match = re.search(r'^(?:static )?(?:int|void) __init ' + name + r'\(void\)', source, re.M) - assert match, name - end = source.index('\n}', match.end()) + 2 - return source[match.start():end] - - -def driver_name(source, symbol): - start = source.index('static struct platform_driver ' + symbol + ' =') - end = source.index('\n};', start) - return re.search(r'\.name\s*=\s*"([^"]+)"', source[start:end])[1] - - -def check(root): - legacy = root / 'drivers/gpu/drm/apple' - neo = root / 'drivers/gpu/drm/apple-neo' - files = {str(p): p.read_text() for d in (legacy, neo) - for p in (d/'audio.c', d/'dcp.c', d/'apple_drv.c')} - code = r''' -#include -#include -#include -#include -#include -#define __init -#define IS_ENABLED(x) x -#define CONFIG_DRM_APPLE_AUDIO AUDIO -#define CONFIG_DRM_APPLE_NEO_AUDIO AUDIO -struct platform_driver { const char *name; }; -static struct platform_driver drivers[6]; -static struct platform_driver *registered[6]; -static unsigned count, attempts, unregisters; -static int fail_at = -1; -static bool t8140, firmware_only; -static bool of_machine_is_compatible(const char *name) { - assert(!strcmp(name,"apple,t8140")); return t8140; -} -static bool drm_firmware_drivers_only(void) { return firmware_only; } -static int platform_driver_register(struct platform_driver *driver) { - if ((int)attempts++ == fail_at) return -EIO; - for (unsigned i=0;iname,registered[i]->name)) return -EBUSY; - assert(count<6);registered[count++]=driver;return 0; -} -static void platform_driver_unregister(struct platform_driver *driver) { - for (unsigned i=0;i 1 ++ { ++ explicit_align = Some(layout.align); ++ } ++ } + } else if is_union && !forward_decl { + if let Some(layout) = layout { + // TODO(emilio): It'd be nice to unify this with the struct path above somehow. diff --git a/tools/aurora-bindgen/patches/0002-anonymous-tag-members.patch b/tools/aurora-bindgen/patches/0002-anonymous-tag-members.patch new file mode 100644 index 00000000000000..7881d882372359 --- /dev/null +++ b/tools/aurora-bindgen/patches/0002-anonymous-tag-members.patch @@ -0,0 +1,90 @@ +--- a/bindgen/clang.rs ++++ b/bindgen/clang.rs +@@ -1204,6 +1204,34 @@ + } + + impl Type { ++ /// Get actual record fields, including implicit anonymous fields omitted ++ /// from `clang_visitChildren` for Microsoft anonymous tag members. ++ pub(crate) fn fields(&self) -> Vec { ++ extern "C" fn collect_field( ++ cur: CXCursor, ++ data: CXClientData, ++ ) -> CXVisitorResult { ++ // SAFETY: clang_Type_visitFields invokes this callback synchronously ++ // with the live Vec pointer supplied below; callbacks do not overlap. ++ let fields = unsafe { &mut *data.cast::>() }; ++ fields.push(Cursor { x: cur }); ++ CXVisit_Continue ++ } ++ ++ let mut fields = Vec::::new(); ++ // SAFETY: self.x is the live Clang type owned by the parsing context. ++ // The visitor only borrows fields during this call, and Clang does not ++ // retain the callback data after returning. ++ unsafe { ++ clang_Type_visitFields( ++ self.x, ++ collect_field, ++ (&mut fields as *mut Vec).cast(), ++ ); ++ } ++ fields ++ } ++ + /// Get this type's kind. + pub(crate) fn kind(&self) -> CXTypeKind { + self.x.kind +--- a/bindgen/ir/comp.rs ++++ b/bindgen/ir/comp.rs +@@ -1340,6 +1340,16 @@ + _ => false, + }); + ++ // Microsoft anonymous named-tag members have real implicit FieldDecls ++ // in the type's field list, but only a non-definition record cursor in ++ // child traversal. Keep positive field evidence: a tag declaration by ++ // itself is not necessarily storage in the containing record. ++ let mut implicit_fields: Vec<_> = ty ++ .fields() ++ .into_iter() ++ .filter(|field| field.spelling().is_empty() && !field.is_bit_field()) ++ .collect(); ++ + let mut maybe_anonymous_struct_field = None; + cursor.visit(|cur| { + if cur.kind() != CXCursor_FieldDecl { +@@ -1463,6 +1473,33 @@ + // definition, it's a valid inner type. + // + // [1]: https://github.com/rust-lang/rust-bindgen/issues/482 ++ if matches!(cur.kind(), CXCursor_StructDecl | CXCursor_UnionDecl) && ++ !cur.is_definition() ++ { ++ if let Some(index) = implicit_fields.iter().position(|field| { ++ field.cur_type().canonical_type() == ++ cur.cur_type().canonical_type() ++ }) { ++ let field = implicit_fields.remove(index); ++ let field_type = Item::from_ty_or_ref( ++ field.cur_type(), ++ field, ++ Some(potential_id), ++ ctx, ++ ); ++ ci.fields.append_raw_field(RawField::new( ++ None, ++ field_type, ++ field.raw_comment(), ++ Annotations::new(&field), ++ None, ++ field.public_accessible(), ++ field.offset_of_field().ok(), ++ )); ++ return CXChildVisit_Continue; ++ } ++ } ++ + let is_inner_struct = + cur.semantic_parent() == cursor || cur.is_definition(); + if !is_inner_struct { diff --git a/tools/aurora-bindgen/sources.json b/tools/aurora-bindgen/sources.json new file mode 100644 index 00000000000000..cf97c6753218ee --- /dev/null +++ b/tools/aurora-bindgen/sources.json @@ -0,0 +1,20 @@ +{ + "version": "0.73.2", + "source_url": "https://codeload.github.com/rust-lang/rust-bindgen/tar.gz/refs/tags/v0.73.2", + "source_sha256": "5fc3277dd334e4bbbd7bdb8cad5d4fb87a667b2e95a2bae350a8d5424c280629", + "source_directory": "rust-bindgen-0.73.2", + "cargo_lock_sha256": "85edb9aab2d5649f39b805dc52e3f27ebf8af4d94b049601aee1c55fcf450393", + "patches": { + "0001-empty-alignment.patch": "eb71efca5c60adc9db6837c933ff377e8aefad53655057406ff9f1464dd3147f", + "0002-anonymous-tag-members.patch": "3c1f6a9fdf02b7cd6f11b1d0787eec7d8a14cf6f140f414cc3069eeb75524562" + }, + "patched_files": { + "bindgen/codegen/mod.rs": "4c100fa631bdd655b509e1797411e341f9a954207d64f8f2efc285201f5f14d9", + "bindgen/clang.rs": "4271adf1fd24f9a8e58e2a69c48f3de6574318461203b513d34308e38852dcd1", + "bindgen/ir/comp.rs": "7d53770db84376699421c8057fbecbfc00e8a6f62ca38919bd4c718e56e40438" + }, + "qualified_private_tool_sha256": "dc794acc7b0d7e3cbfa1533b4b59f2518cd0cc1bff1dbb7fd104229c23c0d146", + "qualified_host_toolchain": "rustc/cargo1.98.1, LLVM/libclang22.1.8, nativeAArch64; kernel Rust1.85.0", + "runtime_tests_performed": false, + "qualified_private_tool_note": "Digest identifies the previously qualified binary before safety-comment-only patch changes; a rebuild from this manifest must record its own binary digest." +} diff --git a/tools/aurora-bindgen/test.py b/tools/aurora-bindgen/test.py new file mode 100644 index 00000000000000..6d93f8b35b9d8c --- /dev/null +++ b/tools/aurora-bindgen/test.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Compile valid generated bindings for explicitly matching C/Rust targets.""" +if not __debug__: + raise SystemExit("optimized Python is unsupported: validation assertions must remain enabled") + +import argparse,hashlib,json,re,subprocess,sys +from pathlib import Path +P=Path(__file__).resolve().parent +ap=argparse.ArgumentParser();ap.add_argument('--bindgen',required=True);ap.add_argument('--output-dir',required=True,type=Path);ap.add_argument('--rustc',default='rustc');ap.add_argument('--clang',default='clang');ap.add_argument('--rust-target');ap.add_argument('--clang-target');ap.add_argument('--rust-sysroot');ap.add_argument('--rust-libdir');ap.add_argument('--baseline-bindgen');a=ap.parse_args() +O=a.output_dir.resolve();assert not O.exists(),'use a fresh evidence directory';O.mkdir(parents=True) +def run(cmd):return subprocess.run(cmd,capture_output=True,text=True) +host=next(x.split(': ',1)[1] for x in subprocess.check_output([a.rustc,'-vV'],text=True).splitlines() if x.startswith('host: '));rt=a.rust_target or host;ct=a.clang_target or rt +arch=lambda t:{'arm64':'aarch64','amd64':'x86_64'}.get(t.split('-')[0],t.split('-')[0]) +assert arch(rt)==arch(ct),('C/Rust architecture mismatch',ct,rt) +rust=[a.rustc,'--edition=2021','--crate-type=rlib','--crate-name','layout_check','--deny=unknown-lints','--emit=obj','-Cpanic=abort','--target='+rt] +if a.rust_sysroot:rust+=['--sysroot='+a.rust_sysroot] +if a.rust_libdir:rust+=['-L',str(Path(a.rust_libdir).resolve())] +probe=O/'target.rs';probe.write_text('#![no_std]\npub fn available() -> usize { core::mem::size_of::() }\n');r=run(rust+[str(probe),'-o',str(O/'target.o')]);(O/'target.log').write_text(r.stdout+r.stderr) +if r.returncode: + status='SKIP' if 'E0463' in r.stderr and 'core' in r.stderr else 'FAIL' + result={'status':status,'reason':'matching Rust target/core unavailable' if status=='SKIP' else 'target preflight compile failed','clang_target':ct,'rust_target':rt,'rust_argv':rust,'log':'target.log'};(O/'result.json').write_text(json.dumps(result,indent=2)+'\n');print(json.dumps(result));sys.exit(77 if status=='SKIP' else 1) +access=''' +unsafe fn check_fields(a: &mut outer, b: &mut wrapper, c: &mut union_member, d: &mut trailing_member) { + a.__bindgen_anon_1.__bindgen_anon_1.words[19] = 1; + b.__bindgen_anon_1.words[19] = 2; + c.__bindgen_anon_1.bytes[23] = 3; + d.__bindgen_anon_1.words[0] = 4; +} +''' +cases=[];equal={};toolset=[('fixed',a.bindgen)]+([('baseline',a.baseline_bindgen)] if a.baseline_bindgen else []) +for fixture,language,extra in [('aligned-empty.h','c',[]),('aligned-empty.hpp','c++',[]),('anonymous-members.h','c',['-fms-extensions']),('tag-only.h','c',[]),('bitfields.h','c',[])]: + header=P/'tests'/fixture + # Also ask the C compiler to accept the ordinary fixture independently. + ccmd=[a.clang,'--target='+ct,'-x',language,*extra,'-Wno-microsoft-anon-tag','-c',str(header),'-o',str(O/(fixture+'.c.o'))];cr=run(ccmd);(O/(fixture+'.c.log')).write_text(cr.stdout+cr.stderr);assert cr.returncode==0,(fixture,cr.stderr) + for label,tool in toolset: + # The baseline comparison covers layout repairs; this fixture checks the fixed tool + # against the selected compiler, including compatibility adapters when needed. + if fixture == 'bitfields.h' and label == 'baseline':continue + cmd=[tool,str(header),'--rust-target','1.85','--no-doc-comments','--use-core','--no-derive-debug','--ctypes-prefix','core::ffi','--','-target',ct,'-x',language,*extra] + g=run(cmd);assert g.returncode==0,(fixture,g.stderr) + assert 'const _: ()' in g.stdout and '#[test]' not in g.stdout,'generator did not emit unconditional const layout checks' + rs=O/(fixture+'.'+label+'.rs');rs.write_text('#![no_std]\n'+g.stdout+(access if fixture=='anonymous-members.h' and label=='fixed' else '')) + rcmd=rust+[str(rs),'-o',str(O/(fixture+'.'+label+'.o'))];r=run(rcmd);(O/(fixture+'.'+label+'.log')).write_text(r.stdout+r.stderr) + expected=label=='fixed' or fixture=='tag-only.h';assert (r.returncode==0)==expected,(fixture,label,r.stderr) + if not expected:assert 'E0080' in r.stderr,'baseline failure must be a const layout mismatch' + cases.append({'fixture':fixture,'tool':label,'clang_argv':ccmd,'generator_argv':cmd,'rust_argv':rcmd,'compile_exit':r.returncode,'expected':'layout compile success' if expected else 'const layout rejection','generated_sha256':hashlib.sha256(rs.read_bytes()).hexdigest()}) + if fixture=='tag-only.h':equal[label]=g.stdout +if a.baseline_bindgen:assert equal['baseline']==equal['fixed'],'ordinary tag-only binding changed' +result={'status':'PASS','clang_target':ct,'rust_target':rt,'rust_host':host,'explicit_rust_target':True,'cases':cases,'baseline_checked':bool(a.baseline_bindgen),'hardware_executed':False};(O/'result.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS matching C/Rust targets:',ct,rt) diff --git a/tools/aurora-bindgen/tests/aligned-empty.h b/tools/aurora-bindgen/tests/aligned-empty.h new file mode 100644 index 00000000000000..9ac1f0546a6236 --- /dev/null +++ b/tools/aurora-bindgen/tests/aligned-empty.h @@ -0,0 +1,13 @@ +struct incomplete; +struct empty1 {} __attribute__((aligned(1))); +struct outer1 { unsigned char prefix[12]; struct empty1 value; unsigned long after; }; +struct empty2 {} __attribute__((aligned(2))); +struct outer2 { unsigned char prefix[12]; struct empty2 value; unsigned long after; }; +struct empty8 {} __attribute__((aligned(8))); +struct outer8 { unsigned char prefix[12]; struct empty8 value; unsigned long after; }; +struct empty64 {} __attribute__((aligned(64))); +struct outer64 { unsigned char prefix[12]; struct empty64 value; unsigned long after; }; +struct empty128 {} __attribute__((aligned(128))); +struct outer128 { unsigned char prefix[12]; struct empty128 value; unsigned long after; }; +struct packed_empty {} __attribute__((packed,aligned(64))); +struct packed_container { char prefix; struct packed_empty value; unsigned long after; }; diff --git a/tools/aurora-bindgen/tests/aligned-empty.hpp b/tools/aurora-bindgen/tests/aligned-empty.hpp new file mode 100644 index 00000000000000..9ac1f0546a6236 --- /dev/null +++ b/tools/aurora-bindgen/tests/aligned-empty.hpp @@ -0,0 +1,13 @@ +struct incomplete; +struct empty1 {} __attribute__((aligned(1))); +struct outer1 { unsigned char prefix[12]; struct empty1 value; unsigned long after; }; +struct empty2 {} __attribute__((aligned(2))); +struct outer2 { unsigned char prefix[12]; struct empty2 value; unsigned long after; }; +struct empty8 {} __attribute__((aligned(8))); +struct outer8 { unsigned char prefix[12]; struct empty8 value; unsigned long after; }; +struct empty64 {} __attribute__((aligned(64))); +struct outer64 { unsigned char prefix[12]; struct empty64 value; unsigned long after; }; +struct empty128 {} __attribute__((aligned(128))); +struct outer128 { unsigned char prefix[12]; struct empty128 value; unsigned long after; }; +struct packed_empty {} __attribute__((packed,aligned(64))); +struct packed_container { char prefix; struct packed_empty value; unsigned long after; }; diff --git a/tools/aurora-bindgen/tests/anonymous-members.h b/tools/aurora-bindgen/tests/anonymous-members.h new file mode 100644 index 00000000000000..56b129a6aab146 --- /dev/null +++ b/tools/aurora-bindgen/tests/anonymous-members.h @@ -0,0 +1,9 @@ +struct payload { unsigned long words[20]; }; +union tag_union { unsigned long word; unsigned char bytes[24]; }; +struct outer { unsigned before; union { struct payload; void *ptr; }; unsigned after; }; +struct wrapper { unsigned prefix; struct payload; unsigned suffix; }; +struct union_member { char prefix; union tag_union; unsigned suffix; }; +struct normal_named { char prefix; struct payload named; unsigned suffix; }; +struct anonymous_definition { char prefix; struct { unsigned long inner; }; unsigned suffix; }; +struct forward_only { struct never_defined *ptr; unsigned suffix; }; +struct trailing_member { unsigned prefix; struct payload; }; diff --git a/tools/aurora-bindgen/tests/bitfields.h b/tools/aurora-bindgen/tests/bitfields.h new file mode 100644 index 00000000000000..47a4332ccf4cf9 --- /dev/null +++ b/tools/aurora-bindgen/tests/bitfields.h @@ -0,0 +1,11 @@ +struct bitfields { + unsigned int enabled : 1; + unsigned int count : 7; + signed int delta : 8; + unsigned int reserved : 16; +}; +struct bitfield_container { + unsigned long before; + struct bitfields value; + unsigned long after; +}; diff --git a/tools/aurora-bindgen/tests/tag-only.h b/tools/aurora-bindgen/tests/tag-only.h new file mode 100644 index 00000000000000..525bc24774be0d --- /dev/null +++ b/tools/aurora-bindgen/tests/tag-only.h @@ -0,0 +1,4 @@ +struct payload { unsigned long words[20]; }; +struct harmless_tag { unsigned before; struct payload; unsigned after; }; +struct ordinary_forward { struct forward_tag; void *ptr; unsigned after; }; +struct normal_named { unsigned before; struct payload named; unsigned after; }; diff --git a/tools/aurora-sep/README.md b/tools/aurora-sep/README.md index 5efa06543beb8a..5de174ba4c7994 100644 --- a/tools/aurora-sep/README.md +++ b/tools/aurora-sep/README.md @@ -34,12 +34,28 @@ systemctl daemon-reload systemctl enable aurora-sep.service ``` -`xart_writes=0` remains the default. `load-driver` explicitly opts into writes -and must only be enabled on a machine whose APFS `.gl` extent passes the -driver's narrow ownership checks: one physical extent, no snapshots or revert +The loader waits up to 120 s for the enclave to seal and unseal a trusted key, +then up to 60 s for `/dev/sep-bio`, and logs how long each took. On J700 the +enclave's key store arrives around 43 s after boot. To change a wait, set +`AURORA_SEP_READY_TIMEOUT` or `AURORA_SEP_DEVICE_TIMEOUT` in whole seconds, +for example with `systemctl edit aurora-sep.service`: + +```ini +[Service] +Environment=AURORA_SEP_READY_TIMEOUT=180 +``` + +The driver defaults to `xart_writes=1` and exports the `apple,sep` module +alias for automatic loading. Writes are accepted only when the APFS `.gl` +extent passes the driver's narrow ownership checks: one physical extent, no snapshots or revert state, and a single extent reference owned by the `.gl` inode. The kernel resolves that extent from checksum-validated APFS metadata; unsupported layouts -fail closed. The opt-in has been tested on m2 with identity-keybag provisioning, +fail closed. Set `options apple_sep xart_writes=0 provision_keybag=0` in a +modprobe configuration to retain read-only operation. `provision_keybag=1` +is also the default: first boot creates the Linux identity needed for trusted +keys, and later boots recover that identity and its machine reference key. +An existing unreadable or invalid identity is not silently replaced. +Enabling writes has been tested on m2 with identity-keybag provisioning, trusted-key seal/unseal, and reboot reload. The raw iBoot container hash did not change in those tests, so an actual xART write was **not** validated. Do not infer that all SEP or Touch ID operations work with a read-only GigaLocker. @@ -74,6 +90,10 @@ ninja -C build meson test -C build --print-errorlogs ``` +The aurora driver for libfprint is also maintained in +https://github.com/aurora-silicon/aurora-sep-userspace, as source and as +distribution packages. + Adjust the patch path for your checkout. The patch includes a fix for an upstream Meson test loop when introspection is disabled. In the Aurora `linux-build` guest, the additional packages needed for this build are @@ -144,3 +164,125 @@ On Omarchy, run `omarchy-apply-lock` once after enrollment. The Quickshell lock screen then selects `omarchy-lock-fingerprint` automatically and accepts Touch ID through fprintd. Hyprlock's separate fingerprint switch is not used by the Quickshell lock screen. + +## Packages + +`packaging/` builds `aurora-sep`, with the loader, its service and the fprintd +policy, as a distribution package. libfprint with the aurora driver comes from +https://github.com/aurora-silicon/aurora-sep-userspace; install both. Installing +`aurora-sep` lets fprintd load the driver with xART writes on, as above. + +On Fedora, `packaging/fedora/build.sh` builds `aurora-sep`. It installs build +dependencies, so run it in a throwaway container from the top of this tree: + +```sh +mkdir rpms +podman run --rm -v "$PWD":/src:ro,z -v "$PWD/rpms":/out:z \ + registry.fedoraproject.org/fedora:45 \ + /src/tools/aurora-sep/packaging/fedora/build.sh /out +sudo dnf install rpms/aurora-sep-1-*.noarch.rpm +``` + +On Arch Linux and Arch Linux ARM, build `aurora-sep`: + +```sh +cd tools/aurora-sep/packaging/arch/aurora-sep && makepkg -si +``` + +fprintd starts the loader the first time it runs. Loading the driver at boot +instead has the enclave ready before the login screen asks for a fingerprint. +The Fedora package enables that with a systemd preset when it is first +installed; on Arch, enable it yourself: + +```sh +sudo systemctl enable aurora-sep.service +``` + +The driver attaches to the enclave once per boot, so reboot rather than +restarting the service or reloading `apple_sep`. To move from the manual +install above, remove `/usr/local/sbin/aurora-sep-load`, +`/etc/systemd/system/aurora-sep.service`, +`/etc/systemd/system/fprintd.service.d/aurora.conf` and any drop-in that points +fprintd at another libfprint, then run `systemctl daemon-reload` and +`systemctl reenable aurora-sep.service`. To go back to the distribution's +libfprint, remove `aurora-sep` and run `dnf distro-sync libfprint` or +`pacman -S libfprint`. + +## Desktop integration + +The first open of `/dev/sep-bio` activates Touch ID: the driver restores the +enrolled fingerprints from the enclave before the open returns, which takes +a few seconds. `aurora-sep-ready.service`, which `aurora-sep.service` pulls +in, makes that open at boot, before fprintd starts. The lock screen, sudo +and polkit then find the reader and the fingerprints ready, and the first +fingerprint request does not wait for the restore. A failed open is only +logged. The packages install it; by hand: + +```sh +install -Dm755 aurora-sep-ready /usr/local/sbin/aurora-sep-ready +install -Dm644 aurora-sep-ready.service /etc/systemd/system/aurora-sep-ready.service +systemctl daemon-reload +``` + +fprintd exits after 30 s without a client, and each new start probes the +reader again, while the login and lock screens ask for the reader as they +come up. `fprintd-no-timeout.conf.in` keeps it running so that it answers +them at once. The packages fill in where the distribution keeps fprintd; by +hand, with `/usr/lib/fprintd` on Arch: + +```sh +sed 's|@FPRINTD@|/usr/libexec/fprintd|' fprintd-no-timeout.conf.in \ + >/etc/systemd/system/fprintd.service.d/aurora-sep-no-timeout.conf +``` + +The Touch ID sensor is the power button. A touch to authenticate goes to the +enclave and never reaches logind as a key press, but a press does, and logind +cannot tell a press meant for the sensor from one meant for the button. Its +default is to power off. `logind-aurora-sep.conf` makes a short press +suspend instead, so pressing while authenticating loses nothing and the next +press wakes the machine, and holding the button for 5 s powers off cleanly. +GNOME sessions and the GDM login screen handle a short press themselves +(`power-button-action`, `suspend` by default); the setting covers consoles +and other desktops. It takes effect at the next boot. By hand: + +```sh +install -Dm644 logind-aurora-sep.conf /etc/systemd/logind.conf.d/aurora-sep.conf +``` + +GNOME Shell 51 offers fingerprint only on the lock screen, never on the +login screen, and `org.gnome.login-screen enable-fingerprint-authentication` +(true by default) cannot change that: the greeter connects its fingerprint +reader only when it re-authenticates a running session. That is GNOME's +choice, because a fingerprint cannot unlock the login keyring: after a +fingerprint login, the first application that wants a stored secret asks +for the password. `aurora-sep-greeter-fingerprint.service` lifts the +restriction for those who want it. At boot it copies the one greeter source +file that holds the check out of the installed GNOME Shell, drops the check, +and serves the copy to the login screen's shell alone through a GResource +overlay. When the check is not exactly as expected, for example after a +GNOME update, it changes nothing. The enable key above still applies. + +The login screen asks fprintd for the reader once, as it comes up, so the +same unit also holds it back until `aurora-sep-ready.service` has finished: +a few seconds on a healthy boot, but as long as the enclave takes when it is +slow, up to 90 s after boot. Without fingerprint at the login screen that +wait buys nothing, which is why both are one switch, off by default. It +needs `gresource` from glib2-devel. With the packages, or after installing +it by hand, turn it on with `systemctl enable`: + +```sh +install -Dm755 aurora-sep-greeter-fingerprint /usr/local/sbin/aurora-sep-greeter-fingerprint +install -Dm644 aurora-sep-greeter-fingerprint.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable aurora-sep-greeter-fingerprint.service +``` + +Disabling it and rebooting returns the login screen to GNOME's behaviour. + +## Biometric userspace header + +The canonical ioctl ABI is `include/uapi/linux/apple-sep.h`. Export this +checkout's headers with `make headers_install INSTALL_HDR_PATH=` +and add that directory's `include` to the libfprint build include path. The +aurora driver in aurora-sep-userspace uses `` directly; it +has no copied ABI header. Interface version, ioctl numbers and fixed-width layouts are unchanged. diff --git a/tools/aurora-sep/XART-SAFETY.md b/tools/aurora-sep/XART-SAFETY.md index e4578a0ec29425..4258fc6db06657 100644 --- a/tools/aurora-sep/XART-SAFETY.md +++ b/tools/aurora-sep/XART-SAFETY.md @@ -1,5 +1,17 @@ # Shared xART gigalocker safety gate +## Current default + +`xart_writes=1` is the driver default. The APFS extent-ownership checks, +record validation and write-ordering checks remain mandatory; unsupported +layouts fail closed. `xart_writes=0 provision_keybag=0` explicitly selects +read-only operation. `provision_keybag=1` is the default: create the Linux +identity if missing and recover an existing identity without replacing it. +The historical evidence below records the earlier opt-in deployments and +does not establish qualification on every supported Mac or APFS layout. + +## Earlier deployment policy + The XART `.gl` file occupies APFS-managed storage. Writes require the explicit `xart_writes=1` opt-in and an APFS-resolved extent with a single owner, no snapshots or clones, and valid existing records. `xart_start_sector` is only diff --git a/tools/aurora-sep/aurora-sep-greeter-fingerprint b/tools/aurora-sep/aurora-sep-greeter-fingerprint new file mode 100644 index 00000000000000..7f04a19d24d20d --- /dev/null +++ b/tools/aurora-sep/aurora-sep-greeter-fingerprint @@ -0,0 +1,85 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only +# +# Offer fingerprint at the GNOME login screen, or with "stop", stop offering +# it from the next time the login screen starts. +# +# GNOME Shell 51 connects the fingerprint reader only when it re-authenticates +# a session that is already running, which is the lock screen. The login +# screen never offers fingerprint, whatever org.gnome.login-screen says: a +# fingerprint cannot unlock the login keyring, so a session started with one +# asks for the password the first time an application wants a stored secret. +# +# This takes the greeter source file that holds that check from the installed +# GNOME Shell, drops the check, and serves the copy to the login screen's +# shell alone through a GResource overlay. When the check is not exactly as +# expected, for example after a GNOME update, it changes nothing and the login +# screen stays as GNOME ships it. Run it before the display manager starts. +set -eu + +res=/org/gnome/shell/gdm/authServicesLegacy.js +out=/run/aurora-sep/greeter +dropin=/run/systemd/user/org.gnome.Shell@gdm.service.d/aurora-sep-fingerprint.conf + +log() { + echo "aurora-sep-greeter-fingerprint: $*" >&2 +} + +rm -rf "$out" "$dropin" +[ "${1:-}" != stop ] || exit 0 + +if ! command -v gresource >/dev/null; then + log "gresource is not installed (it comes with glib2-devel); the login screen stays as it is" + exit 0 +fi + +lib= +for f in /usr/lib64/gnome-shell/libshell-*.so /usr/lib/gnome-shell/libshell-*.so \ + /usr/lib/*/gnome-shell/libshell-*.so; do + [ -f "$f" ] || continue + if gresource list "$f" 2>/dev/null | grep -qxF "$res"; then + lib=$f + break + fi +done +if [ -z "$lib" ]; then + log "no installed GNOME Shell carries $res; the login screen stays as it is" + exit 0 +fi + +install -d -m 0755 "$out" +gresource extract "$lib" "$res" >"$out/stock.js" +if ! awk ' + state == 0 && $0 == " _connectFingerprintManager() {" { + print + state = 1 + next + } + state == 1 { + if ($0 == " if (!this._reauthOnly)") { + state = 2 + next + } + state = 9 + } + state == 2 { + if ($0 == " return;") { + state = 3 + next + } + state = 9 + } + { print } + END { exit state != 3 } +' "$out/stock.js" >"$out/authServicesLegacy.js"; then + log "$lib does not have the expected check; the login screen stays as it is" + rm -rf "$out" + exit 0 +fi +rm -f "$out/stock.js" +chmod 0644 "$out/authServicesLegacy.js" + +install -d -m 0755 "${dropin%/*}" +printf '[Service]\nEnvironment=G_RESOURCE_OVERLAYS=%s=%s\n' \ + "$res" "$out/authServicesLegacy.js" >"$dropin" +log "the login screen offers fingerprint ($res from $lib)" diff --git a/tools/aurora-sep/aurora-sep-greeter-fingerprint.service b/tools/aurora-sep/aurora-sep-greeter-fingerprint.service new file mode 100644 index 00000000000000..108b1e4ccbc875 --- /dev/null +++ b/tools/aurora-sep/aurora-sep-greeter-fingerprint.service @@ -0,0 +1,17 @@ +[Unit] +Description=Offer fingerprint at the GNOME login screen +Documentation=file:///usr/share/doc/aurora-sep/README.md +# The login screen asks fprintd for the reader once, as it comes up. Hold it +# back until Touch ID is up, so that it finds the reader and the fingerprints. +Wants=aurora-sep-ready.service +After=aurora-sep-ready.service +Before=gdm.service + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/local/sbin/aurora-sep-greeter-fingerprint +ExecStop=/usr/local/sbin/aurora-sep-greeter-fingerprint stop + +[Install] +WantedBy=gdm.service diff --git a/tools/aurora-sep/aurora-sep-ready b/tools/aurora-sep/aurora-sep-ready new file mode 100644 index 00000000000000..0363a3a298ecbb --- /dev/null +++ b/tools/aurora-sep/aurora-sep-ready @@ -0,0 +1,44 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only +# +# Bring Touch ID up before anyone asks for a fingerprint. The first open of +# /dev/sep-bio activates it: the driver restores the enrolled fingerprints +# from the enclave and returns only when that is done. Opened here at boot, +# ahead of fprintd, the restore no longer delays the first fingerprint +# request. A failure is only logged. +set -u + +dev=/dev/sep-bio + +log() { + echo "aurora-sep-ready: $*" >&2 +} + +uptime_s() { + read -r up _ &1); then + log "Touch ID is ready after $(($(uptime_s) - start)) s" + exit 0 + fi + case "$err" in + # Another opener is activating it or using it; the unit's start + # timeout bounds this. + *[Bb]usy*) sleep 0.2 ;; + *) + log "could not open $dev: ${err##*: }" + exit 0 + ;; + esac +done diff --git a/tools/aurora-sep/aurora-sep-ready.service b/tools/aurora-sep/aurora-sep-ready.service new file mode 100644 index 00000000000000..91e5a7bce38154 --- /dev/null +++ b/tools/aurora-sep/aurora-sep-ready.service @@ -0,0 +1,18 @@ +[Unit] +Description=Activate Touch ID and restore its fingerprints +Documentation=file:///usr/share/doc/aurora-sep/README.md +Requires=aurora-sep.service +After=aurora-sep.service +Before=fprintd.service +ConditionPathExists=/dev/sep-bio + +# Whatever is ordered after this waits for the enclave, which can take most +# of a minute after boot. Stop waiting 90 s after boot, so that a broken +# enclave cannot hold anything back for longer. +JobTimeoutSec=90 + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/local/sbin/aurora-sep-ready +TimeoutStartSec=60 diff --git a/tools/aurora-sep/aurora-sep.service b/tools/aurora-sep/aurora-sep.service index 4287247cba0da1..34a9f1cd69eb76 100644 --- a/tools/aurora-sep/aurora-sep.service +++ b/tools/aurora-sep/aurora-sep.service @@ -2,6 +2,8 @@ Description=Aurora SEP services After=local-fs.target RequiresMountsFor=/var/lib +# Activate Touch ID as soon as the driver is up. +Wants=aurora-sep-ready.service [Service] Type=oneshot diff --git a/tools/aurora-sep/fprintd-no-timeout.conf.in b/tools/aurora-sep/fprintd-no-timeout.conf.in new file mode 100644 index 00000000000000..35f70927831fa9 --- /dev/null +++ b/tools/aurora-sep/fprintd-no-timeout.conf.in @@ -0,0 +1,7 @@ +# Keep fprintd running instead of letting it exit after 30 s without a +# client. Each new start probes the reader again, and the login and lock +# screens ask it for the reader when they come up, so a resident fprintd +# answers them at once. The packages fill in fprintd's path. +[Service] +ExecStart= +ExecStart=@FPRINTD@ --no-timeout diff --git a/tools/aurora-sep/load-driver b/tools/aurora-sep/load-driver index a70db75e6b25b1..fbf43e5093f005 100755 --- a/tools/aurora-sep/load-driver +++ b/tools/aurora-sep/load-driver @@ -4,6 +4,34 @@ set -eu keybag=/var/lib/aurora-sep-keybag.bin refkey=/var/lib/aurora-sep-refkey.bin +# Seconds to wait for the enclave to seal and unseal a trusted key, then for +# /dev/sep-bio. On J700 the key store arrives around 43 s after boot, and the +# driver waits up to 90 s for it before it brings Touch ID up, so the first +# wait has to outlast that. Set either in the environment, for example with a +# drop-in for aurora-sep.service. +ready_timeout=${AURORA_SEP_READY_TIMEOUT:-120} +device_timeout=${AURORA_SEP_DEVICE_TIMEOUT:-60} + +log() { + echo "aurora-sep-load: $*" >&2 +} + +for budget in "$ready_timeout" "$device_timeout"; do + case "$budget" in + '' | *[!0-9]*) + log "timeouts must be whole seconds, not '$budget'" + exit 2 + ;; + esac +done + +# Seconds since boot. Unlike the wall clock it cannot jump when the time is +# first synchronised during boot. +uptime_s() { + read -r up _ /dev/null || true) if [ -n "$key_id" ]; then if keyctl pipe "$key_id" >"$work/sealed"; then @@ -37,17 +66,27 @@ while [ "$attempts" -gt 0 ]; do keyctl revoke "$key_id" >/dev/null 2>&1 || true fi [ "$verified" = true ] && break + [ "$(uptime_s)" -lt "$deadline" ] || break sleep 0.1 - attempts=$((attempts - 1)) done -[ "$verified" = true ] || exit 1 -[ -s "$keybag" ] || exit 1 -[ -s "$refkey" ] || exit 1 +if [ "$verified" != true ]; then + log "the enclave did not seal and unseal a trusted key within ${ready_timeout} s" + exit 1 +fi +log "trusted keys work after $(($(uptime_s) - start)) s" + +if [ ! -s "$keybag" ] || [ ! -s "$refkey" ]; then + log "$keybag or $refkey is missing or empty" + exit 1 +fi -attempts=300 -while [ "$attempts" -gt 0 ] && [ ! -c /dev/sep-bio ]; do +deadline=$(($(uptime_s) + device_timeout)) +while [ ! -c /dev/sep-bio ] && [ "$(uptime_s)" -lt "$deadline" ]; do sleep 0.1 - attempts=$((attempts - 1)) done -[ -c /dev/sep-bio ] || exit 1 +if [ ! -c /dev/sep-bio ]; then + log "/dev/sep-bio did not appear within ${device_timeout} s" + exit 1 +fi +log "/dev/sep-bio is ready after $(($(uptime_s) - start)) s" diff --git a/tools/aurora-sep/logind-aurora-sep.conf b/tools/aurora-sep/logind-aurora-sep.conf new file mode 100644 index 00000000000000..a175c39f6ca73c --- /dev/null +++ b/tools/aurora-sep/logind-aurora-sep.conf @@ -0,0 +1,11 @@ +# The Touch ID sensor is the power button. Touching it to authenticate goes to +# the enclave and is never a key press, but pressing it is, and logind cannot +# tell a press meant for the sensor from one meant for the button. Suspend +# rather than power off on a short press, so a press while authenticating +# costs nothing: the next press wakes the machine. Holding the button for +# 5 s still powers off, cleanly. GNOME sessions and the GDM login screen +# handle a short press themselves (power-button-action, suspend by default); +# this covers everything else. +[Login] +HandlePowerKey=suspend +HandlePowerKeyLongPress=poweroff diff --git a/tools/aurora-sep/neo/mesa-neo-env b/tools/aurora-sep/neo/mesa-neo-env index b36054c2ac1c56..0ea9eb32902e7a 100644 --- a/tools/aurora-sep/neo/mesa-neo-env +++ b/tools/aurora-sep/neo/mesa-neo-env @@ -48,7 +48,7 @@ mesa_neo_display_admit() { [ -e "$_neo_card" ] || continue if mesa_neo_has_compatible "$_neo_card/device/of_node/compatible" apple,agx-t8140; then [ "$(readlink -f "$_neo_card/device" 2>/dev/null)" = "$_neo_gpu_device" ] && - [ "$(basename "$(readlink -f "$_neo_card/device/driver" 2>/dev/null)")" = asahi_neo ] || return 1 + [ "$(basename "$(readlink -f "$_neo_card/device/driver" 2>/dev/null)")" = asahi ] || return 1 # The GPU primary node has no KMS connectors. for _neo_connector in "$_neo_card"-*; do [ ! -e "$_neo_connector" ] || return 1 @@ -57,7 +57,7 @@ mesa_neo_display_admit() { fi _neo_cards=$((_neo_cards+1)) # A firmware framebuffer cannot use the Asahi render-only KMS route. - [ "$(basename "$(readlink -f "$_neo_card/device/driver" 2>/dev/null)")" = apple-drm-neo ] || return 1 + [ "$(basename "$(readlink -f "$_neo_card/device/driver" 2>/dev/null)")" = apple-drm ] || return 1 mesa_neo_has_compatible "$_neo_card/device/of_node/compatible" apple,t8140-display-subsystem || return 1 [ -r "$_neo_root/dev/dri/${_neo_card##*/}" ] && [ -w "$_neo_root/dev/dri/${_neo_card##*/}" ] || return 1 for _neo_connector in "$_neo_card"-*; do diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/PKGBUILD b/tools/aurora-sep/packaging/arch/aurora-sep/PKGBUILD new file mode 100644 index 00000000000000..bcb0917d589869 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/PKGBUILD @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# The loader and desktop integration for the Apple SEP fingerprint driver. +# The local sources are links into tools/aurora-sep; after changing any of +# them, run updpkgsums here. libfprint with the aurora driver comes from +# https://github.com/aurora-silicon/aurora-sep-userspace. + +pkgname=aurora-sep +pkgver=1 +pkgrel=5 +pkgdesc="Loader and desktop integration for the Apple SEP (Touch ID) driver" +url="https://github.com/aurora-silicon/linux/tree/aurora-wip/tools/aurora-sep" +arch=(any) +license=(GPL-2.0-only) +depends=( + coreutils + fprintd + keyutils + kmod + sh +) +optdepends=('glib2-devel: fingerprint at the GNOME login screen') +source=(load-driver + aurora-sep.service + fprintd-aurora.conf + aurora-sep-ready + aurora-sep-ready.service + fprintd-no-timeout.conf.in + logind-aurora-sep.conf + aurora-sep-greeter-fingerprint + aurora-sep-greeter-fingerprint.service + README.md + XART-SAFETY.md) +sha256sums=('abea60bb264ced349f7cf671d8d26e1609a81a60ea4c34d66ec6f7f73b847dbb' + '447f99cc3c7e683134f31b993aa825fadde24e4854954a3035c96e6631d4d4d0' + 'f8d0c4b9385e6716d5ac5dd9941a57459fbc551a8238383bcde36cd929d0523f' + '2db07d3a0191e8d14581a3056505a8c4225b2fc8e6ea4a1956ae5493abcbb0ae' + '5a5e5c7e93d1462d9d1023446fdb297926aeb93d8988867b515aca7adafe4487' + '8d8df0ba2a6861b2edb2110e606906d84ca4848617621ad3cb48493a1522971d' + '0695bd9ce4c207241db13c3ea07819a032d8213417021378f5015ab9317cde12' + '5936eb76db666496a8700dade7ac6bcb3785b517958dcea10685690f1822c86f' + 'ffab97b20bfd5f0eb1ab641179a131843349cf16890fff0cf2d893f0b5681b37' + 'a87584f87b8b49c19333fd9c5a8e469401756ec08519fd6cc655bef74b5d284f' + 'a950153f3f4e600f29b8e1976bb7de71f96bbf6b922cb88512c40409a8d1b6ef') + +# A unit and the helper it runs ($1, installed as $3), pointed at the +# packaged helper. +_unit() { + local unit="$pkgdir/usr/lib/systemd/system/$2" + + install -Dm755 "$1" "$pkgdir/usr/lib/aurora-sep/$3" + install -Dm644 "$2" "$unit" + sed -i "s|^\(Exec[A-Za-z]*=\)/usr/local/sbin/$3|\1/usr/lib/aurora-sep/$3|" "$unit" + if grep -q /usr/local "$unit"; then + return 1 + fi +} + +package() { + _unit load-driver aurora-sep.service aurora-sep-load + _unit aurora-sep-ready aurora-sep-ready.service aurora-sep-ready + _unit aurora-sep-greeter-fingerprint aurora-sep-greeter-fingerprint.service \ + aurora-sep-greeter-fingerprint + + install -Dm644 fprintd-aurora.conf \ + "$pkgdir/usr/lib/systemd/system/fprintd.service.d/aurora-sep.conf" + grep -qx 'ExecStart=/usr/lib/fprintd' /usr/lib/systemd/system/fprintd.service + sed 's|@FPRINTD@|/usr/lib/fprintd|' fprintd-no-timeout.conf.in \ + >"$pkgdir/usr/lib/systemd/system/fprintd.service.d/50-aurora-sep-no-timeout.conf" + install -Dm644 logind-aurora-sep.conf \ + "$pkgdir/usr/lib/systemd/logind.conf.d/50-aurora-sep.conf" + install -Dm644 -t "$pkgdir/usr/share/doc/$pkgname" README.md XART-SAFETY.md +} diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/README.md b/tools/aurora-sep/packaging/arch/aurora-sep/README.md new file mode 120000 index 00000000000000..8a33348c7d811a --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/README.md @@ -0,0 +1 @@ +../../../README.md \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/XART-SAFETY.md b/tools/aurora-sep/packaging/arch/aurora-sep/XART-SAFETY.md new file mode 120000 index 00000000000000..45a7d77d7e92e2 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/XART-SAFETY.md @@ -0,0 +1 @@ +../../../XART-SAFETY.md \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint new file mode 120000 index 00000000000000..dd1bb065453132 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint @@ -0,0 +1 @@ +../../../aurora-sep-greeter-fingerprint \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint.service b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint.service new file mode 120000 index 00000000000000..bf6aeca94eb787 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-greeter-fingerprint.service @@ -0,0 +1 @@ +../../../aurora-sep-greeter-fingerprint.service \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready new file mode 120000 index 00000000000000..55fb2a87e5d434 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready @@ -0,0 +1 @@ +../../../aurora-sep-ready \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready.service b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready.service new file mode 120000 index 00000000000000..7ab622ca2d8339 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep-ready.service @@ -0,0 +1 @@ +../../../aurora-sep-ready.service \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep.service b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep.service new file mode 120000 index 00000000000000..72bc9aad30d16b --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/aurora-sep.service @@ -0,0 +1 @@ +../../../aurora-sep.service \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-aurora.conf b/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-aurora.conf new file mode 120000 index 00000000000000..41efea20ba890e --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-aurora.conf @@ -0,0 +1 @@ +../../../fprintd-aurora.conf \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-no-timeout.conf.in b/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-no-timeout.conf.in new file mode 120000 index 00000000000000..2e8a7a43c8830d --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/fprintd-no-timeout.conf.in @@ -0,0 +1 @@ +../../../fprintd-no-timeout.conf.in \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/load-driver b/tools/aurora-sep/packaging/arch/aurora-sep/load-driver new file mode 120000 index 00000000000000..3f9fc69e380426 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/load-driver @@ -0,0 +1 @@ +../../../load-driver \ No newline at end of file diff --git a/tools/aurora-sep/packaging/arch/aurora-sep/logind-aurora-sep.conf b/tools/aurora-sep/packaging/arch/aurora-sep/logind-aurora-sep.conf new file mode 120000 index 00000000000000..db6dc4e9530226 --- /dev/null +++ b/tools/aurora-sep/packaging/arch/aurora-sep/logind-aurora-sep.conf @@ -0,0 +1 @@ +../../../logind-aurora-sep.conf \ No newline at end of file diff --git a/tools/aurora-sep/packaging/fedora/aurora-sep.spec b/tools/aurora-sep/packaging/fedora/aurora-sep.spec new file mode 100644 index 00000000000000..b4dd9ca4391b6e --- /dev/null +++ b/tools/aurora-sep/packaging/fedora/aurora-sep.spec @@ -0,0 +1,112 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Built from tools/aurora-sep: rpmbuild --define "_sourcedir " +# (build.sh does this). + +Name: aurora-sep +Version: 1 +Release: 3%{?dist} +Summary: Loader and desktop integration for the Apple SEP fingerprint driver + +License: GPL-2.0-only +URL: https://github.com/aurora-silicon/linux +Source0: load-driver +Source1: aurora-sep.service +Source2: fprintd-aurora.conf +Source3: README.md +Source4: XART-SAFETY.md +Source5: aurora-sep-ready +Source6: aurora-sep-ready.service +Source7: fprintd-no-timeout.conf.in +Source8: logind-aurora-sep.conf +Source9: aurora-sep-greeter-fingerprint +Source10: aurora-sep-greeter-fingerprint.service + +BuildArch: noarch +BuildRequires: systemd-rpm-macros +# Only to check where fprintd's unit runs it from. +BuildRequires: fprintd +Requires: coreutils +Requires: fprintd +Requires: keyutils +Requires: kmod +# Only for aurora-sep-greeter-fingerprint.service, which is off by default. +Suggests: /usr/bin/gresource +%{?systemd_requires} + +%description +Loads the Apple SEP driver (apple_sep) with xART writes enabled, waits until +the secure enclave seals and unseals a trusted key and /dev/sep-bio appears, +and lets fprintd reach /dev/sep-bio. The loader runs at boot, and fprintd +also pulls it in when it starts. Touch ID is then activated, before fprintd +starts, and pressing the Touch ID button suspends rather than powers off. +libfprint needs the aurora driver, which this package does not provide; it is +packaged in https://github.com/aurora-silicon/aurora-sep-userspace. Read XART-SAFETY.md before installing it on +a machine that also runs macOS. + +%prep +%setup -q -c -T +cp -p %{SOURCE3} %{SOURCE4} . + +%build + +%install +# A unit and the helper it runs ($1, installed as $3), pointed at the +# packaged helper. +unit() { + install -Dpm 0755 "$1" %{buildroot}%{_libexecdir}/"$3" + install -Dpm 0644 "$2" %{buildroot}%{_unitdir}/"${2##*/}" + sed -i "s|^\(Exec[A-Za-z]*=\)/usr/local/sbin/$3|\1%{_libexecdir}/$3|" \ + %{buildroot}%{_unitdir}/"${2##*/}" + if grep -q /usr/local %{buildroot}%{_unitdir}/"${2##*/}"; then exit 1; fi +} +unit %{SOURCE0} %{SOURCE1} aurora-sep-load +unit %{SOURCE5} %{SOURCE6} aurora-sep-ready +unit %{SOURCE9} %{SOURCE10} aurora-sep-greeter-fingerprint +install -Dpm 0644 %{SOURCE2} \ + %{buildroot}%{_unitdir}/fprintd.service.d/aurora-sep.conf +grep -qx 'ExecStart=%{_libexecdir}/fprintd' %{_unitdir}/fprintd.service +sed 's|@FPRINTD@|%{_libexecdir}/fprintd|' %{SOURCE7} \ + >%{buildroot}%{_unitdir}/fprintd.service.d/50-aurora-sep-no-timeout.conf +install -Dpm 0644 %{SOURCE8} \ + %{buildroot}%{_prefix}/lib/systemd/logind.conf.d/50-aurora-sep.conf +# Load the driver at boot, so the enclave is up before the login screen asks. +install -d %{buildroot}%{_presetdir} +echo 'enable aurora-sep.service' >%{buildroot}%{_presetdir}/80-aurora-sep.preset + +%post +%systemd_post aurora-sep.service aurora-sep-ready.service aurora-sep-greeter-fingerprint.service + +%preun +%systemd_preun aurora-sep.service aurora-sep-ready.service aurora-sep-greeter-fingerprint.service + +%postun +# Never restart them: the driver attaches to the enclave once per boot. +%systemd_postun aurora-sep.service aurora-sep-ready.service aurora-sep-greeter-fingerprint.service + +%files +%doc README.md XART-SAFETY.md +%{_libexecdir}/aurora-sep-load +%{_libexecdir}/aurora-sep-ready +%{_libexecdir}/aurora-sep-greeter-fingerprint +%{_unitdir}/aurora-sep.service +%{_unitdir}/aurora-sep-ready.service +%{_unitdir}/aurora-sep-greeter-fingerprint.service +%dir %{_unitdir}/fprintd.service.d +%{_unitdir}/fprintd.service.d/aurora-sep.conf +%{_unitdir}/fprintd.service.d/50-aurora-sep-no-timeout.conf +%{_prefix}/lib/systemd/logind.conf.d/50-aurora-sep.conf +%{_presetdir}/80-aurora-sep.preset + +%changelog +* Sun Oct 11 2026 Ryan Murray - 1-3 +- libfprint with the aurora driver is now built in aurora-sep-userspace + +* Sun Oct 11 2026 Ryan Murray - 1-2 +- Activate Touch ID at boot, before fprintd starts +- Keep fprintd running +- Suspend rather than power off on the Touch ID button +- Offer fingerprint at the GNOME 51 login screen, off by default + +* Sun Oct 11 2026 Ryan Murray - 1-1 +- Package the loader, its service and the fprintd policy diff --git a/tools/aurora-sep/packaging/fedora/build.sh b/tools/aurora-sep/packaging/fedora/build.sh new file mode 100755 index 00000000000000..a1331bbdb266b1 --- /dev/null +++ b/tools/aurora-sep/packaging/fedora/build.sh @@ -0,0 +1,38 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only +# +# Build the Fedora package for Touch ID on the Apple SEP: +# +# - aurora-sep: the driver loader, its service and the fprintd policy. +# +# libfprint with the aurora driver is built and packaged in +# https://github.com/aurora-silicon/aurora-sep-userspace. +# +# It installs the build dependencies with dnf, so run it as root in a +# throwaway container of the Fedora release you are building for: +# +# podman run --rm -v :/src:ro,z -v :/out:z \ +# registry.fedoraproject.org/fedora:45 /src/ /out +# +# The binary and source packages land in the output directory. + +set -eu + +out=${1:?usage: build.sh } +here=$(cd "$(dirname "$0")" && pwd) +sep=$(cd "$here/../.." && pwd) +mkdir -p "$out" + +top=$(mktemp -d) +trap 'rm -rf "$top"' EXIT +rpmtop=$top/rpmbuild + +dnf -y install rpm-build systemd-rpm-macros + +# The loader, its units and the desktop integration, from this tree. +dnf -y builddep "$here/aurora-sep.spec" +rpmbuild --define "_topdir $rpmtop" --define "_sourcedir $sep" \ + -ba "$here/aurora-sep.spec" + +cp -p "$rpmtop"/RPMS/*/*.rpm "$rpmtop"/SRPMS/*.rpm "$out/" +ls -l "$out" diff --git a/tools/aurora-sep/tests/test_neo_gpu.py b/tools/aurora-sep/tests/test_neo_gpu.py index 6b9af473d0d52d..a9dddb2488f977 100644 --- a/tools/aurora-sep/tests/test_neo_gpu.py +++ b/tools/aurora-sep/tests/test_neo_gpu.py @@ -144,7 +144,7 @@ def hook(self, intent=True, profile='j700-g17p-hal200', abi=0, display=True): (self.tmp / 'dev').mkdir(exist_ok=True) (self.tmp / 'dev/renderD128').touch() gpu = render.parent - driver = self.tmp / 'drivers/asahi_neo' + driver = self.tmp / 'drivers/asahi' driver.mkdir(parents=True, exist_ok=True) if not (gpu / 'driver').is_symlink(): (gpu / 'driver').symlink_to(driver) @@ -156,7 +156,7 @@ def hook(self, intent=True, profile='j700-g17p-hal200', abi=0, display=True): kms = self.tmp / 'sys/card2/device/of_node' kms.mkdir(parents=True, exist_ok=True) (kms / 'compatible').write_bytes(b'apple,t8140-display-subsystem\0') - dcp = self.tmp / 'drivers/apple-drm-neo' + dcp = self.tmp / 'drivers/apple-drm' dcp.mkdir(parents=True, exist_ok=True) if not (kms.parent / 'driver').is_symlink(): (kms.parent / 'driver').symlink_to(dcp) diff --git a/tools/aurora-sep/tests/test_packaging.py b/tools/aurora-sep/tests/test_packaging.py new file mode 100644 index 00000000000000..961985758ba1e5 --- /dev/null +++ b/tools/aurora-sep/tests/test_packaging.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""The packaging must describe the files in this tree, not stale copies.""" + +import hashlib +import re +import unittest +from pathlib import Path + +SEP = Path(__file__).resolve().parents[1] +ARCH = SEP / "packaging" / "arch" +FEDORA = SEP / "packaging" / "fedora" + + +def bash_array(text, name): + match = re.search(r"^%s=\((.*?)\)" % re.escape(name), text, re.M | re.S) + if match is None: + raise AssertionError("no %s array" % name) + return re.findall(r"'([^']*)'|\"([^\"]*)\"|(\S+)", match.group(1)) + + +def words(text, name): + return ["".join(parts) for parts in bash_array(text, name)] + + +class PkgbuildTest(unittest.TestCase): + def check(self, directory): + text = (directory / "PKGBUILD").read_text() + sources = words(text, "source") + sums = words(text, "sha256sums") + self.assertEqual(len(sources), len(sums)) + pkgver = re.search(r"^pkgver=(\S+)$", text, re.M).group(1) + for source, expected in zip(sources, sums): + if "://" in source: + continue + name = source.replace("$pkgver", pkgver) + path = directory / name + self.assertTrue(path.is_file(), "%s is not a file" % path) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + self.assertEqual( + digest, expected, "%s changed: run updpkgsums in %s" % (name, directory) + ) + + def test_aurora_sep_sums_match_tree(self): + self.check(ARCH / "aurora-sep") + + +class FedoraSpecTest(unittest.TestCase): + def test_sources_are_in_tree(self): + text = (FEDORA / "aurora-sep.spec").read_text() + sources = re.findall(r"^Source\d+:\s+(\S+)$", text, re.M) + self.assertTrue(sources) + for source in sources: + self.assertTrue((SEP / source).is_file(), source) + + def test_build_script_builds_only_aurora_sep(self): + text = (FEDORA / "build.sh").read_text() + self.assertIn("aurora-sep.spec", text) + self.assertNotIn("dnf -y download --source libfprint", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/aurora-sep/tests/test_sbio_exchange.py b/tools/aurora-sep/tests/test_sbio_exchange.py new file mode 100644 index 00000000000000..177246001f7593 --- /dev/null +++ b/tools/aurora-sep/tests/test_sbio_exchange.py @@ -0,0 +1,174 @@ +# SPDX-License-Identifier: MIT +"""Run the driver's actual SBIO transfer method from two threads at once. + +The verify/enrol work item and the ioctl path (GET_INFO and ENROL_START list +the enclave's identities, DELETE removes one) can reach the single SBIO +endpoint at the same time, e.g. when libfprint cancels an identify and starts +the next one while the previous work item is still registering the sensor. +Each exchange must complete with its own reply; none may be refused with +EBUSY or receive another caller's answer. +""" +from pathlib import Path +import re +import shutil +import subprocess +import tempfile +import unittest + +APPLE = Path(__file__).resolve().parents[3] / 'drivers/soc/apple' + + +def method(file, name): + source = (APPLE / file).read_text() + match = re.search(r'^ (?:pub\(crate\) )?fn ' + name + r'\(', source, re.M) + if match is None: + raise AssertionError(f'Missing driver method {name}') + opening = source.index('{', match.start()) + depth, end = 1, opening + 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + return source[match.start():end] + + +HARNESS = r''' +#![allow(dead_code, unused_variables, unused_imports)] +use std::sync::atomic::{AtomicBool, Ordering::Relaxed}; +use std::sync::mpsc; +use std::time::Duration; + +type CStr = str; +type Result = std::result::Result; +const ENODEV: i32 = -19; +const EBUSY: i32 = -16; +const ETIMEDOUT: i32 = -110; +const SBIO_TIMEOUT_MS: u64 = 2000; + +mod time { pub fn msecs_to_jiffies(ms: u64) -> u64 { ms } } + +pub enum CondVarTimeoutResult { Woken { jiffies: u64 }, Signal { jiffies: u64 }, Timeout } + +/// Kernel-shaped mutex: lock() returns a guard a CondVar can wait on. +struct Mutex(std::sync::Mutex); +struct Guard<'a, T>(Option>); +impl Mutex { + fn new(v: T) -> Self { Mutex(std::sync::Mutex::new(v)) } + fn lock(&self) -> Guard<'_, T> { Guard(Some(self.0.lock().unwrap())) } +} +impl<'a, T> std::ops::Deref for Guard<'a, T> { + type Target = T; + fn deref(&self) -> &T { self.0.as_ref().unwrap() } +} +impl<'a, T> std::ops::DerefMut for Guard<'a, T> { + fn deref_mut(&mut self) -> &mut T { self.0.as_mut().unwrap() } +} +struct CondVar(std::sync::Condvar); +impl CondVar { + fn wait_interruptible_timeout(&self, g: &mut Guard<'_, T>, j: u64) -> CondVarTimeoutResult { + let inner = g.0.take().unwrap(); + let (inner, r) = self.0.wait_timeout(inner, Duration::from_millis(j)).unwrap(); + g.0 = Some(inner); + if r.timed_out() { CondVarTimeoutResult::Timeout } else { CondVarTimeoutResult::Woken { jiffies: j / 2 } } + } + fn notify_all(&self) { self.0.notify_all() } +} + +mod transfer { + pub struct Completed { pub status: u32, pub payload: Vec } + /// Same state rules as drivers/soc/apple/transfer.rs Reassembly. + pub struct Reassembly { pub active: Option, pub done: Option } + impl Reassembly { + pub fn begin(&mut self, opcode: u32) -> super::Result<()> { + if self.active.is_some() { return Err(super::EBUSY); } + self.done = None; + self.active = Some(opcode); + Ok(()) + } + pub fn take_done(&mut self) -> Option { self.done.take() } + pub fn abort(&mut self) { + if self.active.take().is_some() { + self.done = Some(Completed { status: 0xdead, payload: Vec::new() }); + } + } + /// The enclave's reply: it echoes the opcode in the payload. + pub fn on_reply(&mut self, opcode: u32) { + if self.active == Some(opcode) { + self.active = None; + self.done = Some(Completed { status: 0, payload: opcode.to_le_bytes().to_vec() }); + } + } + } +} + +struct SepData { + shutting_down: AtomicBool, + sbio_ready: AtomicBool, + sbio_rx: Mutex, + sbio_wq: CondVar, + sbio_exchange_lock: Mutex<()>, + to_enclave: std::sync::Mutex>, +} + +impl SepData { + fn sbio_start(&self, opcode: u16, _name: &'static CStr, _payload: &[u8]) -> Result<()> { + self.to_enclave.lock().unwrap().send(u32::from(opcode)).unwrap(); + Ok(()) + } +METHOD +} + +fn main() { + let (tx, rx) = mpsc::channel::(); + let sep: &'static SepData = Box::leak(Box::new(SepData { + shutting_down: AtomicBool::new(false), + sbio_ready: AtomicBool::new(true), + sbio_rx: Mutex::new(transfer::Reassembly { active: None, done: None }), + sbio_wq: CondVar(std::sync::Condvar::new()), + sbio_exchange_lock: Mutex::new(()), + to_enclave: std::sync::Mutex::new(tx), + })); + // The enclave answers each request after a short delay, as the mailbox does. + std::thread::spawn(move || { + for opcode in rx { + std::thread::sleep(Duration::from_millis(30)); + sep.sbio_rx.lock().on_reply(opcode); + sep.sbio_wq.notify_all(); + } + }); + let mut failures = 0; + for round in 0..20u16 { + let worker = std::thread::spawn(move || { + // e.g. CLEAR_STATE from the work item's sensor registration + sep.sbio_transfer_raw(0x10 + round, "WORKER", &[]) + .map(|d| (d.status, d.payload)) + }); + std::thread::sleep(Duration::from_millis(5)); + // e.g. LIST_IDENTITIES from GET_INFO on the next identify + let ioctl = sep.sbio_transfer_raw(0x40 + round, "IOCTL", &[]).map(|d| (d.status, d.payload)); + let worker = worker.join().unwrap(); + let want_w = Ok((0, u32::from(0x10 + round).to_le_bytes().to_vec())); + let want_i = Ok((0, u32::from(0x40 + round).to_le_bytes().to_vec())); + if worker != want_w || ioctl != want_i { + eprintln!("round {round}: worker {:?} ioctl {:?}", worker, ioctl); + failures += 1; + } + } + assert_eq!(failures, 0, "concurrent SBIO exchanges interfered"); +} +''' + + +@unittest.skipUnless(shutil.which('rustc'), 'rustc required for driver method tests') +class SbioExchangeTest(unittest.TestCase): + def test_concurrent_exchanges_each_get_their_own_reply(self): + program = HARNESS.replace('METHOD', method('sbio.rs', 'sbio_transfer_raw')) + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / 'test.rs').write_text(program) + subprocess.run(['rustc', '--edition=2021', str(root / 'test.rs'), + '-o', str(root / 'test')], check=True) + subprocess.run([str(root / 'test')], check=True) + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/aurora-sep/tests/test_sensor_wake.py b/tools/aurora-sep/tests/test_sensor_wake.py new file mode 100644 index 00000000000000..64fe6898a37fca --- /dev/null +++ b/tools/aurora-sep/tests/test_sensor_wake.py @@ -0,0 +1,117 @@ +# SPDX-License-Identifier: MIT +"""Execute the actual wake_sensor body with synthetic status/registration mocks. + +The post-registration status read applies to nonce-session (J700) profiles; +other profiles keep the patch decision on the status read before registration. +""" +from pathlib import Path +import subprocess +import tempfile +import unittest + +APPLE = Path(__file__).resolve().parents[3] / 'drivers/soc/apple' + + +class SensorWakeTest(unittest.TestCase): + def test_post_registration_status_drives_patch_readiness(self): + source = (APPLE / 'sbio.rs').read_text() + start = source.index(' fn wake_sensor(') + opening = source.index('{', start) + depth, end = 1, opening + 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + function = source[start:end] + program = r''' +#![allow(dead_code, unused_variables)] +use std::cell::RefCell; +macro_rules! dev_warn { ($($args:tt)*) => {} } +macro_rules! dev_info { ($($args:tt)*) => {} } +use std::sync::atomic::{AtomicU64, Ordering::Relaxed}; +mod shim { pub fn boottime_ns()->u64 {0} } +mod module_parameters { + pub struct Param(u8); + impl Param { pub fn value(&self)->&u8 {&self.0} } + #[allow(non_upper_case_globals)] pub static sensor_trace: Param = Param(0); +} +struct Profile { nonce_session: bool } +macro_rules! dev_err { ($($args:tt)*) => {} } +mod kernel { pub mod time { + pub struct Delta; + impl Delta { pub fn from_millis(_:i64)->Self {Self} } + pub mod delay { pub fn fsleep(_:super::Delta) {} } +}} +#[derive(Clone, Copy)] struct PatchLoaded(()); +#[derive(Default)] struct Mock { + reads: usize, registered: bool, patch: Option, next: Option, +} +thread_local! { static MOCK: RefCell = RefCell::new(Mock::default()); } +mod sensor { + pub const POWER_ON_READ_DELAYS_MS: [u32;1] = [0]; + pub const KNOWN_IDENTIFIERS: &[u16] = &[1]; + pub const PATCH_ACCEPTED: u8 = 0x29; + pub const STATE_NEEDS_PATCH: u8 = 9; + #[derive(Clone,Copy)] pub struct Status { pub state:u8, pub ack:u8, pub silent:bool } + pub struct Identifier; + pub enum Offset12 { Identifier(u16), Other } + pub struct Timing; + impl Timing { pub fn is_hardware(&self)->bool {true} pub fn name(&self)->&'static str {"mock"} } + pub fn power_source() {} + pub fn power_cycle()->bool {true} + pub fn cs_timing()->Timing {Timing} + pub fn status()->Result { + super::MOCK.with(|m| { let mut m=m.borrow_mut(); m.reads+=1; + if m.reads==1 {return Ok(Status{state:0,ack:PATCH_ACCEPTED,silent:false})} + assert!(m.registered, "post-registration read preceded registration"); + m.next.ok_or(()) + }) + } + impl Status { + pub fn is_silent(&self)->bool {self.silent} + pub fn offset12(&self)->Offset12 {Offset12::Identifier(1)} + pub fn identifier_to_register(&self)->Option {Some(Identifier)} + pub fn patch_ack(&self)->u8 {self.ack} + } +} +struct Driver { profile: Profile, sensor_power_ns: AtomicU64 } +impl Driver { + fn new()->Self { Driver { profile: Profile { nonce_session: false }, sensor_power_ns: AtomicU64::new(0) } } + fn sensor_ms(&self)->u64 {0} + fn register_sensor(&self,_:&sensor::Identifier)->bool { + MOCK.with(|m| {let mut m=m.borrow_mut();assert_eq!(m.reads,1);m.registered=true;}); true + } + fn apply_sensor_patch(&self,ack:u8)->Option { + MOCK.with(|m| {let mut m=m.borrow_mut();m.patch=Some(ack);}); + Some(PatchLoaded(())) + } +''' + program += function + '\n}\n' + r''' +fn main() { + for (next,ready,patch) in [ + (Some(sensor::Status{state:9,ack:0x29,silent:false}),true,Some(0x29)), + (None,false,None), + (Some(sensor::Status{state:0,ack:0x29,silent:true}),false,None), + (Some(sensor::Status{state:0,ack:0x29,silent:false}),true,None), + (Some(sensor::Status{state:0,ack:0,silent:false}),true,Some(0)), + ] { + MOCK.with(|m| *m.borrow_mut()=Mock{next,..Mock::default()}); + let driver = Driver { profile: Profile { nonce_session: true }, sensor_power_ns: AtomicU64::new(0) }; + assert_eq!(driver.wake_sensor().is_some(),ready); + MOCK.with(|m| {let m=m.borrow();assert_eq!(m.reads,2);assert!(m.registered);assert_eq!(m.patch,patch);}); + } + // Without a nonce session the pre-registration status decides, as before. + MOCK.with(|m| *m.borrow_mut()=Mock::default()); + let driver = Driver::new(); + assert!(driver.wake_sensor().is_some()); + MOCK.with(|m| {let m=m.borrow();assert_eq!(m.reads,1);assert!(m.registered);assert_eq!(m.patch,None);}); +} +''' + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / 'test.rs').write_text(program) + subprocess.run(['rustc', '--edition=2021', str(root / 'test.rs'), '-o', str(root / 'test')], check=True) + subprocess.run([str(root / 'test')], check=True) + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/aurora-userpages/.gitignore b/tools/aurora-userpages/.gitignore new file mode 100644 index 00000000000000..a27b2bf8b140da --- /dev/null +++ b/tools/aurora-userpages/.gitignore @@ -0,0 +1 @@ +/userpages diff --git a/tools/aurora-userpages/Makefile b/tools/aurora-userpages/Makefile new file mode 100644 index 00000000000000..c6d42e2d15efa0 --- /dev/null +++ b/tools/aurora-userpages/Makefile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0-only +CC ?= cc +CFLAGS ?= -O2 -g +CFLAGS += -Wall -Wextra +LDFLAGS += -Wl,-z,max-page-size=65536 + +.PHONY: all clean +all: userpages + +userpages: userpages.c ../../include/uapi/linux/prctl.h + $(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $< $(LDLIBS) -o $@ + +clean: + rm -f userpages diff --git a/tools/aurora-userpages/README.rst b/tools/aurora-userpages/README.rst new file mode 100644 index 00000000000000..64f4fcb0dd12ac --- /dev/null +++ b/tools/aurora-userpages/README.rst @@ -0,0 +1,34 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +Per-process userspace page-size launcher +======================================= + +Build with ``make`` in this directory. The running arm64 kernel must provide +Aurora's experimental per-process granule API. + +Examples:: + + userpages 4k FEXInterpreter ./x86-program + userpages 4k python3 script.py + userpages native program + userpages --status + +The choice becomes the calling thread's inherited exec preference: it applies +to the launched program and subsequent execs in its descendants. Fork retains +the address space and both exec preferences. Other threads and processes are +unchanged. ``native`` clears the inherited alternative preference. Ordinary +processes that never opt in continue to use the kernel's native default. + +``--once`` changes only the next successful exec; an existing inherited +preference remains available for later execs. For example, +``userpages --once native program`` runs one program with native pages and +retains any inherited 4K preference for its subsequent execs. Failed exec does +not consume the pending request. No selection changes a live address space. +AArch32 programs always use 4K when supported, otherwise native pages, +regardless of either preference. + +The kernel checks supported sizes and CPU capabilities. The current branch +supports 4K userspace on 16K/64K kernels and 16K userspace on 64K kernels. +64K userspace on a 16K kernel is not enabled. This launcher uses the private +PR_AURORA_{SET,GET}_{EXEC,DEFAULT}_PAGE_SIZE API in this tree; +it is not an upstream Linux ABI. diff --git a/tools/aurora-userpages/userpages.c b/tools/aurora-userpages/userpages.c new file mode 100644 index 00000000000000..02d9e4357224de --- /dev/null +++ b/tools/aurora-userpages/userpages.c @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include "../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include + +static int usage(const char *name, int status) +{ + fprintf(status ? stderr : stdout, + "Usage: %s [--once] {native|4k|16k|64k} COMMAND [ARG ...]\n" + " %s --status\n" + "Select the userspace page size for this program and its descendants.\n", + name, name); + return status; +} + +int main(int argc, char **argv) +{ + unsigned long size; + int once = argc > 1 && !strcmp(argv[1], "--once"); + int first = once ? 2 : 1; + + if (argc == 2 && !strcmp(argv[1], "--help")) + return usage(argv[0], 0); + if (argc == 2 && !strcmp(argv[1], "--status")) { + int pending = prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL); + int error = errno; + + printf("Current process page size: %lu bytes\n", getauxval(AT_PAGESZ)); + if (pending < 0) { + fprintf(stderr, "Page-size selection unavailable: %s\n", strerror(error)); + return 1; + } + int inherited = prctl(PR_AURORA_GET_DEFAULT_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL); + + if (inherited < 0) { + perror("Cannot read inherited page-size preference"); + return 1; + } + if (inherited) + printf("Inherited exec preference: %d bytes\n", inherited); + else + puts("Inherited exec preference: native default"); + if (pending) + printf("Next exec page size: %d bytes\n", pending); + else + puts("Next exec page size: inherited preference"); + return 0; + } + if (argc < first + 2) + return usage(argv[0], 2); + if (!strcmp(argv[first], "native")) + size = 0; + else if (!strcmp(argv[first], "4k")) + size = 4096; + else if (!strcmp(argv[first], "16k")) + size = 16384; + else if (!strcmp(argv[first], "64k")) + size = 65536; + else + return usage(argv[0], 2); + + if (!once && prctl(PR_AURORA_SET_DEFAULT_PAGE_SIZE, size, 0UL, 0UL, 0UL)) { + fprintf(stderr, "Cannot set inherited %s pages: %s\n", + argv[first], strerror(errno)); + return 1; + } + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, size, 0UL, 0UL, 0UL)) { + fprintf(stderr, "Cannot select %s userspace pages: %s\n", + argv[first], strerror(errno)); + return 1; + } + execvp(argv[first + 1], &argv[first + 1]); + int error = errno; + + fprintf(stderr, "Cannot execute %s: %s\n", argv[first + 1], strerror(error)); + return error == ENOENT ? 127 : 126; +} diff --git a/tools/testing/apple/backlight-golden.json b/tools/testing/apple/backlight-golden.json new file mode 100644 index 00000000000000..d1a8af2ab20118 --- /dev/null +++ b/tools/testing/apple/backlight-golden.json @@ -0,0 +1,297 @@ +[ + { + "nits": 197, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000a06840000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "59bc5374f6e05d6bcf2839b9522bdb9456074f80c7d542c0ecdc7e90cf7c1d16" + }, + { + "nits": 188, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000806740000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "05c226553a34876237f2eef41246a75d013cd29115e68437df03f594f2459941" + }, + { + "nits": 179, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000606640000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "e74693443c8a029a060cf3612556f93837a82cc3f3936975dcd4848432186067" + }, + { + "nits": 170, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000406540000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "67521cee14ce162f0a733a815b1f059f3fe394c0c5e87bfe178f14b3acceb52c" + }, + { + "nits": 161, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000206440000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "f2996dbaff64ac3d653cd4b420df919c418697fcf8dc4f995f562e5839efcef2" + }, + { + "nits": 152, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000006340000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "b134853efd740adc11f258a206b4deb098df1765191c073178e033df83d17148" + }, + { + "nits": 143, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000e06140000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "1488c635d8d55ab49f73f9d9698a4b3e0f8fe95fcf925b73c21f4846fa7188f2" + }, + { + "nits": 134, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000c06040000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "132f2aba31bdc58731510361e6b934284ac81baeed5346dc235b615a43f6d798" + }, + { + "nits": 125, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000405f40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "a4f3e35516ba00acbed9c6c04ebb564334dab983b1a016f570dfe739ea2a9ae3" + }, + { + "nits": 116, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000005d40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "837951003a3632bb5c063e3ee70606583af2f843ef722864cbf461ef01fc5a81" + }, + { + "nits": 106, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000805a40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "a68f25f1ce7f63c335781888df47a8eb12f863c37795b2377d69c270dc063cac" + }, + { + "nits": 97, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000405840000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "6c5108c9aadd93c72b2f0d5b72adb09f55ca63159f909d0876714bbf62f80ea5" + }, + { + "nits": 88, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000005640000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "b2db893e953fda0c249ecf7c674acfd2018d1267985cc09d842b2f38910b2189" + }, + { + "nits": 79, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000c05340000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "321da86969fcfdb11f7d6e508a627bb0f9146defd48c4fa0d0cd76d7b63fd49b" + }, + { + "nits": 70, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000805140000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "66443759f999549d7d9b076c338cc9f5ba7b2606ec5f764f14a17f8a23f28acb" + }, + { + "nits": 61, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000804e40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "4c3447c9290bdda6fc9f38d024f8c8482fc7692b61243680cf6dc2012dd8d8a1" + }, + { + "nits": 52, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000004a40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "ce727d2eb98cbdeba8be5215577c2486ad8d5ad1b7ca1db2358d7c27f0cc2f1f" + }, + { + "nits": 43, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000804540000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "00f94f04f7e431494dddf154cf8e68a1e3fe91afa3ab18d41c52223d582112ae" + }, + { + "nits": 36, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000004240000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "4ae2c36b9d9a773bcfaf68c74b81e3761f4b0da68ffff4f7a55194207b313828" + }, + { + "nits": 59, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000804d40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "aa653a1c3d85ad9cf3a487aebdfafd0332c6d81335e9bc051292cab4610997de" + }, + { + "nits": 92, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000005740000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "a57f482514132d36aa5c46026b7c579bd74b1d21b84e5ed67e4123eab72e1054" + }, + { + "nits": 125, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000405f40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "6260e7be2d29193f51a0fe60337efcdc9c6e73b469f527afb9f0ff83765467ec" + }, + { + "nits": 140, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000806140000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "0eab7f3ebd304ddb5f40d2c8bc1ecd2110b166faf70f89b98a2521e0b6c7ac95" + }, + { + "nits": 200, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000006940000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "98df21464117081715a48b0076304aad0f2b8307ee682c20a7feafd9a1be9202" + }, + { + "nits": 287, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000f07140000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "7f5df7ed6f804a3340c9782698f5292e3b4117f2eb9943778010936a6b2746a4" + }, + { + "nits": 375, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000707740000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "994697bd5bb64697f4d2bbbf2311a7c383f356161184e2a9aeea610515dda25e" + }, + { + "nits": 400, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000007940000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "a9fbdac983180e10843468b14c2cade1ee7d0e8711e02fce56489112c91b38e0" + }, + { + "nits": 335, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000f07440000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "4cf3e7680e17a183e40e54b69d54693f43f3643346de050beb5c9cb73214b8f8" + }, + { + "nits": 249, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000206f40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "9b174a609f7fda9e379923c5e1fba03952b6f44d3ad031dfa8660a015326725e" + }, + { + "nits": 163, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000606440000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "bea9a384a1ceb723ddda8416748e20057e67627d768e9d1a81207d9f1a21d812" + }, + { + "nits": 131, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000606040000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "2445a2a32d0cbf27f6eb394bc48ff08c58ca88a32e6a6f9cbe633e983ab05256" + }, + { + "nits": 185, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000206740000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "1a23772d33bedf58b4c566ae84267a0631f7ced0d881ac9470667ec2eab477bc" + }, + { + "nits": 264, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000807040000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "3bd83da635cd09c608e2124c75e9aaa9ed9337622e0a818c500ddf83d8146d33" + }, + { + "nits": 337, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000107540000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "d44097a52f6cef9ca7c4ddf39897bf67a9915b520b154d50d5210595997f0495" + }, + { + "nits": 357, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000507640000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "51f19fe9d95e6d95b8b81b158a3d5c2db391973a66695858610ae24ade44c0d8" + }, + { + "nits": 0, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000000000000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "3f3d1f6ae52991efa63ac64d74e3e9f28e8b000791732102a69a5c05452d2512" + }, + { + "nits": 140, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000806140000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "5283a0beb8119ca257f5c00ae267598427c2bb01a472395c83b325f8c7a4b81a" + }, + { + "nits": 212, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000806a40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "d1bdce12296d366b8315c73d850fc80e92720748aff8ce3ff28a136e6104f0c8" + }, + { + "nits": 313, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000907340000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "a86f277bb5b34a2b13301d2c3464db76325d0bacd6009b5e46329fe97c82df60" + }, + { + "nits": 413, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000d07940000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "bca47e8efde3063740bd65f174ccc0b502b5ef2350cc821397679724ecf7d2d6" + }, + { + "nits": 454, + "enabled": true, + "maximum": 525, + "fragment": "010000000101010001000000000000607c40000000000000000000000000006880400000000000688040000000000000f03f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f03f", + "payload_sha256": "17c4a25be2d6cefb43cb282859f500415e3902be3d9d9ef5ab9045e6e9c6af53" + }, + { + "nits": 100, + "maximum": 525, + "enabled": false, + "fragment": "00000000000000000000000000000000000000000000000000000000000000688040000000000068804000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + "payload_sha256": "7de62e12dc678b02f3b47b68ab369efc651cc05d765d2e024652583e410461da", + "provenance": "run24 clean ordinary startup presents" + } +] diff --git a/tools/testing/apple/check-backlight-retry.py b/tools/testing/apple/check-backlight-retry.py new file mode 100644 index 00000000000000..32d1fc40580f26 --- /dev/null +++ b/tools/testing/apple/check-backlight-retry.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Native regression for rejected brightness transactions and bounded retries. + +Executes actual reservation, failure, queue-drain, retry-work and completion +functions with mocked scheduling and transport. Does not submit firmware RPCs. +""" +import argparse +import hashlib +import json +from pathlib import Path +import re +import subprocess + +p = argparse.ArgumentParser(description=__doc__) +p.add_argument('--output', type=Path, required=True) +p.add_argument('--cc', default='clang') +a = p.parse_args() +root = Path(__file__).resolve().parents[3] +src = root / 'drivers/gpu/drm/apple' +a.output.mkdir(parents=True, exist_ok=True) + + +def function(file, name): + text = (src / file).read_text() + match = re.search(r'^\w[\w\s*]*?\b' + name + r'\(', text, re.M) + assert match, name + start = text.index('{', match.end()) + pos, depth = start + 1, 1 + while depth: + depth += (text[pos] == '{') - (text[pos] == '}') + pos += 1 + return text[match.start():pos] + + +policy = re.sub(r'^#include ]+>\n', '', + (src / 'dcp_backlight.h').read_text(), flags=re.M) +functions = '\n'.join(function(f, n) for f, n in ( + ('dcp_backlight.c', 'dcp_backlight_complete'), + ('dcp_backlight.c', 'dcp_backlight_pending'), + ('dcp_backlight.c', 'dcp_backlight_retry_delay'), + ('iomfb.c', 'iomfb_present_failed_h17p'), + ('iomfb.c', 'iomfb_present_complete_h17p'), + ('iomfb.c', 'iomfb_queue_advance'), ('iomfb.c', 'iomfb_backlight_retry'))) +prefix = r''' +#include +#include +#include +#include +#include +#include +#include +#include +#include +typedef uint8_t u8;typedef uint32_t u32;typedef uint64_t u64; +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x)=(v)) +#define lockdep_assert_held(x) ((void)(x)) +#define spin_lock_irqsave(x,f) do { (void)(x);(f)=0; } while(0) +#define spin_unlock_irqrestore(x,f) do { (void)(x);(void)(f); } while(0) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#define to_delayed_work(p) container_of(p,struct delayed_work,work) +#define msecs_to_jiffies(x) (x) +#define IOMFB_OPAQUE_X_WAITING 0 +#define IOMFB_OPAQUE_X_NEEDED 1 +struct work_struct {int id;};struct delayed_work {struct work_struct work;}; +struct list_head {int empty;}; +struct iomfb_transaction { + struct dcp_backlight_present backlight; + bool backlight_reserved,backlight_failed,brightness_only,completed; + void (*release)(struct iomfb_transaction *); +}; +struct apple_dcp { + bool crashed;void *dev;struct work_struct bl_update_wq,vblank_wq; + struct {int lock;struct dcp_backlight_state state;} backlight; + struct {bool pending;} present_state_h17p; + struct {int lock;bool stopped,backlight_queued;struct iomfb_transaction *active; + struct delayed_work timeout,backlight_retry;struct work_struct work; + struct list_head pending;int opaque_x_state; + } iomfb; +}; +static unsigned delay,scheduled,released,discarded;static bool idle=true; +static void *system_wq; +static bool iomfb_channels_idle(struct apple_dcp *d) {(void)d;return idle;} +static int iomfb_enqueue_opaque_x(struct apple_dcp *d) {(void)d;return 0;} +static void iomfb_discard_pending(struct apple_dcp *d) {(void)d;discarded++;} +static bool list_empty(struct list_head *l) {return l->empty;} +static void cancel_delayed_work(struct delayed_work *w) {(void)w;} +static void schedule_work(struct work_struct *w) {assert(w->id==1);scheduled++;} +static void mod_delayed_work(void *q,struct delayed_work *w,unsigned ms) { + (void)q;assert(w->work.id==2);delay=ms; +} +static void dev_warn_ratelimited(void *d,const char *msg) {(void)d;(void)msg;} +static void iomfb_scanout_complete_h17p(struct apple_dcp *d) {(void)d;} +static void atomic_cmpxchg(int *p,int old,int value) {if(*p==old)*p=value;} +static void release(struct iomfb_transaction *t) {(void)t;released++;} +''' +# The policy header defines structs before the fixture needs them. +prefix = prefix[:prefix.index('struct work_struct')] + policy + prefix[prefix.index('struct work_struct'):] +main = r''' +static void init(struct apple_dcp *d) { + u32 inherited; + memset(d,0,sizeof(*d));d->bl_update_wq.id=1;d->iomfb.backlight_retry.work.id=2; + assert(!dcp_bl_takeover_nits(509,36000,&inherited)&&inherited==36); + d->iomfb.pending.empty=1;assert(!dcp_bl_init(&d->backlight.state,509,true,inherited,false,0)); + assert(dcp_bl_seed(&d->backlight.state,36));assert(!dcp_bl_dpms(&d->backlight.state,true)); + assert(!dcp_bl_request(&d->backlight.state,140,false,false)); + delay=scheduled=released=discarded=0;idle=true; +} +static void reserve(struct apple_dcp *d,struct iomfb_transaction *t) { + memset(t,0,sizeof(*t));t->brightness_only=true;t->release=release; + assert(!dcp_bl_prepare(&d->backlight.state,true,&t->backlight)); + t->backlight_reserved=true;d->iomfb.active=t;d->iomfb.backlight_queued=true; +} +static void reject(struct apple_dcp *d) { + iomfb_present_failed_h17p(d);d->present_state_h17p.pending=false; + iomfb_queue_advance(d); +} +int main(void) { + struct apple_dcp d;struct iomfb_transaction t; + init(&d);reserve(&d,&t);reject(&d); + assert(delay==100&&!d.iomfb.active&&!d.iomfb.backlight_queued&&released==1); + assert(d.backlight.state.actual==36&&d.backlight.state.dirty&&!t.completed); + /* The deferred worker retries on retained pixels, with NO new user request. */ + iomfb_backlight_retry(&d.iomfb.backlight_retry.work);assert(scheduled==1); + reserve(&d,&t);assert(!iomfb_present_complete_h17p(&d));iomfb_queue_advance(&d); + assert(d.backlight.state.actual==140&&!d.backlight.state.dirty&&!d.backlight.state.retries); + init(&d); + for(unsigned i=0;i<4;i++) { + reserve(&d,&t);delay=0;reject(&d);assert(delay==(i<3?(100U<\n#include \ntypedef uint8_t u8;typedef uint32_t u32;typedef uint64_t u64;\n') +(include / 'limits.h').write_text('#include \n#define U32_MAX UINT32_MAX\n') +(include / 'errno.h').write_text('#include \n#ifndef ENODATA\n#define ENODATA 61\n#endif\n') +prelude = r''' +#include +#include +#include +#include +#include +#include +#include "dcp_backlight.h" +#define DCP_FIRMWARE_H17P 17 +#define DCP_IOMFB_METHODS_H17G 1 +#define MAX_BRIGHTNESS_PART2 510U +#define SWAP_SURFACES 4 +#define READ_ONCE(x) (x) +#define min(a,b) ((a)<(b)?(a):(b)) +#define lockdep_assert_held(x) ((void)(x)) +#define mutex_lock(x) ((void)(x)) +#define mutex_unlock(x) ((void)(x)) +#define spin_lock_irqsave(x,f) do { (void)(x);(f)=0; } while(0) +#define spin_unlock_irqrestore(x,f) do { (void)(x);(void)(f); } while(0) +#define GENMASK_ULL(h,l) ((~UINT64_C(0) >> (63-(h))) & (~UINT64_C(0) << (l))) +#define fls(x) (32-__builtin_clz(x)) +#define BACKLIGHT_PLATFORM 1 +#define BACKLIGHT_SCALE_LINEAR 2 +struct device { int unused; }; +struct backlight_device { int unused; }; +struct backlight_properties { int type;u32 brightness,max_brightness;int scale; }; +struct dcp_present_h17p { u8 swap[0x588];u8 guard[32]; }; +struct dcp_swap_submit_req_h17p { u8 surf_null[SWAP_SURFACES]; }; +struct iomfb_transaction { + struct dcp_backlight_present backlight; + bool backlight_reserved,backlight_failed,brightness_only,completed; +}; +struct apple_dcp { + int fw_compat;struct { int iomfb_method_profile; } hw; + bool panel;void *crtc;struct device *dev; + int bl_register_mutex,bl_register_wq; + struct { u32 maximum;u32 nits;struct backlight_device *bl_dev;u32 dac; } brightness; + struct { int lock;struct dcp_backlight_state state;void (*kick)(struct apple_dcp *); } backlight; + struct { int lock;struct iomfb_transaction *active;int opaque_x_state; } iomfb; +}; +static int schedule_count,register_count,retire_count,dac_count; +static struct backlight_properties registered; +static const int dcp_backlight_ops; +static bool dcp_has_panel(struct apple_dcp *dcp) { return dcp->panel; } +static void schedule_work(int *work) { (void)work;schedule_count++; } +static void iomfb_backlight_kick(struct apple_dcp *dcp) { (void)dcp; } +static void iomfb_scanout_complete_h17p(struct apple_dcp *dcp) { (void)dcp;retire_count++; } +static u32 calculate_dac(struct apple_dcp *dcp,u32 nits) { (void)dcp;dac_count++;return nits; } +static struct backlight_device *devm_backlight_device_register(struct device *dev,const char *name, + struct device *parent,struct apple_dcp *dcp,const int *ops,const struct backlight_properties *props) { + static struct backlight_device device; + (void)dev;(void)parent;(void)dcp;(void)ops; + assert(!strcmp(name,"apple-panel-bl"));registered=*props;register_count++;return &device; +} +#define IS_ERR(p) (!(p)) +#define PTR_ERR(p) ((void)(p),-ENOMEM) +static void put_unaligned_le64(u64 v,void *p) { for(int i=0;i<8;i++)((u8 *)p)[i]=v>>(8*i); } +static void put_unaligned_le32(u32 v,void *p) { for(int i=0;i<4;i++)((u8 *)p)[i]=v>>(8*i); } +static void put_unaligned_le16(uint16_t v,void *p) { for(int i=0;i<2;i++)((u8 *)p)[i]=v>>(8*i); } +enum { IOMFB_OPAQUE_X_WAITING, IOMFB_OPAQUE_X_NEEDED }; +static int atomic_cmpxchg(int *value,int expected,int desired) { + int old=*value;if(old==expected)*value=desired;return old; +} +int iomfb_configure_backlight_h17p(struct apple_dcp *,u32,bool,u32,bool,u32); +''' +functions = [ + ('iomfb_h17p.c', 'iomfb_nits_binary64_h17p'), + ('iomfb_h17p.c', 'iomfb_encode_backlight_h17p'), + ('iomfb.c', 'iomfb_uses_queue'), + ('dcp_backlight.c', 'dcp_backlight_active'), + ('dcp_backlight.c', 'dcp_backlight_configure'), + ('iomfb.c', 'iomfb_configure_backlight_h17p'), + ('dcp_backlight.c', 'dcp_backlight_takeover'), + ('dcp_backlight.c', 'dcp_backlight_register'), + ('dcp_backlight.c', 'dcp_backlight_prepare'), + ('dcp_backlight.c', 'dcp_backlight_complete'), + ('iomfb.c', 'iomfb_apply_backlight_h17p'), + ('iomfb.c', 'iomfb_present_failed_h17p'), + ('iomfb.c', 'iomfb_present_complete_h17p'), +] +main = r''' +static struct apple_dcp fixture(void) { + return (struct apple_dcp){.fw_compat=DCP_FIRMWARE_H17P,.panel=true, + .crtc=(void *)1,.brightness.maximum=525}; +} +static u64 level_bits(const struct dcp_present_h17p *wire) { + u64 value=0;for(int i=0;i<8;i++)value|=(u64)wire->swap[0x35e + i]<<(i*8);return value; +} +static double level(const struct dcp_present_h17p *wire) { + u64 bits=level_bits(wire);double value;memcpy(&value,&bits,sizeof(value));return value; +} +static void outside_unchanged(const struct dcp_present_h17p *wire) { + const u8 *data=(const u8 *)wire; + for(size_t i=0;i=0x32f && i<0x33c)assert(!data[i]); + else if(i<0x354||i>=0x3ee)assert(data[i]==0xa5); + } +} +int main(int argc,char **argv) { + struct dcp_present_h17p wire; + if(argc==3) { + memset(&wire,0xa5,sizeof(wire)); + iomfb_encode_backlight_h17p(&wire,strtoul(argv[1],NULL,0),525,atoi(argv[2])); + outside_unchanged(&wire); + for(int i=0x354;i<0x3ee;i++)printf("%02x",wire.swap[i]);puts("");return 0; + } + if(argc==2) { + for(u32 nits=0;nits<510;nits++)for(int update=0;update<2;update++) { + memset(&wire,0xa5,sizeof(wire)); + iomfb_encode_backlight_h17p(&wire,nits,525,update); + outside_unchanged(&wire); + printf("%u %d ",nits,update); + for(int i=0x354;i<0x3ee;i++)printf("%02x",wire.swap[i]);puts(""); + } + return 0; + } + struct apple_dcp dcp=fixture(); + assert(dcp_backlight_register(&dcp)==-ENODATA); + assert(!register_count); + assert(dcp_backlight_takeover(&dcp,509001)==-ERANGE); + assert(!dcp_backlight_active(&dcp)); + assert(dcp_backlight_takeover(&dcp,100000)==0); + assert(dcp.backlight.state.target==100 && dcp.backlight.state.actual==100); + assert(!dcp.backlight.state.dirty && schedule_count==1); + assert(dcp_backlight_takeover(&dcp,400000)==0); + assert(dcp_backlight_takeover(&dcp,0)==0); + assert(dcp.backlight.state.target==100 && schedule_count==1); + assert(dcp_backlight_register(&dcp)==0 && register_count==1); + assert(registered.brightness==100 && registered.max_brightness==509 && !dac_count); + assert(dcp.backlight.state.controlled); + struct iomfb_transaction transaction={.brightness_only=true}; + struct dcp_swap_submit_req_h17p request={.surf_null={0,1,1,1}}; + dcp.iomfb.active=&transaction; + struct apple_dcp unready=fixture(); + memset(&wire,0xa5,sizeof(wire)); + assert(iomfb_apply_backlight_h17p(&unready,&request,&wire)); + assert(!wire.swap[0x354] && !level_bits(&wire)); + assert(wire.swap[0x374] || wire.swap[0x375]); + unready.brightness.maximum=0; + memset(&wire,0xa5,sizeof(wire)); + assert(!iomfb_apply_backlight_h17p(&unready,&request,&wire)); + assert(wire.swap[0x354]==0xa5); + dcp.brightness.maximum=0; + assert(!iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + dcp.brightness.maximum=525; + memset(&wire,0xa5,sizeof(wire)); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + for(int i=0x354;i<0x366;i++)assert(!wire.swap[i]); + assert(!transaction.backlight_reserved); + assert(dcp_bl_request(&dcp.backlight.state,400,false,false)==0); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==400 && dcp.backlight.state.actual==100); + assert(transaction.backlight_reserved); + assert(!dcp_backlight_complete(&dcp,transaction.backlight.sequence+1,true)); + assert(dcp_bl_request(&dcp.backlight.state,140,false,false)==0); + assert(!iomfb_present_complete_h17p(&dcp)); + assert(dcp.backlight.state.actual==400 && dcp.backlight.state.dirty && !retire_count); + assert(dcp.iomfb.opaque_x_state==IOMFB_OPAQUE_X_WAITING); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==140); + iomfb_present_failed_h17p(&dcp); + assert(dcp.backlight.state.actual==400 && dcp.backlight.state.dirty); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(!iomfb_present_complete_h17p(&dcp)); + assert(dcp.backlight.state.actual==140 && !retire_count); + for(int i=0;i<20;i++) { + assert(dcp_bl_dpms(&dcp.backlight.state,false)==0); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==0 && wire.swap[0x354]==1); + assert(!iomfb_present_complete_h17p(&dcp)); + assert(dcp.backlight.state.actual==0 && dcp.backlight.state.target==140); + assert(dcp_bl_dpms(&dcp.backlight.state,true)==0); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==140); + assert(!iomfb_present_complete_h17p(&dcp)); + } + assert(!retire_count); + assert(dcp_bl_dpms(&dcp.backlight.state,false)==0); + assert(dcp_bl_request(&dcp.backlight.state,400,false,false)==0); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==0 && dcp.backlight.state.target==400); + assert(!iomfb_present_complete_h17p(&dcp)); + assert(dcp_bl_dpms(&dcp.backlight.state,true)==0); + assert(iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(level(&wire)==400); + assert(!iomfb_present_complete_h17p(&dcp)); + assert(dcp_bl_request(&dcp.backlight.state,36,false,false)==0); + memset(request.surf_null,1,sizeof(request.surf_null)); + memset(&wire,0xa5,sizeof(wire)); + assert(!iomfb_apply_backlight_h17p(&dcp,&request,&wire)); + assert(wire.swap[0x354]==0xa5 && dcp.backlight.state.dirty); + struct apple_dcp h17g=fixture();h17g.hw.iomfb_method_profile=DCP_IOMFB_METHODS_H17G; + assert(dcp_backlight_takeover(&h17g,100000)==-EINVAL); + assert(!h17g.backlight.state.ready); + struct apple_dcp external=fixture();external.panel=false; + assert(dcp_backlight_takeover(&external,100000)==-EINVAL); + struct apple_dcp unlinked=fixture();unlinked.crtc=NULL; + assert(dcp_backlight_takeover(&unlinked,100000)==-EINVAL); + struct apple_dcp zero=fixture(); + assert(dcp_backlight_takeover(&zero,0)==0); + assert(zero.backlight.state.target==0 && !zero.backlight.state.dirty); + transaction = (struct iomfb_transaction){.brightness_only=false}; + dcp.iomfb.active = &transaction; + dcp.iomfb.opaque_x_state = IOMFB_OPAQUE_X_WAITING; + assert(iomfb_present_complete_h17p(&dcp)); + assert(dcp.iomfb.opaque_x_state == IOMFB_OPAQUE_X_NEEDED); + + puts("Actual takeover/registration/reservation/accepted completion/rejection/20 DPMS cycles passed"); + return 0; +} +''' +source = prelude + '\n'.join(function(f, n) for f, n in functions) + main +(out / 'fixture.c').write_text(source) +argv = [args.cc, '-std=gnu11', '-Wall', '-Wextra', '-Werror', '-fsanitize=address,undefined', + '-I' + str(out / 'include'), '-I' + str(src), str(out / 'fixture.c'), '-o', str(out / 'fixture')] +build = subprocess.run(argv, text=True, capture_output=True) +(out / 'compile.log').write_text(build.stdout + build.stderr) +assert not build.returncode, build.stderr +run = subprocess.run([str(out / 'fixture')], text=True, capture_output=True) +(out / 'run.log').write_text(run.stdout + run.stderr) +assert not run.returncode, run.stdout + run.stderr +# All measured brightness fragments, including DPMS zero and ordinary startup. +golden = json.loads(Path(__file__).with_name('backlight-golden.json').read_text()) +for case in golden: + fragment = subprocess.check_output([str(out / 'fixture'), str(case['nits']), str(int(case['enabled']))], text=True).strip() + assert fragment == case['fragment'], (case, fragment) +# Exact integer binary64 throughout the exposed range, plus disabled updates. +import struct +range_output = subprocess.check_output([str(out / 'fixture'), 'range'], text=True) +for line in range_output.splitlines(): + n, enabled, encoded = line.split() + nits, enabled = int(n), bool(int(enabled)) + expected = bytearray(0x9a) + expected[0x1a:0x22] = expected[0x22:0x2a] = struct.pack(' +#include +#include +#include +#include +#include +typedef uint8_t u8;typedef uint16_t u16;typedef uint32_t u32;typedef uint64_t u64; +#define GFP_KERNEL 0 +#define BLOCK_SHIFT 6 +#define GETBUF_SIZE 0xffffU +#define GETBUF_TAG 0xffff0000U +#define RBEP_TYPE 0 +#define RBEP_GETBUF_ACK 0 +#define GETBUF_ACK_DVA 0 +#define FIELD_GET(mask,x) ((x)&(mask)) +#define FIELD_PREP(mask,x) ((void)(mask),(u64)(x)) +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x)=(v)) +#define IS_ERR_OR_NULL(x) (!(x)||(uintptr_t)(x)>(uintptr_t)-4096) +#define DRM_MODE_CONNECTOR_eDP 14 +struct work_struct {int id;};struct delayed_work {struct work_struct work;}; +struct drm_device {int refs;bool unplugged;}; +struct apple_drm_private {struct drm_device drm;}; +struct device {void *data;int domain_users;}; +struct platform_device {struct device dev;}; +struct apple_crtc {struct {struct drm_device *dev;} base;}; +struct apple_connector {struct work_struct hotplug_wq;}; +struct apple_rtkit {bool rx_stopped;}; +struct apple_dcp; +struct apple_dcp_afkep { + struct apple_dcp *dcp;void *wq;void *bfr;void *debugfs_entry; + u64 bfr_dma;u32 bfr_size;u16 bfr_tag; +}; +struct apple_dcp { + bool retain_dma,quiescing,drm_retained,piodma_created;int connector_type,hdmi_hpd_irq; + struct device *dev;struct apple_crtc *crtc; + struct apple_connector *connector,*typec_connector; + struct platform_device *piodma;void *iommu_dom,*typec_mux,*shmem,*clk,*clk_194; + struct apple_rtkit *rtk; + struct apple_dcp_afkep *avep,*dptxep,*ibootep,*systemep,*dcpavservep; + struct work_struct bl_register_wq,bl_update_wq,vblank_wq; + struct delayed_work typec_reconnect_wq,typec_fabric_retrain_wq; +}; +static int dma_frees,domain_drops,pio_destroys,pio_puts,firmware_stops,queue_stops; +static int raw_rings,managed_rings,work_drains; +static int context_frees; +static void *ring; +static struct device *component; +static void dcp_comp_unbind(struct device *,struct device *,void *); +static void kfree(void *p) {assert(p);context_frees++;} +static void debugfs_remove_recursive(void *p) {(void)p;} +static void dev_warn(struct device *d,const char *fmt) {(void)d;(void)fmt;} +static void dev_err(struct device *d,const char *fmt,...) {(void)d;(void)fmt;} +static void *dev_get_drvdata(struct device *d) {return d->data;} +static void dev_set_drvdata(struct device *d,void *p) {d->data=p;} +static void drm_dev_unplug(struct drm_device *d) {d->unplugged=true;} +static void drm_atomic_helper_shutdown(struct drm_device *d) {assert(d->unplugged);} +static void component_unbind_all(struct device *m,void *p) {dcp_comp_unbind(component,m,p);} +static void iommu_device_unuse_default_domain(struct device *d) {domain_drops++;d->domain_users--;} +static void apple_rtkit_free(struct apple_rtkit *r) {r->rx_stopped=true;dma_frees++;} +static void apple_rtkit_free_retaining_buffers(struct apple_rtkit *r) {r->rx_stopped=true;} +static void of_platform_device_destroy(struct device *d,void *p) {(void)d;(void)p;pio_destroys++;} +static void put_device(struct device *d) {(void)d;pio_puts++;} +static void drm_dev_get(struct drm_device *d) {d->refs++;} +static void iomfb_queue_stop(struct apple_dcp *d) {assert(d->quiescing||!d->retain_dma);queue_stops++;} +static void destroy_workqueue(void *w) {assert(w);work_drains++;} +static void cancel_work_sync(struct work_struct *w) {(void)w;} +static void cancel_delayed_work_sync(struct delayed_work *w) {(void)w;} +static void disable_irq(int i) {(void)i;} +static void typec_mux_put(void *p) {(void)p;} +static void av_service_disconnect(struct apple_dcp *d) {(void)d;firmware_stops++;} +static void afk_shutdown(struct apple_dcp_afkep *e) {(void)e;firmware_stops++;} +static void iomfb_shutdown(struct apple_dcp *d) {(void)d;firmware_stops++;} +static void devm_clk_put(struct device *d,void *c) {(void)d;(void)c;} +static void trace_afk_getbuf(struct apple_dcp_afkep *e,u32 size,u16 tag) {(void)e;(void)size;(void)tag;} +static void afk_send(struct apple_dcp_afkep *e,u64 msg) {(void)e;(void)msg;} +static void *dma_alloc_coherent(struct device *d,u32 size,u64 *addr,int flags) { + (void)d;(void)flags;raw_rings++;ring=calloc(1,size);assert(ring);*addr=123;return ring; +} +static void *dmam_alloc_coherent(struct device *d,u32 size,u64 *addr,int flags) { + (void)d;(void)flags;managed_rings++;ring=calloc(1,size);assert(ring);*addr=123;return ring; +} +''' +main = r''' +int main(void) { + unsigned cases=0; + for(int live=0;live<2;live++)for(int created=0;created<2;created++)for(int linked=0;linked<2;linked++)for(int master=0;master<2;master++) { + struct device dev={.domain_users=1};struct platform_device pio={0}; + struct apple_drm_private apple={.drm={.refs=1}}; + struct device main={.data=&apple};struct apple_crtc crtc={.base={&apple.drm}}; + struct apple_rtkit rtk={0};struct apple_connector con={0}; + struct apple_dcp d={.retain_dma=live,.dev=&dev,.crtc=linked?&crtc:NULL, + .connector=&con,.piodma=&pio,.piodma_created=created,.iommu_dom=&dev, + .rtk=&rtk,.shmem=&dev,.connector_type=DRM_MODE_CONNECTOR_eDP}; + struct apple_dcp_afkep ep[5]={0}; + for(int i=0;i<5;i++)ep[i]=(struct apple_dcp_afkep){.dcp=&d,.wq=&dev}; + d.avep=&ep[0];d.dptxep=&ep[1];d.ibootep=&ep[2];d.systemep=&ep[3];d.dcpavservep=&ep[4];dev.data=&d; + dma_frees=domain_drops=pio_destroys=pio_puts=firmware_stops=queue_stops=0; + raw_rings=managed_rings=work_drains=context_frees=0; + afk_getbuf(&ep[0],1);assert(ep[0].bfr_size==64); + afk_getbuf(&ep[0],1);assert(raw_rings+managed_rings==1); + component=&dev; + if(master) {apple_drm_uninit(&main);assert(!main.data&&apple.drm.unplugged);} + else dcp_comp_unbind(&dev,&dev,NULL); + /* Component devres release runs even after an early unbind return. */ + dcp_release_rtkit(&d);assert(rtk.rx_stopped&&!d.rtk); + /* Driver removal also releases probe resources and its domain action. */ + dcp_release_dma_domain(&d); + if(live) { + assert(d.quiescing&&!dma_frees&&!domain_drops&&dev.domain_users==1); + assert(d.piodma==&pio&&d.iommu_dom==&dev&&!pio_destroys&&!pio_puts); + assert(raw_rings==1&&!managed_rings&&!firmware_stops&&work_drains==5); + assert(apple.drm.refs==(linked?2:1)); + dcp_comp_unbind(&dev,&dev,NULL);assert(apple.drm.refs==(linked?2:1)); + dcp_release_piodma_iommu_dev(&d);assert(!pio_destroys&&!pio_puts); + } else { + assert(dma_frees==1&&domain_drops==1&&dev.domain_users==0); + assert(!d.piodma&&!d.iommu_dom&&managed_rings==1&&!raw_rings); + assert(pio_destroys==created&&pio_puts==!created&&firmware_stops>0); + assert(apple.drm.refs==1); + } + for(int i=0;i<5;i++)afk_release_context(&ep[i]); + dcp_release_context(&d);assert(context_frees==(live?0:6)); + afk_quiesce(NULL);afk_quiesce((struct apple_dcp_afkep *)(intptr_t)-12); + /* Simulated machine reset: test memory can now be reclaimed. */ + free(ring);cases++; + } + /* A failed bind has no unbind callback: resource cleanup must close rebind. */ + { + struct device dev={0};struct platform_device pio={0};struct apple_rtkit rtk={0}; + struct apple_dcp d={.dev=&dev,.retain_dma=true,.piodma=&pio,.rtk=&rtk}; + assert(!d.quiescing);dcp_release_piodma_iommu_dev(&d); + assert(d.quiescing&&d.piodma==&pio);cases++; + d.quiescing=false;dcp_release_rtkit(&d); + assert(d.quiescing&&rtk.rx_stopped&&!d.rtk);cases++; + } + puts("PASS: 18 actual-source component, master-unwind, driver-resource-release and failed-bind cases; live DMA/domain/rings retained; legacy releases preserved"); + assert(cases==18);return 0; +} +''' +code = a.output / 'lifetime.c' +code.write_text(prefix + functions + main) +command = [a.cc, '-std=gnu11', '-O1', '-g', '-Wall', '-Wextra', '-Werror', + '-Wno-unused-parameter', '-fsanitize=address,undefined', str(code), + '-o', str(a.output / 'lifetime')] +subprocess.run(command, check=True) +result = subprocess.check_output([str(a.output / 'lifetime')], text=True) +(a.output / 'receipt.json').write_text(json.dumps(dict(command=command, result=result, + functions_sha256=hashlib.sha256(functions.encode()).hexdigest(), + scope='Native resource-release simulation; not a live firmware unbind test'), indent=2) + '\n') +print(result) diff --git a/tools/testing/kunit/configs/arm64_user4k.config b/tools/testing/kunit/configs/arm64_user4k.config new file mode 100644 index 00000000000000..de6c8ec2fdff82 --- /dev/null +++ b/tools/testing/kunit/configs/arm64_user4k.config @@ -0,0 +1,33 @@ +CONFIG_KUNIT=y +CONFIG_ARM64_16K_PAGES=y +CONFIG_ARM64_VA_BITS_48=y +CONFIG_ARM64_PA_BITS_48=y +CONFIG_ARM64_USER4K_EXPERIMENTAL=y +CONFIG_ARM64_USER4K_KUNIT_TEST=y +CONFIG_ARM64_SW_TTBR0_PAN=y +# CONFIG_ARM64_CNP is not set +# CONFIG_ARM64_POE is not set +CONFIG_SMP=y +CONFIG_PRINTK=y +CONFIG_SERIAL_AMBA_PL011=y +CONFIG_SERIAL_AMBA_PL011_CONSOLE=y +CONFIG_POWER_RESET=y +CONFIG_POWER_RESET_SYSCON=y +CONFIG_BLK_DEV_INITRD=y +CONFIG_TMPFS=y +CONFIG_TRANSPARENT_HUGEPAGE=y +CONFIG_DEBUG_KERNEL=y +CONFIG_DEBUG_VM=y +CONFIG_DEBUG_VM_MAPLE_TREE=y +CONFIG_DEBUG_VM_RB=y +CONFIG_PROVE_LOCKING=y +CONFIG_PAGE_TABLE_CHECK=y +CONFIG_PAGE_TABLE_CHECK_ENFORCED=y +# Guest-only infrastructure for late swap/reclaim fixtures. +CONFIG_BLOCK=y +CONFIG_VIRTIO=y +CONFIG_VIRTIO_MMIO=y +CONFIG_VIRTIO_BLK=y +CONFIG_DEVTMPFS=y +CONFIG_DEBUG_FS=y +CONFIG_KUNIT_DEBUGFS=y diff --git a/tools/testing/kunit/configs/arm64_user4k_data_kasan.config b/tools/testing/kunit/configs/arm64_user4k_data_kasan.config new file mode 100644 index 00000000000000..2e3e50bd677683 --- /dev/null +++ b/tools/testing/kunit/configs/arm64_user4k_data_kasan.config @@ -0,0 +1,18 @@ +CONFIG_KUNIT=y +CONFIG_ARM64_16K_PAGES=y +CONFIG_ARM64_VA_BITS_48=y +CONFIG_ARM64_PA_BITS_48=y +CONFIG_ARM64_USER4K_EXPERIMENTAL=y +CONFIG_ARM64_USER4K_DATA_KUNIT_TEST=y +# CONFIG_ARM64_USER4K_KUNIT_TEST is not set +CONFIG_KASAN=y +CONFIG_KASAN_GENERIC=y +CONFIG_KASAN_INLINE=y +CONFIG_SMP=y +CONFIG_PRINTK=y +CONFIG_SERIAL_AMBA_PL011=y +CONFIG_SERIAL_AMBA_PL011_CONSOLE=y +CONFIG_POWER_RESET=y +CONFIG_POWER_RESET_SYSCON=y +CONFIG_BLK_DEV_INITRD=y +CONFIG_TMPFS=y diff --git a/tools/testing/kunit/configs/arm64_user4k_mm_kasan.config b/tools/testing/kunit/configs/arm64_user4k_mm_kasan.config new file mode 100644 index 00000000000000..11b9935770f537 --- /dev/null +++ b/tools/testing/kunit/configs/arm64_user4k_mm_kasan.config @@ -0,0 +1,38 @@ +CONFIG_KUNIT=y +CONFIG_ARM64_16K_PAGES=y +CONFIG_ARM64_VA_BITS_48=y +CONFIG_ARM64_PA_BITS_48=y +CONFIG_ARM64_USER4K_EXPERIMENTAL=y +CONFIG_ARM64_USER4K_KUNIT_TEST=y +CONFIG_ARM64_SW_TTBR0_PAN=y +# CONFIG_ARM64_CNP is not set +# CONFIG_ARM64_POE is not set +CONFIG_SMP=y +CONFIG_PRINTK=y +CONFIG_SERIAL_AMBA_PL011=y +CONFIG_SERIAL_AMBA_PL011_CONSOLE=y +CONFIG_POWER_RESET=y +CONFIG_POWER_RESET_SYSCON=y +CONFIG_BLK_DEV_INITRD=y +CONFIG_TMPFS=y +CONFIG_TRANSPARENT_HUGEPAGE=y +CONFIG_DEBUG_KERNEL=y +CONFIG_DEBUG_VM=y +CONFIG_DEBUG_VM_MAPLE_TREE=y +CONFIG_DEBUG_VM_RB=y +CONFIG_PROVE_LOCKING=y +CONFIG_PAGE_TABLE_CHECK=y +CONFIG_PAGE_TABLE_CHECK_ENFORCED=y +# Guest-only infrastructure for late swap/reclaim fixtures. +CONFIG_BLOCK=y +CONFIG_VIRTIO=y +CONFIG_VIRTIO_MMIO=y +CONFIG_VIRTIO_BLK=y +CONFIG_DEVTMPFS=y +CONFIG_DEBUG_FS=y +CONFIG_KUNIT_DEBUGFS=y + +# Software MM fixtures only; the hardware boot opt-in is rejected under KASAN. +CONFIG_KASAN=y +CONFIG_KASAN_GENERIC=y +CONFIG_KASAN_INLINE=y diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 641a180fb35f41..6efdaf9c73cb1e 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -17,7 +17,9 @@ TARGETS += damon TARGETS += devices/error_logs TARGETS += devices/probe TARGETS += dmabuf-heaps +TARGETS += drivers/apple TARGETS += drivers/dma-buf +TARGETS += drivers/gpu/apple-dcp TARGETS += drivers/ntsync TARGETS += drivers/s390x/uvdevice TARGETS += drivers/net diff --git a/tools/testing/selftests/arm64/Makefile b/tools/testing/selftests/arm64/Makefile index e456f3b62fa138..d8948f5fb1ef86 100644 --- a/tools/testing/selftests/arm64/Makefile +++ b/tools/testing/selftests/arm64/Makefile @@ -4,7 +4,7 @@ ARCH ?= $(shell uname -m 2>/dev/null || echo not) ifneq (,$(filter $(ARCH),aarch64 arm64)) -ARM64_SUBTARGETS ?= tags signal pauth fp mte bti abi gcs +ARM64_SUBTARGETS ?= tags signal pauth fp mte bti abi gcs user4k else ARM64_SUBTARGETS := endif diff --git a/tools/testing/selftests/arm64/user4k/.gitignore b/tools/testing/selftests/arm64/user4k/.gitignore new file mode 100644 index 00000000000000..e3e47c93059017 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/.gitignore @@ -0,0 +1,13 @@ +page-contract + +swap-fixture-init +same-quarter +file-cow-consumers +same-native +first-write +clipped-firstwrite +pinned-fork +pagetable-retention +pagetable-concurrent + +/domain-cycle diff --git a/tools/testing/selftests/arm64/user4k/Makefile b/tools/testing/selftests/arm64/user4k/Makefile new file mode 100644 index 00000000000000..889386ca79ffcc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/Makefile @@ -0,0 +1,148 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_GEN_PROGS := page-contract proc-args +TEST_GEN_PROGS_EXTENDED := iommufd-init coarse-shmem-init bpf-uptr-init vhost-log-init uprobes-init user-events-init uffd-move-init swap-fixture-init exec-fixture-init mixed-process memory-interfaces advice reclaim-advice proc-pages proc-memory core-fixture-init context-stress-init mlock-fixture-init lazyfree-fixture-init thread-fixture-init splice-fixture-init dio-fixture-init uring-fixture-init gem-fixture-init compat-fixture-init granule-proc-init granule-swap-init aslr-fixture-init cpu-granule-init user16k-smoke-init cpu-granule16-init granule-threeway-init compat-mlock-init mglru-granule-init memcg-pressure-init memcg-accounting-init runtime-app-init fex-contract-init exec-policy-init uffd-wp-init shmem-uffd-swap-init compat-uffd-init shmem-hole-init shmem-thp-init shmem-collapse-init shmem-pin-init shmem-race-init shmem-pressure-init +TEST_GEN_PROGS_EXTENDED += vfio-fixture-init sysv-shm-init perf-ring-init +TEST_GEN_PROGS_EXTENDED += aio-ring-init numa-policy-init numa-balancing-init cow-pool +TEST_GEN_PROGS_EXTENDED += mte-core-init +CFLAGS += -Wall -Wextra -O2 -ffixed-x18 +TEST_GEN_FILES += compat-probe-4k compat-probe-64k compat-probe-dynamic compat-interp32 compat-aslr-probe compat-mlock-probe +TEST_GEN_FILES += fex-x86-contract libfex-abi.so compat-policy-probe compat-uffd-probe compat-vfio-probe +TEST_GEN_FILES += compat-shm-probe compat-perf-probe compat-numa-probe compat-numa-balancing-probe +TEST_GEN_PROGS_EXTENDED += anon-hole-refill anon-hole-refill-race anon-hole-refill-reclaim +TEST_FILES += README.hole-refill README.hole-refill-race README.hole-refill-reclaim +TEST_FILES += compat-script runtime-app.py +TEST_GEN_PROGS_EXTENDED += same-quarter file-cow-consumers same-native +TEST_FILES += README.any-quarter check-any-quarter.py +TEST_GEN_PROGS_EXTENDED += first-write clipped-firstwrite +TEST_FILES += README.first-write +TEST_GEN_PROGS_EXTENDED += pinned-fork +TEST_FILES += README.pinned-fork +TEST_GEN_PROGS_EXTENDED += pagetable-retention pagetable-concurrent +TEST_FILES += README.pagetable-backing +TEST_GEN_PROGS_EXTENDED += domain-cycle +TEST_FILES += README.domain-cycle +include ../../lib.mk + +$(OUTPUT)/page-contract: page-contract.c x18-probe.S + +$(OUTPUT)/domain-cycle $(OUTPUT)/pagetable-concurrent $(OUTPUT)/first-write $(OUTPUT)/thread-fixture-init $(OUTPUT)/user16k-smoke-init $(OUTPUT)/cow-pool $(OUTPUT)/anon-hole-refill-race $(OUTPUT)/anon-hole-refill-reclaim: LDLIBS += -pthread + +# Freestanding ELF32 probes exercise the compat ABI without an armhf sysroot. +AARCH32_CC ?= clang --target=arm-linux-gnueabihf +AARCH32_FLAGS := -march=armv8-a -marm -mfloat-abi=soft -ffreestanding \ + -fno-builtin -fno-stack-protector -O2 -nostdlib -static -fuse-ld=lld +$(OUTPUT)/compat-probe-4k: compat-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=4096 $< -o $@ +$(OUTPUT)/compat-probe-64k: compat-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ +$(OUTPUT)/compat-interp32: compat-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536,--image-base=0x10000000 $< -o $@ +$(OUTPUT)/compat-probe-dynamic: compat-probe.c + $(AARCH32_CC) $(filter-out -static,$(AARCH32_FLAGS)) -no-pie \ + -Wl,-z,max-page-size=65536,--dynamic-linker=/interp32 $< -o $@ + +$(OUTPUT)/compat-aslr-probe: compat-aslr-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/compat-mlock-probe: compat-mlock-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +# The disposable PID1 fixture also runs with the native 64K process ABI. +$(OUTPUT)/mglru-granule-init $(OUTPUT)/memcg-pressure-init $(OUTPUT)/memcg-accounting-init $(OUTPUT)/runtime-app-init $(OUTPUT)/fex-contract-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +# FEX fixture: the AArch64 emulator runs the freestanding x86-64 contract. +X86_64_CC ?= clang --target=x86_64-linux-gnu +$(OUTPUT)/fex-x86-contract: fex-x86-contract.c + $(X86_64_CC) -O2 -fno-builtin -fno-stack-protector -nostdlib -static \ + -fuse-ld=lld -Wl,-z,max-page-size=4096,-e,_start $< -o $@ + +$(OUTPUT)/libfex-abi.so: fex-native-abi.c + $(CC) $(CFLAGS) -shared -fPIC -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/exec-policy-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/exec-policy-init: LDLIBS += -pthread +$(OUTPUT)/mte-core-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-policy-probe: compat-policy-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +# Optional Windows payload; requires Wine/FEX in a disposable guest to execute. +# Keep its toolchain out of the ordinary Linux selftest build. +WINDOWS_CC ?= clang --target=x86_64-pc-windows-msvc +WINDOWS_DLLTOOL ?= llvm-dlltool +WINDOWS_LD ?= lld-link +.PHONY: wine-contract +wine-contract: $(OUTPUT)/wine-page-contract.exe +$(OUTPUT)/wine-kernel32.lib: wine-kernel32.def + $(WINDOWS_DLLTOOL) -m i386:x86-64 -d $< -l $@ +$(OUTPUT)/wine-page-contract.obj: wine-page-contract.c + $(WINDOWS_CC) -O2 -Wall -Wextra -Werror -ffreestanding -fno-stack-protector -c $< -o $@ +$(OUTPUT)/wine-page-contract.exe: $(OUTPUT)/wine-page-contract.obj $(OUTPUT)/wine-kernel32.lib + $(WINDOWS_LD) /entry:mainCRTStartup /subsystem:console /nodefaultlib /out:$@ $^ +EXTRA_CLEAN += $(OUTPUT)/wine-page-contract.exe $(OUTPUT)/wine-page-contract.obj $(OUTPUT)/wine-kernel32.lib + +$(OUTPUT)/uffd-wp-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/uffd-wp-init: LDLIBS += -pthread + +$(OUTPUT)/shmem-uffd-swap-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/compat-uffd-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-uffd-probe: compat-uffd-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/shmem-hole-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/shmem-hole-init: LDLIBS += -pthread + +$(OUTPUT)/shmem-thp-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/shmem-thp-init: LDLIBS += -pthread + +$(OUTPUT)/shmem-collapse-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/shmem-pin-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/shmem-pin-init: LDLIBS += -pthread + +$(OUTPUT)/shmem-race-init $(OUTPUT)/shmem-pressure-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/shmem-race-init: LDLIBS += -pthread + +$(OUTPUT)/uffd-move-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/uffd-move-init: LDLIBS += -pthread + +$(OUTPUT)/user-events-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/uprobes-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/vhost-log-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/bpf-uptr-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/coarse-shmem-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +$(OUTPUT)/iommufd-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/iommufd-init: LDLIBS += -pthread +$(OUTPUT)/iommufd-init: CFLAGS += $(KHDR_INCLUDES) + +# Privileged recording-domain fixture; run only in a disposable test VM. +$(OUTPUT)/vfio-fixture-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-vfio-probe: compat-vfio-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/sysv-shm-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-shm-probe: compat-shm-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +# Disposable guests exercise perf layout and software sampling across mm granules. +$(OUTPUT)/perf-ring-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-perf-probe: compat-perf-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/aio-ring-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 + +# Requires a disposable guest with two memory nodes and CONFIG_COMPAT. +$(OUTPUT)/numa-policy-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/compat-numa-probe: compat-numa-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ + +$(OUTPUT)/numa-balancing-init: LDFLAGS += -static -Wl,-z,max-page-size=65536 +$(OUTPUT)/numa-balancing-init: LDLIBS += -pthread + +$(OUTPUT)/compat-numa-balancing-probe: compat-numa-balancing-probe.c + $(AARCH32_CC) $(AARCH32_FLAGS) -Wl,-z,max-page-size=65536 $< -o $@ diff --git a/tools/testing/selftests/arm64/user4k/README.any-quarter b/tools/testing/selftests/arm64/user4k/README.any-quarter new file mode 100644 index 00000000000000..305441c8a9db47 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.any-quarter @@ -0,0 +1,37 @@ +Arbitrary-quarter anonymous COW qualification +============================================= + +Run on the audited native16K kernel through an established4K ABI launcher, +with CAP_SYS_ADMIN for visible pagemap PFNs: + userpages 4k ./same-quarter --native-16k > same-quarter.log + python3 check-any-quarter.py same-quarter.log + userpages 4k ./file-cow-consumers --native-16k + userpages 4k ./same-native --native-16k + +same-quarter: two fresh owner+child arms compare4MiB changed data in same versus +balanced virtual quarters of a16MiB dense parent. Exact nativePFN union, no +modified-child alias to parent/unchangedchild, all-byte isolation and histograms. +The binary reports measurements; the checker requires both arms <=6MiB native +for4MiB changed and actual nonzero physical-quarter use in the same-quarter arm. +This catches the old16MiB same-quarter result while allowing bounded pool slack. +30s supervisor/20s child alarms, group/subreaper cleanup, no pressure. + +file-cow-consumers:1MiB owned memfd,two private views; child explicitly READ- +prefaults both views AFTER fork (fork may omit reconstructible file PTEs). +CPU COW, remote typed GUP read-unshare/write-COW each require actual physical/ +logical quarter mismatch and correct file-to-anonymous transition. Then shifted +VA mremap must preserve every actualPFN and byte; post-move GUP and parent/file +isolation are checked.30s supervisor/20s child. Exact stage/byte/PTE diagnostics. +This is typed GUP coverage, not long-term DMA pin or migration qualification. + +same-native:16fresh three-generation trials/16KiB logical payload. B's first COW +is retained by held C while B COWs again. Required distinct4KPFNs sharing native +PFN/4 prove source/destination use one nativefolio; every generation and neighbor +byte stays correct. At least1actual pair is required for PASS; exit77 means +UNEXERCISED, never a coverage pass.20/5/4s supervisor/owner/held-child alarms. + +No fixture changes memory policy, system swap, kernel or GPU state. Native16K +assumption is explicit; do not run density math on an unaudited kernel. Metadata +--selftest modes (where provided) are not kernel-path qualification. UFFD_MOVE +and targeted migration are separate gates; disabled UFFD/single-node systems do +not acquire those qualifications merely by passing these tests. diff --git a/tools/testing/selftests/arm64/user4k/README.domain-cycle b/tools/testing/selftests/arm64/user4k/README.domain-cycle new file mode 100644 index 00000000000000..3e7425f4dbffbf --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.domain-cycle @@ -0,0 +1,78 @@ +Cgroup allocation-domain pool reuse +================================== + +Build domain-cycle with -pthread. Run in the established4K userspace ABI on a +native16K kernel. Root-visible pagemap and kpagecgroup are required for the +controller's density/charge proof. The fixture itself checks every payload byte. +No memory.high/max limits, reclaim, swap-area or GPU operations are needed. + + userpages 4k ./domain-cycle --native-16k [--tail | --race] + +The program prints READY and waits for stdin byte 'c' before allocating payload. +A controller places only this fresh fixture in owned cgroup A, then acknowledges. +Each PHASE prints its index/PID/base/leaf count/stride and waits for 'c'. Both +cgroups must remain alive until all prior ranges have been checked and unmapped. + +Default: three1MiB useful guarded allocation phases. Run in A, move to B, move +back to A between phases. Check all256 written leaves/range (4K every8K), excluding +the inaccessible guards. The old sticky-domain implementation can allocate16K +native backing per4K leaf after a move. Packed acceptance is1MiB..1.25MiB native +PFN union per1MiB phase; the allowance covers bounded shard/partial-owner tails. + +--tail:16 independent4K allocations, switching A/B after EVERY leaf. All16VAs are +selected in the same current16-way allocator shard using its documented hash. +Eight leaves/domain should occupy at most3native folios: two useful plus one +possible preexisting partial boundary. This catches a latest-domain-only cache +that closes old A pools when switching to B, then cannot use A's surviving free +quarters on return. Preserve every earlier leaf throughout all switches. Useful +payload64KiB, maximum256KiB private backing, about8MiB virtual reservation. + +--race:4workers allocate64guarded leaves each, with bounded test-only2ms pacing +per leaf. RACE prints four bases and waits for 'c'. After acknowledgement, move +only this thread group between A/B32times while allocation is active; record +monotonic move intervals. RACE_DONE prints the allocation interval and waits. +Require at least one move overlapping that interval before claiming concurrency +coverage (this does not prove a particular RCU/CAS interleaving occurred). Check +all race PFNs are charged to A or B, never outside those owned domains. Then move +the stopped fixture to B and acknowledge: it allocates one additional1MiB guarded +PHASE. Require packed density after transitions stop, plus unchanged earlier +race PFNs/charges and all bytes. Worst native payload8MiB, useful2MiB; threads +have128KiB stacks. No tight density claim applies during arbitrary in-flight moves. + +For each held range, read pagemap indexed by target4K VA. A present PFN is in4K +units; native PFN=PFN>>2. Read /proc/kpagecgroup at native_PFN*8 and compare with +stat(the intended cgroup).st_ino. Old phases keep their original allocation-time +charge after process moves. During racing allocations either A/B is valid. +Re-read PFNs to reject migration during the charge sample. Do not use total +memory.current as exact payload accounting: it includes code/stacks/page tables. +Normal8KiB guard stride and --tail4KiB stride are emitted explicitly. + +Use an owned fresh uncapped cgroup hierarchy and a bounded controller. If using +a parent unit and childA/B groups, move the held PID into childA BEFORE enabling +memory in that owned parent's subtree_control (parent must be empty). Do not +change unrelated cgroups or global controller settings. Parent15s alarm kills +all fixture threads; partial thread creation releases waiting workers before +joining. Controller should independently terminate/reap the owned process/unit +on protocol error, timeout or failed metadata, and retain the evidence. + +--selftest only validates byte patterns and same-shard address selection; it does +not exercise cgroup moves, kernel pool allocation or RCU. Hardware qualification +must include bulk/tail/race content and exact-charge/density gates, mm-exit cleanup, +existing COW/fork/reclaim controls, and dense/concurrent timing. No source or +parser-only pass closes these runtime/performance requirements. + +Race density must count newly added native owners. Before and after sampling +the final steady range, recheck the original race PFNs and allocation charges. +Reject old/new physical 4K aliases and inconsistent charges in shared native +owners. Subtract only those native owners actually observed in both stable +snapshots. The 80-owner/1,310,720-byte limit applies to the added native PFN +union for the new 1 MiB payload. Reusing an old owner's free quarters should +not count its existing live quarters as additional backing. Record total, +reused and added owners separately, plus combined per-domain quarter masks. +This is payload accounting, not proof of whole-mm alias closure. + +On Neo the corrected controller observed 77 owners in the final range, of +which 14 were reused and 63 new (1,032,192 added bytes). Old mappings and +charges stayed unchanged, reused slots were disjoint and all bytes passed. +The earlier range-only oracle failed at 81 seen owners before its final +checks; that run remains failed, not retrospectively qualified. diff --git a/tools/testing/selftests/arm64/user4k/README.first-write b/tools/testing/selftests/arm64/user4k/README.first-write new file mode 100644 index 00000000000000..2ec69368a6c704 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.first-write @@ -0,0 +1,28 @@ +First-write and clipped-native-window backing tests +================================================== + +Run only on the audited native16K kernel through the established4K ABI launcher: + cd /var/tmp/owned-test-dir + userpages 4k ./first-write --native-16k + userpages 4k ./clipped-firstwrite --native-16k + +first-write: six fresh child arms (anonymous,shmem,regular file x dense/guarded), +eight threads, absent-PTE direct first writes, full untouched-byte verification, +DONTNEED and distinct-byte same-PTE refault racing, all source-file bytes verified. +Regular-file working directory MUST be non-tmpfs; otherwise that arm SKIPs. +All six must pass for qualification.4MiB useful payload/arm,<=16MiB native private +backing under a wasteful baseline plus source file.40s supervisor/15s child alarm +kills the fixture child/threads on timeout. No global pressure or kernel change. +This content/race fixture does not itself assert physical density. + +clipped-firstwrite: root CAP_SYS_ADMIN pagemap visibility required.11 fresh child +arms cover all valid starts for1/2/3leaf windows,3+1tail fill and full4dense control; +two rounds with DONTNEED between. A single initial byte write must prefault every +neighbor in its clipped batch, proven by pagemap before any neighbor read. Full +zero/content checks then native PFN union enforce density.256groups/arm, +<=8MiB payload backing even on baseline;16MiB reserved VA.1MiB native slack bounds +pool sharding/background COW. Count3 alone allows natural16K/12K packing; mixed3+1 +must fill tails.40s supervisor/8s child alarm, no GPU/reclaim/pressure. + +The fixture metadata selftest (--selftest) checks tags/sorting only. It is not a +kernel-path test and does not replace running both fixtures on the target kernel. diff --git a/tools/testing/selftests/arm64/user4k/README.hole-refill b/tools/testing/selftests/arm64/user4k/README.hole-refill new file mode 100644 index 00000000000000..e547a6bfc05ebe --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.hole-refill @@ -0,0 +1,29 @@ +Anonymous hole-refill regression +================================ + +Run anon-hole-refill through the platform's 4K userspace ABI launcher on a +kernel with a native 16K page size. The test does not change its page ABI. + + ./anon-hole-refill + sudo ./anon-hole-refill --native-16k + +The optional --native-16k argument enables physical density checks and requires +visible pagemap PFNs (CAP_SYS_ADMIN). Without it, four density results are +skipped. Do not use it with a different native page size. Pagemap must be +readable even without privileged PFNs: present bits verify neighbor prefault. + +The test owns 16 MiB of logical data. It discards individual 4K leaves and +writes before reading them, exercising missing-PTE refill rather than zero-page +COW. It checks zero initialization, all retained data and neighboring mappings. +Two phased rounds check slot reuse. A two-hole case checks that one fault +prefaults both missing neighbors before the second is accessed. + +Initial backing must be within one native page of 16 MiB. Final backing after +each refill sequence must be at most 20 MiB, allowing bounded pool slack. The +old fresh-private-folio path consumes 64 MiB after single-hole refill rounds +and 32 MiB after the paired-hole sequence. Temporary coexistence of original +and replacement backing is allowed; the test does not require 16 MiB at every +intermediate step. + +Expect eight TAP results. A 30-second alarm bounds the test. It does not fork, +induce pressure or test concurrent faults; see the companion lifetime tests. diff --git a/tools/testing/selftests/arm64/user4k/README.hole-refill-race b/tools/testing/selftests/arm64/user4k/README.hole-refill-race new file mode 100644 index 00000000000000..96f2994a4765b0 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.hole-refill-race @@ -0,0 +1,33 @@ +Concurrent hole-refill lifetime test +=================================== + + sudo ./anon-hole-refill-race + +Run with a 4K userspace ABI on the native 16K kernel. CAP_SYS_ADMIN is required +to distinguish real swap entries from migration and poison markers. The target +configuration has MAX_SWAPFILES=27; types 27 through 31 are excluded and counted +separately. + +Four workers own disjoint 16K virtual windows in a 16 MiB region. Two phased +rounds discard and refill individual 4K leaves. Barriers synchronize quarter +phases, exercising concurrent shared-pool reservations without a userspace data +race. Every refill checks zeroing and unchanged neighbors. This covers pool +contention across windows, not simultaneous PTE changes within one window. + +After joining workers and verifying the region, the allocating process forks a +survivor, modifies one native window privately and exits. A supervisor waits +for that exit before releasing the survivor. The survivor verifies original +contents, rewrites all leaf tags, and verifies them again. This checks that +backing survives its original allocating mm and preserves fork isolation. + +The survivor then applies MADV_COLD/PAGEOUT only to its own region, at most 20 +times with 10 ms between attempts. At least one real swapped leaf and complete +byte-exact readback are required for the reclaim result. The reported count +states coverage; one swapped leaf is not broad reclaim qualification. These +advice operations can skip nonlinear pooled folios because their mappings are +not contiguous. Use README.hole-refill-reclaim for broader controlled coverage. + +Expect three TAP passes and no skips or unexpected signals. The supervisor has +a 35-second alarm and children have 30-second alarms. Failure or timeout kills +the dedicated process group; the supervisor reaps surviving descendants. An +external runner should also bound the entire fixture cgroup. diff --git a/tools/testing/selftests/arm64/user4k/README.hole-refill-reclaim b/tools/testing/selftests/arm64/user4k/README.hole-refill-reclaim new file mode 100644 index 00000000000000..6ede9503192b8b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.hole-refill-reclaim @@ -0,0 +1,36 @@ +Controlled reclaim of hole-refill backing +======================================== + + sudo ./anon-hole-refill-reclaim --external-reclaim + +This separate fixture repeats the concurrent allocation and owner-exit checks +in README.hole-refill-race. It requires the same page ABI, native granule and +visible swap metadata. After the survivor rewrites and verifies its region, it +prints: + + RECLAIM_READY pid= base= bytes=16777216 owner_mm_exited=1 + +It then waits up to 20 seconds for byte r on stdin. Place only the fixture and +its descendants in a fresh cgroup, with the controller outside that cgroup. +The controller may make at most three writes of "24M swappiness=200" to that +cgroup's memory.reclaim. Leave memory.max/high unchanged and do not reclaim +other cgroups or create global pressure. Bound controller work within the +20-second window. + +Record memory.current, memory.swap.current, memory.stat and memory.events, plus +present and valid-swap PTE counts for the reported region. EAGAIN or a partial +reclaim result does not determine success: actual coverage does. Send r after +the bounded attempts, even if coverage is below the required threshold. + +Before reading data back, the survivor snapshots pagemap. It requires at least +3072 of 4096 leaves (75%) to contain real swap entries and verifies every byte +after refault. Insufficient coverage fails this qualification rather than +skipping it. Expect three TAP passes, no skips and no unexpected signals or +kernel warnings. Supervisor and child alarms are 35 and 30 seconds; the +fixture's dedicated process group is killed on failure. + +MADV_COLD/PAGEOUT compares native folio mapcount with a contiguous PTE batch. +Nonlinear pooled quarters can belong to the same requested region while +occupying separate virtual locations, so those advice operations skip them. +This fixture exercises ordinary cgroup reclaim and nonlinear reverse mapping +without removing the advice path's protection for other mappings. diff --git a/tools/testing/selftests/arm64/user4k/README.pagetable-backing b/tools/testing/selftests/arm64/user4k/README.pagetable-backing new file mode 100644 index 00000000000000..cf87f2a64f5aec --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.pagetable-backing @@ -0,0 +1,45 @@ +Page-table physical backing and fragment retention +================================================== + +These bounded fixtures reserve 2 GiB of virtual address space and write one +4 KiB leaf in each of 1,024 distinct 2 MiB table windows. Native payload +backing is at most 16 MiB, with approximately 4 MiB of logical PTE tables. +The concurrent variant uses 32 threads, 32 synchronized fault rounds and +128 KiB thread stacks. Both verify all payload bytes across six checkpoints: +baseline, full, quarter_survivors, refilled, quarter_survivors_again, unmapped. + +Run through the 4 KiB ABI launcher on a native 16 KiB kernel. Each JSON +checkpoint requires one byte 'c' on stdin before continuing. For contents +only, with no physical charge observation: + + yes c | userpages 4k ./pagetable-retention --native-16k + yes c | userpages 4k ./pagetable-concurrent --native-16k + +For physical accounting, run each fixture alone in a fresh uncapped cgroup. +At each checkpoint verify the fixture is the only member, record memory.stat +pagetables and memory.swap.current, and sample at 0, 1 and 3 seconds before +acknowledging. Record logical VmPTE separately: it is not physical backing. +Use an external service lifetime below the fixture's 60-second alarm and +verify all processes exit. Do not change memory.max/high or force reclaim. + +Compare the same complete fixture across kernels. Concurrent thread stacks +and upper-level tables contribute to the measurements; do not subtract them +as though they were pure payload. Deferred table releases can occur between +samples. Contents PASS does not assert a density or performance result. + +The allocator reuses safely retired fragments, including slots from owners +that still contain live tables. RCU-pending slots remain unavailable until +their callback runs; an empty owner is freed. A reused dirty slot is cleared +before it is returned. This avoids allocating new owners while old ones have +reusable holes. It does not compact surviving live tables. + +On Neo, the serial refill checkpoint fell from 7,421,952 to 4,276,224 bytes of +physical page tables; the concurrent refill fell from 6,275,072 to 4,276,224. +The same six-phase contents and post-unmap cleanup passed. Whole-cgroup slab +usage increased and includes allocator metadata plus other kernel objects. +Do not report page-table bytes as the complete memory cost. + +Seven operation-timed rounds after one warmup had serial/concurrent total +medians of 15.625/16.865 ms, versus 15.434/16.679 ms before reuse. The ranges +substantially overlap; a small cost remains possible. These measurements do +not establish a speedup or zero overhead. They exclude checkpoint I/O. diff --git a/tools/testing/selftests/arm64/user4k/README.pinned-fork b/tools/testing/selftests/arm64/user4k/README.pinned-fork new file mode 100644 index 00000000000000..129adde8b005c1 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/README.pinned-fork @@ -0,0 +1,37 @@ +Pinned anonymous fork density and pin isolation +============================================= + +Build pinned-fork with the other arm64/user4k tests. Run as root through the +established 4K userspace launcher on a native16K kernel: + + userpages 4k ./pinned-fork --native-16k --expect-packed + +The fixture initializes one native-aligned1MiB anonymous range, registers real +writable long-term io_uring buffer pins, then forks while the parent retains the +registration. It requires every copied child native PFN to be disjoint from the +pinned parent's native PFNs. It checks every byte before and after distinct +parent/child writes; registration and PFN disjointness, not mlock or VmPin alone, +establish copy-path coverage. Kernel/pin-policy limitations return77 UNEXERCISED. + +--expect-packed requires child backing <=1.25MiB for1MiB copied data, with bounded +pool-shard slack. Without this option the fixture also accepts the old4MiB +backing so a baseline can record both content and density separately. Successful +contents alone do not establish packed density. --selftest checks metadata PFN +grouping only, with no dynamic pin/fork qualification. + +Optional --hold reports the parent PID, child PID and range base after fork PFN +verification, then waits for the byte 'c' on stdin. A controller can hold both +processes in one fresh uncapped non-root cgroup and verify exact payload charges: +read each target's pagemap in4KiB units; normalize PFN>>2 to native16KiB PFN; +read /proc/kpagecgroup at native_PFN*8 and require each returned inode equals +stat(the fixture cgroup).st_ino. Re-read pagemap to reject migration during the +sample. This directly catches incorrectly root-charged fork copies; total +memory.current includes unrelated fixture code/stack/table costs and is not an +exact payload charge oracle. Verify both PIDs belong to the same owned cgroup, +record memory.stat/status, and release promptly (within3seconds). + +Parent15s/child10s alarms bound the protocol; parent exit kills/reaps the child +and the child uses PDEATHSIG. Payload native backing is at most5MiB on the old +path. No forced reclaim, memory.max/high changes, GPU operations, or swap-area +changes are used. Controllers should additionally enforce their own process-group +or service-unit lifetime and preserve logs on any failure. diff --git a/tools/testing/selftests/arm64/user4k/advice.c b/tools/testing/selftests/arm64/user4k/advice.c new file mode 100644 index 00000000000000..4d4cb89350985d --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/advice.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static size_t page; + +static bool wait_child(pid_t pid, int sig) +{ + int status = 0; + + if (pid <= 0 || waitpid(pid, &status, 0) != pid) + return false; + if (sig) + return WIFSIGNALED(status) && WTERMSIG(status) == sig; + return WIFEXITED(status) && !WEXITSTATUS(status); +} + +static void *anon(size_t length, int prot, int extra) +{ + void *p = mmap(NULL, length, prot, MAP_PRIVATE | MAP_ANONYMOUS | extra, + -1, 0); + + if (p == MAP_FAILED) + ksft_exit_fail_msg("anonymous mmap\n"); + return p; +} + +static void discard_and_guards(void) +{ + unsigned char *p = anon(3 * page, PROT_READ | PROT_WRITE, 0); + bool ok; + pid_t child; + + memset(p, 0x61, 3 * page); + ok = madvise(p + page, 1, MADV_DONTNEED) == 0; + ok &= p[0] == 0x61 && p[page] == 0 && p[2 * page] == 0x61; + ksft_test_result(ok, + "DONTNEED discards exactly one rounded user page\n"); + memset(p + page, 0x71, page); + child = fork(); + if (!child) { + ok = madvise(p + page, page, MADV_DONTNEED) == 0; + _exit(!(ok && p[page] == 0 && p[0] == 0x61)); + } + ksft_test_result( + wait_child(child, 0) && p[page] == 0x71, + "discard in a fork child leaves the parent quarter intact\n"); + ok = madvise(p + page, 1, MADV_GUARD_INSTALL) == 0; + ok &= madvise(p + page, page, MADV_DONTNEED) == 0; + child = fork(); + if (!child) { + volatile unsigned char value = p[page]; + + (void)value; + _exit(1); + } + ksft_test_result(ok && wait_child(child, SIGSEGV), + "a user-page guard survives discard and fork\n"); + ok = madvise(p + page, 1, MADV_GUARD_REMOVE) == 0; + ok &= p[page] == 0 && p[0] == 0x61 && p[2 * page] == 0x61; + ksft_test_result(ok, + "guard removal restores only the requested page\n"); + munmap(p, 3 * page); +} + +static void population(void) +{ + unsigned char *p = anon(5 * page, PROT_READ, 0), vec[5]; + bool ok; + unsigned int i; + + ok = mprotect(p + 2 * page, page, PROT_READ | PROT_WRITE) == 0; + ok &= madvise(p + 2 * page, 1, MADV_POPULATE_WRITE) == 0; + ok &= mincore(p, 5 * page, vec) == 0; + for (i = 0; i < 5; i++) + ok &= vec[i] == (i == 2); + ksft_test_result( + ok, "POPULATE_WRITE faults only the selected user-page VMA\n"); + ok = madvise(p + page, 1, MADV_POPULATE_READ) == 0; + ok &= mincore(p + page, page, vec) == 0 && vec[0] == 1; + errno = 0; + ok &= madvise(p, page, MADV_POPULATE_WRITE) == -1 && errno == EINVAL; + ok &= mprotect(p + 3 * page, page, PROT_NONE) == 0; + errno = 0; + ok &= madvise(p + 3 * page, page, MADV_POPULATE_READ) == -1 && + errno == EINVAL; + ok &= munmap(p + 3 * page, page) == 0; + errno = 0; + ok &= madvise(p + 3 * page, page, MADV_POPULATE_READ) == -1 && + errno == ENOMEM; + ksft_test_result( + ok, "POPULATE_READ honors permissions and reports holes\n"); + munmap(p, 5 * page); + p = anon(5 * page, PROT_READ | PROT_WRITE, MAP_POPULATE); + ok = mincore(p, 5 * page, vec) == 0; + for (i = 0; i < 5; i++) + ok &= vec[i] == 1; + ksft_test_result(ok, "MAP_POPULATE materializes every user page\n"); + munmap(p, 5 * page); +} + +static void fork_advice(void) +{ + unsigned char *p = anon(3 * page, PROT_READ | PROT_WRITE, 0); + bool ok; + pid_t child; + + memset(p, 0x58, 3 * page); + ok = madvise(p + page, 1, MADV_WIPEONFORK) == 0; + child = fork(); + if (!child) + _exit(!(p[page] == 0 && p[0] == 0x58 && p[2 * page] == 0x58)); + ok &= wait_child(child, 0) && p[page] == 0x58; + ksft_test_result( + ok, + "WIPEONFORK clears a single child page and preserves neighbors\n"); + ok = madvise(p + page, 1, MADV_KEEPONFORK) == 0; + ok &= madvise(p + page, 1, MADV_DONTFORK) == 0; + child = fork(); + if (!child) { + unsigned char vec; + + errno = 0; + _exit(!(mincore(p + page, page, &vec) == -1 && + errno == ENOMEM && p[0] == 0x58 && + p[2 * page] == 0x58)); + } + ok &= wait_child(child, 0); + ksft_test_result(ok, "DONTFORK omits only its selected user page\n"); + munmap(p, 3 * page); +} + +static void remove_file(void) +{ + int fd = memfd_create("advice-file", 0); + unsigned char *shared, *alias, *private; + bool ok; + size_t i; + + if (fd < 0 || ftruncate(fd, 5 * page)) + ksft_exit_fail_msg("memfd setup\n"); + shared = + mmap(NULL, 5 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + alias = mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + page); + private = mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, + page); + if (shared == MAP_FAILED || alias == MAP_FAILED || + private == MAP_FAILED) + ksft_exit_fail_msg("file mmap\n"); + for (i = 0; i < 5 * page; i++) + shared[i] = 0x40 + i / page; + private[page] = 0x79; + ok = madvise(alias + page, 1, MADV_REMOVE) == 0; + for (i = 0; i < 5 * page; i++) + ok &= shared[i] == (i / page == 2 ? 0 : 0x40 + i / page); + ok &= private[page] == 0x79; + ksft_test_result( + ok, + "REMOVE uses the file byte offset and preserves private COW\n"); + ok = madvise(alias, 3 * page, MADV_WILLNEED) == 0; + ksft_test_result( + ok, "WILLNEED accepts a file range with the process granule\n"); + munmap(shared, 5 * page); + munmap(alias, 3 * page); + munmap(private, 3 * page); + close(fd); +} + +int main(int argc, char **argv) +{ + const struct rlimit no_core = { 0, 0 }; + + page = getauxval(AT_PAGESZ); + if (argc > 2 || (argc == 2 && strcmp(argv[1], "--expect-4k")) || + page != (argc == 2 ? 4096UL : 16384UL)) + ksft_exit_fail_msg("unexpected process page ABI: %zu\n", page); + ksft_print_msg("process page size: %zu\n", page); + if (setrlimit(RLIMIT_CORE, &no_core)) + return 1; + ksft_print_header(); + ksft_set_plan(11); + discard_and_guards(); + population(); + fork_advice(); + remove_file(); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/aio-ring-init.c b/tools/testing/selftests/arm64/user4k/aio-ring-init.c new file mode 100644 index 00000000000000..74f34d16cac299 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/aio-ring-init.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable PID1: legacy AIO ring relocation with per-process page sizes. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct aio_ring_header { + unsigned int id, nr, head, tail; + unsigned int magic, compat_features, incompat_features, header_length; +}; + +static void check(int good, const char *expression, int line) +{ + if (!good) { + fprintf(stderr, "FAIL line %d: %s errno=%d\n", line, expression, + errno); + exit(1); + } +} +#define CHECK(x) check(!!(x), #x, __LINE__) + +static int wait_child(pid_t child) +{ + int status; + pid_t got; + + if (child < 0) + return 1; + do { + got = waitpid(child, &status, 0); + } while (got < 0 && errno == EINTR); + return got != child || !WIFEXITED(status) || WEXITSTATUS(status); +} + +static void complete(aio_context_t context, struct iocb *request, + unsigned long bytes, int event_fd) +{ + struct io_event event; + struct timespec timeout = { .tv_sec = 5 }; + uint64_t notifications; + + CHECK(syscall(SYS_io_getevents, context, 1, 1, &event, &timeout) == 1); + CHECK(event.data == request->aio_data && + event.obj == (uintptr_t)request); + CHECK(event.res == (long long)bytes && event.res2 == 0); + CHECK(read(event_fd, ¬ifications, sizeof(notifications)) == + sizeof(notifications)); + CHECK(notifications == 1); +} + +static void exercise(unsigned long native, unsigned int requested) +{ + unsigned long page = getauxval(AT_PAGESZ), bytes = 2 * page + 37; + aio_context_t context = 0; + int file = memfd_create("aio-granule", 0); + int event_fd = eventfd(0, EFD_NONBLOCK); + unsigned char *source, *destination, *reserve, *target; + struct aio_ring_header *header; + unsigned long ring_size; + struct iocb request = { 0 }, *requests[] = { &request }; + unsigned char residency; + + CHECK(file >= 0 && event_fd >= 0); + CHECK(!ftruncate(file, 4 * page)); + source = mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + destination = mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(source != MAP_FAILED && destination != MAP_FAILED); + for (unsigned long i = 0; i < bytes; i++) + source[page - 19 + i] = i * 17 + i / page; + CHECK(!syscall(SYS_io_setup, requested, &context)); + header = (void *)(uintptr_t)context; + CHECK(header->magic == 0xa10a10a1 && + header->header_length == sizeof(*header)); + CHECK(header->nr >= requested); + ring_size = + header->header_length + header->nr * sizeof(struct io_event); + CHECK(ring_size && !(ring_size % native)); + reserve = mmap(NULL, ring_size + 4 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + target = (void *)(((uintptr_t)reserve + native - 1) & ~(native - 1)); + target += native + (page < native ? page : 0); + target[-1] = 0x51; + target[ring_size] = 0x72; + CHECK(!munmap(target, ring_size)); + request.aio_data = 0x12345678; + request.aio_lio_opcode = IOCB_CMD_PWRITE; + request.aio_fildes = file; + request.aio_buf = (uintptr_t)(source + page - 19); + request.aio_nbytes = bytes; + request.aio_offset = 7; + request.aio_flags = IOCB_FLAG_RESFD; + request.aio_resfd = event_fd; + CHECK(syscall(SYS_io_submit, context, 1, requests) == 1); + /* Retain a queued completion while relocating the ring. */ + CHECK(mremap(header, ring_size, ring_size, + MREMAP_MAYMOVE | MREMAP_FIXED, target) == target); + context = (uintptr_t)target; + header = (void *)target; + CHECK(header->magic == 0xa10a10a1); + CHECK(target[-1] == 0x51 && target[ring_size] == 0x72); + complete(context, &request, bytes, event_fd); + request.aio_data = 0x87654321; + request.aio_lio_opcode = IOCB_CMD_PREAD; + request.aio_buf = (uintptr_t)(destination + page - 11); + CHECK(syscall(SYS_io_submit, context, 1, requests) == 1); + complete(context, &request, bytes, event_fd); + CHECK(!memcmp(source + page - 19, destination + page - 11, bytes)); + CHECK(!syscall(SYS_io_destroy, context)); + errno = 0; + CHECK(mincore(target, page, &residency) == -1 && errno == ENOMEM); + CHECK(target[-1] == 0x51 && target[ring_size] == 0x72); + CHECK(!munmap(reserve, ring_size + 4 * native)); + CHECK(!munmap(source, 4 * page) && !munmap(destination, 4 * page)); + CHECK(!close(file) && !close(event_fd)); + printf("ok - AIO page=%lu requested=%u ring=%lu relocation, I/O, eventfd and teardown\n", + page, requested, ring_size); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + int failed = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--case")) { + native = strtoul(argv[2], NULL, 10); + CHECK(getauxval(AT_PAGESZ) == strtoul(argv[3], NULL, 10)); + exercise(native, 16); + exercise(native, 4096); + return 0; + } + CHECK(getpid() == 1); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + unsigned long sizes[] = { 4096, 16384, 65536 }; + + for (unsigned int i = 0; i < 3; i++) { + if (sizes[i] > native) + continue; + pid_t child = fork(); + + if (!child) { + char n[32], s[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(s, sizeof(s), "%lu", sizes[i]); + CHECK(!prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], + 0UL, 0UL, 0UL)); + execl("/init", "/init", "--case", n, s, NULL); + _exit(2); + } + int bad = wait_child(child); + + printf("%s - AIO native=%lu process=%lu\n", + bad ? "not ok" : "ok", native, sizes[i]); + failed |= bad; + } + printf("AIO GRANULE %s\n", failed ? "FAIL" : "PASS"); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/anon-hole-refill-race.c b/tools/testing/selftests/arm64/user4k/anon-hole-refill-race.c new file mode 100644 index 00000000000000..965141e52e6f77 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/anon-hole-refill-race.c @@ -0,0 +1,298 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Bounded partial-window sharing, original-owner exit and real pageout. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define LEAF 4096UL +#define NATIVE 16384UL +#define BYTES (16UL << 20) +#define LEAVES (BYTES / LEAF) +#define WORKERS 4 +#define ROUNDS 2 + +struct result { + bool concurrent, lifetime, reclaimed; + size_t swapped, non_swap_markers; + unsigned int attempts; +}; + +static unsigned char *data; +static pthread_barrier_t barrier; +static volatile sig_atomic_t fixture_group; + +static void stop_fixture(int signal_number) +{ + if (fixture_group > 0) + kill(-fixture_group, SIGKILL); + _exit(128 + signal_number); +} + +static unsigned char tag(size_t leaf, unsigned int generation) +{ + unsigned char value = leaf % 251 + 1; + + if (generation == 1 && leaf < 4) + value ^= 0xa5; + else if (generation == 2) + value ^= 0x5a; + return value; +} + +static bool verify_leaf(size_t leaf, unsigned int generation) +{ + size_t byte; + + if (data[leaf * LEAF] != tag(leaf, generation)) + return false; + for (byte = 1; byte < LEAF; byte++) + if (data[leaf * LEAF + byte] != 0x5a) + return false; + return true; +} + +static bool verify(unsigned int generation) +{ + size_t leaf; + + for (leaf = 0; leaf < LEAVES; leaf++) + if (!verify_leaf(leaf, generation)) + return false; + return true; +} + +static void *worker(void *arg) +{ + unsigned int id = (uintptr_t)arg, round, quarter, q; + size_t group, leaf, byte; + bool ok = true; + + for (round = 0; round < ROUNDS; round++) { + for (quarter = 0; quarter < 4; quarter++) { + pthread_barrier_wait(&barrier); + for (group = id; group < BYTES / NATIVE; group += WORKERS) { + unsigned char *p; + + leaf = group * 4 + quarter; + p = data + leaf * LEAF; + if (madvise(p, LEAF, MADV_DONTNEED)) { + ok = false; + continue; + } + /* A write fault, without a preceding zero-page read. */ + *(volatile unsigned char *)p = tag(leaf, 0); + for (byte = 1; byte < LEAF; byte++) + if (p[byte]) + ok = false; + memset(p + 1, 0x5a, LEAF - 1); + /* This worker owns the entire native virtual window. */ + for (q = 0; q < 4; q++) + ok &= verify_leaf(group * 4 + q, 0); + } + } + } + return (void *)(uintptr_t)!ok; +} + +static size_t swapped_leaves(size_t *markers) +{ + uint64_t entries[LEAVES]; + size_t i, count = 0; + int fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + ssize_t n; + + if (fd < 0) + return SIZE_MAX; + n = pread(fd, entries, sizeof(entries), (uintptr_t)data / LEAF * 8); + close(fd); + if (n != sizeof(entries)) + return SIZE_MAX; + *markers = 0; + for (i = 0; i < LEAVES; i++) { + if ((entries[i] & (UINT64_C(1) << 63)) || + !(entries[i] & (UINT64_C(1) << 62))) + continue; + /* Audited target config:27..31 are poison/migration/markers. */ + if ((entries[i] & 31) < 27) + count++; + else + (*markers)++; + } + return count; +} + +static bool wait_ok(pid_t pid) +{ + int status; + pid_t got; + + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + return got > 0 && WIFEXITED(status) && !WEXITSTATUS(status); +} + +static void owner(int go, int result_fd) +{ + struct result result = {}; + pthread_t threads[WORKERS]; + void *mapping, *status; + unsigned int i; + bool advice_ok = true, readback_ok; + size_t leaf; + pid_t survivor; + char token; + + alarm(30); + mapping = mmap(NULL, BYTES + NATIVE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping == MAP_FAILED) + _exit(2); + data = (void *)(((uintptr_t)mapping + NATIVE - 1) & ~(NATIVE - 1)); + memset(data, 0x5a, BYTES); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf, 0); + if (pthread_barrier_init(&barrier, NULL, WORKERS)) + _exit(2); + for (i = 0; i < WORKERS; i++) + if (pthread_create(&threads[i], NULL, worker, (void *)(uintptr_t)i)) + _exit(2); + result.concurrent = true; + for (i = 0; i < WORKERS; i++) { + if (pthread_join(threads[i], &status)) + _exit(2); + result.concurrent &= !status; + } + pthread_barrier_destroy(&barrier); + result.concurrent &= verify(0); + if (!result.concurrent) + _exit(3); + survivor = fork(); + if (survivor < 0) + _exit(2); + if (survivor) { + /* The survivor must retain the original contents across this COW. */ + for (leaf = 0; leaf < 4; leaf++) + data[leaf * LEAF] = tag(leaf, 1); + _exit(verify(1) ? 0 : 3); + } + alarm(30); + /* Supervisor waits for the original allocator's mm to disappear first. */ + if (read(go, &token, 1) != 1 || token != 'g') + _exit(2); + close(go); + result.lifetime = verify(0); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf, 2); + result.lifetime &= verify(2); + /* No memory.max reduction, memory.reclaim or unrelated memory pressure. */ + for (i = 0; i < 20; i++) { + result.attempts = i + 1; + if (madvise(data, BYTES, MADV_COLD) || madvise(data, BYTES, MADV_PAGEOUT)) { + advice_ok = false; + break; + } + usleep(10000); + result.swapped = swapped_leaves(&result.non_swap_markers); + if (result.swapped == SIZE_MAX) + break; + if (result.swapped == LEAVES) + break; + } + readback_ok = verify(2); + result.reclaimed = advice_ok && result.swapped && + result.swapped != SIZE_MAX && readback_ok; + if (write(result_fd, &result, sizeof(result)) != sizeof(result)) + _exit(2); + close(result_fd); + munmap(mapping, BYTES + NATIVE); + _exit(result.concurrent && result.lifetime && result.reclaimed ? 0 : 3); +} + +static bool visible_swap_metadata(void) +{ + FILE *file = fopen("/proc/self/status", "re"); + char line[256]; + unsigned long long caps = 0; + + if (!file) + return false; + while (fgets(line, sizeof(line), file)) + if (sscanf(line, "CapEff: %llx", &caps) == 1) + break; + fclose(file); + return caps & (1ULL << 21); /* CAP_SYS_ADMIN, needed for swap type bits. */ +} + +int main(void) +{ + struct result result = {}; + int go[2], results[2]; + pid_t child; + bool owner_ok, survivor_ok = false; + ssize_t n = 0; + + ksft_print_header(); + if (getauxval(AT_PAGESZ) != LEAF) + ksft_exit_skip("Run through the established 4K ABI launcher\n"); + if (!visible_swap_metadata()) + ksft_exit_skip("CAP_SYS_ADMIN required to distinguish real swap entries\n"); + signal(SIGPIPE, SIG_IGN); + signal(SIGALRM, stop_fixture); + signal(SIGTERM, stop_fixture); + alarm(35); + if (prctl(PR_SET_CHILD_SUBREAPER, 1) || pipe(go) || pipe(results)) + ksft_exit_fail_msg("setup: %s\n", strerror(errno)); + child = fork(); + if (child < 0) + ksft_exit_fail_msg("fork: %s\n", strerror(errno)); + if (!child) { + if (setpgid(0, 0)) + _exit(2); + close(go[1]); + close(results[0]); + owner(go[0], results[1]); + } + fixture_group = child; + if (setpgid(child, child) && errno != EACCES && errno != ESRCH) + stop_fixture(SIGTERM); + close(go[0]); + close(results[1]); + owner_ok = wait_ok(child); + if (owner_ok && write(go[1], "g", 1) == 1) { + n = read(results[0], &result, sizeof(result)); + survivor_ok = wait_ok(-1); + } + close(go[1]); + close(results[0]); + if (!owner_ok || !survivor_ok) { + kill(-child, SIGKILL); + while (waitpid(-1, NULL, 0) > 0 || errno == EINTR) + ; + } + fixture_group = 0; + alarm(0); + ksft_set_plan(3); + ksft_test_result(owner_ok && n == sizeof(result) && result.concurrent, + "four-worker partial-hole refill zeroing and neighbors\n"); + ksft_test_result(owner_ok && n == sizeof(result) && result.lifetime, + "original pool owner exits; descendant fork isolation/readwrite\n"); + ksft_print_msg("pageout attempts=%u real swapped=%zu/%lu excluded markers=%zu\n", + result.attempts, result.swapped, LEAVES, result.non_swap_markers); + ksft_test_result(survivor_ok && result.reclaimed, + "actual bounded pageout and full survivor readback\n"); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/anon-hole-refill-reclaim.c b/tools/testing/selftests/arm64/user4k/anon-hole-refill-reclaim.c new file mode 100644 index 00000000000000..4b15b076b0c1c2 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/anon-hole-refill-reclaim.c @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Opt-in own-cgroup reclaim of shared partial-hole backing after owner exit. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define LEAF 4096UL +#define NATIVE 16384UL +#define BYTES (16UL << 20) +#define LEAVES (BYTES / LEAF) +#define WORKERS 4 +#define ROUNDS 2 + +struct result { + bool concurrent, lifetime, reclaimed; + size_t swapped, non_swap_markers; + unsigned int attempts; +}; + +static unsigned char *data; +static pthread_barrier_t barrier; +static volatile sig_atomic_t fixture_group; + +static void stop_fixture(int signal_number) +{ + if (fixture_group > 0) + kill(-fixture_group, SIGKILL); + _exit(128 + signal_number); +} + +static unsigned char tag(size_t leaf, unsigned int generation) +{ + unsigned char value = leaf % 251 + 1; + + if (generation == 1 && leaf < 4) + value ^= 0xa5; + else if (generation == 2) + value ^= 0x5a; + return value; +} + +static bool verify_leaf(size_t leaf, unsigned int generation) +{ + size_t byte; + + if (data[leaf * LEAF] != tag(leaf, generation)) + return false; + for (byte = 1; byte < LEAF; byte++) + if (data[leaf * LEAF + byte] != 0x5a) + return false; + return true; +} + +static bool verify(unsigned int generation) +{ + size_t leaf; + + for (leaf = 0; leaf < LEAVES; leaf++) + if (!verify_leaf(leaf, generation)) + return false; + return true; +} + +static void *worker(void *arg) +{ + unsigned int id = (uintptr_t)arg, round, quarter, q; + size_t group, leaf, byte; + bool ok = true; + + for (round = 0; round < ROUNDS; round++) { + for (quarter = 0; quarter < 4; quarter++) { + pthread_barrier_wait(&barrier); + for (group = id; group < BYTES / NATIVE; group += WORKERS) { + unsigned char *p; + + leaf = group * 4 + quarter; + p = data + leaf * LEAF; + if (madvise(p, LEAF, MADV_DONTNEED)) { + ok = false; + continue; + } + /* A write fault, without a preceding zero-page read. */ + *(volatile unsigned char *)p = tag(leaf, 0); + for (byte = 1; byte < LEAF; byte++) + if (p[byte]) + ok = false; + memset(p + 1, 0x5a, LEAF - 1); + /* This worker owns the entire native virtual window. */ + for (q = 0; q < 4; q++) + ok &= verify_leaf(group * 4 + q, 0); + } + } + } + return (void *)(uintptr_t)!ok; +} + +static size_t swapped_leaves(size_t *markers) +{ + uint64_t entries[LEAVES]; + size_t i, count = 0; + int fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + ssize_t n; + + if (fd < 0) + return SIZE_MAX; + n = pread(fd, entries, sizeof(entries), (uintptr_t)data / LEAF * 8); + close(fd); + if (n != sizeof(entries)) + return SIZE_MAX; + *markers = 0; + for (i = 0; i < LEAVES; i++) { + if ((entries[i] & (UINT64_C(1) << 63)) || + !(entries[i] & (UINT64_C(1) << 62))) + continue; + /* Audited target config:27..31 are poison/migration/markers. */ + if ((entries[i] & 31) < 27) + count++; + else + (*markers)++; + } + return count; +} + +static bool wait_ok(pid_t pid) +{ + int status; + pid_t got; + + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + return got > 0 && WIFEXITED(status) && !WEXITSTATUS(status); +} + +static void owner(int go, int result_fd) +{ + struct result result = {}; + pthread_t threads[WORKERS]; + void *mapping, *status; + unsigned int i; + bool readback_ok; + struct pollfd control = { .fd = STDIN_FILENO, .events = POLLIN }; + size_t leaf; + pid_t survivor; + char token; + + alarm(30); + mapping = mmap(NULL, BYTES + NATIVE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping == MAP_FAILED) + _exit(2); + data = (void *)(((uintptr_t)mapping + NATIVE - 1) & ~(NATIVE - 1)); + memset(data, 0x5a, BYTES); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf, 0); + if (pthread_barrier_init(&barrier, NULL, WORKERS)) + _exit(2); + for (i = 0; i < WORKERS; i++) + if (pthread_create(&threads[i], NULL, worker, (void *)(uintptr_t)i)) + _exit(2); + result.concurrent = true; + for (i = 0; i < WORKERS; i++) { + if (pthread_join(threads[i], &status)) + _exit(2); + result.concurrent &= !status; + } + pthread_barrier_destroy(&barrier); + result.concurrent &= verify(0); + if (!result.concurrent) + _exit(3); + survivor = fork(); + if (survivor < 0) + _exit(2); + if (survivor) { + /* The survivor must retain the original contents across this COW. */ + for (leaf = 0; leaf < 4; leaf++) + data[leaf * LEAF] = tag(leaf, 1); + _exit(verify(1) ? 0 : 3); + } + alarm(30); + /* Supervisor waits for the original allocator's mm to disappear first. */ + if (read(go, &token, 1) != 1 || token != 'g') + _exit(2); + close(go); + result.lifetime = verify(0); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf, 2); + result.lifetime &= verify(2); + /* Controller acts only on this fixture's cgroup. No pressure here. */ + printf("RECLAIM_READY pid=%d base=%p bytes=%lu owner_mm_exited=1\n", + getpid(), data, BYTES); + fflush(stdout); + if (poll(&control, 1, 20000) != 1 || + read(STDIN_FILENO, &token, 1) != 1 || token != 'r') + _exit(2); + result.attempts = 1; /* One post-controller pagemap snapshot. */ + result.swapped = swapped_leaves(&result.non_swap_markers); + readback_ok = verify(2); + result.reclaimed = result.swapped != SIZE_MAX && + result.swapped >= LEAVES * 3 / 4 && readback_ok; + + if (write(result_fd, &result, sizeof(result)) != sizeof(result)) + _exit(2); + close(result_fd); + munmap(mapping, BYTES + NATIVE); + _exit(result.concurrent && result.lifetime && result.reclaimed ? 0 : 3); +} + +static bool visible_swap_metadata(void) +{ + FILE *file = fopen("/proc/self/status", "re"); + char line[256]; + unsigned long long caps = 0; + + if (!file) + return false; + while (fgets(line, sizeof(line), file)) + if (sscanf(line, "CapEff: %llx", &caps) == 1) + break; + fclose(file); + return caps & (1ULL << 21); /* CAP_SYS_ADMIN, needed for swap type bits. */ +} + +int main(int argc, char **argv) +{ + struct result result = {}; + int go[2], results[2]; + pid_t child; + bool owner_ok, survivor_ok = false; + ssize_t n = 0; + + ksft_print_header(); + fflush(stdout); + if (argc != 2 || strcmp(argv[1], "--external-reclaim")) + ksft_exit_skip("Needs explicit --external-reclaim and a bounded own-cgroup controller\n"); + if (getauxval(AT_PAGESZ) != LEAF) + ksft_exit_skip("Run through the established 4K ABI launcher\n"); + if (!visible_swap_metadata()) + ksft_exit_skip("CAP_SYS_ADMIN required to distinguish real swap entries\n"); + signal(SIGPIPE, SIG_IGN); + signal(SIGALRM, stop_fixture); + signal(SIGTERM, stop_fixture); + alarm(35); + if (prctl(PR_SET_CHILD_SUBREAPER, 1) || pipe(go) || pipe(results)) + ksft_exit_fail_msg("setup: %s\n", strerror(errno)); + child = fork(); + if (child < 0) + ksft_exit_fail_msg("fork: %s\n", strerror(errno)); + if (!child) { + if (setpgid(0, 0)) + _exit(2); + close(go[1]); + close(results[0]); + owner(go[0], results[1]); + } + fixture_group = child; + if (setpgid(child, child) && errno != EACCES && errno != ESRCH) + stop_fixture(SIGTERM); + close(go[0]); + close(results[1]); + owner_ok = wait_ok(child); + if (owner_ok && write(go[1], "g", 1) == 1) { + n = read(results[0], &result, sizeof(result)); + survivor_ok = wait_ok(-1); + } + close(go[1]); + close(results[0]); + if (!owner_ok || !survivor_ok) { + kill(-child, SIGKILL); + while (waitpid(-1, NULL, 0) > 0 || errno == EINTR) + ; + } + fixture_group = 0; + alarm(0); + ksft_set_plan(3); + ksft_test_result(owner_ok && n == sizeof(result) && result.concurrent, + "four-worker partial-hole refill zeroing and neighbors\n"); + ksft_test_result(owner_ok && n == sizeof(result) && result.lifetime, + "original pool owner exits; descendant fork isolation/readwrite\n"); + ksft_print_msg("post-controller snapshots=%u real swapped=%zu/%lu excluded markers=%zu\n", + result.attempts, result.swapped, LEAVES, result.non_swap_markers); + ksft_test_result(survivor_ok && result.reclaimed, + "own-cgroup reclaim swaps at least75percent, full survivor readback\n"); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/anon-hole-refill.c b/tools/testing/selftests/arm64/user4k/anon-hole-refill.c new file mode 100644 index 00000000000000..e2d71e5cce602e --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/anon-hole-refill.c @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Missing-PTE refill must retain contents, faultaround and native density. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define LEAF 4096UL +#define NATIVE 16384UL +#define BYTES (16UL << 20) +#define LEAVES (BYTES / LEAF) + +static unsigned char *data; +static int pm = -1; +static bool density; + +static unsigned char tag(size_t leaf) +{ + return leaf % 251 + 1; +} + +static bool refill(size_t leaf) +{ + unsigned char *p = data + leaf * LEAF; + size_t i; + + /* Write first: a read would test zero-page COW instead of hole refill. */ + *(volatile unsigned char *)p = tag(leaf); + for (i = 1; i < LEAF; i++) + if (p[i]) + return false; + memset(p + 1, 0x5a, LEAF - 1); + return true; +} + +static bool contents(void) +{ + size_t leaf, i; + + for (leaf = 0; leaf < LEAVES; leaf++) { + if (data[leaf * LEAF] != tag(leaf)) + return false; + for (i = 1; i < LEAF; i++) + if (data[leaf * LEAF + i] != 0x5a) + return false; + } + return true; +} + +static int cmp(const void *a, const void *b) +{ + uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b; + + return (x > y) - (x < y); +} + +static int native_bytes(size_t *bytes) +{ + uint64_t entries[LEAVES]; + size_t i, unique = 0; + off_t offset = (uintptr_t)data / LEAF * sizeof(uint64_t); + + if (pm < 0 || pread(pm, entries, sizeof(entries), offset) != sizeof(entries)) + return -1; + for (i = 0; i < LEAVES; i++) { + uint64_t pfn = entries[i] & ((UINT64_C(1) << 55) - 1); + + if (!(entries[i] & (UINT64_C(1) << 63)) || !pfn) + return -1; + entries[i] = pfn / (NATIVE / LEAF); + } + qsort(entries, LEAVES, sizeof(*entries), cmp); + for (i = 0; i < LEAVES; i++) + unique += !i || entries[i] != entries[i - 1]; + *bytes = unique * NATIVE; + return 0; +} + +static void check_density(const char *stage, size_t limit) +{ + size_t bytes; + + if (!density) { + ksft_test_result_skip("%s physical density (needs --native-16k and visible PFNs)\n", stage); + return; + } + if (native_bytes(&bytes)) { + ksft_test_result_fail("%s pagemap unavailable, swapped or hidden\n", stage); + return; + } + ksft_print_msg("%s logical=%lu native=%zu limit=%zu\n", stage, BYTES, bytes, limit); + ksft_test_result(bytes <= limit, "%s physical density\n", stage); +} + +static bool phased_refill(void) +{ + unsigned int quarter; + size_t leaf; + + for (quarter = 0; quarter < 4; quarter++) { + for (leaf = quarter; leaf < LEAVES; leaf += 4) + if (madvise(data + leaf * LEAF, LEAF, MADV_DONTNEED) || !refill(leaf)) + return false; + if (!contents()) + return false; + } + return true; +} + +static bool paired_refill(void) +{ + unsigned int parity; + size_t leaf; + + for (parity = 0; parity < 2; parity++) { + for (leaf = parity; leaf < LEAVES; leaf += 4) { + uint64_t entry; + + if (madvise(data + leaf * LEAF, LEAF, MADV_DONTNEED) || + madvise(data + (leaf + 2) * LEAF, LEAF, MADV_DONTNEED) || + !refill(leaf)) + return false; + /* Present is visible without privileged PFNs. Prove the other + * missing neighbor was prefaulted before touching it ourselves. + */ + if (pm < 0 || pread(pm, &entry, sizeof(entry), + ((uintptr_t)data / LEAF + leaf + 2) * 8) != sizeof(entry) || + !(entry & (UINT64_C(1) << 63)) || !refill(leaf + 2)) + return false; + } + } + return contents(); +} + +int main(int argc, char **argv) +{ + void *mapping; + size_t leaf; + + ksft_print_header(); + if (getauxval(AT_PAGESZ) != LEAF) + ksft_exit_skip("Run through the established 4K ABI launcher\n"); + if (argc == 2 && !strcmp(argv[1], "--native-16k")) + density = true; + else if (argc != 1) + ksft_exit_fail_msg("Usage: %s [--native-16k]\n", argv[0]); + alarm(30); + mapping = mmap(NULL, BYTES + NATIVE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping == MAP_FAILED) + ksft_exit_fail_msg("mmap: %s\n", strerror(errno)); + data = (void *)(((uintptr_t)mapping + NATIVE - 1) & ~(NATIVE - 1)); + pm = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + memset(data, 0x5a, BYTES); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf); + ksft_set_plan(8); + ksft_test_result(contents(), "initial dense content\n"); + check_density("initial dense", BYTES + NATIVE); + ksft_test_result(phased_refill(), "phased hole refill zeroing and neighbors\n"); + /* Allow bounded pool-bucket slop, but reject the old 64MiB outcome. */ + check_density("phased refill", BYTES + (4UL << 20)); + ksft_test_result(phased_refill(), "reused-slot refill zeroing and neighbors\n"); + check_density("reused-slot refill", BYTES + (4UL << 20)); + ksft_test_result(paired_refill(), "two-hole mask retains missing-neighbor prefault\n"); + check_density("two-hole refill", BYTES + (4UL << 20)); + munmap(mapping, BYTES + NATIVE); + if (pm >= 0) + close(pm); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/aslr-fixture-init.c b/tools/testing/selftests/arm64/user4k/aslr-fixture-init.c new file mode 100644 index 00000000000000..4ac03136f2ec0a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/aslr-fixture-init.c @@ -0,0 +1,175 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define SAMPLES 64 +struct sample { + unsigned long page_size, stack, mapping, brk; +}; + +static int get_sample(struct sample *s) +{ + char buf[4096], *pos, *end; + unsigned long low, high; + FILE *f; + int fd; + ssize_t size; + void *p; + + s->page_size = getauxval(AT_PAGESZ); + p = mmap(NULL, s->page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + return 1; + s->mapping = (unsigned long)p; + f = fopen("/proc/self/maps", "r"); + if (!f) + return 1; + while (fgets(buf, sizeof(buf), f)) + if (strstr(buf, "[stack]") && + sscanf(buf, "%lx-%lx", &low, &high) == 2) + s->stack = high; + fclose(f); + fd = open("/proc/self/stat", O_RDONLY); + if (fd < 0) + return 1; + size = read(fd, buf, sizeof(buf) - 1); + close(fd); + if (size <= 0) + return 1; + buf[size] = 0; + pos = strrchr(buf, ')'); + if (!pos) + return 1; + pos += 2; + /* Fields 1 and 2 precede the closing parenthesis; start_brk is 47. */ + for (unsigned int field = 3; field < 47; field++) { + pos = strchr(pos, ' '); + if (!pos) + return 1; + pos++; + } + errno = 0; + s->brk = strtoul(pos, &end, 10); + return errno || end == pos || !s->brk || !s->stack; +} + +static int collect(unsigned int mode, bool disabled, struct sample *s) +{ + int pipefd[2], status; + pid_t pid; + ssize_t size; + + if (pipe(pipefd)) + return 1; + pid = fork(); + if (!pid) { + char fd[32]; + close(pipefd[0]); + snprintf(fd, sizeof(fd), "%d", pipefd[1]); + if (prctl(SET_EXEC_PAGE_SIZE, mode == 1 ? 4096UL : 0UL, 0UL, + 0UL, 0UL)) + _exit(2); + if (disabled && personality(ADDR_NO_RANDOMIZE) < 0) + _exit(3); + execl(mode == 2 ? "/compat-aslr" : "/init", "/init", "--sample", + fd, NULL); + _exit(4); + } + close(pipefd[1]); + size = read(pipefd[0], s, sizeof(*s)); + close(pipefd[0]); + return pid < 0 || waitpid(pid, &status, 0) != pid || + !WIFEXITED(status) || WEXITSTATUS(status) || size != sizeof(*s); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + int failed = 0; + + if (argc == 3 && !strcmp(argv[1], "--sample")) { + struct sample s = { 0 }; + if (getppid() != 1 || get_sample(&s)) + return 1; + return write(atoi(argv[2]), &s, sizeof(s)) != sizeof(s); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Dedicated disposable VM PID1 only\n"); + return 2; + } + setvbuf(stdout, NULL, _IONBF, 0); + if (mount("proc", "/proc", "proc", 0, NULL)) + failed++; + int control = open("/proc/sys/kernel/randomize_va_space", O_WRONLY); + if (control < 0 || write(control, "2", 1) != 1) + failed++; + if (control >= 0) + close(control); + int has4k = !prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL); + if (prctl(SET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL)) + failed++; + for (unsigned int mode = 0; mode < 3; mode++) { + if (mode == 1 && !has4k) + continue; + unsigned long ps = mode && has4k ? 4096 : native; + struct sample first = { 0 }; + bool varied[3] = { 0 }, subnative[3] = { 0 }; + int ok = 1; + + for (unsigned int i = 0; i < SAMPLES; i++) { + struct sample s = { 0 }; + if (collect(mode, false, &s)) { + ok = 0; + break; + } + unsigned long address[] = { s.stack, s.mapping, s.brk }; + unsigned long original[] = { first.stack, first.mapping, + first.brk }; + ok &= s.page_size == ps; + for (unsigned int n = 0; n < 3; n++) { + ok &= !(address[n] & (ps - 1)); + varied[n] |= i && address[n] != original[n]; + subnative[n] |= i && + ((address[n] ^ original[n]) & + (native - 1)); + } + if (!i) + first = s; + } + for (unsigned int n = 0; n < 3; n++) + ok &= varied[n] && (ps == native || subnative[n]); + printf("%s - %s %luK ASLR: 64 execs, stack/mmap/brk varied=%d%d%d low-bits-varied=%d%d%d\n", + ok ? "ok" : "not ok", mode == 2 ? "AArch32" : "AArch64", + ps / 1024, varied[0], varied[1], varied[2], subnative[0], + subnative[1], subnative[2]); + failed += !ok; + struct sample a = { 0 }, b = { 0 }; + ok = !collect(mode, true, &a) && !collect(mode, true, &b) && + !memcmp(&a, &b, sizeof(a)) && a.page_size == ps; + printf("%s - %s %luK ADDR_NO_RANDOMIZE retains stable layout\n", + ok ? "ok" : "not ok", mode == 2 ? "AArch32" : "AArch64", + ps / 1024); + failed += !ok; + } + printf("GRANULE ASLR %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/bpf-uptr-init.c b/tools/testing/selftests/arm64/user4k/bpf-uptr-init.c new file mode 100644 index 00000000000000..33793cebc2b8cc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/bpf-uptr-init.c @@ -0,0 +1,583 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(c) \ + do { \ + if (!(c)) { \ + printf("FAIL line %d: %s errno=%d\n", __LINE__, #c, \ + errno); \ + return 1; \ + } \ + } while (0) +struct user_data { + uint64_t a, b, result, marker; +}; +struct value { + struct user_data *p, *q; + uint64_t cookie; + uint8_t trailing; +} __attribute__((packed)); +static char verifier_log[65536]; +static int bpf(enum bpf_cmd cmd, union bpf_attr *a) +{ + size_t size; +#define END(field) (offsetof(union bpf_attr, field) + sizeof(a->field)) + switch (cmd) { + case BPF_BTF_LOAD: + size = END(btf_log_level); + break; + case BPF_MAP_CREATE: + size = END(btf_value_type_id); + break; + case BPF_PROG_LOAD: + size = END(log_buf); + break; + case BPF_RAW_TRACEPOINT_OPEN: + size = END(raw_tracepoint.prog_fd); + break; + case BPF_MAP_UPDATE_ELEM: + size = END(flags); + break; + case BPF_MAP_LOOKUP_ELEM: + size = END(flags); + break; + case BPF_MAP_DELETE_ELEM: + size = END(key); + break; + default: + errno = EINVAL; + return -1; + } +#undef END + return syscall(__NR_bpf, cmd, a, size); +} +struct btf_blob { + unsigned char types[1024]; + char strings[512]; + unsigned nt, ns; +}; +static uint32_t str(struct btf_blob *b, const char *s) +{ + unsigned at = b->ns; + size_t n = strlen(s) + 1; + memcpy(b->strings + b->ns, s, n); + b->ns += n; + return at; +} +static void push(struct btf_blob *b, const void *p, size_t n) +{ + memcpy(b->types + b->nt, p, n); + b->nt += n; +} +static void type(struct btf_blob *b, const char *name, unsigned kind, + unsigned n, unsigned size_type) +{ + struct btf_type t = { .name_off = name ? str(b, name) : 0, + .info = kind << 24 | n, + .size = size_type }; + push(b, &t, sizeof(t)); +} +static void member(struct btf_blob *b, const char *name, unsigned id, + unsigned offset) +{ + struct btf_member m = { .name_off = str(b, name), + .type = id, + .offset = offset * 8 }; + push(b, &m, sizeof(m)); +} +static int create_map(void) +{ + struct btf_blob b = { .ns = 1 }; + uint32_t encoding; + type(&b, "int", BTF_KIND_INT, 0, 4); + encoding = BTF_INT_SIGNED << 24 | 32; + push(&b, &encoding, 4); + type(&b, "u64", BTF_KIND_INT, 0, 8); + encoding = 64; + push(&b, &encoding, 4); + type(&b, "user_data", BTF_KIND_STRUCT, 4, sizeof(struct user_data)); + member(&b, "a", 2, 0); + member(&b, "b", 2, 8); + member(&b, "result", 2, 16); + member(&b, "marker", 2, 24); + type(&b, "uptr", BTF_KIND_TYPE_TAG, 0, 3); + type(&b, NULL, BTF_KIND_PTR, 0, 4); + type(&b, "value_type", BTF_KIND_STRUCT, 4, sizeof(struct value)); + member(&b, "p", 5, 0); + member(&b, "q", 5, 8); + member(&b, "cookie", 2, 16); + member(&b, "trailing", 7, 24); + type(&b, "u8", BTF_KIND_INT, 0, 1); + encoding = 8; + push(&b, &encoding, 4); + unsigned char data[2048]; + struct btf_header h = { .magic = BTF_MAGIC, + .version = BTF_VERSION, + .hdr_len = sizeof(h), + .type_len = b.nt, + .str_off = b.nt, + .str_len = b.ns }; + memcpy(data, &h, sizeof(h)); + memcpy(data + sizeof(h), b.types, b.nt); + memcpy(data + sizeof(h) + b.nt, b.strings, b.ns); + union bpf_attr a = { .btf = (uintptr_t)data, + .btf_size = sizeof(h) + b.nt + b.ns, + .btf_log_buf = (uintptr_t)verifier_log, + .btf_log_size = sizeof(verifier_log), + .btf_log_level = 1 }; + int btf = bpf(BPF_BTF_LOAD, &a); + if (btf < 0) { + printf("BTF load errno=%d: %s\n", errno, verifier_log); + return -1; + } + a = (union bpf_attr){ .map_type = BPF_MAP_TYPE_TASK_STORAGE, + .key_size = sizeof(int), + .value_size = sizeof(struct value), + .map_flags = BPF_F_NO_PREALLOC, + .btf_fd = btf, + .btf_key_type_id = 1, + .btf_value_type_id = 6 }; + int fd = bpf(BPF_MAP_CREATE, &a); + close(btf); + return fd; +} +#define I(c, d, s, o, i) \ + ((struct bpf_insn){ .code = (c), \ + .dst_reg = (d), \ + .src_reg = (s), \ + .off = (o), \ + .imm = (i) }) +static int create_prog(int map, int *link) +{ + struct bpf_insn insns[40]; + unsigned n = 0, jumps[4]; +#define EMIT(c, d, s, o, i) (insns[n++] = I(c, d, s, o, i)) + EMIT(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_current_task_btf); + EMIT(BPF_ALU64 | BPF_MOV | BPF_X, 2, 0, 0, 0); + EMIT(BPF_LD | BPF_DW | BPF_IMM, 1, BPF_PSEUDO_MAP_FD, 0, map); + EMIT(0, 0, 0, 0, 0); + EMIT(BPF_ALU64 | BPF_MOV | BPF_K, 3, 0, 0, 0); + EMIT(BPF_ALU64 | BPF_MOV | BPF_K, 4, 0, 0, 0); + EMIT(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_task_storage_get); + jumps[0] = n; + EMIT(BPF_JMP | BPF_JEQ | BPF_K, 0, 0, 0, 0); + EMIT(BPF_ALU64 | BPF_MOV | BPF_X, 6, 0, 0, 0); + EMIT(BPF_LDX | BPF_DW | BPF_MEM, 7, 6, 16, 0); + jumps[1] = n; + EMIT(BPF_JMP | BPF_JEQ | BPF_K, 7, 0, 0, 0); + EMIT(BPF_LDX | BPF_DW | BPF_MEM, 1, 6, 0, 0); + jumps[2] = n; + EMIT(BPF_JMP | BPF_JEQ | BPF_K, 1, 0, 0, 0); + EMIT(BPF_LDX | BPF_DW | BPF_MEM, 2, 1, 0, 0); + EMIT(BPF_ALU64 | BPF_XOR | BPF_X, 2, 7, 0, 0); + EMIT(BPF_STX | BPF_DW | BPF_MEM, 1, 2, 16, 0); + EMIT(BPF_STX | BPF_DW | BPF_MEM, 1, 7, 24, 0); + insns[jumps[2]].off = n - jumps[2] - 1; + EMIT(BPF_LDX | BPF_DW | BPF_MEM, 1, 6, 8, 0); + jumps[3] = n; + EMIT(BPF_JMP | BPF_JEQ | BPF_K, 1, 0, 0, 0); + EMIT(BPF_LDX | BPF_DW | BPF_MEM, 2, 1, 8, 0); + EMIT(BPF_ALU64 | BPF_ADD | BPF_X, 2, 7, 0, 0); + EMIT(BPF_STX | BPF_DW | BPF_MEM, 1, 2, 16, 0); + EMIT(BPF_STX | BPF_DW | BPF_MEM, 1, 7, 24, 0); + for (unsigned i = 0; i < 4; i++) + if (i != 2) + insns[jumps[i]].off = n - jumps[i] - 1; + EMIT(BPF_ALU64 | BPF_MOV | BPF_K, 0, 0, 0, 0); + EMIT(BPF_JMP | BPF_EXIT, 0, 0, 0, 0); +#undef EMIT + union bpf_attr a = { .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT, + .insn_cnt = n, + .insns = (uintptr_t)insns, + .license = (uintptr_t)"GPL", + .log_buf = (uintptr_t)verifier_log, + .log_size = sizeof(verifier_log), + .log_level = 1 }; + verifier_log[0] = 0; + int fd = bpf(BPF_PROG_LOAD, &a); + if (fd < 0) { + printf("prog load errno=%d: %s\n", errno, verifier_log); + return -1; + } + a = (union bpf_attr){}; + a.raw_tracepoint.name = (uintptr_t)"sys_enter"; + a.raw_tracepoint.prog_fd = fd; + *link = bpf(BPF_RAW_TRACEPOINT_OPEN, &a); + if (*link < 0) { + close(fd); + return -1; + } + return fd; +} +static int update(int map, int key, struct value *v, unsigned flags) +{ + union bpf_attr a = { .map_fd = map, + .key = (uintptr_t)&key, + .value = (uintptr_t)v, + .flags = flags }; + return bpf(BPF_MAP_UPDATE_ELEM, &a); +} +static int delete(int map, int key) +{ + union bpf_attr a = { .map_fd = map, .key = (uintptr_t)&key }; + return bpf(BPF_MAP_DELETE_ELEM, &a); +} +struct pins { + unsigned long long acquired, released; +}; +static int pin_stats(struct pins *p) +{ + int fd = open("/proc/sys/vm/stat_refresh", O_RDONLY); + char c; + if (fd < 0) + return -1; + int ret = read(fd, &c, 1); + close(fd); + if (ret < 0) + return -1; + FILE *f = fopen("/proc/vmstat", "r"); + if (!f) + return -1; + char name[128]; + unsigned long long value; + unsigned found = 0; + while (fscanf(f, "%127s %llu", name, &value) == 2) { + if (!strcmp(name, "nr_foll_pin_acquired")) { + p->acquired = value; + found |= 1; + } + if (!strcmp(name, "nr_foll_pin_released")) { + p->released = value; + found |= 2; + } + } + fclose(f); + return found == 3 ? 0 : -1; +} +static int wait_pins(struct pins base, unsigned active) +{ + struct pins p = {}; + for (unsigned i = 0; i < 500; i++) { + if (pin_stats(&p)) + return 0; + if (p.acquired - base.acquired >= active && + p.acquired - base.acquired - (p.released - base.released) == + active) + return 1; + usleep(10000); + } + printf("pin delta acquired=%llu released=%llu expected-active=%u\n", + p.acquired - base.acquired, p.released - base.released, active); + return 0; +} +static int filled(unsigned char *p, size_t len, unsigned char value) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != value) + return 0; + return 1; +} +static int values(struct user_data *p, struct user_data *q, uint64_t cookie) +{ + return p->result == (p->a ^ cookie) && q->result == q->b + cookie && + p->marker == cookie && q->marker == cookie; +} +static int exercise_case(size_t ps, unsigned shared) +{ + struct pins base; + CHECK(!pin_stats(&base)); + int backing = -1; + if (shared) { + backing = memfd_create("bpf-uptr", 0); + CHECK(backing >= 0 && !ftruncate(backing, 4 * ps)); + } + unsigned char *src = mmap( + (void *)0x20000000, 4 * ps, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | + (shared ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS), + backing, 0); + unsigned char *dst = + mmap((void *)0x30000000, 4 * ps, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(src != MAP_FAILED && dst != MAP_FAILED); + for (unsigned i = 0; i < 4; i++) + memset(src + i * ps, 0x61 + i, ps); + memset(dst, 0x91, 4 * ps); + CHECK(mremap(src + 3 * ps, ps, ps, MREMAP_MAYMOVE | MREMAP_FIXED, + dst + ps) == dst + ps); + unsigned char *leaf = dst + ps; + struct user_data *p = (void *)(leaf + ps - 64), *q = p + 1; + *p = (struct user_data){ .a = 0x12345678, .b = 3 }; + *q = (struct user_data){ .a = 4, .b = 0x87654321 }; + int map = create_map(); + CHECK(map >= 0); + int link = -1, prog = create_prog(map, &link); + CHECK(prog >= 0 && link >= 0); + int key = syscall(SYS_pidfd_open, getpid(), 0); + CHECK(key >= 0); + struct value probe = {}; + int probe_ret = update(map, key, &probe, BPF_NOEXIST); + CHECK(!probe_ret); + CHECK(!delete(map, key)); + struct value v = { + .p = p, .q = q, .cookie = 0xabc001, .trailing = 0x7d + }; + CHECK(!update(map, key, &v, BPF_NOEXIST)); + CHECK(wait_pins(base, 2)); + syscall(SYS_getpid); + CHECK(values(p, q, v.cookie)); + for (unsigned i = 0; i < 64; i++) { + v.cookie++; + CHECK(!update(map, key, &v, BPF_EXIST)); + syscall(SYS_getpid); + CHECK(values(p, q, v.cookie)); + } + CHECK(wait_pins(base, 2)); + struct { + struct value v; + unsigned char tail[16]; + } lookup; + memset(&lookup, 0xa5, sizeof(lookup)); + union bpf_attr a = { .map_fd = map, + .key = (uintptr_t)&key, + .value = (uintptr_t)&lookup }; + CHECK(!bpf(BPF_MAP_LOOKUP_ELEM, &a)); + CHECK(!lookup.v.p && !lookup.v.q && lookup.v.cookie == v.cookie && + lookup.v.trailing == v.trailing && + filled(lookup.tail, sizeof(lookup.tail), 0xa5)); + errno = 0; + CHECK(update(map, key, &v, BPF_NOEXIST) == -1 && errno == EEXIST); + CHECK(wait_pins(base, 2)); + struct value half = v; + half.p = NULL; + CHECK(!update(map, key, &half, BPF_EXIST)); + CHECK(wait_pins(base, 1)); + p->result = p->marker = 0; + syscall(SYS_getpid); + CHECK(!p->result && !p->marker && q->result == q->b + v.cookie); + CHECK(!update(map, key, &v, BPF_EXIST)); + CHECK(wait_pins(base, 2)); + struct value bad = v; + bad.q = (void *)5; + errno = 0; + CHECK(update(map, key, &bad, BPF_EXIST) == -1 && errno == EFAULT); + CHECK(wait_pins(base, 2)); + syscall(SYS_getpid); + CHECK(values(p, q, v.cookie)); + bad.q = (void *)(leaf + ps - 16); + errno = 0; + CHECK(update(map, key, &bad, BPF_EXIST) == -1 && errno == EOPNOTSUPP); + CHECK(wait_pins(base, 2)); + bad.q = (void *)(UINTPTR_MAX - 15); + errno = 0; + CHECK(update(map, key, &bad, BPF_EXIST) == -1 && errno == EFAULT); + CHECK(wait_pins(base, 2)); + CHECK(!mprotect(leaf, ps, PROT_READ)); + errno = 0; + CHECK(update(map, key, &v, BPF_EXIST) == -1 && errno == EFAULT); + CHECK(!mprotect(leaf, ps, PROT_READ | PROT_WRITE)); + CHECK(wait_pins(base, 2)); + struct value empty = {}; + CHECK(!update(map, key, &empty, BPF_EXIST)); + CHECK(wait_pins(base, 0)); + p->result = q->result = p->marker = q->marker = 0; + syscall(SYS_getpid); + CHECK(!p->result && !q->result); + CHECK(!update(map, key, &v, BPF_EXIST)); + CHECK(wait_pins(base, 2)); + /* The child owns storage pointing into the parent's pinned memory. */ + int barrier[2]; + CHECK(!pipe(barrier)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + close(barrier[1]); + char token; + if (read(barrier[0], &token, 1) != 1) + _exit(3); + if (!shared) { + p->a = 1; + q->b = 2; + } + syscall(SYS_getpid); + _exit(0); + } + close(barrier[0]); + int childkey = syscall(SYS_pidfd_open, child, 0); + CHECK(childkey >= 0); + CHECK(!update(map, childkey, &v, BPF_NOEXIST)); + CHECK(!delete(map, key)); + CHECK(wait_pins(base, 2)); + p->result = q->result = p->marker = q->marker = 0; + CHECK(write(barrier[1], "x", 1) == 1); + close(barrier[1]); + int status; + CHECK(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status)); + close(childkey); + CHECK(values(p, q, v.cookie) && p->a == 0x12345678 && + q->b == 0x87654321); + CHECK(wait_pins(base, 0)); + /* The supplying mm exits while a different task retains its UPTRs. */ + int foreign_pipe[2]; + CHECK(!pipe(foreign_pipe)); + pid_t supplier = fork(); + CHECK(supplier >= 0); + if (!supplier) { + close(foreign_pipe[0]); + *p = (struct user_data){ .a = 0x1111, .b = 3 }; + *q = (struct user_data){ .a = 4, .b = 0x2222 }; + struct value foreign = { .p = p, .q = q, .cookie = 0x765432 }; + if (update(map, key, &foreign, BPF_NOEXIST)) + _exit(4); + struct iovec data = { .iov_base = p, .iov_len = 64 }; + if (vmsplice(foreign_pipe[1], &data, 1, 0) != 64) + _exit(5); + _exit(0); + } + close(foreign_pipe[1]); + CHECK(waitpid(supplier, &status, 0) == supplier && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(wait_pins(base, 2)); + syscall(SYS_getpid); + struct user_data retired[2]; + size_t done = 0; + while (done < sizeof(retired)) { + ssize_t got = read(foreign_pipe[0], (char *)retired + done, + sizeof(retired) - done); + CHECK(got > 0); + done += got; + } + close(foreign_pipe[0]); + CHECK(retired[0].a == 0x1111 && retired[1].b == 0x2222 && + values(retired, retired + 1, 0x765432)); + CHECK(!delete(map, key) && wait_pins(base, 0)); + *p = (struct user_data){ .a = 0x12345678, .b = 3 }; + *q = (struct user_data){ .a = 4, .b = 0x87654321 }; + CHECK(!update(map, key, &v, BPF_NOEXIST)); + CHECK(wait_pins(base, 2)); + CHECK(filled(leaf, ps - 64, 0x64)); + int pipefd[2]; + CHECK(!pipe(pipefd)); + struct iovec iov = { .iov_base = leaf, .iov_len = ps }; + CHECK(vmsplice(pipefd[1], &iov, 1, 0) == (ssize_t)ps); + CHECK(!munmap(leaf, ps)); + CHECK(mmap(leaf, ps, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, -1, + 0) == leaf); + memset(leaf, 0xe7, ps); + unsigned char *old = malloc(ps); + CHECK(old); + size_t n = 0; + while (n < ps) { + ssize_t got = read(pipefd[0], old + n, ps - n); + CHECK(got > 0); + n += got; + } + close(pipefd[0]); + close(pipefd[1]); + CHECK(filled(old, ps - 64, 0x64) && + values((void *)(old + ps - 64), (void *)(old + ps - 32), + v.cookie)); + CHECK(filled(leaf, ps, 0xe7)); + free(old); + /* Closing the map alone is insufficient: the program retains its map ref. */ + close(map); + close(link); + close(prog); + close(key); + CHECK(wait_pins(base, 0)); + CHECK(filled(dst, ps, 0x91) && filled(dst + 2 * ps, 2 * ps, 0x91)); + for (unsigned i = 0; i < 3; i++) + CHECK(filled(src + i * ps, ps, 0x61 + i)); + CHECK(!munmap(src, 3 * ps) && !munmap(dst, 4 * ps)); + if (backing >= 0) + close(backing); + printf("ok - %zuK BPF UPTR %s: relocated writes, updates, rollback, storage/supplier exit, unmap and pin balance\n", + ps / 1024, shared ? "shared" : "private"); + return 0; +} +static int exercise(size_t ps) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(90); + for (unsigned shared = 0; shared < 2; shared++) + CHECK(!exercise_case(ps, shared)); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + int fail = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)) { + perror("mount"); + fail = 1; + goto done; + } + size_t sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + for (unsigned i = 0; i < (sizes[0] > 16384 ? 3U : 2U); i++) { + pid_t c = fork(); + int status = 0; + if (!c) { + char size[24]; + snprintf(size, sizeof(size), "%zu", sizes[i]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", size, NULL); + _exit(5); + } + int ok = 0; + for (unsigned ticks = 0; c > 0 && ticks < 1000; ticks++) { + pid_t got = waitpid(c, &status, WNOHANG); + if (got == c) { + ok = WIFEXITED(status) && !WEXITSTATUS(status); + break; + } + if (ticks == 999) { + kill(c, SIGKILL); + waitpid(c, &status, 0); + break; + } + usleep(100000); + } + printf("%s - ABI %zuK BPF UPTR status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + fail += !ok; + } +done: + printf("BPF UPTR %s\n", fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/check-any-quarter.py b/tools/testing/selftests/arm64/user4k/check-any-quarter.py new file mode 100644 index 00000000000000..1b71b3922783f5 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/check-any-quarter.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +"""Validate native backing density from the unchanged same-quarter oracle.""" +import json,sys +from pathlib import Path + +def check(path): + rows=[json.loads(x) for x in Path(path).read_text().splitlines() if x.startswith('{')] + assert len(rows)==2, 'Require exactly two measured arms' + arms={x['arm']:x for x in rows} + assert set(arms)=={'same-quarter-zero','balanced'} + for name,r in arms.items(): + assert r['logical_parent_bytes']==16*1024*1024 and r['modified_child_bytes']==4*1024*1024 + assert r['content_ok'] and r['disjoint_ok'] + assert r['modified_parent_native_overlap']==0 and r['modified_unchanged_native_overlap']==0 + assert r['modified_child']['represented_bytes']==4*1024*1024 + n=r['modified_child']['native_bytes'] + assert 4*1024*1024<=n<=6*1024*1024, (name,'native backing outside packed bound',n) + assert sum(arms['same-quarter-zero']['modified_physical_quarter_counts'][1:])>0 + print('PASS same-quarter/balanced native density, actual relocation and isolation') + +if __name__=='__main__': + if len(sys.argv)!=2:sys.exit('usage: check-any-quarter.py same-quarter.log') + try:check(sys.argv[1]) + except (AssertionError,KeyError,ValueError,OSError) as e:sys.exit('FAIL '+str(e)) diff --git a/tools/testing/selftests/arm64/user4k/clipped-firstwrite.c b/tools/testing/selftests/arm64/user4k/clipped-firstwrite.c new file mode 100644 index 00000000000000..c07cb1a8a980f1 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/clipped-firstwrite.c @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Real clipped-window batching, zeroing, reuse and physical backing density. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define LEAF 4096UL +#define NATIVE 16384UL +#define GROUPS 256UL +#define STRIDE (4 * NATIVE) +#define PFN_MASK ((UINT64_C(1)<<55)-1) +#define PRESENT (UINT64_C(1)<<63) +#define SWAPPED (UINT64_C(1)<<62) +static volatile sig_atomic_t child; +static void stop(int sig) { if(child>0)kill(child,SIGKILL);_exit(128+sig); } +static bool visible(void) +{ + FILE *f=fopen("/proc/self/status","re");char line[256];unsigned long long caps=0; + if(!f)return false; + while(fgets(line,sizeof(line),f))if(sscanf(line,"CapEff: %llx",&caps)==1)break; + fclose(f);return !!(caps&(1ULL<<21)); +} +static bool entries(int fd,unsigned char *base,unsigned n,uint64_t *out) +{ + if(pread(fd,out,n*8,(uintptr_t)base/LEAF*8)!=(ssize_t)n*8)return false; + for(unsigned i=0;iy)-(x>2; + if(mix){ + p=base+g*STRIDE+2*NATIVE; + if(!check_window(p,1,g+GROUPS,round)||!entries(fd,p,1,map))return 4; + native[used++]=(map[0]&PFN_MASK)>>2; + } + } + qsort(native,used,sizeof(*native),compare);size_t unique=0; + for(size_t i=0;itarget+(1UL<<20)){fprintf(stderr,"FAIL physical density\n");return 5;} + if(!round)for(unsigned g=0;g +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "swapfile-fixture.h" + +static int setting(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY | O_CLOEXEC); + ssize_t n; + + if (fd < 0) + return -1; + n = write(fd, value, strlen(value)); + close(fd); + return n == (ssize_t)strlen(value) ? 0 : -1; +} + +static int fault_setting(const char *kind, const char *name, const char *value) +{ + char path[192]; + + snprintf(path, sizeof(path), "/sys/kernel/debug/%s/%s", kind, name); + return setting(path, value); +} + +static int injection_off(void) +{ + int ret = fault_setting("failslab", "probability", "0"); + + ret |= fault_setting("fail_page_alloc", "probability", "0"); + ret |= fault_setting("failslab", "ignore-gfp-wait", "Y"); + ret |= fault_setting("fail_page_alloc", "ignore-gfp-wait", "Y"); + ret |= fault_setting("fail_page_alloc", "ignore-gfp-highmem", "Y"); + return ret; +} + +static int injection_on(bool page) +{ + const char *kind = page ? "fail_page_alloc" : "failslab"; + + if (injection_off() || fault_setting(kind, "task-filter", "Y") || + fault_setting(kind, "verbose", "0") || + fault_setting(kind, "ignore-gfp-wait", "N") || + fault_setting(kind, "times", "-1") || + fault_setting(kind, "interval", "1") || + fault_setting(kind, "space", "0")) + return -1; + if (!page) + return 0; /* Only current->fail_nth selects the slab allocation. */ + return fault_setting(kind, "ignore-gfp-highmem", "N") || + fault_setting(kind, "min-order", "2") || + fault_setting(kind, "probability", "100"); +} + +static int move_worker(const char *text, bool enter, bool swap_limit) +{ + char *end, path[192], pid[32], comm[64]; + long nr = strtol(text, &end, 10); + const char *group = swap_limit ? "coarse-swap-limit" : "coarse-zero"; + int fd; + ssize_t n; + + if (*end || nr <= 1 || nr == getpid()) + return -1; + snprintf(path, sizeof(path), "/proc/%ld/comm", nr); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return -1; + n = read(fd, comm, sizeof(comm) - 1); + close(fd); + if (n < 1) + return -1; + comm[n] = 0; + if (strncmp(comm, "kunit_", 6)) + return -1; + if (enter) { + if (mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL) && + errno != EBUSY) + return -1; + if (setting("/sys/fs/cgroup/cgroup.subtree_control", "+memory")) + return -1; + snprintf(path, sizeof(path), "/sys/fs/cgroup/%s", group); + if (mkdir(path, 0755) && errno != EEXIST) + return -1; + snprintf(path, sizeof(path), "/sys/fs/cgroup/%s/memory.max", + group); + if (setting(path, swap_limit ? "max" : "0")) + return -1; + if (swap_limit && + setting("/sys/fs/cgroup/coarse-swap-limit/memory.swap.max", + "0")) + return -1; + } + snprintf(pid, sizeof(pid), "%ld", nr); + if (enter) + snprintf(path, sizeof(path), "/sys/fs/cgroup/%s/cgroup.procs", + group); + else + snprintf(path, sizeof(path), "/sys/fs/cgroup/cgroup.procs"); + return setting(path, pid); +} + +static int helper(const char *op) +{ + struct stat init, self; + + if (getpid() == 1 || stat("/proc/1/exe", &init) || + stat("/proc/self/exe", &self) || init.st_dev != self.st_dev || + init.st_ino != self.st_ino) + return 1; + if (!strcmp(op, "--coarse-fixture-check")) + return getauxval(AT_PAGESZ) != 16384; + if (!strcmp(op, "--swapoff-helper")) + return swapoff(swap_path()) != 0; + if (!strcmp(op, "--swapon-helper")) + return swapon(swap_path(), 0) != 0; + if (!strncmp(op, "--coarse-swap-enter=", 20)) + return move_worker(op + 20, true, true) != 0; + if (!strncmp(op, "--coarse-swap-leave=", 20)) + return move_worker(op + 20, false, true) != 0; + if (!strcmp(op, "--coarse-swap-unlimit")) + return setting("/sys/fs/cgroup/coarse-swap-limit/memory.swap.max", + "max") != 0; + if (!strcmp(op, "--coarse-injection-off")) + return injection_off() != 0; + if (!strcmp(op, "--coarse-slab-on")) + return injection_on(false) != 0; + if (!strcmp(op, "--coarse-page-on")) + return injection_on(true) != 0; + if (!strncmp(op, "--coarse-memcg-enter=", 21)) + return move_worker(op + 21, true, false) != 0; + if (!strncmp(op, "--coarse-memcg-leave=", 21)) + return move_worker(op + 21, false, false) != 0; + return 1; +} + +static int run_suite(void) +{ + const char *base = "/sys/kernel/debug/kunit/arm64-user4k-coarse-shmem/"; + char path[192], *log = calloc(1, 1024 * 1024); + size_t used = 0, capacity = 1024 * 1024; + ssize_t n = 0; + int fd, ret = -1; + + if (!log) + return -1; + snprintf(path, sizeof(path), "%srun", base); + if (setting(path, "1")) + goto out; + snprintf(path, sizeof(path), "%sresults", base); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + goto out; + while (used < capacity - 1 && + (n = read(fd, log + used, capacity - used - 1)) > 0) + used += n; + close(fd); + fputs(log, stdout); + if (!n && used < capacity - 1 && + strstr(log, "\nok 1 arm64-user4k-coarse-shmem\n") && + !strstr(log, "not ok") && !strstr(log, "# SKIP")) + ret = 0; +out: + free(log); + return ret; +} + +int main(int argc, char **argv) +{ + struct stat st; + bool active = false; + int ret = 1; + unsigned int i; + + if (getpid() != 1 && argc == 2) + return helper(argv[1]); + if (getpid() != 1 || + (argc != 1 && (argc != 2 || strcmp(argv[1], "--swapfile")))) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + if (getauxval(AT_PAGESZ) != 16384) + goto out; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)) + goto out; + for (i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + if (stat("/dev/vda", &st) || !S_ISBLK(st.st_mode)) + goto out; + if (argc == 2 && prepare_swapfile()) + goto out; + printf("COARSE SWAP BACKING=%s\n", swap_path()); + if (swapon(swap_path(), 0)) + goto out; + active = true; + ret = run_suite() ? 1 : 0; + if (injection_off()) + ret = 1; +out: + if (active && swapoff(swap_path())) + ret = 1; + if (active && argc == 2 && umount("/swapdir")) + ret = 1; + printf("COARSE SHMEM SWAP PASS=%d errno=%d\n", !ret, errno); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/compat-aslr-probe.c b/tools/testing/selftests/arm64/user4k/compat-aslr-probe.c new file mode 100644 index 00000000000000..3a88db75112151 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-aslr-probe.c @@ -0,0 +1,134 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Freestanding ELF32 address sampler for the disposable ASLR fixture. */ +typedef unsigned int u32; +typedef unsigned long long u64; +struct sample { + u64 page_size, stack, mapping, brk; +}; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} +static int error(long value) +{ + return (u32)value >= 0xfffff001U; +} +static unsigned number(const char *p) +{ + unsigned n = 0; + while (*p >= '0' && *p <= '9') + n = n * 10 + *p++ - '0'; + return n; +} +static char *find(char *p, char c) +{ + while (*p && *p != c) + p++; + return *p ? p : (char *)0; +} +static char *find_stack(char *p) +{ + const char name[] = "[stack]"; + while (*p) { + unsigned i = 0; + while (name[i] && p[i] == name[i]) + i++; + if (!name[i]) + return p; + p++; + } + return (char *)0; +} +static unsigned hex(char *p) +{ + unsigned n = 0, v; + for (;;) { + if (*p >= '0' && *p <= '9') + v = *p - '0'; + else if (*p >= 'a' && *p <= 'f') + v = *p - 'a' + 10; + else + return n; + n = (n << 4) | v; + p++; + } +} +static int load(const char *path, char *buf, unsigned size) +{ + long fd = call(5, (long)path, 0, 0, 0, 0, 0), n; + if (error(fd)) + return 1; + n = call(3, fd, (long)buf, size - 1, 0, 0, 0); + call(6, fd, 0, 0, 0, 0, 0); + if (n <= 0) + return 1; + buf[n] = 0; + return 0; +} +__attribute__((noreturn)) void probe(u32 *sp) +{ + unsigned argc = *sp++; + char **argv = (void *)sp, buf[4096], *p, *q; + struct sample sample = { 0 }; + long mapping; + int bad = 1; + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + sample.page_size = sp[1]; + sp += 2; + } + if (argc != 3 || call(64, 0, 0, 0, 0, 0, 0) != 1) + goto done; + mapping = call(192, 0, sample.page_size, 3, 0x22, -1, 0); + if (error(mapping)) + goto done; + sample.mapping = (u32)mapping; + if (load("/proc/self/maps", buf, sizeof(buf))) + goto done; + p = find_stack(buf); + if (!p) + goto done; + while (p > buf && p[-1] != '\n') + p--; + p = find(p, '-'); + if (!p) + goto done; + sample.stack = hex(p + 1); + if (load("/proc/self/stat", buf, sizeof(buf))) + goto done; + p = find(buf, ')'); + if (!p) + goto done; + while ((q = find(p + 1, ')'))) + p = q; + p += 2; + for (unsigned field = 3; field < 47; field++) { + p = find(p, ' '); + if (!p) + goto done; + p++; + } + sample.brk = number(p); + if (!sample.page_size || !sample.stack || !sample.brk) + goto done; + bad = call(4, number(argv[2]), (long)&sample, sizeof(sample), 0, 0, + 0) != sizeof(sample); +done: + call(1, bad, 0, 0, 0, 0, 0); + for (;;) { + } +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-fixture-init.c b/tools/testing/selftests/arm64/user4k/compat-fixture-init.c new file mode 100644 index 00000000000000..f576ed93623edd --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-fixture-init.c @@ -0,0 +1,149 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +static int failed; +static unsigned long native_size; +static void result(int ok, const char *s) +{ + printf("%s - launcher %s\n", ok ? "ok" : "not ok", s); + if (!ok) + failed++; +} +static int launch(const char *path, char **args, int request, int expect_errno) +{ + pid_t child = fork(); + int status = 0; + if (!child) { + if (request && prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL)) + _exit(95); + unsigned char *canary = + mmap(NULL, native_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (canary == MAP_FAILED) + _exit(94); + memset(canary, 0x69, native_size); + int before = prctl(0x41555002, 0UL, 0UL, 0UL, 0UL); + for (int i = 0; i < (expect_errno ? 64 : 1); i++) { + execv(path, args); + if (!expect_errno || errno != expect_errno) + _exit(96); + if (getauxval(AT_PAGESZ) != native_size || + prctl(0x41555002, 0UL, 0UL, 0UL, 0UL) != before) + _exit(93); + for (int j = 0; j < native_size; j++) + if (canary[j] != 0x69) + _exit(92); + } + _exit(0); + } + if (child < 0 || waitpid(child, &status, 0) != child) + return 0; + if (status) + printf("# child %s status=%#x\n", path, status); + return WIFEXITED(status) && !WEXITSTATUS(status); +} +int main(int argc, char **argv) +{ + if (argc > 1 && !strcmp(argv[1], "--returned")) + return getauxval(AT_PAGESZ) == + strtoul(getenv("AURORA_NATIVE_PAGE_SIZE"), + NULL, 10) ? + 0 : + 1; + if (argc > 1 && !strcmp(argv[1], "--returned4k")) + return getauxval(AT_PAGESZ) == 4096 ? 0 : 1; + if (getpid() != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + native_size = getauxval(AT_PAGESZ); + char native_text[24]; + snprintf(native_text, sizeof(native_text), "%lu", native_size); + setenv("AURORA_NATIVE_PAGE_SIZE", native_text, 1); + setvbuf(stdout, NULL, _IONBF, 0); + mount("proc", "/proc", "proc", 0, NULL); + int no4k = argc > 1 && !strcmp(argv[1], "--no4k"), + fd = memfd_create("compat-offsets", 0); + if (no4k) { + errno = 0; + result(prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL) == -1 && + errno == EOPNOTSUPP, + "4KB capability is actually masked"); + } + result(fd >= 0 && !ftruncate(fd, 0x100000000LL + 4 * native_size), + "sparse file fixture above 4GB"); + unsigned char page[4096]; + for (unsigned q = 0; q < 4 * native_size / 4096; q++) { + memset(page, 0x40 + q, sizeof(page)); + if (pwrite(fd, page, sizeof(page), q * 4096) != sizeof(page)) + failed++; + memset(page, 0x80 + q, sizeof(page)); + if (pwrite(fd, page, sizeof(page), 0x100000000LL + q * 4096) != + sizeof(page)) + failed++; + } + char fdstr[24]; + snprintf(fdstr, sizeof(fdstr), "%d", fd); + char *arg = malloc(90001); + for (unsigned i = 0; i < 90000; i++) + arg[i] = 'a' + i % 23; + arg[90000] = 0; + char *args[] = { "/probe32-64k", no4k ? native_text : "4096", fdstr, + arg, NULL }; + result(launch(args[0], args, 0, 0), + "64KB-aligned ELF32 automatically selects available ABI"); + if (!no4k) { + args[0] = "/probe32-4k"; + result(launch(args[0], args, 0, 0), + "4KB-aligned ELF32 executes with 4KB translation"); + args[0] = "/probe32-64k"; + result(launch(args[0], args, 1, 0), + "explicit pending 4KB request allows ELF32"); + } + args[0] = "/script32"; + result(launch(args[0], args, 0, 0), + "script final ELF class determines page granule"); + args[0] = "/probe32-dynamic"; + result(launch(args[0], args, 0, 0), + "ELF32 interpreter mapping uses selected page granule"); + char *chain[] = { + "/probe32-64k", "--exec32", no4k ? native_text : "4096", + fdstr, arg, NULL + }; + result(launch(chain[0], chain, 0, 0), + "ELF32 to ELF32 restages arguments and selects ABI"); + char *back[] = { "/probe32-64k", "--exec64", NULL }; + result(launch(back[0], back, 0, 0), + "ELF32 to ELF64 exec restores native default"); + if (!no4k) { + back[1] = "--exec64-4k"; + result(launch(back[0], back, 0, 0), + "ELF32 can request alternative ELF64 exec granule"); + } + char *big = malloc(200001); + memset(big, 'x', 200000); + big[200000] = 0; + args[0] = "/probe32-64k"; + args[3] = big; + if (!no4k) + result(launch(args[0], args, 0, E2BIG), + "64 oversized transfers return E2BIG and preserve old mm and request"); + printf("COMPAT FIXTURE %s mode=%s\n", failed ? "FAIL" : "PASS", + no4k ? "no4k-native-fallback" : "auto4k"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/compat-mlock-init.c b/tools/testing/selftests/arm64/user4k/compat-mlock-init.c new file mode 100644 index 00000000000000..679c39370cd6dc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-mlock-init.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +struct result { + uint32_t pagesize, passed, failed, error; +}; +int main(int argc, char **argv) +{ + struct result out = { 0 }; + unsigned long expected; + int pipes[2], status = -1; + pid_t pid; + size_t done = 0; + int ok; + + if (getpid() != 1 || + (argc != 1 && (argc != 2 || strcmp(argv[1], "--no4k")))) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + expected = argc == 2 ? getauxval(AT_PAGESZ) : 4096; + if (mount("proc", "/proc", "proc", 0, NULL) || pipe(pipes)) + return 3; + pid = fork(); + if (!pid) { + char value[32]; + close(pipes[0]); + if (dup2(pipes[1], 1) < 0) + _exit(4); + close(pipes[1]); + snprintf(value, sizeof(value), "%lu", expected); + execl("/compat-mlock", "/compat-mlock", value, NULL); + _exit(5); + } + close(pipes[1]); + while (done < sizeof(out)) { + ssize_t n = + read(pipes[0], (char *)&out + done, sizeof(out) - done); + if (n < 0 && errno == EINTR) + continue; + if (n <= 0) + break; + done += n; + } + close(pipes[0]); + ok = pid > 0 && waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status) && done == sizeof(out) && + out.pagesize == expected && out.passed == 6 && !out.failed; + printf("%s - AArch32 memlock expected=%lu actual=%u passed=%u stage=%u error=%d status=%d\n", + ok ? "ok" : "not ok", expected, out.pagesize, out.passed, + out.failed, (int32_t)out.error, status); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/compat-mlock-probe.c b/tools/testing/selftests/arm64/user4k/compat-mlock-probe.c new file mode 100644 index 00000000000000..97e76dd94bae95 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-mlock-probe.c @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding ELF32 memlock probe, only for the disposable PID1 fixture. */ +typedef unsigned int u32; +struct result { + u32 pagesize, passed, failed, error; +}; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} +static int error(long n) +{ + return (u32)n >= 0xfffff001U; +} +static unsigned number(const char *p) +{ + unsigned n = 0; + while (*p >= '0' && *p <= '9') + n = n * 10 + *p++ - '0'; + return n; +} +static long locked_kb(void) +{ + char buf[8192]; + long fd = call(5, (long)"/proc/self/status", 0, 0, 0, 0, 0), n; + const char name[] = "VmLck:"; + + if (error(fd)) + return -1; + n = call(3, fd, (long)buf, sizeof(buf) - 1, 0, 0, 0); + call(6, fd, 0, 0, 0, 0, 0); + if (n <= 0) + return -1; + buf[n] = 0; + for (long i = 0; i < n; i++) { + unsigned j = 0; + while (name[j] && i + j < (unsigned)n && buf[i + j] == name[j]) + j++; + if (!name[j]) { + const char *p = buf + i + j; + while (*p == ' ' || *p == '\t') + p++; + return number(p); + } + } + return -1; +} +__attribute__((noreturn)) void probe(u32 *sp) +{ + unsigned argc = *sp++, ps = 0, kb, stage = 1; + char **argv = (void *)sp; + struct result out = { 0 }; + long p, q, pid, ret = -1; + int status = -1; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + ps = sp[1]; + sp += 2; + } + out.pagesize = ps; + if (argc != 2 || call(64, 0, 0, 0, 0, 0, 0) != 1 || + ps != number(argv[1])) + goto done; + kb = ps / 1024; + p = call(192, 0, 8 * ps, 3, 0x22, -1, 0); + if (error(p) || ((u32)p & (ps - 1)) || locked_kb() != 0) + goto done; + ret = call(150, p + ps + 17, ps - 32, 0, 0, 0, 0); + if (ret || locked_kb() != (long)kb) + goto done; + out.passed++; + stage++; + ret = call(150, p + ps, ps, 0, 0, 0, 0); + if (ret || locked_kb() != (long)kb) + goto done; + ret = call(150, p + 2 * ps, ps, 0, 0, 0, 0); + if (ret || locked_kb() != (long)(2 * kb)) + goto done; + ret = call(151, p + ps, ps, 0, 0, 0, 0); + if (ret || locked_kb() != (long)kb) + goto done; + out.passed++; + stage++; + *(volatile unsigned char *)(p + 2 * ps) = 0x63; + q = call(163, p + 2 * ps, ps, 2 * ps, 1, 0, 0); + ret = q; + if (error(q) || locked_kb() != (long)(2 * kb) || + *(volatile unsigned char *)q != 0x63) + goto done; + for (unsigned i = ps; i < 2 * ps; i++) + if (((volatile unsigned char *)q)[i]) + goto done; + out.passed++; + stage++; + pid = call(2, 0, 0, 0, 0, 0, 0); + if (!pid) { + call(1, locked_kb() != 0, 0, 0, 0, 0, 0); + for (;;) { + } + } + ret = call(114, pid, (long)&status, 0, 0, 0, 0); + if (error(pid) || ret != pid || status || locked_kb() != (long)(2 * kb)) + goto done; + out.passed++; + stage++; + ret = call(91, q, 2 * ps, 0, 0, 0, 0); + if (ret || locked_kb() != 0) + goto done; + out.passed++; + stage++; + ret = call(390, p + 4 * ps, 2 * ps, 1, 0, 0, 0); + if (ret || locked_kb() != (long)(2 * kb)) + goto done; + *(volatile unsigned char *)(p + 4 * ps) = 0x75; + ret = call(151, p + 4 * ps, 2 * ps, 0, 0, 0, 0); + if (ret || locked_kb() != 0) + goto done; + out.passed++; + stage = 0; +done: + out.failed = stage; + out.error = stage ? (u32)ret : 0; + call(4, 1, (long)&out, sizeof(out), 0, 0, 0); + call(1, stage != 0, 0, 0, 0, 0, 0); + for (;;) { + } +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-numa-balancing-probe.c b/tools/testing/selftests/arm64/user4k/compat-numa-balancing-probe.c new file mode 100644 index 00000000000000..f2c155a888c15b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-numa-balancing-probe.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 automatic NUMA migration and fallback checks. */ +typedef unsigned int u32; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} + +static unsigned int number(const char *s) +{ + unsigned int n = 0; + + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} + +static _Noreturn void finish(int code) +{ + char text[] = "A32 NUMAB FAIL stage=00\n"; + + if (code) { + text[sizeof(text) - 4] = '0' + code / 10; + text[sizeof(text) - 3] = '0' + code % 10; + call(4, 1, (long)text, sizeof(text) - 1, 0, 0, 0); + } else { + static const char pass[] = "A32 NUMAB PASS\n"; + + call(4, 1, (long)pass, sizeof(pass) - 1, 0, 0, 0); + } + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} + +static void value(const char *label, unsigned int length, u32 n) +{ + char text[16]; + unsigned int pos = sizeof(text); + + text[--pos] = '\n'; + do { + text[--pos] = '0' + n % 10; + n /= 10; + } while (n); + call(4, 1, (long)label, length, 0, 0, 0); + call(4, 1, (long)(text + pos), sizeof(text) - pos, 0, 0, 0); +} + +static u32 seconds(void) +{ + struct { + long long sec, nsec; + } ts; + + /* Use time64 even when CONFIG_COMPAT_32BIT_TIME is disabled. */ + if (call(403, 1, (long)&ts, 0, 0, 0, 0)) + finish(11); + return ts.sec; +} + +static void cpu(unsigned int id) +{ + u32 mask = 1U << id; + + if (call(241, 0, sizeof(mask), (long)&mask, 0, 0, 0)) + finish(12); +} + +static int node_at(unsigned char *address) +{ + int node = -1; + + if (call(320, (long)&node, 0, 0, (long)address, 3, 0)) + finish(13); + return node; +} + +static u32 sample(unsigned char *base, u32 length, u32 page) +{ + u32 moved = 0; + + for (u32 off = 0; off < length; off += page) + if (off != 2 * page) + moved += node_at(base + off + 16) == 1; + return moved; +} + +static void forbidden(unsigned char *address, int write_access) +{ + long child = call(2, 0, 0, 0, 0, 0, 0); + int status = 0; + + if (child < 0) + finish(40); + if (!child) { + /* Force the intended access; this is not thread synchronization. */ + if (write_access) + *(volatile unsigned char *)address = 0x27; + else + (void)*(volatile unsigned char *)address; + call(1, 2, 0, 0, 0, 0, 0); + for (;;) + ; + } + if (call(114, child, (long)&status, 0, 0, 0, 0) != child || + (status & 0x7f) != 11) + finish(41); +} + +static void mapping(u32 page) +{ + const u32 length = 8U << 20; + u32 mask[2] = { 1, 0 }, got[2] = { 0, 0 }; + u32 loops = 0, moved = 0, started, total = length / page - 1; + int mode = -1; + long raw; + unsigned char *base; + + if (call(320, (long)&mode, (long)got, 64, 0, 4, 0) || got[0] != 3 || + got[1]) + finish(20); + cpu(0); + raw = call(192, 0, length, 3, 0x22, -1, 0); + if ((u32)raw >= 0xfffff001U) + finish(21); + base = (void *)raw; + /* A missing 4K capability must retain native alignment restrictions. */ + if (call(125, raw + 4096, 4096, 1, 0, 0, 0) != (page == 4096 ? 0 : -22)) + finish(22); + if (call(125, raw, length, 3, 0, 0, 0)) + finish(23); + if (call(220, raw, length, 15, 0, 0, 0)) /* MADV_NOHUGEPAGE */ + finish(24); + if (call(319, raw, length, 2, (long)mask, 64, 0)) + finish(25); + for (u32 off = 0; off < length; off += page) { + u32 *p = (void *)(base + off); + + p[0] = 0x91720000 + off / page; + p[1] = 0; + p[page / 4 - 1] = 0x51320000 + off / page; + if (node_at(base + off + 16) != 0) + finish(26); + } + if (call(125, raw + page, page, 1, 0, 0, 0) || + call(125, raw + 2 * page, page, 0, 0, 0, 0)) + finish(27); + if (call(319, raw, length, 0, 0, 0, + 0)) /* MPOL_DEFAULT, no MOVE flags */ + finish(28); + cpu(1); + started = seconds(); + do { + for (u32 off = 0; off < length; off += page) { + volatile u32 *p = (void *)(base + off); + + if (off == 2 * page) + continue; + if (p[0] != 0x91720000 + off / page || + p[page / 4 - 1] != 0x51320000 + off / page) + finish(29); + if (off != page) + p[1]++; + } + loops++; + if (!(loops % 128)) { + moved = sample(base, length, page); + if (moved == total) + break; + } + } while (seconds() - started < 30); + moved = sample(base, length, page); + value("A32 NUMAB page=", sizeof("A32 NUMAB page=") - 1, page); + value("A32 NUMAB moved=", sizeof("A32 NUMAB moved=") - 1, moved); + value("A32 NUMAB expected=", sizeof("A32 NUMAB expected=") - 1, total); + value("A32 NUMAB loops=", sizeof("A32 NUMAB loops=") - 1, loops); + if (moved != total) + finish(30); + for (u32 off = 0; off < length; off += page) + if (off != page && off != 2 * page && + *(u32 *)(base + off + 4) != loops) + finish(31); + forbidden(base + page, 1); + forbidden(base + 2 * page, 0); + if (call(125, raw + 2 * page, page, 1, 0, 0, 0) || + *(u32 *)(base + 2 * page) != 0x91720002) + finish(32); + if (call(91, raw, length, 0, 0, 0, 0)) + finish(33); +} + +_Noreturn void probe(u32 *sp) +{ + u32 argc = *sp++, page = 0; + char **argv = (void *)sp; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + page = sp[1]; + sp += 2; + } + if (argc != 2 || page != number(argv[1])) + finish(10); + mapping(page); + finish(0); +} + +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-numa-probe.c b/tools/testing/selftests/arm64/user4k/compat-numa-probe.c new file mode 100644 index 00000000000000..b2652fe348b6ea --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-numa-probe.c @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 NUMA policy and native-granule fallback checks. */ +typedef unsigned int u32; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} + +static unsigned int number(const char *s) +{ + unsigned int n = 0; + + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} + +static _Noreturn void finish(int code) +{ + char text[] = "A32 NUMA FAIL stage=00\n"; + + if (code) { + text[sizeof(text) - 4] = '0' + code / 10; + text[sizeof(text) - 3] = '0' + code % 10; + call(4, 1, (long)text, sizeof(text) - 1, 0, 0, 0); + } else { + static const char pass[] = "A32 NUMA PASS\n"; + + call(4, 1, (long)pass, sizeof(pass) - 1, 0, 0, 0); + } + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} + +static void mapping(unsigned long native, unsigned long page) +{ + unsigned long length = 4 * native; + long raw = call(192, 0, length, 3, 0x22, -1, 0); + unsigned char *base; + u32 mask[2] = { 1, 0 }, got[2] = { 0, 0 }; + int mode = -1, node; + + if ((unsigned long)raw >= 0xfffff001UL) + finish(20); + base = (void *)(((unsigned long)raw + native - 1) & ~(native - 1)); + if (call(320, (long)&mode, (long)got, 64, 0, 4, 0) || got[0] != 3 || + got[1]) + finish(21); + if (call(319, (long)base, 2 * native, 2, (long)mask, 64, 0)) + finish(22); + mask[0] = 2; + if (call(319, (long)(base + page), 1, 2, (long)mask, 64, 0)) + finish(23); + if (call(450, (long)(base + page), 1, 1, 0, 0, 0)) + finish(24); + if (call(320, (long)&mode, (long)got, 64, (long)(base + page), 2, 0) || + mode != 2 || got[0] != 2 || got[1]) + finish(25); + for (unsigned long off = 0; off < 2 * native; off += page) { + base[off + 17] = 0x51 + off / page; + node = -1; + if (call(320, (long)&node, 0, 0, (long)(base + off + 17), 3, + 0) || + node != (off == page ? 1 : 0)) + finish(26); + } + if (call(319, (long)base, 2 * native, 2, (long)mask, 64, 1) != -5) + finish(27); + if (call(319, (long)(base + 4096), 4096, 2, (long)mask, 64, 0) != + (page == 4096 ? 0 : -22)) + finish(28); + if (call(319, (long)base, 2 * native, 2, (long)mask, 64, 5)) + finish(29); + for (unsigned long off = 0; off < 2 * native; off += page) { + node = -1; + if (base[off + 17] != (unsigned char)(0x51 + off / page) || + call(320, (long)&node, 0, 0, (long)(base + off + 17), 3, + 0) || + node != 1) + finish(30); + } + if (call(91, raw, length, 0, 0, 0, 0)) + finish(31); +} + +_Noreturn void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + if (argc != 3 || pages != number(argv[2])) + finish(10); + mapping(number(argv[1]), pages); + finish(0); +} + +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-perf-probe.c b/tools/testing/selftests/arm64/user4k/compat-perf-probe.c new file mode 100644 index 00000000000000..ee02fc34051adb --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-perf-probe.c @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 perf mmap layout, including native-page fallback. */ +typedef unsigned int u32; +typedef unsigned long long u64; + +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} + +static unsigned int number(const char *s) +{ + unsigned int n = 0; + + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} + +static _Noreturn void finish(int code) +{ + char text[] = "A32 PERF FAIL stage=00\n"; + + if (code) { + text[20] = '0' + code / 10; + text[21] = '0' + code % 10; + call(4, 1, (long)text, sizeof(text) - 1, 0, 0, 0); + } else { + static const char pass[] = "A32 PERF PASS\n"; + + call(4, 1, (long)pass, sizeof(pass) - 1, 0, 0, 0); + } + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} + +static void mapping(unsigned long pages, unsigned long native, + unsigned int count) +{ + static const struct { + u32 type, size; + u64 config, sample_period, sample_type, read_format, flags; + u32 wakeup, bp_type; + u64 config1; + } attr = { .type = 1, .size = 64, .flags = 1 }; + long fd = call(364, (long)&attr, 0, -1, -1, 0, 0); + unsigned long length = (1 + count) * pages; + long raw; + volatile u64 *metadata; + + if (fd < 0) + finish(20); + raw = call(192, 0, length, 3, 1, fd, 0); + if ((unsigned long)raw >= 0xfffff001UL) + finish(21); + metadata = (void *)(raw + 1024); + if (metadata[2] != pages || metadata[3] != count * pages) + finish(22); + metadata[1] = 0; + metadata[6] = length; + metadata[7] = native; + /* mmap2 always takes 4K units, including the no4K compat fallback. */ + if (call(192, 0, native, 1, 1, fd, length / 4096) != -95) + finish(23); + if (call(91, raw, length, 0, 0, 0, 0)) + finish(24); + if (call(6, fd, 0, 0, 0, 0, 0)) + finish(25); +} + +_Noreturn void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + if (argc != 3 || pages != number(argv[2])) + finish(10); + unsigned long native = number(argv[1]); + + mapping(pages, native, 0); + mapping(pages, native, 1); + mapping(pages, native, 2); + mapping(pages, native, 2 * native / pages); + finish(0); +} + +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-policy-probe.c b/tools/testing/selftests/arm64/user4k/compat-policy-probe.c new file mode 100644 index 00000000000000..dab5a79abe1302 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-policy-probe.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 -> AArch64 exec with an inherited granule preference. */ +#include "../../../../../include/uapi/linux/prctl.h" +typedef unsigned int u32; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} +static unsigned int number(const char *s) +{ + unsigned int n = 0; + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} +static void fail(int code) +{ + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} +__attribute__((noreturn)) void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + sp += argc + 1; + char **env = (void *)sp; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + if (argc != 4 || pages != number(argv[3])) + fail(10); + if (call(172, PR_AURORA_GET_DEFAULT_PAGE_SIZE, 0, 0, 0, 0, 0) != + number(argv[2])) + fail(11); + if (call(172, PR_AURORA_GET_EXEC_PAGE_SIZE, 0, 0, 0, 0, 0)) + fail(12); + long mem = call(192, 0, 4 * pages, 3, 0x22, -1, 0); + if ((unsigned long)mem >= 0xfffff001UL) + fail(13); + long ret = call(125, mem + 4096, 4096, 1, 0, 0, 0); + if (ret != (pages == 4096 ? 0 : -22)) + fail(14); + if (call(91, mem, 4 * pages, 0, 0, 0, 0)) + fail(15); + char *args[] = { "/init", "--compat-return", argv[1], argv[2], 0 }; + call(11, (long)args[0], (long)args, (long)env, 0, 0, 0); + fail(16); + for (;;) + ; +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-probe.c b/tools/testing/selftests/arm64/user4k/compat-probe.c new file mode 100644 index 00000000000000..14a4deb7164126 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-probe.c @@ -0,0 +1,248 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Freestanding AArch32 process; run only by compat-fixture-init in a guest. */ +typedef unsigned int u32; +typedef unsigned short u16; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f, r7 __asm__("r7") = n; + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} +static unsigned length(const char *s) +{ + unsigned n = 0; + while (s[n]) + n++; + return n; +} +static int equal(const char *a, const char *b) +{ + while (*a && *a == *b) { + a++; + b++; + } + return *a == *b; +} +static unsigned number(const char *s) +{ + unsigned n = 0; + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} +static void out(const char *s) +{ + call(4, 1, (long)s, length(s), 0, 0, 0); +} +static int failures, checks; +static void result(int pass, const char *s) +{ + checks++; + if (!pass) + failures++; + out(pass ? "ok - compat " : "not ok - compat "); + out(s); + out("\n"); +} +static int error(long v) +{ + return (unsigned long)v >= 0xfffff001UL; +} +static long map(unsigned size, int prot, int flags, int fd, unsigned off) +{ + return call(192, 0, size, prot, flags, fd, off); +} +static void unmap(long p, unsigned size) +{ + if (!error(p)) + call(91, p, size, 0, 0, 0, 0); +} +static volatile unsigned signals; +static void handler(int n) +{ + if (n == 10) + signals++; +} +struct action { + u32 handler, flags, restorer, mask[2]; +}; +struct ehdr { + unsigned char ident[16]; + u16 type, machine; + u32 version, entry, phoff, shoff, flags; + u16 ehsize, phentsize, phnum, shentsize, shnum, shstrndx; +}; +struct phdr { + u32 type, offset, vaddr, paddr, filesz, memsz, flags, align; +}; +struct dyn { + u32 tag, val; +}; +struct sym { + u32 name, value, size; + unsigned char info, other; + u16 shndx; +}; +static void *vdso_symbol(unsigned base, const char *name) +{ + struct ehdr *h = (void *)base; + struct phdr *p = (void *)(base + h->phoff); + struct dyn *d = 0; + char *strings = 0; + struct sym *symbols = 0; + u32 *hash = 0; + for (unsigned i = 0; i < h->phnum; i++) + if (p[i].type == 2) + d = (void *)(base + p[i].vaddr); + if (!d) + return 0; + for (; d->tag; d++) { + if (d->tag == 5) + strings = (void *)(base + d->val); + if (d->tag == 6) + symbols = (void *)(base + d->val); + if (d->tag == 4) + hash = (void *)(base + d->val); + } + if (!strings || !symbols || !hash) + return 0; + for (unsigned i = 0; i < hash[1]; i++) + if (equal(strings + symbols[i].name, name)) + return (void *)(base + symbols[i].value); + return 0; +} +__attribute__((noreturn)) void probe(u32 *sp) +{ + unsigned argc = *sp++, pagesz = 0, vdso = 0; + char **argv = (void *)sp; + sp += argc + 1; + char **env = (void *)sp; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pagesz = sp[1]; + if (sp[0] == 33) + vdso = sp[1]; + sp += 2; + } + if (argc > 1 && equal(argv[1], "--exec32")) { + char *a[] = { argv[0], argv[2], argv[3], argv[4], 0 }; + call(11, (long)a[0], (long)a, (long)env, 0, 0, 0); + call(1, 97, 0, 0, 0, 0, 0); + for (;;) + ; + } + if (argc > 1 && + (equal(argv[1], "--exec64") || equal(argv[1], "--exec64-4k"))) { + int alternative = equal(argv[1], "--exec64-4k"); + if (alternative && call(172, 0x41555001, 4096, 0, 0, 0, 0)) { + call(1, 99, 0, 0, 0, 0, 0); + for (;;) + ; + } + char *a[] = { "/init", + alternative ? "--returned4k" : "--returned", 0 }; + call(11, (long)a[0], (long)a, (long)env, 0, 0, 0); + call(1, 98, 0, 0, 0, 0, 0); + for (;;) + ; + } + /* Script interpreter contributes its pathname before the ordinary arguments. */ + unsigned start = argc > 1 && equal(argv[1], "/script32") ? 2 : 1; + unsigned expected = argc > start ? number(argv[start]) : 0; + int fd = argc > start + 1 ? (int)number(argv[start + 1]) : -1; + result(pagesz == expected, "AT_PAGESZ reflects final ELF32 selection"); + int args = argc > start + 2 && length(argv[start + 2]) == 90000; + if (args) + for (unsigned i = 0; i < 90000; i++) + if (argv[start + 2][i] != (char)('a' + i % 23)) { + args = 0; + break; + } + result(args, "90KB staged argument survives granule transfer"); + long mem = map(4 * pagesz, 3, 0x22, -1, 0); + int ok = !error(mem); + if (ok) { + unsigned char *p = (void *)mem; + for (unsigned i = 0; i < 4 * pagesz; i++) + p[i] = 0x30 + i / pagesz; + for (unsigned i = 0; i < 4 * pagesz; i++) + if (p[i] != (unsigned char)(0x30 + i / pagesz)) + ok = 0; + } + result(ok, "anonymous user leaves have independent bytes"); + result(!error(mem) && !call(125, mem + pagesz, pagesz, 1, 0, 0, 0) && + !call(125, mem + pagesz, pagesz, 3, 0, 0, 0), + "mprotect at selected page boundaries"); + result(!error(mem) && call(125, mem + 1, pagesz, 1, 0, 0, 0) == -22 && + call(91, mem + 1, pagesz, 0, 0, 0, 0) == -22, + "unaligned memory operations retain restrictions"); + long q = map(pagesz, 1, 1, fd, 3 * (pagesz >> 12)); + ok = !error(q); + if (ok) + for (unsigned i = 0; i < pagesz; i++) + if (((unsigned char *)q)[i] != + (unsigned char)(0x40 + (3 * pagesz + i) / 4096)) + ok = 0; + result(ok, "mmap2 native-aligned file offset"); + unmap(q, pagesz); + q = map(4096, 1, 1, fd, 1); + ok = pagesz == 4096 ? !error(q) && *(unsigned char *)q == 0x41 : + q == -22; + result(ok, "mmap2 4KB offset accepted only by matching ABI"); + unmap(q, pagesz); + q = map(pagesz, 1, 1, fd, 0x100000 + (pagesz >> 12)); + ok = !error(q); + if (ok) + for (unsigned i = 0; i < pagesz; i++) + if (((unsigned char *)q)[i] != + (unsigned char)(0x80 + (pagesz + i) / 4096)) + ok = 0; + result(ok, "mmap2 offset above 4GB remains full width"); + unmap(q, pagesz); + long child = call(2, 0, 0, 0, 0, 0, 0); + int status = 0; + if (!child) { + if (!error(mem)) + *(unsigned char *)mem = 0xa7; + call(1, 0, 0, 0, 0, 0, 0); + for (;;) + ; + } + ok = child > 0 && + call(114, child, (long)&status, 0, 0, 0, 0) == child && !status && + !error(mem) && *(unsigned char *)mem == 0x30; + result(ok, "fork isolates private COW data"); + unmap(mem, 4 * pagesz); + struct action act = { (u32)handler, 0, 0, { 0, 0 } }; + ok = !call(174, 10, (long)&act, 0, 8, 0, 0); + if (ok) + ok = !call(37, call(20, 0, 0, 0, 0, 0, 0), 10, 0, 0, 0, 0) && + signals == 1; + result(ok, "AArch32 signal frame and kernel sigreturn trampoline"); + volatile unsigned word = 17; + int (*cmpxchg)(unsigned, unsigned, volatile unsigned *) = + (void *)0xffff0fc0; + result(*(volatile unsigned *)0xffff0ffc >= 5 && + !cmpxchg(17, 29, &word) && word == 29, + "kuser helper page and atomic compare exchange"); + int (*gettime)(int, void *) = + vdso ? (void *)vdso_symbol(vdso, "__vdso_clock_gettime") : 0; + long time[2] = { 0, 0 }; + result(gettime && !gettime(1, time) && time[0] >= 0 && time[1] >= 0 && + time[1] < 1000000000, + "compat vDSO clock_gettime and vvar mapping"); + out(failures ? "COMPAT PROBE FAIL\n" : "COMPAT PROBE PASS\n"); + call(1, failures ? 1 : 0, 0, 0, 0, 0, 0); + for (;;) + ; +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-script b/tools/testing/selftests/arm64/user4k/compat-script new file mode 100644 index 00000000000000..9b56d0c7567f53 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-script @@ -0,0 +1 @@ +#!/probe32-64k diff --git a/tools/testing/selftests/arm64/user4k/compat-shm-probe.c b/tools/testing/selftests/arm64/user4k/compat-shm-probe.c new file mode 100644 index 00000000000000..c4ee1da2c5a120 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-shm-probe.c @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 SysV attach/detach and fallback-alignment checks. */ +typedef unsigned int u32; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} + +static unsigned int number(const char *s) +{ + unsigned int n = 0; + + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} + +static _Noreturn void finish(int code) +{ + char text[] = "A32 SHM FAIL stage=00\n"; + + if (code) { + text[19] = '0' + code / 10; + text[20] = '0' + code % 10; + call(4, 1, (long)text, sizeof(text) - 1, 0, 0, 0); + } else { + static const char pass[] = "A32 SHM PASS\n"; + + call(4, 1, (long)pass, sizeof(pass) - 1, 0, 0, 0); + } + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} + +static void mapping(unsigned long pages, int rounded) +{ + unsigned long align = 4 * pages, length = 4 * pages; + long raw = call(192, 0, 12 * align, 3, 0x22, -1, 0); + unsigned char *target; + long id, result; + + if ((unsigned long)raw >= 0xfffff001UL) + finish(20); + result = call(125, raw + 4096, 4096, 1, 0, 0, 0); + if (result != (pages == 4096 ? 0 : -22)) + finish(21); + if (call(125, raw, 12 * align, 3, 0, 0, 0)) + finish(22); + target = (void *)(((unsigned long)raw + align - 1) & ~(align - 1)); + target += align + (rounded ? 0 : pages); + target[-1] = 0x51; + target[length] = 0x72; + if (call(91, (long)target, length, 0, 0, 0, 0)) + finish(23); + id = call(307, 0, length - 1, 0600, 0, 0, 0); + if (id < 0) + finish(24); + result = call(305, id, (long)(target + (rounded ? pages + 17 : 0)), + rounded ? 020000 : 0, 0, 0, 0); + if (result != (long)target) { + call(308, id, 0, 0, 0, 0, 0); + finish(rounded ? 26 : 25); + } + if (call(308, id, 0, 0, 0, 0, 0)) + finish(27); + target[pages + 13] = 0x67; + if (call(125, (long)(target + pages), pages, 1, 0, 0, 0)) + finish(28); + if (call(91, (long)target, pages, 0, 0, 0, 0)) + finish(29); + if (call(306, (long)target, 0, 0, 0, 0, 0)) + finish(30); + if (target[-1] != 0x51 || target[length] != 0x72) + finish(31); + if (call(91, raw, 12 * align, 0, 0, 0, 0)) + finish(32); +} + +_Noreturn void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + if (argc != 3 || pages != number(argv[2])) + finish(10); + mapping(pages, 0); + mapping(pages, 1); + finish(0); +} + +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-uffd-init.c b/tools/testing/selftests/arm64/user4k/compat-uffd-init.c new file mode 100644 index 00000000000000..5f1e90f3e484f3 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-uffd-init.c @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - compat uffd line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +struct wire { + uint32_t address, pagesize; +}; +static void expect_invalid(int fd, uint64_t dst, uint64_t src, uint64_t len) +{ + struct uffdio_copy c = { .dst = dst, .src = src, .len = len }; + errno = 0; + CHECK(ioctl(fd, UFFDIO_COPY, &c) == -1 && errno == EINVAL); +} +static void run(unsigned long native, unsigned long expected, int shared) +{ + int sock[2], file = -1, status; + CHECK(!socketpair(AF_UNIX, SOCK_SEQPACKET, 0, sock)); + if (shared) { + file = memfd_create("compat-uffd", 0); + CHECK(file >= 0 && !ftruncate(file, 3 * expected)); + } + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char page_arg[24], sock_arg[24], file_arg[24]; + close(sock[0]); + snprintf(page_arg, sizeof(page_arg), "%lu", expected); + snprintf(sock_arg, sizeof(sock_arg), "%d", sock[1]); + snprintf(file_arg, sizeof(file_arg), "%d", file); + execl("/compat-uffd", "/compat-uffd", page_arg, sock_arg, + file_arg, NULL); + _exit(99); + } + close(sock[1]); + struct wire data = { 0 }; + struct iovec iov = { &data, sizeof(data) }; + union { + struct cmsghdr align; + char bytes[CMSG_SPACE(sizeof(int))]; + } control = { 0 }; + struct msghdr msg = { .msg_iov = &iov, + .msg_iovlen = 1, + .msg_control = control.bytes, + .msg_controllen = sizeof(control.bytes) }; + CHECK(recvmsg(sock[0], &msg, 0) == sizeof(data)); + CHECK(!(msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC)) && + data.pagesize == expected); + struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg); + CHECK(cmsg && cmsg->cmsg_level == SOL_SOCKET && + cmsg->cmsg_type == SCM_RIGHTS && + cmsg->cmsg_len == CMSG_LEN(sizeof(int))); + int uffd; + memcpy(&uffd, CMSG_DATA(cmsg), sizeof(uffd)); + /* Force the source above every possible AArch32 task limit. */ + unsigned char *source = + mmap((void *)0x1000000000UL, 2 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(source != MAP_FAILED && (uintptr_t)source > UINT32_MAX); + /* Source alignment is independent of both processes' granules. */ + unsigned char *bytes = source + 17; + for (unsigned leaf = 0; leaf < 3; leaf++) { + struct pollfd pfd = { .fd = uffd, .events = POLLIN }; + struct uffd_msg event; + CHECK(poll(&pfd, 1, 10000) == 1 && (pfd.revents & POLLIN)); + CHECK(read(uffd, &event, sizeof(event)) == sizeof(event)); + uint64_t dst = (uint64_t)data.address + leaf * expected; + CHECK(event.event == UFFD_EVENT_PAGEFAULT && + !event.arg.pagefault.flags && + event.arg.pagefault.address == dst); + if (!leaf) { + expect_invalid(uffd, dst, (uintptr_t)bytes, 0); + expect_invalid(uffd, dst, (uintptr_t)bytes, + expected - 1); + expect_invalid(uffd, dst + 1, (uintptr_t)bytes, + expected); + expect_invalid(uffd, 0x100000000ULL, (uintptr_t)bytes, + expected); + expect_invalid(uffd, dst, UINT64_MAX - expected + 1, + expected); + if (expected > 4096) + expect_invalid(uffd, dst, (uintptr_t)bytes, + 4096); + } + if (leaf == 1) { + /* A valid but inaccessible pager source must leave the target missing. */ + CHECK(!mprotect(source, 2 * native, PROT_NONE)); + struct uffdio_copy bad = { .dst = dst, + .src = (uintptr_t)bytes, + .len = expected }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &bad) == -1 && + errno == EFAULT && bad.copy == -EFAULT); + CHECK(!mprotect(source, 2 * native, + PROT_READ | PROT_WRITE)); + } + memset(bytes, 0x51 + leaf, expected); + struct uffdio_copy c = { .dst = dst, + .src = (uintptr_t)bytes, + .len = expected }; + CHECK(!ioctl(uffd, UFFDIO_COPY, &c) && + c.copy == (long long)expected); + } + CHECK(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(!munmap(source, 2 * native)); + CHECK(!close(uffd) && !close(sock[0])); + if (shared) + CHECK(!close(file)); + printf("ok - native %lu pager high unaligned COPY to AArch32 %lu %s, source failure and target alignment\n", + native, expected, shared ? "shared" : "anonymous"); +} +int main(void) +{ + if (getpid() != 1) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + alarm(90); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + char cmdline[2048] = { 0 }; + int fd = open("/proc/cmdline", O_RDONLY); + CHECK(fd >= 0 && read(fd, cmdline, sizeof(cmdline) - 1) > 0 && + !close(fd)); + unsigned long native = getauxval(AT_PAGESZ); + unsigned long expected = + strstr(cmdline, "id_aa64mmfr0.tgran4=f") ? native : 4096; + run(native, expected, 0); + run(native, expected, 1); + puts("COMPAT UFFD PASS"); + reboot(RB_POWER_OFF); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/compat-uffd-probe.c b/tools/testing/selftests/arm64/user4k/compat-uffd-probe.c new file mode 100644 index 00000000000000..6abcdb86bb4d82 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-uffd-probe.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding ELF32 half of compat-uffd-init; no armhf sysroot required. */ +typedef unsigned int u32; +typedef unsigned long long u64; +struct api { + u64 api, features, ioctls; +}; +struct registration { + u64 start, len, mode, ioctls; +}; +struct movement { + u64 dst, src, len, mode; + long long moved; +}; +struct wire { + u32 address, pagesize; +}; +struct iovec32 { + void *base; + u32 len; +}; +struct message32 { + void *name; + u32 namelen; + struct iovec32 *iov; + u32 iovlen; + void *control; + u32 controllen, flags; +}; +struct rights32 { + u32 len, level, type; + int fd; +}; +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} +static u32 number(const char *s) +{ + u32 n = 0; + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} +__attribute__((noreturn)) static void finish(int code) +{ + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} +__attribute__((noreturn)) void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + if (argc != 4 || pages != number(argv[1])) + finish(10); + int sock = number(argv[2]), shared = argv[3][0] != '-'; + int file = shared ? (int)number(argv[3]) : -1; + long map = call(192, 0, 3 * pages, 3, shared ? 1 : 0x22, file, 0); + if ((unsigned long)map >= 0xfffff001UL) + finish(11); + /* Fine operations must retain native alignment restrictions on fallback. */ + if (call(125, map + 4096, 4096, 1, 0, 0, 0) != + (pages == 4096 ? 0 : -22)) + finish(12); + if (call(125, map, 3 * pages, 3, 0, 0, 0)) + finish(13); + long fd = call(388, 0x80801, 0, 0, 0, 0, + 0); /* CLOEXEC|NONBLOCK|USER_MODE_ONLY */ + if (fd < 0) + finish(14); + /* Negotiate shared missing support under both fine and fallback ABIs. */ + struct api api = { 0xaa, (1ULL << 5) | (1ULL << 16), 0 }; + if (call(54, fd, 0xc018aa3fU, (long)&api, 0, 0, 0) || + !(api.features & (1ULL << 5))) + finish(15); + struct registration reg = { (u32)map, 3 * pages, 1, 0 }; + if (call(54, fd, 0xc020aa00U, (long)®, 0, 0, 0)) + finish(16); + if (!(reg.ioctls & (1ULL << 3)) || !(api.features & (1ULL << 16))) + finish(17); /* COPY and MOVE */ + struct wire data = { (u32)map, pages }; + struct iovec32 iov = { &data, sizeof(data) }; + struct rights32 rights = { sizeof(rights), 1, 1, fd }; + struct message32 msg = { 0, 0, &iov, 1, &rights, sizeof(rights), 0 }; + if (call(296, sock, (long)&msg, 0, 0, 0, 0) != sizeof(data)) + finish(18); + volatile unsigned char *p = (void *)map; + /* Each first access waits for the native pager's COPY. */ + for (u32 leaf = 0; leaf < 3; leaf++) + for (u32 i = 0; i < pages; i++) + if (p[leaf * pages + i] != (unsigned char)(0x51 + leaf)) + finish(19); + if (!shared) { + long target = call(192, 0, 3 * pages, 3, 0x22, -1, 0); + if ((unsigned long)target >= 0xfffff001UL) + finish(21); + struct registration dst = { (u32)target, 3 * pages, 1, 0 }; + if (call(54, fd, 0xc020aa00U, (long)&dst, 0, 0, 0) || + !(dst.ioctls & (1ULL << 5))) + finish(22); + struct movement m = { 1ULL << 32, (u32)map + 2 * pages, pages, + 0, 0 }; + if (call(54, fd, 0xc028aa05U, (long)&m, 0, 0, 0) != -22) + finish(23); + m.dst = (u32)target + pages; + if (pages > 4096) { + m.len = 4096; + if (call(54, fd, 0xc028aa05U, (long)&m, 0, 0, 0) != -22) + finish(24); + m.len = pages; + } + if (call(54, fd, 0xc028aa05U, (long)&m, 0, 0, 0) || + m.moved != pages) + finish(25); + volatile unsigned char *q = (void *)target; + for (u32 i = 0; i < pages; i++) + if (q[pages + i] != 0x53 || p[i] != 0x51 || + p[pages + i] != 0x52) + finish(26); + if (call(91, target, 3 * pages, 0, 0, 0, 0)) + finish(27); + static const char ok[] = + "ok - AArch32 MOVE, neighbours and native-fallback alignment\n"; + call(4, 1, (long)ok, sizeof(ok) - 1, 0, 0, 0); + } + if (call(91, map, 3 * pages, 0, 0, 0, 0)) + finish(20); + finish(0); +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/compat-vfio-probe.c b/tools/testing/selftests/arm64/user4k/compat-vfio-probe.c new file mode 100644 index 00000000000000..231e4b79966135 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/compat-vfio-probe.c @@ -0,0 +1,250 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Freestanding AArch32 type1 probe; no armhf sysroot or physical DMA. */ +#define __user +#include "../../../../../include/uapi/linux/vfio.h" +#include "../../../../../drivers/vfio/vfio_iommu_type1_test.h" + +typedef unsigned int u32; +typedef unsigned long long u64; +#define IOVA (256UL * 1024 * 1024) +static unsigned int stage; + +static long call(long n, long a, long b, long c, long d, long e, long f) +{ + register long r0 __asm__("r0") = a, r1 __asm__("r1") = b, + r2 __asm__("r2") = c; + register long r3 __asm__("r3") = d, r4 __asm__("r4") = e, + r5 __asm__("r5") = f; + register long r7 __asm__("r7") = n; + + __asm__ volatile("svc 0" + : "+r"(r0) + : "r"(r1), "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r7) + : "memory", "cc"); + return r0; +} + +static u32 number(const char *s) +{ + u32 n = 0; + + while (*s >= '0' && *s <= '9') + n = n * 10 + *s++ - '0'; + return n; +} + +static void clear(void *data, u32 size) +{ + volatile unsigned char *p = data; + + for (u32 i = 0; i < size; i++) + p[i] = 0; +} + +/* Aggregate initialization may use these compiler-runtime entry points. */ +void __aeabi_memclr8(void *data, u32 size) +{ + clear(data, size); +} +void __aeabi_memclr4(void *data, u32 size) +{ + clear(data, size); +} +void __aeabi_memclr(void *data, u32 size) +{ + clear(data, size); +} + +_Noreturn static void finish(int code) +{ + if (code) { + char msg[] = "not ok - AArch32 type1 stage=00\n"; + + msg[29] = '0' + stage / 10; + msg[30] = '0' + stage % 10; + call(4, 1, (long)msg, sizeof(msg) - 1, 0, 0, 0); + } + call(1, code, 0, 0, 0, 0, 0); + for (;;) + ; +} +#define CHECK(x) \ + do { \ + stage++; \ + if (!(x)) \ + finish(stage); \ + } while (0) + +static long ioctl_test(int fd, u32 request, void *arg) +{ + return call(54, fd, request, (long)arg, 0, 0, 0); +} + +static int state(int fd, struct vfio_type1_test_cmd *cmd) +{ + clear(cmd, sizeof(*cmd)); + return ioctl_test(fd, VFIO_TYPE1_TEST_STATE, cmd); +} + +static long map(int fd, u32 address, u32 size) +{ + struct vfio_iommu_type1_dma_map cmd; + + clear(&cmd, sizeof(cmd)); + cmd.argsz = sizeof(cmd); + cmd.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; + cmd.iova = IOVA; + cmd.vaddr = address; + cmd.size = size; + return ioctl_test(fd, VFIO_IOMMU_MAP_DMA, &cmd); +} + +static long unmap(int fd, u32 size) +{ + struct vfio_iommu_type1_dma_unmap cmd; + long ret; + + clear(&cmd, sizeof(cmd)); + cmd.argsz = sizeof(cmd); + cmd.iova = IOVA; + cmd.size = size; + ret = ioctl_test(fd, VFIO_IOMMU_UNMAP_DMA, &cmd); + return ret ? ret : cmd.size == size ? 0 : -1; +} + +static long rw(int fd, unsigned char *bytes, unsigned int flags) +{ + struct vfio_type1_test_cmd cmd; + + clear(&cmd, sizeof(cmd)); + cmd.iova = IOVA; + cmd.buffer = (u32)bytes; + cmd.size = 16; + cmd.flags = flags; + return ioctl_test(fd, VFIO_TYPE1_TEST_RW, &cmd); +} + +_Noreturn void probe(u32 *sp) +{ + u32 argc = *sp++, pages = 0; + char **argv = (void *)sp; + struct vfio_type1_test_cmd cmd; + struct vfio_iommu_type1_dirty_bitmap dirty; + struct vfio_iommu_type1_info info; + unsigned char bytes[16]; + long fd, file, data, pid; + int status = -1; + u32 native, length; + + sp += argc + 1; + while (*sp) + sp++; + sp++; + while (sp[0]) { + if (sp[0] == 6) + pages = sp[1]; + sp += 2; + } + CHECK(argc == 3 && pages == number(argv[2])); + native = number(argv[1]); + length = native + 4096; + fd = call(5, (long)"/dev/vfio-test", 2, 0, 0, 0, 0); + file = call(385, (long)"compat-vfio", 0, 0, 0, 0, 0); + CHECK(fd >= 0 && file >= 0 && !call(93, file, 2 * native, 0, 0, 0, 0)); + data = call(192, 0, 2 * native, 3, 1, file, 0); + CHECK((u32)data < 0xfffff001U); + CHECK(call(125, data + 4096, 4096, 1, 0, 0, 0) == + (pages == 4096 ? 0 : -22)); + CHECK(!call(125, data, 2 * native, 3, 0, 0, 0)); + for (u32 i = 0; i < 2 * native; i++) + ((volatile unsigned char *)data)[i] = 0x53; + clear(&info, sizeof(info)); + info.argsz = sizeof(info); + CHECK(!ioctl_test(fd, VFIO_IOMMU_GET_INFO, &info) && + (info.iova_pgsizes & 4096)); + CHECK(!map(fd, data, 2 * native)); + CHECK(!state(fd, &cmd) && !cmd.locked_bytes); + CHECK(!rw(fd, bytes, 0) && bytes[0] == 0x53 && bytes[15] == 0x53); + clear(&dirty, sizeof(dirty)); + dirty.argsz = sizeof(dirty); + dirty.flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_START; + CHECK(!ioctl_test(fd, VFIO_IOMMU_DIRTY_PAGES, &dirty)); + clear(&cmd, sizeof(cmd)); + cmd.iova = IOVA + 19; + CHECK(!ioctl_test(fd, VFIO_TYPE1_TEST_PIN, &cmd)); + CHECK(!state(fd, &cmd) && cmd.locked_bytes == native && + cmd.external_refs == 1); + /* Compat pointer padding must not become part of the native pointer. */ + _Alignas(struct vfio_iommu_type1_dirty_bitmap_get) unsigned char + buffer[sizeof(dirty) + + sizeof(struct vfio_iommu_type1_dirty_bitmap_get)]; + struct vfio_iommu_type1_dirty_bitmap *header = (void *)buffer; + struct vfio_iommu_type1_dirty_bitmap_get *range = (void *)header->data; + u64 bits = 0; + + clear(buffer, sizeof(buffer)); + header->argsz = sizeof(buffer); + header->flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; + range->iova = IOVA; + range->size = 2 * native; + range->bitmap.pgsize = 4096; + range->bitmap.size = sizeof(bits); + range->bitmap.data = &bits; + CHECK(!ioctl_test(fd, VFIO_IOMMU_DIRTY_PAGES, buffer) && + bits == ((1ULL << (native >> 12)) - 1)); + _Static_assert( + sizeof(range->bitmap.data) == 4 && sizeof(range->bitmap) == 24, + "AArch32 bitmap has a 32-bit pointer and trailing padding"); + ((unsigned char *)&range + ->bitmap)[__builtin_offsetof(struct vfio_bitmap, data) + + sizeof(range->bitmap.data)] = 0xa5; + bits = 0; + CHECK(!ioctl_test(fd, VFIO_IOMMU_DIRTY_PAGES, buffer) && + bits == ((1ULL << (native >> 12)) - 1)); + _Alignas(struct vfio_bitmap) unsigned char + unmap_buffer[sizeof(struct vfio_iommu_type1_dma_unmap) + + sizeof(struct vfio_bitmap)]; + struct vfio_iommu_type1_dma_unmap *remove = (void *)unmap_buffer; + struct vfio_bitmap *remove_bitmap = (void *)remove->data; + + clear(unmap_buffer, sizeof(unmap_buffer)); + remove->argsz = sizeof(unmap_buffer); + remove->flags = VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; + remove->iova = IOVA; + remove->size = 2 * native; + remove_bitmap->pgsize = 4096; + remove_bitmap->size = sizeof(bits); + remove_bitmap->data = &bits; + ((unsigned char *) + remove_bitmap)[__builtin_offsetof(struct vfio_bitmap, data) + + sizeof(remove_bitmap->data)] = 0x5a; + bits = 0; + CHECK(!ioctl_test(fd, VFIO_IOMMU_UNMAP_DMA, unmap_buffer) && + remove->size == 2 * native && + bits == ((1ULL << (native >> 12)) - 1)); + CHECK(!state(fd, &cmd) && !cmd.locked_bytes && !cmd.external_refs && + cmd.notifications == 1); + clear(&cmd, sizeof(cmd)); + cmd.size = 4096; + CHECK(!ioctl_test(fd, VFIO_TYPE1_TEST_DOMAIN_ADD, &cmd)); + pid = call(2, 0, 0, 0, 0, 0, 0); + if (!pid) + finish(map(fd, data + 4096, length) ? 99 : 0); + CHECK(pid > 0 && call(114, pid, (long)&status, 0, 0, 0, 0) == pid && + !status); + CHECK(!state(fd, &cmd) && !cmd.owner_users && + cmd.locked_bytes == 2 * native && cmd.mapped_bytes[0] == length); + CHECK(rw(fd, bytes, 0) == -14); + CHECK(!rw(fd, bytes, VFIO_TYPE1_TEST_WORKER) && bytes[0] == 0x53 && + bytes[15] == 0x53); + CHECK(!unmap(fd, length)); + CHECK(!state(fd, &cmd) && !cmd.locked_bytes && !cmd.mapped_bytes[0]); + CHECK(!call(91, data, 2 * native, 0, 0, 0, 0)); + CHECK(!call(6, file, 0, 0, 0, 0, 0) && !call(6, fd, 0, 0, 0, 0, 0)); + static const char ok[] = + "ok - AArch32 type1 real ioctls, source exit and fallback alignment\n"; + + call(4, 1, (long)ok, sizeof(ok) - 1, 0, 0, 0); + finish(0); +} +__asm__(".text\n.global _start\n.type _start,%function\n_start:\nmov r0,sp\nbl probe\n"); diff --git a/tools/testing/selftests/arm64/user4k/context-stress-init.c b/tools/testing/selftests/arm64/user4k/context-stress-init.c new file mode 100644 index 00000000000000..869662502c15f5 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/context-stress-init.c @@ -0,0 +1,275 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* PID-1 mixed-granule EL0 stress runner for disposable VMs. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include + +#define DATA_ADDRESS 0x200000000UL +#define CODE_ADDRESS 0x202000000UL +#define CONTEXTS 70000 +#define SHORT_CONTEXTS 2048 + +static sigjmp_buf fault_return; +static volatile uintptr_t expected_fault; +static char generation_path[256]; + +static void fault(int sig, siginfo_t *info, void *context) +{ + (void)context; + if (sig != SIGSEGV || !expected_fault || + (uintptr_t)info->si_addr != expected_fault || + info->si_code != SEGV_ACCERR) + _exit(90); + expected_fault = 0; + siglongjmp(fault_return, 1); +} + +static bool rejects_write(uintptr_t address) +{ + expected_fault = address; + if (!sigsetjmp(fault_return, 1)) { + *(volatile unsigned char *)expected_fault = 1; + expected_fault = 0; + return false; + } + return true; +} + +static int migrate(unsigned int cpu) +{ + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + return sched_setaffinity(0, sizeof(set), &set); +} + +static unsigned char pattern(unsigned long serial, unsigned long offset) +{ + serial ^= (serial >> 8) ^ (serial >> 16); + return (serial * 37 + offset * 17 + (offset >> 8)) & 255; +} + +static bool validate(unsigned char *p, unsigned long serial, unsigned long len) +{ + for (unsigned long i = 0; i < len; i++) + if (p[i] != pattern(serial, i)) + return false; + return true; +} + +static int worker(unsigned long serial, unsigned long ps, int control) +{ + struct sigaction action = { .sa_sigaction = fault, + .sa_flags = SA_SIGINFO }; + unsigned char *data; + uint32_t *code; + unsigned int cpus = sysconf(_SC_NPROCESSORS_ONLN); + struct stat self, init; + unsigned long iterations = 0; + + if (getauxval(AT_PAGESZ) != ps || getppid() != 1 || + stat("/proc/self/exe", &self) || stat("/proc/1/exe", &init) || + self.st_dev != init.st_dev || self.st_ino != init.st_ino || + prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL) != 0) + return 91; + if (cpus < 2 || migrate(serial % cpus) || + sigaction(SIGSEGV, &action, NULL)) + return 92; + data = mmap((void *)DATA_ADDRESS, 4 * ps, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + code = mmap((void *)CODE_ADDRESS, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + if (data == MAP_FAILED || code == MAP_FAILED || + mprotect(data, 3 * ps, PROT_READ | PROT_WRITE)) + return 93; + for (unsigned long i = 0; i < 3 * ps; i++) + data[i] = pattern(serial, i); + if (!rejects_write(DATA_ADDRESS + 3 * ps)) + return 94; + if (mprotect(data + ps, ps, PROT_READ)) + return 95; + if (!rejects_write(DATA_ADDRESS + ps)) + return 96; + do { + unsigned int expected = (serial + iterations * 13) & 65535; + int (*fn)(void) = (void *)code; + char stop; + ssize_t got; + + if (mprotect(code, ps, PROT_READ | PROT_WRITE)) + return 97; + code[0] = 0x52800000 | (expected << 5); /* movz w0, #expected */ + code[1] = 0xd65f03c0; /* ret */ + __builtin___clear_cache((char *)code, (char *)(code + 2)); + if (mprotect(code, ps, PROT_READ | PROT_EXEC) || + migrate((serial + iterations + 1) % cpus)) + return 98; + sched_yield(); + if (fn() != (int)expected || !validate(data, serial, 3 * ps)) + return 99; + if (!(iterations % 64) && + (!rejects_write(DATA_ADDRESS + 3 * ps) || + !rejects_write(DATA_ADDRESS + ps))) + return 101; + iterations++; + if (control < 0) + break; + got = read(control, &stop, 1); + if (got == 1) + break; + if (got != -1 || (errno != EAGAIN && errno != EINTR)) + return 100; + } while (true); + if (control >= 0) + printf("# persistent %luK worker: %lu migration/JIT/data cycles\n", + ps / 1024, iterations); + return 0; +} + +static pid_t spawn(unsigned long serial, unsigned long ps, int control, + int close_fd) +{ + pid_t pid = fork(); + + if (!pid) { + char seq[32], size[32], fd[32]; + + if (close_fd >= 0) + close(close_fd); + if (control >= 0 && fcntl(control, F_SETFD, 0)) + _exit(110); + snprintf(seq, sizeof(seq), "%lu", serial); + snprintf(size, sizeof(size), "%lu", ps); + snprintf(fd, sizeof(fd), "%d", control); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, ps, 0UL, 0UL, 0UL)) + _exit(111); + execl("/init", "/init", "--worker", seq, size, fd, NULL); + _exit(112); + } + return pid; +} + +static bool reap(pid_t pid) +{ + int status = 0; + pid_t got; + + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + if (got == pid && WIFEXITED(status) && !WEXITSTATUS(status)) + return true; + printf("not ok - worker pid=%d wait=%d status=%#x\n", pid, got, status); + return false; +} + +static unsigned long long generation(void) +{ + unsigned long long value = 0; + FILE *f = fopen(generation_path, "re"); + + if (!f || fscanf(f, "%llu", &value) != 1) { + perror("ASID generation"); + return 0; + } + fclose(f); + return value; +} + +int main(int argc, char **argv) +{ + unsigned long long before, after; + unsigned char *data; + pid_t persistent[3]; + int commands[3], pipes[2]; + unsigned long native = getauxval(AT_PAGESZ); + unsigned long granules[] = { native, 4096, 16384 }; + unsigned int nr = native > 16384 ? 3 : 2; + glob_t paths; + bool ok = true; + unsigned int contexts = CONTEXTS; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--worker")) + return worker(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), atoi(argv[4])); + if (argc == 2 && !strcmp(argv[1], "--short")) + contexts = SHORT_CONTEXTS; + if (getpid() != 1 || (argc != 1 && contexts != SHORT_CONTEXTS)) { + fprintf(stderr, "Run only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + glob("/sys/module/*/parameters/user4k_asid_generation", 0, NULL, + &paths) || + paths.gl_pathc != 1) { + perror("fixture setup"); + return 2; + } + snprintf(generation_path, sizeof(generation_path), "%s", + paths.gl_pathv[0]); + globfree(&paths); + data = mmap((void *)DATA_ADDRESS, 4 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + if (data == MAP_FAILED) + return 3; + for (unsigned long i = 0; i < 4 * native; i++) + data[i] = pattern(0, i); + before = generation(); + printf("# mixed context stress: %u fork/exec workers, initial ASID generation=%llu\n", + contexts, before); + for (unsigned int i = 0; i < nr; i++) { + if (pipe2(pipes, O_NONBLOCK | O_CLOEXEC)) + return 4; + persistent[i] = spawn(CONTEXTS + i + 1, granules[i], pipes[0], + pipes[1]); + if (persistent[i] < 0) + return 5; + close(pipes[0]); + commands[i] = pipes[1]; + } + for (unsigned long i = 1; i <= contexts; i++) { + pid_t pid = spawn(i, granules[i % nr], -1, -1); + + if (pid < 0 || !reap(pid) || !validate(data, 0, 4 * native)) { + printf("not ok - churn/parent isolation at worker %lu\n", + i); + ok = false; + break; + } + if (!(i % 128)) + printf("# workers=%lu ASID generation=%llu\n", i, + generation()); + } + after = generation(); + for (unsigned int i = 0; i < nr; i++) { + ok &= write(commands[i], "x", 1) == 1; + close(commands[i]); + ok &= reap(persistent[i]); + } + ok &= before && after >= before + 2 * (1ULL << 16); + printf("%s - mixed context stress: generation=%llu->%llu, measured rollovers=%llu\n", + ok ? "ok" : "not ok", before, after, (after - before) >> 16); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/core-fixture-init.c b/tools/testing/selftests/arm64/user4k/core-fixture-init.c new file mode 100644 index 00000000000000..4caadf98296a65 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/core-fixture-init.c @@ -0,0 +1,447 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Core-dump regression runner: PID 1 in a disposable VM only. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +#define ANON 0x100000000UL +#define FILEMAP 0x102000000UL +#define SPARSE 0x104000000UL +#define EXCLUDED 0x106000000UL +#define SWAPPED 0x108000000UL +#define SWAPPED_SIZE (1UL << 20) +#define SPARSE_SIZE (8UL << 20) + +static void die(const char *what) +{ + perror(what); + _exit(120); +} + +static void write_text(const char *path, const char *text) +{ + int fd = open(path, O_WRONLY); + size_t len = strlen(text); + + if (fd < 0 || write(fd, text, len) != (ssize_t)len) + die(path); + close(fd); +} + +static void *map_at(unsigned long address, size_t len, int prot, int fd, + off_t off) +{ + void *p = mmap((void *)address, len, prot, + MAP_PRIVATE | MAP_FIXED_NOREPLACE | + (fd < 0 ? MAP_ANONYMOUS : 0), + fd, off); + + if (p == MAP_FAILED) + die("mmap"); + return p; +} + +static void crash(bool headers, int bounded) +{ + unsigned long ps = getauxval(AT_PAGESZ); + struct rlimit limit = { 64UL << 20, 64UL << 20 }; + struct stat self, init; + unsigned char *p, *data; + int fd; + + if (getppid() != 1 || stat("/proc/self/exe", &self) || + stat("/proc/1/exe", &init) || self.st_dev != init.st_dev || + self.st_ino != init.st_ino) + _exit(121); + if (bounded) + limit.rlim_cur = limit.rlim_max = ps - (bounded < 0); + if (setrlimit(RLIMIT_CORE, &limit)) + die("rlimit"); + write_text("/proc/self/coredump_filter", headers ? "0x10" : "0x1f"); + p = map_at(ANON, 4 * ps, PROT_READ | PROT_WRITE, -1, 0); + for (unsigned int i = 0; i < 4; i++) + memset(p + i * ps, 0x31 + i, ps); + if (mprotect(p + ps, ps, PROT_NONE) || + mprotect(p + 2 * ps, ps, PROT_READ)) + die("mprotect"); + p = map_at(SPARSE, SPARSE_SIZE, PROT_READ | PROT_WRITE, -1, 0); + memset(p, 0x65, ps); + /* Also exercise an existing read-only zero PTE. */ + if (*(volatile unsigned char *)(p + ps) != 0) + _exit(122); + p = map_at(EXCLUDED, ps, PROT_READ | PROT_WRITE, -1, 0); + memset(p, 0xa7, ps); + if (madvise(p, ps, MADV_DONTDUMP)) + die("dontdump"); + fd = open("/core-input", O_CREAT | O_TRUNC | O_RDWR, 0700); + data = malloc(19 * ps); + if (fd < 0 || !data) + die("file setup"); + for (unsigned int i = 0; i < 19; i++) + memset(data + i * ps, 0x81 + i, ps); + memcpy(data, ELFMAG, SELFMAG); + if (write(fd, data, 19 * ps) != (ssize_t)(19 * ps)) + die("file data"); + /* Offset one user page: three mappings select non-native quarters. */ + map_at(FILEMAP, ps, PROT_READ, fd, headers ? 0 : ps); + map_at(FILEMAP + 2 * ps, 2 * ps, PROT_READ, fd, 2 * ps); + map_at(FILEMAP + 8 * ps, ps, PROT_READ, fd, 17 * ps); + close(fd); + fd = open("/core-library", O_CREAT | O_TRUNC | O_RDWR, 0600); + if (fd < 0 || write(fd, data, ps) != (ssize_t)ps) + die("non-executable ELF file"); + map_at(FILEMAP + 12 * ps, ps, PROT_READ, fd, 0); + close(fd); + free(data); + if (!headers && !bounded) { + unsigned int missing = 0; + uint64_t entry; + int pm = open("/proc/self/pagemap", O_RDONLY); + + p = map_at(SWAPPED, SWAPPED_SIZE, PROT_READ | PROT_WRITE, -1, + 0); + memset(p, 0xc4, SWAPPED_SIZE); + for (unsigned int tries = 0; tries < 200; tries++) { + if (madvise(p, SWAPPED_SIZE, MADV_COLD) || + madvise(p, SWAPPED_SIZE, MADV_PAGEOUT)) + die("pageout"); + missing = 0; + for (unsigned long i = 0; i < SWAPPED_SIZE / ps; i++) { + if (pread(pm, &entry, sizeof(entry), + (SWAPPED / ps + i) * 8) != 8) + die("pagemap"); + missing += !(entry & (1ULL << 62)); + } + if (!missing) + break; + usleep(10000); + } + close(pm); + if (missing) + _exit(124); + printf("# %luK dump precondition: all %lu leaves swapped\n", + ps / 1024, SWAPPED_SIZE / ps); + } + /* The file VMAs remain unfaulted, so dumping must retrieve file data. */ + raise(SIGABRT); + _exit(123); +} + +static Elf64_Phdr *find_load(unsigned char *data, Elf64_Ehdr *elf, + unsigned long addr) +{ + Elf64_Phdr *ph = (void *)(data + elf->e_phoff); + + for (unsigned int i = 0; i < elf->e_phnum; i++) + if (ph[i].p_type == PT_LOAD && addr >= ph[i].p_vaddr && + addr - ph[i].p_vaddr < ph[i].p_memsz) + return ph + i; + return NULL; +} + +static bool bytes_at(unsigned char *data, Elf64_Ehdr *elf, size_t len, + unsigned long addr, size_t count, unsigned char expected) +{ + Elf64_Phdr *p = find_load(data, elf, addr); + unsigned long off; + + if (!p || addr - p->p_vaddr + count > p->p_filesz) + return false; + off = p->p_offset + addr - p->p_vaddr; + if (off > len || count > len - off) + return false; + for (size_t i = 0; i < count; i++) + if (data[off + i] != expected) + return false; + return true; +} + +static void inspect(pid_t pid, unsigned long ps, bool headers) +{ + char path[64]; + struct stat st; + unsigned char *data; + Elf64_Ehdr *elf; + Elf64_Phdr *ph, *p; + bool geometry = true, aux = false; + unsigned int file = 0; + unsigned long vdso = 0; + int fd; + + snprintf(path, sizeof(path), "/core.%d", pid); + fd = open(path, O_RDONLY); + if (fd < 0 || fstat(fd, &st) || st.st_size < (off_t)sizeof(*elf)) + ksft_exit_fail_msg("missing/short core %s\n", path); + data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (data == MAP_FAILED) + die("core read"); + elf = (void *)data; + if (memcmp(elf->e_ident, ELFMAG, SELFMAG) || elf->e_type != ET_CORE || + elf->e_phentsize != sizeof(*ph) || + elf->e_phoff > (uint64_t)st.st_size || + elf->e_phnum * sizeof(*ph) > (uint64_t)st.st_size - elf->e_phoff) + ksft_exit_fail_msg("invalid core ELF\n"); + ph = (void *)(data + elf->e_phoff); + for (unsigned int i = 0; i < elf->e_phnum; i++) { + if (ph[i].p_type == PT_LOAD) { + geometry &= ph[i].p_align == ps && + ph[i].p_filesz <= ph[i].p_memsz && + ph[i].p_offset % ps == ph[i].p_vaddr % ps; + } else if (ph[i].p_type == PT_NOTE) { + size_t at = ph[i].p_offset, end = at + ph[i].p_filesz; + + if (end > (size_t)st.st_size || end < at) + ksft_exit_fail_msg("invalid notes\n"); + while (at + sizeof(Elf64_Nhdr) <= end) { + Elf64_Nhdr *n = (void *)(data + at); + size_t desc = at + sizeof(*n) + + ((n->n_namesz + 3UL) & ~3UL); + uint64_t *words = (void *)(data + desc); + + if (desc > end || n->n_descsz > end - desc) + ksft_exit_fail_msg( + "invalid note length\n"); + if (n->n_type == NT_AUXV) { + for (size_t j = 0; + j + 1 < n->n_descsz / 8; j += 2) { + if (words[j] == AT_PAGESZ) + aux = words[j + 1] == + ps; + else if (words[j] == + AT_SYSINFO_EHDR) + vdso = words[j + 1]; + } + } + if (n->n_type == NT_FILE && n->n_descsz >= 16 && + words[0] <= (n->n_descsz / 8 - 2) / 3 && + words[1] == ps) { + for (size_t j = 0; j < words[0]; j++) { + unsigned long addr = + words[2 + 3 * j]; + unsigned long offset = + words[4 + 3 * j]; + + if (addr == FILEMAP && + offset == (headers ? 0UL : + 1UL)) + file |= 1; + if (addr == FILEMAP + 2 * ps && + offset == 2) + file |= 2; + if (addr == FILEMAP + 8 * ps && + offset == 17) + file |= 4; + } + } + at = desc + ((n->n_descsz + 3UL) & ~3UL); + } + } + } + ksft_test_result(geometry, "%luK core PT_LOAD granule/alignment%s\n", + ps / 1024, headers ? " headers-only" : ""); + ksft_test_result(aux && file == 7, + "%luK core AUXV and NT_FILE offsets%s\n", ps / 1024, + headers ? " headers-only" : ""); + if (headers) { + bool skipped; + + p = find_load(data, elf, FILEMAP + 2 * ps); + skipped = p && !p->p_filesz; + p = find_load(data, elf, FILEMAP + 8 * ps); + ksft_test_result( + skipped && p && !p->p_filesz, + "%luK header filter excludes nonzero quarter/native file offsets\n", + ps / 1024); + p = find_load(data, elf, FILEMAP + 12 * ps); + ksft_test_result( + p && p->p_filesz == ps && + p->p_offset + ps <= (uint64_t)st.st_size && + !memcmp(data + p->p_offset, ELFMAG, SELFMAG), + "%luK header filter recognizes non-executable ELF file\n", + ps / 1024); + p = find_load(data, elf, FILEMAP); + ksft_test_result( + p && p->p_filesz == ps && + p->p_offset + ps <= (uint64_t)st.st_size && + !memcmp(data + p->p_offset, ELFMAG, SELFMAG), + "%luK executable file header limited to one user page\n", + ps / 1024); + } else { + bool anon = true, files = true; + + p = find_load(data, elf, vdso); + ksft_test_result(p && p->p_filesz >= SELFMAG && + p->p_offset + p->p_filesz <= + (uint64_t)st.st_size && + !memcmp(data + p->p_offset, ELFMAG, + SELFMAG), + "%luK core vDSO ELF bytes\n", ps / 1024); + ksft_test_result(bytes_at(data, elf, st.st_size, SWAPPED, + SWAPPED_SIZE, 0xc4), + "%luK core swapped anonymous bytes\n", + ps / 1024); + + for (unsigned int i = 0; i < 4; i++) + anon &= bytes_at(data, elf, st.st_size, ANON + i * ps, + ps, 0x31 + i); + for (unsigned int i = 1; i < 4; i++) + files &= bytes_at(data, elf, st.st_size, + FILEMAP + (i == 1 ? 0 : i * ps), ps, + 0x81 + i); + ksft_test_result( + anon, "%luK core anonymous and PROT_NONE/READ bytes\n", + ps / 1024); + files &= bytes_at(data, elf, st.st_size, FILEMAP + 8 * ps, ps, + 0x81 + 17); + ksft_test_result(files, "%luK core unfaulted file quarters\n", + ps / 1024); + ksft_test_result( + bytes_at(data, elf, st.st_size, SPARSE, ps, 0x65) && + bytes_at(data, elf, st.st_size, SPARSE + ps, + SPARSE_SIZE - ps, 0), + "%luK core sparse and zero-page bytes\n", ps / 1024); + ksft_test_result( + st.st_blocks * 512 < st.st_size / 2, + "%luK core keeps untouched anonymous range sparse\n", + ps / 1024); + p = find_load(data, elf, EXCLUDED); + ksft_test_result(p && !p->p_filesz, + "%luK core DONTDUMP excludes data\n", + ps / 1024); + } + munmap(data, st.st_size); + close(fd); + unlink(path); +} + +int main(int argc, char **argv) +{ + unsigned long native_size = getauxval(AT_PAGESZ); + unsigned long sizes[] = { native_size, 4096, 16384 }; + unsigned int modes = native_size == 65536 ? 3 : 2; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 2 && !strcmp(argv[1], "--crash")) + crash(false, 0); + if (argc == 2 && !strcmp(argv[1], "--headers")) + crash(true, 0); + if (argc == 2 && !strcmp(argv[1], "--limit")) + crash(false, 1); + if (argc == 2 && !strcmp(argv[1], "--below-limit")) + crash(false, -1); + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Use only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) && errno != EBUSY) + die("proc"); + if (mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + swapon("/dev/vda", 0)) + die("VM swap setup"); + write_text("/proc/sys/kernel/core_pattern", "/core.%p"); + ksft_print_header(); + ksft_set_plan(16 * modes); + for (unsigned int mode = 0; mode < modes; mode++) { + unsigned long ps = sizes[mode]; + for (unsigned int headers = 0; headers < 4; headers++) { + pid_t pid = fork(); + int status = 0; + + if (!pid) { + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + ps == native_size ? 0UL : ps, 0UL, + 0UL, 0UL)) + die("exec size"); + execl("/init", "/init", + headers == 3 ? "--below-limit" : + headers == 2 ? "--limit" : + headers ? "--headers" : + "--crash", + NULL); + die("exec"); + } + if (pid < 0 || waitpid(pid, &status, 0) != pid || + !WIFSIGNALED(status) || + WTERMSIG(status) != SIGABRT || + (headers < 3 && !WCOREDUMP(status))) + ksft_exit_fail_msg( + "child/core failure status=%#x\n", + status); + if (headers >= 2) { + char path[64]; + struct stat st; + bool exists, no_data = false; + + snprintf(path, sizeof(path), "/core.%d", pid); + exists = !stat(path, &st); + if (exists && headers == 2) { + Elf64_Ehdr elf; + Elf64_Phdr ph; + uint64_t data_start = UINT64_MAX; + int fd = open(path, O_RDONLY); + + if (fd < 0 || + pread(fd, &elf, sizeof(elf), 0) != + sizeof(elf)) + die("limited core header"); + for (unsigned int i = 0; + i < elf.e_phnum; i++) { + if (pread(fd, &ph, sizeof(ph), + elf.e_phoff + + i * sizeof(ph)) != + sizeof(ph)) + die("limited core phdr"); + if (ph.p_type == PT_LOAD && + ph.p_offset < data_start) + data_start = + ph.p_offset; + } + close(fd); + /* Sparse note padding is not charged to RLIMIT_CORE. */ + no_data = data_start != UINT64_MAX && + st.st_size <= + (off_t)data_start; + printf("# limited core: bytes=%lld first-data=%llu limit=%lu\n", + (long long)st.st_size, + (unsigned long long)data_start, + ps); + } + ksft_test_result( + headers == 2 ? + exists && st.st_size > 0 && + no_data : + !exists && errno == ENOENT, + "%luK core RLIMIT_CORE %s one user page\n", + ps / 1024, + headers == 2 ? "at" : "below"); + unlink(path); + } else { + inspect(pid, ps, headers); + } + } + } + ksft_print_cnts(); + printf("%s - core fixture complete\n", + ksft_get_fail_cnt() ? "not ok" : "ok"); + reboot(RB_POWER_OFF); + return ksft_get_fail_cnt() ? 1 : 0; +} diff --git a/tools/testing/selftests/arm64/user4k/cow-pool.c b/tools/testing/selftests/arm64/user4k/cow-pool.c new file mode 100644 index 00000000000000..bb25904d68c04a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/cow-pool.c @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Exercise shared COW backing under a selected 4K userspace ABI. Run through + * the platform's ABI launcher; this test never changes its own page contract. + * Every selected leaf changes one byte, and all other bytes must survive. + * Physical density and reclaim require privileged, platform-specific controls + * and are deliberately not inferred from this unprivileged contents test. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define BYTES (64UL << 20) +#define LEAF 4096UL +#define LEAVES (BYTES / LEAF) +#define WORKERS 8 + +enum mode { DENSE, SPARSE, PHASED, RANDOM, LOCKED, LATE_LOCKED, LAZYFREE }; +struct scenario { + const char *name; + enum mode mode; + unsigned int workers; +}; + +static unsigned char *data; +static size_t order[LEAVES]; +static pthread_barrier_t barrier; +static const struct scenario *current; + +static unsigned char initial(size_t leaf, size_t byte) +{ + return leaf * 17 + byte * 3 + 0x35; +} + +static unsigned char changed(size_t leaf) +{ + return initial(leaf, 0) ^ 0xa5; +} + +static bool verify(unsigned int generation, bool discard) +{ + unsigned int stride = current->mode == SPARSE ? 4 : 1; + size_t leaf, byte; + + for (leaf = 0; leaf < LEAVES; leaf++) { + if (generation && discard && leaf % 4 == 0) + continue; + for (byte = 0; byte < LEAF; byte++) { + unsigned char want = initial(leaf, byte); + + if (generation && byte == 0 && leaf % stride == 0) + want ^= generation == 1 ? 0xa5 : 0x5a; + if (data[leaf * LEAF + byte] != want) { + ksft_print_msg("pid=%d leaf=%zu byte=%zu got=%u want=%u\n", + getpid(), leaf, byte, data[leaf * LEAF + byte], want); + return false; + } + } + } + return true; +} + +static bool wait_ok(pid_t pid) +{ + pid_t result; + int status; + + do { + result = waitpid(pid, &status, 0); + } while (result < 0 && errno == EINTR); + return result > 0 && WIFEXITED(status) && WEXITSTATUS(status) == 0; +} + +static void *writer(void *arg) +{ + unsigned int id = (uintptr_t)arg; + size_t leaf, index; + unsigned int phase; + + pthread_barrier_wait(&barrier); + if (current->mode == PHASED || current->mode == LATE_LOCKED || + current->mode == LAZYFREE) { + for (phase = 0; phase < 4; phase++) { + for (leaf = phase + 4UL * id; leaf < LEAVES; + leaf += 4UL * current->workers) + data[leaf * LEAF] = changed(leaf); + pthread_barrier_wait(&barrier); + if (phase == 0 && id == 0) { + if (current->mode == LATE_LOCKED && mlock(data, BYTES)) + _exit(2); + if (current->mode == LAZYFREE) + for (leaf = 0; leaf < LEAVES; leaf += 4) + if (madvise(data + leaf * LEAF, LEAF, MADV_FREE)) + _exit(2); + } + pthread_barrier_wait(&barrier); + } + } else { + unsigned int stride = current->mode == SPARSE ? 4 : 1; + + for (index = id; index < LEAVES / stride; index += current->workers) { + leaf = current->mode == RANDOM ? order[index] : index * stride; + data[leaf * LEAF] = changed(leaf); + } + } + return NULL; +} + +static bool exercise(const struct scenario *scenario) +{ + pthread_t workers[WORKERS]; + struct timespec start, end; + unsigned int t; + pid_t child; + + current = scenario; + child = fork(); + if (child < 0) + return false; + if (!child) { + if (current->mode == LOCKED && mlock(data, BYTES)) + _exit(2); + if (pthread_barrier_init(&barrier, NULL, current->workers)) + _exit(2); + clock_gettime(CLOCK_MONOTONIC, &start); + for (t = 0; t < current->workers; t++) + if (pthread_create(&workers[t], NULL, writer, (void *)(uintptr_t)t)) + _exit(2); + for (t = 0; t < current->workers; t++) + if (pthread_join(workers[t], NULL)) + _exit(2); + clock_gettime(CLOCK_MONOTONIC, &end); + ksft_print_msg("%s fault/write phase %.6f s\n", current->name, + end.tv_sec - start.tv_sec + (end.tv_nsec - start.tv_nsec) * 1e-9); + _exit(verify(1, current->mode == LAZYFREE) ? 0 : 3); + } + return wait_ok(child) && verify(0, false); +} + +static bool owner_exit(void) +{ + static const struct scenario scenario = { "owner-mm exit", DENSE, 1 }; + int ready[2]; + pid_t child, descendant; + size_t leaf; + char signal; + + current = &scenario; + if (prctl(PR_SET_CHILD_SUBREAPER, 1) || pipe(ready)) + return false; + child = fork(); + if (child < 0) { + close(ready[0]); + close(ready[1]); + return false; + } + if (!child) { + close(ready[1]); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = changed(leaf); + if (!verify(1, false)) + _exit(3); + descendant = fork(); + if (descendant < 0) + _exit(2); + if (descendant) + _exit(0); + /* Root signals only after the original pool-owning mm has exited. */ + if (read(ready[0], &signal, 1) != 1 || signal != 'g') + _exit(2); + close(ready[0]); + if (!verify(1, false)) + _exit(3); + for (leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] ^= 0xff; + _exit(verify(2, false) ? 0 : 3); + } + close(ready[0]); + if (!wait_ok(child)) { + close(ready[1]); + return false; + } + if (write(ready[1], "g", 1) != 1) { + close(ready[1]); + return false; + } + close(ready[1]); + return wait_ok(-1) && verify(0, false); +} + +int main(void) +{ + static const struct scenario scenarios[] = { + { "dense single-thread COW", DENSE, 1 }, + { "dense concurrent COW", DENSE, WORKERS }, + { "single-offset sparse COW", SPARSE, 1 }, + { "phased-offset concurrent COW", PHASED, WORKERS }, + { "random-order concurrent COW", RANDOM, WORKERS }, + { "locked COW", LOCKED, WORKERS }, + { "mlock after first offset", LATE_LOCKED, WORKERS }, + { "MADV_FREE after first offset", LAZYFREE, WORKERS }, + }; + struct rlimit limit; + uint32_t seed = 0x9148140; + size_t leaf, byte, i, j, tmp; + bool can_lock; + + ksft_print_header(); + if (getauxval(AT_PAGESZ) != LEAF) + ksft_exit_skip("run under a selected 4K userspace ABI\n"); + ksft_set_plan(9); + alarm(180); + data = mmap(NULL, BYTES, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (data == MAP_FAILED) + ksft_exit_fail_msg("mmap: %s\n", strerror(errno)); + for (leaf = 0; leaf < LEAVES; leaf++) + for (byte = 0; byte < LEAF; byte++) + data[leaf * LEAF + byte] = initial(leaf, byte); + for (i = 0; i < LEAVES; i++) + order[i] = i; + for (i = LEAVES; i > 1; i--) { + seed = seed * 1664525 + 1013904223; + j = seed % i; + tmp = order[j]; + order[j] = order[i - 1]; + order[i - 1] = tmp; + } + can_lock = !getrlimit(RLIMIT_MEMLOCK, &limit) && limit.rlim_cur >= BYTES; + for (i = 0; i < sizeof(scenarios) / sizeof(scenarios[0]); i++) { + if (!can_lock && (scenarios[i].mode == LOCKED || + scenarios[i].mode == LATE_LOCKED)) { + ksft_test_result_skip("%s needs a 64MiB memlock limit\n", scenarios[i].name); + continue; + } + ksft_test_result(exercise(&scenarios[i]), "%s preserves contents and parent\n", + scenarios[i].name); + } + ksft_test_result(owner_exit(), "descendant retains COW data after owner mm exits\n"); + munmap(data, BYTES); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/cpu-granule-init.c b/tools/testing/selftests/arm64/user4k/cpu-granule-init.c new file mode 100644 index 00000000000000..1e983f91f74c72 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/cpu-granule-init.c @@ -0,0 +1,187 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +static int failed; +static void result(int ok, const char *name) +{ + printf("%s - %s\n", ok ? "ok" : "not ok", name); + failed += !ok; +} +static int pin_cpu(pid_t pid, unsigned int cpu) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + return sched_setaffinity(pid, sizeof(set), &set); +} +static int online_cpu1(void) +{ + char c = '?'; + int fd = open("/sys/devices/system/cpu/cpu1/online", O_RDONLY); + if (fd >= 0) { + if (read(fd, &c, 1) != 1) + c = '?'; + close(fd); + } + return c == '0' ? 0 : c == '1' ? 1 : -1; +} +static int worker(int ready, int command, unsigned long expected) +{ + unsigned char *p; + unsigned long ps = getauxval(AT_PAGESZ); + char cpu; + int ok = getppid() == 1 && ps == expected; + + p = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + return 2; + for (unsigned long i = 0; i < 4 * ps; i++) + p[i] = i ^ (i >> 12); + if (write(ready, "r", 1) != 1 || read(command, &cpu, 1) != 1) + return 3; + ok &= sched_getcpu() == cpu - '0'; + for (unsigned int round = 0; round < 128; round++) { + for (unsigned long i = 0; i < 4 * ps; i++) + if (p[i] != (unsigned char)(i ^ (i >> 12))) + ok = 0; + sched_yield(); + } + munmap(p, 4 * ps); + return ok ? 0 : 4; +} +static int compat_sample(unsigned long expected) +{ + uint64_t data[4] = { 0 }; + int pipefd[2], status; + pid_t child; + ssize_t size; + + if (pipe(pipefd)) + return 0; + child = fork(); + if (!child) { + char fd[32]; + close(pipefd[0]); + snprintf(fd, sizeof(fd), "%d", pipefd[1]); + execl("/compat-aslr", "/compat-aslr", "--sample", fd, NULL); + _exit(2); + } + close(pipefd[1]); + size = read(pipefd[0], data, sizeof(data)); + close(pipefd[0]); + return child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status) && + size == sizeof(data) && data[0] == expected; +} +int main(int argc, char **argv) +{ + if (argc == 5 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10)); + if (getpid() != 1 || argc != 1) { + fprintf(stderr, + "Dedicated two-CPU disposable VM with maxcpus=1 only\n"); + return 2; + } + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGPIPE, SIG_IGN); + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) { + perror("mount"); + failed++; + goto done; + } + unsigned long native = getauxval(AT_PAGESZ); + int has4k = !prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL); + result(!prctl(SET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL), + "native exec default restored"); + char cmdline[4096] = { 0 }; + int fd = open("/proc/cmdline", O_RDONLY); + if (fd < 0 || read(fd, cmdline, sizeof(cmdline) - 1) < 0) { + failed++; + goto done; + } + close(fd); + int reject = has4k && strstr(cmdline, "arm64.user4k_deny_late_cpu=1"); + result(online_cpu1() == 0, "CPU1 starts offline"); + if (online_cpu1() != 0) + goto done; + int ready[2], command[2]; + if (pipe(ready) || pipe(command)) { + failed++; + goto done; + } + pid_t child = fork(); + if (!child) { + char ready_fd[32], command_fd[32], ps[32]; + close(ready[0]); + close(command[1]); + snprintf(ready_fd, sizeof(ready_fd), "%d", ready[1]); + snprintf(command_fd, sizeof(command_fd), "%d", command[0]); + snprintf(ps, sizeof(ps), "%lu", has4k ? 4096UL : native); + if (prctl(SET_EXEC_PAGE_SIZE, has4k ? 4096UL : 0UL, 0UL, 0UL, + 0UL)) + _exit(5); + execl("/init", "/init", "--worker", ready_fd, command_fd, ps, + NULL); + _exit(6); + } + close(ready[1]); + close(command[0]); + if (child < 0) { + failed++; + goto done; + } + char token; + result(read(ready[0], &token, 1) == 1 && token == 'r', + "worker has live mappings before hotplug"); + close(ready[0]); + fd = open("/sys/devices/system/cpu/cpu1/online", O_WRONLY); + errno = 0; + ssize_t written = fd < 0 ? -1 : write(fd, "1", 1); + int saved_errno = errno; + if (fd >= 0) + close(fd); + int online = online_cpu1(); + printf("# has4k=%d expected-reject=%d write=%zd errno=%d online=%d\n", + has4k, reject, written, saved_errno, online); + result(reject ? written < 0 && online == 0 : + written == 1 && online == 1, + "late CPU admission matches advertised user granule"); + unsigned int cpu = online == 1 ? 1 : 0; + result(!pin_cpu(child, cpu), "worker affinity targets admitted CPU"); + token = '0' + cpu; + result(write(command[1], &token, 1) == 1, + "worker resumes after CPU admission"); + close(command[1]); + int status; + result(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status), + "live mapping bytes survive and worker runs on expected CPU"); + result(!pin_cpu(0, cpu), "launcher affinity targets admitted CPU"); + result(compat_sample(has4k ? 4096 : native), + "AArch32 exec retains selected or native-fallback granule"); +done: + printf("CPU GRANULE %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/cpu-granule16-init.c b/tools/testing/selftests/arm64/user4k/cpu-granule16-init.c new file mode 100644 index 00000000000000..ecf682a22f732f --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/cpu-granule16-init.c @@ -0,0 +1,192 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +static int failed; +static void result(int ok, const char *name) +{ + printf("%s - %s\n", ok ? "ok" : "not ok", name); + failed += !ok; +} +static int pin_cpu(pid_t pid, unsigned int cpu) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + return sched_setaffinity(pid, sizeof(set), &set); +} +static int online_cpu1(void) +{ + char c = '?'; + int fd = open("/sys/devices/system/cpu/cpu1/online", O_RDONLY); + if (fd >= 0) { + if (read(fd, &c, 1) != 1) + c = '?'; + close(fd); + } + return c == '0' ? 0 : c == '1' ? 1 : -1; +} +static int worker(int ready, int command, unsigned long expected) +{ + unsigned char *p; + unsigned long ps = getauxval(AT_PAGESZ); + char cpu; + int ok = getppid() == 1 && ps == expected; + + p = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + return 2; + for (unsigned long i = 0; i < 4 * ps; i++) + p[i] = i ^ (i >> 12); + if (write(ready, "r", 1) != 1 || read(command, &cpu, 1) != 1) + return 3; + ok &= sched_getcpu() == cpu - '0'; + for (unsigned int round = 0; round < 128; round++) { + for (unsigned long i = 0; i < 4 * ps; i++) + if (p[i] != (unsigned char)(i ^ (i >> 12))) + ok = 0; + sched_yield(); + } + munmap(p, 4 * ps); + return ok ? 0 : 4; +} +static int compat_sample(unsigned long expected) +{ + uint64_t data[4] = { 0 }; + int pipefd[2], status; + pid_t child; + ssize_t size; + + if (pipe(pipefd)) + return 0; + child = fork(); + if (!child) { + char fd[32]; + close(pipefd[0]); + snprintf(fd, sizeof(fd), "%d", pipefd[1]); + execl("/compat-aslr", "/compat-aslr", "--sample", fd, NULL); + _exit(2); + } + close(pipefd[1]); + size = read(pipefd[0], data, sizeof(data)); + close(pipefd[0]); + return child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status) && + size == sizeof(data) && data[0] == expected; +} +int main(int argc, char **argv) +{ + if (argc == 5 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10)); + if (getpid() != 1 || argc != 1) { + fprintf(stderr, + "Dedicated two-CPU disposable VM with maxcpus=1 only\n"); + return 2; + } + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGPIPE, SIG_IGN); + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) { + perror("mount"); + failed++; + goto done; + } + unsigned long native = getauxval(AT_PAGESZ); + int has4k = !prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL); + int has16k = native > 16384 && + !prctl(SET_EXEC_PAGE_SIZE, 16384UL, 0UL, 0UL, 0UL); + result(!prctl(SET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL), + "native exec default restored"); + char cmdline[4096] = { 0 }; + int fd = open("/proc/cmdline", O_RDONLY); + if (fd < 0 || read(fd, cmdline, sizeof(cmdline) - 1) < 0) { + failed++; + goto done; + } + close(fd); + int reject = has16k && strstr(cmdline, "arm64.user16k_deny_late_cpu=1"); + result(online_cpu1() == 0, "CPU1 starts offline"); + if (online_cpu1() != 0) + goto done; + int ready[2], command[2]; + if (pipe(ready) || pipe(command)) { + failed++; + goto done; + } + pid_t child = fork(); + if (!child) { + char ready_fd[32], command_fd[32], ps[32]; + close(ready[0]); + close(command[1]); + snprintf(ready_fd, sizeof(ready_fd), "%d", ready[1]); + snprintf(command_fd, sizeof(command_fd), "%d", command[0]); + snprintf(ps, sizeof(ps), "%lu", has16k ? 16384UL : native); + if (prctl(SET_EXEC_PAGE_SIZE, has16k ? 16384UL : 0UL, 0UL, 0UL, + 0UL)) + _exit(5); + execl("/init", "/init", "--worker", ready_fd, command_fd, ps, + NULL); + _exit(6); + } + close(ready[1]); + close(command[0]); + if (child < 0) { + failed++; + goto done; + } + char token; + result(read(ready[0], &token, 1) == 1 && token == 'r', + "worker has live mappings before hotplug"); + close(ready[0]); + fd = open("/sys/devices/system/cpu/cpu1/online", O_WRONLY); + errno = 0; + ssize_t written = fd < 0 ? -1 : write(fd, "1", 1); + int saved_errno = errno; + if (fd >= 0) + close(fd); + int online = online_cpu1(); + printf("# has16k=%d expected-reject=%d write=%zd errno=%d online=%d\n", + has16k, reject, written, saved_errno, online); + result(reject ? written < 0 && online == 0 : + written == 1 && online == 1, + "late CPU admission matches advertised user granule"); + unsigned int cpu = online == 1 ? 1 : 0; + result(!pin_cpu(child, cpu), "worker affinity targets admitted CPU"); + token = '0' + cpu; + result(write(command[1], &token, 1) == 1, + "worker resumes after CPU admission"); + close(command[1]); + int status; + result(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status), + "live mapping bytes survive and worker runs on expected CPU"); + result(!pin_cpu(0, cpu), "launcher affinity targets admitted CPU"); + if (has16k) + result(!prctl(SET_EXEC_PAGE_SIZE, 16384UL, 0UL, 0UL, 0UL), + "16K request staged before compat exec"); + result(compat_sample(has4k ? 4096 : native), + "AArch32 exec selects4K or native despite pending16K"); +done: + printf("CPU GRANULE %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/dio-fixture-init.c b/tools/testing/selftests/arm64/user4k/dio-fixture-init.c new file mode 100644 index 00000000000000..36ebec4350c762 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/dio-fixture-init.c @@ -0,0 +1,266 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static bool use_file; +static unsigned char pattern(size_t i) +{ + return (i * 17 + (i >> 12) * 29 + 71) & 255; +} +static void fill(unsigned char *p, size_t len) +{ + for (size_t i = 0; i < len; i++) + p[i] = pattern(i); +} +static bool check(const unsigned char *p, size_t len, size_t off) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != pattern(i + off)) { + ksft_print_msg("mismatch at %zu: %u vs %u\n", i + off, + p[i], pattern(i + off)); + return false; + } + return true; +} +static void report(bool ok, size_t ps, const char *name) +{ + ksft_test_result(ok, "%zuK %s (errno=%d)\n", ps / 1024, name, errno); +} +static bool aio_transfer(int fd, void *p, size_t len, off_t off, bool write) +{ + aio_context_t ctx = 0; + struct iocb cb = { .aio_lio_opcode = write ? IOCB_CMD_PWRITE : + IOCB_CMD_PREAD, + .aio_fildes = fd, + .aio_buf = (uintptr_t)p, + .aio_nbytes = len, + .aio_offset = off }; + struct iocb *ptr = &cb; + struct io_event ev; + struct timespec timeout = { .tv_sec = 10 }; + bool ok; + + if (syscall(__NR_io_setup, 8, &ctx)) + return false; + ok = syscall(__NR_io_submit, ctx, 1, &ptr) == 1; + if (ok) + ok = syscall(__NR_io_getevents, ctx, 1, 1, &ev, &timeout) == + 1 && + ev.res == (long)len && !ev.res2; + ok &= !syscall(__NR_io_destroy, ctx); + return ok; +} +static int exercise(void) +{ + size_t ps = getauxval(AT_PAGESZ), len = 2 * 1024 * 1024; + unsigned char *p = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + unsigned char *q = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + struct iovec vec[4]; + int fd = open(use_file ? "/fs/test" : "/dev/user4k-test", + O_RDWR | O_DIRECT | (use_file ? O_CREAT | O_TRUNC : 0), + 0600), + memfd; + bool ok; + ssize_t ret; + + ksft_print_header(); + ksft_set_plan(10); + ksft_print_msg("backing=%s\n", use_file ? "ext4" : "raw block"); + if ((ps != 4096 && ps != 16384) || p == MAP_FAILED || q == MAP_FAILED || + fd < 0) + ksft_exit_fail_msg("setup failed errno=%d\n", errno); + if (use_file && ftruncate(fd, 32 * 1024 * 1024)) + ksft_exit_fail_msg("ftruncate failed\n"); + fill(p, len); + memset(q, 0, len); + ok = pwrite(fd, p + ps, 3 * ps, 0) == (ssize_t)(3 * ps) && + pread(fd, q + ps, 3 * ps, 0) == (ssize_t)(3 * ps) && + check(q + ps, 3 * ps, ps); + report(ok, ps, "O_DIRECT exact user offsets and bytes"); + + vec[0] = (struct iovec){ p + 512, ps - 512 }; + vec[1] = (struct iovec){ NULL, 0 }; + vec[2] = (struct iovec){ p + 5 * ps, ps + 512 }; + vec[3] = (struct iovec){ p + 11 * ps + 512, 512 }; + ok = pwritev(fd, vec, 4, 65536) == (ssize_t)(2 * ps + 512); + memset(q, 0, 3 * ps); + ok &= pread(fd, q, 2 * ps + 512, 65536) == (ssize_t)(2 * ps + 512); + ok &= check(q, ps - 512, 512) && + check(q + ps - 512, ps + 512, 5 * ps) && + check(q + 2 * ps, 512, 11 * ps + 512); + report(ok, ps, "fragmented vectors and empty segment"); + + ok = pwrite(fd, p, len, 4 * 1024 * 1024) == (ssize_t)len; + memset(q, 0, len); + ok &= pread(fd, q, len, 4 * 1024 * 1024) == (ssize_t)len && + check(q, len, 0); + report(ok, ps, "multi-bio two-megabyte transfer"); + + ok = !mprotect(p + ps, ps, PROT_NONE); + ret = pwrite(fd, p, 2 * ps, 8 * 1024 * 1024); + ok &= use_file ? ret == -1 && errno == EFAULT : ret == (ssize_t)ps; + ok &= !mprotect(p + ps, ps, PROT_READ | PROT_WRITE); + if (use_file) + ok &= pwrite(fd, p, ps, 8 * 1024 * 1024) == (ssize_t)ps; + memset(q, 0, ps); + ok &= pread(fd, q, ps, 8 * 1024 * 1024) == (ssize_t)ps && + check(q, ps, 0); + report(ok, ps, "source fault and subsequent I/O recovery"); + + ok = !mprotect(q + ps, ps, PROT_NONE); + ret = pread(fd, q, 2 * ps, 4 * 1024 * 1024); + ok &= (use_file ? ret == -1 && errno == EFAULT : ret == (ssize_t)ps) && + check(q, ps, 0); + ok &= !mprotect(q + ps, ps, PROT_READ | PROT_WRITE); + report(ok, ps, "partial destination fault"); + + ok = !mprotect(p + ps, ps, PROT_NONE); + /* Only 256 accessible bytes: none can form a 512-byte device block. */ + ret = pwrite(fd, p + ps - 256, 512, 10 * 1024 * 1024); + ok &= ret == -1 && errno == EFAULT; + ok &= !mprotect(p + ps, ps, PROT_READ | PROT_WRITE); + report(ok, ps, "alignment trimming releases all short-fragment pins"); + + memfd = memfd_create("dio-shared", 0); + ok = memfd >= 0 && !ftruncate(memfd, 4 * ps); + unsigned char *shared = ok ? mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, memfd, 0) : + MAP_FAILED; + ok &= shared != MAP_FAILED; + if (ok) { + memset(shared, 0xa7, 4 * ps); + ok = pread(fd, shared + ps, 2 * ps, 4 * 1024 * 1024) == + (ssize_t)(2 * ps) && + !msync(shared, 4 * ps, MS_SYNC) && + pread(memfd, q, 4 * ps, 0) == (ssize_t)(4 * ps); + ok &= check(q + ps, 2 * ps, 0); + for (size_t i = 0; i < ps; i++) + ok &= q[i] == 0xa7 && q[3 * ps + i] == 0xa7; + munmap(shared, 4 * ps); + } + if (memfd >= 0) + close(memfd); + report(ok, ps, "shared-file destination dirtying and adjacent guards"); + + ok = aio_transfer(fd, p + ps, 7 * ps, 12 * 1024 * 1024, true); + memset(q, 0, 8 * ps); + ok &= aio_transfer(fd, q + ps, 7 * ps, 12 * 1024 * 1024, false) && + check(q + ps, 7 * ps, ps); + report(ok, ps, "asynchronous AIO read/write completion"); + + memset(q, 0x96, 8 * ps); + pid_t child = fork(); + int status = 0; + if (!child) { + _exit(aio_transfer(fd, q + ps, 7 * ps, 12 * 1024 * 1024, + false) && + check(q + ps, 7 * ps, ps) ? + 0 : + 1); + } + ok = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + for (size_t i = 0; i < 8 * ps; i++) + ok &= q[i] == 0x96; + report(ok, ps, "writable-pin COW after fork"); + + ok = true; + for (unsigned i = 0; i < 128 && ok; i++) { + ok = !madvise(q + ps, ps, MADV_DONTNEED) && + pread(fd, q + ps, ps, 4 * 1024 * 1024) == (ssize_t)ps && + check(q + ps, ps, 0); + ok &= !mprotect(q + 2 * ps, ps, PROT_NONE); + ok &= pread(fd, q + 2 * ps, ps, 4 * 1024 * 1024) == -1 && + errno == EFAULT; + ok &= !mprotect(q + 2 * ps, ps, PROT_READ | PROT_WRITE); + } + report(ok, ps, "128 reuse and failed-pin cleanup cycles"); + close(fd); + munmap(p, len); + munmap(q, len); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} +int main(int argc, char **argv) +{ + bool ok = true; + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 2 && + (!strcmp(argv[1], "--test") || !strcmp(argv[1], "--file")) && + getppid() == 1) { + use_file = !strcmp(argv[1], "--file"); + return exercise(); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) + return 3; + for (unsigned disk = 0; disk < 2; disk++) { + char path[80], serial[64]; + unsigned maj, min; + snprintf(path, sizeof(path), "/sys/class/block/vd%c/serial", + 'a' + disk); + FILE *dev = fopen(path, "r"); + if (!dev || fscanf(dev, "%63s", serial) != 1) + return 4; + fclose(dev); + snprintf(path, sizeof(path), "/sys/class/block/vd%c/dev", + 'a' + disk); + dev = fopen(path, "r"); + if (!dev || fscanf(dev, "%u:%u", &maj, &min) != 2) + return 4; + fclose(dev); + const char *node = + !strcmp(serial, "au4k-raw") ? "/dev/user4k-test" : + !strcmp(serial, "au4k-ext4") ? "/dev/user4k-fs" : + NULL; + if (!node || mknod(node, S_IFBLK | 0600, makedev(maj, min))) + return 5; + } + if (mount("/dev/user4k-fs", "/fs", "ext4", 0, NULL)) + return 7; + for (unsigned mode = 0; mode < 4; mode++) { + unsigned small = mode & 1; + pid_t pid = fork(); + int status = 0; + if (!pid) { + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + small ? 4096UL : 0UL, 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", mode >= 2 ? "--file" : "--test", + NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + } + ok &= !umount("/fs"); + printf("%s - direct I/O fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/domain-cycle.c b/tools/testing/selftests/arm64/user4k/domain-cycle.c new file mode 100644 index 00000000000000..e67b9f1b22b332 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/domain-cycle.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define LEAF 4096UL +#define NATIVE 16384UL +#define COUNT 256UL +#define STRIDE (2 * LEAF) +#define LENGTH (COUNT * STRIDE) +#define TAIL_SPAN (8UL * 1024 * 1024) +static unsigned char *maps[16]; +static unsigned long leaves = COUNT, stride = STRIDE; +static void fail(const char *s) { perror(s); exit(1); } +static unsigned bucket(uintptr_t p) +{ return ((uint64_t)(p >> 14) * UINT64_C(0x61c8864680b583eb)) >> 60; } +static unsigned char value(unsigned phase, size_t leaf, size_t byte) +{ return (unsigned char)(1 + (phase * 53 + leaf * 7 + byte) % 251); } +static void check(unsigned count) +{ + for (unsigned phase = 0; phase < count; phase++) + for (size_t leaf = 0; leaf < leaves; leaf++) + for (size_t byte = 0; byte < LEAF; byte++) + if (maps[phase][leaf * stride + byte] != value(phase, leaf, byte)) { + fprintf(stderr, "FAIL bytes phase=%u leaf=%zu byte=%zu\n", phase, leaf, byte); exit(1); + } +} +static void ack(void) +{ + char c; ssize_t n; + do { n = read(0, &c, 1); } while (n < 0 && errno == EINTR); + if (n != 1 || c != 'c') fail("controller ack"); +} +#define WORKERS 4 +#define RACE_LEAVES 64UL +static unsigned char *race_maps[WORKERS]; +static atomic_int race_start; +static uint64_t monotonic_ns(void) +{ + struct timespec t; + if (clock_gettime(CLOCK_MONOTONIC, &t)) fail("clock_gettime"); + return (uint64_t)t.tv_sec * 1000000000 + t.tv_nsec; +} +static void race_check(void) +{ + for (unsigned w = 0; w < WORKERS; w++) + for (size_t leaf = 0; leaf < RACE_LEAVES; leaf++) + for (size_t byte = 0; byte < LEAF; byte++) + if (race_maps[w][leaf * STRIDE + byte] != value(w + 32, leaf, byte)) fail("race bytes"); +} +static void *race_worker(void *arg) +{ + unsigned w = (uintptr_t)arg; + int start; + do { start = atomic_load_explicit(&race_start, memory_order_acquire); if (!start) sched_yield(); } while (!start); + if (start != 1) return NULL; + for (size_t leaf = 0; leaf < RACE_LEAVES; leaf++) { + unsigned char *p = race_maps[w] + leaf * STRIDE; + if (mprotect(p, LEAF, PROT_READ | PROT_WRITE)) fail("race mprotect"); + for (size_t byte = 0; byte < LEAF; byte++) p[byte] = value(w + 32, leaf, byte); + /* Test-only pacing keeps allocation active during bounded controller moves. */ + struct timespec delay = {.tv_nsec = 2000000}; + while (nanosleep(&delay, &delay) && errno == EINTR) {} + } + return NULL; +} +static void race_run(void) +{ + pthread_t threads[WORKERS]; + pthread_attr_t attr; + if (pthread_attr_init(&attr) || pthread_attr_setstacksize(&attr, 128 * 1024)) fail("thread attributes"); + unsigned made = 0; + for (; made < WORKERS; made++) { + race_maps[made] = mmap(NULL, RACE_LEAVES * STRIDE, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (race_maps[made] == MAP_FAILED) fail("race mmap"); + int err = pthread_create(&threads[made], &attr, race_worker, (void *)(uintptr_t)made); + if (err) { errno = err; break; } + } + pthread_attr_destroy(&attr); + if (made != WORKERS) { + atomic_store_explicit(&race_start, 2, memory_order_release); + for (unsigned i = 0; i < made; i++) pthread_join(threads[i], NULL); + fail("pthread_create"); + } + printf("RACE pid=%d base0=%lx base1=%lx base2=%lx base3=%lx leaves=%lu stride=%lu\n", getpid(), (unsigned long)race_maps[0], (unsigned long)race_maps[1], (unsigned long)race_maps[2], (unsigned long)race_maps[3], RACE_LEAVES, STRIDE); + ack(); + uint64_t begin = monotonic_ns(); + atomic_store_explicit(&race_start, 1, memory_order_release); + for (unsigned i = 0; i < WORKERS; i++) if (pthread_join(threads[i], NULL)) abort(); + uint64_t end = monotonic_ns(); + race_check(); + printf("RACE_DONE pid=%d begin_ns=%llu end_ns=%llu\n", getpid(), (unsigned long long)begin, (unsigned long long)end); + ack(); +} +int main(int argc, char **argv) +{ + if (argc == 2 && !strcmp(argv[1], "--selftest")) { + unsigned count = 0; + for (uintptr_t p = 0x100000; p < 0x100000 + TAIL_SPAN; p += NATIVE) count += bucket(p) == 0; + if (count < 16 || value(0, 0, 0) == value(1, 0, 0)) return 1; + puts("PASS patterns/hash selection only; no domain allocation coverage"); return 0; + } + int tail = argc == 3 && !strcmp(argv[2], "--tail"); + int race = argc == 3 && !strcmp(argv[2], "--race"); + if ((argc != 2 && !tail && !race) || strcmp(argv[1], "--native-16k") || getauxval(AT_PAGESZ) != LEAF) return 77; + unsigned phases = tail ? 16 : race ? 1 : 3; + unsigned char *reservation = NULL; + alarm(15); setvbuf(stdout, NULL, _IONBF, 0); + printf("READY pid=%d\n", getpid()); ack(); + if (race) race_run(); + if (tail) { + leaves = 1; stride = LEAF; + reservation = mmap(NULL, TAIL_SPAN + NATIVE, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (reservation == MAP_FAILED) fail("tail reserve"); + uintptr_t begin = ((uintptr_t)reservation + NATIVE - 1) & ~(NATIVE - 1); + unsigned found = 0; + for (uintptr_t p = begin; p < begin + TAIL_SPAN && found < phases; p += NATIVE) + if (!bucket(p)) maps[found++] = (void *)p; + if (found != phases) fail("same-shard tail addresses"); + } + for (unsigned phase = 0; phase < phases; phase++) { + check(phase); + if (!tail) { + maps[phase] = mmap(NULL, LENGTH, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (maps[phase] == MAP_FAILED) fail("mmap"); + } + for (size_t leaf = 0; leaf < leaves; leaf++) { + unsigned char *p = maps[phase] + leaf * stride; + if (mprotect(p, LEAF, PROT_READ | PROT_WRITE)) fail("mprotect"); + for (size_t byte = 0; byte < LEAF; byte++) p[byte] = value(phase, leaf, byte); + } + check(phase + 1); + printf("PHASE index=%u pid=%d base=%lx leaves=%lu stride=%lu\n", phase, getpid(), (unsigned long)maps[phase], leaves, stride); + ack(); + } + check(phases); + if (race) { + race_check(); + for (unsigned w = 0; w < WORKERS; w++) if (munmap(race_maps[w], RACE_LEAVES * STRIDE)) fail("race munmap"); + } + if (tail) { + if (munmap(reservation, TAIL_SPAN + NATIVE)) fail("tail munmap"); + } else for (unsigned phase = 0; phase < phases; phase++) if (munmap(maps[phase], LENGTH)) fail("munmap"); + puts("PASS domain allocation phases and all prior bytes; charge/density separately verified"); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/exec-fixture-init.c b/tools/testing/selftests/arm64/user4k/exec-fixture-init.c new file mode 100644 index 00000000000000..8a390d5df3ddcd --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/exec-fixture-init.c @@ -0,0 +1,167 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* PID-1 runner for the experimental one-shot exec ABI in a disposable VM. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include + +static int exec_size(unsigned long size) +{ + return prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, size, 0UL, 0UL, 0UL); +} + +static int pending_size(void) +{ + return prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL); +} + +static int run(const char *path, const char *arg, unsigned long size) +{ + pid_t pid = fork(), got; + int status = 0; + + if (!pid) { + if (size && exec_size(size)) { + perror("set exec page size"); + _exit(125); + } + execl(path, path, arg, NULL); + perror(path); + _exit(126); + } + if (pid < 0) + return -1; + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + printf("# child %s %s: wait=%d status=%#x\n", path, arg ? arg : "", got, + status); + return got == pid && WIFEXITED(status) && !WEXITSTATUS(status) ? 0 : -1; +} + +static int argument_exec(unsigned long size) +{ + pid_t pid = fork(), got; + int status = 0; + + if (!pid) { + char *payload = malloc(140000), expected[16]; + unsigned int i; + + if (!payload || exec_size(size)) + _exit(120); + for (i = 0; i < 139999; i++) + payload[i] = 'a' + i % 23; + payload[139999] = 0; + if (size == 4096) { + execl("/init", "/init", "--too-long", payload, NULL); + if (errno != E2BIG || pending_size() != 4096) + _exit(121); + } + payload[120000] = 0; + if (setenv("ARG_PAYLOAD", payload, 1)) + _exit(122); + snprintf(expected, sizeof(expected), "%lu", + size ? size : 16384UL); + execl("/init", "/init", "--args-check", expected, payload, + NULL); + perror("argument exec"); + _exit(123); + } + if (pid < 0) + return -1; + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + printf("# large arguments from %lu to %lu: status=%#x\n", + getauxval(AT_PAGESZ), size ? size : 16384UL, status); + return got == pid && WIFEXITED(status) && !WEXITSTATUS(status) ? 0 : -1; +} + +int main(int argc, char **argv) +{ + bool ok; + int failures = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--args-check")) { + const char *env = getenv("ARG_PAYLOAD"); + unsigned int i; + + if (getauxval(AT_PAGESZ) != strtoul(argv[2], NULL, 10) || + pending_size() != 0 || strlen(argv[3]) != 120000 || !env || + strlen(env) != 120000) + return 1; + for (i = 0; i < 120000; i++) + if (argv[3][i] != 'a' + i % 23 || env[i] != argv[3][i]) + return 2; + return 0; + } + if (argc == 2 && !strcmp(argv[1], "--args-from-4k")) { + if (getauxval(AT_PAGESZ) != 4096) + return 1; + return argument_exec(4096) || argument_exec(0); + } + if (argc == 2 && !strcmp(argv[1], "--exec-cycle")) { + if (getauxval(AT_PAGESZ) != 4096 || pending_size() != 0) + return 1; + /* A successful exec consumes the request; ordinary exec is native. */ + execl("/init", "/init", "--native-after-4k", NULL); + return 2; + } + if (argc == 2 && !strcmp(argv[1], "--native-after-4k")) + return getauxval(AT_PAGESZ) != 16384 || pending_size() != 0; + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Run only as /init in a disposable VM.\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) && errno != EBUSY) + return 3; + printf("# exec fixture: native AT_PAGESZ=%lu\n", getauxval(AT_PAGESZ)); + failures += run("/page-contract", NULL, 0) != 0; + ok = pending_size() == 0; + errno = 0; + ok &= exec_size(8192) == -1 && errno == EINVAL; + ok &= prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, 4096UL, 1UL, 0UL, 0UL) == -1; + ok &= exec_size(4096) == 0 && pending_size() == 4096; + execl("/does-not-exist", "/does-not-exist", NULL); + ok &= errno == ENOENT && pending_size() == 4096; + ok &= getauxval(AT_PAGESZ) == 16384; + printf("%s - request validation and failed exec preserve live geometry\n", + ok ? "ok" : "not ok"); + failures += !ok; + /* fork inherits the pending request; successful child exec consumes it. */ + failures += run("/page-contract-4k", "--expect-4k", 0) != 0; + ok = pending_size() == 4096 && exec_size(0) == 0 && pending_size() == 0; + printf("%s - child exec leaves parent request intact; reset works\n", + ok ? "ok" : "not ok"); + failures += !ok; + failures += run("/page-contract-dynamic-4k", "--expect-4k", 4096) != 0; + failures += run("/init", "--exec-cycle", 4096) != 0; + failures += run("/proc-memory", NULL, 0) != 0; + failures += run("/proc-memory", "--expect-4k", 4096) != 0; + failures += run("/proc-pages", NULL, 0) != 0; + failures += run("/proc-pages", "--expect-4k", 4096) != 0; + failures += run("/advice", NULL, 0) != 0; + failures += run("/advice", "--expect-4k", 4096) != 0; + failures += run("/memory-interfaces", NULL, 0) != 0; + failures += run("/memory-interfaces", "--expect-4k", 4096) != 0; + failures += run("/mixed-process", NULL, 0) != 0; + failures += run("/mixed-process", NULL, 4096) != 0; + failures += argument_exec(4096) != 0; + failures += run("/init", "--args-from-4k", 4096) != 0; + failures += run("/page-contract", NULL, 0) != 0; + printf("%s - exec fixture complete: failures=%d\n", + failures ? "not ok" : "ok", failures); + reboot(RB_POWER_OFF); + return failures ? 1 : 0; +} diff --git a/tools/testing/selftests/arm64/user4k/exec-policy-init.c b/tools/testing/selftests/arm64/user4k/exec-policy-init.c new file mode 100644 index 00000000000000..02fd3f95522374 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/exec-policy-init.c @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable guest PID1: per-thread one-shot and inherited exec preferences. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define REQUIRE(x) \ + do { \ + if (!(x)) { \ + fprintf(stderr, "FAIL line %d: %s errno=%d\n", \ + __LINE__, #x, errno); \ + exit(1); \ + } \ + } while (0) + +static int policy(int op, unsigned long value) +{ + return prctl(op, value, 0UL, 0UL, 0UL); +} + +static void state(unsigned long pages, unsigned long preferred, int pending) +{ + REQUIRE(getauxval(AT_PAGESZ) == pages); + REQUIRE(policy(PR_AURORA_GET_DEFAULT_PAGE_SIZE, 0) == (int)preferred); + REQUIRE(policy(PR_AURORA_GET_EXEC_PAGE_SIZE, 0) == pending); +} + +static void chain(int stage, unsigned long native, unsigned long preferred) +{ + char s[32], n[32], p[32]; + + snprintf(s, sizeof(s), "%d", stage); + snprintf(n, sizeof(n), "%lu", native); + snprintf(p, sizeof(p), "%lu", preferred); + execl("/init", "/init", "--chain", s, n, p, NULL); + REQUIRE(0); +} + +static void *thread_policy(void *arg) +{ + unsigned long preferred = *(unsigned long *)arg; + + REQUIRE(policy(PR_AURORA_GET_DEFAULT_PAGE_SIZE, 0) == (int)preferred); + REQUIRE(policy(PR_AURORA_GET_EXEC_PAGE_SIZE, 0) == (int)preferred); + REQUIRE(!policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, 0)); + REQUIRE(!policy(PR_AURORA_SET_EXEC_PAGE_SIZE, 0)); + state(getauxval(AT_PAGESZ), 0, 0); + return NULL; +} + +static void *thread_exec(void *arg) +{ + unsigned long *sizes = arg; + + REQUIRE(!policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, sizes[1])); + chain(0, sizes[0], sizes[1]); + return NULL; +} + +static int wait_child(pid_t pid) +{ + int status; + + REQUIRE(pid > 0); + REQUIRE(waitpid(pid, &status, 0) == pid); + return WIFEXITED(status) && WEXITSTATUS(status) == 0; +} + +static int run_preference(unsigned long native, unsigned long preferred) +{ + pid_t pid = fork(); + + if (!pid) { + pthread_t thread; + REQUIRE(!policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, preferred)); + REQUIRE(!policy(PR_AURORA_SET_EXEC_PAGE_SIZE, preferred)); + REQUIRE(!pthread_create(&thread, NULL, thread_policy, + &preferred)); + REQUIRE(!pthread_join(thread, NULL)); + state(native, preferred, preferred); + /* Rejected values and reserved arguments must preserve both policies. */ + REQUIRE(policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, 8192) == -1 && + errno == EINVAL); + REQUIRE(policy(PR_AURORA_SET_EXEC_PAGE_SIZE, 8192) == -1 && + errno == EINVAL); + int ops[] = { PR_AURORA_SET_DEFAULT_PAGE_SIZE, + PR_AURORA_GET_DEFAULT_PAGE_SIZE, + PR_AURORA_SET_EXEC_PAGE_SIZE, + PR_AURORA_GET_EXEC_PAGE_SIZE }; + for (unsigned int i = 0; i < sizeof(ops) / sizeof(ops[0]); + i++) { + REQUIRE(prctl(ops[i], 0UL, 1UL, 0UL, 0UL) == -1 && + errno == EINVAL); + REQUIRE(prctl(ops[i], 0UL, 0UL, 1UL, 0UL) == -1 && + errno == EINVAL); + REQUIRE(prctl(ops[i], 0UL, 0UL, 0UL, 1UL) == -1 && + errno == EINVAL); + } + REQUIRE(policy(PR_AURORA_GET_DEFAULT_PAGE_SIZE, 1) == -1 && + errno == EINVAL); + REQUIRE(policy(PR_AURORA_GET_EXEC_PAGE_SIZE, 1) == -1 && + errno == EINVAL); + execl("/does-not-exist", "/does-not-exist", NULL); + REQUIRE(errno == ENOENT); + state(native, preferred, preferred); + chain(0, native, preferred); + } + return wait_child(pid); +} + +static int compat(unsigned long native, unsigned long preferred, int no4k) +{ + pid_t pid = fork(); + + if (!pid) { + char n[32], p[32], e[32]; + REQUIRE(!policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, preferred)); + REQUIRE(!policy(PR_AURORA_SET_EXEC_PAGE_SIZE, native)); + snprintf(n, sizeof(n), "%lu", native); + snprintf(p, sizeof(p), "%lu", preferred); + snprintf(e, sizeof(e), "%lu", no4k ? native : 4096UL); + execl("/compat-policy", "/compat-policy", n, p, e, NULL); + REQUIRE(0); + } + return wait_child(pid); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + int failed = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--chain")) { + int stage = atoi(argv[2]); + unsigned long n = strtoul(argv[3], NULL, 10); + unsigned long p = strtoul(argv[4], NULL, 10); + + state(stage == 2 || stage == 5 ? n : p, stage < 4 ? p : 0, 0); + if (stage == 5) + return 0; + if (stage == 1) { + REQUIRE(!policy(PR_AURORA_SET_EXEC_PAGE_SIZE, 0)); + state(p, p, n); + } + if (stage == 3) { + REQUIRE(!policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, 0)); + REQUIRE(!policy(PR_AURORA_SET_EXEC_PAGE_SIZE, p)); + } + chain(stage + 1, n, p); + } + if (argc == 4 && !strcmp(argv[1], "--compat-return")) { + unsigned long n = strtoul(argv[2], NULL, 10); + unsigned long p = strtoul(argv[3], NULL, 10); + state(p ? p : n, p, 0); + return 0; + } + REQUIRE(getpid() == 1); + REQUIRE(!mount("proc", "/proc", "proc", 0, NULL)); + state(native, 0, 0); + int no4k = argc == 2 && !strcmp(argv[1], "--no4k"); + unsigned long preferences[] = { 4096, 16384 }; + for (unsigned int i = 0; i < 2; i++) { + unsigned long p = preferences[i]; + if (p >= native || (no4k && p == 4096)) + continue; + int ok = run_preference(native, p); + printf("%s - exec policy native=%lu alternative=%lu\n", + ok ? "ok" : "not ok", native, p); + failed |= !ok; + pid_t pid = fork(); + if (!pid) { + pthread_t thread; + unsigned long sizes[] = { native, p }; + REQUIRE(!pthread_create(&thread, NULL, thread_exec, + sizes)); + for (;;) + pause(); + } + ok = wait_child(pid); + printf("%s - non-leader exec alternative=%lu\n", + ok ? "ok" : "not ok", p); + failed |= !ok; + ok = compat(native, p, no4k); + printf("%s - compat policy alternative=%lu no4k=%d\n", + ok ? "ok" : "not ok", p, no4k); + failed |= !ok; + } + if (no4k) { + int ok = policy(PR_AURORA_SET_DEFAULT_PAGE_SIZE, 4096) == -1 && + errno == EOPNOTSUPP; + ok &= policy(PR_AURORA_SET_EXEC_PAGE_SIZE, 4096) == -1 && + errno == EOPNOTSUPP; + ok &= compat(native, 0, 1); + printf("%s - no4k preserves native compat restrictions\n", + ok ? "ok" : "not ok"); + failed |= !ok; + } + state(native, 0, 0); + printf("EXEC POLICY %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + return failed; +} diff --git a/tools/testing/selftests/arm64/user4k/fex-contract-init.c b/tools/testing/selftests/arm64/user4k/fex-contract-init.c new file mode 100644 index 00000000000000..3273af30eea7f7 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/fex-contract-init.c @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +int main(void) +{ + if (getpid() != 1) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + int failed = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)) { + perror("mount"); + failed = 1; + goto done; + } + pid_t child = fork(); + if (!child) { + setenv("HOME", "/root", 1); + setenv("PATH", "/usr/bin:/bin", 1); + setenv("XDG_RUNTIME_DIR", "/run", 1); + setenv("FEX_ROOTFS", "/", 1); + setenv("LD_PRELOAD", "/usr/lib/libfex-abi.so", 1); + if (prctl(PR_AURORA_SET_DEFAULT_PAGE_SIZE, 4096UL, 0UL, 0UL, + 0UL) || + prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL)) + _exit(3); + execl("/usr/bin/FEXInterpreter", "/usr/bin/FEXInterpreter", + "/x86-contract", NULL); + perror("FEX exec"); + _exit(4); + } + int status = 0; + int ok = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - FEX 4K native-process / x86 workload status=%#x\n", + ok ? "ok" : "not ok", status); + failed |= !ok; +done: + printf("GRANULE FEX RUNTIME %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/fex-native-abi.c b/tools/testing/selftests/arm64/user4k/fex-native-abi.c new file mode 100644 index 00000000000000..86ccb6e389d9de --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/fex-native-abi.c @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +__attribute__((constructor)) static void check_abi(void) +{ + char path[1024]; + ssize_t n = readlink("/proc/self/exe", path, sizeof(path) - 1); + if (n < 0) + return; + path[n] = 0; + if (!strstr(path, "FEXInterpreter")) + return; + unsigned long ps = getauxval(AT_PAGESZ); + dprintf(2, "FEX NATIVE HOST PROCESS PAGE SIZE %lu\n", ps); + if (ps != 4096) + _exit(90); +} diff --git a/tools/testing/selftests/arm64/user4k/fex-x86-contract.c b/tools/testing/selftests/arm64/user4k/fex-x86-contract.c new file mode 100644 index 00000000000000..047a3c9d28ddb6 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/fex-x86-contract.c @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0-only +typedef unsigned long size_t; +static long sc(long n, long a, long b, long c, long d, long e, long f) +{ + register long r10 __asm__("r10") = d, r8 __asm__("r8") = e, + r9 __asm__("r9") = f; + long r; + __asm__ volatile("syscall" + : "=a"(r) + : "a"(n), "D"(a), "S"(b), "d"(c), "r"(r10), "r"(r8), + "r"(r9) + : "rcx", "r11", "memory"); + return r; +} +static void say(const char *s, size_t n) +{ + sc(1, 1, (long)s, n, 0, 0, 0); +} +#define SAY(s) say(s, sizeof(s) - 1) +static void quit(int n) +{ + sc(231, n, 0, 0, 0, 0, 0); + for (;;) + ; +} +static int check(unsigned char *p, size_t size) +{ + for (size_t i = 0; i < size; i++) + if (p[i] != (unsigned char)(i ^ (i >> 12))) + return 0; + return 1; +} +long run(unsigned long *sp) +{ + unsigned long argc = *sp++, pagesz = 0; + sp += argc + 1; + unsigned long *env = sp; + while (*sp) + sp++; + sp++; + while (*sp) { + if (sp[0] == 6) + pagesz = sp[1]; + sp += 2; + } + if (pagesz != 4096) + return 10; + SAY("ok - FEX x86 guest 4K auxv\n"); + unsigned char *p = (void *)sc(9, 0, 12288, 3, 0x22, -1, 0); + if ((long)p < 0) + return 11; + for (size_t i = 0; i < 12288; i++) + p[i] = (unsigned char)(i ^ (i >> 12)); + if (sc(10, (long)p + 4096, 4096, 0, 0, 0, 0)) + return 12; + long child = sc(57, 0, 0, 0, 0, 0, 0); + int status = 0; + if (!child) { + *(volatile unsigned char *)(p + 4096) = 1; + quit(13); + } + if (child < 0 || sc(61, child, (long)&status, 0, 0, 0, 0) != child || + (status & 127) != 11) + return 14; + if (sc(10, (long)p + 4096, 4096, 3, 0, 0, 0) || !check(p, 12288)) + return 15; + SAY("ok - FEX x86 independent 4K protection fault\n"); + child = sc(57, 0, 0, 0, 0, 0, 0); + if (!child) { + p[4096] = 0x71; + quit(0); + } + if (child < 0 || sc(61, child, (long)&status, 0, 0, 0, 0) != child || + status || !check(p, 12288)) + return 16; + SAY("ok - FEX x86 fork COW\n"); + p = (void *)sc(25, (long)p, 12288, 16384, 1, 0, 0); + if ((long)p < 0 || !check(p, 12288)) + return 17; + for (size_t i = 12288; i < 16384; i++) + if (p[i]) + return 18; + SAY("ok - FEX x86 mremap preserve and zero extension\n"); + long fd = sc(319, (long)"fex-shared", 0, 0, 0, 0, 0); + if (fd < 0 || sc(77, fd, 12288, 0, 0, 0, 0)) + return 19; + unsigned char *a = (void *)sc(9, 0, 4096, 3, 1, fd, 4096), + *b = (void *)sc(9, 0, 4096, 3, 1, fd, 4096); + if ((long)a < 0 || (long)b < 0) + return 20; + a[37] = 0x65; + if (b[37] != 0x65) + return 21; + SAY("ok - FEX x86 memfd mapping at 4K file offset\n"); + unsigned char *code = (void *)sc(9, 0, 4096, 3, 0x22, -1, 0); + if ((long)code < 0) + return 22; + code[0] = 0xb8; + code[1] = 42; + code[2] = 0; + code[3] = 0; + code[4] = 0; + code[5] = 0xc3; + if (sc(10, (long)code, 4096, 5, 0, 0, 0) || + ((int (*)(void))code)() != 42) + return 23; + if (sc(10, (long)code, 4096, 3, 0, 0, 0)) + return 24; + code[1] = 99; + if (sc(10, (long)code, 4096, 5, 0, 0, 0) || + ((int (*)(void))code)() != 99) + return 25; + SAY("ok - FEX x86 JIT permission transition and code replacement\n"); + unsigned char random[32]; + if (sc(318, (long)random, 32, 0, 0, 0, 0) != 32) + return 26; + if (argc == 1) { + child = sc(57, 0, 0, 0, 0, 0, 0); + if (!child) { + char *args[] = { "/x86-contract", "--exec-child", 0 }; + sc(59, (long)args[0], (long)args, (long)env, 0, 0, 0); + quit(28); + } + if (child < 0 || + sc(61, child, (long)&status, 0, 0, 0, 0) != child || status) + return 27; + SAY("ok - FEX x86 fork and re-exec with 4K mappings\n"); + } + SAY("FEX X86 4K CONTRACT PASS\n"); + return 0; +} +__asm__(".global _start\n_start:\nmov %rsp,%rdi\nand $-16,%rsp\ncall run\nmov %rax,%rdi\nmov $231,%rax\nsyscall\n"); diff --git a/tools/testing/selftests/arm64/user4k/file-cow-consumers.c b/tools/testing/selftests/arm64/user4k/file-cow-consumers.c new file mode 100644 index 00000000000000..7d2f82500f6374 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/file-cow-consumers.c @@ -0,0 +1,319 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Owned memfd only: file COW, typed remote GUP and shifted mremap. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define LEAF 4096UL +#define NATIVE 16384UL +#define WINDOWS 64UL +#define BYTES (WINDOWS * NATIVE) +#define LEAVES (BYTES / LEAF) +#define PRESENT (UINT64_C(1) << 63) +#define FILE_PAGE (UINT64_C(1) << 61) +#define PFN_MASK ((UINT64_C(1) << 55) - 1) + +struct message { uintptr_t address; uint64_t phase, detail; }; +static volatile sig_atomic_t child_group; +static const char *stage = "startup"; +static bool child_reaped; +static int child_status; +static void report_wait(pid_t pid, int status) +{ + fprintf(stderr, "CHILD_WAIT pid=%ld raw_status=%d exited=%d exit=%d signaled=%d signal=%d\n", + (long)pid, status, WIFEXITED(status), WIFEXITED(status) ? WEXITSTATUS(status) : -1, + WIFSIGNALED(status), WIFSIGNALED(status) ? WTERMSIG(status) : 0); +} + +static void stop(int sig) +{ + if (child_group > 0) kill(-child_group, SIGKILL); + _exit(128 + sig); +} +static bool io_full(int fd, void *buf, size_t n, bool write_it) +{ + char *p = buf; + while (n) { + ssize_t got = write_it ? write(fd, p, n) : read(fd, p, n); + if (got < 0 && errno == EINTR) continue; + if (got <= 0) return false; + p += got; n -= (size_t)got; + } + return true; +} +static unsigned char original(size_t offset) +{ + size_t leaf = offset / LEAF, byte = offset % LEAF; + return (unsigned char)(leaf * 17 + byte * 13 + byte / 256); +} +static bool verify(const unsigned char *data, unsigned kind, size_t gup_a_leaf) +{ + for (size_t at = 0; at < BYTES; at++) { + size_t leaf = at / LEAF, byte = at % LEAF; + unsigned char value = original(at); + if (kind == 1 && leaf % 4 == 3 && byte == 0) value ^= 0x5a; + if (kind == 1 && leaf == gup_a_leaf && byte >= 17 && byte < 49) value = 0xe3; + if (kind == 2 && leaf % 4 == 2 && byte >= 37 && byte < 69) value = 0xc7; + if (data[at] != value) { + fprintf(stderr, "VERIFY_FAIL stage=%s kind=%u offset=0x%zx leaf=%zu byte=%zu expected=%02x actual=%02x\n", + stage, kind, at, leaf, byte, value, data[at]); + return false; + } + } + return true; +} +static bool visible_pfns(void) +{ + FILE *f = fopen("/proc/self/status", "re"); + char line[256]; unsigned long long caps = 0; + if (!f) return false; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "CapEff: %llx", &caps) == 1) break; + fclose(f); return !!(caps & (1ULL << 21)); +} +static bool entries(pid_t pid, const void *base, uint64_t out[LEAVES]) +{ + char path[64]; snprintf(path, sizeof(path), "/proc/%ld/pagemap", (long)pid); + int fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) { fprintf(stderr, "PAGEMAP_OPEN_FAIL stage=%s pid=%ld errno=%d\n", stage, (long)pid, errno); return false; } + ssize_t n = pread(fd, out, LEAVES * 8, (uintptr_t)base / LEAF * 8); + close(fd); + if (n != LEAVES * 8) { fprintf(stderr, "PAGEMAP_READ_FAIL stage=%s bytes=%zd errno=%d\n", stage, n, errno); return false; } + for (size_t i = 0; i < LEAVES; i++) + if (!(out[i] & PRESENT) || !(out[i] & PFN_MASK)) { + fprintf(stderr, "PAGEMAP_ENTRY_FAIL stage=%s pid=%ld leaf=%zu raw=%016llx present=%d file=%d pfn_visible=%d\n", + stage, (long)pid, i, (unsigned long long)out[i], !!(out[i] & PRESENT), + !!(out[i] & FILE_PAGE), !!(out[i] & PFN_MASK)); + return false; + } + return true; +} +static bool quarter_file_state(const uint64_t *map, unsigned quarter, bool file) +{ + for (size_t leaf = quarter; leaf < LEAVES; leaf += 4) + if (!!(map[leaf] & FILE_PAGE) != file) { + fprintf(stderr, "PTE_KIND_FAIL stage=%s leaf=%zu expected_file=%d raw=%016llx\n", + stage, leaf, file, (unsigned long long)map[leaf]); + return false; + } + return true; +} +static unsigned mismatches(const uint64_t *map, unsigned logical_quarter, size_t *first) +{ + unsigned count = 0; + for (size_t leaf = logical_quarter; leaf < LEAVES; leaf += 4) { + if ((map[leaf] & PFN_MASK) % 4 != logical_quarter) { + if (!count && first) *first = leaf; + count++; + } + } + fprintf(stderr, "MISMATCH_WITNESS stage=%s logical_quarter=%u mismatches=%u/%lu\n", + stage, logical_quarter, count, WINDOWS); + return count; +} +/* Does not fault payload or require present entries: proves the fork premise. */ +static bool entry_summary(const void *base, const char *view) +{ + uint64_t map[LEAVES]; unsigned present = 0, file = 0; + int fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (fd < 0) return false; + ssize_t n = pread(fd, map, sizeof(map), (uintptr_t)base / LEAF * 8); + close(fd); + if (n != sizeof(map)) return false; + for (size_t i = 0; i < LEAVES; i++) { + present += !!(map[i] & PRESENT); + file += !!(map[i] & PRESENT) && !!(map[i] & FILE_PAGE); + } + fprintf(stderr, "PREFLIGHT stage=%s view=%s present=%u file_present=%u leaves=%lu first_raw=%016llx\n", + stage, view, present, file, LEAVES, (unsigned long long)map[0]); + return true; +} +static bool token(int fd, char wanted) +{ + char got; + return io_full(fd, &got, 1, false) && got == wanted; +} +#define CHILD_CHECK(test, name, code) do { stage = name; if (!(test)) { \ + fprintf(stderr, "CHILD_FAIL stage=%s exit=%d errno=%d\n", stage, code, errno); _exit(code); } } while (0) +static void child(unsigned char *a, unsigned char *b, int control, int status) +{ + struct message msg; + uint64_t before[LEAVES], after[LEAVES]; + alarm(20); + CHILD_CHECK(entry_summary(a, "A") && entry_summary(b, "B"), "before-child-prefault", 2); + /* Fork may skip reconstructible file PTEs. Establish them in this mm. */ + CHILD_CHECK(verify(a, 0, SIZE_MAX), "child-prefault-A-original", 3); + CHILD_CHECK(verify(b, 0, SIZE_MAX), "child-prefault-B-original", 3); + CHILD_CHECK(entry_summary(a, "A") && entry_summary(b, "B"), "after-child-prefault", 2); + CHILD_CHECK(entries(getpid(), a, before), "CPU-source-present", 3); + CHILD_CHECK(quarter_file_state(before, 3, true), "CPU-source-file-PTE", 3); + for (size_t window = 0; window < WINDOWS; window++) { + size_t at = (window * 4 + 3) * LEAF; + *(volatile unsigned char *)(a + at) = original(at) ^ 0x5a; + } + CHILD_CHECK(verify(a, 1, SIZE_MAX), "CPU-COW-full-bytes", 3); + CHILD_CHECK(entries(getpid(), a, after), "CPU-destination-present", 3); + CHILD_CHECK(quarter_file_state(after, 3, false), "CPU-destination-anon-PTE", 3); + size_t first = SIZE_MAX; + unsigned cpu_mismatch = mismatches(after, 3, &first); + CHILD_CHECK(cpu_mismatch, "CPU-COW-actual-quarter-mismatch", 5); + msg = (struct message){ .address = first, .phase = 1, .detail = cpu_mismatch }; + CHILD_CHECK(io_full(status, &msg, sizeof(msg), true), "CPU-result-write", 2); + CHILD_CHECK(token(control, 'm'), "wait-mremap-command", 2); + CHILD_CHECK(verify(a, 1, first), "post-GUP-A-full-bytes", 3); + CHILD_CHECK(verify(b, 2, SIZE_MAX), "post-GUP-B-full-bytes", 3); + CHILD_CHECK(entries(getpid(), b, before), "before-mremap-PFNs", 3); + void *reserve = mmap(NULL, BYTES + 2 * NATIVE, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHILD_CHECK(reserve != MAP_FAILED, "mremap-reservation", 2); + unsigned long desired = ((uintptr_t)b + LEAF) % NATIVE; + unsigned char *destination = (void *)((((uintptr_t)reserve + NATIVE - 1) & ~(NATIVE - 1)) + desired); + unsigned char *moved = mremap(b, BYTES, BYTES, MREMAP_MAYMOVE | MREMAP_FIXED, destination); + CHILD_CHECK(moved != MAP_FAILED && moved == destination && + (uintptr_t)moved % NATIVE != (uintptr_t)b % NATIVE, "mremap-shifted-VA", 3); + CHILD_CHECK(verify(moved, 2, SIZE_MAX), "mremap-full-bytes", 3); + CHILD_CHECK(entries(getpid(), moved, after), "after-mremap-PFNs", 3); + for (size_t leaf = 0; leaf < LEAVES; leaf++) { + if ((before[leaf] & PFN_MASK) != (after[leaf] & PFN_MASK)) { + fprintf(stderr, "PFN_CHANGE stage=mremap leaf=%zu before=%016llx after=%016llx\n", + leaf, (unsigned long long)before[leaf], (unsigned long long)after[leaf]); + CHILD_CHECK(false, "mremap-preserved-PFNs", 6); + } + } + msg = (struct message){ .address = (uintptr_t)moved, .phase = 2, .detail = desired }; + CHILD_CHECK(io_full(status, &msg, sizeof(msg), true), "mremap-result-write", 2); + CHILD_CHECK(token(control, 'x'), "wait-final-release", 2); + CHILD_CHECK(verify(a, 1, first), "final-A-full-bytes", 3); + CHILD_CHECK(verify(moved, 2, SIZE_MAX), "final-B-full-bytes", 3); + munmap(reserve, BYTES + 2 * NATIVE); munmap(a, BYTES); + close(control); close(status); _exit(0); +} +static bool remote_one(pid_t pid, void *local, void *remote, size_t bytes, bool write_it) +{ + struct iovec l = { .iov_base = local, .iov_len = bytes }; + struct iovec r = { .iov_base = remote, .iov_len = bytes }; + ssize_t n = write_it ? process_vm_writev(pid, &l, 1, &r, 1, 0) : + process_vm_readv(pid, &l, 1, &r, 1, 0); + return n == (ssize_t)bytes; +} +static bool remote_quarters(pid_t pid, void *local, unsigned char *remote, + unsigned quarter, size_t offset, size_t bytes, bool write_it) +{ + struct iovec l = { .iov_base = local, .iov_len = WINDOWS * bytes }; + struct iovec r[WINDOWS]; + for (size_t window = 0; window < WINDOWS; window++) + r[window] = (struct iovec){ .iov_base = remote + (window * 4 + quarter) * LEAF + offset, .iov_len = bytes }; + ssize_t n = write_it ? process_vm_writev(pid, &l, 1, r, WINDOWS, 0) : + process_vm_readv(pid, &l, 1, r, WINDOWS, 0); + return n == (ssize_t)(WINDOWS * bytes); +} +static bool wait_ok(pid_t pid) +{ + int status; pid_t n; + do { n = waitpid(pid, &status, 0); } while (n < 0 && errno == EINTR); + if (n == pid) { child_reaped = true; child_status = status; report_wait(pid, status); } + else fprintf(stderr, "CHILD_WAIT_ERROR pid=%ld errno=%d\n", (long)pid, errno); + return n == pid && WIFEXITED(status) && !WEXITSTATUS(status); +} +#define CHECK(test, message) do { stage = message; if (!(test)) { fprintf(stderr, "FAIL: %s (errno=%d)\n", message, errno); goto cleanup; } } while (0) +int main(int argc, char **argv) +{ + unsigned char *buffer = NULL, *a = MAP_FAILED, *b = MAP_FAILED; + int fd = -1, control[2] = {-1,-1}, status[2] = {-1,-1}; + pid_t pid = -1; bool ok = false; + uint64_t map[LEAVES]; struct message msg; + unsigned cpu_mismatch = 0, read_mismatch = 0, write_mismatch = 0; + size_t first_a = SIZE_MAX, first_b = SIZE_MAX; + if (argc != 2 || strcmp(argv[1], "--native-16k")) { + fprintf(stderr, "usage: %s --native-16k\n", argv[0]); return 2; + } + if (getauxval(AT_PAGESZ) != LEAF || !visible_pfns()) { + fprintf(stderr, "SKIP: established4K launcher/native16K kernel/visiblePFNs required\n"); return 4; + } + signal(SIGPIPE, SIG_IGN); signal(SIGALRM, stop); signal(SIGTERM, stop); signal(SIGINT, stop); + alarm(30); + buffer = malloc(BYTES); + CHECK(buffer, "buffer allocation"); + for (size_t i = 0; i < BYTES; i++) buffer[i] = original(i); + fd = memfd_create("own-file-cow-consumers", MFD_CLOEXEC); + CHECK(fd >= 0 && !ftruncate(fd, BYTES), "owned memfd"); + CHECK(pwrite(fd, buffer, BYTES, 0) == BYTES, "initialize memfd"); + a = mmap(NULL, BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + b = mmap(NULL, BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + CHECK(a != MAP_FAILED && b != MAP_FAILED, "private file views"); + /* Full reads establish file PTEs before COW and typed pin operations. */ + CHECK(verify(a, 0, SIZE_MAX) && verify(b, 0, SIZE_MAX), "prefault originals"); + CHECK(!pipe(control) && !pipe(status), "control pipes"); + pid = fork(); CHECK(pid >= 0, "fork"); + if (!pid) { + CHILD_CHECK(!setpgid(0, 0), "set-child-group", 2); + close(control[1]); close(status[0]); close(fd); + child(a, b, control[0], status[1]); + } + child_group = pid; + if (setpgid(pid, pid) && errno != EACCES && errno != ESRCH) CHECK(false, "child group"); + close(control[0]); control[0] = -1; close(status[1]); status[1] = -1; + bool received = io_full(status[0], &msg, sizeof(msg), false); + if (!received) { fprintf(stderr, "RESULT_EOF stage=CPU-COW; waiting for exact child status\n"); (void)wait_ok(pid); } + CHECK(received && msg.phase == 1, "CPU COW result"); + cpu_mismatch = msg.detail; first_a = msg.address; + CHECK(cpu_mismatch && first_a < LEAVES && first_a % 4 == 3, "actual CPU COW offset mismatch"); + CHECK(remote_one(pid, buffer, a + first_a * LEAF, LEAF, false), "GUP read of mismatched anonymous slot"); + for (size_t i = 0; i < LEAF; i++) + CHECK(buffer[i] == (unsigned char)(original(first_a * LEAF + i) ^ (i == 0 ? 0x5a : 0)), "GUP read bytes"); + memset(buffer, 0xe3, 32); + CHECK(remote_one(pid, buffer, a + first_a * LEAF + 17, 32, true), "GUP write of mismatched anonymous slot"); + CHECK(entries(pid, b, map) && quarter_file_state(map, 3, true), "read-pin sources remain file PTEs"); + CHECK(remote_quarters(pid, buffer, b, 3, 0, LEAF, false), "typed read-pin file unshare"); + for (size_t w = 0; w < WINDOWS; w++) + for (size_t i = 0; i < LEAF; i++) + CHECK(buffer[w * LEAF + i] == original((w * 4 + 3) * LEAF + i), "read-pin copied bytes"); + CHECK(entries(pid, b, map) && quarter_file_state(map, 3, false), "read-pin produced anonymous slots"); + read_mismatch = mismatches(map, 3, &first_b); + CHECK(read_mismatch && first_b < LEAVES, "actual read-pin file COW offset mismatch"); + CHECK(quarter_file_state(map, 2, true), "write-pin sources remain file PTEs"); + memset(buffer, 0xc7, WINDOWS * 32); + CHECK(remote_quarters(pid, buffer, b, 2, 37, 32, true), "typed write-pin file COW"); + CHECK(entries(pid, b, map) && quarter_file_state(map, 2, false), "write-pin produced anonymous slots"); + write_mismatch = mismatches(map, 2, NULL); + CHECK(write_mismatch, "actual write-pin file COW offset mismatch"); + char command = 'm'; CHECK(io_full(control[1], &command, 1, true), "mremap command"); + received = io_full(status[0], &msg, sizeof(msg), false); + if (!received) { fprintf(stderr, "RESULT_EOF stage=mremap; waiting for exact child status\n"); (void)wait_ok(pid); } + CHECK(received && msg.phase == 2, "mremap content/PFN result"); + unsigned char *moved = (void *)msg.address; + CHECK(remote_one(pid, buffer, moved + first_b * LEAF, LEAF, false), "GUP read after shifted mremap"); + for (size_t i = 0; i < LEAF; i++) CHECK(buffer[i] == original(first_b * LEAF + i), "moved GUP bytes"); + CHECK(verify(a, 0, SIZE_MAX) && verify(b, 0, SIZE_MAX), "parent private views unchanged"); + CHECK(pread(fd, buffer, BYTES, 0) == BYTES && verify(buffer, 0, SIZE_MAX), "original memfd byte exact"); + command = 'x'; CHECK(io_full(control[1], &command, 1, true), "child release"); + CHECK(wait_ok(pid), "child final integrity"); pid = -1; child_group = 0; + printf("{\"file_bytes\":%lu,\"cpu_cow_mismatches\":%u,\"read_pin_cow_mismatches\":%u," + "\"write_pin_cow_mismatches\":%u,\"targets_per_path\":%lu,\"mremap_shifted_va_quarter\":true," + "\"mremap_preserved_pfns\":true,\"gup_after_mremap\":true,\"parent_and_file_unchanged\":true,\"content_ok\":true}\n", + BYTES, cpu_mismatch, read_mismatch, write_mismatch, WINDOWS); + ok = true; +cleanup: + if (pid > 0 && !child_reaped) { + pid_t got = waitpid(pid, &child_status, WNOHANG); + if (got == pid) { child_reaped = true; report_wait(pid, child_status); } + else { kill(-pid, SIGKILL); (void)wait_ok(pid); } + } + child_group = 0; alarm(0); + for (unsigned i = 0; i < 2; i++) { if (control[i] >= 0) close(control[i]); if (status[i] >= 0) close(status[i]); } + if (a != MAP_FAILED) munmap(a, BYTES); + if (b != MAP_FAILED) munmap(b, BYTES); + if (fd >= 0) close(fd); + free(buffer); + return !ok; +} diff --git a/tools/testing/selftests/arm64/user4k/first-write.c b/tools/testing/selftests/arm64/user4k/first-write.c new file mode 100644 index 00000000000000..9bb2cb89b04b29 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/first-write.c @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Concurrent first writes to absent PTEs must preserve the untouched bytes, + * neighbouring private mappings and the source file. Small mappings have an + * inaccessible 4K guard between them, so anonymous fault-around cannot fill + * the whole native page. Density/reclaim are measured separately on hardware. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define LEAF 4096UL +#define LEAVES 1024UL +#define WORKERS 8 +#define TMPFS_MAGIC 0x01021994 + +enum backing { ANONYMOUS, SHMEM, REGULAR }; +static unsigned char *data; +static size_t stride; +static pthread_barrier_t barrier; +static bool race; +static volatile sig_atomic_t running_child; +static void stop_fixture(int sig) +{ + if (running_child > 0) + kill(running_child, SIGKILL); + _exit(128 + sig); +} + +static unsigned char initial(size_t leaf, size_t byte) +{ + return leaf * 17 + byte * 3 + 0x39; +} + +static void *writer(void *arg) +{ + size_t leaf, id = (uintptr_t)arg; + + pthread_barrier_wait(&barrier); + if (race) { + /* Distinct bytes avoid a C data race while faulting the same PTE. */ + for (leaf = 0; leaf < LEAVES; leaf += 3) + data[leaf * stride + id] = initial(leaf, id) ^ 0xa5; + } else { + for (leaf = id; leaf < LEAVES; leaf += WORKERS) + data[leaf * stride] = initial(leaf, 0) ^ 0xa5; + } + return NULL; +} + +static bool verify(enum backing backing) +{ + size_t leaf, byte; + + for (leaf = 0; leaf < LEAVES; leaf++) { + for (byte = 0; byte < LEAF; byte++) { + unsigned char want = backing == ANONYMOUS ? 0 : initial(leaf, byte); + + if (!byte || (race && leaf % 3 == 0 && byte < WORKERS)) + want = initial(leaf, byte) ^ 0xa5; + if (data[leaf * stride + byte] != want) { + ksft_print_msg("leaf=%zu byte=%zu got=%u want=%u\n", leaf, byte, + data[leaf * stride + byte], want); + return false; + } + } + } + return true; +} + +static int exercise(enum backing backing, bool small) +{ + unsigned char buffer[LEAF]; + char name[] = "first-write-XXXXXX"; + pthread_t workers[WORKERS]; + struct statfs fs; + int fd = -1, flags = MAP_PRIVATE; + size_t leaf, byte, span; + unsigned int t; + + stride = small ? 2 * LEAF : LEAF; + span = LEAVES * stride; + if (backing != ANONYMOUS) { + fd = backing == SHMEM ? memfd_create("first-write", MFD_CLOEXEC) : + mkstemp(name); + if (fd < 0) + return 2; + if (backing == REGULAR) + unlink(name); + if (fstatfs(fd, &fs)) + return 2; + if (backing == REGULAR && fs.f_type == TMPFS_MAGIC) { + ksft_print_msg("regular-file arm needs a non-tmpfs working directory\n"); + return KSFT_SKIP; + } + if (ftruncate(fd, span)) + return 2; + for (leaf = 0; leaf < LEAVES; leaf++) { + for (byte = 0; byte < LEAF; byte++) + buffer[byte] = initial(leaf, byte); + if (pwrite(fd, buffer, LEAF, leaf * stride) != LEAF) + return 2; + } + } else { + flags |= MAP_ANONYMOUS; + } + data = mmap(NULL, span, small ? PROT_NONE : PROT_READ | PROT_WRITE, + small ? MAP_PRIVATE | MAP_ANONYMOUS : flags, small ? -1 : fd, 0); + if (data == MAP_FAILED) + return 2; + if (small) + for (leaf = 0; leaf < LEAVES; leaf++) + if (mmap(data + leaf * stride, LEAF, PROT_READ | PROT_WRITE, + flags | MAP_FIXED, fd, backing == ANONYMOUS ? 0 : leaf * stride) == + MAP_FAILED) + return 2; + + /* No reads through the mapping before concurrent direct first writes. */ + if (pthread_barrier_init(&barrier, NULL, WORKERS)) + return 2; + for (t = 0; t < WORKERS; t++) + if (pthread_create(&workers[t], NULL, writer, (void *)(uintptr_t)t)) + return 2; + for (t = 0; t < WORKERS; t++) + if (pthread_join(workers[t], NULL)) + return 2; + if (!verify(backing)) + return 3; + /* Drop selected private slots, leaving the other slots live. */ + for (leaf = 0; leaf < LEAVES; leaf += 3) + if (madvise(data + leaf * stride, LEAF, MADV_DONTNEED)) + return 2; + /* Again no reads before the first writes: exercise abort and slot reuse. */ + race = true; + for (t = 0; t < WORKERS; t++) + if (pthread_create(&workers[t], NULL, writer, (void *)(uintptr_t)t)) + return 2; + for (t = 0; t < WORKERS; t++) + if (pthread_join(workers[t], NULL)) + return 2; + pthread_barrier_destroy(&barrier); + if (!verify(backing)) + return 3; + if (backing != ANONYMOUS) { + for (leaf = 0; leaf < LEAVES; leaf++) { + if (pread(fd, buffer, LEAF, leaf * stride) != LEAF) + return 2; + for (byte = 0; byte < LEAF; byte++) + if (buffer[byte] != initial(leaf, byte)) + return 3; + } + close(fd); + } + munmap(data, span); + return 0; +} + +int main(int argc, char **argv) +{ + static const char * const names[] = { "anonymous", "shmem", "regular-file" }; + unsigned int backing, small; + int status; + pid_t child, result; + + if (argc != 2 || strcmp(argv[1], "--native-16k")) + return 2; + signal(SIGALRM, stop_fixture); + signal(SIGTERM, stop_fixture); + signal(SIGINT, stop_fixture); + ksft_print_header(); + if (getauxval(AT_PAGESZ) != LEAF) + ksft_exit_skip("run under a selected 4K userspace ABI\n"); + ksft_set_plan(6); + fflush(NULL); + alarm(40); + for (backing = ANONYMOUS; backing <= REGULAR; backing++) { + for (small = 0; small <= 1; small++) { + child = fork(); + if (child < 0) + ksft_exit_fail_msg("fork: %s\n", strerror(errno)); + if (!child) { + running_child = 0; + alarm(15); + _exit(exercise(backing, small)); + } + running_child = child; + do { + result = waitpid(child, &status, 0); + } while (result < 0 && errno == EINTR); + running_child = 0; + if (result > 0 && WIFEXITED(status) && WEXITSTATUS(status) == KSFT_SKIP) + ksft_test_result_skip("%s %s first writes\n", names[backing], + small ? "guarded" : "dense"); + else + ksft_test_result(result > 0 && WIFEXITED(status) && + !WEXITSTATUS(status), + "%s %s first writes and slot reuse preserve contents\n", + names[backing], small ? "guarded" : "dense"); + } + } + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/gem-fixture-init.c b/tools/testing/selftests/arm64/user4k/gem-fixture-init.c new file mode 100644 index 00000000000000..aadf343dc7e8d4 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/gem-fixture-init.c @@ -0,0 +1,325 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static unsigned char value(size_t off) +{ + return 0x31 + (off / 4096) % 128; +} +static bool check(unsigned char *p, size_t len, size_t off) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != value(i + off)) { + ksft_print_msg("offset=%zu got=%u expected=%u\n", + i + off, p[i], value(i + off)); + return false; + } + return true; +} +static int exercise(size_t native) +{ + size_t ps = getauxval(AT_PAGESZ), + len = native < 8192 ? 65536 : 8 * native; + int fd = open("/dev/vgem", O_RDWR); + struct drm_mode_create_dumb create = { .width = 4096, + .height = len / (4096 * 4), + .bpp = 32 }; + struct drm_mode_map_dumb map = {}; + struct drm_prime_handle prime = { .flags = DRM_RDWR }; + unsigned char *p = MAP_FAILED, *q = MAP_FAILED, *alias = MAP_FAILED; + bool ok = fd >= 0 && !ioctl(fd, DRM_IOCTL_MODE_CREATE_DUMB, &create); + ksft_print_header(); + ksft_set_plan(12); + ksft_print_msg("ABI=%zuK\n", ps / 1024); + map.handle = create.handle; + ok &= !ioctl(fd, DRM_IOCTL_MODE_MAP_DUMB, &map) && create.size >= len; + if (ok) + p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + map.offset); + ok &= p != MAP_FAILED; + ksft_test_result(ok, "full GEM shmem object mapping\n"); + q = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + map.offset); + bool separate = q != MAP_FAILED; + if (separate) { + memset(q, 0x39, ps); + memset(q + ps, 0xa2, ps); + for (size_t i = 0; i < ps; i++) + separate &= q[i] == 0x39 && q[ps + i] == 0xa2; + munmap(q, 2 * ps); + q = MAP_FAILED; + } + ksft_test_result(separate, + "adjacent user leaves map distinct backing bytes\n"); + if (ok) { + for (size_t i = 0; i < len; i++) + p[i] = value(i); + ok = check(p, len, 0); + } + ksft_test_result(ok, "full object has distinct quarter data\n"); + if (ok) { + q = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + map.offset); + ok = q != MAP_FAILED && !mprotect(q + ps, ps, PROT_READ); + if (ok) + ok = check(q, len, 0); + if (q != MAP_FAILED) + munmap(q, len); + q = MAP_FAILED; + } + ksft_test_result( + ok, + "cold faults after mprotect VMA splitting preserve object offsets\n"); + if (p != MAP_FAILED) { + q = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + map.offset); + ok = q != MAP_FAILED && !munmap(q + ps, ps); + if (ok) + ok = check(q + 2 * ps, len - 2 * ps, 2 * ps) && + check(q, ps, 0); + if (q != MAP_FAILED) + munmap(q, len); + q = MAP_FAILED; + } else + ok = false; + ksft_test_result( + ok, + "cold faults after partial unmap preserve object offsets\n"); + prime.handle = create.handle; + ok = !ioctl(fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, &prime); + if (ok) { + alias = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, + prime.fd, 0); + ok = alias != MAP_FAILED && check(alias, len, 0); + } + ksft_test_result(ok, "PRIME exported full mapping shares GEM data\n"); + if (alias != MAP_FAILED) { + q = mmap(NULL, ps, PROT_READ | PROT_WRITE, MAP_SHARED, prime.fd, + 3 * ps); + ok = q != MAP_FAILED; + if (ok) + ok = check(q, ps, 3 * ps); + if (q != MAP_FAILED) + munmap(q, ps); + q = MAP_FAILED; + } else + ok = false; + ksft_test_result( + ok, "PRIME partial mapping uses requested buffer offset\n"); + if (alias != MAP_FAILED) { + q = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, + prime.fd, 0); + void *target = mmap(NULL, len, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + ok = q != MAP_FAILED && target != MAP_FAILED; + if (ok) { + void *moved = mremap(q, len, len, + MREMAP_MAYMOVE | MREMAP_FIXED, + target); + ok = moved == target; + if (ok) { + q = target; + ok = check(q, len, 0); + } + } + if (q != MAP_FAILED) + munmap(q, len); + if (target != MAP_FAILED && target != q) + munmap(target, len); + q = MAP_FAILED; + } else + ok = false; + ksft_test_result( + ok, + "cold GEM faults after fixed mremap retain object offsets\n"); + if (alias != MAP_FAILED) { + pid_t child = fork(); + int status = 0; + if (!child) { + unsigned char *cold = mmap(NULL, len, + PROT_READ | PROT_WRITE, + MAP_SHARED, prime.fd, 0); + bool pass = cold != MAP_FAILED && check(cold, len, 0); + if (pass) + cold[0] = 0xe1; + _exit(pass ? 0 : 1); + } + ok = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status) && p[0] == 0xe1; + p[0] = value(0); + } else + ok = false; + ksft_test_result( + ok, "forked GEM mapping shares bytes and retains backing\n"); + if (alias != MAP_FAILED) { + q = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE, MAP_SHARED, + prime.fd, len - ps); + ok = q == MAP_FAILED && errno == EINVAL; + if (q != MAP_FAILED) + munmap(q, 2 * ps); + q = mmap(NULL, ps, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + map.offset + ps); + ok &= q == MAP_FAILED && errno == EINVAL; + if (q != MAP_FAILED) + munmap(q, ps); + q = MAP_FAILED; + } else + ok = false; + ksft_test_result(ok, "PRIME bounds and exact GEM offset rejection\n"); + if (alias != MAP_FAILED) { + size_t targets[] = { native, 4096, 16384 }; + + ok = true; + for (unsigned int t = 0; t < 3; t++) { + size_t target = targets[t]; + pid_t child; + int status = 0; + bool pair; + + if (target > native || target == ps || + (t && target == native)) + continue; + child = fork(); + if (!child) { + char number[24], length[24], pages[24]; + + snprintf(number, sizeof(number), "%d", + prime.fd); + snprintf(length, sizeof(length), "%zu", len); + snprintf(pages, sizeof(pages), "%zu", target); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + target == native ? 0UL : target, 0UL, + 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--shared", number, + length, pages, NULL); + _exit(4); + } + pair = child > 0 && + waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + for (size_t i = 0; i < len; i += 4096) { + pair &= p[i] == + (unsigned char)(0x80 + i / 4096); + p[i] = value(i); + } + ksft_print_msg( + "PRIME sharing %zuK -> %zuK status=%d bytes=%zu\n", + ps / 1024, target / 1024, status, len); + ok &= pair; + } + } else { + ok = false; + } + ksft_test_result( + ok, + "PRIME backing shared between native and alternative processes\n"); + struct drm_gem_close close_arg = { .handle = create.handle }; + ok = !ioctl(fd, DRM_IOCTL_GEM_CLOSE, &close_arg); + close(fd); + if (alias != MAP_FAILED) { + close(prime.fd); + ok &= check(alias, len, 0); + munmap(alias, len); + } else + ok = false; + if (p != MAP_FAILED) { + ok &= check(p, len, 0); + munmap(p, len); + } else + ok = false; + ksft_test_result( + ok, + "mappings retain object after handles and descriptors close\n"); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} +int main(int argc, char **argv) +{ + bool ok = true; + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--shared")) { + struct stat init, self; + int fd = atoi(argv[2]); + size_t len = strtoul(argv[3], NULL, 10); + if (stat("/proc/1/exe", &init) || + stat("/proc/self/exe", &self) || + init.st_ino != self.st_ino || init.st_dev != self.st_dev || + len < 65536 || len > 8 * 65536 || + getauxval(AT_PAGESZ) != strtoul(argv[4], NULL, 10)) + return 2; + unsigned char *p = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + bool pass = p != MAP_FAILED && check(p, len, 0); + if (pass) + for (size_t i = 0; i < len; i += 4096) + p[i] = 0x80 + i / 4096; + if (p != MAP_FAILED) + munmap(p, len); + return pass ? 0 : 1; + } + if (argc == 3 && !strcmp(argv[1], "--test") && getppid() == 1) + return exercise(strtoul(argv[2], NULL, 10)); + if (getpid() != 1 || argc != 1) + return 2; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) + return 3; + FILE *dev = fopen("/sys/class/drm/card0/dev", "r"); + unsigned maj, min; + if (!dev || fscanf(dev, "%u:%u", &maj, &min) != 2) + return 4; + fclose(dev); + if (mknod("/dev/vgem", S_IFCHR | 0600, makedev(maj, min))) + return 5; + size_t native = getauxval(AT_PAGESZ), + targets[] = { native, 4096, 16384 }; + + for (unsigned int t = 0; t < 3; t++) { + size_t target = targets[t]; + pid_t pid; + int status = 0; + + if (target > native || (t && target == native)) + continue; + pid = fork(); + if (!pid) { + char number[24]; + + snprintf(number, sizeof(number), "%zu", native); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + target == native ? 0UL : target, 0UL, 0UL, + 0UL)) + _exit(3); + execl("/init", "/init", "--test", number, NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("# GEM child ABI=%zuK native=%zuK status=%d\n", + target / 1024, native / 1024, status); + } + printf("%s - GEM fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/granule-proc-init.c b/tools/testing/selftests/arm64/user4k/granule-proc-init.c new file mode 100644 index 00000000000000..e70f8df18e57a2 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/granule-proc-init.c @@ -0,0 +1,141 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +static int failed; +static void result(int ok, const char *s) +{ + printf("%s - %s\n", ok ? "ok" : "not ok", s); + failed += !ok; +} +static long pss(void *p) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + char *line = NULL; + size_t cap = 0; + unsigned long start, end; + int active = 0; + long value = -1; + if (!f) + return -1; + while (getline(&line, &cap, f) > 0) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) + active = (unsigned long)p >= start && + (unsigned long)p < end; + else if (active && sscanf(line, "Pss: %ld kB", &value) == 1) + break; + } + free(line); + fclose(f); + return value; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--peer")) { + int fd = atoi(argv[2]), ready = atoi(argv[3]), + release = atoi(argv[4]); + size_t native = strtoul(argv[5], NULL, 10); + unsigned char *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, native - 4096); + char token = 'a'; + int ok = getauxval(AT_PAGESZ) == 4096 && p != MAP_FAILED; + if (ok) { + for (unsigned i = 0; i < 4096; i++) + ok &= p[i] == 0x7b; + p[0] = 0xc9; + ok &= pss(p) == 2; + } + token = ok ? 'a' : 'x'; + if (write(ready, &token, 1) != 1) + return 3; + if (read(release, &token, 1) != 1) + return 4; + return ok ? 0 : 2; + } + if (getpid() != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + mount("proc", "/proc", "proc", 0, NULL); + size_t native = getauxval(AT_PAGESZ); + int fd = memfd_create("high-slot-pss", 0); + result(fd >= 0 && !ftruncate(fd, native), + "native file backing created"); + unsigned char *p = + mmap(NULL, native, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (p == MAP_FAILED) { + result(0, "parent mapping"); + goto done; + } + memset(p, 0x7b, native); + long before = pss(p); + printf("# native=%zu initial PSS=%ldKB\n", native, before); + result(before == (long)native / 1024, + "native PSS accounts every physical slot"); + int ready[2], release[2]; + if (pipe(ready) || pipe(release)) { + result(0, "pipes"); + goto done; + } + pid_t child = fork(); + int status = 0; + if (!child) { + char f[24], r[24], w[24], n[24]; + snprintf(f, sizeof(f), "%d", fd); + snprintf(r, sizeof(r), "%d", ready[1]); + snprintf(w, sizeof(w), "%d", release[0]); + snprintf(n, sizeof(n), "%zu", native); + close(ready[0]); + close(release[1]); + if (prctl(SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--peer", f, r, w, n, NULL); + _exit(5); + } + close(ready[1]); + close(release[0]); + char token = 0; + int ok = child > 0 && read(ready[0], &token, 1) == 1 && token == 'a'; + result(ok && p[native - 4096] == 0xc9, + "4K peer shares the final physical slot and observes 2KB PSS"); + long during = pss(p); + printf("# parent mixed PSS=%ldKB expected=%zuKB\n", during, + native / 1024 - 2); + result(during == (long)native / 1024 - 2, + "native PSS weights the high shared slot independently"); + token = 'b'; + if (write(release[1], &token, 1) != 1) + failed++; + ok = waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status); + result(ok && pss(p) == (long)native / 1024, + "PSS returns to native size after peer releases its mapping"); + FILE *f = fopen("/proc/self/smaps_rollup", "r"); + char buf[512]; + ok = f != NULL; + if (f) { + while (fgets(buf, sizeof(buf), f)) { + } + ok &= !ferror(f); + fclose(f); + } + result(ok, "smaps_rollup reads complete native slot accounting"); + munmap(p, native); +done: + printf("GRANULE PROC %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/granule-swap-init.c b/tools/testing/selftests/arm64/user4k/granule-swap-init.c new file mode 100644 index 00000000000000..5791f7e2c0ee9a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/granule-swap-init.c @@ -0,0 +1,248 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "swapfile-fixture.h" +#define SET_EXEC_PAGE_SIZE 0x41555001 +static unsigned char pattern(size_t i, unsigned round) +{ + return (i ^ (i >> 12) ^ (i >> 20) ^ round) & 255; +} +static int check(unsigned char *p, size_t size, unsigned round) +{ + for (size_t i = 0; i < size; i++) + if (p[i] != pattern(i, round)) { + printf("# byte=%zu got=%u expected=%u\n", i, p[i], + pattern(i, round)); + return 0; + } + return 1; +} +static unsigned long swapouts(void) +{ + FILE *f = fopen("/proc/vmstat", "r"); + char k[80]; + unsigned long n = 0, result = 0; + if (!f) + return 0; + while (fscanf(f, "%79s %lu", k, &n) == 2) + if (!strcmp(k, "pswpout")) + result = n; + fclose(f); + return result; +} +static unsigned swapped(void *p, size_t size, size_t ps) +{ + int fd = open("/proc/self/pagemap", O_RDONLY); + uint64_t bits; + unsigned n = 0; + if (fd < 0) + return 0; + for (size_t i = 0; i < size; i += ps) + if (pread(fd, &bits, 8, ((uintptr_t)p + i) / ps * 8) == 8) + n += (bits >> 62) & 1; + close(fd); + return n; +} +static int exercise(unsigned long expected) +{ + size_t ps = getauxval(AT_PAGESZ), size = 8 * 1024 * 1024; + unsigned char *p = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + int failed = 0; + if (ps != expected || p == MAP_FAILED) { + printf("not ok - expected ABI=%lu actual=%zu\n", expected, ps); + return 1; + } + for (unsigned round = 0; round < 4; round++) { + for (size_t i = 0; i < size; i++) + p[i] = pattern(i, round); + unsigned long before = swapouts(); + int advice = 0; + unsigned n = 0, attempts = 0; + /* PAGEOUT is advisory. Require evidence, with a bounded retry window. */ + do { + advice = madvise(p, size, MADV_PAGEOUT); + attempts++; + if (advice) + break; + usleep(20000); + n = swapped(p, size, ps); + } while (!n && attempts < 100); + unsigned long written = swapouts() - before; + int ok = !advice && written > 0 && n > 0; + printf("%s - %zuK round%u real swapped PTEs=%u/%zu writes=%lu attempts=%u\n", + ok ? "ok" : "not ok", ps / 1024, round, n, size / ps, + written, attempts); + failed += !ok; + int release[2], ready[2]; + if (pipe(release) || pipe(ready)) + return 2; + pid_t child = fork(); + int status = 0; + if (!child) { + char token; + close(release[1]); + close(ready[0]); + int ok = read(release[0], &token, 1) == 1 && + check(p, size, round); + if (ok) { + for (size_t i = 0; i < size; i += ps) + p[i] ^= 0x6d; + } + token = ok ? 'a' : 'x'; + write(ready[1], &token, 1); + _exit(ok ? 0 : 3); + } + close(release[0]); + close(ready[1]); + ok = check(p, size, round); + printf("%s - %zuK round%u disk roundtrip bytes\n", + ok ? "ok" : "not ok", ps / 1024, round); + failed += !ok; + char token = 'b'; + write(release[1], &token, 1); + ok = read(ready[0], &token, 1) == 1 && token == 'a' && + waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status) && check(p, size, round); + printf("%s - %zuK round%u forked swap-in and COW retain parent bytes\n", + ok ? "ok" : "not ok", ps / 1024, round); + failed += !ok; + close(release[1]); + close(ready[0]); + } + munmap(p, size); + return failed ? 1 : 0; +} +static int run_suite(bool nonlinear) +{ + const char *base = + nonlinear ? + "/sys/kernel/debug/kunit/arm64-user4k-nonlinear-swap/" : + "/sys/kernel/debug/kunit/arm64-user4k-swap/"; + char path[160]; + snprintf(path, sizeof(path), "%srun", base); + int fd = open(path, O_WRONLY); + if (fd < 0) + return 1; + int ok = write(fd, "1", 1) == 1; + close(fd); + snprintf(path, sizeof(path), "%sresults", base); + FILE *f = fopen(path, "r"); + char *line = NULL; + size_t cap = 0; + int completed = 0; + if (!f) + return 1; + while (getline(&line, &cap, f) > 0) { + fputs(line, stdout); + if (strstr(line, "not ok") || strstr(line, "# SKIP")) + ok = 0; + if (strstr(line, nonlinear ? + "ok 1 arm64-user4k-nonlinear-swap" : + "ok 1 arm64-user4k-swap")) + completed = 1; + } + free(line); + fclose(f); + return !(ok && completed); +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 2 && (!strcmp(argv[1], "--swapoff-helper") || + !strcmp(argv[1], "--swapon-helper"))) { + struct stat a, b; + if (getpid() == 1 || stat("/proc/1/exe", &a) || + stat("/proc/self/exe", &b) || a.st_dev != b.st_dev || + a.st_ino != b.st_ino) + return 1; + return (!strcmp(argv[1], "--swapoff-helper") ? + swapoff(swap_path()) : + swapon(swap_path(), 0)) ? + 1 : + 0; + } + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 1; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || + (argc != 1 && (argc != 2 || (strcmp(argv[1], "--el0-only") && + strcmp(argv[1], "--nonlinear") && + strcmp(argv[1], "--swapfile"))))) { + fprintf(stderr, "Dedicated disposable VM PID1 only\n"); + return 2; + } + int failed = 0, active = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)) { + failed++; + goto done; + } + struct stat st; + for (int i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + if (stat("/dev/vda", &st) || !S_ISBLK(st.st_mode) || + (argc == 2 && !strcmp(argv[1], "--swapfile") && + prepare_swapfile()) || + swapon(swap_path(), 0)) { + perror("swapon"); + failed++; + goto done; + } + active = 1; + printf("GRANULE SWAP BACKING=%s\n", swap_path()); + if (argc == 1 || (argc == 2 && !strcmp(argv[1], "--nonlinear"))) + failed += run_suite(argc == 2); + unsigned long sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + for (unsigned abi = 0; abi < (sizes[0] > 16384 ? 3U : 2U); abi++) { + pid_t c = fork(); + int status = 0; + if (!c) { + char expected[32]; + snprintf(expected, sizeof(expected), "%lu", sizes[abi]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[abi], 0UL, 0UL, + 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", expected, NULL); + _exit(5); + } + int ok = c > 0 && waitpid(c, &status, 0) == c && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - swap ABI %luK child status=%#x\n", + ok ? "ok" : "not ok", sizes[abi] / 1024, status); + failed += !ok; + } +done: + if (active && swapoff(swap_path())) { + perror("swapoff"); + failed++; + } + if (active && argc == 2 && !strcmp(argv[1], "--swapfile") && + umount("/swapdir")) { + perror("umount"); + failed++; + } + printf("GRANULE SWAP %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/granule-threeway-init.c b/tools/testing/selftests/arm64/user4k/granule-threeway-init.c new file mode 100644 index 00000000000000..71a459c9b52219 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/granule-threeway-init.c @@ -0,0 +1,248 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +struct snapshot { + long pss; + int first, last, status; +}; +struct peer { + pid_t pid; + int command, reply; + size_t size; +}; +static int failed; +static void result(int ok, const char *s) +{ + printf("%s - %s\n", ok ? "ok" : "not ok", s); + failed += !ok; +} +static long pss(void *p) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + char *line = NULL; + size_t cap = 0; + unsigned long start, end; + int active = 0; + long value = -1; + if (!f) + return -1; + while (getline(&line, &cap, f) > 0) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) + active = (unsigned long)p >= start && + (unsigned long)p < end; + else if (active && sscanf(line, "Pss: %ld kB", &value) == 1) + break; + } + free(line); + fclose(f); + return value; +} +static int transfer(int fd, void *data, size_t size, int writing) +{ + size_t done = 0; + while (done < size) { + ssize_t n = + writing ? write(fd, (char *)data + done, size - done) : + read(fd, (char *)data + done, size - done); + if (n < 0 && errno == EINTR) + continue; + if (n <= 0) + return 0; + done += n; + } + return 1; +} +static int worker(int fd, int command, int reply, size_t size) +{ + const struct rlimit no_core = { 0, 0 }; + setrlimit(RLIMIT_CORE, &no_core); + if (getppid() != 1 || getauxval(AT_PAGESZ) != size) + return 2; + unsigned char *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, 65536 - size); + if (p == MAP_FAILED) + return 3; + for (;;) { + char op; + struct snapshot s = { .pss = -1, .first = -1, .last = -1 }; + if (!transfer(command, &op, 1, 0)) + return 4; + if (op == 'x') { + s.status = munmap(p, size); + transfer(reply, &s, sizeof(s), 1); + return s.status ? 5 : 0; + } + if (op == 'r') + s.status = mprotect(p, size, PROT_READ); + else if (op == 'w') + p[size - 1] = 0x99; + else if (op == 'd') + s.status = madvise(p, size, MADV_DONTNEED); + else if (op == 't') { + pid_t c = fork(); + int status = 0; + if (!c) { + unsigned char value = *( + volatile unsigned char *)(p + size - 1); + _exit(value); + } + if (c < 0 || waitpid(c, &status, 0) != c) + s.status = -1; + else + s.status = WIFSIGNALED(status) ? + 256 + WTERMSIG(status) : + WIFEXITED(status) ? + WEXITSTATUS(status) : + -2; + } + if (op != 'd' && op != 't') { + s.first = p[0]; + s.last = p[size - 1]; + } + s.pss = pss(p); + if (!transfer(reply, &s, sizeof(s), 1)) + return 6; + } +} +static struct peer spawn(int fd, size_t size) +{ + int command[2], reply[2]; + struct peer p = { .pid = -1, .size = size }; + if (pipe(command) || pipe(reply)) + return p; + p.pid = fork(); + if (!p.pid) { + char f[24], c[24], r[24], s[24]; + close(command[1]); + close(reply[0]); + snprintf(f, sizeof(f), "%d", fd); + snprintf(c, sizeof(c), "%d", command[0]); + snprintf(r, sizeof(r), "%d", reply[1]); + snprintf(s, sizeof(s), "%zu", size); + if (prctl(SET_EXEC_PAGE_SIZE, size, 0UL, 0UL, 0UL)) + _exit(7); + execl("/init", "/init", "--worker", f, c, r, s, NULL); + _exit(8); + } + close(command[0]); + close(reply[1]); + p.command = command[1]; + p.reply = reply[0]; + return p; +} +static struct snapshot command(struct peer *p, char op) +{ + struct snapshot s = { .pss = -1, .first = -1, .last = -1, .status = -1 }; + if (p->pid <= 0 || !transfer(p->command, &op, 1, 1) || + !transfer(p->reply, &s, sizeof(s), 0)) + failed++; + return s; +} +static void finish(struct peer *p) +{ + struct snapshot s = command(p, 'x'); + int status = 0; + result(!s.status && waitpid(p->pid, &status, 0) == p->pid && + WIFEXITED(status) && !WEXITSTATUS(status), + "peer unmap and exit"); + close(p->command); + close(p->reply); +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGPIPE, SIG_IGN); + if (argc == 6 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), atoi(argv[4]), + strtoul(argv[5], NULL, 10)); + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable64K VM PID1 only\n"); + return 2; + } + mount("proc", "/proc", "proc", 0, NULL); + if (getauxval(AT_PAGESZ) != 65536) { + result(0, "native64K required"); + goto done; + } + int fd = memfd_create("three-granules", 0); + if (fd < 0 || ftruncate(fd, 65536)) { + result(0, "file setup"); + goto done; + } + unsigned char *p = + mmap(NULL, 65536, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (p == MAP_FAILED) { + result(0, "native mapping"); + goto done; + } + memset(p, 0x42, 65536); + struct peer medium = spawn(fd, 16384); + struct snapshot s = command(&medium, 's'); + result(s.pss == 8 && s.first == 0x42 && s.last == 0x42 && pss(p) == 56, + "64K/16K initial sharing and PSS"); + struct peer small = spawn(fd, 4096); + s = command(&small, 's'); + result(s.pss == 1 && s.first == 0x42 && s.last == 0x42, + "4K peer shares the final slice with both larger ABIs"); + s = command(&medium, 's'); + result(s.pss == 7 && pss(p) == 55, + "three-way PSS weights partial overlaps independently"); + s = command(&small, 'r'); + result(!s.status, "4K alias becomes read-only"); + s = command(&medium, 'w'); + result(!s.status && p[65535] == 0x99, + "16K alias remains writable across local4K protection"); + s = command(&small, 's'); + result(s.last == 0x99, "read-only4K alias observes16K writes"); + s = command(&small, 'd'); + result(!s.status && s.pss == 0 && pss(p) == 56, + "4K discard removes only its mapping"); + s = command(&medium, 's'); + result(s.pss == 8 && s.last == 0x99, "16K mapping survives4K discard"); + s = command(&small, 's'); + result(s.pss == 1 && s.last == 0x99 && pss(p) == 55, + "4K refault restores coherent three-way sharing"); + result(!ftruncate(fd, 65536 - 4096), + "truncate at4K boundary inside16K leaf"); + s = command(&small, 't'); + result(s.status == 256 + SIGBUS, + "4K mapping wholly beyond EOF faults SIGBUS"); + s = command(&medium, 't'); + result(s.status == 0, "partial16K EOF leaf exposes zero tail"); + result(p[65535] == 0 && p[0] == 0x42, + "native64K sees zero tail and preserved file prefix"); + s = command(&medium, 's'); + result(s.first == 0x42 && s.last == 0 && s.pss == 8 && pss(p) == 56, + "16K EOF refault keeps correct mixed PSS"); + result(!ftruncate(fd, 65536), "restore file length"); + s = command(&small, 's'); + result(s.first == 0 && s.last == 0 && s.pss == 1 && pss(p) == 55, + "4K alias refaults zeroed extension"); + finish(&small); + s = command(&medium, 's'); + result(s.pss == 8 && pss(p) == 56, "16K alias survives4K exit"); + finish(&medium); + result(pss(p) == 64, + "native PSS restored after all alternative aliases exit"); + munmap(p, 65536); + close(fd); +done: + printf("THREE GRANULES %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/iommufd-init.c b/tools/testing/selftests/arm64/user4k/iommufd-init.c new file mode 100644 index 00000000000000..c678f35cf72a46 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/iommufd-init.c @@ -0,0 +1,1314 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable VM PID1: real IOMMUFD ioctls against mock AMDv1 page tables. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../drivers/iommu/iommufd/iommufd_test.h" + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define VA 0x200000000UL +#define IOVA 0x10000000UL +#define ALIAS 0x12000000UL +#define MAP_FLAGS \ + (IOMMU_IOAS_MAP_FIXED_IOVA | IOMMU_IOAS_MAP_READABLE | \ + IOMMU_IOAS_MAP_WRITEABLE) +#define CHECK(c) \ + do { \ + if (!(c)) { \ + printf("FAIL line %d: %s errno=%d\n", __LINE__, #c, \ + errno); \ + return 1; \ + } \ + } while (0) + +static size_t native_ps, user_ps; +struct context { + int fd, access; + uint32_t ioas, hwpt[2], dev[2]; +}; + +static int drop_pin_cap(void) +{ + struct __user_cap_header_struct h = { + .version = _LINUX_CAPABILITY_VERSION_3 + }; + struct __user_cap_data_struct d[2]; + unsigned int mask = 1U << (CAP_IPC_LOCK % 32); + + if (syscall(SYS_capget, &h, d)) + return -1; + d[CAP_IPC_LOCK / 32].effective &= ~mask; + d[CAP_IPC_LOCK / 32].permitted &= ~mask; + return syscall(SYS_capset, &h, d); +} + +static int limit_bytes(size_t bytes) +{ + struct rlimit r; + + if (getrlimit(RLIMIT_MEMLOCK, &r)) + return -1; + r.rlim_cur = bytes; + return setrlimit(RLIMIT_MEMLOCK, &r); +} + +static long stat_kb(pid_t pid, const char *key) +{ + char path[80], line[256]; + long result = -1; + FILE *f; + + snprintf(path, sizeof(path), "/proc/%d/status", pid); + f = fopen(path, "r"); + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (!strncmp(line, key, strlen(key))) { + result = strtol(line + strlen(key), NULL, 10); + break; + } + fclose(f); + return result; +} + +static int new_domain(struct context *c, unsigned int index) +{ + struct iommu_test_cmd cmd = { .size = sizeof(cmd), + .op = IOMMU_TEST_OP_MOCK_DOMAIN, + .id = c->ioas }; + int rc = ioctl(c->fd, IOMMU_TEST_CMD, &cmd); + + if (!rc) { + c->hwpt[index] = cmd.mock_domain.out_hwpt_id; + c->dev[index] = cmd.mock_domain.out_idev_id; + } + return rc; +} + +static int context_open_common(struct context *c, unsigned int mm_account, + bool domain, bool pins) +{ + struct iommu_ioas_alloc a = { .size = sizeof(a) }; + struct iommu_option o = { .size = sizeof(o), + .option_id = IOMMU_OPTION_RLIMIT_MODE, + .op = IOMMU_OPTION_OP_SET, + .val64 = mm_account }; + struct iommu_test_cmd access = { .size = sizeof(access), + .op = IOMMU_TEST_OP_CREATE_ACCESS }; + + memset(c, 0, sizeof(*c)); + c->fd = open("/dev/iommu", O_RDWR); + if (c->fd < 0 || ioctl(c->fd, IOMMU_OPTION, &o) || + ioctl(c->fd, IOMMU_IOAS_ALLOC, &a)) + return -1; + c->ioas = a.out_ioas_id; + if (domain && new_domain(c, 0)) + return -1; + access.id = c->ioas; + access.create_access.flags = + pins ? MOCK_FLAGS_ACCESS_CREATE_NEEDS_PIN_PAGES : 0; + if (ioctl(c->fd, IOMMU_TEST_CMD, &access)) + return -1; + c->access = access.create_access.out_access_fd; + return 0; +} + +static int context_open(struct context *c, unsigned int mm_account) +{ + return context_open_common(c, mm_account, true, false); +} + +static void context_close(struct context *c) +{ + close(c->access); + close(c->fd); +} + +static int map_range(struct context *c, void *p, size_t len, unsigned long iova) +{ + struct iommu_ioas_map m = { .size = sizeof(m), + .flags = MAP_FLAGS, + .ioas_id = c->ioas, + .user_va = (uintptr_t)p, + .length = len, + .iova = iova }; + + return ioctl(c->fd, IOMMU_IOAS_MAP, &m); +} + +static int unmap_range(struct context *c, size_t len, unsigned long iova) +{ + struct iommu_ioas_unmap u = { .size = sizeof(u), + .ioas_id = c->ioas, + .iova = iova, + .length = len }; + int rc = ioctl(c->fd, IOMMU_IOAS_UNMAP, &u); + + return rc ?: (u.length == len ? 0 : -1); +} + +static int check_pa(struct context *c, unsigned int dom, void *p, size_t len, + unsigned long iova) +{ + struct iommu_test_cmd cmd = { .size = sizeof(cmd), + .op = IOMMU_TEST_OP_MD_CHECK_MAP, + .id = c->hwpt[dom], + .check_map = { .iova = iova, + .length = len, + .uptr = (uintptr_t)p } }; + + return ioctl(c->fd, IOMMU_TEST_CMD, &cmd); +} + +static int access_rw(struct context *c, void *p, size_t len, unsigned long iova, + unsigned int flags) +{ + struct iommu_test_cmd cmd = { .size = sizeof(cmd), + .op = IOMMU_TEST_OP_ACCESS_RW, + .id = c->access, + .access_rw = { .iova = iova, + .length = len, + .uptr = (uintptr_t)p, + .flags = flags } }; + + return ioctl(c->fd, IOMMU_TEST_CMD, &cmd); +} + +static bool filled(const void *ptr, unsigned char byte, size_t n) +{ + const unsigned char *p = ptr; + + for (size_t i = 0; i < n; i++) + if (p[i] != byte) + return false; + return true; +} + +static void *relocated(size_t len, unsigned char byte) +{ + void *old = mmap((void *)VA, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, + 0); + + if (old == MAP_FAILED) + return old; + memset(old, byte, len); + size_t distance = len > 32 * native_ps ? len + native_ps : + 32 * native_ps; + + return mremap(old, len, len, MREMAP_MAYMOVE | MREMAP_FIXED, + (void *)(VA + distance + user_ps)); +} + +static int limits_and_lifetime(unsigned int mode) +{ + struct context c; + size_t len = 2 * native_ps; + unsigned char *p = relocated(3 * native_ps, 0x31), *buf = malloc(len); + struct iommu_ioas_copy copy; + + CHECK(p != MAP_FAILED && buf); + CHECK(!context_open(&c, mode)); + CHECK(!limit_bytes(len)); + CHECK(map_range(&c, p, 3 * native_ps, IOVA) == -1 && errno == ENOMEM); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!map_range(&c, p, len, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == (long)len / 1024); + CHECK(stat_kb(getpid(), "VmLck:") == (mode ? (long)len / 1024 : 0)); + CHECK(!check_pa(&c, 0, p, len, IOVA)); + CHECK(map_range(&c, p + len, native_ps, ALIAS) == -1 && + errno == ENOMEM); + CHECK(stat_kb(getpid(), "VmPin:") == (long)len / 1024); + copy = (struct iommu_ioas_copy){ .size = sizeof(copy), + .flags = MAP_FLAGS, + .src_ioas_id = c.ioas, + .dst_ioas_id = c.ioas, + .src_iova = IOVA + user_ps, + .dst_iova = ALIAS, + .length = len - user_ps }; + CHECK(!ioctl(c.fd, IOMMU_IOAS_COPY, ©)); + CHECK(!new_domain(&c, 1)); + CHECK(!check_pa(&c, 1, p + user_ps, len - user_ps, ALIAS)); + CHECK(stat_kb(getpid(), "VmPin:") == (long)len / 1024); + CHECK(mmap(p, len, PROT_READ | PROT_WRITE, + MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0) == p); + memset(p, 0x5a, len); + CHECK(check_pa(&c, 0, p, len, IOVA) == -1 && errno == EINVAL); + CHECK(!access_rw(&c, buf, len, IOVA, MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(filled(buf, 0x31, len)); + CHECK(!access_rw(&c, buf, len, IOVA, 0)); + CHECK(filled(buf, 0x5a, len)); + CHECK(!unmap_range(&c, len, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == + (long)(user_ps < native_ps ? len : len - user_ps) / 1024); + CHECK(!unmap_range(&c, len - user_ps, ALIAS)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + /* Fresh anonymous backing follows the shifted VA's native linear index. + * A non-native-aligned replacement spans three physical backing pages. + */ + size_t replacement_pages = user_ps < native_ps ? 3 : 2; + + if (replacement_pages == 3) { + CHECK(map_range(&c, p, len, IOVA) == -1 && errno == ENOMEM); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + } + CHECK(!limit_bytes(replacement_pages * native_ps)); + CHECK(!map_range(&c, p, len, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == + (long)(replacement_pages * native_ps / 1024)); + CHECK(!unmap_range(&c, len, IOVA)); + CHECK(!munmap(p, 3 * native_ps)); + context_close(&c); + free(buf); + return 0; +} + +static int holes_and_shared(unsigned int mode) +{ + struct context c; + size_t len = 3 * user_ps, file_len = 4 * native_ps; + unsigned char *p = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0), + *buf = malloc(file_len); + int file = memfd_create("iommu-shared", 0); + + CHECK(p != MAP_FAILED && buf && file >= 0); + CHECK(!context_open(&c, mode) && !limit_bytes(8 * native_ps)); + memset(p, 0x31, len); + CHECK(!munmap(p + user_ps, user_ps)); + CHECK(map_range(&c, p, len, IOVA) == -1 && errno == EFAULT); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(mmap(p + user_ps, user_ps, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, + 0) == p + user_ps); + CHECK(map_range(&c, p, len, IOVA) == -1 && errno == EFAULT); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!mprotect(p + user_ps, user_ps, PROT_READ | PROT_WRITE)); + CHECK(!map_range(&c, p, len, IOVA) && !check_pa(&c, 0, p, len, IOVA)); + CHECK(!unmap_range(&c, len, IOVA) && !munmap(p, len)); + CHECK(!ftruncate(file, file_len)); + p = mmap(NULL, 2 * user_ps, PROT_READ | PROT_WRITE, MAP_SHARED, file, + user_ps); + CHECK(p != MAP_FAILED); + CHECK(!map_range(&c, p, 2 * user_ps, IOVA)); + CHECK(!check_pa(&c, 0, p, 2 * user_ps, IOVA)); + CHECK(!munmap(p, 2 * user_ps)); + memset(buf, 0x93, 2 * user_ps); + CHECK(!access_rw(&c, buf, 2 * user_ps, IOVA, + MOCK_ACCESS_RW_WRITE | MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(!unmap_range(&c, 2 * user_ps, IOVA)); + CHECK(pread(file, buf, file_len, 0) == (ssize_t)file_len); + CHECK(filled(buf, 0, user_ps)); + CHECK(filled(buf + user_ps, 0x93, 2 * user_ps)); + CHECK(filled(buf + 3 * user_ps, 0, file_len - 3 * user_ps)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + close(file); + context_close(&c); + free(buf); + return 0; +} + +static int producer(int argc, char **argv) +{ + struct context c = { .fd = atoi(argv[4]), + .ioas = strtoul(argv[5], NULL, 0) }; + int ready = atoi(argv[6]), go = atoi(argv[7]); + char byte = 'R'; + void *p; + + (void)argc; + CHECK(!prctl(PR_SET_PDEATHSIG, SIGKILL)); + CHECK(!drop_pin_cap()); + p = relocated(2 * native_ps, 0x6d); + CHECK(p != MAP_FAILED); + CHECK(!map_range(&c, p, 2 * native_ps, IOVA)); + CHECK(write(ready, &byte, 1) == 1); + CHECK(read(go, &byte, 1) == 1 && byte == 'G'); + return 0; +} + +static int destroy_id(struct context *c, uint32_t id) +{ + struct iommu_destroy cmd = { .size = sizeof(cmd), .id = id }; + + return ioctl(c->fd, IOMMU_DESTROY, &cmd); +} + +static int create_viommu(struct context *c, uint32_t *id, uint32_t *parent) +{ + struct iommu_hwpt_alloc hwpt = { .size = sizeof(hwpt), + .flags = IOMMU_HWPT_ALLOC_NEST_PARENT, + .dev_id = c->dev[0], + .pt_id = c->ioas }; + struct iommu_viommu_alloc cmd = { .size = sizeof(cmd), + .type = IOMMU_VIOMMU_TYPE_SELFTEST, + .dev_id = c->dev[0] }; + + if (ioctl(c->fd, IOMMU_HWPT_ALLOC, &hwpt)) + return -1; + *parent = hwpt.out_hwpt_id; + cmd.hwpt_id = *parent; + if (ioctl(c->fd, IOMMU_VIOMMU_ALLOC, &cmd)) + return -1; + *id = cmd.out_viommu_id; + return 0; +} + +static int queue_alloc(struct context *c, uint32_t viommu, unsigned int index, + unsigned long iova, size_t length, uint32_t *id) +{ + struct iommu_hw_queue_alloc cmd = { .size = sizeof(cmd), + .viommu_id = viommu, + .type = IOMMU_HW_QUEUE_TYPE_SELFTEST, + .index = index, + .nesting_parent_iova = iova, + .length = length }; + int rc = ioctl(c->fd, IOMMU_HW_QUEUE_ALLOC, &cmd); + + if (!rc) + *id = cmd.out_hw_queue_id; + return rc; +} + +static int queue_after_exit(struct context *c) +{ + uint32_t viommu, parent, id; + + CHECK(!create_viommu(c, &viommu, &parent)); + CHECK(!queue_alloc(c, viommu, 0, IOVA + native_ps / 4 + 3, + native_ps / 4, &id)); + CHECK(unmap_range(c, 2 * native_ps, IOVA) == -1 && errno == EBUSY); + CHECK(!destroy_id(c, id)); + CHECK(!destroy_id(c, viommu)); + CHECK(!destroy_id(c, parent)); + return 0; +} + +static int queue_ranges(unsigned int mode) +{ + struct context c; + unsigned char *p = relocated(2 * native_ps, 0x71); + uint32_t viommu, parent, q0, q1, unused; + size_t offset = user_ps / 4 + 3, length = user_ps / 2 + 17; + int file; + + CHECK(p != MAP_FAILED && !context_open(&c, mode)); + CHECK(!limit_bytes(8 * native_ps) && + !create_viommu(&c, &viommu, &parent)); + /* Map only one user leaf: queue pinning must not require surrounding IOVA. */ + CHECK(!map_range(&c, p, user_ps, IOVA)); + CHECK(!check_pa(&c, 0, p, user_ps, IOVA)); + CHECK(queue_alloc(&c, viommu, 1, IOVA + offset, length, &unused) == + -1 && + errno == EIO); + CHECK(!queue_alloc(&c, viommu, 0, IOVA + offset, length, &q0)); + CHECK(queue_alloc(&c, viommu, 0, IOVA + offset, length, &unused) == + -1 && + errno == EEXIST); + CHECK(unmap_range(&c, user_ps, IOVA) == -1 && errno == EBUSY); + CHECK(stat_kb(getpid(), "VmPin:") == (long)native_ps / 1024); + /* Pins keep the old physical bytes after replacing the source VMA. */ + CHECK(mmap(p, user_ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) == p); + memset(p, 0x5a, user_ps); + CHECK(!queue_alloc(&c, viommu, 1, IOVA + offset + 7, length - 7, &q1)); + CHECK(destroy_id(&c, q0) == -1 && errno == EBUSY); + CHECK(!destroy_id(&c, q1)); + CHECK(unmap_range(&c, user_ps, IOVA) == -1 && errno == EBUSY); + CHECK(!destroy_id(&c, q0)); + CHECK(!unmap_range(&c, user_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(queue_alloc(&c, viommu, 0, IOVA + offset, length, &unused) == + -1 && + errno == ENOENT); + CHECK(!munmap(p, 2 * native_ps)); + + /* Separate mappings of the same file byte range cannot form a physical run. + * The second area must fail after the first area's access was registered. + */ + file = memfd_create("queue-alias", 0); + CHECK(file >= 0 && !ftruncate(file, native_ps)); + p = mmap((void *)VA, 2 * user_ps, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(p != MAP_FAILED); + CHECK(mmap(p, user_ps, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, + file, 0) == p); + CHECK(mmap(p + user_ps, user_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED, file, 0) == p + user_ps); + CHECK(!map_range(&c, p, user_ps, IOVA)); + CHECK(queue_alloc(&c, viommu, 0, IOVA, 2 * user_ps, &unused) == -1 && + errno == ENOENT); + CHECK(!map_range(&c, p + user_ps, user_ps, IOVA + user_ps)); + CHECK(queue_alloc(&c, viommu, 0, IOVA, 2 * user_ps, &unused) == -1 && + errno == EFAULT); + CHECK(!unmap_range(&c, user_ps, IOVA)); + CHECK(!unmap_range(&c, user_ps, IOVA + user_ps)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + /* The same discontinuity inside one provider must also release its access. */ + CHECK(!map_range(&c, p, 2 * user_ps, IOVA)); + CHECK(queue_alloc(&c, viommu, 0, IOVA, 2 * user_ps, &unused) == -1 && + errno == EFAULT); + CHECK(!unmap_range(&c, 2 * user_ps, IOVA)); + if (user_ps < native_ps) { + /* Adjacent file quarters form a real physical run across two areas. */ + CHECK(mmap(p + user_ps, user_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED, file, + user_ps) == p + user_ps); + CHECK(!map_range(&c, p, user_ps, IOVA)); + CHECK(!map_range(&c, p + user_ps, user_ps, IOVA + user_ps)); + CHECK(!queue_alloc(&c, viommu, 0, IOVA + 11, 2 * user_ps - 22, + &q0)); + CHECK(unmap_range(&c, user_ps, IOVA) == -1 && errno == EBUSY); + CHECK(unmap_range(&c, user_ps, IOVA + user_ps) == -1 && + errno == EBUSY); + CHECK(!destroy_id(&c, q0)); + CHECK(!unmap_range(&c, user_ps, IOVA)); + CHECK(!unmap_range(&c, user_ps, IOVA + user_ps)); + } + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!munmap(p, 2 * user_ps)); + close(file); + CHECK(!destroy_id(&c, viommu) && !destroy_id(&c, parent)); + context_close(&c); + printf("ok - hardware queue byte spans %zuK/%zuK %s, " + "relocated/overlap/rollback/unmap locks\n", + user_ps / 1024, native_ps / 1024, mode ? "MM" : "USER"); + return 0; +} + +static int cross_process(unsigned int mode) +{ + struct context c; + int ready[2], go[2], status; + pid_t child; + size_t child_ps = user_ps == native_ps ? 4096 : native_ps; + size_t len = 2 * native_ps; + unsigned char *p = mmap((void *)(VA + 128 * native_ps), len, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | + MAP_FIXED_NOREPLACE, + -1, 0), + *buf = malloc(len); + char byte; + + CHECK(p != MAP_FAILED && buf && !pipe(ready) && !pipe(go)); + CHECK(!context_open(&c, mode) && !limit_bytes(len)); + child = fork(); + CHECK(child >= 0); + if (!child) { + char ps[24], native[24], fd[24], ioas[24], r[24], g[24]; + + close(ready[0]); + close(go[1]); + snprintf(ps, sizeof(ps), "%zu", child_ps); + snprintf(native, sizeof(native), "%zu", native_ps); + snprintf(fd, sizeof(fd), "%d", c.fd); + snprintf(ioas, sizeof(ioas), "%u", c.ioas); + snprintf(r, sizeof(r), "%d", ready[1]); + snprintf(g, sizeof(g), "%d", go[0]); + if (prctl(SET_EXEC_PAGE_SIZE, child_ps, 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--producer", ps, native, fd, ioas, r, + g, NULL); + _exit(5); + } + close(ready[1]); + close(go[0]); + CHECK(read(ready[0], &byte, 1) == 1 && byte == 'R'); + CHECK(stat_kb(child, "VmPin:") == (long)len / 1024); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + if (!mode) + CHECK(map_range(&c, p, native_ps, ALIAS) == -1 && + errno == ENOMEM); + else { + CHECK(!map_range(&c, p, native_ps, ALIAS)); + CHECK(!unmap_range(&c, native_ps, ALIAS)); + } + CHECK(!access_rw(&c, buf, len, IOVA, MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(filled(buf, 0x6d, len)); + byte = 'G'; + CHECK(write(go[1], &byte, 1) == 1); + CHECK(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(!new_domain(&c, 1)); + CHECK(!access_rw(&c, buf, len, IOVA, MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(filled(buf, 0x6d, len)); + CHECK(!queue_after_exit(&c)); + if (!mode) + CHECK(map_range(&c, p, native_ps, ALIAS) == -1 && + errno == ENOMEM); + CHECK(!unmap_range(&c, len, IOVA)); + CHECK(!map_range(&c, p, len, ALIAS) && !unmap_range(&c, len, ALIAS)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + close(ready[0]); + close(go[1]); + CHECK(!munmap(p, len)); + context_close(&c); + free(buf); + return 0; +} + +struct stress_arg { + struct context *c; + void *p; + unsigned int index; + atomic_int *failed; +}; +static void *stress_thread(void *opaque) +{ + struct stress_arg *a = opaque; + unsigned long iova = IOVA + a->index * 0x100000UL; + size_t len = 2 * native_ps; + + for (unsigned int round = 0; round < 64; round++) { + if (map_range(a->c, a->p, len, iova) || + (!(round % 16) && check_pa(a->c, 1, a->p, len, iova)) || + unmap_range(a->c, len, iova)) { + printf("FAIL concurrent map thread=%u round=%u errno=%d\n", + a->index, round, errno); + atomic_store(a->failed, 1); + break; + } + } + return NULL; +} + +static int concurrent_maps(unsigned int mode) +{ + struct context c; + pthread_t threads[4]; + struct stress_arg args[4]; + atomic_int failed = 0; + void *p = relocated(2 * native_ps, 0x47); + + CHECK(p != MAP_FAILED && !context_open(&c, mode)); + CHECK(!limit_bytes(32 * native_ps) && !new_domain(&c, 1)); + for (unsigned int i = 0; i < 4; i++) { + args[i] = (struct stress_arg){ &c, p, i, &failed }; + CHECK(!pthread_create(&threads[i], NULL, stress_thread, + &args[i])); + } + for (unsigned int i = 0; i < 4; i++) + CHECK(!pthread_join(threads[i], NULL)); + CHECK(!atomic_load(&failed)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!munmap(p, 2 * native_ps)); + context_close(&c); + return 0; +} + +static int write_attribute(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY); + ssize_t n; + + if (fd < 0) + return -1; + n = write(fd, value, strlen(value)); + close(fd); + return n == (ssize_t)strlen(value) ? 0 : -1; +} + +static int allocation_failures(unsigned int mode) +{ + struct context c; + void *p = relocated(2 * native_ps, 0x62); + int nth = open("/proc/self/fail-nth", O_RDWR); + unsigned int hit = 0, n; + + CHECK(nth >= 0 && p != MAP_FAILED); + CHECK(!context_open(&c, mode) && !limit_bytes(8 * native_ps)); + CHECK(!new_domain(&c, 1)); + CHECK(!write_attribute("/sys/kernel/debug/failslab/ignore-gfp-wait", + "N")); + CHECK(!write_attribute( + "/sys/kernel/debug/fail_page_alloc/ignore-gfp-wait", "N")); + CHECK(!write_attribute( + "/sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem", "N")); + CHECK(!write_attribute("/sys/kernel/debug/failslab/verbose", "0")); + CHECK(!write_attribute("/sys/kernel/debug/fail_page_alloc/verbose", + "0")); + CHECK(!write_attribute("/sys/kernel/debug/fail_iommufd/verbose", "0")); + for (n = 1; n <= 512; n++) { + char number[24], remaining[24] = {}; + int bytes = snprintf(number, sizeof(number), "%u", n); + ssize_t got, disabled; + int rc; + + CHECK(pwrite(nth, number, bytes, 0) == bytes); + rc = map_range(&c, p, 2 * native_ps, IOVA); + got = pread(nth, remaining, sizeof(remaining) - 1, 0); + if (got < 0 && errno == EFAULT) { + strcpy(remaining, "1"); + got = 1; + } + disabled = pwrite(nth, "0", 1, 0); + if (disabled < 0 && errno == EFAULT) { + disabled = pwrite(nth, "0", 1, 0); + strcpy(remaining, "1"); + } + CHECK(disabled == 1 && got > 0); + if (!rc) + CHECK(!unmap_range(&c, 2 * native_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + /* An injected failure must not poison a successful retry or translation. */ + CHECK(!map_range(&c, p, 2 * native_ps, IOVA)); + CHECK(!check_pa(&c, 0, p, 2 * native_ps, IOVA)); + CHECK(!check_pa(&c, 1, p, 2 * native_ps, IOVA)); + CHECK(!unmap_range(&c, 2 * native_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + if (strtoul(remaining, NULL, 0)) + break; + hit++; + } + CHECK(hit > 0 && n <= 512); + printf("ok - IOMMUFD %zuK %s accounting allocation-failure sweep: %u sites\n", + user_ps / 1024, mode ? "MM" : "USER", hit); + CHECK(!munmap(p, 2 * native_ps)); + close(nth); + context_close(&c); + return 0; +} + +static int queue_failure_sweep(unsigned int mode) +{ + struct context c; + void *p = relocated(2 * native_ps, 0x4c); + int nth = open("/proc/self/fail-nth", O_RDWR); + uint32_t viommu, parent, id; + unsigned int n, hit = 0; + + CHECK(p != MAP_FAILED && nth >= 0 && !context_open(&c, mode)); + CHECK(!limit_bytes(8 * native_ps) && + !create_viommu(&c, &viommu, &parent)); + for (n = 1; n <= 512; n++) { + char number[24], remaining[24] = {}; + int bytes = snprintf(number, sizeof(number), "%u", n), rc; + ssize_t got, disabled; + + CHECK(!map_range(&c, p, user_ps, IOVA)); + CHECK(pwrite(nth, number, bytes, 0) == bytes); + rc = queue_alloc(&c, viommu, 0, IOVA + user_ps / 4 + 3, + user_ps / 2, &id); + got = pread(nth, remaining, sizeof(remaining) - 1, 0); + if (got < 0 && errno == EFAULT) { + strcpy(remaining, "1"); + got = 1; + } + disabled = pwrite(nth, "0", 1, 0); + if (disabled < 0 && errno == EFAULT) { + disabled = pwrite(nth, "0", 1, 0); + strcpy(remaining, "1"); + } + CHECK(got > 0 && disabled == 1); + if (!rc) + CHECK(!destroy_id(&c, id)); + /* A failed queue must not leave an internal access blocking unmap. */ + CHECK(!unmap_range(&c, user_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!map_range(&c, p, user_ps, IOVA)); + CHECK(!queue_alloc(&c, viommu, 0, IOVA + user_ps / 4 + 3, + user_ps / 2, &id)); + CHECK(!destroy_id(&c, id) && !unmap_range(&c, user_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + if (strtoul(remaining, NULL, 0)) + break; + hit++; + } + CHECK(hit > 0 && n <= 512); + CHECK(!destroy_id(&c, viommu) && !destroy_id(&c, parent)); + context_close(&c); + close(nth); + CHECK(!munmap(p, 2 * native_ps)); + printf("ok - hardware queue %zuK %s failure sweep: %u sites, no stale locks/pins\n", + user_ps / 1024, mode ? "MM" : "USER", hit); + return 0; +} + +static int page_access(struct context *c, void *p, size_t len, + unsigned long iova, uint32_t *id) +{ + struct iommu_test_cmd cmd = { + .size = sizeof(cmd), + .op = IOMMU_TEST_OP_ACCESS_PAGES, + .id = c->access, + .access_pages = { .iova = iova, + .length = len, + .uptr = (uintptr_t)p, + .flags = MOCK_FLAGS_ACCESS_WRITE } + }; + int rc = ioctl(c->fd, IOMMU_TEST_CMD, &cmd); + + if (!rc) + *id = cmd.access_pages.out_access_pages_id; + return rc; +} + +static int page_release(struct context *c, uint32_t id) +{ + struct iommu_test_cmd cmd = { .size = sizeof(cmd), + .op = IOMMU_TEST_OP_DESTROY_ACCESS_PAGES, + .id = c->access, + .destroy_access_pages = { + .access_pages_id = id } }; + + return ioctl(c->fd, IOMMU_TEST_CMD, &cmd); +} + +static int page_ranges(unsigned int mode) +{ + struct context c; + int file = memfd_create("native-page-array", 0); + unsigned char *p; + uint32_t a, b, d, e; + size_t len = 3 * native_ps, tail = native_ps + user_ps / 2 + 3; + + CHECK(file >= 0 && !ftruncate(file, 5 * native_ps)); + p = mmap((void *)VA, len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, native_ps); + CHECK(p != MAP_FAILED && !limit_bytes(4 * native_ps)); + memset(p, 0x67, len); + CHECK(!context_open_common(&c, mode, false, true)); + /* Separate areas exercise native output stride with alternative indices. */ + CHECK(!map_range(&c, p, native_ps, IOVA)); + CHECK(!map_range(&c, p + native_ps, 2 * native_ps, IOVA + native_ps)); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!page_access(&c, p, len, IOVA, &a)); + CHECK(stat_kb(getpid(), "VmPin:") == (long)len / 1024); + CHECK(!page_access(&c, p, len, IOVA, &b)); + CHECK(!page_access(&c, p + native_ps, tail, IOVA + native_ps, &d)); + CHECK(stat_kb(getpid(), "VmPin:") == (long)len / 1024); + /* Source VA can move without changing its retained physical offset. */ + unsigned char *moved = mremap(p, len, len, + MREMAP_MAYMOVE | MREMAP_FIXED, + (void *)(VA + 32 * native_ps + user_ps)); + CHECK(moved != MAP_FAILED); + CHECK(!page_access(&c, moved, len, IOVA, &e) && !page_release(&c, e)); + CHECK(!page_release(&c, a) && !page_release(&c, b)); + CHECK(stat_kb(getpid(), "VmPin:") == (long)(2 * native_ps) / 1024); + CHECK(!page_release(&c, d)); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(mremap(moved, len, len, MREMAP_MAYMOVE | MREMAP_FIXED, p) == p); + CHECK(!page_access(&c, p, len, IOVA, &a)); + /* Unmap callback releases an access covering both independent areas. */ + CHECK(!unmap_range(&c, len, IOVA)); + CHECK(page_release(&c, a) == -1 && errno == ENOENT); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!munmap(p, len)); + + /* The second area is absent: first area's pin/accounting must roll back. */ + p = mmap((void *)VA, native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, native_ps); + CHECK(p != MAP_FAILED && !map_range(&c, p, native_ps, IOVA)); + CHECK(page_access(&c, NULL, 2 * native_ps, IOVA, &a) == -1 && + errno == ENOENT); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!page_access(&c, p, native_ps, IOVA, &a) && !page_release(&c, a)); + CHECK(!unmap_range(&c, native_ps, IOVA) && !munmap(p, native_ps)); + + p = mmap((void *)VA, 2 * native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, native_ps); + CHECK(p != MAP_FAILED); + CHECK(mmap(p + native_ps, native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED, file, native_ps) == p + native_ps); + CHECK(!map_range(&c, p, 2 * native_ps, IOVA)); + CHECK(!page_access(&c, p, 2 * native_ps, IOVA, &a)); + CHECK(stat_kb(getpid(), "VmPin:") == + (long)((user_ps == native_ps ? 2 : 1) * native_ps) / 1024); + CHECK(!page_release(&c, a) && !unmap_range(&c, 2 * native_ps, IOVA)); + CHECK(!munmap(p, 2 * native_ps)); + + if (user_ps < native_ps) { + /* Native-sized logical range whose physical base has a fragment offset. */ + p = mmap((void *)VA, native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, user_ps); + CHECK(p != MAP_FAILED && !map_range(&c, p, native_ps, IOVA)); + CHECK(page_access(&c, p, native_ps, IOVA, &a) == -1 && + errno == EINVAL); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!unmap_range(&c, native_ps, IOVA) && + !munmap(p, native_ps)); + + /* Aligned first fragment followed by an alias, not consecutive bytes. */ + p = mmap((void *)VA, 2 * native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, 0); + CHECK(p != MAP_FAILED); + CHECK(mmap(p + native_ps + user_ps, user_ps, + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, file, + native_ps) == p + native_ps + user_ps); + CHECK(!map_range(&c, p, native_ps, IOVA)); + CHECK(!map_range(&c, p + native_ps, native_ps, + IOVA + native_ps)); + CHECK(page_access(&c, p, 2 * native_ps, IOVA, &a) == -1 && + errno == EINVAL); + CHECK(stat_kb(getpid(), "VmPin:") == 0); + CHECK(!page_access(&c, p, native_ps, IOVA, &a)); + CHECK(!unmap_range(&c, 2 * native_ps, IOVA)); + CHECK(page_release(&c, a) == -1 && errno == ENOENT); + CHECK(!munmap(p, 2 * native_ps)); + } + context_close(&c); + close(file); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + printf("ok - native page arrays %zuK/%zuK %s, lazy/partial/moved/rollback/callback\n", + user_ps / 1024, native_ps / 1024, mode ? "MM" : "USER"); + return 0; +} + +static int page_lifetime(unsigned int mode) +{ + struct context c; + int ready[2], go[2], status; + pid_t child; + size_t child_ps = user_ps == native_ps ? 4096 : native_ps; + size_t len = 2 * native_ps; + unsigned char *p = mmap((void *)(VA + 128 * native_ps), len, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | + MAP_FIXED_NOREPLACE, + -1, 0), + *buf = malloc(len); + uint32_t a, b; + char byte; + + CHECK(p != MAP_FAILED && buf && !pipe(ready) && !pipe(go)); + memset(p, 0x42, len); + CHECK(!context_open_common(&c, mode, false, true) && !limit_bytes(len)); + child = fork(); + CHECK(child >= 0); + if (!child) { + char ps[24], native[24], fd[24], ioas[24], r[24], g[24]; + + close(ready[0]); + close(go[1]); + snprintf(ps, sizeof(ps), "%zu", child_ps); + snprintf(native, sizeof(native), "%zu", native_ps); + snprintf(fd, sizeof(fd), "%d", c.fd); + snprintf(ioas, sizeof(ioas), "%u", c.ioas); + snprintf(r, sizeof(r), "%d", ready[1]); + snprintf(g, sizeof(g), "%d", go[0]); + if (prctl(SET_EXEC_PAGE_SIZE, child_ps, 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--producer", ps, native, fd, ioas, r, + g, NULL); + _exit(5); + } + close(ready[1]); + close(go[0]); + CHECK(read(ready[0], &byte, 1) == 1 && byte == 'R'); + CHECK(stat_kb(child, "VmPin:") == 0); + CHECK(!page_access(&c, NULL, len, IOVA, &a)); + CHECK(stat_kb(child, "VmPin:") == (long)len / 1024); + CHECK(!access_rw(&c, buf, len, IOVA, MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(filled(buf, 0x6d, len)); + byte = 'G'; + CHECK(write(go[1], &byte, 1) == 1); + CHECK(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status)); + /* Reuse retained anonymous fragments after source mm_users reaches zero. */ + CHECK(!page_access(&c, NULL, native_ps, IOVA, &b)); + CHECK(!page_release(&c, a)); + CHECK(!access_rw(&c, buf, native_ps, IOVA, MOCK_ACCESS_RW_SLOW_PATH)); + CHECK(filled(buf, 0x6d, native_ps)); + CHECK(!map_range(&c, p, len, ALIAS)); + if (!mode) + CHECK(page_access(&c, p, len, ALIAS, &a) == -1 && + errno == ENOMEM); + else + CHECK(!page_access(&c, p, len, ALIAS, &a) && + !page_release(&c, a)); + /* Final access release refunds even a dead provider's account. */ + CHECK(!page_release(&c, b) && !unmap_range(&c, len, IOVA)); + CHECK(!page_access(&c, p, len, ALIAS, &a) && !page_release(&c, a)); + CHECK(!unmap_range(&c, len, ALIAS)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!munmap(p, len)); + context_close(&c); + close(ready[0]); + close(go[1]); + free(buf); + printf("ok - native page array anonymous source %zuK -> %zuK consumer, %s exit/refund\n", + child_ps / 1024, user_ps / 1024, mode ? "MM" : "USER"); + return 0; +} + +static int page_failure_sweep(unsigned int mode) +{ + struct context c; + int file = memfd_create("page-array-failures", 0); + int nth = open("/proc/self/fail-nth", O_RDWR); + unsigned char *p; + uint32_t id; + unsigned int n, hit = 0; + + CHECK(file >= 0 && nth >= 0 && !ftruncate(file, 2 * native_ps)); + p = mmap((void *)VA, 2 * native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, 0); + CHECK(p != MAP_FAILED); + memset(p, 0x53, 2 * native_ps); + CHECK(!context_open_common(&c, mode, false, true) && + !limit_bytes(2 * native_ps)); + CHECK(!map_range(&c, p, native_ps, IOVA)); + CHECK(!map_range(&c, p + native_ps, native_ps, IOVA + native_ps)); + for (n = 1; n <= 512; n++) { + char number[24], remaining[24] = {}; + int bytes = snprintf(number, sizeof(number), "%u", n), rc; + ssize_t got, disabled; + + CHECK(pwrite(nth, number, bytes, 0) == bytes); + rc = page_access(&c, p, 2 * native_ps, IOVA, &id); + got = pread(nth, remaining, sizeof(remaining) - 1, 0); + if (got < 0 && errno == EFAULT) { + strcpy(remaining, "1"); + got = 1; + } + disabled = pwrite(nth, "0", 1, 0); + if (disabled < 0 && errno == EFAULT) { + disabled = pwrite(nth, "0", 1, 0); + strcpy(remaining, "1"); + } + CHECK(got > 0 && disabled == 1); + if (!rc) + CHECK(!page_release(&c, id)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!page_access(&c, p, 2 * native_ps, IOVA, &id) && + !page_release(&c, id)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + if (strtoul(remaining, NULL, 0)) + break; + hit++; + } + CHECK(hit > 0 && n <= 512); + CHECK(!unmap_range(&c, 2 * native_ps, IOVA) && + !munmap(p, 2 * native_ps)); + context_close(&c); + close(file); + close(nth); + printf("ok - native page arrays %zuK %s failure sweep: %u sites, refunds/retries\n", + user_ps / 1024, mode ? "MM" : "USER", hit); + return 0; +} + +struct page_race { + struct context *c; + void *p; + pthread_barrier_t barrier; + atomic_int failed, pinned, absent; +}; + +static void *page_pin_thread(void *opaque) +{ + struct page_race *r = opaque; + + pthread_barrier_wait(&r->barrier); + for (unsigned int i = 0; i < 256; i++) { + uint32_t id; + + if (page_access(r->c, r->p, 2 * native_ps, IOVA, &id)) { + /* Unmap can mark the area inaccessible before notifying drivers. */ + if (errno != ENOENT && errno != EINVAL) + goto fail; + atomic_fetch_add(&r->absent, 1); + } else { + atomic_fetch_add(&r->pinned, 1); + sched_yield(); + if (page_release(r->c, id) && errno != ENOENT) + goto fail; + } + } + return NULL; +fail: + printf("FAIL page pin/unmap race errno=%d\n", errno); + atomic_store(&r->failed, 1); + return NULL; +} + +static void *page_unmap_thread(void *opaque) +{ + struct page_race *r = opaque; + + pthread_barrier_wait(&r->barrier); + for (unsigned int i = 0; i < 256; i++) { + if (unmap_range(r->c, 2 * native_ps, IOVA)) + goto fail; + sched_yield(); + if (map_range(r->c, r->p, 2 * native_ps, IOVA)) + goto fail; + } + return NULL; +fail: + printf("FAIL page unmap/remap race errno=%d\n", errno); + atomic_store(&r->failed, 1); + return NULL; +} + +static int page_concurrency(unsigned int mode) +{ + struct context c; + struct page_race r = { .c = &c }; + pthread_t threads[4]; + int file = memfd_create("page-array-race", 0); + + CHECK(file >= 0 && !ftruncate(file, 2 * native_ps)); + r.p = mmap((void *)VA, 2 * native_ps, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, 0); + CHECK(r.p != MAP_FAILED); + memset(r.p, 0x49, 2 * native_ps); + CHECK(!context_open_common(&c, mode, false, true) && + !limit_bytes(8 * native_ps)); + CHECK(!map_range(&c, r.p, 2 * native_ps, IOVA)); + CHECK(!pthread_barrier_init(&r.barrier, NULL, 4)); + for (unsigned int i = 0; i < 4; i++) + CHECK(!pthread_create( + &threads[i], NULL, + i == 3 ? page_unmap_thread : page_pin_thread, &r)); + for (unsigned int i = 0; i < 4; i++) + CHECK(!pthread_join(threads[i], NULL)); + CHECK(!atomic_load(&r.failed) && atomic_load(&r.pinned) > 0); + CHECK(!unmap_range(&c, 2 * native_ps, IOVA)); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!munmap(r.p, 2 * native_ps)); + context_close(&c); + close(file); + CHECK(!pthread_barrier_destroy(&r.barrier)); + printf("ok - native page arrays %zuK %s race: %d pins, %d unavailable, 256 remaps\n", + user_ps / 1024, mode ? "MM" : "USER", atomic_load(&r.pinned), + atomic_load(&r.absent)); + return 0; +} + +static int dirty_ioctl(struct context *c, uint32_t hwpt, void *bitmap, + unsigned long iova, size_t length, size_t granule, + unsigned int flags) +{ + struct iommu_hwpt_get_dirty_bitmap cmd = { .size = sizeof(cmd), + .hwpt_id = hwpt, + .flags = flags, + .iova = iova, + .length = length, + .page_size = granule, + .data = (uintptr_t)bitmap }; + + return ioctl(c->fd, IOMMU_HWPT_GET_DIRTY_BITMAP, &cmd); +} + +static int dirty_errors(unsigned int mode) +{ + struct context c; + size_t len = 66 * native_ps; + unsigned char *data = relocated(len, 0x34); + unsigned char *out = mmap(NULL, 2 * user_ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + uint64_t *bitmap = (void *)(out + user_ps - 8), seed[2] = { 1, 2 }; + struct iommu_option option = { .size = sizeof(option), + .option_id = IOMMU_OPTION_HUGE_PAGES, + .op = IOMMU_OPTION_OP_SET, + .val64 = 0 }; + struct iommu_hwpt_alloc alloc = { + .size = sizeof(alloc), .flags = IOMMU_HWPT_ALLOC_DIRTY_TRACKING + }; + struct iommu_hwpt_set_dirty_tracking enable = { + .size = sizeof(enable), + .flags = IOMMU_HWPT_DIRTY_TRACKING_ENABLE + }; + struct iommu_test_cmd mark = { .size = sizeof(mark), + .op = IOMMU_TEST_OP_DIRTY, + .dirty = { .iova = IOVA, + .length = len, + .page_size = native_ps, + .uptr = (uintptr_t)seed } }; + unsigned int no_clear = IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR; + int failures = 0, rc, error; + + CHECK(data != MAP_FAILED && out != MAP_FAILED); + CHECK(!context_open(&c, mode) && !limit_bytes(128 * native_ps)); + option.object_id = c.ioas; + CHECK(!ioctl(c.fd, IOMMU_OPTION, &option)); + CHECK(!map_range(&c, data, len, IOVA)); + alloc.dev_id = c.dev[0]; + alloc.pt_id = c.ioas; + CHECK(!ioctl(c.fd, IOMMU_HWPT_ALLOC, &alloc)); + enable.hwpt_id = alloc.out_hwpt_id; + CHECK(!ioctl(c.fd, IOMMU_HWPT_SET_DIRTY_TRACKING, &enable)); + mark.id = alloc.out_hwpt_id; + CHECK(!ioctl(c.fd, IOMMU_TEST_CMD, &mark) && + mark.dirty.out_nr_dirty == 2); + CHECK(!dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, len, native_ps, + no_clear)); + CHECK(bitmap[0] == 1 && bitmap[1] == 2); + CHECK(filled(out, 0, user_ps - 8) && + filled(out + user_ps + 8, 0, user_ps - 8)); + +#define EXPECT_DIRTY_ERROR(call, expected, label) \ + do { \ + errno = 0; \ + rc = (call); \ + error = errno; \ + if (rc != -1 || error != (expected)) { \ + printf("FAIL dirty %s: rc=%d errno=%d expected=%d\n", \ + label, rc, error, expected); \ + failures++; \ + } \ + } while (0) + EXPECT_DIRTY_ERROR(dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, + IOVA + len, native_ps, native_ps, + no_clear), + EINVAL, "unmapped IOVA"); + EXPECT_DIRTY_ERROR(dirty_ioctl(&c, alloc.out_hwpt_id, (void *)1, IOVA, + len, native_ps, no_clear), + EFAULT, "invalid output pointer"); + EXPECT_DIRTY_ERROR(dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, + 4 * native_ps, 3 * native_ps, no_clear), + EINVAL, "non-power-of-two granule"); + CHECK(!mprotect(out, 2 * user_ps, PROT_READ)); + EXPECT_DIRTY_ERROR(dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, len, + native_ps, no_clear), + EFAULT, "read-only output"); + CHECK(!mprotect(out, 2 * user_ps, PROT_READ | PROT_WRITE)); + memset(out, 0, 2 * user_ps); + CHECK(!munmap(out + user_ps, user_ps)); + EXPECT_DIRTY_ERROR(dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, len, + native_ps, no_clear), + EFAULT, "output hole after valid prefix"); + CHECK(bitmap[0] == 1); + CHECK(mmap(out + user_ps, user_ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, + 0) == out + user_ps); + /* NO_CLEAR errors retain dirty state, so fixing the buffer permits retry. */ + CHECK(!dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, len, native_ps, + 0)); + CHECK(bitmap[0] == 1 && bitmap[1] == 2); + memset(out, 0, 2 * user_ps); + CHECK(!dirty_ioctl(&c, alloc.out_hwpt_id, bitmap, IOVA, len, native_ps, + no_clear)); + CHECK(bitmap[0] == 0 && bitmap[1] == 0); + CHECK(!unmap_range(&c, len, IOVA)); + CHECK(!munmap(data, len) && !munmap(out, 2 * user_ps)); + context_close(&c); + CHECK(stat_kb(getpid(), "VmPin:") == 0 && + stat_kb(getpid(), "VmLck:") == 0); + CHECK(!failures); +#undef EXPECT_DIRTY_ERROR + return 0; +} + +static int exercise(void) +{ + CHECK(!drop_pin_cap()); + for (unsigned int mode = 0; mode < 2; mode++) { + CHECK(!limits_and_lifetime(mode)); + CHECK(!holes_and_shared(mode)); + CHECK(!queue_ranges(mode)); + CHECK(!page_ranges(mode)); + CHECK(!page_lifetime(mode)); + CHECK(!cross_process(mode)); + CHECK(!concurrent_maps(mode)); + CHECK(!allocation_failures(mode)); + CHECK(!queue_failure_sweep(mode)); + CHECK(!page_failure_sweep(mode)); + CHECK(!page_concurrency(mode)); + CHECK(!dirty_errors(mode)); + printf("ok - IOMMUFD %zuK userspace/%zuK kernel %s accounting, limits/lifetime/shared/cross-process/256 concurrent cycles\n", + user_ps / 1024, native_ps / 1024, mode ? "MM" : "USER"); + } + return 0; +} + +int main(int argc, char **argv) +{ + int fail = 0; + + setbuf(stdout, NULL); + if (argc >= 4) { + user_ps = strtoul(argv[2], NULL, 0); + native_ps = strtoul(argv[3], NULL, 0); + CHECK(getauxval(AT_PAGESZ) == user_ps); + if (!strcmp(argv[1], "--exercise")) + return exercise(); + if (!strcmp(argv[1], "--producer") && argc == 8) + return producer(argc, argv); + return 2; + } + CHECK(getpid() == 1); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + CHECK(!mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)); + CHECK(!mount("sysfs", "/sys", "sysfs", 0, NULL)); + CHECK(!mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)); + native_ps = getauxval(AT_PAGESZ); + size_t sizes[] = { native_ps, 4096, 16384 }; + + for (unsigned int i = 0; i < (native_ps > 16384 ? 3U : 2U); i++) { + pid_t child = fork(); + int status = 0, ok = 0; + + if (!child) { + char size[24], native[24]; + + snprintf(size, sizeof(size), "%zu", sizes[i]); + snprintf(native, sizeof(native), "%zu", native_ps); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", size, native, + NULL); + _exit(5); + } + for (unsigned int ticks = 0; child > 0 && ticks < 1800; + ticks++) { + if (waitpid(child, &status, WNOHANG) == child) { + ok = WIFEXITED(status) && !WEXITSTATUS(status); + break; + } + if (ticks == 1799) { + kill(child, SIGKILL); + waitpid(child, &status, 0); + } + usleep(100000); + } + printf("%s - IOMMUFD ABI %zuK status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + fail += !ok; + } + printf("IOMMUFD IOCTL CONTRACT %s\n", fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/lazyfree-fixture-init.c b/tools/testing/selftests/arm64/user4k/lazyfree-fixture-init.c new file mode 100644 index 00000000000000..b652200c38b42c --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/lazyfree-fixture-init.c @@ -0,0 +1,274 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static unsigned long stat_value(const char *path, const char *key) +{ + char line[256], name[100]; + unsigned long value = 0; + FILE *f = fopen(path, "re"); + + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "%99s %lu", name, &value) == 2 && + !strcmp(name, key)) + break; + fclose(f); + return value; +} + +static bool all_bytes(const unsigned char *p, size_t len, unsigned char value) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != value) + return false; + return true; +} + +static unsigned long lazy_kb(void *address) +{ + FILE *f = fopen("/proc/self/smaps", "re"); + char line[256]; + unsigned long first, last, value = ~0UL; + bool found = false; + + if (!f) + return value; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%lx-%lx", &first, &last) == 2) { + if (found) + break; + found = (uintptr_t)address >= first && + (uintptr_t)address < last; + } else if (found && + sscanf(line, "LazyFree: %lu", &value) == 1) { + break; + } + } + fclose(f); + return value; +} + +static uint64_t pagemap(void *p, size_t ps) +{ + uint64_t entry = 0; + int fd = open("/proc/self/pagemap", O_RDONLY); + + if (fd < 0 || + pread(fd, &entry, sizeof(entry), (uintptr_t)p / ps * 8) != 8) + ksft_exit_fail_msg("pagemap: %s\n", strerror(errno)); + close(fd); + return entry; +} + +static bool pageout(void *p, size_t len) +{ + cpu_set_t allowed, one; + bool ok = true; + + if (sched_getaffinity(0, sizeof(allowed), &allowed)) + return false; + for (unsigned int i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed)) + continue; + CPU_ZERO(&one); + CPU_SET(i, &one); + ok &= !sched_setaffinity(0, sizeof(one), &one); + ok &= !madvise(p, len, MADV_COLD); + ok &= !madvise(p, len, MADV_PAGEOUT); + } + return !sched_setaffinity(0, sizeof(allowed), &allowed) && ok; +} + +static bool mark_free(void *p, size_t len) +{ + cpu_set_t allowed, one; + bool ok = true; + + if (sched_getaffinity(0, sizeof(allowed), &allowed)) + return false; + for (unsigned int i = 0; i < CPU_SETSIZE; i++) { + if (!CPU_ISSET(i, &allowed)) + continue; + CPU_ZERO(&one); + CPU_SET(i, &one); + ok &= !sched_setaffinity(0, sizeof(one), &one); + ok &= !madvise(p, len, MADV_FREE); + ok &= !madvise(p, len, MADV_COLD); + } + return !sched_setaffinity(0, sizeof(allowed), &allowed) && ok; +} + +static int exercise(unsigned long native) +{ + const size_t ps = getauxval(AT_PAGESZ), len = 4 * ps; + unsigned char *reservation = mmap(NULL, len + 2 * native, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + unsigned char *p = + (void *)(((uintptr_t)reservation + native - 1) & ~(native - 1)); + bool ok; + unsigned long writes; + uint64_t entry; + + if (reservation == MAP_FAILED || + mprotect(p, len, PROT_READ | PROT_WRITE)) + ksft_exit_fail_msg("mapping\n"); + ksft_print_header(); + ksft_set_plan(8); + for (unsigned int i = 0; i < 4; i++) + memset(p + i * ps, 0x40 + i, ps); + ok = mark_free(p + ps, ps); + if (!ok) + ksft_exit_fail_msg("%zuK MADV_FREE: %s\n", ps / 1024, + strerror(errno)); + ksft_test_result(all_bytes(p + ps, ps, 0x41), + "%zuK FREE retains bytes before reclaim\n", ps / 1024); + ksft_test_result(lazy_kb(p) == ps / 1024, + "%zuK partial LazyFree accounting\n", ps / 1024); + ok = pageout(p, len); + entry = pagemap(p + ps, ps); + ok &= !(entry & ((1ULL << 63) | (1ULL << 62))); + ok &= all_bytes(p + ps, ps, 0) && all_bytes(p, ps, 0x40) && + all_bytes(p + 2 * ps, ps, 0x42) && + all_bytes(p + 3 * ps, ps, 0x43); + ksft_test_result(ok, "%zuK partial FREE reclaims only advised bytes\n", + ps / 1024); + /* Drop old swap/cache state, then test cancellation by a real EL0 write. */ + ok = !madvise(p, len, MADV_DONTNEED); + memset(p, 0x62, len); + ok &= mark_free(p + ps, ps); + memset(p + ps, 0xa5, ps); + ksft_test_result(lazy_kb(p) == 0, + "%zuK dirty leaf no longer reported LazyFree\n", + ps / 1024); + ok &= pageout(p, len); + ok &= all_bytes(p + ps, ps, 0xa5) && all_bytes(p, ps, 0x62) && + all_bytes(p + 2 * ps, 2 * ps, 0x62); + ksft_test_result(ok, "%zuK write after FREE cancels quarter discard\n", + ps / 1024); + ok = !madvise(p, len, MADV_DONTNEED); + memset(p, 0x83, len); + writes = stat_value("/proc/vmstat", "pswpout"); + ok &= mark_free(p, len) && pageout(p, len); + for (unsigned int i = 0; i < 4; i++) + ok &= !(pagemap(p + i * ps, ps) & + ((1ULL << 63) | (1ULL << 62))); + printf("# full FREE pswpout=%lu->%lu, entries=%#llx %#llx %#llx %#llx\n", + writes, stat_value("/proc/vmstat", "pswpout"), + (unsigned long long)pagemap(p, ps), + (unsigned long long)pagemap(p + ps, ps), + (unsigned long long)pagemap(p + 2 * ps, ps), + (unsigned long long)pagemap(p + 3 * ps, ps)); + ok &= stat_value("/proc/vmstat", "pswpout") == writes && + all_bytes(p, len, 0); + ksft_test_result( + ok, "%zuK fully free backing reclaimed without swap writes\n", + ps / 1024); + ok = !madvise(p, len, MADV_DONTNEED); + memset(p, 0xb7, len); + ok &= pageout(p, len); + ok &= !!(pagemap(p + ps, ps) & (1ULL << 62)); + ok &= mark_free(p + ps, ps); + ok &= !(pagemap(p + ps, ps) & ((1ULL << 63) | (1ULL << 62))); + ok &= all_bytes(p + ps, ps, 0) && all_bytes(p, ps, 0xb7) && + all_bytes(p + 2 * ps, 2 * ps, 0xb7); + ksft_test_result( + ok, + "%zuK FREE drops selected swap entry without neighbor loss\n", + ps / 1024); + { + int gate[2], status = 0; + pid_t pid; + char ready; + + ok = !madvise(p, len, MADV_DONTNEED); + memset(p, 0xc8, len); + if (pipe(gate)) + ksft_exit_fail_msg("pipe\n"); + pid = fork(); + if (!pid) { + close(gate[1]); + if (read(gate[0], &ready, 1) != 1 || + !all_bytes(p, len, 0xc8)) + _exit(2); + _exit(0); + } + close(gate[0]); + ok &= mark_free(p + ps, ps) && pageout(p, len); + ok &= write(gate[1], "x", 1) == 1; + close(gate[1]); + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + ksft_test_result( + ok, + "%zuK FREE after fork preserves other process data\n", + ps / 1024); + } + munmap(reservation, len + 2 * native); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} + +int main(int argc, char **argv) +{ + bool ok = true; + unsigned long native = getauxval(AT_PAGESZ); + unsigned long granules[] = { native, 4096, 16384 }; + unsigned int nr = native > 16384 ? 3 : 2; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--test") && getppid() == 1) { + if (getauxval(AT_PAGESZ) != strtoul(argv[2], NULL, 10)) + return 42; + return exercise(strtoul(argv[3], NULL, 10)); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Run only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + swapon("/dev/vda", 0)) + return 3; + for (unsigned int small = 0; small < nr; small++) { + pid_t pid = fork(); + int status = 0; + + if (!pid) { + char size[32], host_size[32]; + + snprintf(size, sizeof(size), "%lu", granules[small]); + snprintf(host_size, sizeof(host_size), "%lu", native); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, granules[small], + 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--test", size, host_size, + NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + } + printf("%s - lazyfree fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/memcg-accounting-init.c b/tools/testing/selftests/arm64/user4k/memcg-accounting-init.c new file mode 100644 index 00000000000000..f34602bb730e5c --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/memcg-accounting-init.c @@ -0,0 +1,153 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CG "/sys/fs/cgroup/account" +#define LENGTH (32UL * 1024 * 1024) +static int put(const char *p, const char *v) +{ + int fd = open(p, O_WRONLY); + if (fd < 0) + return 1; + int bad = write(fd, v, strlen(v)) != (ssize_t)strlen(v); + close(fd); + return bad; +} +static unsigned long long stat_value(const char *key) +{ + FILE *f = fopen(CG "/memory.stat", "r"); + char name[128]; + unsigned long long value = 0; + if (!f) + exit(20); + while (fscanf(f, "%127s %llu", name, &value) == 2) + if (!strcmp(key, name)) { + fclose(f); + return value; + } + fclose(f); + exit(21); +} +static int worker(unsigned long expected, int ready, int release) +{ + if (getauxval(AT_PAGESZ) != expected) + return 2; + unsigned char *p = mmap(NULL, LENGTH, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED || madvise(p, LENGTH, MADV_NOHUGEPAGE)) + return 3; + memset(p, 0x71, LENGTH); + char token = 'R'; + if (write(ready, &token, 1) != 1 || read(release, &token, 1) != 1) + return 4; + /* Keep the writes observable independently of the accounting check. */ + for (size_t i = 0; i < LENGTH; i++) + if (p[i] != 0x71) + return 5; + return munmap(p, LENGTH) ? 6 : 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (getppid() == 1 && argc == 5 && !strcmp(argv[1], "--worker")) + return worker(strtoul(argv[2], NULL, 10), atoi(argv[3]), + atoi(argv[4])); + if (getpid() != 1 || argc != 1) + return 2; + signal(SIGPIPE, SIG_IGN); + alarm(60); + int failed = 0; + unsigned long long slab[3] = {}; + unsigned long sizes[3] = { getauxval(AT_PAGESZ), 4096, 16384 }; + unsigned int modes = sizes[0] == 65536 ? 3 : 2; + if ((sizes[0] != 16384 && sizes[0] != 65536) || + mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL) || + put("/sys/fs/cgroup/cgroup.subtree_control", "+memory")) { + perror("setup"); + failed = 1; + goto done; + } + for (unsigned i = 0; i < modes; i++) { + int ready[2], release[2]; + if (mkdir(CG, 0755) || put(CG "/memory.max", "134217728") || + pipe(ready) || pipe(release)) { + failed = 1; + break; + } + pid_t child = fork(); + if (!child) { + char ps[32], rfd[16], cfd[16]; + close(ready[0]); + close(release[1]); + snprintf(ps, sizeof(ps), "%lu", sizes[i]); + snprintf(rfd, sizeof(rfd), "%d", ready[1]); + snprintf(cfd, sizeof(cfd), "%d", release[0]); + if (put(CG "/cgroup.procs", "0") || + prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(7); + execl("/init", "/init", "--worker", ps, rfd, cfd, NULL); + _exit(8); + } + close(ready[1]); + close(release[0]); + char token; + int ok = child > 0 && read(ready[0], &token, 1) == 1; + if (ok) { + /* Let per-CPU slab statistics become visible before the comparison. */ + usleep(1200000); + slab[i] = stat_value("slab_unreclaimable"); + printf("# ABI=%lu anon=%llu slab_unreclaimable=%llu pagetables=%llu\n", + sizes[i], stat_value("anon"), slab[i], + stat_value("pagetables")); + ok = stat_value("anon") >= LENGTH && + write(release[1], "G", 1) == 1; + } + int status = 0; + if (child > 0 && !ok) + kill(child, SIGKILL); + ok = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status) && ok; + close(ready[0]); + close(release[1]); + printf("%s - memcg accounting ABI %luK worker status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + failed |= !ok; + for (unsigned retry = 0; retry < 100 && stat_value("anon"); + retry++) + usleep(20000); + if (stat_value("anon") || rmdir(CG)) + failed = 1; + if (failed) + break; + } + /* Slot metadata must be charged for every alternative granule. The + * loose lower bound tolerates slab/KASAN layout differences. */ + for (unsigned i = 1; !failed && i < modes; i++) { + if (slab[i] < slab[0] + 512 * 1024) { + printf("not ok - slot metadata escaped cgroup accounting ABI=%lu native=%llu alternative=%llu\n", + sizes[i], slab[0], slab[i]); + failed = 1; + } + } + +done: + printf("GRANULE MEMCG ACCOUNTING %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/memcg-pressure-init.c b/tools/testing/selftests/arm64/user4k/memcg-pressure-init.c new file mode 100644 index 00000000000000..c2f50993ff805e --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/memcg-pressure-init.c @@ -0,0 +1,417 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CG "/sys/fs/cgroup/pressure" +#define MIB (1024UL * 1024) +#define LENGTH (32 * MIB) +#define ROUNDS 4 + +static int put(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY); + if (fd < 0) + return -1; + ssize_t ret = write(fd, value, strlen(value)); + int saved = errno; + close(fd); + errno = saved; + return ret == (ssize_t)strlen(value) ? 0 : -1; +} + +static unsigned long long get(const char *path, const char *key) +{ + FILE *f = fopen(path, "r"); + char name[128]; + unsigned long long value = 0, result = 0; + if (!f) { + perror(path); + exit(10); + } + if (!key) { + if (fscanf(f, "%llu", &result) != 1) + exit(11); + } else + while (fscanf(f, "%127s %llu", name, &value) == 2) + if (!strcmp(name, key)) { + result = value; + break; + } + fclose(f); + return result; +} + +static unsigned char pattern(size_t offset, unsigned seed, unsigned round) +{ + return (offset ^ (offset >> 12) ^ (offset >> 20) ^ (seed * 79) ^ + (round * 43)) & + 255; +} + +static int contents(unsigned char *p, size_t ps, unsigned seed, unsigned round, + int write_data, int child) +{ + size_t pages = LENGTH / ps; + /* Odd multiplication permutes every logical page of this power-of-two range. */ + for (size_t i = 0; i < pages; i++) { + size_t page = (i * 4051 + round * 31) & (pages - 1); + for (size_t j = 0; j < ps; j++) { + size_t offset = page * ps + j; + unsigned char want = pattern(offset, seed, round); + if (child && !(page & 3)) + want ^= 0x69; + if (write_data) + p[offset] = want; + else if (p[offset] != want) { + printf("not ok - ABI=%zu round=%u child=%d byte=%zu got=%u expected=%u\n", + ps, round, child, offset, p[offset], + want); + return 1; + } + } + } + return 0; +} + +static int barrier(int ready, int command) +{ + char token = 'R'; + return write(ready, &token, 1) != 1 || read(command, &token, 1) != 1 || + token != 'G'; +} + +static int worker(unsigned long expected, unsigned seed, int ready, int command) +{ + size_t ps = getauxval(AT_PAGESZ); + unsigned char *p; + if (ps != expected) + return 2; + p = mmap(NULL, LENGTH, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + return 3; + if (madvise(p, LENGTH, MADV_NOHUGEPAGE)) { + int error = errno; + struct stat st; + + /* Without THP this advice is unavailable and cannot change policy. */ + if (error != EINVAL || + !stat("/sys/kernel/mm/transparent_hugepage/enabled", &st) || + errno != ENOENT) + return 3; + } + if (contents(p, ps, seed, 0, 1, 0) || barrier(ready, command)) + return 4; + for (unsigned round = 0; round < ROUNDS; round++) { + if (contents(p, ps, seed, round, 0, 0)) + return 5; + pid_t child = fork(); + if (child < 0) + return 6; + if (!child) { + /* Dirty a quarter of the logical pages, retaining untouched siblings. */ + for (size_t page = 0; page < LENGTH / ps; page += 4) + for (size_t j = 0; j < ps; j++) + p[page * ps + j] ^= 0x69; + _exit(contents(p, ps, seed, round, 0, 1) ? 7 : 0); + } + int status; + if (waitpid(child, &status, 0) != child || !WIFEXITED(status) || + WEXITSTATUS(status) || contents(p, ps, seed, round, 0, 0)) + return 8; + printf("ok - memcg ABI %zuK round=%u swap-in bytes and fork COW isolation\n", + ps / 1024, round); + if (round + 1 < ROUNDS && + contents(p, ps, seed, round + 1, 1, 0)) + return 9; + if (barrier(ready, command)) + return 10; + } + return munmap(p, LENGTH) ? 11 : 0; +} + +static void report(unsigned phase) +{ + printf("# memcg phase=%u resident=%llu swap=%llu anon=%llu pgscan=%llu pgsteal=%llu max=%llu oom=%llu oom_kill=%llu\n", + phase, get(CG "/memory.current", NULL), + get(CG "/memory.swap.current", NULL), + get(CG "/memory.stat", "anon"), get(CG "/memory.stat", "pgscan"), + get(CG "/memory.stat", "pgsteal"), + get(CG "/memory.events", "max"), get(CG "/memory.events", "oom"), + get(CG "/memory.events", "oom_kill")); +} + +static void pressure_stacks(void) +{ + FILE *tasks = fopen(CG "/cgroup.procs", "r"); + int pid; + + if (!tasks) + return; + while (fscanf(tasks, "%d", &pid) == 1) { + char path[64], stack[4096]; + int fd; + ssize_t n; + + snprintf(path, sizeof(path), "/proc/%d/stack", pid); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + continue; + n = read(fd, stack, sizeof(stack) - 1); + close(fd); + if (n > 0) { + stack[n] = 0; + printf("# pressure stack pid=%d\n%s", pid, stack); + } + } + fclose(tasks); +} + +static void observe_pressure(void) +{ + unsigned int samples = 0; + + for (;;) { + struct timespec now; + + clock_gettime(CLOCK_MONOTONIC, &now); + printf("# io-pressure ms=%lld resident=%llu swap=%llu writeback=%llu " + "swapcached=%llu scan=%llu steal=%llu max=%llu oom=%llu kill=%llu\n", + (long long)now.tv_sec * 1000 + now.tv_nsec / 1000000, + get(CG "/memory.current", NULL), + get(CG "/memory.swap.current", NULL), + get(CG "/memory.stat", "file_writeback"), + get(CG "/memory.stat", "swapcached"), + get(CG "/memory.stat", "pgscan"), + get(CG "/memory.stat", "pgsteal"), + get(CG "/memory.events", "max"), + get(CG "/memory.events", "oom"), + get(CG "/memory.events", "oom_kill")); + if (!(++samples % 8) && + get(CG "/memory.stat", "file_writeback") > 8 * MIB) + pressure_stacks(); + usleep(250000); + } +} + +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--worker") && getppid() == 1) + return worker(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), atoi(argv[4]), + atoi(argv[5])); + bool classic = false, native_control = false, user16k = false; + + if (getpid() != 1 || argc > 3) { + fprintf(stderr, "Dedicated disposable VM PID1 only\n"); + return 2; + } + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--classic")) + classic = true; + else if (!strcmp(argv[i], "--native-control")) + native_control = true; + else if (!strcmp(argv[i], "--user16k")) + user16k = true; + else + return 2; + } + if (native_control && user16k) + return 2; + signal(SIGPIPE, SIG_IGN); + alarm(300); + int failed = 0, active = 0; + pid_t children[2] = { -1, -1 }, observer = -1; + int ready[2][2], command[2][2]; + unsigned long sizes[2] = { getauxval(AT_PAGESZ), 4096 }; + if (native_control) + sizes[1] = sizes[0]; + if (user16k) + sizes[1] = 16384; + if ((sizes[0] != 16384 && sizes[0] != 65536) || sizes[1] > sizes[0]) { + failed = 1; + goto done; + } + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL) || + put("/sys/fs/cgroup/cgroup.subtree_control", "+memory") || + mkdir(CG, 0755) || put(CG "/memory.max", "50331648") || + put(CG "/memory.swap.max", "268435456")) { + perror("cgroup setup"); + failed = 1; + goto done; + } + /* Explicit diagnostic control, never change the production default. */ + if (classic && put("/sys/kernel/mm/lru_gen/enabled", "0")) { + perror("classic reclaim control"); + failed = 1; + goto done; + } + printf("# reclaim mode=%s\n", + classic ? "classic diagnostic" : "default"); + struct stat st; + for (int i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + if (swapon("/dev/vda", 0)) { + perror("swapon"); + failed = 1; + goto done; + } + active = 1; + for (unsigned i = 0; i < 2; i++) + if (pipe(ready[i]) || pipe(command[i])) { + failed = 1; + goto done; + } + observer = fork(); + if (!observer) { + for (unsigned int i = 0; i < 2; i++) { + close(ready[i][0]); + close(ready[i][1]); + close(command[i][0]); + close(command[i][1]); + } + observe_pressure(); + _exit(0); + } + if (observer < 0) { + failed = 1; + goto done; + } + for (unsigned i = 0; i < 2; i++) { + children[i] = fork(); + if (!children[i]) { + char ps[32], index[16], rfd[16], cfd[16]; + for (unsigned j = 0; j < 2; j++) { + close(ready[j][0]); + close(command[j][1]); + if (i != j) { + close(ready[j][1]); + close(command[j][0]); + } + } + snprintf(ps, sizeof(ps), "%lu", sizes[i]); + snprintf(index, sizeof(index), "%u", i); + snprintf(rfd, sizeof(rfd), "%d", ready[i][1]); + snprintf(cfd, sizeof(cfd), "%d", command[i][0]); + if (put(CG "/cgroup.procs", "0") || + prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(12); + execl("/init", "/init", "--worker", ps, index, rfd, cfd, + NULL); + _exit(13); + } + if (children[i] < 0) { + failed = 1; + goto done; + } + } + for (unsigned i = 0; i < 2; i++) { + close(ready[i][1]); + close(command[i][0]); + } + for (unsigned phase = 0; phase <= ROUNDS; phase++) { + char token; + for (unsigned i = 0; i < 2; i++) + if (read(ready[i][0], &token, 1) != 1 || token != 'R') { + failed = 1; + goto done; + } + report(phase); + if (!get(CG "/memory.swap.current", NULL) || + !get(CG "/memory.events", "max") || + !get(CG "/memory.stat", "pgscan") || + !get(CG "/memory.stat", "pgsteal") || + get(CG "/memory.events", "oom") || + get(CG "/memory.events", "oom_kill")) { + printf("not ok - missing pressure evidence or unexpected OOM\n"); + failed = 1; + goto done; + } + for (unsigned i = 0; i < 2; i++) + if (write(command[i][1], "G", 1) != 1) { + failed = 1; + goto done; + } + } + for (unsigned i = 0; i < 2; i++) { + int status = 0; + int ok = waitpid(children[i], &status, 0) == children[i] && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - memcg ABI %luK exit=%#x\n", ok ? "ok" : "not ok", + sizes[i] / 1024, status); + children[i] = -1; + failed |= !ok; + } + /* Pending swap writeback can outlive exit, especially on a slow device. */ + for (unsigned i = 0; i < 3000 && (get(CG "/memory.stat", "anon") || + get(CG "/memory.swap.current", NULL)); + i++) { + if (!(i % 250)) + printf("# exit drain ms=%u swap=%llu writeback=%llu\n", + i * 20, get(CG "/memory.swap.current", NULL), + get(CG "/memory.stat", "file_writeback")); + /* Exit may leave clean, unmapped swap cache after writeback. Reclaim + * it explicitly; EAGAIN means fewer than the requested bytes existed. */ + if (!get(CG "/memory.stat", "file_writeback") && + put(CG "/memory.reclaim", "67108864 swappiness=200") && + errno != EAGAIN) { + perror("exit cache reclaim"); + failed = 1; + break; + } + usleep(20000); + } + report(ROUNDS + 1); + if (get(CG "/memory.stat", "anon") || + get(CG "/memory.swap.current", NULL)) { + printf("not ok - anonymous or swap charge remains after exit\n"); + failed = 1; + } +done: + if (observer > 0) { + kill(observer, SIGTERM); + waitpid(observer, NULL, 0); + } + if (active && failed) + report(ROUNDS + 2); + for (unsigned i = 0; i < 2; i++) + if (children[i] > 0) { + kill(children[i], SIGKILL); + waitpid(children[i], NULL, 0); + } + if (active && failed) { + put(CG "/cgroup.kill", "1"); + put(CG "/memory.max", "max"); + } + if (active && swapoff("/dev/vda")) { + perror("swapoff"); + failed = 1; + } + printf("GRANULE MEMCG PRESSURE %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/memory-interfaces.c b/tools/testing/selftests/arm64/user4k/memory-interfaces.c new file mode 100644 index 00000000000000..330296ed44499d --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/memory-interfaces.c @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static size_t page; + +static void residency(void) +{ + unsigned char *p, vec[6]; + bool ok; + unsigned int i; + + p = mmap(NULL, 5 * page, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + ksft_exit_fail_msg("anonymous mmap\n"); + memset(vec, 0x55, sizeof(vec)); + ok = mincore(p, 5 * page - 1, vec) == 0; + for (i = 0; i < 5; i++) + ok &= vec[i] == 0; + ok &= vec[5] == 0x55; + ksft_test_result( + ok, "mincore emits exactly one byte per unfaulted user page\n"); + ok = mprotect(p + 2 * page, page, PROT_READ | PROT_WRITE) == 0; + p[2 * page] = 0x37; + ok &= mincore(p, 5 * page, vec) == 0; + for (i = 0; i < 5; i++) + ok &= vec[i] == (i == 2); + ksft_test_result( + ok, + "mincore reports a separately faulted VMA among cold neighbors\n"); + ok = munmap(p + 3 * page, page) == 0; + errno = 0; + ok &= mincore(p, 5 * page, vec) == -1 && errno == ENOMEM; + errno = 0; + ok &= mincore(p + 1, page, vec) == -1 && errno == EINVAL; + ksft_test_result(ok, + "mincore rejects an unaligned start and a VMA hole\n"); + munmap(p, 5 * page); +} + +static void file_residency(void) +{ + const size_t native = 16384, count = 3 * native / page; + unsigned char *p, *guard, *vec, byte = 0x7b; + int fd = memfd_create("residency", 0); + bool ok; + size_t i; + + if (fd < 0 || ftruncate(fd, 4 * native) || + pwrite(fd, &byte, 1, native) != 1) + ksft_exit_fail_msg("memfd setup\n"); + p = mmap(NULL, 3 * native, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + page); + guard = mmap(NULL, 2 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED || guard == MAP_FAILED || + mprotect(guard + page, page, PROT_NONE)) + ksft_exit_fail_msg("file/vector mmap\n"); + vec = guard + page - count; + memset(vec, 0x55, count); + ok = mincore(p, 3 * native, vec) == 0; + for (i = 0; i < count; i++) + ok &= vec[i] == ((page + i * page) / native == 1); + ksft_test_result( + ok, + "unmapped file leaves share residency at native cache indices\n"); + ok = p[native - page] == byte && mincore(p, 3 * native, vec) == 0; + for (i = 0; i < count; i++) + ok &= vec[i] == ((page + i * page) / native == 1); + ksft_test_result( + ok, + "present and cache-only file residency agree at a guarded vector boundary\n"); + ok = msync(p + page, 1, MS_SYNC) == 0; + errno = 0; + ok &= msync(p + page + 1, page, MS_SYNC) == -1 && errno == EINVAL; + ok &= munmap(p + page, page) == 0; + errno = 0; + ok &= msync(p, 3 * page, MS_SYNC) == -1 && errno == ENOMEM; + ksft_test_result(ok, + "msync uses process alignment and reports holes\n"); + munmap(p, 3 * native); + munmap(guard, 2 * page); + close(fd); +} + +static void large_vector(void) +{ + const size_t count = 16384 + 3; + unsigned char *p = mmap(NULL, count * page, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, + 0); + unsigned char *vec = malloc(count + 1); + bool ok; + size_t i; + + if (p == MAP_FAILED || !vec) + ksft_exit_fail_msg("large vector allocation\n"); + memset(vec, 0x55, count + 1); + ok = mincore(p, count * page - 7, vec) == 0; + for (i = 0; i < count; i++) + ok &= vec[i] == 0; + ok &= vec[count] == 0x55; + ksft_test_result( + ok, "mincore advances across multiple native output buffers\n"); + free(vec); + munmap(p, count * page); +} + +static void seals(void) +{ + unsigned char *p = mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + bool ok; + pid_t pid; + int status = 0; + + if (p == MAP_FAILED) + ksft_exit_fail_msg("seal mmap\n"); + ok = syscall(SYS_mseal, p + page, 1UL, 0UL) == 0; + errno = 0; + ok &= mprotect(p + page, page, PROT_READ) == -1 && errno == EPERM; + errno = 0; + ok &= munmap(p + page, page) == -1 && errno == EPERM; + ok &= mprotect(p, page, PROT_READ) == 0; + ok &= mprotect(p + 2 * page, page, PROT_READ) == 0; + ksft_test_result( + ok, + "a one-byte seal covers only its user page and protects it\n"); + pid = fork(); + if (!pid) + _exit(!(munmap(p + page, page) == -1 && errno == EPERM)); + ok = pid > 0 && waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status); + ksft_test_result(ok, "fork inherits the sealed user page\n"); + munmap(p, page); + munmap(p + 2 * page, page); + /* The sealed VMA is released by process exit. */ +} + +int main(int argc, char **argv) +{ + page = getauxval(AT_PAGESZ); + if (argc > 2 || (argc == 2 && strcmp(argv[1], "--expect-4k")) || + page != (argc == 2 ? 4096UL : 16384UL)) + ksft_exit_fail_msg("unexpected process page ABI: %zu\n", page); + ksft_print_msg("process page size: %zu\n", page); + ksft_print_header(); + ksft_set_plan(9); + residency(); + file_residency(); + large_vector(); + seals(); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/mglru-granule-init.c b/tools/testing/selftests/arm64/user4k/mglru-granule-init.c new file mode 100644 index 00000000000000..9bb9bb74e38e86 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/mglru-granule-init.c @@ -0,0 +1,365 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CONTROL "/sys/kernel/debug/lru_gen" +#define FULL "/sys/kernel/debug/lru_gen_full" +static int ready_fd = -1, command_fd = -1; +static unsigned worker_index; +static unsigned char pattern(unsigned region, size_t byte, unsigned round) +{ + return (byte ^ (byte >> 12) ^ (region * 37) ^ (round * 53)) & 255; +} +static int generation(unsigned long long *cg, unsigned *node, + unsigned long *max) +{ + FILE *f = fopen(CONTROL, "r"); + char line[256]; + int have_cg = 0, have_node = 0, have_seq = 0; + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) { + unsigned long long id; + unsigned n; + unsigned long seq, age, anon, file; + if (sscanf(line, "memcg %llu", &id) == 1) { + if (have_cg) + break; + *cg = id; + have_cg = 1; + } else if (sscanf(line, " node %u", &n) == 1) { + if (have_node) + break; + *node = n; + have_node = 1; + } else if (have_node && sscanf(line, "%lu %lu %lu %lu", &seq, + &age, &anon, &file) == 4) { + *max = seq; + have_seq = 1; + } + } + fclose(f); + return have_seq ? 0 : -1; +} +static unsigned long scan_evidence(void) +{ + FILE *f = fopen(FULL, "r"); + char line[512]; + unsigned long total = 0; + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) { + unsigned long t, y, found, added; + if (sscanf(line, " %lut %luy %luf %lua", &t, &y, &found, + &added) == 4) { + printf("# MGLRU scan total=%lu young=%lu tables=%lu added=%lu\n", + t, y, found, added); + total += y; + } + } + fclose(f); + return total; +} +static int age(void) +{ + if (ready_fd >= 0) { + char token = 'A'; + return write(ready_fd, &token, 1) != 1 || + read(command_fd, &token, 1) != 1 || token != 'G'; + } + unsigned long long cg = 0, next_cg = 0; + unsigned node = 0, next_node = 0; + unsigned long seq = 0, next = 0; + char command[128]; + if (generation(&cg, &node, &seq)) + return 1; + /* This tree accepts swappiness here; 200 scans both anonymous and file pages. */ + int len = snprintf(command, sizeof(command), "+ %llu %u %lu 200 1\n", + cg, node, seq); + int fd = open(CONTROL, O_WRONLY); + if (fd < 0) + return 1; + ssize_t written = write(fd, command, len); + close(fd); + if (written != len || generation(&next_cg, &next_node, &next) || + next <= seq) { + printf("not ok - MGLRU generation command=%s result=%zd errno=%d next=%lu\n", + command, written, errno, next); + return 1; + } + printf("ok - MGLRU generation %lu -> %lu\n", seq, next); + return 0; +} +static int check_referenced(unsigned char **regions, unsigned nr, + size_t expected) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + char line[512]; + int region = -1, failed = 0; + unsigned seen = 0; + unsigned long start, end, kb; + if (!f) + return 1; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) { + region = -1; + for (unsigned i = 0; i < nr; i++) + if (start == (uintptr_t)regions[i]) + region = i; + } else if (region >= 0 && + sscanf(line, "Referenced: %lu kB", &kb) == 1) { + printf("%s - region=%d start=%p referenced=%lu expected=%zu kB\n", + kb == expected ? "ok" : "not ok", region, + regions[region], kb, expected); + failed |= kb != expected; + seen++; + } + } + fclose(f); + return failed || seen != nr; +} +static int exercise(unsigned long expected) +{ + static const unsigned shifts[] = { 21, 25, 29, 30, 36, 39, 42, 47 }; + unsigned char *regions[sizeof(shifts) / sizeof(shifts[0])] = {}; + size_t ps = getauxval(AT_PAGESZ), len = 8 * ps; + unsigned nr = sizeof(shifts) / sizeof(shifts[0]); + int fail = 0, fd = memfd_create("granule-aging", 0); + cpu_set_t cpus; + void *scratch = MAP_FAILED; + CPU_ZERO(&cpus); + CPU_SET(ready_fd >= 0 ? worker_index % 2 : 0, &cpus); + if (sched_setaffinity(0, sizeof(cpus), &cpus)) + return 1; + if (ps != expected || fd < 0 || ftruncate(fd, nr * len)) + return 1; + for (unsigned i = 0; i < nr; i++) { + uintptr_t address = (1UL << shifts[i]) - 4 * ps; + regions[i] = mmap((void *)address, len, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | + (i % 2 ? MAP_SHARED : + MAP_PRIVATE | MAP_ANONYMOUS), + i % 2 ? fd : -1, i % 2 ? i * len : 0); + if (regions[i] != (void *)address) { + printf("not ok - %zuK boundary %u mmap errno=%d\n", + ps / 1024, shifts[i], errno); + fail = 1; + goto out; + } + } + for (unsigned i = 0; i < nr; i++) + memset(regions[i], 0x19, len); + /* Publish earlier folios from the bounded per-CPU LRU-add batch. */ + scratch = mmap(NULL, 8UL << 20, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (scratch == MAP_FAILED) { + fail = 1; + goto out; + } + memset(scratch, 0x7b, 8UL << 20); + /* Seed MGLRU's first-access referenced state before testing the next access. */ + if (age()) { + fail = 1; + goto out; + } + for (unsigned round = 0; round < 4; round++) { + if (ready_fd >= 0) { + unsigned cpu = (worker_index + round) % 2; + CPU_ZERO(&cpus); + CPU_SET(cpu, &cpus); + if (sched_setaffinity(0, sizeof(cpus), &cpus) || + sched_getcpu() != (int)cpu) { + fail = 1; + break; + } + printf("# concurrent ABI=%zuK round=%u cpu=%u\n", + ps / 1024, round, cpu); + } + for (unsigned i = 0; i < nr; i++) + for (size_t j = 0; j < len; j++) + regions[i][j] = pattern(i, j, round); + fail |= check_referenced(regions, nr, len / 1024); + if (age() || age()) { + fail = 1; + break; + } + fail |= check_referenced(regions, nr, 0); + int good = 1; + for (unsigned i = 0; i < nr; i++) + for (size_t j = 0; j < len; j++) + if (regions[i][j] != pattern(i, j, round)) { + good = 0; + break; + } + unsigned long young = scan_evidence(); + printf("%s - %zuK round%u eight table boundaries bytes and aging young=%lu\n", + good && young ? "ok" : "not ok", ps / 1024, round, + young); + fail |= !good || !young; + } +out: + if (scratch != MAP_FAILED) + munmap(scratch, 8UL << 20); + for (unsigned i = 0; i < nr; i++) + if (regions[i] && regions[i] != MAP_FAILED) + munmap(regions[i], len); + close(fd); + return fail; +} +static int concurrent(unsigned long *sizes, unsigned nr) +{ + int ready[3][2], commands[3][2], failed = 0; + pid_t workers[3]; + + for (unsigned i = 0; i < nr; i++) + if (pipe(ready[i]) || pipe(commands[i])) + return 1; + for (unsigned i = 0; i < nr; i++) { + workers[i] = fork(); + if (!workers[i]) { + char size[32], index[16], ready_arg[16], + command_arg[16]; + for (unsigned j = 0; j < nr; j++) { + close(ready[j][0]); + close(commands[j][1]); + if (j != i) { + close(ready[j][1]); + close(commands[j][0]); + } + } + snprintf(size, sizeof(size), "%lu", sizes[i]); + snprintf(index, sizeof(index), "%u", i); + snprintf(ready_arg, sizeof(ready_arg), "%d", + ready[i][1]); + snprintf(command_arg, sizeof(command_arg), "%d", + commands[i][0]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--worker", size, index, + ready_arg, command_arg, NULL); + _exit(4); + } + if (workers[i] < 0) + failed = 1; + } + for (unsigned i = 0; i < nr; i++) { + close(ready[i][1]); + close(commands[i][0]); + } + /* One warm-up barrier plus two aging barriers for each of four rounds. */ + for (unsigned phase = 0; phase < 9 && !failed; phase++) { + for (unsigned i = 0; i < nr; i++) { + char token; + if (read(ready[i][0], &token, 1) != 1 || token != 'A') + failed = 1; + } + if (failed || age()) { + failed = 1; + break; + } + for (unsigned i = 0; i < nr; i++) { + char token = 'G'; + if (write(commands[i][1], &token, 1) != 1) + failed = 1; + } + } + for (unsigned i = 0; i < nr; i++) { + close(ready[i][0]); + close(commands[i][1]); + } + for (unsigned i = 0; i < nr; i++) { + int status = 0; + int ok = workers[i] > 0 && + waitpid(workers[i], &status, 0) == workers[i] && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - concurrent MGLRU ABI %luK status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + failed |= !ok; + } + return failed; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGPIPE, SIG_IGN); + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (argc == 6 && !strcmp(argv[1], "--worker")) { + if (getppid() != 1) + return 2; + worker_index = strtoul(argv[3], NULL, 10); + ready_fd = atoi(argv[4]); + command_fd = atoi(argv[5]); + return exercise(strtoul(argv[2], NULL, 10)); + } + int concurrent_mode = argc == 2 && !strcmp(argv[1], "--concurrent"); + if (getpid() != 1 || (argc != 1 && !concurrent_mode)) + return 2; + int fail = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)) { + perror("mount"); + fail = 1; + goto done; + } + FILE *f = fopen("/sys/kernel/mm/lru_gen/enabled", "r"); + unsigned enabled = 0; + if (f) { + fscanf(f, "%x", &enabled); + fclose(f); + } + if ((enabled & 3) != 3) { + printf("not ok - MGLRU/MM walk disabled %#x\n", enabled); + fail = 1; + goto done; + } + printf("ok - MGLRU enabled=%#x\n", enabled); + unsigned long sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + if (concurrent_mode) { + fail = concurrent(sizes, sizes[0] > 16384 ? 3 : 2); + goto done; + } + for (unsigned i = 0; i < (sizes[0] > 16384 ? 3U : 2U); i++) { + pid_t child = fork(); + int status = 0; + if (!child) { + char value[32]; + snprintf(value, sizeof(value), "%lu", sizes[i]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--exercise", value, NULL); + _exit(4); + } + int good = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - MGLRU ABI %luK status=%#x\n", + good ? "ok" : "not ok", sizes[i] / 1024, status); + fail |= !good; + } +done: + printf("GRANULE MGLRU%s %s\n", concurrent_mode ? " CONCURRENT" : "", + fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/mixed-process.c b/tools/testing/selftests/arm64/user4k/mixed-process.c new file mode 100644 index 00000000000000..6d08ab55991b5c --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/mixed-process.c @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Real EL0 mixed-granule sharing, futex queues, and process_vm copies. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include + +#define NATIVE 16384U +#define QUARTER 4096U +#define ROUNDS 2000U +struct channel { + atomic_int turn; + unsigned int value; + unsigned int worker_wakes; +}; +struct ready { + uintptr_t base; + unsigned long page; +}; + +static int wait_turn(struct channel *channel, int wanted) +{ + const struct timespec timeout = { .tv_sec = 5 }; + int value; + + while ((value = atomic_load_explicit(&channel->turn, + memory_order_acquire)) != wanted) { + if (syscall(SYS_futex, &channel->turn, FUTEX_WAIT, value, + &timeout, NULL, 0) < 0 && + errno != EAGAIN && errno != EINTR) + return -1; + } + return 0; +} + +static int send_turn(struct channel *channel, int turn) +{ + atomic_store_explicit(&channel->turn, turn, memory_order_release); + return syscall(SYS_futex, &channel->turn, FUTEX_WAKE, 1, NULL, NULL, 0); +} + +static unsigned char pattern(size_t i) +{ + return (i ^ (i >> 8) ^ (i >> 16)) & 255; +} + +static int worker(int fd, int ready_fd, unsigned long expected) +{ + unsigned long page = getauxval(AT_PAGESZ); + unsigned char *mapping, *remote; + struct channel *channel; + struct ready ready; + size_t i, begin = page - 13, end = 3 * page + 16; + unsigned int round, wakes = 0; + int ret; + + if (page != expected) + return 10; + mapping = mmap(NULL, page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + page == QUARTER ? QUARTER : 0); + remote = mmap(NULL, 5 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping == MAP_FAILED || remote == MAP_FAILED) + return 11; + channel = (void *)(mapping + (page == QUARTER ? 0 : QUARTER)); + for (i = 0; i < 4 * page; i++) + remote[i] = pattern(i); + if (mprotect(remote + 4 * page, page, PROT_NONE)) + return 12; + ready = (struct ready){ (uintptr_t)remote, page }; + if (write(ready_fd, &ready, sizeof(ready)) != sizeof(ready)) + return 13; + close(ready_fd); + for (round = 0; round < ROUNDS; round++) { + if (wait_turn(channel, 1) || + channel->value != (round ^ 0xabcddcbaU)) + return 14; + if (!round) { + for (i = 0; i < 4 * page; i++) + if (remote[i] != + (pattern(i) ^ + (i >= begin && i < end ? 0x5a : 0))) + return 15; + } + channel->value = ~(round ^ 0xabcddcbaU); + ret = send_turn(channel, 0); + if (ret < 0) + return 16; + wakes += ret; + } + channel->worker_wakes = wakes; + return 0; +} + +static bool remote_copies(pid_t pid, const struct ready *ready) +{ + size_t begin = ready->page - 13, len = 2 * ready->page + 29, i; + unsigned char *data = malloc(len), edge[2] = {}; + struct iovec local = { data, len }, + remote = { (void *)(ready->base + begin), len }; + bool ok = data != NULL; + + if (!ok) + return false; + ok = process_vm_readv(pid, &local, 1, &remote, 1, 0) == (ssize_t)len; + for (i = 0; ok && i < len; i++) + ok = data[i] == pattern(i + begin); + for (i = 0; i < len; i++) + data[i] = pattern(i + begin) ^ 0x5a; + ok &= process_vm_writev(pid, &local, 1, &remote, 1, 0) == (ssize_t)len; + local = (struct iovec){ edge, sizeof(edge) }; + remote = (struct iovec){ (void *)(ready->base + 4 * ready->page - 1), + sizeof(edge) }; + ok &= process_vm_readv(pid, &local, 1, &remote, 1, 0) == 1; + ok &= edge[0] == pattern(4 * ready->page - 1); + free(data); + return ok; +} + +int main(int argc, char **argv) +{ + unsigned long page = getauxval(AT_PAGESZ), other; + unsigned char *mapping, *private; + struct channel *channel; + struct ready ready = {}; + int fd, pipefd[2], status = 0, wakes = 0, ret; + unsigned int round = 0; + pid_t pid; + bool ok; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10)); + if (argc != 1 || (page != QUARTER && page != NATIVE)) + return 2; + other = page == NATIVE ? QUARTER : NATIVE; + fd = memfd_create("mixed-process", 0); + if (fd < 0 || ftruncate(fd, 2 * NATIVE) || pipe(pipefd)) + return 3; + mapping = mmap(NULL, 2 * NATIVE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + 0); + private = + mmap(NULL, NATIVE, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (mapping == MAP_FAILED || private == MAP_FAILED) + return 4; + channel = (void *)(mapping + QUARTER); + private[QUARTER] = 0x7e; + pid = fork(); + if (!pid) { + char fdarg[32], pipearg[32], sizearg[32]; + + close(pipefd[0]); + snprintf(fdarg, sizeof(fdarg), "%d", fd); + snprintf(pipearg, sizeof(pipearg), "%d", pipefd[1]); + snprintf(sizearg, sizeof(sizearg), "%lu", other); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, other, 0UL, 0UL, 0UL)) + _exit(20); + execl("/mixed-process", "/mixed-process", "--worker", fdarg, + pipearg, sizearg, NULL); + _exit(21); + } + close(pipefd[1]); + if (pid < 0) + return 5; + ok = read(pipefd[0], &ready, sizeof(ready)) == sizeof(ready) && + ready.page == other; + close(pipefd[0]); + ok = ok && remote_copies(pid, &ready); + printf("%s - process_vm copies %lu -> %lu, including partial inaccessible boundary\n", + ok ? "ok" : "not ok", page, other); + for (round = 0; ok && round < ROUNDS; round++) { + channel->value = round ^ 0xabcddcbaU; + ret = send_turn(channel, 1); + ok = ret >= 0 && !wait_turn(channel, 0) && + channel->value == ~(round ^ 0xabcddcbaU); + if (ret > 0) + wakes += ret; + } + if (!ok) + kill(pid, SIGKILL); + ret = waitpid(pid, &status, 0); + ok &= ret == pid && WIFEXITED(status) && !WEXITSTATUS(status); + ok &= wakes > 0 && channel->worker_wakes > 0 && + private[QUARTER] == 0x7e; + ok &= mapping[0] == 0 && mapping[2 * QUARTER] == 0 && + mapping[NATIVE - 1] == 0; + printf("%s - mixed %lu/%lu shared futex/COW: rounds=%u wakes=%d/%u child=%#x\n", + ok ? "ok" : "not ok", page, other, round, wakes, + channel->worker_wakes, status); + munmap(private, NATIVE); + munmap(mapping, 2 * NATIVE); + close(fd); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/mlock-fixture-init.c b/tools/testing/selftests/arm64/user4k/mlock-fixture-init.c new file mode 100644 index 00000000000000..0dc94eabcc4fbf --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/mlock-fixture-init.c @@ -0,0 +1,249 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static long locked_kb(void) +{ + char line[256]; + long value = -1; + FILE *f = fopen("/proc/self/status", "re"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "VmLck: %ld", &value) == 1) + break; + fclose(f); + return value; +} + +static bool wait_ok(pid_t pid) +{ + int status = 0; + + return pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); +} + +static bool remap_locked(unsigned long ps) +{ + unsigned char *p, *q, *target; + unsigned long kb = ps / 1024; + bool ok; + + p = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED || mlock(p, 2 * ps)) + ksft_exit_fail_msg("mlock remap setup\n"); + memset(p, 0x37, 2 * ps); + ok = locked_kb() == (long)(2 * kb); + q = mremap(p, 2 * ps, 4 * ps, MREMAP_MAYMOVE); + if (q == MAP_FAILED) + ksft_exit_fail_msg("locked remap grow: %s\n", strerror(errno)); + ok &= locked_kb() == (long)(4 * kb); + for (unsigned long i = 0; i < 4 * ps; i++) + ok &= q[i] == (i < 2 * ps ? 0x37 : 0); + p = mremap(q, 4 * ps, 3 * ps, 0); + if (p == MAP_FAILED) + ksft_exit_fail_msg("locked remap shrink: %s\n", + strerror(errno)); + ok &= locked_kb() == (long)(3 * kb); + p[2 * ps] = 0xb6; + ok &= !munmap(p + ps, ps) && locked_kb() == (long)(2 * kb); + target = mmap(NULL, ps, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (target == MAP_FAILED) + ksft_exit_fail_msg("locked remap target\n"); + q = mremap(p, ps, ps, MREMAP_MAYMOVE | MREMAP_FIXED, target); + if (q == MAP_FAILED) + ksft_exit_fail_msg("locked remap move: %s\n", strerror(errno)); + ok &= q == target && locked_kb() == (long)(2 * kb); + for (unsigned long i = 0; i < ps; i++) + ok &= q[i] == 0x37; + ok &= p[2 * ps] == 0xb6; + ok &= !munmap(q, ps) && locked_kb() == (long)kb; + ok &= !munmap(p + 2 * ps, ps) && locked_kb() == 0; + return ok; +} + +static int exercise(void) +{ + unsigned long ps = getauxval(AT_PAGESZ), kb = ps / 1024; + unsigned char *p, *q, residency[4]; + int gate[2]; + pid_t pid; + bool ok; + + ksft_print_header(); + ksft_set_plan(9); + p = mmap(NULL, 32 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED || locked_kb() != 0) + ksft_exit_fail_msg("initial mapping/accounting\n"); + ok = !mlock(p + ps + 17, ps - 32) && locked_kb() == (long)kb; + ok &= !mincore(p + ps, ps, residency) && residency[0] & 1; + p[ps] = 0x51; + ksft_test_result( + ok, "%luK unaligned mlock uses exactly one user leaf\n", kb); + ok = !mlock(p + ps, ps) && locked_kb() == (long)kb; + ok &= !mlock(p + 2 * ps, ps) && locked_kb() == (long)(2 * kb); + ok &= !munlock(p + ps, ps) && locked_kb() == (long)kb; + p[2 * ps] = 0x62; + errno = 0; + ok &= madvise(p + 2 * ps, ps, MADV_PAGEOUT) == -1 && errno == EINVAL && + p[2 * ps] == 0x62; + ok &= !munlock(p + 2 * ps, ps) && locked_kb() == 0; + ksft_test_result( + ok, "%luK repeat lock and independent adjacent unlock\n", kb); + ok = !mlock2(p + 4 * ps, 4 * ps, MLOCK_ONFAULT) && + locked_kb() == (long)(4 * kb); + ok &= !mincore(p + 4 * ps, 4 * ps, residency); + for (unsigned int i = 0; i < 4; i++) + ok &= !(residency[i] & 1); + p[4 * ps] = 0x74; + ok &= !mincore(p + 4 * ps, ps, residency) && residency[0] & 1; + ok &= !munlock(p + 4 * ps, 4 * ps) && locked_kb() == 0; + ksft_test_result( + ok, "%luK MLOCK_ONFAULT accounting and deferred faults\n", kb); + q = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_LOCKED, -1, 0); + ok = q != MAP_FAILED && locked_kb() == (long)(4 * kb); + if (q == MAP_FAILED || pipe(gate)) + ksft_exit_fail_msg("locked mmap/fork setup\n"); + memset(q, 0x8a, 4 * ps); + pid = fork(); + if (!pid) { + char token; + + close(gate[1]); + if (read(gate[0], &token, 1) != 1 || locked_kb() != 0) + _exit(2); + for (unsigned long i = 0; i < 4 * ps; i++) + if (q[i] != 0x8a) + _exit(3); + _exit(0); + } + close(gate[0]); + memset(q + ps, 0x9b, ps); + ok &= write(gate[1], "x", 1) == 1; + close(gate[1]); + ok &= wait_ok(pid) && locked_kb() == (long)(4 * kb); + ok &= !munmap(q, 4 * ps) && locked_kb() == 0; + ksft_test_result( + ok, "%luK MAP_LOCKED, fork noninheritance and parent COW\n", + kb); + ok = !mlockall(MCL_FUTURE | MCL_ONFAULT); + q = mmap(NULL, ps, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, + -1, 0); + ok &= q != MAP_FAILED && locked_kb() == (long)kb; + if (q != MAP_FAILED) + *q = 0xaf; + ok &= !munlockall() && locked_kb() == 0; + if (q != MAP_FAILED) + munmap(q, ps); + ksft_test_result(ok, "%luK future on-fault locks and munlockall\n", kb); + pid = fork(); + if (!pid) { + struct rlimit limit = { ps, ps }; + + if (setrlimit(RLIMIT_MEMLOCK, &limit) || setuid(1000) || + mlock(p, ps)) + _exit(4); + errno = 0; + if (mlock(p + ps, ps) != -1 || errno != ENOMEM || + locked_kb() != (long)kb) + _exit(5); + _exit(0); + } + ksft_test_result( + wait_ok(pid), + "%luK unprivileged RLIMIT_MEMLOCK in user-page units\n", kb); + pid = fork(); + if (!pid) { + if (mlockall(MCL_CURRENT) || locked_kb() <= (long)kb || + munlockall() || locked_kb() != 0) + _exit(6); + _exit(0); + } + ksft_test_result(wait_ok(pid), "%luK lock/unlock current mappings\n", + kb); + { + unsigned long huge = ((1UL << 31) + 1) * ps; + + q = mmap(NULL, huge, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + ok = q != MAP_FAILED; + if (q != MAP_FAILED) { + ok &= !mlock2(q, huge, MLOCK_ONFAULT) && + locked_kb() == (long)(huge / 1024); + ok &= !munlock(q, huge) && locked_kb() == 0; + munmap(q, huge); + } + ksft_test_result( + ok, + "%luK deferred lock count beyond signed 32-bit leaves\n", + kb); + } + ksft_test_result( + remap_locked(ps), + "%luK locked grow/shrink/move and partial unmap accounting\n", + kb); + munmap(p, 32 * ps); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} + +int main(int argc, char **argv) +{ + bool ok = true; + unsigned long native = getauxval(AT_PAGESZ); + unsigned long granules[] = { native, 4096, 16384 }; + unsigned int nr = native > 16384 ? 3 : 2; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--test") && getppid() == 1) { + if (getauxval(AT_PAGESZ) != strtoul(argv[2], NULL, 10)) + return 42; + return exercise(); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Run only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL)) + return 3; + for (unsigned int small = 0; small < nr; small++) { + pid_t pid = fork(); + + if (!pid) { + char size[32], host_size[32]; + + snprintf(size, sizeof(size), "%lu", granules[small]); + snprintf(host_size, sizeof(host_size), "%lu", native); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, granules[small], + 0UL, 0UL, 0UL)) + _exit(7); + execl("/init", "/init", "--test", size, host_size, + NULL); + _exit(8); + } + ok &= wait_ok(pid); + } + printf("%s - mlock fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/mte-core-init.c b/tools/testing/selftests/arm64/user4k/mte-core-init.c new file mode 100644 index 00000000000000..3a6cf5e0723b7b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/mte-core-init.c @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* MTE core-file validation: PID 1 and its children in a disposable VM only. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef PT_AARCH64_MEMTAG_MTE +#define PT_AARCH64_MEMTAG_MTE (PT_LOPROC + 0x2) +#endif +#define TAGGED_BASE 0x200000000UL +#define ZERO_BASE 0x210000000UL +#define EXCLUDED_BASE 0x220000000UL +#define REQUIRE(condition) do { \ + if (!(condition)) { \ + fprintf(stderr, "FAIL line %d: %s errno=%d\n", __LINE__, #condition, errno); \ + exit(1); \ + } \ +} while (0) + +static void write_setting(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY); + + REQUIRE(fd >= 0); + REQUIRE(write(fd, value, strlen(value)) == (ssize_t)strlen(value)); + REQUIRE(!close(fd)); +} + +static unsigned char *tagged_map(unsigned long address, size_t size) +{ + void *p = mmap((void *)address, size, PROT_READ | PROT_WRITE | PROT_MTE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + + REQUIRE(p == (void *)address); + return p; +} + +static void set_tags(unsigned char *p, size_t size, unsigned int tag) +{ + for (size_t i = 0; i < size; i += 16) { + uintptr_t address = (uintptr_t)(p + i) | ((uintptr_t)tag << 56); + + asm volatile(".arch_extension memtag\n\tstg %0, [%0]" + : : "r" (address) : "memory"); + } +} + +static void child(unsigned long ps) +{ + struct stat self, init; + struct rlimit limit = { 64UL << 20, 64UL << 20 }; + unsigned char *p; + + REQUIRE(getppid() == 1); + REQUIRE(!stat("/proc/self/exe", &self) && !stat("/proc/1/exe", &init)); + REQUIRE(self.st_dev == init.st_dev && self.st_ino == init.st_ino); + REQUIRE(getauxval(AT_PAGESZ) == ps); + REQUIRE(!setrlimit(RLIMIT_CORE, &limit)); + REQUIRE(!prctl(PR_SET_DUMPABLE, 1UL, 0UL, 0UL, 0UL)); + REQUIRE(!prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0UL, 0UL, 0UL)); + write_setting("/proc/self/coredump_filter", "1"); + p = tagged_map(TAGGED_BASE, 3 * ps); + for (unsigned int i = 0; i < 3; i++) { + memset(p + i * ps, 0x40 + i, ps); + set_tags(p + i * ps, ps, i + 1); + } + /* Split metadata at a non-native boundary and retain the physical offset. */ + REQUIRE(!mprotect(p + ps, ps, PROT_READ | PROT_MTE)); + p = tagged_map(ZERO_BASE, ps); + *p = 0; + /* Keep anonymous dump eligibility while leaving the leaf absent. */ + REQUIRE(!madvise(p, ps, MADV_DONTNEED)); + p = tagged_map(EXCLUDED_BASE, ps); + memset(p, 0x55, ps); + set_tags(p, ps, 7); + REQUIRE(!madvise(p, ps, MADV_DONTDUMP)); + raise(SIGABRT); + _exit(2); +} + +static void inspect_core(pid_t pid, unsigned long ps) +{ + char path[80]; + Elf64_Ehdr eh; + struct stat st; + unsigned int tagged = 0, loads = 0, zero = 0, excluded = 0; + unsigned char *bytes = malloc(ps); + int fd; + + REQUIRE(bytes); + snprintf(path, sizeof(path), "/tmp/core.%d", pid); + fd = open(path, O_RDONLY); + REQUIRE(fd >= 0 && !fstat(fd, &st)); + REQUIRE(pread(fd, &eh, sizeof(eh), 0) == sizeof(eh)); + REQUIRE(!memcmp(eh.e_ident, ELFMAG, SELFMAG)); + REQUIRE(eh.e_ident[EI_CLASS] == ELFCLASS64 && eh.e_type == ET_CORE); + REQUIRE(eh.e_phentsize == sizeof(Elf64_Phdr) && eh.e_phnum < 4096); + for (unsigned int n = 0; n < eh.e_phnum; n++) { + Elf64_Phdr ph; + unsigned char expected; + unsigned int index; + size_t length; + + REQUIRE(pread(fd, &ph, sizeof(ph), eh.e_phoff + n * sizeof(ph)) == sizeof(ph)); + if (ph.p_type != PT_LOAD && ph.p_type != PT_AARCH64_MEMTAG_MTE) + continue; + if (ph.p_vaddr >= TAGGED_BASE && ph.p_vaddr < TAGGED_BASE + 3 * ps) { + index = (ph.p_vaddr - TAGGED_BASE) / ps; + REQUIRE(ph.p_vaddr == TAGGED_BASE + index * ps && ph.p_memsz == ps); + if (ph.p_type == PT_LOAD) { + loads++; + length = ps; + expected = 0x40 + index; + } else { + tagged++; + length = ps / 32; + expected = (index + 1) * 0x11; + } + } else if (ph.p_vaddr == ZERO_BASE && ph.p_type == PT_AARCH64_MEMTAG_MTE) { + zero++; + length = ps / 32; + expected = 0; + REQUIRE(ph.p_memsz == ps); + } else if (ph.p_vaddr == EXCLUDED_BASE && ph.p_type == PT_AARCH64_MEMTAG_MTE) { + excluded++; + REQUIRE(ph.p_filesz == 0 && ph.p_memsz == ps); + continue; + } else { + continue; + } + if (ph.p_filesz != length) + fprintf(stderr, "segment type=%#x address=%#lx size=%lu expected=%zu\n", + ph.p_type, (unsigned long)ph.p_vaddr, + (unsigned long)ph.p_filesz, length); + REQUIRE(ph.p_filesz == length); + REQUIRE(ph.p_offset <= (uint64_t)st.st_size && length <= st.st_size - ph.p_offset); + REQUIRE(pread(fd, bytes, length, ph.p_offset) == (ssize_t)length); + for (size_t i = 0; i < length; i++) + REQUIRE(bytes[i] == expected); + } + REQUIRE(tagged == 3 && loads == 3 && zero == 1 && excluded == 1); + REQUIRE(!close(fd) && !unlink(path)); + free(bytes); + printf("MTE CORE CASE PASS page-size=%lu\n", ps); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + unsigned long sizes[] = { native, 4096, 16384 }; + unsigned int count = native > 16384 ? 3 : native > 4096 ? 2 : 1; + + setbuf(stdout, NULL); + setbuf(stderr, NULL); + if (argc == 3 && !strcmp(argv[1], "--child")) { + child(strtoul(argv[2], NULL, 10)); + return 1; + } + if (getpid() != 1) { + fprintf(stderr, "SKIP: disposable guest PID1 only\n"); + return 4; + } + REQUIRE(getauxval(AT_HWCAP2) & HWCAP2_MTE); + mkdir("/proc", 0755); + mkdir("/tmp", 0700); + REQUIRE(!mount("proc", "/proc", "proc", 0, NULL)); + REQUIRE(!mount("tmpfs", "/tmp", "tmpfs", 0, "mode=0700")); + write_setting("/proc/sys/kernel/core_pattern", "/tmp/core.%p"); + for (unsigned int i = 0; i < count; i++) { + pid_t pid = fork(); + int status; + + REQUIRE(pid >= 0); + if (!pid) { + char value[32]; + + snprintf(value, sizeof(value), "%lu", sizes[i]); + REQUIRE(!prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)); + execl("/init", "/init", "--child", value, NULL); + _exit(3); + } + REQUIRE(waitpid(pid, &status, 0) == pid); + REQUIRE(WIFSIGNALED(status) && WTERMSIG(status) == SIGABRT && WCOREDUMP(status)); + inspect_core(pid, sizes[i]); + } + printf("MTE CORE PASS cases=%u\n", count); + reboot(RB_POWER_OFF); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/numa-balancing-init.c b/tools/testing/selftests/arm64/user4k/numa-balancing-init.c new file mode 100644 index 00000000000000..2ea31bda12968a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/numa-balancing-init.c @@ -0,0 +1,307 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Disposable two-node guest: real automatic NUMA hinting and migration. + * Volatile accesses intentionally trigger page faults and repeated accesses; + * thread coordination uses an atomic flag, not volatile storage. + */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static void check(int good, const char *expr, int line) +{ + if (!good) { + fprintf(stderr, "FAIL line %d: %s errno=%d\n", line, expr, + errno); + exit(1); + } +} +#define CHECK(x) check(!!(x), #x, __LINE__) + +static void setting(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY); + + CHECK(fd >= 0); + CHECK(write(fd, value, strlen(value)) == (ssize_t)strlen(value)); + CHECK(!close(fd)); +} + +static void cpu(int id) +{ + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(id, &set); + CHECK(!sched_setaffinity(0, sizeof(set), &set)); +} + +static unsigned long counter(const char *key) +{ + FILE *f = fopen("/proc/vmstat", "r"); + char name[128]; + unsigned long value; + + CHECK(f); + while (fscanf(f, "%127s %lu", name, &value) == 2) { + if (!strcmp(name, key)) { + CHECK(!fclose(f)); + return value; + } + } + CHECK(0); + return 0; +} + +static double now(void) +{ + struct timespec ts; + + CHECK(!clock_gettime(CLOCK_MONOTONIC, &ts)); + return ts.tv_sec + ts.tv_nsec / 1e9; +} + +static int node_at(void *address) +{ + int node = -1; + + CHECK(!syscall(SYS_get_mempolicy, &node, NULL, 0UL, address, + MPOL_F_ADDR | MPOL_F_NODE)); + return node; +} + +static unsigned long sample(unsigned char *map, unsigned long length, + unsigned long page) +{ + unsigned long moved = 0; + + for (unsigned long off = 0; off < length; off += page) + if (off != 2 * page) + moved += node_at(map + off + 16) == 1; + return moved; +} + +static void forbidden(unsigned char *address, int write_access) +{ + pid_t child = fork(); + int status; + + CHECK(child >= 0); + if (!child) { + if (write_access) + *(volatile unsigned char *)address = 0x73; + else + (void)*(volatile unsigned char *)address; + _exit(2); + } + CHECK(waitpid(child, &status, 0) == child); + CHECK(WIFSIGNALED(status) && WTERMSIG(status) == SIGSEGV); +} + +struct local_reader { + unsigned char *map; + unsigned long length, page; + atomic_int stop; +}; + +static void *read_local(void *arg) +{ + struct local_reader *r = arg; + + cpu(1); + while (!atomic_load_explicit(&r->stop, memory_order_relaxed)) { + for (unsigned long off = 0; off < r->length; off += r->page) { + if (off == 2 * r->page) + continue; + CHECK(*(volatile uint64_t *)(r->map + off) == + 0x81720000 + off / r->page); + } + } + return NULL; +} + +static void local_restore(unsigned char *map, unsigned long length, + unsigned long page) +{ + struct local_reader r = { .map = map, .length = length, .page = page }; + unsigned long hints = counter("numa_hint_faults_local"); + pthread_t worker; + double deadline = now() + 2; + + /* Two threads permit hinting of already-local private mappings. */ + CHECK(!pthread_create(&worker, NULL, read_local, &r)); + do { + for (unsigned long off = 0; off < length; off += page) { + if (off == 2 * page) + continue; + CHECK(*(volatile uint64_t *)(map + off) == + 0x81720000 + off / page); + } + } while (now() < deadline); + atomic_store_explicit(&r.stop, 1, memory_order_relaxed); + CHECK(!pthread_join(worker, NULL)); + hints = counter("numa_hint_faults_local") - hints; + printf("NUMAB local page=%lu hints=%lu\n", page, hints); + CHECK(hints > 0); +} + +static void exercise(void) +{ + unsigned long page = getauxval(AT_PAGESZ), length = 8UL << 20; + unsigned long mask = 1, allowed = 0, hints, migrated, iterations = 0; + unsigned long moved = 0, total = length / page - 1; + unsigned char *map; + int mode; + + CHECK(!syscall(SYS_get_mempolicy, &mode, &allowed, 64UL, NULL, + MPOL_F_MEMS_ALLOWED)); + CHECK(allowed == 3); + cpu(0); + map = mmap(NULL, length, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(map != MAP_FAILED); + CHECK(!madvise(map, length, MADV_NOHUGEPAGE)); + CHECK(!syscall(SYS_mbind, map, length, MPOL_BIND, &mask, 64UL, 0)); + for (unsigned long off = 0; off < length; off += page) { + *(uint64_t *)(map + off) = 0x81720000 + off / page; + *(uint64_t *)(map + off + 8) = 0; + *(uint64_t *)(map + off + page - 8) = 0x61320000 + off / page; + CHECK(node_at(map + off + 16) == 0); + } + CHECK(!mprotect(map + page, page, PROT_READ)); + CHECK(!mprotect(map + 2 * page, page, PROT_NONE)); + CHECK(!syscall(SYS_mbind, map, length, MPOL_DEFAULT, NULL, 0UL, 0)); + cpu(1); + hints = counter("numa_hint_faults"); + migrated = counter("numa_pages_migrated"); + double started = now(), deadline = started + 20; + + do { + for (unsigned long off = 0; off < length; off += page) { + volatile uint64_t *p = (void *)(map + off); + + if (off == 2 * page) + continue; + CHECK(p[0] == 0x81720000 + off / page); + CHECK(p[page / 8 - 1] == 0x61320000 + off / page); + if (off != page) + p[1]++; + } + iterations++; + if (!(iterations % 128)) { + moved = sample(map, length, page); + if (moved == total) + break; + } + } while (now() < deadline); + moved = sample(map, length, page); + hints = counter("numa_hint_faults") - hints; + migrated = counter("numa_pages_migrated") - migrated; + printf("NUMAB page=%lu moved=%lu/%lu hints=%lu migrated=%lu loops=%lu seconds=%.3f\n", + page, moved, total, hints, migrated, iterations, + now() - started); + CHECK(moved == total && hints > 0 && migrated > 0); + for (unsigned long off = 0; off < length; off += page) { + if (off != page && off != 2 * page) + CHECK(*(uint64_t *)(map + off + 8) == iterations); + } + local_restore(map, length, page); + forbidden(map + page, 1); + forbidden(map + 2 * page, 0); + CHECK(!mprotect(map + 2 * page, page, PROT_READ)); + CHECK(*(uint64_t *)(map + 2 * page) == 0x81720002); + CHECK(!munmap(map, length)); + printf("ok - NUMAB page=%lu automatic placement, bytes and permissions\n", + page); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ), + sizes[] = { 4096, 16384, 65536 }; + int failed = 0, compat_only = 0, no4k = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--case")) { + CHECK(getauxval(AT_PAGESZ) == strtoul(argv[2], NULL, 10)); + exercise(); + return 0; + } + CHECK(getpid() == 1); + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--compat-only")) + compat_only = 1; + else if (!strcmp(argv[i], "--no4k")) + no4k = 1; + else + CHECK(0); + } + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + CHECK(!mount("debugfs", "/debug", "debugfs", 0, NULL)); + setting("/proc/sys/kernel/numa_balancing", "1\n"); + setting("/debug/sched/numa_balancing/scan_delay_ms", "100\n"); + setting("/debug/sched/numa_balancing/scan_period_min_ms", "100\n"); + setting("/debug/sched/numa_balancing/scan_period_max_ms", "100\n"); + setting("/debug/sched/numa_balancing/scan_size_mb", "16\n"); + for (unsigned int i = 0; i < 3; i++) { + pid_t child; + int status; + + if (compat_only || sizes[i] > native || + (no4k && sizes[i] == 4096)) + continue; + child = fork(); + CHECK(child >= 0); + if (!child) { + char s[32]; + + snprintf(s, sizeof(s), "%lu", sizes[i]); + CHECK(!prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], + 0UL, 0UL, 0UL)); + execl("/init", "/init", "--case", s, NULL); + _exit(2); + } + CHECK(waitpid(child, &status, 0) == child); + failed |= !WIFEXITED(status) || WEXITSTATUS(status); + } + CHECK(!access("/compat-numa-balancing-probe", X_OK)); + pid_t child = fork(); + int status; + + CHECK(child >= 0); + if (!child) { + char s[32]; + + snprintf(s, sizeof(s), "%lu", no4k ? native : 4096UL); + execl("/compat-numa-balancing-probe", + "/compat-numa-balancing-probe", s, NULL); + _exit(2); + } + CHECK(waitpid(child, &status, 0) == child); + failed |= !WIFEXITED(status) || WEXITSTATUS(status); + printf("A32 NUMAB expected-page=%lu status=%d\n", + no4k ? native : 4096UL, status); + printf("NUMA BALANCING %s\n", failed ? "FAIL" : "PASS"); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/numa-policy-init.c b/tools/testing/selftests/arm64/user4k/numa-policy-init.c new file mode 100644 index 00000000000000..fedcedb529709f --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/numa-policy-init.c @@ -0,0 +1,236 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable two-node guest: policy ranges and native physical placement. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static void check(int good, const char *expr, int line) +{ + if (!good) { + fprintf(stderr, "FAIL line %d: %s errno=%d\n", line, expr, + errno); + exit(1); + } +} +#define CHECK(x) check(!!(x), #x, __LINE__) + +static long bind_to(void *addr, unsigned long bytes, int node, + unsigned int flags) +{ + unsigned long mask = 1UL << node; + + return syscall(SYS_mbind, addr, bytes, MPOL_BIND, &mask, 64UL, flags); +} + +static void policy_at(void *addr, int node) +{ + unsigned long mask = 0; + int mode = -1; + + CHECK(!syscall(SYS_get_mempolicy, &mode, &mask, 64UL, addr, + MPOL_F_ADDR)); + CHECK(mode == MPOL_BIND && mask == (1UL << node)); +} + +static void node_at(void *addr, int expected) +{ + int node = -1; + + CHECK(!syscall(SYS_get_mempolicy, &node, NULL, 0UL, addr, + MPOL_F_NODE | MPOL_F_ADDR)); + if (node != expected) + fprintf(stderr, "node=%d expected=%d addr=%p\n", node, expected, + addr); + CHECK(node == expected); +} + +static void exercise(unsigned long native, int baseline) +{ + unsigned long page = getauxval(AT_PAGESZ), allowed = 0; + unsigned char *reserve, *base, *shared, *alias; + int mode, fd; + + CHECK(!syscall(SYS_get_mempolicy, &mode, &allowed, 64UL, NULL, + MPOL_F_MEMS_ALLOWED)); + CHECK(allowed == + 3); /* Require two real memory nodes, not a silent skip. */ + reserve = mmap(NULL, 4 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + base = (void *)(((uintptr_t)reserve + native - 1) & ~(native - 1)); + CHECK(!bind_to(base, 2 * native, 0, 0)); + if (baseline) { + errno = 0; + long ret = bind_to(base + page, 1, 1, 0); + + CHECK(page < native ? ret == -1 && errno == EINVAL : ret == 0); + printf("ok - baseline native=%lu page=%lu mbind=%ld errno=%d\n", + native, page, ret, errno); + CHECK(!munmap(reserve, 4 * native)); + return; + } + CHECK(!bind_to(base + page, 1, 1, 0)); + policy_at(base, 0); + policy_at(base + page, 1); + if (page < native) + policy_at(base + 2 * page, 0); + CHECK(!syscall(SYS_set_mempolicy_home_node, base + page, 1UL, 1UL, + 0UL)); + errno = 0; + CHECK(bind_to(base + 1, page, 1, 0) == -1 && errno == EINVAL); + for (unsigned long off = 0; off < 2 * native; off += page) { + base[off + 17] = 0x31 + off / page; + node_at(base + off + 17, off == page ? 1 : 0); + } + CHECK(!bind_to(base + page, page, 1, MPOL_MF_STRICT)); + errno = 0; + CHECK(bind_to(base, 2 * native, 1, MPOL_MF_STRICT) == -1 && + errno == EIO); + policy_at(base, 0); + CHECK(!munmap(reserve, 4 * native)); + + /* A split anonymous VMA must carry its native interleave index correctly. */ + reserve = mmap(NULL, 4 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + base = (void *)(((uintptr_t)reserve + native - 1) & ~(native - 1)); + unsigned long mask = 3, inset = page < native ? page : 0; + + CHECK(!syscall(SYS_mbind, base + inset, 2 * native - 2 * inset, + MPOL_INTERLEAVE, &mask, 64UL, 0)); + for (unsigned long off = inset; off < 2 * native - inset; off += page) { + base[off + 17] = 0x43; + node_at(base + off + 17, ((uintptr_t)base + off) / native % 2); + } + CHECK(!munmap(reserve, 4 * native)); + + /* Moving a fragment moves its native physical owner, retaining bytes. */ + reserve = mmap(NULL, 4 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + base = (void *)(((uintptr_t)reserve + native - 1) & ~(native - 1)); + CHECK(!bind_to(base, 2 * native, 0, 0)); + for (unsigned long off = 0; off < 2 * native; off += page) + base[off + 17] = 0x51 + off / page; + CHECK(!bind_to(base + native - page, page, 1, + MPOL_MF_MOVE_ALL | MPOL_MF_STRICT)); + for (unsigned long off = 0; off < 2 * native; off += page) { + CHECK(base[off + 17] == (unsigned char)(0x51 + off / page)); + node_at(base + off + 17, off < native ? 1 : 0); + } + CHECK(!bind_to(base, 2 * native, 0, MPOL_MF_MOVE_ALL | MPOL_MF_STRICT)); + for (unsigned long off = 0; off < 2 * native; off += page) + node_at(base + off + 17, 0); + CHECK(!munmap(reserve, 4 * native)); + + fd = memfd_create("numa-shared", 0); + CHECK(fd >= 0 && !ftruncate(fd, 2 * native)); + shared = mmap(NULL, 2 * native, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + 0); + CHECK(shared != MAP_FAILED); + CHECK(!bind_to(shared, native, 0, 0)); + CHECK(!bind_to(shared + native, native, 1, 0)); + if (page < native) { + alias = mmap(NULL, 2 * native - page, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, page); + CHECK(alias != MAP_FAILED); + policy_at(alias, 0); + policy_at(alias + native - page, 1); + errno = 0; + CHECK(bind_to(alias, page, 1, 0) == -1 && errno == EINVAL); + policy_at(alias, 0); + policy_at(shared, 0); + CHECK(!bind_to(alias + native - page, native, 1, 0)); + for (unsigned long off = 0; off < 2 * native - page; + off += page) { + alias[off + 17] = 0x72; + node_at(alias + off + 17, off + page < native ? 0 : 1); + CHECK(shared[off + page + 17] == 0x72); + } + CHECK(!munmap(alias, 2 * native - page)); + } + CHECK(!munmap(shared, 2 * native) && !close(fd)); + printf("ok - NUMA native=%lu page=%lu policy, placement, migration and shared offsets\n", + native, page); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ), + sizes[] = { 4096, 16384, 65536 }; + int failed = 0, baseline = 0, no4; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--case")) { + CHECK(getauxval(AT_PAGESZ) == strtoul(argv[3], NULL, 10)); + exercise(strtoul(argv[2], NULL, 10), atoi(argv[4])); + return 0; + } + CHECK(getpid() == 1); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + FILE *cmdline = fopen("/proc/cmdline", "r"); + char line[4096]; + + CHECK(cmdline && fgets(line, sizeof(line), cmdline)); + baseline = strstr(line, "numa_fixture_baseline=1") != NULL; + no4 = strstr(line, "id_aa64mmfr0.tgran4=f") != NULL; + CHECK(!fclose(cmdline)); + for (unsigned int i = 0; i < 3; i++) { + int status; + pid_t child; + + if (sizes[i] > native || (no4 && sizes[i] == 4096)) + continue; + child = fork(); + CHECK(child >= 0); + if (!child) { + char n[32], s[32], b[4]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(s, sizeof(s), "%lu", sizes[i]); + snprintf(b, sizeof(b), "%d", baseline); + CHECK(!prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], + 0UL, 0UL, 0UL)); + execl("/init", "/init", "--case", n, s, b, NULL); + _exit(2); + } + CHECK(waitpid(child, &status, 0) == child); + failed |= !WIFEXITED(status) || WEXITSTATUS(status); + } + if (!baseline) { + int status; + pid_t child = fork(); + + CHECK(child >= 0); + if (!child) { + char n[32], s[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(s, sizeof(s), "%lu", no4 ? native : 4096UL); + execl("/compat-numa-probe", "/compat-numa-probe", n, s, + NULL); + _exit(2); + } + CHECK(waitpid(child, &status, 0) == child); + failed |= !WIFEXITED(status) || WEXITSTATUS(status); + printf("A32 NUMA expected-page=%lu status=%d\n", + no4 ? native : 4096UL, status); + } + printf("NUMA GRANULE %s\n", failed ? "FAIL" : "PASS"); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/page-contract.c b/tools/testing/selftests/arm64/user4k/page-contract.c new file mode 100644 index 00000000000000..316ffe48d0a476 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/page-contract.c @@ -0,0 +1,232 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Run unchanged under the current process ABI, including native 4K/16K/64K. + * Explicit --expect-{4k,16k,64k} options assert the selected size; they never + * launch a different ABI or emulate it. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../kselftest.h" + +#define X18_SENTINEL UINT64_C(0xfedcba9876543210) + +extern long x18_syscall(long nr, long arg0, long arg1, long arg2, + uint64_t *seen); + +static volatile sig_atomic_t signal_seen; +static uint64_t signal_x18; + +static void x18_signal(int sig, siginfo_t *info, void *context) +{ + ucontext_t *uc = context; + + (void)sig; + (void)info; + signal_x18 = uc->uc_mcontext.regs[18]; + signal_seen = 1; + /* rt_sigreturn must restore the interrupted value, not this value. */ + /* Built with -ffixed-x18: the compiler never allocates this register. */ + asm volatile("mov x18, xzr"); +} + +static bool wait_for(pid_t child, int expected_signal) +{ + int status; + pid_t ret; + + if (child < 0) + return false; + do { + ret = waitpid(child, &status, 0); + } while (ret < 0 && errno == EINTR); + if (ret != child) + return false; + if (expected_signal) + return WIFSIGNALED(status) && + WTERMSIG(status) == expected_signal; + return WIFEXITED(status) && WEXITSTATUS(status) == 0; +} + +static bool access_faults(volatile unsigned char *ptr, bool write) +{ + pid_t child = fork(); + + if (!child) { + if (write) + *ptr = 0x5a; + else + (void)*ptr; + _exit(1); + } + return wait_for(child, SIGSEGV); +} + +static void private_mappings(size_t page) +{ + unsigned char *p = mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + bool ok; + pid_t child; + + if (p == MAP_FAILED) + ksft_exit_fail_msg("anonymous mmap: %s\n", strerror(errno)); + memset(p, 0x41, 4 * page); + if (page > 4096) { + errno = 0; + ok = mprotect(p + 4096, 4096, PROT_READ) == -1 && + errno == EINVAL; + } else { + ok = mprotect(p + page, page, PROT_READ) == 0; + } + ksft_test_result(ok, + "mapping alignment agrees with the selected ABI\n"); + ok = mprotect(p + page, page, PROT_READ) == 0 && + mprotect(p + 2 * page, page, PROT_NONE) == 0; + ksft_test_result( + ok && access_faults(p + page, true) && + access_faults(p + 2 * page, false), + "read-only and inaccessible neighbors fault independently\n"); + p[0] = 0x10; + p[3 * page] = 0x30; + ksft_test_result( + p[page] == 0x41 && p[0] == 0x10 && p[3 * page] == 0x30, + "neighbor permissions do not remove writable mappings\n"); + child = fork(); + if (!child) { + p[0] = 0x99; + _exit(p[3 * page] != 0x30); + } + ksft_test_result(wait_for(child, 0) && p[0] == 0x10, + "fork preserves private COW isolation\n"); + ok = munmap(p + page, page) == 0; + ksft_test_result(ok && access_faults(p + page, false) && p[0] == 0x10 && + p[3 * page] == 0x30, + "single-page unmap preserves surrounding mappings\n"); + munmap(p, 4 * page); +} + +static void shared_mappings(size_t page) +{ + int fd = memfd_create("page-contract", MFD_CLOEXEC); + unsigned char *a, *b; + bool ok = true; + pid_t child; + unsigned int i; + + if (fd < 0 || ftruncate(fd, 4 * page)) + ksft_exit_fail_msg("memfd: %s\n", strerror(errno)); + a = mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + b = mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (a == MAP_FAILED || b == MAP_FAILED) + ksft_exit_fail_msg("shared mmap: %s\n", strerror(errno)); + for (i = 0; i < 4; i++) { + a[i * page] = 0x50 + i; + ok &= b[i * page] == 0x50 + i; + } + ksft_test_result(ok, "shared aliases expose the same bytes\n"); + ok = mprotect(a + page, page, PROT_READ) == 0; + child = fork(); + if (!child) { + b[page] = 0x73; + _exit(a[page] != 0x73); + } + ok &= wait_for(child, 0); + ksft_test_result( + ok && a[page] == 0x73 && access_faults(a + page, true), + "shared writes cross fork while alias permissions stay local\n"); + munmap(a, 4 * page); + munmap(b, 4 * page); + close(fd); +} + +static void x18_contract(void) +{ + struct sigaction sa = { .sa_sigaction = x18_signal, + .sa_flags = SA_SIGINFO }; + struct timespec pause = { .tv_nsec = 1000000 }; + struct rusage before, after; + uint64_t seen; + long ret; + bool ok; + + ret = x18_syscall(SYS_getpid, 0, 0, 0, &seen); + ksft_test_result(ret == getpid() && seen == X18_SENTINEL, + "x18 survives a raw syscall\n"); + ok = getrusage(RUSAGE_SELF, &before) == 0; + ret = x18_syscall(SYS_nanosleep, (long)&pause, 0, 0, &seen); + ok &= getrusage(RUSAGE_SELF, &after) == 0; + ksft_test_result(ok && ret == 0 && seen == X18_SENTINEL && + after.ru_nvcsw > before.ru_nvcsw, + "x18 survives a measured voluntary context switch\n"); + sigemptyset(&sa.sa_mask); + ok = sigaction(SIGUSR1, &sa, NULL) == 0; + if (ok) + ret = x18_syscall(SYS_tgkill, getpid(), syscall(SYS_gettid), + SIGUSR1, &seen); + ksft_test_result( + ok && ret == 0 && signal_seen && signal_x18 == X18_SENTINEL && + seen == X18_SENTINEL, + "signal frame and sigreturn preserve x18 after handler clobber\n"); +} + +int main(int argc, char **argv) +{ + const struct rlimit no_core = { 0, 0 }; + size_t expected = getauxval(AT_PAGESZ); + int result; + + if (argc == 2 && !strcmp(argv[1], "--expect-4k")) { + expected = 4096; + } else if (argc == 2 && !strcmp(argv[1], "--expect-16k")) { + expected = 16384; + } else if (argc == 2 && !strcmp(argv[1], "--expect-64k")) { + expected = 65536; + } else if (argc != 1) { + fprintf(stderr, + "usage: %s [--expect-4k|--expect-16k|--expect-64k]\n", + argv[0]); + return KSFT_FAIL; + } + ksft_print_header(); + ksft_set_plan(11); + if (expected != 4096 && expected != 16384 && expected != 65536) + ksft_exit_skip("unsupported process page size: %zu\n", + expected); + if (sysconf(_SC_PAGESIZE) != (long)expected || + getauxval(AT_PAGESZ) != expected) + ksft_exit_fail_msg( + "expected %zu-byte ABI; sysconf=%ld AT_PAGESZ=%lu\n", + expected, sysconf(_SC_PAGESIZE), getauxval(AT_PAGESZ)); + ksft_test_result_pass( + "process reports the expected %zu-byte page ABI\n", expected); + if (setrlimit(RLIMIT_CORE, &no_core)) + ksft_exit_fail_msg("disable child core dumps: %s\n", + strerror(errno)); + private_mappings(expected); + shared_mappings(expected); + x18_contract(); + ksft_print_cnts(); + result = ksft_get_fail_cnt() ? KSFT_FAIL : KSFT_PASS; + /* This static executable can also be /init in a disposable test VM. */ + if (getpid() == 1) { + fflush(NULL); + reboot(RB_POWER_OFF); + } + return result; +} diff --git a/tools/testing/selftests/arm64/user4k/pagetable-concurrent.c b/tools/testing/selftests/arm64/user4k/pagetable-concurrent.c new file mode 100644 index 00000000000000..2573a8882e1651 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/pagetable-concurrent.c @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define LEAF 4096UL +#define WINDOW (2UL * 1024 * 1024) +#define WINDOWS 1024UL +#define LENGTH (WINDOWS * WINDOW) + +static void fail(const char *why) +{ + perror(why); + exit(1); +} + +static unsigned char tag(size_t i) +{ + return (unsigned char)(1 + i % 251); +} + +static void checkpoint(const char *phase) +{ + FILE *f = fopen("/proc/self/status", "re"); + char line[256], ack; + unsigned long pte = 0; + if (!f) + fail("status"); + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "VmPTE: %lu kB", &pte) == 1) + break; + fclose(f); + printf("{\"phase\":\"%s\",\"pid\":%d,\"VmPTE_bytes\":%lu}\n", + phase, getpid(), pte * 1024); + if (read(STDIN_FILENO, &ack, 1) != 1 || ack != 'c') + fail("controller ack"); +} + +static void touch_serial(unsigned char *base, int missing_only) +{ + for (size_t i = 0; i < WINDOWS; i++) { + if (missing_only && !(i % 4)) + continue; + if (missing_only) { + void *p = mmap(base + i * WINDOW, WINDOW, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, + -1, 0); + if (p != base + i * WINDOW) + fail("refill mmap"); + } + memset(base + i * WINDOW, tag(i), LEAF); + } +} + +#define WORKERS 32 +static pthread_barrier_t step; +static atomic_int start_workers; +struct worker { unsigned char *base; unsigned id; }; +static void *fault_worker(void *arg) +{ + struct worker *w = arg; + int start; + do { start = atomic_load_explicit(&start_workers, memory_order_acquire); + if (!start) sched_yield(); } while (!start); + if (start != 1) return NULL; + for (unsigned round = 0; round < WINDOWS / WORKERS; round++) { + int ret = pthread_barrier_wait(&step); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) abort(); + size_t i = round * WORKERS + w->id; + memset(w->base + i * WINDOW, tag(i), LEAF); + } + return NULL; +} +static void touch(unsigned char *base, int missing_only) +{ + if (missing_only) { touch_serial(base, missing_only); return; } + pthread_t threads[WORKERS]; + struct worker args[WORKERS]; + pthread_attr_t attr; + if (pthread_attr_init(&attr) || pthread_attr_setstacksize(&attr, 128 * 1024) || + pthread_barrier_init(&step, NULL, WORKERS)) fail("thread setup"); + unsigned made = 0; + for (; made < WORKERS; made++) { + args[made] = (struct worker){.base = base, .id = made}; + int err = pthread_create(&threads[made], &attr, fault_worker, &args[made]); + if (err) { errno = err; break; } + } + pthread_attr_destroy(&attr); + atomic_store_explicit(&start_workers, made == WORKERS ? 1 : 2, memory_order_release); + for (unsigned i = 0; i < made; i++) if (pthread_join(threads[i], NULL)) abort(); + pthread_barrier_destroy(&step); + if (made != WORKERS) fail("pthread_create"); +} + +static void verify(unsigned char *base, int sparse) +{ + for (size_t i = 0; i < WINDOWS; i++) { + if (sparse && i % 4) + continue; + for (size_t j = 0; j < LEAF; j++) + if (base[i * WINDOW + j] != tag(i)) { + fprintf(stderr, "FAIL content window=%zu byte=%zu\n", i, j); + exit(1); + } + } +} + +static void thin(unsigned char *base) +{ + for (size_t i = 0; i < WINDOWS; i++) + if (i % 4 && munmap(base + i * WINDOW, WINDOW)) + fail("thin munmap"); + verify(base, 1); +} + +int main(int argc, char **argv) +{ + if (argc != 2 || strcmp(argv[1], "--native-16k") || + getauxval(AT_PAGESZ) != LEAF) { + fputs("Requires established 4K launcher on native16K kernel\n", stderr); + return 77; + } + setvbuf(stdout, NULL, _IONBF, 0); + alarm(60); + checkpoint("baseline"); + unsigned char *raw = mmap(NULL, LENGTH + WINDOW, + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw == MAP_FAILED) + fail("reserve mmap"); + unsigned char *base = (void *)(((uintptr_t)raw + WINDOW - 1) & ~(WINDOW - 1)); + size_t head = base - raw, tail = WINDOW - head; + if (head && munmap(raw, head)) + fail("trim head"); + if (tail && munmap(base + LENGTH, tail)) + fail("trim tail"); + touch(base, 0); + verify(base, 0); + checkpoint("full"); + thin(base); + checkpoint("quarter_survivors"); + touch(base, 1); + verify(base, 0); + checkpoint("refilled"); + thin(base); + checkpoint("quarter_survivors_again"); + if (munmap(base, LENGTH)) + fail("final munmap"); + checkpoint("unmapped"); + puts("PASS contents, unmap and refill; concurrent firstfaults, accounting is a separate observation"); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/pagetable-retention.c b/tools/testing/selftests/arm64/user4k/pagetable-retention.c new file mode 100644 index 00000000000000..09f18c5424ef8f --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/pagetable-retention.c @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include + +#define LEAF 4096UL +#define WINDOW (2UL * 1024 * 1024) +#define WINDOWS 1024UL +#define LENGTH (WINDOWS * WINDOW) + +static void fail(const char *why) +{ + perror(why); + exit(1); +} + +static unsigned char tag(size_t i) +{ + return (unsigned char)(1 + i % 251); +} + +static void checkpoint(const char *phase) +{ + FILE *f = fopen("/proc/self/status", "re"); + char line[256], ack; + unsigned long pte = 0; + if (!f) + fail("status"); + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "VmPTE: %lu kB", &pte) == 1) + break; + fclose(f); + printf("{\"phase\":\"%s\",\"pid\":%d,\"VmPTE_bytes\":%lu}\n", + phase, getpid(), pte * 1024); + if (read(STDIN_FILENO, &ack, 1) != 1 || ack != 'c') + fail("controller ack"); +} + +static void touch(unsigned char *base, int missing_only) +{ + for (size_t i = 0; i < WINDOWS; i++) { + if (missing_only && !(i % 4)) + continue; + if (missing_only) { + void *p = mmap(base + i * WINDOW, WINDOW, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, + -1, 0); + if (p != base + i * WINDOW) + fail("refill mmap"); + } + memset(base + i * WINDOW, tag(i), LEAF); + } +} + +static void verify(unsigned char *base, int sparse) +{ + for (size_t i = 0; i < WINDOWS; i++) { + if (sparse && i % 4) + continue; + for (size_t j = 0; j < LEAF; j++) + if (base[i * WINDOW + j] != tag(i)) { + fprintf(stderr, "FAIL content window=%zu byte=%zu\n", i, j); + exit(1); + } + } +} + +static void thin(unsigned char *base) +{ + for (size_t i = 0; i < WINDOWS; i++) + if (i % 4 && munmap(base + i * WINDOW, WINDOW)) + fail("thin munmap"); + verify(base, 1); +} + +int main(int argc, char **argv) +{ + if (argc != 2 || strcmp(argv[1], "--native-16k") || + getauxval(AT_PAGESZ) != LEAF) { + fputs("Requires established 4K launcher on native16K kernel\n", stderr); + return 77; + } + setvbuf(stdout, NULL, _IONBF, 0); + alarm(60); + checkpoint("baseline"); + unsigned char *raw = mmap(NULL, LENGTH + WINDOW, + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw == MAP_FAILED) + fail("reserve mmap"); + unsigned char *base = (void *)(((uintptr_t)raw + WINDOW - 1) & ~(WINDOW - 1)); + size_t head = base - raw, tail = WINDOW - head; + if (head && munmap(raw, head)) + fail("trim head"); + if (tail && munmap(base + LENGTH, tail)) + fail("trim tail"); + touch(base, 0); + verify(base, 0); + checkpoint("full"); + thin(base); + checkpoint("quarter_survivors"); + touch(base, 1); + verify(base, 0); + checkpoint("refilled"); + thin(base); + checkpoint("quarter_survivors_again"); + if (munmap(base, LENGTH)) + fail("final munmap"); + checkpoint("unmapped"); + puts("PASS contents, unmap and refill; accounting is a separate observation"); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/perf-ring-init.c b/tools/testing/selftests/arm64/user4k/perf-ring-init.c new file mode 100644 index 00000000000000..4252f017619ca2 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/perf-ring-init.c @@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable PID1: perf ring layout and sampling with per-mm page sizes. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static void check(int ok, const char *expression, int line) +{ + if (!ok) { + fprintf(stderr, "FAIL line %d: %s errno=%d\n", line, expression, + errno); + exit(1); + } +} + +#define CHECK(x) check(!!(x), #x, __LINE__) + +static int no4k; + +static int wait_child(pid_t child) +{ + int status; + pid_t got; + + if (child < 0) + return 1; + do { + got = waitpid(child, &status, 0); + } while (got < 0 && errno == EINTR); + return got != child || !WIFEXITED(status) || WEXITSTATUS(status); +} + +static int event_open(int sampling) +{ + struct perf_event_attr attr = { + .type = PERF_TYPE_SOFTWARE, + .size = sizeof(attr), + .config = PERF_COUNT_SW_PAGE_FAULTS, + .disabled = 1, + .sample_period = sampling ? 1 : 0, + .sample_type = PERF_SAMPLE_TID | PERF_SAMPLE_TIME | + PERF_SAMPLE_ADDR | PERF_SAMPLE_PERIOD, + }; + int fd = syscall(SYS_perf_event_open, &attr, 0, -1, -1, 0); + + CHECK(fd >= 0); + return fd; +} + +static sigjmp_buf fault_return; +static volatile sig_atomic_t fault_signal; + +static void fault_handler(int signal) +{ + fault_signal = signal; + siglongjmp(fault_return, 1); +} + +static void readonly_data(void *address) +{ + struct sigaction action = { .sa_handler = fault_handler }, previous; + + CHECK(!sigemptyset(&action.sa_mask)); + CHECK(!sigaction(SIGBUS, &action, &previous)); + fault_signal = 0; + if (!sigsetjmp(fault_return, 1)) + *(volatile unsigned char *)address = 0; + CHECK(fault_signal == SIGBUS); + CHECK(!sigaction(SIGBUS, &previous, NULL)); +} + +static void copy_record(void *destination, const unsigned char *ring, + unsigned long capacity, uint64_t position, size_t size) +{ + unsigned long offset = position & (capacity - 1); + size_t first = capacity - offset; + + if (first > size) + first = size; + memcpy(destination, ring + offset, first); + memcpy((unsigned char *)destination + first, ring, size - first); +} + +static void sampling(unsigned long pages) +{ + unsigned long page = getauxval(AT_PAGESZ), capacity = pages * page; + unsigned long length = page + capacity; + int fd = event_open(1); + struct perf_event_mmap_page *map = + mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + unsigned char *region, *ring; + uint64_t tail = 0, samples = 0, wrapped = 0, region_samples = 0; + + CHECK(map != MAP_FAILED); + CHECK(map->data_offset == page && map->data_size == capacity); + map->data_tail = 0; + ring = (unsigned char *)map + map->data_offset; + for (unsigned long i = 0; i < capacity; i += page) + CHECK(ring[i] == 0); + readonly_data(ring); + region = mmap(NULL, 16 * page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(region != MAP_FAILED); + for (unsigned int batch = 0; batch < 10000 && tail < 3 * capacity; + batch++) { + CHECK(!madvise(region, 16 * page, MADV_DONTNEED)); + CHECK(!ioctl(fd, PERF_EVENT_IOC_ENABLE, 0)); + for (unsigned int i = 0; i < 16; i++) + *(volatile unsigned char *)(region + i * page) = 1; + CHECK(!ioctl(fd, PERF_EVENT_IOC_DISABLE, 0)); + uint64_t head = + __atomic_load_n(&map->data_head, __ATOMIC_ACQUIRE); + + CHECK(head >= tail && head - tail <= capacity); + while (tail < head) { + struct perf_event_header header; + struct { + struct perf_event_header header; + uint32_t pid, tid; + uint64_t time, address, period; + } record; + + copy_record(&header, ring, capacity, tail, + sizeof(header)); + CHECK(header.size >= sizeof(header) && + header.size <= head - tail); + if (header.type == PERF_RECORD_SAMPLE) { + CHECK(header.size == sizeof(record)); + copy_record(&record, ring, capacity, tail, + sizeof(record)); + CHECK(record.pid == (uint32_t)getpid()); + CHECK(record.tid == (uint32_t)getpid()); + if (record.address >= (uintptr_t)region && + record.address < + (uintptr_t)region + 16 * page) + region_samples++; + CHECK(record.period == 1); + samples++; + if ((tail & (capacity - 1)) + header.size > + capacity) + wrapped++; + } else { + CHECK(header.type == PERF_RECORD_THROTTLE || + header.type == PERF_RECORD_UNTHROTTLE); + } + tail += header.size; + } + __atomic_store_n(&map->data_tail, tail, __ATOMIC_RELEASE); + } + CHECK(tail >= 3 * capacity && samples && wrapped); + /* Library/stack faults may also occur while a batch is enabled. */ + CHECK(region_samples > samples * 9 / 10); + CHECK(!munmap(region, 16 * page)); + CHECK(!close(fd)); + CHECK(map->data_head == tail); + CHECK(!munmap(map, length)); + printf("ok - perf page=%lu ring=%lu samples=%llu region=%llu wrapped-records=%llu\n", + page, capacity, (unsigned long long)samples, + (unsigned long long)region_samples, (unsigned long long)wrapped); +} + +static void layout(unsigned long native) +{ + unsigned long page = getauxval(AT_PAGESZ); + int fd = event_open(0); + struct perf_event_mmap_page *map = + mmap(NULL, page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + + CHECK(map != MAP_FAILED); + CHECK(map->data_offset == page && map->data_size == 0); + map->data_tail = 0; + CHECK(!munmap(map, page)); + CHECK(!close(fd)); + fd = event_open(0); + errno = 0; + CHECK(mmap(NULL, 4 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0) == + MAP_FAILED); + CHECK(errno == EINVAL); + map = mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + CHECK(map != MAP_FAILED); + struct perf_event_mmap_page *alias = + mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + CHECK(alias != MAP_FAILED); + map->data_tail = 7; + CHECK(alias->data_tail == 7); + map->data_tail = 0; + CHECK(!munmap(alias, 3 * page)); + /* A software event has no AUX provider. Check byte-exact offset routing. */ + map->aux_offset = 3 * page; + map->aux_size = native; + errno = 0; + CHECK(mmap(NULL, native, PROT_READ, MAP_SHARED, fd, map->aux_offset) == + MAP_FAILED); + CHECK(errno == EOPNOTSUPP); + unsigned long sizes[] = { 4096, 16384, 65536 }; + + for (unsigned int i = 0; i < 3; i++) { + if (sizes[i] > native || sizes[i] == page || + (no4k && sizes[i] == 4096)) + continue; + pid_t child = fork(); + + if (!child) { + char f[32], s[32]; + + snprintf(f, sizeof(f), "%d", fd); + snprintf(s, sizeof(s), "%lu", sizes[i]); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], 0UL, + 0UL, 0UL)) + _exit(2); + execl("/init", "/init", "--alias", f, s, NULL); + _exit(3); + } + CHECK(!wait_child(child)); + } + CHECK(map->data_size == 2 * page && map->data_offset == page); + CHECK(!munmap(map, 3 * page)); + CHECK(!close(fd)); + printf("ok - perf page=%lu metadata, invalid size, aliases, AUX routing\n", + page); +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + int failed = 0, compat_only = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + no4k = getenv("PERF_TEST_NO4K") != NULL; + if (argc == 4 && !strcmp(argv[1], "--alias")) { + unsigned long page = strtoul(argv[3], NULL, 10); + + CHECK(getauxval(AT_PAGESZ) == page); + errno = 0; + CHECK(mmap(NULL, 3 * page, PROT_READ | PROT_WRITE, MAP_SHARED, + atoi(argv[2]), 0) == MAP_FAILED); + CHECK(errno == EINVAL); + return 0; + } + if (argc == 4 && !strcmp(argv[1], "--case")) { + native = strtoul(argv[2], NULL, 10); + CHECK(getauxval(AT_PAGESZ) == strtoul(argv[3], NULL, 10)); + layout(native); + sampling(1); + sampling(2); + sampling(2 * native / getauxval(AT_PAGESZ)); + return 0; + } + CHECK(getpid() == 1); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--no4k")) { + no4k = 1; + CHECK(!setenv("PERF_TEST_NO4K", "1", 1)); + } + if (!strcmp(argv[i], "--compat-only")) + compat_only = 1; + } + unsigned long sizes[] = { 4096, 16384, 65536 }; + + for (unsigned int i = 0; i < 3; i++) { + if (compat_only || sizes[i] > native || + (no4k && sizes[i] == 4096)) + continue; + pid_t child = fork(); + + if (!child) { + char n[32], s[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(s, sizeof(s), "%lu", sizes[i]); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, sizes[i], 0UL, + 0UL, 0UL)) + _exit(2); + execl("/init", "/init", "--case", n, s, NULL); + _exit(3); + } + int bad = wait_child(child); + + printf("%s - perf native=%lu process=%lu\n", + bad ? "not ok" : "ok", native, sizes[i]); + failed |= bad; + } + if (compat_only) + CHECK(access("/compat-perf", X_OK) == 0); + if (access("/compat-perf", X_OK) == 0) { + pid_t child = fork(); + + if (!child) { + char n[32], s[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(s, sizeof(s), "%lu", no4k ? native : 4096UL); + execl("/compat-perf", "/compat-perf", n, s, NULL); + _exit(3); + } + int bad = wait_child(child); + + printf("%s - perf AArch32 native=%lu process=%lu\n", + bad ? "not ok" : "ok", native, no4k ? native : 4096UL); + failed |= bad; + } + printf("PERF RING GRANULE %s\n", failed ? "FAIL" : "PASS"); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/pinned-fork.c b/tools/testing/selftests/arm64/user4k/pinned-fork.c new file mode 100644 index 00000000000000..ec91136af8de42 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/pinned-fork.c @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define LEAF 4096UL +#define NATIVE 16384UL +#define SIZE (1024UL * 1024) +#define COUNT (SIZE / LEAF) +#define PFNMASK ((1ULL << 55) - 1) +static pid_t child = -1; +static uint64_t parent_pfn[COUNT], child_pfn[COUNT]; +static void die(const char *s) { perror(s); exit(1); } +static void cleanup(void) +{ + if (child > 0) { + kill(child, SIGKILL); + while (waitpid(child, NULL, 0) < 0 && errno == EINTR) {} + } +} +static void timeout(int sig) +{ + (void)sig; + if (child > 0) kill(child, SIGKILL); + _exit(124); +} +static void transfer(int fd, void *buffer, size_t len, int writing) +{ + unsigned char *p = buffer; + while (len) { + ssize_t n = writing ? write(fd, p, len) : read(fd, p, len); + if (n < 0 && errno == EINTR) continue; + if (n <= 0) die(writing ? "write protocol" : "read protocol"); + p += n; len -= n; + } +} +static unsigned char pattern(size_t i) { return (unsigned char)(1 + (i * 17 + i / LEAF) % 251); } +static void check(unsigned char *p, int mode) +{ + for (size_t i = 0; i < SIZE; i++) { + unsigned char expected = pattern(i); + if ((mode == 1 && i % LEAF == 0) || (mode == 2 && i % LEAF == 1)) expected ^= 0x80; + if (p[i] != expected) { + fprintf(stderr, "FAIL bytes mode=%d offset=%zu expected=%u actual=%u\n", mode, i, expected, p[i]); + exit(1); + } + } +} +static void pfns(unsigned char *p, uint64_t *out) +{ + int fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (fd < 0) die("pagemap open"); + for (size_t i = 0; i < COUNT; i++) { + uint64_t e; + if (pread(fd, &e, sizeof(e), (((uintptr_t)p / LEAF) + i) * sizeof(e)) != sizeof(e)) die("pagemap read"); + if (!(e >> 63) || (e & (1ULL << 62)) || !(e & PFNMASK)) { + fprintf(stderr, "FAIL pagemap leaf=%zu entry=%016llx (root PFNs required)\n", i, (unsigned long long)e); exit(1); + } + out[i] = e & PFNMASK; + } + close(fd); +} +static size_t native_count(uint64_t *pf) +{ + size_t n = 0; + for (size_t i = 0; i < COUNT; i++) { + size_t j; + for (j = 0; j < i; j++) if (pf[j] / 4 == pf[i] / 4) break; + if (j == i) n++; + } + return n; +} +int main(int argc, char **argv) +{ + if (argc == 2 && !strcmp(argv[1], "--selftest")) { + for (size_t i = 0; i < COUNT; i++) parent_pfn[i] = 4 + i; + if (native_count(parent_pfn) != COUNT / 4) return 1; + for (size_t i = 0; i < COUNT; i++) parent_pfn[i] = 4 + 4 * i; + if (native_count(parent_pfn) != COUNT) return 1; + puts("PASS native PFN grouping selftest (no pin/fork coverage)"); return 0; + } + int packed = 0, hold = 0; + if (argc < 2 || strcmp(argv[1], "--native-16k") || getauxval(AT_PAGESZ) != LEAF) return 77; + for (int i = 2; i < argc; i++) { + if (!strcmp(argv[i], "--expect-packed")) packed = 1; + else if (!strcmp(argv[i], "--hold")) hold = 1; + else return 77; + } + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGALRM, timeout); signal(SIGPIPE, SIG_IGN); alarm(15); atexit(cleanup); + unsigned char *raw = mmap(NULL, SIZE + NATIVE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw == MAP_FAILED) die("mmap"); + unsigned char *p = (void *)(((uintptr_t)raw + NATIVE - 1) & ~(NATIVE - 1)); + for (size_t i = 0; i < SIZE; i++) p[i] = pattern(i); + check(p, 0); pfns(p, parent_pfn); + if (native_count(parent_pfn) * NATIVE != SIZE) die("initial dense native backing"); + struct io_uring_params params = {0}; + int ring = syscall(__NR_io_uring_setup, 2, ¶ms); + if (ring < 0) { perror("UNEXERCISED io_uring_setup"); return 77; } + struct iovec iov = { .iov_base = p, .iov_len = SIZE }; + if (syscall(__NR_io_uring_register, ring, IORING_REGISTER_BUFFERS, &iov, 1)) { + perror("UNEXERCISED register real long-term pins"); close(ring); return 77; + } + /* Long-term pin setup can migrate backing; capture the pinned identity. */ + pfns(p, parent_pfn); check(p, 0); + if (native_count(parent_pfn) * NATIVE != SIZE) die("pinned dense native backing"); + int to_child[2], from_child[2]; + if (pipe(to_child) || pipe(from_child)) die("pipe"); + pid_t parent = getpid(); + child = fork(); + if (child < 0) die("fork"); + if (!child) { + prctl(PR_SET_PDEATHSIG, SIGKILL); + if (getppid() != parent) _exit(1); + alarm(10); close(ring); close(to_child[1]); close(from_child[0]); + pfns(p, child_pfn); check(p, 0); + transfer(from_child[1], child_pfn, sizeof(child_pfn), 1); + char token; transfer(to_child[0], &token, 1, 0); + if (token != 'w') die("child token"); + check(p, 0); + for (size_t i = 0; i < COUNT; i++) p[i * LEAF + 1] ^= 0x80; + check(p, 2); + token = 'c'; transfer(from_child[1], &token, 1, 1); + transfer(to_child[0], &token, 1, 0); + if (token != 'e') die("child exit token"); + check(p, 2); _exit(0); + } + close(to_child[0]); close(from_child[1]); + transfer(from_child[0], child_pfn, sizeof(child_pfn), 0); + for (size_t i = 0; i < COUNT; i++) + for (size_t j = 0; j < COUNT; j++) + if (child_pfn[i] / 4 == parent_pfn[j] / 4) die("child copy overlaps pinned parent backing"); + printf("METADATA real_registered_pin=1 parent_native=%zu child_native=%zu copied_leaves=%lu native_overlap=0\n", native_count(parent_pfn) * NATIVE, native_count(child_pfn) * NATIVE, COUNT); + if (hold) { + printf("HOLD parent=%d child=%d base=%lx bytes=%lu (write c to continue)\n", getpid(), child, (unsigned long)p, SIZE); + char ack; transfer(STDIN_FILENO, &ack, 1, 0); + if (ack != 'c') die("controller hold token"); + } + size_t child_bytes = native_count(child_pfn) * NATIVE; + if (child_bytes < SIZE || child_bytes > COUNT * NATIVE || + (packed && child_bytes > SIZE + 256 * 1024)) die("child density bound"); + for (size_t i = 0; i < COUNT; i++) p[i * LEAF] ^= 0x80; + check(p, 1); + char token = 'w'; transfer(to_child[1], &token, 1, 1); + transfer(from_child[0], &token, 1, 0); + if (token != 'c') die("parent token"); + check(p, 1); + token = 'e'; transfer(to_child[1], &token, 1, 1); + int status; + if (waitpid(child, &status, 0) != child) die("waitpid"); + child = -1; + if (!WIFEXITED(status) || WEXITSTATUS(status)) die("child failed"); + if (syscall(__NR_io_uring_register, ring, IORING_UNREGISTER_BUFFERS, NULL, 0)) die("unregister"); + close(ring); close(to_child[1]); close(from_child[0]); + if (munmap(raw, SIZE + NATIVE)) die("munmap"); + puts("PASS real pin/fork copy and full-byte two-way isolation; density is separately reported"); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/proc-args.c b/tools/testing/selftests/arm64/user4k/proc-args.c new file mode 100644 index 00000000000000..300742a2588632 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/proc-args.c @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Compare argv/envp with /proc/PID/cmdline and /proc/PID/environ. + * + * Run it once natively and once under the 4K user granule (for example + * "userpages 4k ./proc-args"). The process reads its own files, and a forked + * child reads the parent's files through /proc/. "--hold SECONDS" + * keeps the process alive so a native process can read it from outside. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +extern char **environ; + +static int failures; + +static ssize_t read_file(const char *path, char *buf, size_t size) +{ + ssize_t got, total = 0; + int fd = open(path, O_RDONLY); + + if (fd < 0) + return -errno; + while ((size_t)total < size) { + got = read(fd, buf + total, size - total); + if (got < 0) { + if (errno == EINTR) + continue; + total = -errno; + break; + } + if (!got) + break; + total += got; + } + close(fd); + return total; +} + +/* Join the complete vector; reserve one extra byte in the read buffer. */ +static char *join(char **vec, size_t *size) +{ + size_t len = 0, offset = 0; + char **entry, *buf; + + for (entry = vec; *entry; entry++) { + size_t n = strlen(*entry) + 1; + + if (n > SSIZE_MAX - 1 - len) { + errno = EOVERFLOW; + return NULL; + } + len += n; + } + buf = malloc(len ? len : 1); + if (!buf) + return NULL; + for (entry = vec; *entry; entry++) { + size_t n = strlen(*entry) + 1; + + memcpy(buf + offset, *entry, n); + offset += n; + } + *size = len; + return buf; +} + +static void check(const char *who, const char *name, pid_t pid, char **vec) +{ + char path[64], *want, *got; + size_t want_len; + ssize_t got_len; + + want = join(vec, &want_len); + got = want ? malloc(want_len + 1) : NULL; + if (!got) { + printf("not ok %s %s allocation: %s\n", who, name, + strerror(errno)); + free(want); + failures++; + return; + } + snprintf(path, sizeof(path), "/proc/%d/%s", pid, name); + /* A trailing byte rejects excess data; exact contents must reach EOF. */ + got_len = read_file(path, got, want_len + 1); + if (got_len == (ssize_t)want_len && !memcmp(want, got, want_len)) { + printf("ok %s %s %zd bytes\n", who, path, got_len); + } else { + printf("not ok %s %s want %zu got %zd", who, path, want_len, + got_len); + if (got_len < 0) + printf(" error %s", strerror(-got_len)); + else if (got_len > 0) + printf(" first [%.*s]", + (int)(got_len < 64 ? got_len : 64), got); + printf("\n"); + failures++; + } + free(got); + free(want); +} + +static void show_layout(pid_t pid, char **argv) +{ + char buf[4096], *p, *field; + ssize_t len; + int i; + + printf("pid %d AT_PAGESZ %lu argv[0] %p environ[0] %p\n", pid, + getauxval(AT_PAGESZ), (void *)argv[0], + environ[0] ? (void *)environ[0] : NULL); + len = read_file("/proc/self/stat", buf, sizeof(buf) - 1); + if (len <= 0) + return; + buf[len] = 0; + /* Fields after the parenthesised comm start at field 3. */ + p = strrchr(buf, ')'); + if (!p || p[1] != ' ') + return; + printf("stat arg_start arg_end env_start env_end:"); + for (i = 3, field = strtok(p + 2, " "); field; + field = strtok(NULL, " "), i++) + if (i >= 48 && i <= 51) + printf(" %#lx", strtoul(field, NULL, 10)); + printf("\n"); +} + +int main(int argc, char **argv) +{ + pid_t self = getpid(), child; + int status; + + setbuf(stdout, NULL); + show_layout(self, argv); + check("self", "cmdline", self, argv); + check("self", "environ", self, environ); + + child = fork(); + if (child < 0) + return 1; + if (!child) { + failures = 0; + check("child", "cmdline", self, argv); + check("child", "environ", self, environ); + _exit(failures ? 1 : 0); + } + if (waitpid(child, &status, 0) != child || !WIFEXITED(status) || + WEXITSTATUS(status)) + failures++; + + if (argc > 2 && !strcmp(argv[1], "--hold")) { + fflush(stdout); + sleep(atoi(argv[2])); + } + printf("%s: %d failure(s)\n", failures ? "FAIL" : "PASS", failures); + return failures ? 1 : 0; +} diff --git a/tools/testing/selftests/arm64/user4k/proc-memory.c b/tools/testing/selftests/arm64/user4k/proc-memory.c new file mode 100644 index 00000000000000..81ca2d0553dd0a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/proc-memory.c @@ -0,0 +1,340 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +struct stats { + unsigned long size, rss, pss, private_clean, private_dirty, + shared_clean, shared_dirty; + unsigned long anonymous, kernel_page, mmu_page; +}; + +static bool get_stats(pid_t pid, uintptr_t address, struct stats *s) +{ + char path[64], line[512]; + unsigned long start, end, value; + unsigned int fields = 0; + char key[64]; + bool found = false; + FILE *f; + + memset(s, 0, sizeof(*s)); + snprintf(path, sizeof(path), "/proc/%d/smaps", pid); + f = fopen(path, "re"); + if (!f) + return false; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) { + if (found) + break; + found = start == address; + continue; + } + if (!found) + continue; + if (sscanf(line, "%63[^:]: %lu", key, &value) != 2) + continue; + if (!strcmp(key, "Size")) { + s->size = value; + fields |= 1U << 0; + } else if (!strcmp(key, "Rss")) { + s->rss = value; + fields |= 1U << 1; + } else if (!strcmp(key, "Pss")) { + s->pss = value; + fields |= 1U << 2; + } else if (!strcmp(key, "Private_Clean")) { + s->private_clean = value; + fields |= 1U << 3; + } else if (!strcmp(key, "Private_Dirty")) { + s->private_dirty = value; + fields |= 1U << 4; + } else if (!strcmp(key, "Shared_Clean")) { + s->shared_clean = value; + fields |= 1U << 5; + } else if (!strcmp(key, "Shared_Dirty")) { + s->shared_dirty = value; + fields |= 1U << 6; + } else if (!strcmp(key, "Anonymous")) { + s->anonymous = value; + fields |= 1U << 7; + } else if (!strcmp(key, "KernelPageSize")) { + s->kernel_page = value; + fields |= 1U << 8; + } else if (!strcmp(key, "MMUPageSize")) { + s->mmu_page = value; + fields |= 1U << 9; + } + } + fclose(f); + return found && fields == (1U << 10) - 1; +} + +static bool check(pid_t pid, void *address, size_t page, size_t length, + unsigned long pss, unsigned long private, + unsigned long shared, bool anon) +{ + struct stats s; + bool ok = get_stats(pid, (uintptr_t)address, &s); + + ok &= s.size == length / 1024 && s.rss == length / 1024 && + s.pss == pss && s.private_clean + s.private_dirty == private && + s.shared_clean + s.shared_dirty == shared && + s.anonymous == (anon ? length / 1024 : 0) && + s.kernel_page == page / 1024 && s.mmu_page == page / 1024; + if (!ok) + printf("# smaps size=%lu rss=%lu pss=%lu private=%lu shared=%lu anon=%lu pages=%lu/%lu\n", + s.size, s.rss, s.pss, s.private_clean + s.private_dirty, + s.shared_clean + s.shared_dirty, s.anonymous, + s.kernel_page, s.mmu_page); + return ok; +} + +static int check_numa(void *address, size_t page) +{ + char line[1024]; + unsigned long start, anonymous = 0, pages = 0, granule = 0; + bool found = false; + FILE *f = fopen("/proc/self/numa_maps", "re"); + + if (!f) + return errno == ENOENT ? -1 : 0; + while (fgets(line, sizeof(line), f)) { + char *save, *token; + + if (sscanf(line, "%lx", &start) != 1 || + start != (uintptr_t)address) + continue; + found = true; + for (token = strtok_r(line, "\n ", &save); token; + token = strtok_r(NULL, "\n ", &save)) { + unsigned int node; + unsigned long count; + + if (sscanf(token, "anon=%lu", &count) == 1) + anonymous = count; + else if (sscanf(token, "kernelpagesize_kB=%lu", + &count) == 1) + granule = count; + else if (sscanf(token, "N%u=%lu", &node, &count) == 2) + pages += count; + } + break; + } + fclose(f); + if (!found || anonymous != 4 || pages != 4 || granule != page / 1024) + printf("# numa found=%d anon=%lu pages=%lu granule=%lu\n", + found, anonymous, pages, granule); + return found && anonymous == 4 && pages == 4 && granule == page / 1024; +} + +static bool reap(pid_t child) +{ + int status; + + return child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); +} + +static bool resident_rewrite_reuses(unsigned char *p, size_t page) +{ + uint64_t before[4], after[4]; + int fd = open("/proc/self/pagemap", O_RDONLY); + unsigned int i; + bool ok = fd >= 0; + + if (ok) + ok = pread(fd, before, sizeof(before), + (uintptr_t)p / page * 8) == sizeof(before); + for (i = 0; i < 4; i++) + p[i * page] ^= 0x21; + if (ok) + ok = pread(fd, after, sizeof(after), (uintptr_t)p / page * 8) == + sizeof(after); + for (i = 0; ok && i < 4; i++) { + const uint64_t frame = (1ULL << 55) - 1; + + ok &= (before[i] & frame) && + (before[i] & frame) == (after[i] & frame) && + (before[i] & (1ULL << 63)) && (after[i] & (1ULL << 63)); + } + if (fd >= 0) + close(fd); + return ok; +} + +static void anonymous_sharing(size_t page) +{ + unsigned char *base, *p; + int ready[2], go[2]; + pid_t child; + char byte; + bool ok; + + base = mmap(NULL, 6 * page, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, + 0); + if (base == MAP_FAILED || + mprotect(base + page, 4 * page, PROT_READ | PROT_WRITE) || + pipe(ready) || pipe(go)) + ksft_exit_fail_msg("anonymous fixture setup\n"); + p = base + page; + memset(p, 0x57, 4 * page); + ksft_test_result( + check(getpid(), p, page, 4 * page, 4 * page / 1024, + 4 * page / 1024, 0, true), + "dense anonymous leaves have exact RSS/PSS/private bytes\n"); + { + int numa = check_numa(p, page); + + if (numa < 0) + ksft_test_result_skip( + "NUMA statistics require CONFIG_NUMA\n"); + else + ksft_test_result( + numa, + "NUMA statistics count target leaves and report their granule\n"); + } + child = fork(); + if (!child) { + close(ready[0]); + close(go[1]); + if (write(ready[1], "a", 1) != 1 || read(go[0], &byte, 1) != 1) + _exit(1); + p[page] = 0x96; + if (write(ready[1], "b", 1) != 1 || read(go[0], &byte, 1) != 1) + _exit(2); + _exit(0); + } + close(ready[1]); + close(go[0]); + ok = child > 0 && read(ready[0], &byte, 1) == 1 && + check(getpid(), p, page, 4 * page, 2 * page / 1024, 0, + 4 * page / 1024, true); + ksft_test_result( + ok, + "fork divides each anonymous leaf's PSS by its own mapcount\n"); + ok = write(go[1], "x", 1) == 1 && read(ready[0], &byte, 1) == 1 && + check(getpid(), p, page, 4 * page, (5 * page) / 2048, page / 1024, + 3 * page / 1024, true) && + p[page] == 0x57; + ksft_test_result( + ok, + "one child COW changes only that leaf's private/shared accounting\n"); + write(go[1], "x", 1); + close(go[1]); + close(ready[0]); + ksft_test_result(reap(child) && check(getpid(), p, page, 4 * page, + 4 * page / 1024, 4 * page / 1024, + 0, true), + "child exit restores full anonymous PSS\n"); + ksft_test_result( + resident_rewrite_reuses(p, page), + "resident writes reuse physical backing after the child exits\n"); + munmap(base, 6 * page); +} + +static int worker(int fd, int output, int input, size_t expected) +{ + size_t page = getauxval(AT_PAGESZ); + unsigned char *p; + uintptr_t addr; + char byte; + + if (page != expected) + return 1; + p = mmap(NULL, page, PROT_READ, MAP_SHARED, fd, + page == 4096 ? 4096 : 0); + if (p == MAP_FAILED || p[0] != 0x78) + return 2; + addr = (uintptr_t)p; + if (write(output, &addr, sizeof(addr)) != sizeof(addr) || + read(input, &byte, 1) != 1) + return 3; + return 0; +} + +static void mixed_file(size_t page) +{ + size_t target = page == 4096 ? 16384 : 4096; + int ready[2], go[2], fd = memfd_create("mixed-pss", 0); + unsigned char data[16384], *p; + uintptr_t remote; + pid_t child; + bool ok; + + memset(data, 0x78, sizeof(data)); + if (fd < 0 || write(fd, data, sizeof(data)) != sizeof(data) || + pipe(ready) || pipe(go)) + ksft_exit_fail_msg("mixed file fixture setup\n"); + p = mmap(NULL, page, PROT_READ | PROT_WRITE, MAP_SHARED, fd, + page == 4096 ? 4096 : 0); + if (p == MAP_FAILED || p[0] != 0x78) + ksft_exit_fail_msg("mixed file mmap\n"); + child = fork(); + if (!child) { + char farg[16], out[16], in[16], expected[16]; + + close(ready[0]); + close(go[1]); + snprintf(farg, sizeof(farg), "%d", fd); + snprintf(out, sizeof(out), "%d", ready[1]); + snprintf(in, sizeof(in), "%d", go[0]); + snprintf(expected, sizeof(expected), "%zu", target); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, target, 0UL, 0UL, 0UL)) + _exit(4); + execl("/proc/self/exe", "/proc/self/exe", "--worker", farg, out, + in, expected, NULL); + _exit(5); + } + close(ready[1]); + close(go[0]); + ok = child > 0 && + read(ready[0], &remote, sizeof(remote)) == sizeof(remote); + ok = ok && + check(getpid(), p, page, page, page == 4096 ? 2 : 14, + page == 4096 ? 0 : 12, 4, false) && + check(child, (void *)remote, target, target, + target == 4096 ? 2 : 14, target == 4096 ? 0 : 12, 4, false); + ksft_test_result( + ok, + "mixed file aliases divide only their overlapping physical quarter\n"); + write(go[1], "x", 1); + close(go[1]); + close(ready[0]); + ksft_test_result(reap(child) && + check(getpid(), p, page, page, page / 1024, + page / 1024, 0, false), + "mixed alias exit restores private file accounting\n"); + munmap(p, page); + close(fd); +} + +int main(int argc, char **argv) +{ + size_t page = getauxval(AT_PAGESZ); + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), atoi(argv[4]), + strtoul(argv[5], NULL, 10)); + if (argc > 2 || (argc == 2 && strcmp(argv[1], "--expect-4k")) || + page != (argc == 2 ? 4096UL : 16384UL)) + return 2; + ksft_print_header(); + ksft_set_plan(8); + anonymous_sharing(page); + mixed_file(page); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/proc-pages.c b/tools/testing/selftests/arm64/user4k/proc-pages.c new file mode 100644 index 00000000000000..d3aaa11101cf2b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/proc-pages.c @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +#define PRESENT (1ULL << 63) +#define EXCLUSIVE (1ULL << 56) +#define GUARD (1ULL << 58) +#define FRAME ((1ULL << 55) - 1) +#define NPAGE 5 + +struct ready { + uintptr_t address; + size_t page; + uint64_t entries[NPAGE]; + uint64_t shared_entry; +}; + +static int open_proc(pid_t pid, const char *name, int flags) +{ + char path[80]; + + snprintf(path, sizeof(path), "/proc/%d/%s", pid, name); + return open(path, flags); +} + +static bool entries(int fd, uintptr_t address, size_t page, uint64_t *values, + int nr) +{ + return pread(fd, values, nr * sizeof(*values), + address / page * sizeof(*values)) == + nr * (ssize_t)sizeof(*values); +} + +static unsigned char *mapping(size_t page) +{ + const size_t boundary = 32 * 1024 * 1024; + size_t reserve = boundary + 8 * page, head, tail; + unsigned char *base, *p; + + base = mmap(NULL, reserve, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, + 0); + if (base == MAP_FAILED) + return NULL; + /* Cross both native 32 MiB and alternative 2 MiB PTE-table boundaries. */ + p = (void *)(((uintptr_t)base + 2 * page + boundary - 1) & + ~(boundary - 1)); + p -= 2 * page; + head = p - base; + tail = reserve - head - NPAGE * page; + if ((head && munmap(base, head)) || + (tail && munmap(p + NPAGE * page, tail)) || + mprotect(p, NPAGE * page, PROT_READ | PROT_WRITE)) + return NULL; + memset(p, 0x67, NPAGE * page); + if (munmap(p + 2 * page, page) || + madvise(p + 3 * page, page, MADV_GUARD_INSTALL) || + mprotect(p + 4 * page, page, PROT_NONE)) + return NULL; + return p; +} + +static bool scan(int fd, uintptr_t addr, size_t page) +{ + struct page_region vec[8]; + struct pm_scan_arg arg = { + .size = sizeof(arg), + .start = addr, + .end = addr + NPAGE * page, + .vec = (uintptr_t)vec, + .vec_len = 8, + .return_mask = PAGE_IS_PRESENT | PAGE_IS_GUARD, + }; + int nr = ioctl(fd, PAGEMAP_SCAN, &arg); + bool ok = nr == 3 && arg.walk_end == arg.end; + if (!ok) + printf("# scan initial nr=%d errno=%d end_delta=%lld\n", nr, + errno, (long long)(arg.walk_end - arg.end)); + + /* The unmapped middle page produces no VMA result. */ + if (nr == 3) + ok &= vec[0].start == addr && vec[0].end == addr + 2 * page && + vec[0].categories == PAGE_IS_PRESENT && + vec[1].start == addr + 3 * page && + vec[1].end == addr + 4 * page && + vec[1].categories == PAGE_IS_GUARD && + vec[2].start == addr + 4 * page && + vec[2].end == arg.end && + vec[2].categories == PAGE_IS_PRESENT; + arg.max_pages = 1; + arg.walk_end = 0; + nr = ioctl(fd, PAGEMAP_SCAN, &arg); + ok &= nr == 1 && vec[0].start == addr && vec[0].end == addr + page && + arg.walk_end == addr + page; + arg.start = addr + 1; + errno = 0; + ok &= ioctl(fd, PAGEMAP_SCAN, &arg) == -1 && errno == EINVAL; + return ok; +} + +static int remote_worker(int output, int input, size_t expected, int shared_fd) +{ + struct ready ready = { .page = getauxval(AT_PAGESZ) }; + unsigned char *p; + int fd = open_proc(getpid(), "pagemap", O_RDONLY); + char byte; + unsigned char *shared = + mmap(NULL, 16384, PROT_READ, MAP_SHARED, shared_fd, 0); + size_t offset = ready.page == 4096 ? 4096 : 0; + + if (shared == MAP_FAILED || shared[offset] != 0x83 || + !entries(fd, (uintptr_t)shared + offset, ready.page, + &ready.shared_entry, 1)) + return 2; + /* Allocate aliases first so mmap cannot reuse our deliberate hole. */ + p = mapping(ready.page); + ready.address = (uintptr_t)p; + if (ready.page != expected || !p || fd < 0 || + !entries(fd, ready.address, ready.page, ready.entries, NPAGE) || + write(output, &ready, sizeof(ready)) != sizeof(ready) || + read(input, &byte, 1) != 1) + return 1; + return 0; +} + +static bool remote_read(size_t own_page) +{ + int ready_pipe[2], done_pipe[2], status, fd, shared_fd, self_fd; + unsigned char *shared; + uint64_t self_entry; + size_t offset = own_page == 4096 ? 4096 : 0; + size_t target = own_page == 4096 ? 16384 : 4096; + struct ready ready; + uint64_t actual[NPAGE]; + pid_t child; + bool ok; + + shared_fd = memfd_create("mixed-pagemap", 0); + if (shared_fd < 0 || ftruncate(shared_fd, 16384)) + return false; + shared = mmap(NULL, 16384, PROT_READ | PROT_WRITE, MAP_SHARED, + shared_fd, 0); + self_fd = open_proc(getpid(), "pagemap", O_RDONLY); + if (shared == MAP_FAILED || self_fd < 0) + return false; + memset(shared, 0x83, 16384); + if (!entries(self_fd, (uintptr_t)shared + offset, own_page, &self_entry, + 1)) + return false; + close(self_fd); + if (pipe(ready_pipe) || pipe(done_pipe)) + return false; + child = fork(); + if (!child) { + char output[16], input[16], expected[16], shared_arg[16]; + + close(ready_pipe[0]); + close(done_pipe[1]); + snprintf(output, sizeof(output), "%d", ready_pipe[1]); + snprintf(input, sizeof(input), "%d", done_pipe[0]); + snprintf(expected, sizeof(expected), "%zu", target); + snprintf(shared_arg, sizeof(shared_arg), "%d", shared_fd); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, target, 0UL, 0UL, 0UL)) + _exit(2); + execl("/proc/self/exe", "/proc/self/exe", "--remote", output, + input, expected, shared_arg, NULL); + _exit(3); + } + close(ready_pipe[1]); + close(done_pipe[0]); + ok = child > 0 && + read(ready_pipe[0], &ready, sizeof(ready)) == sizeof(ready); + fd = open_proc(child, "pagemap", O_RDONLY); + ok = ok && ready.page == target && fd >= 0 && + entries(fd, ready.address, target, actual, NPAGE) && + !memcmp(actual, ready.entries, sizeof(actual)) && + scan(fd, ready.address, target); + if (!ok) + printf("# remote geometry/read failure: target=%zu reported=%zu fd=%d errno=%d\n", + target, ready.page, fd, errno); + if (ok) { + size_t target_offset = target == 4096 ? 4096 : 0; + + /* Compare physical byte addresses, not differently sized PFNs. */ + ok &= (self_entry & FRAME) && (ready.shared_entry & FRAME) && + (self_entry & FRAME) * own_page - offset == + (ready.shared_entry & FRAME) * target - + target_offset; + if (!ok) + printf("# shared PFN mismatch: own_nonzero=%d target_nonzero=%d byte_delta=%lld\n", + !!(self_entry & FRAME), + !!(ready.shared_entry & FRAME), + (long long)((self_entry & FRAME) * own_page - + offset - + (ready.shared_entry & FRAME) * + target + + target_offset)); + } + if (fd >= 0) + close(fd); + write(done_pipe[1], "x", 1); + close(done_pipe[1]); + close(ready_pipe[0]); + ok &= child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + munmap(shared, 16384); + close(shared_fd); + return ok; +} + +int main(int argc, char **argv) +{ + size_t page = getauxval(AT_PAGESZ); + unsigned char *p; + uint64_t values[NPAGE], again[NPAGE]; + int fd, clear, i; + bool ok; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--remote")) + return remote_worker(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10), atoi(argv[5])); + if (argc > 2 || (argc == 2 && strcmp(argv[1], "--expect-4k")) || + page != (argc == 2 ? 4096UL : 16384UL)) + return 2; + ksft_print_header(); + ksft_set_plan(6); + p = mapping(page); + fd = open_proc(getpid(), "pagemap", O_RDONLY); + if (!p || fd < 0) + ksft_exit_fail_msg("proc mapping setup: %s\n", strerror(errno)); + ok = entries(fd, (uintptr_t)p, page, values, NPAGE); + ok &= (values[0] & PRESENT) && (values[1] & PRESENT) && !values[2] && + (values[3] & GUARD) && !(values[3] & PRESENT) && + (values[4] & PRESENT); + ok &= (values[0] & EXCLUSIVE) && (values[1] & EXCLUSIVE) && + (values[4] & EXCLUSIVE); + ksft_test_result( + ok, + "pagemap reports each target leaf, hole, guard and PROT_NONE entry\n"); + ok = true; + for (i = 0; i < NPAGE; i++) { + uint64_t value; + + ok &= entries(fd, (uintptr_t)p + i * page, page, &value, 1) && + value == values[i]; + } + errno = 0; + ok &= pread(fd, again, 7, 0) == -1 && errno == EINVAL; + ksft_test_result( + ok, + "pagemap per-entry offsets agree with batch reads and reject short entries\n"); + ksft_test_result( + scan(fd, (uintptr_t)p, page), + "PAGEMAP_SCAN ranges, max_pages and alignment use target granules\n"); + clear = open_proc(getpid(), "clear_refs", O_WRONLY); + ok = clear >= 0; + for (i = 1; ok && i <= 3; i++) { + char operation = '0' + i; + + ok &= write(clear, &operation, 1) == 1; + } + if (clear >= 0) + close(clear); + ok &= entries(fd, (uintptr_t)p, page, again, NPAGE) && + !memcmp(values, again, sizeof(values)) && p[0] == 0x67 && + p[page] == 0x67; + ksft_test_result( + ok, + "clear_refs traverses short and split VMAs without changing mappings or bytes\n"); + ksft_test_result( + remote_read(page), + "opposite-ABI reader uses target pagemap and scan geometry\n"); + { + pid_t child = fork(); + int status; + + if (!child) { + uint64_t value = 0; + int unprivileged; + + if (setuid(1000) || prctl(PR_SET_DUMPABLE, 1)) + _exit(2); + unprivileged = open_proc(getpid(), "pagemap", O_RDONLY); + ok = unprivileged >= 0 && + entries(unprivileged, (uintptr_t)p, page, &value, + 1) && + (value & PRESENT) && !(value & FRAME); + if (!ok) + printf("# unprivileged fd=%d errno=%d flags=%#llx\n", + unprivileged, errno, + (unsigned long long)(value & ~FRAME)); + _exit(!ok); + } + ok = child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + ksft_test_result( + ok, + "unprivileged pagemap reads retain flags and hide physical addresses\n"); + } + close(fd); + munmap(p, NPAGE * page); + ksft_finished(); +} diff --git a/tools/testing/selftests/arm64/user4k/reclaim-advice.c b/tools/testing/selftests/arm64/user4k/reclaim-advice.c new file mode 100644 index 00000000000000..92f28d49b830c4 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/reclaim-advice.c @@ -0,0 +1,366 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static unsigned long swapouts(void) +{ + FILE *f = fopen("/proc/vmstat", "re"); + char key[128]; + unsigned long value, result = 0; + + if (!f) + return 0; + while (fscanf(f, "%127s %lu", key, &value) == 2) + if (!strcmp(key, "pswpout")) + result = value; + fclose(f); + return result; +} + +static unsigned long swapped_kb(void) +{ + FILE *f = fopen("/proc/self/status", "re"); + char line[256]; + unsigned long result = 0; + + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "VmSwap: %lu kB", &result) == 1) + break; + fclose(f); + return result; +} + +static int advise(void *addr, size_t length, int operation, const char *name) +{ + int ret = madvise(addr, length, operation); + + if (ret) + printf("# %s failed: errno=%d (%s)\n", name, errno, + strerror(errno)); + return ret; +} + +static unsigned char pattern(size_t i, unsigned int round) +{ + return (i ^ (i >> 12) ^ (i >> 20) ^ (round * 0x55)) & 255; +} + +static size_t swapped_pages(unsigned char *p, size_t length, size_t page) +{ + uint64_t *values = calloc(length / page, sizeof(*values)); + int fd = open("/proc/self/pagemap", O_RDONLY); + size_t i, count = length / page, swapped = 0; + + if (!values || fd < 0 || + pread(fd, values, count * sizeof(*values), + (uintptr_t)p / page * sizeof(*values)) != + (ssize_t)(count * sizeof(*values))) { + swapped = SIZE_MAX; + } else { + for (i = 0; i < count; i++) + swapped += !!(values[i] & (1ULL << 62)) && + !(values[i] & (1ULL << 63)); + } + if (fd >= 0) + close(fd); + free(values); + return swapped; +} + +static void diagnose_resident(unsigned char *p, size_t length, size_t page) +{ + int pm = open("/proc/self/pagemap", O_RDONLY); + int flags = open("/proc/kpageflags", O_RDONLY); + int count = open("/proc/kpagecount", O_RDONLY); + + for (size_t i = 0; i < length / page; i++) { + uint64_t entry = 0, state = 0, maps = 0; + unsigned long native_pfn; + + if (pread(pm, &entry, 8, ((uintptr_t)p / page + i) * 8) != 8 || + !(entry & (1ULL << 63))) + continue; + native_pfn = (entry & ((1ULL << 55) - 1)) * page / 16384; + if (pread(flags, &state, 8, native_pfn * 8) != 8 || + pread(count, &maps, 8, native_pfn * 8) != 8) + printf("# resident metadata read failed: errno=%d\n", + errno); + printf("# resident leaf=%zu native_pfn=%#lx flags=%#llx maps=%llu\n", + i, native_pfn, (unsigned long long)state, + (unsigned long long)maps); + } + close(pm); + close(flags); + close(count); +} + +/* PAGEOUT drains only the calling CPU's pending LRU additions. */ +static bool drain_pageout_cpus(unsigned char *p, size_t length) +{ + cpu_set_t allowed, one; + bool ok = true; + + if (sched_getaffinity(0, sizeof(allowed), &allowed)) + return false; + for (unsigned int cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &allowed)) + continue; + CPU_ZERO(&one); + CPU_SET(cpu, &one); + if (sched_setaffinity(0, sizeof(one), &one)) { + ok = false; + break; + } + ok &= advise(p, length, MADV_COLD, "CPU drain COLD") == 0; + ok &= advise(p, length, MADV_PAGEOUT, "CPU drain PAGEOUT") == 0; + } + ok &= sched_setaffinity(0, sizeof(allowed), &allowed) == 0; + return ok; +} + +static bool swapped_pagemap(unsigned char *p, size_t length, size_t page) +{ + uint64_t *values = calloc(length / page, sizeof(*values)); + int fd = open("/proc/self/pagemap", O_RDONLY); + size_t i, count = length / page; + bool ok = values && fd >= 0; + + if (ok) + ok = pread(fd, values, count * sizeof(*values), + (uintptr_t)p / page * sizeof(*values)) == + (ssize_t)(count * sizeof(*values)); + for (i = 0; ok && i < count; i++) { + uint64_t frame = values[i] & ((1ULL << 55) - 1); + + ok &= !!(values[i] & (1ULL << 62)) && + !(values[i] & (1ULL << 63)); + if (page == 4096) { + ok &= ((frame >> 5) & 3) == + (((uintptr_t)p / page + i) & 3); + if (i && (((uintptr_t)p / page + i) & 3)) + ok &= (values[i] & ((1ULL << 55) - 1)) == + (values[i - 1] & ((1ULL << 55) - 1)) + + (1U << 5); + } + } + if (fd >= 0) + close(fd); + free(values); + printf("%s - %zuK pagemap reports swapped leaves and exact quarter offsets\n", + ok ? "ok" : "not ok", page / 1024); + return ok; +} + +static bool check_swap_pss(pid_t pid, void *p, size_t length, + unsigned long expected_swap, + unsigned long expected_pss) +{ + char path[64], line[512]; + unsigned long start, end, swap = 0, pss = 0; + unsigned int fields = 0; + bool found = false, ok; + FILE *f; + + snprintf(path, sizeof(path), "/proc/%d/smaps", pid); + f = fopen(path, "re"); + if (!f) + return false; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) { + if (found) + break; + found = start == (uintptr_t)p && + end == (uintptr_t)p + length; + continue; + } + if (!found) + continue; + if (sscanf(line, "Swap: %lu", &swap) == 1) + fields |= 1; + else if (sscanf(line, "SwapPss: %lu", &pss) == 1) + fields |= 2; + } + fclose(f); + ok = found && fields == 3 && swap == expected_swap && + pss == expected_pss; + if (!ok) + printf("# swap stats found=%d fields=%u Swap=%lu/%lu SwapPss=%lu/%lu\n", + found, fields, swap, expected_swap, pss, expected_pss); + return ok; +} + +static bool swapped_accounting(unsigned char *p, size_t length, size_t page, + bool forked) +{ + int ready[2], go[2], status; + pid_t child; + char byte; + bool ok = check_swap_pss(getpid(), p, length, length / 1024, + length / 1024); + + if (!forked || !ok) + goto out; + if (pipe(ready) || pipe(go)) + return false; + child = fork(); + if (child < 0) + return false; + if (!child) { + close(ready[0]); + close(go[1]); + if (read(go[0], &byte, 1) != 1) + _exit(1); + p[page] ^= 0x37; + if (write(ready[1], "x", 1) != 1 || read(go[0], &byte, 1) != 1) + _exit(2); + _exit(0); + } + close(ready[1]); + close(go[0]); + ok &= check_swap_pss(getpid(), p, length, length / 1024, length / 2048); + ok &= check_swap_pss(child, p, length, length / 1024, length / 2048); + ok &= write(go[1], "x", 1) == 1 && read(ready[0], &byte, 1) == 1; + ok &= check_swap_pss(getpid(), p, length, length / 1024, + (length + page) / 2048); + ok &= check_swap_pss(child, p, length, (length - page) / 1024, + (length - page) / 2048); + write(go[1], "x", 1); + close(go[1]); + close(ready[0]); + ok &= waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status); + ok &= check_swap_pss(getpid(), p, length, length / 1024, length / 1024); +out: + printf("%s - %zuK SwapPss %s\n", ok ? "ok" : "not ok", page / 1024, + forked ? "fork, one-leaf swap-in/COW and child exit" : + "exclusive quarters"); + return ok; +} + +int main(int argc, char **argv) +{ + const size_t length = 4 * 1024 * 1024; + size_t page = getauxval(AT_PAGESZ), count = length / page, i, j, absent; + unsigned char *p, *vec, *reservation; + unsigned long before, after, swapped_before, swapped_after; + unsigned int round, tries; + size_t bad_bytes, bad_residency; + bool ok = true; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc > 2 || (argc == 2 && strcmp(argv[1], "--expect-4k")) || + page != (argc == 2 ? 4096UL : 16384UL)) + return 2; + reservation = mmap(NULL, length + 2 * page, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (reservation == MAP_FAILED) + return 3; + p = reservation + page; + vec = malloc(count); + if (!vec || mprotect(p, length, PROT_READ | PROT_WRITE)) + return 3; + for (round = 0; round < 4 && ok; round++) { + for (i = 0; i < length; i++) + p[i] = pattern(i, round); + before = swapouts(); + swapped_before = swapped_kb(); + ok = advise(p, length, MADV_COLD, "COLD") == 0; + ok &= advise(p, length, MADV_PAGEOUT, "PAGEOUT") == 0; + after = before; + absent = 0; + for (tries = 0; ok && tries < 200; tries++) { + if (mincore(p, length, vec)) { + ok = false; + break; + } + absent = 0; + for (i = 0; i < count; i++) + absent += !(vec[i] & 1); + after = swapouts(); + if (after > before && + swapped_pages(p, length, page) == count) + break; + if (tries == 32) { + printf("# checking pending remote LRU additions before CPU sweep\n"); + diagnose_resident(p, length, page); + ok &= drain_pageout_cpus(p, length); + printf("# after CPU sweep: swapped=%zu/%zu\n", + swapped_pages(p, length, page), count); + } + /* PAGEOUT may skip busy folios; establish the full-swap fixture. */ + usleep(10000); + ok &= advise(p, length, MADV_COLD, "retry COLD") == 0; + ok &= advise(p, length, MADV_PAGEOUT, + "retry PAGEOUT") == 0; + } + printf("# pageout setup attempts=%u\n", tries + 1); + if (tries == 200) + diagnose_resident(p, length, page); + ok &= tries < 200; + swapped_after = swapped_kb(); + /* mincore also counts warm swap-cache data as resident. */ + ok &= after > before && swapped_after > swapped_before; + ok &= swapped_pagemap(p, length, page); + ok &= swapped_accounting(p, length, page, round == 1); + if (!round) + ok &= advise(p + page, page, MADV_WILLNEED, + "WILLNEED") == 0; + else { + if (round == 3) { + ok &= mlock(p, length) == 0; + ok &= swapped_pages(p, length, page) == 0; + } else { + ok &= advise(p, length, MADV_POPULATE_READ, + "POPULATE_READ") == 0; + } + ok &= mincore(p, length, vec) == 0; + bad_residency = 0; + for (i = 0; i < count; i++) + bad_residency += !(vec[i] & 1); + printf("# after %s: absent=%zu/%zu\n", + round == 3 ? "mlock" : "POPULATE_READ", + bad_residency, count); + ok &= bad_residency == 0; + } + bad_bytes = 0; + /* Nonsequential user-page order, every byte checked after swap. */ + for (i = 0; i < count; i++) { + size_t index = ((i * 61) & (count - 1)) * page; + + for (j = index; j < index + page; j++) + if (p[j] != pattern(j, round)) { + if (!bad_bytes) + printf("# first bad byte at %#zx: got=%#x expected=%#x\n", + j, p[j], + pattern(j, round)); + bad_bytes++; + ok = false; + } + } + if (round == 3) + ok &= munlock(p, length) == 0; + printf("# bad bytes=%zu\n", bad_bytes); + printf("%s - %zuK EL0 pageout round %u: pswpout=%lu->%lu VmSwap=%lu->%lu kB absent=%zu/%zu %s\n", + ok ? "ok" : "not ok", page / 1024, round, before, after, + swapped_before, swapped_after, absent, count, + round == 3 ? "mlock/readback" : + round ? "prefault/readback" : + "willneed/readback"); + } + munmap(reservation, length + 2 * page); + free(vec); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/runtime-app-init.c b/tools/testing/selftests/arm64/user4k/runtime-app-init.c new file mode 100644 index 00000000000000..9d1c7c2c49351b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/runtime-app-init.c @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +int main(int argc, char **argv) +{ + if (argc >= 4 && !strcmp(argv[1], "--exec")) { + if (prctl(SET_EXEC_PAGE_SIZE, strtoul(argv[2], NULL, 10), 0UL, + 0UL, 0UL)) + return 8; + execv(argv[3], argv + 3); + perror("exec"); + return 9; + } + if (getpid() != 1) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + int failed = 0; + pid_t children[3]; + unsigned long sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + unsigned count = sizes[0] > 16384 ? 3 : 2; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)) { + perror("mount"); + failed = 1; + goto done; + } + mkdir("/dev/shm", 0777); + mount("tmpfs", "/dev/shm", "tmpfs", 0, NULL); + for (unsigned i = 0; i < count; i++) { + children[i] = fork(); + if (!children[i]) { + char ps[32]; + snprintf(ps, sizeof(ps), "%lu", sizes[i]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(3); + execl("/usr/bin/python3", "/usr/bin/python3", "-I", + "/runtime-app.py", ps, NULL); + perror("exec Python"); + _exit(4); + } + } + for (unsigned i = 0; i < count; i++) { + int status = 0; + int ok = children[i] > 0 && + waitpid(children[i], &status, 0) == children[i] && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - distro Python ABI %luK status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + failed |= !ok; + } +done: + if (failed) { + FILE *f = fopen("/trace4k", "r"); + if (f) { + char line[2048]; + while (fgets(line, sizeof(line), f)) + fputs(line, stdout); + fclose(f); + } + } + printf("GRANULE DISTRO RUNTIME %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/runtime-app.py b/tools/testing/selftests/arm64/user4k/runtime-app.py new file mode 100755 index 00000000000000..47fac03300efcc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/runtime-app.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: GPL-2.0-only +import os,sys,mmap,ctypes,threading,concurrent.futures,tempfile,subprocess,signal,zlib,gzip,lzma,hashlib,socket,select,json,sqlite3 +ps=int(sys.argv[1]);assert os.sysconf('SC_PAGE_SIZE')==ps==mmap.PAGESIZE,(ps,mmap.PAGESIZE) +libc=ctypes.CDLL(None,use_errno=True);libc.prctl.argtypes=[ctypes.c_int,ctypes.c_ulong,ctypes.c_ulong,ctypes.c_ulong,ctypes.c_ulong] +def passed(name):print(f'ok - distro Python {ps//1024}K {name}',flush=True) +passed('dynamic loader and page-size contract') +with mmap.mmap(-1, 2*ps, flags=0x08|mmap.MAP_ANONYMOUS) as droppable: + assert droppable[:]==bytes(2*ps) + droppable[:]=bytes([0x45])*(2*ps) + child=os.fork() + if child==0:os._exit(0 if droppable[:]==bytes(2*ps) else 9) + assert os.waitpid(child,0)[1]==0 + droppable.madvise(mmap.MADV_DONTNEED) + assert droppable[:]==bytes(2*ps) +libc.arc4random_buf.argtypes=[ctypes.c_void_p,ctypes.c_size_t] +random_buffer=ctypes.create_string_buffer(32) +random_samples=set() +for _ in range(128): + libc.arc4random_buf(random_buffer,len(random_buffer));random_samples.add(random_buffer.raw) +assert len(random_samples)==128 +passed('droppable write, fork wipe, discard and glibc random state') + +blob=bytes(range(256))*4096 +with tempfile.TemporaryDirectory() as tmp: + path=tmp+'/mapped' + with open(path,'w+b') as f: + f.truncate(8*ps) + with mmap.mmap(f.fileno(),2*ps,offset=ps) as m: + m[:]=bytes([0x53])*(2*ps);m.flush() + assert os.pread(f.fileno(),2*ps,ps)==m[:] + m.resize(3*ps);m[2*ps:]=bytes([0x72])*ps;m.flush() + assert os.pread(f.fileno(),ps,3*ps)==bytes([0x72])*ps + passed('file mmap offset, flush and resize') + with mmap.mmap(-1,4*ps,flags=mmap.MAP_PRIVATE|mmap.MAP_ANONYMOUS) as m: + m[:]=bytes([0x39])*(4*ps) + c=os.fork() + if c==0: + m[ps:2*ps]=bytes([0x91])*ps + os._exit(0 if m[:ps]==bytes([0x39])*ps else 5) + assert os.waitpid(c,0)[1]==0 and m[:]==bytes([0x39])*(4*ps) + address=ctypes.addressof(ctypes.c_char.from_buffer(m)) + libc.mprotect.argtypes=[ctypes.c_void_p,ctypes.c_size_t,ctypes.c_int] + assert libc.mprotect(address+ps,ps,0)==0 + c=os.fork() + if c==0: + m[ps]=0 + os._exit(6) + status=os.waitpid(c,0)[1];assert os.WIFSIGNALED(status) and os.WTERMSIG(status)==signal.SIGSEGV + assert m[0]==0x39 and m[2*ps]==0x39 + assert libc.mprotect(address+ps,ps,mmap.PROT_READ|mmap.PROT_WRITE)==0 + passed('fork COW and independent protection fault') + def compressed(i): + d=blob+str(i).encode() + for compress,decompress in [(zlib.compress,zlib.decompress),(gzip.compress,gzip.decompress),(lambda d:lzma.compress(d,preset=1),lzma.decompress)]:assert decompress(compress(d))==d + return hashlib.sha256(d).hexdigest() + with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool: + results=list(pool.map(compressed,range(8))) + assert len(set(results))==8 + passed('threaded native extension allocation, compression and hashing') + db=sqlite3.connect(tmp+'/db') + db.execute('pragma journal_mode=wal');db.execute('pragma mmap_size=1048576') + db.execute('create table entries (n integer primary key, data blob)') + db.executemany('insert into entries values (?,?)',[(i,blob[:4096]) for i in range(128)]);db.commit() + assert db.execute('select count(*) from entries').fetchone()[0]==128 + assert all(row[0]==blob[:4096] for row in db.execute('select data from entries'));db.close() + passed('SQLite WAL and mapped database reads') + left,right=socket.socketpair() + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + task=pool.submit(lambda:(right.sendall(blob),right.shutdown(socket.SHUT_WR))) + out=bytearray() + while True: + assert select.select([left],[],[],5)[0] + part=left.recv(997) + if not part:break + out.extend(part) + task.result();assert bytes(out)==blob + left.close();right.close();passed('socket polling and fragmented transfers') + fd=os.memfd_create('cross-abi',0);os.ftruncate(fd,131072);os.pwrite(fd,b'parent',ps) + # Execute the child in the other ABI, retaining the same shared file bytes. + target=16384 if ps==4096 else 4096 + def select_abi(): + if libc.prctl(0x41555001,target,0,0,0):os._exit(7) + code="import os,mmap,sys;fd=int(sys.argv[1]);off=int(sys.argv[2]);want=int(sys.argv[3]);assert os.sysconf('SC_PAGE_SIZE')==want;m=mmap.mmap(fd,131072);assert m[off:off+6]==b'parent';m[off:off+5]=b'child';m.close()" + subprocess.run([sys.executable,'-I','-c',code,str(fd),str(ps),str(target)],pass_fds=(fd,),preexec_fn=select_abi,check=True) + assert os.pread(fd,5,ps)==b'child';os.close(fd);passed('cross-ABI dynamic exec and memfd sharing') + command="set -e; a=(); for ((i=0;i<2048;i++)); do a+=(\"value-$i\"); done; [[ ${#a[@]} == 2048 ]]; [[ ${a[2047]} == value-2047 ]]; printf 'bash-ok\\n'" + def same_abi(): + if libc.prctl(0x41555001,ps,0,0,0):os._exit(8) + result=subprocess.run(['/bin/bash','--noprofile','--norc','-c',command],preexec_fn=same_abi,stdout=subprocess.PIPE,check=True,text=True) + assert result.stdout=='bash-ok\n';passed('distro Bash dynamic exec, allocation and pipes') +print(f'DISTRO PYTHON {ps} PASS',flush=True) diff --git a/tools/testing/selftests/arm64/user4k/same-native.c b/tools/testing/selftests/arm64/user4k/same-native.c new file mode 100644 index 00000000000000..9c592ee98c3a73 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/same-native.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Three generations; actual PFNs distinguish same-native-folio COW coverage. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define LEAF 4096UL +#define NATIVE 16384UL +#define TRIALS 16 +#define PFN_MASK ((UINT64_C(1) << 55) - 1) +struct result { uint64_t old_pfn, new_pfn, held_pfn; unsigned quarter; }; +static volatile sig_atomic_t group; +static unsigned char *data; +static void stop(int sig) +{ + if (group > 0) kill(-group, SIGKILL); + _exit(128 + sig); +} +static bool io_full(int fd, void *buf, size_t bytes, bool output) +{ + unsigned char *p = buf; + while (bytes) { + ssize_t n = output ? write(fd, p, bytes) : read(fd, p, bytes); + if (n < 0 && errno == EINTR) continue; + if (n <= 0) return false; + p += n; bytes -= n; + } + return true; +} +static unsigned char pattern(size_t at, unsigned generation, unsigned quarter) +{ + unsigned salt = at / LEAF == quarter ? generation * 53 : 0; + return (unsigned char)((at * 17 + at / LEAF * 29 + at / 256 + salt) % 251 + 1); +} +static void change(unsigned generation, unsigned quarter) +{ + size_t at; + for (at = quarter * LEAF; at < (quarter + 1) * LEAF; at++) + data[at] = pattern(at, generation, quarter); +} +static bool verify(unsigned generation, unsigned quarter) +{ + size_t at; + for (at = 0; at < NATIVE; at++) + if (data[at] != pattern(at, generation, quarter)) return false; + return true; +} +static bool pfn(int fd, unsigned quarter, uint64_t *out) +{ + uint64_t entry; + off_t where = ((uintptr_t)data / LEAF + quarter) * 8; + if (pread(fd, &entry, 8, where) != 8 || !(entry & (UINT64_C(1) << 63)) || + (entry & (UINT64_C(1) << 62)) || !(entry & PFN_MASK)) return false; + *out = entry & PFN_MASK; + return true; +} +static bool wait_ok(pid_t child) +{ + int status; pid_t got; + do { got = waitpid(child, &status, 0); } while (got < 0 && errno == EINTR); + return got == child && WIFEXITED(status) && WEXITSTATUS(status) == 0; +} +static bool caps(void) +{ + FILE *f = fopen("/proc/self/status", "re"); + char line[256]; unsigned long long effective = 0; + if (!f) return false; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "CapEff: %llx", &effective) == 1) break; + fclose(f); return !!(effective & (1ULL << 21)); +} +static void owner(int output, unsigned quarter) +{ + struct result result = { .quarter = quarter }; + int go[2], answer[2], mapfd; + char token; + pid_t held; + alarm(5); + if (pipe(go) || pipe(answer)) _exit(10); + /* Open in this mm, never inherit an ancestor's already-open pagemap fd. */ + mapfd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (mapfd < 0) _exit(11); + change(1, quarter); + if (!pfn(mapfd, quarter, &result.old_pfn) || !verify(1, quarter)) _exit(12); + held = fork(); + if (held < 0) _exit(13); + if (!held) { + uint64_t held_pfn; + close(go[1]); close(answer[0]); close(output); close(mapfd); + alarm(4); + if (!io_full(go[0], &token, 1, false) || token != 'r') _exit(14); + mapfd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (mapfd < 0 || !pfn(mapfd, quarter, &held_pfn) || + held_pfn != result.old_pfn || !verify(1, quarter) || + !io_full(answer[1], &held_pfn, sizeof(held_pfn), true)) _exit(15); + close(mapfd); close(go[0]); close(answer[1]); _exit(0); + } + /* Before libc/pipe bookkeeping, consume a free sibling if allocator picks it. */ + change(2, quarter); + if (!pfn(mapfd, quarter, &result.new_pfn) || + result.new_pfn == result.old_pfn || !verify(2, quarter)) _exit(16); + close(go[0]); close(answer[1]); token = 'r'; + if (!io_full(go[1], &token, 1, true) || + !io_full(answer[0], &result.held_pfn, sizeof(result.held_pfn), false) || + !wait_ok(held) || !verify(2, quarter)) _exit(17); + close(go[1]); close(answer[0]); close(mapfd); + if (!io_full(output, &result, sizeof(result), true)) _exit(18); + close(output); _exit(0); +} +int main(int argc, char **argv) +{ + unsigned same = 0; + if (argc == 2 && !strcmp(argv[1], "--selftest")) { + for (unsigned q = 0; q < 4; q++) + for (size_t at = 0; at < NATIVE; at++) + if ((pattern(at, 1, q) != pattern(at, 2, q)) != (at / LEAF == q)) return 1; + puts("PASS generation pattern selftest"); return 0; + } + if (argc != 2 || strcmp(argv[1], "--native-16k") || + getauxval(AT_PAGESZ) != LEAF || !caps()) { + fputs("Requires established4K launcher/native16K kernel and visible PFNs\n", stderr); return 77; + } + if (prctl(PR_SET_CHILD_SUBREAPER, 1)) return 1; + signal(SIGPIPE, SIG_IGN); signal(SIGALRM, stop); signal(SIGTERM, stop); signal(SIGINT, stop); + setvbuf(stdout, NULL, _IONBF, 0); alarm(20); + void *map = mmap(NULL, 2 * NATIVE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (map == MAP_FAILED) return 1; + data = (void *)(((uintptr_t)map + NATIVE - 1) & ~(NATIVE - 1)); + for (size_t at = 0; at < NATIVE; at++) data[at] = pattern(at, 0, 0); + for (unsigned trial = 0; trial < TRIALS; trial++) { + int output[2]; struct result result; pid_t pid; + if (pipe(output)) return 1; + pid = fork(); + if (pid < 0) return 1; + if (!pid) { + group = 0; + if (setpgid(0, 0)) _exit(1); + close(output[0]); owner(output[1], trial % 4); + } + group = pid; + if (setpgid(pid, pid) && errno != EACCES && errno != ESRCH) { + kill(pid, SIGKILL); stop(SIGTERM); + } + close(output[1]); + bool received = io_full(output[0], &result, sizeof(result), false); + close(output[0]); + bool good = wait_ok(pid); + /* Reap any orphaned held generation on failure; no child can escape group. */ + kill(-pid, SIGKILL); + while (waitpid(-1, NULL, 0) > 0 || errno == EINTR) {} + group = 0; + if (!received || !good || result.held_pfn != result.old_pfn || + result.new_pfn == result.old_pfn || !verify(0, 0)) { + fprintf(stderr, "FAIL trial %u generation/PFN/isolation/cleanup\n", trial); return 1; + } + bool native_equal = result.old_pfn / 4 == result.new_pfn / 4; + same += native_equal; + printf("{\"trial\":%u,\"logical_quarter\":%u,\"old_pfn4k\":%"PRIu64",\"new_pfn4k\":%"PRIu64",\"held_pfn4k\":%"PRIu64",\"same_native\":%s,\"three_generation_bytes_ok\":true}\n", + trial, result.quarter, result.old_pfn, result.new_pfn, result.held_pfn, native_equal ? "true" : "false"); + } + munmap(map, 2 * NATIVE); alarm(0); + printf("{\"result\":\"%s\",\"trials\":%u,\"same_native_exercised\":%u,\"payload_bytes\":%lu}\n", same ? "PASS" : "UNEXERCISED", TRIALS, same, NATIVE); + return same ? 0 : 77; +} diff --git a/tools/testing/selftests/arm64/user4k/same-quarter.c b/tools/testing/selftests/arm64/user4k/same-quarter.c new file mode 100644 index 00000000000000..ce1f7e80081d04 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/same-quarter.c @@ -0,0 +1,335 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Metadata-only native-backing comparison. Never changes memory policy. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define LEAF 4096UL +#define NATIVE 16384UL +#define BYTES (16UL << 20) +#define LEAVES (BYTES / LEAF) +#define WINDOWS (BYTES / NATIVE) +#define MAX_FRAMES (LEAVES * 2) + +struct coverage { + uint64_t native_units, quarters, occupancy[4]; +}; +struct result { + unsigned arm; + bool content_ok, disjoint_ok; + struct coverage parent, child, modified, unchanged, combined; + uint64_t modified_parent_native_overlap; + uint64_t modified_unchanged_native_overlap; + uint64_t modified_quarter_counts[4]; +}; +static volatile sig_atomic_t fixture_group; + +static void stop_fixture(int sig) +{ + if (fixture_group > 0) + kill(-fixture_group, SIGKILL); + _exit(128 + sig); +} +static bool full_io(int fd, void *buffer, size_t bytes, bool writing) +{ + char *p = buffer; + while (bytes) { + ssize_t n = writing ? write(fd, p, bytes) : read(fd, p, bytes); + if (n < 0 && errno == EINTR) + continue; + if (n <= 0) + return false; + p += n; + bytes -= (size_t)n; + } + return true; +} +static bool wait_ok(pid_t pid) +{ + int status; + pid_t got; + do { got = waitpid(pid, &status, 0); } while (got < 0 && errno == EINTR); + return got > 0 && WIFEXITED(status) && WEXITSTATUS(status) == 0; +} +static int compare_u64(const void *a, const void *b) +{ + uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b; + return (x > y) - (x < y); +} +/* Input is actual4K PFN, never VMA-derived quarter position. Duplicates count once. */ +static struct coverage summarize(uint64_t *frames, size_t count) +{ + struct coverage out = {}; + uint64_t last = UINT64_MAX; + unsigned mask = 0; + qsort(frames, count, sizeof(*frames), compare_u64); + for (size_t i = 0; i < count; i++) { + uint64_t native = frames[i] / 4; + if (native != last) { + if (mask) { + unsigned n = __builtin_popcount(mask); + out.native_units++; + out.quarters += n; + out.occupancy[n - 1]++; + } + last = native; + mask = 0; + } + mask |= 1U << (frames[i] % 4); + } + if (mask) { + unsigned n = __builtin_popcount(mask); + out.native_units++; + out.quarters += n; + out.occupancy[n - 1]++; + } + return out; +} +/* Both arrays were sorted by summarize. Count each common nativeunit once. */ +static uint64_t native_overlap(const uint64_t *a, size_t an, const uint64_t *b, size_t bn) +{ + size_t ai = 0, bi = 0; + uint64_t count = 0; + while (ai < an && bi < bn) { + uint64_t av = a[ai] / 4, bv = b[bi] / 4; + if (av == bv) + count++; + if (av <= bv) + while (ai < an && a[ai] / 4 == av) ai++; + if (bv <= av) + while (bi < bn && b[bi] / 4 == bv) bi++; + } + return count; +} +static bool read_frames(pid_t pid, const unsigned char *data, uint64_t frames[LEAVES]) +{ + char path[64]; + uint64_t entries[LEAVES]; + snprintf(path, sizeof(path), "/proc/%ld/pagemap", (long)pid); + int fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return false; + ssize_t n = pread(fd, entries, sizeof(entries), (uintptr_t)data / LEAF * 8); + close(fd); + if (n != (ssize_t)sizeof(entries)) + return false; + for (size_t i = 0; i < LEAVES; i++) { + uint64_t pfn = entries[i] & ((UINT64_C(1) << 55) - 1); + if (!(entries[i] & (UINT64_C(1) << 63)) || !pfn) + return false; /* Swap/hiddenPFNs invalidate this bounded resident comparison. */ + frames[i] = pfn; + } + return true; +} +static unsigned chosen_quarter(unsigned arm, size_t window) +{ + return arm ? window % 4 : 0; +} +static unsigned char tag(size_t leaf, bool changed) +{ + unsigned char value = (unsigned char)(leaf % 251 + 1); + return changed ? value ^ 0xa5 : value; +} +static bool contents(const unsigned char *data, unsigned arm, bool child) +{ + for (size_t leaf = 0; leaf < LEAVES; leaf++) { + bool changed = child && leaf % 4 == chosen_quarter(arm, leaf / 4); + if (data[leaf * LEAF] != tag(leaf, changed)) + return false; + for (size_t byte = 1; byte < LEAF; byte++) + if (data[leaf * LEAF + byte] != 0x5a) + return false; + } + return true; +} +static void owner(unsigned arm, int output) +{ + uint64_t parent[LEAVES], child_frames[LEAVES], modified[WINDOWS]; + uint64_t unchanged[LEAVES - WINDOWS], combined[MAX_FRAMES]; + struct result result = {.arm = arm}; + int ready[2], release[2]; + char token; + size_t mi = 0, ui = 0; + alarm(20); + void *mapping = mmap(NULL, BYTES + 2 * NATIVE, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping == MAP_FAILED) + _exit(2); + unsigned char *data = (void *)(((uintptr_t)mapping + NATIVE - 1) & ~(NATIVE - 1)); + if (mprotect(data, BYTES, PROT_READ | PROT_WRITE)) + _exit(2); + memset(data, 0x5a, BYTES); + for (size_t leaf = 0; leaf < LEAVES; leaf++) + data[leaf * LEAF] = tag(leaf, false); + if (!contents(data, arm, false) || !read_frames(getpid(), data, parent)) + _exit(3); + struct coverage initial = summarize(parent, LEAVES); + if (initial.native_units * NATIVE != BYTES || initial.quarters != LEAVES) + _exit(4); /* Stop beforefork if the32MiB payload ceiling is not established. */ + if (pipe(ready) || pipe(release)) + _exit(2); + pid_t child = fork(); + if (child < 0) + _exit(2); + if (!child) { + alarm(20); + close(ready[0]); close(release[1]); close(output); + for (size_t window = 0; window < WINDOWS; window++) { + size_t leaf = window * 4 + chosen_quarter(arm, window); + *(volatile unsigned char *)(data + leaf * LEAF) = tag(leaf, true); + } + if (!contents(data, arm, true)) + _exit(3); + token = 'r'; + if (!full_io(ready[1], &token, 1, true)) + _exit(2); + close(ready[1]); + if (!full_io(release[0], &token, 1, false) || token != 'x') + _exit(2); + close(release[0]); + _exit(contents(data, arm, true) ? 0 : 3); + } + close(ready[1]); close(release[0]); + if (!full_io(ready[0], &token, 1, false) || token != 'r') + _exit(2); + close(ready[0]); + result.content_ok = contents(data, arm, false); + if (!result.content_ok || !read_frames(getpid(), data, parent) || + !read_frames(child, data, child_frames)) + _exit(3); + result.disjoint_ok = true; + for (size_t leaf = 0; leaf < LEAVES; leaf++) { + bool changed = leaf % 4 == chosen_quarter(arm, leaf / 4); + if (changed) { + modified[mi++] = child_frames[leaf]; + result.modified_quarter_counts[child_frames[leaf] % 4]++; + result.disjoint_ok &= parent[leaf] != child_frames[leaf]; + } else { + unchanged[ui++] = child_frames[leaf]; + result.disjoint_ok &= parent[leaf] == child_frames[leaf]; + } + } + memcpy(combined, parent, sizeof(parent)); + memcpy(combined + LEAVES, child_frames, sizeof(child_frames)); + result.parent = summarize(parent, LEAVES); + result.child = summarize(child_frames, LEAVES); + result.modified = summarize(modified, mi); + result.unchanged = summarize(unchanged, ui); + result.combined = summarize(combined, MAX_FRAMES); + result.modified_parent_native_overlap = native_overlap(modified, mi, parent, LEAVES); + result.modified_unchanged_native_overlap = native_overlap(modified, mi, unchanged, ui); + result.disjoint_ok &= mi == WINDOWS && result.modified.quarters == WINDOWS && + result.combined.quarters == LEAVES + WINDOWS && + result.modified_parent_native_overlap == 0 && + result.modified_unchanged_native_overlap == 0 && + result.combined.native_units * NATIVE <= 2 * BYTES; + token = 'x'; + if (!full_io(release[1], &token, 1, true)) + _exit(2); + close(release[1]); + bool child_ok = wait_ok(child); + result.content_ok &= child_ok && contents(data, arm, false); + munmap(mapping, BYTES + 2 * NATIVE); + if (!full_io(output, &result, sizeof(result), true)) + _exit(2); + close(output); + _exit(result.content_ok && result.disjoint_ok ? 0 : 3); +} +static bool visible_pfns(void) +{ + FILE *file = fopen("/proc/self/status", "re"); + char line[256]; + unsigned long long caps = 0; + if (!file) return false; + while (fgets(line, sizeof(line), file)) + if (sscanf(line, "CapEff: %llx", &caps) == 1) break; + fclose(file); + return !!(caps & (1ULL << 21)); +} +static void print_coverage(const char *name, struct coverage c) +{ + printf("\"%s\":{\"native_bytes\":%llu,\"represented_bytes\":%llu," + "\"quarter_occupancy\":[%llu,%llu,%llu,%llu]}", name, + (unsigned long long)(c.native_units * NATIVE), + (unsigned long long)(c.quarters * LEAF), + (unsigned long long)c.occupancy[0], (unsigned long long)c.occupancy[1], + (unsigned long long)c.occupancy[2], (unsigned long long)c.occupancy[3]); +} +static int selftest(void) +{ + uint64_t a[] = {400, 401, 401, 403, 404}, b[] = {402, 405}; + uint64_t combined[] = {400, 401, 401, 403, 404, 402, 405}; + struct coverage ca = summarize(a, 5), cb = summarize(b, 2); + struct coverage cc = summarize(combined, 7); + bool ok = ca.native_units == 2 && ca.quarters == 4 && ca.occupancy[0] == 1 && + ca.occupancy[2] == 1 && cb.native_units == 2 && cb.quarters == 2 && + cc.native_units == 2 && cc.quarters == 6 && cc.occupancy[1] == 1 && + cc.occupancy[3] == 1 && native_overlap(a, 5, b, 2) == 2; + uint64_t sparse[] = {400, 404, 408, 412}, dense[] = {400, 401, 402, 403}; + ca = summarize(sparse, 4); cb = summarize(dense, 4); + ok &= ca.native_units == 4 && cb.native_units == 1 && ca.quarters == cb.quarters; + printf("metadata selftest: %s\n", ok ? "PASS" : "FAIL"); + return !ok; +} +int main(int argc, char **argv) +{ + if (argc == 2 && !strcmp(argv[1], "--selftest")) return selftest(); + if (argc != 2 || strcmp(argv[1], "--native-16k")) { fprintf(stderr, "usage: %s --native-16k | --selftest\n", argv[0]); return 2; } + if (getauxval(AT_PAGESZ) != LEAF || !visible_pfns()) { + fprintf(stderr, "SKIP: established4K launcher, native16K kernel and visiblePFNs required\n"); + return 4; + } + signal(SIGPIPE, SIG_IGN); signal(SIGALRM, stop_fixture); signal(SIGTERM, stop_fixture); + signal(SIGINT, stop_fixture); + if (prctl(PR_SET_CHILD_SUBREAPER, 1)) return 2; + alarm(30); + for (unsigned arm = 0; arm < 2; arm++) { + int pipefd[2]; struct result result = {}; bool copied, success; + if (pipe(pipefd)) return 2; + pid_t child = fork(); + if (child < 0) return 2; + if (!child) { + if (setpgid(0, 0)) _exit(2); + close(pipefd[0]); owner(arm, pipefd[1]); + } + fixture_group = child; + if (setpgid(child, child) && errno != EACCES && errno != ESRCH) stop_fixture(SIGTERM); + close(pipefd[1]); + copied = full_io(pipefd[0], &result, sizeof(result), false); + close(pipefd[0]); success = wait_ok(child); + if (!copied || !success) { + kill(-child, SIGKILL); + while (waitpid(-1, NULL, 0) > 0 || errno == EINTR) {} + fprintf(stderr, "FAIL: arm%u incomplete or integrity/coverage check failed\n", arm); + return 1; + } + fixture_group = 0; + printf("{\"arm\":\"%s\",\"logical_parent_bytes\":%lu,\"modified_child_bytes\":%lu,", + arm ? "balanced" : "same-quarter-zero", BYTES, WINDOWS * LEAF); + print_coverage("parent", result.parent); printf(","); + print_coverage("child", result.child); printf(","); + print_coverage("modified_child", result.modified); printf(","); + print_coverage("unchanged_child", result.unchanged); printf(","); + print_coverage("parent_child_union", result.combined); + printf(",\"modified_parent_native_overlap\":%llu,\"modified_unchanged_native_overlap\":%llu," + "\"modified_physical_quarter_counts\":[%llu,%llu,%llu,%llu],\"content_ok\":true,\"disjoint_ok\":true}\n", + (unsigned long long)result.modified_parent_native_overlap, + (unsigned long long)result.modified_unchanged_native_overlap, + (unsigned long long)result.modified_quarter_counts[0], (unsigned long long)result.modified_quarter_counts[1], + (unsigned long long)result.modified_quarter_counts[2], (unsigned long long)result.modified_quarter_counts[3]); + fflush(stdout); + } + alarm(0); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-collapse-init.c b/tools/testing/selftests/arm64/user4k/shmem-collapse-init.c new file mode 100644 index 00000000000000..b040d8fc43a4f3 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-collapse-init.c @@ -0,0 +1,396 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#ifndef MADV_COLLAPSE +#define MADV_COLLAPSE 25 +#endif +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - collapse line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +struct command { + uint64_t op, offset, len, value; +}; +struct ready { + uint64_t base, head; +}; +static uint64_t entry(const char *path, uint64_t index) +{ + uint64_t value; + int fd = open(path, O_RDONLY); + CHECK(fd >= 0 && pread(fd, &value, 8, index * 8) == 8 && !close(fd)); + return value; +} +static uint64_t mapped_pfn(pid_t pid, uintptr_t address, size_t ps) +{ + char path[80]; + snprintf(path, sizeof(path), "/proc/%d/pagemap", pid); + uint64_t e = entry(path, address / ps); + CHECK(e & (1ULL << 63)); + return e & ((1ULL << 55) - 1); +} +static unsigned char *map(int file, size_t len, size_t align, off_t offset) +{ + void *reserve = mmap(NULL, len + align, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + uintptr_t addr = ((uintptr_t)reserve + align - 1) & ~(align - 1); + CHECK(!munmap(reserve, len + align)); + unsigned char *p = mmap((void *)addr, len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, file, offset); + CHECK(p != MAP_FAILED); + return p; +} +static int register_range(unsigned char *p, size_t len, int minor) +{ + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = minor ? UFFD_FEATURE_MINOR_SHMEM : + 0 }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register r = { + .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING | + (minor ? UFFDIO_REGISTER_MODE_MINOR : 0) + }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, &r)); + return fd; +} +static void copy_leaf(int fd, uintptr_t dst, unsigned char value) +{ + unsigned char src[4096]; + memset(src, value, sizeof(src)); + struct uffdio_copy c = { .dst = dst, + .src = (uintptr_t)src, + .len = sizeof(src) }; + CHECK(!ioctl(fd, UFFDIO_COPY, &c) && c.copy == sizeof(src)); +} +static void cpu(unsigned id) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(id, &set); + CHECK(!sched_setaffinity(0, sizeof(set), &set)); +} +static int worker(int file, int sock, size_t native, size_t len) +{ + CHECK(getauxval(AT_PAGESZ) == 4096); + alarm(90); + cpu(1); + unsigned char *p = map(file, len, len, 0); + unsigned char *q = map(file, len - 4096, len, 4096); + CHECK(p[0] == 0x61 && q[0] == 0x61); + uint64_t old_head = + mapped_pfn(getpid(), (uintptr_t)p, 4096) * 4096 / native; + CHECK(entry("/proc/kpageflags", old_head) & + (1ULL << KPF_COMPOUND_HEAD)); + int fd = register_range(p, len, 1); + for (size_t i = 4 * native; i < len; i += native) + copy_leaf(fd, (uintptr_t)p + i, 0x71); + int pipes[2]; + CHECK(!pipe(pipes)); + struct iovec iov = { p, 4096 }; + CHECK(vmsplice(pipes[1], &iov, 1, 0) == 4096); + CHECK(!fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + native, native)); + CHECK(entry("/proc/kpageflags", old_head) & + (1ULL << KPF_COMPOUND_HEAD)); + CHECK(!close(pipes[0]) && !close(pipes[1])); + struct ready r = { (uintptr_t)p, old_head }; + struct iovec v = { &r, sizeof(r) }; + union { + struct cmsghdr align; + char bytes[CMSG_SPACE(sizeof(fd))]; + } control = { 0 }; + struct msghdr msg = { .msg_iov = &v, + .msg_iovlen = 1, + .msg_control = control.bytes, + .msg_controllen = sizeof(control.bytes) }; + struct cmsghdr *c = CMSG_FIRSTHDR(&msg); + c->cmsg_level = SOL_SOCKET; + c->cmsg_type = SCM_RIGHTS; + c->cmsg_len = CMSG_LEN(sizeof(fd)); + memcpy(CMSG_DATA(c), &fd, sizeof(fd)); + CHECK(sendmsg(sock, &msg, 0) == sizeof(r)); + struct command cmd; + while (recv(sock, &cmd, sizeof(cmd), 0) == sizeof(cmd)) { + CHECK(cmd.offset < len && cmd.len <= len - cmd.offset); + unsigned char value = 0; + if (!cmd.op) + break; + if (cmd.op == 1) + value = *(volatile unsigned char *)(p + cmd.offset); + else if (cmd.op == 2) + CHECK(!fallocate(file, + FALLOC_FL_PUNCH_HOLE | + FALLOC_FL_KEEP_SIZE, + cmd.offset, cmd.len)); + else if (cmd.op == 3) { + CHECK(cmd.offset >= 4096); + value = *(volatile unsigned char *)(q + cmd.offset - + 4096); + } else if (cmd.op == 4) { + CHECK(cmd.offset >= 4096); + memset(q + cmd.offset - 4096, cmd.value, cmd.len); + } else + CHECK(0); + CHECK(send(sock, &value, 1, 0) == 1); + } + CHECK(!munmap(q, len - 4096) && !munmap(p, len) && !close(fd) && + !close(file) && !close(sock)); + return 0; +} +static void send_command(int sock, uint64_t op, size_t offset, size_t len) +{ + struct command c = { .op = op, .offset = offset, .len = len }; + CHECK(send(sock, &c, sizeof(c), 0) == sizeof(c)); +} +static void reply(int sock, unsigned char expected) +{ + unsigned char value; + CHECK(recv(sock, &value, 1, 0) == 1 && value == expected); +} +static void fault(int sock, int fd, uintptr_t base, size_t offset, int minor, + unsigned char value) +{ + send_command(sock, 1, offset + 17, 0); + struct pollfd p = { .fd = fd, .events = POLLIN }; + struct uffd_msg m; + CHECK(poll(&p, 1, 10000) == 1 && (p.revents & POLLIN)); + CHECK(read(fd, &m, sizeof(m)) == sizeof(m)); + CHECK(m.event == UFFD_EVENT_PAGEFAULT && + m.arg.pagefault.address == base + offset && + m.arg.pagefault.flags == (minor ? UFFD_PAGEFAULT_FLAG_MINOR : 0)); + if (minor) { + struct uffdio_continue c = { .range = { base + offset, 4096 } }; + CHECK(!ioctl(fd, UFFDIO_CONTINUE, &c) && c.mapped == 4096); + } else + copy_leaf(fd, base + offset, value); + reply(sock, value); +} +static unsigned long pmd_mapped(unsigned char *p) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + CHECK(f); + char *line = NULL; + size_t cap = 0; + unsigned long a, b, kb, result = 0; + int selected = 0; + while (getline(&line, &cap, f) > 0) { + if (sscanf(line, "%lx-%lx", &a, &b) == 2) + selected = a == (uintptr_t)p; + else if (selected && + sscanf(line, "ShmemPmdMapped: %lu kB", &kb) == 1) + result = kb * 1024; + } + free(line); + fclose(f); + return result; +} +static int collapse(unsigned char *p, size_t len) +{ + int ret; + unsigned tries = 0; + do { + errno = 0; + ret = madvise(p, len, MADV_COLLAPSE); + if (!ret || errno != EAGAIN) + break; + usleep(20000); + } while (++tries < 10); + printf("# collapse ret=%d errno=%d PMD-mapped=%lu\n", ret, + ret ? errno : 0, pmd_mapped(p)); + return ret; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--worker")) + return worker(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10), + strtoul(argv[5], NULL, 10)); + if (getpid() != 1 || argc != 1) + return 2; + alarm(120); + cpu(0); + CHECK(!mount("proc", "/proc", "proc", 0, NULL) && + !mount("sysfs", "/sys", "sysfs", 0, NULL)); + CHECK(!mount("tmpfs", "/tmp", "tmpfs", 0, + "huge=within_size,size=256m")); + size_t native = getauxval(AT_PAGESZ), len = 0; + FILE *f = fopen("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", + "r"); + CHECK(f && fscanf(f, "%zu", &len) == 1); + fclose(f); + CHECK(native == 16384 && len == 32 * 1024 * 1024); + int file = open("/tmp/collapse", O_CREAT | O_RDWR | O_TRUNC, 0600); + CHECK(file >= 0 && !ftruncate(file, 4 * native)); + unsigned char *initial = map(file, 4 * native, 4 * native, 0); + memset(initial, 0x61, 4 * native); + CHECK(entry("/proc/kpageflags", + mapped_pfn(getpid(), (uintptr_t)initial, native)) & + (1ULL << KPF_COMPOUND_HEAD)); + CHECK(!munmap(initial, 4 * native) && !unlink("/tmp/collapse") && + !ftruncate(file, len)); + int sock[2]; + CHECK(!socketpair(AF_UNIX, SOCK_SEQPACKET, 0, sock)); + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char fd[24], so[24], ns[24], ls[24]; + close(sock[0]); + snprintf(fd, sizeof(fd), "%d", file); + snprintf(so, sizeof(so), "%d", sock[1]); + snprintf(ns, sizeof(ns), "%zu", native); + snprintf(ls, sizeof(ls), "%zu", len); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, 4096, 0UL, 0UL, 0UL)); + execl("/init", "/init", "--worker", fd, so, ns, ls, NULL); + _exit(3); + } + close(sock[1]); + struct ready ready; + struct iovec v = { &ready, sizeof(ready) }; + union { + struct cmsghdr align; + char bytes[CMSG_SPACE(sizeof(int))]; + } control = { 0 }; + struct msghdr msg = { .msg_iov = &v, + .msg_iovlen = 1, + .msg_control = control.bytes, + .msg_controllen = sizeof(control.bytes) }; + CHECK(recvmsg(sock[0], &msg, 0) == sizeof(ready) && + !(msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC))); + struct cmsghdr *c = CMSG_FIRSTHDR(&msg); + CHECK(c && c->cmsg_level == SOL_SOCKET && c->cmsg_type == SCM_RIGHTS); + int uffd; + memcpy(&uffd, CMSG_DATA(c), sizeof(uffd)); + unsigned char *p = map(file, len, len, 0); + int native_uffd = register_range(p, len, 0); + CHECK(collapse(p, len) == -1 && errno == EINVAL && pmd_mapped(p) == 0); + fault(sock[0], uffd, ready.base, 0, 1, 0x61); + fault(sock[0], uffd, ready.base, len - native, 1, 0x71); + uint64_t head = mapped_pfn(pid, ready.base, 4096) * 4096 / native; + CHECK(head != ready.head && + (entry("/proc/kpageflags", head) & (1ULL << KPF_COMPOUND_HEAD))); + CHECK(mapped_pfn(pid, ready.base + len - native, 4096) * 4096 / + native == + head + len / native - 1); + for (size_t i = 1; i < len / native; i++) + CHECK(entry("/proc/kpageflags", head + i) & + (1ULL << KPF_COMPOUND_TAIL)); + /* Collapse retained the absent native unit and independent fine holes. */ + fault(sock[0], uffd, ready.base, native, 0, 0x92); + fault(sock[0], uffd, ready.base, 5 * native + 4096, 0, 0xa3); + puts("ok - PMD cache collapse preserves missing/minor state and neighbour data"); + /* Every native leaf now contains data: coarse exposure may populate zero subunits. */ + CHECK(!collapse(p, len) && pmd_mapped(p) == len); + fault(sock[0], uffd, ready.base, 6 * native + 4096, 1, 0); + struct uffdio_copy duplicate = { .dst = ready.base + 6 * native + 8192, + .src = (uintptr_t)p, + .len = 4096 }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &duplicate) == -1 && errno == EEXIST); + puts("ok - native PMD promotion exposes coarse leaf zeros without overwriting fine data"); + /* A new fine hole must invalidate the covering native PMD mapping. */ + send_command(sock[0], 2, 7 * native + 4096, 4096); + reply(sock[0], 0); + CHECK(pmd_mapped(p) == 0); + fault(sock[0], uffd, ready.base, 7 * native + 4096, 0, 0xb4); + CHECK(p[7 * native] == 0x71 && p[7 * native + 4096] == 0xb4 && + p[7 * native + 8192] == 0); + CHECK(p[native] == 0x92 && p[5 * native + 4096] == 0xa3); + puts("ok - fine hole splits PMD alias and preserves populated neighbouring leaves"); + for (unsigned round = 0; round < 8; round++) { + CHECK(!collapse(p, len) && pmd_mapped(p) == len); + size_t off = (16 + round) * native + 4096; + /* Prime CPU0's PMD translation before CPU1 punches the fine hole. */ + CHECK(*(volatile unsigned char *)(p + off) == 0); + send_command(sock[0], 3, off + 17, 0); + reply(sock[0], 0); + send_command(sock[0], 2, off, 4096); + reply(sock[0], 0); + CHECK(pmd_mapped(p) == 0); + if (round % 3) { + /* Stale native or shifted fine translations would miss promotion. */ + if (round % 3 == 1) + memset(p + off, 0xc0 + round, 4096); + else { + struct command write = { 4, off, 4096, + 0xc0 + round }; + CHECK(send(sock[0], &write, sizeof(write), 0) == + sizeof(write)); + reply(sock[0], 0); + } + struct uffdio_copy collision = { .dst = ready.base + + off, + .src = (uintptr_t)p, + .len = 4096 }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &collision) == -1 && + errno == EEXIST); + fault(sock[0], uffd, ready.base, off, 1, 0xc0 + round); + } else + fault(sock[0], uffd, ready.base, off, 0, 0xc0 + round); + CHECK(p[off - 4096] == 0x71 && p[off] == 0xc0 + round && + p[off + 4096] == 0); + send_command(sock[0], 3, off + 17, 0); + reply(sock[0], 0xc0 + round); + } + CHECK(!collapse(p, len) && pmd_mapped(p) == len); + for (size_t i = 0; i < len; i++) { + size_t page = i / native, low = i % native; + unsigned char expected = + page < 4 ? + (page == 1 ? (low < 4096 ? 0x92 : 0) : 0x61) : + (low < 4096 ? 0x71 : 0); + if (low >= 4096 && low < 8192) { + if (page == 5) + expected = 0xa3; + if (page == 7) + expected = 0xb4; + if (page >= 16 && page < 24) + expected = 0xc0 + page - 16; + } + CHECK(p[i] == expected); + } + puts("ok - eight cross-CPU PMD cycles, native/shifted-fine writes and complete 32MiB bytes"); + send_command(sock[0], 0, 0, 0); + int status; + CHECK(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(!close(uffd) && !close(native_uffd) && !munmap(p, len) && + !close(file) && !close(sock[0])); + puts("SHMEM COLLAPSE CONTRACT PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-hole-init.c b/tools/testing/selftests/arm64/user4k/shmem-hole-init.c new file mode 100644 index 00000000000000..8831cc30a22c35 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-hole-init.c @@ -0,0 +1,358 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - shmem hole line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +struct access { + volatile unsigned char *p; + unsigned char value; +}; +static void *read_byte(void *opaque) +{ + struct access *a = opaque; + a->value = *a->p; + return NULL; +} +static uint64_t pagemap(pid_t pid, uintptr_t address, size_t granule) +{ + char name[80]; + uint64_t entry; + snprintf(name, sizeof(name), "/proc/%d/pagemap", pid); + int fd = open(name, O_RDONLY); + CHECK(fd >= 0 && pread(fd, &entry, 8, address / granule * 8) == 8); + CHECK(!close(fd)); + return entry; +} +static void fill(int fd, unsigned char *dst, unsigned char *src, size_t ps) +{ + struct uffdio_copy c = { .dst = (uintptr_t)dst, + .src = (uintptr_t)src, + .len = ps }; + CHECK(!ioctl(fd, UFFDIO_COPY, &c) && c.copy == (long long)ps); +} +static void resolve(int fd, unsigned char *p, size_t ps, int minor, + unsigned char value) +{ + struct access a = { .p = p + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &a)); + struct pollfd pollfd = { .fd = fd, .events = POLLIN }; + struct uffd_msg msg; + CHECK(poll(&pollfd, 1, 10000) == 1 && (pollfd.revents & POLLIN)); + CHECK(read(fd, &msg, sizeof(msg)) == sizeof(msg)); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)p && + msg.arg.pagefault.flags == + (minor ? UFFD_PAGEFAULT_FLAG_MINOR : 0)); + if (minor) { + struct uffdio_continue c = { .range = { (uintptr_t)p, ps } }; + CHECK(!ioctl(fd, UFFDIO_CONTINUE, &c) && + c.mapped == (long long)ps); + } else { + unsigned char *source = malloc(ps); + CHECK(source); + memset(source, value, ps); + fill(fd, p, source, ps); + free(source); + } + CHECK(!pthread_join(thread, NULL) && a.value == value); +} +struct command { + uint64_t op, offset, len, value; +}; +static int alias_worker(int file, int input, int output, size_t len, + size_t native) +{ + CHECK(getauxval(AT_PAGESZ) == native); + unsigned char *p = + mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, file, 0); + CHECK(p != MAP_FAILED); + /* Establish every coarse PTE before the fine process punches its hole. */ + for (size_t i = 0; i < len; i++) + CHECK(p[i] == 0x61); + uint64_t base = (uintptr_t)p; + CHECK(write(output, &base, sizeof(base)) == sizeof(base)); + struct command cmd; + while (read(input, &cmd, sizeof(cmd)) == sizeof(cmd)) { + CHECK(cmd.offset <= len && cmd.len <= len - cmd.offset); + if (!cmd.op) + break; + if (cmd.op == 2) + memset(p + cmd.offset, cmd.value, cmd.len); + for (size_t i = 0; i < cmd.len; i++) + CHECK(p[cmd.offset + i] == cmd.value); + CHECK(write(output, "y", 1) == 1); + } + CHECK(!munmap(p, len)); + return 0; +} +static void tell(int input, int output, uint64_t op, size_t offset, size_t len, + unsigned value) +{ + struct command cmd = { op, offset, len, value }; + char reply; + CHECK(write(input, &cmd, sizeof(cmd)) == sizeof(cmd)); + if (op) + CHECK(read(output, &reply, 1) == 1 && reply == 'y'); +} +static unsigned long swapouts(void) +{ + FILE *f = fopen("/proc/vmstat", "r"); + char key[80]; + unsigned long value, result = 0; + CHECK(f); + while (fscanf(f, "%79s %lu", key, &value) == 2) + if (!strcmp(key, "pswpout")) + result = value; + fclose(f); + return result; +} +static unsigned long mapping_swap(void *address) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + char *line = NULL; + size_t cap = 0; + unsigned long start, end, value, result = 0; + int selected = 0; + CHECK(f); + while (getline(&line, &cap, f) > 0) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) + selected = start == (uintptr_t)address; + else if (selected && sscanf(line, "Swap: %lu kB", &value) == 1) + result = value; + } + free(line); + fclose(f); + return result; +} +static void swapped_missing(size_t ps, size_t native) +{ + int file = memfd_create("partial-uffd-swap", 0); + CHECK(file >= 0 && !ftruncate(file, native)); + unsigned char *p = + mmap(NULL, native, PROT_READ | PROT_WRITE, MAP_SHARED, file, 0); + unsigned char *source = malloc(ps); + CHECK(p != MAP_FAILED && source); + memset(source, 0x71, ps); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MINOR_SHMEM }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, native }, + .mode = UFFDIO_REGISTER_MODE_MISSING | + UFFDIO_REGISTER_MODE_MINOR }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + fill(fd, p, source, ps); + unsigned long before = swapouts(), usage = 0, written; + unsigned tries; + for (tries = 0; tries < 100; tries++) { + CHECK(!madvise(p, native, MADV_PAGEOUT)); + usleep(20000); + usage = mapping_swap(p); + if (usage >= native / 1024 && swapouts() > before) + break; + } + written = swapouts() - before; + CHECK(usage >= native / 1024 && written > 0); + /* The selected cache index is swapped; its unfilled neighbour stays missing. */ + resolve(fd, p + ps, ps, 0, 0x92); + resolve(fd, p, ps, 1, 0x71); + for (size_t i = 0; i < ps; i++) + CHECK(p[i] == 0x71 && p[ps + i] == 0x92); + CHECK(mapping_swap(p) == 0); + printf("ok - shared uffd %zuK on %zuK swapped cache=%lukB writes=%lu attempts=%u missing COPY preserves neighbour\n", + ps / 1024, native / 1024, usage, written, tries + 1); + CHECK(!close(fd) && !munmap(p, native) && !close(file)); + free(source); +} +static int worker(size_t ps, size_t native) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(60); + if (getenv("SHMEM_HOLE_SWAP") && ps < native) + swapped_missing(ps, native); + size_t len = 4 * native; + int file = memfd_create("hole-contract", 0); + CHECK(file >= 0 && !ftruncate(file, len)); + unsigned char *source = malloc(len); + CHECK(source); + memset(source, 0x61, len); + CHECK(pwrite(file, source, len, 0) == (ssize_t)len); + unsigned char *p = + mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, file, 0); + unsigned char *q = mmap(NULL, len - ps, PROT_READ | PROT_WRITE, + MAP_SHARED, file, ps); + unsigned char *cow = + mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE, file, 0); + CHECK(p != MAP_FAILED && q != MAP_FAILED && cow != MAP_FAILED); + for (size_t i = 0; i < len; i += ps) + CHECK(p[i] == 0x61); + for (size_t i = 0; i < len - ps; i += ps) + CHECK(q[i] == 0x61); + memset(cow + ps, 0xe1, ps); + int commands[2], replies[2]; + CHECK(!pipe(commands) && !pipe(replies)); + pid_t alias = fork(); + CHECK(alias >= 0); + if (!alias) { + char f[24], in[24], out[24], size[24], ns[24]; + close(commands[1]); + close(replies[0]); + snprintf(f, sizeof(f), "%d", file); + snprintf(in, sizeof(in), "%d", commands[0]); + snprintf(out, sizeof(out), "%d", replies[1]); + snprintf(size, sizeof(size), "%zu", len); + snprintf(ns, sizeof(ns), "%zu", native); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, native, 0UL, 0UL, 0UL)); + execl("/init", "/init", "--alias", f, in, out, size, ns, NULL); + _exit(3); + } + close(commands[0]); + close(replies[1]); + uint64_t alias_base; + CHECK(read(replies[0], &alias_base, sizeof(alias_base)) == + sizeof(alias_base)); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MINOR_SHMEM }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING | + UFFDIO_REGISTER_MODE_MINOR }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + CHECK(pagemap(alias, alias_base + ps, native) & (1ULL << 63)); + CHECK(!fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, ps, + ps)); + CHECK(!(pagemap(getpid(), (uintptr_t)p + ps, ps) & (1ULL << 63))); + CHECK(!(pagemap(getpid(), (uintptr_t)q, ps) & (1ULL << 63))); + CHECK(!(pagemap(alias, alias_base + ps, native) & (1ULL << 63))); + /* Fine neighbour PTEs survive, while COW retains its independent bytes. */ + CHECK(pagemap(getpid(), (uintptr_t)p, ps) & (1ULL << 63)); + CHECK(pagemap(getpid(), (uintptr_t)p + 2 * ps, ps) & (1ULL << 63)); + CHECK(p[0] == 0x61 && p[2 * ps] == 0x61 && cow[ps] == 0xe1); + resolve(fd, p + ps, ps, 0, 0xa2); + CHECK(q[0] == 0xa2 && cow[ps] == 0xe1); + tell(commands[1], replies[0], 1, ps, ps, 0xa2); + tell(commands[1], replies[0], 1, 0, ps, 0x61); + tell(commands[1], replies[0], 1, 2 * ps, ps, 0x61); + /* Refaulting a coarse writable alias makes its covered units populated. */ + CHECK(!fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, ps, + ps)); + tell(commands[1], replies[0], 2, ps, ps, 0xb7); + struct uffdio_copy collision = { .dst = (uintptr_t)p + ps, + .src = (uintptr_t)source, + .len = ps }; + errno = 0; + CHECK(ioctl(fd, UFFDIO_COPY, &collision) == -1 && errno == EEXIST); + resolve(fd, p + ps, ps, 1, 0xb7); + printf("ok - shmem hole %zuK on %zuK precise PTEs, shifted alias, COW, coarse revoke/refault and COPY collision\n", + ps / 1024, native / 1024); + /* Stop the alias before resize, so it cannot intentionally promote holes. */ + tell(commands[1], replies[0], 0, 0, 0, 0); + close(commands[1]); + close(replies[0]); + int status; + CHECK(waitpid(alias, &status, 0) == alias && WIFEXITED(status) && + !WEXITSTATUS(status)); + size_t cut = 2 * ps + ps / 2; + CHECK(!ftruncate(file, cut) && !ftruncate(file, len)); + /* Partial final user leaf remains valid, with only truncated bytes zero. */ + if (!(pagemap(getpid(), (uintptr_t)p + 2 * ps, ps) & (1ULL << 63))) + resolve(fd, p + 2 * ps, ps, 1, 0x61); + CHECK(p[2 * ps] == 0x61 && p[cut - 1] == 0x61 && p[cut] == 0 && + p[3 * ps - 1] == 0); + resolve(fd, p + 3 * ps, ps, 0, 0xc3); + CHECK(q[2 * ps] == 0xc3); + CHECK(cow[ps] == 0xe1); + printf("ok - shmem resize %zuK on %zuK partial tail retained, shrink/regrow missing leaf refilled\n", + ps / 1024, native / 1024); + CHECK(!close(fd) && !munmap(p, len) && !munmap(q, len - ps) && + !munmap(cow, len) && !close(file)); + free(source); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 7 && !strcmp(argv[1], "--alias")) + return alias_worker(atoi(argv[2]), atoi(argv[3]), atoi(argv[4]), + strtoul(argv[5], NULL, 10), + strtoul(argv[6], NULL, 10)); + if (argc == 4 && !strcmp(argv[1], "--worker")) + return worker(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10)); + int swap_test = argc == 2 && !strcmp(argv[1], "--swap"); + if (getpid() != 1 || (argc != 1 && !swap_test)) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + if (swap_test) { + struct stat st; + CHECK(!mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)); + for (unsigned i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + CHECK(!stat("/dev/vda", &st) && S_ISBLK(st.st_mode) && + !swapon("/dev/vda", 0)); + CHECK(!setenv("SHMEM_HOLE_SWAP", "1", 1)); + } + unsigned long native = getauxval(AT_PAGESZ), + sizes[] = { native, 4096, 16384 }; + int failed = 0; + for (unsigned i = 0; i < (native == 65536 ? 3U : 2U); i++) { + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char ps[24], ns[24]; + snprintf(ps, sizeof(ps), "%lu", sizes[i]); + snprintf(ns, sizeof(ns), "%lu", native); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, + 0UL)); + execl("/init", "/init", "--worker", ps, ns, NULL); + _exit(3); + } + int status = 0; + int ok = waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status); + printf("%s - shmem hole ABI %luK status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + failed |= !ok; + } + if (swap_test) + CHECK(!swapoff("/dev/vda")); + printf("SHMEM HOLE CONTRACT %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-pin-init.c b/tools/testing/selftests/arm64/user4k/shmem-pin-init.c new file mode 100644 index 00000000000000..51deda0fde52cc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-pin-init.c @@ -0,0 +1,358 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - shmem pin line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +struct ring { + int fd; + struct io_uring_params p; + void *sq, *cq, *sqes; + size_t sqlen, cqlen, sqelen; +}; +static unsigned load(unsigned *p) +{ + return __atomic_load_n(p, __ATOMIC_ACQUIRE); +} +static void store(unsigned *p, unsigned v) +{ + __atomic_store_n(p, v, __ATOMIC_RELEASE); +} +static size_t align_size(size_t n) +{ + size_t ps = getauxval(AT_PAGESZ); + return (n + ps - 1) & ~(ps - 1); +} +static void destroy(struct ring *r) +{ + if (r->sq && r->sq != MAP_FAILED) + munmap(r->sq, r->sqlen); + if (r->cq && r->cq != MAP_FAILED && r->cq != r->sq) + munmap(r->cq, r->cqlen); + if (r->sqes && r->sqes != MAP_FAILED) + munmap(r->sqes, r->sqelen); + if (r->fd >= 0) + close(r->fd); + *r = (struct ring){ .fd = -1 }; +} +static bool setup(struct ring *r, unsigned entries, unsigned flags) +{ + *r = (struct ring){ .fd = -1 }; + r->p.flags = flags; + r->fd = syscall(__NR_io_uring_setup, entries, &r->p); + if (r->fd < 0) + return false; + r->sqlen = align_size(r->p.sq_off.array + + r->p.sq_entries * sizeof(unsigned)); + r->cqlen = align_size(r->p.cq_off.cqes + + r->p.cq_entries * sizeof(struct io_uring_cqe) * + (flags & IORING_SETUP_CQE32 ? 2 : 1)); + r->sqelen = align_size(r->p.sq_entries * sizeof(struct io_uring_sqe) * + (flags & IORING_SETUP_SQE128 ? 2 : 1)); + if (r->p.features & IORING_FEAT_SINGLE_MMAP) { + if (r->cqlen > r->sqlen) + r->sqlen = r->cqlen; + r->cqlen = r->sqlen; + } + r->sq = mmap(NULL, r->sqlen, PROT_READ | PROT_WRITE, MAP_SHARED, r->fd, + IORING_OFF_SQ_RING); + r->cq = (r->p.features & IORING_FEAT_SINGLE_MMAP) ? + r->sq : + mmap(NULL, r->cqlen, PROT_READ | PROT_WRITE, MAP_SHARED, + r->fd, IORING_OFF_CQ_RING); + r->sqes = mmap(NULL, r->sqelen, PROT_READ | PROT_WRITE, MAP_SHARED, + r->fd, IORING_OFF_SQES); + return r->sq != MAP_FAILED && r->cq != MAP_FAILED && + r->sqes != MAP_FAILED; +} +static bool request(struct ring *r, unsigned op, int fd, void *buf, + unsigned len, uint64_t off, bool async) +{ + unsigned *sqtail = r->sq + r->p.sq_off.tail, + *sqmask = r->sq + r->p.sq_off.ring_mask; + unsigned *cqhead = r->cq + r->p.cq_off.head, + *cqtail = r->cq + r->p.cq_off.tail, + *cqmask = r->cq + r->p.cq_off.ring_mask; + unsigned tail = load(sqtail), head = load(cqhead), + idx = tail & load(sqmask); + unsigned sqstride = sizeof(struct io_uring_sqe) * + (r->p.flags & IORING_SETUP_SQE128 ? 2 : 1); + unsigned cqstride = sizeof(struct io_uring_cqe) * + (r->p.flags & IORING_SETUP_CQE32 ? 2 : 1); + struct io_uring_sqe *sqe = r->sqes + idx * sqstride; + memset(sqe, 0, sqstride); + sqe->opcode = op; + sqe->fd = fd; + sqe->addr = (uintptr_t)buf; + sqe->len = len; + sqe->off = off; + sqe->user_data = (uint64_t)tail + 1234; + if (async) + sqe->flags = IOSQE_ASYNC; + ((unsigned *)(r->sq + r->p.sq_off.array))[idx] = idx; + store(sqtail, tail + 1); + int ret = syscall(__NR_io_uring_enter, r->fd, 1, 1, + IORING_ENTER_GETEVENTS, NULL, 0); + if (ret != 1 || load(cqtail) != head + 1) { + printf("enter=%d errno=%d\n", ret, errno); + return false; + } + struct io_uring_cqe *cqe = + r->cq + r->p.cq_off.cqes + (head & load(cqmask)) * cqstride; + unsigned expected = op == IORING_OP_NOP ? 0 : len; + if (op == IORING_OP_READV_FIXED || op == IORING_OP_WRITEV_FIXED) { + expected = 0; + for (unsigned i = 0; i < len; i++) + expected += ((struct iovec *)buf)[i].iov_len; + } + bool ok = cqe->user_data == (uint64_t)tail + 1234 && + cqe->res == (int)expected; + if (!ok) + printf("cqe res=%d expected=%u\n", cqe->res, expected); + store(cqhead, head + 1); + return ok; +} + +static void fragment_geometry(unsigned char *p, size_t ps, int input) +{ + struct ring r; + CHECK(setup(&r, 8, 0)); + struct iovec reg = { p + 123, 3 * ps - 246 }; + unsigned char *src = malloc(3 * ps), *expected = malloc(3 * ps); + CHECK(src && expected); + memset(expected, 0x61, 3 * ps); + memset(src, 0xb7, 3 * ps); + CHECK(pwrite(input, src, 3 * ps, 0) == (ssize_t)(3 * ps)); + CHECK(!syscall(SYS_io_uring_register, r.fd, IORING_REGISTER_BUFFERS, + ®, 1)); + CHECK(request(&r, IORING_OP_READ_FIXED, input, p + ps + 71, ps + 31, 17, + true)); + memset(expected + ps + 71, 0xb7, ps + 31); + CHECK(!memcmp(p, expected, 3 * ps)); + memset(src, 0xc8, 3 * ps); + CHECK(pwrite(input, src, 3 * ps, 0) == (ssize_t)(3 * ps)); + CHECK(request(&r, IORING_OP_WRITE_FIXED, input, p + ps + 71, ps + 31, 9, + true)); + CHECK(pread(input, src, ps + 31, 9) == (ssize_t)(ps + 31)); + for (size_t i = 0; i < ps + 31; i++) + CHECK(src[i] == 0xb7); + struct iovec vec[2] = { { p + 2 * ps + 17, ps - 400 }, + { p + ps + 91, ps - 600 } }; + memset(src, 0xd5, 3 * ps); + CHECK(pwrite(input, src, 3 * ps, 0) == (ssize_t)(3 * ps)); + CHECK(request(&r, IORING_OP_READV_FIXED, input, vec, 2, 13, true)); + for (unsigned i = 0; i < 2; i++) + memset(expected + ((unsigned char *)vec[i].iov_base - p), 0xd5, + vec[i].iov_len); + CHECK(!memcmp(p, expected, 3 * ps)); + memset(src, 0xc8, 3 * ps); + CHECK(pwrite(input, src, 3 * ps, 0) == (ssize_t)(3 * ps)); + CHECK(request(&r, IORING_OP_WRITEV_FIXED, input, vec, 2, 23, true)); + size_t total = vec[0].iov_len + vec[1].iov_len; + CHECK(pread(input, src, total, 23) == (ssize_t)total); + for (size_t i = 0; i < total; i++) + CHECK(src[i] == 0xd5); + CHECK(!syscall(SYS_io_uring_register, r.fd, IORING_UNREGISTER_BUFFERS, + NULL, 0)); + destroy(&r); + free(src); + free(expected); + memset(p, 0x61, 3 * ps); + printf("ok - %zuK fixed/vector IO uses user fragment geometry and preserves guards\n", + ps / 1024); +} + +struct punch_worker { + int file; + size_t off, ps; + pthread_barrier_t start; +}; +static void bind_cpu(unsigned cpu) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + CHECK(!sched_setaffinity(0, sizeof(set), &set)); +} +static void *punch_repeatedly(void *arg) +{ + struct punch_worker *w = arg; + bind_cpu(1); + pthread_barrier_wait(&w->start); + for (unsigned i = 0; i < 256; i++) { + CHECK(!fallocate(w->file, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + w->off, w->ps)); + sched_yield(); + } + return NULL; +} +static void pin_hole_concurrency(struct ring *r, int file, int input, + unsigned char *p, size_t off, size_t ps) +{ + struct punch_worker w = { .file = file, .off = off, .ps = ps }; + pthread_t thread; + CHECK(!pthread_barrier_init(&w.start, NULL, 2)); + bind_cpu(0); + CHECK(!pthread_create(&thread, NULL, punch_repeatedly, &w)); + pthread_barrier_wait(&w.start); + for (unsigned i = 0; i < 256; i++) + CHECK(request(r, IORING_OP_READ_FIXED, input, p + off, ps, 0, + true)); + CHECK(!pthread_join(thread, NULL) && + !pthread_barrier_destroy(&w.start)); + printf("ok - %zuK concurrent fixed writes and 256 hole punches on separate CPUs\n", + ps / 1024); +} + +static int target(size_t native, size_t ps, int resize) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(30); + size_t len = 4 * native, off = native + ps; + bool retained = ps < native; + int file = memfd_create("pin-cache", 0), + input = memfd_create("pin-input", 0); + CHECK(file >= 0 && input >= 0 && !ftruncate(file, len) && + !ftruncate(input, ps)); + unsigned char *src = malloc(len), *back = malloc(len); + CHECK(src && back); + memset(src, 0x61, len); + CHECK(pwrite(file, src, len, 0) == (ssize_t)len); + memset(src, 0xb7, ps); + CHECK(pwrite(input, src, ps, 0) == (ssize_t)ps); + unsigned char *p = + mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, file, 0); + CHECK(p != MAP_FAILED && !madvise(p, len, MADV_NOHUGEPAGE)); + CHECK(p[off] == 0x61); + fragment_geometry(p, ps, input); + memset(src, 0xb7, ps); + CHECK(pwrite(input, src, ps, 0) == (ssize_t)ps); + struct ring r; + CHECK(setup(&r, 8, 0)); + struct iovec reg = { p + off, ps }; + CHECK(!syscall(SYS_io_uring_register, r.fd, IORING_REGISTER_BUFFERS, + ®, 1)); + int uffd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(uffd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(uffd, UFFDIO_API, &api)); + struct uffdio_register ur = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(uffd, UFFDIO_REGISTER, &ur)); + if (resize) + CHECK(!ftruncate(file, off) && !ftruncate(file, len)); + else + CHECK(!fallocate(file, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + off, ps)); + if (retained && !resize) + pin_hole_concurrency(&r, file, input, p, off, ps); + /* Fixed IO accesses its original FOLL_PIN reference after PTE removal. */ + CHECK(request(&r, IORING_OP_READ_FIXED, input, p + off, ps, 0, true)); + /* Also prove the original pinned object contains the completed write. */ + CHECK(request(&r, IORING_OP_WRITE_FIXED, input, p + off, ps, 0, true)); + CHECK(pread(input, back, ps, 0) == (ssize_t)ps); + for (size_t i = 0; i < ps; i++) + CHECK(back[i] == 0xb7); + CHECK(!syscall(SYS_io_uring_register, r.fd, IORING_UNREGISTER_BUFFERS, + NULL, 0)); + memset(src, 0xc8, ps); + struct uffdio_copy c = { .dst = (uintptr_t)(p + off), + .src = (uintptr_t)src, + .len = ps }; + errno = 0; + int ret = ioctl(uffd, UFFDIO_COPY, &c); + printf("# pin %zuK/%zuK resize=%d retained=%d COPY=%d errno=%d bytes=%lld\n", + ps / 1024, native / 1024, resize, retained, ret, errno, + (long long)c.copy); + if (retained) + CHECK(ret == -1 && errno == EEXIST && c.copy == -EEXIST); + else + CHECK(!ret && c.copy == (long long)ps); + CHECK(pread(file, back, len, 0) == (ssize_t)len); + for (size_t i = 0; i < len; i++) { + unsigned char expected = i >= off && i < off + ps ? + (retained ? 0xb7 : 0xc8) : + resize && i >= off ? 0 : + 0x61; + CHECK(back[i] == expected); + } + for (size_t i = 0; i < ps; i++) + CHECK(p[off + i] == (retained ? 0xb7 : 0xc8)); + /* A new punch after unpin can once again be populated by the pager. */ + CHECK(!fallocate(file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, off, + ps)); + c.copy = 0; + CHECK(!ioctl(uffd, UFFDIO_COPY, &c) && c.copy == (long long)ps); + for (size_t i = 0; i < ps; i++) + CHECK(p[off + i] == 0xc8); + destroy(&r); + CHECK(!close(uffd) && !munmap(p, len) && !close(file) && !close(input)); + free(src); + free(back); + printf("ok - writable pin %zuK on %zuK resize=%d retention and detach\n", + ps / 1024, native / 1024, resize); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--target")) + return target(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), atoi(argv[4])); + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + size_t native = getauxval(AT_PAGESZ); + for (size_t ps = 4096; ps <= native; ps *= 4) + for (int resize = 0; resize < 2; resize++) { + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char n[24], g[24], r[12]; + snprintf(n, sizeof(n), "%zu", native); + snprintf(g, sizeof(g), "%zu", ps); + snprintf(r, sizeof(r), "%d", resize); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, ps, 0, 0, 0)); + execl("/init", "init", "--target", n, g, r, + NULL); + _exit(127); + } + int status; + CHECK(waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status)); + } + puts("ok - shmem writable pin fixture complete"); + reboot(RB_POWER_OFF); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-pressure-init.c b/tools/testing/selftests/arm64/user4k/shmem-pressure-init.c new file mode 100644 index 00000000000000..f321878396f061 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-pressure-init.c @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CG "/sys/fs/cgroup/pressure" +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - shmem pressure line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +static void put(const char *path, const char *value) +{ + int fd = open(path, O_WRONLY); + CHECK(fd >= 0); + CHECK(write(fd, value, strlen(value)) == (ssize_t)strlen(value) && + !close(fd)); +} +static unsigned long long get(const char *path, const char *key) +{ + FILE *f = fopen(path, "r"); + CHECK(f); + char name[128]; + unsigned long long value, result = 0; + if (!key) + CHECK(fscanf(f, "%llu", &result) == 1); + else + while (fscanf(f, "%127s %llu", name, &value) == 2) + if (!strcmp(key, name)) { + result = value; + break; + } + fclose(f); + return result; +} +static void copy_leaf(int uffd, unsigned char *dst, unsigned char *src, + size_t ps, unsigned char value) +{ + memset(src, value, ps); + struct uffdio_copy c = { .dst = (uintptr_t)dst, + .src = (uintptr_t)src, + .len = ps }; + int ret = ioctl(uffd, UFFDIO_COPY, &c); + if (ret) + printf("# COPY destination=%p errno=%d transferred=%lld\n", dst, + errno, (long long)c.copy); + CHECK(!ret && c.copy == (long long)ps); +} +static void checkpoint(int out, int in) +{ + char token; + CHECK(write(out, "R", 1) == 1 && read(in, &token, 1) == 1 && + token == 'G'); +} +static int worker(size_t native, size_t ps, int out, int in) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(120); + size_t records = (16UL << 20) / native, + stride = ps < native ? native : 2 * native; + size_t len = records * stride; + int file = memfd_create("pressure-partial-cache", 0); + CHECK(file >= 0 && !ftruncate(file, len)); + unsigned char *p = + mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, file, 0); + unsigned char *source = malloc(stride), *back = malloc(stride); + CHECK(p != MAP_FAILED && source && back); + CHECK(!madvise(p, len, MADV_NOHUGEPAGE)); + int uffd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(uffd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(uffd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(uffd, UFFDIO_REGISTER, ®)); + for (size_t i = 0; i < records; i++) + copy_leaf(uffd, p + i * stride, source, ps, 0xa1); + checkpoint(out, in); + /* Revisit in permuted order, filling holes in reclaimed cache backing. */ + for (size_t i = 0; i < records; i++) { + size_t slot = (i * 4051) & (records - 1); + copy_leaf(uffd, p + slot * stride + ps, source, ps, 0xb2); + for (size_t j = 0; j < ps; j++) + CHECK(p[slot * stride + j] == 0xa1 && + p[slot * stride + ps + j] == 0xb2); + } + checkpoint(out, in); + for (size_t i = 0; i < records; i += 2) { + CHECK(!fallocate(file, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + i * stride, ps)); + copy_leaf(uffd, p + i * stride, source, ps, 0xc3); + } + checkpoint(out, in); + for (size_t i = 0; i < records; i++) { + size_t slot = (i * 4051 + 31) & (records - 1); + CHECK(pread(file, back, stride, slot * stride) == + (ssize_t)stride); + for (size_t j = 0; j < stride; j++) { + unsigned char expected = + j < ps ? (slot & 1 ? 0xa1 : 0xc3) : + j < 2 * ps ? 0xb2 : + 0; + CHECK(back[j] == expected); + } + struct uffdio_copy duplicate = { + .dst = (uintptr_t)(p + slot * stride), + .src = (uintptr_t)source, + .len = ps + }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &duplicate) == -1 && + errno == EEXIST && duplicate.copy == -EEXIST); + } + checkpoint(out, in); + CHECK(!close(uffd) && !munmap(p, len) && !close(file)); + free(source); + free(back); + printf("ok - shmem pressure %zuK on %zuK partial-cache refill, holes, duplicate rejection and every byte\n", + ps / 1024, native / 1024); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 6 && !strcmp(argv[1], "--worker")) + return worker(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), atoi(argv[4]), + atoi(argv[5])); + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL) && + !mount("sysfs", "/sys", "sysfs", 0, NULL)); + CHECK(!mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) && + !mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL)); + put("/sys/fs/cgroup/cgroup.subtree_control", "+memory"); + CHECK(!mkdir(CG, 0755)); + put(CG "/memory.max", "8388608"); + put(CG "/memory.swap.max", "134217728"); + struct stat st; + for (unsigned i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + CHECK(!swapon("/dev/vda", 0)); + size_t native = getauxval(AT_PAGESZ); + for (size_t ps = 4096; ps <= native; ps *= 4) { + int ready[2], go[2]; + CHECK(!pipe(ready) && !pipe(go)); + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + close(ready[0]); + close(go[1]); + char n[24], p[24], r[24], g[24]; + snprintf(n, sizeof(n), "%zu", native); + snprintf(p, sizeof(p), "%zu", ps); + snprintf(r, sizeof(r), "%d", ready[1]); + snprintf(g, sizeof(g), "%d", go[0]); + put(CG "/cgroup.procs", "0"); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, ps, 0, 0, 0)); + execl("/init", "init", "--worker", n, p, r, g, NULL); + _exit(127); + } + close(ready[1]); + close(go[0]); + for (unsigned phase = 0; phase < 4; phase++) { + char token; + CHECK(read(ready[0], &token, 1) == 1 && token == 'R'); + unsigned long long swap = get(CG "/memory.swap.current", + NULL), + max = get(CG "/memory.events", + "max"); + unsigned long long scan = get(CG "/memory.stat", + "pgscan"), + steal = get(CG "/memory.stat", + "pgsteal"); + printf("# pressure %zuK phase=%u resident=%llu swap=%llu max=%llu scan=%llu steal=%llu\n", + ps / 1024, phase, + get(CG "/memory.current", NULL), swap, max, scan, + steal); + CHECK(swap && max && scan && steal && + !get(CG "/memory.events", "oom") && + !get(CG "/memory.events", "oom_kill")); + CHECK(write(go[1], "G", 1) == 1); + } + int status; + CHECK(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status)); + close(ready[0]); + close(go[1]); + /* Closed/unmapped file identity must release all of its cache/swap. */ + for (unsigned i = 0; + i < 3000 && (get(CG "/memory.swap.current", NULL) || + get(CG "/memory.stat", "shmem")); + i++) + usleep(20000); + CHECK(!get(CG "/memory.swap.current", NULL) && + !get(CG "/memory.stat", "shmem")); + } + CHECK(!swapoff("/dev/vda")); + puts("SHMEM PRESSURE CONTRACT PASS"); + sync(); + reboot(RB_POWER_OFF); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-race-init.c b/tools/testing/selftests/arm64/user4k/shmem-race-init.c new file mode 100644 index 00000000000000..5109654b6ec216 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-race-init.c @@ -0,0 +1,308 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define ROUNDS 512 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - shmem race line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +struct control { + unsigned start, stop, ready; + unsigned long aliases; +}; +struct state { + struct control *control; + int file, uffd, resize; + size_t ps, native, len; + unsigned char *p, *q, *source; + unsigned copied, existed, eof, alloc_eof, retry; +}; +static unsigned load(unsigned *p) +{ + return __atomic_load_n(p, __ATOMIC_ACQUIRE); +} +static void store(unsigned *p, unsigned value) +{ + __atomic_store_n(p, value, __ATOMIC_RELEASE); +} +static void bind_cpu(unsigned cpu) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + CHECK(!sched_setaffinity(0, sizeof(set), &set)); +} +static void start_wait(struct control *c) +{ + __atomic_add_fetch(&c->ready, 1, __ATOMIC_RELEASE); + while (!load(&c->start)) + sched_yield(); +} +static void *mutate(void *arg) +{ + struct state *s = arg; + bind_cpu(0); + start_wait(s->control); + for (unsigned i = 0; i < ROUNDS; i++) { + if (s->resize) + CHECK(!ftruncate(s->file, s->ps) && + !ftruncate(s->file, s->len)); + else + CHECK(!fallocate(s->file, + FALLOC_FL_PUNCH_HOLE | + FALLOC_FL_KEEP_SIZE, + s->ps, s->ps)); + sched_yield(); + } + return NULL; +} +static void *populate(void *arg) +{ + struct state *s = arg; + bind_cpu(1); + start_wait(s->control); + for (unsigned i = 0; i < ROUNDS; i++) { + struct uffdio_copy c = { .dst = (uintptr_t)(s->p + s->ps), + .src = (uintptr_t)s->source, + .len = s->ps }; + errno = 0; + int ret = ioctl(s->uffd, UFFDIO_COPY, &c); + if (!ret) { + CHECK(c.copy == (long long)s->ps); + s->copied++; + } else { + CHECK(ret == -1 && c.copy == -errno); + if (errno == EEXIST) + s->existed++; + else if (errno == EFAULT && s->resize) + s->eof++; + /* Native shmem allocator preserves its NULL/ENOMEM EOF ABI. */ + else if (errno == ENOMEM && s->resize && + s->ps == s->native) + s->alloc_eof++; + else if (errno == EAGAIN) + s->retry++; + else { + printf("# unexpected COPY error=%d bytes=%lld\n", + errno, (long long)c.copy); + CHECK(0); + } + } + sched_yield(); + } + return NULL; +} +static int valid_byte(unsigned char value) +{ + return value == 0 || value == 0x61 || value == 0xa2; +} +static void *fine_alias(void *arg) +{ + struct state *s = arg; + bind_cpu(0); + start_wait(s->control); + for (unsigned i = 0; i < ROUNDS; i++) { + if (!s->resize) { + CHECK(!madvise(s->q, s->ps, MADV_DONTNEED)); + CHECK(valid_byte(*(volatile unsigned char *)s->q)); + } else { + /* The retained prefix remains inside EOF throughout resize. */ + CHECK(*(volatile unsigned char *)s->p == 0x61); + } + sched_yield(); + } + return NULL; +} +static int native_alias(int file, int control, size_t native, size_t ps, + int resize) +{ + CHECK(getauxval(AT_PAGESZ) == native); + bind_cpu(1); + alarm(60); + struct control *c = mmap(NULL, native, PROT_READ | PROT_WRITE, + MAP_SHARED, control, 0); + unsigned char *p = mmap(NULL, 4 * native, PROT_READ | PROT_WRITE, + MAP_SHARED, file, 0); + CHECK(c != MAP_FAILED && p != MAP_FAILED && + !madvise(p, 4 * native, MADV_NOHUGEPAGE)); + start_wait(c); + unsigned long cycles = 0; + while (!load(&c->stop)) { + CHECK(!madvise(p, 2 * native, MADV_DONTNEED)); + CHECK(*(volatile unsigned char *)p == 0x61); + if (!resize) + CHECK(valid_byte(*(volatile unsigned char *)(p + ps))); + cycles++; + sched_yield(); + } + c->aliases = cycles; + CHECK(!munmap(p, 4 * native) && !munmap(c, native)); + return 0; +} +static int target(size_t native, size_t ps, int resize) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(60); + struct state s = { + .native = native, .ps = ps, .len = 4 * native, .resize = resize + }; + s.file = memfd_create("race-cache", 0); + int control = memfd_create("race-control", 0); + CHECK(s.file >= 0 && control >= 0 && !ftruncate(s.file, s.len) && + !ftruncate(control, native)); + s.control = mmap(NULL, native, PROT_READ | PROT_WRITE, MAP_SHARED, + control, 0); + s.source = malloc(s.len); + CHECK(s.control != MAP_FAILED && s.source); + memset(s.source, 0x61, s.len); + CHECK(pwrite(s.file, s.source, s.len, 0) == (ssize_t)s.len); + s.p = mmap(NULL, s.len, PROT_READ | PROT_WRITE, MAP_SHARED, s.file, 0); + s.q = mmap(NULL, s.len - ps, PROT_READ | PROT_WRITE, MAP_SHARED, s.file, + ps); + CHECK(s.p != MAP_FAILED && s.q != MAP_FAILED && + !madvise(s.p, s.len, MADV_NOHUGEPAGE)); + CHECK(s.p[0] == + 0x61); /* Avoid blocking the resize prefix reader on UFFD. */ + unsigned char *cow = MAP_FAILED; + if (!resize) { + cow = mmap(NULL, s.len, PROT_READ | PROT_WRITE, MAP_PRIVATE, + s.file, 0); + CHECK(cow != MAP_FAILED); + memset(cow + ps, 0xe1, ps); + } + s.uffd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(s.uffd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(s.uffd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)s.p, s.len }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(s.uffd, UFFDIO_REGISTER, ®)); + memset(s.source, 0xa2, ps); + pid_t alias = fork(); + CHECK(alias >= 0); + if (!alias) { + char f[24], c[24], n[24], p[24], r[12]; + snprintf(f, sizeof(f), "%d", s.file); + snprintf(c, sizeof(c), "%d", control); + snprintf(n, sizeof(n), "%zu", native); + snprintf(p, sizeof(p), "%zu", ps); + snprintf(r, sizeof(r), "%d", resize); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, native, 0, 0, 0)); + execl("/init", "init", "--alias", f, c, n, p, r, NULL); + _exit(127); + } + pthread_t mutation, copy, fine; + CHECK(!pthread_create(&mutation, NULL, mutate, &s) && + !pthread_create(©, NULL, populate, &s) && + !pthread_create(&fine, NULL, fine_alias, &s)); + while (load(&s.control->ready) != 4) + sched_yield(); + store(&s.control->start, 1); + CHECK(!pthread_join(mutation, NULL) && !pthread_join(copy, NULL) && + !pthread_join(fine, NULL)); + store(&s.control->stop, 1); + int status; + CHECK(waitpid(alias, &status, 0) == alias && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(s.control->aliases && + s.copied + s.existed + s.eof + s.alloc_eof + s.retry == ROUNDS); + printf("# race %zuK/%zuK resize=%d COPY=%u EEXIST=%u EOF=%u native-alloc-EOF=%u retry=%u native-refaults=%lu\n", + ps / 1024, native / 1024, resize, s.copied, s.existed, s.eof, + s.alloc_eof, s.retry, s.control->aliases); + /* Once concurrent actors stop, exact cache/PTE state must be recoverable. */ + CHECK(!ftruncate(s.file, s.len)); + CHECK(!fallocate(s.file, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, ps, + ps)); + memset(s.source, 0xd4, ps); + struct uffdio_copy fill = { .dst = (uintptr_t)(s.p + ps), + .src = (uintptr_t)s.source, + .len = ps }; + CHECK(!ioctl(s.uffd, UFFDIO_COPY, &fill) && fill.copy == (long long)ps); + unsigned char *back = malloc(s.len); + CHECK(back && pread(s.file, back, s.len, 0) == (ssize_t)s.len); + for (size_t i = 0; i < s.len; i++) { + unsigned char expected = i >= ps && i < 2 * ps ? 0xd4 : + resize && i >= ps ? 0 : + 0x61; + if (back[i] != expected) { + printf("# byte=%zu actual=%#x expected=%#x\n", i, + back[i], expected); + CHECK(0); + } + } + for (size_t i = 0; i < ps; i++) { + CHECK(s.p[ps + i] == 0xd4 && s.q[i] == 0xd4); + if (!resize) + CHECK(cow[ps + i] == 0xe1); + } + CHECK(!close(s.uffd) && !munmap(s.p, s.len) && + !munmap(s.q, s.len - ps)); + if (!resize) + CHECK(!munmap(cow, s.len)); + CHECK(!munmap(s.control, native) && !close(control) && !close(s.file)); + free(back); + free(s.source); + printf("ok - concurrent shmem %zuK on %zuK resize=%d cache bytes, aliases and private COW\n", + ps / 1024, native / 1024, resize); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 7 && !strcmp(argv[1], "--alias")) + return native_alias(atoi(argv[2]), atoi(argv[3]), + strtoul(argv[4], NULL, 10), + strtoul(argv[5], NULL, 10), atoi(argv[6])); + if (argc == 5 && !strcmp(argv[1], "--target")) + return target(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), atoi(argv[4])); + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + size_t native = getauxval(AT_PAGESZ); + for (size_t ps = 4096; ps <= native; ps *= 4) + for (int resize = 0; resize < 2; resize++) { + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char n[24], p[24], r[12]; + snprintf(n, sizeof(n), "%zu", native); + snprintf(p, sizeof(p), "%zu", ps); + snprintf(r, sizeof(r), "%d", resize); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, ps, 0, 0, 0)); + execl("/init", "init", "--target", n, p, r, + NULL); + _exit(127); + } + int status; + CHECK(waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status)); + } + puts("SHMEM RACE CONTRACT PASS"); + reboot(RB_POWER_OFF); + return 0; +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-thp-init.c b/tools/testing/selftests/arm64/user4k/shmem-thp-init.c new file mode 100644 index 00000000000000..e08ea92103fcc2 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-thp-init.c @@ -0,0 +1,271 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - shmem THP line=%d errno=%d\n", \ + __LINE__, errno); \ + exit(1); \ + } \ + } while (0) +static uint64_t read_entry(const char *path, uint64_t index) +{ + uint64_t value; + int fd = open(path, O_RDONLY); + CHECK(fd >= 0 && pread(fd, &value, 8, index * 8) == 8 && !close(fd)); + return value; +} +static uint64_t pfn(void *p, size_t ps) +{ + uint64_t entry = read_entry("/proc/self/pagemap", (uintptr_t)p / ps); + CHECK(entry & (1ULL << 63)); + return entry & ((1ULL << 55) - 1); +} +static unsigned char *map_aligned(int fd, size_t len) +{ + void *reserve = mmap(NULL, 2 * len, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + uintptr_t address = ((uintptr_t)reserve + len - 1) & ~(len - 1); + CHECK(!munmap(reserve, 2 * len)); + unsigned char *p = mmap((void *)address, len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED_NOREPLACE, fd, 0); + CHECK(p != MAP_FAILED); + return p; +} +struct reader { + volatile unsigned char *p; + unsigned char value; +}; +static void *read_byte(void *arg) +{ + struct reader *r = arg; + r->value = *r->p; + return NULL; +} +static int target(int file, size_t native, size_t ps, int hold, int zero, + int resize, uint64_t head) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(30); + size_t len = 4 * native, target_off = resize ? 2 * native : native; + size_t cut = native + ps / 2; + unsigned char *p = map_aligned(file, len); + for (size_t i = 0; i < len; i += ps) + CHECK(p[i] == 0x61); + CHECK(pfn(p, ps) * ps / native == head); + int pipes[2]; + CHECK(!pipe(pipes)); + if (hold) { + struct iovec iov = { p, ps }; + CHECK(vmsplice(pipes[1], &iov, 1, 0) == (ssize_t)ps); + } + int uffd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(uffd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(uffd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(uffd, UFFDIO_REGISTER, ®)); + if (resize) + CHECK(!ftruncate(file, cut) && !ftruncate(file, len)); + else + CHECK(!fallocate(file, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + native, ps)); + uint64_t flags = read_entry("/proc/kpageflags", head); + printf("# THP %zuK native=%zuK hold=%d zero=%d resize=%d head_flags=%#llx\n", + ps / 1024, native / 1024, hold, zero, resize, + (unsigned long long)flags); + if (hold) + CHECK((flags & + ((1ULL << KPF_COMPOUND_HEAD) | (1ULL << KPF_THP))) == + ((1ULL << KPF_COMPOUND_HEAD) | (1ULL << KPF_THP))); + struct reader r = { .p = p + target_off + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct pollfd pollfd = { .fd = uffd, .events = POLLIN }; + struct uffd_msg msg; + CHECK(poll(&pollfd, 1, 10000) == 1 && (pollfd.revents & POLLIN)); + CHECK(read(uffd, &msg, sizeof(msg)) == sizeof(msg)); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && !msg.arg.pagefault.flags && + msg.arg.pagefault.address == (uintptr_t)(p + target_off)); + unsigned char *source = malloc(ps); + CHECK(source); + memset(source, 0x92, ps); + int ret; + struct uffdio_copy invalid = { .dst = (uintptr_t)(p + target_off), + .src = 1, + .len = ps }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &invalid) == -1 && errno == EFAULT && + invalid.copy == -EFAULT); + if (zero) { + struct uffdio_zeropage z = { + .range = { (uintptr_t)(p + target_off), ps } + }; + ret = ioctl(uffd, UFFDIO_ZEROPAGE, &z); + printf("# ZEROPAGE returned=%d errno=%d bytes=%lld\n", ret, + errno, (long long)z.zeropage); + CHECK(!ret && z.zeropage == (long long)ps); + } else { + struct uffdio_copy c = { .dst = (uintptr_t)(p + target_off), + .src = (uintptr_t)source, + .len = ps }; + ret = ioctl(uffd, UFFDIO_COPY, &c); + printf("# COPY returned=%d errno=%d bytes=%lld\n", ret, errno, + (long long)c.copy); + CHECK(!ret && c.copy == (long long)ps); + } + CHECK(!pthread_join(thread, NULL) && r.value == (zero ? 0 : 0x92)); + memset(source, 0xd4, ps); + struct uffdio_copy duplicate = { .dst = (uintptr_t)(p + target_off), + .src = (uintptr_t)source, + .len = ps }; + errno = 0; + CHECK(ioctl(uffd, UFFDIO_COPY, &duplicate) == -1 && errno == EEXIST && + duplicate.copy == -EEXIST); + /* File reads verify zeroed tail holes without faulting unfilled UFFD leaves. */ + unsigned char *back = malloc(len); + CHECK(back); + CHECK(pread(file, back, len, 0) == (ssize_t)len); + for (size_t i = 0; i < len; i++) { + unsigned char expected = i >= target_off && + i < target_off + ps ? + (zero ? 0 : 0x92) : + resize && i >= cut ? 0 : + 0x61; + CHECK(back[i] == expected); + if (!resize || (i >= target_off && i < target_off + ps)) + CHECK(p[i] == expected); + } + free(back); + if (hold) { + CHECK(read(pipes[0], source, ps) == (ssize_t)ps); + for (size_t i = 0; i < ps; i++) + CHECK(source[i] == 0x61); + } + CHECK(!close(pipes[0]) && !close(pipes[1]) && !close(uffd) && + !munmap(p, len) && !close(file)); + free(source); + printf("ok - THP %zuK on %zuK hold=%d zero=%d resize=%d missing refill and all neighbour bytes\n", + ps / 1024, native / 1024, hold, zero, resize); + return 0; +} +static int create_file(size_t native, uint64_t *head) +{ + int fd = open("/tmp/huge", O_CREAT | O_TRUNC | O_RDWR, 0600); + CHECK(fd >= 0); + CHECK(!ftruncate(fd, 4 * native)); + unsigned char *p = map_aligned(fd, 4 * native); + CHECK(!madvise(p, 4 * native, MADV_HUGEPAGE)); + memset(p, 0x61, 4 * native); + *head = pfn(p, native); + uint64_t f = read_entry("/proc/kpageflags", *head); + printf("# initial head=%llu flags=%#llx VA=%p native=%zu\n", + (unsigned long long)*head, (unsigned long long)f, p, native); + CHECK((f & ((1ULL << KPF_COMPOUND_HEAD) | (1ULL << KPF_THP))) == + ((1ULL << KPF_COMPOUND_HEAD) | (1ULL << KPF_THP))); + for (unsigned i = 1; i < 4; i++) { + CHECK(pfn(p + i * native, native) == *head + i); + CHECK(read_entry("/proc/kpageflags", *head + i) & + (1ULL << KPF_COMPOUND_TAIL)); + } + /* Keep the name until allocation: unlinked mappings use anon-shmem policy. */ + CHECK(!unlink("/tmp/huge") && !munmap(p, 4 * native)); + return fd; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 9 && !strcmp(argv[1], "--target")) + return target(atoi(argv[2]), strtoul(argv[3], NULL, 10), + strtoul(argv[4], NULL, 10), atoi(argv[5]), + atoi(argv[6]), atoi(argv[7]), + strtoull(argv[8], NULL, 10)); + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + CHECK(!mount("tmpfs", "/tmp", "tmpfs", 0, "huge=within_size,size=32m")); + size_t native = getauxval(AT_PAGESZ), sizes[] = { native, 4096, 16384 }; + int failed = 0; + for (unsigned abi = 0; abi < (native == 65536 ? 3U : 2U); abi++) + for (unsigned hold = 0; hold < 2; hold++) + for (unsigned zero = 0; zero < 2; zero++) + for (unsigned resize = 0; resize < 2; + resize++) { + uint64_t head; + int fd = create_file(native, &head); + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char fs[24], ns[24], ps[24], + hs[24], zs[24], rs[24], + pfs[24]; + snprintf(fs, sizeof(fs), "%d", + fd); + snprintf(ns, sizeof(ns), "%zu", + native); + snprintf(ps, sizeof(ps), "%zu", + sizes[abi]); + snprintf(hs, sizeof(hs), "%u", + hold); + snprintf(zs, sizeof(zs), "%u", + zero); + snprintf(rs, sizeof(rs), "%u", + resize); + snprintf(pfs, sizeof(pfs), + "%llu", + (unsigned long long) + head); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, + sizes[abi], 0UL, + 0UL, 0UL)); + execl("/init", "/init", + "--target", fs, ns, ps, + hs, zs, rs, pfs, NULL); + _exit(3); + } + int status = 0; + int ok = waitpid(pid, &status, 0) == + pid && + WIFEXITED(status) && + !WEXITSTATUS(status); + printf("%s - THP ABI %zuK hold=%u zero=%u resize=%u status=%#x\n", + ok ? "ok" : "not ok", + sizes[abi] / 1024, hold, zero, + resize, status); + failed |= !ok; + CHECK(!close(fd)); + } + printf("SHMEM THP CONTRACT %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/shmem-uffd-swap-init.c b/tools/testing/selftests/arm64/user4k/shmem-uffd-swap-init.c new file mode 100644 index 00000000000000..73ff33d5e0ba13 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/shmem-uffd-swap-init.c @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * PID-1 runner for a disposable VM with a dedicated virtio swap disk. + * Use kunit.autorun=0: tests skipped at boot stay skipped on a debugfs rerun. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int run_swap_suite(void) +{ + const char *base = "/sys/kernel/debug/kunit/arm64-user4k-shmem-swap/"; + char path[128], *log; + size_t used = 0, capacity = 1024 * 1024; + ssize_t n; + int fd, ret = -1; + + snprintf(path, sizeof(path), "%srun", base); + fd = open(path, O_WRONLY | O_CLOEXEC); + if (fd < 0) + return -1; + n = write(fd, "1", 1); + close(fd); + if (n != 1) + return -1; + snprintf(path, sizeof(path), "%sresults", base); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return -1; + log = calloc(1, capacity); + if (!log) { + close(fd); + return -1; + } + while (used < capacity - 1 && + (n = read(fd, log + used, capacity - 1 - used)) > 0) + used += n; + close(fd); + fputs(log, stdout); + if (n == 0 && used < capacity - 1 && + strstr(log, "\nok 1 arm64-user4k-shmem-swap\n") && + !strstr(log, "not ok") && !strstr(log, "# SKIP")) + ret = 0; + free(log); + return ret; +} + +static int fault_setting(const char *name, const char *value) +{ + char path[160]; + snprintf(path, sizeof(path), "/sys/kernel/debug/failslab/%s", name); + int fd = open(path, O_WRONLY | O_CLOEXEC); + if (fd < 0) + return -1; + ssize_t n = write(fd, value, strlen(value)); + close(fd); + return n == (ssize_t)strlen(value) ? 0 : -1; +} +static int configure_failslab(bool enable) +{ + if (!enable) + return fault_setting("probability", "0"); + /* The KUnit worker selects itself only around metadata writeout. */ + return fault_setting("task-filter", "Y") || + fault_setting("verbose", "0") || fault_setting("times", "-1") || + fault_setting("interval", "1") || fault_setting("space", "0") || + fault_setting("probability", "100"); +} + +static int swap_control_helper(const char *operation) +{ + struct stat init, self; + int ret; + + /* A helper is allowed only when this same executable is the VM's PID 1. */ + if (getpid() == 1 || stat("/proc/1/exe", &init) || + stat("/proc/self/exe", &self) || init.st_dev != self.st_dev || + init.st_ino != self.st_ino) { + fprintf(stderr, + "swap helper requires its fixture runner as PID 1\n"); + return 1; + } + if (!strcmp(operation, "--memcg-setup-helper")) { + if (mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL) && + errno != EBUSY) + return 1; + int fd = open("/sys/fs/cgroup/cgroup.subtree_control", + O_WRONLY | O_CLOEXEC); + if (fd < 0) + return 1; + ssize_t n = write(fd, "+memory", 7); + close(fd); + if (n != 7) + return 1; + return mkdir("/sys/fs/cgroup/uffd-metadata", 0755) && + errno != EEXIST; + } + if (!strcmp(operation, "--failslab-on-helper")) + return configure_failslab(true); + if (!strcmp(operation, "--failslab-off-helper")) + return configure_failslab(false); + ret = !strcmp(operation, "--swapoff-helper") ? swapoff("/dev/vda") : + swapon("/dev/vda", 0); + if (ret) + perror(operation); + return ret ? 1 : 0; +} + +int main(int argc, char **argv) +{ + struct stat st; + bool active = false; + int ret = 1; + if (argc == 2 && (!strcmp(argv[1], "--swapon-helper") || + !strcmp(argv[1], "--swapoff-helper") || + !strcmp(argv[1], "--failslab-on-helper") || + !strcmp(argv[1], "--failslab-off-helper") || + !strcmp(argv[1], "--memcg-setup-helper"))) + return swap_control_helper(argv[1]); + if (getpid() != 1 || argc != 1) + return 2; + setvbuf(stdout, NULL, _IONBF, 0); + unsigned long ps = getauxval(AT_PAGESZ); + if (ps != 16384 && ps != 65536) + goto out; + if (mount("sysfs", "/sys", "sysfs", 0, NULL) || + mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)) + goto out; + for (unsigned i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + if (stat("/dev/vda", &st) || !S_ISBLK(st.st_mode) || + swapon("/dev/vda", 0)) + goto out; + active = true; + ret = run_swap_suite() ? 1 : 0; +out: + if (active && swapoff("/dev/vda")) + ret = 1; + printf("SHMEM UFFD STATE SWAP %s errno=%d\n", ret ? "FAIL" : "PASS", + errno); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/splice-fixture-init.c b/tools/testing/selftests/arm64/user4k/splice-fixture-init.c new file mode 100644 index 00000000000000..bac72da15f40ec --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/splice-fixture-init.c @@ -0,0 +1,632 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +static bool use_tcp, use_ipv6, use_udp; + +static int connected_pair(int sockets[2]) +{ + struct sockaddr_in a4 = { .sin_family = AF_INET, + .sin_addr.s_addr = htonl(INADDR_LOOPBACK) }; + struct sockaddr_in6 a6 = { .sin6_family = AF_INET6, + .sin6_addr = IN6ADDR_LOOPBACK_INIT }; + struct sockaddr_storage peer; + struct sockaddr *address = use_ipv6 ? (void *)&a6 : (void *)&a4; + socklen_t size = use_ipv6 ? sizeof(a6) : sizeof(a4), + peer_size = sizeof(peer); + int domain = use_ipv6 ? AF_INET6 : AF_INET; + int listener, saved; + + if (!use_tcp && !use_udp) + return socketpair(AF_UNIX, SOCK_STREAM, 0, sockets); + listener = socket(domain, use_udp ? SOCK_DGRAM : SOCK_STREAM, 0); + if (listener < 0) + return -1; + sockets[0] = -1; + if (bind(listener, address, size) || + getsockname(listener, address, &size)) + goto error; + if (!use_udp && listen(listener, 1)) + goto error; + sockets[0] = socket(domain, use_udp ? SOCK_DGRAM : SOCK_STREAM, 0); + if (sockets[0] < 0 || connect(sockets[0], address, size)) + goto error; + if (use_udp) { + if (getsockname(sockets[0], (void *)&peer, &peer_size) || + connect(listener, (void *)&peer, peer_size)) + goto error; + sockets[1] = listener; + return 0; + } + sockets[1] = accept(listener, NULL, NULL); + if (sockets[1] < 0) + goto error; + close(listener); + return 0; +error: + saved = errno; + if (sockets[0] >= 0) + close(sockets[0]); + close(listener); + errno = saved; + return -1; +} + +static unsigned char value(size_t off) +{ + return (off * 13 + (off >> 12) * 31) & 255; +} +static void fill(unsigned char *p, size_t n) +{ + for (size_t i = 0; i < n; i++) + p[i] = value(i); +} +static bool check(const unsigned char *p, size_t n, size_t off) +{ + for (size_t i = 0; i < n; i++) + if (p[i] != value(i + off)) { + ksft_print_msg( + "data mismatch at %zu: got %02x expected %02x\n", + i + off, p[i], value(i + off)); + return false; + } + return true; +} +static bool read_exact(int fd, unsigned char *p, size_t n) +{ + while (n) { + ssize_t got = read(fd, p, n); + + if (got <= 0) { + ksft_print_msg( + "read fd=%d got=%zd remaining=%zu errno=%d\n", + fd, got, n, errno); + return false; + } + p += got; + n -= got; + } + return true; +} +static bool replace(unsigned char *p, size_t n) +{ + if (munmap(p, n) || + mmap(p, n, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0) != p) + return false; + memset(p, 0xec, n); + return true; +} +static bool cow(unsigned char *p, size_t n) +{ + pid_t child = fork(); + int status = 0; + + if (!child) + _exit(0); + if (child < 0 || waitpid(child, &status, 0) != child || status) + return false; + memset(p, 0xab, n); + return true; +} +static void report(bool ok, size_t ps, const char *name) +{ + ksft_test_result(ok, "%zuK %s\n", ps / 1024, name); +} +static bool tls_pair(int sockets[2], unsigned int version) +{ + struct tls12_crypto_info_aes_gcm_128 crypto = { + .info.version = version, + .info.cipher_type = TLS_CIPHER_AES_GCM_128, + }; + struct timeval timeout = { .tv_sec = 5 }; + int no_padding = 1; + + if (connected_pair(sockets)) + return false; + return !setsockopt(sockets[0], IPPROTO_TCP, TCP_ULP, "tls", 4) && + !setsockopt(sockets[1], IPPROTO_TCP, TCP_ULP, "tls", 4) && + !setsockopt(sockets[0], SOL_TLS, TLS_TX, &crypto, + sizeof(crypto)) && + !setsockopt(sockets[1], SOL_TLS, TLS_RX, &crypto, + sizeof(crypto)) && + (version != TLS_1_3_VERSION || + !setsockopt(sockets[1], SOL_TLS, TLS_RX_EXPECT_NO_PAD, + &no_padding, sizeof(no_padding))) && + !setsockopt(sockets[1], SOL_SOCKET, SO_RCVTIMEO, &timeout, + sizeof(timeout)); +} + +static void tls_checks(size_t ps, unsigned char *p, size_t len, + unsigned char *back) +{ + int sockets[2], fds[2]; + char name[160]; + + if (pipe(fds)) + ksft_exit_fail_msg("TLS pipe errno=%d\n", errno); + for (unsigned int v = 0; v < 2; v++) { + unsigned int version = v ? TLS_1_3_VERSION : TLS_1_2_VERSION; + struct iovec vec; + bool ok; + + if (!tls_pair(sockets, version)) + ksft_exit_fail_msg("TLS %x setup errno=%d\n", version, + errno); + for (unsigned int mode = 0; mode < 2; mode++) { + const size_t count = mode ? 4096 : 4096 - 29; + const size_t offset = ps + (mode ? 0 : 29); + + fill(p, len); + vec = (struct iovec){ p + offset, count }; + if (vmsplice(fds[1], &vec, 1, 0) != (ssize_t)count || + splice(fds[0], NULL, sockets[0], NULL, count, + SPLICE_F_MORE) != (ssize_t)count) + ksft_exit_fail_msg("TLS MORE splice errno=%d\n", + errno); + errno = 0; + if (recv(sockets[1], back, len, MSG_DONTWAIT) != -1 || + errno != EAGAIN) + ksft_exit_fail_msg( + "TLS pending plaintext precondition errno=%d\n", + errno); + ok = mode ? cow(p + ps, 4096) : replace(p + ps, ps); + ok &= send(sockets[0], "!", 1, 0) == 1; + ok &= read_exact(sockets[1], back, count + 1) && + check(back, count, offset) && back[count] == '!'; + snprintf(name, sizeof(name), + "TLS 1.%u pending plaintext survives %s", + v + 2, + mode ? "fork/COW" : "unmap/replacement"); + report(ok, ps, name); + } + { + const size_t bytes = 3 * 16384 + 123, prefix = 137; + const size_t mapping = (bytes + ps - 1) & ~(ps - 1); + unsigned char *source = + mmap(NULL, mapping, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + unsigned char *received = malloc(bytes + prefix); + + if (source == MAP_FAILED || !received) + ksft_exit_fail_msg( + "TLS multi-record allocation\n"); + fill(source, mapping); + ok = send(sockets[0], source, prefix, 0) == + (ssize_t)prefix; + vec = (struct iovec){ source, bytes }; + if (vmsplice(fds[1], &vec, 1, 0) != (ssize_t)bytes || + splice(fds[0], NULL, sockets[0], NULL, bytes, 0) != + (ssize_t)bytes) + ksft_exit_fail_msg( + "TLS multi-record splice errno=%d\n", + errno); + ok &= replace(source, mapping); + ok &= read_exact(sockets[1], received, + bytes + prefix) && + check(received, prefix, 0) && + check(received + prefix, bytes, 0); + snprintf( + name, sizeof(name), + "TLS 1.%u ordinary send and multiple spliced records", + v + 2); + report(ok, ps, name); + munmap(source, mapping); + free(received); + } + { + const size_t bytes = 5003; + int file = memfd_create("tls-receive", 0); + unsigned char *mapped; + + if (file < 0 || ftruncate(file, 3 * ps)) + ksft_exit_fail_msg("TLS receive file setup\n"); + mapped = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, file, 0); + if (mapped == MAP_FAILED) + ksft_exit_fail_msg( + "TLS receive file mapping\n"); + memset(mapped, 0x5c, 3 * ps); + fill(p, len); + ok = send(sockets[0], p, bytes, 0) == (ssize_t)bytes && + read_exact(sockets[1], mapped + 17, bytes); + ok &= !msync(mapped, 3 * ps, MS_SYNC) && + pread(file, back, bytes + 34, 0) == + (ssize_t)(bytes + 34) && + check(back + 17, bytes, 0); + for (unsigned int i = 0; i < 17; i++) + ok &= back[i] == 0x5c && + back[bytes + 17 + i] == 0x5c; + snprintf( + name, sizeof(name), + "TLS 1.%u decrypts into shared file with neighboring bytes intact", + v + 2); + report(ok, ps, name); + munmap(mapped, 3 * ps); + close(file); + } + { + unsigned char *mapped = + mmap(NULL, 6 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + struct iovec vectors[4]; + struct msghdr message = { .msg_iov = vectors, + .msg_iovlen = 4 }; + + if (mapped == MAP_FAILED) + ksft_exit_fail_msg( + "TLS receive iovec mapping\n"); + memset(mapped, 0x81, 6 * ps); + vectors[0] = (struct iovec){ mapped + 13, 5003 }; + vectors[1] = (struct iovec){ NULL, 0 }; + vectors[2] = + (struct iovec){ mapped + 2 * ps + 7, 2222 }; + vectors[3] = + (struct iovec){ mapped + 4 * ps + 5, 1010 }; + fill(p, len); + ok = send(sockets[0], p, 8235, 0) == 8235 && + recvmsg(sockets[1], &message, MSG_WAITALL) == + 8235 && + check(mapped + 13, 5003, 0) && + check(mapped + 2 * ps + 7, 2222, 5003) && + check(mapped + 4 * ps + 5, 1010, 7225); + ok &= mapped[12] == 0x81 && mapped[5016] == 0x81 && + mapped[2 * ps + 6] == 0x81 && + mapped[2 * ps + 2229] == 0x81 && + mapped[4 * ps + 4] == 0x81 && + mapped[4 * ps + 1015] == 0x81; + snprintf( + name, sizeof(name), + "TLS 1.%u scattered receive, empty iovec and subpage boundaries", + v + 2); + report(ok, ps, name); + munmap(mapped, 6 * ps); + } + close(sockets[0]); + close(sockets[1]); + ok = true; + for (unsigned int round = 0; round < 64 && ok; round++) { + if (!tls_pair(sockets, version)) + ksft_exit_fail_msg( + "TLS teardown setup errno=%d\n", errno); + fill(p, len); + vec = (struct iovec){ p + ps, 4096 }; + ok = vmsplice(fds[1], &vec, 1, 0) == 4096 && + splice(fds[0], NULL, sockets[0], NULL, 4096, + SPLICE_F_MORE) == 4096; + close(sockets[1]); + close(sockets[0]); + ok &= !madvise(p + ps, ps, MADV_DONTNEED); + memset(p + ps, 0x75, ps); + } + snprintf(name, sizeof(name), + "TLS 1.%u repeated pending-record teardown and reuse", + v + 2); + report(ok, ps, name); + } + close(fds[0]); + close(fds[1]); +} + +static int exercise(void) +{ + const size_t ps = getauxval(AT_PAGESZ); + size_t len; + unsigned char *p, *back; + int native; + unsigned int mss = 0; + socklen_t mss_len = sizeof(mss); + int fds[2], dup[2], file, sockets[2]; + struct iovec v[3]; + struct stat self, init; + struct timeval timeout = { .tv_sec = 5 }; + bool ok; + ssize_t result; + + if (stat("/proc/self/exe", &self) || stat("/proc/1/exe", &init) || + self.st_dev != init.st_dev || self.st_ino != init.st_ino) + return 2; + if (pipe(fds) || pipe(dup)) + ksft_exit_fail_msg("pipe setup\n"); + /* A pipe has at least one native page, independently of this MM. */ + native = fcntl(fds[1], F_SETPIPE_SZ, 1); + if (native <= 0 || fcntl(fds[1], F_SETPIPE_SZ, 16 * native) < 0) + ksft_exit_fail_msg("pipe capacity setup\n"); + len = 8 * ps > (size_t)native ? 8 * ps : (size_t)native; + back = malloc(len); + p = mmap((void *)0x300000000UL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + if (p == MAP_FAILED || !back) + ksft_exit_fail_msg("setup\n"); + ksft_print_header(); + ksft_set_plan(use_tcp ? 22 : 10); + ksft_print_msg("transport=%s family=%s\n", + use_tcp ? "TCP" : + use_udp ? "UDP" : + "Unix stream", + use_ipv6 ? "IPv6" : "IPv4/Unix"); + fill(p, len); + v[0] = (struct iovec){ NULL, 0 }; + v[1] = (struct iovec){ p + 17, ps - 17 }; + v[2] = (struct iovec){ p + 2 * ps + 31, 2 * ps - 31 }; + result = vmsplice(fds[1], v, 3, 0); + if (result != (ssize_t)(3 * ps - 48)) + ksft_exit_fail_msg("vmsplice returned %zd errno=%d\n", result, + errno); + ok = read_exact(fds[0], back, result) && check(back, ps - 17, 17) && + check(back + ps - 17, 2 * ps - 31, 2 * ps + 31); + report(ok, ps, "unaligned fragmented iovecs and empty segment"); + v[0] = (struct iovec){ p, native }; + ok = fcntl(fds[1], F_SETPIPE_SZ, native) == native; + result = vmsplice(fds[1], v, 1, SPLICE_F_NONBLOCK); + { + int pending = 0; + + ok &= result == native && !ioctl(fds[0], FIONREAD, &pending) && + pending == native; + } + if (result > 0) + ok &= read_exact(fds[0], back, result) && + check(back, result, 0); + report(ok, ps, + "contiguous fragments retain one-native-page pipe capacity"); + fcntl(fds[1], F_SETPIPE_SZ, 16 * native); + v[0] = (struct iovec){ p + 11, 3 * ps - 11 }; + result = vmsplice(fds[1], v, 1, 0); + if (result != (ssize_t)(3 * ps - 11) || + tee(fds[0], dup[1], result, 0) != result) + ksft_exit_fail_msg("tee setup\n"); + ok = replace(p, len); + ok &= read_exact(fds[0], back, 19) && check(back, 19, 11); + ok &= read_exact(fds[0], back + 19, result - 19) && + check(back, result, 11); + ok &= read_exact(dup[0], back, result) && check(back, result, 11); + report(ok, ps, + "tee and partial reads retain data across unmap/replacement"); + fill(p, len); + v[0] = (struct iovec){ p, ps }; + ok = vmsplice(fds[1], v, 1, 0) == (ssize_t)ps && cow(p, ps); + ok &= read_exact(fds[0], back, ps) && check(back, ps, 0); + report(ok, ps, "pipe GET retains pre-COW data after fork sharer exit"); + fill(p, len); + file = memfd_create("splice-target", 0); + v[0] = (struct iovec){ p + ps + 29, 2 * ps - 29 }; + result = vmsplice(fds[1], v, 1, SPLICE_F_GIFT); + ok = file >= 0 && result == (ssize_t)(2 * ps - 29) && + splice(fds[0], NULL, file, NULL, result, SPLICE_F_MOVE) == result; + ok &= pread(file, back, result, 0) == result && + check(back, result, ps + 29); + report(ok, ps, "gifted fragments splice to buffered file"); + close(file); + v[0] = (struct iovec){ p, ps }; + v[1] = (struct iovec){ (void *)0x1000, ps }; + result = vmsplice(fds[1], v, 2, 0); + ok = result == (ssize_t)ps && read_exact(fds[0], back, ps) && + check(back, ps, 0); + report(ok, ps, "partial vmsplice before inaccessible iovec"); + if (connected_pair(sockets)) + ksft_exit_fail_msg("socketpair errno=%d\n", errno); + setsockopt(sockets[1], SOL_SOCKET, SO_RCVTIMEO, &timeout, + sizeof(timeout)); + if (use_tcp && + (getsockopt(sockets[0], IPPROTO_TCP, TCP_MAXSEG, &mss, &mss_len) || mss <= 1)) + ksft_exit_fail_msg("TCP_MAXSEG setup\n"); + for (unsigned int mode = 0; mode < 3; mode++) { + size_t bytes = mode == 1 ? ps : + (use_udp && ps == 4096 ? + 6 * ps - 256 : + 2 * ps - 256); + const size_t offset = mode == 1 ? 0 : ps + 29; + int cork = 1; + + /* Keep the corked payload below one segment; UDP has a byte limit. */ + if (use_tcp && mode < 2 && bytes >= mss) + bytes = mss - 1; + if (use_udp && bytes > 65507) + bytes = 65507; + if ((use_tcp || use_udp) && mode < 2 && + setsockopt(sockets[0], use_tcp ? IPPROTO_TCP : IPPROTO_UDP, + use_tcp ? TCP_CORK : UDP_CORK, &cork, + sizeof(cork))) + ksft_exit_fail_msg("TCP_CORK setup errno=%d\n", errno); + fill(p, len); + v[0] = (struct iovec){ p + offset, bytes }; + ok = vmsplice(fds[1], v, 1, 0) == (ssize_t)bytes && + splice(fds[0], NULL, sockets[0], NULL, bytes, + SPLICE_F_NONBLOCK) == (ssize_t)bytes; + if (!ok) + ksft_exit_fail_msg("socket splice setup errno=%d\n", + errno); + if (mode == 2) { + ok = splice(sockets[1], NULL, fds[1], NULL, bytes, 0) == + (ssize_t)bytes && + tee(fds[0], dup[1], bytes, 0) == (ssize_t)bytes; + if (!ok) + ksft_exit_fail_msg( + "socket-to-pipe setup errno=%d\n", + errno); + } + if (use_tcp && mode < 2) { + int unsent = 0; + + if (ioctl(sockets[0], SIOCOUTQNSD, &unsent) || + unsent != (int)bytes) + ksft_exit_fail_msg( + "TCP unsent precondition: %d expected %zu errno=%d\n", + unsent, bytes, errno); + } + if (use_udp && mode < 2) { + errno = 0; + if (recv(sockets[1], back, len, MSG_DONTWAIT) != -1 || + errno != EAGAIN) + ksft_exit_fail_msg( + "UDP corked precondition errno=%d\n", + errno); + } + ok &= mode == 1 ? cow(p, ps) : replace(p + ps, 2 * ps); + if ((use_tcp || use_udp) && mode < 2) { + cork = 0; + if (setsockopt(sockets[0], + use_tcp ? IPPROTO_TCP : IPPROTO_UDP, + use_tcp ? TCP_CORK : UDP_CORK, &cork, + sizeof(cork))) { + ksft_print_msg("uncork errno=%d\n", errno); + ok = false; + } + } + ok &= read_exact(mode == 2 ? fds[0] : sockets[1], back, + bytes) && + check(back, bytes, offset); + if (mode == 2) + ok &= read_exact(dup[0], back, bytes) && + check(back, bytes, offset); + report(ok, ps, + mode == 0 ? + "socket retains quarters across partial unmap/replacement" : + mode == 1 ? + "socket GET retains pre-COW data after fork sharer exit" : + "socket-to-pipe and tee retain independent quarter ownership"); + } + close(sockets[0]); + close(sockets[1]); + ok = true; + for (unsigned int round = 0; round < 128 && ok; round++) { + const size_t bytes = use_udp && ps > 65507 ? 65507 : ps; + + fill(p, len); + ok = !connected_pair(sockets); + v[0] = (struct iovec){ p + ps, bytes }; + ok &= vmsplice(fds[1], v, 1, 0) == (ssize_t)bytes && + splice(fds[0], NULL, sockets[0], NULL, bytes, 0) == + (ssize_t)bytes; + close(sockets[1]); + close(sockets[0]); + ok &= !madvise(p + ps, ps, MADV_DONTNEED); + memset(p + ps, 0x43, ps); + } + report(ok, ps, "repeated unread socket teardown and quarter reuse"); + if (use_tcp) + tls_checks(ps, p, len, back); + close(fds[0]); + close(fds[1]); + close(dup[0]); + close(dup[1]); + munmap(p, len); + free(back); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} +static bool async_cleanup_balanced(void) +{ + const char *names[] = { "user4k_rx_deferred", "user4k_rx_released" }; + unsigned long counts[2]; + char path[128]; + FILE *file = fopen("/sys/module/tls/parameters/user4k_test_async", "r"); + int enabled; + + if (!file) + return false; + enabled = fgetc(file); + fclose(file); + for (unsigned int i = 0; i < 2; i++) { + snprintf(path, sizeof(path), "/sys/module/tls/parameters/%s", + names[i]); + file = fopen(path, "r"); + if (!file) + return false; + if (fscanf(file, "%lu", &counts[i]) != 1) { + fclose(file); + return false; + } + fclose(file); + } + printf("# TLS async=%c deferred=%lu released=%lu\n", enabled, counts[0], + counts[1]); + return counts[0] == counts[1] && (enabled != 'Y' || counts[0] > 0); +} + +int main(int argc, char **argv) +{ + bool ok = true; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 2 && + (!strcmp(argv[1], "--test") || !strcmp(argv[1], "--tcp") || + !strcmp(argv[1], "--tcp6") || !strcmp(argv[1], "--udp") || + !strcmp(argv[1], "--udp6")) && + getppid() == 1) { + use_tcp = !strncmp(argv[1], "--tcp", 5); + use_udp = !strncmp(argv[1], "--udp", 5); + use_ipv6 = strchr(argv[1], '6') != NULL; + return exercise(); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Run only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) + return 3; + { + struct ifreq request = { .ifr_name = "lo" }; + int fd = socket(AF_INET, SOCK_DGRAM, 0); + + if (fd < 0 || ioctl(fd, SIOCGIFFLAGS, &request)) + return 4; + request.ifr_flags |= IFF_UP; + if (ioctl(fd, SIOCSIFFLAGS, &request)) + return 4; + close(fd); + } + for (unsigned int mode = 0; mode < 10; mode++) { + unsigned int small = mode & 1; + pid_t pid = fork(); + int status = 0; + + if (!pid) { + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + small ? 4096UL : 0UL, 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", + mode >= 8 ? "--udp6" : + mode >= 6 ? "--udp" : + mode >= 4 ? "--tcp6" : + mode >= 2 ? "--tcp" : + "--test", + NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + } + ok &= async_cleanup_balanced(); + printf("%s - splice fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/swap-fixture-init.c b/tools/testing/selftests/arm64/user4k/swap-fixture-init.c new file mode 100644 index 00000000000000..45f6aefdd094e8 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/swap-fixture-init.c @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * PID-1 runner for a disposable VM with a dedicated virtio swap disk. + * Use kunit.autorun=0: tests skipped at boot stay skipped on a debugfs rerun. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static unsigned long swapouts(void) +{ + FILE *f = fopen("/proc/vmstat", "re"); + char key[128]; + unsigned long value, found = 0; + + if (!f) + return 0; + while (fscanf(f, "%127s %lu", key, &value) == 2) + if (!strcmp(key, "pswpout")) + found = value; + fclose(f); + return found; +} + +static int native_swap_smoke(void) +{ + const size_t length = 4 * 1024 * 1024, page = 16384; + unsigned long before = swapouts(), after; + unsigned char *p = mmap(NULL, length, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + size_t i; + unsigned int tries; + int ret = 0; + + if (p == MAP_FAILED) + return -1; + for (i = 0; i < length; i++) + p[i] = (i ^ (i >> 12) ^ (i >> 20)) & 255; + if (madvise(p, length, MADV_PAGEOUT)) { + ret = -1; + goto out; + } + for (tries = 0; tries < 100 && swapouts() == before; tries++) + usleep(20000); + after = swapouts(); + for (i = 0; i < length; i++) + if (p[i] != ((i ^ (i >> 12) ^ (i >> 20)) & 255)) { + ret = -1; + break; + } + if (after <= before) + ret = -1; + printf("%s - native %zuK swap transport: pswpout %lu -> %lu, verified %zu bytes\n", + ret ? "not ok" : "ok", page / 1024, before, after, i); +out: + munmap(p, length); + return ret; +} + +static int run_swap_suite(void) +{ + const char *base = "/sys/kernel/debug/kunit/arm64-user4k-swap/"; + char path[128], *log; + size_t used = 0, capacity = 1024 * 1024; + ssize_t n; + int fd, ret = -1; + + snprintf(path, sizeof(path), "%srun", base); + fd = open(path, O_WRONLY | O_CLOEXEC); + if (fd < 0) + return -1; + n = write(fd, "1", 1); + close(fd); + if (n != 1) + return -1; + snprintf(path, sizeof(path), "%sresults", base); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return -1; + log = calloc(1, capacity); + if (!log) { + close(fd); + return -1; + } + while (used < capacity - 1 && + (n = read(fd, log + used, capacity - 1 - used)) > 0) + used += n; + close(fd); + fputs(log, stdout); + if (n == 0 && used < capacity - 1 && + strstr(log, "\nok 1 arm64-user4k-swap\n") && + !strstr(log, "not ok") && !strstr(log, "# SKIP")) + ret = 0; + free(log); + return ret; +} + +static int swap_control_helper(const char *operation) +{ + struct stat init, self; + int ret; + + /* A helper is allowed only when this same executable is the VM's PID 1. */ + if (getpid() == 1 || stat("/proc/1/exe", &init) || + stat("/proc/self/exe", &self) || init.st_dev != self.st_dev || + init.st_ino != self.st_ino) { + fprintf(stderr, + "swap helper requires its fixture runner as PID 1\n"); + return 1; + } + ret = !strcmp(operation, "--swapoff-helper") ? swapoff("/dev/vda") : + swapon("/dev/vda", 0); + if (ret) + perror(operation); + return ret ? 1 : 0; +} + +static int pageout_exec(bool small) +{ + pid_t pid = fork(); + int status = 0; + + if (!pid) { + if (small && + prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, 4096UL, 0UL, 0UL, 0UL)) + _exit(125); + execl("/reclaim-advice", "/reclaim-advice", + small ? "--expect-4k" : NULL, NULL); + _exit(126); + } + if (pid <= 0 || waitpid(pid, &status, 0) != pid) + return -1; + printf("# EL0 pageout %s child status=%#x\n", small ? "4K" : "native", + status); + return WIFEXITED(status) && !WEXITSTATUS(status) ? 0 : -1; +} + +int main(int argc, char **argv) +{ + bool smoke = argc == 2 && !strcmp(argv[1], "--smoke"); + bool pageout = argc == 2 && !strcmp(argv[1], "--pageout"); + bool native_only = argc == 2 && !strcmp(argv[1], "--pageout-native"); + bool active = false; + unsigned int tries; + struct stat st; + pid_t child; + int status, ret = 1; + + if (argc == 2 && (!strcmp(argv[1], "--swapoff-helper") || + !strcmp(argv[1], "--swapon-helper"))) + return swap_control_helper(argv[1]); + /* This runner must never mount filesystems or enable swap on the host. */ + if (getpid() != 1 || + (argc != 1 && !smoke && !pageout && !native_only)) { + fprintf(stderr, + "Use only as /init in the dedicated disposable test VM.\n"); + return 1; + } + setvbuf(stdout, NULL, _IONBF, 0); + if (getauxval(AT_PAGESZ) != 16384 || sysconf(_SC_PAGESIZE) != 16384) + goto out; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL)) + goto out; + child = fork(); + if (!child) { + execl("/page-contract", "page-contract", NULL); + _exit(127); + } + if (child < 0 || waitpid(child, &status, 0) != child || + !WIFEXITED(status) || WEXITSTATUS(status)) + goto out; + for (tries = 0; tries < 100 && stat("/dev/vda", &st); tries++) + usleep(20000); + if (stat("/dev/vda", &st) || !S_ISBLK(st.st_mode) || + swapon("/dev/vda", 0)) + goto out; + active = true; + if (native_swap_smoke()) + goto out; + if (!smoke && !native_only && run_swap_suite()) + goto out; + if ((pageout || native_only) && pageout_exec(false)) + goto out; + if (pageout && pageout_exec(true)) + goto out; + ret = 0; +out: + if (ret) + printf("not ok - swap fixture runner: errno=%d (%s)\n", errno, + strerror(errno)); + if (active && swapoff("/dev/vda")) { + perror("swapoff"); + ret = 1; + } + printf("%s - swap fixture runner (%s)\n", ret ? "not ok" : "ok", + smoke ? "native transport smoke only" : "4K swap suite"); + fflush(NULL); + reboot(RB_POWER_OFF); + return ret; +} diff --git a/tools/testing/selftests/arm64/user4k/swapfile-fixture.h b/tools/testing/selftests/arm64/user4k/swapfile-fixture.h new file mode 100644 index 00000000000000..258b81fa056958 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/swapfile-fixture.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef USER4K_SWAPFILE_FIXTURE_H +#define USER4K_SWAPFILE_FIXTURE_H + +static const char *swap_path(void) +{ + return access("/swapdir/swapfile", F_OK) ? "/dev/vda" : + "/swapdir/swapfile"; +} + +/* Remove every fourth native page: every 64K group crosses file extents. */ +static int prepare_swapfile(void) +{ + const unsigned long page = getauxval(AT_PAGESZ), initial = 32UL << 20; + unsigned char *buf = calloc(1, page); + unsigned long offset; + int fd = -1, ret = -1; + + if (!buf || (mkdir("/swapdir", 0755) && errno != EEXIST) || + mount("/dev/vda", "/swapdir", "ext4", 0, NULL)) + goto out; + fd = open("/swapdir/swapfile", O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC, + 0600); + if (fd < 0) + goto out; + memset(buf, 0xa5, page); + for (offset = 0; offset < initial; offset += page) + if (write(fd, buf, page) != (ssize_t)page) + goto out; + if (fsync(fd)) + goto out; + for (offset = initial; offset; offset -= 4 * page) + if (fallocate(fd, FALLOC_FL_COLLAPSE_RANGE, offset - 2 * page, + page)) + goto out; + memset(buf, 0, page); + ((unsigned int *)(buf + 1024))[0] = 1; + ((unsigned int *)(buf + 1024))[1] = (initial * 3 / 4) / page - 1; + memcpy(buf + page - 10, "SWAPSPACE2", 10); + if (pwrite(fd, buf, page, 0) != (ssize_t)page || fsync(fd)) + goto out; + ret = 0; +out: + if (fd >= 0) + close(fd); + free(buf); + return ret; +} + +#endif diff --git a/tools/testing/selftests/arm64/user4k/sysv-shm-init.c b/tools/testing/selftests/arm64/user4k/sysv-shm-init.c new file mode 100644 index 00000000000000..3b66d9c738fde0 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/sysv-shm-init.c @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Disposable PID1: SysV shared memory with per-process page granules. */ +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static void check(int ok, const char *expression, int line) +{ + if (!ok) { + fprintf(stderr, "FAIL line %d: %s errno=%d\n", line, expression, + errno); + exit(1); + } +} + +#define CHECK(x) check(!!(x), #x, __LINE__) + +static int unmapped(void *address, size_t page) +{ + unsigned char resident; + + errno = 0; + return mincore(address, page, &resident) == -1 && errno == ENOMEM; +} + +static int mapping_case(unsigned long native, int rounded, int split) +{ + unsigned long page = getauxval(AT_PAGESZ), length = 4 * page; + unsigned char *reserve, *target, *mapping, *alias; + struct shmid_ds info; + int id; + + reserve = mmap(NULL, 16 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(reserve != MAP_FAILED); + target = (void *)(((uintptr_t)reserve + native - 1) & ~(native - 1)); + target += native + (page < native ? page : 0); + target[-1] = 0x51; + target[length] = 0x72; + CHECK(!munmap(target, length)); + id = shmget(IPC_PRIVATE, length - 1, 0600); + CHECK(id >= 0); + mapping = shmat(id, target + (rounded ? page - 1 : 0), + rounded ? SHM_RND : 0); + if (mapping != target) { + int saved = errno; + + shmctl(id, IPC_RMID, NULL); + errno = saved; + CHECK(mapping == target); + } + alias = shmat(id, NULL, 0); + CHECK(alias != (void *)-1); + CHECK(!shmctl(id, IPC_RMID, NULL)); + mapping[0] = 0x31; + mapping[page + 7] = 0x49; + mapping[3 * page] = 0x63; + CHECK(alias[0] == 0x31 && alias[page + 7] == 0x49 && + alias[3 * page] == 0x63); + if (split) { + CHECK(!mprotect(mapping + page, page, PROT_READ)); + CHECK(!munmap(mapping + 2 * page, page)); + if (split == 2) + CHECK(!munmap(mapping, page)); + } + CHECK(!shmdt(mapping)); + for (unsigned long i = 0; i < length; i += page) + CHECK(unmapped(target + i, page)); + CHECK(target[-1] == 0x51 && target[length] == 0x72); + CHECK(alias[page + 7] == 0x49 && alias[3 * page] == 0x63); + CHECK(!shmdt(alias)); + errno = 0; + CHECK(shmctl(id, IPC_STAT, &info) == -1 && + (errno == EINVAL || errno == EIDRM)); + CHECK(!munmap(reserve, 16 * native)); + printf("ok - SysV page=%lu rounded=%d split=%d detach and alias lifetime\n", + page, rounded, split); + return 0; +} + +static int wait_child(pid_t pid) +{ + int status; + pid_t got; + + if (pid < 0) + return 1; + do { + got = waitpid(pid, &status, 0); + } while (got < 0 && errno == EINTR); + return got != pid || !WIFEXITED(status) || WEXITSTATUS(status); +} + +static int sharing_child(int id, unsigned long expected) +{ + unsigned char *map; + + CHECK(getauxval(AT_PAGESZ) == expected); + map = shmat(id, NULL, 0); + CHECK(map != (void *)-1); + CHECK(map[37] == 0x35 && map[4096 + 9] == 0x56); + map[4096 + 9] = 0x78; + CHECK(!shmdt(map)); + return 0; +} + +static int sharing(unsigned long other, unsigned long native) +{ + unsigned long page = getauxval(AT_PAGESZ); + unsigned char *map; + pid_t pid; + int id; + + id = shmget(IPC_PRIVATE, 2 * native, 0600); + CHECK(id >= 0); + map = shmat(id, NULL, 0); + CHECK(map != (void *)-1); + CHECK(!shmctl(id, IPC_RMID, NULL)); + map[37] = 0x35; + map[4096 + 9] = 0x56; + pid = fork(); + if (!pid) { + char idarg[32], sizearg[32]; + + snprintf(idarg, sizeof(idarg), "%d", id); + snprintf(sizearg, sizeof(sizearg), "%lu", other); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, other, 0UL, 0UL, 0UL)) + _exit(2); + execl("/init", "/init", "--share", idarg, sizearg, NULL); + _exit(3); + } + CHECK(!wait_child(pid)); + CHECK(map[4096 + 9] == 0x78); + CHECK(!shmdt(map)); + printf("ok - SysV shared object %lu -> %lu after IPC_RMID\n", page, + other); + return 0; +} + +int main(int argc, char **argv) +{ + unsigned long native = getauxval(AT_PAGESZ); + int failed = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--share")) + return sharing_child(atoi(argv[2]), strtoul(argv[3], NULL, 10)); + if (argc == 5 && !strcmp(argv[1], "--case")) { + unsigned long expected = strtoul(argv[3], NULL, 10); + unsigned long sizes[] = { 4096, 16384, 65536 }; + + native = strtoul(argv[2], NULL, 10); + CHECK(getauxval(AT_PAGESZ) == expected); + failed |= mapping_case(native, 0, 0); + failed |= mapping_case(native, 1, 0); + failed |= mapping_case(native, 0, 1); + failed |= mapping_case(native, 0, 2); + for (unsigned int i = 0; i < 3; i++) + if (sizes[i] <= native && sizes[i] != expected && + !(atoi(argv[4]) && sizes[i] == 4096)) + failed |= sharing(sizes[i], native); + return failed; + } + CHECK(getpid() == 1); + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + int no4k = argc == 2 && !strcmp(argv[1], "--no4k"); + unsigned long sizes[] = { 4096, 16384, 65536 }; + + for (unsigned int i = 0; i < 3; i++) { + unsigned long page = sizes[i]; + pid_t pid; + + if (page > native || (no4k && page == 4096)) + continue; + pid = fork(); + if (!pid) { + char n[32], e[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(e, sizeof(e), "%lu", page); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, page, 0UL, 0UL, + 0UL)) + _exit(2); + execl("/init", "/init", "--case", n, e, + no4k ? "1" : "0", NULL); + _exit(3); + } + int bad = wait_child(pid); + + printf("%s - SysV AArch64 native=%lu process=%lu\n", + bad ? "not ok" : "ok", native, page); + failed |= bad; + } + if (access("/compat-shm", X_OK) == 0) { + pid_t pid = fork(); + + if (!pid) { + char n[32], e[32]; + + snprintf(n, sizeof(n), "%lu", native); + snprintf(e, sizeof(e), "%lu", no4k ? native : 4096UL); + execl("/compat-shm", "/compat-shm", n, e, NULL); + _exit(3); + } + int bad = wait_child(pid); + + printf("%s - SysV AArch32 native=%lu process=%lu\n", + bad ? "not ok" : "ok", native, no4k ? native : 4096UL); + failed |= bad; + } + printf("SYSV SHM GRANULE %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/thread-fixture-init.c b/tools/testing/selftests/arm64/user4k/thread-fixture-init.c new file mode 100644 index 00000000000000..c472ea6697947c --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/thread-fixture-init.c @@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +#define THREADS 4 +#define ROUNDS 128 +#define PAGES 32 + +static pthread_barrier_t barrier; +static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t condition = PTHREAD_COND_INITIALIZER; +static pthread_mutex_t robust; +static _Thread_local unsigned int tls_id; +static _Thread_local volatile sig_atomic_t signals; +static atomic_uint failures; +static unsigned long total; +static unsigned long ps; +static atomic_int waiter_tid; + +static void handler(int sig) +{ + if (sig == SIGUSR1 && tls_id) + signals++; + else + atomic_fetch_add(&failures, 1); +} + +static bool bytes_equal(unsigned char *p, size_t n, unsigned char value) +{ + for (size_t i = 0; i < n; i++) + if (p[i] != value) + return false; + return true; +} + +static void *worker(void *arg) +{ + unsigned int id = (uintptr_t)arg; + unsigned char *p, *heap; + unsigned long request = id & 1 ? 4096 : 0; + bool ok = true; + + tls_id = id; + p = mmap(NULL, PAGES * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0); + if (p == MAP_FAILED) + ok = false; + ok &= !prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, request, 0UL, 0UL, 0UL); + pthread_barrier_wait(&barrier); + for (unsigned int round = 0; round < ROUNDS; round++) { + unsigned char pattern = 1 + (id * 31 + round) % 254; + unsigned char residency[PAGES]; + cpu_set_t cpu; + + if (ok) { + CPU_ZERO(&cpu); + CPU_SET((id + round) % 2, &cpu); + ok &= !sched_setaffinity(0, sizeof(cpu), &cpu); + memset(p, pattern, PAGES * ps); + ok &= !mprotect(p + ps, ps, PROT_READ) && + p[ps] == pattern; + ok &= !mprotect(p + ps, ps, PROT_READ | PROT_WRITE); + ok &= !madvise(p + 2 * ps, ps, MADV_FREE); + memset(p + 2 * ps, pattern, ps); + ok &= !madvise(p + 3 * ps, ps, MADV_DONTNEED); + ok &= bytes_equal(p + 3 * ps, ps, 0); + memset(p + 3 * ps, pattern, ps); + heap = malloc(5000 + id * 777); + if (!heap) { + ok = false; + } else { + memset(heap, pattern, 5000 + id * 777); + ok &= bytes_equal(heap, 5000 + id * 777, + pattern); + free(heap); + } + ok &= !pthread_kill(pthread_self(), SIGUSR1); + ok &= tls_id == id && + signals == (sig_atomic_t)(round + 1); + ok &= prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, + 0UL) == (int)request; + ok &= !mincore(p, PAGES * ps, residency); + ok &= bytes_equal(p, PAGES * ps, pattern); + } + pthread_mutex_lock(&mutex); + total++; + pthread_cond_signal(&condition); + pthread_mutex_unlock(&mutex); + pthread_barrier_wait(&barrier); + } + if (p != MAP_FAILED) + munmap(p, PAGES * ps); + if (!ok) + atomic_fetch_add(&failures, 1); + return NULL; +} + +static void *owner_dies(void *arg) +{ + (void)arg; + if (pthread_mutex_lock(&robust)) + atomic_fetch_add(&failures, 1); + return NULL; +} + +static void *pi_waiter(void *arg) +{ + int error; + + (void)arg; + atomic_store(&waiter_tid, syscall(SYS_gettid)); + error = pthread_mutex_lock(&robust); + if (!error) + error = pthread_mutex_unlock(&robust); + return (void *)(uintptr_t)error; +} + +static bool wait_for_pi_block(void) +{ + for (unsigned int tries = 0; tries < 1000; tries++) { + int tid = atomic_load(&waiter_tid); + char path[80], line[256]; + unsigned long nr, address, operation; + FILE *f; + + if (tid) { + snprintf(path, sizeof(path), + "/proc/self/task/%d/syscall", tid); + f = fopen(path, "re"); + if (f) { + bool blocked = fgets(line, sizeof(line), f) && + sscanf(line, "%lu %lx %lx", &nr, + &address, + &operation) == 3 && + nr == SYS_futex && + ((operation & FUTEX_CMD_MASK) == + FUTEX_LOCK_PI || + (operation & FUTEX_CMD_MASK) == + FUTEX_LOCK_PI2); + + fclose(f); + if (blocked) + return true; + } + } + usleep(1000); + } + return false; +} + +static int exercise(void) +{ + pthread_t threads[THREADS], owner; + pthread_mutexattr_t attributes; + struct sigaction action = { .sa_handler = handler }; + struct stat self, init; + bool ok = true, forks = true; + int error; + + if (stat("/proc/self/exe", &self) || stat("/proc/1/exe", &init) || + self.st_dev != init.st_dev || self.st_ino != init.st_ino) + return 2; + ps = getauxval(AT_PAGESZ); + ksft_print_header(); + ksft_set_plan(6); + if (sigaction(SIGUSR1, &action, NULL) || + pthread_barrier_init(&barrier, NULL, THREADS + 1)) + ksft_exit_fail_msg("thread setup\n"); + for (unsigned int i = 0; i < THREADS; i++) + if (pthread_create(&threads[i], NULL, worker, + (void *)(uintptr_t)(i + 1))) + ksft_exit_fail_msg("pthread_create\n"); + pthread_barrier_wait(&barrier); + for (unsigned int round = 0; round < ROUNDS; round++) { + pthread_mutex_lock(&mutex); + while (total < (round + 1) * THREADS) + pthread_cond_wait(&condition, &mutex); + pthread_mutex_unlock(&mutex); + ok &= prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL) == + 0; + if (!(round % 16)) { + pid_t child = fork(); + int status = 0; + + if (!child) { + unsigned char *p = malloc(32768); + + if (!p || tls_id || + total != (round + 1) * THREADS || + prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, + 0UL, 0UL, 0UL)) + _exit(10); + memset(p, 0x75, 32768); + if (!bytes_equal(p, 32768, 0x75)) + _exit(11); + free(p); + _exit(0); + } + forks &= child > 0 && + waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + } + pthread_barrier_wait(&barrier); + } + for (unsigned int i = 0; i < THREADS; i++) + ok &= !pthread_join(threads[i], NULL); + ksft_test_result( + ok && !atomic_load(&failures) && total == THREADS * ROUNDS, + "%luK %u concurrent memory/TLS/signal/exec-request rounds across %u threads\n", + ps / 1024, ROUNDS, THREADS); + ksft_test_result( + forks, "%luK fork and allocator from a multithreaded process\n", + ps / 1024); + pthread_barrier_destroy(&barrier); + ok = !pthread_mutexattr_init(&attributes); + ok &= !pthread_mutexattr_setrobust(&attributes, PTHREAD_MUTEX_ROBUST); + ok &= !pthread_mutex_init(&robust, &attributes); + ok &= !pthread_create(&owner, NULL, owner_dies, NULL); + if (!ok) + ksft_exit_fail_msg("robust mutex setup\n"); + ok &= !pthread_join(owner, NULL); + error = pthread_mutex_lock(&robust); + ok &= error == EOWNERDEAD; + if (error == EOWNERDEAD) + ok &= !pthread_mutex_consistent(&robust); + if (!error || error == EOWNERDEAD) + ok &= !pthread_mutex_unlock(&robust); + ksft_test_result(ok, "%luK robust futex owner-death recovery\n", + ps / 1024); + pthread_mutex_destroy(&robust); + /* Exercise repeated TLS/stack allocation and set_tid_address clear/wake. */ + for (unsigned int i = 0; i < 128; i++) { + ok = !pthread_mutex_init(&robust, &attributes); + ok &= !pthread_create(&owner, NULL, owner_dies, NULL); + if (!ok) + break; + ok &= !pthread_join(owner, NULL); + error = pthread_mutex_lock(&robust); + ok &= error == EOWNERDEAD; + if (error == EOWNERDEAD) + ok &= !pthread_mutex_consistent(&robust); + if (!error || error == EOWNERDEAD) + pthread_mutex_unlock(&robust); + pthread_mutex_destroy(&robust); + if (!ok) + break; + } + ksft_test_result(ok && !atomic_load(&failures), + "%luK repeated thread exit/join and robust recovery\n", + ps / 1024); + ok = !pthread_mutexattr_setprotocol(&attributes, PTHREAD_PRIO_INHERIT); + ok &= !pthread_mutex_init(&robust, &attributes); + ok &= !pthread_mutex_lock(&robust); + ok &= !pthread_create(&owner, NULL, pi_waiter, NULL); + if (!ok) + ksft_exit_fail_msg("PI mutex setup\n"); + ok &= wait_for_pi_block(); + ok &= !pthread_mutex_unlock(&robust); + { + void *result = (void *)1; + + ok &= !pthread_join(owner, &result) && !result; + } + ksft_test_result( + ok, + "%luK confirmed contended priority-inheritance futex handoff\n", + ps / 1024); + ok = !pthread_create(&owner, NULL, owner_dies, NULL); + if (!ok) + ksft_exit_fail_msg("PI owner setup\n"); + ok &= !pthread_join(owner, NULL); + error = pthread_mutex_lock(&robust); + ok &= error == EOWNERDEAD; + if (error == EOWNERDEAD) + ok &= !pthread_mutex_consistent(&robust); + if (!error || error == EOWNERDEAD) + ok &= !pthread_mutex_unlock(&robust); + pthread_mutex_destroy(&robust); + pthread_mutexattr_destroy(&attributes); + ksft_test_result( + ok, "%luK priority-inheritance robust owner-death recovery\n", + ps / 1024); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} + +static pthread_barrier_t exec_barrier; + +static void *exec_sibling(void *arg) +{ + (void)arg; + pthread_barrier_wait(&exec_barrier); + for (;;) + pause(); + return NULL; +} + +static void *exec_nonleader(void *arg) +{ + unsigned long target = (uintptr_t)arg; + char size[32]; + + snprintf(size, sizeof(size), "%lu", target); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, target, 0UL, 0UL, 0UL)) + _exit(30); + pthread_barrier_wait(&exec_barrier); + execl("/init", "/init", "--after-thread-exec", size, NULL); + _exit(31); +} + +static int thread_exec(unsigned long target) +{ + pthread_t siblings[2], executor; + + if (pthread_barrier_init(&exec_barrier, NULL, 4)) + return 32; + for (unsigned int i = 0; i < 2; i++) + if (pthread_create(&siblings[i], NULL, exec_sibling, NULL)) + return 33; + if (pthread_create(&executor, NULL, exec_nonleader, + (void *)(uintptr_t)target)) + return 34; + pthread_barrier_wait(&exec_barrier); + for (;;) + pause(); + return 35; +} + +int main(int argc, char **argv) +{ + bool ok = true; + unsigned long granules[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + unsigned int nr = granules[0] > 16384 ? 3 : 2; + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--thread-exec") && getppid() == 1) + return thread_exec(strtoul(argv[2], NULL, 10)); + if (argc == 3 && !strcmp(argv[1], "--after-thread-exec") && + getppid() == 1) { + unsigned long expected = strtoul(argv[2], NULL, 10); + + return getauxval(AT_PAGESZ) != expected || + prctl(PR_AURORA_GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, + 0UL) != 0; + } + if (argc == 3 && !strcmp(argv[1], "--test") && getppid() == 1) { + if (getauxval(AT_PAGESZ) != strtoul(argv[2], NULL, 10)) + return 42; + return exercise(); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Run only as PID 1 in a disposable VM\n"); + return 2; + } + if (mount("proc", "/proc", "proc", 0, NULL)) + return 3; + for (unsigned int small = 0; small < nr; small++) { + pid_t pid = fork(); + int status = 0; + + if (!pid) { + char size[32]; + + snprintf(size, sizeof(size), "%lu", granules[small]); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, granules[small], + 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--test", size, NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + } + for (unsigned int source = 0; source < nr; source++) { + for (unsigned int target = 0; target < nr; target++) { + pid_t pid = fork(); + int status = 0; + bool passed; + + if (!pid) { + char size[32]; + + snprintf(size, sizeof(size), "%lu", + granules[target]); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + granules[source], 0UL, 0UL, 0UL)) + _exit(36); + execl("/init", "/init", "--thread-exec", size, + NULL); + _exit(37); + } + passed = pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - non-leader exec %luK -> %luK status=%#x\n", + passed ? "ok" : "not ok", + granules[source] / 1024, granules[target] / 1024, + status); + ok &= passed; + } + } + printf("%s - thread fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/uffd-move-init.c b/tools/testing/selftests/arm64/user4k/uffd-move-init.c new file mode 100644 index 00000000000000..f2649698488b6c --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/uffd-move-init.c @@ -0,0 +1,451 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - MOVE line=%d errno=%d (%s)\n", \ + __LINE__, errno, strerror(errno)); \ + exit(1); \ + } \ + } while (0) +#define PRESENT (1ULL << 63) +#define SWAPPED (1ULL << 62) +#define VALUE_MASK ((1ULL << 55) - 1) +static uint64_t entry(void *p, size_t ps) +{ + uint64_t e; + int fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + CHECK(fd >= 0 && pread(fd, &e, 8, (uintptr_t)p / ps * 8) == 8); + CHECK(!close(fd)); + return e; +} +static uint64_t physical(void *p, size_t ps) +{ + uint64_t e = entry(p, ps); + CHECK(e & PRESENT); + return (e & VALUE_MASK) * ps; +} +static unsigned char pattern(size_t i) +{ + return 1 + ((i ^ (i >> 12) ^ (i >> 16)) % 251); +} +static void bytes(unsigned char *p, size_t original, size_t len) +{ + for (size_t i = 0; i < len; i++) + CHECK(p[i] == pattern(original + i)); +} +static long move(int fd, void *dst, void *src, size_t len, unsigned mode, + int expected_errno) +{ + struct uffdio_move m = { .dst = (uintptr_t)dst, + .src = (uintptr_t)src, + .len = len, + .mode = mode }; + errno = 0; + int r = ioctl(fd, UFFDIO_MOVE, &m); + if (expected_errno) + CHECK(r == -1 && errno == expected_errno); + else + CHECK(r == 0 && m.move == (long long)len); + return m.move; +} +static void on_cpu(unsigned cpu) +{ + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + CHECK(!sched_setaffinity(0, sizeof(set), &set)); +} +struct tlb_reader { + unsigned char *src, *dst; + atomic_int ready, done; +}; +static void *read_after_move(void *arg) +{ + struct tlb_reader *r = arg; + on_cpu(1); + unsigned char old = *(volatile unsigned char *)r->src; + atomic_store(&r->ready, 1); + while (!atomic_load(&r->done)) + sched_yield(); + CHECK(*(volatile unsigned char *)r->src == 0 && + *(volatile unsigned char *)r->dst == old); + return NULL; +} +struct pin_racer { + void *a, *b; + size_t ps; + atomic_int stop; + atomic_uint accepted, rejected; +}; +static void *race_pins(void *arg) +{ + struct pin_racer *r = arg; + struct io_uring_params p = { 0 }; + on_cpu(1); + int fd = syscall(SYS_io_uring_setup, 2, &p); + CHECK(fd >= 0); + unsigned round = 0; + while (!atomic_load(&r->stop)) { + struct iovec iov = { .iov_base = (round++ & 1) ? r->a : r->b, + .iov_len = r->ps }; + int ret = syscall(SYS_io_uring_register, fd, + IORING_REGISTER_BUFFERS, &iov, 1); + if (!ret) { + atomic_fetch_add(&r->accepted, 1); + usleep(50); + CHECK(!syscall(SYS_io_uring_register, fd, + IORING_UNREGISTER_BUFFERS, NULL, 0)); + } else { + CHECK(errno == EFAULT || errno == EAGAIN); + atomic_fetch_add(&r->rejected, 1); + } + } + CHECK(!close(fd)); + return NULL; +} +static void concurrent_move(size_t ps) +{ + unsigned char *a = + mmap((void *)0x50000000, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + unsigned char *b = + mmap((void *)0x60000000, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(a != MAP_FAILED && b != MAP_FAILED); + memset(a, 0x6d, ps); + uint64_t phys = physical(a, ps); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MOVE }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)a, ps }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + reg.range.start = (uintptr_t)b; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + struct pin_racer race = { .a = a, .b = b, .ps = ps }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, race_pins, &race)); + while (!atomic_load(&race.accepted)) + sched_yield(); + unsigned char *cur = a, *next = b; + unsigned moved = 0, busy = 0; + for (unsigned tries = 0; moved < 256 && tries < 100000; tries++) { + struct uffdio_move m = { .src = (uintptr_t)cur, + .dst = (uintptr_t)next, + .len = ps }; + int ret = ioctl(fd, UFFDIO_MOVE, &m); + if (ret) { + CHECK(errno == EBUSY && m.move == -EBUSY); + busy++; + sched_yield(); + continue; + } + CHECK(m.move == (long long)ps && physical(next, ps) == phys && + !(entry(cur, ps) & (PRESENT | SWAPPED))); + unsigned char *tmp = cur; + cur = next; + next = tmp; + moved++; + } + atomic_store(&race.stop, 1); + CHECK(!pthread_join(thread, NULL)); + CHECK(moved == 256 && atomic_load(&race.accepted) > 0); + for (size_t i = 0; i < ps; i++) + CHECK(cur[i] == 0x6d); + printf("ok - MOVE %zuK CPU0/CPU1 race moves=%u busy=%u pins=%u holes=%u unchanged PFN/data\n", + ps / 1024, moved, busy, atomic_load(&race.accepted), + atomic_load(&race.rejected)); + CHECK(!close(fd) && !munmap(a, ps) && !munmap(b, ps)); +} + +static void allocation_failure(size_t ps, size_t native) +{ + if (ps == native) + return; + unsigned char *a = + mmap((void *)0x70000000, 2 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + unsigned char *b = + mmap((void *)0x71000000, 2 * native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(a != MAP_FAILED && b != MAP_FAILED); + memset(a, 0x35, 2 * native); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MOVE }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)a, 2 * native }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + reg.range.start = (uintptr_t)b; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + /* Prepare both roots and target tables using a different native folio. */ + move(fd, b + native, a + native, ps, 0, 0); + uint64_t phys = physical(a, ps); + int inject = open("/proc/self/fail-nth", O_RDWR | O_CLOEXEC); + CHECK(inject >= 0); + unsigned char *src = a, *dst = b; + for (unsigned phase = 0; phase < 2; phase++) { + struct uffdio_move m = { .src = (uintptr_t)src, + .dst = (uintptr_t)dst, + .len = ps }; + char pending[16] = { 0 }; + CHECK(write(inject, "1", 1) == 1); + errno = 0; + int ret = ioctl(fd, UFFDIO_MOVE, &m), saved = errno; + CHECK(pread(inject, pending, sizeof(pending) - 1, 0) > 0); + CHECK(write(inject, "0", 1) == 1); + CHECK(ret == -1 && saved == ENOMEM && m.move == -ENOMEM && + strtoul(pending, NULL, 10) == 0); + CHECK(physical(src, ps) == phys && + !(entry(dst, ps) & (PRESENT | SWAPPED))); + for (size_t i = 0; i < ps; i++) + CHECK(src[i] == 0x35); + move(fd, dst, src, ps, 0, 0); + CHECK(physical(dst, ps) == phys); + unsigned char *tmp = src; + src = dst; + dst = tmp; + } + printf("ok - MOVE %zuK sidecar/retirement allocation failures preserve PTEs and retry\n", + ps / 1024); + CHECK(!close(inject) && !close(fd) && !munmap(a, 2 * native) && + !munmap(b, 2 * native)); +} + +static int exercise(size_t native) +{ + size_t ps = getauxval(AT_PAGESZ), len = 16 * native; + on_cpu(0); + unsigned char *src = + mmap((void *)0x20000000, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + unsigned char *dst = + mmap((void *)0x30000000, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(src != MAP_FAILED && dst != MAP_FAILED); + for (size_t i = 0; i < len; i++) + src[i] = pattern(i); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MOVE }; + CHECK(!ioctl(fd, UFFDIO_API, &api) && + (api.features & UFFD_FEATURE_MOVE)); + struct uffdio_register reg = { .range = { (uintptr_t)dst, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®) && + (reg.ioctls & (1ULL << _UFFDIO_MOVE))); + uint64_t pfns[256]; + CHECK(native / ps <= sizeof(pfns) / sizeof(pfns[0])); + for (size_t i = 0; i < native / ps; i++) + pfns[i] = physical(src + i * ps, ps); + uint64_t p = physical(src + 3 * ps, ps); + struct tlb_reader reader = { .src = src + 3 * ps, .dst = dst + ps }; + pthread_t tlb_thread; + CHECK(!pthread_create(&tlb_thread, NULL, read_after_move, &reader)); + while (!atomic_load(&reader.ready)) + sched_yield(); + move(fd, dst + ps, src + 3 * ps, ps, 0, 0); + CHECK(physical(dst + ps, ps) == p && + !(entry(src + 3 * ps, ps) & (PRESENT | SWAPPED))); + atomic_store(&reader.done, 1); + CHECK(!pthread_join(tlb_thread, NULL)); + bytes(dst + ps, 3 * ps, ps); + for (size_t i = 0; i < native / ps; i++) + if (i != 3) { + CHECK(physical(src + i * ps, ps) == pfns[i]); + bytes(src + i * ps, i * ps, ps); + } + CHECK(src[3 * ps] == + 0); /* Source has an independent zero-fault after MOVE. */ + printf("ok - MOVE %zuK exact PFN, divergent offsets, neighbour PFNs/data and source TLB\n", + ps / 1024); + + struct io_uring_params ur = { 0 }; + int ring = syscall(SYS_io_uring_setup, 2, &ur); + CHECK(ring >= 0); + struct iovec pin = { .iov_base = src + 2 * ps, .iov_len = ps }; + CHECK(!syscall(SYS_io_uring_register, ring, IORING_REGISTER_BUFFERS, + &pin, 1)); + p = physical(src + 2 * ps, ps); + CHECK(move(fd, dst + 2 * ps, src + 2 * ps, ps, 0, EBUSY) == -EBUSY); + CHECK(physical(src + 2 * ps, ps) == p && + !(entry(dst + 2 * ps, ps) & (PRESENT | SWAPPED))); + bytes(src + 2 * ps, 2 * ps, ps); + uint64_t neighbour = physical(src + ps, ps); + move(fd, dst + 3 * ps, src + ps, ps, 0, 0); + CHECK(physical(dst + 3 * ps, ps) == neighbour); + bytes(dst + 3 * ps, ps, ps); + CHECK(!syscall(SYS_io_uring_register, ring, IORING_UNREGISTER_BUFFERS, + NULL, 0)); + CHECK(!close(ring)); + move(fd, dst + 2 * ps, src + 2 * ps, ps, 0, 0); + CHECK(physical(dst + 2 * ps, ps) == p); + bytes(dst + 2 * ps, 2 * ps, ps); + printf("ok - MOVE %zuK source PIN rejected atomically, neighbour PIN allowed, unpin retry\n", + ps / 1024); + + CHECK(move(fd, dst + ps, src + 4 * ps, ps, 0, EEXIST) == -EEXIST); + bytes(src + 4 * ps, 4 * ps, ps); + bytes(dst + ps, 3 * ps, ps); + CHECK(!madvise(src + 5 * native, native, MADV_DONTNEED)); + CHECK(*(volatile unsigned char *)(src + 5 * native) == 0); + move(fd, dst + 5 * native, src + 5 * native, ps, 0, 0); + CHECK(dst[5 * native] == 0); + dst[5 * native] = 0x7d; + CHECK(src[5 * native] == 0); + CHECK(!madvise(src + 6 * native, native, MADV_DONTNEED)); + CHECK(move(fd, dst + 6 * native, src + 6 * native, ps, 0, ENOENT) == + -ENOENT); + move(fd, dst + 6 * native, src + 6 * native, ps, + UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES, 0); + CHECK(!(entry(dst + 6 * native, ps) & (PRESENT | SWAPPED))); + CHECK(!madvise(src + 10 * native + ps, ps, MADV_DONTNEED)); + p = physical(src + 10 * native, ps); + CHECK(move(fd, dst + 12 * native, src + 10 * native, 2 * ps, 0, + EAGAIN) == (long)ps); + CHECK(physical(dst + 12 * native, ps) == p); + bytes(dst + 12 * native, 10 * native, ps); + CHECK(!(entry(dst + 12 * native + ps, ps) & (PRESENT | SWAPPED))); + printf("ok - MOVE %zuK collision, zero, holes and partial progress\n", + ps / 1024); + + int pipefd[2]; + CHECK(!pipe(pipefd)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + close(pipefd[1]); + char c; + _exit(read(pipefd[0], &c, 1) == 1 ? 0 : 1); + } + close(pipefd[0]); + CHECK(move(fd, dst + 11 * native, src + 12 * native, ps, 0, EBUSY) == + -EBUSY); + CHECK(write(pipefd[1], "x", 1) == 1); + close(pipefd[1]); + int status; + CHECK(waitpid(child, &status, 0) == child && status == 0); + *(volatile unsigned char *)(src + 12 * native) = pattern(12 * native); + p = physical(src + 12 * native, ps); + move(fd, dst + 11 * native, src + 12 * native, ps, 0, 0); + CHECK(physical(dst + 11 * native, ps) == p); + bytes(dst + 11 * native, 12 * native, ps); + printf("ok - MOVE %zuK fork-shared source rejected, exclusive write/retry\n", + ps / 1024); + + /* Keep fault-in coverage as well as restoring an untouched PTE at swapoff. */ + for (unsigned int unuse = 0; unuse < 2; unuse++) { + size_t off = ps < native ? ps : 0; + size_t base = (unuse ? 13 : 8) * native; + unsigned char *s = src + base + off; + unsigned char *d = dst + base + 2 * ps; + + *(volatile unsigned char *)s = pattern(base + off); + for (unsigned int tries = 0; + !(entry(s, ps) & SWAPPED) && tries < 100; tries++) { + CHECK(!madvise(src + base, native, MADV_PAGEOUT)); + usleep(20000); + } + uint64_t old = entry(s, ps); + + CHECK(old & SWAPPED); + move(fd, d, s, ps, 0, 0); + uint64_t now = entry(d, ps); + + CHECK((now & SWAPPED) && ((now & VALUE_MASK) == (old & VALUE_MASK)) && + !(entry(s, ps) & (PRESENT | SWAPPED))); + if (unuse) { + /* Do not fault the moved slot in before swapoff restores it. */ + CHECK(!swapoff("/dev/vda")); + CHECK(entry(d, ps) & PRESENT); + } + bytes(d, base + off, ps); + if (unuse) + CHECK(!swapon("/dev/vda", 0)); + printf("ok - MOVE %zuK %s restores transferred swap PTE at new offset\n", + ps / 1024, unuse ? "swapoff" : "fault"); + } + CHECK(!close(fd)); + CHECK(!munmap(src, len)); + CHECK(!munmap(dst, len)); + concurrent_move(ps); + allocation_failure(ps, native); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + CHECK(!mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)); + struct stat st; + for (unsigned i = 0; i < 100 && stat("/dev/vda", &st); i++) + usleep(20000); + CHECK(!stat("/dev/vda", &st) && S_ISBLK(st.st_mode) && + !swapon("/dev/vda", 0)); + size_t native = getauxval(AT_PAGESZ), sizes[] = { native, 4096, 16384 }; + int failed = 0; + for (unsigned i = 0; i < (native > 16384 ? 3U : 2U); i++) { + pid_t c = fork(); + int status = 0; + CHECK(c >= 0); + if (!c) { + char n[32]; + snprintf(n, sizeof(n), "%zu", native); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", n, NULL); + _exit(5); + } + CHECK(waitpid(c, &status, 0) == c); + printf("%s - MOVE ABI %zuK status=%#x\n", + status ? "not ok" : "ok", sizes[i] / 1024, status); + failed += !!status; + } + if (swapoff("/dev/vda")) + failed++; + printf("UFFD MOVE CONTRACT %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/uffd-wp-init.c b/tools/testing/selftests/arm64/user4k/uffd-wp-init.c new file mode 100644 index 00000000000000..f34ab8328721d0 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/uffd-wp-init.c @@ -0,0 +1,917 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define CHECK(x) \ + do { \ + if (!(x)) { \ + printf("not ok - uffd line=%d errno=%d\n", __LINE__, \ + errno); \ + exit(1); \ + } \ + } while (0) +struct writer { + volatile unsigned char *address; + unsigned char value; + atomic_int done, tid; +}; +static void *write_byte(void *opaque) +{ + struct writer *w = opaque; + atomic_store(&w->tid, syscall(SYS_gettid)); + *w->address = w->value; + atomic_store(&w->done, 1); + return NULL; +} +struct forker { + unsigned char *base; + size_t page_size; + pid_t pid; +}; +static void *fork_byte(void *opaque) +{ + struct forker *f = opaque; + f->pid = fork(); + if (!f->pid) { + alarm(10); + f->base[f->page_size + 17] = 0xa5; + _exit(f->base[f->page_size + 17] != 0xa5); + } + return NULL; +} +static struct uffd_msg read_event(int fd) +{ + struct pollfd p = { .fd = fd, .events = POLLIN }; + struct uffd_msg msg; + CHECK(poll(&p, 1, 10000) == 1 && (p.revents & POLLIN)); + CHECK(read(fd, &msg, sizeof(msg)) == sizeof(msg)); + return msg; +} +static void protect(int fd, void *start, size_t len, unsigned long mode) +{ + struct uffdio_writeprotect wp = { .range = { (uintptr_t)start, len }, + .mode = mode }; + CHECK(!ioctl(fd, UFFDIO_WRITEPROTECT, &wp)); +} +static void resolve_write(int fd, unsigned char *base, size_t ps, unsigned page, + unsigned char value, int exact) +{ + struct writer w = { .address = base + page * ps + 17, .value = value }; + struct pollfd pollfd = { .fd = fd, .events = POLLIN }; + struct uffd_msg msg; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, write_byte, &w)); + CHECK(poll(&pollfd, 1, 10000) == 1 && (pollfd.revents & POLLIN)); + CHECK(read(fd, &msg, sizeof(msg)) == sizeof(msg)); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT); + CHECK(msg.arg.pagefault.flags == + (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_WRITE)); + CHECK(msg.arg.pagefault.address == + (uintptr_t)(base + page * ps + (exact ? 17 : 0))); + CHECK(msg.arg.pagefault.feat.ptid == (unsigned int)atomic_load(&w.tid)); + CHECK(!atomic_load(&w.done)); + protect(fd, base + page * ps, ps, UFFDIO_WRITEPROTECT_MODE_DONTWAKE); + struct uffdio_range range = { (uintptr_t)(base + page * ps), ps }; + CHECK(!ioctl(fd, UFFDIO_WAKE, &range)); + CHECK(!pthread_join(thread, NULL) && atomic_load(&w.done)); + CHECK(base[page * ps + 17] == value); + CHECK(poll(&pollfd, 1, 0) == 0); +} +static void fill(int fd, void *dst, void *src, size_t len, unsigned long mode) +{ + struct uffdio_copy copy = { .dst = (uintptr_t)dst, + .src = (uintptr_t)src, + .len = len, + .mode = mode }; + CHECK(!ioctl(fd, UFFDIO_COPY, ©) && copy.copy == (long long)len); +} +static void zero(int fd, void *dst, size_t len) +{ + struct uffdio_zeropage z = { .range = { (uintptr_t)dst, len } }; + CHECK(!ioctl(fd, UFFDIO_ZEROPAGE, &z) && z.zeropage == (long long)len); +} +static uint64_t physical(void *address, size_t ps) +{ + uint64_t entry; + int fd = open("/proc/self/pagemap", O_RDONLY); + CHECK(fd >= 0 && + pread(fd, &entry, 8, (uintptr_t)address / ps * 8) == 8); + CHECK(!close(fd) && (entry & (1ULL << 63))); + return (entry & ((1ULL << 55) - 1)) * ps; +} +static void missing(size_t ps, size_t native) +{ + size_t len = 4 * native; + unsigned char *map = mmap(NULL, len + native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + unsigned char *src = mmap(NULL, len + native, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(map != MAP_FAILED && src != MAP_FAILED); + unsigned char *p = map + ps; + CHECK(!mlock2(p, len, MLOCK_ONFAULT)); + for (size_t i = 0; i < len; i++) + src[i] = (i / ps) ^ (i * 7 + 3); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_PAGEFAULT_FLAG_WP }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING | + UFFDIO_REGISTER_MODE_WP }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + CHECK((reg.ioctls & + ((1ULL << _UFFDIO_COPY) | (1ULL << _UFFDIO_ZEROPAGE))) == + ((1ULL << _UFFDIO_COPY) | (1ULL << _UFFDIO_ZEROPAGE))); + /* A write blocks only on its missing leaf; COPY_MODE_WP causes a second event. */ + struct writer w = { .address = p + 17, .value = 0xd3 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, write_byte, &w)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)p && + msg.arg.pagefault.flags == UFFD_PAGEFAULT_FLAG_WRITE && + !atomic_load(&w.done)); + fill(fd, p, src, ps, UFFDIO_COPY_MODE_WP); + msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)p && + msg.arg.pagefault.flags == + (UFFD_PAGEFAULT_FLAG_WRITE | UFFD_PAGEFAULT_FLAG_WP)); + protect(fd, p, ps, 0); + CHECK(!pthread_join(thread, NULL) && atomic_load(&w.done)); + src[17] = w.value; + CHECK(!memcmp(p, src, ps)); + zero(fd, p + ps, ps); + for (size_t i = 0; i < ps; i++) + CHECK(p[ps + i] == 0); + p[ps + 17] = 0xe4; + CHECK(!memcmp(p, src, ps)); + /* A batch starts at a user-page boundary, not necessarily a native one. */ + fill(fd, p + 2 * ps, src + 2 * ps, len - 2 * ps, 0); + CHECK(!memcmp(p + 2 * ps, src + 2 * ps, len - 2 * ps)); + if (ps < native) { + unsigned char *a = + (void *)(((uintptr_t)(p + 2 * ps) + native - 1) & + ~(native - 1)); + uint64_t start = physical(a, ps); + CHECK(start != 0 && !(start & (native - 1))); + for (size_t i = ps; i < native; i += ps) + CHECK(physical(a + i, ps) == start + i); + } + struct uffdio_copy copy = { .dst = (uintptr_t)p, + .src = (uintptr_t)src, + .len = ps }; + CHECK(ioctl(fd, UFFDIO_COPY, ©) == -1 && errno == EEXIST && + copy.copy == -EEXIST); + CHECK(!munlock(p, len)); + /* Preserve the successfully copied prefix when the next PTE already exists. */ + CHECK(!madvise(p, ps, MADV_DONTNEED)); + copy = (struct uffdio_copy){ .dst = (uintptr_t)p, + .src = (uintptr_t)src, + .len = 2 * ps }; + CHECK(ioctl(fd, UFFDIO_COPY, ©) == -1 && errno == EAGAIN && + copy.copy == (long long)ps); + CHECK(!memcmp(p, src, ps) && p[ps + 17] == 0xe4); + /* An invalid source stops at its leaf, without installing it or touching neighbours. */ + CHECK(!madvise(p + 2 * ps, 2 * ps, MADV_DONTNEED)); + CHECK(!mprotect(src + 3 * ps, ps, PROT_NONE)); + copy = (struct uffdio_copy){ .dst = (uintptr_t)(p + 2 * ps), + .src = (uintptr_t)(src + 2 * ps), + .len = 2 * ps }; + CHECK(ioctl(fd, UFFDIO_COPY, ©) == -1 && errno == EAGAIN && + copy.copy == (long long)ps); + CHECK(!memcmp(p + 2 * ps, src + 2 * ps, ps)); + CHECK(!mprotect(src + 3 * ps, ps, PROT_READ | PROT_WRITE)); + /* A fresh source forces the unlocked copy retry. */ + CHECK(!madvise(src + 3 * ps, ps, MADV_DONTNEED)); + fill(fd, p + 3 * ps, src + 3 * ps, ps, 0); + for (size_t i = 0; i < ps; i++) + CHECK(p[3 * ps + i] == 0); + CHECK(!memcmp(p + 4 * ps, src + 4 * ps, len - 4 * ps)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + p[2 * ps + 9] ^= 0xff; + _exit(p[2 * ps + 9] == src[2 * ps + 9]); + } + int status; + CHECK(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(!memcmp(p + 2 * ps, src + 2 * ps, ps)); + CHECK(!close(fd)); + CHECK(!munmap(map, len + native) && !munmap(src, len + native)); + printf("ok - uffd ABI %zuK missing copy/WP, zero, packed batch, partial EEXIST/EFAULT, source retry and COW\n", + ps / 1024); +} + +struct reader { + volatile unsigned char *address; + unsigned char value; + atomic_int done; +}; +static void *read_byte(void *opaque) +{ + struct reader *r = opaque; + r->value = *r->address; + atomic_store(&r->done, 1); + return NULL; +} +static void resume_minor(int fd, void *start, size_t len, unsigned long mode) +{ + struct uffdio_continue c = { .range = { (uintptr_t)start, len }, + .mode = mode }; + CHECK(!ioctl(fd, UFFDIO_CONTINUE, &c) && c.mapped == (long long)len); +} +static void file_minor(size_t ps, size_t native) +{ + for (int shared = 0; shared < 2; shared++) { + int memfd = memfd_create("uffd-minor-granule", 0); + CHECK(memfd >= 0 && !ftruncate(memfd, 4 * native)); + unsigned char *alias = mmap(NULL, 4 * native, + PROT_READ | PROT_WRITE, MAP_SHARED, + memfd, 0); + unsigned char *p = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + shared ? MAP_SHARED : MAP_PRIVATE, + memfd, ps); + CHECK(alias != MAP_FAILED && p != MAP_FAILED); + CHECK(!mlock2(p, 3 * ps, MLOCK_ONFAULT)); + for (size_t i = 0; i < 4 * native; i++) + alias[i] = (i / ps) ^ (i * 3 + 7); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { + .api = UFFD_API, + .features = UFFD_FEATURE_MINOR_SHMEM | + UFFD_FEATURE_PAGEFAULT_FLAG_WP + }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { + .range = { (uintptr_t)p, 3 * ps }, + .mode = UFFDIO_REGISTER_MODE_MINOR | + UFFDIO_REGISTER_MODE_WP + }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + CHECK(reg.ioctls & (1ULL << _UFFDIO_CONTINUE)); + + for (unsigned i = 0; i < 3; i++) { + struct reader r = { .address = p + i * ps + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == + (uintptr_t)(p + i * ps) && + msg.arg.pagefault.flags == + UFFD_PAGEFAULT_FLAG_MINOR && + !atomic_load(&r.done)); + alias[(i + 1) * ps + 17] ^= 0x3c; + resume_minor(fd, p + i * ps, ps, + UFFDIO_CONTINUE_MODE_WP | + UFFDIO_CONTINUE_MODE_DONTWAKE); + struct uffdio_range range = { (uintptr_t)(p + i * ps), + ps }; + CHECK(!ioctl(fd, UFFDIO_WAKE, &range)); + CHECK(!pthread_join(thread, NULL) && + atomic_load(&r.done)); + CHECK(r.value == alias[(i + 1) * ps + 17]); + CHECK(physical(p + i * ps, ps) == + physical(alias + (i + 1) * ps, ps)); + /* Independent write protection survives installing the file leaf. */ + struct writer w = { .address = p + i * ps + 17, + .value = 0xa1 + i }; + CHECK(!pthread_create(&thread, NULL, write_byte, &w)); + msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == + (uintptr_t)(p + i * ps) && + msg.arg.pagefault.flags == + (UFFD_PAGEFAULT_FLAG_WP | + UFFD_PAGEFAULT_FLAG_WRITE)); + protect(fd, p + i * ps, ps, 0); + CHECK(!pthread_join(thread, NULL) && + atomic_load(&w.done)); + CHECK(p[i * ps + 17] == w.value); + CHECK(shared ? alias[(i + 1) * ps + 17] == w.value : + alias[(i + 1) * ps + 17] == r.value); + } + struct pollfd pollfd = { .fd = fd, .events = POLLIN }; + CHECK(poll(&pollfd, 1, 0) == 0); + CHECK(!munlock(p, 3 * ps)); + CHECK(!madvise(p, 3 * ps, MADV_DONTNEED)); + resume_minor(fd, p, 3 * ps, 0); + CHECK(!memcmp(p, alias + ps, 3 * ps)); + CHECK(!madvise(p, 3 * ps, MADV_DONTNEED)); + CHECK(!ftruncate(memfd, ps + 17)); + resume_minor(fd, p, ps, 0); + struct uffdio_continue c = { .range = { (uintptr_t)(p + ps), + ps } }; + CHECK(ioctl(fd, UFFDIO_CONTINUE, &c) == -1 && errno == EFAULT && + c.mapped == -EFAULT); + CHECK(!close(fd) && !munmap(p, 3 * ps) && + !munmap(alias, 4 * native) && !close(memfd)); + printf("ok - uffd ABI %zuK shmem %s minor/WP, byte offsets, alias coherence, COW, batch continue and EOF\n", + ps / 1024, shared ? "shared" : "private"); + } +} + +static uint64_t pagemap(void *address, size_t ps) +{ + uint64_t entry; + int fd = open("/proc/self/pagemap", O_RDONLY); + CHECK(fd >= 0 && + pread(fd, &entry, 8, (uintptr_t)address / ps * 8) == 8); + CHECK(!close(fd)); + return entry; +} +static sigjmp_buf bus_env; +static void *bus_address; +static void bus_handler(int sig, siginfo_t *info, void *context) +{ + (void)context; + if (sig != SIGBUS) + _exit(91); + bus_address = info->si_addr; + siglongjmp(bus_env, 1); +} +static void markers(size_t ps) +{ + for (int async = 0; async < 2; async++) { + unsigned char *p = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { + .api = UFFD_API, + .features = UFFD_FEATURE_PAGEFAULT_FLAG_WP | + UFFD_FEATURE_THREAD_ID | + UFFD_FEATURE_WP_UNPOPULATED | + (async ? UFFD_FEATURE_WP_ASYNC : 0) + }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { + .range = { (uintptr_t)p, 3 * ps }, + .mode = UFFDIO_REGISTER_MODE_WP + }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + protect(fd, p, 3 * ps, UFFDIO_WRITEPROTECT_MODE_WP); + for (unsigned i = 0; i < 3; i++) + CHECK(pagemap(p + i * ps, ps) & (1ULL << 57)); + CHECK(*(volatile unsigned char *)(p + ps + 17) == 0); + if (async) + p[ps + 17] = 0x67; + else + resolve_write(fd, p, ps, 1, 0x67, 0); + CHECK(p[ps + 17] == 0x67); + CHECK(pagemap(p, ps) & (1ULL << 57)); + CHECK(!(pagemap(p + ps, ps) & (1ULL << 57))); + CHECK(pagemap(p + 2 * ps, ps) & (1ULL << 57)); + struct pollfd pollfd = { .fd = fd, .events = POLLIN }; + CHECK(poll(&pollfd, 1, 0) == 0); + CHECK(!close(fd)); + p[17] = 0x68; + p[2 * ps + 17] = 0x69; + CHECK(!munmap(p, 3 * ps)); + printf("ok - uffd ABI %zuK unpopulated WP async=%d per-leaf pagemap state and close\n", + ps / 1024, async); + } + unsigned char *p = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_POISON }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, 3 * ps }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®) && + (reg.ioctls & (1ULL << _UFFDIO_POISON))); + zero(fd, p, ps); + zero(fd, p + 2 * ps, ps); + struct uffdio_poison poison = { .range = { (uintptr_t)(p + ps), ps } }; + CHECK(!ioctl(fd, UFFDIO_POISON, &poison) && + poison.updated == (long long)ps); + CHECK(ioctl(fd, UFFDIO_POISON, &poison) == -1 && errno == EEXIST && + poison.updated == -EEXIST); + struct sigaction sa = { .sa_sigaction = bus_handler, + .sa_flags = SA_SIGINFO }, + old; + CHECK(!sigemptyset(&sa.sa_mask) && !sigaction(SIGBUS, &sa, &old)); + if (!sigsetjmp(bus_env, 1)) { + (void)*(volatile unsigned char *)(p + ps + 17); + CHECK(0); + } + CHECK(bus_address == p + ps + 17); + CHECK(!sigaction(SIGBUS, &old, NULL)); + p[17] = 0x81; + p[2 * ps + 17] = 0x82; + CHECK(!madvise(p + ps, ps, MADV_FREE)); + CHECK(!close(fd)); + CHECK(p[ps + 17] == 0); + CHECK(p[17] == 0x81 && p[2 * ps + 17] == 0x82); + CHECK(!munmap(p, 3 * ps)); + printf("ok - uffd ABI %zuK poison: SIGBUS, MADV_FREE recovery, neighbours\n", + ps / 1024); +} +struct pager_request { + uintptr_t address; + size_t size; +}; +static void send_pager(int sock, int fd, struct pager_request *request) +{ + union { + struct cmsghdr align; + char bytes[CMSG_SPACE(sizeof(int))]; + } control = { 0 }; + struct iovec iov = { .iov_base = request, .iov_len = sizeof(*request) }; + struct msghdr msg = { .msg_iov = &iov, + .msg_iovlen = 1, + .msg_control = control.bytes, + .msg_controllen = sizeof(control) }; + struct cmsghdr *c = CMSG_FIRSTHDR(&msg); + c->cmsg_level = SOL_SOCKET; + c->cmsg_type = SCM_RIGHTS; + c->cmsg_len = CMSG_LEN(sizeof(int)); + memcpy(CMSG_DATA(c), &fd, sizeof(fd)); + CHECK(sendmsg(sock, &msg, 0) == sizeof(*request)); +} +static unsigned char pager_pattern(size_t i) +{ + return (i / 4096) ^ (i * 11 + 5); +} +static int pager_worker(int sock, size_t expected) +{ + CHECK(getauxval(AT_PAGESZ) == expected); + alarm(30); + struct pager_request request; + union { + struct cmsghdr align; + char bytes[CMSG_SPACE(sizeof(int))]; + } control = { 0 }; + struct iovec iov = { .iov_base = &request, .iov_len = sizeof(request) }; + struct msghdr msg = { .msg_iov = &iov, + .msg_iovlen = 1, + .msg_control = control.bytes, + .msg_controllen = sizeof(control) }; + CHECK(recvmsg(sock, &msg, 0) == sizeof(request)); + struct cmsghdr *c = CMSG_FIRSTHDR(&msg); + CHECK(c && c->cmsg_level == SOL_SOCKET && c->cmsg_type == SCM_RIGHTS && + c->cmsg_len == CMSG_LEN(sizeof(int))); + int fd; + memcpy(&fd, CMSG_DATA(c), sizeof(fd)); + size_t ps = request.size; + unsigned char *source = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(source != MAP_FAILED); + for (size_t i = 0; i < 4 * ps; i++) + source[i] = pager_pattern(i); + for (unsigned i = 0; i < 3; i++) { + struct uffd_msg fault = read_event(fd); + CHECK(fault.event == UFFD_EVENT_PAGEFAULT && + fault.arg.pagefault.address == request.address + i * ps && + !fault.arg.pagefault.flags); + if (i == 1) + zero(fd, (void *)(request.address + ps), ps); + else + fill(fd, (void *)(request.address + i * ps), + source + i * ps, (i == 2 ? 2 : 1) * ps, 0); + } + CHECK(!close(fd) && !close(sock) && !munmap(source, 4 * ps)); + return 0; +} +static void cross_pager(size_t ps, size_t native, int shared) +{ + size_t pager_ps = ps == native ? 4096 : native; + int socks[2]; + CHECK(!socketpair(AF_UNIX, SOCK_SEQPACKET, 0, socks)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + char fd_arg[32], ps_arg[32]; + CHECK(!close(socks[0])); + snprintf(fd_arg, sizeof(fd_arg), "%d", socks[1]); + snprintf(ps_arg, sizeof(ps_arg), "%zu", pager_ps); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, pager_ps, 0UL, 0UL, 0UL)); + execl("/init", "/init", "--pager", fd_arg, ps_arg, NULL); + _exit(3); + } + CHECK(!close(socks[1])); + int memfd = shared ? memfd_create("remote-uffd-shmem", 0) : -1; + if (shared) + CHECK(memfd >= 0 && !ftruncate(memfd, 4 * ps)); + unsigned char *p = mmap( + NULL, 4 * ps, PROT_READ | PROT_WRITE, + shared ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS, memfd, 0); + CHECK(p != MAP_FAILED); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, 4 * ps }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + struct pager_request request = { .address = (uintptr_t)p, .size = ps }; + send_pager(socks[0], fd, &request); + for (size_t i = 0; i < 4 * ps; i++) + CHECK(p[i] == (i >= ps && i < 2 * ps ? 0 : pager_pattern(i))); + int status; + CHECK(waitpid(child, &status, 0) == child && WIFEXITED(status) && + !WEXITSTATUS(status)); + CHECK(!close(fd) && !close(socks[0]) && !munmap(p, 4 * ps)); + if (shared) + CHECK(!close(memfd)); + printf("ok - uffd target %zuK remote pager %zuK shared=%d SCM_RIGHTS copy/zero/batch uses target granule\n", + ps / 1024, pager_ps / 1024, shared); +} + +static void private_missing(size_t ps) +{ + int memfd = memfd_create("uffd-private-hole", 0); + CHECK(memfd >= 0 && !ftruncate(memfd, 4 * ps)); + unsigned char *p = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE, memfd, ps); + unsigned char *src = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED && src != MAP_FAILED); + for (size_t i = 0; i < 3 * ps; i++) + src[i] = pager_pattern(i); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, 3 * ps }, + .mode = UFFDIO_REGISTER_MODE_MISSING }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®) && + (reg.ioctls & (1ULL << _UFFDIO_COPY))); + struct reader r = { .address = p + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)p && + !msg.arg.pagefault.flags); + fill(fd, p, src, 2 * ps, 0); + CHECK(!pthread_join(thread, NULL) && r.value == src[17]); + zero(fd, p + 2 * ps, ps); + CHECK(!memcmp(p, src, 2 * ps)); + unsigned char bytes[256]; + for (size_t off = 0; off < 4 * ps; off += sizeof(bytes)) { + CHECK(pread(memfd, bytes, sizeof(bytes), off) == sizeof(bytes)); + for (unsigned i = 0; i < sizeof(bytes); i++) + CHECK(bytes[i] == 0); + } + CHECK(!madvise(p, ps, MADV_DONTNEED)); + CHECK(!madvise(src, 3 * ps, MADV_DONTNEED)); + fill(fd, p, src, ps, 0); + for (size_t i = 0; i < ps; i++) + CHECK(p[i] == 0); + CHECK(!madvise(p, 3 * ps, MADV_DONTNEED)); + CHECK(!ftruncate(memfd, ps + 17)); + struct uffdio_copy copy = { .dst = (uintptr_t)(p + ps), + .src = (uintptr_t)src, + .len = ps }; + CHECK(ioctl(fd, UFFDIO_COPY, ©) == -1 && errno == EFAULT && + copy.copy == -EFAULT); + fill(fd, p, src, ps, 0); + CHECK(!close(fd) && !close(memfd) && !munmap(p, 3 * ps) && + !munmap(src, 3 * ps)); + printf("ok - uffd ABI %zuK private shmem missing COPY/ZERO, unchanged file bytes, source retry and EOF\n", + ps / 1024); +} + +static int register_shared(void *p, size_t len) +{ + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { .api = UFFD_API, + .features = UFFD_FEATURE_MISSING_SHMEM | + UFFD_FEATURE_MINOR_SHMEM | + UFFD_FEATURE_PAGEFAULT_FLAG_WP }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + struct uffdio_register reg = { .range = { (uintptr_t)p, len }, + .mode = UFFDIO_REGISTER_MODE_MISSING | + UFFDIO_REGISTER_MODE_MINOR | + UFFDIO_REGISTER_MODE_WP }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + CHECK((reg.ioctls & + ((1ULL << _UFFDIO_COPY) | (1ULL << _UFFDIO_ZEROPAGE) | + (1ULL << _UFFDIO_CONTINUE))) == + ((1ULL << _UFFDIO_COPY) | (1ULL << _UFFDIO_ZEROPAGE) | + (1ULL << _UFFDIO_CONTINUE))); + return fd; +} +static void shared_missing(size_t ps, size_t native) +{ + int memfd = memfd_create("uffd-shared-holes", 0); + CHECK(memfd >= 0 && !ftruncate(memfd, 8 * native)); + unsigned char *p = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, memfd, ps); + unsigned char *q = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, memfd, ps); + unsigned char *src = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED && q != MAP_FAILED && src != MAP_FAILED); + for (size_t i = 0; i < 4 * ps; i++) + src[i] = pager_pattern(i); + int fd = register_shared(p, 4 * ps), fd2 = register_shared(q, 4 * ps); + for (unsigned i = 0; i < 2; i++) { + struct reader r = { .address = p + i * ps + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)(p + i * ps) && + !msg.arg.pagefault.flags); + if (!i) + fill(fd, p, src, ps, UFFDIO_COPY_MODE_WP); + else + zero(fd, p + ps, ps); + CHECK(!pthread_join(thread, NULL) && atomic_load(&r.done)); + CHECK(r.value == (i ? 0 : src[17])); + if (!i) { + struct uffdio_continue c = { + .range = { (uintptr_t)(p + ps), ps } + }; + CHECK(ioctl(fd, UFFDIO_CONTINUE, &c) == -1 && + errno == EFAULT && c.mapped == -EFAULT); + } + } + fill(fd, p + 2 * ps, src + 2 * ps, 2 * ps, 0); + CHECK(!memcmp(p, src, ps) && !memcmp(p + 2 * ps, src + 2 * ps, 2 * ps)); + for (size_t i = 0; i < ps; i++) + CHECK(p[ps + i] == 0); + for (unsigned i = 0; i < 4; i++) { + struct reader r = { .address = q + i * ps + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct uffd_msg msg = read_event(fd2); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)(q + i * ps) && + msg.arg.pagefault.flags == UFFD_PAGEFAULT_FLAG_MINOR); + resume_minor(fd2, q + i * ps, ps, 0); + CHECK(!pthread_join(thread, NULL) && r.value == p[i * ps + 17]); + CHECK(physical(p + i * ps, ps) == physical(q + i * ps, ps)); + } + CHECK(!memcmp(p, q, 4 * ps)); + if (ps < native) + CHECK(physical(p + ps, ps) == physical(p, ps) + ps); + struct writer w = { .address = p + 17, .value = 0x9a }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, write_byte, &w)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)p && + msg.arg.pagefault.flags == + (UFFD_PAGEFAULT_FLAG_WRITE | UFFD_PAGEFAULT_FLAG_WP)); + protect(fd, p, ps, 0); + CHECK(!pthread_join(thread, NULL) && p[17] == w.value && + q[17] == w.value); + struct uffdio_copy c = { .dst = (uintptr_t)(p + ps), + .src = (uintptr_t)src, + .len = ps }; + CHECK(ioctl(fd, UFFDIO_COPY, &c) == -1 && errno == EEXIST && + c.copy == -EEXIST); + unsigned char bytes[256]; + CHECK(pread(memfd, bytes, sizeof(bytes), ps) == sizeof(bytes)); + CHECK(!memcmp(bytes, p, sizeof(bytes))); + /* Resolve an untouched source outside VMA locks before taking a cache folio. */ + unsigned char *r = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, memfd, 6 * native); + unsigned char *fresh = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(r != MAP_FAILED && fresh != MAP_FAILED); + int fd3 = register_shared(r, 2 * ps); + fill(fd3, r, fresh, ps, 0); + for (size_t i = 0; i < ps; i++) + CHECK(r[i] == 0); + CHECK(!mprotect(fresh, ps, PROT_NONE)); + c = (struct uffdio_copy){ .dst = (uintptr_t)(r + ps), + .src = (uintptr_t)fresh, + .len = ps }; + CHECK(ioctl(fd3, UFFDIO_COPY, &c) == -1 && errno == EFAULT && + c.copy == -EFAULT); + zero(fd3, r + ps, ps); + for (size_t i = 0; i < ps; i++) + CHECK(r[ps + i] == 0); + CHECK(!close(fd3) && !munmap(r, 2 * ps) && !munmap(fresh, ps)); + CHECK(!ftruncate(memfd, 2 * ps + 17)); + c = (struct uffdio_copy){ .dst = (uintptr_t)(p + 2 * ps), + .src = (uintptr_t)src, + .len = ps }; + CHECK(ioctl(fd, UFFDIO_COPY, &c) == -1 && + errno == (ps == native ? ENOMEM : EFAULT) && + c.copy == -(ps == native ? ENOMEM : EFAULT)); + CHECK(!close(fd) && !close(fd2) && !close(memfd)); + CHECK(!munmap(p, 4 * ps) && !munmap(q, 4 * ps) && !munmap(src, 4 * ps)); + printf("ok - uffd ABI %zuK shared COPY/ZERO, independent missing/minor pagers, CONTINUE holes, packing, WP/coherence and EOF\n", + ps / 1024); +} + +static void shared_buffered_write(size_t ps) +{ + int memfd = memfd_create("uffd-buffered-validity", 0); + CHECK(memfd >= 0 && !ftruncate(memfd, 4 * ps)); + unsigned char *p = mmap(NULL, 4 * ps, PROT_READ | PROT_WRITE, + MAP_SHARED, memfd, 0); + unsigned char *src = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED && src != MAP_FAILED); + memset(src, 0x57, ps); + int fd = register_shared(p, 4 * ps); + fill(fd, p, src, ps, 0); + CHECK(pwrite(memfd, src, 23, ps + 17) == 23); + for (unsigned i = 1; i < 3; i++) { + struct reader r = { .address = p + i * ps + 17 }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, read_byte, &r)); + struct uffd_msg msg = read_event(fd); + CHECK(msg.event == UFFD_EVENT_PAGEFAULT && + msg.arg.pagefault.address == (uintptr_t)(p + i * ps) && + msg.arg.pagefault.flags == + (i == 1 ? UFFD_PAGEFAULT_FLAG_MINOR : 0)); + if (i == 1) + resume_minor(fd, p + ps, ps, 0); + else + zero(fd, p + i * ps, ps); + CHECK(!pthread_join(thread, NULL) && + r.value == (i == 1 ? 0x57 : 0)); + } + for (size_t i = 0; i < ps; i++) + CHECK(p[ps + i] == (i >= 17 && i < 40 ? 0x57 : 0)); + CHECK(!memcmp(p, src, ps)); + CHECK(!close(fd) && !close(memfd) && !munmap(p, 4 * ps) && + !munmap(src, ps)); + printf("ok - uffd ABI %zuK buffered short-range write populates its leaf and leaves neighbours missing\n", + ps / 1024); +} + +static int worker(unsigned long expected, unsigned long native) +{ + size_t ps = getauxval(AT_PAGESZ); + CHECK(ps == expected); + alarm(60); + missing(ps, native); + file_minor(ps, native); + markers(ps); + cross_pager(ps, native, 0); + cross_pager(ps, native, 1); + private_missing(ps); + shared_missing(ps, native); + shared_buffered_write(ps); + for (int exact = 0; exact < 2; exact++) { + unsigned char *p = mmap(NULL, 3 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(p != MAP_FAILED); + for (unsigned i = 0; i < 3; i++) + memset(p + i * ps, 0x40 + i, ps); + int fd = syscall(SYS_userfaultfd, + O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + CHECK(fd >= 0); + struct uffdio_api api = { + .api = UFFD_API, + .features = UFFD_FEATURE_PAGEFAULT_FLAG_WP | + UFFD_FEATURE_THREAD_ID | + (exact ? UFFD_FEATURE_EXACT_ADDRESS | + UFFD_FEATURE_EVENT_FORK : + 0) + }; + CHECK(!ioctl(fd, UFFDIO_API, &api)); + CHECK(api.features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); + struct uffdio_register reg = { + .range = { (uintptr_t)p, 3 * ps }, + .mode = UFFDIO_REGISTER_MODE_WP + }; + CHECK(!ioctl(fd, UFFDIO_REGISTER, ®)); + CHECK(reg.ioctls & (1ULL << _UFFDIO_WRITEPROTECT)); + + for (unsigned i = 0; i < 3; i++) { + protect(fd, p + i * ps, ps, + UFFDIO_WRITEPROTECT_MODE_WP); + /* Neighbours remain directly writable while this page faults. */ + p[((i + 1) % 3) * ps + 31] ^= 1; + resolve_write(fd, p, ps, i, 0x70 + i, exact); + } + protect(fd, p + ps, ps, UFFDIO_WRITEPROTECT_MODE_WP); + pid_t child; + int status; + if (!exact) { + child = fork(); + CHECK(child >= 0); + if (!child) { + alarm(10); + p[ps + 17] = 0xa5; + _exit(p[ps + 17] != 0xa5); + } + } else { + struct forker f = { .base = p, .page_size = ps }; + pthread_t thread; + CHECK(!pthread_create(&thread, NULL, fork_byte, &f)); + struct uffd_msg event = read_event(fd); + CHECK(event.event == UFFD_EVENT_FORK); + int child_fd = event.arg.fork.ufd; + CHECK(child_fd >= 0); + CHECK(!pthread_join(thread, NULL)); + child = f.pid; + CHECK(child > 0); + event = read_event(child_fd); + CHECK(event.event == UFFD_EVENT_PAGEFAULT && + event.arg.pagefault.flags == + (UFFD_PAGEFAULT_FLAG_WP | + UFFD_PAGEFAULT_FLAG_WRITE)); + CHECK(event.arg.pagefault.address == + (uintptr_t)(p + ps + 17)); + CHECK(event.arg.pagefault.feat.ptid == + (unsigned int)child); + protect(child_fd, p + ps, ps, 0); + CHECK(!close(child_fd)); + } + CHECK(waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status)); + CHECK(p[ps + 17] == 0x71); + resolve_write(fd, p, ps, 1, 0x91, exact); + protect(fd, p, 3 * ps, UFFDIO_WRITEPROTECT_MODE_WP); + struct uffdio_range middle = { (uintptr_t)(p + ps), ps }; + CHECK(!ioctl(fd, UFFDIO_UNREGISTER, &middle)); + p[ps + 17] = 0xb2; + resolve_write(fd, p, ps, 0, 0xb1, exact); + CHECK(!close(fd)); + /* Close removes the remaining registration and write protection. */ + p[2 * ps + 17] = 0xb3; + CHECK(p[17] == 0xb1 && p[ps + 17] == 0xb2 && + p[2 * ps + 17] == 0xb3); + CHECK(!munmap(p, 3 * ps)); + printf("ok - uffd ABI %zuK exact=%d events, neighbours, fork delivery/COW, partial unregister and close\n", + ps / 1024, exact); + } + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 4 && !strcmp(argv[1], "--pager")) + return pager_worker(atoi(argv[2]), strtoul(argv[3], NULL, 10)); + if (argc == 4 && !strcmp(argv[1], "--worker")) + return worker(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10)); + if (getpid() != 1 || argc != 1) + return 2; + CHECK(!mount("proc", "/proc", "proc", 0, NULL)); + unsigned long native = getauxval(AT_PAGESZ), + sizes[] = { native, 4096, 16384 }; + unsigned modes = native == 65536 ? 3 : 2; + int failed = 0; + for (unsigned i = 0; i < modes; i++) { + pid_t pid = fork(); + CHECK(pid >= 0); + if (!pid) { + char ps[32], ns[32]; + snprintf(ps, sizeof(ps), "%lu", sizes[i]); + snprintf(ns, sizeof(ns), "%lu", native); + CHECK(!prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, + 0UL)); + execl("/init", "/init", "--worker", ps, ns, NULL); + _exit(3); + } + int status = 0; + int ok = waitpid(pid, &status, 0) == pid && WIFEXITED(status) && + !WEXITSTATUS(status); + printf("%s - uffd ABI %luK status=%#x\n", ok ? "ok" : "not ok", + sizes[i] / 1024, status); + failed |= !ok; + } + printf("UFFD CONTRACT %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/uprobes-init.c b/tools/testing/selftests/arm64/user4k/uprobes-init.c new file mode 100644 index 00000000000000..0c379e4309caf1 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/uprobes-init.c @@ -0,0 +1,318 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define TRACE "/sys/kernel/tracing/" +#define CHECK(c) \ + do { \ + if (!(c)) { \ + printf("FAIL line %d: %s errno=%d\n", __LINE__, #c, \ + errno); \ + return 1; \ + } \ + } while (0) +static int write_file(const char *path, const char *s) +{ + int fd = open(path, O_WRONLY); + if (fd < 0) + return -1; + size_t len = strlen(s); + int ret = write(fd, s, len) == (ssize_t)len ? 0 : -1; + close(fd); + return ret; +} +static int enable(const char *event, int on) +{ + char path[160]; + snprintf(path, sizeof(path), TRACE "events/aurora_up/%s/enable", event); + return write_file(path, on ? "1" : "0"); +} +static uint64_t pagemap(void *p, size_t ps) +{ + uint64_t bits = 0; + int fd = open("/proc/self/pagemap", O_RDONLY); + if (fd >= 0) { + if (pread(fd, &bits, 8, (uintptr_t)p / ps * 8) != 8) + bits = 0; + close(fd); + } + return bits; +} +static unsigned records(const char *event) +{ + FILE *f = fopen(TRACE "trace", "r"); + char *line = NULL; + size_t len = 0; + unsigned count = 0; + char match[96]; + snprintf(match, sizeof(match), "%s:", event); + if (!f) + return 0; + while (getline(&line, &len, f) > 0) + if (strstr(line, match)) + count++; + free(line); + fclose(f); + return count; +} +static int xol_geometry(size_t native) +{ + FILE *f = fopen("/proc/self/maps", "r"); + char *line = NULL; + size_t len = 0; + int ok = 0; + if (!f) + return 0; + while (getline(&line, &len, f) > 0) + if (strstr(line, "[uprobes]")) { + unsigned long a, b; + if (sscanf(line, "%lx-%lx", &a, &b) == 2) { + printf("# XOL %#lx-%#lx native=%zu\n", a, b, + native); + ok = !(a % (unsigned long)native) && + b - a == native; + } + } + free(line); + fclose(f); + return ok; +} +static int exercise(size_t ps, size_t native, unsigned shifted) +{ + CHECK(getauxval(AT_PAGESZ) == ps); + alarm(20); + printf("# start ps=%zu shifted=%u\n", ps, shifted); + int fd = memfd_create("aurora-uprobe-code", 0); + CHECK(fd >= 0); + size_t length = 8 * ps; + unsigned char *blob = malloc(length); + CHECK(blob); + memset(blob, 0x71, length); + size_t code_off = 3 * ps + 64, ctr_off = 5 * ps + 32; + size_t map_off = shifted ? 2 * ps : 0, + map_len = shifted ? 2 * ps : 4 * ps; + uint32_t insns[] = { 0x11001c00, 0xd65f03c0 }; + memcpy(blob + code_off, insns, sizeof(insns)); + memset(blob + ctr_off, 0, 2); + CHECK(write(fd, blob, length) == (ssize_t)length); + unsigned char *code = mmap((void *)(0x20000000 + (shifted ? ps : 0)), + map_len, PROT_READ | PROT_EXEC, + MAP_PRIVATE | MAP_FIXED_NOREPLACE, fd, + map_off); + unsigned char *counter = + mmap((void *)(0x30000000 + 2 * ps), ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_FIXED_NOREPLACE, fd, 5 * ps); + unsigned char *other = mmap((void *)0x40000000, ps, + PROT_READ | PROT_EXEC, + MAP_PRIVATE | MAP_FIXED_NOREPLACE, fd, 0); + CHECK(code != MAP_FAILED && counter != MAP_FAILED && + other != MAP_FAILED); + unsigned long (*target)(unsigned long) = + (void *)(code + code_off - map_off); + volatile unsigned short *ctr = (void *)(counter + 32); + CHECK(*ctr == 0); + CHECK(target(11) == 18); + /* Occupy the default top-of-address-space XOL hint, forcing a new placement. */ + void *guard = mmap((void *)((1UL << 48) - native), ps, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, + -1, 0); + CHECK(guard != MAP_FAILED); + char entry[64], retname[64], cmd[256]; + snprintf(entry, sizeof(entry), "entry_%zu_%u", ps, shifted); + snprintf(retname, sizeof(retname), "return_%zu_%u", ps, shifted); + snprintf(cmd, sizeof(cmd), + "p:aurora_up/%s /proc/self/fd/%d:0x%zx(0x%zx)\n", entry, fd, + code_off, ctr_off); + CHECK(!write_file(TRACE "uprobe_events", cmd)); + snprintf(cmd, sizeof(cmd), + "r:aurora_up/%s /proc/self/fd/%d:0x%zx(0x%zx)\n", retname, fd, + code_off, ctr_off); + CHECK(!write_file(TRACE "uprobe_events", cmd)); + printf("# enabling entry at line %d\n", __LINE__); + CHECK(!enable(entry, 1)); + printf("# entry enabled\n"); + CHECK(*ctr == 1); + printf("# enabling return\n"); + CHECK(!enable(retname, 1)); + printf("# return enabled\n"); + CHECK(*ctr == 1); + uint32_t disk[2]; + CHECK(pread(fd, disk, sizeof(disk), code_off) == sizeof(disk)); + CHECK(!memcmp(disk, insns, sizeof(insns))); + CHECK(!memcmp(other, blob, ps)); + CHECK(!memcmp(counter, blob + 5 * ps, 32)); + CHECK(!memcmp(counter + 34, blob + 5 * ps + 34, ps - 34)); + int tracefd = open(TRACE "trace", O_WRONLY | O_TRUNC); + CHECK(tracefd >= 0); + close(tracefd); + printf("# invoking probes\n"); + for (unsigned i = 0; i < 16; i++) + CHECK(target(i) == i + 7); + printf("# invocations complete\n"); + CHECK(xol_geometry(native)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + int ok = *ctr == 1; + for (unsigned i = 0; i < 8; i++) + ok &= target(100 + i) == 107 + i; + _exit(ok ? 0 : 3); + } + int status; + CHECK(waitpid(child, &status, 0) == child); + CHECK(WIFEXITED(status) && !WEXITSTATUS(status)); + CHECK(*ctr == 1); + unsigned entries = records(entry), returns = records(retname); + printf("# ABI=%zu entry=%u return=%u\n", ps, entries, returns); + CHECK(entries == 24 && returns == 24); + CHECK(!enable(entry, 0)); + CHECK(*ctr == 1); + CHECK(!enable(retname, 0)); + CHECK(*ctr == 0); + /* Identical restored text is dropped so it can be shared with the file again. */ + CHECK(!(pagemap((void *)target, ps) >> 63)); + CHECK(target(10) == 17); + CHECK(!memcmp(code, blob + map_off, map_len)); + CHECK(!memcmp(other, blob, ps)); + CHECK(!memcmp(counter, blob + 5 * ps, ps)); + /* A pipe GET must prevent dropping the mapped code slot on removal. */ + printf("# enabling entry at line %d\n", __LINE__); + CHECK(!enable(entry, 1)); + printf("# entry enabled\n"); + CHECK(*ctr == 1); + int pipefd[2]; + CHECK(!pipe(pipefd)); + struct iovec iov = { .iov_base = + (void *)((uintptr_t)target & ~(ps - 1)), + .iov_len = ps }; + CHECK(vmsplice(pipefd[1], &iov, 1, 0) == (ssize_t)ps); + CHECK(!enable(entry, 0)); + CHECK(*ctr == 0); + CHECK(pagemap((void *)target, ps) >> 63); + unsigned char *held = malloc(ps); + CHECK(held); + CHECK(read(pipefd[0], held, ps) == (ssize_t)ps); + CHECK(!memcmp(held, blob + (code_off & ~(ps - 1)), ps)); + free(held); + close(pipefd[0]); + close(pipefd[1]); + printf("# enabling entry at line %d\n", __LINE__); + CHECK(!enable(entry, 1)); + printf("# entry enabled\n"); + CHECK(!enable(entry, 0)); + CHECK(*ctr == 0); + /* Native folio-wide references may defer this optional optimization. */ + if (ps < native) + CHECK(!(pagemap((void *)target, ps) >> 63)); + CHECK(target(20) == 27); + snprintf(cmd, sizeof(cmd), "-:aurora_up/%s\n", entry); + CHECK(!write_file(TRACE "uprobe_events", cmd)); + snprintf(cmd, sizeof(cmd), "-:aurora_up/%s\n", retname); + CHECK(!write_file(TRACE "uprobe_events", cmd)); + CHECK(!munmap(code, map_len)); + CHECK(!munmap(guard, ps)); + CHECK(!munmap(counter, ps)); + CHECK(!munmap(other, ps)); + close(fd); + free(blob); + printf("ok - %zuK uprobe/uretprobe, byte offsets, counter, fork, restore and file sharing\n", + ps / 1024); + return 0; +} +static int wait_bounded(pid_t child, int *status) +{ + for (unsigned i = 0; i < 250; i++) { + pid_t got = waitpid(child, status, WNOHANG); + if (got == child) + return 1; + if (got < 0) + return 0; + if (i == 50 || i == 200) { + char path[64], buf[4096]; + snprintf(path, sizeof(path), "/proc/%d/stack", child); + int fd = open(path, O_RDONLY); + ssize_t n = fd < 0 ? -1 : + read(fd, buf, sizeof(buf) - 1); + if (fd >= 0) + close(fd); + if (n > 0) { + buf[n] = 0; + printf("# child %d stack:\n%s", child, buf); + } else + printf("# child %d stack unavailable errno=%d\n", + child, errno); + } + if (i == 200) { + printf("# killing stalled child %d\n", child); + kill(child, SIGKILL); + } + usleep(100000); + } + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10), + strtoul(argv[3], NULL, 10), + strtoul(argv[4], NULL, 10)); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + int fail = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("tracefs", TRACE, "tracefs", 0, NULL)) { + perror("mount"); + fail = 1; + goto done; + } + size_t sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + for (unsigned run = 0; run < (sizes[0] > 16384 ? 6U : 4U); run++) { + unsigned i = run / 2, shifted = run & 1; + pid_t c = fork(); + int status = 0; + if (!c) { + char expected[24], native[24], mode[8]; + snprintf(mode, sizeof(mode), "%u", shifted); + snprintf(expected, sizeof(expected), "%zu", sizes[i]); + snprintf(native, sizeof(native), "%zu", sizes[0]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", expected, native, + mode, NULL); + _exit(5); + } + int ok = c > 0 && wait_bounded(c, &status) && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - ABI %zuK uprobes shifted=%u status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, shifted, status); + fail += !ok; + } +done: + printf("UPROBES %s\n", fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/uring-fixture-init.c b/tools/testing/selftests/arm64/user4k/uring-fixture-init.c new file mode 100644 index 00000000000000..cafc11138d2e8f --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/uring-fixture-init.c @@ -0,0 +1,467 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/prctl.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../kselftest.h" + +struct ring { + int fd; + struct io_uring_params p; + void *sq, *cq, *sqes; + size_t sqlen, cqlen, sqelen; +}; +static unsigned load(unsigned *p) +{ + return __atomic_load_n(p, __ATOMIC_ACQUIRE); +} +static void store(unsigned *p, unsigned v) +{ + __atomic_store_n(p, v, __ATOMIC_RELEASE); +} +static size_t align_size(size_t n) +{ + size_t ps = getauxval(AT_PAGESZ); + return (n + ps - 1) & ~(ps - 1); +} +static void destroy(struct ring *r) +{ + if (r->sq && r->sq != MAP_FAILED) + munmap(r->sq, r->sqlen); + if (r->cq && r->cq != MAP_FAILED && r->cq != r->sq) + munmap(r->cq, r->cqlen); + if (r->sqes && r->sqes != MAP_FAILED) + munmap(r->sqes, r->sqelen); + if (r->fd >= 0) + close(r->fd); + *r = (struct ring){ .fd = -1 }; +} +static bool setup(struct ring *r, unsigned entries, unsigned flags) +{ + *r = (struct ring){ .fd = -1 }; + r->p.flags = flags; + r->fd = syscall(__NR_io_uring_setup, entries, &r->p); + if (r->fd < 0) + return false; + r->sqlen = align_size(r->p.sq_off.array + + r->p.sq_entries * sizeof(unsigned)); + r->cqlen = align_size(r->p.cq_off.cqes + + r->p.cq_entries * sizeof(struct io_uring_cqe) * + (flags & IORING_SETUP_CQE32 ? 2 : 1)); + r->sqelen = align_size(r->p.sq_entries * sizeof(struct io_uring_sqe) * + (flags & IORING_SETUP_SQE128 ? 2 : 1)); + if (r->p.features & IORING_FEAT_SINGLE_MMAP) { + if (r->cqlen > r->sqlen) + r->sqlen = r->cqlen; + r->cqlen = r->sqlen; + } + r->sq = mmap(NULL, r->sqlen, PROT_READ | PROT_WRITE, MAP_SHARED, r->fd, + IORING_OFF_SQ_RING); + r->cq = (r->p.features & IORING_FEAT_SINGLE_MMAP) ? + r->sq : + mmap(NULL, r->cqlen, PROT_READ | PROT_WRITE, MAP_SHARED, + r->fd, IORING_OFF_CQ_RING); + r->sqes = mmap(NULL, r->sqelen, PROT_READ | PROT_WRITE, MAP_SHARED, + r->fd, IORING_OFF_SQES); + return r->sq != MAP_FAILED && r->cq != MAP_FAILED && + r->sqes != MAP_FAILED; +} +static bool request(struct ring *r, unsigned op, int fd, void *buf, + unsigned len, uint64_t off, bool async) +{ + unsigned *sqtail = r->sq + r->p.sq_off.tail, + *sqmask = r->sq + r->p.sq_off.ring_mask; + unsigned *cqhead = r->cq + r->p.cq_off.head, + *cqtail = r->cq + r->p.cq_off.tail, + *cqmask = r->cq + r->p.cq_off.ring_mask; + unsigned tail = load(sqtail), head = load(cqhead), + idx = tail & load(sqmask); + unsigned sqstride = sizeof(struct io_uring_sqe) * + (r->p.flags & IORING_SETUP_SQE128 ? 2 : 1); + unsigned cqstride = sizeof(struct io_uring_cqe) * + (r->p.flags & IORING_SETUP_CQE32 ? 2 : 1); + struct io_uring_sqe *sqe = r->sqes + idx * sqstride; + memset(sqe, 0, sqstride); + sqe->opcode = op; + sqe->fd = fd; + sqe->addr = (uintptr_t)buf; + sqe->len = len; + sqe->off = off; + sqe->user_data = (uint64_t)tail + 1234; + if (async) + sqe->flags = IOSQE_ASYNC; + ((unsigned *)(r->sq + r->p.sq_off.array))[idx] = idx; + store(sqtail, tail + 1); + int ret = syscall(__NR_io_uring_enter, r->fd, 1, 1, + IORING_ENTER_GETEVENTS, NULL, 0); + if (ret != 1 || load(cqtail) != head + 1) { + ksft_print_msg("enter=%d errno=%d\n", ret, errno); + return false; + } + struct io_uring_cqe *cqe = + r->cq + r->p.cq_off.cqes + (head & load(cqmask)) * cqstride; + unsigned expected = op == IORING_OP_NOP ? 0 : len; + if (op == IORING_OP_READV_FIXED || op == IORING_OP_WRITEV_FIXED) { + expected = 0; + for (unsigned i = 0; i < len; i++) + expected += ((struct iovec *)buf)[i].iov_len; + } + bool ok = cqe->user_data == (uint64_t)tail + 1234 && + cqe->res == (int)expected; + if (!ok) + ksft_print_msg("cqe res=%d expected=%u\n", cqe->res, expected); + store(cqhead, head + 1); + return ok; +} +static unsigned long pinned_kb(void) +{ + FILE *f = fopen("/proc/self/status", "r"); + char line[160]; + unsigned long n = ~0UL; + if (!f) + return n; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "VmPin: %lu kB", &n) == 1) + break; + fclose(f); + return n; +} +static bool bytes_are(unsigned char *p, size_t len, unsigned char value) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != value) + return false; + return true; +} +static void fixed_checks(int fd, unsigned char *p, size_t len, size_t ps) +{ + struct ring r = { .fd = -1 }; + unsigned char *q = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + struct iovec reg = { p + 512, len - 1024 }; + bool live = false, ok = setup(&r, 8, 0); + unsigned long baseline = pinned_kb(); + memset(p, 0xa6, len); + if (ok) + live = syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS, ®, 1) == 0; + ok &= live; + ksft_test_result(ok, "register unaligned long-term buffer\n"); + if (ok) { + ok = request(&r, IORING_OP_WRITE_FIXED, fd, p + 512, 7 * 512, 0, + false); + memset(p + 512, 0, 7 * 512); + ok &= request(&r, IORING_OP_READ_FIXED, fd, p + 512, 7 * 512, 0, + false) && + bytes_are(p + 512, 7 * 512, 0xa6); + } + ksft_test_result(ok, "fixed first-quarter slice offsets\n"); + if (ok) { + size_t off = len - 3 * ps; + memset(p + off, 0x49, 2 * ps); + ok = request(&r, IORING_OP_WRITE_FIXED, fd, p + off, 2 * ps, + 65536, true); + memset(p + off, 0, 2 * ps); + ok &= request(&r, IORING_OP_READ_FIXED, fd, p + off, 2 * ps, + 65536, true) && + bytes_are(p + off, 2 * ps, 0x49); + } + ksft_test_result(ok, "fixed late-buffer fast indexing and async I/O\n"); + if (ok) { + struct iovec v[3] = { { p + 512, 512 }, + { p + 2 * ps + 512, ps - 512 }, + { p + 17 * ps + 512, 1024 } }; + memset(p, 0x78, len); + ok = request(&r, IORING_OP_WRITEV_FIXED, fd, v, 3, 131072, + true); + memset(p, 0x11, len); + ok &= request(&r, IORING_OP_READV_FIXED, fd, v, 3, 131072, + true); + for (unsigned i = 0; i < 3; i++) + ok &= bytes_are(v[i].iov_base, v[i].iov_len, 0x78); + ok &= p[0] == 0x11 && p[ps + 512] == 0x11 && p[17 * ps] == 0x11; + } + ksft_test_result(ok, "fixed vectored physical offsets and guards\n"); + if (ok) { + memset(p + 512, 0x3e, len - 1024); + ok = request(&r, IORING_OP_WRITE_FIXED, fd, p + 512, len - 1024, + 4 * 1024 * 1024, true); + memset(p + 512, 0, len - 1024); + ok &= request(&r, IORING_OP_READ_FIXED, fd, p + 512, len - 1024, + 4 * 1024 * 1024, true) && + bytes_are(p + 512, len - 1024, 0x3e); + } + ksft_test_result(ok, "fixed multi-bio transfer\n"); + if (ok) { + ok = mmap(p, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) == p; + memset(p, 0x55, len); + ok &= request(&r, IORING_OP_WRITE_FIXED, fd, p + 512, 3 * ps, + 8 * 1024 * 1024, true); + if (q != MAP_FAILED) { + memset(q, 0, 4 * ps); + ok &= request(&r, IORING_OP_READ, fd, q, 3 * ps, + 8 * 1024 * 1024, false) && + bytes_are(q, 3 * ps, 0x3e) && + bytes_are(p, len, 0x55); + } else + ok = false; + } + ksft_test_result(ok, + "registered source survives unmap and replacement\n"); + if (ok) { + memset(q, 0xc1, 3 * ps); + ok = request(&r, IORING_OP_WRITE, fd, q, 3 * ps, + 10 * 1024 * 1024, false); + ok &= request(&r, IORING_OP_READ_FIXED, fd, p + 512, 3 * ps, + 10 * 1024 * 1024, true); + ok &= request(&r, IORING_OP_WRITE_FIXED, fd, p + 512, 3 * ps, + 12 * 1024 * 1024, true); + memset(q, 0, 3 * ps); + ok &= request(&r, IORING_OP_READ, fd, q, 3 * ps, + 12 * 1024 * 1024, false) && + bytes_are(q, 3 * ps, 0xc1) && bytes_are(p, len, 0x55); + } + ksft_test_result( + ok, + "registered destination survives unmap without touching replacement\n"); + if (live) { + ok = syscall(__NR_io_uring_register, r.fd, + IORING_UNREGISTER_BUFFERS, NULL, 0) == 0; + live = false; + } else + ok = false; + ok &= pinned_kb() == baseline; + if (ok) { + ok = !mprotect(p + ps, ps, PROT_NONE); + struct iovec bad[2] = { { p, ps }, { p + ps, ps } }; + ok &= syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS, bad, 2) == -1 && + errno == EFAULT; + ok &= !mprotect(p + ps, ps, PROT_READ | PROT_WRITE) && + pinned_kb() == baseline; + live = syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS, ®, 1) == 0; + ok &= live; + } + ksft_test_result( + ok, + "failed multi-buffer registration rolls back pins and accounting\n"); + if (live) { + ok &= syscall(__NR_io_uring_register, r.fd, + IORING_UNREGISTER_BUFFERS, NULL, 0) == 0; + live = false; + } + for (unsigned i = 0; i < 64 && ok; i++) { + reg = (struct iovec){ p + ps + 512, 2 * ps }; + ok = !madvise(p + ps, 3 * ps, MADV_DONTNEED); + live = syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS, ®, 1) == 0; + ok &= live; + if (live) { + ok &= syscall(__NR_io_uring_register, r.fd, + IORING_UNREGISTER_BUFFERS, NULL, 0) == 0; + live = false; + } + ok &= pinned_kb() == baseline; + } + ksft_test_result(ok, + "64 register/reuse/unregister cycles balance VmPin\n"); + /* The file offset deliberately differs from the virtual quarter index. */ + int mem = memfd_create("uring-fixed-shared", 0); + ok &= mem >= 0 && !ftruncate(mem, len + 3 * ps) && q != MAP_FAILED; + if (ok) { + ok = mmap(p, len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED, mem, 3 * ps) == p; + memset(p, 0x6b, len); + reg = (struct iovec){ p + 512, len - 1024 }; + live = syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS, ®, 1) == 0; + ok &= live; + struct iovec v[3] = { { p + 512, 512 }, + { p + 3 * ps + 512, ps - 512 }, + { p + 23 * ps + 512, 1024 } }; + if (ok) { + ok = request(&r, IORING_OP_WRITEV_FIXED, fd, v, 3, + 14 * 1024 * 1024, true); + memset(p + 512, 0x18, len - 1024); + ok &= request(&r, IORING_OP_READV_FIXED, fd, v, 3, + 14 * 1024 * 1024, true); + for (unsigned i = 0; i < 3; i++) + ok &= bytes_are(v[i].iov_base, v[i].iov_len, + 0x6b); + ok &= !msync(p, len, MS_SYNC) && + pread(mem, q, ps, 3 * ps) == (ssize_t)ps; + ok &= bytes_are(q, 1024, 0x6b) && q[1024] == 0x18 && + p[len - 1] == 0x6b; + } + } + ksft_test_result( + ok, "registered shared-file quarters use physical offsets\n"); + if (ok) { + struct iovec replacement = { q + 512, 3 * ps }; + struct io_uring_rsrc_update2 update = { + .data = (uintptr_t)&replacement, .nr = 1 + }; + memset(q, 0x9d, 4 * ps); + ok = syscall(__NR_io_uring_register, r.fd, + IORING_REGISTER_BUFFERS_UPDATE, &update, + sizeof(update)) == 1; + ok &= request(&r, IORING_OP_WRITE_FIXED, fd, q + 512, 3 * ps, + 16 * 1024 * 1024, true); + memset(q + 512, 0, 3 * ps); + ok &= request(&r, IORING_OP_READ_FIXED, fd, q + 512, 3 * ps, + 16 * 1024 * 1024, true) && + bytes_are(q + 512, 3 * ps, 0x9d); + } + if (live) { + ok &= syscall(__NR_io_uring_register, r.fd, + IORING_UNREGISTER_BUFFERS, NULL, 0) == 0; + live = false; + } + ok &= pinned_kb() == baseline; + ksft_test_result( + ok, + "registered-buffer update transfers ownership and balances accounting\n"); + if (mem >= 0) + close(mem); + if (live) + syscall(__NR_io_uring_register, r.fd, IORING_UNREGISTER_BUFFERS, + NULL, 0); + destroy(&r); + if (q != MAP_FAILED) + munmap(q, len); +} + +static int exercise(void) +{ + struct ring r = { .fd = -1 }; + size_t ps = getauxval(AT_PAGESZ), len = 2 * 1024 * 1024; + unsigned char *p = mmap(NULL, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + int fd = open("/dev/user4k-test", O_RDWR | O_DIRECT); + bool ok; + ksft_print_header(); + ksft_set_plan(19); + ksft_print_msg("ABI=%zuK\n", ps / 1024); + ok = setup(&r, 4, 0); + ksft_test_result(ok, "minimum rounded ring mappings\n"); + void *bad = ok ? mmap(NULL, ps, PROT_READ | PROT_WRITE, MAP_SHARED, + r.fd, 4096) : + MAP_FAILED; + bool rejected = ok && bad == MAP_FAILED && errno == EINVAL; + if (bad != MAP_FAILED) + munmap(bad, ps); + ksft_test_result(rejected, + "reject sub-native offset in ring selector\n"); + if (ok) { + ok = true; + for (unsigned i = 0; i < 64 && ok; i++) + ok = request(&r, IORING_OP_NOP, -1, NULL, 0, 0, false); + } + ksft_test_result(ok, "small submission/completion ring wrap\n"); + if (p == MAP_FAILED || fd < 0) + ksft_exit_fail_msg("buffer/device setup errno=%d\n", errno); + if (ok) { + memset(p + ps, 0x69, 3 * ps); + ok = request(&r, IORING_OP_WRITE, fd, p + ps, 3 * ps, 0, false); + memset(p + ps, 0, 3 * ps); + ok &= request(&r, IORING_OP_READ, fd, p + ps, 3 * ps, 0, false); + for (size_t i = ps; i < 4 * ps; i++) + ok &= p[i] == 0x69; + } + ksft_test_result(ok, + "direct read/write from unregistered user buffers\n"); + if (ok) { + memset(p + ps, 0x37, 3 * ps); + ok = request(&r, IORING_OP_WRITE, fd, p + ps, 3 * ps, 65536, + true); + memset(p + ps, 0, 3 * ps); + ok &= request(&r, IORING_OP_READ, fd, p + ps, 3 * ps, 65536, + true); + for (size_t i = ps; i < 4 * ps; i++) + ok &= p[i] == 0x37; + } + ksft_test_result(ok, "io-worker forced asynchronous read/write\n"); + if (ok) { + memset(p, 0xb6, len); + ok = request(&r, IORING_OP_WRITE, fd, p, len, 4 * 1024 * 1024, + true); + memset(p, 0, len); + ok &= request(&r, IORING_OP_READ, fd, p, len, 4 * 1024 * 1024, + true); + for (size_t i = 0; i < len; i++) + ok &= p[i] == 0xb6; + } + ksft_test_result(ok, "multi-bio io-worker transfer\n"); + destroy(&r); + ok = setup(&r, 4096, 0); + if (ok) + for (unsigned i = 0; i < 8193 && ok; i++) + ok = request(&r, IORING_OP_NOP, -1, NULL, 0, 0, false); + ksft_test_result(ok, "large ring crosses native pages and wraps\n"); + destroy(&r); + ok = setup(&r, 64, IORING_SETUP_SQE128 | IORING_SETUP_CQE32); + if (ok) + for (unsigned i = 0; i < 129 && ok; i++) + ok = request(&r, IORING_OP_NOP, -1, NULL, 0, 0, false); + ksft_test_result(ok, "128-byte SQEs and 32-byte CQEs\n"); + destroy(&r); + fixed_checks(fd, p, len, ps); + close(fd); + munmap(p, len); + ksft_print_cnts(); + return ksft_get_fail_cnt() ? 1 : 0; +} +int main(int argc, char **argv) +{ + bool ok = true; + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 2 && !strcmp(argv[1], "--test") && getppid() == 1) + return exercise(); + if (getpid() != 1 || argc != 1) + return 2; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) + return 3; + FILE *dev = fopen("/sys/class/block/vda/dev", "r"); + unsigned maj, min; + if (!dev || fscanf(dev, "%u:%u", &maj, &min) != 2) + return 4; + fclose(dev); + if (mknod("/dev/user4k-test", S_IFBLK | 0600, makedev(maj, min))) + return 5; + for (unsigned small = 0; small < 2; small++) { + pid_t pid = fork(); + int status = 0; + if (!pid) { + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + small ? 4096UL : 0UL, 0UL, 0UL, 0UL)) + _exit(3); + execl("/init", "/init", "--test", NULL); + _exit(4); + } + ok &= pid > 0 && waitpid(pid, &status, 0) == pid && + WIFEXITED(status) && !WEXITSTATUS(status); + } + printf("%s - io_uring fixture complete\n", ok ? "ok" : "not ok"); + reboot(RB_POWER_OFF); + return ok ? 0 : 1; +} diff --git a/tools/testing/selftests/arm64/user4k/user-events-init.c b/tools/testing/selftests/arm64/user4k/user-events-init.c new file mode 100644 index 00000000000000..4b35954e2a210a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/user-events-init.c @@ -0,0 +1,268 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define TRACE "/sys/kernel/tracing/" +#define CHECK(c) \ + do { \ + if (!(c)) { \ + printf("FAIL line %d: %s errno=%d\n", __LINE__, #c, \ + errno); \ + return 1; \ + } \ + } while (0) + +static int toggle(const char *name, int enabled) +{ + char path[192]; + snprintf(path, sizeof(path), TRACE "events/user_events/%s/enable", + name); + int fd = open(path, O_WRONLY); + if (fd < 0) + return -1; + int ret = write(fd, enabled ? "1" : "0", 1) == 1 ? 0 : -1; + close(fd); + return ret; +} +static uint64_t value(void *p, unsigned bytes) +{ + return bytes == 4 ? __atomic_load_n((uint32_t *)p, __ATOMIC_ACQUIRE) : + __atomic_load_n((uint64_t *)p, __ATOMIC_ACQUIRE); +} +static int wait_value(void *p, unsigned bytes, uint64_t expected) +{ + for (unsigned i = 0; i < 3000; i++) { + if (value(p, bytes) == expected) + return 1; + usleep(1000); + } + return 0; +} +static int check_bytes(unsigned char *p, size_t size, unsigned char fill, + void *word, unsigned bytes) +{ + for (size_t i = 0; i < size; i++) + if ((!word || (uintptr_t)(p + i) < (uintptr_t)word || + (uintptr_t)(p + i) >= (uintptr_t)word + bytes) && + p[i] != fill) { + printf("byte mismatch at %zu got=%#x expected=%#x\n", i, + p[i], fill); + return 0; + } + return 1; +} +static int event_recorded(const char *name) +{ + char buf[16384]; + size_t n = 0; + int fd = open(TRACE "trace", O_RDONLY); + if (fd < 0) + return 0; + while (n < sizeof(buf) - 1) { + ssize_t got = read(fd, buf + n, sizeof(buf) - 1 - n); + if (got < 0) { + close(fd); + return 0; + } + if (!got) + break; + n += got; + } + close(fd); + buf[n] = 0; + char *record = strstr(buf, name), + *payload = record ? strstr(record, "payload=") : NULL; + int ok = payload && strtoul(payload + 8, NULL, 0) == 0x12345678; + if (!ok) + printf("trace mismatch:\n%s\n", buf); + return ok; +} +static int exercise_case(size_t ps, unsigned bytes, unsigned shared) +{ + const size_t len = 4 * ps; + unsigned char *src, *dst; + int memfd = -1; + if (shared) { + memfd = memfd_create("user-event-bitmap", 0); + CHECK(memfd >= 0); + CHECK(!ftruncate(memfd, len)); + } + src = mmap((void *)0x20000000, len, PROT_READ | PROT_WRITE, + (shared ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS) | + MAP_FIXED_NOREPLACE, + memfd, 0); + dst = mmap((void *)0x30000000, len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + CHECK(src != MAP_FAILED && dst != MAP_FAILED); + for (unsigned i = 0; i < 4; i++) + memset(src + i * ps, 0x61 + i, ps); + memset(dst, 0x91, len); + CHECK(mremap(src + 3 * ps, ps, ps, MREMAP_MAYMOVE | MREMAP_FIXED, + dst + ps) == dst + ps); + unsigned char *leaf = dst + ps; + void *word = leaf + ps - bytes; + uint64_t mask = bytes == 4 ? 1ULL << 31 : 1ULL << 63; + memset(word, 0, bytes); + char name[80], spec[120]; + snprintf(name, sizeof(name), "aurora_ue_%zu_%u_%u", ps, bytes, shared); + snprintf(spec, sizeof(spec), "%s u32 payload", name); + int tracefd = open(TRACE "trace", O_WRONLY | O_TRUNC); + CHECK(tracefd >= 0); + close(tracefd); + int fd = open(TRACE "user_events_data", O_RDWR); + CHECK(fd >= 0); + struct user_reg reg = { .size = sizeof(reg), + .enable_bit = bytes * 8 - 1, + .enable_size = bytes, + .enable_addr = (uintptr_t)word, + .name_args = (uintptr_t)spec }; + CHECK(!ioctl(fd, DIAG_IOCSREG, ®)); + CHECK(wait_value(word, bytes, 0)); + CHECK(!toggle(name, 1)); + CHECK(wait_value(word, bytes, mask)); + struct { + uint32_t index, payload; + } event = { reg.write_index, 0x12345678 }; + CHECK(write(fd, &event, sizeof(event)) == sizeof(event)); + CHECK(event_recorded(name)); + CHECK(!toggle(name, 0)); + CHECK(wait_value(word, bytes, 0)); + /* Every other byte, including the adjacent half of the kernel word, is stable. */ + CHECK(check_bytes(leaf, ps, 0x64, word, bytes)); + CHECK(check_bytes(dst, ps, 0x91, NULL, 0)); + CHECK(check_bytes(dst + 2 * ps, 2 * ps, 0x91, NULL, 0)); + int commands[2], reply[2]; + CHECK(!pipe(commands) && !pipe(reply)); + pid_t child = fork(); + CHECK(child >= 0); + if (!child) { + close(commands[1]); + close(reply[0]); + char token; + int ok = 1; + for (unsigned round = 0; round < 32; round++) { + if (read(commands[0], &token, 1) != 1) { + ok = 0; + break; + } + ok = wait_value(word, bytes, token == '1' ? mask : 0) && + check_bytes(leaf, ps, 0x64, word, bytes); + token = ok ? 'y' : 'n'; + if (write(reply[1], &token, 1) != 1 || !ok) + break; + } + _exit(ok ? 0 : 3); + } + close(commands[0]); + close(reply[1]); + int ok = 1; + for (unsigned round = 0; round < 32; round++) { + int on = !(round & 1); + char token = on ? '1' : '0'; + if (toggle(name, on) || + !wait_value(word, bytes, on ? mask : 0) || + write(commands[1], &token, 1) != 1 || + read(reply[0], &token, 1) != 1 || token != 'y') { + ok = 0; + break; + } + } + close(commands[1]); + close(reply[0]); + int status; + CHECK(waitpid(child, &status, 0) == child); + CHECK(ok && WIFEXITED(status) && WEXITSTATUS(status) == 0); + CHECK(check_bytes(leaf, ps, 0x64, word, bytes)); + for (unsigned i = 0; i < 3; i++) + CHECK(check_bytes(src + i * ps, ps, 0x61 + i, NULL, 0)); + /* Force the asynchronous missing-PTE fixup without unregistering. */ + CHECK(!madvise(leaf, ps, MADV_DONTNEED)); + CHECK(!toggle(name, 1)); + CHECK(wait_value(word, bytes, mask)); + CHECK(check_bytes(leaf, ps, shared ? 0x64 : 0, word, bytes)); + CHECK(!toggle(name, 0)); + CHECK(wait_value(word, bytes, 0)); + struct user_unreg unreg = { .size = sizeof(unreg), + .disable_bit = reg.enable_bit, + .disable_addr = (uintptr_t)word }; + CHECK(!ioctl(fd, DIAG_IOCSUNREG, &unreg)); + close(fd); + CHECK(!munmap(src, 3 * ps)); + CHECK(!munmap(dst, len)); + if (memfd >= 0) + close(memfd); + printf("ok - %zuK %u-bit %s relocated bitmap, adjacent bytes, event write and fork toggles\n", + ps / 1024, bytes * 8, shared ? "shared" : "private"); + return 0; +} +static int exercise(size_t expected) +{ + size_t ps = getauxval(AT_PAGESZ); + CHECK(ps == expected); + for (unsigned shared = 0; shared < 2; shared++) + for (unsigned bytes = 4; bytes <= 8; bytes += 4) + CHECK(!exercise_case(ps, bytes, shared)); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + int fail = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL) || + mount("tracefs", TRACE, "tracefs", 0, NULL)) { + perror("mount"); + fail = 1; + goto done; + } + size_t sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + for (unsigned i = 0; i < (sizes[0] > 16384 ? 3U : 2U); i++) { + pid_t c = fork(); + int status = 0; + if (!c) { + char expected[24]; + snprintf(expected, sizeof(expected), "%zu", sizes[i]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", expected, NULL); + _exit(5); + } + int ok = c > 0 && waitpid(c, &status, 0) == c && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("%s - ABI %zuK user events status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + fail += !ok; + } +done: + printf("USER EVENTS %s\n", fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/user16k-smoke-init.c b/tools/testing/selftests/arm64/user4k/user16k-smoke-init.c new file mode 100644 index 00000000000000..b0a2f2bf385f0a --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/user16k-smoke-init.c @@ -0,0 +1,229 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define SET_EXEC_PAGE_SIZE 0x41555001 +#define GET_EXEC_PAGE_SIZE 0x41555002 +static int failed; +static _Thread_local int tls = 7; +static void result(int ok, const char *name) +{ + printf("%s - %s\n", ok ? "ok" : "not ok", name); + failed += !ok; +} +static long pss(void *p) +{ + FILE *f = fopen("/proc/self/smaps", "r"); + char *line = NULL; + size_t cap = 0; + unsigned long start, end; + int active = 0; + long value = -1; + if (!f) + return -1; + while (getline(&line, &cap, f) > 0) { + if (sscanf(line, "%lx-%lx", &start, &end) == 2) + active = (unsigned long)p >= start && + (unsigned long)p < end; + else if (active && sscanf(line, "Pss: %ld kB", &value) == 1) + break; + } + free(line); + fclose(f); + return value; +} +static void *thread_worker(void *arg) +{ + unsigned char *p = arg; + if (tls != 7) + return (void *)1; + tls = 91; + for (unsigned i = 0; i < 16384; i++) + if (p[i] != 0x42) + return (void *)2; + return NULL; +} +static void memory_checks(void) +{ + const size_t ps = 16384; + int status = 0; + pid_t c; + unsigned char *p = mmap(NULL, 6 * ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + result(p != MAP_FAILED, "16K anonymous mmap"); + if (p == MAP_FAILED) + return; + for (unsigned i = 0; i < 6; i++) + memset(p + i * ps, 0x40 + i, ps); + c = fork(); + if (!c) { + memset(p + ps, 0xb7, ps); + _exit(p[0] == 0x40 && p[2 * ps] == 0x42 ? 0 : 1); + } + int ok = c > 0 && waitpid(c, &status, 0) == c && WIFEXITED(status) && + !WEXITSTATUS(status); + for (unsigned i = 0; i < 6 * ps; i++) + ok &= p[i] == 0x40 + i / ps; + result(ok, "16K fork COW preserves every parent byte"); + result(!mprotect(p + ps, ps, PROT_READ), "16K leaf write protection"); + c = fork(); + if (!c) { + *(volatile unsigned char *)(p + ps + 8192) = 3; + _exit(1); + } + result(c > 0 && waitpid(c, &status, 0) == c && WIFSIGNALED(status) && + WTERMSIG(status) == SIGSEGV, + "write protection covers the interior of a16K leaf"); + result(!mprotect(p + ps, ps, PROT_READ | PROT_WRITE), + "16K write permission restored"); + errno = 0; + result(mprotect(p + 4096, ps, PROT_READ) == -1 && errno == EINVAL, + "16K ABI rejects4K-only protection alignment"); + result(!madvise(p + ps, ps, MADV_DONTNEED), + "16K partial unmap via DONTNEED"); + ok = 1; + for (unsigned i = 0; i < ps; i++) + ok &= p[i] == 0x40 && p[ps + i] == 0 && p[2 * ps + i] == 0x42; + result(ok, "16K discard clears one leaf and preserves neighbours"); + pthread_t thread; + void *answer = (void *)3; + int err = pthread_create(&thread, NULL, thread_worker, p + 2 * ps); + if (!err) + err = pthread_join(thread, &answer); + result(!err && !answer && tls == 7, "16K pthread stack and TLS"); + struct timespec ts; + result(!clock_gettime(CLOCK_MONOTONIC, &ts), "16K vDSO clock access"); + munmap(p, 6 * ps); +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + signal(SIGPIPE, SIG_IGN); + if (argc == 6 && !strcmp(argv[1], "--child")) { + int fd = atoi(argv[2]), ready = atoi(argv[3]), + release = atoi(argv[4]); + size_t native = strtoul(argv[5], NULL, 10); + const size_t ps = 16384; + result(getauxval(AT_PAGESZ) == ps && getpagesize() == ps, + "EL0 selects16K pages"); + result(prctl(GET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL) == 0, + "exec consumes alternative selection"); + if (!failed) + memory_checks(); + { + int status = 0; + pid_t c = fork(); + if (!c) { + if (prctl(SET_EXEC_PAGE_SIZE, 16384UL, 0UL, 0UL, + 0UL)) + _exit(4); + execl("/page-contract", "/page-contract", NULL); + _exit(5); + } + result(c > 0 && waitpid(c, &status, 0) == c && + WIFEXITED(status) && + !WEXITSTATUS(status), + "16K-to16K exec and page/x18 contract"); + } + unsigned char *shared = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, native - ps); + unsigned char *private = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE, fd, native - ps); + int ok = shared != MAP_FAILED && private != MAP_FAILED; + if (ok) { + for (unsigned i = 0; i < ps; i++) + ok &= shared[i] == 0x7b && private[i] == 0x7b; + memset(private, 0xd8, ps); + for (unsigned i = 0; i < ps; i++) + ok &= shared[i] == 0x7b && private[i] == 0xd8; + } + result(ok, "16K file COW copies the whole leaf"); + if (private != MAP_FAILED) + munmap(private, ps); + result(shared != MAP_FAILED && pss(shared) == 8, + "16K file PSS shares all four4K slices"); + if (shared != MAP_FAILED) + shared[ps - 1] = 0xc9; + char token = failed ? 'x' : 'a'; + if (write(ready, &token, 1) != 1 || + read(release, &token, 1) != 1) + return 3; + if (shared != MAP_FAILED) + munmap(shared, ps); + return failed ? 1 : 0; + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + mount("proc", "/proc", "proc", 0, NULL); + size_t native = getauxval(AT_PAGESZ); + int fd = memfd_create("mixed16-smoke", 0); + if (native != 65536 || fd < 0 || ftruncate(fd, native)) { + result(0, "64K parent setup"); + goto done; + } + unsigned char *p = + mmap(NULL, native, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (p == MAP_FAILED) { + result(0, "parent shared mmap"); + goto done; + } + memset(p, 0x7b, native); + result(pss(p) == 64, "native64K initial PSS"); + int ready[2], release[2]; + if (pipe(ready) || pipe(release)) { + result(0, "pipes"); + goto done; + } + pid_t c = fork(); + int status = 0; + if (!c) { + char f[24], r[24], w[24], n[24]; + snprintf(f, sizeof(f), "%d", fd); + snprintf(r, sizeof(r), "%d", ready[1]); + snprintf(w, sizeof(w), "%d", release[0]); + snprintf(n, sizeof(n), "%zu", native); + close(ready[0]); + close(release[1]); + if (prctl(SET_EXEC_PAGE_SIZE, 16384UL, 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--child", f, r, w, n, NULL); + _exit(5); + } + close(ready[1]); + close(release[0]); + char token = 0; + int ok = c > 0 && read(ready[0], &token, 1) == 1 && token == 'a'; + result(ok && p[native - 1] == 0xc9, + "native64K and16K shared bytes remain coherent"); + result(pss(p) == 56, + "native64K PSS accounts one16K alias as four shared slices"); + token = 'b'; + if (write(release[1], &token, 1) != 1) + failed++; + result(c > 0 && waitpid(c, &status, 0) == c && WIFEXITED(status) && + !WEXITSTATUS(status), + "16K child completes"); + printf("# child status=%#x\n", status); + result(pss(p) == 64, "native PSS returns after16K exit"); + munmap(p, native); +done: + printf("USER16K SMOKE %s\n", failed ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/vfio-fixture-init.c b/tools/testing/selftests/arm64/user4k/vfio-fixture-init.c new file mode 100644 index 00000000000000..c43e28ca8b21d1 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/vfio-fixture-init.c @@ -0,0 +1,542 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include "../../../../../include/uapi/linux/prctl.h" +#define __user +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../../../../include/uapi/linux/vfio.h" +#include "../../../../../drivers/vfio/vfio_iommu_type1_test.h" + +#define IOVA (256UL * 1024 * 1024) +static unsigned int number, failures; + +static void result(int ok, const char *name) +{ + printf("%s %u - %s (errno=%d)\n", ok ? "ok" : "not ok", ++number, name, + errno); + failures += !ok; +} + +static int state(int fd, struct vfio_type1_test_cmd *cmd) +{ + memset(cmd, 0, sizeof(*cmd)); + return ioctl(fd, VFIO_TYPE1_TEST_STATE, cmd); +} + +static int domain(int fd, unsigned int which, int add, size_t page, size_t fail) +{ + struct vfio_type1_test_cmd cmd = { + .domain = which, + .size = page, + .fail_after = fail, + }; + + return ioctl(fd, + add ? VFIO_TYPE1_TEST_DOMAIN_ADD : + VFIO_TYPE1_TEST_DOMAIN_DEL, + &cmd); +} + +static int rw(int fd, size_t offset, void *data, size_t size, + unsigned int flags) +{ + struct vfio_type1_test_cmd cmd = { + .iova = IOVA + offset, + .size = size, + .buffer = (uintptr_t)data, + .flags = flags, + }; + + return ioctl(fd, VFIO_TYPE1_TEST_RW, &cmd); +} + +static int map(int fd, void *data, size_t size) +{ + struct vfio_iommu_type1_dma_map cmd = { + .argsz = sizeof(cmd), + .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE, + .iova = IOVA, + .vaddr = (uintptr_t)data, + .size = size, + }; + + return ioctl(fd, VFIO_IOMMU_MAP_DMA, &cmd); +} + +static int unmap(int fd, size_t size) +{ + struct vfio_iommu_type1_dma_unmap cmd = { + .argsz = sizeof(cmd), + .iova = IOVA, + .size = size, + }; + int ret = ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &cmd); + + return ret ? ret : cmd.size == size ? 0 : -2; +} + +static int dirty(int fd, unsigned int flags) +{ + struct vfio_iommu_type1_dirty_bitmap cmd = { .argsz = sizeof(cmd), + .flags = flags }; + + return ioctl(fd, VFIO_IOMMU_DIRTY_PAGES, &cmd); +} + +static int bitmap(int fd, size_t size, uint64_t *bits) +{ + unsigned char + buffer[sizeof(struct vfio_iommu_type1_dirty_bitmap) + + sizeof(struct vfio_iommu_type1_dirty_bitmap_get)] = {}; + struct vfio_iommu_type1_dirty_bitmap *cmd = (void *)buffer; + struct vfio_iommu_type1_dirty_bitmap_get *range = (void *)cmd->data; + + cmd->argsz = sizeof(buffer); + cmd->flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; + range->iova = IOVA; + range->size = size; + range->bitmap.pgsize = 4096; + range->bitmap.size = sizeof(*bits); + range->bitmap.data = (void *)bits; + *bits = 0; + return ioctl(fd, VFIO_IOMMU_DIRTY_PAGES, buffer); +} + +static int all_bytes(const unsigned char *data, size_t size, + unsigned char value) +{ + for (size_t i = 0; i < size; i++) + if (data[i] != value) + return 0; + return 1; +} + +static void external_span(int backing, size_t native) +{ + const size_t fine = 4096, count = native / fine; + uint64_t bits; + + for (unsigned int mapped = 0; mapped < 2; mapped++) { + struct vfio_type1_test_cmd cmd; + struct vfio_iommu_type1_dma_unmap remove = { + .argsz = sizeof(remove), + .iova = IOVA, + .size = fine, + }; + int fd = open("/dev/vfio-test", O_RDWR), status = 0; + pid_t child; + + result(fd >= 0 && (!mapped || !domain(fd, 0, 1, fine, 0)), + "span fixture opens lazy or mapped type1 container"); + if (fd < 0) + continue; + result(!dirty(fd, VFIO_IOMMU_DIRTY_PAGES_FLAG_START), + "span dirty tracking starts before fine mappings"); + child = fork(); + if (!child) { + unsigned char *data = mmap(NULL, native, + PROT_READ | PROT_WRITE, + MAP_SHARED, backing, 0); + + if (data == MAP_FAILED) + _exit(1); + for (size_t i = 0; i < count; i++) { + struct vfio_iommu_type1_dma_map area = { + .argsz = sizeof(area), + .flags = VFIO_DMA_MAP_FLAG_READ | + VFIO_DMA_MAP_FLAG_WRITE, + .iova = IOVA + i * fine, + .vaddr = (uintptr_t)data + i * fine, + .size = fine, + }; + + if (ioctl(fd, VFIO_IOMMU_MAP_DMA, &area)) + _exit(2); + } + cmd = (struct vfio_type1_test_cmd){ .iova = IOVA + 17 }; + _exit(ioctl(fd, VFIO_TYPE1_TEST_PIN, &cmd) ? 3 : 0); + } + result(child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status), + "source pins native page across independent fine MAP ioctls"); + result(!state(fd, &cmd) && !cmd.owner_users && + cmd.external_refs == 1 && + cmd.locked_bytes == count * native && + cmd.owner_locked_bytes == count * native, + "every fine DMA area retains its own charge after source exit"); + cmd = (struct vfio_type1_test_cmd){ .iova = IOVA + 31 }; + result(!ioctl(fd, VFIO_TYPE1_TEST_PIN, &cmd) && + !state(fd, &cmd) && cmd.external_refs == 2 && + cmd.locked_bytes == count * native, + "cross-area duplicate pin reuses all cached constituent pins"); + result(!bitmap(fd, native, &bits) && + bits == ((1ULL << count) - 1), + "cross-area dirty bitmap marks only each mapping's fine extent"); + result(!ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &remove) && + remove.size == fine && !state(fd, &cmd) && + !cmd.external_refs && cmd.notifications == 1 && + cmd.locked_bytes == + (mapped ? (count - 1) * native : 0), + "unmapping one constituent drains references in all page constituents"); + remove.iova = IOVA + fine; + remove.size = native - fine; + result(!ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &remove) && + remove.size == native - fine && + !state(fd, &cmd) && !cmd.locked_bytes && + !cmd.mapped_bytes[0], + "remaining fine mappings unmap without pins or translations"); + close(fd); + } +} + +static void native_owner(int fd, int backing, unsigned char *data, + size_t native) +{ + struct vfio_type1_test_cmd cmd = {}; + struct vfio_iommu_type1_dma_unmap invalidate = { + .argsz = sizeof(invalidate), + .flags = VFIO_DMA_UNMAP_FLAG_VADDR, + .iova = IOVA, + .size = 2 * native, + }; + struct vfio_iommu_type1_dma_map update = { + .argsz = sizeof(update), + .flags = VFIO_DMA_MAP_FLAG_VADDR, + .iova = IOVA, + .size = 2 * native, + .vaddr = (uintptr_t)data, + }; + struct __user_cap_header_struct header = { + .version = _LINUX_CAPABILITY_VERSION_3 + }; + struct __user_cap_data_struct saved[2], reduced[2]; + struct rlimit limit, empty; + unsigned char bytes[64]; + int file_flags = fcntl(backing, F_GETFD), status = 0; + pid_t child; + int budget_ready, budget_captured, transferred; + + result(!ioctl(fd, VFIO_TYPE1_TEST_EMULATED, &cmd) && + !domain(fd, 0, 1, 4096, 0), + "VADDR fixture retains production group restrictions"); + result(file_flags >= 0 && + !fcntl(backing, F_SETFD, file_flags & ~FD_CLOEXEC), + "native source inherits exact backing descriptor"); + child = fork(); + if (!child) { + char fd_arg[24], file_arg[24], native_arg[24]; + + snprintf(fd_arg, sizeof(fd_arg), "%d", fd); + snprintf(file_arg, sizeof(file_arg), "%d", backing); + snprintf(native_arg, sizeof(native_arg), "%zu", native); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, 0UL, 0UL, 0UL, 0UL)) + _exit(98); + execl("/init", "/init", "--native-source", fd_arg, file_arg, + native_arg, NULL); + _exit(99); + } + if (file_flags >= 0) + fcntl(backing, F_SETFD, file_flags); + result(child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status) && + !state(fd, &cmd) && !cmd.owner_users && + !(cmd.flags & VFIO_TYPE1_TEST_STATE_FRAGMENTS) && + cmd.locked_bytes == 2 * native && + cmd.mapped_bytes[0] == 2 * native, + "exited native source owns legacy GUP pins before transfer"); + result(!ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &invalidate), + "canonical VADDR invalidation leaves device translations alive"); + budget_captured = !getrlimit(RLIMIT_MEMLOCK, &limit) && + !syscall(SYS_capget, &header, saved); + budget_ready = budget_captured; + if (budget_ready) { + memcpy(reduced, saved, sizeof(saved)); + reduced[CAP_IPC_LOCK / 32].effective &= + ~(1U << (CAP_IPC_LOCK % 32)); + empty = (struct rlimit){ .rlim_cur = 0, + .rlim_max = limit.rlim_max }; + budget_ready = !setrlimit(RLIMIT_MEMLOCK, &empty) && + !syscall(SYS_capset, &header, reduced); + } + errno = 0; + result(budget_ready && ioctl(fd, VFIO_IOMMU_MAP_DMA, &update) < 0 && + errno == ENOMEM && !state(fd, &cmd) && + !cmd.owner_users && + !(cmd.flags & VFIO_TYPE1_TEST_STATE_FRAGMENTS) && + cmd.locked_bytes == 2 * native && + cmd.mapped_bytes[0] == 2 * native, + "failed owner credit preserves native pins, metadata and charge"); + if (budget_captured) { + result(!syscall(SYS_capset, &header, saved) && + !setrlimit(RLIMIT_MEMLOCK, &limit), + "restore original memlock budget and effective capability"); + } else { + result(0, "owner budget setup failed"); + } + transferred = !ioctl(fd, VFIO_IOMMU_MAP_DMA, &update) && + !state(fd, &cmd) && cmd.owner_users && + (cmd.flags & VFIO_TYPE1_TEST_STATE_FRAGMENTS) && + cmd.locked_bytes == 2 * native && + cmd.owner_locked_bytes == 2 * native; + result(transferred, + "canonical VADDR retry adopts native pins for alternative owner"); + /* Dependent CPU access is prohibited while VADDR remains invalid. */ + if (!transferred) + return; + result(!rw(fd, 0, bytes, sizeof(bytes), 0) && + all_bytes(bytes, sizeof(bytes), 0x64), + "new owner accesses retained native buffer with its own mm authorization"); + cmd = (struct vfio_type1_test_cmd){ .flags = 1 }; + result(!ioctl(fd, VFIO_TYPE1_TEST_EMULATED, &cmd), + "external consumer rejoins after VADDR update"); + cmd = (struct vfio_type1_test_cmd){ .iova = IOVA + 17 }; + result(!ioctl(fd, VFIO_TYPE1_TEST_PIN, &cmd) && !state(fd, &cmd) && + cmd.locked_bytes == 2 * native && cmd.external_refs == 1, + "external page array reuses adopted pins without new charge"); + result(!unmap(fd, 2 * native) && !state(fd, &cmd) && + !cmd.locked_bytes && !cmd.owner_locked_bytes && + !cmd.external_refs && !cmd.mapped_bytes[0], + "owner transfer final cleanup refunds recipient and clears domain"); + result(!domain(fd, 0, 0, 0, 0), "owner transfer domain detaches"); +} + +static int exercise(size_t native) +{ + struct vfio_type1_test_cmd cmd; + struct vfio_iommu_type1_info info = { .argsz = sizeof(info) }; + struct vfio_iommu_type1_dma_map bad = { .argsz = 8 }; + unsigned char bytes[64], disk[64]; + uint64_t bits = 0; + int fd = open("/dev/vfio-test", O_RDWR), + backing = memfd_create("vfio", MFD_CLOEXEC); + unsigned char *data = MAP_FAILED; + int status = 0; + pid_t child; + size_t length = native + 4096; + + printf("TAP version 13\n# real type1 ioctl ABI=%luK native=%zuK\n", + getauxval(AT_PAGESZ) / 1024, native / 1024); + result(fd >= 0 && backing >= 0 && !ftruncate(backing, 2 * native), + "open privileged recording-domain fixture and backing"); + if (fd < 0 || backing < 0 || failures) + goto out; + data = mmap(NULL, 2 * native, PROT_READ | PROT_WRITE, MAP_SHARED, + backing, 0); + result(data != MAP_FAILED, + "map native-sized shmem backing in actual process mm"); + if (data == MAP_FAILED) + goto out; + memset(data, 0x64, 2 * native); + result(!ioctl(fd, VFIO_IOMMU_GET_INFO, &info) && + (info.iova_pgsizes & 4096), + "type1 GET_INFO reports fine IOVA capability"); + errno = 0; + result(ioctl(fd, VFIO_IOMMU_MAP_DMA, &bad) < 0 && errno == EINVAL, + "canonical MAP rejects truncated argument"); + errno = 0; + result(ioctl(fd, VFIO_IOMMU_MAP_DMA, (void *)1) < 0 && errno == EFAULT, + "canonical MAP rejects inaccessible userspace argument"); + result(!map(fd, data, 2 * native), "lazy full-range MAP from real EL0"); + result(!state(fd, &cmd) && !cmd.locked_bytes && !cmd.mapped_bytes[0], + "lazy MAP acquires no DMA pins or translations"); + result(!rw(fd, 0, bytes, sizeof(bytes), 0) && + all_bytes(bytes, sizeof(bytes), 0x64), + "CPU-only GET reads actual source bytes"); + result(!state(fd, &cmd) && !cmd.locked_bytes, + "CPU GET does not charge long-term pins"); + result(!dirty(fd, VFIO_IOMMU_DIRTY_PAGES_FLAG_START), + "canonical dirty tracking starts"); + cmd = (struct vfio_type1_test_cmd){ .iova = IOVA + 17 }; + result(!ioctl(fd, VFIO_TYPE1_TEST_PIN, &cmd) && !state(fd, &cmd) && + cmd.locked_bytes == native && cmd.external_refs == 1, + "external page array preserves unaligned caller byte offset"); + cmd = (struct vfio_type1_test_cmd){ .iova = IOVA + 31 }; + result(!ioctl(fd, VFIO_TYPE1_TEST_PIN, &cmd) && !state(fd, &cmd) && + cmd.locked_bytes == native && cmd.external_refs == 2, + "duplicate external pin charges physical native page once"); + result(!bitmap(fd, 2 * native, &bits) && + bits == ((1ULL << (native / 4096)) - 1), + "dirty bitmap covers every fine unit in native external page"); + result(!unmap(fd, 2 * native), + "canonical UNMAP drains callback-held external refs"); + result(!state(fd, &cmd) && !cmd.locked_bytes && !cmd.external_refs && + cmd.notifications == 1, + "external-unpin callback refunds all pins"); + result(!dirty(fd, VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP), + "dirty tracking stops"); + result(!domain(fd, 0, 1, 4096, 0), + "recording4K domain attaches before child MAP"); + child = fork(); + if (!child) { + unsigned char *source = mmap(NULL, 2 * native, + PROT_READ | PROT_WRITE, MAP_SHARED, + backing, 0); + int ret = source == MAP_FAILED ? -1 : + map(fd, source + 4096, length); + + _exit(ret ? 1 : 0); + } + result(child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status), + "fine-offset MAP retains real child pins after exit"); + result(!state(fd, &cmd) && !cmd.owner_users && + cmd.locked_bytes == 2 * native && + cmd.owner_locked_bytes == 2 * native && + cmd.mapped_bytes[0] == length, + "exited source mm retains exact physical-page charge and translation length"); + errno = 0; + result(rw(fd, 0, bytes, sizeof(bytes), 0) < 0 && errno == EFAULT, + "foreign process CPU access preserves source-mm authorization"); + result(!rw(fd, 0, bytes, sizeof(bytes), VFIO_TYPE1_TEST_WORKER) && + all_bytes(bytes, sizeof(bytes), 0x64), + "kernel consumer reads exited source pins"); + errno = 0; + result(domain(fd, 1, 1, 4096, 4096) < 0 && errno == ENOMEM && + !state(fd, &cmd) && !cmd.mapped_bytes[1] && + cmd.mapped_bytes[0] == length && + cmd.locked_bytes == 2 * native, + "failed replay leaves existing domain and pins intact"); + result(!domain(fd, 1, 1, 4096, 0) && !state(fd, &cmd) && + cmd.mapped_bytes[0] == length && + cmd.mapped_bytes[1] == length && + cmd.locked_bytes == 2 * native, + "retry replays cached pins without resurrecting source mm"); + result(!dirty(fd, VFIO_IOMMU_DIRTY_PAGES_FLAG_START), + "fine DMA dirty tracking starts"); + memset(bytes, 0x9a, sizeof(bytes)); + result(!rw(fd, 4096 - 32, bytes, sizeof(bytes), + VFIO_TYPE1_TEST_WRITE | VFIO_TYPE1_TEST_WORKER) && + pread(backing, disk, sizeof(disk), 8192 - 32) == + sizeof(disk) && + all_bytes(disk, sizeof(disk), 0x9a), + "cached worker write dirties original shmem bytes"); + result(!bitmap(fd, length, &bits) && bits == 3, + "canonical bitmap returns exact two4K units touched by worker write"); + result(!unmap(fd, length), + "fine UNMAP synchronizes both recording domains"); + result(!state(fd, &cmd) && !cmd.locked_bytes && !cmd.mapped_bytes[0] && + !cmd.mapped_bytes[1], + "source-exit cleanup refunds pins and clears all translations"); + result(!domain(fd, 1, 0, 0, 0) && !domain(fd, 0, 0, 0, 0), + "both domains detach after complete cleanup"); + external_span(backing, native); + if (getauxval(AT_PAGESZ) != native) + native_owner(fd, backing, data, native); +out: + if (data != MAP_FAILED) + munmap(data, 2 * native); + if (backing >= 0) + close(backing); + if (fd >= 0) + close(fd); + printf("1..%u\n# type1 ioctl failures=%u\n", number, failures); + return !!failures; +} + +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 5 && !strcmp(argv[1], "--native-source")) { + size_t native = strtoul(argv[4], NULL, 10); + int fd = atoi(argv[2]), backing = atoi(argv[3]); + void *data; + + if (getauxval(AT_PAGESZ) != native) + return 2; + data = mmap(NULL, 2 * native, PROT_READ | PROT_WRITE, + MAP_SHARED, backing, 0); + return data == MAP_FAILED || map(fd, data, 2 * native) ? 1 : 0; + } + if (argc == 4 && !strcmp(argv[1], "--test") && getppid() == 1) { + size_t expected = strtoul(argv[3], NULL, 10); + + if (getauxval(AT_PAGESZ) != expected) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || argc != 1) + return 2; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("sysfs", "/sys", "sysfs", 0, NULL)) + return 3; + FILE *dev = fopen("/sys/class/misc/aurora-vfio-type1-test/dev", "r"); + unsigned int major, minor; + + if (!dev || fscanf(dev, "%u:%u", &major, &minor) != 2) + return 4; + fclose(dev); + if (mknod("/dev/vfio-test", S_IFCHR | 0600, makedev(major, minor))) + return 5; + size_t native = getauxval(AT_PAGESZ), + targets[] = { native, 4096, 16384 }; + int ok = 1; + + int compat_only = getenv("VFIO_COMPAT_ONLY") != NULL; + + for (unsigned int i = 0; i < 3; i++) { + if (compat_only) + break; + size_t target = targets[i]; + pid_t child; + int status = 0; + + if (target > native || (i && target == native)) + continue; + child = fork(); + if (!child) { + char native_arg[24], target_arg[24]; + + snprintf(native_arg, sizeof(native_arg), "%zu", native); + snprintf(target_arg, sizeof(target_arg), "%zu", target); + if (prctl(PR_AURORA_SET_EXEC_PAGE_SIZE, + target == native ? 0UL : target, 0UL, 0UL, + 0UL)) + _exit(3); + execl("/init", "/init", "--test", native_arg, + target_arg, NULL); + _exit(4); + } + ok &= child > 0 && waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status); + printf("# actual ABI=%zuK child_status=%d\n", target / 1024, + status); + } + const char *expected_arg = getenv("VFIO_COMPAT_PAGES"); + size_t expected = expected_arg ? strtoul(expected_arg, NULL, 10) : 4096; + pid_t compat = fork(); + int compat_status = 0; + + if (!compat) { + char native_arg[24], page_arg[24]; + + snprintf(native_arg, sizeof(native_arg), "%zu", native); + snprintf(page_arg, sizeof(page_arg), "%zu", expected); + execl("/compat-vfio", "/compat-vfio", native_arg, page_arg, + NULL); + _exit(98); + } + ok &= compat > 0 && waitpid(compat, &compat_status, 0) == compat && + WIFEXITED(compat_status) && !WEXITSTATUS(compat_status); + printf("# AArch32 expected=%zuK child_status=%d\n", expected / 1024, + compat_status); + printf("VFIO TYPE1 IOCTL %s\n", ok ? "PASS" : "FAIL"); + reboot(RB_POWER_OFF); + return !ok; +} diff --git a/tools/testing/selftests/arm64/user4k/vhost-log-init.c b/tools/testing/selftests/arm64/user4k/vhost-log-init.c new file mode 100644 index 00000000000000..07d6a8969bca0e --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/vhost-log-init.c @@ -0,0 +1,392 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SET_EXEC_PAGE_SIZE 0x41555001 +#ifndef VHOST_SET_FORK_FROM_OWNER +#define VHOST_SET_FORK_FROM_OWNER _IOW(VHOST_VIRTIO, 0x84, __u8) +#define VHOST_GET_FORK_FROM_OWNER _IOR(VHOST_VIRTIO, 0x85, __u8) +#endif +#define CHECK(c) \ + do { \ + if (!(c)) { \ + printf("FAIL line %d: %s errno=%d\n", __LINE__, #c, \ + errno); \ + return 1; \ + } \ + } while (0) +#define N 8 +#define ROUNDS 32 +#define PACKET_SIZE 96 +#define PROTOCOL 0x88b5 +#define DATA_GPA (51ULL * 4096) +#define TX_USED_GPA (71ULL * 4096) +#define RX_USED_GPA (70ULL * 4096) + +struct queue { + struct vring_desc desc[N]; + struct { + uint16_t flags, idx, ring[N + 1]; + } avail; + struct { + uint16_t flags, idx; + struct vring_used_elem ring[N]; + uint16_t event; + } used; + int kick, call; +}; + +static int queue_setup(int fd, struct queue *q, unsigned index) +{ + struct vhost_vring_state state = { .index = index, .num = N }; + struct vhost_vring_addr addr = { + .index = index, + .flags = 1U << VHOST_VRING_F_LOG, + .desc_user_addr = (uintptr_t)q->desc, + .avail_user_addr = (uintptr_t)&q->avail, + .used_user_addr = (uintptr_t)&q->used, + .log_guest_addr = index ? TX_USED_GPA : RX_USED_GPA, + }; + q->kick = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK); + q->call = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK); + CHECK(q->kick >= 0 && q->call >= 0); + CHECK(!ioctl(fd, VHOST_SET_VRING_NUM, &state)); + state.num = 0; + CHECK(!ioctl(fd, VHOST_SET_VRING_BASE, &state)); + CHECK(!ioctl(fd, VHOST_SET_VRING_ADDR, &addr)); + struct vhost_vring_file file = { .index = index, .fd = q->kick }; + CHECK(!ioctl(fd, VHOST_SET_VRING_KICK, &file)); + file.fd = q->call; + CHECK(!ioctl(fd, VHOST_SET_VRING_CALL, &file)); + return 0; +} +static int submit(struct queue *q, unsigned round) +{ + uint64_t one = 1; + q->avail.ring[round % N] = 0; + __atomic_store_n(&q->avail.idx, round + 1, __ATOMIC_RELEASE); + return write(q->kick, &one, sizeof(one)) == sizeof(one); +} +static int complete(struct queue *q, unsigned round) +{ + for (unsigned i = 0; i < 2000; i++) { + if (__atomic_load_n(&q->used.idx, __ATOMIC_ACQUIRE) == + round + 1) + return 1; + usleep(1000); + } + printf("queue timeout round=%u used=%u avail=%u\n", round, q->used.idx, + q->avail.idx); + return 0; +} +static int bitmap_ok(unsigned char *leaf, size_t ps, int active, int missing) +{ + for (size_t i = 0; i < ps; i++) { + unsigned char expected = i < ps - 9 ? (missing ? 0 : 0x64) : 0; + if (active && i == ps - 3) + expected = 8; /* RX buffer, GPA page 51 */ + if (active && i == ps - 1) + expected = 0xc0; /* RX/TX rings, pages 70/71 */ + if (__atomic_load_n(leaf + i, __ATOMIC_ACQUIRE) != expected) { + printf("bitmap byte %zu got=%#x expected=%#x\n", i, + leaf[i], expected); + return 0; + } + } + return 1; +} +static int filled(unsigned char *p, size_t len, unsigned char byte) +{ + for (size_t i = 0; i < len; i++) + if (p[i] != byte) + return 0; + return 1; +} +static int exercise_case(size_t ps, unsigned shared, unsigned forked, + uint8_t mode) +{ + int backing = -1; + if (shared) { + backing = memfd_create("vhost-log", 0); + CHECK(backing >= 0 && !ftruncate(backing, 4 * ps)); + } + unsigned char *src = mmap( + (void *)0x20000000, 4 * ps, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | + (shared ? MAP_SHARED : MAP_PRIVATE | MAP_ANONYMOUS), + backing, 0); + unsigned char *dst = + mmap((void *)0x30000000, 4 * ps, PROT_READ | PROT_WRITE, + MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(src != MAP_FAILED && dst != MAP_FAILED); + for (unsigned i = 0; i < 4; i++) + memset(src + i * ps, 0x61 + i, ps); + memset(dst, 0x91, 4 * ps); + CHECK(mremap(src + 3 * ps, ps, ps, MREMAP_MAYMOVE | MREMAP_FIXED, + dst + ps) == dst + ps); + unsigned char *leaf = dst + ps, *bitmap = leaf + ps - 9; + memset(bitmap, 0, 9); + int barrier[2]; + pid_t child = -1; + if (forked) { + CHECK(!pipe(barrier)); + child = fork(); + CHECK(child >= 0); + if (!child) { + close(barrier[1]); + char token; + int ok = read(barrier[0], &token, 1) == 1 && + bitmap_ok(leaf, ps, shared, 0); + _exit(ok ? 0 : 8); + } + close(barrier[0]); + } + int tap = open("/dev/net/tun", O_RDWR | O_CLOEXEC); + CHECK(tap >= 0); + struct ifreq ifr = { .ifr_flags = IFF_TAP | IFF_NO_PI }; + strcpy(ifr.ifr_name, "abi%d"); + CHECK(!ioctl(tap, TUNSETIFF, &ifr)); + int raw = socket(AF_PACKET, SOCK_RAW | SOCK_CLOEXEC, htons(PROTOCOL)); + CHECK(raw >= 0); + CHECK(!ioctl(raw, SIOCGIFINDEX, &ifr)); + int ifindex = ifr.ifr_ifindex; + CHECK(!ioctl(raw, SIOCGIFFLAGS, &ifr)); + ifr.ifr_flags |= IFF_UP; + CHECK(!ioctl(raw, SIOCSIFFLAGS, &ifr)); + int fd = open("/dev/vhost-net", O_RDWR | O_CLOEXEC); + CHECK(fd >= 0); + CHECK(!ioctl(fd, VHOST_SET_FORK_FROM_OWNER, &mode)); + uint8_t gotmode = 9; + CHECK(!ioctl(fd, VHOST_GET_FORK_FROM_OWNER, &gotmode) && + gotmode == mode); + CHECK(!ioctl(fd, VHOST_SET_OWNER, NULL)); + uint64_t log_base = (uintptr_t)bitmap; + CHECK(!ioctl(fd, VHOST_SET_LOG_BASE, &log_base)); + unsigned char *data = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(data != MAP_FAILED); + struct { + struct vhost_memory mem; + struct vhost_memory_region region; + } mem = { + .mem.nregions = 1, + .region = { .guest_phys_addr = DATA_GPA, + .memory_size = 4096, + .userspace_addr = (uintptr_t)data }, + }; + CHECK(!ioctl(fd, VHOST_SET_MEM_TABLE, &mem)); + uint64_t features = (1ULL << VHOST_F_LOG_ALL) | + (1ULL << VHOST_NET_F_VIRTIO_NET_HDR); + uint64_t available; + CHECK(!ioctl(fd, VHOST_GET_FEATURES, &available) && + (available & features) == features); + CHECK(!ioctl(fd, VHOST_SET_FEATURES, &features)); + struct queue *q = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + CHECK(q != MAP_FAILED && sizeof(*q) * 2 < ps); + CHECK(!queue_setup(fd, q, 0) && !queue_setup(fd, q + 1, 1)); + for (unsigned index = 0; index < 2; index++) { + struct vhost_vring_file backend = { .index = index, .fd = tap }; + CHECK(!ioctl(fd, VHOST_NET_SET_BACKEND, &backend)); + } + struct sockaddr_ll dest = { .sll_family = AF_PACKET, + .sll_protocol = htons(PROTOCOL), + .sll_ifindex = ifindex, + .sll_halen = 6 }; + unsigned char packet[PACKET_SIZE]; + memset(packet, 0x5c, sizeof(packet)); + memset(packet, 0xff, 6); + packet[6] = 2; + packet[12] = PROTOCOL >> 8; + packet[13] = PROTOCOL & 0xff; + q[0].desc[0] = (struct vring_desc){ .addr = DATA_GPA, + .len = 512, + .flags = VRING_DESC_F_WRITE }; + q[1].desc[0] = + (struct vring_desc){ .addr = DATA_GPA + 1024, + .len = sizeof(struct virtio_net_hdr) + + PACKET_SIZE }; + memcpy(data + 1024 + sizeof(struct virtio_net_hdr), packet, + PACKET_SIZE); + for (unsigned phase = 0; phase < 2; phase++) { + if (phase) { + memset(bitmap, 0, 9); + CHECK(!madvise(leaf, ps, MADV_DONTNEED)); + int pagemap = open("/proc/self/pagemap", O_RDONLY); + uint64_t entry; + CHECK(pagemap >= 0 && + pread(pagemap, &entry, sizeof(entry), + (uintptr_t)leaf / ps * sizeof(entry)) == + sizeof(entry)); + CHECK(!(entry & (1ULL << 63))); + close(pagemap); + for (unsigned index = 0; index < 2; index++) { + struct vhost_vring_file backend = { + .index = index, .fd = tap + }; + CHECK(!ioctl(fd, VHOST_NET_SET_BACKEND, + &backend)); + } + } + for (unsigned round = phase * ROUNDS; + round < (phase + 1) * ROUNDS; round++) { + packet[20] = round; + memset(data, 0xaa, 512); + CHECK(submit(q, round)); + CHECK(sendto(raw, packet, sizeof(packet), 0, + (void *)&dest, + sizeof(dest)) == sizeof(packet)); + CHECK(submit(q + 1, round)); + CHECK(complete(q, round) && complete(q + 1, round)); + CHECK(q[0].used.ring[round % N].id == 0); + if (q[0].used.ring[round % N].len != + sizeof(struct virtio_net_hdr) + PACKET_SIZE) + printf("RX length=%u expected=%zu ethertype=%02x%02x\n", + q[0].used.ring[round % N].len, + sizeof(struct virtio_net_hdr) + + PACKET_SIZE, + data[sizeof(struct virtio_net_hdr) + 12], + data[sizeof(struct virtio_net_hdr) + 13]); + CHECK(q[0].used.ring[round % N].len == + sizeof(struct virtio_net_hdr) + PACKET_SIZE); + CHECK(!memcmp(data + sizeof(struct virtio_net_hdr), + packet, PACKET_SIZE)); + CHECK(filled(data + sizeof(struct virtio_net_hdr) + + PACKET_SIZE, + 512 - sizeof(struct virtio_net_hdr) - + PACKET_SIZE, + 0xaa)); + } + /* Stop and flush workers before checking the bitmap or discarding backing. */ + for (unsigned index = 0; index < 2; index++) { + struct vhost_vring_file backend = { .index = index, + .fd = -1 }; + CHECK(!ioctl(fd, VHOST_NET_SET_BACKEND, &backend)); + } + CHECK(bitmap_ok(leaf, ps, 1, phase && !shared)); + if (forked && !phase) { + CHECK(write(barrier[1], "x", 1) == 1); + close(barrier[1]); + int status; + CHECK(waitpid(child, &status, 0) == child && + WIFEXITED(status) && !WEXITSTATUS(status)); + } + } + close(fd); + close(raw); + close(tap); + for (unsigned i = 0; i < 2; i++) { + close(q[i].kick); + close(q[i].call); + } + CHECK(filled(dst, ps, 0x91) && filled(dst + 2 * ps, 2 * ps, 0x91)); + for (unsigned i = 0; i < 3; i++) + CHECK(filled(src + i * ps, ps, 0x61 + i)); + CHECK(!munmap(src, 3 * ps) && !munmap(dst, 4 * ps)); + CHECK(!munmap(data, ps) && !munmap(q, ps)); + if (backing >= 0) + close(backing); + printf("ok - %zuK vhost %s fork=%u worker=%u: %u RX/TX packets, relocated dirty bits, missing-PTE repair and neighbors\n", + ps / 1024, shared ? "shared" : "private", forked, mode, + 2 * ROUNDS); + return 0; +} +static int exercise(size_t expected) +{ + CHECK(getauxval(AT_PAGESZ) == expected); + alarm(45); + for (uint8_t mode = 0; mode < 2; mode++) + for (unsigned shared = 0; shared < 2; shared++) + for (unsigned forked = 0; forked < 2; forked++) + CHECK(!exercise_case(expected, shared, forked, + mode)); + return 0; +} +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IONBF, 0); + if (argc == 3 && !strcmp(argv[1], "--exercise")) { + if (getppid() != 1) + return 2; + return exercise(strtoul(argv[2], NULL, 10)); + } + if (getpid() != 1 || argc != 1) { + fprintf(stderr, "Disposable VM PID1 only\n"); + return 2; + } + int fail = 0; + if (mount("proc", "/proc", "proc", 0, NULL) || + mount("devtmpfs", "/dev", "devtmpfs", 0, NULL)) { + perror("mount"); + fail = 1; + goto done; + } + /* Keep unsolicited IPv6 DAD packets out of the isolated TAP queues. */ + int ipv6 = + open("/proc/sys/net/ipv6/conf/default/disable_ipv6", O_WRONLY); + if (ipv6 < 0 || write(ipv6, "1", 1) != 1) { + perror("disable IPv6"); + fail = 1; + goto done; + } + close(ipv6); + size_t sizes[] = { getauxval(AT_PAGESZ), 4096, 16384 }; + for (unsigned i = 0; i < (sizes[0] > 16384 ? 3U : 2U); i++) { + pid_t c = fork(); + int status = 0; + if (!c) { + char expected[24]; + snprintf(expected, sizeof(expected), "%zu", sizes[i]); + if (prctl(SET_EXEC_PAGE_SIZE, sizes[i], 0UL, 0UL, 0UL)) + _exit(4); + execl("/init", "/init", "--exercise", expected, NULL); + _exit(5); + } + int ok = 0; + for (unsigned ticks = 0; c > 0 && ticks < 600; ticks++) { + pid_t got = waitpid(c, &status, WNOHANG); + if (got == c) { + ok = WIFEXITED(status) && !WEXITSTATUS(status); + break; + } + if (ticks == 599) { + kill(c, SIGKILL); + waitpid(c, &status, 0); + break; + } + usleep(100000); + } + printf("%s - ABI %zuK vhost log status=%#x\n", + ok ? "ok" : "not ok", sizes[i] / 1024, status); + fail += !ok; + } +done: + printf("VHOST LOG %s\n", fail ? "FAIL" : "PASS"); + sync(); + reboot(RB_POWER_OFF); + for (;;) + pause(); +} diff --git a/tools/testing/selftests/arm64/user4k/wine-kernel32.def b/tools/testing/selftests/arm64/user4k/wine-kernel32.def new file mode 100644 index 00000000000000..891e48c1fb83bc --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/wine-kernel32.def @@ -0,0 +1,20 @@ +LIBRARY KERNEL32.dll +EXPORTS +VirtualAlloc +VirtualFree +VirtualProtect +VirtualQuery +GetSystemInfo +GetStdHandle +WriteFile +ExitProcess +GetLastError +AddVectoredExceptionHandler +RemoveVectoredExceptionHandler +CreateThread +WaitForSingleObject +GetExitCodeThread +CloseHandle +CreateFileMappingA +MapViewOfFile +UnmapViewOfFile diff --git a/tools/testing/selftests/arm64/user4k/wine-page-contract.c b/tools/testing/selftests/arm64/user4k/wine-page-contract.c new file mode 100644 index 00000000000000..149f619de6651b --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/wine-page-contract.c @@ -0,0 +1,181 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Freestanding Windows x64 PE; Win32 imports, no CRT or external SDK. */ +typedef unsigned long DWORD; +typedef int BOOL; +typedef unsigned long long SIZE_T; +typedef void *HANDLE; +#define API __declspec(dllimport) +API void *VirtualAlloc(void *, SIZE_T, DWORD, DWORD); +API BOOL VirtualFree(void *, SIZE_T, DWORD); +API BOOL VirtualProtect(void *, SIZE_T, DWORD, DWORD *); +API SIZE_T VirtualQuery(const void *, void *, SIZE_T); +API void GetSystemInfo(void *); +API HANDLE GetStdHandle(DWORD); +API BOOL WriteFile(HANDLE, const void *, DWORD, DWORD *, void *); +API void ExitProcess(DWORD); +API DWORD GetLastError(void); +API void *AddVectoredExceptionHandler(DWORD, void *); +API DWORD RemoveVectoredExceptionHandler(void *); +API HANDLE CreateThread(void *, SIZE_T, DWORD (*)(void *), void *, DWORD, + DWORD *); +API DWORD WaitForSingleObject(HANDLE, DWORD); +API BOOL GetExitCodeThread(HANDLE, DWORD *); +API BOOL CloseHandle(HANDLE); +API HANDLE CreateFileMappingA(HANDLE, void *, DWORD, DWORD, DWORD, + const char *); +API void *MapViewOfFile(HANDLE, DWORD, DWORD, DWORD, SIZE_T); +API BOOL UnmapViewOfFile(const void *); +struct system_info { + DWORD arch, page; + void *min, *max; + SIZE_T mask; + DWORD cpus, type, granule; + unsigned short level, revision; +}; +struct memory_info { + void *base, *allocation; + DWORD allocation_protect, pad; + SIZE_T size; + DWORD state, protect, type, pad2; +}; +struct exception_record { + DWORD code, flags; + void *record, *address; + DWORD count, pad; + SIZE_T info[15]; +}; +struct exception_pointers { + struct exception_record *record; + void *context; +}; +/* Keep the freestanding declarations consistent with the Windows x64 ABI. */ +_Static_assert(sizeof(DWORD) == 4, "DWORD must be 32 bits"); +_Static_assert(sizeof(SIZE_T) == 8, "SIZE_T must be 64 bits"); +_Static_assert(sizeof(struct system_info) == 48, "SYSTEM_INFO layout"); +_Static_assert(sizeof(struct memory_info) == 48, + "MEMORY_BASIC_INFORMATION layout"); +_Static_assert(sizeof(struct exception_record) == 152, + "EXCEPTION_RECORD layout"); +static volatile unsigned char *pages; +static volatile DWORD handled; +static void say(const char *s) +{ + DWORD n = 0, w = 0; + while (s[n]) + n++; + WriteFile(GetStdHandle((DWORD)-11), s, n, &w, 0); +} +static void number(DWORD n) +{ + char buf[11], *p = buf + 10; + *p = 0; + do { + *--p = '0' + n % 10; + n /= 10; + } while (n); + say(p); +} +static void fail(DWORD stage) +{ + DWORD e = GetLastError(); + say("WINE PAGE CONTRACT FAIL stage="); + number(stage); + say(" error="); + number(e); + say("\r\n"); + ExitProcess(1); +} +#define CHECK(x, n) \ + do { \ + if (!(x)) \ + fail(n); \ + } while (0) +static long handler(struct exception_pointers *e) +{ + DWORD old; + if (e->record->code != 0xc0000005 || e->record->count < 2 || + e->record->info[0] != 1 || + e->record->info[1] != (SIZE_T)(pages + 4103)) + return 0; + if (!VirtualProtect((void *)(pages + 4096), 4096, 4, &old)) + return 0; + handled++; + return -1; +} +static DWORD thread(void *arg) +{ + volatile unsigned char *p = arg; + for (unsigned int i = 0; i < 4096; i++) + p[i] = (unsigned char)(i ^ 0x5a); + return 37; +} +void mainCRTStartup(void) +{ + struct system_info si; + struct memory_info mi; + DWORD old, status; + GetSystemInfo(&si); + CHECK(si.page == 4096 && si.granule == 65536, 1); + say("ok - Wine reports 4K pages and 64K allocation granularity\r\n"); + pages = VirtualAlloc(0, 65536, 0x2000, 1); + CHECK(pages, 2); + CHECK(VirtualAlloc((void *)pages, 12288, 0x1000, 4) == pages, 3); + for (unsigned int i = 0; i < 12288; i++) + pages[i] = (unsigned char)(i ^ (i >> 12)); + CHECK(VirtualProtect((void *)(pages + 4096), 4096, 2, &old) && old == 4, + 4); + CHECK(VirtualQuery((void *)(pages + 4096), &mi, sizeof(mi)) == + sizeof(mi) && + mi.base == pages + 4096 && mi.size == 4096 && + mi.protect == 2, + 5); + CHECK(VirtualQuery((void *)(pages + 8192), &mi, sizeof(mi)) == + sizeof(mi) && + mi.protect == 4, + 6); + void *veh = AddVectoredExceptionHandler(1, handler); + CHECK(veh, 7); + pages[4103] = 0x6d; + CHECK(handled == 1 && pages[4103] == 0x6d, 8); + CHECK(RemoveVectoredExceptionHandler(veh), 9); + say("ok - Wine independent 4K protection and handled access violation\r\n"); + CHECK(VirtualFree((void *)(pages + 4096), 4096, 0x4000), 10); + CHECK(VirtualQuery((void *)(pages + 4096), &mi, sizeof(mi)) == + sizeof(mi) && + mi.state == 0x2000, + 11); + CHECK(VirtualAlloc((void *)(pages + 4096), 4096, 0x1000, 4) == + pages + 4096, + 12); + for (unsigned int i = 0; i < 4096; i++) + CHECK(!pages[4096 + i], 13); + for (unsigned int i = 0; i < 4096; i++) + CHECK(pages[i] == (unsigned char)i && + pages[8192 + i] == (unsigned char)(i ^ 2), + 14); + say("ok - Wine 4K decommit and zero-filled recommit preserve neighbours\r\n"); + HANDLE t = CreateThread(0, 0, thread, (void *)(pages + 4096), 0, 0); + CHECK(t, 15); + CHECK(WaitForSingleObject(t, 30000) == 0 && + GetExitCodeThread(t, &status) && status == 37, + 16); + CHECK(CloseHandle(t), 17); + for (unsigned int i = 0; i < 4096; i++) + CHECK(pages[4096 + i] == (unsigned char)(i ^ 0x5a), 18); + say("ok - Wine threads share committed 4K pages\r\n"); + HANDLE file = CreateFileMappingA((HANDLE)(SIZE_T)-1, 0, 4, 0, 16384, 0); + CHECK(file, 19); + unsigned char *a = MapViewOfFile(file, 2, 0, 0, 16384), + *b = MapViewOfFile(file, 2, 0, 0, 16384); + CHECK(a && b, 20); + a[4099] = 0xa7; + CHECK(b[4099] == 0xa7, 21); + CHECK(VirtualProtect(a + 4096, 4096, 2, &old), 22); + b[4099] = 0x53; + CHECK(a[4099] == 0x53, 23); + CHECK(UnmapViewOfFile(a) && UnmapViewOfFile(b) && CloseHandle(file), + 24); + CHECK(VirtualFree((void *)pages, 0, 0x8000), 25); + say("ok - Wine shared section aliases and 4K protection\r\nWINE PAGE CONTRACT PASS\r\n"); + ExitProcess(0); +} diff --git a/tools/testing/selftests/arm64/user4k/x18-probe.S b/tools/testing/selftests/arm64/user4k/x18-probe.S new file mode 100644 index 00000000000000..565f34e8323f57 --- /dev/null +++ b/tools/testing/selftests/arm64/user4k/x18-probe.S @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include + + .text + .globl x18_syscall + .type x18_syscall, %function +/* (nr, arg0, arg1, arg2, observed_x18); no libc code inside the probe. */ +x18_syscall: + hint 34 /* bti c */ + stp x18, x19, [sp, #-16]! + mov x19, x4 + movz x18, #0x3210 + movk x18, #0x7654, lsl #16 + movk x18, #0xba98, lsl #32 + movk x18, #0xfedc, lsl #48 + mov x8, x0 + mov x0, x1 + mov x1, x2 + mov x2, x3 + svc #0 + str x18, [x19] + ldp x18, x19, [sp], #16 + ret + .size x18_syscall, .-x18_syscall + .section .note.GNU-stack,"",%progbits diff --git a/tools/testing/selftests/drivers/apple/Makefile b/tools/testing/selftests/drivers/apple/Makefile new file mode 100644 index 00000000000000..76bca086dde7a3 --- /dev/null +++ b/tools/testing/selftests/drivers/apple/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +# Manual physical test: requires user interaction and changes LED brightness. +TEST_FILES := j813-peripherals.sh + +include ../../lib.mk diff --git a/tools/testing/selftests/drivers/apple/j813-peripherals.sh b/tools/testing/selftests/drivers/apple/j813-peripherals.sh new file mode 100644 index 00000000000000..ecad597fe9113e --- /dev/null +++ b/tools/testing/selftests/drivers/apple/j813-peripherals.sh @@ -0,0 +1,104 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# Physical J813 check. Save stdout and the payload SHA-256 manifest together. +# Usage: sh j813-peripherals.sh [event-capture-seconds] +set -eu + +duration=${1:-30} +case "$duration" in *[!0-9]*|'') exit 2;; esac +[ "$duration" -ge 1 ] && [ "$duration" -le 600 ] || exit 2 +[ -r /proc/device-tree/compatible ] || exit 4 +tr '\000' '\n' < /proc/device-tree/compatible | grep -qx apple,j813 || exit 4 +[ "$(id -u)" -eq 0 ] || { echo 'SKIP: run as root'; exit 4; } + +work=$(mktemp -d) +led=/sys/class/leds/kbd_backlight +saved_brightness= +cleanup() +{ + if [ -n "$saved_brightness" ]; then + echo "$saved_brightness" > "$led/brightness" + fi + rm -rf "$work" +} +trap cleanup EXIT +trap 'exit 1' INT TERM + +echo J813_PERIPHERALS_BEGIN +uname -r +zcat /proc/config.gz | sha256sum +cat /sys/devices/system/cpu/online + +hwmon= +for h in /sys/class/hwmon/hwmon*; do + [ -r "$h/name" ] || continue + [ "$(cat "$h/name")" = macsmc_hwmon ] || continue + hwmon=$h +done +[ -n "$hwmon" ] || { echo 'FAIL: no SMC hwmon'; exit 1; } +for sample in 1 2; do + count=0 + for input in "$hwmon"/temp*_input; do + [ -r "$input" ] || continue + value=$(cat "$input") + label=$(cat "${input%_input}_label") + [ "$value" -ge -40000 ] && [ "$value" -le 125000 ] || exit 1 + printf 'temperature sample=%s key=%s millidegrees=%s\n' "$sample" "$label" "$value" + count=$((count + 1)) + done + [ "$count" -gt 0 ] || { echo 'FAIL: no temperature channels'; exit 1; } + sleep 1 +done +echo "PASS: $count readable SMC temperature channels" + +[ -r "$led/brightness" ] || { echo 'FAIL: no keyboard backlight'; exit 1; } +saved_brightness=$(cat "$led/brightness") +maximum=$(cat "$led/max_brightness") +for level in 0 $((maximum / 2)) "$maximum"; do + echo "$level" > "$led/brightness" + [ "$(cat "$led/brightness")" = "$level" ] || exit 1 + printf 'backlight requested=%s readback=%s\n' "$level" "$(cat "$led/brightness")" + sleep 5 +done +echo "$saved_brightness" > "$led/brightness" +echo 'PASS: backlight control and restore (visual confirmation is separate)' + +keyboard= +trackpad= +for e in /sys/class/input/event*; do + [ -r "$e/device/name" ] || continue + case "$(cat "$e/device/name")" in + 'Apple MTP keyboard') keyboard=/dev/input/${e##*/} ;; + 'Apple MTP multi-touch') trackpad=/dev/input/${e##*/} ;; + esac +done +[ -n "$keyboard" ] && [ -n "$trackpad" ] || { + echo 'FAIL: keyboard or trackpad missing'; exit 1; +} +echo "Move/click the built-in trackpad and press/release keys for $duration seconds." +# Count bytes only. Do not retain key codes, pointer coordinates, or raw events. +timeout "$duration" cat "$keyboard" | wc -c > "$work/keyboard" & +kp=$! +timeout "$duration" cat "$trackpad" | wc -c > "$work/trackpad" & +tp=$! +wait "$kp" +wait "$tp" +failed=0 +for name in keyboard trackpad; do + bytes=$(cat "$work/$name") + [ "$bytes" -gt 0 ] && [ $((bytes % 24)) -eq 0 ] || { + echo "FAIL: $name produced no complete arm64 input events" + failed=1 + continue + } + printf 'PASS: %s events=%s\n' "$name" "$((bytes / 24))" +done +dmesg > "$work/dmesg" +fault_pattern='Kernel panic|Oops:|(^|[[:space:]])SError([[:space:]:]|$)' +fault_pattern="$fault_pattern|(^|[[:space:]])BUG:|apple-dart .*:.*(fault|error)" +if grep -E "$fault_pattern" "$work/dmesg"; then + echo 'FAIL: kernel fault detected' + exit 1 +fi +[ "$failed" -eq 0 ] || exit 1 +echo J813_PERIPHERALS_PASS diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/Makefile b/tools/testing/selftests/drivers/gpu/apple-afkv2/Makefile new file mode 100644 index 00000000000000..df1b033d9cb324 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only +HOSTCC ?= cc +export HOSTCC +export AFKV2_SOURCE_ROOT := $(abspath ../../../../../..) + +TEST_PROGS := run-afkv2-tests.sh +TEST_FILES := codec.c owned-prefix.c owned-suffix.c extract-owned.py inline-edid.c extract-service.py extract-send.py extract-edid.py extract-ring.py extract-notify.py host-include + +include ../../../lib.mk diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/README b/tools/testing/selftests/drivers/gpu/apple-afkv2/README new file mode 100644 index 00000000000000..064cdcf0185956 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/README @@ -0,0 +1,40 @@ +AFKV2 host selftests +=================== + +Run ./run-afkv2-tests.sh from this directory, or use make run_tests in a +complete kernel source checkout. HOSTCC, CFLAGS and PYTHON select host tools +(standalone CC is also accepted). +AFKV2_SANITIZERS=1 additionally enables AddressSanitizer and UBSan. +AFKV2_SOURCE_ROOT can name the source checkout when running a copied test +package; the Makefile exports it automatically for separate output builds. + +The codec tests include the production header directly and compare invented, +independently specified golden bytes. They cover truncation, category ranges, +length/padding/overflow, unaligned input, OOB descriptor ownership, response +correlation, token preservation and command ID packing. + +The owned-copy test extracts the actual production function from afk.c, prints +its SHA256, and compiles it with synthetic command slots and CPU buffers. It +checks active/inactive/done slots, high device addresses, boundaries and copies. +Locks and DMA barriers are mocked: this test does not qualify concurrency, +DMA coherency, firmware, a graphics device or any hardware display behavior. + +All build products are temporary. No device node is opened and no GPU, firmware +or target machine is contacted. The extractor fails if production function +boundaries change, requiring review rather than silently testing a stale copy. + +The inline EDID tests cover response context, status, declared/actual lengths, +EDID header/checksum/block limits, and continued rejection of inline responses +by the OOB matcher. The actual service-call body is extracted for short reply, +error status and timeout checks; its wire sender is mocked. + +The real sender is also extracted to test a 128-byte mocked command payload +with a 32 KiB response capacity, OOB behavior, send failures and timeout +retention. The two production EDID readers are tested independently to +prove legacy checksum tolerance remains unchanged. + +The complete encoded EDID request is 160 bytes including the queue entry, +consumes 192 bytes at a 64-byte stride, and needs a 256-byte empty ring. The +ring-admission block is extracted from the real transmitter to check this. +The late-publication notifier is extracted to test asynchronous hotplug +queueing and connector-registration, quiesce, routing and legacy guards. diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/codec.c b/tools/testing/selftests/drivers/gpu/apple-afkv2/codec.c new file mode 100644 index 00000000000000..79598033c084ab --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/codec.c @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Aurora Silicon */ +#undef NDEBUG +#include +#include +#include +#include +#include "afkv2-codec.h" + +static unsigned int assertions; +#define CHECK(x) \ + do { \ + assertions++; \ + assert(x); \ + } while (0) + +/* Invented fixtures with independently specified wire bytes; no device data. */ +static const u8 open_fixture[24] = { 0x7b, 0, 0x34, 0x12, 0x10, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, + 0x12, 0, 1, 0, 0, 0, 0, 0 }; +static const u8 descriptor_fixture[32] = { + 1, 0xab, 0, 0, 0x40, 0, 0, 0, 0x88, 0x77, 0x66, + 0x55, 0x44, 0x33, 0x22, 0x11, 8, 7, 6, 5, 4, 3, + 2, 1, 0x40, 0, 0, 0, 0x10, 0, 0, 0 +}; + +int main(void) +{ + u8 data[128], command[32], unaligned[129], saved[128]; + struct afkv2_frame frame, reply; + struct afkv2_command parsed, response; + size_t written = 999; + int ret; + + memset(data, 0xa5, sizeof(data)); + CHECK(!afkv2_encode_open(data, sizeof(data), 0x7b, 0x1234, &written)); + CHECK(written == sizeof(open_fixture)); + CHECK(!memcmp(data, open_fixture, sizeof(open_fixture))); + CHECK(data[24] == 0xa5); + CHECK(!afkv2_decode_frame(data, written, false, &frame)); + CHECK(frame.interface == 0x1234 && frame.type == AFKV2_OPEN); + CHECK(frame.category == AFKV2_REPORT && frame.interface_flags == 1); + CHECK(frame.payload_size == 0 && frame.padding_size == 0); + memcpy(unaligned + 1, open_fixture, sizeof(open_fixture)); + CHECK(!afkv2_decode_frame(unaligned + 1, sizeof(open_fixture), false, + &frame)); + CHECK(afkv2_decode_frame(data, SIZE_MAX, true, &frame) == -E2BIG); + CHECK(afkv2_decode_frame(NULL, 24, false, &frame) == -EINVAL); + CHECK(afkv2_decode_frame(data, 24, false, NULL) == -EINVAL); + for (size_t i = 0; i < 24; i++) + CHECK(afkv2_decode_frame(data, i, false, &frame) == -EINVAL); + CHECK(afkv2_decode_frame(data, 32, false, &frame) == -EINVAL); + CHECK(!afkv2_decode_frame(data, 32, true, &frame) && + frame.padding_size == 8); + for (u32 i = 0; i < 16; i++) { + afkv2_put32(data + 4, i); + CHECK(afkv2_decode_frame(data, 24, false, &frame) == -EINVAL); + } + afkv2_put32(data + 4, 17); + CHECK(afkv2_decode_frame(data, 24, false, &frame) == -EINVAL); + afkv2_put32(data + 4, UINT32_MAX); + CHECK(afkv2_decode_frame(data, 24, false, &frame) == -EINVAL); + memcpy(data, open_fixture, sizeof(open_fixture)); + for (unsigned int category = 0; category < 256; category++) { + data[17] = category; + ret = afkv2_decode_frame(data, 24, false, &frame); + CHECK(category <= 2 ? ret == 0 : ret == -EPROTO); + } + CHECK(!afkv2_encode_oob_command(command, sizeof(command), 0xab, + 0x1122334455667788ULL, 64, + 0x0102030405060708ULL, 16)); + CHECK(!memcmp(command, descriptor_fixture, sizeof(command))); + CHECK(!afkv2_decode_command(command, sizeof(command), &parsed)); + CHECK(parsed.oob && parsed.id == 0xab && parsed.value == 64); + CHECK(parsed.rx_address == 0x1122334455667788ULL && + parsed.tx_size == 16); + CHECK(parsed.inline_size == 0); + for (size_t i = 0; i < sizeof(command); i++) + CHECK(afkv2_decode_command(command, i, &response) == -EINVAL); + command[0] = 2; + CHECK(afkv2_decode_command(command, sizeof(command), &response) == + -EINVAL); + memcpy(command, descriptor_fixture, sizeof(command)); + command[2] = 1; + CHECK(afkv2_decode_command(command, sizeof(command), &response) == + -EINVAL); + memcpy(command, descriptor_fixture, sizeof(command)); + CHECK(afkv2_encode_oob_command(command, 31, 0, 1, 1, 2, 1) == -ENOSPC); + CHECK(afkv2_encode_oob_command(command, 32, 0, UINT64_MAX, 2, 0, 0) == + -EINVAL); + CHECK(afkv2_encode_oob_command(command, 32, 0, 1, AFKV2_MAX_PAYLOAD + 1, + 0, 0) == -EINVAL); + CHECK(afkv2_encode_oob_command(command, 32, 0, 1, 0, 0, 0) == -EINVAL); + memcpy(command, descriptor_fixture, sizeof(command)); + CHECK(!afkv2_encode_frame(data, sizeof(data), 1, 0x1234, + 0xaabbccddeeff0011ULL, 0xc0, AFKV2_COMMAND, 0, + command, sizeof(command), &written)); + CHECK(written == 56 && afkv2_get32(data + 4) == 48); + CHECK(!afkv2_decode_frame(data, written, false, &frame)); + CHECK(frame.token == 0xaabbccddeeff0011ULL && + frame.category == AFKV2_COMMAND); + CHECK(!afkv2_encode_error_reply(unaligned, sizeof(unaligned), 2, &frame, + &parsed, 0xe00002c7U, &written)); + CHECK(!afkv2_decode_frame(unaligned, written, false, &reply)); + CHECK(reply.token == frame.token && reply.interface == frame.interface); + CHECK(reply.type == frame.type && reply.category == AFKV2_RESPONSE); + CHECK(!afkv2_decode_command(reply.payload, reply.payload_size, + &response)); + CHECK(response.value == 0xe00002c7U && response.rx_size == 0 && + response.tx_size == 0); + CHECK(response.rx_address == parsed.rx_address && + response.tx_address == parsed.tx_address); + CHECK(!afkv2_check_response(&reply, &response, 0x1234, 0xc0, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16)); + response.value = 0; + response.rx_size = 64; + response.tx_size = 16; + CHECK(!afkv2_check_response(&reply, &response, 0x1234, 0xc0, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16)); + response.rx_address++; + CHECK(afkv2_check_response(&reply, &response, 0x1234, 0xc0, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16) == -EPROTO); + response.rx_address--; + response.rx_size++; + CHECK(afkv2_check_response(&reply, &response, 0x1234, 0xc0, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16) == -EPROTO); + response.rx_size--; + CHECK(afkv2_check_response(&reply, &response, 0x1235, 0xc0, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16) == -EPROTO); + CHECK(afkv2_check_response(&reply, &response, 0x1234, 0xc1, 0xab, + parsed.rx_address, 64, parsed.tx_address, + 16) == -EPROTO); + CHECK(afkv2_check_response(&reply, &response, 0x1234, 0xc0, 0xaa, + parsed.rx_address, 64, parsed.tx_address, + 16) == -EPROTO); + memset(data, 0xa5, sizeof(data)); + memcpy(saved, data, sizeof(data)); + CHECK(afkv2_encode_open(data, 23, 0, 1, &written) == -ENOSPC); + CHECK(!memcmp(data, saved, sizeof(data))); + CHECK(afkv2_encode_open(data, sizeof(data), 0, 0, &written) == -EINVAL); + CHECK(!memcmp(data, saved, sizeof(data))); + CHECK(afkv2_encode_frame(data, sizeof(data), 0, 1, 0, 0, 0, 0, data, + SIZE_MAX, &written) == -E2BIG); + CHECK(!memcmp(data, saved, sizeof(data))); + u8 *large = malloc(AFKV2_HEADER_SIZE + AFKV2_MAX_PAYLOAD); + + CHECK(large); + memset(large, 0x5a, AFKV2_HEADER_SIZE + AFKV2_MAX_PAYLOAD); + CHECK(!afkv2_encode_frame(large, AFKV2_HEADER_SIZE + AFKV2_MAX_PAYLOAD, + 0, 1, 0, 0xc0, 1, 0, large, AFKV2_MAX_PAYLOAD, + &written)); + CHECK(!afkv2_decode_frame(large, written, false, &frame)); + CHECK(frame.payload_size == AFKV2_MAX_PAYLOAD && + frame.payload[0] == 0x5a); + free(large); + bool ids[256] = { 0 }; + + for (unsigned int generation = 0; generation < 16; generation++) { + for (unsigned int slot = 0; slot < 16; slot++) { + u8 id = afkv2_command_id(generation, slot); + + CHECK(!ids[id]); + ids[id] = true; + CHECK((id & 15) == slot); + } + } + CHECK(afkv2_command_id(16, 3) == afkv2_command_id(0, 3)); + CHECK(afkv2_owned_span(0x1000, 32, 0x1000, 32)); + CHECK(afkv2_owned_span(0x1000, 32, 0x101c, 4)); + CHECK(!afkv2_owned_span(0x1000, 32, 0x0fff, 1)); + CHECK(!afkv2_owned_span(0x1000, 32, 0x101d, 4)); + CHECK(!afkv2_owned_span(0x1000, 32, 0x1020, 1)); + CHECK(!afkv2_owned_span(0x1000, 0, 0x1000, 0)); + CHECK(afkv2_owned_span(UINT64_MAX - 15, 16, UINT64_MAX, 1)); + CHECK(!afkv2_owned_span(UINT64_MAX - 15, 17, UINT64_MAX, 1)); + struct afkv2_command callback = { .oob = true, .tx_size = 96, + .rx_size = 144, .value = 144 }; + size_t input = 0, output = 0; + + CHECK(!afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + CHECK(input == 96 && output == 144); + callback.value = 143; + CHECK(afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + callback.value = 2048; + callback.rx_size = 2048; + CHECK(afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + callback.rx_size = 67; + CHECK(afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + callback.rx_size = 144; + callback.tx_size = 67; + CHECK(afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + callback.oob = false; + callback.inline_size = 96; + CHECK(!afkv2_callback_lengths(&callback, 68, 1024, &input, &output)); + CHECK(input == 96 && output == 96); + CHECK(afkv2_callback_lengths(NULL, 68, 1024, &input, &output)); + CHECK(afkv2_callback_lengths(&callback, 68, 1024, NULL, &output)); + CHECK(afkv2_callback_lengths(&callback, 68, 1024, &input, NULL)); + CHECK(afkv2_callback_lengths(&callback, 1025, 1024, &input, &output)); + printf("PASS %u assertions\n", assertions); + return 0; +} diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-edid.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-edid.py new file mode 100644 index 00000000000000..0d554d1c82b5da --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-edid.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +"""Verify legacy EDID tolerance and stricter external-inline parsing separately.""" +import pathlib +import re +import sys + +source = pathlib.Path(sys.argv[1]).read_text() + +def function(name): + match = re.search(r"static const struct drm_edid \*" + name + r"\([^;]*?\)\n\{", source) + if not match: + raise RuntimeError("production reader missing: " + name) + end, depth = match.end(), 1 + while depth: + depth += (source[end] == "{") - (source[end] == "}") + end += 1 + return source[match.start():end] + +assert "service->ep->wire_v2 ? dcpavserv_read_edid_inline(service)" in source +prefix = r''' +#undef NDEBUG +#include +#include +#include +#include +#include "epic/dpavserv-edid.h" +#define GFP_KERNEL 0 +#define kzalloc(n,f) calloc(1,n) +#define cpu_to_le64(x) (x) +#define le64_to_cpu(x) (x) +#define EDID_EXT_BLOCK_COUNT_OFFSET 0x7e +#define ERR_PTR(x) ((void*)(intptr_t)(x)) +#define IS_ERR(p) ((uintptr_t)(p)>(uintptr_t)-4096) +static void cleanup_free(void *p) { free(*(void**)p); } +#define __free(x) __attribute__((cleanup(cleanup_free))) +struct apple_epic_service { int unused; }; +struct drm_edid { size_t size; }; +struct dpavserv_copy_edid_cmd { u64 max_size;u8 pad1[24];u64 used_size;u8 pad2[8]; }; +struct dpavserv_copy_edid_resp { u64 max_size;u8 pad1[24];u64 used_size;u8 pad2[8];u8 data[]; }; +static u8 fixture[48+8+256]; +static bool truncated; +static const struct drm_edid *drm_edid_alloc(const void *p,size_t n) +{ struct drm_edid *e=malloc(sizeof(*e));(void)p;e->size=n;return e; } +static int afk_service_call(struct apple_epic_service *s,u16 group,u32 command, + const void *data,size_t data_len,size_t data_pad,void *output,size_t output_len,size_t output_pad) +{ + const struct dpavserv_copy_edid_cmd *cmd=data; + (void)s;(void)data_pad;(void)output_pad; + assert(group==1 && command==7 && data_len==48 && cmd->max_size==EDID_BUF_SIZE); + assert(output_len>=sizeof(fixture));memcpy(output,fixture,sizeof(fixture));return 0; +} +static int afk_service_call_len(struct apple_epic_service *s,u16 group,u32 command, + const void *data,size_t data_len,size_t data_pad,void *output,size_t output_len, + size_t output_pad,size_t *received) +{ int ret=afk_service_call(s,group,command,data,data_len,data_pad,output,output_len,output_pad); + *received=sizeof(fixture)-(truncated?1:0);return ret; } +static void fix_checksum(u8 *p) +{ unsigned int i;u8 sum=0;p[127]=0;for(i=0;i<127;i++)sum+=p[i];p[127]=-sum; } +''' +suffix = r''' +int main(void) +{ + struct apple_epic_service service={0};const struct drm_edid *e; + u8 *base=fixture+56,*extension=base+128; + afkv2_put64(fixture,EDID_BUF_SIZE);afkv2_put64(fixture+32,8+256); + memset(base+1,255,6);base[126]=1;fix_checksum(base); + extension[0]=2; /* deliberately invalid extension checksum */ + e=dcpavserv_read_edid_legacy(&service);assert(!IS_ERR(e) && e->size==256);free((void*)e); + e=dcpavserv_read_edid_inline(&service);assert(IS_ERR(e)); + fix_checksum(extension); + e=dcpavserv_read_edid_inline(&service);assert(!IS_ERR(e) && e->size==256);free((void*)e); + truncated=true; + e=dcpavserv_read_edid_inline(&service);assert(IS_ERR(e)); + e=dcpavserv_read_edid_legacy(&service);assert(!IS_ERR(e));free((void*)e); + puts("PASS: actual legacy EDID checksum tolerance preserved; inline checksum/actual-length validation enforced"); +} +''' +pathlib.Path(sys.argv[2]).write_text(prefix + function("dcpavserv_read_edid_legacy") + function("dcpavserv_read_edid_inline") + suffix) diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-notify.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-notify.py new file mode 100644 index 00000000000000..481aac7d0246a5 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-notify.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +"""Exercise actual late-EDID publication notification guards with mock work.""" +import pathlib +import re +import sys +source = pathlib.Path(sys.argv[1]).read_text() +match = re.search(r"static void dcpavserv_notify_edid\([^;]*?\)\n\{", source) +if not match: + raise RuntimeError("EDID notification helper missing") +end, depth = match.end(), 1 +while depth: + depth += (source[end] == "{") - (source[end] == "}") + end += 1 +prefix = r''' +#undef NDEBUG +#include +#include +#include +#include +#define READ_ONCE(x) (x) +#define DRM_CONNECTOR_REGISTERED 1 +struct base { int registration_state; }; +struct apple_connector { bool connected;struct base base;void *dcp;int hotplug_wq; }; +struct apple_dcp { struct apple_connector *connector;bool quiescing;unsigned int nr_modes;void *dev; }; +struct ep { struct apple_dcp *dcp;bool wire_v2; }; +struct apple_epic_service { struct ep *ep; }; +static unsigned int queued; +static void *to_platform_device(void *p) { return p; } +static void schedule_work(int *w) { (*w)++;queued++; } +''' +suffix = r''' +int main(void) +{ + int device,other;struct apple_dcp dcp={.dev=&device}; + struct ep ep={.dcp=&dcp,.wire_v2=true};struct apple_epic_service service={.ep=&ep}; + struct apple_connector connector={.dcp=&device}; + dcpavserv_notify_edid(&service);assert(queued==0); + dcp.connector=&connector;dcpavserv_notify_edid(&service);assert(queued==0); + dcp.nr_modes=1;connector.connected=true;dcpavserv_notify_edid(&service);assert(queued==0); + connector.base.registration_state=DRM_CONNECTOR_REGISTERED; + dcpavserv_notify_edid(&service);assert(queued==1 && connector.hotplug_wq==1); + dcp.quiescing=true;dcpavserv_notify_edid(&service);assert(queued==1); + dcp.quiescing=false;connector.dcp=&other;dcpavserv_notify_edid(&service);assert(queued==1); + connector.dcp=&device;ep.wire_v2=false;dcpavserv_notify_edid(&service);assert(queued==1); + ep.wire_v2=true;connector.connected=false;dcpavserv_notify_edid(&service);assert(queued==1); + puts("PASS: actual late-EDID notification queues ready connector only; legacy/quiesce/unrouted/unregistered guarded"); +} +''' +pathlib.Path(sys.argv[2]).write_text(prefix + source[match.start():end] + suffix) diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-owned.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-owned.py new file mode 100644 index 00000000000000..21fe63b6d3823c --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-owned.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Compile the production ownership checker with host-only synchronization mocks.""" +import hashlib +from pathlib import Path +import sys + + +def main(): + source = Path(sys.argv[1]).read_text(encoding="utf-8") + output = Path(sys.argv[2]) + first = "static int afk_v2_access_owned(" + last = "static void afk_v2_callback(" + if source.count(first) != 1 or source.count(last) != 1: + raise SystemExit("FAIL: ownership function boundaries changed; review extractor") + start = source.index(first) + end = source.index(last, start) + body = source[start:end] + if "int afk_v2_copy_owned(" not in body: + raise SystemExit("FAIL: production ownership wrapper missing") + directory = Path(__file__).resolve().parent + output.write_text((directory / "owned-prefix.c").read_text() + body + + (directory / "owned-suffix.c").read_text(), encoding="utf-8") + print("Production ownership function SHA256:", hashlib.sha256(body.encode()).hexdigest()) + + +if __name__ == "__main__": + main() diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-ring.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-ring.py new file mode 100644 index 00000000000000..b55eaea38c8d78 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-ring.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +"""Check full AFKV2 frame size against the production ring-admission block.""" +import pathlib +import sys + +source = pathlib.Path(sys.argv[1]).read_text() +start = source.index("\ttotal_epic_size = sizeof(*ehdr) +", source.index("static int afk_send_epic_token", source.index("int afk_receive_message"))) +end = source.index("\n\thdr = hdr2 = NULL;", start) +block = source[start:end] +prefix = r''' +#undef NDEBUG +#include +#include +#include "afkv2-codec.h" +#define IS_ALIGNED(x,a) (!((x)&((a)-1))) +#define ALIGN(x,a) (((x)+(a)-1)&~((a)-1)) +struct epic_hdr { u8 data[16]; }; +struct epic_sub_hdr { u8 data[24]; }; +struct epic_sub_hdr_h17p { u8 data[8]; }; +struct afk_qe { u8 data[16]; }; +struct ring { bool ready; u32 stride,bufsz; }; +struct ep { bool wire_v2; struct ring txbfr; }; +static int check_ring(struct ep *ep,u32 rptr,u32 wptr,size_t payload_len,size_t *total) +{ + struct epic_hdr *ehdr;struct epic_sub_hdr *eshdr;struct afk_qe *hdr; + size_t total_epic_size,total_size;int ret=0; +''' +suffix = r''' + *total=total_size; +out: + return ret; +} +int main(void) +{ + u8 request[112]={0},command[120],frame[144];size_t written,total; + struct ep ep={.wire_v2=true,.txbfr={.ready=true,.stride=64,.bufsz=256}}; + assert(!afkv2_encode_inline_command(command,sizeof(command),1,request,sizeof(request),32888)); + assert(!afkv2_encode_frame(frame,sizeof(frame),0,1,0,0xc0,AFKV2_COMMAND,0,command,sizeof(command),&written)); + assert(written==144 && written+sizeof(struct afk_qe)==160); + assert(!check_ring(&ep,0,0,sizeof(command),&total)); + assert(total==160 && ALIGN(total,64)==192); + ep.txbfr.bufsz=192;assert(check_ring(&ep,0,0,sizeof(command),&total)==-ENOSPC); + ep.txbfr.bufsz=128;assert(check_ring(&ep,0,0,sizeof(command),&total)==-EPROTO); + ep.txbfr.bufsz=256;assert(check_ring(&ep,192,0,sizeof(command),&total)==-ENOSPC); + assert(check_ring(&ep,0,1,sizeof(command),&total)==-EPROTO); + puts("PASS: full request 160 bytes / 192-byte aligned ring consumption; 256-byte empty ring accepted, smaller capacity rejected"); +} +''' +pathlib.Path(sys.argv[2]).write_text(prefix + block + suffix) diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-send.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-send.py new file mode 100644 index 00000000000000..70dbd5d3a69f4f --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-send.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +"""Compile the real command sender against synthetic ring and DMA operations.""" +import pathlib +import re +import sys + +source = pathlib.Path(sys.argv[1]).read_text() +match = re.search(r"static int afk_send_command_timeout_len\([^;]*?\)\n\{", source) +if not match: + raise RuntimeError("production command sender not found") +start, end, depth = match.start(), match.end(), 1 +while depth: + depth += (source[end] == "{") - (source[end] == "}") + end += 1 +reply_match = re.search(r"static void afk_recv_handle_reply\([^;]*?\)\n\{", source) +if not reply_match: + raise RuntimeError("production reply completer not found") +reply_end, reply_depth = reply_match.end(), 1 +while reply_depth: + reply_depth += (source[reply_end] == "{") - (source[reply_end] == "}") + reply_end += 1 +prefix = r''' +#undef NDEBUG +#include +#include +#include +#include +#include "afkv2-codec.h" +#define SZ_1M 1048576U +#define GFP_KERNEL 0 +#define MAX_PENDING_CMDS 16 +#define AFKV2_INTERFACES (1U << 16) +#define lockdep_assert_held(l) ((void)(l)) +#define cpu_to_le32(x) (x) +#define le32_to_cpu(x) (x) +#define cpu_to_le64(x) (x) +#define kmalloc(n,f) malloc(n) +#define kfree(p) free(p) +#define READ_ONCE(x) (x) +#define spin_lock_irqsave(l,f) ((void)(l),(f)=0) +#define spin_unlock_irqrestore(l,f) ((void)(l),(void)(f)) +#define dma_rmb() do {} while(0) +#define dev_warn(...) do {} while(0) +#define dev_err(...) do {} while(0) +#define complete(p) (*(p)=1) +#define EPIC_TYPE_COMMAND 3 +#define EPIC_CAT_COMMAND 0x30 +#define DECLARE_COMPLETION_ONSTACK(n) int n=0 +#define init_completion(p) (*(p)=0) +#define msecs_to_jiffies(x) (x) +#define wait_for_completion_timeout(p,t) ((void)(t),*(p)) +typedef u64 dma_addr_t; +struct epic_cmd { u32 retcode; u64 rxbuf,txbuf; u32 rxlen,txlen; u8 rxcookie,txcookie; }; +struct cmd_info { + u16 tag; u8 type; size_t reply_length; void *rxbuf,*txbuf; + dma_addr_t rxbuf_dma,txbuf_dma; size_t rxlen,txlen; u32 retcode; + bool done,free_on_ack; int *completion; +}; +struct dcp { void *dev; bool quiescing; }; +struct apple_dcp_afkep { struct dcp *dcp; bool wire_v2,inline_v2; int endpoint,lock;u8 *command_tags; }; +struct ops { const char *name; }; +struct apple_epic_service { + struct apple_dcp_afkep *ep; struct ops *ops; unsigned int channel; unsigned long cmd_map[1]; + int lock; unsigned int users; bool enabled,torndown; u8 cmd_tag; struct cmd_info cmds[16]; +}; +static struct apple_epic_service *active; +static unsigned int frees, sends; +static int wire_fail, timeout, reply_status; +static size_t owned_reply_size=32888; +static struct apple_epic_service *afk_service_get(struct apple_epic_service *s) +{ if(!s->enabled) return NULL; s->users++; return s; } +static void afk_service_put(struct apple_epic_service *s) { assert(s->users); s->users--; } +static struct apple_epic_service *afk_epic_find_service(struct apple_dcp_afkep *ep,u32 channel) +{ return active->ep==ep && active->channel==channel && (active->enabled || active->cmd_map[0]) ? active : NULL; } +static int test_bit(unsigned int bit,const unsigned long *map) { return !!(*map & (1UL<cmds[0];struct afkv2_command parsed; + assert(channel==4 && type==3 && category==0x30 && subtype==0xc0); + sends++; + /* The payload-envelope mock bounds the command separately from its response. */ + assert(length<=128); + if(ep->inline_v2) { + assert(length==120); + assert(!afkv2_decode_command(payload,length,&parsed)); + assert(!parsed.oob && parsed.id==tag && parsed.value==32888 && parsed.inline_size==112); + assert(parsed.inline_data[0]==0x5a); + } else { + const struct epic_cmd *legacy=payload; + assert(length==sizeof(*legacy) && legacy->txlen==112 && legacy->rxlen==32888); + } + if(wire_fail)return -ENOSPC; + if(timeout)return 0; + c->reply_length=ep->inline_v2?4:owned_reply_size;c->retcode=reply_status; + memset(c->rxbuf,0x77,c->reply_length);c->done=true;*c->completion=1; + return 0; +} +''' +suffix = r''' +int main(void) +{ + u8 tags[AFKV2_INTERFACES]={0}; + struct dcp dcp={0};struct apple_dcp_afkep ep={.dcp=&dcp,.wire_v2=true,.inline_v2=true,.command_tags=tags}; + struct ops ops={.name="test"};struct apple_epic_service s={.ep=&ep,.ops=&ops,.channel=4,.enabled=true,.users=1}; + u8 request[112]={0x5a},*output=calloc(1,32888);size_t received;u32 status=99; + active=&s; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(received==4 && status==0 && output[0]==0x77 && frees==2 && sends==1 && !s.cmd_map[0]); + /* Existing strict callers still reject a short inline result. */ + assert(afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,NULL)==-EMSGSIZE); + reply_status=5;received=99; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(status==5 && received==0);reply_status=0; + wire_fail=1; + assert(afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)==-ENOSPC && !s.cmd_map[0]); + wire_fail=0;ep.inline_v2=false; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(received==32888 && status==0); + /* Length-aware owned OOB callers copy only the bounded actual reply. */ + owned_reply_size=4;memset(output,0x19,32888); + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(received==4 && output[0]==0x77 && output[3]==0x77 && output[4]==0x19 && s.users==1); + memset(output,0x19,32888); + assert(afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,NULL)==-EMSGSIZE); + assert(output[0]==0x19 && !s.cmd_map[0]); + owned_reply_size=0;received=99; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(received==0 && output[0]==0x19); + /* Legacy framing retains its full-buffer copy contract. */ + ep.wire_v2=false;owned_reply_size=4; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(received==32888 && s.users==1); + ep.wire_v2=true;owned_reply_size=32888;timeout=1; + assert(afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)==-ETIMEDOUT); + assert(s.cmd_map[0] && s.cmds[0].free_on_ack && !s.cmds[0].completion); + /* Execute the real late-ACK completer: mismatch/duplicate cannot free DMA. */ + unsigned int before_late=frees;u16 retained_tag=s.cmds[0].tag; + struct epic_cmd reply={.retcode=0}; + s.enabled=false;s.torndown=true; /* Retired service still owns the pending slot. */ + afk_recv_handle_reply(&ep,4,retained_tag^0x10,&reply,sizeof(reply)); + assert(frees==before_late && s.cmd_map[0] && !s.cmds[0].done && s.users==1); + afk_recv_handle_reply(&ep,4,retained_tag,&reply,sizeof(reply)); + assert(frees==before_late+2 && !s.cmd_map[0] && s.cmds[0].done && s.users==1); + afk_recv_handle_reply(&ep,4,retained_tag,&reply,sizeof(reply)); + assert(frees==before_late+2); + s.enabled=true;s.torndown=false;timeout=0; + assert(!afk_send_command_timeout_len(&s,0xc0,request,112,output,32888,&status,1000,&received)); + assert(!s.cmd_map[0] && s.users==1);free(output); + puts("PASS: extracted production sender, 128-byte command envelope / 32888-byte reply, short OOB, strict/legacy callers and actual late-ACK ownership"); +} +''' +tag_match = re.search(r"static u16 afk_command_tag\([^;]*?\)\n\{", source) +if not tag_match: + raise RuntimeError("production interface tag allocator not found") +tag_end, tag_depth = tag_match.end(), 1 +while tag_depth: + tag_depth += (source[tag_end] == "{") - (source[tag_end] == "}") + tag_end += 1 +pathlib.Path(sys.argv[2]).write_text(prefix + source[tag_match.start():tag_end] + source[start:end] + source[reply_match.start():reply_end] + suffix) diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-service.py b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-service.py new file mode 100644 index 00000000000000..ab95f5c422710e --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/extract-service.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +"""Exercise the production service-call body with a mocked wire reply.""" +import pathlib +import re +import sys + +source = pathlib.Path(sys.argv[1]).read_text() +match = re.search(r"static int afk_service_call_timeout_len\([^;]*?\)\n\{", source) +if not match: + raise RuntimeError("production service-call definition not found") +start = match.start() +end = match.end() +depth = 1 +while depth: + depth += (source[end] == "{") - (source[end] == "}") + end += 1 +prefix = r''' +#undef NDEBUG +#include +#include +#include +#include "afkv2-codec.h" +#define GFP_KERNEL 0 +#define kzalloc(n,f) calloc(1,n) +#define kfree(p) free(p) +#define max(a,b) ((a)>(b)?(a):(b)) +#define cpu_to_le16(x) (x) +#define cpu_to_le32(x) (x) +#define le16_to_cpu(x) (x) +#define le32_to_cpu(x) (x) +#define EPIC_SERVICE_CALL_MAGIC 0x69706378 +#define EPIC_SUBTYPE_STD_SERVICE 0xc0 +#define dev_warn(...) do {} while(0) +#define dev_dbg(...) do {} while(0) +struct epic_service_call { + u8 pad0[2]; u16 group; u32 command, data_len, magic; u8 pad1[48]; +}; +struct dcp { void *dev; }; +struct ops { const char *name; }; +struct ep { struct dcp *dcp; int endpoint; bool inline_v2; }; +struct apple_epic_service { struct ep *ep; struct ops *ops; u32 channel; }; +static int error_status, malformed, short_header, too_long, wire_error; +static int afk_send_command_timeout_len(struct apple_epic_service *s, u8 type, + const void *payload, size_t payload_len, void *output, size_t output_len, + u32 *retcode, unsigned int timeout_ms, size_t *received) +{ + struct epic_service_call *call=output; + (void)payload; + assert(payload_len==(s->ep->inline_v2?68:output_len)); + assert(call->data_len==(s->ep->inline_v2?4:4100)); + assert(type==0xc0 && timeout_ms==1000 && output_len>=68); + if(wire_error) return -ETIMEDOUT; + call->group=1;call->command=7;call->magic=EPIC_SERVICE_CALL_MAGIC; + call->data_len=too_long?5:4; + ((u8*)output)[64]=0xde;((u8*)output)[65]=0xad; + ((u8*)output)[66]=0xbe;((u8*)output)[67]=0xef; + if(malformed)call->command=8; + *retcode=error_status; + if(received)*received=short_header?63:68; + return 0; +} +''' +suffix = r''' +int main(void) +{ + struct ep ep={.inline_v2=true};struct apple_epic_service service={.ep=&ep}; + u8 input[4]={0}, output[128]={0}; size_t received=999; + assert(!afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)); + assert(received==4 && output[0]==0xde && output[3]==0xef && output[4]==0); + error_status=5;received=999; + assert(afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)==-EINVAL && received==0); + error_status=0;malformed=1; + assert(afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)); + malformed=0;short_header=1; + assert(afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)); + short_header=0;too_long=1; + assert(afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)); + too_long=0;wire_error=1; + assert(afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)==-ETIMEDOUT); + wire_error=0; ep.inline_v2=false; + received=999; + assert(!afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,&received)); + assert(received==4 && output[0]==0xde && output[3]==0xef && output[4]==0); + assert(!afk_service_call_timeout_len(&service,1,7,input,4,4096,output,128,0,1000,NULL)); + puts("PASS: extracted production service-call actual request / OOB padding and short reply / legacy / status / context / length tests"); +} +''' +pathlib.Path(sys.argv[2]).write_text(prefix + source[start:end] + suffix) diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/errno.h b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/errno.h new file mode 100644 index 00000000000000..23e11801f1e763 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/errno.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#define EINVAL 22 +#define ENOSPC 28 +#define E2BIG 7 +#define EPROTO 71 +#define EIO 5 +#define EMSGSIZE 90 +#define ENODEV 19 +#define ETIMEDOUT 110 +#define ENOMEM 12 diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/string.h b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/string.h new file mode 100644 index 00000000000000..e74214a66ed831 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/string.h @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#include diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/types.h b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/types.h new file mode 100644 index 00000000000000..4d490ef828ccdb --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/host-include/linux/types.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +#include +#include +#include +typedef uint8_t u8; +typedef uint16_t u16; +typedef uint32_t u32; +typedef uint64_t u64; diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/inline-edid.c b/tools/testing/selftests/drivers/gpu/apple-afkv2/inline-edid.c new file mode 100644 index 00000000000000..b711adf8047944 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/inline-edid.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +#undef NDEBUG +#include +#include +#include "epic/dpavserv-edid.h" + +static unsigned int assertions; +#define CHECK(x) do { assertions++; assert(x); } while (0) + +static void checksum(u8 *block) +{ + u8 sum = 0; + unsigned int i; + + block[127] = 0; + for (i = 0; i < 127; i++) + sum += block[i]; + block[127] = -sum; +} + +int main(void) +{ + u8 command[192], packet[216], body[136] = {0}; + u8 edid[DPAV_EDID_HEADER_SIZE + 8 + 3 * 128] = {0}; + struct afkv2_frame frame; + struct afkv2_command reply; + size_t written, length = 0; + u8 *base = edid + DPAV_EDID_HEADER_SIZE + 8; + + CHECK(!afkv2_encode_inline_command(command, sizeof(command), 0x91, + body, sizeof(body), 400)); + CHECK(command[0] == 0 && command[1] == 0x91); + CHECK(afkv2_get32(command + 4) == 400); + CHECK(!afkv2_encode_frame(packet, sizeof(packet), 5, 4, 0, 0xc0, + AFKV2_RESPONSE, 0, command, 8 + sizeof(body), &written)); + /* Response uses status in the request-capacity field. */ + afkv2_put32(packet + AFKV2_HEADER_SIZE + 4, 0); + CHECK(!afkv2_decode_frame(packet, written, false, &frame)); + CHECK(!afkv2_decode_command(frame.payload, frame.payload_size, &reply)); + CHECK(!afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 135)); + CHECK(afkv2_check_inline_response(&frame, &reply, 5, 0xc0, 0x91, 136)); + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x92, 136)); + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc1, 0x91, 136)); + frame.category = AFKV2_COMMAND; + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + frame.category = AFKV2_RESPONSE; + frame.token = 1; + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + frame.token = 0; + /* R30 responses carry these opaque nonzero header fields. */ + frame.transport_flags = 0x18; + frame.interface_flags = 0x0400; + frame.reserved = 0x3607ebe0; + CHECK(!afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + /* Context and bounds remain mandatory even with opaque header values. */ + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x92, 136)); + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 135)); + frame.transport_flags = 0; + frame.interface_flags = 0; + frame.reserved = 0; + reply.oob = true; + CHECK(afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + reply.oob = false; + /* The original OOB matcher must continue rejecting inline replies. */ + CHECK(afkv2_check_response(&frame, &reply, 4, 0xc0, 0x91, 0, 136, 0, 0)); + reply.value = 7; + CHECK(!afkv2_check_inline_response(&frame, &reply, 4, 0xc0, 0x91, 136)); + CHECK(reply.value != 0); /* Error status is preserved for the caller. */ + CHECK(afkv2_encode_inline_command(command, 7, 1, body, 1, 1)); + CHECK(afkv2_encode_inline_command(command, sizeof(command), 1, NULL, 1, 1)); + CHECK(afkv2_encode_inline_command(command, sizeof(command), 1, body, 1, + AFKV2_MAX_PAYLOAD + 1)); + + afkv2_put16(body + 2, 1); + afkv2_put32(body + 4, 7); + afkv2_put32(body + 8, 72); + afkv2_put32(body + 12, 0x69706378); + CHECK(!afkv2_service_reply_size(body, 136, 1, 7, 72, &length) && length == 72); + CHECK(afkv2_service_reply_size(body, 135, 1, 7, 72, &length)); + CHECK(afkv2_service_reply_size(body, 136, 1, 7, 71, &length)); + CHECK(afkv2_service_reply_size(body, 63, 1, 7, 72, &length)); + CHECK(afkv2_service_reply_size(body, 136, 2, 7, 72, &length)); + CHECK(afkv2_service_reply_size(body, 136, 1, 8, 72, &length)); + body[12] ^= 1; + CHECK(afkv2_service_reply_size(body, 136, 1, 7, 72, &length)); + + afkv2_put64(edid, EDID_BUF_SIZE); + afkv2_put64(edid + 32, 8 + 128); + memset(base + 1, 255, 6); + base[18] = 1; + base[19] = 4; + checksum(base); + CHECK(!dpavserv_edid_size(edid, 48 + 8 + 128, &length) && length == 128); + CHECK(dpavserv_edid_size(edid, 47, &length)); + CHECK(dpavserv_edid_size(edid, 48 + 8 + 127, &length)); + base[0] = 1; + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + base[0] = 0; + base[24] ^= 1; + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + base[24] ^= 1; + base[126] = 1; + checksum(base); + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + afkv2_put64(edid + 32, 8 + 256); + CHECK(!dpavserv_edid_size(edid, 48 + 8 + 256, &length) && length == 256); + /* HF-EEODB can supply more complete blocks than base[126] declares. */ + base[126] = 0; + checksum(base); + CHECK(!dpavserv_edid_size(edid, sizeof(edid), &length) && length == 256); + afkv2_put64(edid + 32, 8 + 255); + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + afkv2_put64(edid + 32, EDID_BUF_SIZE + 1); + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + afkv2_put64(edid + 32, 8 + 128); + afkv2_put64(edid, EDID_BUF_SIZE - 1); + CHECK(dpavserv_edid_size(edid, sizeof(edid), &length)); + printf("PASS: inline AFKV2 and EDID validation (%u checks)\n", assertions); + return 0; +} diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-prefix.c b/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-prefix.c new file mode 100644 index 00000000000000..8ff88ec1ec5b00 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-prefix.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Aurora Silicon */ +#undef NDEBUG +#include +#include +#include "afkv2-codec.h" +#define MAX_PENDING_CMDS 16 +#define SZ_1M 1048576U +#define EACCES 13 +#define spin_lock_irqsave(lock, flags) \ + do { \ + (void)(lock); \ + (flags) = 0; \ + } while (0) +#define spin_unlock_irqrestore(lock, flags) \ + do { \ + (void)(lock); \ + (void)(flags); \ + } while (0) +#define dma_rmb() \ + do { \ + } while (0) +#define dma_wmb() \ + do { \ + } while (0) +#define dev_warn_ratelimited(...) \ + do { \ + } while (0) +static int test_bit(unsigned int bit, const unsigned long *map) +{ + return !!(*map & (1UL << bit)); +} + +struct epic_cmd_info { + bool done; + void *rxbuf, *txbuf; + u64 rxbuf_dma, txbuf_dma; + size_t rxlen, txlen; +}; + +struct apple_dcp_afkep { + bool wire_v2; +}; + +struct apple_epic_service { + int lock; + struct apple_dcp_afkep *ep; + unsigned long cmd_map[1]; + struct epic_cmd_info cmds[MAX_PENDING_CMDS]; +}; diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-suffix.c b/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-suffix.c new file mode 100644 index 00000000000000..770b16bc2fa944 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/owned-suffix.c @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* Copyright 2026 Aurora Silicon */ +int main(void) +{ + u8 rx[16], tx[16], output[16]; + struct apple_dcp_afkep ep = { .wire_v2 = true }; + struct apple_epic_service service = { .ep = &ep }; + struct epic_cmd_info *cmd = &service.cmds[0]; + + memset(rx, 0x12, sizeof(rx)); + memset(tx, 0x34, sizeof(tx)); + memset(output, 0x56, sizeof(output)); + cmd->rxbuf = rx; + cmd->rxbuf_dma = 0x10000001000ULL; + cmd->rxlen = sizeof(rx); + cmd->txbuf = tx; + cmd->txbuf_dma = 0x10000002000ULL; + cmd->txlen = sizeof(tx); + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma, output, 4, false) == + -EACCES); + service.cmd_map[0] = 1; + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma + 12, output, 4, + false) == 0); + assert(output[0] == 0x12 && output[3] == 0x12 && output[4] == 0x56); + assert(afk_v2_copy_owned(&service, cmd->txbuf_dma + 4, output, 4, + false) == 0); + assert(output[0] == 0x34 && output[4] == 0x56); + memset(output, 0x78, 4); + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma + 8, output, 4, + true) == 0); + assert(rx[7] == 0x12 && rx[8] == 0x78 && rx[11] == 0x78 && + rx[12] == 0x12); + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma - 1, output, 1, + true) == -EACCES); + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma + 13, output, 4, + true) == -EACCES); + assert(afk_v2_copy_owned(&service, cmd->txbuf_dma + 16, output, 1, + true) == -EACCES); + cmd->done = true; + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma, output, 1, false) == + -EACCES); + cmd->done = false; + ep.wire_v2 = false; + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma, output, 1, false) == + -EINVAL); + ep.wire_v2 = true; + assert(afk_v2_copy_owned(&service, cmd->rxbuf_dma, NULL, 1, false) == + -EINVAL); + assert(afk_v2_copy_owned(&service, 0, NULL, 0, false) == 0); + assert(afk_v2_access_owned(&service, cmd->rxbuf_dma + 1, NULL, 15, true, + false) == 0); + assert(afk_v2_access_owned(&service, cmd->rxbuf_dma + 1, NULL, 16, true, + false) == -EACCES); + puts("PASS owned-copy boundaries (actual extracted function; lock/barrier mocks)"); + return 0; +} diff --git a/tools/testing/selftests/drivers/gpu/apple-afkv2/run-afkv2-tests.sh b/tools/testing/selftests/drivers/gpu/apple-afkv2/run-afkv2-tests.sh new file mode 100755 index 00000000000000..4348eebc78246e --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-afkv2/run-afkv2-tests.sh @@ -0,0 +1,82 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only +set -eu + +afkv2_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +afkv2_root=${AFKV2_SOURCE_ROOT:-$(CDPATH= cd -- "$afkv2_dir/../../../../../.." && pwd)} +afkv2_cc=${HOSTCC:-${CC:-cc}} +afkv2_cc_command=${afkv2_cc%% *} +afkv2_python=${PYTHON:-python3} +if ! command -v "$afkv2_cc_command" >/dev/null 2>&1 || + ! command -v "$afkv2_python" >/dev/null 2>&1; then + echo "SKIP: a host C compiler and Python 3 are required" + exit 4 +fi +if [ ! -f "$afkv2_root/drivers/gpu/drm/apple/afkv2-codec.h" ]; then + echo "SKIP: run from a source tree containing the AFKV2 codec" + exit 4 +fi +afkv2_tmp=$(mktemp -d "${TMPDIR:-/tmp}/apple-afkv2.XXXXXX") +trap 'rm -rf -- "$afkv2_tmp"' EXIT +trap 'exit 1' HUP INT TERM +afkv2_sanitizers= +if [ "${AFKV2_SANITIZERS:-0}" = 1 ]; then + afkv2_sanitizers="-fsanitize=address,undefined -fno-omit-frame-pointer" +fi + +# Compiler arguments, CFLAGS and sanitizers use normal compiler word splitting. +# shellcheck disable=SC2086 +$afkv2_cc -std=c11 -Wall -Wextra -Werror -pedantic ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_dir/codec.c" -o "$afkv2_tmp/codec" +"$afkv2_tmp/codec" +"$afkv2_python" "$afkv2_dir/extract-owned.py" \ + "$afkv2_root/drivers/gpu/drm/apple/afk.c" "$afkv2_tmp/owned.c" +# The extracted kernel function uses GNU C void-pointer arithmetic. +# shellcheck disable=SC2086 +$afkv2_cc -std=gnu11 -Wall -Wextra -Werror ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_tmp/owned.c" -o "$afkv2_tmp/owned" +"$afkv2_tmp/owned" + +# Inline DPAV transport and actual-length EDID response validation. +# shellcheck disable=SC2086 +$afkv2_cc -std=c11 -Wall -Wextra -Werror -pedantic ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_dir/inline-edid.c" -o "$afkv2_tmp/inline-edid" +"$afkv2_tmp/inline-edid" +"$afkv2_python" "$afkv2_dir/extract-service.py" \ + "$afkv2_root/drivers/gpu/drm/apple/afk.c" "$afkv2_tmp/service.c" +# shellcheck disable=SC2086 +$afkv2_cc -std=gnu11 -Wall -Wextra -Werror ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_tmp/service.c" -o "$afkv2_tmp/service" +"$afkv2_tmp/service" + +"$afkv2_python" "$afkv2_dir/extract-send.py" \ + "$afkv2_root/drivers/gpu/drm/apple/afk.c" "$afkv2_tmp/send.c" +# shellcheck disable=SC2086 +$afkv2_cc -std=gnu11 -Wall -Wextra -Werror ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_tmp/send.c" -o "$afkv2_tmp/send" +"$afkv2_tmp/send" + +"$afkv2_python" "$afkv2_dir/extract-edid.py" \ + "$afkv2_root/drivers/gpu/drm/apple/epic/dpavservep.c" "$afkv2_tmp/edid.c" +# shellcheck disable=SC2086 +$afkv2_cc -std=gnu11 -Wall -Wextra -Werror ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_tmp/edid.c" -o "$afkv2_tmp/edid" +"$afkv2_tmp/edid" + +"$afkv2_python" "$afkv2_dir/extract-ring.py" \ + "$afkv2_root/drivers/gpu/drm/apple/afk.c" "$afkv2_tmp/ring.c" +"$afkv2_python" "$afkv2_dir/extract-notify.py" \ + "$afkv2_root/drivers/gpu/drm/apple/epic/dpavservep.c" "$afkv2_tmp/notify.c" +for afkv2_test in ring notify; do + # shellcheck disable=SC2086 + $afkv2_cc -std=gnu11 -Wall -Wextra -Werror ${CFLAGS:-} $afkv2_sanitizers \ + -I"$afkv2_dir/host-include" -I"$afkv2_root/drivers/gpu/drm/apple" \ + "$afkv2_tmp/$afkv2_test.c" -o "$afkv2_tmp/$afkv2_test" + "$afkv2_tmp/$afkv2_test" +done diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/Makefile b/tools/testing/selftests/drivers/gpu/apple-dcp/Makefile new file mode 100644 index 00000000000000..bbb39cada55bf8 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +TEST_PROGS := run.sh +TEST_FILES := run.py common.py edid.py recovery.py rate.py phy.py lifecycle.py pm.py idle.py boot.py dwc3 README +include ../../../lib.mk diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/README b/tools/testing/selftests/drivers/gpu/apple-dcp/README new file mode 100644 index 00000000000000..b04b6b9934193b --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/README @@ -0,0 +1,77 @@ +Apple external display host regressions +====================================== + +Run from a kernel source checkout: + + make -C tools/testing/selftests TARGETS=drivers/gpu/apple-dcp run_tests + +Or keep translation units, commands, failure logs and source hashes: + + tools/testing/selftests/drivers/gpu/apple-dcp/run.sh --output /tmp/apple-dcp-results + +An installed copy needs --source /path/to/kernel. --group edid, rate, phy or +lifecycle or boot selects one group. HOSTCC selects the host compiler (default clang). +Python 3 and a compiler supporting AddressSanitizer/UndefinedBehaviorSanitizer +are required. No target access, firmware command, captured fixture or kernel +configuration is required. + +One TAP entry point uses shared production extraction and compiler handling. +The modules contain public synthetic host boundaries and retained assertions: + +* edid: actual AFK admission and OPEN/CLOSE handling; native/HDMI providers + alternating over 64 epochs; foreign owners, stale replies, reader/command + pins, ordered receive progress and legacy service behavior. +* rate: actual binary parser, EDID/capacity policy, receipt ingress, getters + and lookup; all arrival orders, conflicting/discarded siblings, rejected + replacement catalogs, HPD retirement and recovery; malformed/duplicate + metadata and independent EDID decoding; eight bounded delayed retries. +* phy: actual parking/mux/init/reset/configure callbacks, masked PMA context, + faulted ACK/BIST cleanup, dirty-route retry, OFF/new-calibration epochs, + warm reset and legacy T8103/T8122/T8140/T8142 route controls. +* lifecycle: actual physical-disconnect authority and ownership despite failed + teardown; DWC3/core/PHY PM and reset-reference recovery; exact firmware + tuple/resource admission, unsupported VRR rejection and power reply bounds; + 15,552 D134 dispatcher scope/length/context and legacy reply cases. + +Transport, MMIO, kernel locks and work scheduling are host mocks. These tests +establish source behavior; electrical readiness, firmware/FDT ownership, +concurrency and system suspend require separate kernel/hardware qualification. +Kernel KUnit tests and apple-afkv2 remain separate tests in their existing +standard layouts. No archived diagnostic script is a dependency of this suite. + +The runner preserves host artifacts on failure as well as success. Existing +physical-disconnect fixture warnings for unused mock boundaries and signed +host/kernel ABI comparisons are handled only by those fixtures' original +compiler options; production source compilation uses the normal Kbuild policy. + +The bounded-PM group shares the PHY fixture and executes the actual PM get/put +and mux wrapper, usb_hc_died and typec_mux_set. The probe group executes real +CD321x register/unregister/rollback and retry-state functions. USB n/y/m uses +the actual optional dependency and a real Kconfig producer (KCONFIG_CONF may +select an existing producer). Legacy lane/teardown groups keep executing the +same locked mux body; they do not pretend their old USB mocks own runtime PM. +The cpuidle group executes the actual Apple table/probe, generic registration +and s2idle selector/entry, and tick_freeze/unfreeze. It checks source behavior +only; RTC, elapsed suspend time and simultaneous hardware execution remain +separate qualification gates. The original human cpuidle patch is preserved. + +The service-rebind group also runs retained reviewer controls through the +actual sender, receive worker, recovery, quiesce and EDID wrapper. It covers +CLOSED cancellation before lookup, first late ACK versus already consumed +duplicates, reader-pinned cross-slot/channel reuse, and the explicit physical +absence window. Per-interface sequence history has a fixed 64-KiB bound per +AFKV2 endpoint; reply routing still ignores reader-only retired services. + +Two wrap controls intentionally retain the command-17 replay assertion +failure, for OOB and inline. The group verifies that these counterexamples +still reproduce, alongside nonvacuous mutations and passing corrected safety +controls. A passing host suite does not establish a replay-exclusion contract +or admit a hardware trial. Token/sequence semantics remain a separate protocol +fact to establish; the tests do not manufacture a firmware echo guarantee. + +The boot group executes the actual OSLOG dispatcher with synthetic fixed-width +words. Known opaque event types remain visible, unknown types retain warnings, +and only the existing buffer-request path can allocate/ACK buffers. The EDID +service-rebind group also distinguishes intentional optional EOPNOTSUPP from +required admission failures and genuine optional-service errors, and proves +refused services never OPEN or publish. No captured firmware values are used. diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/accounting.py b/tools/testing/selftests/drivers/gpu/apple-dcp/accounting.py new file mode 100644 index 00000000000000..ce2791cc3a379c --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/accounting.py @@ -0,0 +1,318 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Source-connected accounting regressions retaining adverse-review assertions. + +SRCU, module and hardware leaves are models; the transaction, DCP and mux core +functions are extracted from the selected public source. No device is accessed. +""" +import quiesce +import transaction +from common import block + +def srcu_index(ROOT): + unit=quiesce.make_unit(ROOT) + main=block(unit,'int main(') + unit=unit.replace(main,main.replace('int main(','int __attribute__((unused)) prior_main(',1),1) + old_lock=block(unit,'static int srcu_read_lock(') + old_unlock=block(unit,'static void srcu_read_unlock(') + old_mutex_unlock=block(unit,'static void mutex_unlock(') + leaves=r''' + static pthread_mutex_t schedule_mutex=PTHREAD_MUTEX_INITIALIZER; + static pthread_cond_t schedule_cond=PTHREAD_COND_INITIALIZER; + static int srcu_bank,bank_readers[2],released_index=-1; + static bool pause_unlock,pause_seen,resume_unlock; + static _Thread_local int expected_index=-1; + static int srcu_read_lock(int *s) { + (void)s;pthread_mutex_lock(&schedule_mutex);int i=srcu_bank;bank_readers[i]++;expected_index=i;pthread_mutex_unlock(&schedule_mutex);return i; + } + static void srcu_read_unlock(int *s,int i) { + (void)s;pthread_mutex_lock(&schedule_mutex);released_index=i; + fprintf(stderr,"SRCU unlock: acquired=%d released=%d bank0=%d bank1=%d\n",expected_index,i,bank_readers[0],bank_readers[1]); + assert(i==expected_index&&bank_readers[i]>0);bank_readers[i]--;expected_index=-1; + pthread_cond_broadcast(&schedule_cond);pthread_mutex_unlock(&schedule_mutex); + } + static void mutex_unlock(struct mutex *m) { + assert(!pthread_mutex_unlock(&m->value)); + if(m==&shared_context.lock&&pause_unlock) { + pthread_mutex_lock(&schedule_mutex);pause_seen=true;pause_unlock=false;pthread_cond_broadcast(&schedule_cond); + while(!resume_unlock)pthread_cond_wait(&schedule_cond,&schedule_mutex); + pthread_mutex_unlock(&schedule_mutex); + } + } + ''' + unit=unit.replace(old_lock,'',1).replace(old_unlock,'',1).replace(old_mutex_unlock,leaves,1) + unit+=r''' + static void *first_actor(void *p) { + struct typec_mux *m=p;struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE};assert(typec_mux_set(m,&safe)==0);return NULL; + } + int main(void) { + struct cd321x c;initial(&c);mux.num_mux_devs=0;pause_unlock=true; + pthread_t thread;assert(!pthread_create(&thread,NULL,first_actor,&mux)); + pthread_mutex_lock(&schedule_mutex);while(!pause_seen)pthread_cond_wait(&schedule_cond,&schedule_mutex); + srcu_bank=1;pthread_mutex_unlock(&schedule_mutex); + struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE};assert(typec_mux_begin(&mux,&safe)==0); + assert(mux.srcu_index==1&&bank_readers[0]==1&&bank_readers[1]==1); + pthread_mutex_lock(&schedule_mutex);resume_unlock=true;pthread_cond_broadcast(&schedule_cond);pthread_mutex_unlock(&schedule_mutex); + assert(!pthread_join(thread,NULL));typec_mux_abort(&mux);assert(bank_readers[0]==0&&bank_readers[1]==0); + puts("PASS same-handle SRCU read-index balance across context unlock"); + } + ''' + return unit + +def resources(ROOT): + actual=next(u for n,u,c,f in transaction.controls(ROOT) if n=='actual-dcp-normal-safe') + main=block(actual,'int main(') + actual=actual.replace(main,main.replace('int main(','int __attribute__((unused)) prior_route_main(',1),1) + actual=actual.replace('struct mux_control { int unused; };',r''' + struct mux_control; + struct mux_ops { int (*set)(struct mux_control *,int); }; + struct mux_chip { const struct mux_ops *ops; }; + struct mux_control { int lock,idle_state,cached_state; unsigned last_change;struct mux_chip *chip; }; + ''',1) + muxcore=(ROOT/'drivers/mux/core.c').read_text() + resource=r''' + #define MUX_IDLE_AS_IS -1 + #define MUX_CACHE_UNKNOWN -2 + static int hw_calls,hw_failures,fixed_restore_calls,fixed_restore_failures; + static unsigned ktime_get(void) { return 0; } + static void up(int *s) { (*s)++; } + static int hw_set(struct mux_control *m,int state) { (void)m;(void)state;hw_calls++;return hw_calls<=hw_failures?-EIO:0; } + ''' + resource+=block(muxcore,'static int mux_control_set(')+'\n'+block(muxcore,'int mux_control_deselect(')+'\n' + actual=actual.replace(block(actual,'static int mux_control_deselect('),resource,1) + actual=actual.replace(block(actual,'static int mux_control_select('),r'''static int mux_control_select(struct mux_control *m,unsigned n) { (void)m;(void)n;fixed_restore_calls++;return fixed_restore_calls<=fixed_restore_failures?-EIO:0; }''',1) + actual=actual.replace('static int gpiod_get_value_cansleep(void *p) { (void)p;return 0; }', 'static int gpio_level;\nstatic int gpiod_get_value_cansleep(void *p) { (void)p;return gpio_level; }',1) + actual+=r''' + static int peer_error; + static int peer_finish(struct typec_mux_dev *p,struct typec_mux_state *s) { (void)p;(void)s;return peer_error; } + int main(int argc,char **argv) { + (void)argc;struct cd321x c;initial(&c); + struct mux_ops ops={.set=hw_set};struct mux_chip chip={.ops=&ops}; + struct mux_control xbar={.lock=0,.idle_state=0,.cached_state=1,.chip=&chip}; + struct mux_control fixed_xbar={.lock=1,.idle_state=0,.cached_state=0,.chip=&chip}; + route.xbar=route.active_xbar=&xbar; + struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE}; + if(argv[1][0]=='n') { + assert(typec_mux_set(&mux,&safe)==0&&xbar.lock==1&&!port.owner&&!port.safe_pending); + puts("PASS actual DCP finalizer + real mux deselect resource balance");return 0; + } + if(argv[1][0]=='l') { + phy_provider.finish=peer_finish;mux.mux_devs[0]=&display_provider;mux.mux_devs[1]=&phy_provider;peer_error=-EIO; + assert(typec_mux_set(&mux,&safe)==-EIO&&xbar.lock==1&&port.owner==&route&&port.safe_pending&&port.safe_finish_ready); + peer_error=0;assert(typec_mux_set(&mux,&safe)==0&&xbar.lock==1&&!port.owner&&!port.safe_pending); + puts("PASS successful DCP finalizer is not repeated after a later provider failure");return 0; + } + if(argv[1][0]=='h')hw_failures=1; + if(argv[1][0]=='f') {display.fixed_connector_type=2;display.hdmi_hpd=(void *)1;gpio_level=1;display.xbar=&fixed_xbar;fixed_restore_failures=1;} + assert(typec_mux_set(&mux,&safe)==-EIO&&xbar.lock==1&&port.owner==&route&&port.safe_pending&&!port.safe_finish_ready); + bool selected_after_error=route.selected; + int ret=typec_mux_set(&mux,&safe); + fprintf(stderr,"finalizer retry: kind=%c ret=%d permits=%d selected_after_error=%d owner=%p pending=%d hw_calls=%d fixed_calls=%d\n",argv[1][0],ret,xbar.lock,selected_after_error,(void *)port.owner,port.safe_pending,hw_calls,fixed_restore_calls); + assert(xbar.lock==1); + if(argv[1][0]=='h')assert(ret==-EIO&&hw_calls==1&&port.owner==&route&&port.safe_pending); + if(argv[1][0]=='f')assert(ret==0&&!port.owner&&!port.safe_pending&&fixed_restore_calls==2); + return 0; + } + ''' + return actual + +def getter_unit(ROOT): + getter=transaction.lifetime(ROOT) + main=block(getter,'int main(') + renamed=main.replace('int main(','int __attribute__((unused)) prior_lifetime_main(',1) + renamed=renamed[:-1]+'return 0;\n}' + getter=getter.replace(main,renamed,1) + getter=getter.replace('struct typec_mux_dev {', 'struct module { unsigned refs;bool going; };\nstruct device { unsigned refs; };\nstruct typec_mux_dev { struct module *owner;struct device dev;',1) + getter=getter.replace(block(getter,'static void typec_mux_put('),'void typec_mux_put(struct typec_mux *mux);',1) + getter=getter.replace('#include ',r''' + #include + static int allocation_count,allocation_failure; + static void *review_alloc(size_t size) { if(++allocation_count==allocation_failure)return NULL;return calloc(1,size); } + ''',1) + getter=getter.replace('#define kzalloc_obj(x) calloc(1,sizeof(x))','#define kzalloc_obj(x) review_alloc(sizeof(x))',1) + getter+=r''' + #undef IS_ERR + #define IS_ERR(p) ((uintptr_t)(p)>=(uintptr_t)-4095) + #define PTR_ERR(p) ((int)(intptr_t)(p)) + #define TYPEC_MUX_MAX_DEVS 3 + #define ARRAY_SIZE(a) (sizeof(a)/sizeof((a)[0])) + static struct typec_mux_dev graph_providers[2]; + static struct module modules[2]; + static bool graph_second_error; + static void *typec_mux_match(const struct fwnode_handle *f,const char *id,void *data) { (void)f;(void)id;(void)data;return NULL; } + static int fwnode_connection_find_matches(struct fwnode_handle *f,const char *id,void *data,void *(*fn)(const struct fwnode_handle *,const char *,void *),void **matches,unsigned max) { + (void)f;(void)id;(void)data;(void)fn;assert(max>=2); + graph_providers[0].dev.refs++;matches[0]=&graph_providers[0]; + if(graph_second_error)matches[1]=ERR_PTR(-EPROBE_DEFER); + else {graph_providers[1].dev.refs++;matches[1]=&graph_providers[1];} + return 2; + } + static bool try_module_get(struct module *m) { if(!m)return true;if(m->going)return false;m->refs++;return true; } + static void module_put(struct module *m) { if(m){assert(m->refs);m->refs--;} } + static void put_device(struct device *d) { assert(d->refs);d->refs--; } + ''' + getter='#define EPROBE_DEFER 517\n'+getter + mux=(ROOT/'drivers/usb/typec/mux.c').read_text() + getter+='\n'+block(mux,'struct typec_mux *fwnode_typec_mux_get(')+'\n'+block(mux,'void typec_mux_put(')+'\n' + getter+=r''' + static void graph_setup(void) { + graph_second_error=false;allocation_count=allocation_failure=0; + for(unsigned i=0;i<2;i++){modules[i]=(struct module){0};graph_providers[i]=(struct typec_mux_dev){.owner=&modules[i],.set=phy_set};} + } + static void graph_balanced(struct fwnode_handle *f) { + assert(!f->refs&&typec_mux_contexts.next==&typec_mux_contexts); + for(unsigned i=0;i<2;i++)assert(!modules[i].refs&&!graph_providers[i].dev.refs); + } + int main(void) { + struct fwnode_handle f={0};graph_setup(); + struct typec_mux *a=fwnode_typec_mux_get(&f),*b=fwnode_typec_mux_get(&f); + assert(!IS_ERR(a)&&!IS_ERR(b)&&a->context==b->context&&a->context->users==2&&f.refs==1); + for(unsigned i=0;i<2;i++)assert(modules[i].refs==2&&graph_providers[i].dev.refs==2); + typec_mux_put(a);typec_mux_put(b);graph_balanced(&f); + graph_setup();modules[1].going=true;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-ENODEV);graph_balanced(&f); + graph_setup();graph_second_error=true;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-EPROBE_DEFER);graph_balanced(&f); + graph_setup();allocation_failure=2;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-ENOMEM);graph_balanced(&f); + graph_setup();a=fwnode_typec_mux_get(&f);graph_providers[0].removed=true; + struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE};unsigned before=phy_calls; + assert(typec_mux_set(a,&safe)==-ENODEV&&phy_calls==before);typec_mux_put(a);graph_balanced(&f); + puts("PASS actual getter/context/module/device balance: same fwnode, failed module conversion, deferred provider, context allocation and stale-provider handle"); + return 0; + } + ''' + return getter + +def controls(root): + clock = transaction.clock_order(root).replace('if(parked)return 0;if(!backend_clock)', 'if(!backend_clock)', 1) + clock = clock.replace('assert(!port.owner&&!port.safe_pending);', 'assert(!port.owner&&!port.safe_pending&&phy_calls==1&&role_calls==2);', 1) + return [ + ('same-handle-srcu-index', srcu_index(root), ((),), ('-pthread',)), + ('mux-release-stages', resources(root), (('normal',), ('late',), ('hardware',), ('fixed',)), ('-pthread',)), + ('single-safe-before-role-exit', clock, ((),), ('-pthread',)), + ('mux-get-put-balance', getter_unit(root), ((),), ('-pthread',)), + ('lookup-unregister-module-lifetime', lookup_lifetime(root), ((),), ('-pthread',)), + ('kernel-module-conversion-balance', module_conversion(root), ((),), ('-pthread',)), + ] + +def lookup_lifetime(ROOT): + getter = getter_unit(ROOT) + mux = (ROOT / "drivers/usb/typec/mux.c").read_text() + module_unit=getter.replace(block(getter,'int main('),block(getter,'int main(').replace('int main(','int __attribute__((unused)) prior_getter_main(',1),1) + module_unit=module_unit.replace('struct module { unsigned refs;bool going; };','struct module { unsigned refs,refcnt;bool going; };',1) + module_unit=module_unit.replace('struct device { unsigned refs; };','struct device { unsigned refs;bool unregistered; };',1) + sync=r''' + static pthread_mutex_t conversion_mutex=PTHREAD_MUTEX_INITIALIZER; + static pthread_cond_t conversion_cond=PTHREAD_COND_INITIALIZER; + static bool lookup_ready,lookup_resume; + static unsigned active_readers; + static void synchronize_srcu(int *s) { (void)s;pthread_mutex_lock(&conversion_mutex);assert(active_readers==0);pthread_mutex_unlock(&conversion_mutex); } + static void device_unregister(struct device *d) { d->unregistered=true;assert(d->refs);d->refs--; } + static bool module_is_live(struct module *m) { return !m->going; } + static bool atomic_inc_not_zero(unsigned *p) { if(!*p)return false;(*p)++;return true; } + #define likely(x) (x) + #define trace_module_get(...) ((void)0) + #define _RET_IP_ 0 + ''' + pos=module_unit.index('static bool try_module_get(') + module_unit=module_unit[:pos]+sync+module_unit[pos:] + modulemain=(ROOT/'kernel/module/main.c').read_text() + module_unit=module_unit.replace(block(module_unit,'static bool try_module_get('),block(modulemain,'bool try_module_get('),1) + needle='return 2;\n }' + assert module_unit.count(needle)==1 + module_unit=module_unit.replace(needle,r''' + pthread_mutex_lock(&conversion_mutex);lookup_ready=true;pthread_cond_broadcast(&conversion_cond); + while(!lookup_resume)pthread_cond_wait(&conversion_cond,&conversion_mutex); + pthread_mutex_unlock(&conversion_mutex);return 2; + }''',1) + + declarations = r''' +static pthread_mutex_t conversion_mutex=PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t conversion_cond=PTHREAD_COND_INITIALIZER; +static bool lookup_ready,lookup_resume,draining; +static unsigned active_readers; +#define WRITE_ONCE(x,v) ((x)=(v)) +''' + start = module_unit.index('static pthread_mutex_t conversion_mutex=') + end = module_unit.index('static void synchronize_srcu(', start) + module_unit = module_unit[:start] + module_unit[end:] + pos = module_unit.index('static int srcu_read_lock(') + module_unit = module_unit[:pos] + declarations + module_unit[pos:] + module_unit = module_unit.replace(block(module_unit, 'static int srcu_read_lock('), r''' +static int srcu_read_lock(int *s) { (void)s;pthread_mutex_lock(&conversion_mutex);active_readers++;pthread_mutex_unlock(&conversion_mutex);return 0; } +''', 1) + module_unit = module_unit.replace(block(module_unit, 'static void srcu_read_unlock('), r''' +static void srcu_read_unlock(int *s,int index) { (void)s;assert(index==0);pthread_mutex_lock(&conversion_mutex);assert(active_readers);active_readers--;pthread_cond_broadcast(&conversion_cond);pthread_mutex_unlock(&conversion_mutex); } +''', 1) + module_unit = module_unit.replace(block(module_unit, 'static void synchronize_srcu('), r''' +static void synchronize_srcu(int *s) { (void)s;pthread_mutex_lock(&conversion_mutex);draining=true;pthread_cond_broadcast(&conversion_cond);while(active_readers)pthread_cond_wait(&conversion_cond,&conversion_mutex);pthread_mutex_unlock(&conversion_mutex); } +''', 1) + module_unit += '\n' + block(mux, 'void typec_mux_unregister(') + r''' +static struct typec_mux *lookup_result; +static void *getter_actor(void *node) { lookup_result=fwnode_typec_mux_get(node);return NULL; } +static void *unregister_actor(void *provider) { typec_mux_unregister(provider);return NULL; } +int main(void) { + struct fwnode_handle f={0};graph_setup(); + struct module *departing=calloc(1,sizeof(*departing));departing->going=true;graph_providers[0].owner=departing;graph_providers[0].dev.refs=1; + pthread_t getter_thread,remove_thread;assert(!pthread_create(&getter_thread,NULL,getter_actor,&f)); + pthread_mutex_lock(&conversion_mutex);while(!lookup_ready)pthread_cond_wait(&conversion_cond,&conversion_mutex); + assert(active_readers==1&&graph_providers[0].dev.refs==2);pthread_mutex_unlock(&conversion_mutex); + assert(!pthread_create(&remove_thread,NULL,unregister_actor,&graph_providers[0])); + pthread_mutex_lock(&conversion_mutex);while(!draining)pthread_cond_wait(&conversion_cond,&conversion_mutex); + assert(active_readers==1&&!graph_providers[0].dev.unregistered); + lookup_resume=true;pthread_cond_broadcast(&conversion_cond);pthread_mutex_unlock(&conversion_mutex); + assert(!pthread_join(getter_thread,NULL));assert(!pthread_join(remove_thread,NULL)); + assert(PTR_ERR(lookup_result)==-ENODEV&&graph_providers[0].dev.unregistered&&graph_providers[0].dev.refs==0&&active_readers==0); + assert(graph_providers[1].dev.refs==0&&!f.refs); + free(departing); + puts("PASS lookup-to-module conversion holds a reader until removed-provider rejection and balanced cleanup"); + return 0; +} +''' + return module_unit + + +def module_conversion(ROOT): + unit = getter_unit(ROOT) + main = block(unit, 'int main(') + unit = unit.replace(main, main.replace('int main(', 'int __attribute__((unused)) prior_getter_main(', 1), 1) + unit = unit.replace('struct module { unsigned refs;bool going; };', 'struct module { unsigned refs,refcnt;bool going; };', 1) + source = (ROOT / 'kernel/module/main.c').read_text() + pos = unit.index('static bool try_module_get(') + leaves = r""" +static bool module_is_live(struct module *m) { return !m->going; } +static bool atomic_inc_not_zero(unsigned *p) { if(!*p)return false;(*p)++;return true; } +static int atomic_dec_if_positive(unsigned *p) { if(!*p)return -1;return --(*p); } +#define likely(x) (x) +#define trace_module_get(...) ((void)0) +#define trace_module_put(...) ((void)0) +#define preempt_disable() ((void)0) +#define preempt_enable() ((void)0) +#define WARN_ON(v) assert(!(v)) +#define _RET_IP_ 0 +""" + unit = unit[:pos] + leaves + unit[pos:] + unit = unit.replace(block(unit, 'static bool try_module_get('), block(source, 'bool try_module_get('), 1) + unit = unit.replace(block(unit, 'static void module_put('), block(source, 'void module_put('), 1) + old = block(unit, 'static int srcu_read_lock(') + '\n' + block(unit, 'static void srcu_read_unlock(') + unit = unit.replace(old, r""" +static unsigned conversion_readers; +static int srcu_read_lock(int *s) { (void)s;conversion_readers++;return 0; } +static void srcu_read_unlock(int *s,int i) { (void)s;assert(i==0&&conversion_readers);conversion_readers--; } +""", 1) + unit += r""" +static void live_graph(void) { graph_setup();for(unsigned i=0;i<2;i++)modules[i].refcnt=1; } +static void module_balance(struct fwnode_handle *f) { graph_balanced(f);assert(!conversion_readers);for(unsigned i=0;i<2;i++)assert(modules[i].refcnt==1); } +int main(void) { + struct fwnode_handle f={0};live_graph(); + struct typec_mux *a=fwnode_typec_mux_get(&f),*b=fwnode_typec_mux_get(&f); + assert(!IS_ERR(a)&&!IS_ERR(b)&&a->context==b->context&&!conversion_readers); + for(unsigned i=0;i<2;i++)assert(modules[i].refcnt==3&&graph_providers[i].dev.refs==2); + typec_mux_put(a);typec_mux_put(b);module_balance(&f); + live_graph();modules[1].going=true;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-ENODEV);module_balance(&f); + live_graph();allocation_failure=2;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-ENOMEM);module_balance(&f); + live_graph();graph_second_error=true;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-EPROBE_DEFER);module_balance(&f); + live_graph();graph_providers[0].removed=true;a=fwnode_typec_mux_get(&f);assert(PTR_ERR(a)==-ENODEV);module_balance(&f); + puts("PASS actual kernel module get/put and mux get/put: shared handles and every failed conversion unwind"); + return 0; +} +""" + return unit diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/boot.py b/tools/testing/selftests/drivers/gpu/apple-dcp/boot.py new file mode 100644 index 00000000000000..bf95df845443d9 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/boot.py @@ -0,0 +1,56 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +# Copyright 2026 Ryan Murray +"""Opaque RTKit event dispatch: no firmware addresses or captured payloads.""" +from common import block + +def oslog(root): + source = (root / 'drivers/soc/apple/rtkit.c').read_text() + mask = next(line for line in source.splitlines() + if line.startswith('#define APPLE_RTKIT_OSLOG_TYPE ')) + return r''' +#include +#include +#include +typedef uint64_t u64; +typedef uint8_t u8; +#define GENMASK_ULL(h,l) ((~0ULL << (l)) & (~0ULL >> (63-(h)))) +#define FIELD_GET(mask,value) (((value) & (mask)) >> __builtin_ctzll(mask)) +#define APPLE_RTKIT_OSLOG_BUFFER_REQUEST 1 +#define APPLE_RTKIT_EP_OSLOG 8 +struct buffer { unsigned sentinel; }; +struct apple_rtkit { void *dev; struct buffer oslog_buffer; }; +static unsigned warnings, information, requests; +static u64 observed; +static u8 observed_type; +#define dev_warn(dev,fmt,msg) do { (void)(dev); (void)(fmt); warnings++; observed=(msg); } while (0) +#define dev_info(dev,fmt,type,msg) do { (void)(dev); (void)(fmt); information++; observed_type=(type); observed=(msg); } while (0) +static void apple_rtkit_common_rx_get_buffer(struct apple_rtkit *rtk, struct buffer *b, u8 ep, u64 msg) +{ assert(b==&rtk->oslog_buffer && ep==8); requests++; observed=msg; } +''' + mask + '\n' + block(source, 'static void apple_rtkit_oslog_rx(') + r''' +static void reset(void) { warnings=information=requests=0; observed=0; observed_type=0; } +int main(void) +{ + struct apple_rtkit rtk={.oslog_buffer={.sentinel=0x12345678}}; + for (u8 type=3; type<=5; type++) { + for (unsigned edge=0; edge<2; edge++) { + u64 msg=((u64)type<<56) | (edge ? ((1ULL<<56)-1) : 0); + reset(); apple_rtkit_oslog_rx(&rtk,msg); + assert(!warnings && information==1 && !requests); + assert(observed==msg && observed_type==type); + assert(rtk.oslog_buffer.sentinel==0x12345678); + } + } + puts("PASS recognized opaque types preserve both payload bounds without buffer/ACK operations"); + const u8 unknown[]={0,2,6,0xff}; + for (unsigned i=0;iusb3_generic_phy[0]) == 0); + assert(phy_exit(d->usb3_generic_phy[0]) == 0); + root_refs = 0; + } +} +/* Boundary mocks acquire references through the production generic PHY API. */ +static int dwc3_core_init(struct dwc3 *d) +{ + starts++; + if (core_init_error) return core_init_error; + for (int i = 0; i < d->num_usb2_ports; i++) assert(phy_init(d->usb2_generic_phy[i]) == 0); + for (int i = 0; i < d->num_usb3_ports; i++) assert(phy_init(d->usb3_generic_phy[i]) == 0); + return dwc3_phy_power_on(d); +} +static int dwc3_apple_core_probe(struct dwc3_apple *a) +{ + int ret = dwc3_core_init(&a->dwc); + if (!ret) a->state = DWC3_APPLE_NO_CABLE; + return ret; +} +static void dwc3_apple_set_role(struct dwc3_apple *a, enum dwc3_apple_state s) +{ assert(atcphy_usb3_set_mode(a->usb3_phy, s == DWC3_APPLE_HOST ? PHY_MODE_USB_HOST : PHY_MODE_USB_DEVICE, 0) == 0); } +struct fixture { struct apple_atcphy atc; struct phy p2, p3, extra; struct dwc3_apple apple; }; +static void initial(struct fixture *f, unsigned int gen, bool fixed, bool pm) +{ + memset(f, 0, sizeof(*f)); + memset(core_regs, 0, sizeof(core_regs)); + memset(pipe_regs, 0, sizeof(pipe_regs)); + memset(usb2_regs, 0, sizeof(usb2_regs)); + hardware = (struct atcphy_hw){ + .gen = gen == 0 ? ATCPHY_GENERATION_T8103 : ATCPHY_GENERATION_T8122, + .has_t8140_dp = gen == 2, .has_usb4 = gen != 2, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, + }; + f->atc = (struct apple_atcphy){ .hw = &hardware, + .regs.core = (unsigned char *)core_regs, .regs.lpdptx = (unsigned char *)core_regs, + .regs.pipehandler = (unsigned char *)pipe_regs, .regs.usb2phy = (unsigned char *)usb2_regs, + .mode = APPLE_ATCPHY_MODE_USB3, .typec_mode = APPLE_ATCPHY_MODE_USB3, + .pipe_state = ATCPHY_PIPEHANDLER_STATE_USB3, + .host_active = true, .fixed_usb2 = fixed, .ss_tunables = true, .dp_tunables = true, + }; + f->p3 = (struct phy){ .data = &f->atc, .ops = &usb3_ops, .power_count = 1, .init_count = 1, + .pm_enabled = pm, .pm_refs = pm ? 1 : 0 }; + f->p2 = (struct phy){ .data = &f->atc, .ops = &usb2_ops, .power_count = 1, .init_count = 1, + .pm_enabled = pm, .pm_refs = pm ? 1 : 0 }; + f->extra = (struct phy){ .ops = &plain_ops, .power_count = 1, .init_count = 1, + .pm_enabled = pm, .pm_refs = pm ? 1 : 0 }; + f->apple = (struct dwc3_apple){ .state = DWC3_APPLE_HOST, .reset = &f->atc.rcdev, + .usb2_phy = &f->p2, .usb3_phy = &f->p3, .core_active = true, .reset_pending = true, + .usb3_power_off_pending = 1, .usb2_power_off_pending = 1, + .dwc = { .num_usb3_ports = 1, .num_usb2_ports = 1, + .usb3_generic_phy = { &f->p3 }, .usb2_generic_phy = { &f->p2 } }, + }; + core_regs[ATCPHY_POWER_CTRL / 4] = ATCPHY_POWER_PHY_RESET_N | ATCPHY_POWER_SLEEP_BIG | + ATCPHY_POWER_SLEEP_SMALL | ATCPHY_POWER_APB_RESET_N; + core_regs[ATCPHY_POWER_STAT / 4] = core_regs[ATCPHY_POWER_CTRL / 4]; + core_regs[AUS_COMMON_DIG_RCAL1 / 4] = AUS_COMMON_DIG_RCAL1_ALL_CODES_DONE; + core_regs[ACIOPHY_TOP_PHY_STAT / 4] = ACIOPHY_TOP_PHY_STAT_LN0_UNK0; + pipe_regs[PIPEHANDLER_AON_GEN / 4] = PIPEHANDLER_AON_GEN_DWC3_RESET_N; + polls = fail_poll = reset_calls = reset_writes = core_writes = warnings = 0; + generic_warnings = 0; + host_exits = clocks_disabled = callback_calls = gadget_exits = event_cleanups = 0; + legacy_suspends = core_mode_exits = free_buffers = starts = deasserts = setup_cio = root_refs = 0; + host_init_error = gadget_init_error = core_init_error = reset_error = other_off_error = 0; + init_cleanup_fault = false; + syscore = true; + stuck_ack = false; +} +static void recover(void) { fail_poll = 0; stuck_ack = false; pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = 0; } +static int exit_apple(struct fixture *f) +{ mutex_lock(&f->apple.lock); int ret = dwc3_apple_exit(&f->apple); mutex_unlock(&f->apple.lock); return ret; } +static int role(struct fixture *f, enum usb_role r) +{ struct usb_role_switch sw = { &f->apple }; return dwc3_usb_role_switch_set(&sw, r); } +static void clean(struct fixture *f) +{ + assert(!f->p3.power_count && !f->p2.power_count && !f->p3.init_count && !f->p2.init_count); + assert(!f->p3.pm_refs && !f->p2.pm_refs && !f->apple.core_active && !f->apple.reset_pending); + assert(!f->apple.usb3_power_off_pending && !f->apple.usb2_power_off_pending); + assert(!f->atc.host_active && !f->atc.mode_dirty && f->atc.mode == APPLE_ATCPHY_MODE_OFF); + assert(!syscore); + assert(!f->atc.lock && !f->p3.mutex && !f->p2.mutex && !f->apple.lock); + assert(!(pipe_regs[PIPEHANDLER_AON_GEN / 4] & PIPEHANDLER_AON_GEN_DWC3_RESET_N)); +} +static void cycles(void) +{ + unsigned int cases = 0; + for (unsigned int gen = 0; gen < 3; gen++) + for (unsigned int fixed = 0; fixed < 2; fixed++) + for (unsigned int pm = 0; pm < 2; pm++) + for (unsigned int fault = 0; fault < 4; fault++) { + struct fixture f; initial(&f, gen, fixed, pm); + if (fault == 1) { + stuck_ack = true; pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + } else if (fault > 1) { + f.atc.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; fail_poll = fault - 1; + } + int ret = exit_apple(&f); + printf("CYCLE gen=%u fixed=%u pm=%u fault=%u exit=%d refs=%d init=%d\n", + gen, fixed, pm, fault, ret, f.p3.power_count, f.p3.init_count); fflush(stdout); + if (fault) { + assert(ret == -ETIMEDOUT && f.p3.power_count == 1 && f.p3.init_count == 1); + assert(f.apple.core_active && clocks_disabled == 0 && event_cleanups == 1 && host_exits == 1); + assert(syscore); + if (fault > 1) assert(f.atc.mode_dirty); + if (fault == 1) { + assert(role(&f, USB_ROLE_HOST) == -ETIMEDOUT && starts == 0 && deasserts == 0); + assert(host_exits == 1 && event_cleanups == 1 && f.p3.power_count == 1); + } + recover(); + assert(exit_apple(&f) == 0); + } else assert(ret == 0); + clean(&f); + assert(host_exits == 1 && clocks_disabled == 1 && event_cleanups == 1); + assert(exit_apple(&f) == 0 && host_exits == 1 && clocks_disabled == 1); + assert(role(&f, USB_ROLE_HOST) == 0 && f.p3.power_count == 1 && f.p3.init_count == 1); + assert(f.atc.mode == APPLE_ATCPHY_MODE_USB3 && !f.atc.mode_dirty); + assert(core_regs[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); + assert(role(&f, USB_ROLE_NONE) == 0); + clean(&f); + assert(!generic_warnings); + if (!fault) assert(!warnings); + cases++; + } + printf("PASS: %u normal/parking/domain shutdown and restart scenarios\n", cases); +} +static void pending_reset(void) +{ + for (unsigned int gen = 0; gen < 3; gen++) { + struct fixture f; initial(&f, gen, false, true); + reset_error = -ETIMEDOUT; + assert(exit_apple(&f) == -ETIMEDOUT); + assert(!f.apple.core_active && f.apple.reset_pending && !f.p3.power_count && !f.p3.init_count); + assert(syscore); + assert(role(&f, USB_ROLE_HOST) == -ETIMEDOUT && !starts && !deasserts); + assert(host_exits == 1 && clocks_disabled == 1 && event_cleanups == 1 && callback_calls == 1); + reset_error = 0; + assert(exit_apple(&f) == 0); + clean(&f); + assert(host_exits == 1 && clocks_disabled == 1 && callback_calls == 1); + assert(reset_calls == 3); + } + puts("PASS: pending reset retry does not repeat host, PHY or core teardown"); +} +static void partial_release(void) +{ + struct fixture f; initial(&f, 1, false, true); + other_off_error = -EIO; + assert(exit_apple(&f) == -EIO); + assert(!f.p3.power_count && f.p3.init_count == 1 && f.p2.power_count == 1); + assert(!f.apple.usb3_power_off_pending && f.apple.usb2_power_off_pending == 1); + assert(callback_calls == 1 && !clocks_disabled); + other_off_error = 0; + assert(exit_apple(&f) == 0); clean(&f); + assert(callback_calls == 1 && event_cleanups == 1 && clocks_disabled == 1); + initial(&f, 1, false, false); + f.apple.dwc.num_usb3_ports = 2; f.apple.dwc.usb3_generic_phy[1] = &f.extra; + f.apple.usb3_power_off_pending = 3; + other_off_error = -EIO; + assert(exit_apple(&f) == -EIO && !f.p3.power_count && f.extra.power_count == 1); + other_off_error = 0; + assert(exit_apple(&f) == 0 && !f.extra.power_count && !f.extra.init_count); + clean(&f); assert(callback_calls == 1); + puts("PASS: successful USB3 references are not released twice after another PHY fails"); +} +static void init_failures(void) +{ + for (unsigned int device = 0; device < 2; device++) { + struct fixture f; initial(&f, 2, false, true); + assert(exit_apple(&f) == 0); + if (device) gadget_init_error = -EIO; else host_init_error = -EIO; + reset_error = -ETIMEDOUT; + assert(role(&f, device ? USB_ROLE_DEVICE : USB_ROLE_HOST) == -EIO); + assert(f.apple.reset_pending && !f.apple.core_active && !f.p3.power_count); + assert(syscore); + host_init_error = gadget_init_error = reset_error = 0; + assert(role(&f, device ? USB_ROLE_DEVICE : USB_ROLE_HOST) == 0); + assert(f.p3.power_count == 1 && f.p3.init_count == 1); + assert(role(&f, USB_ROLE_NONE) == 0); clean(&f); + } + struct fixture f; initial(&f, 1, false, false); assert(exit_apple(&f) == 0); + core_init_error = -EIO; reset_error = -ETIMEDOUT; + assert(role(&f, USB_ROLE_HOST) == -EIO && f.apple.reset_pending && !f.apple.core_active); + core_init_error = reset_error = 0; + assert(role(&f, USB_ROLE_HOST) == 0); + assert(role(&f, USB_ROLE_NONE) == 0); clean(&f); + initial(&f, 1, false, true); assert(exit_apple(&f) == 0); + host_init_error = -EIO; init_cleanup_fault = true; + assert(role(&f, USB_ROLE_HOST) == -EIO && f.apple.core_active); + assert(syscore); + assert(f.p3.power_count == 1 && f.p3.init_count == 1); + recover(); host_init_error = 0; init_cleanup_fault = false; + assert(role(&f, USB_ROLE_HOST) == 0 && f.p3.power_count == 1 && f.p3.init_count == 1); + assert(role(&f, USB_ROLE_NONE) == 0); clean(&f); + puts("PASS: host/gadget/core startup failure reset obligations drain before retry"); +} +static void pending_restart(void) +{ + unsigned int cases = 0; + for (unsigned int gen = 0; gen < 3; gen++) + for (unsigned int fixed = 0; fixed < 2; fixed++) + for (unsigned int pm = 0; pm < 2; pm++) + for (unsigned int fault = 1; fault < 4; fault++) { + struct fixture f; initial(&f, gen, fixed, pm); + if (fault == 1) { + stuck_ack = true; pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + } else { f.atc.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; fail_poll = fault - 1; } + assert(exit_apple(&f) == -ETIMEDOUT); + recover(); + assert(role(&f, USB_ROLE_HOST) == 0 && f.p3.power_count == 1 && f.p3.init_count == 1); + assert(f.atc.mode == APPLE_ATCPHY_MODE_USB3 && !f.atc.mode_dirty); + assert(core_regs[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); + assert(!generic_warnings); + assert(role(&f, USB_ROLE_NONE) == 0); clean(&f); cases++; + } + struct fixture f; initial(&f, 2, true, true); f.apple.fixed_hub = true; + assert(role(&f, USB_ROLE_NONE) == 0 && f.apple.state == DWC3_APPLE_HOST && f.p3.power_count == 1); + assert(exit_apple(&f) == 0); + mutex_lock(&f.apple.lock); + assert(dwc3_apple_init(&f.apple, DWC3_APPLE_HOST) == 0); + mutex_unlock(&f.apple.lock); + assert(f.p3.power_count == 1 && exit_apple(&f) == 0); clean(&f); + printf("PASS: %u pending-failure to HOST transitions and actual fixed-hub role no-op\n", cases); +} +static void negative_power(void) +{ + struct fixture f; initial(&f, 0, false, false); + f.atc.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; fail_poll = 1; + assert(exit_apple(&f) == -ETIMEDOUT && f.p3.power_count == 1 && f.p3.init_count == 1); + recover(); assert(role(&f, USB_ROLE_HOST) == 0 && f.p3.power_count == 1); +} +static void negative_reset(void) +{ + struct fixture f; initial(&f, 0, false, false); reset_error = -ETIMEDOUT; + assert(exit_apple(&f) == -ETIMEDOUT && reset_calls == 1); + reset_error = 0; assert(exit_apple(&f) == 0 && reset_calls == 2); + assert(!(pipe_regs[PIPEHANDLER_AON_GEN / 4] & PIPEHANDLER_AON_GEN_DWC3_RESET_N)); +} +static void negative_legacy(void) +{ + struct fixture f; initial(&f, 0, false, false); + f.atc.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; fail_poll = 1; + assert(phy_power_off(&f.p3) == -ETIMEDOUT && f.atc.mode_dirty); + recover(); + struct typec_mux_dev mux = { &f.atc }; + struct typec_mux_state state = { .mode = TYPEC_STATE_USB }; + assert(atcphy_mux_set(&mux, &state) == 0); + assert(core_regs[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); +} +static void legacy_restore(void) +{ + for (unsigned int gen = 0; gen < 3; gen++) { + struct fixture f; initial(&f, gen, false, false); + f.atc.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; fail_poll = 1; + assert(phy_power_off(&f.p3) == -ETIMEDOUT && f.atc.mode_dirty); + unsigned int before = core_writes; + assert(atcphy_usb3_set_mode(&f.p3, PHY_MODE_USB_HOST, 0) == -EBUSY && core_writes == before); + if (gen < 2) { + f.atc.dp_link_rate = ATCPHY_DP_LINK_RATE_RBR; + assert(atcphy_dp_configure(&f.atc, ATCPHY_DP_LINK_RATE_RBR) == -EBUSY && core_writes == before); + } + recover(); + struct typec_mux_dev mux = { &f.atc }; + struct typec_mux_state state = { .mode = TYPEC_STATE_USB }; + assert(atcphy_mux_set(&mux, &state) == 0 && !f.atc.mode_dirty && core_writes > before); + assert(core_regs[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); + assert(phy_power_off(&f.p3) == 0 && !f.p3.power_count); + } + puts("PASS: M1/M2/J700 aborted shutdown blocks PIPE and restores saved lane mode"); +} +static void removal_and_generic(void) +{ + struct fixture f; initial(&f, 0, false, false); + root_refs = 1; assert(phy_init(&f.p3) == 0 && phy_power_on(&f.p3) == 0); + assert(exit_apple(&f) == 0); clean(&f); + unsigned int callbacks = callback_calls, clocks = clocks_disabled, events = event_cleanups; + dwc3_core_remove_no_shutdown(&f.apple.dwc); + assert(callbacks == callback_calls && clocks == clocks_disabled && events == event_cleanups); + assert(!core_mode_exits && free_buffers == 1 && !f.p3.power_count && !f.p3.init_count); + initial(&f, 1, false, false); + dwc3_core_exit(&f.apple.dwc); + assert(!f.p3.power_count && !f.p3.init_count && !f.p2.power_count && !f.p2.init_count); + assert(event_cleanups == 1 && clocks_disabled == 1 && callback_calls == 1); + puts("PASS: root-hub references, already-exited removal and ordinary generic core exit"); +} +int main(int argc, char **argv) +{ + if (argc == 2 && !strcmp(argv[1], "power")) { negative_power(); return 0; } + if (argc == 2 && !strcmp(argv[1], "reset")) { negative_reset(); return 0; } + if (argc == 2 && !strcmp(argv[1], "legacy")) { negative_legacy(); return 0; } + cycles(); pending_restart(); pending_reset(); partial_release(); init_failures(); legacy_restore(); removal_and_generic(); + puts("PASS: Apple teardown actual-function qualification"); return 0; +} diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/mocks.c b/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/mocks.c new file mode 100644 index 00000000000000..e54e73146cbad6 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/mocks.c @@ -0,0 +1,125 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +struct apple_atcphy { + const struct atcphy_hw *hw; + struct { unsigned char *core, *lpdptx, *pipehandler, *usb2phy; } regs; + struct reset_controller_dev rcdev; + void *dev; + int lock, dp_link_rate; + bool swap_lanes, tunnel_clock_on, tunnel_saved; + bool fixed_usb2, host_active, mode_dirty, ss_tunables, dp_tunables, aux_power_requested; + unsigned int dp_owned_pmas, dp_idle_pmas; + unsigned int dp_usb_context_valid, dp_usb_context_dirty; + u32 dp_usb_context[2][27]; + bool pipe_pending, usb_off_pending, usb_off_drained; + enum atcphy_mode mode, typec_mode; + enum atcphy_pipehandler_state pipe_state; +}; +struct phy; +struct phy_ops { int (*init)(struct phy *); int (*exit)(struct phy *); + int (*power_on)(struct phy *); int (*power_off)(struct phy *); }; +struct phy { struct apple_atcphy *data; const struct phy_ops *ops; + int mutex, power_count, init_count, pm_refs, pwr, dev; bool pm_enabled; }; +#define phy_get_drvdata(p) ((p)->data) +static int phy_pm_runtime_get_sync(struct phy *p) +{ if (!p->pm_enabled) return -ENOTSUPP; p->pm_refs++; return 0; } +static void phy_pm_runtime_put(struct phy *p) { if (p->pm_enabled) p->pm_refs--; } +static void phy_pm_runtime_put_sync(struct phy *p) { phy_pm_runtime_put(p); } +static int regulator_enable(int p) { assert(!p); return 0; } +static void regulator_disable(int p) { assert(!p); } +struct dwc3 { struct phy *usb2_generic_phy[3], *usb3_generic_phy[3]; + int num_usb2_ports, num_usb3_ports, usb2_phy, usb3_phy; + struct reset_controller_dev *reset; + int dev, usb_psy, vbus_draw_work; }; +struct dwc3_apple { struct dwc3 dwc; enum dwc3_apple_state state; + struct reset_controller_dev *reset; int lock; void *dev; + struct phy *usb2_phy, *usb3_phy; + bool fixed_hub, core_active, core_exit_pending, reset_pending; + unsigned long usb2_power_off_pending, usb3_power_off_pending; }; +static unsigned int polls, fail_poll, reset_calls, reset_writes, core_writes; +static unsigned int host_exits, clocks_disabled, callback_calls; +static unsigned int gadget_exits, event_cleanups, legacy_suspends, core_mode_exits, free_buffers; +static unsigned int starts, deasserts, setup_cio, root_refs; +static int host_init_error, gadget_init_error, core_init_error, reset_error, other_off_error; +static bool init_cleanup_fault; +static bool syscore; +static bool stuck_ack; +static u32 core_regs[0x30000 / 4], pipe_regs[0x1000 / 4], usb2_regs[0x1000 / 4]; +static u32 readl(void *p) { return *(u32 *)p; } +static void mask32(void *p, u32 m, u32 v) +{ + *(u32 *)p = (*(u32 *)p & ~m) | v; + if (p == &pipe_regs[PIPEHANDLER_LOCK_REQ / 4] && !stuck_ack) + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = *(u32 *)p & PIPEHANDLER_LOCK_EN; + if (p == &pipe_regs[PIPEHANDLER_AON_GEN / 4]) reset_writes++; + if ((uintptr_t)p >= (uintptr_t)core_regs && + (uintptr_t)p < (uintptr_t)core_regs + sizeof(core_regs)) core_writes++; +} +static void writel(u32 v, void *p) { mask32(p, ~0U, v); } +static void set32(void *p, u32 v) { mask32(p, 0, v); } +static void clear32(void *p, u32 v) { mask32(p, v, 0); } +static void core_mask32(struct apple_atcphy *p, u32 o, u32 m, u32 v) +{ + assert(o < sizeof(core_regs)); + mask32(p->regs.core + o, m, v); + if (o == ATCPHY_POWER_CTRL) core_regs[ATCPHY_POWER_STAT / 4] = core_regs[o / 4]; +} +static void core_set32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, 0, v); } +static void core_clear32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, v, 0); } +#define readl_poll_timeout(a,v,c,i,t) \ + ({ (v) = readl(a); ++polls; polls == fail_poll || !(c) ? -ETIMEDOUT : 0; }) +#define readl_poll_timeout_atomic readl_poll_timeout +/* These dependencies are outside the cross-layer test; no DP ownership is seeded. */ +static int j700_dp_stop(struct apple_atcphy *p) { assert(!p->dp_owned_pmas); return 0; } +static int j700_dp_idle_pending(struct apple_atcphy *p) { (void)p; return 0; } +static void atc_tunnel_stop_t8103(struct apple_atcphy *p) { (void)p; } +static void atc_tunnel_restore(struct apple_atcphy *p) { (void)p; } +/* USB2 power-on is outside this teardown/recovery control. */ +static void atcphy_usb2_power_on(struct apple_atcphy *p) { (void)p; } +static void atcphy_apply_tunables(struct apple_atcphy *p, enum atcphy_mode m) { (void)p; (void)m; } +static int atcphy_configure_lanes(struct apple_atcphy *p, enum atcphy_mode m) { (void)p; (void)m; return 0; } +static int atcphy_enable_dp_aux(struct apple_atcphy *p) { (void)p; return 0; } +static int atcphy_auspll_apb_command(struct apple_atcphy *p, int command) +{ (void)p; (void)command; return 0; } +static int atcphy_dp_configure_lane(struct apple_atcphy *p, int lane, + const struct atcphy_dp_link_rate_configuration *config) +{ (void)p; (void)lane; (void)config; return 0; } +static void usb_phy_set_suspend(int p, int s) { (void)p; (void)s; legacy_suspends++; } +static void usb_phy_shutdown(int p) { (void)p; } +static void dwc3_event_buffers_cleanup(struct dwc3 *d) { (void)d; event_cleanups++; } +static void dwc3_clk_disable(struct dwc3 *d) { (void)d; clocks_disabled++; } +static void dwc3_host_exit(struct dwc3 *d); +static void dwc3_gadget_exit(struct dwc3 *d) { (void)d; gadget_exits++; } +static void dev_pm_syscore_device(void *d, bool s) { (void)d; syscore = s; } +static void dwc3_enable_susphy(struct dwc3 *d, bool s) { (void)d; (void)s; } +static int atcphy_dwc3_reset_assert(struct reset_controller_dev *, unsigned long); +static int reset_control_assert(struct reset_controller_dev *r) +{ if (!r) return 0; reset_calls++; if (reset_error) return reset_error; + return atcphy_dwc3_reset_assert(r, 0); } +static int atcphy_dwc3_reset_deassert(struct reset_controller_dev *, unsigned long); +static int reset_control_deassert(struct reset_controller_dev *r) +{ deasserts++; return atcphy_dwc3_reset_deassert(r, 0); } +static int phy_set_mode(struct phy *p, int m) { (void)p; (void)m; return 0; } +static int dwc3_apple_core_probe(struct dwc3_apple *a); +static int dwc3_core_init(struct dwc3 *d); +static void dwc3_apple_setup_cio(struct dwc3_apple *a) { (void)a; setup_cio++; } +static int dwc3_apple_usb2_mode_before_reset(enum dwc3_apple_state s, enum phy_mode *m) +{ if (s != DWC3_APPLE_HOST && s != DWC3_APPLE_DEVICE) return -EINVAL; + *m = s == DWC3_APPLE_HOST ? PHY_MODE_INVALID : PHY_MODE_USB_DEVICE; return 0; } +static void dwc3_apple_set_role(struct dwc3_apple *a, enum dwc3_apple_state s); +static int dwc3_host_init(struct dwc3 *d) +{ (void)d; if (init_cleanup_fault) { stuck_ack = true; + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; } return host_init_error; } +static int dwc3_gadget_init(struct dwc3 *d) { (void)d; return gadget_init_error; } +#define pm_runtime_get_sync(...) ((void)0) +#define pm_runtime_allow(...) ((void)0) +#define pm_runtime_disable(...) ((void)0) +#define pm_runtime_dont_use_autosuspend(...) ((void)0) +#define pm_runtime_put_noidle(...) ((void)0) +#define pm_runtime_set_suspended(...) ((void)0) +#define dev_set_drvdata(...) ((void)0) +#define cancel_work_sync(...) ((void)0) +#define power_supply_put(...) ((void)0) +static void dwc3_core_exit_mode(struct dwc3 *d) { (void)d; core_mode_exits++; } +static void dwc3_debugfs_exit(struct dwc3 *d) { (void)d; } +static void dwc3_ulpi_exit(struct dwc3 *d) { (void)d; } +static void dwc3_free_event_buffers(struct dwc3 *d) { (void)d; free_buffers++; } diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/prefix.c b/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/prefix.c new file mode 100644 index 00000000000000..a3e62f8a4f12da --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/dwc3/prefix.c @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +typedef uint16_t u16; +typedef uint8_t u8; +#define ENOTSUPP 524 +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define BIT(n) (1U << (n)) +#define GENMASK(h, l) ((~0U << (l)) & (~0U >> (31 - (h)))) +#define FIELD_PREP(m, v) (((u32)(v) << __builtin_ctz(m)) & (m)) +#define FIELD_GET(m, v) (((v) & (m)) >> __builtin_ctz(m)) +#define field_prep FIELD_PREP +#define __iomem +#define udelay(n) ((void)(n)) +#define usleep_range(a,b) ((void)(a), (void)(b)) +#define fallthrough __attribute__((fallthrough)) +static unsigned int warnings, generic_warnings; +static void warning(const char *message) +{ warnings++; if (strstr(message, "phy_power_on was called before phy_init")) generic_warnings++; } +#define dev_warn(d, message, ...) warning(message) +#define dev_err(...) ((void)0) +#define dev_dbg(...) ((void)0) +#define lockdep_assert_held(p) assert(*(p)) +#define WARN_ON_ONCE(v) assert(!(v)) +static void mutex_lock(int *m) { assert(!*m); *m = 1; } +static void mutex_unlock(int *m) { assert(*m); *m = 0; } +struct held { int *m; }; +static struct held hold(int *m) { mutex_lock(m); return (struct held){m}; } +static void release(struct held *h) { mutex_unlock(h->m); } +#define guard(kind) struct held held_guard __attribute__((cleanup(release))) = hold +enum phy_mode { PHY_MODE_USB_HOST, PHY_MODE_USB_DEVICE }; +enum { APPLE_ATCPHY_LANE_0, APPLE_ATCPHY_LANE_1 }; +enum usb_role { USB_ROLE_NONE, USB_ROLE_HOST, USB_ROLE_DEVICE }; +struct usb_role_switch { struct dwc3_apple *data; }; +#define usb_role_switch_get_drvdata(s) ((s)->data) +#define PHY_MODE_INVALID (-1) +enum { TYPEC_STATE_SAFE, TYPEC_STATE_USB, TYPEC_MODE_USB4, + TYPEC_DP_STATE_C, TYPEC_DP_STATE_D, TYPEC_DP_STATE_E }; +enum { USB_TYPEC_TBT_SID = 0x8087, USB_TYPEC_DP_SID = 0xff01 }; +#define EUDO_USB_MODE_MASK 3U +enum { EUDO_USB_MODE_USB2, EUDO_USB_MODE_USB3, EUDO_USB_MODE_USB4 }; +struct enter_usb_data { u32 eudo; }; +struct typec_altmode { unsigned int svid; }; +struct typec_mux_state { unsigned long mode; struct typec_altmode *alt; void *data; }; +struct typec_mux_dev { struct apple_atcphy *data; }; +#define typec_mux_get_drvdata(mux) ((mux)->data) +struct reset_controller_dev { int unused; }; +#define container_of(p,t,m) ((t *)((char *)(p) - offsetof(t,m))) diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/edid.py b/tools/testing/selftests/drivers/gpu/apple-dcp/edid.py new file mode 100644 index 00000000000000..69ebe810f1c2dd --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/edid.py @@ -0,0 +1,934 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +# Copyright 2026 Ryan Murray +"""Actual native EDID admission, service epochs, stale replies and RX progress.""" +from pathlib import Path +from common import block, source_defines + +def rebind(root): + base = root / 'drivers/gpu/drm/apple' + def function(file,signature): + text=(base/file).read_text();start=text.index(signature);opening=text.index('{',start) + while ';' in text[start:opening]: + start=text.index(signature,start+1);opening=text.index('{',start) + end,depth=opening+1,1 + while depth: + depth+=(text[end]=='{')-(text[end]=='}');end+=1 + return text[start:end] + prefix=r''' +#undef NDEBUG +#include +#include +#include +#include +#include +#include +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#include "afkv2-codec.h" +typedef int64_t s64; +typedef u64 dma_addr_t; +#define U16_MAX UINT16_MAX +#define EBUSY 16 +#define EOPNOTSUPP 95 +#define ENOENT 2 +#define MAX_PENDING_CMDS 16 +#define AFK_MAX_CHANNEL 16 +#define AFKV2_INTERFACES (1U << 16) +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x)=(v)) +#define WARN_ON(x) (x) +#define bitmap_empty(p,n) (!*(p)) +#define spin_lock_irqsave(p,f) do { assert(!*(p));*(p)=true;(f)=0; } while(0) +#define spin_unlock_irqrestore(p,f) do { (void)(f);assert(*(p));*(p)=false; } while(0) +static int test_bit(unsigned int bit,const unsigned long *p) { return !!(*p & (1UL<(b)?(a):(b)) +#define strscpy(dst,src,n) snprintf(dst,n,"%s",(char *)(src)) +#define trace_dcpavserv_init(...) ((void)0) +#define dma_rmb() ((void)0) +#define DRM_CONNECTOR_REGISTERED 1 +#define DPAVSERV_ENDPOINT_V2 0x2a +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define EPIC_SUBTYPE_STD_SERVICE 0xc0 +#define EPIC_SUBTYPE_ANNOUNCE_H17P 0x11 +#define to_platform_device(x) (x) +#define ERR_PTR(x) ((void *)(intptr_t)(x)) +#define IS_ERR(x) ((uintptr_t)(x)>(uintptr_t)-4096) +#define lockdep_assert_held(p) assert(*(p)) +static bool *hpd_lock,*av_lock; +static void mutex_lock(bool *p) { if(p==hpd_lock) assert(!*av_lock);assert(!*p);*p=true; } +static void mutex_unlock(bool *p) { assert(*p);*p=false; } +static void complete(int *p) { (*p)++; } +static void reinit_completion(int *p) { *p=0; } +static void schedule_work(unsigned int *p) { (*p)++; } +struct work_struct { bool pending; }; +static bool queue_work(void *wq,struct work_struct *w) { assert(wq);bool fresh=!w->pending;w->pending=true;return fresh; } +struct apple_dcp;struct apple_dcp_afkep;struct apple_epic_service; +void afk_service_retry(struct apple_dcp_afkep *ep); +struct apple_epic_service_ops { char name[32];bool reusable,optional;void (*init)(struct apple_epic_service *,const char *,const char *,s64);int (*prepare)(struct apple_epic_service *,const char *,const char *,s64);int (*revalidate)(struct apple_epic_service *,const char *,const char *,s64);void (*teardown)(struct apple_epic_service *); }; +struct epic_cmd_info { u16 tag;u8 type;size_t reply_length;void *rxbuf,*txbuf;dma_addr_t rxbuf_dma,txbuf_dma;size_t rxlen,txlen;u32 retcode;bool done,free_on_ack;int *completion; }; +struct apple_epic_service { + struct apple_dcp_afkep *ep;const struct apple_epic_service_ops *ops; + bool lock,enabled,torndown;unsigned int users;u32 channel;u8 cmd_tag; + unsigned long cmd_map[1];struct epic_cmd_info cmds[MAX_PENDING_CMDS]; + struct { u64 generation,epoch; } admission;void *cookie; + bool pending_open;const char *pending_name,*pending_class;s64 pending_unit; +}; +struct apple_dcp_afkep { u8 *command_tags;bool lock,stopping;void *wq;struct work_struct service_work;struct apple_dcp *dcp;const struct apple_epic_service_ops *ops;bool wire_v2,inline_v2,match_epic_name;u32 endpoint;unsigned int num_channels;struct apple_epic_service services[AFK_MAX_CHANNEL]; }; +struct apple_connector { void *dcp;bool connected,edid_invalidation_pending;struct { int registration_state; } base;unsigned int hotplug_wq; }; +struct apple_dcp { + void *dev;int fw_compat;struct apple_dcp_afkep *dcpavservep;struct apple_connector *connector; + bool external_physically_absent,hpd_mutex,quiescing,typec_cable_connected;u64 typec_generation;unsigned int nr_modes; + struct { int iomfb_method_profile; } hw; + struct { bool lock,enabled;u64 epoch;int enable_completion;struct apple_epic_service *service; } dcpavserv; +}; +struct epic_cmd { u32 retcode;u64 rxbuf,txbuf;u32 rxlen,txlen; }; +static unsigned int dma_frees,opens; + +static int open_error,open_action; +void dpavservep_detach(struct apple_dcp *dcp); +static void inject_owner(struct apple_dcp_afkep *ep,u32 pending_channel); +static void dma_free_coherent(void *dev,size_t size,void *p,dma_addr_t address) +{ (void)dev;assert(size && (uintptr_t)p==address);free(p);dma_frees++; } +static void afk_remove_service_debugfs(struct apple_epic_service *s) { (void)s; } +static void afk_populate_service_debugfs(struct apple_epic_service *s) { (void)s; } +static int afk_send_epic(struct apple_dcp_afkep *e,u32 channel,u16 tag,int type,int category,u8 subtype,const void *p,size_t n) +{ + assert(channel && !tag && type==EPIC_TYPE_NOTIFY && category==EPIC_CAT_REPORT && subtype==AFKV2_OPEN && !p && !n); + opens++; + assert(!e->dcp->dcpavserv.enabled && !e->dcp->dcpavserv.service); + for(unsigned int i=0;inum_channels;i++) { + struct apple_epic_service *s=&e->services[i]; + if(s->channel==channel && !s->torndown) assert(!s->enabled && !s->users); + } + if(open_action==1) e->dcp->typec_generation++; + if(open_action==2) dpavservep_detach(e->dcp); + if(open_action==4) inject_owner(e,channel); + if(open_action==5) e->dcp->typec_cable_connected=false; + if(open_action==6) e->dcp->quiescing=true; + if(open_action==7) e->dcp->external_physically_absent=true; + if(open_action==3) { + for(unsigned int i=0;inum_channels;i++) + if(e->services[i].channel==channel && !e->services[i].torndown) e->services[i].channel++; + } + return open_error; +} +static void afk_v2_callback(struct apple_dcp_afkep *e,const struct afkv2_frame *f,const struct afkv2_command *c) { (void)e;(void)f;(void)c;assert(!"unexpected callback"); } + +struct item { const char *key,*value; s64 integer; }; +struct dcp_parse_ctx { struct item *items;unsigned int count,index;bool key; }; +struct iterator { struct dcp_parse_ctx *handle;unsigned int idx,len; }; +#define kfree(p) free((void *)(p)) +#define PTR_ERR(p) ((int)(intptr_t)(p)) +static struct item announcement_items[3]; +static int parse(u8 *p,size_t n,struct dcp_parse_ctx *ctx) +{ (void)p;assert(n==37);*ctx=(struct dcp_parse_ctx){.items=announcement_items,.count=3};return 0; } +static int iterator_begin(struct dcp_parse_ctx *h,struct iterator *it,bool dict) +{ assert(dict);h->index=0;h->key=true;*it=(struct iterator){.handle=h,.len=h->count};return 0; } +static char *parse_string(struct dcp_parse_ctx *h) +{ + struct item *i=&h->items[h->index];const char *p; + if(h->key) { p=i->key;h->key=false; } else { p=i->value;h->index++;h->key=true; } + if(!p) + return ERR_PTR(-EINVAL); + char *copy=malloc(strlen(p)+1);assert(copy);strcpy(copy,p);return copy; +} +static int parse_int(struct dcp_parse_ctx *h,s64 *out) +{ assert(!h->key);*out=h->items[h->index++].integer;h->key=true;return 0; } +static int skip(struct dcp_parse_ctx *h) { assert(!h->key);h->index++;h->key=true;return 0; } +''' + parts=[function('afk.c','static u16 afk_command_tag('),function('afk.c','void afk_service_retry('),function('parser.c','int parse_epic_service_init('),function('afk.h','static inline struct apple_epic_service *afk_service_get('), + function('afk.h','static inline void afk_service_put('),function('afk.h','static inline void afk_service_retire('),function('afk.h','static inline void afk_service_disable('), + function('afk.h','static inline bool afk_service_matches('),function('afk.h','static inline bool afk_service_reclaimable('), + function('afk.h','static inline bool\nafk_service_reinit('),function('afk.h','static inline void afk_service_discard('), + function('afk.h','static inline void afk_service_publish('), + function('epic/dpavservep.c','static void dcpavserv_notify_edid('),function('epic/dpavservep.c','static bool dcpavserv_identity('), + function('epic/dpavservep.c','static bool dcpavserv_admission_current('),function('epic/dpavservep.c','static int dcpavserv_prepare('), + function('epic/dpavservep.c','static int dcpavserv_revalidate('),function('epic/dpavservep.c','static void dcpavserv_init('),function('epic/dpavservep.c','static void dcpavserv_invalidate_edid('), + function('epic/dpavservep.c','static void dcpavserv_teardown('),function('epic/dpavservep.c','void dpavservep_detach('), + function('afk.c','static const struct apple_epic_service_ops *\nafk_match_service('), + function('afk.c','static struct apple_epic_service *afk_epic_find_service('), + function('afk.c','static void afk_discard_pending_open('),function('afk.c','static void afk_retry_service_init('),function('afk.c','static void afk_service_worker('),function('afk.c','static void afk_recv_handle_init('), + function('epic/dpavservep.c','static int dcpdpserv_prepare('), + function('epic/dpavservep.c','static void dcpdpserv_teardown('),function('afk.c','static void afk_recv_handle_teardown('), + function('afk.c','static void afk_recv_handle_reply('),function('afk.c','static void afk_v2_receive(')] + suffix=r''' +static struct apple_dcp d; +static struct apple_dcp_afkep ep; +static struct apple_connector con; +static void run_retries(void) { assert(!ep.lock);if(ep.service_work.pending) { ep.service_work.pending=false;afk_service_worker(&ep.service_work); } } +static const struct apple_epic_service_ops ops={.name="dcpav-service-epic",.reusable=true,.prepare=dcpavserv_prepare,.revalidate=dcpavserv_revalidate,.init=dcpavserv_init,.teardown=dcpavserv_teardown}; +static const struct apple_epic_service_ops announcement_ops[]={ + {.name="dcpav-service-epic",.reusable=true,.prepare=dcpavserv_prepare,.revalidate=dcpavserv_revalidate,.init=dcpavserv_init,.teardown=dcpavserv_teardown}, + {.name="dcpdp-service-epic",.reusable=true,.optional=true,.prepare=dcpdpserv_prepare,.teardown=dcpdpserv_teardown}, + {.name=""} +}; +static const char *name="dcpav-service-epic",*class="AppleDCPDP2HDMI"; +static const char *native_class="DCPDP13Service"; +static struct apple_epic_service *prepare(u32 channel,s64 unit,const char *n,const char *c) +{ + for(unsigned int i=0;iusers==1); +} +static void repeated_announcements(void) +{ + ep.ops=announcement_ops;ep.match_epic_name=true;ep.num_channels=0;d.fw_compat=DCP_FIRMWARE_H17P; + /* More than the entire slot budget; wrap channel identity as well. */ + for(unsigned int i=0;i<4*AFK_MAX_CHANNEL;i++) { + u16 channel=1+4*(i%4);unsigned int before=opens; + /* Flip, same-attachment revalidation, return flip; swap both providers. */ + bool native=((i%4==1 || i%4==2) != !!((i/4)%2)); + const char *provider=native?native_class:class; + announce(channel,name,provider,channel); + struct apple_epic_service *owner=d.dcpavserv.service; + if(native) assert(owner && "native DP EDID provider must publish"); + assert(owner->ops==&announcement_ops[0]); + assert(owner && owner->channel==channel && owner->users==1 && opens==before+1); + assert(ep.num_channels<=2); /* Real allocator reuses, no slot exhaustion. */ + u64 epoch=d.dcpavserv.epoch; + announce(channel,name,provider,channel); /* Active duplicate: no extra OPEN/pin. */ + assert(opens==before+1 && owner->users==1 && d.dcpavserv.epoch==epoch); + announce(channel+2,"dcpdp-service-epic",provider,channel+2); + assert(opens==before+1 && d.dcpavserv.service==owner && d.dcpavserv.epoch==epoch); + if(i%4==1) { + u64 generation=d.typec_generation; + closed(channel,0x6a01,0); /* Revalidation retires service, not attachment. */ + assert(!d.dcpavserv.service && d.typec_generation==generation && d.dcpavserv.epoch>epoch); + } else { + disconnect();connect(); + } + } + unsigned int missing_ops_before=opens; + assert(afk_match_service(&ep,"missing-service-epic")==NULL); + assert(afk_match_service(&ep,"dcpav-service-epic")==&announcement_ops[0]); + announce(13,"missing-service-epic",native_class,13); + announce(13,"dcpav-service-epic-extra",native_class,13); + assert(opens==missing_ops_before && !d.dcpavserv.service); + assert(ep.num_channels==2); /* Auxiliary rejection leaves a reusable slot. */ + unsigned int before=opens; + announce(13,name,"synthetic-wrong-class",13);assert(opens==before && !d.dcpavserv.service); + char long_class[161];memset(long_class,'x',160);long_class[160]=0; + announce(13,name,long_class,13);assert(opens==before && !d.dcpavserv.service); + for(unsigned int variant=0;variant<2;variant++) { + const char *provider=variant?native_class:class; + assert(!prepare(0,0,name,provider));assert(!prepare(65536,65536,name,provider)); + assert(!prepare(13,-1,name,provider));assert(!prepare(13,0,name,provider)); + assert(!prepare(13,1,name,provider));assert(!prepare(13,13,NULL,provider)); + assert(!prepare(13,13,"dcpdp-service-epic",provider)); + announce(13,"wrong-compartment",provider,13);assert(opens==before && !d.dcpavserv.service); + ep.endpoint=0x2c;announce(13,name,provider,13);assert(opens==before && !d.dcpavserv.service);ep.endpoint=0x2a; + d.quiescing=true;announce(13,name,provider,13);assert(opens==before && !d.dcpavserv.service);d.quiescing=false; + disconnect();announce(13,name,provider,13);assert(opens==before && !d.dcpavserv.service);connect(); + } + for(unsigned int variant=0;variant<4;variant++) { + const char *forged[]={"DCPDP13ServiceExtra","dcpdp13service","AppleDCPDP2HDMIExtra","unrelated-provider"}; + announce(13,name,forged[variant],13);assert(opens==before && !d.dcpavserv.service); + } + assert(!prepare(13,13,name,NULL)); + open_error=-EIO;announce(13,name,native_class,13);assert(opens==++before && !d.dcpavserv.service);open_error=0; + for(unsigned int window=0;window<5;window++) { + const int actions[]={1,2,3,5,6};open_action=actions[window]; + announce(13,name,native_class,13); + assert(opens==++before && !d.dcpavserv.service && !d.dcpavserv.enabled); + d.typec_cable_connected=true;d.quiescing=false; + } + open_action=4;announce(13,name,native_class,13);assert(opens==++before && d.dcpavserv.service==injected_owner); + assert(injected_owner->users==1);dpavservep_detach(&d);assert(!injected_owner->users); + open_action=0;announce(13,name,native_class,13);assert(opens==++before && d.dcpavserv.service); + struct apple_epic_service *old=d.dcpavserv.service; + assert(afk_service_get(old)==old);ep.command_tags[13]=1;old->cmd_map[0]=1; + disconnect();connect(); + announce(13,name,native_class,13); /* Same channel still owns a pending reply. */ + assert(opens==before && !d.dcpavserv.service && old->users==1 && old->channel==13); + /* The firmware need not announce again after the old reply drains. */ + old->cmds[0]=(struct epic_cmd_info){.tag=0,.type=0xc0,.rxlen=32,.txlen=32,.free_on_ack=true}; + old->cmds[0].rxbuf=calloc(1,32);old->cmds[0].txbuf=calloc(1,32); + old->cmds[0].rxbuf_dma=(uintptr_t)old->cmds[0].rxbuf; + old->cmds[0].txbuf_dma=(uintptr_t)old->cmds[0].txbuf; + u8 drain_oob[32],drain_reply[56];size_t drain_written; + assert(!afkv2_encode_oob_command(drain_oob,sizeof(drain_oob),0,old->cmds[0].rxbuf_dma,32,old->cmds[0].txbuf_dma,32)); + afkv2_put32(drain_oob+4,0); + assert(!afkv2_encode_frame(drain_reply,sizeof(drain_reply),1,13,0,0xc0,AFKV2_RESPONSE,0,drain_oob,sizeof(drain_oob),&drain_written)); + afk_v2_receive(&ep,0,0,drain_reply,drain_written); + run_retries(); + assert(d.dcpavserv.service && "a blocked reannouncement must recover after the old ACK without another announcement"); + assert(d.dcpavserv.service!=old && opens==++before && old->users==1 && !old->cmd_map[0]); + struct apple_epic_service *resumed=d.dcpavserv.service; + resumed->cmd_map[0]=1; + /* The DMA allocator can reuse both old addresses for the new command. */ + resumed->lock=true; + resumed->cmds[0]=(struct epic_cmd_info){.tag=afk_command_tag(resumed,0),.type=0xc0, + .rxbuf_dma=old->cmds[0].rxbuf_dma,.txbuf_dma=old->cmds[0].txbuf_dma,.rxlen=32,.txlen=32}; + resumed->lock=false; + afk_v2_receive(&ep,0,0,drain_reply,drain_written); + assert(!resumed->cmds[0].done && resumed->cmd_map[0] && "old ACK must not complete the next owner's command"); + resumed->cmd_map[0]=0; + dpavservep_detach(&d); + announce(1,name,class,1);assert(opens==++before && d.dcpavserv.service!=old); + struct apple_epic_service *fresh=d.dcpavserv.service;u64 epoch=d.dcpavserv.epoch; + dcpavserv_teardown(old);assert(d.dcpavserv.service==fresh && d.dcpavserv.epoch==epoch); + old->cmd_map[0]=0;afk_service_put(old);disconnect();connect(); + run_retries(); + announce(13,name,native_class,13);assert(opens==++before && d.dcpavserv.service); + struct apple_epic_service *owner=d.dcpavserv.service; + u64 owner_epoch=d.dcpavserv.epoch; + announce(5,name,class,5); /* A second admitted identity cannot displace the owner. */ + assert(opens==before && d.dcpavserv.service==owner && d.dcpavserv.epoch==owner_epoch); + dpavservep_detach(&d); + /* All slots held by retired readers: fail safely, then recover after puts. */ + for(unsigned int i=0;itorndown); + legacy=prepare(13,0,name,native_class);assert(legacy); + d.typec_generation++;afk_service_publish(legacy);dcpavserv_init(legacy,name,native_class,0); + assert(d.dcpavserv.service==legacy && legacy->users==1);dpavservep_detach(&d); + assert(dcpdpserv_prepare(legacy,"dcpdp-service-epic",native_class,0)==0); + ep.wire_v2=true;ep.endpoint=0x2a;d.external_physically_absent=false; + announce(U16_MAX,name,native_class,U16_MAX);assert(d.dcpavserv.service && d.dcpavserv.service->channel==U16_MAX); + dpavservep_detach(&d); + puts("PASS: actual AFKV2 HDMI/native DP alternating over 64 cycles: flip/revalidation/return, strict identity, stale OPEN generation/epoch/owner, auxiliary isolation, reader/command pins and legacy controls"); +} + +static void deferred_announcements(void) +{ + for(unsigned int cycle=0;cycle<64;cycle++) { + unsigned int event=cycle%10,before=opens,freed=dma_frees; + u16 channel=1+4*(cycle%4); + const char *provider=(cycle%2)?native_class:class; + announce(channel,name,provider,channel); + struct apple_epic_service *old=d.dcpavserv.service; + assert(old && opens==++before && afk_service_get(old)==old); + old->cmd_map[0]=1; + old->cmds[0]=(struct epic_cmd_info){.tag=0,.type=0xc0,.rxlen=32,.txlen=32,.free_on_ack=true}; + old->cmds[0].rxbuf=calloc(1,32);old->cmds[0].txbuf=calloc(1,32); + old->cmds[0].rxbuf_dma=(uintptr_t)old->cmds[0].rxbuf; + old->cmds[0].txbuf_dma=(uintptr_t)old->cmds[0].txbuf; + u8 oob[32],reply[56];size_t written; + assert(!afkv2_encode_oob_command(oob,sizeof(oob),0,old->cmds[0].rxbuf_dma,32,old->cmds[0].txbuf_dma,32)); + afkv2_put32(oob+4,0); + assert(!afkv2_encode_frame(reply,sizeof(reply),1,channel,0,0xc0,AFKV2_RESPONSE,0,oob,sizeof(oob),&written)); + u64 generation=d.typec_generation; + closed(channel,0x6a01,0); + assert(!d.dcpavserv.service && old->users==1 && old->torndown && d.typec_generation==generation); + u64 epoch=d.dcpavserv.epoch; + announce(channel,name,provider,channel); + run_retries(); + assert(opens==before && !d.dcpavserv.service && dma_frees==freed); + for(unsigned int i=0;ipending_open) continue; + assert(dcpavserv_revalidate(pending,pending->pending_name,pending->pending_class,pending->pending_unit)==0); + d.dcpavserv.enabled=true; + assert(dcpavserv_revalidate(pending,pending->pending_name,pending->pending_class,pending->pending_unit)==-EBUSY); + d.dcpavserv.enabled=false; + } + if(event==1) d.typec_generation++; + if(event==2) dpavservep_detach(&d); + if(event==3) d.typec_cable_connected=false; + if(event==4) d.quiescing=true; + struct apple_epic_service *other=NULL; + if(event==5) { other=prepare(channel+2,channel+2,name,class);publish(other,channel+2); } + if(event==6) assert(!ep.stopping); + if(event==7) open_error=-EIO; + if(event==8) announce(channel,name,provider,channel); + if(event==9) { d.typec_generation++;announce(channel,name,provider,channel); } + afk_v2_receive(&ep,0,0,reply,written); + if(event==6) { assert(ep.service_work.pending);ep.stopping=true; } + run_retries(); + assert(dma_frees==freed+2 && old->users==1 && !old->cmd_map[0]); + if(event==0 || event==8 || event==9) { + assert(d.dcpavserv.service && d.dcpavserv.service!=old); + assert(d.dcpavserv.service->channel==channel && opens==++before); + assert(d.dcpavserv.epoch>epoch && d.dcpavserv.service->users==1); + } else if(event==5) { + assert(d.dcpavserv.service==other && opens==before); + } else { + assert(!d.dcpavserv.service && opens==before+(event==7)); + } + afk_v2_receive(&ep,0,0,reply,written); /* Duplicate late ACK cannot touch the new owner. */ + assert(dma_frees==freed+2); + afk_service_put(old);run_retries(); + assert(!old->users); + if(event==6) { + for(unsigned int i=0;iusers==1); + assert(afk_service_get(old)==old); /* Delayed EDID reader pins the retired slot. */ + old->cmds[0]=(struct epic_cmd_info){.tag=0,.type=0xc0,.rxlen=32,.txlen=32,.free_on_ack=true}; + old->cmds[0].rxbuf=calloc(1,32);old->cmds[0].txbuf=calloc(1,32);assert(old->cmds[0].rxbuf && old->cmds[0].txbuf); + old->cmds[0].rxbuf_dma=(uintptr_t)old->cmds[0].rxbuf;old->cmds[0].txbuf_dma=(uintptr_t)old->cmds[0].txbuf;old->cmd_map[0]=1; + disconnect();assert(!d.dcpavserv.service && !d.dcpavserv.enabled && old->users==1 && old->torndown && !old->enabled); + assert(con.edid_invalidation_pending && !afk_service_get(old)); + assert(!afk_service_reinit(old,&ep,&ops,5)); + assert(!prepare(5,5,name,class)); /* No admission while physically disconnected. */ + connect();struct apple_epic_service *fresh=prepare(5,5,name,class);assert(fresh && fresh!=old);publish(fresh,5); + assert(d.dcpavserv.service==fresh && fresh->users==1); + u64 epoch=d.dcpavserv.epoch; + dcpavserv_teardown(old);assert(d.dcpavserv.service==fresh && d.dcpavserv.epoch==epoch); + closed(1,0x6a01,0);assert(d.dcpavserv.service==fresh && d.dcpavserv.epoch==epoch && !dma_frees); + u8 oob[32],reply[56];size_t written; + assert(!afkv2_encode_oob_command(oob,sizeof(oob),0,old->cmds[0].rxbuf_dma,32,old->cmds[0].txbuf_dma,32));afkv2_put32(oob+4,0); + assert(!afkv2_encode_frame(reply,sizeof(reply),1,1,0,0xc0,AFKV2_RESPONSE,0,oob,sizeof(oob),&written)); + afk_v2_receive(&ep,0,0,reply,written); + assert(dma_frees==2 && !old->cmd_map[0] && old->users==1 && d.dcpavserv.service==fresh && d.dcpavserv.epoch==epoch); + assert(!afk_service_reinit(old,&ep,&ops,9));afk_service_put(old); + afk_v2_receive(&ep,0,0,reply,written);assert(dma_frees==2); /* Duplicate late ACK. */ + disconnect();connect();struct apple_epic_service *next=prepare(9,9,name,class);assert(next);publish(next,9); + assert(d.dcpavserv.service==next); + closed(1,0x6a01,0);closed(5,0x6a01,0);assert(d.dcpavserv.service==next); + disconnect();connect();next=prepare(13,13,name,class);assert(next);publish(next,13);assert(d.dcpavserv.service==next); + closed(13,0x6a01,1);assert(d.dcpavserv.service==next); /* Nonzero result rejected. */ + closed(13,0x6a00,0);assert(d.dcpavserv.service==next); /* Wrong low flags rejected. */ + d.dcpavservep=NULL;closed(13,0x6a01,0);assert(d.dcpavserv.service==next);d.dcpavservep=&ep; + closed(13,0x6a01,0);assert(!d.dcpavserv.service && !d.dcpavserv.enabled); + u8 result_byte=0;struct afkv2_frame notification={.category=AFKV2_REPORT,.type=AFKV2_CLOSED,.interface_flags=1,.payload=&result_byte,.payload_size=1}; + assert(afkv2_closed_report(¬ification));notification.token=1;assert(!afkv2_closed_report(¬ification)); + notification.token=0;notification.payload_size=0;assert(!afkv2_closed_report(¬ification)); + assert(!prepare(0,0,name,class));assert(!prepare(65536,65536,name,class)); + assert(!prepare(9,1,name,class));assert(!prepare(9,9,"wrong",class));assert(!prepare(9,9,name,"wrong")); + struct apple_epic_service *pending=prepare(9,9,name,class);assert(pending); + disconnect();connect();publish(pending,9); + assert(!d.dcpavserv.service && pending->torndown && !pending->enabled && !pending->users); + pending=prepare(9,9,name,class);assert(pending); + mutex_lock(&d.hpd_mutex);d.typec_generation++;mutex_unlock(&d.hpd_mutex); + publish(pending,9); /* Generation change alone, without registry retirement. */ + assert(!d.dcpavserv.service && pending->torndown && !pending->enabled); + pending=prepare(13,13,name,class);assert(pending); + dpavservep_detach(&d);publish(pending,13); /* Registry retirement with no physical change. */ + assert(!d.dcpavserv.service && pending->torndown && !pending->enabled); + ep.wire_v2=false;ep.endpoint=0x28;d.hw.iomfb_method_profile=0; + assert(!prepare(7,7,name,class));pending=prepare(7,0,name,NULL);assert(pending);publish(pending,0); + assert(d.dcpavserv.service==pending && pending->users==1);dpavservep_detach(&d); + assert(!pending->users && !d.hpd_mutex && !d.dcpavserv.lock); + memset(&ep.services,0,sizeof(ep.services));ep.wire_v2=true;ep.endpoint=0x2a;d.hw.iomfb_method_profile=2; + repeated_announcements(); + deferred_announcements(); + free(ep.command_tags);ep.command_tags=NULL; + puts("PASS: actual dynamic 1/5/9/13 binding, forged identities, prepare/init epochs, physical retirement, reader pins, real old CLOSED/late ACK isolation and legacy unit 0"); +} +''' + return prefix + '\n'.join(parts) + suffix + +def admission(root): + + base = root / 'drivers/gpu/drm/apple' + + def function(file, signature): + source = (base / file).read_text() + start = source.index(signature) + opening = source.index('{', start) + while ';' in source[start:opening]: + start = source.index(signature, start + 1) + opening = source.index('{', start) + end, depth = opening + 1, 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + return source[start:end] + + av = (base / 'epic/dpavservep.c').read_text() + af = (base / 'afk.c').read_text() + announce = af[af.index('static void afk_recv_handle_init('):af.index('static void afk_recv_handle_init(') + 6500] + assert announce.index('ops->prepare(service,') < announce.index('EPIC_CAT_REPORT, AFKV2_OPEN') + assert '.prepare = dcpavserv_prepare,' in av and '.prepare = dcpdpserv_prepare,' in av + prefix = r''' +#undef NDEBUG +#include +#include +#include +#include +#include +#include +#include +#include +typedef uint8_t u8; +typedef uint64_t u64; +typedef int64_t s64; +#define U16_MAX UINT16_MAX +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x)=(v)) +#define ERR_PTR(x) ((void *)(intptr_t)(x)) +#define PTR_ERR(x) ((int)(intptr_t)(x)) +#define IS_ERR(x) ((uintptr_t)(x) > (uintptr_t)-4096) +#define spin_lock_irqsave(p,f) do { assert(!*(p)); *(p)=true; (f)=0; } while(0) +#define spin_unlock_irqrestore(p,f) do { (void)(f); assert(*(p)); *(p)=false; } while(0) +#define lockdep_assert_held(p) assert(*(p)) +static bool *acquire(bool *p) { assert(!*p); *p=true; return p; } +static void release(bool **p) { if(*p) { assert(**p); **p=false; } } +static bool *once(bool *p) { assert(*p); *p=false; return NULL; } +#define scoped_guard(k,p) for(bool *_lock __attribute__((cleanup(release)))=acquire(p); _lock; _lock=once(_lock)) +static void mutex_lock(bool *p) { (void)acquire(p); } +static void mutex_unlock(bool *p) { (void)once(p); } +#define dev_info(...) ((void)0) +#define dev_err(...) ((void)0) +#define trace_dcpavserv_init(...) ((void)0) +#define DRM_CONNECTOR_REGISTERED 1 +#define to_platform_device(x) (x) +static void reinit_completion(int *p) { *p=0; } +static void complete(int *p) { *p=1; } +static void schedule_work(int *p) { (*p)++; } +#define dev_dbg_ratelimited(...) ((void)0) +#define dev_warn_ratelimited(...) ((void)0) +#define trace_dcp_recv_msg(...) ((void)0) +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define DCP_FIRMWARE_H17P 2 +#define AV_ENDPOINT 0x29 +#define SYSTEM_ENDPOINT 0x20 +#define DISP0_ENDPOINT 0x23 +#define DPAVSERV_ENDPOINT 0x28 +#define DPAVSERV_ENDPOINT_V2 0x2a +#define DPTX_ENDPOINT 0x2a +#define DPTX_ENDPOINT_V2 0x2c +#define DPTX_CONTROLLER_ENDPOINT 0x24 +#define IOMFB_ENDPOINT 0x37 +struct apple_dcp; +struct apple_connector { void *dcp; struct { int registration_state; } base; bool connected, edid_invalidation_pending; int hotplug_wq; }; +struct apple_dcp_afkep { struct apple_dcp *dcp; u8 endpoint; bool wire_v2; }; +struct apple_epic_service { struct apple_dcp_afkep *ep; unsigned int users; bool enabled,torndown; struct { u64 generation,epoch; } admission; void *cookie; unsigned int channel; }; +struct apple_dcp { + struct apple_dcp_afkep *dcpavservep, *dptxep, *avep, *systemep, *ibootep, *dptx_controller_ep; + struct { bool lock, enabled; u64 epoch; int enable_completion; struct apple_epic_service *service; } dcpavserv; + struct apple_connector *connector; unsigned int nr_modes; + struct { int iomfb_method_profile; } hw; + bool external_physically_absent,hpd_mutex, typec_cable_connected, quiescing; + u64 typec_generation; + int fw_compat; + void *dev; +}; +struct drm_edid { int identity; }; +static unsigned int reads, allocations, legacy_reads, receives; +static unsigned int event; +static struct apple_dcp_afkep *received_ep; +static struct apple_epic_service *afk_service_get(struct apple_epic_service *s) +{ if(!s || !s->enabled || s->torndown) return NULL; s->users++; return s; } +static void afk_service_put(struct apple_epic_service *s) +{ assert(s->users); s->users--; } +static void afk_service_retire(struct apple_epic_service *s) { s->enabled=false;s->torndown=true; } +static void drm_edid_free(const struct drm_edid *e) +{ if(e) { assert(allocations); allocations--; free((void *)e); } } +static void dcpavserv_teardown(struct apple_epic_service *s); +static const struct drm_edid *dcpavserv_read_edid_inline(struct apple_epic_service *s) +{ + struct apple_dcp *d=s->ep->dcp; + assert(!d->hpd_mutex && !d->dcpavserv.lock && s->users==2); + reads++; + if(event==1 || event==2) { d->typec_generation++; d->typec_cable_connected=(event==2); } + if(event==3) { s->enabled=false; dcpavserv_teardown(s); } + struct drm_edid *e=malloc(sizeof(*e)); assert(e); e->identity=42; allocations++; return e; +} +static const struct drm_edid *dcpavserv_read_edid_legacy(struct apple_epic_service *s) +{ legacy_reads++; return dcpavserv_read_edid_inline(s); } +static void iomfb_recv_msg(struct apple_dcp *d, u64 message) { (void)d; (void)message; } +static void afk_receive_message(struct apple_dcp_afkep *ep, u64 message) +{ (void)message; received_ep=ep; receives++; } +''' + parts = [function('epic/dpavservep.c', 'static void dcpavserv_notify_edid('), + function('epic/dpavservep.c', 'static void dcpavserv_invalidate_edid('), + function('epic/dpavservep.c', 'static void dcpavserv_teardown('), + function('dcp.c', 'static struct apple_dcp_afkep *dcp_afkep('), + function('dcp.c', 'static void dcp_recv_msg('), + function('epic/dpavservep.c', 'static bool dcpavserv_identity('), + function('epic/dpavservep.c', 'static bool dcpavserv_admission_current('), + function('epic/dpavservep.c', 'static int dcpavserv_prepare('), + function('epic/dpavservep.c', 'static int dcpdpserv_prepare('), + function('epic/dpavservep.c', 'static void dcpavserv_init('), + function('epic/dpavservep.c', 'bool dcpavserv_edid_current('), + function('epic/dpavservep.c', 'bool dcpavserv_edid_service_current('), + function('epic/dpavservep.c', 'const struct drm_edid *dcpavserv_copy_edid(')] + suffix = r''' +int main(void) +{ + u64 epoch=0; + int device; + struct apple_connector connector={.dcp=&device,.base.registration_state=DRM_CONNECTOR_REGISTERED,.connected=true}; + struct apple_dcp d={.fw_compat=DCP_FIRMWARE_H17P,.dev=&device,.connector=&connector,.nr_modes=1}; + struct apple_dcp_afkep av={.dcp=&d,.endpoint=0x28}, tx={.dcp=&d,.endpoint=0x2a}; + d.dcpavservep=&av; d.dptxep=&tx; + assert(dcp_afkep(&d,0x28)==&av && dcp_afkep(&d,0x2a)==&tx && !dcp_afkep(&d,0x2c)); + dcp_recv_msg(&d,0x28,1); assert(received_ep==&av && receives==1); + dcp_recv_msg(&d,0x2a,2); assert(received_ep==&tx && receives==2); + d.hw.iomfb_method_profile=DCP_IOMFB_METHODS_H17P_EXTERNAL; + av.endpoint=0x2a; av.wire_v2=true; tx.endpoint=0x2c; + assert(!dcp_afkep(&d,0x28) && dcp_afkep(&d,0x2a)==&av && dcp_afkep(&d,0x2c)==&tx); + dcp_recv_msg(&d,0x28,3); assert(receives==2); + dcp_recv_msg(&d,0x2a,4); assert(received_ep==&av && receives==3); + dcp_recv_msg(&d,0x2c,5); assert(received_ep==&tx && receives==4); + d.dcpavservep=NULL; dcp_recv_msg(&d,0x2a,6); assert(receives==4); d.dcpavservep=&av; + struct apple_epic_service s={.ep=&av,.users=1,.enabled=true,.channel=1}; + const char *name="dcpav-service-epic", *class="AppleDCPDP2HDMI"; + assert(!dcpavserv_prepare(&s,name,class,1)); + assert(dcpavserv_prepare(&s,name,class,0)==-ENODEV); + assert(dcpavserv_prepare(&s,name,class,3)==-ENODEV); + assert(dcpavserv_prepare(&s,name,class,-1)==-ENODEV); + assert(dcpavserv_prepare(&s,"dcpdp-service-epic",class,1)==-ENODEV); + assert(dcpavserv_prepare(&s,name,"wrong",1)==-ENODEV); + assert(dcpavserv_prepare(&s,NULL,class,1)==-ENODEV); + assert(dcpavserv_prepare(&s,name,NULL,1)==-ENODEV); + av.endpoint=0x28; assert(dcpavserv_prepare(&s,name,class,1)==-ENODEV); av.endpoint=0x2a; + assert(dcpdpserv_prepare(&s,"dcpdp-service-epic",class,3)==-EOPNOTSUPP); + d.dcpavserv.enabled=true; assert(dcpavserv_prepare(&s,name,class,1)==-EBUSY); + d.dcpavserv.service=&s; d.dcpavserv.epoch=1; d.typec_generation=1; d.typec_cable_connected=true; + const struct drm_edid *e=dcpavserv_copy_edid(&d,1,&epoch); + assert(!IS_ERR(e) && s.users==1 && reads==1); drm_edid_free(e); + d.typec_cable_connected=false; + assert(PTR_ERR(dcpavserv_copy_edid(&d,1,&epoch))==-ESTALE && reads==1); + d.typec_cable_connected=true; + assert(PTR_ERR(dcpavserv_copy_edid(&d,0,&epoch))==-ESTALE && reads==1); + event=1; assert(PTR_ERR(dcpavserv_copy_edid(&d,1,&epoch))==-ESTALE && !allocations && s.users==1); + event=2; d.typec_cable_connected=true; + assert(PTR_ERR(dcpavserv_copy_edid(&d,2,&epoch))==-ESTALE && !allocations && s.users==1); + event=0; e=dcpavserv_copy_edid(&d,3,&epoch); assert(!IS_ERR(e)); drm_edid_free(e); + event=3; assert(PTR_ERR(dcpavserv_copy_edid(&d,3,&epoch))==-ENODEV && !allocations && !s.users); + assert(PTR_ERR(dcpavserv_copy_edid(&d,3,&epoch))==-ENODEV); + assert(connector.edid_invalidation_pending && connector.hotplug_wq && d.dcpavserv.epoch==2); + /* Same slot and physical generation: CLOSE/re-OPEN gets a fresh ticket. */ + s.enabled=true; assert(!dcpavserv_prepare(&s,name,class,1)); dcpavserv_init(&s,name,class,1); + assert(d.dcpavserv.service==&s && s.users==1 && d.dcpavserv.epoch==3); + mutex_lock(&d.dcpavserv.lock); + assert(!dcpavserv_edid_service_current(&d,1) && dcpavserv_edid_service_current(&d,3)); + mutex_unlock(&d.dcpavserv.lock); + event=0; e=dcpavserv_copy_edid(&d,3,&epoch); assert(!IS_ERR(e) && epoch==3); + /* Simulate retirement in the final validation-to-publication gap. */ + s.enabled=false; dcpavserv_teardown(&s); s.enabled=true; assert(!dcpavserv_prepare(&s,name,class,1)); dcpavserv_init(&s,name,class,1); + mutex_lock(&d.dcpavserv.lock); assert(!dcpavserv_edid_service_current(&d,epoch)); mutex_unlock(&d.dcpavserv.lock); + drm_edid_free(e); assert(d.typec_generation==3 && d.dcpavserv.epoch==5 && s.users==1); + s.enabled=false; dcpavserv_teardown(&s); + av.wire_v2=false; av.endpoint=0x28; d.hw.iomfb_method_profile=0; + d.dcpavserv.enabled=false; assert(!dcpavserv_prepare(&s,name,NULL,0)); + assert(dcpavserv_prepare(&s,name,class,1)==-ENODEV); + assert(!dcpdpserv_prepare(&s,NULL,NULL,3)); + d.dcpavserv.service=&s; d.dcpavserv.enabled=true; s.enabled=true; s.users=1; event=0; + d.typec_cable_connected=false; e=dcpavserv_copy_edid(&d,0,&epoch); + assert(!IS_ERR(e) && legacy_reads==1 && s.users==1); drm_edid_free(e); + d.hw.iomfb_method_profile=0;av.wire_v2=false;d.external_physically_absent=true; + mutex_lock(&d.hpd_mutex);assert(dcpavserv_edid_current(&d,0));mutex_unlock(&d.hpd_mutex); + assert(!allocations && !d.hpd_mutex && !d.dcpavserv.lock); + puts("PASS: production old/new lookup and receive, exact service admission before OPEN, delayed disconnect/replug, service detach pins, legacy transport"); +} +''' + return prefix + '\n'.join(parts) + suffix + +def rx_progress(root): + + def source(path): + return (root / path).read_text() + + def function(text, signature): + start = text.index(signature) + opening = text.index('{', start) + while ';' in text[start:opening]: + start = text.index(signature, start+1) + opening = text.index('{', start) + end, depth = opening+1, 1 + while depth: + depth += (text[end]=='{')-(text[end]=='}') + end += 1 + return text[start:end] + + dcp=source('drivers/gpu/drm/apple/dcp.c') + iomfb=source('drivers/gpu/drm/apple/iomfb.c') + af=source('drivers/gpu/drm/apple/afk.c') + rtkit=source('drivers/soc/apple/rtkit.c') + template=source('drivers/gpu/drm/apple/iomfb_template.c') + assert 'alloc_ordered_workqueue("rtkit-%s"' in rtkit + assert '.recv_message = dcp_recv_msg' in dcp + assert 'dcp_set_dimensions(dcp);' in template[template.index('"DisplayAttributes"'):] + assert 'queue_work(ep->wq, &work->work)' in af + prefix=r''' +#undef NDEBUG +#include +#include +#include +#include +#include +#include +#include +#include +#include +typedef uint8_t u8; +typedef uint32_t u32; +typedef uint64_t u64; +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x)=(v)) +#define GFP_KERNEL 0 +#define kzalloc(n,f) calloc(1,n) +#define kfree(p) free(p) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#define INIT_WORK(p,f) ((p)->fn=(f)) +#define dev_warn(...) ((void)0) +#define DCP_FW_VER 26 +#define DCP_FW_VERSION(a,b,c) (a) +#define SZ_1M (1024*1024) +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define dev_warn_ratelimited(...) ((void)0) +#define dev_dbg_ratelimited(...) ((void)0) +#define trace_dcp_recv_msg(...) ((void)0) +#define DCP_FIRMWARE_H17P 2 +#define IOMFB_ENDPOINT 0x37 +#define AV_ENDPOINT 0x29 +#define SYSTEM_ENDPOINT 0x20 +#define DISP0_ENDPOINT 0x23 +#define DPTX_CONTROLLER_ENDPOINT 0x24 +#define APPLE_RTKIT_EP_MGMT 0 +#define APPLE_RTKIT_EP_CRASHLOG 1 +#define APPLE_RTKIT_EP_SYSLOG 2 +#define APPLE_RTKIT_EP_IOREPORT 4 +#define APPLE_RTKIT_EP_OSLOG 8 +#define APPLE_RTKIT_APP_ENDPOINT_START 0x20 +struct mutex { pthread_mutex_t native; }; +static _Thread_local struct mutex *locks[4]; +static _Thread_local unsigned int depth; +static bool held(struct mutex *m) { for(unsigned int i=0;inative)); locks[depth++]=m; } +static void mutex_unlock(struct mutex *m) { assert(depth && locks[depth-1]==m); depth--; assert(!pthread_mutex_unlock(&m->native)); } +#define lockdep_assert_held(p) assert(held(p)) +#define spin_lock_irqsave(p,f) do { mutex_lock(p); (f)=0; } while(0) +#define spin_unlock_irqrestore(p,f) do { (void)(f); mutex_unlock(p); } while(0) +struct work_struct { void (*fn)(struct work_struct *); }; +struct workqueue_struct { struct work_struct *pending; pthread_cond_t ready; pthread_mutex_t lock; }; +struct drm_device { struct { struct mutex mutex; } mode_config; }; +struct drm_connector { struct drm_device *dev; struct { int width_mm,height_mm; } display_info; }; +struct apple_connector { struct drm_connector base; struct work_struct hotplug_wq; }; +struct apple_dcp; +struct apple_dcp_afkep { struct apple_dcp *dcp; u8 endpoint; struct mutex lock; bool stopping; struct workqueue_struct *wq; }; +struct test_mode_ticket { u64 generation; bool associated; }; +struct apple_dcp { + bool quiescing; int fw_compat; void *dev; + struct { int iomfb_method_profile; } hw; + struct { struct mutex lock; u64 epoch; bool enabled; void *service; } dcpavserv; + struct test_mode_ticket dummy_ticket; + struct { u64 connection_sequence; } rate_proof; + struct { struct mutex lock; struct { u64 generation; bool connected; } epoch; } metadata; + u64 typec_generation; bool typec_cable_connected; + struct { size_t length; void *data; u64 av_service_epoch, automatic_connection_sequence; bool automatic_service_pending; struct test_mode_ticket mode_ticket; } chunks; + int width_mm,height_mm; struct { int width_mm,height_mm; } panel; + unsigned int nr_modes; struct { struct { int width_mm,height_mm; } mode; } modes[1]; + struct apple_connector *connector; + struct apple_dcp_afkep *dcpavservep,*dptxep,*avep,*systemep,*ibootep,*dptx_controller_ep; + struct { struct mutex lock; bool stopped; void *owner; } iomfb; +}; +struct afk_receive_message_work { struct apple_dcp_afkep *ep; u64 message; struct work_struct work; }; +struct apple_rtkit_ops { void (*recv_message)(void *,u8,u64); }; +struct apple_rtkit { bool shutting_down; const struct apple_rtkit_ops *ops; void *cookie,*dev; }; +struct apple_rtkit_rx_work { struct apple_rtkit *rtk; u8 ep; u64 msg; struct work_struct work; }; +#define current ((void *)1) +static pthread_mutex_t reply_lock=PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t reply_ready=PTHREAD_COND_INITIALIZER; +static bool reply_done; +static unsigned int advances, dimension_updates; +static struct workqueue_struct dimensions_queue={.ready=PTHREAD_COND_INITIALIZER,.lock=PTHREAD_MUTEX_INITIALIZER}; +static struct workqueue_struct afk_queue={.ready=PTHREAD_COND_INITIALIZER,.lock=PTHREAD_MUTEX_INITIALIZER}; +static void queue_work(struct workqueue_struct *q,struct work_struct *w) +{ assert(!pthread_mutex_lock(&q->lock)); assert(!q->pending); q->pending=w; pthread_cond_signal(&q->ready); assert(!pthread_mutex_unlock(&q->lock)); } +static void schedule_work(struct work_struct *w) { queue_work(&dimensions_queue,w); } +static void afk_receive_message_worker(struct work_struct *w) +{ + struct afk_receive_message_work *a=container_of(w,struct afk_receive_message_work,work); + assert(a->ep->endpoint==0x2a && a->message==7); + assert(!pthread_mutex_lock(&reply_lock)); reply_done=true; + pthread_cond_signal(&reply_ready); assert(!pthread_mutex_unlock(&reply_lock)); free(a); +} +static bool iomfb_uses_queue(struct apple_dcp *d) { (void)d; return true; } +static void iomfb_queue_advance(struct apple_dcp *d) { assert(held(&d->iomfb.lock)); advances++; } +static void dcp_set_dimensions(struct apple_dcp *d); +static u8 dcpep_cb_prop_start(struct apple_dcp *d,u32 *length); +static void dcp_firmware_metadata_begin(struct apple_dcp *d) { (void)d; } +#define dcp_metadata_begin(e) ((struct test_mode_ticket){.generation=(e)->generation,.associated=(e)->connected}) +static void iomfb_recv_message(struct apple_dcp *d,u64 message) +{ + assert(message==1 && held(&d->iomfb.lock)); dcp_set_dimensions(d); + u32 length=64; + assert(dcpep_cb_prop_start(d,&length)); +} +#define rx_unused(name) static void name(struct apple_rtkit *r,u64 m) { (void)r;(void)m; } +rx_unused(apple_rtkit_management_rx) +rx_unused(apple_rtkit_crashlog_rx) +rx_unused(apple_rtkit_syslog_rx) +rx_unused(apple_rtkit_ioreport_rx) +rx_unused(apple_rtkit_oslog_rx) +''' + parts=[function(dcp,'void dcp_set_dimensions('), function(template,'static u8 dcpep_cb_prop_start(')] + if 'void dcp_update_connector_dimensions(' in dcp: + parts.append(function(dcp,'void dcp_update_connector_dimensions(')) + else: + parts.append('static void dcp_update_connector_dimensions(struct apple_dcp *d, struct drm_connector *c) { (void)d; (void)c; }') + parts += [function(iomfb,'void iomfb_recv_msg('),function(af,'int afk_receive_message('), + function(dcp,'static struct apple_dcp_afkep *dcp_afkep('), + function(dcp,'static void dcp_recv_msg('),function(rtkit,'static void apple_rtkit_rx_work(')] + suffix=r''' +static struct drm_device drm={.mode_config.mutex.native=PTHREAD_MUTEX_INITIALIZER}; +static struct apple_connector connector={.base.dev=&drm}; +static struct apple_dcp d={.connector=&connector,.width_mm=600,.height_mm=340,.nr_modes=1,.fw_compat=DCP_FIRMWARE_H17P,.iomfb.lock.native=PTHREAD_MUTEX_INITIALIZER}; +static struct apple_dcp_afkep av={.dcp=&d,.endpoint=0x2a,.lock.native=PTHREAD_MUTEX_INITIALIZER,.wq=&afk_queue}; +static void dimension_work(struct work_struct *w) +{ + (void)w; + mutex_lock(&drm.mode_config.mutex); + dcp_update_connector_dimensions(&d,&connector.base); dimension_updates++; + mutex_unlock(&drm.mode_config.mutex); +} +static void *worker(void *arg) +{ + struct workqueue_struct *q=arg; + assert(!pthread_mutex_lock(&q->lock)); + while(!q->pending) assert(!pthread_cond_wait(&q->ready,&q->lock)); + struct work_struct *w=q->pending; q->pending=NULL; + assert(!pthread_mutex_unlock(&q->lock)); w->fn(w); return NULL; +} +static const struct apple_rtkit_ops ops={.recv_message=dcp_recv_msg}; +static struct apple_rtkit rtk={.ops=&ops,.cookie=&d}; +static void *ordered_rx(void *arg) +{ + (void)arg; + /* One ordered RX worker must complete DisplayAttributes before AFK. */ + struct apple_rtkit_rx_work *a=calloc(1,sizeof(*a)); assert(a); + a->rtk=&rtk; a->ep=IOMFB_ENDPOINT; a->msg=1; apple_rtkit_rx_work(&a->work); + a=calloc(1,sizeof(*a)); assert(a); + a->rtk=&rtk; a->ep=0x2a; a->msg=7; apple_rtkit_rx_work(&a->work); + return NULL; +} +int main(void) +{ + pthread_t rx,afk,dimensions; + (void)schedule_work; /* The negative baseline does not defer dimensions. */ + d.dcpavservep=&av; connector.hotplug_wq.fn=dimension_work; + d.hw.iomfb_method_profile=DCP_IOMFB_METHODS_H17P_EXTERNAL; + assert(!pthread_mutex_init(&d.dcpavserv.lock.native,NULL)); d.dcpavserv.epoch=1; + assert(!pthread_mutex_init(&d.metadata.lock.native,NULL)); d.metadata.epoch.generation=1; + d.metadata.epoch.connected=true; d.typec_generation=1; d.typec_cable_connected=true; + mutex_lock(&drm.mode_config.mutex); /* DRM get_modes holds this while waiting. */ + assert(!pthread_create(&afk,NULL,worker,&afk_queue)); + assert(!pthread_create(&dimensions,NULL,worker,&dimensions_queue)); + assert(!pthread_mutex_lock(&reply_lock)); + assert(!pthread_create(&rx,NULL,ordered_rx,NULL)); + struct timespec deadline; assert(!clock_gettime(CLOCK_REALTIME,&deadline)); deadline.tv_sec+=2; + int rc=0; + while(!reply_done && !rc) rc=pthread_cond_timedwait(&reply_ready,&reply_lock,&deadline); + if(!reply_done) { fputs("FAIL: ordered RTKit RX blocked before AFK reply while mode_config.mutex held\n",stderr); fflush(stderr); _Exit(1); } + assert(!pthread_mutex_unlock(&reply_lock)); + assert(!dimension_updates); /* Deferred worker cannot take the probe's mutex. */ + mutex_unlock(&drm.mode_config.mutex); + assert(!pthread_join(rx,NULL) && !pthread_join(afk,NULL) && !pthread_join(dimensions,NULL)); + assert(advances==1 && dimension_updates==1 && connector.base.display_info.width_mm==600 && connector.base.display_info.height_mm==340); + assert(d.modes[0].mode.width_mm==600 && d.modes[0].mode.height_mm==340); + assert(d.chunks.length==64 && d.chunks.data); free(d.chunks.data); + assert(!pthread_mutex_destroy(&d.dcpavserv.lock.native)); + assert(!pthread_mutex_destroy(&d.metadata.lock.native)); + puts("PASS: actual ordered RTKit RX -> synchronous IOMFB dimensions/property-start callbacks -> AFK enqueue makes reply progress while DRM mutex held; deferred dimensions update completes after release"); +} +''' + return prefix + '\n'.join(parts) + suffix diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/idle.py b/tools/testing/selftests/drivers/gpu/apple-dcp/idle.py new file mode 100644 index 00000000000000..7a0666fbe92ef6 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/idle.py @@ -0,0 +1,185 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Actual cpuidle driver/registration/selector and tick suspension source replay.""" +import re +def function(text, name): + match=re.search(r'^((?:static\s+)?(?:inline\s+)?(?:__cpuidle\s+|noinstr\s+)?(?:int|void)\s+'+re.escape(name)+r'\([^;{}]*?\))\s*\{',text,re.M) + if not match: raise ValueError(name) + start=match.start(); pos=match.end(); depth=1 + while depth: + if text[pos]=='{': depth+=1 + elif text[pos]=='}': depth-=1 + pos+=1 + return text[start:pos]+'\n' + +MOCKS = r''' +#include +#include +#include +#include +#include +#include +typedef uint64_t u64; typedef int64_t s64; typedef int64_t ktime_t; +#define CONFIG_SUSPEND 1 +#define CPUIDLE_NAME_LEN 16 +#define CPUIDLE_DESC_LEN 32 +#define CPUIDLE_STATE_MAX 10 +#define BIT(n) (1U<<(n)) +#define NSEC_PER_USEC 1000 +#define __cpuidle +#define noinstr +#define THIS_MODULE NULL +#define SYSTEM_SUSPEND 1 +#define SYSTEM_RUNNING 0 +#define TPS(x) x +#define instrumentation_begin() ((void)0) +#define instrumentation_end() ((void)0) +#define raw_spin_lock(x) ((void)0) +#define raw_spin_unlock(x) ((void)0) +#define lock_map_acquire_try(x) ((void)0) +#define lock_map_release(x) ((void)0) +#define trace_suspend_resume(...) ((void)0) +#define WARN_ON(x) ((x)?(warnings++,1):0) +#define WARN_ON_ONCE(x) WARN_ON(x) +#define pr_err(...) ((void)0) +#define for_each_cpu(cpu,mask) for(cpu=0;cpu<(mask)->count;cpu++) +#define per_cpu(v,cpu) v[cpu] +struct platform_device {int dummy;}; +struct module; struct cpumask {int count;}; +struct cpuidle_device; struct cpuidle_driver; +struct cpuidle_device {int cpu; struct cpuidle_state_usage states_usage[10];}; +static struct cpumask possible={6}; static const struct cpumask *cpu_possible_mask=&possible; +static struct cpuidle_device cpuidle_dev[6]; +static int current, irq[6], warnings, wfi, deep, pm, pm_fail, reg_fail=-1, drv_fail; +static int registered[6], unregistered, rcu[6], frozen, stopped[6], online=6; +static int tick_freeze_depth, tick_freeze_lock, tick_freeze_map, system_state; +static int tk_suspend, tk_resume, local_suspend, local_resume, sched_suspend, sched_resume; +static int fanout, faulty_irq, checks; +static uint64_t clock_ns; +static void check(bool b) {checks++; assert(b);} +static int num_online_cpus(void){return online;} +static void sched_clock_suspend(void){sched_suspend++;} +static void sched_clock_resume(void){sched_resume++;} +static void timekeeping_suspend(void){check(tick_freeze_depth==online);tk_suspend++;frozen=1;} +static void timekeeping_resume(void){check(frozen);tk_resume++;frozen=0;} +static void tick_suspend_local(void){local_suspend++;} +static void tick_resume_local(void){local_resume++;} +static void touch_softlockup_watchdog(void){} +static void ct_cpuidle_enter(void){check(irq[current]);rcu[current]++;} +static void ct_cpuidle_exit(void){check(irq[current]);rcu[current]--;} +static bool irqs_disabled(void){return irq[current];} +static void raw_local_irq_disable(void){irq[current]=1;} +static void local_irq_enable(void){check(!frozen);irq[current]=0;} +static void stop_critical_timings(void){stopped[current]++;} +static void start_critical_timings(void){stopped[current]--;} +static ktime_t local_clock_noinstr(void){return clock_ns+=1000;} +static ktime_t ns_to_ktime(ktime_t t){return t;} +static ktime_t ktime_us_delta(ktime_t a,ktime_t b){return (a-b)/1000;} +static u64 div_u64(u64 a,u64 b){return a/b;} +static u64 mul_u32_u32(unsigned a,unsigned b){return (u64)a*b;} +static bool t8140; +static bool of_machine_is_compatible(const char *s){return t8140&&!strcmp(s,"apple,t8140");} +static int cpu_pm_enter(void){pm++;return pm_fail;} +static void cpu_pm_exit(void){pm--;} +static void apple_cpu_deep_wfi(void){deep++;} +static void cpu_do_idle(void); +static int cpuidle_register_driver(struct cpuidle_driver *drv); +static int cpuidle_register_device(struct cpuidle_device *dev){if(dev->cpu==reg_fail)return -5; registered[dev->cpu]=1;return 0;} +static void cpuidle_unregister(struct cpuidle_driver *drv){(void)drv;memset(registered,0,sizeof(registered));unregistered++;} +''' +MAIN = r''' +static int cpuidle_register_driver(struct cpuidle_driver *drv){if(drv_fail)return -5;__cpuidle_driver_init(drv);return 0;} +static void cpu_do_idle(void){ + check(irq[current]);wfi++; + if(fanout && current<5){int previous=current;current++;irq[current]=1; + cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[current],100000);current=previous;} + if(faulty_irq)irq[current]=0; +} +static struct cpuidle_driver original; +static void reset(bool is_t8140){ + apple_idle_driver=original;t8140=is_t8140; current=0; + memset(cpuidle_dev,0,sizeof(cpuidle_dev)); memset(registered,0,sizeof(registered)); + for(int i=0;i<6;i++){irq[i]=1;rcu[i]=stopped[i]=0;} + wfi=deep=pm=pm_fail=warnings=unregistered=0;reg_fail=-1;drv_fail=0; + tick_freeze_depth=frozen=tk_suspend=tk_resume=local_suspend=local_resume=sched_suspend=sched_resume=0; + fanout=faulty_irq=0; +} +int main(void){ + original=apple_idle_driver; + reset(true);check(!apple_cpuidle_probe(NULL)); + check(apple_idle_driver.cpumask==cpu_possible_mask);check(!apple_idle_driver.bctimer); + for(int i=0;i<6;i++)check(registered[i]&&cpuidle_dev[i].cpu==i); + int expected=EXPECT_FIXED?1:0; + check(apple_idle_driver.state_count==(EXPECT_FIXED?2:1)); + check(find_deepest_state(&apple_idle_driver,&cpuidle_dev[0],100000,0,true)==expected); + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000)==expected); + check(tick_freeze_depth==0&&!frozen);check(deep==0&&pm==0); + check(local_suspend==expected&&local_resume==expected); + if(EXPECT_FIXED){ + check(apple_idle_driver.states[1].enter==apple_enter_wfi); + check(apple_idle_driver.states[1].enter_s2idle==apple_enter_wfi); + check(apple_idle_driver.states[1].flags==0); + check(apple_idle_driver.states[1].exit_latency_ns==1000); + check(apple_idle_driver.states[1].target_residency_ns==1000); + check(apple_idle_driver.states[1].power_usage==apple_idle_driver.states[0].power_usage); + reset(true);apple_cpuidle_probe(NULL);fanout=1; + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000)==1); + check(wfi==6&&deep==0&&pm==0);check(tk_suspend==1&&tk_resume==1); + check(sched_suspend==1&&sched_resume==1&&local_suspend==5&&local_resume==5); + check(tick_freeze_depth==0&&!frozen); + for(int i=0;i<6;i++)check(cpuidle_dev[i].states_usage[1].s2idle_usage==1&&rcu[i]==0&&stopped[i]==0); + reset(true);apple_cpuidle_probe(NULL);fanout=1;cpuidle_dev[5].states_usage[1].disable=1; + cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000); + check(wfi==5&&tk_suspend==0&&tk_resume==0&&tick_freeze_depth==0); + reset(true);apple_cpuidle_probe(NULL);cpuidle_dev[0].states_usage[1].disable=1; + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000)==0&&wfi==0); + cpuidle_dev[0].states_usage[1].disable=0;apple_idle_driver.states[1].enter_s2idle=NULL; + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000)==0&&wfi==0); + reset(true);apple_cpuidle_probe(NULL); + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],999)==0); + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],1000)==1); + reset(true);apple_cpuidle_probe(NULL);apple_idle_driver.states[1].exit_latency_ns=0; + check(find_deepest_state(&apple_idle_driver,&cpuidle_dev[0],100000,0,true)==0); + reset(true);apple_cpuidle_probe(NULL);apple_idle_driver.states[1].flags=CPUIDLE_FLAG_TIMER_STOP; + check(find_deepest_state(&apple_idle_driver,&cpuidle_dev[0],100000,CPUIDLE_FLAG_TIMER_STOP,false)==0); + reset(true);apple_cpuidle_probe(NULL);faulty_irq=1; + check(cpuidle_enter_s2idle(&apple_idle_driver,&cpuidle_dev[0],100000)==1); + check(warnings==1&&tick_freeze_depth==0&&rcu[0]==0&&stopped[0]==0); + } + reset(true);apple_cpuidle_probe(NULL); + for(int i=0;i', '') + core = 'drivers/usb/typec/tipd/core.c' + dcp = 'drivers/gpu/drm/apple/dcp.c' + funcs = '\n'.join(function(f, n) for f, n in ( + ('drivers/gpu/drm/apple/iomfb.c', 'iomfb_pause_presentation'), + (dcp, 'dcp_external_report_disconnected'), + (dcp, 'dcp_external_publish_generation'), + (dcp, 'dcp_external_stop_locked'), + (dcp, 'dcp_external_disconnect'), + (dcp, 'dcp_external_stop_work'), + (dcp, 'dcp_external_hotplug'), + (dcp, 'dcp_external_link_locked'), + (dcp, 'dcp_dptx_connect_oob'), + (core, 'cd321x_retry_revalidation'), + (core, 'cd321x_update_work'))) + prefix = r''' +#include +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; typedef uint64_t u64; +#define guard(x) fake_guard +#define fake_guard(x) ((void)(x)) +#define scoped_guard(x,y) for (int once = ((void)(y), 1); once; once = 0) +#define lockdep_assert_held(x) ((void)(x)) +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x,v) ((x) = (v)) +#define container_of(p,t,m) ((t *)((char *)(p) - offsetof(t,m))) +#define to_delayed_work(w) container_of(w,struct delayed_work,work) +#define msecs_to_jiffies(x) (x) +#define IS_REACHABLE(x) 1 +#define CONFIG_DRM 1 +#define IS_ERR(x) false +#define PTR_ERR(x) (-EINVAL) +#define dev_warn(...) ((void)0) +#define dev_err(...) ((void)0) +#define CD321X_DEBOUNCE_DELAY_MS 500 +#define TPS_STATUS_PLUG_PRESENT (1U << 0) +#define TPS_STATUS_PLUG_UPSIDE_DOWN (1U << 1) +#define TPS_DATA_STATUS_USB2_CONNECTION (1U << 0) +#define TPS_DATA_STATUS_USB3_CONNECTION (1U << 1) +#define TPS_DATA_STATUS_TBT_CONNECTION (1U << 2) +#define TPS_DATA_STATUS_DP_CONNECTION (1U << 3) +#define TPS_DATA_STATUS_DP_PIN_ASSIGNMENT_MASK (3U << 4) +#define TPS_DATA_STATUS_USB_DATA_ROLE (1U << 6) +#define CD321X_DATA_STATUS_USB4_CONNECTION (1U << 7) +#define CD321X_DATA_STATUS_HPD_LEVEL (1U << 8) +#define TPS_POWER_STATUS_PWROPMODE(x) (x) +#define TPS_STATUS_TO_UPSIDE_DOWN(x) ((x)&TPS_STATUS_PLUG_UPSIDE_DOWN) +#define TPS_STATUS_TO_TYPEC_PORTROLE(x) 0 +#define TPS_STATUS_TO_TYPEC_VCONN(x) 0 +#define TPS_STATUS_TO_TYPEC_DATAROLE(x) 0 +#define TYPEC_STATE_SAFE 0 +#define TYPEC_STATE_USB 1 +#define TYPEC_THUNDERBOLT_SWITCH_OFF 0 +#define TYPEC_ACCESSORY_NONE 0 +#define DCP_HOTPLUG_NOTIFY 1 +struct work_struct { bool pending; }; +struct delayed_work { struct work_struct work; }; +static int system_freezable_wq; +static int schedule_work(struct work_struct *w) { w->pending = true; return 1; } +static void cancel_work_sync(struct work_struct *w) { w->pending = false; } +static void cancel_delayed_work_sync(struct delayed_work *w) { w->work.pending = false; } +#define cancel_delayed_work cancel_delayed_work_sync +static void mod_delayed_work(int q, struct delayed_work *w, int t) +{ (void)q; (void)t; w->work.pending = true; } +struct apple_connector { bool connected, edid_invalidation_pending; }; +struct apple_dcp { + int external_lock, hpd_mutex; + struct { int lock; bool presentation_paused; } iomfb; + struct { bool connected; void *service; } dptxport[1]; + struct apple_connector *connector; + struct delayed_work typec_reconnect_wq, placeholder_edid_wq; + struct work_struct external_stop_work; + bool external_running, external_stopped, external_power_attempted; + bool external_cable, typec_cable_connected, typec_crtc_off, pipe_enabled_h17p; + bool external_physically_absent, external_prepared, placeholder_retried; + int typec_reconnect_tries; + bool external_hotplug_pending, external_hotplug_connected, quiescing; + u64 external_generation, external_rx_generation, external_crtc_generation; + u64 external_stop_generation, typec_generation; + int external_error, mode_state; void *avep; +}; +struct platform_device { struct apple_dcp *data; }; +static struct apple_dcp *platform_get_drvdata(struct platform_device *p) { return p->data; } +static int raw_error, raw_links; +static int dcp_dptx_connect_raw(struct apple_dcp *d, u32 port, bool *waiting) +{ assert(port == 0); *waiting = false; raw_links++; + if (!raw_error) d->dptxport[0].connected = true; + return raw_error; } +static bool dcp_is_typec_output(struct apple_dcp *d) { (void)d; return true; } +static int dcp_dptx_connect(struct apple_dcp *d, u32 port) { (void)d; (void)port; return 0; } +#define DPTX_RECONNECT_DELAY 500 +static int settle_error, power_error, hpd_error, release_error; +static int settles, powers, releases, detaches, notices; +static bool dcp_is_external_h17p(struct apple_dcp *d) { (void)d; return true; } +static unsigned dcp_mode_hotplug(int *mode, bool connected, bool *sink) +{ (void)mode; if (sink) *sink = connected; return 1; } +static void dcp_handle_hotplug_actions(struct apple_dcp *d, unsigned act) +{ (void)d; notices += !!act; } +static int iomfb_queue_settle(struct apple_dcp *d, int t) +{ (void)d; (void)t; settles++; return settle_error; } +static int iomfb_external_power_h17p(struct apple_dcp *d, bool on) +{ (void)d; assert(!on); powers++; return power_error; } +static bool dptx_service_available(void *p) { (void)p; return true; } +static int dptxport_set_hpd(void *p, bool on) +{ (void)p; assert(!on); return hpd_error; } +static int dptxport_release_display(void *p) +{ (void)p; releases++; return release_error; } +static void av_service_disconnect(struct apple_dcp *d) { (void)d; } +static void dpavservep_detach(struct apple_dcp *d) { (void)d; detaches++; } +static void dcp_firmware_metadata_hotplug(struct apple_dcp *d, bool c) { (void)d; (void)c; } +static void dcp_firmware_metadata_connection(struct apple_dcp *d, u64 g, bool c) +{ (void)d; (void)g; (void)c; } +static int dcp_external_disconnect(struct apple_dcp *, bool); +static struct apple_dcp display; +static void drm_connector_oob_hotplug_event(void *node, int status) +{ assert(node); if (!status) dcp_external_disconnect(&display, true); } +#define connector_status_disconnected 0 +#define connector_status_connected 1 +''' + shim = r''' +enum usb_role { USB_ROLE_NONE, USB_ROLE_HOST, USB_ROLE_DEVICE }; +enum typec_pwr_opmode { TYPEC_PWR_MODE_USB, TYPEC_PWR_MODE_PD }; +enum typec_orientation { TYPEC_ORIENTATION_NONE, TYPEC_ORIENTATION_NORMAL, TYPEC_ORIENTATION_REVERSE }; +struct usb_pd_identity { int id; }; +struct typec_mux_state { void *alt, *data; int mode; }; +struct typec_thunderbolt_switch_data { int state; }; +struct typec_partner_desc { bool usb_pd; int accessory; struct usb_pd_identity *identity; }; +struct cd321x_status { u32 status, data_status, status_changed, data_status_changed, pwr_status; + struct usb_pd_identity partner_identity; }; +struct tps6598x { void *partner, *dev, *port, *psy; int lock; u32 data_status; enum usb_role *role_sw; }; +struct cd321x { struct tps6598x tps; struct cd321x_status update_status; + struct cd321x_pm_state pm; struct delayed_work update_work, resume_work; + struct typec_mux_state state; bool state_valid, display_route_active, cur_partner_is_pd; + struct usb_pd_identity cur_partner_identity; void *connector_fwnode, *tbt_switch; }; +static int tbt_error, role_error, safe_error, mode_error; +static int tbts, roles, safes; +static enum usb_role usb_role_switch_get_role(enum usb_role *r) { return *r; } +static int usb_role_switch_set_role(enum usb_role *r, enum usb_role n) +{ roles++; if (role_error) return role_error; *r = n; return 0; } +static int typec_thunderbolt_switch_set(void *p, void *s) +{ (void)p; (void)s; tbts++; return tbt_error; } +static int typec_set_mode(void *p, int s) +{ (void)p; assert(s == TYPEC_STATE_SAFE); safes++; return safe_error; } +static int cd321x_typec_update_mode(struct tps6598x *t, struct cd321x_status *s, bool *busy) +{ (void)t; (void)s; *busy = mode_error == -EBUSY; return mode_error; } +static void typec_unregister_partner(void *p) { (void)p; } +static void *typec_register_partner(void *p, struct typec_partner_desc *d) +{ (void)p; (void)d; return (void *)1; } +static void typec_partner_set_identity(void *p) { (void)p; } +#define typec_set_pwr_opmode(...) ((void)0) +#define typec_set_pwr_role(...) ((void)0) +#define typec_set_vconn_role(...) ((void)0) +#define typec_set_orientation(...) ((void)0) +#define typec_set_data_role(...) ((void)0) +#define power_supply_changed(...) ((void)0) +''' + tests = r''' +static int failures, checks; +#define CHECK(x) do { checks++; if (!(x)) { printf("FAIL line %d: %s\n", __LINE__, #x); failures++; } } while (0) +static struct apple_connector sink; +static enum usb_role role; +static struct cd321x port; +static void init(void) +{ + memset(&display,0,sizeof(display)); memset(&sink,0,sizeof(sink)); memset(&port,0,sizeof(port)); + sink.connected = true; display.connector = &sink; display.external_prepared = true; + raw_error = raw_links = 0; + display.external_running = display.external_power_attempted = true; + display.external_cable = display.typec_cable_connected = true; + display.dptxport[0].connected = display.pipe_enabled_h17p = true; + display.external_generation = display.external_rx_generation = display.external_crtc_generation = 7; + display.typec_generation = 4; + port.tps.partner = (void *)1; port.tps.role_sw = &role; role = USB_ROLE_HOST; + port.pm.phase = CD321X_PM_RUNNING; port.state_valid = true; port.state.mode = 2; + port.display_route_active = true; port.connector_fwnode = (void *)1; + settle_error = power_error = hpd_error = release_error = 0; + tbt_error = role_error = safe_error = mode_error = 0; + settles = powers = releases = detaches = notices = tbts = roles = safes = 0; +} +static void callback(bool connected) +{ + if (!dcp_external_hotplug(&display,connected)) + dcp_mode_hotplug(&display.mode_state,connected,&sink.connected); +} +int main(void) +{ + /* Unplug at each fallible transition with failing DCP power teardown. */ + for (int which = 0; which < 3; which++) { + init(); power_error = -ETIMEDOUT; + if (which == 0) tbt_error = -ETIMEDOUT; + if (which == 1) role_error = -EIO; + if (which == 2) safe_error = -ETIMEDOUT; + cd321x_update_work(&port.update_work.work); + CHECK(!sink.connected); CHECK(!display.external_cable); + CHECK(display.external_error == -ETIMEDOUT); + CHECK(display.dptxport[0].connected && display.pipe_enabled_h17p); + CHECK(!display.external_stopped && !display.typec_crtc_off); + CHECK(port.display_route_active && !port.state_valid); + CHECK(port.pm.phase == CD321X_PM_REVALIDATE && port.pm.attempts_left == CD321X_RESUME_ATTEMPTS); + CHECK(notices > 0 && releases == 0); + callback(true); CHECK(!sink.connected); + printf("case transition-%d complete\n",which); + } + /* A rejected/nonowner port cannot disconnect the shared owner's sink. */ + for (int which = 0; which < 3; which++) { + init(); port.display_route_active = false; + if (which == 0) tbt_error = -EIO; + if (which == 1) role_error = -EIO; + if (which == 2) safe_error = -EIO; + cd321x_update_work(&port.update_work.work); + CHECK(sink.connected && display.external_cable && !notices); + } + init(); port.display_route_active = false; role = USB_ROLE_NONE; + port.update_status.status = TPS_STATUS_PLUG_PRESENT; + port.update_status.data_status = TPS_DATA_STATUS_DP_CONNECTION | CD321X_DATA_STATUS_HPD_LEVEL; + mode_error = -EBUSY; cd321x_update_work(&port.update_work.work); + CHECK(sink.connected && !notices && !port.display_route_active); + /* Two ports share the connector: busy-port unplug leaves its owner live. */ + init(); + struct cd321x second = port; + enum usb_role second_role = USB_ROLE_NONE; + second.tps.role_sw = &second_role; second.display_route_active = false; + second.update_status.status = TPS_STATUS_PLUG_PRESENT; + second.update_status.data_status = TPS_DATA_STATUS_DP_CONNECTION | CD321X_DATA_STATUS_HPD_LEVEL; + mode_error = -EBUSY; cd321x_update_work(&second.update_work.work); + CHECK(!second.display_route_active && sink.connected && !notices); + second.update_status.status = second.update_status.data_status = 0; + tbt_error = -EIO; cd321x_update_work(&second.update_work.work); + CHECK(port.display_route_active && sink.connected && !notices); + power_error = -EIO; cd321x_update_work(&port.update_work.work); + CHECK(!sink.connected && port.display_route_active); + CHECK(display.dptxport[0].connected && display.pipe_enabled_h17p); + /* Revalidation is bounded even though the failed hardware remains owned. */ + init(); tbt_error = -EIO; + cd321x_update_work(&port.update_work.work); + for (int i = 0; i < CD321X_RESUME_ATTEMPTS; i++) { + CHECK(cd321x_pm_begin_read(&port.pm)); cd321x_pm_snapshot_ready(&port.pm); + cd321x_update_work(&port.update_work.work); + } + CHECK(port.pm.phase == CD321X_PM_STALE && !port.pm.attempts_left); + CHECK(!sink.connected && port.display_route_active); + /* Stop failure stages preserve all state whose shutdown was not acked. */ + for (int which = 0; which < 4; which++) { + init(); + if (which == 0) settle_error = -EIO; + if (which == 1) power_error = -EIO; + if (which == 2) hpd_error = -EIO; + if (which == 3) release_error = -EIO; + CHECK(dcp_external_disconnect(&display,true) == -EIO); + CHECK(!sink.connected && !display.typec_cable_connected); + CHECK(display.external_generation == 8 && display.external_rx_generation == 8); + CHECK(display.external_error == -EIO && display.dptxport[0].connected); + CHECK(!display.external_stopped && !display.typec_crtc_off); + CHECK(display.pipe_enabled_h17p == (which < 2)); + callback(true); CHECK(!sink.connected); + } + /* An existing error returns before pause: late RX still cannot resurrect. */ + init(); display.external_error = -EIO; + CHECK(dcp_external_disconnect(&display,true) == -EIO); + CHECK(!display.iomfb.presentation_paused); + callback(true); CHECK(!sink.connected); + CHECK(display.dptxport[0].connected && display.pipe_enabled_h17p); + /* A queued callback from the old sink cannot be replayed after unplug. */ + init(); display.external_error = -EIO; + display.iomfb.presentation_paused = true; callback(true); + CHECK(display.external_hotplug_pending); + CHECK(dcp_external_disconnect(&display,true) == -EIO); + iomfb_pause_presentation(&display,false); CHECK(!sink.connected); + /* An old stop generation cannot touch a later session. */ + init(); display.external_stop_generation = 6; + dcp_external_stop_work(&display.external_stop_work); + CHECK(sink.connected && settles == 0 && powers == 0 && releases == 0); + CHECK(dcp_external_stop_locked(&display,6) == 0 && sink.connected); + /* Same-cable CRTC off remains distinct from physical absence. */ + init(); CHECK(dcp_external_disconnect(&display,false) == 0); + CHECK(display.external_cable && display.typec_cable_connected); + CHECK(display.external_generation == 7); + /* Genuine discovery and the paused DPMS generation gate still work. */ + init(); sink.connected = false; callback(true); CHECK(sink.connected); + init(); sink.connected = false; display.iomfb.presentation_paused = true; + display.external_power_attempted = false; callback(true); CHECK(sink.connected); + init(); sink.connected = false; display.iomfb.presentation_paused = true; + display.external_stopped = true; display.external_rx_generation = 8; + callback(true); CHECK(sink.connected); + init(); sink.connected = false; display.iomfb.presentation_paused = true; + display.external_stopped = true; callback(true); CHECK(!sink.connected); + init(); sink.connected = false; display.iomfb.presentation_paused = true; + display.external_error = -EIO; callback(true); CHECK(!sink.connected); + /* Fresh firmware discovery precedes OOB: no second RX is required. */ + init(); sink.connected = false; + display.external_cable = display.typec_cable_connected = false; + display.external_generation = display.external_rx_generation = 37; + display.external_power_attempted = false; display.external_stopped = true; + display.dptxport[0].connected = false; + callback(true); CHECK(sink.connected); CHECK(!display.external_physically_absent); + struct platform_device device = { .data = &display }; + CHECK(dcp_dptx_connect_oob(&device,0) == 0); + CHECK(raw_links == 1 && sink.connected); + CHECK(display.external_generation == 38); + /* Initially paused discovery keeps the original pending-replay behavior. */ + init(); sink.connected = false; display.external_cable = false; + display.external_power_attempted = false; display.iomfb.presentation_paused = true; + callback(true); CHECK(display.external_hotplug_pending); CHECK(!sink.connected); + CHECK(dcp_dptx_connect_oob(&device,0) == 0 && raw_links == 1); + iomfb_pause_presentation(&display,false); CHECK(sink.connected); + /* Physical absence consumes late RX and clears an already queued replay. */ + init(); display.iomfb.presentation_paused = true; callback(true); + CHECK(display.external_hotplug_pending); + CHECK(dcp_external_disconnect(&display,true) == 0); + CHECK(display.external_physically_absent && !display.external_hotplug_pending); + callback(true); CHECK(!sink.connected && !display.external_hotplug_pending); + /* A real next OOB connect removes the tombstone and permits discovery. */ + CHECK(dcp_dptx_connect_oob(&device,0) == 0); + CHECK(!display.external_physically_absent && display.external_cable); + CHECK(display.external_generation == 9 && raw_links == 1); + callback(true); CHECK(sink.connected); + /* Same-cable DPMS does not create a physical-absence tombstone. */ + init(); CHECK(dcp_external_disconnect(&display,false) == 0); + CHECK(!display.external_physically_absent && display.external_cable); + sink.connected = false; callback(true); CHECK(!sink.connected); + CHECK(display.external_hotplug_pending); + iomfb_pause_presentation(&display,false); CHECK(sink.connected); + /* A failed hardware stop remains owned, even after a real OOB reconnect. */ + init(); display.external_error = -EIO; + CHECK(dcp_external_disconnect(&display,true) == -EIO); + CHECK(display.external_physically_absent); callback(true); CHECK(!sink.connected); + CHECK(dcp_dptx_connect_oob(&device,0) == -EIO); + CHECK(!display.external_physically_absent && display.external_error == -EIO); + CHECK(raw_links == 0 && display.dptxport[0].connected && display.pipe_enabled_h17p); + /* Successful physical teardown may release ownership normally. */ + init(); CHECK(dcp_external_disconnect(&display,true) == 0); + CHECK(!sink.connected && !display.dptxport[0].connected); + CHECK(display.external_stopped && display.typec_crtc_off && !display.pipe_enabled_h17p); + printf("%d assertions, %d failures\n", checks, failures); return !!failures; +} +''' + source = prefix + pm + shim + funcs + tests + return source + +def teardown(root): + + files = {n: (root / f).read_text() for n, f in { + 'atc': 'drivers/phy/apple/atc.c', + 'dp': 'drivers/phy/apple/atc-j700-dp.h', + 'phy': 'drivers/phy/phy-core.c', + 'core': 'drivers/usb/dwc3/core.c', + 'apple': 'drivers/usb/dwc3/dwc3-apple.c', + 'state': 'drivers/usb/dwc3/dwc3-apple-state.h', + }.items()} + + + def fn(n, name): + s = files[n] + normalized = n == 'atc' and name == 'atcphy_mux_set' and 'static int atcphy_mux_set_locked(' in s + if normalized: + name = 'atcphy_mux_set_locked' + m = re.search(r'^(?:static [^\n]*\b|(?:int|void) )' + name + r'\(', s, re.M) + assert m, (n, name) + result = block(s, m.start()) + return result.replace('atcphy_mux_set_locked(', 'atcphy_mux_set(', 1) if normalized else result + + decls = [] + for name in ('atcphy_generation', 'atcphy_lane_mode', 'atcphy_pipehandler_state', 'atcphy_mode', 'atcphy_dp_link_rate'): + decls.append(block(files['atc'], files['atc'].index('enum ' + name + ' {')) + ';') + for name in ('atcphy_mode_configuration', 'atcphy_hw', 'atcphy_dp_link_rate_configuration'): + decls.append(block(files['atc'], files['atc'].index('struct ' + name + ' {')) + ';') + decls.append(block(files['state'], files['state'].index('enum dwc3_apple_state {')) + ';') + start = files['atc'].index('static const struct {\n\tconst struct atcphy_mode_configuration normal;') + end = files['atc'].index('\n};', start) + 3 + decls.append(files['atc'][start:end]) + start = files['atc'].index('static const struct atcphy_dp_link_rate_configuration dp_lr_config[]') + end = files['atc'].index('\n};', start) + 3 + decls.append(files['atc'][start:end]) + + names = { + 'atc': ('atcphy_pipehandler_lock', 'atcphy_pipehandler_unlock', + 'atcphy_pipehandler_check', 'atcphy_pipehandler_set_mux', + 'atcphy_clear_nonselected_phy_reset', + 'atcphy_configure_pipehandler_usb3_host_t8103', + 'atcphy_configure_pipehandler_usb3_host_t8122', + 'atcphy_configure_pipehandler_usb3', 'atcphy_configure_pipehandler_usb4', + 'atcphy_enable_dummy_phy', 'atcphy_configure_pipehandler_dummy', + 'atcphy_configure_pipehandler', 'atcphy_park_pipehandler', + 'atcphy_get_mode_config', 'atcphy_dp_configure', + 'j700_field_update', 'j700_aux_disable', 'atcphy_disable_dp_aux', + 'atcphy_power_off', 'atcphy_power_on', 'atcphy_configure', 'atcphy_usb2_init', 'atcphy_usb3_power_off', + 'atcphy_usb3_set_mode', 'atcphy_usb2_power_off', + '_atcphy_dwc3_reset_assert', 'atcphy_dwc3_reset_assert', 'atcphy_mux_set'), + 'phy': ('phy_init', 'phy_exit', 'phy_power_on', 'phy_power_off'), + 'core': ('dwc3_phy_power_on', 'dwc3_phy_power_off', 'dwc3_phy_exit', 'dwc3_core_exit'), + 'apple': ('dwc3_apple_core_stop', 'dwc3_apple_core_init', 'dwc3_apple_core_start', + 'dwc3_apple_init', 'dwc3_apple_exit', 'dwc3_usb_role_switch_set'), + } + names['core'] = ('dwc3_phy_power_on', 'dwc3_phy_power_off', 'dwc3_phy_exit', + 'dwc3_core_exit_after_phy_power_off', 'dwc3_core_exit', + 'dwc3_core_remove_common', 'dwc3_core_remove', 'dwc3_core_remove_no_shutdown') + names['atc'] += ('atcphy_dwc3_reset_deassert',) + names['atc'] = tuple(name for name in names['atc'] if name not in ('atcphy_get_mode_config', 'j700_field_update')) + names = {n: tuple(name for name in group if re.search(r'^(?:static [^\n]*\b|(?:int|void) )' + name + r'\(', files[n], re.M)) + for n, group in names.items()} + context = (fn('atc', 'core_read32') + '\n' + fn('atc', 'atcphy_get_mode_config') + '\n' + fn('atc', 'j700_field_update') + '\n' + files['dp'][files['dp'].index('/* Masks are the union'):files['dp'].index('/* Retry only the idle operations')]) + functions = '\n\n'.join(fn(n, name) for n, group in names.items() for name in group) + definitions = [] + lines = iter(files['atc'].splitlines()) + for line in lines: + if not line.startswith('#define '): + continue + definition = [line] + while definition[-1].endswith('\\'): + definition.append(next(lines)) + definitions.append('\n'.join(definition)) + definitions = '\n'.join(definitions) + here = Path(__file__).resolve().parent + fallback = '' if 'dwc3_core_remove_no_shutdown' in names['core'] else '#define dwc3_core_remove_no_shutdown dwc3_core_remove\n' + unit = ((here / 'dwc3/prefix.c').read_text() + '\n' + definitions + '\n' + + '\n'.join(decls) + '\n' + (here / 'dwc3/mocks.c').read_text() + '\n' + + context + '\n' + functions + '\n' + fallback + (here / 'dwc3/cases.c').read_text()) + return unit + +def profile(root): + + base = root / "drivers/gpu/drm/apple" + source = (base / "dcp.c").read_text() + + + def block(marker): + start = source.index(marker) + opening = source.index("{", start) + end, depth = opening + 1, 1 + while depth: + depth += (source[end] == "{") - (source[end] == "}") + end += 1 + return source[start:end] + + + clock = source[source.index("static const struct resource t8140_firmware_clock"): + source.index("static const struct apple_dcp_hw_data apple_dcp_hw_t8140 =")] + profiles = clock + "\n" + "\n".join(block(marker) + ";" for marker in ( + "static const struct apple_dcp_hw_data apple_dcp_hw_t8140 =", + "static const struct apple_dcp_hw_data apple_dcp_hw_t8140_external =", + )) + admission = block("static int dcp_get_disp_regs(") + headers = "\n".join(line for name in ("dcp-external-bw.h", "dcp-link.h") + for line in (base / name).read_text().splitlines() + if not line.startswith("#include")) + prefix = r''' +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; typedef uint64_t u64; +typedef uint32_t __le32; typedef uint64_t __le64; +#define __packed __attribute__((packed)) +#undef static_assert +#define static_assert(condition, ...) _Static_assert(condition, "wire layout") +#define le32_to_cpu(v) (v) +#define cpu_to_le32(v) (v) +#define cpu_to_le64(v) (v) +#define IORESOURCE_MEM 1 +#define MAX_DISP_REGISTERS 7 +#define DCP_FIRMWARE_H17P 3 +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define DEFINE_RES_MEM(s,n) { .start=(s), .end=(s)+(n)-1 } +#define resource_size(r) ((r)->end-(r)->start+1) +struct resource { u64 start,end; }; +struct device_node { bool named; unsigned count; }; +struct device { struct device_node *of_node; }; +struct platform_device { struct device dev; int num_resources; struct resource resources[10]; }; +#define to_platform_device(d) ((struct platform_device *)(d)) +struct apple_dcp_hw_data { + unsigned num_dptx_ports; bool adopt_live_session; + int iomfb_method_profile,firmware_compat; + const struct resource *firmware_clock; + u32 firmware_display_windows,firmware_scratch,firmware_request; +}; +struct apple_dcp { + struct device *dev; struct apple_dcp_hw_data hw; + struct resource *disp_registers[10],disp_bw_scratch_res,disp_bw_doorbell_res; + u32 disp_bw_scratch_index,disp_bw_scratch_offset,disp_bw_doorbell_index,nr_disp_registers; +}; +static bool of_property_present(struct device_node *n,const char *name) +{ return !strcmp(name,"reg-names") && n->named; } +static int of_property_count_strings(struct device_node *n,const char *name) +{ assert(!strcmp(name,"reg-names"));return n->count+1; } +static int of_property_read_string_index(struct device_node *n,const char *name,int i,const char **v) +{ + static const char *names[]={"coproc","disp-0","disp-1","disp-2","disp-3","disp-4","disp-5"}; + assert(!strcmp(name,"reg-names"));if(i<0 || (unsigned)i>n->count)return -EINVAL; + *v=names[i];return 0; +} +static struct resource *platform_get_resource(struct platform_device *p,int type,int i) +{ assert(type==IORESOURCE_MEM);return i>=1 && inum_resources?&p->resources[i-1]:NULL; } +static struct resource *platform_get_resource_byname(struct platform_device *p,int type,const char *name) +{ int i=-1;assert(sscanf(name,"disp-%d",&i)==1);return platform_get_resource(p,type,i+1); } +static int dcp_get_bw_scratch_reg(struct apple_dcp *p,u32 count) { (void)p;(void)count;return -ENOENT; } +static int dcp_get_bw_doorbell_reg(struct apple_dcp *p,u32 count) { (void)p;(void)count;return -ENOENT; } +''' + tests = r''' +static struct platform_device provider; +static struct device_node node; +static struct apple_dcp d; +static void reset(bool external,bool named) +{ + node=(struct device_node){.named=named,.count=5}; + provider=(struct platform_device){.dev.of_node=&node,.num_resources=6}; + for(unsigned i=0;i<5;i++)provider.resources[i]=(struct resource)DEFINE_RES_MEM(0x1000ULL*i,0x1000); + provider.resources[4]=t8140_firmware_clock; + d=(struct apple_dcp){.dev=&provider.dev,.hw=external?apple_dcp_hw_t8140_external:apple_dcp_hw_t8140}; +} +int main(void) +{ + const u32 fw[]={26,6,2},unknown[]={0,0,0}; + assert(apple_dcp_external_bw_valid(0xbc000,0x28000,0x70000)); + assert(!apple_dcp_external_bw_valid(0xbc000,0x28002,0x70000)); + struct apple_dcp_h17p_clock_request clock={.clock_id=APPLE_DCP_H17P_CLOCK_ID_154};u64 rate; + assert(!apple_dcp_h17p_clock_rate(&clock,123,456,&rate) && rate==123); + assert(apple_dcp_external_firmware_supported(fw,3,fw,3)); + assert(!apple_dcp_external_firmware_supported(NULL,3,fw,3)); + assert(!apple_dcp_external_firmware_supported(fw,3,NULL,3)); + assert(!apple_dcp_external_firmware_supported(unknown,3,fw,3)); + assert(apple_dcp_hw_t8140.num_dptx_ports==0 && apple_dcp_hw_t8140.adopt_live_session); + assert(apple_dcp_hw_t8140.firmware_display_windows==5); + assert(apple_dcp_hw_t8140.firmware_scratch==0x20000 && apple_dcp_hw_t8140.firmware_request==0x68000); + assert(apple_dcp_hw_t8140_external.num_dptx_ports==1 && !apple_dcp_hw_t8140_external.adopt_live_session); + assert(apple_dcp_hw_t8140_external.firmware_display_windows==4); + assert(apple_dcp_hw_t8140_external.firmware_scratch==0x28000 && apple_dcp_hw_t8140_external.firmware_request==0x70000); + for(unsigned named=0;named<2;named++) { + reset(true,named);assert(dcp_get_disp_regs(&d)==0); + assert(d.nr_disp_registers==5 && d.disp_bw_scratch_index==4); + assert(d.disp_bw_scratch_res.start+d.disp_bw_scratch_offset==0x302828000ULL); + assert(d.disp_bw_doorbell_res.start==0x302870000ULL && resource_size(&d.disp_bw_doorbell_res)==4); + struct apple_dcp_h17p_rt_bw_request request={0};struct apple_dcp_h17p_rt_bw_reply reply; + assert(!apple_dcp_h17p_rt_bw_encode_reply(&request,&reply,d.disp_bw_scratch_res.start+d.disp_bw_scratch_offset,d.disp_bw_doorbell_res.start)); + assert(reply.config.scratch==0x302828000ULL && reply.config.clock_request==0x302870000ULL && !reply.status); + request.config_null=1;assert(apple_dcp_h17p_rt_bw_encode_reply(&request,&reply,0,0)==-EINVAL); + reset(false,named);assert(dcp_get_disp_regs(&d)==0 && d.nr_disp_registers==6); + assert(d.disp_bw_scratch_offset==0x20000 && d.disp_bw_doorbell_res.start==0x302868000ULL); + reset(true,named);provider.resources[4].start+=4;assert(dcp_get_disp_regs(&d)==-EINVAL); + reset(true,named);provider.resources[4].end-=4;assert(dcp_get_disp_regs(&d)==-EINVAL); + reset(true,named);node.count=4;provider.num_resources=5;assert(dcp_get_disp_regs(&d)==-EINVAL); + reset(true,named);d.hw.firmware_scratch=0xbc000;assert(dcp_get_disp_regs(&d)==-EINVAL); + reset(true,named);d.hw.firmware_request=0xbc000;assert(dcp_get_disp_regs(&d)==-EINVAL); + reset(true,named);d.hw.firmware_display_windows=0;assert(dcp_get_disp_regs(&d)==-EINVAL); + } + puts("PASS: actual internal/external profiles, resource admission and D003 bandwidth reply"); +} +''' + return prefix + headers + "\n" + profiles + admission + tests + +def firmware(root): + + source = (root / "drivers/gpu/drm/apple/iomfb_h17p.c").read_text() + body = source[source.index("struct dcp_h17p_provider_request {"): + source.index("static bool trampoline_luma_span_h17p")] + prefix = r''' +#include +#include +#include +#include +#include +#include +#include +typedef uint8_t u8; +typedef uint32_t u32; +typedef uint32_t __le32; +#define __packed __attribute__((packed)) +#undef static_assert +#define static_assert(cond) _Static_assert(cond, "") +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define sizeof_field(t, f) sizeof(((t *)0)->f) +#define le32_to_cpu(x) (x) +#define cpu_to_le32(x) (x) +#define WRITE_ONCE(x, v) ((x) = (v)) +#define DCP_IOMFB_METHODS_H17G 1 +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define trace_iomfb_callback(d, t, n) ((void)(d), (void)(t)) +#define dev_err(...) ((void)0) +#define dev_warn_once(...) ((void)0) +struct device_node { bool internal, external, present; int length; }; +struct device { struct device_node *of_node; }; +struct apple_dcp { struct { int iomfb_method_profile; } hw; struct device *dev; bool crashed; }; +static bool j700 = true; +static bool of_machine_is_compatible(const char *name) +{ + assert(!strcmp(name, "apple,j700")); + return j700; +} +static bool of_device_is_compatible(struct device_node *n, const char *name) +{ + return (!strcmp(name, "apple,t8140-dcp") && n->internal) || + (!strcmp(name, "apple,t8140-dcpext") && n->external); +} +static bool of_property_present(struct device_node *n, const char *name) +{ + (void)name; + return n->present; +} +static int of_property_count_u8_elems(struct device_node *n, const char *name) +{ + (void)name; + return n->present ? n->length : -EINVAL; +} +static int of_property_read_u8_array(struct device_node *n, const char *name, void *p, unsigned int len) +{ + (void)n; (void)name; + memset(p, 0x5a, len); + return 0; +} +static const void *memchr_inv(const void *p, int value, size_t size) +{ + const u8 *b = p; + for (size_t i = 0; i < size; i++) + if (b[i] != value) + return b + i; + return NULL; +} +''' + suffix = r''' +int main(void) +{ + struct device_node node = { .external = true }; + struct device dev = { .of_node = &node }; + struct apple_dcp dcp = { .dev = &dev, .hw.iomfb_method_profile = 2 }; + struct dcp_h17p_provider_request req = { .scope = { 'V', 'O', 'R', 'P' }, .capacity = 0xc00 }; + struct dcp_h17p_provider_reply reply; + unsigned int cases = 0; + for (unsigned int i = 0; i < ARRAY_SIZE(provider_properties); i++) { + memset(&reply, 0xff, sizeof(reply)); + strcpy(req.name, provider_properties[i].name); + assert(trampoline_provider_property_h17p(&dcp, 400, &reply, &req)); + assert(!dcp.crashed && reply.length == provider_properties[i].length); + if (reply.length) + assert(!memcmp(reply.data, provider_properties[i].legacy_data, reply.length)); + cases++; + } + strcpy(req.name, "power-lut-data-x"); + j700 = false; + assert(!trampoline_provider_property_h17p(&dcp, 400, &reply, &req) && dcp.crashed); + j700 = true; dcp.crashed = false; dcp.hw.iomfb_method_profile = 3; + assert(!trampoline_provider_property_h17p(&dcp, 400, &reply, &req) && dcp.crashed); + node.external = false; node.internal = true; dcp.crashed = false; + assert(trampoline_provider_property_h17p(&dcp, 400, &reply, &req) && !dcp.crashed); + node.internal = false; node.external = true; node.present = true; node.length = 3; + dcp.hw.iomfb_method_profile = 2; dcp.crashed = false; + assert(!trampoline_provider_property_h17p(&dcp, 400, &reply, &req) && dcp.crashed); + node.length = 4; dcp.crashed = false; + assert(trampoline_provider_property_h17p(&dcp, 400, &reply, &req) && !dcp.crashed); + assert(reply.length == 4 && reply.data[0] == 0x5a); + strcpy(req.name, "unqualified-property"); + assert(!trampoline_provider_property_h17p(&dcp, 400, &reply, &req)); + printf("PASS %u provider payloads and scope/length rejection cases\n", cases); + return 0; +} +''' + return prefix + body + suffix + +def readiness(root): + + dcp = (root / 'drivers/gpu/drm/apple/dcp.c').read_text() + drm = (root / 'drivers/gpu/drm/apple/apple_drv.c').read_text() + + + def function(source, marker): + start = source.index(marker) + opening = source.index('{', start) + end, depth = opening + 1, 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + return source[start:end] + + + # This slice is the production probe's publication boundary, including its + # failure exits and prepared-before-component_add ordering, not a copied model. + probe = function(dcp, 'static int dcp_platform_probe(') + publish = probe[probe.rindex('\tif (dcp->hw.iomfb_method_profile ==\n\t DCP_IOMFB_METHODS_H17P_EXTERNAL) {'):] + production = '\n'.join(( + function(dcp, 'bool dcp_h17p_external_supported('), + function(dcp, 'bool dcp_h17p_external_ready('), + function(drm, 'static bool apple_dcp_usable('), + function(drm, 'static int add_dcp_components('), + function(drm, 'static int apple_drm_init_dcp('), + 'static int publish_provider(struct platform_device *pdev, struct apple_dcp *dcp)\n' + '{\n\tstruct device *dev = &pdev->dev;\n\tint ret;\n' + publish, + )) + prefix = r''' +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +typedef uint64_t u64; +#define READ_ONCE(x) (x) +#define WRITE_ONCE(x, v) ((x) = (v)) +#define IS_ENABLED(x) 0 +#define MAX_COPROCESSORS 4 +#define DL_FLAG_AUTOREMOVE_SUPPLIER 0 +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define dev_warn(...) ((void)0) +#define msleep(x) ((void)(x)) +#define component_compare_of 0 +#define apple_dcp_id_tbl 0 +static int dcp_comp_ops; +struct device_node { bool available, external, phys; unsigned int index; u32 firmware[3], compat[3]; int firmware_cells, compat_cells; }; +struct device { struct device_node *of_node; void *data; }; +struct platform_device { struct device dev; bool bound, registered; }; +struct apple_dcp { + bool external_prepared, quiescing, crashed; + struct { int iomfb_method_profile; } hw; +}; +struct drm_device { int unused; }; +struct drm_encoder { int unused; }; +struct apple_drm_private { struct drm_device drm; }; +struct component_match { bool includes[2]; }; +static struct device_node nodes[2]; +static struct platform_device devices[2]; +static struct apple_dcp dcps[2]; +static struct apple_drm_private priv; +static struct device master = { .data = &priv }; +static struct component_match matches; +static bool machine_supported, absent_external; +static unsigned int references, released, node_puts, probes[2], bind_attempts, disables; +static int preflight_error, external_probe_error, bind_result; +static void try_bind(void); +static bool of_machine_is_compatible(const char *s) +{ assert(!strcmp(s,"apple,j700")); return machine_supported; } +static int of_property_read_variable_u32_array(struct device_node *n,const char *s,u32 *v,int min,int max) +{ + assert(min==3 && max==3); + bool fw=!strcmp(s,"apple,firmware-version"); + assert(fw || !strcmp(s,"apple,firmware-compat")); + int count=fw?n->firmware_cells:n->compat_cells; + if(count!=3) return -EINVAL; + memcpy(v,fw?n->firmware:n->compat,3*sizeof(*v)); return count; +} +static bool of_device_is_available(struct device_node *n) { return n->available; } +static bool of_device_is_compatible(struct device_node *n, const char *s) +{ + return n->external && (!strcmp(s, "apple,t8140-dcpext") || !strcmp(s, "apple,dcpext")); +} +static bool of_property_present(struct device_node *n, const char *s) +{ assert(!strcmp(s, "phys")); return n->phys; } +#define for_each_matching_node(np, table) \ + for (unsigned int node_i = 0; node_i < 2 && ((np) = &nodes[node_i]); node_i++) +#define for_each_endpoint_of_node(np, endpoint) \ + for ((endpoint) = NULL; (endpoint) != NULL; (endpoint) = NULL) +static struct device_node *of_graph_get_remote_port_parent(struct device_node *n) +{ return n; } +static void of_node_put(struct device_node *n) { (void)n; node_puts++; } +static struct platform_device *of_find_device_by_node(struct device_node *n) +{ + if (n->external && absent_external) + return NULL; + references++; + return &devices[n->index]; +} +static void *platform_get_drvdata(struct platform_device *p) { return p->dev.data; } +static void *dev_get_drvdata(struct device *d) { return d->data; } +static void put_device(struct device *d) { (void)d; released++; } +/* Deliberately provided: using this during synchronous bind is a regression. */ +#define device_is_bound(dev) (devices[(dev)->of_node->index].bound) +static void drm_of_component_match_add(struct device *d, struct component_match **m, + int compare, struct device_node *n) +{ (void)d; (void)compare; *m = &matches; matches.includes[n->index] = true; } +static void device_link_add(struct device *a, struct device *b, int flags) +{ (void)a; (void)b; (void)flags; } +static int apple_probe_per_dcp(struct device *d, struct drm_device *drm, + struct platform_device *p, int index, bool external, + struct drm_encoder **encoder, u32 *mask) +{ + (void)d; (void)drm; (void)index; (void)encoder; (void)mask; + assert(external == p->dev.of_node->external); + probes[p->dev.of_node->index]++; + return external ? external_probe_error : 0; +} +static int apple_probe_typec_ports(struct drm_device *d, struct platform_device **p, + u32 *masks, int count) +{ (void)d; (void)p; (void)masks; (void)count; return 0; } +static int dcp_start(struct platform_device *p) { (void)p; return 0; } +static int dcp_wait_ready(struct platform_device *p, u64 timeout) +{ (void)p; (void)timeout; return 0; } +static u64 get_jiffies_64(void) { return 0; } +static u64 msecs_to_jiffies(unsigned int ms) { return ms; } +static bool time_after_eq64(u64 a, u64 b) { return a >= b; } +static int dcp_comp_bind(struct device *d, void *master_data, void *data) +{ (void)d; (void)master_data; (void)data; return preflight_error; } +static void dcp_disable_typec_work(struct apple_dcp *d, bool quiesce) +{ disables++; d->quiescing = quiesce; } +static int component_add(struct device *d, const void *ops) +{ + (void)ops; + struct platform_device *p = &devices[d->of_node->index]; + assert(!p->bound); + p->registered = true; + try_bind(); /* Linux component_add may bind the master before probe returns. */ + return 0; +} +''' + suffix = r''' +static void try_bind(void) +{ + for (unsigned int i = 0; i < 2; i++) + if (matches.includes[i] && !devices[i].registered) + return; + bind_attempts++; + bind_result = apple_drm_init_dcp(&master); +} +static void initial(bool supported_firmware) +{ + memset(nodes, 0, sizeof(nodes)); + memset(devices, 0, sizeof(devices)); + memset(dcps, 0, sizeof(dcps)); + memset(&matches, 0, sizeof(matches)); + memset(probes, 0, sizeof(probes)); + for (unsigned int i = 0; i < 2; i++) { + nodes[i] = (struct device_node){ .available = true, .external = i == 1, .index = i }; + devices[i].dev = (struct device){ .of_node = &nodes[i], .data = &dcps[i] }; + } + dcps[1].hw.iomfb_method_profile = DCP_IOMFB_METHODS_H17P_EXTERNAL; + devices[0].registered = devices[0].bound = true; + machine_supported = true; + nodes[1].firmware_cells=nodes[1].compat_cells=3; + if(supported_firmware) { + u32 version[]={26,6,2}; + memcpy(nodes[1].firmware,version,sizeof(version)); + memcpy(nodes[1].compat,version,sizeof(version)); + } + absent_external = false; + references = released = node_puts = bind_attempts = disables = 0; + preflight_error = external_probe_error = bind_result = 0; +} +static int match_providers(void) +{ + struct component_match *match = NULL; + return add_dcp_components(&master, &match); +} +int main(void) +{ + assert(apple_dcp_external_bw_valid(0xbc000,0x28000,0x70000)); + assert(apple_dcp_external_clock_window(0x302800000ULL,0xbc000,0x302800000ULL,0xbc000)); + /* No command line state: exact J700 firmware selects the external profile. */ + initial(true); assert(dcp_h17p_external_supported(&nodes[1])); + machine_supported=false; + assert(!dcp_h17p_external_supported(&nodes[1]) && match_providers()==1); + for(unsigned i=0;i<3;i++) { + initial(true); nodes[1].firmware[i]++; + assert(!dcp_h17p_external_supported(&nodes[1]) && match_providers()==1); + initial(true); nodes[1].compat[i]++; + assert(!dcp_h17p_external_supported(&nodes[1]) && match_providers()==1); + } + for(int count=-1;count<=5;count++) if(count!=3) { + initial(true); nodes[1].firmware_cells=count; + assert(!dcp_h17p_external_supported(&nodes[1])); + initial(true); nodes[1].compat_cells=count; + assert(!dcp_h17p_external_supported(&nodes[1])); + } + + /* Unknown external firmware preserves the normal internal provider. */ + initial(false); + assert(match_providers() == 1 && matches.includes[0] && !matches.includes[1]); + assert(!dcp_h17p_external_ready(&nodes[1]) && references == 0); + try_bind(); + assert(bind_attempts == 1 && bind_result == 0 && probes[0] == 1 && !probes[1]); + + /* Delayed external provider remains in the match; the master must wait. */ + initial(true); + assert(match_providers() == 2 && matches.includes[1]); + try_bind(); + assert(bind_attempts == 0 && !probes[0] && !probes[1]); + /* Last provider is prepared, but driver core bound is still false here. */ + assert(publish_provider(&devices[1], &dcps[1]) == 0); + assert(dcps[1].external_prepared && !devices[1].bound); + assert(bind_attempts == 1 && bind_result == 0 && probes[0] == 1 && probes[1] == 1); + devices[1].bound = true; + + /* Permanent preflight failure cannot silently drop a requested provider. */ + initial(true); + assert(match_providers() == 2); + preflight_error = -EIO; + assert(publish_provider(&devices[1], &dcps[1]) == -EIO); + assert(!devices[1].registered && !dcps[1].external_prepared && disables == 1); + try_bind(); + assert(matches.includes[1] && bind_attempts == 0 && !probes[0]); + + /* An unavailable DT node is excluded; a matched but absent device is fatal. */ + initial(true); + nodes[1].available = false; + assert(!apple_dcp_usable(&nodes[1]) && match_providers() == 1); + initial(true); + absent_external = true; + assert(match_providers() == 2); + assert(apple_drm_init_dcp(&master) == -ENODEV && !probes[1]); + + /* Readiness rejects every unavailable provider state and releases lookup refs. */ + for (unsigned int state = 0; state < 4; state++) { + initial(true); + dcps[1].external_prepared = true; + if (state == 0) devices[1].dev.data = NULL; + if (state == 1) dcps[1].external_prepared = false; + if (state == 2) dcps[1].quiescing = true; + if (state == 3) dcps[1].crashed = true; + assert(!dcp_h17p_external_ready(&nodes[1])); + assert(references == 1 && released == 1); + assert(apple_drm_init_dcp(&master) == -ENODEV && !probes[1]); + } + initial(true); + dcps[1].external_prepared = true; + external_probe_error = -EIO; + assert(apple_drm_init_dcp(&master) == -EIO); + assert(probes[1] == 1 && node_puts == 1); + puts("PASS: automatic hardware/firmware matching, delayed/preflight-failed providers, synchronous publication, readiness rejection"); + return 0; +} +''' + + header = (root / 'drivers/gpu/drm/apple/dcp-external-bw.h').read_text().replace('#include ', '') + return prefix + header + production + suffix + +def vrr(root): + + source = (root / 'drivers/gpu/drm/apple/dcp.c').read_text() + start = source.index('int dcp_crtc_atomic_check(') + opening = source.index('{', start) + end, depth = opening + 1, 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + check = source[start:end] + prefix = r""" +#undef NDEBUG +#include +#include +#include +#include +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define READ_ONCE(x) (x) +#define dev_dbg(...) ((void)0) +struct apple_connector { bool connected; }; +struct apple_dcp { + bool crashed; + struct { int iomfb_method_profile; } hw; + struct { bool valid; } mode_state; + struct apple_connector *connector; + void *dev; +}; +struct platform_device { struct apple_dcp *data; }; +struct drm_crtc { struct platform_device *dcp; }; +struct drm_crtc_state { bool vrr_enabled, modeset; }; +struct drm_atomic_state { struct drm_crtc_state crtc; }; +#define to_apple_crtc(x) (x) +#define platform_get_drvdata(x) ((x)->data) +static struct drm_crtc_state *drm_atomic_get_new_crtc_state( + struct drm_atomic_state *s, struct drm_crtc *c) +{ (void)c; return &s->crtc; } +static bool drm_atomic_crtc_needs_modeset(struct drm_crtc_state *s) +{ return s->modeset; } +""" + suffix = r""" +int main(void) +{ + struct apple_connector connector = { .connected = true }; + struct apple_dcp dcp = { .connector = &connector }; + struct platform_device pdev = { .data = &dcp }; + struct drm_crtc crtc = { .dcp = &pdev }; + struct drm_atomic_state state = {0}; + unsigned int cases = 0; + /* Check initial modesets, plane-only commits and link recovery states. */ + for (unsigned int profile = 0; profile < 3; profile++) + for (unsigned int valid = 0; valid < 2; valid++) + for (unsigned int modeset = 0; modeset < 2; modeset++) + for (unsigned int link = 0; link < 3; link++) + for (unsigned int vrr = 0; vrr < 2; vrr++) { + dcp.hw.iomfb_method_profile = profile; + dcp.mode_state.valid = valid; + state.crtc.modeset = modeset; + state.crtc.vrr_enabled = vrr; + dcp.connector = link == 2 ? NULL : &connector; + connector.connected = link == 1; + int expected = profile == DCP_IOMFB_METHODS_H17P_EXTERNAL && vrr ? -EINVAL : 0; + int actual = dcp_crtc_atomic_check(&crtc, &state); + if (actual != expected) { + fprintf(stderr, "profile=%u valid=%u modeset=%u link=%u vrr=%u: got %d expected %d\n", + profile, valid, modeset, link, vrr, actual, expected); + return 1; + } + cases++; + } + dcp.crashed = true; + state.crtc.vrr_enabled = false; + assert(dcp_crtc_atomic_check(&crtc, &state) == -EINVAL); + printf("PASS: %u production atomic VRR admission cases and crashed-device rejection\n", cases); + return 0; +} +""" + return prefix + check + suffix + + +def power_reply(root): + """Actual external power callback validation with host completion/refcount mocks.""" + source = (root / 'drivers/gpu/drm/apple/iomfb_h17p.c').read_text() + production = block(source, 'struct dcp_external_power_cookie {') + ';\n' + production += block(source, 'static void iomfb_external_power_complete(') + production += block(source, 'static bool iomfb_external_reply_size(') + prefix = r""" +#include +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; typedef uint64_t u64; +#define WRITE_ONCE(p,v) ((p)=(v)) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +struct completion { bool done; }; +struct kref { unsigned count; }; +struct dcp_wait_cookie { struct kref refcount; struct completion done; u32 status; }; +struct dcp_packet_header { char tag[4]; u32 in_len,out_len; }; +struct dcp_set_power_state_req_h17p { u64 unklong; u32 unkint; unsigned char null,padding[3]; } __attribute__((packed)); +struct dcp_set_power_state_resp { u32 unknown,ret; } __attribute__((packed)); +struct apple_dcp { int unused; }; +static unsigned released; +static u64 get_unaligned_le64(const void *p) { u64 v; memcpy(&v,p,sizeof(v)); return v; } +static void complete(struct completion *c) { assert(!c->done); c->done=true; } +static void release_wait_cookie(struct kref *r) { assert(!r->count); released++; } +static void kref_put(struct kref *r,void (*release)(struct kref *)) +{ assert(r->count); if (!--r->count) release(r); } +""" + controls = r""" +int main(void) +{ + struct apple_dcp d={}; + struct { struct dcp_packet_header h; struct dcp_set_power_state_req_h17p q; unsigned char reply[8]; } packet={}; + for(unsigned on=0;on<2;on++) { + const u64 replies[]={0,1,0x123400000000ULL}; + for(unsigned i=0;i<3;i++) { + packet.h.in_len=sizeof(packet.q);packet.h.out_len=sizeof(packet.reply);memcpy(packet.reply,&replies[i],sizeof(packet.reply)); + struct dcp_external_power_cookie c={.on=on,.wait.refcount.count=2}; + assert(iomfb_external_reply_size(&packet.reply,sizeof(packet.q),sizeof(packet.reply))); + iomfb_external_power_complete(&d,&packet.reply,&c); + assert(c.wait.done.done && c.wait.refcount.count==1); + assert((int)c.wait.status==(replies[i]==on?0:-EREMOTEIO)); + kref_put(&c.wait.refcount,release_wait_cookie); + } + } + for(unsigned fault=0;fault<3;fault++) { + packet.h.in_len=sizeof(packet.q);packet.h.out_len=sizeof(packet.reply); + void *out=&packet.reply; + if(fault==0) out=NULL; + if(fault==1) packet.h.in_len--; + if(fault==2) packet.h.out_len--; + struct dcp_external_power_cookie c={.wait.refcount.count=2}; + assert(!iomfb_external_reply_size(out,sizeof(packet.q),sizeof(packet.reply))); + iomfb_external_power_complete(&d,out,&c); + assert(c.wait.done.done && c.wait.refcount.count==1 && (int)c.wait.status==-EPROTO); + kref_put(&c.wait.refcount,release_wait_cookie); + } + assert(released==9); + puts("PASS: actual exact ON/OFF echoes, nonzero status and null/truncated reply rejection preserve callback completion and references"); +} +""" + return prefix + production + controls + + +def dispatcher(root): + source = (root / "drivers/gpu/drm/apple/iomfb.c").read_text() + start = source.index("static bool dcp_external_zero_reply(") + end = source.index("static void dcpep_handle_ack(", start) + prefix = r""" +#include +#include +#include +#include +#include +typedef uint8_t u8; +typedef uint16_t u16; +typedef uint32_t u32; +typedef uint64_t u64; +#define DCP_FIRMWARE_H17P 1 +#define DCP_IOMFB_METHODS_H17P_EXTERNAL 2 +#define DCP_IOMFB_METHODS_H17G 3 +#define DCP_MAX_CALL_DEPTH 8 +#define IOMFB_MAX_CB 600 +#define DCP_PACKET_ALIGNMENT 64 +#define ALIGN(n, a) (((n) + (a) - 1) & ~((a) - 1)) +#define dev_err(...) ((void)dev) +#define dev_info(...) ((void)dev) +#define dev_warn(...) ((void)dev) +enum dcp_context_id { DCP_CONTEXT_CB = 0, DCP_CONTEXT_ASYNC = 3 }; +struct device_node { bool external; }; +struct device { struct device_node *of_node; }; +struct dcp_packet_header { char tag[4]; u32 in_len, out_len; }; +struct dcp_channel { unsigned depth; void *output[8]; unsigned end[8]; }; +struct apple_dcp { + struct { int iomfb_method_profile; } hw; + struct device *dev; + int fw_compat; + bool crashed; + bool (**cb_handlers)(struct apple_dcp *, int, void *, void *); + unsigned long unknown_callbacks[16]; + int start_done; + struct dcp_channel channel; +}; +static bool machine; +static unsigned acks, case_tag; +static struct dcp_channel *dcp_get_channel(struct apple_dcp *d, enum dcp_context_id c) +{ (void)c; return &d->channel; } +static int dcp_parse_tag(char *p) { (void)p; return case_tag; } +static bool of_machine_is_compatible(const char *n) { (void)n; return machine; } +static bool of_device_is_compatible(struct device_node *n, const char *s) +{ (void)s; return n->external; } +static bool iomfb_check_callback_h17p(struct apple_dcp *d, int t, u32 i, u32 o) +{ (void)d; (void)t; (void)i; (void)o; return true; } +static bool iomfb_uses_queue(struct apple_dcp *d) { (void)d; return false; } +static bool test_and_set_bit(int t, unsigned long *b) { (void)t; (void)b; return false; } +static void complete(int *p) { *p = 1; } +static u8 dcp_push_depth(unsigned *d) { return (*d)++; } +static void dcp_ack(struct apple_dcp *d, enum dcp_context_id c) +{ (void)c; acks++; d->channel.depth--; } +""" + suffix = r""" +int main(void) +{ + struct device_node node; + struct device dev = { &node }; + bool (*handlers[IOMFB_MAX_CB])(struct apple_dcp *, int, void *, void *) = {}; + unsigned cases = 0; + for (unsigned scope = 0; scope < 8; scope++) + for (unsigned tag = 133; tag <= 135; tag++) + for (unsigned context = 0; context < 8; context++) + for (unsigned in = 0; in <= 8; in++) + for (unsigned out = 0; out <= 8; out++) { + struct apple_dcp d = { + .dev = &dev, .fw_compat = 1, + .cb_handlers = handlers, .hw.iomfb_method_profile = 2, + }; + union { uint64_t align; unsigned char data[64]; } packet; + struct dcp_packet_header *h = (void *)packet.data; + memset(&packet, 0xa5, sizeof(packet)); + h->in_len = in; h->out_len = out; + machine = scope & 1; node.external = scope & 2; + if (!(scope & 4)) + d.hw.iomfb_method_profile = 0; + case_tag = tag; acks = 0; + dcpep_handle_cb(&d, context, h, sizeof(*h) + in + out, 0); + bool accepted = scope == 7 && tag == 134 && context == 3 && !in && out == 4; + if (scope & 4) { + assert(acks == accepted && d.crashed != accepted); + for (unsigned i = 0; i < out; i++) + assert(packet.data[sizeof(*h) + in + i] == (accepted ? 0 : 0xa5)); + } else { + assert(acks == 1 && !d.crashed); /* Existing legacy behavior. */ + } + assert(packet.data[sizeof(*h) + in + out] == 0xa5); + cases++; + } + return cases != 15552; +} +""" + + return prefix + source[start:end] + suffix diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/phy.py b/tools/testing/selftests/drivers/gpu/apple-dcp/phy.py new file mode 100644 index 00000000000000..fbb1951c0f0b1c --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/phy.py @@ -0,0 +1,2399 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Actual PHY parking, failure/retry, masked context and OFF controls.""" +from pathlib import Path +import re +from common import block, source_defines + +def parking(root): + + atc = (root / 'drivers/phy/apple/atc.c').read_text() + dp = (root / 'drivers/phy/apple/atc-j700-dp.h').read_text() + + + def function(source, name): + if name == 'atcphy_mux_set' and 'static int atcphy_mux_set_locked(' in source: + marker = re.search(r'^static [^\n]+\batcphy_mux_set_locked\(', source, re.M).group() + return block(source, marker).replace('atcphy_mux_set_locked(', 'atcphy_mux_set(', 1) + return block(source, re.search(r'^static [^\n]+\b' + name + r'\(', source, re.M).group()) + + + # Use the production mode table and wrappers too, so SAFE mapping and fast paths + # are tested rather than mirrored by a second implementation. + parts = source_defines(atc) + for name in ('atcphy_generation', 'atcphy_lane_mode', 'atcphy_pipehandler_state', 'atcphy_mode', + 'atcphy_dp_link_rate'): + parts.append(block(atc, 'enum ' + name + ' {') + ';') + for name in ('atcphy_mode_configuration', 'atcphy_hw'): + parts.append(block(atc, 'struct ' + name + ' {') + ';') + parts.append(r''' +struct apple_atcphy { + const struct atcphy_hw *hw; + struct { unsigned char *core, *lpdptx, *pipehandler, *usb2phy; } regs; + struct reset_controller_dev rcdev; + void *dev; + int lock, dp_link_rate; + bool fixed_usb2, host_active, swap_lanes, ss_tunables, dp_tunables; + bool aux_power_requested, mode_dirty, dp_spread_spectrum, dp_presets_valid[5]; + u32 dp_presets[5][16]; + unsigned int dp_idle_pmas, dp_owned_pmas; + unsigned int dp_usb_context_valid, dp_usb_context_dirty; + u32 dp_usb_context[2][27]; + bool pipe_pending, usb_off_pending, usb_off_drained; + enum atcphy_mode mode, typec_mode; + enum atcphy_pipehandler_state pipe_state; +}; +''') + start = atc.index('static const struct {\n\tconst struct atcphy_mode_configuration normal;') + parts.append(atc[start:atc.index('\n};', start) + 3]) + parts.append(r''' +struct phy_configure_opts_dp { + unsigned int link_rate, lanes, voltage[4], pre[4]; + bool set_rate, set_lanes, set_voltages, ssc; +}; +static u32 registers[0x30000 / 4]; +static u32 pipe_regs[0x1000 / 4], usb2_regs[0x1000 / 4]; +static unsigned int polls, fail_poll, writes, usb2_on, usb2_writes, reset_writes, mux_writes; +static bool stuck_ack; +static bool hold_unlock_ack; +static const char *poll_caller[256], *fail_caller; +static bool bist_status_stuck; +static unsigned int bist_faults; +static bool require_parked, expected_j700_swap; +static u32 readl(void *address) { return *(u32 *)address; } +static void core_mask32(struct apple_atcphy *p, u32 offset, u32 mask, u32 value) +{ + (void)p; + assert(offset < sizeof(registers) && offset % 4 == 0); + if (require_parked) { + assert(p->pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert(FIELD_GET(p->hw->pipe_mux_data, pipe_regs[PIPEHANDLER_MUX_CTRL / 4]) == + PIPEHANDLER_MUX_CTRL_DATA_DUMMY); + assert(FIELD_GET(p->hw->pipe_mux_clk, pipe_regs[PIPEHANDLER_MUX_CTRL / 4]) == + PIPEHANDLER_MUX_CTRL_CLK_DUMMY); + } + registers[offset / 4] = (registers[offset / 4] & ~mask) | value; + writes++; + if (offset == ATCPHY_POWER_CTRL) + registers[ATCPHY_POWER_STAT / 4] = registers[offset / 4]; + if (offset == 0x2000) { + if (value == 0x10000001) { + registers[offset / 4] |= BIT(1); + registers[0x7034 / 4] |= BIT(3); + } else { + registers[offset / 4] &= ~BIT(1); + registers[0x7034 / 4] &= ~BIT(3); + } + } +} +static void core_set32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, 0, v); } +static void core_clear32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, v, 0); } +static void mask32(void *p, u32 m, u32 v) +{ + *(u32 *)p = (*(u32 *)p & ~m) | v; + if (p == (void *)&pipe_regs[PIPEHANDLER_LOCK_REQ / 4] && !stuck_ack) + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = hold_unlock_ack && + !(*(u32 *)p & PIPEHANDLER_LOCK_EN) ? PIPEHANDLER_LOCK_EN : + *(u32 *)p & PIPEHANDLER_LOCK_EN; + if (p == (void *)&pipe_regs[PIPEHANDLER_AON_GEN / 4]) reset_writes++; + if (p == (void *)&pipe_regs[PIPEHANDLER_MUX_CTRL / 4]) mux_writes++; + if ((uintptr_t)p >= (uintptr_t)usb2_regs && + (uintptr_t)p < (uintptr_t)usb2_regs + sizeof(usb2_regs)) usb2_writes++; +} +static void writel(u32 value, void *address) { mask32(address, ~0U, value); } +static void set32(void *p, u32 v) { mask32(p, 0, v); } +static void clear32(void *p, u32 v) { mask32(p, v, 0); } +#define readl_poll_timeout_atomic(address, value, condition, interval, timeout) \ + ({ (value) = readl(address); assert(polls < ARRAY_SIZE(poll_caller)); \ + poll_caller[polls++] = __func__; \ + bool bist_fault = bist_status_stuck && (void *)(address) == \ + (void *)®isters[ACIOPHY_BIST_MAC_USBPHY_STATUS2 / 4]; \ + bist_faults += bist_fault; \ + polls == fail_poll || bist_fault || \ + (fail_caller && !strcmp(fail_caller, __func__)) || !(condition) ? -ETIMEDOUT : 0; }) +#define readl_poll_timeout readl_poll_timeout_atomic +/* Observation is read-only and separately tested; retain production caller control flow. */ +static void atcphy_usb2_power_on(struct apple_atcphy *p) { (void)p; usb2_on++; } +static void atcphy_apply_tunables(struct apple_atcphy *p, enum atcphy_mode mode) +{ (void)p; (void)mode; } +static void atc_tunnel_stop_t8103(struct apple_atcphy *p) { (void)p; } +static void atc_tunnel_restore(struct apple_atcphy *p) { (void)p; } +''') + for name in ('atcphy_pipehandler_lock', 'atcphy_pipehandler_unlock', 'atcphy_pipehandler_check', + 'atcphy_pipehandler_set_mux', 'atcphy_clear_nonselected_phy_reset', + 'atcphy_configure_pipehandler_usb3_host_t8122', + 'atcphy_configure_pipehandler_usb3_host_t8103', + 'atcphy_configure_pipehandler_usb3', 'atcphy_configure_pipehandler_usb4', + 'atcphy_enable_dummy_phy', 'atcphy_configure_pipehandler_dummy', + 'atcphy_configure_pipehandler', 'atcphy_park_pipehandler', + 'atcphy_usb2_power_off', '_atcphy_dwc3_reset_assert', 'atcphy_dwc3_reset_assert'): + parts.append(function(atc, name)) + for name in ('core_read32', 'atcphy_get_mode_config', 'j700_field_update', 'j700_pma_idle'): + parts.append(function(atc, name)) + parts.append(dp) + for name in ('atcphy_configure_lanes', 'j700_aux_enable', 'j700_aux_disable', + 'atcphy_enable_dp_aux', 'atcphy_disable_dp_aux', 'atcphy_power_off', + 'atcphy_power_on', 'atcphy_configure', 'atcphy_sw_set', 'atcphy_mux_set', + 'atcphy_usb3_set_mode', 'atcphy_usb3_power_off'): + parts.append(function(atc, name)) + production = '\n'.join(parts) + if 'static int j700_usb_capture(' in dp: + production = '#define HAS_USB_CONTEXT\n' + production + prefix = r''' +#include +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +typedef uint16_t u16; +#define BIT(n) (1U << (n)) +#define GENMASK(h, l) ((~0U << (l)) & (~0U >> (31 - (h)))) +#define FIELD_PREP(m, v) (((u32)(v) << __builtin_ctz(m)) & (m)) +#define FIELD_GET(m, v) (((v) & (m)) >> __builtin_ctz(m)) +#define field_prep FIELD_PREP +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define hweight32(v) __builtin_popcount(v) +#define dev_info_once(...) ((void)0) +#define dev_dbg(...) ((void)0) +#define dev_warn(...) ((void)0) +static void report(void *dev, const char *fmt, ...) { (void)dev; (void)fmt; } +#define dev_err report +#define lockdep_assert_held(...) ((void)0) +#define WARN_ON_ONCE(v) assert(!(v)) +#define guard(kind) (void) +#define fallthrough __attribute__((fallthrough)) +#define ndelay(n) ((void)(n)) +#define udelay(n) ((void)(n)) +#define usleep_range(a, b) ((void)(a), (void)(b)) +enum { TYPEC_STATE_SAFE, TYPEC_STATE_USB, TYPEC_MODE_USB4, + TYPEC_DP_STATE_C, TYPEC_DP_STATE_D, TYPEC_DP_STATE_E }; +enum { USB_TYPEC_TBT_SID = 0x8087, USB_TYPEC_DP_SID = 0xff01 }; +#define EUDO_USB_MODE_MASK 3U +enum { EUDO_USB_MODE_USB2, EUDO_USB_MODE_USB3, EUDO_USB_MODE_USB4 }; +enum typec_orientation { TYPEC_ORIENTATION_NONE, TYPEC_ORIENTATION_NORMAL, + TYPEC_ORIENTATION_REVERSE }; +enum phy_mode { PHY_MODE_USB_HOST, PHY_MODE_USB_DEVICE }; +struct enter_usb_data { u32 eudo; }; +struct typec_altmode { unsigned int svid; }; +struct typec_mux_state { unsigned long mode; struct typec_altmode *alt; void *data; }; +struct typec_mux_dev { struct apple_atcphy *data; }; +struct typec_switch_dev { struct apple_atcphy *data; }; +#define typec_mux_get_drvdata(mux) ((mux)->data) +#define typec_switch_get_drvdata(sw) ((sw)->data) +struct phy { struct apple_atcphy *data; }; +#define phy_get_drvdata(phy) ((phy)->data) +struct reset_controller_dev { int unused; }; +#define container_of(p, t, m) ((t *)((char *)(p) - offsetof(t, m))) +''' + suffix = r''' +static struct atcphy_hw hw = { + .has_t8140_dp = true, .gen = ATCPHY_GENERATION_T8122, + .aciophy_lane_mode = ACIOPHY_LANE_MODE_T8122, + .aciophy_crossbar = ACIOPHY_CROSSBAR_T8122, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, +}; +static void cold_initial(struct apple_atcphy *p, bool fixed, bool swapped) +{ + memset(registers, 0, sizeof(registers)); + memset(pipe_regs, 0, sizeof(pipe_regs)); + memset(usb2_regs, 0, sizeof(usb2_regs)); + registers[0x200 / 4] = 0xee; /* public PMA powered state, synthetic cold fixture */ + registers[AUS_COMMON_DIG_RCAL1 / 4] = AUS_COMMON_DIG_RCAL1_ALL_CODES_DONE; + pipe_regs[PIPEHANDLER_AON_GEN / 4] = PIPEHANDLER_AON_GEN_DWC3_RESET_N; + usb2_regs[USB2PHY_USBCTL / 4] = USB2PHY_USBCTL_RUN; + *p = (struct apple_atcphy) { + .hw = &hw, .regs.core = (unsigned char *)registers, + .regs.lpdptx = (unsigned char *)registers, + .regs.pipehandler = (unsigned char *)pipe_regs, + .regs.usb2phy = (unsigned char *)usb2_regs, + .fixed_usb2 = fixed, .host_active = true, .swap_lanes = swapped, + .ss_tunables = true, .dp_tunables = true, .dp_presets_valid = { true }, + .mode = APPLE_ATCPHY_MODE_USB3_DP, .typec_mode = APPLE_ATCPHY_MODE_USB3_DP, + .pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY, .mode_dirty = false, + .dp_owned_pmas = 0, .dp_idle_pmas = 0, + .aux_power_requested = false, + }; + polls = fail_poll = writes = usb2_on = usb2_writes = reset_writes = mux_writes = 0; + stuck_ack = bist_status_stuck = false; + hold_unlock_ack = false; + fail_caller = NULL; + bist_faults = 0; + require_parked = false; +} +static void initial(struct apple_atcphy *p, bool fixed, bool dirty, bool swapped) +{ + cold_initial(p, fixed, swapped); +#ifdef HAS_USB_CONTEXT + if (fixed && hw.has_t8140_dp) { + /* Two-PMA cleanup stress setup: capture before ownership and actual DP writes. */ + assert(j700_usb_capture(p, APPLE_ATCPHY_MODE_DP) == 0); + assert(atcphy_configure_pipehandler(p, true) == 0); + assert(j700_dp_start(p, 3, 3, false) == 0); + } else +#endif + { + p->pipe_state = ATCPHY_PIPEHANDLER_STATE_USB3; + p->dp_owned_pmas = 3; /* legacy synthetic cleanup ownership, no J700 context path */ + } + p->mode_dirty = dirty; + p->dp_idle_pmas = dirty ? 1 : 0; + p->aux_power_requested = true; + polls = fail_poll = writes = usb2_on = usb2_writes = reset_writes = mux_writes = 0; +} + +static void inject_parking(unsigned int fault) +{ + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + if (fault) + fail_poll = 1; + else + stuck_ack = true; +} +static void recover(void) +{ + stuck_ack = bist_status_stuck = false; + hold_unlock_ack = false; + fail_caller = NULL; + fail_poll = polls = 0; +} +static int request(struct apple_atcphy *p, bool safe) +{ + struct typec_mux_dev mux = { p }; + struct typec_altmode dp = { .svid = USB_TYPEC_DP_SID }; + struct typec_mux_state state = { + .mode = safe ? TYPEC_STATE_SAFE : TYPEC_DP_STATE_C, + .alt = safe ? NULL : &dp, + }; + return atcphy_mux_set(&mux, &state); +} +static void unchanged(struct apple_atcphy *p, bool dirty, bool host) +{ + bool expected_dirty = dirty || (p->hw->has_t8140_dp && p->fixed_usb2 && host); + assert(p->mode == APPLE_ATCPHY_MODE_USB3_DP && p->mode_dirty == expected_dirty); + assert(p->pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && p->host_active == host); + assert(p->dp_owned_pmas == 3 && p->dp_idle_pmas == (dirty ? 1U : 0U)); + assert(p->aux_power_requested); + assert(writes == 0 && usb2_on == 0 && usb2_writes == 0 && reset_writes == 0); + assert(mux_writes == 0); + assert(pipe_regs[PIPEHANDLER_AON_GEN / 4] == PIPEHANDLER_AON_GEN_DWC3_RESET_N); + assert(usb2_regs[USB2PHY_USBCTL / 4] == USB2PHY_USBCTL_RUN); + if (expected_dirty) { + struct phy_configure_opts_dp training = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + assert(j700_dp_configure(p, &training) == -EBUSY && writes == 0); + } +} +static void observed(const char *caller, int result) +{ + printf("%s: result=%d core_writes=%u usb2_writes=%u reset_writes=%u\n", + caller, result, writes, usb2_writes, reset_writes); + fflush(stdout); +} +static void mux_parking(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + for (unsigned int dirty = 0; dirty < 2; dirty++) + for (unsigned int swapped = 0; swapped < 2; swapped++) + for (unsigned int host = 0; host < 2; host++) + for (unsigned int safe = 0; safe < 2; safe++) + for (unsigned int fault = 0; fault < 2; fault++) { + initial(&p, true, dirty, swapped); + p.host_active = host; + inject_parking(fault); + int ret = request(&p, safe); + observed("mux parking failure", ret); + assert(ret == -ETIMEDOUT); + unchanged(&p, dirty, host); + if (!fault) { + assert(request(&p, safe) == -ETIMEDOUT); + unchanged(&p, dirty, host); + } + recover(); + assert(request(&p, safe) == 0); + assert(p.mode == (safe ? APPLE_ATCPHY_MODE_USB2 : APPLE_ATCPHY_MODE_DP)); + assert(!p.mode_dirty && p.host_active == host); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && !p.dp_owned_pmas); + assert(!p.dp_idle_pmas && p.aux_power_requested == !safe); + assert(usb2_on == 1 && usb2_writes == 0 && reset_writes == 0); + assert(usb2_regs[USB2PHY_USBCTL / 4] == USB2PHY_USBCTL_RUN); + unsigned int before = writes; + assert(request(&p, safe) == 0 && writes == before && usb2_on == 1); + cases++; + } + printf("PASS: %u actual mux parking/retry scenarios preserve fixed USB2 and ownership\n", cases); +} +static void power_parking(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + for (unsigned int fixed = 0; fixed < 2; fixed++) + for (unsigned int dirty = 0; dirty < 2; dirty++) + for (unsigned int swapped = 0; swapped < 2; swapped++) + for (unsigned int fault = 0; fault < 2; fault++) { + initial(&p, fixed, dirty, swapped); + struct phy phy = { &p }; + inject_parking(fault); + int ret = atcphy_usb3_power_off(&phy); + observed("power-off parking failure", ret); + assert(ret == -ETIMEDOUT); + unchanged(&p, dirty, true); + if (!fault) { + assert(atcphy_usb3_power_off(&phy) == -ETIMEDOUT); + unchanged(&p, dirty, true); + } + recover(); + assert(atcphy_usb3_power_off(&phy) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_OFF && !p.mode_dirty && !p.host_active); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + assert(usb2_writes == 0 && reset_writes == 0); + unsigned int before = writes; + assert(atcphy_usb3_power_off(&phy) == 0 && writes == before); + cases++; + } + printf("PASS: %u actual power-off parking/retry scenarios\n", cases); +} +static void reset_parking(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + for (unsigned int fixed = 0; fixed < 2; fixed++) + for (unsigned int dirty = 0; dirty < 2; dirty++) + for (unsigned int fault = 0; fault < 2; fault++) { + initial(&p, fixed, dirty, false); + inject_parking(fault); + int ret = atcphy_dwc3_reset_assert(&p.rcdev, 0); + observed("reset parking failure", ret); + assert(ret == -ETIMEDOUT); + unchanged(&p, dirty, true); + if (!fault) { + assert(atcphy_dwc3_reset_assert(&p.rcdev, 0) == -ETIMEDOUT); + unchanged(&p, dirty, true); + } + recover(); + assert(atcphy_dwc3_reset_assert(&p.rcdev, 0) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && !p.host_active); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && p.mode_dirty == (dirty || fixed)); + assert(p.dp_owned_pmas == 3 && p.dp_idle_pmas == (dirty ? 1U : 0U)); + assert(p.aux_power_requested && writes == 0); + assert(reset_writes == 2 && usb2_writes > 0); + assert(!(pipe_regs[PIPEHANDLER_AON_GEN / 4] & PIPEHANDLER_AON_GEN_DWC3_RESET_N)); + assert(pipe_regs[PIPEHANDLER_AON_GEN / 4] & PIPEHANDLER_AON_GEN_DWC3_FORCE_CLAMP_EN); + assert(!(usb2_regs[USB2PHY_USBCTL / 4] & USB2PHY_USBCTL_RUN)); + cases++; + } + printf("PASS: %u actual reset parking/retry scenarios retain DP ownership\n", cases); +} +static void power_cleanup(void) +{ + struct apple_atcphy p; + unsigned int phases = 0; + for (unsigned int fixed = 0; fixed < 2; fixed++) { + initial(&p, fixed, false, false); + p.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + struct phy phy = { &p }; + fail_poll = 1; /* The first DP stop ACK cannot complete. */ + int ret = atcphy_usb3_power_off(&phy); + observed("power-off DP cleanup failure", ret); + assert(ret == -ETIMEDOUT && p.mode_dirty && p.host_active); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && p.dp_owned_pmas == 3); + assert(p.aux_power_requested && usb2_writes == 0 && reset_writes == 0); + struct phy_configure_opts_dp training = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + unsigned int before = writes; + assert(j700_dp_configure(&p, &training) == -EBUSY && writes == before); + recover(); + assert(atcphy_usb3_power_off(&phy) == 0); + assert(!p.mode_dirty && !p.host_active && !p.dp_owned_pmas && !p.aux_power_requested); + /* A failed initial transition can leave OFF recorded with dirty resources. */ + initial(&p, fixed, true, false); + p.mode = APPLE_ATCPHY_MODE_OFF; + p.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + fail_poll = 1; + assert(atcphy_usb3_power_off(&phy) == -ETIMEDOUT); + assert(p.mode_dirty && p.host_active && p.dp_owned_pmas == 3); + recover(); + assert(atcphy_usb3_power_off(&phy) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_OFF && !p.mode_dirty && !p.host_active); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + for (unsigned int fault = 1; fault <= 2; fault++) { + initial(&p, fixed, false, false); + p.mode = APPLE_ATCPHY_MODE_USB2; + p.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + p.dp_owned_pmas = p.dp_idle_pmas = 0; + p.aux_power_requested = false; + fail_poll = fault; /* BIG/SMALL sleep ACK. */ + assert(atcphy_usb3_power_off(&phy) == -ETIMEDOUT); + assert(p.mode == APPLE_ATCPHY_MODE_USB2 && p.mode_dirty && p.host_active); + recover(); + assert(atcphy_usb3_power_off(&phy) == 0 && !p.mode_dirty && !p.host_active); + } + /* Inject every poll in DP stop, pending PMA idle, AUX and domain sleep. */ + initial(&p, fixed, true, false); + p.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + assert(atcphy_usb3_power_off(&phy) == 0); + unsigned int total = polls; + for (unsigned int fault = 1; fault <= total; fault++) { + initial(&p, fixed, true, false); + p.pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + fail_poll = fault; + assert(atcphy_usb3_power_off(&phy) == -ETIMEDOUT); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && p.mode_dirty && p.host_active); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert(usb2_writes == 0 && reset_writes == 0); + unsigned int owned = p.dp_owned_pmas, idle = p.dp_idle_pmas; + bool aux = p.aux_power_requested; + unsigned int before = writes; + assert(j700_dp_configure(&p, &training) == -EBUSY && writes == before); + assert(p.dp_owned_pmas == owned && p.dp_idle_pmas == idle); + assert(p.aux_power_requested == aux); + recover(); + assert(atcphy_usb3_power_off(&phy) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_OFF && !p.mode_dirty && !p.host_active); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + phases++; + } + } + printf("PASS: %u cleanup poll faults retain retry/ownership; dirty OFF retries cleanup\n", phases); +} +static int select_mode(struct apple_atcphy *p, enum atcphy_mode mode) +{ + struct typec_mux_dev mux = { p }; + struct typec_altmode dp = { .svid = USB_TYPEC_DP_SID }; + struct typec_mux_state state = { .mode = TYPEC_STATE_SAFE }; + + if (mode == APPLE_ATCPHY_MODE_USB3) + state.mode = TYPEC_STATE_USB; + else if (mode == APPLE_ATCPHY_MODE_USB3_DP || mode == APPLE_ATCPHY_MODE_DP) { + state.alt = &dp; + state.mode = mode == APPLE_ATCPHY_MODE_DP ? TYPEC_DP_STATE_C : TYPEC_DP_STATE_D; + } + return atcphy_mux_set(&mux, &state); +} +static void orient(struct apple_atcphy *p, bool reverse) +{ + struct typec_switch_dev sw = { p }; + assert(atcphy_sw_set(&sw, reverse ? TYPEC_ORIENTATION_REVERSE : + TYPEC_ORIENTATION_NORMAL) == 0); +} +static void orientation_initial(struct apple_atcphy *p, bool fixed) +{ + cold_initial(p, fixed, false); + p->mode = p->typec_mode = fixed ? APPLE_ATCPHY_MODE_USB2 : APPLE_ATCPHY_MODE_OFF; + p->pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; + p->dp_owned_pmas = p->dp_idle_pmas = 0; + p->aux_power_requested = false; + p->dp_link_rate = -1; + p->host_active = fixed; + registers[ACIOPHY_TOP_PHY_STAT / 4] = ACIOPHY_TOP_PHY_STAT_LN0_UNK0; + registers[ATCPHY_MISC / 4] = BIT(7) | ATCPHY_MISC_LANE_SWAP; + expected_j700_swap = true; + registers[hw.aciophy_lane_mode / 4] = 0xa5000000; + registers[hw.aciophy_crossbar / 4] = 0x80000000; + pipe_regs[PIPEHANDLER_MUX_CTRL / 4] = + FIELD_PREP(hw.pipe_mux_data, PIPEHANDLER_MUX_CTRL_DATA_DUMMY) | + FIELD_PREP(hw.pipe_mux_clk, PIPEHANDLER_MUX_CTRL_CLK_DUMMY); +} +static void check_route(struct apple_atcphy *p, enum atcphy_mode mode, bool reverse) +{ + assert(p->mode == mode && p->swap_lanes == reverse && !p->mode_dirty); + assert(registers[ATCPHY_MISC / 4] & BIT(7)); + /* Fixed public route expectations, independent of the production mode table. */ + assert(mode == APPLE_ATCPHY_MODE_USB3 || mode == APPLE_ATCPHY_MODE_USB3_DP || + mode == APPLE_ATCPHY_MODE_DP); + u32 lane = registers[hw.aciophy_lane_mode / 4]; + u32 crossbar = registers[hw.aciophy_crossbar / 4]; + u32 mask = hw.has_t8140_dp && mode != APPLE_ATCPHY_MODE_DP ? 0x1ffff : 0x3ffff; + assert((lane & 0xfff) == (mode == APPLE_ATCPHY_MODE_DP ? 0x492 : + (reverse ? 0x252 : 0x489))); + assert((lane & 0xfffff000) == 0xa5000000); + assert((crossbar & mask) == (mode == APPLE_ATCPHY_MODE_DP ? + (reverse ? 0x114 : 0x22014) : (0x110 | reverse))); + assert(crossbar & 0x80000000); + /* J700 preserves the boot swap bit; generic routes apply their explicit swap. */ + assert(!!(registers[ATCPHY_MISC / 4] & ATCPHY_MISC_LANE_SWAP) == + (hw.has_t8140_dp ? expected_j700_swap : + (mode != APPLE_ATCPHY_MODE_DP && reverse))); + assert(p->pipe_state == (p->fixed_usb2 && mode != APPLE_ATCPHY_MODE_DP ? + ATCPHY_PIPEHANDLER_STATE_USB3 : ATCPHY_PIPEHANDLER_STATE_DUMMY)); +} +static void orientation_roundtrip(void) +{ + struct apple_atcphy p; + const enum atcphy_mode modes[] = { + APPLE_ATCPHY_MODE_USB3, APPLE_ATCPHY_MODE_USB3_DP, APPLE_ATCPHY_MODE_DP, + }; + unsigned int cases = 0; + + for (unsigned int generation = 0; generation < 4; generation++) { + hw.gen = generation == 0 ? ATCPHY_GENERATION_T8103 : ATCPHY_GENERATION_T8122; + hw.has_t8140_dp = generation == 2; + hw.has_usb4 = !hw.has_t8140_dp; + hw.aciophy_lane_mode = generation == 0 ? ACIOPHY_LANE_MODE_T8103 : + ACIOPHY_LANE_MODE_T8122; + hw.aciophy_crossbar = generation == 0 ? ACIOPHY_CROSSBAR_T8103 : + ACIOPHY_CROSSBAR_T8122; + hw.pipe_mux_clk = generation == 3 ? PIPEHANDLER_MUX_CTRL_CLK_T8142 : + PIPEHANDLER_MUX_CTRL_CLK_T8103; + hw.pipe_mux_data = generation == 3 ? PIPEHANDLER_MUX_CTRL_DATA_T8142 : + PIPEHANDLER_MUX_CTRL_DATA_T8103; + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int m = 0; m < ARRAY_SIZE(modes); m++) { + for (unsigned int first = 0; first < 2; first++) { + orientation_initial(&p, fixed); + for (unsigned int turn = 0; turn < 3; turn++) { + bool reverse = first ^ (turn & 1); + orient(&p, reverse); + if (turn) { + assert(p.mode_dirty); + struct phy phy = { &p }; + unsigned int before = writes; + assert(atcphy_usb3_set_mode(&phy, PHY_MODE_USB_HOST, 0) == -EBUSY); + assert(writes == before); + } + require_parked = true; + assert(select_mode(&p, modes[m]) == 0); + require_parked = false; + check_route(&p, modes[m], reverse); + unsigned int before = writes, before_mux = mux_writes; + orient(&p, reverse); + assert(select_mode(&p, modes[m]) == 0); + assert(writes == before && mux_writes == before_mux); + struct typec_switch_dev sw = { &p }; + assert(atcphy_sw_set(&sw, TYPEC_ORIENTATION_NONE) == 0); + assert(!p.mode_dirty && p.swap_lanes == reverse); + cases++; + } + /* A USB3 protocol change also rebuilds the live backend. */ + if (m == 0) { + require_parked = true; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + require_parked = false; + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, first); + } + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == 0); + assert(p.mode == (fixed ? APPLE_ATCPHY_MODE_USB2 : APPLE_ATCPHY_MODE_OFF)); + assert(!p.mode_dirty && !p.aux_power_requested && !p.dp_idle_pmas); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + } + } + } + } + printf("PASS: %u production orientation routes across T8103/T8122/T8140/T8142, fixed/ordinary ports, both flip roundtrips\n", cases); +} +static void orientation_faults(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int first = 0; first < 2; first++) { + orientation_initial(&p, fixed); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + assert(j700_dp_configure(&p, &rate) == 0); + assert(p.dp_owned_pmas == BIT(!first)); + orient(&p, !first); + unsigned int before = writes; + struct phy_configure_opts_dp training = { .set_voltages = true, .lanes = 2 }; + assert(j700_dp_configure(&p, &training) == -EBUSY && writes == before); + assert(j700_dp_configure(&p, &rate) == -EBUSY && writes == before); + /* Derive all actual poll phases, then inject each in a fresh transition. */ + polls = 0; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + unsigned int total = polls; + bool advisory[256] = { false }; + for (unsigned int i = 0; i < total; i++) { + const char *caller = poll_caller[i]; + /* Every observed BIST and USB3 handoff failure is now fatal. */ + if (fixed && i + 1 == total) { + assert(!strcmp(caller, "atcphy_pipehandler_unlock")); + advisory[i] = false; + } else if (!strcmp(caller, "atcphy_configure_pipehandler_usb3_host_t8122")) { + assert(fixed); + advisory[i] = false; + } else { + assert(!strcmp(caller, "j700_dp_wait") || + !strcmp(caller, "j700_field_update") || + !strcmp(caller, "j700_usb_field_update") || + !strcmp(caller, "atcphy_power_on") || + !strcmp(caller, "atcphy_configure") || + (fixed && (!strcmp(caller, "atcphy_pipehandler_lock") || + !strcmp(caller, "atcphy_pipehandler_unlock")))); + } + } + for (unsigned int fault = 1; fault <= total; fault++) { + orientation_initial(&p, fixed); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + unsigned int owned = p.dp_owned_pmas; + orient(&p, !first); + polls = 0; + fail_poll = fault; + int ret = select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP); + /* A fatal phase must never be accepted as advisory success. */ + assert((ret == 0) == advisory[fault - 1]); + if (!ret) { + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, !first); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas); + cases++; + continue; + } + assert(ret == -ETIMEDOUT); + if (p.mode_dirty || p.pipe_pending) { + before = writes; + assert(j700_dp_configure(&p, &training) == -EBUSY && writes == before); + assert(j700_dp_configure(&p, &rate) == -EBUSY && writes == before); + if (fault == 1 && !fixed) + assert(p.dp_owned_pmas == owned && p.aux_power_requested); + } else { + assert(fixed && p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + } + recover(); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, !first); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas); + assert(j700_dp_configure(&p, &rate) == 0); + assert(p.dp_owned_pmas == BIT(first)); + assert(j700_dp_configure(&p, &training) == 0); + cases++; + } + /* A fatal parking check preserves the old owner and all lane words. */ + if (fixed) { + orientation_initial(&p, true); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + orient(&p, true); + before = writes; + inject_parking(0); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(writes == before && p.dp_owned_pmas == 2 && p.mode_dirty); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && p.host_active); + recover(); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, true); + } + /* Reversing a pending flip cannot erase cleanup obligations. */ + orientation_initial(&p, fixed); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + orient(&p, true); + orient(&p, false); + assert(p.mode_dirty && p.dp_owned_pmas == 2); + p.ss_tunables = false; + before = writes; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -EOPNOTSUPP); + assert(writes == before && p.mode_dirty && p.dp_owned_pmas == 2); + p.ss_tunables = true; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, false); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas); + } + } + printf("PASS: %u orientation transition poll injections, dirty training guard, ownership and retry controls\n", cases); +} +static void backend_change(void) +{ + struct apple_atcphy p; + + orientation_initial(&p, true); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3); + require_parked = true; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + require_parked = false; + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, false); + puts("PASS: a live fixed-hub USB3 backend parks before same-orientation protocol changes"); +} + + +static void failed_flip_return(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int first = 0; first < 2; first++) { + orientation_initial(&p, fixed); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + orient(&p, !first); + polls = 0; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + unsigned int route_fault = 0; + for (unsigned int i = 0; i < polls; i++) + if (!strcmp(poll_caller[i], "j700_field_update")) route_fault = i + 1; + assert(route_fault); + + orientation_initial(&p, fixed); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + orient(&p, !first); + polls = 0; + fail_poll = route_fault; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + /* The route changed before PMA idle completed; completed mode is still old. */ + assert((registers[0x60 / 4] & 0xfff) == (first ? 0x489 : 0x252)); + assert(p.mode_dirty && p.dp_idle_pmas == BIT(first) && !p.dp_owned_pmas); + orient(&p, first); + assert(p.mode_dirty && p.dp_idle_pmas == BIT(first)); + unsigned int before = writes; + assert(j700_dp_configure(&p, &rate) == -EBUSY && writes == before); + recover(); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, first); + assert(!p.dp_idle_pmas && !p.dp_owned_pmas); + assert(j700_dp_configure(&p, &rate) == 0); + assert(p.dp_owned_pmas == BIT(!first)); + cases++; + } + } + printf("PASS: %u partially applied flip cancellations preserve pending PMA ownership\n", cases); +} + +static void safe_first_recovery(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + + for (unsigned int first = 0; first < 2; first++) { + orientation_initial(&p, true); + orient(&p, first); + bist_status_stuck = true; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3) == -ETIMEDOUT); + assert(bist_faults == 2 && p.mode_dirty && p.pipe_pending); + bist_status_stuck = false; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3, first); + polls = 0; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + unsigned int host_lock_poll = 0; + for (unsigned int i = 0; i < polls; i++) + if (!strcmp(poll_caller[i], "atcphy_pipehandler_lock")) host_lock_poll = i + 1; + assert(host_lock_poll > 1); /* Inject the host lock, after acknowledged parking. */ + orientation_initial(&p, true); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3) == 0); + polls = 0; + fail_poll = host_lock_poll; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && !p.mode_dirty); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && p.host_active); + assert(p.aux_power_requested); + + /* Persistent AUX cleanup failure blocks SAFE before orientation publication. */ + fail_caller = "j700_field_update"; + u32 route = registers[0x60 / 4], crossbar = registers[0x64 / 4]; + for (unsigned int attempt = 0; attempt < 3; attempt++) { + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == -ETIMEDOUT); + assert(p.swap_lanes == first && p.mode_dirty && p.aux_power_requested); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && p.host_active); + assert(registers[0x60 / 4] == route && registers[0x64 / 4] == crossbar); + assert(usb2_regs[USB2PHY_USBCTL / 4] == USB2PHY_USBCTL_RUN); + } + /* Returning the plug cannot clear an unavailable cleanup handshake. */ + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == -ETIMEDOUT); + assert(p.swap_lanes == first && p.mode_dirty); + recover(); /* Model hardware readiness returning; production performs the retry. */ + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_USB2 && !p.mode_dirty && !p.aux_power_requested); + /* SAFE uses the existing generic USB2 route and clears its swap control. */ + assert(!(registers[ATCPHY_MISC / 4] & ATCPHY_MISC_LANE_SWAP)); + expected_j700_swap = false; + orient(&p, !first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, !first); + + /* Real reconnect order: SAFE must succeed before the new orientation is published. */ + struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + assert(j700_dp_configure(&p, &rate) == 0); + assert(p.dp_owned_pmas == BIT(first)); + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == 0); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, first); + assert(j700_dp_configure(&p, &rate) == 0); + assert(p.dp_owned_pmas == BIT(!first)); + unsigned int before = writes, before_mux = mux_writes; + orient(&p, first); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(writes == before && mux_writes == before_mux); + cases++; + } + printf("PASS: %u chained BIST/host-ACK failure, persistent SAFE failure and SAFE-first roundtrip controls\n", cases); +} + +static void live_parking_handshake(void) +{ + struct apple_atcphy p; + struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + struct phy_configure_opts_dp training = { .set_voltages = true, .lanes = 2 }; + unsigned int cases = 0; + for (unsigned int reverse = 0; reverse < 2; reverse++) { + for (unsigned int fault = 0; fault < 3; fault++) { + orientation_initial(&p, true); + orient(&p, reverse); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + u32 lanes = registers[0x60 / 4], route = registers[0x64 / 4]; + u32 native = pipe_regs[PIPEHANDLER_NONSELECTED_OVERRIDE / 4]; + unsigned int before = writes, owned = p.dp_owned_pmas; + if (fault == 2) { + /* A previous host unlock left ACK asserted before parking. */ + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + hold_unlock_ack = true; + } else if (fault == 1) { + hold_unlock_ack = true; + } else { + stuck_ack = true; + } + for (unsigned int retry = 0; retry < 3; retry++) { + unsigned int before_mux = mux_writes; + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == -ETIMEDOUT); + assert(p.mode == APPLE_ATCPHY_MODE_USB3_DP && p.mode_dirty); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && p.host_active); + assert(p.dp_owned_pmas == owned && p.aux_power_requested); + assert(writes == before && registers[0x60 / 4] == lanes && registers[0x64 / 4] == route); + assert(pipe_regs[PIPEHANDLER_NONSELECTED_OVERRIDE / 4] == native); + if (fault != 1) assert(mux_writes == before_mux); + /* A failed SAFE cannot be forgotten by returning to the cached mode. */ + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(j700_dp_configure(&p, &training) == -EBUSY && writes == before); + assert(!usb2_writes && !reset_writes); + } + recover(); + assert(select_mode(&p, APPLE_ATCPHY_MODE_OFF) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_USB2 && !p.mode_dirty && p.host_active); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + expected_j700_swap = false; + orient(&p, !reverse); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + check_route(&p, APPLE_ATCPHY_MODE_USB3_DP, !reverse); + assert(j700_dp_configure(&p, &rate) == 0 && p.dp_owned_pmas == BIT(reverse)); + cases++; + } + } + printf("PASS: %u live parking entry-ACK/LOCK/UNLOCK failure, ownership, write ordering and retry roundtrips\n", cases); +} + +int main(int argc, char **argv) +{ + assert(argc == 2); + if (!strcmp(argv[1], "mux")) mux_parking(); + else if (!strcmp(argv[1], "power")) power_parking(); + else if (!strcmp(argv[1], "reset")) reset_parking(); + else if (!strcmp(argv[1], "cleanup")) power_cleanup(); + else if (!strcmp(argv[1], "orientation")) orientation_roundtrip(); + else if (!strcmp(argv[1], "orientation-faults")) orientation_faults(); + else if (!strcmp(argv[1], "backend-change")) backend_change(); + else if (!strcmp(argv[1], "safe-first")) safe_first_recovery(); + else if (!strcmp(argv[1], "live-park")) live_parking_handshake(); + else if (!strcmp(argv[1], "failed-flip-return")) failed_flip_return(); + else assert(false); + return 0; +} +''' + return {**locals(), "block": block} + +def adverse(root): + + # Preserve continued source macros; the parking fixture does not use the + # continued PLL clock macro exercised by the legacy rate helper here. + ns = parking(root) + production = ns['production'] + '\n' + ns['function'](ns['atc'], 'atcphy_usb2_init') + production = production.replace('bool pipe_pending, usb_off_pending, usb_off_drained;', + 'bool pipe_pending, usb_off_pending, usb_off_drained, tunnel_clock_on, tunnel_saved;', 1) + production += '\n' + ns['block'](ns['atc'], 'enum atcphy_lane {') + ';' + production += '\n' + ns['block'](ns['atc'], 'struct atcphy_dp_link_rate_configuration {') + ';' + production += '\n' + ns['block'](ns['atc'], 'static const struct atcphy_dp_link_rate_configuration dp_lr_config[] = {') + ';' + # These leaf operations are outside the pending/cache gate under test. + production += r''' +static int atcphy_auspll_apb_command(struct apple_atcphy *p, u32 command) +{ (void)p; (void)command; assert(false); return -EIO; } +static int atcphy_dp_configure_lane(struct apple_atcphy *p, enum atcphy_lane lane, + const struct atcphy_dp_link_rate_configuration *cfg) +{ (void)p; (void)lane; (void)cfg; assert(false); return -EIO; } +''' + production += '\n' + ns['function'](ns['atc'], 'atcphy_dp_configure') + for profile in ('t8103', 't8122', 't8140', 't8142'): + production += '\n' + ns['block'](ns['atc'], + 'static const struct atcphy_hw atcphy_hw_' + profile + ' = {') + ';' + production = production.replace('static unsigned int polls,', 'static int poll_errors[256];\nstatic unsigned int polls,', 1) + production = production.replace('polls == fail_poll || bist_fault ||', + 'poll_errors[polls] ? poll_errors[polls] : polls == fail_poll || bist_fault ||', 1) + production = production.replace('static u32 readl(void *address) { return *(u32 *)address; }', + '''static unsigned int mmio_reads; +static bool only_power_mmio; +static u32 readl(void *address) +{ + mmio_reads++; + if (only_power_mmio && (uintptr_t)address >= (uintptr_t)registers && + (uintptr_t)address < (uintptr_t)registers + sizeof(registers)) { + uintptr_t offset=(uintptr_t)address-(uintptr_t)registers; + assert(offset==ATCPHY_POWER_CTRL || offset==ATCPHY_POWER_STAT || offset==ATCPHY_MISC); + } + return *(u32 *)address; +}''', 1) + production = production.replace(' (void)p;\n assert(offset < sizeof(registers)', + ' (void)p;\n if (only_power_mmio) assert(offset==ATCPHY_POWER_CTRL || offset==ATCPHY_MISC);\n assert(offset < sizeof(registers)', 1) + controls = r''' +static void legacy_setup(struct apple_atcphy *p, unsigned int gen, bool fixed) +{ + hw = gen==0 ? atcphy_hw_t8103 : gen==1 ? atcphy_hw_t8122 : + gen==2 ? atcphy_hw_t8140 : atcphy_hw_t8142; + cold_initial(p, fixed, false); + p->mode = p->typec_mode = APPLE_ATCPHY_MODE_USB3; + registers[ACIOPHY_TOP_PHY_STAT/4] = ACIOPHY_TOP_PHY_STAT_LN0_UNK0; + memset(poll_errors, 0, sizeof(poll_errors)); +} +static void shared_bist(void) +{ + for (unsigned int gen=0; gen<2; gen++) { + unsigned int count=gen ? 5 : 3; + for (unsigned int fault=1; fault<=count; fault++) { + struct apple_atcphy p; + legacy_setup(&p,gen,false); + poll_errors[fault]=-EIO; + if (fault1 && !strcmp(argv[1],"epoch")) { epoch_controls(); return 0; } + if(argc>1 && !strcmp(argv[1],"bist")) { bist_controls(); return 0; } + if(argc>1)return original_fixture_main(argc,argv); + epoch_controls();bist_controls();return 0; +} +''' + return ns['prefix'] + production + ns['suffix'].replace('int main(', 'int original_fixture_main(', 1) + controls + +def warm_reset(root): + + ns = parking(root) + production = ns['production'] + reset_block = '''\t/* +\t * Fixed-hub SAFE keeps USB2 up rather than taking the block OFF. +\t * With PIPE parked and DP owners released, reinitialize USB3 before +\t * programming its lanes; the existing sequence releases this reset. +\t */ +\tif (atcphy->hw->has_t8140_dp && atcphy->fixed_usb2) +\t\tcore_clear32(atcphy, ATCPHY_POWER_CTRL, ATCPHY_POWER_PHY_RESET_N); + +''' + assert production.count(reset_block) == 1 + production = production.replace('static u32 registers[', '''static unsigned int warm_reset_edges, warm_unreset_lanes, warm_ownership_errors; +static u32 registers[''', 1) + production = production.replace(' registers[offset / 4] = (registers[offset / 4] & ~mask) | value;', ''' if (offset == ATCPHY_POWER_CTRL && (mask & ATCPHY_POWER_PHY_RESET_N) && + !(value & ATCPHY_POWER_PHY_RESET_N)) { + warm_reset_edges++; + if (p->fixed_usb2 && p->hw->has_t8140_dp && + (p->pipe_state != ATCPHY_PIPEHANDLER_STATE_DUMMY || p->dp_owned_pmas || + p->dp_idle_pmas || p->aux_power_requested)) warm_ownership_errors++; + } + if (offset == p->hw->aciophy_lane_mode && p->fixed_usb2 && p->hw->has_t8140_dp && + (registers[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N)) warm_unreset_lanes++; + registers[offset / 4] = (registers[offset / 4] & ~mask) | value;''', 1) + controls = r''' +static void warm_initial(struct apple_atcphy *p) +{ + orientation_initial(p, true); + registers[ATCPHY_POWER_CTRL / 4] |= ATCPHY_POWER_PHY_RESET_N; + warm_reset_edges = warm_unreset_lanes = warm_ownership_errors = 0; +} +static void warm_roundtrip(void) +{ + bool failed = false; + for (unsigned int first = 0; first < 2; first++) { + struct apple_atcphy p; + const struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + warm_initial(&p); + for (unsigned int attach = 0; attach < 3; attach++) { + orient(&p, first ^ (attach & 1)); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && !p.mode_dirty); + assert(j700_dp_configure(&p, &rate) == 0); + unsigned int edges = warm_reset_edges; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(warm_reset_edges == edges); /* cached same-mode must not reset */ + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB2) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && p.host_active); + assert(!p.dp_owned_pmas && !p.dp_idle_pmas && !p.aux_power_requested); + assert(registers[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); + assert(!reset_writes && !usb2_writes); + } + assert(usb2_on == 6); /* no extra USB2 power-on call introduced */ + printf("first=%u reset_assertions=%u unreset_lane_writes=%u ownership_errors=%u\n", + first, warm_reset_edges, warm_unreset_lanes, warm_ownership_errors); + failed |= warm_reset_edges != 6 || warm_unreset_lanes || warm_ownership_errors; + } + fflush(stdout); + assert(!failed); +} +static void warm_failure_controls(void) +{ + struct apple_atcphy p; + const struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 2, + }; + warm_initial(&p); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + unsigned int edges = warm_reset_edges, owned = p.dp_owned_pmas; + stuck_ack = true; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB2) == -ETIMEDOUT); + assert(warm_reset_edges == edges && p.dp_owned_pmas == owned && p.mode_dirty); + assert(!usb2_writes && !reset_writes); + + warm_initial(&p); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(j700_dp_configure(&p, &rate) == 0); + edges = warm_reset_edges; owned = p.dp_owned_pmas; + fail_caller = "j700_field_update"; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB2) == -ETIMEDOUT); + assert(warm_reset_edges == edges && p.dp_owned_pmas == owned && p.mode_dirty); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert(!usb2_writes && !reset_writes); + + warm_initial(&p); + fail_caller = "atcphy_configure"; /* existing RCAL poll, after reset release */ + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(warm_reset_edges == 1 && !warm_ownership_errors && p.mode_dirty); + assert(registers[ATCPHY_POWER_CTRL / 4] & ATCPHY_POWER_PHY_RESET_N); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + struct phy phy = { &p }; + unsigned int before = writes; + assert(atcphy_usb3_set_mode(&phy, PHY_MODE_USB_HOST, 0) == -EBUSY); + assert(j700_dp_configure(&p, &rate) == -EBUSY && writes == before); + fail_caller = NULL; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(!p.mode_dirty && !warm_ownership_errors); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3); + assert(!usb2_writes && !reset_writes); + puts("PASS: park/DP-cleanup failures precede reset; released-reset RCAL failure stays dirty and retryable"); +} +static void warm_scope_controls(void) +{ + struct apple_atcphy p; + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int t8140 = 0; t8140 < 2; t8140++) { + orientation_initial(&p, fixed); + hw.has_t8140_dp = t8140; + registers[ATCPHY_POWER_CTRL / 4] |= ATCPHY_POWER_PHY_RESET_N; + warm_reset_edges = warm_unreset_lanes = warm_ownership_errors = 0; + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + assert(warm_reset_edges == (fixed && t8140)); + assert(!warm_ownership_errors && !usb2_writes && !reset_writes); + } + } + hw.has_t8140_dp = true; + puts("PASS: reset scope requires both fixed hub and T8140; ordinary/legacy paths unchanged"); +} +int main(int argc, char **argv) +{ + if (argc > 1) return original_fixture_main(argc, argv); + warm_roundtrip(); warm_failure_controls(); warm_scope_controls(); + puts("PASS: warm reset source ordering; no electrical recovery is modeled"); + return 0; +} +''' + suffix = ns['suffix'].replace('int main(', 'int original_fixture_main(', 1) + controls + return ns['prefix'] + production + suffix + +def usb_context(root): + ns = parking(root) + production=ns['production'].replace('static u32 registers[', 'static u32 blocked_offset;\nstatic unsigned int mmio_reads;\nstatic bool track_dp_writes;\nstatic u32 dp_write_masks[0x30000 / 4];\nstatic u32 registers[', 1) + production=production.replace('static u32 readl(void *address) { return *(u32 *)address; }', 'static u32 readl(void *address) { mmio_reads++; return *(u32 *)address; }', 1) + production=production.replace(' registers[offset / 4] = (registers[offset / 4] & ~mask) | value;', ' if (track_dp_writes) dp_write_masks[offset / 4] |= mask | value;\n if (blocked_offset && offset == blocked_offset) { writes++; return; }\n registers[offset / 4] = (registers[offset / 4] & ~mask) | value;', 1) + controls = r''' +static void seed_usb(struct apple_atcphy *p, unsigned int first) +{ + orientation_initial(p, true); + registers[0x200 / 4] = 0xee; + registers[0xc / 4] = 0; + /* Synthetic USB context, never a private factory table or captured register image. */ + for (unsigned int offset = 0x9000; offset < 0x15000; offset += 4) + registers[offset / 4] = 0x5a5a1234 ^ offset; + registers[0x8 / 4] = 0xa5000000; + registers[0x10c / 4] = 0xa5000000; + orient(p, first); + assert(select_mode(p, APPLE_ATCPHY_MODE_USB3_DP) == 0); +} + +static void mask_coverage(void) +{ + struct apple_atcphy p; + memset(dp_write_masks, 0, sizeof(dp_write_masks)); + seed_usb(&p, 1); + for (unsigned int rate = 0; rate < 4; rate++) { + track_dp_writes = true; + assert(j700_dp_start(&p, 1, rate, false) == 0); + j700_dp_equalize(&p, 0, true, 0x123); + j700_dp_equalize(&p, 0, false, 0x123); + assert(j700_dp_stop(&p) == 0); + track_dp_writes = false; + } + unsigned int entries = 0; + for (unsigned int offset = 0; offset < sizeof(dp_write_masks); offset += 4) { + if (!((offset >= 0x9000 && offset <= 0xdfff) || offset == 8 || + offset == 0xc || offset == 0x10c || offset == 0x200)) continue; + u32 expected = 0; + for (unsigned int i = 0; i < ARRAY_SIZE(j700_usb_fields); i++) + if (j700_usb_fields[i].offset == offset) expected = j700_usb_fields[i].mask; + assert(dp_write_masks[offset / 4] == expected); + entries += !!expected; + } + assert(entries == 27); + puts("PASS: 27 masks equal the actual public DP start/voltage/stop write union at every rate"); +} +static void masked_roundtrip(void) +{ + const struct phy_configure_opts_dp rate = { + .set_rate=true, .set_lanes=true, .link_rate=8100, .lanes=2, + }; + bool failure=false; + for (unsigned int first=0; first<2; first++) { + struct apple_atcphy p; + seed_usb(&p, first); + unsigned int old_dp=!first; +#ifdef PROPOSAL + assert(p.dp_usb_context_valid & BIT(old_dp)); + assert(p.dp_usb_context[old_dp][1] == 0); + assert(p.dp_usb_context[old_dp][3] == (0xeU << (4 * old_dp))); + assert(((registers[0x200 / 4] >> (4 * old_dp)) & 0xf) == 1); +#endif + u32 saved[27], outside[27]; + for (unsigned int i=0;i<27;i++) { + u32 offset,mask; + j700_usb_field(i,old_dp,&offset,&mask); + saved[i]=core_read32(&p,offset)&mask; + } + /* Power/receiver were saved before route idle, not at this post-route sample. */ + saved[1]=0; saved[3]=0xeU<<(4*old_dp); + assert(j700_dp_configure(&p,&rate)==0); + unsigned int edges_before=writes; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + assert(writes==edges_before); /* same mode never resets or recaptures */ + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + for(unsigned int i=0;i<27;i++) { + u32 offset,mask;j700_usb_field(i,old_dp,&offset,&mask); + u32 other_offset,other_mask; j700_usb_field(i,!old_dp,&other_offset,&other_mask); + outside[i]=registers[offset/4]&~(mask | (i<4 ? other_mask : 0)); + } + orient(&p,!first); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + unsigned int mismatch=0; + for(unsigned int i=0;i<27;i++) { + u32 offset,mask;j700_usb_field(i,old_dp,&offset,&mask); + if((registers[offset/4]&mask)!=saved[i]) mismatch++; + u32 other_offset,other_mask; j700_usb_field(i,!old_dp,&other_offset,&other_mask); + assert((registers[offset/4]&~(mask | (i<4 ? other_mask : 0)))==outside[i]); + } + printf("first=%u former_dp_pma=%u masked_mismatches=%u receiver=%#x power=%#x\n", + first,old_dp,mismatch,registers[0xc/4],registers[0x200/4]); + failure |= mismatch!=0; + assert(!usb2_writes&&!reset_writes); + assert(j700_dp_configure(&p,&rate)==0); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + orient(&p,first); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + assert(!p.mode_dirty&&p.pipe_state==ATCPHY_PIPEHANDLER_STATE_USB3); + } + fflush(stdout);assert(!failure); +} +static void restore_controls(void) +{ +#ifdef PROPOSAL + struct apple_atcphy p; + const struct phy_configure_opts_dp rate={.set_rate=true,.set_lanes=true,.link_rate=8100,.lanes=2}; + seed_usb(&p,1); + assert(j700_dp_configure(&p,&rate)==0); + unsigned int valid=p.dp_usb_context_valid,dirty=p.dp_usb_context_dirty; + u32 snapshot[2][27];memcpy(snapshot,p.dp_usb_context,sizeof(snapshot)); + stuck_ack=true; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==-ETIMEDOUT); + assert(p.dp_owned_pmas==1&&p.dp_usb_context_valid==valid&&p.dp_usb_context_dirty==dirty); + assert(!memcmp(snapshot,p.dp_usb_context,sizeof(snapshot))); + recover(); + fail_caller="j700_field_update"; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==-ETIMEDOUT); + assert(p.dp_owned_pmas==1&&p.dp_usb_context_valid==valid); + assert(!memcmp(snapshot,p.dp_usb_context,sizeof(snapshot))); + recover();assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + orient(&p,0); + assert(j700_usb_capture(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + p.dp_owned_pmas=1; + unsigned int before=writes; + assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==-EBUSY&&writes==before); + p.dp_owned_pmas=0;p.dp_idle_pmas=1; + assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==-EBUSY&&writes==before); + p.dp_idle_pmas=0;p.pipe_state=ATCPHY_PIPEHANDLER_STATE_USB3; + assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==-EBUSY&&writes==before); + p.pipe_state=ATCPHY_PIPEHANDLER_STATE_DUMMY; + fail_caller="j700_usb_field_update"; + assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==-ETIMEDOUT); + assert(p.dp_usb_context_valid&1); + assert(p.dp_usb_context_dirty&1); + recover();assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + assert(!(p.dp_usb_context_valid&1)&&!(p.dp_usb_context_dirty&1)); + seed_usb(&p,1); + assert(j700_dp_configure(&p,&rate)==0); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + orient(&p,0); + u32 old_context[27];memcpy(old_context,p.dp_usb_context[0],sizeof(old_context)); + fail_caller="j700_usb_field_update"; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==-ETIMEDOUT); + assert(p.mode_dirty&&p.pipe_state==ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert((p.dp_usb_context_valid&1)&&(p.dp_usb_context_dirty&1)); + assert(!memcmp(old_context,p.dp_usb_context[0],sizeof(old_context))); + before=writes; + assert(j700_dp_configure(&p,&rate)==-EBUSY&&writes==before); + recover();assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + assert(!p.mode_dirty&&!(p.dp_usb_context_valid&1)); + seed_usb(&p,1); + assert(j700_dp_configure(&p,&rate)==0); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + orient(&p,0); + blocked_offset=0x9244; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==-EIO); + assert(p.mode_dirty&&p.pipe_state==ATCPHY_PIPEHANDLER_STATE_DUMMY); + assert((p.dp_usb_context_valid&1)&&(p.dp_usb_context_dirty&1)); + before=writes; + assert(j700_dp_configure(&p,&rate)==-EBUSY&&writes==before); + blocked_offset=0; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0&&!p.mode_dirty); + puts("PASS: actual mux static/power restore failures keep dirty route/context, block training and retry"); + seed_usb(&p,1); + assert(j700_dp_configure(&p,&rate)==0); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); + orient(&p,0); + registers[ACIOPHY_BIST_MAC_USBPHY_STATUS2/4]=BIT(7); + bist_status_stuck=true; bist_faults=0; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==-ETIMEDOUT); + assert(p.mode_dirty && p.pipe_pending); + assert(bist_faults==2 && (registers[ACIOPHY_BIST_MAC_USBPHY_STATUS2/4]&BIT(7))); + assert((registers[0x200/4]&0xf)==0xe && !(registers[0xc/4]&0x44)); + stuck_ack=true; + before=writes; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==-ETIMEDOUT && writes==before); + puts("PASS: restoration never clears simulated BIST/ACK faults or forces readiness"); + for(unsigned int fixed=0;fixed<2;fixed++) { + seed_usb(&p,0);p.fixed_usb2=fixed;hw.has_t8140_dp=false; + before=writes; + assert(j700_usb_capture(&p,APPLE_ATCPHY_MODE_USB3_DP)==0); + assert(j700_usb_restore(&p,APPLE_ATCPHY_MODE_USB3_DP)==0&&writes==before); + hw.has_t8140_dp=true; + } + puts("PASS: park/DP-idle failure retention, immutable snapshot, live/pending-owner restore guards, power readback failure/retry, legacy exclusions"); +#endif +} + +static void inactive_host_gates(void) +{ +#ifdef PROPOSAL + const struct phy_configure_opts_dp rate = { + .set_rate=true, .set_lanes=true, .link_rate=8100, .lanes=2, + }; + unsigned int violations = 0; + for (unsigned int backend = ATCPHY_PIPEHANDLER_STATE_USB3; + backend <= ATCPHY_PIPEHANDLER_STATE_USB4; backend++) { + for (unsigned int entry = 0; entry < 3; entry++) { + struct apple_atcphy p; + seed_usb(&p, 1); + assert(j700_dp_configure(&p, &rate) == 0); + assert(select_mode(&p, APPLE_ATCPHY_MODE_USB2) == 0); + orient(&p, 0); + p.host_active = false; + p.pipe_state = backend; + unsigned int before_writes = writes, before_reads = mmio_reads; + unsigned int valid = p.dp_usb_context_valid, dirty = p.dp_usb_context_dirty; + u32 snapshot[2][27]; memcpy(snapshot, p.dp_usb_context, sizeof(snapshot)); + int ret = entry == 0 ? j700_usb_capture(&p, APPLE_ATCPHY_MODE_USB3_DP) : + entry == 1 ? j700_usb_restore(&p, APPLE_ATCPHY_MODE_USB3_DP) : + atcphy_configure(&p, APPLE_ATCPHY_MODE_USB3_DP); + bool blocked = ret == -EBUSY && writes == before_writes && + mmio_reads == before_reads && valid == p.dp_usb_context_valid && + dirty == p.dp_usb_context_dirty && + !memcmp(snapshot, p.dp_usb_context, sizeof(snapshot)); + printf("inactive-host backend=%u entry=%u ret=%d new_writes=%u new_reads=%u blocked=%u\n", + backend, entry, ret, writes-before_writes, mmio_reads-before_reads, blocked); + violations += !blocked; + } + } + fflush(stdout); assert(!violations); + puts("PASS: six inactive-host/live-PIPE entry gates retain snapshots and perform zero MMIO"); +#endif +} +static void failure_diagnostics(void) +{ +#ifdef PROPOSAL + struct apple_atcphy p; + const struct phy_configure_opts_dp rate={.set_rate=true,.set_lanes=true,.link_rate=8100,.lanes=2}; + const char *stages[] = { "stage=static", "stage=power", "stage=receiver" }; + for (unsigned int stage=0; stage<3; stage++) { + seed_usb(&p,1); assert(j700_dp_configure(&p,&rate)==0); + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB2)==0); orient(&p,0); + diagnostic_records=0; last_diagnostic[0]=0; + blocked_offset=stage==0 ? 0x9244 : stage==1 ? 0x200 : 0xc; + int ret=select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP); + assert(ret==(stage==0 ? -EIO : -ETIMEDOUT)); + assert(diagnostic_records==1 && strstr(last_diagnostic,stages[stage])); + assert(strstr(last_diagnostic,"pma=0") && strstr(last_diagnostic,"reg=")); + assert(strstr(last_diagnostic,"mask=") && strstr(last_diagnostic,"expected=")); + assert(strstr(last_diagnostic,"observed=")); + assert(p.mode_dirty && (p.dp_usb_context_valid&1) && (p.dp_usb_context_dirty&1)); + blocked_offset=0; + } + puts("PASS: static/power/receiver errors preserve state and report stage/PMA/register/mask/expected/final observed"); +#endif +} +int main(int argc,char **argv) +{ + if(argc>1 && !strcmp(argv[1],"inactive-host")) { inactive_host_gates(); return 0; } + if(argc>1)return original_fixture_main(argc,argv); + mask_coverage();masked_roundtrip();restore_controls();inactive_host_gates();failure_diagnostics(); + puts("PASS: masked restoration roundtrips; no BIST readiness or USB speed is modeled"); + return 0; +} +''' + logger = r''' +#include +static unsigned int diagnostic_records; +static char last_diagnostic[512]; +static void record_debug(const char *format, ...) +{ + if (!strstr(format, "USB restore stage=")) return; + va_list args; va_start(args,format); + vsnprintf(last_diagnostic,sizeof(last_diagnostic),format,args); + va_end(args); diagnostic_records++; + fputs(last_diagnostic,stdout); +} +#undef dev_dbg +#define dev_dbg(dev, format, ...) record_debug(format, ##__VA_ARGS__) +''' + return ns["prefix"] + logger + "#define PROPOSAL\n" + production + ns["suffix"].replace("int main(", "int original_fixture_main(", 1) + controls + +def pipe(root): + + source = (root / 'drivers/phy/apple/atc.c').read_text() + dp = (root / 'drivers/phy/apple/atc-j700-dp.h').read_text() + + + def block(marker): + start = source.index(marker) + opening = source.index('{', start) + end, depth = opening + 1, 1 + while depth: + depth += (source[end] == '{') - (source[end] == '}') + end += 1 + return source[start:end] + + + def function(name): + if name == 'atcphy_mux_set' and 'static int atcphy_mux_set_locked(' in source: + marker = re.search(r'^static [^\n]+\batcphy_mux_set_locked\(', source, re.M).group() + return block(marker).replace('atcphy_mux_set_locked(', 'atcphy_mux_set(', 1) + return block(re.search(r'^static [^\n]+\b' + name + r'\(', source, re.M).group()) + + + parts = re.findall(r'^#define [^\n]*(?:\\\n[^\n]*)*', source, re.M) + for name in ('atcphy_generation', 'atcphy_lane_mode', 'atcphy_pipehandler_state', 'atcphy_mode'): + parts.append(block('enum ' + name + ' {') + ';') + for name in ('atcphy_mode_configuration', 'atcphy_hw'): + parts.append(block('struct ' + name + ' {') + ';') + parts.append(r''' +struct apple_atcphy { + const struct atcphy_hw *hw; + struct { unsigned char *core, *pipehandler; } regs; + void *dev; + int lock; + bool fixed_usb2, host_active, ss_tunables, dp_tunables, mode_dirty; + unsigned int dp_owned_pmas, dp_idle_pmas; + unsigned int dp_usb_context_valid, dp_usb_context_dirty; + u32 dp_usb_context[2][27]; + bool pipe_pending, usb_off_pending, usb_off_drained; + bool swap_lanes; + enum atcphy_mode mode, typec_mode; + enum atcphy_pipehandler_state pipe_state; +}; +''') + start = source.index('static const struct {\n\tconst struct atcphy_mode_configuration normal;') + parts.append(source[start:source.index('\n};', start) + 3]) + parts.append(r''' +static u32 pipe_regs[0x1000 / 4], core_regs[0x30000 / 4]; +static unsigned int polls, fail_poll, mux_writes, lane_calls; +static bool stuck_ack; +static u32 readl(void *address) { return *(u32 *)address; } +static void mask32(void *address, u32 mask, u32 value) +{ + *(u32 *)address = (*(u32 *)address & ~mask) | value; + if (address == (void *)&pipe_regs[PIPEHANDLER_LOCK_REQ / 4] && !stuck_ack) + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = *(u32 *)address & PIPEHANDLER_LOCK_EN; + if (address == (void *)&pipe_regs[PIPEHANDLER_MUX_CTRL / 4]) + mux_writes++; +} +static void set32(void *p, u32 v) { mask32(p, 0, v); } +static void clear32(void *p, u32 v) { mask32(p, v, 0); } +static void writel(u32 v, void *p) { mask32(p, ~0U, v); } +static void core_mask32(struct apple_atcphy *p, u32 o, u32 m, u32 v) +{ assert(o < sizeof(core_regs) && o % 4 == 0); mask32(p->regs.core + o, m, v); } +static void core_set32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, 0, v); } +static void core_clear32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, v, 0); } +#define readl_poll_timeout_atomic readl_poll_timeout +#define readl_poll_timeout(address, value, condition, interval, timeout) \ + ({ (value) = readl(address); polls++; \ + polls == fail_poll || !(condition) ? -ETIMEDOUT : 0; }) +/* Lane configuration is outside this test: count calls and commit its result. */ +/* Observation is read-only and separately tested; retain production caller control flow. */ +static int atcphy_configure(struct apple_atcphy *p, enum atcphy_mode mode); +''') + for name in ('core_read32', 'atcphy_get_mode_config'): + parts.append(function(name)) + # Actual restoration functions and gates; analog lane setup stays outside this PIPE unit. + parts.append(dp[dp.index('/* Masks are the union'):dp.index('/* Retry only the idle operations')]) + parts.append(r''' +static int atcphy_configure(struct apple_atcphy *p, enum atcphy_mode mode) +{ + int ret; + lane_calls++; p->mode_dirty = true; + ret = j700_usb_capture(p, mode); + if (!ret) ret = j700_usb_restore(p, mode); + if (ret) return ret; + p->mode = mode; p->mode_dirty = false; return 0; +} +''') + for name in ('atcphy_pipehandler_lock', 'atcphy_pipehandler_unlock', 'atcphy_pipehandler_check', + 'atcphy_pipehandler_set_mux', 'atcphy_clear_nonselected_phy_reset', + 'atcphy_configure_pipehandler_usb3_host_t8122', + 'atcphy_configure_pipehandler_usb3_host_t8103', + 'atcphy_configure_pipehandler_usb3', 'atcphy_configure_pipehandler_usb4', + 'atcphy_enable_dummy_phy', 'atcphy_configure_pipehandler_dummy', + 'atcphy_configure_pipehandler', 'atcphy_park_pipehandler', + 'atcphy_usb3_set_mode', 'atcphy_mux_set'): + parts.append(function(name)) + production = '\n'.join(parts) + prefix = r''' +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +typedef uint16_t u16; +#define BIT(n) (1U << (n)) +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define GENMASK(h, l) ((~0U << (l)) & (~0U >> (31 - (h)))) +#define FIELD_PREP(m, v) (((u32)(v) << __builtin_ctz(m)) & (m)) +#define FIELD_GET(m, v) (((v) & (m)) >> __builtin_ctz(m)) +#define field_prep FIELD_PREP +#define dev_dbg(...) ((void)0) +#define dev_warn(...) ((void)0) +#define dev_err(...) ((void)0) +#define lockdep_assert_held(...) ((void)0) +#define WARN_ON_ONCE(v) assert(!(v)) +#define guard(kind) (void) +#define fallthrough __attribute__((fallthrough)) +#define usleep_range(...) ((void)0) +#define udelay(n) ((void)(n)) +enum { TYPEC_STATE_SAFE, TYPEC_STATE_USB, TYPEC_MODE_USB4, + TYPEC_DP_STATE_C, TYPEC_DP_STATE_D, TYPEC_DP_STATE_E }; +enum { USB_TYPEC_TBT_SID = 0x8087, USB_TYPEC_DP_SID = 0xff01 }; +#define EUDO_USB_MODE_MASK 3U +enum { EUDO_USB_MODE_USB2, EUDO_USB_MODE_USB3, EUDO_USB_MODE_USB4 }; +enum phy_mode { PHY_MODE_USB_HOST, PHY_MODE_USB_DEVICE, PHY_MODE_INVALID }; +struct enter_usb_data { u32 eudo; }; +struct typec_altmode { unsigned int svid; }; +struct typec_mux_state { unsigned long mode; struct typec_altmode *alt; void *data; }; +struct typec_mux_dev { struct apple_atcphy *data; }; +struct phy { struct apple_atcphy *data; }; +#define typec_mux_get_drvdata(mux) ((mux)->data) +#define phy_get_drvdata(phy) ((phy)->data) +''' + suffix = r''' +static struct atcphy_hw hw = { + .gen = ATCPHY_GENERATION_T8122, .has_t8140_dp = true, + .pipe_mux_clk = PIPEHANDLER_MUX_CTRL_CLK_T8103, + .pipe_mux_data = PIPEHANDLER_MUX_CTRL_DATA_T8103, +}; +static void initial(struct apple_atcphy *p, enum atcphy_mode mode, + enum atcphy_pipehandler_state backend) +{ + memset(pipe_regs, 0, sizeof(pipe_regs)); + memset(core_regs, 0, sizeof(core_regs)); + core_regs[ACIOPHY_TOP_PHY_STAT / 4] = ACIOPHY_TOP_PHY_STAT_LN0_UNK0; + *p = (struct apple_atcphy) { + .hw = &hw, .regs.core = (unsigned char *)core_regs, + .regs.pipehandler = (unsigned char *)pipe_regs, + .mode = mode, .typec_mode = mode, .pipe_state = backend, + .ss_tunables = true, .dp_tunables = true, + }; + polls = fail_poll = mux_writes = lane_calls = 0; + stuck_ack = false; +} +static int request(struct apple_atcphy *p, enum atcphy_mode mode) +{ + struct typec_mux_dev mux = { p }; + struct typec_altmode dp = { .svid = USB_TYPEC_DP_SID }; + struct enter_usb_data data = { .eudo = EUDO_USB_MODE_USB4 }; + struct typec_mux_state state = { .mode = TYPEC_STATE_SAFE }; + if (mode == APPLE_ATCPHY_MODE_USB3) + state.mode = TYPEC_STATE_USB; + else if (mode == APPLE_ATCPHY_MODE_USB3_DP || mode == APPLE_ATCPHY_MODE_DP) { + state.mode = mode == APPLE_ATCPHY_MODE_DP ? TYPEC_DP_STATE_C : TYPEC_DP_STATE_D; + state.alt = &dp; + } else if (mode == APPLE_ATCPHY_MODE_USB4) { + state.mode = TYPEC_MODE_USB4; + state.data = &data; + } + return atcphy_mux_set(&mux, &state); +} +static void backend_failure(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + for (enum atcphy_mode mode = APPLE_ATCPHY_MODE_OFF; mode <= APPLE_ATCPHY_MODE_DP; mode++) { + enum atcphy_pipehandler_state target = atcphy_modes[mode].pipehandler_state; + enum atcphy_pipehandler_state old = target == ATCPHY_PIPEHANDLER_STATE_DUMMY ? + ATCPHY_PIPEHANDLER_STATE_USB3 : ATCPHY_PIPEHANDLER_STATE_DUMMY; + for (unsigned int host = 0; host < 2; host++) { + initial(&p, mode, old); + /* A previous lock ACK cannot be cleared: fatal before mux writes. */ + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + stuck_ack = true; + assert(atcphy_configure_pipehandler(&p, host) == -ETIMEDOUT); + assert(p.pipe_state == old && mux_writes == 0); + stuck_ack = false; + assert(atcphy_configure_pipehandler(&p, host) == 0); + assert(p.pipe_state == target && mux_writes == 3); + const unsigned int data[] = { PIPEHANDLER_MUX_CTRL_DATA_DUMMY, + PIPEHANDLER_MUX_CTRL_DATA_USB3, PIPEHANDLER_MUX_CTRL_DATA_USB4 }; + const unsigned int clk[] = { PIPEHANDLER_MUX_CTRL_CLK_DUMMY, + PIPEHANDLER_MUX_CTRL_CLK_USB3, PIPEHANDLER_MUX_CTRL_CLK_USB4 }; + u32 mux = pipe_regs[PIPEHANDLER_MUX_CTRL / 4]; + assert(FIELD_GET(hw.pipe_mux_data, mux) == data[target]); + assert(FIELD_GET(hw.pipe_mux_clk, mux) == clk[target]); + cases++; + } + } + printf("PASS: %u backend/role check-failure and retry cases\n", cases); +} +static void host_lock_failure(void) +{ + struct apple_atcphy p; + for (unsigned int gen = 0; gen < 2; gen++) { + hw.gen = gen ? ATCPHY_GENERATION_T8122 : ATCPHY_GENERATION_T8103; + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_DUMMY); + struct phy phy = { &p }; + stuck_ack = true; + assert(atcphy_usb3_set_mode(&phy, PHY_MODE_USB_HOST, 0) == -ETIMEDOUT); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && mux_writes == 0); + assert(!(pipe_regs[PIPEHANDLER_LOCK_REQ / 4] & PIPEHANDLER_LOCK_EN)); + stuck_ack = false; + assert(atcphy_usb3_set_mode(&phy, PHY_MODE_USB_HOST, 0) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && mux_writes == 6); + unsigned int before = polls; + assert(atcphy_usb3_set_mode(&phy, PHY_MODE_USB_HOST, 0) == 0); + assert(polls == before && mux_writes == 6); + } + puts("PASS: both USB3 host BIST generations retain fatal lock ACK and retry"); +} +static void mux_retry(void) +{ + struct apple_atcphy p; + const enum atcphy_mode modes[] = { APPLE_ATCPHY_MODE_USB3, APPLE_ATCPHY_MODE_USB3_DP }; + for (unsigned int m = 0; m < 2; m++) { + initial(&p, APPLE_ATCPHY_MODE_USB2, ATCPHY_PIPEHANDLER_STATE_DUMMY); + p.fixed_usb2 = p.host_active = true; + stuck_ack = true; + assert(request(&p, modes[m]) == -ETIMEDOUT); + assert(p.mode == modes[m] && !p.mode_dirty && lane_calls == 1); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && mux_writes == 0); + p.dp_owned_pmas = 3; /* PIPE retry must not reconfigure an owned DP mode. */ + assert(request(&p, modes[m]) == -ETIMEDOUT); + assert(lane_calls == 1 && p.dp_owned_pmas == 3 && mux_writes == 0); + stuck_ack = false; + assert(request(&p, modes[m]) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && lane_calls == 1); + assert(p.dp_owned_pmas == 3 && mux_writes == 6); + unsigned int before = polls; + assert(request(&p, modes[m]) == 0 && polls == before && lane_calls == 1); + } + puts("PASS: fixed-hub same-mode retry preserves lanes and DP ownership"); +} +static void same_mode_pending(void) +{ + struct apple_atcphy p; + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_DUMMY); + p.fixed_usb2 = p.host_active = true; + stuck_ack = true; + assert(request(&p, APPLE_ATCPHY_MODE_USB3) == -ETIMEDOUT); + assert(lane_calls == 0 && mux_writes == 0); + stuck_ack = false; + assert(request(&p, APPLE_ATCPHY_MODE_USB3) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && lane_calls == 0); + puts("PASS: same-mode pending PIPE setup independently retries"); +} +static void parking_and_guards(void) +{ + struct apple_atcphy p; + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_USB3); + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + stuck_ack = true; + assert(atcphy_park_pipehandler(&p) == -ETIMEDOUT); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && mux_writes == 0); + stuck_ack = false; + assert(atcphy_park_pipehandler(&p) == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY && mux_writes == 3); + assert(atcphy_park_pipehandler(&p) == 0); + assert(mux_writes == 3); + initial(&p, APPLE_ATCPHY_MODE_USB2, ATCPHY_PIPEHANDLER_STATE_DUMMY); + p.fixed_usb2 = p.host_active = true; + assert(request(&p, APPLE_ATCPHY_MODE_OFF) == 0); + assert(p.mode == APPLE_ATCPHY_MODE_USB2 && lane_calls == 0 && mux_writes == 0); + p.mode_dirty = true; + assert(request(&p, APPLE_ATCPHY_MODE_OFF) == 0 && lane_calls == 1 && !p.mode_dirty); + /* Inactive hosts and ordinary ports retain their deferred PIPE setup. */ + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_DUMMY); + p.fixed_usb2 = true; + assert(request(&p, APPLE_ATCPHY_MODE_USB3) == 0 && mux_writes == 0); + p.fixed_usb2 = false; + p.host_active = true; + assert(request(&p, APPLE_ATCPHY_MODE_USB3) == 0 && mux_writes == 0); + /* Capability rejection must still precede mode configuration. */ + initial(&p, APPLE_ATCPHY_MODE_USB2, ATCPHY_PIPEHANDLER_STATE_DUMMY); + p.fixed_usb2 = p.host_active = true; + p.ss_tunables = false; + assert(request(&p, APPLE_ATCPHY_MODE_USB3) == -EOPNOTSUPP && lane_calls == 0); + p.ss_tunables = true; + p.dp_tunables = false; + assert(request(&p, APPLE_ATCPHY_MODE_USB3_DP) == -EOPNOTSUPP && lane_calls == 0); + assert(request(&p, APPLE_ATCPHY_MODE_USB4) == -EOPNOTSUPP && lane_calls == 0); + puts("PASS: parking retry, SAFE fixed USB2, dirty-mode and capability guards"); +} +static void requested_lock(void) +{ + struct apple_atcphy p; + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_DUMMY); + pipe_regs[PIPEHANDLER_LOCK_REQ / 4] = PIPEHANDLER_LOCK_EN; + stuck_ack = true; + assert(atcphy_pipehandler_lock(&p) == -ETIMEDOUT && polls == 1); + assert(!(pipe_regs[PIPEHANDLER_LOCK_REQ / 4] & PIPEHANDLER_LOCK_EN)); + pipe_regs[PIPEHANDLER_LOCK_REQ / 4] = PIPEHANDLER_LOCK_EN; + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + assert(atcphy_pipehandler_lock(&p) == 0 && polls == 2); + puts("PASS: an existing LOCK request must still have actual ACK before success"); +} +static void live_policy_scope(void) +{ + struct apple_atcphy p; + for (unsigned int t8140 = 0; t8140 < 2; t8140++) { + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int active = 0; active < 2; active++) { + hw.has_t8140_dp = t8140; + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_USB3); + p.fixed_usb2 = fixed; + p.host_active = active; + stuck_ack = true; /* Request exists, but ACK never asserts. */ + int ret = atcphy_park_pipehandler(&p); + if (t8140 && fixed && active) { + assert(ret == -ETIMEDOUT && p.mode_dirty && mux_writes == 0); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3); + assert(!pipe_regs[PIPEHANDLER_NONSELECTED_OVERRIDE / 4]); + } else { + assert(ret == 0 && !p.mode_dirty && mux_writes == 3); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_DUMMY); + } + /* Entry ACK release is fatal everywhere, but dirties only live T8140 parking. */ + initial(&p, APPLE_ATCPHY_MODE_USB3, ATCPHY_PIPEHANDLER_STATE_USB3); + p.fixed_usb2 = fixed; + p.host_active = active; + pipe_regs[PIPEHANDLER_LOCK_ACK / 4] = PIPEHANDLER_LOCK_EN; + stuck_ack = true; + assert(atcphy_park_pipehandler(&p) == -ETIMEDOUT); + assert(p.mode_dirty == (t8140 && fixed && active)); + assert(p.pipe_state == ATCPHY_PIPEHANDLER_STATE_USB3 && mux_writes == 0); + } + } + } + puts("PASS: eight live/inactive, fixed/ordinary and T8140/legacy LOCK and entry-ACK policy controls"); +} +int main(int argc, char **argv) +{ + assert(argc == 2); + if (!strcmp(argv[1], "backend")) backend_failure(); + else if (!strcmp(argv[1], "host")) host_lock_failure(); + else if (!strcmp(argv[1], "mux")) mux_retry(); + else if (!strcmp(argv[1], "same")) same_mode_pending(); + else if (!strcmp(argv[1], "park")) parking_and_guards(); + else if (!strcmp(argv[1], "requested")) requested_lock(); + else if (!strcmp(argv[1], "scope")) live_policy_scope(); + else assert(false); + return 0; +} +''' + return prefix + production + suffix + +def unwind(root): + + atc = (root / 'drivers/phy/apple/atc.c').read_text() + dp = (root / 'drivers/phy/apple/atc-j700-dp.h').read_text() + + + def function(source, name): + if name == 'atcphy_mux_set' and 'static int atcphy_mux_set_locked(' in source: + marker = re.search(r'^static [^\n]+\batcphy_mux_set_locked\(', source, re.M).group() + return block(source, marker).replace('atcphy_mux_set_locked(', 'atcphy_mux_set(', 1) + return block(source, re.search(r'^static [^\n]+\b' + name + r'\(', source, re.M).group()) + + + # Use the production mode table and wrappers too, so SAFE mapping and fast paths + # are tested rather than mirrored by a second implementation. + parts = re.findall(r'^#define [^\n]*(?:\\\n[^\n]*)*', atc, re.M) + for name in ('atcphy_generation', 'atcphy_lane_mode', 'atcphy_pipehandler_state', 'atcphy_mode', + 'atcphy_dp_link_rate'): + parts.append(block(atc, 'enum ' + name + ' {') + ';') + for name in ('atcphy_mode_configuration', 'atcphy_hw'): + parts.append(block(atc, 'struct ' + name + ' {') + ';') + parts.append(r''' +struct apple_atcphy { + const struct atcphy_hw *hw; + struct { unsigned char *core, *lpdptx; } regs; + void *dev; + int lock, dp_link_rate; + bool fixed_usb2, host_active, swap_lanes, ss_tunables, dp_tunables; + bool aux_power_requested, mode_dirty, dp_spread_spectrum, dp_presets_valid[5]; + u32 dp_presets[5][16]; + unsigned int dp_idle_pmas, dp_owned_pmas; + unsigned int dp_usb_context_valid, dp_usb_context_dirty; + u32 dp_usb_context[2][27]; + bool pipe_pending, usb_off_pending, usb_off_drained; + enum atcphy_mode mode, typec_mode; + enum atcphy_pipehandler_state pipe_state; +}; +''') + start = atc.index('static const struct {\n\tconst struct atcphy_mode_configuration normal;') + parts.append(atc[start:atc.index('\n};', start) + 3]) + parts.append(r''' +struct phy_configure_opts_dp { + unsigned int link_rate, lanes, voltage[4], pre[4]; + bool set_rate, set_lanes, set_voltages, ssc; +}; +static u32 registers[0x30000 / 4]; +static unsigned int polls, fail_poll, writes, usb2_on; +static u32 readl(void *address) { return *(u32 *)address; } +static void core_mask32(struct apple_atcphy *p, u32 offset, u32 mask, u32 value) +{ + (void)p; + assert(offset < sizeof(registers) && offset % 4 == 0); + registers[offset / 4] = (registers[offset / 4] & ~mask) | value; + writes++; + if (offset == ATCPHY_POWER_CTRL) + registers[ATCPHY_POWER_STAT / 4] = registers[offset / 4]; + if (offset == 0x2000) { + if (value == 0x10000001) + registers[offset / 4] |= BIT(1); + else { + registers[offset / 4] &= ~BIT(1); + registers[0x7034 / 4] &= ~BIT(3); + } + } +} +static void core_set32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, 0, v); } +static void core_clear32(struct apple_atcphy *p, u32 o, u32 v) { core_mask32(p, o, v, 0); } +static void writel(u32 value, void *address) { *(u32 *)address = value; } +static void mask32(void *p, u32 m, u32 v) { * (u32 *)p = (*(u32 *)p & ~m) | v; } +static void set32(void *p, u32 v) { mask32(p, 0, v); } +static void clear32(void *p, u32 v) { mask32(p, v, 0); } +#define readl_poll_timeout_atomic(address, value, condition, interval, timeout) \ + ({ (value) = readl(address); polls++; \ + polls == fail_poll || !(condition) ? -ETIMEDOUT : 0; }) +#define readl_poll_timeout readl_poll_timeout_atomic +static void atcphy_usb2_power_on(struct apple_atcphy *p) { (void)p; usb2_on++; } +static void atcphy_apply_tunables(struct apple_atcphy *p, enum atcphy_mode mode) +{ (void)p; (void)mode; } +static void atc_tunnel_stop_t8103(struct apple_atcphy *p) { (void)p; } +static void atc_tunnel_restore(struct apple_atcphy *p) { (void)p; } +static int atcphy_park_pipehandler(struct apple_atcphy *p) +{ p->pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY; return 0; } +/* Observation is read-only and separately tested; retain production caller control flow. */ +static int atcphy_configure_pipehandler(struct apple_atcphy *p, bool host) +{ (void)host; p->pipe_state = atcphy_modes[p->mode].pipehandler_state; return 0; } +''') + for name in ('core_read32', 'atcphy_get_mode_config', 'j700_field_update', 'j700_pma_idle'): + parts.append(function(atc, name)) + parts.append(dp) + for name in ('atcphy_configure_lanes', 'j700_aux_enable', 'j700_aux_disable', + 'atcphy_enable_dp_aux', 'atcphy_disable_dp_aux', 'atcphy_power_off', + 'atcphy_power_on', 'atcphy_configure', 'atcphy_mux_set'): + parts.append(function(atc, name)) + production = '\n'.join(parts) + prefix = r''' +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +typedef uint16_t u16; +#define BIT(n) (1U << (n)) +#define GENMASK(h, l) ((~0U << (l)) & (~0U >> (31 - (h)))) +#define FIELD_PREP(m, v) (((u32)(v) << __builtin_ctz(m)) & (m)) +#define FIELD_GET(m, v) (((v) & (m)) >> __builtin_ctz(m)) +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define hweight32(v) __builtin_popcount(v) +#define dev_info_once(...) ((void)0) +#define dev_dbg(...) ((void)0) +#define dev_warn(...) ((void)0) +static void report(void *dev, const char *fmt, ...) { (void)dev; (void)fmt; } +#define dev_err report +#define lockdep_assert_held(...) ((void)0) +#define WARN_ON_ONCE(v) assert(!(v)) +#define guard(kind) (void) +#define fallthrough __attribute__((fallthrough)) +#define ndelay(n) ((void)(n)) +#define udelay(n) ((void)(n)) +#define usleep_range(a, b) ((void)(a), (void)(b)) +enum { TYPEC_STATE_SAFE, TYPEC_STATE_USB, TYPEC_MODE_USB4, + TYPEC_DP_STATE_C, TYPEC_DP_STATE_D, TYPEC_DP_STATE_E }; +enum { USB_TYPEC_TBT_SID = 0x8087, USB_TYPEC_DP_SID = 0xff01 }; +#define EUDO_USB_MODE_MASK 3U +enum { EUDO_USB_MODE_USB2, EUDO_USB_MODE_USB3, EUDO_USB_MODE_USB4 }; +struct enter_usb_data { u32 eudo; }; +struct typec_altmode { unsigned int svid; }; +struct typec_mux_state { unsigned long mode; struct typec_altmode *alt; void *data; }; +struct typec_mux_dev { struct apple_atcphy *data; }; +#define typec_mux_get_drvdata(mux) ((mux)->data) +''' + suffix = r''' +static struct atcphy_hw hw = { + .has_t8140_dp = true, .gen = ATCPHY_GENERATION_T8122, + .aciophy_lane_mode = 0x60, .aciophy_crossbar = 0x64, +}; +static void initial(struct apple_atcphy *p, bool fixed, bool swapped) +{ + memset(registers, 0, sizeof(registers)); + registers[AUS_COMMON_DIG_RCAL1 / 4] = AUS_COMMON_DIG_RCAL1_ALL_CODES_DONE; + *p = (struct apple_atcphy) { + .hw = &hw, .regs.core = (unsigned char *)registers, + .regs.lpdptx = (unsigned char *)registers, + .fixed_usb2 = fixed, .swap_lanes = swapped, + .ss_tunables = true, .dp_tunables = true, .dp_presets_valid = { true }, + .mode = fixed ? APPLE_ATCPHY_MODE_USB2 : APPLE_ATCPHY_MODE_OFF, + .pipe_state = ATCPHY_PIPEHANDLER_STATE_DUMMY, + }; + polls = fail_poll = writes = usb2_on = 0; +} +static int safe(struct apple_atcphy *p) +{ + struct typec_mux_dev mux = { p }; + struct typec_mux_state state = { .mode = TYPEC_STATE_SAFE }; + return atcphy_mux_set(&mux, &state); +} +int main(void) +{ + struct apple_atcphy p; + unsigned int cases = 0; + for (unsigned int fixed = 0; fixed < 2; fixed++) { + for (unsigned int swapped = 0; swapped < 2; swapped++) { + const enum atcphy_mode modes[] = { + APPLE_ATCPHY_MODE_DP, APPLE_ATCPHY_MODE_USB3_DP, APPLE_ATCPHY_MODE_USB3, + }; + for (unsigned int m = 0; m < ARRAY_SIZE(modes); m++) { + unsigned int aux = modes[m] != APPLE_ATCPHY_MODE_USB3 ? 4 : 0; + unsigned int idle = modes[m] == APPLE_ATCPHY_MODE_DP ? 12 : 6; + for (unsigned int fault = 3; fault <= 2 + aux + idle; fault++) { + enum atcphy_mode old = fixed ? APPLE_ATCPHY_MODE_USB2 : APPLE_ATCPHY_MODE_OFF; + initial(&p, fixed, swapped); + fail_poll = fault; + assert(atcphy_configure(&p, modes[m]) == -ETIMEDOUT); + assert(p.mode == old && p.mode_dirty); + if (fault <= 2 + aux) + assert(p.aux_power_requested && !p.dp_idle_pmas); + else + assert(p.dp_idle_pmas); + bool owned_aux = p.aux_power_requested; + unsigned int owned_idle = p.dp_idle_pmas; + polls = 0; + fail_poll = 1; + usb2_on = 0; + assert(safe(&p) == -ETIMEDOUT); + assert(p.mode == old && p.mode_dirty); + assert(p.aux_power_requested == owned_aux && p.dp_idle_pmas == owned_idle); + assert(usb2_on == 0); /* Cleanup failure must not continue into power changes. */ + polls = fail_poll = 0; + assert(safe(&p) == 0); + assert(p.mode == old && !p.mode_dirty); + assert(!p.aux_power_requested && !p.dp_idle_pmas && !p.dp_owned_pmas); + assert(usb2_on == fixed); /* Fixed-hub SAFE restores USB2, never powers it off. */ + polls = writes = 0; + assert(safe(&p) == 0 && polls == 0 && writes == 0); + cases++; + } + } + } + } + /* A failed route change must not expose the old DP mode to link training. */ + initial(&p, false, false); + p.mode = APPLE_ATCPHY_MODE_DP; + p.aux_power_requested = true; + p.dp_owned_pmas = 3; + assert(atcphy_configure(&p, APPLE_ATCPHY_MODE_USB3_DP) == 0); + unsigned int last_idle_poll = polls - 1; /* The final poll is RCAL, after route idle. */ + initial(&p, false, false); + p.mode = APPLE_ATCPHY_MODE_DP; + p.aux_power_requested = true; + p.dp_owned_pmas = 3; + fail_poll = last_idle_poll; + assert(atcphy_configure(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(p.mode == APPLE_ATCPHY_MODE_DP && p.mode_dirty && p.dp_idle_pmas); + struct phy_configure_opts_dp rate = { + .set_rate = true, .set_lanes = true, .link_rate = 8100, .lanes = 4, + }; + struct phy_configure_opts_dp training = { .set_voltages = true, .lanes = 4 }; + unsigned int writes_before = writes, idle_before = p.dp_idle_pmas; + unsigned int owned_before = p.dp_owned_pmas; + assert(j700_dp_configure(&p, &rate) == -EBUSY); + assert(j700_dp_configure(&p, &training) == -EBUSY); + assert(writes == writes_before && p.dp_idle_pmas == idle_before); + assert(p.dp_owned_pmas == owned_before && p.aux_power_requested && p.mode_dirty); + /* Explicit stop is allowed, but must leave the route cleanup obligation. */ + struct phy_configure_opts_dp stop = { .set_rate = true }; + assert(j700_dp_configure(&p, &stop) == 0); + assert(p.dp_idle_pmas == idle_before && p.mode_dirty); + fail_poll = polls = 0; + assert(safe(&p) == 0 && !p.mode_dirty && !p.dp_idle_pmas && !p.aux_power_requested); + /* If teardown itself fails, reject training without dropping live ownership. */ + initial(&p, false, false); + p.mode = APPLE_ATCPHY_MODE_DP; + p.aux_power_requested = true; + p.dp_owned_pmas = 3; + fail_poll = 1; + assert(atcphy_configure(&p, APPLE_ATCPHY_MODE_USB3_DP) == -ETIMEDOUT); + assert(p.mode_dirty && p.dp_owned_pmas == 3); + writes_before = writes; + assert(j700_dp_configure(&p, &rate) == -EBUSY); + assert(j700_dp_configure(&p, &training) == -EBUSY); + assert(writes == writes_before && p.dp_owned_pmas == 3 && p.aux_power_requested); + fail_poll = polls = 0; + assert(safe(&p) == 0 && !p.mode_dirty && !p.dp_owned_pmas && !p.aux_power_requested); + /* A cleanup failure also dirties an initially successful mode transition. */ + initial(&p, false, false); + p.mode = APPLE_ATCPHY_MODE_DP; + p.aux_power_requested = true; + fail_poll = 1; + assert(safe(&p) == -ETIMEDOUT && p.mode_dirty && p.aux_power_requested); + fail_poll = polls = 0; + assert(safe(&p) == 0 && !p.mode_dirty && !p.aux_power_requested); + /* Existing main-link ownership survives a failed cleanup of a dirty mode. */ + initial(&p, true, false); + p.mode_dirty = p.aux_power_requested = true; + p.dp_owned_pmas = 3; + fail_poll = 1; + assert(safe(&p) == -ETIMEDOUT); + assert(p.mode_dirty && p.aux_power_requested && p.dp_owned_pmas == 3); + fail_poll = polls = 0; + assert(safe(&p) == 0 && !p.mode_dirty && !p.aux_power_requested && !p.dp_owned_pmas); + /* OFF is not committed when domain sleep fails, and SAFE retries it. */ + initial(&p, false, false); + p.mode = APPLE_ATCPHY_MODE_USB2; + fail_poll = 1; + assert(safe(&p) == -ETIMEDOUT && p.mode == APPLE_ATCPHY_MODE_USB2 && p.mode_dirty); + fail_poll = polls = 0; + assert(safe(&p) == 0 && p.mode == APPLE_ATCPHY_MODE_OFF && !p.mode_dirty); + printf("PASS: %u AUX/PMA fault scenarios, failed cleanup, dirty training guard, SAFE retry, fixed USB2, OFF commit\n", cases); + return 0; +} +''' + return prefix + production + suffix + +def aux(root): + + source = root / 'drivers/phy/apple/atc.c' + text = source.read_text() + parts = [line for line in text.splitlines() if line.startswith(('#define J700_AUX_', '#define J700_PMA_'))] + for name in ('j700_field_update', 'j700_pma_idle', 'j700_aux_enable', 'j700_aux_disable'): + start = text.index('static int ' + name + '(') + begin = text.index('{', start) + depth = 1 + end = begin + 1 + while depth: + depth += (text[end] == '{') - (text[end] == '}') + end += 1 + parts.append(text[start:end]) + production = '\n'.join(parts) + prefix = r''' +#include +#include +#include +#include +#include +#include +typedef uint32_t u32; +#define BIT(n) (1U << (n)) +struct apple_atcphy { struct { unsigned char *core; } regs; bool aux_power_requested; }; +struct event { u32 offset, mask, value; } events[32]; +static u32 registers[0x18000 / 4]; +static unsigned int writes, reject; +static void core_mask32(struct apple_atcphy *phy, u32 offset, u32 mask, u32 value) +{ + assert(offset < sizeof(registers) && offset % 4 == 0 && writes < 32); + assert(!(value & ~mask)); + events[writes] = (struct event){offset, mask, value}; + writes++; + if (writes != reject) + registers[offset / 4] = (registers[offset / 4] & ~mask) | value; + (void)phy; +} +static void usleep_range(unsigned int low, unsigned int high) +{ + assert(low && high >= low); +} +#define readl_poll_timeout_atomic(address, value, condition, interval, timeout) \ + ({ (value) = *(u32 *)(address); (condition) ? 0 : -110; }) +''' + suffix = r''' +static void initial(struct apple_atcphy *phy) +{ + memset(registers, 0, sizeof(registers)); + memset(events, 0, sizeof(events)); + registers[J700_AUX_SUPPLY_CONTROL / 4] = 0x80000200; + registers[J700_AUX_ANALOG_CONTROL / 4] = 0x40000001; + *phy = (struct apple_atcphy){ .regs.core = (unsigned char *)registers }; + writes = reject = 0; +} +int main(void) +{ + struct apple_atcphy phy; + initial(&phy); + assert(j700_aux_disable(&phy) == 0 && writes == 0); + assert(j700_aux_enable(&phy) == 0 && phy.aux_power_requested && writes == 4); + assert(registers[J700_AUX_SUPPLY_CONTROL / 4] == 0x8000010f); + assert(registers[J700_AUX_ANALOG_CONTROL / 4] == 0x40000000); + assert(events[0].mask == 3 && events[1].mask == 12); + assert(events[2].mask == 0x300 && events[2].value == 0x100); + assert(events[3].offset == J700_AUX_ANALOG_CONTROL); + writes = 0; + assert(j700_aux_disable(&phy) == 0 && !phy.aux_power_requested && writes == 4); + assert(events[0].offset == J700_AUX_ANALOG_CONTROL); + assert(events[1].mask == 0x300 && events[2].mask == 8 && events[3].mask == 2); + assert(registers[J700_AUX_SUPPLY_CONTROL / 4] == 0x80000305); + assert(registers[J700_AUX_ANALOG_CONTROL / 4] == 0x40000001); + for (unsigned int fault = 1; fault <= 4; fault++) { + initial(&phy); + reject = fault; + assert(j700_aux_enable(&phy) == -110 && phy.aux_power_requested); + reject = 0; + writes = 0; + assert(j700_aux_disable(&phy) == 0 && !phy.aux_power_requested); + initial(&phy); + assert(j700_aux_enable(&phy) == 0); + writes = 0; + reject = fault; + assert(j700_aux_disable(&phy) == -110 && phy.aux_power_requested); + } + initial(&phy); + assert(j700_pma_idle(&phy, 2) == -EINVAL && writes == 0); + for (unsigned int lane = 0; lane < 2; lane++) { + initial(&phy); + registers[J700_PMA_POWER_CONTROL / 4] = 0x800000ee; + registers[J700_PMA_RECEIVER_CONTROL / 4] = 0x5000; + assert(j700_pma_idle(&phy, lane) == 0 && writes == 6); + assert(((registers[J700_PMA_POWER_CONTROL / 4] >> (4 * lane)) & 15) == 1); + assert(((registers[J700_PMA_POWER_CONTROL / 4] >> (4 * (1 - lane))) & 15) == 14); + assert(registers[J700_PMA_POWER_CONTROL / 4] & 0x80000000); + assert(registers[J700_PMA_RECEIVER_CONTROL / 4] == (0x5000U | (0x44U << lane))); + for (unsigned int fault = 1; fault <= 6; fault++) { + initial(&phy); + registers[J700_PMA_POWER_CONTROL / 4] = 0xee; + reject = fault; + assert(j700_pma_idle(&phy, lane) == -110); + } + } + puts("PASS: sequencing, unrelated bits, every failed phase, retained cleanup ownership"); + return 0; +} +''' + return prefix + production + suffix diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/pm.py b/tools/testing/selftests/drivers/gpu/apple-dcp/pm.py new file mode 100644 index 00000000000000..6125dbf312b8f8 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/pm.py @@ -0,0 +1,652 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Actual PM/HCD/mux/probe boundaries using the shared PHY extraction fixture.""" +from pathlib import Path +import os, re, subprocess +import phy +from common import block + +def ownership(root, usb_unreachable=False): + ns=phy.parking(root) + s=ns['atc'];has_pm='static int atcphy_usb3_pm_get(' in s + prod=ns['production'].replace('struct apple_atcphy {','struct apple_atcphy {\n struct device_node *np;',1) + if has_pm: prod=prod.replace('static int atcphy_mux_set(', 'static int atcphy_mux_set_locked(',1) + # Non-mux fixture entry points continue to exercise actual source. Only the new + # wrapper owns PM; the fixture's original helpers call it through this declaration. + prefix=ns['prefix']+'\nstruct device_node;\nstatic int atcphy_mux_set(struct typec_mux_dev *, struct typec_mux_state *);\n' + prefix=prefix.replace('struct typec_mux_dev { struct apple_atcphy *data; };', + '''struct typec_mux_dev { struct apple_atcphy *data; + int (*set)(struct typec_mux_dev *, struct typec_mux_state *); + int (*prepare)(struct typec_mux_dev *, struct typec_mux_state *); + int (*finish)(struct typec_mux_dev *, struct typec_mux_state *); + void (*commit)(struct typec_mux_dev *, struct typec_mux_state *); + void (*abort)(struct typec_mux_dev *, struct typec_mux_state *);bool removed; }; + struct typec_mux_context { struct { bool held; } lock; }; + struct task_struct { int unused; };static struct task_struct test_task; + #define current (&test_task) + static int typec_mux_srcu; + static int srcu_read_lock(int *s) { (void)s;return 0; } + static void srcu_read_unlock(int *s,int i) { (void)s;(void)i; } + #define mutex_trylock(m) ((m)->held ? 0 : ((m)->held=true)) + #define mutex_unlock(m) ((m)->held=false) + struct typec_mux { unsigned int num_mux_devs; struct typec_mux_dev **mux_devs; struct typec_mux_context *context; struct typec_mux_state transaction; struct task_struct *owner; int srcu_index; unsigned prepared; bool applied; }; + #define IS_ERR_OR_NULL(p) (!(p))''') + prefix=prefix.replace('#define WARN_ON_ONCE(v) assert(!(v))', 'static bool checked_warn(bool v) { assert(!v);return v; }\n#define WARN_ON_ONCE(v) checked_warn(v)') + prefix=prefix.replace('#define guard(kind) (void)',r''' +struct lock_guard { int *lock; }; +static struct lock_guard lock_take(int *lock) { assert(!*lock); *lock=1; return (struct lock_guard){lock}; } +static void lock_release(struct lock_guard *g) { assert(*g->lock==1); *g->lock=0; } +#define guard(kind) struct lock_guard __attribute__((cleanup(lock_release))) _guard = lock_take +''') + # Basic USB/device APIs are modeled, but get/put/wrapper and all parking/ + # restoration bodies are extracted verbatim from the chosen public source. + pmfixture=r''' +#define CONFIG_USB 1 +#define READ_ONCE(x) (x) +#define IS_REACHABLE(x) USB_REACHABLE +struct device_node { int id; }; +struct device_driver { const char *name; }; +struct device { int refs; struct device *parent; struct device_driver *driver; }; +struct platform_device { struct device dev; }; +struct usb_device; +struct usb_hcd; +struct usb_bus { struct device *sysdev, *controller; struct usb_hcd *hcd; struct usb_device *root_hub; }; +struct usb_hcd { int state; unsigned int rh_registered:1; + struct usb_bus self; unsigned long flags; + struct usb_hcd *shared_hcd,*primary_hcd; struct {bool pending;} died_work; }; +enum { HC_STATE_RUNNING=1, HC_STATE_QUIESCING=0x85, HC_STATE_HALT=0 }; +enum { HCD_FLAG_RH_RUNNING,HCD_FLAG_DEAD,HCD_FLAG_POLL_RH,HCD_FLAG_HW_ACCESSIBLE }; +static int __attribute__((unused)) hcd_root_hub_lock; +#define spin_lock_irqsave(lock, flags) do { assert(!*(lock));*(lock)=1;(flags)=0; } while(0) +#define spin_unlock_irqrestore(lock, flags) do { assert(*(lock));*(lock)=0;(void)(flags); } while(0) +#define clear_bit(bit, p) (*(p)&=~(1UL<<(bit))) +#define set_bit(bit, p) (*(p)|=1UL<<(bit)) +#define usb_hcd_is_primary_hcd(hcd) (!(hcd)->primary_hcd||(hcd)->primary_hcd==(hcd)) +#define usb_set_device_state(u, s) ((u)->state=(s)) +#define usb_kick_hub_wq(u) ((void)(u)) +#define schedule_work(w) ((w)->pending=true) +#define bus_to_hcd(bus) ((bus)->hcd) +#define HCD_DEAD(hcd) (!!((hcd)->flags&(1UL<flags&(1UL<refs++;return d; } +static void device_lock(struct device *d) { + if(d==&controller.dev){assert(!controller_locked&&!host_locked&&!root_locked);controller_locked=true;} + else {assert(d==&host_device&&controller_locked&&!host_locked&&!root_locked);host_locked=true;} +} +static void device_unlock(struct device *d) { + if(d==&host_device){assert(host_locked&&!root_locked);host_locked=false;} + else {assert(d==&controller.dev&&controller_locked&&!host_locked&&!root_locked);controller_locked=false;} +} +static void put_device(struct device *d) { assert(d->refs>0);d->refs--; } +static struct usb_device *usb_get_dev(struct usb_device *u) { u->refs++;return u; } +static void usb_put_dev(struct usb_device *u) { assert(u->refs>0);u->refs--; } +static int usb_for_each_dev(void *data,int (*fn)(struct usb_device *,void *)) { if(root_missing)return 0;for(unsigned int i=0;i<4;i++){int r=fn(&roots[i],data);if(r)return r;}return 0; } +static struct usb_device *usb_hub_find_child(struct usb_device *u, int port) { assert(u==&roots[3]&&root_locked&&port==1); + if(detach_at_lookup)usb_set_device_state(u,USB_STATE_NOTATTACHED); + if(quiesce_at_lookup)hcd.state=HC_STATE_QUIESCING; + return link_present?&link_device:NULL; } +static void usb_lock_device(struct usb_device *u) { + if(u==&roots[3]){assert(!root_locked);if(REQUIRES_PLATFORM_LOCKS)assert(controller_locked&&host_locked);root_locked=true;if(detach_on_lock)u->state=0;} + else {assert(u==&link_device&&root_locked&&!link_locked);link_locked=true;} +} +static void usb_unlock_device(struct usb_device *u) { + if(u==&roots[3]){assert(root_locked&&!link_locked);root_locked=false;} + else {assert(u==&link_device&&link_locked&&root_locked);link_locked=false;} +} +static int pm_runtime_resume_and_get(struct device *d) { + assert(root_locked&&!pm_phy->lock); + if(d==&link_device.dev){ + assert(link_locked);link_resumes++; + struct lock_guard g=lock_take(&pm_phy->lock);lock_release(&g); + if(link_error)return link_error; + link_pm_refs++;link_suspended=false;stuck_ack=false; + if(link_detach_on_resume)link_device.state=0; + if(link_dead_on_resume)set_bit(HCD_FLAG_DEAD,&hcd.flags); + return 0; + } + assert(d==&roots[3].dev);resumes++; + /* A re-entrant PHY lock acquisition during resume must remain possible. */ + struct lock_guard g=lock_take(&pm_phy->lock);lock_release(&g); + if(resume_error)return resume_error; + pm_refs++;stuck_ack=link_present&&link_device.state!=USB_STATE_NOTATTACHED&&link_suspended;if(detach_on_resume)roots[3].state=0; + if(inaccessible_on_resume)clear_bit(HCD_FLAG_HW_ACCESSIBLE,&hcd.flags); + if(dead_on_resume)set_bit(HCD_FLAG_DEAD,&hcd.flags); + return 0; +} +static void pm_runtime_put_noidle(struct device *d) { + if(d==&link_device.dev){assert(link_pm_refs==1);link_pm_refs--;} + else {assert(d==&roots[3].dev&&pm_refs==1);pm_refs--;} +} +static void usb_mark_last_busy(struct usb_device *u) { + assert(root_locked&&!pm_phy->lock); + if(u==&link_device)assert(link_locked);else assert(u==&roots[3]&&!link_locked); +} +static void pm_runtime_put_autosuspend(struct device *d) { + assert(root_locked&&!pm_phy->lock); + if(d==&link_device.dev){assert(link_locked&&link_pm_refs==1);link_pm_refs--;} + else {assert(d==&roots[3].dev&&!link_locked&&pm_refs==1);pm_refs--;} +} +''' + suffix=ns['suffix'].replace('int main(', 'int __attribute__((unused)) fixture_main(',1) + controls=r''' +static void prepare(struct apple_atcphy *p,bool fixed,bool t8140,bool swap) +{ + hw.has_t8140_dp=t8140;initial(p,fixed,false,swap); + p->np=&ctrl_node;pm_phy=p; + hcd=(struct usb_hcd){.state=HC_STATE_RUNNING,.rh_registered=true,.flags=1UL<lock&&!root_locked&&!link_locked&&!link_pm_refs&&!link_device.refs&&!pm_refs&&!node_refs&&!controller.dev.refs); + for(unsigned int i=0;i<4;i++)assert(!roots[i].refs); +} +int main(void) +{ + struct apple_atcphy p; +#if !USB_REACHABLE + { prepare(&p,true,true,false);unsigned int before=writes; + struct typec_mux_dev provider={.data=&p,.set=atcphy_mux_set}; + struct typec_mux_dev *providers[]={&provider};struct typec_mux_context typec_test_context={0};struct typec_mux mux={.num_mux_devs=1,.mux_devs=providers,.context=&typec_test_context}; + struct typec_mux_state state={.mode=TYPEC_STATE_SAFE}; + assert(typec_mux_set(&mux,&state)==-ENODEV&&writes==before);balanced(&p); + puts("PASS USB-unreachable fixed route blocks before MMIO");return 0; } +#endif + /* HC death before rh_registered is set need not mark the root NOTATTACHED. */ + prepare(&p,true,true,false);hcd.rh_registered=false;usb_hc_died(&hcd); + assert(roots[3].state!=USB_STATE_NOTATTACHED&&HCD_DEAD(&hcd));unsigned int before_hcd=writes; + int hcd_ret=safe(&p);fprintf(stderr,"dead HCD admission returned %d, PM resumes %d\n",hcd_ret,resumes); + assert(hcd_ret==-ENODEV&&writes==before_hcd&&resumes==0);balanced(&p); + if(HCD_DEAD_ONLY){puts("PASS dead HCD rejected before PM/MMIO");return 0;} + for(unsigned int gate=0;gate<2;gate++){ + prepare(&p,true,true,false);before_hcd=writes; + if(gate==0)inaccessible_on_resume=1; + if(gate==1)dead_on_resume=1; + assert(safe(&p)==-ENODEV&&writes==before_hcd);balanced(&p); + } + for(unsigned int swap=0;swap<2;swap++){ + prepare(&p,true,true,swap);stuck_ack=true; + assert(safe(&p)==0);assert(resumes==1);balanced(&p); + } + /* Resume failure must return before ANY lane/core programming. */ + prepare(&p,true,true,false);unsigned int before=writes;resume_error=-EIO; + unsigned int valid=p.dp_usb_context_valid,dirty=p.dp_usb_context_dirty,owned=p.dp_owned_pmas; + assert(safe(&p)==-EIO&&writes==before&&p.dp_usb_context_valid==valid&&p.dp_usb_context_dirty==dirty&&p.dp_owned_pmas==owned);balanced(&p); + resume_error=0;assert(safe(&p)==0);balanced(&p); + /* PM resume is NOT proof of ACK. Real parking timeout stays fatal. */ + prepare(&p,true,true,false);fail_caller="atcphy_pipehandler_lock";before=writes; + assert(safe(&p)==-ETIMEDOUT&&writes==before&&p.pipe_state==ATCPHY_PIPEHANDLER_STATE_USB3&&p.mode_dirty);balanced(&p); + fail_caller=NULL;assert(safe(&p)==0);balanced(&p); + for(unsigned int missing=0;missing<5;missing++){ + prepare(&p,true,true,false);before=writes; + if(missing==0)graph_missing=true; + if(missing==1)controller_missing=true; + if(missing==2)root_missing=true; + if(missing==3)detach_on_lock=1; + if(missing==4)detach_on_resume=1; + assert(safe(&p)==(missing==0||missing>=3?-ENODEV:-EAGAIN)&&writes==before);balanced(&p); + } + /* Root appears after independent fixed-host probe: missing root is retryable. */ + prepare(&p,true,true,false);root_missing=true;before=writes; + assert(safe(&p)==-EAGAIN&&writes==before);balanced(&p); + root_missing=false;assert(safe(&p)==0);balanced(&p); + /* BIST faults are independent of the runtime clock wake; never force readiness. */ + for(unsigned int swap=0;swap<2;swap++){ + prepare(&p,true,true,swap);assert(safe(&p)==0);balanced(&p); + orient(&p,!swap);bist_status_stuck=true; + assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==-ETIMEDOUT&&p.mode_dirty&&p.pipe_pending);balanced(&p); + bist_status_stuck=false;recover();assert(select_mode(&p,APPLE_ATCPHY_MODE_USB3_DP)==0);balanced(&p); + } + /* Scope and clean same-mode preserve MMIO guards; fixed no-op briefly owns PM. */ + struct typec_altmode alt={.svid=USB_TYPEC_DP_SID}; + struct typec_mux_dev mux={&p};struct typec_mux_state same={.mode=TYPEC_DP_STATE_D,.alt=&alt}; + prepare(&p,false,true,false);before=writes;assert(atcphy_mux_set(&mux,&same)==0&&resumes==0&&writes==before);balanced(&p); + prepare(&p,true,false,false);before=writes;assert(atcphy_mux_set(&mux,&same)==0&&resumes==0&&writes==before);balanced(&p); + prepare(&p,true,true,false);before=writes; + assert(atcphy_mux_set(&mux,&same)==0&&writes==before&&resumes==1);balanced(&p); + /* Binding/identity failure must not reach USB PM or PHY programming. */ + for(unsigned int missing=0;missing<3;missing++){ + prepare(&p,true,true,false);before=writes; + if(missing==0)controller.dev.driver=NULL; + if(missing==1)host_device.driver=NULL; + if(missing==2)host_device.parent=&wrong_controller; + assert(safe(&p)==(missing==2?-ENODEV:-EAGAIN)&&writes==before&&resumes==0);balanced(&p); + } + /* Root state can change during lookup despite USB device locks. */ + for(unsigned int event=0;event<2;event++){ + prepare(&p,true,true,false);before=writes; + if(event==0)detach_at_lookup=1;else quiesce_at_lookup=1; + int ret=safe(&p);fprintf(stderr,"post-child admission event %u returned %d writes %u\n",event,ret,writes-before); + assert(ret==-ENODEV&&writes==before);balanced(&p); + } + /* Selectively suspended upstream link is not resumed by root ownership alone. */ + for(unsigned int swap=0;swap<2;swap++){ + prepare(&p,true,true,swap);link_present=link_suspended=stuck_ack=true; + int link_ret=safe(&p);fprintf(stderr,"selective link SAFE returned %d, root resumes %d, link resumes %d\n",link_ret,resumes,link_resumes); + assert(link_ret==0&&resumes==1&&link_resumes==1&&!link_suspended);balanced(&p); + } + for(unsigned int fault=0;fault<3;fault++){ + prepare(&p,true,true,false);link_present=link_suspended=true;before=writes; + unsigned int retained=p.dp_usb_context_valid,owners=p.dp_owned_pmas; + if(fault==0)link_error=-EIO; + if(fault==1)link_detach_on_resume=1; + if(fault==2)link_dead_on_resume=1; + assert(safe(&p)==(fault?-ENODEV:-EIO)&&writes==before&&p.dp_usb_context_valid==retained&&p.dp_owned_pmas==owners);balanced(&p); + } + prepare(&p,true,true,false);link_present=link_suspended=true;link_error=-EIO; + assert(safe(&p)==-EIO);balanced(&p);link_error=0;assert(safe(&p)==0);balanced(&p); + prepare(&p,true,true,false);link_present=link_suspended=true;fail_caller="atcphy_pipehandler_lock";before=writes; + assert(safe(&p)==-ETIMEDOUT&&writes==before&&p.mode_dirty&&p.pipe_state==ATCPHY_PIPEHANDLER_STATE_USB3);balanced(&p); + fail_caller=NULL;assert(safe(&p)==0);balanced(&p); + for(unsigned int count=0;count<2;count++){ + prepare(&p,true,true,false);roots[3].maxchild=count?2:0;before=writes; + assert(safe(&p)==(count?-ENODEV:-EAGAIN)&&writes==before);balanced(&p); + } + prepare(&p,true,true,false);link_present=true;link_device.state=USB_STATE_NOTATTACHED; + assert(safe(&p)==0&&link_resumes==0);balanced(&p); + /* Startup with no old child is allowed; no fake dock or link is required. */ + prepare(&p,true,true,false);assert(safe(&p)==0&&link_resumes==0);balanced(&p); + puts("PASS actual upstream-link ownership: both PMAs, selective suspend, resume/removal/death/ACK failures, retry, unknown port map and no-child startup"); + puts("PASS actual PM wrapper + actual parking/restore: both PMAs, gated clock, PM error/retry, fatal ACK/retry, graph/controller/root/removal gates, legacy/ordinary and cached mode"); + return 0; +} +''' + helpers='' + if has_pm: + helpers=ns['block'](s,'struct atcphy_usb3_pm {')+';\n' + helpers+='#if IS_REACHABLE(CONFIG_USB)\n' + if 'static int atcphy_usb3_link_get(' in s: + helpers+=ns['function'](s,'atcphy_usb3_link_get')+'\n'+ns['function'](s,'atcphy_usb3_link_put')+'\n' + helpers+=ns['function'](s,'atcphy_usb3_pm_find')+'\n#endif\n' + helpers+=ns['function'](s,'atcphy_usb3_pm_get')+'\n'+ns['function'](s,'atcphy_usb3_pm_put')+'\n' + # The fixture extractor adapts the locked body; extract the wrapper directly. + marker=re.search(r'^static int atcphy_mux_set\(',s,re.M).group() + helpers+=ns['block'](s,marker)+'\n' + else: + # Parent runs its ACTUAL mux callback from production, without a substituted wrapper. + pmfixture=pmfixture.replace('static ', 'static __attribute__((unused)) ') + if usb_unreachable: + pmfixture=pmfixture.replace('static ', 'static __attribute__((unused)) ') + typec_source=(root/'drivers/usb/typec/mux.c').read_text() + typec_caller='\n'.join(ns['block'](typec_source, marker) for marker in + ('static void typec_mux_unlock(', 'void typec_mux_abort(', 'int typec_mux_begin(', 'int typec_mux_apply(', 'int typec_mux_end(', 'int typec_mux_set('))+'\n' + hcd_source=(root/'drivers/usb/core/hcd.c').read_text() + hcd_death=ns['block'](hcd_source,'void usb_hc_died (')+'\n' + text=prefix+prod+'\n#define REQUIRES_PLATFORM_LOCKS '+str(int('device_lock(&pm->controller->dev)' in s))+'\n#define USB_REACHABLE '+str(int(not usb_unreachable))+'\n#define HCD_DEAD_ONLY '+str(int(False))+'\n'+pmfixture+helpers+typec_caller+hcd_death+suffix+controls + return text + +def probe(root): + def source(path): + return (root/path).read_text() + + + def function(text, name): + match = re.search(r'^static (?:int|void)\s*\n?' + name + r'\(', text, re.M) + assert match, name + start = match.start() + opening = text.index('{', start) + depth = 1 + end = opening + 1 + while depth: + depth += (text[end] == '{') - (text[end] == '}') + end += 1 + return text[start:end] + + + prefix = r''' +#include +#include +#include +#include +#include +#include +#include +#define EPROBE_DEFER 517 +#define IS_ERR(p) ((intptr_t)(p)<0) +#define PTR_ERR(p) ((int)(intptr_t)(p)) +#define IS_ERR_OR_NULL(p) (!(p)||IS_ERR(p)) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#define guard(kind) (void) +#define TYPEC_STATE_SAFE 0 +#define INIT_DELAYED_WORK(p,fn) ((p)->pending=false) +#define atomic_set(p,n) (*(p)=(n)) +struct work { bool pending; }; +struct fwnode_handle { int refs; }; +struct typec_mux_state { void *alt,*data; unsigned long mode; }; +struct tps6598x { int lock; void *port; }; +struct notifier_block { int (*notifier_call)(struct notifier_block *,unsigned long,void *); }; +''' + pm = source('drivers/usb/typec/tipd/cd321x-pm.h').replace('#include ', '') + mocks = r''' +struct cd321x { + struct tps6598x tps; + struct work update_work,resume_work; + int link_event; + struct cd321x_pm_state pm; + void *mux,*tbt_switch,*port_altmode_dp,*port_altmode_tbt; + struct fwnode_handle *connector_fwnode; + struct typec_mux_state state; + struct notifier_block tbt_notifier; + bool state_valid,tbt_notifier_registered; +}; +static struct fwnode_handle connector; +static int mux_error, notifier_error, switch_error; +static int ports,altmodes,muxes,switches,notifiers,stop_calls; +static bool displayport=true; +static int tps6598x_register_port(struct tps6598x *t,struct fwnode_handle *f) { + (void)f;t->port=t;ports++;return 0; +} +static int cd321x_register_port_altmodes(struct cd321x *c) { + c->port_altmode_dp=c;c->port_altmode_tbt=c;altmodes+=2;return 0; +} +static void *fwnode_typec_mux_get(struct fwnode_handle *f) { (void)f;muxes++;return &muxes; } +static bool fwnode_property_present(struct fwnode_handle *f,const char *s) { (void)f;(void)s;return displayport; } +static struct fwnode_handle *fwnode_find_reference(struct fwnode_handle *f,const char *s,int n) { + (void)f;(void)s;(void)n;connector.refs++;return &connector; +} +static void __attribute__((unused)) fwnode_handle_put(struct fwnode_handle *f) { + if(f){assert(f->refs>0);f->refs--;} +} +static void *fwnode_typec_thunderbolt_switch_get(struct fwnode_handle *f) { + (void)f;if(switch_error)return (void *)(intptr_t)switch_error;switches++;return &switches; +} +static int typec_set_mode(void *p,unsigned long m) { (void)p;assert(m==TYPEC_STATE_SAFE);return mux_error; } +static int cd321x_tbt_notify(struct notifier_block *n,unsigned long e,void *d) { (void)n;(void)e;(void)d;return 0; } +static int typec_thunderbolt_switch_register_notifier(void *s,struct notifier_block *n) { + (void)s;(void)n;if(notifier_error)return notifier_error;notifiers++;return 0; +} +static void typec_thunderbolt_switch_put(void *s) { assert(s&&switches>0);switches--; } +static void typec_mux_put(void *m) { assert(m&&muxes>0);muxes--; } +static void typec_unregister_altmode(void *a) { if(a){assert(altmodes>0);altmodes--;} } +static void typec_unregister_port(void *p) { assert(p&&ports>0);ports--; } +static void cd321x_stop_work(struct cd321x *c) { + stop_calls++;if(c->tbt_notifier_registered){notifiers--;c->tbt_notifier_registered=false;} + cd321x_pm_remove(&c->pm);c->resume_work.pending=c->update_work.pending=false; +} +''' + core = source('drivers/usb/typec/tipd/core.c') + production = function(core, 'cd321x_register_port') + '\n' + function(core, 'cd321x_unregister_port') + controls = r''' +static void reset(struct cd321x *c) { + memset(c,0,sizeof(*c));connector.refs=0; + mux_error=notifier_error=switch_error=0;stop_calls=0; + assert(!ports&&!altmodes&&!muxes&&!switches&&!notifiers); +} +static void balanced(void) { + assert(!ports&&!altmodes&&!muxes&&!switches&&!notifiers&&!connector.refs); +} +int main(void) { + struct cd321x c; + struct cd321x_pm_state budget; + cd321x_pm_init(&budget);assert(cd321x_pm_ready(&budget)); + for(unsigned int i=0;iport=t;ports++;return 0;', '(void)f;t->port=&probe_port;ports++;return 0;') + # Real class dispatch requires the port's mux; its allocator remains a mock. + declarations = 'struct typec_port { struct typec_mux *mux; };\nstatic struct typec_port probe_port;\n' + dispatch = block((root / 'drivers/usb/typec/class.c').read_text(), 'int typec_set_mode(') + marker = port.index('static int tps6598x_register_port(') + port = port[:marker] + declarations + dispatch + '\n' + port[marker:] + import re + # Only helpers used at this boundary belong in this translation unit; + # the separate probe group exercises the complete retry state machine. + used = {'cd321x_pm_new_connection', 'cd321x_pm_init', 'cd321x_pm_remove', 'cd321x_pm_ready', + 'cd321x_pm_begin_read', 'cd321x_pm_snapshot_ready', + 'cd321x_pm_complete', 'cd321x_pm_can_update'} + for match in list(re.finditer(r'^static inline (?:void|bool) (cd321x_\w+)\(', port, re.M))[::-1]: + if match.group(1) not in used: + body = block(port, match.group()) + port = port[:match.start()] + port[match.start() + len(body):] + port = port.replace('static int mux_error, notifier_error, switch_error;', + 'static int notifier_error, switch_error;') + existing = set(re.findall(r'^\s*#define (\w+)', unit, re.M)) + port = '\n'.join(line for line in port.splitlines() + if not (line.startswith('#define ') and line.split()[1].split('(')[0] in existing)) + controls = r""" +static void port_balance(void) +{ + assert(!ports&&!altmodes&&!muxes&&!switches&&!notifiers&&!connector.refs); +} +int main(void) +{ + struct apple_atcphy p; + struct cd321x c={}; + struct typec_mux_dev provider={.data=&p,.set=atcphy_mux_set}; + struct typec_mux_dev *providers[]={&provider}; + struct typec_mux_context typec_test_context={0};struct typec_mux route={.num_mux_devs=1,.mux_devs=providers,.context=&typec_test_context};probe_port.mux=&route; + prepare(&p,true,true,false);root_missing=true;unsigned int before=writes; + assert(cd321x_register_port(&c.tps,&connector)==-EPROBE_DEFER); + assert(!c.state_valid&&!resumes&&writes==before);balanced(&p);port_balance(); + memset(&c,0,sizeof(c));root_missing=false; + assert(cd321x_register_port(&c.tps,&connector)==0 && c.state_valid && resumes==1); + assert(cd321x_pm_ready(&c.pm) && cd321x_pm_begin_read(&c.pm)); + cd321x_pm_snapshot_ready(&c.pm);cd321x_pm_complete(&c.pm); + assert(cd321x_pm_can_update(&c.pm)); + balanced(&p);cd321x_unregister_port(&c.tps);port_balance(); + memset(&c,0,sizeof(c));prepare(&p,true,true,false);notifier_error=-ENOMEM; + assert(cd321x_register_port(&c.tps,&connector)==-ENOMEM); + balanced(&p);port_balance();notifier_error=0; + memset(&c,0,sizeof(c));prepare(&p,true,true,false);resume_error=-EIO;before=writes; + assert(cd321x_register_port(&c.tps,&connector)==0 && !c.state_valid && writes==before); + balanced(&p);cd321x_unregister_port(&c.tps);port_balance(); + puts("PASS: actual CD321x probe -> typec_set_mode -> typec_mux_set -> bounded PM wrapper; delayed root reprobe and rollback references"); +} +""" + return unit + '\n' + port + '\n' + controls + + +def upstream_resume_boundary(root): + """Actual usbcore dispatch: root bus resume does not resume its child.""" + source = (root / 'drivers/usb/core/generic.c').read_text() + function = block(source, 'int usb_generic_driver_resume(') + prefix = r""" +#include +#include +#include +#include +typedef int pm_message_t; +struct usb_device { struct usb_device *parent; }; +static int bus_resumes, port_resumes, notifications, port_error; +static bool child_suspended = true; +static int hcd_bus_resume(struct usb_device *u, pm_message_t m) +{ (void)m; assert(!u->parent); bus_resumes++; return 0; } +static int usb_port_resume(struct usb_device *u, pm_message_t m) +{ (void)m; assert(u->parent); port_resumes++; if(port_error)return port_error; child_suspended=false; return 0; } +static void usbfs_notify_resume(struct usb_device *u) +{ (void)u; notifications++; } +""" + controls = r""" +int main(void) { + struct usb_device root={0}, child={.parent=&root}; + assert(usb_generic_driver_resume(&root,1)==0); + assert(bus_resumes==1&&port_resumes==0&&child_suspended&¬ifications==1); + assert(usb_generic_driver_resume(&child,1)==0); + assert(bus_resumes==1&&port_resumes==1&&!child_suspended&¬ifications==2); + child_suspended=true;port_error=-EIO; + assert(usb_generic_driver_resume(&child,1)==-EIO&&child_suspended&¬ifications==2); + puts("PASS actual usbcore root/child resume dispatch and failed child resume"); + return 0; +} +""" + return prefix + function + controls + + +def controller_lifecycle(root): + from teardown_pm import unit_for + return unit_for(root) + + +def asynchronous_root_state(root): + """Actual USB state API does not use the USB device mutex as its fence.""" + source = (root / 'drivers/usb/core/hub.c').read_text() + prefix = r""" +#include +#include +#include +enum usb_device_state { USB_STATE_NOTATTACHED, USB_STATE_CONFIGURED, USB_STATE_SUSPENDED }; +struct usb_device; +struct usb_port { struct usb_device *child; }; +struct usb_hub { struct usb_port **ports; }; +struct device { int unused; }; +struct config { struct { unsigned bmAttributes; } desc; }; +struct usb_device { struct device dev; struct usb_device *parent; struct usb_hub *hub; enum usb_device_state state; int maxchild; unsigned quirks; struct config *actconfig; unsigned long active_duration; bool device_locked; }; +static unsigned long jiffies=10; +static int device_state_lock; +#define USB_QUIRK_IGNORE_REMOTE_WAKEUP 1 +#define USB_CONFIG_ATT_WAKEUP 2 +#define spin_lock_irqsave(l,f) do { assert(!*(l));*(l)=1;(f)=0; } while(0) +#define spin_unlock_irqrestore(l,f) do { assert(*(l));*(l)=0;(void)(f); } while(0) +static struct usb_hub *usb_hub_to_struct_hub(struct usb_device *u) { return u->hub; } +static void update_port_device_state(struct usb_device *u) { (void)u; } +static void trace_usb_set_device_state(struct usb_device *u) { (void)u; } +static void device_set_wakeup_capable(struct device *d,int w) { (void)d;(void)w; } +""" + controls = r""" +int main(void) { + struct usb_device child={.state=USB_STATE_CONFIGURED,.device_locked=true}; + struct usb_port port={.child=&child};struct usb_port *ports[]={&port}; + struct usb_hub hub={.ports=ports}; + struct usb_device root={.state=USB_STATE_CONFIGURED,.maxchild=1,.hub=&hub,.device_locked=true}; + child.parent=&root; + usb_set_device_state(&root,USB_STATE_NOTATTACHED); + assert(root.state==USB_STATE_NOTATTACHED&&child.state==USB_STATE_NOTATTACHED); + assert(root.device_locked&&child.device_locked&&!device_state_lock); + usb_set_device_state(&child,USB_STATE_CONFIGURED); + assert(child.state==USB_STATE_NOTATTACHED); + puts("PASS actual asynchronous USB detach while device locks are held; no resurrection"); + return 0; +} +""" + return prefix + block(source, 'static void update_usb_device_state(') + block(source, 'static void recursively_mark_NOTATTACHED(') + block(source, 'void usb_set_device_state(') + controls diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/quiesce.py b/tools/testing/selftests/drivers/gpu/apple-dcp/quiesce.py new file mode 100644 index 00000000000000..6e7b9f69b2a254 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/quiesce.py @@ -0,0 +1,147 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Actual mux prepare/set and TPS removal ordering; firmware/PHY leaves are mocks.""" +from common import block + +def make_unit(source, parent=False, reference=None): + reference=source if reference is None else reference + core=(source/'drivers/usb/typec/tipd/core.c').read_text();mux=(source/'drivers/usb/typec/mux.c').read_text() + dcp=(reference/'drivers/gpu/drm/apple/dcp.c').read_text() + prefix=r''' +#include +#include +#include +#include +#include +#include +typedef unsigned u32; +#define IS_ERR(p) false +#define IS_ERR_OR_NULL(p) (!(p)) +#define EXPORT_SYMBOL_GPL(x) +#define guard(k) (void) +#define scoped_guard(k,l) for(int once=((void)(l),1);once;once=0) +#define READ_ONCE(x) (x) +#define WARN_ON_ONCE(x) (x) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#define to_platform_device(p) (p) +#define typec_mux_get_drvdata(m) ((m)->data) +#define dev_warn(...) ((void)0) +#define dev_info(...) ((void)0) +#define dev_name(p) "device" +#define list_for_each_entry(p,h,m) for((void)(h),(p)=sole_port();(p);(p)=NULL) +#define lockdep_assert_held(p) ((void)(p)) +#define dev_err(dev,fmt,ret) do { (void)(dev);last_error=(ret); } while(0) +enum {TYPEC_STATE_SAFE,TYPEC_STATE_USB,TYPEC_ORIENTATION_NONE,TYPEC_PWR_MODE_USB,TYPEC_THUNDERBOLT_SWITCH_OFF}; +enum usb_role {USB_ROLE_NONE,USB_ROLE_HOST,USB_ROLE_DEVICE}; +enum typec_data_role {TYPEC_HOST,TYPEC_DEVICE}; +#define TPS_STATUS_TO_TYPEC_PORTROLE(s) (s) +#define TPS_STATUS_TO_TYPEC_VCONN(s) (s) +#define TPS_STATUS_TO_TYPEC_DATAROLE(s) (s) +struct typec_mux_state { void *alt; unsigned long mode; void *data; }; +struct typec_mux_dev { bool removed; void (*commit)(struct typec_mux_dev *,struct typec_mux_state *); void (*abort)(struct typec_mux_dev *,struct typec_mux_state *); int (*finish)(struct typec_mux_dev *,struct typec_mux_state *); void *data; int (*set)(struct typec_mux_dev *,struct typec_mux_state *); int (*prepare)(struct typec_mux_dev *,struct typec_mux_state *); }; +struct task_struct { int unused; }; +static _Thread_local struct task_struct this_task; +#define current (&this_task) +struct mutex { pthread_mutex_t value; }; +struct typec_mux_context { struct mutex lock; }; +static struct typec_mux_context shared_context={.lock={.value=PTHREAD_MUTEX_INITIALIZER}}; +static int typec_mux_srcu; +static int srcu_read_lock(int *s) { (void)s;return 0; } +static void srcu_read_unlock(int *s,int i) { (void)s;(void)i; } +static int mutex_trylock(struct mutex *m) { return pthread_mutex_trylock(&m->value)==0; } +static void mutex_unlock(struct mutex *m) { assert(!pthread_mutex_unlock(&m->value)); } +struct typec_mux { struct typec_mux_context *context; struct typec_mux_state transaction; struct task_struct *owner; int srcu_index; unsigned prepared; bool applied; unsigned num_mux_devs; struct typec_mux_dev *mux_devs[3]; }; +struct connector { bool connected; }; +struct apple_dcp { int external_lock,external_error; bool external_cable,external_stopped; struct { bool connected; } dptxport[1]; void *hdmi_hpd; bool active; bool external,typec_cable_connected; void *dev; struct connector *typec_connector; }; +struct apple_dcp_typec_route; +struct apple_dcp_typec_port { int link; void *connector_np; struct apple_dcp_typec_route *owner,*target,*secondary_target,*preferred_route; void *applied_alt; unsigned long applied_mode; unsigned applied_status,applied_conf; bool applied_valid,safe_pending,safe_finish_ready,dp_wanted,dp_hpd,hpd; }; +struct apple_dcp_typec_route { struct apple_dcp_typec_port *port; struct apple_dcp *dcp; void *xbar; bool deactivate_xbar_done; int deactivate_error; bool selected,tunnel; }; +static int dcp_typec_fabric_lock,dcp_typec_ports,deactivate_calls; +struct drm_device { int unused; }; +static struct apple_dcp display; +static struct connector connector; +static struct apple_dcp_typec_port port; +static struct apple_dcp_typec_route route; +static struct apple_dcp_typec_port *sole_port(void) { return &port; } +static char events[100];static unsigned count; +static int power_error,phy_error,tbt_error,role_error,last_error,phy_calls,reset_calls,gpio_calls,role_calls,stops,unregisters; +static void event(char c) { assert(count+1external; } +static int dcp_dptx_disconnect_oob(void *dev,u32 index) { (void)dev;assert(index==0);event('Q');stops++;if(power_error){display.external_error=power_error;return power_error;}display.external_cable=false;display.external_stopped=true;display.dptxport[0].connected=false;display.typec_cable_connected=false;connector.connected=false;return 0; } +static struct drm_device *dcp_typec_drm(void) { return NULL; } +static void dcp_typec_plan(struct drm_device *d,struct apple_dcp_typec_port *p,unsigned n) { (void)d;(void)p;(void)n; } +static bool dcp_typec_tunnel_held(struct apple_dcp *d) { (void)d;return false; } +static int dcp_typec_route_deactivate(struct apple_dcp_typec_route *r) { deactivate_calls++;r->selected=false;return 0; } +static void dcp_typec_port_attach(struct apple_dcp_typec_port *p) { (void)p; } +static int gpiod_get_value_cansleep(void *p) { (void)p;return 0; } +static int dcp_dptx_connect(struct apple_dcp *d,unsigned n) { (void)d;(void)n;return 0; } +static bool dcp_typec_route_available(struct apple_dcp_typec_route *r) { return !r->selected; } +static int dcp_typec_route_activate(struct apple_dcp_typec_route *r,void *x) { (void)x;r->selected=true;return 0; } +struct tps6598x; +struct tipd_data { REMOVE_RETURN (*remove)(struct tps6598x *); void (*unregister_port)(struct tps6598x *); int (*reset)(struct tps6598x *); }; +struct typec_port { struct typec_mux *mux; }; +struct tps6598x { bool remove_safe_prepared; struct typec_mux *mux; void *partner,*dev,*psy,*role_sw,*reset; int irq,wq_poll; struct typec_port *port; struct tipd_data *data; }; +struct cd321x { struct tps6598x tps; struct typec_mux *mux; void *tbt_switch; }; +struct typec_thunderbolt_switch_data { int state; }; +static void cd321x_stop_work(struct cd321x *c) { (void)c;event('W'); } +static int typec_thunderbolt_switch_set(void *p,void *s) { (void)p;(void)s;event('B');return tbt_error; } +static int usb_role_switch_set_role(void *p,int r) { (void)p;assert(r==USB_ROLE_NONE);event('R');role_calls++;return role_error; } +static void typec_unregister_partner(void *p) { (void)p;event('P'); } +#define typec_set_pwr_opmode(...) ((void)0) +#define typec_set_pwr_role(...) ((void)0) +#define typec_set_vconn_role(...) ((void)0) +static int typec_set_orientation(void *p,int o) { (void)p;assert(o==TYPEC_ORIENTATION_NONE);return 0; } +static void typec_set_data_role(struct typec_port *p,enum typec_data_role r) { (void)p;(void)r;event('D'); } +static void dcp_typec_pipeline_freed(void) { event('F'); } +static void power_supply_changed(void *p) { (void)p; } +static void tipd_debugfs_unregister(struct tps6598x *t) { (void)t; } +static void cancel_delayed_work_sync(void *w) { (void)w; } +static void devm_free_irq(void *d,int i,void *t) { (void)d;(void)i;(void)t; } +static void typec_mux_put(struct typec_mux *m) { (void)m; } +static void usb_role_switch_put(void *p) { (void)p;event('V'); } +static void gpiod_set_value_cansleep(void *p,int v) { (void)p;assert(v==1);gpio_calls++;event('G'); } + +'''.replace('REMOVE_RETURN','void' if parent else 'int') + functions='\n'.join(block(mux, marker) for marker in ('static void typec_mux_unlock(', 'void typec_mux_abort(', 'int typec_mux_begin(', 'int typec_mux_apply(', 'int typec_mux_end(', 'int typec_mux_prepare(')) if not parent else block((reference/'drivers/usb/typec/mux.c').read_text(),'int typec_mux_prepare(')+'\n' + functions+=block(mux,'int typec_mux_set(')+'\n'+block(dcp,'static int dcp_typec_route_prepare(')+'\n'+block(dcp,'static int dcp_typec_route_finish(')+'\n'+block(dcp,'static void dcp_typec_route_commit(')+'\n' + functions+=block(dcp,'static struct apple_dcp_typec_route *\ndcp_typec_rebalance_locked(')+'\n' + if not parent:functions+=block(core,'static void tps6598x_drop_partner(')+'\n'+block(core,'static int tps6598x_set_data_role(')+'\n' + functions+=block(core,'static '+('void' if parent else 'int')+' tps6598x_disconnect(')+'\n'+block(core,'static '+('void' if parent else 'int')+' cd321x_remove(')+'\n'+block(core,'void tipd_remove(')+'\n' + controls=r''' +static struct typec_mux_dev phy_provider,display_provider; +static struct typec_mux mux; +static struct typec_port class_port; +static struct tipd_data data; + +static int phy_set(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;assert(s->mode==TYPEC_STATE_SAFE);event('L');phy_calls++;return phy_error; } +static int display_set(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;(void)s;event('S');if(port.owner&&display.external&&connector.connected) {int r=dcp_dptx_disconnect_oob(display.dev,0);if(r)return r;}if(port.safe_pending)return 0;port.owner=NULL;return 0; } +static void unregister_port(struct tps6598x *t) { (void)t;unregisters++;event('U'); } +static int reset(struct tps6598x *t) { (void)t;reset_calls++;event('X');return 0; } +static void initial(struct cd321x *c) { + count=0;events[0]=0;power_error=phy_error=tbt_error=role_error=last_error=phy_calls=reset_calls=gpio_calls=role_calls=stops=unregisters=deactivate_calls=0; + connector=(struct connector){.connected=true};display=(struct apple_dcp){.external=true,.external_cable=true,.dptxport={{.connected=true}},.typec_cable_connected=true,.typec_connector=&connector}; + port=(struct apple_dcp_typec_port){.owner=&route,.applied_valid=true,.dp_wanted=true,.dp_hpd=true,.hpd=true};route=(struct apple_dcp_typec_route){.port=&port,.dcp=&display,.selected=true}; + phy_provider=(struct typec_mux_dev){.set=phy_set};display_provider=(struct typec_mux_dev){.data=&route,.set=display_set,.prepare=dcp_typec_route_prepare,.finish=dcp_typec_route_finish,.commit=dcp_typec_route_commit}; + mux=(struct typec_mux){.context=&shared_context,.num_mux_devs=2,.mux_devs={&phy_provider,&display_provider}};class_port.mux=&mux; + data=(struct tipd_data){.remove=cd321x_remove,.unregister_port=unregister_port,.reset=reset}; + *c=(struct cd321x){.tps={.mux=&mux,.partner=(void *)1,.port=&class_port,.data=&data,.reset=(void *)1},.mux=&mux}; +} +static int position(char c) { for(unsigned i=0;i=0&&position('Q')', '') + timing_fixture = [{ + "HorizontalAttributes": {"Total": 2200, "FrontPorch": 88, "SyncWidth": 44, + "Active": 1920, "PreciseSyncRate": 4423680, "SyncPolarity": 1, "PixelRepetition": 0}, + "VerticalAttributes": {"Total": 1125, "FrontPorch": 4, "SyncWidth": 5, + "Active": 1080, "PreciseSyncRate": 3932160, "SyncPolarity": 1, "PixelRepetition": 0}, + "ColorModes": [{"ID": 85, "Score": 1, "IsVirtual": False, "PixelEncoding": 0, + "Colorimetry": 10, "Depth": 10, "DynamicRange": 0, "EOTF": 0}], + "ID": 62, "Score": 16224, "IsPreferred": True, "IsInterlaced": False, "IsVirtual": False, + }] + timing_raw = bytearray(struct.pack(' +#include +#include +#include +#include +#include +#include +#include +typedef uint8_t u8; +typedef uint16_t u16; +typedef uint32_t u32; +typedef uint64_t u64; +typedef int64_t s64; +typedef u16 __le16; +typedef u16 __be16; +typedef u32 __le32; +#define __packed __attribute__((packed)) +#undef static_assert +#define static_assert(c) _Static_assert(c, #c) +#define IS_ENABLED(x) 0 +#define U32_MAX UINT32_MAX +#define U16_MAX UINT16_MAX +#define S64_MAX INT64_MAX +#define S64_MIN INT64_MIN +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) +#define round_up(n, a) (((n) + (a) - 1) & ~((a) - 1)) +#define ERR_PTR(n) ((void *)(intptr_t)(n)) +#define PTR_ERR(p) ((int)(intptr_t)(p)) +#define IS_ERR(p) ((uintptr_t)(p) > (uintptr_t)-4096) +#define IS_ERR_OR_NULL(p) (!(p) || IS_ERR(p)) +#define GFP_KERNEL 0 +#define kmalloc(n, f) malloc(n) +#define kcalloc(n, s, f) calloc(n,s) +#define kfree(p) free(p) +#define mul_u32_u32(a, b) ((u64)(u32)(a) * (u32)(b)) +#define DIV_ROUND_CLOSEST_ULL(n, d) (((n) + (d) / 2) / (d)) +#define DIV_ROUND_CLOSEST(n, d) (((n) + (d) / 2) / (d)) +#define trace_iomfb_color_mode(...) ((void)0) +#define trace_iomfb_parse_mode_fail(...) ((void)0) +#define trace_iomfb_parse_mode_success(...) ((void)0) +#define trace_iomfb_timing_mode(...) ((void)0) +#define DRM_MODE_TYPE_DRIVER 1 +#define DRM_MODE_TYPE_PREFERRED 2 +#define DRM_MODE_FLAG_PHSYNC 1 +#define DRM_MODE_FLAG_NHSYNC 2 +#define DRM_MODE_FLAG_PVSYNC 4 +#define DRM_MODE_FLAG_NVSYNC 8 +#define cpu_to_le16(v) (v) +#define cpu_to_le32(v) (v) +#define cpu_to_be16(v) __builtin_bswap16(v) +struct drm_display_mode { + unsigned int type, flags; + int clock, vdisplay, vsync_start, vsync_end, vtotal; + int hdisplay, hsync_start, hsync_end, htotal, width_mm, height_mm; +}; +static void drm_mode_set_name(struct drm_display_mode *mode) { (void)mode; } +''' + stubs = r''' +struct drm_edid { struct edid raw; }; +static const struct drm_edid *drm_edid_alloc(const void *data, size_t size) +{ + assert(size == 128); + struct drm_edid *result = malloc(sizeof(*result)); + memcpy(&result->raw, data, size); + return result; +} +static void drm_edid_free(const struct drm_edid *edid) { free((void *)edid); } +static bool drm_edid_valid(const struct drm_edid *edid) +{ + const u8 *raw = (const u8 *)&edid->raw; + unsigned int sum = 0; + for (unsigned int i = 0; i < 128; i++) sum += raw[i]; + return !(sum & 255) && !raw[126] && !memcmp(raw, "\0\xff\xff\xff\xff\xff\xff\0", 8); +} +''' + suffix = r''' +static void epochs(void) +{ + struct dcp_metadata_epoch e = {0}; + struct dcp_metadata_ticket before = dcp_metadata_begin(&e); + assert(!dcp_metadata_matches(&e, before)); + dcp_metadata_epoch_connection(&e, 1, true); + struct dcp_metadata_ticket current = dcp_metadata_begin(&e); + assert(dcp_metadata_matches(&e, current)); + assert(!dcp_metadata_matches(&e, before)); /* Never retag pre-association metadata. */ + dcp_metadata_epoch_connection(&e, 1, true); + assert(dcp_metadata_matches(&e, current)); + dcp_metadata_epoch_connection(&e, 2, false); + assert(!dcp_metadata_matches(&e, current)); + dcp_metadata_epoch_connection(&e, 3, true); + assert(!dcp_metadata_matches(&e, dcp_metadata_begin(&e))); /* Later attachment ineligible. */ + e = (struct dcp_metadata_epoch){0}; + dcp_metadata_epoch_connection(&e, 9, true); + current = dcp_metadata_begin(&e); + dcp_metadata_epoch_connection(&e, 10, true); /* A route change also invalidates. */ + assert(!dcp_metadata_matches(&e, current)); +} +int main(int argc, char **argv) +{ + if (argc == 3 && !strcmp(argv[1], "--property")) { + FILE *f=fopen(argv[2],"rb"); assert(f && !fseek(f,0,SEEK_END)); + long n=ftell(f); assert(n>20 && !fseek(f,0,SEEK_SET)); + unsigned char *b=malloc(n); assert(fread(b,n,1,f)==1); fclose(f); + assert(dcp_property_complete(b,n-1,n)); + assert(dcp_property_complete(b,n-1,n-1)); + for(unsigned int cut=1;cut<=16;cut++) + assert(!dcp_property_complete(b,n-1-cut,n-cut)); + assert(!dcp_property_complete(b,n-1,n+1)); + free(b); return 0; + } + assert(argc == 4 || argc == 5); + /* A missing final string byte must not be supplied by zero allocation. */ + unsigned char incomplete[]={0xd3,0,0,0,1,0,0,0x82,1,0,0,0x89,0x42}; + assert(dcp_property_complete(incomplete,sizeof(incomplete),sizeof(incomplete))); + assert(!dcp_property_complete(incomplete,12,13)); + assert(!dcp_property_complete(NULL,0,0)); + epochs(); + timing_proof(); + FILE *f = fopen(argv[1], "rb"); + assert(f && !fseek(f, 0, SEEK_END)); + long length = ftell(f); + assert(length > 0 && !fseek(f, 0, SEEK_SET)); + void *data = malloc(length); + assert(fread(data, length, 1, f) == 1); + fclose(f); + struct dcp_parse_ctx ctx; + struct dcp_display_metadata metadata, unchanged; + memset(&metadata, 0xa5, sizeof(metadata)); unchanged = metadata; + assert(!parse(data, length, &ctx)); + int ret = parse_display_metadata(&ctx, &metadata); + if (atoi(argv[2])) { + assert(ret == -EINVAL && !memcmp(&metadata, &unchanged, sizeof(metadata))); + free(data); return 0; + } + assert(!ret); + assert(dcp_property_complete(data,length,length)); + size_t aligned=round_up((size_t)length,4); + unsigned char *padded=malloc(aligned+4); + memset(padded,0xa5,aligned+4); memcpy(padded,data,length); + assert(dcp_property_complete(padded,aligned,aligned+1)); + assert(!dcp_property_complete(data,length-1,length)); + assert(!dcp_property_complete(padded,aligned,aligned+2)); + assert(!dcp_property_complete(padded,aligned+4,aligned+5)); + assert(!dcp_property_complete(padded,aligned-1,aligned+1)); + free(padded); + struct dcp_parse_ctx timing_ctx; + unsigned int count = 0; + assert(!parse(timing_blob, sizeof(timing_blob), &timing_ctx)); + struct dcp_display_mode *timings = enumerate_modes(&timing_ctx, &count, + metadata.width_cm * 10, metadata.height_cm * 10, 0, false, true); + assert(!IS_ERR(timings) && count == 1 && timings[0].metadata_timing_valid); + struct drm_display_mode mode = timings[0].mode; + assert(mode.flags == (DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_PVSYNC)); + assert(mode.clock == 148500 && mode.hdisplay == 1920 && mode.vdisplay == 1080); + free(timings); + const struct drm_edid *edid = dcp_metadata_edid(&metadata, &mode); + assert(!IS_ERR(edid)); + f = fopen(argv[3], "wb"); assert(f); + assert(fwrite(&edid->raw, 128, 1, f) == 1); fclose(f); + drm_edid_free(edid); + mode.flags = 0; + assert(IS_ERR(dcp_metadata_edid(&metadata, &mode))); + mode.flags = DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_PVSYNC; + mode.clock++; + assert(IS_ERR(dcp_metadata_edid(&metadata, &mode))); + free(data); + if (argc == 5) { + f=fopen(argv[4],"rb"); assert(f && !fseek(f,0,SEEK_END)); + length=ftell(f); assert(length>0 && !fseek(f,0,SEEK_SET)); + data=malloc(length); assert(fread(data,length,1,f)==1); fclose(f); + assert(!parse(data,length,&timing_ctx)); count=0; + timings=enumerate_modes(&timing_ctx,&count,600,330,0,false,true); + assert(!IS_ERR(timings)); bool found60=false, found120=false; + for(unsigned int i=0;iusb4; } +static void schedule_work(struct work_struct *w) { w->scheduled++; } +static const struct edid *drm_edid_raw(const struct drm_edid *e) { return (const void *)e->data; } +static bool dcpavserv_edid_service_current(struct apple_dcp *d, u64 e) { + assert(d->dcpavserv.lock); return e==d->dcpavserv.epoch && d->dcpavserv.enabled && d->dcpavserv.service; +} +''' + callback_stubs=r''' +#define le64_to_cpu(v) (v) +#define le32_to_cpu(v) (v) +#define LINK_RATE_HBR3 0x1e +static int configure_error; +static int dptxport_call_set_active_lane_count(struct apple_epic_service *s, const void *data, size_t size, void *reply, size_t room) { + if (configure_error) return configure_error; + if(sizecookie)->lane_count=((const struct dptxport_apcall_set_active_lane_count *)data)->lane_count; return 0; +} +static int dptxport_call_set_link_rate(struct apple_epic_service *s, const void *data, size_t size, void *reply, size_t room) { + if(configure_error) return configure_error; + if(sizecookie)->link_rate=((const struct dptxport_apcall_link_rate *)data)->link_rate; return 0; +} +static bool dcp_firmware_metadata_enabled(struct apple_dcp *d) { return d->hw.iomfb_method_profile==2; } +#define dcp_trace_metadata(d,p,v) ((void)0) +''' + callback_types='typedef u64 __le64;\n'+callback_types + test=r''' +static void checksum(u8 *p) { unsigned sum=0; for(unsigned i=0;i<127;i++) sum+=p[i]; p[127]=-sum; } +static void qualify(struct apple_dcp *d, struct apple_epic_service *s) { + struct dptxport_apcall_link_rate rate={.link_rate=0x1e}, rate_reply={}; + struct dptxport_apcall_set_active_lane_count lanes={.lane_count=2}, lanes_reply={}; + assert(!dptxport_call_qualified_config(s,false,&rate,sizeof(rate),&rate_reply,sizeof(rate_reply))); + assert(!dptxport_call_qualified_config(s,true,&lanes,sizeof(lanes),&lanes_reply,sizeof(lanes_reply))); + (void)d; +} +static bool current(struct apple_dcp *d,const struct dcp_display_mode *m) { + d->hpd_mutex=d->dcpavserv.lock=d->metadata.lock=true; + bool result=dcp_180hz_mode_current(d,m); + d->hpd_mutex=d->dcpavserv.lock=d->metadata.lock=false; + return result; +} +int main(int argc,char **argv) { + struct dcp_metadata_epoch epoch={}; + dcp_metadata_epoch_connection(&epoch,7,true); + assert(dcp_metadata_matches(&epoch,dcp_metadata_begin(&epoch))); + PARSER_CHECKS + struct drm_edid raw={}; u8 *p=raw.data; + struct dcp_display_mode m={}; + if(argc==2) { FILE *f=fopen(argv[1],"rb"); assert(f); assert(fread(p,1,256,f)==256 && fgetc(f)==EOF); fclose(f); } + else { + memcpy(p,"\0\xff\xff\xff\xff\xff\xff\0",8); + p[8]=4;p[9]=0x72;p[10]=0x4f;p[11]=0x0b;p[18]=1;p[19]=4;p[20]=0xa0;p[126]=1; + u8 *d=p+54; unsigned clock=41409;d[0]=clock;d[1]=clock>>8; + d[2]=1920&255;d[3]=160;d[4]=0x70;d[5]=1080&255;d[6]=26;d[7]=0x40; + d[8]=48;d[9]=32;d[10]=0x35;d[17]=0x1e; + p[75]=0xfc;memcpy(p+77,"QG270 S3\n ",13); + u8 *e=p+128;e[0]=2;e[1]=3;e[2]=12;e[4]=0x67;e[5]=0xd8;e[6]=0x5d;e[7]=0xc4;e[8]=1;e[9]=120;e[10]=0x80; + checksum(p);checksum(e); + } + /* Get the advertised DTD's porch/polarity, without copying identity data. */ + const u8 *dt=NULL; + for(unsigned off=54;off+18<=126;off+=18) if((p[off]|p[off+1]<<8)==41409)dt=p+off; + for(unsigned off=p[130];off+18<=127;off+=18) if((p[128+off]|p[129+off]<<8)==41409)dt=p+128+off; + assert(dt); + m.mode=(struct drm_display_mode){.clock=414090,.hdisplay=1920,.htotal=2080,.vdisplay=1080,.vtotal=1106}; + m.mode.hsync_start=1920+(dt[8]|(dt[11]&0xc0)<<2);m.mode.hsync_end=m.mode.hsync_start+(dt[9]|(dt[11]&0x30)<<4); + m.mode.vsync_start=1080+(dt[10]>>4|(dt[11]&0xc)<<2);m.mode.vsync_end=m.mode.vsync_start+((dt[10]&15)|(dt[11]&3)<<4); + m.mode.flags=(dt[17]&2?DRM_MODE_FLAG_PHSYNC:DRM_MODE_FLAG_NHSYNC)|(dt[17]&4?DRM_MODE_FLAG_PVSYNC:DRM_MODE_FLAG_NVSYNC); + assert(dcp_180hz_edid(p,256,&m.mode)); + assert(!dcp_180hz_edid(p,255,&m.mode)); + u8 saved[256];memcpy(saved,p,256); + p[10]^=1;checksum(p);assert(!dcp_180hz_edid(p,256,&m.mode));memcpy(p,saved,256); + p[8]^=1;checksum(p);assert(!dcp_180hz_edid(p,256,&m.mode));memcpy(p,saved,256); + unsigned hf=0;for(unsigned off=132;off<128+p[130];off+=(p[off]&31)+1) if(p[off+1]==0xd8 && p[off+2]==0x5d && p[off+3]==0xc4)hf=off; + assert(hf);p[hf+5]=82;checksum(p+128);assert(!dcp_180hz_edid(p,256,&m.mode));memcpy(p,saved,256); + p[127]^=1;assert(!dcp_180hz_edid(p,256,&m.mode));memcpy(p,saved,256); + /* Serial changes do not participate in identity matching. */ + p[12]^=0xff;checksum(p);assert(dcp_180hz_edid(p,256,&m.mode));memcpy(p,saved,256); + assert(dcp_180hz_payload(0x1e,2,414090));assert(!dcp_180hz_payload(0x14,2,414090));assert(!dcp_180hz_payload(0x1e,1,414090)); + struct apple_dcp d={.dev=&d,.phy=&raw,.active_typec_route=&m,.typec_cable_connected=true,.typec_generation=7,.nr_modes=1}; + struct apple_connector con={.dcp=&d,.connected=true,.drm_edid=&raw,.drm_edid_generation=7,.drm_edid_service_epoch=1,.base.registration_state=1}; + struct apple_epic ep={.dcp=&d,.endpoint=0x2c,.wire_v2=true}; + struct apple_epic_service s={.ep=&ep,.cookie=&d.dptxport[0],.enabled=true}; + d.connector=&con;d.hw.iomfb_method_profile=2;d.metadata.epoch.connected=true;d.metadata.epoch.generation=7; + d.dcpavserv.enabled=true;d.dcpavserv.service=&raw;d.dcpavserv.epoch=1; + d.dptxport[0]=(struct dptx_port){.enabled=true,.connected=true,.atcphy=d.phy,.service=&s}; + m.unsafe_rgb8_ticket=(struct dcp_metadata_ticket){.associated=true,.generation=7};m.unsafe_rgb8_service_epoch=1; + m.sdr_rgb=(struct dcp_color_mode){.depth=8,.format=0,.range=0,.eotf=0,.colorimetry=10}; + assert(!current(&d,&m));qualify(&d,&s);assert(current(&d,&m));assert(con.hotplug_wq.scheduled==1); + #define DENY(field,value) do { __typeof__(field) old=field;field=value;assert(!current(&d,&m));field=old;assert(current(&d,&m)); } while(0) + DENY(d.typec_generation,8);DENY(con.drm_edid_generation,6);DENY(d.dcpavserv.epoch,3);DENY(con.drm_edid_service_epoch,3); + /* Timings may precede the very first AV announcement. Only fresh epoch-1 + * EDID plus epoch-1 callback proof may admit this initial receipt. */ + m.unsafe_rgb8_service_epoch=0;m.automatic_service_pending=true;assert(current(&d,&m)); + d.dcpavserv.epoch=3;con.drm_edid_service_epoch=3;qualify(&d,&s); + assert(!current(&d,&m)); /* Initial receipt never survives retirement. */ + m.unsafe_rgb8_service_epoch=3;m.automatic_service_pending=false;assert(current(&d,&m)); + d.dcpavserv.epoch=1;con.drm_edid_service_epoch=1;m.unsafe_rgb8_service_epoch=1;qualify(&d,&s); + DENY(con.drm_edid_firmware_derived,true);DENY(d.usb4,true);DENY(d.dptx_tunnel,true); + DENY(d.quiescing,true);DENY(d.metadata.epoch.connected,false);DENY(d.rate_proof.connection_sequence,1); + DENY(d.rate_proof.rate.value,0x14);DENY(d.rate_proof.lanes.value,1);DENY(ep.endpoint,0x28); + DENY(m.sdr_rgb.depth,10);DENY(m.sdr_rgb.colorimetry,1);DENY(m.sdr_rgb.range,1);DENY(m.mode.clock,414091); + struct dptxport_apcall_link_rate rate={.link_rate=0x1e}, response={}; + configure_error=-EIO; + assert(dptxport_call_qualified_config(&s,false,&rate,sizeof(rate),&response,sizeof(response))==-EIO); + assert(!current(&d,&m));configure_error=0;qualify(&d,&s); + assert(dptxport_call_qualified_config(&s,false,&rate,sizeof(rate)-1,&response,sizeof(response))==-EINVAL); + assert(!current(&d,&m));qualify(&d,&s); + dcp_firmware_metadata_connection(&d,7,false);dcp_firmware_metadata_connection(&d,7,true); + assert(!current(&d,&m));qualify(&d,&s);assert(!current(&d,&m)); + m.automatic_connection_sequence=d.rate_proof.connection_sequence;assert(current(&d,&m)); + dcp_firmware_metadata_hotplug(&d,true);dcp_firmware_metadata_hotplug(&d,false); + assert(!current(&d,&m)); + dcp_firmware_metadata_connection(&d,7,true);m.automatic_connection_sequence=d.rate_proof.connection_sequence;con.edid_invalidation_pending=false;qualify(&d,&s);assert(current(&d,&m)); + struct dcp_180hz_token old=dcp_180hz_config_begin(&s,false);d.typec_generation++; + dcp_180hz_config_complete(&s,false,old,0x1e,true);assert(!d.rate_proof.rate.valid);d.typec_generation--;qualify(&d,&s); + old=dcp_180hz_config_begin(&s,true);d.dcpavserv.epoch+=2; + dcp_180hz_config_complete(&s,true,old,2,true);assert(d.rate_proof.lanes.valid);assert(!current(&d,&m));d.dcpavserv.epoch-=2;qualify(&d,&s); + old=dcp_180hz_config_begin(&s,false);dcp_180hz_invalidate(&d); + dcp_180hz_config_complete(&s,false,old,0x1e,true);assert(!current(&d,&m));qualify(&d,&s); + old=dcp_180hz_config_begin(&s,false);dcp_180hz_config_complete(&s,false,old,0x1e,false);assert(!current(&d,&m)); + d.dcpavserv.epoch=0;old=dcp_180hz_config_begin(&s,false);d.dcpavserv.epoch=1; + dcp_180hz_config_complete(&s,false,old,0x1e,true);assert(d.rate_proof.rate.valid);qualify(&d,&s);assert(current(&d,&m)); + puts("PASS: actual binary parser, Acer EDID, capacity, callback success, late-proof event, generation/AV lifetime and foreign-sink controls"); +} +'''.replace('PARSER_CHECKS', checks) + unit=namespace['prefix']+namespace['metadata']+namespace['epoch']+namespace['header']+namespace['edid_header']+namespace['parser']+blobs+policy+proof+state+source+callback_types+callback_stubs+callback+connection+test + return {**locals(), "block": block} + +def publication(root): + + n = policy(root) + n.update({key:n['namespace'][key] for key in ('prefix','metadata','epoch','header','edid_header','parser')}) + base = n['base'] + block = n['block'] + av = (base/'epic/dpavservep.c').read_text() + iomfb = (base/'iomfb.c').read_text() + template = (base/'iomfb_template.c').read_text() + + state = n['state'] + state = state.replace('struct apple_connector {', '''struct drm_device { void *dev; struct { bool mutex; } mode_config; }; +struct drm_connector { struct drm_device *dev; int registration_state; void *state; + struct { struct { unsigned min_vfreq,max_vfreq; } monitor_range; } display_info; +}; +struct apple_connector {''') + state = state.replace(' struct { int registration_state; } base;', ' struct drm_connector base;') + state = state.replace('struct apple_epic_service { struct apple_epic *ep; void *cookie; bool enabled; };', '''struct apple_epic_service { struct apple_epic *ep; void *cookie; bool enabled,torndown; + unsigned users,channel; struct { u64 generation,epoch; } admission; +};''') + state = state.replace('struct { bool lock, enabled; void *service; u64 epoch; } dcpavserv;', 'struct { bool lock, enabled; void *service; u64 epoch; int enable_completion; } dcpavserv;') + state = state.replace('u64 typec_generation; unsigned int nr_modes;', '''u64 typec_generation; unsigned int nr_modes; + struct dcp_display_mode *modes; + struct apple_epic *dcpavservep; + int fw_compat,width_mm,height_mm,fixed_connector_type; unsigned notch_height; + struct { size_t length,received; void *data; bool contiguous,payload_complete; + u64 av_service_epoch,automatic_connection_sequence; bool automatic_service_pending; + struct dcp_metadata_ticket metadata_ticket,mode_ticket; + } chunks;''') + + start = state.index('static bool dcpavserv_edid_service_current(') + state = state[:start] + + stubs = r''' +#define WRITE_ONCE(p,v) ((p)=(v)) +#define platform_device apple_dcp +#define platform_get_drvdata(p) ((struct apple_dcp *)(p)) +#define to_apple_connector(p) ((struct apple_connector *)((char *)(p)-offsetof(struct apple_connector,base))) +#define xchg(p,v) __extension__ ({ __typeof__(*(p)) old=*(p); *(p)=(v); old; }) +#define DRM_MODE_MATCH_TIMINGS 1 +#define DRM_MODE_MATCH_CLOCK 2 +#define DCP_FIRMWARE_H17P 2 +#define DPAVSERV_ENDPOINT_V2 0x2a +#define DRM_CONNECTOR_eDP 14 +#define DRM_MODE_CONNECTOR_eDP 14 +#define DCP_FW_VER 26 +#define DCP_FW_VERSION(a,b,c) (a) +#define SZ_1M (1024*1024) +#define kzalloc(n,f) calloc(1,n) +#define dev_err(...) ((void)0) +#define dev_warn(...) ((void)0) +#define dev_info(...) ((void)0) +#define dev_dbg(...) ((void)0) +#define trace_dcpavserv_init(...) ((void)0) +#include +static bool *hpd_lock,*av_lock; +static void mutex_lock(bool *p) { assert(!*p); if(p==hpd_lock) assert(!*av_lock); *p=true; } +static void mutex_unlock(bool *p) { assert(*p); *p=false; } +static void complete(int *p) { (*p)++; } +static void reinit_completion(int *p) { *p=0; } +static struct apple_epic_service *afk_service_get(struct apple_epic_service *s) { if(!s->enabled) return NULL; s->users++; return s; } +static void afk_service_put(struct apple_epic_service *s) { assert(s->users); s->users--; } +static void afk_service_retire(struct apple_epic_service *s) { assert(!*hpd_lock); s->enabled=false; s->torndown=true; } +static void dpavservep_detach(struct apple_dcp *d); +static struct apple_dcp *probing; +static struct drm_edid transport_edid; +static unsigned raw_allocations,published,reads,retirement_during_read; +static bool fail_update,invalidate_during_duplicate,replace_catalog_during_duplicate; +static void conflicting_catalog_arrived(void); +static const struct drm_edid *dcpavserv_copy_edid(struct apple_dcp *d,u64 generation,u64 *epoch) { + assert(!d->hpd_mutex && !d->dcpavserv.lock && !d->metadata.lock); + if(!d->dcpavserv.enabled || !d->dcpavserv.service || generation!=d->typec_generation) return ERR_PTR(-ENODEV); + *epoch=d->dcpavserv.epoch;reads++; + struct drm_edid *e=malloc(sizeof(*e)); assert(e); *e=transport_edid;raw_allocations++; + if(retirement_during_read) { retirement_during_read--;dpavservep_detach(d); } + return e; +} +static void drm_edid_free(const struct drm_edid *e) { if(e) { assert(raw_allocations);raw_allocations--;free((void *)e); } } +static int drm_edid_connector_update(struct drm_connector *c,const struct drm_edid *e) { + assert(c->dev->mode_config.mutex); (void)e; return fail_update?-EIO:0; +} +static const struct drm_edid *dcp_firmware_edid(struct apple_dcp *d,u64 *generation) { (void)d; (void)generation; return ERR_PTR(-ENODATA); } +static bool dcp_firmware_metadata_current(struct apple_dcp *d,u64 generation) { (void)d; (void)generation; return false; } +static bool dcp_has_panel(struct apple_dcp *d) { return d->fixed_connector_type==DRM_CONNECTOR_eDP; } +static struct drm_display_mode *drm_mode_duplicate(struct drm_device *dev,const struct drm_display_mode *m) { + assert(dev->mode_config.mutex);struct drm_display_mode *copy=malloc(sizeof(*copy));assert(copy);*copy=*m; + if(invalidate_during_duplicate) { invalidate_during_duplicate=false;dcp_180hz_invalidate(probing); } + if(replace_catalog_during_duplicate) { replace_catalog_during_duplicate=false;conflicting_catalog_arrived(); } + return copy; +} +static void drm_mode_probed_add(struct drm_connector *c,struct drm_display_mode *m) { + assert(c->dev->mode_config.mutex && probing->hpd_mutex && probing->dcpavserv.lock && probing->metadata.lock); + published++;free(m); +} +static void drm_mode_destroy(struct drm_device *dev,struct drm_display_mode *m) { (void)dev; free(m); } +static bool drm_mode_match(const struct drm_display_mode *a,const struct drm_display_mode *b,unsigned flags) { + (void)flags;return a->clock==b->clock && a->hdisplay==b->hdisplay && a->vdisplay==b->vdisplay && a->htotal==b->htotal && a->vtotal==b->vtotal && a->hsync_start==b->hsync_start && a->hsync_end==b->hsync_end && a->vsync_start==b->vsync_start && a->vsync_end==b->vsync_end && a->flags==b->flags; +} +static void drm_connector_set_vrr_capable_property(struct drm_connector *c,bool capable) { (void)c; (void)capable; } +static void dcp_retry_placeholder_edid(struct apple_dcp *d,const struct drm_edid *e) { (void)d; (void)e; } +static unsigned pending_probes; +void dcp_180hz_retry(struct apple_dcp *d) { (void)d;pending_probes++; } +static void dcp_firmware_metadata_begin(struct apple_dcp *d) { + unsigned long flags;spin_lock_irqsave(&d->metadata.lock,flags); + d->chunks.metadata_ticket=dcp_metadata_begin(&d->metadata.epoch);spin_unlock_irqrestore(&d->metadata.lock,flags); +} +static void dcp_firmware_metadata_timing(struct apple_dcp *d) { (void)d; } +static void dcp_firmware_metadata_attributes(struct apple_dcp *d,const struct dcp_display_metadata *m) { (void)d; (void)m; } +static void dcp_set_dimensions(struct apple_dcp *d) { (void)d; } +struct dcp_set_dcpav_prop_end_req { char key[64]; }; +''' + transport = '\n'.join(block(av,marker) for marker in [ + 'static void dcpavserv_notify_edid(', 'static bool dcpavserv_identity(', + 'static bool dcpavserv_admission_current(', 'static int dcpavserv_prepare(', + 'static void dcpavserv_init(', 'static void dcpavserv_invalidate_edid(', + 'static void dcpavserv_teardown(', 'void dpavservep_detach(', + 'bool dcpavserv_edid_current(', 'bool dcpavserv_edid_service_current(', + ]) + mode_helpers = iomfb[iomfb.index('static struct dcp_display_mode *dcp_modes_snapshot('):iomfb.index('int dcp_get_modes(')] + publication = block(iomfb,'static void dcp_clear_connector_edid(')+'\n'+block(iomfb,'static void dcp_consume_edid_invalidation(')+mode_helpers+block(iomfb,'int dcp_get_modes(')+block(iomfb,'struct dcp_display_mode *lookup_mode(') + # Drive malformed replacements through the same chunk/end ingress as firmware. + # The dictionary cache is a boundary substitute and takes ownership of the blob. + receipt = r''' +struct dcp_set_dcpav_prop_chunk_req { u32 offset,length; u8 data[4096]; }; +#define dcp_connector_update_dict(con,key,chunks) do { (void)(con); (void)(key); free((chunks)->data); } while (0) +''' + '\n'.join(block(template,marker) for marker in [ + 'static u8 dcpep_cb_prop_start(', 'static u8 dcpep_cb_prop_chunk(', + 'static bool dcpep_process_chunks(', 'static u8 dcpep_cb_prop_end(']) + + main_prefix = n['test'].split('int main(int argc,char **argv) {',1)[1].split(' struct apple_dcp d={',1)[0] + helpers = n['test'].split('int main(int argc,char **argv) {',1)[0] + helpers = helpers.replace(block(helpers, 'static void qualify('), '') + fixture = r''' +static struct apple_dcp d; +static struct apple_connector con; +static struct drm_device drm; +static struct apple_epic ep,av_ep; +static struct apple_epic_service s,av_services[3]; +static void reset(void) { + assert(!raw_allocations); + d=(struct apple_dcp){.dev=&d,.phy=&transport_edid,.active_typec_route=&con,.typec_cable_connected=true,.typec_generation=7,.fw_compat=2,.width_mm=600,.height_mm=330}; + drm=(struct drm_device){}; + con=(struct apple_connector){.dcp=&d,.connected=true,.base={.dev=&drm,.registration_state=1}}; + ep=(struct apple_epic){.dcp=&d,.endpoint=0x2c,.wire_v2=true}; + av_ep=(struct apple_epic){.dcp=&d,.endpoint=0x2a,.wire_v2=true}; + s=(struct apple_epic_service){.ep=&ep,.cookie=&d.dptxport[0],.enabled=true}; + for(unsigned i=0;i<3;i++) av_services[i]=(struct apple_epic_service){.ep=&av_ep,.enabled=true,.channel=1+4*i}; + d.connector=&con;d.hw.iomfb_method_profile=2;d.metadata.epoch.connected=true;d.metadata.epoch.generation=7;dcp_metadata_epoch_connection(&d.metadata.epoch,7,true); + d.dptxport[0]=(struct dptx_port){.enabled=true,.connected=true,.atcphy=d.phy,.service=&s}; + d.dcpavservep=&av_ep;hpd_lock=&d.hpd_mutex;av_lock=&d.dcpavserv.lock;probing=&d; + published=reads=pending_probes=0;fail_update=invalidate_during_duplicate=false; +} +static void admit(unsigned index) { + struct apple_epic_service *a=&av_services[index]; + assert(!dcpavserv_prepare(a,"dcpav-service-epic","AppleDCPDP2HDMI",a->channel)); + dcpavserv_init(a,"dcpav-service-epic","AppleDCPDP2HDMI",a->channel); + assert(d.dcpavserv.enabled && d.dcpavserv.service==a); +} +static void start_receipt(void) { + u32 size=sizeof(input_0);assert(dcpep_cb_prop_start(&d,&size)); + memcpy(d.chunks.data,input_0,size);d.chunks.received=size;d.chunks.contiguous=true; +} +static void finish_receipt(void) { + struct dcp_set_dcpav_prop_end_req request={.key="TimingElements"}; + assert(dcpep_process_chunks(&d,&request));assert(d.nr_modes==1 && d.modes[0].automatic_rgb8_quirk); + free(d.chunks.data);d.chunks.data=NULL; +} +static void timings(void) { start_receipt();finish_receipt(); } +static void callbacks(bool lanes) { + if(lanes) { struct dptxport_apcall_set_active_lane_count a={.lane_count=2},b={}; + assert(!dptxport_call_qualified_config(&s,true,&a,sizeof(a),&b,sizeof(b))); } + else { struct dptxport_apcall_link_rate a={.link_rate=0x1e},b={}; + assert(!dptxport_call_qualified_config(&s,false,&a,sizeof(a),&b,sizeof(b))); } +} +static void probe(bool expected) { + drm.mode_config.mutex=true;unsigned before=published; + int count=dcp_get_modes(&con.base);drm.mode_config.mutex=false; + assert(count==(int)expected && published==before+(unsigned)expected); + if(d.nr_modes) assert(!!lookup_mode(&d,&d.modes[0].mode)==expected); + assert(!d.hpd_mutex && !d.dcpavserv.lock && !d.metadata.lock); +} +static void cleanup(void) { + drm_edid_free(con.drm_edid);con.drm_edid=NULL;free(d.modes);d.modes=NULL; + free(d.chunks.data);d.chunks.data=NULL; + if(d.dcpavserv.service)dpavservep_detach(&d); + assert(!raw_allocations); +} +static void permutations(void) { + unsigned runs=0; + for(unsigned a=0;a<4;a++) for(unsigned b=0;b<4;b++) for(unsigned c=0;c<4;c++) for(unsigned e=0;e<4;e++) { + if(a==b || a==c || a==e || b==c || b==e || c==e)continue; + reset();unsigned order[]={a,b,c,e};unsigned seen=0; + for(unsigned i=0;i<4;i++) { + unsigned notifications=con.hotplug_wq.scheduled; + switch(order[i]) { case 0:callbacks(false);break;case 1:callbacks(true);break;case 2:timings();break;case 3:admit(0);break; } + seen|=1u<notifications); + } + assert(current(&d,&d.modes[0]));cleanup();runs++; + } + assert(runs==24); +} +''' + main=r''' + transport_edid=raw; + /* Runtime EDID porches are encoded into the timing fixture by this + * harness, without embedding identity bytes or fabricating tickets. */ + permutations(); + reset();callbacks(false);callbacks(true);timings();probe(false);admit(0);probe(true); + assert(d.rate_proof.rate.valid && d.rate_proof.lanes.valid); /* No second callback. */ + struct dcp_display_mode old=d.modes[0]; + dcpavserv_teardown(&av_services[0]);probe(false); + admit(1);probe(false);assert(!memcmp(&old,d.modes,sizeof(old))); /* No retagging. */ + free(d.modes);d.modes=NULL;timings();probe(true); /* Existing physical config survives AV-only CLOSE. */ + dcpavserv_teardown(&av_services[0]);probe(true); /* Old CLOSE cannot revoke the new owner. */ + cleanup(); + + reset();callbacks(false);callbacks(true);timings();admit(0);probe(true); + dcp_firmware_metadata_connection(&d,8,false);d.typec_generation=8;d.typec_cable_connected=false;dpavservep_detach(&d);probe(false); + d.typec_generation=9;d.typec_cable_connected=true;dcp_firmware_metadata_connection(&d,9,true); + free(d.modes);d.modes=NULL;timings();admit(1);probe(false); /* Fresh EDID cannot rescue old PHY configuration. */ + callbacks(false);probe(false);callbacks(true);probe(true);cleanup(); + + reset();callbacks(false);callbacks(true);start_receipt();dpavservep_detach(&d);admit(0);finish_receipt();probe(false);cleanup(); + reset();callbacks(false);callbacks(true);admit(0);start_receipt();dcpavserv_teardown(&av_services[0]);admit(1);finish_receipt();probe(false);cleanup(); + reset();callbacks(false);callbacks(true);dpavservep_detach(&d);timings();admit(0);probe(true);cleanup(); /* Legitimate receipt awaiting post-retirement admission. */ + + reset();callbacks(false);callbacks(true);timings();admit(0);retirement_during_read=1;probe(false);cleanup(); + reset();callbacks(false);callbacks(true);timings();admit(0);invalidate_during_duplicate=true;probe(false);cleanup(); + reset();callbacks(false);callbacks(true);timings();admit(0);fail_update=true;probe(false);assert(!con.drm_edid);cleanup(); + reset();callbacks(false);callbacks(true);timings();admit(0);transport_edid.data[10]^=1;checksum(transport_edid.data);probe(false);cleanup();transport_edid=raw; + reset();callbacks(false);callbacks(true);timings();admit(0);configure_error=-EIO; + struct dptxport_apcall_link_rate req={.link_rate=0x1e},resp={};assert(dptxport_call_qualified_config(&s,false,&req,sizeof(req),&resp,sizeof(resp))==-EIO); + configure_error=0;probe(false);cleanup(); + puts("PASS: 24 callback/timing/AV arrival permutations and real get_modes/lookup publication; early configuration needs no second callback; CLOSE, physical reconnect, receipt epochs and publication races deny stale proof"); +} +''' + unit=n['prefix']+n['metadata']+n['epoch']+n['header']+n['edid_header']+n['parser']+n['blobs']+n['policy']+n['proof']+state+stubs+transport+n['source']+n['callback_types']+n['callback_stubs']+n['callback']+n['connection']+publication+receipt+helpers+fixture+'int main(int argc,char **argv) {'+main_prefix+main + # Conflicts are scanned across the full binary receipt, including filtered + # siblings. Exercise both orders, replacement receipts and equivalent aliases. + base_fixture=copy.deepcopy(n['fixture']) + conflict_blobs='' + conflict_checks='' + for kind in ('color', 'timing', 'other-color'): + for reverse in (False, True): + candidate=copy.deepcopy(base_fixture) + other=copy.deepcopy(candidate[0]) + if kind=='color': + other_color=copy.deepcopy(other['ColorModes'][0]) + other_color.update(Depth=10,EOTF=2,Score=999999) + candidate[0]['ColorModes'].insert(0 if reverse else 1,other_color) + elif kind=='other-color': + other.update(ID=79,DSCRequiredColorElementIDs=[80]) + other['ColorModes'][0].update(Depth=10,EOTF=2) + candidate.insert(0 if reverse else 1,other) + else: + other['HorizontalAttributes'].update(Total=2200,FrontPorch=88,SyncWidth=44) + other['VerticalAttributes'].update(Total=1125,FrontPorch=4,SyncWidth=5,PreciseSyncRate=60<<16) + other.update(DSCRequiredColorElementIDs=[80]) + candidate.insert(0 if reverse else 1,other) + raw_blob=bytearray(struct.pack('before);assert(!lookup_mode(&d,&saved));free(old); } + cleanup(); + if(late) { + reset();callbacks(false);callbacks(true);admit(0);timings();probe(true);old=d.modes;saved=old[0].mode; + u32 size=len;assert(dcpep_cb_prop_start(&d,&size));memcpy(d.chunks.data,blob,size); + d.chunks.received=size;d.chunks.contiguous=true; + before=con.hotplug_wq.scheduled; + replace_catalog_during_duplicate=true;probe(false); + assert(con.hotplug_wq.scheduled>before); + assert(!current(&d,old) && !lookup_mode(&d,&saved));free(old);cleanup(); + } +} +static void conflicting_receipts(void) { + CONFLICT_CHECKS + reset();callbacks(false);callbacks(true);admit(0);receipt_blob(equivalent_ids,sizeof(equivalent_ids));probe(true);cleanup(); + puts("PASS: conflicting color/timing IDs, discarded siblings, both receipt orders and late conflicts deny getter/lookup; equivalent color aliases remain eligible"); +} +""".replace('CONFLICT_CHECKS',conflict_checks) + unit=unit.replace('int main(int argc,char **argv) {',conflict_blobs+conflict_helper+'int main(int argc,char **argv) {') + unit=unit.replace(' permutations();',' permutations();conflicting_receipts();') + # Execute the production metadata notifier so a manual probe cannot hide a + # missing connector refresh after the last eligible mode is withdrawn. + unit=unit.replace('bool lock, attributes_valid, timing_valid, hpd_seen; struct dcp_metadata_epoch epoch;', + 'bool lock, attributes_valid, timing_valid, hpd_seen; struct dcp_metadata_ticket attributes_ticket,timing_ticket; struct drm_display_mode preferred; struct dcp_metadata_epoch epoch;') + dcp_source=(base/'dcp.c').read_text() + metadata_notifier='\n'.join(block(dcp_source,marker) for marker in ( + 'static void dcp_firmware_metadata_notify(', + 'static bool dcp_firmware_metadata_complete(', + 'void dcp_firmware_metadata_timing(')) + unit=unit.replace(block(unit,'static void dcp_firmware_metadata_timing('), + '#define dcp_trace_metadata(d,p,v) ((void)0)\nstatic bool dcp_firmware_metadata_enabled(struct apple_dcp *d);\n'+metadata_notifier) + + failed_receipts = r''' +static bool malformed_receipt(const char *key,unsigned fault) { + u32 size=fault==1 ? 3 : fault==2 ? 4 : sizeof(input_0); + assert(dcpep_cb_prop_start(&d,&size)); + struct dcp_set_dcpav_prop_chunk_req chunk={.length=size}; + assert(size<=sizeof(chunk.data));memcpy(chunk.data,input_0,size); + if(fault==0)chunk.data[0]^=1; + assert(dcpep_cb_prop_chunk(&d,&chunk)); + struct dcp_set_dcpav_prop_end_req request={}; + strcpy(request.key,key); + bool accepted=dcpep_cb_prop_end(&d,&request); + assert(!d.chunks.data && !d.chunks.length); + return accepted; +} +static void failed_replacements(void) { + for(unsigned fault=0;fault<3;fault++) { + reset();callbacks(false);callbacks(true);timings();admit(0);probe(true); + struct dcp_display_mode *old=d.modes; + struct drm_display_mode saved=old[0].mode; + unsigned before=con.hotplug_wq.scheduled; + assert(!malformed_receipt("TimingElements",fault)); + assert(!current(&d,old) && !lookup_mode(&d,&saved)); + assert(con.hotplug_wq.scheduled>before);probe(false); + /* Fresh catalog can recover using still-current EDID and link proof. */ + timings();probe(true);free(old);cleanup(); + } + reset();callbacks(false);callbacks(true);timings();admit(0);probe(true); + assert(!malformed_receipt("DisplayAttributes",0));probe(true);cleanup(); + puts("PASS: actual chunk/end callbacks withdraw malformed replacement exceptions and notify DRM; fresh catalogs recover; unrelated bad properties preserve admission"); +} +''' + unit=unit.replace('int main(int argc,char **argv) {',failed_receipts+'int main(int argc,char **argv) {') + unit=unit.replace(' permutations();conflicting_receipts();',' permutations();conflicting_receipts();failed_replacements();') + return {**locals(), "block": block} + +def hotplug(root): + n = publication(root) + unit = n['unit'] + block = n['block'] + base = root / 'drivers/gpu/drm/apple' + dptx = (base / 'dptxep.c').read_text() + training = (base / 'dptx-j700-training.h').read_text() + unit = unit.replace('struct dptx_port { u32 lane_count, link_rate;', ''' +union phy_configure_opts { struct { unsigned lanes,link_rate; bool set_lanes,set_rate,set_voltages; } dp; }; +struct dptx_port { union phy_configure_opts phy_ops; int linkcfg_completion; u32 pending_link_rate; +u32 lane_count, link_rate;''') + unit = unit.replace('int fw_compat,width_mm,height_mm,fixed_connector_type;', 'unsigned dptx_phy; int fw_compat,width_mm,height_mm,fixed_connector_type;') + real_handlers = r''' +#define cpu_to_le32(v) (v) +#define cpu_to_le64(v) (v) +#define LINK_RATE_RBR 0x06 +#define LINK_RATE_HBR 0x0a +#define LINK_RATE_HBR2 0x14 +#define PHY_MODE_DP 1 +#define trace_dptxport_call_set_link_rate(...) ((void)0) +static unsigned phy_capacity=2, phy_calls, hardware_rate, hardware_lanes; +static int validate_error; +static int phy_validate(void *phy,int mode,int submode,union phy_configure_opts *o) { + assert(phy);(void)mode;(void)submode; + if(validate_error)return validate_error; + o->dp.lanes=phy_capacity;return 0; +} +static int phy_configure(void *phy,union phy_configure_opts *o) { + assert(phy);phy_calls++; + if(configure_error)return configure_error; + if(o->dp.set_lanes)hardware_lanes=o->dp.lanes; + if(o->dp.set_rate)hardware_rate=o->dp.link_rate; + return 0; +} +static void dcp_tunnel_set_rate(struct apple_dcp *d,void *phy,unsigned rate) { + (void)d;(void)phy;(void)rate;assert(false); +} +''' + real_handlers += block(training, 'static int dptx_j700_validate_lanes(') + '\n' + real_handlers += block(dptx, 'static int dptxport_call_set_active_lane_count(') + '\n' + real_handlers += block(dptx, 'static int dptxport_call_set_link_rate(') + '\n' + for marker in ['static int dptxport_call_set_active_lane_count(', 'static int dptxport_call_set_link_rate(']: + unit = unit.replace(block(unit, marker), '') + unit = unit.replace('static int configure_error;', 'static int configure_error;\n' + real_handlers) + unit = unit.replace('published=reads=pending_probes=0;', 'phy_capacity=2;phy_calls=hardware_rate=hardware_lanes=0;validate_error=0;published=reads=pending_probes=0;') + unit = unit.replace('assert(current(&d,&d.modes[0]));cleanup();runs++;', 'assert(current(&d,&d.modes[0]));assert(hardware_rate==8100 && hardware_lanes==2 && phy_calls==2);cleanup();runs++;') + unit = unit.replace('struct apple_dcp {', '''struct apple_dcp { + struct { bool lock,presentation_paused; } iomfb; + struct { bool valid,changing; } mode_state; + struct work_struct external_stop_work,vblank_wq; + bool external_physically_absent,main_display,typec_crtc_off,external_cable,external_error,external_power_attempted,external_stopped; + bool external_hotplug_pending,external_hotplug_connected,typec_poweron_hpd; + u64 external_stop_generation,external_rx_generation,external_crtc_generation; + unsigned swap_watchdog_retrains;int typec_iomfb_hpd_ready;''') + stubs = r''' +#define DCP_HOTPLUG_NOTIFY 1 +static bool dcp_is_external_h17p(struct apple_dcp *d) { return d->hw.iomfb_method_profile==2; } +static bool dcp_is_typec_output(struct apple_dcp *d) { (void)d;return true; } +static void complete_all(int *p) { (*p)++; } +#define dcp_mode_invalidate(s) ((s)->valid=false) +#define dcp_mode_hotplug(s,c,p) __extension__ ({ (void)(s);if(p)*(p)=(c);1u; }) +static void dcp_handle_hotplug_actions(struct apple_dcp *d,unsigned action) { (void)d;(void)action; } +static bool dcp_typec_relink(struct apple_dcp *d) { (void)d;return false; } +''' + dcp = (root / 'drivers/gpu/drm/apple/dcp.c').read_text() + template = (root / 'drivers/gpu/drm/apple/iomfb_template.c').read_text() + functions = stubs + block(dcp, 'bool dcp_external_hotplug(') + '\n' + block(template, 'static void dcpep_cb_hotplug(') + + main=r"""int main(int argc,char **argv) { MAIN_PREFIX + transport_edid=raw;permutations();conflicting_receipts();failed_replacements(); + unsigned runs=0; + for(unsigned a=0;a<4;a++)for(unsigned b=0;b<4;b++)for(unsigned c=0;c<4;c++)for(unsigned e=0;e<4;e++) { + if(a==b||a==c||a==e||b==c||b==e||c==e)continue; + reset();callbacks(false);callbacks(true);timings();admit(0);probe(true); + d.external_cable=true;d.external_power_attempted=true;d.external_rx_generation=7; + d.iomfb.lock=true;u64 connected=1;dcpep_cb_hotplug(&d,&connected);d.iomfb.lock=false; + struct dcp_display_mode *old=d.modes;struct drm_display_mode saved=old[0].mode; + u64 sequence=d.rate_proof.sequence,connection=d.rate_proof.connection_sequence; + d.iomfb.lock=true;connected=0;dcpep_cb_hotplug(&d,&connected);d.iomfb.lock=false; + assert(d.external_stop_work.scheduled==1 && d.external_stop_generation==7); + assert(sequence!=d.rate_proof.sequence && connection!=d.rate_proof.connection_sequence); + assert(!d.rate_proof.rate.valid && !d.rate_proof.lanes.valid); + assert(!current(&d,old) && !lookup_mode(&d,&saved)); + assert(con.edid_invalidation_pending);probe(false); /* Work deliberately not executed. */ + dpavservep_detach(&d);dcp_180hz_invalidate(&d); /* Deferred retirement/release. */ + unsigned order[]={a,b,c,e}; + for(unsigned i=0;i<4;i++) { + switch(order[i]) { case 0:callbacks(false);break;case 1:callbacks(true);break;case 2:timings();break;case 3:admit(1);break; } + probe(false);assert(!current(&d,old)); + } + assert(d.rate_proof.rate.valid && d.rate_proof.lanes.valid); + /* Repeated low reports before reassertion are not a new HPD loss. */ + sequence=d.rate_proof.sequence;d.iomfb.lock=true;connected=0; + dcpep_cb_hotplug(&d,&connected);d.iomfb.lock=false; + assert(d.rate_proof.sequence==sequence && d.rate_proof.rate.valid && d.rate_proof.lanes.valid); + d.iomfb.lock=true;connected=1;dcpep_cb_hotplug(&d,&connected);d.iomfb.lock=false; + assert(d.metadata.epoch.connected && d.metadata.epoch.invalidated); + probe(true);assert(!current(&d,old));assert(phy_calls==4 && hardware_rate==8100 && hardware_lanes==2); + free(old);cleanup();runs++; + } + assert(runs==24); + /* Initial bootstrap false and power/paused suppression keep proof intact. */ + reset();callbacks(false);callbacks(true);timings();admit(0);probe(true); + d.iomfb.lock=true;u64 connected=0;u64 sequence=d.rate_proof.sequence; + dcpep_cb_hotplug(&d,&connected);assert(d.rate_proof.sequence==sequence && !con.edid_invalidation_pending); + connected=1;dcpep_cb_hotplug(&d,&connected); + connected=0;d.typec_crtc_off=true;dcpep_cb_hotplug(&d,&connected);assert(d.rate_proof.sequence==sequence); + d.typec_crtc_off=false;d.iomfb.presentation_paused=true;dcpep_cb_hotplug(&d,&connected); + assert(d.rate_proof.sequence==sequence && !con.edid_invalidation_pending);d.iomfb.lock=false;probe(true); + dcpavserv_teardown(&av_services[0]);cleanup(); + puts("PASS: actual RX HPD loss immediately denies getter/lookup before deferred work; 24 fresh-proof recovery orders need only one rate/lane callback; bootstrap/power/paused suppression preserved"); +}""".replace('MAIN_PREFIX',n['main_prefix']) + unit=unit.replace(block(unit,'int main(int argc,char **argv) {'),functions+'\n'+main) + return unit + + +def metadata(root, output): + """Keep malformed/duplicate metadata cases with synthetic bounded wire input.""" + n = parser_fixture(root) + fixture = n['fixture'] + cases = [(fixture, False)] + for path, value in [ + (('MaxHorizontalImageSize',), 256), (('MaxVerticalImageSize',), '33'), + (('Chromaticity', 'Red', 'X'), True), (('Chromaticity', 'Blue', 'Y'), 65536), + (('Chromaticity', 'Green', 'Y'), 64000), (('DefaultWhitePoint', 'Gamma'), 65535), + (('DefaultWhitePoint', 'Gamma'), 240000), (('ProductAttributes', 'ManufacturerID'), 'AC'), + (('ProductAttributes', 'ManufacturerID'), 'A\0R'), (('ProductAttributes', 'LegacyManufacturerID'), 1), + (('ProductAttributes', 'ProductName'), 'x' * 14), (('ProductAttributes', 'ProductID'), 65536), + (('ProductAttributes', 'YearOfManufacture'), 1989), (('ProductAttributes', 'WeekOfManufacture'), 54), + ]: + bad = copy.deepcopy(fixture) + target = bad + for key in path[:-1]: + target = target[key] + target[path[-1]] = value + cases.append((bad, True)) + for key in ('Chromaticity', 'DefaultWhitePoint', 'ProductAttributes'): + bad = copy.deepcopy(fixture) + del bad[key] + cases.append((bad, True)) + second = copy.deepcopy(fixture[key]) + cases.append((n['ObjectPairs'](list(fixture.items()) + [(key, second)]), True)) + for path, field in [(('Chromaticity', 'Red'), 'X'), (('ProductAttributes',), 'ProductName')]: + bad = copy.deepcopy(fixture) + target = bad + for key in path[:-1]: + target = target[key] + value = target[path[-1]] + target[path[-1]] = n['ObjectPairs'](list(value.items()) + [(field, value[field])]) + cases.append((bad, True)) + inputs = output / 'metadata-inputs' + inputs.mkdir(parents=True, exist_ok=True) + arguments = [] + for index, (value, bad) in enumerate(cases): + data = bytearray(struct.pack('H', 1138) + assert raw[21:24] == bytes([60, 33, 120]) + assert raw[38:54] == b'\x01' * 16 and raw[35:38] == b'\0' * 3 + assert struct.unpack_from('> 4) << 8) == 1920 + assert raw[57] + ((raw[58] & 15) << 8) == 280 + assert raw[59] + ((raw[61] >> 4) << 8) == 1080 + assert raw[60] + ((raw[61] & 15) << 8) == 45 + assert raw[62:65] == bytes([88, 44, 0x45]) and raw[71] == 0x1e + chroma = [raw[27] * 4 + ((raw[25] >> 6) & 3), raw[28] * 4 + ((raw[25] >> 4) & 3), + raw[29] * 4 + ((raw[25] >> 2) & 3), raw[30] * 4 + (raw[25] & 3), + raw[31] * 4 + ((raw[26] >> 6) & 3), raw[32] * 4 + ((raw[26] >> 4) & 3), + raw[33] * 4 + ((raw[26] >> 2) & 3), raw[34] * 4 + (raw[26] & 3)] + assert chroma == [697, 317, 265, 632, 146, 81, 321, 337] + assert raw[72:77] == bytes.fromhex('000000fc00') + + +def retry(root): + source = (root / 'drivers/gpu/drm/apple/dcp-180hz.c').read_text() + production = '\n'.join(block(source, marker) for marker in ( + 'static void dcp_180hz_retry_work(', 'void dcp_180hz_retry(', 'static void dcp_180hz_cancel(')) + prefix = r""" +#include +#include +#include +#include +#include +typedef uint64_t u64; +#define READ_ONCE(x) (x) +#define WRITE_ONCE(p,v) ((p)=(v)) +#define guard(kind) (void) +#define spin_lock_irqsave(p,f) do { assert(!*(p)); *(p)=true; (f)=0; } while (0) +#define spin_unlock_irqrestore(p,f) do { (void)(f); assert(*(p)); *(p)=false; } while (0) +#define container_of(p,t,m) ((t *)((char *)(p)-offsetof(t,m))) +#define to_delayed_work(p) container_of(p,struct delayed_work,work) +#define to_platform_device(p) (p) +#define msecs_to_jiffies(x) (x) +struct work_struct { unsigned scheduled; }; +struct delayed_work { struct work_struct work; unsigned delay; bool pending; }; +struct apple_connector { void *dcp; struct work_struct hotplug_wq; }; +struct apple_dcp { + bool hpd_mutex, typec_cable_connected, quiescing, rate_retry_initialized; + struct { bool lock; } metadata; + struct { u64 retry_generation; unsigned retries; } rate_proof; + u64 typec_generation; + struct apple_connector *connector; + void *dev; + struct delayed_work rate_retry_wq; +}; +static int system_freezable_wq; +static unsigned submissions, cancellations; +static void schedule_work(struct work_struct *w) { w->scheduled++; } +static void mod_delayed_work(int q,struct delayed_work *w,unsigned delay) +{ (void)q; submissions++; w->pending=true; w->delay=delay; } +static void cancel_delayed_work_sync(struct delayed_work *w) +{ cancellations++; w->pending=false; } +""" + controls = r""" +int main(void) +{ + struct apple_dcp d={.typec_cable_connected=true,.typec_generation=7}; + struct apple_connector c={.dcp=&d}; d.connector=&c; d.dev=&d; + dcp_180hz_retry(&d); assert(!submissions); + d.rate_retry_initialized=true; + for(unsigned i=0;i<100;i++) dcp_180hz_retry(&d); + assert(submissions==8 && d.rate_proof.retries==8 && d.rate_retry_wq.delay==300); + dcp_180hz_retry_work(&d.rate_retry_wq.work); assert(c.hotplug_wq.scheduled==1); + d.typec_generation++; dcp_180hz_retry_work(&d.rate_retry_wq.work); + assert(c.hotplug_wq.scheduled==1); + d.typec_cable_connected=false; dcp_180hz_retry(&d); assert(submissions==8); + d.typec_cable_connected=true; dcp_180hz_retry(&d); assert(submissions==9 && d.rate_proof.retries==1); + d.quiescing=true; dcp_180hz_retry(&d); dcp_180hz_retry_work(&d.rate_retry_wq.work); + assert(submissions==9 && c.hotplug_wq.scheduled==1); + d.quiescing=false; c.dcp=NULL; dcp_180hz_retry_work(&d.rate_retry_wq.work); + assert(c.hotplug_wq.scheduled==1); + c.dcp=&d; dcp_180hz_retry_work(&d.rate_retry_wq.work); assert(c.hotplug_wq.scheduled==2); + dcp_180hz_cancel(&d); assert(!d.rate_retry_initialized && !d.rate_retry_wq.pending && cancellations==1); + dcp_180hz_retry(&d); assert(submissions==9); + puts("PASS: actual eight-retry bound, 300ms delay, stale-generation/disconnect/quiesce/ownership gates and cancellation"); +} +""" + return prefix + production + controls diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/recovery.py b/tools/testing/selftests/drivers/gpu/apple-dcp/recovery.py new file mode 100644 index 00000000000000..310af408b0f96e --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/recovery.py @@ -0,0 +1,439 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +# Copyright 2026 Ryan Murray +"""Retained reviewer controls: actual AFK sender/receive, queue and EDID functions. + +The wrap controls deliberately retain an expected replay assertion failure. +They document an unresolved peer contract, not successful replay protection. +""" +from pathlib import Path +import json, hashlib, subprocess + +def check(root, output): + import edid + out = output / 'service-recovery-controls' + def extract(text,signature): + start=text.index(signature);opening=text.index('{',start) + while ';' in text[start:opening]: + start=text.index(signature,start+1);opening=text.index('{',start) + end,depth=opening+1,1 + while depth: + depth+=(text[end]=='{')-(text[end]=='}');end+=1 + return text[start:end] + + unit=edid.rebind(root) + # Retain the original exact fixture separately. Correct only its causal ordering. + corrected=unit.replace('assert(afk_service_get(old)==old);old->cmd_map[0]=1;', + 'assert(afk_service_get(old)==old);old->cmd_tag=1;old->cmd_map[0]=1;') + corrected=corrected.replace(' old->cmd_tag=1;\n','') + + sender=extract((root/'drivers/gpu/drm/apple/afk.c').read_text(), + 'static int afk_send_command_timeout_len(') + unit=unit[:unit.index('int main(void)')] + unit=unit.replace('static unsigned int dma_frees,opens;',r''' + static unsigned int dma_frees,opens; + #define SZ_1M (1U<<20) + #define GFP_KERNEL 0 + #define DECLARE_COMPLETION_ONSTACK(c) int c=0 + #define init_completion(c) (*(c)=0) + #define msecs_to_jiffies(x) (x) + #define cpu_to_le64(x) (x) + #define kmalloc(n,g) malloc(n) + #define EPIC_TYPE_COMMAND 3 + #define EPIC_CAT_COMMAND 3 + static int bitmap_find_free_region(unsigned long *p,int n,int order) { + (void)order;for(int i=0;iinline_v2) { + struct afkv2_command decoded;u8 response[40]={0}; + assert(!afkv2_decode_command(p,n,&decoded) && !decoded.oob && decoded.value==32 && decoded.id==tag); + response[1]=decoded.id; + assert(!afkv2_encode_frame(current_reply,sizeof(current_reply),1,channel,0,subtype,AFKV2_RESPONSE,0,response,sizeof(response),¤t_size)); + current_channel=channel;current_tag=tag;return 0; + } + assert(n==sizeof(struct epic_cmd)); + const struct epic_cmd *cmd=p;u8 oob[32]; + assert(!afkv2_encode_oob_command(oob,sizeof(oob),tag,cmd->rxbuf,cmd->rxlen,cmd->txbuf,cmd->txlen)); + afkv2_put32(oob+4,0); + assert(!afkv2_encode_frame(current_reply,sizeof(current_reply),1,channel,0,subtype,AFKV2_RESPONSE,0,oob,sizeof(oob),¤t_size)); + current_channel=channel;current_tag=tag;return 0; + } + '''+marker) + + tests=r''' + static void setup(void) { + d.hw.iomfb_method_profile=2;d.typec_cable_connected=true;d.typec_generation=1; + d.nr_modes=1;d.connector=&con;d.dev=&d; + con.dcp=&d;con.connected=true;con.base.registration_state=1; + ep.dcp=&d;ep.wire_v2=true;ep.endpoint=0x2a;ep.wq=&ep;ep.command_tags=calloc(AFKV2_INTERFACES,1);assert(ep.command_tags); + d.dcpavservep=&ep;ep.ops=announcement_ops;ep.match_epic_name=true; + d.fw_compat=DCP_FIRMWARE_H17P;hpd_lock=&d.hpd_mutex;av_lock=&d.dcpavserv.lock;active_ep=&ep; + } + static int send_actual(struct apple_epic_service *s,bool ack,bool late) { + u8 payload[32]={0},output[32]={0};u32 code; + ack_current=ack;inject_late=late; + int result=afk_send_command_timeout_len(s,0xc0,payload,32,output,32,&code,1,NULL); + inject_late=false;return result; + } + static void save_ack(void) { memcpy(saved_reply,current_reply,current_size);saved_size=current_size; } + static void cleanup(void) { + dpavservep_detach(&d);run_retries(); + for(unsigned int i=0;ichannel==5); + assert(send_actual(middle,true,true)==0); + closed(5,0x6a01,0);announce(13,name,native_class,13);run_retries(); + assert(d.dcpavserv.service && d.dcpavserv.service==middle && d.dcpavserv.service!=old); + assert(send_actual(d.dcpavserv.service,true,true)==0); + afk_service_put(old); + } else if(!strcmp(argv[1],"multiple-first-late-acks")) { + reuse_dma=false;assert(afk_service_get(old)==old); + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + u8 first[56];size_t first_size=saved_size;memcpy(first,saved_reply,first_size); + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + assert(old->cmd_map[0]==3); + closed(13,0x6a01,0);announce(13,name,native_class,13);run_retries();assert(!d.dcpavserv.service); + afk_v2_receive(&ep,0,0,first,first_size);run_retries(); + assert(!d.dcpavserv.service && old->cmd_map[0]==2 && old->users==1); + unsigned int freed=dma_frees; + afk_v2_receive(&ep,0,0,first,first_size);run_retries(); + assert(dma_frees==freed && old->cmd_map[0]==2 && !d.dcpavserv.service); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + assert(d.dcpavserv.service && d.dcpavserv.service!=old && !old->cmd_map[0]); + afk_service_put(old); + } else if(!strcmp(argv[1],"tag-wrap") || !strcmp(argv[1],"inline-tag-wrap")) { + assert(send_actual(old,true,false)==0);save_ack(); + for(unsigned int i=0;i<15;i++) assert(send_actual(old,true,true)==0); + assert(send_actual(old,true,true)==0); + } else if(!strcmp(argv[1],"pending-closed")) { + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + closed(13,0x6a01,0);announce(13,name,native_class,13);run_retries(); + assert(!d.dcpavserv.service); + closed(13,0x6a01,0); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + assert(!d.dcpavserv.service && "CLOSED during pending recovery must invalidate the pending OPEN"); + } else if(!strcmp(argv[1],"actual-tag-before-announcement")) { + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack();assert(ep.command_tags[13]==1); + closed(13,0x6a01,0);announce(13,name,native_class,13); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + assert(d.dcpavserv.service && send_actual(d.dcpavserv.service,true,true)==0); + } else if(!strncmp(argv[1],"open-race-",10)) { + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + closed(13,0x6a01,0);announce(13,name,native_class,13);run_retries(); + if(!strcmp(argv[1],"open-race-generation")) open_action=1; + else if(!strcmp(argv[1],"open-race-epoch")) open_action=2; + else if(!strcmp(argv[1],"open-race-owner")) open_action=4; + else if(!strcmp(argv[1],"open-race-cable")) open_action=5; + else if(!strcmp(argv[1],"open-race-quiescing")) open_action=6; + else if(!strcmp(argv[1],"open-race-absence")) open_action=7; + else assert(!"unknown OPEN race"); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + if(open_action==4) assert(d.dcpavserv.service==injected_owner); + else assert(!d.dcpavserv.service); + open_action=0;d.quiescing=false;d.typec_cable_connected=true;d.external_physically_absent=false; + } else if(!strcmp(argv[1],"physical-absence")) { + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + closed(13,0x6a01,0);announce(13,name,native_class,13);run_retries(); + d.external_physically_absent=true; + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + assert(!d.dcpavserv.service && "physical absence authority must fence deferred OPEN before cable/generation update"); + } else if(!strncmp(argv[1],"fence-",6)) { + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack(); + closed(13,0x6a01,0);announce(13,name,native_class,13);run_retries(); + unsigned int before=opens; + if(!strcmp(argv[1],"fence-epoch")) dpavservep_detach(&d); + else if(!strcmp(argv[1],"fence-generation")) d.typec_generation++; + else if(!strcmp(argv[1],"fence-cable")) d.typec_cable_connected=false; + else if(!strcmp(argv[1],"fence-quiescing")) d.quiescing=true; + else if(!strcmp(argv[1],"fence-owner")) inject_owner(&ep,13); + else if(!strcmp(argv[1],"fence-open-failure")) open_error=-EIO; + else assert(!"unknown fence"); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);run_retries(); + assert(opens==before+(!strcmp(argv[1],"fence-open-failure"))); + if(!strcmp(argv[1],"fence-owner")) assert(d.dcpavserv.service==injected_owner); + else assert(!d.dcpavserv.service); + for(unsigned int i=0;ifn=(f);(w)->pending=false;(w)->next=NULL; } while(0) + static struct work_struct *fifo_head,*fifo_tail;static bool destroyed,rx_running; + ''') + queued=queued.replace('static bool queue_work(void *wq,struct work_struct *w) { assert(wq);bool fresh=!w->pending;w->pending=true;return fresh; }',r''' + static bool queue_work(void *wq,struct work_struct *w) { + assert(wq && !destroyed);if(w->pending) return false; + w->pending=true;w->next=NULL;if(fifo_tail) fifo_tail->next=w;else fifo_head=w;fifo_tail=w;return true; + } + static void run_one_work(void) { + assert(fifo_head);struct work_struct *w=fifo_head;fifo_head=w->next;if(!fifo_head) fifo_tail=NULL; + w->pending=false;w->next=NULL;assert(w->fn);w->fn(w); + } + static void destroy_workqueue(void *wq) { assert(wq && !destroyed);while(fifo_head) run_one_work();destroyed=true; } + #define IS_ERR_OR_NULL(p) (!(p) || IS_ERR(p)) + #define debugfs_remove_recursive(p) ((void)(p)) + #define kzalloc(n,g) calloc(1,n) + #define kvfree(p) free(p) + #define ESHUTDOWN 108 + #define RBEP_TYPE 0xffff + #define FIELD_GET(mask,val) ((val)&(mask)) + #define RBEP_INIT_ACK 0xa0 + #define RBEP_START_ACK 0x86 + #define RBEP_SHUTDOWN_ACK 0xc1 + #define RBEP_GETBUF 0x89 + #define RBEP_INIT_TX 0x8a + #define RBEP_INIT_RX 0x8b + #define RBEP_RECV 0x85 + #define complete_all(p) complete(p) + ''') + queued=queued.replace('struct apple_dcp_afkep { u8 *command_tags;bool lock,stopping;', + 'struct apple_dcp_afkep { u8 *command_tags;int started,stopped,txbfr,rxbfr;void *debugfs_entry;bool lock,stopping;') + queued=queued.replace('bool external_physically_absent,hpd_mutex,quiescing,typec_cable_connected;', + 'bool retain_dma,external_physically_absent,hpd_mutex,quiescing,typec_cable_connected;') + queued=queued.replace('static void run_retries(void) { assert(!ep.lock);if(ep.service_work.pending) { ep.service_work.pending=false;afk_service_worker(&ep.service_work); } }', + 'static void run_retries(void) { assert(!ep.lock);while(fifo_head) run_one_work(); }') + queued=queued.replace(' if(type==EPIC_TYPE_COMMAND) {', + ' if(e->stopping) return -ESHUTDOWN;\n if(type==EPIC_TYPE_COMMAND) {') + queued=queued[:queued.index('int main(int argc')] + queued+=r''' + struct afk_receive_message_work { struct apple_dcp_afkep *ep;u64 message;struct work_struct work; }; + static void afk_getbuf(struct apple_dcp_afkep *e,u64 m) { (void)e;(void)m;assert(!"unexpected GETBUF"); } + static void afk_init_rxtx(struct apple_dcp_afkep *e,u64 m,int *r) { (void)e;(void)m;(void)r;assert(!"unexpected INIT"); } + static unsigned int rx_step;static bool close_pending; + static bool afk_recv(struct apple_dcp_afkep *e) { + assert(e==&ep);rx_running=true; + if(rx_step==0) closed(13,0x6a01,0); + else if(rx_step==1) announce(13,name,native_class,13); + else if(rx_step==2) afk_v2_receive(e,0,0,saved_reply,saved_size); + else if(rx_step==3 && close_pending) closed(13,0x6a01,0); + else { assert(!d.dcpavserv.service && "recovery must wait for the RX drain loop to finish");rx_running=false;return false; } + assert(!d.dcpavserv.service && "recovery must remain behind RX on the ordered queue");rx_step++;return true; + } + ''' + af=(root/'drivers/gpu/drm/apple/afk.c').read_text() + for signature in ['static void afk_receive_message_worker(', 'int afk_receive_message(', 'void afk_quiesce(', 'static void afk_release_context(']: + queued+=extract(af,signature)+'\n' + queued+=r''' + int main(int argc,char **argv) { + assert(argc==2);setup();reuse_dma=true;INIT_WORK(&ep.service_work,afk_service_worker); + announce(13,name,native_class,13);struct apple_epic_service *old=d.dcpavserv.service;assert(old); + assert(send_actual(old,false,false)==-ETIMEDOUT);save_ack();run_retries(); + unsigned int before=opens; + if(!strncmp(argv[1],"ordered-",8)) { + close_pending=!strcmp(argv[1],"ordered-pending-closed"); + assert(!afk_receive_message(&ep,RBEP_RECV));assert(fifo_head); + run_one_work();assert(!rx_running && opens==before && !d.dcpavserv.service); + assert(fifo_head==&ep.service_work);run_retries(); + if(close_pending) assert(!d.dcpavserv.service && "ordered CLOSED must cancel pending OPEN"); + else { assert(d.dcpavserv.service && opens==before+1);assert(send_actual(d.dcpavserv.service,true,true)==0); } + cleanup(); + } else { + assert(!strcmp(argv[1],"quiesce-pending"));assert(afk_service_get(old)==old); + closed(13,0x6a01,0);announce(13,name,native_class,13); + afk_v2_receive(&ep,0,0,saved_reply,saved_size);assert(ep.service_work.pending); + afk_quiesce(&ep);assert(destroyed && !ep.wq && ep.stopping && !fifo_head); + assert(opens==before && !d.dcpavserv.service && old->users==1); + for(unsigned int i=0;iusers); + assert(afk_receive_message(&ep,RBEP_RECV)==-ESHUTDOWN);assert(!fifo_head); + afk_quiesce(&ep);assert(!fifo_head); + struct apple_dcp_afkep *empty=calloc(1,sizeof(*empty));assert(empty);empty->dcp=&d; + afk_release_context(empty); + cleanup(); + } + printf("PASS actual RX worker/quiesce: %s\n",argv[1]); + } + ''' + run('actual-queue-quiesce',queued,[[c] for c in ['ordered-recovery','ordered-pending-closed','quiesce-pending']]) + + # Actual EDID wrapper exercises its owner/reader pins, epoch and generation exits. + reader=unit[:unit.index('int main(int argc')] + reader+=r''' + #define ESTALE 116 + struct drm_edid { int marker; }; + static unsigned int edid_allocations,edid_frees;static int read_action; + struct reader_guard { bool *lock;bool once; }; + static struct reader_guard lock_reader_guard(bool *lock) { mutex_lock(lock);return (struct reader_guard){lock,true}; } + static void unlock_reader_guard(struct reader_guard *guard) { mutex_unlock(guard->lock); } + #define scoped_guard(kind,p) for(struct reader_guard _guard __attribute__((cleanup(unlock_reader_guard)))=lock_reader_guard(p);_guard.once;_guard.once=false) + static void drm_edid_free(const struct drm_edid *edid) { assert(edid && !IS_ERR(edid));free((void *)edid);edid_frees++; } + static const struct drm_edid *dcpavserv_read_edid_inline(struct apple_epic_service *service) { + assert(service->users==2 && "owner plus actual EDID wrapper reader pin"); + assert(send_actual(service,true,false)==0); + if(read_action==1) { closed(service->channel,0x6a01,0);announce(5,name,class,5);assert(d.dcpavserv.service!=service); } + if(read_action==2) d.typec_generation++; + if(read_action==3) d.external_physically_absent=true; + if(read_action==4) { dpavservep_detach(&d);d.quiescing=true; } + struct drm_edid *edid=malloc(sizeof(*edid));assert(edid);edid->marker=1;edid_allocations++;return edid; + } + static const struct drm_edid *dcpavserv_read_edid_legacy(struct apple_epic_service *service) { return dcpavserv_read_edid_inline(service); } + ''' + dpav=(root/'drivers/gpu/drm/apple/epic/dpavservep.c').read_text() + for signature in ['bool dcpavserv_edid_current(', 'bool dcpavserv_edid_service_current(', 'const struct drm_edid *dcpavserv_copy_edid(']: + reader+=extract(dpav,signature)+'\n' + reader+=r''' + int main(int argc,char **argv) { + assert(argc==2);setup();reuse_dma=true;announce(13,name,class,13); + struct apple_epic_service *old=d.dcpavserv.service;assert(old); + if(!strcmp(argv[1],"reader-owner")) read_action=1; + else if(!strcmp(argv[1],"reader-generation")) read_action=2; + else if(!strcmp(argv[1],"reader-physical-absence")) read_action=3; + else if(!strcmp(argv[1],"reader-quiescing")) read_action=4; + else assert(!strcmp(argv[1],"reader-success")); + u64 epoch;const struct drm_edid *edid=dcpavserv_copy_edid(&d,1,&epoch); + if(!read_action) { assert(!IS_ERR(edid));drm_edid_free(edid); } + else { assert(IS_ERR(edid) && "actual EDID wrapper must reject invalidated ownership/physical state"); } + if(read_action==1 || read_action==4) assert(!old->users);else assert(old->users==1); + assert(edid_allocations==edid_frees && !d.hpd_mutex && !d.dcpavserv.lock); + d.quiescing=false;cleanup();printf("PASS actual copy_edid: %s\n",argv[1]); + } + ''' + run('actual-edid-reader',reader,[[c] for c in ['reader-success','reader-owner','reader-generation','reader-quiescing','reader-physical-absence']]) + mutations=[ + ('restart-tags','afkv2_command_id(ep->command_tags[service->channel]++, slot)','afkv2_command_id(0, slot)'), + ('no-drain-retry','\tafk_service_retry(service->ep);',''), + ('ignore-old-command','\t\tif (afk_epic_find_service(ep, service->channel))\n\t\t\tcontinue;',''), + ('ignore-epoch','service->admission.epoch != dcp->dcpavserv.epoch ||\n\t\t ','false ||\n\t\t '), + ('refresh-generation','!dcpavserv_admission_current(dcp, service->admission.generation))\n\t\tret = -ENODEV;','!dcpavserv_admission_current(dcp, dcp->typec_generation))\n\t\tret = -ENODEV;'), + ('ignore-owner','if (dcp->dcpavserv.enabled)\n\t\tret = -EBUSY;\n\telse if (service->admission.epoch','if (false)\n\t\tret = -EBUSY;\n\telse if (service->admission.epoch'), + ('ignore-stop','if (!READ_ONCE(ep->stopping))\n\t\tafk_retry_service_init(ep);','if (true)\n\t\tafk_retry_service_init(ep);'), + ] + + for name,text,case in [ + ('cancel-closed',unit.replace('if (pending->pending_open && pending->channel == channel)','if (false && pending->pending_open && pending->channel == channel)'), 'pending-closed'), + ('reader-lookup',unit.replace('(service->enabled ||','(service->users || service->enabled ||'), 'drained-reader-rebind'), + ('reset-interface-history',unit.replace('service->cookie = NULL;', 'service->cookie = NULL; if(ep->wire_v2 && ep->command_tags) ep->command_tags[channel]=0;',1), 'channel-reuse'), + ('absence-admission',unit.replace('!READ_ONCE(dcp->external_physically_absent) &&','true &&',1), 'initial-absence'), + ('absence-publication',unit.replace('!READ_ONCE(dcp->external_physically_absent) &&','true &&',1), 'initial-open-absence'), + ('absence-reader',reader.replace('!READ_ONCE(dcp->external_physically_absent)', 'true'), 'reader-physical-absence'), + ]: + run('mutation-'+name,text,[[case]]) + + for name,old,new in mutations: + assert corrected.count(old)==1,(name,corrected.count(old)) + run('mutation-'+name,corrected.replace(old,new),[[]]) + negative={ + 'actual-sender':{'tag-wrap','inline-tag-wrap'}, + 'actual-queue-quiesce':set(), + 'actual-edid-reader':set(), + } + for result in results: + case=result.get('case',[]) + fail_expected=bool('case' in result and (result['name'].startswith('mutation-') or (case and case[0] in negative.get(result['name'],set())))) + result['assertion_failure_expected']=fail_expected + result['expectation_met']=result['exit']==(-6 if fail_expected else 0) + if fail_expected: result['expectation_met'] = result['expectation_met'] and 'Assertion' in result.get('output','') + if case and case[0] in {'tag-wrap','inline-tag-wrap'}: + result['expectation_met'] = result['expectation_met'] and 'late ACK completed an actual new production command' in result.get('output','') + (out/'results.json').write_text(json.dumps(results,indent=2)+'\n') + + assert all(r['expectation_met'] for r in results), "retained recovery control expectation failed; see results.json" diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/run.py b/tools/testing/selftests/drivers/gpu/apple-dcp/run.py new file mode 100644 index 00000000000000..e64cc11a1eda4d --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/run.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only OR MIT +# Copyright 2026 Ryan Murray +"""Host regressions for Apple external display ownership and recovery. + +Only public synthetic fixtures and extracted production functions are used. +The transport, MMIO, locks and work queues are mocks, not hardware validation. +""" +import argparse +import copy +import hashlib +import json +import os +from pathlib import Path +import struct +import tempfile +import traceback + +from common import run_unit +import edid +import recovery +import lifecycle +import phy +import pm +import quiesce +import transaction +import accounting +import idle +import rate +import boot + + +def parking_unit(root): + fixture = phy.parking(root) + return ''.join(fixture[k] for k in ('prefix', 'production', 'suffix')) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--source', type=Path, default=Path(__file__).resolve().parents[6]) + parser.add_argument('--output', type=Path, help='retain generated C, commands and results here') + parser.add_argument('--group', choices=('all', 'edid', 'rate', 'phy', 'lifecycle', 'boot', 'quiesce'), default='all') + args = parser.parse_args() + root = args.source.resolve() + output = args.output or Path(tempfile.mkdtemp(prefix='apple-dcp-selftest-')) + output.mkdir(parents=True, exist_ok=True) + apple = root / 'drivers/gpu/drm/apple' + includes = (apple, root / 'tools/testing/selftests/drivers/gpu/apple-afkv2/host-include') + parking_cases = tuple((c,) for c in ('mux', 'power', 'reset', 'cleanup', 'orientation', + 'orientation-faults', 'backend-change', 'safe-first', + 'failed-flip-return', 'live-park')) + tests = [ + ('boot', 'oslog-events', boot.oslog, ((),), ()), + ('edid', 'service-rebind', edid.rebind, ((),), ()), + ('edid', 'provider-admission', edid.admission, ((),), ()), + ('edid', 'ordered-rx', edid.rx_progress, ((),), ('-pthread',)), + ('rate', 'parser-policy', lambda r: rate.policy(r)['unit'], ((),), ()), + ('rate', 'publication-hpd', rate.hotplug, ((),), ()), + ('rate', 'metadata-decoder', lambda r: rate.metadata(r, output), ((),), ('-Wno-sign-compare',)), + ('rate', 'bounded-retry', rate.retry, ((),), ()), + ('phy', 'pm-ownership', pm.ownership, ((),), ()), + ('phy', 'controller-lifecycle', pm.controller_lifecycle, ((),), ('-pthread',)), + ('phy', 'asynchronous-root-state', pm.asynchronous_root_state, ((),), ()), + ('phy', 'upstream-resume-boundary', pm.upstream_resume_boundary, ((),), ()), + ('phy', 'usb-unreachable', lambda r: pm.ownership(r, True), ((),), ()), + ('lifecycle', 'cpuidle-s2idle', idle.s2idle, ((),), ('-Wno-unused-parameter', '-Wno-unused-variable')), + ('lifecycle', 'typec-probe', pm.probe, ((),), ()), + ('lifecycle', 'quiesce-before-safe', quiesce.make_unit, tuple((c,) for c in ('ordered','power','ack','role','bolt','mux','duplicate','error-latched','legacy','fabric','tunnel')), ()), + ('lifecycle', 'probe-mux-pm-dispatch', pm.probe_dispatch, ((),), ()), + ('phy', 'parking', lambda r: parking_unit(r), parking_cases, ()), + ('phy', 'adverse-retry', phy.adverse, tuple((c,) for c in ('bist', 'init', 'mux', 'gate', 'dp-cache', 'off', 'success')), ()), + ('phy', 'masked-context', phy.usb_context, ((),), ()), + ('phy', 'off-hardening', phy.hardening, ((),), ()), + ('phy', 'warm-reset', phy.warm_reset, ((),), ()), + ('phy', 'pipe', phy.pipe, tuple((c,) for c in ('backend', 'host', 'mux', 'same', 'park', 'requested', 'scope')), ()), + ('phy', 'unwind', phy.unwind, ((),), ()), + ('phy', 'aux', phy.aux, ((),), ()), + ('lifecycle', 'physical-disconnect', lifecycle.disconnect, ((),), + ('-Wno-unused-variable', '-Wno-unused-function', '-Wno-unused-but-set-variable')), + ('lifecycle', 'dwc3-teardown', lifecycle.teardown, (('all',),), ()), + ('lifecycle', 'resource-profile', lifecycle.profile, ((),), ('-Wno-sign-compare',)), + ('lifecycle', 'firmware-provider', lifecycle.firmware, ((),), ()), + ('lifecycle', 'readiness', lifecycle.readiness, ((),), ()), + ('lifecycle', 'vrr-rejection', lifecycle.vrr, ((),), ()), + ('lifecycle', 'power-reply', lifecycle.power_reply, ((),), ()), + ('lifecycle', 'dispatcher-scope', lifecycle.dispatcher, ((),), ()), + ] + transaction_tests = transaction.controls(root) + accounting.controls(root) + tests.extend(('lifecycle', name, lambda r, u=unit: u, cases, flags) + for name, unit, cases, flags in transaction_tests) + tests.append(('phy', 'usb-kconfig', lambda r: pm.kconfig(r, output), (), ())) + tests = [t for t in tests if args.group in ('all', t[0]) or (args.group == 'quiesce' and t[1] in ('quiesce-before-safe',) + tuple(n for n, _, _, _ in transaction_tests))] + print('TAP version 13', flush=True) + print('1..' + str(len(tests)), flush=True) + results = [] + for index, (group, name, build, cases, flags) in enumerate(tests, 1): + result = {'name': name, 'group': group, 'passed': False} + try: + unit = build(root) + if unit is None: + result['producer_passed'] = True + elif isinstance(unit, tuple): + unit, cases = unit + if unit is not None: + result['translation_unit_sha256'] = run_unit(name, unit, output, cases, includes if name == 'service-rebind' else (), flags) + if name == 'service-rebind': + recovery.check(root, output) + if name == 'metadata-decoder': + rate.check_metadata(output) + result['passed'] = True + except Exception: + result['failure'] = traceback.format_exc() + (output / (name + '-failure.log')).write_text(result['failure']) + print(('ok ' if result['passed'] else 'not ok ') + str(index) + ' - ' + name, flush=True) + results.append(result) + sources = {} + for area in ('drivers/gpu/drm/apple', 'drivers/phy/apple', 'drivers/usb/dwc3', 'drivers/usb/typec/tipd'): + for path in sorted((root / area).rglob('*')): + if path.is_file() and path.suffix in ('.c', '.h'): + sources[str(path.relative_to(root))] = hashlib.sha256(path.read_bytes()).hexdigest() + (output / 'results.json').write_text(json.dumps({'source': str(root), 'sources': sources, 'results': results}, indent=2) + '\n') + print('# Artifacts: ' + str(output), flush=True) + return int(any(not r['passed'] for r in results)) + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/run.sh b/tools/testing/selftests/drivers/gpu/apple-dcp/run.sh new file mode 100755 index 00000000000000..f4052ba6632f1c --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/run.sh @@ -0,0 +1,4 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only OR MIT +PYTHONDONTWRITEBYTECODE=1; export PYTHONDONTWRITEBYTECODE +exec python3 "$(dirname "$0")/run.py" "$@" diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/teardown_pm.py b/tools/testing/selftests/drivers/gpu/apple-dcp/teardown_pm.py new file mode 100644 index 00000000000000..1f17261136a9c3 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/teardown_pm.py @@ -0,0 +1,156 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Actual wrapper/driver-core/DWC3/xHCI teardown order with modeled scheduling. + +Device mutex APIs use pthread mutexes. Clock, unregister, PM and hardware leaves +are mocks; this is not lockdep, system-PM or electrical atomicity qualification. +""" +import pm +from common import block + +def unit_for(source): + unit=pm.ownership(source) + unit=unit.replace('int main(void)','int __attribute__((unused)) ownership_main(void)',1) + unit=unit.replace('struct device_driver { const char *name; };', 'struct device;\nstruct device_driver { const char *name; void (*remove)(struct device *); void *dev_groups; };') + unit=unit.replace('struct device { int refs;', 'struct bus_type { bool need_parent_lock; void (*remove)(struct device *); void (*dma_cleanup)(struct device *); };\nstruct device_private { int knode_driver; };\nstruct device { struct bus_type *bus; struct device_private *p; int kobj; int refs;') + unit=unit.replace('static struct device host_device;', 'static struct platform_device host_platform_device;\n#define host_device host_platform_device.dev') + unit=unit.replace('static bool controller_locked, host_locked;', 'static _Atomic bool controller_locked, host_locked;') + pre=r''' +#include +#include +static pthread_mutex_t controller_mutex=PTHREAD_MUTEX_INITIALIZER, host_mutex=PTHREAD_MUTEX_INITIALIZER; +static pthread_mutex_t schedule_mutex=PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t schedule_cond=PTHREAD_COND_INITIALIZER; +static _Thread_local bool remover; +static atomic_bool handoff_active, early_write; +static bool removal_started, blocked_attempt, release_remove, inject_remove; +static int remove_target, susphy_calls, host_remove_calls; +static pthread_t remove_thread; +static void start_remove(void); +static void removal_lock_attempt(bool held) { + if(!remover||!held)return; + pthread_mutex_lock(&schedule_mutex);blocked_attempt=true; + pthread_cond_broadcast(&schedule_cond);pthread_mutex_unlock(&schedule_mutex); +} +static void observed_controller_write(void) { + if(atomic_load(&handoff_active))atomic_store(&early_write,true); + pthread_mutex_lock(&schedule_mutex); + removal_started=true;pthread_cond_broadcast(&schedule_cond); + while(atomic_load(&handoff_active)&&!release_remove)pthread_cond_wait(&schedule_cond,&schedule_mutex); + pthread_mutex_unlock(&schedule_mutex); +} +''' + unit=unit.replace('#define CONFIG_USB 1',pre+'\n#define CONFIG_USB 1',1) + old=block(unit,'static void device_lock(') + new=r'''static void device_lock(struct device *d) { + if(d==&controller.dev){ + removal_lock_attempt(atomic_load(&controller_locked)); + pthread_mutex_lock(&controller_mutex);controller_locked=true; + }else{ + assert(d==&host_device);removal_lock_attempt(atomic_load(&host_locked)); + pthread_mutex_lock(&host_mutex);host_locked=true; + } +}''' + unit=unit.replace(old,new) + old=block(unit,'static void device_unlock(') + new=r'''static void device_unlock(struct device *d) { + if(d==&host_device){assert(host_locked);host_locked=false;pthread_mutex_unlock(&host_mutex);} + else {assert(d==&controller.dev&&controller_locked);if(!remover)atomic_store(&handoff_active,false);controller_locked=false;pthread_mutex_unlock(&controller_mutex);} +}''' + unit=unit.replace(old,new) + # The scheduler injection is a child-lookup interleaving, as in the review. + unit=unit.replace('if(detach_at_lookup)usb_set_device_state(u,USB_STATE_NOTATTACHED);','if(inject_remove){inject_remove=false;start_remove();}\n if(detach_at_lookup)usb_set_device_state(u,USB_STATE_NOTATTACHED);',1) + unit=unit.replace('static void pm_runtime_put_noidle(struct device *d) {','static void pm_runtime_put_noidle(struct device *d) {\n if(remover&&d==&host_device)return;',1) + # Old source lacks platform-lock helpers. Keep fixture-only API definitions + # available without claiming they are invoked by the parent. + if 'device_lock(&pm->controller->dev)' not in (source/'drivers/phy/apple/atc.c').read_text(): + unit=unit.replace('static struct device *get_device(', 'static __attribute__((unused)) struct device *get_device(') + prefix=r''' +struct clk { int unused; }; struct reset_control { int unused; }; +struct xhci_hcd { atomic_int xhc_state; struct clk *clk,*reg_clk; struct reset_control *reset; struct usb_hcd *shared_hcd; }; +static struct xhci_hcd xhci_context; +#define XHCI_STATE_REMOVING 1 +#define hcd_to_xhci(h) (&xhci_context) +struct dwc3 { bool sys_wakeup; struct platform_device *xhci; }; +enum { DWC3_APPLE_PROBE_PENDING, DWC3_APPLE_NO_CABLE, DWC3_APPLE_HOST, DWC3_APPLE_DEVICE }; +struct dwc3_apple { struct dwc3 dwc; struct device *dev; int lock,state; void *role_sw; }; +static struct dwc3_apple apple_context; +#define to_dwc3_apple(d) container_of(d,struct dwc3_apple,dwc) +static bool inside_host_remove; +static void *platform_get_drvdata(struct platform_device *d) { return d==&controller?(void *)&apple_context.dwc:(void *)&hcd; } +static void dwc3_enable_susphy(struct dwc3 *d,bool v) { (void)d;assert(v);susphy_calls++;observed_controller_write(); } +static void device_init_wakeup(struct device *d,bool b) { (void)d;(void)b; } +static void platform_device_unregister(struct platform_device *d); +static void dwc3_gadget_exit(struct dwc3 *d) { (void)d;assert(0); } +static int dwc3_apple_core_stop(struct dwc3_apple *d) { (void)d;return 0; } +static void usb_role_switch_unregister(void *s) { (void)s; } +static void dwc3_core_remove_no_shutdown(struct dwc3 *d) { (void)d; } +static void usb_remove_hcd(struct usb_hcd *h) { host_remove_calls++;h->state=HC_STATE_QUIESCING;usb_set_device_state(h->self.root_hub,USB_STATE_NOTATTACHED); } +static void usb_phy_shutdown(void *p) { (void)p; } +static void usb_put_hcd(struct usb_hcd *h) { (void)h; } +static void clk_disable_unprepare(struct clk *c) { (void)c; } +static void reset_control_assert(struct reset_control *r) { (void)r; } +static void pm_runtime_disable(struct device *d) { (void)d; } +static void pm_runtime_set_suspended(struct device *d) { (void)d; } +static int pm_runtime_get_sync(struct device *d) { if(inside_host_remove&&d==&host_device){assert(xhci_context.xhc_state&XHCI_STATE_REMOVING);observed_controller_write();}return 0; } +static void pm_runtime_put_sync(struct device *d) { (void)d; } +static void pm_runtime_put(struct device *d) { (void)d; } +#define dev_attr_state_synced 0 +#define BUS_NOTIFY_UNBIND_DRIVER 0 +#define BUS_NOTIFY_UNBOUND_DRIVER 1 +#define KOBJ_UNBIND 0 +static bool device_links_busy(struct device *d) { (void)d;return false; } +static void device_links_unbind_consumers(struct device *d) { (void)d;assert(0); } +static void driver_sysfs_remove(struct device *d) { (void)d; } +static void bus_notify(struct device *d,int n) { (void)d;(void)n; } +static void device_remove_file(struct device *d,void *a) { (void)d;(void)a; } +static void device_remove_groups(struct device *d,void *a) { (void)d;(void)a; } +static void device_unbind_cleanup(struct device *d) { d->driver=NULL; } +static void device_links_driver_cleanup(struct device *d) { (void)d; } +static void klist_remove(void *k) { (void)k; } +static void device_pm_check_callbacks(struct device *d) { (void)d; } +static void kobject_uevent(void *k,int n) { (void)k;(void)n; } +''' + prefix=prefix.replace('#define dev_attr_state_synced 0','static int dev_attr_state_synced;') + unit=unit.replace('struct usb_hcd { int state;', 'struct usb_hcd { void *usb_phy; int state;',1) + # Extract actual callback order; clock/USB/reset leaves remain mocks. + host=(source/'drivers/usb/dwc3/host.c').read_text() + glue=(source/'drivers/usb/dwc3/dwc3-apple.c').read_text() + plat=(source/'drivers/usb/host/xhci-plat.c').read_text() + dd=(source/'drivers/base/dd.c').read_text() + functions=block(host,'void dwc3_host_exit(')+'\n'+block(glue,'static int dwc3_apple_exit(')+'\n'+block(glue,'static void dwc3_apple_remove(')+'\n'+block(plat,'void xhci_plat_remove(')+'\n' + functions+='\n'.join(block(dd,x) for x in ['static void __device_driver_lock(', 'static void __device_driver_unlock(', 'static void device_remove(', 'static void __device_release_driver(', 'void device_release_driver_internal(']) + controls=r''' +static void controller_remove(struct device *d) { assert(d==&controller.dev);dwc3_apple_remove(&controller); } +static void host_remove(struct device *d) { assert(d==&host_device);inside_host_remove=true;xhci_plat_remove(&host_platform_device);inside_host_remove=false; } +static void platform_device_unregister(struct platform_device *d) { assert(d==&host_platform_device);device_release_driver_internal(&d->dev,NULL,NULL); } +static void *run_remove(void *unused) { (void)unused;remover=true;device_release_driver_internal(remove_target?&host_device:&controller.dev,NULL,NULL);return NULL; } +static void start_remove(void) { + pthread_mutex_lock(&schedule_mutex); + assert(pthread_create(&remove_thread,NULL,run_remove,NULL)==0); + while(!blocked_attempt&&!removal_started)pthread_cond_wait(&schedule_cond,&schedule_mutex); + pthread_mutex_unlock(&schedule_mutex); +} +int main(void) { + for(remove_target=0;remove_target<2;remove_target++) { + struct apple_atcphy p;prepare(&p,true,true,false); + struct device_private private={0};controller.dev.p=&private;host_device.p=&private; + controller_driver.remove=controller_remove;host_driver.remove=host_remove; + apple_context=(struct dwc3_apple){.dwc={.xhci=&host_platform_device},.dev=&controller.dev,.state=DWC3_APPLE_HOST}; + xhci_context=(struct xhci_hcd){0}; + inject_remove=true;removal_started=blocked_attempt=release_remove=false; + atomic_store(&early_write,false);atomic_store(&handoff_active,true); + susphy_calls=host_remove_calls=0; + int ret=safe(&p); + atomic_store(&handoff_active,false); + pthread_mutex_lock(&schedule_mutex);release_remove=true;pthread_cond_broadcast(&schedule_cond);pthread_mutex_unlock(&schedule_mutex); + assert(pthread_join(remove_thread,NULL)==0); + fprintf(stderr,"connected teardown target=%d ret=%d early_write=%d susphy=%d host_remove=%d\n",remove_target,ret,atomic_load(&early_write),susphy_calls,host_remove_calls); + assert(ret==0&&!atomic_load(&early_write)&&host_remove_calls==1); + assert(susphy_calls==(remove_target?0:1));balanced(&p); + } + puts("PASS actual PM wrapper connected to driver-core/DWC3/xHCI removal; synchronous teardown waits until owned transaction ends"); + return 0; +} +''' + return unit+prefix+functions+controls diff --git a/tools/testing/selftests/drivers/gpu/apple-dcp/transaction.py b/tools/testing/selftests/drivers/gpu/apple-dcp/transaction.py new file mode 100644 index 00000000000000..fe4143c84ac571 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/apple-dcp/transaction.py @@ -0,0 +1,407 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright 2026 Ryan Murray +"""Exact production callbacks with the adverse-review counterexample assertions. + +Firmware, MMIO and USB role operations are synthetic leaves. The concurrency +case executes production transactions with real pthread connector/fabric locks. +""" +import quiesce +from common import block + + +def controls(ROOT): + tests = [] + def check(name, unit, cases=((),), negative=False, flags=()): + assert not negative + tests.append((name, unit, cases, flags)) + base=quiesce.make_unit(ROOT) + # Actual generic role helper, omitted by the author's role-leaf fixture. + core=(ROOT/'drivers/usb/typec/tipd/core.c').read_text() + generic=base.replace(block(base,'int main('),block(base,'int main(').replace('int main(','int __attribute__((unused)) prior_main(',1),1) + generic+=r''' + int main(void) { + struct cd321x c;initial(&c);data.remove=NULL;role_error=-EBUSY; + tipd_remove(&c.tps); + fprintf(stderr,"generic role failure: role_calls=%d error=%d reset=%d gpio=%d events=%s\n",role_calls,last_error,reset_calls,gpio_calls,events); + assert(last_error==-EBUSY&&reset_calls==0&&gpio_calls==0&&unregisters==1); + } + ''' + check('generic-role-error-lost',generic) + second_role=generic.replace('data.remove=NULL;role_error=-EBUSY;', 'role_error=-EBUSY;',1) + second_role=second_role.replace("role_calls++;return role_error;", "role_calls++;return role_calls==1?0:role_error;",1) + check('cd-second-role-error-lost',second_role) + + # Replace the modeled DCP set/deactivate bodies with exact production bodies. + dcp=(ROOT/'drivers/gpu/drm/apple/dcp.c').read_text() + actual=base.replace(block(base,'int main('),block(base,'int main(').replace('int main(','int __attribute__((unused)) prior_main(',1),1) + actual=actual.replace('struct typec_mux_state { void *alt;', 'struct typec_altmode { unsigned svid; };\nstruct typec_mux_state { struct typec_altmode *alt;',1) + actual=actual.replace('struct connector { bool connected; };','struct encoder { unsigned possible_crtcs; };\nstruct connector { bool connected; void *dcp; struct encoder *port_encoder; unsigned candidate_crtcs; };\n#define apple_connector connector\nstruct mux_control { int unused; };',1) + actual=actual.replace('struct apple_dcp { int external_lock,external_error;', '''struct apple_dcp { int external_lock,external_error; + bool external_power_attempted,dptx_tunnel,tb_clock_ok,fixed_route_selected; + int tb_lock; unsigned typec_generation,dptx_dfp_port,dptx_phy,fixed_dptx_phy,fixed_mux_index; + void *tb_dpin_set_active,*tb_dpin_ctx,*phy,*fixed_phy; + struct mux_control *xbar; + struct apple_dcp_typec_route *active_typec_route,*typec_routes; + struct connector *connector,*fixed_connector;int connector_type,fixed_connector_type; + ''',1) + actual=actual.replace('bool applied_valid,safe_pending,dp_wanted,dp_hpd,hpd;', 'struct typec_altmode *applied_alt; unsigned long applied_mode; unsigned applied_status,applied_conf; bool applied_valid,safe_pending,dp_wanted,dp_hpd,hpd;',1) + actual=actual.replace('void *xbar; bool deactivate_xbar_done; int deactivate_error; bool selected,tunnel;', 'struct mux_control *xbar,*active_xbar; void *phy; unsigned dptx_phy,tunnel_dpin,mux_index;bool deactivate_xbar_done; int deactivate_error;bool xbar_up,selected,tunnel;',1) + extra=r''' + #define WRITE_ONCE(x,v) ((x)=(v)) + #define USB_TYPEC_DP_SID 0xff01 + #define TYPEC_DP_STATE_A 10 + #define TYPEC_DP_STATE_F 15 + #define DP_STATUS_HPD_STATE 1 + #define DP_STATUS_IRQ_HPD 2 + #define DRM_MODE_CONNECTOR_USB 1 + #define MUX_IDLE_DISCONNECT -1 + #define symbol_get(fn) (&fn) + #define symbol_put(fn) ((void)0) + struct typec_displayport_data { u32 status,conf; }; + static int gpiod_get_value_cansleep(void *p); + static int dcp_dpxbar_preselect(struct mux_control *m,int s) { (void)m;(void)s;return 0; } + static int dcp_dpxbar_tunnel_select_source(struct mux_control *m,int s) { (void)m;(void)s;return 0; } + static int apple_atc_dp_tunnel_rate(void *p,unsigned n,unsigned rate) { (void)p;(void)n;(void)rate;return 0; } + static int xbar_error,xbar_calls; +static int mux_control_deselect(struct mux_control *m) { (void)m;event('C');xbar_calls++;return xbar_error; } + static int mux_control_select(struct mux_control *m,unsigned n) { (void)m;(void)n;return 0; } + static void dpavservep_detach(struct apple_dcp *d) { (void)d; } + static void dcp_firmware_metadata_connection(struct apple_dcp *d,unsigned n,bool c) { (void)d;(void)n;(void)c; } + static bool dcp_typec_keep_order(void) { return false; } + static struct apple_dcp_typec_route *dcp_typec_lowest_free(struct apple_dcp_typec_port *p) { (void)p;return &route; } + static int connect_calls; + static int dcp_dptx_connect_oob(void *dev,unsigned n) { (void)dev;(void)n;connect_calls++;display.external_cable=true;display.dptxport[0].connected=true;display.typec_cable_connected=true;connector.connected=true;return 0; } + static void dcp_retrain_oob(struct connector *c) { (void)c; } + static bool dcp_typec_sink_relink(struct apple_dcp *d) { (void)d;return false; } + ''' + pos=actual.index('static int dcp_typec_route_deactivate(') + actual=actual[:pos]+extra+actual[pos:] + actual=actual.replace(block(actual,'static bool dcp_typec_route_available('), 'static bool dcp_typec_route_fixed_output_busy(struct apple_dcp_typec_route *r) { (void)r;return false; }\n'+block(dcp,'static bool dcp_typec_route_available('),1) + actual=actual.replace(block(actual,'static int dcp_typec_route_deactivate('),block(dcp,dcp.rindex('static int dcp_typec_route_deactivate(')),1) + pos=actual.index('static struct typec_mux_dev phy_provider') + actual=actual[:pos]+block(dcp,'static bool dcp_typec_route_is_dp(')+'\n'+block(dcp,'static int dcp_typec_route_set(')+'\n'+actual[pos:] + actual=actual.replace(block(actual,'static int display_set('),'''static int display_set(struct typec_mux_dev *m,struct typec_mux_state *s) { event('S');return dcp_typec_route_set(m,s); }''',1) + actual=actual.replace("(void)m;assert(s->mode==TYPEC_STATE_SAFE);event('L');", "(void)m;(void)s;event('L');",1) + actual=actual.replace('connector=(struct connector){.connected=true};','connector=(struct connector){.connected=true};',1) + actual=actual.replace('phy_provider=(struct typec_mux_dev){.set=phy_set};','display.typec_routes=&route;display.fixed_connector_type=DRM_MODE_CONNECTOR_USB;display.active_typec_route=&route;\n phy_provider=(struct typec_mux_dev){.set=phy_set};',1) + controls=r''' +static int late_finish_error(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;(void)s;return -EIO; } + int main(int argc,char **argv) { + (void)argc;struct cd321x c;initial(&c);struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE}; + if(argv[1][0]=='o') { + assert(typec_mux_set(&mux,&safe)==0); + assert(stops==1&&!port.owner&&!port.safe_pending&&!route.selected); + puts("PASS actual DCP prepare + route_set + route_deactivate + framework normal SAFE"); + } + if(argv[1][0]=='r') { + mux.mux_devs[0]=&display_provider;mux.mux_devs[1]=&phy_provider;phy_error=-ETIMEDOUT; + assert(typec_mux_set(&mux,&safe)==-ETIMEDOUT); + fprintf(stderr,"display-first failed SAFE: owner=%p lease=%d selected=%d events=%s\n",(void *)port.owner,port.safe_pending,route.selected,events); + assert(port.owner==&route&&port.safe_pending&&route.selected); + } + if(argv[1][0]=='d') { + phy_error=-ETIMEDOUT;assert(typec_mux_set(&mux,&safe)==-ETIMEDOUT&&port.safe_pending&&port.owner==&route); + phy_error=0;struct typec_altmode alt={.svid=USB_TYPEC_DP_SID}; + struct typec_displayport_data data={.status=DP_STATUS_HPD_STATE}; + struct typec_mux_state dp={.alt=&alt,.mode=TYPEC_DP_STATE_A,.data=&data}; + int ret=typec_mux_set(&mux,&dp); + fprintf(stderr,"DP after failed SAFE: ret=%d connects=%d pending=%d wanted=%d hpd=%d\n",ret,connect_calls,port.safe_pending,port.dp_wanted,port.hpd); + assert(ret<0&&connect_calls==0&&!port.dp_wanted); + } + if(argv[1][0]=='a') { + struct altmode alt={.mux=&mux,.adev={.svid=USB_TYPEC_DP_SID}};phy_error=-ETIMEDOUT; + int ret=typec_altmode_set_mux(&alt,TYPEC_STATE_SAFE,NULL); + fprintf(stderr,"altmode SAFE: ret=%d stops=%d phy_calls=%d lease=%d\n",ret,stops,phy_calls,port.safe_pending); + assert(ret==-ETIMEDOUT&&stops==1&&port.safe_pending); + } + if(argv[1][0]=='x') { + xbar_error=-EIO;assert(typec_mux_set(&mux,&safe)==-EIO); + assert(port.owner==&route&&port.safe_pending&&!port.safe_finish_ready&&route.selected&&route.deactivate_xbar_done&&route.deactivate_error==-EIO); + xbar_error=0;assert(typec_mux_set(&mux,&safe)==-EIO&&port.owner==&route&&port.safe_pending&&xbar_calls==1); + } + if(argv[1][0]=='f') { + mux.mux_devs[0]=&display_provider;mux.mux_devs[1]=&phy_provider;phy_provider.finish=late_finish_error; + assert(typec_mux_set(&mux,&safe)==-EIO); + assert(port.owner==&route&&port.safe_pending&&port.safe_finish_ready&&!route.selected&&xbar_calls==1); + assert(!dcp_typec_route_available(&route)); + struct apple_dcp peer={0};struct apple_dcp_typec_route peer_route={.dcp=&peer}; + assert(dcp_typec_route_available(&peer_route)); + phy_provider.finish=NULL; + assert(typec_mux_set(&mux,&safe)==0&&!port.owner&&!port.safe_pending&&xbar_calls==1); + } + return 0; + } + ''' + bus=(ROOT/'drivers/usb/typec/bus.c').read_text() + actual+='\nstruct altmode { struct typec_mux *mux; struct typec_altmode adev; };\n'+block(bus,'static inline int\ntypec_altmode_set_mux(')+'\n'+controls + check('actual-dcp-normal-safe',actual,(('ordered',),)) + check('actual-dcp-finalizer-recovery',actual,(('xbar-error',),('finalizer',))) + check('provider-order-releases-failed-lease',actual,(('release',),)) + check('failed-safe-allows-new-dp-admission',actual,(('dp',),)) + check('altmode-safe-skips-quiescence',actual,(('altmode',),)) + + # Real pthread fabric/consumer mutexes. Pause the SAFE actor after its actual + # DCP prepare has returned and released its locks, before any set callback. + # A second handle can reference the same providers (class/altmode/CD handles + # are separately acquired in the real framework). + concurrent=actual.replace(block(actual,'int main('),block(actual,'int main(').replace('int main(','int __attribute__((unused)) route_controls_main(',1),1) + concurrent=concurrent.replace('#define guard(k) (void)','') + concurrent=concurrent.replace('#define scoped_guard(k,l) for(int once=((void)(l),1);once;once=0)','') + locks=r''' + #include + static pthread_mutex_t fabric_mutex=PTHREAD_MUTEX_INITIALIZER,external_mutex=PTHREAD_MUTEX_INITIALIZER,tb_mutex=PTHREAD_MUTEX_INITIALIZER; + struct real_guard { pthread_mutex_t *mutex; bool held; }; + static struct real_guard take_lock(int *key) { + pthread_mutex_t *m=key==&dcp_typec_fabric_lock?&fabric_mutex:key==&display.external_lock?&external_mutex:&tb_mutex; + assert(pthread_mutex_lock(m)==0);return (struct real_guard){m,true}; + } + static void drop_lock(struct real_guard *g) { if(g->held){g->held=false;assert(pthread_mutex_unlock(g->mutex)==0);} } + #define guard(k) struct real_guard __attribute__((cleanup(drop_lock))) _guard = take_lock + #define scoped_guard(k,l) for(struct real_guard __attribute__((cleanup(drop_lock))) _scope=take_lock(l);_scope.held;drop_lock(&_scope)) + static bool observed_unquiesced_safe; + ''' + pos=concurrent.index('static int dcp_typec_route_deactivate(') + concurrent=concurrent[:pos]+locks+concurrent[pos:] + concurrent=concurrent.replace("(void)m;(void)s;event('L');phy_calls++;", "(void)m;if(s->mode==TYPEC_STATE_SAFE&&display.dptxport[0].connected)observed_unquiesced_safe=true;event('L');phy_calls++;",1) + concurrent+=r''' + static pthread_mutex_t schedule_lock=PTHREAD_MUTEX_INITIALIZER; + static pthread_cond_t schedule_cond=PTHREAD_COND_INITIALIZER; + static bool prepared,continue_safe; + static int safe_ret,dp_ret; + static int scheduled_prepare(struct typec_mux_dev *m,struct typec_mux_state *s) { + int ret=dcp_typec_route_prepare(m,s); + if(s->mode==TYPEC_STATE_SAFE&&!s->alt) { + pthread_mutex_lock(&schedule_lock);prepared=true;pthread_cond_broadcast(&schedule_cond); + while(!continue_safe)pthread_cond_wait(&schedule_cond,&schedule_lock); + pthread_mutex_unlock(&schedule_lock); + } + return ret; + } + static void *safe_actor(void *unused) { + (void)unused;struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE};safe_ret=typec_mux_set(&mux,&safe);return NULL; + } + int main(void) { + struct cd321x c;initial(&c);display_provider.prepare=scheduled_prepare; + pthread_t thread;assert(pthread_create(&thread,NULL,safe_actor,NULL)==0); + pthread_mutex_lock(&schedule_lock);while(!prepared)pthread_cond_wait(&schedule_cond,&schedule_lock);pthread_mutex_unlock(&schedule_lock); + struct typec_mux second_handle=mux; + struct typec_altmode alt={.svid=USB_TYPEC_DP_SID};struct typec_displayport_data data={.status=DP_STATUS_HPD_STATE}; + struct typec_mux_state dp={.alt=&alt,.mode=TYPEC_DP_STATE_A,.data=&data}; + dp_ret=typec_mux_set(&second_handle,&dp); + pthread_mutex_lock(&schedule_lock);continue_safe=true;pthread_cond_broadcast(&schedule_cond);pthread_mutex_unlock(&schedule_lock); + assert(pthread_join(thread,NULL)==0); + fprintf(stderr,"overlapping handles: dp_ret=%d safe_ret=%d connected_at_SAFE_PHY=%d events=%s\n",dp_ret,safe_ret,observed_unquiesced_safe,events); + assert(!observed_unquiesced_safe); + } + ''' + check('concurrent-prepare-set-stale-quiescence',concurrent,flags=('-pthread',)) + check('connector-context-lifetime', lifetime(ROOT), flags=('-pthread',)) + check('probe-transaction-rollback', probe(ROOT)) + check('provider-unregister-drain', unregister(ROOT), flags=('-pthread',)) + check('park-before-host-role-exit', clock_order(ROOT)) + return tests + + +def lifetime(root): + mux = (root / 'drivers/usb/typec/mux.c').read_text() + unit = quiesce.make_unit(root) + unit = unit.replace(block(unit, 'int main('), + block(unit, 'int main(').replace('int main(', 'int __attribute__((unused)) prior_main(', 1), 1) + unit = unit.replace('struct typec_mux_context { struct mutex lock; };', r""" +struct list_head { struct list_head *next,*prev; }; +struct fwnode_handle { unsigned refs; }; +struct typec_mux_context { struct list_head list; struct fwnode_handle *fwnode; struct mutex lock; unsigned users; }; +""") + helpers = r""" +#include +#include +#undef guard +#undef list_for_each_entry +static struct mutex typec_mux_contexts_lock={.value=PTHREAD_MUTEX_INITIALIZER}; +static struct list_head typec_mux_contexts={&typec_mux_contexts,&typec_mux_contexts}; +static struct fwnode_handle *fwnode_handle_get(struct fwnode_handle *f) { f->refs++;return f; } +static void fwnode_handle_put(struct fwnode_handle *f) { assert(f->refs);f->refs--; } +static void mutex_init(struct mutex *m) { assert(!pthread_mutex_init(&m->value,NULL)); } +static struct mutex *registry_lock(struct mutex *m) { assert(!pthread_mutex_lock(&m->value));return m; } +static void registry_unlock(struct mutex **m) { mutex_unlock(*m); } +#define guard(k) struct mutex * __attribute__((cleanup(registry_unlock))) held=registry_lock +#define kzalloc_obj(x) calloc(1,sizeof(x)) +#define kfree(x) free(x) +#define ERR_PTR(e) ((void *)(intptr_t)(e)) +#define list_for_each_entry(p,h,m) for(p=container_of((h)->next,__typeof__(*p),m); &(p)->m!=(h);p=container_of((p)->m.next,__typeof__(*p),m)) +static void list_add_tail(struct list_head *n,struct list_head *h) { n->prev=h->prev;n->next=h;h->prev->next=n;h->prev=n; } +static void list_del(struct list_head *n) { n->prev->next=n->next;n->next->prev=n->prev; } +""" + unit += helpers + block(mux, 'static struct typec_mux_context *typec_mux_context_get(') + '\n' + unit += block(mux, 'static void typec_mux_context_put(') + '\n' + unit += r""" +static struct typec_mux downstream; +static int forwarded,commits,aborts,finish_error; +static int forward_set(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;forwarded++;return typec_mux_set(&downstream,s); } +static int finish(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;(void)s;return finish_error; } +static void commit(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;(void)s;commits++; } +static void abort_callback(struct typec_mux_dev *m,struct typec_mux_state *s) { (void)m;(void)s;aborts++; } +int main(void) { + struct fwnode_handle a={0},b={0}; + struct typec_mux_context *one=typec_mux_context_get(&a),*two=typec_mux_context_get(&a),*other=typec_mux_context_get(&b); + assert(one==two&&one!=other&&one->users==2&&a.refs==1&&b.refs==1); + struct cd321x c;initial(&c);struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE}; + struct typec_mux second=mux; mux.context=one;second.context=two; + assert(typec_mux_begin(&mux,&safe)==0); + unsigned before=phy_calls;assert(typec_mux_set(&second,&safe)==-EBUSY&&phy_calls==before); + typec_mux_abort(&mux);assert(port.safe_pending&&port.owner==&route); + phy_error=-ETIMEDOUT;assert(typec_mux_set(&second,&safe)==-ETIMEDOUT&&port.owner==&route); + phy_error=0;assert(typec_mux_set(&second,&safe)==0&&!port.owner&&!port.safe_pending); + struct typec_mux_dev leaf={.set=phy_set,.finish=finish,.commit=commit,.abort=abort_callback}; + downstream=(struct typec_mux){.context=other,.num_mux_devs=1,.mux_devs={&leaf}}; + struct typec_mux_dev forward={.set=forward_set}; + mux=(struct typec_mux){.context=one,.num_mux_devs=1,.mux_devs={&forward}}; + assert(typec_mux_set(&mux,&safe)==0&&forwarded==1&&commits==1); + finish_error=-EIO;assert(typec_mux_set(&downstream,&safe)==-EIO&&commits==1&&aborts==1); + finish_error=0;assert(typec_mux_set(&downstream,&safe)==0&&commits==2); + downstream.context=one;before=phy_calls; + assert(typec_mux_set(&mux,&safe)==-EBUSY&&phy_calls==before); // cyclic same-connector forwarding + leaf.removed=true;downstream.context=other;before=phy_calls; + assert(typec_mux_set(&downstream,&safe)==-ENODEV&&phy_calls==before&&aborts==1); + typec_mux_context_put(one);assert(a.refs==1&&two->users==1); + typec_mux_context_put(two);assert(a.refs==0); + typec_mux_context_put(other);assert(b.refs==0&&typec_mux_contexts.next==&typec_mux_contexts); + puts("PASS actual connector registry, shared handles, forwarding, finalize/abort and removed-provider guards"); +} +""" + return unit + + +def probe(root): + core = (root / 'drivers/usb/typec/tipd/core.c').read_text() + unit = quiesce.make_unit(root) + unit = unit.replace(block(unit, 'int main('), block(unit, 'int main(').replace('int main(', 'int __attribute__((unused)) prior_main(', 1), 1) + unit = unit.replace('struct tipd_data {', 'struct fwnode_handle;\nstruct tipd_data { unsigned irq_mask1; int (*init)(struct tps6598x *); int (*switch_power_state)(struct tps6598x *, int); int (*register_port)(struct tps6598x *, struct fwnode_handle *); bool (*read_data_status)(struct tps6598x *); int (*connect)(struct tps6598x *,u32); void (*ready)(struct tps6598x *); void *irq_handler;') + unit = unit.replace('struct tps6598x {', 'struct tps6598x { bool dbma_at_probe,wakeup;') + helpers = r""" +#undef dev_err +#define dev_err(...) ((void)0) +#define TPS_SETUP_MS 1000 +#define TPS_SYSTEM_POWER_STATE_S0 0 +#define TPS_MODE_PTCH 2 +#define TPS_MODE_DBMA 3 +#define TPS_REG_VID 0 +#define TPS_REG_INT_MASK1 1 +#define TPS_STATUS_PLUG_PRESENT 1 +#define GPIOD_OUT_LOW 0 +#include +#undef IS_ERR +#define IS_ERR(p) ((uintptr_t)(p)>=(uintptr_t)-4095) +#define PTR_ERR(p) ((int)(intptr_t)(p)) +#define IRQF_SHARED 0 +#define IRQF_ONESHOT 0 +#define POLL_INTERVAL 1000 +#define INIT_DELAYED_WORK(...) ((void)0) +#define tps6598x_poll_work 0 +#define devm_device_init_wakeup(...) ((void)0) +#define enable_irq_wake(...) ((void)0) +#define dev_err_probe(...) (-EINVAL) +struct fwnode_handle { int unused; }; +static int connect_error,irq_error,register_error,node_puts,mux_get_error; +static void *devm_gpiod_get_optional(void *d,const char *s,int v) { (void)d;(void)s;(void)v;return NULL; } +static void msleep(unsigned ms) { (void)ms; } +static bool device_is_compatible(void *d,const char *s) { (void)d;(void)s;return false; } +static int tps6598x_read32(struct tps6598x *t,int reg,u32 *v) { (void)t;(void)reg;*v=1;return 0; } +static int tps6598x_check_mode(struct tps6598x *t) { (void)t;return 0; } +static int tps6598x_write64(struct tps6598x *t,int reg,unsigned v) { (void)t;(void)reg;(void)v;return 0; } +static bool tps6598x_read_status(struct tps6598x *t,u32 *s) { (void)t;*s=TPS_STATUS_PLUG_PRESENT;return true; } +static bool tps6598x_read_power_status(struct tps6598x *t) { (void)t;return true; } +static struct fwnode_handle node; +static struct fwnode_handle *device_get_named_child_node(void *d,const char *s) { (void)d;(void)s;return &node; } +static void fw_devlink_purge_absent_suppliers(struct fwnode_handle *f) { (void)f; } +static void *fwnode_usb_role_switch_get(struct fwnode_handle *f) { (void)f;return (void *)1; } +static struct typec_mux *fwnode_typec_mux_get(struct fwnode_handle *f) { (void)f;return mux_get_error?(struct typec_mux *)(intptr_t)mux_get_error:&mux; } +static void fwnode_handle_put(struct fwnode_handle *f) { (void)f;node_puts++; } +static int devm_tps6598_psy_register(struct tps6598x *t) { (void)t;return 0; } +static int devm_request_threaded_irq(void *d,int irq,void *h,void *th,int flags,const char *name,void *arg) { (void)d;(void)irq;(void)h;(void)th;(void)flags;(void)name;(void)arg;return irq_error; } +static int system_power_efficient_wq; +static unsigned msecs_to_jiffies(unsigned ms) { return ms; } +static void queue_delayed_work(int wq,int *work,unsigned delay) { (void)wq;(void)work;(void)delay; } +static bool device_property_read_bool(void *d,const char *p) { (void)d;(void)p;return false; } +static void tipd_debugfs_register(struct tps6598x *t) { (void)t; } +static int register_port(struct tps6598x *t,struct fwnode_handle *f) { (void)t;(void)f;return register_error; } +static bool read_data_status(struct tps6598x *t) { (void)t;return true; } +static int connect(struct tps6598x *t,u32 status) { (void)t;(void)status;return connect_error; } +""" + unit += helpers + block(core, 'int tipd_init(') + r""" +static void probe_initial(struct cd321x *c) { + initial(c);c->tps.irq=1;data.register_port=register_port;data.read_data_status=read_data_status;data.connect=connect; + connect_error=irq_error=register_error=node_puts=mux_get_error=0; +} +int main(void) { + struct cd321x c; + probe_initial(&c);irq_error=-ENOSPC;phy_error=-ETIMEDOUT; + assert(tipd_init(&c.tps)==-ENOSPC&&reset_calls==0&&gpio_calls==0&&port.owner==&route&&port.safe_pending&&unregisters==1&&node_puts==1); + probe_initial(&c);connect_error=-EAGAIN;role_error=-EBUSY; + assert(tipd_init(&c.tps)==-EAGAIN&&reset_calls==0&&gpio_calls==0&&phy_calls==1&&port.owner==&route&&port.safe_pending); + probe_initial(&c);irq_error=-ENOSPC; + assert(tipd_init(&c.tps)==-ENOSPC&&reset_calls==1&&!port.owner&&!port.safe_pending&&unregisters==1); + probe_initial(&c);mux_get_error=-ENODEV; + assert(tipd_init(&c.tps)==-ENODEV&&reset_calls==0&&stops==0&&phy_calls==0&&unregisters==0&&node_puts==1); + probe_initial(&c);register_error=-EIO; + assert(tipd_init(&c.tps)==-EIO&&reset_calls==0&&stops==0&&phy_calls==0&&unregisters==0); + puts("PASS actual probe rollback preserves primary errors and blocks reset after SAFE/role/registration errors"); +} +""" + return unit + + +def unregister(root): + """Actual unregister/begin/end with sleeping SRCU leaves and two threads.""" + source = (root / 'drivers/usb/typec/mux.c').read_text() + unit = quiesce.make_unit(root) + unit = unit.replace(block(unit, 'int main('), block(unit, 'int main(').replace('int main(', 'int __attribute__((unused)) prior_main(', 1), 1) + unit = unit.replace('struct typec_mux_dev {', 'struct device { bool unregistered; };\nstruct typec_mux_dev { struct device dev;') + old = block(unit, 'static int srcu_read_lock(') + '\n' + block(unit, 'static void srcu_read_unlock(') + leaves = r""" +static pthread_mutex_t lifetime_lock=PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t lifetime_cond=PTHREAD_COND_INITIALIZER; +static unsigned readers; +static bool draining; +static int srcu_read_lock(int *s) { (void)s;pthread_mutex_lock(&lifetime_lock);readers++;pthread_mutex_unlock(&lifetime_lock);return 0; } +static void srcu_read_unlock(int *s,int i) { (void)s;(void)i;pthread_mutex_lock(&lifetime_lock);assert(readers);readers--;pthread_cond_broadcast(&lifetime_cond);pthread_mutex_unlock(&lifetime_lock); } +static void synchronize_srcu(int *s) { (void)s;pthread_mutex_lock(&lifetime_lock);draining=true;pthread_cond_broadcast(&lifetime_cond);while(readers)pthread_cond_wait(&lifetime_cond,&lifetime_lock);pthread_mutex_unlock(&lifetime_lock); } +static void device_unregister(struct device *d) { assert(!readers);d->unregistered=true; } +#define WRITE_ONCE(x,v) ((x)=(v)) +""" + unit = unit.replace(old, leaves, 1) + unit += block(source, 'void typec_mux_unregister(') + r""" +static void *remove_provider(void *arg) { typec_mux_unregister(arg);return NULL; } +int main(void) { + struct cd321x c;initial(&c);struct typec_mux_state safe={.mode=TYPEC_STATE_SAFE}; + assert(typec_mux_begin(&mux,&safe)==0&&readers==1); + pthread_t thread;assert(!pthread_create(&thread,NULL,remove_provider,&phy_provider)); + pthread_mutex_lock(&lifetime_lock);while(!draining)pthread_cond_wait(&lifetime_cond,&lifetime_lock);pthread_mutex_unlock(&lifetime_lock); + assert(phy_provider.removed&&!phy_provider.dev.unregistered&&readers==1); + assert(typec_mux_end(&mux)==0); // pre-existing transaction finishes before removal returns + assert(!pthread_join(thread,NULL)&&phy_provider.dev.unregistered&&readers==0); + unsigned before=phy_calls; + assert(typec_mux_set(&mux,&safe)==-ENODEV&&phy_calls==before&&readers==0); + puts("PASS actual unregister drains the open transaction and tombstone blocks stale handles"); +} +""" + return unit + + +def clock_order(root): + unit = quiesce.make_unit(root) + unit = unit.replace(block(unit, 'int main('), block(unit, 'int main(').replace('int main(', 'int __attribute__((unused)) prior_main(', 1), 1) + unit = unit.replace('static int power_error,', 'static bool backend_clock=true,parked;\nstatic int power_error,') + unit = unit.replace("event('R');role_calls++;return role_error;", "event('R');role_calls++;if(!role_error)backend_clock=false;return role_error;", 1) + unit = unit.replace("event('L');phy_calls++;return phy_error;", "event('L');phy_calls++;if(parked)return 0;if(!backend_clock)return -ENODEV;if(!phy_error)parked=true;return phy_error;", 1) + unit += r""" +int main(void) { + (void)current;struct cd321x c;initial(&c); + tipd_remove(&c.tps); + assert(!last_error&&reset_calls==1&&gpio_calls==1&&parked&&!backend_clock); + assert(position('Q') +#include +#include "dcp-external-bw.h" +int main(void) { + const u64 n=0xbc000; + assert(apple_dcp_external_bw_valid(n,0x28000,0x70000)); + assert(!apple_dcp_external_bw_valid(n,-1,-1)); + assert(!apple_dcp_external_bw_valid(n,0,-1)); + assert(!apple_dcp_external_bw_valid(n,-1,0)); + assert(!apple_dcp_external_bw_valid(n,4,4)); + assert(!apple_dcp_external_bw_valid(n,2,4)); + assert(!apple_dcp_external_bw_valid(n,4,6)); + assert(!apple_dcp_external_bw_valid(n,n,4)); + assert(!apple_dcp_external_bw_valid(n,0,n)); + assert(!apple_dcp_external_bw_valid(3,0,4)); + assert(!apple_dcp_external_bw_valid(n,INT_MAX,0)); + assert(apple_dcp_external_bw_valid(n,0,4)); + assert(apple_dcp_external_bw_valid(n,n-4,0)); + assert(apple_dcp_external_bw_valid(n,0,n-4)); + assert(apple_dcp_external_clock_window(0x302800000ULL,n,0x302800000ULL,n)); + assert(!apple_dcp_external_clock_window(0x302800000ULL,n-4,0x302800000ULL,n)); + assert(!apple_dcp_external_clock_window(0x302800004ULL,n,0x302800000ULL,n)); + return 0; +} +""" + with tempfile.TemporaryDirectory() as directory: + tmp=pathlib.Path(directory);(tmp/'linux').mkdir() + (tmp/'linux/types.h').write_text('#include \n#include \ntypedef uint64_t u64; typedef uint32_t u32;\n') + (tmp/'test.c').write_text(code) + subprocess.run(['clang','-std=c11','-Wall','-Wextra','-Werror','-pedantic','-I'+str(tmp),'-I'+str(root/'drivers/gpu/drm/apple'),str(tmp/'test.c'),'-o',str(tmp/'test')],check=True) + subprocess.run([str(tmp/'test')],check=True) + + +if __name__=='__main__': + unittest.main() diff --git a/tools/testing/selftests/drivers/net/mt7932_e2e.py b/tools/testing/selftests/drivers/net/mt7932_e2e.py index 258baebce12f34..f66ccaac201a67 100644 --- a/tools/testing/selftests/drivers/net/mt7932_e2e.py +++ b/tools/testing/selftests/drivers/net/mt7932_e2e.py @@ -25,6 +25,12 @@ r'SCAN_INTERFACE_SETUP_FAILED|RF_FAILED|recovery-required=1|WARNING:|' r'UBSAN:|hung_task|blocked for more than|WIFI_FLR_REFUSED_OR_FAILED', re.I) +# Both radio drivers report progress at debug level, so a message of theirs at +# info or notice level during scans, association or traffic is log spam. +driver_message = re.compile(r'^\[\s*(\d+\.\d+)\] (?:mt7932-fullmac|mt7932_bt_pcie) ' + r'[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]: ', re.M) +log_stamp = re.compile(r'^\[\s*(\d+\.\d+)\]', re.M) + def run(argv, timeout=30): return subprocess.run(argv, text=True, stdout=subprocess.PIPE, @@ -42,6 +48,17 @@ def kernel_ok(): return p.stdout +def last_log_time(log): + """Timestamp of the newest kernel log message, or 0.0 if there is none.""" + stamps = log_stamp.findall(log) + return float(stamps[-1]) if stamps else 0.0 + + +def driver_info_messages(log, since): + """Radio driver messages in a ``dmesg --level=notice,info`` log after @since.""" + return [m.group(0) for m in driver_message.finditer(log) if float(m.group(1)) > since] + + def dns_name(packet, offset): """Decode a bounded DNS name, rejecting bad offsets and compression cycles.""" labels, seen, end, size = [], set(), None, 1 @@ -178,8 +195,9 @@ def main(): compatible = Path('/sys/firmware/devicetree/base/compatible').read_bytes().split(b'\0') require(b'apple,j700' in compatible and b'apple,t8140' in compatible, 'Wrong device') out.mkdir(mode=0o700) + started = None try: - kernel_ok() + started = last_log_time(kernel_ok()) profile = run(['nmcli', '-g', 'connection.uuid', 'connection', 'show', 'id', args.profile]) profile_uuid = profile.stdout.strip() require(profile.returncode == 0 and re.fullmatch(r'[0-9a-f-]{36}', profile_uuid), @@ -272,6 +290,13 @@ def main(): final_log.returncode != 0 or result['kernel_faults']): result['result'] = 'WIFI_NETWORK_E2E_FAILED' result['error'] = 'Final kernel log unavailable or contains a driver fault' + info_log = run(['dmesg', '--level=notice,info']) + if started is not None and info_log.returncode == 0: + result['driver_info_messages'] = len(driver_info_messages(info_log.stdout, started)) + if result['result'] in ('WIFI_SCAN_OBSERVED', 'WIFI_PHYSICAL_NETWORK_E2E_PASS') and ( + info_log.returncode != 0 or result.get('driver_info_messages')): + result['result'] = 'WIFI_NETWORK_E2E_FAILED' + result['error'] = 'Radio drivers logged routine events at info level' (out / 'result.json').write_text(json.dumps(result, indent=2) + '\n') print(json.dumps(result, indent=2)) print('ARTIFACT=' + str(out / 'result.json')) diff --git a/tools/testing/selftests/drivers/net/mt7932_e2e_test.py b/tools/testing/selftests/drivers/net/mt7932_e2e_test.py index 22b45b154f3655..75d341cad257ff 100644 --- a/tools/testing/selftests/drivers/net/mt7932_e2e_test.py +++ b/tools/testing/selftests/drivers/net/mt7932_e2e_test.py @@ -182,5 +182,27 @@ def test_new_fault_markers_are_terminal(self): self.assertIsNotNone(e2e.fault.search(message)) +class DriverLogTest(unittest.TestCase): + LOG = ('[ 10.000001] mt7932-fullmac 0000:01:00.0: probe message\n' + '[ 20.500000] usb 1-1: new high-speed USB device\n' + '[ 21.000000] mt7932-fullmac 0000:01:00.0: SCAN_REQUEST: channels=38\n' + '[ 21.250000] mt7932_bt_pcie 0000:01:00.1: controller recovered\n' + '[ 22.000000] Bluetooth: hci0: unrelated core message\n') + + def test_last_log_time(self): + self.assertEqual(e2e.last_log_time(self.LOG), 22.0) + self.assertEqual(e2e.last_log_time(''), 0.0) + + def test_only_radio_messages_after_start_count(self): + self.assertEqual(len(e2e.driver_info_messages(self.LOG, 0.0)), 3) + self.assertEqual(e2e.driver_info_messages(self.LOG, 20.5), + ['[ 21.000000] mt7932-fullmac 0000:01:00.0: ', + '[ 21.250000] mt7932_bt_pcie 0000:01:00.1: ']) + self.assertEqual(e2e.driver_info_messages(self.LOG, 22.0), []) + + def test_quiet_log_has_no_driver_messages(self): + self.assertEqual(e2e.driver_info_messages('[ 1.000000] usb 1-1: reset\n', 0.0), []) + + if __name__ == '__main__': unittest.main() diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 9e0dfd3a85b0e2..8d0377cda51f09 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -533,6 +533,9 @@ struct vm_area_desc { /* Mutable fields. Populated with initial state. */ pgoff_t pgoff; +#ifdef CONFIG_MM_SUBPAGE + unsigned int subpage_offset; +#endif struct file *vm_file; vma_flags_t vma_flags; pgprot_t page_prot; @@ -1300,6 +1303,9 @@ static inline void compat_set_desc_from_vma(struct vm_area_desc *desc, desc->end = vma->vm_end; desc->pgoff = vma->vm_pgoff; +#ifdef CONFIG_MM_SUBPAGE + desc->subpage_offset = vma->vm_subpage_offset; +#endif desc->vm_file = vma->vm_file; desc->vma_flags = vma->flags; desc->page_prot = vma->vm_page_prot; @@ -1308,6 +1314,117 @@ static inline void compat_set_desc_from_vma(struct vm_area_desc *desc, desc->action.type = MMAP_NOTHING; } +static inline unsigned int mm_page_shift(const struct mm_struct *mm) +{ + return PAGE_SHIFT; +} + +static inline unsigned long mm_page_size(const struct mm_struct *mm) +{ + return PAGE_SIZE; +} + +static inline unsigned long mm_page_mask(const struct mm_struct *mm) +{ + return PAGE_MASK; +} + +static inline unsigned long mm_page_align(const struct mm_struct *mm, unsigned long size) +{ + return PAGE_ALIGN(size); +} + +static inline unsigned long vma_user_pages(const struct vm_area_struct *vma) +{ + return (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; +} + +#define mm_account_memory(mm, policy_mm, pages) security_vm_enough_memory_mm(policy_mm, pages) +#define mm_unacct_memory(mm, pages) vm_unacct_memory(pages) + +/* File-cache indices remain native-page units even for a 4K VMA. */ +struct vm_page_offset { + pgoff_t index; + unsigned int offset; +}; + +static inline struct vm_page_offset vm_page_offset_add(struct vm_page_offset pos, + unsigned long bytes) +{ + unsigned long low = pos.offset + (bytes & ~PAGE_MASK); + + pos.index += (bytes >> PAGE_SHIFT) + (low >> PAGE_SHIFT); + pos.offset = low & ~PAGE_MASK; + return pos; +} + +static inline struct vm_page_offset vm_page_offset_sub(struct vm_page_offset pos, + unsigned long bytes) +{ + unsigned long low = bytes & ~PAGE_MASK; + + pos.index -= (bytes >> PAGE_SHIFT) + (low > pos.offset); + pos.offset = (pos.offset - low) & ~PAGE_MASK; + return pos; +} + +static inline bool vm_page_offset_equal(struct vm_page_offset a, struct vm_page_offset b) +{ + return a.index == b.index && a.offset == b.offset; +} + +static inline unsigned int vma_subpage_offset(const struct vm_area_struct *vma) +{ +#ifdef CONFIG_MM_SUBPAGE + return vma->vm_subpage_offset; +#else + return 0; +#endif +} + +/* address must be >= vm_start; vm_end computes the next contiguous offset. */ +static inline struct vm_page_offset vma_page_offset_at(const struct vm_area_struct *vma, + unsigned long address) +{ + unsigned long delta = address - vma->vm_start; + unsigned long low = (delta & ~PAGE_MASK) + vma_subpage_offset(vma); + + return (struct vm_page_offset) { + .index = vma->vm_pgoff + (delta >> PAGE_SHIFT) + (low >> PAGE_SHIFT), + .offset = low & ~PAGE_MASK, + }; +} + +static inline void vma_set_page_offset(struct vm_area_struct *vma, struct vm_page_offset pos) +{ + vma->vm_pgoff = pos.index; +#ifdef CONFIG_MM_SUBPAGE + vma->vm_subpage_offset = pos.offset; +#endif +} + +/* Inverse lookup of a single byte; an offset outside the VMA is not clamped. */ +static inline unsigned long vma_address_at_offset(const struct vm_area_struct *vma, + struct vm_page_offset pos) +{ + unsigned long len = vma->vm_end - vma->vm_start; + unsigned long index, delta, last; + unsigned int offset = vma_subpage_offset(vma); + + if (!len || pos.index < vma->vm_pgoff || pos.offset >= PAGE_SIZE) + return -EFAULT; + index = pos.index - vma->vm_pgoff; + last = ((len - 1) >> PAGE_SHIFT) + + ((((len - 1) & ~PAGE_MASK) + offset) >> PAGE_SHIFT); + if (index > last) + return -EFAULT; + delta = (index << PAGE_SHIFT) + pos.offset; + if (delta < offset || delta - offset >= len) + return -EFAULT; + return vma->vm_start + delta - offset; +} + + static inline unsigned long vma_pages(const struct vm_area_struct *vma) { return (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; diff --git a/tools/testing/vma/tests/merge.c b/tools/testing/vma/tests/merge.c index 03b6f9820e0ac9..0b0f4ae413973f 100644 --- a/tools/testing/vma/tests/merge.c +++ b/tools/testing/vma/tests/merge.c @@ -227,7 +227,7 @@ static bool test_simple_shrink(void) ASSERT_FALSE(attach_vma(&mm, vma)); - ASSERT_FALSE(vma_shrink(&vmi, vma, 0, 0x1000, 0)); + ASSERT_FALSE(vma_shrink(&vmi, vma, 0, 0x1000, (struct vm_page_offset) { 0, 0 })); ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x1000); diff --git a/tools/testing/vma/tests/mmap.c b/tools/testing/vma/tests/mmap.c index c85bc000d1cb7a..c87baa48ea26ef 100644 --- a/tools/testing/vma/tests/mmap.c +++ b/tools/testing/vma/tests/mmap.c @@ -12,19 +12,19 @@ static bool test_mmap_region_basic(void) current->mm = &mm; /* Map at 0x300000, length 0x3000. */ - addr = __mmap_region(NULL, 0x300000, 0x3000, vma_flags, 0x300, NULL); + addr = __mmap_region(NULL, 0x300000, 0x3000, vma_flags, (struct vm_page_offset) { 0x300, 0 }, NULL); ASSERT_EQ(addr, 0x300000); /* Map at 0x250000, length 0x3000. */ - addr = __mmap_region(NULL, 0x250000, 0x3000, vma_flags, 0x250, NULL); + addr = __mmap_region(NULL, 0x250000, 0x3000, vma_flags, (struct vm_page_offset) { 0x250, 0 }, NULL); ASSERT_EQ(addr, 0x250000); /* Map at 0x303000, merging to 0x300000 of length 0x6000. */ - addr = __mmap_region(NULL, 0x303000, 0x3000, vma_flags, 0x303, NULL); + addr = __mmap_region(NULL, 0x303000, 0x3000, vma_flags, (struct vm_page_offset) { 0x303, 0 }, NULL); ASSERT_EQ(addr, 0x303000); /* Map at 0x24d000, merging to 0x250000 of length 0x6000. */ - addr = __mmap_region(NULL, 0x24d000, 0x3000, vma_flags, 0x24d, NULL); + addr = __mmap_region(NULL, 0x24d000, 0x3000, vma_flags, (struct vm_page_offset) { 0x24d, 0 }, NULL); ASSERT_EQ(addr, 0x24d000); ASSERT_EQ(mm.map_count, 2); diff --git a/tools/testing/vma/tests/vma.c b/tools/testing/vma/tests/vma.c index 754a2da0632157..38a7cc6702f237 100644 --- a/tools/testing/vma/tests/vma.c +++ b/tools/testing/vma/tests/vma.c @@ -38,7 +38,7 @@ static bool test_copy_vma(void) /* Move backwards and do not merge. */ vma = alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, vma_flags); - vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks); + vma_new = copy_vma(&vma, 0, 0x2000, (struct vm_page_offset) { 0, 0 }, &need_locks); ASSERT_NE(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0); ASSERT_EQ(vma_new->vm_end, 0x2000); @@ -51,7 +51,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0, 0x2000, 0, vma_flags); vma_next = alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, vma_flags); - vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks); + vma_new = copy_vma(&vma, 0x4000, 0x2000, (struct vm_page_offset) { 4, 0 }, &need_locks); vma_assert_attached(vma_new); ASSERT_EQ(vma_new, vma_next);