From 3b0dd96e31dec3a167ac5e476640dbed4a00ecd8 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 16:17:49 +1100 Subject: [PATCH 01/20] media: apple: ave: turn the J700 encoder transport into a V4L2 encoder The AVE driver was a root-only debugfs transport that replays captured firmware commands at a fixed 320x240, and it could not be built into a kernel with suspend or kexec. Rework it into a normal V4L2 stateful encoder driver on top of Acelogic's mailbox and ring transport from 4612e5f8a66a ("media: apple: add an opt-in J700 AVE encoding transport"), which stays as the IPC layer with its start-up handshake. - Probe binds by DT only: no enable or snapshot parameters, no SoC checks beyond the compatible. The five encoder power domains are attached as a list and follow the device's runtime PM. - The firmware is booted on first use instead of through debugfs. The driver keeps a copy of the firmware DATA segment taken while it still matches the loader's checksum and puts it back before every boot, so the coprocessor can lose power and start again from the same state. - Commands are built in the driver. Config, Open, Complete, Stop, Close and Halt are complete. Session start and per-frame Process are left returning -EOPNOTSUPP until their layouts are described. - Firmware replies are checked for their status word. - A V4L2 M2M encoder (NV12 in, H.264 or HEVC out) per dev-encoder.rst, with drain, EOS and bitrate, GOP, profile, level and QP controls. - System sleep marks the firmware lost when the coprocessor lost power; the next session boots it again. Runtime suspend keeps power while the firmware runs until its stop sequence is in. Rename the symbol to VIDEO_APPLE_AVE and drop the !SUSPEND, !HIBERNATION and !KEXEC_CORE dependencies. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- Documentation/admin-guide/media/apple-ave.rst | 102 +-- drivers/media/platform/apple/ave/Kconfig | 22 +- drivers/media/platform/apple/ave/Makefile | 3 +- .../media/platform/apple/ave/apple-ave-ipc.h | 446 ---------- drivers/media/platform/apple/ave/apple-ave.c | 532 ----------- drivers/media/platform/apple/ave/ave-cmd.c | 108 +++ drivers/media/platform/apple/ave/ave-drv.c | 178 ++++ drivers/media/platform/apple/ave/ave-fw.c | 359 ++++++++ drivers/media/platform/apple/ave/ave-ipc.c | 384 ++++++++ drivers/media/platform/apple/ave/ave-regs.h | 128 +++ drivers/media/platform/apple/ave/ave-v4l2.c | 831 ++++++++++++++++++ drivers/media/platform/apple/ave/ave.h | 167 ++++ 12 files changed, 2209 insertions(+), 1051 deletions(-) delete mode 100644 drivers/media/platform/apple/ave/apple-ave-ipc.h delete mode 100644 drivers/media/platform/apple/ave/apple-ave.c create mode 100644 drivers/media/platform/apple/ave/ave-cmd.c create mode 100644 drivers/media/platform/apple/ave/ave-drv.c create mode 100644 drivers/media/platform/apple/ave/ave-fw.c create mode 100644 drivers/media/platform/apple/ave/ave-ipc.c create mode 100644 drivers/media/platform/apple/ave/ave-regs.h create mode 100644 drivers/media/platform/apple/ave/ave-v4l2.c create mode 100644 drivers/media/platform/apple/ave/ave.h diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index d7e63d029de15..dabb4250a72a1 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -1,63 +1,43 @@ .. SPDX-License-Identifier: GPL-2.0-only -Experimental T8140 video encoding -================================ - -The J700 AVE driver exposes a root-only debugfs firmware transport for -hardware encoding experiments. It is not a V4L2 encoder, has no stable -userspace ABI, and does not make an FFmpeg encoder available. The initial -scope is the existing fixed 320x240 NV12 H.264/HEVC protocol experiment. - -Build ``CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL=m`` on a 16 KiB-page Apple kernel -with debugfs and translated DMA. Suspend, hibernation and kexec must be off. -Load with ``modprobe apple-ave enable=1``; probing is disabled otherwise. - -Firmware and boot requirements ------------------------------- - -Use the public Aurora m1n1 J700 AVE handoff. It reserves the firmware already -loaded by iBoot, fills the device's ``memory-region`` references and exports -``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32`` strings under -``/chosen``. No firmware file is included or requested by this driver. -The current transport requires the qualified 25G76 two-segment layout: -0x14c000 text bytes and 0x130000 data bytes, mapped starting at IOVA -0x10000000000. Every mapping and both content CRCs are checked before use. -Text CRCs are restricted to the two inspected 9003.78.0 images (0x2ce7359e -and 0x69c54454). These checks detect build drift, not firmware authenticity. -Different firmware is unsupported, even if the device has the same SoC. - -The driver holds the DMA, pipe4, pipe5, me0 and me1 power domains and owns -the IPC, queue, log and firmware-requested heap allocations. Debugfs lives -under ``/sys/kernel/debug/285100000.video-codec``: - -* ``ave-status`` reports mappings, protocol phase and reply counters. -* ``ave-start-once`` accepts one explicit firmware start per boot. -* ``ave-command`` accepts one bounded packet and returns a framed reply. -* ``ave-arena`` stages image/reference/output surfaces within owned memory. -* ``ave-poll`` drains later firmware replies without another submission. - -Writes require ``CAP_SYS_RAWIO``. Command packets must come from a trusted, -matching local protocol experiment with all embedded surface addresses -relocated into this boot's owned arena. Root access does not make arbitrary -firmware packets valid. Binary command captures are supplied separately and -are not included in this contribution. - -Lifecycle and verification --------------------------- - -After the start attempt, the module, DMA buffers and power references remain -pinned until an external hardware reset, including after failure. Do not -force-unload, unbind the DARTs, reuse uncertain buffers or retry a failed -command. Firmware quiescence and normal production teardown are unfinished. - -A transport acknowledgement is not an encoded frame. For physical tests, -record the exact kernel/module, DT and firmware hashes; submit changing -images; validate each codec reply; independently decode the current DMA -output; check every frame's dimensions and quality; then verify Stop/Close, -kernel health and the existing desktop/radio stack. Keep the generated -inputs, bitstreams, decoded pixels, logs and hashes as a repeatable artifact. - -The prior implementation was authored by Ace (Acelogic). This contribution -ports that existing GPL work. Arbitrary resolutions, standard media APIs, -throughput, concurrent clients and other codecs or SoCs are outside this -experiment. +Apple AVE video encoder +======================= + +The AVE block in Apple T8140 (A18 Pro) SoCs, as used in the MacBook Neo, +encodes H.264 and HEVC. ``CONFIG_VIDEO_APPLE_AVE`` builds the ``apple-ave`` +driver, which registers a V4L2 memory-to-memory stateful encoder following +Documentation/userspace-api/media/v4l/dev-encoder.rst. The OUTPUT queue +takes NV12 frames (64-byte aligned line stride), the CAPTURE queue returns +H.264 or HEVC. + +Firmware +-------- + +The encoder runs firmware on its own coprocessor. The bootloader loads it into +two reserved, DART-mapped segments (TEXT and DATA) but does not start it. The +Aurora Stage 2 loader passes both regions in the device's ``memory-region`` +property and records their CRC32 under ``/chosen`` as +``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. No firmware file is +requested. + +The driver only binds to firmware builds whose command formats it knows, +identified by the TEXT checksum, and only when DATA still matches the +checksum the loader recorded. It keeps a copy of DATA and restores it before +every firmware boot, so the coprocessor can lose power and start again from +the same state. + +Power management +---------------- + +The firmware boots when the first encoding session starts. Runtime PM uses a +two second autosuspend delay. While the firmware runs the encoder stays +powered; across system sleep the power domains go down, open sessions fail +their next frame, and the next session boots the firmware again. + +Status +------ + +Firmware start-up, the command rings, session open/close and the V4L2 +interface are implemented. Session start and per-frame encoding are not yet +wired up, so ``VIDIOC_STREAMON`` on both queues currently fails with +``EOPNOTSUPP``. diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig index 629ce9244039a..59937167b29ce 100644 --- a/drivers/media/platform/apple/ave/Kconfig +++ b/drivers/media/platform/apple/ave/Kconfig @@ -1,16 +1,16 @@ # SPDX-License-Identifier: GPL-2.0-only -config VIDEO_APPLE_AVE_EXPERIMENTAL - tristate "Apple T8140 experimental video encoder transport" - depends on ARCH_APPLE && ARM64_16K_PAGES - depends on PM_GENERIC_DOMAINS && IOMMU_DMA && DEBUG_FS - depends on !SUSPEND && !HIBERNATION && !KEXEC_CORE +config VIDEO_APPLE_AVE + tristate "Apple AVE video encoder" + depends on ARCH_APPLE || COMPILE_TEST + depends on VIDEO_DEV && IOMMU_DMA && PM_GENERIC_DOMAINS select CRC32 + select V4L2_MEM2MEM_DEV + select VIDEOBUF2_DMA_CONTIG help - Experimental root-only AVE firmware and command transport for the - MacBook Neo. This does not register a V4L2 or FFmpeg encoder. + V4L2 memory-to-memory driver for the AVE H.264/HEVC encoder in + Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. The encoder + firmware is loaded by the bootloader. - It requires the bootloader's reserved, preloaded firmware and an - explicit enable=1 module parameter. After firmware startup, memory - and power references remain pinned until a hardware reset. Do not - enable this on a general-purpose kernel requiring suspend or kexec. + To compile this driver as a module, choose M here: the module + will be called apple-ave. diff --git a/drivers/media/platform/apple/ave/Makefile b/drivers/media/platform/apple/ave/Makefile index c8e673e68f080..8e0a5519a520b 100644 --- a/drivers/media/platform/apple/ave/Makefile +++ b/drivers/media/platform/apple/ave/Makefile @@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL) += apple-ave.o +obj-$(CONFIG_VIDEO_APPLE_AVE) += apple-ave.o +apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-v4l2.o diff --git a/drivers/media/platform/apple/ave/apple-ave-ipc.h b/drivers/media/platform/apple/ave/apple-ave-ipc.h deleted file mode 100644 index d317af1dad72b..0000000000000 --- a/drivers/media/platform/apple/ave/apple-ave-ipc.h +++ /dev/null @@ -1,446 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* Experimental 25G76/64-bit AVE mailbox and bidirectional ring transport. - * Included after private driver types. All callers hold ave->lock. - */ -#define AVE_CHANNEL_OFF 0x4000 -#define AVE_HANDSHAKE_OFF 0x24000 -#define AVE_SHARED_OFF 0x28000 -#define AVE_COMMAND_OFF 0x40000 -#define AVE_COMMAND_BYTES SZ_128K -#define AVE_ARENA_OFF 0x80000 -#define AVE_START_MAGIC 0x08042006 - -static int ave_fw_offset(struct apple_ave *ave, u64 address, size_t bytes, u32 *offset) -{ - u64 delta = address - ave->ipc_fw_base; - - if (address < ave->ipc_fw_base || delta > ave->ipc.size || bytes > ave->ipc.size - delta) - return -ERANGE; - *offset = delta; - return 0; -} - -static void ave_mailbox_send(struct apple_ave *ave, u32 a, u32 b, u32 c, u32 d) -{ - writel(a, ave->mailbox + 0x18); - writel(b, ave->mailbox + 0x1c); - writel(c, ave->mailbox + 0x20); - writel(d, ave->mailbox + 0x24); - wmb(); - writel(1, ave->mailbox + 0x0c); -} - -static int ave_mailbox_receive(struct apple_ave *ave) -{ - u32 status; - int ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); - - if (ret) - return ret; - /* Acknowledge the interrupt before reading the four scratch words. */ - writel(1, ave->mailbox + 0x10); - for (unsigned int i = 0; i < 4; i++) - ave->ipc_reply[i] = readl(ave->mailbox + 0x18 + i * 4); - dma_rmb(); - dev_info(ave->dev, "AVE_IPC_MAILBOX phase=%s words=%#x,%#x,%#x,%#x\n", - ave->phase, ave->ipc_reply[0], ave->ipc_reply[1], ave->ipc_reply[2], ave->ipc_reply[3]); - return 0; -} - -static int ave_create_channels(struct apple_ave *ave) -{ - bool found[2] = { false, false }; - - for (unsigned int i = 0; i < 2; i++) { - u8 *desc = ave->ipc.cpu + AVE_CHANNEL_OFF + i * 0x100; - u32 direction = get_unaligned_le32(desc + 0x40); - u32 interrupt = get_unaligned_le32(desc + 0x44); - u32 slots = get_unaligned_le32(desc + 0x48), offset; - u64 address = get_unaligned_le64(desc + 0x4c); - struct ave_channel *c; - unsigned int id; - - if (!memchr(desc, 0, 64)) - return -EPROTO; - if (!strcmp(desc, "IO")) - id = 0; - else if (!strcmp(desc, "IO_T2H")) - id = 1; - else - return -EPROTO; - if (found[id] || direction != id || !interrupt || interrupt >= 32 || - !slots || slots > ave->channel_bytes / 64 || !IS_ALIGNED(address, 64) || - ave_fw_offset(ave, address, (size_t)slots * 64, &offset) || - offset < AVE_CHANNEL_OFF + 0x200 || - offset + slots * 64 > AVE_CHANNEL_OFF + ave->channel_bytes) - return -EPROTO; - found[id] = true; - c = &ave->channel[id]; - strscpy(c->name, desc, sizeof(c->name)); - c->mode = direction ^ 1; - c->interrupt = interrupt; - c->slots = slots; - c->offset = offset; - c->consumer = c->mode ? -1 : 0; - c->producer = c->mode ? 0 : -1; - } - if (!found[0] || !found[1] || - (ave->channel[0].offset < ave->channel[1].offset + ave->channel[1].slots * 64 && - ave->channel[1].offset < ave->channel[0].offset + ave->channel[0].slots * 64)) - return -EPROTO; - /* Initialize only host-originated slots; leave all padding and - * firmware-originated slots unchanged. Ownership is bit zero of address. - */ - for (unsigned int i = 0; i < ave->channel[0].slots; i++) { - __le64 *slot = ave->ipc.cpu + ave->channel[0].offset + i * 64; - slot[0] = cpu_to_le64(1); - slot[1] = 0; - slot[2] = 0; - } - dma_wmb(); - return 0; -} - -static int ave_negotiate_ipc(struct apple_ave *ave) -{ - u64 ticks, time_delta, handshake_address, channel_address; - u32 frequency, status, offset; - u8 *handshake = ave->ipc.cpu + AVE_HANDSHAKE_OFF; - int ret; - - ave->phase = "validate-offer"; - if (ave->hello[0] != 2 || ave->hello[2] != 0x100 || - ave->hello[1] < 0x200 || ave->hello[1] > AVE_HANDSHAKE_OFF - AVE_CHANNEL_OFF || - !ave->hello[3] || ave->hello[3] > 4 * SZ_1M) - return -EPROTO; - ave->channel_bytes = ave->hello[1]; - ret = ave_alloc_buffer(ave, &ave->heap, PAGE_ALIGN(ave->hello[3])); - if (ret) - return ret; - writel(1, ave->mailbox + 0x10); - ticks = readq(ave->coproc + 0x378000); - frequency = readl(ave->coproc + 0x360020); - if (frequency && frequency < 1000000) - return -EPROTO; - ticks = div64_u64(ticks, frequency ? frequency / 1000000 : 24); - time_delta = div_u64(ktime_get_ns(), 1000) - ticks; - writel(lower_32_bits(time_delta), ave->mailbox + 0x28); - writel(upper_32_bits(time_delta), ave->mailbox + 0x2c); - ave->phase = "ipc-base"; - ave_mailbox_send(ave, lower_32_bits(ave->ipc.dma), upper_32_bits(ave->ipc.dma), ave->ipc.size, 0); - ret = ave_mailbox_receive(ave); - if (ret) - return ret; - ave->ipc_fw_base = (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0]; - if (!ave->ipc_fw_base || !IS_ALIGNED(ave->ipc_fw_base, PAGE_SIZE) || - ave->ipc_fw_base > U64_MAX - ave->ipc.size) - return -EPROTO; - memset(ave->ipc.cpu + AVE_CHANNEL_OFF, 0, ave->channel_bytes); - memset(handshake, 0, 0x50); - memset(ave->ipc.cpu + AVE_SHARED_OFF, 0, SZ_64K); - channel_address = ave->ipc_fw_base + AVE_CHANNEL_OFF; - handshake_address = ave->ipc_fw_base + AVE_HANDSHAKE_OFF; - put_unaligned_le64(channel_address, handshake + 8); - put_unaligned_le64(ave->ipc_fw_base + AVE_SHARED_OFF, handshake + 0x10); - put_unaligned_le32(SZ_64K, handshake + 0x18); - put_unaligned_le64(ave->heap.dma, handshake + 0x1c); - put_unaligned_le32(ave->hello[3], handshake + 0x24); - put_unaligned_le32(27, handshake + 0x28); - dma_wmb(); - ave->phase = "channel-offer"; - ave_mailbox_send(ave, lower_32_bits(handshake_address), upper_32_bits(handshake_address), 0, 0); - ret = ave_mailbox_receive(ave); - if (ret) - return ret; - ave->extra_bytes = ave->ipc_reply[2]; - if (ave_fw_offset(ave, (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0], - ave->channel_bytes, &offset) || offset != AVE_CHANNEL_OFF || - ave->extra_bytes > 0x13c000) - return -EPROTO; - ave->phase = "channel-layout"; - ret = ave_create_channels(ave); - if (ret) - return ret; - ave->phase = "ready-flag"; - writel(AVE_START_MAGIC, ave->mailbox + 0x24); - ret = readl_poll_timeout(ave->mailbox + 0x24, status, !status, 50, 5000000); - if (ret) - return ret; - ave->ipc_ready = true; - ave->phase = "ready"; - return 0; -} - -static int ave_ring_send(struct apple_ave *ave, struct ave_channel *c, u64 addr, u32 len, u32 flags) -{ - __le64 *slot; - int next; - - if (c->producer < 0) - return -ENOSPC; - slot = ave->ipc.cpu + c->offset + c->producer * 64; - WRITE_ONCE(slot[1], cpu_to_le64(len)); - WRITE_ONCE(slot[2], cpu_to_le64(flags)); - dma_wmb(); - WRITE_ONCE(slot[0], cpu_to_le64(addr | (c->mode ^ 1))); - dma_wmb(); - if (c->consumer < 0) - c->consumer = c->producer; - next = (c->producer + 1) % c->slots; - c->producer = next == c->consumer ? -1 : next; - writel(BIT(c->interrupt), ave->mailbox + 0x0c); - return 0; -} - -static int ave_ring_receive(struct apple_ave *ave, struct ave_channel *c, u64 *addr, u32 *len, u32 *flags) -{ - __le64 *slot; - u64 word; - int next; - - if (c->consumer < 0) - return -EAGAIN; - slot = ave->ipc.cpu + c->offset + c->consumer * 64; - word = le64_to_cpu(READ_ONCE(slot[0])); - if ((word & 1) != c->mode) - return -EAGAIN; - dma_rmb(); - *addr = word & ~3ULL; - *len = le64_to_cpu(READ_ONCE(slot[1])); - *flags = le64_to_cpu(READ_ONCE(slot[2])); - if (c->producer < 0) - c->producer = c->consumer; - next = (c->consumer + 1) % c->slots; - c->consumer = next == c->producer ? -1 : next; - return 0; -} - -/* Root-only experimental packet transport. The packet is copied into owned - * coherent memory; timeout pins the command too and forbids reuse this boot. - * Firmware command success is established by decoding replies, not this API. - */ -static ssize_t ave_command_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - u8 *cmd = ave->ipc.cpu + AVE_COMMAND_OFF; - u64 deadline, addr; - u32 bytes, flags, offset; - bool ack = false; - int ret = 0; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - static_assert(AVE_COMMAND_OFF + AVE_COMMAND_BYTES <= AVE_ARENA_OFF); - if (len < 0x48 || len > AVE_COMMAND_BYTES) - return -EINVAL; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) { - ret = -EBUSY; - goto out; - } - if (copy_from_user(cmd, data, len)) { - ret = -EFAULT; - goto out; - } - /* The common command timeout is {duration_us, creation_time_us}. - * Stamp it in the same Linux monotonic clock synchronized at startup; - * captured macOS absolute times cannot be replayed on this boot. - */ - put_unaligned_le64(3000000, cmd + 0x30); - put_unaligned_le64(div_u64(ktime_get_ns(), 1000), cmd + 0x38); - ave->response_bytes = 0; - ret = ave_ring_send(ave, &ave->channel[0], ave->ipc_fw_base + AVE_COMMAND_OFF, len, 0); - if (ret) - goto out; - ave->commands++; - deadline = ktime_get_ns() + 5ULL * NSEC_PER_SEC; - do { - u32 pending = readl(ave->mailbox + 0x10); - if (pending) - writel(pending, ave->mailbox + 0x10); - for (unsigned int ch = 0; ch < 2; ch++) { - ret = ave_ring_receive(ave, &ave->channel[ch], &addr, &bytes, &flags); - if (ret == -EAGAIN) - continue; - /* Channel 1 returns the original address with zero payload - * bytes (verified native AVC captures). Only channel 2 is - * copied to response, independent of submitted packet size. - */ - if (ret || (!ch && bytes != 0) || - (ch && bytes > sizeof(ave->response) - 24) || - ave_fw_offset(ave, addr, bytes, &offset)) { - ret = -EPROTO; - goto failed; - } - dev_info(ave->dev, "AVE_COMMAND_REPLY channel=%u address=%#llx bytes=%u flags=%#x\n", - ch + 1, addr, bytes, flags); - if (!ch) { - if (addr != ave->ipc_fw_base + AVE_COMMAND_OFF) { - ret = -EPROTO; - goto failed; - } - ack = true; - ave->transport_acks++; - } else { - put_unaligned_le32(ch + 1, ave->response); - put_unaligned_le32(bytes, ave->response + 4); - put_unaligned_le32(flags, ave->response + 8); - put_unaligned_le32(0, ave->response + 12); - put_unaligned_le64(addr, ave->response + 16); - memcpy(ave->response + 24, ave->ipc.cpu + offset, bytes); - ave->response_bytes = bytes + 24; - ave->replies++; - /* Reply echoes use zero flags on the observed transport. */ - ret = ave_ring_send(ave, &ave->channel[ch], addr, bytes, 0); - if (ret) - goto failed; - } - } - if (ack && ave->response_bytes) { - ret = 0; - goto out; - } - usleep_range(100, 200); - } while (ktime_get_ns() < deadline); - ret = -ETIMEDOUT; -failed: - ave->command_failed = true; -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static ssize_t ave_command_read(struct file *file, char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - mutex_lock(&ave->lock); - ret = simple_read_from_buffer(data, len, pos, ave->response, ave->response_bytes); - mutex_unlock(&ave->lock); - return ret; -} - -static const struct file_operations ave_command_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_command_write, - .read = ave_command_read, -}; - -/* Drain later firmware events without publishing another command. Results - * replace the response buffer and consist of one or more 24-byte-header - * records, each followed by its declared payload. Never discard silently. - */ -static ssize_t ave_poll_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - bool requested; - u64 deadline, addr; - u32 bytes, flags, offset; - int ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - ret = kstrtobool_from_user(data, len, &requested); - if (ret || !requested) - return ret ?: -EINVAL; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) { - ret = -EBUSY; - goto out; - } - ave->response_bytes = 0; - deadline = ktime_get_ns() + NSEC_PER_SEC; - do { - u32 pending = readl(ave->mailbox + 0x10); - u8 *dst; - - if (pending) - writel(pending, ave->mailbox + 0x10); - ret = ave_ring_receive(ave, &ave->channel[1], &addr, &bytes, &flags); - if (ret == -EAGAIN) { - usleep_range(100, 200); - continue; - } - if (ret || ave_fw_offset(ave, addr, bytes, &offset) || - bytes > sizeof(ave->response) - 24 || - ave->response_bytes > sizeof(ave->response) - 24 - bytes) { - ret = -EPROTO; - goto failed; - } - dst = ave->response + ave->response_bytes; - put_unaligned_le32(2, dst); - put_unaligned_le32(bytes, dst + 4); - put_unaligned_le32(flags, dst + 8); - put_unaligned_le32(0, dst + 12); - put_unaligned_le64(addr, dst + 16); - memcpy(dst + 24, ave->ipc.cpu + offset, bytes); - ave->response_bytes += 24 + bytes; - ave->replies++; - ret = ave_ring_send(ave, &ave->channel[1], addr, bytes, 0); - if (ret) - goto failed; - } while (ktime_get_ns() < deadline); - ret = 0; - goto out; -failed: - ave->command_failed = true; -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static const struct file_operations ave_poll_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_poll_write, -}; - -/* Bounded staging area for image/reference/output surfaces. Userspace offsets - * start at zero here, beyond all handshake, ring and command allocations. - * The status file supplies both DART and firmware addresses for this arena. - */ -static ssize_t ave_arena_read(struct file *file, char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - mutex_lock(&ave->lock); - dma_rmb(); - ret = simple_read_from_buffer(data, len, pos, ave->ipc.cpu + AVE_ARENA_OFF, - ave->ipc.size - AVE_ARENA_OFF); - mutex_unlock(&ave->lock); - return ret; -} - -static ssize_t ave_arena_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) - ret = -EBUSY; - else - ret = simple_write_to_buffer(ave->ipc.cpu + AVE_ARENA_OFF, - ave->ipc.size - AVE_ARENA_OFF, pos, data, len); - dma_wmb(); - mutex_unlock(&ave->lock); - return ret; -} - -static loff_t ave_arena_seek(struct file *file, loff_t offset, int whence) -{ - return fixed_size_llseek(file, offset, whence, AVE_IPC_BYTES - AVE_ARENA_OFF); -} - -static const struct file_operations ave_arena_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .read = ave_arena_read, - .write = ave_arena_write, - .llseek = ave_arena_seek, -}; diff --git a/drivers/media/platform/apple/ave/apple-ave.c b/drivers/media/platform/apple/ave/apple-ave.c deleted file mode 100644 index d2cf7abadcadf..0000000000000 --- a/drivers/media/platform/apple/ave/apple-ave.c +++ /dev/null @@ -1,532 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* T8140 AVE bring-up: resources, firmware negotiation and IPC transport. - * Experimental root-only command interface. No registered encoder yet. - */ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#define AVE_IPC_BYTES (20 * SZ_1M) -#define AVE_QUEUE_BYTES SZ_64K -#define AVE_FW_IOVA 0x10000000000ULL -#define AVE_LOG_BYTES 0x24000 -#define AVE_LOG_USED 0x20200 - -static bool enable; -module_param(enable, bool, 0400); -MODULE_PARM_DESC(enable, "Opt in to the experimental AVE transport"); - -/* Read the known Uranus registers once after PM resume. The separate hello - * command pins all resources because reset/quiescence is not yet qualified. - */ -static bool snapshot; -module_param(snapshot, bool, 0400); -MODULE_PARM_DESC(snapshot, "Capture powered AVE registers once without writing them"); - -struct ave_firmware { - phys_addr_t physical; - u64 iova; - size_t size; - u32 crc; - unsigned int verified_pages; -}; - -struct ave_buffer { - void *cpu; - dma_addr_t dma; - size_t size; - unsigned int verified_pages; -}; - -struct ave_channel { - char name[64]; - u32 mode, interrupt, slots; - u32 offset; - int consumer, producer; -}; - -struct apple_ave { - struct device *dev; - struct device *power[5]; - unsigned int num_power; - struct iommu_domain *domain; - struct ave_buffer ipc; - struct ave_buffer queue; - struct ave_buffer log; - struct ave_buffer heap; - struct ave_firmware fw[2]; - bool firmware_verified; - bool start_requested; - int start_result; - struct mutex lock; - void __iomem *coproc, *mailbox; - u32 hello[4]; - u64 ipc_fw_base; - u32 ipc_reply[4], channel_bytes, extra_bytes; - const char *phase; - bool ipc_ready, command_failed; - struct ave_channel channel[2]; - u32 commands, replies, transport_acks; - u8 response[SZ_64K]; - size_t response_bytes; - struct debugfs_blob_wrapper ipc_blob; - struct debugfs_blob_wrapper log_blob; - struct debugfs_blob_wrapper config_blob; - u64 vcbar; - u32 core_control, core_status, mailbox_status, scratch[8]; - struct dentry *debugfs; -}; - -static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size); -#include "apple-ave-ipc.h" - -/* Cold-boot hello only. A successful response still needs the remaining IPC - * negotiation. Keep the module, PM reference and DMA buffers pinned until a - * target reboot, on both success and timeout; no unproven stop/free sequence. - */ -static ssize_t ave_start_write(struct file *file, const char __user *data, - size_t len, loff_t *ppos) -{ - struct apple_ave *ave = file->private_data; - __le32 *cfg = ave->queue.cpu, *log = ave->log.cpu; - bool requested; - u32 status; - int ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - ret = kstrtobool_from_user(data, len, &requested); - if (ret || !requested) - return ret ?: -EINVAL; - mutex_lock(&ave->lock); - if (ave->start_requested) { - ret = -EALREADY; - goto out; - } - if (!ave->firmware_verified || readq(ave->coproc + 0x50000) != (AVE_FW_IOVA | 1) || - readl(ave->coproc + 0x600044) || readl(ave->mailbox + 0x10)) { - ret = -EBUSY; - goto out; - } - for (unsigned int i = 0; i < 8; i++) { - if (readl(ave->mailbox + 0x18 + i * 4)) { - ret = -EBUSY; - goto out; - } - } - /* Startup values observed with the qualified 25G76 firmware: device - * enum 29, one active unit/group and sub-ID mask 1. Firmware maps the - * full page-aligned log allocation; the ring header advertises usage. - */ - memset(cfg, 0, 0x40); - cfg[1] = cpu_to_le32(29); - cfg[2] = cpu_to_le32(1); - cfg[3] = cpu_to_le32(1); - cfg[4] = cpu_to_le32(1); - cfg[8] = cpu_to_le32(lower_32_bits(ave->log.dma)); - cfg[9] = cpu_to_le32(upper_32_bits(ave->log.dma)); - cfg[10] = cpu_to_le32(AVE_LOG_BYTES); - cfg[12] = cpu_to_le32(0x77); - /* Log header: 128 KiB ring after its 512-byte header. */ - log[0] = cpu_to_le32(AVE_LOG_USED); - log[1] = cpu_to_le32(0x200); - log[2] = cpu_to_le32(0x20000); - dma_wmb(); - __module_get(THIS_MODULE); - ave->start_requested = true; - ave->phase = "hello"; - ave->start_result = -EINPROGRESS; - dev_info(ave->dev, "AVE_START_ONCE cfg=%pad log=%pad resources_pinned_until_reboot=1\n", - &ave->queue.dma, &ave->log.dma); - writel(0x08042006, ave->mailbox + 0x18); - writel(lower_32_bits(ave->queue.dma), ave->mailbox + 0x1c); - writel(upper_32_bits(ave->queue.dma), ave->mailbox + 0x20); - /* Preserve the protected firmware mappings inherited from iBoot. */ - writel(1, ave->coproc + 0x600808); - writel(0, ave->coproc + 0x600044); - writel(0x10000, ave->coproc + 0x600400); - writel(0x10, ave->coproc + 0x600044); - ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); - ave->start_result = ret; - ave->core_control = readl(ave->coproc + 0x600044); - ave->core_status = readl(ave->coproc + 0x600048); - ave->mailbox_status = status; - for (unsigned int i = 0; i < 8; i++) - ave->scratch[i] = readl(ave->mailbox + 0x18 + i * 4); - for (unsigned int i = 0; i < 4; i++) - ave->hello[i] = ave->scratch[i]; - dma_rmb(); - dev_info(ave->dev, "AVE_HELLO_RESULT result=%d status=%#x words=%#x,%#x,%#x,%#x core=%#x\n", - ret, status, ave->hello[0], ave->hello[1], ave->hello[2], ave->hello[3], ave->core_status); - if (!ret) - ret = ave_negotiate_ipc(ave); - ave->start_result = ret; - dev_info(ave->dev, "AVE_IPC_RESULT result=%d phase=%s ready=%u\n", - ret, ave->phase, ave->ipc_ready); -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static const struct file_operations ave_start_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_start_write, -}; - -static int ave_verify_firmware(struct apple_ave *ave) -{ - static const char * const crc_names[] = { - "aurora,ave-text-crc32", "aurora,ave-data-crc32", - }; - static const size_t sizes[] = { 0x14c000, 0x130000 }; - struct device_node *chosen, *node; - unsigned int i; - int ret = 0; - - if (of_count_phandle_with_args(ave->dev->of_node, "memory-region", NULL) != 2) - return dev_err_probe(ave->dev, -EINVAL, "Expected two live AVE firmware reservations\n"); - chosen = of_find_node_by_path("/chosen"); - if (!chosen) - return -ENOENT; - for (i = 0; i < ARRAY_SIZE(ave->fw); i++) { - struct ave_firmware *fw = &ave->fw[i]; - struct resource res; - const char *crc_string; - u32 maps[5], expected_crc; - void *cpu; - size_t offset; - - node = of_parse_phandle(ave->dev->of_node, "memory-region", i); - ret = of_address_to_resource(node, 0, &res); - if (!ret) - ret = of_property_read_u32_array(node, "iommu-addresses", maps, ARRAY_SIZE(maps)); - if (!ret && (!of_property_read_bool(node, "no-map") || - !of_device_is_compatible(node, "apple,asc-mem"))) - ret = -EINVAL; - of_node_put(node); - if (ret) - break; - fw->physical = res.start; - fw->size = resource_size(&res); - fw->iova = (u64)maps[1] << 32 | maps[2]; - if (fw->size != sizes[i] || !IS_ALIGNED(fw->physical, PAGE_SIZE) || - maps[0] != ave->dev->of_node->phandle || maps[3] || maps[4] != sizes[i] || - fw->iova != AVE_FW_IOVA + (i ? sizes[0] : 0)) { - ret = -EINVAL; - break; - } - /* The IOMMU core owns these translated reservations across module - * unloads. DMA allocation must never reuse their IOVAs. - */ - for (offset = 0; offset < fw->size; offset += PAGE_SIZE) { - if (iommu_iova_to_phys(ave->domain, fw->iova + offset) != fw->physical + offset) { - dev_err(ave->dev, "Firmware %u mapping mismatch at %#zx\n", i, offset); - ret = -EIO; - break; - } - fw->verified_pages++; - } - if (ret) - break; - ret = of_property_read_string(chosen, crc_names[i], &crc_string); - if (!ret) - ret = kstrtou32(crc_string, 16, &expected_crc); - if (ret) - break; - cpu = memremap(fw->physical, fw->size, MEMREMAP_WB); - if (!cpu) { - ret = -ENOMEM; - break; - } - fw->crc = crc32_le(~0U, cpu, fw->size) ^ ~0U; - memunmap(cpu); - if (fw->crc != expected_crc) { - dev_err(ave->dev, "Firmware %u changed since the boot preflight\n", i); - ret = -EILSEQ; - break; - } - /* Only the two inspected 9003.78.0 text images are supported. - * CRCs detect build drift; they are not firmware authentication. - */ - if (!i && fw->crc != 0x2ce7359e && fw->crc != 0x69c54454) { - dev_err(ave->dev, "Unsupported AVE firmware text CRC %#x\n", fw->crc); - ret = -ENODEV; - break; - } - } - of_node_put(chosen); - if (!ret) - ave->firmware_verified = true; - return ret; -} - -/* Check the real IOMMU page table against the CPU mapping of each allocation. - * This proves mappings, not that firmware has performed a DMA transaction. - */ -static int ave_verify_mapping(struct apple_ave *ave, struct ave_buffer *buf) -{ - size_t offset; - - if (!IS_ALIGNED(buf->dma, PAGE_SIZE) || !IS_ALIGNED(buf->size, PAGE_SIZE)) - return -EINVAL; - for (offset = 0; offset < buf->size; offset += PAGE_SIZE) { - void *cpu = buf->cpu + offset; - struct page *page = is_vmalloc_addr(cpu) ? vmalloc_to_page(cpu) : virt_to_page(cpu); - phys_addr_t physical = iommu_iova_to_phys(ave->domain, buf->dma + offset); - - if (!page || physical != page_to_phys(page)) { - dev_err(ave->dev, "DMA mapping mismatch at offset %#zx\n", offset); - return -EIO; - } - buf->verified_pages++; - } - return 0; -} - -static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size) -{ - buf->size = size; - buf->cpu = dma_alloc_coherent(ave->dev, size, &buf->dma, GFP_KERNEL); - if (!buf->cpu) - return -ENOMEM; - memset(buf->cpu, 0, size); - return ave_verify_mapping(ave, buf); -} - -static void ave_free_buffer(struct apple_ave *ave, struct ave_buffer *buf) -{ - if (buf->cpu) - dma_free_coherent(ave->dev, buf->size, buf->cpu, buf->dma); - buf->cpu = NULL; -} - -static void ave_power_off(struct apple_ave *ave) -{ - while (ave->num_power) { - struct device *pd = ave->power[--ave->num_power]; - - pm_runtime_put_sync(pd); - dev_pm_domain_detach(pd, true); - } -} - -static int ave_power_on(struct apple_ave *ave) -{ - static const char * const names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; - int ret; - - for (unsigned int i = 0; i < ARRAY_SIZE(names); i++) { - struct device *pd = dev_pm_domain_attach_by_name(ave->dev, names[i]); - - if (IS_ERR(pd)) - return PTR_ERR(pd); - if (!pd) - return -ENODEV; - ret = pm_runtime_resume_and_get(pd); - if (ret < 0) { - dev_pm_domain_detach(pd, true); - return ret; - } - ave->power[ave->num_power++] = pd; - } - return 0; -} - -static int ave_status_show(struct seq_file *s, void *unused) -{ - struct apple_ave *ave = s->private; - - mutex_lock(&ave->lock); - seq_printf(s, "state=%s\nfirmware_start_requested=%u\nstart_result=%d\n", - ave->start_requested ? (ave->start_result ? "startup-failed" : "ipc-ready") : - "firmware-mapped", ave->start_requested, ave->start_result); - seq_printf(s, "hardware_dma_tested=%u\nencoder_registered=0\nphase=%s\n", - ave->ipc_ready, ave->phase ?: "not-started"); - seq_printf(s, "ipc_ready=%u\nipc_fw_base=%#llx\nchannel_bytes=%#x\nextra_bytes=%#x\n", - ave->ipc_ready, ave->ipc_fw_base, ave->channel_bytes, ave->extra_bytes); - seq_printf(s, "commands=%u\nreplies=%u\ntransport_acks=%u\ncommand_failed=%u\nresponse_bytes=%zu\n", - ave->commands, ave->replies, ave->transport_acks, ave->command_failed, ave->response_bytes); - seq_printf(s, "arena_bytes=%zu\narena_ipc_offset=%#x\narena_dma=%#llx\narena_fw=%#llx\n", - ave->ipc.size - AVE_ARENA_OFF, AVE_ARENA_OFF, - (u64)ave->ipc.dma + AVE_ARENA_OFF, ave->ipc_fw_base + AVE_ARENA_OFF); - for (unsigned int i = 0; i < 2; i++) { - struct ave_channel *c = &ave->channel[i]; - seq_printf(s, "channel%u_name=%s\nchannel%u_mode=%u\nchannel%u_interrupt=%u\n" - "channel%u_slots=%u\nchannel%u_offset=%#x\nchannel%u_consumer=%d\nchannel%u_producer=%d\n", - i, c->name, i, c->mode, i, c->interrupt, i, c->slots, - i, c->offset, i, c->consumer, i, c->producer); - } - seq_printf(s, "page_size=%lu\n", PAGE_SIZE); - seq_printf(s, "powered_domains=%u\n", ave->num_power); - seq_printf(s, "ipc_bytes=%zu\nipc_dma=%pad\nipc_verified_pages=%u\n", - ave->ipc.size, &ave->ipc.dma, ave->ipc.verified_pages); - seq_printf(s, "queue_bytes=%zu\nqueue_dma=%pad\nqueue_verified_pages=%u\n", - ave->queue.size, &ave->queue.dma, ave->queue.verified_pages); - seq_printf(s, "log_bytes=%zu\nlog_dma=%pad\nlog_verified_pages=%u\n", - ave->log.size, &ave->log.dma, ave->log.verified_pages); - seq_printf(s, "heap_bytes=%zu\nheap_dma=%pad\nheap_verified_pages=%u\n", - ave->heap.size, &ave->heap.dma, ave->heap.verified_pages); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->fw); i++) - seq_printf(s, "fw%u_physical=%pa\nfw%u_iova=%#llx\nfw%u_bytes=%zu\n" - "fw%u_verified_pages=%u\nfw%u_crc32=%08x\n", - i, &ave->fw[i].physical, i, ave->fw[i].iova, i, ave->fw[i].size, - i, ave->fw[i].verified_pages, i, ave->fw[i].crc); - seq_printf(s, "register_snapshot=%u\n", snapshot); - if (snapshot) { - seq_printf(s, "vcbar=%#llx\ncore_control=%#x\ncore_status=%#x\nmailbox_status=%#x\n", - ave->vcbar, ave->core_control, ave->core_status, ave->mailbox_status); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) - seq_printf(s, "scratch%u=%#x\n", i, ave->scratch[i]); - } - seq_printf(s, "resources_pinned_until_reboot=%u\n", ave->start_requested); - mutex_unlock(&ave->lock); - return 0; -} -DEFINE_SHOW_ATTRIBUTE(ave_status); - -static int apple_ave_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct apple_ave *ave; - void __iomem *regs, *mailbox, *coproc; - int ret; - - if (!enable) - return -ENODEV; - if (!of_machine_is_compatible("apple,j700") || PAGE_SIZE != SZ_16K) - return -ENODEV; - ave = devm_kzalloc(dev, sizeof(*ave), GFP_KERNEL); - if (!ave) - return -ENOMEM; - ave->dev = dev; - mutex_init(&ave->lock); - /* Claim only dedicated AVE ranges, never the shared PMGR/SRAM windows. */ - regs = devm_platform_ioremap_resource_byname(pdev, "codec"); - if (IS_ERR(regs)) - return PTR_ERR(regs); - mailbox = devm_platform_ioremap_resource_byname(pdev, "mailbox"); - if (IS_ERR(mailbox)) - return PTR_ERR(mailbox); - coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); - if (IS_ERR(coproc)) - return PTR_ERR(coproc); - ave->coproc = coproc; - ave->mailbox = mailbox; - ave->domain = iommu_get_domain_for_dev(dev); - if (!ave->domain || !(ave->domain->type & __IOMMU_DOMAIN_PAGING)) - return dev_err_probe(dev, -ENODEV, "A translated IOMMU domain is required\n"); - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); - if (ret) - return ret; - ret = ave_power_on(ave); - if (ret) - goto release_power; - ret = devm_pm_runtime_enable(dev); - if (ret) - goto release_power; - ret = pm_runtime_resume_and_get(dev); - if (ret < 0) - goto release_power; - ret = ave_alloc_buffer(ave, &ave->ipc, AVE_IPC_BYTES); - if (ret) - goto free_buffers; - ret = ave_alloc_buffer(ave, &ave->queue, AVE_QUEUE_BYTES); - if (ret) - goto free_buffers; - ret = ave_alloc_buffer(ave, &ave->log, AVE_LOG_BYTES); - if (ret) - goto free_buffers; - ret = ave_verify_firmware(ave); - if (ret) - goto free_buffers; - if (snapshot) { - ave->vcbar = readq(coproc + 0x50000); - ave->core_control = readl(coproc + 0x600044); - ave->core_status = readl(coproc + 0x600048); - ave->mailbox_status = readl(mailbox + 0x10); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) - ave->scratch[i] = readl(mailbox + 0x18 + i * 4); - } - platform_set_drvdata(pdev, ave); - ave->debugfs = debugfs_create_dir(dev_name(dev), NULL); - debugfs_create_file("ave-status", 0400, ave->debugfs, ave, &ave_status_fops); - debugfs_create_file("ave-start-once", 0200, ave->debugfs, ave, &ave_start_fops); - debugfs_create_file("ave-command", 0600, ave->debugfs, ave, &ave_command_fops); - debugfs_create_file("ave-poll", 0200, ave->debugfs, ave, &ave_poll_fops); - debugfs_create_file("ave-arena", 0600, ave->debugfs, ave, &ave_arena_fops); - ave->ipc_blob.data = ave->ipc.cpu; - ave->ipc_blob.size = ave->ipc.size; - debugfs_create_blob("ave-ipc-memory", 0400, ave->debugfs, &ave->ipc_blob); - ave->log_blob.data = ave->log.cpu; - ave->log_blob.size = ave->log.size; - debugfs_create_blob("ave-firmware-log", 0400, ave->debugfs, &ave->log_blob); - ave->config_blob.data = ave->queue.cpu; - ave->config_blob.size = 0x40; - debugfs_create_blob("ave-boot-config", 0400, ave->debugfs, &ave->config_blob); - dev_info(dev, "AVE_DMA_READY ipc=%zu queue=%zu verified_pages=%u firmware_start=0\n", - ave->ipc.size, ave->queue.size, - ave->ipc.verified_pages + ave->queue.verified_pages); - dev_info(dev, "AVE_FIRMWARE_MAPPED pages=%u contents_match_boot=1 register_snapshot=%u\n", - ave->fw[0].verified_pages + ave->fw[1].verified_pages, snapshot); - return 0; - -free_buffers: - ave_free_buffer(ave, &ave->heap); - ave_free_buffer(ave, &ave->log); - ave_free_buffer(ave, &ave->queue); - ave_free_buffer(ave, &ave->ipc); - pm_runtime_put_sync(dev); -release_power: - ave_power_off(ave); - return dev_err_probe(dev, ret, "AVE DMA allocation/verification failed\n"); -} - -static void apple_ave_remove(struct platform_device *pdev) -{ - struct apple_ave *ave = platform_get_drvdata(pdev); - - debugfs_remove_recursive(ave->debugfs); - ave_free_buffer(ave, &ave->heap); - ave_free_buffer(ave, &ave->log); - ave_free_buffer(ave, &ave->queue); - ave_free_buffer(ave, &ave->ipc); - pm_runtime_put_sync(ave->dev); - ave_power_off(ave); - dev_info(ave->dev, "AVE_DMA_RELEASED\n"); -} - -static const struct of_device_id apple_ave_match[] = { - { .compatible = "apple,t8140-ave" }, - { } -}; -MODULE_DEVICE_TABLE(of, apple_ave_match); - -static struct platform_driver apple_ave_driver = { - .probe = apple_ave_probe, - .remove = apple_ave_remove, - .driver = { - .name = "apple-ave", - .of_match_table = apple_ave_match, - .suppress_bind_attrs = true, - }, -}; -module_platform_driver(apple_ave_driver); -MODULE_DESCRIPTION("Apple T8140 experimental video encoder transport"); -MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c new file mode 100644 index 0000000000000..6948bc60496a6 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: firmware commands + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include + +#include + +#include "ave.h" +#include "ave-regs.h" + +static void ave_cmd_header(struct ave_dev *ave, u8 *cmd, enum ave_opcode op, + u32 client, enum ave_codec codec) +{ + put_unaligned_le32(op, cmd + AVE_CMD_OPCODE); + put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); + put_unaligned_le64(client, cmd + AVE_CMD_CLIENT); + put_unaligned_le32(1, cmd + AVE_CMD_UNIT); + put_unaligned_le32(codec, cmd + AVE_CMD_CODEC); + put_unaligned_le32(op, cmd + AVE_CMD_OPCODE_ECHO); +} + +/* Firmware-wide configuration, sent once after every firmware boot. */ +int ave_cmd_config(struct ave_dev *ave) +{ + u8 cmd[AVE_CONFIG_BYTES] = {}; + + put_unaligned_le32(AVE_OP_CONFIG, cmd + AVE_CMD_OPCODE); + put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); + put_unaligned_le32(U32_MAX, cmd + AVE_CONFIG_ALL_UNITS); + cmd[AVE_CONFIG_OPT_A] = 1; + cmd[AVE_CONFIG_OPT_B] = 1; + return ave_ipc_call(ave, cmd, sizeof(cmd)); +} + +/* Firmware-wide stop, sent after every session has closed. */ +int ave_cmd_halt(struct ave_dev *ave) +{ + u8 cmd[AVE_CMD_HDR_BYTES] = {}; + + put_unaligned_le32(AVE_OP_HALT, cmd + AVE_CMD_OPCODE); + put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); + put_unaligned_le32(AVE_HALT_ALL, cmd + AVE_CMD_OPCODE_ECHO); + return ave_ipc_call(ave, cmd, sizeof(cmd)); +} + +static int ave_cmd_simple(struct ave_ctx *ctx, enum ave_opcode op, enum ave_codec codec, + u32 arg0, u32 arg2) +{ + u8 cmd[AVE_CMD_HDR_BYTES] = {}; + + ave_cmd_header(ctx->ave, cmd, op, ctx->client, codec); + put_unaligned_le32(arg0, cmd + AVE_CMD_ARG0); + put_unaligned_le32(AVE_SESSION_ARG1, cmd + AVE_CMD_ARG1); + put_unaligned_le32(arg2, cmd + AVE_CMD_ARG2); + return ave_ipc_call(ctx->ave, cmd, sizeof(cmd)); +} + +int ave_cmd_open(struct ave_ctx *ctx, enum ave_codec codec) +{ + struct ave_dev *ave = ctx->ave; + int ret; + + ctx->client = ++ave->next_client; + ret = ave_cmd_simple(ctx, AVE_OP_OPEN, codec, AVE_SESSION_ARG0, AVE_OPEN_ARG2); + if (!ret) + ctx->session_open = true; + return ret; +} + +/* + * Session parameters. The Start layout (resolution, formats, rate control, + * GOP, the per-session buffer table) is not described yet, so sessions cannot + * start; everything around it is in place. + */ +int ave_cmd_start(struct ave_ctx *ctx, enum ave_codec codec) +{ + return -EOPNOTSUPP; +} + +int ave_cmd_process(struct ave_ctx *ctx, enum ave_codec codec, + struct vb2_v4l2_buffer *src, struct vb2_v4l2_buffer *dst, + u32 *bytesused, bool *keyframe) +{ + return -EOPNOTSUPP; +} + +/* Flush, stop and close the session. */ +int ave_cmd_finish(struct ave_ctx *ctx, enum ave_codec codec) +{ + u32 tail = codec == AVE_CODEC_HEVC ? AVE_FINISH_ARG2_HEVC : AVE_FINISH_ARG2_H264; + int ret, err; + + if (!ctx->session_open) + return 0; + + ret = ave_cmd_simple(ctx, AVE_OP_COMPLETE, codec, 0, tail); + err = ave_cmd_simple(ctx, AVE_OP_STOP, codec, AVE_SESSION_ARG0, tail); + ret = ret ?: err; + err = ave_cmd_simple(ctx, AVE_OP_CLOSE, codec, AVE_SESSION_ARG0, tail); + ctx->session_open = false; + return ret ?: err; +} diff --git a/drivers/media/platform/apple/ave/ave-drv.c b/drivers/media/platform/apple/ave/ave-drv.c new file mode 100644 index 0000000000000..edaaf29aa6739 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-drv.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder + * + * The AVE block runs an encoder firmware on its own coprocessor, which the + * bootloader loads but does not start. This driver starts it on first use, + * talks to it over two shared-memory rings and exposes it as a V4L2 + * stateful encoder. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" + +#define AVE_AUTOSUSPEND_MS 2000 + +static const char * const ave_pd_names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; + +static int ave_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + const struct dev_pm_domain_attach_data pd_data = { + .pd_names = ave_pd_names, + .num_pd_names = ARRAY_SIZE(ave_pd_names), + .pd_flags = PD_FLAG_DEV_LINK_ON, + }; + struct ave_dev *ave; + int ret; + + ave = devm_kzalloc(dev, sizeof(*ave), GFP_KERNEL); + if (!ave) + return -ENOMEM; + ave->dev = dev; + mutex_init(&ave->hw_lock); + platform_set_drvdata(pdev, ave); + + ave->codec = devm_platform_ioremap_resource_byname(pdev, "codec"); + if (IS_ERR(ave->codec)) + return PTR_ERR(ave->codec); + ave->mbox = devm_platform_ioremap_resource_byname(pdev, "mailbox"); + if (IS_ERR(ave->mbox)) + return PTR_ERR(ave->mbox); + ave->coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); + if (IS_ERR(ave->coproc)) + return PTR_ERR(ave->coproc); + + ave->domain = iommu_get_domain_for_dev(dev); + if (!ave->domain || !(ave->domain->type & __IOMMU_DOMAIN_PAGING)) + return dev_err_probe(dev, -ENODEV, "needs a translating IOMMU\n"); + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(42)); + if (ret) + return ret; + + /* The encoder pipes, the DMA block and the motion engines. */ + ret = devm_pm_domain_attach_list(dev, &pd_data, &ave->pd_list); + if (ret < 0) + return dev_err_probe(dev, ret, "cannot attach power domains\n"); + + pm_runtime_set_autosuspend_delay(dev, AVE_AUTOSUSPEND_MS); + pm_runtime_use_autosuspend(dev); + ret = devm_pm_runtime_enable(dev); + if (ret) + return ret; + + ret = ave_fw_probe(ave); + if (ret) + return ret; + + ret = ave_v4l2_register(ave); + if (ret) { + ave_fw_remove(ave); + return ret; + } + return 0; +} + +static void ave_remove(struct platform_device *pdev) +{ + struct ave_dev *ave = platform_get_drvdata(pdev); + + ave_v4l2_unregister(ave); + + /* + * Make sure the coprocessor really loses power before its memory is + * freed: drop to runtime suspend synchronously and check. + */ + pm_runtime_dont_use_autosuspend(ave->dev); + if (pm_runtime_suspended(ave->dev) || pm_runtime_status_suspended(ave->dev)) { + mutex_lock(&ave->hw_lock); + ave_fw_lost(ave); + mutex_unlock(&ave->hw_lock); + ave_fw_remove(ave); + } else { + /* The firmware still owns its buffers; leave them allocated. */ + dev_warn(ave->dev, "firmware still running, keeping its memory\n"); + } +} + +static int ave_runtime_suspend(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + int ret; + + mutex_lock(&ave->hw_lock); + ret = ave_fw_shutdown(ave); + mutex_unlock(&ave->hw_lock); + return ret; +} + +static int ave_runtime_resume(struct device *dev) +{ + return 0; +} + +static int ave_suspend(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + + /* Let a running frame finish; the m2m core holds back new jobs. */ + v4l2_m2m_suspend(ave->m2m_dev); + + mutex_lock(&ave->hw_lock); + if (ave->fw_state != AVE_FW_OFF && ave_fw_shutdown(ave)) + dev_dbg(dev, "firmware will restart after resume\n"); + mutex_unlock(&ave->hw_lock); + return 0; +} + +static int ave_resume(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + + /* + * If the power domains went down with the system, the coprocessor is + * back in reset: open sessions fail their next frame and a new session + * boots the firmware again from its saved state. + */ + mutex_lock(&ave->hw_lock); + ave_fw_resume(ave); + mutex_unlock(&ave->hw_lock); + + v4l2_m2m_resume(ave->m2m_dev); + return 0; +} + +static const struct dev_pm_ops ave_pm_ops = { + SYSTEM_SLEEP_PM_OPS(ave_suspend, ave_resume) + RUNTIME_PM_OPS(ave_runtime_suspend, ave_runtime_resume, NULL) +}; + +static const struct of_device_id ave_of_match[] = { + { .compatible = "apple,t8140-ave" }, + { } +}; +MODULE_DEVICE_TABLE(of, ave_of_match); + +static struct platform_driver ave_driver = { + .probe = ave_probe, + .remove = ave_remove, + .driver = { + .name = "apple-ave", + .of_match_table = ave_of_match, + .pm = pm_ptr(&ave_pm_ops), + /* The firmware cannot be stopped yet, so neither can the driver. */ + .suppress_bind_attrs = true, + }, +}; +module_platform_driver(ave_driver); + +MODULE_DESCRIPTION("Apple AVE video encoder"); +MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c new file mode 100644 index 0000000000000..fa2df964d16f4 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -0,0 +1,359 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: firmware lifecycle + * + * The bootloader leaves the coprocessor firmware in two reserved, DART-mapped + * segments: TEXT and DATA. Running the firmware changes DATA, so the driver + * keeps a copy of DATA taken before the first boot and puts it back before + * every later boot. That lets the coprocessor lose power (runtime PM, system + * sleep) and start again from the same state as the first time. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +/* + * Firmware builds whose command layouts this driver implements, by the CRC32 + * of their TEXT segment. The command formats change between builds. + */ +static const u32 ave_known_text_crc[] = { + 0x2ce7359e, /* AppleAVE2FW 9003.78.0 */ + 0x69c54454, /* AppleAVE2FW 9003.78.0 (macOS 26.0.x) */ +}; + +static int ave_fw_chosen_crc(struct ave_dev *ave, const char *name, u32 *crc) +{ + struct device_node *chosen = of_find_node_by_path("/chosen"); + const char *str; + int ret; + + if (!chosen) + return -ENOENT; + ret = of_property_read_string(chosen, name, &str); + of_node_put(chosen); + if (ret) + return ret; + return kstrtou32(str, 16, crc); +} + +static int ave_fw_segment(struct ave_dev *ave, unsigned int index, + struct ave_fw_segment *seg) +{ + struct device_node *np; + struct resource res; + const __be32 *prop; + int len, ret; + + np = of_parse_phandle(ave->dev->of_node, "memory-region", index); + if (!np) + return -ENOENT; + ret = of_address_to_resource(np, 0, &res); + if (ret) + goto out; + + /* iommu-addresses = <&ave iova-hi iova-lo size-hi size-lo> */ + prop = of_get_property(np, "iommu-addresses", &len); + if (!prop || len != 5 * sizeof(u32) || + be32_to_cpup(prop) != ave->dev->of_node->phandle) { + ret = -EINVAL; + goto out; + } + seg->phys = res.start; + seg->size = resource_size(&res); + seg->iova = of_read_number(prop + 1, 2); + if (of_read_number(prop + 3, 2) != seg->size || + !IS_ALIGNED(seg->phys, SZ_16K) || !IS_ALIGNED(seg->iova, SZ_16K)) + ret = -EINVAL; +out: + of_node_put(np); + return ret; +} + +static int ave_fw_check_mapping(struct ave_dev *ave, const struct ave_fw_segment *seg) +{ + for (size_t off = 0; off < seg->size; off += PAGE_SIZE) + if (iommu_iova_to_phys(ave->domain, seg->iova + off) != seg->phys + off) + return -EIO; + return 0; +} + +static u32 ave_fw_crc(const struct ave_fw_segment *seg, void **copy) +{ + void *va = memremap(seg->phys, seg->size, MEMREMAP_WB); + u32 crc; + + if (!va) + return 0; + crc = crc32_le(~0U, va, seg->size) ^ ~0U; + if (copy) + memcpy(*copy, va, seg->size); + memunmap(va); + return crc; +} + +/* + * Find the firmware, check that it is a build we speak to and that DATA is + * still as the bootloader left it, and keep a copy of DATA for restarts. + */ +int ave_fw_probe(struct ave_dev *ave) +{ + struct device *dev = ave->dev; + u32 text_crc, data_crc, want; + bool known = false; + int ret; + + if (of_count_phandle_with_args(dev->of_node, "memory-region", NULL) != 2) + return dev_err_probe(dev, -ENODEV, "no preloaded firmware\n"); + + ret = ave_fw_segment(ave, 0, &ave->text); + if (!ret) + ret = ave_fw_segment(ave, 1, &ave->data); + if (ret) + return dev_err_probe(dev, ret, "bad firmware reservation\n"); + + if (ave_fw_check_mapping(ave, &ave->text) || ave_fw_check_mapping(ave, &ave->data)) + return dev_err_probe(dev, -EIO, "firmware is not mapped where it was loaded\n"); + + text_crc = ave_fw_crc(&ave->text, NULL); + for (unsigned int i = 0; i < ARRAY_SIZE(ave_known_text_crc); i++) + known |= text_crc == ave_known_text_crc[i]; + if (!known) + return dev_err_probe(dev, -ENODEV, "unsupported firmware (text CRC %08x)\n", + text_crc); + + ave->data_pristine = vmalloc(ave->data.size); + if (!ave->data_pristine) + return -ENOMEM; + data_crc = ave_fw_crc(&ave->data, &ave->data_pristine); + + /* + * The loader records the DATA checksum before anything runs. If it + * differs, the firmware already ran and there is no clean copy left. + */ + ret = ave_fw_chosen_crc(ave, "aurora,ave-data-crc32", &want); + if (ret || data_crc != want) { + vfree(ave->data_pristine); + ave->data_pristine = NULL; + return dev_err_probe(dev, -ENODEV, + "firmware data is not in its boot state\n"); + } + + ave->fw_state = AVE_FW_OFF; + return 0; +} + +static void ave_fw_restore_data(struct ave_dev *ave) +{ + void *va; + + if (!ave->data_dirty) + return; + va = memremap(ave->data.phys, ave->data.size, MEMREMAP_WC); + if (WARN_ON(!va)) + return; + memcpy(va, ave->data_pristine, ave->data.size); + wmb(); + memunmap(va); + ave->data_dirty = false; +} + +static void ave_free(struct ave_dev *ave, struct ave_buf *b) +{ + if (b->cpu) + dma_free_coherent(ave->dev, b->size, b->cpu, b->dma); + b->cpu = NULL; +} + +static int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size) +{ + if (b->cpu) + return 0; + b->size = size; + b->cpu = dma_alloc_coherent(ave->dev, size, &b->dma, GFP_KERNEL); + if (!b->cpu) + return -ENOMEM; + if (!IS_ALIGNED(b->dma, SZ_16K)) { + ave_free(ave, b); + return -EINVAL; + } + return 0; +} + +void ave_fw_remove(struct ave_dev *ave) +{ + ave_free(ave, &ave->heap); + ave_free(ave, &ave->log); + ave_free(ave, &ave->boot_cfg); + ave_free(ave, &ave->ipc); + vfree(ave->data_pristine); + ave->data_pristine = NULL; +} + +static bool ave_fw_idle(struct ave_dev *ave) +{ + if (readq(ave->coproc + AVE_COPROC_VCBAR) != (ave->text.iova | AVE_COPROC_VCBAR_VALID)) + return false; + if (readl(ave->coproc + AVE_COPROC_CTRL) || readl(ave->mbox + AVE_MBOX_STATUS)) + return false; + for (unsigned int i = 0; i < AVE_MBOX_SCRATCH_COUNT; i++) + if (readl(ave->mbox + AVE_MBOX_SCRATCH(i))) + return false; + return true; +} + +/* Boot the coprocessor and bring up its command rings. Caller holds hw_lock. */ +static int ave_fw_boot(struct ave_dev *ave) +{ + __le32 *cfg, *log; + u32 status, hello[4]; + int ret; + + ret = ave_alloc(ave, &ave->ipc, AVE_IPC_BYTES); + if (!ret) + ret = ave_alloc(ave, &ave->boot_cfg, SZ_16K); + if (!ret) + ret = ave_alloc(ave, &ave->log, AVE_LOG_BYTES); + if (ret) + return ret; + + /* + * The bootloader leaves the coprocessor stopped with its code base + * set. Do not start it from any other state. + */ + if (!ave_fw_idle(ave)) { + dev_err(ave->dev, "coprocessor is not in its reset state (vcbar %#llx ctrl %#x)\n", + readq(ave->coproc + AVE_COPROC_VCBAR), + readl(ave->coproc + AVE_COPROC_CTRL)); + return -EIO; + } + + ave_fw_restore_data(ave); + + cfg = ave->boot_cfg.cpu; + log = ave->log.cpu; + memset(cfg, 0, AVE_BOOT_CFG_BYTES); + memset(log, 0, ave->log.size); + cfg[AVE_BOOT_CFG_DEVICE] = cpu_to_le32(AVE_BOOT_CFG_DEVICE_T8140); + cfg[AVE_BOOT_CFG_UNITS] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_GROUPS] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_SUBID_MASK] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_LOG_LO] = cpu_to_le32(lower_32_bits(ave->log.dma)); + cfg[AVE_BOOT_CFG_LOG_HI] = cpu_to_le32(upper_32_bits(ave->log.dma)); + cfg[AVE_BOOT_CFG_LOG_BYTES] = cpu_to_le32(AVE_LOG_BYTES); + cfg[AVE_BOOT_CFG_OPTIONS] = cpu_to_le32(AVE_BOOT_CFG_OPTIONS_VALUE); + log[0] = cpu_to_le32(AVE_LOG_HDR_USED); + log[1] = cpu_to_le32(AVE_LOG_HDR_BYTES); + log[2] = cpu_to_le32(AVE_LOG_RING_BYTES); + dma_wmb(); + + writel(AVE_BOOT_MAGIC, ave->mbox + AVE_MBOX_SCRATCH(0)); + writel(lower_32_bits(ave->boot_cfg.dma), ave->mbox + AVE_MBOX_SCRATCH(1)); + writel(upper_32_bits(ave->boot_cfg.dma), ave->mbox + AVE_MBOX_SCRATCH(2)); + + ave->data_dirty = true; + writel(1, ave->coproc + AVE_COPROC_KEEP_MAPPINGS); + writel(0, ave->coproc + AVE_COPROC_CTRL); + writel(AVE_COPROC_BOOT_ARG_VALUE, ave->coproc + AVE_COPROC_BOOT_ARG); + writel(AVE_COPROC_CTRL_RUN, ave->coproc + AVE_COPROC_CTRL); + + ret = readl_poll_timeout(ave->mbox + AVE_MBOX_STATUS, status, + status & AVE_MBOX_STATUS_PENDING, 50, AVE_MBOX_TIMEOUT_US); + if (ret) { + dev_err(ave->dev, "firmware did not answer (status %#x)\n", + readl(ave->coproc + AVE_COPROC_STATUS)); + return ret; + } + for (unsigned int i = 0; i < 4; i++) + hello[i] = readl(ave->mbox + AVE_MBOX_SCRATCH(i)); + dma_rmb(); + + ret = ave_ipc_negotiate(ave, hello); + if (ret) { + dev_err(ave->dev, "firmware start-up failed: %d\n", ret); + return ret; + } + + ave->fw_state = AVE_FW_RUNNING; + ret = ave_cmd_config(ave); + if (ret) + return ret; + + dev_dbg(ave->dev, "firmware running, %u/%u ring slots\n", + ave->ring[0].slots, ave->ring[1].slots); + return 0; +} + +/* Caller holds hw_lock. */ +int ave_fw_ensure_running(struct ave_dev *ave) +{ + int ret; + + lockdep_assert_held(&ave->hw_lock); + switch (ave->fw_state) { + case AVE_FW_RUNNING: + return 0; + case AVE_FW_BROKEN: + /* Only a power cycle gets the coprocessor back; see runtime PM. */ + return -EIO; + case AVE_FW_OFF: + break; + } + + ret = ave_fw_boot(ave); + if (ret) + ave->fw_state = AVE_FW_BROKEN; + return ret; +} + +/* + * Stop the firmware before its power goes away. Caller holds hw_lock. + * + * The firmware is told to halt once every session is closed. Stopping the + * coprocessor core itself and checking that the fabric is idle are not + * described yet, so until then a running firmware keeps its power (runtime + * suspend fails with -EBUSY). Across system sleep the domains go down anyway; + * ave_fw_resume() notices and the next session boots the firmware again from + * the saved DATA copy. + */ +int ave_fw_shutdown(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state == AVE_FW_OFF) + return 0; + return -EBUSY; +} + +/* The coprocessor lost power behind the firmware's back. Caller holds hw_lock. */ +void ave_fw_lost(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + ave->fw_state = AVE_FW_OFF; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; +} + +/* After system resume: did the coprocessor keep running? Caller holds hw_lock. */ +void ave_fw_resume(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state == AVE_FW_OFF) + return; + /* A running firmware keeps the device runtime active, so this is powered. */ + if (readl(ave->coproc + AVE_COPROC_CTRL) & AVE_COPROC_CTRL_RUN) + return; + dev_dbg(ave->dev, "coprocessor lost power during sleep\n"); + ave_fw_lost(ave); +} diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c new file mode 100644 index 0000000000000..11f0aceb6fb65 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -0,0 +1,384 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: coprocessor mailbox and command rings + * + * The mailbox negotiation and the 64-byte ring protocol here come from + * Acelogic's J700 AVE transport (the first Linux code to drive this + * firmware); this file reshapes it into the driver core. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +static int ave_fw_offset(struct ave_dev *ave, u64 addr, size_t len, u32 *offset) +{ + u64 delta = addr - ave->ipc_fw_base; + + if (addr < ave->ipc_fw_base || delta > ave->ipc.size || + len > ave->ipc.size - delta) + return -ERANGE; + *offset = delta; + return 0; +} + +static void ave_mbox_send(struct ave_dev *ave, u32 a, u32 b, u32 c, u32 d) +{ + writel(a, ave->mbox + AVE_MBOX_SCRATCH(0)); + writel(b, ave->mbox + AVE_MBOX_SCRATCH(1)); + writel(c, ave->mbox + AVE_MBOX_SCRATCH(2)); + writel(d, ave->mbox + AVE_MBOX_SCRATCH(3)); + wmb(); + writel(1, ave->mbox + AVE_MBOX_DOORBELL); +} + +static int ave_mbox_recv(struct ave_dev *ave, u32 reply[4]) +{ + u32 status; + int ret; + + ret = readl_poll_timeout(ave->mbox + AVE_MBOX_STATUS, status, + status & AVE_MBOX_STATUS_PENDING, 50, + AVE_MBOX_TIMEOUT_US); + if (ret) + return ret; + + /* Acknowledge before reading the scratch words. */ + writel(AVE_MBOX_STATUS_PENDING, ave->mbox + AVE_MBOX_STATUS); + for (unsigned int i = 0; i < 4; i++) + reply[i] = readl(ave->mbox + AVE_MBOX_SCRATCH(i)); + dma_rmb(); + return 0; +} + +static int ave_ring_setup(struct ave_dev *ave) +{ + bool found[2] = { false, false }; + + for (unsigned int i = 0; i < 2; i++) { + u8 *desc = ave->ipc.cpu + AVE_IPC_CHANNEL_OFF + i * AVE_CHAN_DESC_STRIDE; + u32 dir = get_unaligned_le32(desc + AVE_CHAN_DESC_DIR); + u32 doorbell = get_unaligned_le32(desc + AVE_CHAN_DESC_IRQ); + u32 slots = get_unaligned_le32(desc + AVE_CHAN_DESC_SLOTS); + u64 addr = get_unaligned_le64(desc + AVE_CHAN_DESC_ADDR); + struct ave_ring *r; + unsigned int id; + u32 offset; + + if (!memchr(desc, 0, AVE_CHAN_DESC_NAME_LEN)) + return -EPROTO; + if (!strcmp(desc, "IO")) + id = 0; + else if (!strcmp(desc, "IO_T2H")) + id = 1; + else + return -EPROTO; + + if (found[id] || dir != id || !doorbell || doorbell >= 32 || !slots || + slots > ave->channel_bytes / AVE_RING_SLOT_BYTES || + !IS_ALIGNED(addr, AVE_RING_SLOT_BYTES) || + ave_fw_offset(ave, addr, (size_t)slots * AVE_RING_SLOT_BYTES, &offset) || + offset < AVE_IPC_CHANNEL_OFF + 2 * AVE_CHAN_DESC_STRIDE || + offset + slots * AVE_RING_SLOT_BYTES > AVE_IPC_CHANNEL_OFF + ave->channel_bytes) + return -EPROTO; + + found[id] = true; + r = &ave->ring[id]; + strscpy(r->name, desc, sizeof(r->name)); + /* Host-to-firmware rings start empty, firmware-to-host rings full. */ + r->mode = dir ^ 1; + r->doorbell = doorbell; + r->slots = slots; + r->offset = offset; + r->consumer = r->mode ? -1 : 0; + r->producer = r->mode ? 0 : -1; + } + + if (!found[0] || !found[1]) + return -EPROTO; + if (ave->ring[0].offset < ave->ring[1].offset + ave->ring[1].slots * AVE_RING_SLOT_BYTES && + ave->ring[1].offset < ave->ring[0].offset + ave->ring[0].slots * AVE_RING_SLOT_BYTES) + return -EPROTO; + + /* Hand every host-to-firmware slot to the host; ownership is bit 0. */ + for (unsigned int i = 0; i < ave->ring[0].slots; i++) { + __le64 *slot = ave->ipc.cpu + ave->ring[0].offset + i * AVE_RING_SLOT_BYTES; + + slot[0] = cpu_to_le64(1); + slot[1] = 0; + slot[2] = 0; + } + dma_wmb(); + return 0; +} + +/* + * Finish the coprocessor start-up once the firmware has answered the boot + * request: hand it the heap and the IPC window, then the channel area, and + * read back the two ring descriptors it writes there. + */ +int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]) +{ + u8 *hs = ave->ipc.cpu + AVE_IPC_HANDSHAKE_OFF; + u64 ticks, offset_us; + u32 freq, status, reply[4], off; + int ret; + + if (hello[0] != AVE_HELLO_VERSION || hello[2] != AVE_HELLO_DESC_BYTES || + hello[1] < 2 * AVE_CHAN_DESC_STRIDE || + hello[1] > AVE_IPC_HANDSHAKE_OFF - AVE_IPC_CHANNEL_OFF || + !hello[3] || hello[3] > 4 * SZ_1M) + return -EPROTO; + ave->channel_bytes = hello[1]; + + if (ave->heap.cpu && ave->heap.size != PAGE_ALIGN(hello[3])) + return -EPROTO; + if (!ave->heap.cpu) { + ave->heap.size = PAGE_ALIGN(hello[3]); + ave->heap.cpu = dma_alloc_coherent(ave->dev, ave->heap.size, + &ave->heap.dma, GFP_KERNEL); + if (!ave->heap.cpu) + return -ENOMEM; + } + memset(ave->heap.cpu, 0, ave->heap.size); + writel(AVE_MBOX_STATUS_PENDING, ave->mbox + AVE_MBOX_STATUS); + + /* Tell the firmware how far its timebase is from the host's. */ + ticks = readq(ave->coproc + AVE_COPROC_TIMEBASE); + freq = readl(ave->coproc + AVE_COPROC_TIMEBASE_FREQ); + if (freq && freq < 1000000) + return -EPROTO; + ticks = div64_u64(ticks, freq ? freq / 1000000 : 24); + offset_us = div_u64(ktime_get_ns(), NSEC_PER_USEC) - ticks; + writel(lower_32_bits(offset_us), ave->mbox + AVE_MBOX_TIME_LO); + writel(upper_32_bits(offset_us), ave->mbox + AVE_MBOX_TIME_HI); + + ave_mbox_send(ave, lower_32_bits(ave->ipc.dma), upper_32_bits(ave->ipc.dma), + ave->ipc.size, 0); + ret = ave_mbox_recv(ave, reply); + if (ret) + return ret; + ave->ipc_fw_base = (u64)reply[1] << 32 | reply[0]; + if (!ave->ipc_fw_base || !IS_ALIGNED(ave->ipc_fw_base, SZ_16K) || + ave->ipc_fw_base > U64_MAX - ave->ipc.size) + return -EPROTO; + + memset(ave->ipc.cpu + AVE_IPC_CHANNEL_OFF, 0, ave->channel_bytes); + memset(hs, 0, AVE_HS_BYTES); + memset(ave->ipc.cpu + AVE_IPC_SHARED_OFF, 0, AVE_IPC_SHARED_BYTES); + put_unaligned_le64(ave->ipc_fw_base + AVE_IPC_CHANNEL_OFF, hs + AVE_HS_CHANNELS); + put_unaligned_le64(ave->ipc_fw_base + AVE_IPC_SHARED_OFF, hs + AVE_HS_SHARED); + put_unaligned_le32(AVE_IPC_SHARED_BYTES, hs + AVE_HS_SHARED_BYTES); + put_unaligned_le64(ave->heap.dma, hs + AVE_HS_HEAP); + put_unaligned_le32(hello[3], hs + AVE_HS_HEAP_BYTES); + put_unaligned_le32(AVE_HS_MAGIC_VALUE, hs + AVE_HS_MAGIC); + dma_wmb(); + + ave_mbox_send(ave, lower_32_bits(ave->ipc_fw_base + AVE_IPC_HANDSHAKE_OFF), + upper_32_bits(ave->ipc_fw_base + AVE_IPC_HANDSHAKE_OFF), 0, 0); + ret = ave_mbox_recv(ave, reply); + if (ret) + return ret; + if (ave_fw_offset(ave, (u64)reply[1] << 32 | reply[0], ave->channel_bytes, &off) || + off != AVE_IPC_CHANNEL_OFF || reply[2] > AVE_HELLO_EXTRA_MAX) + return -EPROTO; + + ret = ave_ring_setup(ave); + if (ret) + return ret; + + writel(AVE_BOOT_MAGIC, ave->mbox + AVE_MBOX_READY); + return readl_poll_timeout(ave->mbox + AVE_MBOX_READY, status, !status, 50, + AVE_MBOX_TIMEOUT_US); +} + +static int ave_ring_send(struct ave_dev *ave, struct ave_ring *r, u64 addr, u32 len) +{ + __le64 *slot; + int next; + + if (r->producer < 0) + return -ENOSPC; + + slot = ave->ipc.cpu + r->offset + r->producer * AVE_RING_SLOT_BYTES; + WRITE_ONCE(slot[1], cpu_to_le64(len)); + WRITE_ONCE(slot[2], 0); + dma_wmb(); + WRITE_ONCE(slot[0], cpu_to_le64(addr | (r->mode ^ 1))); + dma_wmb(); + + if (r->consumer < 0) + r->consumer = r->producer; + next = (r->producer + 1) % r->slots; + r->producer = next == r->consumer ? -1 : next; + writel(BIT(r->doorbell), ave->mbox + AVE_MBOX_DOORBELL); + return 0; +} + +static int ave_ring_recv(struct ave_dev *ave, struct ave_ring *r, u64 *addr, u32 *len) +{ + __le64 *slot; + u64 word; + int next; + + if (r->consumer < 0) + return -EAGAIN; + + slot = ave->ipc.cpu + r->offset + r->consumer * AVE_RING_SLOT_BYTES; + word = le64_to_cpu(READ_ONCE(slot[0])); + if ((word & 1) != r->mode) + return -EAGAIN; + dma_rmb(); + *addr = word & ~3ULL; + *len = le64_to_cpu(READ_ONCE(slot[1])); + + if (r->producer < 0) + r->producer = r->consumer; + next = (r->consumer + 1) % r->slots; + r->consumer = next == r->producer ? -1 : next; + return 0; +} + +/* + * Poll both rings once. Returns 1 when the command at @cmd_addr was + * acknowledged, 2 when a firmware message was copied to ave->msg (and its + * buffer handed back), 0 when nothing was pending. + */ +static int ave_ipc_poll_once(struct ave_dev *ave, u64 cmd_addr) +{ + u32 pending, len, off; + u64 addr; + int ret; + + pending = readl(ave->mbox + AVE_MBOX_STATUS); + if (pending) + writel(pending, ave->mbox + AVE_MBOX_STATUS); + + ret = ave_ring_recv(ave, &ave->ring[0], &addr, &len); + if (!ret) { + /* Transport acks return the command address with no payload. */ + if (len || addr != cmd_addr) + return -EPROTO; + return 1; + } + if (ret != -EAGAIN) + return ret; + + ret = ave_ring_recv(ave, &ave->ring[1], &addr, &len); + if (ret == -EAGAIN) + return 0; + if (ret) + return ret; + if (len > sizeof(ave->msg) || ave_fw_offset(ave, addr, len, &off)) + return -EPROTO; + memcpy(ave->msg, ave->ipc.cpu + off, len); + ave->msg_len = len; + /* Return the message buffer to the firmware. */ + ret = ave_ring_send(ave, &ave->ring[1], addr, len); + return ret ?: 2; +} + +/* + * Send one command and wait for its transport ack and the firmware's reply + * message. The reply is left in ave->msg. Caller holds hw_lock. + */ +int ave_ipc_call(struct ave_dev *ave, const void *cmd, size_t len) +{ + u64 cmd_addr = ave->ipc_fw_base + AVE_IPC_CMD_OFF; + u8 *buf = ave->ipc.cpu + AVE_IPC_CMD_OFF; + bool acked = false, replied = false; + ktime_t deadline; + u32 status; + int ret; + + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) + return -EIO; + if (len < AVE_CMD_HDR_BYTES || len > AVE_IPC_CMD_BYTES || !IS_ALIGNED(len, 4)) + return -EINVAL; + + memcpy(buf, cmd, len); + /* The timeout is {duration, creation} in the clock synced at boot. */ + put_unaligned_le64(AVE_CMD_TIMEOUT_US, buf + AVE_CMD_TIMEOUT); + put_unaligned_le64(div_u64(ktime_get_ns(), NSEC_PER_USEC), buf + AVE_CMD_CREATED); + ave->msg_len = 0; + + ret = ave_ring_send(ave, &ave->ring[0], cmd_addr, len); + if (ret) + goto broken; + + deadline = ktime_add_ms(ktime_get(), AVE_CMD_WAIT_MS); + while (!acked || !replied) { + ret = ave_ipc_poll_once(ave, cmd_addr); + if (ret < 0) + goto broken; + if (ret == 1) + acked = true; + else if (ret == 2) + replied = true; + else if (ktime_after(ktime_get(), deadline)) + goto timeout; + else + usleep_range(50, 150); + } + + /* The firmware refused the command: the session fails, the firmware lives on. */ + if (ave->msg_len < AVE_REPLY_MIN_BYTES) + goto broken_len; + status = get_unaligned_le32(ave->msg + AVE_REPLY_STATUS); + if (status) { + dev_err(ave->dev, "command %u refused: status %#x\n", + get_unaligned_le32(cmd), status); + return -EIO; + } + return 0; + +broken_len: + ret = -EPROTO; + goto broken; +timeout: + ret = -ETIMEDOUT; +broken: + dev_err(ave->dev, "command %u failed: %d (acked %d, replied %d)\n", + get_unaligned_le32(cmd), ret, acked, replied); + ave->fw_state = AVE_FW_BROKEN; + return ret; +} + +/* Wait for one more firmware message (frame completions). */ +int ave_ipc_wait_msg(struct ave_dev *ave, unsigned int timeout_ms) +{ + ktime_t deadline = ktime_add_ms(ktime_get(), timeout_ms); + int ret; + + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) + return -EIO; + + for (;;) { + ret = ave_ipc_poll_once(ave, 0); + if (ret == 2) + return 0; + if (ret < 0) + break; + if (ktime_after(ktime_get(), deadline)) { + ret = -ETIMEDOUT; + break; + } + usleep_range(50, 150); + } + dev_err(ave->dev, "no firmware message: %d\n", ret); + ave->fw_state = AVE_FW_BROKEN; + return ret; +} diff --git a/drivers/media/platform/apple/ave/ave-regs.h b/drivers/media/platform/apple/ave/ave-regs.h new file mode 100644 index 0000000000000..91e7c2c403c3e --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-regs.h @@ -0,0 +1,128 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Apple AVE video encoder: registers and firmware interface constants + * + * Copyright The Aurora Silicon Contributors + */ +#ifndef _APPLE_AVE_REGS_H_ +#define _APPLE_AVE_REGS_H_ + +#include + +/* Mailbox window */ +#define AVE_MBOX_DOORBELL 0x0c +#define AVE_MBOX_STATUS 0x10 +#define AVE_MBOX_STATUS_PENDING BIT(0) +#define AVE_MBOX_SCRATCH(n) (0x18 + (n) * 4) +#define AVE_MBOX_SCRATCH_COUNT 8 +#define AVE_MBOX_READY AVE_MBOX_SCRATCH(3) +#define AVE_MBOX_TIME_LO AVE_MBOX_SCRATCH(4) +#define AVE_MBOX_TIME_HI AVE_MBOX_SCRATCH(5) +#define AVE_MBOX_TIMEOUT_US 5000000 + +/* Coprocessor window */ +#define AVE_COPROC_VCBAR 0x50000 +#define AVE_COPROC_VCBAR_VALID BIT(0) +#define AVE_COPROC_TIMEBASE_FREQ 0x360020 +#define AVE_COPROC_TIMEBASE 0x378000 +#define AVE_COPROC_CTRL 0x600044 +#define AVE_COPROC_CTRL_RUN BIT(4) +#define AVE_COPROC_STATUS 0x600048 +#define AVE_COPROC_BOOT_ARG 0x600400 +#define AVE_COPROC_BOOT_ARG_VALUE 0x10000 +#define AVE_COPROC_KEEP_MAPPINGS 0x600808 + +/* Boot request and the configuration block it points at */ +#define AVE_BOOT_MAGIC 0x08042006 +#define AVE_BOOT_CFG_BYTES 0x40 +#define AVE_BOOT_CFG_DEVICE 1 /* word index */ +#define AVE_BOOT_CFG_DEVICE_T8140 29 +#define AVE_BOOT_CFG_UNITS 2 +#define AVE_BOOT_CFG_GROUPS 3 +#define AVE_BOOT_CFG_SUBID_MASK 4 +#define AVE_BOOT_CFG_LOG_LO 8 +#define AVE_BOOT_CFG_LOG_HI 9 +#define AVE_BOOT_CFG_LOG_BYTES 10 +#define AVE_BOOT_CFG_OPTIONS 12 +#define AVE_BOOT_CFG_OPTIONS_VALUE 0x77 + +/* Firmware log: 512-byte header, then a 128 KiB ring. Mapping is page sized. */ +#define AVE_LOG_BYTES 0x24000 +#define AVE_LOG_HDR_USED 0x20200 +#define AVE_LOG_HDR_BYTES 0x200 +#define AVE_LOG_RING_BYTES 0x20000 + +/* Hello reply: {version, channel area bytes, descriptor bytes, heap bytes} */ +#define AVE_HELLO_VERSION 2 +#define AVE_HELLO_DESC_BYTES 0x100 +#define AVE_HELLO_EXTRA_MAX 0x13c000 + +/* Handshake block in the IPC window */ +#define AVE_HS_BYTES 0x50 +#define AVE_HS_CHANNELS 0x08 +#define AVE_HS_SHARED 0x10 +#define AVE_HS_SHARED_BYTES 0x18 +#define AVE_HS_HEAP 0x1c +#define AVE_HS_HEAP_BYTES 0x24 +#define AVE_HS_MAGIC 0x28 +#define AVE_HS_MAGIC_VALUE 27 + +/* Channel descriptors the firmware writes into the channel area */ +#define AVE_CHAN_DESC_STRIDE 0x100 +#define AVE_CHAN_DESC_NAME_LEN 0x40 +#define AVE_CHAN_DESC_DIR 0x40 +#define AVE_CHAN_DESC_IRQ 0x44 +#define AVE_CHAN_DESC_SLOTS 0x48 +#define AVE_CHAN_DESC_ADDR 0x4c +#define AVE_RING_SLOT_BYTES 64 + +/* Common command header */ +#define AVE_CMD_OPCODE 0x00 +#define AVE_CMD_TOKEN 0x08 +#define AVE_CMD_CLIENT 0x10 +#define AVE_CMD_UNIT 0x18 +#define AVE_CMD_CODEC 0x1c +#define AVE_CMD_OPCODE_ECHO 0x20 +#define AVE_CMD_ARG0 0x24 +#define AVE_CMD_ARG1 0x28 +#define AVE_CMD_ARG2 0x2c +#define AVE_CMD_TIMEOUT 0x30 +#define AVE_CMD_CREATED 0x38 +#define AVE_CMD_HDR_BYTES 0x48 +#define AVE_CMD_TIMEOUT_US 3000000 +#define AVE_CMD_WAIT_MS 5000 + +enum ave_opcode { + AVE_OP_CONFIG = 1, + AVE_OP_HALT = 2, + AVE_OP_OPEN = 3, + AVE_OP_CLOSE = 4, + AVE_OP_START = 6, + AVE_OP_STOP = 7, + AVE_OP_PROCESS = 8, + AVE_OP_COMPLETE = 9, + AVE_OP_PRIORITY = 10, +}; + +/* Firmware-to-host messages */ +#define AVE_REPLY_ID 0x00 +#define AVE_REPLY_CLIENT 0x10 +#define AVE_REPLY_SUBTYPE 0x20 +#define AVE_REPLY_STATUS 0x40 +#define AVE_REPLY_MIN_BYTES 0x48 + +#define AVE_HALT_ALL 0xffffffff + +/* Session command arguments, as the firmware expects them */ +#define AVE_SESSION_ARG0 0xc8 +#define AVE_SESSION_ARG1 0x36 +#define AVE_OPEN_ARG2 0x1e0 +#define AVE_FINISH_ARG2_H264 0x140 +#define AVE_FINISH_ARG2_HEVC 0x20 + +#define AVE_CONFIG_BYTES 0x78 +#define AVE_CONFIG_ALL_UNITS 0x20 +#define AVE_CONFIG_OPT_A 0x49 +#define AVE_CONFIG_OPT_B 0x4a + +#endif diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c new file mode 100644 index 0000000000000..410b86d20b28b --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -0,0 +1,831 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: V4L2 stateful encoder interface + * + * Follows Documentation/userspace-api/media/v4l/dev-encoder.rst: the CAPTURE + * queue carries the coded stream (H.264 or HEVC), the OUTPUT queue raw NV12. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include + +#include +#include +#include +#include + +#include "ave.h" + +#define AVE_NAME "apple-ave" + +/* + * Frame size limits and input stride alignment. Provisional: conservative + * values the hardware is known to cover (up to the 2408x1506 panel and 1080p + * and 4K class streams); refined once the firmware limits are described. + */ +#define AVE_MIN_W 64 +#define AVE_MIN_H 64 +#define AVE_MAX_W 4096 +#define AVE_MAX_H 4096 +#define AVE_STEP 16 +#define AVE_STRIDE_ALIGN 64 +#define AVE_DEFAULT_W 1920 +#define AVE_DEFAULT_H 1080 + +static const u32 ave_coded_fmts[] = { + V4L2_PIX_FMT_H264, + V4L2_PIX_FMT_HEVC, +}; + +static const u32 ave_raw_fmts[] = { + V4L2_PIX_FMT_NV12, +}; + +static const struct v4l2_event ave_eos_event = { + .type = V4L2_EVENT_EOS, +}; + +enum ave_codec ave_ctx_codec(struct ave_ctx *ctx) +{ + return ctx->dst_fmt.pixelformat == V4L2_PIX_FMT_HEVC ? AVE_CODEC_HEVC : + AVE_CODEC_H264; +} + +static bool ave_fmt_listed(const u32 *list, unsigned int n, u32 fourcc) +{ + for (unsigned int i = 0; i < n; i++) + if (list[i] == fourcc) + return true; + return false; +} + +static void ave_clamp_size(u32 *w, u32 *h) +{ + *w = clamp_t(u32, ALIGN(*w, 2), AVE_MIN_W, AVE_MAX_W); + *h = clamp_t(u32, ALIGN(*h, 2), AVE_MIN_H, AVE_MAX_H); +} + +/* Raw NV12, one buffer: Y plane, then interleaved CbCr at half height. */ +static void ave_fill_raw(struct v4l2_pix_format_mplane *f) +{ + u32 stride, height; + + f->pixelformat = V4L2_PIX_FMT_NV12; + ave_clamp_size(&f->width, &f->height); + stride = ALIGN(max_t(u32, f->width, f->plane_fmt[0].bytesperline), AVE_STRIDE_ALIGN); + height = ALIGN(f->height, AVE_STEP); + f->num_planes = 1; + f->plane_fmt[0].bytesperline = stride; + f->plane_fmt[0].sizeimage = stride * height * 3 / 2; + f->field = V4L2_FIELD_NONE; + memset(f->plane_fmt[0].reserved, 0, sizeof(f->plane_fmt[0].reserved)); + memset(f->reserved, 0, sizeof(f->reserved)); +} + +static void ave_fill_coded(struct v4l2_pix_format_mplane *f) +{ + u32 min; + + if (!ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), f->pixelformat)) + f->pixelformat = ave_coded_fmts[0]; + ave_clamp_size(&f->width, &f->height); + /* Room for an intra frame at low compression plus headers. */ + min = max_t(u32, f->width * f->height * 3 / 4, SZ_256K); + f->num_planes = 1; + f->plane_fmt[0].bytesperline = 0; + f->plane_fmt[0].sizeimage = clamp_t(u32, f->plane_fmt[0].sizeimage, min, SZ_32M); + f->field = V4L2_FIELD_NONE; + memset(f->plane_fmt[0].reserved, 0, sizeof(f->plane_fmt[0].reserved)); + memset(f->reserved, 0, sizeof(f->reserved)); +} + +static void ave_reset_fmts(struct ave_ctx *ctx) +{ + struct v4l2_pix_format_mplane *src = &ctx->src_fmt, *dst = &ctx->dst_fmt; + + memset(dst, 0, sizeof(*dst)); + dst->pixelformat = V4L2_PIX_FMT_H264; + dst->width = AVE_DEFAULT_W; + dst->height = AVE_DEFAULT_H; + ave_fill_coded(dst); + + memset(src, 0, sizeof(*src)); + src->width = dst->width; + src->height = dst->height; + ave_fill_raw(src); + src->colorspace = V4L2_COLORSPACE_REC709; + src->xfer_func = V4L2_XFER_FUNC_DEFAULT; + src->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + src->quantization = V4L2_QUANTIZATION_LIM_RANGE; + dst->colorspace = src->colorspace; + dst->xfer_func = src->xfer_func; + dst->ycbcr_enc = src->ycbcr_enc; + dst->quantization = src->quantization; + + ctx->timeperframe = (struct v4l2_fract){ 1, 30 }; +} + +/* ioctls */ + +static int ave_querycap(struct file *file, void *priv, struct v4l2_capability *cap) +{ + strscpy(cap->driver, AVE_NAME, sizeof(cap->driver)); + strscpy(cap->card, "Apple AVE video encoder", sizeof(cap->card)); + return 0; +} + +static int ave_enum_fmt_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f) +{ + if (f->index >= ARRAY_SIZE(ave_coded_fmts)) + return -EINVAL; + f->pixelformat = ave_coded_fmts[f->index]; + f->flags = V4L2_FMT_FLAG_COMPRESSED; + return 0; +} + +static int ave_enum_fmt_out(struct file *file, void *priv, struct v4l2_fmtdesc *f) +{ + if (f->index >= ARRAY_SIZE(ave_raw_fmts)) + return -EINVAL; + f->pixelformat = ave_raw_fmts[f->index]; + return 0; +} + +static int ave_enum_framesizes(struct file *file, void *priv, struct v4l2_frmsizeenum *fs) +{ + if (fs->index || + !ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), fs->pixel_format)) + return -EINVAL; + fs->type = V4L2_FRMSIZE_TYPE_STEPWISE; + fs->stepwise = (struct v4l2_frmsize_stepwise){ + AVE_MIN_W, AVE_MAX_W, 2, AVE_MIN_H, AVE_MAX_H, 2, + }; + return 0; +} + +static int ave_g_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + f->fmt.pix_mp = ave_file_to_ctx(file)->dst_fmt; + return 0; +} + +static int ave_g_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + f->fmt.pix_mp = ave_file_to_ctx(file)->src_fmt; + return 0; +} + +static int ave_try_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + ave_fill_coded(&f->fmt.pix_mp); + return 0; +} + +static int ave_try_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + + ave_fill_raw(pix); + if (pix->colorspace == V4L2_COLORSPACE_DEFAULT) { + pix->colorspace = ctx->src_fmt.colorspace; + pix->xfer_func = ctx->src_fmt.xfer_func; + pix->ycbcr_enc = ctx->src_fmt.ycbcr_enc; + pix->quantization = ctx->src_fmt.quantization; + } + return 0; +} + +static int ave_s_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + + if (vb2_is_busy(vq)) + return -EBUSY; + /* The coded format comes first; the raw side follows its size. */ + ave_fill_coded(pix); + pix->colorspace = ctx->src_fmt.colorspace; + pix->xfer_func = ctx->src_fmt.xfer_func; + pix->ycbcr_enc = ctx->src_fmt.ycbcr_enc; + pix->quantization = ctx->src_fmt.quantization; + ctx->dst_fmt = *pix; + return 0; +} + +static int ave_s_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + int ret; + + if (vb2_is_busy(vq)) + return -EBUSY; + ret = ave_try_fmt_out(file, priv, f); + if (ret) + return ret; + ctx->src_fmt = *pix; + + /* The coded size follows the raw size (crop is the whole frame). */ + ctx->dst_fmt.width = pix->width; + ctx->dst_fmt.height = pix->height; + ctx->dst_fmt.colorspace = pix->colorspace; + ctx->dst_fmt.xfer_func = pix->xfer_func; + ctx->dst_fmt.ycbcr_enc = pix->ycbcr_enc; + ctx->dst_fmt.quantization = pix->quantization; + ave_fill_coded(&ctx->dst_fmt); + return 0; +} + +static int ave_g_selection(struct file *file, void *priv, struct v4l2_selection *s) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT) + return -EINVAL; + switch (s->target) { + case V4L2_SEL_TGT_CROP: + case V4L2_SEL_TGT_CROP_DEFAULT: + case V4L2_SEL_TGT_CROP_BOUNDS: + s->r = (struct v4l2_rect){ 0, 0, ctx->src_fmt.width, ctx->src_fmt.height }; + return 0; + default: + return -EINVAL; + } +} + +static int ave_s_selection(struct file *file, void *priv, struct v4l2_selection *s) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT || s->target != V4L2_SEL_TGT_CROP) + return -EINVAL; + /* Only full-frame encoding for now. */ + s->r = (struct v4l2_rect){ 0, 0, ctx->src_fmt.width, ctx->src_fmt.height }; + return 0; +} + +static int ave_g_parm(struct file *file, void *priv, struct v4l2_streamparm *a) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (!V4L2_TYPE_IS_OUTPUT(a->type)) + return -EINVAL; + a->parm.output.capability = V4L2_CAP_TIMEPERFRAME; + a->parm.output.timeperframe = ctx->timeperframe; + return 0; +} + +static int ave_s_parm(struct file *file, void *priv, struct v4l2_streamparm *a) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_fract *t = &a->parm.output.timeperframe; + + if (!V4L2_TYPE_IS_OUTPUT(a->type)) + return -EINVAL; + if (!t->numerator || !t->denominator) + *t = ctx->timeperframe; + /* 1 to 240 frames per second. */ + if ((u64)t->denominator > (u64)t->numerator * 240) + *t = (struct v4l2_fract){ 1, 240 }; + if (t->numerator > t->denominator) + *t = (struct v4l2_fract){ 1, 1 }; + ctx->timeperframe = *t; + a->parm.output.capability = V4L2_CAP_TIMEPERFRAME; + return 0; +} + +static int ave_encoder_cmd(struct file *file, void *priv, struct v4l2_encoder_cmd *ec) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + int ret; + + ret = v4l2_m2m_ioctl_try_encoder_cmd(file, priv, ec); + if (ret) + return ret; + if (!vb2_is_streaming(v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx)) || + !vb2_is_streaming(v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx))) + return 0; + + ret = v4l2_m2m_ioctl_encoder_cmd(file, priv, ec); + if (ret) + return ret; + if (ec->cmd == V4L2_ENC_CMD_STOP && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); + if (ec->cmd == V4L2_ENC_CMD_START && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx)); + return 0; +} + +static int ave_subscribe_event(struct v4l2_fh *fh, const struct v4l2_event_subscription *sub) +{ + switch (sub->type) { + case V4L2_EVENT_EOS: + return v4l2_event_subscribe(fh, sub, 0, NULL); + default: + return v4l2_ctrl_subscribe_event(fh, sub); + } +} + +static const struct v4l2_ioctl_ops ave_ioctl_ops = { + .vidioc_querycap = ave_querycap, + .vidioc_enum_fmt_vid_cap = ave_enum_fmt_cap, + .vidioc_enum_fmt_vid_out = ave_enum_fmt_out, + .vidioc_enum_framesizes = ave_enum_framesizes, + .vidioc_g_fmt_vid_cap_mplane = ave_g_fmt_cap, + .vidioc_g_fmt_vid_out_mplane = ave_g_fmt_out, + .vidioc_try_fmt_vid_cap_mplane = ave_try_fmt_cap, + .vidioc_try_fmt_vid_out_mplane = ave_try_fmt_out, + .vidioc_s_fmt_vid_cap_mplane = ave_s_fmt_cap, + .vidioc_s_fmt_vid_out_mplane = ave_s_fmt_out, + .vidioc_g_selection = ave_g_selection, + .vidioc_s_selection = ave_s_selection, + .vidioc_g_parm = ave_g_parm, + .vidioc_s_parm = ave_s_parm, + + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, + .vidioc_expbuf = v4l2_m2m_ioctl_expbuf, + .vidioc_streamon = v4l2_m2m_ioctl_streamon, + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, + + .vidioc_try_encoder_cmd = v4l2_m2m_ioctl_try_encoder_cmd, + .vidioc_encoder_cmd = ave_encoder_cmd, + + .vidioc_subscribe_event = ave_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +/* Controls */ + +static int ave_s_ctrl(struct v4l2_ctrl *ctrl) +{ + struct ave_ctx *ctx = container_of(ctrl->handler, struct ave_ctx, hdl); + + switch (ctrl->id) { + case V4L2_CID_MPEG_VIDEO_BITRATE_MODE: + ctx->bitrate_mode = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_BITRATE: + ctx->bitrate = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_BITRATE_PEAK: + ctx->bitrate_peak = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_GOP_SIZE: + ctx->gop_size = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME: + ctx->force_key_frame = true; + break; + case V4L2_CID_MPEG_VIDEO_H264_PROFILE: + ctx->h264_profile = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_LEVEL: + ctx->h264_level = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_I_PERIOD: + ctx->h264_i_period = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_MIN_QP: + ctx->h264_min_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_MAX_QP: + ctx->h264_max_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_PROFILE: + ctx->hevc_profile = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_LEVEL: + ctx->hevc_level = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_MIN_QP: + ctx->hevc_min_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP: + ctx->hevc_max_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEADER_MODE: + case V4L2_CID_MPEG_VIDEO_B_FRAMES: + case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT: + break; + default: + return -EINVAL; + } + return 0; +} + +static const struct v4l2_ctrl_ops ave_ctrl_ops = { + .s_ctrl = ave_s_ctrl, +}; + +static int ave_init_ctrls(struct ave_ctx *ctx) +{ + struct v4l2_ctrl_handler *hdl = &ctx->hdl; + const struct v4l2_ctrl_ops *ops = &ave_ctrl_ops; + struct v4l2_ctrl *c; + + v4l2_ctrl_handler_init(hdl, 18); + + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_MODE, + V4L2_MPEG_VIDEO_BITRATE_MODE_CQ, + 0, V4L2_MPEG_VIDEO_BITRATE_MODE_VBR); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE, 100000, 200000000, 1, + 8000000); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 100000, 400000000, 1, + 16000000); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, 65535, 1, 60); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); + /* No frame reordering until reference handling is described. */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 0, 1, 0); + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEADER_MODE, + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, + BIT(V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE), + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME); + + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_PROFILE, + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, + ~(BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_MAIN) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_HIGH)), + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH); + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_LEVEL, + V4L2_MPEG_VIDEO_H264_LEVEL_5_2, 0, + V4L2_MPEG_VIDEO_H264_LEVEL_4_1); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD, 0, 65535, 1, 0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MIN_QP, 0, 51, 1, 0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MAX_QP, 0, 51, 1, 51); + + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_PROFILE, + V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_10, 0, + V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN); + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_LEVEL, + V4L2_MPEG_VIDEO_HEVC_LEVEL_5_1, 0, + V4L2_MPEG_VIDEO_HEVC_LEVEL_4_1); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MIN_QP, 0, 51, 1, 0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP, 0, 51, 1, 51); + + c = v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, 1); + if (c) + c->flags |= V4L2_CTRL_FLAG_READ_ONLY; + + if (hdl->error) { + int ret = hdl->error; + + v4l2_ctrl_handler_free(hdl); + return ret; + } + ctx->fh.ctrl_handler = hdl; + return v4l2_ctrl_handler_setup(hdl); +} + +/* Queues */ + +static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned int *nplanes, + unsigned int sizes[], struct device *alloc_devs[]) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vq); + const struct v4l2_pix_format_mplane *f = + V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; + + if (*nplanes) + return (*nplanes != 1 || sizes[0] < f->plane_fmt[0].sizeimage) ? -EINVAL : 0; + *nplanes = 1; + sizes[0] = f->plane_fmt[0].sizeimage; + return 0; +} + +static int ave_buf_prepare(struct vb2_buffer *vb) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + const struct v4l2_pix_format_mplane *f = + V4L2_TYPE_IS_OUTPUT(vb->type) ? &ctx->src_fmt : &ctx->dst_fmt; + + if (vb2_plane_size(vb, 0) < f->plane_fmt[0].sizeimage) + return -EINVAL; + if (V4L2_TYPE_IS_OUTPUT(vb->type)) + vb2_set_plane_payload(vb, 0, f->plane_fmt[0].sizeimage); + return 0; +} + +static void ave_buf_queue(struct vb2_buffer *vb) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + + if (V4L2_TYPE_IS_CAPTURE(vb->type) && vb2_is_streaming(vb->vb2_queue) && + v4l2_m2m_dst_buf_is_last(ctx->fh.m2m_ctx)) { + vb2_set_plane_payload(vb, 0, 0); + vbuf->sequence = ctx->dst_sequence++; + v4l2_m2m_last_buffer_done(ctx->fh.m2m_ctx, vbuf); + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); + return; + } + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf); +} + +static void ave_return_bufs(struct ave_ctx *ctx, struct vb2_queue *q, + enum vb2_buffer_state state) +{ + struct vb2_v4l2_buffer *vbuf; + + for (;;) { + vbuf = V4L2_TYPE_IS_OUTPUT(q->type) ? + v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx) : + v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!vbuf) + break; + v4l2_m2m_buf_done(vbuf, state); + } +} + +static int ave_session_begin(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + enum ave_codec codec = ave_ctx_codec(ctx); + int ret; + + ret = pm_runtime_resume_and_get(ave->dev); + if (ret) + return ret; + + mutex_lock(&ave->hw_lock); + ret = ave_fw_ensure_running(ave); + if (!ret) + ret = ave_cmd_open(ctx, codec); + if (!ret) + ret = ave_cmd_start(ctx, codec); + if (ret && ctx->session_open) + ave_cmd_finish(ctx, codec); + mutex_unlock(&ave->hw_lock); + + if (ret) { + dev_dbg(ave->dev, "session start failed: %d\n", ret); + pm_runtime_mark_last_busy(ave->dev); + pm_runtime_put_autosuspend(ave->dev); + } + return ret; +} + +static void ave_session_end(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + + cancel_work_sync(&ctx->run_work); + mutex_lock(&ave->hw_lock); + if (ave->fw_state == AVE_FW_RUNNING) + ave_cmd_finish(ctx, ave_ctx_codec(ctx)); + ctx->session_open = false; + mutex_unlock(&ave->hw_lock); + pm_runtime_mark_last_busy(ave->dev); + pm_runtime_put_autosuspend(ave->dev); +} + +static int ave_start_streaming(struct vb2_queue *q, unsigned int count) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(q); + struct vb2_queue *other = V4L2_TYPE_IS_OUTPUT(q->type) ? + v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx) : + v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx); + int ret = 0; + + v4l2_m2m_update_start_streaming_state(ctx->fh.m2m_ctx, q); + if (V4L2_TYPE_IS_OUTPUT(q->type)) + ctx->src_sequence = 0; + else + ctx->dst_sequence = 0; + + /* The session starts once both queues stream. */ + if (vb2_is_streaming(other) && !ctx->session_open) + ret = ave_session_begin(ctx); + if (ret) + ave_return_bufs(ctx, q, VB2_BUF_STATE_QUEUED); + return ret; +} + +static void ave_stop_streaming(struct vb2_queue *q) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(q); + + if (ctx->session_open) + ave_session_end(ctx); + ave_return_bufs(ctx, q, VB2_BUF_STATE_ERROR); + v4l2_m2m_update_stop_streaming_state(ctx->fh.m2m_ctx, q); + if (V4L2_TYPE_IS_OUTPUT(q->type) && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); +} + +static const struct vb2_ops ave_qops = { + .queue_setup = ave_queue_setup, + .buf_prepare = ave_buf_prepare, + .buf_queue = ave_buf_queue, + .start_streaming = ave_start_streaming, + .stop_streaming = ave_stop_streaming, +}; + +static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *dst) +{ + struct ave_ctx *ctx = priv; + int ret; + + src->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + src->io_modes = VB2_MMAP | VB2_DMABUF; + src->drv_priv = ctx; + src->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + src->ops = &ave_qops; + src->mem_ops = &vb2_dma_contig_memops; + src->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + src->lock = &ctx->ave->vdev_lock; + src->dev = ctx->ave->dev; + ret = vb2_queue_init(src); + if (ret) + return ret; + + dst->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; + dst->io_modes = VB2_MMAP | VB2_DMABUF; + dst->drv_priv = ctx; + dst->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + dst->ops = &ave_qops; + dst->mem_ops = &vb2_dma_contig_memops; + dst->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + dst->lock = &ctx->ave->vdev_lock; + dst->dev = ctx->ave->dev; + return vb2_queue_init(dst); +} + +/* Jobs: one raw frame in, one coded frame out. */ + +static void ave_run_work(struct work_struct *work) +{ + struct ave_ctx *ctx = container_of(work, struct ave_ctx, run_work); + struct ave_dev *ave = ctx->ave; + struct vb2_v4l2_buffer *src, *dst; + enum vb2_buffer_state state = VB2_BUF_STATE_DONE; + bool keyframe = false; + u32 bytes = 0; + int ret; + + src = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); + dst = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + + mutex_lock(&ave->hw_lock); + ret = ave_cmd_process(ctx, ave_ctx_codec(ctx), src, dst, &bytes, &keyframe); + ctx->force_key_frame = false; + mutex_unlock(&ave->hw_lock); + if (ret) { + dev_dbg(ave->dev, "frame %u failed: %d\n", ctx->src_sequence, ret); + state = VB2_BUF_STATE_ERROR; + bytes = 0; + } + + src = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx); + dst = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + src->sequence = ctx->src_sequence++; + dst->sequence = ctx->dst_sequence++; + v4l2_m2m_buf_copy_metadata(src, dst); + dst->flags &= ~(V4L2_BUF_FLAG_KEYFRAME | V4L2_BUF_FLAG_PFRAME | V4L2_BUF_FLAG_LAST); + dst->flags |= keyframe ? V4L2_BUF_FLAG_KEYFRAME : V4L2_BUF_FLAG_PFRAME; + vb2_set_plane_payload(&dst->vb2_buf, 0, bytes); + + if (v4l2_m2m_is_last_draining_src_buf(ctx->fh.m2m_ctx, src)) { + dst->flags |= V4L2_BUF_FLAG_LAST; + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); + v4l2_m2m_mark_stopped(ctx->fh.m2m_ctx); + } + + v4l2_m2m_buf_done(src, state); + v4l2_m2m_buf_done(dst, state); + pm_runtime_mark_last_busy(ave->dev); + v4l2_m2m_job_finish(ave->m2m_dev, ctx->fh.m2m_ctx); +} + +static void ave_device_run(void *priv) +{ + struct ave_ctx *ctx = priv; + + queue_work(ctx->ave->wq, &ctx->run_work); +} + +static const struct v4l2_m2m_ops ave_m2m_ops = { + .device_run = ave_device_run, +}; + +/* File operations */ + +static int ave_open(struct file *file) +{ + struct ave_dev *ave = video_drvdata(file); + struct ave_ctx *ctx; + int ret; + + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return -ENOMEM; + ctx->ave = ave; + INIT_WORK(&ctx->run_work, ave_run_work); + ave_reset_fmts(ctx); + + v4l2_fh_init(&ctx->fh, video_devdata(file)); + ret = ave_init_ctrls(ctx); + if (ret) + goto err_fh; + + ctx->fh.m2m_ctx = v4l2_m2m_ctx_init(ave->m2m_dev, ctx, ave_queue_init); + if (IS_ERR(ctx->fh.m2m_ctx)) { + ret = PTR_ERR(ctx->fh.m2m_ctx); + goto err_ctrls; + } + v4l2_fh_add(&ctx->fh, file); + return 0; + +err_ctrls: + v4l2_ctrl_handler_free(&ctx->hdl); +err_fh: + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + return ret; +} + +static int ave_release(struct file *file) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct ave_dev *ave = ctx->ave; + + mutex_lock(&ave->vdev_lock); + v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); + mutex_unlock(&ave->vdev_lock); + v4l2_ctrl_handler_free(&ctx->hdl); + v4l2_fh_del(&ctx->fh, file); + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + return 0; +} + +static const struct v4l2_file_operations ave_fops = { + .owner = THIS_MODULE, + .open = ave_open, + .release = ave_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +}; + +int ave_v4l2_register(struct ave_dev *ave) +{ + struct video_device *vdev = &ave->vdev; + int ret; + + mutex_init(&ave->vdev_lock); + ave->wq = alloc_ordered_workqueue("apple-ave", WQ_MEM_RECLAIM); + if (!ave->wq) + return -ENOMEM; + + ret = v4l2_device_register(ave->dev, &ave->v4l2_dev); + if (ret) + goto err_wq; + + ave->m2m_dev = v4l2_m2m_init(&ave_m2m_ops); + if (IS_ERR(ave->m2m_dev)) { + ret = PTR_ERR(ave->m2m_dev); + goto err_v4l2; + } + + strscpy(vdev->name, AVE_NAME, sizeof(vdev->name)); + vdev->fops = &ave_fops; + vdev->ioctl_ops = &ave_ioctl_ops; + vdev->release = video_device_release_empty; + vdev->lock = &ave->vdev_lock; + vdev->v4l2_dev = &ave->v4l2_dev; + vdev->vfl_dir = VFL_DIR_M2M; + vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING; + video_set_drvdata(vdev, ave); + + ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1); + if (ret) + goto err_m2m; + return 0; + +err_m2m: + v4l2_m2m_release(ave->m2m_dev); +err_v4l2: + v4l2_device_unregister(&ave->v4l2_dev); +err_wq: + destroy_workqueue(ave->wq); + return ret; +} + +void ave_v4l2_unregister(struct ave_dev *ave) +{ + video_unregister_device(&ave->vdev); + v4l2_m2m_release(ave->m2m_dev); + v4l2_device_unregister(&ave->v4l2_dev); + destroy_workqueue(ave->wq); +} diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h new file mode 100644 index 0000000000000..6421a49e7fdd5 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave.h @@ -0,0 +1,167 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Apple AVE video encoder + * + * Copyright The Aurora Silicon Contributors + */ +#ifndef _APPLE_AVE_H_ +#define _APPLE_AVE_H_ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct dev_pm_domain_list; +struct iommu_domain; + +/* Coprocessor memory every firmware boot uses, all inside one allocation. */ +#define AVE_IPC_BYTES (20 * SZ_1M) +#define AVE_IPC_CHANNEL_OFF 0x4000 +#define AVE_IPC_HANDSHAKE_OFF 0x24000 +#define AVE_IPC_SHARED_OFF 0x28000 +#define AVE_IPC_SHARED_BYTES SZ_64K +#define AVE_IPC_CMD_OFF 0x40000 +#define AVE_IPC_CMD_BYTES SZ_128K +#define AVE_IPC_END (AVE_IPC_CMD_OFF + AVE_IPC_CMD_BYTES) + +#define AVE_MSG_MAX SZ_4K + +enum ave_codec { + AVE_CODEC_H264 = 1, + AVE_CODEC_HEVC = 2, +}; + +struct ave_buf { + void *cpu; + dma_addr_t dma; + size_t size; +}; + +struct ave_fw_segment { + phys_addr_t phys; + u64 iova; + size_t size; +}; + +struct ave_ring { + char name[16]; + u32 mode; + u32 doorbell; + u32 slots; + u32 offset; + int consumer; + int producer; +}; + +enum ave_fw_state { + AVE_FW_OFF, + AVE_FW_RUNNING, + AVE_FW_BROKEN, +}; + +struct ave_dev { + struct device *dev; + void __iomem *codec; + void __iomem *mbox; + void __iomem *coproc; + struct dev_pm_domain_list *pd_list; + struct iommu_domain *domain; + + struct ave_fw_segment text; + struct ave_fw_segment data; + void *data_pristine; + bool data_dirty; + + /* Serialises every firmware and ring access. */ + struct mutex hw_lock; + enum ave_fw_state fw_state; + struct ave_buf ipc; + struct ave_buf boot_cfg; + struct ave_buf log; + struct ave_buf heap; + u64 ipc_fw_base; + u32 channel_bytes; + struct ave_ring ring[2]; + u64 next_token; + u32 next_client; + u8 msg[AVE_MSG_MAX]; + u32 msg_len; + + struct v4l2_device v4l2_dev; + struct video_device vdev; + struct v4l2_m2m_dev *m2m_dev; + struct mutex vdev_lock; + struct workqueue_struct *wq; +}; + +struct ave_ctx { + struct v4l2_fh fh; + struct ave_dev *ave; + struct v4l2_ctrl_handler hdl; + struct v4l2_pix_format_mplane src_fmt; + struct v4l2_pix_format_mplane dst_fmt; + struct v4l2_fract timeperframe; + struct work_struct run_work; + + u32 bitrate_mode; + u32 bitrate; + u32 bitrate_peak; + u32 gop_size; + u32 h264_profile; + u32 h264_level; + u32 h264_i_period; + u32 h264_min_qp; + u32 h264_max_qp; + u32 hevc_profile; + u32 hevc_level; + u32 hevc_min_qp; + u32 hevc_max_qp; + bool force_key_frame; + + u32 client; + bool session_open; + u32 src_sequence; + u32 dst_sequence; +}; + +static inline struct ave_ctx *ave_file_to_ctx(struct file *file) +{ + return container_of(file_to_v4l2_fh(file), struct ave_ctx, fh); +} + +/* ave-fw.c */ +int ave_fw_probe(struct ave_dev *ave); +void ave_fw_remove(struct ave_dev *ave); +int ave_fw_ensure_running(struct ave_dev *ave); +int ave_fw_shutdown(struct ave_dev *ave); +void ave_fw_lost(struct ave_dev *ave); +void ave_fw_resume(struct ave_dev *ave); + +/* ave-ipc.c */ +int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]); +int ave_ipc_call(struct ave_dev *ave, const void *cmd, size_t len); +int ave_ipc_wait_msg(struct ave_dev *ave, unsigned int timeout_ms); + +/* ave-cmd.c */ +int ave_cmd_config(struct ave_dev *ave); +int ave_cmd_halt(struct ave_dev *ave); +int ave_cmd_open(struct ave_ctx *ctx, enum ave_codec codec); +int ave_cmd_start(struct ave_ctx *ctx, enum ave_codec codec); +int ave_cmd_process(struct ave_ctx *ctx, enum ave_codec codec, + struct vb2_v4l2_buffer *src, struct vb2_v4l2_buffer *dst, + u32 *bytesused, bool *keyframe); +int ave_cmd_finish(struct ave_ctx *ctx, enum ave_codec codec); + +/* ave-v4l2.c */ +int ave_v4l2_register(struct ave_dev *ave); +void ave_v4l2_unregister(struct ave_dev *ave); +enum ave_codec ave_ctx_codec(struct ave_ctx *ctx); + +#endif From 0210e6b1b935bc8656e45a0d9cb6c69f69130427 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 16:17:49 +1100 Subject: [PATCH 02/20] arm64: dts: apple: t8140-j700: enable the video encoder aef49ec18ecf ("arm64: dts: apple: t8140-j700: leave the video encoder disabled") turned the AVE encoder, its two DARTs and the six venc0 power domains off because no driver bound apple,t8140-ave and the domains sat in sync_state forever. The apple-ave driver now binds the node, so enable them again. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- arch/arm64/boot/dts/apple/t8140-j700.dts | 12 ++++++++++++ arch/arm64/boot/dts/apple/t8140-pmgr.dtsi | 6 ------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/arm64/boot/dts/apple/t8140-j700.dts b/arch/arm64/boot/dts/apple/t8140-j700.dts index ece0d9c6bfefb..61427cfe7538f 100644 --- a/arch/arm64/boot/dts/apple/t8140-j700.dts +++ b/arch/arm64/boot/dts/apple/t8140-j700.dts @@ -388,6 +388,18 @@ #include "isp-imx558.dtsi" +&ave_dart0 { + status = "okay"; +}; + +&ave_dart1 { + status = "okay"; +}; + +&ave { + status = "okay"; +}; + &serial0 { status = "okay"; }; diff --git a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi index 8a1864e0fabbd..0a479f7e8e8e5 100644 --- a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi +++ b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi @@ -69,7 +69,6 @@ #power-domain-cells = <0>; #reset-cells = <0>; label = "venc0_sys"; - status = "disabled"; }; ps_apcie_gp: power-controller@258 { @@ -408,7 +407,6 @@ #reset-cells = <0>; label = "venc0_dma"; power-domains = <&ps_venc0_sys>; - status = "disabled"; }; ps_venc0_pipe4: power-controller@8008 { @@ -418,7 +416,6 @@ #reset-cells = <0>; label = "venc0_pipe4"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_pipe5: power-controller@8010 { @@ -428,7 +425,6 @@ #reset-cells = <0>; label = "venc0_pipe5"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_me0: power-controller@8018 { @@ -438,7 +434,6 @@ #reset-cells = <0>; label = "venc0_me0"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_me1: power-controller@8020 { @@ -448,7 +443,6 @@ #reset-cells = <0>; label = "venc0_me1"; power-domains = <&ps_venc0_me0>; - status = "disabled"; }; }; From 91c3b9d88c78373eb143b93c613ef38dcddb5805 Mon Sep 17 00:00:00 2001 From: Ryan <81541646+RyanTheTide@users.noreply.github.com> Date: Fri, 30 Oct 2026 16:31:40 +1100 Subject: [PATCH 03/20] arm64: asahi.config: build the Apple video encoder as a module The J700 video encoder (AVE) gets a V4L2 stateful encoder driver in this series. Build it as a module in the universal config, next to the decoder (AVD) and ISP, so the device-tree enable does not leave the encoder and its power domains waiting on a driver that is never built. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- arch/arm64/configs/asahi.config | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/asahi.config b/arch/arm64/configs/asahi.config index fcf0d27611aae..8d9f76581d226 100644 --- a/arch/arm64/configs/asahi.config +++ b/arch/arm64/configs/asahi.config @@ -34,6 +34,7 @@ CONFIG_APPLE_WATCHDOG=y CONFIG_MTD=y CONFIG_MTD_PHRAM=m CONFIG_VIDEO_APPLE_AVD=m +CONFIG_VIDEO_APPLE_AVE=m CONFIG_VIDEO_APPLE_ISP=m CONFIG_DRM=y CONFIG_DRM_ASAHI=m From 42660490d8fe92baae3665c6d4c12f570b233c4d Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 18:33:19 +1100 Subject: [PATCH 04/20] media: apple: ave: start sessions, encode frames and halt the firmware Fill in the parts of the J700 encoder driver that were left returning -EOPNOTSUPP, so a V4L2 client gets H.264 or HEVC out of NV12 frames. Session start builds the Start command from the encoding parameters: picture and coded size, cropping, profile, level (raised to what size, frame rate and bit rate need), VBR with a one-second data rate cap or CBR, QP range, GOP length, the B-frame structure and frame rate. The driver allocates the encoder working buffers with sizes derived from the picture size and lists them in the command. Frames go out as Process commands over a ring of five slots (three without B-frames). The firmware reorders B-frames itself and answers in bitstream order, so CAPTURE buffers come back in that order with the timestamp of the frame they code. The first frame is sent alone and followed by a Complete, as macOS does. Process carries the input planes, frame number, presentation time, the current bit rate (it may change while streaming) and the forced key frame request. HEVC output is assembled from the parameter sets, the slice header and the slice data the firmware leaves in three buffers. For H.264 the length is not known yet, so the coded buffer is read as Annex B up to where the data ends. Odd output is reported once per session with a dump under dynamic debug. Commands no longer wait for their answer one at a time. Each command buffer is tracked until the firmware acknowledges it, answers are matched by command id, and frame answers by session and slot. A short poller collects frame answers while frames are outstanding, since the firmware interrupt is not wired up. Drain sends four Completes, then Priority, Stop and Close, and the firmware is halted after every session. Halt is never answered; after a short wait the coprocessor counts as stopped, runtime suspend may drop the power domains, and the next session restarts the firmware through the boot path from the saved DATA copy. A firmware that stops answering keeps its power and its buffers, and is not used again until reboot. The constant words of Start and Process are not built into the driver. It loads them per firmware build from apple/ave-t8140--{h264,hevc}-{start,process}.bin and overwrites every session and frame field. Without those files STREAMON fails with ENOENT. Field layouts are facts from our private research (macOS traces); no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- Documentation/admin-guide/media/apple-ave.rst | 60 +- drivers/media/platform/apple/ave/Kconfig | 5 +- drivers/media/platform/apple/ave/Makefile | 2 +- drivers/media/platform/apple/ave/ave-cmd.c | 622 ++++++++++++++++-- drivers/media/platform/apple/ave/ave-drv.c | 51 +- drivers/media/platform/apple/ave/ave-fw.c | 180 ++++- drivers/media/platform/apple/ave/ave-ipc.c | 307 ++++++--- drivers/media/platform/apple/ave/ave-regs.h | 29 +- drivers/media/platform/apple/ave/ave-sess.c | 493 ++++++++++++++ drivers/media/platform/apple/ave/ave-v4l2.c | 519 ++++++++++++--- drivers/media/platform/apple/ave/ave.h | 161 ++++- 11 files changed, 2093 insertions(+), 336 deletions(-) create mode 100644 drivers/media/platform/apple/ave/ave-sess.c diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index dabb4250a72a1..d8f9acabb2117 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -8,7 +8,7 @@ encodes H.264 and HEVC. ``CONFIG_VIDEO_APPLE_AVE`` builds the ``apple-ave`` driver, which registers a V4L2 memory-to-memory stateful encoder following Documentation/userspace-api/media/v4l/dev-encoder.rst. The OUTPUT queue takes NV12 frames (64-byte aligned line stride), the CAPTURE queue returns -H.264 or HEVC. +H.264 (Baseline, Main, High) or HEVC (Main) as Annex B. Firmware -------- @@ -17,27 +17,55 @@ The encoder runs firmware on its own coprocessor. The bootloader loads it into two reserved, DART-mapped segments (TEXT and DATA) but does not start it. The Aurora Stage 2 loader passes both regions in the device's ``memory-region`` property and records their CRC32 under ``/chosen`` as -``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. No firmware file is -requested. +``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. The driver only binds to firmware builds whose command formats it knows, identified by the TEXT checksum, and only when DATA still matches the checksum the loader recorded. It keeps a copy of DATA and restores it before -every firmware boot, so the coprocessor can lose power and start again from -the same state. +every firmware boot, so the firmware can be started again from the same state. -Power management ----------------- +Session images +-------------- -The firmware boots when the first encoding session starts. Runtime PM uses a -two second autosuspend delay. While the firmware runs the encoder stays -powered; across system sleep the power domains go down, open sessions fail -their next frame, and the next session boots the firmware again. +The two large session commands, Start and Process, are mostly constant for a +given firmware build. The driver writes every field that depends on the +session or the frame (sizes, rate control, GOP structure, levels, cropping, +buffer addresses, frame numbers and times) and takes the remaining constant +words from four files per firmware build:: -Status + apple/ave-t8140--h264-start.bin (0x3180 bytes) + apple/ave-t8140--h264-process.bin (0x63d8 bytes) + apple/ave-t8140--hevc-start.bin (0x13f28 bytes) + apple/ave-t8140--hevc-process.bin (0xb1c0 bytes) + +Each file is a complete command image. Address and size fields in it are +ignored. Without these files, ``VIDIOC_STREAMON`` fails with ``ENOENT``. + +Sessions and power management +----------------------------- + +Each encoding session boots the firmware, opens and starts a session, encodes +and, at the end, drains, stops and closes the session and halts the firmware. +The next session restarts the firmware through the boot path. One session +runs at a time; a second one gets ``EBUSY``. + +Up to five frames are inside the firmware at once (three without B-frames). +The firmware reorders B-frames itself, so CAPTURE buffers come back in +bitstream order, each carrying the timestamp of the frame it codes. The driver +asks for at least six OUTPUT buffers. + +Runtime PM uses a two second autosuspend delay. A firmware that stops +answering keeps the encoder powered and is not used again until reboot. +Across system sleep an open session ends; the next one starts afresh. + +Limits ------ -Firmware start-up, the command rings, session open/close and the V4L2 -interface are implemented. Session start and per-frame encoding are not yet -wired up, so ``VIDIOC_STREAMON`` on both queues currently fails with -``EOPNOTSUPP``. +Picture sizes from 320x240 to 2560x1600 (in steps of two), up to 240 frames +per second. Rate control is VBR (with ``BITRATE_PEAK`` as a one-second data +rate cap) or CBR; constant quality is not offered. ``BITRATE`` may change +while streaming and applies from the next frame; ``FORCE_KEY_FRAME`` makes +the next frame an IDR. H.264 uses either three B-frames or none (Baseline and +a GOP size of 1 never use them); HEVC always uses them for now. The level +control is a minimum: the driver raises it to what the size, frame rate and +bit rate need. diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig index 59937167b29ce..2b3f380abf752 100644 --- a/drivers/media/platform/apple/ave/Kconfig +++ b/drivers/media/platform/apple/ave/Kconfig @@ -5,12 +5,15 @@ config VIDEO_APPLE_AVE depends on ARCH_APPLE || COMPILE_TEST depends on VIDEO_DEV && IOMMU_DMA && PM_GENERIC_DOMAINS select CRC32 + select FW_LOADER select V4L2_MEM2MEM_DEV select VIDEOBUF2_DMA_CONTIG help V4L2 memory-to-memory driver for the AVE H.264/HEVC encoder in Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. The encoder - firmware is loaded by the bootloader. + firmware is loaded by the bootloader; the constant parts of its + session commands come from files under apple/ in the firmware + search path. To compile this driver as a module, choose M here: the module will be called apple-ave. diff --git a/drivers/media/platform/apple/ave/Makefile b/drivers/media/platform/apple/ave/Makefile index 8e0a5519a520b..542a7adeae360 100644 --- a/drivers/media/platform/apple/ave/Makefile +++ b/drivers/media/platform/apple/ave/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_VIDEO_APPLE_AVE) += apple-ave.o -apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-v4l2.o +apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-sess.o ave-v4l2.o diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index 6948bc60496a6..0b539fc201c3b 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -2,107 +2,621 @@ /* * Apple AVE video encoder: firmware commands * + * Start and Process are large command images. Most of their words never + * change between sessions of a firmware build; the driver takes those from + * a per-build image file (see ave_fw_templates()) and writes every field + * that depends on the session or the frame itself. + * * Copyright The Aurora Silicon Contributors */ #include +#include +#include #include #include #include +#include #include #include "ave.h" #include "ave-regs.h" -static void ave_cmd_header(struct ave_dev *ave, u8 *cmd, enum ave_opcode op, - u32 client, enum ave_codec codec) +/* + * IEEE 754 double for the positive value @num / @den (0 for 0), built with + * integer arithmetic; the firmware takes rates and limits as doubles. + */ +static u64 ave_f64(u64 num, u64 den) +{ + u64 mant = 0, rem; + int exp = 0; + + if (!num || !den) + return 0; + while (num >= den << 1) { + den <<= 1; + exp++; + } + while (num < den) { + num <<= 1; + exp--; + } + /* 1 <= num/den < 2: take 53 significant bits by long division. */ + rem = num; + for (unsigned int i = 0; i < 53; i++) { + mant <<= 1; + if (rem >= den) { + mant |= 1; + rem -= den; + } + rem <<= 1; + } + return (u64)(exp + 1023) << 52 | (mant & GENMASK_ULL(51, 0)); +} + +#define AVE_F64_NEG BIT_ULL(63) + +static void ave_put32(u8 *cmd, u32 off, u32 val) +{ + put_unaligned_le32(val, cmd + off); +} + +static void ave_put64(u8 *cmd, u32 off, u64 val) +{ + put_unaligned_le64(val, cmd + off); +} + +/* A surface descriptor: {u64 DART address, u32 size, u32 0}. */ +static void ave_put_surface(u8 *cmd, u32 off, dma_addr_t addr, u32 size) +{ + ave_put64(cmd, off, addr); + ave_put32(cmd, off + 8, size); + ave_put32(cmd, off + 12, 0); +} + +static void ave_put_buf(u8 *cmd, u32 off, const struct ave_buf *b, u32 size) +{ + ave_put_surface(cmd, off, b->dma, size); +} + +static u32 ave_session_arg2(enum ave_codec codec) { - put_unaligned_le32(op, cmd + AVE_CMD_OPCODE); - put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); - put_unaligned_le64(client, cmd + AVE_CMD_CLIENT); - put_unaligned_le32(1, cmd + AVE_CMD_UNIT); - put_unaligned_le32(codec, cmd + AVE_CMD_CODEC); - put_unaligned_le32(op, cmd + AVE_CMD_OPCODE_ECHO); + return codec == AVE_CODEC_HEVC ? AVE_SESSION_ARG2_HEVC : AVE_SESSION_ARG2_H264; +} + +static void ave_cmd_header(struct ave_ctx *ctx, u8 *cmd, enum ave_opcode op, u32 echo, + u32 arg0, u32 arg2) +{ + struct ave_dev *ave = ctx->ave; + + ave->next_token += AVE_TOKEN_STEP; + ave_put32(cmd, AVE_CMD_OPCODE, op); + ave_put64(cmd, AVE_CMD_TOKEN, ave->next_token); + ave_put64(cmd, AVE_CMD_CLIENT, ctx->sess->client); + ave_put32(cmd, AVE_CMD_UNIT, AVE_SESSION_UNIT); + ave_put32(cmd, AVE_CMD_CODEC, ctx->sess->p.codec); + ave_put32(cmd, AVE_CMD_OPCODE_ECHO, echo); + ave_put32(cmd, AVE_CMD_ARG0, arg0); + ave_put32(cmd, AVE_CMD_ARG1, AVE_SESSION_ARG1); + ave_put32(cmd, AVE_CMD_ARG2, arg2); } /* Firmware-wide configuration, sent once after every firmware boot. */ int ave_cmd_config(struct ave_dev *ave) { - u8 cmd[AVE_CONFIG_BYTES] = {}; + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; + int ret; - put_unaligned_le32(AVE_OP_CONFIG, cmd + AVE_CMD_OPCODE); - put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); - put_unaligned_le32(U32_MAX, cmd + AVE_CONFIG_ALL_UNITS); + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CONFIG_BYTES); + ave_put32(cmd, AVE_CMD_OPCODE, AVE_OP_CONFIG); + ave_put32(cmd, AVE_CONFIG_ALL_UNITS, U32_MAX); cmd[AVE_CONFIG_OPT_A] = 1; cmd[AVE_CONFIG_OPT_B] = 1; - return ave_ipc_call(ave, cmd, sizeof(cmd)); + ave_put32(cmd, AVE_CONFIG_MCC_DSID, AVE_CONFIG_MCC_DSID_T8140); + ret = ave_ipc_submit(ave, buf, AVE_CONFIG_BYTES, AVE_CMD_GLOBAL_TIMEOUT_US, true); + return ret ?: ave_ipc_settle(ave, AVE_CMD_WAIT_MS); } -/* Firmware-wide stop, sent after every session has closed. */ +/* + * Firmware-wide stop, sent after every session has closed. The firmware + * neither acknowledges nor answers it. + */ int ave_cmd_halt(struct ave_dev *ave) { - u8 cmd[AVE_CMD_HDR_BYTES] = {}; + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; - put_unaligned_le32(AVE_OP_HALT, cmd + AVE_CMD_OPCODE); - put_unaligned_le64(++ave->next_token, cmd + AVE_CMD_TOKEN); - put_unaligned_le32(AVE_HALT_ALL, cmd + AVE_CMD_OPCODE_ECHO); - return ave_ipc_call(ave, cmd, sizeof(cmd)); + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CMD_HDR_BYTES); + ave_put32(cmd, AVE_CMD_OPCODE, AVE_OP_HALT); + ave_put32(cmd, AVE_CMD_OPCODE_ECHO, AVE_HALT_ALL); + return ave_ipc_submit(ave, buf, AVE_CMD_HDR_BYTES, AVE_CMD_GLOBAL_TIMEOUT_US, false); } -static int ave_cmd_simple(struct ave_ctx *ctx, enum ave_opcode op, enum ave_codec codec, - u32 arg0, u32 arg2) +static int ave_cmd_scalar(struct ave_ctx *ctx, enum ave_opcode op, u32 arg0, u32 arg2) { - u8 cmd[AVE_CMD_HDR_BYTES] = {}; + struct ave_dev *ave = ctx->ave; + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; - ave_cmd_header(ctx->ave, cmd, op, ctx->client, codec); - put_unaligned_le32(arg0, cmd + AVE_CMD_ARG0); - put_unaligned_le32(AVE_SESSION_ARG1, cmd + AVE_CMD_ARG1); - put_unaligned_le32(arg2, cmd + AVE_CMD_ARG2); - return ave_ipc_call(ctx->ave, cmd, sizeof(cmd)); + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CMD_HDR_BYTES); + ave_cmd_header(ctx, cmd, op, op, arg0, arg2); + return ave_ipc_submit(ave, buf, AVE_CMD_HDR_BYTES, AVE_CMD_SESSION_TIMEOUT_US, true); } -int ave_cmd_open(struct ave_ctx *ctx, enum ave_codec codec) +int ave_cmd_open(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; int ret; - ctx->client = ++ave->next_client; - ret = ave_cmd_simple(ctx, AVE_OP_OPEN, codec, AVE_SESSION_ARG0, AVE_OPEN_ARG2); + ave->next_client += AVE_CLIENT_STEP; + ctx->sess->client = ave->next_client; + ret = ave_cmd_scalar(ctx, AVE_OP_OPEN, AVE_SESSION_ARG0, AVE_SESSION_ARG2_SETUP); if (!ret) - ctx->session_open = true; + ctx->sess->fw_open = true; + return ret ?: ave_ipc_settle(ave, AVE_CMD_WAIT_MS); +} + +/* Drain barrier: answered once every Process sent before it is answered. */ +int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count) +{ + int ret = 0; + + for (unsigned int i = 0; i < count && !ret; i++) + ret = ave_cmd_scalar(ctx, AVE_OP_COMPLETE, 0, ave_session_arg2(ctx->sess->p.codec)); return ret; } -/* - * Session parameters. The Start layout (resolution, formats, rate control, - * GOP, the per-session buffer table) is not described yet, so sessions cannot - * start; everything around it is in place. - */ -int ave_cmd_start(struct ave_ctx *ctx, enum ave_codec codec) +/* Priority and Stop go out together, Close after Stop is answered. */ +int ave_cmd_close(struct ave_ctx *ctx) +{ + u32 arg2 = ave_session_arg2(ctx->sess->p.codec); + int ret, err; + + ret = ave_cmd_scalar(ctx, AVE_OP_PRIORITY, AVE_SESSION_ARG0, arg2); + if (!ret) + ret = ave_cmd_scalar(ctx, AVE_OP_STOP, AVE_SESSION_ARG0, arg2); + err = ave_ipc_settle(ctx->ave, AVE_CMD_WAIT_MS); + ret = ret ?: err; + if (ctx->ave->fw_state != AVE_FW_RUNNING) + return ret; + err = ave_cmd_scalar(ctx, AVE_OP_CLOSE, AVE_SESSION_ARG0, arg2); + if (!err) + err = ave_ipc_settle(ctx->ave, AVE_CMD_WAIT_MS); + ctx->sess->fw_open = false; + return ret ?: err; +} + +/* Start (id 6) */ + +#define AVE_START_H264_BYTES 0x3180 +#define AVE_START_HEVC_BYTES 0x13f28 + +#define AVE_START_SURFACE 0x48 /* and an unused second one at 0x58 */ +#define AVE_START_SURFACE_BYTES 0x12c000 +#define AVE_START_SURFACE2 0x58 +#define AVE_START_REF_A(i) (0x390 + 0x20 * (i)) +#define AVE_START_REF_B(i) (0x3a0 + 0x20 * (i)) +#define AVE_START_REF_C(i) (0x7d0 + 0x20 * (i)) +#define AVE_START_REF_C_ALIAS(i) (0x7e0 + 0x20 * (i)) +#define AVE_START_MB 0xc10 +#define AVE_START_CODED(s) (0xd50 + 0x10 * (s)) +#define AVE_START_META(s) (0xf50 + 0x10 * (s)) +#define AVE_START_ARRAY_LEN 30 +#define AVE_START_REF_D(i) (0x1130 + 0x10 * (i)) +#define AVE_START_ROW_A 0x1350 +#define AVE_START_ROW_B 0x1390 +#define AVE_START_FIXED 0x13d0 +#define AVE_START_FIXED_BYTES 0x4000 +#define AVE_START_HEVC_EXTRA(i) (0x2520 + 0x10 * (i)) + +static const u16 ave_start_tile_off[AVE_TILE_BUFS] = { + 0x1410, 0x1430, 0x1490, 0x14b0, /* both codecs */ + 0x1510, 0x1530, 0x1590, 0x15b0, /* H.264 only */ +}; + +/* Session configuration words */ +#define AVE_START_CODEC 0xc0 +#define AVE_START_8BIT 0x208 +#define AVE_START_CONST_214 0x214 +#define AVE_START_CONST_218 0x218 +#define AVE_START_FPS 0x220 +#define AVE_START_RC_MODE 0x234 +#define AVE_START_BITRATE 0x238 +#define AVE_START_INIT_QP(i) (0x240 + 4 * (i)) +#define AVE_START_INIT_QP_VALUE 26 +#define AVE_START_LIMITS 0x250 +#define AVE_START_LIMIT_BYTES 0x258 +#define AVE_START_LIMIT_SECS 0x260 +#define AVE_START_F64_288 0x288 +#define AVE_START_F64_290 0x290 +#define AVE_START_MIN_QP 0x298 +#define AVE_START_MAX_QP 0x29c +#define AVE_START_GOP_FLAGS 0x2a0 +#define AVE_START_P_FRAMES 0x2ac +#define AVE_START_MAX_B 0x2b0 +#define AVE_START_REF_DEPTH 0x2b4 +#define AVE_START_IDR_INTERVAL 0x2b8 +#define AVE_START_F64_2C0 0x2c0 +#define AVE_START_F64_2C8 0x2c8 +#define AVE_START_RC_FLAGS 0x300 +#define AVE_START_MIN_QP2 0x308 +#define AVE_START_MAX_QP2 0x30c +#define AVE_START_RC_310 0x310 +#define AVE_START_SIZE_CLASS 0x320 +#define AVE_START_CODED_W 0x368 +#define AVE_START_CODED_H 0x36c +#define AVE_START_CODEC_FLAGS 0x370 +#define AVE_START_H264_380 0x380 +#define AVE_START_H264_384 0x384 +#define AVE_START_H264_HIGH 0x2588 +#define AVE_START_BIT_DEPTH 0x2598 +#define AVE_START_CODED_H2 0x25dc + +/* H.264 parameter-set words */ +#define AVE_START_H264_PROFILE 0x291c +#define AVE_START_H264_LEVEL 0x2938 +#define AVE_START_H264_MAX_REFS 0x2d44 +#define AVE_START_H264_WIDTH_MBS 0x2d4c +#define AVE_START_H264_HEIGHT_MBS 0x2d50 +#define AVE_START_H264_CROP 0x2db4 +#define AVE_START_H264_CROP_LEFT 0x2db8 +#define AVE_START_H264_CROP_RIGHT 0x2dbc +#define AVE_START_H264_CROP_TOP 0x2dc0 +#define AVE_START_H264_CROP_BOTTOM 0x2dc4 +#define AVE_START_H264_CABAC 0x2fd8 +#define AVE_START_H264_8X8 0x3023 /* byte */ + +/* HEVC parameter-set words; most appear twice */ +#define AVE_START_HEVC_PAD 0x26e0 +#define AVE_START_HEVC_PROFILE_A 0x2938 +#define AVE_START_HEVC_COMPAT_A 0x293d /* two bytes */ +#define AVE_START_HEVC_8BIT_A 0x2962 /* byte */ +#define AVE_START_HEVC_LEVEL_A 0x296c +#define AVE_START_HEVC_PROFILE_B 0x5904 +#define AVE_START_HEVC_COMPAT_B 0x5909 +#define AVE_START_HEVC_8BIT_B 0x592e +#define AVE_START_HEVC_LEVEL_B 0x5938 +#define AVE_START_HEVC_WIDTH 0x5ab4 +#define AVE_START_HEVC_HEIGHT 0x5ab8 +#define AVE_START_HEVC_WIDTH_CTB 0x5abc +#define AVE_START_HEVC_HEIGHT_CTB 0x5ac0 +#define AVE_START_HEVC_CONF 0x5ac4 +#define AVE_START_HEVC_CONF_LEFT 0x5ac8 +#define AVE_START_HEVC_CONF_RIGHT 0x5acc +#define AVE_START_HEVC_CONF_TOP 0x5ad0 +#define AVE_START_HEVC_CONF_BOTTOM 0x5ad4 +#define AVE_START_HEVC_DEPTH_LUMA 0x5ad8 +#define AVE_START_HEVC_DEPTH_CHROMA 0x5adc +/* HEVC reference structure with B-frames (reordering on) */ +#define AVE_START_HEVC_REFS 0x259c +#define AVE_START_HEVC_REORDER_COPY 0x2e54 +#define AVE_START_HEVC_DPB_MINUS1 0x5b00 +#define AVE_START_HEVC_REORDER 0x5b04 +#define AVE_START_HEVC_REORDER_ON 0x95fc + +static void ave_start_surfaces(struct ave_session *sess, u8 *cmd) { - return -EOPNOTSUPP; + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + unsigned int tiles = hevc ? 4 : AVE_TILE_BUFS; + + /* Clear every descriptor the image may carry, then fill ours. */ + for (unsigned int i = 0; i < AVE_MAX_REFS; i++) { + ave_put_surface(cmd, AVE_START_REF_A(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_B(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_C(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_C_ALIAS(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_D(i), 0, 0); + } + for (unsigned int s = 0; s < AVE_START_ARRAY_LEN; s++) { + ave_put_surface(cmd, AVE_START_CODED(s), 0, 0); + ave_put_surface(cmd, AVE_START_META(s), 0, 0); + } + + ave_put_buf(cmd, AVE_START_SURFACE, &sess->hdr, AVE_START_SURFACE_BYTES); + ave_put_surface(cmd, AVE_START_SURFACE2, 0, 0); + for (unsigned int i = 0; i < p->refs; i++) { + ave_put_buf(cmd, AVE_START_REF_A(i), &sess->ref_a[i], sess->ref_a[i].size); + ave_put_buf(cmd, AVE_START_REF_B(i), &sess->ref_b[i], sess->ref_b[i].size); + ave_put_buf(cmd, AVE_START_REF_C(i), &sess->ref_c[i], sess->ref_c[i].size); + ave_put_surface(cmd, AVE_START_REF_C_ALIAS(i), sess->ref_c[i].dma, 0); + ave_put_buf(cmd, AVE_START_REF_D(i), &sess->ref_d[i], sess->ref_d[i].size); + } + ave_put_buf(cmd, AVE_START_MB, &sess->mb, sess->mb.size); + for (unsigned int s = 0; s < p->slots; s++) { + ave_put_buf(cmd, AVE_START_CODED(s), &sess->slot[s].coded, p->coded_cap); + ave_put_buf(cmd, AVE_START_META(s), &sess->slot[s].meta, sess->slot[s].meta.size); + } + ave_put_buf(cmd, AVE_START_ROW_A, &sess->row_a, sess->row_a.size); + ave_put_buf(cmd, AVE_START_ROW_B, &sess->row_b, sess->row_b.size); + ave_put_buf(cmd, AVE_START_FIXED, &sess->fixed, AVE_START_FIXED_BYTES); + for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) { + u32 off = ave_start_tile_off[i]; + + if (i < tiles) { + ave_put_buf(cmd, off, &sess->tile[i], sess->tile[i].size); + ave_put_buf(cmd, off + 0x10, &sess->tile_aux[i], sess->tile_aux[i].size); + } else { + ave_put_surface(cmd, off, 0, 0); + ave_put_surface(cmd, off + 0x10, 0, 0); + } + } + if (hevc) + for (unsigned int i = 0; i < 2; i++) + ave_put_buf(cmd, AVE_START_HEVC_EXTRA(i), &sess->hevc_extra[i], + sess->hevc_extra[i].size); } -int ave_cmd_process(struct ave_ctx *ctx, enum ave_codec codec, - struct vb2_v4l2_buffer *src, struct vb2_v4l2_buffer *dst, - u32 *bytesused, bool *keyframe) +static void ave_start_config(const struct ave_params *p, u8 *cmd) { - return -EOPNOTSUPP; + bool hevc = p->codec == AVE_CODEC_HEVC; + u32 crop_r = (p->cw - p->width) / 2, crop_b = (p->ch - p->height) / 2; + bool crop = crop_r || crop_b; + + ave_put32(cmd, AVE_START_CODEC, p->codec); + ave_put32(cmd, AVE_START_8BIT, hevc ? 1 : 0); + ave_put32(cmd, AVE_START_CONST_214, 0x39d); + ave_put32(cmd, AVE_START_CONST_218, 0x18); + ave_put32(cmd, AVE_START_FPS, p->fps_int); + + /* Rate control */ + ave_put32(cmd, AVE_START_RC_MODE, p->rc_mode); + ave_put32(cmd, AVE_START_BITRATE, p->bitrate); + for (unsigned int i = 0; i < 3; i++) + ave_put32(cmd, AVE_START_INIT_QP(i), AVE_START_INIT_QP_VALUE); + ave_put32(cmd, AVE_START_LIMITS, p->limit_bytes ? 1 : 0); + if (p->limit_bytes) { + ave_put64(cmd, AVE_START_LIMIT_BYTES, ave_f64(p->limit_bytes, 1)); + ave_put64(cmd, AVE_START_LIMIT_SECS, ave_f64(1, 1)); + } + ave_put64(cmd, AVE_START_F64_288, ave_f64(1, 1)); + ave_put64(cmd, AVE_START_F64_290, ave_f64(1, 2)); + ave_put64(cmd, AVE_START_F64_2C0, AVE_F64_NEG | ave_f64(1, 1)); + ave_put64(cmd, AVE_START_F64_2C8, AVE_F64_NEG | ave_f64(1, 1)); + ave_put32(cmd, AVE_START_MIN_QP, p->min_qp); + ave_put32(cmd, AVE_START_MAX_QP, p->max_qp); + ave_put32(cmd, AVE_START_MIN_QP2, p->min_qp); + ave_put32(cmd, AVE_START_MAX_QP2, p->max_qp); + ave_put32(cmd, AVE_START_RC_FLAGS, hevc ? 0x736 : 0x8202); + ave_put32(cmd, AVE_START_RC_310, 30); + if (!hevc) + ave_put32(cmd, AVE_START_H264_380, 1); + + /* GOP structure: with B-frames as captured, or P-only, or intra-only. */ + ave_put32(cmd, AVE_START_GOP_FLAGS, p->bframes ? 0x600 : 0x200); + ave_put32(cmd, AVE_START_P_FRAMES, p->intra_only ? 0 : 1); + ave_put32(cmd, AVE_START_MAX_B, p->bframes ? 3 : 0); + ave_put32(cmd, AVE_START_REF_DEPTH, p->bframes ? 3 : 1); + ave_put32(cmd, AVE_START_IDR_INTERVAL, p->gop); + + /* Picture size */ + ave_put32(cmd, AVE_START_SIZE_CLASS, p->width * p->height > 1280 * 720 ? 3 : 1); + ave_put32(cmd, AVE_START_CODED_W, p->cw); + ave_put32(cmd, AVE_START_CODED_H, p->ch); + ave_put32(cmd, AVE_START_CODED_H2, p->ch); + ave_put32(cmd, AVE_START_CODEC_FLAGS, hevc ? 0x2000002 : 2); + ave_put32(cmd, AVE_START_BIT_DEPTH, 8); + + if (!hevc) { + ave_put32(cmd, AVE_START_H264_384, p->width * p->height <= 320 * 240 ? 0x40 : 0x10); + ave_put32(cmd, AVE_START_H264_HIGH, p->high ? 2 : 0); + ave_put32(cmd, AVE_START_H264_PROFILE, p->profile); + ave_put32(cmd, AVE_START_H264_LEVEL, p->level); + ave_put32(cmd, AVE_START_H264_MAX_REFS, p->refs - 1); + ave_put32(cmd, AVE_START_H264_WIDTH_MBS, p->cw / 16 - 1); + ave_put32(cmd, AVE_START_H264_HEIGHT_MBS, p->ch / 16 - 1); + ave_put32(cmd, AVE_START_H264_CROP, crop); + ave_put32(cmd, AVE_START_H264_CROP_LEFT, 0); + ave_put32(cmd, AVE_START_H264_CROP_RIGHT, crop_r); + ave_put32(cmd, AVE_START_H264_CROP_TOP, 0); + ave_put32(cmd, AVE_START_H264_CROP_BOTTOM, crop_b); + ave_put32(cmd, AVE_START_H264_CABAC, p->profile != 2); + cmd[AVE_START_H264_8X8] = p->high; + return; + } + + ave_put32(cmd, AVE_START_HEVC_PAD, 1 | + (p->cw > p->width ? BIT(16) | BIT(18) : 0) | + (p->ch > p->height ? BIT(17) | BIT(19) : 0)); + ave_put32(cmd, AVE_START_HEVC_PROFILE_A, p->profile); + ave_put32(cmd, AVE_START_HEVC_PROFILE_B, p->profile); + cmd[AVE_START_HEVC_COMPAT_A] = 1; + cmd[AVE_START_HEVC_COMPAT_A + 1] = 1; + cmd[AVE_START_HEVC_COMPAT_B] = 1; + cmd[AVE_START_HEVC_COMPAT_B + 1] = 1; + cmd[AVE_START_HEVC_8BIT_A] = 1; + cmd[AVE_START_HEVC_8BIT_B] = 1; + ave_put32(cmd, AVE_START_HEVC_LEVEL_A, p->level); + ave_put32(cmd, AVE_START_HEVC_LEVEL_B, p->level); + ave_put32(cmd, AVE_START_HEVC_WIDTH, p->cw); + ave_put32(cmd, AVE_START_HEVC_HEIGHT, p->ch); + ave_put32(cmd, AVE_START_HEVC_WIDTH_CTB, DIV_ROUND_UP(p->cw, 32)); + ave_put32(cmd, AVE_START_HEVC_HEIGHT_CTB, DIV_ROUND_UP(p->ch, 32)); + ave_put32(cmd, AVE_START_HEVC_CONF, crop); + ave_put32(cmd, AVE_START_HEVC_CONF_LEFT, 0); + ave_put32(cmd, AVE_START_HEVC_CONF_RIGHT, crop_r); + ave_put32(cmd, AVE_START_HEVC_CONF_TOP, 0); + ave_put32(cmd, AVE_START_HEVC_CONF_BOTTOM, crop_b); + ave_put32(cmd, AVE_START_HEVC_DEPTH_LUMA, 0); + ave_put32(cmd, AVE_START_HEVC_DEPTH_CHROMA, 0); + /* Only the B-frame structure is described for HEVC (see ave-sess.c). */ + ave_put32(cmd, AVE_START_HEVC_REFS, 4); + ave_put32(cmd, AVE_START_HEVC_DPB_MINUS1, 4); + ave_put32(cmd, AVE_START_HEVC_REORDER, 2); + ave_put32(cmd, AVE_START_HEVC_REORDER_COPY, 2); + ave_put32(cmd, AVE_START_HEVC_REORDER_ON, 1); } -/* Flush, stop and close the session. */ -int ave_cmd_finish(struct ave_ctx *ctx, enum ave_codec codec) +int ave_cmd_start(struct ave_ctx *ctx) { - u32 tail = codec == AVE_CODEC_HEVC ? AVE_FINISH_ARG2_HEVC : AVE_FINISH_ARG2_H264; - int ret, err; + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + const struct firmware *tmpl = ave->tmpl[sess->p.codec - 1][AVE_TMPL_START]; + u8 *cmd; + int ret; - if (!ctx->session_open) - return 0; + cmd = ave_ipc_cmd(ave, AVE_BUF_CMD, tmpl->size); + if (!cmd) + return -EBUSY; + memcpy(cmd, tmpl->data, tmpl->size); + ave_cmd_header(ctx, cmd, AVE_OP_START, AVE_OP_START, AVE_SESSION_ARG0, + AVE_SESSION_ARG2_SETUP); + ave_start_surfaces(sess, cmd); + ave_start_config(&sess->p, cmd); - ret = ave_cmd_simple(ctx, AVE_OP_COMPLETE, codec, 0, tail); - err = ave_cmd_simple(ctx, AVE_OP_STOP, codec, AVE_SESSION_ARG0, tail); - ret = ret ?: err; - err = ave_cmd_simple(ctx, AVE_OP_CLOSE, codec, AVE_SESSION_ARG0, tail); - ctx->session_open = false; - return ret ?: err; + ret = ave_ipc_submit(ave, AVE_BUF_CMD, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, true); + return ret ?: ave_ipc_settle(ave, AVE_START_WAIT_MS); +} + +/* Process (id 8) */ + +struct ave_proc_layout { + u32 first_frame; + u32 y, uv; /* {u64 address, u32 size, u32 stride} */ + u32 in_w, in_h; + u32 coded_cap; + u32 fps_int; + u32 bitrate; + u32 limits, limit_bytes, limit_secs; + u32 const5; + u32 fps; /* f64 */ + u32 timescale; /* u64 */ + u32 slot; + u32 coded, meta; /* u64 addresses */ + u32 ordinal; + u32 pts; /* u64 */ + u32 force_key; /* 1 on a forced key frame */ + u32 force_type; /* 2 on a forced key frame */ + u32 frame_buf; /* H.264 only */ + u32 shdr_table; /* HEVC only */ + u32 crop; /* HEVC only */ +}; + +static const struct ave_proc_layout ave_proc_h264 = { + .first_frame = 0x12c0, + .y = 0x5830, .uv = 0x5850, + .in_w = 0x58e8, .in_h = 0x58ec, + .coded_cap = 0x61c8, + .fps_int = 0x1610, + .bitrate = 0x1620, + .limits = 0x1628, .limit_bytes = 0x1630, .limit_secs = 0x1638, + .const5 = 0x6238, + .fps = 0x6268, .timescale = 0x6278, + .slot = 0x61b4, .coded = 0x61b8, .meta = 0x61c0, + .ordinal = 0x6228, .pts = 0x6270, + .force_key = 0x1658, .force_type = 0x29f8, + .frame_buf = 0x58c8, +}; + +static const struct ave_proc_layout ave_proc_hevc = { + .first_frame = 0x5d90, + .y = 0xa300, .uv = 0xa320, + .in_w = 0xa3b8, .in_h = 0xa3bc, + .coded_cap = 0xac98, + .fps_int = 0x60e0, + .bitrate = 0x60f0, + .limits = 0x60f8, .limit_bytes = 0x6100, .limit_secs = 0x6108, + .const5 = 0xad08, + .fps = 0xad38, .timescale = 0xad48, + .slot = 0xac84, .coded = 0xac88, .meta = 0xac90, + .ordinal = 0xacf8, .pts = 0xad40, + .force_key = 0x6128, .force_type = 0x74c8, + .shdr_table = 0xd88, + .crop = 0x74dc, +}; + +#define AVE_PROC_TIMESCALE 14400000ULL +#define AVE_PROC_SHDR_ENTRIES 256 +#define AVE_PROC_SHDR_STRIDE 0x400 +#define AVE_PROC_FORCED_TYPE 2 + +int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + u32 bitrate, bool force_key) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + const struct ave_proc_layout *l = p->codec == AVE_CODEC_HEVC ? &ave_proc_hevc : + &ave_proc_h264; + const struct firmware *tmpl = ave->tmpl[p->codec - 1][AVE_TMPL_PROCESS]; + struct ave_slot *s = &sess->slot[slot]; + unsigned int buf = AVE_BUF_PROC(slot); + dma_addr_t y; + u64 pts; + u8 *cmd; + int ret; + + y = vb2_dma_contig_plane_dma_addr(&src->vb2_buf, 0) + src->vb2_buf.planes[0].data_offset; + if (!IS_ALIGNED(y, 64)) + return -EINVAL; + + ret = ave_ipc_wait_ack(ave, buf); + if (ret) + return ret; + cmd = ave_ipc_cmd(ave, buf, tmpl->size); + if (!cmd) + return -EBUSY; + memcpy(cmd, tmpl->data, tmpl->size); + ave_cmd_header(ctx, cmd, AVE_OP_PROCESS, AVE_PROC_SUBTYPE + slot, 0, + ave_session_arg2(p->codec)); + + /* Input picture: NV12, chroma right after p->height luma lines. */ + ave_put64(cmd, l->y, y); + ave_put32(cmd, l->y + 8, p->stride * p->height); + ave_put32(cmd, l->y + 12, p->stride); + ave_put64(cmd, l->uv, y + p->stride * p->height); + ave_put32(cmd, l->uv + 8, p->stride * p->height / 2); + ave_put32(cmd, l->uv + 12, p->stride); + ave_put32(cmd, l->in_w, p->width); + ave_put32(cmd, l->in_h, p->height); + + /* Session values repeated in every frame */ + ave_put32(cmd, l->coded_cap, p->coded_cap); + ave_put32(cmd, l->fps_int, p->fps_int); + /* The bit rate may change while streaming; it takes effect here. */ + ave_put32(cmd, l->bitrate, bitrate); + ave_put32(cmd, l->limits, p->limit_bytes ? 1 : 0); + if (p->limit_bytes) { + ave_put64(cmd, l->limit_bytes, ave_f64(p->limit_bytes, 1)); + ave_put64(cmd, l->limit_secs, ave_f64(1, 1)); + } + ave_put32(cmd, l->const5, 5); + ave_put64(cmd, l->fps, ave_f64(p->fps_num, p->fps_den)); + ave_put64(cmd, l->timescale, AVE_PROC_TIMESCALE); + if (l->crop) + ave_put32(cmd, l->crop, p->cw != p->width || p->ch != p->height); + + /* This frame */ + ave_put32(cmd, l->first_frame, sess->frames == 0); + ave_put32(cmd, l->slot, slot); + ave_put64(cmd, l->coded, s->coded.dma); + ave_put64(cmd, l->meta, s->meta.dma); + if (l->frame_buf) + ave_put64(cmd, l->frame_buf, s->aux.dma); + if (l->shdr_table) + for (unsigned int i = 0; i < AVE_PROC_SHDR_ENTRIES; i++) + ave_put64(cmd, l->shdr_table + 8 * i, + s->aux.dma + i * AVE_PROC_SHDR_STRIDE); + ave_put32(cmd, l->ordinal, sess->frames); + pts = mul_u64_u64_div_u64((u64)sess->frames * p->fps_den, AVE_PROC_TIMESCALE, p->fps_num); + ave_put64(cmd, l->pts, pts); + ave_put32(cmd, l->force_key, force_key); + ave_put32(cmd, l->force_type, force_key ? AVE_PROC_FORCED_TYPE : 0); + + ret = ave_ipc_submit(ave, buf, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, false); + if (ret) + return ret; + s->src = src; + s->busy = true; + s->status = 0; + sess->busy++; + sess->frames++; + return 0; } diff --git a/drivers/media/platform/apple/ave/ave-drv.c b/drivers/media/platform/apple/ave/ave-drv.c index edaaf29aa6739..5ed6cf9176a85 100644 --- a/drivers/media/platform/apple/ave/ave-drv.c +++ b/drivers/media/platform/apple/ave/ave-drv.c @@ -88,19 +88,19 @@ static void ave_remove(struct platform_device *pdev) ave_v4l2_unregister(ave); /* - * Make sure the coprocessor really loses power before its memory is - * freed: drop to runtime suspend synchronously and check. + * The coprocessor memory can only go once the firmware never ran or + * was reset by a power cycle. A halted coprocessor may keep power + * (its domain is shared with the DARTs), so its memory stays. */ pm_runtime_dont_use_autosuspend(ave->dev); - if (pm_runtime_suspended(ave->dev) || pm_runtime_status_suspended(ave->dev)) { - mutex_lock(&ave->hw_lock); - ave_fw_lost(ave); + mutex_lock(&ave->hw_lock); + if (ave->fw_state != AVE_FW_OFF) { mutex_unlock(&ave->hw_lock); - ave_fw_remove(ave); - } else { - /* The firmware still owns its buffers; leave them allocated. */ - dev_warn(ave->dev, "firmware still running, keeping its memory\n"); + dev_warn(ave->dev, "firmware was started, keeping its memory\n"); + return; } + mutex_unlock(&ave->hw_lock); + ave_fw_remove(ave); } static int ave_runtime_suspend(struct device *dev) @@ -123,12 +123,21 @@ static int ave_suspend(struct device *dev) { struct ave_dev *ave = dev_get_drvdata(dev); - /* Let a running frame finish; the m2m core holds back new jobs. */ + /* Let a running job finish; the m2m core holds back new ones. */ v4l2_m2m_suspend(ave->m2m_dev); + /* + * An open session cannot survive the power going down: it is ended + * here (its next job fails) and the firmware halted, so the next + * session restarts it from the saved DATA copy. + */ mutex_lock(&ave->hw_lock); - if (ave->fw_state != AVE_FW_OFF && ave_fw_shutdown(ave)) - dev_dbg(dev, "firmware will restart after resume\n"); + if (ave->active) { + ave->active->sess->dead = true; + ave->active = NULL; + } + if (ave_fw_shutdown(ave)) + dev_warn(dev, "firmware not answering, it cannot be stopped for sleep\n"); mutex_unlock(&ave->hw_lock); return 0; } @@ -137,15 +146,7 @@ static int ave_resume(struct device *dev) { struct ave_dev *ave = dev_get_drvdata(dev); - /* - * If the power domains went down with the system, the coprocessor is - * back in reset: open sessions fail their next frame and a new session - * boots the firmware again from its saved state. - */ - mutex_lock(&ave->hw_lock); - ave_fw_resume(ave); - mutex_unlock(&ave->hw_lock); - + /* The next boot checks the coprocessor state itself. */ v4l2_m2m_resume(ave->m2m_dev); return 0; } @@ -168,11 +169,17 @@ static struct platform_driver ave_driver = { .name = "apple-ave", .of_match_table = ave_of_match, .pm = pm_ptr(&ave_pm_ops), - /* The firmware cannot be stopped yet, so neither can the driver. */ + /* A started coprocessor keeps its memory; do not unbind at will. */ .suppress_bind_attrs = true, }, }; module_platform_driver(ave_driver); +/* Constant parts of Start and Process for AppleAVE2FW 9003.78.0 (macOS 26.0.x) */ +MODULE_FIRMWARE("apple/ave-t8140-69c54454-h264-start.bin"); +MODULE_FIRMWARE("apple/ave-t8140-69c54454-h264-process.bin"); +MODULE_FIRMWARE("apple/ave-t8140-69c54454-hevc-start.bin"); +MODULE_FIRMWARE("apple/ave-t8140-69c54454-hevc-process.bin"); + MODULE_DESCRIPTION("Apple AVE video encoder"); MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index fa2df964d16f4..62eb494c149b7 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -20,6 +21,7 @@ #include #include #include +#include #include #include "ave.h" @@ -133,6 +135,8 @@ int ave_fw_probe(struct ave_dev *ave) if (!known) return dev_err_probe(dev, -ENODEV, "unsupported firmware (text CRC %08x)\n", text_crc); + ave->text_crc = text_crc; + mutex_init(&ave->tmpl_lock); ave->data_pristine = vmalloc(ave->data.size); if (!ave->data_pristine) @@ -170,19 +174,24 @@ static void ave_fw_restore_data(struct ave_dev *ave) ave->data_dirty = false; } -static void ave_free(struct ave_dev *ave, struct ave_buf *b) +void ave_free(struct ave_dev *ave, struct ave_buf *b) { if (b->cpu) - dma_free_coherent(ave->dev, b->size, b->cpu, b->dma); + dma_free_attrs(ave->dev, PAGE_ALIGN(b->size), b->cpu, b->dma, b->attrs); b->cpu = NULL; } -static int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size) +/* + * Firmware-visible memory, zeroed and 16 KiB aligned in the encoder's DART. + * @size is what the firmware is told; the allocation is page rounded. + */ +int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs) { if (b->cpu) return 0; b->size = size; - b->cpu = dma_alloc_coherent(ave->dev, size, &b->dma, GFP_KERNEL); + b->attrs = attrs; + b->cpu = dma_alloc_attrs(ave->dev, PAGE_ALIGN(size), &b->dma, GFP_KERNEL, attrs); if (!b->cpu) return -ENOMEM; if (!IS_ALIGNED(b->dma, SZ_16K)) { @@ -192,8 +201,61 @@ static int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size) return 0; } +/* + * Start and Process are mostly constant for a firmware build. Those + * constant words come from per-build image files named after the TEXT + * checksum; the driver fills in every session and frame field itself. + */ +static const char * const ave_codec_name[] = { "h264", "hevc" }; +static const char * const ave_tmpl_name[AVE_TMPL_COUNT] = { "start", "process" }; +static const size_t ave_tmpl_size[2][AVE_TMPL_COUNT] = { + { 0x3180, 0x63d8 }, /* H.264 */ + { 0x13f28, 0xb1c0 }, /* HEVC */ +}; + +#define AVE_TMPL_CODEC 0xc0 /* Start: codec word of the session image */ + +int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec) +{ + unsigned int c = codec - 1; + char name[64]; + int ret = 0; + + mutex_lock(&ave->tmpl_lock); + for (unsigned int t = 0; t < AVE_TMPL_COUNT && !ret; t++) { + const struct firmware *fw; + + if (ave->tmpl[c][t]) + continue; + snprintf(name, sizeof(name), "apple/ave-t8140-%08x-%s-%s.bin", ave->text_crc, + ave_codec_name[c], ave_tmpl_name[t]); + ret = request_firmware(&fw, name, ave->dev); + if (ret) { + dev_err(ave->dev, "cannot load %s: %d\n", name, ret); + break; + } + if (fw->size != ave_tmpl_size[c][t] || + (t == AVE_TMPL_START && + get_unaligned_le32(fw->data + AVE_TMPL_CODEC) != codec)) { + dev_err(ave->dev, "%s does not fit this driver (%zu bytes)\n", name, + fw->size); + release_firmware(fw); + ret = -EINVAL; + break; + } + ave->tmpl[c][t] = fw; + } + mutex_unlock(&ave->tmpl_lock); + return ret; +} + void ave_fw_remove(struct ave_dev *ave) { + for (unsigned int c = 0; c < 2; c++) + for (unsigned int t = 0; t < AVE_TMPL_COUNT; t++) { + release_firmware(ave->tmpl[c][t]); + ave->tmpl[c][t] = NULL; + } ave_free(ave, &ave->heap); ave_free(ave, &ave->log); ave_free(ave, &ave->boot_cfg); @@ -214,6 +276,38 @@ static bool ave_fw_idle(struct ave_dev *ave) return true; } +/* + * After a Halt the firmware no longer runs, but the coprocessor is still + * set going if its power stayed on. A restart uses the normal boot path + * (macOS restarts the firmware for every session the same way): clear the + * run bit, as the boot sequence does first anyway, and the mailbox words + * left from the previous run, so the reset-state checks hold again. The + * code base register must still be set: the driver never writes it. + */ +static int ave_fw_prepare_restart(struct ave_dev *ave) +{ + u32 status; + + if (readq(ave->coproc + AVE_COPROC_VCBAR) != (ave->text.iova | AVE_COPROC_VCBAR_VALID)) { + dev_err(ave->dev, "coprocessor lost its code base (vcbar %#llx)\n", + readq(ave->coproc + AVE_COPROC_VCBAR)); + return -EIO; + } + if (ave->fw_state != AVE_FW_HALTED) + return 0; + + if (readl(ave->coproc + AVE_COPROC_CTRL)) { + writel(0, ave->coproc + AVE_COPROC_CTRL); + usleep_range(1000, 2000); + } + status = readl(ave->mbox + AVE_MBOX_STATUS); + if (status) + writel(status, ave->mbox + AVE_MBOX_STATUS); + for (unsigned int i = 0; i < AVE_MBOX_SCRATCH_COUNT; i++) + writel(0, ave->mbox + AVE_MBOX_SCRATCH(i)); + return 0; +} + /* Boot the coprocessor and bring up its command rings. Caller holds hw_lock. */ static int ave_fw_boot(struct ave_dev *ave) { @@ -221,11 +315,20 @@ static int ave_fw_boot(struct ave_dev *ave) u32 status, hello[4]; int ret; - ret = ave_alloc(ave, &ave->ipc, AVE_IPC_BYTES); + ret = ave_alloc(ave, &ave->ipc, AVE_IPC_BYTES, 0); if (!ret) - ret = ave_alloc(ave, &ave->boot_cfg, SZ_16K); + ret = ave_alloc(ave, &ave->boot_cfg, SZ_16K, 0); if (!ret) - ret = ave_alloc(ave, &ave->log, AVE_LOG_BYTES); + ret = ave_alloc(ave, &ave->log, AVE_LOG_BYTES, 0); + if (ret) + return ret; + + if (ave_fw_check_mapping(ave, &ave->text) || ave_fw_check_mapping(ave, &ave->data)) { + dev_err(ave->dev, "firmware segments are no longer mapped\n"); + return -EIO; + } + + ret = ave_fw_prepare_restart(ave); if (ret) return ret; @@ -240,6 +343,13 @@ static int ave_fw_boot(struct ave_dev *ave) return -EIO; } + memset(ave->ring, 0, sizeof(ave->ring)); + ave->unacked = 0; + memset(ave->want, 0, sizeof(ave->want)); + ave->reply_err = 0; + ave->scalar_next = 0; + ave->next_token = AVE_TOKEN_BASE; + ave_fw_restore_data(ave); cfg = ave->boot_cfg.cpu; @@ -306,9 +416,9 @@ int ave_fw_ensure_running(struct ave_dev *ave) case AVE_FW_RUNNING: return 0; case AVE_FW_BROKEN: - /* Only a power cycle gets the coprocessor back; see runtime PM. */ return -EIO; case AVE_FW_OFF: + case AVE_FW_HALTED: break; } @@ -318,22 +428,47 @@ int ave_fw_ensure_running(struct ave_dev *ave) return ret; } +/* + * Tell the firmware to stop, once every session is closed. Halt gets no + * ack and no answer; after a short wait the coprocessor is treated as + * stopped and the next session restarts it through the boot path. + * Caller holds hw_lock. + */ +void ave_fw_halt(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) + return; + if (ave_cmd_halt(ave)) { + ave->fw_state = AVE_FW_BROKEN; + return; + } + msleep(AVE_HALT_SETTLE_MS); + ave->fw_state = AVE_FW_HALTED; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; +} + /* * Stop the firmware before its power goes away. Caller holds hw_lock. * - * The firmware is told to halt once every session is closed. Stopping the - * coprocessor core itself and checking that the fabric is idle are not - * described yet, so until then a running firmware keeps its power (runtime - * suspend fails with -EBUSY). Across system sleep the domains go down anyway; - * ave_fw_resume() notices and the next session boots the firmware again from - * the saved DATA copy. + * Sessions keep the device runtime active, so the firmware has normally + * been halted already. A firmware that stopped answering keeps its power: + * whether it still has transfers in flight cannot be checked, and only a + * reboot brings it back. */ int ave_fw_shutdown(struct ave_dev *ave) { lockdep_assert_held(&ave->hw_lock); - if (ave->fw_state == AVE_FW_OFF) + switch (ave->fw_state) { + case AVE_FW_RUNNING: + ave_fw_halt(ave); + return ave->fw_state == AVE_FW_HALTED ? 0 : -EBUSY; + case AVE_FW_BROKEN: + return -EBUSY; + default: return 0; - return -EBUSY; + } } /* The coprocessor lost power behind the firmware's back. Caller holds hw_lock. */ @@ -344,16 +479,3 @@ void ave_fw_lost(struct ave_dev *ave) memset(ave->ring, 0, sizeof(ave->ring)); ave->ipc_fw_base = 0; } - -/* After system resume: did the coprocessor keep running? Caller holds hw_lock. */ -void ave_fw_resume(struct ave_dev *ave) -{ - lockdep_assert_held(&ave->hw_lock); - if (ave->fw_state == AVE_FW_OFF) - return; - /* A running firmware keeps the device runtime active, so this is powered. */ - if (readl(ave->coproc + AVE_COPROC_CTRL) & AVE_COPROC_CTRL_RUN) - return; - dev_dbg(ave->dev, "coprocessor lost power during sleep\n"); - ave_fw_lost(ave); -} diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c index 11f0aceb6fb65..d316edca01b3c 100644 --- a/drivers/media/platform/apple/ave/ave-ipc.c +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -250,16 +250,172 @@ static int ave_ring_recv(struct ave_dev *ave, struct ave_ring *r, u64 *addr, u32 return 0; } +static int ave_ipc_buf_offset(unsigned int buf, size_t *max) +{ + if (buf == AVE_BUF_CMD) { + *max = AVE_IPC_CMD_BYTES; + return AVE_IPC_CMD_OFF; + } + if (buf < AVE_BUF_PROC(0)) { + *max = AVE_IPC_SCALAR_BYTES; + return AVE_IPC_SCALAR_OFF + (buf - AVE_BUF_SCALAR(0)) * AVE_IPC_SCALAR_BYTES; + } + *max = AVE_IPC_PROC_BYTES; + return AVE_IPC_PROC_OFF + (buf - AVE_BUF_PROC(0)) * AVE_IPC_PROC_BYTES; +} + +/* Which command buffer does a transport ack refer to? */ +static int ave_ipc_buf_of(struct ave_dev *ave, u64 addr) +{ + u64 off = addr - ave->ipc_fw_base; + size_t max; + + if (addr < ave->ipc_fw_base || off < AVE_IPC_CMD_OFF || off >= AVE_IPC_END) + return -EPROTO; + for (unsigned int buf = 0; buf < AVE_BUF_COUNT; buf++) + if (ave_ipc_buf_offset(buf, &max) == off) + return buf; + return -EPROTO; +} + +/* + * Get command buffer @buf, cleared for @len bytes, or NULL while the + * firmware still owns it. Caller holds hw_lock. + */ +void *ave_ipc_cmd(struct ave_dev *ave, unsigned int buf, size_t len) +{ + size_t max; + int off; + + if (buf >= AVE_BUF_COUNT || (ave->unacked & BIT(buf))) + return NULL; + off = ave_ipc_buf_offset(buf, &max); + if (len > max) + return NULL; + memset(ave->ipc.cpu + off, 0, len); + return ave->ipc.cpu + off; +} + +static bool ave_ipc_acked(struct ave_dev *ave, void *arg) +{ + return !(ave->unacked & BIT(*(unsigned int *)arg)); +} + +/* Wait until the firmware has taken command buffer @buf. Caller holds hw_lock. */ +int ave_ipc_wait_ack(struct ave_dev *ave, unsigned int buf) +{ + return ave_ipc_pump(ave, AVE_CMD_WAIT_MS, ave_ipc_acked, &buf); +} + +static bool ave_ipc_scalar_free(struct ave_dev *ave, void *arg) +{ + return (ave->unacked & GENMASK(AVE_BUF_PROC(0) - 1, AVE_BUF_SCALAR(0))) != + GENMASK(AVE_BUF_PROC(0) - 1, AVE_BUF_SCALAR(0)); +} + +/* Pick a free buffer for a scalar command. Caller holds hw_lock. */ +int ave_ipc_scalar_buf(struct ave_dev *ave) +{ + int ret; + + ret = ave_ipc_pump(ave, AVE_CMD_WAIT_MS, ave_ipc_scalar_free, NULL); + if (ret) + return ret; + for (unsigned int i = 0; i < AVE_IPC_SCALAR_BUFS; i++) { + unsigned int buf = AVE_BUF_SCALAR((ave->scalar_next + i) % AVE_IPC_SCALAR_BUFS); + + if (!(ave->unacked & BIT(buf))) { + ave->scalar_next = (buf - AVE_BUF_SCALAR(0) + 1) % AVE_IPC_SCALAR_BUFS; + return buf; + } + } + return -EBUSY; +} + +static void ave_ipc_mark_broken(struct ave_dev *ave, int err, const char *what) +{ + if (ave->fw_state == AVE_FW_RUNNING) + dev_err(ave->dev, "firmware stopped answering (%s: %d); it restarts after the next power cycle\n", + what, err); + ave->fw_state = AVE_FW_BROKEN; +} + +/* + * Hand the command in buffer @buf to the firmware. The timeout words are + * {duration, creation} in the host clock the firmware was synced to at boot. + * With @reply, the firmware's answer is counted as expected; Halt and + * Process are answered differently. Caller holds hw_lock. + */ +int ave_ipc_submit(struct ave_dev *ave, unsigned int buf, size_t len, u64 timeout_us, + bool reply) +{ + size_t max; + int off = ave_ipc_buf_offset(buf, &max); + u8 *cmd = ave->ipc.cpu + off; + u32 op = get_unaligned_le32(cmd + AVE_CMD_OPCODE); + int ret; + + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) + return -EIO; + if (len < AVE_CMD_HDR_BYTES || len > max || !IS_ALIGNED(len, 4) || + (ave->unacked & BIT(buf)) || (reply && op >= ARRAY_SIZE(ave->want))) + return -EINVAL; + + put_unaligned_le64(timeout_us, cmd + AVE_CMD_TIMEOUT); + put_unaligned_le64(div_u64(ktime_get_ns(), NSEC_PER_USEC), cmd + AVE_CMD_CREATED); + + ret = ave_ring_send(ave, &ave->ring[0], ave->ipc_fw_base + off, len); + if (ret) { + ave_ipc_mark_broken(ave, ret, "command ring full"); + return ret; + } + ave->unacked |= BIT(buf); + if (reply) + ave->want[op]++; + return 0; +} + +static void ave_ipc_dispatch(struct ave_dev *ave) +{ + u32 id, status; + + if (ave->msg_len < AVE_REPLY_MIN_BYTES) { + dev_warn(ave->dev, "short firmware message (%u bytes)\n", ave->msg_len); + ave->reply_err = -EPROTO; + return; + } + id = get_unaligned_le16(ave->msg + AVE_REPLY_ID); + status = get_unaligned_le32(ave->msg + AVE_REPLY_STATUS); + + if (id == AVE_OP_PROCESS) { + if (ave->active) + ave_sess_reply(ave->active, ave->msg, ave->msg_len); + else + dev_warn(ave->dev, "frame reply without a session\n"); + return; + } + if (id < ARRAY_SIZE(ave->want) && ave->want[id]) { + ave->want[id]--; + if (status) { + dev_err(ave->dev, "command %u refused: status %#x\n", id, status); + ave->reply_err = -EIO; + } + return; + } + dev_warn_ratelimited(ave->dev, "unexpected firmware message %u (status %#x)\n", + id, status); +} + /* - * Poll both rings once. Returns 1 when the command at @cmd_addr was - * acknowledged, 2 when a firmware message was copied to ave->msg (and its - * buffer handed back), 0 when nothing was pending. + * Poll both rings once. Returns 1 when an event (ack or message) was + * handled, 0 when nothing was pending. */ -static int ave_ipc_poll_once(struct ave_dev *ave, u64 cmd_addr) +static int ave_ipc_poll_once(struct ave_dev *ave) { u32 pending, len, off; u64 addr; - int ret; + int ret, buf; pending = readl(ave->mbox + AVE_MBOX_STATUS); if (pending) @@ -268,8 +424,12 @@ static int ave_ipc_poll_once(struct ave_dev *ave, u64 cmd_addr) ret = ave_ring_recv(ave, &ave->ring[0], &addr, &len); if (!ret) { /* Transport acks return the command address with no payload. */ - if (len || addr != cmd_addr) + buf = ave_ipc_buf_of(ave, addr); + if (len || buf < 0 || !(ave->unacked & BIT(buf))) { + dev_err(ave->dev, "bad command ack %#llx/%u\n", addr, len); return -EPROTO; + } + ave->unacked &= ~BIT(buf); return 1; } if (ret != -EAGAIN) @@ -282,103 +442,80 @@ static int ave_ipc_poll_once(struct ave_dev *ave, u64 cmd_addr) return ret; if (len > sizeof(ave->msg) || ave_fw_offset(ave, addr, len, &off)) return -EPROTO; + /* Messages must not land in the buffers the host writes commands to. */ + if (off < AVE_IPC_END && off + len > AVE_IPC_CMD_OFF) { + dev_err(ave->dev, "firmware message at %#x overlaps the command area\n", off); + return -EPROTO; + } memcpy(ave->msg, ave->ipc.cpu + off, len); ave->msg_len = len; /* Return the message buffer to the firmware. */ ret = ave_ring_send(ave, &ave->ring[1], addr, len); - return ret ?: 2; + if (ret) + return ret; + ave_ipc_dispatch(ave); + return 1; } /* - * Send one command and wait for its transport ack and the firmware's reply - * message. The reply is left in ave->msg. Caller holds hw_lock. + * Handle acks and messages until @cond holds, for at most @wait_ms. With no + * @cond, handle whatever is pending and return. Running out of time means + * the firmware is stuck: it is marked broken. Caller holds hw_lock. */ -int ave_ipc_call(struct ave_dev *ave, const void *cmd, size_t len) +int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, + bool (*cond)(struct ave_dev *ave, void *arg), void *arg) { - u64 cmd_addr = ave->ipc_fw_base + AVE_IPC_CMD_OFF; - u8 *buf = ave->ipc.cpu + AVE_IPC_CMD_OFF; - bool acked = false, replied = false; - ktime_t deadline; - u32 status; + ktime_t deadline = ktime_add_ms(ktime_get(), wait_ms); int ret; lockdep_assert_held(&ave->hw_lock); - if (ave->fw_state != AVE_FW_RUNNING) - return -EIO; - if (len < AVE_CMD_HDR_BYTES || len > AVE_IPC_CMD_BYTES || !IS_ALIGNED(len, 4)) - return -EINVAL; - - memcpy(buf, cmd, len); - /* The timeout is {duration, creation} in the clock synced at boot. */ - put_unaligned_le64(AVE_CMD_TIMEOUT_US, buf + AVE_CMD_TIMEOUT); - put_unaligned_le64(div_u64(ktime_get_ns(), NSEC_PER_USEC), buf + AVE_CMD_CREATED); - ave->msg_len = 0; - - ret = ave_ring_send(ave, &ave->ring[0], cmd_addr, len); - if (ret) - goto broken; - - deadline = ktime_add_ms(ktime_get(), AVE_CMD_WAIT_MS); - while (!acked || !replied) { - ret = ave_ipc_poll_once(ave, cmd_addr); - if (ret < 0) - goto broken; - if (ret == 1) - acked = true; - else if (ret == 2) - replied = true; - else if (ktime_after(ktime_get(), deadline)) - goto timeout; - else - usleep_range(50, 150); - } - - /* The firmware refused the command: the session fails, the firmware lives on. */ - if (ave->msg_len < AVE_REPLY_MIN_BYTES) - goto broken_len; - status = get_unaligned_le32(ave->msg + AVE_REPLY_STATUS); - if (status) { - dev_err(ave->dev, "command %u refused: status %#x\n", - get_unaligned_le32(cmd), status); - return -EIO; + for (;;) { + if (cond && cond(ave, arg)) + return 0; + if (ave->fw_state != AVE_FW_RUNNING) + return -EIO; + ret = ave_ipc_poll_once(ave); + if (ret < 0) { + ave_ipc_mark_broken(ave, ret, "ring error"); + return ret; + } + if (ret) + continue; + if (!cond) + return 0; + if (ktime_after(ktime_get(), deadline)) { + ave_ipc_mark_broken(ave, -ETIMEDOUT, "no answer"); + return -ETIMEDOUT; + } + usleep_range(50, 150); } - return 0; +} -broken_len: - ret = -EPROTO; - goto broken; -timeout: - ret = -ETIMEDOUT; -broken: - dev_err(ave->dev, "command %u failed: %d (acked %d, replied %d)\n", - get_unaligned_le32(cmd), ret, acked, replied); - ave->fw_state = AVE_FW_BROKEN; - return ret; +static bool ave_ipc_answered(struct ave_dev *ave, void *arg) +{ + u32 mask = *(u32 *)arg; + + if (ave->unacked & mask) + return false; + for (unsigned int i = 0; i < ARRAY_SIZE(ave->want); i++) + if (ave->want[i] && i != AVE_OP_HALT) + return false; + return true; } -/* Wait for one more firmware message (frame completions). */ -int ave_ipc_wait_msg(struct ave_dev *ave, unsigned int timeout_ms) +/* + * Wait until every command except Process has been acknowledged and + * answered. Returns the first refusal seen since the last call. + * Caller holds hw_lock. + */ +int ave_ipc_settle(struct ave_dev *ave, unsigned int wait_ms) { - ktime_t deadline = ktime_add_ms(ktime_get(), timeout_ms); + u32 mask = GENMASK(AVE_BUF_PROC(0) - 1, 0); int ret; - lockdep_assert_held(&ave->hw_lock); - if (ave->fw_state != AVE_FW_RUNNING) - return -EIO; - - for (;;) { - ret = ave_ipc_poll_once(ave, 0); - if (ret == 2) - return 0; - if (ret < 0) - break; - if (ktime_after(ktime_get(), deadline)) { - ret = -ETIMEDOUT; - break; - } - usleep_range(50, 150); - } - dev_err(ave->dev, "no firmware message: %d\n", ret); - ave->fw_state = AVE_FW_BROKEN; + ret = ave_ipc_pump(ave, wait_ms, ave_ipc_answered, &mask); + if (!ret) + ret = ave->reply_err; + ave->reply_err = 0; return ret; } diff --git a/drivers/media/platform/apple/ave/ave-regs.h b/drivers/media/platform/apple/ave/ave-regs.h index 91e7c2c403c3e..894d2e3d4ff0a 100644 --- a/drivers/media/platform/apple/ave/ave-regs.h +++ b/drivers/media/platform/apple/ave/ave-regs.h @@ -89,8 +89,15 @@ #define AVE_CMD_TIMEOUT 0x30 #define AVE_CMD_CREATED 0x38 #define AVE_CMD_HDR_BYTES 0x48 -#define AVE_CMD_TIMEOUT_US 3000000 + +/* Timeouts the firmware is given, and how long the host waits */ +#define AVE_CMD_GLOBAL_TIMEOUT_US 3000000ULL +#define AVE_CMD_SESSION_TIMEOUT_US 94400000ULL #define AVE_CMD_WAIT_MS 5000 +#define AVE_START_WAIT_MS 10000 +#define AVE_FRAME_WAIT_MS 5000 +/* Halt is never answered; give the firmware this long before power-off. */ +#define AVE_HALT_SETTLE_MS 20 enum ave_opcode { AVE_OP_CONFIG = 1, @@ -110,19 +117,31 @@ enum ave_opcode { #define AVE_REPLY_SUBTYPE 0x20 #define AVE_REPLY_STATUS 0x40 #define AVE_REPLY_MIN_BYTES 0x48 +#define AVE_REPLY_PROCESS_BYTES 0x50 #define AVE_HALT_ALL 0xffffffff -/* Session command arguments, as the firmware expects them */ +/* Session command header values */ +#define AVE_SESSION_UNIT 1 #define AVE_SESSION_ARG0 0xc8 #define AVE_SESSION_ARG1 0x36 -#define AVE_OPEN_ARG2 0x1e0 -#define AVE_FINISH_ARG2_H264 0x140 -#define AVE_FINISH_ARG2_HEVC 0x20 +#define AVE_SESSION_ARG2_SETUP 0x1e0 /* Open and Start */ +#define AVE_SESSION_ARG2_H264 0x40 /* every other session command */ +#define AVE_SESSION_ARG2_HEVC 0x20 +#define AVE_TOKEN_STEP 100 +#define AVE_TOKEN_BASE 300 /* the first Open carries 400 */ +#define AVE_CLIENT_STEP 10 + +/* Process slots: the command subtype is this plus the slot index */ +#define AVE_PROC_SUBTYPE 0x15 +/* Completes sent to drain a session */ +#define AVE_DRAIN_COMPLETES 4 #define AVE_CONFIG_BYTES 0x78 #define AVE_CONFIG_ALL_UNITS 0x20 #define AVE_CONFIG_OPT_A 0x49 #define AVE_CONFIG_OPT_B 0x4a +#define AVE_CONFIG_MCC_DSID 0x60 +#define AVE_CONFIG_MCC_DSID_T8140 0x12 #endif diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c new file mode 100644 index 0000000000000..4d65dd09d732a --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -0,0 +1,493 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: encoding sessions + * + * A session owns the encoder's working buffers (sized from the picture + * size), a ring of up to five Process slots and the coded output of each + * slot. The firmware answers frames in bitstream order, so the N-th answer + * is the N-th access unit; with B-frames that is not the input order. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +#define AVE_RC_VBR 4 +#define AVE_RC_CBR 2 + +#define AVE_H264_BASELINE 2 +#define AVE_H264_MAIN 4 +#define AVE_H264_HIGH 6 +#define AVE_HEVC_MAIN 1 + +#define AVE_META_BYTES 0xc000 +#define AVE_META_HEVC_PAYLOAD 0x120 +#define AVE_META_HEVC_SHDR 0x128 +#define AVE_META_HEVC_PAYLOAD2 0x12c +#define AVE_META_HEVC_PARAMS 0xb2a4 +#define AVE_SHDR_BYTES (256 * 0x400) +#define AVE_SHDR_ENTRY_BYTES 0x400 +#define AVE_CODED_MIN 0x3c000 +#define AVE_HDR_BYTES 0x12c000 + +/* Spacing of the H.264 per-frame buffer seen at 1080p; scaled with area. */ +#define AVE_H264_FRAME_BUF_1080P 0x364000 +#define AVE_MBS_1080P (120 * 68) + +/* Level limits: max frame size, max rate, max bit rate (kbit/s, Main). */ +struct ave_level { + u32 max_fs; + u64 max_rate; + u32 max_kbps; +}; + +/* H.264 Table A-1, in V4L2 menu order (1, 1b, 1.1 ... 5.2); macroblocks. */ +static const struct ave_level ave_h264_levels[] = { + { 99, 1485, 64 }, { 99, 1485, 128 }, { 396, 3000, 192 }, { 396, 6000, 384 }, + { 396, 11880, 768 }, { 396, 11880, 2000 }, { 792, 19800, 4000 }, + { 1620, 20250, 4000 }, { 1620, 40500, 10000 }, { 3600, 108000, 14000 }, + { 5120, 216000, 20000 }, { 8192, 245760, 20000 }, { 8192, 245760, 50000 }, + { 8704, 522240, 50000 }, { 22080, 589824, 135000 }, { 36864, 983040, 240000 }, + { 36864, 2073600, 240000 }, +}; + +/* HEVC Table A-8 (Main tier), in V4L2 menu order (1, 2, 2.1 ... 5.1); samples. */ +static const struct ave_level ave_hevc_levels[] = { + { 36864, 552960, 128 }, { 122880, 3686400, 1500 }, { 245760, 7372800, 3000 }, + { 552960, 16588800, 6000 }, { 983040, 33177600, 10000 }, + { 2228224, 66846720, 12000 }, { 2228224, 133693440, 20000 }, + { 8912896, 267386880, 25000 }, { 8912896, 534773760, 40000 }, +}; + +/* + * The lowest level that fits the picture, rate and bit rate, or the + * requested one if that is higher. Returned as the firmware's 1-based enum. + */ +static int ave_pick_level(const struct ave_params *p, u32 requested) +{ + bool hevc = p->codec == AVE_CODEC_HEVC; + const struct ave_level *t = hevc ? ave_hevc_levels : ave_h264_levels; + unsigned int n = hevc ? ARRAY_SIZE(ave_hevc_levels) : ARRAY_SIZE(ave_h264_levels); + u64 size = hevc ? (u64)p->cw * p->ch : p->mbs; + u64 side = hevc ? max(p->cw, p->ch) : p->cw / 16; + u64 rate = DIV_ROUND_UP_ULL(size * p->fps_num, p->fps_den); + u32 kbps = DIV_ROUND_UP(p->bitrate, 1000); + + for (unsigned int i = 0; i < n; i++) { + u32 max_kbps = t[i].max_kbps; + + if (p->high) + max_kbps += max_kbps / 4; + if (size <= t[i].max_fs && side * side <= 8ULL * t[i].max_fs && + rate <= t[i].max_rate && kbps <= max_kbps) + return max(i, requested) + 1; + } + return -EINVAL; +} + +static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) +{ + bool baseline = false; + int level; + + p->codec = ave_ctx_codec(ctx); + p->width = ctx->src_fmt.width; + p->height = ctx->src_fmt.height; + p->cw = ALIGN(p->width, 16); + p->ch = ALIGN(p->height, 16); + p->stride = ctx->src_fmt.plane_fmt[0].bytesperline; + p->mbs = (p->cw / 16) * (p->ch / 16); + p->fps_num = ctx->timeperframe.denominator; + p->fps_den = ctx->timeperframe.numerator; + p->fps_int = max(DIV_ROUND_CLOSEST(p->fps_num, p->fps_den), 1U); + + if (p->codec == AVE_CODEC_H264) { + switch (ctx->h264_profile) { + case V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE: + case V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE: + p->profile = AVE_H264_BASELINE; + baseline = true; + break; + case V4L2_MPEG_VIDEO_H264_PROFILE_HIGH: + p->profile = AVE_H264_HIGH; + p->high = true; + break; + default: + p->profile = AVE_H264_MAIN; + break; + } + p->min_qp = ctx->h264_min_qp; + p->max_qp = ctx->h264_max_qp; + } else { + p->profile = AVE_HEVC_MAIN; + p->min_qp = ctx->hevc_min_qp; + p->max_qp = ctx->hevc_max_qp; + } + if (p->min_qp > p->max_qp) + p->min_qp = p->max_qp; + + /* + * Reference structures as macOS sets them up. H.264 has all three + * (B-frames, P only, intra only). HEVC without B-frames needs a + * reference-set table whose contents are not known yet, so HEVC + * always uses B-frames. + */ + p->intra_only = p->codec == AVE_CODEC_H264 && ctx->gop_size == 1; + p->bframes = p->codec == AVE_CODEC_HEVC || + (!baseline && !p->intra_only && ctx->b_frames); + p->refs = p->intra_only ? 1 : p->bframes ? 5 : 2; + p->slots = p->bframes ? 5 : 3; + p->gop = ctx->gop_size ?: 120; + + p->bitrate = ctx->bitrate; + if (ctx->bitrate_mode == V4L2_MPEG_VIDEO_BITRATE_MODE_CBR) { + p->rc_mode = AVE_RC_CBR; + p->limit_bytes = 0; + } else { + p->rc_mode = AVE_RC_VBR; + p->limit_bytes = max(ctx->bitrate_peak, ctx->bitrate) / 8; + } + + level = ave_pick_level(p, p->codec == AVE_CODEC_H264 ? ctx->h264_level : + ctx->hevc_level); + if (level < 0) + return level; + p->level = level; + + /* Coded-data capacity: 1.5 bytes per (aligned) pixel, 16 KiB pages. */ + p->coded_cap = ALIGN(3 * (p->codec == AVE_CODEC_HEVC ? ALIGN(p->width, 32) : + ALIGN(p->width, 16)) * + ALIGN(p->height, 16) / 2, SZ_16K); + p->coded_cap = max_t(u32, p->coded_cap, AVE_CODED_MIN); + return 0; +} + +/* Working buffer sizes, all from the requested picture size. */ +static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) +{ + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + u32 w = p->width, h = p->height; + u32 w16 = DIV_ROUND_UP(w, 16), w32 = DIV_ROUND_UP(w, 32); + u32 h32 = DIV_ROUND_UP(h, 32), w64 = DIV_ROUND_UP(w, 64), h64 = DIV_ROUND_UP(h, 64); + const unsigned long nomap = DMA_ATTR_NO_KERNEL_MAPPING; + size_t ref_a, ref_b, ref_c, ref_d, tile, frame_buf; + int ret = 0; + + ref_a = ALIGN(w, 32) * ALIGN(h + 16, 32); + ref_b = roundup_pow_of_two(w) * roundup_pow_of_two(h) / 32; + ref_c = hevc ? 512 * w64 * h64 : ALIGN(w, 64) * (ALIGN(h, 16) / 2 + 24); + ref_d = ALIGN(hevc ? 64 * w32 * h32 : 128 * p->mbs, SZ_16K); + tile = hevc ? 2304 * w32 * h64 : ALIGN(w, 16) * ALIGN(h, 64); + + ret |= ave_alloc(ave, &sess->hdr, AVE_HDR_BYTES, nomap); + for (unsigned int i = 0; i < p->refs; i++) { + ret |= ave_alloc(ave, &sess->ref_a[i], ref_a, nomap); + ret |= ave_alloc(ave, &sess->ref_b[i], ref_b, nomap); + ret |= ave_alloc(ave, &sess->ref_c[i], ref_c, nomap); + ret |= ave_alloc(ave, &sess->ref_d[i], ref_d, nomap); + } + ret |= ave_alloc(ave, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), nomap); + ret |= ave_alloc(ave, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), nomap); + ret |= ave_alloc(ave, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), nomap); + ret |= ave_alloc(ave, &sess->fixed, SZ_16K, nomap); + for (unsigned int i = 0; i < (hevc ? 4 : AVE_TILE_BUFS); i++) { + ret |= ave_alloc(ave, &sess->tile[i], tile, nomap); + ret |= ave_alloc(ave, &sess->tile_aux[i], ALIGN(h, 128), nomap); + } + if (hevc) + for (unsigned int i = 0; i < 2; i++) + ret |= ave_alloc(ave, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), nomap); + + /* Per-slot output; the size of the H.264 frame buffer is not known. */ + frame_buf = ALIGN(DIV_ROUND_UP_ULL((u64)AVE_H264_FRAME_BUF_1080P * p->mbs, AVE_MBS_1080P), + SZ_16K); + frame_buf = max_t(size_t, frame_buf, AVE_H264_FRAME_BUF_1080P); + for (unsigned int s = 0; s < p->slots; s++) { + ret |= ave_alloc(ave, &sess->slot[s].coded, p->coded_cap, 0); + ret |= ave_alloc(ave, &sess->slot[s].meta, AVE_META_BYTES, 0); + ret |= ave_alloc(ave, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : frame_buf, + hevc ? 0 : nomap); + } + return ret ? -ENOMEM : 0; +} + +static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) +{ + ave_free(ave, &sess->hdr); + for (unsigned int i = 0; i < AVE_MAX_REFS; i++) { + ave_free(ave, &sess->ref_a[i]); + ave_free(ave, &sess->ref_b[i]); + ave_free(ave, &sess->ref_c[i]); + ave_free(ave, &sess->ref_d[i]); + } + ave_free(ave, &sess->mb); + ave_free(ave, &sess->row_a); + ave_free(ave, &sess->row_b); + ave_free(ave, &sess->fixed); + for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) { + ave_free(ave, &sess->tile[i]); + ave_free(ave, &sess->tile_aux[i]); + } + for (unsigned int i = 0; i < 2; i++) + ave_free(ave, &sess->hevc_extra[i]); + for (unsigned int s = 0; s < AVE_MAX_SLOTS; s++) { + ave_free(ave, &sess->slot[s].coded); + ave_free(ave, &sess->slot[s].meta); + ave_free(ave, &sess->slot[s].aux); + } +} + +/* Work out the parameters and allocate the buffers; no firmware access. */ +struct ave_session *ave_sess_create(struct ave_ctx *ctx) +{ + struct ave_session *sess; + int ret; + + sess = kzalloc(sizeof(*sess), GFP_KERNEL); + if (!sess) + return ERR_PTR(-ENOMEM); + ret = ave_sess_params(ctx, &sess->p); + if (!ret) + ret = ave_sess_alloc(ctx->ave, sess); + if (ret) { + ave_sess_free_bufs(ctx->ave, sess); + kfree(sess); + return ERR_PTR(ret); + } + for (unsigned int s = 0; s < sess->p.slots; s++) + ave_sess_prepare_slot(sess, s); + dev_dbg(ctx->ave->dev, "session %ux%u codec %u profile %u level %u, %u slots, %u refs\n", + sess->p.width, sess->p.height, sess->p.codec, sess->p.profile, sess->p.level, + sess->p.slots, sess->p.refs); + return sess; +} + +/* + * Free the buffers. A firmware that stopped answering may still write to + * them, and a freed DART range is soon handed out again, so in that case + * they stay allocated (until reboot, like the firmware itself). + */ +void ave_sess_destroy(struct ave_ctx *ctx) +{ + struct ave_session *sess = ctx->sess; + + if (!sess) + return; + if (ctx->ave->fw_state == AVE_FW_BROKEN) + dev_warn(ctx->ave->dev, "keeping the buffers of a failed session\n"); + else + ave_sess_free_bufs(ctx->ave, sess); + kfree(sess); + ctx->sess = NULL; +} + +int ave_sess_free_slot(struct ave_session *sess) +{ + for (unsigned int s = 0; s < sess->p.slots; s++) + if (!sess->slot[s].busy && !sess->slot[s].done) + return s; + return -EBUSY; +} + +/* + * Clear what the firmware writes, so the output can be measured: the whole + * metadata buffer, the first slice header and (H.264) the part of the coded + * buffer the previous frame used. + */ +void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot) +{ + struct ave_slot *s = &sess->slot[slot]; + + memset(s->meta.cpu, 0, s->meta.size); + if (sess->p.codec == AVE_CODEC_HEVC) { + memset(s->aux.cpu, 0, AVE_SHDR_ENTRY_BYTES); + } else if (s->coded_dirty) { + memset(s->coded.cpu, 0, min(s->coded_dirty, s->coded.size)); + s->coded_dirty = 0; + } +} + +/* A frame reply: +0x10 our client, +0x20 the slot's subtype, +0x40 status. */ +void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len) +{ + struct ave_session *sess = ctx->sess; + u64 client = get_unaligned_le64(msg + AVE_REPLY_CLIENT); + u32 subtype = get_unaligned_le32(msg + AVE_REPLY_SUBTYPE); + u32 slot = subtype - AVE_PROC_SUBTYPE; + struct ave_slot *s; + + if (!sess || len < AVE_REPLY_PROCESS_BYTES || client != sess->client || + slot >= sess->p.slots || !sess->slot[slot].busy) { + dev_warn(ctx->ave->dev, "stray frame reply (client %llu subtype %#x len %u)\n", + client, subtype, len); + return; + } + s = &sess->slot[slot]; + s->status = get_unaligned_le32(msg + AVE_REPLY_STATUS); + if (s->status) + dev_dbg(ctx->ave->dev, "frame in slot %u: status %#x\n", slot, s->status); + s->busy = false; + s->done = true; + sess->busy--; + sess->done_fifo[(sess->done_head + sess->done_count) % AVE_MAX_SLOTS] = slot; + sess->done_count++; + atomic_inc(&ctx->ready); +} + +/* + * Copy Annex B data from @src to @dst until the data ends: NAL units never + * hold three zero bytes in a row, and the buffer was cleared before the + * firmware wrote it, so a run of @zero_run zero bytes marks the end. + * Returns the length or -ENOSPC. + */ +static ssize_t ave_copy_annexb(u8 *dst, size_t room, const u8 *src, size_t cap, + unsigned int zero_run) +{ + size_t pos = 0, end = 0; + unsigned int zeros = 0; + + while (pos < cap) { + size_t chunk = min3((size_t)SZ_16K, cap - pos, room - pos); + + if (!chunk) + return -ENOSPC; + memcpy(dst + pos, src + pos, chunk); + for (size_t i = pos; i < pos + chunk; i++) { + if (dst[i]) { + end = i + 1; + zeros = 0; + } else if (end && ++zeros >= zero_run) { + return end; + } + } + pos += chunk; + } + return end; +} + +static bool ave_starts_annexb(const u8 *p, size_t len) +{ + return (len >= 3 && !p[0] && !p[1] && p[2] == 1) || + (len >= 4 && !p[0] && !p[1] && !p[2] && p[3] == 1); +} + +/* NAL unit types in an Annex B buffer: is there a key frame / parameter set? */ +static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *params) +{ + for (size_t i = 0; i + 3 < len; i++) { + u32 type; + + if (p[i] || p[i + 1] || p[i + 2] != 1) + continue; + type = hevc ? (p[i + 3] >> 1) & 0x3f : p[i + 3] & 0x1f; + if (hevc) { + *key |= type >= 16 && type <= 21; + *params |= type >= 32 && type <= 34; + } else { + *key |= type == 5; + *params |= type == 7; + } + i += 2; + } +} + +static void ave_sess_dump(struct ave_ctx *ctx, struct ave_slot *s, const char *why) +{ + struct ave_session *sess = ctx->sess; + + if (sess->warned) + return; + sess->warned = true; + dev_warn(ctx->ave->dev, "frame %u output: %s (enable dynamic debug for a dump)\n", + ctx->dst_sequence, why); + print_hex_dump_debug("ave coded: ", DUMP_PREFIX_OFFSET, 16, 1, s->coded.cpu, 128, false); + print_hex_dump_debug("ave meta: ", DUMP_PREFIX_OFFSET, 16, 4, s->meta.cpu, 0x200, false); + if (sess->p.codec == AVE_CODEC_HEVC) + print_hex_dump_debug("ave meta params: ", DUMP_PREFIX_OFFSET, 16, 1, + s->meta.cpu + AVE_META_HEVC_PARAMS, 128, false); +} + +/* + * Assemble the access unit of @slot into @dst. + * + * HEVC: the firmware puts the parameter sets in the metadata buffer, the + * slice header in the first slice-header entry and the slice data in the + * coded buffer; metadata words give the two lengths. + * + * H.264: where the length is reported is not known yet, so the coded buffer + * is read as Annex B until the data ends. + */ +int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, + bool *keyframe) +{ + struct ave_session *sess = ctx->sess; + struct ave_slot *s = &sess->slot[slot]; + bool hevc = sess->p.codec == AVE_CODEC_HEVC; + u8 *out = vb2_plane_vaddr(&dst->vb2_buf, 0); + size_t room = vb2_plane_size(&dst->vb2_buf, 0); + bool params = false; + ssize_t len = 0; + + *keyframe = false; + if (!out) + return -EFAULT; + if (s->status) + return -EIO; + + if (hevc) { + const u8 *meta = s->meta.cpu; + u32 shdr = get_unaligned_le32(meta + AVE_META_HEVC_SHDR); + u32 data = get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD); + ssize_t ps = 0; + + if (ave_starts_annexb(meta + AVE_META_HEVC_PARAMS, + AVE_META_BYTES - AVE_META_HEVC_PARAMS)) { + ps = ave_copy_annexb(out, room, meta + AVE_META_HEVC_PARAMS, + AVE_META_BYTES - AVE_META_HEVC_PARAMS, 16); + if (ps < 0) + return ps; + } + if (shdr > AVE_SHDR_ENTRY_BYTES || data > sess->p.coded_cap || + ps + shdr + data > room) { + dev_err(ctx->ave->dev, "bad HEVC output sizes %u/%u\n", shdr, data); + ave_sess_dump(ctx, s, "bad sizes"); + return -EPROTO; + } + if (get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD2) != data) + dev_dbg(ctx->ave->dev, "HEVC payload words differ: %u/%u\n", data, + get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD2)); + memcpy(out + ps, s->aux.cpu, shdr); + memcpy(out + ps + shdr, s->coded.cpu, data); + len = ps + shdr + data; + } else { + len = ave_copy_annexb(out, room, s->coded.cpu, s->coded.size, 256); + if (len < 0) { + dev_err(ctx->ave->dev, "coded frame larger than the CAPTURE buffer\n"); + return len; + } + s->coded_dirty = len + 256; + } + + if (!len) { + ave_sess_dump(ctx, s, "empty"); + return -ENODATA; + } + if (!ave_starts_annexb(out, len)) + ave_sess_dump(ctx, s, "not Annex B"); + ave_scan_nals(out, len, hevc, keyframe, ¶ms); + if (!ctx->dst_sequence && !params) + ave_sess_dump(ctx, s, "first frame has no parameter sets"); + vb2_set_plane_payload(&dst->vb2_buf, 0, len); + return 0; +} diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 410b86d20b28b..2807d8e3041f1 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -16,18 +16,19 @@ #include #include "ave.h" +#include "ave-regs.h" #define AVE_NAME "apple-ave" /* - * Frame size limits and input stride alignment. Provisional: conservative - * values the hardware is known to cover (up to the 2408x1506 panel and 1080p - * and 4K class streams); refined once the firmware limits are described. + * Frame size limits and input stride alignment: the range the encoder was + * seen working at (320x240 up to the 2408x1506 panel), with a little room. + * Larger sizes need buffer sizes nobody has checked yet. */ -#define AVE_MIN_W 64 -#define AVE_MIN_H 64 -#define AVE_MAX_W 4096 -#define AVE_MAX_H 4096 +#define AVE_MIN_W 320 +#define AVE_MIN_H 240 +#define AVE_MAX_W 2560 +#define AVE_MAX_H 1600 #define AVE_STEP 16 #define AVE_STRIDE_ALIGN 64 #define AVE_DEFAULT_W 1920 @@ -90,8 +91,8 @@ static void ave_fill_coded(struct v4l2_pix_format_mplane *f) if (!ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), f->pixelformat)) f->pixelformat = ave_coded_fmts[0]; ave_clamp_size(&f->width, &f->height); - /* Room for an intra frame at low compression plus headers. */ - min = max_t(u32, f->width * f->height * 3 / 4, SZ_256K); + /* The firmware's coded-data capacity plus parameter sets and headers. */ + min = ALIGN(3 * ALIGN(f->width, 32) * ALIGN(f->height, 16) / 2, SZ_16K) + SZ_64K; f->num_planes = 1; f->plane_fmt[0].bytesperline = 0; f->plane_fmt[0].sizeimage = clamp_t(u32, f->plane_fmt[0].sizeimage, min, SZ_32M); @@ -298,25 +299,39 @@ static int ave_s_parm(struct file *file, void *priv, struct v4l2_streamparm *a) return 0; } +/* + * Drain: frames still inside the firmware (B-frames) only come out after a + * flush, so the driver tracks the drain point itself instead of using the + * m2m helpers, which would mark the next CAPTURE buffer last right away. + */ static int ave_encoder_cmd(struct file *file, void *priv, struct v4l2_encoder_cmd *ec) { struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; int ret; ret = v4l2_m2m_ioctl_try_encoder_cmd(file, priv, ec); if (ret) return ret; - if (!vb2_is_streaming(v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx)) || - !vb2_is_streaming(v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx))) + if (!vb2_is_streaming(v4l2_m2m_get_src_vq(m2m)) || + !vb2_is_streaming(v4l2_m2m_get_dst_vq(m2m))) return 0; - ret = v4l2_m2m_ioctl_encoder_cmd(file, priv, ec); - if (ret) - return ret; - if (ec->cmd == V4L2_ENC_CMD_STOP && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) - v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); - if (ec->cmd == V4L2_ENC_CMD_START && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) - vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx)); + if (ec->cmd == V4L2_ENC_CMD_STOP) { + if (ctx->draining || ctx->stopped) + return 0; + ctx->drain_src = v4l2_m2m_last_src_buf(m2m); + ctx->draining = true; + if (!ctx->drain_src) + WRITE_ONCE(ctx->flush_pending, true); + } else { + if (ctx->draining && !ctx->stopped) + return -EBUSY; + ctx->draining = false; + ctx->stopped = false; + vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(m2m)); + } + v4l2_m2m_try_schedule(m2m); return 0; } @@ -374,7 +389,7 @@ static int ave_s_ctrl(struct v4l2_ctrl *ctrl) ctx->bitrate_mode = ctrl->val; break; case V4L2_CID_MPEG_VIDEO_BITRATE: - ctx->bitrate = ctrl->val; + WRITE_ONCE(ctx->bitrate, ctrl->val); break; case V4L2_CID_MPEG_VIDEO_BITRATE_PEAK: ctx->bitrate_peak = ctrl->val; @@ -383,7 +398,10 @@ static int ave_s_ctrl(struct v4l2_ctrl *ctrl) ctx->gop_size = ctrl->val; break; case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME: - ctx->force_key_frame = true; + WRITE_ONCE(ctx->force_key_frame, true); + break; + case V4L2_CID_MPEG_VIDEO_B_FRAMES: + ctx->b_frames = ctrl->val; break; case V4L2_CID_MPEG_VIDEO_H264_PROFILE: ctx->h264_profile = ctrl->val; @@ -391,18 +409,12 @@ static int ave_s_ctrl(struct v4l2_ctrl *ctrl) case V4L2_CID_MPEG_VIDEO_H264_LEVEL: ctx->h264_level = ctrl->val; break; - case V4L2_CID_MPEG_VIDEO_H264_I_PERIOD: - ctx->h264_i_period = ctrl->val; - break; case V4L2_CID_MPEG_VIDEO_H264_MIN_QP: ctx->h264_min_qp = ctrl->val; break; case V4L2_CID_MPEG_VIDEO_H264_MAX_QP: ctx->h264_max_qp = ctrl->val; break; - case V4L2_CID_MPEG_VIDEO_HEVC_PROFILE: - ctx->hevc_profile = ctrl->val; - break; case V4L2_CID_MPEG_VIDEO_HEVC_LEVEL: ctx->hevc_level = ctrl->val; break; @@ -413,7 +425,7 @@ static int ave_s_ctrl(struct v4l2_ctrl *ctrl) ctx->hevc_max_qp = ctrl->val; break; case V4L2_CID_MPEG_VIDEO_HEADER_MODE: - case V4L2_CID_MPEG_VIDEO_B_FRAMES: + case V4L2_CID_MPEG_VIDEO_HEVC_PROFILE: case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT: break; default: @@ -432,19 +444,27 @@ static int ave_init_ctrls(struct ave_ctx *ctx) const struct v4l2_ctrl_ops *ops = &ave_ctrl_ops; struct v4l2_ctrl *c; - v4l2_ctrl_handler_init(hdl, 18); + v4l2_ctrl_handler_init(hdl, 17); + /* Constant quality needs a quality-to-QP mapping that is not known yet. */ v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_MODE, - V4L2_MPEG_VIDEO_BITRATE_MODE_CQ, - 0, V4L2_MPEG_VIDEO_BITRATE_MODE_VBR); - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE, 100000, 200000000, 1, + V4L2_MPEG_VIDEO_BITRATE_MODE_CBR, + BIT(V4L2_MPEG_VIDEO_BITRATE_MODE_CQ), + V4L2_MPEG_VIDEO_BITRATE_MODE_VBR); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE, 100000, 100000000, 1, + 4000000); + /* In VBR mode the peak caps the data rate over any one second. */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 100000, 200000000, 1, 8000000); - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 100000, 400000000, 1, - 16000000); - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, 65535, 1, 60); + /* 0 means no limit; the firmware then uses 120 frames. */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, 65535, 1, 30); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); - /* No frame reordering until reference handling is described. */ - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 0, 1, 0); + /* + * The firmware either uses up to three B-frames (reordering them + * itself) or none. Baseline and intra-only H.264 never have any; + * HEVC always has them for now. + */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 3, 3, 3); v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEADER_MODE, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, BIT(V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE), @@ -456,24 +476,26 @@ static int ave_init_ctrls(struct ave_ctx *ctx) BIT(V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE) | BIT(V4L2_MPEG_VIDEO_H264_PROFILE_MAIN) | BIT(V4L2_MPEG_VIDEO_H264_PROFILE_HIGH)), - V4L2_MPEG_VIDEO_H264_PROFILE_HIGH); + V4L2_MPEG_VIDEO_H264_PROFILE_MAIN); + /* A minimum: the driver raises it to what size, rate and bit rate need. */ v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_LEVEL, V4L2_MPEG_VIDEO_H264_LEVEL_5_2, 0, - V4L2_MPEG_VIDEO_H264_LEVEL_4_1); - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD, 0, 65535, 1, 0); + V4L2_MPEG_VIDEO_H264_LEVEL_1_0); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MIN_QP, 0, 51, 1, 0); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MAX_QP, 0, 51, 1, 51); v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_PROFILE, - V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_10, 0, + V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN, 0, V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN); v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_LEVEL, V4L2_MPEG_VIDEO_HEVC_LEVEL_5_1, 0, - V4L2_MPEG_VIDEO_HEVC_LEVEL_4_1); + V4L2_MPEG_VIDEO_HEVC_LEVEL_1); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MIN_QP, 0, 51, 1, 0); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP, 0, 51, 1, 51); - c = v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, 1); + /* The firmware holds up to five frames; one more keeps it fed. */ + c = v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, + AVE_MAX_SLOTS + 1); if (c) c->flags |= V4L2_CTRL_FLAG_READ_ONLY; @@ -496,6 +518,10 @@ static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned i const struct v4l2_pix_format_mplane *f = V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; + /* Frames stay with the firmware until encoded; keep one spare. */ + if (V4L2_TYPE_IS_OUTPUT(vq->type) && + vb2_get_num_buffers(vq) + *nbufs < AVE_MAX_SLOTS + 1) + *nbufs = AVE_MAX_SLOTS + 1 - vb2_get_num_buffers(vq); if (*nplanes) return (*nplanes != 1 || sizes[0] < f->plane_fmt[0].sizeimage) ? -EINVAL : 0; *nplanes = 1; @@ -519,17 +545,8 @@ static int ave_buf_prepare(struct vb2_buffer *vb) static void ave_buf_queue(struct vb2_buffer *vb) { struct ave_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); - struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); - if (V4L2_TYPE_IS_CAPTURE(vb->type) && vb2_is_streaming(vb->vb2_queue) && - v4l2_m2m_dst_buf_is_last(ctx->fh.m2m_ctx)) { - vb2_set_plane_payload(vb, 0, 0); - vbuf->sequence = ctx->dst_sequence++; - v4l2_m2m_last_buffer_done(ctx->fh.m2m_ctx, vbuf); - v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); - return; - } - v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf); + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, to_vb2_v4l2_buffer(vb)); } static void ave_return_bufs(struct ave_ctx *ctx, struct vb2_queue *q, @@ -547,30 +564,95 @@ static void ave_return_bufs(struct ave_ctx *ctx, struct vb2_queue *q, } } +/* Sessions */ + +/* Caller holds hw_lock. */ +static void ave_session_close(struct ave_ctx *ctx, bool flush) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + + lockdep_assert_held(&ave->hw_lock); + if (!sess || ave->active != ctx) + return; + if (sess->fw_open && ave->fw_state == AVE_FW_RUNNING) { + /* Complete is a barrier: answered after every frame before it. */ + if (flush && sess->frames && !ave_cmd_complete(ctx, AVE_DRAIN_COMPLETES)) + ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); + if (ave->fw_state == AVE_FW_RUNNING) + ave_cmd_close(ctx); + } + sess->fw_open = false; + ave->active = NULL; + ave_fw_halt(ave); +} + +/* Return frames the firmware never answered; then drop the session. */ +static void ave_session_free(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + + if (!sess) + return; + for (unsigned int i = 0; i < sess->p.slots; i++) { + struct ave_slot *s = &sess->slot[i]; + + if (s->src) + v4l2_m2m_buf_done(s->src, VB2_BUF_STATE_ERROR); + s->src = NULL; + } + atomic_set(&ctx->ready, 0); + ave_sess_destroy(ctx); + pm_runtime_mark_last_busy(ave->dev); + pm_runtime_put_autosuspend(ave->dev); +} + +/* Boot the firmware if needed, then Open and Start a session. */ static int ave_session_begin(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; - enum ave_codec codec = ave_ctx_codec(ctx); + struct ave_session *sess; int ret; - ret = pm_runtime_resume_and_get(ave->dev); + if (ctx->sess) + return 0; + ret = ave_fw_templates(ave, ave_ctx_codec(ctx)); if (ret) return ret; + if (READ_ONCE(ave->active)) + return -EBUSY; + sess = ave_sess_create(ctx); + if (IS_ERR(sess)) + return PTR_ERR(sess); + + ret = pm_runtime_resume_and_get(ave->dev); + if (ret) { + ctx->sess = sess; + ave_sess_destroy(ctx); + return ret; + } mutex_lock(&ave->hw_lock); + ctx->sess = sess; + if (ave->active) { + ret = -EBUSY; + goto out; + } ret = ave_fw_ensure_running(ave); + if (ret) + goto out; + ave->active = ctx; + ret = ave_cmd_open(ctx); if (!ret) - ret = ave_cmd_open(ctx, codec); - if (!ret) - ret = ave_cmd_start(ctx, codec); - if (ret && ctx->session_open) - ave_cmd_finish(ctx, codec); + ret = ave_cmd_start(ctx); + if (ret) + ave_session_close(ctx, false); +out: mutex_unlock(&ave->hw_lock); - if (ret) { - dev_dbg(ave->dev, "session start failed: %d\n", ret); - pm_runtime_mark_last_busy(ave->dev); - pm_runtime_put_autosuspend(ave->dev); + dev_err(ave->dev, "cannot start an encoding session: %d\n", ret); + ave_session_free(ctx); } return ret; } @@ -579,14 +661,19 @@ static void ave_session_end(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; - cancel_work_sync(&ctx->run_work); + cancel_delayed_work_sync(&ctx->poll_work); mutex_lock(&ave->hw_lock); - if (ave->fw_state == AVE_FW_RUNNING) - ave_cmd_finish(ctx, ave_ctx_codec(ctx)); - ctx->session_open = false; + ave_session_close(ctx, true); mutex_unlock(&ave->hw_lock); - pm_runtime_mark_last_busy(ave->dev); - pm_runtime_put_autosuspend(ave->dev); + ave_session_free(ctx); +} + +static void ave_reset_drain(struct ave_ctx *ctx) +{ + ctx->draining = false; + WRITE_ONCE(ctx->flush_pending, false); + ctx->stopped = false; + ctx->drain_src = NULL; } static int ave_start_streaming(struct vb2_queue *q, unsigned int count) @@ -597,14 +684,14 @@ static int ave_start_streaming(struct vb2_queue *q, unsigned int count) v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx); int ret = 0; - v4l2_m2m_update_start_streaming_state(ctx->fh.m2m_ctx, q); if (V4L2_TYPE_IS_OUTPUT(q->type)) ctx->src_sequence = 0; else ctx->dst_sequence = 0; + ave_reset_drain(ctx); - /* The session starts once both queues stream. */ - if (vb2_is_streaming(other) && !ctx->session_open) + /* The session starts once both queues stream; errors show up here. */ + if (vb2_is_streaming(other)) ret = ave_session_begin(ctx); if (ret) ave_return_bufs(ctx, q, VB2_BUF_STATE_QUEUED); @@ -615,12 +702,9 @@ static void ave_stop_streaming(struct vb2_queue *q) { struct ave_ctx *ctx = vb2_get_drv_priv(q); - if (ctx->session_open) - ave_session_end(ctx); + ave_session_end(ctx); + ave_reset_drain(ctx); ave_return_bufs(ctx, q, VB2_BUF_STATE_ERROR); - v4l2_m2m_update_stop_streaming_state(ctx->fh.m2m_ctx, q); - if (V4L2_TYPE_IS_OUTPUT(q->type) && v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) - v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); } static const struct vb2_ops ave_qops = { @@ -661,50 +745,266 @@ static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *d return vb2_queue_init(dst); } -/* Jobs: one raw frame in, one coded frame out. */ +/* + * Jobs. A job delivers finished frames to CAPTURE buffers and sends at most + * one new frame. Up to five frames are inside the firmware at once; their + * answers are collected by the job and, while frames are outstanding, by a + * short poller, since the firmware interrupt is not wired up. + */ + +#define AVE_POLL_MS 2 + +static int ave_job_ready(void *priv) +{ + struct ave_ctx *ctx = priv; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + + if (atomic_read(&ctx->ready) || READ_ONCE(ctx->flush_pending)) + return 1; + if (ctx->stopped || (ctx->draining && !ctx->drain_src)) + return 0; + return v4l2_m2m_num_src_bufs_ready(m2m) > 0; +} + +/* + * Hand finished frames, in bitstream order, to CAPTURE buffers. Returns + * true when the frame that ends a drain went out (with the LAST flag). + */ +static bool ave_deliver(struct ave_ctx *ctx) +{ + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + struct ave_session *sess = ctx->sess; + bool last = false; + + while (sess && sess->done_count) { + unsigned int slot = sess->done_fifo[sess->done_head]; + struct ave_slot *s = &sess->slot[slot]; + struct vb2_v4l2_buffer *dst; + enum vb2_buffer_state state = VB2_BUF_STATE_DONE; + bool key = false; + + dst = v4l2_m2m_dst_buf_remove(m2m); + if (!dst) + break; + if (ave_sess_output(ctx, slot, dst, &key)) { + vb2_set_plane_payload(&dst->vb2_buf, 0, 0); + state = VB2_BUF_STATE_ERROR; + } + v4l2_m2m_buf_copy_metadata(s->src, dst); + dst->flags &= ~(V4L2_BUF_FLAG_KEYFRAME | V4L2_BUF_FLAG_PFRAME | + V4L2_BUF_FLAG_BFRAME | V4L2_BUF_FLAG_LAST); + dst->flags |= key ? V4L2_BUF_FLAG_KEYFRAME : V4L2_BUF_FLAG_PFRAME; + dst->sequence = ctx->dst_sequence++; + + sess->done_head = (sess->done_head + 1) % AVE_MAX_SLOTS; + sess->done_count--; + atomic_dec(&ctx->ready); + s->done = false; + ave_sess_prepare_slot(sess, slot); + + /* The last frame of a drain carries the LAST flag. */ + if (READ_ONCE(ctx->flush_pending) && !sess->fw_open && !sess->done_count) { + dst->flags |= V4L2_BUF_FLAG_LAST; + last = true; + } + v4l2_m2m_buf_done(s->src, state); + s->src = NULL; + v4l2_m2m_buf_done(dst, state); + } + return last; +} + +static bool ave_slot_answered(struct ave_dev *ave, void *arg) +{ + struct ave_session *sess = arg; + + return sess->busy < sess->p.slots || sess->done_count; +} + +static bool ave_frame_answered(struct ave_dev *ave, void *arg) +{ + struct ave_slot *s = arg; + + return !s->busy; +} + +/* Send the next OUTPUT frame. Caller holds hw_lock. */ +static int ave_encode_one(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + struct vb2_v4l2_buffer *src; + bool last; + int slot, ret; + + slot = ave_sess_free_slot(sess); + if (slot < 0 && sess->busy == sess->p.slots) { + /* Every slot is inside the firmware: wait for one to come back. */ + ret = ave_ipc_pump(ave, AVE_FRAME_WAIT_MS, ave_slot_answered, sess); + if (ret) + return ret; + ave_deliver(ctx); + slot = ave_sess_free_slot(sess); + } + if (slot < 0) + return 0; /* waiting for CAPTURE buffers */ + + src = v4l2_m2m_src_buf_remove(m2m); + if (!src) + return 0; + last = src == ctx->drain_src; + src->sequence = ctx->src_sequence++; + + ret = ave_cmd_process(ctx, slot, src, READ_ONCE(ctx->bitrate), + xchg(&ctx->force_key_frame, false)); + if (ret) { + v4l2_m2m_buf_done(src, VB2_BUF_STATE_ERROR); + return ret; + } + + /* Like macOS: the first frame goes alone, then a Complete. */ + if (sess->frames == 1) { + ret = ave_ipc_pump(ave, AVE_FRAME_WAIT_MS, ave_frame_answered, &sess->slot[slot]); + if (!ret) + ret = ave_cmd_complete(ctx, 1); + if (!ret) + ret = ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); + if (ret) + return ret; + } else { + ret = ave_ipc_pump(ave, 0, NULL, NULL); + if (ret) + return ret; + } + + if (last) { + ctx->drain_src = NULL; + WRITE_ONCE(ctx->flush_pending, true); + } + return 0; +} + +/* + * End of a drain: flush and close the session, deliver what is left and + * finish with a LAST buffer. Caller holds hw_lock; returns true once the + * LAST buffer is out. + */ +static bool ave_finish_drain(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + struct vb2_v4l2_buffer *dst; + + if (sess && sess->fw_open) + ave_session_close(ctx, true); + if (sess && sess->busy) { + /* The run job fails the session. */ + dev_err(ave->dev, "%u frames never came back from the firmware\n", sess->busy); + return false; + } + if (sess && sess->done_count) + return ave_deliver(ctx); /* false: waiting for CAPTURE buffers */ + + /* Nothing left to deliver: an empty buffer carries the LAST flag. */ + dst = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!dst) + return false; + vb2_set_plane_payload(&dst->vb2_buf, 0, 0); + dst->flags |= V4L2_BUF_FLAG_LAST; + dst->sequence = ctx->dst_sequence++; + v4l2_m2m_buf_done(dst, VB2_BUF_STATE_DONE); + return true; +} + +/* The firmware or the session failed: return everything, fail the queues. */ +static void ave_job_fail(struct ave_ctx *ctx, int err) +{ + struct ave_dev *ave = ctx->ave; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + + dev_err(ave->dev, "encoding failed: %d\n", err); + mutex_lock(&ave->hw_lock); + ave_session_close(ctx, false); + mutex_unlock(&ave->hw_lock); + ave_session_free(ctx); + ave_return_bufs(ctx, v4l2_m2m_get_src_vq(m2m), VB2_BUF_STATE_ERROR); + ave_return_bufs(ctx, v4l2_m2m_get_dst_vq(m2m), VB2_BUF_STATE_ERROR); + vb2_queue_error(v4l2_m2m_get_src_vq(m2m)); + vb2_queue_error(v4l2_m2m_get_dst_vq(m2m)); + ave_reset_drain(ctx); +} static void ave_run_work(struct work_struct *work) { struct ave_ctx *ctx = container_of(work, struct ave_ctx, run_work); struct ave_dev *ave = ctx->ave; - struct vb2_v4l2_buffer *src, *dst; - enum vb2_buffer_state state = VB2_BUF_STATE_DONE; - bool keyframe = false; - u32 bytes = 0; - int ret; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + bool eos = false, poll = false; + int ret = 0; - src = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); - dst = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + /* After a drain, the next frame starts a new session. */ + if (!ctx->sess && !READ_ONCE(ctx->flush_pending) && v4l2_m2m_num_src_bufs_ready(m2m)) + ret = ave_session_begin(ctx); mutex_lock(&ave->hw_lock); - ret = ave_cmd_process(ctx, ave_ctx_codec(ctx), src, dst, &bytes, &keyframe); - ctx->force_key_frame = false; + if (!ret && ctx->sess && (ctx->sess->dead || ave->active != ctx)) + ret = -EIO; + if (!ret) { + eos = ave_deliver(ctx); + if (!eos && !READ_ONCE(ctx->flush_pending) && ctx->sess && + v4l2_m2m_num_src_bufs_ready(m2m) && v4l2_m2m_num_dst_bufs_ready(m2m)) + ret = ave_encode_one(ctx); + } + if (!ret && !eos) { + eos = ave_deliver(ctx); + if (!eos && READ_ONCE(ctx->flush_pending)) + eos = ave_finish_drain(ctx); + /* Frames outstanding need a running firmware. */ + if (!eos && ctx->sess && ctx->sess->busy && ave->fw_state != AVE_FW_RUNNING) + ret = -EIO; + poll = !ret && !eos && ctx->sess && ctx->sess->busy; + } mutex_unlock(&ave->hw_lock); + if (ret) { - dev_dbg(ave->dev, "frame %u failed: %d\n", ctx->src_sequence, ret); - state = VB2_BUF_STATE_ERROR; - bytes = 0; + ave_job_fail(ctx, ret); + } else if (eos) { + ave_session_free(ctx); + ctx->draining = false; + WRITE_ONCE(ctx->flush_pending, false); + ctx->stopped = true; + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); + } else if (poll) { + mod_delayed_work(ave->wq, &ctx->poll_work, msecs_to_jiffies(AVE_POLL_MS)); } + pm_runtime_mark_last_busy(ave->dev); + v4l2_m2m_job_finish(ave->m2m_dev, m2m); +} - src = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx); - dst = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); - src->sequence = ctx->src_sequence++; - dst->sequence = ctx->dst_sequence++; - v4l2_m2m_buf_copy_metadata(src, dst); - dst->flags &= ~(V4L2_BUF_FLAG_KEYFRAME | V4L2_BUF_FLAG_PFRAME | V4L2_BUF_FLAG_LAST); - dst->flags |= keyframe ? V4L2_BUF_FLAG_KEYFRAME : V4L2_BUF_FLAG_PFRAME; - vb2_set_plane_payload(&dst->vb2_buf, 0, bytes); +/* Collect frame answers while frames are outstanding and no job runs. */ +static void ave_poll_work(struct work_struct *work) +{ + struct ave_ctx *ctx = container_of(to_delayed_work(work), struct ave_ctx, poll_work); + struct ave_dev *ave = ctx->ave; + bool again = false, fail = false; - if (v4l2_m2m_is_last_draining_src_buf(ctx->fh.m2m_ctx, src)) { - dst->flags |= V4L2_BUF_FLAG_LAST; - v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); - v4l2_m2m_mark_stopped(ctx->fh.m2m_ctx); + mutex_lock(&ave->hw_lock); + if (ctx->sess && ave->active == ctx && !ctx->sess->dead) { + fail = ave_ipc_pump(ave, 0, NULL, NULL) != 0; + again = !fail && ctx->sess->busy; + } else if (ctx->sess && ctx->sess->busy) { + fail = true; } + mutex_unlock(&ave->hw_lock); - v4l2_m2m_buf_done(src, state); - v4l2_m2m_buf_done(dst, state); - pm_runtime_mark_last_busy(ave->dev); - v4l2_m2m_job_finish(ave->m2m_dev, ctx->fh.m2m_ctx); + /* Jobs run on the same ordered queue, so none runs right now. */ + if (fail) + ave_job_fail(ctx, -EIO); + else if (atomic_read(&ctx->ready)) + v4l2_m2m_try_schedule(ctx->fh.m2m_ctx); + if (again) + queue_delayed_work(ave->wq, &ctx->poll_work, msecs_to_jiffies(AVE_POLL_MS)); } static void ave_device_run(void *priv) @@ -716,6 +1016,7 @@ static void ave_device_run(void *priv) static const struct v4l2_m2m_ops ave_m2m_ops = { .device_run = ave_device_run, + .job_ready = ave_job_ready, }; /* File operations */ @@ -731,6 +1032,8 @@ static int ave_open(struct file *file) return -ENOMEM; ctx->ave = ave; INIT_WORK(&ctx->run_work, ave_run_work); + INIT_DELAYED_WORK(&ctx->poll_work, ave_poll_work); + atomic_set(&ctx->ready, 0); ave_reset_fmts(ctx); v4l2_fh_init(&ctx->fh, video_devdata(file)); @@ -743,6 +1046,8 @@ static int ave_open(struct file *file) ret = PTR_ERR(ctx->fh.m2m_ctx); goto err_ctrls; } + /* Finished frames can be delivered with no new OUTPUT buffer queued. */ + v4l2_m2m_set_src_buffered(ctx->fh.m2m_ctx, true); v4l2_fh_add(&ctx->fh, file); return 0; @@ -762,6 +1067,8 @@ static int ave_release(struct file *file) mutex_lock(&ave->vdev_lock); v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); mutex_unlock(&ave->vdev_lock); + cancel_delayed_work_sync(&ctx->poll_work); + cancel_work_sync(&ctx->run_work); v4l2_ctrl_handler_free(&ctx->hdl); v4l2_fh_del(&ctx->fh, file); v4l2_fh_exit(&ctx->fh); diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index 6421a49e7fdd5..20d0c6831f5ef 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -7,6 +7,7 @@ #ifndef _APPLE_AVE_H_ #define _APPLE_AVE_H_ +#include #include #include #include @@ -19,17 +20,37 @@ #include struct dev_pm_domain_list; +struct firmware; struct iommu_domain; -/* Coprocessor memory every firmware boot uses, all inside one allocation. */ +/* Frames the firmware can hold at once (Process "slots"). */ +#define AVE_MAX_SLOTS 5 + +/* + * Coprocessor memory every firmware boot uses, all inside one allocation. + * Host commands live in three areas: one large buffer for Start, small + * buffers for the scalar commands and one buffer per Process slot, so a + * command stays untouched until the firmware acknowledges it. + */ #define AVE_IPC_BYTES (20 * SZ_1M) #define AVE_IPC_CHANNEL_OFF 0x4000 #define AVE_IPC_HANDSHAKE_OFF 0x24000 #define AVE_IPC_SHARED_OFF 0x28000 #define AVE_IPC_SHARED_BYTES SZ_64K #define AVE_IPC_CMD_OFF 0x40000 -#define AVE_IPC_CMD_BYTES SZ_128K -#define AVE_IPC_END (AVE_IPC_CMD_OFF + AVE_IPC_CMD_BYTES) +#define AVE_IPC_CMD_BYTES 0x1c000 +#define AVE_IPC_SCALAR_OFF 0x5c000 +#define AVE_IPC_SCALAR_BYTES 0x100 +#define AVE_IPC_SCALAR_BUFS 16 +#define AVE_IPC_PROC_OFF 0x60000 +#define AVE_IPC_PROC_BYTES 0x10000 +#define AVE_IPC_END (AVE_IPC_PROC_OFF + AVE_MAX_SLOTS * AVE_IPC_PROC_BYTES) + +/* Command buffer ids, one bit each in ave_dev.unacked. */ +#define AVE_BUF_CMD 0 +#define AVE_BUF_SCALAR(i) (1 + (i)) +#define AVE_BUF_PROC(s) (1 + AVE_IPC_SCALAR_BUFS + (s)) +#define AVE_BUF_COUNT AVE_BUF_PROC(AVE_MAX_SLOTS) #define AVE_MSG_MAX SZ_4K @@ -42,6 +63,7 @@ struct ave_buf { void *cpu; dma_addr_t dma; size_t size; + unsigned long attrs; }; struct ave_fw_segment { @@ -61,11 +83,21 @@ struct ave_ring { }; enum ave_fw_state { - AVE_FW_OFF, + AVE_FW_OFF, /* coprocessor in reset */ AVE_FW_RUNNING, + AVE_FW_HALTED, /* told to halt; restarts through the boot path */ AVE_FW_BROKEN, }; +/* Session command images that only differ from the listed fields. */ +enum ave_tmpl { + AVE_TMPL_START, + AVE_TMPL_PROCESS, + AVE_TMPL_COUNT, +}; + +struct ave_ctx; + struct ave_dev { struct device *dev; void __iomem *codec; @@ -76,9 +108,14 @@ struct ave_dev { struct ave_fw_segment text; struct ave_fw_segment data; + u32 text_crc; void *data_pristine; bool data_dirty; + /* Protects tmpl[]; the images stay loaded until the driver goes. */ + struct mutex tmpl_lock; + const struct firmware *tmpl[2][AVE_TMPL_COUNT]; + /* Serialises every firmware and ring access. */ struct mutex hw_lock; enum ave_fw_state fw_state; @@ -91,6 +128,11 @@ struct ave_dev { struct ave_ring ring[2]; u64 next_token; u32 next_client; + u32 unacked; + u8 want[16]; + int reply_err; + unsigned int scalar_next; + struct ave_ctx *active; u8 msg[AVE_MSG_MAX]; u32 msg_len; @@ -101,6 +143,69 @@ struct ave_dev { struct workqueue_struct *wq; }; +/* Encoding parameters, fixed for the lifetime of a session. */ +struct ave_params { + enum ave_codec codec; + u32 width, height; /* visible picture */ + u32 cw, ch; /* coded size, multiples of 16 */ + u32 stride; /* NV12 line stride, both planes */ + u32 mbs; /* 16x16 macroblocks */ + u32 fps_num, fps_den; /* frames per second = num / den */ + u32 fps_int; /* the same, rounded */ + u32 slots; + u32 refs; /* max_num_ref_frames + 1 */ + bool bframes; + bool intra_only; + u32 gop; + u32 rc_mode; + u32 bitrate; + u32 limit_bytes; /* 0: no data-rate limit */ + s32 min_qp; + u32 max_qp; + u32 profile; /* H.264: 2, 4, 6; HEVC: 1 */ + u32 level; /* firmware level enum, 1-based */ + bool high; + u32 coded_cap; +}; + +struct ave_slot { + struct ave_buf coded; + struct ave_buf meta; + struct ave_buf aux; /* H.264 per-frame buffer, HEVC slice headers */ + struct vb2_v4l2_buffer *src; + size_t coded_dirty; + u32 status; + bool busy; /* Process sent, reply pending */ + bool done; /* reply in, output not delivered yet */ +}; + +#define AVE_MAX_REFS 5 +#define AVE_TILE_BUFS 8 + +struct ave_session { + struct ave_params p; + struct ave_buf hdr; + struct ave_buf ref_a[AVE_MAX_REFS]; + struct ave_buf ref_b[AVE_MAX_REFS]; + struct ave_buf ref_c[AVE_MAX_REFS]; + struct ave_buf ref_d[AVE_MAX_REFS]; + struct ave_buf mb; + struct ave_buf row_a, row_b, fixed; + struct ave_buf tile[AVE_TILE_BUFS]; + struct ave_buf tile_aux[AVE_TILE_BUFS]; + struct ave_buf hevc_extra[2]; + struct ave_slot slot[AVE_MAX_SLOTS]; + u8 done_fifo[AVE_MAX_SLOTS]; + unsigned int done_head; + unsigned int done_count; + unsigned int busy; + u32 frames; /* Process commands sent */ + u32 client; + bool fw_open; /* Open sent, Close not yet */ + bool dead; /* firmware gone; buffers only */ + bool warned; +}; + struct ave_ctx { struct v4l2_fh fh; struct ave_dev *ave; @@ -109,24 +214,28 @@ struct ave_ctx { struct v4l2_pix_format_mplane dst_fmt; struct v4l2_fract timeperframe; struct work_struct run_work; + struct delayed_work poll_work; u32 bitrate_mode; u32 bitrate; u32 bitrate_peak; u32 gop_size; + u32 b_frames; u32 h264_profile; u32 h264_level; - u32 h264_i_period; u32 h264_min_qp; u32 h264_max_qp; - u32 hevc_profile; u32 hevc_level; u32 hevc_min_qp; u32 hevc_max_qp; bool force_key_frame; - u32 client; - bool session_open; + struct ave_session *sess; + atomic_t ready; /* outputs waiting for a CAPTURE buffer */ + bool draining; + bool flush_pending; + bool stopped; + struct vb2_v4l2_buffer *drain_src; u32 src_sequence; u32 dst_sequence; }; @@ -140,24 +249,42 @@ static inline struct ave_ctx *ave_file_to_ctx(struct file *file) int ave_fw_probe(struct ave_dev *ave); void ave_fw_remove(struct ave_dev *ave); int ave_fw_ensure_running(struct ave_dev *ave); +void ave_fw_halt(struct ave_dev *ave); int ave_fw_shutdown(struct ave_dev *ave); void ave_fw_lost(struct ave_dev *ave); -void ave_fw_resume(struct ave_dev *ave); +int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec); +int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs); +void ave_free(struct ave_dev *ave, struct ave_buf *b); /* ave-ipc.c */ int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]); -int ave_ipc_call(struct ave_dev *ave, const void *cmd, size_t len); -int ave_ipc_wait_msg(struct ave_dev *ave, unsigned int timeout_ms); +void *ave_ipc_cmd(struct ave_dev *ave, unsigned int buf, size_t len); +int ave_ipc_scalar_buf(struct ave_dev *ave); +int ave_ipc_wait_ack(struct ave_dev *ave, unsigned int buf); +int ave_ipc_submit(struct ave_dev *ave, unsigned int buf, size_t len, u64 timeout_us, + bool reply); +int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, + bool (*cond)(struct ave_dev *ave, void *arg), void *arg); +int ave_ipc_settle(struct ave_dev *ave, unsigned int wait_ms); /* ave-cmd.c */ int ave_cmd_config(struct ave_dev *ave); int ave_cmd_halt(struct ave_dev *ave); -int ave_cmd_open(struct ave_ctx *ctx, enum ave_codec codec); -int ave_cmd_start(struct ave_ctx *ctx, enum ave_codec codec); -int ave_cmd_process(struct ave_ctx *ctx, enum ave_codec codec, - struct vb2_v4l2_buffer *src, struct vb2_v4l2_buffer *dst, - u32 *bytesused, bool *keyframe); -int ave_cmd_finish(struct ave_ctx *ctx, enum ave_codec codec); +int ave_cmd_open(struct ave_ctx *ctx); +int ave_cmd_start(struct ave_ctx *ctx); +int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + u32 bitrate, bool force_key); +int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count); +int ave_cmd_close(struct ave_ctx *ctx); + +/* ave-sess.c */ +struct ave_session *ave_sess_create(struct ave_ctx *ctx); +void ave_sess_destroy(struct ave_ctx *ctx); +int ave_sess_free_slot(struct ave_session *sess); +void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len); +int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, + bool *keyframe); +void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); /* ave-v4l2.c */ int ave_v4l2_register(struct ave_dev *ave); From 731f241335e9fcfbcf26423a877c7e34946d963c Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 18:42:36 +1100 Subject: [PATCH 05/20] media: apple: ave: take the H.264 length from the metadata, fail stuck frames The first hardware run encoded frames, and showed three problems. H.264 output was guessed from the coded buffer. The firmware reports the coded length at metadata +0x120 (repeated at +0x12c) for H.264 as for HEVC, and the coded buffer holds only the slice NAL units. Use that length for both codecs, and take the parameter sets from the metadata buffer: at +0xb2a4 where HEVC has them, otherwise from the first SPS (H.264) or VPS (HEVC) start code found past the header words. Frames that the firmware stops answering left the client waiting forever: the poller had no time limit. When frames are outstanding that the firmware must answer (any frame without B-frames, four with them) and nothing comes back for 5 s, the firmware is marked stuck and the session fails, which completes every buffer with an error. The run also showed DART write faults right after Start. To find their source, the driver now logs its buffer map under dynamic debug and warns about any word in the Start or Process images that looks like an encoder DART address but is not one of the session's buffers, which would be a stale address left in a session image. The output dump now also covers the metadata around the parameter sets and every nonzero metadata line. Field layouts are facts from our private research (macOS traces) and our own hardware runs; no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-cmd.c | 3 + drivers/media/platform/apple/ave/ave-sess.c | 248 +++++++++++++++----- drivers/media/platform/apple/ave/ave-v4l2.c | 11 + drivers/media/platform/apple/ave/ave.h | 6 +- 4 files changed, 207 insertions(+), 61 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index 0b539fc201c3b..8514509b13bdf 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -470,6 +470,7 @@ int ave_cmd_start(struct ave_ctx *ctx) AVE_SESSION_ARG2_SETUP); ave_start_surfaces(sess, cmd); ave_start_config(&sess->p, cmd); + ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_START, 0); ret = ave_ipc_submit(ave, AVE_BUF_CMD, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, true); return ret ?: ave_ipc_settle(ave, AVE_START_WAIT_MS); @@ -609,6 +610,7 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff ave_put64(cmd, l->pts, pts); ave_put32(cmd, l->force_key, force_key); ave_put32(cmd, l->force_type, force_key ? AVE_PROC_FORCED_TYPE : 0); + ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_PROCESS, y); ret = ave_ipc_submit(ave, buf, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, false); if (ret) @@ -616,6 +618,7 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff s->src = src; s->busy = true; s->status = 0; + sess->progress = jiffies; sess->busy++; sess->frames++; return 0; diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 4d65dd09d732a..aa93d50fd1143 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -10,6 +10,7 @@ * Copyright The Aurora Silicon Contributors */ #include +#include #include #include #include @@ -31,10 +32,12 @@ #define AVE_HEVC_MAIN 1 #define AVE_META_BYTES 0xc000 -#define AVE_META_HEVC_PAYLOAD 0x120 +/* Metadata words: coded bytes (twice), HEVC slice-header bytes, parameter sets */ +#define AVE_META_PAYLOAD 0x120 #define AVE_META_HEVC_SHDR 0x128 -#define AVE_META_HEVC_PAYLOAD2 0x12c -#define AVE_META_HEVC_PARAMS 0xb2a4 +#define AVE_META_PAYLOAD2 0x12c +#define AVE_META_PARAMS 0xb2a4 +#define AVE_META_SCAN_FROM 0x200 #define AVE_SHDR_BYTES (256 * 0x400) #define AVE_SHDR_ENTRY_BYTES 0x400 #define AVE_CODED_MIN 0x3c000 @@ -248,6 +251,39 @@ static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) } } +/* Debug map of the session buffers, to match DART faults to a buffer. */ +static void ave_log_buf(struct ave_dev *ave, const char *name, int i, const struct ave_buf *b) +{ + if (b->cpu) + dev_dbg(ave->dev, "buf %s%d %pad +%#zx\n", name, i, &b->dma, b->size); +} + +static void ave_sess_log_map(struct ave_dev *ave, struct ave_session *sess) +{ + ave_log_buf(ave, "hdr", 0, &sess->hdr); + ave_log_buf(ave, "mb", 0, &sess->mb); + ave_log_buf(ave, "row_a", 0, &sess->row_a); + ave_log_buf(ave, "row_b", 0, &sess->row_b); + ave_log_buf(ave, "fixed", 0, &sess->fixed); + for (int i = 0; i < 2; i++) + ave_log_buf(ave, "hevc_extra", i, &sess->hevc_extra[i]); + for (int i = 0; i < AVE_MAX_REFS; i++) { + ave_log_buf(ave, "ref_a", i, &sess->ref_a[i]); + ave_log_buf(ave, "ref_b", i, &sess->ref_b[i]); + ave_log_buf(ave, "ref_c", i, &sess->ref_c[i]); + ave_log_buf(ave, "ref_d", i, &sess->ref_d[i]); + } + for (int i = 0; i < AVE_TILE_BUFS; i++) { + ave_log_buf(ave, "tile", i, &sess->tile[i]); + ave_log_buf(ave, "tile_aux", i, &sess->tile_aux[i]); + } + for (int i = 0; i < AVE_MAX_SLOTS; i++) { + ave_log_buf(ave, "coded", i, &sess->slot[i].coded); + ave_log_buf(ave, "meta", i, &sess->slot[i].meta); + ave_log_buf(ave, "aux", i, &sess->slot[i].aux); + } +} + /* Work out the parameters and allocate the buffers; no firmware access. */ struct ave_session *ave_sess_create(struct ave_ctx *ctx) { @@ -267,6 +303,7 @@ struct ave_session *ave_sess_create(struct ave_ctx *ctx) } for (unsigned int s = 0; s < sess->p.slots; s++) ave_sess_prepare_slot(sess, s); + ave_sess_log_map(ctx->ave, sess); dev_dbg(ctx->ave->dev, "session %ux%u codec %u profile %u level %u, %u slots, %u refs\n", sess->p.width, sess->p.height, sess->p.codec, sess->p.profile, sess->p.level, sess->p.slots, sess->p.refs); @@ -301,21 +338,16 @@ int ave_sess_free_slot(struct ave_session *sess) } /* - * Clear what the firmware writes, so the output can be measured: the whole - * metadata buffer, the first slice header and (H.264) the part of the coded - * buffer the previous frame used. + * Clear what the firmware reports through, so a stale value is never taken + * for a new one: the whole metadata buffer and the first slice header. */ void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot) { struct ave_slot *s = &sess->slot[slot]; memset(s->meta.cpu, 0, s->meta.size); - if (sess->p.codec == AVE_CODEC_HEVC) { + if (sess->p.codec == AVE_CODEC_HEVC) memset(s->aux.cpu, 0, AVE_SHDR_ENTRY_BYTES); - } else if (s->coded_dirty) { - memset(s->coded.cpu, 0, min(s->coded_dirty, s->coded.size)); - s->coded_dirty = 0; - } } /* A frame reply: +0x10 our client, +0x20 the slot's subtype, +0x40 status. */ @@ -334,6 +366,7 @@ void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len) return; } s = &sess->slot[slot]; + sess->progress = jiffies; s->status = get_unaligned_le32(msg + AVE_REPLY_STATUS); if (s->status) dev_dbg(ctx->ave->dev, "frame in slot %u: status %#x\n", slot, s->status); @@ -402,9 +435,29 @@ static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *p } } +/* Where in the metadata buffer do the parameter sets start, if anywhere? */ +static int ave_find_params(const u8 *meta, bool hevc) +{ + if (ave_starts_annexb(meta + AVE_META_PARAMS, AVE_META_BYTES - AVE_META_PARAMS)) + return AVE_META_PARAMS; + /* Not at the HEVC offset: look for an SPS (H.264) or VPS (HEVC). */ + for (int i = AVE_META_SCAN_FROM; i + 4 < AVE_META_BYTES; i++) { + u8 type; + + if (meta[i] || meta[i + 1] || meta[i + 2] != 1) + continue; + type = hevc ? (meta[i + 3] >> 1) & 0x3f : meta[i + 3] & 0x1f; + if (type == (hevc ? 32 : 7)) + return i > AVE_META_SCAN_FROM && !meta[i - 1] ? i - 1 : i; + } + return -ENOENT; +} + static void ave_sess_dump(struct ave_ctx *ctx, struct ave_slot *s, const char *why) { struct ave_session *sess = ctx->sess; + const u8 *meta = s->meta.cpu; + int ps; if (sess->warned) return; @@ -412,21 +465,27 @@ static void ave_sess_dump(struct ave_ctx *ctx, struct ave_slot *s, const char *w dev_warn(ctx->ave->dev, "frame %u output: %s (enable dynamic debug for a dump)\n", ctx->dst_sequence, why); print_hex_dump_debug("ave coded: ", DUMP_PREFIX_OFFSET, 16, 1, s->coded.cpu, 128, false); - print_hex_dump_debug("ave meta: ", DUMP_PREFIX_OFFSET, 16, 4, s->meta.cpu, 0x200, false); - if (sess->p.codec == AVE_CODEC_HEVC) - print_hex_dump_debug("ave meta params: ", DUMP_PREFIX_OFFSET, 16, 1, - s->meta.cpu + AVE_META_HEVC_PARAMS, 128, false); + print_hex_dump_debug("ave meta: ", DUMP_PREFIX_OFFSET, 16, 4, meta, 0x200, false); + print_hex_dump_debug("ave meta b200: ", DUMP_PREFIX_OFFSET, 16, 1, meta + 0xb200, + 0x200, false); + ps = ave_find_params(meta, sess->p.codec == AVE_CODEC_HEVC); + if (ps >= 0) { + dev_dbg(ctx->ave->dev, "parameter sets at metadata +%#x\n", ps); + print_hex_dump_debug("ave meta params: ", DUMP_PREFIX_OFFSET, 16, 1, meta + ps, + min(128, AVE_META_BYTES - ps), false); + } + /* Anything else the firmware wrote past the header words. */ + for (int i = AVE_META_SCAN_FROM; i < AVE_META_BYTES; i += 16) + if (memchr_inv(meta + i, 0, 16)) + dev_dbg(ctx->ave->dev, "metadata +%#x: %16ph\n", i, meta + i); } /* - * Assemble the access unit of @slot into @dst. - * - * HEVC: the firmware puts the parameter sets in the metadata buffer, the - * slice header in the first slice-header entry and the slice data in the - * coded buffer; metadata words give the two lengths. - * - * H.264: where the length is reported is not known yet, so the coded buffer - * is read as Annex B until the data ends. + * Assemble the access unit of @slot into @dst: the parameter sets the + * firmware leaves in the metadata buffer (when it emits them), for HEVC the + * slice header from the first slice-header entry, then the coded data. + * Metadata +0x120 (repeated at +0x12c) holds the coded length for both + * codecs, +0x128 the HEVC slice-header length. */ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, bool *keyframe) @@ -436,8 +495,12 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff bool hevc = sess->p.codec == AVE_CODEC_HEVC; u8 *out = vb2_plane_vaddr(&dst->vb2_buf, 0); size_t room = vb2_plane_size(&dst->vb2_buf, 0); + const u8 *meta = s->meta.cpu; + u32 data, data2, shdr = 0; bool params = false; - ssize_t len = 0; + ssize_t ps = 0; + size_t len; + int at; *keyframe = false; if (!out) @@ -445,44 +508,31 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff if (s->status) return -EIO; - if (hevc) { - const u8 *meta = s->meta.cpu; - u32 shdr = get_unaligned_le32(meta + AVE_META_HEVC_SHDR); - u32 data = get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD); - ssize_t ps = 0; - - if (ave_starts_annexb(meta + AVE_META_HEVC_PARAMS, - AVE_META_BYTES - AVE_META_HEVC_PARAMS)) { - ps = ave_copy_annexb(out, room, meta + AVE_META_HEVC_PARAMS, - AVE_META_BYTES - AVE_META_HEVC_PARAMS, 16); - if (ps < 0) - return ps; - } - if (shdr > AVE_SHDR_ENTRY_BYTES || data > sess->p.coded_cap || - ps + shdr + data > room) { - dev_err(ctx->ave->dev, "bad HEVC output sizes %u/%u\n", shdr, data); - ave_sess_dump(ctx, s, "bad sizes"); - return -EPROTO; - } - if (get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD2) != data) - dev_dbg(ctx->ave->dev, "HEVC payload words differ: %u/%u\n", data, - get_unaligned_le32(meta + AVE_META_HEVC_PAYLOAD2)); - memcpy(out + ps, s->aux.cpu, shdr); - memcpy(out + ps + shdr, s->coded.cpu, data); - len = ps + shdr + data; - } else { - len = ave_copy_annexb(out, room, s->coded.cpu, s->coded.size, 256); - if (len < 0) { - dev_err(ctx->ave->dev, "coded frame larger than the CAPTURE buffer\n"); - return len; - } - s->coded_dirty = len + 256; + data = get_unaligned_le32(meta + AVE_META_PAYLOAD); + data2 = get_unaligned_le32(meta + AVE_META_PAYLOAD2); + if (hevc) + shdr = get_unaligned_le32(meta + AVE_META_HEVC_SHDR); + if (data != data2) + dev_dbg(ctx->ave->dev, "coded length words differ: %u/%u\n", data, data2); + + at = ave_find_params(meta, hevc); + if (at >= 0) { + ps = ave_copy_annexb(out, room, meta + at, AVE_META_BYTES - at, 16); + if (ps < 0) + return ps; } - - if (!len) { - ave_sess_dump(ctx, s, "empty"); - return -ENODATA; + if (!data || data > sess->p.coded_cap || shdr > AVE_SHDR_ENTRY_BYTES || + ps + shdr + data > room) { + dev_err(ctx->ave->dev, "bad output sizes %u/%u (capacity %u, room %zu)\n", + shdr, data, sess->p.coded_cap, room); + ave_sess_dump(ctx, s, "bad sizes"); + return -EPROTO; } + if (hevc) + memcpy(out + ps, s->aux.cpu, shdr); + memcpy(out + ps + shdr, s->coded.cpu, data); + len = ps + shdr + data; + if (!ave_starts_annexb(out, len)) ave_sess_dump(ctx, s, "not Annex B"); ave_scan_nals(out, len, hevc, keyframe, ¶ms); @@ -491,3 +541,81 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff vb2_set_plane_payload(&dst->vb2_buf, 0, len); return 0; } + +/* + * Every 64-bit word in a command that looks like an encoder DART address + * should be one of ours. One that is not came from the session image (a + * stale macOS address) and makes the firmware write to an unmapped page. + * Reported once per session and command kind. + */ +static bool ave_buf_has(const struct ave_buf *b, u64 addr) +{ + return b->cpu && addr >= b->dma && addr < b->dma + PAGE_ALIGN(b->size); +} + +static bool ave_sess_owns(struct ave_session *sess, u64 addr) +{ + const struct ave_buf *fixed[] = { + &sess->hdr, &sess->mb, &sess->row_a, &sess->row_b, &sess->fixed, + &sess->hevc_extra[0], &sess->hevc_extra[1], + }; + + for (unsigned int i = 0; i < ARRAY_SIZE(fixed); i++) + if (ave_buf_has(fixed[i], addr)) + return true; + for (unsigned int i = 0; i < AVE_MAX_REFS; i++) + if (ave_buf_has(&sess->ref_a[i], addr) || ave_buf_has(&sess->ref_b[i], addr) || + ave_buf_has(&sess->ref_c[i], addr) || ave_buf_has(&sess->ref_d[i], addr)) + return true; + for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) + if (ave_buf_has(&sess->tile[i], addr) || ave_buf_has(&sess->tile_aux[i], addr)) + return true; + for (unsigned int s = 0; s < AVE_MAX_SLOTS; s++) + if (ave_buf_has(&sess->slot[s].coded, addr) || + ave_buf_has(&sess->slot[s].meta, addr) || + ave_buf_has(&sess->slot[s].aux, addr)) + return true; + return false; +} + +void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsigned int kind, + dma_addr_t input) +{ + struct ave_session *sess = ctx->sess; + struct ave_dev *ave = ctx->ave; + unsigned int bad = 0; + + if (sess->checked[kind]) + return; + sess->checked[kind] = true; + for (size_t off = AVE_CMD_HDR_BYTES; off + 8 <= len; off += 4) { + u64 v = get_unaligned_le64(cmd + off); + + /* The encoder's DART window: 0x100_0000_0000 + 256 MiB. */ + if ((v >> 28) != 0x1000) + continue; + if (ave_sess_owns(sess, v) || (input && v >= input && v < input + SZ_32M) || + (v >= ave->ipc.dma && v < ave->ipc.dma + ave->ipc.size)) + continue; + if (bad++ < 16) + dev_warn(ave->dev, "%s +%#zx holds %#llx, not a buffer of this session\n", + kind == AVE_TMPL_START ? "Start" : "Process", off, v); + } + if (bad) + dev_warn(ave->dev, "%u stale addresses in the %s image\n", bad, + kind == AVE_TMPL_START ? "Start" : "Process"); +} + +/* + * Frames are outstanding, nothing came back for a while, and the firmware + * has enough frames that it should have answered: without B-frames every + * frame is answered on its own; with B-frames, once four are inside (one + * reference plus three B) one of them must come out. + */ +bool ave_sess_stalled(struct ave_session *sess) +{ + unsigned int need = sess->p.bframes ? 4 : 1; + + return sess->busy >= need && + time_after(jiffies, sess->progress + msecs_to_jiffies(AVE_FRAME_WAIT_MS)); +} diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 2807d8e3041f1..36ef6ab0fde77 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -992,6 +992,17 @@ static void ave_poll_work(struct work_struct *work) mutex_lock(&ave->hw_lock); if (ctx->sess && ave->active == ctx && !ctx->sess->dead) { fail = ave_ipc_pump(ave, 0, NULL, NULL) != 0; + if (!fail && ave_sess_stalled(ctx->sess)) { + /* + * The firmware holds frames it should have answered. + * Treat it as stuck, so the session fails now instead + * of leaving the client waiting. + */ + dev_err(ave->dev, "%u frames unanswered for %u ms; firmware is stuck\n", + ctx->sess->busy, AVE_FRAME_WAIT_MS); + ave->fw_state = AVE_FW_BROKEN; + fail = true; + } again = !fail && ctx->sess->busy; } else if (ctx->sess && ctx->sess->busy) { fail = true; diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index 20d0c6831f5ef..e8d433edcb64f 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -173,7 +173,6 @@ struct ave_slot { struct ave_buf meta; struct ave_buf aux; /* H.264 per-frame buffer, HEVC slice headers */ struct vb2_v4l2_buffer *src; - size_t coded_dirty; u32 status; bool busy; /* Process sent, reply pending */ bool done; /* reply in, output not delivered yet */ @@ -204,6 +203,8 @@ struct ave_session { bool fw_open; /* Open sent, Close not yet */ bool dead; /* firmware gone; buffers only */ bool warned; + bool checked[2]; /* Start/Process images checked for stale addresses */ + unsigned long progress; /* jiffies of the last frame sent or answered */ }; struct ave_ctx { @@ -285,6 +286,9 @@ void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len); int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, bool *keyframe); void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); +void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsigned int kind, + dma_addr_t input); +bool ave_sess_stalled(struct ave_session *sess); /* ave-v4l2.c */ int ave_v4l2_register(struct ave_dev *ave); From 46af9cb5733dc06b33d1365c107f26a7fca2bd7d Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 19:29:35 +1100 Subject: [PATCH 06/20] media: apple: ave: map working buffers with room behind them The second 1080p H.264 run (P frames only) still had DART write faults right after the first frame was encoded, from 0x7800 to at least 0xa000 bytes past the end of the first +0x390 buffer, and the firmware stopped answering after that frame. The size the driver puts in Start for that group is the one macOS reports, so the firmware writes beyond the size it is given and macOS must map more than that. Keep the reported sizes, but map the +0x390 group at one and a half times its size and every other working buffer with an eighth (at least 64 KiB) more. Buffers now carry their mapped size separately, and the debug buffer map shows both. The stale address check no longer reports pointers into the firmware image itself (the Start image points at the TEXT segment, which sits at the same address as on macOS). Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-fw.c | 15 +++-- drivers/media/platform/apple/ave/ave-ipc.c | 1 + drivers/media/platform/apple/ave/ave-sess.c | 63 +++++++++++++++------ drivers/media/platform/apple/ave/ave.h | 5 +- 4 files changed, 61 insertions(+), 23 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index 62eb494c149b7..f895b1bd06fd5 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -177,21 +177,23 @@ static void ave_fw_restore_data(struct ave_dev *ave) void ave_free(struct ave_dev *ave, struct ave_buf *b) { if (b->cpu) - dma_free_attrs(ave->dev, PAGE_ALIGN(b->size), b->cpu, b->dma, b->attrs); + dma_free_attrs(ave->dev, b->alloc, b->cpu, b->dma, b->attrs); b->cpu = NULL; } /* * Firmware-visible memory, zeroed and 16 KiB aligned in the encoder's DART. - * @size is what the firmware is told; the allocation is page rounded. + * @size is what the firmware is told; @alloc (at least @size) is mapped. */ -int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs) +int ave_alloc_padded(struct ave_dev *ave, struct ave_buf *b, size_t size, size_t alloc, + unsigned long attrs) { if (b->cpu) return 0; b->size = size; + b->alloc = PAGE_ALIGN(max(size, alloc)); b->attrs = attrs; - b->cpu = dma_alloc_attrs(ave->dev, PAGE_ALIGN(size), &b->dma, GFP_KERNEL, attrs); + b->cpu = dma_alloc_attrs(ave->dev, b->alloc, &b->dma, GFP_KERNEL, attrs); if (!b->cpu) return -ENOMEM; if (!IS_ALIGNED(b->dma, SZ_16K)) { @@ -201,6 +203,11 @@ int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long return 0; } +int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs) +{ + return ave_alloc_padded(ave, b, size, size, attrs); +} + /* * Start and Process are mostly constant for a firmware build. Those * constant words come from per-build image files named after the TEXT diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c index d316edca01b3c..93178824a3604 100644 --- a/drivers/media/platform/apple/ave/ave-ipc.c +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -146,6 +146,7 @@ int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]) return -EPROTO; if (!ave->heap.cpu) { ave->heap.size = PAGE_ALIGN(hello[3]); + ave->heap.alloc = ave->heap.size; ave->heap.cpu = dma_alloc_coherent(ave->dev, ave->heap.size, &ave->heap.dma, GFP_KERNEL); if (!ave->heap.cpu) diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index aa93d50fd1143..4b44f0512cb7a 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -175,6 +175,24 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) return 0; } +/* + * The firmware writes past the sizes it is given. The first 1080p run + * wrote 0x7800..0x8a00 bytes (and maybe more; DART faults are rate + * limited) beyond the end of the first +0x390 buffer, so the sizes stay as + * macOS reports them but the mappings get room behind them: that group is + * mapped at 1.5 times its size, every other working buffer with an eighth + * (at least 64 KiB) more. + */ +static size_t ave_pad(size_t size) +{ + return size + max_t(size_t, SZ_64K, ALIGN(size / 8, SZ_16K)); +} + +static int ave_work_buf(struct ave_dev *ave, struct ave_buf *b, size_t size) +{ + return ave_alloc_padded(ave, b, size, ave_pad(size), DMA_ATTR_NO_KERNEL_MAPPING); +} + /* Working buffer sizes, all from the requested picture size. */ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) { @@ -193,34 +211,39 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ref_d = ALIGN(hevc ? 64 * w32 * h32 : 128 * p->mbs, SZ_16K); tile = hevc ? 2304 * w32 * h64 : ALIGN(w, 16) * ALIGN(h, 64); - ret |= ave_alloc(ave, &sess->hdr, AVE_HDR_BYTES, nomap); + ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc(ave, &sess->ref_a[i], ref_a, nomap); - ret |= ave_alloc(ave, &sess->ref_b[i], ref_b, nomap); - ret |= ave_alloc(ave, &sess->ref_c[i], ref_c, nomap); - ret |= ave_alloc(ave, &sess->ref_d[i], ref_d, nomap); + ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, + max(ref_a * 3 / 2, ave_pad(ref_a)), nomap); + ret |= ave_work_buf(ave, &sess->ref_b[i], ref_b); + ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); + ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); } - ret |= ave_alloc(ave, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), nomap); - ret |= ave_alloc(ave, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), nomap); - ret |= ave_alloc(ave, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), nomap); - ret |= ave_alloc(ave, &sess->fixed, SZ_16K, nomap); + ret |= ave_work_buf(ave, &sess->mb, ALIGN(82 * p->mbs, SZ_16K)); + ret |= ave_work_buf(ave, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K)); + ret |= ave_work_buf(ave, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K)); + ret |= ave_work_buf(ave, &sess->fixed, SZ_16K); for (unsigned int i = 0; i < (hevc ? 4 : AVE_TILE_BUFS); i++) { - ret |= ave_alloc(ave, &sess->tile[i], tile, nomap); - ret |= ave_alloc(ave, &sess->tile_aux[i], ALIGN(h, 128), nomap); + ret |= ave_work_buf(ave, &sess->tile[i], tile); + ret |= ave_work_buf(ave, &sess->tile_aux[i], ALIGN(h, 128)); } if (hevc) for (unsigned int i = 0; i < 2; i++) - ret |= ave_alloc(ave, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), nomap); + ret |= ave_work_buf(ave, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K)); /* Per-slot output; the size of the H.264 frame buffer is not known. */ frame_buf = ALIGN(DIV_ROUND_UP_ULL((u64)AVE_H264_FRAME_BUF_1080P * p->mbs, AVE_MBS_1080P), SZ_16K); frame_buf = max_t(size_t, frame_buf, AVE_H264_FRAME_BUF_1080P); for (unsigned int s = 0; s < p->slots; s++) { - ret |= ave_alloc(ave, &sess->slot[s].coded, p->coded_cap, 0); - ret |= ave_alloc(ave, &sess->slot[s].meta, AVE_META_BYTES, 0); - ret |= ave_alloc(ave, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : frame_buf, - hevc ? 0 : nomap); + ret |= ave_alloc_padded(ave, &sess->slot[s].coded, p->coded_cap, + ave_pad(p->coded_cap), 0); + ret |= ave_alloc_padded(ave, &sess->slot[s].meta, AVE_META_BYTES, + ave_pad(AVE_META_BYTES), 0); + if (hevc) + ret |= ave_alloc(ave, &sess->slot[s].aux, AVE_SHDR_BYTES, 0); + else + ret |= ave_work_buf(ave, &sess->slot[s].aux, frame_buf); } return ret ? -ENOMEM : 0; } @@ -255,7 +278,8 @@ static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) static void ave_log_buf(struct ave_dev *ave, const char *name, int i, const struct ave_buf *b) { if (b->cpu) - dev_dbg(ave->dev, "buf %s%d %pad +%#zx\n", name, i, &b->dma, b->size); + dev_dbg(ave->dev, "buf %s%d %pad +%#zx (mapped %#zx)\n", name, i, &b->dma, + b->size, b->alloc); } static void ave_sess_log_map(struct ave_dev *ave, struct ave_session *sess) @@ -550,7 +574,7 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff */ static bool ave_buf_has(const struct ave_buf *b, u64 addr) { - return b->cpu && addr >= b->dma && addr < b->dma + PAGE_ALIGN(b->size); + return b->cpu && addr >= b->dma && addr < b->dma + b->alloc; } static bool ave_sess_owns(struct ave_session *sess, u64 addr) @@ -597,6 +621,9 @@ void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsign if (ave_sess_owns(sess, v) || (input && v >= input && v < input + SZ_32M) || (v >= ave->ipc.dma && v < ave->ipc.dma + ave->ipc.size)) continue; + /* Pointers into the firmware image are the same as on macOS. */ + if (v >= ave->text.iova && v < ave->data.iova + ave->data.size) + continue; if (bad++ < 16) dev_warn(ave->dev, "%s +%#zx holds %#llx, not a buffer of this session\n", kind == AVE_TMPL_START ? "Start" : "Process", off, v); diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index e8d433edcb64f..eac0ba7901f63 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -62,7 +62,8 @@ enum ave_codec { struct ave_buf { void *cpu; dma_addr_t dma; - size_t size; + size_t size; /* what the firmware is told */ + size_t alloc; /* what is mapped, page rounded */ unsigned long attrs; }; @@ -255,6 +256,8 @@ int ave_fw_shutdown(struct ave_dev *ave); void ave_fw_lost(struct ave_dev *ave); int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec); int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs); +int ave_alloc_padded(struct ave_dev *ave, struct ave_buf *b, size_t size, size_t alloc, + unsigned long attrs); void ave_free(struct ave_dev *ave, struct ave_buf *b); /* ave-ipc.c */ From 1808be7c2b0ef5c1af13451aa489bf9eacde53b1 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 20:17:30 +1100 Subject: [PATCH 07/20] media: apple: ave: restart only from reset, flush held frames, free safe buffers The third hardware run encoded H.264 cleanly twice in a row, with a power cycle between the sessions. HEVC then hung its client for 20 minutes and broke every later session. ffmpeg's v4l2m2m encoder sends a frame and waits for its coded output before sending the next. With B-frames the firmware keeps frames until later ones arrive, so both sides waited, and the stuck-frame timeout only counted four or more held frames. Now, when frames are held and no input came for 50 ms, the driver sends a Complete, which makes the firmware code what it holds (macOS sends one after the first frame of every session). If those frames do not come back within 5 s, the session fails as before. The next session started before runtime suspend had powered the encoder down, and restarting the halted coprocessor while it kept power got no answer. A session now runtime suspends the encoder first when the firmware is halted, and a restart is refused while the coprocessor still runs, instead of trying it. A failed firmware boot marked the firmware broken for good and kept the buffers of the session that triggered it, so later sessions could not allocate. A failed boot now leaves the firmware halted, to be retried after a power cycle. Only buffers the firmware was given in a Start are kept when it stops answering. Field layouts are facts from our private research (macOS traces) and our own hardware runs; no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-cmd.c | 7 +++++- drivers/media/platform/apple/ave/ave-fw.c | 28 ++++++++++++++------- drivers/media/platform/apple/ave/ave-sess.c | 26 +++++++++++++++---- drivers/media/platform/apple/ave/ave-v4l2.c | 10 ++++++++ drivers/media/platform/apple/ave/ave.h | 4 +++ 5 files changed, 60 insertions(+), 15 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index 8514509b13bdf..0aa872b2d720f 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -473,7 +473,10 @@ int ave_cmd_start(struct ave_ctx *ctx) ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_START, 0); ret = ave_ipc_submit(ave, AVE_BUF_CMD, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, true); - return ret ?: ave_ipc_settle(ave, AVE_START_WAIT_MS); + if (ret) + return ret; + sess->handed = true; + return ave_ipc_settle(ave, AVE_START_WAIT_MS); } /* Process (id 8) */ @@ -619,6 +622,8 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff s->busy = true; s->status = 0; sess->progress = jiffies; + sess->last_sent = jiffies; + sess->nudged = false; sess->busy++; sess->frames++; return 0; diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index f895b1bd06fd5..bd434d508f874 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -284,11 +284,11 @@ static bool ave_fw_idle(struct ave_dev *ave) } /* - * After a Halt the firmware no longer runs, but the coprocessor is still - * set going if its power stayed on. A restart uses the normal boot path - * (macOS restarts the firmware for every session the same way): clear the - * run bit, as the boot sequence does first anyway, and the mailbox words - * left from the previous run, so the reset-state checks hold again. The + * After a Halt the firmware is restarted from a power cycle, as macOS does + * for every session: the encoder must have been runtime suspended in + * between, which puts the coprocessor back in reset. Hardware runs showed + * that a restart while it kept power gets no answer, so that is refused + * (-EAGAIN) and the state stays halted until the power really went. The * code base register must still be set: the driver never writes it. */ static int ave_fw_prepare_restart(struct ave_dev *ave) @@ -304,8 +304,8 @@ static int ave_fw_prepare_restart(struct ave_dev *ave) return 0; if (readl(ave->coproc + AVE_COPROC_CTRL)) { - writel(0, ave->coproc + AVE_COPROC_CTRL); - usleep_range(1000, 2000); + dev_warn(ave->dev, "coprocessor kept power after its halt; retry once it suspends\n"); + return -EAGAIN; } status = readl(ave->mbox + AVE_MBOX_STATUS); if (status) @@ -416,6 +416,7 @@ static int ave_fw_boot(struct ave_dev *ave) /* Caller holds hw_lock. */ int ave_fw_ensure_running(struct ave_dev *ave) { + enum ave_fw_state prev; int ret; lockdep_assert_held(&ave->hw_lock); @@ -429,9 +430,18 @@ int ave_fw_ensure_running(struct ave_dev *ave) break; } + /* + * A failed boot has no session buffers in the firmware's hands. If the + * coprocessor was set going, treat it as halted: the next attempt + * then waits for a power cycle. Otherwise nothing changed. + */ + prev = ave->fw_state; ret = ave_fw_boot(ave); - if (ret) - ave->fw_state = AVE_FW_BROKEN; + if (ret) { + ave->fw_state = readl(ave->coproc + AVE_COPROC_CTRL) ? AVE_FW_HALTED : prev; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; + } return ret; } diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 4b44f0512cb7a..99406f1e39013 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -335,9 +335,10 @@ struct ave_session *ave_sess_create(struct ave_ctx *ctx) } /* - * Free the buffers. A firmware that stopped answering may still write to - * them, and a freed DART range is soon handed out again, so in that case - * they stay allocated (until reboot, like the firmware itself). + * Free the buffers. A firmware that stopped answering in the middle of a + * session may still write to them, and a freed DART range is soon handed + * out again, so those stay allocated (until reboot, like the firmware + * itself). Buffers the firmware never saw (no Start sent) always go. */ void ave_sess_destroy(struct ave_ctx *ctx) { @@ -345,7 +346,7 @@ void ave_sess_destroy(struct ave_ctx *ctx) if (!sess) return; - if (ctx->ave->fw_state == AVE_FW_BROKEN) + if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) dev_warn(ctx->ave->dev, "keeping the buffers of a failed session\n"); else ave_sess_free_bufs(ctx->ave, sess); @@ -643,6 +644,21 @@ bool ave_sess_stalled(struct ave_session *sess) { unsigned int need = sess->p.bframes ? 4 : 1; - return sess->busy >= need && + return sess->busy && (sess->busy >= need || sess->nudged) && time_after(jiffies, sess->progress + msecs_to_jiffies(AVE_FRAME_WAIT_MS)); } + +/* + * With B-frames the firmware keeps frames until later ones arrive. A client + * that waits for each coded frame before sending the next (ffmpeg's + * v4l2m2m encoder does) would wait forever. When frames are held and no + * input came for a while, a Complete makes the firmware code what it holds, + * as macOS does after the first frame. + */ +#define AVE_STARVE_MS 50 + +bool ave_sess_starved(struct ave_session *sess) +{ + return sess->busy && !sess->nudged && sess->handed && + time_after(jiffies, sess->last_sent + msecs_to_jiffies(AVE_STARVE_MS)); +} diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 36ef6ab0fde77..6fd3224fbbb74 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -617,6 +617,9 @@ static int ave_session_begin(struct ave_ctx *ctx) if (ctx->sess) return 0; + /* A halted firmware restarts only from reset: let the power go first. */ + if (READ_ONCE(ave->fw_state) == AVE_FW_HALTED) + pm_runtime_suspend(ave->dev); ret = ave_fw_templates(ave, ave_ctx_codec(ctx)); if (ret) return ret; @@ -992,6 +995,13 @@ static void ave_poll_work(struct work_struct *work) mutex_lock(&ave->hw_lock); if (ctx->sess && ave->active == ctx && !ctx->sess->dead) { fail = ave_ipc_pump(ave, 0, NULL, NULL) != 0; + if (!fail && ave_sess_starved(ctx->sess) && + !v4l2_m2m_num_src_bufs_ready(ctx->fh.m2m_ctx)) { + dev_dbg(ave->dev, "no input, flushing %u held frames\n", ctx->sess->busy); + ctx->sess->nudged = true; + ctx->sess->progress = jiffies; + fail = ave_cmd_complete(ctx, 1) != 0; + } if (!fail && ave_sess_stalled(ctx->sess)) { /* * The firmware holds frames it should have answered. diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index eac0ba7901f63..010a8481ae3b3 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -205,7 +205,10 @@ struct ave_session { bool dead; /* firmware gone; buffers only */ bool warned; bool checked[2]; /* Start/Process images checked for stale addresses */ + bool handed; /* Start sent: the firmware knows the buffers */ + bool nudged; /* Complete sent because input stopped */ unsigned long progress; /* jiffies of the last frame sent or answered */ + unsigned long last_sent; /* jiffies of the last frame sent */ }; struct ave_ctx { @@ -292,6 +295,7 @@ void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsigned int kind, dma_addr_t input); bool ave_sess_stalled(struct ave_session *sess); +bool ave_sess_starved(struct ave_session *sess); /* ave-v4l2.c */ int ave_v4l2_register(struct ave_dev *ave); From 593f94d13c246dc2d2bca155434c754484ba4d1e Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 20:21:00 +1100 Subject: [PATCH 08/20] media: apple: ave: report raw frame sizes and cap the buffer counts v4l2-compliance failed the format tests: a stateful encoder must answer VIDIOC_ENUM_FRAMESIZES for every format it enumerates, and the driver only did for the coded ones. Report the same range for NV12. CREATE_BUFS with the queue maximum failed because 32 buffers of each kind do not fit the encoder's 256 MiB DART window next to a session. Allow 16 OUTPUT and 8 CAPTURE buffers. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-v4l2.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 6fd3224fbbb74..13da5a0e7d6ff 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -156,7 +156,8 @@ static int ave_enum_fmt_out(struct file *file, void *priv, struct v4l2_fmtdesc * static int ave_enum_framesizes(struct file *file, void *priv, struct v4l2_frmsizeenum *fs) { if (fs->index || - !ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), fs->pixel_format)) + (!ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), fs->pixel_format) && + !ave_fmt_listed(ave_raw_fmts, ARRAY_SIZE(ave_raw_fmts), fs->pixel_format))) return -EINVAL; fs->type = V4L2_FRMSIZE_TYPE_STEPWISE; fs->stepwise = (struct v4l2_frmsize_stepwise){ @@ -732,6 +733,8 @@ static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *d src->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; src->lock = &ctx->ave->vdev_lock; src->dev = ctx->ave->dev; + /* A full set of both queues must fit the encoder's 256 MiB DART window. */ + src->max_num_buffers = 16; ret = vb2_queue_init(src); if (ret) return ret; @@ -745,6 +748,7 @@ static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *d dst->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; dst->lock = &ctx->ave->vdev_lock; dst->dev = ctx->ave->dev; + dst->max_num_buffers = 8; return vb2_queue_init(dst); } From 01ce954b7f0d7635010c386a62b24c2dc3bb7b98 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 20:44:30 +1100 Subject: [PATCH 09/20] media: apple: ave: cap buffer counts in queue_setup The previous commit set max_num_buffers on both queues, which this kernel's vb2_core_queue_init() refuses below VB2_MAX_FRAME, so every open of the device failed. Leave the queue maximum alone and limit the count in queue_setup instead: 16 OUTPUT and 8 CAPTURE buffers, so a full set still fits the encoder's 256 MiB DART window. Fixes: 593f94d13c24 ("media: apple: ave: report raw frame sizes and cap the buffer counts") Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-v4l2.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 13da5a0e7d6ff..c081734e86151 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -33,6 +33,8 @@ #define AVE_STRIDE_ALIGN 64 #define AVE_DEFAULT_W 1920 #define AVE_DEFAULT_H 1080 +#define AVE_MAX_OUTPUT_BUFS 16 +#define AVE_MAX_CAPTURE_BUFS 8 static const u32 ave_coded_fmts[] = { V4L2_PIX_FMT_H264, @@ -518,11 +520,17 @@ static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned i struct ave_ctx *ctx = vb2_get_drv_priv(vq); const struct v4l2_pix_format_mplane *f = V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; + unsigned int have = vb2_get_num_buffers(vq); + unsigned int max = V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_MAX_OUTPUT_BUFS : + AVE_MAX_CAPTURE_BUFS; /* Frames stay with the firmware until encoded; keep one spare. */ - if (V4L2_TYPE_IS_OUTPUT(vq->type) && - vb2_get_num_buffers(vq) + *nbufs < AVE_MAX_SLOTS + 1) - *nbufs = AVE_MAX_SLOTS + 1 - vb2_get_num_buffers(vq); + if (V4L2_TYPE_IS_OUTPUT(vq->type) && have + *nbufs < AVE_MAX_SLOTS + 1) + *nbufs = AVE_MAX_SLOTS + 1 - have; + /* A full set of both queues must fit the encoder's 256 MiB DART window. */ + if (have >= max) + return -ENOMEM; + *nbufs = min(*nbufs, max - have); if (*nplanes) return (*nplanes != 1 || sizes[0] < f->plane_fmt[0].sizeimage) ? -EINVAL : 0; *nplanes = 1; @@ -733,8 +741,6 @@ static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *d src->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; src->lock = &ctx->ave->vdev_lock; src->dev = ctx->ave->dev; - /* A full set of both queues must fit the encoder's 256 MiB DART window. */ - src->max_num_buffers = 16; ret = vb2_queue_init(src); if (ret) return ret; @@ -748,7 +754,6 @@ static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *d dst->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; dst->lock = &ctx->ave->vdev_lock; dst->dev = ctx->ave->dev; - dst->max_num_buffers = 8; return vb2_queue_init(dst); } From 124be686eb7b65a56d77757bb7174cb1d02f448e Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 20:53:45 +1100 Subject: [PATCH 10/20] media: apple: ave: rate limit, memory budget, low-delay HEVC, restart retry Fixes from the fourth hardware round. Rate control: with only BITRATE set (ffmpeg's case) the one-second data-rate limit equalled the average rate, and the firmware then coded far below the requested rate. macOS always sets the limit to twice the average. BITRATE_PEAK now defaults to 0, which means exactly that; a client value is still used as given (and never below the average). Memory: H.264 at 2408x1506 ran out of room in the encoder's 256 MiB DART window. The DMA layer maps every buffer at a power-of-two alignment, so the size-proportional margins pushed buffers into the next size class. Margins are now 256 KiB on the +0x390 group and 64 KiB elsewhere, and the OUTPUT and CAPTURE queues get a fixed share of the window (64 and 32 MiB, counted at the mapped size, at least 6 OUTPUT and 2 CAPTURE buffers). HEVC with B_FRAMES set to 0, as ffmpeg sets it: the firmware still uses its B-frame structure, so each frame waited for the 50 ms no-input flush. Send a Complete right after every frame in that case instead. Restart: right after a session the coprocessor can still be powered when the next one starts, because the DARTs suspend a little later. Instead of failing with EAGAIN, cycle the power up to ten times, 100 ms apart. Formats: a huge OUTPUT bytesperline wrapped the size computation to 0 (v4l2-compliance). Clamp it to twice the maximum width. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- Documentation/admin-guide/media/apple-ave.rst | 8 ++- drivers/media/platform/apple/ave/ave-sess.c | 29 +++++--- drivers/media/platform/apple/ave/ave-v4l2.c | 67 ++++++++++++++++--- drivers/media/platform/apple/ave/ave.h | 1 + 4 files changed, 81 insertions(+), 24 deletions(-) diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index d8f9acabb2117..8ca9950dbea4b 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -52,7 +52,8 @@ runs at a time; a second one gets ``EBUSY``. Up to five frames are inside the firmware at once (three without B-frames). The firmware reorders B-frames itself, so CAPTURE buffers come back in bitstream order, each carrying the timestamp of the frame it codes. The driver -asks for at least six OUTPUT buffers. +asks for at least six OUTPUT buffers, and limits both queues so a full set fits +the encoder's 256 MiB DMA window (fewer buffers at larger sizes). Runtime PM uses a two second autosuspend delay. A firmware that stops answering keeps the encoder powered and is not used again until reboot. @@ -63,9 +64,10 @@ Limits Picture sizes from 320x240 to 2560x1600 (in steps of two), up to 240 frames per second. Rate control is VBR (with ``BITRATE_PEAK`` as a one-second data -rate cap) or CBR; constant quality is not offered. ``BITRATE`` may change +rate cap, twice the average when left at 0) or CBR; constant quality is not offered. ``BITRATE`` may change while streaming and applies from the next frame; ``FORCE_KEY_FRAME`` makes the next frame an IDR. H.264 uses either three B-frames or none (Baseline and -a GOP size of 1 never use them); HEVC always uses them for now. The level +a GOP size of 1 never use them). HEVC always uses them for now; with +``B_FRAMES`` set to 0 every frame is flushed through at once instead. The level control is a minimum: the driver raises it to what the size, frame rate and bit rate need. diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 99406f1e39013..f20c21570f4b8 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -150,6 +150,8 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) (!baseline && !p->intra_only && ctx->b_frames); p->refs = p->intra_only ? 1 : p->bframes ? 5 : 2; p->slots = p->bframes ? 5 : 3; + /* HEVC asked to go without B-frames: flush every frame instead. */ + p->low_delay = p->codec == AVE_CODEC_HEVC && !ctx->b_frames; p->gop = ctx->gop_size ?: 120; p->bitrate = ctx->bitrate; @@ -157,8 +159,14 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) p->rc_mode = AVE_RC_CBR; p->limit_bytes = 0; } else { + /* + * The data-rate limit macOS sets is twice the average rate over + * one second; a peak equal to the average starved the rate + * control on hardware. A peak of 0 means that default. + */ p->rc_mode = AVE_RC_VBR; - p->limit_bytes = max(ctx->bitrate_peak, ctx->bitrate) / 8; + p->limit_bytes = (ctx->bitrate_peak ? max(ctx->bitrate_peak, ctx->bitrate) : + 2ULL * ctx->bitrate) / 8; } level = ave_pick_level(p, p->codec == AVE_CODEC_H264 ? ctx->h264_level : @@ -176,16 +184,18 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) } /* - * The firmware writes past the sizes it is given. The first 1080p run - * wrote 0x7800..0x8a00 bytes (and maybe more; DART faults are rate - * limited) beyond the end of the first +0x390 buffer, so the sizes stay as - * macOS reports them but the mappings get room behind them: that group is - * mapped at 1.5 times its size, every other working buffer with an eighth - * (at least 64 KiB) more. + * The firmware writes past the sizes it is given. A 1080p run wrote up to + * about 0x8c00 bytes beyond the end of the first +0x390 buffer, so the + * sizes stay as macOS reports them but the mappings get room behind them: + * 256 KiB for that group, 64 KiB for every other working buffer. Larger + * margins cost more than they look: the DMA layer aligns each mapping to a + * power of two in the encoder's 256 MiB window. */ +#define AVE_REF_A_PAD SZ_256K + static size_t ave_pad(size_t size) { - return size + max_t(size_t, SZ_64K, ALIGN(size / 8, SZ_16K)); + return size + SZ_64K; } static int ave_work_buf(struct ave_dev *ave, struct ave_buf *b, size_t size) @@ -213,8 +223,7 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, - max(ref_a * 3 / 2, ave_pad(ref_a)), nomap); + ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ref_a + AVE_REF_A_PAD, nomap); ret |= ave_work_buf(ave, &sess->ref_b[i], ref_b); ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index c081734e86151..0868fc960c8ba 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -7,6 +7,8 @@ * * Copyright The Aurora Silicon Contributors */ +#include +#include #include #include @@ -33,8 +35,16 @@ #define AVE_STRIDE_ALIGN 64 #define AVE_DEFAULT_W 1920 #define AVE_DEFAULT_H 1080 -#define AVE_MAX_OUTPUT_BUFS 16 -#define AVE_MAX_CAPTURE_BUFS 8 +/* + * Buffer counts are limited so a full set fits the encoder's 256 MiB DART + * window next to a session: each queue may use this much, counting every + * buffer at the power-of-two size the DMA layer maps it with. + */ +#define AVE_OUTPUT_BUDGET SZ_64M +#define AVE_CAPTURE_BUDGET SZ_32M +#define AVE_MAX_QUEUE_BUFS 16 +#define AVE_RESTART_TRIES 10 +#define AVE_RESTART_WAIT_MS 100 static const u32 ave_coded_fmts[] = { V4L2_PIX_FMT_H264, @@ -76,7 +86,9 @@ static void ave_fill_raw(struct v4l2_pix_format_mplane *f) f->pixelformat = V4L2_PIX_FMT_NV12; ave_clamp_size(&f->width, &f->height); - stride = ALIGN(max_t(u32, f->width, f->plane_fmt[0].bytesperline), AVE_STRIDE_ALIGN); + /* Keep a client's stride, within reason, rounded to 64 bytes. */ + stride = ALIGN(clamp_t(u32, f->plane_fmt[0].bytesperline, f->width, 2 * AVE_MAX_W), + AVE_STRIDE_ALIGN); height = ALIGN(f->height, AVE_STEP); f->num_planes = 1; f->plane_fmt[0].bytesperline = stride; @@ -456,9 +468,11 @@ static int ave_init_ctrls(struct ave_ctx *ctx) V4L2_MPEG_VIDEO_BITRATE_MODE_VBR); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE, 100000, 100000000, 1, 4000000); - /* In VBR mode the peak caps the data rate over any one second. */ - v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 100000, 200000000, 1, - 8000000); + /* + * In VBR mode the peak caps the data rate over any one second; + * 0 means twice the average, as macOS sets it. + */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 0, 200000000, 1, 0); /* 0 means no limit; the firmware then uses 120 frames. */ v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, 65535, 1, 30); v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); @@ -521,8 +535,12 @@ static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned i const struct v4l2_pix_format_mplane *f = V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; unsigned int have = vb2_get_num_buffers(vq); - unsigned int max = V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_MAX_OUTPUT_BUFS : - AVE_MAX_CAPTURE_BUFS; + unsigned long slot = roundup_pow_of_two(PAGE_ALIGN(f->plane_fmt[0].sizeimage)); + unsigned int max = (V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_OUTPUT_BUDGET : + AVE_CAPTURE_BUDGET) / slot; + + max = clamp(max, V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_MAX_SLOTS + 1U : 2U, + AVE_MAX_QUEUE_BUFS); /* Frames stay with the firmware until encoded; keep one spare. */ if (V4L2_TYPE_IS_OUTPUT(vq->type) && have + *nbufs < AVE_MAX_SLOTS + 1) @@ -638,20 +656,39 @@ static int ave_session_begin(struct ave_ctx *ctx) if (IS_ERR(sess)) return PTR_ERR(sess); + ctx->sess = sess; ret = pm_runtime_resume_and_get(ave->dev); if (ret) { - ctx->sess = sess; ave_sess_destroy(ctx); return ret; } mutex_lock(&ave->hw_lock); - ctx->sess = sess; if (ave->active) { ret = -EBUSY; goto out; } - ret = ave_fw_ensure_running(ave); + /* + * A halted coprocessor restarts only from reset. Its power can take + * a moment to go after a suspend (the DARTs follow later), so cycle + * it again a few times before giving up. + */ + for (unsigned int i = 0; i < AVE_RESTART_TRIES; i++) { + ret = ave_fw_ensure_running(ave); + if (ret != -EAGAIN) + break; + mutex_unlock(&ave->hw_lock); + pm_runtime_put_sync_suspend(ave->dev); + msleep(AVE_RESTART_WAIT_MS); + ret = pm_runtime_resume_and_get(ave->dev); + mutex_lock(&ave->hw_lock); + if (ret) { + mutex_unlock(&ave->hw_lock); + ave_sess_destroy(ctx); + return ret; + } + ret = -EAGAIN; + } if (ret) goto out; ave->active = ctx; @@ -884,6 +921,14 @@ static int ave_encode_one(struct ave_ctx *ctx) ret = ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); if (ret) return ret; + } else if (sess->p.low_delay) { + /* Make the firmware code this frame now, without lookahead. */ + ret = ave_cmd_complete(ctx, 1); + sess->nudged = true; + if (!ret) + ret = ave_ipc_pump(ave, 0, NULL, NULL); + if (ret) + return ret; } else { ret = ave_ipc_pump(ave, 0, NULL, NULL); if (ret) diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index 010a8481ae3b3..018795a5ebfd8 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -157,6 +157,7 @@ struct ave_params { u32 refs; /* max_num_ref_frames + 1 */ bool bframes; bool intra_only; + bool low_delay; /* Complete after every frame */ u32 gop; u32 rc_mode; u32 bitrate; From 44596dca5fdbb353ba1f10403d31a72141291129 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:06:11 +1100 Subject: [PATCH 11/20] media: apple: ave: map the reconstruction buffers at their real size The +0x390 group holds a 4:2:0 picture and its size field only covers the luma part. macOS leaves 1.5 times that size behind each one (0x318000 behind the 0x20d000 buffer at 1080p), and with only 256 KiB of margin the firmware wrote chroma far into unmapped space (from +0x252000 on, with hundreds of thousands of faults suppressed) and stopped answering mid stream. Map 1.5 times the size plus the usual 64 KiB. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-sess.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index f20c21570f4b8..c84aa3bf7cff6 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -184,14 +184,15 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) } /* - * The firmware writes past the sizes it is given. A 1080p run wrote up to - * about 0x8c00 bytes beyond the end of the first +0x390 buffer, so the - * sizes stay as macOS reports them but the mappings get room behind them: - * 256 KiB for that group, 64 KiB for every other working buffer. Larger + * The firmware writes past the sizes it is given, so the sizes stay as + * macOS reports them but the mappings get room behind them. The +0x390 + * group holds a 4:2:0 picture whose size field covers luma only: macOS + * leaves 0x318000 bytes behind the 0x20d000 one at 1080p (1.5 times), and + * the firmware writes chroma there. Every other working buffer gets 64 KiB + * more (macOS leaves 0x28000 behind the 0x20000 +0x3a0 one). Larger * margins cost more than they look: the DMA layer aligns each mapping to a * power of two in the encoder's 256 MiB window. */ -#define AVE_REF_A_PAD SZ_256K static size_t ave_pad(size_t size) { @@ -223,7 +224,7 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ref_a + AVE_REF_A_PAD, nomap); + ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ave_pad(ref_a * 3 / 2), nomap); ret |= ave_work_buf(ave, &sess->ref_b[i], ref_b); ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); From b36da980a378f575149547b8c5a23ce4de248d3e Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:11:50 +1100 Subject: [PATCH 12/20] media: apple: ave: report buffer overruns, refuse sheared NV12 input Encoded streams look right but drift from the source picture over a GOP, and the firmware is known to write past the sizes it is given. A write that runs into a neighbouring mapped buffer raises no DART fault, so it would go unnoticed. Keep a kernel mapping of the working buffers and, when a session ends normally, report for each one how far past its size the firmware wrote, with a warning when it reached the end of the mapping. This costs a scan of the zeroed margins once per session. At 2408x1506, ffmpeg's v4l2m2m encoder produced a sheared picture: it copies NV12 with its own frame line size and ignores the bytesperline from S_FMT, which differs when the width is not a multiple of 64. The encoder needs the 64-byte aligned stride, so the driver now keeps the client's bytesused on OUTPUT buffers, recognises a frame laid out with a smaller stride from it, and fails the stream with a message naming both strides instead of encoding garbage. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-sess.c | 67 +++++++++++++++++++-- drivers/media/platform/apple/ave/ave-v4l2.c | 33 +++++++++- 2 files changed, 94 insertions(+), 6 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index c84aa3bf7cff6..3ea435de52c6a 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -199,9 +199,13 @@ static size_t ave_pad(size_t size) return size + SZ_64K; } +/* + * Working buffers keep a kernel mapping so that, when a session ends, the + * driver can see how far past each reported size the firmware wrote. + */ static int ave_work_buf(struct ave_dev *ave, struct ave_buf *b, size_t size) { - return ave_alloc_padded(ave, b, size, ave_pad(size), DMA_ATTR_NO_KERNEL_MAPPING); + return ave_alloc_padded(ave, b, size, ave_pad(size), 0); } /* Working buffer sizes, all from the requested picture size. */ @@ -212,7 +216,6 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) u32 w = p->width, h = p->height; u32 w16 = DIV_ROUND_UP(w, 16), w32 = DIV_ROUND_UP(w, 32); u32 h32 = DIV_ROUND_UP(h, 32), w64 = DIV_ROUND_UP(w, 64), h64 = DIV_ROUND_UP(h, 64); - const unsigned long nomap = DMA_ATTR_NO_KERNEL_MAPPING; size_t ref_a, ref_b, ref_c, ref_d, tile, frame_buf; int ret = 0; @@ -224,7 +227,7 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ave_pad(ref_a * 3 / 2), nomap); + ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ave_pad(ref_a * 3 / 2), 0); ret |= ave_work_buf(ave, &sess->ref_b[i], ref_b); ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); @@ -318,6 +321,57 @@ static void ave_sess_log_map(struct ave_dev *ave, struct ave_session *sess) } } +/* + * How far did the firmware write past the size it was given? The margin + * behind each buffer was zero when allocated. A write reaching the end of + * the mapping may have gone on into the next buffer. + */ +static void ave_check_tail(struct ave_dev *ave, const char *name, int i, + const struct ave_buf *b) +{ + const u8 *p = b->cpu; + size_t end; + + if (!p || b->attrs & DMA_ATTR_NO_KERNEL_MAPPING || b->alloc <= b->size) + return; + for (end = b->alloc; end > b->size && !p[end - 1]; end--) + ; + if (end == b->size) + return; + if (end == b->alloc) + dev_warn(ave->dev, "buf %s%d: firmware wrote to the end of its mapping (+%#zx, size %#zx)\n", + name, i, b->alloc, b->size); + else + dev_info(ave->dev, "buf %s%d: firmware wrote %#zx bytes past its size %#zx\n", + name, i, end - b->size, b->size); +} + +static void ave_sess_check_tails(struct ave_dev *ave, struct ave_session *sess) +{ + ave_check_tail(ave, "hdr", 0, &sess->hdr); + ave_check_tail(ave, "mb", 0, &sess->mb); + ave_check_tail(ave, "row_a", 0, &sess->row_a); + ave_check_tail(ave, "row_b", 0, &sess->row_b); + ave_check_tail(ave, "fixed", 0, &sess->fixed); + for (int i = 0; i < 2; i++) + ave_check_tail(ave, "hevc_extra", i, &sess->hevc_extra[i]); + for (int i = 0; i < AVE_MAX_REFS; i++) { + ave_check_tail(ave, "ref_a", i, &sess->ref_a[i]); + ave_check_tail(ave, "ref_b", i, &sess->ref_b[i]); + ave_check_tail(ave, "ref_c", i, &sess->ref_c[i]); + ave_check_tail(ave, "ref_d", i, &sess->ref_d[i]); + } + for (int i = 0; i < AVE_TILE_BUFS; i++) { + ave_check_tail(ave, "tile", i, &sess->tile[i]); + ave_check_tail(ave, "tile_aux", i, &sess->tile_aux[i]); + } + for (int i = 0; i < AVE_MAX_SLOTS; i++) { + ave_check_tail(ave, "coded", i, &sess->slot[i].coded); + ave_check_tail(ave, "meta", i, &sess->slot[i].meta); + ave_check_tail(ave, "aux", i, &sess->slot[i].aux); + } +} + /* Work out the parameters and allocate the buffers; no firmware access. */ struct ave_session *ave_sess_create(struct ave_ctx *ctx) { @@ -356,10 +410,13 @@ void ave_sess_destroy(struct ave_ctx *ctx) if (!sess) return; - if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) + if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) { dev_warn(ctx->ave->dev, "keeping the buffers of a failed session\n"); - else + } else { + if (sess->handed) + ave_sess_check_tails(ctx->ave, sess); ave_sess_free_bufs(ctx->ave, sess); + } kfree(sess); ctx->sess = NULL; } diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 0868fc960c8ba..07182d92592f4 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -564,7 +564,8 @@ static int ave_buf_prepare(struct vb2_buffer *vb) if (vb2_plane_size(vb, 0) < f->plane_fmt[0].sizeimage) return -EINVAL; - if (V4L2_TYPE_IS_OUTPUT(vb->type)) + /* Keep what the client filled in, so its layout can be checked. */ + if (V4L2_TYPE_IS_OUTPUT(vb->type) && !vb2_get_plane_payload(vb, 0)) vb2_set_plane_payload(vb, 0, f->plane_fmt[0].sizeimage); return 0; } @@ -877,6 +878,30 @@ static bool ave_frame_answered(struct ave_dev *ave, void *arg) return !s->busy; } +/* + * The encoder reads NV12 with the 64-byte aligned stride S_FMT reported. + * ffmpeg's v4l2m2m encoder copies frames with its own line size instead, + * which differs when the width is not a multiple of 64; the result is a + * sheared picture. Spot that from the payload size and refuse it. + */ +static int ave_check_src_layout(struct ave_ctx *ctx, struct vb2_v4l2_buffer *src) +{ + const struct ave_params *p = &ctx->sess->p; + u32 used = vb2_get_plane_payload(&src->vb2_buf, 0); + u32 rows = p->height + DIV_ROUND_UP(p->height, 2); + u32 stride; + + if (!used || used % rows) + return 0; + stride = used / rows; + if (stride >= p->stride || stride < p->width) + return 0; + dev_err(ctx->ave->dev, + "OUTPUT frame laid out with a %u byte stride, not the %u reported by S_FMT\n", + stride, p->stride); + return -EINVAL; +} + /* Send the next OUTPUT frame. Caller holds hw_lock. */ static int ave_encode_one(struct ave_ctx *ctx) { @@ -905,6 +930,12 @@ static int ave_encode_one(struct ave_ctx *ctx) last = src == ctx->drain_src; src->sequence = ctx->src_sequence++; + ret = ave_check_src_layout(ctx, src); + if (ret) { + v4l2_m2m_buf_done(src, VB2_BUF_STATE_ERROR); + return ret; + } + ret = ave_cmd_process(ctx, slot, src, READ_ONCE(ctx->bitrate), xchg(&ctx->force_key_frame, false)); if (ret) { From 502230fb6915bb4d4036559cdaca4cd31dfd32f5 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:26:00 +1100 Subject: [PATCH 13/20] media: apple: ave: size reconstruction buffers from measured writes The overrun report from the previous commit gave the same numbers in every session. At 1080p the firmware wrote 0x313604 bytes into the 0x20d000 +0x390 buffer and 0x26a6a into the 0x20000 +0x3a0 one; at 2408x1506 0x557e06 into 0x390000 and 0x4c77d into 0x40000. No other buffer was written past its size. So the +0x390 group is a 4:2:0 picture whose size field is the luma part: map 1.5 times it. Map the +0x3a0 group at 1.25 times its size (it is written up to 1.2 times, and macOS spaces it that way). Drop the margins on every other buffer, which saves DART window space. The session debug line now also shows the rate-control mode, bit rate, data-rate limit, GOP length and frame rate the firmware gets. Measurements are from our own hardware runs; no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-sess.c | 37 +++++++++------------ 1 file changed, 16 insertions(+), 21 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 3ea435de52c6a..c4eea08aa437e 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -184,28 +184,22 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) } /* - * The firmware writes past the sizes it is given, so the sizes stay as - * macOS reports them but the mappings get room behind them. The +0x390 - * group holds a 4:2:0 picture whose size field covers luma only: macOS - * leaves 0x318000 bytes behind the 0x20d000 one at 1080p (1.5 times), and - * the firmware writes chroma there. Every other working buffer gets 64 KiB - * more (macOS leaves 0x28000 behind the 0x20000 +0x3a0 one). Larger - * margins cost more than they look: the DMA layer aligns each mapping to a - * power of two in the encoder's 256 MiB window. + * Two groups hold more than their size fields say, measured on hardware + * at 1080p and 2408x1506: the +0x390 group is a 4:2:0 picture whose size + * covers luma only (the firmware writes up to 1.5 times it), and the + * +0x3a0 group is written up to 1.2 times its size (mapped at 1.25 times, + * as macOS spaces them). Every other buffer stays within its size; the + * mapping is only rounded up to whole pages. */ -static size_t ave_pad(size_t size) -{ - return size + SZ_64K; -} - /* * Working buffers keep a kernel mapping so that, when a session ends, the - * driver can see how far past each reported size the firmware wrote. + * driver can see how far past each reported size the firmware wrote (into + * the page rounding, or the room the two groups above get). */ static int ave_work_buf(struct ave_dev *ave, struct ave_buf *b, size_t size) { - return ave_alloc_padded(ave, b, size, ave_pad(size), 0); + return ave_alloc(ave, b, size, 0); } /* Working buffer sizes, all from the requested picture size. */ @@ -227,8 +221,8 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ave_pad(ref_a * 3 / 2), 0); - ret |= ave_work_buf(ave, &sess->ref_b[i], ref_b); + ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ref_a * 3 / 2, 0); + ret |= ave_alloc_padded(ave, &sess->ref_b[i], ref_b, ref_b * 5 / 4, 0); ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); } @@ -249,10 +243,8 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) SZ_16K); frame_buf = max_t(size_t, frame_buf, AVE_H264_FRAME_BUF_1080P); for (unsigned int s = 0; s < p->slots; s++) { - ret |= ave_alloc_padded(ave, &sess->slot[s].coded, p->coded_cap, - ave_pad(p->coded_cap), 0); - ret |= ave_alloc_padded(ave, &sess->slot[s].meta, AVE_META_BYTES, - ave_pad(AVE_META_BYTES), 0); + ret |= ave_alloc(ave, &sess->slot[s].coded, p->coded_cap, 0); + ret |= ave_alloc(ave, &sess->slot[s].meta, AVE_META_BYTES, 0); if (hevc) ret |= ave_alloc(ave, &sess->slot[s].aux, AVE_SHDR_BYTES, 0); else @@ -395,6 +387,9 @@ struct ave_session *ave_sess_create(struct ave_ctx *ctx) dev_dbg(ctx->ave->dev, "session %ux%u codec %u profile %u level %u, %u slots, %u refs\n", sess->p.width, sess->p.height, sess->p.codec, sess->p.profile, sess->p.level, sess->p.slots, sess->p.refs); + dev_dbg(ctx->ave->dev, "rate control mode %u, %u bit/s, limit %u bytes/s, gop %u, %u/%u fps\n", + sess->p.rc_mode, sess->p.bitrate, sess->p.limit_bytes, sess->p.gop, + sess->p.fps_num, sess->p.fps_den); return sess; } From 7962e096edb6864ce2f27ee1744a2bb91715d654 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:30:14 +1100 Subject: [PATCH 14/20] media: apple: ave: keep delivering after a drain, say why a stream fails A GStreamer run stopped at frame 240 with only "encoding failed: -5". A job refused to touch a session whose firmware it no longer owned, but a drain closes the session and halts the firmware first and only then hands out the frames still waiting for CAPTURE buffers. When those buffers came in later, the next job saw the closed session and failed the stream without a word. Only a session that is still open must own the firmware. Every other path that ends a stream with -EIO now logs its reason too: a command or wait on a firmware that is no longer running, a firmware that stopped answering in an earlier session, a failed Halt, a session ended by system sleep, frames outstanding without a firmware, and a frame the firmware answered with an error status. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-fw.c | 2 ++ drivers/media/platform/apple/ave/ave-ipc.c | 10 ++++++++-- drivers/media/platform/apple/ave/ave-sess.c | 5 ++++- drivers/media/platform/apple/ave/ave-v4l2.c | 18 ++++++++++++++++-- 4 files changed, 30 insertions(+), 5 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index bd434d508f874..e1f9a11118ed0 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -424,6 +424,7 @@ int ave_fw_ensure_running(struct ave_dev *ave) case AVE_FW_RUNNING: return 0; case AVE_FW_BROKEN: + dev_err(ave->dev, "firmware stopped answering earlier; reboot to use the encoder\n"); return -EIO; case AVE_FW_OFF: case AVE_FW_HALTED: @@ -457,6 +458,7 @@ void ave_fw_halt(struct ave_dev *ave) if (ave->fw_state != AVE_FW_RUNNING) return; if (ave_cmd_halt(ave)) { + dev_err(ave->dev, "cannot send Halt; firmware treated as stuck\n"); ave->fw_state = AVE_FW_BROKEN; return; } diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c index 93178824a3604..af067fb024e1e 100644 --- a/drivers/media/platform/apple/ave/ave-ipc.c +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -357,8 +357,11 @@ int ave_ipc_submit(struct ave_dev *ave, unsigned int buf, size_t len, u64 timeou int ret; lockdep_assert_held(&ave->hw_lock); - if (ave->fw_state != AVE_FW_RUNNING) + if (ave->fw_state != AVE_FW_RUNNING) { + dev_err_ratelimited(ave->dev, "command %u not sent: firmware state %d\n", op, + ave->fw_state); return -EIO; + } if (len < AVE_CMD_HDR_BYTES || len > max || !IS_ALIGNED(len, 4) || (ave->unacked & BIT(buf)) || (reply && op >= ARRAY_SIZE(ave->want))) return -EINVAL; @@ -473,8 +476,11 @@ int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, for (;;) { if (cond && cond(ave, arg)) return 0; - if (ave->fw_state != AVE_FW_RUNNING) + if (ave->fw_state != AVE_FW_RUNNING) { + dev_err_ratelimited(ave->dev, "waiting on a firmware in state %d\n", + ave->fw_state); return -EIO; + } ret = ave_ipc_poll_once(ave); if (ret < 0) { ave_ipc_mark_broken(ave, ret, "ring error"); diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index c4eea08aa437e..9f5b46704ac4c 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -592,8 +592,11 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff *keyframe = false; if (!out) return -EFAULT; - if (s->status) + if (s->status) { + dev_warn_ratelimited(ctx->ave->dev, "frame %u: firmware status %#x\n", + ctx->dst_sequence, s->status); return -EIO; + } data = get_unaligned_le32(meta + AVE_META_PAYLOAD); data2 = get_unaligned_le32(meta + AVE_META_PAYLOAD2); diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 07182d92592f4..281e9cbad38b0 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -1036,8 +1036,17 @@ static void ave_run_work(struct work_struct *work) ret = ave_session_begin(ctx); mutex_lock(&ave->hw_lock); - if (!ret && ctx->sess && (ctx->sess->dead || ave->active != ctx)) + /* + * A closed session (after a drain) may still hold frames for CAPTURE + * buffers; only an open one must still own the firmware. + */ + if (!ret && ctx->sess && ctx->sess->dead) { + dev_err(ave->dev, "session ended by system sleep\n"); + ret = -EIO; + } else if (!ret && ctx->sess && ctx->sess->fw_open && ave->active != ctx) { + dev_err(ave->dev, "session lost the firmware\n"); ret = -EIO; + } if (!ret) { eos = ave_deliver(ctx); if (!eos && !READ_ONCE(ctx->flush_pending) && ctx->sess && @@ -1049,8 +1058,11 @@ static void ave_run_work(struct work_struct *work) if (!eos && READ_ONCE(ctx->flush_pending)) eos = ave_finish_drain(ctx); /* Frames outstanding need a running firmware. */ - if (!eos && ctx->sess && ctx->sess->busy && ave->fw_state != AVE_FW_RUNNING) + if (!eos && ctx->sess && ctx->sess->busy && ave->fw_state != AVE_FW_RUNNING) { + dev_err(ave->dev, "%u frames outstanding, firmware state %d\n", + ctx->sess->busy, ave->fw_state); ret = -EIO; + } poll = !ret && !eos && ctx->sess && ctx->sess->busy; } mutex_unlock(&ave->hw_lock); @@ -1100,6 +1112,8 @@ static void ave_poll_work(struct work_struct *work) } again = !fail && ctx->sess->busy; } else if (ctx->sess && ctx->sess->busy) { + dev_err(ave->dev, "%u frames outstanding in a session that lost the firmware\n", + ctx->sess->busy); fail = true; } mutex_unlock(&ave->hw_lock); From b2fd1d5d90d6b12c537d23c29d48cae372ea4bb5 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:54:56 +1100 Subject: [PATCH 15/20] media: apple: ave: two streams, survive system sleep, copy odd strides Make the encoder usable as a general V4L2 device: - Two streams at once. One firmware boot serves both, as macOS was seen doing: each session has its own command token counter and Process buffers, replies are routed by session token, header word +0x2c follows the set of open sessions (0x140 on Open and Start and 0 elsewhere while another session is open), and the firmware is halted only when the last session closes. - System sleep. Open sessions are drained, stopped and closed and the firmware halted before the power goes. On resume each session is opened and started again with the same parameters on its next frame, which then starts with an IDR and fresh parameter sets. Clients see no error. - Input with a different line size. ffmpeg's v4l2m2m encoder fills NV12 buffers with its own line size, which differs from the 64-byte aligned stride the encoder needs when the width is not a multiple of 64. Such frames, recognised from the payload size, are now copied to a per-slot buffer in the encoder's layout instead of being refused. - Session images are named after the machine instead of the firmware checksum (apple/ave-j700-{h264,hevc}-{start,process}.bin), since they are extracted from the user's own macOS installation. Errors name the missing or mis-sized file. - Polling backs off to sleeps of up to a millisecond once a wait goes on, so waiting for frames costs next to no CPU time. Which interrupt line signals the rings is still not known. - At the end of an H.264 session the debug log reports how much of the per-frame buffer (Process +0x58c8) the firmware used, to size it later. Also clean up the remaining checkpatch --strict notes. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-cmd.c | 56 +++-- drivers/media/platform/apple/ave/ave-drv.c | 21 +- drivers/media/platform/apple/ave/ave-fw.c | 45 +++- drivers/media/platform/apple/ave/ave-ipc.c | 23 +- drivers/media/platform/apple/ave/ave-regs.h | 1 + drivers/media/platform/apple/ave/ave-sess.c | 96 ++++++- drivers/media/platform/apple/ave/ave-v4l2.c | 263 +++++++++++++------- drivers/media/platform/apple/ave/ave.h | 27 +- 8 files changed, 379 insertions(+), 153 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index 0aa872b2d720f..922db11c98b80 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -16,7 +16,6 @@ #include #include -#include #include #include "ave.h" @@ -79,19 +78,41 @@ static void ave_put_buf(u8 *cmd, u32 off, const struct ave_buf *b, u32 size) ave_put_surface(cmd, off, b->dma, size); } -static u32 ave_session_arg2(enum ave_codec codec) +/* Is another session open in the firmware besides @ctx's? */ +static bool ave_cmd_others_open(struct ave_ctx *ctx) { - return codec == AVE_CODEC_HEVC ? AVE_SESSION_ARG2_HEVC : AVE_SESSION_ARG2_H264; + struct ave_dev *ave = ctx->ave; + + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] && ave->sessions[i] != ctx && ave->sessions[i]->sess->fw_open) + return true; + return false; +} + +/* + * Header word +0x2c depends on the set of open sessions: 0x1e0 on Open and + * Start (0x140 when another session is open), the codec value on every + * other command of a lone session, and 0 while another one is open. + */ +static u32 ave_session_arg2(struct ave_ctx *ctx) +{ + if (ave_cmd_others_open(ctx)) + return 0; + return ctx->sess->p.codec == AVE_CODEC_HEVC ? AVE_SESSION_ARG2_HEVC : + AVE_SESSION_ARG2_H264; +} + +static u32 ave_session_setup_arg2(struct ave_ctx *ctx) +{ + return ave_cmd_others_open(ctx) ? AVE_SESSION_ARG2_SETUP_SHARED : AVE_SESSION_ARG2_SETUP; } static void ave_cmd_header(struct ave_ctx *ctx, u8 *cmd, enum ave_opcode op, u32 echo, u32 arg0, u32 arg2) { - struct ave_dev *ave = ctx->ave; - - ave->next_token += AVE_TOKEN_STEP; + ctx->sess->token += AVE_TOKEN_STEP; ave_put32(cmd, AVE_CMD_OPCODE, op); - ave_put64(cmd, AVE_CMD_TOKEN, ave->next_token); + ave_put64(cmd, AVE_CMD_TOKEN, ctx->sess->token); ave_put64(cmd, AVE_CMD_CLIENT, ctx->sess->client); ave_put32(cmd, AVE_CMD_UNIT, AVE_SESSION_UNIT); ave_put32(cmd, AVE_CMD_CODEC, ctx->sess->p.codec); @@ -157,7 +178,8 @@ int ave_cmd_open(struct ave_ctx *ctx) ave->next_client += AVE_CLIENT_STEP; ctx->sess->client = ave->next_client; - ret = ave_cmd_scalar(ctx, AVE_OP_OPEN, AVE_SESSION_ARG0, AVE_SESSION_ARG2_SETUP); + ctx->sess->token = AVE_TOKEN_BASE; + ret = ave_cmd_scalar(ctx, AVE_OP_OPEN, AVE_SESSION_ARG0, ave_session_setup_arg2(ctx)); if (!ret) ctx->sess->fw_open = true; return ret ?: ave_ipc_settle(ave, AVE_CMD_WAIT_MS); @@ -169,14 +191,14 @@ int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count) int ret = 0; for (unsigned int i = 0; i < count && !ret; i++) - ret = ave_cmd_scalar(ctx, AVE_OP_COMPLETE, 0, ave_session_arg2(ctx->sess->p.codec)); + ret = ave_cmd_scalar(ctx, AVE_OP_COMPLETE, 0, ave_session_arg2(ctx)); return ret; } /* Priority and Stop go out together, Close after Stop is answered. */ int ave_cmd_close(struct ave_ctx *ctx) { - u32 arg2 = ave_session_arg2(ctx->sess->p.codec); + u32 arg2 = ave_session_arg2(ctx); int ret, err; ret = ave_cmd_scalar(ctx, AVE_OP_PRIORITY, AVE_SESSION_ARG0, arg2); @@ -467,7 +489,7 @@ int ave_cmd_start(struct ave_ctx *ctx) return -EBUSY; memcpy(cmd, tmpl->data, tmpl->size); ave_cmd_header(ctx, cmd, AVE_OP_START, AVE_OP_START, AVE_SESSION_ARG0, - AVE_SESSION_ARG2_SETUP); + ave_session_setup_arg2(ctx)); ave_start_surfaces(sess, cmd); ave_start_config(&sess->p, cmd); ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_START, 0); @@ -542,7 +564,7 @@ static const struct ave_proc_layout ave_proc_hevc = { #define AVE_PROC_FORCED_TYPE 2 int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, - u32 bitrate, bool force_key) + dma_addr_t input, u32 bitrate, bool force_key) { struct ave_dev *ave = ctx->ave; struct ave_session *sess = ctx->sess; @@ -551,16 +573,12 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff &ave_proc_h264; const struct firmware *tmpl = ave->tmpl[p->codec - 1][AVE_TMPL_PROCESS]; struct ave_slot *s = &sess->slot[slot]; - unsigned int buf = AVE_BUF_PROC(slot); - dma_addr_t y; + unsigned int buf = AVE_BUF_PROC(sess->index * AVE_MAX_SLOTS + slot); + dma_addr_t y = input; u64 pts; u8 *cmd; int ret; - y = vb2_dma_contig_plane_dma_addr(&src->vb2_buf, 0) + src->vb2_buf.planes[0].data_offset; - if (!IS_ALIGNED(y, 64)) - return -EINVAL; - ret = ave_ipc_wait_ack(ave, buf); if (ret) return ret; @@ -569,7 +587,7 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff return -EBUSY; memcpy(cmd, tmpl->data, tmpl->size); ave_cmd_header(ctx, cmd, AVE_OP_PROCESS, AVE_PROC_SUBTYPE + slot, 0, - ave_session_arg2(p->codec)); + ave_session_arg2(ctx)); /* Input picture: NV12, chroma right after p->height luma lines. */ ave_put64(cmd, l->y, y); diff --git a/drivers/media/platform/apple/ave/ave-drv.c b/drivers/media/platform/apple/ave/ave-drv.c index 5ed6cf9176a85..6d67ddf3780e1 100644 --- a/drivers/media/platform/apple/ave/ave-drv.c +++ b/drivers/media/platform/apple/ave/ave-drv.c @@ -127,15 +127,12 @@ static int ave_suspend(struct device *dev) v4l2_m2m_suspend(ave->m2m_dev); /* - * An open session cannot survive the power going down: it is ended - * here (its next job fails) and the firmware halted, so the next - * session restarts it from the saved DATA copy. + * The firmware does not survive the power going down. Drain and close + * every open session and halt the firmware, as at the end of a stream; + * each session reopens in a cold-started firmware on its next frame. */ mutex_lock(&ave->hw_lock); - if (ave->active) { - ave->active->sess->dead = true; - ave->active = NULL; - } + ave_v4l2_suspend(ave); if (ave_fw_shutdown(ave)) dev_warn(dev, "firmware not answering, it cannot be stopped for sleep\n"); mutex_unlock(&ave->hw_lock); @@ -175,11 +172,11 @@ static struct platform_driver ave_driver = { }; module_platform_driver(ave_driver); -/* Constant parts of Start and Process for AppleAVE2FW 9003.78.0 (macOS 26.0.x) */ -MODULE_FIRMWARE("apple/ave-t8140-69c54454-h264-start.bin"); -MODULE_FIRMWARE("apple/ave-t8140-69c54454-h264-process.bin"); -MODULE_FIRMWARE("apple/ave-t8140-69c54454-hevc-start.bin"); -MODULE_FIRMWARE("apple/ave-t8140-69c54454-hevc-process.bin"); +/* Session images, extracted from the user's macOS installation */ +MODULE_FIRMWARE("apple/ave-j700-h264-start.bin"); +MODULE_FIRMWARE("apple/ave-j700-h264-process.bin"); +MODULE_FIRMWARE("apple/ave-j700-hevc-start.bin"); +MODULE_FIRMWARE("apple/ave-j700-hevc-process.bin"); MODULE_DESCRIPTION("Apple AVE video encoder"); MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index e1f9a11118ed0..5db179711c0a1 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -106,6 +106,21 @@ static u32 ave_fw_crc(const struct ave_fw_segment *seg, void **copy) return crc; } +/* + * The session images are per machine: "j700" from "apple,j700", the first + * root compatible. + */ +static void ave_fw_board_name(struct ave_dev *ave) +{ + const char *compat = NULL, *comma; + + of_property_read_string_index(of_root, "compatible", 0, &compat); + if (!compat) + compat = "unknown"; + comma = strchr(compat, ','); + strscpy(ave->board, comma ? comma + 1 : compat, sizeof(ave->board)); +} + /* * Find the firmware, check that it is a build we speak to and that DATA is * still as the bootloader left it, and keep a copy of DATA for restarts. @@ -137,6 +152,7 @@ int ave_fw_probe(struct ave_dev *ave) text_crc); ave->text_crc = text_crc; mutex_init(&ave->tmpl_lock); + ave_fw_board_name(ave); ave->data_pristine = vmalloc(ave->data.size); if (!ave->data_pristine) @@ -169,6 +185,7 @@ static void ave_fw_restore_data(struct ave_dev *ave) if (WARN_ON(!va)) return; memcpy(va, ave->data_pristine, ave->data.size); + /* The restored image must be in memory before the coprocessor starts. */ wmb(); memunmap(va); ave->data_dirty = false; @@ -209,9 +226,10 @@ int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long } /* - * Start and Process are mostly constant for a firmware build. Those - * constant words come from per-build image files named after the TEXT - * checksum; the driver fills in every session and frame field itself. + * Start and Process are mostly constant for a machine and its firmware. + * Those constant words come from image files named after the machine, + * which the user extracts from their own macOS installation; the driver + * fills in every session and frame field itself. */ static const char * const ave_codec_name[] = { "h264", "hevc" }; static const char * const ave_tmpl_name[AVE_TMPL_COUNT] = { "start", "process" }; @@ -234,18 +252,24 @@ int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec) if (ave->tmpl[c][t]) continue; - snprintf(name, sizeof(name), "apple/ave-t8140-%08x-%s-%s.bin", ave->text_crc, + snprintf(name, sizeof(name), "apple/ave-%s-%s-%s.bin", ave->board, ave_codec_name[c], ave_tmpl_name[t]); ret = request_firmware(&fw, name, ave->dev); if (ret) { - dev_err(ave->dev, "cannot load %s: %d\n", name, ret); + dev_err(ave->dev, "cannot load %s (%d); extract it from macOS first\n", + name, ret); + break; + } + if (fw->size != ave_tmpl_size[c][t]) { + dev_err(ave->dev, "%s is %zu bytes, expected %zu\n", name, fw->size, + ave_tmpl_size[c][t]); + release_firmware(fw); + ret = -EINVAL; break; } - if (fw->size != ave_tmpl_size[c][t] || - (t == AVE_TMPL_START && - get_unaligned_le32(fw->data + AVE_TMPL_CODEC) != codec)) { - dev_err(ave->dev, "%s does not fit this driver (%zu bytes)\n", name, - fw->size); + if (t == AVE_TMPL_START && get_unaligned_le32(fw->data + AVE_TMPL_CODEC) != codec) { + dev_err(ave->dev, "%s is not a %s session image\n", name, + ave_codec_name[c]); release_firmware(fw); ret = -EINVAL; break; @@ -355,7 +379,6 @@ static int ave_fw_boot(struct ave_dev *ave) memset(ave->want, 0, sizeof(ave->want)); ave->reply_err = 0; ave->scalar_next = 0; - ave->next_token = AVE_TOKEN_BASE; ave_fw_restore_data(ave); diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c index af067fb024e1e..974f3b2e8cc90 100644 --- a/drivers/media/platform/apple/ave/ave-ipc.c +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -39,6 +39,7 @@ static void ave_mbox_send(struct ave_dev *ave, u32 a, u32 b, u32 c, u32 d) writel(b, ave->mbox + AVE_MBOX_SCRATCH(1)); writel(c, ave->mbox + AVE_MBOX_SCRATCH(2)); writel(d, ave->mbox + AVE_MBOX_SCRATCH(3)); + /* All four words must land before the doorbell. */ wmb(); writel(1, ave->mbox + AVE_MBOX_DOORBELL); } @@ -393,10 +394,7 @@ static void ave_ipc_dispatch(struct ave_dev *ave) status = get_unaligned_le32(ave->msg + AVE_REPLY_STATUS); if (id == AVE_OP_PROCESS) { - if (ave->active) - ave_sess_reply(ave->active, ave->msg, ave->msg_len); - else - dev_warn(ave->dev, "frame reply without a session\n"); + ave_sess_reply(ave, ave->msg, ave->msg_len); return; } if (id < ARRAY_SIZE(ave->want) && ave->want[id]) { @@ -465,11 +463,19 @@ static int ave_ipc_poll_once(struct ave_dev *ave) * Handle acks and messages until @cond holds, for at most @wait_ms. With no * @cond, handle whatever is pending and return. Running out of time means * the firmware is stuck: it is marked broken. Caller holds hw_lock. + * + * Which interrupt line signals the rings is not known, so this polls. It + * sleeps between polls: briefly at first (commands are acknowledged within + * microseconds), then for up to a millisecond, so a long wait for a frame + * costs next to no CPU time. */ +#define AVE_POLL_FAST_ROUNDS 20 + int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, bool (*cond)(struct ave_dev *ave, void *arg), void *arg) { ktime_t deadline = ktime_add_ms(ktime_get(), wait_ms); + unsigned int idle = 0; int ret; lockdep_assert_held(&ave->hw_lock); @@ -486,15 +492,20 @@ int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, ave_ipc_mark_broken(ave, ret, "ring error"); return ret; } - if (ret) + if (ret) { + idle = 0; continue; + } if (!cond) return 0; if (ktime_after(ktime_get(), deadline)) { ave_ipc_mark_broken(ave, -ETIMEDOUT, "no answer"); return -ETIMEDOUT; } - usleep_range(50, 150); + if (idle++ < AVE_POLL_FAST_ROUNDS) + usleep_range(50, 100); + else + usleep_range(500, 1000); } } diff --git a/drivers/media/platform/apple/ave/ave-regs.h b/drivers/media/platform/apple/ave/ave-regs.h index 894d2e3d4ff0a..a85fde26278a6 100644 --- a/drivers/media/platform/apple/ave/ave-regs.h +++ b/drivers/media/platform/apple/ave/ave-regs.h @@ -126,6 +126,7 @@ enum ave_opcode { #define AVE_SESSION_ARG0 0xc8 #define AVE_SESSION_ARG1 0x36 #define AVE_SESSION_ARG2_SETUP 0x1e0 /* Open and Start */ +#define AVE_SESSION_ARG2_SETUP_SHARED 0x140 /* the same, another session open */ #define AVE_SESSION_ARG2_H264 0x40 /* every other session command */ #define AVE_SESSION_ARG2_HEVC 0x20 #define AVE_TOKEN_STEP 100 diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 9f5b46704ac4c..b96a84ad6d1a8 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -18,6 +18,7 @@ #include #include +#include #include #include "ave.h" @@ -276,9 +277,67 @@ static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) ave_free(ave, &sess->slot[s].coded); ave_free(ave, &sess->slot[s].meta); ave_free(ave, &sess->slot[s].aux); + ave_free(ave, &sess->slot[s].bounce); } } +/* + * The input picture for @slot, as a DART address. The encoder reads NV12 + * with the 64-byte aligned stride S_FMT reported and chroma right after + * height luma lines. Some clients (ffmpeg's v4l2m2m encoder) fill the + * buffer with their own, smaller line size instead, which shows in the + * payload size; such frames are copied to a per-slot buffer in the + * encoder's layout. + */ +int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t *input) +{ + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + struct vb2_buffer *vb = &src->vb2_buf; + u32 off = vb->planes[0].data_offset; + u32 used = vb2_get_plane_payload(vb, 0) - min(off, vb2_get_plane_payload(vb, 0)); + u32 crows = DIV_ROUND_UP(p->height, 2), rows = p->height + crows; + u32 width = ALIGN(p->width, 2), stride; + struct ave_buf *b = &sess->slot[slot].bounce; + const u8 *from; + u8 *to; + int ret; + + *input = vb2_dma_contig_plane_dma_addr(vb, 0) + off; + if (!used || used >= p->stride * rows || used % rows || used / rows < width) { + if (IS_ALIGNED(*input, 64)) + return 0; + dev_err(ctx->ave->dev, "OUTPUT plane at %pad is not 64-byte aligned\n", input); + return -EINVAL; + } + stride = used / rows; + + from = vb2_plane_vaddr(vb, 0); + if (!from) { + dev_err(ctx->ave->dev, "cannot map an OUTPUT buffer to copy its %u byte stride\n", + stride); + return -EFAULT; + } + from += off; + ret = ave_alloc(ctx->ave, b, p->stride * rows, 0); + if (ret) { + dev_err(ctx->ave->dev, "no room to copy %u byte stride input\n", stride); + return ret; + } + if (!sess->copy_noted) { + sess->copy_noted = true; + dev_info(ctx->ave->dev, "input has a %u byte stride; copying to %u\n", stride, + p->stride); + } + + to = b->cpu; + for (u32 y = 0; y < rows; y++) + memcpy(to + y * p->stride, from + y * stride, width); + *input = b->dma; + return 0; +} + /* Debug map of the session buffers, to match DART faults to a buffer. */ static void ave_log_buf(struct ave_dev *ave, const char *name, int i, const struct ave_buf *b) { @@ -362,6 +421,20 @@ static void ave_sess_check_tails(struct ave_dev *ave, struct ave_session *sess) ave_check_tail(ave, "meta", i, &sess->slot[i].meta); ave_check_tail(ave, "aux", i, &sess->slot[i].aux); } + + /* + * The size of the H.264 per-frame buffer is not known; it is + * allocated generously. Report how much of it the firmware used. + */ + if (sess->p.codec == AVE_CODEC_H264 && sess->slot[0].aux.cpu) { + const struct ave_buf *b = &sess->slot[0].aux; + const u64 *w = b->cpu; + size_t n = b->size / 8; + + while (n && !w[n - 1]) + n--; + dev_dbg(ave->dev, "per-frame buffer: %#zx of %#zx bytes used\n", n * 8, b->size); + } } /* Work out the parameters and allocate the buffers; no firmware access. */ @@ -370,7 +443,7 @@ struct ave_session *ave_sess_create(struct ave_ctx *ctx) struct ave_session *sess; int ret; - sess = kzalloc(sizeof(*sess), GFP_KERNEL); + sess = kzalloc_obj(*sess); if (!sess) return ERR_PTR(-ENOMEM); ret = ave_sess_params(ctx, &sess->p); @@ -437,18 +510,27 @@ void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot) memset(s->aux.cpu, 0, AVE_SHDR_ENTRY_BYTES); } -/* A frame reply: +0x10 our client, +0x20 the slot's subtype, +0x40 status. */ -void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len) +/* + * A frame reply: +0x10 the session's client token, +0x20 the slot's + * subtype, +0x40 status. Caller holds hw_lock. + */ +void ave_sess_reply(struct ave_dev *ave, const u8 *msg, u32 len) { - struct ave_session *sess = ctx->sess; u64 client = get_unaligned_le64(msg + AVE_REPLY_CLIENT); u32 subtype = get_unaligned_le32(msg + AVE_REPLY_SUBTYPE); u32 slot = subtype - AVE_PROC_SUBTYPE; + struct ave_session *sess = NULL; + struct ave_ctx *ctx = NULL; struct ave_slot *s; - if (!sess || len < AVE_REPLY_PROCESS_BYTES || client != sess->client || - slot >= sess->p.slots || !sess->slot[slot].busy) { - dev_warn(ctx->ave->dev, "stray frame reply (client %llu subtype %#x len %u)\n", + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] && ave->sessions[i]->sess->client == client) + ctx = ave->sessions[i]; + if (ctx) + sess = ctx->sess; + if (!sess || len < AVE_REPLY_PROCESS_BYTES || slot >= sess->p.slots || + !sess->slot[slot].busy) { + dev_warn(ave->dev, "stray frame reply (client %llu subtype %#x len %u)\n", client, subtype, len); return; } diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 281e9cbad38b0..caa874ba48de3 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -594,14 +594,35 @@ static void ave_return_bufs(struct ave_ctx *ctx, struct vb2_queue *q, /* Sessions */ -/* Caller holds hw_lock. */ +static int ave_session_index(struct ave_dev *ave, struct ave_ctx *ctx) +{ + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] == ctx) + return i; + return -1; +} + +static unsigned int ave_session_count(struct ave_dev *ave) +{ + unsigned int n = 0; + + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + n += !!ave->sessions[i]; + return n; +} + +/* + * Drain (with @flush), stop and close the session in the firmware. The + * last session to go halts the firmware. Caller holds hw_lock. + */ static void ave_session_close(struct ave_ctx *ctx, bool flush) { struct ave_dev *ave = ctx->ave; struct ave_session *sess = ctx->sess; + int idx = ave_session_index(ave, ctx); lockdep_assert_held(&ave->hw_lock); - if (!sess || ave->active != ctx) + if (!sess || idx < 0) return; if (sess->fw_open && ave->fw_state == AVE_FW_RUNNING) { /* Complete is a barrier: answered after every frame before it. */ @@ -611,8 +632,81 @@ static void ave_session_close(struct ave_ctx *ctx, bool flush) ave_cmd_close(ctx); } sess->fw_open = false; - ave->active = NULL; - ave_fw_halt(ave); + ave->sessions[idx] = NULL; + if (!ave_session_count(ave)) + ave_fw_halt(ave); +} + +/* + * Open and Start the session in the firmware, booting the firmware first + * if no other session runs. Returns -EAGAIN when a halted coprocessor has + * not lost power yet. Caller holds hw_lock. + */ +static int ave_session_attach(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + int idx = -1, ret; + + lockdep_assert_held(&ave->hw_lock); + for (unsigned int i = 0; i < AVE_MAX_SESSIONS && idx < 0; i++) + if (!ave->sessions[i]) + idx = i; + if (idx < 0) { + dev_err(ave->dev, "already encoding %u streams\n", AVE_MAX_SESSIONS); + return -EBUSY; + } + if (!ave_session_count(ave)) { + ret = ave_fw_ensure_running(ave); + if (ret) + return ret; + } else if (ave->fw_state != AVE_FW_RUNNING) { + dev_err(ave->dev, "firmware state %d, cannot add a session\n", ave->fw_state); + return -EIO; + } + + sess->index = idx; + sess->frames = 0; + sess->nudged = false; + ave->sessions[idx] = ctx; + ret = ave_cmd_open(ctx); + if (!ret) + ret = ave_cmd_start(ctx); + if (ret) + ave_session_close(ctx, false); + return ret; +} + +/* + * Attach with the device runtime active and one PM reference held by the + * caller. A halted coprocessor restarts only from reset, and its power can + * take a moment to go (the DARTs suspend a little later), so drop the + * reference and cycle the power a few times before giving up. On return + * the reference is still held, unless *pm_lost says otherwise. + */ +static int ave_session_attach_cycled(struct ave_ctx *ctx, bool *pm_lost) +{ + struct ave_dev *ave = ctx->ave; + int ret = -EAGAIN; + + *pm_lost = false; + for (unsigned int i = 0; i < AVE_RESTART_TRIES && ret == -EAGAIN; i++) { + if (i) { + pm_runtime_put_sync_suspend(ave->dev); + msleep(AVE_RESTART_WAIT_MS); + ret = pm_runtime_resume_and_get(ave->dev); + if (ret) { + *pm_lost = true; + return ret; + } + } + mutex_lock(&ave->hw_lock); + ret = ave_session_attach(ctx); + mutex_unlock(&ave->hw_lock); + } + if (ret == -EAGAIN) + dev_err(ave->dev, "coprocessor did not power down; cannot restart it\n"); + return ret; } /* Return frames the firmware never answered; then drop the session. */ @@ -641,21 +735,24 @@ static int ave_session_begin(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; struct ave_session *sess; + bool pm_lost; int ret; if (ctx->sess) return 0; - /* A halted firmware restarts only from reset: let the power go first. */ - if (READ_ONCE(ave->fw_state) == AVE_FW_HALTED) - pm_runtime_suspend(ave->dev); ret = ave_fw_templates(ave, ave_ctx_codec(ctx)); if (ret) return ret; - if (READ_ONCE(ave->active)) + if (READ_ONCE(ave->sessions[0]) && READ_ONCE(ave->sessions[1])) return -EBUSY; + /* A halted firmware restarts only from reset: let the power go first. */ + if (READ_ONCE(ave->fw_state) == AVE_FW_HALTED) + pm_runtime_suspend(ave->dev); sess = ave_sess_create(ctx); - if (IS_ERR(sess)) + if (IS_ERR(sess)) { + dev_err(ave->dev, "cannot set up a session: %ld\n", PTR_ERR(sess)); return PTR_ERR(sess); + } ctx->sess = sess; ret = pm_runtime_resume_and_get(ave->dev); @@ -663,50 +760,59 @@ static int ave_session_begin(struct ave_ctx *ctx) ave_sess_destroy(ctx); return ret; } - - mutex_lock(&ave->hw_lock); - if (ave->active) { - ret = -EBUSY; - goto out; - } - /* - * A halted coprocessor restarts only from reset. Its power can take - * a moment to go after a suspend (the DARTs follow later), so cycle - * it again a few times before giving up. - */ - for (unsigned int i = 0; i < AVE_RESTART_TRIES; i++) { - ret = ave_fw_ensure_running(ave); - if (ret != -EAGAIN) - break; - mutex_unlock(&ave->hw_lock); - pm_runtime_put_sync_suspend(ave->dev); - msleep(AVE_RESTART_WAIT_MS); - ret = pm_runtime_resume_and_get(ave->dev); - mutex_lock(&ave->hw_lock); - if (ret) { - mutex_unlock(&ave->hw_lock); - ave_sess_destroy(ctx); - return ret; - } - ret = -EAGAIN; - } - if (ret) - goto out; - ave->active = ctx; - ret = ave_cmd_open(ctx); - if (!ret) - ret = ave_cmd_start(ctx); - if (ret) - ave_session_close(ctx, false); -out: - mutex_unlock(&ave->hw_lock); + ret = ave_session_attach_cycled(ctx, &pm_lost); if (ret) { dev_err(ave->dev, "cannot start an encoding session: %d\n", ret); - ave_session_free(ctx); + if (pm_lost) + ave_sess_destroy(ctx); + else + ave_session_free(ctx); } return ret; } +/* + * After system sleep: the session was closed when the firmware went down. + * Open and Start it again with the same parameters; its next frame is an + * IDR with fresh parameter sets, so the stream stays decodable. + */ +static int ave_session_reopen(struct ave_ctx *ctx) +{ + bool pm_lost; + int ret; + + ret = ave_session_attach_cycled(ctx, &pm_lost); + if (ret) { + if (pm_lost) { + /* The session's PM reference is gone; do not drop it again. */ + pm_runtime_get_noresume(ctx->ave->dev); + } + return ret; + } + ctx->sess->resume = false; + dev_dbg(ctx->ave->dev, "session reopened after system sleep\n"); + return 0; +} + +/* + * System sleep: drain every open session, close it and halt the firmware, + * as at the end of a stream. Sessions reopen on their next frame. + * Caller holds hw_lock. + */ +void ave_v4l2_suspend(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) { + struct ave_ctx *ctx = ave->sessions[i]; + + if (!ctx) + continue; + cancel_delayed_work(&ctx->poll_work); + ave_session_close(ctx, true); + ctx->sess->resume = true; + } +} + static void ave_session_end(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; @@ -878,30 +984,6 @@ static bool ave_frame_answered(struct ave_dev *ave, void *arg) return !s->busy; } -/* - * The encoder reads NV12 with the 64-byte aligned stride S_FMT reported. - * ffmpeg's v4l2m2m encoder copies frames with its own line size instead, - * which differs when the width is not a multiple of 64; the result is a - * sheared picture. Spot that from the payload size and refuse it. - */ -static int ave_check_src_layout(struct ave_ctx *ctx, struct vb2_v4l2_buffer *src) -{ - const struct ave_params *p = &ctx->sess->p; - u32 used = vb2_get_plane_payload(&src->vb2_buf, 0); - u32 rows = p->height + DIV_ROUND_UP(p->height, 2); - u32 stride; - - if (!used || used % rows) - return 0; - stride = used / rows; - if (stride >= p->stride || stride < p->width) - return 0; - dev_err(ctx->ave->dev, - "OUTPUT frame laid out with a %u byte stride, not the %u reported by S_FMT\n", - stride, p->stride); - return -EINVAL; -} - /* Send the next OUTPUT frame. Caller holds hw_lock. */ static int ave_encode_one(struct ave_ctx *ctx) { @@ -909,6 +991,7 @@ static int ave_encode_one(struct ave_ctx *ctx) struct ave_session *sess = ctx->sess; struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; struct vb2_v4l2_buffer *src; + dma_addr_t input; bool last; int slot, ret; @@ -930,14 +1013,10 @@ static int ave_encode_one(struct ave_ctx *ctx) last = src == ctx->drain_src; src->sequence = ctx->src_sequence++; - ret = ave_check_src_layout(ctx, src); - if (ret) { - v4l2_m2m_buf_done(src, VB2_BUF_STATE_ERROR); - return ret; - } - - ret = ave_cmd_process(ctx, slot, src, READ_ONCE(ctx->bitrate), - xchg(&ctx->force_key_frame, false)); + ret = ave_sess_input(ctx, slot, src, &input); + if (!ret) + ret = ave_cmd_process(ctx, slot, src, input, READ_ONCE(ctx->bitrate), + xchg(&ctx->force_key_frame, false)); if (ret) { v4l2_m2m_buf_done(src, VB2_BUF_STATE_ERROR); return ret; @@ -1031,19 +1110,23 @@ static void ave_run_work(struct work_struct *work) bool eos = false, poll = false; int ret = 0; - /* After a drain, the next frame starts a new session. */ - if (!ctx->sess && !READ_ONCE(ctx->flush_pending) && v4l2_m2m_num_src_bufs_ready(m2m)) - ret = ave_session_begin(ctx); + /* + * After a drain the next frame starts a new session; after system + * sleep it reopens the old one in the restarted firmware. + */ + if (!READ_ONCE(ctx->flush_pending) && v4l2_m2m_num_src_bufs_ready(m2m)) { + if (!ctx->sess) + ret = ave_session_begin(ctx); + else if (ctx->sess->resume) + ret = ave_session_reopen(ctx); + } mutex_lock(&ave->hw_lock); /* - * A closed session (after a drain) may still hold frames for CAPTURE - * buffers; only an open one must still own the firmware. + * A closed session (after a drain or for sleep) may still hold frames + * for CAPTURE buffers; only an open one must still own the firmware. */ - if (!ret && ctx->sess && ctx->sess->dead) { - dev_err(ave->dev, "session ended by system sleep\n"); - ret = -EIO; - } else if (!ret && ctx->sess && ctx->sess->fw_open && ave->active != ctx) { + if (!ret && ctx->sess && ctx->sess->fw_open && ave_session_index(ave, ctx) < 0) { dev_err(ave->dev, "session lost the firmware\n"); ret = -EIO; } @@ -1090,7 +1173,7 @@ static void ave_poll_work(struct work_struct *work) bool again = false, fail = false; mutex_lock(&ave->hw_lock); - if (ctx->sess && ave->active == ctx && !ctx->sess->dead) { + if (ctx->sess && ave_session_index(ave, ctx) >= 0) { fail = ave_ipc_pump(ave, 0, NULL, NULL) != 0; if (!fail && ave_sess_starved(ctx->sess) && !v4l2_m2m_num_src_bufs_ready(ctx->fh.m2m_ctx)) { @@ -1147,7 +1230,7 @@ static int ave_open(struct file *file) struct ave_ctx *ctx; int ret; - ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + ctx = kzalloc_obj(*ctx); if (!ctx) return -ENOMEM; ctx->ave = ave; diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index 018795a5ebfd8..c5dce46faa373 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -23,8 +23,10 @@ struct dev_pm_domain_list; struct firmware; struct iommu_domain; -/* Frames the firmware can hold at once (Process "slots"). */ +/* Frames the firmware can hold at once per session (Process "slots"). */ #define AVE_MAX_SLOTS 5 +/* Sessions one firmware boot serves at once (macOS was seen running two). */ +#define AVE_MAX_SESSIONS 2 /* * Coprocessor memory every firmware boot uses, all inside one allocation. @@ -44,13 +46,14 @@ struct iommu_domain; #define AVE_IPC_SCALAR_BUFS 16 #define AVE_IPC_PROC_OFF 0x60000 #define AVE_IPC_PROC_BYTES 0x10000 -#define AVE_IPC_END (AVE_IPC_PROC_OFF + AVE_MAX_SLOTS * AVE_IPC_PROC_BYTES) +#define AVE_IPC_PROC_BUFS (AVE_MAX_SESSIONS * AVE_MAX_SLOTS) +#define AVE_IPC_END (AVE_IPC_PROC_OFF + AVE_IPC_PROC_BUFS * AVE_IPC_PROC_BYTES) /* Command buffer ids, one bit each in ave_dev.unacked. */ #define AVE_BUF_CMD 0 #define AVE_BUF_SCALAR(i) (1 + (i)) #define AVE_BUF_PROC(s) (1 + AVE_IPC_SCALAR_BUFS + (s)) -#define AVE_BUF_COUNT AVE_BUF_PROC(AVE_MAX_SLOTS) +#define AVE_BUF_COUNT AVE_BUF_PROC(AVE_IPC_PROC_BUFS) #define AVE_MSG_MAX SZ_4K @@ -110,6 +113,7 @@ struct ave_dev { struct ave_fw_segment text; struct ave_fw_segment data; u32 text_crc; + char board[16]; /* from the root compatible, e.g. "j700" */ void *data_pristine; bool data_dirty; @@ -127,19 +131,19 @@ struct ave_dev { u64 ipc_fw_base; u32 channel_bytes; struct ave_ring ring[2]; - u64 next_token; u32 next_client; u32 unacked; u8 want[16]; int reply_err; unsigned int scalar_next; - struct ave_ctx *active; + struct ave_ctx *sessions[AVE_MAX_SESSIONS]; u8 msg[AVE_MSG_MAX]; u32 msg_len; struct v4l2_device v4l2_dev; struct video_device vdev; struct v4l2_m2m_dev *m2m_dev; + /* The video device's ioctl and queue lock. */ struct mutex vdev_lock; struct workqueue_struct *wq; }; @@ -174,6 +178,7 @@ struct ave_slot { struct ave_buf coded; struct ave_buf meta; struct ave_buf aux; /* H.264 per-frame buffer, HEVC slice headers */ + struct ave_buf bounce; /* input copied to the encoder's stride */ struct vb2_v4l2_buffer *src; u32 status; bool busy; /* Process sent, reply pending */ @@ -202,8 +207,11 @@ struct ave_session { unsigned int busy; u32 frames; /* Process commands sent */ u32 client; + u64 token; /* command order token, per session */ + unsigned int index; /* in ave_dev.sessions[] */ bool fw_open; /* Open sent, Close not yet */ - bool dead; /* firmware gone; buffers only */ + bool resume; /* closed for system sleep, reopen on next frame */ + bool copy_noted; bool warned; bool checked[2]; /* Start/Process images checked for stale addresses */ bool handed; /* Start sent: the firmware knows the buffers */ @@ -281,7 +289,7 @@ int ave_cmd_halt(struct ave_dev *ave); int ave_cmd_open(struct ave_ctx *ctx); int ave_cmd_start(struct ave_ctx *ctx); int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, - u32 bitrate, bool force_key); + dma_addr_t input, u32 bitrate, bool force_key); int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count); int ave_cmd_close(struct ave_ctx *ctx); @@ -289,7 +297,9 @@ int ave_cmd_close(struct ave_ctx *ctx); struct ave_session *ave_sess_create(struct ave_ctx *ctx); void ave_sess_destroy(struct ave_ctx *ctx); int ave_sess_free_slot(struct ave_session *sess); -void ave_sess_reply(struct ave_ctx *ctx, const u8 *msg, u32 len); +void ave_sess_reply(struct ave_dev *ave, const u8 *msg, u32 len); +int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t *input); int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, bool *keyframe); void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); @@ -299,6 +309,7 @@ bool ave_sess_stalled(struct ave_session *sess); bool ave_sess_starved(struct ave_session *sess); /* ave-v4l2.c */ +void ave_v4l2_suspend(struct ave_dev *ave); int ave_v4l2_register(struct ave_dev *ave); void ave_v4l2_unregister(struct ave_dev *ave); enum ave_codec ave_ctx_codec(struct ave_ctx *ctx); From 26f65bc170b0542ea178b632e91846dc639f48c2 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 21:54:56 +1100 Subject: [PATCH 16/20] media: apple: ave: document usage and session images, add MAINTAINERS Describe how to use the encoder, where the session images come from (the user's own macOS installation, through the Aurora firmware extractor; no Apple data in the tree), how sessions, power and system sleep behave, and the limits. Update the Kconfig help to match, and add a MAINTAINERS entry. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- Documentation/admin-guide/media/apple-ave.rst | 111 +++++++++++------- MAINTAINERS | 8 ++ drivers/media/platform/apple/ave/Kconfig | 10 +- 3 files changed, 84 insertions(+), 45 deletions(-) diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index 8ca9950dbea4b..788c121d8db51 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -5,10 +5,26 @@ Apple AVE video encoder The AVE block in Apple T8140 (A18 Pro) SoCs, as used in the MacBook Neo, encodes H.264 and HEVC. ``CONFIG_VIDEO_APPLE_AVE`` builds the ``apple-ave`` -driver, which registers a V4L2 memory-to-memory stateful encoder following +driver, a V4L2 memory-to-memory stateful encoder following Documentation/userspace-api/media/v4l/dev-encoder.rst. The OUTPUT queue -takes NV12 frames (64-byte aligned line stride), the CAPTURE queue returns -H.264 (Baseline, Main, High) or HEVC (Main) as Annex B. +takes NV12 frames, the CAPTURE queue returns H.264 (Baseline, Main, High) or +HEVC (Main) as Annex B. + +Usage +----- + +The encoder shows up as a ``/dev/videoN`` node named "Apple AVE video +encoder". GStreamer's ``v4l2h264enc`` and ``v4l2h265enc`` and ffmpeg's +``h264_v4l2m2m`` and ``hevc_v4l2m2m`` work with it, for example:: + + gst-launch-1.0 -e videotestsrc num-buffers=300 ! \ + video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \ + v4l2h264enc extra-controls="controls,video_bitrate=8000000" ! \ + h264parse ! filesink location=out.h264 + + ffmpeg -i in.mkv -pix_fmt nv12 -c:v h264_v4l2m2m -b:v 8M -g 30 out.h264 + +Up to two streams can be encoded at the same time. Firmware -------- @@ -17,57 +33,70 @@ The encoder runs firmware on its own coprocessor. The bootloader loads it into two reserved, DART-mapped segments (TEXT and DATA) but does not start it. The Aurora Stage 2 loader passes both regions in the device's ``memory-region`` property and records their CRC32 under ``/chosen`` as -``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. - -The driver only binds to firmware builds whose command formats it knows, -identified by the TEXT checksum, and only when DATA still matches the -checksum the loader recorded. It keeps a copy of DATA and restores it before -every firmware boot, so the firmware can be started again from the same state. +``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. The driver only binds +to firmware builds whose command formats it knows, identified by the TEXT +checksum, and only while DATA still matches the recorded checksum. It keeps +a copy of DATA and restores it before every firmware boot. Session images -------------- -The two large session commands, Start and Process, are mostly constant for a -given firmware build. The driver writes every field that depends on the -session or the frame (sizes, rate control, GOP structure, levels, cropping, -buffer addresses, frame numbers and times) and takes the remaining constant -words from four files per firmware build:: +Most words of the two large session commands, Start and Process, are +constant for a machine and its firmware. They are not part of the kernel. +The driver loads them from four files named after the machine (the first +root compatible without its vendor prefix, ``j700`` on the MacBook Neo):: - apple/ave-t8140--h264-start.bin (0x3180 bytes) - apple/ave-t8140--h264-process.bin (0x63d8 bytes) - apple/ave-t8140--hevc-start.bin (0x13f28 bytes) - apple/ave-t8140--hevc-process.bin (0xb1c0 bytes) + apple/ave-j700-h264-start.bin (0x3180 bytes) + apple/ave-j700-h264-process.bin (0x63d8 bytes) + apple/ave-j700-hevc-start.bin (0x13f28 bytes) + apple/ave-j700-hevc-process.bin (0xb1c0 bytes) -Each file is a complete command image. Address and size fields in it are -ignored. Without these files, ``VIDIOC_STREAMON`` fails with ``ENOENT``. +The files are extracted from the user's own macOS installation by the +Aurora firmware extractor, in the same way as other firmware the machine +needs, and installed under the firmware search path (usually +``/lib/firmware``). No Apple data is shipped with the kernel. The driver +writes every field that depends on the session or the frame itself, so the +same files serve every size, profile and rate. A missing or wrong-sized +file makes ``VIDIOC_STREAMON`` fail with a message naming it. Sessions and power management ----------------------------- -Each encoding session boots the firmware, opens and starts a session, encodes -and, at the end, drains, stops and closes the session and halts the firmware. -The next session restarts the firmware through the boot path. One session -runs at a time; a second one gets ``EBUSY``. +The firmware boots when the first stream starts and is halted when the last +one ends, as macOS does; the next stream starts it from reset. Runtime PM +powers the encoder down two seconds after the last stream. + +Up to five frames per stream are inside the firmware at once (three without +B-frames). The firmware reorders B-frames itself, so CAPTURE buffers come +back in bitstream order, each with the timestamp of the frame it codes. -Up to five frames are inside the firmware at once (three without B-frames). -The firmware reorders B-frames itself, so CAPTURE buffers come back in -bitstream order, each carrying the timestamp of the frame it codes. The driver -asks for at least six OUTPUT buffers, and limits both queues so a full set fits -the encoder's 256 MiB DMA window (fewer buffers at larger sizes). +Across system sleep, open streams are drained, closed and the firmware is +halted. On resume each stream reopens on its next frame, which starts with +an IDR and fresh parameter sets; clients see no error. -Runtime PM uses a two second autosuspend delay. A firmware that stops -answering keeps the encoder powered and is not used again until reboot. -Across system sleep an open session ends; the next one starts afresh. +Every wait for the firmware is bounded. A command or frame that gets no +answer within 5 seconds (Start: 10 seconds) fails the stream with ``EIO``; +a firmware that stops answering keeps the encoder powered and is not used +again until reboot. The driver polls the firmware rings with sleeping waits. Limits ------ -Picture sizes from 320x240 to 2560x1600 (in steps of two), up to 240 frames -per second. Rate control is VBR (with ``BITRATE_PEAK`` as a one-second data -rate cap, twice the average when left at 0) or CBR; constant quality is not offered. ``BITRATE`` may change -while streaming and applies from the next frame; ``FORCE_KEY_FRAME`` makes -the next frame an IDR. H.264 uses either three B-frames or none (Baseline and -a GOP size of 1 never use them). HEVC always uses them for now; with -``B_FRAMES`` set to 0 every frame is flushed through at once instead. The level -control is a minimum: the driver raises it to what the size, frame rate and -bit rate need. +- Picture sizes from 320x240 to 2560x1600 in steps of two, up to 240 frames + per second. +- The encoder reads NV12 with a 64-byte aligned line stride, which + ``VIDIOC_S_FMT`` reports. Clients that fill buffers with a smaller line + size (ffmpeg does, for widths that are not a multiple of 64) are detected + from the payload size and their frames copied to the encoder's layout. +- Rate control is VBR, with ``BITRATE_PEAK`` as a one-second data-rate cap + (twice the average when left at 0), or CBR. Constant quality is not + offered. ``BITRATE`` may change while streaming and applies from the next + frame; ``FORCE_KEY_FRAME`` makes the next frame an IDR. +- H.264 uses either three B-frames or none (Baseline and a GOP size of 1 never + use them). HEVC always uses them; with ``B_FRAMES`` set to 0 every frame is + flushed through at once instead. +- The level control is a minimum: the driver raises it to what the size, + frame rate and bit rate need. +- Buffers are mapped through the encoder's 256 MiB DMA window, so both queues + are limited to what fits next to the encoder's working memory (fewer + buffers at larger sizes). diff --git a/MAINTAINERS b/MAINTAINERS index 8b2cc48645bf4..7e126d34bed5c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2802,6 +2802,14 @@ F: net/bluetooth/brcm.c F: net/bluetooth/brcm.h F: tools/testing/selftests/drivers/apple/ +ARM/APPLE AVE VIDEO ENCODER DRIVER +M: Ryan Murray +L: linux-media@vger.kernel.org +S: Maintained +W: https://aurora-silicon.github.io +F: Documentation/admin-guide/media/apple-ave.rst +F: drivers/media/platform/apple/ave/ + ARM/APPLE QSPIMC DRIVER M: Ryan Murray W: https://aurora-silicon.github.io diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig index 2b3f380abf752..e1bdc5de54a7f 100644 --- a/drivers/media/platform/apple/ave/Kconfig +++ b/drivers/media/platform/apple/ave/Kconfig @@ -10,10 +10,12 @@ config VIDEO_APPLE_AVE select VIDEOBUF2_DMA_CONTIG help V4L2 memory-to-memory driver for the AVE H.264/HEVC encoder in - Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. The encoder - firmware is loaded by the bootloader; the constant parts of its - session commands come from files under apple/ in the firmware - search path. + Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. + + The encoder firmware is loaded by the bootloader. The constant + parts of its session commands are read from apple/ave--*.bin + in the firmware search path, extracted from the user's own macOS + installation. See Documentation/admin-guide/media/apple-ave.rst. To compile this driver as a module, choose M here: the module will be called apple-ave. From 68c514b2264a98ce9a73b54e3e453bc8ca17e365 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 22:03:34 +1100 Subject: [PATCH 17/20] media: apple: ave: build the session commands in the driver macOS does not store the Start and Process commands anywhere: its driver builds them from the session settings and its own defaults. Apart from the fields this driver already computes, every captured command holds the same small set of constant words. They are now named tables in ave-image.c, grouped and commented, and each command is built in a zeroed buffer from the header, those words and the computed fields. The four session image files and request_firmware() are gone. The constants were checked on one firmware build only (AppleAVE2FW 9003.78.0, TEXT checksum 69c54454), so streams refuse to start on any other build, with a message naming the checksum. With these facts in place: - HEVC can run without B-frames, with the low-delay P reference structure macOS uses when frame reordering is off (Start +0xe120 table, the reorder and reference counts, Process +0xcf8). The per-frame Complete that stood in for it is gone. - H.264 marks a client's B-frame opt-out in Start byte +0x2585, and HEVC sets +0x2588, as macOS does. - CBR pads with filler data once more than 0.5 s of bit rate has gone unused, the rule seen in macOS streams. If the firmware's own output carries no filler, the driver appends a filler-data NAL unit to the access unit; the debug log says which side pads. debugfs apple-ave/last_start_{h264,hevc} and last_process0_{h264,hevc} hold the last Start and the first Process of the last session, for comparison with reference captures. Field values are facts from our private research (macOS traces); no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- Documentation/admin-guide/media/apple-ave.rst | 50 ++-- drivers/media/platform/apple/ave/Kconfig | 7 +- drivers/media/platform/apple/ave/Makefile | 2 +- drivers/media/platform/apple/ave/ave-cmd.c | 53 ++-- drivers/media/platform/apple/ave/ave-drv.c | 54 +++- drivers/media/platform/apple/ave/ave-fw.c | 82 +----- drivers/media/platform/apple/ave/ave-image.c | 278 ++++++++++++++++++ drivers/media/platform/apple/ave/ave-sess.c | 154 +++++----- drivers/media/platform/apple/ave/ave-v4l2.c | 13 +- drivers/media/platform/apple/ave/ave.h | 47 ++- 10 files changed, 496 insertions(+), 244 deletions(-) create mode 100644 drivers/media/platform/apple/ave/ave-image.c diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index 788c121d8db51..2a76d4eac23d1 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -34,30 +34,21 @@ two reserved, DART-mapped segments (TEXT and DATA) but does not start it. The Aurora Stage 2 loader passes both regions in the device's ``memory-region`` property and records their CRC32 under ``/chosen`` as ``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. The driver only binds -to firmware builds whose command formats it knows, identified by the TEXT -checksum, and only while DATA still matches the recorded checksum. It keeps -a copy of DATA and restores it before every firmware boot. - -Session images --------------- - -Most words of the two large session commands, Start and Process, are -constant for a machine and its firmware. They are not part of the kernel. -The driver loads them from four files named after the machine (the first -root compatible without its vendor prefix, ``j700`` on the MacBook Neo):: - - apple/ave-j700-h264-start.bin (0x3180 bytes) - apple/ave-j700-h264-process.bin (0x63d8 bytes) - apple/ave-j700-hevc-start.bin (0x13f28 bytes) - apple/ave-j700-hevc-process.bin (0xb1c0 bytes) - -The files are extracted from the user's own macOS installation by the -Aurora firmware extractor, in the same way as other firmware the machine -needs, and installed under the firmware search path (usually -``/lib/firmware``). No Apple data is shipped with the kernel. The driver -writes every field that depends on the session or the frame itself, so the -same files serve every size, profile and rate. A missing or wrong-sized -file makes ``VIDIOC_STREAMON`` fail with a message naming it. +while DATA still matches the recorded checksum, and keeps a copy of DATA that +it restores before every firmware boot. + +No firmware files are needed. The driver builds every command itself, +including the two large session commands (Start and Process), from the +session settings and a set of constant fields known for one firmware build: +AppleAVE2FW 9003.78.0, TEXT checksum ``69c54454``. With any other firmware +build the device binds, but starting a stream fails with ``ENOTSUPP`` and a +message naming the checksum, until that build's command layout has been +checked. + +With debugfs mounted, ``apple-ave/last_start_h264``, +``last_start_hevc``, ``last_process0_h264`` and ``last_process0_hevc`` hold the +last Start command built and the first Process command of the last session, +for comparison with reference captures. Sessions and power management ----------------------------- @@ -89,12 +80,13 @@ Limits size (ffmpeg does, for widths that are not a multiple of 64) are detected from the payload size and their frames copied to the encoder's layout. - Rate control is VBR, with ``BITRATE_PEAK`` as a one-second data-rate cap - (twice the average when left at 0), or CBR. Constant quality is not - offered. ``BITRATE`` may change while streaming and applies from the next + (twice the average when left at 0), or CBR. In CBR mode, frames are padded + with filler data once more than half a second of bit rate has gone unused, + as macOS does; the driver adds the filler itself if the firmware does not. + Constant quality is not offered. ``BITRATE`` may change while streaming and applies from the next frame; ``FORCE_KEY_FRAME`` makes the next frame an IDR. -- H.264 uses either three B-frames or none (Baseline and a GOP size of 1 never - use them). HEVC always uses them; with ``B_FRAMES`` set to 0 every frame is - flushed through at once instead. +- Either three B-frames or none (``B_FRAMES`` 3 or 0). H.264 Baseline and a + GOP size of 1 never use them. - The level control is a minimum: the driver raises it to what the size, frame rate and bit rate need. - Buffers are mapped through the encoder's 256 MiB DMA window, so both queues diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig index e1bdc5de54a7f..baf6218ecdd5b 100644 --- a/drivers/media/platform/apple/ave/Kconfig +++ b/drivers/media/platform/apple/ave/Kconfig @@ -5,17 +5,14 @@ config VIDEO_APPLE_AVE depends on ARCH_APPLE || COMPILE_TEST depends on VIDEO_DEV && IOMMU_DMA && PM_GENERIC_DOMAINS select CRC32 - select FW_LOADER select V4L2_MEM2MEM_DEV select VIDEOBUF2_DMA_CONTIG help V4L2 memory-to-memory driver for the AVE H.264/HEVC encoder in Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. - The encoder firmware is loaded by the bootloader. The constant - parts of its session commands are read from apple/ave--*.bin - in the firmware search path, extracted from the user's own macOS - installation. See Documentation/admin-guide/media/apple-ave.rst. + The encoder firmware is loaded by the bootloader; the driver builds + every command itself. See Documentation/admin-guide/media/apple-ave.rst. To compile this driver as a module, choose M here: the module will be called apple-ave. diff --git a/drivers/media/platform/apple/ave/Makefile b/drivers/media/platform/apple/ave/Makefile index 542a7adeae360..53bb6b554f770 100644 --- a/drivers/media/platform/apple/ave/Makefile +++ b/drivers/media/platform/apple/ave/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_VIDEO_APPLE_AVE) += apple-ave.o -apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-sess.o ave-v4l2.o +apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-image.o ave-sess.o ave-v4l2.o diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index 922db11c98b80..e453700693f5d 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -2,15 +2,13 @@ /* * Apple AVE video encoder: firmware commands * - * Start and Process are large command images. Most of their words never - * change between sessions of a firmware build; the driver takes those from - * a per-build image file (see ave_fw_templates()) and writes every field - * that depends on the session or the frame itself. + * Start and Process are large command images. Each is built in a zeroed + * buffer: the header, the constant words (ave-image.c), then every field + * that depends on the session or the frame. * * Copyright The Aurora Silicon Contributors */ #include -#include #include #include #include @@ -217,9 +215,6 @@ int ave_cmd_close(struct ave_ctx *ctx) /* Start (id 6) */ -#define AVE_START_H264_BYTES 0x3180 -#define AVE_START_HEVC_BYTES 0x13f28 - #define AVE_START_SURFACE 0x48 /* and an unused second one at 0x58 */ #define AVE_START_SURFACE_BYTES 0x12c000 #define AVE_START_SURFACE2 0x58 @@ -277,7 +272,7 @@ static const u16 ave_start_tile_off[AVE_TILE_BUFS] = { #define AVE_START_CODEC_FLAGS 0x370 #define AVE_START_H264_380 0x380 #define AVE_START_H264_384 0x384 -#define AVE_START_H264_HIGH 0x2588 +#define AVE_START_CODEC_2588 0x2588 /* 2 for H.264 High and HEVC */ #define AVE_START_BIT_DEPTH 0x2598 #define AVE_START_CODED_H2 0x25dc @@ -428,7 +423,7 @@ static void ave_start_config(const struct ave_params *p, u8 *cmd) if (!hevc) { ave_put32(cmd, AVE_START_H264_384, p->width * p->height <= 320 * 240 ? 0x40 : 0x10); - ave_put32(cmd, AVE_START_H264_HIGH, p->high ? 2 : 0); + ave_put32(cmd, AVE_START_CODEC_2588, p->high ? 2 : 0); ave_put32(cmd, AVE_START_H264_PROFILE, p->profile); ave_put32(cmd, AVE_START_H264_LEVEL, p->level); ave_put32(cmd, AVE_START_H264_MAX_REFS, p->refs - 1); @@ -468,33 +463,35 @@ static void ave_start_config(const struct ave_params *p, u8 *cmd) ave_put32(cmd, AVE_START_HEVC_CONF_BOTTOM, crop_b); ave_put32(cmd, AVE_START_HEVC_DEPTH_LUMA, 0); ave_put32(cmd, AVE_START_HEVC_DEPTH_CHROMA, 0); - /* Only the B-frame structure is described for HEVC (see ave-sess.c). */ - ave_put32(cmd, AVE_START_HEVC_REFS, 4); + /* Reference structure: B-frame pyramid or low-delay P (ave-image.c) */ + ave_put32(cmd, AVE_START_HEVC_REFS, p->bframes ? 4 : 1); ave_put32(cmd, AVE_START_HEVC_DPB_MINUS1, 4); - ave_put32(cmd, AVE_START_HEVC_REORDER, 2); - ave_put32(cmd, AVE_START_HEVC_REORDER_COPY, 2); - ave_put32(cmd, AVE_START_HEVC_REORDER_ON, 1); + ave_put32(cmd, AVE_START_HEVC_REORDER, p->bframes ? 2 : 0); + ave_put32(cmd, AVE_START_HEVC_REORDER_COPY, p->bframes ? 2 : 0); + ave_put32(cmd, AVE_START_HEVC_REORDER_ON, p->bframes); } int ave_cmd_start(struct ave_ctx *ctx) { struct ave_dev *ave = ctx->ave; struct ave_session *sess = ctx->sess; - const struct firmware *tmpl = ave->tmpl[sess->p.codec - 1][AVE_TMPL_START]; + const struct ave_params *p = &sess->p; + size_t len = p->codec == AVE_CODEC_HEVC ? AVE_START_HEVC_BYTES : AVE_START_H264_BYTES; u8 *cmd; int ret; - cmd = ave_ipc_cmd(ave, AVE_BUF_CMD, tmpl->size); + cmd = ave_ipc_cmd(ave, AVE_BUF_CMD, len); if (!cmd) return -EBUSY; - memcpy(cmd, tmpl->data, tmpl->size); + ave_image_start(cmd, len, p->codec, p->reorder_off, + p->codec == AVE_CODEC_HEVC && !p->bframes); ave_cmd_header(ctx, cmd, AVE_OP_START, AVE_OP_START, AVE_SESSION_ARG0, ave_session_setup_arg2(ctx)); ave_start_surfaces(sess, cmd); ave_start_config(&sess->p, cmd); - ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_START, 0); + ave_debugfs_save(ave, p->codec, AVE_IMAGE_START, cmd, len); - ret = ave_ipc_submit(ave, AVE_BUF_CMD, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, true); + ret = ave_ipc_submit(ave, AVE_BUF_CMD, len, AVE_CMD_SESSION_TIMEOUT_US, true); if (ret) return ret; sess->handed = true; @@ -523,6 +520,7 @@ struct ave_proc_layout { u32 frame_buf; /* H.264 only */ u32 shdr_table; /* HEVC only */ u32 crop; /* HEVC only */ + u32 low_delay; /* HEVC only: 3 with reordering off */ }; static const struct ave_proc_layout ave_proc_h264 = { @@ -556,12 +554,14 @@ static const struct ave_proc_layout ave_proc_hevc = { .force_key = 0x6128, .force_type = 0x74c8, .shdr_table = 0xd88, .crop = 0x74dc, + .low_delay = 0xcf8, }; #define AVE_PROC_TIMESCALE 14400000ULL #define AVE_PROC_SHDR_ENTRIES 256 #define AVE_PROC_SHDR_STRIDE 0x400 #define AVE_PROC_FORCED_TYPE 2 +#define AVE_PROC_LOW_DELAY 3 int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, dma_addr_t input, u32 bitrate, bool force_key) @@ -571,7 +571,7 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff const struct ave_params *p = &sess->p; const struct ave_proc_layout *l = p->codec == AVE_CODEC_HEVC ? &ave_proc_hevc : &ave_proc_h264; - const struct firmware *tmpl = ave->tmpl[p->codec - 1][AVE_TMPL_PROCESS]; + size_t len = p->codec == AVE_CODEC_HEVC ? AVE_PROCESS_HEVC_BYTES : AVE_PROCESS_H264_BYTES; struct ave_slot *s = &sess->slot[slot]; unsigned int buf = AVE_BUF_PROC(sess->index * AVE_MAX_SLOTS + slot); dma_addr_t y = input; @@ -582,10 +582,10 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff ret = ave_ipc_wait_ack(ave, buf); if (ret) return ret; - cmd = ave_ipc_cmd(ave, buf, tmpl->size); + cmd = ave_ipc_cmd(ave, buf, len); if (!cmd) return -EBUSY; - memcpy(cmd, tmpl->data, tmpl->size); + ave_image_process(cmd, len, p->codec); ave_cmd_header(ctx, cmd, AVE_OP_PROCESS, AVE_PROC_SUBTYPE + slot, 0, ave_session_arg2(ctx)); @@ -631,9 +631,12 @@ int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff ave_put64(cmd, l->pts, pts); ave_put32(cmd, l->force_key, force_key); ave_put32(cmd, l->force_type, force_key ? AVE_PROC_FORCED_TYPE : 0); - ave_sess_check_addrs(ctx, cmd, tmpl->size, AVE_TMPL_PROCESS, y); + if (l->low_delay) + ave_put32(cmd, l->low_delay, p->bframes ? 0 : AVE_PROC_LOW_DELAY); + if (!sess->frames) + ave_debugfs_save(ave, p->codec, AVE_IMAGE_PROCESS, cmd, len); - ret = ave_ipc_submit(ave, buf, tmpl->size, AVE_CMD_SESSION_TIMEOUT_US, false); + ret = ave_ipc_submit(ave, buf, len, AVE_CMD_SESSION_TIMEOUT_US, false); if (ret) return ret; s->src = src; diff --git a/drivers/media/platform/apple/ave/ave-drv.c b/drivers/media/platform/apple/ave/ave-drv.c index 6d67ddf3780e1..b25ac9df48a67 100644 --- a/drivers/media/platform/apple/ave/ave-drv.c +++ b/drivers/media/platform/apple/ave/ave-drv.c @@ -9,6 +9,7 @@ * * Copyright The Aurora Silicon Contributors */ +#include #include #include #include @@ -21,6 +22,51 @@ #define AVE_AUTOSUSPEND_MS 2000 +/* + * debugfs: apple-ave/last_start_{h264,hevc} hold the last Start built and + * last_process0_{h264,hevc} the first Process of the last session, so the + * generated commands can be compared with reference captures. + */ +static const size_t ave_image_bytes[2][AVE_IMAGE_COUNT] = { + { AVE_START_H264_BYTES, AVE_PROCESS_H264_BYTES }, + { AVE_START_HEVC_BYTES, AVE_PROCESS_HEVC_BYTES }, +}; + +void ave_debugfs_save(struct ave_dev *ave, enum ave_codec codec, enum ave_image kind, + const void *cmd, size_t len) +{ + void *dst = ave->dump[codec - 1][kind]; + + if (dst && len == ave_image_bytes[codec - 1][kind]) + memcpy(dst, cmd, len); +} + +static void ave_debugfs_init(struct ave_dev *ave) +{ + static const char * const names[2][AVE_IMAGE_COUNT] = { + { "last_start_h264", "last_process0_h264" }, + { "last_start_hevc", "last_process0_hevc" }, + }; + + if (!IS_ENABLED(CONFIG_DEBUG_FS)) + return; + ave->debugfs = debugfs_create_dir("apple-ave", NULL); + for (unsigned int c = 0; c < 2; c++) + for (unsigned int k = 0; k < AVE_IMAGE_COUNT; k++) { + struct debugfs_blob_wrapper *blob; + + blob = devm_kzalloc(ave->dev, sizeof(*blob), GFP_KERNEL); + ave->dump[c][k] = devm_kzalloc(ave->dev, ave_image_bytes[c][k], GFP_KERNEL); + if (!blob || !ave->dump[c][k]) { + ave->dump[c][k] = NULL; + continue; + } + blob->data = ave->dump[c][k]; + blob->size = ave_image_bytes[c][k]; + debugfs_create_blob(names[c][k], 0400, ave->debugfs, blob); + } +} + static const char * const ave_pd_names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; static int ave_probe(struct platform_device *pdev) @@ -78,6 +124,7 @@ static int ave_probe(struct platform_device *pdev) ave_fw_remove(ave); return ret; } + ave_debugfs_init(ave); return 0; } @@ -85,6 +132,7 @@ static void ave_remove(struct platform_device *pdev) { struct ave_dev *ave = platform_get_drvdata(pdev); + debugfs_remove_recursive(ave->debugfs); ave_v4l2_unregister(ave); /* @@ -172,11 +220,5 @@ static struct platform_driver ave_driver = { }; module_platform_driver(ave_driver); -/* Session images, extracted from the user's macOS installation */ -MODULE_FIRMWARE("apple/ave-j700-h264-start.bin"); -MODULE_FIRMWARE("apple/ave-j700-h264-process.bin"); -MODULE_FIRMWARE("apple/ave-j700-hevc-start.bin"); -MODULE_FIRMWARE("apple/ave-j700-hevc-process.bin"); - MODULE_DESCRIPTION("Apple AVE video encoder"); MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index 5db179711c0a1..4987a32ad5d42 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -13,7 +13,6 @@ #include #include #include -#include #include #include #include @@ -21,7 +20,6 @@ #include #include #include -#include #include #include "ave.h" @@ -106,21 +104,6 @@ static u32 ave_fw_crc(const struct ave_fw_segment *seg, void **copy) return crc; } -/* - * The session images are per machine: "j700" from "apple,j700", the first - * root compatible. - */ -static void ave_fw_board_name(struct ave_dev *ave) -{ - const char *compat = NULL, *comma; - - of_property_read_string_index(of_root, "compatible", 0, &compat); - if (!compat) - compat = "unknown"; - comma = strchr(compat, ','); - strscpy(ave->board, comma ? comma + 1 : compat, sizeof(ave->board)); -} - /* * Find the firmware, check that it is a build we speak to and that DATA is * still as the bootloader left it, and keep a copy of DATA for restarts. @@ -151,8 +134,6 @@ int ave_fw_probe(struct ave_dev *ave) return dev_err_probe(dev, -ENODEV, "unsupported firmware (text CRC %08x)\n", text_crc); ave->text_crc = text_crc; - mutex_init(&ave->tmpl_lock); - ave_fw_board_name(ave); ave->data_pristine = vmalloc(ave->data.size); if (!ave->data_pristine) @@ -226,67 +207,20 @@ int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long } /* - * Start and Process are mostly constant for a machine and its firmware. - * Those constant words come from image files named after the machine, - * which the user extracts from their own macOS installation; the driver - * fills in every session and frame field itself. + * The driver generates Start and Process for one firmware build; other + * builds may lay them out differently, so they are not driven. */ -static const char * const ave_codec_name[] = { "h264", "hevc" }; -static const char * const ave_tmpl_name[AVE_TMPL_COUNT] = { "start", "process" }; -static const size_t ave_tmpl_size[2][AVE_TMPL_COUNT] = { - { 0x3180, 0x63d8 }, /* H.264 */ - { 0x13f28, 0xb1c0 }, /* HEVC */ -}; - -#define AVE_TMPL_CODEC 0xc0 /* Start: codec word of the session image */ - -int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec) +int ave_fw_check_build(struct ave_dev *ave) { - unsigned int c = codec - 1; - char name[64]; - int ret = 0; - - mutex_lock(&ave->tmpl_lock); - for (unsigned int t = 0; t < AVE_TMPL_COUNT && !ret; t++) { - const struct firmware *fw; - - if (ave->tmpl[c][t]) - continue; - snprintf(name, sizeof(name), "apple/ave-%s-%s-%s.bin", ave->board, - ave_codec_name[c], ave_tmpl_name[t]); - ret = request_firmware(&fw, name, ave->dev); - if (ret) { - dev_err(ave->dev, "cannot load %s (%d); extract it from macOS first\n", - name, ret); - break; - } - if (fw->size != ave_tmpl_size[c][t]) { - dev_err(ave->dev, "%s is %zu bytes, expected %zu\n", name, fw->size, - ave_tmpl_size[c][t]); - release_firmware(fw); - ret = -EINVAL; - break; - } - if (t == AVE_TMPL_START && get_unaligned_le32(fw->data + AVE_TMPL_CODEC) != codec) { - dev_err(ave->dev, "%s is not a %s session image\n", name, - ave_codec_name[c]); - release_firmware(fw); - ret = -EINVAL; - break; - } - ave->tmpl[c][t] = fw; - } - mutex_unlock(&ave->tmpl_lock); - return ret; + if (ave->text_crc == AVE_FW_TEXT_CRC_9003_78_0) + return 0; + dev_err(ave->dev, "encoder firmware %08x is not validated for encoding (only %08x is)\n", + ave->text_crc, AVE_FW_TEXT_CRC_9003_78_0); + return -ENOTSUPP; } void ave_fw_remove(struct ave_dev *ave) { - for (unsigned int c = 0; c < 2; c++) - for (unsigned int t = 0; t < AVE_TMPL_COUNT; t++) { - release_firmware(ave->tmpl[c][t]); - ave->tmpl[c][t] = NULL; - } ave_free(ave, &ave->heap); ave_free(ave, &ave->log); ave_free(ave, &ave->boot_cfg); diff --git a/drivers/media/platform/apple/ave/ave-image.c b/drivers/media/platform/apple/ave/ave-image.c new file mode 100644 index 0000000000000..f2f15f3c3d83a --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-image.c @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: constant words of the session commands + * + * macOS builds Start and Process in code, from the session settings and + * its own defaults; nothing is stored. Besides the fields the driver + * computes (sizes, geometry, profile, level, rate control, GOP, buffer + * addresses, per-frame values), every captured command on firmware + * AppleAVE2FW 9003.78.0 holds the same small set of constant words. They + * are listed here as facts, grouped by where they sit; for most of them + * the meaning is not known, only that macOS always sets that value. The + * commands are built in a zeroed buffer: these words first, then the + * computed fields. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +/* @count consecutive u32 words at @off, each set to @val. */ +struct ave_word { + u32 off; + u16 count; + u32 val; +}; + +#define W(o, v) { .off = (o), .count = 1, .val = (v) } +#define RUN(o, n, v) { .off = (o), .count = (n), .val = (v) } + +/* IEEE 754 doubles, high words only (the low words are zero) */ +#define F64_HI_1_0 0x3ff00000 /* 1.0 */ +#define F64_HI_0_5 0x3fe00000 /* 0.5 */ +#define F64_HI_M1_0 0xbff00000 /* -1.0 */ + +/* Session configuration words common to both codecs (Start +0x68..) */ +static const struct ave_word ave_start_common[] = { + /* First surface size: always the 0x12c000 header surface */ + W(0x50, 0x12c000), + W(0x94, 0x36), + W(0x214, 0x39d), + W(0x218, 0x18), + /* Rate-control doubles: 1.0, 0.5 (the 0.5 s VBV buffer), -1, -1 */ + W(0x28c, F64_HI_1_0), + W(0x294, F64_HI_0_5), + W(0x2c4, F64_HI_M1_0), + W(0x2cc, F64_HI_M1_0), + W(0x2d0, 1), + W(0x2e4, 1), + RUN(0x2f4, 3, 1), + RUN(0x338, 2, 0xffffffff), + /* Buffer set: metadata sizes of slots 0..2, +0x13d0 surface size */ + W(0xf58, 0xc000), + W(0xf68, 0xc000), + W(0xf78, 0xc000), + W(0x13d8, 0x4000), +}; + +static const struct ave_word ave_h264_start[] = { + /* Codec (1) and four flags after it */ + RUN(0xc0, 5, 1), + W(0x370, 2), + /* Codec settings block (+0x2580..): see also ave_start_codec_bytes() */ + W(0x258c, 3), + W(0x2594, 1), + W(0x2598, 8), /* bit depth */ + W(0x25a4, 0x100), + W(0x25cc, 1), + W(0x25d4, 1), + W(0x26dc, 4), + W(0x26f8, 0x15), + RUN(0x2704, 2, 1), + W(0x2734, 1), + W(0x2738, 0x10), + W(0x27c0, 8), + /* Sequence and picture parameter block (+0x2900..) */ + W(0x2940, 1), + W(0x2970, 0x100000), + RUN(0x2974, 239, 0x100010), + W(0x2d30, 0x10), + W(0x2d34, 1), + W(0x2d3c, 2), + RUN(0x2d54, 2, 1), + W(0x2d5c, 0x1000000), + W(0x2d60, 5), + W(0x2d68, 1), + RUN(0x2d6c, 3, 2), + W(0x2dc8, 1), + /* Two runs of -13 */ + RUN(0x3004, 7, 0xfffffff3), + RUN(0x302c, 7, 0xfffffff3), + W(0x3048, 1), + /* Execution-unit info */ + W(0x3150, 1), + W(0x3158, 1), +}; + +static const struct ave_word ave_hevc_start[] = { + /* Codec (2) and four flags after it */ + W(0xc0, 2), + RUN(0xc4, 4, 1), + W(0x370, 0x2000002), + W(0x1358, 0x4000), /* +0x1350 surface size, fixed for HEVC */ + /* Codec settings block (+0x2580..): see also ave_start_codec_bytes() */ + W(0x2588, 2), + W(0x2594, 1), + W(0x2598, 8), /* bit depth */ + W(0x25a4, 0x100), + W(0x25cc, 1), + W(0x25d4, 1), + W(0x26dc, 4), + RUN(0x26f8, 2, 0x15), + W(0x2700, 2), + RUN(0x2704, 2, 1), + W(0x2710, 1), + W(0x2734, 1), + W(0x2738, 0x10), + W(0x27c0, 8), + /* Profile, tier and level block, first copy (+0x2920..) */ + W(0x2920, 0x101), + W(0x292c, 1), + W(0x2934, 1), + W(0x295c, 0x1010001), + W(0x2964, 1), + RUN(0x2a8c, 7, 0x1010101), + W(0x2aa8, 0x101), + W(0x2abc, 0x1010101), + W(0x2ac0, 0x101), + RUN(0x2e38, 7, 4), + /* Timing pair 1001 / 24000, also at +0x74f0 */ + W(0x3298, 1001), + W(0x329c, 24000), + /* Sequence parameter block, second copy (+0x58f0..) */ + W(0x58f8, 1), + W(0x5900, 1), + W(0x5928, 0x1010001), + W(0x5930, 1), + W(0x5aac, 1), + W(0x5ae0, 7), /* log2_max_pic_order_cnt_lsb_minus4: 11-bit POC */ + RUN(0x5ae8, 7, 4), /* ends with sps_max_dec_pic_buffering_minus1 */ + W(0x5b40, 2), + W(0x5b48, 3), + W(0x5b4c, 1), + W(0x5b54, 1), + /* Picture parameter and slice defaults (+0x7400..) */ + W(0x7400, 0x100), + RUN(0x7404, 2, 7), + W(0x740c, 1), + W(0x7414, 1), + W(0x749c, 1), + W(0x74b0, 0x100), + W(0x74b4, 5), + W(0x74bc, 1), + RUN(0x74c0, 3, 2), + W(0x74f0, 1001), + W(0x74f4, 24000), + W(0x7500, 0x101), + W(0x7504, 1), + W(0x7508, 3), + W(0x750c, 5), + RUN(0x7510, 3, 0x17), + RUN(0x7524, 2, 0xbebb), + W(0x7558, 0x100), + /* Rate-control block (+0x9600..), with two runs of -13 */ + W(0x960c, 2), + RUN(0x9614, 7, 0xfffffff3), + RUN(0x9634, 7, 0xfffffff3), + W(0x9654, 0x100), + W(0x9e64, 0x10000), + W(0xb734, 2), + W(0xb774, 1), + /* Execution-unit info */ + RUN(0x13ef4, 2, 1), + W(0x13f00, 1), +}; + +/* + * HEVC with frame reordering off: a low-delay P reference structure (the + * reference picture sets hold the delta POCs -1 .. -4). All zero with + * B-frames. + */ +static const struct ave_word ave_hevc_start_low_delay[] = { + W(0xe120, 4), W(0xe154, 4), W(0xe17c, 0x01010101), + W(0xe1dc, 4), W(0xe1e4, 0x01010101), + W(0xe204, -1), W(0xe208, -2), W(0xe20c, -3), W(0xe210, -4), + W(0xe244, 1), W(0xe248, 2), W(0xe24c, 3), W(0xe250, 4), + W(0xe284, 4), W(0xe2b8, 1), W(0xe2e0, 1), W(0xe340, 1), W(0xe348, 1), + W(0xe368, -1), W(0xe3a8, 1), W(0xe3e8, 1), + W(0xe41c, 2), W(0xe444, 0x101), W(0xe4a4, 2), W(0xe4ac, 0x101), + W(0xe4cc, -1), W(0xe4d0, -2), + W(0xe50c, 1), W(0xe510, 2), W(0xe54c, 2), + W(0xe580, 3), W(0xe5a8, 0x10101), W(0xe608, 3), W(0xe610, 0x10101), + W(0xe630, -1), W(0xe634, -2), W(0xe638, -3), + W(0xe670, 1), W(0xe674, 2), W(0xe678, 3), W(0xe6b0, 3), +}; + +/* Process: the first-frame flag and per-frame values are computed. */ +static const struct ave_word ave_h264_process[] = { + /* Codec block */ + W(0x99c, 1), + W(0x9a8, 2), + W(0x9bc, 1), + W(0x9d4, 1), + W(0xeb4, 1), + W(0x12bc, 0xffffffff), + /* Picture-management block */ + W(0x29fc, 0xffffffff), + W(0x6238, 5), + W(0x6278, 14400000), /* presentation timescale, low word */ +}; + +static const struct ave_word ave_hevc_process[] = { + /* Codec block */ + W(0x9a8, 1), + W(0x9b8, 2), + W(0xab8, 0x10100), + W(0xb44, 0x10000), + RUN(0xb4c, 2, 5), + RUN(0xd70, 2, 0xffffffff), + W(0xd7c, 1), + /* Picture-management block */ + W(0x74cc, 0xffffffff), + W(0x74d0, 0x100), + W(0x74e0, 1), + W(0xad08, 5), + W(0xad3c, 0x403e0000), /* frame rate 30.0, rewritten per session */ + W(0xad48, 14400000), /* presentation timescale, low word */ + W(0xad68, 0xffffffff), +}; + +static void ave_put_words(u8 *cmd, size_t len, const struct ave_word *w, size_t n) +{ + for (size_t i = 0; i < n; i++) + for (unsigned int j = 0; j < w[i].count; j++) { + size_t off = w[i].off + 4 * j; + + if (WARN_ON_ONCE(off + 4 > len)) + return; + put_unaligned_le32(w[i].val, cmd + off); + } +} + +/* + * The 8 bytes at Start +0x2580 look like a DART address but are separate + * settings: byte 0 is 4 for HEVC, byte 3 is 1, byte 5 is 1 unless the + * client turned frame reordering off (H.264 only; HEVC keeps 1). + */ +static void ave_start_codec_bytes(u8 *cmd, enum ave_codec codec, bool reorder_off) +{ + cmd[0x2580] = codec == AVE_CODEC_HEVC ? 4 : 0; + cmd[0x2583] = 1; + cmd[0x2585] = codec == AVE_CODEC_H264 && reorder_off ? 0 : 1; +} + +void ave_image_start(u8 *cmd, size_t len, enum ave_codec codec, bool reorder_off, + bool hevc_low_delay) +{ + ave_put_words(cmd, len, ave_start_common, ARRAY_SIZE(ave_start_common)); + if (codec == AVE_CODEC_HEVC) { + ave_put_words(cmd, len, ave_hevc_start, ARRAY_SIZE(ave_hevc_start)); + if (hevc_low_delay) + ave_put_words(cmd, len, ave_hevc_start_low_delay, + ARRAY_SIZE(ave_hevc_start_low_delay)); + } else { + ave_put_words(cmd, len, ave_h264_start, ARRAY_SIZE(ave_h264_start)); + } + ave_start_codec_bytes(cmd, codec, reorder_off); +} + +void ave_image_process(u8 *cmd, size_t len, enum ave_codec codec) +{ + if (codec == AVE_CODEC_HEVC) + ave_put_words(cmd, len, ave_hevc_process, ARRAY_SIZE(ave_hevc_process)); + else + ave_put_words(cmd, len, ave_h264_process, ARRAY_SIZE(ave_h264_process)); +} diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index b96a84ad6d1a8..b3c12bc184dea 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -27,6 +27,10 @@ #define AVE_RC_VBR 4 #define AVE_RC_CBR 2 +/* Filler-data NAL unit types */ +#define AVE_H264_NAL_FILLER 12 +#define AVE_HEVC_NAL_FILLER 38 + #define AVE_H264_BASELINE 2 #define AVE_H264_MAIN 4 #define AVE_H264_HIGH 6 @@ -141,18 +145,14 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) p->min_qp = p->max_qp; /* - * Reference structures as macOS sets them up. H.264 has all three - * (B-frames, P only, intra only). HEVC without B-frames needs a - * reference-set table whose contents are not known yet, so HEVC - * always uses B-frames. + * Reference structures as macOS sets them up: a B-frame pyramid, + * P frames only (low delay), or (H.264) intra only. */ p->intra_only = p->codec == AVE_CODEC_H264 && ctx->gop_size == 1; - p->bframes = p->codec == AVE_CODEC_HEVC || - (!baseline && !p->intra_only && ctx->b_frames); + p->bframes = !baseline && !p->intra_only && ctx->b_frames; + p->reorder_off = !ctx->b_frames; p->refs = p->intra_only ? 1 : p->bframes ? 5 : 2; p->slots = p->bframes ? 5 : 3; - /* HEVC asked to go without B-frames: flush every frame instead. */ - p->low_delay = p->codec == AVE_CODEC_HEVC && !ctx->b_frames; p->gop = ctx->gop_size ?: 120; p->bitrate = ctx->bitrate; @@ -584,8 +584,9 @@ static bool ave_starts_annexb(const u8 *p, size_t len) (len >= 4 && !p[0] && !p[1] && !p[2] && p[3] == 1); } -/* NAL unit types in an Annex B buffer: is there a key frame / parameter set? */ -static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *params) +/* NAL unit types in an Annex B buffer: key frame, parameter sets, filler? */ +static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *params, + bool *filler) { for (size_t i = 0; i + 3 < len; i++) { u32 type; @@ -596,9 +597,11 @@ static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *p if (hevc) { *key |= type >= 16 && type <= 21; *params |= type >= 32 && type <= 34; + *filler |= type == AVE_HEVC_NAL_FILLER; } else { *key |= type == 5; *params |= type == 7; + *filler |= type == AVE_H264_NAL_FILLER; } i += 2; } @@ -649,6 +652,65 @@ static void ave_sess_dump(struct ave_ctx *ctx, struct ave_slot *s, const char *w dev_dbg(ctx->ave->dev, "metadata +%#x: %16ph\n", i, meta + i); } +/* + * CBR padding. macOS streams in CBR mode follow a buffer model: each frame + * has a budget of bitrate / (8 x fps) bytes, and once the bytes left unused + * add up to more than 0.5 s of bit rate (the 0.5 s VBV buffer of Start + * +0x290), every frame is padded with a filler-data NAL unit so the surplus + * stays there. Whether the firmware or macOS user space writes that filler + * is not known; if the firmware's output never carries filler, the driver + * adds it here. + */ +#define AVE_FILLER_OVERHEAD 7 /* start code, NAL header, trailing bits */ + +static size_t ave_sess_cbr_fill(struct ave_ctx *ctx, u8 *out, size_t len, size_t room, + bool fw_filler) +{ + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + s64 budget = div_u64((u64)p->bitrate * p->fps_den, 8ULL * p->fps_num); + s64 vbv = p->bitrate / 16; /* 0.5 s of bit rate, in bytes */ + s64 excess, pad; + u8 *f; + + if (p->rc_mode != AVE_RC_CBR) + return len; + if (fw_filler && !sess->fw_fills) { + sess->fw_fills = true; + dev_dbg(ctx->ave->dev, "CBR: the firmware writes filler data\n"); + } + excess = sess->cbr_surplus + budget - (s64)len - vbv; + if (!sess->fw_fills && excess > AVE_FILLER_OVERHEAD) { + pad = min_t(s64, excess, room - len); + if (pad > AVE_FILLER_OVERHEAD) { + f = out + len; + f[0] = 0; + f[1] = 0; + f[2] = 0; + f[3] = 1; + if (hevc) { + /* nal_unit_type 38, layer 0, temporal id 0 + 1 */ + f[4] = AVE_HEVC_NAL_FILLER << 1; + f[5] = 1; + memset(f + 6, 0xff, pad - AVE_FILLER_OVERHEAD); + } else { + f[4] = AVE_H264_NAL_FILLER; + memset(f + 5, 0xff, pad - AVE_FILLER_OVERHEAD + 1); + } + f[pad - 1] = 0x80; /* rbsp_stop_one_bit */ + len += pad; + if (!sess->host_fill_noted) { + sess->host_fill_noted = true; + dev_dbg(ctx->ave->dev, + "CBR: no filler from the firmware, the driver adds it\n"); + } + } + } + sess->cbr_surplus += budget - (s64)len; + return len; +} + /* * Assemble the access unit of @slot into @dst: the parameter sets the * firmware leaves in the metadata buffer (when it emits them), for HEVC the @@ -666,7 +728,7 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff size_t room = vb2_plane_size(&dst->vb2_buf, 0); const u8 *meta = s->meta.cpu; u32 data, data2, shdr = 0; - bool params = false; + bool params = false, filler = false; ssize_t ps = 0; size_t len; int at; @@ -707,80 +769,14 @@ int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buff if (!ave_starts_annexb(out, len)) ave_sess_dump(ctx, s, "not Annex B"); - ave_scan_nals(out, len, hevc, keyframe, ¶ms); + ave_scan_nals(out, len, hevc, keyframe, ¶ms, &filler); if (!ctx->dst_sequence && !params) ave_sess_dump(ctx, s, "first frame has no parameter sets"); + len = ave_sess_cbr_fill(ctx, out, len, room, filler); vb2_set_plane_payload(&dst->vb2_buf, 0, len); return 0; } -/* - * Every 64-bit word in a command that looks like an encoder DART address - * should be one of ours. One that is not came from the session image (a - * stale macOS address) and makes the firmware write to an unmapped page. - * Reported once per session and command kind. - */ -static bool ave_buf_has(const struct ave_buf *b, u64 addr) -{ - return b->cpu && addr >= b->dma && addr < b->dma + b->alloc; -} - -static bool ave_sess_owns(struct ave_session *sess, u64 addr) -{ - const struct ave_buf *fixed[] = { - &sess->hdr, &sess->mb, &sess->row_a, &sess->row_b, &sess->fixed, - &sess->hevc_extra[0], &sess->hevc_extra[1], - }; - - for (unsigned int i = 0; i < ARRAY_SIZE(fixed); i++) - if (ave_buf_has(fixed[i], addr)) - return true; - for (unsigned int i = 0; i < AVE_MAX_REFS; i++) - if (ave_buf_has(&sess->ref_a[i], addr) || ave_buf_has(&sess->ref_b[i], addr) || - ave_buf_has(&sess->ref_c[i], addr) || ave_buf_has(&sess->ref_d[i], addr)) - return true; - for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) - if (ave_buf_has(&sess->tile[i], addr) || ave_buf_has(&sess->tile_aux[i], addr)) - return true; - for (unsigned int s = 0; s < AVE_MAX_SLOTS; s++) - if (ave_buf_has(&sess->slot[s].coded, addr) || - ave_buf_has(&sess->slot[s].meta, addr) || - ave_buf_has(&sess->slot[s].aux, addr)) - return true; - return false; -} - -void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsigned int kind, - dma_addr_t input) -{ - struct ave_session *sess = ctx->sess; - struct ave_dev *ave = ctx->ave; - unsigned int bad = 0; - - if (sess->checked[kind]) - return; - sess->checked[kind] = true; - for (size_t off = AVE_CMD_HDR_BYTES; off + 8 <= len; off += 4) { - u64 v = get_unaligned_le64(cmd + off); - - /* The encoder's DART window: 0x100_0000_0000 + 256 MiB. */ - if ((v >> 28) != 0x1000) - continue; - if (ave_sess_owns(sess, v) || (input && v >= input && v < input + SZ_32M) || - (v >= ave->ipc.dma && v < ave->ipc.dma + ave->ipc.size)) - continue; - /* Pointers into the firmware image are the same as on macOS. */ - if (v >= ave->text.iova && v < ave->data.iova + ave->data.size) - continue; - if (bad++ < 16) - dev_warn(ave->dev, "%s +%#zx holds %#llx, not a buffer of this session\n", - kind == AVE_TMPL_START ? "Start" : "Process", off, v); - } - if (bad) - dev_warn(ave->dev, "%u stale addresses in the %s image\n", bad, - kind == AVE_TMPL_START ? "Start" : "Process"); -} - /* * Frames are outstanding, nothing came back for a while, and the firmware * has enough frames that it should have answered: without B-frames every diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index caa874ba48de3..3d6b5fa2352b4 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -478,8 +478,7 @@ static int ave_init_ctrls(struct ave_ctx *ctx) v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); /* * The firmware either uses up to three B-frames (reordering them - * itself) or none. Baseline and intra-only H.264 never have any; - * HEVC always has them for now. + * itself) or none. Baseline and intra-only H.264 never have any. */ v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 3, 3, 3); v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEADER_MODE, @@ -740,7 +739,7 @@ static int ave_session_begin(struct ave_ctx *ctx) if (ctx->sess) return 0; - ret = ave_fw_templates(ave, ave_ctx_codec(ctx)); + ret = ave_fw_check_build(ave); if (ret) return ret; if (READ_ONCE(ave->sessions[0]) && READ_ONCE(ave->sessions[1])) @@ -1031,14 +1030,6 @@ static int ave_encode_one(struct ave_ctx *ctx) ret = ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); if (ret) return ret; - } else if (sess->p.low_delay) { - /* Make the firmware code this frame now, without lookahead. */ - ret = ave_cmd_complete(ctx, 1); - sess->nudged = true; - if (!ret) - ret = ave_ipc_pump(ave, 0, NULL, NULL); - if (ret) - return ret; } else { ret = ave_ipc_pump(ave, 0, NULL, NULL); if (ret) diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h index c5dce46faa373..0f6e2b38d86ca 100644 --- a/drivers/media/platform/apple/ave/ave.h +++ b/drivers/media/platform/apple/ave/ave.h @@ -19,8 +19,9 @@ #include #include +struct debugfs_blob_wrapper; +struct dentry; struct dev_pm_domain_list; -struct firmware; struct iommu_domain; /* Frames the firmware can hold at once per session (Process "slots"). */ @@ -93,13 +94,21 @@ enum ave_fw_state { AVE_FW_BROKEN, }; -/* Session command images that only differ from the listed fields. */ -enum ave_tmpl { - AVE_TMPL_START, - AVE_TMPL_PROCESS, - AVE_TMPL_COUNT, +/* The two large session commands, and their sizes per codec */ +enum ave_image { + AVE_IMAGE_START, + AVE_IMAGE_PROCESS, + AVE_IMAGE_COUNT, }; +#define AVE_START_H264_BYTES 0x3180 +#define AVE_START_HEVC_BYTES 0x13f28 +#define AVE_PROCESS_H264_BYTES 0x63d8 +#define AVE_PROCESS_HEVC_BYTES 0xb1c0 + +/* The firmware build whose command layouts the driver generates */ +#define AVE_FW_TEXT_CRC_9003_78_0 0x69c54454 + struct ave_ctx; struct ave_dev { @@ -113,13 +122,12 @@ struct ave_dev { struct ave_fw_segment text; struct ave_fw_segment data; u32 text_crc; - char board[16]; /* from the root compatible, e.g. "j700" */ void *data_pristine; bool data_dirty; - /* Protects tmpl[]; the images stay loaded until the driver goes. */ - struct mutex tmpl_lock; - const struct firmware *tmpl[2][AVE_TMPL_COUNT]; + /* debugfs: the last Start and first Process built, per codec */ + struct dentry *debugfs; + void *dump[2][AVE_IMAGE_COUNT]; /* Serialises every firmware and ring access. */ struct mutex hw_lock; @@ -161,7 +169,7 @@ struct ave_params { u32 refs; /* max_num_ref_frames + 1 */ bool bframes; bool intra_only; - bool low_delay; /* Complete after every frame */ + bool reorder_off; /* the client asked for no B-frames */ u32 gop; u32 rc_mode; u32 bitrate; @@ -212,6 +220,10 @@ struct ave_session { bool fw_open; /* Open sent, Close not yet */ bool resume; /* closed for system sleep, reopen on next frame */ bool copy_noted; + /* CBR: bytes under budget so far, and who pads (see ave_sess_cbr_fill()) */ + s64 cbr_surplus; + bool fw_fills; + bool host_fill_noted; bool warned; bool checked[2]; /* Start/Process images checked for stale addresses */ bool handed; /* Start sent: the firmware knows the buffers */ @@ -266,7 +278,7 @@ int ave_fw_ensure_running(struct ave_dev *ave); void ave_fw_halt(struct ave_dev *ave); int ave_fw_shutdown(struct ave_dev *ave); void ave_fw_lost(struct ave_dev *ave); -int ave_fw_templates(struct ave_dev *ave, enum ave_codec codec); +int ave_fw_check_build(struct ave_dev *ave); int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs); int ave_alloc_padded(struct ave_dev *ave, struct ave_buf *b, size_t size, size_t alloc, unsigned long attrs); @@ -303,11 +315,18 @@ int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffe int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, bool *keyframe); void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); -void ave_sess_check_addrs(struct ave_ctx *ctx, const u8 *cmd, size_t len, unsigned int kind, - dma_addr_t input); bool ave_sess_stalled(struct ave_session *sess); bool ave_sess_starved(struct ave_session *sess); +/* ave-image.c */ +void ave_image_start(u8 *cmd, size_t len, enum ave_codec codec, bool reorder_off, + bool hevc_low_delay); +void ave_image_process(u8 *cmd, size_t len, enum ave_codec codec); + +/* ave-drv.c */ +void ave_debugfs_save(struct ave_dev *ave, enum ave_codec codec, enum ave_image kind, + const void *cmd, size_t len); + /* ave-v4l2.c */ void ave_v4l2_suspend(struct ave_dev *ave); int ave_v4l2_register(struct ave_dev *ave); From 6a99b550f9055eeb643d5ceebfecc5de6b807a6a Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 22:16:08 +1100 Subject: [PATCH 18/20] media: apple: ave: fit the native panel size in the DMA window At 2408x1506 every H.264 setup and HEVC with B-frames ran out of the encoder's 256 MiB DART window. Three changes make room: - The H.264 per-frame buffer of Process +0x58c8 was sized after the macOS slot spacing (0x364000 at 1080p, scaled with the picture). The firmware never wrote a byte of it in any bit-rate session on hardware, so it is now one page. - Session buffers are mapped largest first. The DMA layer aligns every mapping to a power of two, and the smaller buffers now fill the gaps the large ones leave. The debug log reports how much a session maps, and an allocation failure says how far it got. - ffmpeg's lines are spread out in place when the OUTPUT buffer is one the driver allocated (MMAP), which is always large enough for the encoder's layout. That needs no extra DMA space; imported buffers still get a per-slot copy. The OUTPUT and CAPTURE queues get 48 and 24 MiB. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-sess.c | 139 ++++++++++++++------ drivers/media/platform/apple/ave/ave-v4l2.c | 4 +- 2 files changed, 102 insertions(+), 41 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index b3c12bc184dea..27ea45c76080f 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -48,9 +49,8 @@ #define AVE_CODED_MIN 0x3c000 #define AVE_HDR_BYTES 0x12c000 -/* Spacing of the H.264 per-frame buffer seen at 1080p; scaled with area. */ -#define AVE_H264_FRAME_BUF_1080P 0x364000 -#define AVE_MBS_1080P (120 * 68) +/* H.264 per-frame buffer (Process +0x58c8): unused in bit-rate modes */ +#define AVE_H264_FRAME_BUF SZ_16K /* Level limits: max frame size, max rate, max bit rate (kbit/s, Main). */ struct ave_level { @@ -198,12 +198,37 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) * driver can see how far past each reported size the firmware wrote (into * the page rounding, or the room the two groups above get). */ -static int ave_work_buf(struct ave_dev *ave, struct ave_buf *b, size_t size) + +/* One buffer to allocate: what the firmware is told and what is mapped. */ +struct ave_buf_req { + struct ave_buf *b; + size_t size; + size_t alloc; +}; + +#define AVE_MAX_BUF_REQS (1 + 4 * AVE_MAX_REFS + 4 + 2 * AVE_TILE_BUFS + 2 + \ + 3 * AVE_MAX_SLOTS) + +static int ave_buf_req_cmp(const void *a, const void *b) +{ + const struct ave_buf_req *x = a, *y = b; + + return x->alloc < y->alloc ? 1 : x->alloc > y->alloc ? -1 : 0; +} + +static void ave_req(struct ave_buf_req *r, unsigned int *n, struct ave_buf *b, size_t size, + size_t alloc) { - return ave_alloc(ave, b, size, 0); + r[*n] = (struct ave_buf_req){ .b = b, .size = size, .alloc = max(size, alloc) }; + (*n)++; } -/* Working buffer sizes, all from the requested picture size. */ +/* + * Working buffer sizes, all from the requested picture size. The DMA layer + * aligns each mapping to a power of two in the encoder's 256 MiB window, so + * the buffers are mapped largest first: the smaller ones then fill the + * alignment gaps the large ones leave. + */ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) { const struct ave_params *p = &sess->p; @@ -211,47 +236,63 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) u32 w = p->width, h = p->height; u32 w16 = DIV_ROUND_UP(w, 16), w32 = DIV_ROUND_UP(w, 32); u32 h32 = DIV_ROUND_UP(h, 32), w64 = DIV_ROUND_UP(w, 64), h64 = DIV_ROUND_UP(h, 64); - size_t ref_a, ref_b, ref_c, ref_d, tile, frame_buf; + size_t ref_a, ref_b, ref_c, ref_d, tile, mapped = 0; + struct ave_buf_req *r; + unsigned int n = 0; int ret = 0; + r = kcalloc(AVE_MAX_BUF_REQS, sizeof(*r), GFP_KERNEL); + if (!r) + return -ENOMEM; + ref_a = ALIGN(w, 32) * ALIGN(h + 16, 32); ref_b = roundup_pow_of_two(w) * roundup_pow_of_two(h) / 32; ref_c = hevc ? 512 * w64 * h64 : ALIGN(w, 64) * (ALIGN(h, 16) / 2 + 24); ref_d = ALIGN(hevc ? 64 * w32 * h32 : 128 * p->mbs, SZ_16K); tile = hevc ? 2304 * w32 * h64 : ALIGN(w, 16) * ALIGN(h, 64); - ret |= ave_work_buf(ave, &sess->hdr, AVE_HDR_BYTES); + ave_req(r, &n, &sess->hdr, AVE_HDR_BYTES, 0); for (unsigned int i = 0; i < p->refs; i++) { - ret |= ave_alloc_padded(ave, &sess->ref_a[i], ref_a, ref_a * 3 / 2, 0); - ret |= ave_alloc_padded(ave, &sess->ref_b[i], ref_b, ref_b * 5 / 4, 0); - ret |= ave_work_buf(ave, &sess->ref_c[i], ref_c); - ret |= ave_work_buf(ave, &sess->ref_d[i], ref_d); - } - ret |= ave_work_buf(ave, &sess->mb, ALIGN(82 * p->mbs, SZ_16K)); - ret |= ave_work_buf(ave, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K)); - ret |= ave_work_buf(ave, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K)); - ret |= ave_work_buf(ave, &sess->fixed, SZ_16K); + ave_req(r, &n, &sess->ref_a[i], ref_a, ref_a * 3 / 2); + ave_req(r, &n, &sess->ref_b[i], ref_b, ref_b * 5 / 4); + ave_req(r, &n, &sess->ref_c[i], ref_c, 0); + ave_req(r, &n, &sess->ref_d[i], ref_d, 0); + } + ave_req(r, &n, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), 0); + ave_req(r, &n, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), 0); + ave_req(r, &n, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), 0); + ave_req(r, &n, &sess->fixed, SZ_16K, 0); for (unsigned int i = 0; i < (hevc ? 4 : AVE_TILE_BUFS); i++) { - ret |= ave_work_buf(ave, &sess->tile[i], tile); - ret |= ave_work_buf(ave, &sess->tile_aux[i], ALIGN(h, 128)); + ave_req(r, &n, &sess->tile[i], tile, 0); + ave_req(r, &n, &sess->tile_aux[i], ALIGN(h, 128), 0); } if (hevc) for (unsigned int i = 0; i < 2; i++) - ret |= ave_work_buf(ave, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K)); - - /* Per-slot output; the size of the H.264 frame buffer is not known. */ - frame_buf = ALIGN(DIV_ROUND_UP_ULL((u64)AVE_H264_FRAME_BUF_1080P * p->mbs, AVE_MBS_1080P), - SZ_16K); - frame_buf = max_t(size_t, frame_buf, AVE_H264_FRAME_BUF_1080P); + ave_req(r, &n, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), 0); for (unsigned int s = 0; s < p->slots; s++) { - ret |= ave_alloc(ave, &sess->slot[s].coded, p->coded_cap, 0); - ret |= ave_alloc(ave, &sess->slot[s].meta, AVE_META_BYTES, 0); - if (hevc) - ret |= ave_alloc(ave, &sess->slot[s].aux, AVE_SHDR_BYTES, 0); - else - ret |= ave_work_buf(ave, &sess->slot[s].aux, frame_buf); - } - return ret ? -ENOMEM : 0; + ave_req(r, &n, &sess->slot[s].coded, p->coded_cap, 0); + ave_req(r, &n, &sess->slot[s].meta, AVE_META_BYTES, 0); + /* + * HEVC: slice headers. H.264: the per-frame buffer of Process + * +0x58c8, which the firmware never wrote in any bit-rate + * mode on hardware; one page. + */ + ave_req(r, &n, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : AVE_H264_FRAME_BUF, 0); + } + + sort(r, n, sizeof(*r), ave_buf_req_cmp, NULL); + for (unsigned int i = 0; i < n && !ret; i++) { + ret = ave_alloc_padded(ave, r[i].b, r[i].size, r[i].alloc, 0); + mapped += r[i].b->alloc; + } + kfree(r); + if (ret) { + dev_err(ave->dev, "out of encoder DMA space after %zu MiB of session buffers\n", + mapped >> 20); + return -ENOMEM; + } + dev_dbg(ave->dev, "session buffers: %u, %zu KiB mapped\n", n, mapped >> 10); + return 0; } static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) @@ -281,6 +322,15 @@ static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) } } +static void ave_note_copy(struct ave_ctx *ctx, u32 stride, const char *how) +{ + if (ctx->sess->copy_noted) + return; + ctx->sess->copy_noted = true; + dev_info(ctx->ave->dev, "input has a %u byte stride; moving lines to %u %s\n", stride, + ctx->sess->p.stride, how); +} + /* * The input picture for @slot, as a DART address. The encoder reads NV12 * with the 64-byte aligned stride S_FMT reported and chroma right after @@ -320,20 +370,31 @@ int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffe return -EFAULT; } from += off; + + /* + * Buffers the driver allocated (MMAP) are large enough for the + * encoder's layout: spread the lines out in place, last line first, + * so no extra DMA space is needed. Imported buffers may be shared + * with other users and get a copy instead. + */ + if (vb->memory == VB2_MEMORY_MMAP && + vb2_plane_size(vb, 0) >= off + (size_t)p->stride * rows) { + to = (u8 *)from; + for (u32 y = rows; y-- > 0;) + memmove(to + y * p->stride, from + y * stride, width); + ave_note_copy(ctx, stride, "in place"); + return 0; + } + ret = ave_alloc(ctx->ave, b, p->stride * rows, 0); if (ret) { dev_err(ctx->ave->dev, "no room to copy %u byte stride input\n", stride); return ret; } - if (!sess->copy_noted) { - sess->copy_noted = true; - dev_info(ctx->ave->dev, "input has a %u byte stride; copying to %u\n", stride, - p->stride); - } - to = b->cpu; for (u32 y = 0; y < rows; y++) memcpy(to + y * p->stride, from + y * stride, width); + ave_note_copy(ctx, stride, "to a copy"); *input = b->dma; return 0; } diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index 3d6b5fa2352b4..b316489b7294d 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -40,8 +40,8 @@ * window next to a session: each queue may use this much, counting every * buffer at the power-of-two size the DMA layer maps it with. */ -#define AVE_OUTPUT_BUDGET SZ_64M -#define AVE_CAPTURE_BUDGET SZ_32M +#define AVE_OUTPUT_BUDGET (48 * SZ_1M) +#define AVE_CAPTURE_BUDGET (24 * SZ_1M) #define AVE_MAX_QUEUE_BUFS 16 #define AVE_RESTART_TRIES 10 #define AVE_RESTART_WAIT_MS 100 From ec7c3a7f7577ef3b564bffd1cfc3a44e5afa14ff Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 22:24:17 +1100 Subject: [PATCH 19/20] media: apple: ave: match macOS Start byte for byte, give GStreamer its buffers A byte comparison of the generated commands against macOS for the same settings (1080p30 Main, VBR 4 Mb/s, GOP 30, B-frames), with addresses, tokens and timestamps masked, found both Process commands identical and five Start words different: - H.264 +0x3020: byte +0x3020 is 1. The word was not in the constant list because its byte +0x3023 (8x8 transform) varies. - HEVC +0x2960 and +0x592c: the bytes around the computed max-8-bit flags are 1 (+0x2960, +0x2961, +0x2963 and +0x592f). - HEVC +0x270c = 2 and +0x9608 = 0x10000: the rate-control words that only change in constant-quality mode, at their bit-rate mode values. GStreamer stopped after five frames at 2408x1506: the v14 queue limits left it 6 OUTPUT and 3 CAPTURE buffers there. Sessions map much less now, so the queues get 80 and 40 MiB of the DMA window again (10 and 5 buffers at that size), and a trimmed or refused buffer request is logged at debug level. Field values are facts from our private research (macOS traces) and our own hardware runs; no copied code. Assisted-by: LLM Signed-off-by: Ryan Murray Co-Authored-By: Claude Opus 5.5 --- drivers/media/platform/apple/ave/ave-cmd.c | 7 +++++++ drivers/media/platform/apple/ave/ave-image.c | 6 ++++++ drivers/media/platform/apple/ave/ave-v4l2.c | 12 +++++++++--- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c index e453700693f5d..7a03ed53b457b 100644 --- a/drivers/media/platform/apple/ave/ave-cmd.c +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -292,6 +292,9 @@ static const u16 ave_start_tile_off[AVE_TILE_BUFS] = { /* HEVC parameter-set words; most appear twice */ #define AVE_START_HEVC_PAD 0x26e0 +#define AVE_START_HEVC_RC_270C 0x270c /* 2 in bit-rate modes, 5 in CQ */ +#define AVE_START_HEVC_RC_9604 0x9604 /* 0 in bit-rate modes, 2 in CQ */ +#define AVE_START_HEVC_RC_9608 0x9608 /* 0x10000 in bit-rate modes, 0 in CQ */ #define AVE_START_HEVC_PROFILE_A 0x2938 #define AVE_START_HEVC_COMPAT_A 0x293d /* two bytes */ #define AVE_START_HEVC_8BIT_A 0x2962 /* byte */ @@ -450,6 +453,10 @@ static void ave_start_config(const struct ave_params *p, u8 *cmd) cmd[AVE_START_HEVC_COMPAT_B + 1] = 1; cmd[AVE_START_HEVC_8BIT_A] = 1; cmd[AVE_START_HEVC_8BIT_B] = 1; + /* Rate-control words that only change in constant-quality mode */ + ave_put32(cmd, AVE_START_HEVC_RC_270C, 2); + ave_put32(cmd, AVE_START_HEVC_RC_9604, 0); + ave_put32(cmd, AVE_START_HEVC_RC_9608, 0x10000); ave_put32(cmd, AVE_START_HEVC_LEVEL_A, p->level); ave_put32(cmd, AVE_START_HEVC_LEVEL_B, p->level); ave_put32(cmd, AVE_START_HEVC_WIDTH, p->cw); diff --git a/drivers/media/platform/apple/ave/ave-image.c b/drivers/media/platform/apple/ave/ave-image.c index f2f15f3c3d83a..273dd2a69b013 100644 --- a/drivers/media/platform/apple/ave/ave-image.c +++ b/drivers/media/platform/apple/ave/ave-image.c @@ -88,6 +88,8 @@ static const struct ave_word ave_h264_start[] = { W(0x2d68, 1), RUN(0x2d6c, 3, 2), W(0x2dc8, 1), + /* Picture parameters: byte +0x3020 is 1; +0x3023 (8x8 transform) is computed */ + W(0x3020, 1), /* Two runs of -13 */ RUN(0x3004, 7, 0xfffffff3), RUN(0x302c, 7, 0xfffffff3), @@ -123,6 +125,8 @@ static const struct ave_word ave_hevc_start[] = { W(0x292c, 1), W(0x2934, 1), W(0x295c, 0x1010001), + /* Constraint flag bytes; +0x2962 (max 8 bit) is computed */ + W(0x2960, 0x01000101), W(0x2964, 1), RUN(0x2a8c, 7, 0x1010101), W(0x2aa8, 0x101), @@ -136,6 +140,8 @@ static const struct ave_word ave_hevc_start[] = { W(0x58f8, 1), W(0x5900, 1), W(0x5928, 0x1010001), + /* Constraint flag bytes, second copy; +0x592e (max 8 bit) is computed */ + W(0x592c, 0x01000000), W(0x5930, 1), W(0x5aac, 1), W(0x5ae0, 7), /* log2_max_pic_order_cnt_lsb_minus4: 11-bit POC */ diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c index b316489b7294d..0adbaaf8513d5 100644 --- a/drivers/media/platform/apple/ave/ave-v4l2.c +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -40,8 +40,8 @@ * window next to a session: each queue may use this much, counting every * buffer at the power-of-two size the DMA layer maps it with. */ -#define AVE_OUTPUT_BUDGET (48 * SZ_1M) -#define AVE_CAPTURE_BUDGET (24 * SZ_1M) +#define AVE_OUTPUT_BUDGET (80 * SZ_1M) +#define AVE_CAPTURE_BUDGET (40 * SZ_1M) #define AVE_MAX_QUEUE_BUFS 16 #define AVE_RESTART_TRIES 10 #define AVE_RESTART_WAIT_MS 100 @@ -545,8 +545,14 @@ static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned i if (V4L2_TYPE_IS_OUTPUT(vq->type) && have + *nbufs < AVE_MAX_SLOTS + 1) *nbufs = AVE_MAX_SLOTS + 1 - have; /* A full set of both queues must fit the encoder's 256 MiB DART window. */ - if (have >= max) + if (have >= max) { + dev_dbg(ctx->ave->dev, "%s queue already has its %u buffers\n", + V4L2_TYPE_IS_OUTPUT(vq->type) ? "OUTPUT" : "CAPTURE", max); return -ENOMEM; + } + if (*nbufs > max - have) + dev_dbg(ctx->ave->dev, "%s queue: %u buffers asked, %u given\n", + V4L2_TYPE_IS_OUTPUT(vq->type) ? "OUTPUT" : "CAPTURE", *nbufs, max - have); *nbufs = min(*nbufs, max - have); if (*nplanes) return (*nplanes != 1 || sizes[0] < f->plane_fmt[0].sizeimage) ? -EINVAL : 0; From f494c8495ea4cc14e528d963b83623ecbb9923d0 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Fri, 30 Oct 2026 23:11:47 +1100 Subject: [PATCH 20/20] media: apple: ave: drop the bring-up diagnostics The buffer sizes are settled on hardware, so remove what was only there to measure them: the scan of every working buffer's margin at the end of a session, which logged at info level each time, and the usage scan of the H.264 per-frame buffer. Working buffers no longer need a kernel mapping. The stride fix-up note and the retried restart of a still powered coprocessor are normal operation and move to debug level. Assisted-by: LLM Signed-off-by: Ryan Murray --- drivers/media/platform/apple/ave/ave-fw.c | 2 +- drivers/media/platform/apple/ave/ave-sess.c | 125 +++++--------------- 2 files changed, 29 insertions(+), 98 deletions(-) diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c index 4987a32ad5d42..9b4d3c3f67662 100644 --- a/drivers/media/platform/apple/ave/ave-fw.c +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -262,7 +262,7 @@ static int ave_fw_prepare_restart(struct ave_dev *ave) return 0; if (readl(ave->coproc + AVE_COPROC_CTRL)) { - dev_warn(ave->dev, "coprocessor kept power after its halt; retry once it suspends\n"); + dev_dbg(ave->dev, "coprocessor kept power after its halt; retry once it suspends\n"); return -EAGAIN; } status = readl(ave->mbox + AVE_MBOX_STATUS); diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c index 27ea45c76080f..e58d485eb61c1 100644 --- a/drivers/media/platform/apple/ave/ave-sess.c +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -193,17 +193,12 @@ static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) * mapping is only rounded up to whole pages. */ -/* - * Working buffers keep a kernel mapping so that, when a session ends, the - * driver can see how far past each reported size the firmware wrote (into - * the page rounding, or the room the two groups above get). - */ - /* One buffer to allocate: what the firmware is told and what is mapped. */ struct ave_buf_req { struct ave_buf *b; size_t size; size_t alloc; + unsigned long attrs; }; #define AVE_MAX_BUF_REQS (1 + 4 * AVE_MAX_REFS + 4 + 2 * AVE_TILE_BUFS + 2 + \ @@ -217,9 +212,11 @@ static int ave_buf_req_cmp(const void *a, const void *b) } static void ave_req(struct ave_buf_req *r, unsigned int *n, struct ave_buf *b, size_t size, - size_t alloc) + size_t alloc, unsigned long attrs) { - r[*n] = (struct ave_buf_req){ .b = b, .size = size, .alloc = max(size, alloc) }; + r[*n] = (struct ave_buf_req){ + .b = b, .size = size, .alloc = max(size, alloc), .attrs = attrs, + }; (*n)++; } @@ -236,6 +233,7 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) u32 w = p->width, h = p->height; u32 w16 = DIV_ROUND_UP(w, 16), w32 = DIV_ROUND_UP(w, 32); u32 h32 = DIV_ROUND_UP(h, 32), w64 = DIV_ROUND_UP(w, 64), h64 = DIV_ROUND_UP(h, 64); + const unsigned long nomap = DMA_ATTR_NO_KERNEL_MAPPING; size_t ref_a, ref_b, ref_c, ref_d, tile, mapped = 0; struct ave_buf_req *r; unsigned int n = 0; @@ -251,38 +249,39 @@ static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) ref_d = ALIGN(hevc ? 64 * w32 * h32 : 128 * p->mbs, SZ_16K); tile = hevc ? 2304 * w32 * h64 : ALIGN(w, 16) * ALIGN(h, 64); - ave_req(r, &n, &sess->hdr, AVE_HDR_BYTES, 0); + ave_req(r, &n, &sess->hdr, AVE_HDR_BYTES, 0, nomap); for (unsigned int i = 0; i < p->refs; i++) { - ave_req(r, &n, &sess->ref_a[i], ref_a, ref_a * 3 / 2); - ave_req(r, &n, &sess->ref_b[i], ref_b, ref_b * 5 / 4); - ave_req(r, &n, &sess->ref_c[i], ref_c, 0); - ave_req(r, &n, &sess->ref_d[i], ref_d, 0); - } - ave_req(r, &n, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), 0); - ave_req(r, &n, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), 0); - ave_req(r, &n, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), 0); - ave_req(r, &n, &sess->fixed, SZ_16K, 0); + ave_req(r, &n, &sess->ref_a[i], ref_a, ref_a * 3 / 2, nomap); + ave_req(r, &n, &sess->ref_b[i], ref_b, ref_b * 5 / 4, nomap); + ave_req(r, &n, &sess->ref_c[i], ref_c, 0, nomap); + ave_req(r, &n, &sess->ref_d[i], ref_d, 0, nomap); + } + ave_req(r, &n, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->fixed, SZ_16K, 0, nomap); for (unsigned int i = 0; i < (hevc ? 4 : AVE_TILE_BUFS); i++) { - ave_req(r, &n, &sess->tile[i], tile, 0); - ave_req(r, &n, &sess->tile_aux[i], ALIGN(h, 128), 0); + ave_req(r, &n, &sess->tile[i], tile, 0, nomap); + ave_req(r, &n, &sess->tile_aux[i], ALIGN(h, 128), 0, nomap); } if (hevc) for (unsigned int i = 0; i < 2; i++) - ave_req(r, &n, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), 0); + ave_req(r, &n, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), 0, nomap); for (unsigned int s = 0; s < p->slots; s++) { - ave_req(r, &n, &sess->slot[s].coded, p->coded_cap, 0); - ave_req(r, &n, &sess->slot[s].meta, AVE_META_BYTES, 0); + ave_req(r, &n, &sess->slot[s].coded, p->coded_cap, 0, 0); + ave_req(r, &n, &sess->slot[s].meta, AVE_META_BYTES, 0, 0); /* * HEVC: slice headers. H.264: the per-frame buffer of Process * +0x58c8, which the firmware never wrote in any bit-rate * mode on hardware; one page. */ - ave_req(r, &n, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : AVE_H264_FRAME_BUF, 0); + ave_req(r, &n, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : AVE_H264_FRAME_BUF, 0, + hevc ? 0 : nomap); } sort(r, n, sizeof(*r), ave_buf_req_cmp, NULL); for (unsigned int i = 0; i < n && !ret; i++) { - ret = ave_alloc_padded(ave, r[i].b, r[i].size, r[i].alloc, 0); + ret = ave_alloc_padded(ave, r[i].b, r[i].size, r[i].alloc, r[i].attrs); mapped += r[i].b->alloc; } kfree(r); @@ -327,8 +326,8 @@ static void ave_note_copy(struct ave_ctx *ctx, u32 stride, const char *how) if (ctx->sess->copy_noted) return; ctx->sess->copy_noted = true; - dev_info(ctx->ave->dev, "input has a %u byte stride; moving lines to %u %s\n", stride, - ctx->sess->p.stride, how); + dev_dbg(ctx->ave->dev, "input has a %u byte stride; moving lines to %u %s\n", stride, + ctx->sess->p.stride, how); } /* @@ -433,71 +432,6 @@ static void ave_sess_log_map(struct ave_dev *ave, struct ave_session *sess) } } -/* - * How far did the firmware write past the size it was given? The margin - * behind each buffer was zero when allocated. A write reaching the end of - * the mapping may have gone on into the next buffer. - */ -static void ave_check_tail(struct ave_dev *ave, const char *name, int i, - const struct ave_buf *b) -{ - const u8 *p = b->cpu; - size_t end; - - if (!p || b->attrs & DMA_ATTR_NO_KERNEL_MAPPING || b->alloc <= b->size) - return; - for (end = b->alloc; end > b->size && !p[end - 1]; end--) - ; - if (end == b->size) - return; - if (end == b->alloc) - dev_warn(ave->dev, "buf %s%d: firmware wrote to the end of its mapping (+%#zx, size %#zx)\n", - name, i, b->alloc, b->size); - else - dev_info(ave->dev, "buf %s%d: firmware wrote %#zx bytes past its size %#zx\n", - name, i, end - b->size, b->size); -} - -static void ave_sess_check_tails(struct ave_dev *ave, struct ave_session *sess) -{ - ave_check_tail(ave, "hdr", 0, &sess->hdr); - ave_check_tail(ave, "mb", 0, &sess->mb); - ave_check_tail(ave, "row_a", 0, &sess->row_a); - ave_check_tail(ave, "row_b", 0, &sess->row_b); - ave_check_tail(ave, "fixed", 0, &sess->fixed); - for (int i = 0; i < 2; i++) - ave_check_tail(ave, "hevc_extra", i, &sess->hevc_extra[i]); - for (int i = 0; i < AVE_MAX_REFS; i++) { - ave_check_tail(ave, "ref_a", i, &sess->ref_a[i]); - ave_check_tail(ave, "ref_b", i, &sess->ref_b[i]); - ave_check_tail(ave, "ref_c", i, &sess->ref_c[i]); - ave_check_tail(ave, "ref_d", i, &sess->ref_d[i]); - } - for (int i = 0; i < AVE_TILE_BUFS; i++) { - ave_check_tail(ave, "tile", i, &sess->tile[i]); - ave_check_tail(ave, "tile_aux", i, &sess->tile_aux[i]); - } - for (int i = 0; i < AVE_MAX_SLOTS; i++) { - ave_check_tail(ave, "coded", i, &sess->slot[i].coded); - ave_check_tail(ave, "meta", i, &sess->slot[i].meta); - ave_check_tail(ave, "aux", i, &sess->slot[i].aux); - } - - /* - * The size of the H.264 per-frame buffer is not known; it is - * allocated generously. Report how much of it the firmware used. - */ - if (sess->p.codec == AVE_CODEC_H264 && sess->slot[0].aux.cpu) { - const struct ave_buf *b = &sess->slot[0].aux; - const u64 *w = b->cpu; - size_t n = b->size / 8; - - while (n && !w[n - 1]) - n--; - dev_dbg(ave->dev, "per-frame buffer: %#zx of %#zx bytes used\n", n * 8, b->size); - } -} - /* Work out the parameters and allocate the buffers; no firmware access. */ struct ave_session *ave_sess_create(struct ave_ctx *ctx) { @@ -539,13 +473,10 @@ void ave_sess_destroy(struct ave_ctx *ctx) if (!sess) return; - if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) { + if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) dev_warn(ctx->ave->dev, "keeping the buffers of a failed session\n"); - } else { - if (sess->handed) - ave_sess_check_tails(ctx->ave, sess); + else ave_sess_free_bufs(ctx->ave, sess); - } kfree(sess); ctx->sess = NULL; }