diff --git a/Documentation/admin-guide/media/apple-ave.rst b/Documentation/admin-guide/media/apple-ave.rst index d7e63d029de15..2a76d4eac23d1 100644 --- a/Documentation/admin-guide/media/apple-ave.rst +++ b/Documentation/admin-guide/media/apple-ave.rst @@ -1,63 +1,94 @@ .. SPDX-License-Identifier: GPL-2.0-only -Experimental T8140 video encoding -================================ - -The J700 AVE driver exposes a root-only debugfs firmware transport for -hardware encoding experiments. It is not a V4L2 encoder, has no stable -userspace ABI, and does not make an FFmpeg encoder available. The initial -scope is the existing fixed 320x240 NV12 H.264/HEVC protocol experiment. - -Build ``CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL=m`` on a 16 KiB-page Apple kernel -with debugfs and translated DMA. Suspend, hibernation and kexec must be off. -Load with ``modprobe apple-ave enable=1``; probing is disabled otherwise. - -Firmware and boot requirements ------------------------------- - -Use the public Aurora m1n1 J700 AVE handoff. It reserves the firmware already -loaded by iBoot, fills the device's ``memory-region`` references and exports -``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32`` strings under -``/chosen``. No firmware file is included or requested by this driver. -The current transport requires the qualified 25G76 two-segment layout: -0x14c000 text bytes and 0x130000 data bytes, mapped starting at IOVA -0x10000000000. Every mapping and both content CRCs are checked before use. -Text CRCs are restricted to the two inspected 9003.78.0 images (0x2ce7359e -and 0x69c54454). These checks detect build drift, not firmware authenticity. -Different firmware is unsupported, even if the device has the same SoC. - -The driver holds the DMA, pipe4, pipe5, me0 and me1 power domains and owns -the IPC, queue, log and firmware-requested heap allocations. Debugfs lives -under ``/sys/kernel/debug/285100000.video-codec``: - -* ``ave-status`` reports mappings, protocol phase and reply counters. -* ``ave-start-once`` accepts one explicit firmware start per boot. -* ``ave-command`` accepts one bounded packet and returns a framed reply. -* ``ave-arena`` stages image/reference/output surfaces within owned memory. -* ``ave-poll`` drains later firmware replies without another submission. - -Writes require ``CAP_SYS_RAWIO``. Command packets must come from a trusted, -matching local protocol experiment with all embedded surface addresses -relocated into this boot's owned arena. Root access does not make arbitrary -firmware packets valid. Binary command captures are supplied separately and -are not included in this contribution. - -Lifecycle and verification --------------------------- - -After the start attempt, the module, DMA buffers and power references remain -pinned until an external hardware reset, including after failure. Do not -force-unload, unbind the DARTs, reuse uncertain buffers or retry a failed -command. Firmware quiescence and normal production teardown are unfinished. - -A transport acknowledgement is not an encoded frame. For physical tests, -record the exact kernel/module, DT and firmware hashes; submit changing -images; validate each codec reply; independently decode the current DMA -output; check every frame's dimensions and quality; then verify Stop/Close, -kernel health and the existing desktop/radio stack. Keep the generated -inputs, bitstreams, decoded pixels, logs and hashes as a repeatable artifact. - -The prior implementation was authored by Ace (Acelogic). This contribution -ports that existing GPL work. Arbitrary resolutions, standard media APIs, -throughput, concurrent clients and other codecs or SoCs are outside this -experiment. +Apple AVE video encoder +======================= + +The AVE block in Apple T8140 (A18 Pro) SoCs, as used in the MacBook Neo, +encodes H.264 and HEVC. ``CONFIG_VIDEO_APPLE_AVE`` builds the ``apple-ave`` +driver, a V4L2 memory-to-memory stateful encoder following +Documentation/userspace-api/media/v4l/dev-encoder.rst. The OUTPUT queue +takes NV12 frames, the CAPTURE queue returns H.264 (Baseline, Main, High) or +HEVC (Main) as Annex B. + +Usage +----- + +The encoder shows up as a ``/dev/videoN`` node named "Apple AVE video +encoder". GStreamer's ``v4l2h264enc`` and ``v4l2h265enc`` and ffmpeg's +``h264_v4l2m2m`` and ``hevc_v4l2m2m`` work with it, for example:: + + gst-launch-1.0 -e videotestsrc num-buffers=300 ! \ + video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \ + v4l2h264enc extra-controls="controls,video_bitrate=8000000" ! \ + h264parse ! filesink location=out.h264 + + ffmpeg -i in.mkv -pix_fmt nv12 -c:v h264_v4l2m2m -b:v 8M -g 30 out.h264 + +Up to two streams can be encoded at the same time. + +Firmware +-------- + +The encoder runs firmware on its own coprocessor. The bootloader loads it into +two reserved, DART-mapped segments (TEXT and DATA) but does not start it. The +Aurora Stage 2 loader passes both regions in the device's ``memory-region`` +property and records their CRC32 under ``/chosen`` as +``aurora,ave-text-crc32`` and ``aurora,ave-data-crc32``. The driver only binds +while DATA still matches the recorded checksum, and keeps a copy of DATA that +it restores before every firmware boot. + +No firmware files are needed. The driver builds every command itself, +including the two large session commands (Start and Process), from the +session settings and a set of constant fields known for one firmware build: +AppleAVE2FW 9003.78.0, TEXT checksum ``69c54454``. With any other firmware +build the device binds, but starting a stream fails with ``ENOTSUPP`` and a +message naming the checksum, until that build's command layout has been +checked. + +With debugfs mounted, ``apple-ave/last_start_h264``, +``last_start_hevc``, ``last_process0_h264`` and ``last_process0_hevc`` hold the +last Start command built and the first Process command of the last session, +for comparison with reference captures. + +Sessions and power management +----------------------------- + +The firmware boots when the first stream starts and is halted when the last +one ends, as macOS does; the next stream starts it from reset. Runtime PM +powers the encoder down two seconds after the last stream. + +Up to five frames per stream are inside the firmware at once (three without +B-frames). The firmware reorders B-frames itself, so CAPTURE buffers come +back in bitstream order, each with the timestamp of the frame it codes. + +Across system sleep, open streams are drained, closed and the firmware is +halted. On resume each stream reopens on its next frame, which starts with +an IDR and fresh parameter sets; clients see no error. + +Every wait for the firmware is bounded. A command or frame that gets no +answer within 5 seconds (Start: 10 seconds) fails the stream with ``EIO``; +a firmware that stops answering keeps the encoder powered and is not used +again until reboot. The driver polls the firmware rings with sleeping waits. + +Limits +------ + +- Picture sizes from 320x240 to 2560x1600 in steps of two, up to 240 frames + per second. +- The encoder reads NV12 with a 64-byte aligned line stride, which + ``VIDIOC_S_FMT`` reports. Clients that fill buffers with a smaller line + size (ffmpeg does, for widths that are not a multiple of 64) are detected + from the payload size and their frames copied to the encoder's layout. +- Rate control is VBR, with ``BITRATE_PEAK`` as a one-second data-rate cap + (twice the average when left at 0), or CBR. In CBR mode, frames are padded + with filler data once more than half a second of bit rate has gone unused, + as macOS does; the driver adds the filler itself if the firmware does not. + Constant quality is not offered. ``BITRATE`` may change while streaming and applies from the next + frame; ``FORCE_KEY_FRAME`` makes the next frame an IDR. +- Either three B-frames or none (``B_FRAMES`` 3 or 0). H.264 Baseline and a + GOP size of 1 never use them. +- The level control is a minimum: the driver raises it to what the size, + frame rate and bit rate need. +- Buffers are mapped through the encoder's 256 MiB DMA window, so both queues + are limited to what fits next to the encoder's working memory (fewer + buffers at larger sizes). diff --git a/MAINTAINERS b/MAINTAINERS index 8b2cc48645bf4..7e126d34bed5c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2802,6 +2802,14 @@ F: net/bluetooth/brcm.c F: net/bluetooth/brcm.h F: tools/testing/selftests/drivers/apple/ +ARM/APPLE AVE VIDEO ENCODER DRIVER +M: Ryan Murray +L: linux-media@vger.kernel.org +S: Maintained +W: https://aurora-silicon.github.io +F: Documentation/admin-guide/media/apple-ave.rst +F: drivers/media/platform/apple/ave/ + ARM/APPLE QSPIMC DRIVER M: Ryan Murray W: https://aurora-silicon.github.io diff --git a/arch/arm64/boot/dts/apple/t8140-j700.dts b/arch/arm64/boot/dts/apple/t8140-j700.dts index ece0d9c6bfefb..61427cfe7538f 100644 --- a/arch/arm64/boot/dts/apple/t8140-j700.dts +++ b/arch/arm64/boot/dts/apple/t8140-j700.dts @@ -388,6 +388,18 @@ #include "isp-imx558.dtsi" +&ave_dart0 { + status = "okay"; +}; + +&ave_dart1 { + status = "okay"; +}; + +&ave { + status = "okay"; +}; + &serial0 { status = "okay"; }; diff --git a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi index 8a1864e0fabbd..0a479f7e8e8e5 100644 --- a/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi +++ b/arch/arm64/boot/dts/apple/t8140-pmgr.dtsi @@ -69,7 +69,6 @@ #power-domain-cells = <0>; #reset-cells = <0>; label = "venc0_sys"; - status = "disabled"; }; ps_apcie_gp: power-controller@258 { @@ -408,7 +407,6 @@ #reset-cells = <0>; label = "venc0_dma"; power-domains = <&ps_venc0_sys>; - status = "disabled"; }; ps_venc0_pipe4: power-controller@8008 { @@ -418,7 +416,6 @@ #reset-cells = <0>; label = "venc0_pipe4"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_pipe5: power-controller@8010 { @@ -428,7 +425,6 @@ #reset-cells = <0>; label = "venc0_pipe5"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_me0: power-controller@8018 { @@ -438,7 +434,6 @@ #reset-cells = <0>; label = "venc0_me0"; power-domains = <&ps_venc0_dma>; - status = "disabled"; }; ps_venc0_me1: power-controller@8020 { @@ -448,7 +443,6 @@ #reset-cells = <0>; label = "venc0_me1"; power-domains = <&ps_venc0_me0>; - status = "disabled"; }; }; diff --git a/arch/arm64/configs/asahi.config b/arch/arm64/configs/asahi.config index fcf0d27611aae..8d9f76581d226 100644 --- a/arch/arm64/configs/asahi.config +++ b/arch/arm64/configs/asahi.config @@ -34,6 +34,7 @@ CONFIG_APPLE_WATCHDOG=y CONFIG_MTD=y CONFIG_MTD_PHRAM=m CONFIG_VIDEO_APPLE_AVD=m +CONFIG_VIDEO_APPLE_AVE=m CONFIG_VIDEO_APPLE_ISP=m CONFIG_DRM=y CONFIG_DRM_ASAHI=m diff --git a/drivers/media/platform/apple/ave/Kconfig b/drivers/media/platform/apple/ave/Kconfig index 629ce9244039a..baf6218ecdd5b 100644 --- a/drivers/media/platform/apple/ave/Kconfig +++ b/drivers/media/platform/apple/ave/Kconfig @@ -1,16 +1,18 @@ # SPDX-License-Identifier: GPL-2.0-only -config VIDEO_APPLE_AVE_EXPERIMENTAL - tristate "Apple T8140 experimental video encoder transport" - depends on ARCH_APPLE && ARM64_16K_PAGES - depends on PM_GENERIC_DOMAINS && IOMMU_DMA && DEBUG_FS - depends on !SUSPEND && !HIBERNATION && !KEXEC_CORE +config VIDEO_APPLE_AVE + tristate "Apple AVE video encoder" + depends on ARCH_APPLE || COMPILE_TEST + depends on VIDEO_DEV && IOMMU_DMA && PM_GENERIC_DOMAINS select CRC32 + select V4L2_MEM2MEM_DEV + select VIDEOBUF2_DMA_CONTIG help - Experimental root-only AVE firmware and command transport for the - MacBook Neo. This does not register a V4L2 or FFmpeg encoder. + V4L2 memory-to-memory driver for the AVE H.264/HEVC encoder in + Apple T8140 (A18 Pro) SoCs, as in the MacBook Neo. - It requires the bootloader's reserved, preloaded firmware and an - explicit enable=1 module parameter. After firmware startup, memory - and power references remain pinned until a hardware reset. Do not - enable this on a general-purpose kernel requiring suspend or kexec. + The encoder firmware is loaded by the bootloader; the driver builds + every command itself. See Documentation/admin-guide/media/apple-ave.rst. + + To compile this driver as a module, choose M here: the module + will be called apple-ave. diff --git a/drivers/media/platform/apple/ave/Makefile b/drivers/media/platform/apple/ave/Makefile index c8e673e68f080..53bb6b554f770 100644 --- a/drivers/media/platform/apple/ave/Makefile +++ b/drivers/media/platform/apple/ave/Makefile @@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_VIDEO_APPLE_AVE_EXPERIMENTAL) += apple-ave.o +obj-$(CONFIG_VIDEO_APPLE_AVE) += apple-ave.o +apple-ave-y := ave-drv.o ave-fw.o ave-ipc.o ave-cmd.o ave-image.o ave-sess.o ave-v4l2.o diff --git a/drivers/media/platform/apple/ave/apple-ave-ipc.h b/drivers/media/platform/apple/ave/apple-ave-ipc.h deleted file mode 100644 index d317af1dad72b..0000000000000 --- a/drivers/media/platform/apple/ave/apple-ave-ipc.h +++ /dev/null @@ -1,446 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* Experimental 25G76/64-bit AVE mailbox and bidirectional ring transport. - * Included after private driver types. All callers hold ave->lock. - */ -#define AVE_CHANNEL_OFF 0x4000 -#define AVE_HANDSHAKE_OFF 0x24000 -#define AVE_SHARED_OFF 0x28000 -#define AVE_COMMAND_OFF 0x40000 -#define AVE_COMMAND_BYTES SZ_128K -#define AVE_ARENA_OFF 0x80000 -#define AVE_START_MAGIC 0x08042006 - -static int ave_fw_offset(struct apple_ave *ave, u64 address, size_t bytes, u32 *offset) -{ - u64 delta = address - ave->ipc_fw_base; - - if (address < ave->ipc_fw_base || delta > ave->ipc.size || bytes > ave->ipc.size - delta) - return -ERANGE; - *offset = delta; - return 0; -} - -static void ave_mailbox_send(struct apple_ave *ave, u32 a, u32 b, u32 c, u32 d) -{ - writel(a, ave->mailbox + 0x18); - writel(b, ave->mailbox + 0x1c); - writel(c, ave->mailbox + 0x20); - writel(d, ave->mailbox + 0x24); - wmb(); - writel(1, ave->mailbox + 0x0c); -} - -static int ave_mailbox_receive(struct apple_ave *ave) -{ - u32 status; - int ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); - - if (ret) - return ret; - /* Acknowledge the interrupt before reading the four scratch words. */ - writel(1, ave->mailbox + 0x10); - for (unsigned int i = 0; i < 4; i++) - ave->ipc_reply[i] = readl(ave->mailbox + 0x18 + i * 4); - dma_rmb(); - dev_info(ave->dev, "AVE_IPC_MAILBOX phase=%s words=%#x,%#x,%#x,%#x\n", - ave->phase, ave->ipc_reply[0], ave->ipc_reply[1], ave->ipc_reply[2], ave->ipc_reply[3]); - return 0; -} - -static int ave_create_channels(struct apple_ave *ave) -{ - bool found[2] = { false, false }; - - for (unsigned int i = 0; i < 2; i++) { - u8 *desc = ave->ipc.cpu + AVE_CHANNEL_OFF + i * 0x100; - u32 direction = get_unaligned_le32(desc + 0x40); - u32 interrupt = get_unaligned_le32(desc + 0x44); - u32 slots = get_unaligned_le32(desc + 0x48), offset; - u64 address = get_unaligned_le64(desc + 0x4c); - struct ave_channel *c; - unsigned int id; - - if (!memchr(desc, 0, 64)) - return -EPROTO; - if (!strcmp(desc, "IO")) - id = 0; - else if (!strcmp(desc, "IO_T2H")) - id = 1; - else - return -EPROTO; - if (found[id] || direction != id || !interrupt || interrupt >= 32 || - !slots || slots > ave->channel_bytes / 64 || !IS_ALIGNED(address, 64) || - ave_fw_offset(ave, address, (size_t)slots * 64, &offset) || - offset < AVE_CHANNEL_OFF + 0x200 || - offset + slots * 64 > AVE_CHANNEL_OFF + ave->channel_bytes) - return -EPROTO; - found[id] = true; - c = &ave->channel[id]; - strscpy(c->name, desc, sizeof(c->name)); - c->mode = direction ^ 1; - c->interrupt = interrupt; - c->slots = slots; - c->offset = offset; - c->consumer = c->mode ? -1 : 0; - c->producer = c->mode ? 0 : -1; - } - if (!found[0] || !found[1] || - (ave->channel[0].offset < ave->channel[1].offset + ave->channel[1].slots * 64 && - ave->channel[1].offset < ave->channel[0].offset + ave->channel[0].slots * 64)) - return -EPROTO; - /* Initialize only host-originated slots; leave all padding and - * firmware-originated slots unchanged. Ownership is bit zero of address. - */ - for (unsigned int i = 0; i < ave->channel[0].slots; i++) { - __le64 *slot = ave->ipc.cpu + ave->channel[0].offset + i * 64; - slot[0] = cpu_to_le64(1); - slot[1] = 0; - slot[2] = 0; - } - dma_wmb(); - return 0; -} - -static int ave_negotiate_ipc(struct apple_ave *ave) -{ - u64 ticks, time_delta, handshake_address, channel_address; - u32 frequency, status, offset; - u8 *handshake = ave->ipc.cpu + AVE_HANDSHAKE_OFF; - int ret; - - ave->phase = "validate-offer"; - if (ave->hello[0] != 2 || ave->hello[2] != 0x100 || - ave->hello[1] < 0x200 || ave->hello[1] > AVE_HANDSHAKE_OFF - AVE_CHANNEL_OFF || - !ave->hello[3] || ave->hello[3] > 4 * SZ_1M) - return -EPROTO; - ave->channel_bytes = ave->hello[1]; - ret = ave_alloc_buffer(ave, &ave->heap, PAGE_ALIGN(ave->hello[3])); - if (ret) - return ret; - writel(1, ave->mailbox + 0x10); - ticks = readq(ave->coproc + 0x378000); - frequency = readl(ave->coproc + 0x360020); - if (frequency && frequency < 1000000) - return -EPROTO; - ticks = div64_u64(ticks, frequency ? frequency / 1000000 : 24); - time_delta = div_u64(ktime_get_ns(), 1000) - ticks; - writel(lower_32_bits(time_delta), ave->mailbox + 0x28); - writel(upper_32_bits(time_delta), ave->mailbox + 0x2c); - ave->phase = "ipc-base"; - ave_mailbox_send(ave, lower_32_bits(ave->ipc.dma), upper_32_bits(ave->ipc.dma), ave->ipc.size, 0); - ret = ave_mailbox_receive(ave); - if (ret) - return ret; - ave->ipc_fw_base = (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0]; - if (!ave->ipc_fw_base || !IS_ALIGNED(ave->ipc_fw_base, PAGE_SIZE) || - ave->ipc_fw_base > U64_MAX - ave->ipc.size) - return -EPROTO; - memset(ave->ipc.cpu + AVE_CHANNEL_OFF, 0, ave->channel_bytes); - memset(handshake, 0, 0x50); - memset(ave->ipc.cpu + AVE_SHARED_OFF, 0, SZ_64K); - channel_address = ave->ipc_fw_base + AVE_CHANNEL_OFF; - handshake_address = ave->ipc_fw_base + AVE_HANDSHAKE_OFF; - put_unaligned_le64(channel_address, handshake + 8); - put_unaligned_le64(ave->ipc_fw_base + AVE_SHARED_OFF, handshake + 0x10); - put_unaligned_le32(SZ_64K, handshake + 0x18); - put_unaligned_le64(ave->heap.dma, handshake + 0x1c); - put_unaligned_le32(ave->hello[3], handshake + 0x24); - put_unaligned_le32(27, handshake + 0x28); - dma_wmb(); - ave->phase = "channel-offer"; - ave_mailbox_send(ave, lower_32_bits(handshake_address), upper_32_bits(handshake_address), 0, 0); - ret = ave_mailbox_receive(ave); - if (ret) - return ret; - ave->extra_bytes = ave->ipc_reply[2]; - if (ave_fw_offset(ave, (u64)ave->ipc_reply[1] << 32 | ave->ipc_reply[0], - ave->channel_bytes, &offset) || offset != AVE_CHANNEL_OFF || - ave->extra_bytes > 0x13c000) - return -EPROTO; - ave->phase = "channel-layout"; - ret = ave_create_channels(ave); - if (ret) - return ret; - ave->phase = "ready-flag"; - writel(AVE_START_MAGIC, ave->mailbox + 0x24); - ret = readl_poll_timeout(ave->mailbox + 0x24, status, !status, 50, 5000000); - if (ret) - return ret; - ave->ipc_ready = true; - ave->phase = "ready"; - return 0; -} - -static int ave_ring_send(struct apple_ave *ave, struct ave_channel *c, u64 addr, u32 len, u32 flags) -{ - __le64 *slot; - int next; - - if (c->producer < 0) - return -ENOSPC; - slot = ave->ipc.cpu + c->offset + c->producer * 64; - WRITE_ONCE(slot[1], cpu_to_le64(len)); - WRITE_ONCE(slot[2], cpu_to_le64(flags)); - dma_wmb(); - WRITE_ONCE(slot[0], cpu_to_le64(addr | (c->mode ^ 1))); - dma_wmb(); - if (c->consumer < 0) - c->consumer = c->producer; - next = (c->producer + 1) % c->slots; - c->producer = next == c->consumer ? -1 : next; - writel(BIT(c->interrupt), ave->mailbox + 0x0c); - return 0; -} - -static int ave_ring_receive(struct apple_ave *ave, struct ave_channel *c, u64 *addr, u32 *len, u32 *flags) -{ - __le64 *slot; - u64 word; - int next; - - if (c->consumer < 0) - return -EAGAIN; - slot = ave->ipc.cpu + c->offset + c->consumer * 64; - word = le64_to_cpu(READ_ONCE(slot[0])); - if ((word & 1) != c->mode) - return -EAGAIN; - dma_rmb(); - *addr = word & ~3ULL; - *len = le64_to_cpu(READ_ONCE(slot[1])); - *flags = le64_to_cpu(READ_ONCE(slot[2])); - if (c->producer < 0) - c->producer = c->consumer; - next = (c->consumer + 1) % c->slots; - c->consumer = next == c->producer ? -1 : next; - return 0; -} - -/* Root-only experimental packet transport. The packet is copied into owned - * coherent memory; timeout pins the command too and forbids reuse this boot. - * Firmware command success is established by decoding replies, not this API. - */ -static ssize_t ave_command_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - u8 *cmd = ave->ipc.cpu + AVE_COMMAND_OFF; - u64 deadline, addr; - u32 bytes, flags, offset; - bool ack = false; - int ret = 0; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - static_assert(AVE_COMMAND_OFF + AVE_COMMAND_BYTES <= AVE_ARENA_OFF); - if (len < 0x48 || len > AVE_COMMAND_BYTES) - return -EINVAL; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) { - ret = -EBUSY; - goto out; - } - if (copy_from_user(cmd, data, len)) { - ret = -EFAULT; - goto out; - } - /* The common command timeout is {duration_us, creation_time_us}. - * Stamp it in the same Linux monotonic clock synchronized at startup; - * captured macOS absolute times cannot be replayed on this boot. - */ - put_unaligned_le64(3000000, cmd + 0x30); - put_unaligned_le64(div_u64(ktime_get_ns(), 1000), cmd + 0x38); - ave->response_bytes = 0; - ret = ave_ring_send(ave, &ave->channel[0], ave->ipc_fw_base + AVE_COMMAND_OFF, len, 0); - if (ret) - goto out; - ave->commands++; - deadline = ktime_get_ns() + 5ULL * NSEC_PER_SEC; - do { - u32 pending = readl(ave->mailbox + 0x10); - if (pending) - writel(pending, ave->mailbox + 0x10); - for (unsigned int ch = 0; ch < 2; ch++) { - ret = ave_ring_receive(ave, &ave->channel[ch], &addr, &bytes, &flags); - if (ret == -EAGAIN) - continue; - /* Channel 1 returns the original address with zero payload - * bytes (verified native AVC captures). Only channel 2 is - * copied to response, independent of submitted packet size. - */ - if (ret || (!ch && bytes != 0) || - (ch && bytes > sizeof(ave->response) - 24) || - ave_fw_offset(ave, addr, bytes, &offset)) { - ret = -EPROTO; - goto failed; - } - dev_info(ave->dev, "AVE_COMMAND_REPLY channel=%u address=%#llx bytes=%u flags=%#x\n", - ch + 1, addr, bytes, flags); - if (!ch) { - if (addr != ave->ipc_fw_base + AVE_COMMAND_OFF) { - ret = -EPROTO; - goto failed; - } - ack = true; - ave->transport_acks++; - } else { - put_unaligned_le32(ch + 1, ave->response); - put_unaligned_le32(bytes, ave->response + 4); - put_unaligned_le32(flags, ave->response + 8); - put_unaligned_le32(0, ave->response + 12); - put_unaligned_le64(addr, ave->response + 16); - memcpy(ave->response + 24, ave->ipc.cpu + offset, bytes); - ave->response_bytes = bytes + 24; - ave->replies++; - /* Reply echoes use zero flags on the observed transport. */ - ret = ave_ring_send(ave, &ave->channel[ch], addr, bytes, 0); - if (ret) - goto failed; - } - } - if (ack && ave->response_bytes) { - ret = 0; - goto out; - } - usleep_range(100, 200); - } while (ktime_get_ns() < deadline); - ret = -ETIMEDOUT; -failed: - ave->command_failed = true; -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static ssize_t ave_command_read(struct file *file, char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - mutex_lock(&ave->lock); - ret = simple_read_from_buffer(data, len, pos, ave->response, ave->response_bytes); - mutex_unlock(&ave->lock); - return ret; -} - -static const struct file_operations ave_command_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_command_write, - .read = ave_command_read, -}; - -/* Drain later firmware events without publishing another command. Results - * replace the response buffer and consist of one or more 24-byte-header - * records, each followed by its declared payload. Never discard silently. - */ -static ssize_t ave_poll_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - bool requested; - u64 deadline, addr; - u32 bytes, flags, offset; - int ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - ret = kstrtobool_from_user(data, len, &requested); - if (ret || !requested) - return ret ?: -EINVAL; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) { - ret = -EBUSY; - goto out; - } - ave->response_bytes = 0; - deadline = ktime_get_ns() + NSEC_PER_SEC; - do { - u32 pending = readl(ave->mailbox + 0x10); - u8 *dst; - - if (pending) - writel(pending, ave->mailbox + 0x10); - ret = ave_ring_receive(ave, &ave->channel[1], &addr, &bytes, &flags); - if (ret == -EAGAIN) { - usleep_range(100, 200); - continue; - } - if (ret || ave_fw_offset(ave, addr, bytes, &offset) || - bytes > sizeof(ave->response) - 24 || - ave->response_bytes > sizeof(ave->response) - 24 - bytes) { - ret = -EPROTO; - goto failed; - } - dst = ave->response + ave->response_bytes; - put_unaligned_le32(2, dst); - put_unaligned_le32(bytes, dst + 4); - put_unaligned_le32(flags, dst + 8); - put_unaligned_le32(0, dst + 12); - put_unaligned_le64(addr, dst + 16); - memcpy(dst + 24, ave->ipc.cpu + offset, bytes); - ave->response_bytes += 24 + bytes; - ave->replies++; - ret = ave_ring_send(ave, &ave->channel[1], addr, bytes, 0); - if (ret) - goto failed; - } while (ktime_get_ns() < deadline); - ret = 0; - goto out; -failed: - ave->command_failed = true; -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static const struct file_operations ave_poll_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_poll_write, -}; - -/* Bounded staging area for image/reference/output surfaces. Userspace offsets - * start at zero here, beyond all handshake, ring and command allocations. - * The status file supplies both DART and firmware addresses for this arena. - */ -static ssize_t ave_arena_read(struct file *file, char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - mutex_lock(&ave->lock); - dma_rmb(); - ret = simple_read_from_buffer(data, len, pos, ave->ipc.cpu + AVE_ARENA_OFF, - ave->ipc.size - AVE_ARENA_OFF); - mutex_unlock(&ave->lock); - return ret; -} - -static ssize_t ave_arena_write(struct file *file, const char __user *data, size_t len, loff_t *pos) -{ - struct apple_ave *ave = file->private_data; - ssize_t ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - mutex_lock(&ave->lock); - if (!ave->ipc_ready || ave->command_failed) - ret = -EBUSY; - else - ret = simple_write_to_buffer(ave->ipc.cpu + AVE_ARENA_OFF, - ave->ipc.size - AVE_ARENA_OFF, pos, data, len); - dma_wmb(); - mutex_unlock(&ave->lock); - return ret; -} - -static loff_t ave_arena_seek(struct file *file, loff_t offset, int whence) -{ - return fixed_size_llseek(file, offset, whence, AVE_IPC_BYTES - AVE_ARENA_OFF); -} - -static const struct file_operations ave_arena_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .read = ave_arena_read, - .write = ave_arena_write, - .llseek = ave_arena_seek, -}; diff --git a/drivers/media/platform/apple/ave/apple-ave.c b/drivers/media/platform/apple/ave/apple-ave.c deleted file mode 100644 index d2cf7abadcadf..0000000000000 --- a/drivers/media/platform/apple/ave/apple-ave.c +++ /dev/null @@ -1,532 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* T8140 AVE bring-up: resources, firmware negotiation and IPC transport. - * Experimental root-only command interface. No registered encoder yet. - */ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#define AVE_IPC_BYTES (20 * SZ_1M) -#define AVE_QUEUE_BYTES SZ_64K -#define AVE_FW_IOVA 0x10000000000ULL -#define AVE_LOG_BYTES 0x24000 -#define AVE_LOG_USED 0x20200 - -static bool enable; -module_param(enable, bool, 0400); -MODULE_PARM_DESC(enable, "Opt in to the experimental AVE transport"); - -/* Read the known Uranus registers once after PM resume. The separate hello - * command pins all resources because reset/quiescence is not yet qualified. - */ -static bool snapshot; -module_param(snapshot, bool, 0400); -MODULE_PARM_DESC(snapshot, "Capture powered AVE registers once without writing them"); - -struct ave_firmware { - phys_addr_t physical; - u64 iova; - size_t size; - u32 crc; - unsigned int verified_pages; -}; - -struct ave_buffer { - void *cpu; - dma_addr_t dma; - size_t size; - unsigned int verified_pages; -}; - -struct ave_channel { - char name[64]; - u32 mode, interrupt, slots; - u32 offset; - int consumer, producer; -}; - -struct apple_ave { - struct device *dev; - struct device *power[5]; - unsigned int num_power; - struct iommu_domain *domain; - struct ave_buffer ipc; - struct ave_buffer queue; - struct ave_buffer log; - struct ave_buffer heap; - struct ave_firmware fw[2]; - bool firmware_verified; - bool start_requested; - int start_result; - struct mutex lock; - void __iomem *coproc, *mailbox; - u32 hello[4]; - u64 ipc_fw_base; - u32 ipc_reply[4], channel_bytes, extra_bytes; - const char *phase; - bool ipc_ready, command_failed; - struct ave_channel channel[2]; - u32 commands, replies, transport_acks; - u8 response[SZ_64K]; - size_t response_bytes; - struct debugfs_blob_wrapper ipc_blob; - struct debugfs_blob_wrapper log_blob; - struct debugfs_blob_wrapper config_blob; - u64 vcbar; - u32 core_control, core_status, mailbox_status, scratch[8]; - struct dentry *debugfs; -}; - -static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size); -#include "apple-ave-ipc.h" - -/* Cold-boot hello only. A successful response still needs the remaining IPC - * negotiation. Keep the module, PM reference and DMA buffers pinned until a - * target reboot, on both success and timeout; no unproven stop/free sequence. - */ -static ssize_t ave_start_write(struct file *file, const char __user *data, - size_t len, loff_t *ppos) -{ - struct apple_ave *ave = file->private_data; - __le32 *cfg = ave->queue.cpu, *log = ave->log.cpu; - bool requested; - u32 status; - int ret; - - if (!capable(CAP_SYS_RAWIO)) - return -EPERM; - ret = kstrtobool_from_user(data, len, &requested); - if (ret || !requested) - return ret ?: -EINVAL; - mutex_lock(&ave->lock); - if (ave->start_requested) { - ret = -EALREADY; - goto out; - } - if (!ave->firmware_verified || readq(ave->coproc + 0x50000) != (AVE_FW_IOVA | 1) || - readl(ave->coproc + 0x600044) || readl(ave->mailbox + 0x10)) { - ret = -EBUSY; - goto out; - } - for (unsigned int i = 0; i < 8; i++) { - if (readl(ave->mailbox + 0x18 + i * 4)) { - ret = -EBUSY; - goto out; - } - } - /* Startup values observed with the qualified 25G76 firmware: device - * enum 29, one active unit/group and sub-ID mask 1. Firmware maps the - * full page-aligned log allocation; the ring header advertises usage. - */ - memset(cfg, 0, 0x40); - cfg[1] = cpu_to_le32(29); - cfg[2] = cpu_to_le32(1); - cfg[3] = cpu_to_le32(1); - cfg[4] = cpu_to_le32(1); - cfg[8] = cpu_to_le32(lower_32_bits(ave->log.dma)); - cfg[9] = cpu_to_le32(upper_32_bits(ave->log.dma)); - cfg[10] = cpu_to_le32(AVE_LOG_BYTES); - cfg[12] = cpu_to_le32(0x77); - /* Log header: 128 KiB ring after its 512-byte header. */ - log[0] = cpu_to_le32(AVE_LOG_USED); - log[1] = cpu_to_le32(0x200); - log[2] = cpu_to_le32(0x20000); - dma_wmb(); - __module_get(THIS_MODULE); - ave->start_requested = true; - ave->phase = "hello"; - ave->start_result = -EINPROGRESS; - dev_info(ave->dev, "AVE_START_ONCE cfg=%pad log=%pad resources_pinned_until_reboot=1\n", - &ave->queue.dma, &ave->log.dma); - writel(0x08042006, ave->mailbox + 0x18); - writel(lower_32_bits(ave->queue.dma), ave->mailbox + 0x1c); - writel(upper_32_bits(ave->queue.dma), ave->mailbox + 0x20); - /* Preserve the protected firmware mappings inherited from iBoot. */ - writel(1, ave->coproc + 0x600808); - writel(0, ave->coproc + 0x600044); - writel(0x10000, ave->coproc + 0x600400); - writel(0x10, ave->coproc + 0x600044); - ret = readl_poll_timeout(ave->mailbox + 0x10, status, status & 1, 50, 5000000); - ave->start_result = ret; - ave->core_control = readl(ave->coproc + 0x600044); - ave->core_status = readl(ave->coproc + 0x600048); - ave->mailbox_status = status; - for (unsigned int i = 0; i < 8; i++) - ave->scratch[i] = readl(ave->mailbox + 0x18 + i * 4); - for (unsigned int i = 0; i < 4; i++) - ave->hello[i] = ave->scratch[i]; - dma_rmb(); - dev_info(ave->dev, "AVE_HELLO_RESULT result=%d status=%#x words=%#x,%#x,%#x,%#x core=%#x\n", - ret, status, ave->hello[0], ave->hello[1], ave->hello[2], ave->hello[3], ave->core_status); - if (!ret) - ret = ave_negotiate_ipc(ave); - ave->start_result = ret; - dev_info(ave->dev, "AVE_IPC_RESULT result=%d phase=%s ready=%u\n", - ret, ave->phase, ave->ipc_ready); -out: - mutex_unlock(&ave->lock); - return ret ?: len; -} - -static const struct file_operations ave_start_fops = { - .owner = THIS_MODULE, - .open = simple_open, - .write = ave_start_write, -}; - -static int ave_verify_firmware(struct apple_ave *ave) -{ - static const char * const crc_names[] = { - "aurora,ave-text-crc32", "aurora,ave-data-crc32", - }; - static const size_t sizes[] = { 0x14c000, 0x130000 }; - struct device_node *chosen, *node; - unsigned int i; - int ret = 0; - - if (of_count_phandle_with_args(ave->dev->of_node, "memory-region", NULL) != 2) - return dev_err_probe(ave->dev, -EINVAL, "Expected two live AVE firmware reservations\n"); - chosen = of_find_node_by_path("/chosen"); - if (!chosen) - return -ENOENT; - for (i = 0; i < ARRAY_SIZE(ave->fw); i++) { - struct ave_firmware *fw = &ave->fw[i]; - struct resource res; - const char *crc_string; - u32 maps[5], expected_crc; - void *cpu; - size_t offset; - - node = of_parse_phandle(ave->dev->of_node, "memory-region", i); - ret = of_address_to_resource(node, 0, &res); - if (!ret) - ret = of_property_read_u32_array(node, "iommu-addresses", maps, ARRAY_SIZE(maps)); - if (!ret && (!of_property_read_bool(node, "no-map") || - !of_device_is_compatible(node, "apple,asc-mem"))) - ret = -EINVAL; - of_node_put(node); - if (ret) - break; - fw->physical = res.start; - fw->size = resource_size(&res); - fw->iova = (u64)maps[1] << 32 | maps[2]; - if (fw->size != sizes[i] || !IS_ALIGNED(fw->physical, PAGE_SIZE) || - maps[0] != ave->dev->of_node->phandle || maps[3] || maps[4] != sizes[i] || - fw->iova != AVE_FW_IOVA + (i ? sizes[0] : 0)) { - ret = -EINVAL; - break; - } - /* The IOMMU core owns these translated reservations across module - * unloads. DMA allocation must never reuse their IOVAs. - */ - for (offset = 0; offset < fw->size; offset += PAGE_SIZE) { - if (iommu_iova_to_phys(ave->domain, fw->iova + offset) != fw->physical + offset) { - dev_err(ave->dev, "Firmware %u mapping mismatch at %#zx\n", i, offset); - ret = -EIO; - break; - } - fw->verified_pages++; - } - if (ret) - break; - ret = of_property_read_string(chosen, crc_names[i], &crc_string); - if (!ret) - ret = kstrtou32(crc_string, 16, &expected_crc); - if (ret) - break; - cpu = memremap(fw->physical, fw->size, MEMREMAP_WB); - if (!cpu) { - ret = -ENOMEM; - break; - } - fw->crc = crc32_le(~0U, cpu, fw->size) ^ ~0U; - memunmap(cpu); - if (fw->crc != expected_crc) { - dev_err(ave->dev, "Firmware %u changed since the boot preflight\n", i); - ret = -EILSEQ; - break; - } - /* Only the two inspected 9003.78.0 text images are supported. - * CRCs detect build drift; they are not firmware authentication. - */ - if (!i && fw->crc != 0x2ce7359e && fw->crc != 0x69c54454) { - dev_err(ave->dev, "Unsupported AVE firmware text CRC %#x\n", fw->crc); - ret = -ENODEV; - break; - } - } - of_node_put(chosen); - if (!ret) - ave->firmware_verified = true; - return ret; -} - -/* Check the real IOMMU page table against the CPU mapping of each allocation. - * This proves mappings, not that firmware has performed a DMA transaction. - */ -static int ave_verify_mapping(struct apple_ave *ave, struct ave_buffer *buf) -{ - size_t offset; - - if (!IS_ALIGNED(buf->dma, PAGE_SIZE) || !IS_ALIGNED(buf->size, PAGE_SIZE)) - return -EINVAL; - for (offset = 0; offset < buf->size; offset += PAGE_SIZE) { - void *cpu = buf->cpu + offset; - struct page *page = is_vmalloc_addr(cpu) ? vmalloc_to_page(cpu) : virt_to_page(cpu); - phys_addr_t physical = iommu_iova_to_phys(ave->domain, buf->dma + offset); - - if (!page || physical != page_to_phys(page)) { - dev_err(ave->dev, "DMA mapping mismatch at offset %#zx\n", offset); - return -EIO; - } - buf->verified_pages++; - } - return 0; -} - -static int ave_alloc_buffer(struct apple_ave *ave, struct ave_buffer *buf, size_t size) -{ - buf->size = size; - buf->cpu = dma_alloc_coherent(ave->dev, size, &buf->dma, GFP_KERNEL); - if (!buf->cpu) - return -ENOMEM; - memset(buf->cpu, 0, size); - return ave_verify_mapping(ave, buf); -} - -static void ave_free_buffer(struct apple_ave *ave, struct ave_buffer *buf) -{ - if (buf->cpu) - dma_free_coherent(ave->dev, buf->size, buf->cpu, buf->dma); - buf->cpu = NULL; -} - -static void ave_power_off(struct apple_ave *ave) -{ - while (ave->num_power) { - struct device *pd = ave->power[--ave->num_power]; - - pm_runtime_put_sync(pd); - dev_pm_domain_detach(pd, true); - } -} - -static int ave_power_on(struct apple_ave *ave) -{ - static const char * const names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; - int ret; - - for (unsigned int i = 0; i < ARRAY_SIZE(names); i++) { - struct device *pd = dev_pm_domain_attach_by_name(ave->dev, names[i]); - - if (IS_ERR(pd)) - return PTR_ERR(pd); - if (!pd) - return -ENODEV; - ret = pm_runtime_resume_and_get(pd); - if (ret < 0) { - dev_pm_domain_detach(pd, true); - return ret; - } - ave->power[ave->num_power++] = pd; - } - return 0; -} - -static int ave_status_show(struct seq_file *s, void *unused) -{ - struct apple_ave *ave = s->private; - - mutex_lock(&ave->lock); - seq_printf(s, "state=%s\nfirmware_start_requested=%u\nstart_result=%d\n", - ave->start_requested ? (ave->start_result ? "startup-failed" : "ipc-ready") : - "firmware-mapped", ave->start_requested, ave->start_result); - seq_printf(s, "hardware_dma_tested=%u\nencoder_registered=0\nphase=%s\n", - ave->ipc_ready, ave->phase ?: "not-started"); - seq_printf(s, "ipc_ready=%u\nipc_fw_base=%#llx\nchannel_bytes=%#x\nextra_bytes=%#x\n", - ave->ipc_ready, ave->ipc_fw_base, ave->channel_bytes, ave->extra_bytes); - seq_printf(s, "commands=%u\nreplies=%u\ntransport_acks=%u\ncommand_failed=%u\nresponse_bytes=%zu\n", - ave->commands, ave->replies, ave->transport_acks, ave->command_failed, ave->response_bytes); - seq_printf(s, "arena_bytes=%zu\narena_ipc_offset=%#x\narena_dma=%#llx\narena_fw=%#llx\n", - ave->ipc.size - AVE_ARENA_OFF, AVE_ARENA_OFF, - (u64)ave->ipc.dma + AVE_ARENA_OFF, ave->ipc_fw_base + AVE_ARENA_OFF); - for (unsigned int i = 0; i < 2; i++) { - struct ave_channel *c = &ave->channel[i]; - seq_printf(s, "channel%u_name=%s\nchannel%u_mode=%u\nchannel%u_interrupt=%u\n" - "channel%u_slots=%u\nchannel%u_offset=%#x\nchannel%u_consumer=%d\nchannel%u_producer=%d\n", - i, c->name, i, c->mode, i, c->interrupt, i, c->slots, - i, c->offset, i, c->consumer, i, c->producer); - } - seq_printf(s, "page_size=%lu\n", PAGE_SIZE); - seq_printf(s, "powered_domains=%u\n", ave->num_power); - seq_printf(s, "ipc_bytes=%zu\nipc_dma=%pad\nipc_verified_pages=%u\n", - ave->ipc.size, &ave->ipc.dma, ave->ipc.verified_pages); - seq_printf(s, "queue_bytes=%zu\nqueue_dma=%pad\nqueue_verified_pages=%u\n", - ave->queue.size, &ave->queue.dma, ave->queue.verified_pages); - seq_printf(s, "log_bytes=%zu\nlog_dma=%pad\nlog_verified_pages=%u\n", - ave->log.size, &ave->log.dma, ave->log.verified_pages); - seq_printf(s, "heap_bytes=%zu\nheap_dma=%pad\nheap_verified_pages=%u\n", - ave->heap.size, &ave->heap.dma, ave->heap.verified_pages); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->fw); i++) - seq_printf(s, "fw%u_physical=%pa\nfw%u_iova=%#llx\nfw%u_bytes=%zu\n" - "fw%u_verified_pages=%u\nfw%u_crc32=%08x\n", - i, &ave->fw[i].physical, i, ave->fw[i].iova, i, ave->fw[i].size, - i, ave->fw[i].verified_pages, i, ave->fw[i].crc); - seq_printf(s, "register_snapshot=%u\n", snapshot); - if (snapshot) { - seq_printf(s, "vcbar=%#llx\ncore_control=%#x\ncore_status=%#x\nmailbox_status=%#x\n", - ave->vcbar, ave->core_control, ave->core_status, ave->mailbox_status); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) - seq_printf(s, "scratch%u=%#x\n", i, ave->scratch[i]); - } - seq_printf(s, "resources_pinned_until_reboot=%u\n", ave->start_requested); - mutex_unlock(&ave->lock); - return 0; -} -DEFINE_SHOW_ATTRIBUTE(ave_status); - -static int apple_ave_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct apple_ave *ave; - void __iomem *regs, *mailbox, *coproc; - int ret; - - if (!enable) - return -ENODEV; - if (!of_machine_is_compatible("apple,j700") || PAGE_SIZE != SZ_16K) - return -ENODEV; - ave = devm_kzalloc(dev, sizeof(*ave), GFP_KERNEL); - if (!ave) - return -ENOMEM; - ave->dev = dev; - mutex_init(&ave->lock); - /* Claim only dedicated AVE ranges, never the shared PMGR/SRAM windows. */ - regs = devm_platform_ioremap_resource_byname(pdev, "codec"); - if (IS_ERR(regs)) - return PTR_ERR(regs); - mailbox = devm_platform_ioremap_resource_byname(pdev, "mailbox"); - if (IS_ERR(mailbox)) - return PTR_ERR(mailbox); - coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); - if (IS_ERR(coproc)) - return PTR_ERR(coproc); - ave->coproc = coproc; - ave->mailbox = mailbox; - ave->domain = iommu_get_domain_for_dev(dev); - if (!ave->domain || !(ave->domain->type & __IOMMU_DOMAIN_PAGING)) - return dev_err_probe(dev, -ENODEV, "A translated IOMMU domain is required\n"); - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); - if (ret) - return ret; - ret = ave_power_on(ave); - if (ret) - goto release_power; - ret = devm_pm_runtime_enable(dev); - if (ret) - goto release_power; - ret = pm_runtime_resume_and_get(dev); - if (ret < 0) - goto release_power; - ret = ave_alloc_buffer(ave, &ave->ipc, AVE_IPC_BYTES); - if (ret) - goto free_buffers; - ret = ave_alloc_buffer(ave, &ave->queue, AVE_QUEUE_BYTES); - if (ret) - goto free_buffers; - ret = ave_alloc_buffer(ave, &ave->log, AVE_LOG_BYTES); - if (ret) - goto free_buffers; - ret = ave_verify_firmware(ave); - if (ret) - goto free_buffers; - if (snapshot) { - ave->vcbar = readq(coproc + 0x50000); - ave->core_control = readl(coproc + 0x600044); - ave->core_status = readl(coproc + 0x600048); - ave->mailbox_status = readl(mailbox + 0x10); - for (unsigned int i = 0; i < ARRAY_SIZE(ave->scratch); i++) - ave->scratch[i] = readl(mailbox + 0x18 + i * 4); - } - platform_set_drvdata(pdev, ave); - ave->debugfs = debugfs_create_dir(dev_name(dev), NULL); - debugfs_create_file("ave-status", 0400, ave->debugfs, ave, &ave_status_fops); - debugfs_create_file("ave-start-once", 0200, ave->debugfs, ave, &ave_start_fops); - debugfs_create_file("ave-command", 0600, ave->debugfs, ave, &ave_command_fops); - debugfs_create_file("ave-poll", 0200, ave->debugfs, ave, &ave_poll_fops); - debugfs_create_file("ave-arena", 0600, ave->debugfs, ave, &ave_arena_fops); - ave->ipc_blob.data = ave->ipc.cpu; - ave->ipc_blob.size = ave->ipc.size; - debugfs_create_blob("ave-ipc-memory", 0400, ave->debugfs, &ave->ipc_blob); - ave->log_blob.data = ave->log.cpu; - ave->log_blob.size = ave->log.size; - debugfs_create_blob("ave-firmware-log", 0400, ave->debugfs, &ave->log_blob); - ave->config_blob.data = ave->queue.cpu; - ave->config_blob.size = 0x40; - debugfs_create_blob("ave-boot-config", 0400, ave->debugfs, &ave->config_blob); - dev_info(dev, "AVE_DMA_READY ipc=%zu queue=%zu verified_pages=%u firmware_start=0\n", - ave->ipc.size, ave->queue.size, - ave->ipc.verified_pages + ave->queue.verified_pages); - dev_info(dev, "AVE_FIRMWARE_MAPPED pages=%u contents_match_boot=1 register_snapshot=%u\n", - ave->fw[0].verified_pages + ave->fw[1].verified_pages, snapshot); - return 0; - -free_buffers: - ave_free_buffer(ave, &ave->heap); - ave_free_buffer(ave, &ave->log); - ave_free_buffer(ave, &ave->queue); - ave_free_buffer(ave, &ave->ipc); - pm_runtime_put_sync(dev); -release_power: - ave_power_off(ave); - return dev_err_probe(dev, ret, "AVE DMA allocation/verification failed\n"); -} - -static void apple_ave_remove(struct platform_device *pdev) -{ - struct apple_ave *ave = platform_get_drvdata(pdev); - - debugfs_remove_recursive(ave->debugfs); - ave_free_buffer(ave, &ave->heap); - ave_free_buffer(ave, &ave->log); - ave_free_buffer(ave, &ave->queue); - ave_free_buffer(ave, &ave->ipc); - pm_runtime_put_sync(ave->dev); - ave_power_off(ave); - dev_info(ave->dev, "AVE_DMA_RELEASED\n"); -} - -static const struct of_device_id apple_ave_match[] = { - { .compatible = "apple,t8140-ave" }, - { } -}; -MODULE_DEVICE_TABLE(of, apple_ave_match); - -static struct platform_driver apple_ave_driver = { - .probe = apple_ave_probe, - .remove = apple_ave_remove, - .driver = { - .name = "apple-ave", - .of_match_table = apple_ave_match, - .suppress_bind_attrs = true, - }, -}; -module_platform_driver(apple_ave_driver); -MODULE_DESCRIPTION("Apple T8140 experimental video encoder transport"); -MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-cmd.c b/drivers/media/platform/apple/ave/ave-cmd.c new file mode 100644 index 0000000000000..7a03ed53b457b --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-cmd.c @@ -0,0 +1,658 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: firmware commands + * + * Start and Process are large command images. Each is built in a zeroed + * buffer: the header, the constant words (ave-image.c), then every field + * that depends on the session or the frame. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include + +#include + +#include "ave.h" +#include "ave-regs.h" + +/* + * IEEE 754 double for the positive value @num / @den (0 for 0), built with + * integer arithmetic; the firmware takes rates and limits as doubles. + */ +static u64 ave_f64(u64 num, u64 den) +{ + u64 mant = 0, rem; + int exp = 0; + + if (!num || !den) + return 0; + while (num >= den << 1) { + den <<= 1; + exp++; + } + while (num < den) { + num <<= 1; + exp--; + } + /* 1 <= num/den < 2: take 53 significant bits by long division. */ + rem = num; + for (unsigned int i = 0; i < 53; i++) { + mant <<= 1; + if (rem >= den) { + mant |= 1; + rem -= den; + } + rem <<= 1; + } + return (u64)(exp + 1023) << 52 | (mant & GENMASK_ULL(51, 0)); +} + +#define AVE_F64_NEG BIT_ULL(63) + +static void ave_put32(u8 *cmd, u32 off, u32 val) +{ + put_unaligned_le32(val, cmd + off); +} + +static void ave_put64(u8 *cmd, u32 off, u64 val) +{ + put_unaligned_le64(val, cmd + off); +} + +/* A surface descriptor: {u64 DART address, u32 size, u32 0}. */ +static void ave_put_surface(u8 *cmd, u32 off, dma_addr_t addr, u32 size) +{ + ave_put64(cmd, off, addr); + ave_put32(cmd, off + 8, size); + ave_put32(cmd, off + 12, 0); +} + +static void ave_put_buf(u8 *cmd, u32 off, const struct ave_buf *b, u32 size) +{ + ave_put_surface(cmd, off, b->dma, size); +} + +/* Is another session open in the firmware besides @ctx's? */ +static bool ave_cmd_others_open(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] && ave->sessions[i] != ctx && ave->sessions[i]->sess->fw_open) + return true; + return false; +} + +/* + * Header word +0x2c depends on the set of open sessions: 0x1e0 on Open and + * Start (0x140 when another session is open), the codec value on every + * other command of a lone session, and 0 while another one is open. + */ +static u32 ave_session_arg2(struct ave_ctx *ctx) +{ + if (ave_cmd_others_open(ctx)) + return 0; + return ctx->sess->p.codec == AVE_CODEC_HEVC ? AVE_SESSION_ARG2_HEVC : + AVE_SESSION_ARG2_H264; +} + +static u32 ave_session_setup_arg2(struct ave_ctx *ctx) +{ + return ave_cmd_others_open(ctx) ? AVE_SESSION_ARG2_SETUP_SHARED : AVE_SESSION_ARG2_SETUP; +} + +static void ave_cmd_header(struct ave_ctx *ctx, u8 *cmd, enum ave_opcode op, u32 echo, + u32 arg0, u32 arg2) +{ + ctx->sess->token += AVE_TOKEN_STEP; + ave_put32(cmd, AVE_CMD_OPCODE, op); + ave_put64(cmd, AVE_CMD_TOKEN, ctx->sess->token); + ave_put64(cmd, AVE_CMD_CLIENT, ctx->sess->client); + ave_put32(cmd, AVE_CMD_UNIT, AVE_SESSION_UNIT); + ave_put32(cmd, AVE_CMD_CODEC, ctx->sess->p.codec); + ave_put32(cmd, AVE_CMD_OPCODE_ECHO, echo); + ave_put32(cmd, AVE_CMD_ARG0, arg0); + ave_put32(cmd, AVE_CMD_ARG1, AVE_SESSION_ARG1); + ave_put32(cmd, AVE_CMD_ARG2, arg2); +} + +/* Firmware-wide configuration, sent once after every firmware boot. */ +int ave_cmd_config(struct ave_dev *ave) +{ + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; + int ret; + + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CONFIG_BYTES); + ave_put32(cmd, AVE_CMD_OPCODE, AVE_OP_CONFIG); + ave_put32(cmd, AVE_CONFIG_ALL_UNITS, U32_MAX); + cmd[AVE_CONFIG_OPT_A] = 1; + cmd[AVE_CONFIG_OPT_B] = 1; + ave_put32(cmd, AVE_CONFIG_MCC_DSID, AVE_CONFIG_MCC_DSID_T8140); + ret = ave_ipc_submit(ave, buf, AVE_CONFIG_BYTES, AVE_CMD_GLOBAL_TIMEOUT_US, true); + return ret ?: ave_ipc_settle(ave, AVE_CMD_WAIT_MS); +} + +/* + * Firmware-wide stop, sent after every session has closed. The firmware + * neither acknowledges nor answers it. + */ +int ave_cmd_halt(struct ave_dev *ave) +{ + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; + + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CMD_HDR_BYTES); + ave_put32(cmd, AVE_CMD_OPCODE, AVE_OP_HALT); + ave_put32(cmd, AVE_CMD_OPCODE_ECHO, AVE_HALT_ALL); + return ave_ipc_submit(ave, buf, AVE_CMD_HDR_BYTES, AVE_CMD_GLOBAL_TIMEOUT_US, false); +} + +static int ave_cmd_scalar(struct ave_ctx *ctx, enum ave_opcode op, u32 arg0, u32 arg2) +{ + struct ave_dev *ave = ctx->ave; + int buf = ave_ipc_scalar_buf(ave); + u8 *cmd; + + if (buf < 0) + return buf; + cmd = ave_ipc_cmd(ave, buf, AVE_CMD_HDR_BYTES); + ave_cmd_header(ctx, cmd, op, op, arg0, arg2); + return ave_ipc_submit(ave, buf, AVE_CMD_HDR_BYTES, AVE_CMD_SESSION_TIMEOUT_US, true); +} + +int ave_cmd_open(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + int ret; + + ave->next_client += AVE_CLIENT_STEP; + ctx->sess->client = ave->next_client; + ctx->sess->token = AVE_TOKEN_BASE; + ret = ave_cmd_scalar(ctx, AVE_OP_OPEN, AVE_SESSION_ARG0, ave_session_setup_arg2(ctx)); + if (!ret) + ctx->sess->fw_open = true; + return ret ?: ave_ipc_settle(ave, AVE_CMD_WAIT_MS); +} + +/* Drain barrier: answered once every Process sent before it is answered. */ +int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count) +{ + int ret = 0; + + for (unsigned int i = 0; i < count && !ret; i++) + ret = ave_cmd_scalar(ctx, AVE_OP_COMPLETE, 0, ave_session_arg2(ctx)); + return ret; +} + +/* Priority and Stop go out together, Close after Stop is answered. */ +int ave_cmd_close(struct ave_ctx *ctx) +{ + u32 arg2 = ave_session_arg2(ctx); + int ret, err; + + ret = ave_cmd_scalar(ctx, AVE_OP_PRIORITY, AVE_SESSION_ARG0, arg2); + if (!ret) + ret = ave_cmd_scalar(ctx, AVE_OP_STOP, AVE_SESSION_ARG0, arg2); + err = ave_ipc_settle(ctx->ave, AVE_CMD_WAIT_MS); + ret = ret ?: err; + if (ctx->ave->fw_state != AVE_FW_RUNNING) + return ret; + err = ave_cmd_scalar(ctx, AVE_OP_CLOSE, AVE_SESSION_ARG0, arg2); + if (!err) + err = ave_ipc_settle(ctx->ave, AVE_CMD_WAIT_MS); + ctx->sess->fw_open = false; + return ret ?: err; +} + +/* Start (id 6) */ + +#define AVE_START_SURFACE 0x48 /* and an unused second one at 0x58 */ +#define AVE_START_SURFACE_BYTES 0x12c000 +#define AVE_START_SURFACE2 0x58 +#define AVE_START_REF_A(i) (0x390 + 0x20 * (i)) +#define AVE_START_REF_B(i) (0x3a0 + 0x20 * (i)) +#define AVE_START_REF_C(i) (0x7d0 + 0x20 * (i)) +#define AVE_START_REF_C_ALIAS(i) (0x7e0 + 0x20 * (i)) +#define AVE_START_MB 0xc10 +#define AVE_START_CODED(s) (0xd50 + 0x10 * (s)) +#define AVE_START_META(s) (0xf50 + 0x10 * (s)) +#define AVE_START_ARRAY_LEN 30 +#define AVE_START_REF_D(i) (0x1130 + 0x10 * (i)) +#define AVE_START_ROW_A 0x1350 +#define AVE_START_ROW_B 0x1390 +#define AVE_START_FIXED 0x13d0 +#define AVE_START_FIXED_BYTES 0x4000 +#define AVE_START_HEVC_EXTRA(i) (0x2520 + 0x10 * (i)) + +static const u16 ave_start_tile_off[AVE_TILE_BUFS] = { + 0x1410, 0x1430, 0x1490, 0x14b0, /* both codecs */ + 0x1510, 0x1530, 0x1590, 0x15b0, /* H.264 only */ +}; + +/* Session configuration words */ +#define AVE_START_CODEC 0xc0 +#define AVE_START_8BIT 0x208 +#define AVE_START_CONST_214 0x214 +#define AVE_START_CONST_218 0x218 +#define AVE_START_FPS 0x220 +#define AVE_START_RC_MODE 0x234 +#define AVE_START_BITRATE 0x238 +#define AVE_START_INIT_QP(i) (0x240 + 4 * (i)) +#define AVE_START_INIT_QP_VALUE 26 +#define AVE_START_LIMITS 0x250 +#define AVE_START_LIMIT_BYTES 0x258 +#define AVE_START_LIMIT_SECS 0x260 +#define AVE_START_F64_288 0x288 +#define AVE_START_F64_290 0x290 +#define AVE_START_MIN_QP 0x298 +#define AVE_START_MAX_QP 0x29c +#define AVE_START_GOP_FLAGS 0x2a0 +#define AVE_START_P_FRAMES 0x2ac +#define AVE_START_MAX_B 0x2b0 +#define AVE_START_REF_DEPTH 0x2b4 +#define AVE_START_IDR_INTERVAL 0x2b8 +#define AVE_START_F64_2C0 0x2c0 +#define AVE_START_F64_2C8 0x2c8 +#define AVE_START_RC_FLAGS 0x300 +#define AVE_START_MIN_QP2 0x308 +#define AVE_START_MAX_QP2 0x30c +#define AVE_START_RC_310 0x310 +#define AVE_START_SIZE_CLASS 0x320 +#define AVE_START_CODED_W 0x368 +#define AVE_START_CODED_H 0x36c +#define AVE_START_CODEC_FLAGS 0x370 +#define AVE_START_H264_380 0x380 +#define AVE_START_H264_384 0x384 +#define AVE_START_CODEC_2588 0x2588 /* 2 for H.264 High and HEVC */ +#define AVE_START_BIT_DEPTH 0x2598 +#define AVE_START_CODED_H2 0x25dc + +/* H.264 parameter-set words */ +#define AVE_START_H264_PROFILE 0x291c +#define AVE_START_H264_LEVEL 0x2938 +#define AVE_START_H264_MAX_REFS 0x2d44 +#define AVE_START_H264_WIDTH_MBS 0x2d4c +#define AVE_START_H264_HEIGHT_MBS 0x2d50 +#define AVE_START_H264_CROP 0x2db4 +#define AVE_START_H264_CROP_LEFT 0x2db8 +#define AVE_START_H264_CROP_RIGHT 0x2dbc +#define AVE_START_H264_CROP_TOP 0x2dc0 +#define AVE_START_H264_CROP_BOTTOM 0x2dc4 +#define AVE_START_H264_CABAC 0x2fd8 +#define AVE_START_H264_8X8 0x3023 /* byte */ + +/* HEVC parameter-set words; most appear twice */ +#define AVE_START_HEVC_PAD 0x26e0 +#define AVE_START_HEVC_RC_270C 0x270c /* 2 in bit-rate modes, 5 in CQ */ +#define AVE_START_HEVC_RC_9604 0x9604 /* 0 in bit-rate modes, 2 in CQ */ +#define AVE_START_HEVC_RC_9608 0x9608 /* 0x10000 in bit-rate modes, 0 in CQ */ +#define AVE_START_HEVC_PROFILE_A 0x2938 +#define AVE_START_HEVC_COMPAT_A 0x293d /* two bytes */ +#define AVE_START_HEVC_8BIT_A 0x2962 /* byte */ +#define AVE_START_HEVC_LEVEL_A 0x296c +#define AVE_START_HEVC_PROFILE_B 0x5904 +#define AVE_START_HEVC_COMPAT_B 0x5909 +#define AVE_START_HEVC_8BIT_B 0x592e +#define AVE_START_HEVC_LEVEL_B 0x5938 +#define AVE_START_HEVC_WIDTH 0x5ab4 +#define AVE_START_HEVC_HEIGHT 0x5ab8 +#define AVE_START_HEVC_WIDTH_CTB 0x5abc +#define AVE_START_HEVC_HEIGHT_CTB 0x5ac0 +#define AVE_START_HEVC_CONF 0x5ac4 +#define AVE_START_HEVC_CONF_LEFT 0x5ac8 +#define AVE_START_HEVC_CONF_RIGHT 0x5acc +#define AVE_START_HEVC_CONF_TOP 0x5ad0 +#define AVE_START_HEVC_CONF_BOTTOM 0x5ad4 +#define AVE_START_HEVC_DEPTH_LUMA 0x5ad8 +#define AVE_START_HEVC_DEPTH_CHROMA 0x5adc +/* HEVC reference structure with B-frames (reordering on) */ +#define AVE_START_HEVC_REFS 0x259c +#define AVE_START_HEVC_REORDER_COPY 0x2e54 +#define AVE_START_HEVC_DPB_MINUS1 0x5b00 +#define AVE_START_HEVC_REORDER 0x5b04 +#define AVE_START_HEVC_REORDER_ON 0x95fc + +static void ave_start_surfaces(struct ave_session *sess, u8 *cmd) +{ + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + unsigned int tiles = hevc ? 4 : AVE_TILE_BUFS; + + /* Clear every descriptor the image may carry, then fill ours. */ + for (unsigned int i = 0; i < AVE_MAX_REFS; i++) { + ave_put_surface(cmd, AVE_START_REF_A(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_B(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_C(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_C_ALIAS(i), 0, 0); + ave_put_surface(cmd, AVE_START_REF_D(i), 0, 0); + } + for (unsigned int s = 0; s < AVE_START_ARRAY_LEN; s++) { + ave_put_surface(cmd, AVE_START_CODED(s), 0, 0); + ave_put_surface(cmd, AVE_START_META(s), 0, 0); + } + + ave_put_buf(cmd, AVE_START_SURFACE, &sess->hdr, AVE_START_SURFACE_BYTES); + ave_put_surface(cmd, AVE_START_SURFACE2, 0, 0); + for (unsigned int i = 0; i < p->refs; i++) { + ave_put_buf(cmd, AVE_START_REF_A(i), &sess->ref_a[i], sess->ref_a[i].size); + ave_put_buf(cmd, AVE_START_REF_B(i), &sess->ref_b[i], sess->ref_b[i].size); + ave_put_buf(cmd, AVE_START_REF_C(i), &sess->ref_c[i], sess->ref_c[i].size); + ave_put_surface(cmd, AVE_START_REF_C_ALIAS(i), sess->ref_c[i].dma, 0); + ave_put_buf(cmd, AVE_START_REF_D(i), &sess->ref_d[i], sess->ref_d[i].size); + } + ave_put_buf(cmd, AVE_START_MB, &sess->mb, sess->mb.size); + for (unsigned int s = 0; s < p->slots; s++) { + ave_put_buf(cmd, AVE_START_CODED(s), &sess->slot[s].coded, p->coded_cap); + ave_put_buf(cmd, AVE_START_META(s), &sess->slot[s].meta, sess->slot[s].meta.size); + } + ave_put_buf(cmd, AVE_START_ROW_A, &sess->row_a, sess->row_a.size); + ave_put_buf(cmd, AVE_START_ROW_B, &sess->row_b, sess->row_b.size); + ave_put_buf(cmd, AVE_START_FIXED, &sess->fixed, AVE_START_FIXED_BYTES); + for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) { + u32 off = ave_start_tile_off[i]; + + if (i < tiles) { + ave_put_buf(cmd, off, &sess->tile[i], sess->tile[i].size); + ave_put_buf(cmd, off + 0x10, &sess->tile_aux[i], sess->tile_aux[i].size); + } else { + ave_put_surface(cmd, off, 0, 0); + ave_put_surface(cmd, off + 0x10, 0, 0); + } + } + if (hevc) + for (unsigned int i = 0; i < 2; i++) + ave_put_buf(cmd, AVE_START_HEVC_EXTRA(i), &sess->hevc_extra[i], + sess->hevc_extra[i].size); +} + +static void ave_start_config(const struct ave_params *p, u8 *cmd) +{ + bool hevc = p->codec == AVE_CODEC_HEVC; + u32 crop_r = (p->cw - p->width) / 2, crop_b = (p->ch - p->height) / 2; + bool crop = crop_r || crop_b; + + ave_put32(cmd, AVE_START_CODEC, p->codec); + ave_put32(cmd, AVE_START_8BIT, hevc ? 1 : 0); + ave_put32(cmd, AVE_START_CONST_214, 0x39d); + ave_put32(cmd, AVE_START_CONST_218, 0x18); + ave_put32(cmd, AVE_START_FPS, p->fps_int); + + /* Rate control */ + ave_put32(cmd, AVE_START_RC_MODE, p->rc_mode); + ave_put32(cmd, AVE_START_BITRATE, p->bitrate); + for (unsigned int i = 0; i < 3; i++) + ave_put32(cmd, AVE_START_INIT_QP(i), AVE_START_INIT_QP_VALUE); + ave_put32(cmd, AVE_START_LIMITS, p->limit_bytes ? 1 : 0); + if (p->limit_bytes) { + ave_put64(cmd, AVE_START_LIMIT_BYTES, ave_f64(p->limit_bytes, 1)); + ave_put64(cmd, AVE_START_LIMIT_SECS, ave_f64(1, 1)); + } + ave_put64(cmd, AVE_START_F64_288, ave_f64(1, 1)); + ave_put64(cmd, AVE_START_F64_290, ave_f64(1, 2)); + ave_put64(cmd, AVE_START_F64_2C0, AVE_F64_NEG | ave_f64(1, 1)); + ave_put64(cmd, AVE_START_F64_2C8, AVE_F64_NEG | ave_f64(1, 1)); + ave_put32(cmd, AVE_START_MIN_QP, p->min_qp); + ave_put32(cmd, AVE_START_MAX_QP, p->max_qp); + ave_put32(cmd, AVE_START_MIN_QP2, p->min_qp); + ave_put32(cmd, AVE_START_MAX_QP2, p->max_qp); + ave_put32(cmd, AVE_START_RC_FLAGS, hevc ? 0x736 : 0x8202); + ave_put32(cmd, AVE_START_RC_310, 30); + if (!hevc) + ave_put32(cmd, AVE_START_H264_380, 1); + + /* GOP structure: with B-frames as captured, or P-only, or intra-only. */ + ave_put32(cmd, AVE_START_GOP_FLAGS, p->bframes ? 0x600 : 0x200); + ave_put32(cmd, AVE_START_P_FRAMES, p->intra_only ? 0 : 1); + ave_put32(cmd, AVE_START_MAX_B, p->bframes ? 3 : 0); + ave_put32(cmd, AVE_START_REF_DEPTH, p->bframes ? 3 : 1); + ave_put32(cmd, AVE_START_IDR_INTERVAL, p->gop); + + /* Picture size */ + ave_put32(cmd, AVE_START_SIZE_CLASS, p->width * p->height > 1280 * 720 ? 3 : 1); + ave_put32(cmd, AVE_START_CODED_W, p->cw); + ave_put32(cmd, AVE_START_CODED_H, p->ch); + ave_put32(cmd, AVE_START_CODED_H2, p->ch); + ave_put32(cmd, AVE_START_CODEC_FLAGS, hevc ? 0x2000002 : 2); + ave_put32(cmd, AVE_START_BIT_DEPTH, 8); + + if (!hevc) { + ave_put32(cmd, AVE_START_H264_384, p->width * p->height <= 320 * 240 ? 0x40 : 0x10); + ave_put32(cmd, AVE_START_CODEC_2588, p->high ? 2 : 0); + ave_put32(cmd, AVE_START_H264_PROFILE, p->profile); + ave_put32(cmd, AVE_START_H264_LEVEL, p->level); + ave_put32(cmd, AVE_START_H264_MAX_REFS, p->refs - 1); + ave_put32(cmd, AVE_START_H264_WIDTH_MBS, p->cw / 16 - 1); + ave_put32(cmd, AVE_START_H264_HEIGHT_MBS, p->ch / 16 - 1); + ave_put32(cmd, AVE_START_H264_CROP, crop); + ave_put32(cmd, AVE_START_H264_CROP_LEFT, 0); + ave_put32(cmd, AVE_START_H264_CROP_RIGHT, crop_r); + ave_put32(cmd, AVE_START_H264_CROP_TOP, 0); + ave_put32(cmd, AVE_START_H264_CROP_BOTTOM, crop_b); + ave_put32(cmd, AVE_START_H264_CABAC, p->profile != 2); + cmd[AVE_START_H264_8X8] = p->high; + return; + } + + ave_put32(cmd, AVE_START_HEVC_PAD, 1 | + (p->cw > p->width ? BIT(16) | BIT(18) : 0) | + (p->ch > p->height ? BIT(17) | BIT(19) : 0)); + ave_put32(cmd, AVE_START_HEVC_PROFILE_A, p->profile); + ave_put32(cmd, AVE_START_HEVC_PROFILE_B, p->profile); + cmd[AVE_START_HEVC_COMPAT_A] = 1; + cmd[AVE_START_HEVC_COMPAT_A + 1] = 1; + cmd[AVE_START_HEVC_COMPAT_B] = 1; + cmd[AVE_START_HEVC_COMPAT_B + 1] = 1; + cmd[AVE_START_HEVC_8BIT_A] = 1; + cmd[AVE_START_HEVC_8BIT_B] = 1; + /* Rate-control words that only change in constant-quality mode */ + ave_put32(cmd, AVE_START_HEVC_RC_270C, 2); + ave_put32(cmd, AVE_START_HEVC_RC_9604, 0); + ave_put32(cmd, AVE_START_HEVC_RC_9608, 0x10000); + ave_put32(cmd, AVE_START_HEVC_LEVEL_A, p->level); + ave_put32(cmd, AVE_START_HEVC_LEVEL_B, p->level); + ave_put32(cmd, AVE_START_HEVC_WIDTH, p->cw); + ave_put32(cmd, AVE_START_HEVC_HEIGHT, p->ch); + ave_put32(cmd, AVE_START_HEVC_WIDTH_CTB, DIV_ROUND_UP(p->cw, 32)); + ave_put32(cmd, AVE_START_HEVC_HEIGHT_CTB, DIV_ROUND_UP(p->ch, 32)); + ave_put32(cmd, AVE_START_HEVC_CONF, crop); + ave_put32(cmd, AVE_START_HEVC_CONF_LEFT, 0); + ave_put32(cmd, AVE_START_HEVC_CONF_RIGHT, crop_r); + ave_put32(cmd, AVE_START_HEVC_CONF_TOP, 0); + ave_put32(cmd, AVE_START_HEVC_CONF_BOTTOM, crop_b); + ave_put32(cmd, AVE_START_HEVC_DEPTH_LUMA, 0); + ave_put32(cmd, AVE_START_HEVC_DEPTH_CHROMA, 0); + /* Reference structure: B-frame pyramid or low-delay P (ave-image.c) */ + ave_put32(cmd, AVE_START_HEVC_REFS, p->bframes ? 4 : 1); + ave_put32(cmd, AVE_START_HEVC_DPB_MINUS1, 4); + ave_put32(cmd, AVE_START_HEVC_REORDER, p->bframes ? 2 : 0); + ave_put32(cmd, AVE_START_HEVC_REORDER_COPY, p->bframes ? 2 : 0); + ave_put32(cmd, AVE_START_HEVC_REORDER_ON, p->bframes); +} + +int ave_cmd_start(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + size_t len = p->codec == AVE_CODEC_HEVC ? AVE_START_HEVC_BYTES : AVE_START_H264_BYTES; + u8 *cmd; + int ret; + + cmd = ave_ipc_cmd(ave, AVE_BUF_CMD, len); + if (!cmd) + return -EBUSY; + ave_image_start(cmd, len, p->codec, p->reorder_off, + p->codec == AVE_CODEC_HEVC && !p->bframes); + ave_cmd_header(ctx, cmd, AVE_OP_START, AVE_OP_START, AVE_SESSION_ARG0, + ave_session_setup_arg2(ctx)); + ave_start_surfaces(sess, cmd); + ave_start_config(&sess->p, cmd); + ave_debugfs_save(ave, p->codec, AVE_IMAGE_START, cmd, len); + + ret = ave_ipc_submit(ave, AVE_BUF_CMD, len, AVE_CMD_SESSION_TIMEOUT_US, true); + if (ret) + return ret; + sess->handed = true; + return ave_ipc_settle(ave, AVE_START_WAIT_MS); +} + +/* Process (id 8) */ + +struct ave_proc_layout { + u32 first_frame; + u32 y, uv; /* {u64 address, u32 size, u32 stride} */ + u32 in_w, in_h; + u32 coded_cap; + u32 fps_int; + u32 bitrate; + u32 limits, limit_bytes, limit_secs; + u32 const5; + u32 fps; /* f64 */ + u32 timescale; /* u64 */ + u32 slot; + u32 coded, meta; /* u64 addresses */ + u32 ordinal; + u32 pts; /* u64 */ + u32 force_key; /* 1 on a forced key frame */ + u32 force_type; /* 2 on a forced key frame */ + u32 frame_buf; /* H.264 only */ + u32 shdr_table; /* HEVC only */ + u32 crop; /* HEVC only */ + u32 low_delay; /* HEVC only: 3 with reordering off */ +}; + +static const struct ave_proc_layout ave_proc_h264 = { + .first_frame = 0x12c0, + .y = 0x5830, .uv = 0x5850, + .in_w = 0x58e8, .in_h = 0x58ec, + .coded_cap = 0x61c8, + .fps_int = 0x1610, + .bitrate = 0x1620, + .limits = 0x1628, .limit_bytes = 0x1630, .limit_secs = 0x1638, + .const5 = 0x6238, + .fps = 0x6268, .timescale = 0x6278, + .slot = 0x61b4, .coded = 0x61b8, .meta = 0x61c0, + .ordinal = 0x6228, .pts = 0x6270, + .force_key = 0x1658, .force_type = 0x29f8, + .frame_buf = 0x58c8, +}; + +static const struct ave_proc_layout ave_proc_hevc = { + .first_frame = 0x5d90, + .y = 0xa300, .uv = 0xa320, + .in_w = 0xa3b8, .in_h = 0xa3bc, + .coded_cap = 0xac98, + .fps_int = 0x60e0, + .bitrate = 0x60f0, + .limits = 0x60f8, .limit_bytes = 0x6100, .limit_secs = 0x6108, + .const5 = 0xad08, + .fps = 0xad38, .timescale = 0xad48, + .slot = 0xac84, .coded = 0xac88, .meta = 0xac90, + .ordinal = 0xacf8, .pts = 0xad40, + .force_key = 0x6128, .force_type = 0x74c8, + .shdr_table = 0xd88, + .crop = 0x74dc, + .low_delay = 0xcf8, +}; + +#define AVE_PROC_TIMESCALE 14400000ULL +#define AVE_PROC_SHDR_ENTRIES 256 +#define AVE_PROC_SHDR_STRIDE 0x400 +#define AVE_PROC_FORCED_TYPE 2 +#define AVE_PROC_LOW_DELAY 3 + +int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t input, u32 bitrate, bool force_key) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + const struct ave_proc_layout *l = p->codec == AVE_CODEC_HEVC ? &ave_proc_hevc : + &ave_proc_h264; + size_t len = p->codec == AVE_CODEC_HEVC ? AVE_PROCESS_HEVC_BYTES : AVE_PROCESS_H264_BYTES; + struct ave_slot *s = &sess->slot[slot]; + unsigned int buf = AVE_BUF_PROC(sess->index * AVE_MAX_SLOTS + slot); + dma_addr_t y = input; + u64 pts; + u8 *cmd; + int ret; + + ret = ave_ipc_wait_ack(ave, buf); + if (ret) + return ret; + cmd = ave_ipc_cmd(ave, buf, len); + if (!cmd) + return -EBUSY; + ave_image_process(cmd, len, p->codec); + ave_cmd_header(ctx, cmd, AVE_OP_PROCESS, AVE_PROC_SUBTYPE + slot, 0, + ave_session_arg2(ctx)); + + /* Input picture: NV12, chroma right after p->height luma lines. */ + ave_put64(cmd, l->y, y); + ave_put32(cmd, l->y + 8, p->stride * p->height); + ave_put32(cmd, l->y + 12, p->stride); + ave_put64(cmd, l->uv, y + p->stride * p->height); + ave_put32(cmd, l->uv + 8, p->stride * p->height / 2); + ave_put32(cmd, l->uv + 12, p->stride); + ave_put32(cmd, l->in_w, p->width); + ave_put32(cmd, l->in_h, p->height); + + /* Session values repeated in every frame */ + ave_put32(cmd, l->coded_cap, p->coded_cap); + ave_put32(cmd, l->fps_int, p->fps_int); + /* The bit rate may change while streaming; it takes effect here. */ + ave_put32(cmd, l->bitrate, bitrate); + ave_put32(cmd, l->limits, p->limit_bytes ? 1 : 0); + if (p->limit_bytes) { + ave_put64(cmd, l->limit_bytes, ave_f64(p->limit_bytes, 1)); + ave_put64(cmd, l->limit_secs, ave_f64(1, 1)); + } + ave_put32(cmd, l->const5, 5); + ave_put64(cmd, l->fps, ave_f64(p->fps_num, p->fps_den)); + ave_put64(cmd, l->timescale, AVE_PROC_TIMESCALE); + if (l->crop) + ave_put32(cmd, l->crop, p->cw != p->width || p->ch != p->height); + + /* This frame */ + ave_put32(cmd, l->first_frame, sess->frames == 0); + ave_put32(cmd, l->slot, slot); + ave_put64(cmd, l->coded, s->coded.dma); + ave_put64(cmd, l->meta, s->meta.dma); + if (l->frame_buf) + ave_put64(cmd, l->frame_buf, s->aux.dma); + if (l->shdr_table) + for (unsigned int i = 0; i < AVE_PROC_SHDR_ENTRIES; i++) + ave_put64(cmd, l->shdr_table + 8 * i, + s->aux.dma + i * AVE_PROC_SHDR_STRIDE); + ave_put32(cmd, l->ordinal, sess->frames); + pts = mul_u64_u64_div_u64((u64)sess->frames * p->fps_den, AVE_PROC_TIMESCALE, p->fps_num); + ave_put64(cmd, l->pts, pts); + ave_put32(cmd, l->force_key, force_key); + ave_put32(cmd, l->force_type, force_key ? AVE_PROC_FORCED_TYPE : 0); + if (l->low_delay) + ave_put32(cmd, l->low_delay, p->bframes ? 0 : AVE_PROC_LOW_DELAY); + if (!sess->frames) + ave_debugfs_save(ave, p->codec, AVE_IMAGE_PROCESS, cmd, len); + + ret = ave_ipc_submit(ave, buf, len, AVE_CMD_SESSION_TIMEOUT_US, false); + if (ret) + return ret; + s->src = src; + s->busy = true; + s->status = 0; + sess->progress = jiffies; + sess->last_sent = jiffies; + sess->nudged = false; + sess->busy++; + sess->frames++; + return 0; +} diff --git a/drivers/media/platform/apple/ave/ave-drv.c b/drivers/media/platform/apple/ave/ave-drv.c new file mode 100644 index 0000000000000..b25ac9df48a67 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-drv.c @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder + * + * The AVE block runs an encoder firmware on its own coprocessor, which the + * bootloader loads but does not start. This driver starts it on first use, + * talks to it over two shared-memory rings and exposes it as a V4L2 + * stateful encoder. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" + +#define AVE_AUTOSUSPEND_MS 2000 + +/* + * debugfs: apple-ave/last_start_{h264,hevc} hold the last Start built and + * last_process0_{h264,hevc} the first Process of the last session, so the + * generated commands can be compared with reference captures. + */ +static const size_t ave_image_bytes[2][AVE_IMAGE_COUNT] = { + { AVE_START_H264_BYTES, AVE_PROCESS_H264_BYTES }, + { AVE_START_HEVC_BYTES, AVE_PROCESS_HEVC_BYTES }, +}; + +void ave_debugfs_save(struct ave_dev *ave, enum ave_codec codec, enum ave_image kind, + const void *cmd, size_t len) +{ + void *dst = ave->dump[codec - 1][kind]; + + if (dst && len == ave_image_bytes[codec - 1][kind]) + memcpy(dst, cmd, len); +} + +static void ave_debugfs_init(struct ave_dev *ave) +{ + static const char * const names[2][AVE_IMAGE_COUNT] = { + { "last_start_h264", "last_process0_h264" }, + { "last_start_hevc", "last_process0_hevc" }, + }; + + if (!IS_ENABLED(CONFIG_DEBUG_FS)) + return; + ave->debugfs = debugfs_create_dir("apple-ave", NULL); + for (unsigned int c = 0; c < 2; c++) + for (unsigned int k = 0; k < AVE_IMAGE_COUNT; k++) { + struct debugfs_blob_wrapper *blob; + + blob = devm_kzalloc(ave->dev, sizeof(*blob), GFP_KERNEL); + ave->dump[c][k] = devm_kzalloc(ave->dev, ave_image_bytes[c][k], GFP_KERNEL); + if (!blob || !ave->dump[c][k]) { + ave->dump[c][k] = NULL; + continue; + } + blob->data = ave->dump[c][k]; + blob->size = ave_image_bytes[c][k]; + debugfs_create_blob(names[c][k], 0400, ave->debugfs, blob); + } +} + +static const char * const ave_pd_names[] = { "dma", "pipe4", "pipe5", "me0", "me1" }; + +static int ave_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + const struct dev_pm_domain_attach_data pd_data = { + .pd_names = ave_pd_names, + .num_pd_names = ARRAY_SIZE(ave_pd_names), + .pd_flags = PD_FLAG_DEV_LINK_ON, + }; + struct ave_dev *ave; + int ret; + + ave = devm_kzalloc(dev, sizeof(*ave), GFP_KERNEL); + if (!ave) + return -ENOMEM; + ave->dev = dev; + mutex_init(&ave->hw_lock); + platform_set_drvdata(pdev, ave); + + ave->codec = devm_platform_ioremap_resource_byname(pdev, "codec"); + if (IS_ERR(ave->codec)) + return PTR_ERR(ave->codec); + ave->mbox = devm_platform_ioremap_resource_byname(pdev, "mailbox"); + if (IS_ERR(ave->mbox)) + return PTR_ERR(ave->mbox); + ave->coproc = devm_platform_ioremap_resource_byname(pdev, "coproc"); + if (IS_ERR(ave->coproc)) + return PTR_ERR(ave->coproc); + + ave->domain = iommu_get_domain_for_dev(dev); + if (!ave->domain || !(ave->domain->type & __IOMMU_DOMAIN_PAGING)) + return dev_err_probe(dev, -ENODEV, "needs a translating IOMMU\n"); + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(42)); + if (ret) + return ret; + + /* The encoder pipes, the DMA block and the motion engines. */ + ret = devm_pm_domain_attach_list(dev, &pd_data, &ave->pd_list); + if (ret < 0) + return dev_err_probe(dev, ret, "cannot attach power domains\n"); + + pm_runtime_set_autosuspend_delay(dev, AVE_AUTOSUSPEND_MS); + pm_runtime_use_autosuspend(dev); + ret = devm_pm_runtime_enable(dev); + if (ret) + return ret; + + ret = ave_fw_probe(ave); + if (ret) + return ret; + + ret = ave_v4l2_register(ave); + if (ret) { + ave_fw_remove(ave); + return ret; + } + ave_debugfs_init(ave); + return 0; +} + +static void ave_remove(struct platform_device *pdev) +{ + struct ave_dev *ave = platform_get_drvdata(pdev); + + debugfs_remove_recursive(ave->debugfs); + ave_v4l2_unregister(ave); + + /* + * The coprocessor memory can only go once the firmware never ran or + * was reset by a power cycle. A halted coprocessor may keep power + * (its domain is shared with the DARTs), so its memory stays. + */ + pm_runtime_dont_use_autosuspend(ave->dev); + mutex_lock(&ave->hw_lock); + if (ave->fw_state != AVE_FW_OFF) { + mutex_unlock(&ave->hw_lock); + dev_warn(ave->dev, "firmware was started, keeping its memory\n"); + return; + } + mutex_unlock(&ave->hw_lock); + ave_fw_remove(ave); +} + +static int ave_runtime_suspend(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + int ret; + + mutex_lock(&ave->hw_lock); + ret = ave_fw_shutdown(ave); + mutex_unlock(&ave->hw_lock); + return ret; +} + +static int ave_runtime_resume(struct device *dev) +{ + return 0; +} + +static int ave_suspend(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + + /* Let a running job finish; the m2m core holds back new ones. */ + v4l2_m2m_suspend(ave->m2m_dev); + + /* + * The firmware does not survive the power going down. Drain and close + * every open session and halt the firmware, as at the end of a stream; + * each session reopens in a cold-started firmware on its next frame. + */ + mutex_lock(&ave->hw_lock); + ave_v4l2_suspend(ave); + if (ave_fw_shutdown(ave)) + dev_warn(dev, "firmware not answering, it cannot be stopped for sleep\n"); + mutex_unlock(&ave->hw_lock); + return 0; +} + +static int ave_resume(struct device *dev) +{ + struct ave_dev *ave = dev_get_drvdata(dev); + + /* The next boot checks the coprocessor state itself. */ + v4l2_m2m_resume(ave->m2m_dev); + return 0; +} + +static const struct dev_pm_ops ave_pm_ops = { + SYSTEM_SLEEP_PM_OPS(ave_suspend, ave_resume) + RUNTIME_PM_OPS(ave_runtime_suspend, ave_runtime_resume, NULL) +}; + +static const struct of_device_id ave_of_match[] = { + { .compatible = "apple,t8140-ave" }, + { } +}; +MODULE_DEVICE_TABLE(of, ave_of_match); + +static struct platform_driver ave_driver = { + .probe = ave_probe, + .remove = ave_remove, + .driver = { + .name = "apple-ave", + .of_match_table = ave_of_match, + .pm = pm_ptr(&ave_pm_ops), + /* A started coprocessor keeps its memory; do not unbind at will. */ + .suppress_bind_attrs = true, + }, +}; +module_platform_driver(ave_driver); + +MODULE_DESCRIPTION("Apple AVE video encoder"); +MODULE_LICENSE("GPL"); diff --git a/drivers/media/platform/apple/ave/ave-fw.c b/drivers/media/platform/apple/ave/ave-fw.c new file mode 100644 index 0000000000000..9b4d3c3f67662 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-fw.c @@ -0,0 +1,457 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: firmware lifecycle + * + * The bootloader leaves the coprocessor firmware in two reserved, DART-mapped + * segments: TEXT and DATA. Running the firmware changes DATA, so the driver + * keeps a copy of DATA taken before the first boot and puts it back before + * every later boot. That lets the coprocessor lose power (runtime PM, system + * sleep) and start again from the same state as the first time. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +/* + * Firmware builds whose command layouts this driver implements, by the CRC32 + * of their TEXT segment. The command formats change between builds. + */ +static const u32 ave_known_text_crc[] = { + 0x2ce7359e, /* AppleAVE2FW 9003.78.0 */ + 0x69c54454, /* AppleAVE2FW 9003.78.0 (macOS 26.0.x) */ +}; + +static int ave_fw_chosen_crc(struct ave_dev *ave, const char *name, u32 *crc) +{ + struct device_node *chosen = of_find_node_by_path("/chosen"); + const char *str; + int ret; + + if (!chosen) + return -ENOENT; + ret = of_property_read_string(chosen, name, &str); + of_node_put(chosen); + if (ret) + return ret; + return kstrtou32(str, 16, crc); +} + +static int ave_fw_segment(struct ave_dev *ave, unsigned int index, + struct ave_fw_segment *seg) +{ + struct device_node *np; + struct resource res; + const __be32 *prop; + int len, ret; + + np = of_parse_phandle(ave->dev->of_node, "memory-region", index); + if (!np) + return -ENOENT; + ret = of_address_to_resource(np, 0, &res); + if (ret) + goto out; + + /* iommu-addresses = <&ave iova-hi iova-lo size-hi size-lo> */ + prop = of_get_property(np, "iommu-addresses", &len); + if (!prop || len != 5 * sizeof(u32) || + be32_to_cpup(prop) != ave->dev->of_node->phandle) { + ret = -EINVAL; + goto out; + } + seg->phys = res.start; + seg->size = resource_size(&res); + seg->iova = of_read_number(prop + 1, 2); + if (of_read_number(prop + 3, 2) != seg->size || + !IS_ALIGNED(seg->phys, SZ_16K) || !IS_ALIGNED(seg->iova, SZ_16K)) + ret = -EINVAL; +out: + of_node_put(np); + return ret; +} + +static int ave_fw_check_mapping(struct ave_dev *ave, const struct ave_fw_segment *seg) +{ + for (size_t off = 0; off < seg->size; off += PAGE_SIZE) + if (iommu_iova_to_phys(ave->domain, seg->iova + off) != seg->phys + off) + return -EIO; + return 0; +} + +static u32 ave_fw_crc(const struct ave_fw_segment *seg, void **copy) +{ + void *va = memremap(seg->phys, seg->size, MEMREMAP_WB); + u32 crc; + + if (!va) + return 0; + crc = crc32_le(~0U, va, seg->size) ^ ~0U; + if (copy) + memcpy(*copy, va, seg->size); + memunmap(va); + return crc; +} + +/* + * Find the firmware, check that it is a build we speak to and that DATA is + * still as the bootloader left it, and keep a copy of DATA for restarts. + */ +int ave_fw_probe(struct ave_dev *ave) +{ + struct device *dev = ave->dev; + u32 text_crc, data_crc, want; + bool known = false; + int ret; + + if (of_count_phandle_with_args(dev->of_node, "memory-region", NULL) != 2) + return dev_err_probe(dev, -ENODEV, "no preloaded firmware\n"); + + ret = ave_fw_segment(ave, 0, &ave->text); + if (!ret) + ret = ave_fw_segment(ave, 1, &ave->data); + if (ret) + return dev_err_probe(dev, ret, "bad firmware reservation\n"); + + if (ave_fw_check_mapping(ave, &ave->text) || ave_fw_check_mapping(ave, &ave->data)) + return dev_err_probe(dev, -EIO, "firmware is not mapped where it was loaded\n"); + + text_crc = ave_fw_crc(&ave->text, NULL); + for (unsigned int i = 0; i < ARRAY_SIZE(ave_known_text_crc); i++) + known |= text_crc == ave_known_text_crc[i]; + if (!known) + return dev_err_probe(dev, -ENODEV, "unsupported firmware (text CRC %08x)\n", + text_crc); + ave->text_crc = text_crc; + + ave->data_pristine = vmalloc(ave->data.size); + if (!ave->data_pristine) + return -ENOMEM; + data_crc = ave_fw_crc(&ave->data, &ave->data_pristine); + + /* + * The loader records the DATA checksum before anything runs. If it + * differs, the firmware already ran and there is no clean copy left. + */ + ret = ave_fw_chosen_crc(ave, "aurora,ave-data-crc32", &want); + if (ret || data_crc != want) { + vfree(ave->data_pristine); + ave->data_pristine = NULL; + return dev_err_probe(dev, -ENODEV, + "firmware data is not in its boot state\n"); + } + + ave->fw_state = AVE_FW_OFF; + return 0; +} + +static void ave_fw_restore_data(struct ave_dev *ave) +{ + void *va; + + if (!ave->data_dirty) + return; + va = memremap(ave->data.phys, ave->data.size, MEMREMAP_WC); + if (WARN_ON(!va)) + return; + memcpy(va, ave->data_pristine, ave->data.size); + /* The restored image must be in memory before the coprocessor starts. */ + wmb(); + memunmap(va); + ave->data_dirty = false; +} + +void ave_free(struct ave_dev *ave, struct ave_buf *b) +{ + if (b->cpu) + dma_free_attrs(ave->dev, b->alloc, b->cpu, b->dma, b->attrs); + b->cpu = NULL; +} + +/* + * Firmware-visible memory, zeroed and 16 KiB aligned in the encoder's DART. + * @size is what the firmware is told; @alloc (at least @size) is mapped. + */ +int ave_alloc_padded(struct ave_dev *ave, struct ave_buf *b, size_t size, size_t alloc, + unsigned long attrs) +{ + if (b->cpu) + return 0; + b->size = size; + b->alloc = PAGE_ALIGN(max(size, alloc)); + b->attrs = attrs; + b->cpu = dma_alloc_attrs(ave->dev, b->alloc, &b->dma, GFP_KERNEL, attrs); + if (!b->cpu) + return -ENOMEM; + if (!IS_ALIGNED(b->dma, SZ_16K)) { + ave_free(ave, b); + return -EINVAL; + } + return 0; +} + +int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs) +{ + return ave_alloc_padded(ave, b, size, size, attrs); +} + +/* + * The driver generates Start and Process for one firmware build; other + * builds may lay them out differently, so they are not driven. + */ +int ave_fw_check_build(struct ave_dev *ave) +{ + if (ave->text_crc == AVE_FW_TEXT_CRC_9003_78_0) + return 0; + dev_err(ave->dev, "encoder firmware %08x is not validated for encoding (only %08x is)\n", + ave->text_crc, AVE_FW_TEXT_CRC_9003_78_0); + return -ENOTSUPP; +} + +void ave_fw_remove(struct ave_dev *ave) +{ + ave_free(ave, &ave->heap); + ave_free(ave, &ave->log); + ave_free(ave, &ave->boot_cfg); + ave_free(ave, &ave->ipc); + vfree(ave->data_pristine); + ave->data_pristine = NULL; +} + +static bool ave_fw_idle(struct ave_dev *ave) +{ + if (readq(ave->coproc + AVE_COPROC_VCBAR) != (ave->text.iova | AVE_COPROC_VCBAR_VALID)) + return false; + if (readl(ave->coproc + AVE_COPROC_CTRL) || readl(ave->mbox + AVE_MBOX_STATUS)) + return false; + for (unsigned int i = 0; i < AVE_MBOX_SCRATCH_COUNT; i++) + if (readl(ave->mbox + AVE_MBOX_SCRATCH(i))) + return false; + return true; +} + +/* + * After a Halt the firmware is restarted from a power cycle, as macOS does + * for every session: the encoder must have been runtime suspended in + * between, which puts the coprocessor back in reset. Hardware runs showed + * that a restart while it kept power gets no answer, so that is refused + * (-EAGAIN) and the state stays halted until the power really went. The + * code base register must still be set: the driver never writes it. + */ +static int ave_fw_prepare_restart(struct ave_dev *ave) +{ + u32 status; + + if (readq(ave->coproc + AVE_COPROC_VCBAR) != (ave->text.iova | AVE_COPROC_VCBAR_VALID)) { + dev_err(ave->dev, "coprocessor lost its code base (vcbar %#llx)\n", + readq(ave->coproc + AVE_COPROC_VCBAR)); + return -EIO; + } + if (ave->fw_state != AVE_FW_HALTED) + return 0; + + if (readl(ave->coproc + AVE_COPROC_CTRL)) { + dev_dbg(ave->dev, "coprocessor kept power after its halt; retry once it suspends\n"); + return -EAGAIN; + } + status = readl(ave->mbox + AVE_MBOX_STATUS); + if (status) + writel(status, ave->mbox + AVE_MBOX_STATUS); + for (unsigned int i = 0; i < AVE_MBOX_SCRATCH_COUNT; i++) + writel(0, ave->mbox + AVE_MBOX_SCRATCH(i)); + return 0; +} + +/* Boot the coprocessor and bring up its command rings. Caller holds hw_lock. */ +static int ave_fw_boot(struct ave_dev *ave) +{ + __le32 *cfg, *log; + u32 status, hello[4]; + int ret; + + ret = ave_alloc(ave, &ave->ipc, AVE_IPC_BYTES, 0); + if (!ret) + ret = ave_alloc(ave, &ave->boot_cfg, SZ_16K, 0); + if (!ret) + ret = ave_alloc(ave, &ave->log, AVE_LOG_BYTES, 0); + if (ret) + return ret; + + if (ave_fw_check_mapping(ave, &ave->text) || ave_fw_check_mapping(ave, &ave->data)) { + dev_err(ave->dev, "firmware segments are no longer mapped\n"); + return -EIO; + } + + ret = ave_fw_prepare_restart(ave); + if (ret) + return ret; + + /* + * The bootloader leaves the coprocessor stopped with its code base + * set. Do not start it from any other state. + */ + if (!ave_fw_idle(ave)) { + dev_err(ave->dev, "coprocessor is not in its reset state (vcbar %#llx ctrl %#x)\n", + readq(ave->coproc + AVE_COPROC_VCBAR), + readl(ave->coproc + AVE_COPROC_CTRL)); + return -EIO; + } + + memset(ave->ring, 0, sizeof(ave->ring)); + ave->unacked = 0; + memset(ave->want, 0, sizeof(ave->want)); + ave->reply_err = 0; + ave->scalar_next = 0; + + ave_fw_restore_data(ave); + + cfg = ave->boot_cfg.cpu; + log = ave->log.cpu; + memset(cfg, 0, AVE_BOOT_CFG_BYTES); + memset(log, 0, ave->log.size); + cfg[AVE_BOOT_CFG_DEVICE] = cpu_to_le32(AVE_BOOT_CFG_DEVICE_T8140); + cfg[AVE_BOOT_CFG_UNITS] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_GROUPS] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_SUBID_MASK] = cpu_to_le32(1); + cfg[AVE_BOOT_CFG_LOG_LO] = cpu_to_le32(lower_32_bits(ave->log.dma)); + cfg[AVE_BOOT_CFG_LOG_HI] = cpu_to_le32(upper_32_bits(ave->log.dma)); + cfg[AVE_BOOT_CFG_LOG_BYTES] = cpu_to_le32(AVE_LOG_BYTES); + cfg[AVE_BOOT_CFG_OPTIONS] = cpu_to_le32(AVE_BOOT_CFG_OPTIONS_VALUE); + log[0] = cpu_to_le32(AVE_LOG_HDR_USED); + log[1] = cpu_to_le32(AVE_LOG_HDR_BYTES); + log[2] = cpu_to_le32(AVE_LOG_RING_BYTES); + dma_wmb(); + + writel(AVE_BOOT_MAGIC, ave->mbox + AVE_MBOX_SCRATCH(0)); + writel(lower_32_bits(ave->boot_cfg.dma), ave->mbox + AVE_MBOX_SCRATCH(1)); + writel(upper_32_bits(ave->boot_cfg.dma), ave->mbox + AVE_MBOX_SCRATCH(2)); + + ave->data_dirty = true; + writel(1, ave->coproc + AVE_COPROC_KEEP_MAPPINGS); + writel(0, ave->coproc + AVE_COPROC_CTRL); + writel(AVE_COPROC_BOOT_ARG_VALUE, ave->coproc + AVE_COPROC_BOOT_ARG); + writel(AVE_COPROC_CTRL_RUN, ave->coproc + AVE_COPROC_CTRL); + + ret = readl_poll_timeout(ave->mbox + AVE_MBOX_STATUS, status, + status & AVE_MBOX_STATUS_PENDING, 50, AVE_MBOX_TIMEOUT_US); + if (ret) { + dev_err(ave->dev, "firmware did not answer (status %#x)\n", + readl(ave->coproc + AVE_COPROC_STATUS)); + return ret; + } + for (unsigned int i = 0; i < 4; i++) + hello[i] = readl(ave->mbox + AVE_MBOX_SCRATCH(i)); + dma_rmb(); + + ret = ave_ipc_negotiate(ave, hello); + if (ret) { + dev_err(ave->dev, "firmware start-up failed: %d\n", ret); + return ret; + } + + ave->fw_state = AVE_FW_RUNNING; + ret = ave_cmd_config(ave); + if (ret) + return ret; + + dev_dbg(ave->dev, "firmware running, %u/%u ring slots\n", + ave->ring[0].slots, ave->ring[1].slots); + return 0; +} + +/* Caller holds hw_lock. */ +int ave_fw_ensure_running(struct ave_dev *ave) +{ + enum ave_fw_state prev; + int ret; + + lockdep_assert_held(&ave->hw_lock); + switch (ave->fw_state) { + case AVE_FW_RUNNING: + return 0; + case AVE_FW_BROKEN: + dev_err(ave->dev, "firmware stopped answering earlier; reboot to use the encoder\n"); + return -EIO; + case AVE_FW_OFF: + case AVE_FW_HALTED: + break; + } + + /* + * A failed boot has no session buffers in the firmware's hands. If the + * coprocessor was set going, treat it as halted: the next attempt + * then waits for a power cycle. Otherwise nothing changed. + */ + prev = ave->fw_state; + ret = ave_fw_boot(ave); + if (ret) { + ave->fw_state = readl(ave->coproc + AVE_COPROC_CTRL) ? AVE_FW_HALTED : prev; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; + } + return ret; +} + +/* + * Tell the firmware to stop, once every session is closed. Halt gets no + * ack and no answer; after a short wait the coprocessor is treated as + * stopped and the next session restarts it through the boot path. + * Caller holds hw_lock. + */ +void ave_fw_halt(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) + return; + if (ave_cmd_halt(ave)) { + dev_err(ave->dev, "cannot send Halt; firmware treated as stuck\n"); + ave->fw_state = AVE_FW_BROKEN; + return; + } + msleep(AVE_HALT_SETTLE_MS); + ave->fw_state = AVE_FW_HALTED; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; +} + +/* + * Stop the firmware before its power goes away. Caller holds hw_lock. + * + * Sessions keep the device runtime active, so the firmware has normally + * been halted already. A firmware that stopped answering keeps its power: + * whether it still has transfers in flight cannot be checked, and only a + * reboot brings it back. + */ +int ave_fw_shutdown(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + switch (ave->fw_state) { + case AVE_FW_RUNNING: + ave_fw_halt(ave); + return ave->fw_state == AVE_FW_HALTED ? 0 : -EBUSY; + case AVE_FW_BROKEN: + return -EBUSY; + default: + return 0; + } +} + +/* The coprocessor lost power behind the firmware's back. Caller holds hw_lock. */ +void ave_fw_lost(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + ave->fw_state = AVE_FW_OFF; + memset(ave->ring, 0, sizeof(ave->ring)); + ave->ipc_fw_base = 0; +} diff --git a/drivers/media/platform/apple/ave/ave-image.c b/drivers/media/platform/apple/ave/ave-image.c new file mode 100644 index 0000000000000..273dd2a69b013 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-image.c @@ -0,0 +1,284 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: constant words of the session commands + * + * macOS builds Start and Process in code, from the session settings and + * its own defaults; nothing is stored. Besides the fields the driver + * computes (sizes, geometry, profile, level, rate control, GOP, buffer + * addresses, per-frame values), every captured command on firmware + * AppleAVE2FW 9003.78.0 holds the same small set of constant words. They + * are listed here as facts, grouped by where they sit; for most of them + * the meaning is not known, only that macOS always sets that value. The + * commands are built in a zeroed buffer: these words first, then the + * computed fields. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +/* @count consecutive u32 words at @off, each set to @val. */ +struct ave_word { + u32 off; + u16 count; + u32 val; +}; + +#define W(o, v) { .off = (o), .count = 1, .val = (v) } +#define RUN(o, n, v) { .off = (o), .count = (n), .val = (v) } + +/* IEEE 754 doubles, high words only (the low words are zero) */ +#define F64_HI_1_0 0x3ff00000 /* 1.0 */ +#define F64_HI_0_5 0x3fe00000 /* 0.5 */ +#define F64_HI_M1_0 0xbff00000 /* -1.0 */ + +/* Session configuration words common to both codecs (Start +0x68..) */ +static const struct ave_word ave_start_common[] = { + /* First surface size: always the 0x12c000 header surface */ + W(0x50, 0x12c000), + W(0x94, 0x36), + W(0x214, 0x39d), + W(0x218, 0x18), + /* Rate-control doubles: 1.0, 0.5 (the 0.5 s VBV buffer), -1, -1 */ + W(0x28c, F64_HI_1_0), + W(0x294, F64_HI_0_5), + W(0x2c4, F64_HI_M1_0), + W(0x2cc, F64_HI_M1_0), + W(0x2d0, 1), + W(0x2e4, 1), + RUN(0x2f4, 3, 1), + RUN(0x338, 2, 0xffffffff), + /* Buffer set: metadata sizes of slots 0..2, +0x13d0 surface size */ + W(0xf58, 0xc000), + W(0xf68, 0xc000), + W(0xf78, 0xc000), + W(0x13d8, 0x4000), +}; + +static const struct ave_word ave_h264_start[] = { + /* Codec (1) and four flags after it */ + RUN(0xc0, 5, 1), + W(0x370, 2), + /* Codec settings block (+0x2580..): see also ave_start_codec_bytes() */ + W(0x258c, 3), + W(0x2594, 1), + W(0x2598, 8), /* bit depth */ + W(0x25a4, 0x100), + W(0x25cc, 1), + W(0x25d4, 1), + W(0x26dc, 4), + W(0x26f8, 0x15), + RUN(0x2704, 2, 1), + W(0x2734, 1), + W(0x2738, 0x10), + W(0x27c0, 8), + /* Sequence and picture parameter block (+0x2900..) */ + W(0x2940, 1), + W(0x2970, 0x100000), + RUN(0x2974, 239, 0x100010), + W(0x2d30, 0x10), + W(0x2d34, 1), + W(0x2d3c, 2), + RUN(0x2d54, 2, 1), + W(0x2d5c, 0x1000000), + W(0x2d60, 5), + W(0x2d68, 1), + RUN(0x2d6c, 3, 2), + W(0x2dc8, 1), + /* Picture parameters: byte +0x3020 is 1; +0x3023 (8x8 transform) is computed */ + W(0x3020, 1), + /* Two runs of -13 */ + RUN(0x3004, 7, 0xfffffff3), + RUN(0x302c, 7, 0xfffffff3), + W(0x3048, 1), + /* Execution-unit info */ + W(0x3150, 1), + W(0x3158, 1), +}; + +static const struct ave_word ave_hevc_start[] = { + /* Codec (2) and four flags after it */ + W(0xc0, 2), + RUN(0xc4, 4, 1), + W(0x370, 0x2000002), + W(0x1358, 0x4000), /* +0x1350 surface size, fixed for HEVC */ + /* Codec settings block (+0x2580..): see also ave_start_codec_bytes() */ + W(0x2588, 2), + W(0x2594, 1), + W(0x2598, 8), /* bit depth */ + W(0x25a4, 0x100), + W(0x25cc, 1), + W(0x25d4, 1), + W(0x26dc, 4), + RUN(0x26f8, 2, 0x15), + W(0x2700, 2), + RUN(0x2704, 2, 1), + W(0x2710, 1), + W(0x2734, 1), + W(0x2738, 0x10), + W(0x27c0, 8), + /* Profile, tier and level block, first copy (+0x2920..) */ + W(0x2920, 0x101), + W(0x292c, 1), + W(0x2934, 1), + W(0x295c, 0x1010001), + /* Constraint flag bytes; +0x2962 (max 8 bit) is computed */ + W(0x2960, 0x01000101), + W(0x2964, 1), + RUN(0x2a8c, 7, 0x1010101), + W(0x2aa8, 0x101), + W(0x2abc, 0x1010101), + W(0x2ac0, 0x101), + RUN(0x2e38, 7, 4), + /* Timing pair 1001 / 24000, also at +0x74f0 */ + W(0x3298, 1001), + W(0x329c, 24000), + /* Sequence parameter block, second copy (+0x58f0..) */ + W(0x58f8, 1), + W(0x5900, 1), + W(0x5928, 0x1010001), + /* Constraint flag bytes, second copy; +0x592e (max 8 bit) is computed */ + W(0x592c, 0x01000000), + W(0x5930, 1), + W(0x5aac, 1), + W(0x5ae0, 7), /* log2_max_pic_order_cnt_lsb_minus4: 11-bit POC */ + RUN(0x5ae8, 7, 4), /* ends with sps_max_dec_pic_buffering_minus1 */ + W(0x5b40, 2), + W(0x5b48, 3), + W(0x5b4c, 1), + W(0x5b54, 1), + /* Picture parameter and slice defaults (+0x7400..) */ + W(0x7400, 0x100), + RUN(0x7404, 2, 7), + W(0x740c, 1), + W(0x7414, 1), + W(0x749c, 1), + W(0x74b0, 0x100), + W(0x74b4, 5), + W(0x74bc, 1), + RUN(0x74c0, 3, 2), + W(0x74f0, 1001), + W(0x74f4, 24000), + W(0x7500, 0x101), + W(0x7504, 1), + W(0x7508, 3), + W(0x750c, 5), + RUN(0x7510, 3, 0x17), + RUN(0x7524, 2, 0xbebb), + W(0x7558, 0x100), + /* Rate-control block (+0x9600..), with two runs of -13 */ + W(0x960c, 2), + RUN(0x9614, 7, 0xfffffff3), + RUN(0x9634, 7, 0xfffffff3), + W(0x9654, 0x100), + W(0x9e64, 0x10000), + W(0xb734, 2), + W(0xb774, 1), + /* Execution-unit info */ + RUN(0x13ef4, 2, 1), + W(0x13f00, 1), +}; + +/* + * HEVC with frame reordering off: a low-delay P reference structure (the + * reference picture sets hold the delta POCs -1 .. -4). All zero with + * B-frames. + */ +static const struct ave_word ave_hevc_start_low_delay[] = { + W(0xe120, 4), W(0xe154, 4), W(0xe17c, 0x01010101), + W(0xe1dc, 4), W(0xe1e4, 0x01010101), + W(0xe204, -1), W(0xe208, -2), W(0xe20c, -3), W(0xe210, -4), + W(0xe244, 1), W(0xe248, 2), W(0xe24c, 3), W(0xe250, 4), + W(0xe284, 4), W(0xe2b8, 1), W(0xe2e0, 1), W(0xe340, 1), W(0xe348, 1), + W(0xe368, -1), W(0xe3a8, 1), W(0xe3e8, 1), + W(0xe41c, 2), W(0xe444, 0x101), W(0xe4a4, 2), W(0xe4ac, 0x101), + W(0xe4cc, -1), W(0xe4d0, -2), + W(0xe50c, 1), W(0xe510, 2), W(0xe54c, 2), + W(0xe580, 3), W(0xe5a8, 0x10101), W(0xe608, 3), W(0xe610, 0x10101), + W(0xe630, -1), W(0xe634, -2), W(0xe638, -3), + W(0xe670, 1), W(0xe674, 2), W(0xe678, 3), W(0xe6b0, 3), +}; + +/* Process: the first-frame flag and per-frame values are computed. */ +static const struct ave_word ave_h264_process[] = { + /* Codec block */ + W(0x99c, 1), + W(0x9a8, 2), + W(0x9bc, 1), + W(0x9d4, 1), + W(0xeb4, 1), + W(0x12bc, 0xffffffff), + /* Picture-management block */ + W(0x29fc, 0xffffffff), + W(0x6238, 5), + W(0x6278, 14400000), /* presentation timescale, low word */ +}; + +static const struct ave_word ave_hevc_process[] = { + /* Codec block */ + W(0x9a8, 1), + W(0x9b8, 2), + W(0xab8, 0x10100), + W(0xb44, 0x10000), + RUN(0xb4c, 2, 5), + RUN(0xd70, 2, 0xffffffff), + W(0xd7c, 1), + /* Picture-management block */ + W(0x74cc, 0xffffffff), + W(0x74d0, 0x100), + W(0x74e0, 1), + W(0xad08, 5), + W(0xad3c, 0x403e0000), /* frame rate 30.0, rewritten per session */ + W(0xad48, 14400000), /* presentation timescale, low word */ + W(0xad68, 0xffffffff), +}; + +static void ave_put_words(u8 *cmd, size_t len, const struct ave_word *w, size_t n) +{ + for (size_t i = 0; i < n; i++) + for (unsigned int j = 0; j < w[i].count; j++) { + size_t off = w[i].off + 4 * j; + + if (WARN_ON_ONCE(off + 4 > len)) + return; + put_unaligned_le32(w[i].val, cmd + off); + } +} + +/* + * The 8 bytes at Start +0x2580 look like a DART address but are separate + * settings: byte 0 is 4 for HEVC, byte 3 is 1, byte 5 is 1 unless the + * client turned frame reordering off (H.264 only; HEVC keeps 1). + */ +static void ave_start_codec_bytes(u8 *cmd, enum ave_codec codec, bool reorder_off) +{ + cmd[0x2580] = codec == AVE_CODEC_HEVC ? 4 : 0; + cmd[0x2583] = 1; + cmd[0x2585] = codec == AVE_CODEC_H264 && reorder_off ? 0 : 1; +} + +void ave_image_start(u8 *cmd, size_t len, enum ave_codec codec, bool reorder_off, + bool hevc_low_delay) +{ + ave_put_words(cmd, len, ave_start_common, ARRAY_SIZE(ave_start_common)); + if (codec == AVE_CODEC_HEVC) { + ave_put_words(cmd, len, ave_hevc_start, ARRAY_SIZE(ave_hevc_start)); + if (hevc_low_delay) + ave_put_words(cmd, len, ave_hevc_start_low_delay, + ARRAY_SIZE(ave_hevc_start_low_delay)); + } else { + ave_put_words(cmd, len, ave_h264_start, ARRAY_SIZE(ave_h264_start)); + } + ave_start_codec_bytes(cmd, codec, reorder_off); +} + +void ave_image_process(u8 *cmd, size_t len, enum ave_codec codec) +{ + if (codec == AVE_CODEC_HEVC) + ave_put_words(cmd, len, ave_hevc_process, ARRAY_SIZE(ave_hevc_process)); + else + ave_put_words(cmd, len, ave_h264_process, ARRAY_SIZE(ave_h264_process)); +} diff --git a/drivers/media/platform/apple/ave/ave-ipc.c b/drivers/media/platform/apple/ave/ave-ipc.c new file mode 100644 index 0000000000000..974f3b2e8cc90 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-ipc.c @@ -0,0 +1,539 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: coprocessor mailbox and command rings + * + * The mailbox negotiation and the 64-byte ring protocol here come from + * Acelogic's J700 AVE transport (the first Linux code to drive this + * firmware); this file reshapes it into the driver core. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +static int ave_fw_offset(struct ave_dev *ave, u64 addr, size_t len, u32 *offset) +{ + u64 delta = addr - ave->ipc_fw_base; + + if (addr < ave->ipc_fw_base || delta > ave->ipc.size || + len > ave->ipc.size - delta) + return -ERANGE; + *offset = delta; + return 0; +} + +static void ave_mbox_send(struct ave_dev *ave, u32 a, u32 b, u32 c, u32 d) +{ + writel(a, ave->mbox + AVE_MBOX_SCRATCH(0)); + writel(b, ave->mbox + AVE_MBOX_SCRATCH(1)); + writel(c, ave->mbox + AVE_MBOX_SCRATCH(2)); + writel(d, ave->mbox + AVE_MBOX_SCRATCH(3)); + /* All four words must land before the doorbell. */ + wmb(); + writel(1, ave->mbox + AVE_MBOX_DOORBELL); +} + +static int ave_mbox_recv(struct ave_dev *ave, u32 reply[4]) +{ + u32 status; + int ret; + + ret = readl_poll_timeout(ave->mbox + AVE_MBOX_STATUS, status, + status & AVE_MBOX_STATUS_PENDING, 50, + AVE_MBOX_TIMEOUT_US); + if (ret) + return ret; + + /* Acknowledge before reading the scratch words. */ + writel(AVE_MBOX_STATUS_PENDING, ave->mbox + AVE_MBOX_STATUS); + for (unsigned int i = 0; i < 4; i++) + reply[i] = readl(ave->mbox + AVE_MBOX_SCRATCH(i)); + dma_rmb(); + return 0; +} + +static int ave_ring_setup(struct ave_dev *ave) +{ + bool found[2] = { false, false }; + + for (unsigned int i = 0; i < 2; i++) { + u8 *desc = ave->ipc.cpu + AVE_IPC_CHANNEL_OFF + i * AVE_CHAN_DESC_STRIDE; + u32 dir = get_unaligned_le32(desc + AVE_CHAN_DESC_DIR); + u32 doorbell = get_unaligned_le32(desc + AVE_CHAN_DESC_IRQ); + u32 slots = get_unaligned_le32(desc + AVE_CHAN_DESC_SLOTS); + u64 addr = get_unaligned_le64(desc + AVE_CHAN_DESC_ADDR); + struct ave_ring *r; + unsigned int id; + u32 offset; + + if (!memchr(desc, 0, AVE_CHAN_DESC_NAME_LEN)) + return -EPROTO; + if (!strcmp(desc, "IO")) + id = 0; + else if (!strcmp(desc, "IO_T2H")) + id = 1; + else + return -EPROTO; + + if (found[id] || dir != id || !doorbell || doorbell >= 32 || !slots || + slots > ave->channel_bytes / AVE_RING_SLOT_BYTES || + !IS_ALIGNED(addr, AVE_RING_SLOT_BYTES) || + ave_fw_offset(ave, addr, (size_t)slots * AVE_RING_SLOT_BYTES, &offset) || + offset < AVE_IPC_CHANNEL_OFF + 2 * AVE_CHAN_DESC_STRIDE || + offset + slots * AVE_RING_SLOT_BYTES > AVE_IPC_CHANNEL_OFF + ave->channel_bytes) + return -EPROTO; + + found[id] = true; + r = &ave->ring[id]; + strscpy(r->name, desc, sizeof(r->name)); + /* Host-to-firmware rings start empty, firmware-to-host rings full. */ + r->mode = dir ^ 1; + r->doorbell = doorbell; + r->slots = slots; + r->offset = offset; + r->consumer = r->mode ? -1 : 0; + r->producer = r->mode ? 0 : -1; + } + + if (!found[0] || !found[1]) + return -EPROTO; + if (ave->ring[0].offset < ave->ring[1].offset + ave->ring[1].slots * AVE_RING_SLOT_BYTES && + ave->ring[1].offset < ave->ring[0].offset + ave->ring[0].slots * AVE_RING_SLOT_BYTES) + return -EPROTO; + + /* Hand every host-to-firmware slot to the host; ownership is bit 0. */ + for (unsigned int i = 0; i < ave->ring[0].slots; i++) { + __le64 *slot = ave->ipc.cpu + ave->ring[0].offset + i * AVE_RING_SLOT_BYTES; + + slot[0] = cpu_to_le64(1); + slot[1] = 0; + slot[2] = 0; + } + dma_wmb(); + return 0; +} + +/* + * Finish the coprocessor start-up once the firmware has answered the boot + * request: hand it the heap and the IPC window, then the channel area, and + * read back the two ring descriptors it writes there. + */ +int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]) +{ + u8 *hs = ave->ipc.cpu + AVE_IPC_HANDSHAKE_OFF; + u64 ticks, offset_us; + u32 freq, status, reply[4], off; + int ret; + + if (hello[0] != AVE_HELLO_VERSION || hello[2] != AVE_HELLO_DESC_BYTES || + hello[1] < 2 * AVE_CHAN_DESC_STRIDE || + hello[1] > AVE_IPC_HANDSHAKE_OFF - AVE_IPC_CHANNEL_OFF || + !hello[3] || hello[3] > 4 * SZ_1M) + return -EPROTO; + ave->channel_bytes = hello[1]; + + if (ave->heap.cpu && ave->heap.size != PAGE_ALIGN(hello[3])) + return -EPROTO; + if (!ave->heap.cpu) { + ave->heap.size = PAGE_ALIGN(hello[3]); + ave->heap.alloc = ave->heap.size; + ave->heap.cpu = dma_alloc_coherent(ave->dev, ave->heap.size, + &ave->heap.dma, GFP_KERNEL); + if (!ave->heap.cpu) + return -ENOMEM; + } + memset(ave->heap.cpu, 0, ave->heap.size); + writel(AVE_MBOX_STATUS_PENDING, ave->mbox + AVE_MBOX_STATUS); + + /* Tell the firmware how far its timebase is from the host's. */ + ticks = readq(ave->coproc + AVE_COPROC_TIMEBASE); + freq = readl(ave->coproc + AVE_COPROC_TIMEBASE_FREQ); + if (freq && freq < 1000000) + return -EPROTO; + ticks = div64_u64(ticks, freq ? freq / 1000000 : 24); + offset_us = div_u64(ktime_get_ns(), NSEC_PER_USEC) - ticks; + writel(lower_32_bits(offset_us), ave->mbox + AVE_MBOX_TIME_LO); + writel(upper_32_bits(offset_us), ave->mbox + AVE_MBOX_TIME_HI); + + ave_mbox_send(ave, lower_32_bits(ave->ipc.dma), upper_32_bits(ave->ipc.dma), + ave->ipc.size, 0); + ret = ave_mbox_recv(ave, reply); + if (ret) + return ret; + ave->ipc_fw_base = (u64)reply[1] << 32 | reply[0]; + if (!ave->ipc_fw_base || !IS_ALIGNED(ave->ipc_fw_base, SZ_16K) || + ave->ipc_fw_base > U64_MAX - ave->ipc.size) + return -EPROTO; + + memset(ave->ipc.cpu + AVE_IPC_CHANNEL_OFF, 0, ave->channel_bytes); + memset(hs, 0, AVE_HS_BYTES); + memset(ave->ipc.cpu + AVE_IPC_SHARED_OFF, 0, AVE_IPC_SHARED_BYTES); + put_unaligned_le64(ave->ipc_fw_base + AVE_IPC_CHANNEL_OFF, hs + AVE_HS_CHANNELS); + put_unaligned_le64(ave->ipc_fw_base + AVE_IPC_SHARED_OFF, hs + AVE_HS_SHARED); + put_unaligned_le32(AVE_IPC_SHARED_BYTES, hs + AVE_HS_SHARED_BYTES); + put_unaligned_le64(ave->heap.dma, hs + AVE_HS_HEAP); + put_unaligned_le32(hello[3], hs + AVE_HS_HEAP_BYTES); + put_unaligned_le32(AVE_HS_MAGIC_VALUE, hs + AVE_HS_MAGIC); + dma_wmb(); + + ave_mbox_send(ave, lower_32_bits(ave->ipc_fw_base + AVE_IPC_HANDSHAKE_OFF), + upper_32_bits(ave->ipc_fw_base + AVE_IPC_HANDSHAKE_OFF), 0, 0); + ret = ave_mbox_recv(ave, reply); + if (ret) + return ret; + if (ave_fw_offset(ave, (u64)reply[1] << 32 | reply[0], ave->channel_bytes, &off) || + off != AVE_IPC_CHANNEL_OFF || reply[2] > AVE_HELLO_EXTRA_MAX) + return -EPROTO; + + ret = ave_ring_setup(ave); + if (ret) + return ret; + + writel(AVE_BOOT_MAGIC, ave->mbox + AVE_MBOX_READY); + return readl_poll_timeout(ave->mbox + AVE_MBOX_READY, status, !status, 50, + AVE_MBOX_TIMEOUT_US); +} + +static int ave_ring_send(struct ave_dev *ave, struct ave_ring *r, u64 addr, u32 len) +{ + __le64 *slot; + int next; + + if (r->producer < 0) + return -ENOSPC; + + slot = ave->ipc.cpu + r->offset + r->producer * AVE_RING_SLOT_BYTES; + WRITE_ONCE(slot[1], cpu_to_le64(len)); + WRITE_ONCE(slot[2], 0); + dma_wmb(); + WRITE_ONCE(slot[0], cpu_to_le64(addr | (r->mode ^ 1))); + dma_wmb(); + + if (r->consumer < 0) + r->consumer = r->producer; + next = (r->producer + 1) % r->slots; + r->producer = next == r->consumer ? -1 : next; + writel(BIT(r->doorbell), ave->mbox + AVE_MBOX_DOORBELL); + return 0; +} + +static int ave_ring_recv(struct ave_dev *ave, struct ave_ring *r, u64 *addr, u32 *len) +{ + __le64 *slot; + u64 word; + int next; + + if (r->consumer < 0) + return -EAGAIN; + + slot = ave->ipc.cpu + r->offset + r->consumer * AVE_RING_SLOT_BYTES; + word = le64_to_cpu(READ_ONCE(slot[0])); + if ((word & 1) != r->mode) + return -EAGAIN; + dma_rmb(); + *addr = word & ~3ULL; + *len = le64_to_cpu(READ_ONCE(slot[1])); + + if (r->producer < 0) + r->producer = r->consumer; + next = (r->consumer + 1) % r->slots; + r->consumer = next == r->producer ? -1 : next; + return 0; +} + +static int ave_ipc_buf_offset(unsigned int buf, size_t *max) +{ + if (buf == AVE_BUF_CMD) { + *max = AVE_IPC_CMD_BYTES; + return AVE_IPC_CMD_OFF; + } + if (buf < AVE_BUF_PROC(0)) { + *max = AVE_IPC_SCALAR_BYTES; + return AVE_IPC_SCALAR_OFF + (buf - AVE_BUF_SCALAR(0)) * AVE_IPC_SCALAR_BYTES; + } + *max = AVE_IPC_PROC_BYTES; + return AVE_IPC_PROC_OFF + (buf - AVE_BUF_PROC(0)) * AVE_IPC_PROC_BYTES; +} + +/* Which command buffer does a transport ack refer to? */ +static int ave_ipc_buf_of(struct ave_dev *ave, u64 addr) +{ + u64 off = addr - ave->ipc_fw_base; + size_t max; + + if (addr < ave->ipc_fw_base || off < AVE_IPC_CMD_OFF || off >= AVE_IPC_END) + return -EPROTO; + for (unsigned int buf = 0; buf < AVE_BUF_COUNT; buf++) + if (ave_ipc_buf_offset(buf, &max) == off) + return buf; + return -EPROTO; +} + +/* + * Get command buffer @buf, cleared for @len bytes, or NULL while the + * firmware still owns it. Caller holds hw_lock. + */ +void *ave_ipc_cmd(struct ave_dev *ave, unsigned int buf, size_t len) +{ + size_t max; + int off; + + if (buf >= AVE_BUF_COUNT || (ave->unacked & BIT(buf))) + return NULL; + off = ave_ipc_buf_offset(buf, &max); + if (len > max) + return NULL; + memset(ave->ipc.cpu + off, 0, len); + return ave->ipc.cpu + off; +} + +static bool ave_ipc_acked(struct ave_dev *ave, void *arg) +{ + return !(ave->unacked & BIT(*(unsigned int *)arg)); +} + +/* Wait until the firmware has taken command buffer @buf. Caller holds hw_lock. */ +int ave_ipc_wait_ack(struct ave_dev *ave, unsigned int buf) +{ + return ave_ipc_pump(ave, AVE_CMD_WAIT_MS, ave_ipc_acked, &buf); +} + +static bool ave_ipc_scalar_free(struct ave_dev *ave, void *arg) +{ + return (ave->unacked & GENMASK(AVE_BUF_PROC(0) - 1, AVE_BUF_SCALAR(0))) != + GENMASK(AVE_BUF_PROC(0) - 1, AVE_BUF_SCALAR(0)); +} + +/* Pick a free buffer for a scalar command. Caller holds hw_lock. */ +int ave_ipc_scalar_buf(struct ave_dev *ave) +{ + int ret; + + ret = ave_ipc_pump(ave, AVE_CMD_WAIT_MS, ave_ipc_scalar_free, NULL); + if (ret) + return ret; + for (unsigned int i = 0; i < AVE_IPC_SCALAR_BUFS; i++) { + unsigned int buf = AVE_BUF_SCALAR((ave->scalar_next + i) % AVE_IPC_SCALAR_BUFS); + + if (!(ave->unacked & BIT(buf))) { + ave->scalar_next = (buf - AVE_BUF_SCALAR(0) + 1) % AVE_IPC_SCALAR_BUFS; + return buf; + } + } + return -EBUSY; +} + +static void ave_ipc_mark_broken(struct ave_dev *ave, int err, const char *what) +{ + if (ave->fw_state == AVE_FW_RUNNING) + dev_err(ave->dev, "firmware stopped answering (%s: %d); it restarts after the next power cycle\n", + what, err); + ave->fw_state = AVE_FW_BROKEN; +} + +/* + * Hand the command in buffer @buf to the firmware. The timeout words are + * {duration, creation} in the host clock the firmware was synced to at boot. + * With @reply, the firmware's answer is counted as expected; Halt and + * Process are answered differently. Caller holds hw_lock. + */ +int ave_ipc_submit(struct ave_dev *ave, unsigned int buf, size_t len, u64 timeout_us, + bool reply) +{ + size_t max; + int off = ave_ipc_buf_offset(buf, &max); + u8 *cmd = ave->ipc.cpu + off; + u32 op = get_unaligned_le32(cmd + AVE_CMD_OPCODE); + int ret; + + lockdep_assert_held(&ave->hw_lock); + if (ave->fw_state != AVE_FW_RUNNING) { + dev_err_ratelimited(ave->dev, "command %u not sent: firmware state %d\n", op, + ave->fw_state); + return -EIO; + } + if (len < AVE_CMD_HDR_BYTES || len > max || !IS_ALIGNED(len, 4) || + (ave->unacked & BIT(buf)) || (reply && op >= ARRAY_SIZE(ave->want))) + return -EINVAL; + + put_unaligned_le64(timeout_us, cmd + AVE_CMD_TIMEOUT); + put_unaligned_le64(div_u64(ktime_get_ns(), NSEC_PER_USEC), cmd + AVE_CMD_CREATED); + + ret = ave_ring_send(ave, &ave->ring[0], ave->ipc_fw_base + off, len); + if (ret) { + ave_ipc_mark_broken(ave, ret, "command ring full"); + return ret; + } + ave->unacked |= BIT(buf); + if (reply) + ave->want[op]++; + return 0; +} + +static void ave_ipc_dispatch(struct ave_dev *ave) +{ + u32 id, status; + + if (ave->msg_len < AVE_REPLY_MIN_BYTES) { + dev_warn(ave->dev, "short firmware message (%u bytes)\n", ave->msg_len); + ave->reply_err = -EPROTO; + return; + } + id = get_unaligned_le16(ave->msg + AVE_REPLY_ID); + status = get_unaligned_le32(ave->msg + AVE_REPLY_STATUS); + + if (id == AVE_OP_PROCESS) { + ave_sess_reply(ave, ave->msg, ave->msg_len); + return; + } + if (id < ARRAY_SIZE(ave->want) && ave->want[id]) { + ave->want[id]--; + if (status) { + dev_err(ave->dev, "command %u refused: status %#x\n", id, status); + ave->reply_err = -EIO; + } + return; + } + dev_warn_ratelimited(ave->dev, "unexpected firmware message %u (status %#x)\n", + id, status); +} + +/* + * Poll both rings once. Returns 1 when an event (ack or message) was + * handled, 0 when nothing was pending. + */ +static int ave_ipc_poll_once(struct ave_dev *ave) +{ + u32 pending, len, off; + u64 addr; + int ret, buf; + + pending = readl(ave->mbox + AVE_MBOX_STATUS); + if (pending) + writel(pending, ave->mbox + AVE_MBOX_STATUS); + + ret = ave_ring_recv(ave, &ave->ring[0], &addr, &len); + if (!ret) { + /* Transport acks return the command address with no payload. */ + buf = ave_ipc_buf_of(ave, addr); + if (len || buf < 0 || !(ave->unacked & BIT(buf))) { + dev_err(ave->dev, "bad command ack %#llx/%u\n", addr, len); + return -EPROTO; + } + ave->unacked &= ~BIT(buf); + return 1; + } + if (ret != -EAGAIN) + return ret; + + ret = ave_ring_recv(ave, &ave->ring[1], &addr, &len); + if (ret == -EAGAIN) + return 0; + if (ret) + return ret; + if (len > sizeof(ave->msg) || ave_fw_offset(ave, addr, len, &off)) + return -EPROTO; + /* Messages must not land in the buffers the host writes commands to. */ + if (off < AVE_IPC_END && off + len > AVE_IPC_CMD_OFF) { + dev_err(ave->dev, "firmware message at %#x overlaps the command area\n", off); + return -EPROTO; + } + memcpy(ave->msg, ave->ipc.cpu + off, len); + ave->msg_len = len; + /* Return the message buffer to the firmware. */ + ret = ave_ring_send(ave, &ave->ring[1], addr, len); + if (ret) + return ret; + ave_ipc_dispatch(ave); + return 1; +} + +/* + * Handle acks and messages until @cond holds, for at most @wait_ms. With no + * @cond, handle whatever is pending and return. Running out of time means + * the firmware is stuck: it is marked broken. Caller holds hw_lock. + * + * Which interrupt line signals the rings is not known, so this polls. It + * sleeps between polls: briefly at first (commands are acknowledged within + * microseconds), then for up to a millisecond, so a long wait for a frame + * costs next to no CPU time. + */ +#define AVE_POLL_FAST_ROUNDS 20 + +int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, + bool (*cond)(struct ave_dev *ave, void *arg), void *arg) +{ + ktime_t deadline = ktime_add_ms(ktime_get(), wait_ms); + unsigned int idle = 0; + int ret; + + lockdep_assert_held(&ave->hw_lock); + for (;;) { + if (cond && cond(ave, arg)) + return 0; + if (ave->fw_state != AVE_FW_RUNNING) { + dev_err_ratelimited(ave->dev, "waiting on a firmware in state %d\n", + ave->fw_state); + return -EIO; + } + ret = ave_ipc_poll_once(ave); + if (ret < 0) { + ave_ipc_mark_broken(ave, ret, "ring error"); + return ret; + } + if (ret) { + idle = 0; + continue; + } + if (!cond) + return 0; + if (ktime_after(ktime_get(), deadline)) { + ave_ipc_mark_broken(ave, -ETIMEDOUT, "no answer"); + return -ETIMEDOUT; + } + if (idle++ < AVE_POLL_FAST_ROUNDS) + usleep_range(50, 100); + else + usleep_range(500, 1000); + } +} + +static bool ave_ipc_answered(struct ave_dev *ave, void *arg) +{ + u32 mask = *(u32 *)arg; + + if (ave->unacked & mask) + return false; + for (unsigned int i = 0; i < ARRAY_SIZE(ave->want); i++) + if (ave->want[i] && i != AVE_OP_HALT) + return false; + return true; +} + +/* + * Wait until every command except Process has been acknowledged and + * answered. Returns the first refusal seen since the last call. + * Caller holds hw_lock. + */ +int ave_ipc_settle(struct ave_dev *ave, unsigned int wait_ms) +{ + u32 mask = GENMASK(AVE_BUF_PROC(0) - 1, 0); + int ret; + + ret = ave_ipc_pump(ave, wait_ms, ave_ipc_answered, &mask); + if (!ret) + ret = ave->reply_err; + ave->reply_err = 0; + return ret; +} diff --git a/drivers/media/platform/apple/ave/ave-regs.h b/drivers/media/platform/apple/ave/ave-regs.h new file mode 100644 index 0000000000000..a85fde26278a6 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-regs.h @@ -0,0 +1,148 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Apple AVE video encoder: registers and firmware interface constants + * + * Copyright The Aurora Silicon Contributors + */ +#ifndef _APPLE_AVE_REGS_H_ +#define _APPLE_AVE_REGS_H_ + +#include + +/* Mailbox window */ +#define AVE_MBOX_DOORBELL 0x0c +#define AVE_MBOX_STATUS 0x10 +#define AVE_MBOX_STATUS_PENDING BIT(0) +#define AVE_MBOX_SCRATCH(n) (0x18 + (n) * 4) +#define AVE_MBOX_SCRATCH_COUNT 8 +#define AVE_MBOX_READY AVE_MBOX_SCRATCH(3) +#define AVE_MBOX_TIME_LO AVE_MBOX_SCRATCH(4) +#define AVE_MBOX_TIME_HI AVE_MBOX_SCRATCH(5) +#define AVE_MBOX_TIMEOUT_US 5000000 + +/* Coprocessor window */ +#define AVE_COPROC_VCBAR 0x50000 +#define AVE_COPROC_VCBAR_VALID BIT(0) +#define AVE_COPROC_TIMEBASE_FREQ 0x360020 +#define AVE_COPROC_TIMEBASE 0x378000 +#define AVE_COPROC_CTRL 0x600044 +#define AVE_COPROC_CTRL_RUN BIT(4) +#define AVE_COPROC_STATUS 0x600048 +#define AVE_COPROC_BOOT_ARG 0x600400 +#define AVE_COPROC_BOOT_ARG_VALUE 0x10000 +#define AVE_COPROC_KEEP_MAPPINGS 0x600808 + +/* Boot request and the configuration block it points at */ +#define AVE_BOOT_MAGIC 0x08042006 +#define AVE_BOOT_CFG_BYTES 0x40 +#define AVE_BOOT_CFG_DEVICE 1 /* word index */ +#define AVE_BOOT_CFG_DEVICE_T8140 29 +#define AVE_BOOT_CFG_UNITS 2 +#define AVE_BOOT_CFG_GROUPS 3 +#define AVE_BOOT_CFG_SUBID_MASK 4 +#define AVE_BOOT_CFG_LOG_LO 8 +#define AVE_BOOT_CFG_LOG_HI 9 +#define AVE_BOOT_CFG_LOG_BYTES 10 +#define AVE_BOOT_CFG_OPTIONS 12 +#define AVE_BOOT_CFG_OPTIONS_VALUE 0x77 + +/* Firmware log: 512-byte header, then a 128 KiB ring. Mapping is page sized. */ +#define AVE_LOG_BYTES 0x24000 +#define AVE_LOG_HDR_USED 0x20200 +#define AVE_LOG_HDR_BYTES 0x200 +#define AVE_LOG_RING_BYTES 0x20000 + +/* Hello reply: {version, channel area bytes, descriptor bytes, heap bytes} */ +#define AVE_HELLO_VERSION 2 +#define AVE_HELLO_DESC_BYTES 0x100 +#define AVE_HELLO_EXTRA_MAX 0x13c000 + +/* Handshake block in the IPC window */ +#define AVE_HS_BYTES 0x50 +#define AVE_HS_CHANNELS 0x08 +#define AVE_HS_SHARED 0x10 +#define AVE_HS_SHARED_BYTES 0x18 +#define AVE_HS_HEAP 0x1c +#define AVE_HS_HEAP_BYTES 0x24 +#define AVE_HS_MAGIC 0x28 +#define AVE_HS_MAGIC_VALUE 27 + +/* Channel descriptors the firmware writes into the channel area */ +#define AVE_CHAN_DESC_STRIDE 0x100 +#define AVE_CHAN_DESC_NAME_LEN 0x40 +#define AVE_CHAN_DESC_DIR 0x40 +#define AVE_CHAN_DESC_IRQ 0x44 +#define AVE_CHAN_DESC_SLOTS 0x48 +#define AVE_CHAN_DESC_ADDR 0x4c +#define AVE_RING_SLOT_BYTES 64 + +/* Common command header */ +#define AVE_CMD_OPCODE 0x00 +#define AVE_CMD_TOKEN 0x08 +#define AVE_CMD_CLIENT 0x10 +#define AVE_CMD_UNIT 0x18 +#define AVE_CMD_CODEC 0x1c +#define AVE_CMD_OPCODE_ECHO 0x20 +#define AVE_CMD_ARG0 0x24 +#define AVE_CMD_ARG1 0x28 +#define AVE_CMD_ARG2 0x2c +#define AVE_CMD_TIMEOUT 0x30 +#define AVE_CMD_CREATED 0x38 +#define AVE_CMD_HDR_BYTES 0x48 + +/* Timeouts the firmware is given, and how long the host waits */ +#define AVE_CMD_GLOBAL_TIMEOUT_US 3000000ULL +#define AVE_CMD_SESSION_TIMEOUT_US 94400000ULL +#define AVE_CMD_WAIT_MS 5000 +#define AVE_START_WAIT_MS 10000 +#define AVE_FRAME_WAIT_MS 5000 +/* Halt is never answered; give the firmware this long before power-off. */ +#define AVE_HALT_SETTLE_MS 20 + +enum ave_opcode { + AVE_OP_CONFIG = 1, + AVE_OP_HALT = 2, + AVE_OP_OPEN = 3, + AVE_OP_CLOSE = 4, + AVE_OP_START = 6, + AVE_OP_STOP = 7, + AVE_OP_PROCESS = 8, + AVE_OP_COMPLETE = 9, + AVE_OP_PRIORITY = 10, +}; + +/* Firmware-to-host messages */ +#define AVE_REPLY_ID 0x00 +#define AVE_REPLY_CLIENT 0x10 +#define AVE_REPLY_SUBTYPE 0x20 +#define AVE_REPLY_STATUS 0x40 +#define AVE_REPLY_MIN_BYTES 0x48 +#define AVE_REPLY_PROCESS_BYTES 0x50 + +#define AVE_HALT_ALL 0xffffffff + +/* Session command header values */ +#define AVE_SESSION_UNIT 1 +#define AVE_SESSION_ARG0 0xc8 +#define AVE_SESSION_ARG1 0x36 +#define AVE_SESSION_ARG2_SETUP 0x1e0 /* Open and Start */ +#define AVE_SESSION_ARG2_SETUP_SHARED 0x140 /* the same, another session open */ +#define AVE_SESSION_ARG2_H264 0x40 /* every other session command */ +#define AVE_SESSION_ARG2_HEVC 0x20 +#define AVE_TOKEN_STEP 100 +#define AVE_TOKEN_BASE 300 /* the first Open carries 400 */ +#define AVE_CLIENT_STEP 10 + +/* Process slots: the command subtype is this plus the slot index */ +#define AVE_PROC_SUBTYPE 0x15 +/* Completes sent to drain a session */ +#define AVE_DRAIN_COMPLETES 4 + +#define AVE_CONFIG_BYTES 0x78 +#define AVE_CONFIG_ALL_UNITS 0x20 +#define AVE_CONFIG_OPT_A 0x49 +#define AVE_CONFIG_OPT_B 0x4a +#define AVE_CONFIG_MCC_DSID 0x60 +#define AVE_CONFIG_MCC_DSID_T8140 0x12 + +#endif diff --git a/drivers/media/platform/apple/ave/ave-sess.c b/drivers/media/platform/apple/ave/ave-sess.c new file mode 100644 index 0000000000000..e58d485eb61c1 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-sess.c @@ -0,0 +1,799 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: encoding sessions + * + * A session owns the encoder's working buffers (sized from the picture + * size), a ring of up to five Process slots and the coded output of each + * slot. The firmware answers frames in bitstream order, so the N-th answer + * is the N-th access unit; with B-frames that is not the input order. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +#define AVE_RC_VBR 4 +#define AVE_RC_CBR 2 + +/* Filler-data NAL unit types */ +#define AVE_H264_NAL_FILLER 12 +#define AVE_HEVC_NAL_FILLER 38 + +#define AVE_H264_BASELINE 2 +#define AVE_H264_MAIN 4 +#define AVE_H264_HIGH 6 +#define AVE_HEVC_MAIN 1 + +#define AVE_META_BYTES 0xc000 +/* Metadata words: coded bytes (twice), HEVC slice-header bytes, parameter sets */ +#define AVE_META_PAYLOAD 0x120 +#define AVE_META_HEVC_SHDR 0x128 +#define AVE_META_PAYLOAD2 0x12c +#define AVE_META_PARAMS 0xb2a4 +#define AVE_META_SCAN_FROM 0x200 +#define AVE_SHDR_BYTES (256 * 0x400) +#define AVE_SHDR_ENTRY_BYTES 0x400 +#define AVE_CODED_MIN 0x3c000 +#define AVE_HDR_BYTES 0x12c000 + +/* H.264 per-frame buffer (Process +0x58c8): unused in bit-rate modes */ +#define AVE_H264_FRAME_BUF SZ_16K + +/* Level limits: max frame size, max rate, max bit rate (kbit/s, Main). */ +struct ave_level { + u32 max_fs; + u64 max_rate; + u32 max_kbps; +}; + +/* H.264 Table A-1, in V4L2 menu order (1, 1b, 1.1 ... 5.2); macroblocks. */ +static const struct ave_level ave_h264_levels[] = { + { 99, 1485, 64 }, { 99, 1485, 128 }, { 396, 3000, 192 }, { 396, 6000, 384 }, + { 396, 11880, 768 }, { 396, 11880, 2000 }, { 792, 19800, 4000 }, + { 1620, 20250, 4000 }, { 1620, 40500, 10000 }, { 3600, 108000, 14000 }, + { 5120, 216000, 20000 }, { 8192, 245760, 20000 }, { 8192, 245760, 50000 }, + { 8704, 522240, 50000 }, { 22080, 589824, 135000 }, { 36864, 983040, 240000 }, + { 36864, 2073600, 240000 }, +}; + +/* HEVC Table A-8 (Main tier), in V4L2 menu order (1, 2, 2.1 ... 5.1); samples. */ +static const struct ave_level ave_hevc_levels[] = { + { 36864, 552960, 128 }, { 122880, 3686400, 1500 }, { 245760, 7372800, 3000 }, + { 552960, 16588800, 6000 }, { 983040, 33177600, 10000 }, + { 2228224, 66846720, 12000 }, { 2228224, 133693440, 20000 }, + { 8912896, 267386880, 25000 }, { 8912896, 534773760, 40000 }, +}; + +/* + * The lowest level that fits the picture, rate and bit rate, or the + * requested one if that is higher. Returned as the firmware's 1-based enum. + */ +static int ave_pick_level(const struct ave_params *p, u32 requested) +{ + bool hevc = p->codec == AVE_CODEC_HEVC; + const struct ave_level *t = hevc ? ave_hevc_levels : ave_h264_levels; + unsigned int n = hevc ? ARRAY_SIZE(ave_hevc_levels) : ARRAY_SIZE(ave_h264_levels); + u64 size = hevc ? (u64)p->cw * p->ch : p->mbs; + u64 side = hevc ? max(p->cw, p->ch) : p->cw / 16; + u64 rate = DIV_ROUND_UP_ULL(size * p->fps_num, p->fps_den); + u32 kbps = DIV_ROUND_UP(p->bitrate, 1000); + + for (unsigned int i = 0; i < n; i++) { + u32 max_kbps = t[i].max_kbps; + + if (p->high) + max_kbps += max_kbps / 4; + if (size <= t[i].max_fs && side * side <= 8ULL * t[i].max_fs && + rate <= t[i].max_rate && kbps <= max_kbps) + return max(i, requested) + 1; + } + return -EINVAL; +} + +static int ave_sess_params(struct ave_ctx *ctx, struct ave_params *p) +{ + bool baseline = false; + int level; + + p->codec = ave_ctx_codec(ctx); + p->width = ctx->src_fmt.width; + p->height = ctx->src_fmt.height; + p->cw = ALIGN(p->width, 16); + p->ch = ALIGN(p->height, 16); + p->stride = ctx->src_fmt.plane_fmt[0].bytesperline; + p->mbs = (p->cw / 16) * (p->ch / 16); + p->fps_num = ctx->timeperframe.denominator; + p->fps_den = ctx->timeperframe.numerator; + p->fps_int = max(DIV_ROUND_CLOSEST(p->fps_num, p->fps_den), 1U); + + if (p->codec == AVE_CODEC_H264) { + switch (ctx->h264_profile) { + case V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE: + case V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE: + p->profile = AVE_H264_BASELINE; + baseline = true; + break; + case V4L2_MPEG_VIDEO_H264_PROFILE_HIGH: + p->profile = AVE_H264_HIGH; + p->high = true; + break; + default: + p->profile = AVE_H264_MAIN; + break; + } + p->min_qp = ctx->h264_min_qp; + p->max_qp = ctx->h264_max_qp; + } else { + p->profile = AVE_HEVC_MAIN; + p->min_qp = ctx->hevc_min_qp; + p->max_qp = ctx->hevc_max_qp; + } + if (p->min_qp > p->max_qp) + p->min_qp = p->max_qp; + + /* + * Reference structures as macOS sets them up: a B-frame pyramid, + * P frames only (low delay), or (H.264) intra only. + */ + p->intra_only = p->codec == AVE_CODEC_H264 && ctx->gop_size == 1; + p->bframes = !baseline && !p->intra_only && ctx->b_frames; + p->reorder_off = !ctx->b_frames; + p->refs = p->intra_only ? 1 : p->bframes ? 5 : 2; + p->slots = p->bframes ? 5 : 3; + p->gop = ctx->gop_size ?: 120; + + p->bitrate = ctx->bitrate; + if (ctx->bitrate_mode == V4L2_MPEG_VIDEO_BITRATE_MODE_CBR) { + p->rc_mode = AVE_RC_CBR; + p->limit_bytes = 0; + } else { + /* + * The data-rate limit macOS sets is twice the average rate over + * one second; a peak equal to the average starved the rate + * control on hardware. A peak of 0 means that default. + */ + p->rc_mode = AVE_RC_VBR; + p->limit_bytes = (ctx->bitrate_peak ? max(ctx->bitrate_peak, ctx->bitrate) : + 2ULL * ctx->bitrate) / 8; + } + + level = ave_pick_level(p, p->codec == AVE_CODEC_H264 ? ctx->h264_level : + ctx->hevc_level); + if (level < 0) + return level; + p->level = level; + + /* Coded-data capacity: 1.5 bytes per (aligned) pixel, 16 KiB pages. */ + p->coded_cap = ALIGN(3 * (p->codec == AVE_CODEC_HEVC ? ALIGN(p->width, 32) : + ALIGN(p->width, 16)) * + ALIGN(p->height, 16) / 2, SZ_16K); + p->coded_cap = max_t(u32, p->coded_cap, AVE_CODED_MIN); + return 0; +} + +/* + * Two groups hold more than their size fields say, measured on hardware + * at 1080p and 2408x1506: the +0x390 group is a 4:2:0 picture whose size + * covers luma only (the firmware writes up to 1.5 times it), and the + * +0x3a0 group is written up to 1.2 times its size (mapped at 1.25 times, + * as macOS spaces them). Every other buffer stays within its size; the + * mapping is only rounded up to whole pages. + */ + +/* One buffer to allocate: what the firmware is told and what is mapped. */ +struct ave_buf_req { + struct ave_buf *b; + size_t size; + size_t alloc; + unsigned long attrs; +}; + +#define AVE_MAX_BUF_REQS (1 + 4 * AVE_MAX_REFS + 4 + 2 * AVE_TILE_BUFS + 2 + \ + 3 * AVE_MAX_SLOTS) + +static int ave_buf_req_cmp(const void *a, const void *b) +{ + const struct ave_buf_req *x = a, *y = b; + + return x->alloc < y->alloc ? 1 : x->alloc > y->alloc ? -1 : 0; +} + +static void ave_req(struct ave_buf_req *r, unsigned int *n, struct ave_buf *b, size_t size, + size_t alloc, unsigned long attrs) +{ + r[*n] = (struct ave_buf_req){ + .b = b, .size = size, .alloc = max(size, alloc), .attrs = attrs, + }; + (*n)++; +} + +/* + * Working buffer sizes, all from the requested picture size. The DMA layer + * aligns each mapping to a power of two in the encoder's 256 MiB window, so + * the buffers are mapped largest first: the smaller ones then fill the + * alignment gaps the large ones leave. + */ +static int ave_sess_alloc(struct ave_dev *ave, struct ave_session *sess) +{ + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + u32 w = p->width, h = p->height; + u32 w16 = DIV_ROUND_UP(w, 16), w32 = DIV_ROUND_UP(w, 32); + u32 h32 = DIV_ROUND_UP(h, 32), w64 = DIV_ROUND_UP(w, 64), h64 = DIV_ROUND_UP(h, 64); + const unsigned long nomap = DMA_ATTR_NO_KERNEL_MAPPING; + size_t ref_a, ref_b, ref_c, ref_d, tile, mapped = 0; + struct ave_buf_req *r; + unsigned int n = 0; + int ret = 0; + + r = kcalloc(AVE_MAX_BUF_REQS, sizeof(*r), GFP_KERNEL); + if (!r) + return -ENOMEM; + + ref_a = ALIGN(w, 32) * ALIGN(h + 16, 32); + ref_b = roundup_pow_of_two(w) * roundup_pow_of_two(h) / 32; + ref_c = hevc ? 512 * w64 * h64 : ALIGN(w, 64) * (ALIGN(h, 16) / 2 + 24); + ref_d = ALIGN(hevc ? 64 * w32 * h32 : 128 * p->mbs, SZ_16K); + tile = hevc ? 2304 * w32 * h64 : ALIGN(w, 16) * ALIGN(h, 64); + + ave_req(r, &n, &sess->hdr, AVE_HDR_BYTES, 0, nomap); + for (unsigned int i = 0; i < p->refs; i++) { + ave_req(r, &n, &sess->ref_a[i], ref_a, ref_a * 3 / 2, nomap); + ave_req(r, &n, &sess->ref_b[i], ref_b, ref_b * 5 / 4, nomap); + ave_req(r, &n, &sess->ref_c[i], ref_c, 0, nomap); + ave_req(r, &n, &sess->ref_d[i], ref_d, 0, nomap); + } + ave_req(r, &n, &sess->mb, ALIGN(82 * p->mbs, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->row_a, hevc ? SZ_16K : ALIGN(256 * w16, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->row_b, ALIGN(hevc ? 768 * w32 : 1024 * w16, SZ_16K), 0, nomap); + ave_req(r, &n, &sess->fixed, SZ_16K, 0, nomap); + for (unsigned int i = 0; i < (hevc ? 4 : AVE_TILE_BUFS); i++) { + ave_req(r, &n, &sess->tile[i], tile, 0, nomap); + ave_req(r, &n, &sess->tile_aux[i], ALIGN(h, 128), 0, nomap); + } + if (hevc) + for (unsigned int i = 0; i < 2; i++) + ave_req(r, &n, &sess->hevc_extra[i], ALIGN(w * h32, SZ_16K), 0, nomap); + for (unsigned int s = 0; s < p->slots; s++) { + ave_req(r, &n, &sess->slot[s].coded, p->coded_cap, 0, 0); + ave_req(r, &n, &sess->slot[s].meta, AVE_META_BYTES, 0, 0); + /* + * HEVC: slice headers. H.264: the per-frame buffer of Process + * +0x58c8, which the firmware never wrote in any bit-rate + * mode on hardware; one page. + */ + ave_req(r, &n, &sess->slot[s].aux, hevc ? AVE_SHDR_BYTES : AVE_H264_FRAME_BUF, 0, + hevc ? 0 : nomap); + } + + sort(r, n, sizeof(*r), ave_buf_req_cmp, NULL); + for (unsigned int i = 0; i < n && !ret; i++) { + ret = ave_alloc_padded(ave, r[i].b, r[i].size, r[i].alloc, r[i].attrs); + mapped += r[i].b->alloc; + } + kfree(r); + if (ret) { + dev_err(ave->dev, "out of encoder DMA space after %zu MiB of session buffers\n", + mapped >> 20); + return -ENOMEM; + } + dev_dbg(ave->dev, "session buffers: %u, %zu KiB mapped\n", n, mapped >> 10); + return 0; +} + +static void ave_sess_free_bufs(struct ave_dev *ave, struct ave_session *sess) +{ + ave_free(ave, &sess->hdr); + for (unsigned int i = 0; i < AVE_MAX_REFS; i++) { + ave_free(ave, &sess->ref_a[i]); + ave_free(ave, &sess->ref_b[i]); + ave_free(ave, &sess->ref_c[i]); + ave_free(ave, &sess->ref_d[i]); + } + ave_free(ave, &sess->mb); + ave_free(ave, &sess->row_a); + ave_free(ave, &sess->row_b); + ave_free(ave, &sess->fixed); + for (unsigned int i = 0; i < AVE_TILE_BUFS; i++) { + ave_free(ave, &sess->tile[i]); + ave_free(ave, &sess->tile_aux[i]); + } + for (unsigned int i = 0; i < 2; i++) + ave_free(ave, &sess->hevc_extra[i]); + for (unsigned int s = 0; s < AVE_MAX_SLOTS; s++) { + ave_free(ave, &sess->slot[s].coded); + ave_free(ave, &sess->slot[s].meta); + ave_free(ave, &sess->slot[s].aux); + ave_free(ave, &sess->slot[s].bounce); + } +} + +static void ave_note_copy(struct ave_ctx *ctx, u32 stride, const char *how) +{ + if (ctx->sess->copy_noted) + return; + ctx->sess->copy_noted = true; + dev_dbg(ctx->ave->dev, "input has a %u byte stride; moving lines to %u %s\n", stride, + ctx->sess->p.stride, how); +} + +/* + * The input picture for @slot, as a DART address. The encoder reads NV12 + * with the 64-byte aligned stride S_FMT reported and chroma right after + * height luma lines. Some clients (ffmpeg's v4l2m2m encoder) fill the + * buffer with their own, smaller line size instead, which shows in the + * payload size; such frames are copied to a per-slot buffer in the + * encoder's layout. + */ +int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t *input) +{ + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + struct vb2_buffer *vb = &src->vb2_buf; + u32 off = vb->planes[0].data_offset; + u32 used = vb2_get_plane_payload(vb, 0) - min(off, vb2_get_plane_payload(vb, 0)); + u32 crows = DIV_ROUND_UP(p->height, 2), rows = p->height + crows; + u32 width = ALIGN(p->width, 2), stride; + struct ave_buf *b = &sess->slot[slot].bounce; + const u8 *from; + u8 *to; + int ret; + + *input = vb2_dma_contig_plane_dma_addr(vb, 0) + off; + if (!used || used >= p->stride * rows || used % rows || used / rows < width) { + if (IS_ALIGNED(*input, 64)) + return 0; + dev_err(ctx->ave->dev, "OUTPUT plane at %pad is not 64-byte aligned\n", input); + return -EINVAL; + } + stride = used / rows; + + from = vb2_plane_vaddr(vb, 0); + if (!from) { + dev_err(ctx->ave->dev, "cannot map an OUTPUT buffer to copy its %u byte stride\n", + stride); + return -EFAULT; + } + from += off; + + /* + * Buffers the driver allocated (MMAP) are large enough for the + * encoder's layout: spread the lines out in place, last line first, + * so no extra DMA space is needed. Imported buffers may be shared + * with other users and get a copy instead. + */ + if (vb->memory == VB2_MEMORY_MMAP && + vb2_plane_size(vb, 0) >= off + (size_t)p->stride * rows) { + to = (u8 *)from; + for (u32 y = rows; y-- > 0;) + memmove(to + y * p->stride, from + y * stride, width); + ave_note_copy(ctx, stride, "in place"); + return 0; + } + + ret = ave_alloc(ctx->ave, b, p->stride * rows, 0); + if (ret) { + dev_err(ctx->ave->dev, "no room to copy %u byte stride input\n", stride); + return ret; + } + to = b->cpu; + for (u32 y = 0; y < rows; y++) + memcpy(to + y * p->stride, from + y * stride, width); + ave_note_copy(ctx, stride, "to a copy"); + *input = b->dma; + return 0; +} + +/* Debug map of the session buffers, to match DART faults to a buffer. */ +static void ave_log_buf(struct ave_dev *ave, const char *name, int i, const struct ave_buf *b) +{ + if (b->cpu) + dev_dbg(ave->dev, "buf %s%d %pad +%#zx (mapped %#zx)\n", name, i, &b->dma, + b->size, b->alloc); +} + +static void ave_sess_log_map(struct ave_dev *ave, struct ave_session *sess) +{ + ave_log_buf(ave, "hdr", 0, &sess->hdr); + ave_log_buf(ave, "mb", 0, &sess->mb); + ave_log_buf(ave, "row_a", 0, &sess->row_a); + ave_log_buf(ave, "row_b", 0, &sess->row_b); + ave_log_buf(ave, "fixed", 0, &sess->fixed); + for (int i = 0; i < 2; i++) + ave_log_buf(ave, "hevc_extra", i, &sess->hevc_extra[i]); + for (int i = 0; i < AVE_MAX_REFS; i++) { + ave_log_buf(ave, "ref_a", i, &sess->ref_a[i]); + ave_log_buf(ave, "ref_b", i, &sess->ref_b[i]); + ave_log_buf(ave, "ref_c", i, &sess->ref_c[i]); + ave_log_buf(ave, "ref_d", i, &sess->ref_d[i]); + } + for (int i = 0; i < AVE_TILE_BUFS; i++) { + ave_log_buf(ave, "tile", i, &sess->tile[i]); + ave_log_buf(ave, "tile_aux", i, &sess->tile_aux[i]); + } + for (int i = 0; i < AVE_MAX_SLOTS; i++) { + ave_log_buf(ave, "coded", i, &sess->slot[i].coded); + ave_log_buf(ave, "meta", i, &sess->slot[i].meta); + ave_log_buf(ave, "aux", i, &sess->slot[i].aux); + } +} + +/* Work out the parameters and allocate the buffers; no firmware access. */ +struct ave_session *ave_sess_create(struct ave_ctx *ctx) +{ + struct ave_session *sess; + int ret; + + sess = kzalloc_obj(*sess); + if (!sess) + return ERR_PTR(-ENOMEM); + ret = ave_sess_params(ctx, &sess->p); + if (!ret) + ret = ave_sess_alloc(ctx->ave, sess); + if (ret) { + ave_sess_free_bufs(ctx->ave, sess); + kfree(sess); + return ERR_PTR(ret); + } + for (unsigned int s = 0; s < sess->p.slots; s++) + ave_sess_prepare_slot(sess, s); + ave_sess_log_map(ctx->ave, sess); + dev_dbg(ctx->ave->dev, "session %ux%u codec %u profile %u level %u, %u slots, %u refs\n", + sess->p.width, sess->p.height, sess->p.codec, sess->p.profile, sess->p.level, + sess->p.slots, sess->p.refs); + dev_dbg(ctx->ave->dev, "rate control mode %u, %u bit/s, limit %u bytes/s, gop %u, %u/%u fps\n", + sess->p.rc_mode, sess->p.bitrate, sess->p.limit_bytes, sess->p.gop, + sess->p.fps_num, sess->p.fps_den); + return sess; +} + +/* + * Free the buffers. A firmware that stopped answering in the middle of a + * session may still write to them, and a freed DART range is soon handed + * out again, so those stay allocated (until reboot, like the firmware + * itself). Buffers the firmware never saw (no Start sent) always go. + */ +void ave_sess_destroy(struct ave_ctx *ctx) +{ + struct ave_session *sess = ctx->sess; + + if (!sess) + return; + if (sess->handed && ctx->ave->fw_state == AVE_FW_BROKEN) + dev_warn(ctx->ave->dev, "keeping the buffers of a failed session\n"); + else + ave_sess_free_bufs(ctx->ave, sess); + kfree(sess); + ctx->sess = NULL; +} + +int ave_sess_free_slot(struct ave_session *sess) +{ + for (unsigned int s = 0; s < sess->p.slots; s++) + if (!sess->slot[s].busy && !sess->slot[s].done) + return s; + return -EBUSY; +} + +/* + * Clear what the firmware reports through, so a stale value is never taken + * for a new one: the whole metadata buffer and the first slice header. + */ +void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot) +{ + struct ave_slot *s = &sess->slot[slot]; + + memset(s->meta.cpu, 0, s->meta.size); + if (sess->p.codec == AVE_CODEC_HEVC) + memset(s->aux.cpu, 0, AVE_SHDR_ENTRY_BYTES); +} + +/* + * A frame reply: +0x10 the session's client token, +0x20 the slot's + * subtype, +0x40 status. Caller holds hw_lock. + */ +void ave_sess_reply(struct ave_dev *ave, const u8 *msg, u32 len) +{ + u64 client = get_unaligned_le64(msg + AVE_REPLY_CLIENT); + u32 subtype = get_unaligned_le32(msg + AVE_REPLY_SUBTYPE); + u32 slot = subtype - AVE_PROC_SUBTYPE; + struct ave_session *sess = NULL; + struct ave_ctx *ctx = NULL; + struct ave_slot *s; + + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] && ave->sessions[i]->sess->client == client) + ctx = ave->sessions[i]; + if (ctx) + sess = ctx->sess; + if (!sess || len < AVE_REPLY_PROCESS_BYTES || slot >= sess->p.slots || + !sess->slot[slot].busy) { + dev_warn(ave->dev, "stray frame reply (client %llu subtype %#x len %u)\n", + client, subtype, len); + return; + } + s = &sess->slot[slot]; + sess->progress = jiffies; + s->status = get_unaligned_le32(msg + AVE_REPLY_STATUS); + if (s->status) + dev_dbg(ctx->ave->dev, "frame in slot %u: status %#x\n", slot, s->status); + s->busy = false; + s->done = true; + sess->busy--; + sess->done_fifo[(sess->done_head + sess->done_count) % AVE_MAX_SLOTS] = slot; + sess->done_count++; + atomic_inc(&ctx->ready); +} + +/* + * Copy Annex B data from @src to @dst until the data ends: NAL units never + * hold three zero bytes in a row, and the buffer was cleared before the + * firmware wrote it, so a run of @zero_run zero bytes marks the end. + * Returns the length or -ENOSPC. + */ +static ssize_t ave_copy_annexb(u8 *dst, size_t room, const u8 *src, size_t cap, + unsigned int zero_run) +{ + size_t pos = 0, end = 0; + unsigned int zeros = 0; + + while (pos < cap) { + size_t chunk = min3((size_t)SZ_16K, cap - pos, room - pos); + + if (!chunk) + return -ENOSPC; + memcpy(dst + pos, src + pos, chunk); + for (size_t i = pos; i < pos + chunk; i++) { + if (dst[i]) { + end = i + 1; + zeros = 0; + } else if (end && ++zeros >= zero_run) { + return end; + } + } + pos += chunk; + } + return end; +} + +static bool ave_starts_annexb(const u8 *p, size_t len) +{ + return (len >= 3 && !p[0] && !p[1] && p[2] == 1) || + (len >= 4 && !p[0] && !p[1] && !p[2] && p[3] == 1); +} + +/* NAL unit types in an Annex B buffer: key frame, parameter sets, filler? */ +static void ave_scan_nals(const u8 *p, size_t len, bool hevc, bool *key, bool *params, + bool *filler) +{ + for (size_t i = 0; i + 3 < len; i++) { + u32 type; + + if (p[i] || p[i + 1] || p[i + 2] != 1) + continue; + type = hevc ? (p[i + 3] >> 1) & 0x3f : p[i + 3] & 0x1f; + if (hevc) { + *key |= type >= 16 && type <= 21; + *params |= type >= 32 && type <= 34; + *filler |= type == AVE_HEVC_NAL_FILLER; + } else { + *key |= type == 5; + *params |= type == 7; + *filler |= type == AVE_H264_NAL_FILLER; + } + i += 2; + } +} + +/* Where in the metadata buffer do the parameter sets start, if anywhere? */ +static int ave_find_params(const u8 *meta, bool hevc) +{ + if (ave_starts_annexb(meta + AVE_META_PARAMS, AVE_META_BYTES - AVE_META_PARAMS)) + return AVE_META_PARAMS; + /* Not at the HEVC offset: look for an SPS (H.264) or VPS (HEVC). */ + for (int i = AVE_META_SCAN_FROM; i + 4 < AVE_META_BYTES; i++) { + u8 type; + + if (meta[i] || meta[i + 1] || meta[i + 2] != 1) + continue; + type = hevc ? (meta[i + 3] >> 1) & 0x3f : meta[i + 3] & 0x1f; + if (type == (hevc ? 32 : 7)) + return i > AVE_META_SCAN_FROM && !meta[i - 1] ? i - 1 : i; + } + return -ENOENT; +} + +static void ave_sess_dump(struct ave_ctx *ctx, struct ave_slot *s, const char *why) +{ + struct ave_session *sess = ctx->sess; + const u8 *meta = s->meta.cpu; + int ps; + + if (sess->warned) + return; + sess->warned = true; + dev_warn(ctx->ave->dev, "frame %u output: %s (enable dynamic debug for a dump)\n", + ctx->dst_sequence, why); + print_hex_dump_debug("ave coded: ", DUMP_PREFIX_OFFSET, 16, 1, s->coded.cpu, 128, false); + print_hex_dump_debug("ave meta: ", DUMP_PREFIX_OFFSET, 16, 4, meta, 0x200, false); + print_hex_dump_debug("ave meta b200: ", DUMP_PREFIX_OFFSET, 16, 1, meta + 0xb200, + 0x200, false); + ps = ave_find_params(meta, sess->p.codec == AVE_CODEC_HEVC); + if (ps >= 0) { + dev_dbg(ctx->ave->dev, "parameter sets at metadata +%#x\n", ps); + print_hex_dump_debug("ave meta params: ", DUMP_PREFIX_OFFSET, 16, 1, meta + ps, + min(128, AVE_META_BYTES - ps), false); + } + /* Anything else the firmware wrote past the header words. */ + for (int i = AVE_META_SCAN_FROM; i < AVE_META_BYTES; i += 16) + if (memchr_inv(meta + i, 0, 16)) + dev_dbg(ctx->ave->dev, "metadata +%#x: %16ph\n", i, meta + i); +} + +/* + * CBR padding. macOS streams in CBR mode follow a buffer model: each frame + * has a budget of bitrate / (8 x fps) bytes, and once the bytes left unused + * add up to more than 0.5 s of bit rate (the 0.5 s VBV buffer of Start + * +0x290), every frame is padded with a filler-data NAL unit so the surplus + * stays there. Whether the firmware or macOS user space writes that filler + * is not known; if the firmware's output never carries filler, the driver + * adds it here. + */ +#define AVE_FILLER_OVERHEAD 7 /* start code, NAL header, trailing bits */ + +static size_t ave_sess_cbr_fill(struct ave_ctx *ctx, u8 *out, size_t len, size_t room, + bool fw_filler) +{ + struct ave_session *sess = ctx->sess; + const struct ave_params *p = &sess->p; + bool hevc = p->codec == AVE_CODEC_HEVC; + s64 budget = div_u64((u64)p->bitrate * p->fps_den, 8ULL * p->fps_num); + s64 vbv = p->bitrate / 16; /* 0.5 s of bit rate, in bytes */ + s64 excess, pad; + u8 *f; + + if (p->rc_mode != AVE_RC_CBR) + return len; + if (fw_filler && !sess->fw_fills) { + sess->fw_fills = true; + dev_dbg(ctx->ave->dev, "CBR: the firmware writes filler data\n"); + } + excess = sess->cbr_surplus + budget - (s64)len - vbv; + if (!sess->fw_fills && excess > AVE_FILLER_OVERHEAD) { + pad = min_t(s64, excess, room - len); + if (pad > AVE_FILLER_OVERHEAD) { + f = out + len; + f[0] = 0; + f[1] = 0; + f[2] = 0; + f[3] = 1; + if (hevc) { + /* nal_unit_type 38, layer 0, temporal id 0 + 1 */ + f[4] = AVE_HEVC_NAL_FILLER << 1; + f[5] = 1; + memset(f + 6, 0xff, pad - AVE_FILLER_OVERHEAD); + } else { + f[4] = AVE_H264_NAL_FILLER; + memset(f + 5, 0xff, pad - AVE_FILLER_OVERHEAD + 1); + } + f[pad - 1] = 0x80; /* rbsp_stop_one_bit */ + len += pad; + if (!sess->host_fill_noted) { + sess->host_fill_noted = true; + dev_dbg(ctx->ave->dev, + "CBR: no filler from the firmware, the driver adds it\n"); + } + } + } + sess->cbr_surplus += budget - (s64)len; + return len; +} + +/* + * Assemble the access unit of @slot into @dst: the parameter sets the + * firmware leaves in the metadata buffer (when it emits them), for HEVC the + * slice header from the first slice-header entry, then the coded data. + * Metadata +0x120 (repeated at +0x12c) holds the coded length for both + * codecs, +0x128 the HEVC slice-header length. + */ +int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, + bool *keyframe) +{ + struct ave_session *sess = ctx->sess; + struct ave_slot *s = &sess->slot[slot]; + bool hevc = sess->p.codec == AVE_CODEC_HEVC; + u8 *out = vb2_plane_vaddr(&dst->vb2_buf, 0); + size_t room = vb2_plane_size(&dst->vb2_buf, 0); + const u8 *meta = s->meta.cpu; + u32 data, data2, shdr = 0; + bool params = false, filler = false; + ssize_t ps = 0; + size_t len; + int at; + + *keyframe = false; + if (!out) + return -EFAULT; + if (s->status) { + dev_warn_ratelimited(ctx->ave->dev, "frame %u: firmware status %#x\n", + ctx->dst_sequence, s->status); + return -EIO; + } + + data = get_unaligned_le32(meta + AVE_META_PAYLOAD); + data2 = get_unaligned_le32(meta + AVE_META_PAYLOAD2); + if (hevc) + shdr = get_unaligned_le32(meta + AVE_META_HEVC_SHDR); + if (data != data2) + dev_dbg(ctx->ave->dev, "coded length words differ: %u/%u\n", data, data2); + + at = ave_find_params(meta, hevc); + if (at >= 0) { + ps = ave_copy_annexb(out, room, meta + at, AVE_META_BYTES - at, 16); + if (ps < 0) + return ps; + } + if (!data || data > sess->p.coded_cap || shdr > AVE_SHDR_ENTRY_BYTES || + ps + shdr + data > room) { + dev_err(ctx->ave->dev, "bad output sizes %u/%u (capacity %u, room %zu)\n", + shdr, data, sess->p.coded_cap, room); + ave_sess_dump(ctx, s, "bad sizes"); + return -EPROTO; + } + if (hevc) + memcpy(out + ps, s->aux.cpu, shdr); + memcpy(out + ps + shdr, s->coded.cpu, data); + len = ps + shdr + data; + + if (!ave_starts_annexb(out, len)) + ave_sess_dump(ctx, s, "not Annex B"); + ave_scan_nals(out, len, hevc, keyframe, ¶ms, &filler); + if (!ctx->dst_sequence && !params) + ave_sess_dump(ctx, s, "first frame has no parameter sets"); + len = ave_sess_cbr_fill(ctx, out, len, room, filler); + vb2_set_plane_payload(&dst->vb2_buf, 0, len); + return 0; +} + +/* + * Frames are outstanding, nothing came back for a while, and the firmware + * has enough frames that it should have answered: without B-frames every + * frame is answered on its own; with B-frames, once four are inside (one + * reference plus three B) one of them must come out. + */ +bool ave_sess_stalled(struct ave_session *sess) +{ + unsigned int need = sess->p.bframes ? 4 : 1; + + return sess->busy && (sess->busy >= need || sess->nudged) && + time_after(jiffies, sess->progress + msecs_to_jiffies(AVE_FRAME_WAIT_MS)); +} + +/* + * With B-frames the firmware keeps frames until later ones arrive. A client + * that waits for each coded frame before sending the next (ffmpeg's + * v4l2m2m encoder does) would wait forever. When frames are held and no + * input came for a while, a Complete makes the firmware code what it holds, + * as macOS does after the first frame. + */ +#define AVE_STARVE_MS 50 + +bool ave_sess_starved(struct ave_session *sess) +{ + return sess->busy && !sess->nudged && sess->handed && + time_after(jiffies, sess->last_sent + msecs_to_jiffies(AVE_STARVE_MS)); +} diff --git a/drivers/media/platform/apple/ave/ave-v4l2.c b/drivers/media/platform/apple/ave/ave-v4l2.c new file mode 100644 index 0000000000000..0adbaaf8513d5 --- /dev/null +++ b/drivers/media/platform/apple/ave/ave-v4l2.c @@ -0,0 +1,1338 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple AVE video encoder: V4L2 stateful encoder interface + * + * Follows Documentation/userspace-api/media/v4l/dev-encoder.rst: the CAPTURE + * queue carries the coded stream (H.264 or HEVC), the OUTPUT queue raw NV12. + * + * Copyright The Aurora Silicon Contributors + */ +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "ave.h" +#include "ave-regs.h" + +#define AVE_NAME "apple-ave" + +/* + * Frame size limits and input stride alignment: the range the encoder was + * seen working at (320x240 up to the 2408x1506 panel), with a little room. + * Larger sizes need buffer sizes nobody has checked yet. + */ +#define AVE_MIN_W 320 +#define AVE_MIN_H 240 +#define AVE_MAX_W 2560 +#define AVE_MAX_H 1600 +#define AVE_STEP 16 +#define AVE_STRIDE_ALIGN 64 +#define AVE_DEFAULT_W 1920 +#define AVE_DEFAULT_H 1080 +/* + * Buffer counts are limited so a full set fits the encoder's 256 MiB DART + * window next to a session: each queue may use this much, counting every + * buffer at the power-of-two size the DMA layer maps it with. + */ +#define AVE_OUTPUT_BUDGET (80 * SZ_1M) +#define AVE_CAPTURE_BUDGET (40 * SZ_1M) +#define AVE_MAX_QUEUE_BUFS 16 +#define AVE_RESTART_TRIES 10 +#define AVE_RESTART_WAIT_MS 100 + +static const u32 ave_coded_fmts[] = { + V4L2_PIX_FMT_H264, + V4L2_PIX_FMT_HEVC, +}; + +static const u32 ave_raw_fmts[] = { + V4L2_PIX_FMT_NV12, +}; + +static const struct v4l2_event ave_eos_event = { + .type = V4L2_EVENT_EOS, +}; + +enum ave_codec ave_ctx_codec(struct ave_ctx *ctx) +{ + return ctx->dst_fmt.pixelformat == V4L2_PIX_FMT_HEVC ? AVE_CODEC_HEVC : + AVE_CODEC_H264; +} + +static bool ave_fmt_listed(const u32 *list, unsigned int n, u32 fourcc) +{ + for (unsigned int i = 0; i < n; i++) + if (list[i] == fourcc) + return true; + return false; +} + +static void ave_clamp_size(u32 *w, u32 *h) +{ + *w = clamp_t(u32, ALIGN(*w, 2), AVE_MIN_W, AVE_MAX_W); + *h = clamp_t(u32, ALIGN(*h, 2), AVE_MIN_H, AVE_MAX_H); +} + +/* Raw NV12, one buffer: Y plane, then interleaved CbCr at half height. */ +static void ave_fill_raw(struct v4l2_pix_format_mplane *f) +{ + u32 stride, height; + + f->pixelformat = V4L2_PIX_FMT_NV12; + ave_clamp_size(&f->width, &f->height); + /* Keep a client's stride, within reason, rounded to 64 bytes. */ + stride = ALIGN(clamp_t(u32, f->plane_fmt[0].bytesperline, f->width, 2 * AVE_MAX_W), + AVE_STRIDE_ALIGN); + height = ALIGN(f->height, AVE_STEP); + f->num_planes = 1; + f->plane_fmt[0].bytesperline = stride; + f->plane_fmt[0].sizeimage = stride * height * 3 / 2; + f->field = V4L2_FIELD_NONE; + memset(f->plane_fmt[0].reserved, 0, sizeof(f->plane_fmt[0].reserved)); + memset(f->reserved, 0, sizeof(f->reserved)); +} + +static void ave_fill_coded(struct v4l2_pix_format_mplane *f) +{ + u32 min; + + if (!ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), f->pixelformat)) + f->pixelformat = ave_coded_fmts[0]; + ave_clamp_size(&f->width, &f->height); + /* The firmware's coded-data capacity plus parameter sets and headers. */ + min = ALIGN(3 * ALIGN(f->width, 32) * ALIGN(f->height, 16) / 2, SZ_16K) + SZ_64K; + f->num_planes = 1; + f->plane_fmt[0].bytesperline = 0; + f->plane_fmt[0].sizeimage = clamp_t(u32, f->plane_fmt[0].sizeimage, min, SZ_32M); + f->field = V4L2_FIELD_NONE; + memset(f->plane_fmt[0].reserved, 0, sizeof(f->plane_fmt[0].reserved)); + memset(f->reserved, 0, sizeof(f->reserved)); +} + +static void ave_reset_fmts(struct ave_ctx *ctx) +{ + struct v4l2_pix_format_mplane *src = &ctx->src_fmt, *dst = &ctx->dst_fmt; + + memset(dst, 0, sizeof(*dst)); + dst->pixelformat = V4L2_PIX_FMT_H264; + dst->width = AVE_DEFAULT_W; + dst->height = AVE_DEFAULT_H; + ave_fill_coded(dst); + + memset(src, 0, sizeof(*src)); + src->width = dst->width; + src->height = dst->height; + ave_fill_raw(src); + src->colorspace = V4L2_COLORSPACE_REC709; + src->xfer_func = V4L2_XFER_FUNC_DEFAULT; + src->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + src->quantization = V4L2_QUANTIZATION_LIM_RANGE; + dst->colorspace = src->colorspace; + dst->xfer_func = src->xfer_func; + dst->ycbcr_enc = src->ycbcr_enc; + dst->quantization = src->quantization; + + ctx->timeperframe = (struct v4l2_fract){ 1, 30 }; +} + +/* ioctls */ + +static int ave_querycap(struct file *file, void *priv, struct v4l2_capability *cap) +{ + strscpy(cap->driver, AVE_NAME, sizeof(cap->driver)); + strscpy(cap->card, "Apple AVE video encoder", sizeof(cap->card)); + return 0; +} + +static int ave_enum_fmt_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f) +{ + if (f->index >= ARRAY_SIZE(ave_coded_fmts)) + return -EINVAL; + f->pixelformat = ave_coded_fmts[f->index]; + f->flags = V4L2_FMT_FLAG_COMPRESSED; + return 0; +} + +static int ave_enum_fmt_out(struct file *file, void *priv, struct v4l2_fmtdesc *f) +{ + if (f->index >= ARRAY_SIZE(ave_raw_fmts)) + return -EINVAL; + f->pixelformat = ave_raw_fmts[f->index]; + return 0; +} + +static int ave_enum_framesizes(struct file *file, void *priv, struct v4l2_frmsizeenum *fs) +{ + if (fs->index || + (!ave_fmt_listed(ave_coded_fmts, ARRAY_SIZE(ave_coded_fmts), fs->pixel_format) && + !ave_fmt_listed(ave_raw_fmts, ARRAY_SIZE(ave_raw_fmts), fs->pixel_format))) + return -EINVAL; + fs->type = V4L2_FRMSIZE_TYPE_STEPWISE; + fs->stepwise = (struct v4l2_frmsize_stepwise){ + AVE_MIN_W, AVE_MAX_W, 2, AVE_MIN_H, AVE_MAX_H, 2, + }; + return 0; +} + +static int ave_g_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + f->fmt.pix_mp = ave_file_to_ctx(file)->dst_fmt; + return 0; +} + +static int ave_g_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + f->fmt.pix_mp = ave_file_to_ctx(file)->src_fmt; + return 0; +} + +static int ave_try_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + ave_fill_coded(&f->fmt.pix_mp); + return 0; +} + +static int ave_try_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + + ave_fill_raw(pix); + if (pix->colorspace == V4L2_COLORSPACE_DEFAULT) { + pix->colorspace = ctx->src_fmt.colorspace; + pix->xfer_func = ctx->src_fmt.xfer_func; + pix->ycbcr_enc = ctx->src_fmt.ycbcr_enc; + pix->quantization = ctx->src_fmt.quantization; + } + return 0; +} + +static int ave_s_fmt_cap(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + + if (vb2_is_busy(vq)) + return -EBUSY; + /* The coded format comes first; the raw side follows its size. */ + ave_fill_coded(pix); + pix->colorspace = ctx->src_fmt.colorspace; + pix->xfer_func = ctx->src_fmt.xfer_func; + pix->ycbcr_enc = ctx->src_fmt.ycbcr_enc; + pix->quantization = ctx->src_fmt.quantization; + ctx->dst_fmt = *pix; + return 0; +} + +static int ave_s_fmt_out(struct file *file, void *priv, struct v4l2_format *f) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; + int ret; + + if (vb2_is_busy(vq)) + return -EBUSY; + ret = ave_try_fmt_out(file, priv, f); + if (ret) + return ret; + ctx->src_fmt = *pix; + + /* The coded size follows the raw size (crop is the whole frame). */ + ctx->dst_fmt.width = pix->width; + ctx->dst_fmt.height = pix->height; + ctx->dst_fmt.colorspace = pix->colorspace; + ctx->dst_fmt.xfer_func = pix->xfer_func; + ctx->dst_fmt.ycbcr_enc = pix->ycbcr_enc; + ctx->dst_fmt.quantization = pix->quantization; + ave_fill_coded(&ctx->dst_fmt); + return 0; +} + +static int ave_g_selection(struct file *file, void *priv, struct v4l2_selection *s) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT) + return -EINVAL; + switch (s->target) { + case V4L2_SEL_TGT_CROP: + case V4L2_SEL_TGT_CROP_DEFAULT: + case V4L2_SEL_TGT_CROP_BOUNDS: + s->r = (struct v4l2_rect){ 0, 0, ctx->src_fmt.width, ctx->src_fmt.height }; + return 0; + default: + return -EINVAL; + } +} + +static int ave_s_selection(struct file *file, void *priv, struct v4l2_selection *s) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT || s->target != V4L2_SEL_TGT_CROP) + return -EINVAL; + /* Only full-frame encoding for now. */ + s->r = (struct v4l2_rect){ 0, 0, ctx->src_fmt.width, ctx->src_fmt.height }; + return 0; +} + +static int ave_g_parm(struct file *file, void *priv, struct v4l2_streamparm *a) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + + if (!V4L2_TYPE_IS_OUTPUT(a->type)) + return -EINVAL; + a->parm.output.capability = V4L2_CAP_TIMEPERFRAME; + a->parm.output.timeperframe = ctx->timeperframe; + return 0; +} + +static int ave_s_parm(struct file *file, void *priv, struct v4l2_streamparm *a) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_fract *t = &a->parm.output.timeperframe; + + if (!V4L2_TYPE_IS_OUTPUT(a->type)) + return -EINVAL; + if (!t->numerator || !t->denominator) + *t = ctx->timeperframe; + /* 1 to 240 frames per second. */ + if ((u64)t->denominator > (u64)t->numerator * 240) + *t = (struct v4l2_fract){ 1, 240 }; + if (t->numerator > t->denominator) + *t = (struct v4l2_fract){ 1, 1 }; + ctx->timeperframe = *t; + a->parm.output.capability = V4L2_CAP_TIMEPERFRAME; + return 0; +} + +/* + * Drain: frames still inside the firmware (B-frames) only come out after a + * flush, so the driver tracks the drain point itself instead of using the + * m2m helpers, which would mark the next CAPTURE buffer last right away. + */ +static int ave_encoder_cmd(struct file *file, void *priv, struct v4l2_encoder_cmd *ec) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + int ret; + + ret = v4l2_m2m_ioctl_try_encoder_cmd(file, priv, ec); + if (ret) + return ret; + if (!vb2_is_streaming(v4l2_m2m_get_src_vq(m2m)) || + !vb2_is_streaming(v4l2_m2m_get_dst_vq(m2m))) + return 0; + + if (ec->cmd == V4L2_ENC_CMD_STOP) { + if (ctx->draining || ctx->stopped) + return 0; + ctx->drain_src = v4l2_m2m_last_src_buf(m2m); + ctx->draining = true; + if (!ctx->drain_src) + WRITE_ONCE(ctx->flush_pending, true); + } else { + if (ctx->draining && !ctx->stopped) + return -EBUSY; + ctx->draining = false; + ctx->stopped = false; + vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(m2m)); + } + v4l2_m2m_try_schedule(m2m); + return 0; +} + +static int ave_subscribe_event(struct v4l2_fh *fh, const struct v4l2_event_subscription *sub) +{ + switch (sub->type) { + case V4L2_EVENT_EOS: + return v4l2_event_subscribe(fh, sub, 0, NULL); + default: + return v4l2_ctrl_subscribe_event(fh, sub); + } +} + +static const struct v4l2_ioctl_ops ave_ioctl_ops = { + .vidioc_querycap = ave_querycap, + .vidioc_enum_fmt_vid_cap = ave_enum_fmt_cap, + .vidioc_enum_fmt_vid_out = ave_enum_fmt_out, + .vidioc_enum_framesizes = ave_enum_framesizes, + .vidioc_g_fmt_vid_cap_mplane = ave_g_fmt_cap, + .vidioc_g_fmt_vid_out_mplane = ave_g_fmt_out, + .vidioc_try_fmt_vid_cap_mplane = ave_try_fmt_cap, + .vidioc_try_fmt_vid_out_mplane = ave_try_fmt_out, + .vidioc_s_fmt_vid_cap_mplane = ave_s_fmt_cap, + .vidioc_s_fmt_vid_out_mplane = ave_s_fmt_out, + .vidioc_g_selection = ave_g_selection, + .vidioc_s_selection = ave_s_selection, + .vidioc_g_parm = ave_g_parm, + .vidioc_s_parm = ave_s_parm, + + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, + .vidioc_expbuf = v4l2_m2m_ioctl_expbuf, + .vidioc_streamon = v4l2_m2m_ioctl_streamon, + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, + + .vidioc_try_encoder_cmd = v4l2_m2m_ioctl_try_encoder_cmd, + .vidioc_encoder_cmd = ave_encoder_cmd, + + .vidioc_subscribe_event = ave_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +/* Controls */ + +static int ave_s_ctrl(struct v4l2_ctrl *ctrl) +{ + struct ave_ctx *ctx = container_of(ctrl->handler, struct ave_ctx, hdl); + + switch (ctrl->id) { + case V4L2_CID_MPEG_VIDEO_BITRATE_MODE: + ctx->bitrate_mode = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_BITRATE: + WRITE_ONCE(ctx->bitrate, ctrl->val); + break; + case V4L2_CID_MPEG_VIDEO_BITRATE_PEAK: + ctx->bitrate_peak = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_GOP_SIZE: + ctx->gop_size = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME: + WRITE_ONCE(ctx->force_key_frame, true); + break; + case V4L2_CID_MPEG_VIDEO_B_FRAMES: + ctx->b_frames = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_PROFILE: + ctx->h264_profile = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_LEVEL: + ctx->h264_level = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_MIN_QP: + ctx->h264_min_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_H264_MAX_QP: + ctx->h264_max_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_LEVEL: + ctx->hevc_level = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_MIN_QP: + ctx->hevc_min_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP: + ctx->hevc_max_qp = ctrl->val; + break; + case V4L2_CID_MPEG_VIDEO_HEADER_MODE: + case V4L2_CID_MPEG_VIDEO_HEVC_PROFILE: + case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT: + break; + default: + return -EINVAL; + } + return 0; +} + +static const struct v4l2_ctrl_ops ave_ctrl_ops = { + .s_ctrl = ave_s_ctrl, +}; + +static int ave_init_ctrls(struct ave_ctx *ctx) +{ + struct v4l2_ctrl_handler *hdl = &ctx->hdl; + const struct v4l2_ctrl_ops *ops = &ave_ctrl_ops; + struct v4l2_ctrl *c; + + v4l2_ctrl_handler_init(hdl, 17); + + /* Constant quality needs a quality-to-QP mapping that is not known yet. */ + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_MODE, + V4L2_MPEG_VIDEO_BITRATE_MODE_CBR, + BIT(V4L2_MPEG_VIDEO_BITRATE_MODE_CQ), + V4L2_MPEG_VIDEO_BITRATE_MODE_VBR); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE, 100000, 100000000, 1, + 4000000); + /* + * In VBR mode the peak caps the data rate over any one second; + * 0 means twice the average, as macOS sets it. + */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_BITRATE_PEAK, 0, 200000000, 1, 0); + /* 0 means no limit; the firmware then uses 120 frames. */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, 65535, 1, 30); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); + /* + * The firmware either uses up to three B-frames (reordering them + * itself) or none. Baseline and intra-only H.264 never have any. + */ + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 3, 3, 3); + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEADER_MODE, + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, + BIT(V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE), + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME); + + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_PROFILE, + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, + ~(BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_MAIN) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_HIGH)), + V4L2_MPEG_VIDEO_H264_PROFILE_MAIN); + /* A minimum: the driver raises it to what size, rate and bit rate need. */ + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_LEVEL, + V4L2_MPEG_VIDEO_H264_LEVEL_5_2, 0, + V4L2_MPEG_VIDEO_H264_LEVEL_1_0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MIN_QP, 0, 51, 1, 0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_H264_MAX_QP, 0, 51, 1, 51); + + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_PROFILE, + V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN, 0, + V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN); + v4l2_ctrl_new_std_menu(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_LEVEL, + V4L2_MPEG_VIDEO_HEVC_LEVEL_5_1, 0, + V4L2_MPEG_VIDEO_HEVC_LEVEL_1); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MIN_QP, 0, 51, 1, 0); + v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP, 0, 51, 1, 51); + + /* The firmware holds up to five frames; one more keeps it fed. */ + c = v4l2_ctrl_new_std(hdl, ops, V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, + AVE_MAX_SLOTS + 1); + if (c) + c->flags |= V4L2_CTRL_FLAG_READ_ONLY; + + if (hdl->error) { + int ret = hdl->error; + + v4l2_ctrl_handler_free(hdl); + return ret; + } + ctx->fh.ctrl_handler = hdl; + return v4l2_ctrl_handler_setup(hdl); +} + +/* Queues */ + +static int ave_queue_setup(struct vb2_queue *vq, unsigned int *nbufs, unsigned int *nplanes, + unsigned int sizes[], struct device *alloc_devs[]) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vq); + const struct v4l2_pix_format_mplane *f = + V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; + unsigned int have = vb2_get_num_buffers(vq); + unsigned long slot = roundup_pow_of_two(PAGE_ALIGN(f->plane_fmt[0].sizeimage)); + unsigned int max = (V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_OUTPUT_BUDGET : + AVE_CAPTURE_BUDGET) / slot; + + max = clamp(max, V4L2_TYPE_IS_OUTPUT(vq->type) ? AVE_MAX_SLOTS + 1U : 2U, + AVE_MAX_QUEUE_BUFS); + + /* Frames stay with the firmware until encoded; keep one spare. */ + if (V4L2_TYPE_IS_OUTPUT(vq->type) && have + *nbufs < AVE_MAX_SLOTS + 1) + *nbufs = AVE_MAX_SLOTS + 1 - have; + /* A full set of both queues must fit the encoder's 256 MiB DART window. */ + if (have >= max) { + dev_dbg(ctx->ave->dev, "%s queue already has its %u buffers\n", + V4L2_TYPE_IS_OUTPUT(vq->type) ? "OUTPUT" : "CAPTURE", max); + return -ENOMEM; + } + if (*nbufs > max - have) + dev_dbg(ctx->ave->dev, "%s queue: %u buffers asked, %u given\n", + V4L2_TYPE_IS_OUTPUT(vq->type) ? "OUTPUT" : "CAPTURE", *nbufs, max - have); + *nbufs = min(*nbufs, max - have); + if (*nplanes) + return (*nplanes != 1 || sizes[0] < f->plane_fmt[0].sizeimage) ? -EINVAL : 0; + *nplanes = 1; + sizes[0] = f->plane_fmt[0].sizeimage; + return 0; +} + +static int ave_buf_prepare(struct vb2_buffer *vb) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + const struct v4l2_pix_format_mplane *f = + V4L2_TYPE_IS_OUTPUT(vb->type) ? &ctx->src_fmt : &ctx->dst_fmt; + + if (vb2_plane_size(vb, 0) < f->plane_fmt[0].sizeimage) + return -EINVAL; + /* Keep what the client filled in, so its layout can be checked. */ + if (V4L2_TYPE_IS_OUTPUT(vb->type) && !vb2_get_plane_payload(vb, 0)) + vb2_set_plane_payload(vb, 0, f->plane_fmt[0].sizeimage); + return 0; +} + +static void ave_buf_queue(struct vb2_buffer *vb) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, to_vb2_v4l2_buffer(vb)); +} + +static void ave_return_bufs(struct ave_ctx *ctx, struct vb2_queue *q, + enum vb2_buffer_state state) +{ + struct vb2_v4l2_buffer *vbuf; + + for (;;) { + vbuf = V4L2_TYPE_IS_OUTPUT(q->type) ? + v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx) : + v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!vbuf) + break; + v4l2_m2m_buf_done(vbuf, state); + } +} + +/* Sessions */ + +static int ave_session_index(struct ave_dev *ave, struct ave_ctx *ctx) +{ + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + if (ave->sessions[i] == ctx) + return i; + return -1; +} + +static unsigned int ave_session_count(struct ave_dev *ave) +{ + unsigned int n = 0; + + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) + n += !!ave->sessions[i]; + return n; +} + +/* + * Drain (with @flush), stop and close the session in the firmware. The + * last session to go halts the firmware. Caller holds hw_lock. + */ +static void ave_session_close(struct ave_ctx *ctx, bool flush) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + int idx = ave_session_index(ave, ctx); + + lockdep_assert_held(&ave->hw_lock); + if (!sess || idx < 0) + return; + if (sess->fw_open && ave->fw_state == AVE_FW_RUNNING) { + /* Complete is a barrier: answered after every frame before it. */ + if (flush && sess->frames && !ave_cmd_complete(ctx, AVE_DRAIN_COMPLETES)) + ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); + if (ave->fw_state == AVE_FW_RUNNING) + ave_cmd_close(ctx); + } + sess->fw_open = false; + ave->sessions[idx] = NULL; + if (!ave_session_count(ave)) + ave_fw_halt(ave); +} + +/* + * Open and Start the session in the firmware, booting the firmware first + * if no other session runs. Returns -EAGAIN when a halted coprocessor has + * not lost power yet. Caller holds hw_lock. + */ +static int ave_session_attach(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + int idx = -1, ret; + + lockdep_assert_held(&ave->hw_lock); + for (unsigned int i = 0; i < AVE_MAX_SESSIONS && idx < 0; i++) + if (!ave->sessions[i]) + idx = i; + if (idx < 0) { + dev_err(ave->dev, "already encoding %u streams\n", AVE_MAX_SESSIONS); + return -EBUSY; + } + if (!ave_session_count(ave)) { + ret = ave_fw_ensure_running(ave); + if (ret) + return ret; + } else if (ave->fw_state != AVE_FW_RUNNING) { + dev_err(ave->dev, "firmware state %d, cannot add a session\n", ave->fw_state); + return -EIO; + } + + sess->index = idx; + sess->frames = 0; + sess->nudged = false; + ave->sessions[idx] = ctx; + ret = ave_cmd_open(ctx); + if (!ret) + ret = ave_cmd_start(ctx); + if (ret) + ave_session_close(ctx, false); + return ret; +} + +/* + * Attach with the device runtime active and one PM reference held by the + * caller. A halted coprocessor restarts only from reset, and its power can + * take a moment to go (the DARTs suspend a little later), so drop the + * reference and cycle the power a few times before giving up. On return + * the reference is still held, unless *pm_lost says otherwise. + */ +static int ave_session_attach_cycled(struct ave_ctx *ctx, bool *pm_lost) +{ + struct ave_dev *ave = ctx->ave; + int ret = -EAGAIN; + + *pm_lost = false; + for (unsigned int i = 0; i < AVE_RESTART_TRIES && ret == -EAGAIN; i++) { + if (i) { + pm_runtime_put_sync_suspend(ave->dev); + msleep(AVE_RESTART_WAIT_MS); + ret = pm_runtime_resume_and_get(ave->dev); + if (ret) { + *pm_lost = true; + return ret; + } + } + mutex_lock(&ave->hw_lock); + ret = ave_session_attach(ctx); + mutex_unlock(&ave->hw_lock); + } + if (ret == -EAGAIN) + dev_err(ave->dev, "coprocessor did not power down; cannot restart it\n"); + return ret; +} + +/* Return frames the firmware never answered; then drop the session. */ +static void ave_session_free(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + + if (!sess) + return; + for (unsigned int i = 0; i < sess->p.slots; i++) { + struct ave_slot *s = &sess->slot[i]; + + if (s->src) + v4l2_m2m_buf_done(s->src, VB2_BUF_STATE_ERROR); + s->src = NULL; + } + atomic_set(&ctx->ready, 0); + ave_sess_destroy(ctx); + pm_runtime_mark_last_busy(ave->dev); + pm_runtime_put_autosuspend(ave->dev); +} + +/* Boot the firmware if needed, then Open and Start a session. */ +static int ave_session_begin(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess; + bool pm_lost; + int ret; + + if (ctx->sess) + return 0; + ret = ave_fw_check_build(ave); + if (ret) + return ret; + if (READ_ONCE(ave->sessions[0]) && READ_ONCE(ave->sessions[1])) + return -EBUSY; + /* A halted firmware restarts only from reset: let the power go first. */ + if (READ_ONCE(ave->fw_state) == AVE_FW_HALTED) + pm_runtime_suspend(ave->dev); + sess = ave_sess_create(ctx); + if (IS_ERR(sess)) { + dev_err(ave->dev, "cannot set up a session: %ld\n", PTR_ERR(sess)); + return PTR_ERR(sess); + } + + ctx->sess = sess; + ret = pm_runtime_resume_and_get(ave->dev); + if (ret) { + ave_sess_destroy(ctx); + return ret; + } + ret = ave_session_attach_cycled(ctx, &pm_lost); + if (ret) { + dev_err(ave->dev, "cannot start an encoding session: %d\n", ret); + if (pm_lost) + ave_sess_destroy(ctx); + else + ave_session_free(ctx); + } + return ret; +} + +/* + * After system sleep: the session was closed when the firmware went down. + * Open and Start it again with the same parameters; its next frame is an + * IDR with fresh parameter sets, so the stream stays decodable. + */ +static int ave_session_reopen(struct ave_ctx *ctx) +{ + bool pm_lost; + int ret; + + ret = ave_session_attach_cycled(ctx, &pm_lost); + if (ret) { + if (pm_lost) { + /* The session's PM reference is gone; do not drop it again. */ + pm_runtime_get_noresume(ctx->ave->dev); + } + return ret; + } + ctx->sess->resume = false; + dev_dbg(ctx->ave->dev, "session reopened after system sleep\n"); + return 0; +} + +/* + * System sleep: drain every open session, close it and halt the firmware, + * as at the end of a stream. Sessions reopen on their next frame. + * Caller holds hw_lock. + */ +void ave_v4l2_suspend(struct ave_dev *ave) +{ + lockdep_assert_held(&ave->hw_lock); + for (unsigned int i = 0; i < AVE_MAX_SESSIONS; i++) { + struct ave_ctx *ctx = ave->sessions[i]; + + if (!ctx) + continue; + cancel_delayed_work(&ctx->poll_work); + ave_session_close(ctx, true); + ctx->sess->resume = true; + } +} + +static void ave_session_end(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + + cancel_delayed_work_sync(&ctx->poll_work); + mutex_lock(&ave->hw_lock); + ave_session_close(ctx, true); + mutex_unlock(&ave->hw_lock); + ave_session_free(ctx); +} + +static void ave_reset_drain(struct ave_ctx *ctx) +{ + ctx->draining = false; + WRITE_ONCE(ctx->flush_pending, false); + ctx->stopped = false; + ctx->drain_src = NULL; +} + +static int ave_start_streaming(struct vb2_queue *q, unsigned int count) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(q); + struct vb2_queue *other = V4L2_TYPE_IS_OUTPUT(q->type) ? + v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx) : + v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx); + int ret = 0; + + if (V4L2_TYPE_IS_OUTPUT(q->type)) + ctx->src_sequence = 0; + else + ctx->dst_sequence = 0; + ave_reset_drain(ctx); + + /* The session starts once both queues stream; errors show up here. */ + if (vb2_is_streaming(other)) + ret = ave_session_begin(ctx); + if (ret) + ave_return_bufs(ctx, q, VB2_BUF_STATE_QUEUED); + return ret; +} + +static void ave_stop_streaming(struct vb2_queue *q) +{ + struct ave_ctx *ctx = vb2_get_drv_priv(q); + + ave_session_end(ctx); + ave_reset_drain(ctx); + ave_return_bufs(ctx, q, VB2_BUF_STATE_ERROR); +} + +static const struct vb2_ops ave_qops = { + .queue_setup = ave_queue_setup, + .buf_prepare = ave_buf_prepare, + .buf_queue = ave_buf_queue, + .start_streaming = ave_start_streaming, + .stop_streaming = ave_stop_streaming, +}; + +static int ave_queue_init(void *priv, struct vb2_queue *src, struct vb2_queue *dst) +{ + struct ave_ctx *ctx = priv; + int ret; + + src->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + src->io_modes = VB2_MMAP | VB2_DMABUF; + src->drv_priv = ctx; + src->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + src->ops = &ave_qops; + src->mem_ops = &vb2_dma_contig_memops; + src->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + src->lock = &ctx->ave->vdev_lock; + src->dev = ctx->ave->dev; + ret = vb2_queue_init(src); + if (ret) + return ret; + + dst->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; + dst->io_modes = VB2_MMAP | VB2_DMABUF; + dst->drv_priv = ctx; + dst->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + dst->ops = &ave_qops; + dst->mem_ops = &vb2_dma_contig_memops; + dst->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + dst->lock = &ctx->ave->vdev_lock; + dst->dev = ctx->ave->dev; + return vb2_queue_init(dst); +} + +/* + * Jobs. A job delivers finished frames to CAPTURE buffers and sends at most + * one new frame. Up to five frames are inside the firmware at once; their + * answers are collected by the job and, while frames are outstanding, by a + * short poller, since the firmware interrupt is not wired up. + */ + +#define AVE_POLL_MS 2 + +static int ave_job_ready(void *priv) +{ + struct ave_ctx *ctx = priv; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + + if (atomic_read(&ctx->ready) || READ_ONCE(ctx->flush_pending)) + return 1; + if (ctx->stopped || (ctx->draining && !ctx->drain_src)) + return 0; + return v4l2_m2m_num_src_bufs_ready(m2m) > 0; +} + +/* + * Hand finished frames, in bitstream order, to CAPTURE buffers. Returns + * true when the frame that ends a drain went out (with the LAST flag). + */ +static bool ave_deliver(struct ave_ctx *ctx) +{ + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + struct ave_session *sess = ctx->sess; + bool last = false; + + while (sess && sess->done_count) { + unsigned int slot = sess->done_fifo[sess->done_head]; + struct ave_slot *s = &sess->slot[slot]; + struct vb2_v4l2_buffer *dst; + enum vb2_buffer_state state = VB2_BUF_STATE_DONE; + bool key = false; + + dst = v4l2_m2m_dst_buf_remove(m2m); + if (!dst) + break; + if (ave_sess_output(ctx, slot, dst, &key)) { + vb2_set_plane_payload(&dst->vb2_buf, 0, 0); + state = VB2_BUF_STATE_ERROR; + } + v4l2_m2m_buf_copy_metadata(s->src, dst); + dst->flags &= ~(V4L2_BUF_FLAG_KEYFRAME | V4L2_BUF_FLAG_PFRAME | + V4L2_BUF_FLAG_BFRAME | V4L2_BUF_FLAG_LAST); + dst->flags |= key ? V4L2_BUF_FLAG_KEYFRAME : V4L2_BUF_FLAG_PFRAME; + dst->sequence = ctx->dst_sequence++; + + sess->done_head = (sess->done_head + 1) % AVE_MAX_SLOTS; + sess->done_count--; + atomic_dec(&ctx->ready); + s->done = false; + ave_sess_prepare_slot(sess, slot); + + /* The last frame of a drain carries the LAST flag. */ + if (READ_ONCE(ctx->flush_pending) && !sess->fw_open && !sess->done_count) { + dst->flags |= V4L2_BUF_FLAG_LAST; + last = true; + } + v4l2_m2m_buf_done(s->src, state); + s->src = NULL; + v4l2_m2m_buf_done(dst, state); + } + return last; +} + +static bool ave_slot_answered(struct ave_dev *ave, void *arg) +{ + struct ave_session *sess = arg; + + return sess->busy < sess->p.slots || sess->done_count; +} + +static bool ave_frame_answered(struct ave_dev *ave, void *arg) +{ + struct ave_slot *s = arg; + + return !s->busy; +} + +/* Send the next OUTPUT frame. Caller holds hw_lock. */ +static int ave_encode_one(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + struct vb2_v4l2_buffer *src; + dma_addr_t input; + bool last; + int slot, ret; + + slot = ave_sess_free_slot(sess); + if (slot < 0 && sess->busy == sess->p.slots) { + /* Every slot is inside the firmware: wait for one to come back. */ + ret = ave_ipc_pump(ave, AVE_FRAME_WAIT_MS, ave_slot_answered, sess); + if (ret) + return ret; + ave_deliver(ctx); + slot = ave_sess_free_slot(sess); + } + if (slot < 0) + return 0; /* waiting for CAPTURE buffers */ + + src = v4l2_m2m_src_buf_remove(m2m); + if (!src) + return 0; + last = src == ctx->drain_src; + src->sequence = ctx->src_sequence++; + + ret = ave_sess_input(ctx, slot, src, &input); + if (!ret) + ret = ave_cmd_process(ctx, slot, src, input, READ_ONCE(ctx->bitrate), + xchg(&ctx->force_key_frame, false)); + if (ret) { + v4l2_m2m_buf_done(src, VB2_BUF_STATE_ERROR); + return ret; + } + + /* Like macOS: the first frame goes alone, then a Complete. */ + if (sess->frames == 1) { + ret = ave_ipc_pump(ave, AVE_FRAME_WAIT_MS, ave_frame_answered, &sess->slot[slot]); + if (!ret) + ret = ave_cmd_complete(ctx, 1); + if (!ret) + ret = ave_ipc_settle(ave, AVE_FRAME_WAIT_MS); + if (ret) + return ret; + } else { + ret = ave_ipc_pump(ave, 0, NULL, NULL); + if (ret) + return ret; + } + + if (last) { + ctx->drain_src = NULL; + WRITE_ONCE(ctx->flush_pending, true); + } + return 0; +} + +/* + * End of a drain: flush and close the session, deliver what is left and + * finish with a LAST buffer. Caller holds hw_lock; returns true once the + * LAST buffer is out. + */ +static bool ave_finish_drain(struct ave_ctx *ctx) +{ + struct ave_dev *ave = ctx->ave; + struct ave_session *sess = ctx->sess; + struct vb2_v4l2_buffer *dst; + + if (sess && sess->fw_open) + ave_session_close(ctx, true); + if (sess && sess->busy) { + /* The run job fails the session. */ + dev_err(ave->dev, "%u frames never came back from the firmware\n", sess->busy); + return false; + } + if (sess && sess->done_count) + return ave_deliver(ctx); /* false: waiting for CAPTURE buffers */ + + /* Nothing left to deliver: an empty buffer carries the LAST flag. */ + dst = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!dst) + return false; + vb2_set_plane_payload(&dst->vb2_buf, 0, 0); + dst->flags |= V4L2_BUF_FLAG_LAST; + dst->sequence = ctx->dst_sequence++; + v4l2_m2m_buf_done(dst, VB2_BUF_STATE_DONE); + return true; +} + +/* The firmware or the session failed: return everything, fail the queues. */ +static void ave_job_fail(struct ave_ctx *ctx, int err) +{ + struct ave_dev *ave = ctx->ave; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + + dev_err(ave->dev, "encoding failed: %d\n", err); + mutex_lock(&ave->hw_lock); + ave_session_close(ctx, false); + mutex_unlock(&ave->hw_lock); + ave_session_free(ctx); + ave_return_bufs(ctx, v4l2_m2m_get_src_vq(m2m), VB2_BUF_STATE_ERROR); + ave_return_bufs(ctx, v4l2_m2m_get_dst_vq(m2m), VB2_BUF_STATE_ERROR); + vb2_queue_error(v4l2_m2m_get_src_vq(m2m)); + vb2_queue_error(v4l2_m2m_get_dst_vq(m2m)); + ave_reset_drain(ctx); +} + +static void ave_run_work(struct work_struct *work) +{ + struct ave_ctx *ctx = container_of(work, struct ave_ctx, run_work); + struct ave_dev *ave = ctx->ave; + struct v4l2_m2m_ctx *m2m = ctx->fh.m2m_ctx; + bool eos = false, poll = false; + int ret = 0; + + /* + * After a drain the next frame starts a new session; after system + * sleep it reopens the old one in the restarted firmware. + */ + if (!READ_ONCE(ctx->flush_pending) && v4l2_m2m_num_src_bufs_ready(m2m)) { + if (!ctx->sess) + ret = ave_session_begin(ctx); + else if (ctx->sess->resume) + ret = ave_session_reopen(ctx); + } + + mutex_lock(&ave->hw_lock); + /* + * A closed session (after a drain or for sleep) may still hold frames + * for CAPTURE buffers; only an open one must still own the firmware. + */ + if (!ret && ctx->sess && ctx->sess->fw_open && ave_session_index(ave, ctx) < 0) { + dev_err(ave->dev, "session lost the firmware\n"); + ret = -EIO; + } + if (!ret) { + eos = ave_deliver(ctx); + if (!eos && !READ_ONCE(ctx->flush_pending) && ctx->sess && + v4l2_m2m_num_src_bufs_ready(m2m) && v4l2_m2m_num_dst_bufs_ready(m2m)) + ret = ave_encode_one(ctx); + } + if (!ret && !eos) { + eos = ave_deliver(ctx); + if (!eos && READ_ONCE(ctx->flush_pending)) + eos = ave_finish_drain(ctx); + /* Frames outstanding need a running firmware. */ + if (!eos && ctx->sess && ctx->sess->busy && ave->fw_state != AVE_FW_RUNNING) { + dev_err(ave->dev, "%u frames outstanding, firmware state %d\n", + ctx->sess->busy, ave->fw_state); + ret = -EIO; + } + poll = !ret && !eos && ctx->sess && ctx->sess->busy; + } + mutex_unlock(&ave->hw_lock); + + if (ret) { + ave_job_fail(ctx, ret); + } else if (eos) { + ave_session_free(ctx); + ctx->draining = false; + WRITE_ONCE(ctx->flush_pending, false); + ctx->stopped = true; + v4l2_event_queue_fh(&ctx->fh, &ave_eos_event); + } else if (poll) { + mod_delayed_work(ave->wq, &ctx->poll_work, msecs_to_jiffies(AVE_POLL_MS)); + } + pm_runtime_mark_last_busy(ave->dev); + v4l2_m2m_job_finish(ave->m2m_dev, m2m); +} + +/* Collect frame answers while frames are outstanding and no job runs. */ +static void ave_poll_work(struct work_struct *work) +{ + struct ave_ctx *ctx = container_of(to_delayed_work(work), struct ave_ctx, poll_work); + struct ave_dev *ave = ctx->ave; + bool again = false, fail = false; + + mutex_lock(&ave->hw_lock); + if (ctx->sess && ave_session_index(ave, ctx) >= 0) { + fail = ave_ipc_pump(ave, 0, NULL, NULL) != 0; + if (!fail && ave_sess_starved(ctx->sess) && + !v4l2_m2m_num_src_bufs_ready(ctx->fh.m2m_ctx)) { + dev_dbg(ave->dev, "no input, flushing %u held frames\n", ctx->sess->busy); + ctx->sess->nudged = true; + ctx->sess->progress = jiffies; + fail = ave_cmd_complete(ctx, 1) != 0; + } + if (!fail && ave_sess_stalled(ctx->sess)) { + /* + * The firmware holds frames it should have answered. + * Treat it as stuck, so the session fails now instead + * of leaving the client waiting. + */ + dev_err(ave->dev, "%u frames unanswered for %u ms; firmware is stuck\n", + ctx->sess->busy, AVE_FRAME_WAIT_MS); + ave->fw_state = AVE_FW_BROKEN; + fail = true; + } + again = !fail && ctx->sess->busy; + } else if (ctx->sess && ctx->sess->busy) { + dev_err(ave->dev, "%u frames outstanding in a session that lost the firmware\n", + ctx->sess->busy); + fail = true; + } + mutex_unlock(&ave->hw_lock); + + /* Jobs run on the same ordered queue, so none runs right now. */ + if (fail) + ave_job_fail(ctx, -EIO); + else if (atomic_read(&ctx->ready)) + v4l2_m2m_try_schedule(ctx->fh.m2m_ctx); + if (again) + queue_delayed_work(ave->wq, &ctx->poll_work, msecs_to_jiffies(AVE_POLL_MS)); +} + +static void ave_device_run(void *priv) +{ + struct ave_ctx *ctx = priv; + + queue_work(ctx->ave->wq, &ctx->run_work); +} + +static const struct v4l2_m2m_ops ave_m2m_ops = { + .device_run = ave_device_run, + .job_ready = ave_job_ready, +}; + +/* File operations */ + +static int ave_open(struct file *file) +{ + struct ave_dev *ave = video_drvdata(file); + struct ave_ctx *ctx; + int ret; + + ctx = kzalloc_obj(*ctx); + if (!ctx) + return -ENOMEM; + ctx->ave = ave; + INIT_WORK(&ctx->run_work, ave_run_work); + INIT_DELAYED_WORK(&ctx->poll_work, ave_poll_work); + atomic_set(&ctx->ready, 0); + ave_reset_fmts(ctx); + + v4l2_fh_init(&ctx->fh, video_devdata(file)); + ret = ave_init_ctrls(ctx); + if (ret) + goto err_fh; + + ctx->fh.m2m_ctx = v4l2_m2m_ctx_init(ave->m2m_dev, ctx, ave_queue_init); + if (IS_ERR(ctx->fh.m2m_ctx)) { + ret = PTR_ERR(ctx->fh.m2m_ctx); + goto err_ctrls; + } + /* Finished frames can be delivered with no new OUTPUT buffer queued. */ + v4l2_m2m_set_src_buffered(ctx->fh.m2m_ctx, true); + v4l2_fh_add(&ctx->fh, file); + return 0; + +err_ctrls: + v4l2_ctrl_handler_free(&ctx->hdl); +err_fh: + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + return ret; +} + +static int ave_release(struct file *file) +{ + struct ave_ctx *ctx = ave_file_to_ctx(file); + struct ave_dev *ave = ctx->ave; + + mutex_lock(&ave->vdev_lock); + v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); + mutex_unlock(&ave->vdev_lock); + cancel_delayed_work_sync(&ctx->poll_work); + cancel_work_sync(&ctx->run_work); + v4l2_ctrl_handler_free(&ctx->hdl); + v4l2_fh_del(&ctx->fh, file); + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + return 0; +} + +static const struct v4l2_file_operations ave_fops = { + .owner = THIS_MODULE, + .open = ave_open, + .release = ave_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +}; + +int ave_v4l2_register(struct ave_dev *ave) +{ + struct video_device *vdev = &ave->vdev; + int ret; + + mutex_init(&ave->vdev_lock); + ave->wq = alloc_ordered_workqueue("apple-ave", WQ_MEM_RECLAIM); + if (!ave->wq) + return -ENOMEM; + + ret = v4l2_device_register(ave->dev, &ave->v4l2_dev); + if (ret) + goto err_wq; + + ave->m2m_dev = v4l2_m2m_init(&ave_m2m_ops); + if (IS_ERR(ave->m2m_dev)) { + ret = PTR_ERR(ave->m2m_dev); + goto err_v4l2; + } + + strscpy(vdev->name, AVE_NAME, sizeof(vdev->name)); + vdev->fops = &ave_fops; + vdev->ioctl_ops = &ave_ioctl_ops; + vdev->release = video_device_release_empty; + vdev->lock = &ave->vdev_lock; + vdev->v4l2_dev = &ave->v4l2_dev; + vdev->vfl_dir = VFL_DIR_M2M; + vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING; + video_set_drvdata(vdev, ave); + + ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1); + if (ret) + goto err_m2m; + return 0; + +err_m2m: + v4l2_m2m_release(ave->m2m_dev); +err_v4l2: + v4l2_device_unregister(&ave->v4l2_dev); +err_wq: + destroy_workqueue(ave->wq); + return ret; +} + +void ave_v4l2_unregister(struct ave_dev *ave) +{ + video_unregister_device(&ave->vdev); + v4l2_m2m_release(ave->m2m_dev); + v4l2_device_unregister(&ave->v4l2_dev); + destroy_workqueue(ave->wq); +} diff --git a/drivers/media/platform/apple/ave/ave.h b/drivers/media/platform/apple/ave/ave.h new file mode 100644 index 0000000000000..0f6e2b38d86ca --- /dev/null +++ b/drivers/media/platform/apple/ave/ave.h @@ -0,0 +1,336 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Apple AVE video encoder + * + * Copyright The Aurora Silicon Contributors + */ +#ifndef _APPLE_AVE_H_ +#define _APPLE_AVE_H_ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct debugfs_blob_wrapper; +struct dentry; +struct dev_pm_domain_list; +struct iommu_domain; + +/* Frames the firmware can hold at once per session (Process "slots"). */ +#define AVE_MAX_SLOTS 5 +/* Sessions one firmware boot serves at once (macOS was seen running two). */ +#define AVE_MAX_SESSIONS 2 + +/* + * Coprocessor memory every firmware boot uses, all inside one allocation. + * Host commands live in three areas: one large buffer for Start, small + * buffers for the scalar commands and one buffer per Process slot, so a + * command stays untouched until the firmware acknowledges it. + */ +#define AVE_IPC_BYTES (20 * SZ_1M) +#define AVE_IPC_CHANNEL_OFF 0x4000 +#define AVE_IPC_HANDSHAKE_OFF 0x24000 +#define AVE_IPC_SHARED_OFF 0x28000 +#define AVE_IPC_SHARED_BYTES SZ_64K +#define AVE_IPC_CMD_OFF 0x40000 +#define AVE_IPC_CMD_BYTES 0x1c000 +#define AVE_IPC_SCALAR_OFF 0x5c000 +#define AVE_IPC_SCALAR_BYTES 0x100 +#define AVE_IPC_SCALAR_BUFS 16 +#define AVE_IPC_PROC_OFF 0x60000 +#define AVE_IPC_PROC_BYTES 0x10000 +#define AVE_IPC_PROC_BUFS (AVE_MAX_SESSIONS * AVE_MAX_SLOTS) +#define AVE_IPC_END (AVE_IPC_PROC_OFF + AVE_IPC_PROC_BUFS * AVE_IPC_PROC_BYTES) + +/* Command buffer ids, one bit each in ave_dev.unacked. */ +#define AVE_BUF_CMD 0 +#define AVE_BUF_SCALAR(i) (1 + (i)) +#define AVE_BUF_PROC(s) (1 + AVE_IPC_SCALAR_BUFS + (s)) +#define AVE_BUF_COUNT AVE_BUF_PROC(AVE_IPC_PROC_BUFS) + +#define AVE_MSG_MAX SZ_4K + +enum ave_codec { + AVE_CODEC_H264 = 1, + AVE_CODEC_HEVC = 2, +}; + +struct ave_buf { + void *cpu; + dma_addr_t dma; + size_t size; /* what the firmware is told */ + size_t alloc; /* what is mapped, page rounded */ + unsigned long attrs; +}; + +struct ave_fw_segment { + phys_addr_t phys; + u64 iova; + size_t size; +}; + +struct ave_ring { + char name[16]; + u32 mode; + u32 doorbell; + u32 slots; + u32 offset; + int consumer; + int producer; +}; + +enum ave_fw_state { + AVE_FW_OFF, /* coprocessor in reset */ + AVE_FW_RUNNING, + AVE_FW_HALTED, /* told to halt; restarts through the boot path */ + AVE_FW_BROKEN, +}; + +/* The two large session commands, and their sizes per codec */ +enum ave_image { + AVE_IMAGE_START, + AVE_IMAGE_PROCESS, + AVE_IMAGE_COUNT, +}; + +#define AVE_START_H264_BYTES 0x3180 +#define AVE_START_HEVC_BYTES 0x13f28 +#define AVE_PROCESS_H264_BYTES 0x63d8 +#define AVE_PROCESS_HEVC_BYTES 0xb1c0 + +/* The firmware build whose command layouts the driver generates */ +#define AVE_FW_TEXT_CRC_9003_78_0 0x69c54454 + +struct ave_ctx; + +struct ave_dev { + struct device *dev; + void __iomem *codec; + void __iomem *mbox; + void __iomem *coproc; + struct dev_pm_domain_list *pd_list; + struct iommu_domain *domain; + + struct ave_fw_segment text; + struct ave_fw_segment data; + u32 text_crc; + void *data_pristine; + bool data_dirty; + + /* debugfs: the last Start and first Process built, per codec */ + struct dentry *debugfs; + void *dump[2][AVE_IMAGE_COUNT]; + + /* Serialises every firmware and ring access. */ + struct mutex hw_lock; + enum ave_fw_state fw_state; + struct ave_buf ipc; + struct ave_buf boot_cfg; + struct ave_buf log; + struct ave_buf heap; + u64 ipc_fw_base; + u32 channel_bytes; + struct ave_ring ring[2]; + u32 next_client; + u32 unacked; + u8 want[16]; + int reply_err; + unsigned int scalar_next; + struct ave_ctx *sessions[AVE_MAX_SESSIONS]; + u8 msg[AVE_MSG_MAX]; + u32 msg_len; + + struct v4l2_device v4l2_dev; + struct video_device vdev; + struct v4l2_m2m_dev *m2m_dev; + /* The video device's ioctl and queue lock. */ + struct mutex vdev_lock; + struct workqueue_struct *wq; +}; + +/* Encoding parameters, fixed for the lifetime of a session. */ +struct ave_params { + enum ave_codec codec; + u32 width, height; /* visible picture */ + u32 cw, ch; /* coded size, multiples of 16 */ + u32 stride; /* NV12 line stride, both planes */ + u32 mbs; /* 16x16 macroblocks */ + u32 fps_num, fps_den; /* frames per second = num / den */ + u32 fps_int; /* the same, rounded */ + u32 slots; + u32 refs; /* max_num_ref_frames + 1 */ + bool bframes; + bool intra_only; + bool reorder_off; /* the client asked for no B-frames */ + u32 gop; + u32 rc_mode; + u32 bitrate; + u32 limit_bytes; /* 0: no data-rate limit */ + s32 min_qp; + u32 max_qp; + u32 profile; /* H.264: 2, 4, 6; HEVC: 1 */ + u32 level; /* firmware level enum, 1-based */ + bool high; + u32 coded_cap; +}; + +struct ave_slot { + struct ave_buf coded; + struct ave_buf meta; + struct ave_buf aux; /* H.264 per-frame buffer, HEVC slice headers */ + struct ave_buf bounce; /* input copied to the encoder's stride */ + struct vb2_v4l2_buffer *src; + u32 status; + bool busy; /* Process sent, reply pending */ + bool done; /* reply in, output not delivered yet */ +}; + +#define AVE_MAX_REFS 5 +#define AVE_TILE_BUFS 8 + +struct ave_session { + struct ave_params p; + struct ave_buf hdr; + struct ave_buf ref_a[AVE_MAX_REFS]; + struct ave_buf ref_b[AVE_MAX_REFS]; + struct ave_buf ref_c[AVE_MAX_REFS]; + struct ave_buf ref_d[AVE_MAX_REFS]; + struct ave_buf mb; + struct ave_buf row_a, row_b, fixed; + struct ave_buf tile[AVE_TILE_BUFS]; + struct ave_buf tile_aux[AVE_TILE_BUFS]; + struct ave_buf hevc_extra[2]; + struct ave_slot slot[AVE_MAX_SLOTS]; + u8 done_fifo[AVE_MAX_SLOTS]; + unsigned int done_head; + unsigned int done_count; + unsigned int busy; + u32 frames; /* Process commands sent */ + u32 client; + u64 token; /* command order token, per session */ + unsigned int index; /* in ave_dev.sessions[] */ + bool fw_open; /* Open sent, Close not yet */ + bool resume; /* closed for system sleep, reopen on next frame */ + bool copy_noted; + /* CBR: bytes under budget so far, and who pads (see ave_sess_cbr_fill()) */ + s64 cbr_surplus; + bool fw_fills; + bool host_fill_noted; + bool warned; + bool checked[2]; /* Start/Process images checked for stale addresses */ + bool handed; /* Start sent: the firmware knows the buffers */ + bool nudged; /* Complete sent because input stopped */ + unsigned long progress; /* jiffies of the last frame sent or answered */ + unsigned long last_sent; /* jiffies of the last frame sent */ +}; + +struct ave_ctx { + struct v4l2_fh fh; + struct ave_dev *ave; + struct v4l2_ctrl_handler hdl; + struct v4l2_pix_format_mplane src_fmt; + struct v4l2_pix_format_mplane dst_fmt; + struct v4l2_fract timeperframe; + struct work_struct run_work; + struct delayed_work poll_work; + + u32 bitrate_mode; + u32 bitrate; + u32 bitrate_peak; + u32 gop_size; + u32 b_frames; + u32 h264_profile; + u32 h264_level; + u32 h264_min_qp; + u32 h264_max_qp; + u32 hevc_level; + u32 hevc_min_qp; + u32 hevc_max_qp; + bool force_key_frame; + + struct ave_session *sess; + atomic_t ready; /* outputs waiting for a CAPTURE buffer */ + bool draining; + bool flush_pending; + bool stopped; + struct vb2_v4l2_buffer *drain_src; + u32 src_sequence; + u32 dst_sequence; +}; + +static inline struct ave_ctx *ave_file_to_ctx(struct file *file) +{ + return container_of(file_to_v4l2_fh(file), struct ave_ctx, fh); +} + +/* ave-fw.c */ +int ave_fw_probe(struct ave_dev *ave); +void ave_fw_remove(struct ave_dev *ave); +int ave_fw_ensure_running(struct ave_dev *ave); +void ave_fw_halt(struct ave_dev *ave); +int ave_fw_shutdown(struct ave_dev *ave); +void ave_fw_lost(struct ave_dev *ave); +int ave_fw_check_build(struct ave_dev *ave); +int ave_alloc(struct ave_dev *ave, struct ave_buf *b, size_t size, unsigned long attrs); +int ave_alloc_padded(struct ave_dev *ave, struct ave_buf *b, size_t size, size_t alloc, + unsigned long attrs); +void ave_free(struct ave_dev *ave, struct ave_buf *b); + +/* ave-ipc.c */ +int ave_ipc_negotiate(struct ave_dev *ave, const u32 hello[4]); +void *ave_ipc_cmd(struct ave_dev *ave, unsigned int buf, size_t len); +int ave_ipc_scalar_buf(struct ave_dev *ave); +int ave_ipc_wait_ack(struct ave_dev *ave, unsigned int buf); +int ave_ipc_submit(struct ave_dev *ave, unsigned int buf, size_t len, u64 timeout_us, + bool reply); +int ave_ipc_pump(struct ave_dev *ave, unsigned int wait_ms, + bool (*cond)(struct ave_dev *ave, void *arg), void *arg); +int ave_ipc_settle(struct ave_dev *ave, unsigned int wait_ms); + +/* ave-cmd.c */ +int ave_cmd_config(struct ave_dev *ave); +int ave_cmd_halt(struct ave_dev *ave); +int ave_cmd_open(struct ave_ctx *ctx); +int ave_cmd_start(struct ave_ctx *ctx); +int ave_cmd_process(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t input, u32 bitrate, bool force_key); +int ave_cmd_complete(struct ave_ctx *ctx, unsigned int count); +int ave_cmd_close(struct ave_ctx *ctx); + +/* ave-sess.c */ +struct ave_session *ave_sess_create(struct ave_ctx *ctx); +void ave_sess_destroy(struct ave_ctx *ctx); +int ave_sess_free_slot(struct ave_session *sess); +void ave_sess_reply(struct ave_dev *ave, const u8 *msg, u32 len); +int ave_sess_input(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *src, + dma_addr_t *input); +int ave_sess_output(struct ave_ctx *ctx, unsigned int slot, struct vb2_v4l2_buffer *dst, + bool *keyframe); +void ave_sess_prepare_slot(struct ave_session *sess, unsigned int slot); +bool ave_sess_stalled(struct ave_session *sess); +bool ave_sess_starved(struct ave_session *sess); + +/* ave-image.c */ +void ave_image_start(u8 *cmd, size_t len, enum ave_codec codec, bool reorder_off, + bool hevc_low_delay); +void ave_image_process(u8 *cmd, size_t len, enum ave_codec codec); + +/* ave-drv.c */ +void ave_debugfs_save(struct ave_dev *ave, enum ave_codec codec, enum ave_image kind, + const void *cmd, size_t len); + +/* ave-v4l2.c */ +void ave_v4l2_suspend(struct ave_dev *ave); +int ave_v4l2_register(struct ave_dev *ave); +void ave_v4l2_unregister(struct ave_dev *ave); +enum ave_codec ave_ctx_codec(struct ave_ctx *ctx); + +#endif