From be1528aa91837aa272fcb1325e3c51bbf0340086 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:06:15 +1100 Subject: [PATCH 01/11] media: apple: isp: add IMX958 P010 preview and config discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Port the processed config-5 preview command sequence from our m1n1 J616s work, with contiguous V4L2 P010 strides and pool geometry. Query all advertised H17 sensor configs and retain the board defaults while appending consistent output modes. Keep NV12 the default and scope the new path to T6040/IMX958. Retire output and metadata leases after the report ACK, restart an empty P010 queue on requeue, bound ring servicing and latch malformed or timed-out command replies. Add command-byte, geometry, ownership and ring-wrap coverage. These source corrections still require sustained capture validation on 26.6.2. Builds on Jonathan Goyvaerts’ T6040 camera support in PR #384. Protocol reference: our m1n1 PR #20 at 9be380c8a5 and Linux PR #231 at a4e7ee86bc5e. No firmware payload is included. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/drivers/apple-isp-j616s.rst | 258 +++++++++++++++ .../driver-api/media/drivers/index.rst | 1 + drivers/media/platform/apple/isp/Kconfig | 5 +- .../media/platform/apple/isp/isp-cam-test.c | 295 ++++++++++++++++++ drivers/media/platform/apple/isp/isp-cam.c | 141 ++++++++- drivers/media/platform/apple/isp/isp-cmd.c | 60 +++- drivers/media/platform/apple/isp/isp-cmd.h | 14 + drivers/media/platform/apple/isp/isp-drv.c | 1 + drivers/media/platform/apple/isp/isp-drv.h | 55 ++++ drivers/media/platform/apple/isp/isp-fw.c | 1 + .../media/platform/apple/isp/isp-ipc-test.c | 27 ++ drivers/media/platform/apple/isp/isp-ipc.c | 28 +- .../media/platform/apple/isp/isp-v4l2-test.c | 228 ++++++++++++++ drivers/media/platform/apple/isp/isp-v4l2.c | 229 ++++++++++---- drivers/media/platform/apple/isp/isp-v4l2.h | 1 + 15 files changed, 1265 insertions(+), 79 deletions(-) create mode 100644 Documentation/driver-api/media/drivers/apple-isp-j616s.rst create mode 100644 drivers/media/platform/apple/isp/isp-cam-test.c diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst new file mode 100644 index 0000000000000..7413c0e866460 --- /dev/null +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -0,0 +1,258 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +J616s ISP modes and capture protocol +================================== + +Sources and scope +----------------- + +The source reference is Ryan's m1n1 PR 20, branch ``j616s`` at +``9be380c8a5a94609fdbe62d693c7b2121478ad8f``: + +* ``proxyclient/m1n1/fw/isp/isp_sensor.py`` discovers IMX958 and queries + ``CH_CAMERA_CONFIG_GET`` for every configuration reported by ``CH_INFO_GET``. +* ``isp_capture.py:T6040PreviewConfiguration`` contains the successful + processed capture command sequence. ``isp_pixels.py`` decodes its P010 + output as video-range BT.709. +* ``docs/j616s-bringup.md`` records all 13 configuration queries, completed + 1280x720 P010 captures (including an 80-frame ring-wrap trial) and a later + single completed 1920x1080 P010 frame. Intermittent startup/streaming stalls + remain unresolved. Neither the requested AE limits nor the host capture + rate is a measured sensor FPS. + +Both processed output sizes use firmware configuration **5**, whose reported +input dimensions are 1920x2160. The branch does not establish that this input +height is an independently deliverable 2160-row colour image, nor that config +5 is the highest-resolution configuration the sensor offers. It contains no +completed colour result from another configuration. RAW trials produced +stable pixels after stop but no completed RAW report; combined RAW/YUV was +refused at the allocation limit. No sensor full-resolution claim follows. + +Linux PR 231, branch ``j616s/linux-integration-next`` at +``a4e7ee86bc5eed8ff9dab61ef9656adfe000b207``, has the corresponding +``isp-profile.[ch]`` and IPC/buffer preparation. Its T6040 profile is pinned +to **25G76** and remains gated. It is a protocol reference, not an admission +profile for the current firmware. + +This driver change builds on the working T6040 H17 implementation in Linux +PR 384, now in ``integration/aurora``. That PR reports 26.6.x firmware capture +at 1920x1080 NV12 and use of config-5 1080p/720p presets. P010 and additional +firmware-derived output modes still need capture validation on 26.6.2. +Only our source and synthetic protocol tests are included; no Apple binary, +setfile contents, calibration table or firmware image is added. + +Mode and format selection +------------------------- + +On T6040/IMX958, probe queries every advertised configuration, bounded at 64, +using the 0x120-byte H17 ``CH_CAMERA_CONFIG_GET`` exchange. The complete list +is available through dynamic debug at probe and cached for +``VIDIOC_LOG_STATUS`` (``v4l2-ctl --log-status``), without sending commands +during a stream. The log includes +input/output dimensions and the sensor clock word; that word is not FPS. + +The board's existing config-5 1920x1080 and 1280x720 presets stay first and +NV12 stays the default. Additional distinct output sizes are appended only +when the reply's 16-bit and repeated 32-bit dimensions agree, the dimensions +are nonzero, output is even and within input, and output is at most 16384 +in either dimension. The full reported output becomes the crop and output +size for its own config index. Input dimensions alone never add a mode. +Thus a larger consistent firmware output can be selected by ``S_FMT`` and +listed by ``ENUM_FRAMESIZES``; no invented full-resolution preset is added. +These consistency checks cannot establish image quality, supported FPS or +whether a configuration needs additional sensor-specific commands. + +``V4L2_PIX_FMT_P010`` is offered only for T6040/IMX958. The config-5 +1280x720 and 1920x1080 processed previews are enabled when +the runtime input geometry matches the source profile. An unspecified P010 +size selects 1280x720 first. Larger P010 modes require a consistent firmware +output reply. P010 uses two little-endian 16-bit component planes with ten useful +MSB-aligned bits, and equal 64-byte-aligned Y/UV strides. UV immediately +follows Y, as required for contiguous P010. The optional multi-planar V4L2 +API also reports P010 as **one memory plane** containing both component +planes. The firmware still receives two component addresses. + +At 1920x1080, stride is 3840, Y is 4147200 bytes, UV is 2073600 bytes and +``sizeimage`` is 6220800. At 1280x720, stride is 2560 and ``sizeimage`` is +2764800. The proxy's independently page-aligned plane extents were +0x3f8000/0x1fc000 at 1080p and 0x1c4000/0xe4000 at 720p; inserting those +inter-plane gaps into a V4L2 contiguous P010 buffer would misdescribe it. +Linux passes its actual contiguous extents to the rendered pool instead. + +P010 uses the proxy's eight metadata slots (0x8000 bytes each) and two +rendered slots. The queue limits userspace to two rendered buffers, matching +the pool capacity. Completion matches both zero-extended low-32-bit plane +addresses, plane count, pool 9 and a fresh 64-bit submission tag; an old tag +cannot release a new lease. Buffer completion is published to videobuf2 +only after the firmware report ACK and doorbell have been published. +NV12 pool sizes and the T8140/Neo formats, presets and command path retain +their existing behavior. + +Endpoints +--------- + +These are the seven shared-memory ISP channels, not AOP/RTKit endpoints. +The table records the 25G76 source fixture. The current driver negotiates +channel names, types, sources, counts and 64-bit addresses on each boot; +it does not impose the captured ring addresses or counts on 26.6.2. + +================= ==== ====== ===== ======================================= +Channel Type Source Slots Purpose +================= ==== ====== ===== ======================================= +TERMINAL 2 0 768 Firmware log reports / terminal ACKs +IO 0 1 8 Host commands and in-place replies +DEBUG 0 1 8 Discovered; no capture commands sent +BUF_H2T 0 2 64 Host metadata/output descriptor batches +BUF_T2H 1 3 64 Firmware buffer completion/return reports +SHAREDMALLOC 1 3 8 Firmware allocation/free requests +IO_T2H 1 3 8 Discovered; no capture handler installed +================= ==== ====== ===== ======================================= + +A source word selects a doorbell bit, so IO and DEBUG share bit 1; the +three reply channels share bit 3. Each channel descriptor is 0x100 bytes +with a u64 ring address at 0x50. Each slot is 0x40 bytes / eight u64 words. +Publish the payload before arg0, which transfers slot ownership. The +existing IPC parser bounds rings against the allocation, descriptor table, +boot/command storage and other rings. The threaded IRQ services allocations, +terminal messages and buffer reports while a command waits for its ACK. + +Command stream +-------------- + +IO commands have a zero/status u32 followed by the opcode u32, then the +parameters below. Sizes include that eight-byte header. ``ch`` is zero. + +================ =================================== ==== =================== +Opcode Command Size Parameters +================ =================================== ==== =================== +0x0003 CONFIG_GET 0x1c Camera channel count +0x3206 SET_DSID_CLR_MULTI_BC_REG_BASE 0x50 8 slots, range 0x2fc +0x0000 START 0x0c Proxy 0x11, Linux 0 +0x010d CH_INFO_GET 0x190 ch +0x0106 CH_CAMERA_CONFIG_GET 0x120 ch, index (all) +0x0024 FLICKER_SENSOR_SET 0x0c 0 +0x013b CH_SBS_ENABLE 0x10 ch, 1 +0x0107 CH_CAMERA_CONFIG_SELECT 0x10 ch, selected config +0x010e CH_BUFFER_RECYCLE_MODE_SET 0x10 ch, 1 (empty only) +0x010f CH_BUFFER_RECYCLE_START 0x0c ch +0x080c CH_CROP_SCL1_SET 0x1c ch, x, y, width, height +0x0b09 CH_OUTPUT_CONFIG_SCL1_SET 0x38 ch, output geometry +0x0208 CH_AE_FRAME_RATE_MAX_SET 0x10 ch, fps * 256 +0x020a CH_AE_FRAME_RATE_MIN_SET 0x10 ch, min_fps * 256 +0x0117 CH_BUFFER_POOL_CONFIG_SET (meta) 0x9c ch, pool 0, count 8 +0x0117 CH_BUFFER_POOL_CONFIG_SET (output) 0x9c ch, pool 9, count 2 +0x0125 CH_LOCAL_RAW_BUFFER_ENABLE 0x10 ch, 1 +0x0b0d CH_PREVIEW_STREAM_SET 0x10 ch, 1 +0x0138 CH_MASTER_SLAVE_SYNC_MODE_SET 0x10 ch, 0 +0x0100 CH_START 0x0c ch +0x0101 CH_STOP 0x0c ch +0x0104 CH_BUFFER_RETURN 0x0c ch +================ =================================== ==== =================== + +The DSID bases are 0x2201d4000, 0x2221d4000, 0x2241d4000, 0x2261d4000, +then four U64_MAX sentinels. Linux retains its existing boot-time PMP and +logging commands in addition to discovery; it does not replay the proxy's +boot allocations or private power orchestration. + +For config 5, crop is (0, 0, output width, output height). The output command +parameters after ch are width, height, Rec.709=1, **format=0x12**, Y stride, +UV stride, 0, 0, height, 0, width. NV12 continues to use format 0. +Metadata pool sizes/strides are 0x8000, one data block; the output pool +contains actual Y/UV sizes and strides, two data blocks, no compression. + +Before CH_START, BUF_H2T submits metadata and output buffers as a 16-byte +header (type 1, count) and 64-byte descriptors. Each descriptor contains +four u64 addresses, four u32 flags (0x40000000 for each used plane), +u32 plane count, u32 pool and u64 tag. BUF_T2H returns completion reports; +render status 0x10000000 means a completed frame. Submission status is +0x30000000 and return ACK status is 0x80000000. Requeue only a returned lease. +CH_STOP and CH_BUFFER_RETURN retire the stream; resident firmware remains +running for the existing reopen path. + +The proxy used AE max/min 7680/3840 (30/15 fps). Linux retains its V4L2 rate +and exposure-priority controls: the default holds 30/30, and enabling +``exposure_auto_priority`` allows 30/15. These are requested limits, not FPS +measurements. P010 follows the minimal proxy sequence without adding +setfile-load, CNR, MBNR, SIF-format or temporal-filter commands; NV12 keeps +its existing calibrated path. A previously loaded setfile may still be +resident when switching streams. + +26.6.2 comparison +----------------- + +The 25G76 firmware UUID differs from the 26.6.2 image. Matching struct sizes +is source-level ABI evidence, not a new live qualification of P010. + +* Boot block: both use 0x290 bytes, IPC address at 0x08, shared base/size at + 0x10/0x18, extra heap at 0x20/0x28, platform at 0x30, queue offset+1 at + 0x50, 0x40 at 0x68, camera scheme at 0xd0 and the no-optical-card flag + at 0x120. The current H17 path checks descriptor flag bit 1. +* Memory: 25G76 pinned TEXT/DATA sizes 0xca8000/0x1344000, remap 1<<40, + shared base 0x1fec000, and a 256 MiB shared aperture. Linux uses the + bootloader's current TEXT/DATA/main-memory reservations, backs the missing + main-memory tail, and retains T6040's **0xde600000** shared-size descriptor. + No pinned segment size, heap request, IOVA or firmware hash is ported. +* IPC: 0x100-byte descriptors, 0x40-byte slots, 64-bit addresses and the + seven names agree. Counts and addresses remain negotiated, with the + existing overlap checks retained. The driver uses its three-DART IOMMU + domain and bootloader address-filter setup, not proxy DART table writes. +* Commands: DSID's 0x50-byte eight-slot layout, + CH_INFO_GET's 0x190-byte record, scaler-1 crop/output, 0x9c-byte pools, + local RAW, preview, sync and channel start/stop layouts agree with the + current H17 builders. START is an important parameter difference: the + proxy sends 0x11, while the working Linux H17 boot path sends 0. Linux + retains its current boot command; issuing START a second time after + selecting P010 would violate the proxy's guarded startup lifecycle. The new config query explicitly exchanges 0x120, + rather than the unused legacy 0xdc current-config helper. Header/channel/ + index checks and duplicated geometry checks guard decoding. The H17 + appended bytes remain uninterpreted. +* Metadata: CH_INFO_GET reports 0x4d00 on both paths. The proxy rounds its + metadata backing/pool to 0x8000; P010 does the same even on a 4 KiB host. + NV12 retains the current driver's 0x4d00 pool and 16-buffer capacity. + Neither path configures a T6040 capture-metadata pool or capture watchdog. +* Startup: Linux retains its once-per-boot resident-firmware admission, + current reset-vector check and H17 handshake; the 25G76 SHA/UUID admission + and recovery model from PR 231 are not substituted. Linux's command + timeout/IRQ service remain in use; the proxy's 30-second inactivity / + 40-second wall startup budget is not copied. + +Validation on hardware must record the current config list first, then +exercise both P010 presets, requeue, stop/reopen and a longer run. If a larger +firmware output appears, test its format and command requirements separately. +Do not label input dimensions, a successful query, or a requested rate as a +completed full-resolution stream or measured FPS. + +Open qualification and performance checks +----------------------------------------- + +``tools/testing/selftests/media/apple-isp-stress.c`` exercises long runs, +repeated start/stop and format/size switches through MMAP. It logs completed +frames, delivered FPS, maximum inter-frame gap, sequence gaps and buffer errors. +Use the established nominal rate as its explicit rate argument. A driver-generated +sequence is not a firmware frame counter, so timing and sequence checks alone +cannot prove the sensor dropped no frames internally. + +The P010 path requeues even the last returned lease into an otherwise empty +queue, publishes completion ACKs before releasing output or metadata leases, +acquires message validity before reading payload and bounds each IRQ ring drain +to one ring traversal so terminal/allocation traffic cannot starve completions. +Command replies must match the requested reply length, zero status and original +header. A timeout or malformed ACK latches failure; no late ACK can authorize +another command. The existing forced coprocessor stop runs before uncertain +leases are released. Resident firmware then needs a reboot before another capture. +These corrections address source-visible stalls/races. They are not evidence +that the intermittent stalls observed by the USB proxy have been reproduced +or cured on the current firmware. + +No verified IMX958 RAW Bayer packing/order, RAW completion path or nominal FPS +for every configuration is present in the public source. The recorded RAW trial +reads stable memory after stop but receives no completed RAW report. A Bayer +FOURCC cannot be chosen from that evidence. The RAW format and an exhaustive +per-config FPS table remain open pending the original capture evidence and +runtime command replies; neither is inferred from the P010 output or input height. +The 30/15 values in the preview are AE requests, not per-mode nominal measurements. + +The T6040 device tree's current fixed TEXT/DATA reservations should eventually +come from a bootloader handoff describing the loaded image and mapping lifetimes. +They are firmware-build-specific. This change does not implement that handoff +or replace those reservations with 25G76 addresses. diff --git a/Documentation/driver-api/media/drivers/index.rst b/Documentation/driver-api/media/drivers/index.rst index 7f6f3dcd5c90c..f1ead3c9e389f 100644 --- a/Documentation/driver-api/media/drivers/index.rst +++ b/Documentation/driver-api/media/drivers/index.rst @@ -12,6 +12,7 @@ Video4Linux (V4L) drivers .. toctree:: :maxdepth: 5 + apple-isp-j616s bttv-devel cx2341x-devel cx88-devel diff --git a/drivers/media/platform/apple/isp/Kconfig b/drivers/media/platform/apple/isp/Kconfig index b51d19ee115de..7a797a1a7e1a3 100644 --- a/drivers/media/platform/apple/isp/Kconfig +++ b/drivers/media/platform/apple/isp/Kconfig @@ -24,8 +24,9 @@ config VIDEO_APPLE_ISP_KUNIT_TEST depends on KUNIT=y || VIDEO_APPLE_ISP=m default KUNIT_ALL_TESTS help - Builds unit tests for the videobuf2 queue set-up and for the - shared memory the firmware frees into the Apple ISP driver. They + Builds unit tests for queue setup, P010 layouts and buffer identity, + firmware sensor geometry, the capture command stream and shared + memory retirement. The command tests stub the IPC transport and need no hardware. Only useful for kernel developers running KUnit. If unsure, say N. diff --git a/drivers/media/platform/apple/isp/isp-cam-test.c b/drivers/media/platform/apple/isp/isp-cam-test.c new file mode 100644 index 0000000000000..aef545b27c051 --- /dev/null +++ b/drivers/media/platform/apple/isp/isp-cam-test.c @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Synthetic geometry checks: no camera firmware or setfile data. */ +#include +#include +#include +#include "isp-ipc.h" + +static void isp_test_sensor_geometry(struct kunit *test) +{ + struct cmd_ch_camera_config config = { + .in_width = 1920, .in_height = 2160, + .out_width = 1920, .out_height = 1080, + .in_width2 = 1920, .in_height2 = 2160, + .out_width2 = 1920, .out_height2 = 1080, + }; + + KUNIT_EXPECT_TRUE(test, isp_sensor_config_usable(&config)); + /* A future larger output is accepted only when both descriptions agree. */ + config.in_width = 4096; + config.in_width2 = 4096; + config.in_height = 3072; + config.in_height2 = 3072; + config.out_width = 4096; + config.out_width2 = 4096; + config.out_height = 3072; + config.out_height2 = 3072; + KUNIT_EXPECT_TRUE(test, isp_sensor_config_usable(&config)); + config.out_height2 = 2160; + KUNIT_EXPECT_FALSE(test, isp_sensor_config_usable(&config)); + config.out_height2 = 3072; + config.in_height2 = 2160; + KUNIT_EXPECT_FALSE(test, isp_sensor_config_usable(&config)); + config.in_height2 = 3072; + config.out_width = 4097; + config.out_width2 = 4097; + KUNIT_EXPECT_FALSE(test, isp_sensor_config_usable(&config)); + config.out_width = 0; + config.out_width2 = 0; + KUNIT_EXPECT_FALSE(test, isp_sensor_config_usable(&config)); + config.out_width = 8192; + config.out_width2 = 8192; + KUNIT_EXPECT_FALSE(test, isp_sensor_config_usable(&config)); +} + +struct isp_test_command { + u32 words[39]; + u32 size; +}; + +/* Derived from the proxy command fixture, with contiguous V4L2 plane sizes. */ +static const struct isp_test_command isp_test_preview_commands[] = { + {{0, 0x24, 0}, 0xc}, + {{0, 0x13b, 0, 1}, 0x10}, + {{0, 0x107, 0, 5}, 0x10}, + {{0, 0x10e, 0, 1}, 0x10}, + {{0, 0x10f, 0}, 0xc}, + {{0, 0x80c, 0, 0, 0, 1280, 720}, 0x1c}, + {{0, 0xb09, 0, 1280, 720, 1, 0x12, 2560, 2560, 0, 0, 720, 0, 1280}, 0x38}, + {{0, 0x208, 0, 7680}, 0x10}, + {{0, 0x20a, 0, 3840}, 0x10}, + {{[1] = 0x117, [3] = 8 << 16, [4] = 0x8000, [5] = 0x8000, [37] = 1}, 0x9c}, + {{[1] = 0x117, [3] = 9 | 2 << 16, [4] = 1843200, [5] = 2560, + [12] = 921600, [13] = 2560, [37] = 2}, 0x9c}, + {{0, 0x125, 0, 1}, 0x10}, + {{0, 0xb0d, 0, 1}, 0x10}, + {{0, 0x138, 0, 0}, 0x10}, +}; + +struct isp_test_command_trace { + unsigned int count; + unsigned int fail_at; +}; + +static int isp_test_send(struct apple_isp *isp, void *args, u32 insize, + u32 outsize, int timeout) +{ + struct kunit *test = kunit_get_current_test(); + struct isp_test_command_trace *trace = test->priv; + const struct isp_test_command *expected; + + if (trace->count >= ARRAY_SIZE(isp_test_preview_commands)) { + KUNIT_FAIL(test, "unexpected extra command"); + return -EIO; + } + expected = &isp_test_preview_commands[trace->count++]; + KUNIT_EXPECT_EQ(test, insize, expected->size); + KUNIT_EXPECT_EQ(test, outsize, expected->words[1] == 0x117 ? insize : 0); + if (insize == expected->size) + KUNIT_EXPECT_MEMEQ(test, args, expected->words, insize); + if (trace->count == trace->fail_at) + return -ETIME; + return 0; +} + +static void isp_test_free_controls(void *ptr) +{ + v4l2_ctrl_handler_free(ptr); +} + +static struct apple_isp *isp_test_capture_alloc(struct kunit *test) +{ + static const struct apple_isp_hw hw = { + .gen = ISP_GEN_T6040, .fw_abi = ISP_FW_ABI_H17, + .scl1 = true, .meta_size = ISP_META_SIZE_T8140, + }; + static struct isp_preset preset = { + .index = 5, .input_dim = {1920, 2160}, + .output_dim = {1280, 720}, .crop_size = {1280, 720}, + }; + struct apple_isp *isp = kunit_kzalloc(test, sizeof(*isp), GFP_KERNEL); + struct isp_format *fmt; + int err; + + KUNIT_ASSERT_NOT_NULL(test, isp); + isp->dev = kunit_device_register(test, "apple-isp-camera-test"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, isp->dev); + isp->hw = &hw; + isp->frame_rate = 30; + fmt = isp_get_current_format(isp); + fmt->version = 0x958; + fmt->pixelformat = V4L2_PIX_FMT_P010; + fmt->preset = &preset; + fmt->strides[0] = 2560; + fmt->strides[1] = 2560; + fmt->plane_size[0] = 1843200; + fmt->plane_size[1] = 921600; + v4l2_ctrl_handler_init(&isp->ctrl_handler, 1); + err = kunit_add_action_or_reset(test, isp_test_free_controls, &isp->ctrl_handler); + KUNIT_ASSERT_EQ(test, err, 0); + isp->exposure_priority = + v4l2_ctrl_new_std(&isp->ctrl_handler, NULL, + V4L2_CID_EXPOSURE_AUTO_PRIORITY, 0, 1, 1, 1); + KUNIT_ASSERT_EQ(test, isp->ctrl_handler.error, 0); + kunit_activate_static_stub(test, isp_cmd_send, isp_test_send); + return isp; +} + +static void isp_test_preview_stream(struct kunit *test) +{ + struct apple_isp *isp = isp_test_capture_alloc(test); + struct isp_test_command_trace trace = {}; + + test->priv = &trace; + KUNIT_EXPECT_EQ(test, isp_ch_configure_capture(isp, 0), 0); + KUNIT_EXPECT_EQ(test, trace.count, ARRAY_SIZE(isp_test_preview_commands)); +} + +static void isp_test_preview_command_failures(struct kunit *test) +{ + struct apple_isp *isp = isp_test_capture_alloc(test); + struct isp_test_command_trace trace; + + test->priv = &trace; + for (unsigned int i = 1; i <= ARRAY_SIZE(isp_test_preview_commands); i++) { + trace = (struct isp_test_command_trace){ .fail_at = i }; + KUNIT_EXPECT_EQ(test, isp_ch_configure_capture(isp, 0), -ETIME); + KUNIT_EXPECT_EQ(test, trace.count, i); + } +} + +static int isp_test_config_reply(struct apple_isp *isp, void *args, u32 insize, + u32 outsize, int timeout) +{ + struct kunit *test = kunit_get_current_test(); + u32 *count = test->priv; + struct cmd_ch_camera_config_h17 *reply = isp->cmd_virt; + struct cmd_ch_camera_config *config = &reply->config; + + KUNIT_EXPECT_EQ(test, insize, 0x120); + KUNIT_EXPECT_EQ(test, outsize, 0x120); + memcpy(reply, args, sizeof(*reply)); + KUNIT_EXPECT_EQ(test, config->preset, (*count)++); + config->in_width = 1920; + config->in_height = 2160; + config->out_width = 1920; + config->out_height = 1080; + if (config->preset == 1) { + config->in_width = 4096; + config->in_height = 3072; + config->out_width = 4096; + config->out_height = 3072; + } + config->in_width2 = config->in_width; + config->in_height2 = config->in_height; + config->out_width2 = config->out_width; + config->out_height2 = config->out_height; + if (config->preset == 2) + config->out_height2 = 2160; + return 0; +} + +static void isp_test_sensor_config_list(struct kunit *test) +{ + struct apple_isp *isp = isp_test_capture_alloc(test); + struct isp_preset presets[] = { + { .index = 5, .input_dim = {1920, 2160}, + .output_dim = {1920, 1080}, .crop_size = {1920, 1080} }, + { .index = 5, .input_dim = {1920, 2160}, + .output_dim = {1280, 720}, .crop_size = {1280, 720} }, + }; + u32 count = 0; + + isp->cmd_virt = kunit_kzalloc(test, 0x120, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); + isp->presets = presets; + isp->num_presets = ARRAY_SIZE(presets); + test->priv = &count; + kunit_activate_static_stub(test, isp_cmd_send, isp_test_config_reply); + KUNIT_ASSERT_EQ(test, isp_ch_cache_sensor_configs(isp, 0, 6), 0); + KUNIT_EXPECT_EQ(test, count, 6); + KUNIT_EXPECT_EQ(test, isp->num_sensor_configs, 6); + KUNIT_EXPECT_EQ(test, isp->num_presets, 3); + KUNIT_EXPECT_EQ(test, isp->presets[0].index, 5); + KUNIT_EXPECT_EQ(test, isp->presets[0].output_dim.y, 1080); + KUNIT_EXPECT_TRUE(test, isp->presets[1].p010); + KUNIT_EXPECT_TRUE(test, isp->presets[2].p010); + KUNIT_EXPECT_EQ(test, isp->presets[2].index, 1); + KUNIT_EXPECT_EQ(test, isp->presets[2].output_dim.x, 4096); + KUNIT_EXPECT_EQ(test, isp->presets[2].crop_size.y, 3072); + KUNIT_EXPECT_FALSE(test, isp->sensor_configs[2].usable); + KUNIT_EXPECT_EQ(test, isp_ch_cache_sensor_configs(isp, 0, 0), -EINVAL); + KUNIT_EXPECT_EQ(test, isp_ch_cache_sensor_configs(isp, 0, 65), -EINVAL); + KUNIT_EXPECT_EQ(test, count, 6); +} + +struct isp_test_ack_state { + unsigned int fault, calls; +}; + +static int isp_test_ack_reply(struct apple_isp *isp, struct isp_channel *chan, + unsigned long timeout) +{ + struct kunit *test = kunit_get_current_test(); + struct isp_test_ack_state *state = test->priv; + + state->calls++; + chan->rsp = (struct isp_message){ .arg0 = chan->req.arg0 | ISP_IPC_FLAG_ACK, + .arg1 = chan->req.arg2 }; + switch (state->fault) { + case 1: + chan->rsp.arg2 = 1; + break; + case 2: + put_unaligned_le32(1, isp->cmd_virt); + break; + case 3: + chan->rsp.arg1 += 4; + break; + case 4: + return -ETIME; + } + return 0; +} + +static void isp_test_command_ack_failure_latches(struct kunit *test) +{ + struct apple_isp *isp = isp_test_capture_alloc(test); + struct isp_test_ack_state state = {}; + + isp->cmd_virt = kunit_kzalloc(test, 0x400, GFP_KERNEL); + isp->chan_io = kunit_kzalloc(test, sizeof(*isp->chan_io), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); + KUNIT_ASSERT_NOT_NULL(test, isp->chan_io); + isp->chan_io->name = "IO"; + test->priv = &state; + kunit_deactivate_static_stub(test, isp_cmd_send); + kunit_activate_static_stub(test, ipc_chan_send, isp_test_ack_reply); + KUNIT_EXPECT_EQ(test, isp_cmd_ch_camera_config_select(isp, 0, 5), 0); + for (state.fault = 1; state.fault <= 4; state.fault++) { + isp->ipc_failed = false; + state.calls = 0; + KUNIT_EXPECT_EQ(test, isp_cmd_ch_camera_config_select(isp, 0, 5), + state.fault == 4 ? -ETIME : -EIO); + KUNIT_EXPECT_TRUE(test, isp->ipc_failed); + /* A late ACK cannot admit another command on an uncertain slot. */ + KUNIT_EXPECT_EQ(test, isp_cmd_ch_camera_config_select(isp, 0, 5), -EIO); + KUNIT_EXPECT_EQ(test, state.calls, 1); + } +} + +static struct kunit_case isp_cam_test_cases[] = { + KUNIT_CASE(isp_test_command_ack_failure_latches), + KUNIT_CASE(isp_test_sensor_geometry), + KUNIT_CASE(isp_test_sensor_config_list), + KUNIT_CASE(isp_test_preview_stream), + KUNIT_CASE(isp_test_preview_command_failures), + {} +}; + +static struct kunit_suite isp_cam_test_suite = { + .name = "apple-isp-camera", + .test_cases = isp_cam_test_cases, +}; + +kunit_test_suite(isp_cam_test_suite); diff --git a/drivers/media/platform/apple/isp/isp-cam.c b/drivers/media/platform/apple/isp/isp-cam.c index a266c60609614..59f19e571bb64 100644 --- a/drivers/media/platform/apple/isp/isp-cam.c +++ b/drivers/media/platform/apple/isp/isp-cam.c @@ -10,8 +10,6 @@ #include "isp-iommu.h" #include "isp-regs.h" -#define ISP_MAX_PRESETS 32 - struct isp_setfile { u32 version; u32 magic; @@ -217,6 +215,110 @@ static int isp_check_presets(struct apple_isp *isp, u32 num_fw_presets) return 0; } +/* Do not mistake stacked sensor input rows for a processed output mode. */ +static bool isp_sensor_config_usable(const struct cmd_ch_camera_config *config) +{ + return config->in_width && config->in_height && + config->out_width && config->out_height && + config->out_width <= 16384 && config->out_height <= 16384 && + !(config->out_width & 1) && !(config->out_height & 1) && + config->out_width <= config->in_width && + config->out_height <= config->in_height && + config->in_width == config->in_width2 && + config->in_height == config->in_height2 && + config->out_width == config->out_width2 && + config->out_height == config->out_height2; +} + +static int isp_ch_cache_sensor_configs(struct apple_isp *isp, u32 ch, u32 count) +{ + struct cmd_ch_camera_config_h17 args; + struct isp_preset *presets; + unsigned int num = isp->num_presets; + int err; + + /* The extended reply is established only for the IMX958 T6040 path. */ + if (!isp_supports_p010(isp)) + return 0; + if (!count || count > ISP_MAX_SENSOR_CONFIGS) + return -EINVAL; + + presets = devm_kcalloc(isp->dev, num + count, sizeof(*presets), GFP_KERNEL); + if (!presets) + return -ENOMEM; + memcpy(presets, isp->presets, num * sizeof(*presets)); + + for (u32 i = 0; i < count; i++) { + struct cmd_ch_camera_config *config = &args.config; + struct isp_sensor_config *cached = &isp->sensor_configs[i]; + bool duplicate = false; + + err = isp_cmd_ch_camera_config_get(isp, ch, i, &args); + if (err) + return err; + if (config->opcode != (u64)CISP_CMD_CH_CAMERA_CONFIG_GET << 32 || + config->chan != ch || config->preset != i) + return -EIO; + + cached->input_dim = (struct coord){config->in_width, config->in_height}; + cached->output_dim = (struct coord){config->out_width, config->out_height}; + cached->sensor_clk = config->sensor_clk; + cached->usable = isp_sensor_config_usable(config); + dev_dbg(isp->dev, "sensor config %u: input %ux%u output %ux%u clock %u%s\n", + i, config->in_width, config->in_height, + config->out_width, config->out_height, config->sensor_clk, + cached->usable ? "" : " (unverified geometry)"); + print_hex_dump_debug("apple-isp config: ", DUMP_PREFIX_OFFSET, + 16, 4, &args, sizeof(args), false); + /* Both processed preview sizes are recorded in the proxy source. */ + for (unsigned int j = 0; j < num; j++) { + struct isp_preset *preset = &presets[j]; + bool preview = i == 5 && config->in_width == 1920 && + config->in_height == 2160 && + ((preset->output_dim.x == 1280 && + preset->output_dim.y == 720) || + (preset->output_dim.x == 1920 && + preset->output_dim.y == 1080)); + bool reported = cached->usable && + preset->output_dim.x == config->out_width && + preset->output_dim.y == config->out_height; + + if (preset->index == i && + preset->input_dim.x == config->in_width && + preset->input_dim.y == config->in_height && + !preset->crop_offset.x && !preset->crop_offset.y && + preset->crop_size.x == preset->output_dim.x && + preset->crop_size.y == preset->output_dim.y) + preset->p010 = preview || reported; + } + if (!cached->usable) + continue; + + /* Keep the board's config-5 presets, including the default, first. */ + for (unsigned int j = 0; j < num; j++) { + if (presets[j].output_dim.x == config->out_width && + presets[j].output_dim.y == config->out_height) { + duplicate = true; + break; + } + } + if (duplicate) + continue; + + presets[num++] = (struct isp_preset){ + .index = i, + .input_dim = cached->input_dim, + .output_dim = cached->output_dim, + .crop_size = cached->output_dim, + .p010 = true, + }; + } + isp->num_sensor_configs = count; + isp->presets = presets; + isp->num_presets = num; + return 0; +} + static int isp_ch_cache_sensor_info(struct apple_isp *isp, u32 ch) { struct isp_format *fmt = isp_get_format(isp, ch); @@ -264,6 +366,8 @@ static int isp_ch_cache_sensor_info(struct apple_isp *isp, u32 ch) } err = isp_check_presets(isp, args->num_presets); + if (!err) + err = isp_ch_cache_sensor_configs(isp, ch, args->num_presets); exit: kfree(args); @@ -475,7 +579,7 @@ static int isp_ch_configure_pools(struct apple_isp *isp, u32 ch) return 0; return isp_cmd_ch_buffer_pool_config_set_rendered(isp, ch, - ISP_MAX_BUFFERS, + isp_num_rendered(isp), fmt->plane_size[0], fmt->strides[0], fmt->plane_size[1], @@ -488,7 +592,13 @@ static int isp_ch_configure_capture(struct apple_isp *isp, u32 ch) bool h17 = isp->hw->fw_abi == ISP_FW_ABI_H17; int err; - isp_cmd_flicker_sensor_set(isp, 0); + err = isp_cmd_flicker_sensor_set(isp, 0); + if (isp_is_p010(isp) && err) + return err; + + /* The proxy-qualified P010 route uses the firmware defaults. */ + if (isp_is_p010(isp)) + goto select_config; /* * The setfile isn't requisite but then we don't get calibration. If @@ -504,6 +614,7 @@ static int isp_ch_configure_capture(struct apple_isp *isp, u32 ch) return err; } +select_config: err = isp_cmd_ch_sbs_enable(isp, ch, 1); if (err) return err; @@ -531,7 +642,8 @@ static int isp_ch_configure_capture(struct apple_isp *isp, u32 ch) err = isp_cmd_ch_output_config_set(isp, ch, fmt->preset->output_dim.x, fmt->preset->output_dim.y, fmt->strides, CISP_COLORSPACE_REC709, - CISP_OUTPUT_FORMAT_YUV_2PLANE); + isp_is_p010(isp) ? CISP_OUTPUT_FORMAT_P010 : + CISP_OUTPUT_FORMAT_YUV_2PLANE); if (err) return err; @@ -559,6 +671,9 @@ static int isp_ch_configure_capture(struct apple_isp *isp, u32 ch) if (err) return err; + if (isp_is_p010(isp)) + return isp_cmd_ch_master_slave_sync_mode_set(isp, ch, 0); + err = isp_cmd_ch_cnr_start(isp, ch); if (err) return err; @@ -633,7 +748,10 @@ int apple_isp_start_camera(struct apple_isp *isp) err = isp_configure_capture(isp); if (err) { dev_err(isp->dev, "failed to configure capture: %d\n", err); - apple_isp_firmware_shutdown(isp); + if (isp->ipc_failed) + apple_isp_firmware_halt(isp); + else + apple_isp_firmware_shutdown(isp); return err; } @@ -642,7 +760,10 @@ int apple_isp_start_camera(struct apple_isp *isp) void apple_isp_stop_camera(struct apple_isp *isp) { - apple_isp_firmware_shutdown(isp); + if (isp->ipc_failed) + apple_isp_firmware_halt(isp); + else + apple_isp_firmware_shutdown(isp); } /* @@ -714,9 +835,15 @@ int apple_isp_start_capture(struct apple_isp *isp) void apple_isp_stop_capture(struct apple_isp *isp) { + if (isp->ipc_failed || isp->fw_state == ISP_FW_DEAD) + return; isp_cmd_ch_stop(isp, 0); // TODO channel mask isp_cmd_ch_buffer_return(isp, isp->current_ch); /* ... until its buffers are back. */ apple_isp_wdt_stop(isp); } + +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_KUNIT_TEST) +#include "isp-cam-test.c" +#endif diff --git a/drivers/media/platform/apple/isp/isp-cmd.c b/drivers/media/platform/apple/isp/isp-cmd.c index fd7dcfd52899a..a525e68c230f5 100644 --- a/drivers/media/platform/apple/isp/isp-cmd.c +++ b/drivers/media/platform/apple/isp/isp-cmd.c @@ -5,30 +5,54 @@ #include "isp-drv.h" #include "isp-iommu.h" #include "isp-ipc.h" +#include + +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_KUNIT_TEST) +#include +#endif #define CISP_OPCODE_SHIFT 32UL #define CISP_OPCODE(x) (((u64)(x)) << CISP_OPCODE_SHIFT) #define CISP_OPCODE_GET(x) (((u64)(x)) >> CISP_OPCODE_SHIFT) #define CISP_TIMEOUT msecs_to_jiffies(3000) -#define CISP_SEND_IN(x, a) (cisp_send((x), &(a), sizeof(a), 0, CISP_TIMEOUT)) -#define CISP_SEND_INOUT(x, a) (cisp_send((x), &(a), sizeof(a), sizeof(a), CISP_TIMEOUT)) +#define CISP_SEND_IN(x, a) (isp_cmd_send((x), &(a), sizeof(a), 0, CISP_TIMEOUT)) +#define CISP_SEND_INOUT(x, a) (isp_cmd_send((x), &(a), sizeof(a), sizeof(a), CISP_TIMEOUT)) #define CISP_SEND_OUT(x, a) (cisp_send_read((x), (a), sizeof(*a), sizeof(*a))) -#define CISP_POST_IN(x, a) (cisp_send((x), &(a), sizeof(a), 0, 0)) -#define CISP_POST_INOUT(x, a) (cisp_send((x), &(a), sizeof(a), sizeof(a), 0)) +#define CISP_POST_IN(x, a) (isp_cmd_send((x), &(a), sizeof(a), 0, 0)) +#define CISP_POST_INOUT(x, a) (isp_cmd_send((x), &(a), sizeof(a), sizeof(a), 0)) -static int cisp_send(struct apple_isp *isp, void *args, u32 insize, u32 outsize, int timeout) +int isp_cmd_send(struct apple_isp *isp, void *args, u32 insize, u32 outsize, int timeout) { - struct isp_channel *chan = isp->chan_io; - struct isp_message *req = &chan->req; + struct isp_channel *chan; + struct isp_message *req; int err; +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_KUNIT_TEST) + KUNIT_STATIC_STUB_REDIRECT(isp_cmd_send, isp, args, insize, outsize, timeout); +#endif + if (isp->ipc_failed) + return -EIO; + chan = isp->chan_io; + req = &chan->req; + req->arg0 = isp->cmd_iova; req->arg1 = insize; req->arg2 = outsize; memcpy(isp->cmd_virt, args, insize); err = ipc_chan_send(isp, chan, timeout); + if (isp_is_p010(isp)) { + struct isp_message *rsp = &chan->rsp; + + if (!err && timeout && + (rsp->arg1 != outsize || rsp->arg2 || rsp->arg3 || + rsp->arg4 || rsp->arg5 || rsp->arg6 || rsp->arg7 || + get_unaligned_le64(isp->cmd_virt) != get_unaligned_le64(args))) + err = -EIO; + if (err) + isp->ipc_failed = true; + } if (err) { u64 opcode; memcpy(&opcode, args, sizeof(opcode)); @@ -45,7 +69,7 @@ static int cisp_send_read(struct apple_isp *isp, void *args, u32 insize, u32 outsize) { /* TODO do I need to lock the iova space? */ - int err = cisp_send(isp, args, insize, outsize, CISP_TIMEOUT); + int err = isp_cmd_send(isp, args, insize, outsize, CISP_TIMEOUT); if (err) return err; @@ -165,7 +189,7 @@ int isp_cmd_set_dsid_clr_multi_bc(struct apple_isp *isp, const u64 *bases, return -EINVAL; memcpy(args.dsid_clr_base, bases, count * sizeof(*bases)); - return cisp_send(isp, &args, + return isp_cmd_send(isp, &args, offsetof(typeof(args), dsid_clr_base) + count * sizeof(*bases), 0, CISP_TIMEOUT); @@ -232,6 +256,8 @@ int isp_cmd_flicker_sensor_set(struct apple_isp *isp, u32 mode) .opcode = CISP_OPCODE(CISP_CMD_FLICKER_SENSOR_SET), .mode = mode, }; + if (isp_is_p010(isp)) + return CISP_SEND_IN(isp, args); return CISP_SEND_INOUT(isp, args); } @@ -260,6 +286,16 @@ int isp_cmd_ch_info_get(struct apple_isp *isp, u32 chan, return err; } +int isp_cmd_ch_camera_config_get(struct apple_isp *isp, u32 chan, u32 index, + struct cmd_ch_camera_config_h17 *args) +{ + memset(args, 0, sizeof(*args)); + args->config.opcode = CISP_OPCODE(CISP_CMD_CH_CAMERA_CONFIG_GET); + args->config.chan = chan; + args->config.preset = index; + return CISP_SEND_OUT(isp, args); +} + int isp_cmd_ch_camera_config_current_get(struct apple_isp *isp, u32 chan, struct cmd_ch_camera_config *args) { @@ -468,9 +504,9 @@ int isp_cmd_ch_buffer_pool_config_set(struct apple_isp *isp, u32 chan, u16 type) .opcode = CISP_OPCODE(CISP_CMD_CH_BUFFER_POOL_CONFIG_SET), .chan = chan, .type = type, - .count = ISP_MAX_BUFFERS, - .meta_size0 = size, - .meta_size1 = size, + .count = isp_num_meta(isp), + .meta_size0 = isp_is_p010(isp) ? 0x8000 : size, + .meta_size1 = isp_is_p010(isp) ? 0x8000 : size, .unk0 = 0, .unk1 = 0, .unk2 = 0, diff --git a/drivers/media/platform/apple/isp/isp-cmd.h b/drivers/media/platform/apple/isp/isp-cmd.h index 695c0a9f7ab70..a3167108b128e 100644 --- a/drivers/media/platform/apple/isp/isp-cmd.h +++ b/drivers/media/platform/apple/isp/isp-cmd.h @@ -124,6 +124,7 @@ #define CISP_COLORSPACE_REC709 0x1 #define CISP_OUTPUT_FORMAT_YUV_2PLANE 0x0 #define CISP_OUTPUT_FORMAT_YUV_1PLANE 0x1 +#define CISP_OUTPUT_FORMAT_P010 0x12 #define CISP_OUTPUT_FORMAT_RGB 0x2 #define CISP_BUFFER_RECYCLE_MODE_EMPTY_ONLY 0x1 @@ -255,6 +256,8 @@ struct cmd_flicker_sensor_set { } __packed; static_assert(sizeof(struct cmd_flicker_sensor_set) == 0xc); +int isp_cmd_send(struct apple_isp *isp, void *args, u32 insize, u32 outsize, + int timeout); int isp_cmd_start(struct apple_isp *isp, u32 mode); int isp_cmd_stop(struct apple_isp *isp, u32 mode); int isp_cmd_power_down(struct apple_isp *isp); @@ -380,6 +383,15 @@ struct cmd_ch_camera_config { } __packed; static_assert(sizeof(struct cmd_ch_camera_config) == 0xdc); +/* H17 appends 0x44 bytes to the common camera configuration record. */ +struct cmd_ch_camera_config_h17 { + struct cmd_ch_camera_config config; + u8 unknown[0x44]; +} __packed; +static_assert(sizeof(struct cmd_ch_camera_config_h17) == 0x120); +static_assert(offsetof(struct cmd_ch_camera_config, in_width) == 0x10); +static_assert(offsetof(struct cmd_ch_camera_config, out_width2) == 0xd4); + struct cmd_ch_camera_config_select { u64 opcode; u32 chan; @@ -516,6 +528,8 @@ int isp_cmd_ch_start(struct apple_isp *isp, u32 chan); int isp_cmd_ch_stop(struct apple_isp *isp, u32 chan); int isp_cmd_ch_info_get(struct apple_isp *isp, u32 chan, struct cmd_ch_info *args); +int isp_cmd_ch_camera_config_get(struct apple_isp *isp, u32 chan, u32 index, + struct cmd_ch_camera_config_h17 *args); int isp_cmd_ch_camera_config_current_get(struct apple_isp *isp, u32 chan, struct cmd_ch_camera_config *args); int isp_cmd_ch_camera_config_select(struct apple_isp *isp, u32 chan, diff --git a/drivers/media/platform/apple/isp/isp-drv.c b/drivers/media/platform/apple/isp/isp-drv.c index da9d459dd8bdb..04c50011abda1 100644 --- a/drivers/media/platform/apple/isp/isp-drv.c +++ b/drivers/media/platform/apple/isp/isp-drv.c @@ -538,6 +538,7 @@ static int apple_isp_probe(struct platform_device *pdev) mutex_init(&isp->gc_lock); INIT_LIST_HEAD(&isp->bufs_pending); INIT_LIST_HEAD(&isp->bufs_submitted); + INIT_LIST_HEAD(&isp->bufs_returned); isp->wq = alloc_workqueue("apple-isp-wq", WQ_UNBOUND, 0); if (!isp->wq) { dev_err(dev, "failed to create workqueue\n"); diff --git a/drivers/media/platform/apple/isp/isp-drv.h b/drivers/media/platform/apple/isp/isp-drv.h index ed3e48eef569e..e02d675f0e998 100644 --- a/drivers/media/platform/apple/isp/isp-drv.h +++ b/drivers/media/platform/apple/isp/isp-drv.h @@ -30,6 +30,7 @@ /* used to limit the user space buffers to the buffer_pool_config */ #define ISP_MAX_BUFFERS 16 +#define ISP_MAX_SENSOR_CONFIGS 64 /* capture metadata buffers, for firmware that has that pool */ #define ISP_CAPMETA_BUFFERS 8 @@ -139,6 +140,8 @@ struct isp_preset { struct coord output_dim; struct coord crop_offset; struct coord crop_size; + /* Processed P010 geometry established by preview or a config reply. */ + bool p010; }; struct apple_isp_hw { @@ -255,9 +258,17 @@ enum isp_sensor_id { ISP_VD56G0_6222_01, }; +struct isp_sensor_config { + struct coord input_dim; + struct coord output_dim; + u32 sensor_clk; + bool usable; +}; + struct isp_format { enum isp_sensor_id id; u32 version; + u32 pixelformat; struct isp_preset *preset; unsigned int num_planes; u32 strides[VB2_MAX_PLANES]; @@ -274,6 +285,9 @@ struct apple_isp { u32 cam_connections_scheme; struct isp_preset *presets; int num_presets; + /* Read-only sensor configuration replies, cached before video registration. */ + unsigned int num_sensor_configs; + struct isp_sensor_config sensor_configs[ISP_MAX_SENSOR_CONFIGS]; int num_channels; struct isp_format fmts[ISP_MAX_CHANNELS]; @@ -358,16 +372,25 @@ struct apple_isp { spinlock_t buf_lock; struct list_head bufs_pending; struct list_head bufs_submitted; + struct list_head bufs_returned; + u64 buffer_tag; + unsigned long meta_returned; + /* A timed-out command leaves slot ownership ambiguous until reboot. */ + bool ipc_failed; }; struct isp_chan_ops { int (*handle)(struct apple_isp *isp, struct isp_channel *chan); + /* Called after the response slot and its doorbell are published. */ + void (*complete)(struct apple_isp *isp); }; struct isp_buffer { struct vb2_v4l2_buffer vb; struct list_head link; struct isp_surf surfs[VB2_MAX_PLANES]; + u64 tag; + enum vb2_buffer_state return_state; }; #define to_isp_buffer(x) container_of((x), struct isp_buffer, vb) @@ -395,4 +418,36 @@ enum { ((isp)->hw->capture_meta_size ? ISP_CAPMETA_BUFFERS : 0) #define isp_get_current_format(isp) (isp_get_format(isp, isp->current_ch)) +static inline bool isp_supports_p010(struct apple_isp *isp) +{ + return isp->hw->gen == ISP_GEN_T6040 && + isp_get_current_format(isp)->version == 0x958; +} + +static inline bool isp_has_p010(struct apple_isp *isp) +{ + if (!isp_supports_p010(isp)) + return false; + for (int i = 0; i < isp->num_presets; i++) { + if (isp->presets[i].p010) + return true; + } + return false; +} + +static inline bool isp_is_p010(struct apple_isp *isp) +{ + return isp_get_current_format(isp)->pixelformat == V4L2_PIX_FMT_P010; +} + +static inline unsigned int isp_num_rendered(struct apple_isp *isp) +{ + return isp_is_p010(isp) ? 2 : ISP_MAX_BUFFERS; +} + +static inline unsigned int isp_num_meta(struct apple_isp *isp) +{ + return isp_is_p010(isp) ? 8 : ISP_MAX_BUFFERS; +} + #endif /* __ISP_DRV_H__ */ diff --git a/drivers/media/platform/apple/isp/isp-fw.c b/drivers/media/platform/apple/isp/isp-fw.c index 30ff704ef193f..d448225fceec5 100644 --- a/drivers/media/platform/apple/isp/isp-fw.c +++ b/drivers/media/platform/apple/isp/isp-fw.c @@ -212,6 +212,7 @@ static const struct isp_chan_ops sm_ops = { static const struct isp_chan_ops bt_ops = { .handle = ipc_bt_handle, + .complete = ipc_bt_complete, }; static irqreturn_t apple_isp_isr(int irq, void *dev) diff --git a/drivers/media/platform/apple/isp/isp-ipc-test.c b/drivers/media/platform/apple/isp/isp-ipc-test.c index 6521cecb4cfb6..efd24f6794a5a 100644 --- a/drivers/media/platform/apple/isp/isp-ipc-test.c +++ b/drivers/media/platform/apple/isp/isp-ipc-test.c @@ -144,7 +144,34 @@ static void isp_test_sm_free_log_bt(struct kunit *test) KUNIT_EXPECT_NULL(test, t->isp->bt_surf); } +static void isp_test_command_ring_wrap(struct kunit *test) +{ + struct isp_sm_test *t = isp_sm_test_alloc(test); + struct isp_channel *chan = t->chan; + struct isp_message *slots; + + chan->num = 8; + chan->virt = kunit_kcalloc(test, chan->num, sizeof(*slots), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, chan->virt); + slots = chan->virt; + chan->req.arg0 = 0x5000000; + isp_get_current_format(t->isp)->pixelformat = V4L2_PIX_FMT_P010; + for (unsigned int i = 0; i < 100000; i++) { + unsigned int cursor = chan->cursor; + + /* Re-publication must clear the old ACK before waiting on a wrap. */ + slots[cursor].arg0 = chan->req.arg0; + KUNIT_ASSERT_FALSE(test, chan_tx_done(t->isp, chan)); + KUNIT_ASSERT_EQ(test, chan->cursor, cursor); + slots[cursor].arg0 |= ISP_IPC_FLAG_ACK; + KUNIT_ASSERT_TRUE(test, chan_tx_done(t->isp, chan)); + KUNIT_ASSERT_EQ(test, chan->cursor, (cursor + 1) % chan->num); + } + KUNIT_EXPECT_EQ(test, chan->cursor, 0); +} + static struct kunit_case isp_ipc_test_cases[] = { + KUNIT_CASE(isp_test_command_ring_wrap), KUNIT_CASE(isp_test_sm_free_releases), KUNIT_CASE(isp_test_sm_free_alias), KUNIT_CASE(isp_test_sm_free_unknown), diff --git a/drivers/media/platform/apple/isp/isp-ipc.c b/drivers/media/platform/apple/isp/isp-ipc.c index f709f6e305763..da946f360ff5c 100644 --- a/drivers/media/platform/apple/isp/isp-ipc.c +++ b/drivers/media/platform/apple/isp/isp-ipc.c @@ -6,6 +6,10 @@ #include "isp-regs.h" #include "isp-fw.h" +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_KUNIT_TEST) +#include +#endif + #define ISP_IPC_FLAG_TERMINAL_ACK 0x3 /* longer firmware log lines are cut */ #define ISP_IPC_TERMINAL_MAX_LEN 512 @@ -66,7 +70,16 @@ static inline void chan_read_msg_index(struct apple_isp *isp, struct isp_channel *chan, struct isp_message *msg, u32 index) { - memcpy(msg, chan_msg_virt(chan, index), sizeof(*msg)); + if (isp_is_p010(isp)) { + u64 *slot = chan_msg_virt(chan, index); + + /* Acquire validity before consuming the firmware-owned payload. */ + msg->arg0 = READ_ONCE(*slot); + dma_rmb(); + memcpy(&msg->arg1, slot + 1, sizeof(*msg) - sizeof(msg->arg0)); + } else { + memcpy(msg, chan_msg_virt(chan, index), sizeof(*msg)); + } } static inline void chan_read_msg(struct apple_isp *isp, @@ -124,6 +137,8 @@ static int chan_handle_once(struct apple_isp *isp, struct isp_channel *chan) isp_mbox2_write32(isp, ISP_MBOX2_IRQ_DOORBELL, chan->doorbell); chan_update_cursor(chan); + if (chan->ops->complete) + chan->ops->complete(isp); return 0; } @@ -139,10 +154,12 @@ static inline bool chan_rx_done(struct apple_isp *isp, struct isp_channel *chan) int ipc_chan_handle(struct apple_isp *isp, struct isp_channel *chan) { + unsigned int budget = isp_is_p010(isp) ? chan->num : UINT_MAX; int err = 0; mutex_lock(&chan->lock); - while (1) { + /* A busy producer must not starve the other completion/control rings. */ + while (budget--) { chan_read_msg(isp, chan, &chan->req); if (chan_rx_done(isp, chan)) { err = 0; @@ -176,6 +193,11 @@ int ipc_chan_send(struct apple_isp *isp, struct isp_channel *chan, { long t; +#if IS_ENABLED(CONFIG_VIDEO_APPLE_ISP_KUNIT_TEST) + KUNIT_STATIC_STUB_REDIRECT(ipc_chan_send, isp, chan, timeout); +#endif + if (isp->ipc_failed) + return -EIO; chan_write_msg(isp, chan, &chan->req); dma_wmb(); @@ -186,6 +208,8 @@ int ipc_chan_send(struct apple_isp *isp, struct isp_channel *chan, t = wait_event_timeout(isp->wait, chan_tx_done(isp, chan), timeout); if (t == 0) { + if (isp_is_p010(isp)) + isp->ipc_failed = true; dev_err(isp->dev, "%s: timed out on request [0x%llx, 0x%llx, 0x%llx]\n", chan->name, chan->req.arg0, chan->req.arg1, diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c index 895a6d1299c22..47f1f8900d138 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2-test.c +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -7,6 +7,7 @@ #include #include +#include #include #define ISP_TEST_FRAME_SIZE 0x1000 @@ -19,6 +20,8 @@ static struct apple_isp *isp_test_alloc(struct kunit *test) KUNIT_ASSERT_NOT_ERR_OR_NULL(test, isp); mutex_init(&isp->video_lock); + spin_lock_init(&isp->buf_lock); + INIT_LIST_HEAD(&isp->bufs_returned); isp_get_current_format(isp)->total_size = ISP_TEST_FRAME_SIZE; return isp; @@ -107,7 +110,232 @@ static void isp_test_create_bufs_limit(struct kunit *test) sizes), -EINVAL); } +static const struct apple_isp_hw isp_test_t6040 = { + .gen = ISP_GEN_T6040, + .fw_abi = ISP_FW_ABI_H17, +}; + +static void isp_test_p010_format(struct kunit *test) +{ + struct apple_isp *isp = isp_test_alloc(test); + struct isp_format *fmt = isp_get_current_format(isp); + struct isp_preset preset = { .output_dim = { 1920, 1080 }, .p010 = true }; + struct v4l2_format f = {}; + unsigned int sizes[VB2_MAX_PLANES] = {}; + unsigned int nbuffers = 16, num_planes = 0; + + isp->hw = &isp_test_t6040; + fmt->version = 0x958; + KUNIT_ASSERT_EQ(test, isp_set_preset(isp, fmt, &preset, V4L2_PIX_FMT_P010), 0); + isp_get_sp_pix_format(isp, &f, fmt); + KUNIT_EXPECT_EQ(test, f.fmt.pix.pixelformat, V4L2_PIX_FMT_P010); + KUNIT_EXPECT_EQ(test, f.fmt.pix.bytesperline, 3840); + KUNIT_EXPECT_EQ(test, f.fmt.pix.sizeimage, 6220800); + KUNIT_EXPECT_EQ(test, fmt->plane_size[0], (size_t)4147200); + KUNIT_EXPECT_EQ(test, fmt->plane_size[1], (size_t)2073600); + KUNIT_EXPECT_EQ(test, f.fmt.pix.quantization, V4L2_QUANTIZATION_LIM_RANGE); + isp_get_mp_pix_format(isp, &f, fmt); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.pixelformat, V4L2_PIX_FMT_P010); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.num_planes, 1); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.plane_fmt[0].sizeimage, 6220800); + + isp->vbq.drv_priv = isp; + isp_test_set_num_buffers(test, &isp->vbq, 0); + KUNIT_ASSERT_EQ(test, + isp_vb2_queue_setup(&isp->vbq, &nbuffers, &num_planes, sizes, NULL), 0); + KUNIT_EXPECT_EQ(test, nbuffers, 2); + KUNIT_EXPECT_EQ(test, num_planes, 1); + KUNIT_EXPECT_EQ(test, sizes[0], 6220800); + KUNIT_EXPECT_EQ(test, isp_num_meta(isp), 8); + + preset.output_dim = (struct coord){1280, 720}; + KUNIT_ASSERT_EQ(test, isp_set_preset(isp, fmt, &preset, V4L2_PIX_FMT_P010), 0); + KUNIT_EXPECT_EQ(test, fmt->strides[0], 2560); + KUNIT_EXPECT_EQ(test, fmt->total_size, (size_t)2764800); + /* NV12 remains the default and restores the existing pool capacity. */ + KUNIT_ASSERT_EQ(test, isp_set_preset(isp, fmt, &preset, V4L2_PIX_FMT_NV12), 0); + KUNIT_EXPECT_EQ(test, fmt->pixelformat, V4L2_PIX_FMT_NV12); + KUNIT_EXPECT_EQ(test, fmt->total_size, (size_t)1382400); + KUNIT_EXPECT_EQ(test, isp_num_rendered(isp), ISP_MAX_BUFFERS); + KUNIT_EXPECT_EQ(test, isp_num_meta(isp), ISP_MAX_BUFFERS); +} + +static void isp_test_p010_admission(struct kunit *test) +{ + static const struct apple_isp_hw neo = { .gen = ISP_GEN_T8140 }; + struct apple_isp *isp = isp_test_alloc(test); + struct isp_format *fmt = isp_get_current_format(isp); + struct isp_preset preset = { .output_dim = {1280, 720}, .p010 = true }; + struct v4l2_fmtdesc desc = { .index = 1 }; + + isp->presets = &preset; + isp->num_presets = 1; + isp->hw = &isp_test_t6040; + fmt->version = 0x958; + KUNIT_ASSERT_EQ(test, isp_enum_pixelformat(isp, &desc), 0); + KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_P010); + desc.index = 2; + KUNIT_EXPECT_EQ(test, isp_enum_pixelformat(isp, &desc), -EINVAL); + isp->multiplanar = true; + KUNIT_ASSERT_EQ(test, isp_enum_pixelformat(isp, &desc), 0); + KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_P010); + isp->hw = &neo; + KUNIT_EXPECT_EQ(test, isp_enum_pixelformat(isp, &desc), -EINVAL); + KUNIT_ASSERT_EQ(test, isp_set_preset(isp, fmt, &preset, V4L2_PIX_FMT_P010), 0); + KUNIT_EXPECT_EQ(test, fmt->pixelformat, V4L2_PIX_FMT_NV12); + KUNIT_EXPECT_EQ(test, fmt->strides[0], 1280); +} + +static void isp_test_p010_completion_identity(struct kunit *test) +{ + struct isp_format fmt = { .plane_size = {4147200, 2073600} }; + struct isp_buffer buf = { .surfs = {{ .iova = 0x10020000000ULL }}, .tag = 42 }; + struct isp_buflist_buffer desc = { + .iovas = {0x20000000, 0x203f4800}, + .num_planes = 2, .pool_type = CISP_POOL_TYPE_RENDERED_SCL1, .tag = 42, + }; + + KUNIT_EXPECT_TRUE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.tag--; + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.tag++; + desc.iovas[1]++; + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.iovas[1]--; + desc.iovas[0] |= BIT_ULL(40); + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.iovas[0] &= ~BIT_ULL(40); + desc.pool_type = CISP_POOL_TYPE_RENDERED; + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.pool_type = CISP_POOL_TYPE_RENDERED_SCL1; + desc.num_planes = 1; + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); +} + +static void isp_test_p010_report_defers_completion(struct kunit *test) +{ + struct apple_isp *isp = isp_test_alloc(test); + struct isp_format *fmt = isp_get_current_format(isp); + struct isp_buffer *buf = kunit_kzalloc(test, sizeof(*buf), GFP_KERNEL); + struct { + u64 type, count; + struct isp_buflist_buffer desc; + } report = { + .type = 2, .count = 1, + .desc = { .iovas = {0x20000000, 0x203f4800}, + .num_planes = 2, .pool_type = 9, .tag = 42 }, + }; + struct isp_surf report_surf = { + .iova = 0x30000000, .size = sizeof(report), .virt = &report, + }; + struct isp_channel chan = { + .req = { .arg0 = report_surf.iova, .arg1 = sizeof(report), + .arg2 = ISP_IPC_BUFEXC_FLAG_RENDER }, + }; + + KUNIT_ASSERT_NOT_NULL(test, buf); + fmt->pixelformat = V4L2_PIX_FMT_P010; + fmt->plane_size[0] = 4147200; + isp->fw_iova_mask = U64_MAX; + isp->bt_surf = &report_surf; + buf->surfs[0].iova = 0x10020000000ULL; + buf->tag = 42; + INIT_LIST_HEAD(&isp->bufs_submitted); + list_add(&buf->link, &isp->bufs_submitted); + /* There is no vb2 queue: completing it before the ACK would fault. */ + KUNIT_ASSERT_EQ(test, ipc_bt_handle(isp, &chan), 0); + KUNIT_EXPECT_TRUE(test, list_empty(&isp->bufs_submitted)); + KUNIT_EXPECT_PTR_EQ(test, + list_first_entry(&isp->bufs_returned, struct isp_buffer, link), buf); + KUNIT_EXPECT_EQ(test, buf->return_state, VB2_BUF_STATE_DONE); + KUNIT_EXPECT_EQ(test, chan.rsp.arg0, report_surf.iova | ISP_IPC_FLAG_ACK); +} + +static void isp_test_p010_mode_selection(struct kunit *test) +{ + struct apple_isp *isp = isp_test_alloc(test); + struct isp_preset presets[] = { + { .index = 5, .output_dim = {1920, 1080} }, + { .index = 5, .output_dim = {1280, 720}, .p010 = true }, + { .index = 1, .output_dim = {4096, 3072} }, + }; + + isp->hw = &isp_test_t6040; + isp_get_current_format(isp)->version = 0x958; + isp->presets = presets; + isp->num_presets = ARRAY_SIZE(presets); + KUNIT_EXPECT_PTR_EQ(test, isp_select_preset(isp, 0, 0, V4L2_PIX_FMT_P010), &presets[1]); + KUNIT_EXPECT_PTR_EQ(test, isp_select_preset(isp, 1920, 1080, V4L2_PIX_FMT_P010), + &presets[1]); + KUNIT_EXPECT_PTR_EQ(test, isp_select_preset(isp, 4096, 3072, V4L2_PIX_FMT_P010), + &presets[1]); + KUNIT_EXPECT_PTR_EQ(test, isp_select_preset(isp, 1920, 1080, V4L2_PIX_FMT_NV12), + &presets[0]); + /* Only a matching firmware output enables the larger P010 size. */ + presets[2].p010 = true; + KUNIT_EXPECT_PTR_EQ(test, isp_select_preset(isp, 4096, 3072, V4L2_PIX_FMT_P010), + &presets[2]); +} + +static int isp_test_buffer_ack(struct apple_isp *isp, struct isp_channel *chan, + unsigned long timeout) +{ + unsigned int *calls = kunit_get_current_test()->priv; + + (*calls)++; + return 0; +} + +static void isp_test_p010_last_lease_requeue(struct kunit *test) +{ + struct apple_isp *isp = isp_test_alloc(test); + struct isp_format *fmt = isp_get_current_format(isp); + struct isp_buffer *buf = kunit_kzalloc(test, sizeof(*buf), GFP_KERNEL); + struct isp_buffer *submitted; + unsigned int calls = 0; + + KUNIT_ASSERT_NOT_NULL(test, buf); + isp->hw = &isp_test_t6040; + fmt->pixelformat = V4L2_PIX_FMT_P010; + fmt->num_planes = 2; + fmt->plane_size[0] = 1843200; + fmt->plane_size[1] = 921600; + INIT_LIST_HEAD(&isp->bufs_pending); + INIT_LIST_HEAD(&isp->bufs_submitted); + isp->cmd_virt = kunit_kzalloc(test, ISP_CMD_AREA_SIZE(0), GFP_KERNEL); + isp->chan_bh = kunit_kzalloc(test, sizeof(*isp->chan_bh), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); + KUNIT_ASSERT_NOT_NULL(test, isp->chan_bh); + for (unsigned int i = 0; i < isp_num_meta(isp); i++) { + isp->meta_surfs[i] = kunit_kzalloc(test, sizeof(struct isp_surf), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, isp->meta_surfs[i]); + isp->meta_surfs[i]->submitted = true; + } + isp->vbq.drv_priv = isp; + buf->vb.vb2_buf.vb2_queue = &isp->vbq; + buf->surfs[0].iova = 0x20000000; + set_bit(ISP_STATE_STREAMING, &isp->state); + test->priv = &calls; + kunit_activate_static_stub(test, ipc_chan_send, isp_test_buffer_ack); + for (unsigned int i = 1; i <= 10000; i++) { + /* Both lists empty, as after the final completion was dequeued. */ + isp_vb2_buf_queue(&buf->vb.vb2_buf); + KUNIT_ASSERT_EQ(test, calls, i); + KUNIT_ASSERT_EQ(test, buf->tag, (u64)i); + KUNIT_ASSERT_TRUE(test, list_empty(&isp->bufs_pending)); + submitted = list_first_entry(&isp->bufs_submitted, struct isp_buffer, link); + KUNIT_ASSERT_PTR_EQ(test, submitted, buf); + list_del(&buf->link); + } +} + static struct kunit_case isp_v4l2_test_cases[] = { + KUNIT_CASE(isp_test_p010_last_lease_requeue), + KUNIT_CASE(isp_test_p010_report_defers_completion), + KUNIT_CASE(isp_test_p010_mode_selection), + KUNIT_CASE(isp_test_p010_format), + KUNIT_CASE(isp_test_p010_admission), + KUNIT_CASE(isp_test_p010_completion_identity), KUNIT_CASE(isp_test_queue_init), KUNIT_CASE(isp_test_reqbufs_limit), KUNIT_CASE(isp_test_create_bufs_limit), diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index 072b8523279c2..35c30582ab81d 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -21,7 +21,6 @@ #define ISP_MIN_FRAMES 2 #define ISP_MAX_PLANES 4 -#define ISP_MAX_PIX_FORMATS 2 #define ISP_BUFFER_TIMEOUT msecs_to_jiffies(1500) #define ISP_STRIDE_ALIGNMENT 64 @@ -37,8 +36,7 @@ struct isp_buflist_buffer { u32 flags[ISP_MAX_PLANES]; u32 num_planes; u32 pool_type; - u32 tag; - u32 pad; + u64 tag; } __packed; static_assert(sizeof(struct isp_buflist_buffer) == ISP_BUFLIST_DESC_SIZE); @@ -57,6 +55,35 @@ static_assert(ISP_CMD_AREA_SIZE(0) >= ISP_IPC_BUFEXC_STAT_SIZE); */ #define ISP_BUFLIST_POOL_CAPTURE_META 2 +static bool isp_p010_buffer_matches(const struct isp_format *fmt, + const struct isp_buffer *buf, + const struct isp_buflist_buffer *desc) +{ + /* H17 reports zero-extended low 32 bits, not the submitted high IOVA. */ + return desc->num_planes == 2 && + desc->pool_type == CISP_POOL_TYPE_RENDERED_SCL1 && + desc->iovas[0] == (u32)buf->surfs[0].iova && + desc->iovas[1] == (u32)(buf->surfs[0].iova + fmt->plane_size[0]) && + !desc->iovas[2] && !desc->iovas[3] && desc->tag == buf->tag; +} + +void ipc_bt_complete(struct apple_isp *isp) +{ + struct isp_buffer *buf, *tmp; + unsigned long flags, i; + + /* The firmware has received its ACK before userspace can reuse a lease. */ + spin_lock_irqsave(&isp->buf_lock, flags); + for_each_set_bit(i, &isp->meta_returned, ISP_MAX_BUFFERS) + isp->meta_surfs[i]->submitted = false; + isp->meta_returned = 0; + list_for_each_entry_safe(buf, tmp, &isp->bufs_returned, link) { + list_del(&buf->link); + vb2_buffer_done(&buf->vb.vb2_buf, buf->return_state); + } + spin_unlock_irqrestore(&isp->buf_lock, flags); +} + int ipc_bt_handle(struct apple_isp *isp, struct isp_channel *chan) { struct isp_message *req = &chan->req, *rsp = &chan->rsp; @@ -96,11 +123,14 @@ int ipc_bt_handle(struct apple_isp *isp, struct isp_channel *chan) bufd->pool_type); */ if (bufd->pool_type == 0) { - for (int j = 0; j < ARRAY_SIZE(isp->meta_surfs); j++) { + for (int j = 0; j < isp_num_meta(isp); j++) { struct isp_surf *meta = isp->meta_surfs[j]; if ((u32)bufd->iovas[0] == (u32)meta->iova) { WARN_ON(!meta->submitted); - meta->submitted = false; + if (isp_is_p010(isp)) + __set_bit(j, &isp->meta_returned); + else + meta->submitted = false; } } } else if (bufd->pool_type == ISP_BUFLIST_POOL_CAPTURE_META && @@ -114,8 +144,10 @@ int ipc_bt_handle(struct apple_isp *isp, struct isp_channel *chan) } else { list_for_each_entry_safe_reverse( buf, tmp, &isp->bufs_submitted, link) { - if ((u32)buf->surfs[0].iova == - (u32)bufd->iovas[0]) { + if (isp_is_p010(isp) ? + isp_p010_buffer_matches(isp_get_current_format(isp), + buf, bufd) : + (u32)buf->surfs[0].iova == (u32)bufd->iovas[0]) { enum vb2_buffer_state state = VB2_BUF_STATE_ERROR; @@ -126,9 +158,13 @@ int ipc_bt_handle(struct apple_isp *isp, struct isp_channel *chan) if (req->arg2 == ISP_IPC_BUFEXC_FLAG_RENDER) state = VB2_BUF_STATE_DONE; - vb2_buffer_done(&buf->vb.vb2_buf, - state); - list_del(&buf->link); + if (isp_is_p010(isp)) { + buf->return_state = state; + list_move_tail(&buf->link, &isp->bufs_returned); + } else { + vb2_buffer_done(&buf->vb.vb2_buf, state); + list_del(&buf->link); + } } } } @@ -166,7 +202,7 @@ static int isp_submit_buffers(struct apple_isp *isp, bool pre_start) bl->type = 1; spin_lock_irqsave(&isp->buf_lock, flags); - for (int i = 0; i < ARRAY_SIZE(isp->meta_surfs); i++) { + for (int i = 0; i < isp_num_meta(isp); i++) { struct isp_surf *meta = isp->meta_surfs[i]; if (meta->submitted) @@ -205,9 +241,13 @@ static int isp_submit_buffers(struct apple_isp *isp, bool pre_start) } /* One rendered pool per batch; the rest stays pending for the next. */ - while (num_rendered < ISP_MAX_BUFFERS && + while (num_rendered < isp_num_rendered(isp) && (buf = list_first_entry_or_null(&isp->bufs_pending, struct isp_buffer, link))) { + if (isp_is_p010(isp)) { + buf->tag = ++isp->buffer_tag; + bufd->tag = buf->tag; + } bufd->num_planes = fmt->num_planes; bufd->pool_type = isp->hw->scl1 ? CISP_POOL_TYPE_RENDERED_SCL1 : CISP_POOL_TYPE_RENDERED; @@ -249,6 +289,11 @@ static int isp_submit_buffers(struct apple_isp *isp, bool pre_start) err = ipc_chan_send(isp, chan, ISP_BUFFER_TIMEOUT); if (err) { + if (isp_is_p010(isp)) { + /* Keep leases owned until stop_camera halts the coprocessor. */ + isp->ipc_failed = true; + return err; + } /* If we fail, consider the buffer not submitted. */ dev_err(isp->dev, "%s: failed to send bufs: [0x%llx, 0x%llx, 0x%llx]\n", @@ -269,7 +314,7 @@ static int isp_submit_buffers(struct apple_isp *isp, bool pre_start) &isp->bufs_pending); } } - for (int j = 0; j < ARRAY_SIZE(isp->meta_surfs); j++) { + for (int j = 0; j < isp_num_meta(isp); j++) { struct isp_surf *meta = isp->meta_surfs[j]; if (bufd->iovas[0] == meta->iova) { meta->submitted = false; @@ -308,10 +353,10 @@ static int isp_vb2_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers, * REQBUFS frees the old buffers first, CREATE_BUFS adds to them, so * count the buffers the queue already has. */ - if (q_num_bufs >= ISP_MAX_BUFFERS) + if (q_num_bufs >= isp_num_rendered(isp)) return -ENOBUFS; *nbuffers = min_t(unsigned int, *nbuffers, - ISP_MAX_BUFFERS - q_num_bufs); + isp_num_rendered(isp) - q_num_bufs); if (*num_planes) { if (sizes[0] < fmt->total_size) @@ -407,8 +452,15 @@ static void isp_vb2_buf_queue(struct vb2_buffer *vb) list_add_tail(&buf->link, &isp->bufs_pending); spin_unlock_irqrestore(&isp->buf_lock, flags); - if (test_bit(ISP_STATE_STREAMING, &isp->state) && !empty) - isp_submit_buffers(isp, false); + /* A returned last lease must restart an empty P010 queue immediately. */ + if (test_bit(ISP_STATE_STREAMING, &isp->state) && + (isp_is_p010(isp) || !empty)) { + if (isp_submit_buffers(isp, false) && isp_is_p010(isp)) { + apple_isp_firmware_halt(isp); + isp_vb2_release_buffers(isp, VB2_BUF_STATE_ERROR); + vb2_queue_error(&isp->vbq); + } + } } static int apple_isp_start_streaming(struct apple_isp *isp) @@ -565,18 +617,23 @@ static const struct vb2_ops isp_vb2_ops = { }; static int isp_set_preset(struct apple_isp *isp, struct isp_format *fmt, - struct isp_preset *preset) + struct isp_preset *preset, u32 pixelformat) { + unsigned int bytes_per_sample; int i; size_t total_size; fmt->preset = preset; - /* I really fucking hope they all use NV12. */ + /* P010 has MSB-aligned 10-bit samples in little-endian 16-bit words. */ + fmt->pixelformat = pixelformat == V4L2_PIX_FMT_P010 && + isp_supports_p010(isp) && preset->p010 ? + V4L2_PIX_FMT_P010 : V4L2_PIX_FMT_NV12; + bytes_per_sample = fmt->pixelformat == V4L2_PIX_FMT_P010 ? 2 : 1; fmt->num_planes = 2; - fmt->strides[0] = ALIGN(preset->output_dim.x, ISP_STRIDE_ALIGNMENT); - /* UV subsampled interleaved */ - fmt->strides[1] = ALIGN(preset->output_dim.x, ISP_STRIDE_ALIGNMENT); + fmt->strides[0] = ALIGN(preset->output_dim.x * bytes_per_sample, + ISP_STRIDE_ALIGNMENT); + fmt->strides[1] = fmt->strides[0]; fmt->plane_size[0] = fmt->strides[0] * preset->output_dim.y; fmt->plane_size[1] = fmt->strides[1] * preset->output_dim.y / 2; @@ -589,17 +646,25 @@ static int isp_set_preset(struct apple_isp *isp, struct isp_format *fmt, } static struct isp_preset *isp_select_preset(struct apple_isp *isp, u32 width, - u32 height) + u32 height, u32 pixelformat) { struct isp_preset *preset, *best = &isp->presets[0]; int i, score, best_score = INT_MAX; - /* Default if no dimensions */ - if (width == 0 || height == 0) - return &isp->presets[0]; + bool p010 = pixelformat == V4L2_PIX_FMT_P010 && isp_has_p010(isp); + + /* An unspecified P010 size selects the processed 720p preview first. */ + if (width == 0 || height == 0) { + if (!p010) + return &isp->presets[0]; + width = 1280; + height = 720; + } for (i = 0; i < isp->num_presets; i++) { preset = &isp->presets[i]; + if (p010 && !preset->p010) + continue; score = abs((int)preset->output_dim.x - (int)width) + abs((int)preset->output_dim.y - (int)height); if (score < best_score) { @@ -623,22 +688,22 @@ static int isp_vidioc_querycap(struct file *file, void *priv, return 0; } -static int isp_vidioc_enum_format(struct file *file, void *fh, - struct v4l2_fmtdesc *f) +static int isp_enum_pixelformat(struct apple_isp *isp, struct v4l2_fmtdesc *f) { - struct apple_isp *isp = video_drvdata(file); - - if (f->index >= ISP_MAX_PIX_FORMATS) - return -EINVAL; - switch (f->index) { case 0: f->pixelformat = V4L2_PIX_FMT_NV12; break; case 1: - if (!isp->multiplanar) + if (isp->multiplanar) { + f->pixelformat = V4L2_PIX_FMT_NV12M; + break; + } + fallthrough; + case 2: + if (!isp_has_p010(isp) || f->index != (isp->multiplanar ? 2 : 1)) return -EINVAL; - f->pixelformat = V4L2_PIX_FMT_NV12M; + f->pixelformat = V4L2_PIX_FMT_P010; break; default: return -EINVAL; @@ -647,20 +712,36 @@ static int isp_vidioc_enum_format(struct file *file, void *fh, return 0; } +static int isp_vidioc_enum_format(struct file *file, void *fh, + struct v4l2_fmtdesc *f) +{ + return isp_enum_pixelformat(video_drvdata(file), f); +} + static int isp_vidioc_enum_framesizes(struct file *file, void *fh, struct v4l2_frmsizeenum *f) { struct apple_isp *isp = video_drvdata(file); + struct isp_preset *preset = NULL; + u32 index = 0; - if (f->index >= isp->num_presets) + if (f->pixel_format != V4L2_PIX_FMT_NV12 && + f->pixel_format != V4L2_PIX_FMT_NV12M && + !(f->pixel_format == V4L2_PIX_FMT_P010 && isp_has_p010(isp))) return -EINVAL; - if ((f->pixel_format != V4L2_PIX_FMT_NV12) && - (f->pixel_format != V4L2_PIX_FMT_NV12M)) + for (int i = 0; i < isp->num_presets; i++) { + if (f->pixel_format == V4L2_PIX_FMT_P010 && !isp->presets[i].p010) + continue; + if (index++ == f->index) { + preset = &isp->presets[i]; + break; + } + } + if (!preset) return -EINVAL; - - f->discrete.width = isp->presets[f->index].output_dim.x; - f->discrete.height = isp->presets[f->index].output_dim.y; + f->discrete.width = preset->output_dim.x; + f->discrete.height = preset->output_dim.y; f->type = V4L2_FRMSIZE_TYPE_DISCRETE; return 0; @@ -676,10 +757,13 @@ static int isp_vidioc_enum_frameintervals(struct file *file, void *fh, return -EINVAL; if (interval->pixel_format != V4L2_PIX_FMT_NV12 && - interval->pixel_format != V4L2_PIX_FMT_NV12M) + interval->pixel_format != V4L2_PIX_FMT_NV12M && + !(interval->pixel_format == V4L2_PIX_FMT_P010 && isp_has_p010(isp))) return -EINVAL; for (i = 0; i < isp->num_presets; i++) { + if (interval->pixel_format == V4L2_PIX_FMT_P010 && !isp->presets[i].p010) + continue; if (isp->presets[i].output_dim.x == interval->width && isp->presets[i].output_dim.y == interval->height) break; @@ -726,10 +810,12 @@ static inline void isp_get_sp_pix_format(struct apple_isp *isp, f->fmt.pix.sizeimage = fmt->total_size; f->fmt.pix.field = V4L2_FIELD_NONE; - f->fmt.pix.pixelformat = V4L2_PIX_FMT_NV12; + f->fmt.pix.pixelformat = fmt->pixelformat; f->fmt.pix.colorspace = V4L2_COLORSPACE_REC709; f->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_709; f->fmt.pix.xfer_func = V4L2_XFER_FUNC_709; + if (fmt->pixelformat == V4L2_PIX_FMT_P010) + f->fmt.pix.quantization = V4L2_QUANTIZATION_LIM_RANGE; } static inline void isp_get_mp_pix_format(struct apple_isp *isp, @@ -746,9 +832,19 @@ static inline void isp_get_mp_pix_format(struct apple_isp *isp, f->fmt.pix_mp.field = V4L2_FIELD_NONE; f->fmt.pix_mp.pixelformat = V4L2_PIX_FMT_NV12M; + /* P010 is contiguous, including through the multi-planar API. */ + if (fmt->pixelformat == V4L2_PIX_FMT_P010) { + f->fmt.pix_mp.pixelformat = V4L2_PIX_FMT_P010; + f->fmt.pix_mp.num_planes = 1; + f->fmt.pix_mp.plane_fmt[0].sizeimage = fmt->total_size; + memset(&f->fmt.pix_mp.plane_fmt[1], 0, + sizeof(f->fmt.pix_mp.plane_fmt[1])); + } f->fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709; f->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_709; f->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_709; + if (fmt->pixelformat == V4L2_PIX_FMT_P010) + f->fmt.pix_mp.quantization = V4L2_QUANTIZATION_LIM_RANGE; } static int isp_vidioc_get_format(struct file *file, void *fh, @@ -774,8 +870,9 @@ static int isp_vidioc_set_format(struct file *file, void *fh, if (vb2_is_busy(&isp->vbq)) return -EBUSY; - preset = isp_select_preset(isp, f->fmt.pix.width, f->fmt.pix.height); - err = isp_set_preset(isp, fmt, preset); + preset = isp_select_preset(isp, f->fmt.pix.width, f->fmt.pix.height, + f->fmt.pix.pixelformat); + err = isp_set_preset(isp, fmt, preset, f->fmt.pix.pixelformat); if (err) return err; @@ -794,8 +891,9 @@ static int isp_vidioc_try_format(struct file *file, void *fh, struct isp_preset *preset; int err; - preset = isp_select_preset(isp, f->fmt.pix.width, f->fmt.pix.height); - err = isp_set_preset(isp, &fmt, preset); + preset = isp_select_preset(isp, f->fmt.pix.width, f->fmt.pix.height, + f->fmt.pix.pixelformat); + err = isp_set_preset(isp, &fmt, preset, f->fmt.pix.pixelformat); if (err) return err; @@ -832,9 +930,9 @@ static int isp_vidioc_set_format_mplane(struct file *file, void *fh, if (vb2_is_busy(&isp->vbq)) return -EBUSY; - preset = isp_select_preset(isp, f->fmt.pix_mp.width, - f->fmt.pix_mp.height); - err = isp_set_preset(isp, fmt, preset); + preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height, + f->fmt.pix_mp.pixelformat); + err = isp_set_preset(isp, fmt, preset, f->fmt.pix_mp.pixelformat); if (err) return err; @@ -856,9 +954,9 @@ static int isp_vidioc_try_format_mplane(struct file *file, void *fh, if (!isp->multiplanar) return -ENOTTY; - preset = isp_select_preset(isp, f->fmt.pix_mp.width, - f->fmt.pix_mp.height); - err = isp_set_preset(isp, &fmt, preset); + preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height, + f->fmt.pix_mp.pixelformat); + err = isp_set_preset(isp, &fmt, preset, f->fmt.pix_mp.pixelformat); if (err) return err; @@ -940,6 +1038,24 @@ static int isp_vidioc_set_param(struct file *file, void *fh, return 0; } +static int isp_vidioc_log_status(struct file *file, void *fh) +{ + struct apple_isp *isp = video_drvdata(file); + + v4l2_ctrl_log_status(file, fh); + for (unsigned int i = 0; i < isp->num_sensor_configs; i++) { + const struct isp_sensor_config *config = &isp->sensor_configs[i]; + + v4l2_info(&isp->v4l2_dev, + "sensor config %u: input %ux%u output %ux%u clock %u%s\n", + i, config->input_dim.x, config->input_dim.y, + config->output_dim.x, config->output_dim.y, + config->sensor_clk, + config->usable ? "" : " (unverified geometry)"); + } + return 0; +} + static const struct v4l2_ioctl_ops isp_v4l2_ioctl_ops = { .vidioc_querycap = isp_vidioc_querycap, @@ -969,7 +1085,7 @@ static const struct v4l2_ioctl_ops isp_v4l2_ioctl_ops = { .vidioc_streamon = vb2_ioctl_streamon, .vidioc_streamoff = vb2_ioctl_streamoff, - .vidioc_log_status = v4l2_ctrl_log_status, + .vidioc_log_status = isp_vidioc_log_status, .vidioc_subscribe_event = v4l2_ctrl_subscribe_event, .vidioc_unsubscribe_event = v4l2_event_unsubscribe, }; @@ -1044,7 +1160,7 @@ int apple_isp_setup_video(struct apple_isp *isp) struct isp_format *fmt = isp_get_current_format(isp); int err; - err = isp_set_preset(isp, fmt, &isp->presets[0]); + err = isp_set_preset(isp, fmt, &isp->presets[0], V4L2_PIX_FMT_NV12); if (err) { dev_err(isp->dev, "failed to set default preset: %d\n", err); return err; @@ -1054,7 +1170,8 @@ int apple_isp_setup_video(struct apple_isp *isp) for (int i = 0; i < ARRAY_SIZE(isp->meta_surfs); i++) { isp->meta_surfs[i] = - isp_alloc_surface_vmap(isp, isp->hw->meta_size); + isp_alloc_surface_vmap(isp, isp_has_p010(isp) ? 0x8000 : + isp->hw->meta_size); if (!isp->meta_surfs[i]) { isp_err(isp, "failed to alloc meta surface\n"); err = -ENOMEM; diff --git a/drivers/media/platform/apple/isp/isp-v4l2.h b/drivers/media/platform/apple/isp/isp-v4l2.h index 622288296aaae..55135594576b2 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.h +++ b/drivers/media/platform/apple/isp/isp-v4l2.h @@ -9,6 +9,7 @@ int apple_isp_setup_video(struct apple_isp *isp); void apple_isp_remove_video(struct apple_isp *isp); void apple_isp_free_video(struct apple_isp *isp); +void ipc_bt_complete(struct apple_isp *isp); int ipc_bt_handle(struct apple_isp *isp, struct isp_channel *chan); int apple_isp_video_suspend(struct apple_isp *isp); From 910d31f6d65aa1eb497dcfd01fb1ca52e692e169 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:06:16 +1100 Subject: [PATCH 02/11] selftests: media: add Apple ISP capture delivery stress runner Exercise every offered format and discrete size with long MMAP capture, repeated stream restarts and format switches. Record delivered rate, timing gaps and buffer errors, and fail on substitutions or stalls. Allow a mode filter for different nominal rates. A driver-generated sequence alone is not evidence of zero firmware drops. Assisted-by: LLM Signed-off-by: Ryan Murray --- tools/testing/selftests/media/Makefile | 4 + .../testing/selftests/media/README.apple-isp | 34 +++ .../selftests/media/apple-isp-stress.c | 278 ++++++++++++++++++ 3 files changed, 316 insertions(+) create mode 100644 tools/testing/selftests/media/Makefile create mode 100644 tools/testing/selftests/media/README.apple-isp create mode 100644 tools/testing/selftests/media/apple-isp-stress.c diff --git a/tools/testing/selftests/media/Makefile b/tools/testing/selftests/media/Makefile new file mode 100644 index 0000000000000..b63cb0f120e7e --- /dev/null +++ b/tools/testing/selftests/media/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only +CFLAGS += -O2 -Wall -Wextra -Werror +TEST_GEN_PROGS_EXTENDED := apple-isp-stress +include ../lib.mk diff --git a/tools/testing/selftests/media/README.apple-isp b/tools/testing/selftests/media/README.apple-isp new file mode 100644 index 0000000000000..57b95740120c0 --- /dev/null +++ b/tools/testing/selftests/media/README.apple-isp @@ -0,0 +1,34 @@ +Apple ISP capture delivery stress +================================= + +Build with make -C tools/testing/selftests/media. This is a manual hardware +runner; run_tests does not start the camera automatically. No image files are +written. Run only on the agreed J616s test kernel while audio is idle. + +Example for configurations whose expected rate has been established as 30 fps: + + ./apple-isp-stress /dev/video0 600 10 30 | tee capture-results.jsonl + +It enumerates every offered single-planar format and discrete size. Each gets +a ten-minute run, then ten five-second start/stop cycles, switching formats and +sizes between streams. A substituted format/rate, poll timeout, queue error, +V4L2 buffer error, sequence discontinuity or delivery gap exceeding 1.5 nominal +frame periods fails. Delivered FPS must reach 99.8% of the supplied rate, allowing +29.97/30 rounding. The rate argument is an explicit test expectation, not a +rate inferred from the sensor clock or set by the tool as an alleged sensor fact. +For modes with a different nominal rate, filter the run explicitly, for example +`./apple-isp-stress /dev/video0 600 10 25 P010 1920 1080`. A filtered run tests +that mode's restart cycles; the unfiltered run also exercises format switches. This runner stops +on the first failure to preserve evidence and avoid retrying ambiguous firmware. + +The driver's sequence increments for received reports, so consecutive sequence +values alone cannot establish zero sensor/firmware drops. The timing checks +measure delivery gaps and completed frames. A true zero-drop sensor claim also +needs a validated firmware frame counter or an independent sensor-rate oracle. +No RAW packing, Bayer order or nominal per-config FPS is guessed by this tool. +It tests RAW automatically if a verified RAW format is added to ENUM_FMT later. + +Preserve the current firmware config dump, v4l2-ctl --all/--list-formats-ext, +the JSONL output, dmesg before/after, exact kernel and source hashes. Check both +P010 presets for correct image content separately; this runner deliberately +avoids a converter or disk writer in the capture loop. diff --git a/tools/testing/selftests/media/apple-isp-stress.c b/tools/testing/selftests/media/apple-isp-stress.c new file mode 100644 index 0000000000000..8490a7abe3d68 --- /dev/null +++ b/tools/testing/selftests/media/apple-isp-stress.c @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* V4L2 delivery stress test. Does not infer sensor FPS from a driver counter. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_MODES 256 +#define MAX_BUFFERS 16 + +struct mode { + uint32_t fourcc, width, height; +}; + +struct mapping { + void *address; + size_t size; +}; + +static int xioctl(int fd, unsigned long cmd, void *arg) +{ + int ret; + + do { + ret = ioctl(fd, cmd, arg); + } while (ret < 0 && errno == EINTR); + return ret; +} + +static double monotonic(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec + ts.tv_nsec / 1000000000.0; +} + +static int capture(int fd, struct mode mode, unsigned int seconds, + unsigned int fps, unsigned int cycle) +{ + struct v4l2_format format = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE }; + struct v4l2_streamparm parm = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE }; + struct v4l2_requestbuffers req = { + .count = 4, .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .memory = V4L2_MEMORY_MMAP, + }; + struct mapping maps[MAX_BUFFERS] = {}; + enum v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + unsigned int frames = 0, bad = 0, gaps = 0, sequence_gaps = 0, mapped = 0; + unsigned int last_sequence = 0; + double start, end, first = 0, last = 0, max_gap = 0, measured_fps = 0; + bool streaming = false; + int ret = -1; + + format.fmt.pix.width = mode.width; + format.fmt.pix.height = mode.height; + format.fmt.pix.pixelformat = mode.fourcc; + format.fmt.pix.field = V4L2_FIELD_NONE; + if (xioctl(fd, VIDIOC_S_FMT, &format) < 0) + goto error; + /* Fail instead of counting a silently substituted mode as tested. */ + if (format.fmt.pix.width != mode.width || format.fmt.pix.height != mode.height || + format.fmt.pix.pixelformat != mode.fourcc) { + fprintf(stderr, "requested mode was substituted\n"); + goto out; + } + parm.parm.capture.timeperframe.numerator = 1; + parm.parm.capture.timeperframe.denominator = fps; + if (xioctl(fd, VIDIOC_S_PARM, &parm) < 0) + goto error; + if (parm.parm.capture.timeperframe.numerator != 1 || + parm.parm.capture.timeperframe.denominator != fps) { + fprintf(stderr, "requested rate was substituted\n"); + goto out; + } + if (xioctl(fd, VIDIOC_REQBUFS, &req) < 0) + goto error; + if (req.count < 2 || req.count > MAX_BUFFERS) + goto out; + for (unsigned int i = 0; i < req.count; i++) { + struct v4l2_buffer buf = { + .type = type, .memory = V4L2_MEMORY_MMAP, .index = i, + }; + + if (xioctl(fd, VIDIOC_QUERYBUF, &buf) < 0) + goto error; + maps[i].size = buf.length; + maps[i].address = mmap(NULL, buf.length, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, buf.m.offset); + if (maps[i].address == MAP_FAILED) + goto error; + mapped++; + if (xioctl(fd, VIDIOC_QBUF, &buf) < 0) + goto error; + } + if (xioctl(fd, VIDIOC_STREAMON, &type) < 0) + goto error; + streaming = true; + start = monotonic(); + end = start + seconds; + while (monotonic() < end) { + struct pollfd pfd = { .fd = fd, .events = POLLIN }; + struct v4l2_buffer buf = { .type = type, .memory = V4L2_MEMORY_MMAP }; + double timestamp, gap; + int ready = poll(&pfd, 1, 2000); + + if (ready < 0 && errno == EINTR) + continue; + if (ready <= 0 || (pfd.revents & (POLLERR | POLLHUP | POLLNVAL))) { + fprintf(stderr, "stream stalled or queue failed\n"); + goto out; + } + if (xioctl(fd, VIDIOC_DQBUF, &buf) < 0) + goto error; + if (buf.index >= req.count || buf.bytesused > maps[buf.index].size) + goto out; + if (buf.flags & V4L2_BUF_FLAG_ERROR) + bad++; + timestamp = buf.timestamp.tv_sec + buf.timestamp.tv_usec / 1000000.0; + if (frames) { + gap = timestamp - last; + if (gap > max_gap) + max_gap = gap; + if (gap <= 0 || gap > 1.5 / fps) + gaps++; + if ((uint32_t)(buf.sequence - last_sequence) != 1) + sequence_gaps++; + } else { + first = timestamp; + } + last = timestamp; + last_sequence = buf.sequence; + frames++; + /* Requeue immediately; image copies/conversion do not limit the test. */ + if (xioctl(fd, VIDIOC_QBUF, &buf) < 0) + goto error; + } + if (frames > 1 && last > first) + measured_fps = (frames - 1) / (last - first); + /* Allow clock/29.97-vs-30 rounding; gaps and buffer errors must be zero. */ + ret = bad || gaps || sequence_gaps || measured_fps < fps * 0.998 ? 1 : 0; + printf("{\"cycle\":%u,\"fourcc\":\"%.4s\",\"width\":%u,\"height\":%u,", + cycle, (char *)&mode.fourcc, mode.width, mode.height); + printf("\"stride\":%u,\"sizeimage\":%u,\"requested_fps\":%u,\"frames\":%u,", + format.fmt.pix.bytesperline, format.fmt.pix.sizeimage, fps, frames); + printf("\"delivered_fps\":%.6f,\"max_gap_ms\":%.3f,\"timing_gaps\":%u,", + measured_fps, max_gap * 1000, gaps); + printf("\"buffer_errors\":%u,\"sequence_gaps\":%u,\"pass\":%s}\n", + bad, sequence_gaps, ret ? "false" : "true"); + fflush(stdout); + goto out; +error: + perror("V4L2 ioctl"); +out: + if (streaming && xioctl(fd, VIDIOC_STREAMOFF, &type) < 0) { + perror("STREAMOFF"); + ret = -1; + } + for (unsigned int i = 0; i < mapped; i++) + munmap(maps[i].address, maps[i].size); + req.count = 0; + if (xioctl(fd, VIDIOC_REQBUFS, &req) < 0) + ret = -1; + return ret; +} + +int main(int argc, char **argv) +{ + struct mode modes[MAX_MODES]; + unsigned int count = 0, seconds, cycles, fps; + int fd, failed = 0; + char *end; + unsigned long values[3]; + + if (argc != 5 && argc != 8) { + fprintf(stderr, "usage: %s device seconds cycles fps [FOURCC width height]\n", + argv[0]); + return 2; + } + for (int i = 0; i < 3; i++) { + errno = 0; + values[i] = strtoul(argv[i + 2], &end, 10); + if (errno || *end || !values[i] || values[i] > 86400) + return 2; + } + seconds = values[0]; + cycles = values[1]; + fps = values[2]; + if (fps > 240 || cycles > 1000 || seconds < 5) + return 2; + fd = open(argv[1], O_RDWR | O_NONBLOCK); + if (fd < 0) { + perror(argv[1]); + return 2; + } + for (unsigned int i = 0; ; i++) { + struct v4l2_fmtdesc format = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .index = i }; + + if (xioctl(fd, VIDIOC_ENUM_FMT, &format) < 0) { + if (errno != EINVAL) + failed = 1; + break; + } + /* This harness exercises the driver's contiguous, single-planar API. */ + if (format.pixelformat == V4L2_PIX_FMT_NV12M) + continue; + for (unsigned int j = 0; ; j++) { + struct v4l2_frmsizeenum size = { + .pixel_format = format.pixelformat, .index = j, + }; + + if (xioctl(fd, VIDIOC_ENUM_FRAMESIZES, &size) < 0) { + if (errno != EINVAL) + failed = 1; + break; + } + if (size.type != V4L2_FRMSIZE_TYPE_DISCRETE || count == MAX_MODES) { + failed = 1; + break; + } + modes[count++] = (struct mode){format.pixelformat, size.discrete.width, + size.discrete.height}; + } + } + if (argc == 8) { + uint32_t fourcc; + unsigned long width, height; + bool found = false; + + if (strlen(argv[5]) != 4) { + close(fd); + return 2; + } + memcpy(&fourcc, argv[5], sizeof(fourcc)); + errno = 0; + width = strtoul(argv[6], &end, 10); + if (errno || *end || !width || width > UINT32_MAX) { + close(fd); + return 2; + } + errno = 0; + height = strtoul(argv[7], &end, 10); + if (errno || *end || !height || height > UINT32_MAX) { + close(fd); + return 2; + } + for (unsigned int i = 0; i < count; i++) { + if (modes[i].fourcc == fourcc && modes[i].width == width && + modes[i].height == height) { + modes[0] = modes[i]; + found = true; + break; + } + } + count = found ? 1 : 0; + } + if (!count) + failed = 1; + /* Long runs first, then repeated stops/restarts and format/size switches. */ + for (unsigned int cycle = 0; cycle <= cycles && !failed; cycle++) { + for (unsigned int i = 0; i < count; i++) { + if (capture(fd, modes[i], cycle ? 5 : seconds, fps, cycle)) { + failed = 1; + break; + } + } + } + close(fd); + return failed ? 1 : 0; +} From 3bd32e8909402c48eea988b72df229093f2a5cf6 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:09:55 +1100 Subject: [PATCH 03/11] media: apple: isp: reject P010 command ACKs after the deadline Check the command deadline before advancing the ring cursor, including when an ACK arrives at the wait timeout boundary. Keep the failed channel latched until coprocessor stop. Cover the boundary in KUnit. Emit the capture stress result after cleanup so a failed STREAMOFF cannot leave a passing result. Assisted-by: LLM Signed-off-by: Ryan Murray --- drivers/media/platform/apple/isp/isp-drv.h | 2 ++ .../media/platform/apple/isp/isp-ipc-test.c | 19 +++++++++++++++++++ drivers/media/platform/apple/isp/isp-ipc.c | 8 ++++++++ .../selftests/media/apple-isp-stress.c | 18 +++++++++--------- 4 files changed, 38 insertions(+), 9 deletions(-) diff --git a/drivers/media/platform/apple/isp/isp-drv.h b/drivers/media/platform/apple/isp/isp-drv.h index e02d675f0e998..db042f94aed47 100644 --- a/drivers/media/platform/apple/isp/isp-drv.h +++ b/drivers/media/platform/apple/isp/isp-drv.h @@ -117,6 +117,8 @@ struct isp_channel { void *virt; u32 doorbell; u32 cursor; + unsigned long deadline; + bool deadline_active; struct mutex lock; struct isp_message req; struct isp_message rsp; diff --git a/drivers/media/platform/apple/isp/isp-ipc-test.c b/drivers/media/platform/apple/isp/isp-ipc-test.c index efd24f6794a5a..8b6decd63f274 100644 --- a/drivers/media/platform/apple/isp/isp-ipc-test.c +++ b/drivers/media/platform/apple/isp/isp-ipc-test.c @@ -170,7 +170,26 @@ static void isp_test_command_ring_wrap(struct kunit *test) KUNIT_EXPECT_EQ(test, chan->cursor, 0); } +static void isp_test_late_ack_keeps_cursor(struct kunit *test) +{ + struct isp_sm_test *t = isp_sm_test_alloc(test); + struct isp_message response = { .arg0 = 0x5000001 }; + + isp_get_current_format(t->isp)->pixelformat = V4L2_PIX_FMT_P010; + t->chan->virt = &response; + t->chan->num = 8; + t->chan->req.arg0 = 0x5000000; + t->chan->deadline_active = true; + t->chan->deadline = jiffies; + KUNIT_EXPECT_FALSE(test, chan_tx_done(t->isp, t->chan)); + KUNIT_EXPECT_EQ(test, t->chan->cursor, 0); + t->chan->deadline = jiffies + HZ; + KUNIT_EXPECT_TRUE(test, chan_tx_done(t->isp, t->chan)); + KUNIT_EXPECT_EQ(test, t->chan->cursor, 1); +} + static struct kunit_case isp_ipc_test_cases[] = { + KUNIT_CASE(isp_test_late_ack_keeps_cursor), KUNIT_CASE(isp_test_command_ring_wrap), KUNIT_CASE(isp_test_sm_free_releases), KUNIT_CASE(isp_test_sm_free_alias), diff --git a/drivers/media/platform/apple/isp/isp-ipc.c b/drivers/media/platform/apple/isp/isp-ipc.c index da946f360ff5c..92cfc6ab35f5b 100644 --- a/drivers/media/platform/apple/isp/isp-ipc.c +++ b/drivers/media/platform/apple/isp/isp-ipc.c @@ -177,6 +177,9 @@ int ipc_chan_handle(struct apple_isp *isp, struct isp_channel *chan) static inline bool chan_tx_done(struct apple_isp *isp, struct isp_channel *chan) { + if (isp_is_p010(isp) && chan->deadline_active && + time_after_eq(jiffies, chan->deadline)) + return false; dma_rmb(); chan_read_msg(isp, chan, &chan->rsp); @@ -198,6 +201,10 @@ int ipc_chan_send(struct apple_isp *isp, struct isp_channel *chan, #endif if (isp->ipc_failed) return -EIO; + if (isp_is_p010(isp) && timeout) { + chan->deadline = jiffies + timeout; + chan->deadline_active = true; + } chan_write_msg(isp, chan, &chan->req); dma_wmb(); @@ -217,6 +224,7 @@ int ipc_chan_send(struct apple_isp *isp, struct isp_channel *chan, return -ETIME; } + chan->deadline_active = false; isp_dbg(isp, "%s: request success (%ld)\n", chan->name, t); return 0; diff --git a/tools/testing/selftests/media/apple-isp-stress.c b/tools/testing/selftests/media/apple-isp-stress.c index 8490a7abe3d68..cd85e51ecba09 100644 --- a/tools/testing/selftests/media/apple-isp-stress.c +++ b/tools/testing/selftests/media/apple-isp-stress.c @@ -147,15 +147,6 @@ static int capture(int fd, struct mode mode, unsigned int seconds, measured_fps = (frames - 1) / (last - first); /* Allow clock/29.97-vs-30 rounding; gaps and buffer errors must be zero. */ ret = bad || gaps || sequence_gaps || measured_fps < fps * 0.998 ? 1 : 0; - printf("{\"cycle\":%u,\"fourcc\":\"%.4s\",\"width\":%u,\"height\":%u,", - cycle, (char *)&mode.fourcc, mode.width, mode.height); - printf("\"stride\":%u,\"sizeimage\":%u,\"requested_fps\":%u,\"frames\":%u,", - format.fmt.pix.bytesperline, format.fmt.pix.sizeimage, fps, frames); - printf("\"delivered_fps\":%.6f,\"max_gap_ms\":%.3f,\"timing_gaps\":%u,", - measured_fps, max_gap * 1000, gaps); - printf("\"buffer_errors\":%u,\"sequence_gaps\":%u,\"pass\":%s}\n", - bad, sequence_gaps, ret ? "false" : "true"); - fflush(stdout); goto out; error: perror("V4L2 ioctl"); @@ -169,6 +160,15 @@ static int capture(int fd, struct mode mode, unsigned int seconds, req.count = 0; if (xioctl(fd, VIDIOC_REQBUFS, &req) < 0) ret = -1; + printf("{\"cycle\":%u,\"fourcc\":\"%.4s\",\"width\":%u,\"height\":%u,", + cycle, (char *)&mode.fourcc, mode.width, mode.height); + printf("\"stride\":%u,\"sizeimage\":%u,\"requested_fps\":%u,\"frames\":%u,", + format.fmt.pix.bytesperline, format.fmt.pix.sizeimage, fps, frames); + printf("\"delivered_fps\":%.6f,\"max_gap_ms\":%.3f,\"timing_gaps\":%u,", + measured_fps, max_gap * 1000, gaps); + printf("\"buffer_errors\":%u,\"sequence_gaps\":%u,\"pass\":%s}\n", + bad, sequence_gaps, ret ? "false" : "true"); + fflush(stdout); return ret; } From b17a883263f0252e2285836b5d9454065c98a6e1 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:12:47 +1100 Subject: [PATCH 04/11] media: apple: isp: requeue the last IMX958 NV12 buffer immediately Apply the empty-queue restart to both processed IMX958 formats. Exercise 10,000 empty-to-submitted transitions for each of P010 and NV12. Other sensors retain their queue policy. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/platform/apple/isp/isp-v4l2-test.c | 30 +++++++++++-------- drivers/media/platform/apple/isp/isp-v4l2.c | 4 +-- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c index 47f1f8900d138..58af78581bf7e 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2-test.c +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -286,7 +286,7 @@ static int isp_test_buffer_ack(struct apple_isp *isp, struct isp_channel *chan, return 0; } -static void isp_test_p010_last_lease_requeue(struct kunit *test) +static void isp_test_imx958_last_lease_requeue(struct kunit *test) { struct apple_isp *isp = isp_test_alloc(test); struct isp_format *fmt = isp_get_current_format(isp); @@ -296,6 +296,7 @@ static void isp_test_p010_last_lease_requeue(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, buf); isp->hw = &isp_test_t6040; + fmt->version = 0x958; fmt->pixelformat = V4L2_PIX_FMT_P010; fmt->num_planes = 2; fmt->plane_size[0] = 1843200; @@ -306,7 +307,7 @@ static void isp_test_p010_last_lease_requeue(struct kunit *test) isp->chan_bh = kunit_kzalloc(test, sizeof(*isp->chan_bh), GFP_KERNEL); KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); KUNIT_ASSERT_NOT_NULL(test, isp->chan_bh); - for (unsigned int i = 0; i < isp_num_meta(isp); i++) { + for (unsigned int i = 0; i < ISP_MAX_BUFFERS; i++) { isp->meta_surfs[i] = kunit_kzalloc(test, sizeof(struct isp_surf), GFP_KERNEL); KUNIT_ASSERT_NOT_NULL(test, isp->meta_surfs[i]); isp->meta_surfs[i]->submitted = true; @@ -317,20 +318,25 @@ static void isp_test_p010_last_lease_requeue(struct kunit *test) set_bit(ISP_STATE_STREAMING, &isp->state); test->priv = &calls; kunit_activate_static_stub(test, ipc_chan_send, isp_test_buffer_ack); - for (unsigned int i = 1; i <= 10000; i++) { - /* Both lists empty, as after the final completion was dequeued. */ - isp_vb2_buf_queue(&buf->vb.vb2_buf); - KUNIT_ASSERT_EQ(test, calls, i); - KUNIT_ASSERT_EQ(test, buf->tag, (u64)i); - KUNIT_ASSERT_TRUE(test, list_empty(&isp->bufs_pending)); - submitted = list_first_entry(&isp->bufs_submitted, struct isp_buffer, link); - KUNIT_ASSERT_PTR_EQ(test, submitted, buf); - list_del(&buf->link); + for (unsigned int format = 0; format < 2; format++) { + fmt->pixelformat = format ? V4L2_PIX_FMT_NV12 : V4L2_PIX_FMT_P010; + calls = 0; + for (unsigned int i = 1; i <= 10000; i++) { + /* Both lists empty, as after the last completion was dequeued. */ + isp_vb2_buf_queue(&buf->vb.vb2_buf); + KUNIT_ASSERT_EQ(test, calls, i); + if (!format) + KUNIT_ASSERT_EQ(test, buf->tag, (u64)i); + KUNIT_ASSERT_TRUE(test, list_empty(&isp->bufs_pending)); + submitted = list_first_entry(&isp->bufs_submitted, struct isp_buffer, link); + KUNIT_ASSERT_PTR_EQ(test, submitted, buf); + list_del(&buf->link); + } } } static struct kunit_case isp_v4l2_test_cases[] = { - KUNIT_CASE(isp_test_p010_last_lease_requeue), + KUNIT_CASE(isp_test_imx958_last_lease_requeue), KUNIT_CASE(isp_test_p010_report_defers_completion), KUNIT_CASE(isp_test_p010_mode_selection), KUNIT_CASE(isp_test_p010_format), diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index 35c30582ab81d..4fcde29410d2c 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -452,9 +452,9 @@ static void isp_vb2_buf_queue(struct vb2_buffer *vb) list_add_tail(&buf->link, &isp->bufs_pending); spin_unlock_irqrestore(&isp->buf_lock, flags); - /* A returned last lease must restart an empty P010 queue immediately. */ + /* Restart an empty IMX958 queue when its last returned lease is queued. */ if (test_bit(ISP_STATE_STREAMING, &isp->state) && - (isp_is_p010(isp) || !empty)) { + (isp_supports_p010(isp) || !empty)) { if (isp_submit_buffers(isp, false) && isp_is_p010(isp)) { apple_isp_firmware_halt(isp); isp_vb2_release_buffers(isp, VB2_BUF_STATE_ERROR); From 7f571507eb1a8140721067d50321a24efe58d59d Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:23:46 +1100 Subject: [PATCH 05/11] docs: media: fix J616s ISP protocol table markup Size the command table columns for the full H17 record sizes and fix the document title underline. The standalone RST parser now accepts the document with warnings treated as errors. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/drivers/apple-isp-j616s.rst | 58 ++++++++++--------- 1 file changed, 30 insertions(+), 28 deletions(-) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index 7413c0e866460..707ac2c22e6e7 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -1,7 +1,7 @@ .. SPDX-License-Identifier: GPL-2.0-only J616s ISP modes and capture protocol -================================== +==================================== Sources and scope ----------------- @@ -122,32 +122,32 @@ Command stream IO commands have a zero/status u32 followed by the opcode u32, then the parameters below. Sizes include that eight-byte header. ``ch`` is zero. -================ =================================== ==== =================== -Opcode Command Size Parameters -================ =================================== ==== =================== -0x0003 CONFIG_GET 0x1c Camera channel count -0x3206 SET_DSID_CLR_MULTI_BC_REG_BASE 0x50 8 slots, range 0x2fc -0x0000 START 0x0c Proxy 0x11, Linux 0 -0x010d CH_INFO_GET 0x190 ch -0x0106 CH_CAMERA_CONFIG_GET 0x120 ch, index (all) -0x0024 FLICKER_SENSOR_SET 0x0c 0 -0x013b CH_SBS_ENABLE 0x10 ch, 1 -0x0107 CH_CAMERA_CONFIG_SELECT 0x10 ch, selected config -0x010e CH_BUFFER_RECYCLE_MODE_SET 0x10 ch, 1 (empty only) -0x010f CH_BUFFER_RECYCLE_START 0x0c ch -0x080c CH_CROP_SCL1_SET 0x1c ch, x, y, width, height -0x0b09 CH_OUTPUT_CONFIG_SCL1_SET 0x38 ch, output geometry -0x0208 CH_AE_FRAME_RATE_MAX_SET 0x10 ch, fps * 256 -0x020a CH_AE_FRAME_RATE_MIN_SET 0x10 ch, min_fps * 256 -0x0117 CH_BUFFER_POOL_CONFIG_SET (meta) 0x9c ch, pool 0, count 8 -0x0117 CH_BUFFER_POOL_CONFIG_SET (output) 0x9c ch, pool 9, count 2 -0x0125 CH_LOCAL_RAW_BUFFER_ENABLE 0x10 ch, 1 -0x0b0d CH_PREVIEW_STREAM_SET 0x10 ch, 1 -0x0138 CH_MASTER_SLAVE_SYNC_MODE_SET 0x10 ch, 0 -0x0100 CH_START 0x0c ch -0x0101 CH_STOP 0x0c ch -0x0104 CH_BUFFER_RETURN 0x0c ch -================ =================================== ==== =================== +====== ================================== ===== ======================= +Opcode Command Size Parameters +====== ================================== ===== ======================= +0x0003 CONFIG_GET 0x1c Camera channel count +0x3206 SET_DSID_CLR_MULTI_BC_REG_BASE 0x50 8 slots, range 0x2fc +0x0000 START 0x0c Proxy 0x11, Linux 0 +0x010d CH_INFO_GET 0x190 ch +0x0106 CH_CAMERA_CONFIG_GET 0x120 ch, index (all) +0x0024 FLICKER_SENSOR_SET 0x0c 0 +0x013b CH_SBS_ENABLE 0x10 ch, 1 +0x0107 CH_CAMERA_CONFIG_SELECT 0x10 ch, selected config +0x010e CH_BUFFER_RECYCLE_MODE_SET 0x10 ch, 1 (empty only) +0x010f CH_BUFFER_RECYCLE_START 0x0c ch +0x080c CH_CROP_SCL1_SET 0x1c ch, x, y, width, height +0x0b09 CH_OUTPUT_CONFIG_SCL1_SET 0x38 ch, output geometry +0x0208 CH_AE_FRAME_RATE_MAX_SET 0x10 ch, fps * 256 +0x020a CH_AE_FRAME_RATE_MIN_SET 0x10 ch, min_fps * 256 +0x0117 CH_BUFFER_POOL_CONFIG_SET (meta) 0x9c ch, pool 0, count 8 +0x0117 CH_BUFFER_POOL_CONFIG_SET (output) 0x9c ch, pool 9, count 2 +0x0125 CH_LOCAL_RAW_BUFFER_ENABLE 0x10 ch, 1 +0x0b0d CH_PREVIEW_STREAM_SET 0x10 ch, 1 +0x0138 CH_MASTER_SLAVE_SYNC_MODE_SET 0x10 ch, 0 +0x0100 CH_START 0x0c ch +0x0101 CH_STOP 0x0c ch +0x0104 CH_BUFFER_RETURN 0x0c ch +====== ================================== ===== ======================= The DSID bases are 0x2201d4000, 0x2221d4000, 0x2241d4000, 0x2261d4000, then four U64_MAX sentinels. Linux retains its existing boot-time PMP and @@ -205,7 +205,9 @@ is source-level ABI evidence, not a new live qualification of P010. selecting P010 would violate the proxy's guarded startup lifecycle. The new config query explicitly exchanges 0x120, rather than the unused legacy 0xdc current-config helper. Header/channel/ index checks and duplicated geometry checks guard decoding. The H17 - appended bytes remain uninterpreted. + appended bytes remain uninterpreted. The old preview validator also checks + CH_INFO_GET byte 0x138; the current Linux helper retains only the common + 0x118-byte prefix of its 0x190-byte reply and does not make that check. * Metadata: CH_INFO_GET reports 0x4d00 on both paths. The proxy rounds its metadata backing/pool to 0x8000; P010 does the same even on a 4 KiB host. NV12 retains the current driver's 0x4d00 pool and 16-buffer capacity. From 946458f3814ffd01ef9f2f3737e7c455363112f7 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 13:48:05 +1100 Subject: [PATCH 06/11] media: apple: isp: accept full-width P010 completion address pairs The buffer wire carries 64-bit addresses, while the proxy observed zero-extended low words on its firmware. Accept either complete canonical pair for a submitted lease and reject mixed or unrelated upper bits. Keep pool, plane and fresh-tag checks. Also assert that NV12 stays first in format enumeration. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../driver-api/media/drivers/apple-isp-j616s.rst | 5 +++-- drivers/media/platform/apple/isp/isp-v4l2-test.c | 14 ++++++++++++-- drivers/media/platform/apple/isp/isp-v4l2.c | 11 +++++++---- 3 files changed, 22 insertions(+), 8 deletions(-) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index 707ac2c22e6e7..f92cecf30f3e5 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -81,8 +81,9 @@ Linux passes its actual contiguous extents to the rendered pool instead. P010 uses the proxy's eight metadata slots (0x8000 bytes each) and two rendered slots. The queue limits userspace to two rendered buffers, matching -the pool capacity. Completion matches both zero-extended low-32-bit plane -addresses, plane count, pool 9 and a fresh 64-bit submission tag; an old tag +the pool capacity. Completion matches either both full submitted addresses +or both zero-extended low-32-bit plane addresses, plus plane count, pool 9 +and a fresh 64-bit submission tag; an old tag cannot release a new lease. Buffer completion is published to videobuf2 only after the firmware report ACK and doorbell have been published. NV12 pool sizes and the T8140/Neo formats, presets and command path retain diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c index 58af78581bf7e..6af295a7da23e 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2-test.c +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -166,13 +166,16 @@ static void isp_test_p010_admission(struct kunit *test) struct apple_isp *isp = isp_test_alloc(test); struct isp_format *fmt = isp_get_current_format(isp); struct isp_preset preset = { .output_dim = {1280, 720}, .p010 = true }; - struct v4l2_fmtdesc desc = { .index = 1 }; + struct v4l2_fmtdesc desc = { .index = 0 }; isp->presets = &preset; isp->num_presets = 1; isp->hw = &isp_test_t6040; fmt->version = 0x958; KUNIT_ASSERT_EQ(test, isp_enum_pixelformat(isp, &desc), 0); + KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_NV12); + desc.index = 1; + KUNIT_ASSERT_EQ(test, isp_enum_pixelformat(isp, &desc), 0); KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_P010); desc.index = 2; KUNIT_EXPECT_EQ(test, isp_enum_pixelformat(isp, &desc), -EINVAL); @@ -203,8 +206,15 @@ static void isp_test_p010_completion_identity(struct kunit *test) KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); desc.iovas[1]--; desc.iovas[0] |= BIT_ULL(40); + /* Reject mixed encodings, accept both exact full submitted addresses. */ + KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.iovas[1] |= BIT_ULL(40); + KUNIT_EXPECT_TRUE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); + desc.iovas[0] |= BIT_ULL(41); + desc.iovas[1] |= BIT_ULL(41); KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); - desc.iovas[0] &= ~BIT_ULL(40); + desc.iovas[0] = (u32)desc.iovas[0]; + desc.iovas[1] = (u32)desc.iovas[1]; desc.pool_type = CISP_POOL_TYPE_RENDERED; KUNIT_EXPECT_FALSE(test, isp_p010_buffer_matches(&fmt, &buf, &desc)); desc.pool_type = CISP_POOL_TYPE_RENDERED_SCL1; diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index 4fcde29410d2c..b5bfea73e2d82 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -59,11 +59,14 @@ static bool isp_p010_buffer_matches(const struct isp_format *fmt, const struct isp_buffer *buf, const struct isp_buflist_buffer *desc) { - /* H17 reports zero-extended low 32 bits, not the submitted high IOVA. */ + u64 y = buf->surfs[0].iova; + u64 uv = y + fmt->plane_size[0]; + bool full = desc->iovas[0] == y && desc->iovas[1] == uv; + bool low = desc->iovas[0] == (u32)y && desc->iovas[1] == (u32)uv; + + /* Accept the submitted pair or the zero-extended pair observed on H17. */ return desc->num_planes == 2 && - desc->pool_type == CISP_POOL_TYPE_RENDERED_SCL1 && - desc->iovas[0] == (u32)buf->surfs[0].iova && - desc->iovas[1] == (u32)(buf->surfs[0].iova + fmt->plane_size[0]) && + desc->pool_type == CISP_POOL_TYPE_RENDERED_SCL1 && (full || low) && !desc->iovas[2] && !desc->iovas[3] && desc->tag == buf->tag; } From e7b7607ea8fd317858c96c6d701cbc9cff355db3 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 14:03:02 +1100 Subject: [PATCH 07/11] media: apple: isp: remove parameter-controlled format exposure Remove the inherited multiplanar module parameter. T6040/IMX958 supports normal V4L2 negotiation for contiguous NV12 and P010 through either API, with one memory plane containing the two ISP components. Keep NV12 first and initial; other sensors retain their shipped single-planar API. Do not advertise the old non-contiguous NV12M alias. Cache complete sensor configuration replies with the video state and print them on the standard log-status ioctl, so collecting the firmware list needs no driver or boot parameter. Free the cache on probe failure and final video release, and test failed discovery without publishing partial state. No module/kernel parameters or platform Kconfig options are added. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/drivers/apple-isp-j616s.rst | 25 ++++--- .../media/platform/apple/isp/isp-cam-test.c | 41 +++++++++++ drivers/media/platform/apple/isp/isp-cam.c | 20 ++++-- drivers/media/platform/apple/isp/isp-drv.c | 1 + drivers/media/platform/apple/isp/isp-drv.h | 5 +- .../media/platform/apple/isp/isp-v4l2-test.c | 14 +++- drivers/media/platform/apple/isp/isp-v4l2.c | 70 ++++++------------- 7 files changed, 111 insertions(+), 65 deletions(-) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index f92cecf30f3e5..d5342c1d796af 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -46,9 +46,9 @@ Mode and format selection On T6040/IMX958, probe queries every advertised configuration, bounded at 64, using the 0x120-byte H17 ``CH_CAMERA_CONFIG_GET`` exchange. The complete list -is available through dynamic debug at probe and cached for -``VIDIOC_LOG_STATUS`` (``v4l2-ctl --log-status``), without sending commands -during a stream. The log includes +is available through dynamic debug at probe and cached in full for +``VIDIOC_LOG_STATUS`` (``v4l2-ctl --log-status``), without sending firmware +commands. Collect this diagnostic dump before starting a stress run. It includes input/output dimensions and the sensor clock word; that word is not FPS. The board's existing config-5 1920x1080 and 1280x720 presets stay first and @@ -68,9 +68,9 @@ the runtime input geometry matches the source profile. An unspecified P010 size selects 1280x720 first. Larger P010 modes require a consistent firmware output reply. P010 uses two little-endian 16-bit component planes with ten useful MSB-aligned bits, and equal 64-byte-aligned Y/UV strides. UV immediately -follows Y, as required for contiguous P010. The optional multi-planar V4L2 -API also reports P010 as **one memory plane** containing both component -planes. The firmware still receives two component addresses. +follows Y, as required for contiguous P010. The standard multi-planar V4L2 +API on T6040/IMX958 reports both NV12 and P010 as **one memory plane** +containing both component planes. The firmware still receives two component addresses. At 1920x1080, stride is 3840, Y is 4147200 bytes, UV is 2073600 bytes and ``sizeimage`` is 6220800. At 1280x720, stride is 2560 and ``sizeimage`` is @@ -86,8 +86,17 @@ or both zero-extended low-32-bit plane addresses, plus plane count, pool 9 and a fresh 64-bit submission tag; an old tag cannot release a new lease. Buffer completion is published to videobuf2 only after the firmware report ACK and doorbell have been published. -NV12 pool sizes and the T8140/Neo formats, presets and command path retain -their existing behavior. +NV12 pool sizes and the T8140/Neo default formats, presets and command path +retain their existing behavior. + +The inherited ``multiplanar`` module parameter is removed. T6040/IMX958 +supports standard single-planar and multi-planar V4L2 negotiation for these +contiguous formats without a parameter; the non-contiguous NV12M alias is not +advertised. Other sensors retain their shipped single-planar API. No driver +module/kernel parameter, platform flag or per-platform Kconfig option selects +support. NV12 stays first and is the initial format because browsers/WebRTC +and many applications do not support P010 or simply choose the first format. +Applications can select P010 explicitly with ``VIDIOC_S_FMT``. Endpoints --------- diff --git a/drivers/media/platform/apple/isp/isp-cam-test.c b/drivers/media/platform/apple/isp/isp-cam-test.c index aef545b27c051..27c9e23f348aa 100644 --- a/drivers/media/platform/apple/isp/isp-cam-test.c +++ b/drivers/media/platform/apple/isp/isp-cam-test.c @@ -186,9 +186,15 @@ static int isp_test_config_reply(struct apple_isp *isp, void *args, u32 insize, config->out_height2 = config->out_height; if (config->preset == 2) config->out_height2 = 2160; + reply->unknown[sizeof(reply->unknown) - 1] = 0x5a; return 0; } +static void isp_test_free_configs(void *ptr) +{ + kfree(ptr); +} + static void isp_test_sensor_config_list(struct kunit *test) { struct apple_isp *isp = isp_test_capture_alloc(test); @@ -199,6 +205,7 @@ static void isp_test_sensor_config_list(struct kunit *test) .output_dim = {1280, 720}, .crop_size = {1280, 720} }, }; u32 count = 0; + int err; isp->cmd_virt = kunit_kzalloc(test, 0x120, GFP_KERNEL); KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); @@ -207,6 +214,10 @@ static void isp_test_sensor_config_list(struct kunit *test) test->priv = &count; kunit_activate_static_stub(test, isp_cmd_send, isp_test_config_reply); KUNIT_ASSERT_EQ(test, isp_ch_cache_sensor_configs(isp, 0, 6), 0); + err = kunit_add_action_or_reset(test, isp_test_free_configs, isp->sensor_configs); + KUNIT_ASSERT_EQ(test, err, 0); + KUNIT_EXPECT_EQ(test, get_unaligned_le32(isp->sensor_configs[5].reply + 0xc), 5); + KUNIT_EXPECT_EQ(test, isp->sensor_configs[5].reply[ISP_SENSOR_CONFIG_SIZE - 1], 0x5a); KUNIT_EXPECT_EQ(test, count, 6); KUNIT_EXPECT_EQ(test, isp->num_sensor_configs, 6); KUNIT_EXPECT_EQ(test, isp->num_presets, 3); @@ -223,6 +234,35 @@ static void isp_test_sensor_config_list(struct kunit *test) KUNIT_EXPECT_EQ(test, count, 6); } +static int isp_test_bad_config_reply(struct apple_isp *isp, void *args, u32 insize, + u32 outsize, int timeout) +{ + struct cmd_ch_camera_config_h17 *reply = isp->cmd_virt; + + isp_test_config_reply(isp, args, insize, outsize, timeout); + reply->config.preset = 65; + return 0; +} + +static void isp_test_sensor_config_failure(struct kunit *test) +{ + struct apple_isp *isp = isp_test_capture_alloc(test); + struct isp_preset preset = {}; + u32 count = 0; + + isp->cmd_virt = kunit_kzalloc(test, ISP_SENSOR_CONFIG_SIZE, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, isp->cmd_virt); + isp->presets = &preset; + isp->num_presets = 1; + test->priv = &count; + kunit_activate_static_stub(test, isp_cmd_send, isp_test_bad_config_reply); + KUNIT_EXPECT_EQ(test, isp_ch_cache_sensor_configs(isp, 0, 6), -EIO); + KUNIT_EXPECT_EQ(test, count, 1); + KUNIT_EXPECT_EQ(test, isp->num_sensor_configs, 0); + KUNIT_EXPECT_PTR_EQ(test, isp->sensor_configs, NULL); + KUNIT_EXPECT_PTR_EQ(test, isp->presets, &preset); +} + struct isp_test_ack_state { unsigned int fault, calls; }; @@ -282,6 +322,7 @@ static struct kunit_case isp_cam_test_cases[] = { KUNIT_CASE(isp_test_command_ack_failure_latches), KUNIT_CASE(isp_test_sensor_geometry), KUNIT_CASE(isp_test_sensor_config_list), + KUNIT_CASE(isp_test_sensor_config_failure), KUNIT_CASE(isp_test_preview_stream), KUNIT_CASE(isp_test_preview_command_failures), {} diff --git a/drivers/media/platform/apple/isp/isp-cam.c b/drivers/media/platform/apple/isp/isp-cam.c index 59f19e571bb64..a5cbb2b4d8eb0 100644 --- a/drivers/media/platform/apple/isp/isp-cam.c +++ b/drivers/media/platform/apple/isp/isp-cam.c @@ -233,6 +233,7 @@ static bool isp_sensor_config_usable(const struct cmd_ch_camera_config *config) static int isp_ch_cache_sensor_configs(struct apple_isp *isp, u32 ch, u32 count) { struct cmd_ch_camera_config_h17 args; + struct isp_sensor_config *configs; struct isp_preset *presets; unsigned int num = isp->num_presets; int err; @@ -247,19 +248,25 @@ static int isp_ch_cache_sensor_configs(struct apple_isp *isp, u32 ch, u32 count) if (!presets) return -ENOMEM; memcpy(presets, isp->presets, num * sizeof(*presets)); + configs = kcalloc(count, sizeof(*configs), GFP_KERNEL); + if (!configs) + return -ENOMEM; for (u32 i = 0; i < count; i++) { struct cmd_ch_camera_config *config = &args.config; - struct isp_sensor_config *cached = &isp->sensor_configs[i]; + struct isp_sensor_config *cached = &configs[i]; bool duplicate = false; err = isp_cmd_ch_camera_config_get(isp, ch, i, &args); if (err) - return err; + goto free_configs; if (config->opcode != (u64)CISP_CMD_CH_CAMERA_CONFIG_GET << 32 || - config->chan != ch || config->preset != i) - return -EIO; + config->chan != ch || config->preset != i) { + err = -EIO; + goto free_configs; + } + memcpy(cached->reply, &args, sizeof(args)); cached->input_dim = (struct coord){config->in_width, config->in_height}; cached->output_dim = (struct coord){config->out_width, config->out_height}; cached->sensor_clk = config->sensor_clk; @@ -313,10 +320,15 @@ static int isp_ch_cache_sensor_configs(struct apple_isp *isp, u32 ch, u32 count) .p010 = true, }; } + isp->sensor_configs = configs; isp->num_sensor_configs = count; isp->presets = presets; isp->num_presets = num; return 0; + +free_configs: + kfree(configs); + return err; } static int isp_ch_cache_sensor_info(struct apple_isp *isp, u32 ch) diff --git a/drivers/media/platform/apple/isp/isp-drv.c b/drivers/media/platform/apple/isp/isp-drv.c index 04c50011abda1..084cca4dc3707 100644 --- a/drivers/media/platform/apple/isp/isp-drv.c +++ b/drivers/media/platform/apple/isp/isp-drv.c @@ -586,6 +586,7 @@ static int apple_isp_probe(struct platform_device *pdev) detach_genpd: apple_isp_detach_genpd(isp); free_isp: + kfree(isp->sensor_configs); /* Nothing else can reach it before the video device is registered. */ kfree(isp); return err; diff --git a/drivers/media/platform/apple/isp/isp-drv.h b/drivers/media/platform/apple/isp/isp-drv.h index db042f94aed47..dbe6b6ed5a2c4 100644 --- a/drivers/media/platform/apple/isp/isp-drv.h +++ b/drivers/media/platform/apple/isp/isp-drv.h @@ -31,6 +31,7 @@ /* used to limit the user space buffers to the buffer_pool_config */ #define ISP_MAX_BUFFERS 16 #define ISP_MAX_SENSOR_CONFIGS 64 +#define ISP_SENSOR_CONFIG_SIZE 0x120 /* capture metadata buffers, for firmware that has that pool */ #define ISP_CAPMETA_BUFFERS 8 @@ -265,6 +266,7 @@ struct isp_sensor_config { struct coord output_dim; u32 sensor_clk; bool usable; + u8 reply[ISP_SENSOR_CONFIG_SIZE]; }; struct isp_format { @@ -289,7 +291,7 @@ struct apple_isp { int num_presets; /* Read-only sensor configuration replies, cached before video registration. */ unsigned int num_sensor_configs; - struct isp_sensor_config sensor_configs[ISP_MAX_SENSOR_CONFIGS]; + struct isp_sensor_config *sensor_configs; int num_channels; struct isp_format fmts[ISP_MAX_CHANNELS]; @@ -303,7 +305,6 @@ struct apple_isp { struct vb2_queue vbq; struct mutex video_lock; unsigned int sequence; - bool multiplanar; /* capture rate in frames per second, changes under video_lock */ unsigned int frame_rate; diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c index 6af295a7da23e..b3c8d26dfd4aa 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2-test.c +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -102,6 +102,12 @@ static void isp_test_create_bufs_limit(struct kunit *test) KUNIT_EXPECT_EQ(test, isp_test_setup(test, ISP_MAX_BUFFERS, &nbuffers, &num_planes, sizes), -ENOBUFS); + /* The contiguous API must refuse a second independent memory plane. */ + nbuffers = 1; + num_planes = 2; + KUNIT_EXPECT_EQ(test, isp_test_setup(test, 0, &nbuffers, &num_planes, + sizes), -EINVAL); + /* A plane smaller than a frame is still refused. */ nbuffers = 1; num_planes = 1; @@ -158,6 +164,10 @@ static void isp_test_p010_format(struct kunit *test) KUNIT_EXPECT_EQ(test, fmt->total_size, (size_t)1382400); KUNIT_EXPECT_EQ(test, isp_num_rendered(isp), ISP_MAX_BUFFERS); KUNIT_EXPECT_EQ(test, isp_num_meta(isp), ISP_MAX_BUFFERS); + isp_get_mp_pix_format(isp, &f, fmt); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.pixelformat, V4L2_PIX_FMT_NV12); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.num_planes, 1); + KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.plane_fmt[0].sizeimage, 1382400); } static void isp_test_p010_admission(struct kunit *test) @@ -179,9 +189,7 @@ static void isp_test_p010_admission(struct kunit *test) KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_P010); desc.index = 2; KUNIT_EXPECT_EQ(test, isp_enum_pixelformat(isp, &desc), -EINVAL); - isp->multiplanar = true; - KUNIT_ASSERT_EQ(test, isp_enum_pixelformat(isp, &desc), 0); - KUNIT_EXPECT_EQ(test, desc.pixelformat, V4L2_PIX_FMT_P010); + desc.index = 1; isp->hw = &neo; KUNIT_EXPECT_EQ(test, isp_enum_pixelformat(isp, &desc), -EINVAL); KUNIT_ASSERT_EQ(test, isp_set_preset(isp, fmt, &preset, V4L2_PIX_FMT_P010), 0); diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index b5bfea73e2d82..bc270e06be21f 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -27,10 +27,6 @@ /* Capture rates offered to userspace, in frames per second */ static const unsigned int isp_frame_rates[] = { 30, 25, 24, 20, 15 }; -static bool multiplanar = false; -module_param(multiplanar, bool, 0644); -MODULE_PARM_DESC(multiplanar, "Enable multiplanar API"); - struct isp_buflist_buffer { u64 iovas[ISP_MAX_PLANES]; u32 flags[ISP_MAX_PLANES]; @@ -362,7 +358,7 @@ static int isp_vb2_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers, isp_num_rendered(isp) - q_num_bufs); if (*num_planes) { - if (sizes[0] < fmt->total_size) + if (*num_planes != 1 || sizes[0] < fmt->total_size) return -EINVAL; return 0; @@ -693,25 +689,13 @@ static int isp_vidioc_querycap(struct file *file, void *priv, static int isp_enum_pixelformat(struct apple_isp *isp, struct v4l2_fmtdesc *f) { - switch (f->index) { - case 0: + /* Browser/WebRTC compatibility: the shipped first format remains NV12. */ + if (!f->index) f->pixelformat = V4L2_PIX_FMT_NV12; - break; - case 1: - if (isp->multiplanar) { - f->pixelformat = V4L2_PIX_FMT_NV12M; - break; - } - fallthrough; - case 2: - if (!isp_has_p010(isp) || f->index != (isp->multiplanar ? 2 : 1)) - return -EINVAL; + else if (f->index == 1 && isp_has_p010(isp)) f->pixelformat = V4L2_PIX_FMT_P010; - break; - default: + else return -EINVAL; - } - return 0; } @@ -729,7 +713,6 @@ static int isp_vidioc_enum_framesizes(struct file *file, void *fh, u32 index = 0; if (f->pixel_format != V4L2_PIX_FMT_NV12 && - f->pixel_format != V4L2_PIX_FMT_NV12M && !(f->pixel_format == V4L2_PIX_FMT_P010 && isp_has_p010(isp))) return -EINVAL; @@ -760,7 +743,6 @@ static int isp_vidioc_enum_frameintervals(struct file *file, void *fh, return -EINVAL; if (interval->pixel_format != V4L2_PIX_FMT_NV12 && - interval->pixel_format != V4L2_PIX_FMT_NV12M && !(interval->pixel_format == V4L2_PIX_FMT_P010 && isp_has_p010(isp))) return -EINVAL; @@ -827,27 +809,17 @@ static inline void isp_get_mp_pix_format(struct apple_isp *isp, { f->fmt.pix_mp.width = fmt->preset->output_dim.x; f->fmt.pix_mp.height = fmt->preset->output_dim.y; - f->fmt.pix_mp.num_planes = fmt->num_planes; - for (int i = 0; i < fmt->num_planes; i++) { - f->fmt.pix_mp.plane_fmt[i].sizeimage = fmt->plane_size[i]; - f->fmt.pix_mp.plane_fmt[i].bytesperline = fmt->strides[i]; - } - + /* Both contiguous formats have one memory plane and two ISP components. */ + f->fmt.pix_mp.num_planes = 1; + memset(f->fmt.pix_mp.plane_fmt, 0, sizeof(f->fmt.pix_mp.plane_fmt)); + f->fmt.pix_mp.plane_fmt[0].sizeimage = fmt->total_size; + f->fmt.pix_mp.plane_fmt[0].bytesperline = fmt->strides[0]; f->fmt.pix_mp.field = V4L2_FIELD_NONE; - f->fmt.pix_mp.pixelformat = V4L2_PIX_FMT_NV12M; - /* P010 is contiguous, including through the multi-planar API. */ - if (fmt->pixelformat == V4L2_PIX_FMT_P010) { - f->fmt.pix_mp.pixelformat = V4L2_PIX_FMT_P010; - f->fmt.pix_mp.num_planes = 1; - f->fmt.pix_mp.plane_fmt[0].sizeimage = fmt->total_size; - memset(&f->fmt.pix_mp.plane_fmt[1], 0, - sizeof(f->fmt.pix_mp.plane_fmt[1])); - } + f->fmt.pix_mp.pixelformat = fmt->pixelformat; f->fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709; f->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_709; f->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_709; - if (fmt->pixelformat == V4L2_PIX_FMT_P010) - f->fmt.pix_mp.quantization = V4L2_QUANTIZATION_LIM_RANGE; + f->fmt.pix_mp.quantization = V4L2_QUANTIZATION_LIM_RANGE; } static int isp_vidioc_get_format(struct file *file, void *fh, @@ -911,7 +883,7 @@ static int isp_vidioc_get_format_mplane(struct file *file, void *fh, struct apple_isp *isp = video_drvdata(file); struct isp_format *fmt = isp_get_current_format(isp); - if (!isp->multiplanar) + if (!isp_supports_p010(isp)) return -ENOTTY; isp_get_mp_pix_format(isp, f, fmt); @@ -927,7 +899,7 @@ static int isp_vidioc_set_format_mplane(struct file *file, void *fh, struct isp_preset *preset; int err; - if (!isp->multiplanar) + if (!isp_supports_p010(isp)) return -ENOTTY; if (vb2_is_busy(&isp->vbq)) @@ -954,7 +926,7 @@ static int isp_vidioc_try_format_mplane(struct file *file, void *fh, struct isp_preset *preset; int err; - if (!isp->multiplanar) + if (!isp_supports_p010(isp)) return -ENOTTY; preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height, @@ -1001,7 +973,7 @@ static int isp_vidioc_get_param(struct file *file, void *fh, struct apple_isp *isp = video_drvdata(file); if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE && - (!isp->multiplanar || + (!isp_supports_p010(isp) || a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)) return -EINVAL; @@ -1019,7 +991,7 @@ static int isp_vidioc_set_param(struct file *file, void *fh, struct v4l2_fract *tpf = &a->parm.capture.timeperframe; if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE && - (!isp->multiplanar || + (!isp_supports_p010(isp) || a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)) return -EINVAL; @@ -1055,6 +1027,9 @@ static int isp_vidioc_log_status(struct file *file, void *fh) config->output_dim.x, config->output_dim.y, config->sensor_clk, config->usable ? "" : " (unverified geometry)"); + /* This explicit diagnostic ioctl replaces a probe-time parameter. */ + print_hex_dump(KERN_INFO, "apple-isp config: ", DUMP_PREFIX_OFFSET, + 16, 4, config->reply, sizeof(config->reply), false); } return 0; } @@ -1120,6 +1095,7 @@ static void apple_isp_release(struct v4l2_device *v4l2_dev) v4l2_ctrl_handler_free(&isp->ctrl_handler); media_device_cleanup(&isp->mdev); + kfree(isp->sensor_configs); kfree(isp); } @@ -1206,8 +1182,6 @@ int apple_isp_setup_video(struct apple_isp *isp) goto media_cleanup; } - isp->multiplanar = multiplanar; - /* * Auto exposure is always on. By default it holds the frame rate, as * V4L2 specifies; with exposure priority it may slow down in low light. @@ -1239,7 +1213,7 @@ int apple_isp_setup_video(struct apple_isp *isp) vdev->fops = &isp_v4l2_fops; vdev->ioctl_ops = &isp_v4l2_ioctl_ops; vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING; - if (isp->multiplanar) + if (isp_supports_p010(isp)) vdev->device_caps |= V4L2_CAP_VIDEO_CAPTURE_MPLANE; vdev->entity.flags = MEDIA_ENT_FL_DEFAULT; vdev->v4l2_dev = &isp->v4l2_dev; From f57db613d65dd826437a64f190e57135063f2080 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 14:59:51 +1100 Subject: [PATCH 08/11] media: apple: isp: copy complete IPC messages with FORTIFY enabled Acquire the validity word, copy the complete message object after the DMA read barrier and restore that captured word. Copying seven payload words through the arg1 field violates FORTIFY field bounds. Cover all eight copied words and keep the ownership ordering unchanged. Assisted-by: LLM Signed-off-by: Ryan Murray --- drivers/media/platform/apple/isp/isp-ipc-test.c | 16 ++++++++++++++++ drivers/media/platform/apple/isp/isp-ipc.c | 5 +++-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/drivers/media/platform/apple/isp/isp-ipc-test.c b/drivers/media/platform/apple/isp/isp-ipc-test.c index 8b6decd63f274..caa9e92f735f1 100644 --- a/drivers/media/platform/apple/isp/isp-ipc-test.c +++ b/drivers/media/platform/apple/isp/isp-ipc-test.c @@ -188,7 +188,23 @@ static void isp_test_late_ack_keeps_cursor(struct kunit *test) KUNIT_EXPECT_EQ(test, t->chan->cursor, 1); } +static void isp_test_message_payload_copy(struct kunit *test) +{ + struct isp_sm_test *t = isp_sm_test_alloc(test); + struct isp_message response = { + .arg0 = 0x5000001, .arg1 = 1, .arg2 = 2, .arg3 = 3, + .arg4 = 4, .arg5 = 5, .arg6 = 6, .arg7 = 7, + }; + struct isp_message copied = {}; + + isp_get_current_format(t->isp)->pixelformat = V4L2_PIX_FMT_P010; + t->chan->virt = &response; + chan_read_msg_index(t->isp, t->chan, &copied, 0); + KUNIT_EXPECT_MEMEQ(test, &copied, &response, sizeof(response)); +} + static struct kunit_case isp_ipc_test_cases[] = { + KUNIT_CASE(isp_test_message_payload_copy), KUNIT_CASE(isp_test_late_ack_keeps_cursor), KUNIT_CASE(isp_test_command_ring_wrap), KUNIT_CASE(isp_test_sm_free_releases), diff --git a/drivers/media/platform/apple/isp/isp-ipc.c b/drivers/media/platform/apple/isp/isp-ipc.c index 92cfc6ab35f5b..74cc717eadc07 100644 --- a/drivers/media/platform/apple/isp/isp-ipc.c +++ b/drivers/media/platform/apple/isp/isp-ipc.c @@ -72,11 +72,12 @@ static inline void chan_read_msg_index(struct apple_isp *isp, { if (isp_is_p010(isp)) { u64 *slot = chan_msg_virt(chan, index); + u64 arg0 = READ_ONCE(*slot); /* Acquire validity before consuming the firmware-owned payload. */ - msg->arg0 = READ_ONCE(*slot); dma_rmb(); - memcpy(&msg->arg1, slot + 1, sizeof(*msg) - sizeof(msg->arg0)); + memcpy(msg, slot, sizeof(*msg)); + msg->arg0 = arg0; } else { memcpy(msg, chan_msg_virt(chan, index), sizeof(*msg)); } From 1bd911c107446e95e7d4423e0f37a9afe674ebc0 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 15:03:19 +1100 Subject: [PATCH 09/11] media: apple: isp: use the contiguous single-planar capture API Remove the parameter-gated multi-planar hooks, which changed the queue type without reinitialising its cached plane semantics. Both contiguous NV12 and P010 use the existing single-planar queue. Keep format negotiation explicit through S_FMT. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/drivers/apple-isp-j616s.rst | 13 ++- .../media/platform/apple/isp/isp-v4l2-test.c | 11 +-- drivers/media/platform/apple/isp/isp-v4l2.c | 95 +------------------ 3 files changed, 11 insertions(+), 108 deletions(-) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index d5342c1d796af..bb0b9433a6119 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -68,9 +68,9 @@ the runtime input geometry matches the source profile. An unspecified P010 size selects 1280x720 first. Larger P010 modes require a consistent firmware output reply. P010 uses two little-endian 16-bit component planes with ten useful MSB-aligned bits, and equal 64-byte-aligned Y/UV strides. UV immediately -follows Y, as required for contiguous P010. The standard multi-planar V4L2 -API on T6040/IMX958 reports both NV12 and P010 as **one memory plane** -containing both component planes. The firmware still receives two component addresses. +follows Y, as required for contiguous P010. Both NV12 and P010 use the +standard single-planar V4L2 API with one memory plane containing both +component planes. The firmware still receives two component addresses. At 1920x1080, stride is 3840, Y is 4147200 bytes, UV is 2073600 bytes and ``sizeimage`` is 6220800. At 1280x720, stride is 2560 and ``sizeimage`` is @@ -89,10 +89,9 @@ only after the firmware report ACK and doorbell have been published. NV12 pool sizes and the T8140/Neo default formats, presets and command path retain their existing behavior. -The inherited ``multiplanar`` module parameter is removed. T6040/IMX958 -supports standard single-planar and multi-planar V4L2 negotiation for these -contiguous formats without a parameter; the non-contiguous NV12M alias is not -advertised. Other sensors retain their shipped single-planar API. No driver +The inherited ``multiplanar`` module parameter and its inconsistent queue-type +hooks are removed. All sensors use the shipped single-planar V4L2 API; the +non-contiguous NV12M alias is not advertised. No driver module/kernel parameter, platform flag or per-platform Kconfig option selects support. NV12 stays first and is the initial format because browsers/WebRTC and many applications do not support P010 or simply choose the first format. diff --git a/drivers/media/platform/apple/isp/isp-v4l2-test.c b/drivers/media/platform/apple/isp/isp-v4l2-test.c index b3c8d26dfd4aa..398e8e92356ab 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2-test.c +++ b/drivers/media/platform/apple/isp/isp-v4l2-test.c @@ -37,6 +37,8 @@ static void isp_test_queue_init(struct kunit *test) KUNIT_ASSERT_EQ(test, isp_vb2_queue_init(isp), 0); KUNIT_EXPECT_GE(test, isp->vbq.max_num_buffers, VB2_MAX_FRAME); + KUNIT_EXPECT_EQ(test, isp->vbq.type, V4L2_BUF_TYPE_VIDEO_CAPTURE); + KUNIT_EXPECT_FALSE(test, isp->vbq.is_multiplanar); vb2_queue_release(&isp->vbq); } @@ -140,10 +142,6 @@ static void isp_test_p010_format(struct kunit *test) KUNIT_EXPECT_EQ(test, fmt->plane_size[0], (size_t)4147200); KUNIT_EXPECT_EQ(test, fmt->plane_size[1], (size_t)2073600); KUNIT_EXPECT_EQ(test, f.fmt.pix.quantization, V4L2_QUANTIZATION_LIM_RANGE); - isp_get_mp_pix_format(isp, &f, fmt); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.pixelformat, V4L2_PIX_FMT_P010); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.num_planes, 1); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.plane_fmt[0].sizeimage, 6220800); isp->vbq.drv_priv = isp; isp_test_set_num_buffers(test, &isp->vbq, 0); @@ -164,10 +162,7 @@ static void isp_test_p010_format(struct kunit *test) KUNIT_EXPECT_EQ(test, fmt->total_size, (size_t)1382400); KUNIT_EXPECT_EQ(test, isp_num_rendered(isp), ISP_MAX_BUFFERS); KUNIT_EXPECT_EQ(test, isp_num_meta(isp), ISP_MAX_BUFFERS); - isp_get_mp_pix_format(isp, &f, fmt); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.pixelformat, V4L2_PIX_FMT_NV12); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.num_planes, 1); - KUNIT_EXPECT_EQ(test, f.fmt.pix_mp.plane_fmt[0].sizeimage, 1382400); + } static void isp_test_p010_admission(struct kunit *test) diff --git a/drivers/media/platform/apple/isp/isp-v4l2.c b/drivers/media/platform/apple/isp/isp-v4l2.c index bc270e06be21f..ff0133e34dee5 100644 --- a/drivers/media/platform/apple/isp/isp-v4l2.c +++ b/drivers/media/platform/apple/isp/isp-v4l2.c @@ -803,25 +803,6 @@ static inline void isp_get_sp_pix_format(struct apple_isp *isp, f->fmt.pix.quantization = V4L2_QUANTIZATION_LIM_RANGE; } -static inline void isp_get_mp_pix_format(struct apple_isp *isp, - struct v4l2_format *f, - struct isp_format *fmt) -{ - f->fmt.pix_mp.width = fmt->preset->output_dim.x; - f->fmt.pix_mp.height = fmt->preset->output_dim.y; - /* Both contiguous formats have one memory plane and two ISP components. */ - f->fmt.pix_mp.num_planes = 1; - memset(f->fmt.pix_mp.plane_fmt, 0, sizeof(f->fmt.pix_mp.plane_fmt)); - f->fmt.pix_mp.plane_fmt[0].sizeimage = fmt->total_size; - f->fmt.pix_mp.plane_fmt[0].bytesperline = fmt->strides[0]; - f->fmt.pix_mp.field = V4L2_FIELD_NONE; - f->fmt.pix_mp.pixelformat = fmt->pixelformat; - f->fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709; - f->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_709; - f->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_709; - f->fmt.pix_mp.quantization = V4L2_QUANTIZATION_LIM_RANGE; -} - static int isp_vidioc_get_format(struct file *file, void *fh, struct v4l2_format *f) { @@ -877,69 +858,6 @@ static int isp_vidioc_try_format(struct file *file, void *fh, return 0; } -static int isp_vidioc_get_format_mplane(struct file *file, void *fh, - struct v4l2_format *f) -{ - struct apple_isp *isp = video_drvdata(file); - struct isp_format *fmt = isp_get_current_format(isp); - - if (!isp_supports_p010(isp)) - return -ENOTTY; - - isp_get_mp_pix_format(isp, f, fmt); - - return 0; -} - -static int isp_vidioc_set_format_mplane(struct file *file, void *fh, - struct v4l2_format *f) -{ - struct apple_isp *isp = video_drvdata(file); - struct isp_format *fmt = isp_get_current_format(isp); - struct isp_preset *preset; - int err; - - if (!isp_supports_p010(isp)) - return -ENOTTY; - - if (vb2_is_busy(&isp->vbq)) - return -EBUSY; - - preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height, - f->fmt.pix_mp.pixelformat); - err = isp_set_preset(isp, fmt, preset, f->fmt.pix_mp.pixelformat); - if (err) - return err; - - isp_get_mp_pix_format(isp, f, fmt); - - isp->vbq.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; - - return 0; -} - -static int isp_vidioc_try_format_mplane(struct file *file, void *fh, - struct v4l2_format *f) -{ - struct apple_isp *isp = video_drvdata(file); - struct isp_format fmt = *isp_get_current_format(isp); - struct isp_preset *preset; - int err; - - if (!isp_supports_p010(isp)) - return -ENOTTY; - - preset = isp_select_preset(isp, f->fmt.pix_mp.width, f->fmt.pix_mp.height, - f->fmt.pix_mp.pixelformat); - err = isp_set_preset(isp, &fmt, preset, f->fmt.pix_mp.pixelformat); - if (err) - return err; - - isp_get_mp_pix_format(isp, f, &fmt); - - return 0; -} - static int isp_vidioc_enum_input(struct file *file, void *fh, struct v4l2_input *inp) { @@ -972,9 +890,7 @@ static int isp_vidioc_get_param(struct file *file, void *fh, { struct apple_isp *isp = video_drvdata(file); - if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE && - (!isp_supports_p010(isp) || - a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)) + if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE) return -EINVAL; a->parm.capture.capability = V4L2_CAP_TIMEPERFRAME; @@ -990,9 +906,7 @@ static int isp_vidioc_set_param(struct file *file, void *fh, struct apple_isp *isp = video_drvdata(file); struct v4l2_fract *tpf = &a->parm.capture.timeperframe; - if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE && - (!isp_supports_p010(isp) || - a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)) + if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE) return -EINVAL; /* The rate is configured when the stream starts. */ @@ -1041,9 +955,6 @@ static const struct v4l2_ioctl_ops isp_v4l2_ioctl_ops = { .vidioc_g_fmt_vid_cap = isp_vidioc_get_format, .vidioc_s_fmt_vid_cap = isp_vidioc_set_format, .vidioc_try_fmt_vid_cap = isp_vidioc_try_format, - .vidioc_g_fmt_vid_cap_mplane = isp_vidioc_get_format_mplane, - .vidioc_s_fmt_vid_cap_mplane = isp_vidioc_set_format_mplane, - .vidioc_try_fmt_vid_cap_mplane = isp_vidioc_try_format_mplane, .vidioc_enum_framesizes = isp_vidioc_enum_framesizes, .vidioc_enum_frameintervals = isp_vidioc_enum_frameintervals, @@ -1213,8 +1124,6 @@ int apple_isp_setup_video(struct apple_isp *isp) vdev->fops = &isp_v4l2_fops; vdev->ioctl_ops = &isp_v4l2_ioctl_ops; vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING; - if (isp_supports_p010(isp)) - vdev->device_caps |= V4L2_CAP_VIDEO_CAPTURE_MPLANE; vdev->entity.flags = MEDIA_ENT_FL_DEFAULT; vdev->v4l2_dev = &isp->v4l2_dev; vdev->vfl_type = VFL_TYPE_VIDEO; From 7fc12d0823e95d8e8ecd44e504f1f6a59a4689a6 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 15:51:14 +1100 Subject: [PATCH 10/11] docs: media: audit the public J616s capture evidence and AOP map Credit Ryan\x27s m1n1 PR #20, https://github.com/aurora-silicon/m1n1/pull/20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f for the processed preview protocol. Record what each requested file establishes, the diagnostic RAW snapshot limitation and the missing per-config timing records. Distinguish the separate AOP audio sequence from ISP camera configuration. Assisted-by: LLM Signed-off-by: Ryan Murray --- .../media/drivers/apple-isp-j616s.rst | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index bb0b9433a6119..650a109cc7ad1 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -41,6 +41,59 @@ firmware-derived output modes still need capture validation on 26.6.2. Only our source and synthetic protocol tests are included; no Apple binary, setfile contents, calibration table or firmware image is added. +Public source audit +------------------- + +The pinned PR and its discussion were checked again for the RAW layout and +per-configuration timing. The reusable code establishes the processed path; +it does not contain a RAW Bayer packing/CFA contract or the 13 reply records: + +* ``isp_capture.py`` accepts only 1280x720 and 1920x1080 processed geometry. + Its generic pool-3 builder describes a single address, not the pixel packing, + CFA order or RAW receiver command sequence. The qualified preview uses pool 9. +* ``isp_pixels.py`` contains only an MSB-aligned P010 decoder, with no Bayer decoder. +* ``isp_sensor.py`` reads the sensor ID/count and queries an index at runtime. + It has neither saved per-index replies nor a frame-rate decoder. +* ``isp_profile.py`` and ``fixtures/isp_t6040_25g76.json`` describe boot segments, + allocation geometry and channels. The fixture has no camera-info or config replies. +* ``isp_vid.py`` and ``experiments/isp.py`` use the older fixed 720p NV12 receiver. + ``isp_cmd.py`` includes legacy sensor-0x248 examples and a temporary 3.75 FPS + AE request (0x3c0), alongside a comment describing the older 30 FPS request. + These are not IMX958 per-mode timing records. The qualified T6040 helper instead + requests 30/15 FPS AE limits; neither set supplies a measured nominal-rate table. +* ``isp_command.py`` implements ownership, command/reply and deadline handling. + ``src/isp.c`` prepares boot-time power and heap handoff, not a RAW capture client. +* ``docs/j616s-bringup.md`` lines 752-789 explicitly distinguish a stable RAW + snapshot after stop from a completed buffer report and say orchestration remains + private. Commit ``743e4e75`` adds that diagnostic RAW evidence as documentation; + it does not add a RAW receiver implementation. Lines 832-841 qualify one + completed 1080p P010 frame, with measured FPS still unqualified. + +These findings do not deny the recorded RAW snapshot. They mean a V4L2 Bayer +FOURCC, frame-completion contract and per-config rates cannot be derived from +these files without additional runtime replies or capture evidence. In particular, +the old 0x103 SIF example must not be treated as an IMX958 Bayer-format declaration. + +AOP endpoints and separate audio stream +-------------------------------------- + +The modern ``fw/aop/client.py`` map in PR 20 is 0x20 SPUApp, 0x21 wakehint, +0x22 audio, 0x23 voice trigger, 0x24 accelerometer, 0x25 gyroscope, 0x26 lid +angle, 0x27 ALS and four endpoints at 0x28-0x2b with unqualified roles. The +separate ENDPOINTS13 map assigns different roles and is not the T6040 map. + +The documented HQ audio sequence uses the advertised aop-audio service at +endpoint 0x22, EPIC version 2, checked attachment/property replies and owned +DART mappings. It first establishes LP readiness using adpx buffer property 300, +attachments lpai/pdm0/adpx, lpai pw1, lai attachment and lai/lpai runn. HQ then +attaches hpai, requests pw1, prepares stream-10 NS RX0 DMA and starts DMA before +requesting hpai pwrd. Matching reports and two full copies precede RX stop and +return through pw1 to idle. These are audio commands, not camera mode selectors. + +The qualified ISP preview helper contains no AOP command. This camera change +therefore does not replay the AOP audio stream or alter its Linux driver. The +ISP channel and command maps below describe the camera path independently. + Mode and format selection ------------------------- From 2114ed0c6245126cd90a2e5e96e2cddd4e0697e0 Mon Sep 17 00:00:00 2001 From: Ryan Murray Date: Mon, 26 Oct 2026 15:51:35 +1100 Subject: [PATCH 11/11] docs: media: fix AOP section heading width Correct the reStructuredText underline and verify the complete document with warnings treated as errors. Source protocol credit: Ryan Murray, m1n1 PR #20, https://github.com/aurora-silicon/m1n1/pull/20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f. Assisted-by: LLM Signed-off-by: Ryan Murray --- Documentation/driver-api/media/drivers/apple-isp-j616s.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst index 650a109cc7ad1..9a2bf93d7b059 100644 --- a/Documentation/driver-api/media/drivers/apple-isp-j616s.rst +++ b/Documentation/driver-api/media/drivers/apple-isp-j616s.rst @@ -75,7 +75,7 @@ these files without additional runtime replies or capture evidence. In particula the old 0x103 SIF example must not be treated as an IMX958 Bayer-format declaration. AOP endpoints and separate audio stream --------------------------------------- +----------------------------------------- The modern ``fw/aop/client.py`` map in PR 20 is 0x20 SPUApp, 0x21 wakehint, 0x22 audio, 0x23 voice trigger, 0x24 accelerometer, 0x25 gyroscope, 0x26 lid