feat: add rk3568 sim boards - #33
Open
Charliechen114514 wants to merge 29 commits into
Open
Conversation
First heterogeneous machine in the org: 4x Cortex-A55 + 4x Cortex-A76 all up via PSCI (real MIDRs), GICv3 with 8 redistributors, UART2 8250. Ubuntu 26.04 3GiB rootfs mounts and shells on the sim machine. Fixed a cross-board bug: QEMU gicv3 gpio-in n is the SPI index, not the INTID (gicv3_set_irq adds GIC_INTERNAL internally) - both machines were wired +32 off; rk3568 had been surviving on a polling fallback. boot-smoke.py is now dual-board ([board] [mode] [--check]); the QEMU patch is renamed qemu-sim-machines.patch carrying both machines.
The topeet board DT boots to shell on rk3588-lite (three assertions). Ported the rk3568 shadow methodology with rk3588 specifics: PLL lock is CON6 BIT15 (not CON1 BIT10), PLL blocks pre-written by exact offsets from clk-rk3588.c, p=0 self-heal for half-written set_rate, uart2 div prewrite, PMU status mirror. modify_dtb nops the fiq-debugger node and hands uart2 back to the 8250 (no BL31 -> no FIQ console). cpuidle.off=1 works around a TCG heterogeneous hard lockup (root cause open). rk3588 U-Boot dies at SCMI-over-SMCCC (needs BL31 services) - firmware interface emulation is the next topic. Both boards, eight modes green.
Assertions only proved 'reached shell and powered off' - the cpuidle lockup class fires 30s+ later, outside the window. SOAK=<seconds> now boots without the auto-poweroff and heartbeats the shell every 5s, asserting every beat. First SOAK run mis-diagnosed a 'MTTCG heterogeneous race' (with a thread=single crutch). The real bug was the feed code itself: beat delays were cumulative instead of a fixed cadence, so late beats fell past the timeout on BOTH boards. Fixed, crutch reverted: MTTCG passes 60s 12/12 and 120s 24/24 on rk3588-lite board, 60s 12/12 on rk3568. Only cpuidle.off=1 remains as a real workaround (that lockup had a watchdog panic on record).
Interactive use on rk3588 turned up 'clear: not found' - buildroot puts clear/vi/top and friends in usr/bin, which the slim initramfs never packed. Only the symlinks are needed (busybox itself is already in /bin): 108 links, +1KB total. Real files stay out - copying usr/bin wholesale is what blew the image up to 20MB and hit the 190MB unpack wall before. Verified in guest and by regression.
The full real-board experience: topeet DTB + Ubuntu ext4 on 8 cores, two assertions green. The real DTB has no virtio nodes (rootwait trap), so modify_dtb now grafts the four virtio_mmio stand-ins with fdt_add_subnode - QEMU pads the DTB buffer 2x+20K so there is room. Cross-board trap closed: rk3588's GIC is #interrupt-cells=<4> (the 4th cell is the PPI partition, 0 for SPIs) while rk3568 uses 3 - a 3-cell graft parses as malformed, IRQ index 0 returns ENXIO and the disk never appears. Kernel-visible symptom vs dump-verified DTB made it a one-line fix.
init=/bin/sh was a smoke-time shortcut, not a verdict on systemd. With init=/sbin/init on the merged real-DTB + Ubuntu path, systemd 259 boots Ubuntu 26.04 to the serial login prompt (console=ttyS2 auto-spawns serial-getty), adopts and feeds the real-DTB DW watchdog, and queues graphical.target (GNOME stays down - no GPU, as analyzed in note 68). Three assertions: system-mode banner, Welcome to Ubuntu 26, login:. About 4-5 minutes to the prompt under 8-vCPU TCG.
One smoke per board - the board is implied by the script's location, matching the boards/<board>/ self-containment rule. Shared mechanics (qemu discovery, dtb freshness, assertion runner, SOAK) live once in sim/engine.py; the shared busybox initramfs and the two-machine QEMU patch moved to sim/ as well. The rk3588 full-boot mode is named 'ubuntu' after the experience, not the mechanism. Regression green: rk3568 linux/uboot, rk3588 board/ubuntu.
Display pipeline unblock round one. The rockchip-pm-domain driver died with -22 because need_regulator domains (npu etc) deliberately power down at probe, and on real silicon that physically cascades to their children's repair_status bits - our shadow lacked that physics, so parent read OFF while child read ON and genpd_add_subdomain refused (hidden -EINVAL clause in drivers/pmdomain/core.c). The fix is a 26-rule cascade table (pwr mask of self+all ancestors, dual-parent domains unioned) generated from the DOMAIN_RK3588 table and dtsi nesting. With the tree built: VOP2 entered iommu group 5, DSI probes for real, and the chain now bottoms out at pl330 CRn values. Also: AMBA PrimeCell periphid shadow (PL330 0x000413330) for the four dmacs - blanket zeros made the amba bus reject them silently. Route to fbcon mapped end to end; regression green both boards.
rockchip-drm display-subsystem now binds vop2 + dsi2 and prints 'Initialized rockchip 1.0.0' on the rk3588-lite machine. - AMBA fix trio (all from recon): drop the phantom 0xfea00000 dmac shadow (it LIFO-shadowed all four virtio transports), move the periphid answers to base+0x3fe0 (the amba bus reads at size-0x20, which is why 'deferred, reason unknown' was really amba_match failing), correct PID1/PID2, add CR0/CRD. Three pl330 dmacs load; spi falls back to PIO instead of deferring forever. - fdt_delprop on vin-supply livelocks the kernel's OF walker (of_clk_init) - property-level runtime deletion corrupts the blob. The surgery moved to the dts layer: rk3588-topeet-board.dts overlays the real board dts with /delete-property/, compiled by dtc. Label spelled vcc3v3_lcd_n (underscore), and the compiled output is verified by dumping it back. - VOP2 behavioral shadow: VERSION_INFO 0x40176786 (the one hard check in the display stack), FS_FIELD frame injection via a 60fps timer, w1c VP_INT_CLR semantics. Panel chain probes silently once the vin-supply is gone. Remaining freeze point (campaign 3): after the first modeset, runtime PM autosuspends the vop and the ISR bails at pm_runtime_get_if_in_use, leaving VP status stuck at 0x20 and drm_crtc_commit_wait deadlocked - the real-scanout model has to answer this dance. Regression green both boards.
…ture pipeline The fbcon deadlock (first modeset's flip_done needs the FS_FIELD irq, but runtime PM put_sync makes the ISR bail at pm_runtime_get_if_in_use, and under TCG timing the line goes spurious-silent) is bypassed instead of fixed: drm_client_lib.active=none (the module is drm_client_lib - earlier guesses at the param name were inert) leaves the KMS userspace path fully alive. card0-DSI-1 connector stands, Ubuntu rootfs boots two- assertions green, and gdm reaches Started + graphical.target. Tooling banked in sim/: gdbfreezegrab.py (stdlib gdb-remote client that settled the deadlock question - all 8 cores idle, no spin, event never comes), fbdump.py (screen capture: VOP Cluster0-win0 YRGB_MST/DSP_INFO/ VIR exported via qom properties, monitor xp dump, stdlib PNG writer). VOP shadow grew to 0x4000 - window registers live at 0x1000+ and were hitting the blanket before. PMU shadow gained mem-chain mirrors (0x1f8<-0x1a0 same polarity, 0x1f0<-~0x1a0). Open race, honestly logged: genpd_power_off work tangles with the gdm modeset into a hung_task panic (guest 124s, ~80%); one lucky survivor reached login+gdm. An automated hunt loop retries boots to catch the survivor and grab the first sim screenshot. fbcon-native fix and the genpd root cause are campaign-4 topics.
- kernel 0020 (RK3588 cluster windows advertise LINEAR modifier) + 0021 (drop AFBC-only 10bpc formats) unblock mutter/llvmpipe buffer negotiation; 0023 SIM-DIAG plane dump dropped (mission complete), 0022 kept - QEMU shadows: DCPHY PLL lock, vop_mmu status/force-reset semantics, PMU mem-chain translation; VOPDBG fprintf traces removed - sim overlay: drop vop iommus (sim iommu page tables half-modeled); real-board DTS untouched - sim tools: framehunt (KEEP/FAST/SMP timeline), sercmd (serial TCP client), snapshot (savevm workflow, blocked on machine vmstate - see docstring), gbmprobe/gbmshim (format negotiation forensics), klogdump (__log_buf addr update), fbdump XBGR byte order (FBFMT) - measured: cold boot to first frame 79s TCG floor; guest-side levers (animations/LP threads/service masks/quiet) all within noise; gdm hard depends on accountsservice (masking it kills gdm) - note 74
…ked (dxg ioctls fail); smoke.py GPU id + DISPLAY_BACKEND knob
…trio - machine: vop-console bool property (virgl 2nd-display epoxy assert, qemu GitLab #1727); virtio-mmio transports 4->6; scanout NULL guard - kernel: CONFIG_VIRTIO_INPUT=y (cmdline-gated family, dormant on board) - smoke.py: auto virtio-tablet/keyboard on desktop modes, GPU id=gpu0, DISPLAY_BACKEND override, auto vop-console=off for GL device - note 78 rewritten: dxg misdiagnosis corrected (WSL#11293 affects compute path only; GLX healthy - 240 FBConfigs, GL 4.5 llvmpipe), real fix documented; interactive desktop verified end-to-end
- spawns smoke.py behind tail -f hvc0-feed (serial Q&A always available) - virgl+sdl,gl=on host capture is blind (HMP screendump 'no surface', X11 import refuses GL windows) - liveness = guest DRM frame count / gnome-shell journal first entry via serial feed - select-based pump (blocking read stalled deadline checks), serial tee to /tmp/framehunt-serial.log, child-death early exit - measured: banner 8.7s / login 25.9s / shell journal 53.4s (llvmpipe era: 20s / 34s / first frame 79-93s)
user-counted wall time to desktop paint across three tiers; 45s recorded as the current cold-boot floor for the same-DTB virgl desktop
…ocked on rk806 power rail - rk3588-lite-i2c2 sysbus device (rk3x spec): START-phase vs MTXCNT/MRXCNT trigger split matches the driver's fill-after-handle_start timing; FIFO first byte = i2c address (driver prepends); repeated-start reads - gpio3 v2 shadow (double-word halves): ver_id must be 0x010219C8 - byte-swapped value made gpio3 the only bank failing probe - gt911 model ported from imx-forge final state (16-patch chain reapplied cleanly on pinned upstream v11.1); INT routed via named gpio-out - bus enumeration green (2-0014 client live); goodix probe deferred on AVDD28 -> vcc_3v3_s3 (rk806 spi pmic, unmodeled) - the debt the old overlay used to bypass; next beat = rk806 shadow
- hw/ssi/rk806.c: rk8xx-spi command protocol slave (register-file answers, probe trusts compatible so no chip-id magic needed) - rk3588-lite-spi2 sysbus device: DW-style shadow; TXDR write drives the SSI bus full-duplex, RO mode auto-clocks CTRLR1+1 bytes on SSIENR enable (driver writes no TXDR for rx-only), VERSION answers VER2_TYPE2 - i2c2 fixes by trace: register enum must step by 4 (silent-drop bug), CON mode values REGISTER_TX=1/RX=2 (kernel enum order), rx payload now correct (ID '911' + factory config pass checksum) - gpio3: int_status serves plain 32-bit reads (demux uses readl_relaxed, not the v2 half-word protocol) - fixes storm-without-EOI - gt911: rising-edge INT pulse (this DTB configures pin16 edge+pol=1) - verified: Goodix-TS ID 911/version 0020, input0 registered, injected touch report (700,400) read back over real I2C by the driver thread
- FAST bootargs: initcall_blacklist=rockchip_drm_init - the rk806 power chain now brings up the DSI panel (card1-DSI-1 appears) and mutter chokes on the dual-display topology; single-display restored - gt911: host-mouse property + HMP gt911_mouse on|off runtime toggle - simultaneous virtio pointer grab + touch gesture fight over GNOME's grab and break window dragging; drag reports throttled to 33ms guest virtual time (per-event full-price reports saturate mutter) - HMP gt911_release registered (hx docs follow entry-then-SRST rule) - smoke.py: VIRTIO_INPUT=0 knob for pure-touch experiments - M2 verified: host mouse feeds real I2C touch (live rx trace), injected (512,300) read back exactly; daily driver = virtio + bridge off
…ession pass - delete rk3588-topeet-board.dts/dtb (superseded by same-DTB constitution, unreferenced since note 77) - snapshot.py: same-DTB + cmdline-virtio argv (hvc0 via virtconsole); savevm still gated on machine vmstate (documented in docstring) - VOP2 regression on same-DTB flow: rockchipdrm binds, DSI-1 detects connected after rk806 unblocked the power rail - first time the panel chain completes end to end (campaign 1 goal reached as a side effect)
campaign 5: GPU_BACKEND=vop form factors rockchipdrm as the sole DRM card; mutter modesets the real DSI panel (enabled), gnome-shell paints, and the VOP shadow scans out through the real pipeline. three cuts, each trace-verified: - iommu v2 page-table semantics: vop_mmu is rk3568-iommu => v2 ops - DTE index iova>>22 (4MB sections), DTE_ADDR register value carries the valid bit (mask 0xfffff000). v1 walk silently fell back to identity (fb read kernel .text region) - per-page stitched scanout: gem bo pages are NOT physically contiguous across the 4MB DTE boundary (page-table dump shows the topology change) - linear base+stride mapping produced ~30% garbage bands; each 4KB page now walks the iova and lands in a staging buffer - FAST rockchipdrm blacklist made conditional (vop form needs it alive) note 80: campaign 4's iommus-removal premises all overturned - the v1/v2 mismatch was the real disease; every past workaround repaid as proper hardware behavior modeling
System.map moved __log_buf to 0x2ca9140 (phys) after the merged-config rebuild during note 78's dxg-misdiagnosis session; verified against the live System.map before committing
- machine-level vmsd (vop regs+fb / mmu / cru / pmu / dcphy / gpio3 / last_req) with post_load re-arming the frame ticker and clearing last_mst/dsp/vir to force scanout surface rebuild - device vmsds: i2c2 + spi2 (post_load recomputes irq line), rk806, gt911 (full protocol + bridge state) - snapshot.py: six-transport cmdline (two-transport boot deadlocks the kernel - root cause unknown, smoke-parity cmdline sidesteps), FAST tier, earlycon diagnostic - result: create 4.5min / restore 9.2s to first frame vs 80s cold; desktop intact (chroma 8/9), serial login live - open defect: delayed (~2-4min guest) watchdog hardlockup panic on an idle CPU; watchdog=0 at create did not suppress - hypotheses and next-step leads in note 81 (pending-event loss across load the prime suspect)
- loadvm trace: all 397 sections load successfully, machine fields round-trip correctly; the load_bad lines are optional-subsection probe noise. the guest is healthy for ~60s post-restore before cpu0 trips CPU7's buddy watchdog - deferred defect isolated to multi-vCPU timer/cross-cpu state on the TCG savevm path - SMP=1 create/restore verified stable 5+ minutes (blank at 5min is GNOME idle blanking - the flow attaches no input devices, expected); garbage-PC gdbstub reads reclassified as halted-vCPU artifacts - snapshot.py docstring carries the current operating matrix
experiments: post_load cpu kick (no), cold boot on qcow2 (stable - disk exonerated), late snapshot past the 104s workload (still locks, different cpu - workload exonerated), tcg thread=single (still locks - MTTCG exonerated), idle=poll (still locks - sleep/wake exonerated, the strongest theory gone) surviving fact chain: any cold boot is stable; ~1-2min after loadvm one cpu's timer interrupt delivery silently dies while polling - the vtimer QEMUTimer <-> GIC PPI runtime link for that core is not rebuilt by restore even though every field loads. SMP=1 remains the only stable restore shape; FAST cold boot (45s) is the daily driver. next blade: read CNTV_CTL/CVAL/CNTVCT of the dead core at lock time + counter on gicv3_cpuif_update to see whether the timer never expires or the expiry never delivers
…shot stable the instrumentation verdict: on the dying cores gicv3_cpuif_update stops being called (delta +0/+1 over minutes) while healthy cores grow +300K; the vtimer recalc counter sits at exactly 1 for every cpu - the vtimer QEMUTimer fires once at load and is never re-armed, because write_list_to_cpustate restores CNTV_* via raw writes that bypass the writefn which would normally recalc. idle cores waiting for their next tick never wake; busy cores survive on SPI traffic. fix: arm_gt_force_recalc_all_cpus() (exported from helper.c) called in the machine post_load - gt_recalc_timer(VIRT+PHYS) per cpu rebuilds the vtimer<->GIC-PPI runtime link. verified: two independent restores, 10min and 5.5min, zero lockups, serial alive, all eight cpuif counters growing (665K-1300K). SMP=8 snapshot create 4.5min / restore 9s is now a stable workflow. diagnostic counters kept (diag-cpuif-N / diag-vtimer-N qom props)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
因为这边看了一下，后续一些板子的行为，可以刚好丰富一下模拟器，方便各位朋友在板子不在手边的时候，也能上阵学习