[6/7] thunderbolt: apple: Power-cycle the J713 host router for a deep sleep by erikwb · Pull Request #1419 · aurora-silicon/linux

erikwb · 2026-11-10T05:21:19Z

On the J713 a deep sleep powers the ACIO host router off while the dock stays attached. This adds what the connection manager and the Apple NHI need to take the PCIe and USB tunnels through that power cycle, behind thunderbolt_apple.cio_s2r (default off). The stock resume path is unchanged.

Router state transfer: the two vendor router operations that hand the host router's state to the connection manager before the power-off and back after the restart.
Connection manager transaction: prepare (release the DisplayPort tunnels while their owners still run, pause the others with their hops saved), sleep (routers to sleep, state saved, control channel stopped), wake (enumerate the restarted router, restore its state, start its links, replay the hops) and complete (reopen hotplug, retire what the wake found gone, pair the displays again as after a plug).
A dock gone during the sleep is an unplug: routers that did not come back are marked unplugged, their tunnels are skipped by the replay and retired at complete, and the tunneled PCIe port is told not to wait for its link. Only a host-side failure is fatal.
Apple NHI: owns the power cycle of its ACIO block: routers to sleep, DART down, controller off, LSTX held, four power domains down, PHY down; the reverse with a cold restart of the controller. The host router gets its own sleep bit before the vendor enter-sleep request, and after the restart its lane adapters are unlocked and the cable word rewritten, which starts link training. Once the router took its vendor sleep, a failure recovers through a controller restart, with the four domains taken down first, since that is the only reset of the block. Without the PHY reference or the DART the NHI logs it and runs without the transaction.
Platform data: a built-in T8132 policy entry can name the machine it applies to (apple,policy-machine); the J713's ACIO power states use it, so other T8132 boards keep them as before.
Device tree: a "cio" PHY reference on both ACIO nodes, and the ACIO power-state overrides in a file the built-in platform data also includes.

A display does not survive the host router's power cycle, which is why DisplayPort tunnels are released and re-paired instead of replayed.

Part 6/7 of J713 docked S2R, stacked on the native radio series (#1384), which is stacked on #1314.

Series order: #1414 → #1415 → #1416 → #1417 → #1418 → #1419 → #1420.

Depends on #1418.

Validation of the series on this base:

Full arm64 Image, modules and DTBs at every PR tip of the series.
Ten focused host suites with negative controls (DART owner and register sequence, PHY sleep transaction, USB3 host model, router state transfer, tunnel replay including a dock lost during the sleep, NHI and tunneled-PCIe ownership including its failure paths, dock PM callback ordering, DP repair, DART writer gate and its callers) plus the device-tree policy check. checkpatch is clean for the driver code; it notes the Based-on-work-by: trailers and style in the tools/asahi fixtures.
J713 with a CalDigit TS5 (USB4 link, PCIe tunnel with the dock's Ethernet, USB hub with two keyboards, two 4K DisplayPort monitors), on the first version of the series: one suspend-abort test at the platform stage, then one real deep sleep with an RTC wake on the same boot. After the wake the dock link was at 20 Gb/s, dock Ethernet passed traffic at 2.5G, the USB devices were back, both monitors were lit at their previous mode about 10 s after the wake, Wi-Fi and Bluetooth answered, all ten CPUs were online, and there were no kernel warnings or taint. Repeated with one monitor unplugged; a monitor plugged in after the wake came up.

The changes since that version (the DT-selected DART owners and memory-only updates while closed, the unplug handling, the failure unwinds, the stock tb.c resume restored, the PMGR and RTKit hardening) are built and covered by the host suites, but have not run on the J713 yet.

Not covered: repeated sleeps on one boot, wake sources other than the RTC, the desktop's own sleep path (lid, logind), a dock unplugged during the sleep on hardware, and docks other than the TS5. Both native parameters (pcie_apple.cio_s2r, thunderbolt_apple.cio_s2r) default to off.

This series was developed and tested by Claude (Anthropic), working with me on the J713.

DjDeveloperr

[6/7] thunderbolt: apple: review at 323ce0a

Gating: the NHI/CM transaction sits behind thunderbolt_apple.cio_s2r, T8132 and apple,j713, and the tb_cio_*, tb_tunnel_sleep* and tb_path_sleep* additions only run from it. The sleep_replay guards in tunnel.c are no-ops otherwise. In tb_dp_activate(), the newly returned set_hops result is overwritten by tb_dp_port_enable() on the normal path, so the generic path is unchanged. Two changes are not gated, see (2) and (6).

M5: no T6050 impact. Its DP-in policy is disabled, so the tb.c resume change doesn't reach it, and nothing here conflicts with the M5 branch beyond what the series base already does.

Findings (most severe first)

A dock unplugged, swapped or powered off during sleep panics on wake. tb_cio_wait_links() returns -ENOLINK after about 3 s, or tb_cio_check_topology() returns -ENODEV for an unplugged router. tb_cio_wake() then fails and apple_nhi_cio_resume() panics. apple_nhi_cio_complete() even warns about a changed cable, but resume never gets that far. This is the likeliest real-world failure and it isn't in "not covered". When the controller restarted but the links or topology are gone, mark the downstream routers unplugged and the PCIe port dead, finish, and let .complete retire them like a hot-unplug. That needs #1416 to stop panicking on release.
Ungated stock-path change in tb.c. tb_resume_noirq()/tb_complete() defer freeing invalid tunnels and unplugged routers to .complete, and keep hotplug off until then, whenever QUIRK_KEEP_TUNNELS and TB_HOST_DP_NOTIFY are both set. That's the default router_sleep=0 path whenever a PCIe-C link is kept or a host DP tunnel is up. TB_HOST_DP_NOTIFY is set by the M1 (t8103), M3 (t6030) and M4 policies, so this changes stock sleep there with cio_s2r off. It's in 49b5edc (the generic CM commit), whose message doesn't mention it. Please split it into its own patch with its own validation, or gate it.
Unwind before the domains cycled. If the suspend unwind runs after apple_rtkit_poweroff() but before all four domains went off (lstx_hold failure, a domain that stays on, verify_state failure), apple_nhi_cio_power_on() cold-restarts a CPU that never reset. Nothing halts it (cable teardown calls apple_cio_cpu_halt()), and the ACIO reset is only ever deasserted, so HELLO likely never comes and the unwind panics. Halt or reset the CPU before the cold restart when cio_pd_suspended doesn't cover all four domains.
Unwind after the vendor sleep. If apple_router_save_sleep() fails after router_sleep() succeeded, restarted is false, so there is no enumerate and no router wake, although the controller already took the vendor sleep (M3_STAT 0x02000000). tb_switch_resume() then talks to a sleeping controller and recovery panics.
With cio_s2r=1, a failing apple_nhi_cio_bind() (a DT without the new phys/phy-names = "cio", or num_pds != 4) fails the whole NHI probe. Log it and run without CIO sleep, as #1420 does for its DART bind.
t8132-platform-data.dts now includes t8132-j713-usb4-pmgr.dtsi, and the handoff applies that data, status included, to every T8132 machine on the firmware path. The ACIO domain statuses on J604/J623/J624/J773g/J715 now come from a J713 file. Please confirm that's intended.
Simplification: DP tunnels are released in tb_cio_sleep_prepare(), so the DP replay branches (sleep_replay in tb_dp_activate(), sleep_dprx) appear unreachable in this flow.

Verdict: needs changes, (1) and (2) first.

Review only: read at the head commit, not built or run (no J713 here). Written with Claude Code.

DjDeveloperr · 2026-11-10T05:44:05Z

+
+	/* Child resume would otherwise access a dead tunnel or missing translations. */
+	if (ret)
+		panic("Apple CIO: resume failed %d; children remain suspended", ret);


Every wake after the dock was unplugged, swapped or lost power during sleep reaches this panic (-ENOLINK from tb_cio_wait_links(), -ENODEV from the UID or topology checks). Once the controller is back (apple_nhi_cio_power_on() succeeded), a missing peer isn't a DMA-safety problem. It's an unplug. Suggest: mark the downstream routers unplugged and the PCIe-C port dead, finish the DART, PHY and PCIe transactions, and let .complete retire the hierarchy through the normal unplug path. Only a failure to bring the host side back really needs to stay fatal.

DjDeveloperr · 2026-11-10T05:44:05Z

+		tb_info(tb, "CIO port %u link after wake: %d (%u retries)\n", port->port, ret,
+			tries);
+		if (ret <= 0)
+			return ret ?: -ENOLINK;


port->remote is still the pre-sleep router, so an absent dock spends about 3 s here and returns -ENOLINK, and the caller panics (see apple_nhi_cio_resume()). Returning a distinct code for "no peer" would let the NHI take the unplug path.

DjDeveloperr · 2026-11-10T05:44:05Z

+	 * into DCP before it can service the disconnect. Keep the tunnel and
+	 * its ports alive until complete; restore_children skips absent routers.
+	 */
+	if ((tb->nhi->quirks & QUIRK_KEEP_TUNNELS) &&


This isn't gated by cio_s2r. QUIRK_KEEP_TUNNELS is set on the default router_sleep=0 path whenever a PCIe-C link is kept or a host DP tunnel is up, and TB_HOST_DP_NOTIFY comes from the M1 (t8103), M3 (t6030) and M4 DP-in policies. So on those machines, routers and tunnels that vanished during sleep now survive until .complete, and hotplug stays off until then. That may well be the right fix, but it's a separate behaviour change for M1/M3 and deserves its own patch and test, or a gate.

DjDeveloperr · 2026-11-10T05:44:05Z

+	}
+	if (anhi->cio_rtk_off) {
+		dev_info(anhi->dev, "CIO restart controller\n");
+		ret = apple_rtkit_cold_restart(acio->rtk, apple_nhi_cio_start_cpu, anhi);


In the suspend unwind this can run after apple_rtkit_poweroff() but with some or all of the four domains never powered off (lstx_hold failure, dev_pm_genpd_is_on(), verify_state failure). The CPU then never went through reset: nothing calls apple_cio_cpu_halt() the way cable teardown does, and acio->reset is only ever deasserted. apple_rtkit_cold_restart() expects a HELLO from a fresh boot, so this likely times out and the unwind panics. Suggest halting or resetting the CPU first when cio_pd_suspended doesn't cover all four domains.

DjDeveloperr · 2026-11-10T05:44:05Z

+	tb_switch_exit_redrive(tb->root_switch);
+	ret = tb_switch_suspend_cio(tb->root_switch) ?:
+	      tb->nhi->ops->router_sleep(tb->nhi) ?:
+	      apple_router_save_sleep(tb->root_switch, &tcm->cio_router);


If router_sleep() succeeded and apple_router_save_sleep() then fails, the controller has accepted the vendor sleep (M3_STAT 0x02000000) but cio_router.valid is false. tb_cio_wake_locked() then skips the enumerate and the router wake, and tb_switch_resume() runs against a sleeping controller, so recovery likely fails and this panics. That case needs its own wake, or the save needs to come before the vendor request if the firmware allows it.

DjDeveloperr · 2026-11-10T05:44:05Z

+		return 0;
+	if (anhi->acio->pd_list->num_pds != 4)
+		return -EINVAL;
+	anhi->cio_phy = devm_of_phy_get(anhi->dev, anhi->acio->np, "cio");


With cio_s2r=1, any error here (no phy-names = "cio" in an older DT, num_pds != 4 above, a DART bind error) fails apple_nhi_probe(), which loses Thunderbolt entirely. Suggest logging and continuing without router_sleep/router_wake, like #1420's "no DART owner for a deep sleep"; deep sleep with a dock then just takes the stock path.

DjDeveloperr · 2026-11-10T05:44:05Z


 #include "../../../arch/arm64/boot/dts/apple/t8132-pmgr.dtsi"
 #include "../../../arch/arm64/boot/dts/apple/t8132-j713-pmgr.dtsi"
+#include "../../../arch/arm64/boot/dts/apple/t8132-j713-usb4-pmgr.dtsi"


t8132_pmgr_policy() applies this built-in data, status included, on every T8132 machine on the firmware path, not just J713. These ACIO statuses used to live only in the J713 board DT. Please confirm that's intended for J604/J623/J624/J773g/J715; otherwise split them out by board.

An Apple host router hands its connection manager sixteen words of state before it is powered off and takes them back after it has been restarted. Both directions are vendor router operations, 0x8000 to save and 0x8001 to restore, with sixteen data words and no metadata. Add the two operations for the root switch. The words are opaque: they are kept as returned and written back unchanged. A failed restore leaves the saved copy in place so the caller can decide what to do. Add a host-side model of the operation with negative controls. Signed-off-by: Erik Bourget <erik@ebourget.net>

On Apple silicon a deep sleep powers the host router off while a dock stays attached, and the PCIe and USB devices behind the dock stay bound through it. The normal suspend path cannot express that: it either keeps the routers awake or tears the tunnels down. Add a connection-manager transaction for an NHI that owns such a power cycle, in four calls: - prepare, before any device suspends: check the topology, close hotplug and release the DisplayPort tunnels while their owners still run. A display does not survive the host router's power cycle and is paired again afterwards as after a plug. Every other tunnel is paused with its hop entries saved. - sleep, with the tunnels' users stopped: deactivate the paths, put the device routers to sleep, then the host router through the NHI's router_sleep hook, save its state and stop the control channel. - wake, after the NHI has restarted the router: start the control channel, enumerate the router again, restore its state, let the NHI start its links through router_wake and wait for them, resume the routers and replay the saved hops and adapter settings. - complete, after every device has resumed: reopen hotplug, retire what the wake found gone and pair the released DisplayPort resources again. A dock unplugged, swapped or powered off during the sleep is an unplug, not a failure: the routers that did not come back are marked unplugged, their tunnels are skipped by the replay and retired at complete, after their users resumed, and the NHI is told when the host's PCIe tunnel was among them. Only the host side has to come back. The host router's UID is read from its configuration space before the sleep and compared with what it reads after it. An Apple host router has no UID there (its identity comes from the device tree DROM), so both read zero; comparing against the DROM UID would always fail. A failed sleep unwinds through wake. Once the host router took its vendor sleep, only the NHI's restart wakes it; a state save that fails after that does not unwind, and the wake then brings the router back as after a cold start. If the unwind fails, the tunnels' users would resume onto dead paths, so it panics. The stock resume path is unchanged. Add a host-side model of the transaction with negative controls. Signed-off-by: Erik Bourget <erik@ebourget.net>

With thunderbolt_apple.cio_s2r=1 on the J713, the NHI owns the whole power cycle of its ACIO block around a deep sleep with a dock attached. In prepare it closes the connection manager, checks that only the control rings are in use, waits for the released displays' routes to go down and opens the PHY, DART and tunneled PCIe transactions. In its noirq suspend, once the PCIe port reports stopped, it puts the routers to sleep, powers the NHI DART down, powers the controller off, holds the LSTX pin, takes the four ACIO power domains down in order and powers the PHY down. The noirq resume does the reverse: PHY, domains, a cold restart of the controller, tunables, the DART, then the connection manager's wake. Complete closes the three transactions and reopens the connection manager. The host router is told to sleep the way its firmware expects: the router's own sleep bit (ROUTER_CS_5.SLP, with its PCIe and USB wake bits clear) first, then the vendor enter-sleep request, then a wait for the controller to report its sleep state. The vendor request on its own makes the firmware assert. After the restart the router is given its lane adapters unlocked and the cable word again, as at the first start, which is what starts link training. A dock lost during the sleep is retired as an unplug once the host side is back; if the host's PCIe tunnel went with it, the tunneled port is told not to wait for its link. A failure after the routers were changed recovers through a restart of the controller: the router only wakes from its vendor sleep that way. The ACIO reset cannot be asserted again, so the four domains are taken down first if they were not already, which resets the CPU, then the same power-on path runs. If a domain stays on, or the NHI DART could not be saved before the restart, or the recovery fails, the PCIe and USB devices behind the dock cannot be resumed safely, so it panics. A DART that is briefly busy with a mapping is waited for first. The power-on path replays the T8132 ACIO start sequence the driver already has for a cable connect. The parameter defaults to off. Without the PHY reference or the DART, the NHI logs it and runs without its deep-sleep transaction. Add host-side models of the ownership rules and of the dock's PM callback ordering. Based-on-work-by: Diljit Singh <hey@djdev.me> Signed-off-by: Erik Bourget <erik@ebourget.net>

The built-in T8132 PMGR policy is shared by every T8132 machine on the firmware path. Some of its entries describe one board only, such as the J713's ACIO power states, and must not enable those states on the others. Let an entry name the machine it applies to with apple,policy-machine. On every other machine it is treated as absent, as before the entry existed. Signed-off-by: Erik Bourget <erik@ebourget.net>

The ACIO host powers its Type-C PHY down and up itself around a deep sleep with a dock attached, so it needs a reference to it. Add the PHY of each port to its ACIO node as "cio". Move the ACIO and tunneled PCIe power-state overrides into their own file and include it from the built-in T8132 platform data as well, so the policy compiled into the kernel names the same providers as the board's device tree. Those entries name the J713, so the other T8132 boards keep these states as before. Add a check for both. Signed-off-by: Erik Bourget <erik@ebourget.net>

erikwb · 2026-11-13T20:00:19Z

Closing: this commit set landed in integration/aurora through #1314, which carried the whole docked deep-sleep series (#1414 to #1420). The follow-up that turns the docked deep sleep on by default is #1855.

erikwb force-pushed the j713/dock-s2r-thunderbolt branch from 6f9d0c5 to 323ce0a Compare November 10, 2026 05:27

erikwb force-pushed the j713/dock-s2r-pcie branch from 06fc7a3 to 43aec22 Compare November 10, 2026 05:27

erikwb marked this pull request as ready for review November 10, 2026 05:34

DjDeveloperr reviewed Nov 10, 2026

View reviewed changes

erikwb added 5 commits November 10, 2026 01:51

erikwb force-pushed the j713/dock-s2r-pcie branch from 43aec22 to 473e2b4 Compare November 10, 2026 13:31

erikwb force-pushed the j713/dock-s2r-thunderbolt branch from 323ce0a to 903277c Compare November 10, 2026 13:31

erikwb closed this Nov 13, 2026

Navigation Menu

Repository navigation

[6/7] thunderbolt: apple: Power-cycle the J713 host router for a deep sleep - #1419

[6/7] thunderbolt: apple: Power-cycle the J713 host router for a deep sleep#1419
erikwb wants to merge 5 commits into
j713/dock-s2r-pciefrom
j713/dock-s2r-thunderbolt

erikwb commented Nov 10, 2026 •

edited

Loading

Uh oh!

DjDeveloperr left a comment

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

DjDeveloperr Nov 10, 2026

Uh oh!

erikwb commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

erikwb commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DjDeveloperr left a comment

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DjDeveloperr Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

erikwb commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

erikwb commented Nov 10, 2026 •

edited

Loading