[Feature Request]: Multi-core debug - Simultaneous run/step/stop

[jreineckearm]

Description

Investigations into more complex multi-core scenarios showed that applications with parts running on different CPUs can require a more synchronized debug experience. Otherwise, some parts can run away, timeout, etc.

We should investigate use of CoreSight cross-triggers to support synchronization between CPUs in such scenarios.
This touches

• Configurations of cross triggers.
• Debug sequence or scripting extensions to manage cross-trigger HW.
• Embedding cross triggers into connection flows.

Related issues feeding into this

• [Feature Request]: Dual-core debugging with interprocess communication #518
• [Feature Request]: Document typical multi-core load and debug user flows #607

Is this request a Security Requirement?

• Yes

Priority

High

Related Issues (Optional)

#518, #607

Additional Notes (Optional)

No response

[RobertRostohar]

Multi-core Simultaneous Run and Halt

Exploration of how to configure and use Hardware Cross Triggers to simultaneously run/halt cores in multi-core devices. This should be a user option (not enabled by default) which is an alternative to handling all cores independently.

Following is a description of a proposal (could be prototyped and validated).

Single stepping is not covered here (cores handled independently).

Cross Triggers

CoreSight Cross Triggers can be configured to run and halt cores simultaneously.

When any core halts (hits a breakpoint/watchpoint or gets a debug request), all cores receive a debug request automatically and also halt. It is necessary to acknowledge the debug request on all cores before restarting them. All cores can then be restarted simultaneously by setting the restart request using cross triggers.

The configuration is done via the Cross Trigger Interface (CTI).

CTI in Cortex-M:

DEVTYPE=0x14

• CS-400: PART=0x0906 (Cortex-M0+, Cortex-M7, ...)
• CS-600: PART=0x09ED, ARCHID=0x1a14
• Cortex-M23: PART=0x0D20, ARCHID=0x1a14
• Cortex-M33: PART=0x0D21, ARCHID=0x1a14
• Cortex-M35P: PART=0x0D31, ARCHID=0x1a14
• Cortex-M52: PART=0x0D24, ARCHID=0x1a14
• Cortex-M55: PART=0x0D22, ARCHID=0x1a14
• Cortex-M85: PART=0x0D23, ARCHID=0x1a14

Trigger signals related to processor halt and restart:

• CTI Trigger In 0 connected to Processor Halted signal of each core.
• CTI Trigger Out 0 connected to Processor Debug Request signal of each core.
• CTI Trigger Out n connected to Processor Restart Request signal of each core.

n = 1 on Cortex-M23/M33/M35P/M52/M55/M85, 7 on Cortex-M7, Cortex-M0+ (Infineon Traveo)

Typical trigger channel configuration (2 channels out of 4 used):

• Channel 0: Maps all Processor Halted outputs (trigger inputs) to all Processor Debug Request inputs (trigger outputs).
• Channel 1: Maps all Processor Restart Request inputs (trigger outputs). Used to restart all cores simultaneously.

Following is an example in C how to configure and use CTI (Cortex-M7 with CTI at 0xE0042000).

Configuration of each core

CTI on each core needs to be configured as follows:

#define CTI_BASE          0xE0042000
#define CTI_CONTROL_OFS   0x000
#define CTI_INTACK_OFS    0x010
#define CTI_APPSET_OFS    0x014
#define CTI_APPCLEAR_OFS  0x018
#define CTI_APPPULSE_OFS  0x01C
#define CTI_INEN0_OFS     0x020
#define CTI_OUTEN0_OFS    0x0A0

void cti_configure (void) {
  // Enable CTI
  *((uint32_t *)(CTI_BASE + CTI_CONTROL_OFS)) = 1;
  // Enable CTI Trigger In 0 (Processor Halted) to Channel 0
  *((uint32_t *)(CTI_BASE + CTI_INEN0_OFS  + (0 << 2))) = 1 << 0;
  // Enable CTI Trigger Out 0 (Processor Debug Request) to Channel 0
  *((uint32_t *)(CTI_BASE + CTI_OUTEN0_OFS + (0 << 2))) = 1 << 0;
  // Enable CTI Trigger Out 7 (Processor Restart) to Channel 1
  *((uint32_t *)(CTI_BASE + CTI_OUTEN0_OFS + (7 << 2))) = 1 << 1;
}

After each core is configured as outlined, halting any core will cause all cores to halt automatically.

Halting all cores simultaneously

When any core is halted (e.g., hits a breakpoint/watchpoint or was halted via core debug halt), all cores will halt automatically.

Therefore, there is no need to explicitly halt all cores using CTI. However just for completeness, halting all cores can be achieved also by injecting a software pulse into channel 0 from any core:

void cti_halt (void) {
  // Event pulse on Channel 0
  *((uint32_t *)(CTI_BASE + CTI_APPPULSE_OFS)) = 1 << 0;
}

After halting (also when halted due to breakpoint/watchpoint), each core must acknowledge the debug request:

void cti_halt_ack (void) {
  // Acknowledge CTI Trigger Out 0 (Processor Debug Request)
  *((uint32_t *)(CTI_BASE + CTI_INTACK_OFS)) = 0x01;
  *((uint32_t *)(CTI_BASE + CTI_INTACK_OFS)) = 0x00;
}

If a core does not acknowledge the debug request, it will not be able to restart.

Running all cores simultaneously

Running all cores again (after acknowledgement) can be achieved by injecting a software pulse into channel 1 from any core:

void cti_run (void) {
  // Event pulse on Channel 1
  *((uint32_t *)(CTI_BASE + CTI_APPPULSE_OFS)) = 1 << 1;
}

This should be done only on a single core after all cores have acknowledged the debug request.

In practice, the debugger should handle running all cores simultaneously as follows:

• execute cti_halt_ack() after halt on each core to acknowledge the debug request.
• execute cti_run() only on a single core to restart all cores.

Option 1: Debugger manages Cross Triggers internally

Debugger supports Cross Triggers internally when simultaneous multi-core run/halt is enabled. It configures CTI on each core and handles halt acknowledgement and simultaneous restart as described above.

• CTI base address is derived from CoreSight topology.
• CTI trigger mapping is pre-defined based on detected CTI (seems to be only two variants). The mapping seems to be well standardized for Cortex-M7/M23/M33/M35P/M52/M55/M85. To be investigated for Cortex-M0+/M3/M4 and CS-400/600.
• CTI channel usage is pre-defined as described above (Channel 0 for halt, Channel 1 for restart).

It is still possible to use other CTI channels for other purposes (vendor specific).

Option 2: Debug sequences to manage Cross Triggers

Cross Triggers could be configured and utilized through the Debug access sequences. That would provide maximum flexibility to support different cores and CTI implementations. However still needs to be implemented in the debugger and requires more efforts (definition of new sequences, more complex implementation). Debug sequences are also not supported by all debuggers (J-Link, ...) which might already have built-in support for Cross Triggers.

Pre-defined debug access variables

__options: [New] Debug options (read-only)

Bit map:

• Bit 0: Simultaneous multi-core Run/Halt (0=disabled, 1=enabled)
• Bit 1..63: Reserved

Debug access sequences

DebugCoreSetup: [New] Setup core debug system (pre-defined without default implementation)

Executed after having connected and powered up the debug port and before DebugCoreStart.

Can be used to configure Cross Triggers if simultaneous multi-core run/halt is enabled.

Example for Cortex-M7 with CTI at 0xE0042000:

<sequence name="DebugCoreSetup">
  <block>
    __var multi_run_halt = ((__options &amp; 0x00000001) != 0);
  </block>
  <control if="multi_run_halt">
    <block>
      __var CTI_Addr         = 0xE0042000;
      __var CTI_CONTROL_Addr = CTI_Addr + 0x000;
      __var CTI_IEN0_Addr    = CTI_Addr + 0x020;
      __var CTI_OUTEN_Addr   = CTI_Addr + 0x0A0;
   
      // Enable CTI
      Write32(CTI_CONTROL_Addr, 0x00000001);

      // Enable CTI Trigger In 0 (Processor Halted) to Channel 0
      Write32(CTI_IEN0_Addr  + (0 &lt;&lt; 2), 1 &lt;&lt; 0);

      // Enable CTI Trigger Out 0 (Processor Debug Request) to Channel 0
      Write32(CTI_OUTEN_Addr + (0 &lt;&lt; 2), 1 &lt;&lt; 0);

      // Enable CTI Trigger Out 7 (Processor Restart) to Channel 1
      Write32(CTI_OUTEN_Addr + (7 &lt;&lt; 2), 1 &lt;&lt; 1);
    </block>
  </control>
</sequence>

ProcessorStart: [New] Start processor (pre-defined with default implementation)

Executed to restart a core. Can be extended to use Cross Triggers to restart all cores simultaneously when simultaneous multi-core run/halt is enabled.

Default implementation (Cortex-M):

<sequence name="ProcessorStart">
  <block>
    __var SCS_Addr  = 0xE000E000;
    __var DFSR_Addr = SCS_Addr + 0xD30;

    // Resume core execution via DHCSR
    Write32(DHCSR_Addr, 0xA05F0001);
  </block>
</sequence>

Example for Cortex-M7 with CTI at 0xE0042000:

<sequence name="ProcessorStart">
  <block>
    __var multi_run_halt = ((__options &amp; 0x00000001) != 0);
  </block>
  <control if="multi_run_halt">
    <block>
      __var CTI_Addr          = 0xE0042000;
      __var CTI_APPPULSE_Addr = CTI_Addr + 0x01C;

      // Event pulse on Channel 1 to restart all cores
      Write32(CTI_APPPULSE_Addr, 1 &lt;&lt; 1);
    </block>
  </control>
  <control if="!multi_run_halt">
    <block>
      __var SCS_Addr  = 0xE000E000;
      __var DFSR_Addr = SCS_Addr + 0xD30;

      // Resume core execution via DHCSR
      Write32(DHCSR_Addr, 0xA05F0001);
    </block>
  </control>
</sequence>

ProcessorStop: [New] Stop processor (pre-defined with default implementation)

Executed to halt a core. Does not need to be changed when using Cross Triggers, as halting any core will cause all cores to halt automatically when CTI is configured. Defined here for completeness as the counterpart to ProcessorStart.

Default implementation (Cortex-M):

<sequence name="ProcessorStop">
  <block>
    __var SCS_Addr   = 0xE000E000;
    __var DHCSR_Addr = SCS_Addr + 0xDF0;

    // Halt core execution via DHCSR
    Write32(DHCSR_Addr, 0xA05F0003);
  </block>
</sequence>

ProcessorHaltAcknowledge: [New] Acknowledge processor halt (pre-defined without default implementation)

Executed after core is halted (either due to breakpoint/watchpoint or debug halt). Needs to be implemented when using Cross Triggers to acknowledge the debug request via CTI when simultaneous multi-core run/halt is enabled.

Example for Cortex-M7 with CTI at 0xE0042000:

<sequence name="ProcessorHaltAcknowledge">
  <block>
    __var multi_run_halt = ((__options &amp; 0x00000001) != 0);
  </block>
  <control if="multi_run_halt">
    <block>
      __var CTI_Addr        = 0xE0042000;
      __var CTI_INTACK_Addr = CTI_Addr + 0x010;

      // Acknowledge CTI Trigger Out 0 (Processor Debug Request)
      Write32(CTI_INTACK_Addr, 0x01);
      Write32(CTI_INTACK_Addr, 0x00);
    </block>
  </control>
</sequence>

CMSIS-Debugger

• A user option should be provided to enable/disable simultaneous multi-core run/halt (disabled by default). The option could be exposed via GUI and stored in the csolution.yml. It would be also included in the generated cbuild-run.yml and available in scripting for generating launch.json and tasks.json files. The debugger backend would also see this option either via cbuild-run.yml or command line options from .json files.
• Launch configuration could potentially be a compound configuration.
• Play button (continue) of any core or global Play button (if exists) would trigger the simultaneous multi-core run by executing continue command on each core via its gdb client.
• Pause button (halt) of any core or global Pause button (if exist) or any core halting due to hitting a breakpoint/watchpoint would automatically halt all cores simultaneously.

[jreineckearm]

Quick thoughts before leaving for holiday:

Both options can build on each other. First iteration would be option 1. If designing the debugger stack right, it can be expanded later with option 2 for more configurability.

Making assumptions on cross-trigger interfaces for a start sounds reasonable. As you say, they should be well-defined at least for CPUs.

In the longer run we may need more configurability. Either by mapping descriptions as they for example exist in CMSIS-SDF. Or by sequences which could give us even more flexibility.
Cross-triggers are for example also used as trace trigger/start-stop-logic inputs. And they can even be routed to GPIO pins on some devices.