gdbserver: commit breakpoint removals before replying OK

[j4rvisstant]

Fixes #2014.

Problem

When GDB removes a breakpoint with z0/z1 while the core is halted, the gdbserver replies OK once BreakpointManager.remove_breakpoint() has recorded the removal, but nothing has been written to the target yet. The FPB comparator, or the original instruction under a software breakpoint, is restored on the next resume or step. A session that ends without one, as in #2014 (GDB disconnects and the frontend terminates the server), leaves the deleted breakpoint armed, and the target stops on it again later.

Change

• New BreakpointManager.flush_removals() commits pending removals, and only removals, while the core is halted, then drains the probe's queued transfers. Pending additions stay deferred until flush(), as before. If a provider keeps a breakpoint it was asked to remove, DebugError is raised and the breakpoint stays pending for the next flush().
• The z0/z1 handlers call it before replying. If the removal cannot be committed, they log the error and reply E01 instead of OK.
• A removal that arrives while the core is running (non-stop mode) stays pending, as before.

Observed on hardware

Agentic Hardware-in-the-Loop (Agentic HIL) is an open-source toolchain, maintained by us, that puts real embedded boards into an AI agent's build, flash and debug loop. On pyOCD it runs its debug sessions through pyocd gdbserver. A GDB detach resumes the core, so it ends the server while GDB is still connected, which leaves the core halted (backend notes).

Its test suite includes a hardware recorder that drives pyocd gdbserver and GDB directly against a NUCLEO-F446RE (Cortex-M4) to record the behaviour that backend is built against (recorder). With pyOCD 0.45.1 and GDB 14.2.90 it ran this sequence: breakpoint on SysTick_Handler, continue, stop on it, -break-delete, then SIGTERM to the server while GDB was still connected. Three fresh pyocd gdbserver connects followed, two seconds apart, each reading the PC and the firmware's millisecond counter (recording).

• A fresh pyOCD connect lets the core run for a few tens of milliseconds while the pack's DebugCoreStart sequence runs. In the control, where the session halted the core with an interrupt instead of a breakpoint, the counter moved from 3615 to 3641 at the first connect.
• After the session that had deleted its breakpoint, the first connect found the core at SysTick_Handler+4 with the counter unchanged at 3717. The deleted breakpoint was still armed and stopped the core as soon as it ran. The counter moved again only at the following connects (3738, 3765).
• ST's ST-LINK_gdbserver 7.14.0 ran the same sequence on the same board, with one fresh connect after the server ended. In 3 of 3 runs the counter had moved on by about 2.4 seconds, e.g. from 3309 to 5709 ms, so the deleted breakpoint did not stop the core (recording).

The breakpoint manager and the z0/z1 handler are unchanged between 0.45.1 and develop, and the unit tests below fail on develop in the same way. This change itself has not been run on hardware yet.

Trade-off

With GDB's default set breakpoint always-inserted off, GDB removes its breakpoints at every stop and inserts them again before resuming. Before this change such a remove/insert pair cancelled out inside the manager. Now each stop writes the removals to the target, and the next resume writes the breakpoints back. The cost has not been measured.

An alternative is to keep removals deferred and commit them on detach, disconnect and shutdown, including a SIGTERM handler. That leaves the stop/resume path as it is. It does not help a server that is killed without running a handler, though, and the OK would still not mean that the target was updated.

Not addressed

• The server not exiting after the last client disconnects, also mentioned in GDB z1 returns OK but FPB hardware breakpoint remains armed after disconnect #2014.
• A GDBServer created without a core number targets the SoC object, which has no bp_manager, so removals there stay deferred as before. The command-line gdbserver always passes a core.
• FPB.remove_breakpoint() marks a comparator unused before writing it. If that write fails, a retry does not rewrite the comparator. This is pre-existing and unchanged here.

Tests

• New test/unit/test_gdbserver_breakpoints.py drives z0/z1 packets through GDBServer.breakpoint(). It runs against the real BreakpointManager, FPB and software breakpoint provider, over a fake core and a dict-backed fake access port (test/unit/mock_breakpoints.py). On develop without this change, 7 of its 10 tests fail:

  • After an acknowledged z1, the modelled FP_COMP0 is still enabled.
  • After an acknowledged z0 in RAM, the BKPT instruction is still in memory.
  • A removal that cannot be committed is answered OK.

  The other 3 tests cover behaviour that does not change.

• New test/unit/test_breakpoint_manager.py (12 tests) covers flush_removals() itself.

• Full unit suite (pytest): 1122 passed, 41 skipped. flake8 with the CI selection: 0 findings. test/import_all.py: 376 modules imported. These ran on Windows 11 with Python 3.13, on develop f8ca7e4.

Disclosure

We maintain Agentic HIL. This change and its tests were written by AI coding agents (Claude Code and OpenAI Codex) as part of our upstream work on the debug tools Agentic HIL uses, and reviewed by AI agents. No human has reviewed the code yet. Agentic HIL's part is the hardware recording above, made with pyOCD 0.45.1 before this change; the change itself has not been run on hardware. The commit carries Assisted-by: trailers.