hw-native-sys · doraemonmj · Aug 27, 2026
diff --git a/docs/capability-survey.md b/docs/capability-survey.md
@@ -220,8 +220,8 @@ values yield `SIMPLER_ERROR_ASYNC_COMPLETION_INVALID`.
 | Engine | a2a3 | a5 | Status |
 | ------ | ---- | -- | ------ |
 | COUNTER (default) | registered | registered | **Shipped** — `tests/st/worker/comm_domain/async_notify` runs onboard on both architectures; `tests/st/worker/comm_domain/deferred_notify` runs in sim on both and onboard on a2a3, through the `st-onboard-*` / `st-sim-*` jobs in `ci.yml`. Routed by `CASES[*]["platforms"]`, no `skipif` |
-| SDMA | build macro forced ON; runtime opt-in | `option(... OFF)` | a2a3 **Shipped** (the "SDMA pytest (a2a3)" step in `ci.yml`); a5 not built |
-| URMA | absent | full implementation | **Gated** — see below |
+| SDMA | build macro forced ON; runtime opt-in | built and provisioned with communication contexts | a2a3 **Shipped** (the "SDMA pytest (a2a3)" step in `ci.yml`); a5 demo runs in the ordinary A5 sweep |
+| URMA | absent | built and provisioned with communication contexts, best-effort | A5-only; exercised by `urma_deferred_completion_demo` without a build or environment gate. A rank set where registration fails runs SDMA-only, and that demo fails its golden rather than skipping — so a green run is evidence URMA moved data |
 | ROCE, CCU | enum only | enum only | **Name only** |
 
 **a2a3 SDMA is opt-in at runtime**, not "always on": the provider is always
@@ -235,23 +235,33 @@ without, traced to a single 300,000 ms remote TRS event timeout
 ([investigations/2026-07-a2a3-sdma-fault-teardown.md](investigations/2026-07-a2a3-sdma-fault-teardown.md),
 issue #1425).
 
-**a5 URMA is real code that cannot execute.** The scheduler walks `UrmaCqCtx`
+**a5 SDMA and URMA coexist in one communication context.** The scheduler walks `UrmaCqCtx`
 CQEs checking the owner bit, advances the tail and rings the doorbell
 (`src/a5/.../backend/urma/urma_completion_scheduler.h:133-215`); the kernel
-submits `TGET_ASYNC`/`TPUT_ASYNC<DmaEngine::URMA>` with 256 MB chunking. Both
-sit behind `PTO_URMA_SUPPORTED`, which is **defined nowhere in this repo and
-nowhere in the installed CANN pto headers**, so the `#else` branch returns
-`SIMPLER_ERROR_ASYNC_COMPLETION_INVALID` immediately. The host overlay macro is
-fully wired, so turning it on does not help — the device path stays unreachable.
-a5's SDMA and URMA overlays are mutually exclusive by CMake `FATAL_ERROR`
-because `CommContext` exposes a single `workSpace` pair
-(`src/a5/platform/onboard/host/CMakeLists.txt:49-53`).
-
-**HCCL is bootstrap, not data movement.** The complete set of functions called
-is `HcclGetRootInfo`, `HcclCommInitRootInfo`, `HcclBarrier`, `HcclCommDestroy`.
-There is no `HcclAllReduce` / `AllGather` / `Send` / `Recv` anywhere; every
-shipped collective is a hand-written AIV kernel that computes a peer pointer
-from the symmetric window
+submits `TGET_ASYNC`/`TPUT_ASYNC<DmaEngine::URMA>` with 256 MB chunking. The
+pinned PTO-ISA defines `PTO_URMA_SUPPORTED` for DAV_3510. The host provisions
+the process-global SDMA workspace and communicator-scoped URMA workspace before
+uploading `CommContext`; the leading `sdmaWorkSpace` pair carries SDMA,
+and the appended `urmaWorkSpace` pair carries URMA. A derived context also
+carries `urmaWindowOffset`, translating its domain-local window offsets back
+to the registered base MR. URMA metadata is indexed by communicator rank, and
+derived contexts map domain-local ranks onto those communicator ranks. Dynamic
+domains are slices of the registered arena, so sequential and concurrent
+domains do not create extra HCCL memory registrations or channels. Engine-specific kernels run
+from the same default build without an environment selector. Both workspaces
+are best-effort: a transport that fails to initialize leaves its `CommContext`
+pair zero and its kernels self-skip, and the domain is handed out regardless.
+URMA additionally agrees that outcome across ranks through the
+`base_urma_ready` rendezvous, so no rank issues an RDMA against a peer that
+holds no registration.
+
+**HCCL is control-plane setup, not collective data movement.** Communicator
+lifecycle uses `HcclGetRootInfo`, `HcclCommInitRootInfo`, `HcclBarrier`, and
+`HcclCommDestroy`; A5 URMA setup additionally uses `HcclCommMemReg`,
+`HcclRankGraphGetLinks`, and `HcclChannelAcquire` to register memory and create
+transport channels. There is no `HcclAllReduce` / `AllGather` / `Send` / `Recv`
+in the collective data path; every shipped collective is a hand-written AIV
+kernel that computes a peer pointer from the symmetric window
 (`tests/st/worker/collectives/allreduce/kernels/aiv/allreduce_ring_kernel.cpp:59-61`).
 
 **Fabric is undocumented.** a2a3 prefers `alloc_windows_via_fabric()` with
@@ -261,10 +271,10 @@ only when Fabric is unsupported
 occurrences of "fabric". No `.md` in the repo describes Fabric as a
 memory-sharing mechanism.
 
-One ABI fact constrains portability: the AICore→AICPU completion struct
-diverged — `DeferredCompletionEntry` is 24 bytes on a2a3 and 32 on a5, the extra
-8 being the `backend_cookie` URMA's poll needs. A URMA-on-a2a3 port must widen a
-struct that is currently frozen by `static_assert`.
+The AICore→AICPU completion ABI is currently aligned:
+`DeferredCompletionEntry` is 32 bytes on both a2a3 and a5, and both carry
+`backend_cookie`. A2/A3 does not currently register or implement the URMA
+completion backend despite sharing this ABI field.
 
 ## Open questions
 
@@ -273,18 +283,17 @@ Unresolved after this survey, in rough order of how much they block:
 1. **What are `ASYNC_ENGINE_ROCE` and `ASYNC_ENGINE_CCU` for?** No design doc,
    investigation entry, or code comment says whether they are reserved slots or
    leftovers from a dropped design.
-2. **Has a5 URMA ever run on silicon?** No CI run, test artifact, or
-   investigation attests to it.
+2. **How broad is a5 URMA coverage?** The two-rank deferred-completion demo has
+   passed bring-up on A5 and is selected by the ordinary A5 sweep after this
+   change, but there is no retained multi-topology or long-running artifact.
 3. **Which CANN mitigation closed issue #822, and is Path B usable on CANN
    9.0.0?** The doc says "CANN-side mitigation landed" without naming it, and
    nobody re-ran the repro.
 4. **Does CANN's one-inner-SO-per-process latch still exist?** The whole
    one-process-per-`(arch, runtime)` ChipWorker model rests on it, but it is
    asserted only from CANN source paths that are not vendored here and no
    in-repo probe detects it.
-5. **Where would `PTO_URMA_SUPPORTED` ever be defined?** Not in this repo, not
-   in the installed CANN pto headers.
-6. **Which platforms lack Fabric support?** Stated only as a code comment, never
+5. **Which platforms lack Fabric support?** Stated only as a code comment, never
    enumerated, and a5's divergence to the V1 handle route is undocumented.
 
 ## Documentation drift found while compiling this survey

diff --git a/docs/comm-domain.md b/docs/comm-domain.md
@@ -213,21 +213,59 @@ symmetric window is realized:
 
 | Aspect | Sim | HCCL (onboard) |
 | ------ | --- | -------------- |
-| Window memory | POSIX shm + `ftruncate`, mmap'd per rank | a2a3: Fabric V2 handle exchange (`ACL_MEM_SHARE_HANDLE_TYPE_FABRIC`), falling back to VMM + shareable-handle IPC where Fabric is unsupported. a5: VMM shareable handles only. Cross-card P2P via `aclrtDeviceEnablePeerAccess` on both |
+| Window memory | POSIX shm + `ftruncate`, mmap'd per rank | a2a3: Fabric V2 handle exchange (`ACL_MEM_SHARE_HANDLE_TYPE_FABRIC`), falling back to VMM + shareable-handle IPC where Fabric is unsupported. a5: slices of one per-rank VMM arena registered for the communicator lifetime, 200 MiB unless `Worker(comm_arena_size=…)` says otherwise. Cross-card P2P via `aclrtDeviceEnablePeerAccess` on both |
 | Subset barrier | shm-header atomic, `allocation_id`-scoped | file barriers, `allocation_id`-scoped |
-| Window init | window zeroed before the subset barrier (`memset`) | window zeroed before the handle is announced (`aclrtMemset`) |
-| Async-DMA workspace | n/a | a2a3: opt-in per Worker (`enable_sdma`); a5: SDMA by default, URMA as an opt-in alternative |
+| Window init | window zeroed before the subset barrier (`memset`) | a2a3: window zeroed before the handle is announced. a5: the base arena is registered and mapped during communicator init; every newly assigned or reused slice is zeroed at domain allocation before that domain is returned to the caller |
+| Async-DMA workspace | n/a | a2a3: opt-in per Worker (`enable_sdma`); a5: communicator-scoped SDMA and URMA workspaces are inherited by derived contexts, whose rank map supports arbitrary subsets/reorderings. Either may be absent — see below |
 
 The window is zero-initialized on both backends so scratch/signal protocols see
 a known starting state (matching the historical static-path contract).
 
-The wipe happens before the window becomes reachable by any peer — before the
-shareable handle is announced on HCCL, before the `ready_count` barrier on sim.
-A peer that clears the subset barrier can return, launch its kernel and store a
-barrier signal into this rank's window immediately; a wipe issued after that
-point can erase a signal the owner has not yet waited on, and the owner then
-waits on it forever. The rank skew that opens that window grows with host load,
-so the resulting hang shows up only under a loaded box.
+### A5: arena size and transport availability
+
+The A5 arena is allocated once per communicator and every domain is a slice of
+it, so **`comm_arena_size` is the ceiling on all live domains combined**, not a
+per-domain limit. A request that no free run can satisfy raises `MemoryError`
+rather than falling back to a private allocation. The default is 200 MiB; pass
+`Worker(comm_arena_size=<bytes>)` to change it. The backend rounds the request
+up to VMM granularity and reports the granted size back, which is what the
+slice allocator uses.
+
+Neither async transport is required for a domain. SDMA and URMA are each
+provisioned best-effort at `comm_alloc_windows`, and a transport that fails to
+initialize simply leaves its `CommContext` pair (`sdmaWorkSpace` /
+`urmaWorkSpace`) zero; the kernels guard on that and self-skip. A communicator
+that came up SDMA-only still hands out windows normally.
+
+A self-skipping kernel moves no data, so this degrades availability of a
+transport, never correctness of a workload: anything that depended on the
+transfer sees unwritten output and fails its own verification. The a5
+`sdma_async_completion_demo` and `urma_deferred_completion_demo` both fail
+their goldens on a skipped transfer rather than passing vacuously, which is
+what makes a green run of either one evidence that its engine ran.
+
+URMA's outcome is agreed across ranks rather than decided per rank: the
+`base_urma_ready` rendezvous carries each rank's result and every rank takes
+the AND. A rank whose own registration succeeded while a peer's failed
+therefore publishes a zero `urmaWorkSpace` too, because issuing an RDMA against
+a peer that holds no registration is worse than not issuing one. That rank
+still keeps its workspace manager alive, since `Finalize` must run after
+`HcclCommDestroy` either way.
+
+On sim and a2a3, the wipe happens before the window becomes reachable by any
+peer — before the `ready_count` barrier on sim and before the shareable handle
+is announced on a2a3 HCCL. A5 is different: its base arena is already
+registered and peer-mapped when each rank clears a slice, both on the slice's
+first assignment and after reuse. The base arena itself is not published as a
+domain. Safety comes from the parent waiting for every member's allocation RPC
+(including that clear) to complete before it publishes the
+`CommDomainHandle`; no task can access the slice during either wipe.
+
+Once a domain is published, a peer can launch its kernel and store a barrier
+signal into this rank's window immediately. A wipe issued after that point can
+erase a signal the owner has not yet waited on, and the owner then waits on it
+forever. The rank skew that opens that window grows with host load, so the
+resulting hang shows up only under a loaded box.
 
 On a2a3, async-DMA resources are a Worker-level opt-in, not a
 communication-domain property. Construct the Worker with `enable_sdma=True` and

diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -55,8 +55,8 @@ To use a different PTO-ISA revision, update `pto_isa.pin` to the desired
 diff and applies the same revision to install-time runtime builds and run-time
 kernel compilation.
 
-For platforms that embed PTO-ISA headers into onboard host runtimes (a2a3
-always; a5 when an async workspace overlay is ON), builds record the actual
+For platforms that embed PTO-ISA headers into onboard host runtimes (a2a3 and
+a5), builds record the actual
 PTO-ISA git HEAD used for each runtime in `build/lib/pto_isa_build.json`.
 This JSON is artifact provenance, not a second configuration source. Lookup
 of those runtimes **requires** the metadata file: if it is missing, or if it

diff --git a/docs/tensormap-and-ringbuffer-a2a3-vs-a5.md b/docs/tensormap-and-ringbuffer-a2a3-vs-a5.md
@@ -109,7 +109,7 @@ The functional differences group into the following themes:
 | Cache coherence | Hardware coherence model | Yes | Retain the required invalidate/flush operations on A2/A3; do not copy unnecessary maintenance operations to A5 |
 | PMU collection | Hardware PMU and platform collection protocol | Yes | Retain the different counter counts, readers, and FIN submission paths |
 | System counter and DMB | Hardware timing and register layout | Yes | Use the constants for each platform |
-| URMA completion | A5-specific implementation and product capability gate | Yes, for now | Retain the A5 path; do not claim that URMA is available in the default build |
+| URMA completion | A5-specific implementation and product capability | Yes, for now | Retain the A5 path; the default onboard build provisions it together with SDMA |
 | Next-block prefetch | A2/A3-only performance optimization | No | Retain on A2/A3; validate on A5 before considering a port |
 | Scheduler progress publication | AICPU topology and measured publication cost | No | Retain A5's 16-task batching; keep per-advance publication on A2/A3, where the portable implementation showed no significant benefit |
 | Fatal teardown | Software reliability strategy | No | Retain the current implementations; decide whether to converge after measuring the worst-case A5 teardown time |
@@ -184,18 +184,17 @@ platform's `platform/include/common/platform_config.h`.
 | `runtime/scheduler/scheduler_completion.cpp` | After FIN, A2/A3 invokes the AICPU MMIO reader for eight counters; A5 commits the ten-counter slot written by the AICore |
 | `platform/shared/aicpu/pmu_collector_aicpu.cpp` | Implements the A2/A3 direct MMIO read and the A5 staging-slot consumption paths |
 
-### Optional A5-Specific URMA Backend
+### A5-Specific URMA Backend
 
 A5 contains the source path for issuing URMA completion requests, creating
 deferred entries, forwarding FIN, and polling/retiring CQ entries. A2/A3
 currently registers only the COUNTER and SDMA completion backends.
 
-The repository does not currently define `PTO_URMA_SUPPORTED`. A5 therefore
-compiles the shared ABI, mailbox, CQ polling/retirement, and related paths, but
-the kernel path that successfully issues URMA PTO instructions is unreachable.
-The current state is "implemented but disabled by default." It neither means
-that the default A5 build supports URMA nor proves that the A2/A3 hardware does
-not support URMA.
+The pinned PTO-ISA defines `PTO_URMA_SUPPORTED` for DAV_3510. A5 compiles the
+request path together with the shared ABI, mailbox, and CQ polling/retirement
+paths. Its host communication context provisions SDMA and URMA workspaces
+together, so the default build supports both engines without a selector. This
+does not imply that A2/A3 hardware supports URMA.
 
 Both platforms already share `CompletionToken::backend_cookie`,
 `ASYNC_ENGINE_URMA`, the 32-byte `DeferredCompletionEntry`, end-to-end cookie
@@ -341,7 +340,6 @@ comparison. Examples and tests unique to either platform primarily reflect
 chip-feature validation and test-porting progress; they cannot be used to
 infer whether the runtime supports a shared algorithm.
 
-For example, A5 has `urma_deferred_completion_demo`, but this does not mean
-that the current build defines `PTO_URMA_SUPPORTED`. Likewise, the absence of a
-workload on one platform does not automatically mean that the corresponding
-runtime capability is unavailable.
+For example, A5's `urma_deferred_completion_demo` exercises an A5-only
+capability. The absence of a corresponding workload on another platform does
+not automatically establish a hardware limitation there.
diff --git a/docs/user/reference/python-api.md b/docs/user/reference/python-api.md
@@ -39,6 +39,7 @@ into `**config` and validated later. The recognized keys:
 | `device_ids` | L3+ | one chip child process per entry |
 | `num_sub_workers` | L3+ | host-side Python callables to fork |
 | `enable_sdma` | a2a3 | provisions the SDMA workspace; defaults to `False` |
+| `comm_arena_size` | a5 | bytes of the communicator-lifetime arena every communication domain is sliced from; `0` (default) takes the backend's 200 MiB. This is the ceiling on all live domains combined |
 | `heap_ring_size` | all | heap ring sizing |
 | `remote_heap_ring_size`, `remote_session_timeout_s` | L4 | remote-session sizing and timeout |
 

diff --git a/examples/a5/tensormap_and_ringbuffer/README.md b/examples/a5/tensormap_and_ringbuffer/README.md
@@ -38,7 +38,7 @@ direct `Worker` communication-domain walkthrough from construction through
 | Example | Mechanism |
 | ------- | --------- |
 | [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. Enabled by default on a5 onboard. |
-| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** — `kernel_consumer.cpp` is byte-identical to the SDMA demo's, so the transport is the only variable. The two overlays are **mutually exclusive in one build**, so comparing them means rebuilding. |
+| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** — `kernel_consumer.cpp` is byte-identical to the SDMA demo's, so the transport is the only variable. It runs from the same default build as the SDMA demo. |
 
 The cross-architecture notification-counter and deferred-notify watchdogs live
 under [`tests/st/worker/comm_domain/`](../../../tests/st/worker/comm_domain/).

diff --git a/examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md b/examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md
@@ -20,16 +20,12 @@ Every rank's window is laid out identically, so an offset is rank-independent.
 
 ## Requirements
 
-The a5 host runtime includes the async-SDMA workspace by default:
+The A5 host runtime includes both async-SDMA and URMA workspaces by default:
 
 | Gate | Effect |
 | ---- | ------ |
 | `CASES[*]["platforms"] = ["a5"]` | deselected on any other `--platform` |
 | `CASES[*]["config"]["device_count"] = 2` | needs two dies |
-| `@pytest.mark.skipif(_urma_workspace_enabled())` | skipped when `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` selects the URMA backend |
-
-URMA replaces SDMA in a URMA build. Rebuild without
-`SIMPLER_ENABLE_PTO_URMA_WORKSPACE` before running this demo.
 
 ```bash
 pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo \
@@ -40,5 +36,5 @@ Wrap the hardware run in `task-submit` on a shared box.
 
 ## Compare with
 
-- [`../urma_deferred_completion_demo/`](../urma_deferred_completion_demo/) — the same protocol over URMA. `kernel_consumer.cpp` is byte-identical; only the transfer kernel, its completion header, and the build flag differ. **The two overlays are mutually exclusive in one build**, so comparing them means rebuilding — that README has the detail.
+- [`../urma_deferred_completion_demo/`](../urma_deferred_completion_demo/) — the same protocol over URMA. `kernel_consumer.cpp` is byte-identical; only the transfer kernel, its completion header, and its workspace field differ. Both demos run from the same build without environment changes.
 - [`examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/`](../../../a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/) — the a2a3 port of this demo, which needs no overlay flag.