[Performance] hbg: a rank's first bind costs ~4x steady-state host_orch, and pre-faulting at init cannot remove it

[ChaoWao]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

host_build_graph

Summary

A rank's first bind costs about 4x the steady-state host_orch phase, and the excess is paid again by every new process. On dsv4 FLASH decode the phase reads 1632 us / 1480 us on the two ranks' first bind against ~400-450 us once warm, decaying over roughly six binds.

The cost is not on the main thread: of bind 1's 929 process-wide minor faults, 55 are the main thread's and the other 874 belong to the eight Graph recorder threads. Each recorder thread pays ~107 faults and ~360 us of extra time on its own first recording, and which thread gets a job is a cv_.notify_one() race, so that cost smears randomly across a run's first several binds instead of landing once.

Pre-faulting the recorder storage at thread stand-up does not remove it. Five arms were measured; none moved the first-recording cost at all. Details and the root cause below. Filing this so the mechanism is recorded and the two remaining options are not re-derived.

Git Commit ID

3c89eaa

CANN Version

9.0.0

Driver Version

26.0.rc1

Host Platform

Linux (aarch64)

Reproduction

# Twelve rounds, because entering the steady state takes about six binds; a shorter run
# reports the decay as if it were the steady state (docs/dfx/hbg-bind-phases.md, Traps).
export SIMPLER_HBG_BIND_BREAKDOWN_ENABLE=1
export SIMPLER_LOG_LEVEL=TIMING
export SIMPLER_SKIP_DEVICE_RUN=1

task-submit --device auto --device-num 2 --timeout 1800 --max-time 1800 \
    --run '.venv/bin/python examples/a2a3/host_build_graph/deepseek_v4_flash_decode/main.py \
           -p a2a3 -d $TASK_DEVICE --skip-golden --rounds 12'

# Read the per-bind minflt column of the host_orch phase in arrival order. It decays;
# it does not sit at one value.
grep 'bind phase=host_orch ' <log>

Expected Performance

host_orch costs the same on a rank's first bind as on its tenth: one bind is one bind's work. A phase that is 4x more expensive the first time through means a process pays a startup penalty that a warm process does not, and a short run's per-bind average is not the per-bind cost.

Actual Performance

host_orch, dsv4 FLASH decode, 12 rounds = 24 binds (2 ranks), in arrival order:

bind	minor faults (process-wide)	of which main thread	duration
1	929	55	1632 us
2	924	54	1480 us
3	161	3	587 us
4	161	3	543 us
5	10	0	723 us
6	66	0	413 us
7-24	0-14	0-9	402-927 us

Per recorder thread, by that thread's own recording ordinal (16 threads = 8 per rank process):

ordinal	mean minor faults	mean duration
0	106.8	598 us
1	19.9	284 us
2	5.0	261 us
>= 3	~1	~230 us

Profiling Data (Optional)

Five warm-up arms, none of which moved the first-recording cost. Each arm made stand-up
(graph_recorder_stand_up_storage, which already runs at callable registration, before any
bind) write more of the recorder storage:

arm	stand-up faults per thread	ord=0 faults
baseline	47	106.8
new T[N]{} on the tensor pool, resize+clear on the reserved arrays, memset on the hazard-map pools	1438	115.2
tensor pool touched one byte per page through a volatile pointer	2461	114.0
the above plus 1 MB of thread stack touched at thread start (verified: faulted exactly 256 pages)	2461 + 256	114.0
every region touched per page: pool, nodes, the five reserved arrays, both hazard-map pools	3005	110.2

Where the residual faults land. perf record -e page-faults -c 1 -d -k mono on the fully
warmed build, dropping each thread's first 3005 faults (its stand-up), and matching every
faulting address against the [base, base+size) of each region printed at stand-up:

threads: 16   faults kept: 2075   per thread: 129.7
     584   28.1%  <no region>  (the thread's glibc malloc arena)
     507   24.4%  node_tensor_pool
     168    8.1%  thread stack
     139    6.7%  nodes
     135    6.5%  predicates
     124    6.0%  tensormap buckets
     124    6.0%  tensor_sources
     123    5.9%  tensormap entry_pool
      48    2.3%  tensormap task_entry_heads
     ...

The residual is the same regions that were just made resident, at the same addresses --
a probe printing each region's address at stand-up and again at the first recording shows
they are identical, so the storage is not reallocated.

Additional Context

Root cause: the pages are handed back to the kernel between stand-up and first use.

Every recorder region lives in one glibc per-thread arena. The addresses are all inside a
single 64 MB span:

buckets      fffcb4000e20
entry_pool   fffcb4008e80
free_list    fffcb4208eb0
pool         fffcb422af00     <- 4 MB node tensor pool
nodes        fffcb462af50
tsrc         fffcb4644f60
...          all within [fffcb4000000, fffcb8000000)

strace -e trace=madvise,munmap,mprotect over that span shows munmap of the whole 64 MB
arena plus a run of mprotect(PROT_READ|PROT_WRITE) on successive sub-ranges that line up
with the allocation order. glibc trims the arena top past M_TRIM_THRESHOLD (default
128 KB) with mprotect(PROT_NONE), which drops the pages; the next allocation re-opens the
range with mprotect(PROT_READ|PROT_WRITE), so the first write faults again. A page
written at init does not survive to first use.

This is the same mprotect that
docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md
identifies as the in-tree mmap_lock writer.

A second, independent reason the first two arms did nothing: a zero-fill of a fresh
allocation is foldable into calloc, which returns untouched zero pages. new T[N]{} and
the value-initialization inside vector::resize(n) both fold, so nodes.resize(GRAPH_MAX_NODES)
at stand-up makes nothing resident today. Only an unfoldable write works -- a volatile
per-page store, or a memset the compiler cannot pair with the allocation. Measured: stand-up
faulted 47 pages per thread with the foldable spellings and 3005 with the unfoldable ones.

Consequently the comment above graph_recording_reserve_storage
(src/{a2a3,a5}/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp) is wrong
where it says "Only the pool's used prefix ever becomes resident: it is default-initialized
and a body writes the bytes its tensors need" -- no part of the pool is resident after
stand-up, and the used prefix is re-faulted on every process's first recording.

The faults are not most of the duration gap. Bind 1's 924 faults are spread across eight
concurrent recorder threads, so their serial contribution is ~0.2 ms of the 1.2 ms gap. Per
recording: ord=0 is 598 us against ~240 us steady, and 107 faults at ~1.7 us is 182 us of
the 358 us excess. The other half is cold i-cache / branch predictors / first pass through
the code, which no amount of page touching reaches.

Note also that the fault count has already been shown not to buy time: the arm that sets
MALLOC_TRIM_THRESHOLD_ and MALLOC_MMAP_THRESHOLD_ to 1 GiB and MALLOC_TOP_PAD_ to
256 MiB removes 86% of the faults with no measurable latency change (recorded in the
investigation entry above).

Two options left, in the order worth trying:

1. mallopt(M_MMAP_THRESHOLD, ...) to stop glibc's dynamic threshold from growing past the
   pool size. The 4 MB pool should get its own mmap -- never trimmed, only unmapped at free
   -- but the dynamic threshold had grown past 4 MB, which is why it sits in the trimmable
   arena instead. One call at pool creation; directly tests the mechanism.
2. Warm by doing the work: have each recorder thread record one discarded body at
   registration. That holds the arena allocated so a trim cannot take it, and it also warms
   the i-cache and branch predictors -- the half of the gap option 1 cannot touch. It reuses
   no content across binds (the recording is thrown away). The obstacle is that registration
   has no real Graph boundary to record against, so it needs a synthetic one.

Related: #1717 (hbg: overlap device launch with host orchestration) bears on whether the
control plane is on the critical path at all, which decides how much this is worth.

Measured on this box with concurrent NPU jobs from other users present; the fault counts
are load-independent and are what the conclusions rest on, while the durations quoted from
the probe runs carry that noise.