[Performance][A5 TMR] Unexpected cross-commit performance shifts disproportionate to code changes

[yanghaoran29]

Platform

a5 (Ascend 950 hardware), card 1

Runtime Variant

tensormap_and_ringbuffer

Summary

On a local A5 TMR benchmark environment, several commits that were not intended as performance changes produced 5%–8% shifts in Device or Orchestrator time even though they did not add corresponding work to the measured hot path. The scale of the performance change is disproportionate to the scale and execution stage of the code change.

Two cases have been confirmed:

Commit	Functional change	Extra work in the measured hot path	Original anomaly
de32536a	Fix the callable dispatch-table lifetime by clearing an 8 KiB table during Host bind	No; the clearing happens before the Device Orchestrator window	PA Orchestrator became 7.37%–8.47% slower
35f195bd	Preserve dependency flags for dep-gen replay	No; dep-gen was disabled in these benchmarks	BGEMM/PA Device or Orchestrator became 5.23%–8.13% faster

The executed algorithmic steps do not explain these shifts. A hypothesis more consistent with the observations is that A5 TMR performance is sensitive to the binary instruction layout of hot functions, and that unrelated source changes can indirectly alter that layout.

To test this hypothesis without mixing in a one-sided alignment change, I applied the same 64-byte alignment to three hot entry points in both each commit and its direct parent, then compared aligned Commit against aligned Parent. All eight original |delta| >= 5% anomalies moved below 5%:

• de32536a: PA Case1/Case2 Orchestrator changed from 7.37%/8.47% slower to 4.60%/4.18% slower.
• 35f195bd: BGEMM Device/Orchestrator changed from 5.59%/8.13% faster to 2.41%/3.58% faster.
• 35f195bd: PA Case1/Case2 Orchestrator changed from 5.23%/5.91% faster to 1.53%/1.95% slower.
• 35f195bd: manual PA Case1/Case2 Orchestrator changed from 5.23%/6.32% faster to 0.86%/1.23% slower.

This does not prove that 64-byte alignment is the final fix. It does indicate that function layout is likely an important contributor to the unexpected variance and that explicit layout control is worth investigating as a mitigation.

Why this is a performance-debugging problem

This behavior makes performance results difficult to trust and interpret:

• A correctness or DFX commit can appear to provide a 5%–8% optimization or regression.
• An algorithm expected to provide a positive gain can show an additional regression of around 10% in some cases because of layout changes. The result then mixes algorithmic effects with layout noise, making it impossible to determine whether the optimization itself is correct.
• The benchmark direction can disagree with the actual change in executed work.
• Performance bisection can incorrectly identify an unrelated commit as the root cause.
• Small logging, diagnostic, or correctness changes can add substantial performance-debugging cost.

The goal is not to make every benchmark faster through alignment. The goal is to reduce algorithm-independent layout noise so that Device and Orchestrator changes correspond more reliably to actual algorithmic changes.

Git Commit IDs

Case	Commit	Direct parent	PTO ISA
Callable dispatch-table lifetime	de32536a7a67d2148f43560263e2f26d5ef7f308	0d74814d98ea9192d3e6c661a91e4d60c97a3c16	f51c92f610827daad0ddfb383072e03d514b4ae9
dep-gen dependency flags	35f195bd40f3cbe999ae75e85d7fb8c384cb5379	a64147b7a7fbb80e104cade63325d6150a742671	be5ccb765a4ce5d14ca5da8b0e2f182d7f003369

Each commit is compared only with its direct parent so that unrelated intermediate performance changes are not included.

Reproduction and measurement methodology

Benchmark parameters:

--platform a5 --device 1 --rounds 100 --skip-golden --manual include

• Platform: A5 / Ascend950PR_9579
• Runtime: tensormap_and_ringbuffer
• Device: card 1
• 100 iterations per case, arithmetic mean
• All NPU runs were serialized through task-submit with exclusive use of card 1
• Positive delta means Commit is slower than its direct Parent; negative delta means Commit is faster
• Only Device and Orchestrator are used for anomaly detection
• |Commit / Parent - 1| >= 5% is treated as anomalous

The alignment experiment used paired builds:

aligned Parent = Parent + the same three aligned(64) annotations
aligned Commit = Commit + the same three aligned(64) annotations

The comparison is aligned Commit / aligned Parent, not aligned Commit against an unaligned Parent. This keeps the functional commit change separate from the effect of applying alignment on only one side.

Evidence 1: Host-bind change accompanied by a PA Orchestrator regression

Commit de32536a (Fix: bound the dispatch table's lifetime to the active callable (#1847)) prevents stale addresses from remaining in func_id_to_addr_ after callable deregistration. Its main A5 TMR change clears 1,024 function addresses before repopulating the current callable's valid entries:

void Runtime::clear_function_bin_addrs() {
    for (int i = 0; i < RUNTIME_MAX_FUNC_ID; i++) {
        dev.func_id_to_addr_[i] = 0;
    }
}

The Host bind flow changes from:

find callable
-> populate K func_id entries for the current callable
-> set active_callable
-> finish runtime bind

to:

find callable
-> clear 1,024 func_id entries
-> populate K func_id entries for the current callable
-> set active_callable
-> finish runtime bind

This adds 1,024 fixed 64-bit writes, or 8 KiB, during Host bind. It occurs before the Device Orchestrator timing window. Once Device execution begins, the valid function mapping, task count, submit count, TensorMap work, fan-in operations, dependency edges, Scheduler dispatch/completion operations, and release count remain unchanged.

The change may make Host bind slightly slower, but it does not algorithmically explain a stable 7%–8% PA Orchestrator regression.

All times below are in microseconds:

Case	Metric	Original Parent	Original Commit	Original delta	Aligned Parent	Aligned Commit	Aligned delta
PA unroll Case1	Device	1974.2	2016.7	+2.15%	1922.9	1888.3	-1.80%
PA unroll Case1	Orchestrator	1639.5	1760.4	+7.37%	1470.8	1538.5	+4.60%
PA unroll Case2	Device	1096.8	1112.0	+1.39%	1070.1	1067.6	-0.23%
PA unroll Case2	Orchestrator	769.0	834.1	+8.47%	662.2	689.9	+4.18%

The two anomalies moved below the 5% threshold, and the maximum absolute difference dropped from 8.47% to 4.60%.

Alignment did not affect only the Commit:

• PA Case1 Orchestrator: Parent improved by 10.29%, while Commit improved by 12.61%.
• PA Case2 Orchestrator: Parent improved by 13.89%, while Commit improved by 17.29%.

The Commit responded more strongly to the same alignment, compressing the original Commit/Parent gap. This makes it difficult to attribute the original 7%–8% Device-side regression to the new Host clearing loop.

Evidence 2: disabled dep-gen change accompanied by BGEMM/PA improvements

Commit 35f195bd (Fix: preserve explicit dependency flags in dep_gen replay (#2057)) fixes lost dependency flags in the opt-in dep-gen record/replay path. It mainly:

• adds explicit_dep_kinds to DepGenRecord;
• records one additional DepFlags value per explicit dependency when dep-gen is enabled;
• restores and validates the flags during replay;
• increases DepGenRecord from 4,672 B to 4,736 B;
• changes overflow record capacity from 582 dependencies to 524;
• fixes profiler-buffer ownership after a push-gate timeout.

dep-gen was not enabled in these benchmarks. Before and after the commit, the normal submit path is:

check is_dep_gen_enabled()
-> false
-> skip dep-gen recording
-> continue fan-in, TensorMap, and task construction

Therefore, the added dependency flags are not read or copied, the larger record is not used for recording, replay does not execute, and task/dependency/Scheduler/release counts are unchanged. This does not algorithmically explain a 5%–8% Device or Orchestrator improvement while dep-gen is disabled.

All times below are in microseconds:

Case	Metric	Original Parent	Original Commit	Original delta	Aligned Parent	Aligned Commit	Aligned delta
BGEMM Case0	Device	1549.6	1462.9	-5.59%	1499.7	1463.6	-2.41%
BGEMM Case0	Orchestrator	1425.1	1309.3	-8.13%	1361.6	1312.8	-3.58%
PA unroll Case1	Device	1856.7	1851.0	-0.31%	1838.2	1836.0	-0.12%
PA unroll Case1	Orchestrator	1423.6	1349.2	-5.23%	1343.6	1364.2	+1.53%
PA unroll Case2	Device	1062.5	1061.9	-0.06%	1036.2	1033.4	-0.27%
PA unroll Case2	Orchestrator	720.9	678.3	-5.91%	671.4	684.5	+1.95%
manual PA Case1	Device	1872.2	1899.9	+1.48%	1889.3	1889.6	+0.02%
manual PA Case1	Orchestrator	1282.2	1215.1	-5.23%	1225.7	1236.3	+0.86%
manual PA Case2	Device	1061.4	1062.5	+0.10%	1060.9	1067.7	+0.64%
manual PA Case2	Orchestrator	606.0	567.7	-6.32%	576.6	583.7	+1.23%

All six original Device/Orchestrator anomalies moved below the 5% threshold:

• BGEMM Device: 1549.6/1462.9 us (-5.59%) became 1499.7/1463.6 us (-2.41%).
• BGEMM Orchestrator: 1425.1/1309.3 us (-8.13%) became 1361.6/1312.8 us (-3.58%).
• PA Case1 Orchestrator: 1423.6/1349.2 us (-5.23%) became 1343.6/1364.2 us (+1.53%).
• PA Case2 Orchestrator: 720.9/678.3 us (-5.91%) became 671.4/684.5 us (+1.95%).
• manual PA Case1 Orchestrator: 1282.2/1215.1 us (-5.23%) became 1225.7/1236.3 us (+0.86%).
• manual PA Case2 Orchestrator: 606.0/567.7 us (-6.32%) became 576.6/583.7 us (+1.23%).

The maximum absolute difference dropped from 8.13% to 3.58%.

The Parent and Commit again did not receive the same absolute benefit: alignment improved the Parent Orchestrator results by about 4.41%–6.87%, while the aligned Commit Orchestrator results changed by about +0.27%–+2.82% relative to the original Commit. Once the slower Parent caught up, the apparent performance gain of the Commit disappeared.

Therefore, with dep-gen disabled, the original 5%–8% improvements should not be interpreted as algorithmic gains from this commit.

Possible mitigation: explicitly stabilize hot entry-point layout

The minimal intervention tested here was to add:

__attribute__((aligned(64)))

to the same three A5 TMR entry points in both Parent and Commit:

1. submit_task_common(): used by every Orchestrator task submission;
2. SchedulerContext::resolve_and_dispatch(): Scheduler main-loop entry;
3. SchedulerContext::complete_slot_task(): task-completion entry.

No noinline, runtime branches, state variables, or algorithmic logic were added. All 18 target symbols across the six aligned Parent/Commit builds were verified to have addresses divisible by 64.

If 64-byte alignment is not explicitly requested at compile time, whether a function lands on a 64-byte boundary is largely outside the programmer's direct control. It is determined by generated code size, function order within object files, section placement, and linker layout. Consequently, an algorithm-independent change can alter the size or order of code and indirectly move hot function entry addresses or change their alignment.

The complete before/after summary is:

Commit	Case / metric	Original Commit/Parent	Aligned Commit/Parent	Interpretation
de32536a	PA Case1 Orchestrator	+7.37%	+4.60%	7.37% slower before, 4.60% slower after
de32536a	PA Case2 Orchestrator	+8.47%	+4.18%	8.47% slower before, 4.18% slower after
35f195bd	BGEMM Device	-5.59%	-2.41%	5.59% faster before, 2.41% faster after
35f195bd	BGEMM Orchestrator	-8.13%	-3.58%	8.13% faster before, 3.58% faster after
35f195bd	PA Case1 Orchestrator	-5.23%	+1.53%	5.23% faster before, 1.53% slower after
35f195bd	PA Case2 Orchestrator	-5.91%	+1.95%	5.91% faster before, 1.95% slower after
35f195bd	manual PA Case1 Orchestrator	-5.23%	+0.86%	5.23% faster before, 0.86% slower after
35f195bd	manual PA Case2 Orchestrator	-6.32%	+1.23%	6.32% faster before, 1.23% slower after

This is only a possible mitigation and should not be treated as a final fix:

• Entry-point alignment does not guarantee better absolute performance for every case.
• It constrains only the entry address, not the whole function body, neighboring functions, or full linked layout.
• The same alignment can make one version faster and another slower.
• In a separate Main experiment, aligning all three entries caused a 7.63% regression in BGEMM's full AICore window.

Any eventual mitigation should be evaluated by stability across commits, correspondence between measured deltas and actual algorithmic work, and the absence of new stable regressions across the full benchmark set—not by whether one aligned build is the fastest.

Expected Performance

• Commits that do not change the executed algorithm should normally keep Device/Orchestrator changes within a reasonable noise range.
• DFX, logging, and correctness fixes should not produce unexplained 5%–8% shifts.
• Real algorithmic changes should produce repeatable and explainable benchmark results.
• Linux A5 onboard builds should be stable without breaking macOS/simulator or unsupported compilers.
• The build should be able to verify whether the intended layout constraint is present in the final binary.

Actual Performance

The two non-performance commits above produced eight Device/Orchestrator anomalies of at least 5%, with a maximum absolute shift of 8.47%. Applying the same three 64-byte entry alignments to each Commit and its direct Parent reduced all eight Commit/Parent differences below 5%; the remaining maxima were 4.60% for de32536a and 3.58% for 35f195bd.

Local environment and scope

All data was collected on only one local host and one A5 card. The behavior has not yet been reproduced on another machine, card, CI runner, or toolchain.

Item	Environment
Host OS	Ubuntu 24.04.3 LTS
Host kernel	Linux 6.8.0-138-generic
Host architecture	x86_64
Host CPU	2 x AMD EPYC 9575F, 128 physical / 256 logical CPUs
NPU	A5 / Ascend950PR_9579, card 1
CANN toolkit	9.2.0, inner version V100R001C11B134
Python	3.12.13
Host GCC/G++	15.2.0
AArch64 cross G++	13.3.0
CMake	4.4.0
NPU isolation	Runs serialized through task-submit, exclusive card 1

This may be specific to this host, current CANN/firmware/compiler combination, card, or PTO ISA pin. Other A5 systems may have different layout sensitivity, and a2a3 may not show the same behavior. The paired Commit/Parent tests should be repeated on at least a second A5 host or an A5 CI runner before adopting a default code change.

macOS and cross-toolchain considerations

The experiment directly used GNU/Clang-style __attribute__((aligned(64))). Its acceptance and final effect depend on the compiler, object format, and linker. Apple Clang may accept the attribute, but Linux ELF/AArch64 behavior should not be assumed for Mach-O and Apple linker builds.

A production-quality approach should:

1. define a centralized portable macro instead of repeating a raw attribute;
2. enable it by target platform, initially only where it is validated;
3. verify target symbol addresses after linking rather than trusting source annotations alone;
4. evaluate dedicated text sections/linker constraints if stronger control is required;
5. degrade safely to no attribute on unsupported compilers;
6. compile-test Linux GCC/AArch64, Linux Clang, and macOS Apple Clang;
7. rerun the full A5 benchmark suite to check for new Device/AICore regressions.

For now, stable A5 TMR hot-function layout should be treated as an investigation direction, not as a recommendation to add unconditional aligned(64) annotations on all platforms.

Suggested follow-up

• Repeat both paired Commit/Parent tests on a second A5 host or A5 CI runner.
• Align each of the three functions independently to identify which entries matter.
• Compare 16-, 32-, and 64-byte constraints instead of assuming 64 bytes is optimal.
• Evaluate dedicated text sections or linker constraints.
• Add target-compiler capability detection and post-link symbol verification.
• Verify compilation and safe fallback on macOS Apple Clang/simulator.
• Run the complete A5 benchmark suite and reject mitigations that introduce stable absolute regressions.
• Use reduced cross-commit variance with no new regressions as the acceptance criterion.