[Code Health] Onboard AICPU's log threshold is a one-shot snapshot taken at device init

[ChaoWao]

Category

Code Health

Component

Logging / onboard AICPU

Description

A set_level after device init never reaches onboard AICPU. The host side has been live since #1845 — every module reads state()->threshold on each record — but the device side receives a one-shot snapshot:

// src/common/platform/onboard/host/device_runner_base.cpp
init_args.log_level = static_cast<uint32_t>(HostLogger::get_instance().level());

// src/a2a3/platform/onboard/aicpu/kernel.cpp:142
set_log_level(static_cast<int>(init_args->log_level));

simpler_aicpu_init runs once per Worker, so whatever the threshold was at device init is what the AICPU keeps for the Worker's life.

This is the half of #1792's "Related defect, partially resolved" that stayed open when the seven numbered items closed. It is carved out here so closing that issue does not drop it.

Consequence, stated narrowly: any claim that "the log level is live" must scope itself to host modules. A user raising verbosity to chase a device-side problem mid-run silently gets nothing from the AICPU, which is exactly when they most want it.

Location

• src/common/platform/onboard/host/device_runner_base.cpp — ensure_aicpu_init_launched() builds InitArgs.log_level
• src/a2a3/platform/onboard/aicpu/kernel.cpp:142 — the one-shot set_log_level
• src/common/platform/include/aicpu/device_log.h — set_log_level declaration
• src/common/log/include/common/host_log_state.h — threshold, which the host side reads per record

One incidental path existed and is now gone

Worth recording so the history is not misread as "it never worked". Before #2092, provision_dma_workspace cleared aicpu_init_launched_ and re-launched simpler_aicpu_init to republish the DMA workspace addresses, which incidentally re-pushed the log threshold. #2092 folded provisioning into simpler_init so that entry launches exactly once — the point of the change, and correct — but it removed that accidental refresh. Nothing depended on it and it only ever fired for SDMA-enabled Workers at init, so this is not a regression report; it is a note that the accidental path is no longer available as a workaround.

Proposed Fix

Not obvious, and the shape matters more than the speed, so this is a question rather than a plan.

The host cannot simply write into the AICPU's memory: InitArgs is a launch payload, not shared state. Three shapes, with the objection to each:

1. Re-launch simpler_aicpu_init on a threshold change. Cheapest to write, and it is what the accidental path did. But it makes an ordinary set_level do a device launch plus a stream sync, and Refactor: provision the async-DMA workspace inside simpler_init #2092 exists precisely to stop that entry being launched more than once — reintroducing a second launch on a user-facing call is going the wrong way.
2. Put the threshold in a device-resident location the host can poke. The AICPU already reads resident globals that simpler_aicpu_init latched, so a single uint32_t the host can H2D on change is conceivable. Needs someone who knows whether a host write to that region is coherent against an AICPU read without a launch, which is not a question to guess at (see .claude/rules/ascend.md on what AICore/AICPU can and cannot reach).
3. Decide it should stay one-shot and say so. Defensible: the threshold is a run-level decision, a Worker is cheap to recreate, and the device log volume is exactly where an accidental DEBUG is most expensive. If this is the answer, the work is documentation — docs/logging.md should state that the device threshold is fixed at device init, and set_level's own docstring should scope its liveness claim to host modules.

(3) is the likely answer and it is cheap; (2) is the only one that actually delivers the feature. What should not happen is a fourth round of someone assuming the level is live everywhere because the host half is.

Priority

Low. No correctness impact — it costs diagnostic reach at the moment a user is trying to raise it, which is annoying rather than dangerous. The documentation half of (3) is worth doing regardless of which shape is chosen, since the current docs do not scope the claim.

[ChaoWao]

Resolving as (3) from the list above — it is a design decision, not a defect, and the actual gap was that nothing said so. #2117 records it.

I filed this as a defect and that framing was wrong. There is no workflow where someone raises verbosity mid-run to chase a device-side problem; recreating the Worker with the level they want is the answer, it costs nothing a debugging session would notice, and device log volume is exactly where an accidental DEBUG is most expensive — codestyle.md rule 7 forbids AICPU hot-path logging because device_log writes serialize on the single AICPU op and can trip the op-execute timeout. A threshold outside code can raise into that is a liability rather than a feature.

One correction to option (2) as I wrote it. I said it "needs someone who knows whether a host write to that region is coherent against an AICPU read without a launch" — that was too pessimistic, and I went and looked. The threshold could ride KernelArgs, which the host builds per run and simpler_aicpu_exec already reads (k_args->regs, k_args->dump_data_base, and the rest are delivered exactly this way), and onboard set_log_level is a single bool store to a device global, so a per-run refresh would be nearly free and needs no new channel and no coherency question. It would give next-run granularity rather than immediate.

So (2) is cheaper than I claimed, and it still is not worth doing: there is no use case asking for it. That is recorded in #2117's doc so the next person does not re-derive it and mistake "cheap" for "wanted".

Also worth noting for anyone reading the code: the same one-shot site exists in both arch trees — src/a2a3/platform/onboard/aicpu/kernel.cpp and src/a5/platform/onboard/aicpu/kernel.cpp. This issue cited only the a2a3 one. #2117 annotates both.