[sukesh-amd]

Summary

Megatron and Primus no longer write the Hugging Face token value into tokenizer-download or job-launch commands. Each node reads paths.hf_token_file inside the container. If that file is missing, empty, or not mounted, Megatron fails the stage and later stages are skipped. Primus exports an empty HF_TOKEN and continues.

Primus also accepts train_params.primus_cli_overrides as a string or a JSON object. For Primus 405B, decoder first/last pipeline layer flags are derived from train_params.pipeline_parallelism. An 8-node run with pipeline parallelism 4 gets 31/31. User-provided decoder split flags take precedence, and the test fails when no integer split exists.

Motivation

The launch command and the per-node wrapper script were embedding HF_TOKEN=<value>, so orchestrator logs and scripts on disk contained the token. This follows the same in-container read used for vLLM, SGLang, and JAX MaxText.

Primus 405B has 126 decoder layers. Those layers are split across pipeline stages (pipeline_model_parallel_size), not across nodes. When data parallelism makes the node count larger than pipeline_parallelism, a node-count split is wrong. With pipeline parallelism 4 the first and last stages need 31 layers each.

Technical Details

• megatron_lib.py and primus_lib.py read paths.hf_token_file on the node at run time. The token value is not interpolated into the command. Job-launch logs pass through redact_secrets.
• Megatron checks that the file is non-empty inside the container before tokenizer download and before launch. A missing mount fails the stage. Primus exports an empty HF_TOKEN and continues.
• The Megatron suite no longer reads the token file on the machine running cvs run. The hf_token fixture and the hf_token argument are removed from the single-node and distributed stage functions.
• train_params.primus_cli_overrides is appended to the Primus command. A JSON object is rendered as --key value pairs. The nine Megatron templates ship "primus_cli_overrides": {}.
• For model size 405, --decoder_first_pipeline_num_layers and --decoder_last_pipeline_num_layers are added only when 126 is not evenly divisible by pipeline_parallelism. Flags already set in primus_cli_overrides are left unchanged.
• How-to, configuration reference, and the [0.3.0] changelog describe both behaviors.

[github-actions]

Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

[solaiys]

No blocking issues. Combined 0.3.0 backport of the in-container HF token read (#676/#684) and Primus CLI overrides / 405B layer-split (#687).

1. Megatron _hf_token_export_line shlex.quotes the path and reads it on the node (tr -d '[:space:]' < file); a missing/empty file exit 1s. _require_hf_token_file probes test -s via orch.exec(..., detailed=True) before tokenizer download and launch. On this branch that returns {host: {output, exit_code}}. Primus still exports empty HF_TOKEN and continues, as documented. Launch logs go through redact_secrets. hf_token is dropped from the job ctors and the suite fixture.

2. primus_cli_overrides (string or dict) is appended to primus-cli direct. 405B auto-split uses pipeline_stages=self.pipeline_parallelism, so PP=4 is 31/31 even on 8 nodes. User-provided decoder flags win; no integer split fail_tests and raises. How-to, reference, and CHANGELOG match both behaviors.