diff --git a/CHANGELOG.md b/CHANGELOG.md index c92e04fba..cee700204 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Single-node 8-GPU guide configs `mi300x_rccl_single_node_config.json` and `mi350x_rccl_single_node_config.json`. - Platform `test_check_be_nic_link_speed` checks backend NIC link speed against `nic_link_speed` (default `400000` Mb/s). Existing configs fail on NICs running at another speed, or on a node with fewer backend NICs unless `nic_link_interfaces` is set. - Platform host config `fw_dict` includes `PLDM_BUNDLE`. Set it to the version reported by `amd-smi firmware` before the GPU firmware check; the check looks up every firmware ID `amd-smi` reports. +- Primus Megatron configs accept `train_params.primus_cli_overrides` as either a raw CLI string or a JSON object rendered as `--key value` pairs. Primus 405B auto-splits decoder first/last pipeline layers from `train_params.pipeline_parallelism` when needed. ### Changed @@ -53,6 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Aorta script mode (the `aorta_single` default) now applies `training_overrides`; previously runs silently used the YAML values. The applied config is saved as `effective_config.yaml`. - Kernel-log health checks no longer echo each node's full dmesg or journal into the run log and HTML report. Each scan logs one summary line, and detected errors are still reported. +- Megatron and Primus no longer write the Hugging Face token into training, tokenizer-download, or job-launch commands. Each node reads `paths.hf_token_file` inside the container when that command runs. If that file is missing, empty, or not mounted, Megatron fails the test and the remaining training stages are skipped. Primus exports an empty `HF_TOKEN` and the run continues. - ibperf: - Runs on nodes with fewer than 8 GPUs instead of failing with `KeyError: 'card4'`. - `verify_bw` now enforces `expected_results`; previously tests passed regardless of measured bandwidth. Configs with `verify_bw` set to `"True"` may now fail; re-baseline thresholds for your cluster. diff --git a/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.1-8b_single.json b/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.1-8b_single.json index 66abe0d1a..ff1d5a333 100644 --- a/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.1-8b_single.json +++ b/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.1-8b_single.json @@ -50,7 +50,9 @@ "micro_batch_size": "4", "global_batch_size": "128", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", "smoke": { diff --git a/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.3-70b_single.json b/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.3-70b_single.json index 5bf8c8d7f..c1a8c7c1d 100644 --- a/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.3-70b_single.json +++ b/cvs/input/config_file/training/megatron/mi355x_megatron_llama-3.3-70b_single.json @@ -50,7 +50,9 @@ "micro_batch_size": "3", "global_batch_size": "96", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", "smoke": { diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_distributed.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_distributed.json index b1c798491..7777f12d6 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_distributed.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_distributed.json @@ -57,7 +57,9 @@ "micro_batch_size": "4", "global_batch_size": "128", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "verify_network_errors": "True", "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_single.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_single.json index fe207ac4e..92b6e107d 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_single.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_deepseek-v2-lite_single.json @@ -52,7 +52,9 @@ "micro_batch_size": "4", "global_batch_size": "128", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", "smoke": { diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-405b_distributed.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-405b_distributed.json index c8caf21ec..a4c9c10b5 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-405b_distributed.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-405b_distributed.json @@ -57,7 +57,9 @@ "micro_batch_size": "1", "global_batch_size": "64", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "verify_network_errors": "True", "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_distributed.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_distributed.json index d30f69aa5..712f8f691 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_distributed.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_distributed.json @@ -57,7 +57,9 @@ "micro_batch_size": "4", "global_batch_size": "128", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "verify_network_errors": "True", "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_single.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_single.json index 8accbda0e..e27073d3f 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_single.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.1-8b_single.json @@ -52,7 +52,9 @@ "micro_batch_size": "4", "global_batch_size": "128", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", "smoke": { diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_distributed.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_distributed.json index 5be7af277..b5297b7ec 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_distributed.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_distributed.json @@ -57,7 +57,9 @@ "micro_batch_size": "1", "global_batch_size": "64", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "verify_network_errors": "True", "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", diff --git a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_single.json b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_single.json index 89faf5dd5..81763d6c4 100644 --- a/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_single.json +++ b/cvs/input/config_file/training/megatron/mi3xx_megatron_llama-3.3-70b_single.json @@ -52,7 +52,9 @@ "micro_batch_size": "3", "global_batch_size": "96", "precision": "BF16", - "training_iterations": "" + "training_iterations": "", + "_primus_cli_overrides_comment": "Optional Primus-only extra CLI args. Set as a raw string (for example \"--use_backend triton\") or override per sweep combination with the same key.", + "primus_cli_overrides": {} }, "_smoke_comment": "test_smoke loads the model and runs a small fixed cell without metric gating. enabled=false skips it.", "smoke": { diff --git a/cvs/lib/training/megatron/megatron_lib.py b/cvs/lib/training/megatron/megatron_lib.py index 24c430d41..9009e29d6 100644 --- a/cvs/lib/training/megatron/megatron_lib.py +++ b/cvs/lib/training/megatron/megatron_lib.py @@ -11,6 +11,7 @@ import time from cvs.lib import globals +from cvs.lib.env_lib import redact_secrets from cvs.lib.utils_lib import * from cvs.lib.verify_lib import * from cvs.lib import linux_utils @@ -34,6 +35,60 @@ def _default_nic_type(gpu_name): return 'thor2' +def _hf_token_export_line(token_file): + """Shell line that exports ``HF_TOKEN`` by reading ``token_file`` on the + node the command runs on (as SGLang/vLLM/jaxmaxtext do). + + Embedding HF_TOKEN's value directly in a shell command would print it, + because the orchestrator logs the full command. A missing or empty file + fails the shell command instead of continuing with an empty token. + """ + path = str(token_file or '').strip() + if not path: + return "echo 'paths.hf_token_file is not configured' >&2; exit 1; " + quoted = shlex.quote(path) + return ( + f"if [ -s {quoted} ]; then export HF_TOKEN=\"$(tr -d '[:space:]' < {quoted})\"; " + f"else echo 'HF token file is missing or not mounted inside the container' >&2; exit 1; fi; " + ) + + +def _require_hf_token_file(orch, token_file): + """Fail the test when token_file is missing or empty inside the container. + + The training command reads the file on the node, so a path that is not + mounted into the container must stop the test before the job is launched. + """ + path = str(token_file or '').strip() + if not path: + msg = ( + 'paths.hf_token_file is not configured. Megatron reads the token inside the ' + 'container on each node and fails the test when the file is missing.' + ) + fail_test(msg) + raise FileNotFoundError(msg) + quoted = shlex.quote(path) + result = orch.exec(f'test -s {quoted}', detailed=True, timeout=30) or {} + missing = [] + seen = set() + for host, info in result.items(): + seen.add(str(host)) + code = info.get('exit_code') if isinstance(info, dict) else None + if code != 0: + missing.append(str(host)) + for host in getattr(orch, 'hosts', None) or []: + if str(host) not in seen: + missing.append(str(host)) + if not result or missing: + where = ', '.join(missing) if missing else 'no node returned a result' + msg = ( + f'HF token file {path} is missing, empty, or not mounted inside the container on {where}. ' + 'Megatron fails the test here. Mount paths.hf_token_file into the container on each node.' + ) + fail_test(msg) + raise FileNotFoundError(msg) + + # Ordered fallback chains for parsing Megatron-LM training output. # Each chain is tried in order; first non-empty match wins. Seeded with # [new, old] so newer Megatron output (e.g. `throughput per GPU @@ -225,7 +280,6 @@ def __init__( self, orch, variant_config, - hf_token, micro_batch_size, global_batch_size, precision=None, @@ -241,7 +295,6 @@ def __init__( Args: orch: Orchestrator handle for container and host command execution. variant_config: MegatronVariantConfig holding container, config, train_params, sweep. - hf_token: Hugging Face token passed to the job environment. batch_size: Global batch size for this sweep cell (overrides train_params). micro_batch_size: Micro batch size for this sweep cell (overrides train_params). precision: Optional precision override for this sweep cell. When None, falls @@ -253,7 +306,6 @@ def __init__( self.orch = orch self.model_name = variant_config.train_params["model_name"] - self.hf_token = hf_token self.tune_model_params = tune_model_params self.job_cmd = '' @@ -306,6 +358,8 @@ def __init__( self.megatron_root = tdict['megatron_root'] self.training_scripts = tdict['training_scripts'] + self.hf_token_file = tdict['hf_token_file'] + pdict = dict(variant_config.train_params) pdict.update(sweep_overrides or {}) pdict['micro_batch_size'] = micro_batch_size @@ -412,9 +466,9 @@ def download_tokenizer_model(self): local_dir = f'{self.data_cache_dir}/{self.model_name}' log.info('Downloading tokenizer.model for %s into %s', self.model_name, local_dir) + _require_hf_token_file(self.orch, self.hf_token_file) self.orch.exec( - f'export HF_TOKEN={shlex.quote(self.hf_token)}; ' - f'huggingface-cli download {self.tokenizer_model} ' + _hf_token_export_line(self.hf_token_file) + f'huggingface-cli download {self.tokenizer_model} ' f'--include "tokenizer.model" ' f'--local-dir {local_dir}' ) @@ -535,7 +589,7 @@ def build_training_job_cmd( cmd + f'cd {self.megatron_root}; export MOCK_DATA=1; ' + f'export IMAGE={self.container_image}; ' - + f'export HF_TOKEN="{self.hf_token}"; ' + + _hf_token_export_line(self.hf_token_file) + f'export DATA_CACHE_PATH={self.data_cache_dir}; ' + f'export TOKENIZER_MODEL={self.local_tokenizer_path if self.local_tokenizer_path else self.tokenizer_model}; ' + f'export LD_LIBRARY_PATH=/usr/local/lib/:{self.rocm_path}/lib:$LD_LIBRARY_PATH; ' @@ -605,8 +659,9 @@ def start_training_job(self, timeout=500): by build_training_job_cmd() prior to invocation. """ log.info('start training job') - log.info('%s', self.job_cmd_list) - log.info("%s", self.job_cmd) + _require_hf_token_file(self.orch, self.hf_token_file) + log.info('%s', [redact_secrets(c) for c in self.job_cmd_list]) + log.info("%s", redact_secrets(self.job_cmd)) n = len(self.orch.hosts) # Create per-node log dirs inside container on all hosts in parallel diff --git a/cvs/lib/training/megatron/primus_lib.py b/cvs/lib/training/megatron/primus_lib.py index ab891982a..cd56a4481 100644 --- a/cvs/lib/training/megatron/primus_lib.py +++ b/cvs/lib/training/megatron/primus_lib.py @@ -25,11 +25,28 @@ from cvs.lib import globals from cvs.lib import linux_utils +from cvs.lib.env_lib import redact_secrets from cvs.lib.utils_lib import fail_test from cvs.lib.verify_lib import verify_dmesg_for_errors log = globals.log + +def _hf_token_export_line(token_file): + """Shell line that exports ``HF_TOKEN`` by reading ``token_file`` on the + node the command runs on (as SGLang/vLLM/jaxmaxtext do). + + Embedding HF_TOKEN's value directly in a shell command would print it, + because the orchestrator logs the full command. A missing or empty file + exports an empty HF_TOKEN and the run continues. + """ + path = str(token_file or '').strip() + if not path: + return 'export HF_TOKEN=; ' + quoted = shlex.quote(path) + return f"if [ -f {quoted} ]; then export HF_TOKEN=\"$(tr -d '[:space:]' < {quoted})\"; else export HF_TOKEN=; fi; " + + # --------------------------------------------------------------------------- # Primus-Megatron log patterns # --------------------------------------------------------------------------- @@ -105,6 +122,75 @@ def _parse_mean_from_iterations(log_text, pattern, skip_warmup=True): return str(sum(values) / len(values)) +def _format_primus_cli_overrides(overrides): + """Normalize primus_cli_overrides into a CLI arg string. + + Accepts a pre-formatted string (used as-is) or a dict rendered as + ``--flag value`` pairs in insertion order. + """ + if isinstance(overrides, dict): + return ' '.join(f'--{key} {value}' for key, value in overrides.items()) + return str(overrides or '') + + +def _decoder_layer_split_flags(total_layers, stages): + """Return first/last decoder-layer flags with an integer middle split. + + For pipeline stages > 2, pick equal first/last layer counts that are as + close as possible to uniform split while keeping + (total_layers - first - last) divisible across middle stages. + """ + n = int(stages) + total = int(total_layers) + if n < 2 or total <= 0: + return "" + if total % n == 0: + return "" + if n == 2: + first = total // 2 + last = total - first + return f' --decoder_first_pipeline_num_layers {first} --decoder_last_pipeline_num_layers {last}' + + ideal = total / n + candidates = [] + for first_last in range(1, total): + middle_total = total - (2 * first_last) + if middle_total <= 0: + continue + if middle_total % (n - 2) != 0: + continue + middle_each = middle_total // (n - 2) + if middle_each <= 0: + continue + candidates.append((abs(first_last - ideal), first_last)) + + if not candidates: + return None + + _, chosen = min(candidates) + return f' --decoder_first_pipeline_num_layers {chosen} --decoder_last_pipeline_num_layers {chosen}' + + +def _auto_primus_405_layer_split_flags(model_size, pipeline_stages, primus_cli_overrides): + """Auto-add 405B decoder first/last layer flags unless user set them.""" + size_text = str(model_size or "").strip() + if size_text != "405": + return "" + overrides = str(primus_cli_overrides or "") + if "--decoder_first_pipeline_num_layers" in overrides or "--decoder_last_pipeline_num_layers" in overrides: + return "" + split_flags = _decoder_layer_split_flags(total_layers=126, stages=pipeline_stages) + if split_flags is None: + stages_text = str(pipeline_stages) + raise ValueError( + "Unable to derive integer 405B decoder first/last pipeline layer split for " + f"pipeline_parallelism={stages_text}. " + "Set --decoder_first_pipeline_num_layers and " + "--decoder_last_pipeline_num_layers in train_params.primus_cli_overrides." + ) + return split_flags + + def _parse_step_losses(log_text): """Return {step: lm_loss} from a Primus training log. @@ -133,7 +219,6 @@ def __init__( self, orch, variant_config, - hf_token, micro_batch_size, global_batch_size, precision=None, @@ -145,7 +230,6 @@ def __init__( ): self.orch = orch self.model_name = variant_config.train_params['model_name'] - self.hf_token = hf_token self.tune_model_params = tune_model_params self.distributed_training = distributed_training @@ -195,6 +279,8 @@ def __init__( self.primus_cli = tdict['primus_cli'] self.gpu_arch = variant_config.gpu_name + self.hf_token_file = tdict['hf_token_file'] + pdict = dict(variant_config.train_params) pdict.update(sweep_overrides or {}) pdict['micro_batch_size'] = micro_batch_size @@ -209,14 +295,18 @@ def __init__( pdict.setdefault('training_iterations', 10) pdict.setdefault('tensor_parallelism', '1') pdict.setdefault('pipeline_parallelism', '1') + pdict.setdefault('primus_cli_overrides', '') + pdict.setdefault('model_size', '70') self.tokenizer_model = pdict['tokenizer_model'] + self.model_size = pdict['model_size'] self.precision = pdict['precision'] self.micro_batch_size = pdict['micro_batch_size'] self.global_batch_size = pdict['global_batch_size'] self.iterations = int(pdict['training_iterations']) self.tensor_parallelism = pdict['tensor_parallelism'] self.pipeline_parallelism = pdict['pipeline_parallelism'] + self.primus_cli_overrides = _format_primus_cli_overrides(pdict['primus_cli_overrides']) raw_label = run_label or f"{self.model_name}_mbs{micro_batch_size}_gbs{global_batch_size}_{self.precision}" self.run_label = re.sub(r'[^A-Za-z0-9._-]', '_', str(raw_label)) @@ -368,7 +458,7 @@ def _build_primus_cmd(self): exp_path = self._exp_config_path() log.info('Primus EXP config path: %s', exp_path) - env_exports = f'export HF_TOKEN="{self.hf_token}"; export LOG_DIR={self.log_dir}; ' + env_exports = _hf_token_export_line(self.hf_token_file) + f'export LOG_DIR={self.log_dir}; ' if re.search(r'MI3(00|25)X', self.gpu_arch, re.I): env_exports += 'export PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32=1; export NVTE_CK_IS_V3_ATOMIC_FP32=1; ' @@ -397,6 +487,17 @@ def _build_primus_cmd(self): batch_args += f' --load {self.checkpoint_dir}/checkpoints' if self.distributed_training: batch_args += ' --auto_detect_ckpt_format' + try: + batch_args += _auto_primus_405_layer_split_flags( + model_size=self.model_size, + pipeline_stages=self.pipeline_parallelism, + primus_cli_overrides=self.primus_cli_overrides, + ) + except ValueError as err: + fail_test(str(err)) + raise + if self.primus_cli_overrides: + batch_args += f' {self.primus_cli_overrides}' if self.distributed_training: for i in range(len(self.orch.hosts)): @@ -428,8 +529,8 @@ def _build_primus_cmd(self): def _start_primus_job(self): """Launch the Primus training job (distributed or single-node).""" log.info('start primus training job') - log.info('%s', self.job_cmd_list) - log.info('%s', self.job_cmd) + log.info('%s', [redact_secrets(c) for c in self.job_cmd_list]) + log.info('%s', redact_secrets(self.job_cmd)) exp_full_path = f'{self.primus_root}/{self._exp_config_path()}' out_dict = self.orch.exec(f'test -f {exp_full_path} && echo EXISTS || echo MISSING') diff --git a/cvs/tests/training/megatron/conftest.py b/cvs/tests/training/megatron/conftest.py index ee1d4d1c9..b78ac158b 100644 --- a/cvs/tests/training/megatron/conftest.py +++ b/cvs/tests/training/megatron/conftest.py @@ -97,15 +97,6 @@ def variant_config(pytestconfig, cluster_dict): return load_training_variant(config_file, cluster_dict) -@pytest.fixture(scope="module") -def hf_token(variant_config): - path = variant_config.paths.hf_token_file - if not os.path.isfile(path): - pytest.skip(f"hf_token file missing: {path}") - with open(path) as fp: - return fp.read().strip() - - class _Lifecycle: """Cross-test state for the lifecycle-as-tests model. diff --git a/cvs/tests/training/megatron/megatron_distributed.py b/cvs/tests/training/megatron/megatron_distributed.py index 6cf17f67a..b23773a42 100644 --- a/cvs/tests/training/megatron/megatron_distributed.py +++ b/cvs/tests/training/megatron/megatron_distributed.py @@ -104,7 +104,7 @@ def test_backend_nic_speed(orch, lifecycle, request): log.info("backend NIC link speed check PASSED") -def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): +def test_download_tokenizer(orch, variant_config, lifecycle, request): """Stage 2: download the tokenizer model once if the model family requires it.""" if lifecycle.failed: pytest.skip("a prior lifecycle stage failed") @@ -112,7 +112,6 @@ def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size="1", global_batch_size="1", precision="BF16", @@ -141,7 +140,7 @@ def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): log.info("test_download_tokenizer: tokenizer ready at %s", lifecycle.tokenizer_path) -def test_smoke(orch, variant_config, hf_token, lifecycle, request): +def test_smoke(orch, variant_config, lifecycle, request): """Stage 3: smoke-test — model loads and runs smoke.iters steps without error. Knobs come from the top-level ``smoke`` block. Empty ``global_batch_size`` @@ -163,7 +162,6 @@ def test_smoke(orch, variant_config, hf_token, lifecycle, request): mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=mbs, global_batch_size=gbs, precision=precision, @@ -192,7 +190,7 @@ def test_smoke(orch, variant_config, hf_token, lifecycle, request): log.info("smoke PASSED | iters=%s mbs=%s gbs=%s precision=%s", iters, mbs, gbs, precision) -def test_checkpoint(orch, variant_config, hf_token, lifecycle, request): +def test_checkpoint(orch, variant_config, lifecycle, request): """Stage 4: checkpoint save + resume — verify step counter and loss continuity. Two phases: @@ -237,7 +235,6 @@ def _run(run_label, iters, checkpoint_dir=None, save_interval=None, load_checkpo job = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=mbs, global_batch_size=gbs, precision=precision, @@ -401,7 +398,7 @@ def _run(run_label, iters, checkpoint_dir=None, save_interval=None, load_checkpo log.info("checkpoint dir retained for debugging: %s", ckpt_dir) -def test_training(orch, variant_config, hf_token, sweep_name, train_res_dict, lifecycle, request): +def test_training(orch, variant_config, sweep_name, train_res_dict, lifecycle, request): """Stage 5 (parametrized): run one sweep combo inside the shared container. stop_training_processes() runs in a finally block after every combo so GPU @@ -416,7 +413,6 @@ def test_training(orch, variant_config, hf_token, sweep_name, train_res_dict, li mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=combo.micro_batch_size, global_batch_size=combo.global_batch_size, precision=combo.precision, diff --git a/cvs/tests/training/megatron/megatron_single.py b/cvs/tests/training/megatron/megatron_single.py index cc318a3b2..c61d6e4e6 100644 --- a/cvs/tests/training/megatron/megatron_single.py +++ b/cvs/tests/training/megatron/megatron_single.py @@ -104,7 +104,7 @@ def test_backend_nic_speed(orch, lifecycle, request): log.info("backend NIC link speed check PASSED") -def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): +def test_download_tokenizer(orch, variant_config, lifecycle, request): """Stage 2: download the tokenizer model once if the model family requires it.""" if lifecycle.failed: pytest.skip("a prior lifecycle stage failed") @@ -112,7 +112,6 @@ def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size="1", global_batch_size="1", precision="BF16", @@ -141,7 +140,7 @@ def test_download_tokenizer(orch, variant_config, hf_token, lifecycle, request): log.info("test_download_tokenizer: tokenizer ready at %s", lifecycle.tokenizer_path) -def test_smoke(orch, variant_config, hf_token, lifecycle, request): +def test_smoke(orch, variant_config, lifecycle, request): """Stage 3: smoke-test — model loads and runs smoke.iters steps without error. Knobs come from the top-level ``smoke`` block (defaults: 10 iters, MBS 1, @@ -163,7 +162,6 @@ def test_smoke(orch, variant_config, hf_token, lifecycle, request): mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=mbs, global_batch_size=gbs, precision=precision, @@ -192,7 +190,7 @@ def test_smoke(orch, variant_config, hf_token, lifecycle, request): log.info("smoke PASSED | iters=%s mbs=%s gbs=%s precision=%s", iters, mbs, gbs, precision) -def test_checkpoint(orch, variant_config, hf_token, lifecycle, request): +def test_checkpoint(orch, variant_config, lifecycle, request): """Stage 4: checkpoint save + resume — verify step counter and loss continuity. Two phases: @@ -235,7 +233,6 @@ def _run(run_label, iters, checkpoint_dir=None, save_interval=None, load_checkpo job = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=mbs, global_batch_size=gbs, precision=precision, @@ -396,7 +393,7 @@ def _run(run_label, iters, checkpoint_dir=None, save_interval=None, load_checkpo log.info("checkpoint dir retained for debugging: %s", ckpt_dir) -def test_training(orch, variant_config, hf_token, sweep_name, train_res_dict, lifecycle, request): +def test_training(orch, variant_config, sweep_name, train_res_dict, lifecycle, request): """Stage 5 (parametrized): run one sweep combo inside the shared container. stop_training_processes() runs in a finally block after every combo so GPU @@ -411,7 +408,6 @@ def test_training(orch, variant_config, hf_token, sweep_name, train_res_dict, li mt_obj = _make_training_job( orch, variant_config, - hf_token=hf_token, micro_batch_size=combo.micro_batch_size, global_batch_size=combo.global_batch_size, precision=combo.precision, diff --git a/docs/how-to/test-suites/training/megatron.rst b/docs/how-to/test-suites/training/megatron.rst index b10c053d9..6058b284a 100644 --- a/docs/how-to/test-suites/training/megatron.rst +++ b/docs/how-to/test-suites/training/megatron.rst @@ -27,9 +27,16 @@ In addition to the :ref:`common training prerequisites `` with cluster-specific values. For MI300X/MI325X shared templates, set ``gpu_name`` to ``MI300X`` or ``MI325X`` and ``threshold_json`` to the matching ``mi300x_*`` or ``mi325x_*`` threshold file. Also set ``container.image`` and the ``container.env`` NIC fields (templates ship example interface/HCA/GID/debug strings that still contain ```` — keep or edit the example and remove the placeholder). Packaged ``sweep.combinations`` cells already set ``training_iterations`` to ``"20"`` (that overlay wins over ``train_params.training_iterations``). Change the overlay per combo if needed. Do not add ``NNODES``; the suite sets it from the cluster host count at ``docker run``. Distributed configs also need ``MASTER_ADDR``, ``NCCL_IB_HCA``, and a shared ``paths.data_cache_dir``. +3. Replace every ```` with cluster-specific values. For MI300X/MI325X shared templates, set ``gpu_name`` to ``MI300X`` or ``MI325X`` and ``threshold_json`` to the matching ``mi300x_*`` or ``mi325x_*`` threshold file. Also set ``container.image`` and the ``container.env`` NIC fields (templates ship example interface/HCA/GID/debug strings that still contain ```` — keep or edit the example and remove the placeholder). Packaged ``sweep.combinations`` cells already set ``training_iterations`` to ``"20"`` (that overlay wins over ``train_params.training_iterations``). Change the overlay per combo if needed. ``train_params.primus_cli_overrides`` is optional and only applies to Primus runs (string or key/value map). Do not add ``NNODES``; the suite sets it from the cluster host count at ``docker run``. Distributed configs also need ``MASTER_ADDR``, ``NCCL_IB_HCA``, and a shared ``paths.data_cache_dir``. 4. Change any other parameters relevant to your testing requirements. The same folder also has DeepSeek V2 Lite (single and distributed; Megatron-LM or Primus) and Llama 3.1 405B (distributed, **Primus only**). See `Config and threshold files`_ for the full inventory. @@ -103,6 +110,16 @@ The same ordered test stages run for both backends. ``_make_training_job`` in `` Logs for both backends use per-node files: ``/{megatron-logs|primus-logs}//out-node/training.log``. On disk ```` is the sweep key with ``=`` and ``,`` replaced by ``_`` (for example ``MBS_4_GBS_128_PRECISION_FP8``). +For Primus 405B distributed runs, CVS auto-adds +``--decoder_first_pipeline_num_layers`` and +``--decoder_last_pipeline_num_layers`` only when 126 layers are not evenly +divisible by ``train_params.pipeline_parallelism`` (pipeline stage count). +Example: with ``pipeline_parallelism=4``, it appends ``31`` and ``31`` so +middle stages get ``32`` and ``32``. ``pipeline_parallelism`` may differ from +cluster node count when data parallelism is greater than 1. If you set either +decoder split flag in ``train_params.primus_cli_overrides``, your value is +used as-is. + .. _megatron-run-tests: Run tests @@ -254,7 +271,9 @@ Tests run in this pinned order. ``[combo]`` = one row per enabled sweep combo. - once - Tear the container down -A training failure is isolated to that combo's ``test_training`` row; other combos still run. When a combo's training does not complete, its downstream ``test_metric`` and ``test_loss_curve`` rows are skipped. If an early lifecycle stage fails, all subsequent stages are skipped using ``lifecycle.failed``. +A training failure is isolated to that combo's ``test_training`` row; other combos still run. When a combo's training does not complete, its downstream ``test_metric`` and ``test_loss_curve`` rows are skipped. If an early lifecycle stage fails, all subsequent stages are skipped using ``lifecycle.failed``. ``test_teardown`` still runs. + +On Megatron, a token file that is missing, empty, or not mounted inside the container is one of those early failures. DeepSeek and Mixtral fail in ``test_download_tokenizer``; other models fail in ``test_smoke``. If ``smoke.enabled`` is ``false``, the same check runs in ``test_training`` instead: that combo fails without setting ``lifecycle.failed``, its metric and loss-curve rows are skipped, and other combos still run. Primus exports an empty ``HF_TOKEN`` and does not fail the suite when the file is not visible. On a training failure, lingering GPU processes are killed (``stop_training_processes``) so the next combo does not launch on top of them. diff --git a/docs/reference/configuration-files/training/megatron.rst b/docs/reference/configuration-files/training/megatron.rst index d0f7420c2..9476f7776 100644 --- a/docs/reference/configuration-files/training/megatron.rst +++ b/docs/reference/configuration-files/training/megatron.rst @@ -82,7 +82,8 @@ The following fields must be customized to match your cluster and hardware befor * ``container.image`` — Megatron-LM or Primus ROCm image on all nodes. * ``train_params.training_iterations`` — default training steps (for example ``"30"``). Each ``sweep.combinations`` body sets ``training_iterations`` (packaged value ``"20"``); that overlay wins for that cell. * ``train_params.micro_batch_size`` / ``global_batch_size`` / ``precision`` — required when ``sweep`` is omitted. Packaged files already set them; with a declared sweep the combination key supplies those values for each cell. -* ``paths.hf_token_file`` — Hugging Face token path on the nodes. +* ``train_params.primus_cli_overrides`` — optional Primus-only extra ``primus-cli`` args. Set as a raw string (for example ``"--use_backend triton"``) or as a JSON object rendered to ``--key value`` pairs. +* ``paths.hf_token_file`` — Hugging Face token file, read inside the container on each node when the tokenizer download or training command runs. CVS does not read it on the machine running ``cvs run``. The path must be mounted into the container. If the file is not visible inside the container, Megatron fails that stage and sets ``lifecycle.failed``, so later stages are skipped. Primus exports an empty ``HF_TOKEN`` and the run continues. * ``container.env.NCCL_SOCKET_IFNAME`` / ``GLOO_SOCKET_IFNAME`` / ``NCCL_IB_GID_INDEX`` / ``NCCL_DEBUG`` — templates include example values plus ```` (for example ``enp193s0f1np1 ``, ``3 ``, ``ERROR ``). Remove ```` and keep or edit the example. Required on single-node and distributed. * ``sweep.runs`` — combination keys to execute (same ``MBS=…,GBS=…,PRECISION=…`` strings as in ``sweep.combinations``). Must be non-empty when ``combinations`` is present. * Do not set ``NNODES`` in JSON. @@ -419,6 +420,19 @@ Llama 3.1 405B Available as ``mi3xx_megatron_llama-3.1-405b_distributed.json`` (MI300X/MI325X; distributed only). Unlike the other models, 405B runs on **Primus only**: ``container.image`` must contain ``primus``. Megatron-LM does not support 405B. +For Primus 405B, when total decoder layers (126) are not evenly divisible by +``train_params.pipeline_parallelism`` (pipeline stage count), CVS +automatically appends +``--decoder_first_pipeline_num_layers`` and +``--decoder_last_pipeline_num_layers`` to keep middle stages evenly split. +Example: with ``pipeline_parallelism=4``, CVS appends ``31`` and ``31`` so the +middle stages get ``32`` and ``32``. ``pipeline_parallelism`` may differ from +node count when data parallelism is greater than 1. If layers are already +evenly divisible, those flags are not added. If no integer first/last split is +possible for the configured pipeline stage count, the test fails and asks you +to set both flags explicitly in ``primus_cli_overrides``. User-provided decoder +split flags in ``primus_cli_overrides`` take precedence. + .. dropdown:: ``mi3xx_megatron_llama-3.1-405b_distributed.json`` (representative) .. code:: json @@ -504,6 +518,11 @@ Available as ``mi3xx_megatron_llama-3.1-405b_distributed.json`` (MI300X/MI325X; * - ``training_iterations`` - ```` - Training steps. + * - ``primus_cli_overrides`` + - ``{}`` + - Optional Primus-only extra CLI args appended to ``primus-cli direct``. + Accepts either a raw string (for example ``"--use_backend triton"``) + or an object rendered as ``--key value`` pairs. Common parameters @@ -525,7 +544,7 @@ Host paths used by the job. They must be volume-mounted into the container (typi - Description * - ``hf_token_file`` - ``/home/{user-id}/.hf_token`` - - Path to a Hugging Face token file for gated models and datasets. + - Path to a Hugging Face token file for gated models and datasets, as seen inside the container. CVS does not read the file on the machine running ``cvs run``. Each node reads it when the tokenizer download or training command runs. The packaged ``/home/{user-id}/.hf_token`` path works when that home directory is bind-mounted. If the file is not visible inside the container, Megatron fails that stage and sets ``lifecycle.failed``, so later stages are skipped. Primus exports an empty ``HF_TOKEN`` and the run continues. * - ``log_dir`` - ``/home/{user-id}/LOGS/megatron`` - Host path where per-node training logs are written. Megatron-LM writes ``/megatron-logs//out-node/training.log``; Primus writes ``/primus-logs//out-node/training.log``. On disk ```` is the sanitized combination key (see `Backends`_).