From c8df05c57443bfe1277de53241aec84eec390101 Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Tue, 20 Oct 2026 18:32:39 +0000 Subject: [PATCH 1/3] Give the GPU, regression and example jobs more time before their job cap kills them A job cap close to the typical runtime turns every slow runner into a failed required check that only a manual re-run clears. Over 2026-10-06..10-20, the caps below killed these PR jobs (killed = cancelled within 1.5 min after the cap; minutes): lane old cap median p95 killed / finished new cap gpu 60 41.6 52 9 / 609 75 gpu_trtllm 15 6.5 10 34 / 715 30 regression 15 10.0 14 41 / 630 30 example llm_qat 30 19.3 25 9 / 615 45 example megatron (PR) 30 19.6 25 26 / 589 45 The kills follow the load on the runner fleet: all 34 gpu_trtllm kills and 37 of the 41 regression kills fall on 10-14..10-17, the days with the most PR pushes. For gpu_trtllm, 32 of the 34 hit during container initialization or after one that took more than 12 minutes: pulling the image takes 4 min at the median, while the test step takes 2 min. A slow job is not a hung one, and the per-test timeouts still cut a hung test long before any of these caps. The new caps are 1.8x to 4.6x the median. The nightly megatron example lane gets 90 min: its manually dispatched runs took up to 69 min against a cap of 75. The example torch lanes share one cap, so llm_qat's 45 min applies to the whole matrix. The unit jobs keep their 20 min, which CONTRIBUTING.md documents as the limit that keeps tests/unit fast. Signed-off-by: Shengliang Xu --- .github/workflows/example_tests.yml | 6 +++--- .github/workflows/gpu_tests.yml | 4 ++-- .github/workflows/regression_tests.yml | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/example_tests.yml b/.github/workflows/example_tests.yml index 67b98cbd230..e646af86ac6 100644 --- a/.github/workflows/example_tests.yml +++ b/.github/workflows/example_tests.yml @@ -133,7 +133,7 @@ jobs: with: docker_image: "nvcr.io/nvidia/pytorch:26.09-py3" example: ${{ matrix.example }} - timeout_minutes: 30 + timeout_minutes: 45 pip_install_extras: "[hf,dev-test]" runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }} allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }} @@ -191,8 +191,8 @@ jobs: with: docker_image: "nvcr.io/nvidia/nemo:26.08" example: megatron_bridge - # PR runs single-GPU (~11 min); the nightly runs every test multi-GPU (~58 min). - timeout_minutes: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 30 || 75 }} + # PR runs single-GPU (~20 min with the setup); the nightly runs every test multi-GPU (~45 min). + timeout_minutes: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 45 || 90 }} pip_install_extras: "[hf,puzzletron,dev-test]" runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }} allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), ',megatron_bridge,') }} diff --git a/.github/workflows/gpu_tests.yml b/.github/workflows/gpu_tests.yml index b73f85affeb..8a7b3cf4581 100644 --- a/.github/workflows/gpu_tests.yml +++ b/.github/workflows/gpu_tests.yml @@ -44,13 +44,13 @@ jobs: matrix: include: - example: gpu - timeout: 60 + timeout: 75 container_image: nvcr.io/nvidia/pytorch:26.09-py3 - example: gpu_megatron timeout: 75 container_image: nvcr.io/nvidia/nemo:26.08 - example: gpu_trtllm - timeout: 15 + timeout: 30 container_image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 - example: gpu_vllm timeout: 30 diff --git a/.github/workflows/regression_tests.yml b/.github/workflows/regression_tests.yml index 9c1154a2b65..ea174f86557 100644 --- a/.github/workflows/regression_tests.yml +++ b/.github/workflows/regression_tests.yml @@ -41,7 +41,7 @@ jobs: needs: [pr-gate] if: needs.pr-gate.outputs.any_changed == 'true' runs-on: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }} - timeout-minutes: 15 + timeout-minutes: 30 container: image: nvcr.io/nvidia/pytorch:26.09-py3 env: From 1f47ac0da50c5f9ef2e515d8dc5b7fc2fab7039f Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Tue, 20 Oct 2026 18:28:54 +0000 Subject: [PATCH 2/3] Raise the default per-test timeouts of the gpu, gpu_megatron, gpu_vllm and example groups Besides test_mamba_search_space and test_overfits_a_single_batch, which are fixed on their own, 21 tests hit their per-test timeout in the CI runs of 2026-10-06..10-20: 10 in gpu (120 s), 5 in gpu_megatron (120 s), 5 in the examples (300 s) and 1 in gpu_vllm (60 s). Five of them passed on a re-run. The "slowest durations" of recent passing runs show how little room the slowest tests have before a slow runner pushes them over (call time, longest of the last 4 runs of each lane): test_qwen3_eval_fp8 (examples) 291 s of 300 s, median 228 s test_llama_eagle3[1-False] (examples) 233 s of 300 s test_serving_measurement_matches_production_autotuning (vllm) 49 s of 60 s test_quant_lora_distributed_checkpoint[1] (gpu_megatron) 88 s of 120 s test_unified_hf_export_and_check_safetensors (gpu) 66 s of 120 s (timed out 4 times) The new defaults are at least 2x the slowest of these: examples 600 s, gpu 180 s, gpu_megatron 180 s, gpu_vllm 120 s. A hung test is still cut there rather than by its job cap, and MODELOPT_QA_TEST_TIMEOUT still overrides the examples default. Signed-off-by: Shengliang Xu --- tests/conftest.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/conftest.py b/tests/conftest.py index af77546a244..6046a3e8b2d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -52,11 +52,11 @@ def pytest_addoption(parser): # Every collectible test group must be listed here else collection errors occur # A test can override its cap by adding ``@pytest.mark.timeout(...)`` _DEFAULT_TIMEOUT = { - "examples": int(os.environ.get("MODELOPT_QA_TEST_TIMEOUT", 300)), - "gpu": 120, - "gpu_megatron": 120, + "examples": int(os.environ.get("MODELOPT_QA_TEST_TIMEOUT", 600)), + "gpu": 180, + "gpu_megatron": 180, "gpu_trtllm": 60, - "gpu_vllm": 60, + "gpu_vllm": 120, "regression": 180, "unit": 120 if platform.system() == "Windows" else 60, } From 9bf800b1ca8643d382540d5e06d045bac375a71c Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Tue, 20 Oct 2026 18:48:18 +0000 Subject: [PATCH 3/3] Give the gpu_megatron lane 90 minutes on the 2-GPU runner The 2-GPU runs of the lane (the nightly and the dispatched runs) take longer than the PR runs on the 1-GPU runner (about 42 min). Of the 17 runs of the standard lane on 2026-10-14..10-20 under the old 60 min cap, 8 were killed by it, including 4 of the 7 nightlies on main, and the 9 that finished took 45 to 58 min (median 51). With a median near 58 min over all 17, the 75 min cap from #2836 leaves 1.3x. The runs that did finish under the 75 min cap (10 dispatched runs on 10-20) took 42 to 60 min, but 10-14..10-17 pushed 8 of 17 runs past 60 min, so a loaded day can still reach 75. 90 min is 1.5x the typical 2-GPU run. PR runs finish in about 42 min and do not use it; a hung job is still cut earlier by the per-test timeout. Signed-off-by: Shengliang Xu --- .github/workflows/gpu_tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/gpu_tests.yml b/.github/workflows/gpu_tests.yml index 8a7b3cf4581..e4265ede836 100644 --- a/.github/workflows/gpu_tests.yml +++ b/.github/workflows/gpu_tests.yml @@ -47,7 +47,7 @@ jobs: timeout: 75 container_image: nvcr.io/nvidia/pytorch:26.09-py3 - example: gpu_megatron - timeout: 75 + timeout: 90 container_image: nvcr.io/nvidia/nemo:26.08 - example: gpu_trtllm timeout: 30