Skip to content

Keep evaluation coverage after GPU memory retries - #234

Merged
anle2024 merged 5 commits into
mainfrom
codex/preserve-evaluation-row-budgets
Nov 7, 2026
Merged

anle2024 merged 5 commits into
mainfrom
codex/preserve-evaluation-row-budgets

Conversation

@anle2024

@anle2024 anle2024 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

GPU memory retries reduce batch size. Fixed evaluation step limits then cover fewer rows. Refit tests also rebuild their data module, so retry settings must survive that rebuild. Partial outputs from failed evaluation attempts must be discarded.

This change preserves the configured evaluation row budget. For example:

Evaluation attempt Batch size Step limit Configured row budget
Before the memory error 64 1,000 64,000
Retry before this fix 32 1,000 32,000
Retry with this fix 32 2,000 64,000

What changes

  • Scale finite validation and test step limits as batches shrink. Forward the adjusted trial limits to the Trainer.
  • Keep test retry settings on both the Trainer and current data module. Refit-created modules inherit them. Restore the original settings after success, exhausted retries, or another error.
  • Clear partial validation and test outputs before each evaluation attempt, so a retry does not duplicate saved prediction rows.
  • Remove the 8,000-step API ceiling while preserving positive integer, nullable and default behavior. Regenerate the affected API clients.

Addresses both points in Vedaant's review.

Scope and limits

This preserves the configured row budget. Timestamp bucket boundaries can still change the exact rows evaluated. Prediction and explanation populations retain their existing behavior.

Testing Done

At 962fbb3b, 82 focused CPU tests pass. Coverage includes recreated refit data modules, finite and unlimited test limits, caller-owned loader options, all retry exits, public/internal plan conversion, and saved output rows after partial failures.

Before the fixes, the new cases reproduced retries restarting at batch 64 and duplicate output IDs [10, 10, 11] instead of [10, 11]. Focused Pyflakes and whitespace checks pass.

The tests simulate GPU memory errors. Real GPU recovery and full-scale metric behavior remain unqualified for this revision.

Change map

Maintained file-by-file change map

@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

Optimization plans now accept any positive integer for validation and test step limits. Automatic-training and test OOM retries reduce the batch size and scale applicable step limits. The test OOM handler restores the original test batch size, step limit, and trainer evaluation configuration when the call exits. Evaluation starts clear accumulated validation and test outputs.

Sequence Diagram(s)

sequenceDiagram
  participant AutoTrainer
  participant TrialConfig
  participant DiskgraphNestedTrainer
  AutoTrainer->>TrialConfig: Read current validation and test step limits
  AutoTrainer->>DiskgraphNestedTrainer: Start trial with batch size and step limits
  DiskgraphNestedTrainer-->>AutoTrainer: CUDA OOM
  AutoTrainer->>TrialConfig: Reduce batch size and scale set step limits
  AutoTrainer->>DiskgraphNestedTrainer: Retry trial with updated configuration
Loading
Merge Risk: Moderate · up to c94f1

After a GPU memory retry during testing, logged test metrics may include rows from the failed attempt. Reset the stage's metric collection when an evaluation starts before merging.

Security architecture risk: Low · up to c94f1

The change preserves evaluation coverage after memory retries without establishing a new security attack path. A narrow interruption case can leave temporary retry settings installed. Production workload limits and tenant isolation could not be verified.

Retained concerns

  • Low · reliability · inferred: If interruption or an exception exits retry preparation after retrying is set to true, the finally block skips restoration. The PR newly stores retry settings on the Trainer itself, so temporary configuration can survive the aborted call and be inherited by later refit initialization. The base already lacked interruption cleanup for the current data module’s batch size; the added effect is persistence in the configuration owner used to rebuild modules. No cross-tenant impact or attacker-triggered path is established.
Security review details

Security Blast Radius

  • inferred — The demonstrated resource effect concerns validation and test evaluation for the configured workload. The source establishes split-level execution limits, but does not establish the independently attackable tenant, service or environment scope.

Security Findings and Attack Paths

  • inferred — Removing the finite-input ceiling expands accepted values, but does not by itself establish a new resource-exhaustion capability because whole-split evaluation was already expressible with None. Production acceptance and enforcement remain unverified.

Trust Boundaries and Controls

  • observed — The changed boundary is public configuration validation feeding core evaluation, not an observed identity or privilege transition. Positive-value validation remains; the former finite upper bound does not.

Resilience and Maintainability Implications

  • observed — Evaluation retries remain bounded to three retries. Attempt-start output cleanup and ordinary terminal-state restoration improve failure containment, subject to the interruption rollback concern.

Hardening Proposals

  • proposed — Place final restoration around the entire retry loop so aborts during retry preparation or logging cannot leave temporary Trainer-owned configuration installed.
Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive The change map comment is not available in the supplied conversation, and live comment retrieval is unavailable. The diff has 341 additions and deletions, so the check requires scope-level design revi… Provide the complete conversation comment beginning with <!-- kumo-pr-change-map --> and any linked problem details needed to assess its design.
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The pull request changes 15 files. The limit is 40 files.
Public Resource Access Passed The pull request adds or modifies no external resource references. The added imports do not specify external resource locations.
Full details: Change Map Design Review

Explanation

The change map comment is not available in the supplied conversation, and live comment retrieval is unavailable. The diff has 341 additions and deletions, so the check requires scope-level design review. The available PR description does not replace the comment.

Resolution

Provide the complete conversation comment beginning with &lt;!-- kumo-pr-change-map --&gt; and any linked problem details needed to assess its design.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@anle2024

anle2024 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

GPU memory retries must preserve the nominal evaluation row budget. Refit test initialization rebuilds the data module from Trainer settings. Failed evaluation attempts also retain output rows unless their buffers are cleared. These are the two paths identified in the review.

The smallest fix keeps temporary test settings on both owners and clears partial outputs at the shared evaluation start hook. It leaves metric reset with the existing CUDA OOM callback.

File Function / scope What changed Why
packages/kumo-api/kumoapi/model_plan.py OptimizationPlan evaluation limits Accept positive integer caps above 8,000 and describe memory retry behavior. Allow public callers to request larger evaluation budgets.
packages/kumo-core/src/kumo_core/automl/auto_trainer.py _handle_oom_retry, _train Scale retry limits, forward the trial configuration and remove unused cached limits. Retain nominal validation and test row budgets during training retries.
packages/kumo-core/src/kumo_core/train/trainer.py handle_gpu_oom Scale test limits, synchronize Trainer and current data-module settings, and restore both on every exit. Make reduced batches and preserved budgets reach rebuilt refit modules.
packages/kumo-core/tests/test_evaluation_budget.py Retry and constructor regressions Cover retained and recreated modules, loader-option restoration, finite/unlimited caps and all retry exits. Catch refit retries that discard reduced batches or scaled limits without running a model.
packages/kumo-core/src/kumo_core/train/base_module.py on_evaluation_start Clear partial validation and test output buffers before evaluation starts. Prevent failed prefixes from being appended again on retry.
test/train/test_base_node_module.py test_evaluation_retry_discards_outputs_from_failed_attempt Reproduce a mid-evaluation OOM through the real start, step and end hooks for validation and test. Verify saved IDs, targets and predictions contain each retry row once.
packages/kumo-core/tests/test_model_plan_conversion.py Evaluation limit validation Reuse the existing full plan round-trip for large, None and missing evaluation limits. Keep the public and internal plan contracts consistent.
ui/src/gen/spec/v0_openapi.json Generated OptimizationPlan component Regenerate the affected evaluation limit fields from the API schema. Match the positive-integer and nullable contract.
ui/src/gen/spec/v1_openapi.json Generated OptimizationPlan component Regenerate the affected evaluation limit fields from the API schema. Match the positive-integer and nullable contract.
ui/src/gen/v0/schemas/training-jobs.ts Generated Zod evaluation fields Regenerate validation and test limit validators. Accept large positive limits and null while rejecting invalid values.
ui/src/gen/v0/types/optimizationPlanMaxTestSteps.ts Generated nullable step-limit alias Include null in the generated TypeScript type. Match the public API contract.
ui/src/gen/v0/types/optimizationPlanMaxValSteps.ts Generated nullable step-limit alias Include null in the generated TypeScript type. Match the public API contract.
ui/src/gen/v1/schemas/predictive-queries.ts Generated Zod evaluation fields Regenerate validation and test limit validators. Accept large positive limits and null while rejecting invalid values.
ui/src/gen/v1/types/optimizationPlanMaxTestSteps.ts Generated nullable step-limit alias Include null in the generated TypeScript type. Match the public API contract.
ui/src/gen/v1/types/optimizationPlanMaxValSteps.ts Generated nullable step-limit alias Include null in the generated TypeScript type. Match the public API contract.

Verification: 82 focused CPU tests pass. The recreated-module cases failed before the Trainer state fix; both output-buffer cases failed before the shared reset. Tests cover success, exhausted retries, other errors, finite/unlimited limits, exact loader-option restoration and aligned saved row IDs, targets and predictions. Focused Pyflakes and whitespace checks pass.

Stopping condition: both review reproductions pass and existing retry, plan-conversion and nearby module checks stay green. These simulated errors do not qualify real GPU OOM recovery or full-scale metric behavior.

Merge main, remove unused evaluation fields, and retain the existing inclusive minimum while allowing larger retry limits. Regenerate affected contracts and simplify the tests. Cover an ordinary error after an initial memory retry.
@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 00:25 — with GitHub Actions Inactive
@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 00:25 — with GitHub Actions Inactive
@anle2024
anle2024 marked this pull request as ready for review November 7, 2026 00:27
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

⚠️ OpenAPI spec changes detected in this PR

This means the UI client code needs to be regenerated.

To regenerate the UI client code in this PR, comment:
/trigger-ui-codegen

@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 00:45 — with GitHub Actions Inactive
@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 00:45 — with GitHub Actions Inactive
@anle2024

anle2024 commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test fee7bd3

@anle2024
anle2024 requested review from Vedaant-J and Victsm and removed request for Vedaant-J November 7, 2026 01:01
@Vedaant-J

Copy link
Copy Markdown
Contributor

The scaling and restore look right at fee7bd36, and the tests fail on main as expected. Two points:

  1. The retry doesn't reach the refit test pass. test() sets self.fit_mode and calls self.init() (trainer.py:2014-2016). In refit mode init() doesn't return early (diskgraph_nested_trainer.py:899); it builds a new DiskgraphLightningDataModule from self.eval_loader_kwargs and self.max_test_steps (:1026-1035). handle_gpu_oom sets the halved batch size and the scaled limit on the old datamodule (trainer.py:235-240), so each retry of test(fit_mode_to_use=FitMode.refit_trainval) (training/orchestration.py:385) runs again with the original batch size and limit. That is the pass whose metrics are published. The batch-size part already fails this way on main. Keeping the retry state on the Trainer (eval_loader_kwargs['batch_size'] and max_test_steps), so init() builds from it, would fix both. A test where func replaces the datamodule on each call would cover it.

  2. A retry after a mid-test OOM duplicates output rows. _test_step_outputs (base_module.py:1028) is only cleared in on_test_epoch_end, which doesn't run when the step raises; CudaOOMHandler resets metrics, not outputs. After an OOM at step k, the retry appends batches 0 to k-1 again, so saved test predictions and the holdout file get duplicate rows (metrics are unaffected). This predates the PR, so it could be a separate fix, but it lands on the same retry path: clearing _test_step_outputs and _validation_step_outputs in on_evaluation_start (base_module.py:399) or in CudaOOMHandler.on_exception would cover it.

@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 01:36 — with GitHub Actions Inactive
@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 01:36 — with GitHub Actions Inactive
@anle2024
anle2024 requested a review from Vedaant-J November 7, 2026 01:42
@anle2024

anle2024 commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

@Vedaant-J Addressed both points from your comment in 962fbb3b.

  1. Test retries now keep the reduced batch size and scaled test limit on the Trainer as well as the current datamodule. Refit initialization inherits those settings. Both owners restore their original settings on success, exhausted retries, or another error. The regression replaces the datamodule on every attempt.
  2. on_evaluation_start clears both partial output buffers. Validation and test regressions reproduce a mid-evaluation OOM and check that saved IDs, targets, and predictions contain each retry row once. Metric reset stays with the existing CUDA OOM callback.

Both failures were reproduced before the fixes. All 82 focused CPU checks pass. These tests simulate CUDA OOM; real GPU recovery remains unqualified.

@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 02:37 — with GitHub Actions Inactive
@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 7, 2026 02:37 — with GitHub Actions Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 962fbb3 and c94f135.

Files ignored due to path filters: 8

  • ui/src/gen/spec/v0_openapi.json is excluded by !**/gen/**
  • ui/src/gen/spec/v1_openapi.json is excluded by !**/gen/**
  • ui/src/gen/v0/schemas/training-jobs.ts is excluded by !**/gen/**
  • ui/src/gen/v0/types/optimizationPlanMaxTestSteps.ts is excluded by !**/gen/**
  • ui/src/gen/v0/types/optimizationPlanMaxValSteps.ts is excluded by !**/gen/**
  • ui/src/gen/v1/schemas/predictive-queries.ts is excluded by !**/gen/**
  • ui/src/gen/v1/types/optimizationPlanMaxTestSteps.ts is excluded by !**/gen/**
  • ui/src/gen/v1/types/optimizationPlanMaxValSteps.ts is excluded by !**/gen/**
Files selected for processing: 3

  • packages/kumo-core/src/kumo_core/automl/auto_trainer.py
  • packages/kumo-core/src/kumo_core/train/base_module.py
  • packages/kumo-core/src/kumo_core/train/trainer.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

UI Test Jobs Completed on cb08827

@anle2024
anle2024 merged commit 91dd4b9 into main Nov 7, 2026
151 checks passed
@anle2024
anle2024 deleted the codex/preserve-evaluation-row-budgets branch November 7, 2026 07:35

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — c94f1351 Deployed Nov 7, 2026 by anle2024 via Check OpenAPI Changes #1040
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants