You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Keep evaluation coverage after GPU memory retries - #234
GPU memory retries reduce batch size. Fixed evaluation step limits then cover fewer rows. Refit tests also rebuild their data module, so retry settings must survive that rebuild. Partial outputs from failed evaluation attempts must be discarded.
This change preserves the configured evaluation row budget. For example:
Evaluation attempt
Batch size
Step limit
Configured row budget
Before the memory error
64
1,000
64,000
Retry before this fix
32
1,000
32,000
Retry with this fix
32
2,000
64,000
What changes
Scale finite validation and test step limits as batches shrink. Forward the adjusted trial limits to the Trainer.
Keep test retry settings on both the Trainer and current data module. Refit-created modules inherit them. Restore the original settings after success, exhausted retries, or another error.
Clear partial validation and test outputs before each evaluation attempt, so a retry does not duplicate saved prediction rows.
Remove the 8,000-step API ceiling while preserving positive integer, nullable and default behavior. Regenerate the affected API clients.
This preserves the configured row budget. Timestamp bucket boundaries can still change the exact rows evaluated. Prediction and explanation populations retain their existing behavior.
Testing Done
At 962fbb3b, 82 focused CPU tests pass. Coverage includes recreated refit data modules, finite and unlimited test limits, caller-owned loader options, all retry exits, public/internal plan conversion, and saved output rows after partial failures.
Before the fixes, the new cases reproduced retries restarting at batch 64 and duplicate output IDs [10, 10, 11] instead of [10, 11]. Focused Pyflakes and whitespace checks pass.
The tests simulate GPU memory errors. Real GPU recovery and full-scale metric behavior remain unqualified for this revision.
Optimization plans now accept any positive integer for validation and test step limits. Automatic-training and test OOM retries reduce the batch size and scale applicable step limits. The test OOM handler restores the original test batch size, step limit, and trainer evaluation configuration when the call exits. Evaluation starts clear accumulated validation and test outputs.
Sequence Diagram(s)
sequenceDiagram
participant AutoTrainer
participant TrialConfig
participant DiskgraphNestedTrainer
AutoTrainer->>TrialConfig: Read current validation and test step limits
AutoTrainer->>DiskgraphNestedTrainer: Start trial with batch size and step limits
DiskgraphNestedTrainer-->>AutoTrainer: CUDA OOM
AutoTrainer->>TrialConfig: Reduce batch size and scale set step limits
AutoTrainer->>DiskgraphNestedTrainer: Retry trial with updated configuration
Loading
Merge Risk: Moderate · up to c94f1
After a GPU memory retry during testing, logged test metrics may include rows from the failed attempt. Reset the stage's metric collection when an evaluation starts before merging.
Security architecture risk: Low · up to c94f1
The change preserves evaluation coverage after memory retries without establishing a new security attack path. A narrow interruption case can leave temporary retry settings installed. Production workload limits and tenant isolation could not be verified.
Retained concerns
Low · reliability · inferred: If interruption or an exception exits retry preparation after retrying is set to true, the finally block skips restoration. The PR newly stores retry settings on the Trainer itself, so temporary configuration can survive the aborted call and be inherited by later refit initialization. The base already lacked interruption cleanup for the current data module’s batch size; the added effect is persistence in the configuration owner used to rebuild modules. No cross-tenant impact or attacker-triggered path is established.
Security review details
Security Blast Radius
inferred — The demonstrated resource effect concerns validation and test evaluation for the configured workload. The source establishes split-level execution limits, but does not establish the independently attackable tenant, service or environment scope.
Security Findings and Attack Paths
inferred — Removing the finite-input ceiling expands accepted values, but does not by itself establish a new resource-exhaustion capability because whole-split evaluation was already expressible with None. Production acceptance and enforcement remain unverified.
Trust Boundaries and Controls
observed — The changed boundary is public configuration validation feeding core evaluation, not an observed identity or privilege transition. Positive-value validation remains; the former finite upper bound does not.
Resilience and Maintainability Implications
observed — Evaluation retries remain bounded to three retries. Attempt-start output cleanup and ordinary terminal-state restoration improve failure containment, subject to the interruption rollback concern.
Hardening Proposals
proposed — Place final restoration around the entire retry loop so aborts during retry preparation or logging cannot leave temporary Trainer-owned configuration installed.
Pre-merge checks: 2 · 1
Failed checks: 1 inconclusive
Check name
Status
Explanation
Resolution
Change Map Design Review
The change map comment is not available in the supplied conversation, and live comment retrieval is unavailable. The diff has 341 additions and deletions, so the check requires scope-level design revi…
Provide the complete conversation comment beginning with <!-- kumo-pr-change-map --> and any linked problem details needed to assess its design.
Passed checks: 2 passed
Check name
Status
Explanation
Pr Size
The pull request changes 15 files. The limit is 40 files.
Public Resource Access
The pull request adds or modifies no external resource references. The added imports do not specify external resource locations.
Full details: Change Map Design Review
Explanation
The change map comment is not available in the supplied conversation, and live comment retrieval is unavailable. The diff has 341 additions and deletions, so the check requires scope-level design review. The available PR description does not replace the comment.
Resolution
Provide the complete conversation comment beginning with <!-- kumo-pr-change-map --> and any linked problem details needed to assess its design.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
GPU memory retries must preserve the nominal evaluation row budget. Refit test initialization rebuilds the data module from Trainer settings. Failed evaluation attempts also retain output rows unless their buffers are cleared. These are the two paths identified in the review.
The smallest fix keeps temporary test settings on both owners and clears partial outputs at the shared evaluation start hook. It leaves metric reset with the existing CUDA OOM callback.
Verification: 82 focused CPU tests pass. The recreated-module cases failed before the Trainer state fix; both output-buffer cases failed before the shared reset. Tests cover success, exhausted retries, other errors, finite/unlimited limits, exact loader-option restoration and aligned saved row IDs, targets and predictions. Focused Pyflakes and whitespace checks pass.
Stopping condition: both review reproductions pass and existing retry, plan-conversion and nearby module checks stay green. These simulated errors do not qualify real GPU OOM recovery or full-scale metric behavior.
Merge main, remove unused evaluation fields, and retain the existing inclusive minimum while allowing larger retry limits. Regenerate affected contracts and simplify the tests. Cover an ordinary error after an initial memory retry.
The scaling and restore look right at fee7bd36, and the tests fail on main as expected. Two points:
The retry doesn't reach the refit test pass.test() sets self.fit_mode and calls self.init() (trainer.py:2014-2016). In refit mode init() doesn't return early (diskgraph_nested_trainer.py:899); it builds a new DiskgraphLightningDataModule from self.eval_loader_kwargs and self.max_test_steps (:1026-1035). handle_gpu_oom sets the halved batch size and the scaled limit on the old datamodule (trainer.py:235-240), so each retry of test(fit_mode_to_use=FitMode.refit_trainval) (training/orchestration.py:385) runs again with the original batch size and limit. That is the pass whose metrics are published. The batch-size part already fails this way on main. Keeping the retry state on the Trainer (eval_loader_kwargs['batch_size'] and max_test_steps), so init() builds from it, would fix both. A test where func replaces the datamodule on each call would cover it.
A retry after a mid-test OOM duplicates output rows._test_step_outputs (base_module.py:1028) is only cleared in on_test_epoch_end, which doesn't run when the step raises; CudaOOMHandler resets metrics, not outputs. After an OOM at step k, the retry appends batches 0 to k-1 again, so saved test predictions and the holdout file get duplicate rows (metrics are unaffected). This predates the PR, so it could be a separate fix, but it lands on the same retry path: clearing _test_step_outputs and _validation_step_outputs in on_evaluation_start (base_module.py:399) or in CudaOOMHandler.on_exception would cover it.
Test retries now keep the reduced batch size and scaled test limit on the Trainer as well as the current datamodule. Refit initialization inherits those settings. Both owners restore their original settings on success, exhausted retries, or another error. The regression replaces the datamodule on every attempt.
on_evaluation_start clears both partial output buffers. Validation and test regressions reproduce a mid-evaluation OOM and check that saved IDs, targets, and predictions contain each retry row once. Metric reset stays with the existing CUDA OOM callback.
Both failures were reproduced before the fixes. All 82 focused CPU checks pass. These tests simulate CUDA OOM; real GPU recovery remains unqualified.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GPU memory retries reduce batch size. Fixed evaluation step limits then cover fewer rows. Refit tests also rebuild their data module, so retry settings must survive that rebuild. Partial outputs from failed evaluation attempts must be discarded.
This change preserves the configured evaluation row budget. For example:
What changes
Addresses both points in Vedaant's review.
Scope and limits
This preserves the configured row budget. Timestamp bucket boundaries can still change the exact rows evaluated. Prediction and explanation populations retain their existing behavior.
Testing Done
At
962fbb3b, 82 focused CPU tests pass. Coverage includes recreated refit data modules, finite and unlimited test limits, caller-owned loader options, all retry exits, public/internal plan conversion, and saved output rows after partial failures.Before the fixes, the new cases reproduced retries restarting at batch 64 and duplicate output IDs
[10, 10, 11]instead of[10, 11]. Focused Pyflakes and whitespace checks pass.The tests simulate GPU memory errors. Real GPU recovery and full-scale metric behavior remain unqualified for this revision.
Change map
Maintained file-by-file change map