Repository navigation
feat!: model averaging, out-of-fold cross-validation with per-fold stopping; faster quantile scale pass - #165
Merged
Merged
Conversation
…intercept residual_scales (reg:quantileerror, reg:absoluteerror) forms its square-root terms eight rows at a time so they vectorize and sums the total weight in the same pass as output 0's terms; every f64 sum still adds in row order from -0.0, so scales are bit-identical. The quantile intercept's stable label order comes from a stable radix sort on canonical (+0) keys instead of a comparison sort. Models and margins are bit-identical to before.
… stopping BoostedModel::average (Python Booster.average) combines compatible members into one model predicting their mean margin: trees concatenated with every leaf the f32 nearest v/k, intercepts averaged in f64; incompatible members (objective, features, layout, DART weights, shrinkage, gblinear, Boulevard, EBM, early stopping) are refused with IncompatibleModel. CrossValidation::run now returns a CvReport with each fold's round count, scores, and test margins, and CvReport::out_of_fold combines them per row. CrossValidation::early_stopping(CvStopping) replaces early_stopping_rounds: fold-mean stopping as before, or per-fold stopping on a PurgedTail of each fold's own training rows, refitting each fold on all its rows. Python adds hessboost.cross_validate, CvReport, FoldReport, OutOfFold, TailStopping, and folds.PurgedTail; hessboost.cv keeps its XGBoost-shaped results. BREAKING CHANGE: CrossValidation::run returns CvReport (results in CvReport::results), CvRefit::results is CvRefit::report, and CrossValidation::early_stopping_rounds is early_stopping(CvStopping::fold_mean).
- average: compare leaf kinds against the first member with trees, and validate the result's structure as a last guard (returning the error). - cv: refuse EBM bag stopping under every CvStopping; refuse per-fold stopping for Boulevard fits up front; name tail-split refusals `purged tail` everywhere. - PurgedTail: Fold::purged_forward's single-block convention (fraction of distinct decision times, split at a decision time); length mismatches (also purged_forward's) are DimensionMismatch. - Python: cross_validate's CvReport.refit is a Refit without the history; CvRefit is cv()'s record again; Booster.average raises TypeError for non-Boosters; PurgedTail.split refuses non-integer rows.
PurgedTail starts the tail at the first decision time at or after first + (1 - fraction) * (last - first), as slutbot's early_stopping_split does, instead of by the count of distinct decision times (purged_forward's rule), so an irregular gap no longer shifts the tail. split_at stays a decision time and the purge stays purged_forward's (inclusive).
…hon Refit Check that DART folds stopped early on the fold means report the margins of the fold retrained for the chosen rounds, and that per-fold stopping with target statistics reports the hand-built trial's best score. Rename the Python hessboost.Refit to CvReportRefit and reword the README's per-fold stopping paragraph.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
CvReport's fields are public, so out_of_fold checks each fold's margin rows against its test rows, every fold's width against fold 0's, and the test rows' bounds before indexing by them.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c11cf3fb78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- CrossValidation::history_only reports the fold-mean history alone; cv and Python's hessboost.cv run it, so they keep no fold model past its run and replay no DART, EBM or Boulevard fold for margins they discard (hessboost.cv with DART and early stopping took about 1.5x as long). Refused with per-fold stopping, which has no fold-mean history. - Without early stopping, a fold's model is dropped once its margins are taken instead of every fold's being kept until the last trains. - PurgedTail::split compares decision-time offsets as integers against the cutoff rounded up to a whole tick and capped at the span: past 2^53 ticks (nanosecond times over months) an f64 offset one tick before the cutoff rounded onto it and moved into the tail. - Python: cross-validation of zero rounds builds empty fold scores instead of raising ValueError (hessboost.cv(params, dtrain, 0) did); PurgedTail.split refuses negative rows with HessboostError, not OverflowError. - Docs: a fold's EBM round count counts both stages, each capped at num_boost_round; the EBM bag-stopping refusal no longer names the removed early_stopping_rounds builder; Booster.average refuses DART models with non-unit tree weights, and max_delta_step mismatches.
Owner
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three slutbot-driven items from the training workstream:
perf(objective): the quantile/absolute-error residual-scale pass forms its square-root terms 8 rows at a time and sums the total weight in the same pass as output 0; everyf64sum still adds in row order from-0.0. The quantile intercept's 4M-label comparison sort becomesdata::sort::stable_order, a stable radix sort (-0/+0one key). Bit for bit unchanged.BoostedModel::average(PythonBooster.average): native model averaging. Every leaf (vector leaves and linear-leaf coefficients included) is divided by the member count, trees are concatenated, and intercepts are averaged in margin space. Members must agree on objective (with its parameters),max_delta_step, features, outputs, targets, classes,num_parallel_treeand leaf kind; gblinear, model shrinkage, non-unit tree weights (DART), Boulevard, EBM andbest_iterationare refused (IncompatibleModel { what: "average" }). For log links the average is the geometric mean of the members' predictions, for multiclass the softmax of the mean logits (documented).CrossValidation::early_stopping(CvStopping)replacesearly_stopping_rounds, withCvStopping::fold_mean(patience)(the existing mode) andCvStopping::per_fold(patience, PurgedTail). In per-fold mode each fold early-stops on a purged chronological tail of its own training rows, then retrains on all of them for the chosen rounds.run()returns aCvReportwhosefoldscarry each fold's test rows, round count, scores, margins and tail split;CvReport::out_of_fold()assembles per-row OOF margins.CrossValidation::history_only()reports the fold-mean history alone, whichcvand Python'shessboost.cvrun. Python:hessboost.cross_validate(..., per_fold_stopping=folds.PurgedTail(...), refit=)returningCvReport(CvReportRefitfor the refit);hessboost.cvkeeps its XGBoost-shaped results.PurgedTailcuts the tail by time span, as slutbot'searly_stopping_splitdoes (the first decision time at or afterfirst + (1 − fraction)·(last − first)), and purges withpurged_forward's inclusive rule.Version: the CV API change is breaking, so both manifests go to 0.4.0 (crates.io's latest release is 0.3.0; main also carries the unreleased
feat(python)!of #162).Measurements (
/home/admin/hbbench, 4M × 173 panel, Neoverse-V3)Residual-scale pass, per call:
End to end (
prof.py 8 41, five alternating runs, min / median per round; the shifts are within run-to-run noise, squared error moved as much without being touched):The intercept sort shows in the first round: quantile 3.18 → 2.95 s, weighted 3.58 → 3.35 s.
Bit identity:
ws5_bitident.pyhashes model JSON and margins of quantile (one and three alphas) and absolute error (one and two targets), weighted with zero weights and unweighted, at 1/4/16 threads: all 24 lines equal before and after.averageagainst slutbot'saverage_models(3 seeds × 30 rounds, every 40th panel row):reg:quantileerrorreg:tweedieEvery native leaf is the correctly rounded
v/k;average_modelsscales the shortest-decimal JSON leaves, which parse tof64values other than the storedf32(all 1435 checked), so its products miss by one ulp. Intercepts agree exactly.Review fixes
history_only: routinghessboost.cvthrough the report path kept every fold's model until the best fold-mean round was known and, for DART/EBM/Boulevard under early stopping, retrained every fold for margins it then discarded. History-only runs keep nothing past each fold's scores (refused with per-fold stopping, which has no fold-mean history).hessboost.cvwith DART (20k × 20, 5 folds, 150 rounds; same process,early_stopping_rounds=10vs none): 1.85 s vs 1.26 s (1.47×) before, 1.48 s vs 1.52 s (0.97×) after.PurgedTail::splitcompares offsets as integers against the cutoff rounded up to a whole tick, capped at the span: past 2^53 ticks (datetime64[ns]times over ~104 days) anf64offset one tick before the cutoff rounded onto it and joined the tail ([0, t − 1, t, 2t],t = 2^54, fraction 0.5 split att − 1). Unchanged below 2^53.ValueErrorbuilding fold scores (hessboost.cv(params, dtrain, 0)included);PurgedTail.splitraisedOverflowErrorfor negative rows instead ofHessboostError.out_of_foldrefuses an edited report (public fields) instead of indexing out of bounds.num_boost_round(so not "at mostnum_boost_round"); the EBM bag-stopping refusal no longer names the removedearly_stopping_roundsbuilder;Booster.averagerefuses DART only for non-unit tree weights.Checks
cargo fmt --check(root, python, fuzz, cuda-kernels), clippy--all-targets --features wgpu -D warnings(root) and--all-targets -D warnings(python),cargo nextest run --cargo-profile ci --features wgpu(828 passed), doctests,RUSTDOCFLAGS=-D warnings cargo doc --features wgpu,uv run --locked pytest(440 passed), ruff, ty,pyright --verifytypes(100%),cargo semver-checks --features wgpu(0.3.0 → 0.4.0: major, passes).