Repository navigation
feat(training): opt-in exact quantile leaves (leaf_estimation_method = exact) - #164
Conversation
GHistIndex::column_bins() is Some only for dense indexes, so any matrix with missing values sent every vector-leaf (multi_output_tree) node through the serial CSR loop, whatever the thread count. NodeHistograms now sweeps the feature-major copy whenever the index keeps one: dense columns, or missing_columns() skipping the width's sentinel. Tasks of four pool features read each row's split gradients once, and only the tree's colsample_bytree pool is built (every node's candidate features are a subset of it). Each bin still adds its rows in one ascending chain, so models are bit-identical at every thread count.
…= exact) reg:quantileerror and reg:absoluteerror grow their structure on smooth surrogates and keep Newton leaves by default. LeafEstimation::Exact (CatBoost's leaf_estimation_method) sets output j of every leaf to eta times the alpha_j residual quantile (the median for absolute error) of the training rows the finished tree routes there: every row, sampled or not, by raw value, with the intercept's interpolated or weighted-step quantile. A positive max_delta_step bounds the quantile; lambda, alpha, gamma and min_child_weight act on the structure only. Scalar and vector leaves, several levels, label matrices and sample weights are supported; the all-row partition also adds the tree to the training margins. dart, gblinear, Boulevard, EBMs, monotone constraints, linear_tree, path_smooth, quantized leaf renewal, SGLB and refresh are refused. Python takes it as a params key, the scikit-learn estimators as a constructor parameter. The default (Newton) path is unchanged bit for bit.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ed0b44b5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…n's margins With `leaf_estimation_method = exact`, `multi_output_tree`, and `num_parallel_tree > 1`, each vector-leaf tree was added to the training margins before the next one was fitted, so later trees of the iteration took their residual quantiles against margins that already held the earlier ones while their gradients came from the iteration's start. The vector path now grows all of an iteration's trees before any joins the margins, as the scalar path does; Newton leaves never read the margins, so their models are unchanged. The exact-leaf reference tests now cover forests (scalar and vector leaves, `eta / num_parallel_tree`, outputs grouped by forest) and every builder (hist depth-wise and symmetric, approx, exact).
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…eaves # Conflicts: # AGENTS.md # src/data/sort.rs # src/tree/builder/multi.rs
Was stacked on #163 (merged); now targets
main, which is merged in.Change
New opt-in setting
TrainingParams::leaf_estimation, aLeafEstimationenum named after CatBoost'sleaf_estimation_method(also the flat key).Newtonstays the default and is bit-identical to before. WithExact, forreg:quantileerrorandreg:absoluteerror, the surrogate gradients still grow each tree's structure. Once the structure is fixed, outputj's value in each leaf is set toeta / num_parallel_tree× thealpha_j-quantile (the median for absolute error) of the residualslabel - margin_jof the leaf's rows.Where it's available:
TrainingParams::builder().leaf_estimation(LeafEstimation::Exact)leaf_estimation_method="exact"inparamsleaf_estimation_methodas a constructor parameterDecisions
Which rows: every training row, sampled or not. Each row is routed through the finished tree by its raw values, the same way prediction routes it. XGBoost's adaptive leaves and LightGBM's leaf renewal use the round's sample instead. I measured both on the bands split (all rows vs. sampled rows, test pinball):
All rows wins clearly on q10 at eta 0.06, the regime where exact leaves pay off. It also gives each leaf twice the rows at
subsample = 0.5, and half of those rows did not choose the leaf's splits.The same routing pass then updates the training margins. In a sampled round that replaces the walk Newton leaves need anyway; an unsampled round adds one walk per tree, since Newton leaves read the builder's partitions there. On 1M × 20 rows (depth 6, 30 rounds, 64 threads), exact leaves add 0.12 s with or without sampling, so the walk costs little next to the quantile fit.
Quantile definition: the same as the intercept's, i.e. XGBoost's
common::QuantileandWeightedQuantile.(n + 1)αgrid.f32cumulative weight reachesαof the leaf's total.total_cmpwith a stable radix sort, so ties keep row order.weighted_step_quantile. Its result is unchanged;partition_pointsemantics are kept exactly, including for a NaN threshold.Regularization:
lambda,alpha,gammaandmin_child_weightaffect only the structure, through the surrogate gradients.max_delta_stepclamps the quantile to±max_delta_stepbeforeetascales it.Refused (
invalid_param("leaf_estimation_method")): every objective other thanreg:quantileerrorandreg:absoluteerror. Alsodart,gblinear, Boulevard, EBMs, monotone constraints,linear_tree,path_smooth, quantized leaf renewal, SGLB / posterior sampling, andprocess_type = update, because these bound, replace or recompute the leaves themselves. Online updates and budget mode also refuse it, through their existing allow-lists.Unchanged: the model format; models still export to XGBoost (leaves are plain values).
Determinism: leaves (and each output of a vector leaf) are fit in parallel. Each is computed from its own sorted residuals, so results do not depend on the thread count.
Numbers
Native bands benchmark:
hbbench/exactbands.py, the bands split (2.77M train / 615k valid / 616k test rows), early stopping with patience 40, 16 threads, one scalar model per alpha. Columns are test pinball loss and rounds to early stop.Against the shipped default (Newton, eta 0.12):
So the acceptance bar ("about 2× fewer rounds at equal test pinball, or better pinball at fewer rounds than the default at eta 0.12") is met at eta 0.06. Exact leaves want a smaller eta: at 0.12 the q10 band overshoots.
Vector leaves (one
multi_output_treemodel for both bands):With recency sample weights (
WEIGHTED=1, half-life 72 h, eta 0.06): q10 newton 1.42985 at 780 rounds vs. exact 1.42523 at 200 rounds; q90 newton 1.44296 at 783 rounds vs. exact 1.44355 at 155 rounds.Your Python emulation (
leafrefit.py, 500k rows) found the same pattern: about a quarter of the rounds at eta 0.12 with slightly worse pinball, and better pinball at eta 0.03.Tests
tests/exact_leaves.rs: a brute-force reference routes every row by raw value, recomputes each leaf's quantile at the margins of the iterations before its tree's, and compares bit for bit. Cases:subsample 0.6andcolsample_bytree 0.75, under every builder (hist depth-wise and symmetric, approx, exact)num_parallel_tree = 3), scalar and vector leaves: every tree of an iteration fits the same starting marginsreg:absoluteerroron a two-column label matrix (each output's own column)max_delta_stepclamping (the bound binds)Perturbing alpha in the implementation makes five of the reference tests fail.
Review fix (Codex): a vector-leaf forest added each tree to the training margins before fitting the next, so its later trees took their exact leaves against margins that already held the earlier ones. The vector path now grows all of an iteration's trees before any joins the margins, as the scalar path does. The forest case above failed before the fix (
tree 1 leaf 7 output 0). Newton vector-leaf forests (subsampled and not) and a scalar forest keep the same model and margin hashes.Flat-key round trip in
config/xgboost/tests.rs.Default path:
bitident.py(vector and scalar configs at 1, 4 and 16 threads) gives the same model and margin hashes as main. With the review fix:cargo nextest run --all-features(867 passed),cargo test --doc --all-features, clippy--all-features -D warnings(CUDA headers installed locally) and rustdoc-D warningsare clean. Python: pytest (435 passed), ruff and ty are clean.