fix(trees): make DecisionTree::fit deterministic - #452
Open
mysma-9403 wants to merge 1 commit into
Open
Conversation
Repeated `fit` calls on identical data could return differently shaped
trees. On the iris dataset, 60 fits in one process produced 4 distinct
trees, differing in leaf count (9, 10, 11 and 12) and therefore in their
predictions.
Class frequencies were carried in a `HashMap<L, f32>`, and `HashMap`
iteration order is randomised per instance. Every consumer of those
frequencies depended on the order:
* `find_modal_class` folded with `best_freq > freq`, so on a tie it kept
the *last* maximum it happened to visit — the opposite of the "first
class found" its doc comment promises.
* `gini_impurity` and `entropy` sum the frequencies in `f32`, and
floating-point addition is not associative, so the same frequencies
summed in a different order produce slightly different scores.
Those score differences flip the `score < best_score` comparison in
`TreeNode::fit` at near-ties, which selects a different split and changes
the shape of the whole subtree.
Frequencies are now `Vec<f32>` indexed by position in a sorted class list
built once per fit, so the iteration order is pinned. The tree this
produces is one of the trees the previous code already produced, just
consistently rather than at random.
As a side effect this is also faster: the innermost split-search loop ran
`right_class_freq.get_mut(class)` and `left_class_freq.entry(class.clone())`
for every (sample, feature) pair, i.e. two hash lookups plus a label clone,
which for `String` labels meant an allocation. Indexing a `Vec` removes all
three.
n=2000 d=20 k=3 707.2 ms -> 271.5 ms (2.6x)
n=8000 d=20 k=3 2570.1 ms -> 860.9 ms (3.0x)
n=8000 d=20 k=10 2198.7 ms -> 861.1 ms (2.6x)
n=16000 d=50 k=5 14652.5 ms -> 4494.8 ms (3.3x)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4uUK6s1CWFwx6u7Rx28go
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #452 +/- ##
==========================================
+ Coverage 77.53% 78.04% +0.51%
==========================================
Files 106 104 -2
Lines 7585 7543 -42
==========================================
+ Hits 5881 5887 +6
+ Misses 1704 1656 -48 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DecisionTree::fitis not reproducible: repeated calls on identical data return differently shaped trees.On the iris dataset, 60 fits in a single process produce 4 distinct trees, differing in leaf count (9, 10, 11, 12) and therefore in their predictions:
Cause
Class frequencies were carried in a
HashMap<L, f32>, andHashMapiteration order is randomised per instance. Both consumers of those frequencies depend on the order:find_modal_classfolded withif best_freq > freq { acc } else { Some((idx, freq)) }, so on a tie it kept the last maximum it happened to visit — the opposite of the "first class found with that frequency is returned" its own doc comment promises.gini_impurityandentropysum the frequencies inf32, and floating-point addition is not associative. The same frequencies summed in a different order give slightly different scores.The second one is what changes the tree shape: those small score differences flip the
score < best_scorecomparison inTreeNode::fitat near-ties, selecting a different split and changing the whole subtree below it.Fix
Frequencies become
Vec<f32>indexed by position in a sorted class list (ClassIndex) built once per fit, so iteration order is pinned.find_modal_classnow also uses>=so that ties resolve to the first class, matching its documentation.The tree this produces is one of the trees the previous code already produced — I verified this by collecting the fingerprint set of 60 fits on master and checking that the now-deterministic output is a member of it. This changes which of several equally-valid trees you get, consistently, rather than introducing a new one.
Side effect: it is also faster
The innermost split-search loop ran, for every (sample, feature) pair:
Two hash lookups plus a label clone — for
Stringlabels, a heap allocation. Indexing aVecremoves all three.Benchmark context
lowpowermode 0)DecisionTree::params().fit(&ds), default hyperparameters, measured with a throwaway harness (not part of this PR); A/B taken on this branch with and without the change so the binary shape is identicalNotes
ClassIndex::frequenciesreplacesDatasetBase::label_frequencies_with_maskinside the tree. The only behavioural difference is that classes absent from a node are present with frequency0.0instead of being omitted, which is inert: a zero frequency contributes nothing to either impurity measure, and can never be the modal class of a non-empty node.fit_is_deterministicas a regression test, plusmodal_class_ties_pick_the_first_classto pin the documented tie-break.gini_impurity/entropy/find_modal_classwere updated for the new slice signatures; their expected values are unchanged.Checks
cargo fmt --all -- --check— cleancargo clippy --workspace --all-targets -- -D warnings— cleancargo test --release --workspace— 546 tests, no failures🤖 Generated with Claude Code
https://claude.ai/code/session_01L4uUK6s1CWFwx6u7Rx28go