Repository navigation
Cluster-robust standard errors (builds on #71) - #72
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #72 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 31 32 +1
Lines 2301 2618 +317
==========================================
+ Hits 2301 2618 +317 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The previous commit's editor re-indented nine existing files (function bodies moved from two to four spaces, and blank lines picked up trailing spaces), which made the diff about 4,000 lines and would have rewritten git blame for vim-factors.R and vim-numerics.R. This keeps every code change and puts it back on master's formatting: `git diff -w` against the previous commit shows only DESCRIPTION and a restored blank line at the end of R/estimate_tmle2.R. Also restores RoxygenNote: 7.3.2 in DESCRIPTION, strips trailing whitespace from the new files, adds the missing final newline to R/create_cv_folds.R, and adds Pratik Khadse to Authors@R as a contributor for cluster-robust standard errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
ck37
force-pushed
the
claude/loving-brahmagupta-xdc7g1
branch
3 times, most recently
from
October 11, 2026 22:58
bde942d to
cc0e5b1
Compare
Variance. The clustered standard error was the average of per-fold clustered variances. A fold holding a single cluster then had no variance, and the variable vanished from the results (21 hospitals, one with 55% of the rows, V = 2: results_all had no rows); and a cluster that spans folds, as with crossed clustering variables, was split and its covariance dropped (88% coverage for 95% intervals). cv_cluster_var() now computes one clustered variance over every fold's influence curve, each scaled by n / (V n_v) to match the average of fold estimates and re-centered at the overall estimate, since centering within folds erases the between-cluster variation of a fold with one cluster. Each cluster's sum is divided by 1 - h, its share of the weight, with a (G - 1) / G factor: the CR3 (jackknife-style) variance, equal to sandwich::vcovCL(type = "HC3", cadjust = FALSE) for one-, two- and three-way clustering and to the usual G / (G - 1) version when clusters are equal in size. Degrees of freedom. Each variable's t reference uses the clusters behind its estimate: in each fold, the fewer of the clusters holding its low and its high level, summed over folds, minus one, capped at (fewest clusters - 1). It is shown in a df column of results_all. A variable whose low or high level comes from a single cluster gets no interval or p-value, with a message. Folds. Every fold gets at least two clusters when there are enough, so a large cluster no longer fills a fold alone; with clusters of similar size the folds are unchanged. SuperLearner's grouped folds for g and g.Delta stay stratified, built with the same grouped assignment, which now balances whichever outcome value is rarer. Other fixes: - The macOS test failure: relabeled cluster ids reorder rowsum()'s sums, which changes the last bits without extended precision; the test now uses expect_equal(). - The "id is NULL" warning fires only for data$column or data[["column"]], not for a wrapper forwarding its own NULL default. - A factor whose levels each fall within one cluster, such as the cluster id left in data, is set aside with a message instead of vanishing. - apply_tmle_to_validation()'s row-number argument is renamed row_index. - print() notes the clustering and the largest df; ?varimpact and the README have an example; the id docs note that learners' own internal folds ignore clusters. Tests cover the new code to 100%, including the variance against sandwich::vcovCL() and the cluster jackknife, continuous and missing outcomes with id, and fold assignment with a dominant cluster. The choices were made by simulation (varimpact() with id on clustered data; false-positive rates for null variables at the 5% level): - with id they ranged from 0% to 9%, against 12% to 73% without; - CR3 cut a null covariate's rate from 7.6% to 4.6% when one cluster held half the rows, and left other designs within a point; - the per-variable df kept cluster-level factors nearer 5% than one global df or a Welch-type df (6.7% and 6.0% against 7.7% and 8.5%), at the cost of being conservative with 15 clusters. The variance formula alone covered 92% to 95% in every design, where the per-fold average fell to 87% with crossed clustering. The code comments and ?varimpact's references cite the literature behind the variance across folds (the cross-validated TMLE's influence curve: Zheng & van der Laan 2010; Hubbard, Kherad-Pajouh & van der Laan 2016), the t reference, the CR3 variance and the degrees of freedom (Cameron, Gelbach & Miller 2008, 2011; Cameron & Miller 2015; Bell & McCaffrey 2002; Imbens & Kolesar 2016; MacKinnon, Nielsen & Webb 2023). They also say that the per-variable df count is a simple, conservative heuristic rather than a published rule. With crossed clustering variables that cannot all stay within folds, the fallback warning now names the variable whose clusters span folds and says that training and validation folds then share them, which can bias variables that vary with them; the folds_by docs and NEWS explain how to choose the variable to fold on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
tests/testthat/test-cluster-simulation.R holds two simulation studies that run only when VARIMPACT_RUN_SIMULATIONS=true, so CI and R CMD check skip them: - The full pipeline: clustered data with two null predictors (one varying within clusters, one constant within them) and one real effect, fitted with and without id across five designs (40 clusters, 15 clusters, one cluster holding 30% or 50% of the rows, no cluster effect on the outcome, and site crossed with school, also fitted with folds_by = "site"). It checks that with id the null variables' false-positive rates are within Monte Carlo error of 5%. About an hour on 3 cores. - The variance formula alone, without fitting varimpact(): 95% interval coverage for the average of per-fold variances, one variance centered within folds, and varimpact's cv_cluster_var(), across crossed, one-way and dominant-cluster designs. About a minute. The reference results are recorded at the end of the file. With id, null false-positive rates ranged from 0% to 9% (from 12% to 73% without), and the variance formula covered 91.9% to 94.6% in every design. The documentation of the clustered variance, the per-variable degrees of freedom and the id argument points to this study. The crossed design shows the limitation of folds that cannot keep every clustering variable intact: folding on school (the default), a factor constant within sites was flagged 13% of the time with id (36.5% without), from bias rather than variance, since training and validation folds share sites; folding on site brought it to 3.0%. The folds_by documentation points to these results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
ck37
force-pushed
the
claude/loving-brahmagupta-xdc7g1
branch
from
October 11, 2026 23:23
cc0e5b1 to
4be483d
Compare
ck37
marked this pull request as ready for review
October 11, 2026 23:42
This was referenced Oct 11, 2026
ck37
pushed a commit
that referenced
this pull request
Oct 12, 2026
Release version with today's date, and the NEWS heading for everything since 1.2.4, including cluster-robust standard errors (#71, #72, #73). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
ck37
pushed a commit
that referenced
this pull request
Oct 12, 2026
Release version with today's date, and the NEWS heading for the changes recorded since NEWS.md was started in 2025, including cluster-robust standard errors (#71, #72, #73). The old "1.2.4" NEWS heading never matched a DESCRIPTION version, which went from 1.2 to 1.3 in January 2017 and has been a 1.3.0 development version since, so it now reads 1.3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch carries Pratik Khadse's commit from #71 (
idargument for cluster-robust standard errors) unchanged, with three follow-up commits on top. It is a draft so that CI runs on all three platforms; #71 failed on macOS.What the follow-up commits change
RoxygenNote: 7.3.2.sandwich::vcovCL(type = "HC3", cadjust = FALSE)for one-, two- and three-way clustering, and the usual G/(G-1) version when clusters are equal in size.dfcolumn ofresults_allwhen clustering. This count is a conservative heuristic motivated by Bell & McCaffrey (2002) and Imbens & Kolesár (2016), not a published rule.folds_by.id = NULLwarning only fordata$columntypos, factors whose levels each sit in one cluster are set aside with a message, SuperLearner's grouped folds stay stratified for g and g.Delta,print()notes clustering, README and?varimpactexamples and references, Pratik added toAuthors@R.Validation
R CMD checkpasses locally (Linux); CI green on macOS, Windows and Ubuntu.id, and for the variance againstsandwich::vcovCL()and the cluster jackknife.tests/testthat/test-cluster-simulation.R(skipped unlessVARIMPACT_RUN_SIMULATIONS=true):id, against 12-73% without.id): training and validation folds share sites, which biases it. Withfolds_by = "site"it was 3.0%.id, results are bit-identical to master.🤖 Generated with Claude Code
https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T