Skip to content

Cluster-robust standard errors (builds on #71) - #72

Merged
ck37 merged 4 commits into
masterfrom
claude/loving-brahmagupta-xdc7g1
Oct 11, 2026
Merged

ck37 merged 4 commits into
masterfrom
claude/loving-brahmagupta-xdc7g1

Conversation

@ck37

@ck37 ck37 commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

This branch carries Pratik Khadse's commit from #71 (id argument for cluster-robust standard errors) unchanged, with three follow-up commits on top. It is a draft so that CI runs on all three platforms; #71 failed on macOS.

What the follow-up commits change

  • Formatting: restores the original indentation in the nine files Add cluster-robust standard errors #71 re-indented, so the diff against master is about 1,600 lines rather than 7,000, and restores RoxygenNote: 7.3.2.
  • Clustered variance over all folds: one cluster-robust variance over every fold's influence curve, re-centered at the overall estimate, replaces the average of per-fold variances. This follows from the cross-validated TMLE's influence curve with clusters as the independent units (Zheng & van der Laan 2010; Hubbard, Kherad-Pajouh & van der Laan 2016). Folds holding a single cluster no longer make variables disappear, and clusters that span folds count once.
  • Small-sample correction: jackknife-style (CR3) variance; equals sandwich::vcovCL(type = "HC3", cadjust = FALSE) for one-, two- and three-way clustering, and the usual G/(G-1) version when clusters are equal in size.
  • Per-variable degrees of freedom: from the clusters holding each variable's low and high level, shown in a new df column of results_all when clustering. This count is a conservative heuristic motivated by Bell & McCaffrey (2002) and Imbens & Kolesár (2016), not a published rule.
  • Fold assignment: every fold gets at least two clusters when there are enough, so a cluster holding about 1/V of the data no longer fills a fold by itself.
  • Crossed clustering: when crossed variables cannot all stay within folds, the warning now names the variable whose clusters span folds and explains how to choose folds_by.
  • Smaller fixes: macOS test failure (floating-point summation order), id = NULL warning only for data$column typos, factors whose levels each sit in one cluster are set aside with a message, SuperLearner's grouped folds stay stratified for g and g.Delta, print() notes clustering, README and ?varimpact examples and references, Pratik added to Authors@R.

Validation

  • Full test suite passes with 100% line coverage; R CMD check passes locally (Linux); CI green on macOS, Windows and Ubuntu.
  • Tests for continuous and missing outcomes with id, and for the variance against sandwich::vcovCL() and the cluster jackknife.
  • An opt-in simulation study in tests/testthat/test-cluster-simulation.R (skipped unless VARIMPACT_RUN_SIMULATIONS=true):
    • Where every clustering variable stays within folds, null variables were flagged 0-9% of the time at the 5% level with id, against 12-73% without.
    • With site crossed with school and the default folds on school, a site-level null factor was flagged 13% of the time (36.5% without id): training and validation folds share sites, which biases it. With folds_by = "site" it was 3.0%.
    • The variance formula alone kept 92-95% interval coverage across crossed, one-way and dominant-cluster designs.
    • Reference results are at the end of the file.
  • Without id, results are bit-identical to master.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (dcc3a6d) to head (4be483d).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##            master       #72    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           31        32     +1     
  Lines         2301      2618   +317     
==========================================
+ Hits          2301      2618   +317     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The previous commit's editor re-indented nine existing files (function
bodies moved from two to four spaces, and blank lines picked up trailing
spaces), which made the diff about 4,000 lines and would have rewritten
git blame for vim-factors.R and vim-numerics.R. This keeps every code
change and puts it back on master's formatting: `git diff -w` against the
previous commit shows only DESCRIPTION and a restored blank line at the end
of R/estimate_tmle2.R.

Also restores RoxygenNote: 7.3.2 in DESCRIPTION, strips trailing whitespace
from the new files, adds the missing final newline to R/create_cv_folds.R,
and adds Pratik Khadse to Authors@R as a contributor for cluster-robust
standard errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
@ck37
ck37 force-pushed the claude/loving-brahmagupta-xdc7g1 branch 3 times, most recently from bde942d to cc0e5b1 Compare October 11, 2026 22:58
Variance. The clustered standard error was the average of per-fold
clustered variances. A fold holding a single cluster then had no variance,
and the variable vanished from the results (21 hospitals, one with 55% of
the rows, V = 2: results_all had no rows); and a cluster that spans folds,
as with crossed clustering variables, was split and its covariance dropped
(88% coverage for 95% intervals). cv_cluster_var() now computes one
clustered variance over every fold's influence curve, each scaled by
n / (V n_v) to match the average of fold estimates and re-centered at the
overall estimate, since centering within folds erases the between-cluster
variation of a fold with one cluster. Each cluster's sum is divided by
1 - h, its share of the weight, with a (G - 1) / G factor: the CR3
(jackknife-style) variance, equal to sandwich::vcovCL(type = "HC3",
cadjust = FALSE) for one-, two- and three-way clustering and to the usual
G / (G - 1) version when clusters are equal in size.

Degrees of freedom. Each variable's t reference uses the clusters behind
its estimate: in each fold, the fewer of the clusters holding its low and
its high level, summed over folds, minus one, capped at (fewest clusters -
1). It is shown in a df column of results_all. A variable whose low or high
level comes from a single cluster gets no interval or p-value, with a
message.

Folds. Every fold gets at least two clusters when there are enough, so a
large cluster no longer fills a fold alone; with clusters of similar size
the folds are unchanged. SuperLearner's grouped folds for g and g.Delta
stay stratified, built with the same grouped assignment, which now balances
whichever outcome value is rarer.

Other fixes:
- The macOS test failure: relabeled cluster ids reorder rowsum()'s sums,
  which changes the last bits without extended precision; the test now
  uses expect_equal().
- The "id is NULL" warning fires only for data$column or data[["column"]],
  not for a wrapper forwarding its own NULL default.
- A factor whose levels each fall within one cluster, such as the cluster
  id left in data, is set aside with a message instead of vanishing.
- apply_tmle_to_validation()'s row-number argument is renamed row_index.
- print() notes the clustering and the largest df; ?varimpact and the README
  have an example; the id docs note that learners' own internal folds
  ignore clusters.

Tests cover the new code to 100%, including the variance against
sandwich::vcovCL() and the cluster jackknife, continuous and missing
outcomes with id, and fold assignment with a dominant cluster.

The choices were made by simulation (varimpact() with id on clustered
data; false-positive rates for null variables at the 5% level):
- with id they ranged from 0% to 9%, against 12% to 73% without;
- CR3 cut a null covariate's rate from 7.6% to 4.6% when one cluster held
  half the rows, and left other designs within a point;
- the per-variable df kept cluster-level factors nearer 5% than one global
  df or a Welch-type df (6.7% and 6.0% against 7.7% and 8.5%), at the cost
  of being conservative with 15 clusters.
The variance formula alone covered 92% to 95% in every design, where the
per-fold average fell to 87% with crossed clustering.

The code comments and ?varimpact's references cite the literature behind
the variance across folds (the cross-validated TMLE's influence curve:
Zheng & van der Laan 2010; Hubbard, Kherad-Pajouh & van der Laan 2016),
the t reference, the CR3 variance and the degrees of freedom (Cameron,
Gelbach & Miller 2008, 2011; Cameron & Miller 2015; Bell & McCaffrey 2002;
Imbens & Kolesar 2016; MacKinnon, Nielsen & Webb 2023). They also say that
the per-variable df count is a simple, conservative heuristic rather than a
published rule.

With crossed clustering variables that cannot all stay within folds, the
fallback warning now names the variable whose clusters span folds and says
that training and validation folds then share them, which can bias
variables that vary with them; the folds_by docs and NEWS explain how to
choose the variable to fold on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
tests/testthat/test-cluster-simulation.R holds two simulation studies that
run only when VARIMPACT_RUN_SIMULATIONS=true, so CI and R CMD check skip
them:

- The full pipeline: clustered data with two null predictors (one varying
  within clusters, one constant within them) and one real effect, fitted
  with and without id across five designs (40 clusters, 15 clusters, one
  cluster holding 30% or 50% of the rows, no cluster effect on the outcome,
  and site crossed with school, also fitted with folds_by = "site").
  It checks that with id the null variables' false-positive rates are within
  Monte Carlo error of 5%. About an hour on 3 cores.
- The variance formula alone, without fitting varimpact(): 95% interval
  coverage for the average of per-fold variances, one variance centered
  within folds, and varimpact's cv_cluster_var(), across crossed, one-way
  and dominant-cluster designs. About a minute.

The reference results are recorded at the end of the file. With id, null
false-positive rates ranged from 0% to 9% (from 12% to 73% without), and
the variance formula covered 91.9% to 94.6% in every design.

The documentation of the clustered variance, the per-variable degrees of
freedom and the id argument points to this study.

The crossed design shows the limitation of folds that cannot keep every
clustering variable intact: folding on school (the default), a factor
constant within sites was flagged 13% of the time with id (36.5% without),
from bias rather than variance, since training and validation folds share
sites; folding on site brought it to 3.0%. The folds_by documentation
points to these results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
@ck37
ck37 force-pushed the claude/loving-brahmagupta-xdc7g1 branch from cc0e5b1 to 4be483d Compare October 11, 2026 23:23
@ck37
ck37 marked this pull request as ready for review October 11, 2026 23:42
@ck37
ck37 merged commit c21b4bb into master Oct 11, 2026
8 checks passed
ck37 pushed a commit that referenced this pull request Oct 12, 2026
Release version with today's date, and the NEWS heading for everything
since 1.2.4, including cluster-robust standard errors (#71, #72, #73).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
ck37 pushed a commit that referenced this pull request Oct 12, 2026
Release version with today's date, and the NEWS heading for the changes
recorded since NEWS.md was started in 2025, including cluster-robust
standard errors (#71, #72, #73).

The old "1.2.4" NEWS heading never matched a DESCRIPTION version, which
went from 1.2 to 1.3 in January 2017 and has been a 1.3.0 development
version since, so it now reads 1.3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UcpuhTnMKP8W19CLUUyU2T
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants