Skip to content

Add a top-level generic "solver" option with a Frank-Wolfe solver for the SCM weights - #117

Open
paulgp wants to merge 1 commit into
ebenmichael:masterfrom
paulgp:frank-wolfe-solver
Open

Add a top-level generic "solver" option with a Frank-Wolfe solver for the SCM weights#117
paulgp wants to merge 1 commit into
ebenmichael:masterfrom
paulgp:frank-wolfe-solver

Conversation

@paulgp

@paulgp paulgp commented Jul 20, 2026

Copy link
Copy Markdown

Following up on our email exchange — and to answer your question directly: yes, it's memory (and time) with large donor pools. synth_qp() forms the dense n0 x n0 donor Gram matrix before handing it to OSQP: 200 MB at 5,000 donors, 3.2 GB at 20,000. In our financial applications (S&P index events against an all-CRSP donor pool) each event has 4,000–7,000 donors and there are hundreds of events, so the Gram construction dominates everything.

What this PR does

1. A top-level, generic solver option (your first ask). single_augsynth() gains solver = "osqp" — the default is the existing behavior, with the current synth_qp body moved verbatim to R/solvers.R as synth_qp_osqp(). Solvers are plain functions with signature (X1, X0, V) -> weights, resolved by get_synth_solver(); solver also accepts such a function directly, so anyone can plug in a different solver without touching the package:

single_augsynth(..., solver = function(X1, X0, V) my_solver(X1, X0, V))

The option threads through every place single-treated SCM weights are solved — fit_synth_formatted(), the ridge base weights, the ridge lambda CV refits (cv_lambda/get_lambda_errors), and fit_augsyn() — and it lands in extra_args, so the jackknife/conformal/permutation/cv refits automatically reuse it. augsynth_multiout inherits it through fit_augsynth_internal. multisynth is untouched, and the augsynth() dispatcher warns that solver is ignored there (mirroring the existing progfunc warning), per your note about the bespoke staggered-adoption optimization.

2. The "frank_wolfe" solver. Frank–Wolfe with exact line search (the synthdid algorithm — Arkhangelsky, Athey, Hirshberg, Imbens & Wager 2021, dual BSD-3/GPL≥2; implementation written for this PR) to identify the sparse active donor set, then the QP solved exactly on that support (OSQP on a problem of at most 5·t0 donors), plus a KKT gradient screen that re-admits any donors the support restriction missed. Pure FW alone has a sublinear tail and is not accurate enough — the support-restricted polish is what makes the two solvers agree tightly. The full Gram matrix is never formed, so memory stays O(n0·t0):

n0 (t0 = 100) osqp frank_wolfe rel. objective gap Gram size
1,000 3.0s 0.7s −1.8e−08 (fw better) 8 MB
5,000 14.6s 3.9s +5.3e−04 200 MB
20,000 5.5s (osqp not run) 3.2 GB

3. Unit tests (your second ask). tests/testthat/test_solvers.R checks, on the basque fixture and a simulated QP: solver agreement on the raw QP objective; SCM without ridge augmentation (imbalance and ATT path); ridge augmentation with a fixed lambda; ridge with CV lambda (the CV refits run through the requested solver and select the same lambda); a fixedeff = TRUE variant; that a custom solver function reproduces the default exactly; and that inference refits remember the solver via extra_args. All pass.

Test status

On my machine a fresh clone of current master has 19 failing tests (dependency-version drift / missing optional Suggests, none related to this area). I ran the full suite against stock master and against this branch into separate libraries and diffed the failing sets: they are identical — this PR introduces zero new failures and its 16 new assertions all pass.

Default behavior is unchanged: solver = "osqp" runs the exact code that ran before.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S4N6wSPuTLbSVDtsnEafKT

… the SCM weights

`solver = "osqp"` (default) is the existing behavior, with synth_qp's
body moved unchanged to R/solvers.R. Solvers are plain functions
(X1, X0, V) -> weights resolved by get_synth_solver(), and `solver`
also accepts such a function directly, so new solvers can be plugged
in without touching the package.

`solver = "frank_wolfe"` runs Frank-Wolfe with exact line search (the
synthdid algorithm; Arkhangelsky et al. 2021, dual BSD-3/GPL>=2) to
identify the active donor set, solves the QP exactly on that support
(OSQP on a problem of at most 5*t0 donors), and verifies KKT
optimality with a full gradient screen that re-admits missed donors.
The n0 x n0 donor Gram matrix is never formed, so memory stays
O(n0 t0): at 20,000 donors the Gram alone is 3.2 GB while the
Frank-Wolfe path runs in ~5s.

The option threads through everywhere single-treated SCM weights are
solved (fit_synth_formatted, the ridge base weights, the ridge lambda
CV refits, and fit_augsyn), lands in extra_args so inference /
permutation / cv refits reuse it, and is inherited by
augsynth_multiout; multisynth is untouched and warns that the option
is ignored. tests/testthat/test_solvers.R checks solver agreement on
the raw QP, SCM with and without ridge augmentation (fixed and CV
lambda), with a fixed effect, the custom-function hook, and that
refits remember the solver.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4N6wSPuTLbSVDtsnEafKT
@paulgp paulgp changed the title Add a top-level generic option with a Frank-Wolfe solver for the SCM weights Add a top-level generic "solver" option with a Frank-Wolfe solver for the SCM weights Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant