lmackey.github.io/research.html at main · lmackey/lmackey.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />
    <title>Lester Mackey: Research</title>
    <meta name="keywords" content="lester mackey, computer science, machine learning, statistics, social good, high-dimensional, probability, concentration, research, stanford, microsoft research" />
    <meta name="description" content="Lester Mackey's website" />
    <link rel="stylesheet" type="text/css" href="default.css" />

    <!-- Google analytics tracking code -->
    <script type="text/javascript">

      var _gaq = _gaq || [];
      _gaq.push(['_setAccount', 'UA-9971253-1']);
      _gaq.push(['_trackPageview']);

      (function() {
      var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
      ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'https://www') + '.google-analytics.com/ga.js';
      var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
      })();

    </script>
  </head>
  <body>
    <div id="upbg"></div>
    <div id="outer">
      <div id="header">
	<div id="headercontent">
	  <h1>Lester Mackey</h1>
    <h2>&#108;&#119;&#109;&#97;&#99;&#107;&#101;&#121;&#64;&#103;&#109;&#97;&#105;&#108;&#46;&#99;&#111;&#109;</h2>
	  <!--<h2>Personal homepage</h2>-->
	</div>
      </div>

      <div id="headerpic"></div>

      <div id="menu">
	<ul>
	  <li><a href="index.html">Home</a></li>
	  <li><a href="research.html" class="active">Research</a></li>
    <li><a href="talks.html">Talks</a></li>
	  <li><a href="teaching.html">Teaching</a></li>
	  <li><a href="dinosaurplanet.html">Netflix Prize</a></li>
	  <li><a href="alsprize4life.html">ALS Prize4Life</a></li>
	  <li><a href="forecastrodeo.html">Forecast Rodeo</a></li>
	  <li><a href="aiweatherquest.html">AI Weather Quest</a></li>
	  <li><a href="contact.html">Contact</a></li>
	  <li><a href="bio.html">Bio</a></li>
	  <li><a href="papers/Lester_Mackey-CV.pdf">CV</a></li>
	</ul>
      </div>
      <div id="menubottom"></div>

      <div id="content">
	<div id="normalcontent">
      <!--
	  <h4>Research Interests</h4>
	  <div class="contentarea">
	    <ul class="normal">
	      <li><b>Statistical machine learning</b></li>
          <li><b>Scalable algorithms</b></li>
	      <li><b>High-dimensional statistics</b></li>
          <li><b>Approximate inference</b></li>
	      <li><b>Probability</b></li>
	    </ul>
	  </div>
	  <div class="divider1"></div>
       -->
	  <h4>In the Pipeline</h4>
	  <div class="contentarea">
	    <ul class="normal">
        <li><b>Universal Bounds for Self-Normalized Kullback-Leibler Estimation.  (<a href="https://arxiv.org/pdf/2610.12851">arxiv</a>)</b>
          <br /><i>Buu Phan, Lester Mackey, and David Alvarez-Melis.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#snkl">View details &raquo</a>
              <!--colapsed text-->
              <div id="snkl" class="collapse">
              <p></p>
              <p>
              This work studies an entropy-based estimator of Kullback-Leibler (KL) divergence constructed from normalized importance weights. We prove a universal finite-sample bound showing that its expectation can exceed the target-proposal KL divergence by at most log 2 for any sample size. We then characterize the KL divergence between the distribution induced by importance-weighted resampling and the proposal, showing that it is upper bounded by both the expectation of the entropy-based estimator and the target-proposal KL divergence. Importantly, these bounds require neither bounded importance weights nor finite higher moments. We apply these results to test-time alignment of generative models via Soft Best-of-N sampling, an importance-sampling-based scaling technique proposed by Verdun et al. (2025), and to relative entropy coding, obtaining finite-sample policy-divergence certificates and moment-free bounds on coding cost.
              </p>
              </div>
              <p />
              </li>
        </li>
        <li><b>KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning.  (<a href="https://arxiv.org/pdf/2512.19605">arxiv</a>)</b>
          <br /><i>Eric Zimmermann, Harley Wiltzer, Justin Szeto, David Alvarez-Melis, and Lester Mackey.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#kerjepa">View details &raquo</a>
              <!--colapsed text-->
              <div id="kerjepa" class="collapse">
              <p></p>
              <p>
              Breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of Euclidean joint embedding self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the SIGReg Epps-Pulley regularizer which approximates a sliced Maximum Mean Discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced discrepancies, we develop alternative methods with improved training stability and design flexibility and leverage those techniques to train strong vision learners at scale.
              </p>
              </div>
              <p />
              </li>
        <li><b>Probabilistic Inference and Learning with Stein's Method.  (<a href="https://arxiv.org/pdf/2603.07467">arxiv</a>)</b>
          <br /><i>Qiang Liu, Lester Mackey, and Chris Oates.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#stein">View details &raquo</a>
              <!--colapsed text-->
              <div id="stein" class="collapse">
              <p></p>
              <p>
              This monograph provides a rigorous overview of theoretical and methodological aspects of probabilistic inference and learning with Stein's method. Recipes are provided for constructing Stein discrepancies from Stein operators and Stein sets, and properties of these discrepancies such as computability, separation, convergence detection, and convergence control are discussed. Further, the connection between Stein operators and Stein variational gradient descent is set out in detail. The main definitions and results are precisely stated, and references to all proofs are provided.
              </p>
              </div>
              <p />
              </li>
        <li><b>Estimating Treatment Effects with Independent Component Analysis.  (<a href="https://arxiv.org/pdf/2507.16467">arxiv</a>, <a href="https://github.com/rpatrik96/ica_causal_effect">code</a>)</b>
          <br /><i>Patrik Reizinger, Lester Mackey, Wieland Brendel, and Rahul Krishnan.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#ica">View details &raquo</a>
              <!--colapsed text-->
              <div id="ica" class="collapse">
              <p></p>
              <p>
              Independent Component Analysis (ICA) uses a measure of non-Gaussianity to identify latent sources from data and estimate their mixing coefficients (Shimizu et al., 2006). Meanwhile, higher-order Orthogonal Machine Learning (OML) exploits non-Gaussian treatment noise to provide more accurate estimates of treatment effects in the presence of confounding nuisance effects (Mackey et al., 2018). Remarkably, we find that the two approaches rely on the same moment conditions for consistent estimation. We then seize upon this connection to show how ICA can be effectively used for treatment effect estimation. Specifically, we prove that linear ICA can consistently estimate multiple treatment effects, even in the presence of Gaussian confounders, and identify regimes in which ICA is provably more sample-efficient than OML for treatment effect estimation. Our synthetic demand estimation experiments confirm this theory and demonstrate that linear ICA can accurately estimate treatment effects even in the presence of nonlinear nuisance.
              </p>
              </div>
              <p />
              </li>
        <li><b>Reflections from the Workshop on AI-Assisted Decision Making for Conservation.  (<a href="https://arxiv.org/pdf/2307.08774.pdf">arxiv</a>)</b>
          <br /><i>Lily Xu, Esther Rolf, Sara Beery, Joseph R. Bennett, Tanya Berger-Wolf, Tanya Birch, Elizabeth Bondi-Kelly, Justin Brashares, Melissa Chapman, Anthony Corso, Andrew Davies, Nikhil Garg, Angela Gaylard, Robert Heilmayr, Hannah Kerner, Konstantin Klemmer, Vipin Kumar, Lester Mackey, Claire Monteleoni, Paul Moorcroft, Jonathan Palmer, Andrew Perrault, David Thau, and Milind Tambe.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#aiconservation">View details &raquo</a>
              <!--colapsed text-->
              <div id="aiconservation" class="collapse">
              <p></p>
              <p>
              In this white paper, we synthesize key points made during presentations and discussions from the AI-Assisted Decision Making for Conservation workshop, hosted by the Center for Research on Computation and Society at Harvard University on October 20-21, 2022. We identify key open research questions in resource allocation, planning, and interventions for biodiversity conservation, highlighting conservation challenges that not only require AI solutions, but also require novel methodological advances. In addition to providing a summary of the workshop talks and discussions, we hope this document serves as a call-to-action to orient the expansion of algorithmic decision-making approaches to prioritize real-world conservation challenges, through collaborative efforts of ecologists, conservation decision-makers, and AI researchers.
              </p>
        </div>
        <p />
        </li>
        <li><b>Budget-Constrained Bounds for Mini-Batch Estimation of Optimal Transport.  (<a href="https://arxiv.org/pdf/2210.13630.pdf">arxiv</a>)</b>
          <br /><i>David Alvarez-Melis, Nicolo Fusi, Lester Mackey, and Tal Wagner.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#mbot">View details &raquo</a>
              <!--colapsed text-->
              <div id="mbot" class="collapse">
              <p></p>
              <p>
              Optimal Transport (OT) is a fundamental tool for comparing probability distributions, but its exact computation remains prohibitive for large datasets. In this work, we introduce novel families of upper and lower bounds for the OT problem constructed by aggregating solutions of mini-batch OT problems. The upper bound family contains traditional minibatch averaging at one extreme and a tight bound found by optimal coupling of mini-batches at the other. In between these extremes, we propose various methods to construct bounds based on a fixed computational budget. Through various experiments, we explore the tradeoff between computational budget and bound tightness and show the usefulness of these bounds in computer vision applications.
              </p>
        </div>
        <p />
        </li>
        <li><b>Weighted Meta-Learning. (<a href="https://arxiv.org/pdf/2003.09465.pdf">arxiv</a>)</b>
          <br /><i>Diana Cai, Rishit Sheth, Lester Mackey, and Nicolo Fusi.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#alphamaml">View details &raquo</a>
              <!--colapsed text-->
              <div id="alphamaml" class="collapse">
              <p></p>
              <p>
              Meta-learning leverages related source tasks to learn an initialization that can be quickly fine-tuned to a target task with limited labeled examples. However, many popular meta-learning algorithms, such as model-agnostic meta-learning (MAML), only assume access to the target samples for fine-tuning. In this work, we provide a general framework for meta-learning based on weighting the loss of different source tasks, where the weights are allowed to depend on the target samples. In this general setting, we provide upper bounds on the distance of the weighted empirical risk of the source tasks and expected target risk in terms of an integral probability metric (IPM) and Rademacher complexity, which apply to a number of meta-learning settings including MAML and a weighted MAML variant. We then develop a learning algorithm based on minimizing the error bound with respect to an empirical IPM, including a weighted MAML algorithm, &alpha;-MAML. Finally, we demonstrate empirically on several regression problems that our weighted meta-learning algorithm is able to find better initializations than uniformly-weighted meta-learning algorithms, such as MAML.
              </p>
              </div>
              <p />
        </li>
        <li><b>Teacher-Student Compression with Generative Adversarial Networks. (<a href="https://arxiv.org/pdf/1812.02271.pdf">arxiv</a>, <a href="https://github.com/RuishanLiu/GAN-TSC">GAN-TSC code</a>, <a href="https://github.com/RuishanLiu/GAN-TSC-Score">TSC Score code</a>)</b>
          <br /><i>Ruishan Liu, Nicolo Fusi, and Lester Mackey.</i>
          <br/><a class="btn" data-toggle="collapse" data-target="#ganmc">View details &raquo</a>
              <!--colapsed text-->
              <div id="ganmc" class="collapse">
              <p></p>
              <p>
              More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric---the TSC Score---to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.
              </p>
              </div>
              <p />
        </li>
	    </ul>
	  </div>
	  <div class="divider1"></div>
	  <h4>Publications</h4>
	  <div class="contentarea">
	    <ul class="normal">
        <li><b>Advancing Subseasonal Forecasting with Machine Learning. (<a href="https://arxiv.org/pdf/2604.16238">arxiv</a>, <a href="https://www.nature.com/articles/s41586-026-11126-x">nature</a>, <a href="https://www.nature.com/articles/d41586-026-03077-0">briefing</a>, <a href="https://github.com/mouatadid/pbc">code</a>, <a href="https://www.microsoft.com/en-us/research/story/new-ai-framework-bridges-a-critical-gap-in-mid-range-weather-forecasts/">blog</a>, <a href="https://www.microsoft.com/en-us/research/project/subseasonal-climate-forecasting/">website</a>, <a href="https://youtu.be/7Qv1RTPm2-0?si=tVbpCxxv01Fh6JXY&t=1690">Hannah's talk</a>)</b>
          <br /><i>Hannah Guan, Soukayna Mouatadid, Paulo Orenstein, Judah Cohen, Haiyu Dong, Zekun Ni, Jeremy Berman, Genevieve Flaspohler, Alex Lu, Jakob Schloer, Joshua Talib, Jonathan A. Weyn, and Lester Mackey.</i>
          <br />Nature. October 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#pbc">View details &raquo</a>
              <!--colapsed text-->
              <div id="pbc" class="collapse">
              <p></p>
              <p>
              Decision-makers rely on weather forecasts to plant crops, manage wildfires, allocate water and energy, and prepare for weather extremes. Today, such forecasts enjoy unprecedented accuracy out to two weeks thanks to steady advances in physics-based dynamical models and data-driven artificial intelligence (AI) models. However, model skill drops precipitously at subseasonal timescales (2 - 6 weeks ahead), due to compounding errors, systemic model biases, and the chaotic nature of the atmosphere. To counter this degradation, we introduce probabilistic bias correction (PBC), a machine learning framework that substantially reduces systematic error by learning to correct historical probabilistic forecasts. When applied to the leading dynamical and AI models from the European Centre for Medium-Range Weather Forecasts (ECMWF), PBC doubles the modest subseasonal skill of the AI Forecasting System and improves the skill of the operationally-debiased dynamical model for 91% of pressure, 92% of temperature, and 98% of precipitation targets. We designed PBC for operational deployment, and, in ECMWF's 2025 real-time forecasting competition, its global forecasts placed first for all weather variables and lead times, outperforming the dynamical models from six operational forecasting centers, an international dynamical multi-model ensemble, ECMWF's AI Forecasting System, and the forecasting systems of 34 teams worldwide. These probabilistic skill gains translate into more accurate prediction of extreme events and have the potential to improve agricultural planning, energy management, and disaster preparedness in vulnerable communities.
              </p>
              </div>
              <p />
              </li>
        <li><b>Express Language Modeling. (<a href="https://arxiv.org/pdf/2606.10944">arxiv</a>, <a href="https://github.com/microsoft/thinformer">code</a>)</b>
          <br /><i>Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi, and Lester Mackey.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2026.
          <br /><small><b>COLM 2026 Workshop on Efficient Reasoning Best Paper Award.</b></small>
          <br/><a class="btn" data-toggle="collapse" data-target="#express">View details &raquo</a>
              <!--colapsed text-->
              <div id="express" class="collapse">
              <p></p>
              <p>
              We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering log<sup>3/2</sup>(n)/s approximation error with only O(s) memory and O(s<sup>2</sup>log<sup>2</sup>(n)) compression overhead for a sequence of length n. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.
              </p>
              </div>
              <p />
              </li>
        <li><b>Cheap Permutation Testing.  (<a href="https://arxiv.org/pdf/2502.07672">arxiv</a>, <a href="https://github.com/microsoft/cheap-permutations">code</a>)</b>
          <br /><i>Carles Domingo-Enrich, Raaz Dwivedi, and Lester Mackey.</i>
          <br />Annals of Statistics. To appear.
          <br/><a class="btn" data-toggle="collapse" data-target="#cheaper">View details &raquo</a>
              <!--colapsed text-->
              <div id="cheaper" class="collapse">
              <p></p>
              <p>
              Permutation tests are a popular choice for distinguishing distributions and testing independence, due to their exact, finite-sample control of false positives and their minimax optimality when paired with U-statistics. However, standard permutation tests are also expensive, requiring a test statistic to be computed hundreds or thousands of times to detect a separation between distributions. In this work, we offer a simple approach to accelerate testing: group your datapoints into bins and permute only those bins. For U and V-statistics, we prove that these cheap permutation tests have two remarkable properties. First, by storing appropriate sufficient statistics, a cheap test can be run in time comparable to evaluating a single test statistic. Second, cheap permutation power closely approximates standard permutation power. As a result, cheap tests inherit the exact false positive control and minimax optimality of standard permutation tests while running in a fraction of the time. We complement these findings with improved power guarantees for standard permutation testing and experiments demonstrating the benefits of cheap permutations over standard maximum mean discrepancy (MMD), Hilbert-Schmidt independence criterion (HSIC), random Fourier feature, Wilcoxon-Mann-Whitney, aggregated MMD, cross-MMD, and cross-HSIC tests.
              </p>
              </div>
              <p />
              </li>
        <li><b>Domain-Aware Scaling Laws Uncover Data Synergy.  (<a href="https://data-syn.github.io/">website</a>, <a href="https://arxiv.org/pdf/2607.11052">arxiv</a>, <a href="https://github.com/kimiah/DataSynergy">code</a>)</b>
        <br /><i>Kimia Hamidieh, David Alvarez-Melis, and Lester Mackey.</i>
        <br />Conference on Language Modeling (COLM). October 2026.
        <br/><a class="btn" data-toggle="collapse" data-target="#datasynergy">View details &raquo</a>
            <!--colapsed text-->
            <div id="datasynergy" class="collapse">
            <p></p>
            <p>
            Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential. Empirical findings repeatedly show that combining datasets from different domains yields nontrivial interactions. For instance, adding code improves mathematical reasoning, while certain mixtures introduce interference that reduces model performance. We refer to these effects collectively as data synergy, where the contribution of multiple domains exceeds or falls short of the sum of their isolated contributions. In this work, we formalize and quantify data synergy in language model pretraining. Leveraging observational variation across open-weight LLMs with diverse pretraining mixtures, we estimate both direct domain-to-benchmark synergy (how one domain contributes to performance on another) and a second-order domain-domain synergy (capabilities that require co-occurrence of multiple domains). Our framework improves predictive accuracy over domain-agnostic scaling laws and recovers stable synergy estimates. We validate these estimates by training models on predicted optimal and predicted anti-optimal mixtures and confirm that our synergy estimates correctly predict performance rankings.
            </p>
            </div>
            <p />
        </li>
        <li><b>A sequence of odd events.  (<a href="https://www.sciencenews.org/article/math-puzzle-2026-august-odd-family">puzzle</a>)</b>
        <br /><i>Lilly Fang, Lester Mackey, Noah Mackey, and Isaiah Mackey.</i>
        <br />Science News. July 2026.
            <p />
        </li>
        <li><b>Earth Embeddings: Towards AI-centric Representations of our Planet.  (<a href="https://eartharxiv.org/repository/view/11083/">eartharxiv</a>)</b>
        <br /><i>Konstantin Klemmer, Esther Rolf, Marc Russwurm, Gustau Camps-Valls, Mikolaj Czerkawski, Stefano Ermon, Alistair Francis, Nathan Jacobs, Hannah Rae Kerner, Lester Mackey, Gengchen Mai, Oisin Mac Aodha, Markus Reichstein, Caleb Robinson, David Rolnick, Evan Shelhamer, Vincent Sitzmann, Devis Tuia, and Xiaoxiang Zhu.</i>
        <br />IEEE Geoscience and Remote Sensing Magazine. To appear.
        <br/><a class="btn" data-toggle="collapse" data-target="#earthembeddings">View details &raquo</a>
            <!--colapsed text-->
            <div id="earthembeddings" class="collapse">
            <p></p>
            <p>
            This paper presents a new perspective for the flexible and efficient representation of geospatial data, tailored to and empowered by AI: Earth embeddings. Earth embeddings provide a unified and accessible vector representation of local geographic characteristics. They fuse different geospatial data sources across time and space, compress highly-correlated raw geospatial data into one dense representation, can be used to guide interpolation between data observations, and can serve as a universal location token for foundation models. This provides a powerful alternative to existing geospatial workflows that rely on heterogeneous data, hard-to-acquire expertise, and significant computation by the user: embeddings instead provide convenient representations, easily adaptable for numerous downstream tasks. We posit that Earth embeddings redefine geospatial analytics, transforming it from fragmented, task-specific modeling into a coherent, generalizable framework for AI. We approach this from both the users' and developers' perspectives, outlining a path for how the rapidly developing technology of Earth embeddings can reshape the way we store, represent, and use geospatial data, evidenced by recent research charting initial directions. We call on Earth embedding users and developers to align methodological and applied development and deployment within an interdisciplinary, open-source oriented research community.
            </p>
            </div>
            <p />
        </li>
        <li><b>Efficient Concentration with Gaussian Approximation.  (<a href="https://arxiv.org/pdf/2208.09922.pdf">arxiv</a>, <a href="https://github.com/lmackey/gauss_conc">code</a>)</b>
          <br /><i>Morgane Austern and Lester Mackey.</i>
          <br />Annals of Applied Probability. To appear.
          <br/><a class="btn" data-toggle="collapse" data-target="#gaussconc">View details &raquo</a>
              <!--colapsed text-->
              <div id="gaussconc" class="collapse">
              <p></p>
              <p>
                Concentration inequalities for the sample mean, like those due to Bernstein, Hoeffding, and Bentkus, are valid for any sample size but overly conservative, yielding confidence intervals that are unnecessarily wide. The central limit theorem (CLT) provides asymptotic confidence intervals with optimal width, but these are invalid for all sample sizes. To resolve this tension, we develop new computable concentration inequalities for bounded variables with asymptotically optimal size, finite-sample validity, and sub-Gaussian decay. These bounds enable the construction of efficient confidence intervals with correct coverage for any sample size and efficient empirical Berry-Esseen bounds that require no prior knowledge of the population variance. We derive our inequalities by tightly bounding non-uniform Kolmogorov and Wasserstein distances to a Gaussian using zero-bias couplings and Stein's method of exchangeable pairs and demonstrate practical improvements over the Bernstein, Hoeffding, Bentkus, Berry-Esseen, Feller-Cramér, Romano-Wolf, empirical Bernstein, empirical Bentkus, and coin-betting inequalities.
              </p>
              </div>
              <p />
        </li>
        <li><b>Bounding Hellinger Distance with Stein's Method.  (<a href="https://arxiv.org/pdf/2411.04322">arxiv</a>)</b>
          <br /><i>Morgane Austern and Lester Mackey.</i>
          <br />Electronic Communications in Probability. August 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#hellinger">View details &raquo</a>
              <!--colapsed text-->
              <div id="hellinger" class="collapse">
              <p></p>
              <p>
              This work introduces a new, explicit bound on the Hellinger distance between a continuous random variable and a Gaussian with matching mean and variance. As example applications, we derive a quantitative Hellinger central limit theorem and efficient concentration inequalities for U-statistics.
              </p>
              </div>
              <p />
              </li>
        <li><b>Integrating chemical structures as treatments improves representations of microscopy images for morphological profiling.  (<a href="https://arxiv.org/pdf/2504.09544">arxiv</a>, <a href="https://github.com/microsoft/micon">code</a>)</b>
          <br /><i>Yemin Yu, Neil Tenenholtz, Lester Mackey, Ying Wei, David Alvarez-Melis, Ava P. Amini, and Alex X. Lu.</i>
          <br />PLOS Computational Biology. July 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#micon">View details &raquo</a>
              <!--colapsed text-->
              <div id="micon" class="collapse">
              <p></p>
              <p>
              Recent advances in self-supervised deep learning have improved our ability to quantify cellular morphological changes in high-throughput microscopy screens, a process known as morphological profiling. However, most current methods only learn from images, despite many screens being inherently multimodal, as they involve both a chemical or genetic perturbation as well as an image-based readout. We hypothesized that incorporating chemical compound structures during self-supervised pre-training could improve learned representations of images from high-throughput microscopy screens. We introduce a representation learning framework, MICON (Molecular-Image Contrastive Learning), that models chemical compounds as treatments that induce transformations of cell phenotypes. MICON significantly outperforms classical hand-crafted features such as CellProfiler and existing deep-learning-based representation learning methods in challenging evaluation settings where models must identify reproducible effects of drugs across independent replicates and data-generating centers. We demonstrate that incorporating chemical compound information into the learning process provides small, but consistent improvements in performance and that modeling compounds specifically as treatments outperforms approaches that directly align images and compounds in a single representation space. Our findings point to a new direction for representation learning in morphological profiling, suggesting that methods should explicitly account for the multimodal nature of microscopy screening data.
              </p>
              </div>
              <p />
              </li>
        <li><b>WildCat: Near-Linear Attention in Theory and Practice.  (<a href="https://www.arxiv.org/pdf/2602.10056">arxiv</a>, <a href="https://github.com/microsoft/wildcat">code</a>)</b>
          <br /><i>Tobias Schr&ouml;der and Lester Mackey.</i>
          <br />International Conference on Machine Learning (ICML). July 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#wildcat">View details &raquo</a>
              <!--colapsed text-->
              <div id="wildcat" class="collapse">
              <p></p>
              <p>
              We introduce WildCat, a high-accuracy, low-cost approach to compressing the attention mechanism in neural networks. While attention is a staple of modern network architectures, it is also notoriously expensive to deploy due to resource requirements that scale quadratically with the input sequence length n. WildCat avoids these quadratic costs by only attending over a small weighted coreset. Crucially, we select the coreset using a fast but spectrally-accurate subsampling algorithm -- randomly pivoted Cholesky -- and weight the elements optimally to minimise reconstruction error. Remarkably, given bounded inputs, WildCat approximates exact attention with super-polynomial
              <!--$O(n^{-\sqrt{\log(\log(n))}})$-->
              <math display="inline">
              <mi>O</mi>
              <mo stretchy="false">(</mo>
              <msup>
                <mi>n</mi>
                <mrow>
                  <mo>-</mo>
                  <msqrt>
                    <mrow>
                      <mi>log(log(n))</mi>
                    </mrow>
                  </msqrt>
                </mrow>
              </msup>
              <mo stretchy="false">)</mo>
            </math>
            error decay while running in near-linear
            <!--$O(n^{1+o(1)})$-->
            <math display="inline">
              <mi>O</mi>
              <mo stretchy="false">(</mo>
              <msup>
                <mi>n</mi>
                <mrow>
                  <mi>1+o(1)</mi>
                </mrow>
              </msup>
              <mo stretchy="false">)</mo>
            </math>
            time. In contrast, prior practical approximations either lack error guarantees or require quadratic runtime to guarantee such high fidelity. We couple this advance with a GPU-optimized PyTorch implementation and a suite of benchmark experiments demonstrating the benefits of WildCat for image generation, image classification, and language model KV cache compression.
              </p>
              </div>
              <p />
              </li>
        <li><b>Thinned Mean Field Langevin Dynamics.  (<a href="https://arxiv.org/pdf/2605.28589">arxiv</a>, <a href="https://github.com/hudsonchen/thinned_mfld">code</a>)</b>
          <br /><i>Zonghao Chen, Heishiro Kanagawa, François-Xavier Briol, Chris J. Oates, and Lester Mackey.</i>
          <br />International Conference on Machine Learning (ICML). July 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#ktmfld">View details &raquo</a>
              <!--colapsed text-->
              <div id="ktmfld" class="collapse">
              <p></p>
              <p>
              Several important learning tasks can be formulated as minimizing an entropy-regularized objective over an appropriate space of probability distributions.
              Mean-field Langevin dynamics (MFLD) facilitate computation in this general context, casting the minimizer as the invariant distribution of a McKean-Vlasov process,
              which can be numerically discretized using N particles and thus simulated.
              However, simulating this interacting particle system has computational complexity of order N^2.
              Motivated by recent research into kernel thinning, we propose KT-MFLD, in which each particle interacts only with a thinned particle coreset of size N^{1/2}.
              KT-MFLD thus reduces the computational complexity to order N^{3/2} while, under mild regularity conditions, achieving the same convergence guarantees (up to logarithmic factors) as MFLD.
              Our theoretical analysis is empirically confirmed on tasks including the training of student-teacher neural networks, quantization with maximum mean discrepancy,
              and computation of predictively-oriented posteriors in a post-Bayesian framework.
              </p>
              </div>
              <p />
              </li>
        <li><b>The Relative Instability of Model Comparison with Cross-validation.  (<a href="https://arxiv.org/pdf/2508.04409">arxiv</a>, <a href="https://github.com/alexandre-bayle/ricv">code</a>)</b>
          <br /><i>Alexandre Bayle, Lucas Janson, and Lester Mackey.</i>
          <br />International Conference on Machine Learning (ICML). July 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#ricv">View details &raquo</a>
              <!--colapsed text-->
              <div id="ricv" class="collapse">
              <p></p>
              <p>
              Cross-validation (CV) is known to provide asymptotically exact tests and confidence intervals for model improvement but only when the model comparison is relatively stable. Surprisingly, we prove that even simple, individually stable models can generate relatively unstable comparisons, calling into question the validity of CV inference. Specifically, we show that the Lasso and its close cousin, soft-thresholding, generate relatively unstable comparisons and invalid CV inferences, even in the most favorable of learning settings and even when both models are individually stable. These findings highlight the importance of verifying relative stability before deploying CV for model comparison.
              </p>
              </div>
              <p />
              </li>
          <li><b>Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner.  (<a href="https://www.arxiv.org/pdf/2510.03206">arxiv</a>, <a href="https://github.com/zhouc20/CCDD">code</a>)</b>
          <br /><i>Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang.</i>
          <br />International Conference on Machine Learning (ICML). July 2026.
          <br/><a class="btn" data-toggle="collapse" data-target="#ccdd">View details &raquo</a>
              <!--colapsed text-->
              <div id="ccdd" class="collapse">
              <p></p>
              <p>
              Diffusion language models, especially masked discrete diffusion models, have achieved great success recently. While there are some theoretical and primary empirical results showing the advantages of latent reasoning with looped transformers or continuous chain-of-thoughts, continuous diffusion models typically underperform their discrete counterparts. In this paper, we argue that diffusion language models do not necessarily need to be in the discrete space. In particular, we prove that continuous diffusion models have stronger expressivity than discrete diffusions and looped transformers. We attribute the contradiction between the theoretical expressiveness and empirical performance to their practical trainability: while continuous diffusion provides intermediate supervision that looped transformers lack, they introduce additional difficulty decoding tokens into the discrete token space from the continuous representation space. We therefore propose Coevolutionary Continuous Discrete Diffusion (CCDD), which defines a joint multimodal diffusion process on the union of a continuous representation space and a discrete token space, leveraging a single model to simultaneously denoise in the joint space. By combining two modalities, CCDD is expressive with rich semantics in the latent space, as well as good trainability and sample quality with the help of explicit discrete tokens. We also propose effective architectures and advanced training/sampling techniques for CCDD, which reveals strong empirical performance in extensive language modeling experiments on real-world tasks.
              </p>
              </div>
              <p />
              </li>
        <li><b>It's Hard to Be Normal: The Impact of Noise on Structure-agnostic Estimation.  (<a href="https://arxiv.org/pdf/2507.02275">arxiv</a>, <a href="https://github.com/JikaiJin/ACE">code</a>)</b>
          <br /><i>Jikai Jin, Lester Mackey, and Vasilis Syrgkanis.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#ace">View details &raquo</a>
              <!--colapsed text-->
              <div id="ace" class="collapse">
              <p></p>
              <p>
              Structure-agnostic causal inference studies how well one can estimate a treatment effect given black-box machine learning estimates of nuisance functions (like the impact of confounders on treatment and outcomes). Here, we find that the answer depends in a surprising way on the distribution of the treatment noise. Focusing on the partially linear model of Robinson (1988), we first show that the widely adopted double machine learning (DML) estimator is minimax rate-optimal for Gaussian treatment noise, resolving an open problem of Mackey et al. (2018). Meanwhile, for independent non-Gaussian treatment noise, we show that DML is always suboptimal by constructing new practical procedures with higher-order robustness to nuisance errors. These ACE procedures use structure-agnostic cumulant estimators to achieve r-th order insensitivity to nuisance errors whenever the (r+1)-st treatment cumulant is non-zero. We complement these core results with novel minimax guarantees for binary treatments in the partially linear model. Finally, using synthetic demand estimation experiments, we demonstrate the practical benefits of our higher-order robust estimators.
              </p>
              </div>
        <p />
        </li>
        <li><b>Informed Correctors for Discrete Diffusion Models.  (<a href="https://arxiv.org/pdf/2407.21243.pdf">arxiv</a>, <a href="https://github.com/lindermanlab/informed-correctors">code</a>)</b>
          <br /><i>Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#informed">View details &raquo</a>
              <!--colapsed text-->
              <div id="informed" class="collapse">
              <p></p>
              <p>
                Discrete diffusion has emerged as a powerful framework for generative modeling in discrete domains, yet efficiently sampling from these models remains challenging. Existing sampling strategies often struggle to balance computation and sample quality when the number of sampling steps is reduced, even when the model has learned the data distribution well. To address these limitations, we propose a predictor-corrector sampling scheme where the corrector is informed by the diffusion model to more reliably counter the accumulating approximation errors. To further enhance the effectiveness of our informed corrector, we introduce complementary architectural modifications based on hollow transformers and a simple tailored training objective that leverages more training signal. We use a synthetic example to illustrate the failure modes of existing samplers and show how informed correctors alleviate these problems. On the text8 and tokenized ImageNet 256x256 datasets, our informed corrector consistently produces superior samples with fewer errors or improved FID scores for discrete diffusion models. These results underscore the potential of informed correctors for fast and high-fidelity generation using discrete diffusion.
              </p>
              </div>
        <p />
        </li>
        <li><b>Controlling Moments with Kernel Stein Discrepancies.  (<a href="https://arxiv.org/pdf/2211.05408.pdf">arxiv</a>, <a href="https://github.com/noukoudashisoup/ksd-moment">code</a>)</b>
          <br /><i>Heishiro Kanagawa, Alessandro Barp, Arthur Gretton, and Lester Mackey.</i>
          <br />Annals of Applied Probability. December 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#ksdmoment">View details &raquo</a>
              <!--colapsed text-->
              <div id="ksdmoment" class="collapse">
              <p></p>
              <p>
              Kernel Stein discrepancies (KSDs) measure the quality of a distributional approximation and can be computed even when the target density has an intractable normalizing constant. Notable applications include the diagnosis of approximate MCMC samplers and goodness-of-fit tests for unnormalized statistical models. The present work analyzes the convergence control properties of KSDs. We first show that standard KSDs used for weak convergence control fail to control moment convergence. To address this limitation, we next provide sufficient conditions under which alternative diffusion KSDs control both moment and weak convergence. As an immediate consequence we develop, for each q>0, the first KSDs known to exactly characterize q-Wasserstein convergence.
              </p>
        </div>
        <p />
        </li>
        <li><b>Near-optimal inference in adaptive linear regression. (<a href="https://arxiv.org/pdf/2107.02266.pdf">arxiv</a>, <a href="https://github.com/KoulikBerkeley/online-debiasing/">code</a>)</b>
          <br /><i>Koulik Khamaru, Yash Deshpande, Lester Mackey, and Martin J. Wainwright.</i>
          <br />Annals of Statistics. December 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#onlinedebias">View details &raquo</a>
              <!--colapsed text-->
              <div id="onlinedebias" class="collapse">
              <p></p>
              <p>
              When data is collected in an adaptive manner, even simple methods likeordinary  least  squares  can  exhibit  non-normal  asymptotic  behavior.  As  anundesirable consequence, hypothesis tests and confidence intervals based on asymptotic normality can lead to erroneous results. We propose a family ofonline debiasing estimators to correct these distributional anomalies in least squares estimation. Our proposed methods take advantage of the covariance structure present in the dataset and provide sharper estimates in directions for which more information has accrued. We establish an asymptotic normality property for our proposed online debiasing estimators under mild conditions on the data collection process and provide asymptotically exact confidence intervals. We additionally prove a minimax lower bound for the adaptive linear regression problem, thereby providing a baseline by which to compare estimators. There are various conditions under which our proposed estimators achieve the minimax lower bounds up to logarithmic factors. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
          </p>
              </div>
              <p />
              </li>
        <li><b>Low-Rank Thinning.  (<a href="https://arxiv.org/pdf/2502.12063">arxiv</a>, <a href="https://github.com/microsoft/thinformer">thinformer</a>, <a href="https://github.com/microsoft/khsgd">khsgd</a>, <a href="https://github.com/microsoft/deepctt">deepctt</a>)</b>
          <br /><i>Annabelle Michael Carrell, Albert Gong, Abhishek Shetty, Raaz Dwivedi, and Lester Mackey.</i>
          <br />International Conference on Machine Learning (ICML). July 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#thinformer">View details &raquo</a>
              <!--colapsed text-->
              <div id="thinformer" class="collapse">
              <p></p>
              <p>
              The goal in thinning is to summarize a dataset using a small set of representative points. Remarkably, sub-Gaussian thinning algorithms like Kernel Halving and Compress can match the quality of uniform subsampling while substantially reducing the number of summary points. However, existing guarantees cover only a restricted range of distributions and kernel-based quality measures and suffer from pessimistic dimension dependence. To address these deficiencies, we introduce a new low-rank analysis of sub-Gaussian thinning that applies to any distribution and any kernel, guaranteeing high-quality compression whenever the kernel or data matrix is approximately low-rank. To demonstrate the broad applicability of the techniques, we design practical sub-Gaussian thinning approaches that improve upon the best known guarantees for approximating attention in transformers, accelerating stochastic gradient training through reordering, and distinguishing distributions in near-linear time.
              </p>
              </div>
              <p />
              </li>
        <li><b>SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery.  (<a href="https://arxiv.org/pdf/2311.17179.pdf">arxiv</a>, <a href="https://github.com/microsoft/satclip">code</a>)</b>
          <br /><i>Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Marc Ru&#223;wurm.</i>
          <br />AAAI Conference on Artificial Intelligence (AAAI). February 2025.
          <br/><a class="btn" data-toggle="collapse" data-target="#satclip">View details &raquo</a>
              <!--colapsed text-->
              <div id="satclip" class="collapse">
              <p></p>
              <p>
                Geographic information is essential for modeling tasks in fields ranging from ecology to epidemiology. However, extracting relevant location characteristics for a given task can be challenging, often requiring expensive data fusion or distillation from massive global imagery datasets. To address this challenge, we introduce Satellite Contrastive Location-Image Pretraining (SatCLIP). This global, general-purpose geographic location encoder learns an implicit representation of locations by matching CNN and ViT inferred visual patterns of openly available satellite imagery with their geographic coordinates. The resulting SatCLIP location encoder efficiently summarizes the characteristics of any given location for convenient use in downstream tasks. In our experiments, we use SatCLIP embeddings to improve prediction performance on nine diverse location-dependent tasks including temperature prediction, animal recognition, and population density estimation. Across tasks, SatCLIP consistently outperforms alternative location encoders and improves geographic generalization by encoding visual similarities of spatially distant environments. These results demonstrate the potential of vision-location models to learn meaningful representations of our planet from the vast, varied, and largely untapped modalities of geospatial data.
              </p>
        </div>
        <p />
        </li>
        <li><b>SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation. (<a href="https://arxiv.org/pdf/2411.09730">arxiv</a>, <a href="https://github.com/mkhodak/SureMap">code</a>)</b>
          <br /><i>Mikhail Khodak, Lester Mackey, Alexandra Chouldechova, and Miroslav Dudik.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#suremap">View details &raquo</a>
              <!--colapsed text-->
              <div id="suremap" class="collapse">
              <p></p>
              <p>
                Disaggregated evaluation---estimation of performance of a machine learning model on different subpopulations---is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually.  This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task).  In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models.  SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients.  Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).	We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.          </p>
              </div>
              <p />
              </li>
        <li><b>Targeted Separation and Convergence with Kernel Discrepancies.  (<a href="https://arxiv.org/pdf/2209.12835.pdf">arxiv</a>)</b>
          <br /><i>Alessandro Barp, Carl-Johann Simon-Gabriel, Mark Girolami, and Lester Mackey.</i>
          <br />Journal of Machine Learning Research (JMLR). December 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#mmdksd">View details &raquo</a>
              <!--colapsed text-->
              <div id="mmdksd" class="collapse">
              <p></p>
              <p>
                Maximum mean discrepancies (MMDs) like the kernel Stein discrepancy (KSD) have grown central to a wide range of applications, including hypothesis testing, sampler selection, distribution approximation, and variational inference. In each setting, these kernel-based discrepancy measures are required to (i) separate a target P from other probability measures or even (ii) control weak convergence to P. In this article we derive new sufficient and necessary conditions to ensure (i) and (ii). For MMDs on separable metric spaces, we characterize those kernels that separate Bochner embeddable measures and introduce simple conditions for separating all measures with unbounded kernels and for controlling convergence with bounded kernels. We use these results on R^d to substantially broaden the known conditions for KSD separation and convergence control and to develop the first KSDs known to exactly metrize weak convergence to P. Along the way, we highlight the implications of our results for hypothesis testing, measuring and improving sample quality, and sampling with Stein variational gradient descent.
          </p>
        </div>
        <p />
        </li>
        <li><b>Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.  (<a href="https://arxiv.org/pdf/2310.02304.pdf">arxiv</a>, <a href="https://github.com/microsoft/stop">code</a>)</b>
          <br /><i>Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai.</i>
          <br />Conference on Language Models (COLM). October 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#stop">View details &raquo</a>
              <!--colapsed text-->
              <div id="stop" class="collapse">
              <p></p>
              <p>
                Several recent advances in AI systems solve problems by providing a
                "scaffolding" program that structures multiple calls to language models (LMs)
                to generate better outputs. A scaffolding program is written in a programming
                language such as Python. In this work, we use a language-model-infused
                scaffolding program to improve itself. We start with a seed "improver" that
                improves an input program according to a given utility function by querying an
                LM several times and returning the best solution. We then run this seed
                improver to improve itself. Across a small set of downstream tasks, the
                resulting improved improver generates programs with significantly better
                performance than its seed improver. A variety of self-improvement strategies
                are proposed by the language model, including beam search, genetic algorithms,
                and simulated annealing. Since the language models themselves are not altered,
                this is not full recursive self-improvement. Nonetheless, it demonstrates that
                a modern language model, GPT-4 in our experiments, is capable of writing code
                that can call itself to improve itself. We consider concerns around the
                development of self-improving technologies and evaluate the frequency with
                which the generated code bypasses a sandbox.
              </p>
        </div>
        <p />
        </li>
        <li><b>Debiased Distribution Compression. (<a href="https://arxiv.org/pdf/2404.12290.pdf">arxiv</a>, <a href="https://github.com/microsoft/goodpoints">code</a>)</b>
          <br /><i>Lingxiao Li, Raaz Dwivedi, and Lester Mackey.</i>
          <br />International Conference on Machine Learning (ICML). July 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#ddc">View details &raquo</a>
              <!--colapsed text-->
              <div id="ddc" class="collapse">
              <p></p>
              <p>
                Modern compression methods can summarize a target distribution P more succinctly than i.i.d. sampling but require access to a low-bias input sequence like a Markov chain converging quickly to P. We introduce a new suite of compression methods suitable for compression with biased input sequences. Given n points targeting the wrong distribution and quadratic time, Stein Kernel Thinning (SKT) returns sqrt(n) equal-weighted points with &#213;(1/sqrt(n)) maximum mean discrepancy (MMD) to P. For larger-scale compression tasks, Low-rank SKT achieves the same feat in sub-quadratic time using an adaptive low-rank debiasing procedure that may be of independent interest. For downstream tasks that support simplex or constant-preserving weights, Stein Recombination and Stein Cholesky achieve even greater parsimony, matching the guarantees of SKT with as few as poly-log(n) weighted points. Underlying these advances are new guarantees for the quality of simplex-weighted coresets, the spectral decay of kernel matrices, and the covering numbers of Stein kernel Hilbert spaces. In our experiments, our techniques provide succinct and accurate posterior summaries while overcoming biases due to burn-in, approximate Markov chain Monte Carlo, and tempering.
              </p>
        </div>
        <p />
        </li>
        <li><b>Kernel Thinning. (<a href="https://arxiv.org/pdf/2105.05842.pdf">arxiv</a>, <a href="https://github.com/microsoft/goodpoints">code</a>, <a href="papers/thinning-slides.pdf">slides</a>)</b>
          <br /><i>Raaz Dwivedi and Lester Mackey.</i>
          <br />Journal of Machine Learning Research (JMLR). June 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#kernelthinning">View details &raquo</a>
              <!--colapsed text-->
              <div id="kernelthinning" class="collapse">
              <p></p>
              <p>
              We introduce kernel thinning, a new procedure for compressing a distribution P more effectively than i.i.d. sampling or standard thinning.
              Given a suitable reproducing kernel k and O(n^2) time, kernel thinning compresses an n-point approximation to P into a sqrt(n)-point approximation with comparable worst-case integration error across the associated reproducing kernel Hilbert space.
              With high probability, the maximum discrepancy in integration error is O_d(n^{-1/2} sqrt(log n)) for compactly supported P and O_d(n^{-1/2} sqrt((log n)^{d+1} log log n)) for sub-exponential P on R^d.
              In contrast, an equal-sized i.i.d. sample from P suffers Omega(n^{-1/4}) integration error.
              Our sub-exponential guarantees resemble
              the classical quasi-Monte Carlo error rates for uniform P on [0,1]^d but apply to general distributions on R^d
              and a wide range of common kernels.
              We use our results to derive explicit non-asymptotic maximum mean discrepancy bounds for Gaussian, Matern, and B-spline kernels
              and present two vignettes illustrating the practical benefits of kernel thinning over i.i.d. sampling and standard Markov chain Monte Carlo thinning, in dimensions d=2 through 100.
          </p>
              </div>
              <p />
              </li>
        <li><b> Do Language Models Know When They're Hallucinating References?  (<a href="https://arxiv.org/pdf/2305.18248.pdf">arxiv</a>, <a href="https://github.com/microsoft/hallucinated-references">code</a>)</b>
          <br /><i>Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Tauman Kalai.</i>
          <br />Findings of the Association for Computational Linguistics: EACL. Mar. 2024.
          <br/><a class="btn" data-toggle="collapse" data-target="#llmref">View details &raquo</a>
              <!--colapsed text-->
              <div id="llmref" class="collapse">
              <p></p>
              <p>
                State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature.
              </p>
        </div>
        <p />
        </li>
        <li><b>SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking.  (<a href="https://arxiv.org/pdf/2109.10399.pdf">arxiv</a>, <a href="https://github.com/microsoft/subseasonal_toolkit">code</a>, <a href="https://github.com/microsoft/subseasonal_data">SubseasonalClimateUSA dataset</a>, <a href="https://communities.springernature.com/posts/improving-subseasonal-forecasting-with-machine-learning">blog</a>, <a href="https://www.microsoft.com/en-us/research/project/subseasonal-climate-forecasting/">website</a>)</b>
          <br /><i>Soukayna Mouatadid, Paulo Orenstein, Genevieve Flaspohler, Miruna
              Oprescu, Judah Cohen, Franklyn Wang, Sean Knight, Maria Geogdzhayeva, Sam Levang, Ernest Fraenkel, and Lester Mackey.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#subseasonalusa">View details &raquo</a>
              <!--colapsed text-->
              <div id="subseasonalusa" class="collapse">
              <p></p>
              <p>
                Subseasonal forecasting of the weather two to six weeks in advance is critical for resource allocation and advance disaster notice but poses many challenges for the forecasting community.
                At this forecast horizon, physics-based dynamical models have limited skill, and the targets for prediction depend in a complex manner on both local weather and global climate variables.
                Recently, machine learning methods have shown promise in advancing the state of the art but  only at the cost of complex data curation, integrating expert knowledge with aggregation across multiple relevant data sources, file formats, and temporal and spatial resolutions.
                To streamline this process and accelerate future development, we introduce SubseasonalClimateUSA, a curated dataset for training and benchmarking subseasonal forecasting models in the United States.
                We use this dataset to benchmark a diverse suite of subseasonal models,
                including operational dynamical models, classical meteorological baselines, and ten state-of-the-art machine learning and deep learning-based methods from the literature.
                Overall, our benchmarks suggest simple and effective ways to extend the accuracy of current operational models.
                SubseasonalClimateUSA is regularly updated and accessible via the <a href="https://github.com/microsoft/subseasonal_data/">https://github.com/microsoft/subseasonal_data/</a> Python package.
          </p>
              </div>
              <p />
        </li>
        <li><b> Should I Stop or Should I Go: Early Stopping with Heterogeneous Populations.  (<a href="https://arxiv.org/pdf/2306.11839.pdf">arxiv</a>, <a href="https://github.com/hammaadadam1/clash">code</a>)</b>
          <br /><i>Hammaad Adam, Fan Yin, Mary Hu, Neil Tenenholtz, Lorin Crawford, Lester Mackey, and Allison Koenecke.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#clash">View details &raquo</a>
              <!--colapsed text-->
              <div id="clash" class="collapse">
              <p></p>
              <p>
                Randomized experiments often need to be stopped prematurely due to the treatment
                having an unintended harmful effect. Existing methods that determine when to
                stop an experiment early are typically applied to the data in aggregate and do
                not account for treatment effect heterogeneity. In this paper, we study the early
                stopping of experiments for harm on heterogeneous populations. We first establish
                that current methods often fail to stop experiments when the treatment harms a
                minority group of participants. We then use causal machine learning to develop
                CLASH, the first broadly-applicable method for heterogeneous early stopping. We
                demonstrate CLASH's performance on simulated and real data and show that it
                yields effective early stopping for both clinical trials and A/B tests.
              </p>
        </div>
        <p />
        </li>
        <li><b>Learning Rate Free Sampling in Constrained Domains.  (<a href="https://arxiv.org/pdf/2305.14943.pdf">arxiv</a>, <a href="https://github.com/louissharrock/constrained-coin-sampling">code</a>)</b>
          <br /><i>Louis Sharrock, Lester Mackey, and Christopher Nemeth.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#coinmsvgd">View details &raquo</a>
              <!--colapsed text-->
              <div id="coinmsvgd" class="collapse">
              <p></p>
              <p>
                We introduce a suite of new particle-based algorithms for sampling in constrained domains which are entirely learning rate free. Our approach leverages coin betting ideas from convex optimisation, and the viewpoint of constrained sampling as a mirrored optimisation problem on the space of probability measures. Based on this viewpoint, we also introduce a unifying framework for several existing constrained sampling algorithms, including mirrored Langevin dynamics and mirrored Stein variational gradient descent. We demonstrate the performance of our algorithms on a range of numerical examples, including sampling from targets on the simplex, sampling with fairness constraints, and constrained sampling problems in post-selection inference. Our results indicate that our algorithms achieve competitive performance with existing constrained sampling methods, without the need to tune any hyperparameters.
              </p>
        </div>
        <p />
        </li>
        <li><b> A Finite-Particle Convergence Rate for Stein Variational Gradient
          Descent.  (<a href="https://arxiv.org/pdf/2211.09721.pdf">arxiv</a>)</b>
          <br /><i>Jiaxin Shi and Lester Mackey.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#svgdrate">View details &raquo</a>
              <!--colapsed text-->
              <div id="svgdrate" class="collapse">
              <p></p>
              <p>
                We provide a first finite-particle convergence rate for Stein variational
                gradient descent (SVGD). Specifically, whenever the target distribution
                is sub-Gaussian with a Lipschitz score, SVGD with n particles and an appropriate
                step size sequence drives the kernel Stein discrepancy to zero at an order
                1/sqrt(log log n) rate. We suspect that the dependence on n can be improved,
                and we hope that our explicit, non-asymptotic proof strategy will serve as a
                template for future refinements.
              </p>
        </div>
        <p />
        </li>
        <li><b>Bounding Wasserstein distance with couplings. (<a href="https://arxiv.org/pdf/2112.03152.pdf">arxiv</a>, <a href="https://github.com/niloyb/BoundWasserstein">code</a>, <a href="papers/cub-neurips2021-poster.pdf">poster</a>)</b>
          <br /><i>Niloy Biswas and Lester Mackey.</i>
          <br />Journal of the American Statistical Association (JASA). November 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#cub">View details &raquo</a>
              <!--colapsed text-->
              <div id="cub" class="collapse">
              <p></p>
              <p>
                Markov chain Monte Carlo (MCMC) provides asymptotically consistent estimates of
                intractable posterior expectations as the number of iterations tends to infinity.
                However, in large data applications, MCMC can be computationally expensive per iteration.
                This has catalyzed interest in approximating MCMC in a manner that improves computational speed per iteration
                but does not produce asymptotically consistent estimates.
                In this article, we propose estimators based on couplings of Markov chains to assess the
                quality of such asymptotically biased sampling methods. The estimators give empirical upper
                bounds of the Wasserstein distance between the limiting distribution of the asymptotically
                biased sampling method and the original target distribution of interest.
                We establish theoretical guarantees for our upper bounds and show that our estimators
                can remain effective in high dimensions.
                We apply our quality measures to stochastic gradient MCMC, variational Bayes,
                and Laplace approximations for tall data and to approximate MCMC for Bayesian logistic regression in 4500 dimensions and Bayesian linear regression in 50000 dimensions.
              </p>
              </div>
              <p />
              </li>
        <li><b> Evaluation of Dependency Structure for Multivariate Weather Predictors using Copulas.  (<a href="https://arxiv.org/pdf/2211.09721.pdf">arxiv</a>, <a href="https://github.com/SamChege/Vine-Copulas-Weather-Dependency">code</a>)</b>
          <br /><i>Samuel C. Maina, Dorcas Mwigereri, Jonathan Weyn, Lester Mackey, and Millicent Ochieng.</i>
          <br />ACM Journal on Computing and Sustainable Societies (COMPASS).  September 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#copulas">View details &raquo</a>
              <!--colapsed text-->
              <div id="copulas" class="collapse">
              <p></p>
              <p>
              In the Global South, the effects of climate change have resulted in more frequent and severe weather events such as droughts, floods, and storms, leading to crop failures, food insecurity, and job loss. These effects are expected to increase in intensity in the future, further disadvantaging already marginalized communities and exacerbating existing inequalities. Hence the need for prevention and adaptation is urgent, but accurate weather forecasting remains challenging, despite advances in machine learning and numerical modeling, due to complex interaction of atmospheric and oceanic variables. This research aims to explore the potential of vine copulas in explaining complex relationships of different weather variables in three African locations. Copulas separate marginal distributions from the dependency structure, offering a flexible way to model dependence between random variables for improved risk assessments and simulations. Vine copulas are based on a variety of bivariate copulas, including Gaussian, Student's t, Clayton, Gumbel, and Frank copulas, and they are effective in high-dimensional problems and offer a hierarchy of trees to express conditional dependence. In addition, we propose how this framework can be applied within the subseasonal forecasting models to enhance the prediction of different weather events or variables.
              </p>
        </div>
        <p />
        </li>
        <li><b>Adaptive Bias Correction for Improved Subseasonal Forecasting.  (<a href="https://arxiv.org/pdf/2209.10666.pdf">arxiv</a>, <a href="https://www.nature.com/articles/s41467-023-38874-y">ncomms</a>, <a href="https://github.com/microsoft/subseasonal_toolkit">code</a>, <a href="https://communities.springernature.com/posts/improving-subseasonal-forecasting-with-machine-learning">blog</a>, <a href="https://www.microsoft.com/en-us/research/project/subseasonal-climate-forecasting/">website</a>)</b>
          <br /><i>Soukayna Mouatadid, Paulo Orenstein, Genevieve Flaspohler, Judah Cohen, Miruna Oprescu, Ernest Fraenkel, and Lester Mackey.</i>
          <br />Nature Communications. June 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#abc">View details &raquo</a>
              <!--colapsed text-->
              <div id="abc" class="collapse">
              <p></p>
              <p>
                Subseasonal forecasting---predicting temperature and precipitation 2 to 6 weeks ahead---is critical for effective water allocation, wildfire management, and drought and flood mitigation.
                Recent international research efforts have advanced the subseasonal capabilities of operational dynamical models, yet temperature and precipitation prediction skills remain poor,
                partly due to stubborn errors in representing atmospheric dynamics and physics inside dynamical models.
                To counter these errors, we introduce an adaptive bias correction (ABC) method that combines state-of-the-art dynamical forecasts with observations using machine learning.
                We show that, when applied to the leading subseasonal model from the European Centre for Medium-Range Weather Forecasts (ECMWF), ABC improves temperature forecasting skill by 60-90% (over baseline skills of 0.18-0.25) and precipitation forecasting skill by 40-69% (over baseline skills of 0.11-0.15) in the contiguous U.S.
                We couple these performance improvements with a practical workflow for explaining ABC skill gains and identifying higher-skill windows of opportunity based on specific climate conditions.
              </p>
              </div>
              <p />
              </li>
        <li><b>A Kernel Stein Test for Comparing Latent Variable Models. (<a href="https://arxiv.org/pdf/1907.00586.pdf">arxiv</a>, <a href="https://github.com/noukoudashisoup/lkgof">code</a>)</b>
          <br /><i>Heishiro Kanagawa, Wittawat Jitkrittum, Lester Mackey, Kenji Fukumizu, and Arthur Gretton.</i>
          <br />Journal of the Royal Statistical Society, Series B (JRSSB). May 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#latentstein">View details &raquo</a>
              <!--colapsed text-->
              <div id="latentstein" class="collapse">
              <p></p>
              <p>
                We propose a kernel-based nonparametric test of relative goodness of fit, where the goal is to compare two models, both of which may have unobserved latent variables, such that the marginal distribution of the observed variables is intractable. The proposed test generalizes the recently proposed kernel Stein discrepancy (KSD) tests
                (Liu et al., 2016; Chwialkowski et al., 2016; Yang et al., 2018) to the case of latent variable models, a much more general class than the fully observed models treated previously. The new test, with a properly calibrated threshold, has a well-controlled type-I error. In the case of certain models with low-dimensional latent structures and high-dimensional observations, our test significantly outperforms the relative maximum mean discrepancy test, which is based on samples from the models and does not exploit the latent structure.
              </p>
              </div>
              <p />
              </li>
        <li><b>Independent finite approximations for Bayesian nonparametric inference. (<a href="https://arxiv.org/pdf/2009.10780.pdf">arxiv</a>)</b>
          <br /><i>Tin D. Nguyen, Jonathan Huggins, Lorenzo Masoero, Lester Mackey, and Tamara Broderick.</i>
          <br />Bayesian Analysis. May 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#ifa">View details &raquo</a>
              <!--colapsed text-->
              <div id="ifa" class="collapse">
              <p></p>
              <p>
              Bayesian nonparametric priors based on completely random measures (CRMs) offer a flexible modeling approach when the number of latent components in a dataset is unknown. However, managing the infinite dimensionality of CRMs typically requires practitioners to derive ad-hoc algorithms, preventing the use of general-purpose inference methods and often leading to long compute times. We propose a general but explicit recipe to construct a simple finite-dimensional approximation that can replace the infinite-dimensional CRMs. Our independent finite approximation (IFA) is a generalization of important cases that are used in practice. The independence of atom weights in our approximation (i) makes the construction well-suited for parallel and distributed computation and (ii) facilitates more convenient inference schemes. We quantify the approximation error between IFAs and the target nonparametric prior. We compare IFAs with an alternative approximation scheme -- truncated finite approximations (TFAs), where the atom weights are constructed sequentially. We prove that, for worst-case choices of observation likelihoods, TFAs are a more efficient approximation than IFAs. However, in real-data experiments with image denoising and topic modeling, we find that IFAs perform very similarly to TFAs in terms of task-specific accuracy metrics.
          </p>
              </div>
              <p />
              </li>
        <li><b> Compress Then Test: Powerful Kernel Testing in Near-linear Time.  (<a href="https://arxiv.org/pdf/2301.05974.pdf">arxiv</a>, <a href="https://github.com/microsoft/goodpoints">code</a>)</b>
          <br /><i>Carles Domingo-Enrich, Raaz Dwivedi, and Lester Mackey.</i>
          <br />International Conference on Artificial Intelligence and Statistics (AISTATS). April 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#ctt">View details &raquo</a>
              <!--colapsed text-->
              <div id="ctt" class="collapse">
              <p></p>
              <p>
                Kernel two-sample testing provides a powerful framework for distinguishing any pair of distributions based on $n$ sample points.
                However, existing kernel tests either run in $n^2$ time or sacrifice undue power to improve runtime.
                To address these shortcomings, we introduce Compress Then Test (CTT), a new framework for high-powered kernel testing based on sample compression.
                CTT cheaply approximates an expensive test by compressing each $n$ point sample into a small but provably high-fidelity coreset.
                For standard kernels and subexponential distributions, CTT inherits the statistical behavior of a quadratic-time test---recovering the same optimal detection boundary---while running in near-linear time.
                We couple these advances with cheaper permutation testing, justified by new power analyses;
                improved time-vs.-quality guarantees for low-rank approximation;
                and a fast aggregation procedure for identifying especially discriminating kernels.
                In our experiments with real and simulated data, CTT and its extensions provide 20--200x speed-ups over state-of-the-art approximate MMD tests with no loss of power.
              </p>
        </div>
        <p />
        </li>
        <li><b>Metrizing Weak Convergence with Maximum Mean Discrepancies. (<a href="https://arxiv.org/pdf/2006.09268.pdf">arxiv</a>)</b>
          <br /><i>Carl-Johann Simon-Gabriel, Alessandro Barp, Bernhard Sch&#246;lkopf, and Lester Mackey.</i>
          <br />Journal of Machine Learning Research (JMLR). To appear.
          <br/><a class="btn" data-toggle="collapse" data-target="#weakmmd">View details &raquo</a>
              <!--colapsed text-->
              <div id="weakmmd" class="collapse">
              <p></p>
              <p>Theorem 12 of Simon-Gabriel and Scholkopf (JMLR, 2018) seemed to close a 40-year-old quest to characterize maximum mean discrepancies (MMD) that metrize the weak convergence of probability measures. We prove, however, that the theorem is incorrect and provide a correction. We show that, on a locally compact, non-compact, Hausdorff space, the MMD of a bounded continuous Borel measurable kernel k, whose RKHS-functions vanish at infinity, metrizes the weak convergence of probability measures if and only if k is continuous and integrally strictly positive definite (ISPD) over all signed, finite, regular Borel measures. We also show that, contrary to the claim of the aforementioned Theorem 12, there exist both bounded continuous ISPD kernels that do not metrize weak convergence and bounded continuous non-ISPD kernels that do metrize it.
              </p>
              </div>
              <p />
              </li>
        <li><b>Social Norm Bias: Residual Harms of Fairness-Aware Algorithms.  (<a href="https://arxiv.org/pdf/2108.11056.pdf">arxiv</a>)</b>
          <br /><i>Myra Cheng, Maria De-Arteaga, Lester Mackey, and Adam Tauman Kalai.</i>
          <br />WIREs Data Mining and Knowledge Discovery. January 2023.
          <br/><a class="btn" data-toggle="collapse" data-target="#snob">View details &raquo</a>
              <!--colapsed text-->
              <div id="snob" class="collapse">
              <p></p>
              <p>
              Many modern learning algorithms mitigate bias by enforcing fairness across coarsely-defined groups related to a sensitive attribute like gender or race. However, the same algorithms seldom account for the within-group biases that arise due to the heterogeneity of group members. In this work, we characterize Social Norm Bias (SNoB), a subtle but consequential type of discrimination that may be exhibited by automated decision-making systems, even when these systems achieve group fairness objectives. We study this issue through the lens of gender bias in occupation classification from biographies. We quantify SNoB by measuring how an algorithm's predictions are associated with conformity to gender norms, which is measured using a machine learning approach. This framework reveals that for classification tasks related to male-dominated occupations, fairness-aware classifiers favor biographies written in ways that align with masculine gender norms. We compare SNoB across fairness intervention techniques and show that post-processing interventions do not mitigate this type of bias at all.
          </p>
              </div>
              <p />
              </li>
        <li><b>Gradient Estimation with Discrete Stein Operators. (<a href="https://arxiv.org/pdf/2202.09497.pdf">arxiv</a>, <a href="https://github.com/thjashin/rodeo">code</a>)</b>
          <br /><i>Jiaxin Shi, Yuhao Zhou, Jessica Hwang, Michalis K. Titsias, and Lester Mackey.</i>
          <br />Advances in Neural Processing Systems (NeurIPS). December 2022.
          <br /><small><b>NeurIPS 2022 Outstanding Paper Award.</b></small>
          <br/><a class="btn" data-toggle="collapse" data-target="#rodeo">View details &raquo</a>
              <!--colapsed text-->
              <div id="rodeo" class="collapse">
              <p></p>
              <p>
  Gradient estimation -- approximating the gradient of an expectation with
  respect to the parameters of a distribution -- is central to the solution of
  many machine learning problems. However, when the distribution is discrete,
  most common gradient estimators suffer from excessive variance. To improve the
  quality of gradient estimation, we introduce a variance reduction technique
  based on Stein operators for discrete distributions. We then use this technique
  to build flexible control variates for the REINFORCE leave-one-out estimator.
  Our control variates can be adapted online to minimize the variance and do not
  require extra evaluations of the target function. In benchmark generative
  modeling tasks such as training binary variational autoencoders, our gradient
  estimator achieves substantially lower variance than state-of-the-art
  estimators with the same number of function evaluations.
              </p>
              </div>
              <p />
              </li>
            <li><b>Scalable Spike-and-Slab. (<a href="https://arxiv.org/pdf/2204.01668.pdf">arxiv</a>, <a href="https://github.com/niloyb/ScaleSpikeSlab">code</a>)</b>
              <br /><i>Niloy Biswas, Lester Mackey, and Xiao-Li Meng.</i>
	      <br />International Conference on Machine Learning (ICML). July 2022.
                <br/><a class="btn" data-toggle="collapse" data-target="#s3">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="s3" class="collapse">
                    <p></p>
                    <p>
		    Spike-and-slab priors are commonly used for Bayesian variable selection, due to their interpretability and favorable statistical properties. However, existing samplers for spike-and-slab posteriors incur prohibitive computational costs when the number of variables is large. In this article, we propose Scalable Spike-and-Slab (S^3), a scalable Gibbs sampling implementation for high-dimensional Bayesian regression with the continuous spike-and-slab prior of George and McCulloch (1993). For a dataset with n observations and p covariates, S^3 has order max(n^2 p_t, np) computational cost at iteration t where p_t never exceeds the number of covariates switching spike-and-slab states between iterations t and t-1 of the Markov chain. This improves upon the order n^2 p per-iteration cost of state-of-the-art implementations as, typically, p_t is substantially smaller than p. We apply S^3 on synthetic and real-world datasets, demonstrating orders of magnitude speed-ups over existing exact samplers and significant gains in inferential quality over approximate samplers with comparable cost.
                    </p>
                    </div>
                    <p />
            </li>
	    <li><b>Stein's Method Meets Statistics: A Review of Some Recent Developments. (<a href="https://arxiv.org/pdf/2105.03481.pdf">arxiv</a>)</b>
              <br /><i>Andreas Anastasiou, Alessandro Barp, Francois-Xavier Briol, Bruno Ebner, Robert E. Gaunt, Fatemeh Ghaderinezhad, Jackson Gorham, Arthur Gretton, Christophe Ley, Qiang Liu, Lester Mackey, Chris. J. Oates, Gesine Reinert, and Yvik Swan.</i>
	      <br />Statistical Science. February 2023.
                <br/><a class="btn" data-toggle="collapse" data-target="#steinstat">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="steinstat" class="collapse">
                    <p></p>
                    <p>
		      Stein's method compares probability distributions through the study of a class of linear operators called Stein operators. While mainly studied in probability and used to underpin theoretical statistics, Stein's method has led to significant advances in computational statistics in recent years. The goal of this survey is to bring together some of these recent developments and, in doing so, to stimulate further research into the successful field of Stein's method and statistics. The topics we discuss include tools to benchmark and compare sampling methods such as approximate Markov chain Monte Carlo, deterministic alternatives to sampling methods, control variate techniques, parameter estimation and goodness-of-fit testing.
                </p>
                    </div>
                    <p />
                    </li>
            <li><b>Distribution Compression in Near-linear Time. (<a href="https://arxiv.org/pdf/2111.07941.pdf">arxiv</a>, <a href="https://github.com/microsoft/goodpoints">code</a>, <a href="papers/thinning-slides.pdf">slides</a>)</b>
                <br /><i>Abhishek Shetty, Raaz Dwivedi, and Lester Mackey.</i>
                <br />International Conference on Learning Representations (ICLR). April 2022.
                  <br /><small><b>2022 American Statistical Association SCSG Student Paper Award.</b></small>
                <br/><a class="btn" data-toggle="collapse" data-target="#compress">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="compress" class="collapse">
                    <p></p>
                    <p>
            In distribution compression, one aims to accurately summarize a probability distribution P using a small number of representative points. Near-optimal thinning procedures achieve this goal by sampling n points from a Markov chain and identifying sqrt(n) points with &Otilde(1/sqrt(n)) discrepancy to P. Unfortunately, these algorithms suffer from quadratic or super-quadratic runtime in the sample size n. To address this deficiency, we introduce Compress++, a simple meta-procedure for speeding up any thinning algorithm while suffering at most a factor of 4 in error. When combined with the quadratic-time kernel halving and kernel thinning algorithms of Dwivedi and Mackey (2021), Compress++ delivers sqrt(n) points with O(sqrt(log n/n)) integration error and better-than-Monte-Carlo maximum mean discrepancy in O(n log^3 n) time and O(sqrt(n) log^2 n) space. Moreover, Compress++ enjoys the same near-linear runtime given any quadratic-time input and reduces the runtime of super-quadratic algorithms by a square-root factor. In our benchmarks with high-dimensional Monte Carlo samples and Markov chains targeting challenging differential equation posteriors, Compress++ matches or nearly matches the accuracy of its input algorithm in orders of magnitude less time.
                </p>
                    </div>
                    <p />
                    </li>
            <li><b>Generalized Kernel Thinning. (<a href="https://arxiv.org/pdf/2110.01593.pdf">arxiv</a>, <a href="https://github.com/microsoft/goodpoints">code</a>, <a href="papers/thinning-slides.pdf">slides</a>)</b>
                <br /><i>Raaz Dwivedi and Lester Mackey.</i>
                <br />International Conference on Learning Representations (ICLR). April 2022.
                <br/><a class="btn" data-toggle="collapse" data-target="#gkt">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="gkt" class="collapse">
                    <p></p>
                    <p>
            The kernel thinning (KT) algorithm of Dwivedi and Mackey (2021) compresses a probability distribution more effectively than independent sampling by targeting a reproducing kernel Hilbert space (RKHS) and leveraging a less smooth square-root kernel. Here we provide four improvements. First, we show that KT applied directly to the target RKHS yields tighter, dimension-free guarantees for any kernel, any distribution, and any fixed function in the RKHS. Second, we show that, for analytic kernels like Gaussian, inverse multiquadric, and sinc, target KT admits maximum mean discrepancy (MMD) guarantees comparable to or better than those of square-root KT without making explicit use of a square-root kernel. Third, we prove that KT with a fractional power kernel yields better-than-Monte-Carlo MMD guarantees for non-smooth kernels, like Laplace and Matern, that do not have square-roots. Fourth, we establish that KT applied to a sum of the target and power kernels (a procedure we call KT+) simultaneously inherits the improved MMD guarantees of power KT and the tighter individual function guarantees of target KT. In our experiments with target KT and KT+, we witness significant improvements in integration error even in 100 dimensions and when compressing challenging differential equation posteriors.
                </p>
                    </div>
                    <p />
                    </li>
            <li><b>Sampling with Mirrored Stein Operators. (<a href="https://arxiv.org/pdf/2106.12506.pdf">arxiv</a>, <a href="https://github.com/thjashin/mirror-stein-samplers">code</a>)</b>
                <br /><i>Jiaxin Shi, Chang Liu, and Lester Mackey.</i>
                <br />International Conference on Learning Representations (ICLR). April 2022.
                <br/><a class="btn" data-toggle="collapse" data-target="#mirrorstein">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="mirrorstein" class="collapse">
                    <p></p>
                    <p>
                    We introduce a new family of particle evolution samplers suitable for constrained domains and non-Euclidean geometries. Stein Variational Mirror Descent and Mirrored Stein Variational Gradient Descent minimize the Kullback-Leibler (KL) divergence to constrained target distributions by evolving particles in a dual space defined by a mirror map. Stein Variational Natural Gradient exploits non-Euclidean geometry to more efficiently minimize the KL divergence to unconstrained targets. We derive these samplers from a new class of mirrored Stein operators and adaptive kernels developed in this work. We demonstrate that these new samplers yield accurate approximations to distributions on the simplex, deliver valid confidence intervals in post-selection inference, and converge more rapidly than prior methods in large-scale unconstrained posterior inference. Finally, we establish the convergence of our new procedures under verifiable conditions on the target distribution.
                </p>
                    </div>
                    <p />
                    </li>
              <li><b>Optimal Thinning of MCMC Output. (<a href="https://arxiv.org/pdf/2005.03952.pdf">arxiv</a>,
                  <a href="https://stein-thinning.org/">website</a>,
                  <a href="https://www.youtube.com/watch?v=WwmTeLrNmOQ">video</a>, <a href="papers/thinning-slides.pdf">slides</a>)</b>
                <br /><i>Marina Riabiz, Wilson Chen, Jon Cockayne, Pawel Swietach, Steven A. Niederer, Lester Mackey, and Chris. J. Oates.</i>
                <br />Journal of the Royal Statistical Society, Series B (JRSSB). April 2022.
                <br/><a class="btn" data-toggle="collapse" data-target="#steinthinning">View details &raquo</a>
                <!--colapsed text-->
                <div id="steinthinning" class="collapse">
                  <p></p>
                  <p>
		    The use of heuristics to assess the convergence and compress the output of Markov chain Monte Carlo can be sub-optimal in terms of the empirical approximations that are produced. Typically a number of the initial states are attributed to "burn in" and removed, whilst the chain can be "thinned" if compression is also required. In this paper we consider the problem of selecting a subset of states, of fixed cardinality, such that the approximation provided by their empirical distribution is close to optimal. A novel method is proposed, based on greedy minimisation of a kernel Stein discrepancy, that is suitable for problems where heavy compression is required. Theoretical results guarantee consistency of the method and its effectiveness is demonstrated in the challenging context of parameter inference for ordinary differential equations. Software is available in the "Stein Thinning" package in both Python and MATLAB, and example code is included.
                  </p>
                </div>
                <p />
              </li>
	      <li><b>An Open Repository of Real-Time COVID-19 Indicators.  (<a href="https://doi.org/10.1101/2021.07.12.21259660">medrxiv</a>)</b>
                <br /><i>Alex Reinhart, Logan Brooks, Maria Jahja, Aaron Rumack, Jingjing Tang, Wael Al Saeed, Taylor Arnold, Amartya Basu, Jacob Bien, Angel A. Cabrera, Andrew Chin, Eu Jing Chua, Brian Clark, Nat DeFries, Jodi Forlizzi, Samuel Gratzl, Alden Green, George Haff, Robin Han, Addison J. Hu, Sangwon Hyun, Ananya Joshi, Jimi Kim, Andrew Kuznetsov, Wichada La Motte-Kerr, Yeon Jin Lee, Kenneth Lee, Zachary C. Lipton, Michael X. Liu, Lester Mackey, Kathryn Mazaitis, Daniel J. McDonald, Balasubramanian Narasimhan, Natalia L. Oliveira, Pratik Patil, Adam Perer, Collin A. Politsch, Samyak Rajanala, Dawn Rucker, Nigam H. Shah, Vishnu Shankar, James Sharpnack, Dmitry Shemetov, Noah Simon, Vishakha Srivastava, Shuyi Tan, Robert Tibshirani, Elena Tuzhilina, Ana Karina Van Nortwick, Valerie Ventura, Larry Wasserman, Jeremy C. Weiss, Kristin Williams, Roni Rosenfeld, and Ryan J. Tibshirani.</i>
		<br />Proceedings of the National Academy of Sciences. December 2021.
                <br/><a class="btn" data-toggle="collapse" data-target="#covidcast">View details &raquo</a>
                <!--colapsed text-->
                <div id="covidcast" class="collapse">
                  <p></p>
                  <p>
                    The COVID-19 pandemic presented enormous data challenges in the United States. Policy makers, epidemiological modelers, and health researchers all require up-to-date data on the pandemic and relevant public behavior, ideally at fine spatial and temporal resolution. The COVIDcast API is our attempt to fill this need: operational since April 2020, it provides open access to both traditional public health surveillance signals (cases, deaths, and hospitalizations) and many auxiliary indicators of COVID-19 activity, such as signals extracted from de-identified medical claims data, massive online surveys, cell phone mobility data, and internet search trends. These are available at a fine geographic resolution (mostly at the county level) and are updated daily. The COVIDcast API also tracks all revisions to historical data, allowing modelers to account for the frequent revisions and backfill that are common for many public health data sources. All of the data is available in a common format through the API and accompanying R and Python software packages. This paper describes the data sources and signals, and provides examples demonstrating that the auxiliary signals in the COVIDcast API present information relevant to tracking COVID activity, augmenting traditional public health reporting and empowering research and decision-making.
                  </p>
                </div>
                <p />
	      </li>
            <li><b>DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation. (<a href="https://arxiv.org/pdf/1805.12323.pdf">arxiv</a>, <a href="https://github.com/jimmyyhwu/ddsm-visual-primitives">code</a>)</b>
              <br /><i>Jimmy Wu, Bolei Zhou, Diondra Peck, Scott Hsieh, Vandana Dialani, Lester Mackey, and Genevieve Patterson.</i>
              <br />Harvard Data Science Review. October 2021.
              <br/><a class="btn" data-toggle="collapse" data-target="#deepminer">View details &raquo</a>
                  <!--colapsed text-->
                  <div id="deepminer" class="collapse">
                  <p></p>
                  <p>We propose DeepMiner, a framework to discover interpretable representations in deep neural networks and to build explanations for medical predictions. By probing convolutional neural networks (CNNs) trained to classify cancer in mammograms, we show that many individual units in the final convolutional layer of a CNN respond strongly to diseased tissue concepts specified by the BI-RADS lexicon. After expert annotation of the interpretable units, our proposed method is able to generate explanations for CNN mammogram classification that are correlated with ground truth radiology reports on the DDSM dataset. We show that DeepMiner not only enables better understanding of the nuances of CNN classification decisions, but also possibly discovers new visual knowledge relevant to medical diagnosis.</p>
                  </div>
                  <p />
                  </li>
            <li><b>Online Learning with Optimism and Delay. (<a href="https://arxiv.org/pdf/2106.06885.pdf">arxiv</a>, <a href="https://github.com/geflaspohler/poold">code</a>, <a href="papers/poold-slides.pdf">slides</a>)</b>
                <br /><i>Genevieve Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, and Lester Mackey.</i>
                <br />International Conference on Machine Learning (ICML). July 2021.
                <br/><a class="btn" data-toggle="collapse" data-target="#poold">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="poold" class="collapse">
                    <p></p>
                    <p>
                    Inspired by the demands of real-time climate and weather forecasting, we develop optimistic online learning algorithms that require no parameter tuning and have optimal regret guarantees under delayed feedback. Our algorithms -- DORM, DORM+, and AdaHedgeD -- arise from a novel reduction of delayed online learning to optimistic online learning that reveals how optimistic hints can mitigate the regret penalty caused by delay. We pair this delay-as-optimism perspective with a new analysis of optimistic learning that exposes its robustness to hinting errors and a new meta-algorithm for learning effective hinting strategies in the presence of delay. We conclude by benchmarking our algorithms on four subseasonal climate forecasting tasks, demonstrating low regret relative to state-of-the-art forecasting models.
                    </p>
                    </div>
                    <p />
            </li>
            <li><b>Knowledge Distillation as Semiparametric Inference. (<a href="https://arxiv.org/pdf/2104.09732.pdf">arxiv</a>, <a href="https://github.com/microsoft/semiparametric-distillation">code</a>, <a href="papers/enhanced_kd_gantsc-slides.pdf">slides</a>)</b>
              <br /><i>Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, and Lester Mackey.</i>
              <br />International Conference on Learning Representations (ICLR). May 2021.
                <br/><a class="btn" data-toggle="collapse" data-target="#enhancedkd">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="enhancedkd" class="collapse">
                    <p></p>
                    <p>
                    A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model.
                    Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data.
                    To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate.
                    By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements - cross-fitting and loss correction - to mitigate the impact of teacher overfitting and underfitting on student performance.
                    We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
                    </p>
                    </div>
                    <p />
            </li>
            <li><b>Initialization and Regularization of Factorized Neural Layers. (<a href="https://arxiv.org/pdf/2105.01029.pdf">arxiv</a>, <a href="https://github.com/microsoft/fnl_paper">code</a>, <a href="https://www.microsoft.com/en-us/research/blog/factorized-layers-revisited-compressing-deep-networks-without-playing-the-lottery/">blog</a>)</b>
              <br /><i>Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi.</i>
              <br />International Conference on Learning Representations (ICLR). May 2021.
                <br/><a class="btn" data-toggle="collapse" data-target="#fnl">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="fnl" class="collapse">
                    <p></p>
                    <p>
                    Factorized layers - operations parameterized by products of two or more matrices - occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed counterparts; we back this intuition with an analysis of how the initialization and regularization schemes impact training with gradient descent, drawing on modern attempts to understand the interplay of weight-decay and batch-normalization. Empirically, we highlight the benefits of spectral initialization and Frobenius decay across a variety of settings. In model compression, we show that they enable low-rank methods to significantly outperform both unstructured sparsity and tensor methods on the task of training low-memory residual networks; analogs of the schemes also improve the performance of tensor decomposition techniques. For knowledge distillation, Frobenius decay enables a simple, overcomplete baseline that yields a compact model from over-parameterized training without requiring retraining with or pruning a teacher network. Finally, we show how both schemes applied to multi-head attention lead to improved performance on both translation and unsupervised pre-training.
                    </p>
                    </div>
                    <p />
            </li>
            <li><b>Cross-validation Confidence Intervals for Test Error. (<a href="https://arxiv.org/pdf/2007.12671.pdf">arxiv</a>, <a href="https://github.com/alexandre-bayle/cvci">code</a>,  <a href="papers/cvci-slides.pdf">slides</a>)</b>
              <br /><i>Pierre Bayle, Alexandre Bayle, Lucas Janson, and Lester Mackey.</i>
	      <br />Advances in Neural Information Processing Systems (NeurIPS). December 2020.
                <br/><a class="btn" data-toggle="collapse" data-target="#cvci">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="cvci" class="collapse">
                    <p></p>
                    <p>
                    This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.
                    </p>
                    </div>
                    <p />
            </li>
            <li><b>Stochastic Stein Discrepancies. (<a href="https://arxiv.org/pdf/2007.02857.pdf">arxiv</a>, <a href="https://github.com/jgorham/stochastic_stein_discrepancy">code</a>)</b>
              <br /><i>Jackson Gorham, Anant Raj, and Lester Mackey.</i>
	      <br />Advances in Neural Information Processing Systems (NeurIPS). December 2020.
                <br/><a class="btn" data-toggle="collapse" data-target="#ssd">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="ssd" class="collapse">
                    <p></p>
                    <p>
		      Stein discrepancies (SDs) monitor convergence and non-convergence in approximate inference when exact integration and sampling are intractable. However, the computation of a Stein discrepancy can be prohibitive if the Stein operator -- often a sum over likelihood terms or potentials -- is expensive to evaluate. To address this deficiency, we show that stochastic Stein discrepancies (SSDs) based
on subsampled approximations of the Stein operator inherit the convergence control properties of standard SDs with probability 1. Along the way, we establish the convergence of Stein variational gradient descent (SVGD) on unbounded domains, resolving an open question of Liu (2017). In our experiments with biased Markov chain Monte Carlo (MCMC) hyperparameter tuning, approximate MCMC sampler selection, and stochastic SVGD, SSDs deliver comparable inferences to standard SDs with orders of magnitude fewer likelihood evaluations.
	            </p>
                    </div>
                    <p />
                    </li>
            <li><b>Minimax Estimation of Conditional Moment Models. (<a href="https://arxiv.org/abs/2006.07201">arxiv</a>, <a href="https://github.com/microsoft/AdversarialGMM">code</a>, <a href="https://www.microsoft.com/en-us/research/blog/adversarial-machine-learning-and-instrumental-variables-for-flexible-causal-modeling/">blog</a>)</b>
              <br /><i>Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis.</i>
	      <br />Advances in Neural Information Processing Systems (NeurIPS). December 2020.
                <br/><a class="btn" data-toggle="collapse" data-target="#minicmm">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="minicmm" class="collapse">
                    <p></p>
                    <p>
		      We develop an approach for estimating models described via conditional moment restrictions, with a prototypical application being non-parametric instrumental
variable regression. We introduce a min-max criterion function, under which the estimation problem can be thought of as solving a zero-sum game between a modeler who is optimizing over the hypothesis space of the target model and an adversary who identifies violating moments over a test function space. We analyze the statistical estimation rate of the resulting estimator for arbitrary hypothesis spaces, with respect to an appropriate analogue of the mean squared error metric, for ill-posed inverse problems. We show that when the minimax criterion is regularized with a second moment penalty on the test function and the test function space is sufficiently rich, then the estimation rate scales with the critical radius of the hypothesis and test function spaces, a quantity which typically gives tight fast rates. Our main result follows from a novel localized Rademacher analysis of statistical learning problems defined via minimax objectives. We provide applications of our main results for several hypothesis spaces used in practice such as: reproducing kernel Hilbert spaces, high dimensional sparse linear functions, spaces defined via shape constraints, ensemble estimators such as random forests, and neural networks. For each of these applications we provide computationally efficient optimization methods for solving the corresponding minimax problem (e.g. stochastic first-order heuristics for neural networks). In several applications, we show how our modified mean squared error rate, combined with conditions that bound the ill-posedness of the inverse problem, lead to mean squared error rates. We conclude with an extensive experimental analysis of the proposed methods.
	            </p>
                    </div>
                    <p />
                    </li>
            <li><b>Single Point Transductive Prediction. (<a href="https://arxiv.org/pdf/1908.02341.pdf">arxiv</a>, <a href="https://github.com/nileshtrip/SPTransducPredCode">code</a>, <a href="https://slideslive.com/38928039">talk</a>)</b>
              <br /><i>Nilesh Tripuraneni and Lester Mackey.</i>
	      <br />International Conference on Machine Learning (ICML). July 2020.
                <br/><a class="btn" data-toggle="collapse" data-target="#singlepointtransduction">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="singlepointtransduction" class="collapse">
                    <p></p>
                    <p>
		      Standard methods in supervised learning separate training and prediction: the model is fit independently of any test points it may encounter. However, can knowledge of the next test point <b>x</b><sub>&#9733;</sub> be exploited to improve prediction accuracy? We address this question in the context of linear prediction, showing how techniques from semiparametric inference can be used transductively to combat regularization bias. We first lower bound the <b>x</b><sub>&#9733;</sub> prediction error of ridge regression and the Lasso, showing that they must incur significant bias in certain test directions. We then provide non-asymptotic upper bounds on the <b>x</b><sub>&#9733;</sub> prediction error of two transductive prediction rules. We conclude by showing the efficacy of our methods on both synthetic and real data, highlighting the improvements single point transductive prediction can provide in settings with distribution shift.
                    </p>
                    </div>
                    <p />
                    </li>
            <li><b>Opportunities and Challenges for Analyzing Cancer Data at the Inter- and Intra-Institutional Levels. (<a href="https://doi.org/10.1200/PO.19.00394">article</a>)</b>
              <br /><i>Julie Wu, Jordan Bryan, Samuel M. Rubinstein, Lucy Wang, Michele Lenoue-Newton, Raed Zuhour, Mia Levy, Christine Micheel, Yaomin Xu, Suresh K. Bhavnani, Lester Mackey, and Jeremy L. Warner.</i>
	      <br />JCO Precision Oncology. June 2020.
                <br/><a class="btn" data-toggle="collapse" data-target="#geniemetastasis">View details &raquo</a>
                    <!--colapsed text-->
                    <div id="geniemetastasis" class="collapse">
                      <p></p>
                    <p>
		      <b>PURPOSE:</b> Our goal was to identify the opportunities and challenges in analyzing data from the American Association of Cancer Research Project Genomics Evidence Neoplasia Information Exchange (GENIE), a multi-institutional database derived from clinically driven genomic testing, at both the inter- and the intra-institutional level. Inter-institutionally, we identified genotypic differences between primary and metastatic tumors across the 3 most represented cancers in GENIE. Intra-institutionally, we analyzed the clinical characteristics of the Vanderbilt-Ingram Cancer Center (VICC) subset of GENIE to inform the interpretation of GENIE as a whole.
		      <br/><b>METHODS:</b> We performed overall cohort matching on the basis of age, ethnicity, and sex of 13,208 patients stratified by cancer type (breast, colon, or lung) and sample site (primary or metastatic). We then determined whether detected variants, at the gene level, were associated with primary or metastatic tumors. We extracted clinical data for the VICC subset from VICC's clinical data warehouse. Treatment exposures were mapped to a 13-class schema derived from the HemOnc ontology.
		      <br/><b>RESULTS:</b> Across 756 genes, there were significant differences in all cancer types. In breast cancer, ESR1 variants were over-represented in metastatic samples (odds ratio, 5.91; q < 10e-6). TP53 mutations were over-represented in metastatic samples across all cancers. VICC had a significantly different cancer type distribution than that of GENIE but patients were well matched with respect to age, sex, and sample type. Treatment data from VICC was used for a bipartite network analysis, demonstrating clusters with a mix of histologies and others being more histology specific.
		      <br/><b>CONCLUSION:</b> This article demonstrates the feasibility of deriving meaningful insights from GENIE at the inter- and intra-institutional level and illuminates the opportunities and challenges of the data GENIE contains. The results should help guide future development of GENIE, with the goal of fully realizing its potential for accelerating precision medicine.
                    </p>
                    </div>