-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathfunction_approximation.html
More file actions
1514 lines (1379 loc) · 67.8 KB
/
Copy pathfunction_approximation.html
File metadata and controls
1514 lines (1379 loc) · 67.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>How Activations Shape What a Network Learns</title>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.17.0/dist/tf.min.js"></script>
<style>
:root {
--bg: #0d1117;
--surface: #161b22;
--border: #30363d;
--text: #e6edf3;
--muted: #8b949e;
--accent: #7c83ff;
--accent2: #58d6ff;
--red: #ff5c5c;
--green: #39d353;
--panel: #1c2128;
--orange: #f0883e;
--yellow: #ffdb5c;
--purple: #c084fc;
--ood-color: rgba(255, 92, 92, 0.08);
--ood-border: rgba(255, 92, 92, 0.4);
--func-canvas-bg: #12181f;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body { background: var(--bg); color: var(--text); font-family: 'SF Mono', 'Fira Code', monospace; font-size: 13px; min-height: 100vh; }
/* ── HEADER ── */
#header {
background: var(--surface);
border-bottom: 1px solid var(--border);
padding: 10px 20px;
display: flex;
align-items: center;
gap: 16px;
flex-wrap: wrap;
}
#header h1 { font-size: 15px; color: var(--accent); font-weight: 700; letter-spacing: 0.5px; }
#header .subtitle { color: var(--muted); font-size: 11px; }
#header a { color: var(--accent2); font-size: 11px; text-decoration: none; margin-left: auto; }
#header a:hover { text-decoration: underline; }
/* ── CONTROLS BAR ── */
#controls {
background: var(--panel);
border-bottom: 1px solid var(--border);
padding: 8px 20px;
display: flex;
align-items: center;
gap: 10px;
flex-wrap: wrap;
}
.ctrl-group {
display: flex;
align-items: center;
gap: 6px;
background: #21262d;
border: 1px solid var(--border);
border-radius: 6px;
padding: 5px 10px;
}
.ctrl-group label { color: var(--accent); font-size: 10px; font-weight: 700; text-transform: uppercase; letter-spacing: 0.8px; }
/* column variant: lets a control carry a small helper caption underneath */
.ctrl-group.col { flex-direction: column; align-items: flex-start; gap: 2px; }
.ctrl-group.col .row { display: flex; align-items: center; gap: 6px; }
.ctrl-help { color: var(--muted); font-size: 9px; font-weight: 400; text-transform: none; letter-spacing: 0; line-height: 1.3; max-width: 180px; }
.ctrl-group .val { color: var(--text); font-size: 12px; font-weight: 700; min-width: 24px; text-align: center; }
.ctrl-group input[type=range] { width: 80px; accent-color: var(--accent); }
.ctrl-group select {
background: var(--bg);
border: 1px solid var(--border);
color: var(--text);
border-radius: 4px;
padding: 3px 6px;
font-size: 12px;
font-family: inherit;
cursor: pointer;
}
button {
background: #21262d;
border: 1px solid var(--border);
color: var(--text);
border-radius: 6px;
padding: 6px 14px;
cursor: pointer;
font-size: 12px;
font-family: inherit;
font-weight: 600;
transition: background 0.15s, border-color 0.15s;
}
button:hover { background: #2d333b; border-color: var(--accent); }
button.active { background: var(--accent); color: #fff; border-color: var(--accent); }
button:disabled { opacity: 0.4; cursor: not-allowed; }
#btn-play { min-width: 80px; }
.divider { width: 1px; height: 28px; background: var(--border); margin: 0 2px; }
/* ── MAIN LAYOUT ── */
#main {
display: grid;
grid-template-columns: 1fr 280px;
gap: 0;
min-height: calc(100vh - 100px);
}
#left-col {
display: flex;
flex-direction: column;
border-right: 1px solid var(--border);
}
/* ── PLOT AREA ── */
#plot-wrap {
flex: 1;
padding: 14px 16px;
display: flex;
flex-direction: column;
gap: 10px;
}
.plot-title {
font-size: 10px;
color: var(--muted);
text-transform: uppercase;
letter-spacing: 1px;
display: flex;
align-items: center;
gap: 10px;
}
#canvas-fit {
width: 100%;
display: block;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--func-canvas-bg);
}
/* ── LEGEND ── */
#legend {
display: flex;
gap: 14px;
flex-wrap: wrap;
font-size: 10px;
color: var(--muted);
align-items: center;
}
.leg { display: flex; align-items: center; gap: 5px; }
.leg-line { width: 22px; height: 3px; border-radius: 2px; }
.leg-dot { width: 8px; height: 8px; border-radius: 50%; }
.leg-box { width: 14px; height: 14px; border-radius: 2px; }
/* ── OOD ANNOTATION ── */
.ood-badge {
display: inline-flex;
align-items: center;
gap: 5px;
background: rgba(255,92,92,0.12);
border: 1px solid rgba(255,92,92,0.4);
border-radius: 4px;
padding: 2px 8px;
font-size: 10px;
color: var(--red);
font-weight: 700;
}
/* ── LOSS CHART ── */
#loss-chart-wrap {
padding: 10px 16px 14px;
border-top: 1px solid var(--border);
}
#canvas-loss {
width: 100%;
display: block;
border: 1px solid var(--border);
border-radius: 4px;
background: var(--func-canvas-bg);
}
/* ── RIGHT SIDEBAR ── */
#right-col {
display: flex;
flex-direction: column;
overflow-y: auto;
}
.side-section {
border-bottom: 1px solid var(--border);
padding: 12px 14px;
}
.side-title {
font-size: 10px;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 1px;
color: var(--muted);
margin-bottom: 10px;
padding-bottom: 5px;
border-bottom: 1px solid var(--border);
}
/* ── METRICS ── */
.metric-row {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 6px;
font-size: 11px;
}
.metric-label { color: var(--muted); }
.metric-val { font-weight: 700; font-size: 13px; font-family: monospace; }
.metric-val.good { color: var(--green); }
.metric-val.warn { color: var(--orange); }
.metric-val.epoch { color: var(--accent2); }
.metric-val.loss { color: var(--accent); }
.metric-val.ood-val { color: var(--red); }
/* progress bar for loss */
.loss-bar-wrap { margin: 4px 0 8px; }
.loss-bar-bg { height: 4px; background: #21262d; border-radius: 2px; overflow: hidden; }
.loss-bar-fill { height: 100%; border-radius: 2px; transition: width 0.3s, background 0.3s; }
/* ── ACTIVATION INFO CARD ── */
.act-card {
background: #12181f;
border: 1px solid var(--border);
border-radius: 6px;
padding: 10px 12px;
font-size: 11px;
color: var(--muted);
line-height: 1.6;
}
.act-card .act-name {
font-size: 13px;
font-weight: 700;
color: var(--accent);
margin-bottom: 4px;
}
.act-card .act-formula {
font-family: monospace;
color: var(--yellow);
font-size: 11px;
margin-bottom: 6px;
}
.act-card .act-extrapolation {
background: rgba(255,92,92,0.08);
border-left: 2px solid var(--red);
padding: 5px 8px;
border-radius: 0 4px 4px 0;
margin-bottom: 6px;
font-size: 10px;
color: #ff9090;
}
.act-card .act-used-in {
background: rgba(88,214,255,0.06);
border-left: 2px solid var(--accent2);
padding: 5px 8px;
border-radius: 0 4px 4px 0;
font-size: 10px;
color: var(--accent2);
line-height: 1.5;
}
.act-card .act-used-in strong { color: var(--text); }
/* ── PATCH TOGGLE ── */
.toggle-row {
display: flex;
align-items: center;
gap: 10px;
margin-top: 6px;
}
.toggle-label { font-size: 11px; color: var(--text); flex: 1; }
.toggle-sub { font-size: 10px; color: var(--muted); display: block; }
.toggle-switch {
position: relative;
width: 38px;
height: 20px;
flex-shrink: 0;
}
.toggle-switch input { opacity: 0; width: 0; height: 0; }
.toggle-slider {
position: absolute;
cursor: pointer;
inset: 0;
background: #21262d;
border: 1px solid var(--border);
border-radius: 20px;
transition: background 0.2s;
}
.toggle-slider:before {
content: '';
position: absolute;
width: 14px;
height: 14px;
left: 2px;
top: 2px;
background: var(--muted);
border-radius: 50%;
transition: transform 0.2s, background 0.2s;
}
.toggle-switch input:checked + .toggle-slider { background: rgba(255,92,92,0.25); border-color: var(--red); }
.toggle-switch input:checked + .toggle-slider:before { transform: translateX(18px); background: var(--red); }
/* ── STATUS ── */
#status-bar {
font-size: 11px;
padding: 4px 20px;
background: #0a0e14;
border-top: 1px solid var(--border);
color: var(--muted);
display: flex;
gap: 16px;
align-items: center;
min-height: 26px;
}
#status-text { color: var(--accent2); flex: 1; }
#tfjs-runner-wrap {
padding: 0 16px 16px;
}
/* act selector tabs */
.act-tabs {
display: flex;
gap: 5px;
flex-wrap: wrap;
}
.act-tab {
padding: 4px 10px;
font-size: 11px;
border-radius: 4px;
border: 1px solid var(--border);
background: transparent;
color: var(--muted);
cursor: pointer;
font-family: inherit;
font-weight: 600;
transition: all 0.15s;
}
.act-tab:hover { border-color: var(--accent); color: var(--text); background: #21262d; }
.act-tab.active { background: var(--accent); border-color: var(--accent); color: #fff; }
.target-tabs {
display: flex;
gap: 5px;
flex-wrap: wrap;
}
.target-tab {
padding: 4px 10px;
font-size: 11px;
border-radius: 4px;
border: 1px solid var(--border);
background: transparent;
color: var(--muted);
cursor: pointer;
font-family: inherit;
font-weight: 600;
transition: all 0.15s;
}
.target-tab:hover { border-color: var(--accent2); color: var(--text); background: #21262d; }
.target-tab.active { background: rgba(88,214,255,0.18); border-color: var(--accent2); color: var(--accent2); }
html[data-theme="light"] .ctrl-group,
html[data-theme="light"] button,
html[data-theme="light"] .toggle-slider,
html[data-theme="light"] .act-card,
html[data-theme="light"] #status-bar {
background: var(--surface);
color: var(--text);
border-color: var(--border);
}
html[data-theme="light"] button:hover,
html[data-theme="light"] .act-tab:hover,
html[data-theme="light"] .target-tab:hover {
background: var(--surface2);
color: var(--text);
}
html[data-theme="light"] button.active,
html[data-theme="light"] .act-tab.active {
background: var(--accent);
border-color: var(--accent);
color: #fff;
}
html[data-theme="light"] #canvas-fit,
html[data-theme="light"] #canvas-loss {
--func-canvas-bg: #f6f8fa;
}
@media (max-width: 800px) {
#main { grid-template-columns: 1fr; }
#right-col { border-top: 1px solid var(--border); }
}
@media (max-width: 768px) {
#header { padding: 10px 14px; flex-wrap: wrap; }
#header h1 { font-size: 14px; }
#controls { flex-wrap: wrap; padding: 8px 12px; gap: 8px; }
canvas { max-width: 100%; }
button { min-height: 40px; }
.target-tabs { flex-wrap: wrap; }
}
@media (max-width: 480px) {
#header h1 { font-size: 13px; }
#controls { padding: 6px 10px; }
}
/* ══════════════════════════════════════════
LESSON NARRATIVE (additive — sits BELOW the simulator)
Component styles that lesson-components.css does not already provide.
All colors read the page's existing CSS variables, so they follow the
light/dark theme automatically.
══════════════════════════════════════════ */
#lesson {
max-width: 940px;
margin: 0 auto;
padding: 28px 24px 64px;
font-family: 'Segoe UI', system-ui, sans-serif;
}
#lesson .lesson-head {
font-size: 19px;
font-weight: 700;
color: var(--text);
margin-bottom: 6px;
}
#lesson .lesson-sub { color: var(--muted); font-size: 12px; line-height: 1.7; margin-bottom: 18px; }
#lesson-tabs { margin-bottom: 18px; }
.section-h2 {
font-size: 18px;
font-weight: 700;
color: var(--text);
margin-bottom: 6px;
padding-bottom: 8px;
border-bottom: 1px solid var(--border);
}
.card {
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
padding: 16px 20px;
margin-bottom: 16px;
}
.card-title {
font-size: 13px;
font-weight: 700;
color: var(--accent2);
margin-bottom: 10px;
display: flex;
align-items: center;
gap: 8px;
}
.card-title .num-badge {
background: var(--accent);
color: #fff;
border-radius: 4px;
padding: 1px 7px;
font-size: 10px;
font-weight: 700;
}
.card p { color: var(--text); font-size: 13px; line-height: 1.75; margin-bottom: 8px; }
.card p:last-child { margin-bottom: 0; }
.card ul { margin: 8px 0 8px 18px; }
.card li { color: var(--text); font-size: 13px; line-height: 1.7; margin-bottom: 5px; }
.lesson-link { color: var(--accent2); text-decoration: none; }
.lesson-link:hover { text-decoration: underline; }
.highlight-box {
background: rgba(124,131,255,0.08);
border: 1px solid rgba(124,131,255,0.3);
border-radius: 6px;
padding: 12px 16px;
margin: 10px 0;
font-size: 12px;
line-height: 1.7;
color: var(--text);
}
.highlight-box b { color: var(--accent2); }
.analogy-box {
background: rgba(88,214,255,0.07);
border-left: 3px solid var(--accent2);
border-radius: 0 6px 6px 0;
padding: 12px 16px;
margin: 12px 0;
font-size: 12px;
line-height: 1.7;
color: var(--text);
}
.analogy-box .analogy-label {
font-size: 10px;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.8px;
color: var(--accent2);
margin-bottom: 6px;
}
.good-box {
background: rgba(57,211,83,0.07);
border-left: 3px solid var(--green);
border-radius: 0 6px 6px 0;
padding: 10px 14px;
margin: 10px 0;
font-size: 12px;
color: var(--text);
line-height: 1.65;
}
.good-box b { color: var(--green); }
.warn-box {
background: rgba(255,92,92,0.07);
border-left: 3px solid var(--red);
border-radius: 0 6px 6px 0;
padding: 10px 14px;
margin: 10px 0;
font-size: 12px;
color: var(--text);
line-height: 1.65;
}
.warn-box b { color: var(--red); }
.math-box {
background: var(--bg);
border: 1px solid var(--border);
border-radius: 6px;
padding: 12px 14px;
font-family: 'SF Mono', 'Fira Code', monospace;
font-size: 11.5px;
color: var(--yellow);
line-height: 1.9;
margin: 10px 0;
overflow-x: auto;
}
.math-box .comment { color: var(--muted); }
.math-box .hl { color: var(--accent2); font-weight: 700; }
.num-table { width: 100%; border-collapse: collapse; margin: 12px 0; font-size: 12px; }
.num-table th, .num-table td {
border: 1px solid var(--border);
padding: 7px 9px;
text-align: center;
font-family: 'SF Mono', 'Fira Code', monospace;
}
.num-table th { background: var(--panel); color: var(--accent); font-weight: 700; }
.num-table td { color: var(--text); }
.num-table td.lbl { text-align: left; color: var(--muted); font-family: 'Segoe UI', system-ui, sans-serif; }
.num-table tr.total td { color: var(--orange); font-weight: 700; background: rgba(240,136,62,0.06); }
html[data-theme="light"] .card p,
html[data-theme="light"] .card li,
html[data-theme="light"] .highlight-box,
html[data-theme="light"] .analogy-box,
html[data-theme="light"] .good-box,
html[data-theme="light"] .warn-box,
html[data-theme="light"] .num-table td { color: #1f2328; }
@media (max-width: 768px) {
#lesson { padding: 18px 14px 44px; }
.card { padding: 14px; }
.num-table { display: block; overflow-x: auto; }
}
</style>
<link rel="stylesheet" href="lesson-components.css">
<link rel="stylesheet" href="site-polish.css">
<link rel="stylesheet" href="code-runner.css">
<script src="canvas-hidpi.js"></script>
<script src="catalog.js"></script>
<script src="theme.js" defer></script>
<script src="search.js" defer></script>
<script src="nav.js" defer></script>
<script src="glossary.js" defer></script>
<script src="controls.js" defer></script>
<script src="code-runner.js" defer></script>
<script src="lesson-tabs.js" defer></script>
</head>
<body>
<!-- ══════════════════════ HEADER ══════════════════════ -->
<div id="header">
<div>
<h1>How Activations Shape What a Network Learns</h1>
<div class="subtitle">Real TF.js · sin / abs / step / x² / damped-sin · ReLU / tanh / sin / GELU · OOD regions + missing-patch test</div>
</div>
<a href="index.html">← Suite Index</a>
</div>
<!-- ══════════════════════ CONTROLS ══════════════════════ -->
<div id="controls">
<button id="btn-play">▶ Play</button>
<button id="btn-step">Step</button>
<button id="btn-reset">↺ Reset</button>
<div class="divider"></div>
<div class="ctrl-group" title="Learning rate — how big a step the optimizer takes each update. Higher = faster but can overshoot; lower = slower but more stable.">
<label>Learning rate</label>
<select id="lr-sel" data-control="select" data-state-key="lr">
<option value="0.003">0.003</option>
<option value="0.01" selected>0.01</option>
<option value="0.03">0.03</option>
<option value="0.1">0.1</option>
</select>
</div>
<div class="ctrl-group">
<label>Depth</label>
<input type="range" id="depth-slider" min="1" max="5" value="3" step="1"
data-control="range" data-state-key="depth" data-label="#depth-val" data-format="integer">
<span class="val" id="depth-val">3</span>
</div>
<div class="ctrl-group">
<label>Width</label>
<input type="range" id="width-slider" min="4" max="64" value="16" step="4"
data-control="range" data-state-key="width" data-label="#width-val" data-format="integer">
<span class="val" id="width-val">16</span>
</div>
<div class="ctrl-group col" title="Epochs per step — how many full training passes (epochs) run each time you press Step or each animation frame of Play.">
<div class="row">
<label>Epochs/step</label>
<input type="range" id="eps-slider" min="1" max="20" value="5" step="1"
data-control="range" data-state-key="eps" data-label="#eps-val" data-format="integer">
<span class="val" id="eps-val">5</span>
</div>
<div class="ctrl-help">Training epochs run per Step / Play frame</div>
</div>
</div>
<!-- ══════════════════════ MAIN LAYOUT ══════════════════════ -->
<div id="main">
<!-- LEFT: plot + loss chart -->
<div id="left-col">
<div id="plot-wrap">
<div class="plot-title">
<span>Function Fit & OOD Regions</span>
<span class="ood-badge">◀ OOD region</span>
<span class="ood-badge">OOD region ▶</span>
</div>
<!-- Function (target) selector -->
<div class="target-tabs" id="target-tabs" data-control="button-group" data-state-key="targetFn" role="group" aria-label="Target function">
<button class="target-tab active" data-fn="sin">sin(x)</button>
<button class="target-tab" data-fn="abs">|x|</button>
<button class="target-tab" data-fn="step">step(x)</button>
<button class="target-tab" data-fn="x2">x²</button>
<button class="target-tab" data-fn="dsin">damped sin</button>
</div>
<!-- Activation selector -->
<div class="act-tabs" id="act-tabs" data-control="button-group" data-state-key="actKey" role="group" aria-label="Activation function">
<button class="act-tab active" data-act="relu">ReLU</button>
<button class="act-tab" data-act="tanh">tanh</button>
<button class="act-tab" data-act="sin">sin</button>
<button class="act-tab" data-act="gelu">GELU</button>
</div>
<canvas id="canvas-fit" height="340"></canvas>
<div id="legend">
<div class="leg"><div class="leg-line" style="background:var(--accent2)"></div><span>Target function</span></div>
<div class="leg"><div class="leg-line" style="background:var(--orange)"></div><span>Network fit</span></div>
<div class="leg"><div class="leg-dot" style="background:var(--accent)"></div><span>Train samples</span></div>
<div class="leg"><div class="leg-box" style="background:rgba(255,92,92,0.15);border:1px solid rgba(255,92,92,0.4)"></div><span>OOD (extrapolation) zone</span></div>
<div class="leg"><div class="leg-box" style="background:rgba(255,219,92,0.12);border:1px dashed rgba(255,219,92,0.5)"></div><span>Missing patch</span></div>
</div>
</div>
<!-- LOSS CHART -->
<div id="loss-chart-wrap">
<div class="plot-title" style="margin-bottom:6px;">Training Loss History</div>
<canvas id="canvas-loss" height="90"></canvas>
</div>
</div>
<!-- RIGHT SIDEBAR -->
<div id="right-col">
<!-- METRICS -->
<div class="side-section">
<div class="side-title">Live Metrics</div>
<div class="metric-row">
<span class="metric-label">Epoch</span>
<span class="metric-val epoch" id="epoch-display">0</span>
</div>
<div class="metric-row">
<span class="metric-label">Train loss (in-dist)</span>
<span class="metric-val loss" id="train-loss-display">—</span>
</div>
<div class="loss-bar-wrap">
<div class="loss-bar-bg"><div class="loss-bar-fill" id="train-loss-bar" style="width:0%;background:var(--accent)"></div></div>
</div>
<div class="metric-row">
<span class="metric-label">OOD extrapolation error</span>
<span class="metric-val ood-val" id="ood-loss-display">—</span>
</div>
<div class="loss-bar-wrap">
<div class="loss-bar-bg"><div class="loss-bar-fill" id="ood-loss-bar" style="width:0%;background:var(--red)"></div></div>
</div>
<div class="metric-row">
<span class="metric-label">OOD / train ratio</span>
<span class="metric-val warn" id="ratio-display">—</span>
</div>
</div>
<!-- MISSING PATCH TOGGLE -->
<div class="side-section">
<div class="side-title">Missing Patch (Interpolation)</div>
<div class="toggle-row">
<div class="toggle-label">
Remove middle interval
<span class="toggle-sub">Deletes x ∈ [−0.5, 0.5] training points. Does the net recover the missing patch?</span>
</div>
<label class="toggle-switch">
<input type="checkbox" id="patch-toggle">
<span class="toggle-slider"></span>
</label>
</div>
<div id="patch-note" style="display:none;margin-top:8px;font-size:10px;background:rgba(255,219,92,0.06);border:1px solid rgba(255,219,92,0.3);border-radius:4px;padding:6px 8px;color:var(--yellow);line-height:1.5;">
Patch gap active. Points in x ∈ [−0.5, 0.5] removed. The network must <strong>interpolate</strong> this region from surrounding context — watch whether the orange curve recovers the shape or diverges.
</div>
</div>
<!-- ACTIVATION INFO CARD -->
<div class="side-section">
<div class="side-title">Activation: Behavior & Real-world Use</div>
<div class="act-card" id="act-info-card">
<div class="act-name" id="act-name">ReLU</div>
<div class="act-formula" id="act-formula">f(x) = max(0, x)</div>
<div class="act-extrapolation" id="act-extrap">
Extrapolates linearly — the last "slope" of each piece continues outward. Left of all training data → flat (saturated to 0). Right → grows linearly. Good for well-scoped inputs; dangerous if distribution shifts far.
</div>
<div class="act-used-in" id="act-used-in">
<strong>Used in:</strong> ResNets, VGG, most CNNs, standard MLPs — anywhere speed and simplicity matter. Default activation of the deep-learning era.
</div>
</div>
</div>
<!-- EXPLANATION -->
<div class="side-section">
<div class="side-title">Why Each Activation Extrapolates Differently</div>
<div style="font-size:10px;color:var(--muted);line-height:1.65;">
<p style="margin-bottom:7px;"><span style="color:var(--accent);font-weight:700;">ReLU</span> — piecewise linear; the network is a sum of half-planes. OOD = the last linear segment extends to infinity. No periodicity, no saturation right-ward.</p>
<p style="margin-bottom:7px;"><span style="color:var(--accent2);font-weight:700;">tanh</span> — bounded [−1,+1]. Saturates to a flat constant far from training range. OOD prediction is nearly constant — safe to not explode, but useless for true extrapolation.</p>
<p style="margin-bottom:7px;"><span style="color:var(--yellow);font-weight:700;">sin</span> — periodic. OOD can <em>continue the periodicity</em> if the network learns the right phase/frequency — the SIREN behaviour. Can over-oscillate on non-periodic targets.</p>
<p><span style="color:var(--purple);font-weight:700;">GELU</span> — smooth gate: x·Φ(x). Behaves like ReLU for large positive x (nearly linear); like 0 for large negative x. Less jagged than ReLU, but still extrapolates approximately linearly past the training range.</p>
</div>
</div>
</div><!-- /right-col -->
</div><!-- /main -->
<div id="tfjs-runner-wrap">
<div class="code-runner" data-runtime="tfjs" data-title="Run a tiny TF.js regression step">
<script type="text/plain" class="code-runner-source">
const xs = tf.tensor2d([-2, -1, 0, 1, 2, 3], [6, 1]);
const ys = tf.tensor2d([4, 1, 0, 1, 4, 9], [6, 1]);
const model = tf.sequential();
model.add(tf.layers.dense({ units: 8, activation: "relu", inputShape: [1] }));
model.add(tf.layers.dense({ units: 1 }));
model.compile({ optimizer: tf.train.adam(0.08), loss: "meanSquaredError" });
const before = model.evaluate(xs, ys).dataSync()[0];
await model.fit(xs, ys, { epochs: 80, verbose: 0 });
const after = model.evaluate(xs, ys).dataSync()[0];
const query = tf.tensor2d([-1.5, 0.5, 2.5], [3, 1]);
const preds = model.predict(query);
log(`loss before: ${before.toFixed(4)}`);
log(`loss after: ${after.toFixed(4)}`);
log("predictions for -1.5, 0.5, 2.5:");
log(preds);
tf.dispose([xs, ys, query, preds]);
model.dispose();
</script>
</div>
</div>
<!-- STATUS BAR -->
<div id="status-bar">
<span id="status-text">Ready — press Play or Step to begin training.</span>
<span id="arch-display" style="color:var(--accent);font-size:10px;"></span>
</div>
<!-- ══════════════════════════════════════════════════════════════════
LESSON NARRATIVE (ADDITIVE — the simulator above is unchanged)
Tabbed deep-dive driven by the shared lesson-tabs.js controller via
data-tabs / data-tab-target / data-tab-panel. No effect on the TF.js viz.
══════════════════════════════════════════════════════════════════ -->
<div id="lesson" data-tabs="funcapprox" data-tabs-hash data-tab-active-class="active">
<div class="lesson-head">Function Approximation — the idea behind the simulator</div>
<div class="lesson-sub">The plot above is the whole of deep learning in one picture: a network is just a tunable function, and training bends it until it matches a target. This section explains <em>why</em> a pile of simple units can copy almost any curve, what you are actually watching, and a fully worked-by-hand example you can verify with a calculator.</div>
<div id="lesson-tabs" class="nav-bar" role="tablist" aria-label="Function-approximation lesson sections"
style="display:flex; gap:0; flex-wrap:wrap; border-bottom:1px solid var(--border);">
<button class="nav-tab active" data-tab-target="fa-prereq">1 · Prerequisites</button>
<button class="nav-tab" data-tab-target="fa-idea">2 · The Big Idea</button>
<button class="nav-tab" data-tab-target="fa-read">3 · Reading the Simulator</button>
<button class="nav-tab" data-tab-target="fa-worked">4 · Worked Example</button>
<button class="nav-tab" data-tab-target="fa-depth">5 · Width vs. Depth</button>
</div>
<!-- ── 1 · PREREQUISITES ───────────────────────────────────────────── -->
<section class="section-panel" data-tab-panel="fa-prereq">
<div class="section-h2">Prerequisites</div>
<div class="section-intro">This lesson is the payoff of four earlier ones. If any row below feels shaky, follow its link first — none takes long, and they make the simulator above read like a story instead of a wall of knobs.</div>
<div class="prereq-row">
<div class="prereq-concept">What a function is</div>
<div class="prereq-why">A function is just a rule that turns an input into an output — feed it an <code>x</code>, get back a <code>y</code>. The blue curve above is a target function (e.g. <code>y = sin(x)</code>); the network is a second function with adjustable knobs. "Approximation" means making the network's output close to the target's at every <code>x</code> we care about. Build the visual intuition for inputs, outputs, and slopes in the <a class="lesson-link" href="math_visual_lab.html">Math Visual Lab</a>.</div>
</div>
<div class="prereq-row">
<div class="prereq-concept">Weights & a forward pass</div>
<div class="prereq-why">Each layer multiplies its input by learnable <em>weights</em>, adds a <em>bias</em>, and passes the result on. Running an input through every layer to get a prediction is a <em>forward pass</em>. The orange curve is one forward pass evaluated across the whole x-axis at once. See it happen weight-by-weight in <a class="lesson-link" href="watch_network_train.html">Watch a Network Train</a>.</div>
</div>
<div class="prereq-row">
<div class="prereq-concept">Activations & nonlinearity</div>
<div class="prereq-why">After each weighted sum, a network applies a nonlinear <em>activation</em> (ReLU, tanh, …). This is the load-bearing ingredient: without it, stacking layers does nothing (next tab proves why). The activation buttons above swap which nonlinearity the hidden units use — watch how the fit's "texture" changes. Compare them side by side in the <a class="lesson-link" href="activation_loss_explorer.html">Activation & Loss Explorer</a>.</div>
</div>
<div class="prereq-row">
<div class="prereq-concept">Fitting by gradient descent</div>
<div class="prereq-why">"Training" means measuring how wrong the fit is (the <em>loss</em>, here mean-squared error) and nudging every weight a little in the direction that lowers it, over and over. The loss chart below the plot is that number falling. The mechanics — gradients, learning rate, epochs — are in <a class="lesson-link" href="watch_network_train.html">Watch a Network Train</a> and the <a class="lesson-link" href="optimizer_playground.html">Optimizer Playground</a>.</div>
</div>
<div class="highlight-box" style="margin-top:16px;">
<b>One-line map:</b> a function is a rule <code>x → y</code>; a network is a function with knobs; <b>activations</b> are what let those knobs bend the line into a curve; <b>gradient descent</b> turns the knobs until the curve matches the target. Everything else on this page is detail on those four ideas.
</div>
</section>
<!-- ── 2 · THE BIG IDEA (Universal Approximation) ──────────────────── -->
<section class="section-panel" data-tab-panel="fa-idea">
<div class="section-h2">The Big Idea: stack simple bends, get any curve</div>
<div class="section-intro">Why should adding more identical little units let a network copy a sine wave, a step, or a jagged signal? Two facts answer it — one about why nonlinearity is non-negotiable, and one classic theorem about what a single layer of nonlinear units can reach.</div>
<div class="card">
<div class="card-title"><span class="num-badge">1</span>Without a nonlinearity, depth is an illusion</div>
<p>A layer with no activation is a linear map: <code>output = W·input + b</code>. Stack two of them and you get <code>W₂(W₁x + b₁) + b₂</code>, which multiplies out to <code>(W₂W₁)x + (W₂b₁ + b₂)</code> — that is just <em>one</em> linear map with a combined weight matrix. A hundred linear layers still collapse to a single straight-line (or flat-plane) function. It could never bend to follow <code>sin(x)</code>, and famously cannot even separate XOR.</p>
<p>The nonlinear activation between layers is what breaks this collapse. Once each unit can "switch on" only past a threshold (ReLU) or saturate (tanh), the layers stop merging and the network can assemble genuinely curved, composite shapes. <b style="color:var(--accent2);">Nonlinearity is not a tuning detail — it is the entire reason depth buys you anything.</b></p>
<div class="math-box"><span class="comment"># two linear layers, no activation — multiply them out</span><br>
f(x) = W₂·(W₁·x + b₁) + b₂<br>
= <span class="hl">(W₂W₁)</span>·x + <span class="hl">(W₂b₁ + b₂)</span> <span class="comment"># ← one matrix, one bias: still linear</span><br>
<br><span class="comment"># insert a nonlinearity σ and the collapse is broken</span><br>
f(x) = W₂·<span class="hl">σ(</span>W₁·x + b₁<span class="hl">)</span> + b₂ <span class="comment"># now it can curve</span></div>
<p style="margin-top:4px;">Flip the activation buttons above to <em>any</em> setting and the fit can chase the curve. There is no "linear" option on purpose — a linear network is the one configuration that <em>cannot</em> do this job.</p>
</div>
<div class="card">
<div class="card-title"><span class="num-badge">2</span>The Universal Approximation Theorem</div>
<p>The reassuring result: a feed-forward network with a <em>single hidden layer</em> of nonlinear units can approximate <em>any</em> continuous function on a closed interval to any accuracy you like — given enough units. Cybenko proved it in 1989 for sigmoid activations; Hornik showed in 1991 that the magic is the multi-layer architecture itself, not the specific activation — essentially any non-polynomial activation works (ReLU, tanh, sigmoid, GELU all qualify).</p>
<div class="analogy-box">
<div class="analogy-label">Intuition</div>
Think of approximating a smooth curve with a row of narrow rectangular tiles (a Riemann sum) — enough thin tiles trace any shape. Each pair of nonlinear units can build one such localized "tile" or bend; sum enough of them and you tile out the whole target. <em>Width</em> = how many tiles you have. The Worked Example tab builds one tile by hand from three ReLU units.
</div>
<p>There is a sibling "arbitrary-depth" version too: with a bounded width and unlimited depth, ReLU networks of width just <code>n + 4</code> (for <code>n</code> inputs) are also universal approximators. Two roads to the same destination — pile up width, or pile up depth.</p>
<div class="warn-box">
<b>The honest small print:</b> the theorem is an <em>existence</em> result. It promises a good-enough network <em>exists</em>; it does <strong>not</strong> say how many units you need, and it says <strong>nothing</strong> about whether gradient descent will actually <em>find</em> it. "Can be approximated" ≠ "will be learned." That gap — between what is representable and what is trainable — is exactly what you feel when you shrink the width above and the fit degrades, or when the out-of-distribution region goes wild.
</div>
</div>
</section>
<!-- ── 3 · READING THE SIMULATOR ───────────────────────────────────── -->
<section class="section-panel" data-tab-panel="fa-read">
<div class="section-h2">Reading the Simulator</div>
<div class="section-intro">Everything in the plot maps onto one idea. Here is what each color, region, and knob is telling you — so the live picture above becomes legible at a glance.</div>
<div class="card">
<div class="card-title">The two curves and the dots</div>
<ul>
<li><b style="color:var(--accent2);">Blue line</b> — the <em>target</em> function you picked (sin, |x|, step, x², damped-sin). This is the ground truth the network is trying to copy.</li>
<li><b style="color:var(--orange);">Orange line</b> — the network's current output, swept across the whole x-axis. At the start it is nearly a straight, random line; as training runs it bends to overlap the blue.</li>
<li><b style="color:var(--accent);">Blue dots</b> — the training samples the network actually sees (120 points inside the training range). The network only ever gets these; the curve everywhere else is its <em>guess</em>.</li>
</ul>
<p>When orange lies on top of blue through the dotted training band, the network has learned the function <em>there</em>. The whole drama is watching it get there — and watching what it does where it has no data.</p>
</div>
<div class="card">
<div class="card-title">The red OOD bands and the loss chart</div>
<p>The shaded red strips on the far left and right are <em>out-of-distribution</em> (OOD) regions — x-values <em>beyond</em> the training data. The network was never shown points there, so its behavior is pure extrapolation, and it depends entirely on the activation: ReLU shoots off in a straight line, tanh flattens to a constant, sin keeps oscillating. The "OOD extrapolation error" metric quantifies how badly it guesses out there. <b style="color:var(--accent2);">Key lesson: a network learns the shape of its data, not the underlying law</b> — push past the data and the guarantee evaporates.</p>
<p>The lower <em>Training Loss History</em> chart plots two lines over time: the in-distribution loss (falls fast as the fit locks on) and the OOD error (often stays high or grows). A big gap between them is the visual signature of a model that interpolates well but cannot extrapolate.</p>
</div>
<div class="card">
<div class="card-title">The knobs — and the experiment to run</div>
<ul>
<li><b>Width</b> — units per layer = how many "bends/tiles" are available. Too few and the orange curve is too stiff to follow a wiggly target (underfitting).</li>
<li><b>Depth</b> — number of layers = how many times bends get composed on top of bends.</li>
<li><b>Activation</b> — which nonlinearity each unit uses; this is what sets the extrapolation personality.</li>
<li><b>Missing-patch toggle</b> — deletes the middle slice of training data so you can watch whether the net <em>interpolates</em> across a gap (it usually does, smoothly) versus <em>extrapolates</em> past the edges (it usually can't).</li>
</ul>
<div class="good-box">
<b>Try this:</b> pick the <code>sin</code> target with <code>ReLU</code>, set width to the minimum (4), and Play. The fit is a crude set of straight segments. Now raise width to 32 — the segments multiply and the orange curve smooths into the sine. You just watched the Universal Approximation Theorem happen: more units → finer piecewise approximation. Then switch the activation to <code>sin</code> and watch the OOD bands suddenly extrapolate correctly, because a periodic activation matches a periodic target.
</div>
</div>
</section>
<!-- ── 4 · WORKED EXAMPLE ──────────────────────────────────────────── -->
<section class="section-panel" data-tab-panel="fa-worked">
<div class="section-h2">Worked Example: building one "bump" from three ReLU units by hand</div>
<div class="section-intro">The theorem says "sum enough bends and you get any curve." Let's make that concrete with numbers you can check on a calculator: a one-hidden-layer network with three ReLU units that produces a single triangular bump — the localized tile the approximation argument is built from. No training, no mystery: we just pick the weights and watch the units compose.</div>
<div class="card">
<div class="card-title"><span class="num-badge">1</span>The three units</div>
<p>Our network has one input <code>x</code>, three hidden ReLU units, and one linear output. Recall <code>ReLU(z) = max(0, z)</code> — it passes positive values through and clamps everything negative to 0. We hand-pick these hidden units (each is <code>ReLU(x − shift)</code>, i.e. a ramp that "switches on" at a different x):</p>
<div class="math-box">h₁ = ReLU(x − 0) <span class="comment"># ramp turns on at x = 0</span><br>
h₂ = ReLU(x − 1) <span class="comment"># ramp turns on at x = 1</span><br>
h₃ = ReLU(x − 2) <span class="comment"># ramp turns on at x = 2</span><br>
<br><span class="comment"># output layer: a weighted sum of the three ramps</span><br>
y = (1)·h₁ + (−2)·h₂ + (1)·h₃</div>
<p>Three ramps with output weights <code>+1, −2, +1</code>. That tiny combination is the whole trick. Let's evaluate it.</p>
</div>
<div class="card">
<div class="card-title"><span class="num-badge">2</span>Evaluate at a few points</div>
<p>Plug in several x-values. Each cell is just <code>max(0, x − shift)</code>, then the bottom row sums them with weights <code>+1, −2, +1</code>:</p>
<table class="num-table">
<thead>
<tr><th>x</th><th>h₁ = ReLU(x)</th><th>h₂ = ReLU(x−1)</th><th>h₃ = ReLU(x−2)</th><th>y = h₁ − 2h₂ + h₃</th></tr>
</thead>
<tbody>
<tr><td>−1</td><td>0</td><td>0</td><td>0</td><td class="">0</td></tr>
<tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>
<tr><td>0.5</td><td>0.5</td><td>0</td><td>0</td><td>0.5</td></tr>
<tr class="total"><td>1</td><td>1</td><td>0</td><td>0</td><td>1 ← peak</td></tr>
<tr><td>1.5</td><td>1.5</td><td>0.5</td><td>0</td><td>0.5</td></tr>
<tr><td>2</td><td>2</td><td>1</td><td>0</td><td>0</td></tr>
<tr><td>3</td><td>3</td><td>2</td><td>1</td><td>0</td></tr>
</tbody>
</table>
<p>Read the last column: the output is 0 everywhere up to <code>x = 0</code>, rises to a peak of <code>1</code> at <code>x = 1</code>, falls back to 0 by <code>x = 2</code>, and stays 0 afterward. That is a clean <b style="color:var(--orange);">triangular bump on [0, 2] peaking at (1, 1)</b> — assembled purely from three straight ReLU ramps. The <code>−2</code> middle weight is what bends the rising ramp back down; the <code>+1</code> at the end flattens the tail.</p>
</div>
<div class="card">
<div class="card-title"><span class="num-badge">3</span>From one bump to any curve</div>
<p>This is the Universal Approximation Theorem in miniature. One bump is a tile. Add three more units to make a second bump somewhere else, scale each bump's height with its output weight, and place enough of them and you can outline any continuous curve — exactly like sketching a function with a row of adjustable triangles. To trace <code>sin(x)</code>, you would stack a positive bump over its hump and a negative one under its trough, and so on.</p>
<div class="good-box">
<b>Connect it to the simulator:</b> "Width" up top is literally <em>how many of these ramps the network has to work with</em>. With ReLU selected, look closely at the orange fit at low width — it is visibly made of straight segments meeting at kinks, each kink the point where one ReLU unit switched on. Training does by gradient descent what we did by hand here: it <em>discovers</em> the shifts and output weights that tile the target. The only differences are that the network finds the numbers itself, and it has many more tiles.</p>
</div>
<p>And the limits tab's warning is visible here too: each ReLU ramp is a <em>straight line</em> past its kink, so far outside <code>[0, 2]</code> the building blocks only ever produce straight lines — which is exactly why a ReLU network extrapolates linearly into the OOD bands.</p>
</div>
</section>
<!-- ── 5 · WIDTH VS DEPTH ──────────────────────────────────────────── -->
<section class="section-panel" data-tab-panel="fa-depth">
<div class="section-h2">Width vs. Depth — two ways to add power</div>
<div class="section-intro">The simulator gives you both a Width and a Depth slider. They both increase what the network can represent, but they are not interchangeable. Here is the difference, and what the research actually shows.</div>
<div class="card">
<div class="card-title">Width: more tiles, side by side</div>
<p>Adding width adds more units in the same layer — more bends available <em>at once</em>, laid out across the input. This is the lever the single-hidden-layer theorem pulls: in principle, enough width alone can approximate anything. In the simulator, widening a 1-layer-ish fit visibly adds more straight segments to the orange ReLU curve, smoothing it toward the target. The cost is that pure width can be wildly inefficient — some functions need an enormous number of units in a shallow net.</p>
</div>
<div class="card">
<div class="card-title">Depth: bends composed on bends</div>
<p>Adding depth feeds one layer's bends into the next, so later layers bend combinations of earlier bends. This compounding is why deep networks can be dramatically more parameter-efficient than shallow ones. The classic result (depth-efficiency / depth separation) is striking: there are functions a depth-3 network represents with a modest, polynomial number of units that a depth-2 network would need an <em>exponential</em> number of units to match. Slightly reducing depth can force a huge increase in width to keep the same accuracy.</p>
<div class="highlight-box">
<b>The asymmetry:</b> trading depth for width is expensive (you may pay an exponential blow-up in width), while trading width for depth is comparatively cheap. In short — for many real targets, <b>depth is the more efficient way to buy expressive power</b>, which is a big part of why the field is called <em>deep</em> learning. (Lu et al., "The Expressive Power of Neural Networks: A View from the Width", NeurIPS 2017; Vardi et al., "Width is Less Important than Depth in ReLU Neural Networks", 2022.)
</div>
</div>
<div class="card">
<div class="card-title">What to feel in the simulator</div>
<ul>
<li>On a simple target (a single bend like <code>|x|</code>), a little width at depth 1–2 already fits well — extra depth is overkill.</li>
<li>On a harder target (damped-sin, with several wiggles), increasing <em>depth</em> often locks the shape on faster and cleaner than the same parameter budget spent on raw width.</li>
<li>Push either too far for the data and you stop learning the law and start memorizing the 120 dots — the cue for the next lesson, <a class="lesson-link" href="generalization_train_test.html">Generalization: Train / Val / Test</a>.</li>
</ul>
<div class="good-box"><b>Bottom line:</b> width and depth both raise the ceiling on what's representable; depth tends to raise it more cheaply; and neither one guarantees the network will <em>learn</em> the target or behave outside its data. The simulator lets you feel all three of those truths in about a minute.</div>
</div>
<div class="card">
<div class="card-title">Sources</div>
<p style="font-size:12px;">
Universal Approximation Theorem (Cybenko 1989; Hornik 1991; arbitrary-depth width-<code>n+4</code> ReLU form; existence-only caveat) — <a class="lesson-link" href="https://en.wikipedia.org/wiki/Universal_approximation_theorem">Wikipedia: Universal approximation theorem</a>.<br>
Depth vs. width expressivity and depth-efficiency — <a class="lesson-link" href="https://arxiv.org/abs/1709.02540">Lu, Pu, Wang, Hu & Wang, "The Expressive Power of Neural Networks: A View from the Width" (arXiv 1709.02540, NeurIPS 2017)</a> and <a class="lesson-link" href="https://arxiv.org/abs/2202.03841">Vardi, Reichman, Pitassi & Shamir, "Width is Less Important than Depth in ReLU Neural Networks" (arXiv 2202.03841)</a>.<br>
Why stacked linear layers collapse / the need for nonlinearity — standard result; see <a class="lesson-link" href="https://en.wikipedia.org/wiki/Universal_approximation_theorem">the theorem's non-polynomial-activation requirement</a> (a polynomial-only or linear network is not a universal approximator).
</p>
</div>
</section>
</div>
<!-- ══════════════════════ SCRIPT ══════════════════════ -->
<script>
// ═══════════════ CONFIG ═══════════════
const TRAIN_X_MIN = -3, TRAIN_X_MAX = 3; // training range
const OOD_X_MIN = -5, OOD_X_MAX = 5; // full plot range
const N_TRAIN = 120;
const PATCH_MIN = -0.5, PATCH_MAX = 0.5;
// ═══════════════ STATE ═══════════════
let model = null;
let epoch = 0;
let running = false;
let runTimer = null;
let lossHistory = [], trainLossHistory = [], oodLossHistory = [];
let targetFn = 'sin';