Update docs with temperature and ResNet50 findings

- Temperature ablation: T=4 optimal, T=6/T=8 perform worse
- ResNet50 conv1+KD: 77-81% gap recovery (vs 88-123% for ResNet18)
- Bump experiment count to 153
- Update CLAUDE.md with lambda server workflow

Author: dariocazzani

CLAUDE.md

@@ -14,7 +14,7 @@ Research project studying **BitNet b1.58** (1.58-bit ternary quantization) appli
 
 ## Project Structure
 
-```
+```text
 experiments/
 ├── train.py          # Standard training script
 ├── train_kd.py       # Knowledge Distillation training
@@ -59,7 +59,7 @@ uv run python -m analysis.generate_figures
 
 ## Results Directory Structure
 
-```
+```text
 results/raw/{dataset}/{model}/{version}_s{seed}/
 ├── config.json      # Training config
 ├── results.json     # Final metrics
@@ -69,15 +69,20 @@ results/raw/{dataset}/{model}/{version}_s{seed}/
 
 ## Server Workflow
 
-Code lives locally, experiments run on remote server with GPUs.
+Code lives locally, experiments run on `lambda` server with GPUs.
 
 ```bash
-# On server: archive results (JSON only, not model weights)
+# On server (lambda): archive results (JSON only, not model weights)
 find results/raw -name "*.json" | tar -czvf results_json.tar.gz -T -
 
 # Locally: download and extract
-scp server:~/code/.../results_json.tar.gz .
+scp lambda:/home/dcazzani/code/lab-strange-loop/bitnet/results_json.tar.gz .
 tar -xzvf results_json.tar.gz
+
+# Regenerate analysis artifacts
+uv run python -m analysis.aggregate_results   # reads raw/, saves processed/
+uv run python -m analysis.generate_tables     # reads raw/, writes paper/tables/
+uv run python -m analysis.generate_figures    # reads raw/, writes paper/figures/
 ```
 
 ## Important Notes

PLAN.md

@@ -1,8 +1,36 @@
 # BitNet CNN Research Plan
 
-**Last Updated**: Feb 2, 2026
-**Primary Target**: WACV 2027 Round 2 (Sept 2026)
+**Last Updated**: Feb 5, 2026
+**Primary Target**: CVPR 2026 Workshop (~Apr 2026)
 **Parallel Track**: NeurIPS 2026 Efficient ML Workshop (Aug 2026)
+**Backup**: WACV 2027 Round 2 (Sept 2026)
+
+---
+
+## Paper Status: 📝 FIRST DRAFT COMPLETE
+
+**Title**: "When Augmentation Fails: Knowledge Distillation for Ternary CNNs"
+
+**Paper file**: `paper/main.tex` (13 pages, builds successfully)
+
+### Sections Completed
+
+- ✅ Title and author info (Dario Cazzani, Cisco Systems Inc.)
+- ✅ Abstract
+- ✅ Introduction (framing the augmentation paradox)
+- ✅ Related Work (with TTQ, QKD, HAQ citations)
+- ✅ Method (BitNet b1.58 implementation details)
+- ✅ Section 4: "What Doesn't Work: The Augmentation Paradox"
+- ✅ Section 5: "What Works: A Practical Recipe" (conv1 + KD)
+- ✅ Discussion
+- ✅ Conclusion
+- ✅ Reproducibility Appendix (code → results → paper pipeline)
+
+### Still Needed
+
+- 📝 Complete figures (Fig 4: Recipe comparison bar chart)
+- 📝 Polish and proofread
+- 🔄 Wait for ImageNet validation results
 
 ---
 
@@ -12,9 +40,11 @@ Systematic study of BitNet b1.58 (1.58-bit ternary quantization) applied to stan
 
 **Core Narrative**: "For ternary CNNs, invest in distillation not augmentation."
 
+**Practical Recipe**: Keep conv1 in FP32 + apply KD → recovers 88% of accuracy gap on CIFAR-10, **exceeds FP32 by 1.0% on CIFAR-100**.
+
 ---
 
-## Current State (134 experiments completed)
+## Current State (153 experiments completed)
 
 ### Results Summary
 
@@ -88,7 +118,38 @@ Systematic study of BitNet b1.58 (1.58-bit ternary quantization) applied to stan
 
 **Status**: ✅ Complete (6 runs: 2 datasets × 3 seeds)
 
-### Key Finding #5: ImageNet Validation 🔄 IN PROGRESS
+### Key Finding #5: conv1 + KD Combo ⭐ COMPLETE
+
+**Combining conv1 in FP32 with KD recovers 88-123% of the accuracy gap.**
+
+**CIFAR-10 Results (ResNet18)**:
+| Method | Accuracy | Gap Recovery |
+|--------|----------|--------------|
+| FP32 (target) | 88.89% | - |
+| BitNet (baseline) | 85.40% | 0% |
+| BitNet + KD | 86.66 ± 0.18% | 36% |
+| BitNet + keep_conv1 | 87.40% | 58% |
+| **BitNet + keep_conv1 + KD** | **88.48 ± 0.17%** | **88%** |
+
+**CIFAR-100 Results (ResNet18)** ⭐ NEW:
+| Method | Accuracy | Gap Recovery |
+|--------|----------|--------------|
+| FP32 (target) | 62.40% | - |
+| BitNet (baseline) | 58.06% | 0% |
+| BitNet + KD | 60.55 ± 0.21% | 57% |
+| BitNet + keep_conv1 | 61.27% | 74% |
+| **BitNet + keep_conv1 + KD** | **63.40 ± 0.09%** | **123% (exceeds FP32!)** |
+
+**Key insights**:
+
+- Benefits are largely additive: conv1 (58%) + KD (36%) = 94% theoretical, achieved 88% on CIFAR-10
+- On CIFAR-100, the combo **exceeds FP32 by 1.0 percentage point** (63.40% vs 62.40%)
+- KD provides stronger regularization on harder tasks, pushing beyond FP32 baseline
+- This is the **practical recipe**: keep conv1 in FP32 + apply KD
+
+**Status**: ✅ Complete (6 runs: ResNet18 × CIFAR-10/100 × seeds 42, 123, 456)
+
+### Key Finding #6: ImageNet Validation 🔄 IN PROGRESS
 
 **Goal**: Validate that findings scale beyond CIFAR to large-scale datasets.
 
@@ -104,11 +165,11 @@ Systematic study of BitNet b1.58 (1.58-bit ternary quantization) applied to stan
 
 **YES**: "Systematic analysis of BitNet scaling to standard deep CNN architectures (11M-25M params), revealing limitations not apparent in smaller networks"
 
-### Paper Framing Options
+### Paper Framing ✅ FINALIZED
+
+**Title**: "When Augmentation Fails: Knowledge Distillation for Ternary CNNs"
 
-1. "The Augmentation Paradox: Why Data Augmentation Fails to Help Ternary Neural Networks"
-2. "BitNet b1.58 for CNNs: A Comprehensive Evaluation and Layer-Wise Analysis"
-3. "When 1.58 Bits Aren't Enough: Understanding the Limits of Extreme CNN Quantization"
+**Core message**: Standard data augmentation doesn't help ternary CNNs, but conv1 in FP32 + knowledge distillation recovers 88% of the accuracy gap on CIFAR-10 and **exceeds FP32 accuracy on CIFAR-100**.
 
 ---
 
@@ -121,7 +182,7 @@ Systematic study of BitNet b1.58 (1.58-bit ternary quantization) applied to stan
 | **Layer-wise ablation** (quantize all-but-one layer) | 2-3 days | **High** | ✅ DONE |
 | **Efficiency proxy metrics** (BOPs, memory, model size) | Hours | Medium | ✅ DONE |
 | **KD experiment** (FP32 teacher → BitNet student) | 1-2 weeks | **High** | ✅ DONE (36% gap recovery) |
-| **conv1 + KD combo** (practical recipe) | ~6h | **High** | 🎯 PRIORITY |
+| **conv1 + KD combo** (practical recipe) | ~6h | **High** | ✅ DONE (88% gap recovery) |
 | **TTQ comparison** (address in Related Work) | Text only | **High** | 📝 Write explanation |
 
 ### Tier 2: Strengthens Paper
@@ -130,17 +191,35 @@ Systematic study of BitNet b1.58 (1.58-bit ternary quantization) applied to stan
 |------------|--------|--------|--------|
 | ImageNet validation (ResNet-18, 2 seeds) | 24-48h each | Medium | 🔄 Running |
 | KD on CIFAR-100 (harder task) | ~6h | Medium | ✅ DONE (57% gap recovery) |
+| **conv1 + KD on CIFAR-100** | ~6h | **High** | ✅ DONE (123% gap recovery - exceeds FP32!) |
 
 ### Tier 3: Optional (If Time Permits)
 
 | Experiment | Effort | Signal | Status |
 |------------|--------|--------|--------|
-| KD on ResNet50/CIFAR-10 | ~12h | Low | Not started |
-| KD on ResNet50/CIFAR-100 | ~12h | Low | Not started |
+| KD on ResNet50/CIFAR-10 | ~12h | Low | ✅ DONE (77% recovery) |
+| KD on ResNet50/CIFAR-100 | ~12h | Low | ✅ DONE (81% recovery) |
 | Implement TTQ/DoReFa for direct comparison | 1-2 weeks | Medium | Not started |
 | Real inference measurements (BitBLAS) | Days | Medium | Not started |
 | Additional architectures (MobileNetV2) | Days | Low | Not started |
 
+### Key Finding #7: ResNet50 Recipe Validation ⭐ NEW
+
+**The conv1+KD recipe generalizes to larger models, though with lower recovery.**
+
+| Model | Dataset | Result | FP32 | Gap Recovery |
+|-------|---------|--------|------|--------------|
+| ResNet50 | CIFAR-10 | 89.46% | 90.42% | **77%** |
+| ResNet50 | CIFAR-100 | 63.87% | 65.58% | **81%** |
+
+**Comparison with ResNet18:**
+- ResNet18 CIFAR-10: 88% recovery
+- ResNet18 CIFAR-100: 123% recovery (exceeds FP32!)
+- ResNet50 CIFAR-10: 77% recovery
+- ResNet50 CIFAR-100: 81% recovery
+
+**Insight**: Larger models are harder to fully recover. More parameters = more information lost to quantization. ResNet18 remains the sweet spot for ternary deployment.
+
 ### TTQ Comparison Strategy
 
 **Don't implement TTQ.** Instead, explain the design tradeoff in Related Work:
@@ -153,32 +232,82 @@ This reframes the gap as an **intentional design tradeoff** for deployment effic
 
 ## Immediate Next Steps
 
-### 🎯 PRIORITY 1: conv1 + KD Combo Experiment
+### 🎯 PRIORITY 1: Deep Literature Research ✅ COMPLETE
+
+**Status**: All 4 research prompts completed and integrated into paper.
+
+| File | Topic | Key Finding |
+|------|-------|-------------|
+| `01_ttq_comparison.md` | TTQ vs BitNet gap | TTQ uses learned asymmetric scales (W^p, W^n) + keeps conv1/FC in FP32. Our gap explained by simpler formulation. |
+| `02_layer_sensitivity_literature.md` | Layer sensitivity | conv1 FP32 is standard since 2016. **Our contribution: precise quantification (54-74%)**. |
+| `03_kd_for_quantization.md` | KD literature | T=4 may be suboptimal (T=6-8 better). **Feature distillation could add 5-20% more recovery**. |
+| `04_bitnet_cnn_prior_work.md` | Prior BitNet CNN work | Novelty confirmed: first ResNet study with full training + augmentation analysis. |
+
+**Paper updates made**:
+- ✅ Related Work rewritten with proper framing and 6 new citations
+- ✅ Introduction updated to acknowledge conv1 FP32 as established practice
+- ✅ Contributions list refined ("quantified layer sensitivity" not "discovery")
+- ✅ Discussion added TTQ comparison paragraph (design tradeoff)
+- ✅ Future Work expanded with feature distillation and temperature tuning
+
+### 🎯 PRIORITY 2: New Experiments from Research Findings
+
+Based on KD research, these experiments could further improve results:
+
+#### 2a. Higher Temperature KD ✅ COMPLETE
+
+**Results**: T=4 is already optimal. Higher temperatures perform slightly worse.
+
+| Temperature | Accuracy |
+|-------------|----------|
+| T=4 (default) | **88.66%** |
+| T=6 | 88.23% |
+| T=8 | 88.34% |
+
+**Conclusion**: No benefit from higher temperatures. Our default T=4 is optimal for ternary networks.
+
+#### 2b. CIFAR-100 conv1+KD (Validates Recipe) ✅ COMPLETE
+
+**Results**: 63.40 ± 0.09% accuracy - **exceeds FP32 (62.40%) by 1.0 percentage point!**
+
+| Seed | Accuracy |
+|------|----------|
+| 42 | 63.41% |
+| 123 | 63.48% |
+| 456 | 63.30% |
+
+**Gap recovery**: 123% (exceeds 100% because KD regularization pushes beyond FP32)
+
+**Key insight**: On harder tasks, KD provides stronger regularization, enabling ternary networks to surpass full-precision baselines.
 
-**Goal**: Test if combining conv1 in FP32 with KD gives additive benefits.
+#### 2c. Feature Distillation (Bigger Effort, Bigger Gain)
+**Goal**: Implement DCQ/OFF-style feature distillation.
 
-**Hypothesis**: conv1 (58% recovery) + KD (36% recovery) could recover ~85-90% of the gap.
+**Potential gain**: 5-20% additional improvement (research consensus).
 
-**Method**:
+**Implementation**:
+- Add intermediate feature extraction to teacher/student
+- Cosine similarity loss on conv layer outputs
+- Combined loss: α×KD_logits + (1-α)×CE + β×feature_loss
 
-- Add `--ablation keep_conv1` support to `train_kd.py`
-- Run 3 seeds on ResNet18/CIFAR-10
-- Compare: baseline → +conv1 → +KD → +both
+**Status**: Parked for now. Could push from 88% to ~95% recovery. Good for future work or camera-ready revision.
 
-**Expected outcome**: Practical recipe for maximum accuracy with minimal overhead.
+### 🎯 PRIORITY 3: Paper Polish
 
-### 🎯 PRIORITY 2: TTQ Explanation in Paper
+**Goal**: Final polish before submission.
 
-**Goal**: Address why BitNet underperforms TTQ (which claimed to beat FP32).
+**Tasks**:
 
-**Approach**: Explain in Related Work as design tradeoff, not implement.
+- [ ] Generate Fig 4 (Recipe comparison bar chart)
+- [ ] Verify paper compiles with new citations
+- [ ] Final proofread
+- [ ] Wait for ImageNet validation results
 
 ---
 
 ### Currently Running
 
 - 🔄 **ImageNet validation**: 4 runs (ResNet18, seeds 42/123, FP32/BitNet)
-- ✅ **KD on CIFAR-100**: 60.55 ± 0.21% (+2.49%, 57% gap recovery) - **KD more effective on harder tasks!**
 
 ---
 
@@ -187,8 +316,15 @@ This reframes the gap as an **intentional design tradeoff** for deployment effic
 - ✅ **Layer-wise ablation**: conv1 accounts for 54-74% of accuracy gap (confirmed across all configs)
 - ✅ **Ablation generalization**: Tested on ResNet18/50 × CIFAR-10/100 (36 ablation runs)
 - ✅ **Efficiency metrics**: 20.3x compression, 64x theoretical speedup
-- ✅ **134 total experiments**: Main experiments + comprehensive ablation study
-- ✅ **Knowledge Distillation**: CIFAR-10 (36% recovery), CIFAR-100 (57% recovery) - KD more effective on harder tasks
+- ✅ **Knowledge Distillation**: CIFAR-10 (36% recovery), CIFAR-100 (57% recovery)
+- ✅ **conv1 + KD combo (CIFAR-10)**: 88.48 ± 0.17% accuracy (88% gap recovery)
+- ✅ **conv1 + KD combo (CIFAR-100)**: 63.40 ± 0.09% accuracy (**exceeds FP32 by 1.0%!**)
+- ✅ **Temperature ablation**: T=4 is optimal, T=6/T=8 perform worse
+- ✅ **ResNet50 recipe validation**: 77-81% gap recovery (lower than ResNet18 but still substantial)
+- ✅ **153 total experiments**: Main + ablation + KD studies
+- ✅ **Paper first draft**: All main sections written (13 pages)
+- ✅ **Research prompts**: Created 4 deep research prompts for AI collaboration
+- ✅ **Reproducibility appendix**: Documented code → results → paper pipeline
 
 ---
 
@@ -209,12 +345,13 @@ This reframes the gap as an **intentional design tradeoff** for deployment effic
 
 | Month | Tasks |
 |-------|-------|
-| **Jan-Feb** | ✅ Layer-wise ablation, efficiency metrics, KD experiment |
-| **Feb** | conv1+KD combo, ImageNet results, CIFAR-100 KD |
-| **Mar-Apr** | Paper writing, figures, internal review |
-| **May-Jun** | Revisions, polish |
-| **Jul-Aug** | Submit to NeurIPS 2026 Efficient ML Workshop |
-| **Sept** | Submit to WACV 2027 Round 2 |
+| **Jan** | ✅ Layer-wise ablation, efficiency metrics, KD experiment |
+| **Feb (now)** | ✅ conv1+KD combo, ✅ Paper first draft, 🔄 ImageNet, 📝 CIFAR-100 conv1+KD |
+| **Feb-Mar** | Deep research integration, figures, CIFAR-100 conv1+KD results |
+| **Mar** | Polish, internal review |
+| **~Apr** | Submit to **CVPR 2026 Workshop** (primary target) |
+| **Aug** | Submit to NeurIPS 2026 Efficient ML Workshop (if needed) |
+| **Sept** | Submit to WACV 2027 Round 2 (backup) |
 
 ---
 
@@ -229,8 +366,9 @@ This reframes the gap as an **intentional design tradeoff** for deployment effic
 
 | File | Purpose |
 |------|---------|
+| `paper/main.tex` | LaTeX paper (first draft complete) |
 | `paper/notes.md` | Detailed paper notes and TODO |
-| `paper/main.tex` | LaTeX paper skeleton |
+| `paper/research/` | Deep research prompts for AI collaboration |
 | `paper/prompts_for_review.md` | Prompts for external review |
 | `results/processed/aggregated.csv` | Current experiment results |