Skip to content

Latest commit

 

History

History
253 lines (211 loc) · 6.93 KB

VISUAL_GUIDE.md

File metadata and controls

253 lines (211 loc) · 6.93 KB

Micrograd-Java: Visual Explanation

Computational Graph Example

When you write:

Value a = new Value(2.0);
Value b = new Value(-3.0);
Value c = a.mul(b);
Value d = c.add(a);

This creates a computational graph:

Forward Pass (computing values):
    
    a = 2.0          b = -3.0
      |                |
      |                |
      +-------[*]------+
              |
          c = -6.0
              |
              |     a = 2.0
              |       |
              +--[+]--+
                  |
              d = -4.0

Backpropagation Flow

When you call d.backward():

Backward Pass (computing gradients):

Initialize: d.grad = 1.0

Step 1: [+] operation
    d.grad = 1.0
    ↓
    c.grad += d.grad = 1.0
    a.grad += d.grad = 1.0

Step 2: [*] operation  
    c.grad = 1.0
    ↓
    a.grad += b.data × c.grad = -3.0 × 1.0 = -3.0
    b.grad += a.data × c.grad = 2.0 × 1.0 = 2.0

Final gradients:
    a.grad = 1.0 + (-3.0) = -2.0  (accumulated from both paths!)
    b.grad = 2.0

Neural Network Architecture

MLP(3, 4, 4, 1) creates:

Input Layer (3 neurons)
    ↓
[Neuron 1: w1, w2, w3, b] → ReLU
[Neuron 2: w1, w2, w3, b] → ReLU    } Hidden Layer 1 (4 neurons)
[Neuron 3: w1, w2, w3, b] → ReLU
[Neuron 4: w1, w2, w3, b] → ReLU
    ↓
[Neuron 5: w1, w2, w3, w4, b] → ReLU
[Neuron 6: w1, w2, w3, w4, b] → ReLU    } Hidden Layer 2 (4 neurons)
[Neuron 7: w1, w2, w3, w4, b] → ReLU
[Neuron 8: w1, w2, w3, w4, b] → ReLU
    ↓
[Neuron 9: w1, w2, w3, w4, b] → Linear  } Output Layer (1 neuron)
    ↓
Output (1 value)

Total parameters: (3×4 + 4) + (4×4 + 4) + (4×1 + 1) = 16 + 20 + 5 = 41 parameters

Single Neuron Detail

Neuron with 3 inputs:

x1 ──[w1]──┐
x2 ──[w2]──┤
x3 ──[w3]──┼──[Σ]──[+ b]──[ReLU]──> output
           │
         [bias]

Computation:
1. Weighted sum: z = w1×x1 + w2×x2 + w3×x3 + b
2. Activation: output = ReLU(z) = max(0, z)

Gradients flow backward through:
- ReLU: ∂L/∂z = ∂L/∂output × (z > 0 ? 1 : 0)
- Bias: ∂L/∂b = ∂L/∂z
- Weights: ∂L/∂wi = ∂L/∂z × xi
- Inputs: ∂L/∂xi = ∂L/∂z × wi

Training Loop Visualization

┌─────────────────────────────────────┐
│  Start Training Loop (epoch = 0)    │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│  1. Forward Pass                    │
│  - Feed inputs through network      │
│  - Compute predictions              │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│  2. Compute Loss                    │
│  - Compare predictions vs targets   │
│  - loss = Σ(prediction - target)²  │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│  3. Zero Gradients                  │
│  - model.zeroGrad()                 │
│  - Clear all previous gradients     │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│  4. Backward Pass                   │
│  - loss.backward()                  │
│  - Compute all gradients            │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│  5. Update Parameters               │
│  - for each parameter p:            │
│  -   p -= learning_rate × p.grad    │
└────────────┬────────────────────────┘
             │
             ▼
        ┌────┴────┐
        │ epoch++ │
        └────┬────┘
             │
        ┌────▼────────┐
        │ More epochs?│
        └────┬────┬───┘
          Yes│    │No
             │    │
             ▼    ▼
          (loop) (end)

How Gradients Flow: Chain Rule Example

Expression: f(x) = (x + 1)²

Graph:
x ──[+1]──> t ──[²]──> f

Forward:
x = 2.0  →  t = 3.0  →  f = 9.0

Backward (chain rule):
∂f/∂f = 1.0
       ↓
∂f/∂t = ∂f/∂f × ∂(t²)/∂t = 1.0 × 2t = 1.0 × 6.0 = 6.0
       ↓
∂f/∂x = ∂f/∂t × ∂(x+1)/∂x = 6.0 × 1 = 6.0

Verify: f(x) = (x+1)² = x² + 2x + 1
        ∂f/∂x = 2x + 2 = 2(2) + 2 = 6.0 ✓

Activation Functions Compared

Input range: x ∈ [-2, 2]

ReLU(x):
  2 │         ╱
    │       ╱
  1 │     ╱
    │   ╱
  0 ├─────────
    │
 -1 │
    │
 -2 │_______________
   -2  -1  0  1  2

tanh(x):
  1 │      ────────
    │    ╱
  0 ├───╯
    │  ╱
 -1 │──────
   -2  -1  0  1  2

Properties:
- ReLU: Fast, simple, but gradient is 0 for x < 0
- tanh: Smooth, bounded [-1, 1], symmetric around 0

Memory Layout: Value Object

Value object for: c = a * b

┌─────────────────────────────┐
│ Value instance              │
├─────────────────────────────┤
│ data: -6.0                  │ ← The actual value
│ grad: 0.0 → 1.0 (after .backward())
│ op: "*"                     │ ← Operation that created this
│ prev: {a, b}                │ ← Parent values in graph
│ backward: (Consumer<Value>) │ ← Function to compute gradients
│ label: "c"                  │ ← Optional name for debugging
└─────────────────────────────┘

Comparison: Python vs Java

# Python micrograd
a = Value(2.0)
b = Value(-3.0)
c = a * b + a
c.backward()
print(a.grad)  # -2.0
// Java micrograd
Value a = new Value(2.0);
Value b = new Value(-3.0);
Value c = a.mul(b).add(a);
c.backward();
System.out.println(a.getGrad());  // -2.0

Key differences:

  • Python uses operator overloading (a * b)
  • Java uses explicit methods (a.mul(b))
  • Same underlying algorithm and concepts!