When you write:
Value a = new Value(2.0);
Value b = new Value(-3.0);
Value c = a.mul(b);
Value d = c.add(a);This creates a computational graph:
Forward Pass (computing values):
a = 2.0 b = -3.0
| |
| |
+-------[*]------+
|
c = -6.0
|
| a = 2.0
| |
+--[+]--+
|
d = -4.0
When you call d.backward():
Backward Pass (computing gradients):
Initialize: d.grad = 1.0
Step 1: [+] operation
d.grad = 1.0
↓
c.grad += d.grad = 1.0
a.grad += d.grad = 1.0
Step 2: [*] operation
c.grad = 1.0
↓
a.grad += b.data × c.grad = -3.0 × 1.0 = -3.0
b.grad += a.data × c.grad = 2.0 × 1.0 = 2.0
Final gradients:
a.grad = 1.0 + (-3.0) = -2.0 (accumulated from both paths!)
b.grad = 2.0
MLP(3, 4, 4, 1) creates:
Input Layer (3 neurons)
↓
[Neuron 1: w1, w2, w3, b] → ReLU
[Neuron 2: w1, w2, w3, b] → ReLU } Hidden Layer 1 (4 neurons)
[Neuron 3: w1, w2, w3, b] → ReLU
[Neuron 4: w1, w2, w3, b] → ReLU
↓
[Neuron 5: w1, w2, w3, w4, b] → ReLU
[Neuron 6: w1, w2, w3, w4, b] → ReLU } Hidden Layer 2 (4 neurons)
[Neuron 7: w1, w2, w3, w4, b] → ReLU
[Neuron 8: w1, w2, w3, w4, b] → ReLU
↓
[Neuron 9: w1, w2, w3, w4, b] → Linear } Output Layer (1 neuron)
↓
Output (1 value)
Total parameters: (3×4 + 4) + (4×4 + 4) + (4×1 + 1) = 16 + 20 + 5 = 41 parameters
Neuron with 3 inputs:
x1 ──[w1]──┐
x2 ──[w2]──┤
x3 ──[w3]──┼──[Σ]──[+ b]──[ReLU]──> output
│
[bias]
Computation:
1. Weighted sum: z = w1×x1 + w2×x2 + w3×x3 + b
2. Activation: output = ReLU(z) = max(0, z)
Gradients flow backward through:
- ReLU: ∂L/∂z = ∂L/∂output × (z > 0 ? 1 : 0)
- Bias: ∂L/∂b = ∂L/∂z
- Weights: ∂L/∂wi = ∂L/∂z × xi
- Inputs: ∂L/∂xi = ∂L/∂z × wi
┌─────────────────────────────────────┐
│ Start Training Loop (epoch = 0) │
└────────────┬────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 1. Forward Pass │
│ - Feed inputs through network │
│ - Compute predictions │
└────────────┬────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 2. Compute Loss │
│ - Compare predictions vs targets │
│ - loss = Σ(prediction - target)² │
└────────────┬────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 3. Zero Gradients │
│ - model.zeroGrad() │
│ - Clear all previous gradients │
└────────────┬────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 4. Backward Pass │
│ - loss.backward() │
│ - Compute all gradients │
└────────────┬────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 5. Update Parameters │
│ - for each parameter p: │
│ - p -= learning_rate × p.grad │
└────────────┬────────────────────────┘
│
▼
┌────┴────┐
│ epoch++ │
└────┬────┘
│
┌────▼────────┐
│ More epochs?│
└────┬────┬───┘
Yes│ │No
│ │
▼ ▼
(loop) (end)
Expression: f(x) = (x + 1)²
Graph:
x ──[+1]──> t ──[²]──> f
Forward:
x = 2.0 → t = 3.0 → f = 9.0
Backward (chain rule):
∂f/∂f = 1.0
↓
∂f/∂t = ∂f/∂f × ∂(t²)/∂t = 1.0 × 2t = 1.0 × 6.0 = 6.0
↓
∂f/∂x = ∂f/∂t × ∂(x+1)/∂x = 6.0 × 1 = 6.0
Verify: f(x) = (x+1)² = x² + 2x + 1
∂f/∂x = 2x + 2 = 2(2) + 2 = 6.0 ✓
Input range: x ∈ [-2, 2]
ReLU(x):
2 │ ╱
│ ╱
1 │ ╱
│ ╱
0 ├─────────
│
-1 │
│
-2 │_______________
-2 -1 0 1 2
tanh(x):
1 │ ────────
│ ╱
0 ├───╯
│ ╱
-1 │──────
-2 -1 0 1 2
Properties:
- ReLU: Fast, simple, but gradient is 0 for x < 0
- tanh: Smooth, bounded [-1, 1], symmetric around 0
Value object for: c = a * b
┌─────────────────────────────┐
│ Value instance │
├─────────────────────────────┤
│ data: -6.0 │ ← The actual value
│ grad: 0.0 → 1.0 (after .backward())
│ op: "*" │ ← Operation that created this
│ prev: {a, b} │ ← Parent values in graph
│ backward: (Consumer<Value>) │ ← Function to compute gradients
│ label: "c" │ ← Optional name for debugging
└─────────────────────────────┘
# Python micrograd
a = Value(2.0)
b = Value(-3.0)
c = a * b + a
c.backward()
print(a.grad) # -2.0// Java micrograd
Value a = new Value(2.0);
Value b = new Value(-3.0);
Value c = a.mul(b).add(a);
c.backward();
System.out.println(a.getGrad()); // -2.0Key differences:
- Python uses operator overloading (
a * b) - Java uses explicit methods (
a.mul(b)) - Same underlying algorithm and concepts!