You now have a complete Java implementation of micrograd - a tiny autograd engine that can:
- Track gradients automatically through mathematical operations
- Train neural networks using backpropagation
- Build computational graphs dynamically
-
Value.java- The heart of the autograd engine- Wraps scalar values
- Tracks operations to build computational graph
- Computes gradients via backpropagation
-
Module.java- Base class for neural network components- Provides
parameters()andzeroGrad()methods
- Provides
-
Neural Network Classes (
nn/package)Neuron.java- Single neuron with weights and biasLayer.java- Layer of multiple neuronsMLP.java- Multi-Layer Perceptron (full network)
mvn exec:java -Dexec.mainClass="micrograd.examples.BasicExample"This demonstrates:
- Simple gradient computation
- Complex mathematical expressions
- Activation functions (ReLU, tanh)
Sample Output:
Example 1: Simple Gradients
Computing: f(a, b) = a * b + a
a.grad (∂d/∂a) = -2.0
b.grad (∂d/∂b) = 2.0
java -cp target/classes micrograd.examples.NeuralNetworkExampleThis shows:
- Building a 3-layer neural network
- Training loop with backpropagation
- Gradient descent optimization
Sample Output:
Epoch 0: Loss = 6.123456
Epoch 10: Loss = 3.456789
...
Epoch 99: Loss = 2.666667
mvn testAll 10 tests should pass, validating:
- Basic arithmetic operations
- Gradient computations
- Activation functions
- Chain rule implementation
import micrograd.Value;
// Create values
Value a = new Value(2.0);
Value b = new Value(-3.0);
// Build expression: c = a * b + a
Value c = a.mul(b).add(a);
// Compute gradients
c.backward();
// Access gradients
System.out.println("∂c/∂a = " + a.getGrad()); // -2.0
System.out.println("∂c/∂b = " + b.getGrad()); // 2.0import micrograd.nn.MLP;
import micrograd.Value;
import java.util.*;
// Create network: 3 inputs -> 4 hidden -> 1 output
MLP model = new MLP(3, 4, 1);
// Prepare input
List<Value> input = Arrays.asList(
new Value(1.0),
new Value(2.0),
new Value(-1.0)
);
// Forward pass
List<Value> output = model.call(input);
// Compute loss (e.g., MSE)
Value target = new Value(1.0);
Value loss = output.get(0).sub(target).pow(2);
// Backpropagation
model.zeroGrad();
loss.backward();
// Gradient descent update
double learningRate = 0.01;
for (Value p : model.parameters()) {
p.setData(p.getData() - learningRate * p.getGrad());
}Value x = new Value(0.5);
// ReLU
Value relu = x.relu();
System.out.println("ReLU(0.5) = " + relu.getData()); // 0.5
// Tanh
Value tanh = x.tanh();
System.out.println("tanh(0.5) = " + tanh.getData()); // 0.462...
// Custom: sigmoid(x) = 1 / (1 + e^(-x))
Value negX = x.mul(-1);
Value expNegX = negX.exp();
Value onePlusExp = expNegX.add(1);
Value sigmoid = new Value(1.0).div(onePlusExp);When you call value.backward():
- Topological Sort: Orders all operations from input to output
- Initialize Output Gradient: Sets
value.grad = 1.0 - Backward Pass: Traverses in reverse, applying chain rule
- Accumulate Gradients: Each variable accumulates gradients from all paths
| Operation | Forward | Gradient |
|---|---|---|
a.add(b) |
a + b |
∂L/∂a += ∂L/∂out, ∂L/∂b += ∂L/∂out |
a.mul(b) |
a × b |
∂L/∂a += b × ∂L/∂out, ∂L/∂b += a × ∂L/∂out |
a.pow(n) |
a^n |
∂L/∂a += n × a^(n-1) × ∂L/∂out |
a.relu() |
max(0, a) |
∂L/∂a += (a > 0 ? 1 : 0) × ∂L/∂out |
a.tanh() |
tanh(a) |
∂L/∂a += (1 - tanh²(a)) × ∂L/∂out |
- Too high: Training diverges
- Too low: Training is slow
- Typical range:
0.001to0.1
double learningRate = 0.01; // Adjust based on results// Small problem: 2-3 layers
MLP small = new MLP(2, 4, 1);
// Medium problem: 3-4 layers
MLP medium = new MLP(10, 32, 16, 1);
// More complex: deeper networks
MLP deep = new MLP(20, 64, 32, 16, 1);for (int epoch = 0; epoch < epochs; epoch++) {
// 1. Forward pass - compute predictions
// 2. Compute loss
// 3. Zero gradients
model.zeroGrad();
// 4. Backward pass
loss.backward();
// 5. Update parameters
for (Value p : model.parameters()) {
p.setData(p.getData() - learningRate * p.getGrad());
}
}- Add more activations: Sigmoid, LeakyReLU, etc.
- Implement optimizers: SGD with momentum, Adam
- Add regularization: L2 regularization, dropout
- Better loss functions: Cross-entropy, hinge loss
// In Value.java
public Value sigmoid() {
Value negX = this.mul(-1);
Value expNegX = negX.exp();
Value onePlusExp = expNegX.add(1);
Value out = new Value(1.0).div(onePlusExp);
out.backward = (self) -> {
double s = out.data;
this.grad += s * (1 - s) * out.grad;
};
return out;
}- Forgetting to zero gradients: Always call
model.zeroGrad()before each backward pass - Not calling backward: Gradients are only computed after
loss.backward() - Reusing values across iterations: Create fresh
Valueobjects for each training iteration - Wrong learning rate: Start with
0.01and adjust
Value x = new Value(2.0);
x.setLabel("x");
Value y = x.pow(2);
y.setLabel("y");
System.out.println("Value: " + y);
System.out.println("Previous nodes: " + y.getPrev().size());
System.out.println("Operation: " + y.getOp());loss.backward();
for (Value p : model.parameters()) {
System.out.printf("Param: %.4f, Grad: %.4f\n",
p.getData(), p.getGrad());
}- Original Python Micrograd: https://github.com/karpathy/micrograd
- Andrej Karpathy's Tutorial: https://www.youtube.com/watch?v=VMj-3S1tku0
- Backpropagation Explained: https://cs231n.github.io/optimization-2/
You've successfully created a working autograd engine in Java! This is the same fundamental technology that powers modern deep learning frameworks like PyTorch and TensorFlow, just simplified to work with scalar values instead of tensors.
The key concepts you've learned:
- ✅ Automatic differentiation
- ✅ Computational graphs
- ✅ Backpropagation
- ✅ Gradient descent
- ✅ Neural network architecture
Happy learning! 🚀