Skip to content

Latest commit

 

History

History
299 lines (220 loc) · 7.01 KB

QUICKSTART.md

File metadata and controls

299 lines (220 loc) · 7.01 KB

Quick Start Guide - Micrograd Java

What You Just Built

You now have a complete Java implementation of micrograd - a tiny autograd engine that can:

  1. Track gradients automatically through mathematical operations
  2. Train neural networks using backpropagation
  3. Build computational graphs dynamically

Project Overview

Core Components

  1. Value.java - The heart of the autograd engine

    • Wraps scalar values
    • Tracks operations to build computational graph
    • Computes gradients via backpropagation
  2. Module.java - Base class for neural network components

    • Provides parameters() and zeroGrad() methods
  3. Neural Network Classes (nn/ package)

    • Neuron.java - Single neuron with weights and bias
    • Layer.java - Layer of multiple neurons
    • MLP.java - Multi-Layer Perceptron (full network)

Running the Examples

1. Basic Autograd Example

mvn exec:java -Dexec.mainClass="micrograd.examples.BasicExample"

This demonstrates:

  • Simple gradient computation
  • Complex mathematical expressions
  • Activation functions (ReLU, tanh)

Sample Output:

Example 1: Simple Gradients
Computing: f(a, b) = a * b + a
a.grad (∂d/∂a) = -2.0
b.grad (∂d/∂b) = 2.0

2. Neural Network Training Example

java -cp target/classes micrograd.examples.NeuralNetworkExample

This shows:

  • Building a 3-layer neural network
  • Training loop with backpropagation
  • Gradient descent optimization

Sample Output:

Epoch   0: Loss = 6.123456
Epoch  10: Loss = 3.456789
...
Epoch  99: Loss = 2.666667

3. Run Tests

mvn test

All 10 tests should pass, validating:

  • Basic arithmetic operations
  • Gradient computations
  • Activation functions
  • Chain rule implementation

Quick Usage Examples

Example 1: Simple Gradient Computation

import micrograd.Value;

// Create values
Value a = new Value(2.0);
Value b = new Value(-3.0);

// Build expression: c = a * b + a
Value c = a.mul(b).add(a);

// Compute gradients
c.backward();

// Access gradients
System.out.println("∂c/∂a = " + a.getGrad()); // -2.0
System.out.println("∂c/∂b = " + b.getGrad()); // 2.0

Example 2: Building a Neural Network

import micrograd.nn.MLP;
import micrograd.Value;
import java.util.*;

// Create network: 3 inputs -> 4 hidden -> 1 output
MLP model = new MLP(3, 4, 1);

// Prepare input
List<Value> input = Arrays.asList(
    new Value(1.0),
    new Value(2.0),
    new Value(-1.0)
);

// Forward pass
List<Value> output = model.call(input);

// Compute loss (e.g., MSE)
Value target = new Value(1.0);
Value loss = output.get(0).sub(target).pow(2);

// Backpropagation
model.zeroGrad();
loss.backward();

// Gradient descent update
double learningRate = 0.01;
for (Value p : model.parameters()) {
    p.setData(p.getData() - learningRate * p.getGrad());
}

Example 3: Custom Activation Functions

Value x = new Value(0.5);

// ReLU
Value relu = x.relu();
System.out.println("ReLU(0.5) = " + relu.getData()); // 0.5

// Tanh
Value tanh = x.tanh();
System.out.println("tanh(0.5) = " + tanh.getData()); // 0.462...

// Custom: sigmoid(x) = 1 / (1 + e^(-x))
Value negX = x.mul(-1);
Value expNegX = negX.exp();
Value onePlusExp = expNegX.add(1);
Value sigmoid = new Value(1.0).div(onePlusExp);

Understanding the Code

How Backpropagation Works

When you call value.backward():

  1. Topological Sort: Orders all operations from input to output
  2. Initialize Output Gradient: Sets value.grad = 1.0
  3. Backward Pass: Traverses in reverse, applying chain rule
  4. Accumulate Gradients: Each variable accumulates gradients from all paths

Key Operations and Their Gradients

Operation Forward Gradient
a.add(b) a + b ∂L/∂a += ∂L/∂out, ∂L/∂b += ∂L/∂out
a.mul(b) a × b ∂L/∂a += b × ∂L/∂out, ∂L/∂b += a × ∂L/∂out
a.pow(n) a^n ∂L/∂a += n × a^(n-1) × ∂L/∂out
a.relu() max(0, a) ∂L/∂a += (a > 0 ? 1 : 0) × ∂L/∂out
a.tanh() tanh(a) ∂L/∂a += (1 - tanh²(a)) × ∂L/∂out

Training Tips

1. Learning Rate

  • Too high: Training diverges
  • Too low: Training is slow
  • Typical range: 0.001 to 0.1
double learningRate = 0.01; // Adjust based on results

2. Network Architecture

// Small problem: 2-3 layers
MLP small = new MLP(2, 4, 1);

// Medium problem: 3-4 layers
MLP medium = new MLP(10, 32, 16, 1);

// More complex: deeper networks
MLP deep = new MLP(20, 64, 32, 16, 1);

3. Training Loop Pattern

for (int epoch = 0; epoch < epochs; epoch++) {
    // 1. Forward pass - compute predictions
    // 2. Compute loss
    // 3. Zero gradients
    model.zeroGrad();
    // 4. Backward pass
    loss.backward();
    // 5. Update parameters
    for (Value p : model.parameters()) {
        p.setData(p.getData() - learningRate * p.getGrad());
    }
}

Next Steps

Extend the Library

  1. Add more activations: Sigmoid, LeakyReLU, etc.
  2. Implement optimizers: SGD with momentum, Adam
  3. Add regularization: L2 regularization, dropout
  4. Better loss functions: Cross-entropy, hinge loss

Example: Adding Sigmoid

// In Value.java
public Value sigmoid() {
    Value negX = this.mul(-1);
    Value expNegX = negX.exp();
    Value onePlusExp = expNegX.add(1);
    Value out = new Value(1.0).div(onePlusExp);
    
    out.backward = (self) -> {
        double s = out.data;
        this.grad += s * (1 - s) * out.grad;
    };
    
    return out;
}

Common Pitfalls

  1. Forgetting to zero gradients: Always call model.zeroGrad() before each backward pass
  2. Not calling backward: Gradients are only computed after loss.backward()
  3. Reusing values across iterations: Create fresh Value objects for each training iteration
  4. Wrong learning rate: Start with 0.01 and adjust

Debugging Tips

Print Computational Graph

Value x = new Value(2.0);
x.setLabel("x");

Value y = x.pow(2);
y.setLabel("y");

System.out.println("Value: " + y);
System.out.println("Previous nodes: " + y.getPrev().size());
System.out.println("Operation: " + y.getOp());

Monitor Gradients

loss.backward();

for (Value p : model.parameters()) {
    System.out.printf("Param: %.4f, Grad: %.4f\n", 
        p.getData(), p.getGrad());
}

Resources

Summary

You've successfully created a working autograd engine in Java! This is the same fundamental technology that powers modern deep learning frameworks like PyTorch and TensorFlow, just simplified to work with scalar values instead of tensors.

The key concepts you've learned:

  • ✅ Automatic differentiation
  • ✅ Computational graphs
  • ✅ Backpropagation
  • ✅ Gradient descent
  • ✅ Neural network architecture

Happy learning! 🚀