[SomeB1oody]

Summary

This PR adds 3 activations: GELU, SiLU (also named Swish), and Mish. Each one is an Activation variant and a standalone layer. Dense, all 10 convolution layers, SimpleRNN, LSTM, and GRU accept them.

Why the activation contract changes

Before this PR, Activation::backward read the activated output a alone. The derivative of GELU, SiLU, and Mish has no closed form in a, because these functions have the shape a = z * g(z). They need the pre-activation z.

• Activation::saves() returns an ActivationInput, which names the tensor that the backward pass reads. It is PreActivation for the 3 new variants and Output for the 12 others.
• Activation::forward_train(z) returns the output and an ActivationCache. The cache holds a copy of the output, or z itself.
• Activation::backward(&cache, &grad) returns Error::InvalidInput for a cache that holds the other tensor. A wrong tensor therefore gives an error, not a wrong gradient.
• A host layer still caches 1 activation tensor. An inference pass makes no cache and no copy.

Changes

• Activations.
  • Activation::GELU { approximate }: the exact form uses erfcf from libm, so the negative tail keeps its precision. The tanh approximation uses x * sigmoid(2u), which does not cancel in the negative tail.
  • Activation::SiLU and Activation::Mish.
• Standalone layers. GELU::new() gives the exact form, and with_approximate(true) selects the tanh approximation. The other 2 layers are SiLU::new() and Mish::new().
• Host layers. Each host layer caches an ActivationCache in place of the activated output. The fused ReLU path of Dense and SimpleRNN stays.
• Recurrent layers. RnnCell::RECORD_SLOTS is now the method record_slots(). SimpleRNN records z for each timestep with the 3 new activations. LSTM and GRU record the candidate pre-activation in place of the candidate.
• Docs. The guide (chapters 1.5, 3.2, 3.7, and 6.2, in English and Chinese) and the CHANGELOG are updated.

Example

use rustyml::neural_network::layers::activation::GELU;
use rustyml::neural_network::layers::{Activation, Dense};

let exact = Dense::new(64, Activation::GELU { approximate: false }).unwrap();
let tanh = Dense::new(64, GELU::new().with_approximate(true)).unwrap();
let silu = Dense::new(64, Activation::SiLU).unwrap();

Breaking changes

• Activation::backward takes &ActivationCache in place of the activated output tensor. Get the cache from Activation::forward_train(z).
• Activation::backward returns Error::ShapeMismatch for a gradient of another shape. Before, most variants panicked.
• Dense applies the activation to the output at its restored rank, not to the folded rank-2 matrix. The result is the same, because an embedded Softmax accepts only axis -1.