Add GELU, SiLU and Mish activations

CHANGELOG.md

@@ -13,6 +13,8 @@ Please view [SomeB1oody/RustyML](https://github.com/SomeB1oody/RustyML) for more
 - **`LayerPath`, the address of 1 layer inside a model.** It holds the model position and the sublayer names below it, and its text form is `2.forward`. `Ctx::layer_path` gives the current path. `Ctx::state_left_under` lists the state values of a model position that no layer took back. `neural_network` re-exports `LayerPath`, `Sublayer`, `SublayerMut`, and `SublayerName`.
 - **Public tree walks for a layer that a caller drives by hand.** `layer_path::walk` and `walk_mut` visit every node of 1 layer tree in pre-order. `update_tree` runs `Optimizer::update` on every node, and `apply_state_tree` moves the state of every node. `Ctx::state_left_below` lists the state values at a path, or below it, that no layer took back.
 - **`layer_path::total_param_count` sums the parameters of a layer and all of its sublayers.** `Sequential::summary` and `Graph::summary` use it. `LayerBase::param_count` still counts the arrays of 1 layer alone.
+- **3 new activations: GELU, SiLU, and Mish.** `Activation::GELU { approximate }` is `x * Phi(x)`, where `Phi` is the standard normal cumulative distribution function. With `approximate: true` it uses the tanh approximation `x * sigmoid(2 * sqrt(2/pi) * (x + 0.044715 * x^3))`. `Activation::SiLU`, also named Swish, is `x * sigmoid(x)`. `Activation::Mish` is `x * tanh(softplus(x))`. Each one is also a standalone layer: `GELU`, `SiLU`, and `Mish`. `GELU::new()` gives the exact form, and `with_approximate(true)` selects the tanh approximation. `Dense`, all 10 convolution layers, `SimpleRNN`, `LSTM`, and `GRU` accept the 3 new activations.
+- **`ActivationInput` and `ActivationCache` name the tensor that the backward pass of an activation reads.** `Activation::saves()` returns `ActivationInput::PreActivation` for GELU, SiLU, and Mish, and `ActivationInput::Output` for every other variant. `Activation::forward_train(z)` returns the activated output and an `ActivationCache` that holds the saved tensor. `Activation::output_of(&cache)` gives the activated output that a cache stands for. A host layer caches 1 activation tensor, never both.
 
 ### Changed
 - **Breaking: `ParamId` is `{ layer: LayerPath, name }` in place of `{ scope, name }`, and it is no longer `Copy`.** `ParamId::new(scope, name)` still addresses the layer at a model position. `ParamId::at(path, name)` and `LayerPath::param(name)` address a sublayer. `ParamId` implements `Display` and prints its checkpoint path.
@@ -21,6 +23,8 @@ Please view [SomeB1oody/RustyML](https://github.com/SomeB1oody/RustyML) for more
 - **Breaking: `Ctx::has_state` and `Ctx::state_slot` take `&LayerPath` in place of a model position.** Pass `&LayerPath::root(scope)` for the layer at a model position.
 - **Breaking: `MODEL_FORMAT_VERSION` is 4, and no version 3 checkpoint loads.** `LayerCheckpoint` gains a `sublayers` field of the new `SublayerCheckpoint` records, so a file holds the same tree as the model. A checkpoint path is `<layer path>.<name>`, such as `2.kernel` or `2.forward.kernel`. A layer with no sublayer keeps the path it had in version 3. Rebuild the model, retrain, and save again.
 - **Breaking: `weight_path` moves from `layers::checkpoint` to `layer_path`, and it takes a `&LayerPath` in place of a model position.**
+- **Breaking: `Activation::backward` takes `&ActivationCache` in place of the activated output tensor.** Get the cache from `Activation::forward_train(z)`. The inference path still calls `Activation::forward(&z)`, which makes no cache and no copy.
+- **Behavior change: `Activation::backward` returns an error for a bad argument.** It returns `Error::InvalidInput` for a cache that holds the other tensor than `Activation::saves()` names. It returns `Error::ShapeMismatch` for a gradient whose shape is not the shape of the cache. Before, a gradient of another shape caused a panic in most variants.
 - **Behavior change: a state value that no layer takes back stops the training step.** `Sequential::train_batch` and `Graph::train_batch` return `Error::Computation`, and the message names each value. Before, a debug assertion caught this case, and a release build ignored the value.
 - **Behavior change: `UnaryLayer::forward_mut` and `Layer::forward_many_mut` return `Error::Computation` for a state value that no layer takes back.**
 - **Behavior change: a training step refuses a sublayer call that reaches another layer than the tree holds at its path.** A training pass records the layer of each `Ctx::sublayer` call. The model compares each record against the tree after the forward pass and after the backward pass. A call of sublayer `b` under the name of sublayer `a` therefore stops the step, also when `a` and `b` have the same type and shape.

Cargo.toml

@@ -19,7 +19,7 @@ autobenches = false
 # Classical machine learning algorithms for supervised and unsupervised learning
 machine_learning = ["math", "dep:ndarray", "dep:rayon", "dep:ndarray-rand", "dep:ahash", "dep:serde", "dep:postcard", "dep:thiserror"]
 # Complete neural network framework with flexible architecture design
-neural_network = ["math", "dep:ndarray", "dep:rayon", "dep:ndarray-rand", "dep:indicatif", "dep:serde", "dep:postcard", "dep:thiserror"]
+neural_network = ["math", "dep:ndarray", "dep:rayon", "dep:ndarray-rand", "dep:indicatif", "dep:libm", "dep:serde", "dep:postcard", "dep:thiserror"]
 # Data preprocessing and dimensionality reduction utilities
 utils = ["math", "dep:ndarray", "dep:rayon", "dep:ahash", "dep:serde", "dep:postcard", "dep:ndarray-rand", "dep:thiserror"]
 # Comprehensive evaluation metrics for model performance assessment
@@ -44,6 +44,7 @@ serde = { version = "1.0.229", features = ["derive"], optional = true }
 postcard = { version = "1.1.3", features = ["use-std"], optional = true }
 thiserror = { version = "2.0.20", optional = true }
 gemmkit-ndarray = { version = "0.1.2", features = ["epilogue"], optional = true }
+libm = { version = "0.2.16", optional = true }
 
 [dev-dependencies]
 approx = "0.5.1"

guide/en/src/Chapter-01/1.5._The_Prelude_and_Imports.md

@@ -66,7 +66,7 @@ The `prelude` is a hand-picked list: it re-exports only the items you reach for
 | Graph model | `GraphBuilder` (collects nodes and layers for a non-chain topology), `Graph` (what `build` gives back), `NodeId` (a handle to 1 node of the graph) |
 | Core layers | `Dense`, `Embedding`, `Flatten`, `Identity`, `Reshape`, `Reverse`, `Permute`, `RepeatVector`, `Rescaling`, `Activation` |
 | Merge layers | `Add`, `Average`, `Concatenate`, `Maximum`, `Minimum`, `Multiply`, `Subtract` |
-| Activation layers | `Linear`, `ReLU`, `LeakyReLU`, `ELU`, `SELU`, `Sigmoid`, `HardSigmoid`, `Tanh`, `Softplus`, `Softsign`, `Exponential`, `Softmax`, `PReLU` |
+| Activation layers | `Linear`, `ReLU`, `LeakyReLU`, `ELU`, `SELU`, `Sigmoid`, `HardSigmoid`, `Tanh`, `Softplus`, `Softsign`, `Exponential`, `GELU`, `SiLU`, `Mish`, `Softmax`, `PReLU` |
 | Convolution | `Conv1D`, `Conv2D`, `Conv3D`, `Conv1DTranspose`, `Conv2DTranspose`, `Conv3DTranspose`, `DepthwiseConv1D`, `DepthwiseConv2D`, `SeparableConv1D`, `SeparableConv2D`, `PaddingType`, `ConvPadding` |
 | Border | `ZeroPadding1D/2D/3D`, `Cropping1D/2D/3D`, `Border1D/2D/3D` |
 | Pooling | `MaxPooling1D/2D/3D`, `AveragePooling1D/2D/3D`, `GlobalMaxPooling1D/2D/3D`, `GlobalAveragePooling1D/2D/3D` |