From 30f47127aabe47d5a7d070cbecfae438f4174ddc Mon Sep 17 00:00:00 2001 From: Michael Kraus Date: Sun, 23 Aug 2026 13:37:09 +0900 Subject: [PATCH 1/2] Remove the recurrent and LSTM architectures `RecurrentNeuralNetwork` and `LSTMNeuralNetwork` go, with `src/architectures/recurrent_neural_network.jl`, `src/architectures/LSTM_neural_network.jl` and the driver scripts under `scripts/Script_using_fully_GML/{RNN,LSTM}/`. Both were already unusable, and had been for some time. `Chain(::RecurrentNeuralNetwork)` returned an `AbstractNeuralNetworks.GridCell` rather than a `Chain`, and the cells it held still defined the pre-0.6 `initialparameters(cell, backend, T; init, rng)` signature, so constructing a `NeuralNetwork` from either architecture errored out. Nothing in the test suite referenced them, so nothing noticed. AbstractNeuralNetworks 0.7 deleted `src/cells/` outright -- `Recurrent`, `LSTM`, `GRU`, `IdentityCell` and `GridCell` (JuliaGNI/AbstractNeuralNetworks.jl#34) -- so the imports at the top of `GeometricMachineLearning.jl` have to go regardless of what happens to the architectures that used them. Removing them here means the 0.7 bump does not have to carry a behavioural change on top of a dependency change. The transformer-derived architectures are what GML maintains for time series; the footnote in `docs/src/architectures/neural_network_integrators.md` had flagged the LSTM implementation as likely to be deprecated since it was written, and now says it is gone. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 14 ++++ .../neural_network_integrators.md | 2 +- .../LSTM/generate_data.jl | 68 ---------------- .../Script_using_fully_GML/LSTM/training.jl | 69 ---------------- .../Script_using_fully_GML/LSTM/validation.jl | 75 ------------------ .../RNN/generate_data.jl | 68 ---------------- .../Script_using_fully_GML/RNN/training.jl | 75 ------------------ .../Script_using_fully_GML/RNN/validation.jl | 78 ------------------- src/GeometricMachineLearning.jl | 10 +-- src/architectures/LSTM_neural_network.jl | 41 ---------- src/architectures/recurrent_neural_network.jl | 42 ---------- 11 files changed, 18 insertions(+), 524 deletions(-) delete mode 100644 scripts/Script_using_fully_GML/LSTM/generate_data.jl delete mode 100644 scripts/Script_using_fully_GML/LSTM/training.jl delete mode 100644 scripts/Script_using_fully_GML/LSTM/validation.jl delete mode 100644 scripts/Script_using_fully_GML/RNN/generate_data.jl delete mode 100644 scripts/Script_using_fully_GML/RNN/training.jl delete mode 100644 scripts/Script_using_fully_GML/RNN/validation.jl delete mode 100644 src/architectures/LSTM_neural_network.jl delete mode 100644 src/architectures/recurrent_neural_network.jl diff --git a/CHANGELOG.md b/CHANGELOG.md index 52ec8e0cf..5bf6c0b08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,20 @@ breaking release). ## [Unreleased] +### Removed (breaking) + +- **`RecurrentNeuralNetwork` and `LSTMNeuralNetwork`**, along with + `src/architectures/recurrent_neural_network.jl`, `src/architectures/LSTM_neural_network.jl` and the + driver scripts under `scripts/Script_using_fully_GML/{RNN,LSTM}/`. + + Both were already unusable. `Chain(::RecurrentNeuralNetwork)` returned an + `AbstractNeuralNetworks.GridCell` rather than a `Chain`, and the cells it held still defined the + pre-0.6 `initialparameters(cell, backend, T; init, rng)` signature, so building a `NeuralNetwork` + from one errored out. Nothing in the test suite touched them. AbstractNeuralNetworks 0.7 removes + `src/cells/` — `Recurrent`, `LSTM`, `GRU`, `IdentityCell` and `GridCell` — so the imports have to go + regardless. The transformer-derived architectures are what GML maintains for time series; the docs + had flagged the LSTM implementation as likely to be deprecated since it was written. + ### Fixed - **Zygote 0.7 silently zeroed every gradient that flows through `assign_q_and_p`.** Its `rrule` diff --git a/docs/src/architectures/neural_network_integrators.md b/docs/src/architectures/neural_network_integrators.md index 44d690c7c..0aa88b5c5 100644 --- a/docs/src/architectures/neural_network_integrators.md +++ b/docs/src/architectures/neural_network_integrators.md @@ -39,7 +39,7 @@ where `sl` is short for *sequence length* and `pw` is short for *prediction wind In `GeometricMachineLearning` all multi-step methods, as is the case with one-step methods, are explicit. There are essentially two ways to construct multi-step methods with neural networks: the older one is using recurrent neural networks such as long short-term memory cells (LSTMs) [hochreiter1997long](@cite) and the newer one is using transformer neural networks [vaswani2017attention](@cite). Both of these approaches have been successfully employed to learn multi-step methods (see [fresca2021comprehensive, lee2020model](@cite) for the former and [hemmasian2023reduced, solera2023beta, brantner2025volume](@cite) for the latter), but because the transformer architecture exhibits superior performance on modern hardware and can be imbued with geometric properties we almost always use a transformer-derived architecture when dealing with time series[^2]. -[^2]: `GeometricMachineLearning` also has an LSTM implementation, but this may be deprecated in the future. +[^2]: `GeometricMachineLearning` used to ship an LSTM implementation. It was removed in favour of the transformer-derived architectures, which are the ones maintained for time series here. Explicit multi-step methods derived from the transformer are always subtypes of the type [`TransformerIntegrator`](@ref) in `GeometricMachineLearning`. In `GeometricMachineLearning` the [standard transformer](@ref "Standard Transformer"), the [volume-preserving transformer](@ref "Volume-Preserving Transformer") and the [linear symplectic transformer](@ref "Linear Symplectic Transformer") are implemented. diff --git a/scripts/Script_using_fully_GML/LSTM/generate_data.jl b/scripts/Script_using_fully_GML/LSTM/generate_data.jl deleted file mode 100644 index 9a2b7acc8..000000000 --- a/scripts/Script_using_fully_GML/LSTM/generate_data.jl +++ /dev/null @@ -1,68 +0,0 @@ -using GeometricIntegrators, KernelAbstractions, JLD2 - -T = Float64 - -initial_conditions_collection = ( (q=[T(1.),T(0.)], p=[T(2.),T(0.)]), ) - -m1 = T(2.) -m2 = T(1.) -k1 = T(1.5) -k2 = T(0.3) -k = T(0.5) #k = T.(0.0:0.1:4) - -params_collection = Tuple([(m1=m1,m2=m2,k1=k1,k2=k2,k=k_val) for k_val in k]) - -const t_integration = 100 -const time_step = T(.4) - -function q̇(v, t, q, p, params) - v[1] = p[1]/params.m1 - v[2] = p[2]/params.m2 -end - -function sigmoid(x::T) where {T<:Real} - T(1)/(T(1) + exp(-x)) -end - -function ṗ(f, t, q, p, params) - f[1] = -params.k1 * q[1] - params.k * (q[1] - q[2]) * sigmoid(q[1]) - params.k /2 * (q[1] - q[2])^2 * sigmoid(q[1])^2 * exp(-q[1]) - f[2] = -params.k2 * q[2] + params.k * (q[1] - q[2]) * sigmoid(q[1]) -end - -@kernel function create_tensor_kernel_q!(data_tensor, sols) - i,j,k = @index(Global, NTuple) - data_tensor[i,j,k] = sols[j].q[k-1][i] -end -@kernel function create_tensor_kernel_p!(data_tensor, sols) - i,j,k = @index(Global, NTuple) - data_tensor[i+2,j,k] = sols[j].p[k-1][i] -end -function assign_tensor(data_tensor, sols) - assign_q! = create_tensor_kernel_q!(CPU()) - assign_p! = create_tensor_kernel_p!(CPU()) - dims = (2, size(data_tensor,2), size(data_tensor,3)) - assign_q!(data_tensor, sols, ndrange=dims) - assign_p!(data_tensor, sols, ndrange=dims) -end - -function generate_data(params_collection=params_collection, initial_conditions=initial_conditions, t_integration=t_integration, Float32_not_working_yet=true) - Float32_not_working_yet ? (@warn "Float32 not used in integration!!") : nothing - sols = [] - for params in params_collection - for initial_conditions in initial_conditions_collection - pode = PODEProblem(q̇, ṗ, (0.0, t_integration), time_step, initial_conditions; parameters = params) - sol = integrate(pode,ImplicitMidpoint()) - push!(sols, sol) - end - end - - time_steps = length(sols[1].q) - data_tensor = zeros(4, length(sols), time_steps) - assign_tensor(data_tensor, sols) - Float32_not_working_yet ? Float32.(data_tensor) : data_tensor -end - -data_tensor = generate_data() - -jldsave("data", tensor=data_tensor, params=params_collection, initial_conditions=initial_conditions_collection, - time_interval=(0, t_integration), time_step=time_step) \ No newline at end of file diff --git a/scripts/Script_using_fully_GML/LSTM/training.jl b/scripts/Script_using_fully_GML/LSTM/training.jl deleted file mode 100644 index 460dbd9fb..000000000 --- a/scripts/Script_using_fully_GML/LSTM/training.jl +++ /dev/null @@ -1,69 +0,0 @@ -using GeometricMachineLearning -using JLD2 -using KernelAbstractions -using LinearAlgebra - -# Generation of the data -backend = CPU() -T = Float32 - -file = jldopen("data", "r") -data_raw = file["tensor"] -sys_dim, n_params, n_time_steps = size(data_raw) -generated_data = KernelAbstractions.allocate(backend, T, size(data_raw)) -copyto!(generated_data, data_raw) - -get_Data = Dict( - :shape => TrajectoryData, - :nb_trajectory => Data -> size(Data)[2], - :length_trajectory => (Data,i) -> size(Data)[3], - :Δt => Data -> 0.4, - :q => (Data,i,n) -> [Data[1,i,n], Data[2,i,n]], - :p => (Data,i,n) -> [Data[3,i,n], Data[4,i,n]] -) -data = TrainingData(generated_data, get_Data) - -# Creation of the architecture - -dimin = 4 -ssize = (1, 6) - -arch = LSTMNeuralNetwork(dimin, ssize) - -# Creation of the NeuralNetwork - -rnn = NeuralNetwork(arch, backend, T) - -using Test -@test_nowarn rnn([[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]]) - -# Creation of the Optimizer - -opt = AdamOptimizer() - -# Creation of the Loss Function - -function loss(rnn, data, batch = ((1,1),(1,2)), params = rnn.params) - s = 0 - for (i,j) in batch - x = [[get_data(data, :q, i, y)...,get_data(data, :p, i, y)...] for y in j:(j+5)] - y = [get_data(data, :q, i, j+6)...,get_data(data, :p, i, j+6)...] - s += norm(y - rnn(x, params)) - end - s -end - -# Paramters of the training - -batch_size = (1,7,7) -nruns = 500 - -training_parameters = TrainingParameters(nruns, loss, opt; batch_size = batch_size) - -# Training of the NeuralNetwork - -train!(rnn, data, training_parameters; showprogress = true, timer = true) - - - -jldsave("nn_model", model=rnn, params = rnn.params, seq_length = 6) diff --git a/scripts/Script_using_fully_GML/LSTM/validation.jl b/scripts/Script_using_fully_GML/LSTM/validation.jl deleted file mode 100644 index c89e87b9e..000000000 --- a/scripts/Script_using_fully_GML/LSTM/validation.jl +++ /dev/null @@ -1,75 +0,0 @@ -using CairoMakie, JLD2, GeometricIntegrators, AbstractNeuralNetworks, GeometricMachineLearning - -# this file stores parameters relevant for the NN -file_nn = jldopen("nn_model", "r") -model = file_nn["model"] -params = file_nn["params"] -seq_length = file_nn["seq_length"] - - -# this file stores parameters relevant for dataset/vector field (theoretically only needed for validation!) -file_data = jldopen("data", "r") -time_step = file_data["time_step"] - -# Computing the reference solution - -function sigmoid(x::T) where {T<:Real} - T(1)/(T(1) + exp(-x)) -end - -function q̇(v, t, q, p, params) - v[1] = p[1]/params.m1 - v[2] = p[2]/params.m2 -end - -function ṗ(f, t, q, p, params) - f[1] = -params.k1 * q[1] - params.k * (q[1] - q[2]) * sigmoid(q[1]) - params.k /2 * (q[1] - q[2])^2 * sigmoid(q[1])^2 * exp(-q[1]) - f[2] = -params.k2 * q[2] + params.k * (q[1] - q[2]) * sigmoid(q[1]) -end - -T = Float64 - -m1 = T(2.) -m2 = T(1.) -k1 = T(1.5) -k2 = T(0.3) -k = T(0.5) -params = (m1=m1, m2=m2, k1=k1, k2=k2, k=k) - -initial_conditions_val = (q=[T(1.),T(0.)], p=[T(2.),T(0.)]) - -t_integration = 50 - -pode = PODEProblem(q̇, ṗ, (T(0.0), T(t_integration)), time_step, initial_conditions_val; parameters = params) -sol = integrate(pode, ImplicitMidpoint()) - -data_matrix = zeros(T, 4, Int(t_integration/time_step) + 1) -for i in 1:seq_length data_matrix[1:2, i] = sol.q[i-1] end -for i in 1:seq_length data_matrix[3:4, i] = sol.p[i-1] end - -# Computing the prediction - -total_steps = Int(t_integration/time_step) - seq_length + 1 - -for i in 1:total_steps - x = [[data_matrix[:, i + j - 1]] for j in 1:6] - data_matrix[:,i + seq_length] = model(x) -end - -q1 = zeros(size(data_matrix,2)) -t = zeros(size(data_matrix,2)) -for i in 1:length(q1) - q1[i] = sol.q[i-1][1] - t[i] = sol.t[i-1] -end - -# Plot of the validation - -plt = Figure(size = (1000, 600)) -ax = Axis(plt[1, 1]) -lines!(ax, t, q1; label = "Numeric Integration") -lines!(ax, t, data_matrix[1,:]; label = "Neural Network") -vlines!(ax, [seq_length*time_step]; color = :red, label = "Start of Prediction") -axislegend(ax) - -#png(plot1, "seq_length"*string(seq_length)*"_prediction_window"*string(prediction_window)) diff --git a/scripts/Script_using_fully_GML/RNN/generate_data.jl b/scripts/Script_using_fully_GML/RNN/generate_data.jl deleted file mode 100644 index 1d9d40c32..000000000 --- a/scripts/Script_using_fully_GML/RNN/generate_data.jl +++ /dev/null @@ -1,68 +0,0 @@ -using GeometricIntegrators, KernelAbstractions, JLD2 - -T = Float64 - -initial_conditions_collection = ( (q=[T(1.),T(0.)], p=[T(2.),T(0.)]), ) - -m1 = T(2.) -m2 = T(1.) -k1 = T(1.5) -k2 = T(0.3) -k = T.(0.0:0.01:4) - -params_collection = Tuple([(m1=m1,m2=m2,k1=k1,k2=k2,k=k_val) for k_val in k]) - -const t_integration = 100 -const time_step = T(.4) - -function q̇(v, t, q, p, params) - v[1] = p[1]/params.m1 - v[2] = p[2]/params.m2 -end - -function sigmoid(x::T) where {T<:Real} - T(1)/(T(1) + exp(-x)) -end - -function ṗ(f, t, q, p, params) - f[1] = -params.k1 * q[1] - params.k * (q[1] - q[2]) * sigmoid(q[1]) - params.k /2 * (q[1] - q[2])^2 * sigmoid(q[1])^2 * exp(-q[1]) - f[2] = -params.k2 * q[2] + params.k * (q[1] - q[2]) * sigmoid(q[1]) -end - -@kernel function create_tensor_kernel_q!(data_tensor, sols) - i,j,k = @index(Global, NTuple) - data_tensor[i,j,k] = sols[j].q[k-1][i] -end -@kernel function create_tensor_kernel_p!(data_tensor, sols) - i,j,k = @index(Global, NTuple) - data_tensor[i+2,j,k] = sols[j].p[k-1][i] -end -function assign_tensor(data_tensor, sols) - assign_q! = create_tensor_kernel_q!(CPU()) - assign_p! = create_tensor_kernel_p!(CPU()) - dims = (2, size(data_tensor,2), size(data_tensor,3)) - assign_q!(data_tensor, sols, ndrange=dims) - assign_p!(data_tensor, sols, ndrange=dims) -end - -function generate_data(params_collection=params_collection, initial_conditions=initial_conditions, t_integration=t_integration, Float32_not_working_yet=true) - Float32_not_working_yet ? (@warn "Float32 not used in integration!!") : nothing - sols = [] - for params in params_collection - for initial_conditions in initial_conditions_collection - pode = PODEProblem(q̇, ṗ, (0.0, t_integration), time_step, initial_conditions; parameters = params) - sol = integrate(pode,ImplicitMidpoint()) - push!(sols, sol) - end - end - - time_steps = length(sols[1].q) - data_tensor = zeros(4, length(sols), time_steps) - assign_tensor(data_tensor, sols) - Float32_not_working_yet ? Float32.(data_tensor) : data_tensor -end - -data_tensor = generate_data() - -jldsave("data", tensor=data_tensor, params=params_collection, initial_conditions=initial_conditions_collection, - time_interval=(0, t_integration), time_step=time_step) \ No newline at end of file diff --git a/scripts/Script_using_fully_GML/RNN/training.jl b/scripts/Script_using_fully_GML/RNN/training.jl deleted file mode 100644 index a4f2a85ed..000000000 --- a/scripts/Script_using_fully_GML/RNN/training.jl +++ /dev/null @@ -1,75 +0,0 @@ -using GeometricMachineLearning -using JLD2 -using KernelAbstractions -using LinearAlgebra - -# Generation of the data -backend = CPU() -T = Float32 - -file = jldopen("data", "r") -data_raw = file["tensor"] -sys_dim, n_params, n_time_steps = size(data_raw) -generated_data = KernelAbstractions.allocate(backend, T, size(data_raw)) -copyto!(generated_data, data_raw) - -get_Data = Dict( - :shape => TrajectoryData, - :nb_trajectory => Data -> size(Data)[2], - :length_trajectory => (Data,i) -> size(Data)[3], - :Δt => Data -> 0.4, - :q => (Data,i,n) -> [Data[1,i,n], Data[2,i,n]], - :p => (Data,i,n) -> [Data[3,i,n], Data[4,i,n]] -) -data = TrainingData(generated_data, get_Data) - -# Creation of the architecture - -len = 1 -dimin = 4 -dimout = 4 -ssize = (1, len) - -arch = RecurrentNeuralNetwork(dimin, dimout, ssize) - -# Creation of the NeuralNetwork - -rnn = NeuralNetwork(arch, backend, T) - - -# Creation of the Optimizer - -opt = AdamOptimizer() - -# Creation of the Loss Function - -function loss(rnn, data, batch = vcat([[(i,j) for j in 1:get_length_trajectory(data,i)-len-1] for i in 1:get_nb_trajectory(data)]...), params = rnn.params) - s = 0 - for (i,j) in batch - x = [[get_data(data, :q, i, y)...,get_data(data, :p, i, y)...] for y in j:(j+len-1)] - y = [get_data(data, :q, i, j+len)...,get_data(data, :p, i, j+len)...] - s += norm(y - rnn(x, params)) - end - s -end - -pss = rnn.params -@show loss(rnn, data) - -# Paramters of the training - -batch_size = (30,10*(len+1), len+1) -nruns = 1000 - -training_parameters = TrainingParameters(nruns, loss, opt; batch_size = batch_size) - - -# Training of the NeuralNetwork - -total_loss = train!(rnn, data, training_parameters; showprogress = true, timer = true) - -@show loss(rnn, data) - - - -jldsave("nn_model2", model=rnn, params = rnn.params, seq_length = ssize[2]) diff --git a/scripts/Script_using_fully_GML/RNN/validation.jl b/scripts/Script_using_fully_GML/RNN/validation.jl deleted file mode 100644 index 08766a068..000000000 --- a/scripts/Script_using_fully_GML/RNN/validation.jl +++ /dev/null @@ -1,78 +0,0 @@ -using CairoMakie, JLD2, GeometricIntegrators, AbstractNeuralNetworks, GeometricMachineLearning - -# this file stores parameters relevant for the NN -file_nn = jldopen("nn_model2", "r") -model = file_nn["model"] -params = file_nn["params"] -seq_length = file_nn["seq_length"] - - -# this file stores parameters relevant for dataset/vector field (theoretically only needed for validation!) -file_data = jldopen("data", "r") -time_step = file_data["time_step"] - -# Computing the reference solution - -function sigmoid(x::T) where {T<:Real} - T(1)/(T(1) + exp(-x)) -end - -function q̇(v, t, q, p, params) - v[1] = p[1]/params.m1 - v[2] = p[2]/params.m2 -end - -function ṗ(f, t, q, p, params) - f[1] = -params.k1 * q[1] - params.k * (q[1] - q[2]) * sigmoid(q[1]) - params.k /2 * (q[1] - q[2])^2 * sigmoid(q[1])^2 * exp(-q[1]) - f[2] = -params.k2 * q[2] + params.k * (q[1] - q[2]) * sigmoid(q[1]) -end - -T = Float64 - -m1 = T(2.) -m2 = T(1.) -k1 = T(1.5) -k2 = T(0.3) -k = T(0.5) -params = (m1=m1, m2=m2, k1=k1, k2=k2, k=k) - -initial_conditions_val = (q=[T(4.),T(2.)], p=[T(0.5),T(0.)]) - -t_integration = 10 - -pode = PODEProblem(q̇, ṗ, (T(0.0), T(t_integration)), time_step, initial_conditions_val; parameters = params) -sol = integrate(pode, ImplicitMidpoint()) - -data_matrix = zeros(T, 4, Int(t_integration/time_step) + 1) -for i in 1:seq_length data_matrix[1:2, i] = sol.q[i-1] end -for i in 1:seq_length data_matrix[3:4, i] = sol.p[i-1] end - -# Computing the prediction - -total_steps = Int(t_integration/time_step) - seq_length + 1 - -for i in 1:total_steps - x = [data_matrix[:, i + j - 1] for j in 1:seq_length] - data_matrix[:,i + seq_length] = model(x) -end - - - - -q1 = zeros(size(data_matrix,2)) -t = zeros(size(data_matrix,2)) -for i in 1:length(q1) - q1[i] = sol.q[i-1][1] - t[i] = sol.t[i-1] -end - -# Plot of the validation - -plt = Figure(size = (1000, 600)) -ax = Axis(plt[1, 1]) -lines!(ax, t, q1; label = "Numeric Integration") -lines!(ax, t, data_matrix[1,:]; label = "Neural Network") -vlines!(ax, [seq_length*time_step-time_step]; color = :red, label = "Start of Prediction") -axislegend(ax) - -#png(plot1, "seq_length"*string(seq_length)*"_prediction_window"*string(prediction_window)) diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl index 50f9011d1..948751f3f 100644 --- a/src/GeometricMachineLearning.jl +++ b/src/GeometricMachineLearning.jl @@ -61,13 +61,13 @@ import GeometricOptimizers: solve! # them as `GeometricOptimizers.AdamCache` where it needs to name one, and no longer re-exports them. import AbstractNeuralNetworks: Architecture, Model, AbstractExplicitLayer, - AbstractExplicitCell, AbstractNeuralNetwork, NeuralNetwork, + AbstractNeuralNetwork, NeuralNetwork, UnknownArchitecture, FeedForwardLoss -import AbstractNeuralNetworks: Chain, GridCell +import AbstractNeuralNetworks: Chain # `input_dimension`/`output_dimension` are AbstractNeuralNetworks' since v0.6.4; the `Chain` # methods GML uses are added to them by SymbolicNeuralNetworks. import AbstractNeuralNetworks: input_dimension, output_dimension -import AbstractNeuralNetworks: Dense, Linear, Recurrent +import AbstractNeuralNetworks: Dense, Linear import AbstractNeuralNetworks: IdentityActivation, ZeroVector # `update!` used to be imported here too, from `AbstractNeuralNetworks`, and re-exported. GML never # added a method to it, so all the export did was shadow `GeometricOptimizers.update!` — which is @@ -304,8 +304,6 @@ include("architectures/fixed_width_network.jl") include("architectures/hamiltonian_neural_network.jl") include("architectures/lagrangian_neural_network.jl") include("architectures/variable_width_network.jl") -include("architectures/recurrent_neural_network.jl") -include("architectures/LSTM_neural_network.jl") include("architectures/transformer_neural_network.jl") include("architectures/volume_preserving_feedforward.jl") include("architectures/volume_preserving_transformer.jl") @@ -313,8 +311,6 @@ include("architectures/volume_preserving_transformer.jl") export HamiltonianArchitecture export LagrangianNeuralNetwork export SympNet, LASympNet, GSympNet -export RecurrentNeuralNetwork -export LSTMNeuralNetwork export ClassificationTransformer, ClassificationLayer export VolumePreservingFeedForward export SymplecticAutoencoder, PSDArch diff --git a/src/architectures/LSTM_neural_network.jl b/src/architectures/LSTM_neural_network.jl deleted file mode 100644 index 61b6f3ed2..000000000 --- a/src/architectures/LSTM_neural_network.jl +++ /dev/null @@ -1,41 +0,0 @@ - -struct LSTMNeuralNetwork{TS, TAO, TAM} <: Architecture - dimin::Int - dimst::Int - size::TS - σ₀::TAO - σ₋₁::TAM - function LSTMNeuralNetwork(dimin::Int, size::Tuple{<:Int, <:Int} = (1,1); dimst::Int = dimin, act0 = AbstractNeuralNetworks.SigmoidActivation(), act1 = tanh) - @assert size[1] > 0 && size[2] > 0 - new{typeof(size), typeof(act0), typeof(act1)}(dimin, dimst, size, act0, act1) - end -end - -@inline AbstractNeuralNetworks.dim(arch::LSTMNeuralNetwork) = arch.dimin - -function Chain(rnn::LSTMNeuralNetwork) - N, M = rnn.size - if N == 1 - cell_upper = reshape([LSTM(rnn.dimin, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 1:M-1], (1,M-1)) - cell_output = [LSTM(rnn.dimin, rnn.dimst, rnn.σ₀, rnn.σ₋₁)] - return GridCell([hcat(cell_upper, cell_output);]) - elseif M == 1 - cell_upper = reshape([LSTM(rnn.dimin, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 1:M], (1,M)) - cell_left = [LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 2:N-1] - cell_bot = reshape([LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 1:M-1], (1,M-1)) - cell_output = [LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁)] - matrix = vcat(cell_upper, cell_left) - matrix = vcat(matrix, hcat(cell_bot, cell_output)) - return GridCell(matrix) - else - cell_upper = reshape([LSTM(rnn.dimin, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 1:M], (1,M)) - cell_left = [LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 2:N-1] - cell_bot = reshape([LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 1:M-1], (1,M-1)) - cell_inner = [LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁) for _ in 2:N-1, _ in 2:M] - cell_output = [LSTM(rnn.dimst, rnn.dimst, rnn.σ₀, rnn.σ₋₁)] - matrix = hcat(cell_left, cell_inner) - matrix = vcat(cell_upper, matrix) - matrix = vcat(matrix, hcat(cell_bot, cell_output)) - return GridCell(matrix) - end -end diff --git a/src/architectures/recurrent_neural_network.jl b/src/architectures/recurrent_neural_network.jl deleted file mode 100644 index 087c1c413..000000000 --- a/src/architectures/recurrent_neural_network.jl +++ /dev/null @@ -1,42 +0,0 @@ - -struct RecurrentNeuralNetwork{TS, TAO, TAM} <: Architecture - dimin::Int - dimout::Int - dimst::Int - size::TS - act_output::TAO - act_st::TAM - function RecurrentNeuralNetwork(dimin::Int, dimout::Int, size::Tuple{<:Int, <:Int} = (1,1); dimst::Int = dimin, act_out = tanh, act_st =act_out) - @assert size[1] > 0 && size[2] > 0 - new{typeof(size), typeof(act_out), typeof(act_st)}(dimin, dimout, dimst, size, act_out, act_st) - end -end - -@inline AbstractNeuralNetworks.dim(arch::RecurrentNeuralNetwork) = arch.dimin - -function Chain(rnn::RecurrentNeuralNetwork) - N, M = rnn.size - if N == 1 - cell_upper = reshape([Recurrent(rnn.dimin, rnn.dimst, 0, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 1:M-1], (1,M-1)) - cell_output = [Recurrent(rnn.dimin, rnn.dimst, rnn.dimout, rnn.dimst, IdentityActivation(), rnn.act_st)] - return GridCell([hcat(cell_upper, cell_output);]) - elseif M == 1 - cell_upper = reshape([Recurrent(rnn.dimin, rnn.dimst,rnn.dimst, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 1:M], (1,M)) - cell_left = [Recurrent(rnn.dimst, rnn.dimst, rnn.dimst, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 2:N-1] - cell_bot = reshape([Recurrent(rnn.dimst, rnn.dimst,0, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 1:M-1], (1,M-1)) - cell_output = [Recurrent(rnn.dimin, rnn.dimst, rnn.dimout, rnn.dimst, IdentityActivation(), rnn.act_st)] - matrix = vcat(cell_upper, cell_left) - matrix = vcat(matrix, hcat(cell_bot, cell_output)) - return GridCell(matrix) - else - cell_upper = reshape([Recurrent(rnn.dimin, rnn.dimst,rnn.dimst, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 1:M], (1,M)) - cell_left = [Recurrent(rnn.dimst, rnn.dimst, rnn.dimst, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 2:N-1] - cell_bot = reshape([Recurrent(rnn.dimst, rnn.dimst, 0, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 1:M-1], (1,M-1)) - cell_inner = [Recurrent(rnn.dimst, rnn.dimst, rnn.dimst, rnn.dimst, rnn.act_output, rnn.act_st) for _ in 2:N-1, _ in 2:M] - cell_output = [Recurrent(rnn.dimin, rnn.dimst, rnn.dimout, rnn.dimst, IdentityActivation(), rnn.act_st)] - matrix = hcat(cell_left, cell_inner) - matrix = vcat(cell_upper, matrix) - matrix = vcat(matrix, hcat(cell_bot, cell_output)) - return GridCell(matrix) - end -end From df89a6e61e0d8f9468bf2ad9adfdd525df5ec72e Mon Sep 17 00:00:00 2001 From: Michael Kraus Date: Sun, 23 Aug 2026 15:30:23 +0900 Subject: [PATCH 2/2] Drop the import the removal left dead, and tighten the footnote `import AbstractNeuralNetworks: IdentityActivation, ZeroVector` had `Chain(::RecurrentNeuralNetwork)` as its only reader for `IdentityActivation`, and `ZeroVector`'s last use in `src/` moved to `legacy/` in `ec8e8fa5`. Neither is exported by GML, and `IdentityActivation` is an `AbstractNeuralNetworks` export that the blanket `using` at the top of the module already provides, so `GeometricMachineLearning.IdentityActivation` keeps resolving without it. The footnote in `docs/src/architectures/neural_network_integrators.md` had grown a second half that repeated the sentence it hangs off -- the body already says transformer-derived architectures are what gets used for time series -- so it now records only the fact it is there for. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 7 +++++++ docs/src/architectures/neural_network_integrators.md | 2 +- src/GeometricMachineLearning.jl | 1 - 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5bf6c0b08..ddad2dbcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,13 @@ breaking release). regardless. The transformer-derived architectures are what GML maintains for time series; the docs had flagged the LSTM implementation as likely to be deprecated since it was written. + `import AbstractNeuralNetworks: IdentityActivation, ZeroVector` went with them. + `Chain(::RecurrentNeuralNetwork)` was the only thing in `src/` that named `IdentityActivation`, + and `ZeroVector`'s last use moved to `legacy/` in `ec8e8fa5`. Neither was ever exported by GML, + and `IdentityActivation` is an `AbstractNeuralNetworks` export that the blanket `using` at the top + of the module already provides, so `GeometricMachineLearning.IdentityActivation` still resolves — + the line was redundant, not load-bearing. + ### Fixed - **Zygote 0.7 silently zeroed every gradient that flows through `assign_q_and_p`.** Its `rrule` diff --git a/docs/src/architectures/neural_network_integrators.md b/docs/src/architectures/neural_network_integrators.md index 0aa88b5c5..b6e39ba33 100644 --- a/docs/src/architectures/neural_network_integrators.md +++ b/docs/src/architectures/neural_network_integrators.md @@ -39,7 +39,7 @@ where `sl` is short for *sequence length* and `pw` is short for *prediction wind In `GeometricMachineLearning` all multi-step methods, as is the case with one-step methods, are explicit. There are essentially two ways to construct multi-step methods with neural networks: the older one is using recurrent neural networks such as long short-term memory cells (LSTMs) [hochreiter1997long](@cite) and the newer one is using transformer neural networks [vaswani2017attention](@cite). Both of these approaches have been successfully employed to learn multi-step methods (see [fresca2021comprehensive, lee2020model](@cite) for the former and [hemmasian2023reduced, solera2023beta, brantner2025volume](@cite) for the latter), but because the transformer architecture exhibits superior performance on modern hardware and can be imbued with geometric properties we almost always use a transformer-derived architecture when dealing with time series[^2]. -[^2]: `GeometricMachineLearning` used to ship an LSTM implementation. It was removed in favour of the transformer-derived architectures, which are the ones maintained for time series here. +[^2]: `GeometricMachineLearning` used to ship an LSTM implementation. It has been removed. Explicit multi-step methods derived from the transformer are always subtypes of the type [`TransformerIntegrator`](@ref) in `GeometricMachineLearning`. In `GeometricMachineLearning` the [standard transformer](@ref "Standard Transformer"), the [volume-preserving transformer](@ref "Volume-Preserving Transformer") and the [linear symplectic transformer](@ref "Linear Symplectic Transformer") are implemented. diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl index 948751f3f..ae185a293 100644 --- a/src/GeometricMachineLearning.jl +++ b/src/GeometricMachineLearning.jl @@ -68,7 +68,6 @@ import AbstractNeuralNetworks: Chain # methods GML uses are added to them by SymbolicNeuralNetworks. import AbstractNeuralNetworks: input_dimension, output_dimension import AbstractNeuralNetworks: Dense, Linear -import AbstractNeuralNetworks: IdentityActivation, ZeroVector # `update!` used to be imported here too, from `AbstractNeuralNetworks`, and re-exported. GML never # added a method to it, so all the export did was shadow `GeometricOptimizers.update!` — which is # `GeometricBase.update!`, a different generic function, and the one that actually has methods for