Skip to content

Networks

Jump to bottom
H.P. Gansevoort edited this page Oct 14, 2026 · 13 revisions

Networks

The learning half: the layers, the losses, the optimizers and the loop that trains them; a network described in Keras's words, written as code or declared in the chain of the pipeline that prepares its rows, which is the same network whichever way it is written; and a network trained behind a pipeline, measured by its report, drawn, and kept with that pipeline as one file. The layers and the loop live in DeepSharp, beside the tensors. Where a network meets a pipeline is DeepSharp.Learners.Networks, and the charts are DeepSharp.Charts — each a package of its own, so a project carries only what it uses. The arithmetic runs on the light engine unless you hand it another, such as libtorch — TorchSharp backend — and a network PyTorch, Keras or an ONNX exporter saved is read into the same network — Importing a model.

dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines
dotnet add package DeepSharp.Learners.Networks
dotnet add package DeepSharp.Charts

One network, start to finish

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    // ---- nothing above this line learns from the rows ----
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes every feature between -1 and 1: the default
    .Report(report => report
        .Measure(Metric.Accuracy, Metric.Precision, Metric.Recall, Metric.ConfusionMatrix)
        .On(Part.Train, Part.Validation, Part.Test)
        .As(Shown.Numbers, Shown.Drawn))
    .WithTensorflow(network => network                // and the network those rows train
        .Dense(16).Relu().Dense(1)
        .Adam(0.01)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 100)
        .StoppingAfter(patience: 10))
    .Build();

var trained = pipeline.Train();

var test = trained.Measures!.Parts[^1];               // the test rows' measures, each beside the average's
File.WriteAllText("titanic.network.json", trained.ToJson());
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());

The network learns from the training rows and is judged by the validation rows once an epoch; the test rows never reach the loop, and the report measures them once it has learned. Run on the published file, it stopped after 46 epochs, kept the network of its 36th, and got 0.815 of the test passengers right, where predicting the training rows' most common answer gets 0.615. The networks sample in the repository trains it, a price five days on and a day's bikes hour by hour, and prints what each did.

Three ways to describe one network

Keras's words, code, or the chain of the pipeline that prepares the rows. All three lower to the same network, number for number, and the third has a section of its own, below.

In Keras's words. new Sequential() and then the layers — .Dense(units), .Relu(), .Tanh(), .Sigmoid(), .Dropout(rate), .BatchNorm(), .LayerNorm(), .Reshape(shape), .Flatten(), and for a series, an image or a volume .Conv1D(filters, window), .Conv2D(filters, window, groups), .Conv3D(filters, window), .DepthwiseConv2D(multiplier, window), .Conv2DTranspose(filters, window), .Pad2D(border), .MaxPool2D(2), .AvgPool1D(window), .AdaptiveAvgPool2D(rows, columns), .GlobalAvgPool2D() and .SpatialDropout2D(rate) — set out below, under A series, an image, a volume — and nothing about widths: each is worked out from the shape of an example as it reaches the layer, a row of numbers for a dense layer, an image of rows, columns and channels for a convolution. A layer that starts otherwise or has no bias takes one more argument, .Dense(units, new Beginning { WithoutBias = true }), and one that carries a penalty, a bound or a hold is followed by the word that says it, .Dense(units).Ruled(new SlotRules { L2 = 0.0001 }) or .Dense(units).Held() — set out below, under How a layer starts and Penalising, bounding and holding still. A word an example cannot reach is refused, naming the word, where it stands and what reached it:

This network cannot be lowered at 36: word 2, dense, takes each example as a row of numbers, and each reaching it is 6x6x1: flatten it first.

Compile takes the description as it stands and draws nothing. The network is built at its first fit, from the shape of the rows it learns from and the run's seed, so the one seed the history records reproduces the whole run — what the layers start at, the order of every epoch's rows, what every dropout leaves out. .Input(shape) states the shape of an example, so every word is checked where the description is compiled; to hold the network before any rows, lower it by hand — description.Lower(new Shape(14), new RandomStream(42)) — and compile that.

The words take Keras's settings where Keras's differ: .BatchNorm(momentum, epsilon) takes Keras's momentum — the share of the running statistics a training batch leaves as they were, 0.99 in Keras — and keeps its complement, with Keras's epsilon, and .LayerNorm(epsilon) takes Keras's epsilon; without settings, both are PyTorch's. A window padded as TensorFlow's padding='same' is new Window(3, 3) { Stride = 2, PaddingMode = PaddingMode.Same }. And a last .Sigmoid() before BinaryCrossEntropy, Keras's habit, is lifted into the loss, which applies the sigmoid itself: the network ends at the layer before it, so every prediction goes through one sigmoid, not two. A stack that ends in a Sigmoid layer is refused where it is compiled with such a loss, naming the layer.

As code. A network is a layer that holds layers, and a LayerStack holds them in order:

using DeepSharp.Networks;

var stream = new RandomStream(42);
var network = new LayerStack(
    new Dense(14, 16, stream.Draw("initialise:0", 0, 0)),
    new Relu(),
    new Dense(16, 1, stream.Draw("initialise:2", 0, 0)));

That is exactly what the words above lower to under the same stream: the same numbers, bit for bit, and the same file. A network with a forward pass of its own derives from Network, adds its layers by name with AddLayer and says what it does in Compute; to be saved in its file, it says the name the file knows it by and how it is built again from what it wrote:

using System.Text.Json;
using DeepSharp.Networks;
using DeepSharp.Tensors;

public sealed class Passenger : Network, ISaved<Passenger>
{
    private readonly Dense _hidden;
    private readonly Dense _out;

    public Passenger(Draws draws)
    {
        _hidden = AddLayer("hidden", new Dense(14, 16, draws));
        _out = AddLayer("out", new Dense(16, 1, draws));
    }

    public static string Name => "passenger";                   // what its file calls it

    public static Passenger Rebuild(JsonElement settings, Rebuilding rebuilding) => new(rebuilding.Draws);

    public void WriteSettings(Utf8JsonWriter writer)             // nothing: its layers write their own
    {
    }

    protected override Tensor Compute(Tensor input, Pass pass) =>
        _out.Forward(pass.Backend.Relu(_hidden.Forward(input, pass)), pass);
}

A program that reads its file back registers it with the catalog it reads with: TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn().Register<Passenger>(), StepCatalog.BuiltIn()). One that is not registered is refused, naming it, rather than read as something else.

Every number a network holds is a slot with a dotted path — hidden.weight, 0.bias, 2.running_mean — its own first, then those of the layers it holds; Slots() lists them, and HeldLayers() every layer a layer holds, each with its path, so the layer holding any slot is named by the slot's path. A Parameter is moved by the optimizer; a RunningStatistic, such as a batch normalisation's mean, only by a training pass, so the validation rows never shape what the network keeps. A pass says whether it trains or evaluates, which is why no layer has a mode to switch.

Declared with the pipeline

This is the door the page opened with, written out. The chain that prepares the rows names the network they train, below its output — .WithTensorflow(…) in the words TensorFlow and Keras use, .WithTorch(…) in the words PyTorch uses — and pipeline.Train() runs the pipeline and trains exactly that network. A network written beside its pipeline and fitted by hand is the other door, and the sections below it are about both:

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("btceur-1d.csv")                                // declared, not opened
    .Declare(schema => schema
        .Timestamp("timestamp")
        .Number("close")
        .Optional("trades", ColumnKind.Number))              // a column that may have gaps
    .OrderBy("timestamp")
    .SettleGaps(gaps => gaps.Zero("trades"))                 // the gaps first: a quiet day traded nothing
    .AddFeature("turnover", "close", Arithmetic.Times, "trades") // then the features, worked out from settled columns
    .SplitByTime("timestamp", train: 0.70, validation: 0.15, gap: 1) // then the split: test is the rest
    .Ahead("close", 1, AheadAs.Return)                       // the answer: tomorrow's return
    .Drop("timestamp")                                       // what no model reads, before what learns
    .Normalise(scale => scale                                // and below it what learns from the training rows alone
        .Columns("close", "turnover")                        // where this pipeline's features land: -1 to 1 by default
        .MaxAbs("trades"))                                   // a count, divided by its largest, so a quiet day stays nought
    .Report(report => report.Measure(Metric.Rmse).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network                       // and the network these rows train
        .Dense(8).Relu().Dense(1)                            // Keras's words: the widths come from the rows
        .Adam()
        .MeanSquaredError()
        .Run(seed: 42, epochs: 20))
    .Build();

var trained = pipeline.Train();                              // the course, from the file to the trained model

var file = trained.ToJson();                                 // the network and its pipeline, one file

btceur-1d.csv is a file of your own, and none ships here: a row a day of bitcoin's price in euros, under a header that names timestamp, close and trades. Live sources lands a file with those columns among its ten.

The network is one more step of the declaration, so the file says what was trained behind its steps, and training it again from the file gives the same network, number for number. Train() runs the pipeline for the learner the declaration names, trains the network, measures it as the pipeline's report declares, and hands back the same TrainedNetwork a network written as code gives — a test holds the declared network and the hand-written one to the same numbers. prepared.Train() does it for a run already in hand, and a pipeline that names no learner says so rather than guessing one. Pipeline has the step as a file holds it, and a block of the Notebook holds the same step.

The words. They are the ones a network's own file writes, so a declaration and the network it trains say dense and adam alike and nothing is translated between them, and each takes the settings it takes in that file:
.Dense(units, bias, weights, biasStart) dense: units, bias, weights, biasStart, l1, l2, maxNorm, learns A layer that answers with that many numbers, each a weighted sum of the ones below it, plus a bias of its own unless bias is false. weights and biasStart name how its numbers start; l1, l2, maxNorm and learns are the rules its weights carry — the chain says them with .L1(…), .L2(…), .MaxNorm(…) and .Held() on the layer written last. Every setting but units is left out when it says what the layer always was.
.Relu(), .Tanh(), .Sigmoid() relu, tanh, sigmoid: nothing The activations: what is below nought left at nought, every number squashed between minus one and one, or between nought and one.
.Dropout(rate) dropout: rate Leaves a share of the numbers out at every pass while training, and none while serving.
.BatchNorm(momentum, epsilon, scale, shift) batchNorm: momentum, epsilon, scale, shift, learns Brings every number to the mean and spread of the rows it was trained on, then scales and shifts it by numbers it learns — unless scale or shift is false, which leaves that one out.
.LayerNorm(epsilon, scale, shift) layerNorm: epsilon, scale, shift, learns Brings every number of one row to that row's own mean and spread, and scales and shifts it the same way.
.Sgd(rate, momentum) sgd: rate, momentum, weightDecay, dampening, nesterov Moves every number against its gradient, carrying some of the last step's velocity.
.Adam(rate, epsilon) adam: rate, firstMoment, secondMoment, epsilon, weightDecay, amsgrad Moves every number by a running mean of its gradients over the root of a running mean of their squares.
.AdamW(rate, weightDecay, betas, epsilon) adamw: rate, firstMoment, secondMoment, epsilon, weightDecay, amsgrad Adam, with every number shrunk towards nought before its step, apart from its gradient.
.RmsProp(rate, alpha, momentum, epsilon) rmsprop: rate, alpha, epsilon, momentum, weightDecay, centered Moves every number by its gradient over the root of a running mean of the squares of its gradients. PyTorch's words only.
.Nadam(rate, betas, momentumDecay, epsilon) nadam: rate, firstMoment, secondMoment, epsilon, momentumDecay, weightDecay Adam with Nesterov's momentum, warmed up over the steps. PyTorch's words only.
.WeightDecay(decay), .Dampening(share), .Nesterov(), .AmsGrad(), .Centered() weightDecay, dampening, nesterov, amsgrad, centered on the optimizer The options of PyTorch's optimizers, said before or after the optimizer and refused by an optimizer that has no such option. PyTorch's words only: .WithTorch(…). Each is left out when it does nothing, and nothing is written.
.MeanSquaredError(), .CrossEntropy(), .BinaryCrossEntropy() meanSquaredError, crossEntropy, binaryCrossEntropy: nothing What the answers are judged by: an answer that is an amount, one among several classes read through a softmax, or a chance read through a sigmoid.
.EarthMoversDistance() earthMoversDistance: nothing An answer of shares of a whole in an order, read through a softmax: how far along the order the predicted shares have to move to be the answer's. Refused where the output is not ordered.
.StepDecay(every, factor), .ExponentialDecay(factor), .CosineDecay(epochs, minimum), .WarmUp(epochs, start) schedule: stepDecay, exponentialDecay, cosineDecay, linearWarmup How the rate changes from epoch to epoch; one schedule a declaration, the last said. Left out, the optimizer's own rate every epoch, and nothing is written.
.ClipGradients(norm) clip: a number The most norm the gradients of a step may have together before they move the network. Left out, they are not clipped, and nothing is written.
.ClipValue(bound) clipValue: a number The most any single gradient value may reach, either side of nothing, before the norm is clipped. Left out, no value is clipped, and nothing is written.
.StepEvery(batches) stepEvery: a whole number How many batches' gradients are averaged into one step. Left out, a step follows every batch, and nothing is written.
.Average(share) average: a number The share of itself an average of the weights keeps at every step, from nothing to below one. Left out, no average is kept, and nothing is written.
.StoppingAfter(patience, least, restoreBest) patience: patience, least, best Stops once the validation loss has not fallen for as many passes as it waits; least is how far it has to fall below the best to count, and best whether the run ends holding the numbers of its best pass.
nothing said never: nothing The run goes on to its last pass.
.Run(seed, epochs, batch) seed, epochs, batch The number every random draw of the run is worked out from, how many times it goes over the training rows at most, and how many rows one step is worked out from.
.On(engine) engine The name of the engine the run happens on.

Every number is written down, the ones a chain names and the ones it leaves unsaid alike — .Run(seed) writes a hundred epochs and batches of thirty-two, and a chain that never says when to stop goes on to its last pass — and a file that leaves a setting out is refused where it is read. The exception is every setting added in 0.10.0, which a table above marks as left out when it says nothing new: a dense layer with a bias, a normalisation with a scale and a shift, an optimizer with no decay, a run that clips no value, steps after every batch and keeps no average. They mean what the layer or the run always was, so a file that leaves them out is read as that, and a chain that says them at that meaning writes the file it always wrote. Said at any other meaning they are written, and the file is then one 0.9.0 refuses by its number.

The same words say it all at once, in the Titanic pipeline's chain: a first layer with no bias and a small penalty on its weights, a normalisation after it, momentum with Nesterov's look-ahead and a weight decay, and a run that takes each step from four batches, clips each gradient value and keeps an average of the weights:

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .WithTorch(network => network                     // PyTorch's words: the options below are PyTorch's
        .Dense(16, bias: false).L2(0.0001)            // no bias of its own, and a penalty on its weights
        .BatchNorm()                                  // its shift makes up for the bias that is not there
        .Relu()
        .Dense(1)
        .Sgd(0.05, momentum: 0.9).Nesterov().WeightDecay(0.0001)
        .BinaryCrossEntropy()
        .ClipValue(1).StepEvery(4).Average(0.99)      // the run: each value clipped, a step from four batches, an average
        .Run(seed: 20260929, epochs: 100))
    .Build();

var trained = pipeline.Train();                       // the network the average of its weights ends as

What the two doors write differently. The words TensorFlow and Keras use and the words PyTorch uses name the same layers, optimizers and losses, and both run on the light engine and on libtorch alike: a vocabulary is a way of speaking, never an engine. They differ in the number each writes where a chain says none, because the two libraries leave different ones:

Where a chain says nothing .WithTensorflow(…) .WithTorch(…)
.Adam()'s epsilon 1e-07 1e-08
.Sgd()'s rate 0.01 0.001
.BatchNorm()'s momentum and epsilon 0.99 and 0.001 0.9 and 1e-05
.LayerNorm()'s epsilon 0.001 1e-05

So the two write different files, which is right, since they are different runs. The file names no vocabulary: it says which run was asked for rather than which library asked for it, and nothing would read a name.

What is not declared. A pipeline hands over a flat row of numbers, so the layers that work on a row are here and the ones that work on an image — a window over it, a reshaping of it — are not: a network that takes images is written as code and trained on data a pipeline did not prepare.

Which engine. The step names the engine its run happens on — light unless said, .On("torch") in the chain — and the application says what the name stands for before the run: new Engines().Use("torch", TorchBackend.OnCpu()) gives the name an engine, and pipeline.Train(engines) trains on it. So one declared network runs on the light engine on a machine with nothing installed and on libtorch where libtorch is there, without a word of the file changing — TorchSharp backend has the engine — and which device it works on is the application's to say. A name nothing was given is refused by name, with what to do, rather than answered by an engine that would quietly give other numbers.

Read back. trained.ToJson() writes the network and its pipeline as it always did, and the pipeline carries the step, so a program that reads the file teaches its catalog the verb — TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks()) — and a catalog that was never taught says which package brings it.

The layers
Dense(inputs, outputs, beginning) Every value an example holds, weighed into each of so many, and a bias of its own added unless the beginning says there is none. Dense(inputs, outputs, draws) is the beginning that starts it as PyTorch starts a linear layer.
Relu, Tanh, Sigmoid The activations.
Dropout(rate) Leaves out a share of the values while training, and scales what it keeps, so evaluation lets everything through.
BatchNorm(features) Each feature normalised over the batch while training, and by the means and spreads it kept while evaluating; its Momentum is PyTorch's, the share a new batch takes. Scaled and shifted by numbers it learns, unless BatchNorm(features, new Affine { WithoutScale = true }) or WithoutShift leaves one out.
LayerNorm(features) Each example normalised over its own features, scaled and shifted the same way and left without either the same way.
Conv1D, Conv2D, Conv3D (channels, window, beginning) A window slid along each series, over each image or through each volume, channels last, as Keras keeps them; padded by the border it states, as TensorFlow's 'same', or — along a series — as Keras's 'causal'. Its Channels say how many it reads, how many it makes and in how many groups: one group weighs every channel into every channel, a group for every channel is a depthwise convolution. The two numbers alone, (inChannels, outChannels, window, draws), mean one group.
Conv1DTranspose, Conv2DTranspose, Conv3DTranspose (channels, window, draws, outputPadding) Gives back the places an ordinary convolution takes away: each place of the example spread over a window's worth of places and added where they overlap. Its border is stated at each end — it makes the example a worked-out border would be read from — and outputPadding keeps so many places at the end of each axis.
Pad1D, Pad2D, Pad3D (border, value) or (border, filling) Adds places at each end of each axis and does nothing else: nothing in them, the value given, or — with a Filling — places taken from the example itself, mirrored, repeated or gone round.
AdaptiveAvgPool1D…3D, AdaptiveMaxPool1D…3D (sizes) Pools to a size rather than by a window: place i takes the stretch of the example that belongs to it, so the stretches overlap where the size does not divide the length, and the example comes out the sizes it is given whatever it was handed.
MaxPool1D, MaxPool2D, MaxPool3D (window) The largest of what a window covers, for each channel on its own.
AvgPool1D, AvgPool2D, AvgPool3D (window) The average of it; a border the window pads with is left out of the average unless CountsPadding says it counts.
GlobalMaxPool1D, GlobalMaxPool2D, GlobalMaxPool3D, GlobalAvgPool1D, GlobalAvgPool2D, GlobalAvgPool3D The largest, or the average, of a whole channel: a row of channels for each example, or — with KeepsAxes — the axes it pooled kept as axes of one place.
SpatialDropout1D, SpatialDropout2D, SpatialDropout3D (rate) Leaves out whole channels of an example, in every place at once, while training; what is kept is scaled as dropout's is.
Flatten, Reshape(each) An example laid out as one row, or in another shape holding as many values.

Each starts as PyTorch's does — weights uniform by the number of values that reach them, biases within one over its square root — and each forward pass and its gradients were matched against PyTorch's on the same rows. That is what a layer starts as when nothing is said, and it is what it was in every earlier release; the next section says how to start it another way, or without a bias.

How a layer starts, and leaving out its bias

A layer's start can be told three things: how its weights begin, how its bias begins, and that it has no bias. They travel together as a Beginning, which takes the place the draws had in a layer's constructor — new Beginning(draws) is the start a layer always had — and, in a description, where the run supplies the draws, as new Beginning { … } with none.

Start Draws the numbers Reach for it when
KaimingUniform Evenly within one over the root of the values that reach a weight: PyTorch's start for a linear layer and a convolution. You say nothing. It is the weights' start unless told.
FanInUniform Evenly within one over the root of the values that reach the layer. You say nothing about a bias. It is PyTorch's bias start, and so the bias' unless told.
GlorotUniform Evenly within the root of six over the values that reach a weight and those it feeds. You are reproducing Keras's recipe, whose weights start so.
Zeros All nought. You are reproducing Keras's recipe, whose bias starts so, or a bias should begin neutral.
Ones All one. A number that should pass its input through at first.
Constant(value) All one value you give. A bias that should begin at a small positive number, say.
Normal(deviation, mean) A normal draw around nought: a deviation of five hundredths unless given. A recipe that names a normal start and its spread.
TruncatedNormal(deviation, mean) The same, redrawn when it falls beyond two deviations: Keras's. A recipe that names it, for a start with no outliers.
Orthogonal(gain) A matrix whose columns are at right angles to each other, scaled by the gain. A recipe that names it. Only a weight has one: a bias is a row of numbers, not a matrix.

The weights are drawn first and the bias second whatever starts them, so a layer that changes how its bias starts starts its weights where it did. A start never reaches a file: a file holds the numbers a layer learned, not how it began. A word of a declaration can name the starts that need no number of their own — weights: InitialiserWord.GlorotUniform, biasStart: InitialiserWord.Zeros, and kaimingUniform, fanInUniform, ones, normal, truncatedNormal and orthogonal — and a constant or a spread of your own is code.

Neither vocabulary changes what is left unsaid. A dense layer in Keras's words starts as PyTorch's does, as it always did, because a different default would change the numbers of every network written before. Keras's own start, Glorot's weights and a bias of nought, is asked for:

using DeepSharp.Networks;

var description = new Sequential()
    .Dense(32, new Beginning { Weights = new GlorotUniform(), Bias = new Zeros() })   // Keras's start for this layer
    .Relu()
    .Dense(16, new Beginning { WithoutBias = true })                                  // a layer with no bias at all
    .BatchNorm()                                                                       // whose shift makes up for it
    .Relu()
    .Dense(1);

var stream = new RandomStream(42);
var constant = new Dense(14, 16, new Beginning(stream.Draw("initialise:0", 0, 0)) { Bias = new Constant(0.1) });
var bare = new Dense(14, 16, new Beginning(stream.Draw("initialise:1", 0, 0)) { WithoutBias = true });

var hasBias = bare.Bias is not null;                  // false: the slot is not there, and so nothing is written for it

Leaving out a bias. A layer without a bias has one set of numbers fewer, adds nothing of its own to a sum, and writes "bias": false into the file; no slot is renamed, so every other number of the network keeps its path. Reach for it when a normalisation follows the layer, whose shift makes the bias redundant, when a model must pass through the origin, and when you are reproducing a model that was made without one (Keras's use_bias=False, PyTorch's bias=False). It costs nothing and changes nothing else; what it does not do is make a layer cheaper to train in any way worth measuring. The same goes for a normalisation: new Affine { WithoutScale = true } or WithoutShift keeps that one number out, as Keras's scale=False and center=False do, and leaving out both is PyTorch's affine=False. A bias cannot be given a start it will not use: a layer that has no bias and also says how its bias begins is refused, by name.

A layer built with its numbers in hand — new Dense(weights, bias) — takes a null bias for the same thing, and a layer's Bias is null when it has none, so code that reads it asks.

Penalising, bounding and holding still

Three rules can ride on a layer's numbers, each about the numbers of one slot — a dense layer's weights, a convolution's kernel, a normalisation's scale — and each said where Keras says it:

Rule Say it What it does Reach for it when
A penalty, L1 new SlotRules { L1 = 0.001 }, .L1(0.001) Adds that share of the sum of the absolute values of the numbers to the loss. Many weights should go to nought exactly, and the ones that matter stand out.
A penalty, L2 new SlotRules { L2 = 0.0001 }, .L2(0.0001) Adds that share of the sum of their squares, with no half in it. Weights grow large and a net memorises; a gentle pull of every weight towards nought.
A bound, MaxNorm new SlotRules { MaxNorm = 3 }, .MaxNorm(3) After every step, each unit's numbers — a dense layer's column, a convolution's filter — are scaled back when their norm is over the bound. No unit should be allowed to grow without limit.
A hold, Held layer.Hold(), .Held(), new SlotRules { Held = true } The numbers are moved by nothing: no gradient is worked out for them and the optimizer remembers nothing of them. A layer read from another framework whose numbers should stay while the layers after it learn, or a normalisation that should only apply what it kept. 
using DeepSharp.Networks;

var description = new Sequential()
    .Dense(32).Ruled(new SlotRules { L2 = 0.0001, MaxNorm = 3 })    // a penalty and a bound on this layer's weights
    .Relu()
    .Dense(16, new Beginning { WithoutBias = true })
    .Ruled(new SlotRules { L1 = 0.001 })                             // a penalty on the weights of the last layer written
    .Relu()
    .Dense(1);

// A network written as code puts the same rules on the parameters themselves, and holds a layer still with Hold().
var stream = new RandomStream(42);
var first = new Dense(14, 16, stream.Draw("initialise:0", 0, 0));
var last = new Dense(16, 1, stream.Draw("initialise:2", 0, 0));
var network = new LayerStack(first, new Relu(), last);

first.Hold();                                                        // nothing moves its numbers
last.Weight.Rules = new SlotRules { L2 = 0.0001, MaxNorm = 3 };      // a penalty and a bound on the weights of the last

A penalty or a weight decay. They look alike and are two things. A penalty is Keras's regularizer: it is added to the loss, so the loss a run reports includes it — in training and in validation, so early stopping compares losses that include it — and it is said for the weights of a layer or for its bias alone. A weight decay is PyTorch's: the optimizer adds a share of every number it moves to that number's gradient, which the loss never sees, and it reaches every number the optimizer moves, scales and biases too. Plain Sgd under a decay of twice an L2 moves the same way. Reach for a penalty when you want the pull in the reported loss or on one slot; for a decay when you reproduce a PyTorch recipe or want it on all of them.

What a hold costs and does. A held layer is excluded from the loop altogether — no gradient is worked out for it — and a normalisation all of whose numbers are held reads its running statistics while training too, so a frozen normalisation behaves as it does when serving. The held layers are part of a run: a checkpoint records them, and a run that goes on from it with another set held is refused, naming the layer. Letting a held layer learn again is a new fit, whose optimizer starts with nothing remembered for it, as in PyTorch. A file records "learns": false beside the slot.

What the rules do not do: a bound by unit is not defined on a transposed convolution's kernel — it is laid out by the channels it takes, not by each unit it makes — and is refused there, by name. Nothing penalises what a layer makes (Keras's activity regularizer), and no constraint but the largest norm is built. A declared dense layer takes l1, l2, maxNorm and learns for its weights, and the chain has .L1(…), .L2(…), .MaxNorm(…) and .Held() for the layer written last, in both vocabularies; a declared normalisation takes learns.

A series, an image, a volume

The same layers walk one axis, two or three. A window is a Window1D(length) for a series, a Window(height, width) for an image and a Window3D(depth, height, width) for a volume; each moves Stride places at a time — one number for every axis, or a number of its own for each, new Window(3, 3) { Stride = new Steps(2, 1) } — holds cells Dilation places apart (one, neighbours), takes the last place whose patch overhangs the end when RoundsUp says to, and pads by Padding places on every side, by a Border of before and after for each axis, or as TensorFlow's and Keras's padding='same' does (PaddingMode.Same: as many places as the stride fits, the odd place after), or — for a series — as Keras's padding='causal' (PaddingMode.Causal): the window's length less one places before the series and none after it, so no place sees what follows its own. A series is laid out step by channel, an image row by column by channel, a volume plane by row by column by channel: channels last, as Keras keeps them.

using DeepSharp.Networks;
using DeepSharp.Tensors;

// A series of 64 steps of 3 channels, in which a bump is somewhere.
var description = new Sequential()
    .Input(new Shape(64, 3))
    .Conv1D(16, new Window1D(5) { PaddingMode = PaddingMode.Causal })   // no place sees what follows it
    .Relu()
    .MaxPool1D(2)                                                        // runs of two, not overlapping
    .SpatialDropout1D(0.2)                                               // whole channels left out while training
    .Conv1D(16, new Window1D(3) { PaddingMode = PaddingMode.Same })
    .Relu()
    .GlobalMaxPool1D()                                                   // the strongest place of each channel
    .Dense(1);

var compiled = description.Compile(new Adam(0.01), new BinaryCrossEntropy());

A pooling written with a size, MaxPool2D(2), takes runs that do not overlap, as Keras's pooling does by default; written with a window, MaxPool2D(new Window(3, 3) { Stride = 2, Padding = 1 }), it walks as that window does. The largest a window covers is the first of values that tie, and the gradient of a loss goes to it alone, as PyTorch's max pooling sends it. A window over a border sees the values of the example only: the largest is never a nought the border supplies, and an average leaves the border out of its count, as TensorFlow's and ONNX's do. PyTorch counts a border it pads with, so a network trained there is AvgPool2D(window) { CountsPadding = true }. A global pooling gives a row of channels, which a dense layer takes directly; KeepsAxes keeps the axes it pooled as axes of one place, as Keras's keepdims and ONNX's global pools do.

A convolution's kernel is one matrix for every axis: as many rows as the window holds values, channels in included, by as many columns as channels out, which is Keras's kernel flattened. Its start is PyTorch's. Every value and every gradient of the convolutions and the poolings was matched against PyTorch's on every engine, to a ten-thousandth for a convolution and a hundred-thousandth for a pooling; the border PyTorch cannot say — a stride with 'same', a causal border — is worked out as TensorFlow works it out and padded by hand, so the numbers are still PyTorch's arithmetic over it. The kinds in a network's file are conv1d, conv2d, conv3d, conv1dtranspose…3dtranspose, pad1d…3d, maxpool1d…3d, avgpool1d…3d, adaptivemaxpool1d…3d, adaptiveavgpool1d…3d, globalmaxpool1d…3d, globalavgpool1d…3d and spatialdropout1d…3d; a library older than this one refuses them by name.

A convolution of groups holds a kernel as small as its groups — the values of its window for the channels of one group, by the channels it makes — on purpose: a kernel of the full size with noughts off the diagonal would receive gradient in those noughts, and a convolution trained that way stops being grouped. DepthwiseConv1D(multiplier, window), DepthwiseConv2D and DepthwiseConv3D are the description's words for a group of every channel making so many channels of each; Keras's separable convolution is one of those and a convolution of a window of one place after it, which is how such a file is read.

What is not built is refused where a source says it: a max pooling's storage_order, and the places a max pooling picked as a second output. The rest of what this page used to list — a stride of its own for each axis, a dilated window, channels in groups, a pooling that rounds its number of places up (ceil_mode), a pooling to a size, a transposed, depthwise or separable convolution, and a layer that only pads — is built, each with its values and its gradient matched to the framework that has it, on every engine.

Borders taken from the example. A padding can fill the places it adds from the example itself, which a border of noughts cannot: Pad1D(sides, Filling.Mirror), Pad2D(margins, Filling.Repeat), Pad3D(margins, Filling.Wrap).

Filling A place is filled with PyTorch's Reach for it when Limit
Constant Nought, or the value given. constant The usual: an edge that reads as nothing. None.
Mirror The example reflected at its edge, the edge not repeated. reflect A hard edge of noughts would be a feature the network learns, as at the border of a picture or a signal. The padding is at most the length less one.
Repeat The edge value, again and again. replicate A series continues at its last value. None.
Wrap The other end of the example. circular The signal goes round: an hour of the day, an angle. The padding is at most the length. 
using DeepSharp.Networks;
using DeepSharp.Tensors;

// A series of 64 steps of 3 channels, padded by two places at each end with the series itself, mirrored, before a window of five.
var description = new Sequential()
    .Input(new Shape(64, 3))
    .Pad1D(new Sides(2, 2), Filling.Mirror)
    .Conv1D(16, new Window1D(5))
    .Relu()
    .GlobalMaxPool1D()
    .Dense(1);

var images = new Sequential()
    .Input(new Shape(28, 28, 1))
    .Pad2D(new Margins(new Sides(1, 1), new Sides(1, 1)), Filling.Wrap)   // a picture that wraps round its edges
    .Conv2D(8, new Window(3, 3))
    .Relu()
    .Flatten()
    .Dense(10);

Every value is the example's own, copied, and the gradient of a place copied twice is the sum of what its copies did; the three were matched to PyTorch's values and gradients on every engine. A padding that asks for more than the example can give — a mirror of the length or more — is refused by name when the layer is shown an example, and a value beside a filling other than the constant is refused. A padding with a filling is written to the file as "border"; a constant padding is written as it always was. ONNX's Pad is read as these: reflect, edge and wrap (from opset 19).

Losses, optimizers and schedules
MeanSquaredError For amounts: the numbers as they are.
CrossEntropy For shares of a whole: the numbers through a softmax.
BinaryCrossEntropy For chances: each number through a sigmoid.
EarthMoversDistance For shares of a whole in an order, as bands of weight are: the numbers through a softmax, and how far, along the order, the predicted shares have to move to be the answer's.
Sgd(rate) { Momentum, WeightDecay, Dampening, Nesterov } PyTorch's stochastic gradient descent.
Adam(rate) { Betas, Epsilon, WeightDecay, AmsGrad } PyTorch's Adam.
AdamW(rate) { Betas, Epsilon, WeightDecay, AmsGrad } PyTorch's AdamW: every number is shrunk by the rate times the decay before Adam's step, apart from its gradient.
RmsProp(rate) { Alpha, Epsilon, Momentum, WeightDecay, Centered } PyTorch's RMSprop.
Nadam(rate) { Betas, Epsilon, MomentumDecay, WeightDecay } PyTorch's NAdam.
ConstantRate, StepDecay, ExponentialDecay, CosineDecay, LinearWarmup How the rate changes from epoch to epoch; a warm-up rises in a straight line from a share of the rate to the whole of it, and may hand over to another schedule.

A loss takes the network's raw numbers and says how they are read, so there is no softmax layer to forget or to apply twice, and a prediction goes through the same activation. It is always named, and it refuses a row whose answers it could not have meant — shares that do not sum to one, a chance above one — naming the row as it was read, before anything is trained on it.

The earth mover's distance is the loss for an answer whose columns are in an order. It adds up the predicted shares and the answer's from the first column, and the loss is the size of the difference between the two, summed over every column but the last — where both have added up to one — and averaged over the batch: a prediction one band off costs one, and ten bands off costs ten, which a cross-entropy cannot tell apart. It is the distance Metric.Emd measures. A pipeline whose output does not say its columns are ordered is refused it, by name, before anything is trained. With a remainder on the output the bands are compared as shares of what they hold together and the last answer, what is left, by the size of how far its share is off, so moving a share into what is left is never taken for moving it along the order. The squared form of the distance was measured on synthetic flocks and ended behind the cross-entropy, so it is not built.

AdamW, RMSprop and Nadam are matched to PyTorch 2.14.1 step by step, on the light engine and on libtorch alike, and each remembers its slots by parameter under PyTorch's names — exp_avg, exp_avg_sq, square_avg, momentum_buffer and the steps — so a checkpoint carries them. Where a vocabulary cannot name one honestly the door refuses: RMSprop and Nadam are declared in PyTorch's words only, because Keras's differ in arithmetic, and AdamW in Keras's words writes PyTorch's numbers.

Which option, when. Each does nothing unless you ask, as in PyTorch, each is held to torch's own numbers step by step on every engine, and each is remembered in a checkpoint under PyTorch's name for it.

Option On What it does Reach for it when What it costs or does not do
WeightDecay Sgd, Adam, RmsProp, Nadam Adds that share of every number to its gradient before the step, once, whichever optimizer. You reproduce a PyTorch recipe that says weight_decay, or want a pull towards nought on every number. Not AdamW's decay, which shrinks apart from the gradient and is AdamW's own WeightDecay; not a penalty, which sits in the loss.
Dampening Sgd Lets the velocity take in less of each new gradient, by that share; the first gradient is taken whole. A recipe that says so. Needs a momentum; refused beside Nesterov.
Nesterov Sgd Takes the step from the gradient plus the momentum times the velocity. Momentum training where looking ahead helps, or a recipe that says so. Needs a momentum above nothing and no dampening; refused otherwise, by name, wherever the optimizer is made.
AmsGrad Adam, AdamW Takes the step over the largest mean of squares there has been. An Adam run that fails to settle where a smaller step later would have helped. One more tensor for every number in the optimizer's memory.
Centered RmsProp Divides by the root of the mean of the squares less the square of the mean gradient. A recipe that says centered. One more tensor for every number in the optimizer's memory.

These are PyTorch's options, and they are declared in PyTorch's words — .WithTorch(…). A network declared in Keras's words is refused them by name, saying why: Keras's weight_decay shrinks the number itself, its SGD has no dampening, and its AMSGrad places its epsilon otherwise. An option an optimizer does not have is refused by that optimizer. A checkpoint now also refuses a tensor its optimizer does not keep, as it already refused one it lacks, so a checkpoint of one optimizer is not gone on from under another.

An optimizer of your own. Optimizer is open: a subclass says what it remembers of each parameter with MemoryOf and takes it back with Recall, which hand and take a SlotMemory, and registered with a catalog by name it goes through a checkpoint file and on from it bit for bit. A loss of your own derives from Loss as it did. Neither is a word of a declared network: a pipeline's file names the shipped words, because a vocabulary a file names must be one every reader has, and a network of your own is compiled and fitted by hand.

The loop

compiled.Fit(train, validation, new FitOptions(seed)) trains on rows of tensors, and compiled.Fit(prepared, options) on the rows a pipeline hands over. The seed is asked for and never assumed; the rest is Keras's fit: one epoch and batches of thirty-two unless said, the rows shuffled afresh every epoch and the last batch kept short. EarlyStopping { Patience, MinDelta, RestoreBest } stops the run as Keras's does, and restoring the best epoch brings back every slot. A loss that is not a finite number stops the run with the epoch and the batch it came from. The options hold the light engine unless they name another — new FitOptions(seed) { Backend = engine } — and that engine does the whole run: the training passes, the looks at the validation rows and the report's measures. Every batch's gradients are worked out by a recording of its pass, and only towards the slots the optimizer moves: nobody asks how a batch's own rows moved the loss, so that way back is never worked out.

History holds what the run did: every epoch's training loss, validation loss and rate, the seed, the best epoch and why it stopped.

Clipping, watching and stopping. GradientClip in the options — new FitOptions(seed) { GradientClip = new GradientClip(1.0) } — scales a step's gradients down to a norm together when they are over it, as PyTorch's clip_grad_norm_ does, and passes them untouched, to the last bit, when they are under; ClipValue, below, clips each value on its own. OnEpoch is handed each epoch as it ends, in order, on the training thread, and Cancellation stops the run before its next batch or its next look at the validation rows by throwing an OperationCanceledException and returning nothing — no history, no best epoch put back. The network then holds what the last finished batch left it, so fitting it again goes on from there, and a checkpoint taken before is as it was. A run that neither stops is the run without them, bit for bit.

Clipping each value, stepping once every several batches, averaging the weights. Three more things a run can be told, all off unless asked, and all of them part of what a checkpoint records:

Option Say it What it does Reach for it when What it costs or does not do
ClipValue ClipValue = 1, .ClipValue(1) Brings each gradient value to within the bound either side of nothing, before the norm is clipped. A few spikes in a gradient should not move a number far, or a recipe clips by value. A value inside the bound passes unchanged. A gradient that holds a value that is not a finite number is refused by name rather than clipped.
StepEvery StepEvery = 4, .StepEvery(4) Takes one step from the mean of the gradients of that many batches. The batch you want does not fit in memory, or a step should come from more rows than a batch holds. A window never goes across the end of an epoch: the last batches of one are stepped as a window of however many are left. A normalisation still counts each batch on its own, so it is not the run of a batch that many times larger.
Average Average = 0.99, .Average(0.99) Keeps an exponential moving average of the weights; when the run has finished, the network holds the averages. The last steps' weights are noisy and a smoother end serves better. A copy of the weights. Every epoch is still judged by the weights as they are, and it is refused beside RestoreBest, since both would say what the run ends holding. 
using DeepSharp.Networks;

var options = new FitOptions(seed: 7)
{
    Epochs = 50,
    ClipValue = 1,                           // no single gradient value beyond one, clipped first
    GradientClip = new GradientClip(5.0),    // then the norm of all of them together, if over five
    StepEvery = 4,                           // a step from the mean of four batches' gradients
    Average = 0.99,                          // and an average of the weights, which the network holds at the end
};

In a step the order is always the same: the gradients of the window are averaged, each value is clipped, the norm is clipped, the optimizer steps, and the average takes in the new weights. The average is part of a checkpoint, so a run goes on from one with it to the bit; no window is open across an epoch, so a checkpoint holds none.

Checkpoints are taken every epoch, or only when the validation loss improves, and a run goes on from one exactly as if it had never stopped — on the engine it was taken on, bit for bit. A checkpoint records what its run went under: the seed, the batch size, the early stopping, the clip, the clip of each value, the window of batches, the average, the layers that were held and the engine, with the version and the device an engine that implements INamesItsVersionAndDevice names. Handed another engine — or another seed, batch size, early stopping, clip, window of batches, average or set of held layers — a run going on from it is refused, naming each difference, before anything is put back, since it would be another run under the same seed: gone on from on libtorch, the networks sample's Titanic run ended with weights up to 3.0e−7 from those of the run it was taken of. The epochs are not among them, since going on to more of them is what a resume is for:

var prepared = pipeline.Run();                       // the rows alone, for a network written here rather than declared
var compiled = new Sequential().Dense(16).Relu().Dense(1).Compile(new Adam(0.01), new BinaryCrossEntropy());

compiled.Fit(prepared, new FitOptions(seed: 7)
{
    Epochs = 50,
    Checkpoints = new Checkpoints(checkpoint => File.WriteAllText("run.json", CheckpointFile.Write(compiled, prepared, checkpoint))),
});

var resumed = CheckpointFile.Read(File.ReadAllText("run.json"), NetworkCatalog.BuiltIn(), prepared);
var more = resumed.Compiled.Fit(prepared, new FitOptions(seed: 7) { Epochs = 100, ResumeFrom = resumed.Checkpoint });

CheckpointFile.Read reads the file once, whole, before it holds it to the pipeline handed over: its network is held to the pipeline the file carries, and that to the one handed over. A checkpoint kept under keys of your own is read with NetworkDocument.ReadCheckpoint(json, network, training, catalog), its network and its run from one reading of the text. A checkpoint 0.4.0 wrote records neither the batch size, nor the early stopping, nor the engine, and goes on under whatever it is handed, as it did.

Behind a pipeline

What both doors hand back — pipeline.Train() and Fit(prepared, options) — is a TrainedNetwork: the network, its history, how the pipeline's report measured it, and what it predicts for rows that arrive later — in the answer's own units, through the loss's activation and the pipeline's way back:

var passengers = new InMemoryRowSource(
    ["pclass", "sex", "age", "sibsp", "parch", "fare"],
    [["3", "male", "22", "1", "0", "7.25"], ["3", "Male", "22", "1", "0", "7.25"]]);

var predictions = trained.Predict(passengers);       // the chance each passenger survived: 0.16 for the first, as it ran here
var unfamiliar = predictions.Unfamiliar;              // nothing for the first; sex_other for the second

A target comes back as a value or the chance of one, a label as its chance, a share of a whole as a count by the row's own total, and a return as a price by the row's own price. A class is never guessed: where the line between classes lies is for the use to say. trained.Predict(rows, engine) serves on the engine handed in, for that call alone, and trained.Predict(rows) on a light engine of its own; either asks the engine for thirty-two rows at a time, Keras's own default, and no answer depends on which rows share its pass.

What the network learned nothing about. A feature that held one value on every training row taught the network nothing. The encoder keeps a place for a category the training rows never held — sex_other — and marks a cell that was empty — sex_was_missing — and on the Titanic passengers those are nought on all 623 training rows, so the first layer's weights for them are still the random start's after training. A passenger written 'Male' is answered from there, by the seed rather than by anything learned: across six seeds the same passenger was given 0.257 to 0.495 as 'Male', against 0.162 to 0.186 as 'male'. Unfamiliar names, for each served row, the features it moves away from the one value the training rows held, and the report counts the rows of each part that do, as UnfamiliarRows beside Rows. The answer itself is left as it is: a pipeline that should answer no category its training rows never held declares EncodeCategories(unseen: Unseen.Refuse).

A network takes every feature on one scale, so it is refused a run made for a learner that does without the scales or takes categories itself — RunFor(Needs.NoScale) or RunFor(Needs.Categories), which Pipeline has — and fits behind a run of every step. A pipeline read from its file holds no rows, and fitting behind it says so, and how to have them: run its declaration over the rows, new Pipeline(declaration, rows, folder).Run().

One file. trained.ToJson() writes the network and the pipeline it was trained behind together, and TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn()) reads them back in a program that has never seen the data — with StepCatalog.BuiltIn().WithNetworks() when the pipeline names its network. The network's part says what it was trained on: its features, its answers and the output that named them, the seed, the epoch its numbers come from, each feature that held one value on every training row, and the SHA-256 of the pipeline's own file as it was fitted. The same names are not the same fit — fitted again on a few more rows, a pipeline keeps every feature's name and moves the numbers it learned — so the network is read only beside the very fit it was trained behind, and refused beside any other, when it is loaded and when a run goes on. A checkpoint is the same file with what the run needs to go on, and serves as a trained network too. The file names no engine, so a network trained on one engine is read back and served on another, and every line of it ends with a line feed, on every system.

The file is one text, and .NET makes no text longer than 1,073,741,791 characters, so the one file holds a network of up to about 45 million parameters — its numbers take about 24 characters each — and a checkpoint under Adam, which keeps two more numbers a parameter, one of about 14 million, or about 10 million keeping the best epoch's. Reading a network's file takes 36 bytes a parameter — the text's own copy and the numbers — and a checkpoint under Adam 96, or 128 keeping the best epoch's.

The report and the charts

The measures a network is held to are the pipeline's, declared before any number exists: .Report(...) names them — RMSE, MAE and R² for amounts; accuracy, precision, recall and the confusion matrix for classes; Emd, Kl and Rps for shares of a whole, Emd and Rps along an order — the parts they are taken on, and how they are shown. Pipeline has the rules. Each measure is in the answer's own units, beside what predicting the training rows' average would score, by scikit-learn's definitions for amounts and classes and scipy's for shares; the network's measures are taken on the engine its run was handed, by the same evaluation it serves with.

trained.Measures!.Report(), in DeepSharp.Charts, is the report as it says its measures are shown — the numbers, the charts or both — as one value whose ToHtml() gives its HTML, so a notebook's C# cell that ends with it shows the report. What the report measured can also cross as text: trained.Measures!.PredictionsToJson() writes, for each part, the key of each row and what was predicted for it, with the fit the predictions were made behind, and prepared.MeasureAgain(text) measures that text again on another run of the same pipeline — refused beside any other fit, as the network's file is. That is how a notebook's report block draws a network a C# cell trained: Notebook.

DeepSharp.Charts draws them, as the text of an SVG, drawn with MatPlotLibNet:
history.LossCurve() The training loss of every epoch, and the validation loss beside it.
history.LearningRates() The rate every epoch took: the schedule.
measures.Bars() Every measure, each part's beside the average's.
measures.ConfusionMatrices() Each confusion matrix as a heatmap of counts, a row to each class the rows held.
measures.PredictedAgainstActual() What was predicted against what was there, part by part, beside the line where they agree.
measures.Residuals() What was left over against what was predicted.
correlation.Heatmap() A correlation, on the whole of its scale.

A network written down

A network's file holds the kinds it is made of — each by the name it is registered under, with the settings it is rebuilt from — every slot's numbers by its path, and its loss. NetworkDocument writes and reads it, through a NetworkCatalog of the kinds a reader knows; a network written as code is saved by a name of its own — it implements ISaved<T> and is registered with NetworkCatalog.BuiltIn().Register<Passenger>() — and nothing is ever made from a type looked up by its name. Read back, it is the same network to the last bit. Everything wrong with a file is said at once, each at its line and column:

(9,13): 'dens' is not a kind this catalog knows. The nearest one it knows is 'dense'.
(15,7): '0.bias' is missing: every slot of the network is written.
(17,9): '0.weight' is a 4x1 slot here, and is written as 2x4x1.

A part is written as the oldest version that says what it holds. A layer without a bias writes "bias": false, a normalisation without a scale or a shift "scale": false or "shift": false, a padding that fills from the example "border", a number with a penalty, a bound or a hold l1, l2, maxNorm or "learns": false beside its values, an optimizer with an option weightDecay, dampening, nesterov, amsgrad or centered, and a run clipValue, stepEvery, average and the averaged numbers — each only when it says something, and each makes its part the fifth version, which 0.9.0 refuses by its number. A network and a run that say none of them are written as they always were, and the repository holds files the published 0.9.0 packages wrote, which every release reads, writes again and serves.

Numbers trained somewhere else go into a network the same way: network.Load(entries) puts each into the slot its path names, all of them or none, as PyTorch's load_state_dict does with strict=True, and refuses a missing, extra or misshapen slot in the same words as the file does. A model PyTorch, Keras or an ONNX exporter saved is read so, into a network written here or one its file describes — Importing a model has each reader.

What is not here

  • A schedule that watches the validation loss. The rows a model is chosen on would then shape the weights it is chosen for.
  • Every regularizer and constraint Keras has. A penalty is L1, L2 or both, and a bound is a largest norm for each unit. An orthogonal regularizer, a unit norm, a non-negative constraint and a penalty on what a layer makes are refused by name where a Keras file says one, rather than read as something they are not.
  • A norm clipped tensor by tensor. The norm of all the gradients together is PyTorch's clip_grad_norm_, and each value can be clipped on its own; nothing else says a clip.
  • A start in a file. A file holds the numbers a layer learned, never how they began.
  • A word of your own for a loss or an optimizer in a pipeline's file. Yours is code: compile and fit it by hand.
  • Hand-written GPU code. A graphics card is reached through an engine that already reaches one: libtorch, through the TorchSharp backend.

DeepSharp

Clone this wiki locally