Repository navigation
Getting Started
dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines
dotnet add package DeepSharp.Learners.Networks
dotnet add package DeepSharp.Charts
A network trained behind a pipeline lives in DeepSharp.Learners.Networks, and its charts in DeepSharp.Charts. The
data half lives in DeepSharp.Pipelines, with DeepSharp.Pipelines.Parquet, .Excel and .Json for reading those
files, DeepSharp.Pipelines.DataFrame for reading through Microsoft's DataFrame and DeepSharp.Pipelines.Indicators
for indicators over a series — add the ones you use. DeepSharp.Backends.TorchSharp runs the arithmetic on libtorch —
TorchSharp backend — and DeepSharp.Import.PyTorch, .Keras and .Onnx read a model trained elsewhere —
Importing a model.
To write a pipeline as a notebook, install DeepSharp.Verso.Notebooks from Verso's Extensions panel; it runs in
Verso's VS Code extension, in verso serve and in an application of your own, through DeepSharp.Verso.Api: see
Notebook. Or let DeepSharp's own server show it in your browser, with nothing else to install — run it
beside a notebook, or in a folder of them; from the repository's folder, this opens the sample notebook:
dotnet tool install --global DeepSharp.Verso.Serve
deepsharp-serve Samples/titanic.verso
Or build it all from source:
git clone https://github.com/xkqg/DeepSharp.git
cd DeepSharp
dotnet build DeepSharp.slnx -c Release
It runs on .NET 8 and .NET 10, on Windows, Linux and macOS. The default engine needs nothing installed; libtorch, the heavier one, is a package of its own, and choosing it is one line.
using DeepSharp.Tensors;
var maths = new CpuBackend();
var a = Tensor.From(new Shape(2, 2), [1f, 2f, 3f, 4f]);
var b = Tensor.From(new Shape(2, 2), [10f, 20f, 30f, 40f]);
var sum = maths.Add(a, b); // 11, 22, 33, 44
var product = maths.Multiply(a, b);
Console.WriteLine(sum); // Tensor 2x2A network learns by knowing which way to move each of its numbers. Hand the pass a RecordingBackend instead of the
plain one, work out a loss of one value, and ask which way each number moved it:
var pass = new RecordingBackend(maths);
var rows = Tensor.From(new Shape(2, 2), [0.80f, -0.57f, 0.79f, -0.67f]);
var weights = Tensor.From(new Shape(2, 1), [0.1f, -0.2f]);
var bias = Tensor.From(new Shape(1), [0f]);
var wanted = Tensor.From(new Shape(2, 1), [0f, 0.01f]);
var misses = pass.Subtract(pass.AddRow(pass.MatMul(rows, weights), bias), wanted);
var loss = pass.Mean(pass.Multiply(misses, misses));
var gradients = pass.GradientsOf(loss, [weights, bias]);
Console.WriteLine(gradients[weights]); // Tensor 2x1: how much each weight moves the lossThe recording is made for one pass and dropped after it; nothing else holds it.
A wrong size is caught where you write it. new Shape(2, 3) means two rows of three. Add a 2x3 to a
3x2 and you get an error naming both, not a number that quietly went wrong three layers later. A shape with
no sizes at all is a single number — that is what a loss is, and it behaves like one everywhere.
You hand over the maths; nothing goes looking for it. There is no DeepSharp.Backend.Current and there
never will be. You create a backend and pass it to what needs it. A shared one that anything could reach and
replace is how two halves of a program end up disagreeing about where their work runs. It is also the whole
point of the design: your model talks to a backend, so pointing it at libtorch instead of at the built-in
engine is one line — new FitOptions(seed: 42) { Backend = TorchBackend.OnCpu() } to train on it,
trained.Predict(rows, engine) to serve on it — and the model itself does not change.
Tensor.From copies the numbers you give it. So if you reuse your own array afterwards, the tensor you
already handed over stays as it was — and giving the same tensor to two layers is safe. Nothing writes into
what you passed in.
The ten steps below are the course this library is named after, one page each: what the step is for, the one line you write, what the row looks like after it, and where every option lives. Walked in order they build one pipeline from a file to a trained model; read on their own they answer "where does this belong?".
| step | what it decides | |
|---|---|---|
| 1 | Reading a file | where the rows come from, and what each column looks like before you decide |
| 2 | Declaring the columns | what takes part, what kind each column holds, and which may be absent |
| 3 | Settling the gaps | the gaps a feature will be worked out from, with a value nobody learned |
| 4 | Working out the features | the columns worked out from other columns, above the split so the split divides them |
| 5 | Splitting the rows | the line: which rows may teach, which may choose, which may only measure |
| 6 | Naming the answer | what the model is asked to predict, kept apart from what it is given, named right below the line |
| 7 | Filling and scaling | everything learned from the training rows and replayed unchanged |
| 8 | Measuring and drawing | how every run is measured, declared before a number exists |
| 9 | Naming the network | which network these rows are prepared for, in Keras's words or PyTorch's |
| 10 | Saving and replaying | the one file, and what a replay is held to |
To start a pipeline from these steps rather than from memory — every one present, in this order, waiting for what only you know — see Starting from a course.
The rest of this page is the short version: install it, add two tensors, and run the whole course at once.
using DeepSharp.Pipelines;
var passengers = Pdd.Create()
.ReadCsv("titanic.csv")
.Declare(schema => schema.Integer("survived", "pclass").Optional("age", ColumnKind.Number).Number("fare"))
.SplitStratified("survived", train: 0.70, validation: 0.15)
.Target("survived")
.FillMissing("age", With.Median)
.Normalise("age", "fare")
.Build()
.Run();
var train = passengers.Batch(Part.Train); // the numbers, their names, and the answers apartEverything below the split is fitted on the training rows alone and replayed unchanged on the rest — PDD says why, and Pipeline has every verb.
One line does one thing to many columns. .Normalise("age", "fare") scales both the way a caller who names no kind
means it — between minus one and one, which is where a network takes its features — and
.Normalise(scale => scale.MidRange("age", "fare").Robust("volume")) names a kind per group;
.FillMissing(fill => fill.Median("age").Mean("trades")) is the same shape for the gaps, and so are .Reshape,
.Cyclical, .ClipOutliers, .FillNaN and .TimeParts. Each writes one step a column, as writing the verb once a
column does. .SettleGaps(gaps => gaps.Zero("trades")) has that shape as well and stands above the split: a nought,
a number you choose or a refusal is not a value any row decided, so a feature worked out after it is worked out from
settled columns. .ScaleGiven(scale => scale.Between("fare", 0, 512)) stands there for the same reason — the bounds are
yours rather than the rows' — and .Shuffle(seed) puts the rows in an order drawn from a seed before they are divided,
so every part holds the same mixture; without it they keep the order they were read in. Where the features land is said once for the whole pipeline, above the split:
.DefaultFeatures(Form.Unit) puts them between nothing and one instead, and a column that names its own kind keeps
it.
Not sure what a column holds? Pdd.Create().ReadCsv("titanic.csv").ProposedKinds() reads every cell and proposes a
kind for each column — alive true or false, embarked a category, age a number — to write the schema from. It
is a proposal and nothing more: the schema is still yours to write.
A network takes every feature declared to land between minus one and one, so the scales say so; the pipeline declares what the network will be held to before anything learns, and then the network itself — the layers, what moves them, what judges them and when the run stops. Running it trains exactly that network:
using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;
var pipeline = Pdd.Create()
.ReadCsv("titanic.csv")
.Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
.SplitStratified("survived", train: 0.70, validation: 0.15)
.Target("survived")
.FillMissing("age", With.Median)
.EncodeCategories()
.Normalise("age", "fare", "sibsp", "parch") // a network takes them between -1 and 1, which is the default
.Report(report => report.Measure(Metric.Accuracy, Metric.ConfusionMatrix).On(Part.Validation, Part.Test).As(Shown.Numbers))
.WithTensorflow(network => network // the network these rows train
.Dense(16).Relu().Dense(1) // the widths come from the rows
.Adam(0.01)
.BinaryCrossEntropy()
.Run(seed: 20260929, epochs: 100)
.StoppingAfter(patience: 10))
.Build();
var trained = pipeline.Train(); // the steps, and then the network they name
var chance = trained.Predict(new InMemoryRowSource(
["pclass", "sex", "age", "sibsp", "parch", "fare"], [["3", "male", "22", "1", "0", "7.25"]]));
File.WriteAllText("titanic.network.json", trained.ToJson()); // the network and its pipeline, one file
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());The whole course is one declaration, so the file says what was trained behind its steps and training it again from the
file gives the same network. It learns from the training rows, is judged by the validation rows, and stops once they no
longer improve; the test rows reach it only when the report measures it, each measure beside what predicting the
training rows' average would score. Predict answers a passenger nobody trained on with the chance they survived, and
the file reads back with TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks())
— beside the very pipeline fit it was trained behind, and no other; the catalog is told the verb the declaration names.
The same network written as code trains the same, bit for bit: new Sequential().Dense(16).Relu().Dense(1) .Compile(new Adam(0.01), new BinaryCrossEntropy()).Fit(prepared, options) over a pipeline that was only .Run(), which
is the door to take when the network is not yours to write — one read from a Keras or PyTorch file, say. The run happens
on the engine the declaration names, light unless said, and an application gives that name an engine before the run:
pipeline.Train(new Engines().Use("torch", TorchBackend.OnCpu())). trained.Predict(rows, engine) serves on whichever
engine it is handed. Networks has all of it.
Two programs and a notebook in the repository run all of this on the published files in Samples/data:
dotnet run --project Samples/DeepSharp.Sample.Pipelines -c Release -f net10.0
dotnet run --project Samples/DeepSharp.Sample.Networks -c Release -f net10.0
The first asks the passenger list what each of its columns holds before anything is declared, has a profile say what
should not be there and how each is answered, and prepares the passengers and a price series, reaching its pipelines
the way an application reaches a service. The second trains a network on each of three datasets, has each pipeline's
report measure it, saves it as its one file, reads it back and serves a row, and writes its charts beside itself. The
notebook, Samples/titanic.verso, writes the Titanic pipeline block by block; its C# cell trains a network on what the
blocks hand over, and its report block draws the measures — open it in Verso, or with deepsharp-serve as above. A
test runs each as it stands.
A test project is a program you run, not a suite you point a tool at — so dotnet test is not how it works
here. There are six: the library's; the libtorch engine's, which carries libtorch, as the library's never does; the
ML.NET learner's, which carries ML.NET and the native side it comes with; and the notebook's, the host's and the
server's, which run on Verso's own engine.
Each is built for .NET 8 and for .NET 10, so -f says which to run on; -f net8.0 runs the same tests on .NET 8:
dotnet run --project Tst/DeepSharp/DeepSharp.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Backends.TorchSharp/DeepSharp.Backends.TorchSharp.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Learners.MLNet/DeepSharp.Learners.MLNet.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Notebooks/DeepSharp.Verso.Notebooks.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Api/DeepSharp.Verso.Api.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Serve/DeepSharp.Verso.Serve.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp/DeepSharp.Tests.csproj -c Release -f net10.0 -- -class DeepSharp.Tests.Tensors.ShapeTests
The library's suite compiles every C# block of this wiki against the packages as built, and the notebook's runs every
block marked as a notebook's cell in Verso's own engine, so both need the wiki cloned beside the repository, as
DeepSharp.wiki.
To see how much of the library the tests actually reach — which is a check that fails, not a number to admire — see Quality.