Repository navigation
PDD
Pipeline-driven design is the idea DeepSharp is built around. It is not a feature you switch on; it is the shape everything else here takes. The whole course from raw data to a checked model is declared in advance as one artefact, the pipeline, and then replayed.
The one rule to remember: anything that learns from the data is fitted on the training split alone, and then replayed unchanged — on validation, on test and on live data.
| You want to… | Write | Where it stands |
|---|---|---|
| Settle a gap with a value nobody learned | .SettleGaps(gaps => gaps.Zero("trades")) |
above the split |
| Scale a column between bounds you know | .ScaleGiven("fare", 0, 512) |
above the split |
| Take out the rows with a gap | .DropGaps("close") |
above the split |
| Take out the warm-up rows | .DropWarmUp() |
above the split |
| Draw the line | .SplitByTime("timestamp", train: 0.70, validation: 0.15) |
the split |
| Name the answer | .Target("close") |
right below the split |
| Fill a gap from the training rows | .FillMissing("age", FillStrategy.Median) |
below the split |
| Normalise from the training rows | .Normalise("close", "turnover") |
below the split |
The rest of this page explains why the line falls where it does, what the artefact holds, and what it leaves out.
Any piece of machine-learning work runs the same course:
- Collect the data.
- Deal with the gaps: the missing values, the ones that are missing for a reason, the ones that are not.
- Add the features — the columns that are derived rather than measured.
- Normalise, so that a value in euros and a value in seconds carry comparable weight.
- Split into training, validation and test — and, when you want it, a part held back to predict on.
- Build the model and train it.
- Check it against data it has never seen.
The gaps come before the features because a feature is derived from the columns as they are. A sum of two columns one of which has a gap is a gap, and a feature worked out from a value nobody measured is worse than one that is plainly absent.
Nobody argues with that list. The trouble is never the list — it is that most projects perform it rather than declare it. The steps live in a notebook, in a script, in a service, and in someone's head, and the four copies stop agreeing quietly.
The library writes that course, and the one thing it insists on is where the line falls:
- A value learned from the data — a mean, a median, the categories an encoder found, the bounds of a clip — is learned from the training rows alone, so the step that learns it stands below the split.
- A value nobody learned — a nought, a number you chose, a refusal — may stand wherever you like. Settling a gap
that way is
settle.gaps, above the split, where the features are worked out. So is scaling a column between bounds you give rather than bounds read from the rows,scale.given.
One column takes one of the two answers: settle it up there, or fill it down below. The pipeline refuses the pair rather than run a fill that has nothing left to find.
using DeepSharp.Pipelines;
var course = Pdd.Create()
.ReadCsv("prices.csv")
.Declare(schema => schema.Timestamp("timestamp").Number("close").Optional("trades", ColumnKind.Number))
.OrderBy("timestamp")
.SettleGaps(gaps => gaps.Zero("trades")) // a value no row decided, so it may stand here
.AddFeature("turnover", "close", Arithmetic.Times, "trades") // and the feature is worked out from settled columns
.SplitByTime("timestamp", train: 0.70, validation: 0.15)
.Target("close") // the answer, right below the line
.Normalise("close", "turnover", "trades") // what the training rows decide, below the line
.Build();These are the two halves of the same work, with the split between them.
Leave a gap unsettled and the feature worked out from it is a gap too, and filling the column afterwards does not reach back into it. The handover says so when it meets one, naming the columns the feature came from.
DropGaps and DropWarmUp stand up there as well, and take the rows out instead of settling them.
The whole course is declared in advance as one artefact, and then replayed.
That artefact is the pipeline. It is not a log of what happened; it is the thing that makes it happen. It can be handed to somebody, saved beside a model, and run again in a year on data that did not exist yet.
The difference between a pipeline and a script is what you can do with it afterwards:
- A script does the steps once, for whoever ran it.
- A pipeline is still there when the model comes back from production behaving oddly, and it can tell you exactly what the model was fed.
Anything that learns from the data is fitted on the training split alone, and then replayed unchanged.
Several of those steps look like plain arithmetic and are in fact learning:
| Step | What it learns | Where the number must come from |
|---|---|---|
| Normalise | a mean and a spread per column | the training rows only |
| Fill a gap | the value that fills it | the training rows only |
| Encode a category | which categories exist, and their order | the training rows only |
| Clip an outlier | the bounds | the training rows only |
| Select features | which columns survive | the training rows only |
Fitted on everything. Fit those steps on everything and the validation set has quietly taught the model about itself. The model then scores beautifully in validation and disappoints on the day it meets real data — and nothing anywhere went red, which is what makes this the most expensive mistake in the field.
Computed two ways. The same fault has a second form, at the other end. A feature computed one way while training and another way in the live service produces a model that was never wrong, fed numbers it was never trained on.
A declared pipeline removes both at once, because training and serving run the same artefact rather than two implementations that are meant to agree.
- The ordered steps, exactly as declared.
- What each step learned while being fitted — the means, the fill values, the category lists.
- How the split was made, so the same rows land in the same places when it is rerun — by what each row says rather than where it stands, so the same rows land in the same parts even when the file lists them in another order.
It is saved beside the model, and a model without it is not usable: the numbers reaching it would not be the numbers it was trained on.
A run that finishes and leaves you to decide what to look at has not finished. So the pipeline also declares what a run has to produce as proof: which numbers are computed, and whether they are shown as a table or drawn.
- The measures. Root-mean-square error, mean absolute error, R² for a regression; accuracy, precision, recall and the confusion matrix for a classification — named in the declaration, not chosen afterwards by whoever is looking.
- Where they are measured. On the training split, on validation, on test. The same measure on the three splits side by side is the single most useful thing a run can tell you: a low error on training and a high one on test is overfitting, and you see it in one glance rather than deduce it.
- How they are shown. A grid of numbers when they are to be compared or pasted into a report; a chart when the shape is the point — a loss curve over the epochs, predicted against actual, the residuals, a confusion matrix. Both, when both help.
- Every run produces the same evidence, so two runs are comparable without anybody remembering what was shown last time.
- The report cannot quietly shrink to whatever happened to look good — the measures were named before the numbers existed.
- About the data. A profile of the columns, which names what should not be there and how each is answered, and the rows a correlation between columns is drawn from. Both are measured on the training rows, so the rows a model will be measured on do not shape what it is shown.
- About a trained model. Its report: the measures named in the declaration, taken once the model has predicted, each in the answer's own units and beside what predicting the training rows' average would score.
The drawing itself is done by MatPlotLibNet, in a package of its own, DeepSharp.Charts, so a trainer on a machine
with no screen carries no renderer. It draws the loss curve, predicted against actual, the residuals, a confusion matrix
and the measures as bars, and the Notebook draws with it.
What the pipeline holds is the declaration: these measures, on these splits, in this form.
The datasets in between. Naming and storing every state between steps multiplies the things that can drift apart and answers a question nobody asks — and a replayable pipeline produces any of those states on demand, the same way every time. Keep the recipe, not the leftovers.
The pipeline came before gradients and layers, because everything after it is written in its terms — and because the leak it prevents is invisible once it has happened.
The data half. The data half is there end to end:
- reading a file — a comma-separated one, Parquet, Excel or JSON, each column's kind proposed from its cells, or one that Live sources landed from an exchange;
- handing prepared numbers and the answers a model is asked for over, run for what the learner needs and saying which steps it left out;
- the same rules for every way of writing it;
- a Notebook to see the data at any step and decide its columns.
A course to start from. A pipeline written from nothing can start from a course instead — every step in the order the steps belong, each waiting for what only you know: Starting from a course.
The model. The pipeline closes the course, because the model is declared with the pipeline rather than written beside it. Below the output, the declaration names the network these rows train — its layers, what moves them, what judges them, when the run stops and the engine it runs on — and running the pipeline trains exactly that network, on whichever engine you hand it.
That network learns from the training rows alone, is judged by the validation rows, is measured by the report, and is kept with its pipeline as one file, since a model without its pipeline cannot be used — see Networks. A network written as code still stands behind a pipeline the same way, and so does a model trained elsewhere once it is fitted there.
The Roadmap says what is done, what is still to come, and what is left out on purpose.
Pipeline is the other half of this page: the verbs themselves, in the order you write them, from reading a file to declaring what the run has to prove.