DataForge 2026 · Pathway Track submission
An interactive explainer for one idea: in-context memory can live in a fixed-size synapse matrix that is rewritten as the model reads, instead of in a KV cache that grows with the context. The learner writes associations into a real matrix, reads them back, and watches recall degrade through interference rather than eviction.
- Live artifact: https://claude.ai/code/artifact/510de275-a8d7-4fc3-9bd7-b5edce282bc0 (opens without sign-in)
- Source repository: https://github.com/tanmaysahare/dataforge
- One-page concept summary (PDF):
concept-summary.pdfin this repo - Selected approved topic: Synaptic Plasticity as Short-Term Memory (Memory and learning), touching Associative Memory and Fast Weights.
In-context memory can be held in a fixed-size synapse matrix updated by a Hebbian outer-product write while reading, so per-token recall cost stops growing with context length; the price is that memories are not evicted but superposed, and recall degrades smoothly through interference once the number of stored associations approaches the matrix's capacity.
The claim is falsifiable inside the artifact. If a fixed-size state gave free unbounded memory, the recall-versus-load curve would stay flat. It does not, and the learner can find the crossing point in under a minute.
Audience: someone who understands Transformer attention and a KV cache, and can read matrix-vector notation. Undergraduate ML or a working data scientist. No neuroscience background and no prior BDH knowledge is required.
Prerequisites: Q/K/V attention; dot products and outer products; what a KV cache is and why it grows.
Time to complete: about 12 minutes, including the sixty-second test at the end.
After using the artifact, a learner can:
- Write and read an associative memory as a single matrix, and state the read equation
as
signal + crosstalk. - Predict, before touching a slider, the load at which recall fails for a given sparsity, and check the prediction against the live curve.
- State precisely what Dragon Hatchling (BDH) changes relative to a Transformer, and what BDH-CQ adds on top of BDH.
- Name at least one limitation of the fixed-state idea and one limitation of this artifact's analogy.
Single-file, dependency-free HTML page. No build step, no network calls at runtime, no sign-in. Roughly 200 lines of vanilla JavaScript drive all computation.
| Component | What it does | Status |
|---|---|---|
| Section 1, KV-cache calculator | Computes 2 · layers · tokens · d_model · 2 bytes against d_model² · 2 bytes |
Live, exact arithmetic, formula printed on the page |
| Section 2, associative-memory substrate | Generates sparse non-negative keys and Gaussian values, writes σ ← λσ + v xᵀ, reads σx / ‖x‖² |
Live, computed in-browser on every slider move |
| Recalled-vs-truth bar chart | Ground truth beside model output, same axes | Live |
| Recall-versus-load curve | Re-writes prefixes of the sequence and measures mean cosine at ~24 loads | Live |
| Top-1 recall stat | Re-reads every stored pair and counts how many return themselves | Live |
| Verdict banner | Classifies the current regime: clean / degrading / retrieval failure | Live, derived from the measured numbers |
| Section 3, BDH module | Equations, mapping table, faithful-vs-not-faithful list | Literature, from primary sources, labelled as reported evidence |
| Section 5, sixty-second test | Three questions with explanations | Static, but each answer refers to something the learner just measured |
Nothing on this page is a pre-rendered animation. There are no recorded runs and no scripted motion. Every chart is redrawn from freshly computed numbers.
Every control maps to exactly one variable in the model. There are no decorative sliders.
| Control | Model variable |
|---|---|
Neurons n |
Dimension of the key space; state is D × n in code, conceptually n × n |
| Active fraction | Number of non-zero entries per key, k = round(n · fraction) |
| Associations written | Number of outer-product writes, i.e. the context load |
Decay λ |
Multiplicative decay applied to σ before each write |
| Probe | Which stored association is read back |
Context length / d_model (Section 1) |
Inputs to the KV-cache byte formula |
BDH is connected in Section 3, and the connection is structural rather than decorative: the substrate the learner manipulates is the mechanism BDH reformulates attention into. The section states plainly which system is meant (BDH and BDH-GPU, then BDH-CQ), what is changing inside it (the fast synapse state, not the trained weights), which properties are faithful in our toy, and which are not.
The toy model in this artifact is an independent reimplementation of a generic Hebbian associative memory, written by us for teaching. It is not an official BDH or BDH-CQ model, uses no BDH weights, and must not be presented as one. BDH-specific results quoted in the text (≈5% activation sparsity, monosemantic synapses, heavy-tailed connectivity, BDH-CQ's ARC-AGI and effort-scaling behaviour) are cited as developer-reported results from primary sources, not as things this page demonstrates.
git clone https://github.com/tanmaysahare/dataforge
cd pathway-submission
python -m http.server 8000 # or just open index.html in a browser
# visit http://localhost:8000There are no dependencies, no package manager and no build. The page works offline.
Reproducing a specific figure. The random draw is seeded (seed = 12345), so a fresh
page load produces identical numbers for identical slider settings. The "New random draw"
button reseeds from the clock; reload the page to return to the deterministic seed.
Reproducing the headline result in under a minute: open the page, press Break it: 150 pairs, 20% dense (retrieval fails, top-1 points at the wrong association), then press Fix it with sparsity: 5% active (same load, same state size, recall returns). That contrast is the claim.
nis capped at 768 so a full read stays inside one animation frame.- Keys are random sparse non-negative vectors, not learned embeddings. Real learned keys are correlated, which makes interference worse than shown here, not better. The artifact therefore gives an optimistic picture of capacity.
- Values are dense 16-dimensional Gaussian vectors, chosen for chart legibility.
- The substrate has one layer and one state; BDH is a deep stack with a specific ReLU low-rank parameterisation.
- The decay
λis a scalar teaching knob. BDH is not "linear attention plus a decay slider", and BDH-GPU should not be classified as a Mamba-style SSM. - The KV-cache calculator assumes 24 layers and fp16 and prints its formula. It is a memory-footprint argument only, not an accuracy claim about any model.
At least three primary papers from 2022–2026 that use, extend, test or rely on the
concept, cited beside the claims they support in CITATIONS.md and on the page itself:
- Kosowski et al., The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain, Pathway, 2025, arXiv:2509.26507.
- Pathway, BDH-CQ technical report, 2025/2026.
- Schlag, Irie, Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021 (foundational for the write rule implemented here).
- Gu and Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023.
- Yang et al., Gated Linear Attention Transformers with Hardware-Efficient Training, ICML 2024, and Parallelizing Linear Transformers with the Delta Rule, NeurIPS 2024.
- Behrouz et al., Titans: Learning to Memorize at Test Time, 2024.
See CITATIONS.md for which claim each source supports and the evidence level of each.
- Code and text in this repository: MIT Licence, see
LICENSE. - No third-party code, model weights, datasets, fonts or images are bundled. Typography uses system font stacks only. All graphics are inline SVG generated by our own code.
- AI assistance, data, asset and licence disclosure:
DISCLOSURE.md. - Team: Abhiyanta, Tanmay Sahare, Anuj Kumar Gupta · mentorship: none.
Every claim, sentence and citation in this submission can be explained and defended by the registered team.