How to download data?

[lucasjinreal]

How to download data?

[Mapika]

There is no single download of the training data. The repository contains the builders, and they download the sources:

uv venv --python 3.12 .venv312 && uv pip install -p .venv312/bin/python -e ".[serve,train]"
.venv312/bin/python -m decider.data.core --out data/tasks.pkl        # downloads and converts about 95 public datasets
.venv312/bin/python -m decider.data.mixture --base data/tasks.pkl --mode full --out data/mixture_full.pkl --probes data/probes.pkl

The data labelled by our local 27B teacher is in the repository itself, in teacher_data/ (22 MB of JSONL). decider/data/mixture.py lists every component of the mixture with its size. scripts/train.sh full runs these steps and then the training run.

The public builders reproduce the mixture of decider-2b. decider-4b was trained on a larger mixture (26 more public datasets and ten programmatic families), and those builders are not in the package yet.

If you meant the evaluation data, or the weights, say which and I will point you to it.