This repository builds on Self-SoftCoT: A Self-Consistent Framework via
Position-Aware Latent Space Reinforcement Learning (Dong, Shan & Li, ACL 2026;
paper/2026.acl-long.1496.pdf). It
re-implements the method as one composable package and adds the four ideas
of improvement.md, so that every idea, and every
combination of them, can be trained and evaluated under the same protocol as
the paper's baseline:
| Idea | Method key | Self-SoftCoT limitation it targets |
|---|---|---|
| 1 · DyLaR | dylar |
static budget of N = 2 soft tokens → entropy-gated N(q) ∈ {1, 2, 4, 8} |
| 2 · Deep-SteerCoT | deep |
re-injection only at the input embeddings → projections into the keys/values of layers L/2…L |
| 3 · PRM-SoftCoT | prm |
sparse binary outcome reward → reward blended with a process reward model |
| 4 · MoP-SoftCoT | mop |
one projector per task → mixture of projector experts with a router, trained on a task mixture |
| 5 · combinations | e.g. dylar+deep |
any “+”-joined set of the four |
The original code of the paper is kept untouched in upstream/.
cd ~/Self-Soft
./self_beat.sh --plan # what the SELECTION expands to; writes nothing
sbatch self_beat.sh --smoke # every method end-to-end on Qwen2.5-0.5B (~1 h), plus the GPU tests
sbatch self_beat.sh # the real experiments; runs indefinitely, re-submits itself every 24 h
./reporting.sh # refresh the report by hand (the heart beat does it every hour)self_beat.sh is the heart beat: a CPU-only Slurm job that every hour builds
whatever is missing (Python env, data, model downloads), trains every method
the SELECTION block at its top asks for, evaluates the trained heads,
resubmits jobs that died (training and evaluation both resume where they
stopped), merges finished runs and rewrites the report. Choose what runs by
editing that block — methods, combinations, models, datasets, seeds, the
sweep phase and every hyper-parameter (with the grid of improvement.md next
to its default). The running heart beat notices the edit and hands over to a
fresh job.
Storage. Everything the experiments produce — the virtualenv, model
weights, caches, checkpoints, logs, per-problem results — goes to Lustre,
/lustre/scratch/data/s76kalav_hpc-memory/self-soft/. The only generated file
in the repository is the report, result/selfsoft_report.md.
self_beat.sh the heart beat (SELECTION + hyper-parameters at the top)
reporting.sh writes result/selfsoft_report.md
selfsoft/ the implementation
config.py every knob (CLI + checkpoint contents)
tasks.py data files, prompt templates, answer grading (verbatim upstream logic)
model.py frozen backbone, thinking pass, KV steering, the trainable head
rewards.py process reward models (Qwen2.5-Math-PRM-7B, Math-Shepherd)
train.py GSPO trainer, Config A/B, all ideas; resumable python -m selfsoft.train
evaluate.py sharded, resumable evaluation python -m selfsoft.evaluate
scripts/
ss_data.py prepares all benchmarks on Lustre
ss_tools.py merge + status (stdlib only; used by the heart beat)
ss_report.py the report (stdlib only)
tests/test_selfsoft.py GPU tests, incl. token-exact comparison with the upstream code
docs/ the documentation (zensical.toml)
upstream/ the paper's original code (buffer = Config B, Streaming = Config A) and README
paper/ the paper
improvement.md the research blueprint
result/ the report
Start with docs/index.md. Rendered site:
pip install zensical && zensical serve in the repository root. Key pages:
- Running the heart beat and the SELECTION options
- Every hyper-parameter · compute budget
- The methods: baselines, DyLaR, Deep-SteerCoT, PRM-SoftCoT, MoP-SoftCoT, combinations
- What the upstream code really does — details that decide numbers
- What is saved where · the report
@inproceedings{dong2026selfsoftcot,
title = {Self-SoftCoT: A Self-Consistent Framework via Position-Aware Latent Space Reinforcement Learning},
author = {Dong, Liangliang and Shan, Lianlei and Li, Shuaimin},
booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics},
pages = {32393--32414},
year = {2026}
}