Socrates: Structured Questioning Unlocks Latent Knowledge in AI Research Agents

Pair a tool-using research agent with a question-only advisor that can never give answers, never issue directives, and has no tools of its own. The advisor must approve every plan via [APPROVED] before the Scientist runs the next experiment. On five MLE-bench Kaggle competitions this lifts test scores by an average of +55.9% over the same agent running alone.

Left: Socrates asks questions only and is stateful across sessions; the Scientist is stateless, executes code, and reads/writes the shared environment. Right: Statoil example — Socrates asks whether incremental tuning is closing the gap, the Scientist pivots to domain features (+10.2%); the Baseline PI offers generic encouragement and the Scientist stays on pixel statistics (+1.6%).

Note

The asciinema badge above is a placeholder. To record your own: bash scripts/record_demo.sh, then asciinema upload and paste the returned cast ID into this README in place of YOUR_CAST_ID (two occurrences).

---

Table of contents

• Quick start
• Repository layout
• The two scaffolds
• The three conditions
• Reproducing the paper results
• Configuration reference
• Running tests
• Citation
• License

---

Quick start

Tested on Python 3.10–3.12, Linux/macOS. GPU is optional (only required for tasks that train deep models — Statoil and NFL benefit, the others run fine on CPU).

# 1. Clone the repo
git clone https://github.com/hexo-ai/socrates.git
cd socrates

# 2. Create an isolated environment (conda or venv — pick one)
conda create -n socrates python=3.11 -y
conda activate socrates
#   or
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt
pip install --no-deps -r socratic-evolve/public-repo/requirements_base.txt
pip install --no-deps -r socratic-evolve/public-repo/requirements_ml.txt
pip install --no-deps -r socratic-evolve/public-repo/requirements_domain.txt

# 4. Set API keys
export ANTHROPIC_API_KEY="sk-ant-..."        # required
export OPENAI_API_KEY="sk-..."               # optional, only if you use OpenAI models

# 5. Create a local test config (gitignored)
cp socratic-evolve/test_config.yaml.example socratic-evolve/test_config.yaml
cp discover/test_config.yaml.example          discover/test_config.yaml
# Edit each to set dataset_dir and model.

# 6. Smoke-test the sequential scaffold
python discover/test_agent_locally.py

If step 6 prints a Socrates question and an [APPROVED] from a short discussion loop, the install is good.

---

Repository layout

socrates/
├── discover/                 # Sequential scaffold (single agent, one experiment at a time)
│   ├── custom_agent.py       # Agent implementation
│   ├── base_agent.py         # Base class with webhook protocol
│   ├── models.py             # Message models
│   └── test_agent_locally.py # Local smoke test
│
├── socratic-evolve/          # Evolutionary scaffold (MLevolve + MCGS tree search)
│   ├── custom_agent.py       # Agent wrapper
│   ├── base_agent.py         # Base class
│   ├── models.py             # Message models
│   └── public-repo/          # MLevolve core
│       ├── run.py            # Main entry point for full experiments
│       ├── config/           # Default configuration
│       ├── engine/           # MCGS tree search, code execution
│       ├── agents/           # Multi-agent subsystem
│       │   ├── socrates/     # Socrates PI implementation
│       │   │   ├── prompts.py        # Question-only system prompt + [APPROVED] gate
│       │   │   ├── approval_loop.py  # Multi-round discussion loop
│       │   │   └── config.py         # Toggle flags
│       │   ├── evolution_agent.py    # Paradigm-shift mutations
│       │   └── fusion_agent.py       # Cross-branch solution merging
│       └── llm/              # LLM client wrappers
│
├── assets/
│   └── protocol.png          # Protocol diagram
├── scripts/
│   └── record_demo.sh        # Records the asciinema demo cast
├── conda.sh                  # Quick env activation helper
├── requirements.txt          # Top-level dependency manifest
├── LICENSE                   # MIT
└── README.md                 # This file

---

The two scaffolds

Sequential (discover/)

A single agent writes and executes experiments one at a time. No built-in exploration mechanism. The Scientist retains tool access during Socratic review, so when Socrates asks "how many features have zero importance?" the Scientist runs the analysis right then. Best when per-step quality matters more than raw experiment volume.

Evolutionary (socratic-evolve/)

An evolutionary code-generation system (MLevolve) maintaining a tree of candidate solutions across parallel branches. Includes evolution stages (paradigm-shift mutations), fusion stages (cross-branch solution merging), and runs multiple branches in parallel. During review, the Scientist can only revise plan text (no tool access). Best when the search space rewards high iteration volume.

---

The three conditions

All controlled via configuration flags (use_socrates_review and use_baseline_pi in config.yaml / config.py):

Condition	Flags	Behavior
Scientist-only	use_socrates_review=False, use_baseline_pi=False	Single agent, no supervision.
Baseline PI	use_socrates_review=False, use_baseline_pi=True	Second agent giving generic encouragement (control condition).
Socrates	use_socrates_review=True	Full protocol: question-only PI, [APPROVED] gate.

---

Reproducing the paper results

We evaluate on five tasks from MLE-bench:

Task	Metric	Notes
Statoil Iceberg	Log Loss ↓	Radar imagery
Stanford COVID Vaccine	MCRMSE ↓	RNA degradation
Ventilator Pressure	MAE ↓	Tabular time-series
NFL Contact Detection	MCC ↑	Player tracking + video
Smartphone Decimeter	Haversine ↓	GPS positioning

1. Get the datasets

Follow the MLE-bench instructions to download the five competition datasets. Place each one under a local directory and remember its path — you'll plug it into the config in the next step.

2. Run the sequential scaffold

cd discover/
# Edit test_config.yaml to set:
#   AGENT_CONFIG.exp_id        -> the MLE-bench task id (e.g. "statoil-iceberg-classifier-challenge")
#   AGENT_CONFIG.dataset_dir   -> the local path you put the data in
#   AGENT_CONFIG.model         -> the LLM (default: claude-opus-4-6)
python test_agent_locally.py

This writes per-experiment folders, a best_score.txt, and a submission.csv in dataset_dir. Submit submission.csv to Kaggle to get the test score.

3. Run the evolutionary scaffold

cd socratic-evolve/public-repo/
python run.py \
  exp_id="statoil-iceberg-classifier-challenge" \
  agent.use_socrates_review=True \
  agent.steps=50

For each task, run it once per condition (toggling the flags above) so you can compare Scientist-only / Baseline PI / Socrates side by side.

4. Collecting and plotting

cd socratic-evolve/public-repo/
python collect_and_plot.py   # aggregates per-experiment logs into the paper's tables/plots
python dashboard.py          # optional live dashboard

Expected results

Task	Scientist-only (test)	Baseline PI (test)	Socrates (test)	Δ vs Scientist
Statoil	0.255	0.251	0.229	+10.5%
COVID	0.389	0.308	0.294	+24.4%
Ventilator	1.534	0.815	0.853	+44.4%
NFL	0.198	0.537	0.584	+195.4%
Smartphone	6.285	5.993	5.984	+4.8%

Note: LLM agents are high-variance run-to-run. We saw a standard deviation of ~15% of the mean across 10 Scientist-only seeds on Smartphone. Expect single-seed numbers to vary; the direction of the effect (Socrates ≥ Baseline PI > Scientist-only) is the reproducible claim.

---

Configuration reference

The key flags live in socratic-evolve/public-repo/config/config.yaml and discover/test_config.yaml:

Flag	Default	Meaning
agent.use_socrates_review	false	Enable the full Socrates question-only protocol.
agent.use_baseline_pi	false	Enable the generic-encouragement control condition.
agent.steps	50 (evolve) / 30 (seq)	Total experiment budget.
agent.K	3	Max discussion rounds before forced approval.
agent.model	claude-opus-4-6	Scientist LLM.
agent.feedback_model	(same as model)	Socrates LLM (can differ from the Scientist).
agent.respect_finished	true	Whether the agent may stop early via [FINISHED].
agent.enforce_gpu_usage	false	Inject the GPU-required block into the system prompt.

A more detailed flag-level reference for the prompts (which blocks get injected when) is in socratic-evolve/public-repo/agents/socrates/.

---

Running tests

# Sequential scaffold smoke test (no real run; mocks the LLM):
python discover/test_agent_locally.py --dry-run

# Evolutionary scaffold live test (requires API key):
cd socratic-evolve/public-repo/
pytest tests/test_socrates_live.py -k "test_socrates_basic"

---

Citation

@inproceedings{vrabac2026socrates,
  title     = {Socrates: Structured Questioning Unlocks Latent Knowledge in AI Research Agents},
  author    = {Vrabac, Damir and Hebbar, Prannay and Manawat, Yogendra and Palanimalai, Selvam and Verboomen, Samuel and Juneja, Gurusha and Bhatia, Kunal and Baskaran, Vignesh},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}

---

License

MIT. See LICENSE.