Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,25 @@ jobs:
run: python -m pip install -r requirements-dev.txt
- name: Validate catalog and generated files
run: make check

website:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22"
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install website dependencies
working-directory: website
run: npm ci
- name: Lint website
working-directory: website
run: npm run lint
- name: Build and test website
working-directory: website
run: npm test
- name: Audit production dependencies
working-directory: website
run: npm audit --omit=dev --audit-level=high
5 changes: 4 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Source of truth

- Edit `data/papers.yaml`; do not hand-edit generated catalog tables, `data/papers.json`, `papers/*.md`, or `views/*.md`.
- Edit `data/papers.yaml`; do not hand-edit generated catalog tables, `data/papers.json`, `website/data/catalog.json`, `papers/*.md`, or `views/*.md`.
- Update `data/schema.json` and `docs/taxonomy.md` together when adding fields, products, statuses, or method tags.
- A product mention means the complete agent or harness, not merely a model from the same vendor.

Expand All @@ -20,6 +20,9 @@
python -m pip install -r requirements-dev.txt
make build
make check
npm --prefix website install
make site-check
```

Run `make build` after catalog edits and include all generated changes in the same commit.
The website reads the generated `website/data/catalog.json`; never maintain a second paper list in the frontend.
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ All notable catalog and schema changes are recorded here.

## [Unreleased]

### Added

- Interactive bilingual research catalog with full-text search and product, evidence, fairness, venue, and method filters
- Expandable paper evidence dossiers and recurring-method summaries
- Host-aware social sharing metadata and a dedicated Open Graph card
- Website build, server-rendering, catalog-parity, lint, and dependency-audit checks in CI

### Planned

- Deeper audits of product versions, model snapshots, and execution budgets
Expand Down
7 changes: 4 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@ same_budget: unknown
3. Record authors, identifiers, system name, artifact status, and publication date from primary sources.
4. Choose the narrowest correct evidence class and comparison scope.
5. Record where the supporting result appears in the paper.
6. Run `make build` to regenerate READMEs, JSON, views, and paper dossiers.
7. Run `make check` before opening a pull request.
6. Run `make build` to regenerate READMEs, both JSON exports, views, and paper dossiers.
7. Run `make check` and `make site-check` before opening a pull request.

Do not manually edit content between `CATALOG:*:START` and `CATALOG:*:END` markers in either README. Files under `papers/` and `views/`, plus `data/papers.json`, are also generated.
Do not manually edit content between `CATALOG:*:START` and `CATALOG:*:END` markers in either README. Files under `papers/` and `views/`, plus `data/papers.json` and `website/data/catalog.json`, are also generated.

## Required source checks

Expand Down Expand Up @@ -73,4 +73,5 @@ Future products must be added to both `data/schema.json` and `docs/taxonomy.md`
- [ ] The evidence location and comparison scope are recorded.
- [ ] Comparison caveats are explicit.
- [ ] `make check` passes.
- [ ] `make site-check` passes when the website or catalog changed.
- [ ] No copyrighted PDF has been committed.
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: install build validate test lint check
.PHONY: install build validate test lint check site-check

install:
python -m pip install -r requirements-dev.txt
Expand All @@ -22,3 +22,8 @@ check: validate
python -m pytest
python -m ruff check scripts tests
python -m ruff format --check scripts tests

site-check:
npm --prefix website run lint
npm --prefix website test
npm --prefix website audit --omit=dev --audit-level=high
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
</p>
<!-- CATALOG:STATS:END -->

<p align="center">
<a href="https://awesome-claude-code-codex-papers.osako6947.chatgpt.site"><strong>Open the interactive research catalog →</strong></a>
</p>

This repository is an evidence-first catalog of papers that study **Claude Code** and **Codex CLI** as complete products—not merely Claude or GPT-family models. It records the baseline configuration, task, intervention, reported result, comparison controls, caveats, and official artifacts.

> [!IMPORTANT]
Expand Down Expand Up @@ -81,6 +85,9 @@ Excluded from the primary catalog:

These papers numerically compare a proposed method with Claude Code or Codex CLI. “Direct” does not imply same-model or same-budget parity—check the controls column and linked dossier.

<details>
<summary><strong>Open the raw direct-comparison table</strong></summary>

<!-- CATALOG:DIRECT:START -->
| System / paper | Venue | Product baseline | Task | What changed | Reported evidence | Controls |
|---|---|---|---|---|---|---|
Expand All @@ -99,6 +106,8 @@ These papers numerically compare a proposed method with Claude Code or Codex CLI
| **[MLZero](papers/mlzero-2025.md)**<br>[paper](https://papers.neurips.cc/paper_files/paper/2025/hash/63ed15a46a143ff57484b38cd6b85d91-Abstract-Conference.html) · [artifact](https://github.com/autogluon/autogluon-assistant) | NeurIPS 2025<br>main | Codex CLI (GPT-4.1 and o4-mini) | End-to-end machine learning automation | Hierarchical agents with multimodal perception, semantic and episodic memory, and a planning-coding-evaluation loop<br>`multi-agent` `memory` `verifier-loop` `test-feedback` | MLZero led MLE-Bench Lite with six gold medals and achieved 0.92 success on its multimodal benchmark; one appendix setting reports 6.5% errors versus 26.9% for Codex CLI. | model: unknown<br>budget: unknown<br>evidence: medium |
<!-- CATALOG:DIRECT:END -->

</details>

## What repeatedly works

Across the current catalog, successful methods repeatedly add structure around the model:
Expand All @@ -116,17 +125,25 @@ See the generated [method view](views/by-method.md) and [taxonomy](docs/taxonomy

These papers contain useful product-level evidence, but their central numeric result is a component comparison or another agent's result rather than a clean Claude Code/Codex CLI head-to-head.

<details>
<summary><strong>Open the raw related-method table</strong></summary>

<!-- CATALOG:RELATED:START -->
| System / paper | Venue | Product baseline | Task | What changed | Reported evidence | Controls |
|---|---|---|---|---|---|---|
| **[CodeGrep](papers/codegrep-2026.md)**<br>[paper](https://arxiv.org/abs/2608.05886) | arXiv 2026<br>preprint | Claude Code (Claude-Code-like agent) | Repository retrieval for issue resolution | GRPO-trained 14B retrieval agent that issues multi-turn parallel grep, glob, and read operations for a frozen downstream coding agent<br>`retrieval` `context-reduction` `parallelism` | Resolve rate 27.0% versus 25.8% without retrieval; resolved tasks use 15% fewer rounds and 19% fewer tokens. | model: same<br>budget: unknown<br>evidence: contextual |
| **[SCATE](papers/scate-2026.md)**<br>[paper](https://arxiv.org/abs/2607.08983) | arXiv 2026<br>preprint | Claude Code | Automated test generation | Contextual-bandit supervisor that selects testing actions from current coverage and testability state<br>`test-feedback` `verifier-loop` | Line coverage improves 32.3% and branch coverage 30.9% over agent-only Gemini CLI; adaptation is also evaluated with Claude Code. | model: unknown<br>budget: unknown<br>evidence: contextual |
<!-- CATALOG:RELATED:END -->

</details>

## Evaluation-only papers

These papers benchmark or diagnose the products without proposing a method intended to outperform them.

<details>
<summary><strong>Open the raw evaluation table</strong></summary>

<!-- CATALOG:EVALUATION:START -->
| System / paper | Venue | Product baseline | Task | What changed | Reported evidence | Controls |
|---|---|---|---|---|---|---|
Expand All @@ -135,16 +152,23 @@ These papers benchmark or diagnose the products without proposing a method inten
| **[BountyBench](papers/bountybench-2025.md)**<br>[paper](https://proceedings.neurips.cc/paper_files/paper/2025/hash/faed4276b52ef762879db4142655c699-Abstract-Datasets_and_Benchmarks_Track.html) · [artifact](https://github.com/bountybench/bountybench) | NeurIPS 2025<br>dataset benchmark | Claude Code<br>Codex CLI (o3-high and o4-mini) | Vulnerability detection, exploitation, and patching | Evaluation-only benchmark of offensive and defensive cybersecurity agents<br>`verifier-loop` | Codex o3-high detects 12.5% and patches 90%; Claude Code patches 87.5% and exploits 57.5% in the reported settings. | model: unknown<br>budget: unknown<br>evidence: high |
<!-- CATALOG:EVALUATION:END -->

</details>

## Historical OpenAI Codex model papers

These entries study the pre-CLI OpenAI Codex model and are never mixed with modern Codex CLI results.

<details>
<summary><strong>Open the raw historical-model table</strong></summary>

<!-- CATALOG:HISTORICAL:START -->
| System / paper | Venue | Product baseline | Task | What changed | Reported evidence | Controls |
|---|---|---|---|---|---|---|
| **[TiCoder](papers/ticoder-2022.md)**<br>[paper](https://arxiv.org/abs/2208.05950) · [artifact](https://github.com/microsoft/TiCoder) | AST at ICSE 2022<br>workshop | OpenAI Codex model (OpenAI Codex) | Interactive function-level code generation | Generate tests to formalize user intent, collect lightweight feedback, and rank or prune candidate programs<br>`test-feedback` `verifier-loop` | With one to five simulated user queries, Codex pass@1 gains 22.49 to 37.71 points on MBPP and 24.79 to 53.98 points on HumanEval. | model: same<br>budget: different<br>evidence: high |
<!-- CATALOG:HISTORICAL:END -->

</details>

## Evidence classes

- **Direct comparison** — numeric comparison with the production product.
Expand Down
24 changes: 24 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
</p>
<!-- CATALOG:STATS:END -->

<p align="center">
<a href="https://awesome-claude-code-codex-papers.osako6947.chatgpt.site"><strong>打开可交互论文目录 →</strong></a>
</p>

这是一个以证据为核心的论文库,研究对象是完整的 **Claude Code** 与 **Codex CLI** 产品,而不只是 Claude 或 GPT 系列模型。每篇论文都记录 baseline 配置、任务、新增方法、论文结果、实验控制、风险说明和官方 artifact。

> [!IMPORTANT]
Expand Down Expand Up @@ -81,6 +85,9 @@

这些论文把提出的方法与 Claude Code 或 Codex CLI 做了数值比较。“直接”不代表模型和预算相同,请同时查看可比性字段和论文证据页。

<details>
<summary><strong>展开原始直接对比表</strong></summary>

<!-- CATALOG:DIRECT:START -->
| 系统 / 论文 | 会议 | 产品 baseline | 任务 | 新增方法 | 论文结果 | 可比性 |
|---|---|---|---|---|---|---|
Expand All @@ -99,6 +106,8 @@
| **[MLZero](papers/mlzero-2025.md)**<br>[paper](https://papers.neurips.cc/paper_files/paper/2025/hash/63ed15a46a143ff57484b38cd6b85d91-Abstract-Conference.html) · [artifact](https://github.com/autogluon/autogluon-assistant) | NeurIPS 2025<br>main | Codex CLI (GPT-4.1 and o4-mini) | End-to-end machine learning automation | Hierarchical agents with multimodal perception, semantic and episodic memory, and a planning-coding-evaluation loop<br>`multi-agent` `memory` `verifier-loop` `test-feedback` | MLZero led MLE-Bench Lite with six gold medals and achieved 0.92 success on its multimodal benchmark; one appendix setting reports 6.5% errors versus 26.9% for Codex CLI. | model: unknown<br>budget: unknown<br>evidence: medium |
<!-- CATALOG:DIRECT:END -->

</details>

## 反复出现的有效方法

1. **持久化仓库状态**:把计划、架构、依赖和设计知识放到上下文窗口之外。
Expand All @@ -114,17 +123,25 @@

这些论文包含有用的产品证据,但主要数值可能来自组件实验或其他 agent,不构成严格的 Claude Code/Codex CLI 正面对比。

<details>
<summary><strong>展开原始相关方法表</strong></summary>

<!-- CATALOG:RELATED:START -->
| 系统 / 论文 | 会议 | 产品 baseline | 任务 | 新增方法 | 论文结果 | 可比性 |
|---|---|---|---|---|---|---|
| **[CodeGrep](papers/codegrep-2026.md)**<br>[paper](https://arxiv.org/abs/2608.05886) | arXiv 2026<br>preprint | Claude Code (Claude-Code-like agent) | Repository retrieval for issue resolution | GRPO-trained 14B retrieval agent that issues multi-turn parallel grep, glob, and read operations for a frozen downstream coding agent<br>`retrieval` `context-reduction` `parallelism` | Resolve rate 27.0% versus 25.8% without retrieval; resolved tasks use 15% fewer rounds and 19% fewer tokens. | model: same<br>budget: unknown<br>evidence: contextual |
| **[SCATE](papers/scate-2026.md)**<br>[paper](https://arxiv.org/abs/2607.08983) | arXiv 2026<br>preprint | Claude Code | Automated test generation | Contextual-bandit supervisor that selects testing actions from current coverage and testability state<br>`test-feedback` `verifier-loop` | Line coverage improves 32.3% and branch coverage 30.9% over agent-only Gemini CLI; adaptation is also evaluated with Claude Code. | model: unknown<br>budget: unknown<br>evidence: contextual |
<!-- CATALOG:RELATED:END -->

</details>

## 仅评测论文

这些论文负责 benchmark、测量或故障分析,没有提出专门超越产品的新方法。

<details>
<summary><strong>展开原始评测论文表</strong></summary>

<!-- CATALOG:EVALUATION:START -->
| 系统 / 论文 | 会议 | 产品 baseline | 任务 | 新增方法 | 论文结果 | 可比性 |
|---|---|---|---|---|---|---|
Expand All @@ -133,16 +150,23 @@
| **[BountyBench](papers/bountybench-2025.md)**<br>[paper](https://proceedings.neurips.cc/paper_files/paper/2025/hash/faed4276b52ef762879db4142655c699-Abstract-Datasets_and_Benchmarks_Track.html) · [artifact](https://github.com/bountybench/bountybench) | NeurIPS 2025<br>dataset benchmark | Claude Code<br>Codex CLI (o3-high and o4-mini) | Vulnerability detection, exploitation, and patching | Evaluation-only benchmark of offensive and defensive cybersecurity agents<br>`verifier-loop` | Codex o3-high detects 12.5% and patches 90%; Claude Code patches 87.5% and exploits 57.5% in the reported settings. | model: unknown<br>budget: unknown<br>evidence: high |
<!-- CATALOG:EVALUATION:END -->

</details>

## 旧 OpenAI Codex 模型论文

这些论文研究的是 Codex CLI 出现前的 OpenAI Codex 模型,不与现代 Codex CLI 结果混合。

<details>
<summary><strong>展开原始历史模型表</strong></summary>

<!-- CATALOG:HISTORICAL:START -->
| 系统 / 论文 | 会议 | 产品 baseline | 任务 | 新增方法 | 论文结果 | 可比性 |
|---|---|---|---|---|---|---|
| **[TiCoder](papers/ticoder-2022.md)**<br>[paper](https://arxiv.org/abs/2208.05950) · [artifact](https://github.com/microsoft/TiCoder) | AST at ICSE 2022<br>workshop | OpenAI Codex model (OpenAI Codex) | Interactive function-level code generation | Generate tests to formalize user intent, collect lightweight feedback, and rank or prune candidate programs<br>`test-feedback` `verifier-loop` | With one to five simulated user queries, Codex pass@1 gains 22.49 to 37.71 points on MBPP and 24.79 to 53.98 points on HumanEval. | model: same<br>budget: different<br>evidence: high |
<!-- CATALOG:HISTORICAL:END -->

</details>

## 数据与自动校验

`data/papers.yaml` 是唯一数据源。构建流程会校验 Schema、生成中英文 README、导出 JSON、生成每篇论文的证据页和研究视图,并检查内部链接、执行测试与 lint:
Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ The catalog optimizes for evidence quality before paper count.
- [x] English and Simplified Chinese entry points
- [x] Schema, tests, linting, and CI
- [x] Structured paper-suggestion workflow
- [x] Interactive bilingual site generated from the same catalog
- [x] Matched-model and matched-budget comparison filters

## v0.2 — Evidence depth

Expand Down Expand Up @@ -45,7 +47,5 @@ Model-only papers remain out of scope unless they study a product harness direct

## Long-term

- Interactive static site generated from the same catalog
- Comparison matrices with matched-model and matched-budget filters
- Reproduction packs for a small set of high-impact claims
- Community-maintained venue watchlist and artifact-status audits
4 changes: 3 additions & 1 deletion scripts/build_readme.py
Original file line number Diff line number Diff line change
Expand Up @@ -519,10 +519,12 @@ def render_by_venue(catalog: dict) -> str:

def render_all() -> dict[Path, str]:
catalog = load_catalog()
catalog_json = json.dumps(catalog, indent=2, ensure_ascii=False) + "\n"
outputs = {
README_PATH: render_marked_readme(README_PATH, catalog, "en"),
README_ZH_PATH: render_marked_readme(README_ZH_PATH, catalog, "zh"),
ROOT / "data" / "papers.json": json.dumps(catalog, indent=2, ensure_ascii=False) + "\n",
ROOT / "data" / "papers.json": catalog_json,
ROOT / "website" / "data" / "catalog.json": catalog_json,
ROOT / "papers" / "README.md": render_paper_index(catalog),
ROOT / "views" / "README.md": render_views_index(),
ROOT / "views" / "by-product.md": render_by_product(catalog),
Expand Down
12 changes: 11 additions & 1 deletion scripts/check_links.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,17 @@
ROOT = Path(__file__).resolve().parents[1]
MARKDOWN_LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)")
HTML_LINK = re.compile(r'(?:href|src)="([^"]+)"')
EXCLUDED_PARTS = {".git", ".venv"}
EXCLUDED_PARTS = {
".git",
".next",
".venv",
".vinext",
".wrangler",
"coverage",
"dist",
"node_modules",
"out",
}


def markdown_files() -> list[Path]:
Expand Down
13 changes: 13 additions & 0 deletions tests/test_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from pathlib import Path
from xml.etree import ElementTree

from scripts import check_links
from scripts.discover_candidates import parse_feed

ROOT = Path(__file__).resolve().parents[1]
Expand All @@ -30,6 +31,18 @@ def test_generated_readme_is_current() -> None:
assert result.returncode == 0, result.stderr


def test_link_scan_excludes_dependencies_and_build_outputs(tmp_path: Path, monkeypatch) -> None:
(tmp_path / "README.md").write_text("# Project\n", encoding="utf-8")
for directory in ("node_modules", "dist", ".next", ".wrangler"):
path = tmp_path / directory
path.mkdir()
(path / "README.md").write_text("[missing](missing.md)\n", encoding="utf-8")

monkeypatch.setattr(check_links, "ROOT", tmp_path)

assert check_links.markdown_files() == [tmp_path / "README.md"]


def test_arxiv_feed_parser_strips_version_and_normalizes_text() -> None:
feed = ElementTree.fromstring(
"""<feed xmlns="http://www.w3.org/2005/Atom">
Expand Down
40 changes: 40 additions & 0 deletions website/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.

# dependencies
/node_modules
/.pnp
.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/versions

# testing
/coverage

# next.js
/.next/
/.vinext/
/out/

# misc
.DS_Store
*.pem

# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*

# env files (can opt-in for committing if needed)
.env*

# vercel
.vercel

/dist/
/.wrangler/
/outputs/
/work/
5 changes: 5 additions & 0 deletions website/.openai/hosting.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"project_id": "appgprj_6a880c0383448191a8e115732079a231",
"d1": null,
"r2": null
}
10 changes: 10 additions & 0 deletions website/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Research catalog website

Interactive HTML front end for the evidence catalog in `../data/papers.yaml`.

The site provides full-text search, product and evidence filters, method views,
comparison-control summaries, expandable paper dossiers, English/Chinese UI,
and a small GitHub Star call to action.

`../scripts/build_readme.py` generates `data/catalog.json`; do not edit that
file by hand.
Loading