diff --git a/README.md b/README.md index ea240b6..aeccc1b 100644 --- a/README.md +++ b/README.md @@ -2,219 +2,201 @@ [中文](README.md) | [English](README_en.md) -![Python](https://img.shields.io/badge/python-3.10%2B-blue) -![Status](https://img.shields.io/badge/status-final%20lockbox%20reported-0b6e69) -![Model](https://img.shields.io/badge/model-Qwen2.5--7B%20LoRA-6f42c1) -![Scope](https://img.shields.io/badge/scope-evidence--first-f59e0b) +中文语音 / ASR 指令到可验证 Browser Task Contract 的大模型后训练与可信评测系统。 -Voice2Task 是一个中文 spoken command / ASR transcript 到浏览器任务合约的 -post-training 项目。 -任务不是控制浏览器，而是把用户口语命令转换成严格的 -Browser Task Contract JSON，供后续浏览器 agent 决定搜索、打开 URL、 -填写表单、抽取页面信息、澄清或拒绝高风险动作。 +基于 Qwen2.5-7B-Instruct + LoRA，覆盖数据构建、assistant-only SFT、 +gold-free 推理、严格合约评测、等步数消融与错误归因。 -## Recruiter Summary +Python · PyTorch · Transformers · TRL · PEFT · Qwen2.5-7B · LoRA -| Area | What this project built | -| --- | --- | -| Problem | 中文语音/ASR 浏览器命令 -> schema-valid browser task contract JSON | -| Model pipeline | Qwen2.5-7B-Instruct + LoRA SFT；adapter 私有，不随仓库发布 | -| Data/training | 247 seeds / 696 SFT rows / 2,100 preference pairs；public dev/test 为 `DEVELOPMENT_ONLY_SPENT`；final SFT 只使用既有训练数据 | -| Prompt/eval hardening | 统一 gold-free prompt policy `unified_gold_free_v1`；严格 JSON parse -> strict schema -> semantic contract -> exact match 分层验证 | -| Frozen evaluation | 120-row `lockbox-v1`，120 semantic families，manifest frozen，one-look final evaluation | -| Result boundary | final SFT 没有提升 strict contract exact match；`no overall model improvement claim` | - -## Final Lockbox v1 Result - -Frozen protocol: -`lockbox_hash=06114cf3ad6029930284af5f2245fb2c4a8174fd35c6a1107f4c73482b555b33`, -prompt policy `unified_gold_free_v1`, greedy decoding, -schema guard + one schema retry, strict evaluator, -two pre-registered arms only. - -| Metric | Base Qwen2.5-7B | Final SFT adapter | Delta | +| 247 seeds -> 696 SFT rows | 282 manifest-bound train-only rows | 2100 preference pairs | +| --- | --- | --- | +| 120 rows / 120 families frozen lockbox | 真实私有 A100 bounded smoke | 1485 automated tests | + +## 项目价值 + +Voice2Task 不控制浏览器。它把中文口语命令或 ASR transcript 转换成严格、 +机器可校验的 Browser Task Contract，供下游系统决定搜索、打开 URL、 +填写表单、抽取页面信息、澄清请求或拒绝高风险动作。 + +项目的核心不是生成一段“看起来合理”的文本，而是让模型输出可解析、 +可做 schema 检查、可逐字段比较并能追溯失败原因的结构化任务合约。 + +这使后训练目标、推理输出和评测口径落在同一份明确契约上， +同时把浏览器执行、线上自动化与生产部署留在项目边界之外。 + +## 核心结果 + +最终结果来自 120 rows / 120 semantic families 的冻结 one-look lockbox。 +Base 与 Final SFT 使用预注册 prompt、greedy decode、schema guard 和严格评测器。 + +| 冻结 lockbox 指标 | Base Qwen2.5-7B | Final SFT | 变化 | | --- | ---: | ---: | ---: | -| `contract_exact_match` | 0.0167 | 0.0083 | -0.0083 | -| `semantic_contract_valid_rate` | 0.8250 | 0.8667 | +0.0417 | -| `task_type_accuracy` | 0.7917 | 0.8583 | +0.0667 | -| `route_accuracy` | 0.8000 | 0.8583 | +0.0583 | -| `confirmation_accuracy` | 0.7083 | 0.7917 | +0.0833 | -| `strict_schema_valid_rate` | 1.0000 | 0.9833 | -0.0167 | -| `slot_f1` | 0.0417 | 0.0500 | +0.0083 | -| `slot_f1_soft` | 0.3783 | 0.3867 | +0.0084 | - -Interpretation: - -- Final SFT did **not** improve strict contract exact match on the frozen lockbox. -- Final SFT did improve several semantic/channel metrics: - `semantic_contract_valid_rate +0.0417`, `task_type_accuracy +0.0667`, - `route_accuracy +0.0583`, `confirmation_accuracy +0.0833`. -- This is aggregate-only one-look evidence. Public reports do not include row-level failure analysis；不能据此推断 row-level failure cause、natural-ASR generalization 或 overall SFT causal effect。 - -Evidence links: - -- [Final comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json) -- [Final run card](reports/lockbox-v1/final-evaluation/run-card.json) -- [Final comparison Markdown](reports/lockbox-v1/final-evaluation/comparison.md) -- [Current status and evidence](docs/current-status.md) -- [Lockbox protocol](docs/lockbox.md) - -## Explicit Non-Claims - -This repository does **not** claim: - -- overall model improvement from final SFT (`no overall model improvement claim`); -- production readiness; -- safety readiness; -- executable browser quality; -- DPO success; -- adapter/checkpoint release; -- live-browser benchmark improvement. - -The strongest supported claim is narrower: -under a frozen 120-row lockbox and a gold-free strict evaluator, -final SFT improved several semantic/channel aggregate metrics -but reduced strict full-contract exact match. - -## Repository Role - -| This repo is | This repo is not | -| --- | --- | -| A speech/ASR-to-contract post-training evidence repository | A generic chat fine-tuning project | -| A strict JSON contract generation and evaluation pipeline | A GUI action policy or browser controller | -| A public-safe SFT/DPO data, training, prediction, and evaluation workflow | A checkpoint or adapter release | -| A place where negative, blocked, and superseded evidence stays auditable | A success story built by deleting inconvenient results | +| Semantic contract valid rate | 82.50% | 86.67% | **+4.17 pp** | +| Task type accuracy | 79.17% | 85.83% | **+6.67 pp** | +| Route accuracy | 80.00% | 85.83% | **+5.83 pp** | +| Confirmation accuracy | 70.83% | 79.17% | **+8.33 pp** | +| Strict contract exact match | 1.67% | 0.83% | **-0.83 pp** | -## Method Overview +Final SFT arm 在 semantic validity、task type、route 与 confirmation 指标上的 +聚合得分高于 Base arm，但 strict full-contract exact 得分更低。因此项目不声称 +整体模型质量提升，也不把局部指标差异包装成完整合约生成能力已经改善。 -1. Build public-safe Voice2Task data from seed traces into SFT and preference rows. -2. Render Qwen chat prompts with no gold contract in prediction prompts. -3. Train LoRA SFT adapters on existing training data only. -4. Decode greedily with `max_new_tokens=256`, schema guard enabled, and at most one schema retry. -5. Score with strict layered metrics: JSON parse, strict schema validity, - semantic contract validity, exact match, slot-level metrics, - route/task/confirmation/safety metrics. -6. Freeze lockbox rows and manifest before the final one-look evaluation. +权威结果见 [lockbox comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json)； +完整解释与限制见 [current status](docs/current-status.md)。 -## Quick Start +## 我完成的核心工作 -Install local tooling: +### 1. 数据与后训练流水线 -```bash -python -m venv .venv -source .venv/bin/activate -pip install -e '.[dev,dataset]' -``` +- 基于 Qwen2.5-7B-Instruct + LoRA，使用 PyTorch、Transformers、TRL 与 PEFT + 实现 assistant-only loss 后训练路径；把 247 条 seeds 构建为 696 条 SFT rows + 与 2100 组 preference pairs，并保留严格的数据和合约校验。 -Rebuild and validate the committed public sample: +- 将 282 条 canonical train-only rows 绑定到正式 manifest 与 SHA-256，避免训练 + 输入悄然漂移。真实训练仅接受 ignored private config、本地模型权重和私有 A100； + adapter/checkpoint 不公开，也不允许运行时下载替代本地模型。 -```bash -PYTHONPATH=src python -m voice2task.cli.data build-public \ - --seed data/public-samples/seed_traces.jsonl \ - --output data/public-samples - -PYTHONPATH=src python -m voice2task.cli.data validate \ - --sft data/public-samples/sft_public_sample.jsonl \ - --dpo data/public-samples/dpo_public_sample.jsonl \ - --manifest data/public-samples/manifest_public_sample.json \ - --public - -PYTHONPATH=src python -m voice2task.cli.data audit-splits \ - --seed data/public-samples/seed_traces.jsonl \ - --sft data/public-samples/sft_public_sample.jsonl \ - --dpo data/public-samples/dpo_public_sample.jsonl \ - --manifest data/public-samples/manifest_public_sample.json \ - --output reports/public-sample/split-integrity-audit -``` +[归档的 public-safe A100 smoke 证据](openspec/changes/archive/2026-07-15-rerun-real-a100-sft-smoke-after-cli-fix-v1/tasks.md) +记录了恰好一次 launch、一个 optimizer step、两条训练 rows；224 个 adapter tensors +中 112 个发生变化且全部有限。它只证明 bounded training path 与参数更新可行。 -Run local baselines and metrics: +### 2. 可信评测与实验设计 -```bash -PYTHONPATH=src python -m voice2task.cli.eval baseline \ - --gold data/public-samples/sft_public_sample.jsonl \ - --output reports/public-sample/rule_baseline_predictions.jsonl - -PYTHONPATH=src python -m voice2task.cli.eval metrics \ - --gold data/public-samples/sft_public_sample.jsonl \ - --predictions reports/public-sample/rule_baseline_predictions.jsonl \ - --output reports/public-sample -``` +- 构建 120 rows / 120 families 的冻结 one-look lockbox；固定 gold-free prompt、 + greedy decode、schema guard 与 strict evaluator，分层衡量 JSON、schema、语义、 + 路由、确认和 strict full-contract exact。 -Dry-run training metadata export remains available, but real heavy training is gated by explicit config: +- 设计 Control/Treatment 等步数消融，每个 arm 固定 3132 optimizer steps， + 明确不是 token matched。实验没有调参评测器、语义放宽、预测修复或选择性报告， + 正向指标与 strict exact 回退同时保留。 -```bash -PYTHONPATH=src python -m voice2task.cli.train sft \ - --config configs/sft-dev.json \ - --manifest data/public-samples/manifest_public_sample.json \ - --output-dir reports/public-sample/sft-dry-run \ - --dry-run - -PYTHONPATH=src python -m voice2task.cli.train dpo \ - --config configs/dpo-dev.json \ - --manifest data/public-samples/manifest_public_sample.json \ - --output-dir reports/public-sample/dpo-dry-run \ - --dry-run -``` +### 3. 错误归因与安全边界 -## Metric Interpretation Boundaries +- 错误分析显示 68.79% 的 V1 strict failures 集中在 core slots；据此区分 + copy-backed、bounded structured 与 unresolved 表示。实现的 + [observe-only provenance shadow hook](reports/public-sample/copy-backed-prediction-shadow-hook/summary.json) + 默认关闭，不改预测、不参与执行决策。 -`contract_exact_match` is a hard full-contract exact-match metric. Future evaluator runs use recursive JSON type-strict equality: booleans, integers, and floating-point values are distinct; object key order and serialization whitespace are ignored; array order is preserved; non-finite or non-JSON values fail closed. Historical metrics were not re-scored. -`normalized_command` string-mismatch diagnostics are explanatory row-level -evidence only: they do not relax, normalize, semantically score, repair, replace, -or re-score predictions, and they do not automatically mark Chinese phrase differences such as `搜索/查询` or `明天的天气/明天天气` as equivalent. +- [template-disjoint challenge 的已观测结果](reports/public-sample/copy-shadow-template-disjoint-challenge-v1/adapter-evaluation/challenge-evaluation-summary.json) + 记录 3 个 source-absent、6 个 normalization-collision 和 3 个 partial-span + false-trust。它是 adversarial verifier fixture 证据，不是自然语言 ASR 或模型质量证据。 -`normalized_command` gold targets are canonical Chinese intent phrases, not -verbatim transcripts or ASR text. This is target-writing guidance for SFT/DPO -data and prompts, not evaluator-side normalization, semantic-equivalence -scoring, prediction repair, or re-scoring. +训练与证据导出还使用 fail-closed GPU/path/data/adapter/public-output gates； +任一模型身份、数据绑定、输出边界或 smoke 后置条件不满足时都会停止。 -### Normalized Command Target Policy +## 系统流程 + +```mermaid +flowchart LR + A["Chinese Voice / ASR"] --> B["Dataset & Contract Validation"] + B --> C["Qwen2.5-7B LoRA SFT"] + C --> D["Gold-free Prediction"] + D --> E["JSON / Schema Guard"] + E --> F["Strict Contract Evaluation"] + F --> G["Error Analysis & Provenance Shadow Audit"] +``` + +流程终点是严格评测与错误审计，不包含浏览器执行、线上部署或生产自动化。 -Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR -text. Representative forms include `搜索北京明天天气`, `打开示例网站`, -`填写邮箱并确认`, and `拒绝代替用户付款`. This is authoring guidance, not -evaluator-side normalization or semantic-equivalence scoring. +## 为什么结果可信 -## Evidence Archive +- 数据切分采用 family-aware 规则，并通过 + [split integrity audit](reports/public-sample/split-integrity-audit/summary.json) + 显式记录跨 split 风险，而不是默认宣称数据天然干净。 -Longer-running internal evidence remains documented below the headline result: +- 最终评测使用 120 rows / 120 families 的冻结 one-look lockbox； + 只公开聚合结果，不使用 lockbox 逐行错误做再次调优。 -- Public split integrity: the current 282/207/207 dev/test boundary is - `DEVELOPMENT_ONLY_SPENT`, not blind, independent, leakage-free, or - final-generalization evidence. The audit preserves historical rows and - metrics; lockbox-v1 remains the distinct frozen one-look aggregate boundary. -- Contract V2 projection: `PARTIAL_SCHEMA_BENEFIT`; derived-field-only strict - failures are 14.65%, normalized-command-only strict failures are 14.65%, and - core slot failures remain 68.79% of V1 strict failures. This is useful - schema-burden evidence, not model-quality evidence. -- Copy-backed verification and shadow mode: observe-only provenance/interface evidence, not runtime enforcement. -- Copy-shadow template-disjoint challenge v1: adversarial verifier fixture, not a naturalistic language benchmark. -- Earlier step-matched SFT ablations: mixed/inconclusive; no stable broad canonical-slot benefit. +- recursive `JSON type-strict` equality 是未来 evaluator runs 的 exact 边界； + 本页展示的历史 lockbox metrics 未重新计分。 Object key order and serialization + whitespace 可忽略，数组顺序保留，布尔、整数与浮点数区分，异常值 fail closed。 -See [current status](docs/current-status.md) and -[public evidence index](reports/public-sample/EVIDENCE_INDEX.md) for the -complete archived map. +- step-matched 实验固定 prompt、decoder、schema guard 与 evaluator， + 每个 arm 使用 3132 optimizer steps，并明确披露 not token matched。 -## A100 Boundary +- public dev/test 标记为 `DEVELOPMENT_ONLY_SPENT`；它们不是干净独立 held-out。 + 页面同时报告局部正向变化和 strict exact 负向结果。 -GPU-heavy training and prediction are designed for a private A100 development -machine. -Public repo artifacts intentionally omit checkpoints, LoRA adapters, raw logs, -remote caches, private corpus rows, hostnames, SSH details, credentials, -private paths, private override configs, and production-readiness claims. +## 仓库导航 -## Validation +| 入口 | 用途 | +| --- | --- | +| [Public manifest](data/public-samples/manifest_public_sample.json) | 247 / 696 / 2100 计数、split 与文件哈希 | +| [Train-only SFT artifact](data/public-samples/sft_train_public_sample.jsonl) | 282 条 manifest-bound canonical train rows | +| [Training CLI](src/voice2task/cli/train.py) | SFT preflight、训练与 gold-free prediction 入口 | +| [Evaluation CLI](src/voice2task/cli/eval.py) | 严格分层评测入口 | +| [Lockbox comparison](reports/lockbox-v1/final-evaluation/comparison.json) | Base 与 Final SFT 冻结聚合结果 | +| [Step-matched ablation](reports/public-sample/step-matched-canonical-slot-ablation/comparison.json) | 3132-step Control/Treatment 对照与边界 | +| [Slot-error summary](reports/public-sample/slot-error-mechanism-analysis/summary.json) | core-slot 瓶颈与表示分析 | +| [Evidence index](reports/public-sample/EVIDENCE_INDEX.md) | CURRENT、HISTORICAL、BLOCKED 等证据地图 | +| [Training spec](openspec/specs/supervised-contract-tuning/spec.md) / [dataset spec](openspec/specs/voice2task-dataset-preparation/spec.md) | 当前 OpenSpec 训练与数据契约 | +| [Tests](tests) | 数据、训练、评测、证据与边界回归测试 | -Useful local checks: +## 本地验证 + +以下命令只验证公开仓库，不下载模型，也不启动训练： ```bash PYTHONPATH=src pytest -q -PYTHONPATH=src ruff check src tests +PYTHONPATH=src ruff check . OPENSPEC_TELEMETRY=0 openspec validate --all --strict PYTHONPATH=src python scripts/check_current_truth_surface.py -git diff --check ``` +真实训练需要 ignored private config 与本地模型权重；仓库不公开主机、SSH、路径、 +token、私有配置、原始日志或 adapter 位置。 + +## 项目状态 + +**已完成：Portfolio-ready research and engineering project。** + +- 完成公开数据构建、manifest/SHA-256 绑定与 assistant-only LoRA SFT 路径。 +- 完成 fail-closed 私有 A100 bounded smoke，证明单步训练和 adapter 参数更新。 +- 完成 gold-free 推理、strict evaluator 与冻结 one-look lockbox 聚合评测。 +- 完成 3132-step matched A/B、slot-error attribution 与 mixed representation 设计。 +- 完成 observe-only provenance shadow audit 与 verifier false-trust challenge。 +- 当前自动化基线为 1485 tests；adapter 与 checkpoint 仍为私有且未发布。 + +**证据边界：** + +- Contract V2 离线投影结论是 `PARTIAL_SCHEMA_BENEFIT`，不是模型质量结论。 +- derived-field-only strict failures 为 14.65%；这只说明部分 schema burden。 +- core slot failures 仍占 68.79%，是完整合约 strict failure 的主要瓶颈。 +- public dev/test 状态是 `DEVELOPMENT_ONLY_SPENT`，不构成 clean held-out evidence。 +- 后续 exact 采用 `JSON type-strict` 递归比较；历史指标没有重算。 +- `strict exact remains canonical`：局部指标不能替代完整合约严格一致性。 + +### Metric Interpretation Boundaries + +`contract_exact_match` is a hard full-contract exact-match metric. Future runs use +recursive `JSON type-strict` equality: object key order and serialization whitespace +are ignored, array order is preserved, and non-finite or non-JSON values fail closed. + +`normalized_command` string-mismatch diagnostics are explanatory row-level evidence only. +They do not relax, normalize, semantically score, repair, replace, or re-score predictions. +They do not automatically mark Chinese phrase differences such as `搜索/查询` or +`明天的天气/明天天气` as equivalent. + +### Normalized Command Target Policy + +Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR text. +Representative forms include `搜索北京明天天气`、`打开示例网站`、`填写邮箱并确认` +和 `拒绝代替用户付款`。 This is authoring guidance only: not evaluator-side normalization, +semantic-equivalence scoring, prediction repair, or rescoring. + +**明确不声称：** + +- 不声称整体模型质量提升或 final SFT 的总体因果收益。 +- 不声称 production readiness 或 safety certification。 +- 不声称 live-browser benchmark gain；项目没有控制浏览器。 +- 不声称 DPO benefit，也没有以本结果授权 DPO/GRPO。 +- 不声称 clean held-out generalization 或 naturalistic ASR generalization。 +- 不声称已经发布 checkpoint、adapter 或可复现的私有模型产物。 + +项目展示的是可审计的数据、后训练、严格评测和负结果处理能力， +不是把一次 bounded smoke 或若干正向 aggregate metrics 包装成上线能力。 + ## License 本项目采用 [MIT License](LICENSE)。 diff --git a/README_en.md b/README_en.md index 64824e1..d98b621 100644 --- a/README_en.md +++ b/README_en.md @@ -2,220 +2,226 @@ [中文](README.md) | [English](README_en.md) -![Python](https://img.shields.io/badge/python-3.10%2B-blue) -![Status](https://img.shields.io/badge/status-final%20lockbox%20reported-0b6e69) -![Model](https://img.shields.io/badge/model-Qwen2.5--7B%20LoRA-6f42c1) -![Scope](https://img.shields.io/badge/scope-evidence--first-f59e0b) +A post-training and trustworthy-evaluation system that maps Chinese voice / ASR +instructions to verifiable Browser Task Contracts. -Voice2Task is a post-training project for Chinese spoken commands / ASR -transcripts to browser task contracts. -It does not control a browser. -It converts user commands into strict Browser Task Contract JSON so a downstream -browser agent can decide whether to search, open a URL, fill a form, -extract page information, clarify, or refuse a risky action. +Built on Qwen2.5-7B-Instruct + LoRA, the project covers data construction, +assistant-only SFT, gold-free inference, strict evaluation, step-matched +ablation, and error attribution. -## Recruiter Summary +Python · PyTorch · Transformers · TRL · PEFT · Qwen2.5-7B · LoRA -| Area | What this project built | -| --- | --- | -| Problem | Chinese voice/ASR browser commands -> schema-valid browser task contract JSON | -| Model pipeline | Qwen2.5-7B-Instruct + LoRA SFT; the adapter remains private and is not released in this repo | -| Data/training | 247 seeds / 696 SFT rows / 2,100 preference pairs; public dev/test are `DEVELOPMENT_ONLY_SPENT`; final SFT used existing training data only | -| Prompt/eval hardening | Unified gold-free prompt policy `unified_gold_free_v1`; strict layered validation from JSON parse -> strict schema -> semantic contract -> exact match | -| Frozen evaluation | 120-row `lockbox-v1`, 120 semantic families, frozen manifest, one-look final evaluation | -| Result boundary | Final SFT did not improve strict contract exact match; no overall model improvement claim is made | - -## Final Lockbox v1 Result - -Frozen protocol: -`lockbox_hash=06114cf3ad6029930284af5f2245fb2c4a8174fd35c6a1107f4c73482b555b33`, -prompt policy `unified_gold_free_v1`, greedy decoding, -schema guard + one schema retry, strict evaluator, -and exactly two pre-registered arms. - -| Metric | Base Qwen2.5-7B | Final SFT adapter | Delta | -| --- | ---: | ---: | ---: | -| `contract_exact_match` | 0.0167 | 0.0083 | -0.0083 | -| `semantic_contract_valid_rate` | 0.8250 | 0.8667 | +0.0417 | -| `task_type_accuracy` | 0.7917 | 0.8583 | +0.0667 | -| `route_accuracy` | 0.8000 | 0.8583 | +0.0583 | -| `confirmation_accuracy` | 0.7083 | 0.7917 | +0.0833 | -| `strict_schema_valid_rate` | 1.0000 | 0.9833 | -0.0167 | -| `slot_f1` | 0.0417 | 0.0500 | +0.0083 | -| `slot_f1_soft` | 0.3783 | 0.3867 | +0.0084 | - -Interpretation: - -- Final SFT did **not** improve strict contract exact match on the frozen lockbox. -- Final SFT improved several semantic/channel metrics: - `semantic_contract_valid_rate +0.0417`, `task_type_accuracy +0.0667`, - `route_accuracy +0.0583`, `confirmation_accuracy +0.0833`. -- This is aggregate-only one-look evidence. Public reports do not include row-level failure analysis, and it cannot establish row-level failure causes, natural-ASR generalization, or an overall SFT causal effect. - -Evidence links: - -- [Final comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json) -- [Final run card](reports/lockbox-v1/final-evaluation/run-card.json) -- [Final comparison Markdown](reports/lockbox-v1/final-evaluation/comparison.md) -- [Current status and evidence](docs/current-status.md) -- [Lockbox protocol](docs/lockbox.md) - -## Explicit Non-Claims - -This repository does **not** claim: - -- overall model improvement from final SFT; -- production readiness; -- safety readiness; -- executable browser quality; -- DPO success; -- adapter/checkpoint release; -- live-browser benchmark improvement. - -The strongest supported claim is narrower: -under a frozen 120-row lockbox and a gold-free strict evaluator, -final SFT improved several semantic/channel aggregate metrics -but reduced strict full-contract exact match. - -## Repository Role - -| This repo is | This repo is not | -| --- | --- | -| A speech/ASR-to-contract post-training evidence repository | A generic chat fine-tuning project | -| A strict JSON contract generation and evaluation pipeline | A GUI action policy or browser controller | -| A public-safe SFT/DPO data, training, prediction, and evaluation workflow | A checkpoint or adapter release | -| A place where negative, blocked, and superseded evidence stays auditable | A success story built by deleting inconvenient results | +| 247 seeds -> 696 SFT rows | 282 manifest-bound train-only rows | 2100 preference pairs | +| --- | --- | --- | +| 120 rows / 120 families frozen lockbox | Private, bounded real A100 smoke run | 1485 automated tests | -## Method Overview +## Project Value -1. Build public-safe Voice2Task data from seed traces into SFT and preference rows. -2. Render Qwen chat prompts with no gold contract in prediction prompts. -3. Train LoRA SFT adapters on existing training data only. -4. Decode greedily with `max_new_tokens=256`, schema guard enabled, and at most one schema retry. -5. Score with strict layered metrics: JSON parse, strict schema validity, - semantic contract validity, exact match, slot-level metrics, - route/task/confirmation/safety metrics. -6. Freeze lockbox rows and manifest before the final one-look evaluation. +Voice2Task does not control a browser. It converts Chinese spoken commands or +ASR transcripts into strict, machine-verifiable Browser Task Contracts so a +downstream system can decide whether to search, open a URL, fill a form, extract +page information, clarify a request, or refuse a high-risk action. -## Quick Start +The goal is not to generate text that merely looks plausible. Model output must +be parseable, schema-checkable, comparable field by field, and traceable when a +failure occurs. -Install local tooling: +This places post-training targets, inference outputs, and evaluation criteria +under one explicit contract while keeping browser execution, online automation, +and production deployment outside the project boundary. -```bash -python -m venv .venv -source .venv/bin/activate -pip install -e '.[dev,dataset]' -``` +## Key Results -Rebuild and validate the committed public sample: +The final result comes from a frozen one-look lockbox with 120 rows across 120 +semantic families. Base and Final SFT used the preregistered prompt, greedy +decode, schema guard, and strict evaluator. -```bash -PYTHONPATH=src python -m voice2task.cli.data build-public \ - --seed data/public-samples/seed_traces.jsonl \ - --output data/public-samples - -PYTHONPATH=src python -m voice2task.cli.data validate \ - --sft data/public-samples/sft_public_sample.jsonl \ - --dpo data/public-samples/dpo_public_sample.jsonl \ - --manifest data/public-samples/manifest_public_sample.json \ - --public - -PYTHONPATH=src python -m voice2task.cli.data audit-splits \ - --seed data/public-samples/seed_traces.jsonl \ - --sft data/public-samples/sft_public_sample.jsonl \ - --dpo data/public-samples/dpo_public_sample.jsonl \ - --manifest data/public-samples/manifest_public_sample.json \ - --output reports/public-sample/split-integrity-audit +| Frozen lockbox metric | Base Qwen2.5-7B | Final SFT | Change | +| --- | ---: | ---: | ---: | +| Semantic contract valid rate | 82.50% | 86.67% | **+4.17 pp** | +| Task type accuracy | 79.17% | 85.83% | **+6.67 pp** | +| Route accuracy | 80.00% | 85.83% | **+5.83 pp** | +| Confirmation accuracy | 70.83% | 79.17% | **+8.33 pp** | +| Strict contract exact match | 1.67% | 0.83% | **-0.83 pp** | + +The Final SFT arm scored higher on semantic validity, task type, routing, and confirmation accuracy, but lower on strict full-contract exact match; the project therefore makes no overall model-quality improvement claim. + +See the authoritative [lockbox comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json) +and the full interpretation and limitations in [current status](docs/current-status.md). + +## What I Built + +### 1. Data and Post-Training Pipeline + +- Implemented a Qwen2.5-7B-Instruct + LoRA post-training path with PyTorch, + Transformers, TRL, PEFT, and assistant-only loss. The data pipeline expands + 247 seeds into 696 SFT rows and 2100 preference pairs while retaining strict + data and contract validation. + +- Bound 282 canonical train-only rows to the formal manifest and SHA-256 so + training inputs cannot drift silently. Real training runs only with an ignored + private config, local model weights, and an explicitly selected private A100; + runtime downloads do not replace the local model. The adapter and checkpoint are private. + +The archived [public-safe A100 smoke evidence](openspec/changes/archive/2026-07-15-rerun-real-a100-sft-smoke-after-cli-fix-v1/tasks.md) +records exactly one launch, one optimizer step, and two training rows. Of 224 +adapter tensors, 112 changed and all were finite. This proves only that the +bounded training path ran and updated parameters. + +### 2. Trustworthy Evaluation and Experimental Design + +- Built a frozen one-look lockbox with 120 rows across 120 families. The fixed + protocol uses a gold-free prompt, greedy decode, schema guard, and strict + evaluator to measure JSON, schema, semantics, routing, confirmation, and + strict full-contract exact match separately. + +- Designed a step-matched Control/Treatment ablation with exactly 3132 optimizer + steps per arm; the experiment is explicitly not token matched. It did not tune + the evaluator, relax semantic rules, repair predictions, or selectively report + only favorable metrics. Positive metrics and the strict-exact regression remain visible. + +### 3. Error Attribution and Safety Boundaries + +- Error analysis found that 68.79% of V1 strict failures were concentrated in + core slots. This motivated separate copy-backed, bounded structured, and + unresolved representations. The implemented + [observe-only provenance shadow hook](reports/public-sample/copy-backed-prediction-shadow-hook/summary.json) + is disabled by default, does not alter predictions, and does not participate + in execution decisions. + +- The observed [template-disjoint challenge result](reports/public-sample/copy-shadow-template-disjoint-challenge-v1/adapter-evaluation/challenge-evaluation-summary.json) + records 3 source-absent, 6 normalization-collision, and 3 partial-span + false-trust cases. It is an adversarial verifier fixture, not naturalistic ASR + or model-quality evidence. + +Training and evidence export also use fail-closed GPU, path, data, adapter, and +public-output gates. The process stops when model identity, data binding, output +boundaries, or smoke postconditions do not match the declared contract. + +## System Overview + +```mermaid +flowchart LR + A["Chinese Voice / ASR"] --> B["Dataset & Contract Validation"] + B --> C["Qwen2.5-7B LoRA SFT"] + C --> D["Gold-free Prediction"] + D --> E["JSON / Schema Guard"] + E --> F["Strict Contract Evaluation"] + F --> G["Error Analysis & Provenance Shadow Audit"] ``` -Run local baselines and metrics: +The workflow ends with strict evaluation and error auditing. It contains no +browser execution, online deployment, or production automation stage. -```bash -PYTHONPATH=src python -m voice2task.cli.eval baseline \ - --gold data/public-samples/sft_public_sample.jsonl \ - --output reports/public-sample/rule_baseline_predictions.jsonl - -PYTHONPATH=src python -m voice2task.cli.eval metrics \ - --gold data/public-samples/sft_public_sample.jsonl \ - --predictions reports/public-sample/rule_baseline_predictions.jsonl \ - --output reports/public-sample -``` +## Why the Results Are Credible + +- Data splits follow family-aware rules. The + [split integrity audit](reports/public-sample/split-integrity-audit/summary.json) + records cross-split risks explicitly instead of assuming the data is clean. + +- Final evaluation uses the frozen one-look lockbox of 120 rows / 120 families. + Only aggregate results are public; lockbox row-level errors were not used for + another tuning pass. + +- Recursive `JSON type-strict` equality is the exact-match boundary for future + evaluator runs; the historical lockbox metrics displayed here were not re-scored. + Object key order and serialization whitespace are ignored, array order is preserved, + booleans, integers, and floats remain distinct, and non-finite or non-JSON values fail closed. + +- The step-matched experiment fixes the prompt, decoder, schema guard, and + evaluator, with 3132 optimizer steps per arm, and explicitly states that token + counts were not matched. -Dry-run training metadata export remains available, but real heavy training is gated by explicit config: +- Public dev/test is marked `DEVELOPMENT_ONLY_SPENT`, not clean independent + held-out evidence. Both positive metric changes and the negative strict-exact + result are reported. + +## Repository Map + +| Entry | Purpose | +| --- | --- | +| [Public manifest](data/public-samples/manifest_public_sample.json) | Counts for 247 / 696 / 2100, splits, and file hashes | +| [Train-only SFT artifact](data/public-samples/sft_train_public_sample.jsonl) | 282 manifest-bound canonical training rows | +| [Training CLI](src/voice2task/cli/train.py) | SFT preflight, training, and gold-free prediction entry point | +| [Evaluation CLI](src/voice2task/cli/eval.py) | Strict layered evaluation entry point | +| [Lockbox comparison](reports/lockbox-v1/final-evaluation/comparison.json) | Frozen Base and Final SFT aggregate results | +| [Step-matched ablation](reports/public-sample/step-matched-canonical-slot-ablation/comparison.json) | 3132-step Control/Treatment comparison and boundaries | +| [Slot-error summary](reports/public-sample/slot-error-mechanism-analysis/summary.json) | Core-slot bottleneck and representation analysis | +| [Evidence index](reports/public-sample/EVIDENCE_INDEX.md) | Evidence map for CURRENT, HISTORICAL, BLOCKED, and related states | +| [Training spec](openspec/specs/supervised-contract-tuning/spec.md) / [dataset spec](openspec/specs/voice2task-dataset-preparation/spec.md) | Current OpenSpec training and data contracts | +| [Tests](tests) | Regression coverage for data, training, evaluation, evidence, and boundaries | + +## Local Verification + +These commands validate the public repository only. They do not download a model +or start training. ```bash -PYTHONPATH=src python -m voice2task.cli.train sft \ - --config configs/sft-dev.json \ - --manifest data/public-samples/manifest_public_sample.json \ - --output-dir reports/public-sample/sft-dry-run \ - --dry-run - -PYTHONPATH=src python -m voice2task.cli.train dpo \ - --config configs/dpo-dev.json \ - --manifest data/public-samples/manifest_public_sample.json \ - --output-dir reports/public-sample/dpo-dry-run \ - --dry-run +PYTHONPATH=src pytest -q +PYTHONPATH=src ruff check . +OPENSPEC_TELEMETRY=0 openspec validate --all --strict +PYTHONPATH=src python scripts/check_current_truth_surface.py ``` -## Metric Interpretation Boundaries +Real training requires an ignored private config and local model weights. No +private values, host details, SSH details, private runtime paths, tokens, raw logs, +or adapter locations are published. -`contract_exact_match` is a hard full-contract exact-match metric. Future evaluator runs use recursive JSON type-strict equality: booleans, integers, and floating-point values are distinct; object key order and serialization whitespace are ignored; array order is preserved; non-finite or non-JSON values fail closed. Historical metrics were not re-scored. -`normalized_command` string-mismatch diagnostics are explanatory row-level -evidence only: they do not relax, normalize, semantically score, repair, replace, -or re-score predictions, and they do not automatically mark Chinese phrase -differences such as `搜索/查询` or `明天的天气/明天天气` as equivalent. +## Project Status -`normalized_command` gold targets are canonical Chinese intent phrases, not -verbatim transcripts or ASR text. This is target-writing guidance for SFT/DPO -data and prompts, not evaluator-side normalization, semantic-equivalence -scoring, prediction repair, or re-scoring. +**Completed: Portfolio-ready research and engineering project.** -### Normalized Command Target Policy +- Completed public data construction, manifest/SHA-256 binding, and the + assistant-only LoRA SFT path. +- Completed a fail-closed private A100 bounded smoke that demonstrates a + single-step training path and adapter parameter update. +- Completed gold-free inference, the strict evaluator, and frozen one-look + lockbox aggregate evaluation. +- Completed the 3132-step-matched A/B experiment, slot-error attribution, and + mixed-representation design. +- Completed the observe-only provenance shadow audit and verifier false-trust challenge. +- The current automated baseline is 1485 tests; the adapter and checkpoint + remain private and unreleased. -Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR -text. Representative forms include `搜索北京明天天气`, `打开示例网站`, -`填写邮箱并确认`, and `拒绝代替用户付款`. This is authoring guidance, not -evaluator-side normalization or semantic-equivalence scoring. +**Evidence boundaries:** -## Evidence Archive +- The Contract V2 offline projection is `PARTIAL_SCHEMA_BENEFIT`, not a model-quality conclusion. +- Derived-field-only strict failures are 14.65%; this measures only part of the schema burden. +- Core-slot failures remain 68.79%, the main bottleneck in full-contract strict failures. +- Public dev/test is `DEVELOPMENT_ONLY_SPENT`, not clean held-out evidence. +- Future exact comparison uses `JSON type-strict`; historical metrics were not recomputed. +- `strict exact remains canonical`: local metrics cannot replace strict full-contract agreement. -Longer-running internal evidence remains documented below the headline result: +### Metric Interpretation Boundaries -- Public split integrity: the current 282/207/207 dev/test boundary is - `DEVELOPMENT_ONLY_SPENT`, not blind, independent, leakage-free, or - final-generalization evidence. The audit preserves historical rows and - metrics; lockbox-v1 remains the distinct frozen one-look aggregate boundary. -- Contract V2 projection: `PARTIAL_SCHEMA_BENEFIT`; derived-field-only strict - failures are 14.65%, normalized-command-only strict failures are 14.65%, and - core slot failures remain 68.79% of V1 strict failures. This is useful - schema-burden evidence, not model-quality evidence. -- Copy-backed verification and shadow mode: observe-only provenance/interface evidence, not runtime enforcement. -- Copy-shadow template-disjoint challenge v1: adversarial verifier fixture, not a naturalistic language benchmark. -- Earlier step-matched SFT ablations: mixed/inconclusive; no stable broad canonical-slot benefit. +`contract_exact_match` is a hard full-contract exact-match metric. Future runs +use recursive `JSON type-strict` equality: object key order and serialization whitespace +are ignored, array order is preserved, and non-finite or non-JSON values fail closed. -See [current status](docs/current-status.md) and the -[public evidence index](reports/public-sample/EVIDENCE_INDEX.md) for the -complete archived map. +`normalized_command` string-mismatch diagnostics are explanatory row-level evidence only. +They do not relax, normalize, semantically score, repair, replace, or re-score predictions. +They do not automatically mark Chinese phrase differences such as `搜索/查询` or +`明天的天气/明天天气` as equivalent. -## A100 Boundary +### Normalized Command Target Policy -GPU-heavy training and prediction are designed for a private A100 development -machine. -Public repo artifacts intentionally omit checkpoints, LoRA adapters, raw logs, -remote caches, private corpus rows, hostnames, SSH details, credentials, -private paths, private override configs, and production-readiness claims. +Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR text. +Representative forms include `搜索北京明天天气`, `打开示例网站`, `填写邮箱并确认`, +and `拒绝代替用户付款`. This is target-authoring guidance only, not evaluator-side +normalization, semantic-equivalence scoring, prediction repair, or re-scoring. -## Validation +**Explicit non-claims:** -Useful local checks: +- No overall model improvement or overall causal benefit from Final SFT. +- No production readiness or safety certification. +- No live-browser benchmark gain; the project does not control a browser. +- No DPO benefit, and these results do not authorize DPO or GRPO. +- No clean held-out generalization or naturalistic ASR generalization. +- No released checkpoint, adapter, or reproducible private model artifact. -```bash -PYTHONPATH=src pytest -q -PYTHONPATH=src ruff check src tests -OPENSPEC_TELEMETRY=0 openspec validate --all --strict -PYTHONPATH=src python scripts/check_current_truth_surface.py -git diff --check -``` +The project demonstrates auditable data, post-training, strict evaluation, and +responsible handling of negative results. It does not present a bounded smoke or +several positive aggregate metrics as production capability. ## License diff --git a/docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md b/docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md new file mode 100644 index 0000000..e7a2901 --- /dev/null +++ b/docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md @@ -0,0 +1,762 @@ +# Voice2Task Recruiter-Facing Project Page V1 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Rewrite the Chinese and English GitHub homepages so recruiting and ML readers can understand the project, measured result, evidence quality, and repository navigation within 30 seconds. + +**Architecture:** Treat the READMEs as synchronized views over immutable repository evidence. Extract every number from committed manifests and reports, keep the frozen-lockbox regression visible, and validate through existing truth-surface, link, leak, lint, OpenSpec, and full-test checks. Keep the Draft PR stacked on `codex/materialize-manifest-bound-train-only-sft-v1` so its diff is documentation-only while required evidence links resolve. + +**Tech Stack:** GitHub-flavored Markdown, Mermaid, Python 3.10+, pytest, Ruff, OpenSpec, repository truth-surface and leak-scan utilities. + +--- + +## File Map + +- Modify `README.md`: primary Chinese recruiting page. +- Modify `README_en.md`: English page with identical facts and section order. +- Preserve `docs/superpowers/specs/2026-07-16-optimize-recruiter-facing-project-page-v1-design.md`. +- Create this plan; do not add scripts, tests, images, data, reports, or model artifacts. + +### Task 1: Freeze the Documentation Truth Inputs + +**Files:** +- Read: `data/public-samples/manifest_public_sample.json` +- Read: `reports/lockbox-v1/final-evaluation/comparison.json` +- Read: `reports/public-sample/step-matched-canonical-slot-ablation/comparison.json` +- Read: `reports/public-sample/slot-error-mechanism-analysis/summary.json` + +- [x] **Step 1: Confirm branch ancestry and starting scope** + +Run: + +```bash +git merge-base --is-ancestor \ + origin/codex/materialize-manifest-bound-train-only-sft-v1 HEAD +git status --short --branch +``` + +Expected: ancestry exits 0; no source, data, metric, or report file is modified. + +- [x] **Step 2: Extract dataset and lockbox facts** + +Run: + +```bash +jq '{counts, split_counts, files}' \ + data/public-samples/manifest_public_sample.json +jq '{comparison, metrics}' \ + reports/lockbox-v1/final-evaluation/comparison.json +``` + +Expected: + +```text +247 seeds; 696 SFT rows; 2100 DPO pairs +train/dev/test = 282/207/207 +sft_train = data/public-samples/sft_train_public_sample.jsonl +semantic valid: 0.8250 -> 0.8667 +task type: 0.7917 -> 0.8583 +route: 0.8000 -> 0.8583 +confirmation: 0.7083 -> 0.7917 +strict exact: 0.0167 -> 0.0083 +``` + +- [x] **Step 3: Extract experiment and error-analysis boundaries** + +Run: + +```bash +jq '{decision_label, step_matching, claims}' \ + reports/public-sample/step-matched-canonical-slot-ablation/comparison.json +jq '{decision_label, summary_metrics, claims}' \ + reports/public-sample/slot-error-mechanism-analysis/summary.json +``` + +Expected: 3132 optimizer steps per arm, `not_token_matched=true`, no DPO/GRPO, +`MIXED_SLOT_REPRESENTATION_REQUIRED`, and no overall-improvement authorization. + +- [x] **Step 4: Confirm the full pre-edit baseline** + +Run: `PYTHONPATH=src pytest -q` + +Expected: `1485 passed`. + +### Task 2: Rewrite the Chinese Homepage + +**Files:** +- Modify: `README.md` + +- [x] **Step 1: Replace the opening with this compact Hero model** + +```markdown +# Voice2Task Post-Training + +[中文](README.md) | [English](README_en.md) + +中文语音 / ASR 指令到可验证 Browser Task Contract 的大模型后训练与可信评测系统。 + +基于 Qwen2.5-7B-Instruct + LoRA，覆盖数据构建、assistant-only SFT、 +gold-free 推理、严格合约评测、等步数消融与错误归因。 + +`Python` · `PyTorch` · `Transformers` · `TRL` · `PEFT` · `Qwen2.5-7B` · `LoRA` + +| 247 seeds -> 696 SFT rows | 282 manifest-bound train-only rows | 2100 preference pairs | +| --- | --- | --- | +| 120 rows / 120 families frozen lockbox | 真实私有 A100 bounded smoke | 1485 automated tests | +``` + +Do not use external badge images. Follow the Hero immediately with project +value and results. + +- [x] **Step 2: Add project value and the exact result table** + +```markdown +## 项目价值 + +Voice2Task 不控制浏览器。它把中文口语命令或 ASR transcript 转换成严格、 +机器可校验的 Browser Task Contract，让下游 agent 决定搜索、打开 URL、 +填写表单、抽取页面信息、澄清或拒绝高风险动作。 + +## 核心结果 + +| 冻结 lockbox 指标 | Base Qwen2.5-7B | Final SFT | 变化 | +| --- | ---: | ---: | ---: | +| Semantic contract valid rate | 82.50% | 86.67% | **+4.17 pp** | +| Task type accuracy | 79.17% | 85.83% | **+6.67 pp** | +| Route accuracy | 80.00% | 85.83% | **+5.83 pp** | +| Confirmation accuracy | 70.83% | 79.17% | **+8.33 pp** | +| Strict contract exact match | 1.67% | 0.83% | **-0.83 pp** | + +模型在语义结构、任务分类、路由和确认判断上取得提升，但 strict +full-contract exact match 未同步提升，因此项目不声称整体模型质量提升。 +``` + +Link the final comparison JSON and current status directly below. Never call +confirmation accuracy overall accuracy. + +- [x] **Step 3: Add exactly three contribution modules** + +Use: + +```markdown +## 我完成的核心工作 +### 1. 数据与后训练流水线 +### 2. 可信评测与实验设计 +### 3. 错误归因与安全边界 +``` + +Module 1 must include Qwen2.5-7B LoRA, PyTorch/Transformers/TRL/PEFT, +assistant-only loss, 247 -> 696 + 2100, the 282-row manifest/SHA-256 binding, +and private local-only A100 operation. + +Module 2 must include the frozen 120/120 one-look lockbox, fixed prompt, greedy +decoding, schema guard, strict evaluator, 3132 steps per arm, and no evaluator +tuning or selective reporting. + +Module 3 must include 68.79% core-slot failures, copy-backed/structured/ +unresolved representations, observe-only provenance shadow hooks, false-trust +findings, and fail-closed execution gates. + +- [x] **Step 4: Add the exact GitHub-native Mermaid flow** + +```mermaid +flowchart LR + A["Chinese Voice / ASR"] --> B["Dataset & Contract Validation"] + B --> C["Qwen2.5-7B LoRA SFT"] + C --> D["Gold-free Prediction"] + D --> E["JSON / Schema Guard"] + E --> F["Strict Contract Evaluation"] + F --> G["Error Analysis & Provenance Shadow Audit"] +``` + +Do not add browser execution, deployment, production, or online automation. + +- [x] **Step 5: Add the remaining sections in this order** + +```markdown +## 为什么结果可信 +## 仓库导航 +## 本地验证 +## 项目状态 +## License +``` + +Use at most five trust bullets. Link the manifest, train-only artifact, +training CLI, evaluation CLI, lockbox comparison, step-matched ablation, +slot-error analysis, evidence index, OpenSpec specs, and tests. + +The verification block must contain only: + +```bash +PYTHONPATH=src pytest -q +PYTHONPATH=src ruff check . +OPENSPEC_TELEMETRY=0 openspec validate --all --strict +PYTHONPATH=src python scripts/check_current_truth_surface.py +``` + +Keep these machine-checked markers in the lower evidence/status section: +`PARTIAL_SCHEMA_BENEFIT`, `14.65%`, `68.79%`, `DEVELOPMENT_ONLY_SPENT`, +`JSON type-strict`, and `strict exact remains canonical`. + +Explicitly reject overall model improvement, production readiness, +live-browser benchmark gain, DPO benefit, clean held-out generalization, and +released checkpoints/adapters. + +Use project facts and individual-contribution framing, never “we.” Keep each +paragraph to four rendered lines where practical. The completed-work status may +use `Portfolio-ready research and engineering project`. + +- [x] **Step 6: Check Chinese page size and order** + +Run: + +```bash +wc -l README.md +rg -n '^## ' README.md +``` + +Expected: 180-260 lines and approved heading order. + +### Task 3: Rewrite the English Homepage in Lockstep + +**Files:** +- Modify: `README_en.md` + +- [x] **Step 1: Mirror the section structure** + +Use these top-level headings: + +```markdown +## Project Value +## Key Results +## What I Built +## System Overview +## Why the Results Are Credible +## Repository Map +## Local Verification +## Project Status +## License +``` + +Mirror all Hero facts, labels, result rows, three modules, Mermaid nodes, +repository links, commands, and non-claims. + +- [x] **Step 2: Use this exact result boundary** + +```text +The Final SFT arm scored higher on semantic validity, task type, routing, and +confirmation accuracy, but lower on strict full-contract exact match; the +project therefore makes no overall model-quality improvement claim. +``` + +- [x] **Step 3: Preserve machine-checked vocabulary and page size** + +Include the same six lower-page truth markers as Chinese. Run: + +```bash +wc -l README.md README_en.md +rg -n '^## ' README.md README_en.md +``` + +Expected: both files are 180-260 lines with matching structure. + +### Task 4: Validate Facts, Links, Claims, and Public Safety + +**Files:** +- Verify: `README.md` +- Verify: `README_en.md` + +- [x] **Step 1: Derive every displayed result row from JSON, per file, and prove mutation sensitivity** + +```bash +PYTHONPATH=src python - <<'PY' +import json +from pathlib import Path + +root = Path.cwd() +comparison = json.loads((root / "reports/lockbox-v1/final-evaluation/comparison.json").read_text()) +metric_labels = [ + ("Semantic contract valid rate", "semantic_contract_valid_rate"), + ("Task type accuracy", "task_type_accuracy"), + ("Route accuracy", "route_accuracy"), + ("Confirmation accuracy", "confirmation_accuracy"), + ("Strict contract exact match", "contract_exact_match"), +] + +def pct(value: float) -> str: + return f"{value * 100:.2f}%" + +expected_rows = [] +for label, key in metric_labels: + base = pct(comparison["metrics"]["base"][key]) + final = pct(comparison["metrics"]["final_sft"][key]) + delta = f"{comparison['delta'][key] * 100:+.2f} pp" + expected_rows.append(f"| {label} | {base} | {final} | **{delta}** |") + +def validate_rows(text: str) -> None: + actual_rows = [ + line + for line in text.splitlines() + if any(line.startswith(f"| {label} |") for label, _ in metric_labels) + ] + assert actual_rows == expected_rows, (actual_rows, expected_rows) + +for name in ("README.md", "README_en.md"): + text = (root / name).read_text(encoding="utf-8") + validate_rows(text) + for expected_row in expected_rows: + mutated_row = expected_row.replace("%", "X", 1) + mutated_text = text.replace(expected_row, mutated_row, 1) + try: + validate_rows(mutated_text) + except AssertionError: + continue + raise AssertionError(f"{name}: metric-row mutation escaped validation") +print("README metric rows match authoritative JSON and mutations fail closed") +PY +``` + +Expected: `README metric rows match authoritative JSON and mutations fail closed`. +This validates all five complete Markdown rows independently in each README. +It does not authorize `1485 automated tests`; that fresh authorization is +deliberately deferred to the full-suite gate in Task 5. + +- [x] **Step 2: Source-bind the remaining displayed counts and boundaries** + +Read and cross-check the authoritative sources, rather than checking that bare +number tokens appear somewhere in the combined READMEs: + +```bash +PYTHONPATH=src python - <<'PY' +import json +from pathlib import Path + +root = Path.cwd() +load = lambda path: json.loads((root / path).read_text(encoding="utf-8")) +manifest = load("data/public-samples/manifest_public_sample.json") +lockbox = load("data/lockbox/lockbox-v1.manifest.json") +lockbox_result = load("reports/lockbox-v1/final-evaluation/comparison.json") +ablation = load("reports/public-sample/step-matched-canonical-slot-ablation/comparison.json") +control = load("reports/public-sample/step-matched-canonical-slot-ablation/control/training-summary.json") +treatment = load("reports/public-sample/step-matched-canonical-slot-ablation/treatment/training-summary.json") +challenge = load("reports/public-sample/copy-shadow-template-disjoint-challenge-v1/adapter-evaluation/challenge-evaluation-summary.json") +diagnosis = load("reports/public-sample/copy-shadow-false-trust-diagnosis/summary.json") +a100_tasks = (root / "openspec/changes/archive/2026-07-15-rerun-real-a100-sft-smoke-after-cli-fix-v1/tasks.md").read_text(encoding="utf-8") + +assert manifest["counts"] == {"seed_rows": 247, "sft_rows": 696, "dpo_pairs": 2100} +assert manifest["split_counts"]["train"] == 282 +assert manifest["files"]["sft_train"] == "data/public-samples/sft_train_public_sample.jsonl" +assert lockbox["row_count"] == lockbox["family_count"] == 120 +assert lockbox_result["row_count"] == 120 +assert ablation["step_matching"]["max_steps"] == 3132 +assert ablation["step_matching"]["not_token_matched"] is True +assert ablation["step_matching"]["unit"] == "optimizer_steps" +assert control["observed_optimizer_steps"] == 3132 +assert treatment["observed_optimizer_steps"] == 3132 +assert "112 changed adapter tensors out of 224" in a100_tasks +observed = challenge["observed_metrics"] +assert observed["source_absent_false_trust_count"] == 3 +assert observed["normalization_collision_false_trust_count"] == 6 +assert observed["partial_span_false_trust_count"] == 3 +assert challenge["row_count"] == 120 +# Secondary cross-check only; the README-linked challenge JSON above is primary. +assert diagnosis["mechanism_counts"]["SOURCE_ABSENT_SUBSTITUTION"] == 3 +assert diagnosis["mechanism_counts"]["NORMALIZATION_EQUIVALENCE_COLLISION"] == 6 +assert diagnosis["mechanism_counts"]["OVERLONG_SOURCE_SPAN"] == 3 + +zh = " ".join((root / "README.md").read_text(encoding="utf-8").split()) +en = " ".join((root / "README_en.md").read_text(encoding="utf-8").split()) +assert "247 条 seeds 构建为 696 条 SFT rows 与 2100 组 preference pairs" in zh +assert "247 seeds into 696 SFT rows and 2100 preference pairs" in en +assert "282 条 canonical train-only rows 绑定到正式 manifest 与 SHA-256" in zh +assert "Bound 282 canonical train-only rows to the formal manifest and SHA-256" in en +assert "每个 arm 固定 3132 optimizer steps" in zh +assert "exactly 3132 optimizer steps per arm" in en +assert "224 个 adapter tensors 中 112 个发生变化" in zh +assert "Of 224 adapter tensors, 112 changed" in en +assert "3 个 source-absent、6 个 normalization-collision 和 3 个 partial-span" in zh +assert "3 source-absent, 6 normalization-collision, and 3 partial-span" in en +assert "120 rows / 120 semantic families" in zh +assert "120 rows across 120 semantic families" in en +print("README source bindings verified") +PY +``` + +Expected: `README source bindings verified`. This binds the 282 train-only +rows, 3132 steps per arm, A100 112/224 tensor boundary, and frozen +120-row/120-family lockbox to their specific sources. The README-linked +challenge evaluation JSON is primary for source-absent=3, +normalization-collision=6, and partial-span=3; the later diagnosis summary is +only a secondary consistency cross-check. + +- [x] **Step 3: Run truth-surface, link, and public leak validation** + +Run: `PYTHONPATH=src python scripts/check_current_truth_surface.py` + +Expected: exit 0 with no broken link or missing truth marker. + +```bash +PYTHONPATH=src python - <<'PY' +from pathlib import Path +from voice2task.leak_scan import scan_paths + +result = scan_paths([Path("README.md"), Path("README_en.md")]) +assert result.ok, result.to_dict() +print("README leak scan: 0 findings") +PY +``` + +Expected: `README leak scan: 0 findings`. + +- [x] **Step 4: Run the reviewed-risk-block allowlist and complete image scans** + +Normalize each Markdown block by splitting on blank lines and collapsing +internal whitespace, then select every block containing a broad protected-risk +lexeme. Compare the selected block SHA-256 list, including order and occurrence +count, against the per-file reviewed allowlist below. Every selected block must +match and every allowlisted block must occur. Any protected wording addition, +deletion, edit, or reordering fails closed and requires human review plus an +explicit allowlist refresh; the gate uses only reviewed block identity. + +Scan all external-image forms, not only inline Markdown: `![alt](https://...)`, +reference/collapsed/shortcut image references whose definition resolves to +HTTP(S), and HTML ``. Expected: zero affirmative +overclaims and zero external image references in either README. + +```bash +PYTHONPATH=src python - <<'PY' +import hashlib +import re +from pathlib import Path + +protected_risk = re.compile( + r"overall|model(?:[- ]quality| improvement)|production|deploy(?:able|ment|ed|ing)?|" + r"held[- ]out|generalization|\bDPO\b|\bGRPO\b|live[- ]browser|checkpoint|adapter|" + r"safety|state[- ]of[- ]the[- ]art|\bSOTA\b|整体|模型(?:提升|质量)|生产|部署|" + r"可部署|泛化|真实浏览器|适配器|检查点|安全|最先进|业界领先", + re.IGNORECASE, +) + +expected_selected_counts = {"README.md": 13, "README_en.md": 14} +reviewed_hashes = { + "README.md": [ + "a73fd8f8320790aa6e12efc6efea8bf77854ccc69c387ba0cec1d39b7a3b92d8", + "5fd35967177356d03fc85fd68177809e26a241be2ffce2d1cdac16dc7ae33e7e", + "6af06b9f5e269098b0af8807fb291e9b17b667c610f0ef8698b2ac9cdf1f0da4", + "a79908c9855cb3df3406bb3485a819ce7d6367d4d28cb8bb0f48a4e64a01b7ed", + "2a073072a4983629f9442cbddf01bc691e1266f2f8a02d52f9fd15814759eaa5", + "de20ff9be05d4bc7595ffc63a3fc847ba26577506284426be371563a29d5b101", + "f7cbff9d7f4df1a89166d7afc5f0bcdfb56f30ccca42c7a369eedeabaf4a4228", + "ca54bc76b66911f2495c71a2c4832a1627db461d0059348fa1fb199faea2e4f9", + "667c5f05eaa2bbe2a4b5e5b7820c0503f12baa32b9c91c5cd85fe2a52350b23c", + "ca4ebe2673c4ff9266671c9511b2b80ecb4076c6f8f25bc56106e8512f2d0d74", + "03047b091b58fc4056d699a4c246be9d50f3d9e89e53820a6f62d433ec83d6a8", + "98905c2896db03f5ea66fc8231eb23a6f8d8567f3e0abbd9a88b3c88c7e817c3", + "3fa73da1f69222eed424e1baf1b2aa32aea701d817b0dd8238ed58d2b547a2ea", + ], + "README_en.md": [ + "19e440c84d196dc63bd456128d60dca455cd10e470f7551991f16731e3877fe0", + "a041fdeb9df9ab447c48088c45080f35def611636db9b8cbc0900a6d9502310a", + "6e42b912bc1f68ea0341e1c9900a8c112c107b0fa80078c5765b9a9495b4c807", + "a88636f9172a62b4a821bd1a75635149a47838435099237b9e2fce2eeb1a104b", + "b471ffd62104ad80cc4c073ac3a495b2bfddf3493dc129e2b6767601ac6f74fb", + "ce423082b203f54813dba88f03b318b17707fbc1f52491c84e7efcb4b2e5944c", + "05a7bdd42f390cacf6f3272ba01afa0bb2fbdf523a01770867452ceea5543fb6", + "09eb77c09846bc14617b17456f962baf6d083416a368b8e7adf74da833ccea28", + "028c726b946a6375a8de677675f9410524435ce2a50c85bd8b1bb5177fed0f7c", + "1309794515e3fca2304d7694fb9291a3615d7ed9a54033c0a8344063754b5051", + "d5b8d56728e3bfa50059cd479991b8d2d2ef4d7ec4e88184b1ea25330c8575f7", + "bb9e72ecafff93149115c737d791094275f310c8c1a3424d8504dc0ccb23392c", + "3a176ee1c650b0ca46cef62f03fe3f69a22494844e1285662ee380a0c127cee3", + "64c72a20541193ea1d50aae6c06de6c577d23886e32eb2959ee1868e9596b00b", + ], +} + +required_nonclaims = { + "README.md": { + "overall_and_sota": "不声称整体模型质量提升或 final SFT 的总体因果收益", + "production_and_safety": "不声称 production readiness 或 safety certification", + "live_browser": "不声称 live-browser benchmark gain", + "dpo": "不声称 DPO benefit，也没有以本结果授权 DPO/GRPO", + "heldout": "不声称 clean held-out generalization 或 naturalistic ASR generalization", + "release": "不声称已经发布 checkpoint、adapter 或可复现的私有模型产物", + }, + "README_en.md": { + "overall_and_sota": "No overall model improvement or overall causal benefit from Final SFT", + "production_and_safety": "No production readiness or safety certification", + "live_browser": "No live-browser benchmark gain", + "dpo": "No DPO benefit, and these results do not authorize DPO or GRPO", + "heldout": "No clean held-out generalization or naturalistic ASR generalization", + "release": "No released checkpoint, adapter, or reproducible private model artifact", + }, +} + +def normalized_blocks(text: str) -> list[str]: + return [ + re.sub(r"\s+", " ", raw.strip()) + for raw in re.split(r"\n[ \t]*\n", text.strip()) + if raw.strip() + ] + +def validate_reviewed_blocks(name: str, text: str) -> int: + fragments = required_nonclaims[name] + missing = {category: fragment for category, fragment in fragments.items() if fragment not in text} + assert not missing, (name, missing) + selected = [block for block in normalized_blocks(text) if protected_risk.search(block)] + actual_hashes = [hashlib.sha256(block.encode("utf-8")).hexdigest() for block in selected] + assert len(selected) == expected_selected_counts[name], (name, len(selected), selected) + assert actual_hashes == reviewed_hashes[name], (name, actual_hashes, reviewed_hashes[name]) + return len(selected) + +texts = { + name: Path(name).read_text(encoding="utf-8") + for name in ("README.md", "README_en.md") +} +selected_counts = { + name: validate_reviewed_blocks(name, text) + for name, text in texts.items() +} + +new_paragraph_probes = [ + ("README_en.md", "The project does not control a browser, but it is production ready."), + ("README.md", "项目不控制浏览器，但已经生产就绪。"), + ("README_en.md", "No released checkpoint exists, but DPO improvement was achieved."), + ("README_en.md", "The project is not a browser controller and is production ready."), + ("README.md", "项目不是浏览器控制器并且生产就绪。"), + ("README_en.md", "The checkpoint is publicly available and DPO yielded gains."), + ("README_en.md", "The system is state-of-the-art (SOTA)."), + ("README_en.md", "The project demonstrates overall model improvement."), + ("README.md", "项目实现了整体模型质量提升。"), + ("README_en.md", "Held-out generalization improved."), + ("README_en.md", "Live-browser performance improved."), + ("README_en.md", "Safety certification is complete."), +] + +def assert_rejected(name: str, mutated_text: str, label: str) -> None: + try: + validate_reviewed_blocks(name, mutated_text) + except AssertionError: + return + raise AssertionError(f"reviewed-risk mutation escaped: {label}") + +rejected = 0 +for name, probe in new_paragraph_probes: + assert_rejected(name, texts[name].rstrip() + "\n\n" + probe + "\n", probe) + rejected += 1 + +approved_fragment = required_nonclaims["README_en.md"]["overall_and_sota"] +mutated_existing = texts["README_en.md"].replace( + approved_fragment, + approved_fragment + " The project demonstrates overall model improvement.", + 1, +) +assert mutated_existing != texts["README_en.md"] +assert_rejected("README_en.md", mutated_existing, "append to approved protected paragraph") +rejected += 1 + +for name in ("README.md", "README_en.md"): + text = texts[name] + + inline_external = re.findall(r"!\[[^]]*\]\(\s*]*\bsrc\s*=\s*['\"]?https?://[^\s>'\"]+", + text, + re.IGNORECASE, + ) + image_refs = { + (reference or alt).strip().casefold() + for alt, reference in re.findall(r"!\[([^]]*)\]\[([^]]*)\]", text) + } + image_refs.update( + alt.strip().casefold() + for alt in re.findall(r"!\[([^]]+)\](?![\[(])", text) + ) + definitions = { + key.strip().casefold(): target + for key, target in re.findall( + r"^\[([^]]+)\]:\s*&1) +plan_status=$? +set -e +test "$plan_status" -eq 1 +test -z "$plan_check" + +git status --short --branch +git diff --name-only \ + origin/codex/materialize-manifest-bound-train-only-sft-v1...HEAD +git diff --name-only +git ls-files --others --exclude-standard + +{ + git diff --name-only \ + origin/codex/materialize-manifest-bound-train-only-sft-v1...HEAD + git diff --name-only + git ls-files --others --exclude-standard +} | LC_ALL=C sort -u +``` + +For the untracked plan, `git diff --no-index --check` returns status `1` because +the file differs from `/dev/null`; status `1` with empty diagnostic output means +the file is whitespace-clean. Status greater than `1`, status `0`, or any +diagnostic output is a gate failure. The normal and committed-range +`git diff --check` commands must both return status `0` with empty output. + +Expected union, exactly: + +```text +README.md +README_en.md +docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md +docs/superpowers/specs/2026-07-16-optimize-recruiter-facing-project-page-v1-design.md +``` + +No source, test, data, report, config, or metric file may differ. The committed +`base...HEAD` range sees only committed tracked paths; it cannot see unstaged +README edits or the untracked plan, so all three name sources must be reported +separately and then unioned. + +### Task 6: Commit, Push, and Open the Draft PR + +**Files:** +- Commit: `docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md` +- Commit: `README.md` +- Commit: `README_en.md` + +- [x] **Step 1: Commit the implementation plan** + +```bash +git add docs/superpowers/plans/2026-07-16-optimize-recruiter-facing-project-page-v1.md +git commit -m "Plan recruiter-facing README rewrite" +``` + +- [x] **Step 2: Commit the synchronized READMEs** + +```bash +git add README.md README_en.md +git commit -m "Optimize recruiter-facing project page" +``` + +- [x] **Step 3: Run final short verification** + +```bash +PYTHONPATH=src python scripts/check_current_truth_surface.py +git diff --check \ + origin/codex/materialize-manifest-bound-train-only-sft-v1...HEAD +git status --short --branch +``` + +Expected: checks pass and the worktree is clean. + +- [x] **Step 4: Push the branch** + +```bash +git push -u origin codex/optimize-recruiter-facing-project-page-v1 +``` + +- [x] **Step 5: Create a Draft PR with this topology and content** + +Create a public-safe PR body outside the repository, then run: + +```bash +gh pr create \ + --repo Raidriar7170/voice2task-posttraining \ + --base codex/materialize-manifest-bound-train-only-sft-v1 \ + --head codex/optimize-recruiter-facing-project-page-v1 \ + --draft \ + --title "Optimize recruiter-facing Voice2Task project page" \ + --body-file "/path/outside/repository/voice2task-recruiter-pr-body.md" +``` + +The body must contain Summary, Scope, Verification, and Result boundary +sections. It must say the PR is docs-only and stacked, list the passing checks, +and disclose that strict exact decreased from 1.67% to 0.83%, so no overall +model-improvement claim is made. + +- [x] **Step 6: Verify the Draft PR without merging** + +```bash +gh pr view \ + --repo Raidriar7170/voice2task-posttraining \ + --json number,title,isDraft,baseRefName,headRefName,url +``` + +Expected: `isDraft=true`, approved stacked base and head names, and no merge. + +- [ ] **Step 7: Deliver the evidence-backed handoff without changing settings** + +The final response must include the bilingual README structure, every displayed +metric and its authoritative file, any broken links repaired, every validation +command and result, `git diff --check`, final `git status --short`, branch, and +Draft PR URL. + +Recommend, but do not apply, this repository description: + +```text +Chinese voice/ASR to structured browser task contracts with Qwen2.5-7B LoRA post-training, strict evaluation, and evidence-first ML engineering. +``` + +Recommend, but do not apply, these topics: + +```text +llm qwen lora sft speech asr structured-output model-evaluation machine-learning pytorch +``` diff --git a/docs/superpowers/specs/2026-07-16-optimize-recruiter-facing-project-page-v1-design.md b/docs/superpowers/specs/2026-07-16-optimize-recruiter-facing-project-page-v1-design.md new file mode 100644 index 0000000..249e871 --- /dev/null +++ b/docs/superpowers/specs/2026-07-16-optimize-recruiter-facing-project-page-v1-design.md @@ -0,0 +1,279 @@ +# Voice2Task Recruiter-Facing Project Page V1 Design + +## Status + +- Approved by the user on 2026-07-16. +- Change type: documentation-only GitHub project-page optimization. +- Working branch: `codex/optimize-recruiter-facing-project-page-v1`. +- Stacked base: `codex/materialize-manifest-bound-train-only-sft-v1` at + `38db124`. +- Intended Draft PR base: `codex/materialize-manifest-bound-train-only-sft-v1`. +- Merge is explicitly out of scope. + +## Objective + +Rewrite the Chinese and English repository homepages so an HR reviewer, +interviewer, or machine-learning engineer can understand within 30 seconds: + +1. the problem Voice2Task solves; +2. the technically substantive work completed; +3. the real quantitative results, including the negative result; +4. why the evidence is credible; and +5. where to inspect code, experiments, and evidence. + +The page must lead with project value and measured results instead of internal +phase names. It must remain conservative enough for public recruiting use. + +## Delivery Topology + +The required 282-row manifest-bound train-only artifact, the latest bounded +real A100 smoke evidence, and its CLI fix are not present on `origin/main`. +They are available on the approved stacked base. + +The project-page branch therefore starts from that base, and its Draft PR will +target that base. Relative to the Draft PR base, the change remains docs-only. +After the prerequisite lineage reaches `main`, the Draft PR can be retargeted +without changing the README content. + +The project-page PR must not absorb, rewrite, or duplicate prerequisite code, +data, metrics, or OpenSpec history. + +## Scope + +### Files to change + +- `README.md` +- `README_en.md` +- this approved design record + +Purely presentational local assets are permitted only if required. The planned +design uses GitHub-native Markdown and Mermaid, so no asset is currently needed. + +### Explicitly excluded + +- model or training logic; +- prediction or evaluation logic; +- dataset rows or manifest content; +- metrics, reports, historical evidence, or experimental conclusions; +- training, prediction, evaluation, DPO, or GRPO execution; +- repository settings, description, or topics mutation; +- merge, release, deploy, checkpoint publication, or adapter publication. + +## Sources of Truth + +All public claims must be derived from the approved stacked base. The primary +authority order for this page is: + +1. `reports/lockbox-v1/final-evaluation/comparison.json` +2. `data/public-samples/manifest_public_sample.json` +3. `reports/public-sample/step-matched-canonical-slot-ablation/comparison.json` +4. `reports/public-sample/slot-error-mechanism-analysis/summary.json` +5. `reports/public-sample/EVIDENCE_INDEX.md` +6. `docs/current-status.md` +7. `openspec/specs/supervised-contract-tuning/spec.md` +8. `openspec/specs/voice2task-dataset-preparation/spec.md` +9. the archived July 2026 A100 smoke, CLI-fix, and train-only changes +10. `CONTEXT.md` + +When existing README wording conflicts with these files, the README must be +corrected without mutating the evidence. + +## Verified Facts for the Homepage + +### Data and infrastructure + +- Model path: Qwen2.5-7B-Instruct with LoRA. +- Public dataset: 247 seed rows, 696 SFT rows, and 2,100 preference pairs. +- Bound training derivative: 282 canonical train-only SFT rows. +- Frozen evaluation: 120 rows across 120 semantic families. +- Step-matched ablation: 3,132 optimizer steps per arm; not token matched. +- Fresh approved-base test baseline: 1,485 passing tests. +- A bounded private A100 smoke completed exactly one optimizer step on two + training rows and verified non-zero finite adapter updates. +- The adapter and checkpoint remain private and are not released. + +### Frozen lockbox results + +The compact result table will show decimal scores and percentage-point deltas: + +| Metric | Base | Final SFT | Delta | +| --- | ---: | ---: | ---: | +| Semantic contract valid rate | 82.50% | 86.67% | +4.17 pp | +| Task type accuracy | 79.17% | 85.83% | +6.67 pp | +| Route accuracy | 80.00% | 85.83% | +5.83 pp | +| Confirmation accuracy | 70.83% | 79.17% | +8.33 pp | +| Strict contract exact match | 1.67% | 0.83% | -0.83 pp | + +The interpretation immediately below the table must state that semantic +structure, task classification, routing, and confirmation judgment improved, +while strict full-contract exact match did not. The repository therefore does +not claim overall model-quality improvement. + +Confirmation accuracy must never be described as overall accuracy. + +## Homepage Information Architecture + +Both language versions will use the same section order and evidence links. +Wording may be idiomatic rather than sentence-for-sentence literal. + +### 1. Hero + +- Title: `Voice2Task Post-Training`. +- One-sentence input/output positioning. +- One short sentence covering data, assistant-only SFT, gold-free prediction, + strict evaluation, step-matched ablation, and error attribution. +- A plain-text technical stack line: Python, PyTorch, Transformers, TRL, PEFT, + Qwen2.5-7B, and LoRA. +- Four to six compact highlight items. Qwen/LoRA belongs in the stack line so + the numeric highlights can prioritize 247 -> 696, 282, 2,100, 120/120, + bounded A100 training, and 1,485 tests. + +The page will not use externally hosted badge images. This avoids CI-like badge +implications and external image dependencies. + +### 2. Key Results + +Use the five-row frozen-lockbox table above, followed by the explicit negative +result and links to the comparison JSON and current status. + +### 3. What I Built + +Keep exactly three high-value modules: + +1. Data and post-training pipeline: Qwen2.5-7B LoRA, assistant-only loss, + public data construction, manifest/SHA-256 binding, and private offline A100 + operation. +2. Trustworthy evaluation and experimental design: frozen one-look lockbox, + fixed prompt/decoder/evaluator, strict layered metrics, and the 3,132-step + matched A/B. +3. Error attribution and safety boundaries: core-slot failure concentration, + mixed slot representation, observe-only provenance hooks, false-trust + findings, and fail-closed execution gates. + +Each module will use at most two short paragraphs or bullets. + +### 4. System Overview + +Use one GitHub-native Mermaid flowchart: + +`Chinese Voice / ASR -> Dataset & Contract Validation -> Qwen2.5-7B LoRA SFT +-> Gold-free Prediction -> JSON / Schema Guard -> Strict Contract Evaluation +-> Error Analysis & Provenance Shadow Audit` + +The diagram must not show browser execution, production deployment, or online +automation. + +### 5. Evaluation Design + +Use no more than five bullets covering: + +- family-aware splits and split-integrity audit; +- frozen one-look lockbox; +- recursive JSON type-strict equality; +- step-matched experiment with fixed prompt, decoder, schema guard, and + evaluator; and +- `DEVELOPMENT_ONLY_SPENT` public dev/test plus symmetric reporting of positive + and negative metrics. + +### 6. Repository Map + +Use a compact table with verified relative links to: + +- public data manifest; +- train-only SFT artifact; +- training CLI; +- evaluation CLI; +- lockbox comparison; +- step-matched ablation; +- slot-error analysis; +- evidence index; +- current OpenSpec specifications; and +- tests. + +Every linked path must exist on the Draft PR base and final branch. + +### 7. Reproducibility + +Only list public-safe commands that do not download a model or initiate private +training: + +```bash +PYTHONPATH=src pytest -q +PYTHONPATH=src ruff check . +OPENSPEC_TELEMETRY=0 openspec validate --all --strict +PYTHONPATH=src python scripts/check_current_truth_surface.py +``` + +Mention that real training requires an ignored private config and local model +weights. Do not expose hosts, SSH commands, paths, tokens, configs, raw logs, +or adapter locations. + +### 8. Project Status + +Separate completed engineering/evidence work from explicit non-claims. + +Completed work may include dataset construction, the LoRA training path, +bounded real A100 smoke, strict evaluation, frozen lockbox, step-matched A/B, +slot-error analysis, and provenance shadow audit. + +The page must explicitly reject claims of: + +- overall model improvement; +- production readiness or safety certification; +- live-browser benchmark improvement; +- DPO benefit; +- clean held-out generalization; and +- released checkpoints or adapters. + +`Portfolio-ready research and engineering project` is permitted if status text +needs a concise summary. + +## Writing and Layout Constraints + +- Chinese README is the primary recruiting page; English matches its structure. +- Each file should remain approximately 180 to 260 lines. +- No paragraph should exceed four rendered lines where practical. +- Use compact tables and short bullets; avoid uninterrupted prose blocks. +- Use project facts and first-person contribution framing, never “we.” +- All repository links must be relative. +- The result table and code blocks must remain usable on mobile widths. +- Do not add an unverified CI badge or external image dependency. +- Do not expose private infrastructure details. + +## Validation Design + +Before publication, run and report: + +1. Markdown structure and Mermaid fence checks. +2. Relative-link existence checks for both READMEs. +3. Public leak and prohibited-private-detail scans. +4. Existing README/evidence-surface tests. +5. Full `PYTHONPATH=src pytest -q`. +6. `PYTHONPATH=src ruff check .`. +7. `OPENSPEC_TELEMETRY=0 openspec validate --all --strict`. +8. `PYTHONPATH=src python scripts/check_current_truth_surface.py`. +9. `git diff --check` and `git status --short`. +10. A full claim scan for unsupported overall-improvement, production, + held-out-generalization, DPO, live-browser, checkpoint-release, and + safety-certification wording. + +The numeric table will also be checked directly against the lockbox comparison +JSON, and the dataset highlights against the current manifest. + +## Acceptance Criteria + +- A recruiter can identify the input, output, model, data scale, result, and + result limitation from the first screen and Key Results section. +- A technical reviewer can find the training, evaluation, experimental-design, + and error-analysis evidence without searching the repository. +- Chinese and English versions contain the same facts and section order. +- Every metric and dataset count matches an authoritative committed artifact. +- The strict exact-match regression is visible, not buried or softened. +- The Draft PR diff against its approved stacked base contains documentation + changes only. +- The branch is pushed and a Draft PR is opened, but not merged. + +## Repository Metadata Recommendation + +The final handoff may recommend a repository description and topic list, but +must not change repository settings without separate authorization.