Skip to content

Optimize recruiter-facing Voice2Task project page #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Raidriar7170 wants to merge 5 commits into
base: codex/materialize-manifest-bound-train-only-sft-v1
Choose a base branch
from
Draft

Optimize recruiter-facing Voice2Task project page #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
06770fa
Document recruiter-facing README design
Raidriar7170 Jul 15, 2026
5aa4fea
Plan recruiter-facing README rewrite
Raidriar7170 Jul 15, 2026
67982a4
Optimize recruiter-facing project page
Raidriar7170 Jul 15, 2026
c53d0dd
Record project page publication
Raidriar7170 Jul 15, 2026
4af79ce
Remove host-specific path from publication plan
Raidriar7170 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
338 changes: 160 additions & 178 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,219 +2,201 @@

[中文](README.md) | [English](README_en.md)

![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![Status](https://img.shields.io/badge/status-final%20lockbox%20reported-0b6e69)
![Model](https://img.shields.io/badge/model-Qwen2.5--7B%20LoRA-6f42c1)
![Scope](https://img.shields.io/badge/scope-evidence--first-f59e0b)
中文语音 / ASR 指令到可验证 Browser Task Contract 的大模型后训练与可信评测系统。

Voice2Task 是一个中文 spoken command / ASR transcript 到浏览器任务合约的
post-training 项目。
任务不是控制浏览器，而是把用户口语命令转换成严格的
Browser Task Contract JSON，供后续浏览器 agent 决定搜索、打开 URL、
填写表单、抽取页面信息、澄清或拒绝高风险动作。
基于 Qwen2.5-7B-Instruct + LoRA，覆盖数据构建、assistant-only SFT、
gold-free 推理、严格合约评测、等步数消融与错误归因。

## Recruiter Summary
Python · PyTorch · Transformers · TRL · PEFT · Qwen2.5-7B · LoRA

| Area | What this project built |
| --- | --- |
| Problem | 中文语音/ASR 浏览器命令 -> schema-valid browser task contract JSON |
| Model pipeline | Qwen2.5-7B-Instruct + LoRA SFT；adapter 私有，不随仓库发布 |
| Data/training | 247 seeds / 696 SFT rows / 2,100 preference pairs；public dev/test 为 `DEVELOPMENT_ONLY_SPENT`；final SFT 只使用既有训练数据 |
| Prompt/eval hardening | 统一 gold-free prompt policy `unified_gold_free_v1`；严格 JSON parse -> strict schema -> semantic contract -> exact match 分层验证 |
| Frozen evaluation | 120-row `lockbox-v1`，120 semantic families，manifest frozen，one-look final evaluation |
| Result boundary | final SFT 没有提升 strict contract exact match；`no overall model improvement claim` |

## Final Lockbox v1 Result

Frozen protocol:
`lockbox_hash=06114cf3ad6029930284af5f2245fb2c4a8174fd35c6a1107f4c73482b555b33`,
prompt policy `unified_gold_free_v1`, greedy decoding,
schema guard + one schema retry, strict evaluator,
two pre-registered arms only.

| Metric | Base Qwen2.5-7B | Final SFT adapter | Delta |
| 247 seeds -> 696 SFT rows | 282 manifest-bound train-only rows | 2100 preference pairs |
| --- | --- | --- |
| 120 rows / 120 families frozen lockbox | 真实私有 A100 bounded smoke | 1485 automated tests |

## 项目价值

Voice2Task 不控制浏览器。它把中文口语命令或 ASR transcript 转换成严格、
机器可校验的 Browser Task Contract，供下游系统决定搜索、打开 URL、
填写表单、抽取页面信息、澄清请求或拒绝高风险动作。

项目的核心不是生成一段“看起来合理”的文本，而是让模型输出可解析、
可做 schema 检查、可逐字段比较并能追溯失败原因的结构化任务合约。

这使后训练目标、推理输出和评测口径落在同一份明确契约上，
同时把浏览器执行、线上自动化与生产部署留在项目边界之外。

## 核心结果

最终结果来自 120 rows / 120 semantic families 的冻结 one-look lockbox。
Base 与 Final SFT 使用预注册 prompt、greedy decode、schema guard 和严格评测器。

| 冻结 lockbox 指标 | Base Qwen2.5-7B | Final SFT | 变化 |
| --- | ---: | ---: | ---: |
| `contract_exact_match` | 0.0167 | 0.0083 | -0.0083 |
| `semantic_contract_valid_rate` | 0.8250 | 0.8667 | +0.0417 |
| `task_type_accuracy` | 0.7917 | 0.8583 | +0.0667 |
| `route_accuracy` | 0.8000 | 0.8583 | +0.0583 |
| `confirmation_accuracy` | 0.7083 | 0.7917 | +0.0833 |
| `strict_schema_valid_rate` | 1.0000 | 0.9833 | -0.0167 |
| `slot_f1` | 0.0417 | 0.0500 | +0.0083 |
| `slot_f1_soft` | 0.3783 | 0.3867 | +0.0084 |

Interpretation:

- Final SFT did **not** improve strict contract exact match on the frozen lockbox.
- Final SFT did improve several semantic/channel metrics:
`semantic_contract_valid_rate +0.0417`, `task_type_accuracy +0.0667`,
`route_accuracy +0.0583`, `confirmation_accuracy +0.0833`.
- This is aggregate-only one-look evidence. Public reports do not include row-level failure analysis；不能据此推断 row-level failure cause、natural-ASR generalization 或 overall SFT causal effect。

Evidence links:

- [Final comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json)
- [Final run card](reports/lockbox-v1/final-evaluation/run-card.json)
- [Final comparison Markdown](reports/lockbox-v1/final-evaluation/comparison.md)
- [Current status and evidence](docs/current-status.md)
- [Lockbox protocol](docs/lockbox.md)

## Explicit Non-Claims

This repository does **not** claim:

- overall model improvement from final SFT (`no overall model improvement claim`);
- production readiness;
- safety readiness;
- executable browser quality;
- DPO success;
- adapter/checkpoint release;
- live-browser benchmark improvement.

The strongest supported claim is narrower:
under a frozen 120-row lockbox and a gold-free strict evaluator,
final SFT improved several semantic/channel aggregate metrics
but reduced strict full-contract exact match.

## Repository Role

| This repo is | This repo is not |
| --- | --- |
| A speech/ASR-to-contract post-training evidence repository | A generic chat fine-tuning project |
| A strict JSON contract generation and evaluation pipeline | A GUI action policy or browser controller |
| A public-safe SFT/DPO data, training, prediction, and evaluation workflow | A checkpoint or adapter release |
| A place where negative, blocked, and superseded evidence stays auditable | A success story built by deleting inconvenient results |
| Semantic contract valid rate | 82.50% | 86.67% | **+4.17 pp** |
| Task type accuracy | 79.17% | 85.83% | **+6.67 pp** |
| Route accuracy | 80.00% | 85.83% | **+5.83 pp** |
| Confirmation accuracy | 70.83% | 79.17% | **+8.33 pp** |
| Strict contract exact match | 1.67% | 0.83% | **-0.83 pp** |

## Method Overview
Final SFT arm 在 semantic validity、task type、route 与 confirmation 指标上的
聚合得分高于 Base arm，但 strict full-contract exact 得分更低。因此项目不声称
整体模型质量提升，也不把局部指标差异包装成完整合约生成能力已经改善。

1. Build public-safe Voice2Task data from seed traces into SFT and preference rows.
2. Render Qwen chat prompts with no gold contract in prediction prompts.
3. Train LoRA SFT adapters on existing training data only.
4. Decode greedily with `max_new_tokens=256`, schema guard enabled, and at most one schema retry.
5. Score with strict layered metrics: JSON parse, strict schema validity,
semantic contract validity, exact match, slot-level metrics,
route/task/confirmation/safety metrics.
6. Freeze lockbox rows and manifest before the final one-look evaluation.
权威结果见 [lockbox comparison JSON](reports/lockbox-v1/final-evaluation/comparison.json)；
完整解释与限制见 [current status](docs/current-status.md)。

## Quick Start
## 我完成的核心工作

Install local tooling:
### 1. 数据与后训练流水线

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,dataset]'
```
- 基于 Qwen2.5-7B-Instruct + LoRA，使用 PyTorch、Transformers、TRL 与 PEFT
实现 assistant-only loss 后训练路径；把 247 条 seeds 构建为 696 条 SFT rows
与 2100 组 preference pairs，并保留严格的数据和合约校验。

Rebuild and validate the committed public sample:
- 将 282 条 canonical train-only rows 绑定到正式 manifest 与 SHA-256，避免训练
输入悄然漂移。真实训练仅接受 ignored private config、本地模型权重和私有 A100；
adapter/checkpoint 不公开，也不允许运行时下载替代本地模型。

```bash
PYTHONPATH=src python -m voice2task.cli.data build-public \
--seed data/public-samples/seed_traces.jsonl \
--output data/public-samples

PYTHONPATH=src python -m voice2task.cli.data validate \
--sft data/public-samples/sft_public_sample.jsonl \
--dpo data/public-samples/dpo_public_sample.jsonl \
--manifest data/public-samples/manifest_public_sample.json \
--public

PYTHONPATH=src python -m voice2task.cli.data audit-splits \
--seed data/public-samples/seed_traces.jsonl \
--sft data/public-samples/sft_public_sample.jsonl \
--dpo data/public-samples/dpo_public_sample.jsonl \
--manifest data/public-samples/manifest_public_sample.json \
--output reports/public-sample/split-integrity-audit
```
[归档的 public-safe A100 smoke 证据](openspec/changes/archive/2026-07-15-rerun-real-a100-sft-smoke-after-cli-fix-v1/tasks.md)
记录了恰好一次 launch、一个 optimizer step、两条训练 rows；224 个 adapter tensors
中 112 个发生变化且全部有限。它只证明 bounded training path 与参数更新可行。

Run local baselines and metrics:
### 2. 可信评测与实验设计

```bash
PYTHONPATH=src python -m voice2task.cli.eval baseline \
--gold data/public-samples/sft_public_sample.jsonl \
--output reports/public-sample/rule_baseline_predictions.jsonl

PYTHONPATH=src python -m voice2task.cli.eval metrics \
--gold data/public-samples/sft_public_sample.jsonl \
--predictions reports/public-sample/rule_baseline_predictions.jsonl \
--output reports/public-sample
```
- 构建 120 rows / 120 families 的冻结 one-look lockbox；固定 gold-free prompt、
greedy decode、schema guard 与 strict evaluator，分层衡量 JSON、schema、语义、
路由、确认和 strict full-contract exact。

Dry-run training metadata export remains available, but real heavy training is gated by explicit config:
- 设计 Control/Treatment 等步数消融，每个 arm 固定 3132 optimizer steps，
明确不是 token matched。实验没有调参评测器、语义放宽、预测修复或选择性报告，
正向指标与 strict exact 回退同时保留。

```bash
PYTHONPATH=src python -m voice2task.cli.train sft \
--config configs/sft-dev.json \
--manifest data/public-samples/manifest_public_sample.json \
--output-dir reports/public-sample/sft-dry-run \
--dry-run

PYTHONPATH=src python -m voice2task.cli.train dpo \
--config configs/dpo-dev.json \
--manifest data/public-samples/manifest_public_sample.json \
--output-dir reports/public-sample/dpo-dry-run \
--dry-run
```
### 3. 错误归因与安全边界

## Metric Interpretation Boundaries
- 错误分析显示 68.79% 的 V1 strict failures 集中在 core slots；据此区分
copy-backed、bounded structured 与 unresolved 表示。实现的
[observe-only provenance shadow hook](reports/public-sample/copy-backed-prediction-shadow-hook/summary.json)
默认关闭，不改预测、不参与执行决策。

`contract_exact_match` is a hard full-contract exact-match metric. Future evaluator runs use recursive JSON type-strict equality: booleans, integers, and floating-point values are distinct; object key order and serialization whitespace are ignored; array order is preserved; non-finite or non-JSON values fail closed. Historical metrics were not re-scored.
`normalized_command` string-mismatch diagnostics are explanatory row-level
evidence only: they do not relax, normalize, semantically score, repair, replace,
or re-score predictions, and they do not automatically mark Chinese phrase differences such as `搜索/查询` or `明天的天气/明天天气` as equivalent.
- [template-disjoint challenge 的已观测结果](reports/public-sample/copy-shadow-template-disjoint-challenge-v1/adapter-evaluation/challenge-evaluation-summary.json)
记录 3 个 source-absent、6 个 normalization-collision 和 3 个 partial-span
false-trust。它是 adversarial verifier fixture 证据，不是自然语言 ASR 或模型质量证据。

`normalized_command` gold targets are canonical Chinese intent phrases, not
verbatim transcripts or ASR text. This is target-writing guidance for SFT/DPO
data and prompts, not evaluator-side normalization, semantic-equivalence
scoring, prediction repair, or re-scoring.
训练与证据导出还使用 fail-closed GPU/path/data/adapter/public-output gates；
任一模型身份、数据绑定、输出边界或 smoke 后置条件不满足时都会停止。

### Normalized Command Target Policy
## 系统流程

```mermaid
flowchart LR
A["Chinese Voice / ASR"] --> B["Dataset & Contract Validation"]
B --> C["Qwen2.5-7B LoRA SFT"]
C --> D["Gold-free Prediction"]
D --> E["JSON / Schema Guard"]
E --> F["Strict Contract Evaluation"]
F --> G["Error Analysis & Provenance Shadow Audit"]
```

流程终点是严格评测与错误审计，不包含浏览器执行、线上部署或生产自动化。

Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR
text. Representative forms include `搜索北京明天天气`, `打开示例网站`,
`填写邮箱并确认`, and `拒绝代替用户付款`. This is authoring guidance, not
evaluator-side normalization or semantic-equivalence scoring.
## 为什么结果可信

## Evidence Archive
- 数据切分采用 family-aware 规则，并通过
[split integrity audit](reports/public-sample/split-integrity-audit/summary.json)
显式记录跨 split 风险，而不是默认宣称数据天然干净。

Longer-running internal evidence remains documented below the headline result:
- 最终评测使用 120 rows / 120 families 的冻结 one-look lockbox；
只公开聚合结果，不使用 lockbox 逐行错误做再次调优。

- Public split integrity: the current 282/207/207 dev/test boundary is
`DEVELOPMENT_ONLY_SPENT`, not blind, independent, leakage-free, or
final-generalization evidence. The audit preserves historical rows and
metrics; lockbox-v1 remains the distinct frozen one-look aggregate boundary.
- Contract V2 projection: `PARTIAL_SCHEMA_BENEFIT`; derived-field-only strict
failures are 14.65%, normalized-command-only strict failures are 14.65%, and
core slot failures remain 68.79% of V1 strict failures. This is useful
schema-burden evidence, not model-quality evidence.
- Copy-backed verification and shadow mode: observe-only provenance/interface evidence, not runtime enforcement.
- Copy-shadow template-disjoint challenge v1: adversarial verifier fixture, not a naturalistic language benchmark.
- Earlier step-matched SFT ablations: mixed/inconclusive; no stable broad canonical-slot benefit.
- recursive `JSON type-strict` equality 是未来 evaluator runs 的 exact 边界；
本页展示的历史 lockbox metrics 未重新计分。 Object key order and serialization
whitespace 可忽略，数组顺序保留，布尔、整数与浮点数区分，异常值 fail closed。

See [current status](docs/current-status.md) and
[public evidence index](reports/public-sample/EVIDENCE_INDEX.md) for the
complete archived map.
- step-matched 实验固定 prompt、decoder、schema guard 与 evaluator，
每个 arm 使用 3132 optimizer steps，并明确披露 not token matched。

## A100 Boundary
- public dev/test 标记为 `DEVELOPMENT_ONLY_SPENT`；它们不是干净独立 held-out。
页面同时报告局部正向变化和 strict exact 负向结果。

GPU-heavy training and prediction are designed for a private A100 development
machine.
Public repo artifacts intentionally omit checkpoints, LoRA adapters, raw logs,
remote caches, private corpus rows, hostnames, SSH details, credentials,
private paths, private override configs, and production-readiness claims.
## 仓库导航

## Validation
| 入口 | 用途 |
| --- | --- |
| [Public manifest](data/public-samples/manifest_public_sample.json) | 247 / 696 / 2100 计数、split 与文件哈希 |
| [Train-only SFT artifact](data/public-samples/sft_train_public_sample.jsonl) | 282 条 manifest-bound canonical train rows |
| [Training CLI](src/voice2task/cli/train.py) | SFT preflight、训练与 gold-free prediction 入口 |
| [Evaluation CLI](src/voice2task/cli/eval.py) | 严格分层评测入口 |
| [Lockbox comparison](reports/lockbox-v1/final-evaluation/comparison.json) | Base 与 Final SFT 冻结聚合结果 |
| [Step-matched ablation](reports/public-sample/step-matched-canonical-slot-ablation/comparison.json) | 3132-step Control/Treatment 对照与边界 |
| [Slot-error summary](reports/public-sample/slot-error-mechanism-analysis/summary.json) | core-slot 瓶颈与表示分析 |
| [Evidence index](reports/public-sample/EVIDENCE_INDEX.md) | CURRENT、HISTORICAL、BLOCKED 等证据地图 |
| [Training spec](openspec/specs/supervised-contract-tuning/spec.md) / [dataset spec](openspec/specs/voice2task-dataset-preparation/spec.md) | 当前 OpenSpec 训练与数据契约 |
| [Tests](tests) | 数据、训练、评测、证据与边界回归测试 |

Useful local checks:
## 本地验证

以下命令只验证公开仓库，不下载模型，也不启动训练：

```bash
PYTHONPATH=src pytest -q
PYTHONPATH=src ruff check src tests
PYTHONPATH=src ruff check .
OPENSPEC_TELEMETRY=0 openspec validate --all --strict
PYTHONPATH=src python scripts/check_current_truth_surface.py
git diff --check
```

真实训练需要 ignored private config 与本地模型权重；仓库不公开主机、SSH、路径、
token、私有配置、原始日志或 adapter 位置。

## 项目状态

**已完成：Portfolio-ready research and engineering project。**

- 完成公开数据构建、manifest/SHA-256 绑定与 assistant-only LoRA SFT 路径。
- 完成 fail-closed 私有 A100 bounded smoke，证明单步训练和 adapter 参数更新。
- 完成 gold-free 推理、strict evaluator 与冻结 one-look lockbox 聚合评测。
- 完成 3132-step matched A/B、slot-error attribution 与 mixed representation 设计。
- 完成 observe-only provenance shadow audit 与 verifier false-trust challenge。
- 当前自动化基线为 1485 tests；adapter 与 checkpoint 仍为私有且未发布。

**证据边界：**

- Contract V2 离线投影结论是 `PARTIAL_SCHEMA_BENEFIT`，不是模型质量结论。
- derived-field-only strict failures 为 14.65%；这只说明部分 schema burden。
- core slot failures 仍占 68.79%，是完整合约 strict failure 的主要瓶颈。
- public dev/test 状态是 `DEVELOPMENT_ONLY_SPENT`，不构成 clean held-out evidence。
- 后续 exact 采用 `JSON type-strict` 递归比较；历史指标没有重算。
- `strict exact remains canonical`：局部指标不能替代完整合约严格一致性。

### Metric Interpretation Boundaries

`contract_exact_match` is a hard full-contract exact-match metric. Future runs use
recursive `JSON type-strict` equality: object key order and serialization whitespace
are ignored, array order is preserved, and non-finite or non-JSON values fail closed.

`normalized_command` string-mismatch diagnostics are explanatory row-level evidence only.
They do not relax, normalize, semantically score, repair, replace, or re-score predictions.
They do not automatically mark Chinese phrase differences such as `搜索/查询` or
`明天的天气/明天天气` as equivalent.

### Normalized Command Target Policy

Targets use canonical Chinese intent phrases, not verbatim transcripts or ASR text.
Representative forms include `搜索北京明天天气`、`打开示例网站`、`填写邮箱并确认`
和 `拒绝代替用户付款`。 This is authoring guidance only: not evaluator-side normalization,
semantic-equivalence scoring, prediction repair, or rescoring.

**明确不声称：**

- 不声称整体模型质量提升或 final SFT 的总体因果收益。
- 不声称 production readiness 或 safety certification。
- 不声称 live-browser benchmark gain；项目没有控制浏览器。
- 不声称 DPO benefit，也没有以本结果授权 DPO/GRPO。
- 不声称 clean held-out generalization 或 naturalistic ASR generalization。
- 不声称已经发布 checkpoint、adapter 或可复现的私有模型产物。

项目展示的是可审计的数据、后训练、严格评测和负结果处理能力，
不是把一次 bounded smoke 或若干正向 aggregate metrics 包装成上线能力。

## License

本项目采用 [MIT License](LICENSE)。
Loading