Skip to content

Harden private A100 SFT smoke preflight #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Raidriar7170 wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Harden private A100 SFT smoke preflight #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
41c53b8
Harden private A100 SFT smoke preflight
Raidriar7170 Jul 13, 2026
3a4bf38
Harden private A100 SFT smoke preflight
Raidriar7170 Jul 14, 2026
f214b9b
Fix container-safe SFT GPU occupancy probe
Raidriar7170 Jul 14, 2026
df25648
fix training CLI single-JSON output
Raidriar7170 Jul 15, 2026
d86475b
Record successful real A100 SFT smoke rerun
Raidriar7170 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions configs/sft-a100-smoke.example.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"base_model_public_id": "Qwen/Qwen2.5-7B-Instruct",
"base_model_runtime_path": "<private-local-qwen2.5-7b-instruct-path>",
"allow_heavy_training": false,
"local_files_only": true,
"trust_remote_code": false,
"dtype": "bfloat16",
"torch_dtype": "bfloat16",
"dataset_split": "train",
"dataset_manifest_id": "public-sample-20260619T090925Z",
"max_train_rows": 2,
"max_steps": 1,
"per_device_train_batch_size": 1,
"gradient_accumulation_steps": 1,
"max_seq_length": 1024,
"bf16": true,
"fp16": false,
"tf32": true,
"gradient_checkpointing": true,
"use_cache": false,
"low_cpu_mem_usage": true,
"save_strategy": "no",
"logging_steps": 1,
"seed": 42,
"report_to": [],
"min_free_disk_gib": 20,
"output_root": "<approved-private-output-root>",
"lora": {
"r": 8,
"alpha": 16,
"dropout": 0.05,
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj"
]
}
}
93 changes: 93 additions & 0 deletions docs/human-briefs/2026-07-13-enable-private-a100-sft-smoke-v1.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Voice2Task Private A100 SFT Smoke V1 · 人类简报</title>
<style>
:root { color-scheme: light; --ink:#17202a; --muted:#4b5563; --accent:#9a3412; --line:#e5e7eb; --paper:#fff; }
* { box-sizing:border-box; }
body { margin:0; color:var(--ink); background:var(--paper); font:16px/1.7 -apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; }
header,main,footer { width:min(760px,calc(100% - 2rem)); margin:auto; }
header { padding:4rem 0 2rem; border-bottom:1px solid var(--line); }
main { padding-bottom:3rem; }
section { padding:2rem 0; border-bottom:1px solid var(--line); }
h1,h2 { line-height:1.2; text-wrap:balance; }
h1 { margin:.4rem 0 1rem; font-size:clamp(2rem,7vw,3.5rem); letter-spacing:-.035em; }
h2 { margin:0 0 .8rem; }
.eyebrow,.status { color:var(--accent); font-size:.78rem; font-weight:750; letter-spacing:.08em; text-transform:uppercase; }
.lede { max-width:62ch; color:var(--muted); font-size:1.08rem; }
code { padding:.08em .3em; background:#f3f4f6; overflow-wrap:anywhere; }
ul { padding-left:1.25rem; }
li+li { margin-top:.45rem; }
.callout { padding:1rem; border-left:4px solid var(--accent); background:#fff7ed; }
dl { display:grid; grid-template-columns:minmax(12rem,1fr) 2fr; border-top:1px solid var(--line); }
dt,dd { margin:0; padding:.65rem 0; border-bottom:1px solid var(--line); }
dt { font-weight:700; }
@media (max-width:620px) { dl { display:block; } dd { color:var(--muted); } }
</style>
</head>
<body>
<header>
<p class="eyebrow">OpenSpec · enable-private-a100-sft-smoke-v1</p>
<h1>真实训练之前，先把边界变成可证明的</h1>
<p class="lede">本阶段只允许一次 Qwen2.5-7B-Instruct LoRA SFT 基础设施 smoke：formal public sample 的 train split、最多 2 条、恰好 1 optimizer step、单张显式 GPU。</p>
<p class="status">FINAL_REVIEW_READY · PREFLIGHT_BLOCKED · A100_UNREACHABLE · REAL_SMOKE_NOT_RUN</p>
</header>
<main>
<section>
<h2>为什么要修</h2>
<p>旧路径可能在未配置输出根、未规范化路径或阻断结果退出码为 0 的情况下继续。新合同把输出目录、共享 preflight、CLI 退出码和 7B 本地加载串成同一个 fail-closed 门。</p>
</section>
<section>
<h2>本阶段交付</h2>
<ul>
<li>绝对、已存在、非 symlink 的 output root；候选必须是规范化后的全新严格子目录，任何已存在候选（包括空目录）都拒绝。runner 排他创建并再次校验路径 hash。</li>
<li><code>sft-preflight</code> 与真实 run 共用同一 core；core 同时产生 public report 与 immutable private context。真实 run 只消费已绑定 config、manifest、SFT、rows、model inventory 和 output facts，并在加载前重新 hash。</li>
<li>真实 run 在任何 legacy config/metadata/dataset 读取前先进入 exception-safe shared core；阻断结果不写 candidate，ready metadata 只来自 report/context，并绑定模型 probe 已验证的同一份 facts。</li>
<li>稳定 blocker code、单一 stdout JSON、真实非成功退出码。</li>
<li><code>max_seq_length</code> 只接受 1–4096 的非布尔整数；public config facts 使用白名单。共享 core 的意外异常只返回 <code>PREFLIGHT_INTERNAL_ERROR</code>，不回显私有值或原始异常。</li>
<li>所有真实训练预算采用严格 JSON 整数语义：rows 只能是整数 1/2，step、batch、accumulation 只能是整数 1；<code>seed</code> 必须为整数，<code>logging_steps</code> 必须为正整数。<code>true</code> 与 <code>1.0</code> 不再被隐式转换。</li>
<li>严格 A100 与 Qwen2.5-7B geometry/weight inventory、BF16、单 GPU、1 step、1–2 rows 的 LoRA 参数与审计 metadata；成功还要求 finite non-bool loss、adapter config+weights，并扫描整个 run tree 排除完整模型权重。</li>
<li>DPO real mode 从调用方提供的 manifest checkout 推导 repo root，不依赖进程 cwd；无法确认 checkout 时在 config/metadata/dependency/runner 之前 fail closed。本阶段仍不授权或执行 DPO。</li>
</ul>
</section>
<section>
<h2>硬停止</h2>
<p class="callout">不会自动完整训练，不会运行 DPO、GRPO、prediction、evaluation，不访问 lockbox，不声明模型改善，不提交私有 config、adapter、日志、缓存或私有路径。</p>
</section>
<section>
<h2>clean-evaluation 事实不变</h2>
<dl>
<dt>acquisition_source_status</dt><dd><code>UNAVAILABLE</code></dd>
<dt>authoritatively_bound_binding_count</dt><dd><code>0</code></dd>
<dt>human_acceptance_status</dt><dd><code>NOT_RECORDED</code></dd>
<dt>protocol_freeze_status</dt><dd><code>NOT_FROZEN</code></dd>
<dt>clean_population_status</dt><dd><code>NOT_MATERIALIZED</code></dd>
<dt>freeze_authorized</dt><dd><code>false</code></dd>
<dt>execution_readiness</dt><dd><code>false</code></dd>
</dl>
</section>
<section>
<h2>执行门</h2>
<p>只有 ignored private config 已存在、共享 preflight 返回退出码 0 且 <code>ready=true</code>、并确认一张空闲 A100 后，才允许恰好一次真实 smoke。任一条件缺失都必须以 blocker 停止，不得用 fixture、mock、dry-run 或小模型替代成功。</p>
</section>
<section>
<h2>验证结果</h2>
<ul>
<li>最终 fresh reviewer module：<code>123 passed</code>；Reviewer 最终合并聚焦回归：<code>214 passed</code>，Final Verdict 为 <code>Pass</code>，无 Must Fix 或 Should Fix。</li>
<li>提交前独立全量 pytest：<code>1381 passed, 4 failed</code>。四个失败均为 active OpenSpec change 触发的既有生命周期/冻结边界 guard；未增加永久 allowlist，也未绕过。</li>
<li>提交前独立 ruff、严格 OpenSpec validation（<code>16 passed, 0 failed</code>）和 <code>git diff --check</code> 通过。</li>
<li>用户指定 targeted mypy 命令只报告两个既有 imported-file diagnostic（<code>slot_error_analysis.py:574</code>、<code>copy_backed_shadow_interface.py:357</code>）；本次触及的 <code>training.py</code> 与 <code>cli/train.py</code> 没有诊断。</li>
<li>truth-surface check 仅因本 change 仍处于 active 生命周期而非绿；该状态保留为真实结果。</li>
</ul>
</section>
<section>
<h2>真实执行状态</h2>
<p class="callout">ignored private config 不存在。clean-HEAD 上实际 <code>sft-preflight</code> 退出码为 1，stdout 为单个 JSON document、stderr 为 0 字节，唯一 blocker 为 <code>CONFIG_FILE_MISSING</code>。既有授权 A100 alias 的只读占用探测另行重试三次，均连接超时，因此没有取得 GPU 或本地 7B 模型事实，也没有执行真实 1-step smoke。</p>
<p>没有 adapter、训练日志、缓存、prediction 或 evaluation 产物进入 Git；没有完整训练、DPO、GRPO、lockbox 访问或模型改进声明。</p>
</section>
</main>
<footer><p>更新时间：2026-07-13 · 当前为 final-review-ready 的代码交付；共享 preflight 的唯一 blocker 为 <code>CONFIG_FILE_MISSING</code>，A100 alias 同时不可达。</p></footer>
</body>
</html>
10 changes: 10 additions & 0 deletions docs/human-briefs/2026-07-14-enable-private-a100-sft-smoke-v1.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="zh-CN"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Private A100 SFT Smoke V1 · Archive Closeout</title><style>body{max-width:760px;margin:4rem auto;padding:0 1rem;font:16px/1.7 system-ui;color:#17202a}h1{line-height:1.15}.status{color:#166534;font-weight:700}code{background:#f3f4f6;padding:.1em .3em}.callout{border-left:4px solid #9a3412;padding:1rem;background:#fff7ed}</style></head>
<body><p>OpenSpec · enable-private-a100-sft-smoke-v1</p><h1>审阅收口：把真实更新证据和运行时边界做成硬门</h1><p class="status">ARCHIVED_CODE_GREEN · REAL_SMOKE_NOT_RUN · execution_readiness=false</p>
<p>本轮只修复审阅指出的五项：public stdout 严格白名单、descriptor-relative 输出 claim、空闲显存与占用门、offset 驱动的 assistant-only 标签验证、adapter 参数真实变化证据。</p>
<p class="callout">不执行完整训练、DPO、GRPO、prediction、evaluation 或 lockbox 访问；未通过所有共享 preflight 与 postconditions 时不得使用 <code>SMOKE_COMPLETED</code>。</p>
<p>clean-evaluation 保持：acquisition unavailable、binding count 0、human acceptance not recorded、protocol not frozen、clean population not materialized、freeze unauthorized、execution readiness false。</p>
<p class="callout">输出 claim 的威胁模型已由用户明确收窄：代码继续防御所有检查点可观察的 drift、symlink、inode exchange 与 concurrent leaf；不声称最后一次 identity checkpoint 到 <code>mkdirat</code> syscall 之间对同 UID 恶意 rename 具备内核原子性。部署前必须保证该窗口内 output root/parent namespace 不被同 UID 对手恶意改名。</p>
<p>本轮新回归先得到 <code>17 failed</code>；最终 focused suite 为 <code>203 passed</code>，完整 pytest 为 <code>1413 passed</code>。Ruff、OpenSpec strict（15/15）、truth-surface、git diff check 均通过。Targeted mypy 仅保留两个未触及文件中的既有诊断，本次训练文件零诊断。</p>
<p>主 <code>supervised-contract-tuning</code> spec 已同步，change 已归档到 <code>openspec/changes/archive/2026-07-14-enable-private-a100-sft-smoke-v1/</code>，active changes 为 0。未运行真实训练。</p>
</body></html>
26 changes: 26 additions & 0 deletions docs/human-briefs/2026-07-14-fix-container-pid-aware-sft-gpu-occupancy-v1.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Container PID-aware SFT GPU Occupancy · 人类简报</title>
<style>
body { max-width: 760px; margin: 4rem auto; padding: 0 1rem; color: #17202a; font: 16px/1.7 system-ui; }
h1 { line-height: 1.15; }
.status { color: #9a3412; font-weight: 700; }
.callout { border-left: 4px solid #9a3412; padding: 1rem; background: #fff7ed; }
code { background: #f3f4f6; padding: .1em .3em; }
</style>
</head>
<body>
<p>OpenSpec · fix-container-pid-aware-sft-gpu-occupancy-v1</p>
<h1>让空闲 GPU 检查不再依赖容器 PID 映射</h1>
<p class="status">LOCAL_IMPLEMENTATION · FOCUSED_GREEN · REMOTE_NOT_RUN</p>
<p>父进程先确认显式选择的 GPU 没有 compute process，再以 <code>sys.executable -I</code>、绝对 helper 脚本和固定最小环境启动短生命周期内部 Python helper，采集 CUDA、A100、BF16、总显存和可用显存事实。helper 不继承 proxy、token、<code>PYTHONPATH</code> 或动态加载注入变量，本身不预建 socket，并在导入 torch 前安装阻断 CPython 真实 socket creation 与网络事件的 audit hook。helper 退出并释放自己的 CUDA context 后，父进程再次确认 occupancy 为空。</p>
<p>每次 shared preflight 都完整重复 <code>occupancy → helper → occupancy</code>，不推断、不学习、不缓存或公开任何 PID。任一已确认占用返回 <code>GPU_BUSY</code>；采样、helper timeout/nonzero 或协议异常返回 <code>GPU_OCCUPANCY_PROBE_FAILED</code>。</p>
<p>该合同只证明 pre/post 两个 occupancy 样本为空，并记录 helper 期间的可用显存；不声称两个样本之间持续绝对没有外部进程。</p>
<p>torch 导入失败或 CUDA 不可用返回 <code>CUDA_UNAVAILABLE</code>；CUDA API 异常返回 <code>CUDA_PROBE_FAILED</code>。helper 只输出一个严格 typed JSON，所有失败均不包含原始异常或私有运行值。</p>
<p class="callout">本阶段没有运行真实训练、完整 SFT、DPO、GRPO、prediction 或 evaluation；没有访问 lockbox，没有远端操作，也没有模型改进、clean-evaluation ready 或 production-ready 声明。</p>
<p>clean-evaluation 七项事实保持阻断，<code>execution_readiness=false</code>。</p>
</body>
</html>
60 changes: 60 additions & 0 deletions docs/human-briefs/2026-07-15-fix-real-sft-cli-single-json-v1.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>真实 SFT CLI 单一 JSON 修复 · Human Brief</title>
<style>
:root { color-scheme: light; --ink:#17202a; --muted:#475569; --line:#cbd5e1; --warn:#9a3412; --panel:#fff7ed; }
body { max-width:760px; margin:4rem auto; padding:0 1rem; font:16px/1.7 system-ui,sans-serif; color:var(--ink); }
h1 { line-height:1.2; } h2 { margin-top:2rem; } code { background:#f1f5f9; padding:.1em .3em; overflow-wrap:anywhere; }
.status { color:var(--warn); font-weight:700; } .callout { border-left:4px solid var(--warn); padding:1rem; background:var(--panel); }
a { color:#1d4ed8; } ul { padding-left:1.3rem; } footer { border-top:1px solid var(--line); margin-top:2rem; padding-top:1rem; color:var(--muted); }
</style>
</head>
<body>
<p>OpenSpec · fix-real-sft-cli-single-json-v1</p>
<h1>真实 SFT CLI stdout 单一 JSON 修复</h1>
<p class="status">ARCHIVED_CODE_GREEN · PRIOR_SMOKE_RUNTIME_POSTCONDITION_FAILED · SECOND_SMOKE_NOT_AUTHORIZED · execution_readiness=false</p>

<p class="callout">第一次获授权的真实 1-step SFT 已证明 adapter 参数发生有限、非零且有限值的更新，但进程 stdout 在最终 JSON 前包含 Trainer 的两行 Python 字典输出，因此完整 stdout 不是单一 JSON document。该运行不得对外描述为 <code>SMOKE_COMPLETED</code>，也不得据此进入阶段二。</p>

<h2>本轮边界</h2>
<ul>
<li>只修复 CLI 输出隔离与回归测试；不修改模型、数据、训练预算、preflight 或 adapter 证据。</li>
<li>统一 backend dispatch 内的 Python stdout 被逐行加 <code>voice2task-backend: </code> 前缀并流式转发到 stderr；最终成功或失败 JSON 保留在隔离范围外。</li>
<li>不再次运行真实 smoke，不运行完整 SFT、DPO、GRPO、prediction、evaluation，不访问 test、lockbox 或 clean-evaluation population。</li>
</ul>

<h2>TDD 与当前证据</h2>
<ul>
<li>RED：3 个回归按预期失败，分别复现 noisy SFT success、noisy runtime exception、mock-only DPO dispatch 的 stdout 污染。</li>
<li>GREEN：3 个新增回归通过；A100 review-hardening 与 private-preflight 组合回归为 154 passed。</li>
<li>独立只读审阅：Pass，Must Fix 0，Should Fix 0；pre-archive 完整测试为 1450 passed，另有 4 个只由 active change 触发的既有生命周期门失败，未增加 allowlist。</li>
<li>Ruff、strict OpenSpec 16/16、formal public data 696 SFT / 2100 DPO、88 个数据与 schema 回归、public leak scan 和 diff check 均通过；targeted Mypy 仅保留两个未触及 imported module 的既有诊断。</li>
<li>归档后：active changes=0，完整 pytest 1454 passed，strict OpenSpec 15/15，truth-surface、5/5 本地链接、private ignore/untracked/status guard 与 Stop hook 双路径 exit 0 均通过。</li>
<li>测试不下载模型、不要求 GPU，也不执行训练或 DPO。</li>
</ul>

<h2>不变的 clean-evaluation 事实</h2>
<ul>
<li><code>acquisition_source_status=UNAVAILABLE</code></li>
<li><code>authoritatively_bound_binding_count=0</code></li>
<li><code>human_acceptance_status=NOT_RECORDED</code></li>
<li><code>protocol_freeze_status=NOT_FROZEN</code></li>
<li><code>clean_population_status=NOT_MATERIALIZED</code></li>
<li><code>freeze_authorized=false</code>；<code>execution_readiness=false</code></li>
</ul>

<h2>审阅入口</h2>
<ul>
<li><a href="../../openspec/changes/archive/2026-07-15-fix-real-sft-cli-single-json-v1/proposal.md">Proposal</a></li>
<li><a href="../../openspec/changes/archive/2026-07-15-fix-real-sft-cli-single-json-v1/design.md">Design</a></li>
<li><a href="../../openspec/changes/archive/2026-07-15-fix-real-sft-cli-single-json-v1/specs/supervised-contract-tuning/spec.md">Delta spec</a></li>
<li><a href="../../openspec/changes/archive/2026-07-15-fix-real-sft-cli-single-json-v1/tasks.md">Tasks</a></li>
<li><a href="../../openspec/specs/supervised-contract-tuning/spec.md">Current main spec</a></li>
</ul>

<footer>本 bounded code fix 已审阅并归档，但先前真实 smoke 仍是 Runtime Postcondition Failed。任何第二次真实 smoke 都需要新的明确授权；本 brief 不构成训练、评测、发布、生产就绪或模型改进声明。</footer>
</body>
</html>
Loading